diff --git a/docs/browser-observation.md b/docs/browser-observation.md index 042afabfc9..9bd8527919 100644 --- a/docs/browser-observation.md +++ b/docs/browser-observation.md @@ -54,6 +54,26 @@ from a protocol omission. Rejected DOM facts also mark derived URL and agent-hin metadata incomplete, while valid facts from other nodes and documents remain available in the capture. +Passive comparison keeps lookup identity, cross-capture correspondence, content +evidence, and inventory coverage separate: + +| Observation | Comparison outcome | +| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | +| Renamed WebMCP tool keys/frame IDs with one matching name, owner origin, and frame URL | Compare the registration content; local IDs do not prove change | +| Duplicate WebMCP owners with the same name, origin, and frame URL | Unknown correspondence; complete group count differences still prove change | +| Matching script inventory keys with retained source digests | Compare the retained SHA-256 values even without a producer hash | +| One corresponding script source is unavailable, or neither source nor producer hash is available | Source equality remains unknown; inventory changes and other proven differences survive | +| A partial HTTP response group is a subset of the complete peer | Unknown; omitted repeated responses can explain the difference | +| An observed HTTP response fingerprint is absent from the complete peer group | Changed; omissions cannot explain that retained response | +| A WebSocket connection or frame inventory is incomplete | Do not align retained payloads by position | + +Storage fingerprints decode supported CDP primitive/JSON values before hashing. +Debugger descriptions, previews, and remote references do not represent storage +content. Actual JSON properties with those names remain content. Reference-only +objects, unsupported values, and opaque cookie partition identities remain +incomplete. Previously exported fingerprints are not recomputed; recapture both +sides when comparing captures produced across this decoder change. + - `capture_web_screenshot` / `rea capture-web-screenshot` returns a content-addressed visible-viewport PNG. - `compare_web_screenshots` / `rea compare-web-screenshots` performs bounded local PNG pixel comparison without OCR or external services. - The CLI comparison commands accept each capture, screenshot artifact or normalization policy as inline JSON or a JSON file path. Use files for screenshot artifacts and scenario captures, whose embedded PNG bytes can exceed the host's command-line length limit. @@ -356,8 +376,8 @@ REA retains local observation data and removes only URL userinfo credentials, ex request additions and removals remain observable even when selected body shapes, response metadata, or WebSocket coverage are unavailable. HTTP request and WebSocket inventory coverage are evaluated separately; complete HTTP coverage - cannot prove a missing WebSocket connection. Limited request inventory still - prevents request-absence claims. This comparison rule does not make live CDP's + cannot prove a missing WebSocket connection. A missing request in a limited inventory cannot prove absence; a retained + request absent from a complete peer can prove a change. This comparison rule does not make live CDP's attach-limited inventory complete or establish body-shape equality. An unavailable HTTP resource type also leaves request identity incomplete: observations with the same method and URL remain possible matches across types, @@ -372,11 +392,15 @@ REA retains local observation data and removes only URL userinfo credentials, ex uses `websocket_connections` and `websocket_frames`. Drops in other event kinds and the aggregate total do not by themselves limit HTTP inventory. Proven modifications remain reportable despite incomplete inventory. - Repeated identical HTTP content samples establish multiplicity changes only - with complete request inventories; limited inventories retain known content - comparisons without interpreting duplicate sample counts as changes. + Repeated HTTP samples retain multiplicity. A partial group that is contained + in a complete peer cannot prove modification, even when the omitted response + had different content. A retained sample absent from the complete peer can + prove modification. Two partial groups cannot establish response + correspondence from their different retained samples alone. WebSocket stream inventories and connection counts likewise require complete - WebSocket coverage. Redirect resource types use only shared known metadata: + WebSocket coverage. Retained frame positions are compared only with complete + connection/frame inventories; matching direction/opcode sequences alone do + not prove alignment after drops. Redirect resource types use only shared known metadata: an omitted type leaves equality unknown rather than proving a modification. - Console observations with a stack source retain call type, argument types, the CDP `Runtime.Timestamp` in milliseconds since Unix epoch, and source location. An absent or invalid timestamp is `null` and makes `console_events` incomplete; an observed zero remains zero. Argument types are listed by CDP argument position. An absent or non-array argument inventory is `null`, and a malformed argument keeps its position as `unknown`; either makes `console_events` incomplete, while an observed empty inventory remains `[]`. A malformed stack or top stack frame is not replaced by its caller: the console event is excluded, and a request initiator location or WebMCP `registration_source` is `null`; each is reported as `invalid_protocol_value`. This epoch-millisecond clock is separate from the monotonic-second timestamps in `observe-web-session`. When console text capture is selected, already-delivered primitive values are retained verbatim with their original `argument_index`; objects, getters, and remote properties are never expanded. Text capture is `unavailable` when the argument inventory is unknown. - WebSocket observations retain direction, opcode, and payload byte length. A missing or malformed CDP opcode is returned as `null`, with opcode-dependent payload byte length and shape also `null`, and marks `websocket_frames` incomplete. Missing or wrong-typed payload data is not treated as empty; its byte length is `null`, and a text payload shape is unknown. Numeric opcode `0` and an observed empty payload remain distinct from missing data. Opcode is nullable in the public inspection schema, so clients must handle unknown protocol values. When shape capture is selected, text messages are classified as text or complete value-free JSON shape; binary bytes, hashes, prefixes, and raw payloads are never retained. diff --git a/docs/tool-design.md b/docs/tool-design.md index e2563137a8..d36be85c60 100644 --- a/docs/tool-design.md +++ b/docs/tool-design.md @@ -80,6 +80,24 @@ satisfy. A plan without execution does not establish runtime behavior. ### Interpret boundary facts once +Comparison needs separate proofs for lookup, correspondence, content, and +coverage. A hash cannot supply the proof that two retained values belong to +corresponding objects. + +| Comparison fact | What it establishes | What remains unknown | +| ----------------------------------------------------------------------------- | ----------------------------------------------- | --------------------------------------------- | +| Same capture-local frame or request ID | Lookup within that capture | Cross-capture identity | +| Same HTTP method, URL, and resource type | A repeated-request group | Which individual responses correspond | +| A partial HTTP response multiset is contained in a complete peer | Omissions can explain every retained sample | Equality or modification | +| A retained HTTP sample is absent from a complete peer group | An observed difference independent of omissions | Other unobserved differences | +| Complete WebSocket connection/frame inventories and matching metadata streams | Ordered payload positions can be compared | Equality of unavailable payload facts | +| Unique WebMCP name, owner origin, and frame URL | Tool correspondence without transient frame IDs | Correspondence between duplicate owners | +| A complete source digest or decoded storage value | Comparable content | Missing source bytes or opaque remote objects | + +Complete traversal or inventory does not make unavailable values equal. A +partial aggregate fingerprint difference cannot establish modification when +omissions alone explain it; independently proven differences still survive. + The format or protocol reader owns interpretation. Pass its parsed facts and explicit uncertainty to consumers; do not make adapters reparse display text, reconstruct omitted values, or infer success from missing diagnostics. diff --git a/src/browser/CdpCaptureStorageFingerprints.ts b/src/browser/CdpCaptureStorageFingerprints.ts index 538db86c23..169dc62e76 100644 --- a/src/browser/CdpCaptureStorageFingerprints.ts +++ b/src/browser/CdpCaptureStorageFingerprints.ts @@ -157,7 +157,7 @@ const addCookies = async ( partition_key: cookie.partitionKey ?? null, }, value, - complete: true, + complete: cookie.partitionKeyOpaque !== true, }); } }; @@ -300,12 +300,9 @@ const addIndexedDbRecords = async ( if (entries.length !== result.objectStoreDataEntries.length) run.state.complete = false; for (const entry of entries) { - const key = stableValue(entry.key); - const primaryKey = stableValue(entry.primaryKey); - // A missing remote object is an omission, not the JavaScript value - // "undefined". Keep the unavailable digest null and coverage partial. - const value = - entry.value === undefined ? undefined : stableValue(entry.value); + const key = decodeRemoteObject(entry.key); + const primaryKey = decodeRemoteObject(entry.primaryKey); + const value = decodeRemoteObject(entry.value); addFingerprint(run.state, { scope: "indexed_db_record", identity: { @@ -316,9 +313,7 @@ const addIndexedDbRecords = async ( }, value, complete: - remoteObjectComplete(entry.key) && - remoteObjectComplete(entry.primaryKey) && - remoteObjectComplete(entry.value), + key !== undefined && primaryKey !== undefined && value !== undefined, }); } if (result.hasMore !== true) return; @@ -470,7 +465,6 @@ const stableValue = (value: unknown): unknown => { if (record === undefined) return String(value); return Object.fromEntries( Object.entries(record) - .filter(([key]) => key !== "objectId" && key !== "customPreview") .sort(([left], [right]) => compareUnicodeCodePoints(left, right)) .map(([key, item]) => [key, stableValue(item)]), ); @@ -490,55 +484,56 @@ const stableValueComplete = (value: unknown): boolean => { return value.every((item) => stableValueComplete(item)); const record = recordValue(value); if (record === undefined) return false; - const entries = Object.entries(record).filter( - ([key]) => key !== "objectId" && key !== "customPreview", - ); + const entries = Object.entries(record); return entries.every(([, item]) => stableValueComplete(item)); }; -const remoteObjectComplete = (value: unknown): boolean => { +/** Decode CDP value evidence once; debugger labels/references are not content. + * https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#type-RemoteObject + */ +const decodeRemoteObject = (value: unknown): unknown => { const record = recordValue(value); - if ( - record === undefined || - record.objectId !== undefined || - record.preview !== undefined || - record.customPreview !== undefined || - !stableValueComplete(record) - ) - return false; - switch (record.type) { + if (record === undefined) return undefined; + const { type, unserializableValue } = record; + const serialized = record.value; + const primitive = (valid: boolean): unknown => + valid ? { type, value: serialized } : undefined; + switch (type) { case "undefined": - return ( - record.value === undefined && record.unserializableValue === undefined - ); + return serialized === undefined && unserializableValue === undefined + ? { type } + : undefined; case "string": case "boolean": - return ( - typeof record.value === record.type && - record.unserializableValue === undefined + return primitive( + typeof serialized === type && unserializableValue === undefined, ); case "number": - return record.unserializableValue === undefined - ? typeof record.value === "number" && Number.isFinite(record.value) - : record.value === undefined && - typeof record.unserializableValue === "string" && - ["-0", "NaN", "Infinity", "-Infinity"].includes( - record.unserializableValue, - ); + if (unserializableValue === undefined) + return primitive( + typeof serialized === "number" && Number.isFinite(serialized), + ); + return serialized === undefined && + typeof unserializableValue === "string" && + ["-0", "NaN", "Infinity", "-Infinity"].includes(unserializableValue) + ? { type, unserializableValue } + : undefined; case "bigint": - return ( - record.value === undefined && - typeof record.unserializableValue === "string" && - /^-?\d+n$/u.test(record.unserializableValue) - ); + return serialized === undefined && + typeof unserializableValue === "string" && + /^-?\d+n$/u.test(unserializableValue) + ? { type, unserializableValue } + : undefined; case "object": - return ( - record.unserializableValue === undefined && - (record.subtype === "null" - ? record.value === null - : record.value !== null && typeof record.value === "object") + return primitive( + unserializableValue === undefined && + (record.subtype === "null" + ? serialized === null + : serialized !== null && + typeof serialized === "object" && + stableValueComplete(serialized)), ); default: - return false; + return undefined; } }; diff --git a/src/domain/webCaptureDiff.network.test.ts b/src/domain/webCaptureDiff.network.test.ts index a736414f74..a2f4970d1f 100644 --- a/src/domain/webCaptureDiff.network.test.ts +++ b/src/domain/webCaptureDiff.network.test.ts @@ -298,7 +298,7 @@ describe("passive network coverage uncertainty", () => { (section) => { const before = passiveNetworkCapture(); const after = passiveNetworkCapture(["/a", "/b"]); - after.completeness[section].push("network_requests"); + before.completeness[section].push("network_requests"); for (const [left, right] of [ [before, after], [after, before], @@ -625,6 +625,15 @@ describe("passive dropped-event inventory coverage", () => { }); const changed = structuredClone(before); changed.network.requests[0]!.status = 503; + expect(compare(before, changed)).toMatchObject({ + status: "unknown", + total_changes: 0, + changes: [], + }); + // Once one complete inventory excludes the peer's observed response, + // omissions cannot explain that response difference. + changed.completeness.dropped_events.network_requests = 0; + changed.completeness.dropped_events.total = 0; expect(compare(before, changed)).toMatchObject({ status: "changed", total_changes: 1, diff --git a/src/domain/webCaptureDiff.ts b/src/domain/webCaptureDiff.ts index 7bad18e855..1e8138b384 100644 --- a/src/domain/webCaptureDiff.ts +++ b/src/domain/webCaptureDiff.ts @@ -44,7 +44,7 @@ export const compareWebCaptures = ( limitations: [ "A changed status proves an observed difference; an unknown status means necessary comparison facts or coverage were unavailable.", "Network comparison covers only activity observed after each CDP attachment. WebSocket connections with the same URL are compared as an unordered collection of ordered frame streams, retaining connection counts; their individual identities are not stable across captures.", - "Accessibility roles, ignored state, text, and hierarchy are compared only when the accessibility tree was fully captured and text capture was selected and not truncated. A missing role leaves that node's role comparison unknown; independently observed differences can still establish a change.", + "Accessibility structure is compared when the tree is complete; text additionally requires complete selected text capture. A missing role leaves that node's role comparison unknown; independently observed differences can still establish a change.", "Storage key inventories are compared only when selected and complete; usage and quota are compared only when reported. Redacted content is compared through complete SHA-256 fingerprints.", ], }); @@ -194,19 +194,7 @@ const compareWebCaptureDimensions = ( const after = input.after.inspection; return { dom_structure: domDimension(before, after), - scripts: compareDimension( - keyed( - before.scripts.items.map(scriptProjection), - (item) => item.script_key, - ), - keyed( - after.scripts.items.map(scriptProjection), - (item) => item.script_key, - ), - sectionsComplete(before, ["scripts"]) && - sectionsComplete(after, ["scripts"]), - "Script inventory was incomplete in at least one observation.", - ), + scripts: scriptsDimension(before, after), resources: compareResources( before.resources, after.resources, @@ -472,6 +460,12 @@ const networkDimension = ( const requestsComplete = [before, after].every((inspection) => networkInventoryComplete(inspection, ["network_requests"]), ); + const webSocketFramesComplete = [before, after].every((inspection) => + networkInventoryComplete(inspection, [ + "websocket_connections", + "websocket_frames", + ]), + ); const webSocketInventoryComplete = [before, after].every( (inspection) => networkInventoryComplete(inspection, [ @@ -487,7 +481,9 @@ const networkDimension = ( const projection = { metadataFields: metadata.fields, ambiguousIdentities: requestIdentities.ambiguousIdentities, - inventoryComplete: requestsComplete, + initiatorsComplete: [before, after].every((inspection) => + sectionsComplete(inspection, ["network_initiators"]), + ), }; const sources = new Map(); if (selected) { @@ -545,17 +541,24 @@ const networkDimension = ( beforeWebSockets, afterWebSockets, webSocketShapes, + webSocketFramesComplete, + ); + const [beforeRequests, afterRequests] = networkComparisonMaps( + networkMap(before, sources, projection), + networkMap(after, sources, projection), + networkInventoryComplete(before, ["network_requests"]), + networkInventoryComplete(after, ["network_requests"]), ); const compared = compareDimension( new Map([ - ...networkMap(before, sources, projection), + ...beforeRequests, ...(webSocketInventoryComplete ? webSocketMetadataMap(beforeWebSockets) : []), ...beforeWebSocketPayload, ]), new Map([ - ...networkMap(after, sources, projection), + ...afterRequests, ...(webSocketInventoryComplete ? webSocketMetadataMap(afterWebSockets) : []), @@ -563,6 +566,7 @@ const networkDimension = ( ]), requestsComplete && requestIdentities.complete && + projection.initiatorsComplete && webSocketInventoryComplete && shapesComplete && webSocketPayloadBytesComplete && @@ -570,11 +574,11 @@ const networkDimension = ( requestIdentities.complete ? "Network capture or selected HTTP/WebSocket payload-shape coverage is limited, unavailable, incomplete, or includes dropped network events." : "HTTP request identity is incomplete because a resource type is unavailable; potentially matching type groups cannot establish membership/content changes or complete network equality.", - // HTTP and WebSocket identities have independent inventory coverage. Shape - // selection can prevent equality without erasing proven membership changes. + // HTTP maps already retain only directionally proven membership/content + // changes. WebSocket inventory uses its own coverage independently. (identity) => identity.startsWith("net_") - ? requestsComplete + ? true : identity.startsWith("ws_") && webSocketInventoryComplete, ); if (metadata.complete) return compared; @@ -601,9 +605,9 @@ const networkMap = ( ReadonlySet >; readonly ambiguousIdentities: ReadonlySet; - readonly inventoryComplete: boolean; + readonly initiatorsComplete: boolean; }, -): ReadonlyMap => { +): ReadonlyMap => { const grouped = new Map(); for (const request of inspection.network.requests) { const identity = networkIdentity(request); @@ -642,7 +646,9 @@ const networkMap = ( }; }, ), - initiator: request.initiator, + ...(projection.initiatorsComplete + ? { initiator: request.initiator } + : {}), ...(shapes?.request === true ? { request_shape: request.body_shapes.request } : {}), @@ -653,16 +659,48 @@ const networkMap = ( ); grouped.set(identity, values); } - return new Map( - [...grouped].map(([identity, values]) => [ - identity, - digestCanonicalValue( - // Repeated samples do not establish multiplicity across omissions; - // keep their known content without interpreting missing rows as change. - (projection.inventoryComplete ? values : [...new Set(values)]).sort(), - ), - ]), - ); + return grouped; +}; + +/** A partial multiset may omit any sample, including a distinct response. */ +const networkComparisonMaps = ( + before: ReadonlyMap, + after: ReadonlyMap, + beforeComplete: boolean, + afterComplete: boolean, +): readonly [ReadonlyMap, ReadonlyMap] => { + const left = new Map(); + const right = new Map(); + for (const identity of new Set([...before.keys(), ...after.keys()])) { + const previous = before.get(identity) ?? []; + const next = after.get(identity) ?? []; + if ( + !(beforeComplete && afterComplete) && + !(beforeComplete && !networkSamplesContained(next, previous)) && + !(afterComplete && !networkSamplesContained(previous, next)) + ) + continue; + if (before.has(identity)) + left.set(identity, digestCanonicalValue([...previous].sort())); + if (after.has(identity)) + right.set(identity, digestCanonicalValue([...next].sort())); + } + return [left, right]; +}; + +const networkSamplesContained = ( + samples: readonly string[], + complete: readonly string[], +): boolean => { + const counts = new Map(); + for (const sample of complete) + counts.set(sample, (counts.get(sample) ?? 0) + 1); + for (const sample of samples) { + const count = counts.get(sample) ?? 0; + if (count === 0) return false; + counts.set(sample, count - 1); + } + return true; }; type WebSocketGroup = { @@ -717,9 +755,13 @@ const webSocketPayloadMaps = ( beforeGroups: ReadonlyMap, afterGroups: ReadonlyMap, selected: boolean, + inventoryComplete: boolean, ): readonly [ReadonlyMap, ReadonlyMap] => { const left = new Map(); const right = new Map(); + // Retained positions cannot identify frames after any frame or connection + // omission. A matching direction/opcode sequence does not prove alignment. + if (!inventoryComplete) return [left, right]; for (const [groupKey, beforeGroup] of beforeGroups) { const afterGroup = afterGroups.get(groupKey); if ( @@ -783,46 +825,81 @@ const webMcpDimension = ( before: WebMcpDiscovery | null, after: WebMcpDiscovery | null, ): Dimension => { - const afterTools = new Map( - after?.tools.items.map((tool) => [tool.tool_key, tool]), - ); - const unknownHints = new Set( - (before?.tools.items ?? []) - .filter((tool) => { - const other = afterTools.get(tool.tool_key); - return ( - other !== undefined && - (tool.annotations.consequential === undefined || - other.annotations.consequential === undefined) - ); - }) - .map((tool) => tool.tool_key), - ); - const project = ( - discovery: WebMcpDiscovery | null, - ): ReadonlyMap => - keyed( - (discovery?.tools.items ?? []).map((tool) => { - if (!unknownHints.has(tool.tool_key)) return tool; - const { consequential: _unknown, ...annotations } = tool.annotations; - return { ...tool, annotations }; - }), - (tool) => tool.tool_key, - ); - const result = compareDimension( - project(before), - project(after), - webMcpComplete(before) && webMcpComplete(after), - "WebMCP discovery was unavailable or incomplete in at least one capture.", - ); - return result.status === "unchanged" && unknownHints.size > 0 - ? { - ...result, - status: "unknown", - reason: - "Consequential annotations were not recorded or were invalid in at least one capture.", + type Tool = WebMcpDiscovery["tools"]["items"][number]; + const groups = (discovery: WebMcpDiscovery | null) => { + const result = new Map(); + for (const tool of discovery?.tools.items ?? []) { + const identity = `webmcp_${digestCanonicalValue({ + name: tool.name, + frame_url: tool.frame_url, + owner_origin: tool.owner_origin, + })}`; + const tools = result.get(identity) ?? []; + tools.push(tool); + result.set(identity, tools); + } + return result; + }; + const previous = groups(before); + const next = groups(after); + const inventoryComplete = webMcpComplete(before) && webMcpComplete(after); + const left = new Map(); + const right = new Map(); + let valuesComplete = true; + for (const identity of new Set([...previous.keys(), ...next.keys()])) { + const beforeTools = previous.get(identity) ?? []; + const afterTools = next.get(identity) ?? []; + // Multiple same-name owners at the same URL cannot be paired by frame ID. + // Keep this uncertainty rather than selecting a duplicate by capture order. + if (beforeTools.length > 1 || afterTools.length > 1) { + valuesComplete = false; + // Complete counts can contradict one another even when individual + // owners cannot be paired. Do not discard that independent proof. + if (inventoryComplete) { + if (beforeTools.length > 0) + left.set(identity, String(beforeTools.length)); + if (afterTools.length > 0) + right.set(identity, String(afterTools.length)); } - : result; + continue; + } + const beforeTool = beforeTools[0]; + const afterTool = afterTools[0]; + const consequentialKnown = + beforeTool?.annotations.consequential !== undefined && + afterTool?.annotations.consequential !== undefined; + if ( + beforeTool !== undefined && + afterTool !== undefined && + !consequentialKnown + ) + valuesComplete = false; + const project = (tool: Tool) => { + const { + tool_key: _key, + frame_id: _frameId, + annotations, + ...content + } = tool; + const { consequential, ...otherAnnotations } = annotations; + return digestCanonicalValue({ + ...content, + annotations: { + ...otherAnnotations, + ...(consequentialKnown ? { consequential } : {}), + }, + }); + }; + if (beforeTool !== undefined) left.set(identity, project(beforeTool)); + if (afterTool !== undefined) right.set(identity, project(afterTool)); + } + return compareDimension( + left, + right, + inventoryComplete && valuesComplete, + "WebMCP discovery, owner correspondence, or consequential annotation evidence was unavailable or incomplete in at least one capture.", + inventoryComplete, + ); }; const webMcpComplete = (discovery: WebMcpDiscovery | null): boolean => @@ -922,6 +999,53 @@ const scriptProjection = ( source_map_url: script.source_map_url, }); +const scriptsDimension = ( + before: WebPageInspection, + after: WebPageInspection, +): Dimension => { + const inventoryComplete = [before, after].every((inspection) => + sectionsComplete(inspection, ["scripts"]), + ); + const byKey = (inspection: WebPageInspection) => + new Map( + inspection.scripts.items.map((script) => [script.script_key, script]), + ); + const left = byKey(before); + const right = byKey(after); + let sourcesComplete = true; + const project = (inspection: WebPageInspection, peer: typeof left) => + keyed( + inspection.scripts.items.map((script) => { + const other = peer.get(script.script_key); + const sourceComparable = + script.source.included && other?.source.included === true; + if ( + other !== undefined && + (((script.source.included || other.source.included) && + !sourceComparable) || + (script.cdp_hash === null && !sourceComparable)) + ) + sourcesComplete = false; + return { + ...scriptProjection(script), + ...(sourceComparable && script.source.included + ? { source_sha256: script.source.artifact.sha256 } + : {}), + }; + }), + (script) => script.script_key, + ); + const previous = project(before, right); + const next = project(after, left); + return compareDimension( + previous, + next, + inventoryComplete && sourcesComplete, + "Script inventory or corresponding source digest evidence was incomplete in at least one observation.", + inventoryComplete, + ); +}; + const metadataProjection = (inspection: WebPageInspection) => ({ responses: inspection.metadata.responses .map(({ request_id: _requestId, ...response }) => response) diff --git a/tests/acceptance/tools/passiveCaptureCorrespondence.test.ts b/tests/acceptance/tools/passiveCaptureCorrespondence.test.ts new file mode 100644 index 0000000000..521bc7d564 --- /dev/null +++ b/tests/acceptance/tools/passiveCaptureCorrespondence.test.ts @@ -0,0 +1,408 @@ +import { writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect } from "vitest"; + +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { createWebTextArtifact } from "../../../src/domain/webContentArtifact.js"; +import type { WebMcpDiscovery } from "../../../src/domain/webMcpDiscovery.js"; +import { + type CompareWebCapturesInput, + type WebCaptureDiff, + webCaptureDiffSchema, +} from "../../../src/domain/webCaptureDiffSchemas.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { + passiveNetworkCapture, + withPassiveDroppedEvents, +} from "../../fixtures/passiveNetworkCapture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { cliTest } from "../../support/cli/cliFixture.js"; + +type Capture = { + inspection: CompareWebCapturesInput["before"]["inspection"]; + webmcp?: WebMcpDiscovery | null; +}; +type Case = { + name: string; + before: Capture; + after: Capture; + dimension: keyof WebCaptureDiff["dimensions"]; + status: "changed" | "unknown" | "unchanged"; +}; +const cases: Case[] = []; +const add = ( + name: string, + before: Capture, + after: Capture, + dimension: Case["dimension"], + status: Case["status"], +) => cases.push({ name, before, after, dimension, status }); +const unselectedCapture = (paths: readonly string[] = []) => { + const capture = passiveNetworkCapture(paths); + capture.completeness.excluded.push({ + section: "json_body_shapes", + reason: "not_approved", + count: null, + }); + for (const request of capture.network.requests) + request.body_shapes.status = "not_approved"; + return capture; +}; +const full = unselectedCapture(["/a", "/a"]); +full.network.requests[1]!.status = 503; +for (const retained of [0, 1]) { + const omitted = structuredClone(full); + omitted.network.requests = [omitted.network.requests[retained]!]; + const partial = withPassiveDroppedEvents(omitted, { network_requests: 1 }); + add( + `heterogeneous HTTP omission ${retained}`, + { inspection: full }, + { inspection: partial }, + "network", + "unknown", + ); + add( + `heterogeneous HTTP omission ${retained} reversed`, + { inspection: partial }, + { inspection: full }, + "network", + "unknown", + ); +} +const changedResponse = structuredClone(full); +const small = unselectedCapture(["/a"]); +const largerPartial = withPassiveDroppedEvents( + unselectedCapture(["/a", "/b"]), + { network_requests: 1 }, +); +add( + "partial inventory retains a request absent from complete peer", + { inspection: small }, + { inspection: largerPartial }, + "network", + "changed", +); +add( + "partial inventory retains a request absent from complete peer reversed", + { inspection: largerPartial }, + { inspection: small }, + "network", + "changed", +); +changedResponse.network.requests[1]!.status = 404; +add( + "complete heterogeneous HTTP change", + { inspection: full }, + { inspection: changedResponse }, + "network", + "changed", +); +const partialChanged = withPassiveDroppedEvents(changedResponse, { + network_requests: 1, +}); +add( + "observed HTTP value absent from complete peer", + { inspection: full }, + { inspection: partialChanged }, + "network", + "changed", +); +add( + "two partial HTTP groups cannot prove correspondence", + { inspection: withPassiveDroppedEvents(full, { network_requests: 1 }) }, + { inspection: partialChanged }, + "network", + "unknown", +); + +const socket = unselectedCapture(); +const initiatorBefore = unselectedCapture(["/a"]); +const initiatorAfter = structuredClone(initiatorBefore); +initiatorAfter.network.requests[0]!.initiator = { + type: "other", + url: null, + line: null, + column: null, +}; +initiatorAfter.completeness.unavailable_sections.push("network_initiators"); +add( + "unavailable initiator is not a changed request", + { inspection: initiatorBefore }, + { inspection: initiatorAfter }, + "network", + "unknown", +); +const responseDespiteInitiator = structuredClone(initiatorAfter); +responseDespiteInitiator.network.requests[0]!.status = 503; +add( + "response change survives unavailable initiator", + { inspection: initiatorBefore }, + { inspection: responseDespiteInitiator }, + "network", + "changed", +); +socket.network.websocket_connections.push({ + request_id: "local-socket", + url: "wss://app.example.test/live", + events: [1, 2, 3].map((bytes) => ({ + direction: "received", + opcode: 1, + payload_bytes: bytes, + payload_shape: { format: "text", json_shape: null }, + })), +}); +for (const index of [0, 1, 2]) { + const omitted = structuredClone(socket); + omitted.network.websocket_connections[0]!.events.splice(index, 1); + const partial = withPassiveDroppedEvents(omitted, { websocket_frames: 1 }); + add( + `WebSocket dropped frame ${index}`, + { inspection: socket }, + { inspection: partial }, + "network", + "unknown", + ); + add( + `WebSocket dropped frame ${index} reversed`, + { inspection: partial }, + { inspection: socket }, + "network", + "unknown", + ); +} +// Same retained positions/directions/opcodes can describe entirely different +// original frames when both captures dropped a frame. +const leftSocket = structuredClone(socket); +leftSocket.network.websocket_connections[0]!.events.shift(); +const rightSocket = structuredClone(socket); +rightSocket.network.websocket_connections[0]!.events.pop(); +add( + "same-length partial WebSocket streams", + { inspection: withPassiveDroppedEvents(leftSocket, { websocket_frames: 1 }) }, + { + inspection: withPassiveDroppedEvents(rightSocket, { websocket_frames: 1 }), + }, + "network", + "unknown", +); +const changedSocket = structuredClone(socket); +changedSocket.network.websocket_connections[0]!.events[1]!.payload_bytes = 4; +add( + "complete WebSocket payload change", + { inspection: socket }, + { inspection: changedSocket }, + "network", + "changed", +); + +const inspection = unselectedCapture(); +const discovery: WebMcpDiscovery = { + browser: inspection.browser, + target: { + target_id: "local-page", + url: inspection.target.url, + origin: inspection.target.origin!, + }, + status: "available", + completeness: structuredClone(inspection.completeness), + limitations: [], + tools: { + total: 1, + items: [ + { + tool_key: `webmcp_${"a".repeat(64)}`, + frame_id: "local-frame-a", + frame_url: inspection.target.url, + owner_origin: inspection.target.origin!, + name: "search", + description: "Search records", + declaration_kind: "imperative", + input_schema: null, + input_schema_sha256: null, + annotations: { + read_only: true, + untrusted_content: true, + consequential: false, + autosubmit: false, + }, + registration_source: null, + trust: "page-declared-untrusted", + }, + ], + }, +}; +const renamed = structuredClone(discovery); +renamed.tools.items[0]!.tool_key = `webmcp_${"b".repeat(64)}`; +renamed.tools.items[0]!.frame_id = "local-frame-b"; +add( + "WebMCP capture-local identity renamed", + { inspection, webmcp: discovery }, + { inspection, webmcp: renamed }, + "webmcp", + "unchanged", +); +const changedTool = structuredClone(renamed); +changedTool.tools.items[0]!.description = "Changed description"; +add( + "WebMCP content change survives local identity rename", + { inspection, webmcp: discovery }, + { inspection, webmcp: changedTool }, + "webmcp", + "changed", +); +const duplicateOwner = structuredClone(discovery); +duplicateOwner.tools.total = 2; +duplicateOwner.tools.items.push({ + ...duplicateOwner.tools.items[0]!, + tool_key: `webmcp_${"c".repeat(64)}`, + frame_id: "duplicate-owner", +}); +add( + "WebMCP complete duplicate owner count changes", + { inspection, webmcp: duplicateOwner }, + { inspection, webmcp: renamed }, + "webmcp", + "changed", +); + +const script = unselectedCapture(); +const duplicateRenamed = structuredClone(duplicateOwner); +duplicateRenamed.tools.items.reverse(); +for (const [index, tool] of duplicateRenamed.tools.items.entries()) { + tool.frame_id = `new-owner-${index}`; + tool.tool_key = `webmcp_${String(index + 1).repeat(64)}`; +} +add( + "equal-count WebMCP duplicate owners stay unresolved", + { inspection, webmcp: duplicateOwner }, + { inspection, webmcp: duplicateRenamed }, + "webmcp", + "unknown", +); +script.scripts = { + total: 1, + items: [ + { + script_key: `scr_${"a".repeat(64)}`, + frame_id: null, + url: "https://app.example.test/app.js", + origin: inspection.target.origin, + cdp_hash: null, + length: 1, + is_module: false, + language: "JavaScript", + source_map_url: null, + resource_reconciliation: { + status: "unmatched", + reason: "no_exact_sanitized_url", + }, + source: { + included: true, + artifact: createWebTextArtifact("1", "text/javascript"), + }, + }, + ], +}; +const scriptChanged = structuredClone(script); +scriptChanged.scripts.items[0]!.source = { + included: true, + artifact: createWebTextArtifact("2", "text/javascript"), +}; +add( + "script retained digest change without producer hash", + { inspection: script }, + { inspection: scriptChanged }, + "scripts", + "changed", +); +add( + "script retained digest agreement", + { inspection: script }, + { inspection: script }, + "scripts", + "unchanged", +); +const unavailableScript = structuredClone(script); +unavailableScript.scripts.items[0]!.source = { + included: false, + reason: "not selected", +}; +add( + "script digest unavailable", + { inspection: script }, + { inspection: unavailableScript }, + "scripts", + "unknown", +); +add( + "no source digest is unknown", + { inspection: unavailableScript }, + { inspection: unavailableScript }, + "scripts", + "unknown", +); + +const expectComparison = (document: unknown, scenario: Case) => { + const evidence = parseEvidence(document); + expect(evidence.operation).toBe("compare_web_captures"); + const result = webCaptureDiffSchema.parse(evidence.normalized_result); + expect(result.dimensions[scenario.dimension], scenario.name).toMatchObject({ + status: scenario.status, + ...(scenario.status === "changed" ? {} : { total_changes: 0, changes: [] }), + }); +}; + +// The single MCP connection below exercises every semantic permutation. CLI +// subprocesses cover the distinct adapter outcomes without repeating startup +// for each reversed input or dropped-frame position. +const cliCaseNames = new Set([ + "heterogeneous HTTP omission 0", + "partial inventory retains a request absent from complete peer", + "observed HTTP value absent from complete peer", + "same-length partial WebSocket streams", + "WebMCP capture-local identity renamed", + "script retained digest change without producer hash", + "script digest unavailable", +]); + +cliTest.for(cases.filter(({ name }) => cliCaseNames.has(name)))( + "CLI preserves capture correspondence: $name", + async (scenario, { cli }) => { + const directory = await createTestTempDirectory( + "rea-capture-correspondence-", + ); + const paths = [ + join(directory, "before.json"), + join(directory, "after.json"), + ]; + await Promise.all( + paths.map((path, index) => + writeFile( + path, + JSON.stringify(index === 0 ? scenario.before : scenario.after), + ), + ), + ); + const response = await cli.run({ + arguments: ["compare-web-captures", ...paths, "--json"], + environment: { REA_LOG_LEVEL: "silent" }, + }); + expect(response.exitCode, response.stderr).toBe(0); + expectComparison(response.json, scenario); + }, +); + +cliTest( + "stdio MCP preserves capture correspondence on one connection", + async () => { + const mcp = await connectLocalToolsMcp(); + for (const scenario of cases) { + const response = await mcp.call("compare_web_captures", { + before: scenario.before, + after: scenario.after, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expectComparison(response.structuredContent, scenario); + } + }, +); diff --git a/tests/boundary/browser/cdpBrowserProvider.storage_completeness.test.ts b/tests/boundary/browser/cdpBrowserProvider.storage_completeness.test.ts index b1c5d7495f..4da71be2da 100644 --- a/tests/boundary/browser/cdpBrowserProvider.storage_completeness.test.ts +++ b/tests/boundary/browser/cdpBrowserProvider.storage_completeness.test.ts @@ -250,6 +250,17 @@ const completeRemoteObjects = [ { type: "bigint", unserializableValue: "-42n" }, { type: "object", subtype: "null", value: null }, { type: "object", value: { nested: [false, null, 0] } }, + { + type: "string", + value: "same", + description: "display", + objectId: "optional-reference", + }, + { + type: "object", + value: { objectId: "business-value", customPreview: "content" }, + preview: {}, + }, ] as const; it.each([ @@ -284,6 +295,87 @@ it.each([ }, ); +it("compares decoded IndexedDB content independently of debugger metadata", async () => { + let metadata = false; + let content = "before"; + const { capture } = await inspect({ + commandResult: (command) => + command.method === "IndexedDB.requestData" + ? { + objectStoreDataEntries: [ + { + key: { + ...validKey, + ...(metadata ? { description: "key display" } : {}), + }, + primaryKey: validKey, + value: { + type: "object", + value: { + objectId: content, + customPreview: "retained property", + }, + ...(metadata + ? { + description: "Object", + preview: {}, + objectId: "session-only", + } + : {}), + }, + }, + ], + hasMore: false, + } + : undefined, + }); + const before = await capture(); + metadata = true; + const same = await capture(); + expect(compareStorage(before, same)).toMatchObject({ + status: "unchanged", + total_changes: 0, + }); + content = "after"; + const after = await capture(); + expect(compareStorage(before, after)).toMatchObject({ + status: "changed", + total_changes: 1, + }); +}); + +it.each([undefined, false, true])( + "keeps opaque cookie partition identity incomplete: %s", + async (partitionKeyOpaque) => { + const { capture } = await inspect({ + commandResult: (command) => + command.method === "Network.getCookies" + ? { + cookies: [ + { + name: "cookie", + value: "value", + domain: "app.example.test", + path: "/", + partitionKeyOpaque, + }, + ], + } + : undefined, + }); + const before = await capture(); + const after = await capture(); + expect( + before.storage.content_fingerprints.find( + ({ scope }) => scope === "cookie", + )?.complete, + ).toBe(partitionKeyOpaque !== true); + expect(compareStorage(before, after).status).toBe( + partitionKeyOpaque === true ? "unknown" : "unchanged", + ); + }, +); + it.each([ [undefined, false], [{}, false], diff --git a/tests/boundary/browser/webCaptureDiff.test.ts b/tests/boundary/browser/webCaptureDiff.test.ts index 955b63714d..851cc25874 100644 --- a/tests/boundary/browser/webCaptureDiff.test.ts +++ b/tests/boundary/browser/webCaptureDiff.test.ts @@ -308,8 +308,9 @@ describe("web capture dimension uncertainty", () => { changes: [expect.objectContaining({ change: "removed" })], }); expect(result.dimensions.network).toMatchObject({ - status: "changed", - changes: [expect.objectContaining({ change: "modified" })], + status: "unknown", + total_changes: 0, + changes: [], }); expect(result.dimensions.webmcp).toMatchObject({ status: "unknown",