diff --git a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md index baddcda226..8f5303cc9d 100644 --- a/.agents/skills/reverse-engineer-anything/references/runtime-observation.md +++ b/.agents/skills/reverse-engineer-anything/references/runtime-observation.md @@ -86,6 +86,8 @@ Page-declared WebMCP tools are untrusted inventory and are never invoked by REA. browser and profile. Check `browser.cleanup` for the outcome. Request only the DOM, accessibility, or network details needed for the question. The default final capture contains the URL; step outcomes are reported separately. + For multi-step flows with trees or events, select `detail: "summary"` and page + exact steps, transactions, text slices or events with `inspect_analysis_view` kind `browser-scenario`. Add a wait action for the expected page state before capturing it. Report that REA performed the declared actions. diff --git a/docs/browser-scenario-contract.md b/docs/browser-scenario-contract.md index 1a3a0d703a..0be3b4b493 100644 --- a/docs/browser-scenario-contract.md +++ b/docs/browser-scenario-contract.md @@ -621,3 +621,58 @@ rules and preserve the same missing fact. Network normalization also commits the removal of capture-local transaction IDs, attribution to the requesting step, and unordered transaction comparison. These built-in rule identities participate in the normalization digest. + +## Summary delivery and retained views + +`capture_browser_scenario` accepts `detail: "summary"` to retain the complete +capture in the MCP session and return its summary. The default remains +`complete`. Delivery detail does not change scenario parameters or identity. + +The summary reports each step's status, error, URL and captured/missing/truncated +section names. It omits `not_requested` sections and repeated `before_url` values. +The accessibility tree comes from the first unsuccessful step, or the final +step on success; when that step captured no tree, its nearest earlier captured +tree is identified explicitly. Event and network counts cover retained records, +with capture omissions reported separately. + +For the one-shot CLI, save the complete Evidence before selecting a summary: + +```bash +rea capture-browser-scenario scenario.json --detail summary --evidence_output /absolute/path/scenario-evidence.json --json +``` + +Use the returned `normalized_result.parent_evidence_id` with +`inspect_analysis_view` source `{kind: "retained-evidence", evidence_id: "..."}`. +For a CLI bundle, use an `evidence-file` source with its absolute path and the +parent Evidence ID. Views read the recorded capture; they never replay actions. + +- `{kind: "summary"}` returns the same overview. +- `{kind: "browser-scenario", collection: "steps", step_indices: [2], artifacts: ["accessibility"], offset: 0, limit: 1}` returns selected recorded steps and artifacts. +- `{kind: "browser-scenario", collection: "events", event_kinds: ["response"], transaction_ids: ["request-2"], offset: 0, limit: 20}` pages selected events with original item indices. + +Omitted filters select all; empty arrays select none. Pagination counts matching +retained rows. Exhausting a page does not establish capture completeness or +current application state. + +Transaction pages use `{kind: "browser-scenario", collection: "transactions", limit: 20}`. +Their stable identity is the parent Evidence ID plus `transaction_id`; each row +keeps original event indices. Metadata events are returned by default. Select +`event_kinds: ["network-content"]` for recorded bodies and header values, or +`event_kinds: []` for identity and indices alone. A `step_indices` filter selects +a whole transaction when any retained event belongs to a selected step; this +is receipt attribution, not proof that the action caused the request. + +For a large retained tree, request `{kind: "browser-scenario", collection: "text", +step_index: 10, artifact: "accessibility", offset: 0, limit: 2048}`. `dom` is also +supported. Offsets and limits count UTF-16 code units. The full artifact digest +and byte size remain attached to each page. Missing, truncated and unrequested +text stay explicit; source truncation counts are preserved without treating an +exhausted page as a complete capture. + +Network summary counts include every retained transaction identity, including +those whose request event was dropped. Missing request events, conflicting +metadata and observed request failures have separate counts. `transactions` is +`null` when network observation was unrequested or its legacy selection is +unknown and there are no retained transactions; an observed empty inventory is +`0`. Full capture omissions remain available and each page carries source +completeness, dropped-event counts and omitted-transaction accounting. diff --git a/experiments/analysis-efficiency/browser-scenario-summary.md b/experiments/analysis-efficiency/browser-scenario-summary.md new file mode 100644 index 0000000000..008001d0e0 --- /dev/null +++ b/experiments/analysis-efficiency/browser-scenario-summary.md @@ -0,0 +1,63 @@ +# Browser scenario summary verification + +The verifier runs the actual compiled CLI/MCP and caller-selected Chromium against +TodoMVC's JavaScript ES5 distribution served on loopback. Ten actions add five +todos. Every step captures URL, DOM, accessibility and a screenshot; the oracle +checks all completed steps, the five authored labels, retained events and +transaction identities. Summary runs also read step, text and transaction pages +and authenticate an exported complete parent. + +Download the public distribution at the pinned revision into a disposable folder: + +```bash +export REA_TODOMVC_APP="$(mktemp -d)" +python3 - "$REA_TODOMVC_APP" <<'PY' +import hashlib, json, sys, urllib.request +from pathlib import Path +root = Path(sys.argv[1]) +revision = "ff43b02e59dfa604386bb382034b2cd07c2bcd8a" +files = ["index.html", "app.js", "base.css", "base.js", "controller.js", + "helpers.js", "index.css", "model.js", "store.js", "template.js", "view.js"] +manifest = [] +for name in files: + url = f"https://raw.githubusercontent.com/tastejs/todomvc/{revision}/examples/javascript-es5/dist/{name}" + with urllib.request.urlopen(url, timeout=30) as reply: + data = reply.read() + (root / name).write_bytes(data) + manifest.append({"path": name, "url": url, "bytes": len(data), + "sha256": hashlib.sha256(data).hexdigest()}) +(root / "provenance.json").write_text(json.dumps({"revision": revision, "files": manifest}, indent=2)) +PY +``` + +Build each selected checkout with the repository's pinned Node/npm toolchain. +Use new output directories; exports never overwrite an earlier run: + +```bash +node experiments/analysis-efficiency/browser-scenario-summary.mjs \ + --repository /absolute/path/to/baseline --mode complete \ + --application "$REA_TODOMVC_APP" --browser /absolute/path/to/chromium \ + --output /absolute/path/to/new-baseline-report +node experiments/analysis-efficiency/browser-scenario-summary.mjs \ + --mode summary --application "$REA_TODOMVC_APP" \ + --browser /absolute/path/to/chromium \ + --output /absolute/path/to/new-summary-report +``` + +`cold` is the first capture in a fresh MCP process; `warm` repeats the capture in +that process with a new owned browser/profile. Neither label establishes cold OS +file caches. Summary mode intentionally performs additional retained queries. +Compare equivalent answers and include those calls when reporting task cost; +smaller replies alone do not establish a faster task. + +Add `--verify-boundaries` to a separate summary run for a real failed-step focus, +CLI summary publication, and MCP text readback from the CLI's complete bundle. +Those extra checks make its whole-run resource cost unsuitable for comparison +against a complete-only run. + +Reports retain the actual catalog, arguments, MCP text and complete Evidence. +When estimating tokens, name the tokenizer and count discovery, arguments and +all replies needed for the chosen task. Raw serialized catalog estimates are +not actual model usage or a measurement of a client's rendered tool definitions. +Measure CPU and peak process-family memory externally, state the accounting +method, and include cleanup; the verifier's elapsed phase excludes final teardown. diff --git a/experiments/analysis-efficiency/browser-scenario-summary.mjs b/experiments/analysis-efficiency/browser-scenario-summary.mjs new file mode 100644 index 0000000000..a7addfd438 --- /dev/null +++ b/experiments/analysis-efficiency/browser-scenario-summary.mjs @@ -0,0 +1,412 @@ +import assert from "node:assert/strict"; +import { createServer } from "node:http"; +import { readFile, writeFile, mkdir, readdir } from "node:fs/promises"; +import { resolve, join, extname } from "node:path"; +import { pathToFileURL, fileURLToPath } from "node:url"; +import { createRequire } from "node:module"; +import { performance } from "node:perf_hooks"; +import { execFile } from "node:child_process"; +import { promisify, parseArgs } from "node:util"; +// A developer verifier: the caller selects the pinned TodoMVC distribution, +// browser executable, output directory and baseline/final repository checkout. +const { values } = parseArgs({ + options: { + repository: { + type: "string", + default: fileURLToPath(new URL("../../", import.meta.url)), + }, + mode: { type: "string", default: "summary" }, + application: { type: "string" }, + browser: { type: "string" }, + output: { type: "string" }, + port: { type: "string", default: "0" }, + "verify-boundaries": { type: "boolean", default: false }, + }, +}); +assert.ok( + values.application && values.browser && values.output, + "Pass --application --browser --output ", +); +const repo = resolve(values.repository); +const mode = values.mode; +const output = resolve(values.output); +const appRoot = resolve(values.application); +const executablePath = resolve(values.browser); +assert.ok(["complete", "summary"].includes(mode)); +assert.ok( + !values["verify-boundaries"] || mode === "summary", + "--verify-boundaries requires summary mode", +); +process.chdir(repo); +const require = createRequire(join(repo, "package.json")); +const { Client } = await import( + pathToFileURL(require.resolve("@modelcontextprotocol/client")).href +); +const { StdioClientTransport } = await import( + pathToFileURL(require.resolve("@modelcontextprotocol/client/stdio")).href +); +await mkdir(output); +const files = await readdir(appRoot); +const server = createServer(async (request, response) => { + try { + const name = + decodeURIComponent( + new URL(request.url, "http://127.0.0.1").pathname, + ).slice(1) || "index.html"; + assert.ok(files.includes(name)); + const data = await readFile(join(appRoot, name)); + response.setHeader( + "Content-Type", + { ".html": "text/html", ".js": "text/javascript", ".css": "text/css" }[ + extname(name) + ] ?? "application/octet-stream", + ); + response.end(data); + } catch { + response.writeHead(404); + response.end(); + } +}); +await new Promise((done, reject) => { + server.once("error", reject); + server.listen(Number(values.port), "127.0.0.1", done); +}); +const address = server.address(); +assert.ok(address && typeof address === "object"); +const origin = `http://127.0.0.1:${address.port}`; +const environment = { + PATH: process.env.PATH, + NODE_OPTIONS: "--max-old-space-size=3072", + UV_THREADPOOL_SIZE: "2", + REA_LOG_LEVEL: "silent", + REA_MCP_INPUT_SCHEMA_PROFILE: "compact", +}; +const transport = new StdioClientTransport({ + command: process.execPath, + args: [join(repo, "scripts/rea.mjs"), "mcp"], + cwd: repo, + env: environment, + stderr: "pipe", +}); +let stderr = ""; +transport.stderr?.on("data", (chunk) => { + if (stderr.length < 65536) stderr += chunk; +}); +const client = new Client({ + name: "rea-browser-scenario-summary-verifier", + version: "1", +}); +const calls = []; +const invoke = async (name, args, tools) => { + const toolDefinition = tools.find((tool) => tool.name === name); + assert.ok( + toolDefinition?.outputSchema, + `Missing named output schema: ${name}`, + ); + const started = performance.now(); + const result = await client.callTool( + { name, arguments: args }, + { toolDefinition, timeout: 90000 }, + ); + calls.push({ + name, + arguments: args, + content: result.content, + elapsed_ms: performance.now() - started, + }); + assert.notEqual(result.isError, true, JSON.stringify(result)); + return result.structuredContent; +}; +const allStarted = performance.now(); +try { + await client.connect(transport); + const discoveryStarted = performance.now(); + const catalog = await client.listTools(); + const discoveryMs = performance.now() - discoveryStarted; + await writeFile(join(output, "catalog.json"), JSON.stringify(catalog)); + const actions = Array.from({ length: 5 }, (_, index) => [ + { + step_id: `fill_${index}`, + action: "fill", + locator: { kind: "css", selector: ".new-todo" }, + value: { source: "literal", value: `Review evidence ${index + 1}` }, + timeout_ms: 10000, + }, + { + step_id: `add_${index}`, + action: "press", + locator: { kind: "css", selector: ".new-todo" }, + key: "Enter", + timeout_ms: 10000, + }, + ]).flat(); + const scenario = { + browser: { + mode: "launch", + executable_path: executablePath, + headless: true, + }, + start_url: { url: origin, query: [] }, + actions, + capture: { + after_each_step: ["url", "dom", "accessibility", "screenshot"], + at_end: ["url", "dom", "accessibility"], + events: ["network"], + }, + }; + const runs = []; + for (const temperature of ["cold", "warm"]) { + const callStart = calls.length; + const result = await invoke( + "capture_browser_scenario", + { ...scenario, ...(mode === "summary" ? { detail: "summary" } : {}) }, + catalog.tools, + ); + const parentId = + mode === "summary" + ? result.normalized_result.parent_evidence_id + : result.evidence_id; + const bundlePath = join(output, `${temperature}-evidence.json`); + await invoke("export_evidence_bundle", { path: bundlePath }, catalog.tools); + const bundle = JSON.parse(await readFile(bundlePath, "utf8")); + const parent = bundle.records.find((row) => row.evidence_id === parentId); + assert.ok(parent); + const capture = parent.normalized_result; + assert.equal(capture.steps.length, 11); + assert.ok(capture.steps.every((step) => step.status === "completed")); + assert.equal(capture.browser.cleanup, "terminated-owned-process"); + const final = capture.steps.at(-1).artifacts.accessibility; + assert.equal(final.state, "captured"); + for (let index = 1; index <= 5; index++) + assert.ok(final.value.text.includes(`Review evidence ${index}`)); + assert.ok(final.value.text.includes('strong: "5"')); + assert.ok(final.value.text.includes("text: items left")); + const transactionIds = new Set( + capture.events.items + .filter((event) => "transaction_id" in event) + .map((event) => event.transaction_id), + ); + const details = { + temperature, + parentId, + steps: capture.steps.length, + retained_events: capture.events.retained, + transactions: transactionIds.size, + }; + if (mode === "summary") { + const summary = result.normalized_result.summary; + assert.equal(summary.steps.length, 11); + assert.equal( + summary.final_accessibility.accessibility.value.text, + final.value.text, + ); + assert.equal(summary.events.retained, capture.events.retained); + assert.equal(summary.limitation_count, parent.limitations.length); + assert.equal(summary.network.transactions, transactionIds.size); + const source = { kind: "retained-evidence", evidence_id: parentId }; + const pagingStarted = performance.now(); + for (const stepIndex of [1, 5, 10]) { + const page = await invoke( + "inspect_analysis_view", + { + source, + view: { + kind: "browser-scenario", + collection: "steps", + step_indices: [stepIndex], + artifacts: [], + limit: 1, + }, + }, + catalog.tools, + ); + assert.equal( + page.normalized_result.items.rows[0].step_id, + capture.steps[stepIndex].step_id, + ); + } + details.warm_page_total_ms = performance.now() - pagingStarted; + const text = await invoke( + "inspect_analysis_view", + { + source, + view: { + kind: "browser-scenario", + collection: "text", + step_index: 10, + artifact: "accessibility", + offset: 0, + limit: 32, + }, + }, + catalog.tools, + ); + assert.equal( + text.normalized_result.items.text, + final.value.text.slice(0, 32), + ); + const tx = await invoke( + "inspect_analysis_view", + { + source, + view: { + kind: "browser-scenario", + collection: "transactions", + limit: 2, + }, + }, + catalog.tools, + ); + assert.deepEqual( + tx.normalized_result.items.rows.map((row) => row.transaction_id), + [...transactionIds].slice(0, 2), + ); + details.returned_summary = summary; + } + details.calls = calls.slice(callStart); + runs.push(details); + } + const boundaryChecks = []; + if (values["verify-boundaries"]) { + const failed = await invoke( + "capture_browser_scenario", + { + ...scenario, + actions: [ + { + step_id: "missing", + action: "click", + locator: { + kind: "css", + selector: "#rea-control-that-does-not-exist", + }, + timeout_ms: 100, + }, + ], + capture: { + after_each_step: ["url", "accessibility"], + at_end: ["url", "accessibility"], + events: ["network"], + }, + detail: "summary", + }, + catalog.tools, + ); + assert.equal(failed.normalized_result.summary.steps[1].status, "failed"); + assert.equal( + failed.normalized_result.summary.final_accessibility.focus_step_index, + 1, + ); + assert.equal( + failed.normalized_result.summary.completeness.equality_eligible, + false, + ); + boundaryChecks.push("real failed-step focus and incomplete coverage"); + const execute = promisify(execFile); + const bundlePath = join(output, "cli-evidence.json"); + const { stdout } = await execute( + process.execPath, + [ + join(repo, "scripts/rea.mjs"), + "capture-browser-scenario", + JSON.stringify(scenario), + "--detail", + "summary", + "--evidence_output", + bundlePath, + "--json", + ], + { + cwd: repo, + env: environment, + maxBuffer: 64 * 1024 * 1024, + timeout: 90000, + }, + ); + const cli = JSON.parse(stdout); + const parentId = cli.summary.normalized_result.parent_evidence_id; + const bundle = JSON.parse(await readFile(bundlePath, "utf8")); + const parent = bundle.records.find((row) => row.evidence_id === parentId); + assert.ok(parent); + assert.equal(parent.normalized_result.steps.length, 11); + assert.ok( + parent.normalized_result.steps.every( + (step) => step.status === "completed", + ), + ); + assert.ok(!("detail" in parent.parameters)); + assert.equal( + cli.summary.normalized_result.summary.final_accessibility.accessibility + .value.text, + parent.normalized_result.steps.at(-1).artifacts.accessibility.value.text, + ); + const text = await invoke( + "inspect_analysis_view", + { + source: { + kind: "evidence-file", + path: bundlePath, + evidence_id: parentId, + }, + view: { + kind: "browser-scenario", + collection: "text", + step_index: 10, + artifact: "accessibility", + offset: 0, + limit: 32, + }, + }, + catalog.tools, + ); + assert.equal( + text.normalized_result.items.text, + parent.normalized_result.steps + .at(-1) + .artifacts.accessibility.value.text.slice(0, 32), + ); + boundaryChecks.push( + "real CLI summary, complete bundle and MCP evidence-file readback", + ); + } + await writeFile( + join(output, "result.json"), + JSON.stringify( + { + repo, + mode, + application: appRoot, + browser: executablePath, + input_paths: { + repository: values.repository, + application: values.application, + browser: values.browser, + output: values.output, + }, + origin, + discovery_ms: discoveryMs, + elapsed_ms: performance.now() - allStarted, + runs, + boundaryChecks, + stderr, + }, + null, + 2, + ), + ); + console.log( + JSON.stringify({ + mode, + discovery_ms: discoveryMs, + runs: runs.map( + ({ calls: _calls, returned_summary: _summary, ...rest }) => rest, + ), + boundaryChecks, + elapsed_ms: performance.now() - allStarted, + }), + ); +} finally { + await client.close(); + server.closeAllConnections(); + await new Promise((done) => server.close(done)); +} diff --git a/src/application/analysisView/BrowserScenarioViews.test.ts b/src/application/analysisView/BrowserScenarioViews.test.ts new file mode 100644 index 0000000000..4a01e56046 --- /dev/null +++ b/src/application/analysisView/BrowserScenarioViews.test.ts @@ -0,0 +1,521 @@ +import { createHash } from "node:crypto"; + +import { describe, expect, it } from "vitest"; + +import { + SCENARIO_ORIGIN, + scenarioCaptureEvidence, + scenarioCaptureFixture, +} from "../../../tests/fixtures/browserScenarioCapture.js"; +import { inspectAnalysisView } from "./AnalysisViewService.js"; +import { projectAnalysisError } from "../../domain/analysisErrorProjection.js"; +import { sanitizeBrowserUrl } from "../../domain/browserObservation.js"; +import type { BrowserScenarioCapture } from "../../domain/browserScenarioCapture.js"; +import type { AnalysisViewRequest } from "../../domain/analysisView/analysisView.js"; +import { projectBrowserScenarioView } from "../../domain/analysisView/browserScenarioView.js"; + +const tree = (text: string) => ({ + state: "captured" as const, + value: { + sha256: createHash("sha256").update(text).digest("hex"), + bytes: Buffer.byteLength(text), + text, + }, +}); + +/** Three steps: navigate to /quiz, capture its tree, then fail on a missing control. */ +const quizCapture = ( + thirdStep: "failed" | "completed" = "failed", +): BrowserScenarioCapture => { + const base = scenarioCaptureFixture(); + const [initial, browse] = base.steps; + if (initial === undefined || browse === undefined) + throw new Error("Fixture lacks steps"); + const quiz = sanitizeBrowserUrl(`${SCENARIO_ORIGIN}/quiz`); + return { + ...base, + scenario: { ...base.scenario, action_count: 2 }, + steps: [ + { + ...initial, + artifacts: { + ...initial.artifacts, + url: { state: "captured", value: initial.after_url }, + }, + }, + { + ...browse, + after_url: quiz, + artifacts: { + ...browse.artifacts, + accessibility: tree('- heading "Question 1"'), + screenshot: { state: "missing", reason: "page closed" }, + }, + }, + { + ...browse, + step_index: 2, + step_id: "answer", + action: "click", + before_url: quiz, + after_url: quiz, + ...(thirdStep === "failed" + ? { status: "failed" as const, error: "locator not found" } + : { status: "completed" as const, error: null }), + }, + ], + }; +}; + +const view = async ( + capture: BrowserScenarioCapture, + request: AnalysisViewRequest, +) => { + const projected = await inspectAnalysisView({ + source: { kind: "inline", evidence: scenarioCaptureEvidence(capture) }, + view: request, + }); + if (!projected.ok) throw projected.error; + return projected.value.normalized_result as Record; +}; + +describe("browser scenario analysis views", () => { + it("counts inherited Evidence limitations alongside capture-local caveats", async () => { + const capture = scenarioCaptureFixture(); + expect(capture.limitations).toEqual([]); + const summary = await view(capture, { kind: "summary" }); + expect(summary).toMatchObject({ + limitations: [ + "Artifact identity is unavailable for this observation.", + "Browser host platform and architecture were not observed.", + ], + summary: { limitation_count: 2 }, + }); + }); + + it("summarizes steps compactly and focuses the first failing step's nearest captured tree", async () => { + const summary = await view(quizCapture(), { kind: "summary" }); + expect(summary).toMatchObject({ + kind: "summary", + parent_operation: "capture_browser_scenario", + artifact: null, + coverage: { status: "complete-within-view", exhausted: true }, + }); + const { steps, final_accessibility, network, events } = + summary.summary as Record & { + steps: Record[]; + }; + // before_url repeats only across a discontinuity; not_requested sections are omitted. + expect(steps[0]).toHaveProperty("before_url"); + expect(steps[1]).not.toHaveProperty("before_url"); + expect(steps[2]).not.toHaveProperty("before_url"); + expect(steps[0]?.sections).toEqual({ + captured: ["url"], + missing: [], + truncated: [], + }); + expect(steps[1]?.sections).toEqual({ + captured: ["accessibility"], + missing: ["screenshot"], + truncated: [], + }); + expect(steps[2]).toMatchObject({ + status: "failed", + error: "locator not found", + after_url: { url: `${SCENARIO_ORIGIN}/quiz` }, + }); + expect(final_accessibility).toEqual({ + focus_step_index: 2, + focus: "first-unsuccessful-step", + step_index: 1, + accessibility: tree('- heading "Question 1"'), + }); + expect(network).toEqual({ + selected: true, + transactions: 8, + without_request: 0, + ambiguous_transactions: 0, + failed_transactions: 1, + by_origin: [{ origin: SCENARIO_ORIGIN, transactions: 8 }], + by_resource_type: [ + { resource_type: "document", transactions: 1 }, + { resource_type: "fetch", transactions: 7 }, + ], + by_status_class: [ + { status_class: "2xx", transactions: 7 }, + { status_class: "failed", transactions: 1 }, + ], + }); + expect(events).toMatchObject({ retained: 24, dropped: 0 }); + }); + + it("reports the last step's own state when no step captured a tree", async () => { + const capture = scenarioCaptureFixture(); + expect(await view(capture, { kind: "summary" })).toMatchObject({ + summary: { + final_accessibility: { + focus_step_index: 1, + focus: "last-step", + step_index: null, + accessibility: { state: "not_requested" }, + }, + }, + }); + }); +}); + +describe("browser scenario analysis view pages", () => { + it("pages exact steps with only the selected captured sections", async () => { + const page = await view(quizCapture("completed"), { + kind: "browser-scenario", + collection: "steps", + step_indices: [1, 2], + artifacts: ["accessibility"], + offset: 0, + limit: 1, + }); + expect(page).toMatchObject({ + kind: "browser-scenario", + source_completeness: { status: "complete" }, + source_events: { retained: 24, dropped: 0 }, + coverage: { status: "page", examined: 1, total: 2, next_offset: 1 }, + items: { + collection: "steps", + rows: [ + { + step_index: 1, + artifacts: { accessibility: tree('- heading "Question 1"') }, + omitted_artifacts: [{ section: "screenshot", state: "missing" }], + }, + ], + }, + }); + }); + + it("filters events by step, kind and transaction, keeping original item indices", async () => { + const capture = scenarioCaptureFixture(); + const responses = (await view(capture, { + kind: "browser-scenario", + collection: "events", + event_kinds: ["response", "request-failed"], + transaction_ids: ["request-2", "request-8"], + offset: 0, + limit: 64, + })) as { items: { rows: { item_index: number; event: unknown }[] } }; + expect( + responses.items.rows.map(({ item_index, event }) => ({ + item_index, + event, + })), + ).toEqual( + capture.events.items.flatMap((event, item_index) => + (event.kind === "response" || event.kind === "request-failed") && + ["request-2", "request-8"].includes(event.transaction_id) + ? [{ item_index, event }] + : [], + ), + ); + expect(responses.items.rows).toHaveLength(2); + for (const empty of [ + { step_indices: [] }, + { event_kinds: [] }, + { transaction_ids: [] }, + { step_indices: [0] }, + ]) + expect( + await view(capture, { + kind: "browser-scenario", + collection: "events", + offset: 0, + limit: 64, + ...empty, + }), + ).toMatchObject({ + items: { rows: [] }, + coverage: { status: "empty", total: 0, exhausted: true }, + }); + }); + + it("states dropped events and refuses views of other parents", async () => { + const base = scenarioCaptureFixture(); + const dropped = { + ...base, + events: { + ...base.events, + dropped: 3, + omissions: { + ...base.events.omissions, + by_kind: [ + { + kind: "console" as const, + count: 3, + reasons: ["max-events" as const], + }, + ], + }, + }, + completeness: { + status: "truncated" as const, + equality_eligible: false as const, + missing_sections: [], + truncated_sections: ["events" as const], + }, + }; + const summary = await view(dropped, { kind: "summary" }); + expect(summary.unknowns).toContain( + "3 events were dropped by capture budgets; counts and pages cover retained events only.", + ); + expect(summary).toMatchObject({ + summary: { + events: { + dropped: 3, + omissions: { + by_kind: [{ kind: "console", count: 3 }], + network_transactions: 0, + }, + }, + }, + }); + const refused = await inspectAnalysisView({ + source: { kind: "inline", evidence: scenarioCaptureEvidence() }, + view: { kind: "go", collection: "dependencies" }, + }); + if (refused.ok) throw new Error("Expected an unsupported view refusal"); + expect(projectAnalysisError(refused.error)).toMatchObject({ + code: "invalid_request", + }); + }); +}); + +describe("browser scenario retained boundaries", () => { + it("counts transactions whose request event was omitted and keeps source omissions on an exhausted page", async () => { + const base = scenarioCaptureFixture(); + const omitted = ( + event: BrowserScenarioCapture["events"]["items"][number], + ) => + event.kind === "request" || + (event.kind === "network-content" && event.phase === "request"); + const removed = base.events.items.filter(omitted); + const items = base.events.items.filter((event) => !omitted(event)); + const capture: BrowserScenarioCapture = { + ...base, + events: { + ...base.events, + items, + retained: items.length, + dropped: removed.length, + omissions: { + ...base.events.omissions, + by_kind: [ + { kind: "request", count: 8, reasons: ["max-events"] }, + { kind: "network-content", count: 1, reasons: ["max-events"] }, + ], + }, + }, + completeness: { + status: "truncated", + equality_eligible: false, + missing_sections: [], + truncated_sections: ["events"], + }, + }; + expect(await view(capture, { kind: "summary" })).toMatchObject({ + summary: { + network: { + transactions: 8, + without_request: 8, + failed_transactions: 1, + }, + }, + }); + expect( + await view(capture, { + kind: "browser-scenario", + collection: "transactions", + transaction_ids: ["request-8"], + event_kinds: [], + offset: 0, + limit: 1, + }), + ).toMatchObject({ + source_completeness: { status: "truncated" }, + source_events: { dropped: 9 }, + coverage: { total: 1, exhausted: true }, + items: { + rows: [ + { + transaction_id: "request-8", + events: [], + omitted_events: [{ kind: "request-failed" }], + }, + ], + }, + }); + }); + + it("keeps network bodies opt-in while retaining each transaction's event identities", async () => { + const capture = scenarioCaptureFixture(); + const request = { + kind: "browser-scenario" as const, + collection: "transactions" as const, + transaction_ids: ["request-2"], + offset: 0, + limit: 1, + }; + // Parse defaults at the public boundary rather than supplying a manually normalized request. + const page = await inspectAnalysisView({ + source: { kind: "inline", evidence: scenarioCaptureEvidence(capture) }, + view: request, + }); + if (!page.ok) throw page.error; + const result = page.value.normalized_result; + expect(result).toMatchObject({ + items: { + collection: "transactions", + rows: [{ transaction_id: "request-2" }], + }, + }); + expect(JSON.stringify(result)).not.toContain('"content":'); + const selected = await view(capture, { + ...request, + event_kinds: ["network-content"], + }); + expect(JSON.stringify(selected)).toContain('"content":'); + expect(selected).toMatchObject({ + items: { + rows: [ + { + omitted_events: expect.arrayContaining([ + { item_index: expect.any(Number), kind: "request" }, + ]), + }, + ], + }, + }); + }); + + it("distinguishes unrequested and legacy-unknown network observation from an observed empty inventory", () => { + const base = scenarioCaptureFixture([], { + request_body: false, + response_body: false, + header_values: false, + }); + const project = (capture: unknown) => { + const result = projectBrowserScenarioView( + { + evidenceId: `ev:${"a".repeat(64)}`, + operation: "capture_browser_scenario", + normalizedResult: capture, + limitations: [], + }, + { kind: "summary" }, + ); + if (!result.ok) throw result.error; + return result.value; + }; + expect(project(base)).toMatchObject({ + summary: { network: { selected: true, transactions: 0 } }, + }); + expect( + project({ ...base, scenario: { ...base.scenario, event_families: [] } }), + ).toMatchObject({ + summary: { network: { selected: false, transactions: null } }, + }); + const { event_families: _families, ...legacyScenario } = base.scenario; + expect(project({ ...base, scenario: legacyScenario })).toMatchObject({ + summary: { network: { selected: null, transactions: null } }, + }); + }); +}); + +describe("browser scenario text and selection boundaries", () => { + it("pages exact Unicode text with the full artifact digest and reports unavailable text explicitly", async () => { + const capture = quizCapture("completed"); + const step = capture.steps[1]!; + step.artifacts.accessibility = tree("A💡B\nC"); + const request = { + kind: "browser-scenario" as const, + collection: "text" as const, + step_index: 1, + artifact: "accessibility" as const, + offset: 1, + limit: 2, + }; + expect(await view(capture, request)).toMatchObject({ + items: { + state: "captured", + text: "💡", + sha256: tree("A💡B\nC").value.sha256, + bytes: Buffer.byteLength("A💡B\nC"), + }, + coverage: { examined: 2, total: 6, next_offset: 3 }, + }); + expect(await view(capture, { ...request, offset: 99 })).toMatchObject({ + items: { text: "" }, + coverage: { status: "empty", total: 6, exhausted: true }, + }); + expect(await view(capture, { ...request, step_index: 0 })).toMatchObject({ + items: { state: "not_requested", text: null, sha256: null, bytes: null }, + }); + }); + + it("rejects irrelevant selection fields and nonexistent text steps with typed diagnostics", async () => { + for (const request of [ + { kind: "browser-scenario", collection: "events", artifacts: [] }, + { + kind: "browser-scenario", + collection: "text", + step_index: 99, + artifact: "dom", + }, + ]) { + const result = await inspectAnalysisView({ + source: { kind: "inline", evidence: scenarioCaptureEvidence() }, + view: request, + }); + if (result.ok) + throw new Error("Expected the invalid view to be rejected"); + expect(projectAnalysisError(result.error)).toMatchObject({ + code: "invalid_request", + }); + } + }); + + it("keeps source truncation facts when a text page has no retained payload", async () => { + const capture = quizCapture("completed"); + const completeness = { + status: "truncated" as const, + equality_eligible: false as const, + missing_sections: [], + truncated_sections: ["accessibility" as const], + }; + capture.steps[1]!.artifacts.accessibility = { + state: "truncated", + observed: 1000, + retained: 0, + reason: "capture budget", + }; + capture.steps[1]!.completeness = completeness; + capture.completeness = completeness; + expect( + await view(capture, { + kind: "browser-scenario", + collection: "text", + step_index: 1, + artifact: "accessibility", + offset: 0, + limit: 20, + }), + ).toMatchObject({ + source_completeness: { status: "truncated" }, + items: { + state: "truncated", + observed: 1000, + retained: 0, + reason: "capture budget", + text: null, + }, + coverage: { status: "empty", exhausted: true }, + }); + }); +}); diff --git a/src/application/environment/environmentScenario.test.ts b/src/application/environment/environmentScenario.test.ts index 2f0d542c14..2debd00892 100644 --- a/src/application/environment/environmentScenario.test.ts +++ b/src/application/environment/environmentScenario.test.ts @@ -76,15 +76,15 @@ describe("environment manifests from capture_browser_scenario parents", () => { kind: "inline", evidence: scenarioCaptureEvidence(scenarioCaptureFixture()), }, - view: { kind: "summary" }, + view: { kind: "dart-aot", collection: "package_uris" }, }); if (refused.ok) throw new Error("Expected the unsupported view refusal"); expect(projectAnalysisError(refused.error)).toMatchObject({ - code: "unsupported_target", - message: expect.stringContaining( - "supports only the environment-manifest, environment-state and environment-coverage views", - ), + code: "invalid_request", }); + expect(JSON.stringify(projectAnalysisError(refused.error))).toContain( + "supports summary and browser-scenario views", + ); }); it("serves and exports a scenario-built manifest from the same capture", () => { diff --git a/src/cli/finishAnalysisCommand.ts b/src/cli/finishAnalysisCommand.ts index 5a529fd0a3..c05979eacb 100644 --- a/src/cli/finishAnalysisCommand.ts +++ b/src/cli/finishAnalysisCommand.ts @@ -21,6 +21,7 @@ export const finishAnalysisCommand = async < canonicalOperation: string, signal: AbortSignal, fallbackResource?: string, + options: { readonly failureProjection?: "initial" | "final" } = {}, ): Promise> => { let result: Result; let cleanup: AnalysisError | undefined; @@ -35,6 +36,12 @@ export const finishAnalysisCommand = async < : new ProviderAdapterError(providerId, canonicalOperation, { cause }), }; } + // Some one-shot commands promise the original failure snapshot even when + // close retries and mutates the producer's live cleanup status. + const initialFailure = + !result.ok && options.failureProjection === "initial" + ? projectAnalysisError(result.error) + : undefined; try { // Resource owners may resolve their retained errors during close. This // adapter cannot infer resolution of a producer's earlier cleanup failure. @@ -61,6 +68,7 @@ export const finishAnalysisCommand = async < } } await dispatchQueuedInterrupts(); + if (initialFailure !== undefined) return initialFailure; if (result.ok && signal.aborted) result = { ok: false, diff --git a/src/cliBrowserScenarioCommands.ts b/src/cliBrowserScenarioCommands.ts index 20e8923203..11234a26ac 100644 --- a/src/cliBrowserScenarioCommands.ts +++ b/src/cliBrowserScenarioCommands.ts @@ -2,15 +2,17 @@ import { Cli } from "incur"; import { z } from "zod"; import { captureBrowserScenario } from "./application/BrowserScenarioCaptureService.js"; +import { finishAnalysisCommand } from "./cli/finishAnalysisCommand.js"; import { withCommandCancellation } from "./cli/commandCancellation.js"; +import { + analysisEvidenceDeliveryOptions, + cliEvidenceDeliveryInputError, + publishCliAnalysisEvidence, +} from "./cli/analysisEvidenceDelivery.js"; import { CLI_COMMANDS } from "./cliCommandNames.js"; import { parseCliJsonInput, resolveCliJsonPaths } from "./cliJsonInput.js"; import { logCliCommand } from "./cliLogging.js"; import { projectAnalysisError } from "./domain/analysisErrorProjection.js"; -import { - BrowserObservationError, - browserCleanupObservation, -} from "./domain/browserObservationError.js"; import { analysisInputErrorFromIssues } from "./domain/inputIssueProjection.js"; import { browserScenarioSchema } from "./domain/browserScenario.js"; import type { BrowserScenarioCapturePort } from "./application/BrowserScenarioCapturePort.js"; @@ -38,9 +40,16 @@ export const registerBrowserScenarioCommands = ( .string() .describe("Inline browser scenario JSON or JSON file path"), }), - run: ({ args }) => + options: z.object(analysisEvidenceDeliveryOptions), + run: ({ args, options: delivery }) => withCommandCancellation((signal) => logCliCommand(logger, CLI_COMMANDS.captureBrowserScenario, async () => { + const deliveryError = cliEvidenceDeliveryInputError({ + detail: delivery.detail, + evidenceOutput: delivery.evidence_output, + operation: OPERATION, + }); + if (deliveryError !== undefined) return cliError(deliveryError); const input = await parseCliJsonInput( args.inputJson, OPERATION, @@ -67,28 +76,28 @@ export const registerBrowserScenarioCommands = ( createBrowserScenarioProvider(environment), ) : options.createProvider(environment); - let result: - | Awaited> - | undefined; - try { - result = await captureBrowserScenario(provider, scenario.data, { - signal, - }); - return result.ok ? result.value : cliError(result.error); - } finally { - try { - await provider?.close?.(); - } catch (cause: unknown) { - // Failed captures already retain their primary error and partial - // observations. A cleanup retry must not replace that outcome. - if (result?.ok) - throw new BrowserObservationError(OPERATION, "cleanup_failed", { - cause, - cleanup: browserCleanupObservation(cause), - partialObservation: result.value, - }); - } - } + const result = await finishAnalysisCommand( + () => captureBrowserScenario(provider, scenario.data, { signal }), + { + close: async () => { + await provider?.close?.(); + }, + }, + "rea-playwright-browser-scenario", + OPERATION, + signal, + undefined, + { failureProjection: "initial" }, + ); + if (!("evidence_id" in result)) + return { error: "Browser scenario capture failed", ...result }; + return publishCliAnalysisEvidence(result, { + providerId: "rea-playwright-browser-scenario", + operation: OPERATION, + signal, + detail: delivery.detail, + evidenceOutput: delivery.evidence_output, + }); }), ), }); diff --git a/src/contracts/analysisViewToolContracts.ts b/src/contracts/analysisViewToolContracts.ts index 8218722f25..7de273e6bf 100644 --- a/src/contracts/analysisViewToolContracts.ts +++ b/src/contracts/analysisViewToolContracts.ts @@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [ ...toolContractMetadata("inspect_analysis_view"), kind: "application", description: - 'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state (a list also learns the filters, search, sort and paging its recordings verify, and answers new values of them from the state), and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming its effect and none contradicting it. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. The environment-coverage view, view {kind: "environment-coverage", template_rules: [...]}, summarizes that state without its entities: reads.state and reads.recorded count read tools answered from the state and replayed (recorded_reasons per view reason), writes.served lists the served write tools, and writes.withheld lists each other write tool with its reason, plan record, contradicting read and a hint about what to record next; it refuses what the environment-state view refuses. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. The file may also be an Evidence bundle written by CLI --evidence_output or export_evidence_bundle; select evidence_id when it contains more than one record. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. Protobuf views use kind protobuf, collection bodies/frames/fields, with record_ordinals, body_pointers and fields; field pages also select frame_indices, field_ids, field_numbers or schema_names. Byte identity, offset basis and body status remain present. Text and schema_value are opt-in page columns. APK signing views use kind apk-signature and collection signers/digests/lineage, with ids, signer_ids, schemes, fields and paging. Identity links remain when fields is empty. Declared identities and digest matches do not establish signature verification or authenticity. Model views use kind model-artifact and collection metadata (keys), tensors (names/types) or types (derived totals), with indices, fields and paging. Header SHA-256 scopes declarations; full-file identity remains unknown unless full hashing was selected. Arrays retain typed eight-element previews with explicit omissions. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods/fields/properties/events/parameters/string_literals/code_gen_modules/method_addresses/return_types/field_types/parameter_types/generic_method_instances/generic_method_issues. Select indices, fields and pages; sections/images/types/methods/fields/properties/events/parameters/return_types/field_types/parameter_types/generic_method_instances accept exact names (native type names match retained rendered values, preserving same-name rows; null names never match), types also accept image_indices/namespaces/declaring_types, and methods/members accept declaring_types. Parameters and parameter_types accept method_indices; code_gen_modules accept image_indices; method_addresses and generic_method_instances accept addresses/states (generic_method_instances also method_indices), preserving every folded alternative; generic_method_issues accept rows. Row index is scoped to the metadata digest; native pages add binary_source. Uncollected native tables have null counts and remain unavailable. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native inventory Evidence from list_names, list/search_procedures or list/search_strings uses view: {kind: "native-inventory", offset: 0, limit: 64, fields: ["value"]}. Rows keep source_index and address even with selected fields. Summary counts describe retained rows; provider discovery coverage remains unknown. Native function Evidence accepts kind summary for identity, availability and facet counts, or kind native for selected facts: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', + 'Use javascript-routes collections to query route patterns, source-linked occurrences, source status, unknowns or summary counts from retained JavaScript application or web-bundle Evidence. Inspect selected views of supported completed Evidence without repeating analysis. Choose a view compatible with the parent operation described below. Command summary views preserve ordered GDB records or opaque objdump stdout/stderr once, source parameters, completion facts and parent identity. They do not establish current target state. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. Native export tables use kind native-exports, collection exports/issues, exact names or original indices, selected fields and matching-row pagination; parent Evidence plus source_index identifies each row. Source coverage stays separate from page coverage. BEAM views use kind beam, collections atoms/exports/imports/lambdas/chunks, exact names/indices/fields and matching-row pages; import modules intersects function names. source_index identifies original parent-and-collection rows, preserving every arity and raw atom index. beam.counts preserves null missing tables versus decoded empty tables; runtime absence remains unknown. PYC views use kind python-pyc, collection code_objects, names/indices/fields and paging; source_index scopes every same-name alternative to the parent, nested_code keeps code-constant references, and pyc.coverage preserves source coverage. Go metadata uses kind go, collection dependencies/settings, exact paths/keys or indices, fields and pagination. Row index refers to the original collection; empty selections return no rows, and dependency coverage remains unknown. Rust metadata uses kind rust, collection packages/edges, exact names/versions or original indices, selected fields, edge from_indices/to_indices and matching-row pagination. Embedded-record coverage is separate from unknown runtime dependencies. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent also supports summary and browser-scenario steps/events/transactions/text pages, preserving step_indices, event_kinds, transaction_ids and selected artifacts. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized reads that are not entities and have one initial request line (item-shaped paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state (a list also learns the filters, search, sort and paging its recordings verify, and answers new values of them from the state), and every other read tool is a recorded view with a reason. It also reports a write plan per state-changing tool (create, update, delete or document, built from its first successful call) and write_clock; a plan is served only when replaying the capture shows a later read confirming its effect and none contradicting it. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. The environment-coverage view, view {kind: "environment-coverage", template_rules: [...]}, summarizes that state without its entities: reads.state and reads.recorded count read tools answered from the state and replayed (recorded_reasons per view reason), writes.served lists the served write tools, and writes.withheld lists each other write tool with its reason, plan record, contradicting read and a hint about what to record next; it refuses what the environment-state view refuses. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. The file may also be an Evidence bundle written by CLI --evidence_output or export_evidence_bundle; select evidence_id when it contains more than one record. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. Protobuf views use kind protobuf, collection bodies/frames/fields, with record_ordinals, body_pointers and fields; field pages also select frame_indices, field_ids, field_numbers or schema_names. Byte identity, offset basis and body status remain present. Text and schema_value are opt-in page columns. APK signing views use kind apk-signature and collection signers/digests/lineage, with ids, signer_ids, schemes, fields and paging. Identity links remain when fields is empty. Declared identities and digest matches do not establish signature verification or authenticity. Model views use kind model-artifact and collection metadata (keys), tensors (names/types) or types (derived totals), with indices, fields and paging. Header SHA-256 scopes declarations; full-file identity remains unknown unless full hashing was selected. Arrays retain typed eight-element previews with explicit omissions. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods/fields/properties/events/parameters/string_literals/code_gen_modules/method_addresses/return_types/field_types/parameter_types/generic_method_instances/generic_method_issues. Select indices, fields and pages; sections/images/types/methods/fields/properties/events/parameters/return_types/field_types/parameter_types/generic_method_instances accept exact names (native type names match retained rendered values, preserving same-name rows; null names never match), types also accept image_indices/namespaces/declaring_types, and methods/members accept declaring_types. Parameters and parameter_types accept method_indices; code_gen_modules accept image_indices; method_addresses and generic_method_instances accept addresses/states (generic_method_instances also method_indices), preserving every folded alternative; generic_method_issues accept rows. Row index is scoped to the metadata digest; native pages add binary_source. Uncollected native tables have null counts and remain unavailable. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native inventory Evidence from list_names, list/search_procedures or list/search_strings uses view: {kind: "native-inventory", offset: 0, limit: 64, fields: ["value"]}. Rows keep source_index and address even with selected fields. Summary counts describe retained rows; provider discovery coverage remains unknown. Native function Evidence accepts kind summary for identity, availability and facet counts, or kind native for selected facts: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', inputSchema: inspectAnalysisViewInputSchema, outputSchema: evidenceResultOf(analysisViewResultSchema), examples: [ diff --git a/src/contracts/browserScenarioToolContracts.ts b/src/contracts/browserScenarioToolContracts.ts index 4e19ef730d..6955cb13df 100644 --- a/src/contracts/browserScenarioToolContracts.ts +++ b/src/contracts/browserScenarioToolContracts.ts @@ -1,6 +1,10 @@ import type { ToolContract } from "./toolContractTypes.js"; import { toolContractMetadata } from "./toolEffects.js"; -import { evidenceResultOf } from "./toolOutputSchemaPrimitives.js"; +import { + analysisDetailOutputSchemaOf, + analysisDetailSchema, +} from "./analysisDetail.js"; +import { browserScenarioSummaryViewResultSchema } from "../domain/analysisView/analysisView.js"; import { browserScenarioInputSchema } from "../domain/browserScenario.js"; import { browserScenarioCaptureSchema } from "../domain/browserScenarioCapture.js"; import type { JsonValue } from "../domain/jsonValue.js"; @@ -26,10 +30,15 @@ export const BROWSER_SCENARIO_TOOL_CONTRACTS = [ name: "capture_browser_scenario", ...toolContractMetadata("capture_browser_scenario"), description: - "Run a controlled browser scenario when passive observation cannot exercise the application. Launch the selected executable or connect to a loopback CDP target, provide a start URL and actions, and select storage or artifacts when needed. A request action sends one HTTP request to the start origin through the page's own fetch, with header values from secrets, page cookies or local storage; write methods change application state. Environment, empty storage, a URL-only final capture, and measured event-retention budgets are defaults. capture.network selects request_body, response_body, and header_values independently; capture.limits lets callers raise or lower the event count, retained-allocation estimates, network transaction count, and network content budgets. Selected content is returned inline with transaction and event references, byte digests, and explicit unavailable states. Budget omissions identify affected event kinds and network transactions. Credentials and declared secrets are redacted. Agents can run their own local fixture servers to control application responses. Actions run until completion or cancellation unless timeout_ms is set. Results are returned inline.", + "Run a controlled browser scenario when passive observation cannot exercise the application. Launch the selected executable or connect to a loopback CDP target, provide a start URL and actions, and select storage or artifacts when needed. A request action sends one HTTP request to the start origin through the page's own fetch, with header values from secrets, page cookies or local storage; write methods change application state. Environment, empty storage, a URL-only final capture, and measured event-retention budgets are defaults. capture.network selects request_body, response_body, and header_values independently; capture.limits lets callers raise or lower the event count, retained-allocation estimates, network transaction count, and network content budgets. Selected content is returned inline with transaction and event references, byte digests, and explicit unavailable states. Budget omissions identify affected event kinds and network transactions. Credentials and declared secrets are redacted. Agents can run their own local fixture servers to control application responses. Actions run until completion or cancellation unless timeout_ms is set. Default detail complete returns the capture inline. Detail summary retains complete Evidence and returns step outcomes, the final or first unsuccessful step’s accessibility tree, and event/network counts. Read exact steps with selected artifacts, UTF-16 text pages, transactions with opt-in content, or events filtered by step, kind and transaction, through inspect_analysis_view kind browser-scenario.", kind: "browser-provider", - inputSchema: browserScenarioInputSchema, - outputSchema: evidenceResultOf(browserScenarioCaptureSchema), + inputSchema: browserScenarioInputSchema.extend({ + detail: analysisDetailSchema, + }), + outputSchema: analysisDetailOutputSchemaOf( + browserScenarioCaptureSchema, + browserScenarioSummaryViewResultSchema, + ), examples: [ { title: "Capture a deterministic browser scenario", input: example }, ], diff --git a/src/domain/analysisView/analysisView.ts b/src/domain/analysisView/analysisView.ts index a77aa29f4d..d9d2a8b13d 100644 --- a/src/domain/analysisView/analysisView.ts +++ b/src/domain/analysisView/analysisView.ts @@ -122,6 +122,13 @@ import { import { absoluteLocalPathStringSchema } from "../localPath.js"; import { z } from "zod"; +import { + browserScenarioSummarySchema, + browserScenarioViewContextSchema, + browserScenarioViewItemsSchema, + browserScenarioViewRequestSchema, +} from "./browserScenarioViewSchemas.js"; +import { projectBrowserScenarioView } from "./browserScenarioView.js"; import { desktopViewRequestSchema, desktopViewItemsSchema, @@ -393,6 +400,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [ environmentStateViewRequestSchema, environmentCoverageViewRequestSchema, captureViewRequestSchema, + browserScenarioViewRequestSchema, protobufViewRequestSchema, qlogViewRequestSchema, pcapViewRequestSchema, @@ -729,6 +737,14 @@ export const webNetworkCaptureSummaryViewResultSchema = z.strictObject({ summary: captureSummarySchema, }); +/** Compact outcomes and the focus screen from a retained browser scenario. */ +export const browserScenarioSummaryViewResultSchema = z.strictObject({ + ...summaryViewResultBase, + artifact: z.null(), + parent_operation: z.literal("capture_browser_scenario"), + summary: browserScenarioSummarySchema, +}); + /** Exact package declarations with retained decoded manifest text. */ export const androidPackageSummaryViewResultSchema = z.strictObject({ ...summaryViewResultBase, @@ -1022,6 +1038,7 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ managedArtifactSummaryViewResultSchema, managedMemberSummaryViewResultSchema, webNetworkCaptureSummaryViewResultSchema, + browserScenarioSummaryViewResultSchema, androidPackageSummaryViewResultSchema, pcapCaptureSummaryViewResultSchema, dartAotSummaryViewResultSchema, @@ -1076,6 +1093,15 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ ...viewResultBase, parent_operation: z.literal("inspect_web_network_capture"), }), + z.strictObject({ + kind: z.literal("browser-scenario"), + view: browserScenarioViewRequestSchema, + items: browserScenarioViewItemsSchema, + ...viewResultBase, + ...browserScenarioViewContextSchema.shape, + artifact: z.null(), + parent_operation: z.literal("capture_browser_scenario"), + }), z.strictObject({ kind: z.literal("qlog"), view: qlogViewRequestSchema, @@ -1283,9 +1309,7 @@ const unsupportedParent = ( new AnalysisUnsupportedTargetError( INSPECT_ANALYSIS_VIEW_OPERATION, parentArtifactPath(parent), - parent.operation === "capture_browser_scenario" - ? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest, environment-state and environment-coverage views.` - : `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed Evidence from a supported parent operation with a compatible view. For a summary receipt, use its normalized_result.parent_evidence_id rather than the receipt's own evidence_id.`, + `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed Evidence from a supported parent operation with a compatible view. For a summary receipt, use its normalized_result.parent_evidence_id rather than the receipt's own evidence_id.`, ); const parentArtifactPath = (parent: AnalysisViewParent): string => { @@ -1331,6 +1355,7 @@ const retainedViewProjectors = new Map([ ["inspect_managed_artifact", projectManagedArtifactView], ["inspect_managed_members", projectManagedMemberView], ["inspect_android_package", projectAndroidPackageView], + ["capture_browser_scenario", projectBrowserScenarioView], ["inspect_dart_aot", projectDartAotView], ["inspect_il2cpp_metadata", projectIl2cppMetadataView], ["inspect_apk_signature", projectApkSignatureView], diff --git a/src/domain/analysisView/browserScenarioNetworkView.ts b/src/domain/analysisView/browserScenarioNetworkView.ts new file mode 100644 index 0000000000..34be1d06ea --- /dev/null +++ b/src/domain/analysisView/browserScenarioNetworkView.ts @@ -0,0 +1,193 @@ +import { z } from "zod"; +import type { RetainedBrowserScenarioCapture } from "../browserScenarioCapture.js"; +import { isImmutableJsonSnapshot } from "../immutableJson.js"; +import { compareAsciiStrings } from "../canonicalOrdering.js"; + +const count = z.int().nonnegative(); +const statusClass = z.enum([ + "1xx", + "2xx", + "3xx", + "4xx", + "5xx", + "failed", + "unknown", +]); + +/** Counts concern retained transaction identities, including those without a request event. */ +export const browserScenarioNetworkSummarySchema = z.strictObject({ + selected: z.boolean().nullable(), + transactions: count.nullable(), + without_request: count, + ambiguous_transactions: count, + failed_transactions: count, + by_origin: z.array( + z.strictObject({ origin: z.string().nullable(), transactions: count }), + ), + by_resource_type: z.array( + z.strictObject({ + resource_type: z.string().nullable(), + transactions: count, + }), + ), + by_status_class: z.array( + z.strictObject({ status_class: statusClass, transactions: count }), + ), +}); + +interface TransactionGroup { + readonly transaction_id: string; + readonly source_item_indices: readonly number[]; + readonly origin: string | null; + readonly resource_type: string | null; + readonly status_class: z.infer; + readonly has_request: boolean; + readonly failed: boolean; + readonly ambiguous: boolean; +} + +interface TransactionFacts { + indices: number[]; + origins: Set; + resourceTypes: Set; + statuses: Set; + requests: number; + failed: boolean; +} + +/** Sealed historical captures own this metadata-only index until their retained lifetime ends. */ +const transactionsByCapture = new WeakMap< + object, + readonly TransactionGroup[] +>(); + +/** Keep all observed alternatives; one missing event never erases its transaction. */ +export const browserScenarioTransactionGroups = ( + capture: RetainedBrowserScenarioCapture, +): readonly TransactionGroup[] => { + const reusable = isImmutableJsonSnapshot(capture); + const known = reusable ? transactionsByCapture.get(capture) : undefined; + if (known !== undefined) return known; + const groups = new Map(); + for (const [index, event] of capture.events.items.entries()) { + if (!("transaction_id" in event)) continue; + let group = groups.get(event.transaction_id); + if (group === undefined) { + group = { + indices: [], + origins: new Set(), + resourceTypes: new Set(), + statuses: new Set(), + requests: 0, + failed: false, + }; + groups.set(event.transaction_id, group); + } + group.indices.push(index); + if (event.kind === "network-content") continue; + group.origins.add(event.url.origin); + group.resourceTypes.add(event.resource_type); + if (event.kind === "request") group.requests += 1; + if (event.kind === "response") group.statuses.add(event.status); + if (event.kind === "request-failed") group.failed = true; + } + const result = [...groups].map( + ([transaction_id, group]): TransactionGroup => { + const status = group.statuses.values().next().value; + return Object.freeze({ + transaction_id, + source_item_indices: Object.freeze(group.indices), + origin: + group.origins.size === 1 + ? (group.origins.values().next().value ?? null) + : null, + resource_type: + group.resourceTypes.size === 1 + ? (group.resourceTypes.values().next().value ?? null) + : null, + status_class: + group.statuses.size === 1 && status !== undefined + ? (`${String(Math.floor(status / 100))}xx` as z.infer< + typeof statusClass + >) + : group.statuses.size === 0 && group.failed + ? "failed" + : "unknown", + has_request: group.requests > 0, + failed: group.failed, + ambiguous: + group.requests > 1 || + group.statuses.size > 1 || + group.origins.size > 1 || + group.resourceTypes.size > 1, + }); + }, + ); + if (reusable) transactionsByCapture.set(capture, result); + return result; +}; + +const increment = (values: Map, key: Key): void => { + values.set(key, (values.get(key) ?? 0) + 1); +}; +const sorted = ( + values: ReadonlyMap, +): [Key, number][] => + [...values].sort(([left], [right]) => + left === right + ? 0 + : left === null + ? -1 + : right === null + ? 1 + : compareAsciiStrings(left, right), + ); + +/** Null selection/counts preserve legacy unknowns and unrequested network observation. */ +export const summarizeBrowserScenarioNetwork = ( + capture: RetainedBrowserScenarioCapture, +): z.infer => { + const groups = browserScenarioTransactionGroups(capture); + const contentSelected = Object.values(capture.scenario.network_content).some( + Boolean, + ); + const selected = + contentSelected || + capture.scenario.event_families?.includes("network") === true + ? true + : capture.scenario.event_families === undefined + ? null + : false; + const origins = new Map(); + const resources = new Map(); + const statuses = new Map, number>(); + let withoutRequest = 0; + let ambiguous = 0; + let failed = 0; + for (const group of groups) { + if (!group.has_request) withoutRequest += 1; + if (group.ambiguous) ambiguous += 1; + if (group.failed) failed += 1; + increment(origins, group.origin); + increment(resources, group.resource_type); + increment(statuses, group.status_class); + } + return { + selected, + transactions: selected === true || groups.length > 0 ? groups.length : null, + without_request: withoutRequest, + ambiguous_transactions: ambiguous, + failed_transactions: failed, + by_origin: sorted(origins).map(([origin, transactions]) => ({ + origin, + transactions, + })), + by_resource_type: sorted(resources).map( + ([resource_type, transactions]) => ({ resource_type, transactions }), + ), + by_status_class: sorted(statuses).map(([status_class, transactions]) => ({ + status_class, + transactions, + })), + }; +}; diff --git a/src/domain/analysisView/browserScenarioView.ts b/src/domain/analysisView/browserScenarioView.ts new file mode 100644 index 0000000000..4b66c6bf82 --- /dev/null +++ b/src/domain/analysisView/browserScenarioView.ts @@ -0,0 +1,388 @@ +import type { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { + retainedBrowserScenarioCaptureSchema, + type RetainedBrowserScenarioCapture as BrowserScenarioCapture, +} from "../browserScenarioCapture.js"; +import type { BrowserScenarioStep as Step } from "../browserScenarioCaptureValues.js"; +import { compareAsciiStrings } from "../canonicalOrdering.js"; +import { err, ok, type Result } from "../result.js"; +import { analysisViewInputError } from "./analysisViewErrors.js"; +import { + completeWithinViewCoverage, + pageViewCoverage, +} from "./analysisViewCoverage.js"; +import { createAnalysisViewResultReader } from "./analysisViewParent.js"; +import type { + AnalysisViewCoverage, + AnalysisViewParent, + AnalysisViewRequest, + UnsignedAnalysisView, +} from "./analysisView.js"; +import { + ARTIFACT_SECTIONS, + stepRowSchema, + browserScenarioSummarySchema, + stepSectionsSchema, + summaryStepSchema, + focusAccessibilitySchema, + browserScenarioViewItemsSchema, + type BrowserScenarioViewRequest, + type ArtifactSection, +} from "./browserScenarioViewSchemas.js"; + +import { + browserScenarioTransactionGroups, + summarizeBrowserScenarioNetwork, +} from "./browserScenarioNetworkView.js"; + +const OPERATION = "capture_browser_scenario" as const; +const readCapture = createAnalysisViewResultReader( + retainedBrowserScenarioCaptureSchema, +); + +const unsupportedView = (): AnalysisError => + analysisViewInputError([ + { + path: ["view"], + reason: "invalid_value", + message: + 'Parent Evidence operation "capture_browser_scenario" supports summary and browser-scenario views, plus compatible environment views.', + }, + ]); + +const sameUrl = ( + left: Step["after_url"] | undefined, + right: Step["before_url"], +): boolean => + left !== undefined && + left.url === right.url && + left.redacted === right.redacted; + +const stepSections = (step: Step): z.infer => { + const sections: z.infer = { + captured: [], + missing: [], + truncated: [], + }; + for (const section of ARTIFACT_SECTIONS) { + const { state } = step.artifacts[section]; + if (state !== "not_requested") sections[state].push(section); + } + return sections; +}; + +const summaryStep = ( + step: Step, + previous: Step | undefined, +): z.infer => ({ + step_index: step.step_index, + step_id: step.step_id, + action: step.action, + status: step.status, + error: step.error, + elapsed_ms: step.elapsed_ms, + ...(sameUrl(previous?.after_url, step.before_url) + ? {} + : { before_url: step.before_url }), + after_url: step.after_url, + events: { + sequence_start: step.event_sequence_start, + sequence_end: step.event_sequence_end, + }, + sections: stepSections(step), + completeness: step.completeness.status, +}); + +/** The first unsuccessful step explains a failure; otherwise the final screen does. */ +const focusAccessibility = ( + steps: readonly Step[], +): z.infer => { + const unsuccessful = steps.findIndex(({ status }) => status !== "completed"); + const focusIndex = unsuccessful === -1 ? steps.length - 1 : unsuccessful; + const focus = steps[focusIndex]; + if (focus === undefined) throw new Error("Capture has no steps"); + const kind = + unsuccessful === -1 ? ("last-step" as const) : "first-unsuccessful-step"; + for (let index = focusIndex; index >= 0; index -= 1) { + const accessibility = steps[index]?.artifacts.accessibility; + if (accessibility?.state === "captured") + return { + focus_step_index: focusIndex, + focus: kind, + step_index: index, + accessibility, + }; + } + return { + focus_step_index: focusIndex, + focus: kind, + step_index: null, + accessibility: focus.artifacts.accessibility, + }; +}; + +/** Counts in canonical key order; a null origin sorts first. */ +const sortedCounts = ( + counts: ReadonlyMap, +): [Key, number][] => + [...counts].sort(([left], [right]) => + left === right + ? 0 + : left === null + ? -1 + : right === null + ? 1 + : compareAsciiStrings(left, right), + ); + +const increment = (counts: Map, key: Key): void => { + counts.set(key, (counts.get(key) ?? 0) + 1); +}; + +const eventCounts = ( + capture: BrowserScenarioCapture, +): z.infer["events"] => { + const kinds = new Map< + BrowserScenarioCapture["events"]["items"][number]["kind"], + number + >(); + for (const event of capture.events.items) increment(kinds, event.kind); + const { omissions } = capture.events; + return { + retained: capture.events.retained, + dropped: capture.events.dropped, + by_kind: sortedCounts(kinds).map(([kind, count]) => ({ kind, count })), + omissions: { + by_kind: omissions.by_kind, + network_transactions: omissions.network_transactions.length, + network_transaction_range: omissions.network_transaction_range, + }, + }; +}; + +const selected = ( + filter: readonly Value[] | undefined, +): ((value: Value) => boolean) => { + if (filter === undefined) return () => true; + const values = new Set(filter); + return (value) => values.has(value); +}; + +const stepRow = ( + { artifacts: recorded, ...step }: Step, + selection: ReadonlySet, +): z.infer => { + const artifacts: z.infer["artifacts"] = {}; + const omitted: z.infer["omitted_artifacts"] = []; + for (const section of ARTIFACT_SECTIONS) { + const artifact = recorded[section]; + if (selection.has(section)) + Object.assign(artifacts, { [section]: artifact }); + else if (artifact.state !== "not_requested") + omitted.push({ section, state: artifact.state }); + } + return { ...step, artifacts, omitted_artifacts: omitted }; +}; + +/** Scan matching rows once and materialize only the requested page. */ +const selectedPage = ( + source: readonly Input[], + view: { readonly offset: number; readonly limit: number }, + matches: (item: Input, index: number) => boolean, + project: (item: Input, index: number) => Row, +): { rows: Row[]; coverage: AnalysisViewCoverage } => { + const rows: Row[] = []; + let total = 0; + for (const [index, item] of source.entries()) { + if (!matches(item, index)) continue; + if (total >= view.offset && rows.length < view.limit) + rows.push(project(item, index)); + total += 1; + } + return { rows, coverage: pageViewCoverage(view.offset, rows.length, total) }; +}; + +/** Text offsets are UTF-16 positions in the exact digest-bound retained artifact. */ +const textPage = ( + capture: BrowserScenarioCapture, + view: Extract, +) => { + const artifact = capture.steps[view.step_index]!.artifacts[view.artifact]; + const value = artifact.state === "captured" ? artifact.value : null; + const text = value?.text.slice(view.offset, view.offset + view.limit) ?? null; + return { + items: { + collection: "text" as const, + step_index: view.step_index, + artifact: view.artifact, + state: artifact.state, + reason: "reason" in artifact ? artifact.reason : null, + sha256: value?.sha256 ?? null, + bytes: value?.bytes ?? null, + observed: artifact.state === "truncated" ? artifact.observed : null, + retained: artifact.state === "truncated" ? artifact.retained : null, + text, + }, + coverage: pageViewCoverage( + view.offset, + text?.length ?? 0, + value?.text.length ?? 0, + ), + }; +}; + +/** One stable page of exact recorded facts; no whole-result filtering arrays. */ +const pageRecords = ( + capture: BrowserScenarioCapture, + view: BrowserScenarioViewRequest, +): { + readonly items: z.infer; + readonly coverage: AnalysisViewCoverage; +} => { + if (view.collection === "text") return textPage(capture, view); + const stepSelected = selected(view.step_indices); + if (view.collection === "steps") { + const artifacts = new Set(view.artifacts); + const page = selectedPage( + capture.steps, + view, + ({ step_index }) => stepSelected(step_index), + (step) => stepRow(step, artifacts), + ); + return { + items: { collection: "steps", rows: page.rows }, + coverage: page.coverage, + }; + } + const kindSelected = selected(view.event_kinds); + const transactionSelected = selected(view.transaction_ids); + if (view.collection === "transactions") { + const page = selectedPage( + browserScenarioTransactionGroups(capture), + view, + (group) => + transactionSelected(group.transaction_id) && + (view.step_indices === undefined || + group.source_item_indices.some((index) => + stepSelected(capture.events.items[index]!.step_index), + )), + (group) => { + const events: { + item_index: number; + event: BrowserScenarioCapture["events"]["items"][number]; + }[] = []; + const omitted: { + item_index: number; + kind: BrowserScenarioCapture["events"]["items"][number]["kind"]; + }[] = []; + for (const item_index of group.source_item_indices) { + const event = capture.events.items[item_index]!; + if (kindSelected(event.kind)) events.push({ item_index, event }); + else omitted.push({ item_index, kind: event.kind }); + } + return { + transaction_id: group.transaction_id, + source_item_indices: [...group.source_item_indices], + events, + omitted_events: omitted, + }; + }, + ); + return { + items: { collection: "transactions", rows: page.rows }, + coverage: page.coverage, + }; + } + const page = selectedPage( + capture.events.items, + view, + (event) => + stepSelected(event.step_index) && + kindSelected(event.kind) && + (view.transaction_ids === undefined || + ("transaction_id" in event && + transactionSelected(event.transaction_id))), + (event, item_index) => ({ item_index, event }), + ); + return { + items: { collection: "events", rows: page.rows }, + coverage: page.coverage, + }; +}; + +/** Project retained scenario Evidence without relaunching the browser or replaying actions. */ +export const projectBrowserScenarioView = ( + parent: AnalysisViewParent, + view: AnalysisViewRequest, +): Result => { + if (view.kind !== "summary" && view.kind !== "browser-scenario") + return err(unsupportedView()); + const parsed = readCapture(parent); + if (!parsed.ok) return parsed; + const capture = parsed.value; + const shared = { + parent_evidence_id: parent.evidenceId, + parent_operation: OPERATION, + parent_digest: parent.evidenceId.slice(3), + artifact: null, + limitations: [...parent.limitations], + unknowns: [ + "Step states record one controlled replay; they do not establish the application's current state.", + "Event step references record receipt attribution; they do not prove an action caused a transaction.", + ...(capture.events.dropped > 0 + ? [ + `${String(capture.events.dropped)} events were dropped by capture budgets; counts and pages cover retained events only.`, + ] + : []), + ], + }; + if (view.kind === "summary") + return ok({ + kind: "summary", + view, + ...shared, + summary: { + browser: capture.browser, + scenario: capture.scenario, + duration_ms: capture.duration_ms, + completeness: capture.completeness, + steps: capture.steps.map((step, index) => + summaryStep(step, capture.steps[index - 1]), + ), + final_accessibility: focusAccessibility(capture.steps), + events: eventCounts(capture), + network: summarizeBrowserScenarioNetwork(capture), + limitation_count: shared.limitations.length, + }, + coverage: completeWithinViewCoverage(1, 1), + }); + if ( + view.collection === "text" && + capture.steps[view.step_index] === undefined + ) + return err( + analysisViewInputError([ + { + path: ["view", "step_index"], + reason: "invalid_value", + message: "The selected step is not present in the retained capture.", + }, + ]), + ); + return ok({ + kind: "browser-scenario", + view, + ...shared, + source_completeness: capture.completeness, + source_events: { + retained: capture.events.retained, + dropped: capture.events.dropped, + }, + source_network_omissions: { + transactions: capture.events.omissions.network_transactions.length, + transaction_range: capture.events.omissions.network_transaction_range, + }, + ...pageRecords(capture, view), + }); +}; diff --git a/src/domain/analysisView/browserScenarioViewSchemas.ts b/src/domain/analysisView/browserScenarioViewSchemas.ts new file mode 100644 index 0000000000..1e483dd304 --- /dev/null +++ b/src/domain/analysisView/browserScenarioViewSchemas.ts @@ -0,0 +1,250 @@ +import { z } from "zod"; +import { digestSchema } from "../digests.js"; +import { browserScenarioNetworkSummarySchema } from "./browserScenarioNetworkView.js"; +import { sanitizedBrowserUrlSchema } from "../browserObservation.js"; +import { retainedBrowserScenarioCaptureSchema } from "../browserScenarioCapture.js"; +import { + browserScenarioCompletenessSchema, + browserScenarioEventKindSchema, + browserScenarioEventSchema, + browserStepArtifactsSchema, +} from "../browserScenarioCaptureValues.js"; + +export const ARTIFACT_SECTIONS = browserStepArtifactsSchema.keyof().options; +const artifactSectionSchema = browserStepArtifactsSchema.keyof(); +export type ArtifactSection = z.infer; + +const countSchema = z.number().int().nonnegative(); +/** Section names by recorded state; not_requested sections are omitted. */ +export const stepSectionsSchema = z.strictObject({ + captured: z.array(artifactSectionSchema), + missing: z.array(artifactSectionSchema), + truncated: z.array(artifactSectionSchema), +}); + +export const summaryStepSchema = z.strictObject({ + step_index: countSchema, + step_id: z.string().min(1), + action: z.string().min(1), + status: z.enum(["completed", "failed", "cancelled"]), + error: z.string().nullable(), + elapsed_ms: countSchema, + before_url: sanitizedBrowserUrlSchema + .exactOptional() + .describe( + "Present only when it differs from the previous step's after_url (always for step 0).", + ), + after_url: sanitizedBrowserUrlSchema, + events: z.strictObject({ + sequence_start: z.number().int().min(1), + sequence_end: countSchema, + }), + sections: stepSectionsSchema, + completeness: z.enum(["complete", "incomplete", "truncated"]), +}); + +export const focusAccessibilitySchema = z.strictObject({ + focus_step_index: countSchema, + focus: z.enum(["first-unsuccessful-step", "last-step"]), + /** The step whose tree is returned: the focus step, or the nearest earlier step that captured one. */ + step_index: countSchema.nullable(), + accessibility: browserStepArtifactsSchema.shape.accessibility, +}); + +/** Compact capture overview; complete records stay in the retained parent. */ +export const browserScenarioSummarySchema = z.strictObject({ + browser: retainedBrowserScenarioCaptureSchema.shape.browser, + scenario: retainedBrowserScenarioCaptureSchema.shape.scenario, + duration_ms: countSchema, + completeness: browserScenarioCompletenessSchema, + steps: z.array(summaryStepSchema), + final_accessibility: focusAccessibilitySchema, + events: z.strictObject({ + retained: countSchema, + dropped: countSchema, + by_kind: z.array( + z.strictObject({ + kind: browserScenarioEventKindSchema, + count: countSchema, + }), + ), + omissions: z.strictObject({ + by_kind: + retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape + .by_kind, + network_transactions: countSchema, + network_transaction_range: + retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape + .network_transaction_range, + }), + }), + network: browserScenarioNetworkSummarySchema, + limitation_count: countSchema.describe( + "Number of parent Evidence limitations returned with this summary, including inherited provenance caveats.", + ), +}); + +const pageShape = { + kind: z.literal("browser-scenario"), + offset: countSchema.default(0), + limit: z.int().positive().default(64), +}; +const stepSelection = { + step_indices: z + .array(countSchema) + .optional() + .describe( + "Exact recorded step indices; omitted selects all, [] selects none.", + ), +}; +const transactionSelection = { + transaction_ids: z + .array(z.string().min(1)) + .optional() + .describe( + "Exact recorded transaction IDs; omitted selects all, [] selects none.", + ), +}; + +/** Collection-specific fields are validated rather than silently ignored. */ +export const browserScenarioViewRequestSchema = z.discriminatedUnion( + "collection", + [ + z.strictObject({ + ...pageShape, + ...stepSelection, + collection: z.literal("steps"), + artifacts: z + .array(artifactSectionSchema) + .default([]) + .describe( + "Return only selected artifact payloads; omitted_artifacts names the other recorded sections.", + ), + }), + z.strictObject({ + ...pageShape, + ...stepSelection, + ...transactionSelection, + collection: z.literal("events"), + event_kinds: z + .array(browserScenarioEventKindSchema) + .optional() + .describe("Omitted selects all event kinds; [] selects none."), + }), + z.strictObject({ + ...pageShape, + ...stepSelection, + ...transactionSelection, + collection: z.literal("transactions"), + event_kinds: z + .array(browserScenarioEventKindSchema) + .default([ + "request", + "response", + "request-finished", + "request-failed", + "request-unfinished", + ]) + .describe( + "Metadata events by default; select network-content explicitly for bodies and header values. [] returns identity and source indices only.", + ), + }), + z.strictObject({ + ...pageShape, + collection: z.literal("text"), + step_index: countSchema, + artifact: z.enum(["accessibility", "dom"]), + limit: z + .int() + .positive() + .default(2048) + .describe("Caller-selected UTF-16 code units, matching offset."), + }), + ], +); +export type BrowserScenarioViewRequest = z.output< + typeof browserScenarioViewRequestSchema +>; + +/** One exact step whose unselected captured sections are named rather than returned. */ +export const stepRowSchema = z.strictObject({ + step_index: countSchema, + step_id: z.string().min(1), + action: z.string().min(1), + status: z.enum(["completed", "failed", "cancelled"]), + error: z.string().nullable(), + elapsed_ms: countSchema, + before_url: sanitizedBrowserUrlSchema, + after_url: sanitizedBrowserUrlSchema, + event_sequence_start: z.number().int().min(1), + event_sequence_end: countSchema, + completeness: browserScenarioCompletenessSchema, + artifacts: browserStepArtifactsSchema.partial(), + omitted_artifacts: z.array( + z.strictObject({ + section: artifactSectionSchema, + state: z.enum(["captured", "missing", "truncated"]), + }), + ), +}); +const eventRowSchema = z.strictObject({ + item_index: countSchema, + event: browserScenarioEventSchema, +}); + +export const browserScenarioViewItemsSchema = z.discriminatedUnion( + "collection", + [ + z.strictObject({ + collection: z.literal("steps"), + rows: z.array(stepRowSchema), + }), + z.strictObject({ + collection: z.literal("events"), + rows: z.array(eventRowSchema), + }), + z.strictObject({ + collection: z.literal("transactions"), + rows: z.array( + z.strictObject({ + transaction_id: z.string().min(1), + source_item_indices: z.array(countSchema), + events: z.array(eventRowSchema), + omitted_events: z.array( + z.strictObject({ + item_index: countSchema, + kind: browserScenarioEventKindSchema, + }), + ), + }), + ), + }), + z.strictObject({ + collection: z.literal("text"), + step_index: countSchema, + artifact: z.enum(["accessibility", "dom"]), + state: z.enum(["captured", "missing", "truncated", "not_requested"]), + reason: z.string().nullable(), + sha256: digestSchema.nullable(), + bytes: countSchema.nullable(), + observed: countSchema.nullable(), + retained: countSchema.nullable(), + text: z.string().nullable(), + }), + ], +); + +/** Capture-wide facts every page repeats so exhaustion is not read as completeness. */ +export const browserScenarioViewContextSchema = z.strictObject({ + source_completeness: browserScenarioCompletenessSchema, + source_events: z.strictObject({ + retained: countSchema, + dropped: countSchema, + }), + source_network_omissions: z.strictObject({ + transactions: countSchema, + transaction_range: + retainedBrowserScenarioCaptureSchema.shape.events.shape.omissions.shape + .network_transaction_range, + }), +}); diff --git a/src/server/createServer.ts b/src/server/createServer.ts index bb9f772b7e..86180f3965 100644 --- a/src/server/createServer.ts +++ b/src/server/createServer.ts @@ -1290,6 +1290,7 @@ const registerObservationTools = ({ }); registerBrowserScenarioTool(server, { ...common, + evidenceById, provider: options.browserScenarioCapture, loadFailure: options.optionalProviderLoadFailures?.browserScenarioCapture, }); diff --git a/src/server/registerBrowserScenarioTool.ts b/src/server/registerBrowserScenarioTool.ts index 8c2015bc4a..811bb9f5f5 100644 --- a/src/server/registerBrowserScenarioTool.ts +++ b/src/server/registerBrowserScenarioTool.ts @@ -1,82 +1,67 @@ import type { EvidenceMcpServer } from "./EvidenceMcpServer.js"; import type { EvidenceWriter } from "../application/investigation/InvestigationRecordPort.js"; +import type { EvidenceLookup } from "../application/EvidenceInputResolver.js"; import { optionalProviderUnavailable, type OptionalProviderLoadFailure, } from "../application/OptionalObservationProviders.js"; import { err } from "../domain/result.js"; - +import { browserScenarioSchema } from "../domain/browserScenario.js"; +import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; import type { BrowserScenarioCapturePort } from "../application/BrowserScenarioCapturePort.js"; import { captureBrowserScenario } from "../application/BrowserScenarioCaptureService.js"; import { toolContract } from "../contracts/toolContracts.js"; -import { analysisInputErrorFromIssues } from "../domain/inputIssueProjection.js"; -import { browserScenarioSchema } from "../domain/browserScenario.js"; import type { Logger } from "pino"; -import { logToolExecution } from "./toolLogging.js"; import { toolRegistrationOptions } from "./toolRegistrationOptions.js"; import type { WithAdmittedAnalysis } from "./analysisAdmission.js"; -import { runAdmittedToolOperation } from "./admittedToolOperation.js"; -import { deliverEvidenceResult } from "./contractToolHandler.js"; +import { createRetainedSummaryToolHandler } from "./contractToolHandler.js"; interface BrowserScenarioToolRegistration { readonly logger: Logger; readonly loadFailure?: OptionalProviderLoadFailure | undefined; readonly provider: BrowserScenarioCapturePort | undefined; readonly recordEvidence: EvidenceWriter["recordEvidence"] | undefined; + readonly evidenceById?: EvidenceLookup | undefined; readonly withAdmittedAnalysis?: WithAdmittedAnalysis; } -/** Register the browser scenario tool with execution-time provider diagnostics. */ +/** Validate one scenario, then share complete/retained delivery with other analyses. */ export const registerBrowserScenarioTool = ( server: EvidenceMcpServer, options: BrowserScenarioToolRegistration, ): void => { const contract = toolContract("capture_browser_scenario"); + const handler = createRetainedSummaryToolHandler( + server, + contract, + async (input, execution) => { + // The SDK admits the advertised structure. The canonical semantic parser + // owns field-path diagnostics, shared with the CLI rather than flattened + // into the SDK's pre-handler validation message. + const scenario = browserScenarioSchema.safeParse(input); + if (!scenario.success) + return err( + analysisInputErrorFromIssues( + contract.name, + scenario.error.issues, + input, + { cause: scenario.error }, + ), + ); + return options.loadFailure === undefined + ? captureBrowserScenario(options.provider, scenario.data, execution) + : err(optionalProviderUnavailable(options.loadFailure, contract.name)); + }, + options.logger, + { + recordEvidence: options.recordEvidence, + evidenceById: options.evidenceById, + }, + options.withAdmittedAnalysis, + ); server.registerTool( contract.name, toolRegistrationOptions(contract), - async (input, context) => { - const parsedScenario = browserScenarioSchema.safeParse(input); - if (!parsedScenario.success) - return server.delivery.toCallToolResult( - err( - analysisInputErrorFromIssues( - contract.name, - parsedScenario.error.issues, - input, - { cause: parsedScenario.error }, - ), - ), - contract, - ); - const scenario = parsedScenario.data; - if (options.loadFailure !== undefined) - return server.delivery.toCallToolResult( - err(optionalProviderUnavailable(options.loadFailure, contract.name)), - contract, - ); - return runAdmittedToolOperation( - server, - options.withAdmittedAnalysis, - contract.name, - context.mcpReq.signal, - async () => { - const result = await logToolExecution( - options.logger, - contract.name, - () => - captureBrowserScenario(options.provider, scenario, { - signal: context.mcpReq.signal, - }), - ); - return deliverEvidenceResult( - server, - result, - contract, - options.recordEvidence, - ); - }, - ); - }, + handler, ); }; diff --git a/tests/boundary/cli/browserScenarioSummaryCli.test.ts b/tests/boundary/cli/browserScenarioSummaryCli.test.ts new file mode 100644 index 0000000000..905b37bd78 --- /dev/null +++ b/tests/boundary/cli/browserScenarioSummaryCli.test.ts @@ -0,0 +1,146 @@ +import { access } from "node:fs/promises"; +import { join } from "node:path"; +import { Cli } from "incur"; +import pino from "pino"; +import { expect, it } from "vitest"; + +import { registerBrowserScenarioCommands } from "../../../src/cliBrowserScenarioCommands.js"; +import { ok } from "../../../src/domain/result.js"; +import { ProviderCleanupError } from "../../../src/domain/providerCleanupError.js"; +import { scenarioCaptureFixture } from "../../fixtures/browserScenarioCapture.js"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; + +it.each(["complete", "summary"] as const)( + "retains complete Evidence on cancellation during CLI cleanup with %s delivery", + async (detail) => { + const root = await createTestTempDirectory("rea-browser-summary-cli-"); + const destination = join(root, "evidence.json"); + const originalExit = process.exitCode; + const listeners = new Set(process.listeners("SIGINT")); + const cli = Cli.create("rea-browser-summary-test", { sync: false }); + let stdout = ""; + let closed = false; + registerBrowserScenarioCommands( + cli, + pino({ level: "silent" }), + {}, + { + createProvider: () => ({ + identity: () => ({ id: "fixture", name: "Fixture", version: "1" }), + captureScenario: async () => ok(scenarioCaptureFixture()), + close: async () => { + closed = true; + // Invoke only this command's newly installed listener, without emitting + // a process-wide signal into the shared test worker. + const installed = process + .listeners("SIGINT") + .filter((listener) => !listeners.has(listener)); + expect(installed).toHaveLength(1); + installed[0]!("SIGINT"); + }, + }), + }, + ); + try { + await cli.serve( + [ + "capture-browser-scenario", + JSON.stringify({ + browser: { mode: "launch", executable_path: process.execPath }, + start_url: { url: "http://127.0.0.1:3101/" }, + actions: [ + { step_id: "browse", action: "wait_for_timeout", duration_ms: 1 }, + ], + }), + "--detail", + detail, + "--evidence_output", + destination, + "--json", + ], + { + env: {}, + stdout: (text) => { + stdout += text; + }, + exit: () => undefined, + }, + ); + expect(JSON.parse(stdout)).toMatchObject({ + code: "cancelled", + details: { + partial_observation: { + operation: "capture_browser_scenario", + normalized_result: { steps: expect.any(Array) }, + }, + }, + }); + expect(closed).toBe(true); + await expect(access(destination)).rejects.toMatchObject({ + code: "ENOENT", + }); + expect(new Set(process.listeners("SIGINT"))).toEqual(listeners); + } finally { + process.exitCode = originalExit; + } + }, +); + +it("keeps the provider's cleanup failure and completed Evidence without publishing a summary bundle", async () => { + const root = await createTestTempDirectory("rea-browser-summary-cleanup-"); + const destination = join(root, "evidence.json"); + const cli = Cli.create("rea-browser-cleanup-test", { sync: false }); + let stdout = ""; + const failure = new ProviderCleanupError( + "owned-browser", + ["owned-profile"], + { reason: "cleanup denied" }, + { operation: "capture_browser_scenario" }, + ); + registerBrowserScenarioCommands( + cli, + pino({ level: "silent" }), + {}, + { + createProvider: () => ({ + identity: () => ({ id: "fixture", name: "Fixture", version: "1" }), + captureScenario: async () => ok(scenarioCaptureFixture()), + close: async () => { + throw failure; + }, + }), + }, + ); + await cli.serve( + [ + "capture-browser-scenario", + JSON.stringify({ + browser: { mode: "launch", executable_path: process.execPath }, + start_url: { url: "http://127.0.0.1:3101/" }, + actions: [ + { step_id: "browse", action: "wait_for_timeout", duration_ms: 1 }, + ], + }), + "--detail", + "summary", + "--evidence_output", + destination, + "--json", + ], + { + env: {}, + stdout: (text) => { + stdout += text; + }, + exit: () => undefined, + }, + ); + expect(JSON.parse(stdout)).toMatchObject({ + code: "cleanup_incomplete", + details: { + resources: ["owned-profile"], + partial_observation: { operation: "capture_browser_scenario" }, + }, + }); + await expect(access(destination)).rejects.toMatchObject({ code: "ENOENT" }); +}); diff --git a/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts b/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts index 80dccb290b..b693a10168 100644 --- a/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts +++ b/tests/boundary/cli/scenarioCliInputDiagnostics.test.ts @@ -38,6 +38,38 @@ describe("scenario CLI schema diagnostics", () => { }); }); + cliTest( + "refuses summary delivery without a complete Evidence bundle before launching", + async ({ cli }) => { + const scenario = { + browser: { mode: "launch", executable_path: "/nonexistent/chromium" }, + start_url: { url: "https://app.example.test/", query: [] }, + actions: [ + { step_id: "wait", action: "wait_for_timeout", duration_ms: 1 }, + ], + }; + const result = await cli.run({ + arguments: [ + "capture-browser-scenario", + JSON.stringify(scenario), + "--detail", + "summary", + "--json", + ], + environment: { REA_LOG_LEVEL: "silent" }, + }); + + expect(result.exitCode).not.toBe(0); + expect(result.json).toMatchObject({ + code: "invalid_request", + details: { + operation: "capture_browser_scenario", + issues: [{ path: ["evidence_output"], reason: "invalid_value" }], + }, + }); + }, + ); + cliTest("reports Electron scenario schema paths", async ({ cli }) => { const result = await cli.run({ arguments: ["capture-electron-scenario", "{}", "--json"], diff --git a/tests/boundary/mcp/browserScenarioMcp.test.ts b/tests/boundary/mcp/browserScenarioMcp.test.ts index 6ed007354f..e8caca9da6 100644 --- a/tests/boundary/mcp/browserScenarioMcp.test.ts +++ b/tests/boundary/mcp/browserScenarioMcp.test.ts @@ -367,3 +367,111 @@ test("records a completed late-close cancellation as server Evidence", async () await Promise.all([client.close(), server.close(), session.close()]); } }); + +test("delivers a retained summary and pages its exact steps and events", async () => { + const provider = new FakeBrowserScenarioProvider(); + const session = createTestBinarySession(() => ({ + execute: () => Promise.resolve(observed(null)), + close: () => Promise.resolve(ok(null)), + })); + const server = createServer( + { kind: "session", session }, + { + browserScenarioCapture: provider, + availabilityPolicy: () => ({ + processCaptureEnabled: false, + investigationInputRoots: 0, + }), + }, + ); + const client = new Client({ name: "scenario-summary", version: "1" }); + const [clientTransport, serverTransport] = + InMemoryTransport.createLinkedPair(); + await server.connect(serverTransport); + await client.connect(clientTransport); + try { + const tools = (await client.listTools()).tools; + const advertised = tools.find( + ({ name }) => name === "capture_browser_scenario", + ); + if (advertised?.outputSchema === undefined) + throw new Error("Missing advertised scenario output schema"); + const ajv = new Ajv2020({ strict: false, validateFormats: false }); + const validateInput = ajv.compile(advertised.inputSchema); + const validateOutput = ajv.compile( + z.record(z.string(), z.unknown()).parse(advertised.outputSchema), + ); + const request = { ...minimalScenario(), detail: "summary" }; + expect(validateInput(request), JSON.stringify(validateInput.errors)).toBe( + true, + ); + + const response = await client.callTool({ + name: "capture_browser_scenario", + arguments: request, + }); + expect(response.isError, JSON.stringify(response)).not.toBe(true); + expect( + validateOutput(response.structuredContent), + JSON.stringify(validateOutput.errors), + ).toBe(true); + const summary = z + .object({ + normalized_result: z.object({ + kind: z.literal("summary"), + parent_evidence_id: z.string(), + summary: z.object({ + steps: z.array(z.object({ step_id: z.string() })), + }), + }), + }) + .parse(response.structuredContent).normalized_result; + expect(summary.summary.steps.map(({ step_id }) => step_id)).toEqual([ + "scenario_start", + "settle", + ]); + // Delivery detail is neither sent to the provider nor recorded as a scenario parameter. + expect(provider.scenarios[0]).not.toHaveProperty("detail"); + const parent = session.evidenceById(summary.parent_evidence_id); + expect(parent?.operation).toBe("capture_browser_scenario"); + expect(parent?.parameters).not.toHaveProperty("detail"); + + const page = await client.callTool({ + name: "inspect_analysis_view", + arguments: { + source: { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }, + view: { + kind: "browser-scenario", + collection: "steps", + step_indices: [1], + }, + }, + }); + expect(page.isError, JSON.stringify(page)).not.toBe(true); + expect(page.structuredContent).toMatchObject({ + normalized_result: { + kind: "browser-scenario", + parent_evidence_id: summary.parent_evidence_id, + items: { + collection: "steps", + rows: [{ step_index: 1, step_id: "settle", artifacts: {} }], + }, + coverage: { total: 1, exhausted: true }, + }, + }); + + const invalid = await client.callTool({ + name: "capture_browser_scenario", + arguments: { ...minimalScenario(), detail: "brief" }, + }); + expect(parseMcpToolError(invalid)).toMatchObject({ + error: { code: "invalid_request" }, + }); + expect(provider.scenarios).toHaveLength(1); + } finally { + await Promise.all([client.close(), server.close(), session.close()]); + } +});