From 3fca2e0ca6e54ba700c286d9a85a9600dee6a19d Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Mon, 19 Oct 2026 20:33:46 -0700 Subject: [PATCH] feat(environment): extract the initial state a capture's reads observed (#5319) * feat(environment): extract the initial state a capture's reads observed Add the environment-state analysis view. From the parents the environment manifest accepts, it extracts the data the capture's reads observed before the first related write: entity types with an identity field (proved by a path parameter or inferred from an identity name), entities with their recorded values and provenance, documents for written non-entity paths, and a render plan per read tool that is kept only when it reproduces the tool's latest initial recordings from the state; every other read tool is a recorded view with the reason. The Evidence is always inferred. Response-body decoding moves to a domain owner shared with replay, exact JSON parsing moves to the owner of the exact-number policy, the manifest and state views share parent admission and record grouping, and projectAnalysisView takes its accumulators as one object. Co-Authored-By: Claude Opus 5.5 * fix(environment): bound state at redacted writes; infer identity only from items A scenario write whose URL was redacted has no request line, so it was dropped and reads after it counted as initial state. Such a successful write now joins the timeline with an empty path, whose root relates to every type, so it ends every type's state; a limitation counts them. A list whose recordings were all empty passed the "every listed item carries it" check vacuously and published an identity field nothing showed; it now needs at least one item, else the view stays recorded as no-identity. A value that differed between two request lines of one item tool, such as a query variant, was called volatile, and the gate then substituted each recording's own value; only repeats of one request line now show volatility, so differing lines go through fields-differ. List membership is compared only while a list can still be the whole collection, and merged entities are indexed by type and collection once, so paged captures no longer cost pages times entities (8,000 pages of 40,000 entities: 5.7 s to 0.23 s). Docs and the tool contract say scenario state keeps the capture's redaction markers and how redacted writes bound it. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep first sightings, one list pointer and linear field unions Entities took the order of the first plan observed, and item plans run before list plans, so a complete list read before its detail reads came out reordered and lost "all" membership; each entity now keeps its earliest recorded sighting. A list plan reused the first recording's pointer for later recordings, so a later response that was itself an entity wrapping a list was read as that list; every recording must now locate the same list. The field union scanned its ordered list for every key, quadratic in an object's width; a set now answers membership. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep post-write values and partial lists out of the initial state A value that changed between repeats of one request line was named volatile whenever no write under the type's own root lay between, so a write under another root (for example POST /api/dashboards/db before a second GET /api/search) left its effect in the initial state. Any recorded write between the two observations now leaves every tool of the type recorded with reason fields-differ instead. A list was published with membership "all" even when a total or count envelope member exceeded the number it listed. Such a counter now makes the membership "recorded". Also drop a stale doc comment on ObservedEntity.order. Co-Authored-By: Claude Opus 5.5 * docs(environment): scope item-shaped and deleted written paths out of documents State documents are held only for list-shaped paths a POST, PUT or PATCH tool writes. Reads of item-shaped written paths and of paths that are only deleted stay recorded views; the guide now says so and the spec defers them to a later increment of #4933. Co-Authored-By: Claude Opus 5.5 * fix(environment): count writes in flight when a value changes between reads The volatility check compared write start positions with the two reads, so a write already in flight when the earlier read started was missed and its effect was named volatile. A write now counts when its interval overlaps the span from the earlier read's start to the later read's end. The check also ran once per changed field and scanned every write. It now runs once per pair of observations, answered by a binary search over the writes in timeline order and the latest end among each prefix. Co-Authored-By: Claude Opus 5.5 * fix(environment): catch cross-tool changes across writes; read only item totals A value that changed across a write was caught only between repeats of one tool's request line, so a list read before a write and an item read after it put the post-write value into the initial state. Changes are now classified per entity in timeline order after observation: across any recorded write, a change between repeats of a line or between two lines of the type rewrites the type (every tool recorded as fields-differ); with no write between, a repeat's change is volatile. The classification lives in environmentStateMerge.ts. Any numeric member whose name contained "total" or "count" counted as an item total, so total_pages, account_id or page_count made complete lists partial. Item totals are now members whose name tokens include total or count and none of page, pages or per; a string of digits counts as a number; and a set has_more, next or next-page token member also marks the list partial. Co-Authored-By: Claude Opus 5.5 * fix(environment): keep post-write observations out of initial state * fix(environment): retain field provenance and bound state inference work --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> Co-authored-by: gensokyo-dev --- .../references/evidence-workflows.md | 6 + docs/agent-environments.md | 12 + docs/mcp-contracts.md | 7 + .../agent-environments/state-view/plan.md | 28 + .../agent-environments/state-view/spec.md | 74 ++ .../agent-environments/state-view/tasks.md | 12 + .../state-view/verification.md | 118 +++ docs/web-network-captures.md | 97 +++ scripts/ci/scopes.mjs | 2 +- .../analysisView/AnalysisViewService.test.ts | 159 ++-- .../analysisView/AnalysisViewService.ts | 23 +- .../environment/environmentReplay.ts | 23 +- .../environment/environmentScenario.test.ts | 4 +- .../environment/environmentSource.ts | 65 +- src/contracts/analysisDetail.test.ts | 2 + src/contracts/analysisViewToolContracts.ts | 2 +- .../EnvironmentManifestAccumulator.ts | 11 +- .../EnvironmentStateAccumulator.ts | 283 +++++++ .../EnvironmentStateExtraction.ts | 587 ++++++++++++++ .../WebNetworkProjectionAccumulator.ts | 8 +- src/domain/analysisView/analysisView.ts | 80 +- src/domain/analysisView/binaryLayoutView.ts | 2 +- .../analysisView/environmentManifestView.ts | 123 +-- .../analysisView/environmentStateGate.ts | 320 ++++++++ .../analysisView/environmentStateMerge.ts | 135 ++++ .../analysisView/environmentStateRules.ts | 437 +++++++++++ .../analysisView/environmentStateView.test.ts | 715 ++++++++++++++++++ .../analysisView/environmentStateView.ts | 188 +++++ .../analysisView/javascriptApplicationView.ts | 2 +- .../analysisView/recordedResponseBody.ts | 63 ++ .../scenarioCaptureRecords.test.ts | 2 + .../analysisView/scenarioCaptureRecords.ts | 15 +- .../analysisView/webNetworkCaptureView.ts | 2 +- src/domain/exactJsonNumber.ts | 16 + tests/acceptance/tools/qlogCapture.test.ts | 6 +- tests/boundary/cli/analysisViewCli.test.ts | 47 ++ tests/boundary/mcp/analysisViewMcp.test.ts | 74 ++ tests/fixtures/analysisView.ts | 16 +- tests/fixtures/historicalHar.ts | 11 + 39 files changed, 3556 insertions(+), 221 deletions(-) create mode 100644 docs/specs/agent-environments/state-view/plan.md create mode 100644 docs/specs/agent-environments/state-view/spec.md create mode 100644 docs/specs/agent-environments/state-view/tasks.md create mode 100644 docs/specs/agent-environments/state-view/verification.md create mode 100644 src/domain/analysisView/EnvironmentStateAccumulator.ts create mode 100644 src/domain/analysisView/EnvironmentStateExtraction.ts create mode 100644 src/domain/analysisView/environmentStateGate.ts create mode 100644 src/domain/analysisView/environmentStateMerge.ts create mode 100644 src/domain/analysisView/environmentStateRules.ts create mode 100644 src/domain/analysisView/environmentStateView.test.ts create mode 100644 src/domain/analysisView/environmentStateView.ts create mode 100644 src/domain/analysisView/recordedResponseBody.ts diff --git a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md index 447832b995..5d6f374dd1 100644 --- a/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md +++ b/.agents/skills/reverse-engineer-anything/references/evidence-workflows.md @@ -77,3 +77,9 @@ body shapes and record pointers for an agent environment. Opt into `template_rules` only when recorded identifiers need them; every template, parameter and merged sibling is an inference, no header, cookie or body value is copied, and the three stated unknowns stay unresolved. + +Use `view: {kind: "environment-state"}` with the manifest's `template_rules` to +extract the data those reads observed instead: entity types and identity +fields, entities with recorded values, documents, and per-tool render plans +checked against the recordings. The state comes only from reads before the +first related write; types, identity and list membership are inferences. diff --git a/docs/agent-environments.md b/docs/agent-environments.md index 67b9909570..5ebaf7f4a3 100644 --- a/docs/agent-environments.md +++ b/docs/agent-environments.md @@ -213,6 +213,18 @@ That same warning prefixes every tool description in the standalone card, so card discovery carries it before an MCP client initializes. It labels the captured text without changing the tool names or schema fields clients use. +## The state a capture observed + +Replay answers each call from one recording, so the answers reflect whichever +moment of the capture recorded them. The +[environment state](web-network-captures.md#environment-state) view extracts +instead the data the capture's reads observed before the first related write: +entity types with an identity field, entities with their recorded values, +documents for written paths, and a render plan per read tool, kept only where +it reproduces that tool's recordings from the state. It is the starting state +an environment can hold, load and report; serving from it is a later increment +and does not change replay. + ## Serve a manifest ```sh diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index 9f25a28891..9e0159d9e1 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -225,6 +225,13 @@ names are inferences; the Evidence is inferred whenever any tool or excluded gro is templated, and derived otherwise. See [environment manifests](web-network-captures.md#environment-manifest). +`inspect_analysis_view` also accepts `environment-state` for the same parents, +extracting the initial state the recorded reads observed: entity types with an +identity field, entities, documents for written non-entity paths, and a render +plan per read tool that reproduces its recordings from the state or names why it +cannot. Types and identity are inferences, so the Evidence is always inferred. +See [environment state](web-network-captures.md#environment-state). + `inspect_analysis_view` projects a caller-selected view of already completed `inspect_binary_layout`, `analyze_javascript_application`, or `analyze_function` Evidence. Native views select procedure, pseudocode, assembly, basic blocks, diff --git a/docs/specs/agent-environments/state-view/plan.md b/docs/specs/agent-environments/state-view/plan.md new file mode 100644 index 0000000000..518e6b35ec --- /dev/null +++ b/docs/specs/agent-environments/state-view/plan.md @@ -0,0 +1,28 @@ +# Implementation plan + +1. Move response-body decoding from the environment source into + `recordedResponseBody.ts` with the caller's byte limit, and exact JSON + parsing into `exactJsonNumber.ts`, so replay and state extraction share one + owner each. +2. Add `environmentStateRules.ts` as pure helpers: write classification, API + bases and roots, entity location, identity names, JSON pointers of depth one, + value bytes and the entity and list renderers. +3. Add `EnvironmentStateExtraction.ts` for planning item and list tools, + observing entities and assembling the result, `environmentStateMerge.ts` for + merging observations, and `environmentStateGate.ts` for reproducing each + plan's recordings; every body is parsed through a budget callback and + released before the next. +4. Add `EnvironmentStateAccumulator.ts`, extending the manifest accumulator so + grouping is shared, retaining record lines and references to the admitted + records, and charging bodies and kept values. +5. Add `environmentStateView.ts` with the request schema (sharing the + manifest's `template_rules` schema), the result fields and the projector; + share capture admission and record grouping with the manifest view; register + the kind and pass accumulators to `projectAnalysisView` as one object; + always seal the result inferred. +6. Cover domain, service, MCP, CLI, catalog and acceptance behavior; document + the view in the capture guide, the agent environments guide, the contracts + reference and the authored skill. + +No new tool, provider, side effect or network activity is introduced. The state +is a projection of already retained Evidence; serving it is deferred. diff --git a/docs/specs/agent-environments/state-view/spec.md b/docs/specs/agent-environments/state-view/spec.md new file mode 100644 index 0000000000..18c0e46f74 --- /dev/null +++ b/docs/specs/agent-environments/state-view/spec.md @@ -0,0 +1,74 @@ +# Environment state view from captures + +Issue: https://github.com/morluto/rea/issues/4933 + +## Outcome + +`inspect_analysis_view` accepts `view: {"kind": "environment-state"}` for the +parents the environment manifest accepts and extracts the initial state the +capture's reads observed: entity types with an identity field, entities with +their recorded values, documents for written paths whose reads are not +entities, and a render plan per read tool that reproduces the tool's latest +initial recordings from the state, or a recorded view with the reason it +cannot. This increment adds the view kind only; serving from the state, loading +another state through the AgentEnv data plane and modelling write effects are +later increments. + +## Requirements + +- Accept `inspect_web_network_capture` HAR parents and + `capture_browser_scenario` parents that selected response bodies, with the + manifest's admission: the same format, subject and sensitive-value refusals. +- Group records exactly as the manifest does for the same `template_rules`, so + tool names match the manifest the state accompanies. +- Order primary-origin records by recorded start time, or by record order when + any start time is missing, and say which. Bound each type's state by the first + write under its root (API base, version segments, one segment); a query + `POST` is not a write, a read overlapping the write is left out, and a write + whose URL was redacted bounds every type, with a limitation counting it. +- Prove an item type's identity field from the requested path value its + responses repeat; otherwise infer it from a fixed list of identity names that + every listed item carries distinctly, which needs at least one listed item. + Never treat an envelope that is itself an entity as a list. +- Merge observations into one entity per collection and identity: item views' + latest values win over list views', differing views keep their own values, + and a value that changed between repeats of one request line with no recorded + write is volatile and keeps the latest; a change across a write under another + root, including between two tools of the type, leaves every tool of the type + recorded. Compare list membership, including empty lists, and field presence before selecting each request line's latest recording; a change across a write also leaves the type recorded. Different query strings remain distinct; omitted fields do not erase their earlier observations on other request lines. +- A list has `all` membership only when no item-total envelope member (`total`, + `count` and their compounds, not page counts) differs from the number of + entities it lists and no `has_more`, `next` or next-page token is set. +- Keep an item or list plan only when it reproduces each request line's latest + initial recording from the state, comparing JSON values without regard to + member order; otherwise record the tool with a reason. Parse bodies only when + JSON-like, UTF-8, at most 256 KiB, exactly representable and free of + `__proto__` members. +- Mark the Evidence inferred; state unknowns for authorization, response + semantics and write effects. +- Charge the manifest's admission and grouping, every record line, every body + as temporary bytes before decoding and parsing, and every kept value, within + the existing 96 MiB budget. +- Read response bodies through one domain owner shared with replay, and parse + exact JSON through the owner of the exact-number policy. + +## Acceptance and limits + +The Grafana-shaped HAR fixture with distinct start times drives domain, +service, MCP, CLI and acceptance coverage: folders merge their detail and list +views under a proved `uid`, the dashboard entity is read from its `/dashboard` +wrapper, and the annotation list after the annotation write has no initial +recording. Synthetic captures cover write bounds, volatile fields, complete +lists with counters, each recorded reason and documents; the scenario fixture +covers record order and the response-body requirement. The view performs no +network activity and serves nothing. Values copied outside entity objects, +derived values and cross-type effects are not tied to the state. + +## Deferred issue criteria + +Serving tool calls from the state, `data/reset`, `data/add` and `data/get` on +the environment data plane, exporting the state with a bundle, and write +effects are later increments of #4933. So are documents for item-shaped +written paths (one per identity rather than per template) and for paths that +are only deleted; until then their reads stay recorded. This PR must reference rather than +close the issue. diff --git a/docs/specs/agent-environments/state-view/tasks.md b/docs/specs/agent-environments/state-view/tasks.md new file mode 100644 index 0000000000..10605c5b07 --- /dev/null +++ b/docs/specs/agent-environments/state-view/tasks.md @@ -0,0 +1,12 @@ +# Tasks + +- [x] Record the view contract, requirements and deferred increments. +- [x] Move body decoding and exact JSON parsing to their domain owners without + changing replay. +- [x] Implement the extraction rules, merge, gate and accumulator. +- [x] Register the `environment-state` kind, result schema, service wiring and + confidence rule. +- [x] Add the timed Grafana-shaped fixture and synthetic captures. +- [x] Add domain, service, MCP, CLI, catalog and acceptance coverage. +- [x] Document the view in the guides, contracts reference and skill. +- [x] Run static, formatting, duplication, documentation and test gates. diff --git a/docs/specs/agent-environments/state-view/verification.md b/docs/specs/agent-environments/state-view/verification.md new file mode 100644 index 0000000000..26101f651a --- /dev/null +++ b/docs/specs/agent-environments/state-view/verification.md @@ -0,0 +1,118 @@ +# Verification + +Verified base: bf1eeee9 (fetched main). + +Executed on the implementation worktree after `npm ci` and `npm run +build:cached`; every check below ran as written in this pass, after the last +source change. + +- `npm run check:fast`: typecheck and lint passed. +- `npm run format:check`: passed. +- `npx knip`: passed (exit 0). +- `npm run jscpd`: 3.40% duplicated lines over 3331 files, under the 3.5% + gate; no clone involves a file this change adds. +- `npm run scan:todos`: passed. +- `npm run docs:check`: the built documentation checks and the README + consistency conformance test (5 tests) passed. +- `npx vitest run --project domain --project services`: 338 files, 5830 tests + passed, 5 pre-existing skips. +- `npm run test:focused --` the analysis-view MCP and CLI boundaries, the + environment serve and export CLI boundaries, the environment serve MCP + boundary, the compact input-schema MCP budget, the analysis-detail and + advertised-input-example contracts and the qlog acceptance test: 9 files, 68 + tests passed, 1 pre-existing skip. + +Real captures (outside the test suite): the built view ran through the +application service on 44 recorded captures (39 HAR, 5 browser scenario) with +all three template rules, in 7 to 46 ms each. 42 produced a state: 568 types, +2,213 entities, 2 documents, and 710 of 1,258 read tools state-backed (211 +item, 497 list, 2 document views). The other read tools are recorded views: +306 not entities (singletons, configuration and computed views), 129 without +an identity field, 32 with unusable bodies (most are Nomad bodies with +integers beyond exact native conversion), 31 lists and 7 items that do not +reproduce, 26 without an initial recording, 12 query `POST`s and 5 documents +read under several queries. The two browser-scenario captures that did not +produce a state are refused inline with `resource_constraint`, as the +environment manifest view refuses them on unchanged main. + +Not covered here: serving tool calls from the state, the AgentEnv data plane +and write effects, which are later increments. + +## Linux follow-up (2026-10-20) + +Node 24.18.0 / npm 11.16.0, with the author's `47f8a27a9` main integration +preserved and main `d72021b0d` integrated. Eight regression cases failed across +two pre-fix runs: membership additions/removals across a write, field presence, +sparse field observations, and compact compound totals. The fixes compare +membership before selecting latest recordings, retain each field's observation +across sparse reads, and recognize `totalcount`/`itemcount`. API-base grouping +appends to its owned arrays; identity intersection uses a set and skips the +first response's self-intersection. + +Executed checks: + +- Relevant domain, application and contract suites: 39 files, 493 passed, + 3 existing skips. Includes empty/stable lists, query variants, field additions + and removals, sparse reads, identity proof and deferred document paths. +- Broader compiled CLI/MCP/environment, compact schema and qlog acceptance: + 7 files, 65 passed, 1 existing skip. After the final state fixes, both selected + compiled state CLI/MCP cases passed again (36 other cases deselected). +- Final serial static/build/artifact gate: 11 tasks passed. Typecheck, lint, + format, knip, package metadata and runtime/catalog generation all ran. +- Final documentation/catalog/format validation: 9 tasks passed; architecture + guards passed across 1,602 source files; README consistency: 5 tests passed. + +Real Gitea 28.2.0 used an isolated SQLite database. An owned Node fetch HAR +recorder captured actual API exchanges, omitting transport credentials: + +- Three exchanges: empty repository search, creation through `/user/repos`, + then the same search. On exactly the same authenticated Evidence, pre-fix + `29e745923` published one created repository as initial state; the final + source publishes none and keeps the search recorded with `fields-differ`. +- Eleven exchanges: initial issue list/details and topics, PATCH, PUT and + DELETE, then subsequent reads. The state keeps the two original issues + before ordinal 4 and the original topics document at ordinal 3. File CLI and + default stdio MCP agree; six HTTP SDK replays reproduce original responses. +- Actual Chromium `capture_browser_scenario`: two transactions, eight events, + zero dropped. The state uses record order, extracts one issue, and keeps list + and item plans; file CLI and default stdio MCP agree. The earlier 12.6 MB + real browser capture also projects, keeping oversized bodies recorded with + `body-unusable` rather than parsing them into state. Owned servers and browser + processes were stopped. + +### Resource and performance measurements + +One CPU and one worker. Process-family RSS sums were sampled every 50 ms and +may count shared pages twice. Final source suites: 39.669 s wall / 39.486 CPU s / +898.42 MiB peak; broader boundaries: 270.171 s / 267.136 CPU s / 2,130.78 MiB; +final static/build: 199.683 s / 199.523 CPU s / 4,689.07 MiB; final live HAR +workflow: 22.836 s / 19.880 CPU s / 1,072.43 MiB; actual browser-state workflow: +12.734 s / 10.941 CPU s / 1,549.80 MiB. The qlog acceptance lane intentionally +exhausts an isolated decoder heap and verifies that MCP remains usable; no +workspace OOM occurred. + +Before/after used immutable pre-fix `29e745923` and the final source, the same +completed Evidence files, Node and CPU affinity. One fresh process per revision, +with a first call and two warm repeats; filesystem caches were not cleared. +Complete normalized output digests matched for both synthetic projection tasks. + +| Work | Before: first / warm (ms) | After: first / warm (ms) | +| ----------------------------------------------- | ------------------------- | ------------------------ | +| Group 8,000 templates (helper only) | 320.08 / 81.01–81.46 | 11.89 / 3.88–4.33 | +| File-to-state projection, 2,000 collections | 1,321.57 / 732.00–939.57 | 1,268.76 / 781.34–811.05 | +| File-to-state projection, one 15,001-field item | 657.05 / 494.82–501.19 | 377.78 / 347.62–380.87 | + +Combined benchmark processes, including imports, three helper calls, six +projections and the live-capture comparison: before 6.536 s wall / 6.496 CPU s / +370.82 MiB peak; after 5.077 s / 5.056 CPU s / 386.53 MiB. Grouping and wide +identity inference improved; the 2,000-collection projection has overlapping +warm timings, and peak memory did not decrease. These few runs do not establish +a general speedup. Model token usage was not measured; no model-task scoring +was run. Initial external probe attempts had incorrect CLI positional arguments +and assumed coverage counted records rather than grouped tools; those driver +errors were corrected before the successful complete runs. + +No new macOS/Windows runtime, state serving, AgentEnv data-plane or write-effects +claim is made. A type first observed only after an unrelated-root write may +still lack the earlier observation needed to detect its effect. Item-shaped +and delete-only document paths remain recorded, as specified. diff --git a/docs/web-network-captures.md b/docs/web-network-captures.md index 124b63b9ca..4a9667bd6a 100644 --- a/docs/web-network-captures.md +++ b/docs/web-network-captures.md @@ -433,6 +433,103 @@ once per value in the capture clone, not per character. A `capture_browser_scenario` parent is also charged, before validation, for checking its largest body and for the decoded text of its request bodies. +## Environment state + +Pass the same parent to `inspect_analysis_view` with +`view: {"kind":"environment-state"}` to extract the initial state the +capture's reads observed, and a render plan per read tool that reproduces those +reads from it. Where the environment manifest names an application's tools, the +state holds its data: entities by type and identity, plus documents for written +paths whose reads are not entities. Use the `template_rules` of the manifest the +state accompanies so tool names match; the view groups records the same way. + +```json +{ + "name": "inspect_analysis_view", + "arguments": { + "source": { "kind": "retained-evidence", "evidence_id": "ev_..." }, + "view": { + "kind": "environment-state", + "template_rules": [ + "after-uid-literal", + "opaque-token", + "sibling-variance" + ] + } + } +} +``` + +Records on the primary origin are ordered by recorded start time, or by record +order when any start time is missing (`timeline`), as for a +`capture_browser_scenario` parent. A `POST`, `PUT`, `PATCH` or `DELETE` with a +2xx or 3xx status, or no status, may change state, unless it is a `POST` whose +path has a segment such as `search`, `query`, `list`, `find`, `fuzzy`, +`count` or `stats`, which queries state instead. A type's state comes from its +successful reads that finished before the first such write under the same root: +the path's API base (the literal segments every tool sharing its first segment +has), any `v1`-style version segments, then one more segment. A read that +overlaps that write in time is left out. A write whose URL a scenario redacted +has an unknown target, so it ends the state of every type, and a limitation +counts such writes. Values from a `capture_browser_scenario` parent keep the +capture's redaction markers in place of declared secrets; the state is not +refused for them, as the manifest is not. + +Each `GET` tool whose template ends in a parameter is an item tool, and its +type's `collection_template` is the template without that parameter; any other +`GET` tool is a list tool of its own template. An item tool proves its type's +`id_field` when every initial response carries the requested path value in the +same field, at the root or in one wrapper member (`id_basis: "path-parameter"`). +Otherwise the field is the first of `id`, `uid`, `uuid`, `_id`, `ID`, `UID`, +`Id`, `guid`, `key`, `slug` and `name` that every listed item carries as a +distinct string or number (`id_basis: "field-name"`). A list is the response +array, or the one array of objects in an envelope that is not itself an entity. + +`entities` merge every initial observation of one identity within one +collection path: an item view's latest values win over a list view's, a view +whose latest values differ keeps them in `view_fields`, and a field whose value +changed between one tool's observations with no recorded write between is named +in the type's `volatile_fields`, keeping the latest value. A value that changed +across a write under another root may be that write's effect, so every tool of +its type is a recorded view with reason `fields-differ`; so is a value one tool +read before a write and another tool read after it. Repeated reads of one full request line also keep the type recorded when list membership changes across a write, including transitions to or from an empty list, or when entity fields are added or removed. Different query strings remain distinct request lines. Sparse responses do not erase the last observation of a field supplied by another request line. Entities are ordered +by type, collection and first sighting; that order is a list's state order. +Every entity cites its `record_ordinals`. + +`views` lists one plan per read tool in tool order: + +| Shape | Plan | +| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `item` | The entity object at `pointer` (the root or one member), the ordered `fields` the view shows and the `required_fields` every recording had, and `envelope_ordinal`: the recording whose response, with the entity replaced, every item response shares | +| `list` | The array at `pointer`, its `fields`, `membership` (`all`: every entity of the collection in state order, when each recording listed exactly that, had no query, no item-total member (`total`, `count`, `totalCount`, `item_count`, `totalcount`, `itemcount`, but not `total_pages` or `per_page`; a number or a string of digits) differing from the number listed, and no `has_more`, `next` or next-page token member set; `recorded`: the entities each recording listed) and `counters` that equal the number listed | +| `document` | The path's latest initial value, held in `documents` | +| `recorded` | A `reason` the tool replays its recordings instead | + +An item or list plan is kept only when it reproduces the latest initial +recording of each request line from the state, comparing JSON values without +regard to member order; a volatile field compares its recorded value. Each +plan lists the `checked_record_ordinals`. Otherwise the tool is a recorded view +with reason `envelope-varies`, `fields-differ` or `list-differs`; the other +reasons are `query-body` (a query `POST`), `no-init-recording`, +`body-unusable`, `not-entity`, `no-identity` and `query-varies` (a document +read under several queries). Documents are held only for list-shaped paths +that a `POST`, `PUT` or `PATCH` tool writes: a non-entity read of an item path +such as `/api/files/{id}`, or of a path that is only deleted, stays recorded +with `no-identity` or `not-entity`. A body is usable when its content type is +JSON-like, it is UTF-8 text of at most 256 KiB, every number survives native +conversion exactly, and no member is named `__proto__`. + +Values in the state are recorded data, so the view refuses the same parents as +the manifest, including those inspected with `sensitive_values`, and a +`capture_browser_scenario` parent must have selected response bodies. Types and +identity are inferences, so the Evidence is always inferred. `unknowns` states +that the state establishes no authorization model, no way the application +computes responses or derived values, and no effects of writes. The state +accumulator charges the manifest's admission and grouping, every record line, +each body as temporary bytes before it is decoded and parsed, and every value +it keeps, within the same 96 MiB budget. The view performs no network activity +and serves nothing. + ## Resource and lifecycle boundaries Complete evidence has a 32 MiB input, 96 MiB decoder reply and 64-level nesting diff --git a/scripts/ci/scopes.mjs b/scripts/ci/scopes.mjs index 8cd4a99533..fcb4a0dab0 100644 --- a/scripts/ci/scopes.mjs +++ b/scripts/ci/scopes.mjs @@ -448,7 +448,7 @@ export const SCOPES = { "src/server/http/**", "src/{cli,cliCommandNames,cli/environmentCommands}.ts", "src/config/{environmentName,mcpHttpEndpoint}.ts", - "src/domain/analysisView/{environmentManifest*,EnvironmentManifest*,scenarioCaptureRecords,WebNetworkProjectionAccumulator,analysisViewDataAdmission}.ts", + "src/domain/analysisView/{environmentManifest*,EnvironmentManifest*,environmentState*,EnvironmentState*,recordedResponseBody,scenarioCaptureRecords,WebNetworkProjectionAccumulator,analysisViewDataAdmission}.ts", "src/process/{OwnedCommand,ProviderProcess,PrivateRuntimeRoot}.ts", "src/process/ProcessOwnership*.ts", "tests/fixtures/historicalHar.ts", diff --git a/src/application/analysisView/AnalysisViewService.test.ts b/src/application/analysisView/AnalysisViewService.test.ts index 728aada4cb..27500e6855 100644 --- a/src/application/analysisView/AnalysisViewService.test.ts +++ b/src/application/analysisView/AnalysisViewService.test.ts @@ -6,6 +6,7 @@ import { analysisViewCaptureEvidence, analysisViewObjectDenseCapture, environmentManifestCaptureFixture, + environmentStateCaptureFixture, } from "../../../tests/fixtures/analysisView.js"; import { describe, expect, it, vi } from "vitest"; @@ -464,6 +465,7 @@ it("refuses an object-dense borrowed capture before making a full schema clone", for (const view of [ { kind: "api-surface", limit: 1 }, { kind: "environment-manifest" }, + { kind: "environment-state" }, ] as const) { const result = inspectAnalysisView( { @@ -536,42 +538,75 @@ it("seals an inferred environment manifest and a derived manifest without templa }); }); -it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)( - "rejects inline accessors before parsing a %s view", - (kind) => { - let accessed = false; - const evidence = { - get normalized_result() { - accessed = true; - throw new Error("Evidence must be admitted before parsing"); - }, - }; - const input = { - source: { kind: "inline", evidence }, - view: { kind }, - }; - const parsed = inspectAnalysisViewInputSchema.parse(input); - if (parsed.source.kind !== "inline") - throw new Error("Expected inline input"); - expect(parsed.source.evidence).toBe(evidence); - expect(inspectAnalysisView(input)).toMatchObject({ - ok: false, - error: { - _tag: "AnalysisInputError", - issues: [{ path: ["source", "evidence", "normalized_result"] }], - }, - }); - expect(accessed).toBe(false); - expect( - inspectAnalysisView({ - source: { kind: "inline", evidence: analysisViewCaptureEvidence() }, - view: { kind }, - }).ok, - ).toBe(true); - }, -); +it("seals an inferred environment state whose values are the recorded ones", () => { + const parent = analysisViewCaptureEvidence(environmentStateCaptureFixture()); + const sealed = inspectAnalysisView({ + source: { kind: "inline", evidence: parent }, + view: { kind: "environment-state", template_rules: ["opaque-token"] }, + }); + if (!sealed.ok) throw sealed.error; + expect(sealed.value.confidence).toBe("inferred"); + expect(sealed.value.authority).toBe(parent.authority); + expect(sealed.value.evidence_links).toEqual([parent.evidence_id]); + expect(sealed.value.parameters).toMatchObject({ + view: { kind: "environment-state", template_rules: ["opaque-token"] }, + }); + expect(sealed.value.normalized_result).toMatchObject({ + kind: "environment-state", + name: "grafana-example", + parent_evidence_id: parent.evidence_id, + timeline: "recorded-start-time", + }); + expect(JSON.stringify(sealed.value.normalized_result)).toContain( + "probe folder", + ); + expect(parseEvidence(sealed.value)).toEqual(sealed.value); +}); -it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)( +it.each([ + "summary", + "capture", + "api-surface", + "environment-manifest", + "environment-state", +] as const)("rejects inline accessors before parsing a %s view", (kind) => { + let accessed = false; + const evidence = { + get normalized_result() { + accessed = true; + throw new Error("Evidence must be admitted before parsing"); + }, + }; + const input = { + source: { kind: "inline", evidence }, + view: { kind }, + }; + const parsed = inspectAnalysisViewInputSchema.parse(input); + if (parsed.source.kind !== "inline") throw new Error("Expected inline input"); + expect(parsed.source.evidence).toBe(evidence); + expect(inspectAnalysisView(input)).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisInputError", + issues: [{ path: ["source", "evidence", "normalized_result"] }], + }, + }); + expect(accessed).toBe(false); + expect( + inspectAnalysisView({ + source: { kind: "inline", evidence: analysisViewCaptureEvidence() }, + view: { kind }, + }).ok, + ).toBe(true); +}); + +it.each([ + "summary", + "capture", + "api-surface", + "environment-manifest", + "environment-state", +] as const)( "rejects nested, hidden and inherited inline accessors for %s", (kind) => { const parent = analysisViewCaptureEvidence(); @@ -653,6 +688,7 @@ it.each([ "capture", "api-surface", "environment-manifest", + "environment-state", ] as const) { const result = inspectAnalysisView({ source: { kind: "inline", evidence: { ...parent, raw_result: array } }, @@ -689,6 +725,7 @@ it("rejects callable array overrides and subclass species before parsing", () => "capture", "api-surface", "environment-manifest", + "environment-state", ] as const) { const result = inspectAnalysisView({ source: { kind: "inline", evidence: { ...parent, raw_result: array } }, @@ -702,26 +739,29 @@ it("rejects callable array overrides and subclass species before parsing", () => expect(invoked).toBe(false); }); -it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)( - "preserves ordinary inline Evidence for %s", - (kind) => { - const parent = analysisViewCaptureEvidence(); - const before = JSON.stringify(parent); - const result = inspectAnalysisView({ - source: { kind: "inline", evidence: parent }, - view: { kind }, - }); - if (!result.ok) throw result.error; - expect(parseEvidence(result.value)).toEqual(result.value); - expect(result.value.evidence_links).toEqual([parent.evidence_id]); - expect(result.value.normalized_result).toMatchObject({ - kind, - parent_evidence_id: parent.evidence_id, - }); - expect(JSON.stringify(parent)).toBe(before); - expect(Object.isFrozen(parent)).toBe(false); - }, -); +it.each([ + "summary", + "capture", + "api-surface", + "environment-manifest", + "environment-state", +] as const)("preserves ordinary inline Evidence for %s", (kind) => { + const parent = analysisViewCaptureEvidence(); + const before = JSON.stringify(parent); + const result = inspectAnalysisView({ + source: { kind: "inline", evidence: parent }, + view: { kind }, + }); + if (!result.ok) throw result.error; + expect(parseEvidence(result.value)).toEqual(result.value); + expect(result.value.evidence_links).toEqual([parent.evidence_id]); + expect(result.value.normalized_result).toMatchObject({ + kind, + parent_evidence_id: parent.evidence_id, + }); + expect(JSON.stringify(parent)).toBe(before); + expect(Object.isFrozen(parent)).toBe(false); +}); it("rejects executable constructor values before schema plain-object checks", () => { const parent = analysisViewCaptureEvidence(); @@ -738,6 +778,7 @@ it("rejects executable constructor values before schema plain-object checks", () "capture", "api-surface", "environment-manifest", + "environment-state", ] as const) { expect( inspectAnalysisView({ @@ -771,6 +812,7 @@ it("rejects inline proxy values before reflection or parsing can run traps", () "capture", "api-surface", "environment-manifest", + "environment-state", ] as const) { const result = inspectAnalysisView({ source: { kind: "inline", evidence: { ...parent, raw_result: value } }, @@ -815,6 +857,7 @@ it("keeps ignored array metadata inert through parsing and projection", () => { "capture", "api-surface", "environment-manifest", + "environment-state", ] as const) { const result = inspectAnalysisView({ source: { kind: "inline", evidence: parent }, @@ -862,7 +905,11 @@ it("admits raw_result and parameters before authenticating a portable capture", const dense = analysisViewObjectDenseCapture(); const parent = analysisViewCaptureEvidence(); for (const field of ["raw_result", "parameters"] as const) - for (const kind of ["api-surface", "environment-manifest"] as const) { + for (const kind of [ + "api-surface", + "environment-manifest", + "environment-state", + ] as const) { const result = inspectAnalysisView({ source: { kind: "inline", evidence: { ...parent, [field]: dense } }, view: { kind }, diff --git a/src/application/analysisView/AnalysisViewService.ts b/src/application/analysisView/AnalysisViewService.ts index fd2410f781..cdabb800f8 100644 --- a/src/application/analysisView/AnalysisViewService.ts +++ b/src/application/analysisView/AnalysisViewService.ts @@ -32,6 +32,7 @@ import { import { err, ok, type Result } from "../../domain/result.js"; import { WebNetworkApiSurfaceAccumulator } from "../../domain/analysisView/WebNetworkApiSurfaceAccumulator.js"; import { EnvironmentManifestAccumulator } from "../../domain/analysisView/EnvironmentManifestAccumulator.js"; +import { EnvironmentStateAccumulator } from "../../domain/analysisView/EnvironmentStateAccumulator.js"; import { admitAnalysisViewData } from "../../domain/analysisView/analysisViewDataAdmission.js"; import { readEvidenceFileSource, @@ -128,7 +129,12 @@ export const inspectAnalysisViewValidated = ( input.view.template_rules, ) : undefined; - const accumulator = apiSurfaceAccumulator ?? manifestAccumulator; + const stateAccumulator = + input.view.kind === "environment-state" + ? new EnvironmentStateAccumulator(undefined, input.view.template_rules) + : undefined; + const accumulator = + apiSurfaceAccumulator ?? manifestAccumulator ?? stateAccumulator; if (input.source.kind === "inline") { const admitted = accumulator === undefined @@ -160,8 +166,11 @@ export const inspectAnalysisViewValidated = ( }, }, input.view, - apiSurfaceAccumulator, - manifestAccumulator, + { + apiSurface: apiSurfaceAccumulator, + manifest: manifestAccumulator, + state: stateAccumulator, + }, ); if (!projected.ok) return projected; if ( @@ -314,10 +323,12 @@ const createAnalysisViewEvidence = ( result: jsonValueSchema.parse(result), rawResult: null, confidence: - // Grouping is whole-capture, so any templated group or parameter makes the result inferred. - (result.kind === "api-surface" || + // Grouping is whole-capture, so any templated group or parameter makes the result inferred; + // entity types and identity in an extracted state always are. + result.kind === "environment-state" || + ((result.kind === "api-surface" || result.kind === "environment-manifest") && - result.templated_groups > 0 + result.templated_groups > 0) ? "inferred" : "derived", authority: parent.authority, diff --git a/src/application/environment/environmentReplay.ts b/src/application/environment/environmentReplay.ts index c3f709562c..1be9c8fe11 100644 --- a/src/application/environment/environmentReplay.ts +++ b/src/application/environment/environmentReplay.ts @@ -2,13 +2,12 @@ import { isJsonLike } from "../../domain/analysisView/EnvironmentManifestAccumul import { mediaTypeEssence } from "../../domain/analysisView/webNetworkRouteTemplates.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; import { AnalysisInputError } from "../../domain/analysisErrorCore.js"; -import { isExactJsonNumber } from "../../domain/exactJsonNumber.js"; -import { jsonValueSchema, type JsonValue } from "../../domain/jsonValue.js"; +import { parseExactJson } from "../../domain/exactJsonNumber.js"; +import type { JsonValue } from "../../domain/jsonValue.js"; import { iterateUrlQueryParameters, recordedPathSegments, } from "../../domain/recordedTransportUrl.js"; -import { safeParseJson } from "../../domain/safeJson.js"; import { err, ok, type Result } from "../../domain/result.js"; import { environmentToolArguments } from "./environmentTools.js"; import { @@ -190,22 +189,8 @@ const replayBody = ( exchange.contentType !== null && isJsonLike(mediaTypeEssence(exchange.contentType)) ) { - const parsed = safeParseJson(text, (_key, value, context) => { - // The native parser supplies each primitive's original spelling. Share - // HAR's exact-number policy rather than accepting an already-rounded ID. - if ( - typeof value === "number" && - !isExactJsonNumber(value, context?.source) - ) - throw new RangeError( - "JSON numeric conversion would lose recorded data", - ); - return value; - }); - const value = parsed.ok - ? jsonValueSchema.safeParse(parsed.value) - : undefined; - if (value?.success === true) return { body: value.data, truncated: false }; + const value = parseExactJson(text); + if (value !== undefined) return { body: value, truncated: false }; } return { body: text, truncated: false }; }; diff --git a/src/application/environment/environmentScenario.test.ts b/src/application/environment/environmentScenario.test.ts index 592db02976..594240ef82 100644 --- a/src/application/environment/environmentScenario.test.ts +++ b/src/application/environment/environmentScenario.test.ts @@ -69,7 +69,7 @@ describe("environment manifests from capture_browser_scenario parents", () => { }); }); - it("names the one view a scenario capture supports when another is requested", () => { + it("names the views a scenario capture supports when another is requested", () => { const refused = inspectAnalysisView({ source: { kind: "inline", @@ -81,7 +81,7 @@ describe("environment manifests from capture_browser_scenario parents", () => { expect(projectAnalysisError(refused.error)).toMatchObject({ code: "unsupported_target", message: expect.stringContaining( - "supports only the environment-manifest view", + "supports only the environment-manifest and environment-state views", ), }); }); diff --git a/src/application/environment/environmentSource.ts b/src/application/environment/environmentSource.ts index 569c2299ea..175c3f18d2 100644 --- a/src/application/environment/environmentSource.ts +++ b/src/application/environment/environmentSource.ts @@ -5,6 +5,10 @@ import { type AnalysisViewResult, } from "../../domain/analysisView/analysisView.js"; import { scenarioCaptureRecords } from "../../domain/analysisView/scenarioCaptureRecords.js"; +import { + readRecordedResponseBody, + type RecordedResponseBody, +} from "../../domain/analysisView/recordedResponseBody.js"; import { readRecordedRequestLine } from "../../domain/analysisView/webNetworkRouteTemplates.js"; import { browserScenarioCaptureSchema } from "../../domain/browserScenarioCapture.js"; import type { AnalysisError } from "../../domain/analysisErrorBase.js"; @@ -30,28 +34,19 @@ export type EnvironmentManifest = Extract< >; export type EnvironmentManifestTool = EnvironmentManifest["tools"][number]; -/** One retained parent record reduced to what a replay returns. */ -export interface RecordedExchange { +/** One retained parent record reduced to what a replay returns; a cut body is reported truncated. */ +export interface RecordedExchange extends RecordedResponseBody { readonly ordinal: number; readonly path: string; /** WHATWG query spelling including `?`, or null when the request had none. */ readonly query: string | null; readonly status: number | null; readonly contentType: string | null; - /** Producer-decoded response text, or null when the record retained none. */ - readonly bodyText: string | null; - /** Set when the retained bytes are not UTF-8, so `bodyText` holds their base64 rather than a lossy decoding. */ - readonly bodyEncoding?: "base64"; - /** Set when only a prefix of a large base64 body was decoded; replay reports it truncated. */ - readonly bodyCut?: true; } /** Replayed bodies larger than this are cut at a UTF-8 boundary and flagged. */ export const REPLAY_BODY_LIMIT_BYTES = 256 * 1024; -/** Base64 characters that decode to just over the replay limit, so a cut prefix still fills it. */ -const BASE64_PREFIX_CHARS = Math.ceil((REPLAY_BODY_LIMIT_BYTES + 4) / 3) * 4; - /** A HAR inspected by inspect_web_network_capture, or a session recorded by capture_browser_scenario. */ export type EnvironmentCaptureKind = "web-network-capture" | "browser-scenario"; @@ -107,52 +102,6 @@ const parseEvidenceFile = ( } }; -const contentTextSchema = z.object({ - response: z.object({ - content: z.object({ - text: z.unknown().optional(), - encoding: z.unknown().optional(), - }), - }), -}); - -/** Strict, and keeping a leading byte-order mark, so replay returns the recorded text unchanged. */ -const strictUtf8 = (): TextDecoder => - new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); - -/** The retained response text of one record, decoding HAR base64 content when the decoder kept it as valid UTF-8. */ -const recordedBody = ( - record: WebNetworkCaptureRecord, -): Pick => { - const content = contentTextSchema.safeParse(record.reported); - if (!content.success) return { bodyText: null }; - const { text, encoding } = content.data.response.content; - if (typeof text !== "string") return { bodyText: null }; - if (encoding !== "base64") return { bodyText: text }; - const binary = record.binary_fields.find( - (field) => - field.pointer === "/response/content/text" && field.state === "retained", - ); - if (binary === undefined || binary.content_base64 === null) - return { bodyText: null }; - // Replay returns at most REPLAY_BODY_LIMIT_BYTES, so a large body is decoded only that far. - const cut = binary.content_base64.length > BASE64_PREFIX_CHARS; - const encoded = cut - ? binary.content_base64.slice(0, BASE64_PREFIX_CHARS) - : binary.content_base64; - const flags = cut ? { bodyCut: true as const } : {}; - try { - // A cut prefix may end inside a character; streaming leaves that tail undecoded. - const text = strictUtf8().decode(Buffer.from(encoded, "base64"), { - stream: cut, - }); - return { bodyText: text, ...flags }; - } catch { - // Binary or non-UTF-8 bytes: keep them exactly, as base64, instead of inventing replacement characters. - return { bodyText: encoded, bodyEncoding: "base64", ...flags }; - } -}; - const recordedExchange = ( record: WebNetworkCaptureRecord, ): RecordedExchange | undefined => { @@ -164,7 +113,7 @@ const recordedExchange = ( query: line.query === undefined || line.query === "" ? null : line.query, status: line.status, contentType: line.rawMime === "" ? null : line.rawMime, - ...recordedBody(record), + ...readRecordedResponseBody(record, REPLAY_BODY_LIMIT_BYTES), }; }; diff --git a/src/contracts/analysisDetail.test.ts b/src/contracts/analysisDetail.test.ts index 22c0e72f92..275abc7ce9 100644 --- a/src/contracts/analysisDetail.test.ts +++ b/src/contracts/analysisDetail.test.ts @@ -38,6 +38,7 @@ it("advertises only that operation's summary from analysis detail outputs", () = ); expect(schema).not.toContain("api-surface"); expect(schema).not.toContain("environment-manifest"); + expect(schema).not.toContain("environment-state"); expect(schema).not.toContain("inspect_web_network_capture"); } const viewSchema = JSON.stringify( @@ -45,6 +46,7 @@ it("advertises only that operation's summary from analysis detail outputs", () = ); expect(viewSchema).toContain("api-surface"); expect(viewSchema).toContain("environment-manifest"); + expect(viewSchema).toContain("environment-state"); const layout = projectSummary(analysisViewLayoutEvidence()); const javascript = projectSummary(analysisViewJavaScriptEvidence()); diff --git a/src/contracts/analysisViewToolContracts.ts b/src/contracts/analysisViewToolContracts.ts index e904022e8b..ad6bf4fa89 100644 --- a/src/contracts/analysisViewToolContracts.ts +++ b/src/contracts/analysisViewToolContracts.ts @@ -17,7 +17,7 @@ export const ANALYSIS_VIEW_TOOL_CONTRACTS = [ ...toolContractMetadata("inspect_analysis_view"), kind: "application", description: - 'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods, with exact indices/names, fields and paging; types also select image_indices/namespaces/declaring_types and methods select declaring_types. Rows always keep their original index scoped to the parent artifact digest. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native function Evidence requires kind native: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', + 'Inspect selected views of completed binary-layout, JavaScript application, native analyze_function, inspect_artifact, inventory_artifact, inspect_wasm_artifact, inspect_macho, decode_interface_builder, inspect_evm_interface, inspect_managed_artifact, inspect_managed_members, inspect_android_package, inspect_dart_aot, inspect_il2cpp_metadata or inspect_web_network_capture Evidence without repeating analysis. Historical capture summaries return identity and counts without bodies or archive inventories. Capture views page retained records: view {kind: "capture", detail: "preview" | "complete", offset: 0, limit: 20}; preview selects recorded headline fields and binary descriptors, complete returns exact records with all sidecars. Offsets follow the selected parent record order, preserving original producer ordinals. Paging never reopens the source or reruns the decoder. For inspect_pcap_capture parents, kind pcap pages endpoints, dns-queries, tls-sni, protocols, http-uris or unattributed-addresses. Select an exact query for dns-addresses, or a tshark/capinfos stream for UTF-16 diagnostics text. Page totals count retained facts; source coverage remains explicit. Qlog summaries keep profile/scope and counts; kind qlog pages traces, streams, loss-declarations, incomplete-records, contributions, ranges or final-sizes. Except traces, select trace_index; stream-specific facets also require exact stream_id and direction. Facts retain all-recorded-events-before-selection scope, and interval overlap or endpoint loss declarations do not prove retransmission or network loss. The api-surface view groups retained HAR requests by method, origin and format-rule path template; templated segments are inferences. It reads no headers or bodies and reports no query values. The environment-manifest view projects the same HAR requests on the primary origin (most JSON-like records) into a replayable tool surface: view {kind: "environment-manifest", template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], include_query_parameter_names: true}. Each tool has a deterministic name, a path template and parameters from the four format rules plus the selected opt-in rules, observed query parameter names, a value-free body shape for POST/PUT/PATCH, status and content-type sets and record ordinals; records are pointers into the parent, excluded groups carry a reason and three fixed unknowns are always stated. Templates, parameters and names are inferences; the view copies no header, cookie or body values and refuses parents inspected with sensitive values. A capture_browser_scenario parent that selected response bodies or header values projects the same way, one record per network transaction in request order; it relies on capture-time redaction of declared secrets instead of that refusal, and a transaction whose URL was redacted is left unrepresented. The environment-state view extracts, from the same parents, the initial state the recorded reads observed: view {kind: "environment-state", template_rules: [...]} with the rules of the manifest it accompanies. It reads response bodies recorded before the first related write and reports entity types with an identity field (proved by a path parameter or inferred from a field name), entities in collection order with per-view values where views differ, documents for non-parameterized non-entity paths written by POST/PUT/PATCH (item-shaped and delete-only paths stay recorded), and a render plan per read tool: item and list views reproduce their latest recordings from the state, and every other read tool is a recorded view with a reason. Types and identity are inferences; values are recorded data, so it refuses parents inspected with sensitive values, and a scenario parent must have selected response bodies; its values keep the capture redaction markers, and a write whose URL it redacted ends the state of every type. For a retained record use source: {kind: "retained-evidence", evidence_id: ""}. Use normalized_result.parent_evidence_id from a summary receipt. Source kind is required. Portable source uses {kind: "inline", evidence: }. A completed Evidence JSON file on this host uses {kind: "evidence-file", path: ""}: REA reads and authenticates it, then projects it like retained Evidence, so a large capture is bounded by the file reader rather than by inline-input accounting. For layout, JavaScript, inventory, WASM, Mach-O, Interface Builder, EVM, managed artifact, Android package and Dart AOT sources, request a summary with view: {kind: "summary"}; summary is a view kind and has no facet. The WASM summary gives validation, counts, WAT digest/length, tool profile and execution/network states without reading WAT text. Select a layout mitigations/linkage facet, one section/symbol/module, or a stable page. Module pages include assets, bundled and source modules; use node_id for ambiguous or unavailable paths. Typed module references use view {kind: "javascript", collection: "modules", projection: "references", where: {path: "main.js", kind: "javascript-module"}, offset: 0, limit: 64}; where matches exact retained strings (null selects unavailable paths). Rows include canonical identity, retained source-range count and observation support states. Source coverage and selection remain explicit; exhausting matching rows does not prove runtime loading or complete analysis. For exact file/native facts and processing gaps, use view {kind: "javascript-facts", where: {paths: ["package.json"]}, property_keys: ["name", "operation", "analysis_status"]}. Rows page original observations with canonical identities and Evidence; missing properties remain omitted. Paths match recorded artifact-path/source-range locations; optional kinds filter nodes. To discover field names without values, opt into include_available_property_keys: true with property_keys: []. Canonical identity and Evidence are separate from properties; keep each node_id/observation_id pair together when composing results. Inventory uses view: {kind: "inventory", collection: "occurrences", logical_paths: ["package.json"], offset: 0, limit: 64}. Inventory views page nodes, occurrences, edges or integrity contradictions. Occurrences accept exact logical_paths and occurrence_ids; nodes accept artifact_ids. Batch filters intersect (omitted means all available, [] means none); fields selects columns while retaining identity links. source_complete describes retained graph membership; source_coverage describes producer coverage, including unknown encrypted PCK directories. Page exhaustion does not prove absence in a partial source. WASM views use kind wasm and facet sections/imports/exports/candidates/glue/wat/headers/details/tool_profile; text facets use UTF-16 code units and source forms remain exact. Mach-O views use kind macho and facet entrypoints/architectures/build_metadata/load_commands/dependencies/imports/exports/segments/provenance; source_exhaustive false means unreturned rows may exist. Interface Builder views use kind interface-builder and collection documents/nodes/edges/coverage; nodes and edges accept an exact document relative_path. EVM summary returns all inferred interface candidates, carrier/decoded identities and explicit unknowns without hex or duplicate stdout. EVM views use kind evm and facet functions/bytecode_hex/stdout/stderr; function pages preserve the original candidate rows, and text offsets/limits count UTF-16 code units (hex has two characters per decoded byte). Exhausting candidate pages does not establish global interface completeness. Managed artifact summaries preserve complete identity/classification, metadata status, module/MVID, frameworks, coverage and limitations without unrelated inventories. Managed pages use kind managed and collection references/resources/attributes, retaining source coverage and build-local token scope; page exhaustion does not establish complete metadata. Managed member summaries keep identity, selection with unmatched names, coverage and per-collection counts; managed-members pages use collection types/exported_types/fields/methods/member_refs/call_edges/field_accesses/user_strings and carry the source selection and coverage. Android package summaries retain all declarations, engine identity and counts without decoded manifest text. Android manifest pages use kind android, facet manifest, offset and limit in UTF-16 code units; source_status preserves partial decoded coverage even when the retained text page is exhausted. Dart AOT summary preserves target/ABI/library identity, snapshot headers and pool lower bounds. String pages use kind dart-aot, collection package_uris/dart_uris/class_like_tokens; source coverage remains partial when the producer says partial. IL2CPP metadata views use kind il2cpp-metadata and collection sections/images/types/methods, with exact indices/names, fields and paging; types also select image_indices/namespaces/declaring_types and methods select declaring_types. Rows always keep their original index scoped to the parent artifact digest. Desktop directory views use kind desktop-application and collection files/omissions/candidates/components/next_probes, with fields and pagination; files and omissions accept exact paths or their stable IDs, and candidates accept families. Source coverage and stable row IDs remain available. Native function Evidence requires kind native: view: {kind: "native", facet: "procedure"}. Facets select procedure, pseudocode, assembly, basic blocks, comments, callers/callees, references, unresolved calls, referenced strings/names, native API and high-pcode. Use nonnegative offset and a positive caller-selected limit (defaults 0 and 64); pseudocode uses UTF-16 code units, other native pages use rows. Returns artifact identity, parent Evidence ID, a distinct view digest, coverage, limitations and unknowns. Actual serialized size determines MCP transport admission; choose a smaller page or export the complete retained Evidence if needed.', inputSchema: inspectAnalysisViewInputSchema, outputSchema: evidenceResultOf(analysisViewResultSchema), examples: [ diff --git a/src/domain/analysisView/EnvironmentManifestAccumulator.ts b/src/domain/analysisView/EnvironmentManifestAccumulator.ts index ff59a3410b..d9d8928021 100644 --- a/src/domain/analysisView/EnvironmentManifestAccumulator.ts +++ b/src/domain/analysisView/EnvironmentManifestAccumulator.ts @@ -195,11 +195,17 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes, readonly includeQueryParameterNames = true, rules: readonly TemplateRule[] = [], + budgetSubject = "Environment-manifest", ) { - super(maximumBytes, "Environment-manifest"); + super(maximumBytes, budgetSubject); this.#rules = new Set(rules); } + /** Whether the finished projection publishes body shapes, and so states their limitation. */ + protected get reportsBodyShapes(): boolean { + return true; + } + add(record: WebNetworkCaptureRecord): Result { const limitations = this.addLimitations(record.limitations); if (!limitations.ok) return limitations; @@ -500,7 +506,8 @@ export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumula compare(left.reason, right.reason), ); const limitations = this.addLimitations([ - ...(tools.some((tool) => tool.body_shape !== null) + ...(this.reportsBodyShapes && + tools.some((tool) => tool.body_shape !== null) ? [ "Body shapes list top-level JSON key types from one recorded request each; they do not establish required fields or a complete schema.", ] diff --git a/src/domain/analysisView/EnvironmentStateAccumulator.ts b/src/domain/analysisView/EnvironmentStateAccumulator.ts new file mode 100644 index 0000000000..93c115e1c1 --- /dev/null +++ b/src/domain/analysisView/EnvironmentStateAccumulator.ts @@ -0,0 +1,283 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { parseExactJson } from "../exactJsonNumber.js"; +import { ok, type Result } from "../result.js"; +import { + WEB_NETWORK_CAPTURE_LIMITS, + type WebNetworkCaptureRecord, +} from "../webNetworkCapture.js"; +import type { UnsignedAnalysisView } from "./analysisView.js"; +import { + EnvironmentManifestAccumulator, + isJsonLike, +} from "./EnvironmentManifestAccumulator.js"; +import { EnvironmentStateExtraction } from "./EnvironmentStateExtraction.js"; +import type { TemplateRule } from "./environmentManifestRules.js"; +import { + isStateWrite, + type ParsedBody, + type StateRecordFacts, + type StateTimeline, +} from "./environmentStateRules.js"; +import { + base64PrefixChars, + readRecordedResponseBody, +} from "./recordedResponseBody.js"; +import { + ENTRY_BYTES, + RECORD_SCRATCH_BYTES, + mediaTypeEssence, + readRecordedRequestLine, + textBytes, +} from "./webNetworkRouteTemplates.js"; + +type StateView = Extract; +type StateFields = Pick< + StateView, + | "origin" + | "name" + | "timeline" + | "types" + | "entities" + | "documents" + | "views" + | "unknowns" + | "coverage" + | "limitations" +>; + +/** Bodies beyond this many UTF-8 bytes are not parsed into state; their views stay recorded. */ +export const STATE_BODY_LIMIT_BYTES = 256 * 1024; + +const timingSchema = z.object({ + startedDateTime: z.string(), + time: z.number().nonnegative(), +}); +const redactedWriteSchema = z.object({ + request: z.object({ method: z.string() }), + response: z.object({ status: z.number() }).optional(), +}); +const contentSchema = z.object({ + response: z.object({ content: z.object({ text: z.string() }) }), +}); + +interface RecordEntry { + readonly facts: StateRecordFacts; + /** Null for a write whose URL was redacted: its target is unknown, so it bounds every type. */ + readonly origin: string | null; + readonly contentType: string; + readonly timed: boolean; + readonly record: WebNetworkCaptureRecord; +} + +const UNKNOWNS: StateView["unknowns"] = [ + { + kind: "authorization-model", + statement: + "Authorization headers and cookies are structurally excluded from captures; the state states no authorization model.", + }, + { + kind: "response-semantics", + statement: + "Views reproduce their recorded responses from the state; they do not establish how the application computes responses, derived values or values copied outside entity objects.", + }, + { + kind: "state-transition", + statement: + "The state is one snapshot from before the first related recorded write; it states no effects of writes.", + }, +]; + +/** The manifest's grouping plus the record facts and bodies the init state is extracted from. */ +export class EnvironmentStateAccumulator extends EnvironmentManifestAccumulator { + readonly #entries = new Map(); + + constructor( + maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes, + rules: readonly TemplateRule[] = [], + ) { + super(maximumBytes, true, rules, "Environment-state"); + } + + /** The state publishes no body shapes. */ + protected override get reportsBodyShapes(): boolean { + return false; + } + + override add(record: WebNetworkCaptureRecord): Result { + const grouped = super.add(record); + if (!grouped.ok) return grouped; + try { + const line = readRecordedRequestLine(record.reported, true, this.scratch); + if (line === null) return this.overBudget(); + const timing = timingSchema.safeParse(record.reported); + const start = timing.success + ? Date.parse(timing.data.startedDateTime) + : Number.NaN; + const timed = timing.success && Number.isFinite(start); + if (line.kind === "excluded") + return line.reason === "url-redacted" + ? this.#redactedWrite( + record, + timing.success && timed + ? { start, end: start + timing.data.time } + : undefined, + ) + : ok(undefined); + // The entry keeps the line's strings and a reference to the already admitted record. + if ( + !this.retain( + ENTRY_BYTES * 8 + + textBytes( + line.method.length + + line.origin.length + + line.path.length + + (line.query?.length ?? 0) + + line.rawMime.length, + ), + ) + ) + return this.overBudget(); + this.#entries.set(record.ordinal, { + facts: { + ordinal: record.ordinal, + method: line.method, + path: line.path, + query: line.query === "" ? undefined : line.query, + status: line.status, + start: timed ? start : record.ordinal, + end: timed ? start + timing.data.time : record.ordinal, + }, + origin: line.origin, + contentType: line.rawMime, + timed, + record, + }); + return ok(undefined); + } finally { + this.releaseTemporary(); + } + } + + /** Keep a successful write whose URL was redacted, with an empty path whose root relates to every type. */ + #redactedWrite( + record: WebNetworkCaptureRecord, + timing: { readonly start: number; readonly end: number } | undefined, + ): Result { + const parsed = redactedWriteSchema.safeParse(record.reported); + if (!parsed.success) return ok(undefined); + const status = parsed.data.response?.status ?? 0; + const facts: StateRecordFacts = { + ordinal: record.ordinal, + method: parsed.data.request.method, + path: "", + query: undefined, + status: status > 0 ? status : null, + start: timing?.start ?? record.ordinal, + end: timing?.end ?? record.ordinal, + }; + if (!isStateWrite(facts)) return ok(undefined); + if (!this.retain(ENTRY_BYTES * 8 + textBytes(facts.method.length))) + return this.overBudget(); + this.#entries.set(record.ordinal, { + facts, + origin: null, + contentType: "", + timed: timing !== undefined, + record, + }); + return ok(undefined); + } + + /** Parse one retained response body as exact JSON, charging its decoding and tree as temporary work. */ + #body(ordinal: number): ParsedBody | null { + const entry = this.#entries.get(ordinal); + if (entry === undefined || !isJsonLike(mediaTypeEssence(entry.contentType))) + return { kind: "unusable" }; + const content = contentSchema.safeParse(entry.record.reported); + if (!content.success) return { kind: "unusable" }; + // Larger bodies are never parsed; within the limit, decoding copies at most the retained text and the tree is linear in it. + const length = content.data.response.content.text.length; + if (length > base64PrefixChars(STATE_BODY_LIMIT_BYTES)) + return { kind: "unusable" }; + if (!this.temporary(RECORD_SCRATCH_BYTES + textBytes(length) * 4)) + return null; + const body = readRecordedResponseBody(entry.record, STATE_BODY_LIMIT_BYTES); + if ( + body.bodyText === null || + body.bodyEncoding === "base64" || + body.bodyCut === true || + Buffer.byteLength(body.bodyText, "utf8") > STATE_BODY_LIMIT_BYTES + ) + return { kind: "unusable" }; + const value = parseExactJson(body.bodyText); + return value === undefined ? { kind: "unusable" } : { kind: "json", value }; + } + + /** Project the init state of the manifest's primary origin; records keep timeline order. */ + finishState(): Result { + const manifest = this.finish(); + if (!manifest.ok) return manifest; + const entries = [...this.#entries.values()].filter( + (entry) => + entry.origin === manifest.value.origin || entry.origin === null, + ); + const redactedWrites = entries.filter( + (entry) => entry.origin === null, + ).length; + const timeline: StateTimeline = + entries.length > 0 && entries.every((entry) => entry.timed) + ? "recorded-start-time" + : "record-order"; + const records = entries + .map((entry) => entry.facts) + .sort((left, right) => + timeline === "record-order" + ? left.ordinal - right.ordinal + : left.start - right.start || left.ordinal - right.ordinal, + ) + .map((facts) => + timeline === "record-order" + ? { ...facts, start: facts.ordinal, end: facts.ordinal } + : facts, + ); + const state = new EnvironmentStateExtraction({ + tools: manifest.value.tools, + records, + timeline, + budget: { + body: (ordinal) => this.#body(ordinal), + retain: (bytes) => this.retain(bytes), + release: () => this.releaseTemporary(), + }, + }).run(); + if (state === null) return this.overBudget(); + const limitations = this.addLimitations([ + ...(redactedWrites > 0 + ? [ + `${String(redactedWrites)} of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.`, + ] + : []), + ...(timeline === "record-order" + ? [ + "Some records carry no start time, so the state's timeline is record order and overlapping requests are not detected.", + ] + : []), + ...(state.views.some((view) => view.shape === "recorded") + ? [ + "Tools whose recordings do not reproduce from the extracted state are listed as recorded views with the reason; they replay their recordings.", + ] + : []), + ]); + if (!limitations.ok) return limitations; + return ok({ + origin: manifest.value.origin, + name: manifest.value.name, + timeline, + ...state, + unknowns: UNKNOWNS.map((unknown) => ({ ...unknown })), + coverage: manifest.value.coverage, + limitations: this.sortedLimitations(), + }); + } +} diff --git a/src/domain/analysisView/EnvironmentStateExtraction.ts b/src/domain/analysisView/EnvironmentStateExtraction.ts new file mode 100644 index 0000000000..7ed3ef79ec --- /dev/null +++ b/src/domain/analysisView/EnvironmentStateExtraction.ts @@ -0,0 +1,587 @@ +import type { JsonValue } from "../jsonValue.js"; +import { + compareUnicodeCodePoints as compare, + compositeKey, +} from "../unicodeCodePointOrder.js"; +import { + IDENTITY_NAMES, + at, + collectionPathOf, + identifies, + isObject, + isParam, + isQueryPost, + isStateWrite, + jsonBytes, + hasPrototypeKey, + lastSegmentValue, + listItems, + normalizedPath, + locateItem, + locateList, + rootOf, + rootsRelated, + scalarText, + segmentsOf, + apiBaseLengths, + writeSpanCheck, + type ExtractedState, + type JsonObject, + type RecordedViewReason, + type StateBudget, + type StateDocument, + type StateEntity, + type StateRecordFacts, + type StateTimeline, + type StateToolFacts, + type StateType, + type StateView, +} from "./environmentStateRules.js"; +import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js"; +import { + classifyEntityChanges, + compareObservedEntities, + mergeObservedEntity, + type ObservedEntity, + type StateObservation, +} from "./environmentStateMerge.js"; +import { + checkItemPlan, + checkListPlan, + type StateGateContext, + type StatePlan, +} from "./environmentStateGate.js"; + +class StateBudgetExhausted extends Error {} + +type Candidate = Pick; + +interface ObservedMembership { + record: StateRecordFacts; + readonly ids: Set; + capacity: number; +} + +const entityKey = (type: string, collection: string, id: string): string => + compositeKey([type, collection, id]); + +const requestLine = (record: StateRecordFacts): string => + record.path + (record.query ?? ""); + +/** Derive the init state and render plans from manifest tools and timeline-ordered records. */ +export class EnvironmentStateExtraction { + readonly #tools: readonly StateToolFacts[]; + readonly #timeline: StateTimeline; + readonly #budget: StateBudget; + readonly #position: ReadonlyMap; + readonly #byOrdinal: ReadonlyMap; + readonly #writes: readonly StateRecordFacts[]; + readonly #writeWithin: ReturnType; + readonly #bases: ReadonlyMap; + readonly #identity = new Map< + string, + { readonly field: string; readonly basis: StateType["id_basis"] } + >(); + readonly #views = new Map(); + readonly #plans: StatePlan[] = []; + readonly #entities = new Map(); + readonly #volatile = new Map>(); + /** Types with a value that changed across a recorded write, which may be that write's effect. */ + readonly #rewritten = new Set(); + readonly #merged = new Map(); + /** Merged entities by type and collection, built once on first use after merging. */ + #collections: Map | undefined; + + constructor(input: { + readonly tools: readonly StateToolFacts[]; + readonly records: readonly StateRecordFacts[]; + readonly timeline: StateTimeline; + readonly budget: StateBudget; + }) { + this.#tools = input.tools; + this.#timeline = input.timeline; + this.#budget = input.budget; + this.#position = new Map( + input.records.map((record, index) => [record.ordinal, index]), + ); + this.#byOrdinal = new Map( + input.records.map((record) => [record.ordinal, record]), + ); + this.#writes = input.records.filter(isStateWrite); + this.#writeWithin = writeSpanCheck(this.#writes); + this.#bases = apiBaseLengths(input.tools.map((tool) => tool.path_template)); + } + + /** The extracted state, or null when the representation budget is exhausted. */ + run(): ExtractedState | null { + try { + const candidates = this.#candidates(); + for (const candidate of candidates.filter((entry) => entry.item)) + this.#planItem(candidate); + for (const candidate of candidates.filter((entry) => !entry.item)) + this.#planList(candidate); + for (const plan of this.#plans) this.#observePlan(plan); + this.#classifyChanges(); + this.#merge(); + const gate = this.#gateContext(); + for (const plan of this.#plans) + if (this.#rewritten.has(plan.type)) + this.#recorded(plan.tool, "fields-differ"); + else + this.#views.set( + plan.tool.name, + plan.item ? checkItemPlan(gate, plan) : checkListPlan(gate, plan), + ); + const documents = this.#documents(candidates); + for (const { tool } of candidates) + if (!this.#views.has(tool.name)) this.#recorded(tool, "not-entity"); + return this.#result(documents); + } catch (cause: unknown) { + if (cause instanceof StateBudgetExhausted) return null; + throw cause; + } + } + + #retain(bytes: number): void { + if (!this.#budget.retain(bytes)) throw new StateBudgetExhausted(); + } + + /** Parse one body and hand it to `use` before its temporary work is dropped; unusable bodies arrive undefined. */ + #withBody(ordinal: number, use: (body: JsonValue | undefined) => T): T { + const parsed = this.#budget.body(ordinal); + if (parsed === null) throw new StateBudgetExhausted(); + try { + return use( + parsed.kind === "json" && !hasPrototypeKey(parsed.value) + ? parsed.value + : undefined, + ); + } finally { + this.#budget.release(); + } + } + + #recorded(tool: StateToolFacts, reason: RecordedViewReason): void { + this.#views.set(tool.name, { tool: tool.name, shape: "recorded", reason }); + } + + /** GET tools may render from state; query POSTs stay recorded. */ + #candidates(): Candidate[] { + const candidates: Candidate[] = []; + for (const tool of this.#tools) { + if (isQueryPost(tool.method, tool.path_template)) + this.#recorded(tool, "query-body"); + if (tool.method !== "GET") continue; + const segments = segmentsOf(tool.path_template); + const item = segments.length > 0 && isParam(segments.at(-1)!); + candidates.push({ + tool, + item, + type: item + ? `/${segments.slice(0, -1).join("/")}` + : normalizedPath(tool.path_template), + }); + } + return candidates; + } + + /** The first write whose root relates to the template's bounds its init recordings. */ + #initLimit(template: string): StateRecordFacts | undefined { + const root = rootOf(template, this.#bases); + return this.#writes.find((write) => + rootsRelated(root, rootOf(write.path, this.#bases)), + ); + } + + /** Successful recordings that finished before the first related write began. */ + #initRecordings(tool: StateToolFacts, template: string): StateRecordFacts[] { + const limit = this.#initLimit(template); + const before = (record: StateRecordFacts): boolean => + limit === undefined || + (this.#position.get(record.ordinal)! < + this.#position.get(limit.ordinal)! && + (this.#timeline === "record-order" || record.end <= limit.start)); + return tool.record_ordinals + .map((ordinal) => this.#byOrdinal.get(ordinal)) + .filter( + (record): record is StateRecordFacts => + record !== undefined && + record.status !== null && + record.status >= 200 && + record.status < 300 && + before(record), + ) + .sort( + (left, right) => + this.#position.get(left.ordinal)! - + this.#position.get(right.ordinal)!, + ); + } + + #latestPerLine(recordings: readonly StateRecordFacts[]): StateRecordFacts[] { + const latest = new Map(); + for (const record of recordings) { + latest.delete(requestLine(record)); + latest.set(requestLine(record), record); + } + return [...latest.values()]; + } + + /** A candidate's initial recordings, or undefined after recording that it has none. */ + #initRecordingsOf(candidate: Candidate): StateRecordFacts[] | undefined { + const recordings = this.#initRecordings(candidate.tool, candidate.type); + if (recordings.length > 0) return recordings; + this.#recorded(candidate.tool, "no-init-recording"); + return undefined; + } + + #addPlan( + candidate: Candidate, + pointer: string, + recordings: readonly StateRecordFacts[], + ): void { + this.#plans.push({ + ...candidate, + pointer, + recordings, + latest: this.#latestPerLine(recordings), + }); + } + + /** Item tools prove a type's identity field by the path value their bodies repeat. */ + #planItem(candidate: Candidate): void { + const { tool, type } = candidate; + const recordings = this.#initRecordingsOf(candidate); + if (recordings === undefined) return; + let pointer: string | undefined; + let keys: string[] | undefined; + for (const record of recordings) { + const located = this.#withBody(record.ordinal, (body) => + body === undefined + ? null + : locateItem(body, lastSegmentValue(record.path)), + ); + if (located === null) { + this.#recorded(tool, "body-unusable"); + return; + } + if ( + located === undefined || + (pointer ?? located.pointer) !== located.pointer + ) { + this.#recorded(tool, "no-identity"); + return; + } + pointer = located.pointer; + if (keys === undefined) keys = located.keys; + else { + const matching = new Set(located.keys); + keys = keys.filter((key) => matching.has(key)); + } + } + const field = + IDENTITY_NAMES.find((name) => keys?.includes(name) === true) ?? keys?.[0]; + const known = this.#identity.get(type); + if ( + field === undefined || + pointer === undefined || + (known !== undefined && known.field !== field) + ) { + this.#recorded(tool, "no-identity"); + return; + } + this.#identity.set(type, { field, basis: "path-parameter" }); + this.#addPlan(candidate, pointer, recordings); + } + + /** List tools use the proved field, else the first identity name every non-empty recording's items carry. */ + #planList(candidate: Candidate): void { + const { tool, type } = candidate; + const recordings = this.#initRecordingsOf(candidate); + if (recordings === undefined) return; + const proved = this.#identity.get(type)?.field; + let pointer: string | undefined; + let names: readonly string[] | undefined; + let sawItems = false; + for (const record of recordings) { + const found = this.#withBody(record.ordinal, (body) => { + if (body === undefined) return null; + // Every recording must locate the same list, so a later entity envelope is not taken as one. + const located = locateList(body, proved); + if (pointer !== undefined && located !== pointer) return undefined; + const items = + located === undefined ? undefined : listItems(body, located); + if (items === undefined) return undefined; + pointer = located; + if (items.length > 0) sawItems = true; + return items.length === 0 + ? (names ?? (proved === undefined ? IDENTITY_NAMES : [proved])) + : (proved === undefined ? IDENTITY_NAMES : [proved]).filter((name) => + identifies(items, name), + ); + }); + if (found === null || found === undefined) { + this.#recorded(tool, found === null ? "body-unusable" : "not-entity"); + return; + } + names = (names ?? found).filter((name) => found.includes(name)); + } + const field = names?.[0]; + // An identity name is only inferred from items that carry it; empty lists prove none. + if ( + field === undefined || + pointer === undefined || + (proved === undefined && !sawItems) + ) { + this.#recorded(tool, "no-identity"); + return; + } + if (!this.#identity.has(type)) + this.#identity.set(type, { field, basis: "field-name" }); + this.#addPlan(candidate, pointer, recordings); + } + + #observePlan(plan: StatePlan): void { + // Only lists whose observations span a write need membership snapshots, including empty lists. + const memberships = + !plan.item && + this.#writeWithin(plan.recordings[0]!, plan.recordings.at(-1)!) + ? new Map() + : undefined; + for (const record of plan.recordings) + this.#withBody(record.ordinal, (body) => { + const located = body === undefined ? undefined : at(body, plan.pointer); + const objects = plan.item + ? isObject(located) + ? [located] + : [] + : body === undefined + ? [] + : (listItems(body, plan.pointer) ?? []); + if (memberships !== undefined) + this.#observeMembership(plan, record, objects, memberships); + objects.forEach((object, index) => + this.#observe(plan, record, object, index), + ); + }); + } + + /** Compare one request line's membership before discarding older recordings; reuse its bounded set. */ + #observeMembership( + plan: StatePlan, + record: StateRecordFacts, + objects: readonly JsonObject[], + memberships: Map, + ): void { + const line = requestLine(record); + let previous = memberships.get(line); + const earlier = previous?.record; + if (previous === undefined) { + this.#retain(ENTRY_BYTES * 2 + textBytes(line.length)); + previous = { record, ids: new Set(), capacity: 0 }; + memberships.set(line, previous); + } + // Reserve the peak membership representation before growing the set, including numeric ID text. + if (objects.length > previous.capacity) { + this.#retain((objects.length - previous.capacity) * ENTRY_BYTES * 4); + previous.capacity = objects.length; + } + const field = this.#identity.get(plan.type)!.field; + let changed = previous.ids.size !== objects.length; + for (const object of objects) + if (!previous.ids.delete(scalarText(object[field])!)) changed = true; + if (previous.ids.size > 0) changed = true; + if (earlier !== undefined && changed && this.#writeWithin(earlier, record)) + this.#rewritten.add(plan.type); + previous.ids.clear(); + for (const object of objects) previous.ids.add(scalarText(object[field])!); + previous.record = record; + } + + /** Record one observed entity object; a value that changes between one tool's own observations is volatile. */ + #observe( + plan: StatePlan, + record: StateRecordFacts, + object: JsonObject, + index: number, + ): void { + const field = this.#identity.get(plan.type)!.field; + const id = scalarText(object[field]); + if (id === undefined) return; + const collection = collectionPathOf(record, plan.item); + const key = entityKey(plan.type, collection, id); + const position = this.#position.get(record.ordinal)!; + let entity = this.#entities.get(key); + if (entity === undefined) { + this.#retain(ENTRY_BYTES * 4 + textBytes(key.length)); + entity = { + type: plan.type, + collection, + id, + order: [position, index], + observations: [], + }; + this.#entities.set(key, entity); + } else { + // Plans are observed item tools first, so keep the earliest sighting in timeline order. + const [first, at] = entity.order; + if (position < first || (position === first && index < at)) + entity.order = [position, index]; + } + this.#retain(ENTRY_BYTES + jsonBytes(object)); + entity.observations.push({ + position, + ordinal: record.ordinal, + tool: plan.tool.name, + item: plan.item, + object, + }); + } + + /** A change across any recorded write rewrites the type; one between repeats of a line with no write is volatile. */ + #classifyChanges(): void { + const recordOf = (observation: StateObservation): StateRecordFacts => + this.#byOrdinal.get(observation.ordinal)!; + const changes = classifyEntityChanges( + this.#entities.values(), + (observation) => [observation.tool, requestLine(recordOf(observation))], + (earlier, later) => this.#writeWithin(recordOf(earlier), recordOf(later)), + ); + for (const type of changes.rewritten) this.#rewritten.add(type); + for (const [type, names] of changes.volatile) + this.#volatile.set(type, names); + } + + #merge(): void { + for (const entity of [...this.#entities.values()].sort( + compareObservedEntities, + )) + this.#merged.set( + entityKey(entity.type, entity.collection, entity.id), + mergeObservedEntity(entity), + ); + } + + #gateContext(): StateGateContext { + return { + withBody: (ordinal, use) => this.#withBody(ordinal, use), + retain: (bytes) => this.#retain(bytes), + entity: (type, collection, id) => + this.#merged.get(entityKey(type, collection, id)), + inCollection: (type, collection) => this.#inCollection(type, collection), + volatileOf: (type) => this.#volatileOf(type), + idField: (type) => this.#identity.get(type)!.field, + }; + } + + #volatileOf(type: string): ReadonlySet { + return this.#volatile.get(type) ?? new Set(); + } + + #inCollection(type: string, collection: string): StateEntity[] { + if (this.#collections === undefined) { + this.#retain(ENTRY_BYTES * (this.#merged.size + 1)); + this.#collections = new Map(); + for (const entity of this.#merged.values()) { + const key = JSON.stringify([entity.type, entity.collection]); + const members = this.#collections.get(key); + if (members === undefined) this.#collections.set(key, [entity]); + else members.push(entity); + } + } + return this.#collections.get(JSON.stringify([type, collection])) ?? []; + } + + /** A written path whose read is not an entity holds one document value: its latest init recording. */ + #documents(candidates: readonly Candidate[]): StateDocument[] { + const written = new Set( + this.#tools + .filter( + (tool) => + ["POST", "PUT", "PATCH"].includes(tool.method) && + !isQueryPost(tool.method, tool.path_template), + ) + .map((tool) => tool.path_template), + ); + const documents: StateDocument[] = []; + for (const { tool, item } of candidates) { + const current = this.#views.get(tool.name); + const entityless = + current === undefined || + (current.shape === "recorded" && + (current.reason === "not-entity" || + current.reason === "no-identity")); + if (item || !entityless || !written.has(tool.path_template)) continue; + const latest = this.#latestPerLine( + this.#initRecordings(tool, tool.path_template), + ); + if (latest.length !== 1) { + if (latest.length > 1) this.#recorded(tool, "query-varies"); + continue; + } + const record = latest[0]!; + const value = this.#withBody(record.ordinal, (body) => body); + if (value === undefined) { + this.#recorded(tool, "body-unusable"); + continue; + } + this.#retain( + ENTRY_BYTES + jsonBytes(value) + textBytes(record.path.length), + ); + documents.push({ + tool: tool.name, + path: record.path, + value, + record_ordinal: record.ordinal, + }); + this.#views.set(tool.name, { + tool: tool.name, + shape: "document", + checked_record_ordinals: [record.ordinal], + }); + } + return documents.sort((left, right) => compare(left.tool, right.tool)); + } + + /** Types and entities that at least one state-backed view renders; view values only for those views. */ + #result(documents: StateDocument[]): ExtractedState { + const backed = new Set( + [...this.#views.values()].flatMap((view) => + view.shape === "item" || view.shape === "list" ? [view.tool] : [], + ), + ); + const types = [...this.#identity.entries()] + .filter(([type]) => + this.#plans.some( + (plan) => plan.type === type && backed.has(plan.tool.name), + ), + ) + .map(([type, { field, basis }]): StateType => ({ + collection_template: type, + id_field: field, + id_basis: basis, + init_before_ordinal: this.#initLimit(type)?.ordinal ?? null, + volatile_fields: [...this.#volatileOf(type)].sort(compare), + })) + .sort((left, right) => + compare(left.collection_template, right.collection_template), + ); + const typed = new Set(types.map((type) => type.collection_template)); + return { + types, + entities: [...this.#merged.values()] + .filter((entity) => typed.has(entity.type)) + .map((entity) => ({ + ...entity, + view_fields: entity.view_fields.filter((view) => + backed.has(view.tool), + ), + })), + documents, + views: this.#tools.flatMap((tool) => { + const view = this.#views.get(tool.name); + return view === undefined ? [] : [view]; + }), + }; + } +} diff --git a/src/domain/analysisView/WebNetworkProjectionAccumulator.ts b/src/domain/analysisView/WebNetworkProjectionAccumulator.ts index 89a8b0d4d7..e512ba4f69 100644 --- a/src/domain/analysisView/WebNetworkProjectionAccumulator.ts +++ b/src/domain/analysisView/WebNetworkProjectionAccumulator.ts @@ -99,7 +99,7 @@ export abstract class WebNetworkProjectionAccumulator { /** Authenticate and gate a retained HAR parent before a grouping view reads any record. */ export const admitHarCaptureParent = ( parent: AnalysisViewParent, - viewKind: "api-surface" | "environment-manifest", + viewKind: "api-surface" | "environment-manifest" | "environment-state", accumulator: WebNetworkProjectionAccumulator, ): Result => { const unsupported = (reason: string) => @@ -112,7 +112,7 @@ export const admitHarCaptureParent = ( ); if (parent.operation !== "inspect_web_network_capture") return unsupported( - `View ${viewKind} is unsupported for parent operation ${parent.operation}; ${viewKind} requires ${viewKind === "environment-manifest" ? "inspect_web_network_capture or capture_browser_scenario" : "inspect_web_network_capture"}.`, + `View ${viewKind} is unsupported for parent operation ${parent.operation}; ${viewKind} requires ${viewKind === "api-surface" ? "inspect_web_network_capture" : "inspect_web_network_capture or capture_browser_scenario"}.`, ); const admitted = accumulator.admitCaptureClone(parent.normalizedResult); if (!admitted.ok) return admitted; @@ -132,11 +132,11 @@ export const admitHarCaptureParent = ( return unsupported( `${viewKind} does not support capture format ${parsed.data.format}.`, ); - // The manifest cites the report path and digest, so its parent's subject must be exactly them; + // Environment views cite the report path and digest, so their parent's subject must be exactly them; // the api-surface view keeps the admission its existing callers rely on. const { artifact } = parsed.data; if ( - viewKind === "environment-manifest" && + viewKind !== "api-surface" && artifact.sha256 !== null && (parent.subject == null || parent.subject.local_path !== artifact.path || diff --git a/src/domain/analysisView/analysisView.ts b/src/domain/analysisView/analysisView.ts index 58cb333d48..9a30bde79c 100644 --- a/src/domain/analysisView/analysisView.ts +++ b/src/domain/analysisView/analysisView.ts @@ -105,11 +105,17 @@ import { } from "./webNetworkCaptureView.js"; import type { WebNetworkApiSurfaceAccumulator } from "./WebNetworkApiSurfaceAccumulator.js"; import type { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js"; +import type { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js"; import { projectEnvironmentManifestView, environmentManifestFields, environmentManifestViewRequestSchema, } from "./environmentManifestView.js"; +import { + environmentStateFields, + environmentStateViewRequestSchema, + projectEnvironmentStateView, +} from "./environmentStateView.js"; import { pcapViewRequestSchema, pcapViewPageSchema, @@ -239,6 +245,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [ androidViewRequestSchema, apiSurfaceViewSchema, environmentManifestViewRequestSchema, + environmentStateViewRequestSchema, captureViewRequestSchema, qlogViewRequestSchema, pcapViewRequestSchema, @@ -512,6 +519,16 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [ ]), view: environmentManifestViewRequestSchema, }), + z.strictObject({ + ...viewResultBase, + ...environmentStateFields, + kind: z.literal("environment-state"), + parent_operation: z.enum([ + "inspect_web_network_capture", + "capture_browser_scenario", + ]), + view: environmentStateViewRequestSchema, + }), z.strictObject({ ...summaryViewResultBase, summary: z.union([ @@ -706,6 +723,10 @@ export type UnsignedAnalysisView = Extract, "view_digest" > + | Omit< + Extract, + "view_digest" + > | Omit< Extract, "view_digest" @@ -807,7 +828,7 @@ const unsupportedParent = ( INSPECT_ANALYSIS_VIEW_OPERATION, parentArtifactPath(parent), parent.operation === "capture_browser_scenario" - ? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest view.` + ? `Parent Evidence operation "capture_browser_scenario" supports only the environment-manifest and environment-state views.` : `Parent Evidence operation ${JSON.stringify(parent.operation)} does not support analysis views. Use completed binary layout, JavaScript application, native function, artifact inventory/inspection, WASM, Mach-O, Interface Builder, EVM, managed artifact or member, Android package, Dart AOT, IL2CPP metadata, or web network capture Evidence.`, ); @@ -848,12 +869,47 @@ const retainedViewProjectors = new Map([ ["inventory_artifact", projectArtifactInventoryView], ]); +/** Budgets a caller created before admitting the source; each grouping view uses its own. */ +export interface AnalysisViewAccumulators { + readonly apiSurface?: WebNetworkApiSurfaceAccumulator | undefined; + readonly manifest?: EnvironmentManifestAccumulator | undefined; + readonly state?: EnvironmentStateAccumulator | undefined; +} + +type GroupingViewRequest = Extract< + AnalysisViewRequest, + { kind: "api-surface" | "environment-manifest" | "environment-state" } +>; + +const isGroupingView = ( + view: AnalysisViewRequest, +): view is GroupingViewRequest => + view.kind === "api-surface" || + view.kind === "environment-manifest" || + view.kind === "environment-state"; + +/** Views that group a capture's records, each under the budget its caller created. */ +const projectGroupingView = ( + parent: AnalysisViewParent, + view: GroupingViewRequest, + accumulators: AnalysisViewAccumulators, +): Result => { + if (view.kind === "api-surface") + return projectWebNetworkApiSurfaceView( + parent, + view, + accumulators.apiSurface, + ); + if (view.kind === "environment-manifest") + return projectEnvironmentManifestView(parent, view, accumulators.manifest); + return projectEnvironmentStateView(parent, view, accumulators.state); +}; + /** Project one selected view from authenticated parent analysis Evidence. */ export const projectAnalysisView = ( parent: AnalysisViewParent, view: AnalysisViewRequest, - apiSurfaceAccumulator?: WebNetworkApiSurfaceAccumulator, - manifestAccumulator?: EnvironmentManifestAccumulator, + accumulators: AnalysisViewAccumulators = {}, ): Result => { const digest = parentDigest(parent.evidenceId); if (!digestSchema.safeParse(digest).success) @@ -866,21 +922,9 @@ export const projectAnalysisView = ( }, ]), ); - if (view.kind === "api-surface") { - const projected = projectWebNetworkApiSurfaceView( - parent, - view, - apiSurfaceAccumulator, - ); - return projected.ok ? ok(sealAnalysisView(projected.value)) : projected; - } - if (view.kind === "environment-manifest") { - const projected = projectEnvironmentManifestView( - parent, - view, - manifestAccumulator, - ); - return projected.ok ? ok(sealAnalysisView(projected.value)) : projected; + if (isGroupingView(view)) { + const grouped = projectGroupingView(parent, view, accumulators); + return grouped.ok ? ok(sealAnalysisView(grouped.value)) : grouped; } if (parent.operation === "inspect_web_network_capture") { const projected = projectWebNetworkCaptureView(parent, view); diff --git a/src/domain/analysisView/binaryLayoutView.ts b/src/domain/analysisView/binaryLayoutView.ts index 17e425915f..b441524ddd 100644 --- a/src/domain/analysisView/binaryLayoutView.ts +++ b/src/domain/analysisView/binaryLayoutView.ts @@ -159,7 +159,7 @@ export const projectBinaryLayoutView = ( layout: BinaryLayout, view: Exclude< AnalysisViewRequest, - { kind: "api-surface" | "environment-manifest" } + { kind: "api-surface" | "environment-manifest" | "environment-state" } >, ): Result => { const shared = parentFields(parent, layout); diff --git a/src/domain/analysisView/environmentManifestView.ts b/src/domain/analysisView/environmentManifestView.ts index 5204274f17..dd68558c4f 100644 --- a/src/domain/analysisView/environmentManifestView.ts +++ b/src/domain/analysisView/environmentManifestView.ts @@ -10,23 +10,28 @@ import type { UnsignedAnalysisView, } from "./analysisView.js"; import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js"; -import { TEMPLATE_RULES } from "./environmentManifestRules.js"; +import { + TEMPLATE_RULES, + type TemplateRule, +} from "./environmentManifestRules.js"; import { admitHarCaptureParent } from "./WebNetworkProjectionAccumulator.js"; import { admitScenarioCaptureParent } from "./scenarioCaptureRecords.js"; import { ROUTE_RULES } from "./webNetworkRouteTemplates.js"; +/** Opt-in template rules, each listed at most once; shared by every view that groups tools. */ +export const templateRulesSchema = z + .array(z.enum(TEMPLATE_RULES)) + .refine((rules) => new Set(rules).size === rules.length, { + message: "template_rules must not repeat a rule", + }) + .meta({ uniqueItems: true }) + .default([]); + export const environmentManifestViewRequestSchema = z.strictObject({ kind: z.literal("environment-manifest"), - template_rules: z - .array(z.enum(TEMPLATE_RULES)) - .refine((rules) => new Set(rules).size === rules.length, { - message: "template_rules must not repeat a rule", - }) - .meta({ uniqueItems: true }) - .default([]) - .describe( - "Opt-in template rules applied after the four format rules, each listed at most once: after-uid-literal templates the segment following a literal uid, opaque-token templates long mixed alphanumeric segments, sibling-variance merges groups that differ in one identifier-like segment, sibling-pair-variance merges groups that differ in two adjacent literals between a plural literal and another literal when at least three distinct pairs vary in both. Literal sibling folding into existing opt-in parameters also requires sibling-variance. Merged parameters are inferences.", - ), + template_rules: templateRulesSchema.describe( + "Opt-in template rules applied after the four format rules, each listed at most once: after-uid-literal templates the segment following a literal uid, opaque-token templates long mixed alphanumeric segments, sibling-variance merges groups that differ in one identifier-like segment, sibling-pair-variance merges groups that differ in two adjacent literals between a plural literal and another literal when at least three distinct pairs vary in both. Literal sibling folding into existing opt-in parameters also requires sibling-variance. Merged parameters are inferences.", + ), include_query_parameter_names: z .boolean() .default(true) @@ -122,7 +127,7 @@ export const environmentManifestFields = { }; /** The records, coverage and identity a manifest projects from, for either parent kind. */ -interface CaptureParent { +export interface CaptureParent { readonly operation: | "inspect_web_network_capture" | "capture_browser_scenario"; @@ -135,10 +140,11 @@ interface CaptureParent { /** HAR captures keep their own gate; a browser scenario has no captured file, so its artifact is empty. */ const admitCaptureParent = ( parent: AnalysisViewParent, + viewKind: "environment-manifest" | "environment-state", accumulator: EnvironmentManifestAccumulator, ): Result => { if (parent.operation === "capture_browser_scenario") { - const scenario = admitScenarioCaptureParent(parent, accumulator); + const scenario = admitScenarioCaptureParent(parent, viewKind, accumulator); return scenario.ok ? ok({ operation: "capture_browser_scenario", @@ -149,16 +155,64 @@ const admitCaptureParent = ( }) : scenario; } - const har = admitHarCaptureParent( - parent, - "environment-manifest", - accumulator, - ); + const har = admitHarCaptureParent(parent, viewKind, accumulator); return har.ok ? ok({ operation: "inspect_web_network_capture", ...har.value }) : har; }; +/** Admit the parent, the shared and view-specific limitations, then group every retained record. */ +export const groupCaptureRecords = ( + parent: AnalysisViewParent, + viewKind: "environment-manifest" | "environment-state", + rules: readonly TemplateRule[], + target: { + readonly accumulator: EnvironmentManifestAccumulator; + readonly noun: "manifest" | "state"; + readonly limitations: readonly string[]; + }, +): Result => { + const admitted = admitCaptureParent(parent, viewKind, target.accumulator); + if (!admitted.ok) return admitted; + const added = addCaptureRecords(parent, admitted.value, rules, target); + return added.ok ? admitted : added; +}; + +/** Admit the shared and view-specific limitations, then group every retained record. */ +const addCaptureRecords = ( + parent: AnalysisViewParent, + capture: CaptureParent, + rules: readonly TemplateRule[], + target: { + readonly accumulator: EnvironmentManifestAccumulator; + readonly noun: "manifest" | "state"; + readonly limitations: readonly string[]; + }, +): Result => { + const limitations = target.accumulator.addLimitations([ + ...parent.limitations, + ...capture.limitations, + "Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.", + ...target.limitations, + ...(rules.length > 0 + ? [ + `Opt-in template rules ${uniqueSorted(rules).join(", ")} merge recorded literals by shape; merged parameters may conflate distinct resources.`, + ] + : []), + ...(capture.records.length < capture.total_records + ? [ + `Parent contains an ordinal subset; the ${target.noun} covers only retained records.`, + ] + : []), + ]); + if (!limitations.ok) return limitations; + for (const record of capture.records) { + const added = target.accumulator.add(record); + if (!added.ok) return added; + } + return ok(undefined); +}; + /** Project retained request lines, from a HAR or a browser scenario, into a replayable tool surface; templates are inferences. */ export const projectEnvironmentManifestView = ( parent: AnalysisViewParent, @@ -179,34 +233,17 @@ export const projectEnvironmentManifestView = ( `View ${view.kind} is unsupported for parent operation ${parent.operation}; environment-manifest requires inspect_web_network_capture or capture_browser_scenario.`, ), ); - const admitted = admitCaptureParent(parent, accumulator); - if (!admitted.ok) return admitted; - const capture = admitted.value; - const limitations = accumulator.addLimitations([ - ...parent.limitations, - ...capture.limitations, - "Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.", - ...(view.include_query_parameter_names + const grouped = groupCaptureRecords(parent, view.kind, view.template_rules, { + accumulator, + noun: "manifest", + limitations: view.include_query_parameter_names ? [ "Observed query parameter names do not establish required parameters, value types or a complete API schema.", ] - : []), - ...(view.template_rules.length > 0 - ? [ - `Opt-in template rules ${uniqueSorted(view.template_rules).join(", ")} merge recorded literals by shape; merged parameters may conflate distinct resources.`, - ] - : []), - ...(capture.records.length < capture.total_records - ? [ - "Parent contains an ordinal subset; the manifest covers only retained records.", - ] - : []), - ]); - if (!limitations.ok) return limitations; - for (const record of capture.records) { - const added = accumulator.add(record); - if (!added.ok) return added; - } + : [], + }); + if (!grouped.ok) return grouped; + const capture = grouped.value; const manifest = accumulator.finish(); if (!manifest.ok) return manifest; return ok({ diff --git a/src/domain/analysisView/environmentStateGate.ts b/src/domain/analysisView/environmentStateGate.ts new file mode 100644 index 0000000000..4df98d3506 --- /dev/null +++ b/src/domain/analysisView/environmentStateGate.ts @@ -0,0 +1,320 @@ +import { canonicalJson } from "../comparisonSemantics.js"; +import type { JsonValue } from "../jsonValue.js"; +import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js"; +import { + at, + collectionPathOf, + isObject, + lastSegmentValue, + listItems, + renderStateEntity, + renderStateList, + replaceAt, + same, + scalarText, + type JsonObject, + type RecordedViewReason, + type StateEntity, + type StateRecordFacts, + type StateToolFacts, + type StateView, +} from "./environmentStateRules.js"; +import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js"; + +/** A read tool whose recordings locate entities of one type at one pointer. */ +export interface StatePlan { + readonly tool: StateToolFacts; + readonly item: boolean; + readonly type: string; + readonly pointer: string; + /** Successful init recordings in timeline order. */ + readonly recordings: readonly StateRecordFacts[]; + /** The latest of them per request line, in timeline order. */ + readonly latest: readonly StateRecordFacts[]; +} + +/** What the gate reads from an extraction; `withBody` drops each body's temporary work after `use`. */ +export interface StateGateContext { + readonly withBody: ( + ordinal: number, + use: (body: JsonValue | undefined) => T, + ) => T; + readonly retain: (bytes: number) => void; + readonly entity: ( + type: string, + collection: string, + id: string, + ) => StateEntity | undefined; + readonly inCollection: (type: string, collection: string) => StateEntity[]; + readonly volatileOf: (type: string) => ReadonlySet; + readonly idField: (type: string) => string; +} + +/** Ordered key union and always-present keys, grown one object at a time. */ +class FieldUnion { + readonly fields: string[] = []; + /** Membership beside the ordered list, so a wide object is not scanned per key. */ + readonly #seen = new Set(); + #required: Set | undefined; + + add(object: JsonObject): void { + for (const key of Object.keys(object)) + if (!this.#seen.has(key)) { + this.#seen.add(key); + this.fields.push(key); + } + const keys = new Set(Object.keys(object)); + this.#required = + this.#required === undefined + ? keys + : new Set([...this.#required].filter((key) => keys.has(key))); + } + + get required(): string[] { + return this.fields.filter((key) => this.#required?.has(key) === true); + } + + /** Retained bytes of the field lists a plan keeps. */ + get bytes(): number { + return this.fields.reduce( + (sum, name) => sum + ENTRY_BYTES * 2 + textBytes(name.length) * 2, + ENTRY_BYTES * 2, + ); + } +} + +const recorded = (plan: StatePlan, reason: RecordedViewReason): StateView => ({ + tool: plan.tool.name, + shape: "recorded", + reason, +}); + +/** Why one latest item recording does not render from the shared envelope and the entity's state, if it does not. */ +const itemMismatch = ( + context: StateGateContext, + plan: StatePlan, + check: { + readonly view: { readonly tool: string; readonly fields: string[] }; + readonly envelope: { skeleton?: string }; + }, + record: StateRecordFacts, +): RecordedViewReason | undefined => { + const { view, envelope } = check; + const entity = context.entity( + plan.type, + collectionPathOf(record, true), + lastSegmentValue(record.path), + ); + return context.withBody(record.ordinal, (body) => { + const object = body === undefined ? undefined : at(body, plan.pointer); + if (entity === undefined || body === undefined || !isObject(object)) + return "fields-differ"; + const skeleton = canonicalJson( + replaceAt(body, plan.pointer, null), + "Environment state", + ); + if (envelope.skeleton === undefined) { + context.retain(textBytes(skeleton.length)); + envelope.skeleton = skeleton; + } + if (skeleton !== envelope.skeleton) return "envelope-varies"; + const shown = renderStateEntity( + entity, + view, + context.volatileOf(plan.type), + object, + ); + return same(shown, object) ? undefined : "fields-differ"; + }); +}; + +/** An item plan must reproduce each latest recording from the first one's envelope and the entity's state. */ +export const checkItemPlan = ( + context: StateGateContext, + plan: StatePlan, +): StateView => { + const union = new FieldUnion(); + for (const record of plan.latest) + context.withBody(record.ordinal, (body) => { + const object = body === undefined ? undefined : at(body, plan.pointer); + if (isObject(object)) union.add(object); + }); + context.retain(union.bytes); + const view = { tool: plan.tool.name, fields: union.fields }; + const envelope: { skeleton?: string } = {}; + for (const record of plan.latest) { + const reason = itemMismatch(context, plan, { view, envelope }, record); + if (reason !== undefined) return recorded(plan, reason); + } + return { + tool: plan.tool.name, + shape: "item", + type: plan.type, + pointer: plan.pointer, + fields: union.fields, + required_fields: union.required, + envelope_ordinal: plan.latest[0]!.ordinal, + checked_record_ordinals: plan.latest.map((record) => record.ordinal), + }; +}; + +/** Whether every latest recording lists its whole collection in state order, and which counters equal the item count. */ +const listFacts = ( + context: StateGateContext, + plan: StatePlan, + union: FieldUnion, +): { readonly all: boolean; readonly counters: string[] } => { + const field = context.idField(plan.type); + let all = true; + let counters: string[] | undefined; + for (const record of plan.latest) + context.withBody(record.ordinal, (body) => { + const items = + body === undefined ? undefined : listItems(body, plan.pointer); + if (body === undefined || items === undefined) return; + items.forEach((item) => union.add(item)); + // Once a list cannot be the whole collection, its membership is not compared again. + if (all && record.query === undefined) { + const ids = items.map((item) => scalarText(item[field]) ?? ""); + const expected = context + .inCollection(plan.type, collectionPathOf(record, false)) + .map((entity) => entity.id); + if (!same(ids, expected)) all = false; + } else all = false; + const parent = plan.pointer === "" || !isObject(body) ? {} : body; + const counted = Object.keys(parent).filter( + (name) => isItemTotal(name) && countOf(parent[name]) !== undefined, + ); + // A total that differs from the number listed, or a next page, shows the list is partial. + if ( + counted.some((name) => countOf(parent[name]) !== items.length) || + Object.keys(parent).some((name) => continues(name, parent[name])) + ) + all = false; + const names = counted.filter((name) => parent[name] === items.length); + counters = (counters ?? names).filter((name) => names.includes(name)); + }); + return { all, counters: (counters ?? []).sort(compare) }; +}; + +const nameTokens = (name: string): string[] => + name + .split(/[_\-\s]|(?<=[a-z0-9])(?=[A-Z])/u) + .map((token) => token.toLowerCase()); + +/** Item totals, including compact `totalcount`/`itemcount`; page counts and account IDs are excluded. */ +const isItemTotal = (name: string): boolean => { + const tokens = nameTokens(name); + return ( + tokens.some( + (token) => + token === "total" || + token === "count" || + token === "totalcount" || + token === "itemcount", + ) && + !tokens.some( + (token) => token === "page" || token === "pages" || token === "per", + ) + ); +}; + +/** A count recorded as a number or as a string of digits. */ +const countOf = (value: JsonValue | undefined): number | undefined => + typeof value === "number" + ? value + : typeof value === "string" && /^\d+$/u.test(value) + ? Number(value) + : undefined; + +const CONTINUATION_NAMES = new Set([ + "hasmore", + "next", + "nextcursor", + "nextpage", + "nextpagetoken", + "nexttoken", +]); + +/** A top-level `has_more`, `next` or next-page token member that says more items follow. */ +const continues = (name: string, value: JsonValue | undefined): boolean => + CONTINUATION_NAMES.has(nameTokens(name).join("")) && + value !== undefined && + value !== null && + value !== false && + value !== ""; + +/** Whether one latest list recording renders from its own envelope and the listed entities' state. */ +const listReproduces = ( + context: StateGateContext, + plan: StatePlan, + view: { + readonly tool: string; + readonly fields: string[]; + readonly pointer: string; + readonly counters: string[]; + readonly all: boolean; + }, + record: StateRecordFacts, +): boolean => { + const field = context.idField(plan.type); + return context.withBody(record.ordinal, (body) => { + const items = + body === undefined ? undefined : listItems(body, plan.pointer); + if (body === undefined || items === undefined) return false; + const observed = new Map( + items.map((item) => [scalarText(item[field]) ?? "", item]), + ); + const members = view.all + ? context.inCollection(plan.type, collectionPathOf(record, false)) + : items.flatMap((item) => { + const entity = context.entity( + plan.type, + collectionPathOf(record, false), + scalarText(item[field]) ?? "", + ); + return entity === undefined ? [] : [entity]; + }); + const rendered = members.map((entity) => + renderStateEntity( + entity, + view, + context.volatileOf(plan.type), + observed.get(entity.id), + ), + ); + return same(renderStateList(view, body, rendered), body); + }); +}; + +/** A list plan must reproduce each latest recording from its own envelope and the listed entities' state. */ +export const checkListPlan = ( + context: StateGateContext, + plan: StatePlan, +): StateView => { + const union = new FieldUnion(); + const { all, counters } = listFacts(context, plan, union); + context.retain(union.bytes); + const view = { + tool: plan.tool.name, + fields: union.fields, + pointer: plan.pointer, + counters, + all, + }; + if ( + !plan.latest.every((record) => listReproduces(context, plan, view, record)) + ) + return recorded(plan, "list-differs"); + return { + tool: plan.tool.name, + shape: "list", + type: plan.type, + pointer: plan.pointer, + fields: union.fields, + required_fields: union.required, + membership: all ? "all" : "recorded", + counters, + checked_record_ordinals: plan.latest.map((record) => record.ordinal), + }; +}; diff --git a/src/domain/analysisView/environmentStateMerge.ts b/src/domain/analysisView/environmentStateMerge.ts new file mode 100644 index 0000000000..f0bb102dfb --- /dev/null +++ b/src/domain/analysisView/environmentStateMerge.ts @@ -0,0 +1,135 @@ +import type { JsonValue } from "../jsonValue.js"; +import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js"; +import { + same, + type JsonObject, + type StateEntity, +} from "./environmentStateRules.js"; + +/** One entity object as one tool recorded it. */ +export interface StateObservation { + readonly position: number; + readonly ordinal: number; + readonly tool: string; + readonly item: boolean; + readonly object: JsonObject; +} + +/** Every init observation of one entity, with where it was first seen. */ +export interface ObservedEntity { + readonly type: string; + readonly collection: string; + readonly id: string; + /** The earliest recorded sighting: timeline position, then position in its list. */ + order: readonly [number, number]; + readonly observations: StateObservation[]; +} + +/** Collection order: type, collection, then first sighting. */ +export const compareObservedEntities = ( + left: ObservedEntity, + right: ObservedEntity, +): number => + compare(left.type, right.type) || + compare(left.collection, right.collection) || + left.order[0] - right.order[0] || + left.order[1] - right.order[1]; + +/** One entity: item views' latest values win over list views', and a view whose latest values differ keeps them. */ +export const mergeObservedEntity = (entity: ObservedEntity): StateEntity => { + const latest = new Map(); + for (const observation of entity.observations) + latest.set(observation.tool, observation); + const fields = new Map(); + const ranked = [...latest.values()].sort( + (left, right) => + Number(left.item) - Number(right.item) || left.position - right.position, + ); + for (const observation of ranked) + for (const [name, value] of Object.entries(observation.object)) + fields.set(name, value); + return { + type: entity.type, + collection: entity.collection, + id: entity.id, + fields: Object.fromEntries(fields), + view_fields: [...latest.values()] + .map((observation) => ({ + tool: observation.tool, + fields: Object.fromEntries( + Object.entries(observation.object).filter( + ([name, value]) => !same(fields.get(name), value), + ), + ), + })) + .filter((view) => Object.keys(view.fields).length > 0) + .sort((left, right) => compare(left.tool, right.tool)), + record_ordinals: [ + ...new Set(entity.observations.map((observation) => observation.ordinal)), + ], + }; +}; + +/** + * How observed values changed, per entity in timeline order. A shared field that differs from the + * previous repeat of the same request line is volatile, unless a recorded write lies between them; + * a field that differs from its previous observation on another line across a write rewrites the + * type. Either across a write means the reads do not agree on one initial state. + */ +export const classifyEntityChanges = ( + entities: Iterable, + lineOf: (observation: StateObservation) => readonly string[], + writeBetween: (earlier: StateObservation, later: StateObservation) => boolean, +): { + readonly rewritten: ReadonlySet; + readonly volatile: ReadonlyMap>; +} => { + const rewritten = new Set(); + const volatile = new Map>(); + const changed = (before: StateObservation, after: StateObservation) => + Object.keys(after.object).filter( + (name) => + Object.hasOwn(before.object, name) && + !same(before.object[name], after.object[name]), + ); + for (const entity of entities) { + const timeline = [...entity.observations].sort( + (left, right) => left.position - right.position, + ); + const lastOfLine = new Map(); + // Sparse projections do not erase where each field was last observed. + const lastOfField = new Map< + string, + { readonly observation: StateObservation; readonly line: string } + >(); + timeline.forEach((after) => { + const line = JSON.stringify(lineOf(after)); + const repeat = lastOfLine.get(line); + if (repeat !== undefined) { + if (writeBetween(repeat, after)) { + // Key additions/removals are changes too, even when every shared field is unchanged. + if (!same(repeat.object, after.object)) rewritten.add(entity.type); + } else + for (const name of changed(repeat, after)) + volatile.set( + entity.type, + (volatile.get(entity.type) ?? new Set()).add(name), + ); + } + const sighting = { observation: after, line }; + for (const name of Object.keys(after.object)) { + const previous = lastOfField.get(name); + if ( + previous !== undefined && + previous.line !== line && + !same(previous.observation.object[name], after.object[name]) && + writeBetween(previous.observation, after) + ) + rewritten.add(entity.type); + lastOfField.set(name, sighting); + } + lastOfLine.set(line, after); + }); + } + return { rewritten, volatile }; +}; diff --git a/src/domain/analysisView/environmentStateRules.ts b/src/domain/analysisView/environmentStateRules.ts new file mode 100644 index 0000000000..ce7006be01 --- /dev/null +++ b/src/domain/analysisView/environmentStateRules.ts @@ -0,0 +1,437 @@ +import { canonicalJson } from "../comparisonSemantics.js"; +import type { JsonValue } from "../jsonValue.js"; +import { ENTRY_BYTES, textBytes } from "./webNetworkRouteTemplates.js"; + +/** How records are ordered: by recorded start time, or by record order when any start time is missing. */ +export type StateTimeline = "recorded-start-time" | "record-order"; + +/** What extraction reads of one primary-origin record; times are timeline coordinates. */ +export interface StateRecordFacts { + readonly ordinal: number; + readonly method: string; + readonly path: string; + /** WHATWG query spelling including `?`, or undefined when the request had none. */ + readonly query: string | undefined; + readonly status: number | null; + readonly start: number; + readonly end: number; +} + +/** The manifest tool facts extraction uses. */ +export interface StateToolFacts { + readonly name: string; + readonly method: string; + readonly path_template: string; + readonly record_ordinals: readonly number[]; +} + +/** A record's response body as JSON, or unusable when absent, cut, not JSON or not exactly representable. */ +export type ParsedBody = + | { readonly kind: "json"; readonly value: JsonValue } + | { readonly kind: "unusable" }; + +/** Budgeted access the accumulator lends extraction; null and false mean the budget is exhausted. */ +export interface StateBudget { + readonly body: (ordinal: number) => ParsedBody | null; + readonly retain: (bytes: number) => boolean; + /** Drop the temporary work of the body just read. */ + readonly release: () => void; +} + +export type RecordedViewReason = + | "query-body" + | "no-init-recording" + | "body-unusable" + | "not-entity" + | "no-identity" + | "query-varies" + | "fields-differ" + | "envelope-varies" + | "list-differs"; + +export interface StateType { + readonly collection_template: string; + readonly id_field: string; + readonly id_basis: "path-parameter" | "field-name"; + readonly init_before_ordinal: number | null; + readonly volatile_fields: string[]; +} + +export type JsonObject = Record; + +export interface StateEntity { + readonly type: string; + readonly collection: string; + readonly id: string; + readonly fields: JsonObject; + readonly view_fields: { + readonly tool: string; + readonly fields: JsonObject; + }[]; + readonly record_ordinals: number[]; +} + +export interface StateDocument { + readonly tool: string; + readonly path: string; + readonly value: JsonValue; + readonly record_ordinal: number; +} + +interface EntityViewPlan { + readonly tool: string; + readonly type: string; + readonly pointer: string; + readonly fields: string[]; + readonly required_fields: string[]; + readonly checked_record_ordinals: number[]; +} + +export type StateItemView = EntityViewPlan & { + readonly shape: "item"; + readonly envelope_ordinal: number; +}; + +export type StateListView = EntityViewPlan & { + readonly shape: "list"; + readonly membership: "all" | "recorded"; + readonly counters: string[]; +}; + +export type StateView = + | StateItemView + | StateListView + | { + readonly tool: string; + readonly shape: "document"; + readonly checked_record_ordinals: number[]; + } + | { + readonly tool: string; + readonly shape: "recorded"; + readonly reason: RecordedViewReason; + }; + +export interface ExtractedState { + readonly types: StateType[]; + readonly entities: StateEntity[]; + readonly documents: StateDocument[]; + readonly views: StateView[]; +} + +const WRITE_METHODS = new Set(["POST", "PUT", "PATCH", "DELETE"]); +/** A POST whose path names one of these segments queries state rather than changing it. */ +const QUERY_SEGMENTS = new Set([ + "search", + "query", + "list", + "find", + "fuzzy", + "lookup", + "count", + "aggregate", + "stats", + "_search", + "_count", + "_msearch", +]); +/** Identity field names tried, in order, when no path parameter proves one. */ +export const IDENTITY_NAMES = [ + "id", + "uid", + "uuid", + "_id", + "ID", + "UID", + "Id", + "guid", + "key", + "slug", + "name", +]; +const VERSION_SEGMENT = /^v\d+(?:\.\d+)*$/u; + +export const segmentsOf = (path: string): string[] => + path.split("/").filter((segment) => segment.length > 0); +export const isParam = (segment: string): boolean => + segment.startsWith("{") && segment.endsWith("}"); +export const isObject = (value: JsonValue | undefined): value is JsonObject => + typeof value === "object" && value !== null && !Array.isArray(value); +export const scalarText = (value: JsonValue | undefined): string | undefined => + typeof value === "string" + ? value + : typeof value === "number" + ? String(value) + : undefined; +/** Equal JSON values; object member order carries no meaning. */ +export const same = ( + left: JsonValue | undefined, + right: JsonValue | undefined, +): boolean => + left === undefined || right === undefined + ? left === right + : canonicalJson(left, "Environment state") === + canonicalJson(right, "Environment state"); + +/** The decoded final path segment, which an item request names its entity by. */ +export const lastSegmentValue = (path: string): string => { + const segment = segmentsOf(path).at(-1) ?? ""; + try { + return decodeURIComponent(segment); + } catch { + return segment; + } +}; + +/** A path's non-empty segments, so a trailing slash names the same collection. */ +export const normalizedPath = (path: string): string => + `/${segmentsOf(path).join("/")}`; + +/** An item record's collection is its path without the identity segment; a list record's is its path. */ +export const collectionPathOf = ( + record: StateRecordFacts, + item: boolean, +): string => + item + ? `/${segmentsOf(record.path).slice(0, -1).join("/")}` + : normalizedPath(record.path); + +/** A POST whose path names a query segment observes state rather than changing it. */ +export const isQueryPost = (method: string, path: string): boolean => + method === "POST" && + segmentsOf(path).some((segment) => QUERY_SEGMENTS.has(segment.toLowerCase())); + +/** Whether a record may change state: a successful or unanswered write that is not a query POST. */ +export const isStateWrite = (record: StateRecordFacts): boolean => + WRITE_METHODS.has(record.method) && + !isQueryPost(record.method, record.path) && + (record.status === null || (record.status >= 200 && record.status < 400)); + +/** Whether any write overlaps the span from one record's start to a later record's end; writes come in timeline order. */ +export const writeSpanCheck = ( + writes: readonly StateRecordFacts[], +): ((earlier: StateRecordFacts, later: StateRecordFacts) => boolean) => { + // The latest end among each prefix of writes, so one search answers a span. + const reach: number[] = []; + for (const write of writes) + reach.push(Math.max(reach.at(-1) ?? Number.NEGATIVE_INFINITY, write.end)); + return (earlier, later) => { + let low = 0; + let high = writes.length; + while (low < high) { + const middle = (low + high) >>> 1; + if (writes[middle]!.start < later.end) low = middle + 1; + else high = middle; + } + return low > 0 && reach[low - 1]! > earlier.start; + }; +}; + +/** Literal leading segments shared by every template with the same first segment: each path's API base. */ +export const apiBaseLengths = ( + templates: readonly string[], +): ReadonlyMap => { + const groups = new Map(); + for (const template of templates) { + const segments = segmentsOf(template); + const first = segments[0] ?? ""; + const group = groups.get(first); + if (group === undefined) groups.set(first, [segments]); + else group.push(segments); + } + return new Map( + [...groups.entries()].map(([first, members]) => { + const head = members[0] ?? []; + let length = 0; + while ( + length < head.length && + !isParam(head[length]!) && + members.every((segments) => segments[length] === head[length]) + ) + length += 1; + return [first, length]; + }), + ); +}; + +/** The path's API base, then any version segments, then one segment: the scope a write may affect. */ +export const rootOf = ( + path: string, + bases: ReadonlyMap, +): string[] => { + const segments = segmentsOf(path); + let end = Math.min(bases.get(segments[0] ?? "") ?? 0, segments.length); + while (end < segments.length && VERSION_SEGMENT.test(segments[end]!)) + end += 1; + return segments.slice(0, end + 1); +}; + +/** Roots relate when they agree on their common length; template parameters match any segment. */ +export const rootsRelated = ( + left: readonly string[], + right: readonly string[], +): boolean => + left + .slice(0, right.length) + .every( + (segment, index) => + segment === right[index] || isParam(segment) || isParam(right[index]!), + ); + +/** Value bytes as retained state, charged before a value is kept. */ +export const jsonBytes = (value: JsonValue): number => { + if (typeof value === "string") return textBytes(value.length); + if (Array.isArray(value)) + return value.reduce( + (sum, item) => sum + jsonBytes(item), + ENTRY_BYTES, + ); + if (isObject(value)) + return Object.entries(value).reduce( + (sum, [name, item]) => sum + textBytes(name.length) + jsonBytes(item), + ENTRY_BYTES, + ); + return ENTRY_BYTES; +}; + +/** JSON parsers keep a `__proto__` member that plain object construction would turn into a prototype. */ +export const hasPrototypeKey = (value: JsonValue): boolean => + Array.isArray(value) + ? value.some(hasPrototypeKey) + : isObject(value) && + Object.entries(value).some( + ([name, item]) => name === "__proto__" || hasPrototypeKey(item), + ); + +export const pointerTo = (name: string): string => + `/${name.replaceAll("~", "~0").replaceAll("/", "~1")}`; +const pointerName = (pointer: string): string => + pointer.slice(1).replaceAll("~1", "/").replaceAll("~0", "~"); + +/** The value at a pointer of depth at most one. */ +export const at = (body: JsonValue, pointer: string): JsonValue | undefined => { + if (pointer === "") return body; + if (!isObject(body)) return undefined; + const name = pointerName(pointer); + return Object.hasOwn(body, name) ? body[name] : undefined; +}; + +/** A copy of `body` with the value at a pointer of depth at most one replaced. */ +export const replaceAt = ( + body: JsonValue, + pointer: string, + value: JsonValue, +): JsonValue => { + if (pointer === "" || !isObject(body)) return value; + const name = pointerName(pointer); + return Object.fromEntries( + Object.entries(body).map(([key, item]) => [ + key, + key === name ? value : item, + ]), + ); +}; + +const matchingKeys = (object: JsonObject, value: string): string[] => + Object.keys(object).filter((key) => scalarText(object[key]) === value); + +/** The entity object an item response carries, at the root or in one wrapper property, with the keys repeating its path value. */ +export const locateItem = ( + body: JsonValue, + value: string, +): { readonly pointer: string; readonly keys: string[] } | undefined => { + if (!isObject(body)) return undefined; + const root = matchingKeys(body, value); + if (root.length > 0) return { pointer: "", keys: root }; + const wrapped = Object.entries(body).flatMap(([name, item]) => { + if (!isObject(item)) return []; + const keys = matchingKeys(item, value); + return keys.length > 0 ? [{ pointer: pointerTo(name), keys }] : []; + }); + return wrapped.length === 1 ? wrapped[0] : undefined; +}; + +/** Whether every item carries `name` as a distinct string or number. */ +export const identifies = ( + items: readonly JsonObject[], + name: string, +): boolean => { + const values = items.map((item) => scalarText(item[name])); + return ( + values.every((value) => value !== undefined) && + new Set(values).size === values.length + ); +}; + +/** The array of entity objects a list response carries, at the root or in one property of an envelope that is not itself an entity. */ +export const locateList = ( + body: JsonValue, + idField: string | undefined, +): string | undefined => { + if (Array.isArray(body)) return body.every(isObject) ? "" : undefined; + if ( + !isObject(body) || + IDENTITY_NAMES.some((name) => scalarText(body[name]) !== undefined) + ) + return undefined; + const arrays = Object.entries(body).filter( + (entry): entry is [string, JsonObject[]] => + Array.isArray(entry[1]) && entry[1].every(isObject), + ); + const carrying = arrays.filter( + ([, items]) => + items.length > 0 && + (idField === undefined + ? IDENTITY_NAMES.some((name) => identifies(items, name)) + : identifies(items, idField)), + ); + const chosen = + carrying.length === 1 + ? carrying[0] + : carrying.length === 0 && arrays.length === 1 + ? arrays[0] + : undefined; + return chosen === undefined ? undefined : pointerTo(chosen[0]); +}; + +export const listItems = ( + body: JsonValue, + pointer: string, +): JsonObject[] | undefined => { + const items = at(body, pointer); + return Array.isArray(items) && items.every(isObject) ? items : undefined; +}; + +/** One entity as a view shows it: the view's fields it has, view-specific values first; volatile fields take `observed` values when given. */ +export const renderStateEntity = ( + entity: StateEntity, + view: Pick, + volatileFields: ReadonlySet, + observed?: JsonObject, +): JsonObject => { + const own = + entity.view_fields.find((candidate) => candidate.tool === view.tool) + ?.fields ?? {}; + return Object.fromEntries( + view.fields.flatMap((name): [string, JsonValue][] => { + const source = + observed !== undefined && volatileFields.has(name) + ? observed + : Object.hasOwn(own, name) + ? own + : entity.fields; + return Object.hasOwn(source, name) ? [[name, source[name]!]] : []; + }), + ); +}; + +/** A list response rendered from `base`: its array replaced by `items` and its counters set to their number. */ +export const renderStateList = ( + view: Pick, + base: JsonValue, + items: readonly JsonObject[], +): JsonValue => + view.counters.reduce( + (body, name) => replaceAt(body, pointerTo(name), items.length), + replaceAt(base, view.pointer, [...items]), + ); diff --git a/src/domain/analysisView/environmentStateView.test.ts b/src/domain/analysisView/environmentStateView.test.ts new file mode 100644 index 0000000000..7a0f5cdec2 --- /dev/null +++ b/src/domain/analysisView/environmentStateView.test.ts @@ -0,0 +1,715 @@ +import { expect, it } from "vitest"; +import { + analysisViewCaptureEvidence, + environmentManifestCaptureFixture, + environmentStateCaptureFixture, +} from "../../../tests/fixtures/analysisView.js"; +import { + SCENARIO_EXCHANGES, + scenarioCaptureEvidence, + scenarioCaptureFixture, +} from "../../../tests/fixtures/browserScenarioCapture.js"; +import type { Evidence } from "../evidence.js"; +import type { JsonValue } from "../jsonValue.js"; +import { + projectAnalysisView, + type AnalysisViewParent, + type AnalysisViewResult, +} from "./analysisView.js"; +import { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js"; + +type State = Extract; +const ALL_RULES = [ + "after-uid-literal", + "opaque-token", + "sibling-variance", +] as const; + +const parentOf = ( + evidence: Evidence = analysisViewCaptureEvidence( + environmentStateCaptureFixture(), + ), +): AnalysisViewParent => ({ + evidenceId: evidence.evidence_id, + operation: evidence.operation, + normalizedResult: evidence.normalized_result, + subject: evidence.subject, + limitations: [], + parameters: evidence.parameters, +}); + +const stateOf = (parent: AnalysisViewParent = parentOf()): State => { + const result = projectAnalysisView(parent, { + kind: "environment-state", + template_rules: [...ALL_RULES], + }); + if (!result.ok || result.value.kind !== "environment-state") + throw new Error("Expected an environment state"); + return result.value; +}; + +type Row = readonly [ + method: string, + path: string, + body?: JsonValue, + status?: number, +]; + +/** HAR entries one second apart on one origin; a string body is recorded as written. */ +const captureOf = (rows: readonly Row[]): AnalysisViewParent => + parentOf( + analysisViewCaptureEvidence( + environmentManifestCaptureFixture( + rows.map(([method, path, body = null, status = 200], index) => ({ + startedDateTime: new Date( + Date.parse("2026-10-17T18:00:00.000Z") + index * 1000, + ).toISOString(), + time: 5, + request: { method, url: `http://app.example${path}` }, + response: { + status, + content: { + mimeType: "application/json", + text: typeof body === "string" ? body : JSON.stringify(body), + }, + }, + })), + ), + ), + ); + +const viewOf = (state: State, tool: string) => + state.views.find((view) => view.tool === tool); + +it("extracts the state the Grafana-shaped capture's reads observed", () => { + const state = stateOf(); + expect(state.timeline).toBe("recorded-start-time"); + expect(state.types).toContainEqual({ + collection_template: "/api/folders", + id_field: "uid", + id_basis: "path-parameter", + init_before_ordinal: 20, + volatile_fields: [], + }); + expect(state.types).toContainEqual({ + collection_template: "/api/search", + id_field: "id", + id_basis: "field-name", + init_before_ordinal: null, + volatile_fields: [], + }); + expect( + state.entities.filter((entity) => entity.type === "/api/folders"), + ).toEqual([ + { + type: "/api/folders", + collection: "/api/folders", + id: "fg1jo91e57e2ob", + fields: { id: 5, uid: "fg1jo91e57e2ob", title: "probe folder" }, + view_fields: [], + record_ordinals: [12, 14], + }, + { + type: "/api/folders", + collection: "/api/folders", + id: "general", + fields: { id: 0, uid: "general", title: "Dashboards" }, + view_fields: [], + record_ordinals: [13], + }, + ]); + expect(viewOf(state, "get_folders")).toMatchObject({ + shape: "list", + pointer: "", + membership: "recorded", + }); + expect(viewOf(state, "get_folders_by_uid")).toMatchObject({ + shape: "item", + envelope_ordinal: 12, + checked_record_ordinals: [12, 13], + }); + expect(viewOf(state, "get_dashboards_by_uid")).toMatchObject({ + shape: "item", + pointer: "/dashboard", + }); + expect(viewOf(state, "get_annotations")).toEqual({ + tool: "get_annotations", + shape: "recorded", + reason: "no-init-recording", + }); + expect(state.documents).toEqual([]); + expect(state.unknowns.map((unknown) => unknown.kind)).toEqual([ + "authorization-model", + "response-semantics", + "state-transition", + ]); + expect( + state.limitations.filter((limitation) => + limitation.startsWith("Body shapes"), + ), + ).toEqual([]); +}); + +it("ends a type's state at the first write under its root, not at unrelated writes", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }], + ["PUT", "/api/notes/9", { id: 9 }], + ["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }], + ["PUT", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }], + ["GET", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }], + ]), + ); + expect(state.types).toEqual([ + { + collection_template: "/api/items", + id_field: "id", + id_basis: "path-parameter", + init_before_ordinal: 3, + volatile_fields: [], + }, + ]); + expect(state.entities).toMatchObject([ + { id: "1", fields: { title: "Granite Ledger" }, record_ordinals: [0, 2] }, + ]); + expect(state.views).toContainEqual( + expect.objectContaining({ shape: "item", checked_record_ordinals: [2] }), + ); +}); + +it("keeps the latest value of a field that changed with no related write and names it volatile", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/items/1", { id: 1, title: "Lighthouse Rota", seen: 10 }], + ["GET", "/api/items/1", { id: 1, title: "Lighthouse Rota", seen: 11 }], + ["GET", "/api/items/2", { id: 2, title: "Blue Heron", seen: 3 }], + ]), + ); + expect(state.types[0]).toMatchObject({ volatile_fields: ["seen"] }); + expect(state.entities[0]).toMatchObject({ + id: "1", + fields: { seen: 11 }, + }); + expect(state.views).toContainEqual( + expect.objectContaining({ + shape: "item", + checked_record_ordinals: [1, 2], + }), + ); +}); + +it("leaves a type recorded when a value changed across a write under another root", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/search", [{ id: 1, title: "Granite Ledger" }]], + ["POST", "/api/dashboards/db", { id: 1, title: "Saltmarsh Depot" }], + ["GET", "/api/search", [{ id: 1, title: "Saltmarsh Depot" }]], + ]), + ); + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); + expect(viewOf(state, "get_search")).toEqual({ + tool: "get_search", + shape: "recorded", + reason: "fields-differ", + }); +}); + +it("leaves a type recorded when a value changed across a write already in flight", () => { + const exchange = ([method, path, seconds, time, title]: readonly [ + string, + string, + number, + number, + string, + ]) => ({ + startedDateTime: new Date( + Date.parse("2026-10-17T18:00:00.000Z") + seconds * 1000, + ).toISOString(), + time, + request: { method, url: `http://app.example${path}` }, + response: { + status: 200, + content: { + mimeType: "application/json", + text: JSON.stringify([{ id: 1, title }]), + }, + }, + }); + const state = stateOf( + parentOf( + analysisViewCaptureEvidence( + environmentManifestCaptureFixture([ + exchange([ + "POST", + "/api/dashboards/db", + 0, + 30_000, + "Saltmarsh Depot", + ]), + exchange(["GET", "/api/search", 10, 5, "Granite Ledger"]), + exchange(["GET", "/api/search", 40, 5, "Saltmarsh Depot"]), + ]), + ), + ), + ); + expect(viewOf(state, "get_search")).toEqual({ + tool: "get_search", + shape: "recorded", + reason: "fields-differ", + }); +}); + +it("does not list the whole collection when a counter exceeds the number listed", () => { + const state = stateOf( + captureOf([ + [ + "GET", + "/api/items", + { total: 50, items: [{ id: 1, title: "Ada Finch" }] }, + ], + ]), + ); + expect(viewOf(state, "get_items")).toMatchObject({ + shape: "list", + membership: "recorded", + counters: [], + }); +}); + +it("lists the whole collection with its counter only when every recording did", () => { + const rows: Row[] = [ + [ + "GET", + "/api/items", + { + total: 2, + items: [ + { id: 1, title: "Ada Finch" }, + { id: 2, title: "Tomas Vey" }, + ], + }, + ], + ["GET", "/api/items/1", { title: "Ada Finch", id: 1, notes: "first" }], + ["GET", "/api/items/2", { id: 2, notes: "second", title: "Tomas Vey" }], + ]; + const whole = stateOf(captureOf(rows)); + expect(whole.views).toContainEqual( + expect.objectContaining({ + shape: "list", + pointer: "/items", + fields: ["id", "title"], + membership: "all", + counters: ["total"], + }), + ); + expect(whole.entities.map((entity) => entity.fields)).toEqual([ + { id: 1, title: "Ada Finch", notes: "first" }, + { id: 2, notes: "second", title: "Tomas Vey" }, + ]); + const filtered = stateOf( + captureOf([ + ...rows, + [ + "GET", + "/api/items?title=Ada", + { total: 1, items: [{ id: 1, title: "Ada Finch" }] }, + ], + ]), + ); + expect(filtered.views).toContainEqual( + expect.objectContaining({ shape: "list", membership: "recorded" }), + ); +}); + +it("leaves recorded the views it cannot reproduce, with the reason", () => { + const state = stateOf( + captureOf([ + [ + "GET", + "/api/boards/1", + { meta: { viewer: "Ada Finch" }, board: { id: 1, name: "Rota" } }, + ], + [ + "GET", + "/api/boards/2", + { meta: { viewer: "Mina Okafor" }, board: { id: 2, name: "Depot" } }, + ], + ["GET", "/api/ledgers/1", '{"id":1,"total":12345678901234567890}'], + ["POST", "/api/boards/search", [{ id: 1, name: "Rota" }]], + ]), + ); + expect( + state.views.map((view) => + view.shape === "recorded" ? [view.tool, view.reason] : view.tool, + ), + ).toEqual([ + ["post_boards_search", "query-body"], + ["get_boards_by_id", "envelope-varies"], + ["get_ledgers_by_id", "body-unusable"], + ]); + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); +}); + +it("holds a written path's non-entity read as a document", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/settings", { theme: "dark", density: "compact" }], + ["PUT", "/api/settings", { theme: "light", density: "compact" }], + ["GET", "/api/settings", { theme: "light", density: "compact" }], + ]), + ); + expect(state.documents).toEqual([ + { + tool: "get_settings", + path: "/api/settings", + value: { theme: "dark", density: "compact" }, + record_ordinal: 0, + }, + ]); + expect(viewOf(state, "get_settings")).toEqual({ + tool: "get_settings", + shape: "document", + checked_record_ordinals: [0], + }); +}); + +it.each([ + ["/api/files/123", "PUT", "get_files_by_id"], + ["/api/settings", "DELETE", "get_settings"], +] as const)( + "leaves a deferred document path recorded (%s, %s)", + (path, method, tool) => { + const state = stateOf( + captureOf([ + ["GET", path, { text: "initial document" }], + [method, path, null], + ]), + ); + expect(state.documents).toEqual([]); + expect(viewOf(state, tool)).toMatchObject({ shape: "recorded" }); + }, +); + +it("orders a scenario capture by request order and needs its response bodies", () => { + const state = stateOf(parentOf(scenarioCaptureEvidence())); + expect(state.timeline).toBe("record-order"); + expect(state.types).toEqual([ + { + collection_template: "/api/users", + id_field: "id", + id_basis: "path-parameter", + init_before_ordinal: null, + volatile_fields: [], + }, + ]); + expect(state.entities.map((entity) => entity.fields)).toEqual([ + { id: 2, login: "probe" }, + { id: 3, login: "viewer" }, + ]); + const headersOnly = scenarioCaptureEvidence( + scenarioCaptureFixture(SCENARIO_EXCHANGES, { + request_body: true, + response_body: false, + header_values: true, + }), + ); + expect( + projectAnalysisView(parentOf(headersOnly), { + kind: "environment-state", + template_rules: [], + }), + ).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisUnsupportedTargetError", + message: expect.stringContaining("response_body selected"), + }, + }); +}); + +it("refuses when extraction would exceed the projection budget", () => { + expect( + projectAnalysisView( + parentOf(), + { kind: "environment-state", template_rules: [...ALL_RULES] }, + { state: new EnvironmentStateAccumulator(160 * 1024, [...ALL_RULES]) }, + ), + ).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisResourceConstraintError", + resource: "memory", + message: expect.stringContaining("Environment-state"), + }, + }); +}); + +it("ends every type's state at a write whose URL was redacted, and says so", () => { + const parent = captureOf([ + ["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }], + ["PUT", "/api/items/1", { id: 1 }], + ["GET", "/api/items/1", { id: 1, title: "Saltmarsh Depot" }], + ]); + const result = structuredClone(parent.normalizedResult) as { + records: { reported: { request: { url: string | null } } }[]; + }; + // A scenario keeps a redacted URL as null, so the write's target is unknown. + result.records[1]!.reported.request.url = null; + const state = stateOf({ ...parent, normalizedResult: result }); + expect(state.types).toMatchObject([{ init_before_ordinal: 1 }]); + expect(state.entities).toMatchObject([ + { fields: { title: "Granite Ledger" }, record_ordinals: [0] }, + ]); + expect(state.limitations).toContain( + "1 of the recorded writes had a redacted URL; each ends the initial state of every type, since its target is unknown.", + ); +}); + +it("infers no identity for a list whose recordings held no items", () => { + const state = stateOf(captureOf([["GET", "/api/widgets", []]])); + expect(state.types).toEqual([]); + expect(state.views).toEqual([ + { tool: "get_widgets", shape: "recorded", reason: "no-identity" }, + ]); +}); + +it("does not call a value volatile when it differs between request lines", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/items/1?lang=fr", { id: 1, name: "Bonjour" }], + ["GET", "/api/items/1", { id: 1, name: "Hello" }], + ]), + ); + expect(state.types).toEqual([]); + expect( + state.views.map((view) => + view.shape === "recorded" ? [view.tool, view.reason] : view.tool, + ), + ).toEqual([["get_items_by_id", "fields-differ"]]); +}); + +it("orders entities by their earliest sighting, so a complete list keeps all membership", () => { + const state = stateOf( + captureOf([ + [ + "GET", + "/api/items", + [ + { id: 1, title: "Granite Ledger" }, + { id: 2, title: "Blue Heron" }, + ], + ], + ["GET", "/api/items/2", { id: 2, title: "Blue Heron" }], + ["GET", "/api/items/1", { id: 1, title: "Granite Ledger" }], + ]), + ); + expect(state.entities.map((entity) => entity.id)).toEqual(["1", "2"]); + expect(state.views).toContainEqual( + expect.objectContaining({ tool: "get_items", membership: "all" }), + ); +}); + +it("does not take a later recording that is itself an entity as the list", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/items", { items: [{ id: 1, title: "Granite Ledger" }] }], + [ + "GET", + "/api/items", + { id: 99, items: [{ id: 2, title: "Blue Heron" }] }, + ], + ]), + ); + expect(state.views).toContainEqual( + expect.objectContaining({ tool: "get_items", shape: "recorded" }), + ); + expect(state.entities).toEqual([]); +}); + +it("leaves a type recorded when two of its tools read different values across a write", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/search", [{ id: 1, title: "Granite Ledger" }]], + ["POST", "/api/dashboards/db", { id: 9 }], + ["GET", "/api/search/1", { id: 1, title: "Saltmarsh Depot" }], + ]), + ); + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); + expect( + state.views.map((view) => + view.shape === "recorded" ? [view.tool, view.reason] : view.tool, + ), + ).toEqual([ + ["get_search", "fields-differ"], + ["get_search_by_id", "fields-differ"], + ]); +}); + +it.each(["old", "new"])( + "retains field observations through sparse reads across a write (%s)", + (title) => { + const state = stateOf( + captureOf([ + ["GET", "/api/search", [{ id: 1, title: "old" }]], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search?fields=id", [{ id: 1 }]], + ["GET", "/api/search/1", { id: 1, title }], + ]), + ); + if (title === "new") { + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); + expect( + state.views.every( + (view) => + view.shape === "recorded" && view.reason === "fields-differ", + ), + ).toBe(true); + } else { + expect(viewOf(state, "get_search_by_id")).toMatchObject({ + shape: "item", + }); + expect(state.entities[0]?.fields.title).toBe("old"); + } + }, +); + +it("proves the identity field against every item response", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/widgets/1", { left: 1, right: 1 }], + ["GET", "/api/widgets/2", { right: 2, left: "other" }], + ]), + ); + expect(state.types).toMatchObject([ + { id_field: "right", id_basis: "path-parameter" }, + ]); + expect(state.entities.map((entity) => entity.id)).toEqual(["1", "2"]); +}); + +it.each(["totalcount", "itemcount"])( + "recognizes a compact compound item total (%s)", + (name) => { + const state = stateOf( + captureOf([["GET", "/api/items", { [name]: 50, items: [{ id: 1 }] }]]), + ); + expect(viewOf(state, "get_items")).toMatchObject({ + shape: "list", + membership: "recorded", + counters: [], + }); + }, +); + +it.each([ + [[], [{ id: 1 }]], + [[{ id: 1 }], []], + [[{ id: 1 }], [{ id: 2 }]], +] as const)( + "keeps changed list membership across another root's write recorded (%j -> %j)", + (before, after) => { + const state = stateOf( + captureOf([ + ["GET", "/api/search", [...before]], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search", [...after]], + ]), + ); + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); + expect(viewOf(state, "get_search")).toEqual({ + tool: "get_search", + shape: "recorded", + reason: "fields-differ", + }); + }, +); + +it("accepts stable membership across another root's write and changes confined before it", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/search", []], + ["GET", "/api/search", [{ id: 1 }]], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search", [{ id: 1 }]], + ]), + ); + expect(viewOf(state, "get_search")).toMatchObject({ shape: "list" }); + expect(state.entities.map((entity) => entity.id)).toEqual(["1"]); +}); + +it("keeps membership checks specific to the full request line", () => { + const state = stateOf( + captureOf([ + ["GET", "/api/search?page=1", [{ id: 1 }]], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search?page=2", [{ id: 2 }]], + ]), + ); + expect(viewOf(state, "get_search")).toMatchObject({ + shape: "list", + membership: "recorded", + }); +}); + +it("does not treat a first observation as a membership change", () => { + const parent = captureOf([ + ["GET", "/api/search", [{ id: 1 }]], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search", [{ id: 1 }]], + ]); + const data = structuredClone(parent.normalizedResult) as { + records: { reported: { time: number } }[]; + }; + data.records[0]!.reported.time = 1500; + const state = stateOf({ ...parent, normalizedResult: data }); + expect(viewOf(state, "get_search")).toMatchObject({ shape: "list" }); +}); + +it.each([ + [{ id: 1 }, { id: 1, title: "Created" }], + [{ id: 1, title: "Original" }, { id: 1 }], +])( + "keeps fields added or removed across another root's write recorded (%j -> %j)", + (before, after) => { + const state = stateOf( + captureOf([ + ["GET", "/api/search/1", before], + ["POST", "/api/items", { id: 2 }], + ["GET", "/api/search/1", after], + ]), + ); + expect(state.types).toEqual([]); + expect(state.entities).toEqual([]); + expect(viewOf(state, "get_search_by_id")).toMatchObject({ + shape: "recorded", + reason: "fields-differ", + }); + }, +); + +it("reads only item totals and next-page markers as signs of a partial list", () => { + const membership = (body: JsonValue) => + stateOf(captureOf([["GET", "/api/items", body]])).views.map((view) => + view.shape === "list" ? [view.membership, view.counters] : view.shape, + ); + // Page and id numbers are not item totals, so a complete page stays whole. + expect( + membership({ + page: 1, + total: 2, + total_pages: 1, + account_id: 77, + items: [{ id: 1 }, { id: 2 }], + }), + ).toEqual([["all", ["total"]]]); + expect(membership({ has_more: true, items: [{ id: 1 }, { id: 2 }] })).toEqual( + [["recorded", []]], + ); + expect(membership({ total: "50", items: [{ id: 1 }] })).toEqual([ + ["recorded", []], + ]); +}); diff --git a/src/domain/analysisView/environmentStateView.ts b/src/domain/analysisView/environmentStateView.ts new file mode 100644 index 0000000000..a43a7e05c5 --- /dev/null +++ b/src/domain/analysisView/environmentStateView.ts @@ -0,0 +1,188 @@ +import { z } from "zod"; +import type { AnalysisError } from "../analysisErrorBase.js"; +import { jsonValueSchema } from "../jsonValue.js"; +import { ok, type Result } from "../result.js"; +import type { + AnalysisViewParent, + AnalysisViewRequest, + UnsignedAnalysisView, +} from "./analysisView.js"; +import { EnvironmentStateAccumulator } from "./EnvironmentStateAccumulator.js"; +import { + groupCaptureRecords, + templateRulesSchema, +} from "./environmentManifestView.js"; + +export const environmentStateViewRequestSchema = z.strictObject({ + kind: z.literal("environment-state"), + template_rules: templateRulesSchema.describe( + "Opt-in template rules as for environment-manifest, each listed at most once. Use the rules of the manifest the state accompanies so tool names match.", + ), +}); + +const jsonObjectFieldsSchema = z.record(z.string(), jsonValueSchema); +const checkedSchema = z + .array(z.number().int().nonnegative()) + .describe( + "Latest init record per request line that the view reproduced from the state.", + ); +const entityViewFields = { + tool: z.string(), + type: z.string(), + pointer: z + .string() + .describe( + "JSON pointer, of depth at most one, to the entity object or array within the response.", + ), + fields: z + .array(z.string()) + .describe("Ordered fields the view shows of each entity."), + required_fields: z.array(z.string()), + checked_record_ordinals: checkedSchema, +}; +const viewSchema = z.discriminatedUnion("shape", [ + z.strictObject({ + ...entityViewFields, + shape: z.literal("item"), + envelope_ordinal: z + .number() + .int() + .nonnegative() + .describe( + "Record whose response, with the entity object replaced, every item response shares.", + ), + }), + z.strictObject({ + ...entityViewFields, + shape: z.literal("list"), + membership: z + .enum(["all", "recorded"]) + .describe( + "all: every entity of the collection in state order; recorded: the entities each recorded response listed.", + ), + counters: z + .array(z.string()) + .describe("Envelope members that count the listed entities."), + }), + z.strictObject({ + tool: z.string(), + shape: z.literal("document"), + checked_record_ordinals: checkedSchema, + }), + z.strictObject({ + tool: z.string(), + shape: z.literal("recorded"), + reason: z.enum([ + "query-body", + "no-init-recording", + "body-unusable", + "not-entity", + "no-identity", + "query-varies", + "fields-differ", + "envelope-varies", + "list-differs", + ]), + }), +]); + +export const environmentStateFields = { + origin: z + .string() + .describe("Primary origin, as the environment manifest selects it."), + name: z.string(), + timeline: z + .enum(["recorded-start-time", "record-order"]) + .describe( + "Records are ordered by recorded start time, or by record order when any start time is missing.", + ), + types: z.array( + z.strictObject({ + collection_template: z.string(), + id_field: z.string(), + id_basis: z + .enum(["path-parameter", "field-name"]) + .describe( + "path-parameter: item responses repeat the requested path value in this field; field-name: an inferred identity name every listed item carries distinctly.", + ), + init_before_ordinal: z + .number() + .int() + .nonnegative() + .nullable() + .describe( + "First recorded write whose path root relates to the type; state comes from records that finished before it.", + ), + volatile_fields: z + .array(z.string()) + .describe( + "Fields whose recorded values changed with no recorded write between; the state keeps the latest.", + ), + }), + ), + entities: z + .array( + z.strictObject({ + type: z.string(), + collection: z.string(), + id: z.string(), + fields: jsonObjectFieldsSchema, + view_fields: z.array( + z.strictObject({ tool: z.string(), fields: jsonObjectFieldsSchema }), + ), + record_ordinals: z.array(z.number().int().nonnegative()), + }), + ) + .describe( + "Entities in collection order: type, collection, then first sighting.", + ), + documents: z.array( + z.strictObject({ + tool: z.string(), + path: z.string(), + value: jsonValueSchema, + record_ordinal: z.number().int().nonnegative(), + }), + ), + views: z.array(viewSchema), + unknowns: z.array( + z.strictObject({ + kind: z.enum([ + "authorization-model", + "response-semantics", + "state-transition", + ]), + statement: z.string(), + }), + ), +}; + +/** Extract the init state a capture's reads observed and each read tool's render plan; types and identity are inferences. */ +export const projectEnvironmentStateView = ( + parent: AnalysisViewParent, + view: Extract, + accumulator = new EnvironmentStateAccumulator(undefined, view.template_rules), +): Result => { + const grouped = groupCaptureRecords(parent, view.kind, view.template_rules, { + accumulator, + noun: "state", + limitations: [ + "Entity types, identity fields, list membership and counters are inferences from recorded response bodies; the state does not establish server-side schemas, derived values or values copied outside entity objects.", + ], + }); + if (!grouped.ok) return grouped; + const state = accumulator.finishState(); + if (!state.ok) return state; + return ok({ + kind: "environment-state", + view, + parent_evidence_id: parent.evidenceId, + parent_operation: grouped.value.operation, + parent_digest: parent.evidenceId.slice(3), + artifact: { + path: grouped.value.artifact.path, + sha256: grouped.value.artifact.sha256, + }, + ...state.value, + }); +}; diff --git a/src/domain/analysisView/javascriptApplicationView.ts b/src/domain/analysisView/javascriptApplicationView.ts index 3a36e4b812..ca9d96b76e 100644 --- a/src/domain/analysisView/javascriptApplicationView.ts +++ b/src/domain/analysisView/javascriptApplicationView.ts @@ -309,7 +309,7 @@ export const projectJavaScriptApplicationView = ( analysis: JavaScriptApplicationAnalysisResult, view: Exclude< AnalysisViewRequest, - { kind: "api-surface" | "environment-manifest" } + { kind: "api-surface" | "environment-manifest" | "environment-state" } >, ): Result => { if ( diff --git a/src/domain/analysisView/recordedResponseBody.ts b/src/domain/analysisView/recordedResponseBody.ts new file mode 100644 index 0000000000..1adf258774 --- /dev/null +++ b/src/domain/analysisView/recordedResponseBody.ts @@ -0,0 +1,63 @@ +import { z } from "zod"; +import type { WebNetworkCaptureRecord } from "../webNetworkCapture.js"; + +/** One record's retained response text, as replay and state extraction both read it. */ +export interface RecordedResponseBody { + /** Producer-decoded response text, or null when the record retained none. */ + readonly bodyText: string | null; + /** Set when the retained bytes are not UTF-8, so `bodyText` holds their base64 rather than a lossy decoding. */ + readonly bodyEncoding?: "base64"; + /** Set when only a prefix of a large base64 body was decoded. */ + readonly bodyCut?: true; +} + +const contentTextSchema = z.object({ + response: z.object({ + content: z.object({ + text: z.unknown().optional(), + encoding: z.unknown().optional(), + }), + }), +}); + +/** Strict, and keeping a leading byte-order mark, so callers see the recorded text unchanged. */ +const strictUtf8 = (): TextDecoder => + new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); + +/** Base64 characters that decode to just over `limitBytes`, so a cut prefix still fills it. */ +export const base64PrefixChars = (limitBytes: number): number => + Math.ceil((limitBytes + 4) / 3) * 4; + +/** The retained response text of one record, decoding HAR base64 content when the decoder kept it as valid UTF-8; base64 beyond `limitBytes` is decoded only that far. */ +export const readRecordedResponseBody = ( + record: WebNetworkCaptureRecord, + limitBytes: number, +): RecordedResponseBody => { + const content = contentTextSchema.safeParse(record.reported); + if (!content.success) return { bodyText: null }; + const { text, encoding } = content.data.response.content; + if (typeof text !== "string") return { bodyText: null }; + if (encoding !== "base64") return { bodyText: text }; + const binary = record.binary_fields.find( + (field) => + field.pointer === "/response/content/text" && field.state === "retained", + ); + if (binary === undefined || binary.content_base64 === null) + return { bodyText: null }; + const prefixChars = base64PrefixChars(limitBytes); + const cut = binary.content_base64.length > prefixChars; + const encoded = cut + ? binary.content_base64.slice(0, prefixChars) + : binary.content_base64; + const flags = cut ? { bodyCut: true as const } : {}; + try { + // A cut prefix may end inside a character; streaming leaves that tail undecoded. + const text = strictUtf8().decode(Buffer.from(encoded, "base64"), { + stream: cut, + }); + return { bodyText: text, ...flags }; + } catch { + // Binary or non-UTF-8 bytes: keep them exactly, as base64, instead of inventing replacement characters. + return { bodyText: encoded, bodyEncoding: "base64", ...flags }; + } +}; diff --git a/src/domain/analysisView/scenarioCaptureRecords.test.ts b/src/domain/analysisView/scenarioCaptureRecords.test.ts index 9579dd118b..bb3a006374 100644 --- a/src/domain/analysisView/scenarioCaptureRecords.test.ts +++ b/src/domain/analysisView/scenarioCaptureRecords.test.ts @@ -158,6 +158,7 @@ it("reserves retained request-body expansion before schema decoding or projectio normalizedResult: capture, limitations: [], }, + "environment-manifest", new EnvironmentManifestAccumulator(128 * 1024), ); expect(projected).toMatchObject({ @@ -205,6 +206,7 @@ it("reserves request text even when another event's body was not captured", () = normalizedResult: { ...capture, events: { ...capture.events, items } }, limitations: [], }, + "environment-manifest", new EnvironmentManifestAccumulator(8 * 1024 * 1024), ); expect(projected).toMatchObject({ diff --git a/src/domain/analysisView/scenarioCaptureRecords.ts b/src/domain/analysisView/scenarioCaptureRecords.ts index 19e108417d..f5d0d2751e 100644 --- a/src/domain/analysisView/scenarioCaptureRecords.ts +++ b/src/domain/analysisView/scenarioCaptureRecords.ts @@ -293,9 +293,10 @@ const derivedTextBytes = (value: unknown): number => { return 64 + largestBody * 2 + requestText; }; -/** Admit a capture_browser_scenario parent whose network content identifies response media types. */ +/** Admit a capture_browser_scenario parent whose network content identifies response media types; state also needs bodies. */ export const admitScenarioCaptureParent = ( parent: AnalysisViewParent, + viewKind: "environment-manifest" | "environment-state", accumulator: WebNetworkProjectionAccumulator, ): Result => { const admitted = accumulator.admitCaptureClone(parent.normalizedResult); @@ -321,13 +322,19 @@ export const admitScenarioCaptureParent = ( ]), ); const content = parsed.data.scenario.network_content; - // Tools are chosen by response media type, which only bodies or header values carry. - if (!content.response_body && !content.header_values) + // Tools are chosen by response media type, which only bodies or header values carry; state is read from bodies. + if ( + viewKind === "environment-state" + ? !content.response_body + : !content.response_body && !content.header_values + ) return err( new AnalysisUnsupportedTargetError( "inspect_analysis_view", parent.evidenceId, - "environment-manifest needs a capture_browser_scenario capture with capture.network.response_body or header_values selected.", + viewKind === "environment-state" + ? "environment-state needs a capture_browser_scenario capture with capture.network.response_body selected." + : "environment-manifest needs a capture_browser_scenario capture with capture.network.response_body or header_values selected.", ), ); return ok(scenarioCaptureRecords(parsed.data, "manifest")); diff --git a/src/domain/analysisView/webNetworkCaptureView.ts b/src/domain/analysisView/webNetworkCaptureView.ts index 02da9d59b1..f962110c31 100644 --- a/src/domain/analysisView/webNetworkCaptureView.ts +++ b/src/domain/analysisView/webNetworkCaptureView.ts @@ -437,7 +437,7 @@ export const projectWebNetworkCaptureView = ( path: ["view"], reason: "invalid_value", message: - "Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface or environment-manifest views.", + "Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface, environment-manifest or environment-state views.", }, ]), ); diff --git a/src/domain/exactJsonNumber.ts b/src/domain/exactJsonNumber.ts index 411d2f8f91..19f828023e 100644 --- a/src/domain/exactJsonNumber.ts +++ b/src/domain/exactJsonNumber.ts @@ -1,4 +1,6 @@ import { isSafeNumber } from "lossless-json"; +import { jsonValueSchema, type JsonValue } from "./jsonValue.js"; +import { safeParseJson } from "./safeJson.js"; /** Whether native JSON conversion and serialization preserve the recorded number. */ export const isExactJsonNumber = ( @@ -6,3 +8,17 @@ export const isExactJsonNumber = ( source: string | undefined, ): boolean => source !== undefined && isSafeNumber(source) && !Object.is(value, -0); + +/** Parse JSON text whose every number survives native conversion exactly; undefined otherwise. */ +export const parseExactJson = (text: string): JsonValue | undefined => { + const parsed = safeParseJson(text, (_key, value, context) => { + // The native parser supplies each primitive's original spelling. Share + // HAR's exact-number policy rather than accepting an already-rounded ID. + if (typeof value === "number" && !isExactJsonNumber(value, context?.source)) + throw new RangeError("JSON numeric conversion would lose recorded data"); + return value; + }); + if (!parsed.ok) return undefined; + const value = jsonValueSchema.safeParse(parsed.value); + return value.success ? value.data : undefined; +}; diff --git a/tests/acceptance/tools/qlogCapture.test.ts b/tests/acceptance/tools/qlogCapture.test.ts index 1765dcedce..eafaf0ad4f 100644 --- a/tests/acceptance/tools/qlogCapture.test.ts +++ b/tests/acceptance/tools/qlogCapture.test.ts @@ -302,7 +302,11 @@ it("runs built CLI and real stdio SDK with exact evidence parity, selection, ret covered_ranges: [{ start: "10", end: "15" }], }); // An unsupported view over an existing retained parent proves lookup without pretending qlog is HAR. - for (const kind of ["api-surface", "environment-manifest"] as const) { + for (const kind of [ + "api-surface", + "environment-manifest", + "environment-state", + ] as const) { const retained = await call("inspect_analysis_view", { source: { kind: "retained-evidence", evidence_id: mcp.evidence_id }, view: { kind }, diff --git a/tests/boundary/cli/analysisViewCli.test.ts b/tests/boundary/cli/analysisViewCli.test.ts index ca37b4f6af..ca10bd9dcf 100644 --- a/tests/boundary/cli/analysisViewCli.test.ts +++ b/tests/boundary/cli/analysisViewCli.test.ts @@ -22,6 +22,7 @@ import { analysisViewJavaScriptAnalysisWithSource, analysisViewNativeEvidence, environmentManifestCaptureFixture, + environmentStateCaptureFixture, } from "../../fixtures/analysisView.js"; import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js"; import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js"; @@ -661,6 +662,52 @@ cliTest( }, ); +cliTest( + "extracts an environment state through the compiled CLI with recorded values", + async ({ cli }) => { + const root = await createTestTempDirectory("rea-environment-state-cli-"); + const path = join(root, "view.json"); + await writeFile( + path, + JSON.stringify({ + source: { + kind: "inline", + evidence: analysisViewCaptureEvidence( + environmentStateCaptureFixture(), + ), + }, + view: { + kind: "environment-state", + template_rules: [ + "after-uid-literal", + "opaque-token", + "sibling-variance", + ], + }, + }), + ); + const result = await cli.run({ + arguments: ["inspect-analysis-view", path, "--json"], + }); + expect(result.exitCode, result.stdout).toBe(0); + const evidence = parseEvidence(result.json); + expect(evidence.confidence).toBe("inferred"); + expect(evidence.normalized_result).toMatchObject({ + kind: "environment-state", + origin: "http://grafana.example", + timeline: "recorded-start-time", + types: expect.arrayContaining([ + expect.objectContaining({ + collection_template: "/api/folders", + id_field: "uid", + }), + ]), + }); + for (const value of ["probe folder", "probe@example.test"]) + expect(result.stdout).toContain(value); + }, +); + cliTest( "decodes a Grafana-shaped HAR and projects its environment manifest through the compiled workflows", async ({ cli }) => { diff --git a/tests/boundary/mcp/analysisViewMcp.test.ts b/tests/boundary/mcp/analysisViewMcp.test.ts index 4457a6c291..749e9665b5 100644 --- a/tests/boundary/mcp/analysisViewMcp.test.ts +++ b/tests/boundary/mcp/analysisViewMcp.test.ts @@ -43,6 +43,7 @@ import { analysisViewObjectDenseCapture, analysisViewNativeEvidence, environmentManifestCaptureFixture, + environmentStateCaptureFixture, } from "../../fixtures/analysisView.js"; import { createTestBinarySession } from "../../fixtures/binarySession.js"; import { BINARY_LAYOUT_TEST_PROVIDER } from "../../fixtures/binaryDiagnostics/layout.js"; @@ -1167,6 +1168,79 @@ it.each(PUBLISHED_EXAMPLES)( }, ); +it("extracts an environment state identically from retained, portable and file MCP Evidence", async () => { + const { client, session } = await connect(); + const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas( + client, + "inspect_analysis_view", + ); + const capture = environmentStateCaptureFixture(); + const parent = analysisViewCaptureEvidence(capture); + expect(session.recordEvidence(parent).ok).toBe(true); + const evidencePath = join( + await createTestTempDirectory("rea-state-file-mcp-"), + "capture.json", + ); + await writeFile(evidencePath, JSON.stringify(parent)); + const view = { + kind: "environment-state", + template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"], + }; + const outputs = []; + for (const source of [ + { kind: "inline", evidence: parent }, + { kind: "retained-evidence", evidence_id: parent.evidence_id }, + { kind: "evidence-file", path: evidencePath }, + ]) { + expect(ajv.validate(inputSchema, { source, view })).toBe(true); + const response = await client.callTool({ + name: "inspect_analysis_view", + arguments: { source, view }, + }); + expect(response.isError).not.toBe(true); + expect( + ajv.validate(outputSchema, response.structuredContent), + JSON.stringify(ajv.errors), + ).toBe(true); + const evidence = parseEvidence(response.structuredContent); + expect(evidence.confidence).toBe("inferred"); + expect(evidence.normalized_result).toMatchObject({ + kind: "environment-state", + views: expect.arrayContaining([ + expect.objectContaining({ + tool: "get_folders_by_uid", + shape: "item", + }), + ]), + }); + expect(session.evidenceById(evidence.evidence_id)).toEqual(evidence); + outputs.push(evidence.normalized_result); + } + expect(outputs[0]).toEqual(outputs[1]); + expect(outputs[2]).toEqual(outputs[0]); + const hidden = createEvidence( + { + path: capture.artifact.path, + sha256: capture.artifact.sha256!, + format: "file", + }, + { id: "fixture", name: "fixture", version: "1" }, + { + operation: "inspect_web_network_capture", + parameters: { sensitive_value_count: 1 }, + result: capture, + confidence: "observed", + }, + ); + const refused = await client.callTool({ + name: "inspect_analysis_view", + arguments: { source: { kind: "inline", evidence: hidden }, view }, + }); + expect(parseMcpToolError(refused).error.code).toBe("unsupported_target"); + expect(JSON.stringify(refused)).not.toContain("probe folder"); + await client.ping(); +}); + it("projects an environment manifest identically from retained, portable and file MCP Evidence", async () => { const { client, session } = await connect(); const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas( diff --git a/tests/fixtures/analysisView.ts b/tests/fixtures/analysisView.ts index e9eea8f5b8..9a88e33fe9 100644 --- a/tests/fixtures/analysisView.ts +++ b/tests/fixtures/analysisView.ts @@ -1,4 +1,9 @@ -import { apiSurfaceHar, environmentManifestHar } from "./historicalHar.js"; +import type { JsonValue } from "../../src/domain/jsonValue.js"; +import { + apiSurfaceHar, + environmentManifestHar, + environmentStateHar, +} from "./historicalHar.js"; import { webNetworkCaptureSchema } from "../../src/domain/webNetworkCapture.js"; import { BINARY_LAYOUT_TEST_PROVIDER } from "./binaryDiagnostics/layout.js"; import { binaryLayoutFixture } from "./binaryDiagnostics/layout.js"; @@ -290,9 +295,10 @@ export const analysisViewObjectDenseCapture = (headerCount = 100_000) => { }; /** Capture parent decoded from the Grafana-shaped HAR with producer ordinals intact. */ -export const environmentManifestCaptureFixture = () => { +export const environmentManifestCaptureFixture = ( + entries: readonly JsonValue[] = environmentManifestHar().log.entries, +) => { const capture = analysisViewCaptureFixture(); - const entries = environmentManifestHar().log.entries; capture.artifact.path = "/fixtures/environment-manifest.har"; capture.total_records = entries.length; capture.records = entries.map((reported, ordinal) => ({ @@ -303,3 +309,7 @@ export const environmentManifestCaptureFixture = () => { })); return capture; }; + +/** The same capture with distinct start times, as the environment-state view orders it. */ +export const environmentStateCaptureFixture = () => + environmentManifestCaptureFixture(environmentStateHar().log.entries); diff --git a/tests/fixtures/historicalHar.ts b/tests/fixtures/historicalHar.ts index 4ecc606633..a7ae76bd95 100644 --- a/tests/fixtures/historicalHar.ts +++ b/tests/fixtures/historicalHar.ts @@ -309,3 +309,14 @@ export const environmentManifestHar = () => ({ ), }, }); + +/** The Grafana-shaped HAR with one second between requests, so its timeline orders reads before the writes that follow them. */ +export const environmentStateHar = () => { + const har = environmentManifestHar(); + const start = Date.parse("2026-10-17T18:00:00.000Z"); + har.log.entries = har.log.entries.map((entry, index) => ({ + ...entry, + startedDateTime: new Date(start + index * 1000).toISOString(), + })); + return har; +};