mirror of
https://github.com/morluto/rea
synced 2026-10-25 22:08:11 -04:00
feat(analysis-view): project an environment manifest from web network captures (#4954)
* feat(analysis-view): project an environment manifest from web network captures Add the environment-manifest view kind to inspect_analysis_view for retained HAR Evidence: the primary origin's recorded request lines become a deterministic tool surface (names, path templates, parameters with their rule and basis, observed query names, value-free body shapes, record pointers), with excluded groups, three fixed unknowns and the shared projection budget. The four format rules always apply; after-uid-literal, opaque-token and sibling-variance are opt-in and labelled, and literal fold-in never touches a format-rule parameter. The api-surface accumulator moves onto shared route, media-type and budget helpers without behaviour change. Refs morluto/rea#4933 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(web-network): check the manifest refusal text like the api-surface refusal The real-capture verifier stringified the whole MCP result for the environment-manifest refusal, so a declared "[" literal matched the result envelope itself; compare the parsed refusal as the api-surface branch does. Refs morluto/rea#4933 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): record the hosted web-network lane run for the manifest view Refs morluto/rea#4933 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(specs): reconcile the manifest view verification scope note with the hosted run Refs morluto/rea#4933 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): bound body, route and name work in the environment manifest Addresses the automated review findings on the PR: read request bodies at the top level only instead of walking nested values; admit every literal route part before materializing it; compare at most 32 segments in sibling-variance and say so; keep stop words literal after a uid literal; keep the next collision suffix per tool-name base; preserve a templated exclusion when two groups print the same row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): keep manifest folding, naming and media-type reads linear Addresses the second automated review round: fold literal siblings through set membership, key collision suffixes by the truncated name they share, and charge a request media type before it is scanned. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): charge the manifest group key before building it Addresses the third automated review round: the composite key copies the method, origin, template and rules, so that copy is reserved before the key is stringified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): accept a +json suffix only on application media types Addresses the fourth automated review round: `image/example+json` or `text/example+json` no longer counts as JSON for origin selection, exclusion or body shapes, matching the documented `application/json` and `application/*+json` contract. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): require the capture parent's exact subject for a manifest Addresses the fifth automated review round: the manifest cites the report path and digest, so a parent whose subject is absent or names another path or SHA-256 is refused at admission, as the capture view already does; the api-surface view keeps its existing admission. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): share truncated suffix allocation work Track suffix progress by the actual truncated prefix and suffix width. Distinct full names can collide only after suffix truncation; sharing that progress prevents quadratic rescanning while preserving shorter free names and existing allocation order. Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * fix(analysis-view): merge only identifier-like siblings under sibling-variance Addresses the sixth automated review finding: a bucket merged whenever one literal carried a digit, folding plain words such as `mine` into the parameter. The rule now partitions the bucket, merges the identifier-like siblings and leaves the other literals as their own tools. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(analysis-view): combine qualifying sibling cohorts Keep coded identifiers and a qualifying plural hyphenated cohort in one eligibility Set before checking the minimum merge size. Read each literal once and reuse the same membership at the consumer, preserving non-identifier routes and the three-hyphen minimum. Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> * docs(analysis-view): clarify opt-in sibling folding State the additional sibling-variance selection in the guide, acceptance specification and MCP parameter description. Preserve the existing single-rule behavior and caller-selected inference rules. Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
Edgar Arakelyan
@@ -69,3 +69,11 @@ When the connected schema advertises `include_query_parameter_names`, select
|
||||
it in the view for distinct observed names with original record ordinals. Query
|
||||
values are omitted; the names establish neither required parameters nor a
|
||||
complete server schema. Keep the original parent Evidence for exact URLs.
|
||||
|
||||
Use `inspect_analysis_view` with `view: {kind: "environment-manifest"}` on the
|
||||
same retained HAR Evidence to project the primary origin's request lines into
|
||||
named tools with path templates, parameters, observed query names, value-free
|
||||
body shapes and record pointers for an agent environment. Opt into
|
||||
`template_rules` only when recorded identifiers need them; every template,
|
||||
parameter and merged sibling is an inference, no header, cookie or body value is
|
||||
copied, and the three stated unknowns stay unresolved.
|
||||
|
||||
@@ -210,6 +210,14 @@ inferred Evidence whenever any group is templated, and derived Evidence only whe
|
||||
no group was templated.
|
||||
See [derived API surfaces](web-network-captures.md#derived-api-surface).
|
||||
|
||||
`inspect_analysis_view` also accepts `environment-manifest` for retained HAR capture
|
||||
Evidence, projecting the primary origin's request lines into named tools with path
|
||||
templates, parameters, observed query names, value-free body shapes and record
|
||||
pointers, plus excluded groups and three fixed unknowns. Templates, parameters and
|
||||
names are inferences; the Evidence is inferred whenever any tool or excluded group
|
||||
is templated, and derived otherwise.
|
||||
See [environment manifests](web-network-captures.md#environment-manifest).
|
||||
|
||||
`inspect_analysis_view` projects a caller-selected view of already completed
|
||||
`inspect_binary_layout`, `analyze_javascript_application`, or `analyze_function`
|
||||
Evidence. Native views select procedure, pseudocode, assembly, basic blocks,
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
# Implementation plan
|
||||
|
||||
1. Lift the api-surface accumulator's shared pieces into
|
||||
`webNetworkRouteTemplates.ts` (format rules, media-type essence, request-line
|
||||
and query-name readers, route scanning, byte allowances) and an abstract
|
||||
`WebNetworkProjectionAccumulator` (retained/temporary budget, admission,
|
||||
limitations) with `admitHarCaptureParent` for the shared parent guards, so
|
||||
both grouping views own one interpretation.
|
||||
2. Add `environmentManifestRules.ts` as pure functions: the opt-in template
|
||||
rules, re-bucketing and sibling merging, parameter naming, path templates and
|
||||
deterministic tool names.
|
||||
3. Add `EnvironmentManifestAccumulator.ts`: per-record pointers, group state,
|
||||
primary-origin selection, body-shape inference through `inferJsonShape`,
|
||||
exclusion reasons, fixed unknowns and the finished manifest fields.
|
||||
4. Add `environmentManifestView.ts` with the strict request schema (defaults
|
||||
applied in the echoed view), the result fields and the projector; register
|
||||
the kind first in `projectAnalysisView`, extend the result union and
|
||||
`UnsignedAnalysisView`, and wire the accumulator and confidence rule in
|
||||
`AnalysisViewService`.
|
||||
5. Publish the `inspect_analysis_view` description and a fixture-generated
|
||||
example; port the Grafana-shaped HAR into a source-owned TypeScript builder.
|
||||
6. Cover domain, accumulator budget, service, MCP boundary, CLI boundary and
|
||||
acceptance behavior; extend the real-capture verifier; document the view in
|
||||
the capture guide, the contracts reference and the authored skill.
|
||||
|
||||
No new tool, provider, side effect or network activity is introduced. The
|
||||
manifest is a projection of already retained Evidence; replaying or serving it
|
||||
is deferred.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Environment manifest view from web network captures
|
||||
|
||||
Issue: https://github.com/morluto/rea/issues/4933
|
||||
|
||||
## Outcome
|
||||
|
||||
`inspect_analysis_view` accepts `view: {"kind": "environment-manifest"}` for
|
||||
retained HAR Evidence and projects the primary origin's recorded request lines
|
||||
into a replayable tool surface: deterministic tool names, path templates,
|
||||
parameters, observed query names, value-free body shapes, status and
|
||||
content-type sets and record pointers, with excluded groups, three fixed
|
||||
unknowns and the shared projection representation budget. This increment adds
|
||||
the view kind only; serving and exporting an agent environment from the
|
||||
manifest are later increments.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Accept only `inspect_web_network_capture` HAR parents; refuse native
|
||||
mitmproxy, SAZ and qlog parents, and parents whose Evidence does not record
|
||||
`sensitive_value_count: 0`, with the same typed unsupported errors as the
|
||||
api-surface view.
|
||||
- Without opt-in rules, every tool template must equal the api-surface template
|
||||
of the same group; the four format rules always apply with `basis: inferred`.
|
||||
- Apply `after-uid-literal`, `opaque-token` and `sibling-variance` only when
|
||||
listed, each at most once, in that order. With `sibling-variance` selected,
|
||||
fold literal siblings within its comparison scope into existing opt-in
|
||||
parameters unless they are stop words, whichever was recorded first. With only
|
||||
`opaque-token`, a non-opaque literal remains a separate tool. Never fold a
|
||||
literal into a format-rule parameter or template a stop word;
|
||||
`sibling-variance` compares routes of at most 32 segments and the manifest
|
||||
states when a route exceeded that.
|
||||
- Derive tool names from the method and route literals deterministically, at
|
||||
most 60 characters, unique within one manifest.
|
||||
- Derive `body_shape` for `POST`, `PUT` and `PATCH` from the tool's earliest
|
||||
record when it carries a JSON-typed request body, as top-level key types
|
||||
only without walking nested values; copy no header, cookie, query or body
|
||||
value anywhere in the result.
|
||||
- List records as pointers (`ordinal`, `method`, `path`, `status`,
|
||||
`content_type`) and excluded groups with reason `other-origin`,
|
||||
`ui-page-or-asset` or `non-json`.
|
||||
- Mark the Evidence inferred whenever any tool or excluded group is templated,
|
||||
derived otherwise; keep the parent's authority.
|
||||
- Admit the capture clone, the inline envelope, every retained pointer, group,
|
||||
route part, observed value, query name and body-shape key before allocation,
|
||||
and charge body parsing as temporary bytes, within the existing 96 MiB budget.
|
||||
|
||||
## Acceptance and limits
|
||||
|
||||
A source-owned Grafana-shaped HAR fixture (28 recorded exchanges on two
|
||||
origins) drives domain, service, MCP, CLI and acceptance coverage. Without
|
||||
rules the templates equal the api-surface grouping; with all three rules the
|
||||
fixture yields `/api/dashboards/uid/{uid}`, `/api/folders/{uid}` folding
|
||||
`general`, `/api/plugins/{plugin_id}/settings` and
|
||||
`/api/datasources/uid/{uid}/health`. A published contract example is
|
||||
reproduced byte-for-byte from fixture bytes through the real decoder and view
|
||||
service. The view performs no network activity and replays nothing; it does
|
||||
not establish handler identity, schemas, authorization or state transitions.
|
||||
|
||||
## Deferred issue criteria
|
||||
|
||||
Serving a manifest over streamable HTTP, exporting an agent environment from a
|
||||
manifest and the hosted verification lane are later independent increments of
|
||||
#4933. This PR must reference rather than close the whole issue.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Tasks
|
||||
|
||||
- [x] Record the view contract, requirements and deferred increments.
|
||||
- [x] Factor shared route, media-type and budget helpers out of the api-surface
|
||||
accumulator without changing its behavior.
|
||||
- [x] Implement the opt-in template rules, naming and manifest accumulator.
|
||||
- [x] Register the `environment-manifest` kind, result schema, service wiring and
|
||||
confidence rule.
|
||||
- [x] Port the Grafana-shaped HAR fixture and publish a reproducible example.
|
||||
- [x] Add domain, service, MCP, CLI and acceptance coverage.
|
||||
- [x] Document the view in the capture guide, contracts reference and skill.
|
||||
- [x] Run static, formatting, duplication, documentation and test gates.
|
||||
@@ -0,0 +1,186 @@
|
||||
# Verification
|
||||
|
||||
Verified base: 18aa9af2b2c26fd114113dd39528a91224801840 (fetched main).
|
||||
|
||||
Executed on the implementation worktree after `npm ci`, `npm run build` and
|
||||
`npm run test:prepare`; every check below ran as written in this pass. The
|
||||
built-group projects and the built-CLI checks ran against a `dist/` rebuilt
|
||||
after the last source change.
|
||||
|
||||
- `npm run check:fast`: typecheck and lint passed (2 tasks successful).
|
||||
- `npm run format` then `npm run format:check`: all 4028 matched files use the
|
||||
correct format.
|
||||
- `npx knip`: ok (exit 0, no unused files, exports or dependencies reported).
|
||||
- `npm run check:ci:tasks`: typecheck, lint:built (3730 files, 0 errors), format:check, knip,
|
||||
metadata:check, jscpd (3.35% duplicated lines over 3140 files and 2581
|
||||
clones, under the 3.5% gate) and the tech-debt marker scan passed (8 tasks,
|
||||
3 replayed from this worktree's cache).
|
||||
- `npm run docs:check`: the six built documentation checks and the README
|
||||
consistency conformance test (5 tests) passed; no tracked generated file
|
||||
changed.
|
||||
- `npx vitest run --project domain --project services` (full source group):
|
||||
312 files, 5412 tests passed, 3 pre-existing skips.
|
||||
- `npx vitest run --project cli-boundary` (full project): 111 files; 659 tests passed, 2
|
||||
pre-existing skips, 8 failed in three files that drive no analysis-view
|
||||
command (`closedStdout`, `commandFinalizationCancellation`,
|
||||
`javascriptApplicationJsonOutput`). Re-run in isolation on this worktree the
|
||||
three WASM interrupt cases and the four JavaScript JSON-output cases passed
|
||||
and only `closedStdout` "stops quietly when the reader closes during streamed
|
||||
JSON" (`inspect-sqlite-database`) exited 1 instead of 141. The same three
|
||||
files run on a clean export of 18aa9af2 (`git archive`, fresh `npm ci`,
|
||||
`npm run build`, `npm run test:prepare`, nothing from this slice) reproduced
|
||||
exactly that one failure and passed the other 16, so none of the eight is
|
||||
attributable to this slice and none was changed here.
|
||||
- `npx vitest run --project mcp-boundary` (full project): 153 files passed, 2
|
||||
pre-existing file skips; 901 tests passed, 7 pre-existing skips, no failures.
|
||||
- `npx vitest run --project acceptance` (full project): 123 files passed, 1
|
||||
pre-existing file skip; 608 tests passed, 13 pre-existing skips, no failures.
|
||||
- Built CLI end-to-end on the 28-record Grafana-shaped capture:
|
||||
`inspect-web-network-capture ... har --format json` then
|
||||
`inspect-analysis-view` with `environment-manifest`, with and without the
|
||||
three opt-in rules, and with `api-surface`, twice each. Without rules the 22
|
||||
tool templates plus the 3 primary-origin exclusions equal the 25 api-surface
|
||||
groups on that origin and every `path_template` equals its
|
||||
`api_surface_template`; with rules `/api/dashboards/uid/{uid}`,
|
||||
`/api/folders/{uid}` (observed values `fg1jo91e57e2ob` and `general`, records
|
||||
12 and 13), `/api/plugins/{plugin_id}/settings` and
|
||||
`/api/datasources/uid/{uid}/health` appear across 19 tools with 13 templated
|
||||
groups; `normalized_result` and `evidence_id` are identical across the two
|
||||
runs of each view; records carry only pointer keys; no recorded body or query
|
||||
value appears; three unknowns and `view_digest` are present; confidence is
|
||||
`inferred` with authority `historical-reference`.
|
||||
- Built CLI on the same capture extended with `GET /api/teams/search` and
|
||||
`GET /api/teams/mine` (30 records), all three rules selected:
|
||||
`/api/teams/{numeric}` keeps observed value `1` only and `get_teams_search`
|
||||
and `get_teams_mine` stay their own tools; 21 tools, 13 templated groups.
|
||||
|
||||
## Independent review
|
||||
|
||||
Findings repaired at their owning boundary:
|
||||
|
||||
1. A `__proto__` key in a recorded JSON body became a prototype rather than an
|
||||
own property, and the schema clone then dropped it silently. The manifest
|
||||
accumulator builds body shapes with `Object.fromEntries`, omits that key and
|
||||
states the omission as a limitation; a domain regression covers it.
|
||||
2. The contract example carried a readonly tuple that the mutable JSON example
|
||||
type rejects; the example keeps a literal-typed mutable tuple.
|
||||
3. Folding a literal sibling into an already templated position took the parts
|
||||
of the earliest group, so a literal recorded before the token kept its
|
||||
literal, dropped the parameter and claimed the token's record under a
|
||||
template it does not match. `mergeGroups` now takes the templated group as
|
||||
the shape for parts while the earliest group still names
|
||||
`api_surface_template`, `body_shape` and record order; a domain regression
|
||||
with the literal recorded first covers it.
|
||||
4. The request-body parse accepted `text/plain` and `text/markdown` bodies; it
|
||||
accepts only `application/json` and `application/*+json`, and the capture
|
||||
guide states the earliest-record rule; a domain test covers both media
|
||||
types.
|
||||
5. A repeated entry in `template_rules` was accepted, echoed and hashed into
|
||||
`view_digest` while the accumulator deduplicated it; the request schema
|
||||
refuses repeats with `template_rules must not repeat a rule`.
|
||||
6. The sibling-variance fold-in absorbed a non-conforming literal into a
|
||||
parameter produced by one of the four format rules, so `/api/teams/mine`
|
||||
became an observed value of `/api/teams/{numeric}` and the published rule no
|
||||
longer described every observed value. Fold-in is restricted to parameters
|
||||
produced by the opt-in rules; literal siblings of a format-rule parameter
|
||||
stay their own tools or merge among themselves under `sibling-variance`
|
||||
only when identifier-like. A domain test with `/api/teams/1`,
|
||||
`/api/teams/2`, `/api/teams/search` and `/api/teams/mine`, the capture
|
||||
guide and the spec state the rule; the built-CLI check above exercises it.
|
||||
|
||||
## Hosted lane
|
||||
|
||||
- `real-web-network-captures` (hosted ubuntu-24.04, workflow run
|
||||
https://github.com/earakely-scale/rea/actions/runs/39097105064 on the fork
|
||||
at 2d95597f) passed with the manifest block of
|
||||
`scripts/verify-web-network-captures.mjs` executed against the lane's
|
||||
producer and mitmproxy captures. The first run at 61d35369 failed only in
|
||||
that block: the refusal check stringified the whole MCP result, so the
|
||||
declared `[` literal matched the result envelope; the follow-up compares the
|
||||
parsed refusal as the api-surface branch does.
|
||||
|
||||
## Scope and measurement limits
|
||||
|
||||
macOS `/usr/bin/time -l`, command accounting only: `npm run check:fast`
|
||||
reported 5.23 seconds elapsed, 24.20 user and 2.92 system seconds, and
|
||||
4449140736 bytes maximum RSS; `npm run format` 0.74 seconds elapsed, 4.14
|
||||
user and 0.73 system seconds, and 223363072 bytes maximum RSS;
|
||||
`npm run format:check` 0.66 seconds elapsed, 4.03 user and 0.61 system
|
||||
seconds, and 218759168 bytes maximum RSS; the full source group reported
|
||||
25.10 seconds elapsed, 67.26 user and 2.72 system seconds, and 2659172352
|
||||
bytes maximum RSS; `npx knip` 3.68 seconds elapsed, 5.24 user and 0.70 system
|
||||
seconds, and 865599488 bytes maximum RSS; `npm run check:ci:tasks` 4.65
|
||||
seconds elapsed, 15.03 user and 4.36 system seconds, and 843333632 bytes
|
||||
maximum RSS; `npm run docs:check` 3.61 seconds elapsed, 3.81 user and 0.99
|
||||
system seconds, and 289472512 bytes maximum RSS; the full `cli-boundary`
|
||||
project 425.66 seconds elapsed, 703.52 user and 206.77 system seconds, and
|
||||
630456320 bytes maximum RSS; the full `mcp-boundary` project 164.19 seconds
|
||||
elapsed, 302.85 user and 47.08 system seconds, and 4457529344 bytes maximum
|
||||
RSS; the full `acceptance` project 960.25 seconds elapsed, 1520.91 user and
|
||||
520.68 system seconds, and 800063488 bytes maximum RSS. `npx knip` ran while
|
||||
the `cli-boundary` project was executing, and the clean 18aa9af2 export's
|
||||
`npm ci`, build and test preparation overlapped the `mcp-boundary` and
|
||||
`acceptance` runs, so those elapsed figures include host contention. Token usage and full process-family peak are
|
||||
unmeasured. Local real-tool verification ran on macOS arm64; the hosted Linux lane result is recorded above.
|
||||
|
||||
## Automated review findings on the PR (2026-10-18)
|
||||
|
||||
Codex and Greptile left seven inline findings on morluto/rea#4954 at
|
||||
b6d18d23; each was checked against the code and addressed in one follow-up
|
||||
commit, with a regression test where the behaviour is observable:
|
||||
|
||||
- Body shapes walked nested values (both reviewers, P1): `#bodyShape` passed
|
||||
the text to `inferJsonShape`, whose per-node path copies grow with the square
|
||||
of depth, so a small deeply nested body could exceed the budget it had
|
||||
reserved. The reader now parses once and reads top-level key types from the
|
||||
parsed object; a test feeds a 2,500-level nested body and gets the four
|
||||
top-level types back.
|
||||
- Literal route parts were not admitted (Codex, P2): a path of many short
|
||||
literal segments materialized one part object per segment without a charge.
|
||||
Every route part is now charged before the group is created; a test shows a
|
||||
5,000-segment path refused under a 1 MiB budget while a 100-segment path is
|
||||
retained.
|
||||
- `sibling-variance` rebuilt a key over the whole route at every position
|
||||
(Greptile, P1): routes longer than 32 segments are no longer compared, the
|
||||
manifest states it in a limitation, and a test shows two 39-segment siblings
|
||||
left unmerged beside a merged short pair.
|
||||
- `after-uid-literal` templated stop words (Greptile, P1): the rule now skips
|
||||
them; a test keeps `/api/items/uid/search` literal beside
|
||||
`/api/items/uid/{uid}`.
|
||||
- Tool-name suffixes restarted at 2 on every collision (Greptile, P2): the
|
||||
registry keeps the next suffix per base; a test projects 300 dotted literals
|
||||
into `get_item` and `get_item_2` through `get_item_300`.
|
||||
- An exclusion row could lose its templated flag when two groups printed the
|
||||
same row (Codex, P2): the flag is now the logical or of both. The scenario
|
||||
needs a literal segment spelled `{numeric}`, which the WHATWG URL parser
|
||||
percent-encodes in recorded paths, so no capture reaches it; the one-line
|
||||
defence is kept without a test.
|
||||
|
||||
Checks after the change: `oxfmt --check .` clean; `tsc --noEmit` clean;
|
||||
`npm run lint` and `verify:module-boundaries` clean (the complexity warnings
|
||||
on `#append` predate the change); `npx knip` no findings; `npm run jscpd`
|
||||
3.35 % duplicated lines; `scan:todos` none; `verify:test-discovery` 1,580
|
||||
files once; `vitest run` over the view, accumulator, JSON-shape and contract
|
||||
example tests 32 passed in 4 files; `vitest run` over the service, CLI and
|
||||
MCP boundary tests for analysis views 73 passed in 3 files.
|
||||
|
||||
A sixth automated finding (Codex) was addressed after the collaborator's
|
||||
merge of main: `sibling-variance` merged a whole bucket when any one literal
|
||||
carried a digit, so `/api/items/mine` could be folded into `{item_id}`. The
|
||||
rule now partitions the bucket and merges only the identifier-like siblings
|
||||
(each with a digit or colon, or three or more hyphenated names under a plural
|
||||
literal), leaving the other literals as their own tools; with the rule alone
|
||||
the fixture therefore keeps `/api/folders/general` and
|
||||
`/api/folders/fg1jo91e57e2ob` separate, while all three rules still fold
|
||||
`general` into `/api/folders/{uid}` through the opaque-token parameter. Tests:
|
||||
the new partition case and the updated single-rule table.
|
||||
|
||||
Follow-up mixed-cohort regressions reproduced two omissions on `dd87f54e`
|
||||
with main `8307ea37`: two coded siblings prevented three qualifying hyphenated
|
||||
siblings from joining the parameter, and one coded sibling was omitted when
|
||||
three hyphenated siblings merged. Eligibility now belongs to one Set that
|
||||
combines both qualifying cohorts before the minimum group size is checked.
|
||||
The consumer reuses that Set and materializes its merge order once. Three
|
||||
cases cover both omissions and preserve the three-hyphen minimum when only
|
||||
two hyphenated siblings accompany coded identifiers; `mine` and `search`
|
||||
remain literal and every source record remains represented.
|
||||
@@ -315,6 +315,94 @@ parent envelope, capture and examined records are deduplicated onto the view.
|
||||
Portable captures must retain unique ordinals within their declared total;
|
||||
caller-selected subsets can preserve any requested ordinal order.
|
||||
|
||||
## Environment manifest
|
||||
|
||||
Pass retained HAR Evidence to `inspect_analysis_view` with
|
||||
`view: {"kind":"environment-manifest"}` to project one application's recorded
|
||||
request lines into a replayable tool surface: the environment manifest that an
|
||||
agent environment later serves from retained Evidence. The view reuses the
|
||||
api-surface grouping, then keeps the groups on the primary origin, the origin
|
||||
with the most JSON-like records (`application/json`, `application/*+json`,
|
||||
`text/plain`, `text/markdown`), whose path starts with `/api/` or `/apis/` or
|
||||
whose content types are JSON-like. Every other group is listed under
|
||||
`excluded` with reason `other-origin`, `ui-page-or-asset` or `non-json`, and
|
||||
retained records without a usable request line are counted in a limitation.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "inspect_analysis_view",
|
||||
"arguments": {
|
||||
"source": { "kind": "retained-evidence", "evidence_id": "ev_..." },
|
||||
"view": {
|
||||
"kind": "environment-manifest",
|
||||
"template_rules": [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance"
|
||||
],
|
||||
"include_query_parameter_names": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`origin` and `name` identify the environment; the name is the kebab-case
|
||||
primary host with dots and ports replaced by hyphens, `grafana-example` for
|
||||
`http://grafana.example`. Each tool carries a deterministic `name`, `method`,
|
||||
`path_template`, `api_surface_template` (the four-rule template of its
|
||||
earliest record), `params`, `query_parameter_names`, `body_shape`,
|
||||
`statuses`, `content_types` and `record_ordinals`. `records` lists one pointer
|
||||
`{ordinal, method, path, status, content_type}` per record a tool cites and
|
||||
never a header, cookie or body. Tools sort by template then method.
|
||||
|
||||
Without `template_rules`, every `path_template` equals the api-surface
|
||||
template and `params` carry the four format rules with names `id`, `uuid`,
|
||||
`hash` and `date`; a repeated name gets a numeric suffix. Opt-in rules apply
|
||||
only when listed, each at most once (a repeated rule is an `invalid_request`),
|
||||
in this order:
|
||||
|
||||
| Rule | Applies to | Parameter |
|
||||
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
|
||||
| `after-uid-literal` | The literal segment right after a literal `uid`, unless it is a stop word | `uid` |
|
||||
| `opaque-token` | A lowercase alphanumeric segment of twelve or more characters mixing letters and digits that is not a `v1`-style version | `uid` |
|
||||
| `sibling-variance` | Groups equal in method, origin and segment count that differ in one literal at index 2 or later; the siblings whose literal is identifier-like (a digit or `:`, or three or more hyphenated names under a plural literal) merge, never a stop word, and the other literals stay their own tools | singular previous literal plus `_id`, else `id` |
|
||||
|
||||
With `sibling-variance` selected, literal siblings within its comparison scope
|
||||
fold into existing opt-in parameters as observed values unless they are stop words.
|
||||
For example, select both `opaque-token` and `sibling-variance` so
|
||||
`/api/folders/{uid}` lists `general` beside the recorded token. With only
|
||||
`opaque-token`, `/api/folders/general` stays a separate literal tool.
|
||||
A position templated by one of the four format rules
|
||||
absorbs no literal, so `/api/teams/{numeric}` keeps its exact-format meaning
|
||||
and `/api/teams/search` stays its own tool. Opt-in placeholders use the
|
||||
parameter name; format-rule placeholders keep their api-surface spelling. Tool names are
|
||||
`<method>_<literals>` with `api`, `apis`, `namespaces`, `default`, `v`-prefixed
|
||||
version tokens and a `uid` literal before a parameter dropped, dotted literals
|
||||
cut to their first label, `_by_<params>` appended when parameters exist,
|
||||
sanitized to `[a-z0-9_]`, at most 60 characters and made unique with numeric
|
||||
suffixes: `get_dashboards_by_uid`, `get_plugins_settings_by_plugin_id`.
|
||||
|
||||
`sibling-variance` compares routes of at most 32 segments; longer routes keep
|
||||
their grouped templates and the manifest says so in a limitation. `body_shape`
|
||||
is derived for `POST`, `PUT` and `PATCH` from the tool's earliest record when it
|
||||
carries an `application/json` or `application/*+json` request body: top-level
|
||||
keys and their JSON type names, never values and never nested structure; a body
|
||||
on a later record adds nothing. `unknowns` always states three facts: the authorization model, response
|
||||
semantics and state transitions are not established by a capture.
|
||||
`templated_groups` counts tools and excluded groups with a templated segment;
|
||||
whenever it is nonzero the Evidence is inferred, otherwise derived. The view
|
||||
performs no network activity and replays nothing. Native mitmproxy, SAZ and
|
||||
qlog captures, and parents inspected with `sensitive_values`, return the same
|
||||
typed unsupported errors as the api-surface view, including the canonical
|
||||
token `sensitive_values_unretained`.
|
||||
|
||||
The manifest accumulator shares the api-surface admission model: the capture
|
||||
clone, the inline envelope, record pointers, groups, every route part, observed
|
||||
values, query names and body-shape keys are admitted before allocation, and each
|
||||
recorded body is charged as temporary bytes before it is parsed. Retained state and
|
||||
temporary work must fit together within the 96 MiB projection representation
|
||||
budget; exceeding it returns `resource_constraint` without a partial manifest.
|
||||
|
||||
## Resource and lifecycle boundaries
|
||||
|
||||
Complete evidence has a 32 MiB input, 96 MiB decoder reply and 64-level nesting
|
||||
|
||||
@@ -936,6 +936,32 @@ async function inspect(mode, input, errorCategory) {
|
||||
value.normalized_result.records.length,
|
||||
);
|
||||
}
|
||||
const manifest = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: {
|
||||
source: { kind: "retained-evidence", evidence_id: value.evidence_id },
|
||||
view: { kind: "environment-manifest" },
|
||||
},
|
||||
});
|
||||
if ((input.sensitive_values ?? []).length > 0) {
|
||||
assert.equal(manifest.isError, true);
|
||||
const refusedManifest = JSON.parse(mcpTextValue(manifest));
|
||||
for (const literal of input.sensitive_values)
|
||||
assert.ok(!JSON.stringify(refusedManifest).includes(literal));
|
||||
} else {
|
||||
assert.notEqual(manifest.isError, true, mcpTextValue(manifest));
|
||||
const projected = JSON.parse(mcpTextValue(manifest));
|
||||
assert.equal(projected.normalized_result.kind, "environment-manifest");
|
||||
assert.equal(projected.normalized_result.unknowns.length, 3);
|
||||
for (const record of projected.normalized_result.records)
|
||||
assert.deepEqual(Object.keys(record).sort(), [
|
||||
"content_type",
|
||||
"method",
|
||||
"ordinal",
|
||||
"path",
|
||||
"status",
|
||||
]);
|
||||
}
|
||||
}
|
||||
return value.normalized_result;
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ import {
|
||||
analysisViewCaptureFixture,
|
||||
analysisViewCaptureEvidence,
|
||||
analysisViewObjectDenseCapture,
|
||||
environmentManifestCaptureFixture,
|
||||
} from "../../../tests/fixtures/analysisView.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
@@ -446,24 +447,82 @@ it("refuses an object-dense borrowed capture before making a full schema clone",
|
||||
if (!recorded.ok) throw recorded.error;
|
||||
const borrowed = ledger.forAnalysis(parent.evidence_id)!;
|
||||
expect(Object.isFrozen(borrowed.normalized_result)).toBe(true);
|
||||
const result = inspectAnalysisView(
|
||||
{
|
||||
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
view: { kind: "api-surface", limit: 1 },
|
||||
},
|
||||
(id) => ledger.forAnalysis(id),
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisResourceConstraintError",
|
||||
resource: "memory",
|
||||
},
|
||||
});
|
||||
for (const view of [
|
||||
{ kind: "api-surface", limit: 1 },
|
||||
{ kind: "environment-manifest" },
|
||||
] as const) {
|
||||
const result = inspectAnalysisView(
|
||||
{
|
||||
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
view,
|
||||
},
|
||||
(id) => ledger.forAnalysis(id),
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
_tag: "AnalysisResourceConstraintError",
|
||||
resource: "memory",
|
||||
},
|
||||
});
|
||||
}
|
||||
expect(ledger.forAnalysis(parent.evidence_id)).toBe(borrowed);
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface"] as const)(
|
||||
it("seals an inferred environment manifest and a derived manifest without templated groups", () => {
|
||||
const parent = analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture(),
|
||||
);
|
||||
const inferred = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind: "environment-manifest", template_rules: ["opaque-token"] },
|
||||
});
|
||||
if (!inferred.ok) throw inferred.error;
|
||||
expect(inferred.value.confidence).toBe("inferred");
|
||||
expect(inferred.value.authority).toBe(parent.authority);
|
||||
expect(inferred.value.evidence_links).toEqual([parent.evidence_id]);
|
||||
expect(inferred.value.parameters).toMatchObject({
|
||||
view: {
|
||||
kind: "environment-manifest",
|
||||
template_rules: ["opaque-token"],
|
||||
include_query_parameter_names: true,
|
||||
},
|
||||
});
|
||||
expect(inferred.value.normalized_result).toMatchObject({
|
||||
kind: "environment-manifest",
|
||||
name: "grafana-example",
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
});
|
||||
expect(parseEvidence(inferred.value)).toEqual(inferred.value);
|
||||
const literal = environmentManifestCaptureFixture();
|
||||
literal.artifact.sha256 = null;
|
||||
literal.records = [3, 4, 8].map((ordinal) => literal.records[ordinal]!);
|
||||
const derived = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: analysisViewCaptureEvidence(literal) },
|
||||
view: { kind: "environment-manifest" },
|
||||
});
|
||||
if (!derived.ok) throw derived.error;
|
||||
expect(derived.value.confidence).toBe("derived");
|
||||
expect(derived.value.subject).toBeNull();
|
||||
expect(derived.value.normalized_result).toMatchObject({
|
||||
templated_groups: 0,
|
||||
tools: [{ name: "get_org" }, { name: "get_search" }, { name: "get_user" }],
|
||||
limitations: expect.arrayContaining([
|
||||
"Parent contains an ordinal subset; the manifest covers only retained records.",
|
||||
]),
|
||||
});
|
||||
expect(
|
||||
inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: analysisViewLayoutEvidence() },
|
||||
view: { kind: "environment-manifest" },
|
||||
}),
|
||||
).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisUnsupportedTargetError" },
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
"rejects inline accessors before parsing a %s view",
|
||||
(kind) => {
|
||||
let accessed = false;
|
||||
@@ -498,7 +557,7 @@ it.each(["summary", "capture", "api-surface"] as const)(
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["summary", "capture", "api-surface"] as const)(
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
"rejects nested, hidden and inherited inline accessors for %s",
|
||||
(kind) => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
@@ -575,7 +634,12 @@ it.each([
|
||||
},
|
||||
});
|
||||
if (inherited) Object.setPrototypeOf(array, owner);
|
||||
for (const kind of ["summary", "capture", "api-surface"] as const) {
|
||||
for (const kind of [
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: array } },
|
||||
view: { kind },
|
||||
@@ -606,7 +670,12 @@ it("rejects callable array overrides and subclass species before parsing", () =>
|
||||
Object.defineProperty([], Symbol.iterator, { value: callback }),
|
||||
new CustomArray(),
|
||||
])
|
||||
for (const kind of ["summary", "capture", "api-surface"] as const) {
|
||||
for (const kind of [
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: array } },
|
||||
view: { kind },
|
||||
@@ -619,7 +688,7 @@ it("rejects callable array overrides and subclass species before parsing", () =>
|
||||
expect(invoked).toBe(false);
|
||||
});
|
||||
|
||||
it.each(["summary", "capture", "api-surface"] as const)(
|
||||
it.each(["summary", "capture", "api-surface", "environment-manifest"] as const)(
|
||||
"preserves ordinary inline Evidence for %s",
|
||||
(kind) => {
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
@@ -650,7 +719,12 @@ it("rejects executable constructor values before schema plain-object checks", ()
|
||||
throw new Error("Constructor metadata must remain inert");
|
||||
},
|
||||
});
|
||||
for (const kind of ["summary", "capture", "api-surface"] as const) {
|
||||
for (const kind of [
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
] as const) {
|
||||
expect(
|
||||
inspectAnalysisView({
|
||||
source: {
|
||||
@@ -678,7 +752,12 @@ it("rejects inline proxy values before reflection or parsing can run traps", ()
|
||||
new Proxy(() => undefined, { get: trap }),
|
||||
revoked.proxy,
|
||||
])
|
||||
for (const kind of ["summary", "capture", "api-surface"] as const) {
|
||||
for (const kind of [
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, raw_result: value } },
|
||||
view: { kind },
|
||||
@@ -717,7 +796,12 @@ it("keeps ignored array metadata inert through parsing and projection", () => {
|
||||
for (let index = 0; index < 10_000; index += 1)
|
||||
Object.defineProperty(array, `metadata-${index}`, { get: getter });
|
||||
Object.defineProperty(array, Symbol("metadata"), { get: getter });
|
||||
for (const kind of ["summary", "capture", "api-surface"] as const) {
|
||||
for (const kind of [
|
||||
"summary",
|
||||
"capture",
|
||||
"api-surface",
|
||||
"environment-manifest",
|
||||
] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind },
|
||||
@@ -763,15 +847,16 @@ it("keeps inline-envelope admission live alongside capture validation and groupi
|
||||
it("admits raw_result and parameters before authenticating a portable capture", () => {
|
||||
const dense = analysisViewObjectDenseCapture();
|
||||
const parent = analysisViewCaptureEvidence();
|
||||
for (const field of ["raw_result", "parameters"] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, [field]: dense } },
|
||||
view: { kind: "api-surface" },
|
||||
});
|
||||
// Refuse expansion before hashing can report the deliberately stale ID.
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
}
|
||||
for (const field of ["raw_result", "parameters"] as const)
|
||||
for (const kind of ["api-surface", "environment-manifest"] as const) {
|
||||
const result = inspectAnalysisView({
|
||||
source: { kind: "inline", evidence: { ...parent, [field]: dense } },
|
||||
view: { kind },
|
||||
});
|
||||
// Refuse expansion before hashing can report the deliberately stale ID.
|
||||
expect(result).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
@@ -27,6 +27,7 @@ import {
|
||||
} from "../../domain/immutableJson.js";
|
||||
import { err, ok, type Result } from "../../domain/result.js";
|
||||
import { WebNetworkApiSurfaceAccumulator } from "../../domain/analysisView/WebNetworkApiSurfaceAccumulator.js";
|
||||
import { EnvironmentManifestAccumulator } from "../../domain/analysisView/EnvironmentManifestAccumulator.js";
|
||||
import { admitAnalysisViewData } from "../../domain/analysisView/analysisViewDataAdmission.js";
|
||||
|
||||
/** Authenticate a selected-view request and project already completed Evidence. */
|
||||
@@ -52,13 +53,22 @@ export const inspectAnalysisViewValidated = (
|
||||
input: InspectAnalysisViewInput,
|
||||
lookup?: EvidenceLookup,
|
||||
): Result<Evidence, AnalysisError> => {
|
||||
const accumulator =
|
||||
const apiSurfaceAccumulator =
|
||||
input.view.kind === "api-surface"
|
||||
? new WebNetworkApiSurfaceAccumulator(
|
||||
undefined,
|
||||
input.view.include_query_parameter_names === true,
|
||||
)
|
||||
: undefined;
|
||||
const manifestAccumulator =
|
||||
input.view.kind === "environment-manifest"
|
||||
? new EnvironmentManifestAccumulator(
|
||||
undefined,
|
||||
input.view.include_query_parameter_names,
|
||||
input.view.template_rules,
|
||||
)
|
||||
: undefined;
|
||||
const accumulator = apiSurfaceAccumulator ?? manifestAccumulator;
|
||||
if (input.source.kind === "inline") {
|
||||
const admitted =
|
||||
accumulator === undefined
|
||||
@@ -90,7 +100,8 @@ export const inspectAnalysisViewValidated = (
|
||||
},
|
||||
},
|
||||
input.view,
|
||||
accumulator,
|
||||
apiSurfaceAccumulator,
|
||||
manifestAccumulator,
|
||||
);
|
||||
if (!projected.ok) return projected;
|
||||
if (
|
||||
@@ -223,8 +234,10 @@ const createAnalysisViewEvidence = (
|
||||
result: jsonValueSchema.parse(result),
|
||||
rawResult: null,
|
||||
confidence:
|
||||
// Paging follows the full grouping, so any templated group makes every page inferred.
|
||||
result.kind === "api-surface" && result.templated_groups > 0
|
||||
// Grouping is whole-capture, so any templated group or parameter makes the result inferred.
|
||||
(result.kind === "api-surface" ||
|
||||
result.kind === "environment-manifest") &&
|
||||
result.templated_groups > 0
|
||||
? "inferred"
|
||||
: "derived",
|
||||
authority: parent.authority,
|
||||
|
||||
@@ -37,13 +37,14 @@ it("advertises only that operation's summary from analysis detail outputs", () =
|
||||
z.toJSONSchema(toolContract(name).outputSchema),
|
||||
);
|
||||
expect(schema).not.toContain("api-surface");
|
||||
expect(schema).not.toContain("environment-manifest");
|
||||
expect(schema).not.toContain("inspect_web_network_capture");
|
||||
}
|
||||
expect(
|
||||
JSON.stringify(
|
||||
z.toJSONSchema(toolContract("inspect_analysis_view").outputSchema),
|
||||
),
|
||||
).toContain("api-surface");
|
||||
const viewSchema = JSON.stringify(
|
||||
z.toJSONSchema(toolContract("inspect_analysis_view").outputSchema),
|
||||
);
|
||||
expect(viewSchema).toContain("api-surface");
|
||||
expect(viewSchema).toContain("environment-manifest");
|
||||
|
||||
const layout = projectSummary(analysisViewLayoutEvidence());
|
||||
const javascript = projectSummary(analysisViewJavaScriptEvidence());
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
/** Generated from three environmentManifestHar records (dashboard read, folder create, avatar asset) through the HAR decoder and view service. */
|
||||
export const ENVIRONMENT_MANIFEST_EXAMPLE = {
|
||||
input: {
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id:
|
||||
"ev_3a9d300b14dc39c182f1ad6438e84870eed21da719c550bb9661958f69ce6277",
|
||||
},
|
||||
view: {
|
||||
kind: "environment-manifest",
|
||||
// Contract examples carry mutable JSON values; the tuple stays literal-typed.
|
||||
template_rules: ["after-uid-literal"] as ["after-uid-literal"],
|
||||
},
|
||||
},
|
||||
output: {
|
||||
parent_evidence_id:
|
||||
"ev_3a9d300b14dc39c182f1ad6438e84870eed21da719c550bb9661958f69ce6277",
|
||||
parent_operation: "inspect_web_network_capture",
|
||||
parent_digest:
|
||||
"3a9d300b14dc39c182f1ad6438e84870eed21da719c550bb9661958f69ce6277",
|
||||
view_digest:
|
||||
"c5c7ea379b9ec8cbb1b6f6d2a2b37d251782d68ce3610166cb7112f830b9c6c3",
|
||||
artifact: {
|
||||
path: "/fixtures/environment-manifest.har",
|
||||
sha256:
|
||||
"5bee3e93d69fa5f8650737417615da7fda09f1d8f3e39c1fae8ee69fbcb03420",
|
||||
},
|
||||
coverage: {
|
||||
status: "complete-within-view",
|
||||
examined: 2,
|
||||
total: 2,
|
||||
next_offset: null,
|
||||
exhausted: true,
|
||||
},
|
||||
limitations: [
|
||||
"Body shapes list top-level JSON key types from one recorded request each; they do not establish required fields or a complete schema.",
|
||||
"HAR text is producer-decoded Unicode. Original body bytes remain unknown unless content explicitly uses valid base64; declared sizes are preserved independently of retained bytes. mitmproxy 12.2.3 SaveHar may report null optional postData.text; that null is retained and omitted only from the upstream validation copy.",
|
||||
"Observed query parameter names do not establish required parameters, value types or a complete API schema.",
|
||||
"Opt-in template rules after-uid-literal merge recorded literals by shape; merged parameters may conflate distinct resources.",
|
||||
"Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.",
|
||||
],
|
||||
unknowns: [
|
||||
{
|
||||
kind: "authorization-model",
|
||||
statement:
|
||||
"Authorization headers and cookies are structurally excluded from captures; the manifest states no authorization model.",
|
||||
},
|
||||
{
|
||||
kind: "response-semantics",
|
||||
statement:
|
||||
"Recorded statuses and bodies are retained only as pointers to parent records; the manifest states no response semantics.",
|
||||
},
|
||||
{
|
||||
kind: "state-transition",
|
||||
statement:
|
||||
"Recorded exchanges are independent observations; the manifest states no ordering, side effects or state transitions between tools.",
|
||||
},
|
||||
],
|
||||
origin: "http://grafana.example",
|
||||
name: "grafana-example",
|
||||
tools: [
|
||||
{
|
||||
name: "get_dashboards_by_uid",
|
||||
method: "GET",
|
||||
path_template: "/api/dashboards/uid/{uid}",
|
||||
api_surface_template: "/api/dashboards/uid/acb4v6",
|
||||
params: [
|
||||
{
|
||||
index: 3,
|
||||
name: "uid",
|
||||
rule: "after-uid-literal",
|
||||
basis: "inferred",
|
||||
observed_values: ["acb4v6"],
|
||||
},
|
||||
],
|
||||
query_parameter_names: [],
|
||||
body_shape: null,
|
||||
statuses: [200],
|
||||
content_types: ["application/json"],
|
||||
record_ordinals: [0],
|
||||
},
|
||||
{
|
||||
name: "post_folders",
|
||||
method: "POST",
|
||||
path_template: "/api/folders",
|
||||
api_surface_template: "/api/folders",
|
||||
params: [],
|
||||
query_parameter_names: [],
|
||||
body_shape: {
|
||||
type: "object",
|
||||
properties: {
|
||||
title: "string",
|
||||
},
|
||||
},
|
||||
statuses: [200],
|
||||
content_types: ["application/json"],
|
||||
record_ordinals: [1],
|
||||
},
|
||||
],
|
||||
records: [
|
||||
{
|
||||
ordinal: 0,
|
||||
method: "GET",
|
||||
path: "/api/dashboards/uid/acb4v6",
|
||||
status: 200,
|
||||
content_type: "application/json",
|
||||
},
|
||||
{
|
||||
ordinal: 1,
|
||||
method: "POST",
|
||||
path: "/api/folders",
|
||||
status: 200,
|
||||
content_type: "application/json",
|
||||
},
|
||||
],
|
||||
excluded: [
|
||||
{
|
||||
method: "GET",
|
||||
origin: "http://grafana.example",
|
||||
path_template: "/avatar/{hex}",
|
||||
reason: "ui-page-or-asset",
|
||||
},
|
||||
],
|
||||
templated_groups: 2,
|
||||
kind: "environment-manifest",
|
||||
view: {
|
||||
kind: "environment-manifest",
|
||||
template_rules: ["after-uid-literal"],
|
||||
include_query_parameter_names: true,
|
||||
},
|
||||
},
|
||||
} as const;
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,122 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { environmentManifestCaptureFixture } from "../../../tests/fixtures/analysisView.js";
|
||||
import type { WebNetworkCaptureRecord } from "../webNetworkCapture.js";
|
||||
import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
|
||||
const sample = (
|
||||
ordinal: number,
|
||||
path: string,
|
||||
postText?: string,
|
||||
): WebNetworkCaptureRecord => ({
|
||||
...environmentManifestCaptureFixture().records[0]!,
|
||||
ordinal,
|
||||
limitations: [],
|
||||
reported: {
|
||||
request: {
|
||||
method: postText === undefined ? "GET" : "POST",
|
||||
url: `https://manifest.test${path}`,
|
||||
...(postText === undefined
|
||||
? {}
|
||||
: { postData: { mimeType: "application/json", text: postText } }),
|
||||
},
|
||||
response: { status: 200, content: { mimeType: "application/json" } },
|
||||
},
|
||||
});
|
||||
|
||||
const finished = (accumulator: EnvironmentManifestAccumulator) => {
|
||||
const result = accumulator.finish();
|
||||
if (!result.ok) throw result.error;
|
||||
return result.value;
|
||||
};
|
||||
|
||||
it("refuses a body parse that exceeds the budget without retaining a partial tool", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(32 * 1024);
|
||||
expect(accumulator.add(sample(0, "/api/items")).ok).toBe(true);
|
||||
const before = finished(accumulator);
|
||||
const body = `{"name":"${"x".repeat(2_000)}"}`;
|
||||
expect(accumulator.add(sample(1, "/api/items", body))).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
expect(finished(accumulator)).toEqual(before);
|
||||
expect(accumulator.add(sample(2, "/api/items", '{"name":"n"}')).ok).toBe(
|
||||
true,
|
||||
);
|
||||
expect(finished(accumulator).tools.map((tool) => tool.body_shape)).toEqual([
|
||||
null,
|
||||
{ type: "object", properties: { name: "string" } },
|
||||
]);
|
||||
});
|
||||
|
||||
it("charges every retained record pointer so many records exhaust a small budget", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(64 * 1024);
|
||||
let accepted = 0;
|
||||
for (; accepted < 1000; accepted += 1) {
|
||||
const result = accumulator.add(
|
||||
sample(accepted, `/api/route${String(accepted)}`),
|
||||
);
|
||||
if (result.ok) continue;
|
||||
expect(result.error).toMatchObject({
|
||||
_tag: "AnalysisResourceConstraintError",
|
||||
});
|
||||
break;
|
||||
}
|
||||
expect(accepted).toBeGreaterThan(0);
|
||||
expect(accepted).toBeLessThan(1000);
|
||||
const manifest = finished(accumulator);
|
||||
expect(manifest.records).toHaveLength(accepted);
|
||||
expect(manifest.tools).toHaveLength(accepted);
|
||||
expect(manifest.coverage.total).toBe(accepted);
|
||||
});
|
||||
|
||||
it("retains repeated observed values, statuses and query names once across many samples", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(1024 * 1024);
|
||||
for (let ordinal = 0; ordinal < 1000; ordinal += 1)
|
||||
expect(
|
||||
accumulator.add(sample(ordinal, "/api/items/7?page=1&page=2")).ok,
|
||||
).toBe(true);
|
||||
const manifest = finished(accumulator);
|
||||
expect(manifest.tools).toHaveLength(1);
|
||||
expect(manifest.tools[0]).toMatchObject({
|
||||
params: [{ observed_values: ["7"] }],
|
||||
query_parameter_names: ["page"],
|
||||
statuses: [200],
|
||||
});
|
||||
expect(manifest.tools[0]?.record_ordinals).toHaveLength(1000);
|
||||
expect(manifest.records).toHaveLength(1000);
|
||||
});
|
||||
|
||||
it("charges every literal route part so a path of many short segments fails closed", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(1024 * 1024);
|
||||
expect(accumulator.add(sample(0, "/a".repeat(100))).ok).toBe(true);
|
||||
expect(accumulator.add(sample(1, "/a".repeat(5_000)))).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
expect(finished(accumulator).tools).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("charges a request media type before scanning it so an oversized one fails closed", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(64 * 1024);
|
||||
const record = sample(0, "/api/items", "{}");
|
||||
(
|
||||
record.reported as { request: { postData: { mimeType: string } } }
|
||||
).request.postData.mimeType = `application/json${"x".repeat(100_000)}`;
|
||||
expect(accumulator.add(record)).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
expect(finished(accumulator).tools).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("charges the group key before building it so an oversized request method fails closed", () => {
|
||||
const accumulator = new EnvironmentManifestAccumulator(64 * 1024);
|
||||
const record = sample(0, "/api/items");
|
||||
(record.reported as { request: { method: string } }).request.method =
|
||||
`GET${"X".repeat(100_000)}`;
|
||||
expect(accumulator.add(record)).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "AnalysisResourceConstraintError", resource: "memory" },
|
||||
});
|
||||
expect(finished(accumulator).tools).toHaveLength(0);
|
||||
});
|
||||
@@ -0,0 +1,543 @@
|
||||
import { z } from "zod";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { jsonValueType, parseJsonObject } from "../jsonShape.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
import {
|
||||
compareUnicodeCodePoints as compare,
|
||||
compositeKey,
|
||||
} from "../unicodeCodePointOrder.js";
|
||||
import {
|
||||
WEB_NETWORK_CAPTURE_LIMITS,
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../webNetworkCapture.js";
|
||||
import { completeWithinViewCoverage } from "./analysisViewCoverage.js";
|
||||
import type { UnsignedAnalysisView } from "./analysisView.js";
|
||||
import { WebNetworkProjectionAccumulator } from "./WebNetworkProjectionAccumulator.js";
|
||||
import {
|
||||
applyTemplateRules,
|
||||
compareTools,
|
||||
parameterNames,
|
||||
pathTemplate,
|
||||
SIBLING_VARIANCE_SEGMENT_LIMIT,
|
||||
toolName,
|
||||
toolNameRegistry,
|
||||
type BodyShape,
|
||||
type JsonTypeName,
|
||||
type LiteralPart,
|
||||
type ManifestRule,
|
||||
type TemplateRule,
|
||||
type ToolGroup,
|
||||
} from "./environmentManifestRules.js";
|
||||
import {
|
||||
ENTRY_BYTES,
|
||||
GROUP_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
SEGMENT_BYTES,
|
||||
mediaTypeEssence,
|
||||
readQueryParameterNames,
|
||||
readRecordedRequestLine,
|
||||
routePartCount,
|
||||
scanRouteTemplate,
|
||||
textBytes,
|
||||
type RecordedRequestLine,
|
||||
type RouteTemplate,
|
||||
} from "./webNetworkRouteTemplates.js";
|
||||
|
||||
type Manifest = Extract<UnsignedAnalysisView, { kind: "environment-manifest" }>;
|
||||
type ManifestFields = Pick<
|
||||
Manifest,
|
||||
| "origin"
|
||||
| "name"
|
||||
| "tools"
|
||||
| "records"
|
||||
| "excluded"
|
||||
| "unknowns"
|
||||
| "templated_groups"
|
||||
| "coverage"
|
||||
| "limitations"
|
||||
>;
|
||||
type Tool = Manifest["tools"][number];
|
||||
type Exclusion = Manifest["excluded"][number];
|
||||
type RequestLine = Extract<RecordedRequestLine, { kind: "line" }>;
|
||||
|
||||
interface RecordLine {
|
||||
readonly ordinal: number;
|
||||
readonly method: string;
|
||||
readonly path: string;
|
||||
readonly status: number | null;
|
||||
readonly contentType: string | null;
|
||||
}
|
||||
|
||||
type MutablePart =
|
||||
| LiteralPart
|
||||
| {
|
||||
readonly kind: "param";
|
||||
readonly rule: ManifestRule;
|
||||
readonly values: Set<string>;
|
||||
};
|
||||
|
||||
interface GroupState extends ToolGroup {
|
||||
readonly parts: readonly MutablePart[];
|
||||
readonly positions: number[];
|
||||
readonly statuses: Set<number>;
|
||||
readonly contentTypes: Set<string>;
|
||||
readonly queryNames: Set<string>;
|
||||
readonly apiPath: boolean;
|
||||
jsonLike: boolean;
|
||||
}
|
||||
|
||||
const postDataSchema = z.object({
|
||||
request: z.object({
|
||||
postData: z
|
||||
.object({
|
||||
mimeType: z.unknown().optional(),
|
||||
text: z.unknown().optional(),
|
||||
})
|
||||
.nullable()
|
||||
.optional(),
|
||||
}),
|
||||
});
|
||||
const JSON_LIKE_TEXT = new Set(["text/plain", "text/markdown"]);
|
||||
const ASSET_ESSENCES = new Set([
|
||||
"text/html",
|
||||
"application/xhtml+xml",
|
||||
"text/css",
|
||||
"text/javascript",
|
||||
"application/javascript",
|
||||
"application/x-javascript",
|
||||
"application/wasm",
|
||||
]);
|
||||
const ASSET_PREFIXES = ["image/", "font/", "audio/", "video/"];
|
||||
const ASSET_PATH =
|
||||
/\.(?:m?js|css|map|html?|png|jpe?g|gif|svg|webp|ico|woff2?|ttf|otf)$/iu;
|
||||
const WRITE_METHODS = new Set(["POST", "PUT", "PATCH"]);
|
||||
/** Quotes, separators and brackets a four-part composite key adds around its texts. */
|
||||
const KEY_OVERHEAD_CHARS = 32;
|
||||
const UNKNOWNS: Manifest["unknowns"] = [
|
||||
{
|
||||
kind: "authorization-model",
|
||||
statement:
|
||||
"Authorization headers and cookies are structurally excluded from captures; the manifest states no authorization model.",
|
||||
},
|
||||
{
|
||||
kind: "response-semantics",
|
||||
statement:
|
||||
"Recorded statuses and bodies are retained only as pointers to parent records; the manifest states no response semantics.",
|
||||
},
|
||||
{
|
||||
kind: "state-transition",
|
||||
statement:
|
||||
"Recorded exchanges are independent observations; the manifest states no ordering, side effects or state transitions between tools.",
|
||||
},
|
||||
];
|
||||
|
||||
/** `application/json` or `application/*+json`; a `+json` suffix on another type is not JSON here. */
|
||||
const isJson = (essence: string | null): boolean =>
|
||||
essence === "application/json" ||
|
||||
(essence !== null &&
|
||||
essence.startsWith("application/") &&
|
||||
essence.endsWith("+json"));
|
||||
|
||||
const isJsonLike = (essence: string | null): boolean =>
|
||||
isJson(essence) || (essence !== null && JSON_LIKE_TEXT.has(essence));
|
||||
|
||||
const isApiPath = (path: string): boolean =>
|
||||
path.startsWith("/api/") || path.startsWith("/apis/");
|
||||
|
||||
const isAsset = (group: GroupState): boolean =>
|
||||
[...group.contentTypes].some(
|
||||
(essence) =>
|
||||
ASSET_ESSENCES.has(essence) ||
|
||||
ASSET_PREFIXES.some((prefix) => essence.startsWith(prefix)),
|
||||
) || ASSET_PATH.test(group.api_surface_template);
|
||||
|
||||
/** Kebab-case name from the origin host; dots and ports become hyphens. */
|
||||
const environmentName = (origin: string): string =>
|
||||
new URL(origin).host
|
||||
.toLowerCase()
|
||||
.replaceAll(/[^a-z0-9]+/gu, "-")
|
||||
.replace(/^-/u, "")
|
||||
.replace(/-$/u, "");
|
||||
|
||||
const routeParts = (route: RouteTemplate): MutablePart[] => {
|
||||
const byIndex = new Map(
|
||||
route.segments.map((segment) => [segment.segment_index, segment]),
|
||||
);
|
||||
return route.template
|
||||
.slice(1)
|
||||
.split("/")
|
||||
.map((text, index): MutablePart => {
|
||||
const segment = byIndex.get(index);
|
||||
return segment === undefined
|
||||
? { kind: "literal", value: text }
|
||||
: {
|
||||
kind: "param",
|
||||
rule: segment.rule,
|
||||
values: new Set([segment.value]),
|
||||
};
|
||||
});
|
||||
};
|
||||
|
||||
const hasParam = (parts: readonly MutablePart[]): boolean =>
|
||||
parts.some((part) => part.kind === "param");
|
||||
|
||||
/** Own tool grouping, record pointers and admission for the entire manifest. */
|
||||
export class EnvironmentManifestAccumulator extends WebNetworkProjectionAccumulator {
|
||||
readonly #groups = new Map<string, GroupState>();
|
||||
readonly #records: RecordLine[] = [];
|
||||
readonly #origins = new Map<string, { records: number; jsonLike: number }>();
|
||||
readonly #rules: ReadonlySet<TemplateRule>;
|
||||
#unrepresented = 0;
|
||||
#unrepresentableBodyKey = false;
|
||||
|
||||
constructor(
|
||||
maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
readonly includeQueryParameterNames = true,
|
||||
rules: readonly TemplateRule[] = [],
|
||||
) {
|
||||
super(maximumBytes, "Environment-manifest");
|
||||
this.#rules = new Set(rules);
|
||||
}
|
||||
|
||||
add(record: WebNetworkCaptureRecord): Result<void, AnalysisError> {
|
||||
const limitations = this.addLimitations(record.limitations);
|
||||
if (!limitations.ok) return limitations;
|
||||
try {
|
||||
const line = readRecordedRequestLine(
|
||||
record.reported,
|
||||
this.includeQueryParameterNames,
|
||||
this.scratch,
|
||||
);
|
||||
if (line === null) return this.overBudget();
|
||||
if (line.kind === "excluded") {
|
||||
this.#unrepresented += 1;
|
||||
return ok(undefined);
|
||||
}
|
||||
const route = scanRouteTemplate(line.path, this.scratch);
|
||||
if (route === null) return this.overBudget();
|
||||
const queryNames = readQueryParameterNames(line.query, this.scratch);
|
||||
if (queryNames === null) return this.overBudget();
|
||||
// The key copies the method, origin, template and rules; charge that copy before it exists.
|
||||
if (
|
||||
!this.scratch(
|
||||
textBytes(
|
||||
line.method.length +
|
||||
line.origin.length +
|
||||
route.template.length +
|
||||
route.rules.length +
|
||||
KEY_OVERHEAD_CHARS,
|
||||
),
|
||||
)
|
||||
)
|
||||
return this.overBudget();
|
||||
const key = compositeKey([
|
||||
line.method,
|
||||
line.origin,
|
||||
route.template,
|
||||
route.rules,
|
||||
]);
|
||||
const group = this.#groups.get(key);
|
||||
const bodyShape =
|
||||
group === undefined && WRITE_METHODS.has(line.method)
|
||||
? this.#bodyShape(record.reported)
|
||||
: ok(null);
|
||||
if (!bodyShape.ok) return bodyShape;
|
||||
return this.#append(
|
||||
record.ordinal,
|
||||
line,
|
||||
route,
|
||||
queryNames,
|
||||
{ key, group },
|
||||
bodyShape.value,
|
||||
);
|
||||
} finally {
|
||||
this.releaseTemporary();
|
||||
}
|
||||
}
|
||||
|
||||
/** Read the top-level key types of the group's first recorded JSON-typed request body; nested values are never walked. */
|
||||
#bodyShape(reported: unknown): Result<BodyShape | null, AnalysisError> {
|
||||
const parsed = postDataSchema.safeParse(reported);
|
||||
const body = parsed.success ? parsed.data.request.postData : undefined;
|
||||
if (
|
||||
body === undefined ||
|
||||
body === null ||
|
||||
typeof body.text !== "string" ||
|
||||
typeof body.mimeType !== "string"
|
||||
)
|
||||
return ok(null);
|
||||
// The media type is scanned and lowercased before it is known to be JSON; charge it first.
|
||||
if (!this.scratch(textBytes(body.mimeType.length)))
|
||||
return this.overBudget();
|
||||
if (!isJson(mediaTypeEssence(body.mimeType))) return ok(null);
|
||||
// The parsed tree is linear in the text and coexists with it and the key entries.
|
||||
if (!this.scratch(RECORD_SCRATCH_BYTES + textBytes(body.text.length) * 3))
|
||||
return this.overBudget();
|
||||
const root = parseJsonObject(body.text);
|
||||
if (root === null) return ok(null);
|
||||
const entries: [string, JsonTypeName][] = [];
|
||||
for (const [name, value] of Object.entries(root)) {
|
||||
// JSON parsers and schema clones drop a __proto__ own property; omit it explicitly.
|
||||
if (name === "__proto__") this.#unrepresentableBodyKey = true;
|
||||
else entries.push([name, jsonValueType(value)]);
|
||||
}
|
||||
return ok({ type: "object", properties: Object.fromEntries(entries) });
|
||||
}
|
||||
|
||||
#append(
|
||||
ordinal: number,
|
||||
line: RequestLine,
|
||||
route: RouteTemplate,
|
||||
queryNames: ReadonlySet<string> | undefined,
|
||||
bucket: { readonly key: string; readonly group: GroupState | undefined },
|
||||
bodyShape: BodyShape | null,
|
||||
): Result<void, AnalysisError> {
|
||||
const mime = mediaTypeEssence(line.rawMime);
|
||||
let group = bucket.group;
|
||||
// One retained record pointer, including its projected output.
|
||||
let bytes =
|
||||
ENTRY_BYTES * 2 +
|
||||
textBytes(line.path.length) +
|
||||
textBytes(line.method.length);
|
||||
const origin = this.#origins.get(line.origin);
|
||||
if (origin === undefined)
|
||||
bytes += ENTRY_BYTES * 2 + textBytes(line.origin.length);
|
||||
if (group === undefined) {
|
||||
bytes +=
|
||||
GROUP_BYTES +
|
||||
textBytes(bucket.key.length) +
|
||||
textBytes(line.method.length) +
|
||||
textBytes(line.origin.length) +
|
||||
textBytes(route.template.length) * 2 +
|
||||
textBytes(route.rules.length);
|
||||
// Every part, literal or templated, is one object in the group and in each rule, template and name pass.
|
||||
bytes += ENTRY_BYTES * 4 * routePartCount(route.template);
|
||||
for (const segment of route.segments)
|
||||
bytes += SEGMENT_BYTES + ENTRY_BYTES + textBytes(segment.value.length);
|
||||
if (bodyShape !== null)
|
||||
for (const name of Object.keys(bodyShape.properties))
|
||||
bytes += ENTRY_BYTES + textBytes(name.length);
|
||||
} else {
|
||||
for (const segment of route.segments) {
|
||||
const part = group.parts[segment.segment_index];
|
||||
if (part?.kind === "param" && !part.values.has(segment.value))
|
||||
bytes += ENTRY_BYTES + textBytes(segment.value.length);
|
||||
}
|
||||
}
|
||||
if (line.status !== null && !group?.statuses.has(line.status))
|
||||
bytes += ENTRY_BYTES;
|
||||
if (mime !== null && !group?.contentTypes.has(mime))
|
||||
bytes += ENTRY_BYTES + textBytes(mime.length);
|
||||
for (const name of queryNames ?? [])
|
||||
if (!group?.queryNames.has(name))
|
||||
bytes += ENTRY_BYTES + textBytes(name.length);
|
||||
// Reserve the whole mutation while route scratch is still live.
|
||||
if (!this.retain(bytes)) return this.overBudget();
|
||||
const position = this.#records.length;
|
||||
this.#records.push({
|
||||
ordinal,
|
||||
method: line.method,
|
||||
path: line.path,
|
||||
status: line.status,
|
||||
contentType: mime,
|
||||
});
|
||||
const jsonLike = isJsonLike(mime);
|
||||
const stats = origin ?? { records: 0, jsonLike: 0 };
|
||||
stats.records += 1;
|
||||
if (jsonLike) stats.jsonLike += 1;
|
||||
this.#origins.set(line.origin, stats);
|
||||
if (group === undefined) {
|
||||
group = {
|
||||
method: line.method,
|
||||
origin: line.origin,
|
||||
api_surface_template: route.template,
|
||||
parts: routeParts(route),
|
||||
positions: [],
|
||||
statuses: new Set(),
|
||||
contentTypes: new Set(),
|
||||
queryNames: new Set(),
|
||||
bodyShape,
|
||||
apiPath: isApiPath(route.template),
|
||||
jsonLike: false,
|
||||
};
|
||||
this.#groups.set(bucket.key, group);
|
||||
} else {
|
||||
for (const segment of route.segments) {
|
||||
const part = group.parts[segment.segment_index];
|
||||
if (part?.kind === "param") part.values.add(segment.value);
|
||||
}
|
||||
}
|
||||
group.positions.push(position);
|
||||
group.jsonLike ||= jsonLike;
|
||||
if (line.status !== null) group.statuses.add(line.status);
|
||||
if (mime !== null) group.contentTypes.add(mime);
|
||||
for (const name of queryNames ?? []) group.queryNames.add(name);
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** The origin with the most JSON-like records, then most records, then lowest code points. */
|
||||
#primaryOrigin(): string | null {
|
||||
let best: { origin: string; records: number; jsonLike: number } | null =
|
||||
null;
|
||||
for (const [origin, stats] of this.#origins) {
|
||||
if (
|
||||
best === null ||
|
||||
stats.jsonLike > best.jsonLike ||
|
||||
(stats.jsonLike === best.jsonLike &&
|
||||
(stats.records > best.records ||
|
||||
(stats.records === best.records &&
|
||||
compare(origin, best.origin) < 0)))
|
||||
)
|
||||
best = { origin, ...stats };
|
||||
}
|
||||
return best?.origin ?? null;
|
||||
}
|
||||
|
||||
#tools(groups: readonly ToolGroup[]): Tool[] {
|
||||
const registry = toolNameRegistry();
|
||||
return applyTemplateRules(groups, this.#rules)
|
||||
.map((group) => {
|
||||
const names = parameterNames(group.parts);
|
||||
return { group, names, template: pathTemplate(group.parts, names) };
|
||||
})
|
||||
.sort((left, right) =>
|
||||
compareTools(
|
||||
{ template: left.template, method: left.group.method },
|
||||
{ template: right.template, method: right.group.method },
|
||||
),
|
||||
)
|
||||
.map(({ group, names, template }): Tool => {
|
||||
let next = 0;
|
||||
return {
|
||||
name: toolName(group.method, group.parts, names, registry),
|
||||
method: group.method,
|
||||
path_template: template,
|
||||
api_surface_template: group.api_surface_template,
|
||||
params: group.parts.flatMap((part, index) => {
|
||||
if (part.kind !== "param") return [];
|
||||
const name = names[next] ?? "";
|
||||
next += 1;
|
||||
return [
|
||||
{
|
||||
index,
|
||||
name,
|
||||
rule: part.rule,
|
||||
basis: "inferred" as const,
|
||||
observed_values: [...part.values].sort(compare),
|
||||
},
|
||||
];
|
||||
}),
|
||||
query_parameter_names: [...group.queryNames].sort(compare),
|
||||
body_shape: group.bodyShape,
|
||||
statuses: [...group.statuses].sort((left, right) => left - right),
|
||||
content_types: [...group.contentTypes].sort(compare),
|
||||
record_ordinals: group.positions.map(
|
||||
(position) => this.#records[position]!.ordinal,
|
||||
),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** Project the complete manifest; tools, records and exclusions are sorted deterministically. */
|
||||
finish(): Result<ManifestFields, AnalysisError> {
|
||||
const primary = this.#primaryOrigin();
|
||||
const toolGroups: GroupState[] = [];
|
||||
const exclusions = new Map<string, Exclusion & { templated: boolean }>();
|
||||
for (const group of this.#groups.values()) {
|
||||
const reason: Exclusion["reason"] | null =
|
||||
group.origin !== primary
|
||||
? "other-origin"
|
||||
: group.apiPath || group.jsonLike
|
||||
? null
|
||||
: isAsset(group)
|
||||
? "ui-page-or-asset"
|
||||
: "non-json";
|
||||
if (reason === null) {
|
||||
toolGroups.push(group);
|
||||
continue;
|
||||
}
|
||||
const exclusion = {
|
||||
method: group.method,
|
||||
origin: group.origin,
|
||||
path_template: group.api_surface_template,
|
||||
reason,
|
||||
};
|
||||
// Distinct groups can print the same exclusion row; a templated one keeps the row inferred.
|
||||
const key = compositeKey(Object.values(exclusion));
|
||||
exclusions.set(key, {
|
||||
...exclusion,
|
||||
templated:
|
||||
(exclusions.get(key)?.templated ?? false) || hasParam(group.parts),
|
||||
});
|
||||
}
|
||||
const tools = this.#tools(toolGroups);
|
||||
const cited = [...new Set(tools.flatMap((tool) => tool.record_ordinals))];
|
||||
const positions = new Map(
|
||||
this.#records.map((record, position) => [record.ordinal, position]),
|
||||
);
|
||||
const excluded = [...exclusions.values()].sort(
|
||||
(left, right) =>
|
||||
compare(left.origin, right.origin) ||
|
||||
compare(left.path_template, right.path_template) ||
|
||||
compare(left.method, right.method) ||
|
||||
compare(left.reason, right.reason),
|
||||
);
|
||||
const limitations = this.addLimitations([
|
||||
...(tools.some((tool) => tool.body_shape !== null)
|
||||
? [
|
||||
"Body shapes list top-level JSON key types from one recorded request each; they do not establish required fields or a complete schema.",
|
||||
]
|
||||
: []),
|
||||
...(this.#unrepresentableBodyKey
|
||||
? [
|
||||
"A top-level __proto__ key in a recorded JSON body is omitted from body_shape; it cannot be an own property of the published object.",
|
||||
]
|
||||
: []),
|
||||
...(this.#rules.has("sibling-variance") &&
|
||||
toolGroups.some(
|
||||
(group) => group.parts.length > SIBLING_VARIANCE_SEGMENT_LIMIT,
|
||||
)
|
||||
? [
|
||||
`Routes with more than ${String(SIBLING_VARIANCE_SEGMENT_LIMIT)} segments are not compared by sibling-variance; they keep their grouped templates.`,
|
||||
]
|
||||
: []),
|
||||
...(this.#unrepresented > 0
|
||||
? [
|
||||
`${String(this.#unrepresented)} retained records without a usable request line are not represented in the manifest.`,
|
||||
]
|
||||
: []),
|
||||
...(primary === null
|
||||
? [
|
||||
"No retained record had a parseable origin; the manifest names no primary origin.",
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
if (!limitations.ok) return limitations;
|
||||
return ok({
|
||||
origin: primary ?? "",
|
||||
name: primary === null ? "" : environmentName(primary),
|
||||
tools,
|
||||
records: cited
|
||||
.sort((left, right) => positions.get(left)! - positions.get(right)!)
|
||||
.map((ordinal) => {
|
||||
const record = this.#records[positions.get(ordinal)!]!;
|
||||
return {
|
||||
ordinal: record.ordinal,
|
||||
method: record.method,
|
||||
path: record.path,
|
||||
status: record.status,
|
||||
content_type: record.contentType,
|
||||
};
|
||||
}),
|
||||
excluded: excluded.map(
|
||||
({ templated: _templated, ...exclusion }) => exclusion,
|
||||
),
|
||||
unknowns: UNKNOWNS.map((unknown) => ({ ...unknown })),
|
||||
templated_groups:
|
||||
tools.filter((tool) => tool.params.length > 0).length +
|
||||
excluded.filter((exclusion) => exclusion.templated).length,
|
||||
coverage: completeWithinViewCoverage(tools.length, tools.length),
|
||||
limitations: this.sortedLimitations(),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -1,41 +1,38 @@
|
||||
import { z } from "zod";
|
||||
import { AnalysisResourceConstraintError } from "../analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
import {
|
||||
parseRecordedTransportUrl,
|
||||
parseUrlQueryParameterNames,
|
||||
} from "../recordedTransportUrl.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { admitAnalysisViewData } from "./analysisViewDataAdmission.js";
|
||||
import { ok, type Result } from "../result.js";
|
||||
import {
|
||||
WEB_NETWORK_CAPTURE_LIMITS,
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../webNetworkCapture.js";
|
||||
import { pageViewCoverage } from "./analysisViewCoverage.js";
|
||||
import type { UnsignedAnalysisView } from "./analysisView.js";
|
||||
import { WebNetworkProjectionAccumulator } from "./WebNetworkProjectionAccumulator.js";
|
||||
import {
|
||||
CAPTURE_CONTAINER_BYTES,
|
||||
ENTRY_BYTES,
|
||||
GROUP_BYTES,
|
||||
RECORD_SCRATCH_BYTES,
|
||||
SEGMENT_BYTES,
|
||||
mediaTypeEssence,
|
||||
readQueryParameterNames,
|
||||
readRecordedRequestLine,
|
||||
scanRouteTemplate,
|
||||
textBytes,
|
||||
type RouteRule,
|
||||
type RouteTemplate,
|
||||
} from "./webNetworkRouteTemplates.js";
|
||||
|
||||
type Surface = Extract<UnsignedAnalysisView, { kind: "api-surface" }>;
|
||||
type Group = Surface["groups"][number];
|
||||
type Rule = Group["templated_segments"][number]["rule"];
|
||||
type Exclusion = Surface["excluded_records"][number];
|
||||
interface Segment {
|
||||
readonly segment_index: number;
|
||||
readonly rule: Rule;
|
||||
readonly value: string;
|
||||
}
|
||||
interface Route {
|
||||
readonly template: string;
|
||||
readonly rules: string;
|
||||
readonly segments: readonly Segment[];
|
||||
}
|
||||
interface GroupState {
|
||||
readonly method: string;
|
||||
readonly origin: string;
|
||||
readonly path_template: string;
|
||||
readonly segments: readonly {
|
||||
readonly segment_index: number;
|
||||
readonly rule: Rule;
|
||||
readonly rule: RouteRule;
|
||||
readonly values: Set<string>;
|
||||
}[];
|
||||
readonly ordinals: number[];
|
||||
@@ -45,259 +42,65 @@ interface GroupState {
|
||||
}
|
||||
type Buckets = Map<string, Map<string, Map<string, Map<string, GroupState>>>>;
|
||||
|
||||
const requestSchema = z.object({
|
||||
request: z.object({ url: z.unknown(), method: z.unknown() }),
|
||||
});
|
||||
const responseSchema = z.object({
|
||||
response: z.object({
|
||||
status: z.unknown().optional(),
|
||||
content: z.unknown().optional(),
|
||||
}),
|
||||
});
|
||||
const contentSchema = z.object({ mimeType: z.unknown().optional() });
|
||||
const MEDIA_TYPE_ESSENCE =
|
||||
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
|
||||
|
||||
const mediaTypeEssence = (reported: string): string | null => {
|
||||
const separator = reported.indexOf(";");
|
||||
let start = 0;
|
||||
let end = separator < 0 ? reported.length : separator;
|
||||
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
|
||||
// whitespace cannot cause repeated trailing-regex searches.
|
||||
while (start < end && (reported[start] === " " || reported[start] === "\t"))
|
||||
start += 1;
|
||||
while (
|
||||
end > start &&
|
||||
(reported[end - 1] === " " || reported[end - 1] === "\t")
|
||||
)
|
||||
end -= 1;
|
||||
const essence = reported.slice(start, end);
|
||||
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
|
||||
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
|
||||
};
|
||||
|
||||
// Conservative representation allowances, not an enforced V8 heap limit.
|
||||
// Fixed charges include the accumulator, bucket/set entries, array growth and
|
||||
// page/schema projection. Text includes UTF-16 storage and two worst-case JSON
|
||||
// encodings (transport text and UTF-8 bytes); no serialization is needed to count.
|
||||
const GROUP_BYTES = 2048;
|
||||
const SEGMENT_BYTES = 1024;
|
||||
const ENTRY_BYTES = 128;
|
||||
const RECORD_SCRATCH_BYTES = 512;
|
||||
const SEGMENT_SCRATCH_BYTES = 512;
|
||||
const CAPTURE_CONTAINER_BYTES = 512;
|
||||
const textBytes = (length: number): number => 32 + length * 14;
|
||||
|
||||
const segmentRule = (value: string): Rule | null => {
|
||||
if (/^[0-9]+$/u.test(value)) return "numeric";
|
||||
if (
|
||||
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(
|
||||
value,
|
||||
)
|
||||
)
|
||||
return "uuid";
|
||||
if (/^[0-9a-f]{32,}$/iu.test(value)) return "hex";
|
||||
if (/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/u.test(value)) {
|
||||
const date = new Date(`${value}T00:00:00.000Z`);
|
||||
if (
|
||||
Number.isFinite(date.getTime()) &&
|
||||
date.toISOString().slice(0, 10) === value
|
||||
)
|
||||
return "iso-date";
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
/** Own group identity, deduplication and admission for the entire projection. */
|
||||
export class WebNetworkApiSurfaceAccumulator {
|
||||
export class WebNetworkApiSurfaceAccumulator extends WebNetworkProjectionAccumulator {
|
||||
readonly #buckets: Buckets = new Map();
|
||||
readonly #groups: GroupState[] = [];
|
||||
readonly #excluded: Exclusion[] = [];
|
||||
readonly #limitations = new Set<string>();
|
||||
#unavailableResponse = false;
|
||||
#retainedBytes = 0;
|
||||
#temporaryBytes = 0;
|
||||
|
||||
constructor(
|
||||
readonly maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
maximumBytes = WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
readonly includeQueryParameterNames = false,
|
||||
) {}
|
||||
|
||||
#retain(bytes: number): boolean {
|
||||
if (this.#retainedBytes + this.#temporaryBytes + bytes > this.maximumBytes)
|
||||
return false;
|
||||
this.#retainedBytes += bytes;
|
||||
return true;
|
||||
}
|
||||
|
||||
#temporary(bytes: number): boolean {
|
||||
if (this.#retainedBytes + this.#temporaryBytes + bytes > this.maximumBytes)
|
||||
return false;
|
||||
this.#temporaryBytes += bytes;
|
||||
return true;
|
||||
}
|
||||
|
||||
#overBudget(): Result<never, AnalysisError> {
|
||||
return err(
|
||||
new AnalysisResourceConstraintError(
|
||||
"inspect_analysis_view",
|
||||
"memory",
|
||||
"API-surface retained state and temporary work exceed the projection representation budget.",
|
||||
{ output_bytes: this.maximumBytes },
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
/** Reserve schema output and validation scratch while the borrowed parent stays live. */
|
||||
admitCaptureClone(value: unknown): Result<void, AnalysisError> {
|
||||
return admitAnalysisViewData(value, false, (bytes) =>
|
||||
this.#retain(bytes) ? ok(undefined) : this.#overBudget(),
|
||||
);
|
||||
}
|
||||
|
||||
/** Admit the complete portable envelope before cloning or authenticating it. */
|
||||
admitInlineEvidence(value: unknown): Result<void, AnalysisError> {
|
||||
return admitAnalysisViewData(value, true, (bytes) =>
|
||||
this.#retain(bytes) ? ok(undefined) : this.#overBudget(),
|
||||
);
|
||||
}
|
||||
|
||||
addLimitations(limitations: readonly string[]): Result<void, AnalysisError> {
|
||||
for (const limitation of limitations) {
|
||||
if (this.#limitations.has(limitation)) continue;
|
||||
if (!this.#retain(ENTRY_BYTES + textBytes(limitation.length)))
|
||||
return this.#overBudget();
|
||||
this.#limitations.add(limitation);
|
||||
}
|
||||
return ok(undefined);
|
||||
) {
|
||||
super(maximumBytes, "API-surface");
|
||||
}
|
||||
|
||||
#exclude(
|
||||
ordinal: number,
|
||||
reason: Exclusion["reason"],
|
||||
): Result<void, AnalysisError> {
|
||||
if (!this.#retain(ENTRY_BYTES + RECORD_SCRATCH_BYTES))
|
||||
return this.#overBudget();
|
||||
if (!this.retain(ENTRY_BYTES + RECORD_SCRATCH_BYTES))
|
||||
return this.overBudget();
|
||||
this.#excluded.push({ ordinal, reason });
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** Scan literal spans; only matched segments need descriptors or template parts. */
|
||||
#route(path: string): Result<Route, AnalysisError> {
|
||||
const segments: Segment[] = [];
|
||||
const parts: string[] = [];
|
||||
const rules: string[] = [];
|
||||
let templateLength = path.length;
|
||||
let rulesLength = 0;
|
||||
let copied = 0;
|
||||
let index = 0;
|
||||
let start = 1;
|
||||
for (;;) {
|
||||
const slash = path.indexOf("/", start);
|
||||
const end = slash < 0 ? path.length : slash;
|
||||
const value = path.slice(start, end);
|
||||
const rule = segmentRule(value);
|
||||
if (rule !== null) {
|
||||
if (!this.#temporary(SEGMENT_SCRATCH_BYTES)) return this.#overBudget();
|
||||
const placeholder = `{${rule}}`;
|
||||
const identity = `${index}:${rule};`;
|
||||
segments.push({ segment_index: index, rule, value });
|
||||
parts.push(path.slice(copied, start), placeholder);
|
||||
rules.push(identity);
|
||||
templateLength += placeholder.length - value.length;
|
||||
rulesLength += identity.length;
|
||||
copied = end;
|
||||
}
|
||||
if (slash < 0) break;
|
||||
start = slash + 1;
|
||||
index += 1;
|
||||
}
|
||||
if (segments.length === 0)
|
||||
return ok({ template: path, rules: "", segments });
|
||||
if (!this.#temporary(textBytes(templateLength) + textBytes(rulesLength)))
|
||||
return this.#overBudget();
|
||||
parts.push(path.slice(copied));
|
||||
return ok({ template: parts.join(""), rules: rules.join(""), segments });
|
||||
}
|
||||
|
||||
add(record: WebNetworkCaptureRecord): Result<void, AnalysisError> {
|
||||
const limitations = this.addLimitations(record.limitations);
|
||||
if (!limitations.ok) return limitations;
|
||||
const request = requestSchema.safeParse(record.reported);
|
||||
if (!request.success)
|
||||
return this.#exclude(record.ordinal, "request-unavailable");
|
||||
const { url, method } = request.data.request;
|
||||
if (url === null) return this.#exclude(record.ordinal, "url-redacted");
|
||||
if (typeof url !== "string")
|
||||
return this.#exclude(record.ordinal, "url-unparseable");
|
||||
// Admission precedes URL parsing, text normalization and path materialization.
|
||||
if (!this.#temporary(RECORD_SCRATCH_BYTES + textBytes(url.length)))
|
||||
return this.#overBudget();
|
||||
try {
|
||||
const parsed = parseRecordedTransportUrl(url, {
|
||||
includeQuery: this.includeQueryParameterNames,
|
||||
});
|
||||
if (parsed.origin === null)
|
||||
return this.#exclude(record.ordinal, "url-unparseable");
|
||||
if (typeof method !== "string" || method.length === 0)
|
||||
return this.#exclude(record.ordinal, "method-unavailable");
|
||||
if (parsed.path === null)
|
||||
return this.#exclude(record.ordinal, "url-unparseable");
|
||||
const response = responseSchema.safeParse(record.reported);
|
||||
const status = response.success
|
||||
? response.data.response.status
|
||||
: undefined;
|
||||
const content = contentSchema.safeParse(
|
||||
response.success ? response.data.response.content : undefined,
|
||||
const line = readRecordedRequestLine(
|
||||
record.reported,
|
||||
this.includeQueryParameterNames,
|
||||
this.scratch,
|
||||
);
|
||||
const mime = content.success ? content.data.mimeType : undefined;
|
||||
const rawMime = typeof mime === "string" ? mime : "";
|
||||
if (!this.#temporary(textBytes(rawMime.length)))
|
||||
return this.#overBudget();
|
||||
const route = this.#route(parsed.path);
|
||||
if (!route.ok) return route;
|
||||
const queryNames = this.#queryNames(parsed.query);
|
||||
if (!queryNames.ok) return queryNames;
|
||||
if (line === null) return this.overBudget();
|
||||
if (line.kind === "excluded")
|
||||
return this.#exclude(record.ordinal, line.reason);
|
||||
const route = scanRouteTemplate(line.path, this.scratch);
|
||||
if (route === null) return this.overBudget();
|
||||
const queryNames = readQueryParameterNames(line.query, this.scratch);
|
||||
if (queryNames === null) return this.overBudget();
|
||||
return this.#append(
|
||||
record.ordinal,
|
||||
method,
|
||||
parsed.origin,
|
||||
route.value,
|
||||
typeof status === "number" && status !== 0 ? status : null,
|
||||
mediaTypeEssence(rawMime),
|
||||
queryNames.value,
|
||||
line.method,
|
||||
line.origin,
|
||||
route,
|
||||
line.status,
|
||||
mediaTypeEssence(line.rawMime),
|
||||
queryNames,
|
||||
);
|
||||
} finally {
|
||||
// Per-record strings and descriptors never become cumulative scratch.
|
||||
this.#temporaryBytes = 0;
|
||||
this.releaseTemporary();
|
||||
}
|
||||
}
|
||||
|
||||
#queryNames(
|
||||
query: string | undefined,
|
||||
): Result<ReadonlySet<string> | undefined, AnalysisError> {
|
||||
if (query === undefined) return ok(undefined);
|
||||
// Reserve form-parser tuples, decoded strings, the iterator and the name
|
||||
// set before URLSearchParams materializes them. Delimiter count is an upper
|
||||
// bound: empty query pairs are ignored by the WHATWG parser.
|
||||
let pairs = query.length > 1 ? 1 : 0;
|
||||
for (const character of query) if (character === "&") pairs += 1;
|
||||
if (
|
||||
!this.#temporary(
|
||||
RECORD_SCRATCH_BYTES +
|
||||
textBytes(query.length) +
|
||||
pairs * ENTRY_BYTES * 4,
|
||||
)
|
||||
)
|
||||
return this.#overBudget();
|
||||
return ok(new Set(parseUrlQueryParameterNames(query)));
|
||||
}
|
||||
|
||||
#append(
|
||||
ordinal: number,
|
||||
method: string,
|
||||
origin: string,
|
||||
route: Route,
|
||||
route: RouteTemplate,
|
||||
status: number | null,
|
||||
mime: string | null,
|
||||
queryNames: ReadonlySet<string> | undefined,
|
||||
@@ -336,7 +139,7 @@ export class WebNetworkApiSurfaceAccumulator {
|
||||
}
|
||||
// Reserve the whole mutation while route scratch is still live. No sets,
|
||||
// map entries, arrays or new observed values are retained before admission.
|
||||
if (!this.#retain(bytes)) return this.#overBudget();
|
||||
if (!this.retain(bytes)) return this.overBudget();
|
||||
if (group === undefined) {
|
||||
group = {
|
||||
method,
|
||||
@@ -437,7 +240,7 @@ export class WebNetworkApiSurfaceAccumulator {
|
||||
0,
|
||||
),
|
||||
excluded_records: [...this.#excluded],
|
||||
limitations: [...this.#limitations].sort(compare),
|
||||
limitations: this.sortedLimitations(),
|
||||
unknowns: this.#unavailableResponse
|
||||
? [
|
||||
"Some retained records have unavailable status or content type values.",
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
import { z } from "zod";
|
||||
import {
|
||||
AnalysisResourceConstraintError,
|
||||
AnalysisUnsupportedTargetError,
|
||||
} from "../analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { compareUnicodeCodePoints as compare } from "../unicodeCodePointOrder.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import {
|
||||
webNetworkCaptureSchema,
|
||||
type WebNetworkCapture,
|
||||
} from "../webNetworkCapture.js";
|
||||
import { admitAnalysisViewData } from "./analysisViewDataAdmission.js";
|
||||
import { analysisViewInputError } from "./analysisViewErrors.js";
|
||||
import type { AnalysisViewParent } from "./analysisView.js";
|
||||
import {
|
||||
ENTRY_BYTES,
|
||||
textBytes,
|
||||
type Reserve,
|
||||
} from "./webNetworkRouteTemplates.js";
|
||||
|
||||
/** Shared retained/temporary byte accounting for views that group retained HAR records. */
|
||||
export abstract class WebNetworkProjectionAccumulator {
|
||||
readonly #limitations = new Set<string>();
|
||||
#retainedBytes = 0;
|
||||
#temporaryBytes = 0;
|
||||
/** Per-record scratch reservation handed to the shared record readers. */
|
||||
protected readonly scratch: Reserve = (bytes) => this.temporary(bytes);
|
||||
|
||||
protected constructor(
|
||||
readonly maximumBytes: number,
|
||||
private readonly budgetSubject: string,
|
||||
) {}
|
||||
|
||||
protected retain(bytes: number): boolean {
|
||||
if (this.#retainedBytes + this.#temporaryBytes + bytes > this.maximumBytes)
|
||||
return false;
|
||||
this.#retainedBytes += bytes;
|
||||
return true;
|
||||
}
|
||||
|
||||
protected temporary(bytes: number): boolean {
|
||||
if (this.#retainedBytes + this.#temporaryBytes + bytes > this.maximumBytes)
|
||||
return false;
|
||||
this.#temporaryBytes += bytes;
|
||||
return true;
|
||||
}
|
||||
|
||||
/** Per-record strings and descriptors never become cumulative scratch. */
|
||||
protected releaseTemporary(): void {
|
||||
this.#temporaryBytes = 0;
|
||||
}
|
||||
|
||||
protected overBudget(): Result<never, AnalysisError> {
|
||||
return err(
|
||||
new AnalysisResourceConstraintError(
|
||||
"inspect_analysis_view",
|
||||
"memory",
|
||||
`${this.budgetSubject} retained state and temporary work exceed the projection representation budget.`,
|
||||
{ output_bytes: this.maximumBytes },
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
/** Reserve schema output and validation scratch while the borrowed parent stays live. */
|
||||
admitCaptureClone(value: unknown): Result<void, AnalysisError> {
|
||||
return admitAnalysisViewData(value, false, (bytes) =>
|
||||
this.retain(bytes) ? ok(undefined) : this.overBudget(),
|
||||
);
|
||||
}
|
||||
|
||||
/** Admit the complete portable envelope before cloning or authenticating it. */
|
||||
admitInlineEvidence(value: unknown): Result<void, AnalysisError> {
|
||||
return admitAnalysisViewData(value, true, (bytes) =>
|
||||
this.retain(bytes) ? ok(undefined) : this.overBudget(),
|
||||
);
|
||||
}
|
||||
|
||||
addLimitations(limitations: readonly string[]): Result<void, AnalysisError> {
|
||||
for (const limitation of limitations) {
|
||||
if (this.#limitations.has(limitation)) continue;
|
||||
if (!this.retain(ENTRY_BYTES + textBytes(limitation.length)))
|
||||
return this.overBudget();
|
||||
this.#limitations.add(limitation);
|
||||
}
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
protected sortedLimitations(): string[] {
|
||||
return [...this.#limitations].sort(compare);
|
||||
}
|
||||
}
|
||||
|
||||
/** Authenticate and gate a retained HAR parent before a grouping view reads any record. */
|
||||
export const admitHarCaptureParent = (
|
||||
parent: AnalysisViewParent,
|
||||
viewKind: "api-surface" | "environment-manifest",
|
||||
accumulator: WebNetworkProjectionAccumulator,
|
||||
): Result<WebNetworkCapture, AnalysisError> => {
|
||||
const unsupported = (reason: string) =>
|
||||
err(
|
||||
new AnalysisUnsupportedTargetError(
|
||||
"inspect_analysis_view",
|
||||
parent.evidenceId,
|
||||
reason,
|
||||
),
|
||||
);
|
||||
if (parent.operation !== "inspect_web_network_capture")
|
||||
return unsupported(
|
||||
`View ${viewKind} is unsupported for parent operation ${parent.operation}; ${viewKind} requires inspect_web_network_capture.`,
|
||||
);
|
||||
const admitted = accumulator.admitCaptureClone(parent.normalizedResult);
|
||||
if (!admitted.ok) return admitted;
|
||||
const parsed = webNetworkCaptureSchema.safeParse(parent.normalizedResult);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["source"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Parent Evidence result does not match inspect_web_network_capture.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
if (parsed.data.format !== "har")
|
||||
return unsupported(
|
||||
`${viewKind} does not support capture format ${parsed.data.format}.`,
|
||||
);
|
||||
// The manifest cites the report path and digest, so its parent's subject must be exactly them;
|
||||
// the api-surface view keeps the admission its existing callers rely on.
|
||||
const { artifact } = parsed.data;
|
||||
if (
|
||||
viewKind === "environment-manifest" &&
|
||||
artifact.sha256 !== null &&
|
||||
(parent.subject == null ||
|
||||
parent.subject.local_path !== artifact.path ||
|
||||
parent.subject.digest.sha256 !== artifact.sha256)
|
||||
)
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["source", "subject"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Capture parent subject must identify the report's exact path and SHA-256.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
// The view would write path templates it cannot check against declarations
|
||||
// the parent does not retain, so the refusal does not repeat declaration text.
|
||||
const declarations = z
|
||||
.object({ sensitive_value_count: z.number().int().nonnegative() })
|
||||
.safeParse(parent.parameters);
|
||||
if (!declarations.success || declarations.data.sensitive_value_count > 0)
|
||||
return unsupported("sensitive_values_unretained");
|
||||
return ok(parsed.data);
|
||||
};
|
||||
@@ -92,6 +92,12 @@ import {
|
||||
captureSummarySchema,
|
||||
} from "./webNetworkCaptureView.js";
|
||||
import type { WebNetworkApiSurfaceAccumulator } from "./WebNetworkApiSurfaceAccumulator.js";
|
||||
import type { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
import {
|
||||
projectEnvironmentManifestView,
|
||||
environmentManifestFields,
|
||||
environmentManifestViewRequestSchema,
|
||||
} from "./environmentManifestView.js";
|
||||
import {
|
||||
pcapViewRequestSchema,
|
||||
pcapViewPageSchema,
|
||||
@@ -234,6 +240,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
|
||||
managedMembersViewRequestSchema,
|
||||
androidViewRequestSchema,
|
||||
apiSurfaceViewSchema,
|
||||
environmentManifestViewRequestSchema,
|
||||
captureViewRequestSchema,
|
||||
qlogViewRequestSchema,
|
||||
pcapViewRequestSchema,
|
||||
@@ -496,6 +503,13 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
sha256: digestSchema.nullable(),
|
||||
}),
|
||||
}),
|
||||
z.strictObject({
|
||||
...viewResultBase,
|
||||
...environmentManifestFields,
|
||||
kind: z.literal("environment-manifest"),
|
||||
parent_operation: z.literal("inspect_web_network_capture"),
|
||||
view: environmentManifestViewRequestSchema,
|
||||
}),
|
||||
z.strictObject({
|
||||
...summaryViewResultBase,
|
||||
summary: z.union([
|
||||
@@ -670,6 +684,10 @@ export type UnsignedAnalysisView =
|
||||
Extract<AnalysisViewResult, { readonly kind: "api-surface" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "environment-manifest" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "summary" }>,
|
||||
"view_digest"
|
||||
@@ -809,6 +827,7 @@ export const projectAnalysisView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: AnalysisViewRequest,
|
||||
apiSurfaceAccumulator?: WebNetworkApiSurfaceAccumulator,
|
||||
manifestAccumulator?: EnvironmentManifestAccumulator,
|
||||
): Result<AnalysisViewResult, AnalysisError> => {
|
||||
const digest = parentDigest(parent.evidenceId);
|
||||
if (!digestSchema.safeParse(digest).success)
|
||||
@@ -829,6 +848,14 @@ export const projectAnalysisView = (
|
||||
);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (view.kind === "environment-manifest") {
|
||||
const projected = projectEnvironmentManifestView(
|
||||
parent,
|
||||
view,
|
||||
manifestAccumulator,
|
||||
);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
}
|
||||
if (parent.operation === "inspect_web_network_capture") {
|
||||
const projected = projectWebNetworkCaptureView(parent, view);
|
||||
return projected.ok ? ok(sealAnalysisView(projected.value)) : projected;
|
||||
|
||||
@@ -157,7 +157,10 @@ const layoutSummary = (layout: BinaryLayout) => ({
|
||||
export const projectBinaryLayoutView = (
|
||||
parent: AnalysisViewParent,
|
||||
layout: BinaryLayout,
|
||||
view: Exclude<AnalysisViewRequest, { kind: "api-surface" }>,
|
||||
view: Exclude<
|
||||
AnalysisViewRequest,
|
||||
{ kind: "api-surface" | "environment-manifest" }
|
||||
>,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
const shared = parentFields(parent, layout);
|
||||
if (view.kind === "native")
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { toolName, toolNameRegistry } from "./environmentManifestRules.js";
|
||||
|
||||
it("shares suffix work when distinct names truncate to the same suffixed prefix", () => {
|
||||
const registry = toolNameRegistry();
|
||||
const has = vi.spyOn(registry.used, "has");
|
||||
const names: string[] = [];
|
||||
for (let index = 0; index < 256; index += 1) {
|
||||
const literal = "a".repeat(53) + index.toString(36).padStart(3, "0");
|
||||
for (const suffix of ["", ".json"])
|
||||
names.push(
|
||||
toolName(
|
||||
"GET",
|
||||
[{ kind: "literal", value: literal + suffix }],
|
||||
[],
|
||||
registry,
|
||||
),
|
||||
);
|
||||
}
|
||||
expect(new Set(names).size).toBe(512);
|
||||
expect(names.every((name) => name.length <= 60)).toBe(true);
|
||||
// Counts the owner operation directly, independent of host timing or GC.
|
||||
expect(has.mock.calls.length).toBeLessThanOrEqual(names.length * 2);
|
||||
});
|
||||
|
||||
it("keeps free shorter suffixes and skips occupied names across suffix widths", () => {
|
||||
const registry = toolNameRegistry();
|
||||
const stem = "a".repeat(53);
|
||||
const allocate = (literal: string) =>
|
||||
toolName("GET", [{ kind: "literal", value: literal }], [], registry);
|
||||
expect(allocate(`${stem}_10`)).toBe(`get_${stem}_10`);
|
||||
const longCollisions: string[] = [];
|
||||
for (let index = 0; index < 9; index += 1) {
|
||||
const literal = stem + index.toString(36).padStart(3, "0");
|
||||
allocate(literal);
|
||||
longCollisions.push(allocate(`${literal}.json`));
|
||||
}
|
||||
expect(longCollisions[0]).toBe(`get_${stem}0_2`);
|
||||
expect(longCollisions[7]).toBe(`get_${stem}0_9`);
|
||||
expect(longCollisions[8]).toBe(`get_${stem}_11`);
|
||||
expect(allocate(stem)).toBe(`get_${stem}`);
|
||||
for (let suffix = 2; suffix <= 9; suffix += 1)
|
||||
expect(allocate(`${stem}.json`)).toBe(`get_${stem}_${String(suffix)}`);
|
||||
expect(allocate(`${stem}.json`)).toBe(`get_${stem}_12`);
|
||||
});
|
||||
@@ -0,0 +1,445 @@
|
||||
import {
|
||||
compareUnicodeCodePoints as compare,
|
||||
compositeKey,
|
||||
} from "../unicodeCodePointOrder.js";
|
||||
import { ROUTE_RULES, type RouteRule } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
/** Opt-in template rules applied after the four exact format rules. */
|
||||
export const TEMPLATE_RULES = [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance",
|
||||
] as const;
|
||||
export type TemplateRule = (typeof TEMPLATE_RULES)[number];
|
||||
export type ManifestRule = RouteRule | TemplateRule;
|
||||
|
||||
export type JsonTypeName =
|
||||
| "string"
|
||||
| "number"
|
||||
| "boolean"
|
||||
| "object"
|
||||
| "array"
|
||||
| "null";
|
||||
|
||||
/** Value-free top-level JSON key types of one recorded request body. */
|
||||
export interface BodyShape {
|
||||
readonly type: "object";
|
||||
readonly properties: Readonly<Record<string, JsonTypeName>>;
|
||||
}
|
||||
|
||||
export interface LiteralPart {
|
||||
readonly kind: "literal";
|
||||
readonly value: string;
|
||||
}
|
||||
export interface ParamPart {
|
||||
readonly kind: "param";
|
||||
readonly rule: ManifestRule;
|
||||
readonly values: ReadonlySet<string>;
|
||||
}
|
||||
export type RoutePart = LiteralPart | ParamPart;
|
||||
|
||||
/** One candidate tool: a merged request-line group on the primary origin. */
|
||||
export interface ToolGroup {
|
||||
readonly method: string;
|
||||
readonly origin: string;
|
||||
readonly api_surface_template: string;
|
||||
readonly parts: readonly RoutePart[];
|
||||
/** Record positions in admission order; the first is the earliest record. */
|
||||
readonly positions: readonly number[];
|
||||
readonly statuses: ReadonlySet<number>;
|
||||
readonly contentTypes: ReadonlySet<string>;
|
||||
readonly queryNames: ReadonlySet<string>;
|
||||
readonly bodyShape: BodyShape | null;
|
||||
}
|
||||
|
||||
const OPAQUE_TOKEN = /^(?=.*\d)(?=.*[a-z])[a-z0-9]{12,}$/u;
|
||||
const VERSION_TOKEN = /^v\d/u;
|
||||
const STOP_WORDS = new Set([
|
||||
"search",
|
||||
"settings",
|
||||
"errors",
|
||||
"list",
|
||||
"members",
|
||||
"versions",
|
||||
"health",
|
||||
"merged",
|
||||
"usage",
|
||||
"checks",
|
||||
"dto",
|
||||
"markdown",
|
||||
]);
|
||||
const DROPPED_NAME_LITERALS = new Set(["api", "apis", "namespaces", "default"]);
|
||||
const TOOL_NAME_LIMIT = 60;
|
||||
/** Longest route sibling-variance compares; every position rebuilds a key over the whole route. */
|
||||
export const SIBLING_VARIANCE_SEGMENT_LIMIT = 32;
|
||||
const BASE_RULES: ReadonlySet<string> = new Set(ROUTE_RULES);
|
||||
const OPT_IN_RULES: ReadonlySet<string> = new Set(TEMPLATE_RULES);
|
||||
|
||||
const param = (rule: ManifestRule, values: Iterable<string>): ParamPart => ({
|
||||
kind: "param",
|
||||
rule,
|
||||
values: new Set(values),
|
||||
});
|
||||
|
||||
const partKey = (part: RoutePart): unknown =>
|
||||
part.kind === "literal" ? part.value : ["param", part.rule];
|
||||
|
||||
const withParts = (
|
||||
group: ToolGroup,
|
||||
parts: readonly RoutePart[],
|
||||
): ToolGroup => ({
|
||||
...group,
|
||||
parts,
|
||||
});
|
||||
|
||||
const earliest = (groups: readonly ToolGroup[]): ToolGroup =>
|
||||
groups.reduce((best, group) =>
|
||||
(group.positions[0] ?? Infinity) < (best.positions[0] ?? Infinity)
|
||||
? group
|
||||
: best,
|
||||
);
|
||||
|
||||
/** Merge groups around `shape`; literal siblings of its templated positions become observed values. */
|
||||
const mergeGroups = (
|
||||
groups: readonly ToolGroup[],
|
||||
shape = earliest(groups),
|
||||
): ToolGroup => {
|
||||
if (groups.length === 1) return groups[0]!;
|
||||
const primary = earliest(groups);
|
||||
return {
|
||||
method: primary.method,
|
||||
origin: primary.origin,
|
||||
api_surface_template: primary.api_surface_template,
|
||||
parts: shape.parts.map((part, index) =>
|
||||
part.kind === "literal"
|
||||
? part
|
||||
: param(
|
||||
part.rule,
|
||||
groups.flatMap((group) => {
|
||||
const sibling = group.parts[index];
|
||||
if (sibling === undefined) return [];
|
||||
return sibling.kind === "param"
|
||||
? [...sibling.values]
|
||||
: [sibling.value];
|
||||
}),
|
||||
),
|
||||
),
|
||||
positions: groups
|
||||
.flatMap((group) => group.positions)
|
||||
.sort((left, right) => left - right),
|
||||
statuses: new Set(groups.flatMap((group) => [...group.statuses])),
|
||||
contentTypes: new Set(groups.flatMap((group) => [...group.contentTypes])),
|
||||
queryNames: new Set(groups.flatMap((group) => [...group.queryNames])),
|
||||
bodyShape: primary.bodyShape,
|
||||
};
|
||||
};
|
||||
|
||||
const rebucket = (groups: readonly ToolGroup[]): ToolGroup[] => {
|
||||
const buckets = new Map<string, ToolGroup[]>();
|
||||
for (const group of groups) {
|
||||
const key = compositeKey([
|
||||
group.method,
|
||||
group.origin,
|
||||
...group.parts.map(partKey),
|
||||
]);
|
||||
const bucket = buckets.get(key);
|
||||
if (bucket === undefined) buckets.set(key, [group]);
|
||||
else bucket.push(group);
|
||||
}
|
||||
return [...buckets.values()].map((bucket) => mergeGroups(bucket));
|
||||
};
|
||||
|
||||
const templateAfterUidLiteral = (parts: readonly RoutePart[]): RoutePart[] =>
|
||||
parts.map((part, index) => {
|
||||
const previous = parts[index - 1];
|
||||
return part.kind === "literal" &&
|
||||
previous?.kind === "literal" &&
|
||||
previous.value === "uid" &&
|
||||
!STOP_WORDS.has(part.value)
|
||||
? param("after-uid-literal", [part.value])
|
||||
: part;
|
||||
});
|
||||
|
||||
const templateOpaqueTokens = (parts: readonly RoutePart[]): RoutePart[] =>
|
||||
parts.map((part) =>
|
||||
part.kind === "literal" &&
|
||||
OPAQUE_TOKEN.test(part.value) &&
|
||||
!VERSION_TOKEN.test(part.value)
|
||||
? param("opaque-token", [part.value])
|
||||
: part,
|
||||
);
|
||||
|
||||
const literalAt = (group: ToolGroup, position: number): string => {
|
||||
const part = group.parts[position];
|
||||
return part?.kind === "literal" ? part.value : "";
|
||||
};
|
||||
|
||||
/** The siblings whose literal looks like an identifier: each carries a digit or colon, or three or more are hyphenated names under a plural literal. Stop words never qualify. */
|
||||
const identifierLikeSiblings = (
|
||||
literal: readonly ToolGroup[],
|
||||
position: number,
|
||||
): ReadonlySet<ToolGroup> => {
|
||||
const eligible = new Set<ToolGroup>();
|
||||
const previous = literal[0]?.parts[position - 1];
|
||||
const plural = previous?.kind === "literal" && previous.value.endsWith("s");
|
||||
const hyphenated: ToolGroup[] = [];
|
||||
for (const group of literal) {
|
||||
const value = literalAt(group, position);
|
||||
if (STOP_WORDS.has(value)) continue;
|
||||
if (/[\d:]/u.test(value)) eligible.add(group);
|
||||
if (plural && value.includes("-")) hyphenated.push(group);
|
||||
}
|
||||
if (hyphenated.length >= 3)
|
||||
for (const group of hyphenated) eligible.add(group);
|
||||
return eligible;
|
||||
};
|
||||
|
||||
const optInParamAt = (group: ToolGroup, position: number): boolean => {
|
||||
const part = group.parts[position];
|
||||
return part?.kind === "param" && OPT_IN_RULES.has(part.rule);
|
||||
};
|
||||
|
||||
/** Merge the identifier-like literal siblings into one sibling-variance parameter; other literals stay their own tools. */
|
||||
const mergeLiteralSiblings = (
|
||||
literal: readonly ToolGroup[],
|
||||
position: number,
|
||||
): ToolGroup[] => {
|
||||
const eligible = identifierLikeSiblings(literal, position);
|
||||
if (eligible.size < 2) return [...literal];
|
||||
const groups = [...eligible];
|
||||
const first = earliest(groups);
|
||||
const shape = withParts(
|
||||
first,
|
||||
first.parts.map((part, index) =>
|
||||
index === position
|
||||
? param("sibling-variance", [literalAt(first, position)])
|
||||
: part,
|
||||
),
|
||||
);
|
||||
return [
|
||||
mergeGroups(
|
||||
groups.map((group) => (group === first ? shape : group)),
|
||||
shape,
|
||||
),
|
||||
...literal.filter((group) => !eligible.has(group)),
|
||||
];
|
||||
};
|
||||
|
||||
/** Fold literal siblings into an opt-in parameter only; format-rule parameters absorb no literal. */
|
||||
const mergeSiblings = (
|
||||
bucket: readonly ToolGroup[],
|
||||
position: number,
|
||||
): ToolGroup[] => {
|
||||
if (bucket.length < 2) return [...bucket];
|
||||
const templated = bucket.filter(
|
||||
(group) => group.parts[position]?.kind === "param",
|
||||
);
|
||||
const literal = bucket.filter(
|
||||
(group) => group.parts[position]?.kind === "literal",
|
||||
);
|
||||
const targets = templated.filter((group) => optInParamAt(group, position));
|
||||
if (targets.length === 0)
|
||||
return [...templated, ...mergeLiteralSiblings(literal, position)];
|
||||
const target = earliest(targets);
|
||||
const foldable = new Set(
|
||||
literal.filter((group) => !STOP_WORDS.has(literalAt(group, position))),
|
||||
);
|
||||
return [
|
||||
mergeGroups([target, ...foldable], target),
|
||||
...templated.filter((group) => group !== target),
|
||||
...literal.filter((group) => !foldable.has(group)),
|
||||
];
|
||||
};
|
||||
|
||||
/** Merge groups that differ in exactly one identifier-like segment at index two or later. */
|
||||
const mergeSiblingVariance = (groups: readonly ToolGroup[]): ToolGroup[] => {
|
||||
const beyondLimit = groups.filter(
|
||||
(group) => group.parts.length > SIBLING_VARIANCE_SEGMENT_LIMIT,
|
||||
);
|
||||
let current = groups.filter(
|
||||
(group) => group.parts.length <= SIBLING_VARIANCE_SEGMENT_LIMIT,
|
||||
);
|
||||
const longest = Math.max(0, ...current.map((group) => group.parts.length));
|
||||
for (let position = 2; position < longest; position += 1) {
|
||||
const buckets = new Map<string, ToolGroup[]>();
|
||||
const output: ToolGroup[] = [];
|
||||
for (const group of current) {
|
||||
if (position >= group.parts.length) {
|
||||
output.push(group);
|
||||
continue;
|
||||
}
|
||||
const key = compositeKey([
|
||||
group.method,
|
||||
group.origin,
|
||||
group.parts.length,
|
||||
...group.parts.map((part, index) =>
|
||||
index === position ? null : partKey(part),
|
||||
),
|
||||
]);
|
||||
const bucket = buckets.get(key);
|
||||
if (bucket === undefined) buckets.set(key, [group]);
|
||||
else bucket.push(group);
|
||||
}
|
||||
for (const bucket of buckets.values())
|
||||
output.push(...mergeSiblings(bucket, position));
|
||||
current = output;
|
||||
}
|
||||
return [...current, ...beyondLimit];
|
||||
};
|
||||
|
||||
/** Apply the selected opt-in rules in fixed order; every merge is an inference. */
|
||||
export const applyTemplateRules = (
|
||||
groups: readonly ToolGroup[],
|
||||
rules: ReadonlySet<TemplateRule>,
|
||||
): ToolGroup[] => {
|
||||
let current = [...groups];
|
||||
if (rules.has("after-uid-literal"))
|
||||
current = current.map((group) =>
|
||||
withParts(group, templateAfterUidLiteral(group.parts)),
|
||||
);
|
||||
if (rules.has("opaque-token"))
|
||||
current = current.map((group) =>
|
||||
withParts(group, templateOpaqueTokens(group.parts)),
|
||||
);
|
||||
current = rebucket(current);
|
||||
if (rules.has("sibling-variance")) current = mergeSiblingVariance(current);
|
||||
return current;
|
||||
};
|
||||
|
||||
const sanitizeToken = (text: string): string =>
|
||||
text
|
||||
.toLowerCase()
|
||||
.replaceAll(/[^a-z0-9_]+/gu, "_")
|
||||
.replaceAll(/_+/gu, "_")
|
||||
.replace(/^_/u, "")
|
||||
.replace(/_$/u, "");
|
||||
|
||||
const singular = (word: string): string => {
|
||||
if (word.endsWith("ies") && word.length > 3) return `${word.slice(0, -3)}y`;
|
||||
if (word.endsWith("s") && !word.endsWith("ss")) return word.slice(0, -1);
|
||||
return word;
|
||||
};
|
||||
|
||||
const paramBaseName = (parts: readonly RoutePart[], index: number): string => {
|
||||
const part = parts[index];
|
||||
if (part?.kind !== "param") return "";
|
||||
switch (part.rule) {
|
||||
case "numeric":
|
||||
return "id";
|
||||
case "uuid":
|
||||
return "uuid";
|
||||
case "hex":
|
||||
return "hash";
|
||||
case "iso-date":
|
||||
return "date";
|
||||
case "after-uid-literal":
|
||||
case "opaque-token":
|
||||
return "uid";
|
||||
case "sibling-variance": {
|
||||
const previous = parts[index - 1];
|
||||
const stem =
|
||||
previous?.kind === "literal"
|
||||
? sanitizeToken(singular(previous.value))
|
||||
: "";
|
||||
return stem.length > 0 ? `${stem}_id` : "id";
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
/** Parameter names in segment order; repeated names get numeric suffixes. */
|
||||
export const parameterNames = (parts: readonly RoutePart[]): string[] => {
|
||||
const used = new Map<string, number>();
|
||||
return parts.flatMap((part, index) => {
|
||||
if (part.kind !== "param") return [];
|
||||
const base = paramBaseName(parts, index);
|
||||
const count = used.get(base) ?? 0;
|
||||
used.set(base, count + 1);
|
||||
return [count === 0 ? base : `${base}_${String(count + 1)}`];
|
||||
});
|
||||
};
|
||||
|
||||
/** Format-rule placeholders keep their api-surface spelling; opt-in rules use the parameter name. */
|
||||
export const pathTemplate = (
|
||||
parts: readonly RoutePart[],
|
||||
names: readonly string[],
|
||||
): string => {
|
||||
let next = 0;
|
||||
return `/${parts
|
||||
.map((part) => {
|
||||
if (part.kind === "literal") return part.value;
|
||||
const name = names[next] ?? "";
|
||||
next += 1;
|
||||
return BASE_RULES.has(part.rule) ? `{${part.rule}}` : `{${name}}`;
|
||||
})
|
||||
.join("/")}`;
|
||||
};
|
||||
|
||||
/** Names assigned in one manifest and the next suffix to try for each colliding truncated base. */
|
||||
export interface ToolNameRegistry {
|
||||
readonly used: Set<string>;
|
||||
readonly nextSuffix: Map<string, number>;
|
||||
}
|
||||
export const toolNameRegistry = (): ToolNameRegistry => ({
|
||||
used: new Set(),
|
||||
nextSuffix: new Map(),
|
||||
});
|
||||
/** Deterministic tool name from the method and route literals, unique within a manifest. */
|
||||
export const toolName = (
|
||||
method: string,
|
||||
parts: readonly RoutePart[],
|
||||
names: readonly string[],
|
||||
registry: ToolNameRegistry,
|
||||
): string => {
|
||||
const literals = parts.flatMap((part, index) => {
|
||||
if (part.kind !== "literal") return [];
|
||||
const value = part.value;
|
||||
if (
|
||||
value.length === 0 ||
|
||||
DROPPED_NAME_LITERALS.has(value) ||
|
||||
VERSION_TOKEN.test(value) ||
|
||||
(value === "uid" && parts[index + 1]?.kind === "param")
|
||||
)
|
||||
return [];
|
||||
return [value.split(".")[0] ?? ""];
|
||||
});
|
||||
const base =
|
||||
sanitizeToken(
|
||||
[
|
||||
method.toLowerCase(),
|
||||
...literals,
|
||||
...(names.length > 0 ? ["by", ...names] : []),
|
||||
].join("_"),
|
||||
) || "tool";
|
||||
const truncate = (text: string, limit: number): string =>
|
||||
text.slice(0, limit).replace(/_+$/u, "");
|
||||
let candidate = truncate(base, TOOL_NAME_LIMIT);
|
||||
if (registry.used.has(candidate)) {
|
||||
let suffix = 2;
|
||||
for (;;) {
|
||||
const tail = `_${String(suffix)}`;
|
||||
const prefix = truncate(base, TOOL_NAME_LIMIT - tail.length);
|
||||
// Suffixes can truncate distinct full names to the same prefix. Share
|
||||
// progress for that actual candidate family, separately for each width:
|
||||
// a longer suffix can change the prefix and must not skip free short names.
|
||||
const namespace = compositeKey([prefix, tail.length]);
|
||||
const next = registry.nextSuffix.get(namespace) ?? suffix;
|
||||
if (next > suffix) {
|
||||
suffix = next;
|
||||
continue;
|
||||
}
|
||||
candidate = `${prefix}${tail}`;
|
||||
registry.nextSuffix.set(namespace, suffix + 1);
|
||||
if (!registry.used.has(candidate)) break;
|
||||
suffix += 1;
|
||||
}
|
||||
}
|
||||
registry.used.add(candidate);
|
||||
return candidate;
|
||||
};
|
||||
|
||||
/** Sort tools by template then method with locale-independent ordering. */
|
||||
export const compareTools = (
|
||||
left: { readonly template: string; readonly method: string },
|
||||
right: { readonly template: string; readonly method: string },
|
||||
): number =>
|
||||
compare(left.template, right.template) || compare(left.method, right.method);
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,185 @@
|
||||
import { z } from "zod";
|
||||
import { AnalysisUnsupportedTargetError } from "../analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { uniqueSorted } from "../canonicalOrdering.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import type {
|
||||
AnalysisViewParent,
|
||||
AnalysisViewRequest,
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import { EnvironmentManifestAccumulator } from "./EnvironmentManifestAccumulator.js";
|
||||
import { TEMPLATE_RULES } from "./environmentManifestRules.js";
|
||||
import { admitHarCaptureParent } from "./WebNetworkProjectionAccumulator.js";
|
||||
import { ROUTE_RULES } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
export const environmentManifestViewRequestSchema = z.strictObject({
|
||||
kind: z.literal("environment-manifest"),
|
||||
template_rules: z
|
||||
.array(z.enum(TEMPLATE_RULES))
|
||||
.refine((rules) => new Set(rules).size === rules.length, {
|
||||
message: "template_rules must not repeat a rule",
|
||||
})
|
||||
.meta({ uniqueItems: true })
|
||||
.default([])
|
||||
.describe(
|
||||
"Opt-in template rules applied after the four format rules, each listed at most once: after-uid-literal templates the segment following a literal uid, opaque-token templates long mixed alphanumeric segments, sibling-variance merges groups that differ in one identifier-like segment. Literal sibling folding into existing opt-in parameters also requires sibling-variance. Merged parameters are inferences.",
|
||||
),
|
||||
include_query_parameter_names: z
|
||||
.boolean()
|
||||
.default(true)
|
||||
.describe(
|
||||
"Include distinct observed query parameter names per tool; names do not establish required parameters or a complete API schema.",
|
||||
),
|
||||
});
|
||||
|
||||
const manifestRuleSchema = z.enum([...ROUTE_RULES, ...TEMPLATE_RULES]);
|
||||
const jsonTypeNameSchema = z.enum([
|
||||
"string",
|
||||
"number",
|
||||
"boolean",
|
||||
"object",
|
||||
"array",
|
||||
"null",
|
||||
]);
|
||||
const toolSchema = z.strictObject({
|
||||
name: z.string(),
|
||||
method: z.string(),
|
||||
path_template: z.string(),
|
||||
api_surface_template: z
|
||||
.string()
|
||||
.describe("Four-rule template of the tool's earliest record."),
|
||||
params: z.array(
|
||||
z.strictObject({
|
||||
index: z.number().int().nonnegative(),
|
||||
name: z.string(),
|
||||
rule: manifestRuleSchema,
|
||||
basis: z.literal("inferred"),
|
||||
observed_values: z.array(z.string()),
|
||||
}),
|
||||
),
|
||||
query_parameter_names: z.array(z.string()),
|
||||
body_shape: z
|
||||
.strictObject({
|
||||
type: z.literal("object"),
|
||||
properties: z.record(z.string(), jsonTypeNameSchema),
|
||||
})
|
||||
.nullable(),
|
||||
statuses: z.array(z.number()),
|
||||
content_types: z.array(z.string()),
|
||||
record_ordinals: z.array(z.number().int().nonnegative()),
|
||||
});
|
||||
export const environmentManifestFields = {
|
||||
origin: z
|
||||
.string()
|
||||
.describe("Primary origin: the origin with the most JSON-like records."),
|
||||
name: z
|
||||
.string()
|
||||
.describe(
|
||||
"Kebab-case environment name derived from the primary origin host.",
|
||||
),
|
||||
tools: z.array(toolSchema),
|
||||
records: z
|
||||
.array(
|
||||
z.strictObject({
|
||||
ordinal: z.number().int().nonnegative(),
|
||||
method: z.string(),
|
||||
path: z.string(),
|
||||
status: z.number().nullable(),
|
||||
content_type: z.string().nullable(),
|
||||
}),
|
||||
)
|
||||
.describe(
|
||||
"Pointers to the retained parent records cited by tools; no bodies.",
|
||||
),
|
||||
excluded: z.array(
|
||||
z.strictObject({
|
||||
method: z.string(),
|
||||
origin: z.string(),
|
||||
path_template: z.string(),
|
||||
reason: z.enum(["other-origin", "ui-page-or-asset", "non-json"]),
|
||||
}),
|
||||
),
|
||||
unknowns: z.array(
|
||||
z.strictObject({
|
||||
kind: z.enum([
|
||||
"authorization-model",
|
||||
"response-semantics",
|
||||
"state-transition",
|
||||
]),
|
||||
statement: z.string(),
|
||||
}),
|
||||
),
|
||||
templated_groups: z
|
||||
.number()
|
||||
.int()
|
||||
.nonnegative()
|
||||
.describe(
|
||||
"Tools and excluded groups that carry at least one templated segment.",
|
||||
),
|
||||
};
|
||||
|
||||
/** Project retained HAR request lines into a replayable tool surface; templates are inferences. */
|
||||
export const projectEnvironmentManifestView = (
|
||||
parent: AnalysisViewParent,
|
||||
view: AnalysisViewRequest,
|
||||
accumulator = view.kind === "environment-manifest"
|
||||
? new EnvironmentManifestAccumulator(
|
||||
undefined,
|
||||
view.include_query_parameter_names,
|
||||
view.template_rules,
|
||||
)
|
||||
: new EnvironmentManifestAccumulator(),
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (view.kind !== "environment-manifest")
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(
|
||||
"inspect_analysis_view",
|
||||
parent.evidenceId,
|
||||
`View ${view.kind} is unsupported for parent operation ${parent.operation}; environment-manifest requires inspect_web_network_capture.`,
|
||||
),
|
||||
);
|
||||
const admitted = admitHarCaptureParent(
|
||||
parent,
|
||||
"environment-manifest",
|
||||
accumulator,
|
||||
);
|
||||
if (!admitted.ok) return admitted;
|
||||
const capture = admitted.value;
|
||||
const limitations = accumulator.addLimitations([
|
||||
...parent.limitations,
|
||||
...capture.limitations,
|
||||
"Tool templates, parameters and names are inferences from recorded request lines; they do not establish handler identity, URL-space schemas or causality.",
|
||||
...(view.include_query_parameter_names
|
||||
? [
|
||||
"Observed query parameter names do not establish required parameters, value types or a complete API schema.",
|
||||
]
|
||||
: []),
|
||||
...(view.template_rules.length > 0
|
||||
? [
|
||||
`Opt-in template rules ${uniqueSorted(view.template_rules).join(", ")} merge recorded literals by shape; merged parameters may conflate distinct resources.`,
|
||||
]
|
||||
: []),
|
||||
...(capture.records.length < capture.total_records
|
||||
? [
|
||||
"Parent contains an ordinal subset; the manifest covers only retained records.",
|
||||
]
|
||||
: []),
|
||||
]);
|
||||
if (!limitations.ok) return limitations;
|
||||
for (const record of capture.records) {
|
||||
const added = accumulator.add(record);
|
||||
if (!added.ok) return added;
|
||||
}
|
||||
const manifest = accumulator.finish();
|
||||
if (!manifest.ok) return manifest;
|
||||
return ok({
|
||||
kind: "environment-manifest",
|
||||
view,
|
||||
parent_evidence_id: parent.evidenceId,
|
||||
parent_operation: "inspect_web_network_capture",
|
||||
parent_digest: parent.evidenceId.slice(3),
|
||||
artifact: { path: capture.artifact.path, sha256: capture.artifact.sha256 },
|
||||
...manifest.value,
|
||||
});
|
||||
};
|
||||
@@ -307,7 +307,10 @@ const selectModule = (
|
||||
export const projectJavaScriptApplicationView = (
|
||||
parent: AnalysisViewParent,
|
||||
analysis: JavaScriptApplicationAnalysisResult,
|
||||
view: Exclude<AnalysisViewRequest, { kind: "api-surface" }>,
|
||||
view: Exclude<
|
||||
AnalysisViewRequest,
|
||||
{ kind: "api-surface" | "environment-manifest" }
|
||||
>,
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
if (
|
||||
view.kind !== "summary" &&
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
import { analysisViewInputError } from "./analysisViewErrors.js";
|
||||
import { z } from "zod";
|
||||
import { AnalysisUnsupportedTargetError } from "../analysisErrorCore.js";
|
||||
import type { AnalysisError } from "../analysisErrorBase.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { webNetworkCaptureSchema } from "../webNetworkCapture.js";
|
||||
import type {
|
||||
AnalysisViewParent,
|
||||
AnalysisViewRequest,
|
||||
UnsignedAnalysisView,
|
||||
} from "./analysisView.js";
|
||||
import { WebNetworkApiSurfaceAccumulator } from "./WebNetworkApiSurfaceAccumulator.js";
|
||||
import { admitHarCaptureParent } from "./WebNetworkProjectionAccumulator.js";
|
||||
import { ROUTE_RULES } from "./webNetworkRouteTemplates.js";
|
||||
|
||||
export const apiSurfaceViewSchema = z.strictObject({
|
||||
kind: z.literal("api-surface"),
|
||||
@@ -22,7 +22,7 @@ export const apiSurfaceViewSchema = z.strictObject({
|
||||
"Include distinct observed query parameter names and their original record ordinals; names do not establish required parameters or a complete API schema.",
|
||||
),
|
||||
});
|
||||
const ruleSchema = z.enum(["numeric", "uuid", "hex", "iso-date"]);
|
||||
const ruleSchema = z.enum(ROUTE_RULES);
|
||||
const groupSchema = z.strictObject({
|
||||
method: z.string(),
|
||||
origin: z.string(),
|
||||
@@ -85,45 +85,17 @@ export const projectWebNetworkApiSurfaceView = (
|
||||
view.kind === "api-surface" && view.include_query_parameter_names === true,
|
||||
),
|
||||
): Result<UnsignedAnalysisView, AnalysisError> => {
|
||||
const unsupported = (reason: string) =>
|
||||
err(
|
||||
if (view.kind !== "api-surface")
|
||||
return err(
|
||||
new AnalysisUnsupportedTargetError(
|
||||
"inspect_analysis_view",
|
||||
parent.evidenceId,
|
||||
reason,
|
||||
`View ${view.kind} is unsupported for parent operation ${parent.operation}; api-surface requires inspect_web_network_capture.`,
|
||||
),
|
||||
);
|
||||
if (
|
||||
parent.operation !== "inspect_web_network_capture" ||
|
||||
view.kind !== "api-surface"
|
||||
)
|
||||
return unsupported(
|
||||
`View ${view.kind} is unsupported for parent operation ${parent.operation}; api-surface requires inspect_web_network_capture.`,
|
||||
);
|
||||
const admitted = accumulator.admitCaptureClone(parent.normalizedResult);
|
||||
const admitted = admitHarCaptureParent(parent, "api-surface", accumulator);
|
||||
if (!admitted.ok) return admitted;
|
||||
const parsed = webNetworkCaptureSchema.safeParse(parent.normalizedResult);
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisViewInputError([
|
||||
{
|
||||
path: ["source"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Parent Evidence result does not match inspect_web_network_capture.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
const capture = parsed.data;
|
||||
if (capture.format !== "har")
|
||||
return unsupported(
|
||||
`api-surface does not support capture format ${capture.format}.`,
|
||||
);
|
||||
const declarations = z
|
||||
.object({ sensitive_value_count: z.number().int().nonnegative() })
|
||||
.safeParse(parent.parameters);
|
||||
if (!declarations.success || declarations.data.sensitive_value_count > 0)
|
||||
return unsupported("sensitive_values_unretained");
|
||||
const capture = admitted.value;
|
||||
const limitations = accumulator.addLimitations([
|
||||
...parent.limitations,
|
||||
...capture.limitations,
|
||||
|
||||
@@ -437,7 +437,7 @@ export const projectWebNetworkCaptureView = (
|
||||
path: ["view"],
|
||||
reason: "invalid_value",
|
||||
message:
|
||||
"Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface views.",
|
||||
"Historical captures support summary, capture record pages, qlog fact pages and admitted api-surface or environment-manifest views.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
import { z } from "zod";
|
||||
import {
|
||||
parseRecordedTransportUrl,
|
||||
parseUrlQueryParameterNames,
|
||||
} from "../recordedTransportUrl.js";
|
||||
|
||||
/** Exact segment formats shared by the api-surface and environment-manifest views. */
|
||||
export const ROUTE_RULES = ["numeric", "uuid", "hex", "iso-date"] as const;
|
||||
export type RouteRule = (typeof ROUTE_RULES)[number];
|
||||
|
||||
export interface RouteSegment {
|
||||
readonly segment_index: number;
|
||||
readonly rule: RouteRule;
|
||||
readonly value: string;
|
||||
}
|
||||
|
||||
export interface RouteTemplate {
|
||||
readonly template: string;
|
||||
readonly rules: string;
|
||||
readonly segments: readonly RouteSegment[];
|
||||
}
|
||||
|
||||
export type RequestLineExclusion =
|
||||
| "url-redacted"
|
||||
| "url-unparseable"
|
||||
| "request-unavailable"
|
||||
| "method-unavailable";
|
||||
|
||||
/** One retained request line, or the reason it cannot join a grouping view. */
|
||||
export type RecordedRequestLine =
|
||||
| { readonly kind: "excluded"; readonly reason: RequestLineExclusion }
|
||||
| {
|
||||
readonly kind: "line";
|
||||
readonly method: string;
|
||||
readonly origin: string;
|
||||
readonly path: string;
|
||||
readonly query: string | undefined;
|
||||
readonly status: number | null;
|
||||
readonly rawMime: string;
|
||||
};
|
||||
|
||||
/** Temporary-byte reservation; false means the projection budget is exhausted. */
|
||||
export type Reserve = (bytes: number) => boolean;
|
||||
|
||||
const requestSchema = z.object({
|
||||
request: z.object({ url: z.unknown(), method: z.unknown() }),
|
||||
});
|
||||
const responseSchema = z.object({
|
||||
response: z.object({
|
||||
status: z.unknown().optional(),
|
||||
content: z.unknown().optional(),
|
||||
}),
|
||||
});
|
||||
const contentSchema = z.object({ mimeType: z.unknown().optional() });
|
||||
const MEDIA_TYPE_ESSENCE =
|
||||
/^[A-Za-z0-9!#$%&'*+.^_`|~-]+\/[A-Za-z0-9!#$%&'*+.^_`|~-]+$/u;
|
||||
|
||||
// Conservative representation allowances, not an enforced V8 heap limit.
|
||||
// Fixed charges include the accumulator, bucket/set entries, array growth and
|
||||
// page/schema projection. Text includes UTF-16 storage and two worst-case JSON
|
||||
// encodings (transport text and UTF-8 bytes); no serialization is needed to count.
|
||||
export const GROUP_BYTES = 2048;
|
||||
export const SEGMENT_BYTES = 1024;
|
||||
export const ENTRY_BYTES = 128;
|
||||
export const RECORD_SCRATCH_BYTES = 512;
|
||||
export const CAPTURE_CONTAINER_BYTES = 512;
|
||||
const SEGMENT_SCRATCH_BYTES = 512;
|
||||
export const textBytes = (length: number): number => 32 + length * 14;
|
||||
|
||||
/** Lowercase type/subtype without parameters; malformed essences are unknown. */
|
||||
export const mediaTypeEssence = (reported: string): string | null => {
|
||||
const separator = reported.indexOf(";");
|
||||
let start = 0;
|
||||
let end = separator < 0 ? reported.length : separator;
|
||||
// HTTP OWS is SP/HTAB. Scan from both ends so malformed internal runs of
|
||||
// whitespace cannot cause repeated trailing-regex searches.
|
||||
while (start < end && (reported[start] === " " || reported[start] === "\t"))
|
||||
start += 1;
|
||||
while (
|
||||
end > start &&
|
||||
(reported[end - 1] === " " || reported[end - 1] === "\t")
|
||||
)
|
||||
end -= 1;
|
||||
const essence = reported.slice(start, end);
|
||||
// Validate ASCII tokens before case folding can turn Unicode into ASCII.
|
||||
return MEDIA_TYPE_ESSENCE.test(essence) ? essence.toLowerCase() : null;
|
||||
};
|
||||
|
||||
/** Route parts of a template: one more than its separators. */
|
||||
export const routePartCount = (template: string): number => {
|
||||
let count = 1;
|
||||
for (const character of template) if (character === "/") count += 1;
|
||||
return count;
|
||||
};
|
||||
/** The first exact format rule a recorded path segment satisfies, if any. */
|
||||
export const segmentRule = (value: string): RouteRule | null => {
|
||||
if (/^[0-9]+$/u.test(value)) return "numeric";
|
||||
if (
|
||||
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/iu.test(
|
||||
value,
|
||||
)
|
||||
)
|
||||
return "uuid";
|
||||
if (/^[0-9a-f]{32,}$/iu.test(value)) return "hex";
|
||||
if (/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/u.test(value)) {
|
||||
const date = new Date(`${value}T00:00:00.000Z`);
|
||||
if (
|
||||
Number.isFinite(date.getTime()) &&
|
||||
date.toISOString().slice(0, 10) === value
|
||||
)
|
||||
return "iso-date";
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
/** Scan literal spans; only matched segments need descriptors or template parts. */
|
||||
export const scanRouteTemplate = (
|
||||
path: string,
|
||||
temporary: Reserve,
|
||||
): RouteTemplate | null => {
|
||||
const segments: RouteSegment[] = [];
|
||||
const parts: string[] = [];
|
||||
const rules: string[] = [];
|
||||
let templateLength = path.length;
|
||||
let rulesLength = 0;
|
||||
let copied = 0;
|
||||
let index = 0;
|
||||
let start = 1;
|
||||
for (;;) {
|
||||
const slash = path.indexOf("/", start);
|
||||
const end = slash < 0 ? path.length : slash;
|
||||
const value = path.slice(start, end);
|
||||
const rule = segmentRule(value);
|
||||
if (rule !== null) {
|
||||
if (!temporary(SEGMENT_SCRATCH_BYTES)) return null;
|
||||
const placeholder = `{${rule}}`;
|
||||
const identity = `${index}:${rule};`;
|
||||
segments.push({ segment_index: index, rule, value });
|
||||
parts.push(path.slice(copied, start), placeholder);
|
||||
rules.push(identity);
|
||||
templateLength += placeholder.length - value.length;
|
||||
rulesLength += identity.length;
|
||||
copied = end;
|
||||
}
|
||||
if (slash < 0) break;
|
||||
start = slash + 1;
|
||||
index += 1;
|
||||
}
|
||||
if (segments.length === 0) return { template: path, rules: "", segments };
|
||||
if (!temporary(textBytes(templateLength) + textBytes(rulesLength)))
|
||||
return null;
|
||||
parts.push(path.slice(copied));
|
||||
return { template: parts.join(""), rules: rules.join(""), segments };
|
||||
};
|
||||
|
||||
/** Read one record's request line, charging URL and MIME text before parsing them. */
|
||||
export const readRecordedRequestLine = (
|
||||
reported: unknown,
|
||||
includeQuery: boolean,
|
||||
temporary: Reserve,
|
||||
): RecordedRequestLine | null => {
|
||||
const excluded = (reason: RequestLineExclusion): RecordedRequestLine => ({
|
||||
kind: "excluded",
|
||||
reason,
|
||||
});
|
||||
const request = requestSchema.safeParse(reported);
|
||||
if (!request.success) return excluded("request-unavailable");
|
||||
const { url, method } = request.data.request;
|
||||
if (url === null) return excluded("url-redacted");
|
||||
if (typeof url !== "string") return excluded("url-unparseable");
|
||||
// Admission precedes URL parsing, text normalization and path materialization.
|
||||
if (!temporary(RECORD_SCRATCH_BYTES + textBytes(url.length))) return null;
|
||||
const parsed = parseRecordedTransportUrl(url, { includeQuery });
|
||||
if (parsed.origin === null) return excluded("url-unparseable");
|
||||
if (typeof method !== "string" || method.length === 0)
|
||||
return excluded("method-unavailable");
|
||||
if (parsed.path === null) return excluded("url-unparseable");
|
||||
const response = responseSchema.safeParse(reported);
|
||||
const status = response.success ? response.data.response.status : undefined;
|
||||
const content = contentSchema.safeParse(
|
||||
response.success ? response.data.response.content : undefined,
|
||||
);
|
||||
const mime = content.success ? content.data.mimeType : undefined;
|
||||
const rawMime = typeof mime === "string" ? mime : "";
|
||||
if (!temporary(textBytes(rawMime.length))) return null;
|
||||
return {
|
||||
kind: "line",
|
||||
method,
|
||||
origin: parsed.origin,
|
||||
path: parsed.path,
|
||||
query: parsed.query,
|
||||
status: typeof status === "number" && status !== 0 ? status : null,
|
||||
rawMime,
|
||||
};
|
||||
};
|
||||
|
||||
/** Reserve form-parser scratch before decoding distinct query parameter names. */
|
||||
export const readQueryParameterNames = (
|
||||
query: string | undefined,
|
||||
temporary: Reserve,
|
||||
): ReadonlySet<string> | undefined | null => {
|
||||
if (query === undefined) return undefined;
|
||||
// Reserve form-parser tuples, decoded strings, the iterator and the name
|
||||
// set before URLSearchParams materializes them. Delimiter count is an upper
|
||||
// bound: empty query pairs are ignored by the WHATWG parser.
|
||||
let pairs = query.length > 1 ? 1 : 0;
|
||||
for (const character of query) if (character === "&") pairs += 1;
|
||||
if (
|
||||
!temporary(
|
||||
RECORD_SCRATCH_BYTES + textBytes(query.length) + pairs * ENTRY_BYTES * 4,
|
||||
)
|
||||
)
|
||||
return null;
|
||||
return new Set(parseUrlQueryParameterNames(query));
|
||||
};
|
||||
+14
-1
@@ -129,7 +129,20 @@ export const inferJsonShape = (text: string): JsonShape | null => {
|
||||
};
|
||||
};
|
||||
|
||||
const jsonValueType = (value: unknown): JsonValueType => {
|
||||
/** Parse text as a JSON object; anything else, including invalid JSON, is null. */
|
||||
export const parseJsonObject = (
|
||||
text: string,
|
||||
): Readonly<Record<string, unknown>> | null => {
|
||||
let root: unknown;
|
||||
try {
|
||||
root = JSON.parse(text);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return isRecord(root) ? root : null;
|
||||
};
|
||||
/** The JSON type name of one parsed value. */
|
||||
export const jsonValueType = (value: unknown): JsonValueType => {
|
||||
if (value === null) return "null";
|
||||
if (Array.isArray(value)) return "array";
|
||||
switch (typeof value) {
|
||||
|
||||
@@ -301,14 +301,16 @@ it("runs built CLI and real stdio SDK with exact evidence parity, selection, ret
|
||||
unique_bytes: "5",
|
||||
covered_ranges: [{ start: "10", end: "15" }],
|
||||
});
|
||||
const retained = await call("inspect_analysis_view", {
|
||||
source: { kind: "retained-evidence", evidence_id: mcp.evidence_id },
|
||||
view: { kind: "api-surface" },
|
||||
});
|
||||
// An unsupported view over an existing retained parent proves lookup without pretending qlog is HAR.
|
||||
expect(parseMcpToolError(retained)).toMatchObject({
|
||||
error: { code: "unsupported_target" },
|
||||
});
|
||||
for (const kind of ["api-surface", "environment-manifest"] as const) {
|
||||
const retained = await call("inspect_analysis_view", {
|
||||
source: { kind: "retained-evidence", evidence_id: mcp.evidence_id },
|
||||
view: { kind },
|
||||
});
|
||||
expect(parseMcpToolError(retained)).toMatchObject({
|
||||
error: { code: "unsupported_target" },
|
||||
});
|
||||
}
|
||||
await client.ping();
|
||||
for (const record_ordinals of [[2], [0, 0]]) {
|
||||
expect(
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { apiSurfaceHar } from "../../fixtures/historicalHar.js";
|
||||
import {
|
||||
apiSurfaceHar,
|
||||
environmentManifestHar,
|
||||
} from "../../fixtures/historicalHar.js";
|
||||
import { inspectAnalysisView } from "../../../src/application/analysisView/AnalysisViewService.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, unlink, writeFile } from "node:fs/promises";
|
||||
@@ -18,6 +21,7 @@ import {
|
||||
analysisViewJavaScriptEvidence,
|
||||
analysisViewJavaScriptAnalysisWithSource,
|
||||
analysisViewNativeEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
} from "../../fixtures/analysisView.js";
|
||||
import { createEvidence, parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import { ghidraFunctionDossier } from "../../../src/domain/ghidraValues.fixture.js";
|
||||
@@ -542,3 +546,126 @@ cliTest(
|
||||
expect(JSON.stringify(result.json)).not.toContain("ordinary");
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"projects an environment manifest through the compiled CLI without recorded values",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-environment-manifest-cli-");
|
||||
const path = join(root, "view.json");
|
||||
await writeFile(
|
||||
path,
|
||||
JSON.stringify({
|
||||
source: {
|
||||
kind: "inline",
|
||||
evidence: analysisViewCaptureEvidence(
|
||||
environmentManifestCaptureFixture(),
|
||||
),
|
||||
},
|
||||
view: {
|
||||
kind: "environment-manifest",
|
||||
template_rules: [
|
||||
"after-uid-literal",
|
||||
"opaque-token",
|
||||
"sibling-variance",
|
||||
],
|
||||
},
|
||||
}),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-analysis-view", path, "--json"],
|
||||
});
|
||||
expect(result.exitCode, result.stdout).toBe(0);
|
||||
const evidence = parseEvidence(result.json);
|
||||
expect(evidence.confidence).toBe("inferred");
|
||||
expect(evidence.normalized_result).toMatchObject({
|
||||
kind: "environment-manifest",
|
||||
origin: "http://grafana.example",
|
||||
name: "grafana-example",
|
||||
coverage: { total: 19, exhausted: true },
|
||||
excluded: expect.arrayContaining([
|
||||
expect.objectContaining({ reason: "other-origin" }),
|
||||
]),
|
||||
});
|
||||
for (const template of [
|
||||
"/api/dashboards/uid/{uid}",
|
||||
"/api/folders/{uid}",
|
||||
"/api/plugins/{plugin_id}/settings",
|
||||
"/api/datasources/uid/{uid}/health",
|
||||
])
|
||||
expect(result.stdout).toContain(JSON.stringify(template));
|
||||
for (const value of ["probe folder", "Main Org", "JFIF", "console.log"])
|
||||
expect(result.stdout).not.toContain(value);
|
||||
await writeFile(
|
||||
path,
|
||||
JSON.stringify({
|
||||
source: {
|
||||
kind: "inline",
|
||||
evidence: analysisViewCaptureEvidence(
|
||||
analysisViewObjectDenseCapture(),
|
||||
),
|
||||
},
|
||||
view: { kind: "environment-manifest" },
|
||||
}),
|
||||
);
|
||||
const refused = await cli.run({
|
||||
arguments: ["inspect-analysis-view", path, "--json"],
|
||||
});
|
||||
expect(refused.exitCode).toBe(1);
|
||||
expect(refused.json).toMatchObject({
|
||||
code: "resource_constraint",
|
||||
details: { resource: "memory" },
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"decodes a Grafana-shaped HAR and projects its environment manifest through the compiled workflows",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-manifest-chain-cli-");
|
||||
const capturePath = join(root, "capture.har");
|
||||
await writeFile(capturePath, JSON.stringify(environmentManifestHar()));
|
||||
const capture = await cli.run({
|
||||
arguments: ["inspect-web-network-capture", capturePath, "har", "--json"],
|
||||
});
|
||||
expect(capture.exitCode, capture.stdout).toBe(0);
|
||||
const parent = parseEvidence(capture.json);
|
||||
const input = join(root, "manifest.json");
|
||||
await writeFile(
|
||||
input,
|
||||
JSON.stringify({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: { kind: "environment-manifest" },
|
||||
}),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-analysis-view", input, "--json"],
|
||||
});
|
||||
expect(result.exitCode, result.stdout).toBe(0);
|
||||
expect(result.json).toMatchObject({
|
||||
confidence: "inferred",
|
||||
evidence_links: [parent.evidence_id],
|
||||
normalized_result: {
|
||||
kind: "environment-manifest",
|
||||
name: "grafana-example",
|
||||
templated_groups: 6,
|
||||
coverage: { total: 22, exhausted: true },
|
||||
artifact: { path: capturePath, sha256: parent.subject?.digest.sha256 },
|
||||
records: expect.arrayContaining([
|
||||
{
|
||||
ordinal: 3,
|
||||
method: "GET",
|
||||
path: "/api/org",
|
||||
status: 200,
|
||||
content_type: "application/json",
|
||||
},
|
||||
]),
|
||||
unknowns: [
|
||||
expect.objectContaining({ kind: "authorization-model" }),
|
||||
expect.objectContaining({ kind: "response-semantics" }),
|
||||
expect.objectContaining({ kind: "state-transition" }),
|
||||
],
|
||||
},
|
||||
});
|
||||
expect(JSON.stringify(result.json)).not.toContain("probe folder");
|
||||
},
|
||||
);
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
import { advertisedMcpToolSchemas } from "../../fixtures/advertisedMcpTool.js";
|
||||
import { API_SURFACE_EXAMPLE } from "../../../src/contracts/analysisViewApiSurfaceExample.js";
|
||||
import { ENVIRONMENT_MANIFEST_EXAMPLE } from "../../../src/contracts/analysisViewEnvironmentManifestExample.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { WebNetworkCaptureService } from "../../../src/application/WebNetworkCaptureService.js";
|
||||
import { decodeHarCapture } from "../../../src/browser/history/HarCapture.js";
|
||||
import { join } from "node:path";
|
||||
import { apiSurfaceHar } from "../../fixtures/historicalHar.js";
|
||||
import {
|
||||
apiSurfaceHar,
|
||||
environmentManifestHar,
|
||||
} from "../../fixtures/historicalHar.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import { Client, InMemoryTransport } from "@modelcontextprotocol/client";
|
||||
@@ -38,6 +42,7 @@ import {
|
||||
analysisViewCaptureEvidence,
|
||||
analysisViewObjectDenseCapture,
|
||||
analysisViewNativeEvidence,
|
||||
environmentManifestCaptureFixture,
|
||||
} from "../../fixtures/analysisView.js";
|
||||
import { createTestBinarySession } from "../../fixtures/binarySession.js";
|
||||
import { BINARY_LAYOUT_TEST_PROVIDER } from "../../fixtures/binaryDiagnostics/layout.js";
|
||||
@@ -1092,37 +1097,153 @@ it("projects an actual HAR capture retained by the public MCP workflow", async (
|
||||
});
|
||||
});
|
||||
|
||||
it("reproduces the published API surface example from decoded fixture bytes", async () => {
|
||||
const fixture = apiSurfaceHar();
|
||||
fixture.log.entries = fixture.log.entries.slice(0, 2);
|
||||
const bytes = Buffer.from(JSON.stringify(fixture));
|
||||
const service = new WebNetworkCaptureService({
|
||||
inspect: async () =>
|
||||
ok({
|
||||
...decodeHarCapture(bytes.toString("utf8"), []),
|
||||
artifact: {
|
||||
path: "/fixtures/api-surface.har",
|
||||
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
bytes: bytes.length,
|
||||
},
|
||||
format: "har",
|
||||
runtime_attribution: "unknown",
|
||||
limitations: [],
|
||||
const PUBLISHED_EXAMPLES: readonly (readonly [
|
||||
name: string,
|
||||
example: {
|
||||
readonly input: {
|
||||
readonly source: { readonly evidence_id: string };
|
||||
readonly view: Record<string, unknown>;
|
||||
};
|
||||
readonly output: { readonly artifact: { readonly path: string } };
|
||||
},
|
||||
fixture: () => unknown,
|
||||
])[] = [
|
||||
[
|
||||
"API surface",
|
||||
API_SURFACE_EXAMPLE,
|
||||
() => {
|
||||
const fixture = apiSurfaceHar();
|
||||
fixture.log.entries = fixture.log.entries.slice(0, 2);
|
||||
return fixture;
|
||||
},
|
||||
],
|
||||
[
|
||||
"environment manifest",
|
||||
ENVIRONMENT_MANIFEST_EXAMPLE,
|
||||
() => {
|
||||
const fixture = environmentManifestHar();
|
||||
fixture.log.entries = [10, 20, 27].map(
|
||||
(index) => fixture.log.entries[index]!,
|
||||
);
|
||||
return fixture;
|
||||
},
|
||||
],
|
||||
];
|
||||
|
||||
it.each(PUBLISHED_EXAMPLES)(
|
||||
"reproduces the published %s example from decoded fixture bytes",
|
||||
async (_name, example, fixture) => {
|
||||
const bytes = Buffer.from(JSON.stringify(fixture()));
|
||||
const path = example.output.artifact.path;
|
||||
const service = new WebNetworkCaptureService({
|
||||
inspect: async () =>
|
||||
ok({
|
||||
...decodeHarCapture(bytes.toString("utf8"), []),
|
||||
artifact: {
|
||||
path,
|
||||
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
bytes: bytes.length,
|
||||
},
|
||||
format: "har",
|
||||
runtime_attribution: "unknown",
|
||||
limitations: [],
|
||||
}),
|
||||
});
|
||||
const { client } = await connect(undefined, service);
|
||||
const capture = await client.callTool({
|
||||
name: "inspect_web_network_capture",
|
||||
arguments: { capture_path: path, format: "har" },
|
||||
});
|
||||
expect(parseEvidence(capture.structuredContent).evidence_id).toBe(
|
||||
example.input.source.evidence_id,
|
||||
);
|
||||
const response = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: example.input,
|
||||
});
|
||||
expect(parseEvidence(response.structuredContent).normalized_result).toEqual(
|
||||
example.output,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("projects an environment manifest identically from retained and portable MCP Evidence", async () => {
|
||||
const { client, session } = await connect();
|
||||
const { ajv, inputSchema, outputSchema } = await advertisedMcpToolSchemas(
|
||||
client,
|
||||
"inspect_analysis_view",
|
||||
);
|
||||
const capture = environmentManifestCaptureFixture();
|
||||
const parent = analysisViewCaptureEvidence(capture);
|
||||
expect(session.recordEvidence(parent).ok).toBe(true);
|
||||
const view = {
|
||||
kind: "environment-manifest",
|
||||
template_rules: ["after-uid-literal", "opaque-token", "sibling-variance"],
|
||||
};
|
||||
const outputs = [];
|
||||
for (const source of [
|
||||
{ kind: "inline", evidence: parent },
|
||||
{ kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
]) {
|
||||
expect(ajv.validate(inputSchema, { source, view })).toBe(true);
|
||||
expect(
|
||||
ajv.validate(inputSchema, {
|
||||
source,
|
||||
view: { ...view, template_rules: ["glob"] },
|
||||
}),
|
||||
});
|
||||
const { client } = await connect(undefined, service);
|
||||
const capture = await client.callTool({
|
||||
name: "inspect_web_network_capture",
|
||||
arguments: { capture_path: "/fixtures/api-surface.har", format: "har" },
|
||||
});
|
||||
expect(parseEvidence(capture.structuredContent).evidence_id).toBe(
|
||||
API_SURFACE_EXAMPLE.input.source.evidence_id,
|
||||
).toBe(false);
|
||||
const response = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: { source, view },
|
||||
});
|
||||
expect(response.isError).not.toBe(true);
|
||||
expect(
|
||||
ajv.validate(outputSchema, response.structuredContent),
|
||||
JSON.stringify(ajv.errors),
|
||||
).toBe(true);
|
||||
const evidence = parseEvidence(response.structuredContent);
|
||||
expect(evidence.confidence).toBe("inferred");
|
||||
expect(evidence.normalized_result).toMatchObject({
|
||||
kind: "environment-manifest",
|
||||
name: "grafana-example",
|
||||
templated_groups: 13,
|
||||
tools: expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
name: "get_plugins_settings_by_plugin_id",
|
||||
path_template: "/api/plugins/{plugin_id}/settings",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
name: "get_folders_by_uid",
|
||||
path_template: "/api/folders/{uid}",
|
||||
}),
|
||||
]),
|
||||
});
|
||||
expect(session.evidenceById(evidence.evidence_id)).toEqual(evidence);
|
||||
for (const value of ["probe folder", "Main Org", "JFIF", "console.log"])
|
||||
expect(JSON.stringify(response)).not.toContain(value);
|
||||
outputs.push(evidence.normalized_result);
|
||||
}
|
||||
expect(outputs[0]).toEqual(outputs[1]);
|
||||
const hidden = createEvidence(
|
||||
{
|
||||
path: capture.artifact.path,
|
||||
sha256: capture.artifact.sha256!,
|
||||
format: "file",
|
||||
},
|
||||
{ id: "fixture", name: "fixture", version: "1" },
|
||||
{
|
||||
operation: "inspect_web_network_capture",
|
||||
parameters: { sensitive_value_count: 1 },
|
||||
result: capture,
|
||||
confidence: "observed",
|
||||
},
|
||||
);
|
||||
const response = await client.callTool({
|
||||
const refused = await client.callTool({
|
||||
name: "inspect_analysis_view",
|
||||
arguments: API_SURFACE_EXAMPLE.input,
|
||||
arguments: { source: { kind: "inline", evidence: hidden }, view },
|
||||
});
|
||||
expect(parseEvidence(response.structuredContent).normalized_result).toEqual(
|
||||
API_SURFACE_EXAMPLE.output,
|
||||
);
|
||||
expect(parseMcpToolError(refused).error.code).toBe("unsupported_target");
|
||||
expect(JSON.stringify(refused)).toContain("sensitive_values_unretained");
|
||||
expect(JSON.stringify(refused)).not.toContain("/api/folders");
|
||||
await client.ping();
|
||||
});
|
||||
|
||||
Vendored
+16
-1
@@ -1,4 +1,4 @@
|
||||
import { apiSurfaceHar } from "./historicalHar.js";
|
||||
import { apiSurfaceHar, environmentManifestHar } from "./historicalHar.js";
|
||||
import { webNetworkCaptureSchema } from "../../src/domain/webNetworkCapture.js";
|
||||
import { BINARY_LAYOUT_TEST_PROVIDER } from "./binaryDiagnostics/layout.js";
|
||||
import { binaryLayoutFixture } from "./binaryDiagnostics/layout.js";
|
||||
@@ -288,3 +288,18 @@ export const analysisViewObjectDenseCapture = (headerCount = 100_000) => {
|
||||
};
|
||||
return capture;
|
||||
};
|
||||
|
||||
/** Capture parent decoded from the Grafana-shaped HAR with producer ordinals intact. */
|
||||
export const environmentManifestCaptureFixture = () => {
|
||||
const capture = analysisViewCaptureFixture();
|
||||
const entries = environmentManifestHar().log.entries;
|
||||
capture.artifact.path = "/fixtures/environment-manifest.har";
|
||||
capture.total_records = entries.length;
|
||||
capture.records = entries.map((reported, ordinal) => ({
|
||||
...capture.records[0]!,
|
||||
ordinal,
|
||||
location: { kind: "json-pointer", pointer: `/log/entries/${ordinal}` },
|
||||
reported,
|
||||
}));
|
||||
return capture;
|
||||
};
|
||||
|
||||
Vendored
+237
@@ -72,3 +72,240 @@ export const apiSurfaceHar = () => {
|
||||
}));
|
||||
return fixture;
|
||||
};
|
||||
|
||||
type ManifestRow = readonly [
|
||||
method: string,
|
||||
url: string,
|
||||
mimeType: string,
|
||||
text: string,
|
||||
postText?: string,
|
||||
];
|
||||
|
||||
/** Twenty-eight recorded exchanges of a Grafana-shaped application: UI assets, a second origin, JSON API reads and writes. */
|
||||
const ENVIRONMENT_MANIFEST_ROWS: readonly ManifestRow[] = [
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/dashboards",
|
||||
"text/html",
|
||||
"<!doctype html><html><body>Grafana</body></html>",
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/public/build/app.js",
|
||||
"text/javascript",
|
||||
"console.log('app');",
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"https://grafana.example.com/api/plugins/announcements",
|
||||
"application/json",
|
||||
'{"items":[]}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/org",
|
||||
"application/json",
|
||||
'{"id":1,"name":"Main Org."}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/user",
|
||||
"application/json",
|
||||
'{"id":1,"login":"admin","isGrafanaAdmin":true}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/users/2",
|
||||
"application/json",
|
||||
'{"id":2,"uid":"fg1jo96rvtvk0e","login":"probe","email":"probe@example.test"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/users/3",
|
||||
"application/json",
|
||||
'{"id":3,"uid":"ag1jo31kg1ekgc","login":"viewer","email":"viewer@example.test"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/users/2/orgs",
|
||||
"application/json",
|
||||
'[{"orgId":1,"name":"Main Org.","role":"Viewer"}]',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/search?query=probe&type=dash-db",
|
||||
"application/json",
|
||||
'[{"id":7,"uid":"acb4v6","title":"probe dashboard","type":"dash-db"}]',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/search?query=&type=dash-folder",
|
||||
"application/json",
|
||||
'[{"id":5,"uid":"fg1jo91e57e2ob","title":"probe folder","type":"dash-folder"}]',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/dashboards/uid/acb4v6",
|
||||
"application/json",
|
||||
'{"meta":{"type":"db","folderUid":"fg1jo91e57e2ob"},"dashboard":{"uid":"acb4v6","title":"probe dashboard","panels":[]}}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/dashboards/uid/acb4v6/versions",
|
||||
"application/json",
|
||||
'{"versions":[{"id":1,"version":1}]}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/folders/fg1jo91e57e2ob?accesscontrol=true",
|
||||
"application/json",
|
||||
'{"id":5,"uid":"fg1jo91e57e2ob","title":"probe folder"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/folders/general?accesscontrol=true",
|
||||
"application/json",
|
||||
'{"id":0,"uid":"general","title":"Dashboards"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/folders?limit=100&page=1",
|
||||
"application/json",
|
||||
'[{"id":5,"uid":"fg1jo91e57e2ob","title":"probe folder"}]',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/plugins/grafana-testdata-datasource/settings",
|
||||
"application/json",
|
||||
'{"id":"grafana-testdata-datasource","enabled":true}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/plugins/grafana-lokiexplore-app/settings",
|
||||
"application/json",
|
||||
'{"id":"grafana-lokiexplore-app","enabled":true}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/plugins/grafana-pyroscope-app/settings",
|
||||
"application/json",
|
||||
'{"id":"grafana-pyroscope-app","enabled":false}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/teams/1",
|
||||
"application/json",
|
||||
'{"id":1,"name":"probe team","memberCount":1}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/teams/1/members",
|
||||
"application/json",
|
||||
'[{"userId":2,"login":"probe"}]',
|
||||
],
|
||||
[
|
||||
"POST",
|
||||
"http://grafana.example/api/folders",
|
||||
"application/json",
|
||||
'{"id":5,"uid":"fg1jo91e57e2ob","title":"probe folder"}',
|
||||
'{"title":"probe folder"}',
|
||||
],
|
||||
[
|
||||
"POST",
|
||||
"http://grafana.example/api/annotations",
|
||||
"application/json",
|
||||
'{"id":11,"message":"Annotation added"}',
|
||||
'{"dashboardUID":"acb4v6","panelId":1,"time":1792000000000,"text":"probe","tags":["rea"]}',
|
||||
],
|
||||
[
|
||||
"PUT",
|
||||
"http://grafana.example/api/folders/fg1jo91e57e2ob",
|
||||
"application/json",
|
||||
'{"id":5,"uid":"fg1jo91e57e2ob","title":"probe folder renamed"}',
|
||||
'{"title":"probe folder renamed","overwrite":true}',
|
||||
],
|
||||
[
|
||||
"DELETE",
|
||||
"http://grafana.example/api/dashboards/uid/acb4v6",
|
||||
"application/json",
|
||||
'{"title":"probe dashboard","message":"Dashboard probe dashboard deleted"}',
|
||||
],
|
||||
[
|
||||
"DELETE",
|
||||
"http://grafana.example/api/teams/1",
|
||||
"application/json",
|
||||
'{"message":"Team deleted"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/annotations?dashboardUID=acb4v6&limit=10",
|
||||
"application/json",
|
||||
'[{"id":11,"dashboardUID":"acb4v6","text":"probe"}]',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/api/datasources/uid/dg1jo96tern5sb/health",
|
||||
"application/json",
|
||||
'{"status":"OK","message":"Data source is working"}',
|
||||
],
|
||||
[
|
||||
"GET",
|
||||
"http://grafana.example/avatar/46d229b033af06a191ff2267bca9ae56",
|
||||
"image/jpeg",
|
||||
"JFIF",
|
||||
],
|
||||
];
|
||||
|
||||
/** Playwright-declared HAR 1.2 fixture for the environment-manifest view; every body is synthetic. */
|
||||
export const environmentManifestHar = () => ({
|
||||
log: {
|
||||
version: "1.2",
|
||||
creator: { name: "Playwright", version: "1.58.2" },
|
||||
browser: { name: "chromium", version: "143.0.7499.4" },
|
||||
pages: [
|
||||
{
|
||||
startedDateTime: "2026-10-17T18:00:00.000Z",
|
||||
id: "page@1",
|
||||
title: "Grafana",
|
||||
pageTimings: { onContentLoad: -1, onLoad: -1 },
|
||||
},
|
||||
],
|
||||
entries: ENVIRONMENT_MANIFEST_ROWS.map(
|
||||
([method, url, mimeType, text, postText]) => ({
|
||||
startedDateTime: "2026-10-17T18:00:00.000Z",
|
||||
time: 7.5,
|
||||
request: {
|
||||
method,
|
||||
url,
|
||||
httpVersion: "HTTP/1.1",
|
||||
cookies: [],
|
||||
headers: [
|
||||
{ name: "accept", value: "application/json, text/plain, */*" },
|
||||
],
|
||||
queryString: [...new URL(url).searchParams].map(([name, value]) => ({
|
||||
name,
|
||||
value,
|
||||
})),
|
||||
headersSize: -1,
|
||||
bodySize: postText === undefined ? -1 : postText.length,
|
||||
...(postText === undefined
|
||||
? {}
|
||||
: { postData: { mimeType: "application/json", text: postText } }),
|
||||
},
|
||||
response: {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
httpVersion: "HTTP/1.1",
|
||||
cookies: [],
|
||||
headers: [{ name: "content-type", value: mimeType }],
|
||||
content: { size: text.length, mimeType, text },
|
||||
redirectURL: "",
|
||||
headersSize: -1,
|
||||
bodySize: text.length,
|
||||
},
|
||||
cache: {},
|
||||
timings: { send: 0.5, wait: 5.0, receive: 2.0 },
|
||||
}),
|
||||
),
|
||||
},
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user