mirror of
https://github.com/morluto/rea
synced 2026-10-25 22:08:11 -04:00
feat(capture): decode selected protobuf and gRPC bodies (#5031)
Retain exact bounded wire facts and optional supplied-schema interpretations for selected historical captures. Share CLI/MCP views and preserve Evidence, partial coverage, credential exclusions and source provenance. Fixes #4986 Co-authored-by: AayushJB03 <aayushjb14@gmail.com> Co-authored-by: gensokyo-dev <gensokyo-dev@users.noreply.github.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Aayush Bhadbhade
@@ -69,6 +69,12 @@ with `inspect_analysis_view` and `view: {kind: "capture", detail: "preview", off
|
||||
Select complete record pages only for needed headers, metadata or payloads; views
|
||||
read the historical parent without repeating capture decoding.
|
||||
|
||||
For selected protobuf/gRPC bodies, keep raw bytes as the source and use
|
||||
`inspect_analysis_view` kind `protobuf`, collection `bodies`, `frames` or
|
||||
`fields`. Batch exact record ordinals and field numbers/IDs, and select
|
||||
`schema_value` or `text` only when needed. Supply a local descriptor set and
|
||||
message type for schema interpretation; compressed payloads remain unknown.
|
||||
|
||||
Inspect native `.saz` snapshots with `inspect_web_network_capture` and
|
||||
`format: "saz"`; encrypted archives are unsupported. For live
|
||||
traffic analysis, connect Everywhere's separate MCP server and use its
|
||||
|
||||
@@ -122,29 +122,96 @@ XML node tree; optional `_w.txt` and `_g.txt` files retain opaque WebSocket and
|
||||
gRPC message bytes. Missing members remain null, distinct from empty bodies.
|
||||
See Telerik's [SAZ format reference](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/knowledge-base/fiddler-archives).
|
||||
|
||||
The existing ZIP reader verifies read members; expanded archive contents are
|
||||
limited to 32 MiB in addition to the input/output budgets below. Exceeding this
|
||||
expanded-byte budget returns `invalid_request` / `invalid_input` through both
|
||||
CLI and MCP, with an `out_of_range` issue at
|
||||
`capture_path` and `maximum_expanded_capture_bytes`. Selecting records does not
|
||||
bypass the complete-archive admission check or return partial Evidence. Encrypted
|
||||
archives and unrecognized raw session members are unsupported. SAZ requires an
|
||||
enforced decoder memory bound: Node's fixed heap, or the Windows Job Object
|
||||
commit limit. Bun on hosts without that Job Object limit returns capability
|
||||
unavailable before starting the SAZ decoder. Non-session
|
||||
members are inventoried without decoding their content. Original files remain
|
||||
unchanged. Linux verification includes independently checked public recorded
|
||||
Mattermost captures from [FOREST](https://github.com/dfrc-korea/FOREST/tree/405c9154658a128ea203a6c10040cec55111ed03/api-results/dataset)
|
||||
and the Fiddler author's [encoding corpus](https://github.com/ericlaw1979/Clearinet/blob/b6faf78cde1e72042980364f180ceb34b77d1434/Test/data/Encodings.saz),
|
||||
plus synthetic malformed/limit fixtures. Their exporting product versions are
|
||||
not authenticated. An interactive export from a current Fiddler Everywhere
|
||||
installation and its live MCP server remain unverified. Support here is the
|
||||
bounded documented raw HTTP/XML SAZ layout, not every product/version profile.
|
||||
## Protobuf and gRPC bodies
|
||||
|
||||
For visual inspection, use Everywhere's **Import Snapshot**, then **Inspectors**
|
||||
and **Overview**. HAR remains an alternative: select sessions, choose
|
||||
**Export > HTTPArchive**, and confirm `log.version` is `"1.2"`. Preserve the
|
||||
original SAZ when exporting. See the [import/export instructions](https://www.telerik.com/fiddler/fiddler-everywhere/documentation/collaboration/exporting-and-importing).
|
||||
Selecting `record_ordinals` (CLI `--record`) also decodes retained bodies with
|
||||
recognized protobuf or gRPC content types. Whole-capture inspection does not
|
||||
perform this extra work. The original binary sidecar, SHA-256 and producer
|
||||
ordinal remain the source; decoding adds flat body, frame and field facts.
|
||||
|
||||
Native gRPC admits only data-frame flags 0/1; gRPC-web additionally admits its
|
||||
terminal trailer frame. `grpc-web-text` supports independently padded base64
|
||||
chunks. Its offsets refer to **base64-decoded body bytes**; other offsets refer
|
||||
to retained body bytes. Every field has a body-scoped `field_id` derived from
|
||||
its byte offset, a `frame_index`, and a `parent_field_id` for schema-driven
|
||||
nested messages or groups. Repeated field numbers remain separate occurrences.
|
||||
|
||||
Without a schema, varints and fixed-width values retain exact unsigned decimal
|
||||
representations. Length-delimited UTF-8, nested-message and packed readings
|
||||
are candidates; REA does not choose one or invent field names or signedness.
|
||||
Nested candidate ranges remain accessible without recursively expanding them.
|
||||
Malformed messages retain the parsed prefix and an exact error offset.
|
||||
Compressed frames and encoded HTTP bodies remain unknown; they are not
|
||||
silently treated as empty messages.
|
||||
|
||||
For schema interpretation, supply a local compiled `FileDescriptorSet` and its
|
||||
fully qualified message type. All imports must be included. REA never fetches
|
||||
schemas or executes the selected application. Descriptor identity and parser
|
||||
version remain in the body facts. Scalars, repeated numeric packing and nested
|
||||
messages are interpreted alongside the unchanged wire facts; disagreements
|
||||
remain explicit. Map semantics, extensions and required-field validation are
|
||||
unsupported. A supplied schema does not establish the sender's actual schema.
|
||||
An invalid nested interpretation records `schema_status: "invalid-message"`
|
||||
and `schema_error` locally; later outer fields remain available. Valid string
|
||||
contents, including a leading U+FEFF, are preserved.
|
||||
|
||||
```bash
|
||||
# Compile your own schema with your existing protoc, including local imports:
|
||||
protoc --include_imports --descriptor_set_out=/evidence/api.pb api.proto
|
||||
rea inspect-web-network-capture /evidence/session.har har --record 3 \
|
||||
--protobuf-schema /evidence/api.pb --protobuf-message example.Response \
|
||||
--detail summary --evidence_output /evidence/session-evidence.json --json
|
||||
```
|
||||
|
||||
MCP accepts the same choices as `protobuf: {schema: {descriptor_set_path:
|
||||
"/evidence/api.pb", message_type: "example.Response"}}`, with explicit
|
||||
`record_ordinals`. Optional `body_pointers` batch-select exact binary pointers;
|
||||
`framing` can explicitly select `raw`, `grpc`, `grpc-web` or `grpc-web-text`
|
||||
when MIME metadata is unavailable. CLI exposes `--protobuf-body-pointer` and
|
||||
`--protobuf-framing`. Filters never invent bytes excluded by source redaction. Decoding an extra
|
||||
base64 layer also checks explicit exclusions; newly recognized trailer
|
||||
credentials exclude the entire affected binary sidecar and its reported
|
||||
encoding, preserving a redaction coordinate instead of a credential digest.
|
||||
Credential inspection covers every selected body before descriptor reads or
|
||||
field expansion. Later capacity or schema failures can therefore retain the
|
||||
sanitized source even with explicit sensitivity options. If that source pass
|
||||
is interrupted, unchecked bodies are not published as partial Evidence.
|
||||
Framing overrides apply only to format-owned body coordinates and preserve
|
||||
exclusions for the transport declared in MIME metadata.
|
||||
Unknown or explicitly excluded HTTP coding metadata is not treated as absent:
|
||||
SAZ/native bodies remain opaque until identity coding is established. Raw
|
||||
header values and retained body bytes remain available in source Evidence.
|
||||
Skipping wire analysis still runs the independent credential check. If HTTP
|
||||
coding metadata is unavailable and that check cannot complete, the operation
|
||||
returns unavailable without publishing unchecked capture bodies as Evidence.
|
||||
|
||||
Use `detail: "summary"`, then `inspect_analysis_view` with its retained parent:
|
||||
|
||||
```json
|
||||
{"kind":"protobuf","collection":"bodies","offset":0,"limit":16}
|
||||
{"kind":"protobuf","collection":"frames","record_ordinals":[3],"offset":0,"limit":16}
|
||||
{"kind":"protobuf","collection":"fields","record_ordinals":[3],"field_numbers":[1,4],"fields":["number","schema_name","schema_value"],"offset":0,"limit":32}
|
||||
```
|
||||
|
||||
Filters intersect; an empty filter selects nothing. Field pages also accept
|
||||
`field_ids`, `frame_indices` and `schema_names`. Default field pages omit `text`,
|
||||
`schema_value` and enum alias inventories; select these columns when needed. Identity, source offset
|
||||
basis and body status survive column selection. Complete Evidence retains
|
||||
all decoded values, and ordinary capture previews expose counts. Page exhaustion
|
||||
covers retained facts only; a partial source does not establish absence.
|
||||
|
||||
Authenticated immutable parents reuse validation and the most recent field
|
||||
filter per capture. These caches live only with retained Evidence, hold references
|
||||
to existing facts and never reopen either file. Cached selector text is bounded
|
||||
by retained field cardinality; larger selections use an uncached scan. Mutable
|
||||
input is revalidated.
|
||||
The existing 96 MiB report budget includes the original decoder reply plus
|
||||
conservative charges for new facts, strings and descriptor expansion. The explicit
|
||||
`protobuf_coverage` reports selected and inspected record counts, preserving
|
||||
partial decoding through composed views. Capacity failure retains the source
|
||||
observation as partial Evidence; cancellation yields
|
||||
between bounded work batches. Real elapsed time and process-family memory
|
||||
still depend on the selected capture and schema.
|
||||
|
||||
## Fiddler Everywhere MCP and public skills
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ upstream texts; they do not change [REA's MIT license](../LICENSE).
|
||||
| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. |
|
||||
| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. |
|
||||
| `protobuf-es-v2.16.0-LICENSE.txt` | Original Apache license omitted from the protobuf-es npm archive, preserved from its v2.16.0 source commit. Its embedded Google BSD notice is retained in `npm-runtime-notices.txt`. |
|
||||
| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. |
|
||||
| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. |
|
||||
| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. |
|
||||
|
||||
@@ -2,7 +2,7 @@ REA npm dependency inventory
|
||||
Prepared from the root package-lock.json; refresh this inventory when locked dependencies change.
|
||||
License identifiers are upstream lockfile declarations, not a legal assessment.
|
||||
Runtime includes transitive and optional platform packages; development includes build, test and website tooling.
|
||||
Locked package entries: 750; runtime: 178.
|
||||
Locked package entries: 751; runtime: 179.
|
||||
Full runtime copyright/license/NOTICE texts: npm-runtime-notices.txt.
|
||||
|
||||
@algolia/abtesting@1.25.0 | development | MIT
|
||||
@@ -335,6 +335,11 @@ obug@3.0.0 | development | MIT
|
||||
Package: https://www.npmjs.com/package/@bcoe/v8-coverage/v/1.0.2
|
||||
Archive: https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-1.0.2.tgz
|
||||
Integrity: sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==
|
||||
@bufbuild/protobuf@2.16.0 | runtime | (Apache-2.0 AND BSD-3-Clause)
|
||||
Lockfile path: node_modules/@bufbuild/protobuf
|
||||
Package: https://www.npmjs.com/package/@bufbuild/protobuf/v/2.16.0
|
||||
Archive: https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.16.0.tgz
|
||||
Integrity: sha512-FWa0sPlqYGJgpTs6OxBRcL/AW4JT0OIO+W1ajerluoTVh8CV0B7XM1qsNCRgnkSCkRCgOICKHRQlY17jrsm0+Q==
|
||||
@cfworker/json-schema@4.1.1 | runtime | MIT
|
||||
Lockfile path: node_modules/@cfworker/json-schema
|
||||
Package: https://www.npmjs.com/package/@cfworker/json-schema/v/4.1.1
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
REA third-party npm runtime copyright, license and NOTICE texts
|
||||
Collected from integrity-verified locked npm archives. Release-specific supplements and upstream notice gaps are identified in their entries.
|
||||
External engines are separately obtained; see THIRD_PARTY_NOTICES.md.
|
||||
Runtime dependency fingerprint (SHA-256): 131100b2e1e6ad5e55b10a0baf4be2d7a0838fd724fb307b9bafd73446b0d3ec
|
||||
Runtime dependency fingerprint (SHA-256): 4125777159e02cb749e17c959e77f59528cfffe3f7a5220b8a21ce3c49ad3061
|
||||
|
||||
===== @babel/helper-string-parser@8.0.6 =====
|
||||
Declared license: MIT
|
||||
@@ -126,6 +126,254 @@ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
|
||||
|
||||
===== @bufbuild/protobuf@2.16.0 =====
|
||||
Declared license: (Apache-2.0 AND BSD-3-Clause)
|
||||
Archive: https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.16.0.tgz
|
||||
Integrity: sha512-FWa0sPlqYGJgpTs6OxBRcL/AW4JT0OIO+W1ajerluoTVh8CV0B7XM1qsNCRgnkSCkRCgOICKHRQlY17jrsm0+Q==
|
||||
Scope: npm runtime dependency, including optional platform packages.
|
||||
|
||||
The locked archive omits standalone license files. The Apache license is preserved from its v2.16.0 upstream commit; the BSD notice is embedded in the integrity-verified archive.
|
||||
|
||||
--- licenses/protobuf-es-v2.16.0-LICENSE.txt (see upstream-sources.json) ---
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2021-2026 Buf Technologies, Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
--- package/dist/esm/wire/varint.js (unchanged leading BSD notice) ---
|
||||
|
||||
// Copyright 2008 Google Inc. All rights reserved.
|
||||
//
|
||||
// Redistribution and use in source and binary forms, with or without
|
||||
// modification, are permitted provided that the following conditions are
|
||||
// met:
|
||||
//
|
||||
// * Redistributions of source code must retain the above copyright
|
||||
// notice, this list of conditions and the following disclaimer.
|
||||
// * Redistributions in binary form must reproduce the above
|
||||
// copyright notice, this list of conditions and the following disclaimer
|
||||
// in the documentation and/or other materials provided with the
|
||||
// distribution.
|
||||
// * Neither the name of Google Inc. nor the names of its
|
||||
// contributors may be used to endorse or promote products derived from
|
||||
// this software without specific prior written permission.
|
||||
//
|
||||
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
//
|
||||
// Code generated by the Protocol Buffer compiler is owned by the owner
|
||||
// of the input file used when generating it. This code is not
|
||||
// standalone and requires a support library to be linked with it. This
|
||||
// support library is itself covered by the above license.
|
||||
|
||||
|
||||
===== @cfworker/json-schema@4.1.1 =====
|
||||
Declared license: MIT
|
||||
Archive: https://registry.npmjs.org/@cfworker/json-schema/-/json-schema-4.1.1.tgz
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright 2021-2026 Buf Technologies, Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -68,5 +68,10 @@
|
||||
"file": "licenses/LGPL-2.0.txt",
|
||||
"source": "https://www.gnu.org/licenses/old-licenses/lgpl-2.0.txt",
|
||||
"sha256": "cc535c21133c895b56b374c8a1dc1eb948d99003ed2b47372069456b62f42b24"
|
||||
},
|
||||
{
|
||||
"file": "licenses/protobuf-es-v2.16.0-LICENSE.txt",
|
||||
"source": "https://raw.githubusercontent.com/bufbuild/protobuf-es/b3b232f7f466eda27c66e8f63c82f930578b6cc6/LICENSE",
|
||||
"sha256": "c04b4216f1cd4c5a4f7fb2f2a1b0ae70d847e9e0cac7c9dee9bf8cc03177c449"
|
||||
}
|
||||
]
|
||||
|
||||
Generated
+7
@@ -11,6 +11,7 @@
|
||||
"dependencies": {
|
||||
"@babel/parser": "8.0.4",
|
||||
"@babel/types": "8.0.4",
|
||||
"@bufbuild/protobuf": "2.16.0",
|
||||
"@clack/prompts": "1.7.0",
|
||||
"@electron/asar": "4.2.0",
|
||||
"@jridgewell/trace-mapping": "0.3.31",
|
||||
@@ -1090,6 +1091,12 @@
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/@bufbuild/protobuf": {
|
||||
"version": "2.16.0",
|
||||
"resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.16.0.tgz",
|
||||
"integrity": "sha512-FWa0sPlqYGJgpTs6OxBRcL/AW4JT0OIO+W1ajerluoTVh8CV0B7XM1qsNCRgnkSCkRCgOICKHRQlY17jrsm0+Q==",
|
||||
"license": "(Apache-2.0 AND BSD-3-Clause)"
|
||||
},
|
||||
"node_modules/@cfworker/json-schema": {
|
||||
"version": "4.1.1",
|
||||
"resolved": "https://registry.npmjs.org/@cfworker/json-schema/-/json-schema-4.1.1.tgz",
|
||||
|
||||
@@ -323,6 +323,7 @@
|
||||
"dependencies": {
|
||||
"@babel/parser": "8.0.4",
|
||||
"@babel/types": "8.0.4",
|
||||
"@bufbuild/protobuf": "2.16.0",
|
||||
"@clack/prompts": "1.7.0",
|
||||
"@electron/asar": "4.2.0",
|
||||
"@jridgewell/trace-mapping": "0.3.31",
|
||||
|
||||
@@ -543,3 +543,29 @@ it("reports malformed retained facts while keeping their cleanup owner retryable
|
||||
expect(result.error.cleanupIncomplete).toBe(false);
|
||||
expect(projectAnalysisError(result.error).code).toBe("unreadable_output");
|
||||
});
|
||||
|
||||
it("rejects relative descriptor paths before starting a capture decoder", async () => {
|
||||
const service = new WebNetworkCaptureService({
|
||||
inspect: () => {
|
||||
throw new Error("Invalid input must fail before provider admission");
|
||||
},
|
||||
});
|
||||
const result = await service.inspect({
|
||||
capture_path: "/capture.har",
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
protobuf: {
|
||||
schema: {
|
||||
descriptor_set_path: "relative.pb",
|
||||
message_type: "capture.Message",
|
||||
},
|
||||
},
|
||||
});
|
||||
if (result.ok) throw new Error("Expected typed invalid input");
|
||||
expect(projectAnalysisError(result.error)).toMatchObject({
|
||||
code: "invalid_request",
|
||||
details: {
|
||||
issues: [{ path: ["protobuf", "schema", "descriptor_set_path"] }],
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
@@ -80,6 +80,30 @@ export class WebNetworkCaptureService {
|
||||
},
|
||||
]),
|
||||
);
|
||||
if (input.protobuf !== undefined && input.record_ordinals === undefined)
|
||||
return failure(
|
||||
new AnalysisInputError(OPERATION, undefined, [
|
||||
{
|
||||
path: ["record_ordinals"],
|
||||
reason: "invalid_format",
|
||||
message:
|
||||
"Protobuf options require explicit producer record ordinals.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
if (
|
||||
input.protobuf?.schema !== undefined &&
|
||||
!isAbsolute(input.protobuf.schema.descriptor_set_path)
|
||||
)
|
||||
return failure(
|
||||
new AnalysisInputError(OPERATION, undefined, [
|
||||
{
|
||||
path: ["protobuf", "schema", "descriptor_set_path"],
|
||||
reason: "invalid_format",
|
||||
message: "Expected an absolute descriptor-set path on this host.",
|
||||
},
|
||||
]),
|
||||
);
|
||||
const loaded = await this.port.inspect(input, options);
|
||||
if (!loaded.ok) {
|
||||
const partial = webNetworkCaptureSchema.safeParse(
|
||||
@@ -201,6 +225,30 @@ export class WebNetworkCaptureService {
|
||||
? {}
|
||||
: { format: input.format }),
|
||||
record_ordinals: selected,
|
||||
...(input.protobuf === undefined
|
||||
? {}
|
||||
: {
|
||||
protobuf: {
|
||||
framing: input.protobuf.framing,
|
||||
...(input.protobuf.body_pointers === undefined
|
||||
? {}
|
||||
: {
|
||||
body_pointers: input.protobuf.body_pointers.map(redactText),
|
||||
}),
|
||||
...(input.protobuf.schema === undefined
|
||||
? {}
|
||||
: {
|
||||
schema: {
|
||||
descriptor_set_path: redactText(
|
||||
input.protobuf.schema.descriptor_set_path,
|
||||
),
|
||||
message_type: redactText(
|
||||
input.protobuf.schema.message_type,
|
||||
),
|
||||
},
|
||||
}),
|
||||
},
|
||||
}),
|
||||
sensitive_value_count: input.sensitive_values.length,
|
||||
});
|
||||
return ok(
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import type { ExecutionOptions } from "../../application/AnalysisProvider.js";
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
|
||||
import { bindArtifactFileCleanup } from "../../artifacts/ArtifactFileOperation.js";
|
||||
import { artifactFileCancellationCause } from "../../artifacts/ArtifactFileOperation.js";
|
||||
import { AnalysisError } from "../../domain/analysisErrorBase.js";
|
||||
import {
|
||||
@@ -33,10 +35,18 @@ export const CAPTURE_PROCESS_MEMORY_LIMIT_REASON =
|
||||
export const historicalCaptureFailure = (
|
||||
input: InspectWebNetworkCaptureInput,
|
||||
cause: unknown,
|
||||
phase: "capture-read" | "decoder",
|
||||
phase: "capture-read" | "descriptor-read" | "decoder",
|
||||
options?: ExecutionOptions,
|
||||
): AnalysisError => {
|
||||
if (cause instanceof AnalysisError) return cause;
|
||||
const descriptorRead = phase === "descriptor-read";
|
||||
const sourcePath = descriptorRead
|
||||
? (input.protobuf?.schema?.descriptor_set_path ?? input.capture_path)
|
||||
: input.capture_path;
|
||||
const inputPath = descriptorRead
|
||||
? ["protobuf", "schema", "descriptor_set_path"]
|
||||
: ["capture_path"];
|
||||
const sourceLabel = descriptorRead ? "descriptor set" : "capture";
|
||||
if (cause instanceof OwnedCommandFailure) {
|
||||
if (cause.cleanupFailure !== null)
|
||||
return new ProviderCleanupError(
|
||||
@@ -99,18 +109,18 @@ export const historicalCaptureFailure = (
|
||||
);
|
||||
}
|
||||
if (
|
||||
phase === "capture-read" &&
|
||||
phase !== "decoder" &&
|
||||
cause instanceof ArtifactReaderFailure &&
|
||||
cause.reason === "integrity"
|
||||
)
|
||||
return new AnalysisArtifactChangedError(
|
||||
OPERATION,
|
||||
input.capture_path,
|
||||
sourcePath,
|
||||
cause.message,
|
||||
{ cause },
|
||||
);
|
||||
if (
|
||||
phase === "capture-read" &&
|
||||
phase !== "decoder" &&
|
||||
cause instanceof ArtifactReaderFailure &&
|
||||
cause.reason === "cancelled"
|
||||
)
|
||||
@@ -118,7 +128,7 @@ export const historicalCaptureFailure = (
|
||||
cause: artifactFileCancellationCause(cause),
|
||||
});
|
||||
if (
|
||||
phase === "capture-read" &&
|
||||
phase !== "decoder" &&
|
||||
cause instanceof ArtifactReaderFailure &&
|
||||
(cause.reason === "limit" ||
|
||||
cause.reason === "format" ||
|
||||
@@ -126,44 +136,44 @@ export const historicalCaptureFailure = (
|
||||
)
|
||||
return new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
path: inputPath,
|
||||
reason: cause.reason === "limit" ? "out_of_range" : "invalid_format",
|
||||
message:
|
||||
cause.reason === "limit"
|
||||
? `${cause.message}. Select a capture within the ${WEB_NETWORK_CAPTURE_LIMITS.inputBytes}-byte input budget.`
|
||||
? `${cause.message}. Select a ${sourceLabel} within the ${WEB_NETWORK_CAPTURE_LIMITS.inputBytes}-byte input budget.`
|
||||
: cause.message,
|
||||
...(cause.reason === "limit"
|
||||
? {
|
||||
expected: {
|
||||
maximum_capture_bytes: WEB_NETWORK_CAPTURE_LIMITS.inputBytes,
|
||||
[descriptorRead
|
||||
? "maximum_descriptor_set_bytes"
|
||||
: "maximum_capture_bytes"]:
|
||||
WEB_NETWORK_CAPTURE_LIMITS.inputBytes,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
},
|
||||
]);
|
||||
if (
|
||||
phase === "capture-read" &&
|
||||
phase !== "decoder" &&
|
||||
cause instanceof Error &&
|
||||
"code" in cause &&
|
||||
(cause.code === "EACCES" || cause.code === "EPERM")
|
||||
)
|
||||
return new AnalysisAccessDeniedError(
|
||||
OPERATION,
|
||||
input.capture_path,
|
||||
cause.code,
|
||||
{ cause },
|
||||
);
|
||||
return new AnalysisAccessDeniedError(OPERATION, sourcePath, cause.code, {
|
||||
cause,
|
||||
});
|
||||
if (
|
||||
phase === "capture-read" &&
|
||||
phase !== "decoder" &&
|
||||
cause instanceof Error &&
|
||||
"code" in cause &&
|
||||
["ENOENT", "ENOTDIR"].includes(String(cause.code))
|
||||
)
|
||||
return new AnalysisInputError(OPERATION, { cause }, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
path: inputPath,
|
||||
reason: "invalid_value",
|
||||
message: `Selected capture could not be read (${String(cause.code)}): ${input.capture_path}.`,
|
||||
message: `Selected ${sourceLabel} could not be read (${String(cause.code)}): ${sourcePath}.`,
|
||||
},
|
||||
]);
|
||||
if (
|
||||
@@ -182,6 +192,9 @@ export const historicalCaptureFailure = (
|
||||
diagnostics: {
|
||||
phase,
|
||||
capture_path: input.capture_path,
|
||||
...(descriptorRead
|
||||
? { source_path: sourcePath, input_path: inputPath }
|
||||
: {}),
|
||||
reason: cause instanceof Error ? cause.message : String(cause),
|
||||
...(cause instanceof ArtifactReaderFailure
|
||||
? { failure_kind: cause.reason }
|
||||
@@ -248,3 +261,36 @@ export const captureMemoryFailure = (
|
||||
: {}),
|
||||
},
|
||||
);
|
||||
|
||||
/** Preserve the exact file owner when capture or descriptor acquisition needs cleanup. */
|
||||
export const withCaptureReaderCleanup = (
|
||||
primary: AnalysisError,
|
||||
cause: unknown,
|
||||
): AnalysisError => {
|
||||
if (!(cause instanceof ArtifactReaderFailure) || cause.cleanup === undefined)
|
||||
return primary;
|
||||
if (cause.reason === "unavailable") {
|
||||
const cleanup = new ProviderCleanupError(
|
||||
"web-network-capture",
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ operation: OPERATION, cause },
|
||||
);
|
||||
bindArtifactFileCleanup(cause, cleanup);
|
||||
return cleanup;
|
||||
}
|
||||
const cleanup = new ProviderCleanupError(
|
||||
"web-network-capture",
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ operation: OPERATION, cause },
|
||||
);
|
||||
bindArtifactFileCleanup(cause, cleanup);
|
||||
const composed = analysisErrorWithCleanupFailure(
|
||||
primary,
|
||||
cleanup,
|
||||
OPERATION,
|
||||
).bindCleanupDependencies([primary, cleanup]);
|
||||
bindArtifactFileCleanup(cause, composed);
|
||||
return composed;
|
||||
};
|
||||
|
||||
@@ -9,6 +9,31 @@ const CREDENTIAL_HEADERS = new Set([
|
||||
"cookie",
|
||||
"set-cookie",
|
||||
]);
|
||||
/** HTTP credential names apply equally to recorded headers and gRPC-web trailers. */
|
||||
export const isCaptureCredentialHeader = (name: string): boolean =>
|
||||
CREDENTIAL_HEADERS.has(name.toLowerCase());
|
||||
|
||||
/** Match an ASCII header name without allocating a potentially enormous line. */
|
||||
export const isCaptureCredentialHeaderBytes = (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
end: number,
|
||||
): boolean => {
|
||||
for (const name of CREDENTIAL_HEADERS) {
|
||||
if (end - start !== name.length) continue;
|
||||
let matches = true;
|
||||
for (let index = 0; index < name.length; index++) {
|
||||
const byte = bytes[start + index]!;
|
||||
const lower = byte >= 65 && byte <= 90 ? byte + 32 : byte;
|
||||
if (lower !== name.charCodeAt(index)) {
|
||||
matches = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (matches) return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
/** Literal redaction and structural authentication exclusions preserve unrelated local evidence. */
|
||||
export class CaptureRedaction {
|
||||
@@ -75,7 +100,7 @@ export class CaptureRedaction {
|
||||
return (
|
||||
(/^\/(request|response)\/headers\/\d+\/value$/.test(pointer) &&
|
||||
typeof parent.name === "string" &&
|
||||
CREDENTIAL_HEADERS.has(parent.name.toLowerCase())) ||
|
||||
isCaptureCredentialHeader(parent.name)) ||
|
||||
/^\/(request|response)\/cookies\/\d+\/value$/.test(pointer)
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,7 +1,13 @@
|
||||
import {
|
||||
decodeCaptureProtobuf,
|
||||
prepareCaptureProtobuf,
|
||||
} from "./protobuf/ProtobufCapture.js";
|
||||
import { CaptureFormatError } from "./CaptureFormatError.js";
|
||||
import {
|
||||
CAPTURE_PROCESS_MEMORY_LIMIT_REASON,
|
||||
historicalCaptureFailure,
|
||||
captureMemoryFailure,
|
||||
withCaptureReaderCleanup,
|
||||
} from "./CaptureFailures.js";
|
||||
import {
|
||||
NODE_CAPTURE_HEAP_ENFORCED,
|
||||
@@ -17,7 +23,6 @@ import { z } from "zod";
|
||||
import type { ExecutionOptions } from "../../application/AnalysisProvider.js";
|
||||
import { readStableArtifact } from "../../artifacts/readStableArtifact.js";
|
||||
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
|
||||
import { bindArtifactFileCleanup } from "../../artifacts/ArtifactFileOperation.js";
|
||||
import {
|
||||
ArtifactResourceScope,
|
||||
type ArtifactCleanupAttempt,
|
||||
@@ -29,6 +34,7 @@ import {
|
||||
AnalysisCancelledError,
|
||||
AnalysisInputError,
|
||||
AnalysisOutputError,
|
||||
AnalysisResourceConstraintError,
|
||||
} from "../../domain/analysisErrorCore.js";
|
||||
import { ProviderAdapterError } from "../../domain/providerAdapterError.js";
|
||||
import { ProviderCleanupError } from "../../domain/providerCleanupError.js";
|
||||
@@ -104,7 +110,7 @@ export class HistoricalCaptureDecoder {
|
||||
);
|
||||
return err(
|
||||
redactExplicitFailure(
|
||||
this.withReaderCleanup(error, cause),
|
||||
withCaptureReaderCleanup(error, cause),
|
||||
input.sensitive_values,
|
||||
),
|
||||
);
|
||||
@@ -118,6 +124,7 @@ export class HistoricalCaptureDecoder {
|
||||
let runtime: Pick<PrivateRuntimeRoot, "path" | "close"> | undefined;
|
||||
let result: Result<WebNetworkCapture, AnalysisError>;
|
||||
let phase: "capture-read" | "decoder" = "capture-read";
|
||||
let encodedReplyBytes = 0;
|
||||
try {
|
||||
const adapter = this.adapters.find(
|
||||
(candidate) => candidate.format === input.format,
|
||||
@@ -186,6 +193,7 @@ export class HistoricalCaptureDecoder {
|
||||
undefined,
|
||||
this.#resources,
|
||||
);
|
||||
encodedReplyBytes = replyFile.bytes.length;
|
||||
reply = replySchema.parse(JSON.parse(replyFile.bytes.toString("utf8")));
|
||||
} catch (cause: unknown) {
|
||||
if (cause instanceof ArtifactReaderFailure) throw cause;
|
||||
@@ -295,11 +303,58 @@ export class HistoricalCaptureDecoder {
|
||||
);
|
||||
result = ok({ ...reply.value, ...common, format: input.format });
|
||||
}
|
||||
if (result.ok) {
|
||||
let safePartialSource = false;
|
||||
try {
|
||||
const prepared = await prepareCaptureProtobuf(
|
||||
result.value,
|
||||
input,
|
||||
options?.signal,
|
||||
);
|
||||
safePartialSource = true;
|
||||
await decodeCaptureProtobuf(
|
||||
result.value,
|
||||
input,
|
||||
this.#resources,
|
||||
prepared,
|
||||
options?.signal,
|
||||
encodedReplyBytes + Buffer.byteLength(JSON.stringify(common)) + 16,
|
||||
);
|
||||
} catch (cause) {
|
||||
const error =
|
||||
cause instanceof CaptureFormatError && cause.reason === "limit"
|
||||
? new AnalysisResourceConstraintError(
|
||||
OPERATION,
|
||||
"capacity",
|
||||
cause.message,
|
||||
{
|
||||
maximum_encoded_capture_bytes:
|
||||
WEB_NETWORK_CAPTURE_LIMITS.outputBytes,
|
||||
},
|
||||
{
|
||||
retryable: false,
|
||||
remediationAction:
|
||||
"Select smaller records, or inspect the retained original source bytes.",
|
||||
},
|
||||
)
|
||||
: cause instanceof CaptureFormatError
|
||||
? new AnalysisInputError(OPERATION, undefined, [
|
||||
{
|
||||
path: ["protobuf"],
|
||||
reason: "invalid_format",
|
||||
message: cause.message,
|
||||
},
|
||||
])
|
||||
: historicalCaptureFailure(input, cause, phase, options);
|
||||
if (safePartialSource) error.retainPartialObservation(result.value);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
} catch (cause: unknown) {
|
||||
const error = historicalCaptureFailure(input, cause, phase, options);
|
||||
result = err(
|
||||
redactExplicitFailure(
|
||||
this.withReaderCleanup(error, cause),
|
||||
withCaptureReaderCleanup(error, cause),
|
||||
input.sensitive_values,
|
||||
),
|
||||
);
|
||||
@@ -358,41 +413,6 @@ export class HistoricalCaptureDecoder {
|
||||
result.error.retainPartialObservation(completedCapture);
|
||||
return result;
|
||||
}
|
||||
|
||||
private withReaderCleanup(
|
||||
primary: AnalysisError,
|
||||
cause: unknown,
|
||||
): AnalysisError {
|
||||
if (
|
||||
!(cause instanceof ArtifactReaderFailure) ||
|
||||
cause.cleanup === undefined
|
||||
)
|
||||
return primary;
|
||||
if (cause.reason === "unavailable") {
|
||||
const cleanup = new ProviderCleanupError(
|
||||
"web-network-capture",
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ operation: OPERATION, cause },
|
||||
);
|
||||
bindArtifactFileCleanup(cause, cleanup);
|
||||
return cleanup;
|
||||
}
|
||||
const cleanup = new ProviderCleanupError(
|
||||
"web-network-capture",
|
||||
cause.cleanup.resources,
|
||||
{ reason: cause.cleanup.reason },
|
||||
{ operation: OPERATION, cause },
|
||||
);
|
||||
bindArtifactFileCleanup(cause, cleanup);
|
||||
const composed = analysisErrorWithCleanupFailure(
|
||||
primary,
|
||||
cleanup,
|
||||
OPERATION,
|
||||
).bindCleanupDependencies([primary, cleanup]);
|
||||
bindArtifactFileCleanup(cause, composed);
|
||||
return composed;
|
||||
}
|
||||
}
|
||||
|
||||
/** The Node capture decoder runs on this executable; only Node.js enforces its V8 heap flags. */
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
import {
|
||||
readGrpcFrame,
|
||||
prepareProtobufBody,
|
||||
type PreparedProtobufBody,
|
||||
} from "./ProtobufFraming.js";
|
||||
import { CaptureFormatError } from "../CaptureFormatError.js";
|
||||
import type {
|
||||
ProtobufDecoding,
|
||||
ProtobufFrame,
|
||||
} from "../../../domain/protobufWire.js";
|
||||
import {
|
||||
decodeProtobufWire,
|
||||
ProtobufBudget,
|
||||
type WireSchema,
|
||||
} from "./ProtobufWire.js";
|
||||
|
||||
/** Framing is protocol-specific: native gRPC has no in-body trailer frames. */
|
||||
export const PROTOBUF_FRAMING = {
|
||||
"application/grpc": "grpc",
|
||||
"application/grpc+proto": "grpc",
|
||||
"application/grpc-web": "grpc-web",
|
||||
"application/grpc-web+proto": "grpc-web",
|
||||
"application/grpc-web-text": "grpc-web-text",
|
||||
"application/grpc-web-text+proto": "grpc-web-text",
|
||||
"application/x-protobuf": "raw",
|
||||
"application/protobuf": "raw",
|
||||
"application/vnd.google.protobuf": "raw",
|
||||
} as const;
|
||||
|
||||
export const PROTOBUF_LIMITATIONS = [
|
||||
"Wire numbers and ranges are observations. Length-delimited string, nested-message and packed readings are candidates without a supplied schema. Raw decoding does not infer field names, signedness or business meaning; nested candidate bytes remain accessible through their ranges.",
|
||||
"Schema values are interpretations of the caller-supplied descriptor, not proof that the sender used that schema. Map semantics, extensions, required-field validation and unknown schema fields are not reconstructed.",
|
||||
] as const;
|
||||
|
||||
export const decodeProtobufBody = async (
|
||||
body: Uint8Array,
|
||||
framing: ProtobufDecoding["framing"],
|
||||
source: Pick<ProtobufDecoding, "pointer" | "content_type" | "body_sha256">,
|
||||
redact: (value: string) => string,
|
||||
budget: ProtobufBudget,
|
||||
schema?: WireSchema,
|
||||
prepared?: PreparedProtobufBody,
|
||||
): Promise<ProtobufDecoding> => {
|
||||
const result: ProtobufDecoding = {
|
||||
...source,
|
||||
decoder: "rea-protobuf-wire-v1",
|
||||
framing,
|
||||
offset_basis:
|
||||
framing === "grpc-web-text" ? "base64-decoded-body" : "retained-body",
|
||||
decoded_bytes: body.length,
|
||||
status: "complete",
|
||||
examined_bytes: 0,
|
||||
frames: [],
|
||||
fields: [],
|
||||
limitations: [],
|
||||
};
|
||||
const stop = (offset: number, reason: string): ProtobufDecoding => {
|
||||
result.status = "partial";
|
||||
result.error = { offset, reason: redact(reason) };
|
||||
return result;
|
||||
};
|
||||
// The production source pass already decoded text. Charge representation expansion
|
||||
// here only after that pass made the retained source safe on failure.
|
||||
if (framing === "grpc-web-text") budget.charge(body.length * 6);
|
||||
const data =
|
||||
prepared === undefined
|
||||
? (
|
||||
await prepareProtobufBody(
|
||||
body,
|
||||
framing,
|
||||
{ decoded: () => false, header: () => false },
|
||||
budget.signal,
|
||||
)
|
||||
).data
|
||||
: prepared.data;
|
||||
if (data === undefined)
|
||||
return stop(
|
||||
0,
|
||||
"gRPC-web text is not canonical base64 or independently padded base64 chunks",
|
||||
);
|
||||
result.decoded_bytes = data.length;
|
||||
let at = 0;
|
||||
const addMessage = async (frame: ProtobufFrame) => {
|
||||
if (frame.compressed) {
|
||||
frame.status = "unknown";
|
||||
frame.error = {
|
||||
offset: frame.payload_offset,
|
||||
reason:
|
||||
"Compressed gRPC payload requires its negotiated codec; source bytes remain retained",
|
||||
};
|
||||
return;
|
||||
}
|
||||
const decoded = await decodeProtobufWire(
|
||||
data,
|
||||
frame.payload_offset,
|
||||
frame.payload_offset + frame.bytes,
|
||||
frame.index,
|
||||
budget,
|
||||
redact,
|
||||
schema,
|
||||
result.fields.length,
|
||||
);
|
||||
for (const field of decoded.fields) result.fields.push(field);
|
||||
frame.examined_bytes = decoded.examined_bytes;
|
||||
if (decoded.error !== undefined) {
|
||||
frame.status = "partial";
|
||||
frame.error = decoded.error;
|
||||
}
|
||||
};
|
||||
if (framing === "raw") {
|
||||
const frame: ProtobufFrame = {
|
||||
index: 0,
|
||||
kind: "data",
|
||||
offset: 0,
|
||||
payload_offset: 0,
|
||||
bytes: data.length,
|
||||
compressed: false,
|
||||
status: "complete",
|
||||
examined_bytes: 0,
|
||||
};
|
||||
result.frames.push(frame);
|
||||
await addMessage(frame);
|
||||
result.status = frame.status;
|
||||
result.examined_bytes = frame.examined_bytes;
|
||||
return result;
|
||||
}
|
||||
let trailerSeen = false;
|
||||
while (at < data.length) {
|
||||
await budget.checkpoint();
|
||||
if (trailerSeen)
|
||||
return stop(at, "A gRPC-web trailer frame must terminate the body");
|
||||
const parsed = readGrpcFrame(data, at, framing);
|
||||
if (!parsed.ok) return stop(parsed.error.offset, parsed.error.reason);
|
||||
const { flags, bytes: length } = parsed.value;
|
||||
budget.charge(1024);
|
||||
const frame: ProtobufFrame = {
|
||||
index: result.frames.length,
|
||||
kind: flags & 128 ? "trailer" : "data",
|
||||
offset: at,
|
||||
payload_offset: at + 5,
|
||||
bytes: length,
|
||||
compressed: (flags & 1) !== 0,
|
||||
status: "complete",
|
||||
examined_bytes: 0,
|
||||
};
|
||||
result.frames.push(frame);
|
||||
if (frame.kind === "data") await addMessage(frame);
|
||||
else {
|
||||
trailerSeen = true;
|
||||
if (frame.compressed) {
|
||||
frame.status = "unknown";
|
||||
frame.error = {
|
||||
offset: at + 5,
|
||||
reason: "Compressed gRPC-web trailers remain opaque",
|
||||
};
|
||||
} else {
|
||||
budget.charge(length * 8);
|
||||
try {
|
||||
const text = new TextDecoder("utf-8", {
|
||||
fatal: true,
|
||||
ignoreBOM: true,
|
||||
}).decode(data.subarray(at + 5, at + 5 + length));
|
||||
frame.trailers = [];
|
||||
for (const line of text.split("\r\n")) {
|
||||
if (line === "") continue;
|
||||
const colon = line.indexOf(":");
|
||||
if (
|
||||
colon <= 0 ||
|
||||
!/^[!#$%&'*+.^_`|~0-9A-Za-z-]+$/.test(line.slice(0, colon))
|
||||
)
|
||||
throw new Error("Malformed trailer header");
|
||||
budget.charge(512);
|
||||
frame.trailers.push({
|
||||
name: redact(line.slice(0, colon)),
|
||||
value: redact(line.slice(colon + 1).replace(/^[ \t]+/, "")),
|
||||
});
|
||||
}
|
||||
frame.examined_bytes = length;
|
||||
} catch (cause) {
|
||||
if (cause instanceof CaptureFormatError) throw cause;
|
||||
frame.status = "partial";
|
||||
frame.error = {
|
||||
offset: at + 5,
|
||||
reason: "Invalid gRPC-web trailer header block",
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
if (frame.status !== "complete") result.status = "partial";
|
||||
result.examined_bytes += 5 + frame.examined_bytes;
|
||||
at += 5 + length;
|
||||
}
|
||||
return result;
|
||||
};
|
||||
@@ -0,0 +1,77 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { expect, it } from "vitest";
|
||||
import { ArtifactResourceScope } from "../../../artifacts/ArtifactResourceScope.js";
|
||||
import {
|
||||
inspectWebNetworkCaptureInputSchema,
|
||||
type WebNetworkCapture,
|
||||
} from "../../../domain/webNetworkCapture.js";
|
||||
import {
|
||||
prepareCaptureProtobuf,
|
||||
decodeCaptureProtobuf,
|
||||
} from "./ProtobufCapture.js";
|
||||
|
||||
it("restricts framing overrides to native body facts while preserving header bytes", async () => {
|
||||
const binary = (pointer: string, bytes: Buffer) => ({
|
||||
pointer,
|
||||
representation: "producer-bytes" as const,
|
||||
state: "retained" as const,
|
||||
bytes: bytes.length,
|
||||
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
content_base64: bytes.toString("base64"),
|
||||
});
|
||||
const headers = [
|
||||
binary("/response/headers/0/0", Buffer.from("content-type")),
|
||||
binary("/response/headers/0/1", Buffer.from("application/grpc")),
|
||||
];
|
||||
const report: WebNetworkCapture = {
|
||||
artifact: { path: "/fixture.mitm", bytes: 100, sha256: "a".repeat(64) },
|
||||
decoder: { id: "fixture", name: "Native producer facts", version: "1" },
|
||||
container: {
|
||||
reported: null,
|
||||
numeric_literals: [],
|
||||
redactions: [],
|
||||
records_pointer: "/flows",
|
||||
},
|
||||
format: "mitmproxy",
|
||||
total_records: 1,
|
||||
runtime_attribution: "unknown",
|
||||
limitations: [],
|
||||
records: [
|
||||
{
|
||||
ordinal: 0,
|
||||
location: { kind: "byte-range", offset: 0, bytes: 100 },
|
||||
reported: { response: { headers: [[null, null]], content: null } },
|
||||
binary_fields: [
|
||||
...headers,
|
||||
binary("/response/content", Buffer.from([0, 0, 0, 0, 2, 8, 1])),
|
||||
],
|
||||
numeric_literals: [],
|
||||
redactions: [],
|
||||
limitations: [],
|
||||
},
|
||||
],
|
||||
};
|
||||
const input = inspectWebNetworkCaptureInputSchema.parse({
|
||||
capture_path: report.artifact.path,
|
||||
format: report.format,
|
||||
record_ordinals: [0],
|
||||
protobuf: { framing: "grpc" },
|
||||
});
|
||||
const resources = new ArtifactResourceScope();
|
||||
try {
|
||||
const prepared = await prepareCaptureProtobuf(report, input);
|
||||
await decodeCaptureProtobuf(report, input, resources, prepared);
|
||||
} finally {
|
||||
await resources.close();
|
||||
}
|
||||
expect(report.records[0]?.protobuf).toMatchObject([
|
||||
{
|
||||
pointer: "/response/content",
|
||||
status: "complete",
|
||||
fields: [{ number: 1, value: "1" }],
|
||||
},
|
||||
]);
|
||||
expect(report.records[0]?.protobuf).toHaveLength(1);
|
||||
expect(report.records[0]?.binary_fields.slice(0, 2)).toEqual(headers);
|
||||
expect(report.protobuf_coverage?.status).toBe("complete");
|
||||
});
|
||||
@@ -0,0 +1,501 @@
|
||||
import { isAbsolute } from "node:path";
|
||||
import { CaptureFormatError } from "../CaptureFormatError.js";
|
||||
import {
|
||||
CaptureRedaction,
|
||||
isCaptureCredentialHeaderBytes,
|
||||
} from "../CaptureRedaction.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readStableArtifact } from "../../../artifacts/readStableArtifact.js";
|
||||
import {
|
||||
historicalCaptureFailure,
|
||||
withCaptureReaderCleanup,
|
||||
} from "../CaptureFailures.js";
|
||||
import type { ArtifactResourceScope } from "../../../artifacts/ArtifactResourceScope.js";
|
||||
import {
|
||||
AnalysisInputError,
|
||||
AnalysisCapabilityUnavailableError,
|
||||
} from "../../../domain/analysisErrorCore.js";
|
||||
import { redactExplicitText } from "../../../domain/explicitSensitiveValues.js";
|
||||
import {
|
||||
WEB_NETWORK_CAPTURE_LIMITS,
|
||||
type InspectWebNetworkCaptureInput,
|
||||
type WebNetworkCapture,
|
||||
type WebNetworkCaptureRecord,
|
||||
} from "../../../domain/webNetworkCapture.js";
|
||||
import type { JsonValue } from "../../../domain/jsonValue.js";
|
||||
import { ProtobufBudget } from "./ProtobufWire.js";
|
||||
import type { ProtobufSchema } from "./ProtobufSchema.js";
|
||||
import {
|
||||
PROTOBUF_FRAMING,
|
||||
PROTOBUF_LIMITATIONS,
|
||||
decodeProtobufBody,
|
||||
} from "./ProtobufBody.js";
|
||||
import {
|
||||
prepareProtobufBody,
|
||||
type PreparedProtobufBody,
|
||||
} from "./ProtobufFraming.js";
|
||||
import type { ProtobufDecoding } from "../../../domain/protobufWire.js";
|
||||
|
||||
const object = (
|
||||
value: JsonValue | undefined,
|
||||
): Readonly<Record<string, JsonValue>> | undefined =>
|
||||
value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? value
|
||||
: undefined;
|
||||
|
||||
/** Exclude the original encoded representation as well as its decoding sidecar. */
|
||||
const excludeBody = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
pointer: string,
|
||||
reason: "explicit-sensitive-value" | "transport-credential",
|
||||
) => {
|
||||
record.binary_fields = record.binary_fields.map((field) =>
|
||||
field.pointer !== pointer
|
||||
? field
|
||||
: {
|
||||
pointer: field.pointer,
|
||||
representation: field.representation,
|
||||
state: "redacted",
|
||||
content_base64: null,
|
||||
bytes: null,
|
||||
sha256: null,
|
||||
},
|
||||
);
|
||||
const parts = pointer
|
||||
.split("/")
|
||||
.slice(1)
|
||||
.map((part) => part.replaceAll("~1", "/").replaceAll("~0", "~"));
|
||||
let parent: JsonValue | undefined = record.reported;
|
||||
for (const part of parts.slice(0, -1)) {
|
||||
if (
|
||||
parent === null ||
|
||||
typeof parent !== "object" ||
|
||||
!Object.hasOwn(parent, part)
|
||||
) {
|
||||
parent = undefined;
|
||||
break;
|
||||
}
|
||||
parent = Array.isArray(parent) ? parent[Number(part)] : parent[part];
|
||||
}
|
||||
const key = parts.at(-1);
|
||||
if (
|
||||
parent !== null &&
|
||||
typeof parent === "object" &&
|
||||
key !== undefined &&
|
||||
Object.hasOwn(parent, key)
|
||||
)
|
||||
Object.defineProperty(parent, key, {
|
||||
value: null,
|
||||
writable: true,
|
||||
configurable: true,
|
||||
enumerable: true,
|
||||
});
|
||||
record.redactions.push({ pointer, reason });
|
||||
};
|
||||
|
||||
/** Format-aware owner of body metadata; callers never reinterpret producer JSON. */
|
||||
const bodyMetadata = (
|
||||
record: WebNetworkCaptureRecord,
|
||||
pointer: string,
|
||||
format: string,
|
||||
binaryByPointer: ReadonlyMap<
|
||||
string,
|
||||
WebNetworkCaptureRecord["binary_fields"][number]
|
||||
>,
|
||||
) => {
|
||||
const side = /^\/(request|response)\//.exec(pointer)?.[1];
|
||||
if (
|
||||
side === undefined ||
|
||||
![
|
||||
"/request/postData/text",
|
||||
"/response/content/text",
|
||||
"/request/body",
|
||||
"/response/body",
|
||||
"/request/content",
|
||||
"/response/content",
|
||||
].includes(pointer)
|
||||
)
|
||||
return undefined;
|
||||
const message = object(object(record.reported)?.[side]);
|
||||
if (message === undefined) return undefined;
|
||||
const headers = new Map<string, string | undefined>();
|
||||
let headerNamesKnown = Array.isArray(message.headers);
|
||||
if (Array.isArray(message.headers)) {
|
||||
for (const [index, item] of message.headers.entries()) {
|
||||
const pair = object(item);
|
||||
const text = (
|
||||
value: JsonValue | undefined,
|
||||
coordinate: string,
|
||||
): string | undefined => {
|
||||
const binary = binaryByPointer.get(coordinate);
|
||||
if (binary !== undefined)
|
||||
return binary.state === "retained"
|
||||
? Buffer.from(binary.content_base64, "base64").toString("latin1")
|
||||
: undefined;
|
||||
return typeof value === "string" ? value : undefined;
|
||||
};
|
||||
const name = Array.isArray(item)
|
||||
? text(item[0], `/${side}/headers/${index}/0`)
|
||||
: pair?.name;
|
||||
const value = Array.isArray(item)
|
||||
? text(item[1], `/${side}/headers/${index}/1`)
|
||||
: pair?.value;
|
||||
if (typeof name === "string") {
|
||||
const key = name.toLowerCase();
|
||||
const knownValue = typeof value === "string" ? value : undefined;
|
||||
headers.set(
|
||||
key,
|
||||
headers.has(key) && headers.get(key) !== knownValue
|
||||
? undefined
|
||||
: knownValue,
|
||||
);
|
||||
} else headerNamesKnown = false;
|
||||
}
|
||||
}
|
||||
if (format === "har") {
|
||||
if (
|
||||
pointer !== `/${side}/${side === "request" ? "postData" : "content"}/text`
|
||||
)
|
||||
return undefined;
|
||||
const content = object(
|
||||
message[side === "request" ? "postData" : "content"],
|
||||
);
|
||||
return {
|
||||
contentType:
|
||||
typeof content?.mimeType === "string"
|
||||
? content.mimeType
|
||||
: headers.get("content-type"),
|
||||
httpIdentityKnown: true,
|
||||
httpCodingKnown: true,
|
||||
};
|
||||
}
|
||||
if (pointer !== `/${side}/body` && pointer !== `/${side}/content`)
|
||||
return undefined;
|
||||
return {
|
||||
contentType: headers.get("content-type"),
|
||||
httpCodingKnown:
|
||||
headerNamesKnown &&
|
||||
["content-encoding", "transfer-encoding"].every(
|
||||
(name) => !headers.has(name) || headers.get(name) !== undefined,
|
||||
),
|
||||
httpIdentityKnown:
|
||||
headerNamesKnown &&
|
||||
["content-encoding", "transfer-encoding"].every(
|
||||
(name) =>
|
||||
// HTTP OWS is SP/HTAB; require the entire identity token, including EOF.
|
||||
!headers.has(name) ||
|
||||
/^[ \t]*identity[ \t]*(?![\s\S])/i.test(headers.get(name) ?? ""),
|
||||
),
|
||||
};
|
||||
};
|
||||
|
||||
type RetainedBody = Extract<
|
||||
WebNetworkCaptureRecord["binary_fields"][number],
|
||||
{ state: "retained" }
|
||||
>;
|
||||
interface PreparedBody {
|
||||
readonly field: RetainedBody;
|
||||
readonly body: Buffer;
|
||||
readonly contentType: string | undefined;
|
||||
readonly framing: ProtobufDecoding["framing"];
|
||||
readonly prepared: PreparedProtobufBody;
|
||||
}
|
||||
export interface PreparedProtobufCapture {
|
||||
readonly records: readonly {
|
||||
record: WebNetworkCaptureRecord;
|
||||
bodies: readonly PreparedBody[];
|
||||
}[];
|
||||
readonly incomplete: boolean;
|
||||
}
|
||||
|
||||
/** Sanitize every selected body before any schema acquisition or field-expansion failure.
|
||||
* Until this stage completes, the caller must not publish the original source as partial Evidence.
|
||||
*/
|
||||
export const prepareCaptureProtobuf = async (
|
||||
capture: WebNetworkCapture,
|
||||
input: InspectWebNetworkCaptureInput,
|
||||
signal?: AbortSignal,
|
||||
): Promise<PreparedProtobufCapture | undefined> => {
|
||||
if (input.record_ordinals === undefined) return undefined;
|
||||
capture.protobuf_coverage = {
|
||||
status: "partial",
|
||||
selected_records: input.record_ordinals.length,
|
||||
inspected_records: 0,
|
||||
};
|
||||
capture.limitations.push(...PROTOBUF_LIMITATIONS);
|
||||
let incomplete = false;
|
||||
const records: { record: WebNetworkCaptureRecord; bodies: PreparedBody[] }[] =
|
||||
[];
|
||||
const options = input.protobuf;
|
||||
const pointers =
|
||||
options?.body_pointers === undefined
|
||||
? undefined
|
||||
: new Set(options.body_pointers);
|
||||
const redactions = new CaptureRedaction(input.sensitive_values);
|
||||
const work = new ProtobufBudget(signal);
|
||||
for (const ordinal of input.record_ordinals) {
|
||||
await work.checkpoint();
|
||||
const record = capture.records[ordinal];
|
||||
if (record === undefined) continue;
|
||||
const bodies: PreparedBody[] = [];
|
||||
records.push({ record, bodies });
|
||||
const binaryByPointer = new Map(
|
||||
record.binary_fields.map((field) => [field.pointer, field]),
|
||||
);
|
||||
for (const field of record.binary_fields) {
|
||||
await work.checkpoint();
|
||||
if (pointers !== undefined && !pointers.has(field.pointer)) continue;
|
||||
const metadata = bodyMetadata(
|
||||
record,
|
||||
field.pointer,
|
||||
input.format,
|
||||
binaryByPointer,
|
||||
);
|
||||
// Framing overrides interpret bodies; header and extension bytes remain their own facts.
|
||||
if (metadata === undefined) continue;
|
||||
if (field.state !== "retained") {
|
||||
incomplete = true;
|
||||
continue;
|
||||
}
|
||||
const contentType = metadata.contentType;
|
||||
const essence = contentType?.split(";", 1)[0]?.trim().toLowerCase();
|
||||
const automatic =
|
||||
essence === undefined
|
||||
? undefined
|
||||
: PROTOBUF_FRAMING[essence as keyof typeof PROTOBUF_FRAMING];
|
||||
const framing =
|
||||
options?.framing === undefined || options.framing === "auto"
|
||||
? automatic
|
||||
: options.framing;
|
||||
if (framing === undefined) continue;
|
||||
if (
|
||||
field.bytes > WEB_NETWORK_CAPTURE_LIMITS.inputBytes ||
|
||||
field.content_base64.length >
|
||||
Math.ceil(WEB_NETWORK_CAPTURE_LIMITS.inputBytes / 3) * 4
|
||||
)
|
||||
throw new CaptureFormatError(
|
||||
"limit",
|
||||
"Protobuf body exceeds the admitted 32 MiB source bound",
|
||||
);
|
||||
const body = Buffer.from(field.content_base64, "base64");
|
||||
if (
|
||||
body.length !== field.bytes ||
|
||||
createHash("sha256").update(body).digest("hex") !== field.sha256
|
||||
)
|
||||
throw new AnalysisInputError("inspect_web_network_capture", undefined, [
|
||||
{
|
||||
path: ["capture_path"],
|
||||
reason: "invalid_format",
|
||||
message:
|
||||
"Retained protobuf bytes contradict their source descriptor",
|
||||
},
|
||||
]);
|
||||
const exclusions = {
|
||||
decoded: (data: Uint8Array) =>
|
||||
redactions.sensitiveBytes(
|
||||
Buffer.from(data.buffer, data.byteOffset, data.byteLength),
|
||||
),
|
||||
header: isCaptureCredentialHeaderBytes,
|
||||
};
|
||||
const prepared = await prepareProtobufBody(
|
||||
body,
|
||||
framing,
|
||||
exclusions,
|
||||
signal,
|
||||
);
|
||||
// Caller overrides do not turn off exclusions for the transport declared by the source.
|
||||
const declared =
|
||||
prepared.excluded === undefined &&
|
||||
automatic !== undefined &&
|
||||
automatic !== framing
|
||||
? await prepareProtobufBody(body, automatic, exclusions, signal)
|
||||
: prepared;
|
||||
const excluded = prepared.excluded ?? declared.excluded;
|
||||
if (excluded !== undefined) {
|
||||
excludeBody(record, field.pointer, excluded);
|
||||
binaryByPointer.delete(field.pointer);
|
||||
incomplete = true;
|
||||
continue;
|
||||
}
|
||||
// Analysis eligibility never bypasses the independent source exclusions.
|
||||
if (
|
||||
!metadata.httpCodingKnown &&
|
||||
(!prepared.credentialsChecked || !declared.credentialsChecked)
|
||||
)
|
||||
throw new AnalysisCapabilityUnavailableError(
|
||||
input.format,
|
||||
"inspect_web_network_capture",
|
||||
"Credential inspection could not complete while HTTP coding metadata was unavailable; unchecked capture bodies were withheld.",
|
||||
{
|
||||
diagnostics: { body_pointer: field.pointer },
|
||||
userMessage:
|
||||
"Use a capture with available HTTP coding metadata, or select records whose transport can be inspected. Unchecked source bytes were not published.",
|
||||
},
|
||||
);
|
||||
if (!metadata.httpIdentityKnown) {
|
||||
incomplete = true;
|
||||
record.limitations.push(
|
||||
"Selected protobuf body's HTTP content/transfer coding is encoded or unknown, so identity is not established; its exact stored bytes remain opaque.",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
bodies.push({ field, body, contentType, framing, prepared });
|
||||
}
|
||||
}
|
||||
return { records, incomplete };
|
||||
};
|
||||
|
||||
/** Interpret the sanitized source once, inside the capture adapter's owned operation. */
|
||||
export const decodeCaptureProtobuf = async (
|
||||
capture: WebNetworkCapture,
|
||||
input: InspectWebNetworkCaptureInput,
|
||||
resources: ArtifactResourceScope,
|
||||
prepared: PreparedProtobufCapture | undefined,
|
||||
signal?: AbortSignal,
|
||||
encodedSourceBytes = 0,
|
||||
): Promise<void> => {
|
||||
if (prepared === undefined || capture.protobuf_coverage === undefined) return;
|
||||
let incomplete = prepared.incomplete;
|
||||
const budget = new ProtobufBudget(signal);
|
||||
budget.charge(
|
||||
encodedSourceBytes +
|
||||
256 +
|
||||
Buffer.byteLength(JSON.stringify(PROTOBUF_LIMITATIONS)),
|
||||
);
|
||||
for (const entry of prepared.records)
|
||||
for (const item of entry.bodies)
|
||||
budget.charge(
|
||||
item.body.length +
|
||||
(item.prepared.data === item.body
|
||||
? 0
|
||||
: (item.prepared.data?.length ?? 0)),
|
||||
);
|
||||
const redact = (text: string) => {
|
||||
if (input.sensitive_values.length === 0) return text;
|
||||
if (input.sensitive_values.some((literal) => text.includes(literal)))
|
||||
budget.charge(text.length * 60);
|
||||
return redactExplicitText(text, input.sensitive_values);
|
||||
};
|
||||
const options = input.protobuf;
|
||||
let schema: ProtobufSchema | undefined;
|
||||
let schemaSource: ProtobufDecoding["schema"];
|
||||
if (options?.schema !== undefined) {
|
||||
const path = options.schema.descriptor_set_path;
|
||||
if (!isAbsolute(path))
|
||||
throw new AnalysisInputError("inspect_web_network_capture", undefined, [
|
||||
{
|
||||
path: ["protobuf", "schema", "descriptor_set_path"],
|
||||
reason: "invalid_format",
|
||||
message: "Expected an absolute descriptor-set path",
|
||||
},
|
||||
]);
|
||||
const snapshot = await readStableArtifact(
|
||||
path,
|
||||
WEB_NETWORK_CAPTURE_LIMITS.inputBytes,
|
||||
signal,
|
||||
undefined,
|
||||
resources,
|
||||
).catch((cause: unknown) => {
|
||||
throw withCaptureReaderCleanup(
|
||||
historicalCaptureFailure(
|
||||
input,
|
||||
cause,
|
||||
"descriptor-read",
|
||||
signal === undefined ? undefined : { signal },
|
||||
),
|
||||
cause,
|
||||
);
|
||||
});
|
||||
const { ProtobufSchema } = await import("./ProtobufSchema.js");
|
||||
try {
|
||||
schema = await ProtobufSchema.read(
|
||||
snapshot.bytes,
|
||||
options.schema.message_type,
|
||||
budget,
|
||||
redact,
|
||||
);
|
||||
} catch (cause) {
|
||||
if (signal?.aborted === true && cause === signal.reason) throw cause;
|
||||
if (cause instanceof CaptureFormatError) throw cause;
|
||||
throw new AnalysisInputError("inspect_web_network_capture", undefined, [
|
||||
{
|
||||
path: ["protobuf", "schema"],
|
||||
reason: "invalid_format",
|
||||
message:
|
||||
"The selected local FileDescriptorSet or message type is invalid or exceeds decoding budgets",
|
||||
},
|
||||
]);
|
||||
}
|
||||
schemaSource = {
|
||||
decoder: "protobuf-es-2.16.0",
|
||||
descriptor_set_path: redact(path),
|
||||
descriptor_set_sha256: snapshot.sha256,
|
||||
descriptor_set_bytes: snapshot.bytes.length,
|
||||
message_type: redact(options.schema.message_type),
|
||||
};
|
||||
}
|
||||
for (const { record, bodies } of prepared.records) {
|
||||
const decoded: ProtobufDecoding[] = [];
|
||||
for (const {
|
||||
field,
|
||||
body,
|
||||
contentType,
|
||||
framing,
|
||||
prepared: bodySource,
|
||||
} of bodies) {
|
||||
await budget.checkpoint();
|
||||
if (
|
||||
input.sensitive_values.some(
|
||||
(value) =>
|
||||
field.pointer.includes(value) ||
|
||||
field.sha256.includes(value) ||
|
||||
schemaSource?.descriptor_set_sha256.includes(value),
|
||||
)
|
||||
) {
|
||||
incomplete = true;
|
||||
continue;
|
||||
}
|
||||
budget.charge(
|
||||
2048 +
|
||||
8 *
|
||||
(field.pointer.length +
|
||||
(contentType?.length ?? 0) +
|
||||
(schemaSource?.descriptor_set_path.length ?? 0) +
|
||||
(schemaSource?.message_type.length ?? 0)),
|
||||
);
|
||||
schema?.reset();
|
||||
const result = await decodeProtobufBody(
|
||||
body,
|
||||
framing,
|
||||
{
|
||||
pointer: field.pointer,
|
||||
content_type: contentType ?? null,
|
||||
body_sha256: field.sha256,
|
||||
},
|
||||
redact,
|
||||
budget,
|
||||
schema,
|
||||
bodySource,
|
||||
);
|
||||
if (schemaSource !== undefined) result.schema = schemaSource;
|
||||
if (
|
||||
result.status !== "complete" ||
|
||||
result.fields.some(
|
||||
(field) =>
|
||||
field.schema_status !== undefined &&
|
||||
field.schema_status !== "decoded",
|
||||
)
|
||||
)
|
||||
incomplete = true;
|
||||
decoded.push(result);
|
||||
}
|
||||
if (decoded.length > 0) record.protobuf = decoded;
|
||||
capture.protobuf_coverage.inspected_records++;
|
||||
}
|
||||
if (
|
||||
!incomplete &&
|
||||
capture.protobuf_coverage.inspected_records ===
|
||||
capture.protobuf_coverage.selected_records
|
||||
)
|
||||
capture.protobuf_coverage.status = "complete";
|
||||
};
|
||||
@@ -0,0 +1,144 @@
|
||||
import { err, ok } from "../../../domain/result.js";
|
||||
import type { ProtobufDecoding } from "../../../domain/protobufWire.js";
|
||||
import { WEB_NETWORK_CAPTURE_LIMITS } from "../../../domain/webNetworkCapture.js";
|
||||
import { CaptureFormatError } from "../CaptureFormatError.js";
|
||||
import { ProtobufBudget } from "./ProtobufWire.js";
|
||||
|
||||
/** One owner for frame boundaries used by exclusion and field interpretation. */
|
||||
export const readGrpcFrame = (
|
||||
data: Uint8Array,
|
||||
at: number,
|
||||
framing: Exclude<ProtobufDecoding["framing"], "raw">,
|
||||
) => {
|
||||
if (data.length - at < 5)
|
||||
return err({ offset: at, reason: "Truncated gRPC frame header" });
|
||||
const flags = data[at]!;
|
||||
if (
|
||||
(framing === "grpc" && flags !== 0 && flags !== 1) ||
|
||||
(flags & ~0x81) !== 0
|
||||
)
|
||||
return err({
|
||||
offset: at,
|
||||
reason: "Invalid flags for the selected gRPC framing",
|
||||
});
|
||||
const length =
|
||||
data[at + 1]! * 2 ** 24 +
|
||||
data[at + 2]! * 2 ** 16 +
|
||||
data[at + 3]! * 256 +
|
||||
data[at + 4]!;
|
||||
if (length > data.length - at - 5)
|
||||
return err({ offset: at, reason: "gRPC frame exceeds the retained body" });
|
||||
return ok({
|
||||
flags,
|
||||
bytes: length,
|
||||
payload_offset: at + 5,
|
||||
end: at + 5 + length,
|
||||
});
|
||||
};
|
||||
|
||||
/** Decode padded stream chunks into one source-bounded buffer, never a chunk inventory. */
|
||||
const decodeText = async (
|
||||
body: Uint8Array,
|
||||
work: ProtobufBudget,
|
||||
): Promise<Uint8Array | undefined> => {
|
||||
const text = Buffer.from(body.buffer, body.byteOffset, body.byteLength)
|
||||
.toString("latin1")
|
||||
.replace(/[ \r\n\t]/g, "");
|
||||
if (/[^A-Za-z0-9+/=]/.test(text)) return undefined;
|
||||
const output = Buffer.alloc(Math.floor((text.length * 3) / 4));
|
||||
let at = 0;
|
||||
let written = 0;
|
||||
while (at < text.length) {
|
||||
await work.checkpoint();
|
||||
let end = text.indexOf("=", at);
|
||||
if (end < 0) end = text.length;
|
||||
else {
|
||||
end++;
|
||||
if (text[end] === "=") end++;
|
||||
}
|
||||
const chunk = text.slice(at, end);
|
||||
if (
|
||||
chunk.length % 4 === 1 ||
|
||||
(chunk.includes("=") && chunk.length % 4 !== 0)
|
||||
)
|
||||
return undefined;
|
||||
const bytes = Buffer.from(chunk, "base64");
|
||||
if (
|
||||
bytes.toString("base64").replace(/=+$/, "") !== chunk.replace(/=+$/, "")
|
||||
)
|
||||
return undefined;
|
||||
output.set(bytes, written);
|
||||
written += bytes.length;
|
||||
at = end;
|
||||
}
|
||||
return output.subarray(0, written);
|
||||
};
|
||||
|
||||
export interface PreparedProtobufBody {
|
||||
readonly data: Uint8Array | undefined;
|
||||
readonly credentialsChecked: boolean;
|
||||
readonly excluded?: "explicit-sensitive-value" | "transport-credential";
|
||||
}
|
||||
|
||||
/** Complete credential inspection before schema I/O, fact budgets or recoverable errors.
|
||||
* Its temporary storage is bounded by admitted source bytes, independently of field expansion.
|
||||
*/
|
||||
export const prepareProtobufBody = async (
|
||||
body: Uint8Array,
|
||||
framing: ProtobufDecoding["framing"],
|
||||
exclusions: {
|
||||
decoded(bytes: Uint8Array): boolean;
|
||||
header(bytes: Uint8Array, start: number, end: number): boolean;
|
||||
},
|
||||
signal?: AbortSignal,
|
||||
): Promise<PreparedProtobufBody> => {
|
||||
if (body.length > WEB_NETWORK_CAPTURE_LIMITS.inputBytes)
|
||||
throw new CaptureFormatError(
|
||||
"limit",
|
||||
"Protobuf body exceeds the admitted 32 MiB source bound",
|
||||
);
|
||||
const work = new ProtobufBudget(signal);
|
||||
await work.checkpoint();
|
||||
const data =
|
||||
framing === "grpc-web-text" ? await decodeText(body, work) : body;
|
||||
if (data === undefined) return { data, credentialsChecked: false };
|
||||
if (exclusions.decoded(data))
|
||||
return {
|
||||
data,
|
||||
credentialsChecked: true,
|
||||
excluded: "explicit-sensitive-value",
|
||||
};
|
||||
if (framing === "raw" || framing === "grpc")
|
||||
return { data, credentialsChecked: true };
|
||||
let at = 0;
|
||||
let credentialsChecked = true;
|
||||
while (at < data.length) {
|
||||
await work.checkpoint();
|
||||
const parsed = readGrpcFrame(data, at, framing);
|
||||
if (!parsed.ok) {
|
||||
credentialsChecked = false;
|
||||
break;
|
||||
}
|
||||
const frame = parsed.value;
|
||||
if (frame.flags === 129) credentialsChecked = false;
|
||||
if (frame.flags === 128) {
|
||||
let start = frame.payload_offset;
|
||||
for (let index = start; index <= frame.end; index++) {
|
||||
// Large/malformed lines remain source-bounded and cancellation stays cooperative.
|
||||
if ((index - start) % 256 === 0) await work.checkpoint();
|
||||
if (index !== frame.end && data[index] !== 10) continue;
|
||||
let colon = start;
|
||||
while (colon < index && data[colon] !== 58) colon++;
|
||||
if (colon < index && exclusions.header(data, start, colon))
|
||||
return {
|
||||
data,
|
||||
credentialsChecked: true,
|
||||
excluded: "transport-credential",
|
||||
};
|
||||
start = index + 1;
|
||||
}
|
||||
}
|
||||
at = frame.end;
|
||||
}
|
||||
return { data, credentialsChecked };
|
||||
};
|
||||
@@ -0,0 +1,299 @@
|
||||
import {
|
||||
createFileRegistry,
|
||||
fromBinary,
|
||||
ScalarType,
|
||||
type DescField,
|
||||
type DescEnum,
|
||||
type DescMessage,
|
||||
} from "@bufbuild/protobuf";
|
||||
import {
|
||||
FileDescriptorSetSchema,
|
||||
type FileDescriptorSet,
|
||||
} from "@bufbuild/protobuf/wkt";
|
||||
import { BinaryReader } from "@bufbuild/protobuf/wire";
|
||||
import type { ProtobufField } from "../../../domain/protobufWire.js";
|
||||
import { WEB_NETWORK_CAPTURE_LIMITS } from "../../../domain/webNetworkCapture.js";
|
||||
import { CaptureFormatError } from "../CaptureFormatError.js";
|
||||
import {
|
||||
decodeProtobufWire,
|
||||
readProtobufVarint,
|
||||
type ProtobufBudget,
|
||||
type WireSchema,
|
||||
} from "./ProtobufWire.js";
|
||||
|
||||
const scalarWire = (scalar: ScalarType): ProtobufField["wire_type"] => {
|
||||
if (scalar === ScalarType.STRING || scalar === ScalarType.BYTES) return "len";
|
||||
if (
|
||||
[ScalarType.DOUBLE, ScalarType.FIXED64, ScalarType.SFIXED64].includes(
|
||||
scalar,
|
||||
)
|
||||
)
|
||||
return "i64";
|
||||
if (
|
||||
[ScalarType.FLOAT, ScalarType.FIXED32, ScalarType.SFIXED32].includes(scalar)
|
||||
)
|
||||
return "i32";
|
||||
return "varint";
|
||||
};
|
||||
|
||||
const floatingValue = (value: number): string =>
|
||||
Object.is(value, -0) ? "-0" : String(value);
|
||||
|
||||
/** Qualified names can expand a small descriptor into a much larger registry. */
|
||||
const chargeRegistryExpansion = async (
|
||||
descriptor: FileDescriptorSet,
|
||||
budget: ProtobufBudget,
|
||||
): Promise<void> => {
|
||||
for (const file of descriptor.file) {
|
||||
budget.charge(file.name.length * 8 + 128);
|
||||
const stack = file.messageType.map((message) => ({
|
||||
message,
|
||||
prefix: file.package.length,
|
||||
}));
|
||||
const chargeQualifiedName = (name: string, prefix: number) =>
|
||||
budget.charge((prefix + 1 + name.length) * 8 + 128);
|
||||
for (const enumeration of file.enumType)
|
||||
chargeQualifiedName(enumeration.name, file.package.length);
|
||||
for (const extension of file.extension)
|
||||
chargeQualifiedName(extension.name, file.package.length);
|
||||
for (const service of file.service) {
|
||||
const length = file.package.length + 1 + service.name.length;
|
||||
budget.charge(length * 8 + 128);
|
||||
for (const method of service.method)
|
||||
chargeQualifiedName(method.name, length);
|
||||
}
|
||||
while (stack.length > 0) {
|
||||
await budget.checkpoint();
|
||||
const { message, prefix } = stack.pop()!;
|
||||
const length = prefix + 1 + message.name.length;
|
||||
budget.charge(length * 8 + 128);
|
||||
for (const field of message.field)
|
||||
chargeQualifiedName(field.name, length);
|
||||
for (const extension of message.extension)
|
||||
chargeQualifiedName(extension.name, length);
|
||||
for (const enumeration of message.enumType)
|
||||
chargeQualifiedName(enumeration.name, length);
|
||||
for (const nested of message.nestedType)
|
||||
stack.push({ message: nested, prefix: length });
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const scalarValue = (reader: BinaryReader, scalar: ScalarType): string => {
|
||||
switch (scalar) {
|
||||
case ScalarType.DOUBLE:
|
||||
return floatingValue(reader.double());
|
||||
case ScalarType.FLOAT:
|
||||
return floatingValue(reader.float());
|
||||
case ScalarType.INT64:
|
||||
return String(reader.int64());
|
||||
case ScalarType.UINT64:
|
||||
return String(reader.uint64());
|
||||
case ScalarType.INT32:
|
||||
return String(reader.int32());
|
||||
case ScalarType.FIXED64:
|
||||
return String(reader.fixed64());
|
||||
case ScalarType.FIXED32:
|
||||
return String(reader.fixed32());
|
||||
case ScalarType.BOOL:
|
||||
return String(reader.bool());
|
||||
case ScalarType.UINT32:
|
||||
return String(reader.uint32());
|
||||
case ScalarType.SFIXED32:
|
||||
return String(reader.sfixed32());
|
||||
case ScalarType.SFIXED64:
|
||||
return String(reader.sfixed64());
|
||||
case ScalarType.SINT32:
|
||||
return String(reader.sint32());
|
||||
case ScalarType.SINT64:
|
||||
return String(reader.sint64());
|
||||
default:
|
||||
throw new Error("Not a numeric protobuf scalar");
|
||||
}
|
||||
};
|
||||
|
||||
/** The supplied schema interprets occurrences; it never overwrites raw wire facts. */
|
||||
export class ProtobufSchema implements WireSchema {
|
||||
readonly #fields = new Map<DescMessage, ReadonlyMap<number, DescField>>();
|
||||
readonly #messages = new Map<number, DescMessage>();
|
||||
readonly #enums = new WeakMap<
|
||||
DescEnum,
|
||||
ReadonlyMap<string, readonly string[]>
|
||||
>();
|
||||
constructor(
|
||||
readonly message: DescMessage,
|
||||
readonly budget: ProtobufBudget,
|
||||
readonly redact: (text: string) => string,
|
||||
) {}
|
||||
|
||||
static async read(
|
||||
bytes: Uint8Array,
|
||||
typeName: string,
|
||||
budget: ProtobufBudget,
|
||||
redact: (text: string) => string,
|
||||
): Promise<ProtobufSchema> {
|
||||
// Bound the object expansion of the descriptor before the SDK allocates its message tree.
|
||||
const checked = await decodeProtobufWire(
|
||||
bytes,
|
||||
0,
|
||||
bytes.length,
|
||||
0,
|
||||
budget,
|
||||
redact,
|
||||
new ProtobufSchema(FileDescriptorSetSchema, budget, redact),
|
||||
);
|
||||
if (
|
||||
checked.error !== undefined ||
|
||||
checked.fields.some((field) => field.schema_status === "invalid-message")
|
||||
)
|
||||
throw new Error("Malformed FileDescriptorSet wire format");
|
||||
const descriptor = fromBinary(FileDescriptorSetSchema, bytes, {
|
||||
readUnknownFields: false,
|
||||
recursionLimit: WEB_NETWORK_CAPTURE_LIMITS.depth,
|
||||
});
|
||||
await chargeRegistryExpansion(descriptor, budget);
|
||||
const message = createFileRegistry(descriptor).getMessage(
|
||||
typeName.replace(/^\./, ""),
|
||||
);
|
||||
if (message === undefined)
|
||||
throw new Error(
|
||||
"The supplied descriptor set does not define the selected message type",
|
||||
);
|
||||
return new ProtobufSchema(message, budget, redact);
|
||||
}
|
||||
|
||||
reset(): void {
|
||||
this.#messages.clear();
|
||||
}
|
||||
|
||||
nested(field: ProtobufField): boolean {
|
||||
return this.#messages.has(field.index);
|
||||
}
|
||||
|
||||
async annotate(
|
||||
field: ProtobufField,
|
||||
bytes: Uint8Array,
|
||||
parent: ProtobufField | undefined,
|
||||
): Promise<void> {
|
||||
const message =
|
||||
parent === undefined ? this.message : this.#messages.get(parent.index);
|
||||
if (message === undefined) return;
|
||||
let fields = this.#fields.get(message);
|
||||
if (fields === undefined) {
|
||||
fields = new Map(message.fields.map((item) => [item.number, item]));
|
||||
this.#fields.set(message, fields);
|
||||
}
|
||||
const desc = fields.get(field.number);
|
||||
if (desc === undefined) {
|
||||
field.schema_status = "unknown-field";
|
||||
return;
|
||||
}
|
||||
this.budget.charge(
|
||||
8 *
|
||||
(desc.name.length +
|
||||
(desc.message?.typeName.length ?? 0) +
|
||||
(desc.enum?.typeName.length ?? 0)),
|
||||
);
|
||||
field.schema_name = this.redact(desc.name);
|
||||
field.schema_status = "decoded";
|
||||
if (desc.fieldKind === "map") {
|
||||
field.schema_type = "map";
|
||||
field.schema_status = "unsupported";
|
||||
return;
|
||||
}
|
||||
if (desc.message !== undefined) {
|
||||
field.schema_type = this.redact(desc.message.typeName);
|
||||
const expected = desc.delimitedEncoding ? "group" : "len";
|
||||
if (field.wire_type !== expected)
|
||||
field.schema_status = "wire-type-mismatch";
|
||||
else this.#messages.set(field.index, desc.message);
|
||||
return;
|
||||
}
|
||||
const scalar = desc.enum === undefined ? desc.scalar : ScalarType.INT32;
|
||||
if (scalar === undefined) {
|
||||
field.schema_status = "unsupported";
|
||||
return;
|
||||
}
|
||||
field.schema_type =
|
||||
desc.enum === undefined
|
||||
? ScalarType[scalar].toLowerCase()
|
||||
: this.redact(desc.enum.typeName);
|
||||
const packed =
|
||||
desc.fieldKind === "list" &&
|
||||
field.wire_type === "len" &&
|
||||
scalar !== ScalarType.STRING &&
|
||||
scalar !== ScalarType.BYTES;
|
||||
if (!packed && field.wire_type !== scalarWire(scalar)) {
|
||||
field.schema_status = "wire-type-mismatch";
|
||||
return;
|
||||
}
|
||||
const payload = bytes.subarray(
|
||||
field.payload_offset,
|
||||
field.payload_offset + field.payload_bytes,
|
||||
);
|
||||
try {
|
||||
if (scalar === ScalarType.BYTES) return; // Exact bytes already have an accessible source range.
|
||||
if (scalar === ScalarType.STRING) {
|
||||
// Wire decoding owns UTF-8 (including BOM) and literal exclusions.
|
||||
// Reuse that fact; charge only its additional serialized representation.
|
||||
this.budget.charge(payload.length * 6);
|
||||
if (field.text === undefined)
|
||||
field.schema_status = "wire-type-mismatch";
|
||||
else field.schema_value = field.text;
|
||||
return;
|
||||
}
|
||||
const reader = new BinaryReader(payload);
|
||||
const enumValues = new Map<string, readonly string[]>();
|
||||
const read = () => {
|
||||
if (
|
||||
scalarWire(scalar) === "varint" &&
|
||||
readProtobufVarint(payload, reader.pos) === undefined
|
||||
)
|
||||
throw new Error("Invalid packed varint");
|
||||
const value = scalarValue(reader, scalar);
|
||||
if (desc.enum !== undefined) {
|
||||
let names = this.#enums.get(desc.enum);
|
||||
if (names === undefined) {
|
||||
this.budget.charge(desc.enum.values.length * 256);
|
||||
const indexed = new Map<string, string[]>();
|
||||
for (const item of desc.enum.values) {
|
||||
const number = String(item.number);
|
||||
const aliases = indexed.get(number) ?? [];
|
||||
aliases.push(item.name);
|
||||
indexed.set(number, aliases);
|
||||
}
|
||||
names = indexed;
|
||||
this.#enums.set(desc.enum, names);
|
||||
}
|
||||
const aliases = names.get(value);
|
||||
if (aliases !== undefined && !enumValues.has(value)) {
|
||||
this.budget.charge(
|
||||
256 + aliases.reduce((sum, name) => sum + name.length * 8, 0),
|
||||
);
|
||||
enumValues.set(value, aliases.map(this.redact));
|
||||
}
|
||||
}
|
||||
return value;
|
||||
};
|
||||
if (packed) {
|
||||
const values: string[] = [];
|
||||
while (reader.pos < reader.len) {
|
||||
await this.budget.checkpoint();
|
||||
this.budget.charge(192);
|
||||
values.push(this.redact(read()));
|
||||
}
|
||||
field.schema_value = JSON.stringify(values);
|
||||
} else field.schema_value = this.redact(read());
|
||||
if (enumValues.size > 0)
|
||||
field.schema_enum_values = [...enumValues].map(([number, names]) => ({
|
||||
number: this.redact(number),
|
||||
names: [...names],
|
||||
}));
|
||||
} catch (cause) {
|
||||
if (cause instanceof CaptureFormatError) throw cause;
|
||||
if (this.budget.signal?.aborted && cause === this.budget.signal.reason)
|
||||
throw cause;
|
||||
field.schema_status = "wire-type-mismatch";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,522 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { create, toBinary } from "@bufbuild/protobuf";
|
||||
import { FileDescriptorSetSchema } from "@bufbuild/protobuf/wkt";
|
||||
import { decodeProtobufWire, ProtobufBudget } from "./ProtobufWire.js";
|
||||
import { decodeProtobufBody } from "./ProtobufBody.js";
|
||||
import { ProtobufSchema } from "./ProtobufSchema.js";
|
||||
import { protobufCaptureDescriptor } from "../../../../tests/fixtures/protobufCapture.js";
|
||||
|
||||
const same = (value: string) => value;
|
||||
const wire = (hex: string) => {
|
||||
const bytes = Buffer.from(hex, "hex");
|
||||
return decodeProtobufWire(
|
||||
bytes,
|
||||
0,
|
||||
bytes.length,
|
||||
0,
|
||||
new ProtobufBudget(),
|
||||
same,
|
||||
);
|
||||
};
|
||||
const source = {
|
||||
pointer: "/response/content/text",
|
||||
content_type: "application/grpc-web+proto",
|
||||
body_sha256: "a".repeat(64),
|
||||
};
|
||||
const frame = (bytes: Buffer, flag = 0) => {
|
||||
const header = Buffer.alloc(5);
|
||||
header[0] = flag;
|
||||
header.writeUInt32BE(bytes.length, 1);
|
||||
return Buffer.concat([header, bytes]);
|
||||
};
|
||||
|
||||
it("rejects overflowing uint64 values without masking and preserves the maximum value", async () => {
|
||||
expect(await wire("0880808080808080808002")).toMatchObject({
|
||||
fields: [],
|
||||
error: { offset: 0 },
|
||||
});
|
||||
expect((await wire("08ffffffffffffffffff01")).fields[0]?.value).toBe(
|
||||
"18446744073709551615",
|
||||
);
|
||||
expect(await wire("0880808080808080808080")).toHaveProperty("error");
|
||||
expect(await wire("0880")).toHaveProperty("error");
|
||||
});
|
||||
|
||||
it("preserves repeated field occurrences, source ranges, ambiguous bytes and exact fixed values", async () => {
|
||||
const result = await wire("0896010801120568656c6c6f1a02080125ffffffff");
|
||||
expect(result.error).toBeUndefined();
|
||||
expect(result.fields.map(({ number }) => number)).toEqual([1, 1, 2, 3, 4]);
|
||||
expect(result.fields[2]).toMatchObject({
|
||||
field_id: "offset:5",
|
||||
offset: 5,
|
||||
payload_offset: 7,
|
||||
payload_bytes: 5,
|
||||
text: "hello",
|
||||
interpretations: ["utf8-string", "packed-varint", "bytes"],
|
||||
});
|
||||
expect(result.fields[3]?.interpretations).toContain("nested-message");
|
||||
expect(result.fields[3]?.text).toBe("\b\u0001");
|
||||
expect(result.fields[4]?.value).toBe("4294967295");
|
||||
});
|
||||
|
||||
it("handles proto2 groups and reports mismatched or missing end groups", async () => {
|
||||
expect((await wire("0b10010c")).fields).toMatchObject([
|
||||
{ wire_type: "group", bytes: 4 },
|
||||
{ parent_field_id: "offset:0", number: 2, value: "1" },
|
||||
]);
|
||||
expect(await wire("0b100114")).toHaveProperty("error");
|
||||
expect(await wire("0b1001")).toHaveProperty("error");
|
||||
expect(await wire("0c")).toHaveProperty("error");
|
||||
});
|
||||
|
||||
it("keeps empty and control-containing UTF-8 among ambiguous length-delimited readings", async () => {
|
||||
const empty = await wire("0a00");
|
||||
expect(empty.fields[0]).toMatchObject({
|
||||
text: "",
|
||||
interpretations: [
|
||||
"utf8-string",
|
||||
"nested-message",
|
||||
"packed-varint",
|
||||
"packed-fixed32",
|
||||
"packed-fixed64",
|
||||
"bytes",
|
||||
],
|
||||
});
|
||||
expect((await wire("0a020801")).fields[0]).toMatchObject({
|
||||
text: "\b\u0001",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps enum numbers separate from every declared alias", async () => {
|
||||
const bytes = toBinary(
|
||||
FileDescriptorSetSchema,
|
||||
create(FileDescriptorSetSchema, {
|
||||
file: [
|
||||
{
|
||||
name: "aliases.proto",
|
||||
package: "aliases",
|
||||
syntax: "proto3",
|
||||
enumType: [
|
||||
{
|
||||
name: "Mode",
|
||||
options: { allowAlias: true },
|
||||
value: [
|
||||
{ name: "ZERO", number: 0 },
|
||||
{ name: "ALSO_ZERO", number: 0 },
|
||||
],
|
||||
},
|
||||
],
|
||||
messageType: [
|
||||
{
|
||||
name: "Message",
|
||||
field: [
|
||||
{
|
||||
name: "mode",
|
||||
number: 1,
|
||||
type: 14,
|
||||
typeName: ".aliases.Mode",
|
||||
label: 1,
|
||||
},
|
||||
{ name: "negative_zero", number: 2, type: 2, label: 1 },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const budget = new ProtobufBudget();
|
||||
const schema = await ProtobufSchema.read(
|
||||
bytes,
|
||||
"aliases.Message",
|
||||
budget,
|
||||
same,
|
||||
);
|
||||
const decoded = await decodeProtobufBody(
|
||||
Buffer.from([8, 0, 21, 0, 0, 0, 128]),
|
||||
"raw",
|
||||
source,
|
||||
same,
|
||||
budget,
|
||||
schema,
|
||||
);
|
||||
expect(decoded.fields[1]).toMatchObject({
|
||||
value: "2147483648",
|
||||
schema_value: "-0",
|
||||
});
|
||||
expect(decoded.fields[0]).toMatchObject({
|
||||
value: "0",
|
||||
schema_type: "aliases.Mode",
|
||||
schema_value: "0",
|
||||
schema_enum_values: [{ number: "0", names: ["ZERO", "ALSO_ZERO"] }],
|
||||
});
|
||||
});
|
||||
|
||||
it("bounds qualified descriptor-name expansion before creating the SDK registry", async () => {
|
||||
const bytes = toBinary(
|
||||
FileDescriptorSetSchema,
|
||||
create(FileDescriptorSetSchema, {
|
||||
file: [
|
||||
{
|
||||
name: "expansion.proto",
|
||||
syntax: "proto3",
|
||||
messageType: [
|
||||
{
|
||||
name: "A".repeat(10000),
|
||||
nestedType: Array.from({ length: 2000 }, (_, index) => ({
|
||||
name: `Child${index}`,
|
||||
})),
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
await expect(
|
||||
ProtobufSchema.read(bytes, "Missing", new ProtobufBudget(), same),
|
||||
).rejects.toMatchObject({ reason: "limit" });
|
||||
});
|
||||
|
||||
it("distinguishes native gRPC flags from web trailers and retains partial messages", async () => {
|
||||
const body = Buffer.concat([
|
||||
frame(Buffer.from("0801", "hex")),
|
||||
frame(Buffer.from("grpc-status: 0\r\ngrpc-message: ok\r\n"), 128),
|
||||
]);
|
||||
const web = await decodeProtobufBody(
|
||||
body,
|
||||
"grpc-web",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
);
|
||||
expect(web.status).toBe("complete");
|
||||
expect(web.frames[1]?.trailers).toEqual([
|
||||
{ name: "grpc-status", value: "0" },
|
||||
{ name: "grpc-message", value: "ok" },
|
||||
]);
|
||||
expect(
|
||||
(await decodeProtobufBody(body, "grpc", source, same, new ProtobufBudget()))
|
||||
.error,
|
||||
).toMatchObject({ offset: 7 });
|
||||
expect(
|
||||
(
|
||||
await decodeProtobufBody(
|
||||
frame(Buffer.from("0880", "hex")),
|
||||
"grpc",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
)
|
||||
).frames[0],
|
||||
).toMatchObject({ status: "partial", error: { offset: 5 } });
|
||||
expect(
|
||||
(
|
||||
await decodeProtobufBody(
|
||||
frame(Buffer.from("0801", "hex"), 2),
|
||||
"grpc-web",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
)
|
||||
).status,
|
||||
).toBe("partial");
|
||||
});
|
||||
|
||||
it("decodes independently padded base64 chunks with explicit coordinates and rejects noncanonical text", async () => {
|
||||
const first = frame(Buffer.from("0801", "hex"));
|
||||
const second = frame(Buffer.from("0802", "hex"));
|
||||
const body = Buffer.from(
|
||||
first.toString("base64") + second.toString("base64"),
|
||||
);
|
||||
const decoded = await decodeProtobufBody(
|
||||
body,
|
||||
"grpc-web-text",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
);
|
||||
expect(decoded).toMatchObject({
|
||||
offset_basis: "base64-decoded-body",
|
||||
decoded_bytes: 14,
|
||||
status: "complete",
|
||||
});
|
||||
expect(
|
||||
decoded.fields.map(({ field_id, value }) => [field_id, value]),
|
||||
).toEqual([
|
||||
["offset:5", "1"],
|
||||
["offset:12", "2"],
|
||||
]);
|
||||
expect(decoded.fields.map(({ index }) => index)).toEqual([0, 1]);
|
||||
expect(
|
||||
(
|
||||
await decodeProtobufBody(
|
||||
Buffer.from("AB=="),
|
||||
"grpc-web-text",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
)
|
||||
).status,
|
||||
).toBe("partial");
|
||||
});
|
||||
|
||||
it("keeps compressed payloads explicitly unknown and rejects frames beyond the body", async () => {
|
||||
const compressed = await decodeProtobufBody(
|
||||
frame(Buffer.from("0801", "hex"), 1),
|
||||
"grpc",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
);
|
||||
expect(compressed.frames[0]).toMatchObject({
|
||||
status: "unknown",
|
||||
compressed: true,
|
||||
examined_bytes: 0,
|
||||
});
|
||||
expect(compressed.fields).toEqual([]);
|
||||
expect(
|
||||
(
|
||||
await decodeProtobufBody(
|
||||
Buffer.from("00000000090801", "hex"),
|
||||
"grpc",
|
||||
source,
|
||||
same,
|
||||
new ProtobufBudget(),
|
||||
)
|
||||
).error,
|
||||
).toBeDefined();
|
||||
});
|
||||
|
||||
it("uses supplied descriptors without replacing wire values and preserves schema disagreements", async () => {
|
||||
const descriptor = toBinary(
|
||||
FileDescriptorSetSchema,
|
||||
create(FileDescriptorSetSchema, {
|
||||
file: [
|
||||
{
|
||||
name: "sample.proto",
|
||||
package: "sample",
|
||||
syntax: "proto3",
|
||||
messageType: [
|
||||
{
|
||||
name: "Child",
|
||||
field: [{ name: "count", number: 1, type: 5, label: 1 }],
|
||||
},
|
||||
{
|
||||
name: "Message",
|
||||
field: [
|
||||
{ name: "signed", number: 1, type: 17, label: 1 },
|
||||
{ name: "text", number: 2, type: 9, label: 1 },
|
||||
{
|
||||
name: "child",
|
||||
number: 3,
|
||||
type: 11,
|
||||
label: 1,
|
||||
typeName: ".sample.Child",
|
||||
},
|
||||
{ name: "values", number: 4, type: 13, label: 3 },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const budget = new ProtobufBudget();
|
||||
const schema = await ProtobufSchema.read(
|
||||
descriptor,
|
||||
"sample.Message",
|
||||
budget,
|
||||
same,
|
||||
);
|
||||
const body = Buffer.from("0801120568656c6c6f1a020801220201023801", "hex");
|
||||
const decoded = await decodeProtobufBody(
|
||||
body,
|
||||
"raw",
|
||||
source,
|
||||
same,
|
||||
budget,
|
||||
schema,
|
||||
);
|
||||
expect(decoded.fields[0]).toMatchObject({
|
||||
value: "1",
|
||||
schema_name: "signed",
|
||||
schema_type: "sint32",
|
||||
schema_value: "-1",
|
||||
});
|
||||
expect(
|
||||
decoded.fields.find(({ schema_name }) => schema_name === "text")
|
||||
?.schema_value,
|
||||
).toBe("hello");
|
||||
expect(
|
||||
decoded.fields.find(({ schema_name }) => schema_name === "count"),
|
||||
).toMatchObject({ parent_field_id: "offset:9", value: "1" });
|
||||
expect(
|
||||
decoded.fields.find(({ schema_name }) => schema_name === "values")
|
||||
?.schema_value,
|
||||
).toBe('["1","2"]');
|
||||
expect(decoded.fields.at(-1)?.schema_status).toBe("unknown-field");
|
||||
schema.reset();
|
||||
expect(
|
||||
(
|
||||
await decodeProtobufBody(
|
||||
Buffer.from("1001", "hex"),
|
||||
"raw",
|
||||
source,
|
||||
same,
|
||||
budget,
|
||||
schema,
|
||||
)
|
||||
).fields[0]?.schema_status,
|
||||
).toBe("wire-type-mismatch");
|
||||
});
|
||||
|
||||
it("stops before exhausting fact capacity and yields for caller cancellation", async () => {
|
||||
const budget = new ProtobufBudget();
|
||||
budget.charge(96 * 1024 * 1024 - 1023);
|
||||
await expect(
|
||||
decodeProtobufWire(Buffer.from([8, 1]), 0, 2, 0, budget, same),
|
||||
).rejects.toMatchObject({ reason: "limit" });
|
||||
const abort = new AbortController();
|
||||
const reason = new Error("caller stopped selected decoding");
|
||||
const data = Buffer.alloc(8192);
|
||||
for (let at = 0; at < data.length; at += 2) {
|
||||
data[at] = 8;
|
||||
data[at + 1] = 1;
|
||||
}
|
||||
const task = decodeProtobufWire(
|
||||
data,
|
||||
0,
|
||||
data.length,
|
||||
0,
|
||||
new ProtobufBudget(abort.signal),
|
||||
same,
|
||||
);
|
||||
setImmediate(() => abort.abort(reason));
|
||||
await expect(task).rejects.toBe(reason);
|
||||
});
|
||||
|
||||
it("preserves a leading Unicode BOM in both the wire string and schema interpretation", async () => {
|
||||
const budget = new ProtobufBudget();
|
||||
const schema = await ProtobufSchema.read(
|
||||
protobufCaptureDescriptor(),
|
||||
"capture.Message",
|
||||
budget,
|
||||
same,
|
||||
);
|
||||
const bytes = Buffer.from("0a08efbbbf68656c6c6f", "hex");
|
||||
const result = await decodeProtobufWire(
|
||||
bytes,
|
||||
0,
|
||||
bytes.length,
|
||||
0,
|
||||
budget,
|
||||
same,
|
||||
schema,
|
||||
);
|
||||
expect(result.fields[0]).toMatchObject({
|
||||
text: "\uFEFFhello",
|
||||
schema_value: "\uFEFFhello",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps bad nested schema interpretations local and preserves subsequent outer fields", async () => {
|
||||
const descriptor = toBinary(
|
||||
FileDescriptorSetSchema,
|
||||
create(FileDescriptorSetSchema, {
|
||||
file: [
|
||||
{
|
||||
name: "nested.proto",
|
||||
package: "nested",
|
||||
syntax: "proto3",
|
||||
messageType: [
|
||||
{
|
||||
name: "Child",
|
||||
field: [{ name: "value", number: 1, type: 13, label: 1 }],
|
||||
},
|
||||
{
|
||||
name: "Message",
|
||||
field: [
|
||||
{
|
||||
name: "child",
|
||||
number: 1,
|
||||
type: 11,
|
||||
label: 1,
|
||||
typeName: ".nested.Child",
|
||||
},
|
||||
{ name: "value", number: 2, type: 13, label: 1 },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const budget = new ProtobufBudget();
|
||||
const schema = await ProtobufSchema.read(
|
||||
descriptor,
|
||||
"nested.Message",
|
||||
budget,
|
||||
same,
|
||||
);
|
||||
const bytes = Buffer.from("0a01ff1001", "hex");
|
||||
const result = await decodeProtobufWire(
|
||||
bytes,
|
||||
0,
|
||||
bytes.length,
|
||||
0,
|
||||
budget,
|
||||
same,
|
||||
schema,
|
||||
);
|
||||
expect(result.error).toBeUndefined();
|
||||
expect(result.examined_bytes).toBe(bytes.length);
|
||||
expect(result.fields).toMatchObject([
|
||||
{
|
||||
number: 1,
|
||||
schema_status: "invalid-message",
|
||||
schema_error: { offset: 2 },
|
||||
},
|
||||
{ number: 2, value: "1", schema_value: "1", parent_field_id: null },
|
||||
]);
|
||||
});
|
||||
|
||||
it("yields and preserves the cancellation reason inside one large packed interpretation", async () => {
|
||||
const abort = new AbortController();
|
||||
const budget = new ProtobufBudget(abort.signal);
|
||||
const schema = await ProtobufSchema.read(
|
||||
protobufCaptureDescriptor(),
|
||||
"capture.Message",
|
||||
budget,
|
||||
same,
|
||||
);
|
||||
// Field 2, length 100000, followed by 100000 independently expected uint32 ones.
|
||||
const bytes = Buffer.concat([
|
||||
Buffer.from([18, 160, 141, 6]),
|
||||
Buffer.alloc(100000, 1),
|
||||
]);
|
||||
const parsed = await decodeProtobufWire(
|
||||
bytes,
|
||||
0,
|
||||
bytes.length,
|
||||
0,
|
||||
new ProtobufBudget(),
|
||||
same,
|
||||
);
|
||||
const field = parsed.fields[0];
|
||||
if (field === undefined) throw new Error("Expected the packed wire field");
|
||||
const task = schema.annotate(field, bytes, undefined);
|
||||
const reason = new Error("stop during packed annotation");
|
||||
setImmediate(() => abort.abort(reason));
|
||||
await expect(task).rejects.toBe(reason);
|
||||
});
|
||||
|
||||
it("rejects malformed descriptor wire even when the SDK would skip the unknown field", async () => {
|
||||
// File t.proto defines empty M, then unknown field 99 carries an overflowing uint64.
|
||||
const bytes = Buffer.from(
|
||||
"0a1a0a07742e70726f746f22030a014d980680808080808080808002",
|
||||
"hex",
|
||||
);
|
||||
await expect(
|
||||
ProtobufSchema.read(bytes, "M", new ProtobufBudget(), same),
|
||||
).rejects.toThrow("Malformed FileDescriptorSet wire format");
|
||||
});
|
||||
@@ -0,0 +1,269 @@
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
import type { ProtobufField } from "../../../domain/protobufWire.js";
|
||||
import { CaptureFormatError } from "../CaptureFormatError.js";
|
||||
import { WEB_NETWORK_CAPTURE_LIMITS } from "../../../domain/webNetworkCapture.js";
|
||||
|
||||
/** One charge covers all selected records, including temporary interpretation strings. */
|
||||
export class ProtobufBudget {
|
||||
#remaining = WEB_NETWORK_CAPTURE_LIMITS.outputBytes;
|
||||
#work = 0;
|
||||
constructor(readonly signal?: AbortSignal) {}
|
||||
charge(bytes: number): void {
|
||||
if (!Number.isSafeInteger(bytes) || bytes < 0 || bytes > this.#remaining)
|
||||
throw new CaptureFormatError(
|
||||
"limit",
|
||||
"Protobuf decoding exceeds the shared 96 MiB encoded-result and working-set budget.",
|
||||
);
|
||||
this.#remaining -= bytes;
|
||||
}
|
||||
async checkpoint(): Promise<void> {
|
||||
this.signal?.throwIfAborted();
|
||||
if (++this.#work % 256 === 0) {
|
||||
await setImmediate();
|
||||
this.signal?.throwIfAborted();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Reject overflowing uint64s before conversion; never wrap malformed bytes. */
|
||||
export const readProtobufVarint = (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
end = bytes.length,
|
||||
): { value: bigint; end: number } | undefined => {
|
||||
let value = 0n;
|
||||
for (let index = 0; index < 10 && start + index < end; index++) {
|
||||
const byte = bytes[start + index]!;
|
||||
if (index === 9 && byte > 1) return undefined;
|
||||
value |= BigInt(byte & 127) << BigInt(index * 7);
|
||||
if (byte < 128) return { value, end: start + index + 1 };
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
const utf8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
const textOf = (bytes: Uint8Array): string | undefined => {
|
||||
try {
|
||||
return utf8.decode(bytes);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
/** An allocation-free syntactic candidate check, not a claim about the payload's type. */
|
||||
const messageCandidate = async (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
end: number,
|
||||
budget: ProtobufBudget,
|
||||
): Promise<boolean> => {
|
||||
if (start === end) return true;
|
||||
while (start < end) {
|
||||
await budget.checkpoint();
|
||||
const tag = readProtobufVarint(bytes, start, end);
|
||||
if (
|
||||
tag === undefined ||
|
||||
tag.value >> 3n === 0n ||
|
||||
tag.value >> 3n > 0x1fffffffn
|
||||
)
|
||||
return false;
|
||||
const type = Number(tag.value & 7n);
|
||||
start = tag.end;
|
||||
if (type === 0) {
|
||||
const value = readProtobufVarint(bytes, start, end);
|
||||
if (value === undefined) return false;
|
||||
start = value.end;
|
||||
} else if (type === 1 || type === 5) start += type === 1 ? 8 : 4;
|
||||
else if (type === 2) {
|
||||
const size = readProtobufVarint(bytes, start, end);
|
||||
if (size === undefined || size.value > BigInt(end - size.end))
|
||||
return false;
|
||||
start = size.end + Number(size.value);
|
||||
} else return false;
|
||||
}
|
||||
return start === end;
|
||||
};
|
||||
|
||||
const packedVarintCandidate = async (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
end: number,
|
||||
budget: ProtobufBudget,
|
||||
): Promise<boolean> => {
|
||||
while (start < end) {
|
||||
await budget.checkpoint();
|
||||
const value = readProtobufVarint(bytes, start, end);
|
||||
if (value === undefined) return false;
|
||||
start = value.end;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
export interface WireSchema {
|
||||
annotate(
|
||||
field: ProtobufField,
|
||||
bytes: Uint8Array,
|
||||
parent: ProtobufField | undefined,
|
||||
): void | Promise<void>;
|
||||
nested(field: ProtobufField): boolean;
|
||||
}
|
||||
|
||||
/** Decode one message into flat occurrence facts with offsets in the decoded body. */
|
||||
export const decodeProtobufWire = async (
|
||||
bytes: Uint8Array,
|
||||
start: number,
|
||||
end: number,
|
||||
frameIndex: number,
|
||||
budget: ProtobufBudget,
|
||||
redact: (text: string) => string,
|
||||
schema?: WireSchema,
|
||||
baseIndex = 0,
|
||||
): Promise<{
|
||||
fields: ProtobufField[];
|
||||
examined_bytes: number;
|
||||
error?: { offset: number; reason: string };
|
||||
}> => {
|
||||
const fields: ProtobufField[] = [];
|
||||
const walk = async (
|
||||
from: number,
|
||||
until: number,
|
||||
parent: ProtobufField | undefined,
|
||||
depth: number,
|
||||
group?: number,
|
||||
): Promise<{
|
||||
end: number;
|
||||
payloadEnd?: number;
|
||||
error?: { offset: number; reason: string };
|
||||
}> => {
|
||||
let at = from;
|
||||
const fail = (reason: string) => ({
|
||||
end: at,
|
||||
error: { offset: at, reason },
|
||||
});
|
||||
while (at < until) {
|
||||
await budget.checkpoint();
|
||||
const tag = readProtobufVarint(bytes, at, until);
|
||||
if (tag === undefined) return fail("Truncated or overflowing field tag");
|
||||
const number = Number(tag.value >> 3n);
|
||||
const type = Number(tag.value & 7n);
|
||||
if (number === 0 || tag.value >> 3n > 0x1fffffffn)
|
||||
return fail("Invalid field number");
|
||||
if (type === 4)
|
||||
return group === number
|
||||
? { end: tag.end, payloadEnd: at }
|
||||
: fail("Unexpected or mismatched end-group tag");
|
||||
let payloadOffset = tag.end;
|
||||
let next: number;
|
||||
let value: string | undefined;
|
||||
if (type === 0) {
|
||||
const parsed = readProtobufVarint(bytes, tag.end, until);
|
||||
if (parsed === undefined)
|
||||
return fail("Truncated or overflowing uint64 varint");
|
||||
next = parsed.end;
|
||||
value = parsed.value.toString();
|
||||
} else if (type === 1 || type === 5) {
|
||||
next = tag.end + (type === 1 ? 8 : 4);
|
||||
if (next > until) return fail("Truncated fixed-width field");
|
||||
let integer = 0n;
|
||||
for (let p = next - 1; p >= tag.end; p--)
|
||||
integer = (integer << 8n) | BigInt(bytes[p]!);
|
||||
value = integer.toString();
|
||||
} else if (type === 2) {
|
||||
const length = readProtobufVarint(bytes, tag.end, until);
|
||||
if (length === undefined)
|
||||
return fail("Truncated or overflowing length prefix");
|
||||
if (length.value > BigInt(until - length.end))
|
||||
return fail("Length-delimited field exceeds its message");
|
||||
payloadOffset = length.end;
|
||||
next = payloadOffset + Number(length.value);
|
||||
} else if (type === 3) {
|
||||
if (depth >= WEB_NETWORK_CAPTURE_LIMITS.depth)
|
||||
return fail("Group nesting exceeds the 64-level decoder budget");
|
||||
next = tag.end;
|
||||
} else return fail(`Invalid wire type ${type}`);
|
||||
budget.charge(1024);
|
||||
const field: ProtobufField = {
|
||||
index: baseIndex + fields.length,
|
||||
frame_index: frameIndex,
|
||||
parent_index: parent?.index ?? null,
|
||||
field_id: `offset:${at}`,
|
||||
parent_field_id:
|
||||
parent === undefined ? null : `offset:${parent.offset}`,
|
||||
number,
|
||||
wire_type:
|
||||
type === 0
|
||||
? "varint"
|
||||
: type === 1
|
||||
? "i64"
|
||||
: type === 2
|
||||
? "len"
|
||||
: type === 3
|
||||
? "group"
|
||||
: "i32",
|
||||
offset: at,
|
||||
bytes: next - at,
|
||||
payload_offset: payloadOffset,
|
||||
payload_bytes: next - payloadOffset,
|
||||
...(value === undefined ? {} : { value: redact(value) }),
|
||||
};
|
||||
fields.push(field);
|
||||
if (type === 2) {
|
||||
// Account for UTF-16 allocation and JSON escaping before decoding any string.
|
||||
budget.charge(field.payload_bytes * 8);
|
||||
const text = textOf(bytes.subarray(payloadOffset, next));
|
||||
const candidate = await messageCandidate(
|
||||
bytes,
|
||||
payloadOffset,
|
||||
next,
|
||||
budget,
|
||||
);
|
||||
const packed = await packedVarintCandidate(
|
||||
bytes,
|
||||
payloadOffset,
|
||||
next,
|
||||
budget,
|
||||
);
|
||||
field.interpretations = [
|
||||
...(text === undefined ? [] : ["utf8-string" as const]),
|
||||
...(candidate ? ["nested-message" as const] : []),
|
||||
...(packed ? ["packed-varint" as const] : []),
|
||||
...(field.payload_bytes % 4 === 0 ? ["packed-fixed32" as const] : []),
|
||||
...(field.payload_bytes % 8 === 0 ? ["packed-fixed64" as const] : []),
|
||||
"bytes",
|
||||
];
|
||||
if (text !== undefined) field.text = redact(text);
|
||||
}
|
||||
if (schema !== undefined) await schema.annotate(field, bytes, parent);
|
||||
if (type === 3) {
|
||||
const nested = await walk(tag.end, until, field, depth + 1, number);
|
||||
field.bytes = nested.end - at;
|
||||
field.payload_bytes = (nested.payloadEnd ?? nested.end) - payloadOffset;
|
||||
if (nested.error !== undefined) return nested;
|
||||
next = nested.end;
|
||||
} else if (type === 2 && schema?.nested(field)) {
|
||||
const error =
|
||||
depth >= WEB_NETWORK_CAPTURE_LIMITS.depth
|
||||
? {
|
||||
offset: payloadOffset,
|
||||
reason:
|
||||
"Schema message nesting exceeds the 64-level decoder budget",
|
||||
}
|
||||
: (await walk(payloadOffset, next, field, depth + 1)).error;
|
||||
// The outer length is an observed boundary. A bad interpretation of its
|
||||
// payload must never hide valid fields following that boundary.
|
||||
if (error !== undefined) {
|
||||
field.schema_status = "invalid-message";
|
||||
field.schema_error = error;
|
||||
}
|
||||
}
|
||||
at = next;
|
||||
}
|
||||
return group === undefined ? { end: at } : fail("Missing end-group tag");
|
||||
};
|
||||
const result = await walk(start, end, undefined, 0);
|
||||
return {
|
||||
fields,
|
||||
examined_bytes: result.end - start,
|
||||
...(result.error === undefined ? {} : { error: result.error }),
|
||||
};
|
||||
};
|
||||
@@ -7,7 +7,11 @@ import type { Logger } from "pino";
|
||||
import type { CliInstance } from "./types.js";
|
||||
import { cliIntegerArgument } from "./options.js";
|
||||
import { withCommandCancellation } from "./commandCancellation.js";
|
||||
import { finishAnalysisCommand } from "./finishAnalysisCommand.js";
|
||||
import {
|
||||
analysisEvidenceDeliveryOptions,
|
||||
deliverCliAnalysisEvidence,
|
||||
} from "./analysisEvidenceDelivery.js";
|
||||
import { absoluteCallerPath } from "../filesystem/callerPath.js";
|
||||
|
||||
/** Expose offline historical capture inspection with the same selection contract as MCP. */
|
||||
export const registerWebNetworkCaptureCommands = (
|
||||
@@ -26,6 +30,23 @@ export const registerWebNetworkCaptureCommands = (
|
||||
.describe("Producer capture format"),
|
||||
}),
|
||||
options: z.object({
|
||||
...analysisEvidenceDeliveryOptions,
|
||||
protobufFraming: z
|
||||
.enum(["auto", "raw", "grpc", "grpc-web", "grpc-web-text"])
|
||||
.optional()
|
||||
.describe("Override framing for selected retained binary bodies"),
|
||||
protobufBodyPointer: z
|
||||
.array(z.string())
|
||||
.optional()
|
||||
.describe("Exact body pointers; repeat to select several"),
|
||||
protobufSchema: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe("Absolute local compiled FileDescriptorSet path"),
|
||||
protobufMessage: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe("Fully qualified message type in the descriptor set"),
|
||||
record: z
|
||||
.array(cliIntegerArgument(0))
|
||||
.optional()
|
||||
@@ -43,13 +64,42 @@ export const registerWebNetworkCaptureCommands = (
|
||||
logger,
|
||||
CLI_COMMANDS.inspectWebNetworkCapture,
|
||||
async () => {
|
||||
return finishAnalysisCommand(
|
||||
() =>
|
||||
return deliverCliAnalysisEvidence({
|
||||
run: () =>
|
||||
service.inspect(
|
||||
{
|
||||
capture_path: args.capturePath,
|
||||
capture_path: absoluteCallerPath(args.capturePath),
|
||||
format: args.format,
|
||||
sensitive_values: options.sensitiveValue,
|
||||
...([
|
||||
options.protobufFraming,
|
||||
options.protobufBodyPointer,
|
||||
options.protobufSchema,
|
||||
options.protobufMessage,
|
||||
].every((value) => value === undefined)
|
||||
? {}
|
||||
: {
|
||||
protobuf: {
|
||||
framing: options.protobufFraming ?? "auto",
|
||||
...(options.protobufBodyPointer === undefined
|
||||
? {}
|
||||
: { body_pointers: options.protobufBodyPointer }),
|
||||
...(options.protobufSchema === undefined &&
|
||||
options.protobufMessage === undefined
|
||||
? {}
|
||||
: {
|
||||
schema: {
|
||||
descriptor_set_path:
|
||||
options.protobufSchema === undefined
|
||||
? undefined
|
||||
: absoluteCallerPath(
|
||||
options.protobufSchema,
|
||||
),
|
||||
message_type: options.protobufMessage,
|
||||
},
|
||||
}),
|
||||
},
|
||||
}),
|
||||
...(options.record === undefined
|
||||
? {}
|
||||
: { record_ordinals: options.record }),
|
||||
@@ -57,11 +107,12 @@ export const registerWebNetworkCaptureCommands = (
|
||||
{ signal },
|
||||
),
|
||||
service,
|
||||
"web-network-capture",
|
||||
"inspect_web_network_capture",
|
||||
providerId: "web-network-capture",
|
||||
operation: "inspect_web_network_capture",
|
||||
signal,
|
||||
"Web network capture resources",
|
||||
);
|
||||
detail: options.detail,
|
||||
evidenceOutput: options.evidence_output,
|
||||
});
|
||||
},
|
||||
),
|
||||
),
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -16,7 +16,7 @@ export const WEB_NETWORK_CAPTURE_TOOL_CONTRACTS = [
|
||||
name: "inspect_web_network_capture",
|
||||
...toolContractMetadata("inspect_web_network_capture"),
|
||||
description:
|
||||
"Request detail summary for capture identity and counts; then page retained records with inspect_analysis_view kind capture (preview or complete), avoiding repeated decoding. Default detail complete preserves existing inline delivery. Inspect retained local HAR 1.2, native mitmproxy, Fiddler SAZ or aioquic 1.5.0 JSON qlog 0.3 events without contacting recorded URLs. Returns original ordinals, pointers/byte ranges/archive member names, preserved producer fields, precise numeric representations and exact exposed binary bytes inline. Excludes known transport credentials and explicitly marked values. Decodes every record before optional ordinal selection; malformed or oversized captures return no partial success. Duplicate JSON members are invalid; prototype-named members such as __proto__ are preserved as ordinary own JSON members without prototype mutation. Qlog reports trace-scoped STREAM interval unions and endpoint loss declarations; it does not recover bodies, prove network loss or correlate PCAP. Other qlog layouts are unsupported. Uses pinned upstream parsers, owned temporary files and bounded offline processes. SAZ preserves HTTP fields, XML metadata and encoded body/message bytes without replay or HAR conversion; encrypted archives are unsupported. Native mitmproxy requires caller-supplied mitmdump 12.2.3 on Linux. Browser transaction IDs, runtime attribution and original bytes of HAR Unicode text remain unknown.",
|
||||
"Explicit record_ordinals also decode retained protobuf/gRPC bodies whose declared content type is recognized. Optional protobuf selects exact body_pointers, framing, and a caller-supplied local FileDescriptorSet with message_type. Native gRPC and gRPC-web framing remain distinct. Wire fields are flat byte-range facts; ambiguous readings stay candidates, compressed payloads stay unknown. Page retained decoding with inspect_analysis_view kind protobuf, collection bodies/frames/fields, exact batch selectors and fields. Large text and schema values require explicit fields selection in those pages. Request detail summary for capture identity and counts; then page retained records with inspect_analysis_view kind capture (preview or complete), avoiding repeated decoding. Default detail complete preserves existing inline delivery. Inspect retained local HAR 1.2, native mitmproxy, Fiddler SAZ or aioquic 1.5.0 JSON qlog 0.3 events without contacting recorded URLs. Returns original ordinals, pointers/byte ranges/archive member names, preserved producer fields, precise numeric representations and exact exposed binary bytes inline. Excludes known transport credentials and explicitly marked values. Decodes every record before optional ordinal selection; malformed or oversized captures return no partial success. Duplicate JSON members are invalid; prototype-named members such as __proto__ are preserved as ordinary own JSON members without prototype mutation. Qlog reports trace-scoped STREAM interval unions and endpoint loss declarations; it does not recover bodies, prove network loss or correlate PCAP. Other qlog layouts are unsupported. Uses pinned upstream parsers, owned temporary files and bounded offline processes. SAZ preserves HTTP fields, XML metadata and encoded body/message bytes without replay or HAR conversion; encrypted archives are unsupported. Native mitmproxy requires caller-supplied mitmdump 12.2.3 on Linux. Browser transaction IDs, runtime attribution and original bytes of HAR Unicode text remain unknown.",
|
||||
kind: "browser-provider",
|
||||
inputSchema: inspectWebNetworkCaptureInputSchema.extend({
|
||||
detail: analysisDetailSchema,
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import {
|
||||
protobufViewRequestSchema,
|
||||
protobufViewItemsSchema,
|
||||
protobufViewContextSchema,
|
||||
} from "./protobufCaptureView.js";
|
||||
import { isAbsolute } from "node:path";
|
||||
|
||||
import { localPathStringSchema } from "../localPath.js";
|
||||
@@ -247,6 +252,7 @@ export const analysisViewRequestSchema = z.discriminatedUnion("kind", [
|
||||
environmentManifestViewRequestSchema,
|
||||
environmentStateViewRequestSchema,
|
||||
captureViewRequestSchema,
|
||||
protobufViewRequestSchema,
|
||||
qlogViewRequestSchema,
|
||||
pcapViewRequestSchema,
|
||||
dartAotViewRequestSchema,
|
||||
@@ -549,6 +555,14 @@ export const analysisViewResultSchema = z.discriminatedUnion("kind", [
|
||||
il2cppMetadataSummarySchema,
|
||||
]),
|
||||
}),
|
||||
z.strictObject({
|
||||
kind: z.literal("protobuf"),
|
||||
view: protobufViewRequestSchema,
|
||||
items: protobufViewItemsSchema,
|
||||
...viewResultBase,
|
||||
...protobufViewContextSchema.shape,
|
||||
parent_operation: z.literal("inspect_web_network_capture"),
|
||||
}),
|
||||
z.strictObject({
|
||||
kind: z.literal("capture"),
|
||||
view: captureViewRequestSchema,
|
||||
@@ -710,79 +724,10 @@ export interface AnalysisViewParent {
|
||||
}
|
||||
|
||||
/** View payload before the content digest is attached. */
|
||||
export type UnsignedAnalysisView =
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "desktop-application" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "api-surface" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "environment-manifest" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "environment-state" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "summary" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "facet" }>, "view_digest">
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "item" }>, "view_digest">
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "page" }>, "view_digest">
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "javascript" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "javascript-facts" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "native" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "inventory" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "wasm" }>, "view_digest">
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "macho" }>, "view_digest">
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "interface-builder" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "evm" }>, "view_digest">
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "qlog" }>, "view_digest">
|
||||
| Omit<Extract<AnalysisViewResult, { readonly kind: "pcap" }>, "view_digest">
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "capture" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "managed" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "managed-members" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "android" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "dart-aot" }>,
|
||||
"view_digest"
|
||||
>
|
||||
| Omit<
|
||||
Extract<AnalysisViewResult, { readonly kind: "il2cpp-metadata" }>,
|
||||
"view_digest"
|
||||
>;
|
||||
type WithoutViewDigest<T> = T extends { readonly view_digest: string }
|
||||
? Omit<T, "view_digest">
|
||||
: never;
|
||||
export type UnsignedAnalysisView = WithoutViewDigest<AnalysisViewResult>;
|
||||
|
||||
/** Attach the content digest that must change when projected bytes change. */
|
||||
export const sealAnalysisView = (
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { createEvidence } from "../evidence.js";
|
||||
import { projectAnalysisView } from "./analysisView.js";
|
||||
import { webNetworkCaptureSchema } from "../webNetworkCapture.js";
|
||||
|
||||
const capture = async () => {
|
||||
const report = webNetworkCaptureSchema.parse({
|
||||
artifact: { path: "/recordings/a.har", bytes: 100, sha256: "b".repeat(64) },
|
||||
decoder: {
|
||||
id: "projection-fixture",
|
||||
name: "Original projection facts",
|
||||
version: "1",
|
||||
},
|
||||
container: {
|
||||
reported: null,
|
||||
numeric_literals: [],
|
||||
redactions: [],
|
||||
records_pointer: "/log/entries",
|
||||
},
|
||||
format: "har",
|
||||
total_records: 1,
|
||||
runtime_attribution: "unknown",
|
||||
limitations: [],
|
||||
records: [
|
||||
{
|
||||
ordinal: 0,
|
||||
location: { kind: "json-pointer", pointer: "/log/entries/0" },
|
||||
reported: null,
|
||||
binary_fields: [],
|
||||
numeric_literals: [],
|
||||
redactions: [],
|
||||
limitations: [],
|
||||
protobuf: [
|
||||
{
|
||||
decoder: "rea-protobuf-wire-v1",
|
||||
pointer: "/response/content/text",
|
||||
content_type: "application/grpc-web+proto",
|
||||
body_sha256: "c".repeat(64),
|
||||
framing: "grpc-web",
|
||||
offset_basis: "retained-body",
|
||||
decoded_bytes: 37,
|
||||
status: "complete",
|
||||
examined_bytes: 37,
|
||||
limitations: [],
|
||||
frames: [
|
||||
{
|
||||
index: 0,
|
||||
kind: "data",
|
||||
offset: 0,
|
||||
payload_offset: 5,
|
||||
bytes: 11,
|
||||
compressed: false,
|
||||
status: "complete",
|
||||
examined_bytes: 11,
|
||||
},
|
||||
{
|
||||
index: 1,
|
||||
kind: "trailer",
|
||||
offset: 16,
|
||||
payload_offset: 21,
|
||||
bytes: 16,
|
||||
compressed: false,
|
||||
status: "complete",
|
||||
examined_bytes: 16,
|
||||
},
|
||||
],
|
||||
fields: [
|
||||
{
|
||||
index: 0,
|
||||
frame_index: 0,
|
||||
parent_index: null,
|
||||
field_id: "offset:5",
|
||||
parent_field_id: null,
|
||||
number: 1,
|
||||
wire_type: "len",
|
||||
offset: 5,
|
||||
bytes: 7,
|
||||
payload_offset: 7,
|
||||
payload_bytes: 5,
|
||||
text: "hello",
|
||||
interpretations: ["utf8-string", "packed-varint", "bytes"],
|
||||
},
|
||||
{
|
||||
index: 1,
|
||||
frame_index: 0,
|
||||
parent_index: null,
|
||||
field_id: "offset:12",
|
||||
parent_field_id: null,
|
||||
number: 2,
|
||||
wire_type: "len",
|
||||
offset: 12,
|
||||
bytes: 4,
|
||||
payload_offset: 14,
|
||||
payload_bytes: 2,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
const evidence = createEvidence(
|
||||
{ ...report.artifact, sha256: "b".repeat(64), format: "file" },
|
||||
report.decoder,
|
||||
{
|
||||
operation: "inspect_web_network_capture",
|
||||
parameters: { record_ordinals: [0] },
|
||||
result: report,
|
||||
confidence: "observed",
|
||||
authority: "historical-reference",
|
||||
},
|
||||
);
|
||||
return {
|
||||
report,
|
||||
parent: {
|
||||
evidenceId: evidence.evidence_id,
|
||||
operation: evidence.operation,
|
||||
normalizedResult: report,
|
||||
subject: evidence.subject,
|
||||
limitations: evidence.limitations,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
it("keeps coordinate identity and coverage with empty fields, batch filters and offset paging", async () => {
|
||||
const { parent } = await capture();
|
||||
const viewed = projectAnalysisView(parent, {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
field_ids: ["offset:5", "offset:12"],
|
||||
fields: [],
|
||||
offset: 1,
|
||||
limit: 1,
|
||||
});
|
||||
if (!viewed.ok) throw viewed.error;
|
||||
expect(viewed.value).toMatchObject({
|
||||
kind: "protobuf",
|
||||
coverage: { total: 2, examined: 1, exhausted: true },
|
||||
items: [
|
||||
{
|
||||
field_id: "offset:12",
|
||||
record_ordinal: 0,
|
||||
body_pointer: "/response/content/text",
|
||||
body_status: "complete",
|
||||
offset_basis: "retained-body",
|
||||
},
|
||||
],
|
||||
});
|
||||
const empty = projectAnalysisView(parent, {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
field_numbers: [],
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
});
|
||||
if (!empty.ok) throw empty.error;
|
||||
expect(empty.value).toMatchObject({
|
||||
items: [],
|
||||
coverage: { total: 0, exhausted: true },
|
||||
});
|
||||
});
|
||||
|
||||
it("omits large values by default, returns requested text inline and preserves parser uncertainty", async () => {
|
||||
const { parent, report } = await capture();
|
||||
const brief = projectAnalysisView(parent, {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
});
|
||||
if (!brief.ok) throw brief.error;
|
||||
expect(brief.value).toMatchObject({
|
||||
items: [{ interpretations: ["utf8-string", "packed-varint", "bytes"] }],
|
||||
});
|
||||
expect(JSON.stringify(brief.value)).not.toContain('"hello"');
|
||||
const preview = projectAnalysisView(parent, {
|
||||
kind: "capture",
|
||||
detail: "preview",
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
});
|
||||
if (!preview.ok) throw preview.error;
|
||||
expect(preview.value).toMatchObject({
|
||||
items: [
|
||||
{
|
||||
protobuf_summary: {
|
||||
bodies: 1,
|
||||
frames: 2,
|
||||
fields: 2,
|
||||
incomplete_bodies: 0,
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(JSON.stringify(preview.value)).not.toContain('"field_id"');
|
||||
report.records[0]!.protobuf![0]!.status = "partial";
|
||||
const full = projectAnalysisView(parent, {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
fields: ["text"],
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
});
|
||||
if (!full.ok) throw full.error;
|
||||
expect(full.value).toMatchObject({
|
||||
source_status: "partial",
|
||||
items: [{ text: "hello", body_status: "partial" }],
|
||||
});
|
||||
const changedSubject = projectAnalysisView(
|
||||
{ ...parent, subject: { ...parent.subject!, local_path: "/other.har" } },
|
||||
{ kind: "protobuf", collection: "bodies", offset: 0, limit: 1 },
|
||||
);
|
||||
expect(changedSubject.ok).toBe(false);
|
||||
report.records[0]!.protobuf![0]!.fields[0]!.offset = 1000;
|
||||
expect(
|
||||
projectAnalysisView(parent, {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
}).ok,
|
||||
).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,315 @@
|
||||
import { isImmutableJsonSnapshot } from "../immutableJson.js";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
protobufDecodingSchema,
|
||||
protobufDecodingObjectSchema,
|
||||
protobufFieldSchema,
|
||||
protobufFrameSchema,
|
||||
type ProtobufDecoding,
|
||||
type ProtobufField,
|
||||
} from "../protobufWire.js";
|
||||
import type { WebNetworkCapture } from "../webNetworkCapture.js";
|
||||
import { pageViewCoverage } from "./analysisViewCoverage.js";
|
||||
|
||||
const body = {
|
||||
record_ordinal: z.number().int().nonnegative(),
|
||||
body_pointer: z.string(),
|
||||
body_sha256: z.string(),
|
||||
body_status: protobufDecodingSchema.shape.status,
|
||||
offset_basis: protobufDecodingSchema.shape.offset_basis,
|
||||
};
|
||||
const bodyRow = protobufDecodingObjectSchema
|
||||
.omit({
|
||||
pointer: true,
|
||||
body_sha256: true,
|
||||
offset_basis: true,
|
||||
frames: true,
|
||||
fields: true,
|
||||
})
|
||||
.extend({
|
||||
...body,
|
||||
frame_count: z.number().int().nonnegative(),
|
||||
field_count: z.number().int().nonnegative(),
|
||||
});
|
||||
const frameRow = protobufFrameSchema.extend(body);
|
||||
const fieldRow = protobufFieldSchema.extend(body);
|
||||
const page = {
|
||||
kind: z.literal("protobuf"),
|
||||
record_ordinals: z.array(z.number().int().nonnegative()).optional(),
|
||||
body_pointers: z.array(z.string()).optional(),
|
||||
offset: z.number().int().nonnegative().default(0),
|
||||
limit: z.number().int().positive().default(64),
|
||||
};
|
||||
|
||||
/** Batch selectors intersect; omitted filters select all retained decoding facts. */
|
||||
export const protobufViewRequestSchema = z.discriminatedUnion("collection", [
|
||||
z.strictObject({
|
||||
...page,
|
||||
collection: z.literal("bodies"),
|
||||
fields: z.array(bodyRow.keyof()).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
...page,
|
||||
collection: z.literal("frames"),
|
||||
frame_indices: z.array(z.number().int().nonnegative()).optional(),
|
||||
fields: z.array(frameRow.keyof()).optional(),
|
||||
}),
|
||||
z.strictObject({
|
||||
...page,
|
||||
collection: z.literal("fields"),
|
||||
frame_indices: z.array(z.number().int().nonnegative()).optional(),
|
||||
field_ids: z.array(z.string()).optional(),
|
||||
field_numbers: z.array(z.number().int().positive()).optional(),
|
||||
schema_names: z.array(z.string()).optional(),
|
||||
fields: z.array(fieldRow.keyof()).optional(),
|
||||
}),
|
||||
]);
|
||||
export const protobufViewItemsSchema = z.union([
|
||||
z.array(
|
||||
bodyRow.partial().required({
|
||||
record_ordinal: true,
|
||||
body_pointer: true,
|
||||
body_sha256: true,
|
||||
body_status: true,
|
||||
offset_basis: true,
|
||||
}),
|
||||
),
|
||||
z.array(
|
||||
frameRow.partial().required({
|
||||
record_ordinal: true,
|
||||
body_pointer: true,
|
||||
body_sha256: true,
|
||||
body_status: true,
|
||||
offset_basis: true,
|
||||
index: true,
|
||||
offset: true,
|
||||
}),
|
||||
),
|
||||
z.array(
|
||||
fieldRow.partial().required({
|
||||
record_ordinal: true,
|
||||
body_pointer: true,
|
||||
body_sha256: true,
|
||||
body_status: true,
|
||||
offset_basis: true,
|
||||
index: true,
|
||||
frame_index: true,
|
||||
field_id: true,
|
||||
parent_field_id: true,
|
||||
offset: true,
|
||||
payload_offset: true,
|
||||
payload_bytes: true,
|
||||
}),
|
||||
),
|
||||
]);
|
||||
export const protobufViewContextSchema = z.strictObject({
|
||||
source_status: z.enum(["complete", "partial", "unknown"]),
|
||||
retained_decoded_bodies: z.number().int().nonnegative(),
|
||||
});
|
||||
type Request = z.output<typeof protobufViewRequestSchema>;
|
||||
type Body = { recordOrdinal: number; decoding: ProtobufDecoding };
|
||||
const checkedBodies = new WeakMap<object, readonly Body[]>();
|
||||
interface FieldSelection {
|
||||
readonly key: string;
|
||||
readonly frames: ReadonlySet<number> | undefined;
|
||||
readonly ids: ReadonlySet<string> | undefined;
|
||||
readonly numbers: ReadonlySet<number> | undefined;
|
||||
readonly names: ReadonlySet<string> | undefined;
|
||||
readonly rows: WeakMap<object, readonly ProtobufField[]> | undefined;
|
||||
}
|
||||
const fieldQueries = new WeakMap<object, FieldSelection>();
|
||||
const fieldSelection = (
|
||||
report: WebNetworkCapture,
|
||||
bodies: readonly Body[],
|
||||
view: Extract<Request, { collection: "fields" }>,
|
||||
reusable: boolean,
|
||||
): FieldSelection | undefined => {
|
||||
if (
|
||||
view.frame_indices === undefined &&
|
||||
view.field_ids === undefined &&
|
||||
view.field_numbers === undefined &&
|
||||
view.schema_names === undefined
|
||||
)
|
||||
return undefined;
|
||||
const key = JSON.stringify([
|
||||
view.frame_indices,
|
||||
view.field_ids,
|
||||
view.field_numbers,
|
||||
view.schema_names,
|
||||
]);
|
||||
const cached = reusable ? fieldQueries.get(report) : undefined;
|
||||
if (cached?.key === key) return cached;
|
||||
// Selector strings must fit within one 8-byte reference per retained field.
|
||||
// One filter per parent avoids multiplying large keys/sets by its body count.
|
||||
const referenceBytes = bodies.reduce(
|
||||
(sum, body) => sum + body.decoding.fields.length * 8,
|
||||
0,
|
||||
);
|
||||
const cache = reusable && key.length * 2 <= referenceBytes;
|
||||
const selection: FieldSelection = {
|
||||
key,
|
||||
frames:
|
||||
view.frame_indices === undefined
|
||||
? undefined
|
||||
: new Set(view.frame_indices),
|
||||
ids: view.field_ids === undefined ? undefined : new Set(view.field_ids),
|
||||
numbers:
|
||||
view.field_numbers === undefined
|
||||
? undefined
|
||||
: new Set(view.field_numbers),
|
||||
names:
|
||||
view.schema_names === undefined ? undefined : new Set(view.schema_names),
|
||||
rows: cache ? new WeakMap() : undefined,
|
||||
};
|
||||
if (cache) fieldQueries.set(report, selection);
|
||||
return selection;
|
||||
};
|
||||
const selectFields = (
|
||||
decoding: ProtobufDecoding,
|
||||
selection: FieldSelection | undefined,
|
||||
): readonly ProtobufField[] => {
|
||||
if (selection === undefined) return decoding.fields;
|
||||
const cached = selection.rows?.get(decoding);
|
||||
if (cached !== undefined) return cached;
|
||||
const { frames, ids, numbers, names } = selection;
|
||||
const rows = decoding.fields.filter(
|
||||
(field) =>
|
||||
(frames === undefined || frames.has(field.frame_index)) &&
|
||||
(ids === undefined || ids.has(field.field_id)) &&
|
||||
(numbers === undefined || numbers.has(field.number)) &&
|
||||
(names === undefined ||
|
||||
(field.schema_name !== undefined && names.has(field.schema_name))),
|
||||
);
|
||||
selection.rows?.set(decoding, rows);
|
||||
return rows;
|
||||
};
|
||||
|
||||
/** Pages select owner-parsed facts; neither payloads nor descriptors are decoded again. */
|
||||
export const projectProtobufCapturePage = (
|
||||
report: WebNetworkCapture,
|
||||
view: Request,
|
||||
) => {
|
||||
const reusable = isImmutableJsonSnapshot(report);
|
||||
let bodies = reusable ? checkedBodies.get(report) : undefined;
|
||||
if (bodies === undefined) {
|
||||
bodies = report.records.flatMap((record) =>
|
||||
(record.protobuf ?? []).map((decoding) => ({
|
||||
recordOrdinal: record.ordinal,
|
||||
decoding,
|
||||
})),
|
||||
);
|
||||
if (reusable) checkedBodies.set(report, bodies);
|
||||
}
|
||||
const selection =
|
||||
view.collection === "fields"
|
||||
? fieldSelection(report, bodies, view, reusable)
|
||||
: undefined;
|
||||
const ordinals =
|
||||
view.record_ordinals === undefined
|
||||
? undefined
|
||||
: new Set(view.record_ordinals);
|
||||
const pointers =
|
||||
view.body_pointers === undefined ? undefined : new Set(view.body_pointers);
|
||||
const frames =
|
||||
"frame_indices" in view && view.frame_indices !== undefined
|
||||
? new Set(view.frame_indices)
|
||||
: undefined;
|
||||
let total = 0;
|
||||
const items: Record<string, unknown>[] = [];
|
||||
let sourceStatus: "complete" | "partial" | "unknown" =
|
||||
report.protobuf_coverage?.status === "partial"
|
||||
? "partial"
|
||||
: bodies.length === 0
|
||||
? "unknown"
|
||||
: "complete";
|
||||
for (const entry of bodies) {
|
||||
const { recordOrdinal, decoding } = entry;
|
||||
if (
|
||||
(ordinals !== undefined && !ordinals.has(recordOrdinal)) ||
|
||||
(pointers !== undefined && !pointers.has(decoding.pointer))
|
||||
)
|
||||
continue;
|
||||
if (decoding.status !== "complete") sourceStatus = "partial";
|
||||
const rows =
|
||||
view.collection === "bodies"
|
||||
? [decoding]
|
||||
: view.collection === "frames"
|
||||
? frames === undefined
|
||||
? decoding.frames
|
||||
: decoding.frames.filter((frame) => frames.has(frame.index))
|
||||
: selectFields(decoding, selection);
|
||||
const from = Math.max(0, view.offset - total);
|
||||
const to = Math.min(rows.length, view.offset + view.limit - total);
|
||||
for (let index = from; index < to; index++) {
|
||||
const source = rows[index]!;
|
||||
const row: Record<string, unknown> =
|
||||
view.collection === "bodies"
|
||||
? {
|
||||
...decoding,
|
||||
frame_count: decoding.frames.length,
|
||||
field_count: decoding.fields.length,
|
||||
}
|
||||
: source;
|
||||
const mandatory = [
|
||||
"record_ordinal",
|
||||
"body_pointer",
|
||||
"body_sha256",
|
||||
"body_status",
|
||||
"offset_basis",
|
||||
...(view.collection === "fields"
|
||||
? [
|
||||
"index",
|
||||
"frame_index",
|
||||
"field_id",
|
||||
"parent_field_id",
|
||||
"offset",
|
||||
"payload_offset",
|
||||
"payload_bytes",
|
||||
]
|
||||
: view.collection === "frames"
|
||||
? ["index", "offset"]
|
||||
: []),
|
||||
];
|
||||
const keys =
|
||||
view.fields ??
|
||||
Object.keys(row).filter(
|
||||
(key) =>
|
||||
![
|
||||
"frames",
|
||||
"fields",
|
||||
"pointer",
|
||||
"text",
|
||||
"schema_value",
|
||||
"schema_enum_values",
|
||||
].includes(key),
|
||||
);
|
||||
const chosen = new Set([...mandatory, ...keys]);
|
||||
const projected: Record<string, unknown> = {
|
||||
record_ordinal: recordOrdinal,
|
||||
body_pointer: decoding.pointer,
|
||||
body_sha256: decoding.body_sha256,
|
||||
body_status: decoding.status,
|
||||
offset_basis: decoding.offset_basis,
|
||||
};
|
||||
for (const key of keys)
|
||||
if (
|
||||
chosen.has(key) &&
|
||||
Object.hasOwn(row, key) &&
|
||||
!["frames", "fields", "pointer"].includes(key)
|
||||
)
|
||||
projected[key] = row[key];
|
||||
for (const key of mandatory)
|
||||
if (Object.hasOwn(row, key)) projected[key] = row[key];
|
||||
items.push(projected);
|
||||
}
|
||||
total += rows.length;
|
||||
}
|
||||
return {
|
||||
kind: "protobuf" as const,
|
||||
view,
|
||||
items: protobufViewItemsSchema.parse(items),
|
||||
source_status: sourceStatus,
|
||||
retained_decoded_bodies: bodies.length,
|
||||
coverage: pageViewCoverage(view.offset, items.length, total),
|
||||
};
|
||||
};
|
||||
@@ -1,3 +1,4 @@
|
||||
import { projectProtobufCapturePage } from "./protobufCaptureView.js";
|
||||
import { analysisViewInputError } from "./analysisViewErrors.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { z } from "zod";
|
||||
@@ -11,6 +12,7 @@ import {
|
||||
safeParseOwnedWebNetworkCapture,
|
||||
webNetworkCaptureArtifactSchema,
|
||||
webNetworkCaptureDecoderSchema,
|
||||
protobufCaptureCoverageSchema,
|
||||
WEB_NETWORK_CAPTURE_FORMATS,
|
||||
type WebNetworkCaptureRecord,
|
||||
type WebNetworkCapture,
|
||||
@@ -38,21 +40,43 @@ export const captureViewRequestSchema = z.strictObject({
|
||||
limit: z.number().int().positive().default(64),
|
||||
});
|
||||
|
||||
const previewSchema = webNetworkCaptureRecordSchema.extend({
|
||||
binary_fields: z.array(webNetworkCaptureBinaryDescriptorSchema),
|
||||
const protobufSummarySchema = z.strictObject({
|
||||
bodies: z.number().int().nonnegative(),
|
||||
frames: z.number().int().nonnegative(),
|
||||
fields: z.number().int().nonnegative(),
|
||||
incomplete_bodies: z.number().int().nonnegative(),
|
||||
});
|
||||
const summarizeProtobuf = (records: readonly WebNetworkCaptureRecord[]) => {
|
||||
const counts = { bodies: 0, frames: 0, fields: 0, incomplete_bodies: 0 };
|
||||
for (const record of records)
|
||||
for (const body of record.protobuf ?? []) {
|
||||
counts.bodies++;
|
||||
counts.frames += body.frames.length;
|
||||
counts.fields += body.fields.length;
|
||||
counts.incomplete_bodies += Number(body.status !== "complete");
|
||||
}
|
||||
return counts.bodies === 0 ? undefined : counts;
|
||||
};
|
||||
const previewSchema = webNetworkCaptureRecordSchema
|
||||
.omit({ protobuf: true })
|
||||
.extend({
|
||||
binary_fields: z.array(webNetworkCaptureBinaryDescriptorSchema),
|
||||
protobuf_summary: protobufSummarySchema.exactOptional(),
|
||||
});
|
||||
export const captureViewItemsSchema = z.array(
|
||||
z.union([webNetworkCaptureRecordSchema, previewSchema]),
|
||||
);
|
||||
|
||||
/** Count/identity overview; bodies and the archive member inventory stay in the parent. */
|
||||
export const captureSummarySchema = z.strictObject({
|
||||
protobuf_coverage: protobufCaptureCoverageSchema.exactOptional(),
|
||||
artifact: webNetworkCaptureArtifactSchema,
|
||||
decoder: webNetworkCaptureDecoderSchema,
|
||||
format: z.enum(WEB_NETWORK_CAPTURE_FORMATS),
|
||||
total_records: z.number().int().nonnegative(),
|
||||
retained_records: z.number().int().nonnegative(),
|
||||
runtime_attribution: z.literal("unknown"),
|
||||
protobuf: protobufSummarySchema.exactOptional(),
|
||||
qlog: qlogSummarySchema.optional(),
|
||||
});
|
||||
|
||||
@@ -369,8 +393,13 @@ const preview = (record: WebNetworkCaptureRecord, format: string) => {
|
||||
);
|
||||
}
|
||||
}
|
||||
const { protobuf: _protobuf, ...sourceRecord } = record;
|
||||
const protobufSummary = summarizeProtobuf([record]);
|
||||
return {
|
||||
...record,
|
||||
...sourceRecord,
|
||||
...(protobufSummary === undefined
|
||||
? {}
|
||||
: { protobuf_summary: protobufSummary }),
|
||||
reported: source === null ? record.reported : reported,
|
||||
binary_fields: record.binary_fields.map(
|
||||
({ content_base64: _content, ...field }) => field,
|
||||
@@ -389,6 +418,7 @@ const parseCaptureParent = (
|
||||
parent: AnalysisViewParent,
|
||||
): Result<WebNetworkCapture, AnalysisError> => {
|
||||
const parsed = safeParseOwnedWebNetworkCapture(parent.normalizedResult);
|
||||
|
||||
if (!parsed.success)
|
||||
return err(
|
||||
analysisInputErrorFromIssues(
|
||||
@@ -429,6 +459,7 @@ export const projectWebNetworkCaptureView = (
|
||||
if (
|
||||
view.kind !== "summary" &&
|
||||
view.kind !== "capture" &&
|
||||
view.kind !== "protobuf" &&
|
||||
view.kind !== "qlog"
|
||||
)
|
||||
return err(
|
||||
@@ -463,6 +494,15 @@ export const projectWebNetworkCaptureView = (
|
||||
: []),
|
||||
],
|
||||
};
|
||||
if (view.kind === "protobuf")
|
||||
return ok({
|
||||
...shared,
|
||||
...projectProtobufCapturePage(report, view),
|
||||
unknowns: [
|
||||
...shared.unknowns,
|
||||
"Decoding covers selected retained bodies only. Exhausting wire fields does not establish schema completeness or business meaning; compressed bodies and unsupported schema facets remain unknown.",
|
||||
],
|
||||
});
|
||||
if (view.kind === "summary")
|
||||
return ok({
|
||||
kind: "summary",
|
||||
@@ -475,6 +515,12 @@ export const projectWebNetworkCaptureView = (
|
||||
total_records: report.total_records,
|
||||
retained_records: report.records.length,
|
||||
runtime_attribution: report.runtime_attribution,
|
||||
...(report.protobuf_coverage === undefined
|
||||
? {}
|
||||
: { protobuf_coverage: report.protobuf_coverage }),
|
||||
...(summarizeProtobuf(report.records) === undefined
|
||||
? {}
|
||||
: { protobuf: summarizeProtobuf(report.records)! }),
|
||||
...(report.format === "qlog"
|
||||
? { qlog: summarizeQlogCapture(report.qlog) }
|
||||
: {}),
|
||||
|
||||
@@ -14,6 +14,7 @@ import { ProviderCleanupError } from "./providerCleanupError.js";
|
||||
import { ProviderAdapterError } from "./providerAdapterError.js";
|
||||
import { analysisErrorWithCleanupFailure } from "./analysisErrorCleanup.js";
|
||||
import { resolvedAnalysisError } from "./analysisErrorBase.js";
|
||||
import { createEvidence } from "./evidence.js";
|
||||
import {
|
||||
redactExplicitFailure,
|
||||
redactExplicitPointer,
|
||||
@@ -55,7 +56,20 @@ it.each([
|
||||
),
|
||||
new ProviderAdapterError("provider", "inspect", context),
|
||||
])("preserves sanitized common context for $._tag", (error) => {
|
||||
const observation = createEvidence(
|
||||
undefined,
|
||||
{ id: "fixture", name: "Owner facts", version: "1" },
|
||||
{
|
||||
operation: "inspect",
|
||||
parameters: {},
|
||||
result: { source: "already sanitized owner facts" },
|
||||
confidence: "observed",
|
||||
authority: "historical-reference",
|
||||
},
|
||||
);
|
||||
error.retainPartialObservation(observation);
|
||||
const sanitized = redactExplicitFailure(error, ["explicit-secret"]);
|
||||
expect(sanitized.partialObservation).toBe(observation);
|
||||
expect(sanitized._tag).toBe(error._tag);
|
||||
const projected = projectAnalysisError(sanitized);
|
||||
expect(projected.details).toMatchObject({
|
||||
|
||||
@@ -41,6 +41,10 @@ export const redactExplicitFailure = (
|
||||
): AnalysisError => {
|
||||
if (values.length === 0) return error;
|
||||
const redacted = redactFailure(error, values);
|
||||
// Observation owners already apply format-aware exclusions. Keep their facts
|
||||
// and identity across every diagnostic replacement, including input/capacity.
|
||||
if (redacted !== error && error.partialObservation !== undefined)
|
||||
redacted.retainPartialObservation(error.partialObservation);
|
||||
return redacted === error
|
||||
? error
|
||||
: mirrorCleanup(error, redacted, (value) =>
|
||||
@@ -118,12 +122,10 @@ const redactFailure = (
|
||||
return new AnalysisCancelledError(error.operation, {
|
||||
...context,
|
||||
cause: error.cause,
|
||||
partialObservation: error.partialObservation,
|
||||
});
|
||||
if (error instanceof AnalysisTimeoutError)
|
||||
return new AnalysisTimeoutError(error.operation, error.timeoutMs, {
|
||||
...context,
|
||||
partialObservation: error.partialObservation,
|
||||
});
|
||||
if (error instanceof AnalysisAccessDeniedError)
|
||||
return new AnalysisAccessDeniedError(
|
||||
@@ -199,17 +201,11 @@ const redactFailure = (
|
||||
{
|
||||
...context,
|
||||
operation: error.operation,
|
||||
...(error.partialObservation === undefined
|
||||
? {}
|
||||
: { partialObservation: error.partialObservation }),
|
||||
},
|
||||
);
|
||||
if (error instanceof ProviderAdapterError)
|
||||
return new ProviderAdapterError(error.providerId, error.operation, {
|
||||
...context,
|
||||
...(error.partialObservation === undefined
|
||||
? {}
|
||||
: { partialObservation: error.partialObservation }),
|
||||
...(error.userMessage === undefined
|
||||
? {}
|
||||
: { userMessage: text(error.userMessage) }),
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
import { z } from "zod";
|
||||
import { localPathStringSchema } from "./localPath.js";
|
||||
|
||||
/** Optional decoding applies only to explicitly selected capture records. */
|
||||
export const protobufOptionsSchema = z.strictObject({
|
||||
framing: z
|
||||
.enum(["auto", "raw", "grpc", "grpc-web", "grpc-web-text"])
|
||||
.default("auto"),
|
||||
body_pointers: z.array(z.string()).optional(),
|
||||
schema: z
|
||||
.strictObject({
|
||||
descriptor_set_path: localPathStringSchema,
|
||||
message_type: z.string().min(1),
|
||||
})
|
||||
.optional(),
|
||||
});
|
||||
|
||||
const index = z.number().int().nonnegative();
|
||||
const issueSchema = z.strictObject({ offset: index, reason: z.string() });
|
||||
|
||||
/** Indices are scoped to the capture artifact, producer ordinal and body pointer. */
|
||||
export const protobufFieldSchema = z.strictObject({
|
||||
index,
|
||||
frame_index: index,
|
||||
field_id: z.string(),
|
||||
parent_field_id: z.string().nullable(),
|
||||
parent_index: index.nullable(),
|
||||
number: z.number().int().positive().max(0x1fffffff),
|
||||
wire_type: z.enum(["varint", "i64", "len", "i32", "group"]),
|
||||
offset: index,
|
||||
bytes: index,
|
||||
payload_offset: index,
|
||||
payload_bytes: index,
|
||||
value: z.string().exactOptional(),
|
||||
text: z.string().exactOptional(),
|
||||
interpretations: z
|
||||
.array(
|
||||
z.enum([
|
||||
"utf8-string",
|
||||
"nested-message",
|
||||
"packed-varint",
|
||||
"packed-fixed32",
|
||||
"packed-fixed64",
|
||||
"bytes",
|
||||
]),
|
||||
)
|
||||
.exactOptional(),
|
||||
schema_name: z.string().exactOptional(),
|
||||
schema_type: z.string().exactOptional(),
|
||||
schema_value: z.string().exactOptional(),
|
||||
schema_enum_values: z
|
||||
.array(z.strictObject({ number: z.string(), names: z.array(z.string()) }))
|
||||
.exactOptional(),
|
||||
schema_status: z
|
||||
.enum([
|
||||
"decoded",
|
||||
"unknown-field",
|
||||
"wire-type-mismatch",
|
||||
"invalid-message",
|
||||
"unsupported",
|
||||
])
|
||||
.exactOptional(),
|
||||
schema_error: issueSchema.exactOptional(),
|
||||
});
|
||||
|
||||
export const protobufFrameSchema = z.strictObject({
|
||||
index,
|
||||
kind: z.enum(["data", "trailer"]),
|
||||
offset: index,
|
||||
payload_offset: index,
|
||||
bytes: index,
|
||||
compressed: z.boolean(),
|
||||
status: z.enum(["complete", "partial", "unknown"]),
|
||||
examined_bytes: index,
|
||||
error: issueSchema.exactOptional(),
|
||||
trailers: z
|
||||
.array(z.strictObject({ name: z.string(), value: z.string() }))
|
||||
.exactOptional(),
|
||||
});
|
||||
|
||||
/** Flat facts avoid recursive advertised schemas and preserve same-number occurrences. */
|
||||
export const protobufDecodingObjectSchema = z.strictObject({
|
||||
pointer: z.string(),
|
||||
content_type: z.string().nullable(),
|
||||
decoder: z.literal("rea-protobuf-wire-v1"),
|
||||
body_sha256: z.string().regex(/^[0-9a-f]{64}$/),
|
||||
framing: z.enum(["raw", "grpc", "grpc-web", "grpc-web-text"]),
|
||||
offset_basis: z.enum(["retained-body", "base64-decoded-body"]),
|
||||
decoded_bytes: index,
|
||||
status: z.enum(["complete", "partial", "unknown"]),
|
||||
examined_bytes: index,
|
||||
frames: z.array(protobufFrameSchema),
|
||||
fields: z.array(protobufFieldSchema),
|
||||
error: issueSchema.exactOptional(),
|
||||
limitations: z.array(z.string()),
|
||||
schema: z
|
||||
.strictObject({
|
||||
decoder: z.literal("protobuf-es-2.16.0"),
|
||||
descriptor_set_path: z.string(),
|
||||
descriptor_set_sha256: z.string().regex(/^[0-9a-f]{64}$/),
|
||||
descriptor_set_bytes: index,
|
||||
message_type: z.string(),
|
||||
})
|
||||
.exactOptional(),
|
||||
});
|
||||
/** Validate coordinate relationships at the owner, before any consumer projects rows. */
|
||||
export const protobufDecodingSchema = protobufDecodingObjectSchema.superRefine(
|
||||
(body, context) => {
|
||||
const issue = (path: (string | number)[], message: string) =>
|
||||
context.addIssue({ code: "custom", path, message });
|
||||
if (body.examined_bytes > body.decoded_bytes)
|
||||
issue(["examined_bytes"], "Examined bytes exceed the decoded body");
|
||||
if (
|
||||
body.status === "complete" &&
|
||||
(body.error !== undefined ||
|
||||
body.frames.some((frame) => frame.status !== "complete"))
|
||||
)
|
||||
issue(["status"], "Complete body contradicts frame coverage");
|
||||
let expectedOffset = 0;
|
||||
for (const [index, frame] of body.frames.entries()) {
|
||||
if (
|
||||
frame.offset !== expectedOffset ||
|
||||
frame.payload_offset !== frame.offset + (body.framing === "raw" ? 0 : 5)
|
||||
)
|
||||
issue(
|
||||
["frames", index, "offset"],
|
||||
"Frames must preserve contiguous protocol ranges",
|
||||
);
|
||||
expectedOffset = frame.payload_offset + frame.bytes;
|
||||
if (
|
||||
frame.index !== index ||
|
||||
frame.payload_offset < frame.offset ||
|
||||
frame.payload_offset + frame.bytes > body.decoded_bytes ||
|
||||
frame.examined_bytes > frame.bytes
|
||||
)
|
||||
issue(
|
||||
["frames", index],
|
||||
"Frame identity or range contradicts the body",
|
||||
);
|
||||
if (
|
||||
frame.status === "complete" &&
|
||||
(frame.error !== undefined ||
|
||||
frame.examined_bytes !== frame.bytes ||
|
||||
frame.compressed)
|
||||
)
|
||||
issue(
|
||||
["frames", index, "status"],
|
||||
"Complete frame contradicts decoding coverage",
|
||||
);
|
||||
if (body.framing === "grpc" && frame.kind === "trailer")
|
||||
issue(
|
||||
["frames", index, "kind"],
|
||||
"Native gRPC has no in-body trailer frame",
|
||||
);
|
||||
}
|
||||
if (
|
||||
body.status === "complete" &&
|
||||
(body.examined_bytes !== body.decoded_bytes ||
|
||||
expectedOffset !== body.decoded_bytes)
|
||||
)
|
||||
issue(["status"], "Complete decoding must cover the whole decoded body");
|
||||
const ids = new Set<string>();
|
||||
for (const [index, field] of body.fields.entries()) {
|
||||
const frame = body.frames[field.frame_index];
|
||||
if (
|
||||
field.index !== index ||
|
||||
field.field_id !== `offset:${field.offset}` ||
|
||||
ids.has(field.field_id)
|
||||
)
|
||||
issue(["fields", index], "Field occurrence identity is inconsistent");
|
||||
ids.add(field.field_id);
|
||||
if (
|
||||
frame === undefined ||
|
||||
frame.kind !== "data" ||
|
||||
field.offset < frame.payload_offset ||
|
||||
field.offset + field.bytes > frame.payload_offset + frame.bytes ||
|
||||
field.payload_offset < field.offset ||
|
||||
field.payload_offset + field.payload_bytes > field.offset + field.bytes
|
||||
)
|
||||
issue(["fields", index], "Field range contradicts its frame");
|
||||
const parent =
|
||||
field.parent_index === null
|
||||
? undefined
|
||||
: body.fields[field.parent_index];
|
||||
if (
|
||||
field.parent_index === null
|
||||
? field.parent_field_id !== null
|
||||
: parent === undefined ||
|
||||
parent.index >= index ||
|
||||
parent.field_id !== field.parent_field_id ||
|
||||
parent.frame_index !== field.frame_index ||
|
||||
field.offset < parent.payload_offset ||
|
||||
field.offset + field.bytes >
|
||||
parent.payload_offset + parent.payload_bytes
|
||||
)
|
||||
issue(
|
||||
["fields", index, "parent_field_id"],
|
||||
"Field parent contradicts its source range",
|
||||
);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
export type ProtobufDecoding = z.output<typeof protobufDecodingSchema>;
|
||||
export type ProtobufField = z.output<typeof protobufFieldSchema>;
|
||||
export type ProtobufFrame = z.output<typeof protobufFrameSchema>;
|
||||
@@ -1,6 +1,10 @@
|
||||
import { z } from "zod";
|
||||
import { jsonValueSchema } from "./jsonValue.js";
|
||||
import { localPathStringSchema } from "./localPath.js";
|
||||
import {
|
||||
protobufDecodingSchema,
|
||||
protobufOptionsSchema,
|
||||
} from "./protobufWire.js";
|
||||
import { qlogCaptureFactsSchema } from "./qlogCapture.js";
|
||||
import { canBorrowParsedJsonSnapshot } from "./immutableJson.js";
|
||||
|
||||
@@ -32,6 +36,7 @@ export const inspectWebNetworkCaptureInputSchema = z.strictObject({
|
||||
.describe(
|
||||
"Zero-based producer record ordinals, in caller-selected order. Omit to inspect every record.",
|
||||
),
|
||||
protobuf: protobufOptionsSchema.optional(),
|
||||
sensitive_values: z
|
||||
.array(z.string().min(1))
|
||||
.default([])
|
||||
@@ -128,6 +133,12 @@ export const webNetworkCaptureRecordSchema = z.strictObject({
|
||||
numeric_literals: z.array(numberSchema),
|
||||
redactions: z.array(redactionSchema),
|
||||
limitations: z.array(z.string()),
|
||||
protobuf: z
|
||||
.array(protobufDecodingSchema)
|
||||
.exactOptional()
|
||||
.describe(
|
||||
"Flat decoding facts for explicitly selected records. Field indices are local to a body; offset_basis states whether ranges refer to retained or base64-decoded bytes. Interpretations without a caller-supplied schema are candidates.",
|
||||
),
|
||||
});
|
||||
|
||||
/** The selected offline parser profile, independent of authenticated producer identity. */
|
||||
@@ -138,7 +149,26 @@ export const webNetworkCaptureDecoderSchema = z.strictObject({
|
||||
});
|
||||
|
||||
/** Source coverage is shared by every offline decoder. */
|
||||
export const protobufCaptureCoverageSchema = z
|
||||
.strictObject({
|
||||
status: z.enum(["complete", "partial"]),
|
||||
selected_records: z.number().int().nonnegative(),
|
||||
inspected_records: z.number().int().nonnegative(),
|
||||
})
|
||||
.superRefine((coverage, context) => {
|
||||
if (
|
||||
coverage.inspected_records > coverage.selected_records ||
|
||||
(coverage.status === "complete" &&
|
||||
coverage.inspected_records !== coverage.selected_records)
|
||||
)
|
||||
context.addIssue({
|
||||
code: "custom",
|
||||
path: ["status"],
|
||||
message: "Protobuf coverage contradicts its inspected selection.",
|
||||
});
|
||||
});
|
||||
const payloadShape = {
|
||||
protobuf_coverage: protobufCaptureCoverageSchema.exactOptional(),
|
||||
decoder: webNetworkCaptureDecoderSchema,
|
||||
container: z.strictObject({
|
||||
reported: jsonValueSchema,
|
||||
|
||||
@@ -0,0 +1,530 @@
|
||||
import { z } from "zod";
|
||||
import { readFile, unlink, writeFile, mkdir, truncate } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect } from "vitest";
|
||||
import { cliTest } from "../../support/cli/cliFixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import {
|
||||
protobufCaptureDescriptor,
|
||||
protobufHar,
|
||||
protobufFrame,
|
||||
protobufCaptureBody,
|
||||
} from "../../fixtures/protobufCapture.js";
|
||||
import { parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import {
|
||||
webNetworkCaptureSchema,
|
||||
inspectWebNetworkCaptureInputSchema,
|
||||
} from "../../../src/domain/webNetworkCapture.js";
|
||||
import { readEvidenceBundle } from "../../../src/application/EvidenceBundleFiles.js";
|
||||
import { analysisViewResultSchema } from "../../../src/domain/analysisView/analysisView.js";
|
||||
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
|
||||
type CaptureCall = Awaited<ReturnType<typeof connectLocalToolsMcp>>["call"];
|
||||
type CaptureRequest = z.input<typeof inspectWebNetworkCaptureInputSchema>;
|
||||
|
||||
const retainedCaptureDetailsSchema = z.object({
|
||||
partial_observation: z.strictObject({
|
||||
kind: z.literal("retained-evidence"),
|
||||
evidence_id: z.string(),
|
||||
}),
|
||||
});
|
||||
|
||||
async function verifyCaptureExclusions(
|
||||
call: CaptureCall,
|
||||
args: CaptureRequest,
|
||||
root: string,
|
||||
): Promise<void> {
|
||||
const redacted = await call("inspect_web_network_capture", {
|
||||
...args,
|
||||
sensitive_values: ["hello"],
|
||||
});
|
||||
expect(redacted.isError, JSON.stringify(redacted)).not.toBe(true);
|
||||
const excluded = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(redacted.structuredContent).normalized_result,
|
||||
);
|
||||
expect(excluded.records[0]?.protobuf).toBeUndefined();
|
||||
expect(excluded.protobuf_coverage).toMatchObject({
|
||||
status: "partial",
|
||||
selected_records: 1,
|
||||
inspected_records: 1,
|
||||
});
|
||||
expect(JSON.stringify(redacted)).not.toContain("hello");
|
||||
const textPath = join(root, "text.har");
|
||||
await writeFile(
|
||||
textPath,
|
||||
protobufHar(
|
||||
Buffer.from(protobufCaptureBody().toString("base64")),
|
||||
"application/grpc-web-text",
|
||||
),
|
||||
);
|
||||
const textExcluded = await call("inspect_web_network_capture", {
|
||||
capture_path: textPath,
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
sensitive_values: ["hello"],
|
||||
});
|
||||
expect(textExcluded.isError, JSON.stringify(textExcluded)).not.toBe(true);
|
||||
const excludedText = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(textExcluded.structuredContent).normalized_result,
|
||||
);
|
||||
expect(excludedText.records[0]?.binary_fields[0]).toMatchObject({
|
||||
state: "redacted",
|
||||
content_base64: null,
|
||||
sha256: null,
|
||||
});
|
||||
expect(excludedText.records[0]?.reported).toMatchObject({
|
||||
response: { content: { text: null } },
|
||||
});
|
||||
expect(excludedText.records[0]?.protobuf).toBeUndefined();
|
||||
const credentialPath = join(root, "trailer.har");
|
||||
await writeFile(
|
||||
credentialPath,
|
||||
protobufHar(
|
||||
Buffer.concat([
|
||||
protobufFrame(Buffer.from("0a0568656c6c6f", "hex")),
|
||||
protobufFrame(
|
||||
Buffer.from(
|
||||
"grpc-status: 0\r\nauthorization: Bearer fixture-only-credential\r\n",
|
||||
),
|
||||
128,
|
||||
),
|
||||
]),
|
||||
),
|
||||
);
|
||||
const credential = await call("inspect_web_network_capture", {
|
||||
capture_path: credentialPath,
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
});
|
||||
expect(credential.isError, JSON.stringify(credential)).not.toBe(true);
|
||||
const excludedCredential = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(credential.structuredContent).normalized_result,
|
||||
);
|
||||
expect(excludedCredential.records[0]?.binary_fields[0]).toMatchObject({
|
||||
state: "redacted",
|
||||
content_base64: null,
|
||||
});
|
||||
expect(excludedCredential.records[0]?.redactions).toContainEqual({
|
||||
pointer: "/response/content/text",
|
||||
reason: "transport-credential",
|
||||
});
|
||||
expect(excludedCredential.records[0]?.protobuf).toBeUndefined();
|
||||
expect(JSON.stringify(credential)).not.toContain("fixture-only-credential");
|
||||
}
|
||||
|
||||
async function verifyPartialCaptureSources(
|
||||
call: CaptureCall,
|
||||
args: CaptureRequest,
|
||||
root: string,
|
||||
descriptor: string,
|
||||
): Promise<void> {
|
||||
const missing = await call("inspect_web_network_capture", {
|
||||
...args,
|
||||
sensitive_values: ["unrelated fixture exclusion"],
|
||||
protobuf: {
|
||||
schema: {
|
||||
descriptor_set_path: descriptor,
|
||||
message_type: "capture.Missing",
|
||||
},
|
||||
},
|
||||
});
|
||||
const missingError = parseMcpToolError(missing).error;
|
||||
expect(missingError.code).toBe("invalid_request");
|
||||
const retainedSource = retainedCaptureDetailsSchema.parse(
|
||||
missingError.details,
|
||||
).partial_observation;
|
||||
const retainedSummary = await call("inspect_analysis_view", {
|
||||
source: retainedSource,
|
||||
view: { kind: "summary" },
|
||||
});
|
||||
expect(retainedSummary.isError, JSON.stringify(retainedSummary)).not.toBe(
|
||||
true,
|
||||
);
|
||||
expect(
|
||||
analysisViewResultSchema.parse(
|
||||
parseEvidence(retainedSummary.structuredContent).normalized_result,
|
||||
),
|
||||
).toMatchObject({
|
||||
summary: {
|
||||
protobuf_coverage: { status: "partial", inspected_records: 0 },
|
||||
},
|
||||
});
|
||||
const missingView = await call("inspect_analysis_view", {
|
||||
source: retainedSource,
|
||||
view: { kind: "protobuf", collection: "fields", offset: 0, limit: 1 },
|
||||
});
|
||||
expect(missingView.isError, JSON.stringify(missingView)).not.toBe(true);
|
||||
expect(
|
||||
analysisViewResultSchema.parse(
|
||||
parseEvidence(missingView.structuredContent).normalized_result,
|
||||
),
|
||||
).toMatchObject({ source_status: "partial", items: [] });
|
||||
const capacityPath = join(root, "capacity.har");
|
||||
const repeated = Buffer.alloc(200000);
|
||||
for (let index = 0; index < repeated.length; index += 2) {
|
||||
repeated[index] = 8;
|
||||
repeated[index + 1] = 1;
|
||||
}
|
||||
await writeFile(capacityPath, protobufHar(protobufFrame(repeated)));
|
||||
const capacity = await call("inspect_web_network_capture", {
|
||||
capture_path: capacityPath,
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
sensitive_values: ["unrelated fixture exclusion"],
|
||||
});
|
||||
const capacityError = parseMcpToolError(capacity).error;
|
||||
expect(capacityError.code).toBe("resource_constraint");
|
||||
expect(capacityError.retryable).toBe(false);
|
||||
const capacitySource = retainedCaptureDetailsSchema.parse(
|
||||
capacityError.details,
|
||||
).partial_observation;
|
||||
const capacitySummary = await call("inspect_analysis_view", {
|
||||
source: capacitySource,
|
||||
view: { kind: "summary" },
|
||||
});
|
||||
expect(capacitySummary.isError, JSON.stringify(capacitySummary)).not.toBe(
|
||||
true,
|
||||
);
|
||||
expect(
|
||||
analysisViewResultSchema.parse(
|
||||
parseEvidence(capacitySummary.structuredContent).normalized_result,
|
||||
),
|
||||
).toMatchObject({
|
||||
summary: {
|
||||
protobuf_coverage: { status: "partial", inspected_records: 0 },
|
||||
},
|
||||
});
|
||||
expect(JSON.stringify(capacity)).not.toContain("content_base64");
|
||||
}
|
||||
|
||||
async function verifyCredentialPreflight(
|
||||
call: CaptureCall,
|
||||
root: string,
|
||||
descriptor: string,
|
||||
): Promise<void> {
|
||||
const repeated = Buffer.alloc(200000);
|
||||
for (let index = 0; index < repeated.length; index += 2) {
|
||||
repeated[index] = 8;
|
||||
repeated[index + 1] = 1;
|
||||
}
|
||||
const large = protobufFrame(repeated);
|
||||
const credential = protobufFrame(
|
||||
Buffer.from(
|
||||
"grpc-status: 0\r\nauthorization: fixture-only-trailer-credential\r\n",
|
||||
),
|
||||
128,
|
||||
);
|
||||
const small = protobufFrame(Buffer.from([8, 1]));
|
||||
for (const [name, body, mimeType, framing] of [
|
||||
[
|
||||
"large",
|
||||
Buffer.concat([large, credential]),
|
||||
"application/grpc-web+proto",
|
||||
"auto",
|
||||
],
|
||||
[
|
||||
"malformed",
|
||||
Buffer.concat([
|
||||
small,
|
||||
protobufFrame(
|
||||
Buffer.from(
|
||||
"malformed\r\nauthorization: fixture-only-trailer-credential\r\n",
|
||||
),
|
||||
128,
|
||||
),
|
||||
]),
|
||||
"application/grpc-web+proto",
|
||||
"auto",
|
||||
],
|
||||
[
|
||||
"text-override",
|
||||
Buffer.from(Buffer.concat([small, credential]).toString("base64")),
|
||||
"application/grpc-web-text",
|
||||
"raw",
|
||||
],
|
||||
] as const) {
|
||||
const path = join(root, `credential-${name}.har`);
|
||||
await writeFile(path, protobufHar(body, mimeType));
|
||||
const response = await call("inspect_web_network_capture", {
|
||||
capture_path: path,
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
protobuf: { framing },
|
||||
});
|
||||
expect(response.isError, JSON.stringify(response)).not.toBe(true);
|
||||
const capture = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(response.structuredContent).normalized_result,
|
||||
);
|
||||
expect(capture.records[0]?.binary_fields[0]).toMatchObject({
|
||||
state: "redacted",
|
||||
content_base64: null,
|
||||
sha256: null,
|
||||
});
|
||||
expect(capture.records[0]?.protobuf).toBeUndefined();
|
||||
expect(capture.protobuf_coverage?.status).toBe("partial");
|
||||
}
|
||||
const path = join(root, "credential-later-record.har");
|
||||
await writeFile(
|
||||
path,
|
||||
protobufHar(large, "application/grpc-web+proto", [
|
||||
Buffer.concat([small, credential]),
|
||||
]),
|
||||
);
|
||||
for (const schema of [
|
||||
undefined,
|
||||
{ descriptor_set_path: descriptor, message_type: "capture.Missing" },
|
||||
]) {
|
||||
const response = await call("inspect_web_network_capture", {
|
||||
capture_path: path,
|
||||
format: "har",
|
||||
record_ordinals: [0, 1],
|
||||
sensitive_values: ["unrelated fixture exclusion"],
|
||||
...(schema === undefined ? {} : { protobuf: { schema } }),
|
||||
});
|
||||
const error = parseMcpToolError(response).error;
|
||||
expect(error.code).toBe(
|
||||
schema === undefined ? "resource_constraint" : "invalid_request",
|
||||
);
|
||||
const source = retainedCaptureDetailsSchema.parse(
|
||||
error.details,
|
||||
).partial_observation;
|
||||
const retained = await call("inspect_analysis_view", {
|
||||
source,
|
||||
view: { kind: "capture", detail: "complete", offset: 0, limit: 2 },
|
||||
});
|
||||
expect(retained.isError, JSON.stringify(retained)).not.toBe(true);
|
||||
const view = analysisViewResultSchema.parse(
|
||||
parseEvidence(retained.structuredContent).normalized_result,
|
||||
);
|
||||
expect(view).toMatchObject({
|
||||
items: [
|
||||
expect.anything(),
|
||||
{
|
||||
binary_fields: [
|
||||
{ state: "redacted", content_base64: null, sha256: null },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyDescriptorReadFailures(
|
||||
call: CaptureCall,
|
||||
args: CaptureRequest,
|
||||
root: string,
|
||||
): Promise<void> {
|
||||
const directory = join(root, "descriptor-directory");
|
||||
const oversized = join(root, "oversized.pb");
|
||||
await mkdir(directory);
|
||||
await writeFile(oversized, "");
|
||||
await truncate(oversized, 32 * 1024 * 1024 + 1);
|
||||
for (const [path, reason] of [
|
||||
[join(root, "missing.pb"), "invalid_value"],
|
||||
[directory, "invalid_format"],
|
||||
[oversized, "out_of_range"],
|
||||
] as const) {
|
||||
const response = await call("inspect_web_network_capture", {
|
||||
...args,
|
||||
sensitive_values: ["unrelated fixture exclusion"],
|
||||
protobuf: {
|
||||
schema: { descriptor_set_path: path, message_type: "capture.Message" },
|
||||
},
|
||||
});
|
||||
const error = parseMcpToolError(response).error;
|
||||
expect(error.code).toBe("invalid_request");
|
||||
expect(error.details).toMatchObject({
|
||||
issues: [{ path: ["protobuf", "schema", "descriptor_set_path"], reason }],
|
||||
});
|
||||
expect(
|
||||
retainedCaptureDetailsSchema.parse(error.details).partial_observation
|
||||
.kind,
|
||||
).toBe("retained-evidence");
|
||||
}
|
||||
}
|
||||
|
||||
cliTest(
|
||||
"decodes selected capture bytes with a supplied descriptor and pages exported facts after both sources disappear",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-protobuf-cli-");
|
||||
const capture = join(root, "capture.har");
|
||||
const descriptor = join(root, "capture.pb");
|
||||
const bundle = join(root, "evidence.json");
|
||||
await writeFile(capture, protobufHar());
|
||||
await writeFile(descriptor, protobufCaptureDescriptor());
|
||||
const original = await cli.run({
|
||||
arguments: ["inspect-web-network-capture", capture, "har", "--json"],
|
||||
});
|
||||
expect(original.exitCode, original.stderr).toBe(0);
|
||||
expect(
|
||||
webNetworkCaptureSchema.parse(
|
||||
parseEvidence(original.json).normalized_result,
|
||||
).records[0],
|
||||
).not.toHaveProperty("protobuf");
|
||||
const decoded = await cli.run({
|
||||
arguments: [
|
||||
"inspect-web-network-capture",
|
||||
capture,
|
||||
"har",
|
||||
"--record",
|
||||
"0",
|
||||
"--protobuf-schema",
|
||||
descriptor,
|
||||
"--protobuf-message",
|
||||
"capture.Message",
|
||||
"--detail",
|
||||
"summary",
|
||||
"--evidence_output",
|
||||
bundle,
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(decoded.exitCode, decoded.stdout + decoded.stderr).toBe(0);
|
||||
const loaded = await readEvidenceBundle(bundle);
|
||||
if (!loaded.ok) throw loaded.error;
|
||||
const parent = loaded.value.records.find(
|
||||
(record) => record.operation === "inspect_web_network_capture",
|
||||
);
|
||||
if (parent === undefined)
|
||||
throw new Error("Complete capture Evidence must be retained");
|
||||
const report = webNetworkCaptureSchema.parse(parent.normalized_result);
|
||||
expect(report.records[0]?.protobuf?.[0]).toMatchObject({
|
||||
status: "complete",
|
||||
schema: { message_type: "capture.Message" },
|
||||
});
|
||||
expect(report.records[0]?.protobuf?.[0]?.fields[0]).toMatchObject({
|
||||
schema_name: "sentence",
|
||||
schema_value: "hello",
|
||||
field_id: "offset:5",
|
||||
});
|
||||
await unlink(capture);
|
||||
await unlink(descriptor);
|
||||
const request = join(root, "view.json");
|
||||
await writeFile(
|
||||
request,
|
||||
JSON.stringify({
|
||||
source: { kind: "inline", evidence: parent },
|
||||
view: {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
field_ids: ["offset:5"],
|
||||
fields: ["schema_name", "schema_value"],
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
},
|
||||
}),
|
||||
);
|
||||
const viewed = await cli.run({
|
||||
arguments: ["inspect-analysis-view", request, "--json"],
|
||||
});
|
||||
expect(viewed.exitCode, viewed.stdout + viewed.stderr).toBe(0);
|
||||
const view = analysisViewResultSchema.parse(
|
||||
parseEvidence(viewed.json).normalized_result,
|
||||
);
|
||||
expect(view).toMatchObject({
|
||||
kind: "protobuf",
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
source_status: "complete",
|
||||
coverage: { total: 1, examined: 1, exhausted: true },
|
||||
items: [
|
||||
{
|
||||
record_ordinal: 0,
|
||||
body_pointer: "/response/content/text",
|
||||
field_id: "offset:5",
|
||||
schema_name: "sentence",
|
||||
schema_value: "hello",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(JSON.stringify(view)).not.toContain("content_base64");
|
||||
expect((await readFile(bundle)).length).toBeGreaterThan(0);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"uses production MCP schemas, retained pages and CLI-equivalent decoding without reopening sources",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-protobuf-mcp-");
|
||||
const capture = join(root, "capture.har");
|
||||
const descriptor = join(root, "capture.pb");
|
||||
await writeFile(capture, protobufHar());
|
||||
await writeFile(descriptor, protobufCaptureDescriptor());
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const args: CaptureRequest = {
|
||||
capture_path: capture,
|
||||
format: "har",
|
||||
record_ordinals: [0],
|
||||
protobuf: {
|
||||
schema: {
|
||||
descriptor_set_path: descriptor,
|
||||
message_type: "capture.Message",
|
||||
},
|
||||
},
|
||||
};
|
||||
const complete = await call("inspect_web_network_capture", args);
|
||||
expect(complete.isError, JSON.stringify(complete)).not.toBe(true);
|
||||
const parent = parseEvidence(complete.structuredContent);
|
||||
const cliResult = await cli.run({
|
||||
arguments: [
|
||||
"inspect-web-network-capture",
|
||||
capture,
|
||||
"har",
|
||||
"--record",
|
||||
"0",
|
||||
"--protobuf-schema",
|
||||
descriptor,
|
||||
"--protobuf-message",
|
||||
"capture.Message",
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(cliResult.exitCode, cliResult.stdout + cliResult.stderr).toBe(0);
|
||||
expect(parseEvidence(cliResult.json).evidence_id).toBe(parent.evidence_id);
|
||||
await verifyCaptureExclusions(call, args, root);
|
||||
await verifyPartialCaptureSources(call, args, root, descriptor);
|
||||
await verifyCredentialPreflight(call, root, descriptor);
|
||||
await verifyDescriptorReadFailures(call, args, root);
|
||||
const summary = await call("inspect_web_network_capture", {
|
||||
...args,
|
||||
detail: "summary",
|
||||
});
|
||||
expect(summary.isError, JSON.stringify(summary)).not.toBe(true);
|
||||
const result = analysisViewResultSchema.parse(
|
||||
parseEvidence(summary.structuredContent).normalized_result,
|
||||
);
|
||||
expect(result).toMatchObject({
|
||||
kind: "summary",
|
||||
parent_evidence_id: parent.evidence_id,
|
||||
});
|
||||
await unlink(capture);
|
||||
await unlink(descriptor);
|
||||
const viewed = await call("inspect_analysis_view", {
|
||||
source: { kind: "retained-evidence", evidence_id: parent.evidence_id },
|
||||
view: {
|
||||
kind: "protobuf",
|
||||
collection: "fields",
|
||||
field_numbers: [2],
|
||||
fields: ["schema_name", "schema_value"],
|
||||
offset: 0,
|
||||
limit: 1,
|
||||
},
|
||||
});
|
||||
expect(viewed.isError, JSON.stringify(viewed)).not.toBe(true);
|
||||
expect(
|
||||
analysisViewResultSchema.parse(
|
||||
parseEvidence(viewed.structuredContent).normalized_result,
|
||||
),
|
||||
).toMatchObject({
|
||||
kind: "protobuf",
|
||||
items: [{ schema_name: "values", schema_value: '["1","2"]' }],
|
||||
coverage: { total: 1, exhausted: true },
|
||||
});
|
||||
const invalid = await call("inspect_web_network_capture", {
|
||||
...args,
|
||||
record_ordinals: undefined,
|
||||
});
|
||||
expect(invalid.isError).toBe(true);
|
||||
expect(parseMcpToolError(invalid).error.code).toBe("invalid_request");
|
||||
},
|
||||
);
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
} from "../../../src/domain/webNetworkCapture.js";
|
||||
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import { protobufFrame } from "../../fixtures/protobufCapture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
|
||||
import {
|
||||
@@ -113,6 +114,181 @@ const cli = async (path: string, extra: string[] = []) => {
|
||||
return parseEvidence(JSON.parse(result.stdout));
|
||||
};
|
||||
|
||||
it("uses HTTP optional whitespace for SAZ identity codings without changing raw headers", async () => {
|
||||
const body = Buffer.from([0, 0, 0, 0, 2, 8, 1]);
|
||||
const codings = [
|
||||
" \tidentity\t ",
|
||||
"\tIdEnTiTy ",
|
||||
" gzip ",
|
||||
" \u00a0identity",
|
||||
];
|
||||
const entries: [string, Buffer][] = codings.map((coding, index) => [
|
||||
`raw/${index + 1}_s.txt`,
|
||||
Buffer.concat([
|
||||
Buffer.from(
|
||||
`HTTP/1.1 200 OK\r\nContent-Type: application/grpc\r\nContent-Encoding:${coding}\r\nContent-Length: ${body.length}\r\n\r\n`,
|
||||
"latin1",
|
||||
),
|
||||
body,
|
||||
]),
|
||||
]);
|
||||
const path = await fixture(await archive(entries));
|
||||
const evidence = await cli(path, [
|
||||
"--record",
|
||||
"0",
|
||||
"--record",
|
||||
"1",
|
||||
"--record",
|
||||
"2",
|
||||
"--record",
|
||||
"3",
|
||||
"--protobuf-framing",
|
||||
"grpc",
|
||||
]);
|
||||
const capture = webNetworkCaptureSchema.parse(evidence.normalized_result);
|
||||
for (const [index, coding] of codings.entries()) {
|
||||
expect(capture.records[index]?.reported).toMatchObject({
|
||||
response: {
|
||||
headers: expect.arrayContaining([
|
||||
{ name: "Content-Encoding", value: coding },
|
||||
]),
|
||||
},
|
||||
});
|
||||
if (index < 2)
|
||||
expect(capture.records[index]?.protobuf).toMatchObject([
|
||||
{
|
||||
pointer: "/response/body",
|
||||
status: "complete",
|
||||
fields: [{ number: 1, value: "1" }],
|
||||
},
|
||||
]);
|
||||
else expect(capture.records[index]?.protobuf).toBeUndefined();
|
||||
}
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const response = await call("inspect_web_network_capture", {
|
||||
capture_path: path,
|
||||
format: "saz",
|
||||
record_ordinals: [0, 1, 2, 3],
|
||||
protobuf: { framing: "grpc" },
|
||||
});
|
||||
expect(response.isError, JSON.stringify(response)).not.toBe(true);
|
||||
expect(parseEvidence(response.structuredContent).evidence_id).toBe(
|
||||
evidence.evidence_id,
|
||||
);
|
||||
for (const sensitive of ["gzip", "Content-Encoding"]) {
|
||||
const excluded = await call("inspect_web_network_capture", {
|
||||
capture_path: path,
|
||||
format: "saz",
|
||||
record_ordinals: [0, 1, 2, 3],
|
||||
protobuf: { framing: "grpc" },
|
||||
sensitive_values: [sensitive],
|
||||
});
|
||||
expect(excluded.isError, JSON.stringify(excluded)).not.toBe(true);
|
||||
const report = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(excluded.structuredContent).normalized_result,
|
||||
);
|
||||
expect(report.protobuf_coverage).toMatchObject({
|
||||
status: "partial",
|
||||
inspected_records: 4,
|
||||
});
|
||||
if (sensitive === "gzip")
|
||||
expect(report.records[2]?.protobuf).toBeUndefined();
|
||||
else
|
||||
expect(
|
||||
report.records.every((record) => record.protobuf === undefined),
|
||||
).toBe(true);
|
||||
for (const record of report.records)
|
||||
expect(record.binary_fields).toContainEqual(
|
||||
expect.objectContaining({
|
||||
pointer: "/response/body",
|
||||
state: "retained",
|
||||
}),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("checks trailer credentials before hidden metadata blocks analysis and withholds unchecked source", async () => {
|
||||
const credential = Buffer.concat([
|
||||
protobufFrame(Buffer.from([8, 1])),
|
||||
protobufFrame(Buffer.from("authorization: Bearer fixture-only\r\n"), 128),
|
||||
]);
|
||||
const responseBytes = (body: Buffer, contentType: string, extra = "") =>
|
||||
Buffer.concat([
|
||||
Buffer.from(
|
||||
`HTTP/1.1 200 OK\r\nContent-Type: ${contentType}\r\nX-Debug: harmless\r\n${extra}\r\n`,
|
||||
),
|
||||
body,
|
||||
]);
|
||||
const bytes = await archive([
|
||||
["raw/1_s.txt", responseBytes(credential, "application/grpc-web+proto")],
|
||||
[
|
||||
"raw/2_s.txt",
|
||||
responseBytes(Buffer.from([0, 0, 0]), "application/grpc-web+proto"),
|
||||
],
|
||||
[
|
||||
"raw/3_s.txt",
|
||||
responseBytes(
|
||||
gzipSync(credential),
|
||||
"application/grpc-web+proto",
|
||||
"Content-Encoding: gzip\r\n",
|
||||
),
|
||||
],
|
||||
[
|
||||
"raw/4_s.txt",
|
||||
responseBytes(Buffer.from("invalid*base64"), "application/grpc-web-text"),
|
||||
],
|
||||
]);
|
||||
const path = await fixture(bytes);
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const base = {
|
||||
capture_path: path,
|
||||
format: "saz",
|
||||
sensitive_values: ["X-Debug"],
|
||||
};
|
||||
const protectedReply = await call("inspect_web_network_capture", {
|
||||
...base,
|
||||
record_ordinals: [0],
|
||||
});
|
||||
expect(protectedReply.isError, JSON.stringify(protectedReply)).not.toBe(true);
|
||||
const protectedCapture = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(protectedReply.structuredContent).normalized_result,
|
||||
);
|
||||
expect(protectedCapture.records[0]?.binary_fields).toContainEqual(
|
||||
expect.objectContaining({
|
||||
pointer: "/response/body",
|
||||
state: "redacted",
|
||||
content_base64: null,
|
||||
sha256: null,
|
||||
}),
|
||||
);
|
||||
expect(protectedCapture.records[0]?.protobuf).toBeUndefined();
|
||||
for (const ordinals of [[0, 1], [2], [3]]) {
|
||||
const response = await call("inspect_web_network_capture", {
|
||||
...base,
|
||||
record_ordinals: ordinals,
|
||||
});
|
||||
const error = parseMcpToolError(response).error;
|
||||
expect(error.code).toBe("capability_unavailable");
|
||||
expect(JSON.stringify(response)).not.toContain("content_base64");
|
||||
expect(error.details).not.toHaveProperty("partial_observation");
|
||||
}
|
||||
const opaqueReply = await call("inspect_web_network_capture", {
|
||||
capture_path: path,
|
||||
format: "saz",
|
||||
record_ordinals: [2],
|
||||
});
|
||||
expect(opaqueReply.isError, JSON.stringify(opaqueReply)).not.toBe(true);
|
||||
const opaque = webNetworkCaptureSchema.parse(
|
||||
parseEvidence(opaqueReply.structuredContent).normalized_result,
|
||||
);
|
||||
expect(opaque.records[0]?.binary_fields).toContainEqual(
|
||||
expect.objectContaining({ pointer: "/response/body", state: "retained" }),
|
||||
);
|
||||
expect(opaque.records[0]?.protobuf).toBeUndefined();
|
||||
expect(opaque.protobuf_coverage?.status).toBe("partial");
|
||||
expect(await readFile(path)).toEqual(bytes);
|
||||
});
|
||||
|
||||
it("inspects native SAZ through the public CLI and MCP with equal evidence, bytes and archive coordinates", async () => {
|
||||
const bytes = await archive();
|
||||
const path = await fixture(bytes);
|
||||
|
||||
Vendored
+83
@@ -0,0 +1,83 @@
|
||||
import { create, toBinary } from "@bufbuild/protobuf";
|
||||
import { FileDescriptorSetSchema } from "@bufbuild/protobuf/wkt";
|
||||
|
||||
export const protobufFrame = (body: Buffer, flag = 0): Buffer => {
|
||||
const header = Buffer.alloc(5);
|
||||
header[0] = flag;
|
||||
header.writeUInt32BE(body.length, 1);
|
||||
return Buffer.concat([header, body]);
|
||||
};
|
||||
|
||||
export const protobufCaptureBody = () =>
|
||||
Buffer.concat([
|
||||
protobufFrame(Buffer.from("0a0568656c6c6f12020102", "hex")),
|
||||
protobufFrame(Buffer.from("grpc-status: 0\r\n"), 128),
|
||||
]);
|
||||
|
||||
/** Original fixture schema; independent expected field values come from the literal wire bytes. */
|
||||
export const protobufCaptureDescriptor = () =>
|
||||
toBinary(
|
||||
FileDescriptorSetSchema,
|
||||
create(FileDescriptorSetSchema, {
|
||||
file: [
|
||||
{
|
||||
name: "capture.proto",
|
||||
package: "capture",
|
||||
syntax: "proto3",
|
||||
messageType: [
|
||||
{
|
||||
name: "Message",
|
||||
field: [
|
||||
{ name: "sentence", number: 1, type: 9, label: 1 },
|
||||
{ name: "values", number: 2, type: 13, label: 3 },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
|
||||
export const protobufHar = (
|
||||
body: Buffer = protobufCaptureBody(),
|
||||
mimeType = "application/grpc-web+proto",
|
||||
additionalBodies: readonly Buffer[] = [],
|
||||
) =>
|
||||
JSON.stringify({
|
||||
log: {
|
||||
version: "1.2",
|
||||
creator: { name: "REA original fixture", version: "1" },
|
||||
entries: [body, ...additionalBodies].map((body) => ({
|
||||
startedDateTime: "2026-10-20T00:00:00Z",
|
||||
time: 1,
|
||||
request: {
|
||||
method: "POST",
|
||||
url: "https://example.invalid/api",
|
||||
httpVersion: "HTTP/1.1",
|
||||
cookies: [],
|
||||
headers: [],
|
||||
queryString: [],
|
||||
headersSize: -1,
|
||||
bodySize: 0,
|
||||
},
|
||||
response: {
|
||||
status: 200,
|
||||
statusText: "OK",
|
||||
httpVersion: "HTTP/1.1",
|
||||
cookies: [],
|
||||
headers: [],
|
||||
redirectURL: "",
|
||||
headersSize: -1,
|
||||
bodySize: body.length,
|
||||
content: {
|
||||
size: body.length,
|
||||
mimeType,
|
||||
text: body.toString("base64"),
|
||||
encoding: "base64",
|
||||
},
|
||||
},
|
||||
cache: {},
|
||||
timings: { send: 0, wait: 1, receive: 0 },
|
||||
})),
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user