Normalize raw URL separators after admitting an original relative ESM
request, before artifact path validation and root confinement. Preserve
the authored declaration and the existing encoded-separator checks.
Retain Node-backed static and dynamic loader controls and compare the
application graph with the historical importer already present on main.
Narrow PR #4323 to this remaining case after the upstream #4269 fix.
Signed-off-by: Jiakai Xu <jiakaiPeanut@gmail.com>
U-Boot's mkimage writes the multi-file size table for script images
(`mkimage -T script`, the boot.scr format) exactly as it does for multi
images, `mkimage -l` lists the script as "Image 0", and the `source`
command skips the table before running the script ("scripts are just
multi-image files with one component", boot/image-board.c).
The uImage reader parsed the table only for IH_TYPE_MULTI, so a script
image reported sub_image_count 0 and exposed only payload.bin, whose
first bytes are the binary size table rather than the script text.
Treat IH_TYPE_SCRIPT like IH_TYPE_MULTI: parse the same bounded table,
expose image-<n>.bin components and report the size-table limitation.
payload.bin is unchanged. A script payload without a valid table now
fails closed, as a multi image already does.
* fix(cli): align MCP doctor completion with its options
* fix(cli): explain zsh completion initialization
* fix(cli): include launcher timing in doctor completion
* style(cli): format doctor completion schema
* feat(artifacts): inventory and extract ESP32 firmware images
ESP32-family app and bootloader images (magic 0xE9) had no reader, so
open_binary refused OTA downloads, app partitions and bootloaders.
Add the esp-image container format behind inspect_artifact and
extract_artifact. A pure decoder reads each byte once and keeps only
fixed-size fields. Derived facts cover the chip, from a table cited to
ESP-IDF v6.1; segment classification as ESP-IDF's should_map and
should_load; checksum and appended SHA-256 results; app and bootloader
descriptors; Secure Boot v2 signature blocks, observed and unverified;
and undescribed trailing byte ranges. Truncation and undescribed bytes
make coverage unknown. ESP8266 images are refused with a specific
reason, and ciphertext stays unrecognized.
Add the format to every target, Evidence, snapshot, live-summary and
artifact vocabulary, document it, and add an optional verify:esp-images
lane that compares esptool-built images with esptool image-info and
espsecure signature-info-v2.
Refs #3615
* test(artifacts): run the ESP image lane on esptool v5.6.0
The lane compared descriptor fields only when esptool image-info printed
one, so a descriptor that REA reports and esptool does not print passed
without comparison. Compare the descriptor kind in both directions.
Record that the lane passed for all 13 images with esptool and espsecure
v5.6.0, the version the layout facts cite, as well as with 4.12.0.
* refactor(artifacts): define ESP image layout sizes once
The decoder and the derived facts each defined the segment-header,
appended-digest and flash-sector sizes, the erased-byte scan and the
offset formatter. Define them once with the other fixed-layout structures
and import them in both places. The signature slot count is the one
constant that drives the slot scan.
Correct the observedEnd comment: it is the end of the last chunk read.
* fix(artifacts): report ESP-IDF's segment length limit by chip
ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more
unless the chip maps flash through 32-bit cache addresses
(SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and
S31 in v6.1). Record that capability in the chip table and add a
limitation for such segments; unknown chips get none.
`loaded` repeats should_load. It does not mean the bootloader accepts the
segment: verify_load_addresses still rejects one outside the chip's RAM,
which REA does not model. Say so in the docs, the skill reference and the
inventory limitation.
* test(artifacts): check ESP lane tools first and verify extraction
The lane started work before checking its tools, so a missing RISC-V
target, ld.lld, esptool or espsecure surfaced as a raw spawn or command
error. Check each one before building and name the variable to set. Warn
when esptool is not v5.6.0, the release the layout facts cite.
Assert each segment's class for the ESP32-C3 app, its variants and the
bootloader, as the Xtensa image already did. Extract the app and the
signed image through the CLI and compare every file with the bytes at
esptool's segment offsets or in the appended signature sector; the
source's SHA-256 must be unchanged and equal the reported root_sha256.
* docs(artifacts): state what the ESP image lane does not establish
The Verification section listed the lane's checks but not its gaps: only
ESP32-C3 and ESP32 images come from a real producer, no real project
build, OTA download or flash dump is inspected, no signature is verified,
and the lane does not run in CI.
Record the timing of the final lane run on esptool v5.6.0.
* fix(artifacts): compare ESP segment declarations
Segment entries hash only their data, so compare_artifacts reported a
segment that moved to another load address with the same bytes as
unchanged; only the image itself changed. Compare each segment's
load_addr and data_len as its declaration, as Node SEA and PyInstaller
members are compared, and count a changed declaration as a metadata
change, also when content is unavailable.
Header offsets are layout and the classification is derived, so neither
is compared.
* test(artifacts): run ESP lane commands as owned process trees
The lane ran its tools through execFile. A timeout killed only the direct
child, so a linker or assembler that Clang or GCC had started could keep
running while the lane removed its workspace. Run every command through
createVerifierCommands, check the tools inside withVerifierCleanup, and
remove the workspace only after commands.close() has stopped every tree;
a failed close keeps the workspace.
Only a nonzero exit now counts as a refusal or an expected esptool
failure; timeouts and spawn failures propagate. Supervising each command
roughly doubles the lane's run time.
* test(artifacts): cancel ESP decoding after its first decoded chunk
The cancellation test requested cancel or close on setImmediate, which
can land while the file is still being opened and checked, so it never
showed that decoding had started. It also passed against a reader that
ignored both requests while decoding. Acknowledge the decoder's first
chunk, then cancel or close, join the operation, and assert that no
later chunk reaches the decoder and that the reader ends closed.
* docs(artifacts): record which ESP metadata comparison ignores
Segments are compared by their declaration. The image's and the
signature blocks' format_metadata are not: every field comes from their
own bytes, which content comparison already covers, and a block's
image_digest_matches follows the image's content.
* docs(artifacts): state that ESP mmu_page_size is a base-2 logarithm
The app descriptor's mmu_page_size keeps ESP-IDF's name and value, the
MMU page size in log base 2 as esp_app_desc_t declares it, so 16 means
64 KiB. Say so in docs/esp-image.md and the schema comment; a reader
could otherwise take 16 as 16 KiB.
* fix(esp-image): retain member consumption and share lifecycle regressions
* test(artifacts): scope MCP catalogs and budget full-schema validation
* test(artifacts): verify compact output profiles and document bounded catalogs
* test(artifacts): close readers when cleanup-test admission fails
* test(artifacts): compare ESP image limitations with their owners
The decoder and reader tests froze complete limitation sentences, so a
harmless rewording failed them. The wording now has two owners:
ESP_IMAGE_LIMITATIONS in espImageFacts.ts, which the decoder, the inventory
and the reader use, and ESP_DESCRIPTOR_LIMITATIONS beside the descriptor
decoder. Tests compare delivered limitations with the owner, called with the
facts each case establishes (segment index, declared and available bytes,
offsets, chip, page range), next to the structured assertions they already
make. Delivered text is unchanged.
* test(esp-image): assert structured facts and owned guidance
* fix(esp-image): keep spi_flash_mode and descriptor fields before a truncated tail
ESP-IDF v6.1's esp_app_desc_t declares spi_flash_mode at byte 181, after
mmu_page_size, as the committed reference layout shows; the descriptor now
keeps it as stored. The decoder also required all 256 descriptor bytes,
though it reads none past byte 181, so a file ending in the reserved tail
lost the version, project, digest and revision facts it does hold. The app
descriptor now needs only its fields; the segment's truncation is reported
as before.
* test(esp-image): pin the P-192 public key digest to espsecure's layout
espsecure packs ECDSA signature blocks as "<BBBx32sB64s64s1031x" and
zero-pads a 48-byte P-192 point to 64 bytes, so the signature starts at
byte 101 for every SHA-256 curve. signature_info_v2 hashes sig_block[36:101]
and _digest_sbv2_public_key the curve ID with the padded point. No test built
a P-192 block; this one checks public_key_sha256 against that definition.
* fix(esp-image): keep each app descriptor field the file holds in full
A file can end anywhere inside the 256-byte esp_app_desc_t. The decoder
needed every field byte, so a cut at byte 176 dropped the version, project
and app_elf_sha256 it does hold. Once the 4-byte magic identifies an app
descriptor, each field is now kept when the file holds all of its bytes and
is null otherwise; observed_length records where the descriptor ends, so a
cut field is told apart from text that is not UTF-8. The fields after the
magic become nullable in the metadata schema.
* fix(esp-image): report a signature block whose image digest does not match
A CRC-valid Secure Boot v2 block can carry another image's digest. The
block recorded image_digest_matches false, but no limitation said so, unlike
a checksum or appended-SHA mismatch, so CLI and MCP summaries showed only
the generic unverified note. The mismatch is now a limitation; coverage
stays complete and signature_status stays unverified. The three other
signature-sector limitations move into ESP_IMAGE_LIMITATIONS with their
text unchanged, so all ESP image wording has one owner.
* refactor(esp-image): remove redundant descriptor byte guards
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Seal sensitive terminal chunks before deferred rendering, apply one bounded
capture-local literal matcher, and retain exact redacted byte accounting.
Keep private input facts in explicit fields and separate comparison policies.
Retain distinct terminal failures once and verify structured refusal facts.
Fixes#7782
BREAKING CHANGE: sensitive interaction data is empty with sensitive and input_bytes fields instead of a byte-count placeholder. Sensitive raw_data is also redacted. Read input_bytes and recapture both comparison sides when migrating older private-input Evidence.
Assisted-by: Reimu (gensokyo-skills:reimu-incident-triage)
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
Assisted-by: Yukari (gensokyo-skills:yukari-boundary-analysis)
CSP3 parsing keeps the first directive of each name in a policy and ignores
later repeats; Chrome reports "Ignoring duplicate Content-Security-Policy
directive". parseCsp flattened every policy's directives into one stream, so
a header such as `script-src 'self'; script-src 'unsafe-inline'` reported the
ignored `'unsafe-inline'` directive as declared and counted nonces and hashes
from it. Track directive names per policy so only the effective directive is
retained, while the same name in separate comma-separated or repeated-field
policies is still listed once per policy.
* fix(javascript): keep both-direction feature traces from crossing shared containers
trace_application_feature with direction "both" walked the application
graph as if it were undirected. From a seed, one outgoing or incoming step
into a shared container (an asset or the artifact root) followed by steps
in the other direction reached every sibling of that container, so a trace
seeded on one storage key returned the whole connected graph.
"both" now returns the union of the outgoing and incoming closures. Each
walk keeps its own predecessor chains, so a terminal path never changes
direction midway. "outgoing" and "incoming" are unchanged.
* test(javascript): check both-direction trace paths end to end
The existing tests checked predecessor maps but no returned path. Add a
trace whose seed reaches one storage terminal outgoing and another
incoming, and assert each path's complete node_ids and edge_ids. A
pathTo that reads only the first walk's map returns a one-node path for
the incoming terminal and fails this test.
* fix(javascript): confine only ascended containers in both-direction traces
Making "both" the union of separate outgoing and incoming walks dropped
modules that a feature reaches by crossing directions. In the Notes
known-answer evaluation, the notes:export channel is pointed at by both
the preload asset (invokes) and the handler (handles); its preload,
renderer and csv modules are reached only through relationships that
change direction, so the trace kept main.js alone and the feature_modules
claim failed in tests/boundary/mcp/agentKnownAnswerMcp.test.ts.
Walk both directions in one traversal again, and confine only the
containment case the change was meant to stop: a node reached by stepping
up a contains edge from one of its children follows only further contains
steps up. A shared asset or artifact root then no longer leads into its
other children, while relationships across the feature are kept. The
Notes trace returns csv.js, main.js, preload.js and renderer.js again
(30 nodes; the previous main returned the whole 32-node graph).
outgoing and incoming traces are unchanged, and paths come from one
predecessor map again, as before this branch.
* fix(javascript): stop double-expanding application projection estimates
estimateJavaScriptStaticApplicationProjection multiplied the per-record
metadata allowance (2 KB plus the path) by the 8x expansion that it also
applies to checkpoint bytes. A projected record serializes to about 0.46 KB,
so the allowance alone already covers retention; expanded again, the
charge was about 25x the serialized application relationships.
The charge accumulates in the parent's shared result budget. On a
synthetic 2.8 MB Vite-style build (88 files, one 456 KB chunk), the first
10 files used 2.0 GB of a 2.08 GB budget while the parent heap held about
300 MiB. The other 78 files lost their application relationships and were
offered a 0-byte semantic budget. On a 201-file build the charges exceeded
the budget at file 93 and the analysis failed with resource_constraint,
leaving the remaining sources without static facts.
Keep the 8x expansion for checkpoint and repeated bytes and count the
per-record allowance once. The same builds now refuse 14 and 102
projections instead of 78 and a failed run; peak RSS stays at 1.2 GB and
2.0 GB under a 4.2 GB parent heap limit.
workerFailureComposition simulated 64 KiB of parent result budget, which
the corrected charge for its one-finding checkpoint fits. Lower that
headroom to 32 KiB so the case still exercises a refused projection.
* test(javascript): cover application projections near the shared parent budget
The estimate test compares one file's charge with its serialized graph,
but no test put several sources against one parent result budget. Replay
one recorded static checkpoint for five sources against an exact budget:
sources 1-3 fit, source 4 misses by half a source and keeps its static
facts with the reported required and budget bytes, and the last source
fits again once no checkpoint reserve is held back. A worker that charges
the projection twice, or not at all, passes the existing tests and fails
this one.
* test(javascript): cover a malformed source before a valid one under the shared budget
The testing guide asks representation-expansion and capacity changes for
a large malformed input followed by a valid input, and for escaped or
expanded representations.
Under an exact parent budget, a 3 MiB checkpoint cut off inside a string
comes first and is rejected as unreadable output with no static facts.
The valid source after it keeps its application relationships; it fits
only if the rejected bytes took no charge. A worker that charges the
static checkpoint on receipt, before parsing, passes the existing tests
and fails this one.
The estimate ratio test also runs with a nested source path and storage
keys holding quotes, backslashes, a tab and multibyte text, which expand
in serialized JSON; the charge stays within 3x to 8x of the serialized
application graph (5.3x).
* fix(javascript): resolve Vite build-output preload dependencies
With an absolute base, Vite writes __vite__mapDeps entries relative to the
build output directory (for example assets/chunk.js), and its preload helper
prefixes the configured base at runtime. REA emitted these entries as
dynamic-import module specifiers, so a scheme-less path without a leading
dot resolved as a bare package and every entry was reported as an
unresolved "external" module, even when the build output directory was
selected.
The static analyzer now marks such entries with a vite-output: specifier,
as Webpack chunk loads are marked with chunk:. Entries starting with ./,
../ or /, and scheme-qualified entries, keep their previous handling. The
graph resolves a marked entry as a URL path under each ancestor directory
of the importing file; exactly one ancestor holding the file establishes
the build output directory. Otherwise the entry is not-found (or
unavailable when several ancestors match) with a limitation that names
the build output directory.
On a synthetic 7-file Vite build, the 12 preload entries that were
"external" now resolve when the build output directory is selected, and
are not-found with that guidance when only assets/ is selected.
* fix(javascript): keep unanalyzed Vite output candidates
resolveViteOutputDependency considered only resolved ancestors. A file
that exists under an ancestor but was not analyzed (for example
dist/assets/lazy.css for the entry assets/lazy.css) resolves there as
unavailable, and that result was replaced by the root's not-found with a
limitation saying that no ancestor holds the file.
Keep resolved and unavailable ancestor candidates together. One candidate
is returned with its own path and limitations; several are reported as
unavailable with every candidate path and limitation. Only when no
ancestor can hold the file does the root's not-found result stand.
Apktool 2.9.0 replaced SnakeYAML with the YamlWriter that 3.x still uses:
apktool.yml starts with an untagged `version: 2.x.y` line and strings are
written unquoted with Java-style escapes. Only a `version: 3.x.y` header
selected the producer dialect, so 2.9.0-2.12.x output went through YAML
core parsing: a versionName of `1.0 #beta` became `1.0`, escaped quotes kept
their backslashes, an empty name became null, and `1.0: beta` made the whole
decode fail as invalid metadata.
Select the line-oriented reader for 2.9.0 and later as well. Older tagged
SnakeYAML documents keep YAML scalar semantics.
Every source string literal without an established Next root was tested
against /(?:^|\/)app\/(?:[\s\S]*\/)?page(?:\.[cm]?[jt]sx?|-[^/?#]+\.js|)(?:[?#].*)?$/u.
The unanchored search restarted at every app/ segment and rescanned to the
end of the literal, and each page? or page# candidate scanned to the next
line terminator before $ failed. A long literal therefore took quadratic
time on the analysis thread: about 6 s for a 256 KB literal of repeated
app/ segments, and 3.5 s for 224 KB of x/page? followed by a newline.
The first app/ segment admits every later page segment, and . cannot cross
a line terminator, so a query or fragment after the page filename starts on
the last line. Test the tail after the first app/ segment one page segment
at a time instead; results are unchanged.
Use one command owner for setup, client writers, Gemini policy and CLI defaults.
Migrate recognized runner registrations with backups and remove runner prewarm.
Retain unknown identities and exercise real offline npm, CLI and MCP workflows.
Fixes#7757
BREAKING CHANGE: managed MCP registrations use the selected absolute runtime
and REA entry script, including package-runner cache paths. Keep that installation
available or restore it before setup; unknown registrations require confirmed
manual repair. Setup no longer returns prewarm or emits its progress events.
Assisted-by: Reimu (gensokyo-skills:reimu-incident-triage)
Assisted-by: Kogasa (gensokyo-skills:kogasa-surprise-testing)
Assisted-by: Yukari (gensokyo-skills:yukari-boundary-analysis)
* feat(artifacts): add U-Boot legacy uImage reader (part of #7129)
Recognize legacy uImages by their 0x27051956 magic and inventory them as containers: 64-byte header facts (arch/os/type/compression raw and named, load/entry addresses, name, timestamp) with header and data CRC32 verified, stored payload extractable as payload.bin, and multi-file images split from the stored size table into image-<n>.bin. Compressed payloads stay as stored; nothing is executed or decompressed. Wires the format through classification, the artifact reader dispatch, extraction, and the evidence/snapshot format vocabularies. Related to #7129 (uImage part only); HEX/SREC and FIT are out of scope.
* fix(uimage): preserve aligned component ranges and exact header facts
* fix(uimage): retain buffered members through consumption and cancellation
* fix(uimage): preserve active read and validation failure precedence
---------
Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
gensokyo-dev