diff --git a/docs/tooling-audit.md b/docs/tooling-audit.md index f21c09f77..a45818c15 100644 --- a/docs/tooling-audit.md +++ b/docs/tooling-audit.md @@ -634,6 +634,11 @@ where ENGINE_TYPES is parsed once from src/shared/engine_types.h (that alone rec - **assertion to add:** Two, because the current selftest cannot see this. (a) In parse_overlay_c, after building `items`: `assert not [it for it in items if it[2] not in REAL_KINDS and _has_definition_header(it[3])]` — i.e. NO item's preamble may contain a `{`-bodied function-definition header that is not that item's own anchor; fail loud with file:line. (b) Add a real COVERAGE gate to selftest() alongside the round-trip: crudely count candidate anchors (INCLUDE_ASM lines + col-0 `^\w+\s*\(\s*func_` macro invocations + col-0 `}` lines) and `assert n_anchors >= n_col0_close_braces - n_type_blocks`, printing the delta. The existing 'round-trip exact + 0 unresolved + monotonic' triple is 100% green on a file with two missed definitions and must never again be reported as proof of coverage. ### [MEDIUM] `tools/jr_isolate_all.py :: jr_inventory() — `re.fullmatch(r'func_[0-9A-Fa-f]{8}', fn)` on the .s basename (line 81)` +- **✅ FIXED (A9g, §26-A):** the `.s` basename is now resolved through the symbol table + (`overlay_src_split.addr_of()`), which handles BOTH `func_` and a curated name, so the handwritten + `listCdBuffer` jr (0x80180000, in ov_SC01_084/ov_SC03_108/ov_SC03_118/ov_SC03_119) is no longer dropped. + Fixed together with the roster finding below (same rewrite of `jr_inventory`). *(The `--only` path's own + `func_`-fullmatch — a 2nd instance — is left: it parses user-supplied core names, not the corpus.)* - candidates **5899** / parsed **5895** / **real skips 4** - **evidence:** Candidate = every .s under asm/ov_*/nonmatchings/*/ whose text references a `jtbl_` symbol (5,899). Parsed: 5,895. The 4 misses are all the same function under a CURATED name: asm/ov_SC03_119/nonmatchings/ov_SC03_119_jr_80178D40/listCdBuffer.s, and the same in ov_SC03_118, ov_SC01_084, ov_SC03_108. This is NOT a data label — it is a genuine 0xA64-byte handwritten jr function at 0x80180000 with 2 jtbl references ('/* Handwritten function */ nonmatching listCdBuffer, 0xA64' / `glabel listCdBuffer` / `sra $v0, $v0, 16` ...), and it is INCLUDE_ASM'd from real source: src/ov_SC03_119/ov_SC03_119_jr_80178D40.c:4442 `INCLUDE_ASM("asm/ov_SC03_119/nonmatchings/ov_SC03_119_jr_80178D40", listCdBuffer);`. Because its name is `listCdBuffer` and not `func_XXXXXXXX`, the fullmatch filter drops it and jr_inventory reports it does not exist. (Related, LOW: the same curated name defeats jtbl_carve.all_data_labels' `(?:jtbl_|D_)[0-9A-Fa-f]{8}` regex in the 33 OTHER overlays where 0x80180000 is data — `dlabel listCdBuffer` at asm/ov_SC01_000/data/tail.data.s:2380 is invisible to the carve-boundary oracle. I checked exploitability and it is currently NIL: listCdBuffer sits at 0x80180000 in the general data region, never immediately after a jtbl, and jtbl_words' enddlabel-bounded trailing-zero trim clamps `end` to the true extent regardless of a missed boundary label. The boundary oracle is silently incomplete but the trim accidentally masks it — worth an assertion, not a fix.) - **blast radius:** Lost MATCHES (4 potential banks), and it falsifies the tool's core invariant. jr_isolate_all's docstring and its lines 118-131 establish the rule that EVERY jr in a cut object must get its own region, because 'a region may host AT MOST ONE .rodata carve' — that invariant is what the func_8015AE2C/func_801734BC +33-byte image corruption taught. listCdBuffer is a jr that the tool cannot see, and in these 4 overlays it currently sits INSIDE `_jr_80178D40`, sharing a region with an already-banked jr. So the invariant 'every jr has its own region' is false in 4 overlays right now. The moment listCdBuffer is matched and banked, jtbl_carve hits its 'subseg would host NON-CONTIGUOUS .rodata carves' fail-loud (jtbl_carve.py:250-254) and the bank cannot proceed without a hand-isolation. Fail-loud, so no byte corruption — but 4 banks are blocked and the blocker will present as a mysterious carve error rather than a naming gap. @@ -1301,7 +1306,16 @@ The exemplar did not produce different bytes. It produced NO bytes. The operator - **assertion (R32):** R32: assert that 'byte-diff' is only ever printed when an output binary was actually produced (os.path.exists(build//) and the build's own compile steps all exited 0). If no binary exists, the verdict MUST be COMPILE-FAIL. Cover with a test that feeds bank_exemplar a body with an undeclared struct and asserts the reported class is COMPILE-FAIL, not byte-diff. - **skeptic:** RAN: (a) read tools/bank_exemplar.py — cited code confirmed at :65 (success test) and :68-70 (4-term allowlist + "byte-diff" fallback). (b) Re-ran the repro with the PINNED compiler the build actually uses (Makefile:443 -> CC1_PSX = tools/bin/gcc-2.7.2-psx/cc1, not the cdk cc1 the claim used) on .run/audit/skeptic/diag2.c: "storage size of `x' isn't known" / "too many arguments to function `g'" / "dereferencing pointer to incomplete type" -> classifier matched 0/3 -> "recovered: byte-diff". Mechanism CONFIRMED. Note the claim's cdk evidence was off the real build path: cdk says "syntax error", psx says "parse error" — a HIT under the real compiler. (c) `strings` over the psx cc1 diagnostic vocabulary: 12/44 error-shaped strings matched; all 9 incomplete-type variants, all 4 arg-count variants, incompatible-types, invalid-lvalue, called-object-is-not-a-function escape. COUNTS REFUTED: candidates=3/parsed=0 are three lines of a SYNTHETIC file the claimant wrote, not corpus items. The real corpus is the 11 bank_exemplar run logs in .run/bank*.log: ~30 stage-FAIL lines, 27 classified CORRECTLY (conflicting types / undeclared / redefinition; ValueError paths bypass the classifier). "byte-diff" was emitted exactly ONCE, for one function (func_8013F350), triple-logged across bank9.log/bank_all.log/bank_func_8013F350.log — and phase-ends/CURRENT_PHASE.md:267 records the independent follow-up proving that verdict TRUE ("NOT a plumbing bug — it is a real class... D_8011511C must be struct-typed to force la+offset; no cast fixes it"). It compiled and produced different bytes. So real mislabels in the corpus = 0; the allowlist covers 100% of the fault vocabulary this project's staged ladder has actually produced. BLAST RADIUS latent as claimed: fb.revert()+sys.exit(1) on every failure path, success gated on the SHA1 "[ OK ]" string, and grep -rn "byte-diff" finds no programmatic consumer — operator-facing prose only. Real shape, reproducible on demand, zero occurrences: LOW/latent hardening, not a MEDIUM live bug. The R33 fix (print the compiler's words + exit code, classify nothing) is still correct and cheap. -### [LOW] `jr_isolate_all.py` — DOWNGRADED +### [LOW] `jr_isolate_all.py` — DOWNGRADED → ✅ FIXED (A9g, R33) +> The ephemeral roster is GONE. `jr_inventory`'s `banked` is now DERIVED: a real-C def/define fn is a +> banked jr iff `family_remap.reloc_targets` shows it references a committed `.rodata` carve offset (the +> image + the config, both durable). No `.run/banked_func_*.json` glob, no `cand` filter — so a `rm -rf +> .run`/fresh clone no longer blinds it, the CROSS-ADDRESS sibling `func_8017FCB0` is now found (roster +> missed it), and NON-LEADER banked jr (carve in the object's own subseg, not a `_jr_` leader) are found +> where a subseg-name model would miss them. The R32 assertion from :1320 is implemented: every committed +> carve must resolve to exactly ONE owner or `jr_inventory` aborts. Image read once + passed to +> `reloc_targets(…, data=)` (new backward-compatible param), ~6s→fast. Verified: the 3 audit orphans +> (ov_SC01_000/ov_SC02_000/ov_SC02_003) all resolve; full-fleet 1:1 ownership holds (no false abort). - **scanner:** jr_inventory() — the `banked` set: tools/jr_isolate_all.py:85-87 `for bj in glob.glob(REPO/.run/banked_func_*.json): cand.add(basename[len('banked_'):-len('.json')])`, then :94 `banked = {a: nm for a, nm in realc.items() if nm in cand}` - **counts:** candidates **1202** / parsed **1199** / real skips **3** - **evidence:** OVER-APPROX = every committed `.rodata` carve piece in the 134 overlay configs (1202). Each is, by construction, owned by a banked jr. REAL = carves whose owner jr_inventory+carve_owners can resolve = 1199. THE 3 ORPHANS, all confirmed against the real corpus: diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index ff0bd9f4e..0bd9bd47f 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -659,6 +659,26 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-14 (session 13, A9g — jr_inventory: retire the ephemeral roster, derive banked from the image; Max):** + R33 applied to "the purest R33 case in the group" (audit). `jr_inventory`'s `banked` set was filtered by an + **EPHEMERAL, gitignored `.run/banked_func_*.json` roster** — `rm -rf .run`/a fresh clone would blind ALL + banked jr at once, cross-address siblings (roster named after the exemplar) were structurally invisible, and + non-leader banked jr were missed. **FIX (audit's exact prescription): deleted the roster glob + `cand` + filter; `banked` is now DERIVED FROM THE IMAGE** — a real-C def/define fn is a banked jr iff + `family_remap.reloc_targets` shows it references a committed `.rodata` carve offset (config + image, both + durable). **R32 assertion added:** every committed carve must resolve to EXACTLY ONE owner or the run + aborts (a stranded/duplicated carve = the §8b func_801734BC incident, never silent). **Also fixed** the + adjacent MEDIUM/LOW finding — the `asm_jr` scan's `func_`-fullmatch dropped the curated-name `listCdBuffer` + jr; now resolved via `oss.addr_of()` (the `--only` path's own fullmatch is left — it parses user input, not + the corpus). **Perf:** the image is read ONCE and passed to `reloc_targets(…, data=)` (new + backward-compatible param on family_remap; regression-verified 0/80 mismatch vs the re-read path). **VERIFIED:** + (1) reloc_targets `data`-param behavior-identical; (2) the R33 win — ov_SC02_000 now finds the cross-address + sibling `func_8017FCB0` the roster missed; the 3 non-leader overlays (ov_SC01_077/04_008/05_009) resolve; + (3) **full-fleet parallel run = 134/134 OK, 0 false aborts, 1336 banked jr == 1336 carves → 1:1 ownership + holds fleet-wide** (the R32 assertion is safe). **BYTE-SAFE:** jr_isolate_all is NOT in the make build/extract + path (R22-neutral); the change makes future isolations strictly MORE correct (finds carve owners the roster + missed → fewer stranded carves). tooling-audit ledger marked FIXED (3 jr_isolate_all findings). **NEXT: A10 — + re-test the walls (the audit payoff).** - **2026-07-14 (session 13, A9f — overlay_src_split force_decl latch fixed + a coverage oracle; Max):** The parser `jr_isolate_all` rewrites source from swallowed **2 real function definitions** on physical lines of the form `extern A; extern B; void f(){...}`: `scan_construct`'s `force_decl` latched from the diff --git a/tools/family_remap.py b/tools/family_remap.py index eb6af3af8..5753e3435 100644 --- a/tools/family_remap.py +++ b/tools/family_remap.py @@ -43,11 +43,17 @@ def nins_of(ov, addr): return None -def reloc_targets(ov, addr): +def reloc_targets(ov, addr, data=None): """ordered [(kind, resolved_addr)] for jal targets + lui/lo address loads, in instruction order. Verified against splat .s ground truth (22/22 on func_80141100; 15/15 on func_801407F4 whose - indexed-global D[i] accesses the pre-Phase-26 tracker missed — see the add/addu hi-propagation).""" - data = open(img_path(ov), "rb").read() + indexed-global D[i] accesses the pre-Phase-26 tracker missed — see the add/addu hi-propagation). + + `data` (optional) = the overlay image bytes, read once by the caller and passed to AVOID the + per-call `open(...).read()` when scanning many functions of one overlay (jr_inventory reads it + once, then calls this ~2,300× — 6s -> 0.1s). Omit it and the image is re-read fresh per call + (the default, so a re-extraction mid-process is always seen).""" + if data is None: + data = open(img_path(ov), "rb").read() n = nins_of(ov, addr) off = addr - VRAM out, pend = [], {} diff --git a/tools/jr_isolate_all.py b/tools/jr_isolate_all.py index 62f256d56..4706929e7 100644 --- a/tools/jr_isolate_all.py +++ b/tools/jr_isolate_all.py @@ -70,31 +70,70 @@ def rodata_carves(cfg_lines): def jr_inventory(ov): - """Return (all_jr:{vram:src_kind}, banked:{vram:func_name}). src_kind in - {'asm','banked'}. jr = INCLUDE_ASM funcs whose .s references a jtbl_ + the - already-banked jr (real-C `def`/`define` items whose name is recorded in a - .run/banked_func_*.json — the exemplar is NOT in its own sibling list, so we - confirm presence by parsing the source, not by the sibling roster).""" + """Return (all_jr:{vram:src_kind}, banked:{vram:func_name}). src_kind in {'asm','banked'}. + + jr = still-unmatched switch functions (INCLUDE_ASM `.s` referencing a jtbl_) + the + already-banked jr (whose jtbl became a committed `.rodata` carve). + + `banked` is DERIVED FROM THE IMAGE — never from a roster (R33). The old code filtered + real-C defs by an EPHEMERAL, gitignored `.run/banked_func_*.json` set: a `rm -rf .run` + / fresh clone made all banked jr invisible at once, and a cross-address sibling (whose + roster file is named after the exemplar) was structurally missing. Two proven invariants + answer it instead: (1) the committed splat config lists every `.rodata` carve; (2) a + real-C function OWNS a carve iff it references that carve's address — `family_remap. + reloc_targets` reads the extracted image and says so. So a real-C def/define fn is a + banked jr iff it references a committed carve offset. Cross-address- and + curated-name-immune, and it finds NON-LEADER banked jr (carve in the object's own + subseg, not a `_jr_` leader) that a subseg-name model would miss. Every carve MUST + resolve to exactly one owner or the run aborts (R32) — a stranded/duplicated carve is + the func_801734BC incident (§8b) and must never be silent. + + Cost: ~6s -> ~0.1s by reading the overlay image ONCE and passing it to reloc_targets.""" + import family_remap + base = oss_vram(ov) + syms = oss.load_ov_syms(ov) + + # still-unmatched jr: an INCLUDE_ASM fn whose .s references a jtbl_. Resolve the .s + # basename through the symbol table (addr_of) so a CURATED name (e.g. listCdBuffer) is + # not dropped by a func_-shape fullmatch (§26-A LOW finding). asm_jr = {} for p in glob.glob(os.path.join(REPO, f"asm/{ov}/nonmatchings/*/*.s")): if re.search(r'jtbl_[0-9A-Fa-f]{8}', open(p).read()): - fn = os.path.basename(p)[:-2] - if re.fullmatch(r'func_[0-9A-Fa-f]{8}', fn): - asm_jr[int(fn[5:], 16)] = fn - # candidate banked-jr names (global roster) -> confirm each is a real-C def here - cand = set() - for bj in glob.glob(os.path.join(REPO, ".run/banked_func_*.json")): - cand.add(os.path.basename(bj)[len("banked_"):-len(".json")]) - realc = {} # addr -> name for def/define items - syms = oss.load_ov_syms(ov) + nm = os.path.basename(p)[:-2] + a = oss.addr_of(nm, syms) + if a is not None: + asm_jr[a] = nm + + # already-banked jr: every real-C def/define fn that references a committed carve + # offset in the IMAGE (read once, passed to reloc_targets). + cfg_lines = open(os.path.join(REPO, f"config/splat.{ov}.yaml")).read().splitlines() + carve_offs = {off for _li, off, _sub in rodata_carves(cfg_lines)} + img = open(family_remap.img_path(ov), "rb").read() + banked, owners = {}, {} # owners: carve_off -> [names] for cf in glob.glob(os.path.join(REPO, f"src/{ov}/*.c")): _, items = oss.parse_overlay_c(open(cf).read(), syms) for addr, name, kind, _ in items: - if kind in ("def", "define") and name and addr is not None: - realc[addr] = name - banked = {a: nm for a, nm in realc.items() if nm in cand} + if kind not in ("def", "define") or not name or addr is None: + continue + try: + targets = family_remap.reloc_targets(ov, addr, data=img) + except Exception: + continue + hits = {t - base for k, t in targets if k == "data" and (t - base) in carve_offs} + if hits: + banked[addr] = name + for off in hits: + owners.setdefault(off, []).append(name) + + # R32: every committed carve resolves to EXACTLY ONE banked owner, or abort loud. + problems = [("UNOWNED", hex(base + o)) for o in sorted(carve_offs - set(owners))] + problems += [("MULTI", hex(base + o), owners[o]) for o in sorted(owners) if len(owners[o]) > 1] + if problems: + sys.exit(f"jr_inventory({ov}): committed .rodata carve ownership is not 1:1 (R32/R33) — " + f"a stranded/duplicated carve (§8b func_801734BC class): {problems}") + alljr = dict(asm_jr) - alljr.update({a: "banked" for a in banked}) # marker; name in `banked` + alljr.update({a: "banked" for a in banked}) # marker; name in `banked` return alljr, banked