The audit's evidence (6 agents + 6 skeptics, 1.2M tokens, 40 findings with file:line proof and measured candidate/parsed/skip counts) existed ONLY in a workflow journal OUTSIDE the repo. A fresh session would have inherited my SUMMARY of the audit, not the audit — exactly the R30 failure mode (capture context-dependent artifacts DURING the session that produced them). Drew caught it. Now committed as the plannable input to the audit phase, with: - the method (measure found-vs-candidates against an OVER-approximating detector; never "review the regex" — that is the failure mode that wrote these bugs); - why it gates the matching work (the byte-gate is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle: green since Phase 5 at 0% decompiled, so it is compatible with ANY decomp %); - THE QUESTION IT ANSWERS: how many walls we have already "byte-proven" across 26 phases were lookup misses wearing a wall's clothes? (the def-side loose-typing wall, the 159 arity conflicts, the type-heavy tail were ALL diagnosed on top of the 10% callee-oracle hole); - R32 (coverage assertion) + R33 (derive, don't re-derive — apply FIRST: the best outcome is a DELETED scanner, not a fixed regex); - the priority order (dedup_integrate FIRST — a fail-closed validator that can print a FALSE GREEN); - the 7 bugs already fixed (do not redo) and the 63 tools not yet audited, with the filter for which matter.
138 KiB
Tooling-Integrity Audit — measured findings (Phase 26 session 8, 2026-07-14)
STATUS: DIAGNOSIS ONLY. Almost nothing here is fixed. This is the input to the audit phase Drew gated ahead of all further matching work ("we should do T14 now, before the rest of the work").
Method (do not audit by reading regexes — that is the failure mode that WROTE these bugs). For each scanner: build a deliberately OVER-APPROXIMATING candidate detector, run both over the real corpus, compute
gap = candidates − parsed, and classify every item in the gap as a real silent skip or a justified exclusion. 6 auditor agents, each followed by an independent SKEPTIC told to refute (their corrections are recorded per group — several magnitudes were wrong in BOTH directions, and one whole class was refuted).Coverage of this audit: 19 of 82 tools (23%), chosen by risk. NOT comprehensive. See "Not yet audited".
Why this gates the matching work
The whole-binary byte-gate is a perfect correctness oracle and a null coverage oracle. It has never once
accepted a wrong match — and it is blind by construction to work never attempted (it has been green since
Phase 5, when 0% was decompiled, because INCLUDE_ASM pastes the ORIGINAL assembly: a green byte-gate is
compatible with any decomp %). Every silent skip is therefore invisible to the one instrument we trust
absolutely.
The cost is not wrong answers. It is invisible work, and walls that aren't there. One 10% hole in the
callee-signature oracle (SIG_IN_BODY_RE — a \s that cannot match a line-continuation backslash) made nine
byte-exact functions look like an intrinsic compiler wall. We would have written them up as such.
THE QUESTION THIS AUDIT ACTUALLY ANSWERS: how many of the walls we have already "byte-proven" across 26 phases were lookup misses wearing a wall's clothes? The def-side loose-typing wall (§20/§41, "triple- confirmed" in Phase 23), the 159 arity/narrow-param conflicts (Phase-15 "documented dead-end"), the 3,098 type-heavy tail, and the 9 zero-bank type-using families were all diagnosed on top of that hole. Phase 16 byte-proved that genuinely contradictory typings DO exist, so the wall is real in part — but "some of it was our tooling" is now the prior, not the long shot. Re-test the cheap ones.
The two rules this produced (P10 candidates — Drew ratifies at PhaseEnd)
- R32 — Coverage assertion. A tool that scans the corpus must assert its own coverage (found vs. an over-approximating candidate set) and fail loud on unparsed input. A silent skip is a DEFECT, not a no-op.
- R33 — Derive, don't re-derive. Where a proven invariant answers the question, derive the answer from it
rather than re-parsing the source. Apply R33 to each tool BEFORE R32: the best audit outcome is not a fixed
regex — it is a DELETED SCANNER.
harvest_verifyis the model (it derives frommake build+ SHA1, so a parse hole makes it conservative, not wrong).progress.pyis the counter-example:weighted_metrics()derives from the invariant and was correct;classify()re-parsed C and was not.
Priority order for the audit phase
dedup_integrate.py— UNAUDITED, and the most dangerous thing in the toolchain. It is the FAIL-CLOSED byte-honesty validator; a silent skip there prints "1813 validated, 0 failed" — a false green from a gate. Nothing downstream can catch it.jtbl_family_bank.py— UNAUDITED. Three bugs found in it BY HAND this session.- The SELECTION tools —
family_hseq.py,wave_targets.py,exemplar_miner.py,worklist.py,build_fuel_manifest.py. A hole here makes work invisible to planning — the worst kind, because you never know to look. masked_diff.py/match_one.py— the closeness oracle every crack agent trusts.- Then the findings below (the 23% already measured), highest severity first.
Already fixed this session (do not redo)
| tool | bug | commit |
|---|---|---|
gen_harvest_targets.SIG_IN_BODY_RE |
)\s*{ missed own-line braces → 186 of 1801 (10%) of the callee oracle |
commit:0561 |
family_remap.extract_unit |
read an m2c DECLARATION as a DEFINITION → 15 of 35 exemplars phantom | commit:0552 |
scope_data_externs (NEW, §8d) |
carried data externs at FILE scope established a global the TU never had | commit:0551 |
scope_data_externs._body_open_brace |
own-line-brace only → silently no-op'd on every ANSI draft | commit:0555 |
progress.py classify() |
K&R defs invisible (; before {); phantom dedup members; + coverage assertion |
commit:0574 |
jtbl_family_bank.revert() / jr_isolate_all |
config residue → duplicate subseg → "segments out of order"; + fail-loud validation | commit:0558 |
tools/reconcile_tu.py (NEW) |
TU-visible decl oracle + fn-ptr parsing + coverage assertion — WRITTEN, VALIDATED, NOT WIRED IN | commit:0580 |
Not yet audited (63 of 82 tools)
The filter is "does it PARSE something, and does it GATE or SELECT work?" — not all 82 (many are dead
LLM-tier scripts: api_draft, export_pairs, format_finetune, eval_lora, ab_score, glm_reconcile, …).
The ~15 that matter: dedup_integrate · jtbl_family_bank · bank_exemplar · gate_stage · harvest_verify
· match_one · masked_diff · family_hseq · family_manifest · wave_targets · exemplar_miner ·
worklist · build_fuel_manifest · dup_report · difficulty · sig_image · canon_draft_decls ·
derive_canonical_sigs · census_conflict_callees.
THE 40 MEASURED FINDINGS
(candidates = the over-approximating detector's count; parsed = what the tool actually extracts;
real skips = items in the gap the tool SHOULD have parsed. Skeptic corrections are recorded per group —
read them: several auditor magnitudes were wrong, and one severity was correctly downgraded to "latent".)
GROUP: data-decls (tools/reconcile_decls.py, tools/canon_sig_reconcile.py)
Headline: The filed "fn-ptr-extern gap" is far bigger than filed: reconcile_decls' oracle is blind to 27 of engine_core.h's 497 data symbols and to 6,384 of the fleet's 39,440 — and worse than silently skipping, it returns an actively WRONG canonical decl for 3,717 symbols; one blind sized-array decl (engine_core.h:2114) is single-handedly blocking func_801387B8, which is still INCLUDE_ASM in 134 TUs.
Scanners measured: 14 Verified CLEAN: Four scanners measured FULL coverage and should NOT be spent on:
-
canon_sig_reconcile._file_scope_statements(the cpp + brace-depth-0 scanner) — CLEAN. Emitted 949/949 file-scope statements for src/ov_SC01_077/ov_SC01_077.c and 497/497 for src/ov_SC01_000/ov_SC01_000.c with no truncation, no swallowed definitions, and correct handling of the def-vs-initializer brace ambiguity (the v2.1 fix holds). Every hole I found at this layer was in the CLASSIFIER regexes above it (tu_ambient/visible_above), not in the scanner — the fn-ptr statements ARE produced correctly and then dropped on the floor. Do not touch this function. -
canon_sig_reconciledef-locator (there.finditer(r'... \bfn\s*\([^;{]*\)\s*\n?\{')in reconcile()) — CLEAN, and crucially it FAILS LOUD. 10,603 drafts contain a definition of their own fn; 10,600 located (99.97%). The 3 misses (.run/drafts-*-uni/func_80161208.c, all containing the syntactically garbageif ( func_80161208() == 0 a0)) are malformed drafts, and the toolraise ValueError(f'no definition of {fn} found in draft')rather than silently skipping. This is the one scanner in the group that already behaves the way the whole class should — it is the model for the assertions proposed above. -
canon_sig_reconcile.parse_sig / split_params / parse_param— CLEAN over the corpus. split_params' depth-tracking correctly handles nested parens (fn-ptr params) and brackets; parse_sig raised on 0 of the 10,600 located definitions. -
reconcile_decls.data_access_subsaccess-rewrite regex(&?)\bD_x\b(\s*\[)?— CLEAN for the three shapes it is reachable for (scalar, array, ptr). Probed directly:v = D_1[i];->v = ((u8 *)D_1)[i];,D_2 = 3;->(*(s32 *)&D_2) = 3;, and the single-pass design does correctly prevent double-wrapping. CAVEAT, not a finding: it would mangle a fn-ptr call-through-array (D_1[i]()->((u8 *)D_1)[i]()), but that path is unreachable today because DATA_DECL_LINE_RE never parses a fn-ptr decl in the first place. Fixing DATA_DECL_LINE_RE (finding 3) WILL make it reachable — so the fnptr kind must be added to data_access_subs in the SAME change, or fixing the parser will introduce a new bug here.
[CRITICAL] tools/gen_harvest_targets.py :: DATA_DECL_RE (as consumed by reconcile_decls.canonical_data_map — BOTH the engine_core.h authoritative tier via collect_data_decls AND the fleet plurality tier)
- candidates 39440 / parsed 33056 / real skips 6384
- evidence: The regex
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;has no(and no[N]in its character class, so three whole decl shapes are invisible.
AUTHORITATIVE TIER (src/shared/engine_core.h): 497 candidate D_ names, collect_data_decls sees 470, BLIND to 27 — 21 fnptr[], 4 fnptr, 2 sized array:
src/shared/engine_core.h:2114 extern s32 D_80127530[4]; \ <- sized array, invisible
src/shared/engine_core.h:2554 extern void (*D_80127088)(void); \ <- fn-ptr, invisible
src/shared/engine_core.h:3517 extern void (*D_8011DB28)(s32 a0);
src/shared/engine_core.h:21319 extern s32 (*D_801274D0)(s32);
full blind set (27): D_8011DB10 D_8011DB28 D_80127088 D_801274D0 D_80127530 D_80127540 D_8018E208 D_8018E858 D_8018E8A0 D_8018EA0C D_8018EAD8 D_8018ED5C D_8018F2FC D_8018F31C D_8018F804 D_8018F824 D_8018FE80 D_801903DC D_80190CCC D_80190D1C D_8019155C D_801918E4 D_80191EE4 D_801B8A3C D_801E04F0 D_801E051C D_801E0554
FLEET TIER (src/ov_/.c + src/resident/resident.c): 287,801 candidate decl lines, 231,405 parsed, GAP 56,533 (19.6%). Shapes: 54,918 FNPTR, 1,077 SIZED-ARRAY, 404 MULTI-NAME, 134 ASM-ALIAS. 6,384 distinct D_ names get ZERO canonical decl. src/ov_SC01_000/ov_SC01_000.c:30 extern void (*D_8017EEA0[])(void); src/ov_SC01_000/ov_SC01_000.c:322 extern M2C_UNK (*D_8017EBDC)(); src/ov_SC01_000/ov_SC01_000.c:2632 extern s32 D_801A445C, D_801A4460, D_801A4464, D_801A4468; src/ov_SC01_000/ov_SC01_000_jr_8015AE2C.c:3239 extern unsigned char D_801A3E54[8];
END-TO-END, the x134 loss:
engine_core.h:2114 declares extern s32 D_80127530[4];
.run/drafts-wave-cn/func_801387B8.c:4 declares extern s32 D_80127530[4]; (and .run/drafts-clean/func_801387B8.c: extern M2C_UNK D_80127530;)
canonical_data_map()['D_80127530'] -> None (invisible in BOTH tiers)
-> transform()'s if name not in canon: continue leaves the draft AS-IS -> conflicting types vs engine_core.h -> never banks.
grep -rl 'INCLUDE_ASM("…", func_801387B8)' src/ | wc -l = 134
- blast radius: LOST MATCHES + corrupt BUILD. 6 distinct draft functions are hard-blocked by the 27-symbol engine_core.h blind set (65 draft x symbol shape-conflicts the tool provably cannot repair, vs 1,034 it can). The live one is func_801387B8: still INCLUDE_ASM in 134 TUs => a x134 core bank is blocked by a single unparsed
[4]. The other 5 (func_80144B9C, func_80145CEC, func_80153800, func_801726D0, func_8013E448, func_801536DC) were routed around by hand and are now banked, which is exactly why this read as a small lever. Forward-looking: 6,384 fleet symbols have no canon at all, so ANY future draft touching one is unreconcilable. - fix: In tools/gen_harvest_targets.py, replace DATA_DECL_RE with a shape-aware alternation instead of a single character-class:
DATA_DECL_RE = re.compile(
r'extern\s+('
r'[A-Za-z_][\w\s*]?(\s*\sD_[0-9A-Fa-f]+\s(?:^])?\s)\s*([^;])' # fn-ptr and fn-ptr array
r'|[A-Za-z_][\w\s*]?\bD_[0-9A-Fa-f]+\s*(?:^])?' # scalar/ptr/array, ANY [N]
r')\s;')
and loosen the array bracket from
\[\s*\]to\[[^\]]*\]so sized arrays parse. Then teach collect_data_decls to emit one entry per D_ name on MULTI-NAME lines (split m.group(1) on top-level commas and re-synthesizeextern <base> <name><arr>;per name) — today aextern s32 D_a, D_b;line contributes nothing at all. reconcile_decls.parse_data_decl / DATA_DECL_LINE_RE must gain a matchingfnptrkind so _norm_type can compare array-vs-fnptr-vs-scalar (a fnptr canon must never be 'repaired' into a scalar). - assertion to add: In collect_data_decls: run the permissive detector
^[ \t]*extern\b[^;]*\bD_[0-9A-Fa-f]+[^;]*;(comment-stripped) over the same paths; assert the set of D_ names it yields == the set collect_data_decls returns. On mismatch, fail loud with the unparsed lines:raise SystemExit(f'DATA_DECL_RE blind to {len(gap)} decls: {gap[:10]}'). This single assertion would have caught all 27 engine_core.h holes on day one.
[CRITICAL] tools/reconcile_decls.py :: canonical_data_map (the fleet-MAJORITY oracle design)
- candidates 249615 / parsed 195438 / real skips 54177
- evidence: This is not a regex hole — the oracle is asking the WRONG QUESTION, exactly as the decision-log suspected. C's constraint is PER-TRANSLATION-UNIT: each overlay .c is its own TU and may legally declare the same address differently. A fleet-wide plurality therefore cannot be right for every TU, by construction.
Measured with a permissive ground-truth extractor over src/ov_/.c + resident.c: distinct D_ symbols declared in the fleet : 39,447 symbols with >=2 CONFLICTING decl spellings : 14,521 (36.8%) <- one fleet-wide answer is wrong for SOME TU (TU, symbol) decl pairs : 249,615 oracle has NO canon (silent skip) : 19,768 (7.9%) oracle DISAGREES with the TU's own file-scope decl : 54,177 (21.7%) TUs with >=1 broken oracle answer : 678 / 678 (every single TU)
Worse than 'missing': for 3,717 symbols the oracle returns an actively WRONG canon, because the majority real-world form is invisible to DATA_DECL_RE so the vote is won by a MINORITY parseable spelling from an unrelated overlay:
src/ov_SC01_000/ov_SC01_000.c:323 TU declares extern s16 (*D_8017EBE0)();
oracle says extern u8 D_8017EBE0[]; (invisible=2, visible=1)
src/ov_SC01_000/ov_SC01_000.c TU declares extern void (*D_8017EF98[])(void);
oracle says extern char *D_8017EF98;
D_8011DB10: engine_core.h says extern void (*D_8011DB10)(s32); but the oracle, blind to it, falls through to the fleet vote and returns extern s32 D_8011DB10; — it CONFIRMS a decl that conflicts with the header the TU includes.
(full list: .run/audit/data-decls/wrong_canon.txt)
- blast radius: corrupt BUILD (and silently caps the recovery pipeline). reconcile_decls WRITES canon[name] into the draft. Where canon disagrees with the target TU, the tool converts a draft that would have compiled into one that cannot — a self-inflicted
conflicting types. 21.7% of all (TU,symbol) pairs are exposed; every one of the 678 TUs has at least one. This is the mechanism behind the 780 h_seq rejections (task #12) and it silently bounds the whole family-remap endgame, because the byte-gate cannot distinguish 'reconcile poisoned the decl' from 'the body doesn't match'. - fix: Make the oracle TU-scoped, not fleet-scoped. canonical_data_map(extra_src) should become canonical_data_map(tu_path): resolve each D_ name against what THAT TU actually sees, in this precedence — (1) the TU's own file-scope decl (obtained the way canon_sig_reconcile already does it: cpp the pristine TU and brace-depth-0-scan, which correctly expands the DEFINE_ macros, so engine_core.h decls arrive automatically and the macro-body problem disappears); (2) only if the TU declares it nowhere, fall back to the fleet plurality as a guess. Concretely: reuse canon_sig_reconcile.tu_ambient(tu_path)['data'] as the primary source and demote canonical_data_map to the fallback. reconcile_decls' --src-file flag is already the hook — promote it from 'also votes' to 'is authoritative'. NOTE: tu_ambient must first be fixed per the fnptr finding below, or it will hand back the same blind spot.
- assertion to add: After building canon for a TU, assert every symbol the draft declares that the TU ALSO declares resolves to the TU's spelling:
for n in draft_syms & tu_syms: assert _norm(canon[n]) == _norm(tu_decl[n]), f'{n}: oracle {canon[n]!r} conflicts with TU {tu_decl[n]!r}'. Additionally emit a one-line stat per run —canon: N resolved from TU, M guessed from fleet, K UNRESOLVED— and make K>0 a hard warning, so a symbol with no canon can never again be a silentcontinue.
[HIGH] tools/reconcile_decls.py :: DATA_DECL_LINE_RE (draft-side, drives transform + parse_data_decl)
- candidates 2500 / parsed 2385 / real skips 115
- evidence: Same character-class disease as the oracle regex, measured over the live draft corpora (.run/drafts-recov, drafts-wave-cn, drafts-giants, drafts-t5, drafts-full): 2,500 candidate
extern ... D_x ...;lines, 2,385 parsed, GAP 115 (4.6%) — 84 FNPTR, 25 MULTI-NAME, 4 SIZED-ARRAY, 2 other. .run/drafts-recov/func_8013E448.c:3 extern s32 (*D_801274D0)(s32); .run/drafts-recov/func_801536DC.c:5 extern void (*D_8011DB28)(s32); .run/drafts-wave-cn/func_80144B9C.c:29 extern s32 D_800A5E88, D_800A5E8C, D_800A5E90; .run/drafts-wave-cn/func_801387B8.c:4 extern s32 D_80127530[4]; .run/drafts-wave-cn/func_801502EC.c:17 extern Ent D_801202A0[96]; Note the skipped names are LITERALLY the same symbols the oracle is blind to (D_801274D0, D_8011DB28, D_80127530) — the two holes compound: the draft's decl is unparseable AND its canon is missing, so nothing can rescue it. MULTI-NAME is the nastiest sub-shape: DATA_DECL_LINE_RE matches nothing onextern s32 D_a, D_b, D_c;(the comma is outside[\w \t]), so ALL THREE symbols are skipped from a single line — and canon_sig_reconcile's _DATA_EXTERN_RE explicitly handles this shape (v2 note #9), so the two tools in the same pipeline disagree about what a data decl is. - blast radius: LOST MATCHES. 115 draft decl lines silently pass through untransformed; the draft keeps a decl that may conflict with the TU, and — critically — the ACCESS SITES are never cast either (transform only builds
subsfor symbols whose decl line matched), so even a correct decl rewrite by another tool would then byte-drift. Directly implicated in the 6 hard-blocked functions above. - fix: Rewrite DATA_DECL_LINE_RE as an alternation over the four real shapes and return a 4-valued kind:
DATA_DECL_LINE_RE = re.compile(
r'^([ \t])extern\s+(?:'
r'(?P[A-Za-z_][\w \t*]?)(\s**\s*(?PD_[0-9A-Fa-f]+)\s*(?P^])?\s)\s*((?P[^;]))'
r'|(?P[A-Za-z_][\w \t]?)\s*(?P*?)\s*\b(?PD_[0-9A-Fa-f]+)\b\s*(?P^])?'
r')\s;[ \t](?:/*[^\n]*/)?[ \t]*$')
and add a separate MULTI-NAME pre-pass that splits
extern T a, b, c;into one synthetic single-name decl per symbol before the main scan (mirroring canon_sig_reconcile._DATA_EXTERN_RE's piece-splitting, so the two tools agree). parse_data_decl must return kind in {sca,ptr,arr,fnptr} and _norm_type must treat fnptr as its own kind so a fnptr is never 'reconciled' into a scalar. - assertion to add: In transform(): count candidate lines with the permissive
^[ \t]*extern\b[^;]*\bD_[0-9A-Fa-f]+[^;]*;detector and compare to the number DATA_DECL_LINE_RE matched. Assert equal; on mismatch print each unparsed line and exit non-zero (SystemExit(f'{path}: {n} data externs UNPARSED: {lines}')). Today the tool's only signal is theUNPARSEABLE decl, left as-isnote — which is emitted ONLY when parse_data_decl fails on a line DATA_DECL_LINE_RE already matched, i.e. it is structurally incapable of reporting the lines that never matched at all. That is the whole silent-skip class in one line of code.
[HIGH] tools/canon_sig_reconcile.py :: _reconcile_data — the if not base or '(' in m.group(0): return m.group(0) guard
- candidates 14332 / parsed 13809 / real skips 523
- evidence: TWO separate holes, both measured over every .run/drafts* corpus (14,332 candidate data-extern lines, 13,809 fully processed, GAP 523 = 3.6%):
(F1) 436 lines: _DATA_EXTERN_RE does not match fn-ptr data externs AT ALL (its type group is [^;\n()]*? — parens excluded), so the entire data-reconcile contract (strip-if-identical / access-cast-if-different / block-scope-if-invisible) is skipped for 100% of fn-ptr data externs:
.run/drafts/func_800CEDFC.c:12 extern void (*D_800D3430[])(void);
.run/drafts/func_800CEE40.c:4 extern void (*D_800D3480[])(void);
.run/drafts/func_800CEFD0.c:15 extern void (*D_800D3488[])(void);
(F2) 87 lines — the one that is pure accident: the line MATCHES, then the '(' in m.group(0) guard bails because there is a parenthesis IN A TRAILING COMMENT. m.group(0) spans the comment, which _DATA_EXTERN_RE deliberately allows. An ordinary scalar extern is silently abandoned because a human wrote '(' in prose:
.run/drafts/func_800D27DC.c:64 extern s32 D_800AE7BC; /* ordering-table base (array of OT-entry pointers, stride 20) /
.run/drafts-giants/func_8015126C.c:20 extern s32 D_800AE6A8; / +0x8 == D_800AE6B0 (s32) /
.run/drafts-giants/func_8015126C.c:21 extern u8 D_80078E78[]; / +0x49 == D_80078EC1 (u8) /
PROVED by calling the real function (.run/audit/data-decls/a7_proof.py):
ambient = {'D_800AE7BC': ('u8', is_array=True)}, visible = {'D_800AE7BC'}
draft 'extern s32 D_800AE7BC;\nvoid f(void){ g(D_800AE7BC); }'
-> 'void f(void){ g(((s32*)D_800AE7BC)); }' (extern stripped, use access-cast: CORRECT)
same draft + ' /* ordering-table base (array of pointers) */'
-> UNTOUCHED, extern kept verbatim (-> conflicting types vs the u8[] ambient)
- blast radius: LOST MATCHES + corrupt BUILD. 523 draft data externs are silently passed through unreconciled in the live recovery pipeline. F2's 87 are the cruellest: identical code banks or fails depending on whether a drafter's comment happened to contain a parenthesis, which is invisible to every downstream signal — the byte-gate just reports 'did not compile'.
- fix: (F2) Compute the guard on the COMMENT-STRIPPED decl text, not on the raw match: change
if not base or '(' in m.group(0): return m.group(0)to strip the trailing comment first — the regex already isolates the payload, so test the fields instead of the whole match: decl_txt = m.group(0).split('/*')[0] if not base or '(' in decl_txt: return m.group(0) (F1) Add a fn-ptr branch. Give _DATA_EXTERN_RE an alternation forextern <ret> (*NAME[opt])(args);and give _reconcile_data a 'fnptr' kind so it can do the same three-way decision (identical -> strip; different -> access-cast; not visible -> block-scope move). A fn-ptr extern must NEVER be silently emitted verbatim into a TU that declares the symbol differently. - assertion to add: In _reconcile_data, before returning: re-scan the ORIGINAL draft with the permissive
^[ \t]*extern\b[^;]*\bD_[0-9A-Fa-f]+[^;]*;detector and assert every candidate line was either consumed (stripped/moved/cast-registered) or explicitly recorded in askipped=[]list with a REASON. Thenassert not skipped, f'{fn}: {len(skipped)} data externs silently passed through: {skipped}'. The bail-out branch must never be reachable without leaving a trace.
[HIGH] tools/canon_sig_reconcile.py :: tu_ambient + visible_above (the file-scope statement CLASSIFIER)
- candidates 25 / parsed 0 / real skips 25
- evidence: file_scope_statements (the cpp + brace-depth-0 scanner) is CLEAN — it correctly emits every file-scope statement. The bug is one layer up: the two classifier regexes in tu_ambient and visible_above (
^(?:extern\s+)?([A-Za-z_][\w \*]*?)\s*\b([A-Za-z_]\w*(\s*\[[^\]]*\])?...)\s*;$) have no(in their type class, so a file-scope fn-ptr DATA decl falls into NO bucket — not funcs, not data, not typedefs. It is invisible to the ambient map AND absent fromvisible. src/ov_SC01_077/ov_SC01_077.c : 949 file-scope stmts, 569 names classified, 18 stmts naming a D/func_ landed in NO bucket — all fn-ptr: extern s32 (*D_801274D0)(s32); extern void (*D_80187ED0[])(void); extern void (*D_80187F10[])(void); (+15 more) src/ov_SC01_000/ov_SC01_000.c : 497 stmts, 279 classified, 7 unclassified: extern void (*D_8017EEA0[])(void); extern M2C_UNK (*D_8017EBDC)(); extern s16 (*D_8017EBE0)();
The CONSEQUENCE is not a no-op, it is an active miscompile-inducing branch. Because the name is missing from visible, _reconcile_data takes if name not in visible: moved.append(...) and MOVES the draft's (wrong-typed) extern to BLOCK SCOPE inside the function body — while the file-scope fn-ptr decl is still in scope. PROVED against the real cross-compiler (.run/audit/data-decls/blk.c):
typedef int s32;
extern s32 (*D_801274D0)(s32);
void f(void){ extern s32 D_801274D0; g(D_801274D0); }
-> blk.c:3:26: error: conflicting types for 'D_801274D0'; have 's32'
The block-scope idiom, which is the tool's safety valve, is precisely what detonates here.
- blast radius: corrupt BUILD. Every draft that touches any of the ~25 file-scope fn-ptr data symbols per TU gets a guaranteed-uncompilable block-scope redeclaration. Fleet-wide there are 54,918 fn-ptr extern lines across 678 TUs, so this branch is reachable from a large fraction of the recovery corpus. It also means
visible_aboveUNDER-reports the TU's namespace, which biases every downstream 'not visible -> move to block scope' decision the tool makes. - fix: Add a fn-ptr branch to BOTH classifiers, ordered BEFORE the generic data-decl regex (a fn-ptr decl otherwise looks like a func decl to the func regex, or like nothing):
m = re.match(r'^(?:extern\s+)?([A-Za-z_][\w *]?)(\s*\s*([A-Za-z_]\w*)\s*(^])?\s)\s*(([^;]))\s;$', stn)
if m:
data[m.group(2)] = (f'{m.group(1).strip()} (*)({m.group(4)})', m.group(3) is not None)
continue
and in visible_above add the same pattern to
names.add(...). Give tu_ambient's data map a third element (kind='fnptr') so _reconcile_data can compare like with like rather than treating a fn-ptr as a scalar of type ''. - assertion to add: At the end of tu_ambient(), assert the classifier is TOTAL over the statements the scanner produced: for every file-scope statement that declares an identifier (permissive detector: mentions a D_/func_/named symbol and ends in ';' or '{'), assert that identifier landed in exactly one of funcs/data/typedefs.
unbucketed = [st for st in stmts if names_in(st) and not (names_in(st) & classified)]; assert not unbucketed, f'{tu_path}: {len(unbucketed)} file-scope decls in NO bucket: {unbucketed[:5]}'. A classifier that silently drops a declaration is strictly worse than one that crashes.
[MEDIUM] tools/reconcile_decls.py :: _norm_type (the "already compatible -> return []" gate)
- candidates 197 / parsed 171 / real skips 26
- evidence: _norm_type collapses
s32|u32|int|unsigned int|unsigned|long|unsigned long|u_longall to the single token 'int' (line 110), so it declares a draft/canon pair COMPATIBLE across a SIGNEDNESS change and returns[]— no decl rewrite, no cast. gcc does not agree. Verified with the real toolchain: typedef int s32; typedef unsigned int u32; extern s32 D_1; extern u32 D_1; -> error: conflicting types for 'D_1'; have 'u32' {aka 'unsigned int'} Measured over the draft corpora: 197 (draft, canon) pairs where _norm_type says 'compatible' but the spellings differ; 26 of them differ in SIGNEDNESS and are therefore hard gcc errors the tool refuses to repair: .run/drafts-ov077/func_8016F2C8.c draft 'extern u32 D_80126D50;' vs canon 'extern s32 D_80126D50;' .run/drafts-ov077/func_8016F30C.c draft 'extern u32 D_80126D50;' vs canon 'extern s32 D_80126D50;' .run/drafts-ov077/func_8016F350.c draft 'extern u32 D_80126D50;' vs canon 'extern s32 D_80126D50;' .run/drafts-T6-fail/func_801718AC.c draft 'extern u32 D_80126D50;' vs canon 'extern s32 D_80126D50;' The collapse is right about CODEGEN (same width, same load opcode, so the cast really is byte-neutral) and wrong about the C FRONT END (which rejects the redeclaration before codegen is ever reached). The tool skips the repair for exactly the reason the repair is unnecessary — and the compile then fails anyway. - blast radius: LOST MATCHES. 26 draft x symbol pairs currently in the corpus compile-fail with
conflicting typesand the tool, by design, declines to touch them. Note func_801718AC's draft is in a *-fail directory. The failure mode is maximally confusing to a human: the tool reports 0 reconciles, the build errors on a decl the tool looked at and consciously approved. - fix: Split the equivalence into two distinct predicates. Keep _norm_type's int-family collapse ONLY for the question 'is the access cast byte-neutral' (it is). Add a separate, STRICT predicate for the question 'will gcc accept the draft's decl alongside the canonical one' — no signedness collapse, no width collapse; s32!=u32, int!=unsigned. In transform(): rewrite the decl to canonical whenever the STRICT predicate says they conflict, and emit the access cast whenever _norm_type says the types differ in a codegen-relevant way. Today one predicate is doing both jobs and gets one of them wrong. Concretely, change the guard at line 133 from
if _norm_type(...) == _norm_type(...): return []toif _strict_same(idecl, cdecl): return []and keep _norm_type only to decide whethersubsneeds to be non-empty. - assertion to add: Whenever data_access_subs returns [] ('already compatible'), assert the two decl strings are genuinely interchangeable to the C front end:
assert _strict_same(idecl, cdecl), f'{name}: claimed compatible but {idecl!r} vs {cdecl!r} differ in signedness/width — gcc will reject'. Cheaper still, add a one-off CI check that compilesextern <canon>; extern <draft>;for every pair the tool declares compatible, and fails if gcc does.
[MEDIUM] tools/canon_sig_reconcile.py :: _uniquify_draft_types (the fn-ptr / alias typedef arm)
- candidates 36 / parsed 0 / real skips 36
- evidence: The 'simple/alias/fnptr/array typedefs' regex
^[ \t]*typedef\s+[^;{}\n]*?\b([A-Za-z_]\w*)\s*(?:\[[^\]]*\])?\s*;requires an IDENTIFIER immediately before the;(modulo an array suffix). A fn-ptr typedef ends in), so it never matches — despite the docstring explicitly claiming 'simple/alias/fnptr/array typedefs'. Verified by calling the real function: csr._uniquify_draft_types('typedef int (*Fn)(void);', {}, 'func_80000000') -> 'typedef int (*Fn)(void);' (NOT stripped, NOT uniquified — the name Fn is untouched) csr._uniquify_draft_types('typedef struct { int a; } S;', {}, 'func_80000000') -> 'typedef struct { int a; } S_80000000;' (the aggregate arm works correctly) Corpus: 36 fn-ptr typedef lines across the draft dirs, 7 distinct names — ActorFn, DispatchFn, FuncPtr, Handler, VoidFn, code, code_fn: .run/drafts-recover-a-cn-cast/func_80131CA8.c:4 typedef void (*code)(int); .run/drafts-recover-a-cn/func_80131CA8.c:4 typedef void (*code)(int); Because the name is neither stripped (when identical to an ambient type) nor uniquified (when colliding), a fn-ptr typedef is the one type-name class that can still collide — which is exactly theredefinition of ...failure the T7-M1 uniquify pass was built to eliminate. Generic names likecode,Handler,FuncPtrare precisely the ones two independently-drafted exemplars landing in ONE split will both invent. - blast radius: corrupt BUILD, bounded. 36 draft lines / 7 names. Only bites when two drafts sharing a split both define the same fn-ptr typedef name with different signatures, or when one collides with an ambient engine_types.h name. Low frequency today, but it is a latent landmine that scales with the family-remap endgame (more exemplars per split = more collisions), and the docstring asserts a guarantee the code does not provide.
- fix: Add a third re.sub arm to uniquify_draft_types, before the generic alias arm, matching the fn-ptr shape and extracting the name from inside the parens:
draft = re.sub(r'^[ \t]typedef\s+[^;{}\n]?(\s**\s*([A-Za-z]\w*)\s*)\s*([^;])\s;[ \t]*\n?',
strip_or_mark, draft, flags=re.M)
strip_or_mark already does the right thing with the captured name (strip if identical to ambient, else add to the rename set). No other change needed — the trailing
for name in rename: draft = re.sub(...)loop then uniquifies it everywhere. - assertion to add: After _uniquify_draft_types, assert no typedef NAME survives that is neither ambient-identical nor suffixed:
leftover = [n for n in re.findall(r'typedef[^;{}]*?(?:\(\s*\*\s*)?([A-Za-z_]\w*)\)?\s*(?:\([^;]*\))?\s*;', draft) if not n.endswith(suffix) and n not in ambient_typedefs]; assert not leftover, f'{fn}: typedef names neither stripped nor uniquified: {leftover}'. This makes the docstring's guarantee machine-checked.
[LOW] tools/canon_sig_reconcile.py :: _FN_EXTERN_RE (callee externs)
- candidates 22481 / parsed 22470 / real skips 11
- evidence: Very nearly clean — 22,470 of 22,481 candidate callee-extern lines parse (99.95%). The only gap is a WRAPPED PARAMETER LIST: the regex's param group is
([^;\n]*), which forbids a newline, so an extern whose params wrap across lines never matches and is therefore never stripped, never block-scope-moved, and never call-site cast — it is emitted verbatim at TU file scope. .run/drafts-T6b/func_80157158.c:1 extern void func_801571C4(s32 a0, u16 a1, u16 a2, s32 a3, .run/drafts-recov/func_80156670.c:12 extern void func_80157158(s32 a0, u16 a1, u16 a2, s32 a3, .run/drafts-ov077/func_801466B4.c:1 extern void func_801466F0(s32 a0, s32 a1, s32 a2, s32 a3, .run/drafts-wave/func_8015F118.c:11 extern void func_801466F0(s32 a0, s32 a1, s32 a2, s32 a3, All 11 are 7+-arg callees, i.e. the ones a drafter is most likely to line-wrap. If the TU's canonical sig for that callee differs, the verbatim extern is aconflicting types. - blast radius: LOST MATCHES, small. 11 draft lines across ~6 functions (func_801571C4, func_80157158, func_801466F0). Real but rare — it only bites on wide-signature callees. Worth fixing in the same pass since it is a two-character change.
- fix: Change the param group in _FN_EXTERN_RE from
([^;\n]*)to([^;]*)and add re.S is NOT needed (the class already spans lines once \n is allowed); the trailing[ \t]*\nanchor still terminates the match at the real end of the decl: FN_EXTERN_RE = re.compile( r'^[ \t]extern\s+([^;\n]?)\b([A-Za-z]\w*)\s*(([^;]))\s;[ \t](?:/*[^\n]*/)?[ \t]*\n', re.M) (only the third group loses its \n exclusion — the return-type group must keep \n excluded so the match cannot start on a previous line). - assertion to add: In _reconcile_callees: count candidate lines with a permissive
^[ \t]*extern\b[^;]*\w+\s*\(detector (excluding fn-ptr DATA decls) and assert every one was visited.assert visited == candidates, f'{fn}: {candidates-visited} callee externs never parsed'.
[LOW] tools/canon_sig_reconcile.py :: SCALAR_TYPEDEF_RE
- candidates 4774 / parsed 4719 / real skips 55
- evidence: 4,774 candidate scalar-typedef lines in the draft corpus, 4,719 stripped, 55 skipped. The regex is anchored
^[ \t]*typedef\b[^;]*\b(u8|...)[ \t]*;[ \t]*\n— it requires the;to be immediately followed by end-of-line, so MULTIPLE typedefs on one line are all missed (the[^;]*cannot cross the first;, and the\nanchor then fails): .run/drafts-p18-3b/func_8012C2D0_v15a.c:1 typedef unsigned char u8; typedef unsigned short u16; typedef int s32; .run/drafts-p18-3b/func_8012C2D0_v15b.c:1 (same) .run/drafts-p18-3b/_tc.c:1 (same) All 55 are confined to one stale drafter batch (.run/drafts-p18-3b) whose emitter packed typedefs onto one line; current drafters emit one per line. Left unstripped, these areredefinition of typedef 'u8'errors against common.h (a hard error in C89/gcc-2.7.2). - blast radius: LOST MATCHES, contained. 55 lines, all in a single legacy corpus (.run/drafts-p18-3b) that is not on the live path. No current drafter produces this shape. Fix is cheap insurance against a future drafter regressing to one-line typedefs; it is not blocking anything today.
- fix: Drop the end-of-line anchor and let the regex strip each typedef independently:
SCALAR_TYPEDEF_RE = re.compile(
r'[ \t]typedef\b[^;{}\n]\b(u8|u16|u32|u64|s8|s16|s32|s64|f32|f64|'
r'M2C_UNK|M2C_UNK8|M2C_UNK16|M2C_UNK32|M2C_UNK64)[ \t];[ \t](?:\n)?')
(the
{}exclusion prevents it from eating an aggregate typedef body; the newline becomes optional so a mid-line typedef is stripped in place). - assertion to add: After the SCALAR_TYPEDEF_RE.sub pass, assert none survive:
assert not re.search(r'typedef\b[^;{}\n]*\b(u8|u16|u32|s8|s16|s32|f32|M2C_UNK\w*)[ \t]*;', draft), f'{fn}: scalar typedef survived the strip pass'. gcc will reject it anyway; failing here names the cause instead of making a human read a redefinition error.
GROUP: callee-sigs
Headline: The DATA_DECL_RE char class [\w\s\*] cannot contain (, so the canonical-data oracle is 100% blind to function-pointer / jump-table extern void (*D_X[])(void); decls — and for 13 of the 25 that engine_core.h declares AUTHORITATIVELY it doesn't just miss them, it hands back a WRONG scalar canonical (extern s32 D_8018E858;) elected by a fleet plurality vote in which every correctly-shaped decl is invisible — guaranteeing conflicting types against the very macro it was supposed to conform to, on exactly the jtbl symbols the in-flight jtbl_family_bank is trying to bank.
Scanners measured: 10
Verified CLEAN: VERIFIED CLEAN — do not spend time here: (1) gen_harvest_targets :: SIG_IN_BODY_RE / collect_define_sigs — M=1801 #define DEFINE_func_ candidates, N=1801 parsed, GAP=0. The Phase-26 [\s\\]* fix is COMPLETE. I also audited sig QUALITY, not just count: all 1801 round-trip cleanly through split_sig_string→parse_sig, 0 failures, 0 control-keyword leaks (if (/while ( never captured as a signature). (2) cast_call_sites :: DECL_LINE_RE — 26,261 strict-prototype candidates across all 162 real draft dirs in .run/, 26,261 parsed, 0 real skips. Over src/**/*.c: 779,215 candidates, 2 non-matches, and both are DEFINITION lines it correctly declines to rewrite. Genuinely clean — the (?:/\*[^\n]*\*/)? trailing-comment allowance covers every form m2c and the drafters actually emit (no //-comment or multi-line prototype exists in the corpus). (3) cast_call_sites :: parse_sig / split_sig_string / norm_sig (+ sig_unify's split_top_commas / param_type) — 1801/1801 canonical sigs parse without exception, including function-pointer params. (4) gen_harvest_targets :: INCLUDE_ASM_RE — the regex itself is clean (13/13 on the file it is given; the fault in F4 is the CORPUS it is pointed at, not the pattern).
[CRITICAL] tools/gen_harvest_targets.py :: DATA_DECL_RE / collect_data_decls
- candidates 38529 / parsed 33312 / real skips 5219
- evidence: Candidate detector = every line containing
extern+ aD_[0-9A-Fa-f]+token ending in;. Corpus-wide (src/**/*.c + engine_core.h): M=38529 sites, N=33312 parsed, GAP=5219 (13.5%). I classified 100% of the gap — there are NO justified exclusions, all 5219 are real: (a) function-pointer / jump-table arrays, 4614 sites —extern void (*D_8018E858[])(void);src/shared/engine_core.h:24818 ;extern void (*D_8011DB10)(s32);engine_core.h:23629 ;extern void (*D_80187ED0[])(void);src/ov_SC01_077/ov_SC01_077.c:830 ;extern void (*D_800D3430[])(void);src/resident/resident.c:12. (b) sized arrays, 249 sites —extern s32 D_80127530[4];engine_core.h:2114 (the regex's(?:\[\s*\])?only accepts EMPTY brackets). (c) multi-declarator lines, 356 sites —extern short D_800AE710, D_800AE712;src/800.c:1119 (the,is outside[\w\s\*], so the WHOLE line is dropped, not just the 2nd declarator). Root cause: the char class[\w\s\*]betweenexternandD_Xcannot contain(,),,or a non-empty[N]. THE KILLER: this is not merely a missing entry. reconcile_decls.canonical_data_map() is built entirely on this scanner (tools/reconcile_decls.py:67_ght.collect_data_decls([ec])= the AUTHORITATIVE tier; :78_ght.DATA_DECL_RE.finditer= the fleet plurality tier). engine_core.h declares 25 D_ symbols as function pointers inside DEFINE macros; the oracle sees ZERO of them, so the authoritative tier is silently empty and all 25 fall through to the plurality vote. 12 come out MISSING; 13 come out WRONG: D_8018E858 truthextern void (*D_8018E858[])(void);(engine_core.h:24818, dereferenced asD_8018E858[*(u16*)(a0+2)]();at :24820) -> oracle returnsextern s32 D_8018E858;. I verified the vote: 12 overlay files carry the scalar mis-decl (e.g. src/ov_SC03_126/ov_SC03_126_jr_80178D40.c:1245) and 0 carry the correct func-ptr form — the plurality is decided ENTIRELY by wrong-shaped decls because every right-shaped one is invisible to the regex. Also D_801903DC ->extern struct packed_word D_801903DC;, D_8018EAD8 ->extern char D_8018EAD8[];, D_8018F824 ->extern int D_8018F824;. Sized arrays D_80127530/D_80127540 -> oracle None. - blast radius: Corrupts the BUILD and loses MATCHES. reconcile_decls hands a draft
extern s32 D_8018E858;; the same TU also instantiates the engine_core.h DEFINE macro that declares itvoid (*[])(void)-> in-TUconflicting types for D_8018E858-> the draft is discarded as 'won't compile', and the reconcile pass reports nothing to fix (the identical signature of the SIG_IN_BODY_RE bug). Affected: 25 authoritative func-ptr symbols (13 actively mis-canonicalized, 12 absent) + 2 sized arrays + 5219 decl sites fleet-wide. These D_ symbols ARE the per-overlay jump-table dispatch arrays — i.e. precisely the classjtbl_family_bank(the in-flight recovery stage) exists to bank, and a strong candidate for a chunk of the open '780 h_seq rejections' (task #12). Worse than a silent skip: a silently WRONG oracle that looks authoritative. - fix: Rewrite DATA_DECL_RE to admit the fn-ptr and sized-array forms, and pre-split multi-declarator lines. TEXT (do not apply — fleet build in flight):
DATA_DECL_RE = re.compile( r'extern\s+(' r'[A-Za-z_][\w\s*]?(\s*\sD_[0-9A-Fa-f]+\s(?:^])?\s)\s*([^;])' # fn-ptr / jtbl array r'|[A-Za-z_][\w\s*]?\bD_[0-9A-Fa-f]+\s*(?:^])?' # scalar / sized array r')\s;')
and in collect_data_decls, before matching, expand extern T A, B, C; into one synthetic decl per declarator (split on top-level commas). MIRROR THE SAME FIX in tools/reconcile_decls.py:51 DATA_DECL_LINE_RE (see the separate spillover finding) or the repair is only half-done — canonical_data_map reads BOTH.
- assertion to add: In collect_data_decls: build cand = {every
D_[0-9A-Fa-f]+on a line matching^\s*extern\b.*;} and assert set(out) == cand, else raise with the unparsed lines printed. Additionally assert that every D_ symbol declared inside an engine_core.h DEFINE macro appears in the AUTHORITATIVE tier of canonical_data_map — an empty authoritative tier for a symbol that engine_core.h demonstrably declares is a hard failure, never a silent fall-through to the plurality vote.
[CRITICAL] tools/reconcile_decls.py :: DATA_DECL_LINE_RE (SPILLOVER — outside this group, but the F1 fix is void without it)
- candidates 300617 / parsed 243283 / real skips 57334
- evidence: tools/reconcile_decls.py:51-53 carries an INDEPENDENT copy of the same disease:
r'^([ \t]*)extern\s+([A-Za-z_][\w \t]*?)\s*(\*?)\s*\b(D_[0-9A-Fa-f]+)\b\s*(\[\s*\])?\s*;'— the prefix class[\w \t]again has no(, and the bracket group again only accepts EMPTY[]. Candidate = every line matching^\s*extern\b.*D_XXXX.*;: M=300617 decl lines fleet-wide, N=243283 parsed, GAP=57334 (19.1%) — same three classes (fn-ptr/jtbl, sized array, multi-declarator). canonical_data_map reads BOTH scanners (_ght.collect_data_declsfor the authoritative tier, and this one is used by the reconcile/rewrite side), so patching DATA_DECL_RE alone produces an oracle that KNOWS the correct func-ptr canonical but a rewriter that still cannot recognise the decl line it must replace. - blast radius: Corrupts the BUILD. This is the rewrite half of the F1 loop: even with a correct canonical in hand, reconcile_decls cannot locate/replace a draft's
extern void (*D_X[])(void);line, so the reconcile silently no-ops on exactly the jtbl symbols. Gates the same jtbl_family_bank / 780-h_seq-rejection work as F1. - fix: Mirror the F1 fix here:
DATA_DECL_LINE_RE = re.compile( r'^([ \t])extern\s+(?:' r'([A-Za-z_][\w \t*]?)\s*(\s**\s*\b(D_[0-9A-Fa-f]+)\b\s*(^])?\s)\s*(([^;]))' # fn-ptr / jtbl r'|([A-Za-z_][\w \t]?)\s*(*?)\s*\b(D_[0-9A-Fa-f]+)\b\s*(^])?' # scalar / sized array r')\s;[ \t](?:/*[^\n]*/)?[ \t]*$')
and handle multi-declarator lines by expansion. Fix F1 and this in the SAME change — they are one loop.
- assertion to add: assert the set of D_ symbols this regex extracts from a file == the set of D_ symbols on lines matching
^\s*extern\b.*;in that file, else fail loud. Cross-assert against canonical_data_map: every symbol in the canonical map must be RECOGNISABLE by DATA_DECL_LINE_RE, otherwise the reconciler can produce a canonical it can never apply.
[HIGH] tools/gen_harvest_targets.py :: EXTERN_DECL_RE / collect_extern_sigs
- candidates 2723 / parsed 2706 / real skips 24
- evidence: EXTERN_DECL_RE requires the literal keyword
extern. But m2c and several hand-written/reconciled decls emit BARE prototypes — valid C, fully TU-visible, and completely invisible to the canonical-callee oracle. Measured PER-TU (the unit that matters: cast_call_sites takes --src-file) across the 9 ov_SC01_077 TUs = 24 distinct bare-proto decl sites skipped; 18 of those also escape the define/inline rescue in cast_call_sites.canonical_map and are therefore TRULY invisible to the oracle. Evidence:M2C_UNK func_80178D40(s32, s32); /* extern */src/ov_SC01_077/ov_SC01_077_jr_801734BC.c:3508 (the x134 core just banked);void func_8012A418(void); /* extern (Phase-18 reconciled) */src/ov_SC01_077/ov_SC01_077_jr_80178D40.c:3823;void func_8013B83C(s32 a0, s32 a1, s32 a2);andvoid func_8013BD74(void *a0, s32 a1);src/ov_SC01_077/ov_SC01_077_o0.c:22-23;void func_8012A048(void *a0, s32 a1, u8 a2);src/ov_SC01_077/ov_SC01_077_a.c:580;void func_80154C24(s32 a0, s32 *a1, s32 *a2);src/ov_SC01_077/ov_SC01_077_after.c:3057;s32 func_80019198();src/ov_SC01_077/ov_SC01_077.c:46;s32 func_800D02D0(u8);src/ov_SC01_077/ov_SC01_077_after.c:3120. (Corpus-wide single-pass over engine_core.h + all 134 overlays: M=2723 addrs, N=2706, GAP=17 — that number UNDERSTATES the harm because it lets anexternin a DIFFERENT overlay's file rescue an addr that is bare in the TU actually being compiled; the TU-accurate figure is the 24/18 above.) Note ov_SC01_077 is the only overlay with a nonzero lost set — the other 133 are template-instantiated and carry onlyexternforms — so the whole hole sits squarely on the one overlay every draft is written against. - blast radius: Loses MATCHES. cast_call_sites.transform:
if addr not in canon: continue # pure stub callee: draft's decl is the only one -> no conflict, leave it (don't cast). For these 18 the premise is FALSE — the TU does declare them — so a draft that declares the callee with its own byte-matching signature gets NO decl rewrite and NO call-site cast, hits in-TUconflicting types, and is thrown away as 'won't compile'. That is the EXACT §20 failure class cast_call_sites was built to eliminate, silently re-opened for 18 callees. The same collector also feeds gen_harvest_targets (emits status='stub', signature=None -> the drafting agent guesses), sig_unify, canon_draft_decls and census_conflict_callees — 5 consumers, one hole. - fix: Make
externoptional and anchor at line start (the^anchor is what stops it eating call statements onceexternno longer gates it):
EXTERN_DECL_RE = re.compile( r'^[ \t](?:extern[ \t]+)?([A-Za-z_][\w \t*]?\bfunc_[0-9A-Fa-f]+\s*([^;{]))\s;', re.M)
Keep the existing setdefault first-wins precedence; an explicit extern and a bare prototype for the same addr are the same declaration in C.
- assertion to add: In collect_extern_sigs: cand = every line matching
^\s*(?:extern\s+)?[A-Za-z_][\w \t\*]*\bfunc_[0-9A-Fa-f]{8}\s*\([^;{]*\)\s*;(a prototype: a type run, then func_X, then a param list, then;). assert set(cand_addrs) <= set(sigs), else fail loud listing the unparsed file:line — a declaration the TU can see but the oracle cannot is never acceptable.
[MEDIUM] tools/gen_harvest_targets.py :: collect_stubs + main() corpus scoping (main .c only, no split-file awareness)
- candidates 270 / parsed 13 / real skips 257
- evidence: collect_stubs (and every other collector called from main(), and the default asm_dir) is hardcoded to
src/<overlay>/<overlay>.candasm/<overlay>/nonmatchings/<overlay>/. ov_SC01_077 has 270 INCLUDE_ASM stubs spread over 8 .c files — ov_SC01_077.c:13, _a.c:51, _after.c:58, _jr_8015AE2C.c:68, _jr_801734BC.c:25, _jr_80178D40.c:26, _jr_80182268.c:22, _o0.c:7 — and the asm tree mirrors that split (asm/ov_SC01_077/nonmatchings/{ov_SC01_077:13, ov_SC01_077_a:51, ov_SC01_077_after:56, ov_SC01_077_jr_8015AE2C:68, ...}). The tool sees 13 of 270 = 4.8% of the overlay's remaining work and reports success. The--asm-dirflag re-points only the .s scan, NOT the .c corpus, so pointing it at a split subdir still yields an EMPTY manifest — those addrs are filtered out atif addr not in stubs: continue. cast_call_sites already fixed this class at the consumer end (it grew--src-fileprecisely because 'the TU sees THAT file's local decls, NOT the main .c's'); gen_harvest_targets never got the same treatment. - blast radius: Corrupts the METRICS / target selection, not the build. The leverage-ranked target manifest silently enumerates 4.8% of the candidate pool, so 'we've worked the high-leverage tail' is unfounded for 257 functions. Mitigating: the manifest path looks semi-dormant (.run/t6_targets.json last written Jun 21; no workflow or Makefile target invokes the CLI — the live surface of this module is its collect_* functions imported as a library by 5 tools). census_conflict_callees.py imports collect_stubs and inherits the same 13/270 blindness.
- fix: Add
--src-file(mirroring cast_call_sites) or, better, glob the whole TU family:c_paths = sorted(glob.glob(f'src/{ov}/{ov}*.c')), union collect_stubs over all of them, and derive each target's asm subdir from the .c file its INCLUDE_ASM lives in (the INCLUDE_ASM first argument already names it:INCLUDE_ASM("asm/ov_SC01_077/nonmatchings/ov_SC01_077_jr_801734BC", func_80174684)) instead of assumingasm/<ov>/nonmatchings/<ov>. - assertion to add: assert len(stubs) == sum of INCLUDE_ASM occurrences over glob(f'src/{ov}/{ov}*.c'), else fail loud naming the .c files that were never scanned. And in the .s scan: assert os.path.exists(sp) for every target, else fail loud rather than silently emitting data=[] — a missing .s is the tool telling you it is pointed at the wrong subdir.
[MEDIUM] tools/gen_harvest_targets.py :: INLINE_DEF_RE / collect_inline_sigs
- candidates 693 / parsed 690 / real skips 3
- evidence: Candidate detector = brace-balanced scan for any
func_XXXXXXXX(...)whose matching)is followed (after whitespace/newlines) by{, preceded by a type token rather than a keyword/operator. Across the 9 ov_SC01_077 TUs: M=693, N=690, GAP=3 — and all 3 are REAL skips, no false positives. Two shapes defeat the^([A-Za-z_][\w \t\*]*?...column-0 + type-only-prefix anchor: (a) an INDENTED file-scope definition —s32 func_80174650(s32 _arg0)followed by a column-0{, src/ov_SC01_077/ov_SC01_077_jr_801734BC.c:2822. This is a template body replicated fleet-wide: 138 occurrences corpus-wide (src/ov_SC01_000/ov_SC01_000_jr_801734BC.c:2632, ov_SC01_001:2630, ov_SC01_004:2630, ov_SC01_005:2631, ...). (b) a definition preceded on the SAME LINE by extern decls —extern void func_80150170(void *a0); extern s32 func_8014FE60(void *a0); void func_8014FDF4(struct S8014FDF4 *a0) { ... }src/ov_SC01_077/ov_SC01_077_after.c:2151, and the same shape at :3299 for func_80155FF8. The;and(are outside the[\w \t\*]prefix class, so the line matches nowhere. 2 occurrences. - blast radius: Corrupts the sig PRECEDENCE (a latent match-loss). canonical_map ranks inline-def > DEFINE macro > extern, because the in-TU DEFINITION is the authoritative signature. A skipped inline def means a looser/stale
extern(or nothing) silently wins. Today the damage is contained: func_80174650 and func_80155FF8 happen to also carry a matchingextern, so they are rescued; func_8014FDF4 is not — no extern, no DEFINE — so its true signaturevoid func_8014FDF4(struct S8014FDF4 *a0)is simply absent from the oracle, and any draft calling it guesses and conflicts. The 138-site indented form is one symbol replicated, so it is 1 symbol at risk, not 138 — but the shape will recur the moment a drafter indents a definition. - fix: Re-admit a leading-whitespace start and a start-after-a-semicolon, while keeping the exclusion of
if (/=/,(which never sit directly before a type run):
INLINE_DEF_RE = re.compile( r'(?:^|;)[ \t]([A-Za-z_][\w \t*]?\bfunc_[0-9A-Fa-f]+\s*([^;{]))\s{', re.M)
(The (?:^|;) alternation is the whole fix: ^ + optional indent covers shape (a), ; covers shape (b). The Phase-19 bug this anchor was added to prevent — matching if (func_X(...) == 0x2000 && ...) { — stays excluded because if is followed by (, which the [\w \t\*] run cannot cross.)
- assertion to add: In collect_inline_sigs: cand = the brace-balanced scan above (find every
func_X(whose matching)is followed by{and whose preceding non-space token is not one of {if,for,while,switch,return,else,do,sizeof} and does not end in an operator). assert set(cand) == set(sigs), else fail loud printing the unparsed file:line.
[LOW] tools/gen_harvest_targets.py :: ASM_DATA_REF_RE
- candidates 3035 / parsed 2883 / real skips 24
- evidence: Corpus = the 267 per-fn .s under asm/ov_SC01_077/nonmatchings/**. Candidate = every
%hi(...)/%lo(...)operand (the only way a .s names a global). M=3035 operands, of which\b(D_[0-9A-Fa-f]+)\bparses N=2883. GAP=152, classified: 124 =jtbl_XXXXXXXX-> JUSTIFIED EXCLUSION (compiler-generated switch tables; a C draft regenerates its own, there is no C declaration to resolve — the tool is right to ignore them). 14 =func_XXXXXXXX-> REAL SKIP: an address-TAKEN function, not ajal, so it is in neither rec['calls'] (nocalleesentry) nor the D_datalist — the draft must declare it to take its address with ZERO signature guidance from the manifest. Evidence: asm/ov_SC01_077/nonmatchings/ov_SC01_077_jr_801734BC/func_80172C50.s -> %hi(func_80172CA0); .../func_8017F290.s -> %hi(func_8017E1D4) and %hi(func_8017E748); .../func_8017D98C.s -> %hi(func_8017DA08); .../func_80173A60.s -> %hi(func_80173B4C). 10 = renamed non-D_globals -> REAL SKIP:currentLocationIdx6,cdFileLocTablex4 — invisible to ASM_DATA_REF_RE (D_-prefix only) AND to DATA_DECL_RE, so as the project renames symbols away from D_ this hole grows monotonically. - blast radius: Loses MATCHES, small and bounded today. 14 address-taken callees + 2 renamed globals get
decl: None/ no callee entry -> the drafting agent guesses the declaration ->conflicting typesat the whole-binary gate. Note this rides in the semi-dormant main() manifest path (see F4), so it is a latent rather than in-flight loss; the renamed-global class is the one that will grow. - fix: Scan only
%hi/%looperands rather than the whole .s text, broaden the symbol pattern, and route the classes:
ASM_SYM_RE = re.compile(r'%(?:hi|lo)(\s*([A-Za-z_]\w*)')
... for each sym:
if sym.startswith('jtbl_'): continue # compiler-generated; no C decl (documented exclusion)
elif sym.startswith('func_'): -> resolve through defined_sigs/extern_sigs and append to callees with status/signature (address-taken, not a jal)
else: -> resolve through data_decls (covers D_XXXX and renamed globals alike)
- assertion to add: assert every
%hi/%looperand in the target's .s is accounted for in exactly one of {callees, data, the explicitjtbl_exclusion list}, else fail loud naming the operand — the exclusion set must be an explicit named list, never 'whatever the regex happened not to match'.
GROUP: recovery-rest
Headline: sig_unify's DATA_DECL_RE cannot parse function-pointer or sized-array data externs — 44 of the 646 D_ symbols in the ov_SC01_077 canon corpus (including the engine's callback slots D_80127088, D_8011DB28, D_801274D0) never enter data_canon, so 80 draft files keep an m2c-guessed decl that TYPE-CONFLICTS with the TU and are silently reverted by the gate as if they were codegen failures.
Scanners measured: 16
Verified CLEAN: MEASURED FULL COVERAGE (no action needed, do not spend time here): (1) sig_unify.rewrite_def def-header rx — 10600/10600 draft definitions matched; its [^()]* param class would break on a fn-ptr param but ZERO exist in 10731 drafts. (2) sig_unify.PROTO_DECL_RE — 5297/5297 bare m2c prototypes matched, GAP=0. (3) sig_unify.parse_canon_sig — 2341/2341 canonical func decls parsed. (4) recover_giant.canon_sig — 1801/1801 DEFINE_func_X() macros parsed; all 1801 DEFINE_* are zero-arg and ZERO have a { before the signature (the failure mode I hypothesised does not exist). (5) canon_resident_calls.SYM_RE — 991 parsed of 1039 assignment lines; the 48-line gap is 100% // data symbols = a JUSTIFIED exclusion, exactly as documented. (6) canon_resident_calls.FUNC_TOK — 69,876 func_ tokens across the draft corpus, every one exactly 8 hex; the {8} quantifier loses nothing. (7) inject_capped_externs.def_span — 57,472/57,472 top-level defs located across all 811 overlay TUs. (8) fix_arity_callers.draft_is_promotion_safe — 3998/4000 defs found; the 2 misses are corrupt drafts (if ( func_80161208() == 0 a0)), not a regex hole.
[CRITICAL] tools/sig_unify.py :: DATA_DECL_RE (line 36) -> collect_data_decls
- candidates 646 / parsed 602 / real skips 44
- evidence: Corpus = the tool's own canon set (src/ov_SC01_077/ov_SC01_077.c + src/shared/engine_core.h). The char class
[A-Za-z_][\w\s\*]*?contains no(,[,]or,, and the array suffix is\[\s*\](EMPTY brackets only). So three declarator forms are invisible: (a) fn-ptr / fn-ptr-array — 42 syms, e.g. src/shared/engine_core.h:2554extern void (*D_80127088)(void);and src/ov_SC01_077/ov_SC01_077.c:830extern void (*D_80187ED0[])(void);; (b) SIZED arrays — 2 syms, e.g. engine_core.hextern s32 D_80127530[4];; (c) multi-symbol lines, e.g. src/ov_SC01_000/ov_SC01_000_jr_8012ACE0.cextern s32 D_801A445C, D_801A4460, D_801A4464, D_801A4468;(only the 1st sym would even be a candidate, and the comma kills the whole match). NONE of the 44 is a justified exclusion — every one is a data symbol drafts reference. Counterfactual, type-normalized (u8==unsigned char etc.): 80 draft files declare a GAP symbol with a genuinely TYPE-CONFLICTING spelling. Concrete: .run/drafts-T6b-fail/func_801726D0.c:3extern void *D_80127088;vs engine_core.h:2554extern void (*D_80127088)(void);->conflicting types for D_80127088. Also .run/drafts-T6c-fail/func_801387B8.cextern u8 D_80127530[];vsextern s32 D_80127530[4];, and .run/drafts-T6c-fail/func_80153800.cextern s32 D_8011DB28;vsextern void (*D_8011DB28)(s32 a0);. Per-symbol draft counts: D_80127088 x45, D_8011DB28 x19, D_801274D0 x10, D_80127530 x9, D_80127540 x3. - blast radius: LOST MATCHES. sig_unify is the recovery pass whose entire job is to stop
conflicting typesfrom failing the whole-binary gate. For these 44 symbols it silently does nothing, so up to 80 drafts (each a candidate byte-exact bank) fail the TU compile and are reverted — and the failure reads as 'this draft is unrecoverable', identical in the logs to a real codegen wall. The affected symbols are the engine's fn-ptr dispatch/callback slots, i.e. the exact class scope_data_externs' own docstring already flags as unparseable by reconcile_decls — the same disease, still unfixed in sig_unify. Multiply by the family-sweep reach (x113-x134 per core) for the ones that gate a sibling family. - fix: Replace line 36-37 with a declarator-agnostic form:
DATA_DECL_RE = re.compile(r'extern\s+([A-Za-z_][^;{}\n]?\bD_[0-9A-Fa-f]+[^;{}\n]?)\s*;')
This accepts
(*D_x[])(void *),D_x[4]andD_x[]. Additionally, in collect_data_decls, emit one entry PER symbol found on the line (for s in re.findall(r'\bD_[0-9A-Fa-f]+\b', decl)) so multi-symbolextern s32 D_a, D_b, D_c;lines are not dropped wholesale — or, safer, skip multi-symbol lines EXPLICITLY and count them as a known exclusion rather than a silent one. - assertion to add: In collect_data_decls, build an over-approximating candidate set first:
cands = {s for line in <every line matching r'^[ \t]*extern\b.*;'> for s in re.findall(r'\bD_[0-9A-Fa-f]+\b', line)}. Thenmissing = cands - set(out);assert not missing, f'DATA_DECL_RE failed to parse {len(missing)} declared data symbols: {sorted(missing)[:10]}'(or print a LOUD warning listing the offending source lines). The tool must never report 'canonical decls: N data' without also reporting how many declared data symbols it could not parse.
[MEDIUM] tools/recover_giant.py :: recover() extern-gather (line 49: [l for l in lines if l.strip().startswith('extern')])
- candidates 5297 / parsed 0 / real skips 2401
- evidence: recover_giant's stated purpose is to canonicalize a giant draft's guessed callee decls against engine_core.h's DEFINE_ sigs and block-scope them. It collects ONLY lines starting with
extern. But m2c also emits callee prototypes with NOexternkeyword (<ret> func_X(<params>);— the very form sig_unify had to add PROTO_DECL_RE for in Phase 16). Across the draft corpus there are 5297 such col-0 bare prototypes in 1819 files; recover() neither canonicalizes nor block-scopes ANY of them, and still printsrecovered <fn>: N externs canonicalized(success). Type-normalized, 2401 of them (in 979 files) carry a REAL type conflict with the DEFINE canonical sig, e.g. .run/drafts-T6w1-fail/func_80153C44.c:2void func_80153C74(s32 a0, s32 a1);vs engine_core.h:1266 DEFINE_func_80153C74 ->void func_80153C74(s16, s16); .run/drafts-T6w1-fail/func_801653B8.cvoid func_801653B8(s32);vs canonicalvoid func_801653B8(s32 *); .run/drafts-T6w1-fail/func_801699D0.cvoid func_80169A4C(s32,s32);vs canonicals32 func_80169A4C(s32,s32). HONEST CAVEAT: in recover_giant's ACTUAL corpus today (.run/drafts-giants*/) there are ZERO bare prototypes — all 426 callee decls there are extern-prefixed. So this hole is currently LATENT: the tool is clean by luck, not by construction. It bites the moment recover_giant is pointed at a mainstream m2c draft (which its CLI signaturerecover_giant.py <fn> <in> <out>invites). - blast radius: LOST MATCHES (latent). Zero cost today (0/426 in drafts-giants*). If the giant-recovery lever is reused on any mainstream draft — 1819 of 10731 drafts carry bare protos — recover_giant silently leaves the guessed sigs at FILE scope: the draft then fails the whole-binary gate with
conflicting types(the exact wall recover_giant exists to remove) AND the file-scope leftovers break the find_site/compiles_standalone lift, so dedup_propagate's xN reach is lost too. It reports success either way. - fix: In recover(), gather BOTH decl forms:
PROTO = re.compile(r'^[ \t](?!extern\b)(?!return\b|if\b|while\b|for\b|switch\b|do\b|else\b)[A-Za-z_][\w \t*]?\bfunc_[0-9A-Fa-f]+\s*([^;{])\s;.*$')
is_decl = lambda l: l.strip().startswith('extern') or PROTO.match(l)
and use is_decl in place of the startswith('extern') test for BOTH the
externsand therestpartition. canon_extern already handles a missingexternprefix correctly (it rebuilds 'extern ' + cs + ';'). - assertion to add: After building
rest, assert no file-scope declaration of a func_/D_ symbol survives in it:leftovers = [l for l in rest[:di] if re.match(r'^[A-Za-z_][\w \t\*]*\b(func_|D_)[0-9A-Fa-f]+\s*[\(;]', l) and l.rstrip().endswith(';')]; assert not leftovers, f'recover_giant left {len(leftovers)} file-scope decls uncanonicalized/unmoved: {leftovers[:5]}'. Also print the count of decls it moved AND the count it could not parse, never just the former.
[MEDIUM] tools/inject_capped_externs.py :: file_scope_externs (line 60: r'^[ \t]*(extern[^\n;]*;)[ \t]*$')
- candidates 292 / parsed 263 / real skips 29
- evidence: Corpus = the default --src-file src/ov_SC01_077/ov_SC01_077.c. The
[ \t]*$anchor demands the;be the last non-space char on the line, so every extern carrying a TRAILING COMMENT is invisible. All 29 gap symbols are that class, e.g. src/ov_SC01_077/ov_SC01_077.c:12extern s32 func_8016EC0C(s32 a0, s32 a1); /* match-first, arity 2 */and :14extern s32 func_801670E4(s32 a0, s32 a1, s32 a2, s32 a3); /* derive-decl, arity 4 */. Two further sub-classes exist and are also silently dropped: multi-line externs (src/ov_SC01_077/ov_SC01_077.c:269extern unsigned char D_801DA9B8, D_801DA9B9, ...wrapping to the next line) and multi-symbol lines (SYM_RE.search takes only the FIRST symbol, soextern s32 D_801A445C, D_801A4460, D_801A4464, D_801A4468;maps 1 of 4). Across the whole overlay TU corpus (which --src-file can target) the same regex loses 3667 trailing-comment (sym,decl) pairs + 1872 multi-symbol pairs. Measured downstream effect on the default file TODAY: 1 top-level def references a GAP symbol -> it lands in themissing extern: ...skip bucket instead of being freed. - blast radius: LOST MATCHES (small today, systemic by construction). A symbol absent from ext_map goes to
missing, compiles_standalone fails, and the capped function is reported asskip 0x... (missing extern: func_X)— indistinguishable from a genuine type wall. Each such skip forfeits that function's dedup_propagate reach (x2..x134 overlays). Currently 1 def on the canonical overlay; the number scales directly with how many hand-annotated externs (the trailing/* match-first, arity 2 */style, which is the project's own convention) exist in whichever --src-file is passed. - fix: Line 60: drop the end anchor and map every symbol on the line:
for m in re.finditer(r'^[ \t](extern\b(?:[^;{}]|\n)?;)', text, re.M):
line = ' '.join(m.group(1).split())
for s in SYM_RE.findall(line):
out.setdefault(s, line)
The
(?:[^;{}]|\n)*?also picks up the wrapped multi-symbol externs, and SYM_RE.findall (instead of .search) fixes the multi-symbol case. - assertion to add:
cands = {s for m in re.finditer(r'^[ \t]*extern\b.*$', text, re.M) for s in SYM_RE.findall(m.group(0))}; missing = cands - set(out); assert not missing, f'file_scope_externs failed to map {len(missing)} declared symbols: {sorted(missing)[:10]}'. Any function later skipped with reason 'missing extern: X' where X IS in cands must be a hard error, not a skip line.
[MEDIUM] tools/sig_unify.py :: DRAFT_EXTERN_LINE_RE (line 38: r'^[ \t]*extern\b[^;]*;[ \t]*$')
- candidates 37022 / parsed 36777 / real skips 245
- evidence: Corpus = 10731 draft .c files. The
[ \t]*$anchor means the;must end the line. Gap breakdown: 206 externs with a trailing /* */ comment (e.g. .run/drafts-T6b-fail/func_8013E588.c:7extern u8 D_800B9A15; /* sb */), 15 WRAPPED multi-line externs, 15 multi-decl lines (.run/drafts-o0/func_8013B598.cextern s32 D_801DAA08; extern u16 D_801DAA0C; extern u16 D_801DAA0E;— NONE of the three is rewritten), 9 with a trailing // comment. All 245 are real skips (the tool should have canonicalized them and did not) — but I checked the counterfactual honestly: after normalizing typedef aliases (u8==unsigned char) and param names, only 10 statements in 10 draft files carry a TYPE-level conflict that the rewrite would actually have resolved. The other 235 are cosmetic (the draft already agrees with canon; the comment merely blocked the no-op rewrite). The 10 real ones are the wrapped long signatures: .run/drafts-T6b/func_80157158.cextern void func_801571C4(s32 a0, u16 a1, u16 a2, s32 a3,(wraps) vs canonextern u32 func_801571C4(...)— return type void vs u32, a guaranteedconflicting types; and .run/drafts-T6c-fail/func_80156670.c full-proto vs canon no-protoextern void func_80157158();. - blast radius: LOST MATCHES. 10 drafts today whose canonicalization silently no-ops -> whole-binary gate revert. Structurally it is the same class as the SIG_IN_BODY bug: the moment any tool or human puts a comment on an extern line — which the project's own convention encourages — that decl becomes invisible to the recovery pass forever. Note the 15 wrapped externs are exactly the >4-arg engine signatures, i.e. the hardest-to-match call-heavy functions.
- fix: Make it a STATEMENT matcher, not a line matcher, and stop requiring EOL after the
;: DRAFT_EXTERN_LINE_RE = re.compile(r'^[ \t]extern\b(?:[^;{}]|\n)?;', re.M) The sub() callback already preserves leading whitespace and replaces the matched span, so a multi-line match collapses correctly to the canonical one-liner. (Keep a guard so a{never enters the span.) - assertion to add: Per draft, before writing:
cands = len(re.findall(r'(?m)^[ \t]*extern\b', txt_before)); parsed = len(DRAFT_EXTERN_LINE_RE.findall(txt_before)); assert parsed == cands, f'{p}: {cands-parsed} extern statements not parsed by DRAFT_EXTERN_LINE_RE'— and print the aggregate 'externs seen / externs parsed' in the final summary line so a regression shows up as a number, not as a silent zero.
[MEDIUM] tools/fix_arity_callers.py :: void_re / anyproto_re / noproto_re (lines 97-99)
- candidates 128351 / parsed 128345 / real skips 6
- evidence: All three regexes hard-require the literal
externkeyword before the return type. Corpus = src/shared/engine_core.h + all 811 overlay .c (the files reachable via --binary). Of 128,351(void)-prototype declarations of a func_X, 6 lackexternand are therefore invisible: src/ov_SC01_077/ov_SC01_077_jr_8015AE2C.c:2092s32 func_80029AAC(void);, plus the same decl in ov_SC01_077_jr_801734BC.c, ov_SC01_077_jr_80178D40.c, ov_SC01_077_jr_80182268.c, andvoid func_8012A418(void);in the latter two. Two distinct callers (func_80029AAC, func_8012A418). Separately verified as NOT a problem: zero extern func decls in engine_core.h wrap across lines, so the newline-free return-type class[\w \t\*]costs nothing today. When this fires the tool prints[no caller (void) decl found] func_X— which is factually FALSE and reads to the operator as 'nothing to do here'. - blast radius: LOST MATCHES (bounded). If func_80029AAC or func_8012A418 is ever a match target that takes an argument, the no-prototype arity recovery cannot fire in the four jr carve TUs — and those are precisely the TUs the heavy-jr family sweep builds. --revert is equally blind, so a bad state there cannot be undone by the tool either. Bounded to 2 functions x 4 TUs today, but the shape means any future non-extern prototype in a carve TU is silently unrecoverable.
- fix: Make
externoptional in all three patterns and anchor the decl at line start: void_re = re.compile(rf'(?m)^([ \t](?:extern\s+)?[A-Za-z_][\w \t*]?\bfunc_{ad}\s*()\svoid\s()\s*;)', re.I) anyproto_re = re.compile(rf'(?m)^([ \t](?:extern\s+)?[A-Za-z_][\w \t*]?\bfunc_{ad}\s*()\s*[^;)]+?\s*()\s*;)', re.I) noproto_re = re.compile(rf'(?m)^([ \t](?:extern\s+)?[A-Za-z_][\w \t*]?\bfunc_{ad}\s*()\s*()\s*;)', re.I) (the(?m)^anchor prevents a call site from matching now thatexternis optional). - assertion to add: Before printing
[no caller (void) decl found] fn, cross-check with an over-approximating detector:cand = re.search(rf'(?m)^[ \t\\]*(?:extern\s+)?[A-Za-z_][\w \t\*]*?\bfunc_{ad}\s*\(\s*void\s*\)\s*;', text). Ifcandfires but the rewrite did not,sys.exit(f'FAIL: a (void) decl of {fn} EXISTS but void_re could not match it: {cand.group(0)!r}'). A 'not found' message must be provable, not assumed.
[LOW] tools/scope_data_externs.py :: _body_open_brace (line 74)
- candidates 10600 / parsed 10590 / real skips 10
- evidence: The ANSI/own-line/K&R brace fix documented in the docstring is CORRECT (I verified 10590/10600 land exactly on the definition's opening brace, 0 no-matches). But the sig regex
^[^\n]*\b{func}\s*\(takes the FIRST line in the body that mentionsfunc(— including a line inside a COMMENT. 10 drafts carry a header comment that names the function with a paren and thereby mis-anchor the search: .run/drafts/func_800D27DC.c:3// void *func_800D27DC(u32 flags /*a0*/, u8 *out /*a1=t4*/, u16 *src /*a2*/,—body.find('{', sig.end())then returns a brace that is NOT the def's, so the demoted externs are injected into whatever construct owns it. Same in .run/drafts-o0/func_8013BCDC.c, .run/drafts-t5-pilot/func_8017B490.c, .run/drafts2/func_800D2CA8.c, .run/drafts3/func_800D1B80.c, and 5 more. - blast radius: CORRUPT BUILD -> lost match (small, 0.09%). A misplaced block of
externlines yields a compile error, the sibling reverts, and the family sweep records it as a failure of the CORE rather than of the placement. Zero risk of a wrong match (the gate catches it), but it costs the bank silently. Risk grows with the exemplar bodies carried by family_remap, which are lifted from src/ where analysis-comment headers are the norm. - fix: Strip comments before locating the signature (do NOT strip them from the returned body — compute the index on a masked copy of equal length):
masked = re.sub(r'/*.?*/', lambda m: ' 'len(m.group(0)), body, flags=re.S)
masked = re.sub(r'//[^\n]', lambda m: ' 'len(m.group(0)), masked)
sig = re.search(rf'^[^\n]\b{re.escape(func)}\s(', masked, re.M)
i = masked.find('{', sig.end())
Offsets stay valid against
bodybecause the masks preserve length. - assertion to add: After computing
at, prove the brace really closes the signature: walk back from the{at index i to the matching)and assert the text between them is whitespace only, and that the paren at sig.end()-1 is balanced-closed before i.assert re.fullmatch(r'\s*', body[close_paren+1:i]), f'_body_open_brace anchored on a brace that is not {func}\'s body: {body[i-60:i+20]!r}'— a wrong anchor must raise, never silently inject.
[LOW] tools/scope_data_externs.py :: FILE_EXTERN_RE / _file_scope_data_syms (lines 58, 62)
- candidates 1017508 / parsed 1017503 / real skips 5
- evidence:
^extern\b[^;{}\n]*;cannot span lines, so a WRAPPED file-scope extern is invisible to the TU oracle. Only 5 statements out of 1,017,508 col-0 extern lines — but they are SYSTEMIC and multi-symbol: src/ov_SC01_077/ov_SC01_077.c:269extern unsigned char D_801DA9B8, D_801DA9B9, D_801DA9BA, D_801DA9BB, D_801DA9BC,(wraps). Every one of the 134 overlay TUs carries the same shape, and each loses exactly 5 file-scope D_ symbols from the oracle (measured: 200 true -> 195 seen in ov_SC06_014.c, ov_SC07_000.c, ...; 306 -> 301 in the jr_8013FFD8 carves). A second, quieter hole in the same function:DATA_SYM_RE.searchrecords only the FIRST D token of a decl, soextern s32 D_a, D_b, D_c;registers 1 of 3 even when it IS parsed. - blast radius: CORRUPT ORACLE, benign consequence TODAY. When fix() wrongly believes the TU has no file-scope decl for one of those 5 symbols, it DEMOTES the carried extern to block scope. By the tool's own (correct) 'never worse than the status quo' argument both branches are survivable — identical spelling is a legal duplicate at block scope, a differing spelling is an error at either scope (the §41 reconcile class). So I am NOT claiming lost banks here. What IS true: the oracle _file_scope_data_syms is silently wrong on 5 syms x 134 TUs, and anything built on it later (a stricter placement rule, a reconcile decision) inherits a false negative it has no way to detect.
- fix: FILE_EXTERN_RE = re.compile(r'^extern\b(?:[^;{}]|\n)*?;', re.M) # statement, not line
and in _file_scope_data_syms use
out.update(DATA_SYM_RE.findall(m.group(0)))instead of taking only the first token. In fix(), keep the col-0 line test for the BODY partition (that half is correct — gather_externs always emits one-line col-0 decls) but guard it: if a body extern statement wraps, join it before classifying. - assertion to add:
crude = set(re.findall(r'\bD_[0-9A-Fa-f]{6,8}\b', ' '.join(l for l in text.split(chr(10)) if l.startswith('extern') or _in_wrapped_extern(l))))is awkward; simpler permanent guard:assert not re.search(r'(?m)^extern\b[^;{}]*$', text), 'FILE_EXTERN_RE: TU contains a WRAPPED file-scope extern this scanner cannot see'— i.e. fail loud on the existence of the unparseable shape rather than silently under-reporting the symbol set.
[LOW] tools/sig_unify.py :: stub_re (line 176) -> cur_stubs gate (line 184)
- candidates 59201 / parsed 59101 / real skips 100
- evidence:
INCLUDE_ASM\([^,]*,\s*(func_[0-9A-Fa-f]+)\)only recognizes auto-named stubs. 100 INCLUDE_ASM stubs across src/ov_*/ name a CURATED symbol instead — all 100 arelistCdBuffer, e.g. src/ov_SC01_077/ov_SC01_077_jr_80178D40.cINCLUDE_ASM("asm/ov_SC01_077/nonmatchings/ov_SC01_077_jr_80178D40", listCdBuffer);(one per overlay's _jr_80178D40 carve). Becausecur_stubsnever contains such a symbol, the guardif fn not in cur_stubs: continuedrops the draft — and note it drops it BEFORE the write, so the draft is not even COPIED to --out. It vanishes from the pipeline entirely with no message; the run summary just says 'drafts unified: N' with a smaller N. - blast radius: LOST MATCHES (1 function today, unbounded going forward). Any function that has been given a curated name in config/symbols.* becomes permanently un-recoverable by sig_unify — and curated naming is a thing the project does more of over time, so this hole WIDENS as RE quality improves. It also interacts badly with canon_resident_calls, whose whole job is to rewrite func_ to the curated name: a draft that tool just canonicalized is one sig_unify may then silently discard.
- fix: Broaden the stub regex and reverse-map curated names to addresses:
stub_re = re.compile(r'INCLUDE_ASM([^,],\s([A-Za-z_]\w*))')
then build
cur_stubsas a set of BOTH the raw stub names and, for curated names,func_{addr:08X}resolved via canon_resident_calls.load_name_map (inverted). Also, when a draft is skipped for not being a current stub, COPY it through to --out unchanged (or log it) so nothing silently disappears from the pipeline. - assertion to add:
skipped = [p for p in drafts if basename(p)[:-2] not in cur_stubs]; assert not skipped or all(<addr has no INCLUDE_ASM stub at all in c_path>), f'sig_unify dropped {len(skipped)} drafts that have no matching stub: {skipped[:5]}'— and unconditionally printdrafts in: X, unified: N, dropped (not a stub): X-Nso a nonzero drop count is always visible rather than being folded into a smaller success number.
GROUP: metrics
Headline: progress.py's headline REAL count is wrong in BOTH directions: 532 function-instances are reported as banked matches while still INCLUDE_ASM stubs (4 dedup groups whose DEFINE_ macro exists nowhere in src), and 670 genuinely byte-exact banked instances (243k instructions — the hardest-won Phase-26 cracks) are counted as NOTHING because classify() reads a K&R-style definition as a forward declaration.
Scanners measured: 9
Verified CLEAN: MEASURED AND FULLY COVERED (do not spend time here):
(1) classify()'s INCLUDE_ASM regex — 59,248 INCLUDE_ASM sites fleet-wide, all parsed; independently gated against the ".s oracle" (every asm//nonmatchings//.s = one un-banked symbol): 0 orphan .s files across all 136 binaries (main: 1034 stubs + 959 linked + 2 blobs + 7 NM = 2002 = exactly the .s count). This scanner is complete.
(2) linked_subsegs() — the Makefile psyq_integrate + NAME := <comma,list> mvars parse: 49 candidate segments (every src/.c stem named lib/snd*/apicard* reachable from a psyq_integrate line, with $(LIBGTE_STUBS)/$(SND_STUBS)/$(APICARD_STUBS) expanded) vs 49 parsed. GAP = 0, EXTRA = 0. The Makefile-lockstep regex is currently correct.
(3) BINARIES dict — 134 src/ov_* dirs = 134 ov_ entries = 134 sig.ov_*.jsonl. No overlay is silently missing from the fleet aggregate.
(4) SRCS glob (src/<bin>/*.c, top-level only) — justified exclusion: the only .c files below the top level of main's src/ are the overlay subdirs, which are separate binaries with their own BINARIES entries. No orphan .c anywhere.
(5) is_data_blob() — applied to all 2002 main .s files it returns exactly the 2 blobs classify() reports (func_8005CE38, func_80047CAC); self-consistent and complete over the .s corpus.
(6) asm_is_trivial()'s REGEX — parses the real .s format correctly (0/2002 false "trivial"). The regex is fine; its CALLER is broken (see finding 5).
(7) weighted_metrics() address alignment — resident's 21 src stub addresses all 21 land in .run/sig.resident.jsonl; the fun_ vs func_ name-prefix difference in the resident sig is a sig-generation artifact and does NOT affect the metric, which joins on ADDRESS. The instruction-weighted / distinct-code headline numbers are SOUND: they use "not an INCLUDE_ASM stub ⇒ matched", which independently gets both the K&R functions (correctly matched) and the 4 phantom-dedup functions (correctly unmatched) right. Only the fn-count metric is corrupted.
(8) SIG's same-line definition form — classifying every uncounted sig function fleet-wide produced NO residue outside the K&R and macro-only classes: the ordinary s32 func_X(s32 a) { form is parsed with full coverage.
[CRITICAL] tools/progress.py :: dedup_members() / _DEDUP_CACHE (config/dedup.us.yaml via dedup_integrate.group_members)
- candidates 223725 / parsed 223725 / real skips 532
- evidence: DIRECTION NOTE: this gap is an OVER-claim, not a skip — the tool parses 532 registry claims it should have rejected. 7 registry groups name a
func:token that occurs in ZERO files under src/: E_func_80128ED8, E_func_8012C098, E_func_8012C0EC, E_func_8012C750, E_func_8012F14C, E_func_8012F038, E_func_8012E5CC (grep -rl 'DEFINE_func_80128ED8' src/-> 0 files). Four of them are still live INCLUDE_ASM stubs, so the same function is counted TWICE — once as a REAL dedup-shared match, once as a stub: config/dedup.us.yaml:11548- id: E_func_80128ED8/func: DEFINE_func_80128ED8/ binaries: [ ...134 overlays... ] src/ov_SC01_005/ov_SC01_005.c:401 INCLUDE_ASM("asm/ov_SC01_005/nonmatchings/ov_SC01_005", func_80128ED8); Measured over the fleet: func_80128ED8 x133, func_8012C098 x133, func_8012C0EC x133, func_8012C750 x133 = 532 (binary,fn) pairs that appear in BOTH dedup_members() and classify()'sstubsbucket. report() doesshared = dedup_members(BINARY) - set(real)— it subtractsrealbut neverstubs, so the registry claim wins unchallenged. - blast radius: Corrupt METRICS + lost MATCHES. (a) REAL is inflated by 532 function-instances that are NOT matched; every docs/progress.ov_*.md "REAL matches:" line names func_80128ED8/func_8012C098/func_8012C0EC/func_8012C750 as banked. (b)
matchableis double-counted by the same 532 (measured: matchable - |union of buckets| = 532 across the fleet). (c) LOST WORK: these 4 distinct functions x133 overlays are REPORTED DONE, so no sweep or target-picker driven off progress.py will ever queue them — 532 free instances hidden behind a false green. - fix: In dedup_members(), verify the registry against reality instead of trusting it. Two changes, both fail-loud:
(1) After building _DEDUP_CACHE, drop-and-report any claim that is still stubbed:
stubbed = {m.group(1) for cf in (ROOT/f'src/{binary}').glob('.c')
for m in re.finditer(r'INCLUDE_ASM([^)],\s*(\w+))', cf.read_text())}
names = _DEDUP_CACHE.get(binary, set())
bogus = names & stubbed
if bogus: sys.exit(f"progress.py: dedup registry claims {len(bogus)} fns MATCHED in {binary} that are still INCLUDE_ASM: {sorted(bogus)[:5]}")
return names - bogus
(2) Once, at load: assert every group's
func:token occurs at least once in src/**/*.[ch]; the 7 groups listed above fail this today. (Separately, and outside progress.py's scope: those 7 registry entries should be removed or re-integrated — they are stale claims in the source of truth.) - assertion to add: assert dedup_members(b) & include_asm_symbols(b) == set(), f'dedup registry claims {n} matched fns that src still stubs' AND assert all(group['func'] in ALL_SRC_TEXT for group in registry), 'registry claims a match whose macro does not exist in src'
[HIGH] tools/progress.py :: classify() — the SIG definition-vs-declaration discriminator (progress.py:440-445)
- candidates 534 / parsed 0 / real skips 534
- evidence: K&R (old-style) function definitions are read as forward DECLARATIONS and skipped. classify() scans forward from the SIG line for the first
{or;and calls it a decl if the;comes first — but in a K&R definition the parameter declarations sit BETWEEN the)and the{: src/ov_SC01_005/ov_SC01_005_after.c:3778 s32 func_8015A3C8(arg0) src/ov_SC01_005/ov_SC01_005_after.c:3779 s32 arg0; <--;found first -> kind='decl' ->i = j + 1; continuesrc/ov_SC01_005/ov_SC01_005_after.c:3780 { Over-approximating candidate detector (col-0<type> name(a, b)with an untyped arg list, followed by<type> arg;lines then{): 533 K&R heads across 399 files. Cross-checked against the sig oracle (every function that EXISTS in each binary): 534 (binary,fn) pairs, 243,296 instructions, are in NO bucket at all — not real, not empty, not stub, not shared. Distinct symbols: func_8015AE2C x134, func_80133CD4 x134, func_80166994 x134, func_8015A3C8 x132, func_80180000 x1. These are byte-exact BANKED matches (Phase-26's 562/493/399/369-instruction cracks). main has 0 K&R defs; this is entirely an overlay/m2c-output phenomenon. - blast radius: Corrupt METRICS. 534 function-instances / 243,296 instructions of genuinely banked, byte-exact code are invisible: REAL, byteident and matchable are each 534 too low. The per-binary REAL number gates the Gen1-exit ">=25 REAL" bar and any "which overlay next" decision. It also silently punishes exactly the highest-value work: the biggest hand-won functions in the project are the ones m2c emits in K&R form. LATENT: every future K&R draft vanishes the same way, and it reads as "the crack didn't land".
- fix: Replace the def/decl discriminator so that a
;only means 'declaration' when it appears on or before the line that CLOSES the parameter list; K&R parameter declarations between)and{must be skipped. Concretely, in classify() replace thewhile j < n: ... br/smloop with: (a) walk forward tracking paren depth from the SIG line until the parameter list's)closes -> linecl; (b) if a;appears in strip_comments(lines[cl]) AFTER that closing), kind='decl'; (c) otherwise walk forward from cl+1 skipping blank lines and lines matching^[A-Za-z_][\w \t\*\[\],]*;$(K&R param decls) — if the next non-such line contains{, kind='def', else kind='decl'. - assertion to add: assert set(real) | set(empty) | set(nonmatching) | set(stubs) | set(linked) | set(blobs) | dedup_members(b) >= {r['name'] for r in sig(b) if r['nins'] > 0}, f'{n} functions exist in {b} but are classified as NOTHING: {sorted(missing)[:5]}' — this single sig-reconciliation assertion catches findings 2 and 3 and the false side of finding 1.
[MEDIUM] tools/progress.py :: classify() (macro-emitted functions) + dedup_members() (registry binaries: list completeness)
- candidates 136 / parsed 0 / real skips 136
- evidence: Functions that exist in a binary, are matched (no INCLUDE_ASM anywhere, fleet builds byte-identical), and are counted in NO bucket — they are emitted by a shared macro in src/shared/engine_core.h, which classify() cannot parse, AND the dedup registry group that covers them omits those binaries from its
binaries:list. Measured against the sig oracle: 136 (binary,fn) pairs, 1,632 instructions. Distinct symbols: func_80128EA8 x118, func_80132EC4 x118, func_80180000 x3. Evidence that they are macro-only: src/ov_SC01_004/ov_SC01_004_jr_8013FFD8.c:178 extern void func_80128EA8(s32 a0, s32 a1, s32 a2); <- declaration only src/ov_SC01_004/ov_SC01_004_jr_8013FFD8.c:1724 func_80128EA8(...); <- call only (no definition and no INCLUDE_ASM anywhere in src/ov_SC01_004/ — the body comes from an engine_core.h macro) Contrast: in ov_SC01_005 the same function IS accounted for, because the registry lists ov_SC01_005 for that group. So the registry's per-groupbinaries:list is short by ~118 overlays. - blast radius: Corrupt METRICS (undercount). REAL / byteident / matchable each 136 too low. Smaller than the K&R class but the same shape, and it means the dedup registry — the declared 'source of truth for code shares' — is not actually reconciled against the binaries it claims to describe.
- fix: Do not try to teach classify() to expand macros. Instead make the sig reconciliation authoritative: after classify()+dedup_members(), any sig function with nins>0 that is not in any bucket and is not an INCLUDE_ASM stub is BY DEFINITION a byte-identical match (the fleet gate proves it) — fold it into
realand report the count separately, e.g.(of which macro-emitted, sig-reconciled : N). Fail loud if that reconciliation count is nonzero for a binary that has no sig (i.e. main), where it cannot be verified. - assertion to add: assert (sig_fns(b) - accounted(b) - stub_syms(b)) == set(), f'{n} matched fns in {b} counted in no bucket (macro-emitted / registry binaries: list short): {sorted(...)[:5]}'
[MEDIUM] tools/progress.py :: classify() — preprocessor handling (only #ifdef NON_MATCHING is understood)
- candidates 1 / parsed 1 / real skips 1
- evidence: DIRECTION NOTE: over-claim — the tool COUNTS an item it should have excluded. classify() special-cases
#ifdef NON_MATCHINGbut is otherwise preprocessor-blind, and it runs SIG.match on the RAW line. A function definition parked inside#if 0 ... #endifis therefore counted as a REAL match: src/resident/resident.c:861 /* func_800D00E4: left as INCLUDE_ASM — genuine structural mismatch. ... */ src/resident/resident.c:868 #if 0 src/resident/resident.c:903 void func_800D00E4(s32 arg0) { <- counted REAL src/resident/resident.c:925 #endif src/resident/resident.c:926 INCLUDE_ASM("asm/resident/nonmatchings/resident", func_800D00E4); <- also counted as a stub Measured fleet-wide: exactly 1 (binary,fn) pair today, and it is simultaneously inrealandstubs(the resident row's matchable double-count of 1). - blast radius: Corrupt METRICS, small today (resident REAL +1, matchable +1) but it is a FALSE MATCH claim: a function the source explicitly documents as un-matched is reported as banked. LATENT and growing:
#if 0is the natural place to park a near-miss draft, and every one parked there will silently inflate REAL. Same class as finding 1 — the tool cannot tell 'code that ships' from 'code that is commented out'. - fix: Give classify()'s top-level loop a preprocessor skip that mirrors the existing NON_MATCHING block skip: on a line matching
^#if\s+0\b, consume forward tracking nested#if/#ifdef/#ifndefdepth until the matching#endif, andcontinue— exactly as the#ifdef NON_MATCHINGbranch already does (progress.py:413-420). Do NOT harvest an INCLUDE_ASM out of the#if 0body. - assertion to add: assert set(real).isdisjoint(set(stubs)) and set(real).isdisjoint(set(linked)) and set(empty).isdisjoint(set(stubs)), 'a function is counted in two buckets' — bucket disjointness; today it fires 533 times (532 phantom-dedup + this 1).
[MEDIUM] tools/progress.py :: --audit (report() lines 497-501 calling asm_is_trivial() over the empty bucket)
- candidates 570 / parsed 0 / real skips 570
- evidence:
--auditis documented as 'verify every empty no-op's asm is exactly {jr,nop}'. It verifies ZERO of them and always prints 'all clean'. Mechanism: asm_is_trivial(name) doesp = find_s(name); if p is None: return None, and the caller filters withbad = [n for n in empty if asm_is_trivial(n) is False]—Noneis notFalse, so a missing .s silently PASSES. And the .s files only exist for UN-banked symbols: anemptyfunction is by definition already in C, so its .s is gone. Measured across all 136 binaries: 570emptyno-ops, of which 0 have a .s file to audit. Every one returns None. The report line then printsempties audit: 570/570 genuine jr;nop (all clean)having checked nothing. (The regex inside asm_is_trivial is NOT the bug — it parses the real .s format correctly, 0/2002 false trivials on main's stub .s files.) - blast radius: Corrupt METRICS (a check that cannot fail). The
emptybucket feeds byteident (REAL+LINKED+empty), so a mis-emittedvoid f(void) {}over a non-trivial function would inflate the byte-identical headline. In practice the full-binary SHA1 byte-gate WOULD catch such a function, so no wrong match can ship — this is a redundant guard, not the last line of defence. But it is the purest instance of the class: a tool that no-ops on input it cannot find is indistinguishable from a tool that found nothing wrong. - fix: Audit the empties against the sig (which still lists them) instead of the deleted .s, and never let 'not found' read as 'clean':
checked = [n for n in empty if find_s(n) is not None]
unverif = [n for n in empty if find_s(n) is None]
bad = [n for n in checked if asm_is_trivial(n) is False]
out.append(f"empties audit: {len(checked)-len(bad)}/{len(checked)} verified jr;nop; {len(unverif)} UNVERIFIABLE (no .s)")
Better: join
emptyto the sig by address and assert nins <= 2 — that is checkable for all 570. - assertion to add: assert len(checked) == len(empty), f'--audit verified only {len(checked)}/{len(empty)} empties; the rest had no .s and were silently passed' — a check must report its own coverage or fail.
[LOW] tools/progress.py :: weighted_metrics().src_stubs() — INCLUDE_ASM name regex (progress.py:533)
- candidates 59248 / parsed 59148 / real skips 100
- evidence: The regex is
INCLUDE_ASM\([^)]*,\s*func_([0-9A-Fa-f]+)\)— it only recognises stubs whose symbol is literallyfunc_<hex>, and it recovers the address by parsing hex out of the NAME. Any stub with a real (renamed) symbol is invisible to it, anda not in stthen reads as MATCHED. Over-approximating candidate detectorINCLUDE_ASM\([^)]*,\s*(\w+)\)over every src//*.c: 59,248 sites; src_stubs parses 59,148. GAP = 100, all the same symbol: src/ov_SC01_005/ov_SC01_005_jr_80178D40.c:4444 INCLUDE_ASM("asm/ov_SC01_005/nonmatchings/ov_SC01_005_jr_80178D40", listCdBuffer); (x100 overlays) I then checked whether any of the 100 is present in its binary's sig (which is what would actually miscredit it): 0 of 100. Total instructions wrongly credited TODAY: 0. - blast radius: Corrupt METRICS — LATENT, zero live impact. listCdBuffer is absent from the sigs, so the 100 skipped sites currently misattribute 0 instructions in the FLEET instr-weighted headline (the decomp.dev-display number). But the failure is silent and armed: the first renamed symbol that appears in both an INCLUDE_ASM and a sig is counted as MATCHED code it has not matched, inflating the project's most public number with no way to notice. Renaming symbols away from func_ is a normal, desirable decomp activity.
- fix: Stop deriving the address from the symbol name. Match any symbol —
INCLUDE_ASM\([^)]*,\s*(\w+)\)— and resolve it to an address via the sig's ownnamefield (case-insensitively), falling back to int(name[5:],16) only for func_: stub_names = {n.lower() for n in RE_ANY.findall(text)} st = {int(r['addr'],16) for r in recs if r['name'].lower() in stub_names} Then assert every stub name resolved. - assertion to add: assert len(stub_names_unresolved) == 0, f'{n} INCLUDE_ASM symbols in src/{b} could not be resolved to a sig address (renamed stubs are being counted as MATCHED): {sorted(...)[:5]}'
GROUP: split-infra
Headline: build_engine_types hard-exits on 573 of 709 type-bearing overlay .c files (81%, all 134 overlays) because 1,929 tagged-struct typedefs trip its "handle manually" overlap guard — the documented unblocker for the type-heavy tail cannot run on the corpus the tail lives in.
Scanners measured: 18
Verified CLEAN: VERIFIED CLEAN (measured, full corpus — do NOT spend time here):
• jtbl_carve.PIECE_RE + EOF_RE — 2,583/2,583 config - [...] piece lines parsed across all 134 overlay configs, gap 0.
• jtbl_carve.parse_config data-region assumption ("data/.rodata are the trailing run after the last c") — 0/134 configs have a data/.rodata piece before the last c piece. The assumption holds.
• jtbl_carve.jtbl_words — 5,671/5,671 jtbls return a non-empty word list; 17,013/17,013 entry lines are .word 0x<hex>. I specifically hunted the .word <symbol> form (which the 0x-only regex would miss, killing the trailing-.align-pad trim and re-creating the +4 image corruption): it occurs ONLY under D_/listCdBuffer pointer tables, never under a jtbl_ dlabel. The trim fires on 100% of jtbls.
• jtbl_carve.all_data_labels (jtbl half) — 0 non-8-hex jtbl labels, 0 glabel'd jtbls; 5,653/5,653.
• jr_isolate_all.code_objects — 910/910 code pieces. jr_isolate_all.rodata_carves — 722/722 .rodata carves. Gap 0 on both, all 134 configs.
• overlay_src_split.macro_table / _MACRO_SIG — 1,804/1,804 function-like macros across engine_core.h + ov_setters.h + clearTbl40.h, all with a definition body parsed (def_lines != [] for every entry). This is the exact scanner shape that produced the SIG_IN_BODY_RE bug in a sibling tool; _MACRO_SIG is clean because it does NOT require )\s*{ between signature and brace.
• overlay_src_split.INCLUDE_ASM / DEFINE_FUNC / MACRO_ARG_ANCHOR — 59,246 INCLUDE_ASM lines and 6 col-0 macro-arg anchors, gap 0. (I over-approximated MACRO_ARG_ANCHOR with a space-tolerant ^\w+\s*\(\s*func_ to catch SETTER (func_X,...); no such form exists.)
• build_engine_types.find_defs / find_typedefs / typedef_name — the FINDERS are clean: 6,339/6,339 named struct|union defs, 14,701/14,701 typedefs. I hunted 4 hazards and all came back zero: 0 named enum X { in the corpus (so the struct|union-only regex loses nothing), 0 nested/overlapping def spans (the unchecked def-vs-def --strip corruption is not reachable), 0 multi-declarator typedefs (typedef struct{...} A, *PA;), 0 block-scope aggregates. The tool's blocker is PURELY the overlap guard (finding #1), not the parsing.
[CRITICAL] tools/build_engine_types.py :: main() tagged-struct-typedef overlap guard (lines 164-167), against find_defs + find_typedefs
-
candidates 709 / parsed 136 / real skips 573
-
evidence: Corpus = 875 overlay .c; 166 have no types; 709 are type-bearing. 573 of those 709 (81%) contain >=1 TAGGED-struct typedef and therefore hit
sys.exit('[overlap] a tagged-struct typedef matched both finders — handle manually'). 1,929 tagged typedefs total; all 134/134 overlays have >=1 bailing file. Examples: src/ov_SC01_000/ov_SC01_000_jr_8012ACE0.c:609typedef struct Entry_8012DDA4 { u16 active; unsigned char pad[0x10C-2]; } Entry_8012DDA4;; src/ov_SC01_000/ov_SC01_000_jr_8015AE2C.c:5077-5082typedef struct SubB_8016A73C {...} SubB_8016A73C;/SubA_8016A73C/Mat_8016A73C/Obj_8016A73C; src/ov_SC01_000/ov_SC01_000_jr_80178D40.c (5 tagged typedefs). The tool's own comment asserts 'Our source has only ANONYMOUS-struct typedefs (no tag)' — that was true in Phase 20 and is now false: the harvest agents write their guessed overlay-local structs in the tagged form. -
blast radius: Lost MATCHES, at the largest remaining scale. tools/inject_capped_externs.py:126 explicitly routes every type-bearing body away from extern-injection with reason 'inline type def -> build_engine_types.py (§28b)' — i.e. the type-heavy tail's ONLY sanctioned unblocker is this tool, and it hard-exits on 81% of the files that tail lives in. This is the '3,098 type-heavy tail' and the 9 zero-bank type-using families. NOTE (honest): this is FAIL-LOUD, not a silent skip — it prints '[overlap] ... handle manually' and exits. It went unfixed because the message reads like an edge case rather than an 81% coverage failure.
-
fix: The guard is over-conservative: a
typedef struct Tag {...} Alias;is perfectly liftable — the typedef span ALREADY CONTAINS the whole struct body, so it just must not be double-counted. Replace the sys.exit with containment handling. In main(), after computing defs/tdefs:contained = [d for d in defs if any(ts <= d[2] and d[3] <= te for ,,ts,te in tdefs)] partial = [d for d in defs if any(ts < d[3] and d[2] < te for ,,ts,te in tdefs) and d not in contained] if partial: sys.exit('[overlap] PARTIAL def/typedef span overlap — handle manually') defs = [d for d in defs if d not in contained] # the typedef carries the body; don't lift or strip twice
Then add a struct <Tag>; forward decl for each contained tag alongside the existing forward-decl block (safe, and lets pointer-only refs resolve). The strip spans stay disjoint by construction, so --strip is safe. Keep the sys.exit ONLY for a genuine PARTIAL overlap, which is the malformed case the guard was actually written for.
- assertion to add: assert every named-struct def span is either DISJOINT from every typedef span or FULLY CONTAINED in one; count the contained ones and print
lifted N named + M typedefs (K tagged-struct typedefs folded into their typedef). Fail loud ONLY on a partial overlap. Additionally:assert (len(defs)+len(tdefs)) == n_candidates - n_containedwhere n_candidates is a crudere.findall(r'\b(struct|union|enum)\s+\w+\s*\{|\btypedef\b', blank_comments(text))count — so a future type form that the finders cannot see fails the run instead of being quietly lifted-minus-one.
[HIGH] tools/jr_isolate_all.py :: _file_scope_decls() — the _HOIST_RE.match(line) and _SAFE_TYPE.match(line) predicate (line 336)
-
candidates 1126761 / parsed 1062837 / real skips 4040
-
evidence: Over-approximating candidate = every col-0, brace-free,
;-terminated line in src/ov_/.c (1,126,761). Hoisted into the carried decl layer: 1,062,837. 4,040 lines MATCH _HOIST_RE but are REJECTED by _SAFE_TYPE and silently dropped — 683 function PROTOTYPES + 3,357 DATA externs. The dropped-base-type histogram: struct 1313, S801563EC 678, B8 544, Blk20 274, S8 272, SV4 268, uint 139, S16 134, ImgRect8 134, Entry 134, DStruct 134, code_fn 21, volatile 3, +8 more. Concrete: src/ov_SC01_000/ov_SC01_000_jr_8015AE2C.c:969extern S801563EC *func_801563EC(u16 idx);(PROTO — dropped); :990extern B8 D_80128120[];; src/ov_SC01_000/ov_SC01_000_jr_80178D40.c:2662extern uint * func_80177EA4(uint *param_1, int param_2, uint param_3, int param_4);(PROTO — dropped); src/ov_SC01_000/ov_SC01_000_jr_8012ACE0.c:224extern struct BigCopy D_80126DB8;. TWO of the dropped base types are NOT file-local at all:uint(139 drops) andcode_fn(21 drops) are BOTH DEFINED IN src/shared/engine_types.h, which engine_core.h pulls into every region — so _SAFE_TYPE rejects include-provided types it has no reason to reject.volatile(3 drops) falls off because the whitelist hasconstbut notvolatile. Most damning: the comment ON _SAFE_TYPE (lines 296-301) states these decls DO ride along — 'one naming a FILE-LOCAL type is only safe once that type is carried too — which file_scope_types() now does, so such decls ride along after their typedef'. The predicate never implements it. The comment documents a fix that was never applied to the code. -
blast radius: Corrupt BUILD, and specifically the SILENT half of it. The 3,357 dropped DATA externs are loud (undeclared identifier -> compile error, someone notices). The 683 dropped FUNCTION PROTOTYPES are not: in C89 an undeclared function is implicitly
int f(), so the TU still COMPILES — with the wrong return type and lost pointer-ness. This project has already byte-proven that return type drives codegen (cookbook '@class: schedule (delay-slot fill via void return type)'; src/ov_SC01_077/ov_SC01_077_after.c:3301 carries an engine_core.hextern int->extern voidflip described as byte-neutral precisely because the return type moves the delay slot). So a dropped prototype is a silent byte-changer on any future cut. Today's split is green only because the source redundantly re-declares externs per fn-group, so most items carry their own decl — this is a latent landmine that fires on the NEXT jr_isolate_all cut, not a live fire. -
fix: Make the predicate accept a decl whose base type is carried in the SAME layer, which is what the comment already promises. In _file_scope_decls(), collect the carried types first, then widen the test:
carried = set() for ,,kind,text in items: for block in oss.file_scope_types(text): carried |= set(re.findall(r'}\s*([A-Za-z_]\w*)\s*;|\b(?:struct|union|enum)\s+([A-Za-z_]\w*)', block)) ... base = re.match(r'^\s*(?:extern\s+)?(?:const\s+|volatile\s+)(?:struct\s+|union\s+|enum\s+)?([A-Za-z_]\w)', line) if _HOIST_RE.match(line) and (_SAFE_TYPE.match(line) or (base and base.group(1) in carried | ENGINE_TYPES)): out.append((line.rstrip(), False))
where ENGINE_TYPES is parsed once from src/shared/engine_types.h (that alone recovers uint and code_fn). Also add volatile to _SAFE_TYPE's optional-qualifier group alongside const (it is currently absent — a one-token omission costing 3 drops).
- assertion to add: assert that EVERY col-0 line matching _HOIST_RE is either hoisted or its base type is provably unavailable; i.e.
dropped = [l for l in hoist_matches if not emitted]; assert not dropped, f'{len(dropped)} file-scope decls dropped from the carried layer: {sorted({base_type(l) for l in dropped})}'. Print the base-type histogram of any drop and FAIL — a decl the tool recognised as hoistable but could not place is a bug, never a silent no-op. (This assertion alone would have surfaced all 4,040 today.)
[HIGH] tools/lint_symbol_refs.py :: main() glob src/**/*.c + load_symbols() (2 of 138 symbol files) + no __asm__-label model
- candidates 967 / parsed 962 / real skips 5
- evidence: THE TOOL IS CURRENTLY RED AND UNWIRED.
python3 tools/lint_symbol_refs.pyprints '43 STALE func_ ref(s) — clean-build will FAIL' and exits 1.grep -rn lint_symbol_refs Makefile config/*.mk tools/*.py=> ZERO hits outside the file itself;make report(Makefile:148-156) runs progress.py/difficulty.py/dup_report.py and never calls it. Its own docstring says 'Run it after any symbols rename and inmake report' — it is not there. All 43 hits are FALSE POSITIVES, from two distinct blind spots: (a) ASM-LABEL ALIASES — src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c:603extern void func_8005C324(int dst, int src, int n) __asm__("memcpy"); /* Phase-24: ... keep the non-builtin C name here (else built-in codegen), emit via asm-label */. config/symbols.resident.txt:21-24 documents this as the SANCTIONED pattern ('func_8005C324 aliases the SAME address ... asm("memcpy") label so they emit this same 0x8005C324 call. One symbol resolves both.'). The lint has no model for asm labels. (b) PER-BINARY SYMBOL STACKS — src/libc2_1.c:10INCLUDE_ASM("asm/nonmatchings/libc2_1", func_8005C324);is flagged because load_symbols() unions symbols.us.txt + symbols.resident.txt into ONE map, butmemcpy = 0x8005C324lives ONLY in symbols.resident.txt, which the main EXE deliberately does NOT stack (symbols.resident.txt:20: 'so main — where 0x8005C324 is DEFINED by MEMCPY.o — is untouched'). Proof it resolves: asm/nonmatchings/libc2_1/func_8005C324.s EXISTS on disk. COVERAGE GAPS on top: (c) the glob issrc/**/*.c, so src/shared/.h is NEVER scanned — yet the docstring names 'func_(...) calls in shared macros (src/shared/engine_core.h)' as breakage source (b) of the very bug it exists to catch; engine_core.h holds 10,360 func_/D_ tokens, func_80144B9C.h 180, clearTbl40.h 2. (d) load_symbols reads 2 of 138 config/symbols..txt; the 136 ignored per-overlay files hold 1,598 curated names that can never be checked against. - blast radius: Corrupt BUILD. The Phase-24 T5b/T5c class (a symbols rename leaves committed src/ refs dangling; clean rebuilds fail, incremental builds mask it — the R22 failure mode) is presently UNGUARDED: the only detector is unwired, and if someone wired it today it would fail immediately on 43 false positives and be reverted. The header blindness is the sharper hole — engine_core.h is the single most-shared file in the project (1,801 macros ×134 overlays), so ONE dangling func_ in it breaks every overlay's clean build at once, and that file is exactly the one the lint cannot see.
- fix: Four changes: (1) ASM-LABEL MODEL — before flagging, skip any token on a line carrying
__asm__("<name>")where == curated[a]; strip_comments_strings currently blanks the label's contents, so capture it BEFORE blanking (scan the raw line for__asm__\s*\(\s*"([^"]+)"). (2) PER-BINARY SYMBOL STACK — build the curated map per source tree from that binary's splat configsymbol_addrs_pathlist (overlay_src_split.load_ov_syms already implements exactly this walk); src/.c at the root -> splat.us.exe.yaml's stack (which excludes symbols.resident.txt), src//.c -> splat..yaml's stack. This kills the libc2_1 class of false positive AND makes the 136 per-overlay symbol files (1,598 curated names) actually checked. (3) CORPUS — change the glob tosrc/**/*.[ch]so src/shared/*.h is linted (10,542 tokens currently invisible). (4) WIRE IT — add$(VENV_PY) tools/lint_symbol_refs.pyto thereport:target in the Makefile (its docstring already claims it is there). - assertion to add: Two assertions. (a) COVERAGE:
scanned = glob('src/**/*.[ch]'); assert set(scanned) == set(all_committed_sources), f'lint blind to {set(all_committed_sources)-set(scanned)}'— a source file the lint cannot see must fail the run, not be silently omitted. (b) GREEN-BY-DEFAULT: once the false positives are fixed the tool must exit 0 on HEAD, andmake reportmust invoke it — so add a CI-ish guardassert lint_symbol_refs.main() == 0to the report target. A guard that is allowed to sit red is a guard that does not exist.
[MEDIUM] tools/overlay_src_split.py :: scan_construct() force_decl short-circuit (lines 157-158, 188-194) — and the selftest that cannot see it
- candidates 1021298 / parsed 1021296 / real skips 2
- evidence: Candidate = every top-level construct parse_overlay_c classifies as a non-anchor DECL (1,021,298 across 811 overlay .c). I classified all of them: 1,003,781 are genuine prototypes, 17,515 are type defs (both justified). TWO contain a
{body and are NOT type keywords — i.e. real function DEFINITIONS that were never anchored: src/ov_SC01_077/ov_SC01_077_after.c:2151extern void func_80150170(void *a0); extern s32 func_8014FE60(void *a0); void func_8014FDF4(struct S8014FDF4 *a0) { ... }and src/ov_SC01_077/ov_SC01_077_after.c:3299extern void func_80156044(int arg, int a1); int func_80155FF8(int arg, int a1) { __asm__ __volatile__( ... ); }. Both put externs and a DEFINITION on ONE physical line; scan_construct'sforce_declfires on the leadingexternkeyword and returns at the FIRST depth-0;, so the definition that follows on the same line is never seen. VERIFIED:parse_overlay_con that file yields 731 items and neither 0x8014FDF4 nor 0x80155FF8 is among them. AND THE SELFTEST PASSES GREEN ANYWAY:tools/overlay_src_split.py selftest src/ov_SC01_077/ov_SC01_077_after.cexits 0 with 'round-trip exact: True / addressed: 730 / unresolved(non-footer): 0 / non-monotonic transitions: 0'. This is the direct answer to the audit's question about the 404/404, 341,902-item selftest: it is a SERIALISATION check, not a coverage check. Unrecognised text is absorbed into the PREAMBLE of the next anchor, so the round-trip is exact BY CONSTRUCTION even when an anchor is missed. The selftest is structurally incapable of detecting this bug class. - blast radius: Corrupt BUILD (latent), plus a metrics hole in the EXEMPLAR overlay. ov_SC01_077 is the family-template source (build_engine_types' default --source; the ×134 remap exemplar), so its item stream is the one everything else is derived from — and it is short two definitions. Two consequences: (1) partition()/jr_isolate_all — the swallowed definition rides in the preamble of the NEXT anchor, so if a cut is ever placed between the swallowed def and its host anchor, the function BODY moves into the wrong object -> .text address shift -> SHA1 break. Not reachable today (neither 0x8014FE60 nor 0x80156044 is a jr vram), so this is a landmine, not a live fire. (2) jr_inventory's
realcmap is built from parse_overlay_c's def/define items — a definition missing from that map is a banked function the isolator cannot see, which is EXACTLY the mechanism that stranded func_801734BC's .rodata carve (jr_isolate_all lines 228-237 document that incident). Neither of these two is a jr today, so no carve is stranded now. - fix: force_decl must not survive a top-level
;. Today it is computed ONCE from the first token of the construct and then latches for the whole scan. Make the top-level;a construct BOUNDARY and re-classify from the next token. Minimal surgical change in parse_overlay_c's loop, before calling scan_construct: split the physical line on depth-0;(comment/string/paren/brace aware — _strip already gives the machinery) and treat each resulting fragment as its own top-level construct. Equivalently, inside scan_construct, on hitting a depth-0;while force_decl is set and there is REMAINING CODE on the line, recompute force_decl from the remaining text and continue the scan rather than returning. Thenextern A; extern B; void f(){...}yields three constructs: two decls and one anchored def. - assertion to add: Two, because the current selftest cannot see this. (a) In parse_overlay_c, after building
items:assert not [it for it in items if it[2] not in REAL_KINDS and _has_definition_header(it[3])]— i.e. NO item's preamble may contain a{-bodied function-definition header that is not that item's own anchor; fail loud with file:line. (b) Add a real COVERAGE gate to selftest() alongside the round-trip: crudely count candidate anchors (INCLUDE_ASM lines + col-0^\w+\s*\(\s*func_macro invocations + col-0}lines) andassert n_anchors >= n_col0_close_braces - n_type_blocks, printing the delta. The existing 'round-trip exact + 0 unresolved + monotonic' triple is 100% green on a file with two missed definitions and must never again be reported as proof of coverage.
[MEDIUM] tools/jr_isolate_all.py :: jr_inventory() — re.fullmatch(r'func_[0-9A-Fa-f]{8}', fn) on the .s basename (line 81)
-
candidates 5899 / parsed 5895 / real skips 4
-
evidence: Candidate = every .s under asm/ov_/nonmatchings// whose text references a
jtbl_symbol (5,899). Parsed: 5,895. The 4 misses are all the same function under a CURATED name: asm/ov_SC03_119/nonmatchings/ov_SC03_119_jr_80178D40/listCdBuffer.s, and the same in ov_SC03_118, ov_SC01_084, ov_SC03_108. This is NOT a data label — it is a genuine 0xA64-byte handwritten jr function at 0x80180000 with 2 jtbl references ('/* Handwritten function */ nonmatching listCdBuffer, 0xA64' /glabel listCdBuffer/sra $v0, $v0, 16...), and it is INCLUDE_ASM'd from real source: src/ov_SC03_119/ov_SC03_119_jr_80178D40.c:4442INCLUDE_ASM("asm/ov_SC03_119/nonmatchings/ov_SC03_119_jr_80178D40", listCdBuffer);. Because its name islistCdBufferand notfunc_XXXXXXXX, the fullmatch filter drops it and jr_inventory reports it does not exist. (Related, LOW: the same curated name defeats jtbl_carve.all_data_labels'(?:jtbl_|D_)[0-9A-Fa-f]{8}regex in the 33 OTHER overlays where 0x80180000 is data —dlabel listCdBufferat asm/ov_SC01_000/data/tail.data.s:2380 is invisible to the carve-boundary oracle. I checked exploitability and it is currently NIL: listCdBuffer sits at 0x80180000 in the general data region, never immediately after a jtbl, and jtbl_words' enddlabel-bounded trailing-zero trim clampsendto the true extent regardless of a missed boundary label. The boundary oracle is silently incomplete but the trim accidentally masks it — worth an assertion, not a fix.) -
blast radius: Lost MATCHES (4 potential banks), and it falsifies the tool's core invariant. jr_isolate_all's docstring and its lines 118-131 establish the rule that EVERY jr in a cut object must get its own region, because 'a region may host AT MOST ONE .rodata carve' — that invariant is what the func_8015AE2C/func_801734BC +33-byte image corruption taught. listCdBuffer is a jr that the tool cannot see, and in these 4 overlays it currently sits INSIDE
<ov>_jr_80178D40, sharing a region with an already-banked jr. So the invariant 'every jr has its own region' is false in 4 overlays right now. The moment listCdBuffer is matched and banked, jtbl_carve hits its 'subseg would host NON-CONTIGUOUS .rodata carves' fail-loud (jtbl_carve.py:250-254) and the bank cannot proceed without a hand-isolation. Fail-loud, so no byte corruption — but 4 banks are blocked and the blocker will present as a mysterious carve error rather than a naming gap. -
fix: Resolve the .s basename through the symbol table instead of demanding the func_ shape. jr_inventory already loads
syms = oss.load_ov_syms(ov)(line 88) for the realc walk — hoist it above the asm scan and use overlay_src_split.addr_of(), which handles BOTH the func_ shape and a curated-name lookup:syms = oss.load_ov_syms(ov) for p in glob.glob(f'asm/{ov}/nonmatchings//.s'): if not re.search(r'jtbl_[0-9A-Fa-f]{8}', open(p).read()): continue fn = os.path.basename(p)[:-2] a = oss.addr_of(fn, syms) # was: re.fullmatch(r'func_[0-9A-Fa-f]{8}', fn) if a is None: unresolved.append(fn) # and FAIL, see the assertion continue asm_jr[a] = fn
Same class of fix in jtbl_carve.all_data_labels: accept any (?:dlabel|glabel)\s+(\S+) and resolve the name via the overlay's symbol stack, so a curated data name still bounds a carve.
- assertion to add:
assert not unresolved, f'jr_inventory: {len(unresolved)} jtbl-referencing .s files whose symbol could not be resolved to an address: {unresolved}'— a jr function the isolator cannot NAME must abort the run, never be silently dropped from the cut list. Pair it with a post-plan invariant that is the tool's actual contract: for every -O2 object being cut,assert len(jr_in_region) <= 1 for each emitted region— i.e. re-derive the jr set from the asm AFTER planning and assert no region hosts two, so a jr the inventory missed is caught by the invariant even if the naming fix is ever regressed.
GROUP: family-engine
Headline: family_remap.extract_unit cannot read a DEFINE_func macro body out of src/shared/engine_core.h — so 93 of the 218 "matched" h_seq exemplars (43%) are PHANTOM and 1834 still-stubbed, PURE/IMM-clean, symbol_map-clean member templates are never even attempted by the sweep.
Scanners measured: 23
Verified CLEAN: MEASURED AND CLEAN — do not spend time here: (1) dedup_propagate.onboarded_overlays — the ^OVERLAY_BINARIES\s*:=\s*(.*)$ regex parses 134 overlays; there are exactly 134 src/ov_* dirs and config/overlays.mk contains zero += and zero line-continuations, so the single-line assumption holds. (2) family_sweep.load_sigs (glob .run/sig.ov_*.jsonl) — 139 sig files exist, 134 parsed; the 5 excluded are sig.SLUS_007.26, sig.resident, sig.resident_image, sig.sep8_SLUS_007.26, sig.aug31_USA_DEMO.EXE — genuinely not overlays, fully justified. (3) family_sweep's hard-reg PIN GUARD (__asm__\s*\(\s*"\$) — crude detector (any line with register + asm) finds 18,319 candidates across src/ + all 5 raw-crack draft dirs, the guard catches 17,908; all 411 gap items classified: 408 are COMMENT lines mentioning "register-asm", and 3 are register s32 r asm("$2"); inside src/shared/engine_core.h macros, which this guard never scans. Zero real skips. (Latent-only: the bare asm("$N") and __asm__ volatile ("$N") spellings would evade it — worth widening to \b(?:__asm__|asm)\b[^;]*"\$ for free.) (4) dedup_propagate's macro-safety plan filters (the "//" in l / trailing-backslash / (\b(struct|union)\s+\w+\s*\{)|(\btypedef\b) rejects) — measured on ov_SC01_077's visible defs: 18 candidates (def, unregistered, h_exact reach>=2), and ALL 18 survive all three filters. They cost nothing today and their skips are printed, not silent. (5) family_remap.apply_remap — single-pass simultaneous longest-first substitution; no coverage gap found. (6) family_remap.classify_member / reg_fields / imm_map_tier1 — their skips (STRUCT 137, LEN, "unresolved immediate: asm-ambiguous" 179 / "not-in-C" 8 / "non-imm diff op=0xf" 3) are all COUNTED and REPORTED by the sweep, i.e. declared capability limits, not silent skips. Likewise the sweep's largest single loss bucket, pinned-exemplar (3,427 members), is a declared §42e refusal. (7) extract_unit's _DECL_LAYER_END guard and its post-R14 trailing-comment strip both behave correctly: over 2,030 matched functions in 4 overlays, extract_unit produced ZERO malformed units (no unbalanced braces, no unit containing two function definitions) — the R14 declaration-swallowing bug is genuinely fixed. NOT MEASURED (deliberately): dedup_propagate.compiles_standalone — exercising it writes .run/dpcc/t.c and forks cc1, which I would not do while the fleet build is mid-flight; its skips are at least printed ("[skip] N not self-contained"), so it is not silent, but its true rejection rate is unaudited. Two LOW/latent items not worth a finding slot: dedup_propagate's o0_skip guard tests _p.name.endswith("_o0.c") while _o0b.c is ALSO compiled -O0 (Makefile WHALE_O0B_OBJS) — currently vacuous because every _o0b.c contains 0 inline defs, but it is a live trap the moment a second whale-region function is matched; and find_site's stub_line() hardcodes nonmatchings/{ov}, so it can never recognise a stub in a split file ({ov}_after, {ov}_jr_*) — harmless today because no caller consumes its 'stub' verdict, but it will silently mislead the moment one does.
[CRITICAL] tools/family_remap.py :: extract_unit (glob src/<ov>/<ov>*.c + the ^\s*[A-Za-z_][\w *]\bfunc_\s( def scan)
- candidates 218 / parsed 122 / real skips 96
- evidence: Ground truth = .run/family_hseq.json exemplars with kind matched/matched-ov077 (218). extract_unit returns a unit for only 122. Classified all 96 of the gap: (a) 93 families = the exemplar is matched via a DEFINE_func_() macro whose BODY lives in src/shared/engine_core.h, not in any .c — extract_unit only globs src//*.c, so it returns None. e.g. src/ov_SC01_077/ov_SC01_077_after.c:1797
DEFINE_func_8014FDF4()(family 0x80151FB4 = 268 members; 0x80174784 = 243; 0x8016B448 / 0x80172780 / 0x80146AB4 = 134 each) with the body at src/shared/engine_core.h:13266#define DEFINE_func_8014FDF4() \. (b) 3 families = the exemplar is a NAMED INCLUDE_ASM stub (see the stub_map finding). ZERO justified exclusions in the gap. - blast radius: Lost MATCHES. The 96 phantom families carry 2157 candidate members; of those, 1834 are measured to be (i) still an INCLUDE_ASM stub, (ii) classify_member PURE/IMM, (iii) symbol_map-clean, (iv) unpinned — i.e. they would be STAGED and byte-gated today and are instead dropped before the first build. Bands: mid=801, tiny=1033 — this lands squarely on the NEXT queued task ("Task 9 — mid-band"). It also corrupts METRICS: the manifest's "218 matched exemplars" is really 122; every sweep run silently spends its skip budget on families that can never produce a draft. These are the HIGHEST-reach functions in the project (they were dedup'd into engine_core.h precisely because reach>=2), so the loss is concentrated in the largest families.
- fix: In extract_unit, after the per-.c scan fails, fall back to the shared macro: search src/shared/engine_core.h for
^#define DEFINE_func_<ADDR>\(\) \\\n((?:.*\\\n)*.*)$(re.M) and reconstruct the unit by stripping the trailing line-continuation backslashes and the 4-space indent from each captured line — make_macro() built it from exactlyexterns + def block, so the reconstructed text IS the unit extract_unit wants. Return (body, 'src/shared/engine_core.h'). Also scan the other shared bodies (src/shared/func_*.h) the same way. NOTE: the caller in family_sweep must keep using the SIBLING's src_rel for the reconcile/scope_data step (the exemplar's cf is now a header, not a TU). - assertion to add: extract_unit must never return None for an addr that the caller believes is MATCHED. Add:
assert unit is not None, f"{ov} func_{addr:08X}: believed matched but no source form found (stub? macro? named stub?)"— and in family_sweep/family_hseq, assert that every exemplar classified matched/matched-ov077 yields a unit, failing loud with the count (phantom exemplars: N of M) instead of silently incrementing a skip counter.
[HIGH] tools/dedup_propagate.py :: overlay_files (hardcoded suffix allowlist for suf in ("_a", "_o0", "_o0b", "_after"))
- candidates 811 / parsed 404 / real skips 407
- evidence: Over-approximating detector = every file matching src//*.c across the 134 onboarded overlays: 811 files. overlay_files() returns 404 of them. The 407-file gap is ENTIRELY the Phase-26 jr-isolation splits, which the allowlist predates: _jr_8015AE2C ×134, _jr_801734BC ×134, _jr_80178D40 ×134, _jr_8017FCB0 ×2, _jr_80182268 ×2, _jr_8017BEBC ×1. Those invisible files contain 36,135 INCLUDE_ASM stubs and ~32,000 inline defs — half the corpus. Zero justified exclusions. This is the SAME bug as the Phase-24
_o0b/_aftermiss, re-opened by the new split family. - blast radius: Lost MATCHES (free ones). overlay_files gates source_text/find_site/apply_plan/struct_check/reconcile_caller_extern, i.e. ALL of dedup_propagate. Measured on the propagation source ov_SC01_077: 435 of its 689 inline defs live in jr* files and are invisible to
--auto-from. Two of them are h_exact reach-134 AND still INCLUDE_ASM stubs in 133 overlays each — 266 free ×N member banks that --auto-from cannot even nominate today: func_801749C8 (105 ins, def at src/ov_SC01_077/ov_SC01_077_jr_801734BC.c:2916, stub at src/ov_SC01_000/ov_SC01_000_jr_801734BC.c:2681) and func_80165CA0 (99 ins, def at src/ov_SC01_077/ov_SC01_077_jr_8015AE2C.c:4117, stub at src/ov_SC01_000/ov_SC01_000_jr_8015AE2C.c:3922). Every future jr-region crack is likewise un-propagatable. (--addr 0x801749C8 --source-overlay ov_SC01_077also fails: it fails CLOSED with "not inline-def", so no registry corruption is reachable — it just does nothing.) - fix: Replace the allowlist with a glob:
out = [(c_path(ov), ov)] + [(p, p.stem) for p in sorted(ROOT.glob(f"src/{ov}/{ov}_*.c"))]— the asm_subdir is always the file stem (ov_SC01_000_jr_8015AE2C.c -> asm subdir ov_SC01_000_jr_8015AE2C), which is exactly the invariant the existing four entries already satisfy. Do NOT add_jr_*to the tuple: the next split family will re-open the same hole. MUST be fixed together with the find_site K&R bug below — fixing overlay_files alone exposes the jr files, whose biggest matched functions (func_8015AE2C, func_80166994) are K&R and would still be silently dropped. - assertion to add:
assert {p for p,_ in overlay_files(ov)} == set(Path(f"src/{ov}").glob(f"{ov}*.c")), f"overlay_files missed {...}"— a hardcoded suffix list must be proven exhaustive against the directory on every call, and fail loud when a new split kind appears.
[HIGH] tools/dedup_propagate.py :: find_site (defre ^\s*[A-Za-z_][\w *]\b{s}\s([^;{{])\s({{)?\s*$ + the "brace on this line or the next non-blank line" rule)
- candidates 2030 / parsed 2002 / real skips 21
- evidence: Ground truth over 4 overlays (ov_SC01_077/000, ov_SC02_000, ov_SC06_008), scanning ALL files (so this isolates the regex from the overlay_files bug): M = addrs in the sig that are neither an INCLUDE_ASM stub nor a DEFINE macro = 2030. find_site returns kind 'def' for 2002. Every one of the 28 gap items classified: 16 = K&R OLD-STYLE definitions — the sig line matches, but the next non-blank line is a parameter DECLARATION, not '{', so the
if not lines[j].lstrip().startswith("{"): continuedrops it silently. src/ov_SC01_077/ov_SC01_077_jr_8015AE2C.c:1327s32 func_8015AE2C(arg0)(THE ×134 whale), :4357s32 func_80166994(param_1, param_2, param_3, param_4), ov_SC01_077_a.c:2747s32 func_80133CD4(arg0, cmd, base, arr), ov_SC01_077_after.c:4279s32 func_8015A3C8(arg0). 5 = signature not terminated by ')' on its own line: ov_SC01_077_after.c:3420void func_80157158(s32 a0, u16 a1, u16 a2, s32 a3,(multi-line params, reach-134) and ov_SC01_077_o0.c:36void func_8013B7AC(int a0) { D_801DAA08[a0].f0 = 0; }(one-line body). 7 = JUSTIFIED (no source form at all: func_80144B9C lives in src/shared/func_80144B9C.h; 2 addrs are splat-merged into a neighbouring asm chunk). family_remap.extract_unit finds ALL of these (its pattern has no end-anchor and its brace scan is a pure forward count) — so the two tools disagree, and only find_site is wrong. - blast radius: Lost MATCHES — currently LATENT (0 live lost banks: the 4 affected reach>=2 fns are already banked everywhere), but it is the second half of the overlay_files fix. K&R is the project's house style for exactly the hard, high-reach functions the cookbook says to write that way, and the jr* files are full of them; the moment overlay_files is fixed, find_site will silently drop the biggest prizes in the newly-visible half of the corpus. It also silently caps the o0_skip guard (which is built from find_site).
- fix: Two changes to find_site's defre block. (1) Relax the anchor so a multi-line signature is admitted: match
^\s*[A-Za-z_][\w \*]*\b{s}\s*\((as extract_unit does) and then, if the line has no ')' , consume forward lines until the paren depth returns to 0 — that consumed span is the signature. (2) Replace the strict "next non-blank line must start with '{'" test with a scan forward over K&R parameter-declaration lines (lines matching^\s*[A-Za-z_][\w \*]*\b\w+\s*(\[[^\]]*\])?\s*;\s*$) until the first line containing '{' — if a ';'-only prototype terminator or another def signature is hit first, it is a prototype (reject). Add the single-line-body case by allowing the line itself to contain '{'. - assertion to add: find_site and family_remap.extract_unit must agree:
assert (find_site(...)[0]=='def') == (extract_unit(ov,addr)[0] is not None)for every non-stub, non-macro sig addr — run it as a fleet-wide self-check in the tool's --check-only path and fail loud on any disagreement. Independently:assert defs_found + stubs_found + macros_found + known_shared == len(sig)per overlay.
[HIGH] tools/family_remap.py :: symbol_map (prefix chosen from the RELOC KIND: pfx = "func_" if ke == "call" else "D_") — fed by reloc_targets, which labels every lui/%lo pair "data"
- candidates 7 / parsed 0 / real skips 7
- evidence: Over-approximating oracle = splat's own .s ground truth.
grep -rhoE '%lo\(func_[0-9A-Fa-f]+\)' asm/ov_SC01_077/nonmatchings/= 7 occurrences: a FUNCTION's address taken via lui/%lo (address-taken callbacks), e.g. asm/ov_SC01_077/nonmatchings/ov_SC01_077_jr_80178D40/func_8017F290.s%lo(func_8017E1D4), .../func_8017D98C.s%lo(func_8017DA08), .../ov_SC01_077_jr_801734BC/func_80173A60.s%lo(func_80173B4C). reloc_targets emits kind='data' for all of them, so symbol_map names them D_ — but the exemplar's C writesfunc_<ADDR>. apply_remap's word-bounded rx therefore matches NOTHING and the substitution silently no-ops. 0 of the 7 are named correctly. - blast radius: Lost MATCHES, live TODAY. Measured over the manifest's non-phantom matched families: 3 families (ov_SC01_077 exemplars 0x8017D840, 0x8017DCEC, 0x8017DD9C) x 37 still-stubbed members produce a symbol_map containing a D_-keyed FUNCTION address that the C body writes as func_. e.g. ov_SC01_077 0x8017DCEC -> ov_SC01_080 0x8017ED48: map has
D_8017DE84 -> D_8017EEE0while the unit saysfunc_8017DE84. The staged sibling body keeps the EXEMPLAR's function pointer, the whole-binary gate rejects it, and the loss is booked as a "byte failure" — indistinguishable from a genuine compiler wall. It also burns a bisect slot in every chunk it lands in. - fix: Name the symbol by WHAT THE ADDRESS IS, not by how it was loaded. In symbol_map, load the overlay's sig once (the set of function addrs) and use
pfx = "func_" if (ke == "call" or ae in func_addrs) else "D_". Belt-and-braces (and cheaper): for a data-kind reloc emit BOTH keys into the table —m[f"D_{ae:08X}"] = f"D_{at:08X}"ANDm[f"func_{ae:08X}"] = f"func_{at:08X}"— addresses are unique, so only the token that actually appears in the C can ever match, and apply_remap is a single simultaneous pass so the extra key is free. - assertion to add: After apply_remap, assert that NO per-overlay exemplar address survives in the output:
for ex_addr in symbol_map keys: assert not re.search(rf'\b(func_|D_){ex_addr:08X}\b', remapped)— a remap that leaves an exemplar-only symbol in the sibling body is a tool failure, not a byte failure, and must fail loud rather than be handed to the gate.
[MEDIUM] tools/family_remap.py :: gather_externs (^\sextern\b[^\n;{}]\b{sym}\b[^\n;{}]*; over the exemplar TU — single-line only)
- candidates 199 / parsed 190 / real skips 9
- evidence: Over-approximating detector = for every func_/D_ ref in a matched exemplar's unit that the unit does not itself declare, is it declared ANYWHERE at file scope in the TU (any line ending in ';', excluding assignments/calls)? M=199 across the matched-exemplar corpus; gather_externs carries N=190 extern lines. All 9 gap items are the same shape: a MULTI-LINE comma-separated extern. src/ov_SC01_077/ov_SC01_077.c:271-272 —
extern unsigned char D_801DAA78, D_801DAA79, D_801DAA7A, D_801DAA7B, D_801DAA7C,\n D_801DAA7D, D_801DAA7E, D_801DAA7F, D_801DAA80;. The[^\n;{}]*class cannot cross the newline, so line 271 has no ';' and line 272 has noextern— the ENTIRE group of 9 symbols is invisible, including the ones on the first line. - blast radius: Lost MATCHES. The exemplar that references them, ov_SC01_077 func_8013D178, is a band=substantial family with 133 members. Its remapped siblings are staged with NO declaration for D_801DAA78..D_801DAA80, so every sibling TU fails to compile (
D_801DAA7D undeclared), which fails/bisect-storms the whole (overlay,split) gate group it is chunked with — so the damage is not even confined to the one function. Booked as a compile failure, i.e. invisible. - fix: Make the extern scan statement-oriented rather than line-oriented. Pre-join the TU's file-scope declarations by scanning from each
^\s*extern\bto the next unbraced ';' (a small hand loop, or regex with re.S:^\s*extern\b[^;{}]*?;with re.M|re.S), then test\b{sym}\bagainst each joined statement and carry the joined statement verbatim (newlines preserved — they are legal C). The same single-line assumption exists in dedup_propagate.reconcile_caller_extern's rx ([\w \t\*]*?) and in find_site's preceding-extern walk (^\s*extern\b.*;\s*(/\*.*\*/\s*)?$) — fix all three from one shared helper. - assertion to add:
assert every func_/D_ symbol referenced in the emitted draft is either declared in the draft, declared in the target TU, or resident/common.h— i.e. gather_externs should return a second valueunresolvedand family_sweep should refuse (loudly, with the symbol names) rather than stage a body with a symbol it could not find a declaration for.
[MEDIUM] tools/family_sweep.py :: stub_map (INCLUDE_ASM("([^"]+)",\s*(func_[0-9A-Fa-f]+)))
- candidates 59226 / parsed 59123 / real skips 3
- evidence: Over-approximating detector = every line in src/ov_/.c containing the token INCLUDE_ASM: 59,226. stub_map's regex extracts 59,123. The 103-item gap classified in full: 3 = comment lines merely mentioning INCLUDE_ASM (justified); 100 = REAL stubs whose symbol is NOT named func_XXXXXXXX —
INCLUDE_ASM("asm/ov_SC01_005/nonmatchings/ov_SC01_005_jr_80178D40", listCdBuffer);(src/ov_SC01_005/ov_SC01_005_jr_80178D40.c:4444, and 99 more overlays). 97 of those 100 are JUSTIFIED (0x80180000 is not a function in those overlays' sigs, so the address-keyed engine never asks about it). 3 are REAL SKIPS: in ov_SC03_094, ov_SC06_008 and ov_SC02_041 the sig DOES contain a function at 0x80180000, and it is stubbed under the name listCdBuffer — so stub_map reports "not a stub", and family_hseq therefore classifies it as a MATCHED exemplar. - blast radius: Corrupt METRICS + lost MATCHES. Three PHANTOM matched-exemplar families are created (ov_SC03_094 0x80180000 band=mid with 19 members, ov_SC06_008 with 7, ov_SC02_041 with 2 = 28 members). extract_unit returns None for all of them, so every run of the sweep re-nominates them, produces nothing, and books 28 silent skips; the "matched exemplars" headline count is inflated. Symmetrically, if such a function ever became a real sweep TARGET,
to_addr not in stubs[ov]would silently classify it as already-matched and skip it. - fix: Widen the symbol group to any C identifier and resolve it to an address via the overlay's symbol table:
INCLUDE_ASM\("([^"]+)",\s*([A-Za-z_]\w*)\), thenaddr = int(m[1].split("_")[1],16) if m[1].startswith(("func_","D_")) else symbols[ov][m[1]](config/symbols..txt already haslistCdBuffer = 0x80180000;). Skip only names that resolve to no address, and COUNT them. - assertion to add:
assert len(stub_map(ov)) + n_named_skipped == text.count("INCLUDE_ASM") - n_comment_linesper overlay, and separatelyassert extract_unit(ov,addr) is not Nonefor every addr family_hseq is about to label MATCHED (the two checks together make a named stub impossible to mistake for a match).