diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 7690a17c3..5ce8607a4 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -17830,3 +17830,84 @@ The fleet-wide draw hit the doctrine's 6k target *and* concentrated into two TUs gate-group concentration once the band is wide enough to have choices. **A binary running dry is not the frontier running dry** — it is a signal to widen the binary set, not the instruction band. Widening the band instead buys smaller functions at *more* gate groups, which is the wrong trade twice over. + +--- + +## §181 — WHAT A WAVE'S GATE ACTUALLY REJECTS (P31 S53, measured on wave R's 45-draft main pile) +### Only ONE of 27 blocked drafts was wrong. The other 26 were correct and unbankable. + +Wave R drafted **92/110 MATCH** and its main pile carried 45 byte-verified drafts. **18 banked in the +first slate.** Every rejection was catalogued; five distinct classes, four of which `match_one`, +`reloc_identity`, `fragment_check` and `pregate_check` are ALL structurally blind to: + +| class | n | how it announces itself | who could have caught it | +|---|--:|---|---| +| **MIRROR-FRAGMENT** | 2 | `undefined reference to '.L80050D5C'` at LINK | nothing we had | +| duplicate typedef, TU's copy BELOW | 7 | C89 duplicate-typedef at COMPILE | `pregate_check` did | +| draft-vs-draft data-symbol type clash | 1 | `previous declaration of D_80072784` | nothing we had | +| draft-vs-file signature conflict | 4 | `previous declaration of func_80032A74` | nothing we had | +| **genuinely byte-wrong** | 1 | whole-binary SHA differs | only the build | + +**1. THE MIRROR FRAGMENT — the reverse of §176i, and a new refusal we do not yet compute.** +`fragment_check` asks *"does another symbol live INSIDE my range?"*. The opposite is equally fatal: +another stub's `.s` **branches into a label inside YOUR range**, and converting you to C deletes that +label. Two instances, both silent until the linker spoke: +* `gfx2D_BG0_OBJ_4D8` — `gfx2D_BG0_OBJ_1B4.s` branches to `.L80050D5C` / `.L80050D70` inside it; +* `SYS_OBJ_26EC` — `SYS_OBJ_25C8.s` branches to `.L8005B9B8` inside it. +The test is as cheap as the forward one: for each still-stubbed sibling `.s`, collect the `.L` +targets it references and refuse any draft whose `[addr, addr+4*nins)` contains one. **Shipped as +`fragment_check.branched_into()`**, negative-controlled on both known-bad drafts (it names them and +the exact referring sibling) — and then measured across the corpus, which is the number that matters: + +| binary | stubs owning a branched-into label | +|---|--:| +| `main` | **99 of 1,745 (5.7%)** — incl. `SaveLoadRoutine`, `GsSortFastBg`, the `gfx2D_BG0_*` cluster | +| `ov_SC04_011` | 0 of 229 | +| `ov_SC03_028` | 0 of 194 | + +So it is a **main-specific hazard at ~1 in 18**, and effectively absent from the overlay fleet. That +asymmetry is itself informative: main is where splat's symbol table names the most non-function +addresses. Two notes on the semantics — the check excludes labels a function defines itself, and it +excludes siblings that are IN THE SAME SLATE (converting both at once removes the referencing `.s`, +so the hazard evaporates). A population scan that puts every stub in one slate therefore correctly +reports zero, which is a right answer to a different question. + +**2. THE TYPEDEF-BELOW CASE.** `gate_main.strip_dup_typedefs` reuses a definition only when it is +visible ABOVE the insertion point — correct, since stripping a below-survivor leaves the name +undefined (that trap is documented in its own docstring). But it then KEEPS the draft's copy, and +two definitions of one typedef name is a C89 error wherever they sit. Both horns are wrong; the +missing third option is to rename the draft's private copy, or hoist the file's definition (typedefs +emit no code, so hoisting is byte-neutral). **7 correct drafts are parked on this.** + +**3. THE BISECT ECONOMICS ARE SETTLED.** `bisect_slate.py` — null control first, then true binary +search — isolated the single byte-wrong draft (`SYS_OBJ_1DC0`) in **5 steps / 90 seconds**, at ~13 s +per incremental build. `gate_main`'s built-in bisect on a comparable slate ran **3 hours and named +nothing** (§176i). Never use the built-in one; always pass `--no-bisect` and drive `bisect_slate`. + +**4. THE STRATEGIC READING.** 26 of 27 blocked drafts are byte-correct work that only the *plumbing* +rejects, which is the §180b ratio again from a third independent direction. The lesson is not "draft +better" — the drafting is done. It is that **every hour spent making the integration layer compute a +refusal is worth more than an hour of drafting**, because drafting is already at 84–93%. + +## §182 — §177's HONEST NEGATIVE: the epilogue lever cracked 4 of 16, and the `800c3` cluster held +Wave R ran §177 (the saved-register-set → epilogue-delay-slot law) as a dedicated 16-card repair lane +against pre-classified near-misses — 10 of them the exact `addiu $sp / jr $ra / nop` vs +`jr $ra / addiu $sp` signature, all in `800c3`. Result: **4 MATCH, 10 still at closeness 2–3, 2 +IMMOVABLE.** Every survivor kept its original signature. + +So §177's *mechanism* is source-confirmed and its *lever* ("change what is live across the call") is +not sufficient for this cluster. That is a refutation of the lever's reach, not of the rule. **Do not +re-run this lane as-is** (R38: the verdicts are recorded in `.run/s53_epi_class.json` and the wave +journals). The next probe should ask what ELSE forces those frames — an argument spill, a callee's +clobber set, or a `$s`-register requirement the C shape cannot avoid — before another agent-hour is +spent. A lane that converts 25% is a lane that needs a new hypothesis, not another pass. + +## §180d — THE `pgrep` BRACKET TRICK PROTECTS THE PATTERN, NOT THE COMMAND LINE +S52 banked "`pgrep -f` self-matches its own shell wrapper — use the `[g]ate_main` bracket trick." S53 +found the trick's limit the hard way: a waiter written as +`until ! pgrep -f '[g]ate_lane'; do sleep 20; done; tail -8 .run/gate_lane_sc02.log` +never exits, because its own command line contains the plain string `gate_lane` **in the `tail` +argument**. The bracket only de-fangs the occurrence inside the pattern itself. Either name the file +nowhere else on that line, or match something unforgeable (`pgrep -f 'tools/gate_lane\.py'` from a +wrapper that does not mention the path twice). Cost here: a 46-minute loop against a job that had +already finished, and a false "still running" in two status reports. diff --git a/tools/fragment_check.py b/tools/fragment_check.py index 9f372151c..aa04b44cf 100644 --- a/tools/fragment_check.py +++ b/tools/fragment_check.py @@ -72,6 +72,61 @@ def defines_foreign_symbol(slate, binary='main'): return out +LOCAL_LABEL = re.compile(r'\.L([0-9A-Fa-f]{8})\b') + + +def branched_into(slate, binary='main'): + """Drafts that own a `.L` label ANOTHER still-stubbed function branches to — the MIRROR of the + subsume test, and the class that cost wave R two clean rebuilds (cookbook §181). + + `check()` above asks "does another symbol live inside MY range?". The reverse is just as fatal: + a neighbour's .s contains `bnez $v0, .L8005B9B8` where `.L8005B9B8` is defined inside the + function YOU are converting to C. Convert it and the label ceases to exist, so the neighbour's + still-assembled .s fails at LINK time with `undefined reference to '.L8005B9B8'`. Nothing in the + C-level ladder can see this: the draft is byte-perfect, its symbols are right, and the file it + lands in is textually valid. Only the linker knows, and it charges a full rebuild to say so. + + Measured instances (wave R): `gfx2D_BG0_OBJ_4D8` (branched into by gfx2D_BG0_OBJ_1B4.s at + .L80050D5C/.L80050D70) and `SYS_OBJ_26EC` (by SYS_OBJ_25C8.s at .L8005B9B8). + + Cost: one pass over the binary's still-stubbed .s files, seconds. It is deliberately scoped to + STUBS ONLY -- a label referenced by an already-banked function is not a hazard, because that + function no longer ships an .s that could reference it.""" + stubs = {st.symbol: st for st in corpus.stubs(binary).values()} + slate_names = {e['fn'] for e in slate} + + # every .L target referenced by a stub that is NOT itself being converted in this slate + referenced = {} + for name, st in stubs.items(): + if name in slate_names: + continue + try: + text = open(st.asm_path, errors='replace').read() + except OSError: + continue + defined_here = {m.group(1).upper() for m in re.finditer(r'^\.L([0-9A-Fa-f]{8}):', text, re.M)} + for m in LOCAL_LABEL.finditer(text): + lab = m.group(1).upper() + if lab in defined_here: + continue # a purely internal label, harmless + referenced.setdefault(int(lab, 16), set()).add(name) + + out = [] + for e in slate: + st = stubs.get(e['fn']) + if not st: + continue + n = corpus.s_ins_count(st.asm_path) if hasattr(corpus, 's_ins_count') else None + if not n: + continue + lo, hi = st.addr, st.addr + 4 * n + hits = sorted((a, sorted(who)) for a, who in referenced.items() if lo <= a < hi) + if hits: + out.append({'fn': e['fn'], 'addr': '0x%08X' % lo, + 'labels': [{'label': '.L%08X' % a, 'referenced_by': who} for a, who in hits]}) + return out + + def check(slate, binary='main'): stubs, byaddr = interior_symbols(binary) findings = [] @@ -102,6 +157,7 @@ def main(): slate = json.load(open(a.slate)) bad = check(slate, a.binary) foreign = defines_foreign_symbol(slate, a.binary) + mirror = branched_into(slate, a.binary) print(f'{len(slate)} drafts checked against {a.binary} symbol space') for f in bad: syms = ', '.join(f"{s['symbol']}@{s['addr']}" for s in f['subsumes'][:4]) @@ -110,13 +166,18 @@ def main(): for f in foreign: print(f" [FAIL] {f['fn']} DEFINES another stub's symbol in asm: {', '.join(f['defines'])} " f"— that symbol still has its own .s (assembler: 'already defined')") - if not bad and not foreign: - print(' clean — no draft subsumes or redefines another symbol') + for f in mirror: + first = f['labels'][0] + print(f" [FAIL] {f['fn']} ({f['addr']}) OWNS {first['label']}, which " + f"{', '.join(first['referenced_by'][:3])} branches to — converting it to C deletes " + f"the label (linker: \"undefined reference to `{first['label']}'\")") + if not bad and not foreign and not mirror: + print(' clean — no draft subsumes, redefines, or owns a branched-into label') if a.prune: - names = {f['fn'] for f in bad} | {f['fn'] for f in foreign} + names = {f['fn'] for f in bad} | {f['fn'] for f in foreign} | {f['fn'] for f in mirror} json.dump([e for e in slate if e['fn'] not in names], open(a.prune, 'w'), indent=1) print(f' pruned slate ({len(slate) - len(names)} drafts) -> {a.prune}') - sys.exit(1 if (bad or foreign) else 0) + sys.exit(1 if (bad or foreign or mirror) else 0) if __name__ == '__main__': diff --git a/tools/pregate_check.py b/tools/pregate_check.py index 8f8a1a1a7..5a8e45ddb 100644 --- a/tools/pregate_check.py +++ b/tools/pregate_check.py @@ -98,7 +98,14 @@ def check_text(path, text): findings = [] masked = cdecl._mask(text) # R33: the one comment/string masking oracle depth = _depth_map(masked) - tds = _typedefs(text) + # SCAN THE MASKED TEXT, NEVER THE RAW TEXT (P31 S53, R35). This originally read + # `_typedefs(text)`, so a typedef quoted in a COMMENT counted as a definition: src/800.c's own + # bank note at line 2409 quotes `typedef struct { s32 a; s32 b[4]; } OtBlk_80016450;` inside a + # /* */ block, and the tool reported DUPLICATE-TYPEDEF against a file that has compiled + # byte-identically for weeks. Seven of nine FAILs on a real wave-R slate were comment-borne. + # The masking oracle was already computed one line above and simply was not used here; because + # _mask is length-preserving, every reported offset stays valid. + tds = _typedefs(masked) # 3. duplicate typedef -- ANY redefinition, identical body or not. # C89 has no "compatible redefinition" allowance for typedefs: `typedef struct {...} T;` twice @@ -164,7 +171,7 @@ def check_text(path, text): decls.setdefault(s, (m.start(), sig)) # 5. definition vs a visible prototype - for name, (off, sig) in _func_defs(text).items(): + for name, (off, sig) in _func_defs(masked).items(): # masked, not raw — see the note above sig = _norm_sig(sig) if name in decls and decls[name][1] != sig: # SEVERITY CALIBRATED AGAINST THE COMPILER, not against C89 pedantry. Measured on the