diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 2605ed600..7da6eb920 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -1974,3 +1974,30 @@ deep frame-RE is a dead end — only the permuter explores that space.** Net for (cheap-recovery, permuter, giant-wave, hand-finish, deep-RE) are byte-proven exhausted at fleet 63.17%** for the reach-134 tail; the residual is the gcc-2.7.2 scheduler/regalloc wall, addressable only by the (low-yield) permuter or by accepting it as the matching ceiling at this fleet level. + +## §28 — Banking a "close=0 gate-rejected" giant: the canonical-extern recovery (Phase 22 T1, byte-proven on func_8015126C ×134) + +Phase-21 (§25/§27 cont.7d) logged a class of giants as "close=0 (match_one MATCH) but gate-REJECTED" and treated them ALL as masked-residual/permuter fuel. **That was incomplete (R14).** A close=0-gate-rejected giant is one of FOUR distinct walls — and one is deterministically bankable BY HAND: + +1. **PURE-EXTERN PLUMBING (bankable — the NEW lever).** The draft's self-contained **file-scope** externs conflict with engine_core.h's canonical decls (the giant calls already-matched engine fns; the draft guessed their sigs, e.g. `extern void func_8015173C(void*)` vs canonical `void func_8015173C(s32*)`). match_one's isolated compile (own externs) MATCHes; the whole-binary TU fails to compile (`conflicting types for func_X`). `sig_unify`/`cast_call_sites` do NOT canonicalize these (the gap that stranded them). **Fix = `tools/recover_giant.py`:** for each callee with a `DEFINE_func_X()` in engine_core.h, rewrite the draft's extern to that macro's exact def-sig; then move ALL externs (callee + `D_` data) **block-scope** (inside the body, after `{`). `find_site`/`compiles_standalone`/`dedup_propagate` lift the body + its *internal* externs as one unit — **file-scope externs are excluded from the lifted body → false "not self-contained" skip.** Forward-refs (higher-addr callees) MUST stay declared. Then `harvest_verify` → `dedup_propagate` ×134. +2. **MASKED RESIDUAL (permuter fuel).** Compiles whole-binary but bytes differ — the relocation-mask hid a real codegen diff (func_8014F74C, §25). The recovery won't help → grinder. +3. **STRUCT-WALLED (type reconciliation).** The draft uses a local named `struct S8`/`B8` (the array-of-struct %lo idiom, §18) that collides with the TU's other defs (`conflicting types for S8`) — func_80156B74/func_8014F74C/func_80163C2C. Needs the type lifted to `engine_types.h` (named once) or rewritten anonymous/raw-cast. Not near-free. +4. **REGRESSED DRAFT.** The saved best_draft was clobbered by a later worse attempt (func_80178004: worklist close=0 but saved draft is DIFF 91). Re-derive. + +**Triage:** `recover_giant.py` → `match_one` (DIFF → #4) → whole-binary build. byte-identical → #1 banked · `conflicting types for func_X` → was-#1, recovery fixes · `conflicting types for ` → #3 · compiles but bytes differ → #2. **Honest yield:** of the 5 close=0 giants, only func_8015126C was #1 (pure-extern); the rest are #2/#3/#4 — the canonical-extern lever is real but the close=0-giant group is NOT uniformly near-free. + +**The coalescing pin (extends §25 — func_8015126C's last mile).** The `(s16)p[0x79] != 1000` compare temp wanted `$a0` (coalesced with the soon-to-be-angle arg); gcc gave `$v1`. An eager named temp forces `$a0` but HOISTS the load (+95 ins). Fix — pin AND keep the load lazy inside the `&&`: +```c +register s32 cmp __asm__("$4"); /* $a0 */ +if (cond1 && ((cmp = *(s16*)(p + 0x79)) != 1000)) { ... } +``` +The in-`&&` assignment stays inside the short-circuit (not hoisted) yet lands in the pinned reg. Reusable for any coalesced-compare-temp residual. + +**Op gotchas (Phase 22 T1):** run `dedup_propagate` in the **BACKGROUND** (134 builds > the 2-min foreground cap; a SIGTERM/interrupt leaves a non-atomic partial state — macro+instantiations applied, registry unwritten). And `git checkout src/` does **NOT** revert `config/dedup.us.yaml`, so on a redo reset BOTH (`git checkout src/ config/dedup.us.yaml`) or `registered_addrs()` stays dirty and re-skips the function as "already shared." + +### §28a — decomp.wiki GCC patterns worth trying on BFM giants (decomp.wiki/compilers/GCC, raw at decompals/decompedia; PS1-applicable subset) +- **Negative struct offsets in loops** — `for (i=…; …; i++, p++)` makes gcc *advance* the pointer + use negative member offsets instead of offset-folding. Directly targets func_801412A8's "OFFSET-FOLDS the 4× prim stores instead of ADVANCING $t6" residual. +- **Branch-invariant code duplication** — when gcc hoists a shared tail (a call) out of two branches and swaps regs, *duplicating that tail inside both branches* fixes the regalloc. An alternative to §17 pins for the call-crossing swap class. +- **Load coalescing** — adjacent struct fields compared together (`if (t->a || t->b)`) fold into one `lw` (+lui/ori/and mask when <4 bytes). Recognition aid for engine code. +- **div-by-constant magic table** (0x66666667→/10, 0x55555556→/3, …) + **s16/s8 div-by-2** sign-extension forms; **gcc-2.7.2.x** `slti …,0` ⇒ `(x & (1<<31)) != 0`. +- **N/A to PS1** (don't chase): `bnel`/likely branches (MIPS II+; R3000 has none), `.lit4` float-literal NOPs (PS2), C++ `bool` load/store (BFM is C). diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 80f987c28..e1b5f65fd 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -54,3 +54,4 @@ orchestrator / wave_targets / worker_wave.js / distill.js / gate_stage / idiom_l - **(T4 hypothesis)** the other close=0 giants (func_80156B74 214, func_8014F74C 174, func_80163C2C 167, func_80178004 165) likely unblock with the SAME canonical-extern + block-scope recovery → a deterministic near-free wave. - 2026-06-26: **T1 CLOSED — func_8015126C banked ×134, fleet 63.18%→63.22%, `make check-all` 136/136 byte-identical, dedup-check 1617 validated/0 failed.** The full loop is proven end-to-end (hand-crack → bank → ×134 propagate → fleet-verify). Calib draft saved at `.run/drafts-t1-calib6/`. Reusable canonical-extern recovery: extract each engine_core.h callee's def-sig, rewrite the draft's extern to it, move ALL externs block-scope; then harvest_verify + dedup_propagate (run propagate in BACKGROUND — foreground gets killed; and `git checkout src/` does NOT revert `config/dedup.us.yaml`, so reset BOTH on a redo). - 2026-06-26: Drew Q — decomp.wiki/compilers/GCC patterns: ~6 useful for us (PS1 gcc-2.7.2-psx). Top for giants: **negative-struct-offset-in-loops** (`for(...;ptr++)` → directly addresses func_801412A8's offset-fold residual), **branch-invariant code duplication** (alt to §17 pins for call-crossing regalloc swaps), **load-coalescing** (`if(t->a||t->b)`→`lw`). Also: div magic-constant table, s16/s8 div-by-2, gcc-2.7.2.x `slti …,0`. N/A to PS1: `bnel`-likely (MIPS II+), `.lit4` NOPs (PS2), C++ bool (C only). → fold the 3 top into cookbook in T3. +- 2026-06-26: **close=0-giant fan-out probed — R14 REFUTES "near-free" (only func_8015126C was pure-extern).** Of the 5 close=0 giants: func_8015126C #1 pure-extern (BANKED); func_80156B74/func_8014F74C/func_80163C2C #3 STRUCT-walled (local `struct S8`/`B8` collide w/ TU — need type-lift); func_8014F74C also #2 masked-residual (§27, permuter); func_80178004 #4 regressed draft (DIFF 91, re-derive). The canonical-extern lever is real but the group is NOT uniformly near-free. **T3 captured:** cookbook §28 (the 4-way close=0 triage + the canonical-extern recovery + the coalescing pin) + §28a (decomp.wiki PS1 patterns) + `tools/recover_giant.py` (promoted). Net giant strategy unchanged: genuine per-giant hand-work (T2), grinder for the masked/coalescing tail (T5). diff --git a/tools/recover_giant.py b/tools/recover_giant.py new file mode 100644 index 000000000..7482bddf8 --- /dev/null +++ b/tools/recover_giant.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""recover_giant.py — the canonical-extern recovery for a "close=0 gate-rejected" giant (cookbook §28). + +A giant draft that is match_one-MATCH but fails the WHOLE-binary gate with `conflicting types for +func_X` is a PURE-EXTERN PLUMBING wall: its self-contained file-scope externs guessed sigs that +clash with engine_core.h's canonical decls of the same callees (the giant calls already-matched +engine functions). sig_unify/cast_call_sites don't canonicalize these. This tool does: + + 1. strip // comments + scalar-typedef redefinitions; uppercase func_ to the splat convention. + 2. for each callee that has a `DEFINE_func_X()` in engine_core.h, rewrite the draft's extern to + that macro's exact definition signature (`extern func_X();`). + 3. move ALL externs (callee + D_ data) BLOCK-scope, just inside the function body's opening brace, + so find_site/compiles_standalone/dedup_propagate lift the body + its internal externs as ONE + unit (file-scope externs are excluded from the lifted body -> false "not self-contained" skip). + +Then gate it: `harvest_verify` (banks whole-binary, the sole arbiter G3/P9) -> `dedup_propagate` ×134. +This ONLY fixes the pure-extern-plumbing wall (§28 case 1). A struct-type collision (`conflicting +types for S8`, case 3), a masked residual (compiles but wrong bytes, case 2 -> permuter), or a +regressed draft (match_one DIFF, case 4 -> re-derive) are NOT addressed here — the gate decides. + +Usage: tools/recover_giant.py +""" +import re, sys, os + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +EC = os.path.join(REPO, "src/shared/engine_core.h") + + +def canon_sig(name, ec): + """The canonical definition signature ` func_X()` from DEFINE_func_X in engine_core.h.""" + m = re.search(r'#define DEFINE_%s\(\)(.*?)\{' % re.escape(name), ec, re.S) + if not m: + return None + for raw in m.group(1).split('\\\n'): + line = raw.strip().rstrip('\\').strip() + if not line or line.startswith('extern') or line.startswith('#'): + continue + mm = re.match(r'([A-Za-z_].*\b%s\s*\(.*\))\s*\{?\s*$' % re.escape(name), line) + if mm: + return mm.group(1).strip() + return None + + +def recover(fn, src, ec): + s = re.sub(r'^[ \t]*//.*\n', '', src, flags=re.M) + s = re.sub(r'^[ \t]*typedef\b.*\b(u8|u16|u32|u64|s8|s16|s32|s64|f32|f64)[ \t]*;[ \t]*\n', '', s, flags=re.M) + s = re.sub(r'func_([0-9A-Fa-f]+)', lambda m: 'func_' + m.group(1).upper(), s) + lines = s.split('\n') + externs = [l for l in lines if l.strip().startswith('extern')] + rest = [l for l in lines if not l.strip().startswith('extern')] + + def canon_extern(line): + m = re.search(r'\b(func_[0-9A-F]+)\b', line) + if not m: + return line.strip() # D_ data extern — keep as-is + cs = canon_sig(m.group(1), ec) + return ('extern ' + cs + ';') if cs else line.strip() + + externs = [canon_extern(l) for l in externs] + di = next((i for i, l in enumerate(rest) + if re.match(r'^[A-Za-z_][\w \*]*\b' + re.escape(fn) + r'\s*\(', l) and ';' not in l), None) + if di is None: + raise SystemExit(f"recover_giant: could not find the definition of {fn} in the draft") + bi = di + while bi < len(rest) and '{' not in rest[bi]: + bi += 1 + new = rest[:bi + 1] + [' ' + e for e in externs] + rest[bi + 1:] + return '\n'.join(new), len(externs) + + +def main(): + if len(sys.argv) != 4: + raise SystemExit(__doc__) + fn, inp, outp = sys.argv[1], sys.argv[2], sys.argv[3] + out, n = recover(fn, open(inp).read(), open(EC).read()) + os.makedirs(os.path.dirname(outp) or ".", exist_ok=True) + open(outp, 'w').write(out) + print(f"recovered {fn}: {n} externs canonicalized + block-scoped -> {outp}") + + +if __name__ == "__main__": + main()