docs(phase-22): T3 — distill §28 canonical-extern recovery + recover_giant.py + decomp.wiki patterns

- cookbook §28: the 4-way triage of a close=0 gate-rejected giant (pure-extern plumbing /
  masked-residual / struct-walled / regressed-draft) + the canonical-extern recovery (the NEW
  lever that banked func_8015126C, which Phase-21 left as a close=0 wall) + the coalescing pin
  (register __asm__ + lazy in-&& assign). §28a: PS1-applicable decomp.wiki GCC patterns
  (negative-offset loops, branch-duplication, load-coalescing, div-magic table, slti-0)
- tools/recover_giant.py: canonicalize a draft's engine_core.h-callee externs to def-sigs +
  move all externs block-scope (validated func_8015126C -> MATCH)
- R14: close=0-giant fan-out NOT uniformly near-free — only func_8015126C was pure-extern;
  the rest are struct-walled / masked-residual / regressed -> genuine per-giant hand-work
This commit is contained in:
Drew T
2026-06-26 03:06:54 -06:00
parent 75dc869099
commit 0375cd529b
3 changed files with 110 additions and 0 deletions
+27
View File
@@ -1974,3 +1974,30 @@ deep frame-RE is a dead end — only the permuter explores that space.** Net for
(cheap-recovery, permuter, giant-wave, hand-finish, deep-RE) are byte-proven exhausted at fleet 63.17%** for the
reach-134 tail; the residual is the gcc-2.7.2 scheduler/regalloc wall, addressable only by the (low-yield) permuter
or by accepting it as the matching ceiling at this fleet level.
## §28 — Banking a "close=0 gate-rejected" giant: the canonical-extern recovery (Phase 22 T1, byte-proven on func_8015126C ×134)
Phase-21 (§25/§27 cont.7d) logged a class of giants as "close=0 (match_one MATCH) but gate-REJECTED" and treated them ALL as masked-residual/permuter fuel. **That was incomplete (R14).** A close=0-gate-rejected giant is one of FOUR distinct walls — and one is deterministically bankable BY HAND:
1. **PURE-EXTERN PLUMBING (bankable — the NEW lever).** The draft's self-contained **file-scope** externs conflict with engine_core.h's canonical decls (the giant calls already-matched engine fns; the draft guessed their sigs, e.g. `extern void func_8015173C(void*)` vs canonical `void func_8015173C(s32*)`). match_one's isolated compile (own externs) MATCHes; the whole-binary TU fails to compile (`conflicting types for func_X`). `sig_unify`/`cast_call_sites` do NOT canonicalize these (the gap that stranded them). **Fix = `tools/recover_giant.py`:** for each callee with a `DEFINE_func_X()` in engine_core.h, rewrite the draft's extern to that macro's exact def-sig; then move ALL externs (callee + `D_` data) **block-scope** (inside the body, after `{`). `find_site`/`compiles_standalone`/`dedup_propagate` lift the body + its *internal* externs as one unit — **file-scope externs are excluded from the lifted body → false "not self-contained" skip.** Forward-refs (higher-addr callees) MUST stay declared. Then `harvest_verify` → `dedup_propagate` ×134.
2. **MASKED RESIDUAL (permuter fuel).** Compiles whole-binary but bytes differ — the relocation-mask hid a real codegen diff (func_8014F74C, §25). The recovery won't help → grinder.
3. **STRUCT-WALLED (type reconciliation).** The draft uses a local named `struct S8`/`B8` (the array-of-struct %lo idiom, §18) that collides with the TU's other defs (`conflicting types for S8`) — func_80156B74/func_8014F74C/func_80163C2C. Needs the type lifted to `engine_types.h` (named once) or rewritten anonymous/raw-cast. Not near-free.
4. **REGRESSED DRAFT.** The saved best_draft was clobbered by a later worse attempt (func_80178004: worklist close=0 but saved draft is DIFF 91). Re-derive.
**Triage:** `recover_giant.py` → `match_one` (DIFF → #4) → whole-binary build. byte-identical → #1 banked · `conflicting types for func_X` → was-#1, recovery fixes · `conflicting types for <Type>` → #3 · compiles but bytes differ → #2. **Honest yield:** of the 5 close=0 giants, only func_8015126C was #1 (pure-extern); the rest are #2/#3/#4 — the canonical-extern lever is real but the close=0-giant group is NOT uniformly near-free.
**The coalescing pin (extends §25 — func_8015126C's last mile).** The `(s16)p[0x79] != 1000` compare temp wanted `$a0` (coalesced with the soon-to-be-angle arg); gcc gave `$v1`. An eager named temp forces `$a0` but HOISTS the load (+95 ins). Fix — pin AND keep the load lazy inside the `&&`:
```c
register s32 cmp __asm__("$4"); /* $a0 */
if (cond1 && ((cmp = *(s16*)(p + 0x79)) != 1000)) { ... }
```
The in-`&&` assignment stays inside the short-circuit (not hoisted) yet lands in the pinned reg. Reusable for any coalesced-compare-temp residual.
**Op gotchas (Phase 22 T1):** run `dedup_propagate` in the **BACKGROUND** (134 builds > the 2-min foreground cap; a SIGTERM/interrupt leaves a non-atomic partial state — macro+instantiations applied, registry unwritten). And `git checkout src/` does **NOT** revert `config/dedup.us.yaml`, so on a redo reset BOTH (`git checkout src/ config/dedup.us.yaml`) or `registered_addrs()` stays dirty and re-skips the function as "already shared."
### §28a — decomp.wiki GCC patterns worth trying on BFM giants (decomp.wiki/compilers/GCC, raw at decompals/decompedia; PS1-applicable subset)
- **Negative struct offsets in loops** — `for (i=…; …; i++, p++)` makes gcc *advance* the pointer + use negative member offsets instead of offset-folding. Directly targets func_801412A8's "OFFSET-FOLDS the 4× prim stores instead of ADVANCING $t6" residual.
- **Branch-invariant code duplication** — when gcc hoists a shared tail (a call) out of two branches and swaps regs, *duplicating that tail inside both branches* fixes the regalloc. An alternative to §17 pins for the call-crossing swap class.
- **Load coalescing** — adjacent struct fields compared together (`if (t->a || t->b)`) fold into one `lw` (+lui/ori/and mask when <4 bytes). Recognition aid for engine code.
- **div-by-constant magic table** (0x66666667→/10, 0x55555556→/3, …) + **s16/s8 div-by-2** sign-extension forms; **gcc-2.7.2.x** `slti …,0` ⇒ `(x & (1<<31)) != 0`.
- **N/A to PS1** (don't chase): `bnel`/likely branches (MIPS II+; R3000 has none), `.lit4` float-literal NOPs (PS2), C++ `bool` load/store (BFM is C).
+1
View File
@@ -54,3 +54,4 @@ orchestrator / wave_targets / worker_wave.js / distill.js / gate_stage / idiom_l
- **(T4 hypothesis)** the other close=0 giants (func_80156B74 214, func_8014F74C 174, func_80163C2C 167, func_80178004 165) likely unblock with the SAME canonical-extern + block-scope recovery → a deterministic near-free wave.
- 2026-06-26: **T1 CLOSED — func_8015126C banked ×134, fleet 63.18%→63.22%, `make check-all` 136/136 byte-identical, dedup-check 1617 validated/0 failed.** The full loop is proven end-to-end (hand-crack → bank → ×134 propagate → fleet-verify). Calib draft saved at `.run/drafts-t1-calib6/`. Reusable canonical-extern recovery: extract each engine_core.h callee's def-sig, rewrite the draft's extern to it, move ALL externs block-scope; then harvest_verify + dedup_propagate (run propagate in BACKGROUND — foreground gets killed; and `git checkout src/` does NOT revert `config/dedup.us.yaml`, so reset BOTH on a redo).
- 2026-06-26: Drew Q — decomp.wiki/compilers/GCC patterns: ~6 useful for us (PS1 gcc-2.7.2-psx). Top for giants: **negative-struct-offset-in-loops** (`for(...;ptr++)` → directly addresses func_801412A8's offset-fold residual), **branch-invariant code duplication** (alt to §17 pins for call-crossing regalloc swaps), **load-coalescing** (`if(t->a||t->b)`→`lw`). Also: div magic-constant table, s16/s8 div-by-2, gcc-2.7.2.x `slti …,0`. N/A to PS1: `bnel`-likely (MIPS II+), `.lit4` NOPs (PS2), C++ bool (C only). → fold the 3 top into cookbook in T3.
- 2026-06-26: **close=0-giant fan-out probed — R14 REFUTES "near-free" (only func_8015126C was pure-extern).** Of the 5 close=0 giants: func_8015126C #1 pure-extern (BANKED); func_80156B74/func_8014F74C/func_80163C2C #3 STRUCT-walled (local `struct S8`/`B8` collide w/ TU — need type-lift); func_8014F74C also #2 masked-residual (§27, permuter); func_80178004 #4 regressed draft (DIFF 91, re-derive). The canonical-extern lever is real but the group is NOT uniformly near-free. **T3 captured:** cookbook §28 (the 4-way close=0 triage + the canonical-extern recovery + the coalescing pin) + §28a (decomp.wiki PS1 patterns) + `tools/recover_giant.py` (promoted). Net giant strategy unchanged: genuine per-giant hand-work (T2), grinder for the masked/coalescing tail (T5).
+82
View File
@@ -0,0 +1,82 @@
#!/usr/bin/env python3
"""recover_giant.py — the canonical-extern recovery for a "close=0 gate-rejected" giant (cookbook §28).
A giant draft that is match_one-MATCH but fails the WHOLE-binary gate with `conflicting types for
func_X` is a PURE-EXTERN PLUMBING wall: its self-contained file-scope externs guessed sigs that
clash with engine_core.h's canonical decls of the same callees (the giant calls already-matched
engine functions). sig_unify/cast_call_sites don't canonicalize these. This tool does:
1. strip // comments + scalar-typedef redefinitions; uppercase func_<hex> to the splat convention.
2. for each callee that has a `DEFINE_func_X()` in engine_core.h, rewrite the draft's extern to
that macro's exact definition signature (`extern <ret> func_X(<params>);`).
3. move ALL externs (callee + D_ data) BLOCK-scope, just inside the function body's opening brace,
so find_site/compiles_standalone/dedup_propagate lift the body + its internal externs as ONE
unit (file-scope externs are excluded from the lifted body -> false "not self-contained" skip).
Then gate it: `harvest_verify` (banks whole-binary, the sole arbiter G3/P9) -> `dedup_propagate` ×134.
This ONLY fixes the pure-extern-plumbing wall (§28 case 1). A struct-type collision (`conflicting
types for S8`, case 3), a masked residual (compiles but wrong bytes, case 2 -> permuter), or a
regressed draft (match_one DIFF, case 4 -> re-derive) are NOT addressed here — the gate decides.
Usage: tools/recover_giant.py <func_NAME> <in_draft.c> <out_draft.c>
"""
import re, sys, os
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
EC = os.path.join(REPO, "src/shared/engine_core.h")
def canon_sig(name, ec):
"""The canonical definition signature `<ret> func_X(<params>)` from DEFINE_func_X in engine_core.h."""
m = re.search(r'#define DEFINE_%s\(\)(.*?)\{' % re.escape(name), ec, re.S)
if not m:
return None
for raw in m.group(1).split('\\\n'):
line = raw.strip().rstrip('\\').strip()
if not line or line.startswith('extern') or line.startswith('#'):
continue
mm = re.match(r'([A-Za-z_].*\b%s\s*\(.*\))\s*\{?\s*$' % re.escape(name), line)
if mm:
return mm.group(1).strip()
return None
def recover(fn, src, ec):
s = re.sub(r'^[ \t]*//.*\n', '', src, flags=re.M)
s = re.sub(r'^[ \t]*typedef\b.*\b(u8|u16|u32|u64|s8|s16|s32|s64|f32|f64)[ \t]*;[ \t]*\n', '', s, flags=re.M)
s = re.sub(r'func_([0-9A-Fa-f]+)', lambda m: 'func_' + m.group(1).upper(), s)
lines = s.split('\n')
externs = [l for l in lines if l.strip().startswith('extern')]
rest = [l for l in lines if not l.strip().startswith('extern')]
def canon_extern(line):
m = re.search(r'\b(func_[0-9A-F]+)\b', line)
if not m:
return line.strip() # D_ data extern — keep as-is
cs = canon_sig(m.group(1), ec)
return ('extern ' + cs + ';') if cs else line.strip()
externs = [canon_extern(l) for l in externs]
di = next((i for i, l in enumerate(rest)
if re.match(r'^[A-Za-z_][\w \*]*\b' + re.escape(fn) + r'\s*\(', l) and ';' not in l), None)
if di is None:
raise SystemExit(f"recover_giant: could not find the definition of {fn} in the draft")
bi = di
while bi < len(rest) and '{' not in rest[bi]:
bi += 1
new = rest[:bi + 1] + [' ' + e for e in externs] + rest[bi + 1:]
return '\n'.join(new), len(externs)
def main():
if len(sys.argv) != 4:
raise SystemExit(__doc__)
fn, inp, outp = sys.argv[1], sys.argv[2], sys.argv[3]
out, n = recover(fn, open(inp).read(), open(EC).read())
os.makedirs(os.path.dirname(outp) or ".", exist_ok=True)
open(outp, 'w').write(out)
print(f"recovered {fn}: {n} externs canonicalized + block-scoped -> {outp}")
if __name__ == "__main__":
main()