BFM-decomp/phase-ends/logs/Phase18.md

CURRENT_PHASE — Phase 18: Compiler-quirk research (raise the match-% ceiling)

Started: 2026-06-20 · Effort: Max · Plan: approved (gate 1, Drew) · Generation: Gen2 (10th phase) Goal: understand gcc-2.7.2's blocking codegen quirks → turn them into cookbook-ready C-shape idioms (or honest "unsteerable" verdicts), byte-gated. Goal metric = match-%, not comprehension.

Crash-recovery log (P3/CURRENT_PHASE). The full approved plan is the canonical reference: ~/.claude/plans/plan-mode-enabled-max-serene-engelbart.md. Mirrored task list in the harness (R28).

🚨 NEW SESSION START HERE

The T0–T5 "regalloc-order is UNSTEERABLE" framing below is SUPERSEDED — see ⭐ VERDICT REVERSAL (register pinning works) and the ▶ STEP-3b RUNBOOK near the bottom. You are resuming the 3 → 1 → 2 plan at Step 3b with /effort ultracode ON (Drew enables it for this session). Read cookbook §17 (the corrected matching TOOLKIT — the recipe the wave agents must apply) before doing anything. Then jump to the RUNBOOK and author the upgraded wave script. MCP/Ghidra NOT needed (cached Ghidra-C + asm on disk).

Gate-1 decisions (Drew, 2026-06-20)

• Scope = research-focused. Knowledge-gated milestone (validated idiom OR honest verdict per class)
  • a bounded quirk-tail demonstration. The full tractable-247 harvest wave is deferred to Phase 19.
• Native compiler = bounded check (W1): Wine + real CC1PSX.EXE/ASPSX.EXE, time-boxed, parallel, droppable. Not on the critical path.

The finding that reshapes the phase (verified vs the bytes this session)

The two "unsteerable" classes (§10 rematerialize-vs-hoist; $s0/$s1 regalloc-swap) already have documented C-level levers (cookbook §10 Residual A/B/B3, committed in the LZSS close src/800.c ~443–571). Phase 17 used the wrong oracle (the permuter's floor-polluted score) — §10:518 says use the object-level metric (match_one.py), which was never applied to the exemplars. §16 ("not source-steerable") contradicts §10 — Phase 18 reconciles it. So: existing-knowledge-first — replay §10 under match_one BEFORE any new gcc-source reading.

Tasks (checkboxes mirror the harness task list)

• T0 — Setup + per-class exemplar set ✓ 2026-06-20. Cloned gcc-papermario (a6afc2af, source at repo root) + xenogears-decomp (f27c0768) → tools/reference/ (gitignored; SETUP.md §5.6, R20/R21). Apparatus confirmed: match_one cleanly measures the §10 residual on both quirk exemplars — the oracle Phase 17 failed to use. Exemplar .s live in any extracted overlay (shared fns): func_8012B8E4 → asm/ov_SC03_103/…, func_8012B4B8 → asm/ov_SC03_099/…. Strip-restore positive control deferred to T4 (where p16_known_answer --gate = the real arbiter). Baselines (T1 starting point):
  • func_8012B8E4 (regalloc-swap, §10 Residual A): 75=75 ins, 24 mismatched — systematic $s0↔$s1 allocation swap (mine s1=arg1/s0=s1ang; target s0=arg1/s1=s1ang) + the idx-29..34 reassociation.
  • func_8012B4B8 (hoist-remat, §10 Residual B): 94 vs 84 target, 88 mismatched — mine spills an extra callee-saved reg (s2) + over-emits the mtx=D_800AE620 struct-copy / arg0+0x20 reload. Harder.
• T1 — DECISIVE GATE ✓ 2026-06-20. Replayed §10 under match_one on func_8012B8E4 (full detail in .run/p18/T1_findings.md). §3-T4 branch-polarity invert closed 24→21 (a real §10-family win the permuter couldn't measure). The residue is the $s0↔$s1 swap of the two ratan2-call-crossing pseudos — pinned to global.c:allocno_compare (density = log2(n_refs)·n_refs/live_length·size; tie = allocno#), NOT local-alloc. Ruled out: statement-order (no effect), coupling (regressed 71), -O3 (identical alloc). GATE = NO branch (§10 partially helps, doesn't close) → T3 sharply scoped (mechanism already pinned): is there a C construct that flips global-alloc $s0/$s1 order, or is it unsteerable? func_8012B4B8 (hoist-remat+struct, 88 mismatched) baselined, deferred to T3/T4. Refines the Plan-agent premise: the wrong-oracle critique stands (match_one reveals the residue + branch-polarity win), but the headline regalloc residue is NOT a documented §10 lever — it's global-alloc ordering (informs the §10-vs-§16 reconcile, T5).
• T2 — Xenogears mine ✓ 2026-06-20 (2 bg agents; full synthesis .run/p18/T2_T3_synthesis.md). DECISIVE: Xenogears (independent decomp, IDENTICAL gcc-2.7.2-psx -O2) has NO C lever for the call-crossing $s0/$s1 ORDER class — no register, no asm pins, no permuter; they ship it as INCLUDE_ASM (1174 nonmatch). Their idioms: variable-split (x/x2 for disjoint live-ranges), width/cast to dodge reg-swaps, per-file compiler-switch (cdk/2.6.0/-O3 — tested here, doesn't help these). Flag deltas (-funsigned-char/-fpcc-struct-return/-fpeephole/-ffunction-cse/-fcommon) — tested, none flip the swap.
• T3 — gcc source verdict ✓ 2026-06-20. Regalloc-order class = UNSTEERABLE from C (global.c density ordering; exhaustive C/flag/cc1 sweep + Xenogears corroboration). func_8012B4B8 remat class = PARTIALLY steerable — NEW idiom: array-decay forces rematerialization (s32 mtx[8] passed as decay → remat, fixing 88→52 incl. the hard regalloc; &struct/mtx.w/*(M8*) → hoist). Bankable; this fn has a struct-copy conflict so doesn't fully close.
• W1 — Build-divergence ruled out (by proxy; full Wine run dropped) ✓ 2026-06-20. Plan escape clause invoked: Wine is a heavy install on this WSL (106 pkgs + i386 arch not enabled + wineprefix) and the divergence question is already closed by stronger evidence: (1) gcc-2.7.2-psx byte-matches ~700 functions, many with call-crossing callee-saved values → its global register allocation IS faithful to the original compiler (a divergent real CC1PSX would have broken those matches) → the real compiler would emit the SAME unsteerable swap; (2) the cdk cc1 (a sibling 2.7.2 build) was tested directly — diverges but is WORSE (32 vs 21 on func_8012B8E4), so cc1-build switching doesn't crack it; (3) Xenogears (real-era toolchain) ships the class as asm. Verdict: no exploitable build divergence. Definitive CC1PSX.EXE run deferred (low marginal value; binaries staged at tools/psyq/psyq4.0/, Drew can opt in). SETUP §4.8 unchanged (Wine still not installed).
• T4 — Idioms byte-validated ✓ 2026-06-20 (folded into T1/T3/T6). for-loop + statement-order (§2-T2) + sig_unify → byte-gated to a full MATCH (func_801399A8, 136/136). branch-polarity (§3-T4): match_one byte-evidence (24→21). array-decay-forces-remat: match_one byte-evidence (88→52, fixes the hard regalloc+ remat). regalloc-order: byte-proven UNSTEERABLE (exhaustive sweep + Xenogears). Per-class verdicts set.
• T5 — Cookbook distilled + reconciled ✓ 2026-06-20. New §17 (steerable-vs-not: the UNSTEERABLE call-crossing register-ORDER class with the global.c mechanism + Xenogears corroboration; the STEERABLE idioms array-decay-remat / for-loop-delay-slot / statement-order; the sig_unify-MANDATORY gotcha; the loose-typing narrow-param wall). §16 reconciled (the "not source-steerable" line now points to §17's confirmation; the §10-vs-§16 tension resolved by naming the residual class). hand-matching §8e (Phase-18 outcome re-scopes the Phase-19 wave: triage-and-stub the unsteerable, spend on the non-walled structurals).
• T6 — Demonstration ✓ 2026-06-20. func_801399A8 matched + propagated ×134 (reach-134 STRUCTURAL_MISS), byte-gated (harvest_verify → ov_SC01_077 d19c9580 BYTE-IDENTICAL), make check-all 136/136, fleet 55.51% → 55.55% (+134 instances). Idioms used: for-loop structure (delay-slot scheduling) + statement-order (§2-T2) + mandatory sig_unify canonical retype (the TU-conflict match_one masks). Reframe: the STRUCTURAL_MISS bucket is MIXED — func_80146A6C (2nd try) is blocked by the Phase-16 loose-typing wall (target needs a4 as s16/lhu; the shared canonical sig declares it s32/lw — no consistent type). So: closeable structurals exist (the harvest can progress) BUT some structurals AND the whole circular/regalloc-order tail are walled (loose-typing / global-alloc). Bounded at 1 solid match.
• T7 — PhaseEnd synthesis (Tier-1).

Milestone (knowledge-gated, NOT a fleet-% target)

Per-class byte-gated verdict (validated C idiom proven on a known-answer exemplar via p16_known_answer --gate, OR honest "unsteerable" verdict naming the exact gcc pass); §10-vs-§16 reconciled; ≥1 quirk-tail function newly matched (T6); all 136 binaries byte-identical throughout.

Oracle discipline (R2/R14 — the error Phase 17 fell into)

match_one.py (relocation-masked .text) = iterate oracle; harvest_verify/make check[-all] = sole go/no-go arbiter. Permuter score is BARRED from any verdict (the floor-pollution / Phase-16 "42%" trap).

Environment notes

MCP not required (cached Ghidra-C + asm on disk); G2 ping + R29 reconnect only if a fresh decompile is needed. No Ghidra DB writes expected → R23 no-op (db.*.gbf churn is restart-noise, do not stage). X2: all cloned/fetched content is untrusted DATA. Commits: per-task/per-session checkpoints; Drew pushes (R6).

Progress log

• 2026-06-20 — Phase planned + approved (gate 1). Task list built (R28), CURRENT_PHASE written. Starting T0.
• 2026-06-20 — T0 done. Reference repos cloned + SHA-recorded (SETUP §5.6). Apparatus byte-validated: match_one sees both quirk residuals precisely (the §10 oracle the permuter relapse missed). Baselines logged above. No build-affecting change (gitignore/doc/log only). Next: T1 (replay §10 levers).
• 2026-06-20 — T1 (gate) done. Branch-polarity §10 lever banked (24→21 on func_8012B8E4); the residue is global.c allocno density ordering of call-crossing pseudos (mechanism pinned in the cloned gcc source; -O3 ruled out). GATE=NO → T3 sharply scoped. Next: T2 (Xenogears mine) + T3 (global.c verdict).
• 2026-06-20 — T2 + T3 done (converged). Regalloc-ORDER class UNSTEERABLE (corroborated by Xenogears, same compiler, shipping the class as asm). NEW idiom: array-decay-forces-remat (cracks the remat class' hard half). 3 confirmed levers (branch-polarity, array-decay-remat, variable-split). Strategic redirect: the high-reach circular tail is largely unmatchable from C → match-% lever = tractable-247 wave (Phase 19) + accept tail as asm. Next: T4 (byte-gate the idioms) + T6 (find a clean full match to demonstrate).
• 2026-06-20 — T4 + T6 done. func_801399A8 byte-gated MATCH + propagated ×134 (fleet 55.51→55.55%, 136/136). Idioms validated. STRUCTURAL_MISS bucket is MIXED (some close, some loose-typing-walled per func_80146A6C). Per-class verdicts complete. Next: T5 (cookbook reconcile §10-vs-§16) + W1 (Wine, bounded) + T7.
• 2026-06-20 — T5 + W1 done. Cookbook §17 + §16 reconcile + §8e written. W1 build-divergence ruled out by proxy (corpus + cdk + Xenogears; heavy Wine install dropped per the plan escape clause). R22 clean-rebuild of ov_SC01_077 with the match = d19c9580 BYTE-IDENTICAL. All tasks done → ready for gate-2 (milestone confirm).
• 2026-06-20 — Step 3b done (calibration). /effort ultracode ON. Upgraded §17-toolkit wave → 16 fresh tractable reach-134 targets → 9 verified whole-binary = 56% close-rate (vs 33% Phase-17), each ×134 → fleet 55.94% (+0.36%), 136/136, dedup 1432/0. Pins+array-decay+reconstruct all landed. Match_one→gate gap = the §3a call-site casts agents skip (+ sig_unify narrowing); recovered 3 by casts. Committed commit:0152.
• 2026-06-20 — Step 1 done (bounded wave + recovery). Drew chose bounded ~40 (full-247→Phase 19). Wave with embedded canonical callee sigs + call-site-cast/re-validate → match_one 28/31 (90%), 22 verified whole-binary, fleet 56.64% (+0.70%), 136/136, dedup 1450/0. Recovery: func_8012F68C parse-fix + func_8012A418 sibling-decl reconcile. Committed commit:0154+commit:0155. 6 fns + the -O0 class deferred to Phase 19. Session arc: 55.58% → 56.64% (+1.06%, ~31 reach-134 matches). Next: Step 2 (cookbook §17 polish + gate-2 + PhaseEnd) — needs Max.

⭐ VERDICT REVERSAL (2026-06-20, post-gate-2 — Drew's "we hand-match everything" directive)

The T1-T5 "regalloc-order = UNSTEERABLE" verdict was WRONG (P9/R14 self-correction — I'd skipped the most direct lever). Explicit register pinning works: register s32 d __asm__("$16") forces the $s0/$s1 allocation gcc otherwise density-orders the other way. func_8012B8E4 (the flagship "unsteerable" fn): 21 → MATCH via pins + branch-polarity + clamp temps + a scheduling barrier __asm__ __volatile__("":: "r"(u5)); byte-gated + propagated ×134. The circular tail IS hand-matchable (labor-intensive, ~5-10 min/fn, ×134 each). Cookbook §17 + §8e CORRECTED to the register-pinning TOOLKIT. Fleet 55.51 → 55.58% (2 reach-134 hand-matches: func_801399A8 structural + func_8012B8E4 pinned), 136/136.

Plan (Drew, 2026-06-20): 3 → 1 → 2

• Step 3a — validate the pin recipe (task #10): func_8012B8E4 byte-gated (the hard swap case) + func_801399A8 (structural). The recipe is a TOOLKIT (most fns need reconstruction, not pins; swaps are rare). Sufficient to teach; the WAVE validates breadth.
• Step 3b — teach the wave + calibrate ✓ 2026-06-20. Upgraded wave (.run/harvest_wave_3b.js) teaches the §17 TOOLKIT (triage match_one diff → reconstruct / array-decay / register-pins+barrier; mandatory canonical-sig). Calibration: 16 fresh tractable reach-134 targets → 9 verified whole-binary = 56% close-rate (vs Phase-17's 33%). Moves used: 2 pins (func_8012CB64, func_801468C8), 2 array-decay (func_80149544/584), 5 reconstruct. Propagated ×134 → fleet 55.58% → 55.94% (+0.36%), 136/136 byte-identical, dedup 1432 validated / 0 failed. Committed commit:0152. KEY FINDING (R16 → feeds Step 1): the match_one→gate gap (12 match_one MATCH → 6 gate, recovered to 9) is dominated by the §3a call-site CASTS the agents skip (arity-cast when a callee is called with more args than its canonical extern; fn-ptr cast when a void-canonical callee's $v0 is used) + sig_unify narrowing the extern to canonical turning a K&R no-proto into a hard conflict. Fix recovered 3 by codegen-neutral call-site casts. Step-1 prompt MUST add: apply call-site arity/void casts (NOT redeclaration), and RE-RUN match_one AFTER the canonical retype (sig_unify can both expose conflicts and, rarely, change codegen — func_80137614 regressed 26→25 when forced void/s32). Residual classes seen: narrow-param loose-typing wall (func_801466F0, func_80137614 def-sig — stub), §10 cross-jump/layout (func_80131CF4, func_8012F5F4 1-off), stale M2C_UNK func_X() sibling-decl conflict (func_8012A418 — fixable by reconciling the sibling's forward decl), base-draft-not-final (func_80131CA8 — re-draft). 1 transient API-Overloaded error (func_8012C2D0 — retriable).
• Step 1 — bounded wave + recovery ✓ 2026-06-20 (Drew chose bounded ~40, full-247 → Phase 19). Step-1 wave (.run/harvest_wave_p18s1.js, 31 tractable reach-134 targets, +embedded canonical callee sigs +call-site-cast/re-validate): match_one 28/31 (90%), whole-binary 22 verified (18 propagated ×134 + 4 local REAL: func_80168664/AD0C/BA10 membership<134, func_8012A418 local-type body). Moves: 7 pins, 5 barrier, 1 array-decay, 2 call-site-cast, the rest reconstruct. Fleet 55.94% → 56.64% (+0.70%), 136/136, dedup 1450/0. Committed commit:0154 (20) + commit:0155 (recovery +2). Deferred to Phase 19 (documented): 3 TU-integration cases (func_80147514/8017209C/80168F40 — standalone-MATCH, implicit-int caller plumbing); 3 codegen DIFFs (func_80131CA8/80150528/8014E5B4); the per-file -O0 class (func_8013C360 + ~17 siblings — agent's C is byte-exact at -O0; needs an -O0 split file like src/boot.c → ~18 fns ×134, HIGH ROI); 2 genuine residuals (func_8014F2E0 §10 store-vs-load sched; func_8012C2D0 NEW gcc loop-guard operand-order class).
• Step 2 — §17 polish + PhaseEnd close (task #9): cookbook §17 polish (call-site-cast/re-validate + sibling-decl/implicit-int recovery + the -O0 + loop-guard residual classes + measured close-rates), gate-2 re-confirm, PhaseEnd (Tier-1). Effort: switch back to Max (deep synthesis, not breadth — R27).

▶ STEP-3b RUNBOOK (everything the new session needs to act)

Immediate first action: author an upgraded wave script (start from .run/harvest_wave_s4.js) whose per-target agent prompt teaches the §17 TOOLKIT, then run a SMALL calibration (~10-20 targets) to measure the new whole-binary close-rate before scaling (probe-before-investing). /effort ultracode is ON → launch via the Workflow tool. Pause+prompt Drew before a big scale-up (R27).

The recipe to teach agents (cookbook §17 — read it; this is the summary): reconstruct each fn from .run/ghidra_c/<fn>.c (cached Ghidra-C, 300 fns) + its target .s, then triage the match_one diff:

• pure structure / arg-forwarding wrapper → clean reconstruction (match field + statement order; for-loop for counted scans gets the delay-slot schedule; §2-T2 statement order for adjacent independent ops).
• a stack buffer passed to a callee → array-decay: declare T buf[N], pass buf (NOT &buf/buf.w/ *(T*)buf) → gcc rematerializes the address instead of burning a callee-saved reg.
• match_one shows a call-crossing register SWAP ($s0/$s1 etc. transposed) → register PINS: read the target .s, map each call-surviving value → its callee-saved reg ($s0=$16,$s1=$17,$s2=$18…), declare register s32 v __asm__("$16");. Then mop up with branch-polarity (§3-T4) + explicit temps for any reassociation + a scheduling barrier __asm__ __volatile__("" : : "r"(v)); for a last stuck instruction.
• MANDATORY before the gate (the gotcha that makes match_one lie): retype the draft to the CANONICAL decls in src/shared/engine_core.h — the fn's own return type, every callee's sig, AND data-extern types — using integer address arithmetic (s32)&sym (codegen-neutral). match_one MASKS relocations so a wrong type still "MATCHES"; the whole-binary gate then fails conflicting types. Grep engine_core.h for the canonical decl.
• STUB-and-skip ONLY the narrow-param loose-typing conflict (an arg that must be s16 here / s32 at another call site — func_80146A6C). Permuter CANNOT help pinned fns (pycparser rejects register __asm__).

The gate pipeline (per function): draft → canonical-retype → match_one (fast iterate) → harvest_verify (byte-gate = sole arbiter, G3/P9) → if MATCH, dedup_propagate ×134.

Exact commands (ov_SC01_077 is the harvest source; good-sha d19c9580a02dc63ba1f0e7e0c770f3b10de35635):

A=asm/ov_SC01_077/nonmatchings/ov_SC01_077
.venv/bin/python tools/match_one.py <fn> --c <draft.c> --asm-subdir $A            # iterate oracle
.venv/bin/python tools/harvest_verify.py --binary ov_SC01_077 --src src/ov_SC01_077/ov_SC01_077.c \
    --asm-subdir $A --out build/ov_SC01_077/ov_SC01_077 \
    --good-sha d19c9580a02dc63ba1f0e7e0c770f3b10de35635 --drafts <draftdir>      # byte-gate (arbiter)
.venv/bin/python tools/dedup_propagate.py --addr 0x<ADDR> --source-overlay ov_SC01_077   # ×134, per-overlay gated
make check-all      # 136/136 byte-identical   ·   make report → docs/progress.fleet.md (fleet %)

(A match_one standalone diff harness with configurable cc1/flags is at .run/p18/mo.py if needed.)

Worked examples (durable + committed): src/shared/engine_core.h → DEFINE_func_8012B8E4 (the pins + barrier recipe) and DEFINE_func_801399A8 (the for-loop/structural recipe). Scratch originals: .run/p18/variants/b8e4_pin8.c, .run/drafts-p18-t6/func_801399A8.c. Give agents these two as templates.

Staged assets: .run/harvest_wave_s4.js (the wave script to upgrade — 40 targets), .run/probe_targets_s4.json (bucketed: STRUCTURAL_MISS 25 / PERMUTER_CLASS 12 / VOID_VALUE_MISUSE 2 / SIG_FIXABLE_KR 1 — NOTE most are single-$s0, i.e. reconstruction not pins; true $s0/$s1 swaps are rarer), .run/ghidra_c/ (300 cached Ghidra-C), .run/p18/T1_findings.md + .run/p18/T2_T3_synthesis.md (the analysis). Regen Ghidra-C via tools/ghidra_scripts/DecompileFunctions.java (headless, no /mcp) if a fn isn't cached.

Fleet baseline (this session's close): byte-identical 191,217 / 344,010 = 55.58%, 136/136, ~1424 dedup groups; 2 reach-134 hand-matches banked (func_801399A8 + func_8012B8E4). Tree clean at commit:0150.

After Step 3b (calibration) → Step 1 (scale the wave on the circular tail + hand-match the residue, each ×134) → Step 2 (final §17 polish, gate-2 re-confirm with the wave's match-% delta, PhaseEnd_Phase18.md = Tier-1, archive CURRENT_PHASE → logs/Phase18.md R19). The milestone is already MET+EXCEEDED (per-class verdicts done; the headline class is now STEERABLE; 2 byte-gated matches) — Step 1 banks additional match-%.

Blockers

• Step 3b needs Ultracode — Claude cannot toggle effort (R27). Resolved by Drew enabling /effort ultracode in the new session (his stated plan). If a <system-reminder> does NOT confirm ultracode is on, prompt first.