BFM-decomp/phase-ends/logs/Phase22.md

CURRENT PHASE — Phase 22: Hand-grind the giants (×134) + idiom-distill-then-fan-out flywheel

Generation: Gen2 (14th phase of the arc; Phase 14 deferred to Gen3+) · Started: 2026-06-26 · Effort: Max (giants/synthesis) ↔ Ultracode (idiom waves, prompt per R27) · Plan approved: gate 1 ✅ (Drew)

Per-task crash-recovery log (P3). The full approved plan is at ~/.claude/plans/plan-mode-enabled-max-shimmering-thimble.md. This file is the committable state a fresh session resumes from.

Goal

Drive the byte-weighted progress number (~30% decomp.dev-comparable; the function-count metric is at 63.18%) by hand-grinding the giants (reach-134, >150 ins — each banks ×134), distilling each crack into a reusable idiom, then fanning that idiom out across its class via the existing wave harness. Open-ended; monotonic; zero regressions; close at a clean checkpoint when giant ROI drops.

Decisions (Drew, gate 1)

• Calibrate on a mid-giant first: func_8015126C (254 ins / 135K, already a close=2 backlog near-miss) → func_8014EE14 (248 / 131K, close=6) → escalate toward func_80144B9C (770 / 410K).
• Background grinder ON: token-free permuter daemon on the regalloc/coalescing residual, alongside the hand-loop.

The operating loop

A. Hand-decomp next giant (Max) → §27 recipe → match_one → harvest_verify byte-gate → dedup_propagate ×134 → commit. ROI-gate: irreducible class → bank closest draft to backlog (P9), move on. B. Distill the last-mile idiom → cookbook (R16/R30). C. If general → fan out a few waves (Ultracode, prompt R26/R27) applying it → gate_stage → distill.js → backlog. D. Rinse and repeat (next giant, highest byte-weight first).

Tasks

• T0 — Unified byte-weighted ranked worklist (done — commit:0296) — tools/worklist.py → docs/worklist.md + .run/worklist.json.
• T1 — Calibrate on func_8015126C (done — banked ×134, fleet 63.18→63.22%, check-all 136/136) — pin idiom + canonical-extern/block-scope recovery cracked it; all 23 giants confirmed Ghidra-C-cached (no prefetch needed).
• T2 — Hand-decomp giants by byte-weight (loop) — func_8014EE14 → … → func_80144B9C; one at a time, byte-gated, propagated ×134, committed. Report each (P3).
• T3 — Distill idioms → cookbook (done for func_8015126C — cookbook §28/§28a + tools/recover_giant.py, commit:0298) — recurring per future crack.
• T4 — Idiom fan-out waves (Ultracode) — a few waves per learned idiom over its class (prompt R27). (close=0 canonical-extern "wave" was probed → R14 refuted near-free; only func_8015126C banked.)
• T5 — Background grinder (started + STOPPED for handoff; STOP sentinel set; banked 0 on the hard giants it sampled) — RESUME: rm .run/auto/STOP then DRIVER=tools/grinder.py setsid nohup bash tools/auto_supervisor.sh --permute-secs 120 -j 14 >/dev/null 2>&1 & disown; monitor bash tools/auto_status.sh.
• T6 — Progress honesty + PhaseEnd — keep worklist/backlog/progress.fleet.md fresh; track both %s; PhaseEnd_Phase22.md at close (Tier-1 Max).

▶ RESUME HERE (fresh session)

State: Phase 22 in progress (NOT a phase end — no PhaseEnd file). 3 commits this session on top of Phase-21 close commit:0295: commit:0296 (T0 worklist) · commit:0297 (T1 func_8015126C ×134) · commit:0298 (T3 cookbook §28 + recover_giant.py) + a reports-regen checkpoint commit. All 136 binaries byte-identical (make check-all 136/136); fleet 63.22% function-count; dedup-check 1617/0. Working tree clean except the R23 db.*.gbf churn (do NOT stage). Drew commits AND pushes the session work (R6/R8). Grinder STOPPED (STOP sentinel set).

NEXT TASK — T2: hand-decomp the next giants by byte-weight (effort Max; the loop is A hand-crack → B distill → C fan-out). Use docs/worklist.md (the GIANT queue, refreshed) as the decision spine. Concrete queue:

1. func_80132784 (400 ins, #2 byte-weight giant; backlog close=240: prologue+GTE pipelines match, residual = else-branch $s0 pointer regalloc + GTE-section stack-ptr allocation → §17 pins). The biggest reachable hand target.
2. The struct-walled close=0 trio func_80156B74 (214), func_8014F74C (174), func_80163C2C (167): match_one MATCH but blocked by a local struct S8/B8 colliding with the TU (§28 case 3). Lever: lift S8/B8 to src/shared/engine_types.h (or reuse the existing def / rename), then tools/recover_giant.py + gate. Could bank ~3 giants if the type-lift resolves it.
3. func_80178004 (165): saved draft regressed (DIFF 91) — re-derive from cached Ghidra-C (it had register-pins-per-sibling-func_80177EA4 keys).
4. decomp.wiki levers to try (§28a): func_801412A8 (198, offset-fold-vs-advance residual) → the negative-struct-offset-in-loop idiom (for(...; p++)); the branch-duplication lever for call-crossing regalloc swaps.
5. After each bank: dedup_propagate ×134 (BACKGROUND), make check-all, commit, distill any new idiom (T3), then optionally a fan-out wave (T4, prompt for Ultracode R27).

Reusable this session: tools/recover_giant.py (canonical-extern recovery for pure-extern close=0 giants); tools/worklist.py --refresh (the ranked decision table); calib drafts in .run/drafts-t1-calib6/. Op gotchas (cookbook §28): run dedup_propagate in BACKGROUND; git checkout src/ does NOT revert config/dedup.us.yaml (reset both on a redo). Pending tooling polish: gate_stage.py commit message hardcodes "phase-21" → make phase-agnostic before the T4 worker waves.

Verification invariant (every bank)

make check-all → 136/136 byte-identical from a CLEAN tree (R22); dedup-check validated, 0 failed; 0 NON_MATCHING (G4); byte-gate (harvest_verify) is the sole arbiter (G3/P9); each bank = checkpoint commit; the db.*.gbf churn is R23 restart-noise (do NOT stage).

Reuse (no rewrites)

orchestrator / wave_targets / worker_wave.js / distill.js / gate_stage / idiom_loop / grinder + supervisor / harvest_verify / match_one / dedup_propagate / build_fuel_manifest / backlog; cookbook §17–§27 + hand-matching-process.md + automation-runbook.md.

Guardrails

• Don't grind confirmed-irreducible classes (store-vs-load sched, hoist-vs-remat, IV-combine, hoisted-invariant order, narrow-param loose-typing, cross-jump merge) — ROI-gate, bank closest, move on.
• Actor struct is byte-NEUTRAL for matching (comprehension only).
• R27 effort transitions: prompt Drew to toggle Max↔Ultracode at every A/B↔C boundary; never launch a Workflow on a verbal yes.

Blockers

(none)

Progress log

• 2026-06-26: Phase plan approved (gate 1). Task list built (R28). CURRENT_PHASE.md written. Starting T0.
• 2026-06-26: T0 done (commit commit:0296) — tools/worklist.py + docs/worklist.md + .run/worklist.json. 451 live stubs / 1.85M ins remaining gain; GIANT queue = 23 fns = 37.2% of remaining gain; top = func_80144B9C (770 ins, 5.58%).
• 2026-06-26: T1 calibration on func_8015126C (254 ins, top-5 giant) — CRACKED, byte-identical in ov_SC01_077; ×134 propagation running (bg bl73feigi). Key findings (→ T3 distill):
  • (idiom) §17 pin for the coalescing residual: the (s16)p[0x79]!=1000 compare wanted $a0 (coalesced) not $v1. Lever: register s32 cmp79 __asm__("$4") + assign inside the && (lazy, not hoisted) → MATCH. Reusable for the regalloc-coalescing giant group.
  • (THE close=0 wall, byte-proven) why match_one MATCH ≠ whole-binary bank: the saved draft's self-contained file-scope externs conflict with engine_core.h's canonical sigs (e.g. my extern void func_8015173C(void*) vs canonical void func_8015173C(s32*)). cast_call_sites/sig_unify do NOT canonicalize these → the close=0 giants sit unbanked. Fix (deterministic): rewrite each engine_core.h-callee extern to its canonical def-sig AND move ALL externs block-scope (inside the body) so find_site/compiles_standalone/dedup_propagate accept them (file-scope externs are excluded from the lifted body → "not self-contained"). This is a missing gate-stage recovery step.
  • (gotcha) saved best_drafts get clobbered by later worse attempts (func_8015126C's close=0 winner was overwritten by a close=2/case-broken copy). Trust the gate, re-derive from match_one.
  • (gotcha) propagation is slow (134 overlay builds) — run dedup_propagate in the background, not a 2-min foreground (a SIGTERM leaves a partial non-atomic state: macro + instantiations applied, registry unwritten).
  • (T4 hypothesis) the other close=0 giants (func_80156B74 214, func_8014F74C 174, func_80163C2C 167, func_80178004 165) likely unblock with the SAME canonical-extern + block-scope recovery → a deterministic near-free wave.
• 2026-06-26: T1 CLOSED — func_8015126C banked ×134, fleet 63.18%→63.22%, make check-all 136/136 byte-identical, dedup-check 1617 validated/0 failed. The full loop is proven end-to-end (hand-crack → bank → ×134 propagate → fleet-verify). Calib draft saved at .run/drafts-t1-calib6/. Reusable canonical-extern recovery: extract each engine_core.h callee's def-sig, rewrite the draft's extern to it, move ALL externs block-scope; then harvest_verify + dedup_propagate (run propagate in BACKGROUND — foreground gets killed; and git checkout src/ does NOT revert config/dedup.us.yaml, so reset BOTH on a redo).
• 2026-06-26: Drew Q — decomp.wiki/compilers/GCC patterns: ~6 useful for us (PS1 gcc-2.7.2-psx). Top for giants: negative-struct-offset-in-loops (for(...;ptr++) → directly addresses func_801412A8's offset-fold residual), branch-invariant code duplication (alt to §17 pins for call-crossing regalloc swaps), load-coalescing (if(t->a||t->b)→lw). Also: div magic-constant table, s16/s8 div-by-2, gcc-2.7.2.x slti …,0. N/A to PS1: bnel-likely (MIPS II+), .lit4 NOPs (PS2), C++ bool (C only). → fold the 3 top into cookbook in T3.
• 2026-06-26: close=0-giant fan-out probed — R14 REFUTES "near-free" (only func_8015126C was pure-extern). Of the 5 close=0 giants: func_8015126C #1 pure-extern (BANKED); func_80156B74/func_8014F74C/func_80163C2C #3 STRUCT-walled (local struct S8/B8 collide w/ TU — need type-lift); func_8014F74C also #2 masked-residual (§27, permuter); func_80178004 #4 regressed draft (DIFF 91, re-derive). The canonical-extern lever is real but the group is NOT uniformly near-free. T3 captured: cookbook §28 (the 4-way close=0 triage + the canonical-extern recovery + the coalescing pin) + §28a (decomp.wiki PS1 patterns) + tools/recover_giant.py (promoted). Net giant strategy unchanged: genuine per-giant hand-work (T2), grinder for the masked/coalescing tail (T5).
• 2026-06-26: Session paused at a clean checkpoint (Drew). T0/T1/T3 committed; T5 grinder started then STOPPED for handoff (banked 0). Reports regenerated. Tree clean (only R23 db churn). Resume from the ▶ RESUME HERE block → T2 giants. Grinder is OFF (STOP sentinel) until a fresh session restarts it.
• 2026-06-26 (new session, effort xHigh — Drew testing xHigh vs Max on giants): tasklist rebuilt (R28); T2/T3/T4 reframed as a recurring cycle (Drew). T2 giant #1 BANKED: func_80156B74 (214 ins) ×134 — the struct-walled close=0 trio's lever proven. Method (→ cookbook §28b): lifted the S8/B8 typedefs from ov_SC01_077.c's prelude block to src/shared/engine_types.h (shared fleet-wide; byte-neutral verified on resident + 2 overlays), removed the dup typedefs, recover_giant.py (block-scope externs) → whole-binary harvest_verify MATCH → dedup_propagate ×134. Fleet 63.22% → 63.25% (function-count; +28,676 ins byte-weighted), make check-all 136/136, dedup-check 1618/0 failed, 0 NON_MATCHING. Gotcha hit + fixed: a concurrent make job (check-all racing dedup_propagate) corrupted an .o AND dropped the propagation's registry write — re-ran dedup_propagate (idempotent) to register; lesson: serialize all make jobs (cookbook §28b-6). NEXT: func_80163C2C (167, Blk16/Buf32 — same type-lift lever; recovered draft staged in .run/drafts-t2/).
• 2026-06-26 (xHigh): T2 giant #2 BANKED: func_80163C2C (167 ins) ×134 — same type-lift lever. Lifted Blk16(u32×4) + Buf32(2×Blk16) to engine_types.h, removed the Blk16 dupes from ov_SC01_077.c AND ov_SC01_077_a.c (the -O0 split), byte-neutral verified → recover_giant → harvest_verify MATCH → dedup_propagate ×134. Fleet 63.25% → 63.29% (+22,378 ins byte-weighted), make check-all 136/136, dedup-check 1619/0 failed. Struct-walled close=0 trio DONE (func_80156B74 ✓, func_80163C2C ✓; func_8014F74C is masked-residual → grinder fuel, not type-liftable). Tool quirk (recurring, benign): dedup_propagate's first full run applies source + registers but its per-overlay byte-gate output didn't appear in-log; the authoritative gate is the post-bank make check-all (136/136). NEXT: restart grinder (T5) on permuter-class giants; hand-decomp func_80132784 (400, close=240, §17 pins GTE-section regalloc — the big one).
• 2026-06-26 (xHigh): func_80132784 HARD-DEFER (R14, byte-evidenced) — residual = hoist-vs-remat (irreducible §10) + register-LIFETIME-reuse (target reuses $s0/$s2 for GTE pointers after s0p/s2v die; function-scoped register __asm__ pins can't express this). Too big (400) + far for the grinder. Logged to backlog. Grinder restarted (T5, permuting) then PAUSED for the batch below.
• 2026-06-26 (xHigh): T2 close=0 RECOVERY BATCH — 14 functions ×134 (R14 corrects §26 "exhausted"). Surveyed the 43 still-live close=0 reach-134 fns; recover_giant + whole-binary gate banked 7 (func_80156ECC, func_80147E44, func_8015ADB0, func_801661CC, func_80166054, func_8012CFA8, func_8012A62C); dedup_propagate --auto-from propagated those + 7 pre-existing inline-matched bonus (func_8012C098/F14C/F038/C0EC/E5CC/C750, func_80128ED8) ×134. The other ~33 are the genuine DEF-side/masked wall. Fleet 63.29% → 63.66%, make check-all 136/136 byte-identical, dedup-check 1633/0 failed. Hit the recurring dedup_propagate registry-skip (source propagated, registry partial) → recovered by direct append_groups registration (cookbook §28c; bytes were already correct per check-all). Distilled §28c (close=0 not exhausted + the registry-skip recovery).
• 2026-06-26 (xHigh): from-scratch giant test — func_801372B0 (207) STRUCTURALLY CRACKED but scheduler-walled (Drew chose 'keep hand-decomp at xHigh'). Decompiled from raw Ghidra-C: a 3D-gizmo/compass HUD drawer (SVEC in/out + GsLINE prim, 4 axis projections via ApplyMatrixSV → GsSortLine + func_80137030/178). First draft was structurally PERFECT (206/207 ins, logic 100%); 4 pin iterations placed the regs but the residual is the giant scheduler/regalloc last-mile (10 held callee-saved regs; gcc's list-scheduler orders prologue-saves+materializations differently → pervasive positional diff, ~173). NOT C-steerable to byte-exact; too far for the grinder (173>30). Banked the structural draft to backlog (high-value head-start). xHigh finding (Drew's test): xHigh reached structural match FAST on giants — the wall is gcc determinism (§27/§17), not reasoning depth, so xHigh appears sufficient for this work (the bottleneck isn't depth). Grinder running on the permuter tail. SESSION DELIVERED: 16 functions banked ×134 (2 giants + 14 close=0), fleet 63.22%→63.66%, §28b/§28c, 2 hard giants diagnosed+logged.
• 2026-06-27: Grinder STOPPED (clean, rc=0) + ROI audited (Drew asked for the total). All-time grinder banks across 11 runs = 7 functions — ALL during Phase 21 (~06-22, fleet 61-62% when tractable close≤30 fuel existed); 0 banks since (late Phase 21 + all of Phase 22). Efficiency: 62 "permuter WON" events but only 7 banks (~11%) — the rest were gate-rejected (masked-residual §25) or pure CHURN (func_8014F3E8 won ~22×, func_8014FE60 ~18× across runs, never banked AND never blacklisted). Two grinder BUGS found (fix before next use): (1) split-file-blind — it looks for .s only in asm/ov_SC01_077/nonmatchings/ov_SC01_077/, NOT the _a/_o0 split subdirs → this session it errored on 100% of targets (all _a-region); (2) churn-without-blacklist on permuter-won-but-gate-rejected-for-non-plumbing fns. Net: the grinder is a real-but-modest token-free contributor (7 all-time), now exhausted on the hard tail — keep for future fresh tractable fuel only after the 2 fixes. T5 is OFF (STOP sentinel set).