BFM-decomp/phase-ends/logs/Phase19.md

# CURRENT PHASE — Phase 19: Scale the toolkit (wave-harvest at scale + cheap front-loaded levers)

**Started:** 2026-06-20 · **Generation:** Gen2 (11th phase) · **Plan approved (gate 1):** Drew, 2026-06-20
**Effort:** Max (planning/synthesis) · xHigh (T1/T2 execution) · **Ultracode for T3 wave batches** (prompt at the transition, R26/R27)
**Plan file:** `~/.claude/plans/plan-mode-enabled-max-golden-truffle.md`

## Scope decisions (Drew, gate 1)
- **Bounded flywheel** — T1 + T2 + **2–3 wave batches of 50**, bank gains, close at a clean checkpoint with a Phase-20 backlog. (Not open-ended; not infra-only.)
- **gcc-research spike (T4) = CONDITIONAL in-phase** — runs ONLY if the waves plateau on a residual class blocking meaningful reach.

## Baseline at phase start (verified this session)
- Fleet **56.64%** byte-identical-from-source (194,839 / 344,010); 136/136 binaries byte-identical; 1450 dedup groups (0 failed); 0 NON_MATCHING.
- `ov_SC01_077`: 1719/2586 = 66.5%, 867 INCLUDE_ASM stubs remaining; clean-rebuild SHA `d19c9580…`.
- Staged: `.run/harvest_wave_p18s1.js` (120 tractable reach-134 targets; P18 ran 31). Ghidra-C cache `.run/ghidra_c/` = 300 fns (fresh).

## Tasks
- [x] **T1 — Per-file -O0 split-file lever** — DONE (re-scoped by Drew: bank infra + 6, defer rest). The 3-object split works (`ov_SC01_077_a` before / `ov_SC01_077_o0` -O0 / `ov_SC01_077` after — a single object's .text can't be split around a middle object). **6/16 -O0 fns matched** byte-perfect (`func_8013B568/B7F4/BD34/C360/C938/C964`). **DEFERRED to Phase 20:** the 10 remaining -O0 fns hit an indexed-global `%lo`-folding codegen quirk (gcc-source research, R17 — our cc1 materializes the address, the original folds %lo); AND the ×134 per-overlay rollout (engine_core.h compiles -O2, can't carry -O0 fns → each overlay needs its own -O0 split). Two findings corrected the Phase-18 backlog's "afternoon, free ×134" premise (R14).
- [x] **T2 — Recovery tooling (light) — DONE.** R14 finding overturned the Phase-18 "implicit-int / propagate-first" framing: reproducing the 3 deferred near-misses through the gate showed **3 distinct real classes** (none implicit-int). (a) **Resident-callee link-miss** (func_8017209C calls `func_8004CFEC` = curated `ratan2`) → built **`tools/canon_resident_calls.py`** (rewrite `func_<ADDR>`→curated name; pipeline runs it FIRST). (b) **Shared-caller arity conflict** (func_80147514 `s32`, func_80168F40 `void*`): an engine_core.h caller macro declares the callee file-scope `extern void func_X(void)` → `conflicting types`; **FIX = caller extern → no-proto `extern void func_X();`** (byte-neutral, promotion-safe params). `cast_call_sites.py` NOT needed (the §17a "call-site cast = #1 miss" did not appear here; deferred to T3 if real near-misses need it). **3 matches banked** in ov_SC01_077 (byte-gated); 2 no-proto lines in engine_core.h; **`make check-all` 136/136 byte-identical** (R22); dedup-check 1450 validated/0 failed. Recovery pipeline = `draft → canon_resident_calls → sig_unify → harvest_verify --chunk 1` (cookbook §17a corrected, SETUP inventory, R16/R21).
- [ ] **T3 — Scale waves (2–3 batches of 50, Ultracode).** Regen Ghidra-C for fresh targets, `gen_harvest_targets.py` manifest, Step-1 wave prompt → recovery (T2) → gate → `dedup_propagate` ×134 → checkpoint → improve. Each batch = a progress report.
- [ ] **T4 — (CONDITIONAL) gcc-research spike.** Only on a blocking residual class (loop-guard order `func_8012C2D0`; §10 schedule `func_8014F2E0`).
- [ ] **PhaseEnd** — verify all checkboxes (P7), demonstrate milestone, write `PhaseEnd_Phase19.md` (+ `## Plain-English Recap`, R25; Phase-20 backlog), archive worklog (R19).

## Milestone (gate 2, Drew confirms)
-O0 lever banked + recovery tooling proven + toolkit scaled across 2–3 batches of 50, fleet up materially (target +3–5%, ~56.6% → ~60%), 136/136 byte-identical from clean (R22), 0 NON_MATCHING (G4), closed at a clean checkpoint with a Phase-20 backlog.

## Progress log
*(append one line per task as completed)*
- 2026-06-20 — Phase started; plan approved (gate 1); task list built (R28); CURRENT_PHASE.md created. Beginning T1.
- 2026-06-20 — T1: built + proved the 3-object -O0 split infra; matched 6/16 -O0 fns (ov_SC01_077 d19c9580). Two consults with Drew on T1 scope: (1) full -O0 rollout chosen, then (2) on discovering the remaining fns are research-grade (the %lo quirk), Drew chose "bank 6 + infra, defer rest to Phase 20." Fleet unchanged (the 6 are ov_SC01_077-local until rollout). Verified main/resident/ov_SC01_077/+2 byte-identical. Committed (be33b887f + cookbook §18 c8e925842). → T2.
- 2026-06-20 — T2 scoping finding (R14): examined Phase-18's actual near-misses (.run/drafts-p18s1-recover*/) — the recoveries were mostly **def-signature canonicalization** (ret→void, params→void*/int), which **`sig_unify` already does**, + a data-type reconcile (`struct Q16`→`BigCopy`) + a whitespace fix. The §17a "call-site casts = #1 miss" did NOT show up in these. So T2 is lighter than planned: confirm sig_unify is wired draft→`sig_unify`→gate, add the implicit-int **propagate-first** helper (3 fns: func_80147514/80168F40/8017209C → def into engine_core.h, gate after) + a small data-type-reconcile, and treat call-site-casts as a thin add only if T3's real near-misses need it. **SESSION CHECKPOINT here (Drew) — resume fresh.**
- 2026-06-20 — **T2 DONE.** Resumed fresh, rebuilt task list (R28), /effort max. Reproduced the 3 deferred near-misses through the gate (R14) → the "implicit-int / propagate-first" premise was WRONG: real classes are (a) resident-callee link-miss → new `tools/canon_resident_calls.py` (func_8017209C ✓), (b) shared-caller arity conflict → caller-extern no-proto fix (func_80147514 + func_80168F40 ✓). func_8012A418 isn't a stub in ov_SC01_077 (not a target; already Phase-18-reconciled). **3 matches banked**, 2 engine_core.h no-proto lines, **make check-all 136/136 byte-identical** (full extract + R22), dedup 1450/0, fleet 56.64% (194,848 — the 3 are ov_SC01_077-local, ×134 propagation rides T3/Phase-20: func_80147514/F40 inline named structs need engine_types.h lifting, func_8017209C propagatable). Cookbook §17a corrected, SETUP inventory updated. Pipeline now `draft → canon_resident_calls → sig_unify → harvest_verify`. → **T3 (the +3-5% lever).**

- 2026-06-20 — **T3 batch-1 PREPPED, awaiting Ultracode toggle (R27 boundary).** 268 reach-134 tractable (nins≤150) stubs in ov_SC01_077; 91 are cached+eligible (Ghidra-C present, excl. the 2 known-hard func_8012C2D0/func_8014F2E0). Batch 1 = top 50 by leverage, nins 24–88 (`.run/t3_b1_targets.json`); batches 1–2 need NO Ghidra-C regen / NO MCP stop (batch 3 would, R23/R29). Proven Step-1 wave recipe staged (`.run/harvest_wave_p18s1.js`); recovery pipeline now `draft → canon_resident_calls → sig_unify → harvest_verify`. Realistic gain ~+1.2–1.7%/batch (matches×134/344,010). **PAUSED for `/effort ultracode` (R26/R27) — will build the batch-1 wave JS + launch on the toggle.**

- 2026-06-20 — **T3 batch-1 wave LAUNCHED** (Ultracode, Drew toggled; R27 satisfied). 50 agents, `.run/harvest_wave_t3b1.js` (run `wf_372a2738-773`), drafts → `.run/drafts-t3b1/`.
- 2026-06-20 — **T3 batch-1 DONE.** Wave: **44/50 match_one MATCH** (88%); 6 DIFF (genuine gcc-quirk tail: fold-const constant-last, GCSE/reload, loop-IV combine, reg time-sharing, narrow-param — correctly stubbed, no false matches). Whole-binary gate (the arbiter): **30 banked** (28 via the improved 2-stage gate + 2 arity via no-proto). **Two pipeline findings (R14):** (1) sig_unify REGRESSES already-canonical agent drafts → gate **canon-only FIRST, sig_unify FALLBACK** (recovered +5 vs unconditional sig_unify); (2) gen_harvest_targets emits **garbled callee sigs** (use-site `if(...)` exprs) that agents paste as externs → 7 PARSE fails (root-cause fix queued for batch 2). **Propagated 25 ×134** (`dedup_propagate --auto-from`; incl. the T2-deferred func_8017209C; ~5 stayed local — inline named structs). **check-all 136/136 byte-identical** (R22); dedup **1450→1475** validated/0 failed; **fleet 56.64% → 57.62% (+0.98%)**. 14 still-failing (3 agent-garble, 1 struct, 10 genuine codegen near-miss) = batch-2 retry/recovery candidates. Recovery tools used: canon_resident_calls (T2), no-proto caller fix (T2), the 2-stage gate (`.run/t3_gate2.sh`). → commit batch 1, then improve (gen_harvest_targets sigs + fold canon-first into the pipeline), then batch 2.

- 2026-06-20 — **Between-batch improvements + T3 batch-2 LAUNCHED.** Fixed `gen_harvest_targets.py` `INLINE_DEF_RE` (was matching indented `if(func_X(...)){` call-exprs as defs → garbled callee sigs; now column-0 + type-only prefix → **0 garbled** of 749, was 9). Parameterized the canon-first 2-stage gate (`.run/t3_gate2.sh <draftsdir>`). Batch 2 = **38 targets** (`.run/t3_b2_targets.json`, nins 10–88; cache-limited after batch-1's 25 propagated + 19 genuine-hard excluded), incl. the 3 garble-retries (clean hints now). Wave `wf_68f3e807-d75` → `.run/drafts-t3b2/`. On completion: `t3_gate2.sh .run/drafts-t3b2` → recover (no-proto/garble) → `dedup_propagate ×134` → check-all → commit (incl. the gen_harvest_targets fix, uncommitted). Then assess batch 3 (needs Ghidra-C regen, R23/R29) vs close.

- 2026-06-20 — **T3 batch-2 DONE.** Wave: **35/38 match_one MATCH (92%)**, up from 88% (clean hints — the gen_harvest_targets fix worked; **2 of 3 garble-retries matched**). 3 DIFF = genuine (delay-slot schedule, call-crossing hoist-vs-remat, narrow-param STOP). Gate: 17 (2-stage) + **8 via NEW `tools/fix_arity_callers.py`** (the no-proto recovery: **ALL 18 stage failures were the shared-caller arity class** — automated it) = **25 banked**. Remaining 10 = genuine loose-typing conflicts (caller sig ≠ def sig, call-site-cast walls — Phase-16 documented). Propagated **8 ×134** (17 stayed local — inline named structs, need engine_types lifting). check-all **136/136** (R22); dedup 1475→1483; **fleet 57.62% → 57.93% (+0.31%)**. New/changed tools: `fix_arity_callers.py` (no-proto), `gen_harvest_targets.py` (garbled-sig fix), `.run/t3_gate2.sh` (canon-first 2-stage). → commit, then struct-lift to propagate the ~22 local struct matches (batches 1+2), then assess batch 3 vs close.

- 2026-06-20 — **Struct-lift + propagation top-up.** `build_engine_types.py --strip` lifted 48 named types → `engine_types.h` (byte-neutral, ov_SC01_077 d19c9580); propagated **+4** more (incl. the T2/batch-2 struct-locals like func_80147514). **check-all 136/136 (R22); fleet 57.93% → 58.00%.** **KEY FINDING (the phase's real bottleneck): propagation, not matching, now caps the fleet-% yield** — match_one is 88–92% and the gate banks 60–71%, but ~16 banked matches stay ov_SC01_077-LOCAL because their bodies use **typedef'd / anonymous / sibling local types** that `build_engine_types` (named-struct-only) can't lift → `dedup_propagate`'s `compiles_standalone` rejects them. Phase total: **56.64% → 58.00% (+1.36%)**; the +3–5% target was gated by (a) the cache-limited pool (~88 eligible, all run) and (b) this propagation gap. **Phase-20 lever: extend the type-lift to typedefs/local types** → recovers ~16+ deferred local matches ×134 (~+0.6%) for ~0 agent tokens, and lifts every future batch's yield.

## ▶ RESUME HERE (fresh session)
1. Re-read load order + this file. Rebuild the harness task list (R28). `/effort max`.
2. ~~T2~~ **DONE** (committed) — `canon_resident_calls.py` + the no-proto caller-extern fix; 3 matches banked; check-all 136/136. Recovery pipeline = `draft → canon_resident_calls → sig_unify → harvest_verify`.
3. **T3 (the main lever, +3-5%):** regen Ghidra-C for fresh tractable reach-134 fns (`DecompileFunctions.java`, headless — MCP stopped, R23/R29); `gen_harvest_targets.py` manifest; then **PROMPT Drew for `/effort ultracode` (R26/R27) and WAIT** before launching the wave. Run 2-3 batches of 50 (recover→gate→`dedup_propagate` ×134→checkpoint→improve between batches).
4. Then T4 (conditional) / PhaseEnd.
- State after T2: commits be33b887f, c8e925842, dce768a26 (T1) + the T2 commit landed; fleet 56.64% (T1's 6 + T2's 3 are ov_SC01_077-local, not yet fleet-counted — they bank fleet-% only after the ×134 rollout). Next = T3.

## Blockers
*(none)*

## Notes / carried to Phase 20
- **The -O0 lever completion (NEW, from T1):** infra is built + proven in ov_SC01_077 (the 3-object split). Remaining: (a) **match the 10 indexed-global -O0 fns** — blocked by the `%lo`-folding quirk (our cc1 materializes `lui;addiu;addu;sw 0(reg)`; original folds `lui;addu;sw %lo(sym)(reg)`); needs gcc-2.7.2 source research (R17/§17), maybe unsteerable from C. Stubs: func_8013B598/B6A0/B7AC/B83C/BC7C/BCDC/BD74/C08C/C0F8/C414. (b) **The ×134 rollout** — each overlay needs its own -O0 split (uniform offsets 0x13410/0x14834, shared bodies header); engine_core.h can't carry -O0 fns. ~+0.6% fleet when complete. (c) the 2 -O0 outliers (func_80144B9C giant + func_801457A4).
- Most giants (28 reach-134 >150-ins); per-overlay unique remainder (×1); comprehension/emulator actor-field naming; Wine/CC1PSX (parked); public flip (Phase 14, Gen3+).