BFM-decomp/phase-ends/logs/Phase19.md

CURRENT PHASE — Phase 19: Scale the toolkit (wave-harvest at scale + cheap front-loaded levers)

Started: 2026-06-20 · Generation: Gen2 (11th phase) · Plan approved (gate 1): Drew, 2026-06-20 Effort: Max (planning/synthesis) · xHigh (T1/T2 execution) · Ultracode for T3 wave batches (prompt at the transition, R26/R27) Plan file: ~/.claude/plans/plan-mode-enabled-max-golden-truffle.md

Scope decisions (Drew, gate 1)

• Bounded flywheel — T1 + T2 + 2–3 wave batches of 50, bank gains, close at a clean checkpoint with a Phase-20 backlog. (Not open-ended; not infra-only.)
• gcc-research spike (T4) = CONDITIONAL in-phase — runs ONLY if the waves plateau on a residual class blocking meaningful reach.

Baseline at phase start (verified this session)

• Fleet 56.64% byte-identical-from-source (194,839 / 344,010); 136/136 binaries byte-identical; 1450 dedup groups (0 failed); 0 NON_MATCHING.
• ov_SC01_077: 1719/2586 = 66.5%, 867 INCLUDE_ASM stubs remaining; clean-rebuild SHA d19c9580….
• Staged: .run/harvest_wave_p18s1.js (120 tractable reach-134 targets; P18 ran 31). Ghidra-C cache .run/ghidra_c/ = 300 fns (fresh).

Tasks

• T1 — Per-file -O0 split-file lever — DONE (re-scoped by Drew: bank infra + 6, defer rest). The 3-object split works (ov_SC01_077_a before / ov_SC01_077_o0 -O0 / ov_SC01_077 after — a single object's .text can't be split around a middle object). 6/16 -O0 fns matched byte-perfect (func_8013B568/B7F4/BD34/C360/C938/C964). DEFERRED to Phase 20: the 10 remaining -O0 fns hit an indexed-global %lo-folding codegen quirk (gcc-source research, R17 — our cc1 materializes the address, the original folds %lo); AND the ×134 per-overlay rollout (engine_core.h compiles -O2, can't carry -O0 fns → each overlay needs its own -O0 split). Two findings corrected the Phase-18 backlog's "afternoon, free ×134" premise (R14).
• T2 — Recovery tooling (light) — DONE. R14 finding overturned the Phase-18 "implicit-int / propagate-first" framing: reproducing the 3 deferred near-misses through the gate showed 3 distinct real classes (none implicit-int). (a) Resident-callee link-miss (func_8017209C calls func_8004CFEC = curated ratan2) → built tools/canon_resident_calls.py (rewrite func_<ADDR>→curated name; pipeline runs it FIRST). (b) Shared-caller arity conflict (func_80147514 s32, func_80168F40 void*): an engine_core.h caller macro declares the callee file-scope extern void func_X(void) → conflicting types; FIX = caller extern → no-proto extern void func_X(); (byte-neutral, promotion-safe params). cast_call_sites.py NOT needed (the §17a "call-site cast = #1 miss" did not appear here; deferred to T3 if real near-misses need it). 3 matches banked in ov_SC01_077 (byte-gated); 2 no-proto lines in engine_core.h; make check-all 136/136 byte-identical (R22); dedup-check 1450 validated/0 failed. Recovery pipeline = draft → canon_resident_calls → sig_unify → harvest_verify --chunk 1 (cookbook §17a corrected, SETUP inventory, R16/R21).
• T3 — Scale waves (2–3 batches of 50, Ultracode). Regen Ghidra-C for fresh targets, gen_harvest_targets.py manifest, Step-1 wave prompt → recovery (T2) → gate → dedup_propagate ×134 → checkpoint → improve. Each batch = a progress report.
• T4 — (CONDITIONAL) gcc-research spike. Only on a blocking residual class (loop-guard order func_8012C2D0; §10 schedule func_8014F2E0).
• PhaseEnd — verify all checkboxes (P7), demonstrate milestone, write PhaseEnd_Phase19.md (+ ## Plain-English Recap, R25; Phase-20 backlog), archive worklog (R19).

Milestone (gate 2, Drew confirms)

-O0 lever banked + recovery tooling proven + toolkit scaled across 2–3 batches of 50, fleet up materially (target +3–5%, ~56.6% → ~60%), 136/136 byte-identical from clean (R22), 0 NON_MATCHING (G4), closed at a clean checkpoint with a Phase-20 backlog.

Progress log

(append one line per task as completed)

• 2026-06-20 — Phase started; plan approved (gate 1); task list built (R28); CURRENT_PHASE.md created. Beginning T1.

• 2026-06-20 — T1: built + proved the 3-object -O0 split infra; matched 6/16 -O0 fns (ov_SC01_077 d19c9580). Two consults with Drew on T1 scope: (1) full -O0 rollout chosen, then (2) on discovering the remaining fns are research-grade (the %lo quirk), Drew chose "bank 6 + infra, defer rest to Phase 20." Fleet unchanged (the 6 are ov_SC01_077-local until rollout). Verified main/resident/ov_SC01_077/+2 byte-identical. Committed (commit:0158 + cookbook §18 commit:0159). → T2.

• 2026-06-20 — T2 scoping finding (R14): examined Phase-18's actual near-misses (.run/drafts-p18s1-recover*/) — the recoveries were mostly def-signature canonicalization (ret→void, params→void*/int), which sig_unify already does, + a data-type reconcile (struct Q16→BigCopy) + a whitespace fix. The §17a "call-site casts = #1 miss" did NOT show up in these. So T2 is lighter than planned: confirm sig_unify is wired draft→sig_unify→gate, add the implicit-int propagate-first helper (3 fns: func_80147514/80168F40/8017209C → def into engine_core.h, gate after) + a small data-type-reconcile, and treat call-site-casts as a thin add only if T3's real near-misses need it. SESSION CHECKPOINT here (Drew) — resume fresh.

• 2026-06-20 — T2 DONE. Resumed fresh, rebuilt task list (R28), /effort max. Reproduced the 3 deferred near-misses through the gate (R14) → the "implicit-int / propagate-first" premise was WRONG: real classes are (a) resident-callee link-miss → new tools/canon_resident_calls.py (func_8017209C ✓), (b) shared-caller arity conflict → caller-extern no-proto fix (func_80147514 + func_80168F40 ✓). func_8012A418 isn't a stub in ov_SC01_077 (not a target; already Phase-18-reconciled). 3 matches banked, 2 engine_core.h no-proto lines, make check-all 136/136 byte-identical (full extract + R22), dedup 1450/0, fleet 56.64% (194,848 — the 3 are ov_SC01_077-local, ×134 propagation rides T3/Phase-20: func_80147514/F40 inline named structs need engine_types.h lifting, func_8017209C propagatable). Cookbook §17a corrected, SETUP inventory updated. Pipeline now draft → canon_resident_calls → sig_unify → harvest_verify. → T3 (the +3-5% lever).

• 2026-06-20 — T3 batch-1 PREPPED, awaiting Ultracode toggle (R27 boundary). 268 reach-134 tractable (nins≤150) stubs in ov_SC01_077; 91 are cached+eligible (Ghidra-C present, excl. the 2 known-hard func_8012C2D0/func_8014F2E0). Batch 1 = top 50 by leverage, nins 24–88 (.run/t3_b1_targets.json); batches 1–2 need NO Ghidra-C regen / NO MCP stop (batch 3 would, R23/R29). Proven Step-1 wave recipe staged (.run/harvest_wave_p18s1.js); recovery pipeline now draft → canon_resident_calls → sig_unify → harvest_verify. Realistic gain ~+1.2–1.7%/batch (matches×134/344,010). PAUSED for /effort ultracode (R26/R27) — will build the batch-1 wave JS + launch on the toggle.

• 2026-06-20 — T3 batch-1 wave LAUNCHED (Ultracode, Drew toggled; R27 satisfied). 50 agents, .run/harvest_wave_t3b1.js (run wf_372a2738-773), drafts → .run/drafts-t3b1/.

• 2026-06-20 — T3 batch-1 DONE. Wave: 44/50 match_one MATCH (88%); 6 DIFF (genuine gcc-quirk tail: fold-const constant-last, GCSE/reload, loop-IV combine, reg time-sharing, narrow-param — correctly stubbed, no false matches). Whole-binary gate (the arbiter): 30 banked (28 via the improved 2-stage gate + 2 arity via no-proto). Two pipeline findings (R14): (1) sig_unify REGRESSES already-canonical agent drafts → gate canon-only FIRST, sig_unify FALLBACK (recovered +5 vs unconditional sig_unify); (2) gen_harvest_targets emits garbled callee sigs (use-site if(...) exprs) that agents paste as externs → 7 PARSE fails (root-cause fix queued for batch 2). Propagated 25 ×134 (dedup_propagate --auto-from; incl. the T2-deferred func_8017209C; ~5 stayed local — inline named structs). check-all 136/136 byte-identical (R22); dedup 1450→1475 validated/0 failed; fleet 56.64% → 57.62% (+0.98%). 14 still-failing (3 agent-garble, 1 struct, 10 genuine codegen near-miss) = batch-2 retry/recovery candidates. Recovery tools used: canon_resident_calls (T2), no-proto caller fix (T2), the 2-stage gate (.run/t3_gate2.sh). → commit batch 1, then improve (gen_harvest_targets sigs + fold canon-first into the pipeline), then batch 2.

• 2026-06-20 — Between-batch improvements + T3 batch-2 LAUNCHED. Fixed gen_harvest_targets.py INLINE_DEF_RE (was matching indented if(func_X(...)){ call-exprs as defs → garbled callee sigs; now column-0 + type-only prefix → 0 garbled of 749, was 9). Parameterized the canon-first 2-stage gate (.run/t3_gate2.sh <draftsdir>). Batch 2 = 38 targets (.run/t3_b2_targets.json, nins 10–88; cache-limited after batch-1's 25 propagated + 19 genuine-hard excluded), incl. the 3 garble-retries (clean hints now). Wave wf_68f3e807-d75 → .run/drafts-t3b2/. On completion: t3_gate2.sh .run/drafts-t3b2 → recover (no-proto/garble) → dedup_propagate ×134 → check-all → commit (incl. the gen_harvest_targets fix, uncommitted). Then assess batch 3 (needs Ghidra-C regen, R23/R29) vs close.

• 2026-06-20 — T3 batch-2 DONE. Wave: 35/38 match_one MATCH (92%), up from 88% (clean hints — the gen_harvest_targets fix worked; 2 of 3 garble-retries matched). 3 DIFF = genuine (delay-slot schedule, call-crossing hoist-vs-remat, narrow-param STOP). Gate: 17 (2-stage) + 8 via NEW tools/fix_arity_callers.py (the no-proto recovery: ALL 18 stage failures were the shared-caller arity class — automated it) = 25 banked. Remaining 10 = genuine loose-typing conflicts (caller sig ≠ def sig, call-site-cast walls — Phase-16 documented). Propagated 8 ×134 (17 stayed local — inline named structs, need engine_types lifting). check-all 136/136 (R22); dedup 1475→1483; fleet 57.62% → 57.93% (+0.31%). New/changed tools: fix_arity_callers.py (no-proto), gen_harvest_targets.py (garbled-sig fix), .run/t3_gate2.sh (canon-first 2-stage). → commit, then struct-lift to propagate the ~22 local struct matches (batches 1+2), then assess batch 3 vs close.

• 2026-06-20 — Struct-lift + propagation top-up. build_engine_types.py --strip lifted 48 named types → engine_types.h (byte-neutral, ov_SC01_077 d19c9580); propagated +4 more (incl. the T2/batch-2 struct-locals like func_80147514). check-all 136/136 (R22); fleet 57.93% → 58.00%. KEY FINDING (the phase's real bottleneck): propagation, not matching, now caps the fleet-% yield — match_one is 88–92% and the gate banks 60–71%, but ~16 banked matches stay ov_SC01_077-LOCAL because their bodies use typedef'd / anonymous / sibling local types that build_engine_types (named-struct-only) can't lift → dedup_propagate's compiles_standalone rejects them. Phase total: 56.64% → 58.00% (+1.36%); the +3–5% target was gated by (a) the cache-limited pool (~88 eligible, all run) and (b) this propagation gap. Phase-20 lever: extend the type-lift to typedefs/local types → recovers 16+ deferred local matches ×134 (+0.6%) for ~0 agent tokens, and lifts every future batch's yield.

▶ RESUME HERE (fresh session)

1. Re-read load order + this file. Rebuild the harness task list (R28). /effort max.
2. T2 DONE (committed) — canon_resident_calls.py + the no-proto caller-extern fix; 3 matches banked; check-all 136/136. Recovery pipeline = draft → canon_resident_calls → sig_unify → harvest_verify.
3. T3 (the main lever, +3-5%): regen Ghidra-C for fresh tractable reach-134 fns (DecompileFunctions.java, headless — MCP stopped, R23/R29); gen_harvest_targets.py manifest; then PROMPT Drew for /effort ultracode (R26/R27) and WAIT before launching the wave. Run 2-3 batches of 50 (recover→gate→dedup_propagate ×134→checkpoint→improve between batches).
4. Then T4 (conditional) / PhaseEnd.

• State after T2: commits commit:0158, commit:0159, commit:0160 (T1) + the T2 commit landed; fleet 56.64% (T1's 6 + T2's 3 are ov_SC01_077-local, not yet fleet-counted — they bank fleet-% only after the ×134 rollout). Next = T3.

Blockers

(none)

Notes / carried to Phase 20

• The -O0 lever completion (NEW, from T1): infra is built + proven in ov_SC01_077 (the 3-object split). Remaining: (a) match the 10 indexed-global -O0 fns — blocked by the %lo-folding quirk (our cc1 materializes lui;addiu;addu;sw 0(reg); original folds lui;addu;sw %lo(sym)(reg)); needs gcc-2.7.2 source research (R17/§17), maybe unsteerable from C. Stubs: func_8013B598/B6A0/B7AC/B83C/BC7C/BCDC/BD74/C08C/C0F8/C414. (b) The ×134 rollout — each overlay needs its own -O0 split (uniform offsets 0x13410/0x14834, shared bodies header); engine_core.h can't carry -O0 fns. ~+0.6% fleet when complete. (c) the 2 -O0 outliers (func_80144B9C giant + func_801457A4).
• Most giants (28 reach-134 >150-ins); per-overlay unique remainder (×1); comprehension/emulator actor-field naming; Wine/CC1PSX (parked); public flip (Phase 14, Gen3+).