BFM-decomp/phase-ends/CURRENT_PHASE.md

CURRENT PHASE — Phase 15: Overlay long-tail (ONGOING, not a Gen2 gate)

Started: 2026-06-16 · Project Version (in progress): 1.13.0 → 1.14.0 · Generation: Gen2 (7th phase) Effort: Max (planning + deep tasks); Ultracode for the breadth harvests (T5/T6/T7-unique), with the R27 pause-and-wait toggle at each transition. Plan file: ~/.claude/plans/defer-phase14-till-gen3-cheeky-yeti.md (approved by Drew, gate 1). Decisions (Drew, 2026-06-16): Phase 14 deferred to Gen3+; scope = "build machine + full fleet sweep"; Ghidra-free by default, escalate to per-overlay → per-SC-area Ghidra import only when the Ghidra-free loop stalls (T7).

Strategy (one line)

Overlays are position-locked at vram 0x80128158 and share an enormous engine core byte-identical across all 134 (9366 h_exact groups, ~29.7 MB collapsible), mostly still unmatched → match the shared core ONCE → propagate fleet-wide via a new tools/dedup_propagate.py, not re-harvest each overlay. The per-overlay make check byte-gate is the sole arbiter (G3/P9).

Verified ground truth (this session, R14)

• 134 overlays on disc; 129 distinct by sha1; 5 byte-identical dup pairs (free 2-for-1): SC01_005≡006 (done), SC02_000≡003, SC03_014≡015, SC03_118≡119, SC04_018≡019.
• Onboarded (4): ov_SC01_077 (704 REAL, d19c9580), ov_SC01_005 & 006 (3 REAL, 56760dbe), ov_SC03_001 (0 REAL, f8fd92f5). Resident 123/146. Main 54 REAL/959 LINKED.
• Top cross groups: ex0=770 ins @0x80144b9c ×134; ex1=476 @0x80141ca4; ex3=41 ins ×1104 — mostly unmatched even in 077.

Tasks (harness IDs in brackets; one at a time, P4)

• T0 [#1] — DONE (commit commit:0101) — Phase-14 deferral recorded in docs/gen2-roadmap.md.
• T1 [#2] — DONE — onboarded diverse wave-1 (12 overlays, all 7 SC areas + 2 dup pairs); check-all 18/18 byte-identical, 0 anomalies.
• T2 [#3] — DONE — tools/dedup_propagate.py + src/shared/engine_core.h built & proven: 4 funcs propagated across all 16 onboarded overlays, clean check-all 18/18, dedup_integrate green, idempotent, negative-test fails-closed. Also added Makefile -MMD header-dep tracking (closed a stale-build gap the negative test exposed).
• T3 [#4] — DONE — onboarded all remaining 118 overlays (0 failed) → full fleet 136 binaries; clean rebuild check-all 136/136 byte-identical (R22, 0 extract fails, ~8 min).
• T3 [#4] — onboard remaining ~118 overlays (full fleet); 136/136 check-all from clean (R22).
• T4 [#5] — DONE — progress.py --fleet → docs/progress.fleet.md (fleet 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%); refactor report(binary)+index (6m38s→7s, single-binary output unchanged); wired into make report; fixed dup_report --cross double-count (overlay counted as named binary AND glob → collapsible bytes were 2× inflated; back to accurate 28.6M). dedup-check 8/0. (P6 rules check done before T4.)
• T5 [#6] — DONE (Tier-1 harvest banked across passes) — (a) bulk-propagated 553 already-matched shared funcs fleet-wide; (b) Ultracode harvest pass 1 (300 small mixed → 49 verified) + pass 2 (52 pure-LEAF → 36 verified; 34 propagated). All byte-gated; 644 groups, 0 failed. Fleet 3.82% → 25.38% (REAL 85,651). Lesson: leaf-first yields ~69%, mixed/call-heavy ~16% — the call-heavy residual hits the extern-type-conflict wall (cookbook §14b). Pure leaves now ≈exhausted (52 remained, 36 matched).
• [~] T6 [#7] — IN PROGRESS — Ultracode shared-core harvest of the call-heavy residual + propagate. New tool tools/gen_harvest_targets.py builds a callee-signature-aware target manifest (the §14b wall-breaker): resolves each callee to its EXACT sig from engine_core.h DEFINE bodies + extern decls + inline defs, so agents never guess a conflicting extern. Targets split into .run/t6_small.json (616, nins 8-30) + .run/t6_med.json (430, nins 31-150), ~all reach=134. Pipeline validated on 3 hand-pilots (2 verified byte-identical; 1 = narrow-return-widening T6d case). Harvest Workflow (44 agents) running on the small band.
• T7 [#8] — ongoing tail: giants (match_one) + per-overlay unique remainder; Ghidra escalation when stuck.
• T8 [#9] — phase close: milestone proof (136/136 clean) + PhaseEnd (Tier 1 / Max).

Milestone (machine-checkable, P9; tail stays ongoing)

1. All 134 onboarded; make clean && extract-all && check-all → 136/136 byte-identical from clean (R22); negative control fails loud.
2. dedup_propagate.py proven fleet-wide (≥1 shared fn propagated across the full fleet, all byte-identical, dedup_integrate green, idempotent + negative test fail-closed).
3. docs/progress.fleet.md deterministic fleet totals (byte-identical %, collapsible bytes retired/remaining).
4. Shared-core harvest banked + reported honestly; giants + per-overlay unique tails logged as ongoing remainder.

RESUME HANDOFF (fresh session — start here; T0–T5 done, T6+T7 remain, then T8 PhaseEnd)

State: 136 binaries byte-identical from clean (make check-all 136/136, last verified after the leaf pass). Fleet 25.38% byte-identical (REAL 85,651 / matchable 344,010; docs/progress.fleet.md). Effort: Ultracode is ON (Drew) — good for the harvest; drop to Max for the PhaseEnd (T8). All work T0–T5 committed (Drew pushes, R6); commits commit:0101 commit:0102 commit:0103 commit:0104 commit:0105 commit:0106 commit:0107 + the leaf-pass commit.

The harvest machine (all proven this phase):

• tools/dedup_propagate.py --auto-from ov_SC01_077 — propagates every NEW matched+shared+self-contained inline-def in 077 across the fleet (skips registered; per-overlay byte-gate; shorthand registry; compile pre-filter). Re-run anytime to top up.
• Harvest loop (cookbook §12/§14b): (1) generate target batches (python filter on .run/sig.ov_*.jsonl + 077 stubs); (2) Ultracode Workflow — agents draft C → .run/drafts-X/<fn>.c and self-verify with tools/match_one.py <fn> --c … --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077; (3) byte-gate: tools/harvest_verify.py --binary ov_SC01_077 --src src/ov_SC01_077/ov_SC01_077.c --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077 --out build/ov_SC01_077/ov_SC01_077 --good-sha d19c9580a02dc63ba1f0e7e0c770f3b10de35635 --drafts .run/drafts-X; (4) dedup_propagate --auto-from ov_SC01_077; (5) make report; (6) R22 clean check-all; (7) commit.
• Workflow gotcha: the args global was unreliable — HARDCODE constants (e.g. const NB = N) in the script, don't read args.x.

T6 — Tier-2 / call-heavy shared core (the bulk of what's left, ~1097 unmatched-shared call-heavy fns, 8–120 ins, ~22 MB collapsible). Pure leaves are exhausted (matched). The blocker is the extern-type-conflict wall (cookbook §14b): a draft's extern for a callee ≠ the callee's real signature in the same TU. To break it, the next harvest must be callee-signature-aware — agents, for each jal func_X in the .s, must reuse func_X's EXACT signature if func_X is already C-defined (grep src/shared/engine_core.h for DEFINE_func_X / src/ov_SC01_077/ov_SC01_077.c); a still-stub callee takes any consistent extern. match_one CANNOT verify callees (masks jal) → trust only the whole-binary harvest_verify gate. Expect lower yield than leaves; this is genuine decomp-permuter-class work for the residual.

T7 — giants + per-overlay unique remainder. The 3 giants (ex0 770-ins @0x80144b9c, ex1 476 @0x80141ca4, ex2 400 @0x80132784 — byte-identical in all 134, ~0.9 MB) via match_one hand-iteration (Max), then propagate. Then opportunistic per-overlay unique (non-shared) function harvest, phase-disjoint per overlay. Ghidra escalation (Drew): if stuck on an overlay, tools/ghidra_import_raw.sh <0.4.dec> 0x80128158 <ov> + MCP (G2/R29/R23); per-SC-area if area-specific.

T8 — PhaseEnd (Tier-1, Max): R22 clean check-all 136/136 + make report; write phase-ends/PhaseEnd_Phase15.md (incl. ## Plain-English Recap, R25); archive this file → phase-ends/logs/Phase15.md (R19). Candidate rules to propose: none new required so far (techniques live in cookbook §14/§14a/§14b per the §9.6/§11 precedent).

Blockers / open

• None. (Pure-leaf shared functions are exhausted; remaining shared core is call-heavy — T6 needs callee-signature-aware drafting.)
• Minor: the 4 Phase-13/T2 proof groups (E_func_80128EA8/8012A568/80132EC4/80138C30) are at 16 members not 134 (registered before the bulk, so --auto-from skips them) — negligible; top up later if desired.

Log

• 2026-06-17 — T6 v3 missing-harvest (GROUPS OF 10) — fleet 48.02% → 51.41% (OVER HALF). (+20.2 session.) Drew: "do groups of 10 instead of 100+". Rebuilt the harvest workflow to process the 375 never-drafted shared stubs (leaf-first sorted) in SEQUENTIAL WAVES of ≤10 agents w/ intra-wave retry. Result: 0 dead batches (vs 33 from the 48-batch burst), ~42 drafts/60s (vs ~1/45s), 374/375 drafted. Gate-fill found 1 missing (func_8017C710). Gate: 85 verified (incl. high-yield leaf), → 79 propagated fleet-wide (7 struct/local skipped), 134 byte-identical, dedup 1237→1316 (0 failed). REAL ~164k → 175,212; fleet byte-identical 51.41%. Cookbook §12 updated (the rate-limit-gentle wave pattern). R22 clean running pre-commit. Residual: 289 quarantined (hard) + the 33 earlier-dead from the burst (now in land_missing) + 077-unique + giants — all permuter-class / low-leverage tail.
• 2026-06-17 — T6 v3 final harvest — SERVER-RATE-LIMITED, consolidated at 47.70%. Landscape: 568 remaining shared stubs (110 leaf + 458 call-heavy, richly resolved). Launched the v3 harvest (48 agents, .run/land_all.json); sustained server-side API rate limiting (93 agent failures) stalled it at ~31/568 drafts; gated → only 4 verified (the 31 were the harder call-heavy residual; the easy leaf ones never drafted). Stopped the workflow, reverted the 4 (kept the clean committed milestone), preserved .run/drafts-T6d + .run/land_all.json for resume. CONSOLIDATED: committed milestone commit:0114, fleet 47.70%, R22 136/136 clean; session 31.22% → 47.70% (+16.5). RESUME (when rate limiting clears): re-run the v3 harvest on .run/land_all.json (gap-fill + gate the 32 already-drafted), then OPTIONALLY T8 PhaseEnd to close Phase 15. The remaining shared core is the hard tail (regalloc/GTE/§10 — low yield even unthrottled, permuter-class). 4 typedef'd-struct fns + 27 giants/large + 077-unique are minor residual.
• 2026-06-17 — T6 struct follow-up — fleet 46.27% → 47.70% (+1.43). Built tools/build_engine_types.py → src/shared/engine_types.h (the 34 named struct/union types, forward decls + defs in source order; consumes trailing __attribute__((packed)) + aliases through the ;). Included it via engine_core.h; STRIPPED the 34 inline defs from 077.c (byte-neutral, verified). Relaxed dedup_propagate's guard to skip only inline named-struct/typedef DEFINITIONS (header-type USAGES + anonymous local structs propagate) and added engine_types.h to its compile pre-filter. → 37 struct-using shared fns propagated fleet-wide (was [skip] 41 → now 4, the overlay-local typedef'd types — tiny residual), 134 byte-identical, dedup 1183→1220 (0 failed). REAL 157,541 → 162,462. Session total: 31.22% → 47.70% (+16.5). R22 clean running pre-commit.
• 2026-06-17 — T6 tail: quarantine recovery — fleet 44.64% → 46.27% (+1.63). canon_draft_decls on the combined 505 quarantined (now canonicalizing against the much-larger banked set: 268 decls rewritten) → 47 verified (1175→1128 stubs) → 42 propagated fleet-wide (41 struct/local skipped), dedup 1141→1183 (0 failed). REAL 151,908 → 157,541. Session total: 31.22% → 46.27% (+15.0). Struct-landscape scoped (read-only): 34 named struct/union types in 077, ZERO same-name-different-layout collisions + 9 anonymous → a shared-types-header (src/shared/engine_types.h) is a CLEAN path to propagate the ~100 struct-using shared fns (the recurring "[skip] N local types"), ≈+3.8% fleet. NEXT (after this commit, H4): build engine_types.h, strip inline struct defs from 077.c (byte-neutral), relax dedup_propagate's struct guard for header-shared types, propagate.
• 2026-06-17 — T6g medium band — fleet 42.14% → 44.64% (+2.5). Regenerated .run/t6_med2.json (293 medium still-stub call-heavy, nins 41-150; fresh callee+data resolution). Harvest Workflow (30 agents) + 3-target gap-fill (R: mandatory reconcile). Gate: 73 verified (25% — harder fns: GTE/matrix/regalloc residuals; 220 quarantined .run/drafts-T6c-fail/). dedup_propagate (struct-guarded) → 65 propagated fleet-wide (37 struct/local-type skipped), 134 byte-identical, dedup 1076→1141 (0 failed). REAL 143,322 → 151,908. R22 clean running pre-commit. Session total: 31.22% → 44.64% (+13.4). Open tail: 285+220 quarantine (regalloc/struct/§10 residuals), struct-using shared fns (shared-types-header follow-up), 077-unique remainder, T7 giants.
• 2026-06-17 — T6 v2 (resumed after 2 crashes) — fleet 31.22% → 42.14% (+10.83 this pass). Extended gen_harvest_targets.py to ALSO resolve canonical DATA-symbol declarations per target (scan the .s for D_XXXX refs → extern <type> D_XXXX; from banked code) — the §14c(c) fix at the source. Regenerated the manifest (884 still-stub call-heavy; 1836 callees + 374 data refs resolved). v2 harvest Workflow (43 agents) on the 591 small still-stub (nins 8-40) + a gap-fill agent for 6 dropped-by-connection-failure targets (R: the retry-wave misses truncated-non-null results → now a MANDATORY cookbook post-step). Gate: 306 verified (1554→1248 stubs). Propagation hit a struct-redefinition abort (agents named structs with colliding generic names struct S/struct vec inline) → broadened dedup_propagate.py to skip any struct/union-touching body (34 skip) → 281 propagated fleet-wide, all 134 byte-identical. REAL 105,764 → 143,322; dedup 795 → 1076 groups (0 failed); fleet 42.14%. R22 clean check-all running pre-commit. NEXT: commit, then T6g medium band (293) + the struct-using-fns shared-types-header follow-up.
• 2026-06-16 — NEW SESSION (resume T6). Read full load order + cookbook + SETUP §5.4. Drew: Ultracode ON (R27 toggle landed), build task list (R28: tasks #1-7). T6a done — built tools/gen_harvest_targets.py (callee-sig-aware manifest; §14b wall-breaker). Found+fixed a resolver bug (macro-body split artifact → 0/640 sigs extracted; line-by-line fix → 479 defined + 376 extern-declared resolved on the small band) AND added extern-declaration harvesting (a stub callee already extern-declared elsewhere must reuse that sig — caught by the func_80150BA4 pilot, void-vs-s32). 3 pilots gated: func_8012A0E0 ✓, func_80150BA4 ✓, func_8017AE08 = T6D-WIDEN (narrow s16 return forces sll/sra the target lacks → widen the def, byte-identical). T6b Workflow (44 agents) launched on .run/t6_small.json. Drafts → .run/drafts-T6/; gate = harvest_verify --binary ov_SC01_077 --good-sha d19c9580….
• 2026-06-17 — T6 pass-1 (small band) BANKED — fleet 25.38% → 30.48% (+5.10). The callee-sig-aware manifest (gen_harvest_targets.py) gave 60-67% gate yield on the call-heavy residual (vs §14b naive 16%) by pre-resolving every callee's EXACT sig (defined bodies + extern decls + inline defs). Despite SERVER-SIDE API rate limiting that stalled the draft Workflow at ~246/616 (stopped it deliberately), the incremental gate→propagate loop (deterministic, throttle-immune) banked 142 functions in ov_SC01_077 (1716→1574 stubs) → ~132 propagated fleet-wide (134 overlays each, every round byte-gated). REAL 85,651 → 103,217; dedup 644 → 776 groups (0 failed). Cookbook §14c written (the wall-breaker + 4 conflict flavors + the git checkout hazard + the rate-limit-resilient loop). Committed commit:0109 (R22 136/136 clean). Then T6d deterministic recovery (no API): new tools/canon_draft_decls.py rewrites each quarantined draft's decls to the banked-consistent set (byte-gate = safety net) → +19 more recovered + propagated (of the 90 match_one-MATCH-but-conflicting; the other ~70 need definition-WIDENING §14c(d) or true data-type resolution — next pass). Fleet → 31.22% (+5.84 session); REAL substantive 85,651 → 105,764; dedup 644 → 795 groups (0 failed); 162 fns banked in ov_SC01_077 (1716→1554 stubs). (Correction: commit:0110's message said "1535 stubs" — arithmetic slip conflating the 19 fleet-propagated with stub reduction; on-disk truth is 1554, verified post-crash. Fleet % is authoritative from make report.) Residual for next session: (a) narrow-return-widening pass (§14c(d), recovers the ~70); (b) the data-symbol canonical-decls structural fix; (c) the medium band (430, un-started); (d) the 54 genuine instruction-misses → permuter/§3a. R22 #2 running pre-commit-of-the-+19.
• 2026-06-16 — Session start: read load order (PROJECT_CONTEXT + all 14 PhaseEnds, no prior CURRENT_PHASE). Drew: skip Phase 14, do Phase 15, Max effort, plan mode. Verified ground truth (overlay count, dup pairs, cross-report, per-overlay counts) directly, correcting several sub-agent overclaims (R14). Plan approved (gate 1). Task list created (R28). Starting T0.
• 2026-06-16 — T0 done (commit:0101): Phase-14 deferral propagated through gen2-roadmap (additive, H5).
• 2026-06-16 — T1 done: onboarded wave-1 = ov_{SC01_000, SC01_001, SC02_000, SC02_003, SC03_002, SC03_006, SC04_000, SC04_018, SC04_019, SC05_000, SC06_000, SC07_000} (all byte-identical at 100% INCLUDE_ASM in ~50s; dup pairs SC02_000≡003=5ece4bca, SC04_018≡019=fe9b413f). Fleet make check-all = 18/18 passed. Position-lock risk retired across all 7 areas — no anomalies. Next: T2 (dedup_propagate.py).
• 2026-06-16 — KEY FINDING (reshapes T5): of ov_SC01_077's 785 matched signed functions, 577 are h_exact-identical across ALL 134 overlays (2.19 MB collapsible) — already matched, just need propagating (free credit, no harvest). 204 are 077-unique. So T5 has two parts: (a) bulk-propagate these 577 already-matched shared funcs fleet-wide (mechanical, after T3); (b) Ultracode-harvest the still-UNMATCHED shared core (the big ex0–ex50 giants/mediums, mostly unmatched even in 077).
• 2026-06-16 — T5(b) pass 2 (leaf-first) done: identified 52 pure-LEAF unmatched-shared funcs (no jal/%hi/%lo → match_one is an exact gate); 5-agent Workflow → 36/52 verified (69%) in ov_SC01_077 → 34 propagated fleet-wide (644 groups, 0 failed). Fleet 24.05% → 25.38% (REAL 85,651). R22 clean-verify running pre-commit. Stopping point here (Drew): no PhaseEnd; T6 + T7 remain. Fresh session resumes — see RESUME HANDOFF below.
• 2026-06-16 — T5(b) pass 1 done: 25-agent Workflow drafted+match_one-verified 300 small unmatched-shared → byte-gate → 49 verified → propagated. Fleet 22.14% → 24.05%. Diagnosed the call-heavy wall (match_one masks callees; extern-type conflicts).
• 2026-06-16 — T5(a) done (the big free-credit run): dedup_propagate --auto-from ov_SC01_077 bulk-propagated 553 already-matched shared functions across all 134 overlays, each byte-gated. Hardened the tool for fleet scale: registry shorthand (vram+binaries; dedup.us.yaml 4050 lines vs ~77k verbose), per-overlay apply (one read/write/file), gate-only-changed, skip already-registered (additive/resumable), and a compile pre-filter (body must build with common.h alone — 9 functions use 077-local struct types and were skipped honestly; the byte-gate first caught one mid-run → fail-closed revert → added the filter). group_members() reader (both forms) in dedup_integrate + progress; progress _DEDUP_CACHE (fleet 136× parse → 1). Fleet: REAL 947→74,527, byte-identical 13,132→76,174 = 22.14%; dedup-check 561 validated / 0 failed. R22 clean rebuild verifying before commit. Next: T5(b) Ultracode harvest of the unmatched core (R27 toggle).
• 2026-06-16 — T4 done: refactored progress.py → report(binary) + --fleet (aggregates docs/progress.fleet.md, deterministic source-derived) + a per-binary .s index (fleet run 6m38s→7s; single-binary docs/progress.md byte-unchanged). Wired --fleet into make report. Honesty fix (R14): dup_report --cross was double-counting each onboarded overlay (named BINARIES entry AND .run/sig.ov_* glob) → collapsible bytes 2× inflated (58.7M); deduped by alias → accurate 9366 h_exact groups / 28.6M collapsible. make report green, dedup-check 8/0. Fleet now: 947 REAL / 959 LINKED / 3.82% byte-identical. Rules check (P6) done. Next: T5.
• 2026-06-16 — T3 done: onboarded all remaining 118 overlays (sequential new_overlay.sh, 0 failed) → fleet = 134 overlays + main + resident = 136 binaries. R22 CLEAN rebuild (make clean → extract all 136 → check-all) = 136 passed / 0 failed, 0 extract fails. Every overlay byte-identical at 100% INCLUDE_ASM. Report-tool dicts + overlays.mk auto-registered by new_overlay.sh. Next: T4 (fleet roll-up + regenerate fleet-wide cross-report).
• 2026-06-16 — T2 done: built tools/dedup_propagate.py (extract body → DEFINE_func macro in src/shared/engine_core.h → in-place stub/def replace at every h_exact member → snapshot+byte-gate fail-closed → register dedup.us.yaml). Proven on 4 medium funcs (0x80128EA8/8012A568/80132EC4/80138C30) across all 16 onboarded: clean check-all 18/18, dedup_integrate 8 validated/0 failed, idempotent re-run no-op, negative test (corrupt body) → make check FAILS then restores. Fixed 2 bugs found by self-verification: (i) edit-accumulation read from snapshot not disk (only last fn landed); (ii) Makefile had no header-dep tracking → header-only edits were stale/false-pass → added cpp -MMD -MP + -include $(C_DEPS) (output-neutral; R22-clean 18/18). Cookbook §14, SETUP inventory updated. Next: T3.