BFM-decomp/phase-ends/logs/Phase15.md

# CURRENT PHASE — Phase 15: Overlay long-tail (ONGOING, not a Gen2 gate)

**Started:** 2026-06-16 · **Project Version (in progress):** 1.13.0 → 1.14.0 · **Generation:** Gen2 (7th phase)
**Effort:** Max (planning + deep tasks); **Ultracode** for the breadth harvests (T5/T6/T7-unique), with the R27 pause-and-wait toggle at each transition.
**Plan file:** `~/.claude/plans/defer-phase14-till-gen3-cheeky-yeti.md` (approved by Drew, gate 1).
**Decisions (Drew, 2026-06-16):** Phase 14 deferred to Gen3+; scope = "build machine + full fleet sweep"; **Ghidra-free by default, escalate to per-overlay → per-SC-area Ghidra import only when the Ghidra-free loop stalls** (T7).

## Strategy (one line)
Overlays are position-locked at vram 0x80128158 and share an enormous engine core byte-identical across all 134 (9366 h_exact groups, ~29.7 MB collapsible), mostly still unmatched → **match the shared core ONCE → propagate fleet-wide** via a new `tools/dedup_propagate.py`, not re-harvest each overlay. The per-overlay `make check` byte-gate is the sole arbiter (G3/P9).

## Verified ground truth (this session, R14)
- 134 overlays on disc; **129 distinct by sha1**; 5 byte-identical dup pairs (free 2-for-1): SC01_005≡006 (done), SC02_000≡003, SC03_014≡015, SC03_118≡119, SC04_018≡019.
- Onboarded (4): ov_SC01_077 (704 REAL, d19c9580), ov_SC01_005 & 006 (3 REAL, 56760dbe), ov_SC03_001 (0 REAL, f8fd92f5). Resident 123/146. Main 54 REAL/959 LINKED.
- Top cross groups: ex0=770 ins @0x80144b9c ×134; ex1=476 @0x80141ca4; ex3=41 ins ×1104 — mostly unmatched even in 077.

## Tasks (harness IDs in brackets; one at a time, P4)
- [x] **T0 [#1] — DONE** (commit commit:0101) — Phase-14 deferral recorded in `docs/gen2-roadmap.md`.
- [x] **T1 [#2] — DONE** — onboarded diverse wave-1 (12 overlays, all 7 SC areas + 2 dup pairs); `check-all` **18/18 byte-identical**, 0 anomalies.
- [x] **T2 [#3] — DONE** — `tools/dedup_propagate.py` + `src/shared/engine_core.h` built & proven: 4 funcs propagated across all 16 onboarded overlays, clean `check-all` **18/18**, dedup_integrate green, idempotent, negative-test fails-closed. Also added Makefile `-MMD` header-dep tracking (closed a stale-build gap the negative test exposed).
- [x] **T3 [#4] — DONE** — onboarded all remaining 118 overlays (0 failed) → full fleet **136 binaries**; **clean rebuild `check-all` 136/136 byte-identical** (R22, 0 extract fails, ~8 min).
- [ ] T3 [#4] — onboard remaining ~118 overlays (full fleet); 136/136 `check-all` from clean (R22).
- [x] **T4 [#5] — DONE** — `progress.py --fleet` → `docs/progress.fleet.md` (fleet **947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%**); refactor `report(binary)`+index (6m38s→7s, single-binary output unchanged); wired into `make report`; **fixed dup_report --cross double-count** (overlay counted as named binary AND glob → collapsible bytes were 2× inflated; back to accurate 28.6M). dedup-check 8/0. **(P6 rules check done before T4.)**
- [x] **T5 [#6] — DONE (Tier-1 harvest banked across passes)** — (a) bulk-propagated **553** already-matched shared funcs fleet-wide; (b) Ultracode harvest **pass 1** (300 small mixed → **49** verified) + **pass 2** (52 pure-LEAF → **36** verified; **34** propagated). All byte-gated; 644 groups, 0 failed. **Fleet 3.82% → 25.38%** (REAL 85,651). **Lesson:** leaf-first yields ~69%, mixed/call-heavy ~16% — the call-heavy residual hits the **extern-type-conflict wall** (cookbook §14b). Pure leaves now ≈exhausted (52 remained, 36 matched).
- [~] **T6 [#7] — IN PROGRESS** — Ultracode shared-core harvest of the **call-heavy** residual + propagate. New tool `tools/gen_harvest_targets.py` builds a **callee-signature-aware** target manifest (the §14b wall-breaker): resolves each callee to its EXACT sig from engine_core.h DEFINE bodies + extern decls + inline defs, so agents never guess a conflicting extern. Targets split into `.run/t6_small.json` (616, nins 8-30) + `.run/t6_med.json` (430, nins 31-150), ~all reach=134. Pipeline validated on 3 hand-pilots (2 verified byte-identical; 1 = narrow-return-widening T6d case). Harvest Workflow (44 agents) running on the small band.
- [ ] T7 [#8] — ongoing tail: giants (match_one) + per-overlay unique remainder; Ghidra escalation when stuck.
- [ ] T8 [#9] — phase close: milestone proof (136/136 clean) + PhaseEnd (Tier 1 / Max).

## Milestone (machine-checkable, P9; tail stays ongoing)
1. All 134 onboarded; `make clean && extract-all && check-all` → 136/136 byte-identical from clean (R22); negative control fails loud.
2. `dedup_propagate.py` proven fleet-wide (≥1 shared fn propagated across the full fleet, all byte-identical, dedup_integrate green, idempotent + negative test fail-closed).
3. `docs/progress.fleet.md` deterministic fleet totals (byte-identical %, collapsible bytes retired/remaining).
4. Shared-core harvest banked + reported honestly; giants + per-overlay unique tails logged as ongoing remainder.

## RESUME HANDOFF (fresh session — start here; T0–T5 done, T6+T7 remain, then T8 PhaseEnd)

**State:** 136 binaries byte-identical from clean (`make check-all` 136/136, last verified after the leaf pass). Fleet **25.38% byte-identical** (REAL 85,651 / matchable 344,010; `docs/progress.fleet.md`). Effort: **Ultracode is ON** (Drew) — good for the harvest; drop to Max for the PhaseEnd (T8). All work T0–T5 committed (Drew pushes, R6); commits `commit:0101 commit:0102 commit:0103 commit:0104 commit:0105 commit:0106 commit:0107` + the leaf-pass commit.

**The harvest machine (all proven this phase):**
- `tools/dedup_propagate.py --auto-from ov_SC01_077` — propagates every NEW matched+shared+self-contained inline-def in 077 across the fleet (skips registered; per-overlay byte-gate; shorthand registry; compile pre-filter). Re-run anytime to top up.
- Harvest loop (cookbook §12/§14b): (1) generate target batches (python filter on `.run/sig.ov_*.jsonl` + 077 stubs); (2) Ultracode Workflow — agents draft C → `.run/drafts-X/<fn>.c` and self-verify with `tools/match_one.py <fn> --c … --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077`; (3) byte-gate: `tools/harvest_verify.py --binary ov_SC01_077 --src src/ov_SC01_077/ov_SC01_077.c --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077 --out build/ov_SC01_077/ov_SC01_077 --good-sha d19c9580a02dc63ba1f0e7e0c770f3b10de35635 --drafts .run/drafts-X`; (4) `dedup_propagate --auto-from ov_SC01_077`; (5) `make report`; (6) R22 clean `check-all`; (7) commit.
- **Workflow gotcha:** the `args` global was unreliable — HARDCODE constants (e.g. `const NB = N`) in the script, don't read `args.x`.

**T6 — Tier-2 / call-heavy shared core (the bulk of what's left, ~1097 unmatched-shared call-heavy fns, 8–120 ins, ~22 MB collapsible).** Pure leaves are exhausted (matched). The blocker is the **extern-type-conflict wall** (cookbook §14b): a draft's `extern` for a callee ≠ the callee's real signature in the same TU. To break it, the next harvest must be **callee-signature-aware** — agents, for each `jal func_X` in the `.s`, must reuse func_X's EXACT signature if func_X is already C-defined (grep `src/shared/engine_core.h` for `DEFINE_func_X` / `src/ov_SC01_077/ov_SC01_077.c`); a still-stub callee takes any consistent extern. `match_one` CANNOT verify callees (masks jal) → trust only the whole-binary `harvest_verify` gate. Expect lower yield than leaves; this is genuine decomp-permuter-class work for the residual.

**T7 — giants + per-overlay unique remainder.** The 3 giants (ex0 770-ins @0x80144b9c, ex1 476 @0x80141ca4, ex2 400 @0x80132784 — byte-identical in all 134, ~0.9 MB) via `match_one` hand-iteration (Max), then propagate. Then opportunistic per-overlay *unique* (non-shared) function harvest, phase-disjoint per overlay. **Ghidra escalation (Drew):** if stuck on an overlay, `tools/ghidra_import_raw.sh <0.4.dec> 0x80128158 <ov>` + MCP (G2/R29/R23); per-SC-area if area-specific.

**T8 — PhaseEnd** (Tier-1, Max): R22 clean `check-all` 136/136 + `make report`; write `phase-ends/PhaseEnd_Phase15.md` (incl. `## Plain-English Recap`, R25); archive this file → `phase-ends/logs/Phase15.md` (R19). Candidate rules to propose: none new required so far (techniques live in cookbook §14/§14a/§14b per the §9.6/§11 precedent).

## Blockers / open
- None. (Pure-leaf shared functions are exhausted; remaining shared core is call-heavy — T6 needs callee-signature-aware drafting.)
- Minor: the 4 Phase-13/T2 proof groups (E_func_80128EA8/8012A568/80132EC4/80138C30) are at 16 members not 134 (registered before the bulk, so --auto-from skips them) — negligible; top up later if desired.

## Log
- 2026-06-17 — **T6 v3 missing-harvest (GROUPS OF 10) — fleet 48.02% → 51.41% (OVER HALF). (+20.2 session.)** Drew: "do groups of 10 instead of 100+". Rebuilt the harvest workflow to process the 375 never-drafted shared stubs (leaf-first sorted) in SEQUENTIAL WAVES of ≤10 agents w/ intra-wave retry. Result: **0 dead batches** (vs 33 from the 48-batch burst), ~42 drafts/60s (vs ~1/45s), 374/375 drafted. Gate-fill found 1 missing (func_8017C710). Gate: **85 verified** (incl. high-yield leaf), → **79 propagated** fleet-wide (7 struct/local skipped), 134 byte-identical, dedup 1237→1316 (0 failed). REAL ~164k → 175,212; **fleet byte-identical 51.41%**. Cookbook §12 updated (the rate-limit-gentle wave pattern). R22 clean running pre-commit. Residual: 289 quarantined (hard) + the 33 earlier-dead from the burst (now in land_missing) + 077-unique + giants — all permuter-class / low-leverage tail.
- 2026-06-17 — **T6 v3 final harvest — SERVER-RATE-LIMITED, consolidated at 47.70%.** Landscape: 568 remaining shared stubs (110 leaf + 458 call-heavy, richly resolved). Launched the v3 harvest (48 agents, `.run/land_all.json`); sustained server-side API rate limiting (93 agent failures) stalled it at ~31/568 drafts; gated → only 4 verified (the 31 were the harder call-heavy residual; the easy leaf ones never drafted). Stopped the workflow, reverted the 4 (kept the clean committed milestone), preserved `.run/drafts-T6d` + `.run/land_all.json` for resume. **CONSOLIDATED: committed milestone commit:0114, fleet 47.70%, R22 136/136 clean; session 31.22% → 47.70% (+16.5).** RESUME (when rate limiting clears): re-run the v3 harvest on `.run/land_all.json` (gap-fill + gate the 32 already-drafted), then OPTIONALLY T8 PhaseEnd to close Phase 15. The remaining shared core is the hard tail (regalloc/GTE/§10 — low yield even unthrottled, permuter-class). 4 typedef'd-struct fns + 27 giants/large + 077-unique are minor residual.
- 2026-06-17 — **T6 struct follow-up — fleet 46.27% → 47.70% (+1.43).** Built `tools/build_engine_types.py` → `src/shared/engine_types.h` (the 34 named struct/union types, forward decls + defs in source order; consumes trailing `__attribute__((packed))` + aliases through the `;`). Included it via `engine_core.h`; STRIPPED the 34 inline defs from 077.c (byte-neutral, verified). Relaxed `dedup_propagate`'s guard to skip only inline named-struct/typedef DEFINITIONS (header-type USAGES + anonymous local structs propagate) and added `engine_types.h` to its compile pre-filter. → **37 struct-using shared fns propagated** fleet-wide (was [skip] 41 → now 4, the overlay-local typedef'd types — tiny residual), 134 byte-identical, dedup 1183→1220 (0 failed). REAL 157,541 → 162,462. **Session total: 31.22% → 47.70% (+16.5).** R22 clean running pre-commit.
- 2026-06-17 — **T6 tail: quarantine recovery — fleet 44.64% → 46.27% (+1.63).** `canon_draft_decls` on the combined 505 quarantined (now canonicalizing against the much-larger banked set: 268 decls rewritten) → **47 verified** (1175→1128 stubs) → **42 propagated** fleet-wide (41 struct/local skipped), dedup 1141→1183 (0 failed). REAL 151,908 → 157,541. **Session total: 31.22% → 46.27% (+15.0).** **Struct-landscape scoped (read-only): 34 named struct/union types in 077, ZERO same-name-different-layout collisions + 9 anonymous → a shared-types-header (`src/shared/engine_types.h`) is a CLEAN path to propagate the ~100 struct-using shared fns (the recurring "[skip] N local types"), ≈+3.8% fleet.** NEXT (after this commit, H4): build engine_types.h, strip inline struct defs from 077.c (byte-neutral), relax dedup_propagate's struct guard for header-shared types, propagate.
- 2026-06-17 — **T6g medium band — fleet 42.14% → 44.64% (+2.5).** Regenerated `.run/t6_med2.json` (293 medium still-stub call-heavy, nins 41-150; fresh callee+data resolution). Harvest Workflow (30 agents) + 3-target gap-fill (R: mandatory reconcile). Gate: **73 verified** (25% — harder fns: GTE/matrix/regalloc residuals; 220 quarantined `.run/drafts-T6c-fail/`). `dedup_propagate` (struct-guarded) → **65 propagated fleet-wide** (37 struct/local-type skipped), 134 byte-identical, dedup 1076→1141 (0 failed). REAL 143,322 → 151,908. R22 clean running pre-commit. **Session total: 31.22% → 44.64% (+13.4).** Open tail: 285+220 quarantine (regalloc/struct/§10 residuals), struct-using shared fns (shared-types-header follow-up), 077-unique remainder, T7 giants.
- 2026-06-17 — **T6 v2 (resumed after 2 crashes) — fleet 31.22% → 42.14% (+10.83 this pass).** Extended `gen_harvest_targets.py` to ALSO resolve canonical DATA-symbol declarations per target (scan the .s for D_XXXX refs → `extern <type> D_XXXX;` from banked code) — the §14c(c) fix at the source. Regenerated the manifest (884 still-stub call-heavy; 1836 callees + 374 data refs resolved). v2 harvest Workflow (43 agents) on the 591 small still-stub (nins 8-40) + a **gap-fill agent** for 6 dropped-by-connection-failure targets (R: the retry-wave misses truncated-non-null results → now a MANDATORY cookbook post-step). Gate: **306 verified** (1554→1248 stubs). Propagation hit a struct-redefinition abort (agents named structs with colliding generic names `struct S`/`struct vec` inline) → broadened `dedup_propagate.py` to skip any struct/union-touching body (34 skip) → **281 propagated fleet-wide**, all 134 byte-identical. **REAL 105,764 → 143,322; dedup 795 → 1076 groups (0 failed); fleet 42.14%.** R22 clean check-all running pre-commit. NEXT: commit, then T6g medium band (293) + the struct-using-fns shared-types-header follow-up.
- 2026-06-16 — **NEW SESSION (resume T6).** Read full load order + cookbook + SETUP §5.4. Drew: Ultracode ON (R27 toggle landed), build task list (R28: tasks #1-7). **T6a done** — built `tools/gen_harvest_targets.py` (callee-sig-aware manifest; §14b wall-breaker). Found+fixed a resolver bug (macro-body split artifact → 0/640 sigs extracted; line-by-line fix → 479 defined + 376 extern-declared resolved on the small band) AND added extern-declaration harvesting (a stub callee already extern-declared elsewhere must reuse that sig — caught by the func_80150BA4 pilot, void-vs-s32). 3 pilots gated: func_8012A0E0 ✓, func_80150BA4 ✓, func_8017AE08 = T6D-WIDEN (narrow s16 return forces sll/sra the target lacks → widen the def, byte-identical). **T6b** Workflow (44 agents) launched on `.run/t6_small.json`. Drafts → `.run/drafts-T6/`; gate = `harvest_verify --binary ov_SC01_077 --good-sha d19c9580…`.
- 2026-06-17 — **T6 pass-1 (small band) BANKED — fleet 25.38% → 30.48% (+5.10).** The callee-sig-aware manifest (`gen_harvest_targets.py`) gave **60-67% gate yield** on the call-heavy residual (vs §14b naive 16%) by pre-resolving every callee's EXACT sig (defined bodies + extern decls + inline defs). Despite SERVER-SIDE API rate limiting that stalled the draft Workflow at ~246/616 (stopped it deliberately), the incremental gate→propagate loop (deterministic, throttle-immune) banked **142 functions in ov_SC01_077** (1716→1574 stubs) → **~132 propagated fleet-wide** (134 overlays each, every round byte-gated). **REAL 85,651 → 103,217; dedup 644 → 776 groups (0 failed).** Cookbook §14c written (the wall-breaker + 4 conflict flavors + the `git checkout` hazard + the rate-limit-resilient loop). **Committed `commit:0109`** (R22 136/136 clean). Then **T6d deterministic recovery** (no API): new `tools/canon_draft_decls.py` rewrites each quarantined draft's decls to the banked-consistent set (byte-gate = safety net) → **+19 more** recovered + propagated (of the 90 match_one-MATCH-but-conflicting; the other ~70 need definition-WIDENING §14c(d) or true data-type resolution — next pass). **Fleet → 31.22% (+5.84 session); REAL substantive 85,651 → 105,764; dedup 644 → 795 groups (0 failed); 162 fns banked in ov_SC01_077 (1716→1554 stubs).** (Correction: commit:0110's message said "1535 stubs" — arithmetic slip conflating the 19 fleet-propagated with stub reduction; on-disk truth is 1554, verified post-crash. Fleet % is authoritative from make report.) Residual for next session: (a) narrow-return-widening pass (§14c(d), recovers the ~70); (b) the data-symbol canonical-decls structural fix; (c) the medium band (430, un-started); (d) the 54 genuine instruction-misses → permuter/§3a. R22 #2 running pre-commit-of-the-+19.
- 2026-06-16 — Session start: read load order (PROJECT_CONTEXT + all 14 PhaseEnds, no prior CURRENT_PHASE). Drew: skip Phase 14, do Phase 15, Max effort, plan mode. Verified ground truth (overlay count, dup pairs, cross-report, per-overlay counts) directly, correcting several sub-agent overclaims (R14). Plan approved (gate 1). Task list created (R28). Starting T0.
- 2026-06-16 — **T0 done** (commit:0101): Phase-14 deferral propagated through gen2-roadmap (additive, H5).
- 2026-06-16 — **T1 done**: onboarded wave-1 = ov_{SC01_000, SC01_001, SC02_000, SC02_003, SC03_002, SC03_006, SC04_000, SC04_018, SC04_019, SC05_000, SC06_000, SC07_000} (all byte-identical at 100% INCLUDE_ASM in ~50s; dup pairs SC02_000≡003=5ece4bca, SC04_018≡019=fe9b413f). Fleet `make check-all` = **18/18 passed**. Position-lock risk retired across all 7 areas — no anomalies. Next: T2 (dedup_propagate.py).
- 2026-06-16 — **KEY FINDING (reshapes T5):** of `ov_SC01_077`'s 785 matched signed functions, **577 are h_exact-identical across ALL 134 overlays** (2.19 MB collapsible) — already matched, just need propagating (free credit, no harvest). 204 are 077-unique. So T5 has two parts: (a) bulk-propagate these 577 already-matched shared funcs fleet-wide (mechanical, after T3); (b) Ultracode-harvest the still-UNMATCHED shared core (the big ex0–ex50 giants/mediums, mostly unmatched even in 077).
- 2026-06-16 — **T5(b) pass 2 (leaf-first) done**: identified 52 pure-LEAF unmatched-shared funcs (no jal/%hi/%lo → match_one is an exact gate); 5-agent Workflow → **36/52 verified (69%)** in ov_SC01_077 → **34 propagated** fleet-wide (644 groups, 0 failed). Fleet 24.05% → **25.38%** (REAL 85,651). R22 clean-verify running pre-commit. **Stopping point here (Drew): no PhaseEnd; T6 + T7 remain. Fresh session resumes — see RESUME HANDOFF below.**
- 2026-06-16 — **T5(b) pass 1 done**: 25-agent Workflow drafted+match_one-verified 300 small unmatched-shared → byte-gate → 49 verified → propagated. Fleet 22.14% → 24.05%. Diagnosed the call-heavy wall (match_one masks callees; extern-type conflicts).
- 2026-06-16 — **T5(a) done** (the big free-credit run): `dedup_propagate --auto-from ov_SC01_077` bulk-propagated **553** already-matched shared functions across all 134 overlays, each byte-gated. Hardened the tool for fleet scale: registry **shorthand** (vram+binaries; dedup.us.yaml 4050 lines vs ~77k verbose), **per-overlay apply** (one read/write/file), **gate-only-changed**, **skip already-registered** (additive/resumable), and a **compile pre-filter** (body must build with common.h alone — 9 functions use 077-local struct types and were skipped honestly; the byte-gate first caught one mid-run → fail-closed revert → added the filter). `group_members()` reader (both forms) in dedup_integrate + progress; progress `_DEDUP_CACHE` (fleet 136× parse → 1). **Fleet: REAL 947→74,527, byte-identical 13,132→76,174 = 22.14%**; dedup-check 561 validated / 0 failed. R22 clean rebuild verifying before commit. Next: T5(b) Ultracode harvest of the unmatched core (R27 toggle).
- 2026-06-16 — **T4 done**: refactored `progress.py` → `report(binary)` + `--fleet` (aggregates `docs/progress.fleet.md`, deterministic source-derived) + a per-binary `.s` index (fleet run 6m38s→7s; single-binary `docs/progress.md` byte-unchanged). Wired `--fleet` into `make report`. **Honesty fix (R14):** `dup_report --cross` was double-counting each onboarded overlay (named BINARIES entry AND `.run/sig.ov_*` glob) → collapsible bytes 2× inflated (58.7M); deduped by alias → accurate **9366 h_exact groups / 28.6M collapsible**. `make report` green, dedup-check 8/0. Fleet now: 947 REAL / 959 LINKED / 3.82% byte-identical. Rules check (P6) done. Next: T5.
- 2026-06-16 — **T3 done**: onboarded all remaining 118 overlays (sequential new_overlay.sh, 0 failed) → fleet = 134 overlays + main + resident = **136 binaries**. R22 CLEAN rebuild (make clean → extract all 136 → check-all) = **136 passed / 0 failed**, 0 extract fails. Every overlay byte-identical at 100% INCLUDE_ASM. Report-tool dicts + overlays.mk auto-registered by new_overlay.sh. Next: T4 (fleet roll-up + regenerate fleet-wide cross-report).
- 2026-06-16 — **T2 done**: built `tools/dedup_propagate.py` (extract body → DEFINE_func macro in src/shared/engine_core.h → in-place stub/def replace at every h_exact member → snapshot+byte-gate fail-closed → register dedup.us.yaml). Proven on 4 medium funcs (0x80128EA8/8012A568/80132EC4/80138C30) across all 16 onboarded: clean `check-all` 18/18, dedup_integrate 8 validated/0 failed, idempotent re-run no-op, negative test (corrupt body) → `make check` FAILS then restores. Fixed 2 bugs found by self-verification: (i) edit-accumulation read from snapshot not disk (only last fn landed); (ii) **Makefile had no header-dep tracking** → header-only edits were stale/false-pass → added `cpp -MMD -MP` + `-include $(C_DEPS)` (output-neutral; R22-clean 18/18). Cookbook §14, SETUP inventory updated. Next: T3.