BFM-decomp/phase-ends/PhaseEnd_Phase13.md

PhaseEnd — Phase 13: Representative location overlays (end-to-end + fleet runbook)

Date: 2026-06-16 · Project Version: 1.13.0 · Phase Status: Complete (milestone confirmed by Drew, gate 2) · Generation: Gen2 (6th phase)

Gen2 phase 6 of the arc (8→9→10→11→12→13→14→15). The granular per-task trail (the harvest pass-by-pass logs, the non-4-aligned-overlay diagnostics, the dedup sig-hash hunt, the effort-experiment runs) is preserved on-demand at phase-ends/logs/Phase13.md (R19 — NOT auto-loaded; consult only when researching a mechanism). Owner decisions (2026-06-16): scope = "Representative + credit" (not the full 134-sweep — that's Phase 15); import the representative overlays into Ghidra; the harvest ran under Ultracode (Drew enabled it); a controlled Max-vs-Ultracode effort experiment (T7) was added as the final task.

Build Log

Files created/changed and complete — do not recreate:

Reusable overlay machinery (the Phase-15 fleet recipe):

• config/splat.us.overlay.template.yaml — new. The flat-blob splat template for any location overlay (@TOKEN@ placeholders, unquoted so vram lands as an int): no header/gp_value (-G0), single code seg @ vram 0x80128158, asset_path: assets/@ALIAS@, stacked [symbols.us, symbols.resident, symbols.@ALIAS@], @TRAILING@ marker for the non-4-aligned bin carve. Code at file 0x0 (overlays open with a prologue).
• tools/new_overlay.sh — new. One-command onboarding <SCxx> <FILE_nnn>: sha1/size/code_end (sig_image) → instantiate the config (+ inject the bin carve when size%4≠0) → config/check.<ov>.sha + empty config/symbols.<ov>.txt → append the <ov>_* block to the generated config/overlays.mk + OVERLAY_BINARIES → sentinel-insert the 4 report/diff dicts (grep-guarded + ast.parse-checked) → make extract && build byte-check. Idempotent. Proven on SC01/005, /006, SC03/001.
• config/overlays.mk — new (generated, committed). -included by the Makefile before BINARIES := (so BINARIES := main resident $(OVERLAY_BINARIES)); holds the 4 onboarded overlays' var blocks. The Makefile body is never hand-edited to add an overlay.
• Makefile — modified. The overlays.mk indirection; the build/assets/%.o incbin rule (+ --set-section-alignment .data=1) + ASSET_OBJS prereq for splat bin subsegs; the objcopy end-align TRIM (shrink-only ≤3 B, removes splat's . = ALIGN(.,4) segment-end pad for non-4-aligned overlays — never touches the 4-aligned EXE/resident); build-all/check-all (recursive $(MAKE) BINARY=<b> over $(BINARIES), one PASS/FAIL); clean removes assets/.
• .gitignore — /assets/ (regenerable splat output).
• diff_settings.py, tools/{progress,difficulty,dup_report}.py — modified. Each gained the 4 overlay BINARIES dict entries + the # <<< overlays: new_overlay.sh inserts … >>> sentinel anchor.

The representative overlay set (4 binaries, all byte-identical):

• config/splat.ov_SC01_077.yaml + …ov_SC01_005/006/SC03_001.yaml, config/check.ov_*.sha, config/symbols.ov_*.txt — new. The 4 onboarded overlays' configs (SC01_077 inlines the bin carve; the others generated by new_overlay.sh).
• src/ov_SC01_077/ov_SC01_077.c — the matched-C record: REAL 1→704 (was 100% INCLUDE_ASM). All hand-written C, each byte-gated by harvest_verify.py. ~1815 stubs remain (the Phase-15 hard tail).
• src/shared/ov_setters.h — new. Generic SETTER/RETCONST macro bodies (cookbook §11 source-level share), reusable fleet-wide.
• src/ov_SC01_005/ov_SC01_005.c + …006.c — the dup-pair collapse: 3 accessors instantiated from the one src/shared/ov_setters.h (via #include "../shared/…", in-place to preserve address order). REAL 3 each.
• src/ov_SC03_001/ov_SC03_001.c — new (100% INCLUDE_ASM record).
• config/dedup.us.yaml — modified. D1/D2/D3 cross-overlay share groups (members in BOTH 005 & 006).

Docs / knowledge base:

• docs/matching-cookbook.md §13 — "Add a location overlay — the canonical runbook" (new_overlay.sh, the flat-blob config, the non-4-aligned bin-carve/asset/trim gotcha, the A→B→C workflow, dedup-credit two shapes, build-all/check-all). The Phase-15 fleet recipe.
• docs/effort-map.md — the per-task effort-mapping + pause-at-every-transition discipline (R27) + the T7 experiment Evidence (Max≈Ultracode yield → prefer Ultracode for harvests).
• docs/SETUP.md — §2.8 MCP client-reconnect note (R29) + new_overlay.sh in the tooling inventory (R21).
• docs/gen2-roadmap.md — Phase-13 milestone ACHIEVED amendment.
• docs/{progress,difficulty,duplicates}.ov_*.md — new, committed per-overlay digests.
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase13.md (R19 archive). phase-ends/PhaseEnd_Phase13.md — this file.

Local artifacts (gitignored / regenerable): asm/<ov>/**, assets/<ov>/*.bin, build/, expected/, .run/drafts-ov077/*, .run/drafts-exp-{A,B}/*, .run/sig.ov_*.jsonl, .run/<ov>_funcs.txt, the effort-experiment logs (.run/arm{A,B}*.txt). The 4 overlay Ghidra programs are script-reproducible (ghidra_import_raw.sh + DefineFunctions.java); DB-commit deferred as regenerable (no manual RE on them; avoids ~14 MB/overlay bloat).

Tools/packages installed: None — the entire phase used the existing Phase-4/6/10/11/12 toolchain.

Verification results (literal) — the milestone proof (R22 CLEAN fleet rebuild, all green):

• make clean && (make extract BINARY=<b> for all 6) && make check-all → 6 passed, 0 failed of 6: main 143dbb89…, resident 8e17e02f…, ov_SC01_077 d19c9580…, ov_SC01_005 56760dbe…, ov_SC01_006 56760dbe…, ov_SC03_001 f8fd92f5… — all BYTE-IDENTICAL from a fully clean tree. Negative control (corrupt one sha) → [FAIL] + "5 passed, 1 failed" + nonzero exit.
• Harvest: ov_SC01_077 REAL 704/2586 (Ultracode §12, 5 passes: 110+205+217+52(throttled)+106 + 14 from the T7 bank). Fleet REAL 887 (main 54 + resident 123 + ov_SC01_077 704 + 005/006 3+3 + SC03_001 0).
• Dup-pair collapse: dedup_integrate --check → 4 validated, 0 failed (I0 + D1/D2/D3); 005 & 006 both 56760dbe… from the one src/shared/ov_setters.h; REAL 3 credited to both.
• T7 effort experiment (controlled): 80 fixed fns, identical Workflow, only session effort varied → Arm A (Ultracode/xHigh) 13/80, Arm B (Max) 14/80, A∩B=12 → yield difference is noise; per-agent depth doesn't change blind-draft harvest yield. (The "pass-1-at-Max-did-better" observation was difficulty-confounded.)
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged (asm/, assets/, build/, .run/ gitignored). No Ghidra DB committed (overlay DBs regenerable).

Milestone achieved (confirmed by Drew, gate 2): 6 binaries rebuild byte-for-byte from a clean tree (check-all 6/6) — a representative location-overlay set matched end-to-end (SC01/077 byte-identical + 704 functions; the SC01/005 ≡ 006 duplicate pair collapsed to one shared source, both green; the large free-roam SC03/001 byte-identical), the cross-binary dedup mechanism proven on the dup pair, and a one-command fleet runbook in place (tools/new_overlay.sh + make build-all/check-all). The ~134-overlay long-tail is the open-ended Phase 15 (not a gate).

Next: Phase 14 — Public flip via a curated public repo (Gen2 EXIT capstone) per the gen2-roadmap critical path (8→9→10→11→{12,13}→14). Alternatively/concurrently Phase 15 — overlay long-tail (open-ended, not a gate) can grind down more overlays first. Whether to go public now vs. harvest more overlays first is a Phase-Start decision for Drew. Start a fresh session (effort Max, plan mode).

Deviations

Item	Plan	Actual	Reason
Non-4-aligned overlays	(not anticipated)	a 3-part fix baked into the template + Makefile (bin carve + incbin asset rule + objcopy trim)	~75% of overlays aren't 4-aligned; spimdisasm drops the trailing partial word, the asset needs a build rule + .data align fix, and splat's .ld pads the segment end. The resident (4-aligned) never hit it.
Harvest depth	"harvest the unique remainder"	704 matched (the easy/medium tier, 5 Ultracode passes), then consolidated	yield fell 92%→54% as difficulty rose; the hard tail is genuine compiler-residual / large-function work = Phase 15. "Bank, don't strand."
Pass 4	match_one iterate pass	rate-limited (20/36 batches lost to a transient server throttle) → +52 only; resumed with paced blind-draft + retry-wave	server-side throttle on the token-heavy iterate fan-out; the retry-wave (§12) recovered cleanly on pass 5
Dup-pair credit	"fork psyq_integrate" (roadmap §11 sketch)	source-level macro share (src/shared/ + dedup.us.yaml), the proven §11 mechanism	game-code fns are interior to one object per binary; the object-swap is library-only (Phase 11 D1). The macro must be instantiated IN PLACE (overlays link in source order).
Ghidra DB	(commit per R23)	deferred as regenerable (overlay programs are script-reproducible; no manual RE)	avoids ~14 MB/overlay .git bloat; the import is reproducible from committed tooling (Phase-10 precedent)
T7 experiment	(not in the original plan)	added (Drew) — Max vs Ultracode yield, controlled	Drew's question after I ran pass 1 at Max by mistake; settled it empirically (= noise)
scriptPath args	(forward args)	the scriptPath re-invoke didn't forward args.draftdir → Arm B overwrote Arm A's drafts dir	a Workflow-tool gotcha; harmless here (Arm A was already gated/recorded before the overwrite)

Commit Message

(Phase 13 landed as ~18 per-task/per-batch checkpoint commits 50c5c9ea3..52514b5f4 + this PhaseEnd.)

feat(phase-13): representative location overlays — 6 binaries byte-identical + fleet runbook (v1.13.0)

- T0: overlay splat template + generated config/overlays.mk indirection (Makefile body untouched)
- T1: ov_SC01_077 byte-identical d19c9580 + REAL 1->704 (Ultracode §12, 5 passes) + Ghidra import;
  reusable non-4-aligned-overlay handling (bin carve + incbin asset rule + objcopy end-align trim)
- T2: dup-pair SC01/005 ≡ 006 collapsed to ONE shared source (src/shared/ov_setters.h +
  dedup.us.yaml D1/D2/D3); both 56760dbe; dedup_integrate 4 validated/0 failed
- T3: free-roam ov_SC03_001 byte-identical f8fd92f5 (recipe scales)
- T4: tools/new_overlay.sh one-command onboarding (idempotent; proven on 3 overlays)
- T5: make build-all/check-all (6/6 clean-fleet R22 proof; negative control fails loud)
- T7: controlled Max-vs-Ultracode harvest experiment = noise (14 vs 13) -> prefer Ultracode for breadth
- cookbook §13 overlay runbook; SETUP/gen2-roadmap/effort-map; rules R27-R29
- MILESTONE (gate 2): 6 binaries byte-identical from clean; fleet REAL 887; dedup proven; runbook live
- bumps project version 1.12.0 -> 1.13.0

Rules Added This Phase

Rule	Reason
R27 — Effort-transition discipline (pause+prompt at EVERY boundary). Effort is mapped per-task at plan time AND re-evaluated continuously; at every transition — into Ultracode (mid-task breadth revealed), back to Max (breadth done), and at task hand-off (next task doesn't need the current mode) — STOP, prompt Drew to toggle, and WAIT for the actual /effort command (a system-reminder confirms). Never launch a Workflow on a verbal "yes" before the toggle lands; never grind deep synthesis at Ultracode/xHigh. Claude cannot toggle effort itself. Governing doc docs/effort-map.md; memory effort-prompt-ultracode-on-breadth. Extends R7/R26.	Two failures this phase: launched the harvest Workflow on Drew's verbal "Enable Ultracode" before he toggled it; and continued from the Ultracode harvest straight into the deep T2–T6 work without prompting the switch back to Max (Drew had to ask).
R28 — Build the harness task list right after plan approval, before starting work. On plan approval (ExitPlanMode), create the task list (one TaskCreate per plan task) so Drew has a live, monitorable view; mark the first task in_progress and proceed. In addition to CURRENT_PHASE.md (the committable crash-recovery log), not instead. Memory build-tasklist-after-plan-approval.	Drew (2026-06-16): "build tasklist so I can monitor progress." He tracks progress via the task list, not the prose plan.
R29 — Pause + prompt for /mcp reconnect after an MCP server restart / program switch. Restarting the headless MCP server (R23 import cycle) or restarting it to serve a different program drops the Claude Code client's SSE connection → every mcp__ghidra__* call times out until the client reconnects. Claude can't run /mcp. So after any server restart / program switch: pause, ask Drew to run /mcp, then one cheap get_binary_info (G2) before continuing — don't try calls, hit timeouts, and work around them. SETUP §2.8; memory mcp-reconnect-after-restart.	Restarted MCP to serve the freshly-imported ov_SC01_077, my calls timed out, and I started working around it instead of pausing to ask for the reconnect.

(Not elevated to rules — captured as cookbook §13 techniques: the non-4-aligned-overlay 3-part fix, the overlays.mk indirection, the per-function vs whole-overlay dedup shapes, the address-order-preserving in-place macro instantiation, the sig_image-vs-splat name-case / addr-keyed validator detail.)

PhaseEnd Changelog

v1.12.0 → v1.13.0 — Phase 13 complete (Gen2 phase 6). The project rebuilds 6 binaries byte-for-byte from a clean tree (was 2): the EXE, the resident engine, and a representative location-overlay set — SC01/077 (byte-identical + 704 functions matched via an Ultracode parallel-draft/byte-gate harvest), the byte-identical duplicate pair SC01/005 ≡ 006 collapsed to one shared source (proving the cross-binary dedup mechanism end-to-end), and the large free-roam SC03/001. A reusable one-command fleet runbook is in place: tools/new_overlay.sh (idempotent overlay onboarding via a flat-blob template + the generated config/overlays.mk indirection) and make build-all/check-all. The phase's hardest-won reusable lesson is the non-4-aligned-overlay handling (≈75% of the fleet): spimdisasm drops the trailing partial word, so a bin carve + an incbin asset rule (+ .data align fix) + an objcopy end-align trim recover the exact bytes. A controlled Max-vs-Ultracode experiment (T7) found the two efforts produce equal harvest yield (noise) → Ultracode is preferred for breadth harvests; Max for deep synthesis. Three process rules (R27 effort-transition discipline, R28 task-list-after-plan, R29 MCP-reconnect). The ~134-overlay long-tail is the open-ended Phase 15.

Plain-English Recap

The game's first playable "level" now rebuilds perfectly, bit-for-bit, from our own source and is about a quarter decompiled — and so do a duplicate level and a big open level, so six pieces of the game now reconstruct exactly (up from two at the start of the session). The biggest practical win is a one-command tool that adds any of the remaining ~130 levels to the build in seconds, plus a single command that re-checks every piece at once. We also proved that when two levels are identical, we can rebuild both from one shared source file — which is how the project will avoid re-doing the same work 130 times. Along the way we hit and permanently solved a fiddly snag that affects most levels (their size isn't a round number, so the tools were dropping the last few bytes). The level's functions were matched by a swarm of AI agents drafting code against an automatic bit-for-bit referee — and we ran a clean side experiment that answered a real question: running that swarm at maximum reasoning depth did not beat the faster parallel mode (same results), so we'll use the faster mode for this kind of bulk work and save the deep mode for hard thinking. The remaining levels are a big, optional to-do list — deliberately not blocking the project.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase13.md (R19). The Phase-13 work is committed (~18 checkpoints 50c5c9ea3..52514b5f4 + this PhaseEnd commit); Drew pushes (R6). Gen2 continues — do NOT start the next phase here. Start a fresh session (effort Max, plan mode) to plan Phase 14 (public flip — Gen2 EXIT) or Phase 15 (overlay long-tail) — Drew's choice at the Phase-Start gate. Keep this file forever.