mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
Drew T
a0865e78ca
Milestone confirmed (gate 2): make check-all 6/6 from clean (R22); ov_SC01_077 704 matched, dup-pair collapse, free-roam, new_overlay.sh + build-all/check-all. Rules R27 (effort-transition discipline), R28 (task-list-after-plan), R29 (MCP reconnect). CURRENT_PHASE.md archived -> phase-ends/logs/Phase13.md (R19).
170 lines
17 KiB
Markdown
170 lines
17 KiB
Markdown
# PhaseEnd — Phase 13: Representative location overlays (end-to-end + fleet runbook)
|
||
**Date:** 2026-06-16 · **Project Version:** 1.13.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (6th phase)
|
||
|
||
> Gen2 phase 6 of the arc (8→9→10→11→12→**13**→14→15). The granular per-task trail (the harvest pass-by-pass
|
||
> logs, the non-4-aligned-overlay diagnostics, the dedup sig-hash hunt, the effort-experiment runs) is preserved
|
||
> on-demand at **`phase-ends/logs/Phase13.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism).
|
||
> Owner decisions (2026-06-16): scope = **"Representative + credit"** (not the full 134-sweep — that's Phase 15);
|
||
> **import the representative overlays into Ghidra**; the harvest ran under **Ultracode** (Drew enabled it); a
|
||
> controlled **Max-vs-Ultracode effort experiment** (T7) was added as the final task.
|
||
|
||
## Build Log
|
||
|
||
**Files created/changed and complete — do not recreate:**
|
||
|
||
*Reusable overlay machinery (the Phase-15 fleet recipe):*
|
||
- `config/splat.us.overlay.template.yaml` — **new.** The flat-blob splat template for any location overlay
|
||
(`@TOKEN@` placeholders, unquoted so `vram` lands as an int): no header/gp_value (-G0), single `code` seg @
|
||
vram `0x80128158`, `asset_path: assets/@ALIAS@`, stacked `[symbols.us, symbols.resident, symbols.@ALIAS@]`,
|
||
`@TRAILING@` marker for the non-4-aligned `bin` carve. Code at file 0x0 (overlays open with a prologue).
|
||
- `tools/new_overlay.sh` — **new.** One-command onboarding `<SCxx> <FILE_nnn>`: sha1/size/`code_end` (sig_image)
|
||
→ instantiate the config (+ inject the `bin` carve when `size%4≠0`) → `config/check.<ov>.sha` + empty
|
||
`config/symbols.<ov>.txt` → append the `<ov>_*` block to the generated `config/overlays.mk` + `OVERLAY_BINARIES`
|
||
→ sentinel-insert the 4 report/diff dicts (grep-guarded + `ast.parse`-checked) → `make extract && build`
|
||
byte-check. **Idempotent.** Proven on SC01/005, /006, SC03/001.
|
||
- `config/overlays.mk` — **new (generated, committed).** `-include`d by the Makefile **before** `BINARIES :=`
|
||
(so `BINARIES := main resident $(OVERLAY_BINARIES)`); holds the 4 onboarded overlays' var blocks. The Makefile
|
||
body is never hand-edited to add an overlay.
|
||
- `Makefile` — **modified.** The `overlays.mk` indirection; the `build/assets/%.o` incbin rule (+
|
||
`--set-section-alignment .data=1`) + `ASSET_OBJS` prereq for splat `bin` subsegs; the objcopy **end-align TRIM**
|
||
(shrink-only ≤3 B, removes splat's `. = ALIGN(.,4)` segment-end pad for non-4-aligned overlays — never touches
|
||
the 4-aligned EXE/resident); **`build-all`/`check-all`** (recursive `$(MAKE) BINARY=<b>` over `$(BINARIES)`,
|
||
one PASS/FAIL); `clean` removes `assets/`.
|
||
- `.gitignore` — `/assets/` (regenerable splat output).
|
||
- `diff_settings.py`, `tools/{progress,difficulty,dup_report}.py` — **modified.** Each gained the 4 overlay
|
||
`BINARIES` dict entries + the `# <<< overlays: new_overlay.sh inserts … >>>` sentinel anchor.
|
||
|
||
*The representative overlay set (4 binaries, all byte-identical):*
|
||
- `config/splat.ov_SC01_077.yaml` + `…ov_SC01_005/006/SC03_001.yaml`, `config/check.ov_*.sha`,
|
||
`config/symbols.ov_*.txt` — **new.** The 4 onboarded overlays' configs (SC01_077 inlines the `bin` carve; the
|
||
others generated by new_overlay.sh).
|
||
- `src/ov_SC01_077/ov_SC01_077.c` — **the matched-C record: REAL 1→704** (was 100% INCLUDE_ASM). All hand-written
|
||
C, each byte-gated by `harvest_verify.py`. ~1815 stubs remain (the Phase-15 hard tail).
|
||
- `src/shared/ov_setters.h` — **new.** Generic `SETTER`/`RETCONST` macro bodies (cookbook §11 source-level share),
|
||
reusable fleet-wide.
|
||
- `src/ov_SC01_005/ov_SC01_005.c` + `…006.c` — **the dup-pair collapse:** 3 accessors instantiated from the one
|
||
`src/shared/ov_setters.h` (via `#include "../shared/…"`, in-place to preserve address order). REAL 3 each.
|
||
- `src/ov_SC03_001/ov_SC03_001.c` — **new** (100% INCLUDE_ASM record).
|
||
- `config/dedup.us.yaml` — **modified.** D1/D2/D3 cross-overlay share groups (members in BOTH 005 & 006).
|
||
|
||
*Docs / knowledge base:*
|
||
- `docs/matching-cookbook.md` **§13** — "Add a location overlay — the canonical runbook" (new_overlay.sh, the
|
||
flat-blob config, the non-4-aligned bin-carve/asset/trim gotcha, the A→B→C workflow, dedup-credit two shapes,
|
||
build-all/check-all). The Phase-15 fleet recipe.
|
||
- `docs/effort-map.md` — the **per-task effort-mapping + pause-at-every-transition discipline** (R27) + the **T7
|
||
experiment Evidence** (Max≈Ultracode yield → prefer Ultracode for harvests).
|
||
- `docs/SETUP.md` — §2.8 MCP **client-reconnect** note (R29) + `new_overlay.sh` in the tooling inventory (R21).
|
||
- `docs/gen2-roadmap.md` — Phase-13 milestone **ACHIEVED** amendment.
|
||
- `docs/{progress,difficulty,duplicates}.ov_*.md` — **new, committed** per-overlay digests.
|
||
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase13.md`** (R19 archive). `phase-ends/PhaseEnd_Phase13.md` — this file.
|
||
|
||
**Local artifacts (gitignored / regenerable):** `asm/<ov>/**`, `assets/<ov>/*.bin`, `build/`, `expected/`,
|
||
`.run/drafts-ov077/*`, `.run/drafts-exp-{A,B}/*`, `.run/sig.ov_*.jsonl`, `.run/<ov>_funcs.txt`, the effort-experiment
|
||
logs (`.run/arm{A,B}*.txt`). The 4 overlay **Ghidra programs** are script-reproducible (`ghidra_import_raw.sh` +
|
||
`DefineFunctions.java`); **DB-commit deferred as regenerable** (no manual RE on them; avoids ~14 MB/overlay bloat).
|
||
|
||
**Tools/packages installed:** None — the entire phase used the existing Phase-4/6/10/11/12 toolchain.
|
||
|
||
**Verification results (literal) — the milestone proof (R22 CLEAN fleet rebuild, all green):**
|
||
- **`make clean && (make extract BINARY=<b> for all 6) && make check-all`** → **6 passed, 0 failed of 6:** main
|
||
`143dbb89…`, resident `8e17e02f…`, ov_SC01_077 `d19c9580…`, ov_SC01_005 `56760dbe…`, ov_SC01_006 `56760dbe…`,
|
||
ov_SC03_001 `f8fd92f5…` — all **BYTE-IDENTICAL** from a fully clean tree. Negative control (corrupt one sha) →
|
||
`[FAIL]` + "5 passed, 1 failed" + nonzero exit.
|
||
- **Harvest:** `ov_SC01_077` **REAL 704/2586** (Ultracode §12, 5 passes: 110+205+217+52(throttled)+106 + 14 from
|
||
the T7 bank). Fleet REAL **887** (main 54 + resident 123 + ov_SC01_077 704 + 005/006 3+3 + SC03_001 0).
|
||
- **Dup-pair collapse:** `dedup_integrate --check` → **4 validated, 0 failed** (I0 + D1/D2/D3); 005 & 006 both
|
||
`56760dbe…` from the one `src/shared/ov_setters.h`; REAL 3 credited to both.
|
||
- **T7 effort experiment (controlled):** 80 fixed fns, identical Workflow, only session effort varied →
|
||
**Arm A (Ultracode/xHigh) 13/80, Arm B (Max) 14/80, A∩B=12** → yield difference is **noise**; per-agent depth
|
||
doesn't change blind-draft harvest yield. (The "pass-1-at-Max-did-better" observation was difficulty-confounded.)
|
||
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk
|
||
staged** (`asm/`, `assets/`, `build/`, `.run/` gitignored). No Ghidra DB committed (overlay DBs regenerable).
|
||
|
||
**Milestone achieved (confirmed by Drew, gate 2):** **6 binaries rebuild byte-for-byte from a clean tree**
|
||
(`check-all` 6/6) — a representative location-overlay set matched end-to-end (`SC01/077` byte-identical + 704
|
||
functions; the `SC01/005 ≡ 006` duplicate pair collapsed to one shared source, both green; the large free-roam
|
||
`SC03/001` byte-identical), the **cross-binary dedup mechanism proven** on the dup pair, and a **one-command fleet
|
||
runbook** in place (`tools/new_overlay.sh` + `make build-all`/`check-all`). The ~134-overlay long-tail is the
|
||
open-ended Phase 15 (not a gate).
|
||
|
||
**Next:** **Phase 14 — Public flip via a curated public repo (Gen2 EXIT capstone)** per the gen2-roadmap critical
|
||
path (8→9→10→11→{12,13}→14). Alternatively/concurrently **Phase 15 — overlay long-tail** (open-ended, not a gate)
|
||
can grind down more overlays first. Whether to go public now vs. harvest more overlays first is a Phase-Start
|
||
decision for Drew. Start a fresh session (effort **Max**, plan mode).
|
||
|
||
## Deviations
|
||
| Item | Plan | Actual | Reason |
|
||
|---|---|---|---|
|
||
| Non-4-aligned overlays | (not anticipated) | a **3-part fix** baked into the template + Makefile (bin carve + incbin asset rule + objcopy trim) | ~75% of overlays aren't 4-aligned; spimdisasm drops the trailing partial word, the asset needs a build rule + .data align fix, and splat's `.ld` pads the segment end. The resident (4-aligned) never hit it. |
|
||
| Harvest depth | "harvest the unique remainder" | **704 matched** (the easy/medium tier, 5 Ultracode passes), then consolidated | yield fell 92%→54% as difficulty rose; the hard tail is genuine compiler-residual / large-function work = Phase 15. "Bank, don't strand." |
|
||
| Pass 4 | match_one iterate pass | **rate-limited** (20/36 batches lost to a transient server throttle) → +52 only; resumed with paced blind-draft + retry-wave | server-side throttle on the token-heavy iterate fan-out; the retry-wave (§12) recovered cleanly on pass 5 |
|
||
| Dup-pair credit | "fork psyq_integrate" (roadmap §11 sketch) | **source-level macro share** (`src/shared/` + `dedup.us.yaml`), the proven §11 mechanism | game-code fns are interior to one object per binary; the object-swap is library-only (Phase 11 D1). The macro must be instantiated IN PLACE (overlays link in source order). |
|
||
| Ghidra DB | (commit per R23) | **deferred as regenerable** (overlay programs are script-reproducible; no manual RE) | avoids ~14 MB/overlay `.git` bloat; the import is reproducible from committed tooling (Phase-10 precedent) |
|
||
| T7 experiment | (not in the original plan) | **added** (Drew) — Max vs Ultracode yield, controlled | Drew's question after I ran pass 1 at Max by mistake; settled it empirically (= noise) |
|
||
| scriptPath args | (forward `args`) | the `scriptPath` re-invoke **didn't forward `args.draftdir`** → Arm B overwrote Arm A's drafts dir | a Workflow-tool gotcha; harmless here (Arm A was already gated/recorded before the overwrite) |
|
||
|
||
## Commit Message
|
||
```
|
||
(Phase 13 landed as ~18 per-task/per-batch checkpoint commits commit:0084..commit:0099 + this PhaseEnd.)
|
||
|
||
feat(phase-13): representative location overlays — 6 binaries byte-identical + fleet runbook (v1.13.0)
|
||
|
||
- T0: overlay splat template + generated config/overlays.mk indirection (Makefile body untouched)
|
||
- T1: ov_SC01_077 byte-identical d19c9580 + REAL 1->704 (Ultracode §12, 5 passes) + Ghidra import;
|
||
reusable non-4-aligned-overlay handling (bin carve + incbin asset rule + objcopy end-align trim)
|
||
- T2: dup-pair SC01/005 ≡ 006 collapsed to ONE shared source (src/shared/ov_setters.h +
|
||
dedup.us.yaml D1/D2/D3); both 56760dbe; dedup_integrate 4 validated/0 failed
|
||
- T3: free-roam ov_SC03_001 byte-identical f8fd92f5 (recipe scales)
|
||
- T4: tools/new_overlay.sh one-command onboarding (idempotent; proven on 3 overlays)
|
||
- T5: make build-all/check-all (6/6 clean-fleet R22 proof; negative control fails loud)
|
||
- T7: controlled Max-vs-Ultracode harvest experiment = noise (14 vs 13) -> prefer Ultracode for breadth
|
||
- cookbook §13 overlay runbook; SETUP/gen2-roadmap/effort-map; rules R27-R29
|
||
- MILESTONE (gate 2): 6 binaries byte-identical from clean; fleet REAL 887; dedup proven; runbook live
|
||
- bumps project version 1.12.0 -> 1.13.0
|
||
```
|
||
|
||
## Rules Added This Phase
|
||
| Rule | Reason |
|
||
|---|---|
|
||
| **R27 — Effort-transition discipline (pause+prompt at EVERY boundary).** Effort is mapped per-task at plan time AND re-evaluated continuously; at every transition — into Ultracode (mid-task breadth revealed), back to Max (breadth done), and at task hand-off (next task doesn't need the current mode) — **STOP, prompt Drew to toggle, and WAIT for the actual `/effort` command** (a system-reminder confirms). Never launch a Workflow on a verbal "yes" before the toggle lands; never grind deep synthesis at Ultracode/xHigh. Claude cannot toggle effort itself. Governing doc `docs/effort-map.md`; memory `effort-prompt-ultracode-on-breadth`. Extends R7/R26. | Two failures this phase: launched the harvest Workflow on Drew's verbal "Enable Ultracode" before he toggled it; and continued from the Ultracode harvest straight into the deep T2–T6 work without prompting the switch back to Max (Drew had to ask). |
|
||
| **R28 — Build the harness task list right after plan approval, before starting work.** On plan approval (ExitPlanMode), create the task list (one TaskCreate per plan task) so Drew has a live, monitorable view; mark the first task in_progress and proceed. In addition to `CURRENT_PHASE.md` (the committable crash-recovery log), not instead. Memory `build-tasklist-after-plan-approval`. | Drew (2026-06-16): "build tasklist so I can monitor progress." He tracks progress via the task list, not the prose plan. |
|
||
| **R29 — Pause + prompt for `/mcp` reconnect after an MCP server restart / program switch.** Restarting the headless MCP server (R23 import cycle) or restarting it to serve a different program drops the Claude Code client's SSE connection → every `mcp__ghidra__*` call times out until the client reconnects. Claude can't run `/mcp`. So after any server restart / program switch: **pause, ask Drew to run `/mcp`, then one cheap `get_binary_info` (G2) before continuing** — don't try calls, hit timeouts, and work around them. SETUP §2.8; memory `mcp-reconnect-after-restart`. | Restarted MCP to serve the freshly-imported `ov_SC01_077`, my calls timed out, and I started working around it instead of pausing to ask for the reconnect. |
|
||
|
||
*(Not elevated to rules — captured as cookbook §13 techniques: the non-4-aligned-overlay 3-part fix, the
|
||
`overlays.mk` indirection, the per-function vs whole-overlay dedup shapes, the address-order-preserving in-place
|
||
macro instantiation, the sig_image-vs-splat name-case / addr-keyed validator detail.)*
|
||
|
||
## PhaseEnd Changelog
|
||
**v1.12.0 → v1.13.0 — Phase 13 complete (Gen2 phase 6).** The project rebuilds **6 binaries byte-for-byte from a
|
||
clean tree** (was 2): the EXE, the resident engine, and a **representative location-overlay set** — `SC01/077`
|
||
(byte-identical + **704 functions** matched via an Ultracode parallel-draft/byte-gate harvest), the
|
||
**byte-identical duplicate pair `SC01/005 ≡ 006`** collapsed to **one shared source** (proving the cross-binary
|
||
dedup mechanism end-to-end), and the large free-roam `SC03/001`. A reusable **one-command fleet runbook** is in
|
||
place: `tools/new_overlay.sh` (idempotent overlay onboarding via a flat-blob template + the generated
|
||
`config/overlays.mk` indirection) and `make build-all`/`check-all`. The phase's hardest-won reusable lesson is the
|
||
**non-4-aligned-overlay handling** (≈75% of the fleet): spimdisasm drops the trailing partial word, so a `bin`
|
||
carve + an incbin asset rule (+ `.data` align fix) + an objcopy end-align trim recover the exact bytes. A
|
||
controlled **Max-vs-Ultracode experiment** (T7) found the two efforts produce **equal harvest yield** (noise) →
|
||
Ultracode is preferred for breadth harvests; Max for deep synthesis. Three process rules (R27 effort-transition
|
||
discipline, R28 task-list-after-plan, R29 MCP-reconnect). The ~134-overlay long-tail is the open-ended Phase 15.
|
||
|
||
## Plain-English Recap
|
||
The game's first playable "level" now rebuilds **perfectly, bit-for-bit, from our own source** and is about a
|
||
quarter decompiled — and so do a duplicate level and a big open level, so **six pieces of the game** now
|
||
reconstruct exactly (up from two at the start of the session). The biggest practical win is a **one-command tool**
|
||
that adds any of the remaining ~130 levels to the build in seconds, plus a single command that re-checks every
|
||
piece at once. We also proved that when two levels are identical, we can rebuild both from **one shared source
|
||
file** — which is how the project will avoid re-doing the same work 130 times. Along the way we hit and
|
||
permanently solved a fiddly snag that affects most levels (their size isn't a round number, so the tools were
|
||
dropping the last few bytes). The level's functions were matched by a **swarm of AI agents drafting code against
|
||
an automatic bit-for-bit referee** — and we ran a clean side experiment that answered a real question: running
|
||
that swarm at maximum reasoning depth did **not** beat the faster parallel mode (same results), so we'll use the
|
||
faster mode for this kind of bulk work and save the deep mode for hard thinking. The remaining levels are a big,
|
||
optional to-do list — deliberately not blocking the project.
|
||
|
||
## 🛑 Stop Here
|
||
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase13.md` (R19). The Phase-13 work is committed
|
||
(~18 checkpoints `commit:0084`..`commit:0099` + this PhaseEnd commit); **Drew pushes** (R6). Gen2 continues — do **NOT**
|
||
start the next phase here. Start a **fresh session** (effort **Max**, plan mode) to plan **Phase 14 (public flip —
|
||
Gen2 EXIT)** or **Phase 15 (overlay long-tail)** — Drew's choice at the Phase-Start gate. Keep this file forever.
|