From a0865e78cac54d7397502f69198532c8e2c8aa72 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 16 Jun 2026 18:00:25 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-13):=20PhaseEnd=20=E2=80=94=20repres?= =?UTF-8?q?entative=20overlays=20complete;=206=20binaries=20byte-identical?= =?UTF-8?q?=20(v1.13.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone confirmed (gate 2): make check-all 6/6 from clean (R22); ov_SC01_077 704 matched, dup-pair collapse, free-roam, new_overlay.sh + build-all/check-all. Rules R27 (effort-transition discipline), R28 (task-list-after-plan), R29 (MCP reconnect). CURRENT_PHASE.md archived -> phase-ends/logs/Phase13.md (R19). --- phase-ends/PhaseEnd_Phase13.md | 169 ++++++++++++++++++ .../{CURRENT_PHASE.md => logs/Phase13.md} | 0 2 files changed, 169 insertions(+) create mode 100644 phase-ends/PhaseEnd_Phase13.md rename phase-ends/{CURRENT_PHASE.md => logs/Phase13.md} (100%) diff --git a/phase-ends/PhaseEnd_Phase13.md b/phase-ends/PhaseEnd_Phase13.md new file mode 100644 index 0000000000..ab61c49420 --- /dev/null +++ b/phase-ends/PhaseEnd_Phase13.md @@ -0,0 +1,169 @@ +# PhaseEnd — Phase 13: Representative location overlays (end-to-end + fleet runbook) +**Date:** 2026-06-16 · **Project Version:** 1.13.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (6th phase) + +> Gen2 phase 6 of the arc (8→9→10→11→12→**13**→14→15). The granular per-task trail (the harvest pass-by-pass +> logs, the non-4-aligned-overlay diagnostics, the dedup sig-hash hunt, the effort-experiment runs) is preserved +> on-demand at **`phase-ends/logs/Phase13.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). +> Owner decisions (2026-06-16): scope = **"Representative + credit"** (not the full 134-sweep — that's Phase 15); +> **import the representative overlays into Ghidra**; the harvest ran under **Ultracode** (Drew enabled it); a +> controlled **Max-vs-Ultracode effort experiment** (T7) was added as the final task. + +## Build Log + +**Files created/changed and complete — do not recreate:** + +*Reusable overlay machinery (the Phase-15 fleet recipe):* +- `config/splat.us.overlay.template.yaml` — **new.** The flat-blob splat template for any location overlay + (`@TOKEN@` placeholders, unquoted so `vram` lands as an int): no header/gp_value (-G0), single `code` seg @ + vram `0x80128158`, `asset_path: assets/@ALIAS@`, stacked `[symbols.us, symbols.resident, symbols.@ALIAS@]`, + `@TRAILING@` marker for the non-4-aligned `bin` carve. Code at file 0x0 (overlays open with a prologue). +- `tools/new_overlay.sh` — **new.** One-command onboarding ` `: sha1/size/`code_end` (sig_image) + → instantiate the config (+ inject the `bin` carve when `size%4≠0`) → `config/check..sha` + empty + `config/symbols..txt` → append the `_*` block to the generated `config/overlays.mk` + `OVERLAY_BINARIES` + → sentinel-insert the 4 report/diff dicts (grep-guarded + `ast.parse`-checked) → `make extract && build` + byte-check. **Idempotent.** Proven on SC01/005, /006, SC03/001. +- `config/overlays.mk` — **new (generated, committed).** `-include`d by the Makefile **before** `BINARIES :=` + (so `BINARIES := main resident $(OVERLAY_BINARIES)`); holds the 4 onboarded overlays' var blocks. The Makefile + body is never hand-edited to add an overlay. +- `Makefile` — **modified.** The `overlays.mk` indirection; the `build/assets/%.o` incbin rule (+ + `--set-section-alignment .data=1`) + `ASSET_OBJS` prereq for splat `bin` subsegs; the objcopy **end-align TRIM** + (shrink-only ≤3 B, removes splat's `. = ALIGN(.,4)` segment-end pad for non-4-aligned overlays — never touches + the 4-aligned EXE/resident); **`build-all`/`check-all`** (recursive `$(MAKE) BINARY=` over `$(BINARIES)`, + one PASS/FAIL); `clean` removes `assets/`. +- `.gitignore` — `/assets/` (regenerable splat output). +- `diff_settings.py`, `tools/{progress,difficulty,dup_report}.py` — **modified.** Each gained the 4 overlay + `BINARIES` dict entries + the `# <<< overlays: new_overlay.sh inserts … >>>` sentinel anchor. + +*The representative overlay set (4 binaries, all byte-identical):* +- `config/splat.ov_SC01_077.yaml` + `…ov_SC01_005/006/SC03_001.yaml`, `config/check.ov_*.sha`, + `config/symbols.ov_*.txt` — **new.** The 4 onboarded overlays' configs (SC01_077 inlines the `bin` carve; the + others generated by new_overlay.sh). +- `src/ov_SC01_077/ov_SC01_077.c` — **the matched-C record: REAL 1→704** (was 100% INCLUDE_ASM). All hand-written + C, each byte-gated by `harvest_verify.py`. ~1815 stubs remain (the Phase-15 hard tail). +- `src/shared/ov_setters.h` — **new.** Generic `SETTER`/`RETCONST` macro bodies (cookbook §11 source-level share), + reusable fleet-wide. +- `src/ov_SC01_005/ov_SC01_005.c` + `…006.c` — **the dup-pair collapse:** 3 accessors instantiated from the one + `src/shared/ov_setters.h` (via `#include "../shared/…"`, in-place to preserve address order). REAL 3 each. +- `src/ov_SC03_001/ov_SC03_001.c` — **new** (100% INCLUDE_ASM record). +- `config/dedup.us.yaml` — **modified.** D1/D2/D3 cross-overlay share groups (members in BOTH 005 & 006). + +*Docs / knowledge base:* +- `docs/matching-cookbook.md` **§13** — "Add a location overlay — the canonical runbook" (new_overlay.sh, the + flat-blob config, the non-4-aligned bin-carve/asset/trim gotcha, the A→B→C workflow, dedup-credit two shapes, + build-all/check-all). The Phase-15 fleet recipe. +- `docs/effort-map.md` — the **per-task effort-mapping + pause-at-every-transition discipline** (R27) + the **T7 + experiment Evidence** (Max≈Ultracode yield → prefer Ultracode for harvests). +- `docs/SETUP.md` — §2.8 MCP **client-reconnect** note (R29) + `new_overlay.sh` in the tooling inventory (R21). +- `docs/gen2-roadmap.md` — Phase-13 milestone **ACHIEVED** amendment. +- `docs/{progress,difficulty,duplicates}.ov_*.md` — **new, committed** per-overlay digests. +- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase13.md`** (R19 archive). `phase-ends/PhaseEnd_Phase13.md` — this file. + +**Local artifacts (gitignored / regenerable):** `asm//**`, `assets//*.bin`, `build/`, `expected/`, +`.run/drafts-ov077/*`, `.run/drafts-exp-{A,B}/*`, `.run/sig.ov_*.jsonl`, `.run/_funcs.txt`, the effort-experiment +logs (`.run/arm{A,B}*.txt`). The 4 overlay **Ghidra programs** are script-reproducible (`ghidra_import_raw.sh` + +`DefineFunctions.java`); **DB-commit deferred as regenerable** (no manual RE on them; avoids ~14 MB/overlay bloat). + +**Tools/packages installed:** None — the entire phase used the existing Phase-4/6/10/11/12 toolchain. + +**Verification results (literal) — the milestone proof (R22 CLEAN fleet rebuild, all green):** +- **`make clean && (make extract BINARY= for all 6) && make check-all`** → **6 passed, 0 failed of 6:** main + `143dbb89…`, resident `8e17e02f…`, ov_SC01_077 `d19c9580…`, ov_SC01_005 `56760dbe…`, ov_SC01_006 `56760dbe…`, + ov_SC03_001 `f8fd92f5…` — all **BYTE-IDENTICAL** from a fully clean tree. Negative control (corrupt one sha) → + `[FAIL]` + "5 passed, 1 failed" + nonzero exit. +- **Harvest:** `ov_SC01_077` **REAL 704/2586** (Ultracode §12, 5 passes: 110+205+217+52(throttled)+106 + 14 from + the T7 bank). Fleet REAL **887** (main 54 + resident 123 + ov_SC01_077 704 + 005/006 3+3 + SC03_001 0). +- **Dup-pair collapse:** `dedup_integrate --check` → **4 validated, 0 failed** (I0 + D1/D2/D3); 005 & 006 both + `56760dbe…` from the one `src/shared/ov_setters.h`; REAL 3 credited to both. +- **T7 effort experiment (controlled):** 80 fixed fns, identical Workflow, only session effort varied → + **Arm A (Ultracode/xHigh) 13/80, Arm B (Max) 14/80, A∩B=12** → yield difference is **noise**; per-agent depth + doesn't change blind-draft harvest yield. (The "pass-1-at-Max-did-better" observation was difficulty-confounded.) +- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk + staged** (`asm/`, `assets/`, `build/`, `.run/` gitignored). No Ghidra DB committed (overlay DBs regenerable). + +**Milestone achieved (confirmed by Drew, gate 2):** **6 binaries rebuild byte-for-byte from a clean tree** +(`check-all` 6/6) — a representative location-overlay set matched end-to-end (`SC01/077` byte-identical + 704 +functions; the `SC01/005 ≡ 006` duplicate pair collapsed to one shared source, both green; the large free-roam +`SC03/001` byte-identical), the **cross-binary dedup mechanism proven** on the dup pair, and a **one-command fleet +runbook** in place (`tools/new_overlay.sh` + `make build-all`/`check-all`). The ~134-overlay long-tail is the +open-ended Phase 15 (not a gate). + +**Next:** **Phase 14 — Public flip via a curated public repo (Gen2 EXIT capstone)** per the gen2-roadmap critical +path (8→9→10→11→{12,13}→14). Alternatively/concurrently **Phase 15 — overlay long-tail** (open-ended, not a gate) +can grind down more overlays first. Whether to go public now vs. harvest more overlays first is a Phase-Start +decision for Drew. Start a fresh session (effort **Max**, plan mode). + +## Deviations +| Item | Plan | Actual | Reason | +|---|---|---|---| +| Non-4-aligned overlays | (not anticipated) | a **3-part fix** baked into the template + Makefile (bin carve + incbin asset rule + objcopy trim) | ~75% of overlays aren't 4-aligned; spimdisasm drops the trailing partial word, the asset needs a build rule + .data align fix, and splat's `.ld` pads the segment end. The resident (4-aligned) never hit it. | +| Harvest depth | "harvest the unique remainder" | **704 matched** (the easy/medium tier, 5 Ultracode passes), then consolidated | yield fell 92%→54% as difficulty rose; the hard tail is genuine compiler-residual / large-function work = Phase 15. "Bank, don't strand." | +| Pass 4 | match_one iterate pass | **rate-limited** (20/36 batches lost to a transient server throttle) → +52 only; resumed with paced blind-draft + retry-wave | server-side throttle on the token-heavy iterate fan-out; the retry-wave (§12) recovered cleanly on pass 5 | +| Dup-pair credit | "fork psyq_integrate" (roadmap §11 sketch) | **source-level macro share** (`src/shared/` + `dedup.us.yaml`), the proven §11 mechanism | game-code fns are interior to one object per binary; the object-swap is library-only (Phase 11 D1). The macro must be instantiated IN PLACE (overlays link in source order). | +| Ghidra DB | (commit per R23) | **deferred as regenerable** (overlay programs are script-reproducible; no manual RE) | avoids ~14 MB/overlay `.git` bloat; the import is reproducible from committed tooling (Phase-10 precedent) | +| T7 experiment | (not in the original plan) | **added** (Drew) — Max vs Ultracode yield, controlled | Drew's question after I ran pass 1 at Max by mistake; settled it empirically (= noise) | +| scriptPath args | (forward `args`) | the `scriptPath` re-invoke **didn't forward `args.draftdir`** → Arm B overwrote Arm A's drafts dir | a Workflow-tool gotcha; harmless here (Arm A was already gated/recorded before the overwrite) | + +## Commit Message +``` +(Phase 13 landed as ~18 per-task/per-batch checkpoint commits commit:0084..commit:0099 + this PhaseEnd.) + +feat(phase-13): representative location overlays — 6 binaries byte-identical + fleet runbook (v1.13.0) + +- T0: overlay splat template + generated config/overlays.mk indirection (Makefile body untouched) +- T1: ov_SC01_077 byte-identical d19c9580 + REAL 1->704 (Ultracode §12, 5 passes) + Ghidra import; + reusable non-4-aligned-overlay handling (bin carve + incbin asset rule + objcopy end-align trim) +- T2: dup-pair SC01/005 ≡ 006 collapsed to ONE shared source (src/shared/ov_setters.h + + dedup.us.yaml D1/D2/D3); both 56760dbe; dedup_integrate 4 validated/0 failed +- T3: free-roam ov_SC03_001 byte-identical f8fd92f5 (recipe scales) +- T4: tools/new_overlay.sh one-command onboarding (idempotent; proven on 3 overlays) +- T5: make build-all/check-all (6/6 clean-fleet R22 proof; negative control fails loud) +- T7: controlled Max-vs-Ultracode harvest experiment = noise (14 vs 13) -> prefer Ultracode for breadth +- cookbook §13 overlay runbook; SETUP/gen2-roadmap/effort-map; rules R27-R29 +- MILESTONE (gate 2): 6 binaries byte-identical from clean; fleet REAL 887; dedup proven; runbook live +- bumps project version 1.12.0 -> 1.13.0 +``` + +## Rules Added This Phase +| Rule | Reason | +|---|---| +| **R27 — Effort-transition discipline (pause+prompt at EVERY boundary).** Effort is mapped per-task at plan time AND re-evaluated continuously; at every transition — into Ultracode (mid-task breadth revealed), back to Max (breadth done), and at task hand-off (next task doesn't need the current mode) — **STOP, prompt Drew to toggle, and WAIT for the actual `/effort` command** (a system-reminder confirms). Never launch a Workflow on a verbal "yes" before the toggle lands; never grind deep synthesis at Ultracode/xHigh. Claude cannot toggle effort itself. Governing doc `docs/effort-map.md`; memory `effort-prompt-ultracode-on-breadth`. Extends R7/R26. | Two failures this phase: launched the harvest Workflow on Drew's verbal "Enable Ultracode" before he toggled it; and continued from the Ultracode harvest straight into the deep T2–T6 work without prompting the switch back to Max (Drew had to ask). | +| **R28 — Build the harness task list right after plan approval, before starting work.** On plan approval (ExitPlanMode), create the task list (one TaskCreate per plan task) so Drew has a live, monitorable view; mark the first task in_progress and proceed. In addition to `CURRENT_PHASE.md` (the committable crash-recovery log), not instead. Memory `build-tasklist-after-plan-approval`. | Drew (2026-06-16): "build tasklist so I can monitor progress." He tracks progress via the task list, not the prose plan. | +| **R29 — Pause + prompt for `/mcp` reconnect after an MCP server restart / program switch.** Restarting the headless MCP server (R23 import cycle) or restarting it to serve a different program drops the Claude Code client's SSE connection → every `mcp__ghidra__*` call times out until the client reconnects. Claude can't run `/mcp`. So after any server restart / program switch: **pause, ask Drew to run `/mcp`, then one cheap `get_binary_info` (G2) before continuing** — don't try calls, hit timeouts, and work around them. SETUP §2.8; memory `mcp-reconnect-after-restart`. | Restarted MCP to serve the freshly-imported `ov_SC01_077`, my calls timed out, and I started working around it instead of pausing to ask for the reconnect. | + +*(Not elevated to rules — captured as cookbook §13 techniques: the non-4-aligned-overlay 3-part fix, the +`overlays.mk` indirection, the per-function vs whole-overlay dedup shapes, the address-order-preserving in-place +macro instantiation, the sig_image-vs-splat name-case / addr-keyed validator detail.)* + +## PhaseEnd Changelog +**v1.12.0 → v1.13.0 — Phase 13 complete (Gen2 phase 6).** The project rebuilds **6 binaries byte-for-byte from a +clean tree** (was 2): the EXE, the resident engine, and a **representative location-overlay set** — `SC01/077` +(byte-identical + **704 functions** matched via an Ultracode parallel-draft/byte-gate harvest), the +**byte-identical duplicate pair `SC01/005 ≡ 006`** collapsed to **one shared source** (proving the cross-binary +dedup mechanism end-to-end), and the large free-roam `SC03/001`. A reusable **one-command fleet runbook** is in +place: `tools/new_overlay.sh` (idempotent overlay onboarding via a flat-blob template + the generated +`config/overlays.mk` indirection) and `make build-all`/`check-all`. The phase's hardest-won reusable lesson is the +**non-4-aligned-overlay handling** (≈75% of the fleet): spimdisasm drops the trailing partial word, so a `bin` +carve + an incbin asset rule (+ `.data` align fix) + an objcopy end-align trim recover the exact bytes. A +controlled **Max-vs-Ultracode experiment** (T7) found the two efforts produce **equal harvest yield** (noise) → +Ultracode is preferred for breadth harvests; Max for deep synthesis. Three process rules (R27 effort-transition +discipline, R28 task-list-after-plan, R29 MCP-reconnect). The ~134-overlay long-tail is the open-ended Phase 15. + +## Plain-English Recap +The game's first playable "level" now rebuilds **perfectly, bit-for-bit, from our own source** and is about a +quarter decompiled — and so do a duplicate level and a big open level, so **six pieces of the game** now +reconstruct exactly (up from two at the start of the session). The biggest practical win is a **one-command tool** +that adds any of the remaining ~130 levels to the build in seconds, plus a single command that re-checks every +piece at once. We also proved that when two levels are identical, we can rebuild both from **one shared source +file** — which is how the project will avoid re-doing the same work 130 times. Along the way we hit and +permanently solved a fiddly snag that affects most levels (their size isn't a round number, so the tools were +dropping the last few bytes). The level's functions were matched by a **swarm of AI agents drafting code against +an automatic bit-for-bit referee** — and we ran a clean side experiment that answered a real question: running +that swarm at maximum reasoning depth did **not** beat the faster parallel mode (same results), so we'll use the +faster mode for this kind of bulk work and save the deep mode for hard thinking. The remaining levels are a big, +optional to-do list — deliberately not blocking the project. + +## 🛑 Stop Here +PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase13.md` (R19). The Phase-13 work is committed +(~18 checkpoints `commit:0084`..`commit:0099` + this PhaseEnd commit); **Drew pushes** (R6). Gen2 continues — do **NOT** +start the next phase here. Start a **fresh session** (effort **Max**, plan mode) to plan **Phase 14 (public flip — +Gen2 EXIT)** or **Phase 15 (overlay long-tail)** — Drew's choice at the Phase-Start gate. Keep this file forever. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase13.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase13.md