# PhaseEnd — Phase 9: Binary-agnostic toolchain refactor **Date:** 2026-06-15 · **Project Version:** 1.9.0 · **Phase Status:** Complete · **Generation:** Gen2 (2nd phase) > Gen2 phase 2 of the arc (8→**9**→10→11→{12,13}→14). The granular per-task trail (per-tool edits, > exact gates, the transitional-default reasoning) is preserved on-demand at > **`phase-ends/logs/Phase9.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). > Owner decisions (2026-06-15): scope = **de-risked middle**; commits = **per-tool checkpoints**; > at phase-end **stop the Ghidra MCP + commit the DB** (option a → R23). ## Build Log **Files created/changed and complete — do not recreate:** *The refactor pattern (locked, gate-1 approved):* alias-key Makefile namespacing (`BINARIES := main` + `main_*` vars + `$(BINARY)`-resolved aliases, borrowing `match_protos.py`'s idiom); EXE artifact paths **preserved verbatim** (no rename churn against the byte oracle); **no new config-file format** (Make vars + existing splat yaml); **required params, no EXE default** threaded **leaf-first**; `--vram-base` a single fileoff→vram scalar (window is orthogonal). *Makefile (T1, T5, T6, T7):* the data-driven `BINARIES`/`BINARY ?= main` block + `$(filter)` guard + the `main_*` set (paths/`VRAM_BASE 0x8000F800`/`TEXT_LO/HI` preserved) + selected-binary aliases. The 9 PsyQ `psyq_integrate` calls gained `--vram-base $(main_VRAM_BASE) --exe $(main_EXE) --symbols $(main_SYMBOLS)` (flags before positionals) and are wrapped in `ifeq ($(BINARY),main)`; the `extract` `ld_interleave` call likewise gated + `--front 53198.data.o --tail 6324C.data.o`; the `report` target threads `--binary $(BINARY)`. *Link pipeline — `--vram-base`/`--exe` REQUIRED, no module globals (T2–T5, T8b):* - `tools/psyq_link.py` — `recover_sym_addrs`/`unique_byte_vram`/`link_object` take `vram_base` (+ `link_object` `exe_path`); module `EXE`/`VRAM_BASE` removed; CLI `--vram-base`/`--exe` required (argparse). - `tools/psyq_identify.py` — argparse; `--vram-base`/`--exe` required; `[lo hi]` window optional positional. - `tools/psyq_link_region.py` + `tools/psyq_link_lib.py` — thread `vram_base`/`exe` through `build_region`/`placement`/`classify`; dropped the `VRAM_BASE` import + `EXE`; pass `--vram-base`/`--exe` to the `psyq_identify` subprocess; CLI required. - `tools/psyq_integrate.py` — `integrate()` `vram_base`/`exe_path`/`symbols_path` keyword-required; argparse (`nargs="*"` window) with `--vram-base`/`--exe`/`--symbols` before positionals; dropped `VRAM_BASE` import + `EXE`. *Report scripts — `--binary` selector (T5, T6):* `tools/progress.py` (the Makefile `psyq_integrate` stub-list regex updated **in lockstep** to consume leading `--flag value` pairs: `psyq_integrate\.py(?:\s+--\S+\s+\S+)*\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)`), `tools/dup_report.py`, `tools/difficulty.py` — each gains `--binary ` (default `main`) + a `BINARIES` path table. *EXE-curation helpers — de-defaulted (T7, T8a):* `tools/ld_interleave.py` (`--front`/`--tail`/`[ld]`), `tools/make_snd_used.py` + `tools/make_apicard_used.py` (`--exe`/`--vram-base`; region bounds/excludes kept as EXE data), `tools/gen_lib_subsegs.py` (`--vram-base`/`--exe`; its own `VRAM_BASE` const removed), `tools/split_src_region.py` (`--symbols`). *asm-differ (T9):* `diff_settings.py` — `BFM_BINARY` env-var selector + `BINARIES` → `{baseimg,myimg,mapfile}` (env var because asm-differ owns the `apply()` signature); `main` = the EXE images. *Docs (T10, R16/R21):* `docs/SETUP.md` **§6.7** (the binary-agnostic toolchain: `make build BINARY=`, the required flags, `BFM_BINARY`, and the future-binary path convention `config/splat..yaml` / `build//` / `config/check..sha` / `config/symbols..txt` / `.run/sig..jsonl`); `docs/matching-cookbook.md` **§9.7** (the reusable Gen2 refactor pattern: required-params-no-default, the transitional-default + leaf-first technique, the negative control, the progress.py regex lockstep gotcha); `docs/psyq-worklist.md` (per-library recipe updated with the now-required flags); `README.md` (status → 52 REAL / 959 LINKED / ~50% / Gen2-underway); `docs/difficulty.md` regenerated (resolves the Phase-8 deferred regen — 9 harvested fns leaving the unmatched queue, old-code==new-code verified). - `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase9.md`** (R19 archive). `phase-ends/PhaseEnd_Phase9.md` — this file. **Local artifacts (gitignored / regenerable — a fresh clone redoes them):** `build/`, `expected/`, `asm/`, the generated `include/` macros, `undefined_*_auto.txt`, `.run/obj40/*` (curated SDK dirs), `build/psyq/*`. **Tools/packages installed:** None (used the existing Phase-4/6 toolchain entirely). **Ghidra DB (R23, this phase):** the MCP server ran (SessionStart hook) but Phase 9 made **zero** RE writes (pure build-tooling). At phase close the MCP was cleanly stopped (`tools/ghidra_mcp_stop.sh` — save + lock release) and the current `ghidra/` program DB committed to reconcile git with disk (the tracked `db.12` → the live `db.N`); content unchanged, a one-time ~10 MB reconcile + protocol bank. **Verification results (literal) — the milestone proof (all green):** - **(1) with-SDK** `make clean && extract && build` → `build/us/SLUS_007.26` SHA1 `143dbb89f34491258bbc27810d0a12ec8b43a8dd` (BYTE-IDENTICAL). - **(2) `make report`** → REAL **52** / LINKED **959** / NON_MATCHING **7** / **50.24%** (exact Phase-8 counts; the progress regex still parses all 9 stub lists through the new flags). - **(3) without-SDK** (`.run/obj40` moved aside, fresh-clone stub fallback) → `143dbb89…` (BYTE-IDENTICAL). - **(4) NEGATIVE CONTROL** `make build main_VRAM_BASE=0x8000F804` → `cae22f7ee81e534b4af83290750a8c4e7e651b70` (≠ target — proves `--vram-base` is load-bearing, not accepted-and-ignored). Per-tool: `psyq_link.py ` with no flags → argparse `error: the following arguments are required: --vram-base, --exe` (loud). - **(5) restore + `make expected`** → `143dbb89…` baseline refreshed. - `git status`: only `config/`/`Makefile`/`tools/`/`src`-free docs tracked per commit; Ghidra DB reconciled at close; **zero ROM-derived/generated bulk staged** in the source commits. **Milestone achieved:** The toolchain is **binary-agnostic** — `make build` rebuilds `SLUS_007.26` **byte-for-byte (`143dbb89…`)** through the fully parameterized path **with AND without** the SDK objects, `make report` reproduces the Phase-8 counts exactly, and **every binary-specific value is a required parameter** (no EXE default an overlay could inherit) proven load-bearing by the negative control — a pure behavior-preserving no-op. Confirmed by Drew (gate 2). **Next:** **Phase 10 — Resident engine blob (splat config + all-asm byte-match).** Start a fresh session (effort **Max**, plan mode). First target: the resident engine blob `MAIN.CD/FILE_010/1.1` @ vram `0x800CEDF8` — author `config/splat.resident.yaml` + the `resident` `BINARIES` instance (clean `` convention; its own `resident_VRAM_BASE`, **not** `0x8000F800`), build at 100% INCLUDE_ASM, SHA1-match the extracted payload. The Phase-9 parameterized tools + the documented convention (SETUP §6.7) feed directly in. ## Deviations | Item | Plan | Actual | Reason | |---|---|---|---| | T8 commit count | one commit | **split T8a + T8b** | in-process coupling: the no-default removal (T8b) can only land after the curation helpers (T8a) pass explicit args — two green commits | | Default removal | "remove module globals in each tool's commit" | **transitional-default technique** (params default to kept globals T2–T7; removed → required at T8b) | `psyq_integrate`/`link_region` import from `psyq_link` IN-PROCESS — removing a global in isolation breaks the build mid-sequence; defaults keep every per-tool commit green, then a final hardening reaches the no-default end state | | SDK-region Makefile vars | namespace under `main_` | **left un-namespaced** (LIB*_ELF…) | already main-only by the `ifeq ($(BINARY),main)` gate; namespacing a hypothetical future collision is speculative churn (de-risked scope) — deferred to when a 2nd binary needs SDK regions | | `psyq_identify` caller argv | "update all callers in T3" | **distributed to each caller's own task** (T4/T8a) | one-touch-per-file; psyq_identify's transitional defaults kept callers green meanwhile | | `ld_interleave`/curation `--exe`/`--vram-base` | "required" | **CLI-default to EXE values** (entry-point convenience) | these are EXE-curation tools / EXE-gated; they thread EXPLICIT values down to the now-required pipeline — the "no silent default" holds where it matters (the pipeline) | | `difficulty.md` | not in plan | **regenerated** (committed) | the Phase-8 deferred regen surfaced; old-code==new-code output verified → the diff is the deferral, not a T6 bug | | Ghidra DB at phase-end | (not anticipated) | **stop MCP + commit DB** (R23) | Drew raised the lock/backup gap: the `.rep` lock holds until MCP closes, and SessionEnd is too late for a mid-session phase-end commit | ## Commit Message ``` (Phase 9 landed as 11 per-tool checkpoint commits commit:0042..commit:0052 + this PhaseEnd.) feat(phase-9): binary-agnostic toolchain refactor — one pipeline builds any binary (v1.9.0) - Makefile: data-driven BINARIES (alias keys) + main_* vars + $(BINARY) aliases; EXE paths preserved verbatim; 9 psyq_integrate calls + ld_interleave gated ifeq(BINARY,main) - link pipeline (psyq_link/identify/link_region/link_lib/integrate): --vram-base/--exe (+ integrate --symbols) REQUIRED, module EXE/VRAM_BASE removed, threaded leaf-first via a transitional-default technique (every per-tool commit byte-identical) - reports (progress/difficulty/dup_report): --binary selector; progress regex consumes the new leading flags in lockstep. diff_settings: BFM_BINARY env selector - curation helpers (ld_interleave --front/--tail; make_snd/apicard/gen_lib_subsegs --vram-base/--exe; split_src_region --symbols) de-defaulted - docs: SETUP §6.7 + cookbook §9.7 (the reusable Gen2 pattern) + psyq-worklist recipe + README; difficulty.md deferred-regen resolved - MILESTONE: EXE rebuilds 143dbb89 WITH and WITHOUT SDK objects; report 52/959/7/50.24%; wrong --vram-base -> cae22f7e (negative control); required params fail loud - rule R23 (stop Ghidra MCP + commit DB at phase-end/RE-checkpoint); bumps 1.8.0 -> 1.9.0 ``` ## Rules Added This Phase | Rule | Reason | |---|---| | **R23 — Stop the Ghidra MCP before any phase-end / RE-checkpoint commit; commit the DB if RE work changed it.** The headless MCP server holds the `.rep` lock with an open transaction — the saved program DB only materializes on a clean stop (`tools/ghidra_mcp_stop.sh`, which waits for *"Save succeeded"* + lock release). Phase-end/checkpoint commits happen **mid-session**, so the **SessionEnd** hook is too late — run `ghidra_mcp_stop.sh` explicitly first, then commit the current `ghidra/` DB (the live `db.N.gbf`) when RE work changed it. Between RE checkpoints, the `db.*.gbf` rename-on-save churn in `git status` is noise — don't commit no-op renumbers. Extends R20 (backup) + R9 (verify persistence). | Drew (2026-06-15): the DB is the irreplaceable RE artifact (types/comments/imported overlays beyond `symbols.us.txt`), it's tracked-but-not-actually-saved-when-current, and the lock won't release until MCP closes — so a phase-end commit must stop MCP itself, not wait for session close. | *(Not elevated to rules — captured in cookbook §9.7: the transitional-default + leaf-first technique for refactoring a shared in-process global without breaking the byte oracle; the negative-control discipline (a no-op can pass for the wrong reason — pass a deliberately wrong value too); the progress.py-regex/Makefile lockstep. SETUP §6.7 documents the future-binary path convention.)* ## PhaseEnd Changelog **v1.8.0 → v1.9.0 — Phase 9 complete (Gen2 phase 2).** The single-binary toolchain is now **binary-agnostic**: a data-driven `BINARIES` Makefile (`make build BINARY=`) + every link-pipeline, report, curation, and asm-differ tool parameterized by **required** `--vram-base`/`--exe`/`--symbols`/ `--binary`/`BFM_BINARY` — **no EXE default an overlay could silently inherit** (the roadmap's #1 risk, eliminated by construction; a miss fails loud). Proven a **pure no-op**: the EXE rebuilds `143dbb89…` through the parameterized path with and without the SDK objects, `make report` reproduces 52 REAL / 959 LINKED / 50.24%, and a deliberately wrong `--vram-base` diverges (`cae22f7e…`, the negative control). Landed as 11 per-tool checkpoint commits (each byte-gated), using a **transitional-default technique** (params default to kept globals, removed → required once every caller passes explicitly) so the byte-locked EXE never broke mid-refactor. The EXE's artifact paths are preserved verbatim; the clean `` convention + the EXE-only `ifeq ($(BINARY),main)` gates are documented (SETUP §6.7, cookbook §9.7) for Phase 10's resident blob. One rule (R23, stop-MCP-then-commit-DB at phase-end). No tools installed. The toolchain is ready to point at the first overlay binary. ## Notes for Future Phases - **Gen2 candidate (raised Phase 9):** extend `ExportSymbols.java` to also export Ghidra **types + comments** to text. Today only function names/addresses are text-exported (`config/symbols.us.txt`, R15); types/ comments/imported-overlays live only in the binary `db.*.gbf`, which **renames on every save** → committing it bloats `.git` ~10 MB/checkpoint (R20 accepts this; clean public mirror later). Text-export would make the DB regenerable, eliminate the binary-blob backup + the R23 lock/commit dance, and be public-clean from day one. - **Phase 10 first step:** `config/splat.resident.yaml` + the `resident` `BINARIES` instance for `MAIN.CD/FILE_010/1.1` @ `0x800CEDF8` (its own `resident_VRAM_BASE`); all-asm byte-match vs the extracted payload. The Phase-3 overlay map (`docs/memory-map.md §4.3`) + the Phase-9 parameterized tools feed directly in. ## 🛑 Stop Here PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase9.md` (R19). The Phase-9 work is committed (11 checkpoints `commit:0042`..`commit:0052` + this PhaseEnd commit, which also reconciles the Ghidra DB per R23); **Drew pushes** (R6). Gen2 continues — do **NOT** start Phase 10 here. Start a **fresh session** (effort **Max**, plan mode) for **Phase 10 — Resident engine blob**. Keep this file forever.