BFM-decomp/phase-ends/PhaseEnd_Phase9.md

PhaseEnd — Phase 9: Binary-agnostic toolchain refactor

Date: 2026-06-15 · Project Version: 1.9.0 · Phase Status: Complete · Generation: Gen2 (2nd phase)

Gen2 phase 2 of the arc (8→9→10→11→{12,13}→14). The granular per-task trail (per-tool edits, exact gates, the transitional-default reasoning) is preserved on-demand at phase-ends/logs/Phase9.md (R19 — NOT auto-loaded; consult only when researching a mechanism). Owner decisions (2026-06-15): scope = de-risked middle; commits = per-tool checkpoints; at phase-end stop the Ghidra MCP + commit the DB (option a → R23).

Build Log

Files created/changed and complete — do not recreate:

The refactor pattern (locked, gate-1 approved): alias-key Makefile namespacing (BINARIES := main + main_* vars + $(BINARY)-resolved aliases, borrowing match_protos.py's idiom); EXE artifact paths preserved verbatim (no rename churn against the byte oracle); no new config-file format (Make vars

• existing splat yaml); required params, no EXE default threaded leaf-first; --vram-base a single fileoff→vram scalar (window is orthogonal).

Makefile (T1, T5, T6, T7): the data-driven BINARIES/BINARY ?= main block + $(filter) guard + the main_* set (paths/VRAM_BASE 0x8000F800/TEXT_LO/HI preserved) + selected-binary aliases. The 9 PsyQ psyq_integrate calls gained --vram-base $(main_VRAM_BASE) --exe $(main_EXE) --symbols $(main_SYMBOLS) (flags before positionals) and are wrapped in ifeq ($(BINARY),main); the extract ld_interleave call likewise gated + --front 53198.data.o --tail 6324C.data.o; the report target threads --binary $(BINARY).

Link pipeline — --vram-base/--exe REQUIRED, no module globals (T2–T5, T8b):

• tools/psyq_link.py — recover_sym_addrs/unique_byte_vram/link_object take vram_base (+ link_object exe_path); module EXE/VRAM_BASE removed; CLI --vram-base/--exe required (argparse).
• tools/psyq_identify.py — argparse; --vram-base/--exe required; [lo hi] window optional positional.
• tools/psyq_link_region.py + tools/psyq_link_lib.py — thread vram_base/exe through build_region/placement/classify; dropped the VRAM_BASE import + EXE; pass --vram-base/--exe to the psyq_identify subprocess; CLI required.
• tools/psyq_integrate.py — integrate() vram_base/exe_path/symbols_path keyword-required; argparse (nargs="*" window) with --vram-base/--exe/--symbols before positionals; dropped VRAM_BASE import + EXE.

Report scripts — --binary selector (T5, T6): tools/progress.py (the Makefile psyq_integrate stub-list regex updated in lockstep to consume leading --flag value pairs: psyq_integrate\.py(?:\s+--\S+\s+\S+)*\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)), tools/dup_report.py, tools/difficulty.py — each gains --binary <alias> (default main) + a BINARIES path table.

EXE-curation helpers — de-defaulted (T7, T8a): tools/ld_interleave.py (--front/--tail/[ld]), tools/make_snd_used.py + tools/make_apicard_used.py (--exe/--vram-base; region bounds/excludes kept as EXE data), tools/gen_lib_subsegs.py (--vram-base/--exe; its own VRAM_BASE const removed), tools/split_src_region.py (--symbols).

asm-differ (T9): diff_settings.py — BFM_BINARY env-var selector + BINARIES → {baseimg,myimg,mapfile} (env var because asm-differ owns the apply() signature); main = the EXE images.

Docs (T10, R16/R21): docs/SETUP.md §6.7 (the binary-agnostic toolchain: make build BINARY=<alias>, the required flags, BFM_BINARY, and the future-binary path convention config/splat.<bin>.yaml / build/<bin>/ / config/check.<bin>.sha / config/symbols.<bin>.txt / .run/sig.<bin>.jsonl); docs/matching-cookbook.md §9.7 (the reusable Gen2 refactor pattern: required-params-no-default, the transitional-default + leaf-first technique, the negative control, the progress.py regex lockstep gotcha); docs/psyq-worklist.md (per-library recipe updated with the now-required flags); README.md (status → 52 REAL / 959 LINKED / ~50% / Gen2-underway); docs/difficulty.md regenerated (resolves the Phase-8 deferred regen — 9 harvested fns leaving the unmatched queue, old-code==new-code verified).

• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase9.md (R19 archive). phase-ends/PhaseEnd_Phase9.md — this file.

Local artifacts (gitignored / regenerable — a fresh clone redoes them): build/, expected/, asm/, the generated include/ macros, undefined_*_auto.txt, .run/obj40/* (curated SDK dirs), build/psyq/*.

Tools/packages installed: None (used the existing Phase-4/6 toolchain entirely).

Ghidra DB (R23, this phase): the MCP server ran (SessionStart hook) but Phase 9 made zero RE writes (pure build-tooling). At phase close the MCP was cleanly stopped (tools/ghidra_mcp_stop.sh — save + lock release) and the current ghidra/ program DB committed to reconcile git with disk (the tracked db.12 → the live db.N); content unchanged, a one-time ~10 MB reconcile + protocol bank.

Verification results (literal) — the milestone proof (all green):

• (1) with-SDK make clean && extract && build → build/us/SLUS_007.26 SHA1 143dbb89f34491258bbc27810d0a12ec8b43a8dd (BYTE-IDENTICAL).
• (2) make report → REAL 52 / LINKED 959 / NON_MATCHING 7 / 50.24% (exact Phase-8 counts; the progress regex still parses all 9 stub lists through the new flags).
• (3) without-SDK (.run/obj40 moved aside, fresh-clone stub fallback) → 143dbb89… (BYTE-IDENTICAL).
• (4) NEGATIVE CONTROL make build main_VRAM_BASE=0x8000F804 → cae22f7ee81e534b4af83290750a8c4e7e651b70 (≠ target — proves --vram-base is load-bearing, not accepted-and-ignored). Per-tool: psyq_link.py <obj> <vram> with no flags → argparse error: the following arguments are required: --vram-base, --exe (loud).
• (5) restore + make expected → 143dbb89… baseline refreshed.
• git status: only config//Makefile/tools//src-free docs tracked per commit; Ghidra DB reconciled at close; zero ROM-derived/generated bulk staged in the source commits.

Milestone achieved: The toolchain is binary-agnostic — make build rebuilds SLUS_007.26 byte-for-byte (143dbb89…) through the fully parameterized path with AND without the SDK objects, make report reproduces the Phase-8 counts exactly, and every binary-specific value is a required parameter (no EXE default an overlay could inherit) proven load-bearing by the negative control — a pure behavior-preserving no-op. Confirmed by Drew (gate 2).

Next: Phase 10 — Resident engine blob (splat config + all-asm byte-match). Start a fresh session (effort Max, plan mode). First target: the resident engine blob MAIN.CD/FILE_010/1.1 @ vram 0x800CEDF8 — author config/splat.resident.yaml + the resident BINARIES instance (clean <bin> convention; its own resident_VRAM_BASE, not 0x8000F800), build at 100% INCLUDE_ASM, SHA1-match the extracted payload. The Phase-9 parameterized tools + the documented convention (SETUP §6.7) feed directly in.

Deviations

Item	Plan	Actual	Reason
T8 commit count	one commit	split T8a + T8b	in-process coupling: the no-default removal (T8b) can only land after the curation helpers (T8a) pass explicit args — two green commits
Default removal	"remove module globals in each tool's commit"	transitional-default technique (params default to kept globals T2–T7; removed → required at T8b)	psyq_integrate/link_region import from psyq_link IN-PROCESS — removing a global in isolation breaks the build mid-sequence; defaults keep every per-tool commit green, then a final hardening reaches the no-default end state
SDK-region Makefile vars	namespace under main_	left un-namespaced (LIB*_ELF…)	already main-only by the ifeq ($(BINARY),main) gate; namespacing a hypothetical future collision is speculative churn (de-risked scope) — deferred to when a 2nd binary needs SDK regions
psyq_identify caller argv	"update all callers in T3"	distributed to each caller's own task (T4/T8a)	one-touch-per-file; psyq_identify's transitional defaults kept callers green meanwhile
ld_interleave/curation --exe/--vram-base	"required"	CLI-default to EXE values (entry-point convenience)	these are EXE-curation tools / EXE-gated; they thread EXPLICIT values down to the now-required pipeline — the "no silent default" holds where it matters (the pipeline)
difficulty.md	not in plan	regenerated (committed)	the Phase-8 deferred regen surfaced; old-code==new-code output verified → the diff is the deferral, not a T6 bug
Ghidra DB at phase-end	(not anticipated)	stop MCP + commit DB (R23)	Drew raised the lock/backup gap: the .rep lock holds until MCP closes, and SessionEnd is too late for a mid-session phase-end commit

Commit Message

(Phase 9 landed as 11 per-tool checkpoint commits 170586d6b..6345b5012 + this PhaseEnd.)

feat(phase-9): binary-agnostic toolchain refactor — one pipeline builds any binary (v1.9.0)

- Makefile: data-driven BINARIES (alias keys) + main_* vars + $(BINARY) aliases; EXE
  paths preserved verbatim; 9 psyq_integrate calls + ld_interleave gated ifeq(BINARY,main)
- link pipeline (psyq_link/identify/link_region/link_lib/integrate): --vram-base/--exe
  (+ integrate --symbols) REQUIRED, module EXE/VRAM_BASE removed, threaded leaf-first via a
  transitional-default technique (every per-tool commit byte-identical)
- reports (progress/difficulty/dup_report): --binary selector; progress regex consumes
  the new leading flags in lockstep. diff_settings: BFM_BINARY env selector
- curation helpers (ld_interleave --front/--tail; make_snd/apicard/gen_lib_subsegs
  --vram-base/--exe; split_src_region --symbols) de-defaulted
- docs: SETUP §6.7 + cookbook §9.7 (the reusable Gen2 pattern) + psyq-worklist recipe +
  README; difficulty.md deferred-regen resolved
- MILESTONE: EXE rebuilds 143dbb89 WITH and WITHOUT SDK objects; report 52/959/7/50.24%;
  wrong --vram-base -> cae22f7e (negative control); required params fail loud
- rule R23 (stop Ghidra MCP + commit DB at phase-end/RE-checkpoint); bumps 1.8.0 -> 1.9.0

Rules Added This Phase

Rule

Reason

R23 — Stop the Ghidra MCP before any phase-end / RE-checkpoint commit; commit the DB if RE work changed it. The headless MCP server holds the .rep lock with an open transaction — the saved program DB only materializes on a clean stop (tools/ghidra_mcp_stop.sh, which waits for "Save succeeded" + lock release). Phase-end/checkpoint commits happen mid-session, so the SessionEnd hook is too late — run ghidra_mcp_stop.sh explicitly first, then commit the current ghidra/ DB (the live db.N.gbf) when RE work changed it. Between RE checkpoints, the db.*.gbf rename-on-save churn in git status is noise — don't commit no-op renumbers. Extends R20 (backup) + R9 (verify persistence).

Drew (2026-06-15): the DB is the irreplaceable RE artifact (types/comments/imported overlays beyond symbols.us.txt), it's tracked-but-not-actually-saved-when-current, and the lock won't release until MCP closes — so a phase-end commit must stop MCP itself, not wait for session close.

(Not elevated to rules — captured in cookbook §9.7: the transitional-default + leaf-first technique for refactoring a shared in-process global without breaking the byte oracle; the negative-control discipline (a no-op can pass for the wrong reason — pass a deliberately wrong value too); the progress.py-regex/Makefile lockstep. SETUP §6.7 documents the future-binary path convention.)

PhaseEnd Changelog

v1.8.0 → v1.9.0 — Phase 9 complete (Gen2 phase 2). The single-binary toolchain is now binary-agnostic: a data-driven BINARIES Makefile (make build BINARY=<alias>) + every link-pipeline, report, curation, and asm-differ tool parameterized by required --vram-base/--exe/--symbols/ --binary/BFM_BINARY — no EXE default an overlay could silently inherit (the roadmap's #1 risk, eliminated by construction; a miss fails loud). Proven a pure no-op: the EXE rebuilds 143dbb89… through the parameterized path with and without the SDK objects, make report reproduces 52 REAL / 959 LINKED / 50.24%, and a deliberately wrong --vram-base diverges (cae22f7e…, the negative control). Landed as 11 per-tool checkpoint commits (each byte-gated), using a transitional-default technique (params default to kept globals, removed → required once every caller passes explicitly) so the byte-locked EXE never broke mid-refactor. The EXE's artifact paths are preserved verbatim; the clean <bin> convention + the EXE-only ifeq ($(BINARY),main) gates are documented (SETUP §6.7, cookbook §9.7) for Phase 10's resident blob. One rule (R23, stop-MCP-then-commit-DB at phase-end). No tools installed. The toolchain is ready to point at the first overlay binary.

Notes for Future Phases

• Gen2 candidate (raised Phase 9): extend ExportSymbols.java to also export Ghidra types + comments to text. Today only function names/addresses are text-exported (config/symbols.us.txt, R15); types/ comments/imported-overlays live only in the binary db.*.gbf, which renames on every save → committing it bloats .git ~10 MB/checkpoint (R20 accepts this; clean public mirror later). Text-export would make the DB regenerable, eliminate the binary-blob backup + the R23 lock/commit dance, and be public-clean from day one.
• Phase 10 first step: config/splat.resident.yaml + the resident BINARIES instance for MAIN.CD/FILE_010/1.1 @ 0x800CEDF8 (its own resident_VRAM_BASE); all-asm byte-match vs the extracted payload. The Phase-3 overlay map (docs/memory-map.md §4.3) + the Phase-9 parameterized tools feed directly in.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase9.md (R19). The Phase-9 work is committed (11 checkpoints 170586d6b..6345b5012 + this PhaseEnd commit, which also reconciles the Ghidra DB per R23); Drew pushes (R6). Gen2 continues — do NOT start Phase 10 here. Start a fresh session (effort Max, plan mode) for Phase 10 — Resident engine blob. Keep this file forever.