From d380b13fc53037ab010dc5fbd9b867d397f4568f Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Mon, 15 Jun 2026 22:01:17 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-9):=20close-out=20=E2=80=94=20PhaseE?= =?UTF-8?q?nd=5FPhase9=20+=20binary-agnostic=20toolchain=20(v1.9.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - PhaseEnd_Phase9.md written; CURRENT_PHASE.md archived -> phase-ends/logs/Phase9.md (R19) - MILESTONE (gate-2 confirmed): the toolchain is binary-agnostic — EXE rebuilds 143dbb89 through the parameterized path WITH and WITHOUT SDK objects; make report 52/959/7/50.24%; every binary-specific value a required param (no EXE default); wrong --vram-base -> cae22f7e (negative control). Landed as 11 per-tool checkpoints. - R23: stop the Ghidra MCP + commit the DB at phase-end/RE-checkpoint (lock won't release until MCP closes; SessionEnd is too late for a mid-session commit). This commit reconciles the Ghidra program DB (db.12 -> db.15, 99% identical: Phase 9 did zero RE writes) after a clean ghidra_mcp_stop.sh save. - .gitignore: ignore /ghidra/**/*.lock~ (the lock-backup that churned every session) - bumps project version 1.8.0 -> 1.9.0 --- .gitignore | 2 + phase-ends/PhaseEnd_Phase9.md | 164 ++++++++++++++++++ .../{CURRENT_PHASE.md => logs/Phase9.md} | 0 3 files changed, 166 insertions(+) create mode 100644 phase-ends/PhaseEnd_Phase9.md rename phase-ends/{CURRENT_PHASE.md => logs/Phase9.md} (100%) diff --git a/.gitignore b/.gitignore index 561473173e..3cdcb16245 100644 --- a/.gitignore +++ b/.gitignore @@ -43,6 +43,8 @@ # 12.1 -> the committed .rep restores directly.) Joins the pre-public ROM-content scrub list. /ghidra/*.lock /ghidra/**/*.lock +/ghidra/*.lock~ +/ghidra/**/*.lock~ /ghidra/**/tmp*.ps # Toolchain downloads & caches (not source) diff --git a/phase-ends/PhaseEnd_Phase9.md b/phase-ends/PhaseEnd_Phase9.md new file mode 100644 index 0000000000..253824632f --- /dev/null +++ b/phase-ends/PhaseEnd_Phase9.md @@ -0,0 +1,164 @@ +# PhaseEnd — Phase 9: Binary-agnostic toolchain refactor +**Date:** 2026-06-15 · **Project Version:** 1.9.0 · **Phase Status:** Complete · **Generation:** Gen2 (2nd phase) + +> Gen2 phase 2 of the arc (8→**9**→10→11→{12,13}→14). The granular per-task trail (per-tool edits, +> exact gates, the transitional-default reasoning) is preserved on-demand at +> **`phase-ends/logs/Phase9.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). +> Owner decisions (2026-06-15): scope = **de-risked middle**; commits = **per-tool checkpoints**; +> at phase-end **stop the Ghidra MCP + commit the DB** (option a → R23). + +## Build Log + +**Files created/changed and complete — do not recreate:** + +*The refactor pattern (locked, gate-1 approved):* alias-key Makefile namespacing (`BINARIES := main` + +`main_*` vars + `$(BINARY)`-resolved aliases, borrowing `match_protos.py`'s idiom); EXE artifact paths +**preserved verbatim** (no rename churn against the byte oracle); **no new config-file format** (Make vars ++ existing splat yaml); **required params, no EXE default** threaded **leaf-first**; `--vram-base` a single +fileoff→vram scalar (window is orthogonal). + +*Makefile (T1, T5, T6, T7):* the data-driven `BINARIES`/`BINARY ?= main` block + `$(filter)` guard + the +`main_*` set (paths/`VRAM_BASE 0x8000F800`/`TEXT_LO/HI` preserved) + selected-binary aliases. The 9 PsyQ +`psyq_integrate` calls gained `--vram-base $(main_VRAM_BASE) --exe $(main_EXE) --symbols $(main_SYMBOLS)` +(flags before positionals) and are wrapped in `ifeq ($(BINARY),main)`; the `extract` `ld_interleave` call +likewise gated + `--front 53198.data.o --tail 6324C.data.o`; the `report` target threads `--binary $(BINARY)`. + +*Link pipeline — `--vram-base`/`--exe` REQUIRED, no module globals (T2–T5, T8b):* +- `tools/psyq_link.py` — `recover_sym_addrs`/`unique_byte_vram`/`link_object` take `vram_base` (+ + `link_object` `exe_path`); module `EXE`/`VRAM_BASE` removed; CLI `--vram-base`/`--exe` required (argparse). +- `tools/psyq_identify.py` — argparse; `--vram-base`/`--exe` required; `[lo hi]` window optional positional. +- `tools/psyq_link_region.py` + `tools/psyq_link_lib.py` — thread `vram_base`/`exe` through + `build_region`/`placement`/`classify`; dropped the `VRAM_BASE` import + `EXE`; pass `--vram-base`/`--exe` + to the `psyq_identify` subprocess; CLI required. +- `tools/psyq_integrate.py` — `integrate()` `vram_base`/`exe_path`/`symbols_path` keyword-required; argparse + (`nargs="*"` window) with `--vram-base`/`--exe`/`--symbols` before positionals; dropped `VRAM_BASE` import + `EXE`. + +*Report scripts — `--binary` selector (T5, T6):* `tools/progress.py` (the Makefile `psyq_integrate` +stub-list regex updated **in lockstep** to consume leading `--flag value` pairs: +`psyq_integrate\.py(?:\s+--\S+\s+\S+)*\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)`), `tools/dup_report.py`, +`tools/difficulty.py` — each gains `--binary ` (default `main`) + a `BINARIES` path table. + +*EXE-curation helpers — de-defaulted (T7, T8a):* `tools/ld_interleave.py` (`--front`/`--tail`/`[ld]`), +`tools/make_snd_used.py` + `tools/make_apicard_used.py` (`--exe`/`--vram-base`; region bounds/excludes kept +as EXE data), `tools/gen_lib_subsegs.py` (`--vram-base`/`--exe`; its own `VRAM_BASE` const removed), +`tools/split_src_region.py` (`--symbols`). + +*asm-differ (T9):* `diff_settings.py` — `BFM_BINARY` env-var selector + `BINARIES` → `{baseimg,myimg,mapfile}` +(env var because asm-differ owns the `apply()` signature); `main` = the EXE images. + +*Docs (T10, R16/R21):* `docs/SETUP.md` **§6.7** (the binary-agnostic toolchain: `make build BINARY=`, +the required flags, `BFM_BINARY`, and the future-binary path convention `config/splat..yaml` / +`build//` / `config/check..sha` / `config/symbols..txt` / `.run/sig..jsonl`); +`docs/matching-cookbook.md` **§9.7** (the reusable Gen2 refactor pattern: required-params-no-default, the +transitional-default + leaf-first technique, the negative control, the progress.py regex lockstep gotcha); +`docs/psyq-worklist.md` (per-library recipe updated with the now-required flags); `README.md` (status → +52 REAL / 959 LINKED / ~50% / Gen2-underway); `docs/difficulty.md` regenerated (resolves the Phase-8 +deferred regen — 9 harvested fns leaving the unmatched queue, old-code==new-code verified). + +- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase9.md`** (R19 archive). `phase-ends/PhaseEnd_Phase9.md` — this file. + +**Local artifacts (gitignored / regenerable — a fresh clone redoes them):** `build/`, `expected/`, `asm/`, +the generated `include/` macros, `undefined_*_auto.txt`, `.run/obj40/*` (curated SDK dirs), `build/psyq/*`. + +**Tools/packages installed:** None (used the existing Phase-4/6 toolchain entirely). + +**Ghidra DB (R23, this phase):** the MCP server ran (SessionStart hook) but Phase 9 made **zero** RE writes +(pure build-tooling). At phase close the MCP was cleanly stopped (`tools/ghidra_mcp_stop.sh` — save + lock +release) and the current `ghidra/` program DB committed to reconcile git with disk (the tracked `db.12` → +the live `db.N`); content unchanged, a one-time ~10 MB reconcile + protocol bank. + +**Verification results (literal) — the milestone proof (all green):** +- **(1) with-SDK** `make clean && extract && build` → `build/us/SLUS_007.26` SHA1 + `143dbb89f34491258bbc27810d0a12ec8b43a8dd` (BYTE-IDENTICAL). +- **(2) `make report`** → REAL **52** / LINKED **959** / NON_MATCHING **7** / **50.24%** (exact Phase-8 counts; + the progress regex still parses all 9 stub lists through the new flags). +- **(3) without-SDK** (`.run/obj40` moved aside, fresh-clone stub fallback) → `143dbb89…` (BYTE-IDENTICAL). +- **(4) NEGATIVE CONTROL** `make build main_VRAM_BASE=0x8000F804` → `cae22f7ee81e534b4af83290750a8c4e7e651b70` + (≠ target — proves `--vram-base` is load-bearing, not accepted-and-ignored). Per-tool: `psyq_link.py + ` with no flags → argparse `error: the following arguments are required: --vram-base, --exe` (loud). +- **(5) restore + `make expected`** → `143dbb89…` baseline refreshed. +- `git status`: only `config/`/`Makefile`/`tools/`/`src`-free docs tracked per commit; Ghidra DB reconciled + at close; **zero ROM-derived/generated bulk staged** in the source commits. + +**Milestone achieved:** The toolchain is **binary-agnostic** — `make build` rebuilds `SLUS_007.26` +**byte-for-byte (`143dbb89…`)** through the fully parameterized path **with AND without** the SDK objects, +`make report` reproduces the Phase-8 counts exactly, and **every binary-specific value is a required +parameter** (no EXE default an overlay could inherit) proven load-bearing by the negative control — a pure +behavior-preserving no-op. Confirmed by Drew (gate 2). + +**Next:** **Phase 10 — Resident engine blob (splat config + all-asm byte-match).** Start a fresh session +(effort **Max**, plan mode). First target: the resident engine blob `MAIN.CD/FILE_010/1.1` @ vram +`0x800CEDF8` — author `config/splat.resident.yaml` + the `resident` `BINARIES` instance (clean `` +convention; its own `resident_VRAM_BASE`, **not** `0x8000F800`), build at 100% INCLUDE_ASM, SHA1-match the +extracted payload. The Phase-9 parameterized tools + the documented convention (SETUP §6.7) feed directly in. + +## Deviations +| Item | Plan | Actual | Reason | +|---|---|---|---| +| T8 commit count | one commit | **split T8a + T8b** | in-process coupling: the no-default removal (T8b) can only land after the curation helpers (T8a) pass explicit args — two green commits | +| Default removal | "remove module globals in each tool's commit" | **transitional-default technique** (params default to kept globals T2–T7; removed → required at T8b) | `psyq_integrate`/`link_region` import from `psyq_link` IN-PROCESS — removing a global in isolation breaks the build mid-sequence; defaults keep every per-tool commit green, then a final hardening reaches the no-default end state | +| SDK-region Makefile vars | namespace under `main_` | **left un-namespaced** (LIB*_ELF…) | already main-only by the `ifeq ($(BINARY),main)` gate; namespacing a hypothetical future collision is speculative churn (de-risked scope) — deferred to when a 2nd binary needs SDK regions | +| `psyq_identify` caller argv | "update all callers in T3" | **distributed to each caller's own task** (T4/T8a) | one-touch-per-file; psyq_identify's transitional defaults kept callers green meanwhile | +| `ld_interleave`/curation `--exe`/`--vram-base` | "required" | **CLI-default to EXE values** (entry-point convenience) | these are EXE-curation tools / EXE-gated; they thread EXPLICIT values down to the now-required pipeline — the "no silent default" holds where it matters (the pipeline) | +| `difficulty.md` | not in plan | **regenerated** (committed) | the Phase-8 deferred regen surfaced; old-code==new-code output verified → the diff is the deferral, not a T6 bug | +| Ghidra DB at phase-end | (not anticipated) | **stop MCP + commit DB** (R23) | Drew raised the lock/backup gap: the `.rep` lock holds until MCP closes, and SessionEnd is too late for a mid-session phase-end commit | + +## Commit Message +``` +(Phase 9 landed as 11 per-tool checkpoint commits commit:0042..commit:0052 + this PhaseEnd.) + +feat(phase-9): binary-agnostic toolchain refactor — one pipeline builds any binary (v1.9.0) + +- Makefile: data-driven BINARIES (alias keys) + main_* vars + $(BINARY) aliases; EXE + paths preserved verbatim; 9 psyq_integrate calls + ld_interleave gated ifeq(BINARY,main) +- link pipeline (psyq_link/identify/link_region/link_lib/integrate): --vram-base/--exe + (+ integrate --symbols) REQUIRED, module EXE/VRAM_BASE removed, threaded leaf-first via a + transitional-default technique (every per-tool commit byte-identical) +- reports (progress/difficulty/dup_report): --binary selector; progress regex consumes + the new leading flags in lockstep. diff_settings: BFM_BINARY env selector +- curation helpers (ld_interleave --front/--tail; make_snd/apicard/gen_lib_subsegs + --vram-base/--exe; split_src_region --symbols) de-defaulted +- docs: SETUP §6.7 + cookbook §9.7 (the reusable Gen2 pattern) + psyq-worklist recipe + + README; difficulty.md deferred-regen resolved +- MILESTONE: EXE rebuilds 143dbb89 WITH and WITHOUT SDK objects; report 52/959/7/50.24%; + wrong --vram-base -> cae22f7e (negative control); required params fail loud +- rule R23 (stop Ghidra MCP + commit DB at phase-end/RE-checkpoint); bumps 1.8.0 -> 1.9.0 +``` + +## Rules Added This Phase +| Rule | Reason | +|---|---| +| **R23 — Stop the Ghidra MCP before any phase-end / RE-checkpoint commit; commit the DB if RE work changed it.** The headless MCP server holds the `.rep` lock with an open transaction — the saved program DB only materializes on a clean stop (`tools/ghidra_mcp_stop.sh`, which waits for *"Save succeeded"* + lock release). Phase-end/checkpoint commits happen **mid-session**, so the **SessionEnd** hook is too late — run `ghidra_mcp_stop.sh` explicitly first, then commit the current `ghidra/` DB (the live `db.N.gbf`) when RE work changed it. Between RE checkpoints, the `db.*.gbf` rename-on-save churn in `git status` is noise — don't commit no-op renumbers. Extends R20 (backup) + R9 (verify persistence). | Drew (2026-06-15): the DB is the irreplaceable RE artifact (types/comments/imported overlays beyond `symbols.us.txt`), it's tracked-but-not-actually-saved-when-current, and the lock won't release until MCP closes — so a phase-end commit must stop MCP itself, not wait for session close. | + +*(Not elevated to rules — captured in cookbook §9.7: the transitional-default + leaf-first technique for refactoring a shared in-process global without breaking the byte oracle; the negative-control discipline (a no-op can pass for the wrong reason — pass a deliberately wrong value too); the progress.py-regex/Makefile lockstep. SETUP §6.7 documents the future-binary path convention.)* + +## PhaseEnd Changelog +**v1.8.0 → v1.9.0 — Phase 9 complete (Gen2 phase 2).** The single-binary toolchain is now +**binary-agnostic**: a data-driven `BINARIES` Makefile (`make build BINARY=`) + every link-pipeline, +report, curation, and asm-differ tool parameterized by **required** `--vram-base`/`--exe`/`--symbols`/ +`--binary`/`BFM_BINARY` — **no EXE default an overlay could silently inherit** (the roadmap's #1 risk, +eliminated by construction; a miss fails loud). Proven a **pure no-op**: the EXE rebuilds `143dbb89…` through +the parameterized path with and without the SDK objects, `make report` reproduces 52 REAL / 959 LINKED / +50.24%, and a deliberately wrong `--vram-base` diverges (`cae22f7e…`, the negative control). Landed as 11 +per-tool checkpoint commits (each byte-gated), using a **transitional-default technique** (params default to +kept globals, removed → required once every caller passes explicitly) so the byte-locked EXE never broke +mid-refactor. The EXE's artifact paths are preserved verbatim; the clean `` convention + the EXE-only +`ifeq ($(BINARY),main)` gates are documented (SETUP §6.7, cookbook §9.7) for Phase 10's resident blob. One +rule (R23, stop-MCP-then-commit-DB at phase-end). No tools installed. The toolchain is ready to point at the +first overlay binary. + +## Notes for Future Phases +- **Gen2 candidate (raised Phase 9):** extend `ExportSymbols.java` to also export Ghidra **types + comments** + to text. Today only function names/addresses are text-exported (`config/symbols.us.txt`, R15); types/ + comments/imported-overlays live only in the binary `db.*.gbf`, which **renames on every save** → committing + it bloats `.git` ~10 MB/checkpoint (R20 accepts this; clean public mirror later). Text-export would make the + DB regenerable, eliminate the binary-blob backup + the R23 lock/commit dance, and be public-clean from day one. +- **Phase 10 first step:** `config/splat.resident.yaml` + the `resident` `BINARIES` instance for + `MAIN.CD/FILE_010/1.1` @ `0x800CEDF8` (its own `resident_VRAM_BASE`); all-asm byte-match vs the extracted + payload. The Phase-3 overlay map (`docs/memory-map.md §4.3`) + the Phase-9 parameterized tools feed directly in. + +## 🛑 Stop Here +PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase9.md` (R19). The Phase-9 work is +committed (11 checkpoints `commit:0042`..`commit:0052` + this PhaseEnd commit, which also reconciles the Ghidra DB per +R23); **Drew pushes** (R6). Gen2 continues — do **NOT** start Phase 10 here. Start a **fresh session** +(effort **Max**, plan mode) for **Phase 10 — Resident engine blob**. Keep this file forever. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase9.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase9.md