# SETUP.md — Environment Setup & Daily Operations Reference > **This file is the EVOLVABLE reference layer.** Unlike `PROJECT_CONTEXT.md` (permanent, never edited), this document holds volatile facts — pinned versions, URLs, commands, ports — and **may be updated freely** as tools move. Note each change in the active phase log (`CURRENT_PHASE.md`). Items marked **TBD** / **UNVERIFIED** / **JP-only — re-derive for US** are honest gaps: confirm before relying on them, then update this file. Last full revision: 2026-06-10 (initial authoring, pre-Phase-1 — nothing below is installed yet except the repo itself; same-day conversion to the all-in-WSL / Linux-first architecture — everything now runs inside a single WSL2 Ubuntu 24.04 clone, no Windows/WSL split). **Refresh 2026-06-15:** added a tooling/MCP-lifecycle/session-hooks/backup-posture pass after the doc had drifted past the as-built reality — new §1a (`.run/` scratch), §2.8 (MCP lifecycle, persistence model & session hooks), a `## Tooling inventory` table, and a `## Backup & private-repo posture` section. Rule **R21** (added this session) now requires keeping THIS file current whenever tooling / MCP / hooks / env change. ## Version pin summary All components run inside WSL2 Ubuntu 24.04 (single ext4 clone). The "Side" column below is retained for historical continuity but every entry is now **WSL2 (Linux)**. | Component | Pinned version | Side | Phase | |---|---|---|---| | JDK | 21 (Temurin) | WSL2 (Linux) | 1 | | Ghidra | **12.1 PUBLIC exactly** (`ghidra_12.1_PUBLIC_20260513.zip`) — NOT 12.1.2 | WSL2 (Linux) | 1 | | GhidrAssistMCP | v2.8.0 (`ghidra_12.1_PUBLIC_20260530_GhidrAssistMCP.zip`) | WSL2 (Linux) | 1 | | ghidra_psx_ldr | release 2026.06.04 (`ghidra_12.1_PUBLIC_20260604_ghidra_psx_ldr.zip`) | WSL2 (Linux) | 1 | | PCSX-Redux | Linux build (AppImage/flatpak; no version pin; record build on install) | WSL2 (Linux) | 3 | | WSL2 distro | Ubuntu-24.04 (Python 3.12 ships with it) | WSL2 (Linux) | 4 | | splat | pip `splat64[mips]` `>=0.41.0,<1.0.0` — **as-built 0.41.0** (Phase 4); freeze exact once Phase 5 green | WSL2 (Linux) | 4 | | Vintage compiler | decompals/old-gcc **release 0.17**: `gcc-2.7.2-psx` + `gcc-2.7.2-cdk` | WSL2 (Linux) | 4 | | maspsx | git submodule, `mkst/maspsx` (decomp.me pins commit `874855c53f65f8fa57447e1da6bde6236dbef9d5` — reasonable default pin) | WSL2 (Linux) | 4 | | asm-differ / m2c / decomp-permuter | git submodules (URLs in §4.6) | WSL2 (Linux) | 4 | | binutils (mipsel) | apt `binutils-mipsel-linux-gnu` — **as-built 2.42** (Phase 4; ≥2.38 → check-env WARN, verdict deferred to Phase 5), 2.35 known-good | WSL2 (Linux) | 4 | --- ## §1 The all-in-WSL environment (single ext4 clone) ``` ┌─ WSL2 Ubuntu 24.04 (ext4) — everything lives here ──────────────────────────┐ │ │ │ ~/bfm-decomp ← THE clone (one git working tree, one remote) │ │ ├─ Ghidra 12.1 PUBLIC (JDK 21 Temurin, GUI via WSLg) │ │ │ ├─ ghidra_psx_ldr 2026.06.04 (PSX loader, PsyQ signatures, │ │ │ │ .gdt type archives) │ │ │ └─ GhidrAssistMCP v2.8.0 ── SSE 127.0.0.1:8080 (local to WSL) │ │ │ │ │ ├─ Ghidra project .gpr/.rep (under ~/bfm-decomp or ~/.ghidra) │ │ ├─ PCSX-Redux (Linux build: runtime oracle — debugger, Lua, │ │ │ web API RAM dumps, GDB server :3333) │ │ ├─ Claude Code ← .mcp.json → 127.0.0.1:8080/sse (same box, no boundary) │ │ │ │ │ ├─ .venv/ splat64[mips] etc. │ │ ├─ tools/maspsx, tools/asm-differ, tools/m2c, tools/decomp-permuter (subm.)│ │ ├─ tools/bin/gcc-2.7.2-psx, tools/bin/gcc-2.7.2-cdk │ │ │ (old-gcc 0.17, Linux x86-64 binaries) │ │ ├─ disks/ BIN/CUE dump, copied in ONCE (gitignored) │ │ └─ asm/ build/ expected/ (generated, never committed) │ │ │ │ apt: binutils/gcc-mipsel-linux-gnu, make, ninja, python3.12, JDK 21, ... │ └───────────────────────────────────┬─────────────────────────────────────────┘ │ └── git remote (push/pull off-box only) ──► ``` One repository clone on ext4 at `~/bfm-decomp` holds everything — RE stack, build toolchain, runtime oracle, docs, and Claude Code's working directory. There is **no Windows/WSL split**: no second clone, no cross-OS working-tree edits, no `/mnt` build path. The git remote is used only to push/pull off the machine (backup, collaboration), never to synchronize two local trees. Rationale for keeping the clone on ext4 (not on a `/mnt/*` drvfs mount): - **9P filesystem penalty.** File access on `/mnt/*` drvfs mounts goes through the Plan 9 protocol and is the slowest path WSL2 offers — Microsoft's own numbers put ext4 at 2–20× faster, and community benchmarks measured builds up to 375% faster after moving off `/mnt`. Every `make`, `git status`, and splat extract pays the tax if the build tree sits on a mounted Windows drive. Keeping the clone on ext4 sidesteps this entirely. - **inotify works on ext4.** Linux file-watchers get no events for changes on `/mnt/*` drvfs mounts (microsoft/WSL [#4739](https://github.com/microsoft/WSL/issues/4739), [#5424](https://github.com/microsoft/WSL/issues/5424)). asm-differ watch mode fires reliably only when source *and* build outputs live on ext4 — which, all-in-WSL, they always do. - **No ghost-file hazard.** anthropics/claude-code [#28015](https://github.com/anthropics/claude-code/issues/28015): Write/Edit on WSL2 drvfs mounts can hit a statx/9P cache-poisoning kernel bug producing "ghost files" (ENOENT for files that exist). Editing only on ext4 avoids the bug; Claude Code runs inside WSL and never touches a drvfs path. One-shot file copies onto ext4 (e.g. the disc dump into `disks/`) from any source are fine — it is sustained/random IO and watchers on drvfs that break, not correctness. Ghidra project files (`.gpr`/`.rep`) live on ext4 alongside the clone (or under `~/.ghidra`), consistent with Ghidra's local-drive-storage recommendation; their `~lock` files behave correctly on a native Linux filesystem. Git hygiene: the committed `.gitattributes` (`* text=auto eol=lf` + binary exclusions) keeps line endings consistent. On the Linux clone `core.filemode true` is fine (ext4) and `core.autocrlf` is `false` by default (LF-native), so no line-ending dance is needed. ### §1a Runtime scratch (`.run/`, rule R12) Gitignored project-local scratch directory at `~/bfm-decomp/.run/` — the standing replacement for `/tmp` (rule R12: never write to `/tmp`; all runtime data is project-local). Holds purely regenerable transients: the headless MCP server log (`ghidra-mcp.log`), the clean-shutdown sentinel (`mcp-stop.req`, see §2.8), signature dumps (`sig.*.jsonl`, produced by `make sig-refresh`), decomp-permuter scratch (`.run/permuter/`), build/extract logs, and assorted `tmp*` working files. Everything here is regenerated on demand and **never committed** — the directory exists only so no working data ever lands in `/tmp` or pollutes the tree. --- ## §2 RE stack install (Phase 1, all in WSL) Order matters: JDK → Ghidra → both extensions → import → MCP wiring. ### §2.1 JDK 21 (Temurin) ```bash sudo apt-get update && sudo apt-get install -y openjdk-21-jdk ``` Required by Ghidra 12.x. (The Temurin apt repo — `https://packages.adoptium.net` — is an equivalent alternative if you prefer Temurin specifically.) Verify: `java -version` reports 21.x, and ensure it is the JDK on `PATH` for the shell that launches Ghidra (`ghidraRun` needs JDK 21 visible). ### §2.2 Ghidra 12.1 PUBLIC — pin EXACTLY Download **`ghidra_12.1_PUBLIC_20260513.zip`** from `https://github.com/NationalSecurityAgency/ghidra/releases/tag/Ghidra_12.1_build` — plain zip, no installer; the same cross-platform Java distribution on every OS. Unzip and launch the GUI under WSLg: ```bash cd ~ && unzip ghidra_12.1_PUBLIC_20260513.zip ~/ghidra_12.1_PUBLIC/ghidraRun # WSLg supplies the X11/Wayland display ``` `./ghidraRun` needs JDK 21 on `PATH` (§2.1); the Ghidra GUI renders through WSLg with no extra X-server setup. > ⚠️ **Extension version-lock warning.** Ghidra extensions are point-version-locked via `extension.properties`. Both extension zips below are built against `12.1_PUBLIC`; installing them on 12.1.2 (current latest, 2026-06-05) may trip the version check (red "incompatible" flag — same failure mode as LaurieWired issue #83 on 11.4.1). Whether the 12.1 zips happen to load on 12.1.2 is **UNVERIFIED** — pin **12.1**, and only move to a point release when both extensions ship matching assets or you rebuild them from source (`gradle installExtension` with `GHIDRA_INSTALL_DIR` set). The extension assets are platform-independent Java (`ghidra_12.1_PUBLIC_..._GhidrAssistMCP.zip` / `..._ghidra_psx_ldr.zip` — the "12.1" is the Ghidra version, not the OS), so the same asset filenames install on Linux Ghidra 12.1. ### §2.3 GhidrAssistMCP v2.8.0 - Asset for our Ghidra: **`ghidra_12.1_PUBLIC_20260530_GhidrAssistMCP.zip`** from `https://github.com/symgraph/GhidrAssistMCP/releases/tag/2.8.0` (repo moved from `jtang613/GhidrAssistMCP` to `symgraph/GhidrAssistMCP`; GitHub redirects. The release carries TWO assets — take the `12.1` one, not `12.0`.) - Install: Ghidra → **File → Install Extensions… → `+`** → select zip → restart Ghidra. - Enable plugin: **File → Configure → Configure Plugins → check `GhidrAssistMCP`**. - Control panel: **Window → GhidrAssistMCP** → set **Host = localhost, Port = 8080**. Server exposes `/sse` (SSE), `/message`, and `/mcp` (streamable HTTP). No Python bridge process — Claude Code connects directly. - This is the server the psxrecomp proof-of-concept actually ran with Claude Code (their port was 7777, a user setting; we standardize on 8080 to match the committed `.mcp.json`). **Fallbacks (documented, not installed):** | Server | Status | Why fallback only | |---|---|---| | bethington/ghidra-mcp v5.13.x | active, claims Ghidra 12.1, 249 tools | Unaudited, single-maintainer with day-apart releases, needs a Python bridge (plugin HTTP :8089), 249 tools swamp Claude's tool context. Consider only if a missing niche tool (mass `create_enum`, `batch_decompile`) becomes a bottleneck. | | clearbluejar/pyghidra-mcp | active, pip/uvx, headless multi-binary | **No struct-creation or apply-type-at-address tools** — dead end as primary server for matching-decomp data work; fine for scripted headless batch passes. | | LaurieWired/GhidraMCP 1.4 | **unmaintained — DO NOT INSTALL** | Caps at Ghidra 11.3.2 (issues #83/#131 open, no fix since 2025-06-23); ghidra_psx_ldr dropped 11.x. Older write-ups (including the 1379.tech psxrecomp blog post) credit it incorrectly. | | ismaelcaraballo-afk/GhidraMCP-12 | stale one-off recompile | 12.0.1 only, no struct tools, not viable. | ### §2.4 ghidra_psx_ldr release 2026.06.04 - Asset: **`ghidra_12.1_PUBLIC_20260604_ghidra_psx_ldr.zip`** (~9.15 MB, PsyQ signatures bundled — no separate OBJ download needed) from `https://github.com/lab313ru/ghidra_psx_ldr/releases/tag/2026.06.04` - Install via the same **File → Install Extensions…** path → restart. - Coexists with GhidrAssistMCP without conflicts (PSX loader/analyzer/SLEIGH vs HTTP-server plugin). ### §2.5 PSX-EXE import flow (SLUS_007.26) 1. Extract `SLUS_007.26` from the disc image (LBA 24, 0x65000 bytes, track 1 is MODE2/2352) and import into Ghidra. The loader auto-selects **"PSX Executables Loader"**, language `PSX:LE:32:default`, and builds the full PS1 memory map: RAM around the image at 0x80000000, scratchpad 0x1F800000, all IO/DMA/timer/CD/GPU/SPU register blocks, **and a synthetic GTEMAC segment at 0x20000000 automatically** — do NOT run the `CreateGteMacSegment` script (that is only for migrating legacy non-PSX projects). 2. Run auto-analysis with the **"PsyQ Signatures"** analyzer enabled (auto-enabled for PSX-language programs). Analyzer options: "Only first match", "Minimal signature entropy" (default 3.0 — can skip tiny low-entropy library stubs), "PsyQ Version if not found" (manual override). 3. Read the detected PsyQ version: **Edit → Options for Program → Program Information → "PsyQ Version"**. Expected: **4.0** (our own EXE scan found 12 genuine `Ps` stamps: 9× 4.0, one 4.0.1x on libnum 16, one 4.2 on libnum 0, one 4.2.1x on libnum 12 — i.e. PsyQ 4.0 libs + 4.2 library updates; see §5.1). If detection errors with `'psyq/xx' cannot be found`, append `.0` to the version field. Record the detected value in the phase log. **CONFIRMED 2026-06-13 (Phase 1):** DetectPsyQ at headless import recorded `PsyQ Version = 4.0.0` on the extracted US EXE (resolves ledger #12). Import also reported: loader `PSX Executables Loader`, language `PSX:LE:32:default`, ImageBase `80000000`, address range `1f800000–801fffff`, 1726 functions, ~177 s analysis. *(Note: a raw track-1 scan during research reported slightly different per-libnum details — raw 2352-byte-sector scans produce false positives; the extracted-EXE scan is the ground truth, and DetectPsyQ at import is the final word.)* 4. **One-time manual `.gdt` attach (GUI only — no MCP tool opens archives):** in the CodeBrowser Data Type Manager, attach the bundled PsyQ type archive for the detected version — `psyq400.gdt` (`psyq420.gdt` also exists). This gives PsyQ struct/typedef types for retyping work. 5. **Early MCP type-resolution test (run before any bulk typing):** via MCP, run the `types` tool with `action=set` applying a PsyQ type (e.g. apply a known PsyQ struct at some address) and confirm it resolves. **RESOLVED 2026-06-13 (Phase 1) — answer: import the types into the program.** The `.gdt` attach is doable **headlessly** (no GUI needed) via `tools/ghidra_scripts/ImportPsyqGdt.java`, which opens `psyq400.gdt` as a `FileDataTypeManager` and `resolve()`s all its types into the program's DataTypeManager — this both copies the types in **and** registers `psyq400` as a SourceArchive. After that, the MCP `types` tool fully resolves them: `types action=list filter=DRAWENV` and `types action=get name=DRAWENV` return the complete struct (`DRAWENV` [/LIBGPU.H], 92 bytes, all fields incl. `RECT`/`DR_ENV`), and `SVECTOR` [/LIBGTE.H]. psyq400.gdt carries **2599 types** (program DTM went 205 → 2609). **Takeaway for the typing workflow:** don't rely on a bare archive *attachment* — import (resolve) the PsyQ types into the program once, and all MCP type/struct tools see them. The bundled archives live at `/Ghidra/Extensions/ghidra_psx_ldr/data/psyq*.gdt` (psyq400.gdt for our SDK). ### §2.6 `.mcp.json` wiring + verification The committed repo-root `.mcp.json` (already present): ```json { "mcpServers": { "ghidra": { "type": "sse", "url": "http://127.0.0.1:8080/sse" } } } ``` Single `ghidra` entry only — do NOT copy psxrecomp's duplicated `ghidra` + `ghidra_psx` pair (same URL twice = every tool duplicated in context). Verify in Claude Code with `/mcp`: **~38–41 `mcp__ghidra__*` tools** should appear. *(CONFIRMED 2026-06-13: GhidrAssistMCP v2.8.0 reports **41 tools** headless. Operational notes from Phase 1: tool responses come back as SSE frames (`event: message` / `data: {json}`) on the streamable `/mcp` endpoint; heavy tools like `get_code` run **asynchronously** — the `tools/call` returns a `task_id` and you must poll `get_task_status` for the result. `get_code` takes `{function: "0x80018730", format: "decompiler|disassembly|pcode"}`.)* These are the **v2.8.0 names** — e.g. `get_binary_info`, `get_code` (format: disassembly|decompiler), `disassemble_at`, `analyze_function`, `xrefs`, `get_functions` (paginated), `struct` (actions: create/modify/merge/set_field/name_gap/auto_create/rename_field/field_xrefs), `types` (list/get_info/set/delete), `variables` (list/rename/set_type/set_prototype), `rename_symbol`, `batch_rename`, `create_data_var`, `create_function`, `search_bytes`, `patch_bytes`, `assemble_code`, `export_program`. **NOT the pre-2.4.0 names** (`get_function_info`, `list_data`) that appear in psxrecomp's PLAN.md and older write-ups — those were renamed in v2.4.0 (upstream commit `aa3ffc7d`, 2026-03-14). Operational cautions: - Tools operate on the program currently open in CodeBrowser and fail (sometimes silently) if none is open. - Do not run Ghidra auto-analysis concurrently with MCP-driven writes (renames/comments) — both mutate the program DB and can stall each other. ### §2.7 Headless mode (batch passes) For unattended batch passes, GhidrAssistMCP runs headless (supported since v2.3.0): ```bash /support/analyzeHeadless McpHeadless \ -import \ -scriptPath "/ghidra_scripts" \ -preScript GAMCPStartServerScript.java "host=127.0.0.1" "port=8080" "wait=true" ``` - **`wait=true` is mandatory** — without it the analyzeHeadless process exits right after the prescript instead of serving MCP clients. - On subsequent runs use **`-process SLUS_007.26`** (not `-import`) to reuse the existing project. Pair with **`-noanalysis`** so it doesn't re-run analysis on every server start. - **CONFIRMED 2026-06-13:** this headless flow works end-to-end (server "started on port 8080 … with 41 tools"; `get_binary_info` and `get_code` operate on the `-process` program). **The headless server holds the project `.rep` lock while serving** — stop it (cancel the analyzeHeadless process) before opening the same project in the GUI. - **Extension install (headless-compatible):** extract each extension zip into `/Ghidra/Extensions/` (e.g. `unzip ext.zip -d ~/ghidra_12.1_PUBLIC/Ghidra/Extensions/`). Both GUI and `analyzeHeadless` then load the extracted module dirs (no GUI "Install Extensions" step needed). Verified for GhidrAssistMCP + ghidra_psx_ldr on Ghidra 12.1. - **Loader selection in headless:** `-loader "PSX Executables Loader"` is **rejected** (`InvalidInputException: Invalid loader name specified`) even though that is the loader's display name. **Omit `-loader` and let auto-detection pick** — for a real `PS-X EXE` it correctly selects "PSX Executables Loader" over Raw Binary (log line: `Using Loader: PSX Executables Loader`). ### §2.8 MCP server lifecycle, persistence model & session hooks We drive RE through our own headless MCP server, `tools/ghidra_scripts/BfmMcpServer.java`, run under `analyzeHeadless` (the §2.7 flow). **Persistence model — read this before doing RE:** the server holds an **OPEN TRANSACTION** the entire time it serves, so there is **NO mid-session save** — MCP writes (renames, retypes, comments, structs) live in RAM and are flushed to the program DB **only on a clean shutdown**. A hard crash (or `SIGKILL`) loses every RAM-only write since the last clean stop. Therefore: take **clean-stop checkpoints during long RE** rather than trusting a single save at the end. Lifecycle scripts (under `tools/`): - **`ghidra_mcp_start.sh`** — spawns the headless server detached, logging to `.run/ghidra-mcp.log`, serving on port **8080**. - **`ghidra_mcp_stop.sh`** — the **only persistence event**: it requests a clean save+close by dropping the `.run/mcp-stop.req` sentinel, waits for the server to report **"Save succeeded"**, then releases the project `.rep` lock. **Never `SIGKILL` the server to stop it** — that skips the save and loses the work. Clean stop is the save. - **`ghidra_mcp_verify.sh `** — read-only persistence re-check (rule R9): after a clean stop, re-reads the named symbol at the address to confirm the write actually landed on disk. **Client reconnect after a server restart (operational, Phase 13).** Restarting the server (`stop` then `start` — e.g. for a headless raw-blob import per R23, or to serve a different program) **drops the Claude Code MCP client's SSE connection**: every `mcp__ghidra__*` call then **times out** until the client reconnects, and Claude **cannot** run `/mcp` itself. So the rhythm after any server restart / program switch is: **pause and ask Drew to run `/mcp`**, then make one cheap `get_binary_info` call (G2) before continuing. Do not try the calls, hit timeouts, and work around them. (Memory: `mcp-reconnect-after-restart`.) **Session hooks (committed `.claude/settings.json`, as of 2026-06-15):** - `SessionStart` → runs `ghidra_mcp_start.sh` (auto-starts the MCP server when a Claude Code session begins). - `SessionEnd` → runs `ghidra_mcp_stop.sh` with **`timeout: 150`** s (auto-saves Ghidra on a clean session exit). These hooks live in the **committed `settings.json`** — NOT the gitignored `settings.local.json` — specifically so they are backed up to the remote. Consequence to internalize: **closing Claude Code does NOT save unless the `SessionEnd` hook fires**, and it fires only on *clean* exits — a hard crash of the CC process still loses RAM-only writes. This is exactly why mid-RE clean-stop checkpoints matter. **Committing Ghidra RE work:** clean-stop (which saves) **FIRST**, then commit `ghidra/`. The `.rep` lock never blocks git (the lock is gitignored, and ext4 reads open files fine), but do **not** commit `ghidra/` mid-RE — that snapshots a stale on-disk DB that predates the in-RAM writes. --- ## §3 PCSX-Redux (Linux) — the runtime oracle **Role:** Ghidra is the static oracle; PCSX-Redux is the **runtime oracle**. It proves overlay load addresses by live RAM comparison, validates decompressed blobs byte-for-byte against what the game actually loads, and drives the debug-menu/loader RE. - **Install:** Linux AppImage from `https://distrib.app/pub/org/pcsx-redux/project/dev-linux-x64` (no stable pin — record the build date). **As-built (2026-06-13):** `tools/pcsx-redux/PCSX-Redux-HEAD-x86_64.AppImage` (87.5 MB, ELF static-pie, BuildID `db3b55ac…`). **distrib.app is a JS SPA — not curl-able headlessly** (the `dev-linux-x64` URL returns only an HTML shell; the real file is numeric-ID-gated); download via a browser. Runs directly under WSLg (FUSE 3.14.0 present; no `--appimage-extract` needed). Docs: `https://pcsx-redux.consoledev.net`. Ships OpenBIOS (boots without a retail BIOS dump). Launch with disc + auto-run: `./tools/pcsx-redux/PCSX-Redux-HEAD-x86_64.AppImage -iso "" -run`. - **RAM dumps (primary use):** enable the built-in web server, then **`GET http://127.0.0.1:8081/api/v1/cpu/ram/raw`** returns the full 2 MB RAM image — diff slices of it against our extractor's output to prove byte-identity. (Port **8081**, resolved at Phase 3 install — ledger #4 — moved off 8080 to avoid the GhidrAssistMCP collision. **Verified 2026-06-13:** returns exactly 2,097,152 B; RAM[0x10000:] == extracted `SLUS_007.26`[0x800:] byte-for-byte.) > ✅ **Port collision (RESOLVED 2026-06-13):** PCSX-Redux's web server defaults to `localhost:8080` — same as GhidrAssistMCP. GhidrAssistMCP stays on 8080 (`.mcp.json`); the Redux web server moves to **8081** via `~/.config/pcsx-redux/pcsx.json` → `emulator.Debug.WebServer=true` + `emulator.Debug.WebServerPort=8081` (GUI: Configuration → Emulation → Web Server). Edit `pcsx.json` only while Redux is **stopped** (it rewrites the file on exit). - **Lua scripting:** LuaJIT 2.1.0-beta3 (Lua 5.2 compat), FFI direct memory access; console + editor under the Debug menu. Gotcha: the editor autosaves to `pcsx.lua` and reloads it at startup — a crashing script wedges the emulator across restarts (delete `pcsx.lua` to recover); long-running scripts must yield via coroutines (~15 ms/frame budget). - **Debugger:** fully featured MIPS debugger, VRAM/SPU viewers. GDB server on port 3333 (Configuration → Emulation → Enable GDB server); for Ghidra-attached debugging launch Redux with `-interpreter -debugger -gdb` (dynarec breaks debugging) and connect `gdb-multiarch -i mi2` → `target remote localhost:3333`. - PSX RAM at 0x80000000 mirrors physical 0x0 (0x800425D0 ≡ 0x000425D0) — relevant when reading dump offsets. --- ## §4 Build environment (Phase 4) Everything below installs inside the same WSL2 Ubuntu 24.04 that already hosts Ghidra and Claude Code (§1). Where Phase 1 (§2) already set up the distro and JDK, this phase adds the build toolchain on top. ### §4.1 WSL2 + Ubuntu 24.04 (already present from Phase 1) The all-in-WSL architecture means WSL2 Ubuntu 24.04 is the single host for the whole project, so it exists before Phase 4 begins (it is the same environment §2 installed Ghidra into). Confirm it is the expected distro and version: ```bash cat /etc/os-release # Ubuntu 24.04 uname -a # Linux kernel (WSL2) whoami # the Linux username; ~ resolves to /home/ ``` All project paths are plain Linux paths under `~/bfm-decomp` — there is no Windows distro name or `wsl.exe --cd` target to track. ### §4.2 Networking: MCP is local Under the all-in-WSL architecture there is **no cross-OS networking**. Ghidra/GhidrAssistMCP, PCSX-Redux, and Claude Code all run inside the same WSL2 instance, so the MCP endpoint is plain loopback: **MCP is local to WSL at `http://127.0.0.1:8080`; no mirrored-mode `.wslconfig`, firewall rule, or host-IP discovery is needed.** Smoke-test with Ghidra running: `curl http://127.0.0.1:8080/` from any shell in the same WSL instance. ### §4.3 The clone on ext4 ```bash cd ~ && git clone bfm-decomp ``` The single clone lives at `~/bfm-decomp` (ext4). Builds, splat, asm-differ, Ghidra, and Claude Code all run here. In this clone: `git config core.filemode true`. **TBD:** the canonical remote URL (GitHub private repo planned; not created as of this writing). ### §4.4 Copy the disc dump into the clone One-shot copy onto ext4 is fine (and required once): ```bash mkdir -p ~/bfm-decomp/disks cp '/Brave Fencer Musashi (USA)/'*.bin \ '/Brave Fencer Musashi (USA)/'*.cue ~/bfm-decomp/disks/ ``` `` is wherever the disc dump currently lives (e.g. a one-time download into `~/Downloads`, or a one-shot copy from external media). `disks/` is gitignored — no ROM-derived bytes ever reach the remote (rule H1). **Status (Phase 2, 2026-06-13):** the disc was staged early — extraction needs it before Phase 4. Track 1 alone (it holds all 27 root files) was copied once from the `/mnt/z` dump to ext4 at `disks/Brave Fencer Musashi (USA) (Track 1).bin` (364,846,944 bytes). WSL `extract_exe.py --bin "disks/…(Track 1).bin" --verify-disc` **PASSED** — SHA1 `b44f0f0a19936f23b26188b658e13201a6a9c211`, CRC32 `c238191b`, both == redump — which **closes the Phase-1 deferral** (verify-disc had previously only run on Windows; PhaseEnd_Phase1 Deviations). ### §4.5 apt packages Adapted from sotn-decomp's `tools/requirements-debian.txt` (dropped Saturn/PSP-only items `binutils-sh-elf`, `xfonts-utils`; Rust/Go deferred until a duplicate-detector or asset tool needs them): ```bash sudo apt-get update && sudo apt-get install -y \ bchunk binutils-mipsel-linux-gnu bsdmainutils clang-format coreutils curl \ gcc-mipsel-linux-gnu git libelf-dev make ninja-build p7zip-full \ python3-pip python3-venv unzip wget ``` > ⚠️ **binutils regression check (mandatory before trusting builds):** open-ribbon documents that `binutils-mipsel-linux-gnu >= 2.38` generated broken binaries; **2.35 is the known-good reference**. Ubuntu 24.04 ships newer binutils — **VERIFY on 24.04**: after Phase 5's first full build, if the SHA1 check mysteriously fails with correct-looking asm, suspect the assembler first (`mipsel-linux-gnu-as --version`), and pin/downgrade or build binutils 2.35 if confirmed. Record the verdict here. > > **As-built (Phase 4, 2026-06-14, ledger #6):** apt installed **binutils-mipsel-linux-gnu 2.42** (as/ld/objcopy all 2.42; mipsel-gcc 12.4.0). 2.42 ≥ 2.38, so `make check-env` emits a **[WARN]** (not FAIL) and the regression verdict is **deferred to Phase 5's first full build** exactly as above — no preemptive downgrade. > > **✅ VERDICT (Phase 5, 2026-06-14): binutils 2.42 is byte-clean — no regression with our flags.** The all-asm `make build` reproduces `SLUS_007.26` **SHA1-identical** (`143dbb89…`) using `mipsel-as` 2.42 with `-march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0`. The open-ribbon "≥2.38 broken" warning does **not** bite here; **no downgrade to 2.35 needed.** (Revisit only if Phase-6 C-compiled objects ever diff where the asm is right.) ### §4.6 Python venv + splat + submodules ```bash cd ~/bfm-decomp python3 -m venv .venv # Python >= 3.12 required (24.04 ships 3.12; older = f-string SyntaxError mid-build) .venv/bin/pip install -U 'splat64[mips]>=0.41.0,<1.0.0' ``` The PyPI package is **`splat64`** (not `splat`), and the `[mips]` extra is required for PSX (pulls spimdisasm/rabbitizer). Always invoke as `.venv/bin/splat` or `.venv/bin/python3 -m splat` — `splat: command not found` means you're outside the venv. Once Phase 5 builds green, freeze the exact working version in a committed `tools/requirements-python.txt`. Submodules (add under `tools/`): | Submodule | URL | Pin | |---|---|---| | `tools/maspsx` | `https://github.com/mkst/maspsx.git` | commit `874855c53f65f8fa57447e1da6bde6236dbef9d5` (decomp.me's pin — keeps local results comparable to decomp.me scratches) | | `tools/asm-differ` | `https://github.com/simonlindholm/asm-differ.git` | pin current HEAD at adoption | | `tools/m2c` | `https://github.com/matt-kempster/m2c.git` | pin current HEAD at adoption | | `tools/decomp-permuter` | `https://github.com/simonlindholm/decomp-permuter` | sotn pins `b44b0622269fb4bff29e79fbbad26b9f47beda79` — sane default | Pin all four (sotn precedent: blindly updating submodules breaks tooling). Note: sotn's asm-differ `--overlay` flag is **sotn-fork-specific**, not upstream — for BFM overlay diffing use upstream's `-o` object mode or port their fork later. **As-built (Phase 4, 2026-06-14):** `.venv` created (Python 3.12.3); installed **splat64 0.41.0** (`splat64[mips]`) — deps spimdisasm 1.41.0, rabbitizer 1.16.2, PyYAML 6.0.3, colorama 0.4.6, intervaltree 3.1.0, tqdm 4.67.1; `import splat` OK. Submodule pins as adopted: maspsx `874855c5`, decomp-permuter `b44b0622` (both per the table); **asm-differ `2ad4a4a4`** and **m2c `4266cc28`** (each HEAD-at-adoption). Their pip deps are **not** installed yet (Phase 6, when first invoked); `tools/requirements-python.txt` is frozen only after Phase 5 is green. ### §4.7 Vintage compilers (old-gcc 0.17) Linux prebuilts from decompals/old-gcc, **release 0.17** (32-bit i386 static — see the correction below): ```bash mkdir -p ~/bfm-decomp/tools/bin && cd ~/bfm-decomp/tools/bin wget https://github.com/decompals/old-gcc/releases/download/0.17/gcc-2.7.2-psx.tar.gz wget https://github.com/decompals/old-gcc/releases/download/0.17/gcc-2.7.2-cdk.tar.gz sha256sum gcc-2.7.2-*.tar.gz # record hashes in a committed tools/bin/*.sha256 on first download, # then verify with `sha256sum --check` on every fresh setup (sotn pattern) # The 0.17 tarballs are FLAT (no top-level dir) and SHARE filenames (cc1, cpp, gcc, ...) # -> extract each into its OWN subdir, or the second clobbers the first (Phase-4 finding): mkdir -p gcc-2.7.2-psx gcc-2.7.2-cdk tar xzf gcc-2.7.2-psx.tar.gz -C gcc-2.7.2-psx tar xzf gcc-2.7.2-cdk.tar.gz -C gcc-2.7.2-cdk ``` - `gcc-2.7.2-psx` = community GCC 2.7.2 PSX build (primary candidate). - `gcc-2.7.2-cdk` = **cygnus-2.7.2-970404**, the exact base of PsyQ 4.0/4.1's CC1PSX (added in old-gcc 0.14). - **sha256 (RECORDED Phase 4, old-gcc 0.17, ledger #7):** `gcc-2.7.2-psx.tar.gz` = `500a459b3485e885a8d302cac23c2a4632f3900e03a09153f6190699fd723571`; `gcc-2.7.2-cdk.tar.gz` = `42bb0df96db11a9b5d2e23d78bdc962791f40046280d3d360da93fe5eef6f0bb`. Committed to `tools/bin/CHECKSUMS.sha256` (gitignore exception `!/tools/bin/*.sha256`); re-verify with `sha256sum --check tools/bin/CHECKSUMS.sha256`. - **CORRECTION (Phase 4):** these are **32-bit i386 statically-linked** ELF binaries (NOT x86-64 as previously written) — they run on x86-64 WSL2 via the kernel's IA-32 emulation (verified: `cc1` smoke-compiles to MIPS asm and self-identifies as `GNU C 2.7.2 [AL 1.1, MM 40] Sony Playstation`). Still Linux-only — *why* the build side must be Linux/WSL2. As-built layout: `tools/bin/gcc-2.7.2-psx/cc1` + `tools/bin/gcc-2.7.2-cdk/cc1` (matches the §6.2 path). ### §4.8 Optional: PsyQ 4.0/4.1 binaries for arbitration (via Wine) For byte-exact arbitration when maspsx output is in doubt, the **real** PsyQ Win32 tools can be driven from WSL under Wine (`sudo apt-get install -y wine`): - `https://github.com/mkst/esa/releases/download/psyq-binaries/psyq4.0.tar.gz` - `https://github.com/mkst/esa/releases/download/psyq-binaries/psyq4.1.tar.gz` (contain `CC1PSX.EXE`, `ASPSX.EXE`, `CCPSX.EXE`, `PSYLINK.EXE`, `PSYLIB.EXE`; 1–2.3 MB each) - Their `.OBJ` output converts to ELF with **psyq-obj-parser** (part of pcsx-redux; prebuilt Linux binary: `https://github.com/decompme/compilers/releases/download/compilers/psyq-obj-parser.tar.gz`). Keep these under `tools/` on ext4 (not committed); they are a tie-breaker, not the daily pipeline. (decomp.me runs these same Win32 tools under Wine for its psyq presets — the precedent that this works headless.) > **DEFERRED to Phase 6 (Drew decision, Phase 4):** not staged in Phase 4 — fetched only if/when maspsx output is disputed during fingerprinting. Wine is not installed. The §4.8 "optional native PsyQ binaries" checkbox is consciously skipped for Phase 4. ### §4.9 `make check-env` (Phase 4 exit milestone) Phase 4's observable milestone: a `check-env` make target that asserts every §4 component (venv + splat import, cc1 binaries executable, maspsx present, mipsel-as/ld/objcopy on PATH, python >= 3.12) and exits 0 when invoked directly in the WSL clone (see §6.1). **As-built (Phase 4, 2026-06-14):** the root `Makefile` implements `check-env` (`.ONESHELL` bash; default goal `help`). Beyond the components above it also asserts **sha1(committed `extracted/retail/SLUS_007.26`) == `EXPECTED_EXE_SHA1`** (imported from `tools/bfm_extract/extract_exe.py` — fresh-clone-safe; the disc-walk `--verify-disc` needs the gitignored `disks/` and is intentionally NOT in check-env) and WARNs on binutils ≥ 2.38. **`make check-env` exits 0** (milestone met). `extract/build/check/expected/clean` exist as loud-failing Phase-5 stubs (names fixed per §6.3). --- ## §5 Compiler candidate ladder (Phase 6 fingerprinting) ### §5.1 The evidence Locally verified on the extracted US EXE (DetectPsyQ-style masked-pattern scan): **13 pattern hits, of which 12 are genuine `Ps` library stamps** — 9× PsyQ **4.0** (libnums 2, 3, 4, 6, 7, 8, 9, 17, 24), 1× **4.0.1x** (libnum 16), 1× **4.2** (libnum 0), 1× **4.2.1x** (libnum 12); the remaining hit (ver 0x0000 at vaddr 0x8005CD20) is a code false positive. A raw-track scan during research reported 16 hits with extra spurious 4.0 stamps — the extracted-EXE scan is ground truth, and ghidra_psx_ldr's detection at import is the final word (ledger #12). Library copyright string `(c) 1993-1997 Sony` corroborates the era. Conclusion: BFM links **PsyQ 4.0 libraries with 4.2 library updates** ⇒ the GCC 2.7.2/SN32-era toolchain — **NOT sotn's GCC 2.6.3** (the starting-point doc's claim is corrected). PsyQ 4.2 was a library-only refresh: no 4.2 toolchain disc survives (absent from redump/arthus sets), so 4.2 stamps still mean the 4.0/4.1 toolchain. Caveat: `Ps` stamps date the **linked libraries**, not the compiler that built game code — Square mixed cc1 builds within one EXE (see §5.5). The final triple is pinned only by Phase-6 fingerprinting. ### §5.2 SDK → GCC → ASPSX mapping (verified from the actual binaries in mkst/esa psyq-binaries tarballs) | PsyQ SDK | CC1PSX identifies as | ASPSX | old-gcc 0.17 artifact | maspsx flag | |---|---|---|---|---| | 3.3 | GNU C 2.6.0 [AL 1.1, MM 40] | 2.21 | `gcc-2.6.0-psx` | `--aspsx-version=2.21` | | 3.5 | GNU C 2.6.0 (same binary) | 2.34 | `gcc-2.6.0-psx` | `--aspsx-version=2.34` | | 3.6 | GNU C 2.7.2.SN.1 | 2.34 | `gcc-2.7.2` (vanilla-ish) | `--aspsx-version=2.34` | | **4.0** | **GNU C 2.7.2.SN32.3.7.0002** | **2.56** | **`gcc-2.7.2-psx` / `gcc-2.7.2-cdk`** | **`--aspsx-version=2.56`** | | **4.1** | **cygnus-2.7.2-970404 SN32.3.7.0004 (SonyPSX)** | **2.67** | **`gcc-2.7.2-cdk`** (exact base) | **`--aspsx-version=2.67`** | | 4.2 | *library-only release — no toolchain exists* | n/a | use 4.0/4.1 row | use 4.0/4.1 row | | 4.3 | GNU C 2.8.0 SN32 Build 4.0.0007 (community shorthand "2.8.1" is wrong — that's 4.4) | 2.77 | `gcc-2.8.0-psx` | `--aspsx-version=2.77` | | 4.4 | GNU C 2.8.1 SN32 BUILD 4.0.0010 | 2.79 | `gcc-2.8.1-psx` | `--aspsx-version=2.79` | | 4.5 | egcs-2.91.66 (egcs-1.1.2) | 2.81 | `gcc-2.91.66-psx` | `--aspsx-version=2.81` | | 4.6 | GNU C 2.95.2 BUILD 4.0.0030 | 2.86 | `gcc-2.95.2-psx` | `--aspsx-version=2.86` | > ⚠️ **`--aspsx-version` MUST always be passed explicitly.** maspsx with no flag is *not* "latest behavior" — the dataclass defaults approximate ASPSX ~2.3x (`expand_li=True`, `sltu_at=True`, `nop_mflo_mfhi=True`). Behavior thresholds: `expand_li` off ≥2.50; `sltu_at` off ≥2.60; `$gp` symbol+offset ≥2.70; `$gp` for `la` ≥2.80. The observable 2.56 vs 2.67 tell: "$at for `sltu < 0`" present at 2.56, gone at 2.67, and %hi/%lo support arrives at 2.67 — decide on functions with unsigned comparisons / li/sltu idioms. ### §5.3 -G0 vs -G8 Read `gp_value` from the SLUS_007.26 EXE header and check for `$gp`-relative loads in Ghidra **before** fixing the flag (**RESOLVED Phase 5 — see the ✅ verdict below: -G0**). Precedent: FF7 used `-G 0`; Xenogears used `-G8` for game code. maspsx forces `-G0` to GNU `as` by default — non-zero `$gp` needs `-G8` passed to maspsx and a look at `--dont-force-G0`. **✅ RESOLVED (Phase 5, 2026-06-14): -G0.** The header `gp_value` is 0, and the splat disassembly has **zero `($gp)` base-register accesses and zero `%gp_rel` relocations** (the 4 `$gp` mentions are crt0 register setup) — no small-data/$gp-relative addressing, i.e. the FF7-style `-G0`. The linker's `_gp=0x80074750` (splat's computed Initial-GP) is therefore inert, and the all-asm build is byte-identical. Carry `-G0` into Phase-6 cc1 fingerprinting (swap only if asm-differ ever shows otherwise). ### §5.4 Candidate ladder (try in this order) > **✅ PINNED (Phase 6, 2026-06-14) — rung 1 is the answer (G8).** Triple: > `tools/bin/gcc-2.7.2-psx/cc1 -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker` > → `maspsx --aspsx-version=2.56 **--expand-div**` → `mipsel-as -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0`. > Pinned in the `Makefile` (`CC1FLAGS` / `ASPSX_VERSION` / `MASPSX_FLAGS`). Evidence: byte-exact on `func_80018F20` > (the `sltiu` range-check probe) + instruction-identical across 2 more idiom classes (division via `--expand-div`; > memset). **`--expand-div` is required** for any div/rem (without it maspsx emits a bare `divu` with no zero-check > and div functions never match). **psx≈cdk and 2.56≈2.67 are byte-equivalent** on functions lacking the > discriminating idioms, so rungs 2–5 went unused (kept below for per-module-mixing escalation, §5.5). Reusable > codegen findings: `docs/matching-cookbook.md`. 1. **`gcc-2.7.2-psx` cc1 + `--aspsx-version=2.56`**, flags `-O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker` (FF7 style; swap to -G8 if §5.3 says so). 2. Same cc1 + `--aspsx-version=2.67` (PsyQ 4.1 assembler era). 3. **`gcc-2.7.2-cdk`** (cygnus-2.7.2-970404, the exact CC1PSX 4.0/4.1 base) × 2.56, then × 2.67. 4. Real PsyQ 4.0/4.1 `CC1PSX.EXE` + `ASPSX.EXE` (under Wine, §4.8) + psyq-obj-parser — byte-exact arbitration when maspsx emulation is in question. 5. Only if diffs show GCC 2.8-style codegen: `gcc-2.8.0-psx` + 2.77 (unlikely — the JP master predates PsyQ 4.3-era adoption). **Library-code preset** (for PsyQ SDK functions linked into the EXE): `--aspsx-version=2.56 --expand-div`, `-O3 -G0` — the Xenogears precedent, consistent with our 4.0 lib stamps. ### §5.5 Per-module mixing warning Xenogears (the closest comparable: Square US, Oct 1998) mixes **three** cc1 builds in one EXE (`gcc-2.7.2-psx`, `gcc-2.6.0-psx`, `gcc-2.7.2-cdk`) with different maspsx flags per module. Expect per-module variation in BFM: fingerprint **several modules independently** (3–5 medium leaf functions each), and design the build config for per-file compiler/flag overrides from day one (sotn does this with `//!` comments in line 1–2 of a .c file; Xenogears with `gears.toml` presets). Reference repos for build-config patterns: `https://github.com/ladysilverberg/xenogears-decomp` (gears.toml presets), `https://github.com/Drahsid/ffvii` (Makefile). ### §5.6 Compiler-quirk research reference clones (Phase 18, 2026-06-20) Phase 18 (raise the match-% ceiling by understanding gcc-2.7.2's blocking codegen quirks) reads the **real compiler source** and mines a peer decomp built with **our exact compiler**. Cloned shallow, gitignored under `/tools/reference/` (re-clonable — SHAs pinned here for reproducibility, R20/R21): | Repo | Purpose | Path | Branch | SHA (pinned) | |---|---|---|---|---| | `pmret/gcc-papermario` | ⚠️ **actually gcc 2.8.1, NOT 2.7.2** (behavioral diff: 2.8.1 `&&0`-disables biv-elim paths ENABLED in real 2.7.2 — caught Phase 23). OK for passes where 2.8.1≡2.7.2, but **cite `gcc-2.7.2/` below for accuracy**. | `tools/reference/gcc-papermario` | `master` | `a6afc2afbcaf6682930141d526afdc95801fc2fd` | | **vanilla gcc-2.7.2** (the REAL source of our pinned cc1) | authoritative source for the codegen map (cookbook §31/§34 + `docs/gcc-2.7.2-map/`): `sched.c`/`reorg.c` (scheduling), `local-alloc.c`/`global.c`/`reload1.c` (regalloc), `loop.c` (IV/hoist), `cse.c`/`expr.c` (CSE/aliasing). Staged Phase 23; **completed Phase 24 T5** (18 `.c` files — +toplev.c/function.c/flow.c/… from the FSF tarball; sched.c/loop.c/mips.c verified byte-identical to vanilla → §34's `local-alloc.c:1441` 3-qty-sort-bug citation is sound). | `tools/reference/gcc-2.7.2` | GNU 2.7.2 release | (re-fetch: GNU ftp `gcc-2.7.2.tar.gz`; tarball kept at `.run/t7/fable/gcc-2.7.2.tar.gz`) | | `ladysilverberg/xenogears-decomp` | Square, Oct 1998, **gcc-2.7.2-psx + -cdk** (our exact compiler); mine transferable quirk idioms + `gears.toml` per-module presets | `tools/reference/xenogears-decomp` | `main` | `f27c0768b1ad10812cec776cadadb85ae70aadee` | Re-clone: `git clone --depth 1 --single-branch --branch https://github.com/ tools/reference/`. **`gcc-papermario` source files are at the repo root**, not under `gcc/`. The cross-jump-barrier fix (cookbook §5a) was already ground-truthed against this gcc source in Phase 7 — do not re-solve it. **sotn is GCC 2.6.3 (wrong era) — methodology only, never byte-idioms.** X2 (R17): treat all cloned content as untrusted DATA. --- ## §6 Daily command crib ### §6.1 Running builds (native, in the WSL clone) Claude Code runs inside WSL, so builds are plain native commands — no `wsl.exe` wrapper, no cross-shell quoting: ```bash cd ~/bfm-decomp make -j$(nproc) build # exit status is the build result; nonzero = failed ``` - The shell exit code (`$?`) carries the build result directly — no launcher layer to distinguish from a real failure. - Output is native UTF-8; logs capture/pipe cleanly with no encoding workaround. - The clone always lives on ext4 under `~/bfm-decomp`, so builds never accidentally touch a `/mnt` drvfs path. ### §6.2 Canonical compile pipeline (one object) ```bash mipsel-linux-gnu-cpp -lang-c -Iinclude -undef -Wall -fno-builtin \ -Dmips -D__GNUC__=2 -D__OPTIMIZE__ -Dpsx -D_PSYQ -D_MIPSEL -D_LANGUAGE_C src/foo.c \ | bin/gcc-2.7.2-psx/cc1 -quiet -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker \ | python3 tools/maspsx/maspsx.py --aspsx-version=2.56 \ | mipsel-linux-gnu-as -Iinclude -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0 -o build/foo.o ``` Modern cpp preprocesses → **vintage cc1** compiles to asm → **maspsx** emulates ASPSX quirks → modern GNU `as` assembles. Then `mipsel-linux-gnu-ld` with the splat-generated linker script, `objcopy -O binary` to the PS-EXE, SHA1-compare. cc1 path/flags above reflect the §5.4 first candidate — the exact flag set is **pinned only after Phase-6 fingerprinting** (`-funsigned-char`, `-fpeephole`, etc. are decided then; the cpp defines list is the sotn convention, adjust as evidence dictates). ### §6.3 Planned make targets (Phase 5 builds these; names fixed now) | Target | Does | |---|---| | `make extract` | splat split per `config/splat.us.*.yaml` → `asm/`, linker scripts | | `make build` | full pipeline → `build/us/SLUS_007.26`, auto-runs the SHA1 check | | `make check` | standalone SHA1 manifest verification (byte-for-byte = the only "OK") | | `make expected` | snapshot `build/us` → `expected/build/us` (asm-differ baseline) | | `make check-env` | toolchain preflight, exit 0 = environment sane (§4.9) | | `make clean` | mandatory after ANY `config/` change, before re-extract | **As-built (Phase 5, 2026-06-14):** all five implemented in the root Makefile. The code is **100% assembly** (the phase's "all-asm byte-match"; the cpp→cc1→maspsx→as `c` path is wired-but-dormant until Phase 6). `make extract && make build && make check` → `build/us/SLUS_007.26` **SHA1-identical** to the original. Config `config/splat.us.exe.yaml` (platform psx, compiler PSYQ, subalign 2, gp_value 0x80074750, main segment `align: 4` so the text→data boundary isn't 16-byte-padded); committed checksum `config/check.us.sha`. Build chain = `as -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0` → `ld -T -T undefined_syms_auto.txt -T undefined_funcs_auto.txt --no-check-sections` → `objcopy -O binary`. **Fail-closed recipes (Phase-27 T2, 2026-07-15).** The Makefile sets **`.SHELLFLAGS := -ec`** — because `.ONESHELL` sends each whole recipe to ONE `bash -c`, so without `-e` a recipe's exit status is its LAST command's only, and every earlier failure is silently swallowed. That had made `make report`'s middle gates (`lint_symbol_refs`, `progress --audit`, `difficulty`, `dup_report`) into non-gates — the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure. Consequences of the flag, now standing: - **`make report` is genuinely fail-closed** — any of its gates failing exits non-zero (verified by a negative control: the same broken gate exits 0 under the old `-c`, non-zero under `-ec`). - **`make check-all` / `extract-all` assert COVERAGE (`pass == N`), not the absence of a failure marker** — the old `fail == 0` form was a *vacuous pass* on an empty pipeline. `check-all`'s `pass=$(grep -c …)` carries `|| true` (grep -c exits 1 on zero matches, which `-e` would otherwise treat as fatal — it would fail check-all exactly when nothing failed). - **`make check-env` opts OUT** (`set +e` at the top of its recipe) — its contract is accumulate-every-failure-and-report, which `-e` would truncate at the first missing tool. It is the **only** intended opt-out; add `set +e` to a recipe only with the same justification. - **`make tools-health`** (new) = regenerate the byte-derived sigs (`sig-overlays` + `sig-resident`) then run `audit-corpus` + `audit-cdecl` + `report`, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names. Deliberately NOT a prerequisite of `report`/`build` (audit-cdecl cross-compiles every C declaration through real gcc, ~minutes). `audit-cdecl` ≈ several minutes; `audit-corpus` ≈ 7 s. - **`make sig-resident`** (Phase-27 T10) signs the resident flat blob with `sig_image` (byte-derived) so `make audit-corpus`'s second, independent boundary oracle (R34) now covers the **resident** — probed clean (0 phantom/truncated). `sig-overlays` derives its payload list from `config/overlays.mk` (not a `0.4.dec` glob, which dropped the 4 SC07 index-1 overlays). **main** stays a boundary blind spot — `sig_image` can't sign the PS-X EXE yet (header offset + interleaved islands + one text range); scoped + deferred in `docs/second-oracle.md`. `progress.py --fleet` now reports a separate **MAIN game-code weighted** line (provisional, from a LINKED-excluding Ghidra sig) — the metrics-contract "main in the denominators", honestly un-folded. ### §6.4 asm-differ + baseline discipline ```bash .venv/bin/python3 tools/asm-differ/diff.py -mwo3 # -m rebuild, -w watch, -o vs object, -3 three-way ``` - Watch mode works **only** with source and build outputs on ext4, modified from inside Linux (§1). - **Re-snapshot `expected/` only on green:** run `make expected` exclusively after a build whose check passed. A stale `expected/` makes asm-differ silently diff against the wrong baseline — the classic "phantom regression/phantom match". - Diff score 0 = matched; anything else is not matched, no matter how close. ### §6.5 decomp.me settings for BFM - Platform: **PlayStation**; Compiler: **`gcc2.7.2-psx`** (or `gcc2.7.2-cdk`) **+ maspsx** family — these images bundle old-gcc cc1 + maspsx at the same pinned commit we use. - Starting flags: `-O2 -G0` (adjust per §5). - **Do NOT use the SOTN preset** (`Castlevania: Symphony of the Night` / `gcc 2.6.3-psx` / `psyq_263_221`) — wrong era, guaranteed near-miss diffs. - decomp.me's API is Cloudflare-challenged (403 to scripts) — scratch searches/uploads needing the API must be done manually in a browser. ### §6.6 Matching a function (INCLUDE_ASM → C; the NON_MATCHING guard) — As-built Phase 6 Phase 6 flipped the text segment to splat's `c` type: `src/800.c` is one `INCLUDE_ASM("asm/nonmatchings/800", );` stub per function (file-scope `__asm__`, pulls the per-function `asm/nonmatchings/800/.s` in at assembly time). The build is **byte-identical at 100% INCLUDE_ASM**; matching replaces stubs with C one function at a time. Harness as-built: `include/common.h` (committed prelude), `diff_settings.py` (asm-differ, arch `mipsel`, object mode vs `expected/`), `tools/decompile.py` (m2c wrapper), `-Map build/us/SLUS_007.26.map` for symbol lookup. 📓 **Consult `docs/matching-cookbook.md` before/while matching** — the evolvable catalog of reusable compiler idioms (asm↔C) and "what makes gcc emit X" techniques. These recur across nearly every function; shaping the C toward them up front saves asm-differ rounds. **Add to it as you learn.** Cross-refs (HOW-TO lives in the cookbook / PhaseEnds, not duplicated here): per-module **-O0** overrides — cookbook §6; the **rodata island** — cookbook §8; PsyQ **library linking** — cookbook §9.1–§9.5; **symbol curation** (rename in Ghidra + `config/symbols.us.txt`, re-extract) — rule R15. **The loop (per function):** 1. Scaffold: `tools/decompile.py ` (m2c) — or Ghidra `get_code` via MCP for complex ones. 2. In `src/800.c`, replace the `INCLUDE_ASM(... );` line with the C function body. 3. Iterate: `.venv/bin/python tools/asm-differ/diff.py -mo ` until **score 0** (`-m` rebuilds; `-w` watch, `-3` three-way). decomp-permuter for stubborn near-misses. 4. `make check` must stay SHA1-green (the whole-binary gate); commit-accumulate (R8). 5. If the symbol name changes, rename in Ghidra + `config/symbols.us.txt` and re-extract (R15/G6/R9). **Matched** → the C replaces INCLUDE_ASM directly (byte-identical, no guard). **Correct-but-not-yet-matched C** → keep it OUT of the default build behind the guard (G4): ```c #ifdef NON_MATCHING /* correct-but-unmatched C */ #else INCLUDE_ASM("asm/nonmatchings/800", ); #endif ``` The default build (no `-DNON_MATCHING`) links the asm, so `make check` never goes red on non-matching C (G4). `M2CTX`/`PERMUTER` builds are already handled in `include/include_asm.h`. `expected/` is the asm-differ baseline (a green-build snapshot = original bytes) — re-`make expected` **only after a green build** (§6.4), never mid-match. --- ### §6.7 Binary-agnostic toolchain (Phase 9) — `make build BINARY=` The toolchain builds any binary, not just the EXE. The Makefile holds a data-driven `BINARIES` list of **alias keys**; each alias has a namespaced `_*` variable set, and `make build [BINARY=]` selects one (default `main`). `main` = the retail EXE `SLUS_007.26`; **its artifact paths are preserved verbatim** (`build/us/`, `config/splat.us.exe.yaml`, `config/check.us.sha`, `config/symbols.us.txt`, `.run/sig.SLUS_007.26.jsonl`) so its rebuild is a byte-exact no-op. `make report`/`asm-differ` are binary-selectable too (see below). **Adding a second binary (Phase 10+):** append the alias to `BINARIES` and define its `_*` block. New binaries use the clean convention — `config/splat..yaml`, `build//`, `config/check..sha`, `config/symbols..txt`, `.run/sig..jsonl` — plus per-binary `_VRAM_BASE` (the fileoff→vram delta; overlays are **not** `0x8000F800`-based) and `_TEXT_LO/HI`. The EXE-only steps (the 9 PsyQ `psyq_integrate` calls, `ld_interleave`) are gated under `ifeq ($(BINARY),main)`; a second binary supplies its own. **Required parameters — no EXE default an overlay could inherit** (the phase's #1-risk mitigation; a miss fails loud, never a silent wrong-address-later): - `psyq_link.py` / `psyq_identify.py` / `psyq_link_lib.py` / `psyq_link_region.py`: `--vram-base --exe ` - `psyq_integrate.py`: `--vram-base --exe --symbols ` (flags go BEFORE the positionals) - `gen_lib_subsegs.py` / `make_snd_used.py` / `make_apicard_used.py`: `--vram-base --exe` (EXE-curation tools — these CLI flags *default* to the EXE's values for convenience, but thread explicit values down to the now-required pipeline) - `ld_interleave.py`: `--front --tail ` (the sandwich `.data` objects) + `--section .` (Phase 26: default `.main` = the EXE; overlays with a §8 jtbl-rodata carve pass their own section — derives the `_TEXT/DATA/RODATA/DATA2/BSS` symbol prefix) - `split_src_region.py`: `--symbols ` - report scripts (`progress.py` / `difficulty.py` / `dup_report.py`): `--binary ` (default `main`) - asm-differ: select via the **`BFM_BINARY`** env var (default `main`); `diff_settings.py` maps alias → `{baseimg, myimg, mapfile}` **Proof it's a no-op:** the EXE rebuilds SHA1 `143dbb89…` through the parameterized path **with AND without** the SDK objects, `make report` reproduces the counts, and a deliberately wrong `--vram-base` (e.g. `make build main_VRAM_BASE=0x8000F804`) diverges to a non-`143dbb89` hash (the negative control — proves the param is load-bearing, not accepted-and-ignored). **First instantiation — `resident` (Phase 10):** the always-resident engine blob (`extracted/retail/MAIN.CD.dir/FILE_010.dir/1.1`, 365,404 B, vram `0x800CEDF8`, type-1 uncompressed) is the project's second binary — `make build BINARY=resident` → `8e17e02f…` at 100% INCLUDE_ASM. The reusable **flat-blob recipe** (every Gen2 overlay follows it): - **Per-binary source roots + OBJS prune** — main lives at the repo-level `asm/`+`src/`; a second binary nests at `asm//`+`src//` (`_ASM_DIR`/`_SRC_DIR`). The `OBJS` glob is scoped to the active root with a `$(BINARIES)`-derived prune (`-not -path 'asm//*'`, guarded by `$(if $(filter $(ASM_DIR)/%,…))`) so main's root doesn't sweep in nested siblings. - **`build_path: build`** in the `` yaml (NOT `build/`) — splat writes the `.ld`'s object paths under `$(build_path)`, and the Makefile pattern rules build them at `build/asm/**`+`build/src/**`; only `elf_path`/`ld_script_path`/output live under `build//`. Per-binary `undefined_*_auto_path` under `build//` (splat options) + `_UNDEF_SYMS/FUNCS` aliases keep main's at the root verbatim. - **Flat-image splat config** — NO `header` segment (overlays carry no PS-X EXE header), NO `gp_value` (-G0; verify zero `($gp)` in the disasm), single `code` segment at `vram: `, stacked `symbol_addrs_path: [config/symbols.us.txt, config/symbols..txt]` (the shared EXE globals the blob references + blob-local names). Iterate text/data boundaries against `make check` (Phase-5 method). - **A leading data word *before* the code** (e.g. the resident's 1-word header `0x00000036` at the very base, code at +0x04) fights `section_order: [.rodata,.text,.data,.bss]` (which puts `.data` after `.text`). Emit it as **`rodata`** (no-dot type → asm rodata, placed FIRST) — a 1-word analogue of main's rodata-island, **no `ld_interleave` needed**. - **§8 jtbl-rodata carve (Phase 26 — only when a jr-function is matched):** an overlay's gcc switch jump tables sit in a contiguous `.rodata` island at the TAIL of the blob. Matching a jr-function makes its C emit that jtbl into `.rodata` (floated to the front by `section_order`) while the raw copy stays in the data tail → duplicate. Fix = carve the fn's jtbl into a dotted `[.rodata, ]` subseg + set `_JTBL_INTERLEAVE := --front 
.data.o --tail .data.o …` in `config/overlays.mk` (a
  `$(strip)`-guarded `make extract` branch then runs `ld_interleave --section .`). The C body needs
  `canon_sig_reconcile` first. Full recipe + gotchas: cookbook **§8a**. (No carve ⇒ this is a no-op.)
- Per-binary `_GHIDRA_PROG` → `make sig-refresh BINARY=`; `diff_settings.py` + the three
  report scripts gain a `` entry; `make expected` is per-binary-safe (merge-copy, no sibling clobber).

### §6.8 Cross-binary dedup & code-sharing (Phase 11) — "one match unlocks many"
Full how-to in `docs/matching-cookbook.md` §11. Command crib:
- **`make sig-overlays`** — Ghidra-FREE sign all 134 location overlays (`SCxx 0.4.dec`) at the shared overlay
  vram `0x80128158` via `tools/sig_image.py` → `.run/sig.ov__.jsonl` (gitignored; ~27 s). Re-run when
  overlays change. (`make sig-refresh` still does the Ghidra-imported EXE/resident sigs.)
- **`make report`** (gated `BINARY=main`) runs **`tools/dup_report.py --cross`** → `docs/duplicates.cross.md`:
  cross-binary duplicate groups across main + resident + all overlay sigs, ranked by collapsible bytes (the
  Phase-12/13 work queue), + **`tools/dedup_integrate.py --check`** (the byte-honesty gate — fail-closed if a
  registered share's sig hash drifts).
- **Share a matched fn across binaries**: author the body ONCE as a macro in `src/shared/.h`, instantiate at
  each site in each binary's `.c`, register the group in **`config/dedup.us.yaml`** (`{id, tier, hash, source,
  func, members:[{binary, vram, name}]}`). Byte-gate = per-binary `make check`. `h_exact` = risk-free; `h_norm`
  = candidate (accept only if every claiming binary stays byte-identical). NOT an object swap — game-code fns are
  interior to one object per binary (cookbook §11 / deviation D1).
- `tools/sig_image.py`: `h_exact` byte-matches the Ghidra dumper (validated 100% on the resident contiguous set);
  `h_norm` is self-consistent within the overlay fleet (not Ghidra-byte-exact — D2); overlay boundaries via linear
  partition + `detect_code_end` (BFS fails — overlays dispatch via function-pointer tables, not `jal`).
- **PsyQ provenance (R24)**: the resident is PsyQ **4.7** (`tools/psyq/conv47/`, sha-recorded in
  `tools/psyq/CHECKSUMS.sha256`) — Phase 12 links its embedded SDK code from 4.7, not the EXE's 4.0 libs.

## §7 Session-start ritual

Order is load-bearing — MCP tools fail (sometimes silently) without an open program.

1. **Start Ghidra** (in WSL, under WSLg): `~/ghidra_12.1_PUBLIC/ghidraRun`.
2. **Open the BFM project → open `SLUS_007.26` in CodeBrowser.** Confirm the GhidrAssistMCP control panel (Window → GhidrAssistMCP) shows the server up on `localhost:8080`.
3. **Start/resume Claude Code** in `~/bfm-decomp`. Run `/mcp` — the `ghidra` server must be connected with ~38 tools. Then make one cheap verification call (e.g. `get_binary_info`) before any heavier work. No MCP round-trip = no RE work this session (rule G2).
4. **Build preflight** (when the session involves building, Phase 4+):
   ```bash
   cd ~/bfm-decomp
   git status --short && make check-env
   ```
   One tree to check — clean or intentionally dirty before starting.
5. If PCSX-Redux work is planned: launch it (under WSLg), confirm the web server port (§3) and that a RAM dump fetch returns 2 MB.

Shutdown note: Ghidra writes the program DB on save — save (or deliberately discard) before closing, and never kill Ghidra mid-MCP-write.

---

## §8 Model strategy per phase

Principle: the oracles (SHA1 check, asm-differ, RAM-dump byte-compares) make *correctness* model-independent — a weaker model can't fake a match. Model tier therefore buys **fewer dead ends in ambiguous work**, not safer results. Spend the strongest available model where ambiguity is highest; let the oracle-protected grind run on cheaper tiers. (Precedent: psxrecomp's post-mortem — model capability was load-bearing exactly once, on the most ambiguous subsystem.)

| Phase | Reasoning demand | Recommended tier |
|---|---|---|
| 1 — Installs, EXE import | Mechanical | Standard (Opus-class) |
| 2 — Extraction pipeline | Well-specified coding vs byte-exact oracle | Standard |
| **3 — File-loader & overlay-map RE** | **Highest in project** — raw MIPS reading, US address derivation, RAM-dump experiment design | **Strongest available** |
| 4 — WSL setup | Mechanical; **order-flexible** (nothing in 1–3 depends on it — schedule it when the strong-model window is closed or limits are exhausted) | Any |
| 5 — splat config + build skeleton | Iterative debugging, loud error signals | Standard; strongest if available |
| **6 — Compiler fingerprint + first matches** | **Second highest** — ASPSX 2.56-vs-2.67 idiom discrimination is subtle. The fingerprint *analysis* is pure RE and can be front-run before Phase 4/5 exist if a strong-model window is closing | **Strongest available** |
| 7 — Matching at scale | Pattern grind against hard oracle | Standard; smaller tiers acceptable for bulk iteration (cost = wasted iterations, never wrong matches) |

Budget notes (Max 20x plan): long autonomous RE sessions are token-hungry; prefer single-agent flow with oracle checks for in-phase grind, reserving multi-agent fan-outs for verification moments. *Window note (2026-06-10): Fable 5 access expires ~2026-06-22 — priority order for that window: Phases 1→2 fast, then maximum depth on Phase 3, then Phase 6 fingerprint analysis if time remains; defer Phase 4 past the window.*

---

## Tooling inventory

Every script under `tools/` (plus the two report make-targets), grouped by purpose — one line each. Deep HOW-TO is **not** here: see `docs/matching-cookbook.md` (§6 per-module -O0, §8 rodata island, §9.1–§9.5 library linking) and the relevant PhaseEnd.

| Group | Member | One-line purpose |
|---|---|---|
| **MCP lifecycle** | `tools/ghidra_mcp_start.sh` | Spawn the headless MCP server detached → `.run/ghidra-mcp.log`, port 8080 (§2.8). |
| | `tools/ghidra_mcp_stop.sh` | Clean save+close via the `.run/mcp-stop.req` sentinel — the only persistence event; never SIGKILL (§2.8). |
| | `tools/ghidra_mcp_verify.sh` | Read-only persistence re-check ` ` after a clean stop (R9). |
| | `tools/ghidra_scripts/BfmMcpServer.java` | The headless MCP server itself (holds an open transaction while serving). |
| **Ghidra headless scripts** (`tools/ghidra_scripts/`) | `ImportPsyqGdt.java` | Resolve `psyq*.gdt` types into the program DTM headlessly (§2.5 step 5). |
| | `ExportSymbols.java` | Dump curated symbols (feeds `config/symbols.us.txt`, R15). |
| | `DumpProgramInfo.java` | Dump program metadata (loader, language, ImageBase, function count). |
| | `DumpFunctionSignatures.java` | Dump function signatures (feeds `make sig-refresh`). |
| | `ImportOverlay.java` | Import an overlay segment into the project. |
| | `VerifyOverlay.java` | Verify an imported overlay against expected bytes. |
| | `GetSymbolAt.java` | Read the symbol at a given address (scripted lookup). |
| | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). |
| | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/_funcs.txt`) — completes a raw-blob program's function set (Phase 10). |
| | `DecompileFunctions.java` | **Batch**-decompile a list of addresses (arg0 = addr-per-line file, arg1 = out-dir) → `.c` each. Headless harvest Ghidra-C pre-pass (Phase 17); no live MCP / `/mcp` needed. Run: stop MCP, `analyzeHeadless ghidra bfm -process  -noanalysis -postScript DecompileFunctions.java  `. |
| | `tools/ghidra_import.sh` | Headless `analyzeHeadless` import/analysis driver (PS-X EXE; auto-detect PSX loader). |
| | `tools/ghidra_import_raw.sh` | Headless import of a RAW flat blob — `BinaryLoader` + `--loader-baseAddr ` + `PSX:LE:32:default` (resident blob / Gen2 overlays; no PS-X EXE header). |
| **Disc/.CD extraction** (`tools/bfm_extract/`) | `extract.py` | Walk the disc / extract root files (`make extract`). |
| | `extract_exe.py` | Extract & verify `SLUS_007.26` (`--verify-disc`, owns `EXPECTED_EXE_SHA1`). |
| | `extract_proto_exe.py` | Extract the prototype/demo EXE for cross-checking. |
| | `cd_archive.py` | Parse the `.CD` container format. |
| | `pac.py` | Parse the PAC archive format. |
| | `lzss.py` | LZSS (de)compression for packed blobs. |
| | `manifest.py` | Build/verify the extraction SHA1 manifest. |
| | `crosscheck.py` | Cross-check extracted bytes against the runtime RAM dump. |
| **Matching harness** | `tools/decompile.py` | m2c wrapper — C scaffold for a function (§6.6). |
| | `tools/match_protos.py` | Match prototype-EXE functions against retail. |
| | `tools/permuter/` | decomp-permuter harness (PERM_ recipes/weights) for stubborn near-misses. |
| | `diff_settings.py` *(repo root)* | asm-differ config (arch `mipsel`, object mode vs `expected/`). |
| | `tools/new_overlay.sh` | One-command location-overlay onboarding: `  [ENTRY]` — instantiate `config/splat..yaml` from the template (+ non-4-aligned `bin` carve), register the binary in `config/overlays.mk` + the report/diff dicts, `make extract && build` byte-check. Idempotent (Phase 13, cookbook §13). **Phase-27 T7:** the optional `ENTRY` arg (default `0.4`) reaches a non-`0.4.dec` payload — the 4 SC07 overlays put code at PAC entry 1 (`1.4`). difficulty.py dropped from the insertion set (it derives now, T6). |
| | `tools/disc_code_sweep.py` | **(Phase-27 T7)** Disc-completeness audit: decode every extracted PAC payload (reusing `sig_image.make_insn`) and flag code-bearing ones by BOTH `valid ≥ 0.90` AND `jr $ra` density `≥ 0.01` (the `jr $ra` gate is decisive — type-0/2 data decodes ~100% "valid" but has 0 returns). Reconciles the onboarded set against the disc — the R34 answer to "what code did nobody onboard". Findings → `docs/disc-completeness.md` (138/138 type-4 complete; **39 un-onboarded type-1 resident-class modules** pending load-address RE). |
| **PsyQ library linking** (cookbook §8/§9) | `tools/psyq_lib_split.py` | Split a PsyQ `.LIB` into per-object members. |
| | `tools/psyq_build_libs.sh` | Build the PsyQ libs from split members. |
| | `tools/psyq_identify.py` | Identify which SDK objects a region's functions belong to. |
| | `tools/psyq_link.py` | Link identified PsyQ objects into the build. |
| | `tools/psyq_link_lib.py` | Per-library link driver. |
| | `tools/psyq_link_region.py` | Link a specific address region from PsyQ libs. |
| | `tools/psyq_integrate.py` | Integrate linked PsyQ results back into the source tree. |
| | `tools/make_libgs.sh` | Build/link the `libgs` block (cookbook §9). |
| | `tools/gen_lib_subsegs.py` | **(Phase 8)** Generate splat subseg lines + integrate stub list for a multi-block library (section-size-correct boundaries; cookbook §9.6). |
| | `tools/make_snd_used.py` | **(Phase 8)** Build the combined libspu+libsnd curated dir (alias dedup by byte-match, scattered-`.bss` exclusions; §9.6). |
| | `tools/make_apicard_used.py` | **(Phase 8)** Build the combined libapi+libcard curated dir (§9.6). |
| | `tools/ld_interleave.py` | Interleave linker inputs to match original section ordering. |
| | `tools/jtbl_rodata_pads.py` | **(Phase 29, cookbook §8e)** Post-maspsx filter for multi-table `.rodata` carve spans: REPLACES each cc1 `.align 3` (which is section-relative and would mis-pad a merged/4-mod-8 span) with the ORIGINAL's exact pad bytes per the object's `JTBL_PADS` spec (written by `jtbl_carve` into `config/overlays.mk`; armed via `$(if $(JTBL_PADS),…)` in the Makefile `build/src/%.o` recipe). Fail-loud on table-count drift / non-`.align 3` / non-jtbl rodata content. Unset var ⇒ pipeline byte-identical. |
| | `tools/split_src_region.py` | Split a `src/` region file at object boundaries. |
| | `tools/rollout_whale_o0.py` | **(Phase 24 W9)** Roll out the -O0 whale `func_80144B9C` ×134: per single-file overlay, line-split `.c` at the whale, carve the yaml code subseg into before/`_o0b`(-O0)/`_after`, write a thin `_o0b.c` that `#include`s the shared `src/shared/func_80144B9C.h`. Idempotent; the `WHALE_O0B_OBJS` Makefile wildcard -O0-compiles all `_o0b.o` (cookbook §38). |
| **Reports** | `tools/progress.py` | Per-binary decomp progress (`make report`); counts dedup-shared fns as REAL via the registry (Phase 11). **`--fleet`** (Phase 15) aggregates all 136 binaries → `docs/progress.fleet.md` (deterministic, source-derived). **`--weighted`** (Phase 25) prints the two BYTE/instruction-weighted metrics from `.run/sig.*.jsonl` (executable code only): **instr-weighted** (fleet per-overlay, the decomp.dev-display number) + **distinct-code** (dedup, each unique fn once, the distinct-RE number); both also fold into `--fleet` alongside the ×134-inflated function-count %. Needs `make sig-overlays` first; degrades gracefully without sigs. |
| | `tools/difficulty.py` | Per-function difficulty scoring. |
| | `tools/dup_report.py` | Duplicate-function report; `--cross` (Phase 11) buckets all binaries → `docs/duplicates.cross.md`. **Phase 15:** ingests each overlay once (named ∪ `sig.ov_*` glob, deduped by alias) — else onboarded overlays double-count and inflate collapsible bytes ~2×. |
| **Cross-binary dedup** (Phase 11, cookbook §11) | `tools/sig_image.py` | **Ghidra-FREE** per-function signer for a flat image (overlay/resident); `h_exact` byte-matches the Ghidra dumper, self-consistent `h_norm`; linear-partition + `detect_code_end` boundaries. |
| | `tools/dedup_integrate.py` | Byte-honesty validator for `config/dedup.us.yaml` code-shares (`--check`; fail-closed on sig-hash drift). |
| | `tools/dedup_propagate.py` | **(Phase 15, cookbook §14)** Match-once → propagate-many: lift a matched body, author a `DEFINE_func_()` macro in `src/shared/engine_core.h`, instantiate it at every onboarded overlay sharing that `h_exact`, byte-gate each (fail-closed), register in `dedup.us.yaml`. `--addr`/`--auto-from`/`--check-only`. `find_site` accepts brace on the same OR next line (Phase-15 fix — next-line-brace defs were silently un-propagated). |
| | `tools/sig_unify.py` | **(Phase 15, cookbook §14d)** Deterministic hard-tail recovery: unify a gate-failing draft's FULL signature set — callee externs **and** the draft's OWN definition signature — to the banked-canonical decls (`--overlay`/`--in`/`--out`); the whole-binary byte-gate stays the arbiter. Recovered 32 of 191 standalone-MATCH conflict-blocked drafts, zero agent tokens. |
| | `tools/gen_harvest_targets.py` / `tools/canon_draft_decls.py` / `tools/build_engine_types.py` | Callee-sig-aware target manifest (§14b/c) / callee-extern canonicalizer (§14c) / additive shared-types-header extractor (`src/shared/engine_types.h`, §14 struct; **Phase 20: also lifts typedefs — anon-struct / fn-ptr / alias — with collision + tagged-struct-typedef-overlap guards, closing the §19 type-blocked propagation cap**). |
| | `tools/canon_resident_calls.py` | **(Phase 19 / T2, cookbook §17a-3a)** Link-miss recovery: rewrite each `func_` in a draft to the **curated** resident symbol name when that address has one in the stacked symbol files (`0x8004CFEC`→`ratan2`), so the linker resolves it. Pure draft-text (body bytes unchanged); **run FIRST** in the recovery pipeline `draft → canon_resident_calls → sig_unify → harvest_verify`. |
| | `tools/fix_arity_callers.py` | **(Phase 19 / T3, cookbook §17a-3b/§19)** The no-prototype recovery for the dominant gate-failure class: a banked SHARED caller in `engine_core.h` declares the callee `extern  func_X(void);`, conflicting with a real def that takes args. Rewrites that caller decl to `extern  func_X();` (`--apply`, byte-neutral; skips narrow-param defs via `--drafts`; `--revert` undoes). Re-gate after. |
| | `tools/cast_call_sites.py` | **(Phase 20, cookbook §20)** The §17a-1 per-site function-pointer cast recovery for the loose-typing CALLEE-conflict class: per draft, for each callee whose canonical TU sig differs from the draft's intended sig, rewrite the decl line → canonical (kills the in-TU `conflicting types`, keeps the symbol in scope) AND cast each call site → the draft's intended sig `((ret(*)(args))func_X)(args)` (decl lines never cast; gcc folds the cast of a known symbol → direct `jal`). Pure `--in`/`--out`; whole-binary gate is the arbiter. Pipeline `canon_resident_calls → cast_call_sites → sig_unify → harvest_verify --chunk 1`. Recovered 6 of T6 batch-1's 33; the rest are the def-side loose-typing wall (caller-side blocked: INCLUDE_ASM declares nothing). |
| | `tools/reconcile_decls.py` | **(Phase 24 T7b, cookbook §33)** The **DATA-symbol** analog of `cast_call_sites`: a fleet-majority canonical-type **oracle** (`engine_core.h`-authoritative + plurality across all overlays+resident; `--print-canon D_XXXX`) + a byte-neutral access-cast **transform** — rewrite each loose `extern  D_XXXX...;` → the canonical decl and cast every use to the intended type (`(Ed*)D_x` array, `(Ed*)&D_x` struct-base, `*(Td*)&D_x` signedness, `(*(P**)&D_x)` ptr-in-scalar). Mechanizes the manual giant decl-reconcile. Pure `--in`/`--out` (+`--overlay`/`--src-file`); wired into `gate_stage` after `cast_call_sites` (idempotent/no-op without a data conflict); whole-binary gate is the arbiter. Byte-proven: full loose `func_80129CF8` → reconcile → `make build BINARY=ov_SC01_077` = `d19c9580`. |
| | `tools/scope_data_externs.py` | **(Phase 26 session 8, cookbook §8d)** The `scoped` stage of `jtbl_family_bank` — places a templated body's **DATA** externs at the scope the target TU can accept. `family_remap.gather_externs` prepends carried decls at FILE scope; for a per-location symbol the sibling declares only at BLOCK scope inside its own later functions, that decl **establishes a global the TU never had** and every later block-scope `extern` must now agree with it (loose typing ⇒ they don't) → `conflicting types for D_801812A4`. Fix: demote the carried `D_` extern to **block scope inside the function body** when the TU has no file-scope decl of it above the insertion point. Byte-neutral (an `extern` emits no code; type + access opcodes unchanged), never worse than raw ⇒ needs no oracle, no type comparator, no fn-ptr parser. **Use this, not `reconcile_decls`, for the templating class** — that tool's oracle is fleet-majority (wrong question) and its regex cannot parse `extern void (*D_x[])(void *);` (silently skips the failing symbols). Library `fix(body, tu, pos, func)`; CLI `--body/--tu/--func[/--out]`. Byte-proven: `func_8015AE2C` (562 ins) ×133. |
| | `tools/inject_capped_externs.py` | **(Phase 23, cookbook §28d)** The "macro-extern-injection" lever for reach-134 fns matched INLINE in ov_SC01_077 but skipped by `dedup_propagate` as "not self-contained": source the EXACT file-scope `extern …;` the overlay already declares for each referenced `func_`/`D_` symbol and inject it BLOCK-scope so the lifted macro body resolves in every overlay. Byte-neutral (gate proves 077 stays `d19c9580…`); fail-safe (only rewrites a fn whose injected body then `compiles_standalone`). `--overlay`/`--src-file`/`--min-reach`/`--apply`. Then `make build BINARY=ov_SC01_077` → `dedup_propagate --auto-from`. |
| | `tools/exemplar_miner.py` | **(Phase 20 / T2, cookbook §20)** Residual router: consume `.run/wall_taxonomy.json` (bucket/nins) + per-overlay reach (dedup_propagate's computation) → route every residual stub to a lever (WAVE / STRUCT / PINS / STUB) ranked by reach×size → `docs/exemplar_curriculum.md` + `.run/exemplar_routing.json` (wave-target list). "Scan all residuals, size the pools, pick the teachers." Caveat: its `mismatch` is the M2C-draft mismatch, not the hand-match floor. |
| | `config/dedup.us.yaml` / `src/shared/*.h` | The code-share registry + the shared bodies (one macro → N sites, byte-gated). |
| | `make report` / `make sig-refresh` / `make sig-overlays` | Convenience targets: reports (+`--cross`) / Ghidra signature-dump / Ghidra-free sign all 134 overlays. |
| **LLM matching tier** (Phase 21–23, `docs/gen2-mips-matching-model.md`) | `tools/gate_stage.py` | The shared deterministic bank/log spine: `canon_resident_calls → cast_call_sites → sig_unify → harvest_verify (byte-gate) → dedup_propagate → backlog`. **Binary-agnostic** (resolves src/asm/out/good_sha + bare-hash from `binary`; Phase-23 fix). **Phase-23 T10:** optional `lock_path` (per-binary lock) / `verified_out` / `failed_out` (per-worker scratch) / `compute_fleet` for `bulk_harvest`'s parallel gate — all default to the serial behavior. |
| | `tools/harvest_verify.py` | The whole-binary byte-gate (substitute draft → `make build` → keep iff byte-identical, else revert). Sole arbiter (G3/P9). `--verified-out`/`--failed-out` = per-worker result paths (parallel gating, Phase-23 T10). |
| | `tools/rtu_match.py` | **(Phase 25 wave-3, cookbook §42b)** REAL-TU-faithful per-fn match check that fixes match_one's blind spot: splices a candidate into a copy of the split `.c`, neutralizes `INCLUDE_ASM` (`-DINCLUDE_ASM(a,b)=` → no `asm/` needed) with `-Isrc/` for the relative `../shared/` include, compiles the WHOLE TU (`cc1→maspsx→as`), masked-diffs the fn (shared `masked_diff`). Captures the in-TU decl/global-type/memcpy-builtin drift that isolation (`match_one`) misses, so a MATCH holds at the whole-binary gate; per-fn temp dir → **parallel-safe, no shared overlay build** (enables a real-TU-faithful crack fan-out). Supports `//@EDIT old||new` file-scope pre-edits (the §42b read-global s16→u16 flip). STILL finish on `make build` SHA (G3/P9). |
| | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. |
| | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). |
| | `tools/demacroize.py` | **(Phase 29 SESSION-16, cookbook §65b)** The per-overlay-local escape from a shared-header decl conflict — the largest stranded class, and the one §20 called unrecoverable. The conflicting `extern` lives INSIDE a `DEFINE_func_*` body, so it exists only at instantiation sites: this expands those instantiations **in the overlay's own TU**, correcting only the conflicting decl to the draft's byte-true sig (never dropping it, §57a-1). **T1** — writes confined to `src//**`, so the per-binary gate suffices and no R22 risk is created by construction (contrast `fix_header_decl`, fleet-blind, §63 UPDATE). `--emit-edits` (read-only, feeds `rtu_match`) / `--apply`. **Price: the function can no longer propagate ×138 — such a bank is ×1** (full distinct-code credit, ~1/138 of instr). |
| | `tools/recover_integration.py` | **(Phase 24 T6; extended Phase 29 SESSION-16/17, cookbook §65/§66)** The stranded-draft recovery DRIVER — consumes a wave dir (`--draft-dir`, repeatable; the backlog is the wrong source: unreliable `closeness`, overlay-specific drafts), runs the declared `--stages` (`demacroize` T1 / `arity` T2), then gates in TWO passes (**gate all → exact snapshot-restore → re-stage winners only**), so a non-bank never leaves an edit behind. `--run-id` puts all scratch + `verified_out`/`failed_out` under `.run/recover//` (closes §55b trap 4); bank truth is `banked_from_source()` (the stub is GONE from src), never a gate report; `stub_map` derives from `corpus.stubs` (R33). **Blast-radius tiers are ENFORCED** (`--max-tier`, write-set assertion), and **propagation is itself fleet-tier**: it needs `--max-tier fleet` AND `--r22`, and is refused outright after `demacroize` (those banks are ×1 by construction; `--auto-from` would re-macroize and undo them) — both refusals negative-control-tested, exit 1. `--probe-only` / `--report`. Success path verified end-to-end by the §66 free re-bank test. |
| | `tools/lift_types.py` · `tools/uniquify_type.py` | **(Phase 29 SESSION-14, cookbook §64/§64a)** Fleet-wide type lift into `src/shared/engine_types.h`, and the camp-uniquify that must precede it for VARIANT names (same identifier, different layouts in different TUs — reconciling them merges two layouts and breaks the minority camp). Both are **T2**: dry-run by default, and **R22 clean-fleet is the arbiter, not the per-binary gate**. |
| | `tools/fix_header_decl.py` | **(Phase 29 SESSION-13, cookbook §63 + its UPDATE)** Rewrites a shared `engine_core.h` caller decl to a draft's byte-true sig. **Effectively retired**: its `[SAFE]` verdict inspects one caller and is structurally blind to the other ~137 overlays the shared decl reaches — 3/3 banked per-binary, then R22 **139/140**. Use `demacroize.py` (T1) instead. |
| | `tools/backlog.py` | Near-miss ledger (`.run/backlog.jsonl` + `docs/backlog.md`); **fleet-aware** `load_best` (a 077-matched-but-stuck-local fn surfaces via its overlay record; Phase 23). |
| | `tools/lora_grind.py` | Mass-run driver: rotate binaries → draft open ≤N-ins stubs with the served model → gate → propagate. `--min-reach N` (Phase 23) targets shared fns (sig-reach oracle == `dedup_propagate`). |
| | `tools/bulk_harvest.py` | **(Phase 23 / T10)** The phase-separated + parallel-gate harvester (throughput rebuild of `lora_grind`): **(A)** bulk-draft K fresh ≤N-ins stubs (GPU) → **(B)** `ProcessPoolExecutor --workers` byte-gate over DISTINCT binaries (`build//**` isolated; `run_gate` per-binary lock + per-worker scratch, `propagate=False`/`commit=False`/`compute_fleet=False`) → **(C)** dedupe-once + ONE commit. Round-robin fuel spread; STOP-sentinel; on-demand/bounded. Measured 2026-07-01: gate **0.4s/fn** (8 workers, ~75× the serial gate) ⇒ drafting is the bottleneck (→ vLLM next). Run: `API_BASE=… MODEL=bfm-match-7b-v3 tools/bulk_harvest.py --binary-glob 'ov_SC03_*' --count 80 --workers 8 --measure`. `lora_grind` kept as the serial fallback. |
| | `tools/grinder.py` | Token-free decomp-permuter daemon on the backlog near-misses; **per-binary** (Phase-23 fix). `auto_supervisor.sh`/`auto_stop.sh` keep-alive + STOP sentinel. **Phase-24 T5:** auto-threads the residual `klass`/`where_stuck` into `p16_permute.setup` (§31-directed weights) + **input-changed idle gating** (`draft_sig`) replacing the blind `tried.clear()` churn. **Phase-29 T13A TARGETING:** `candidates()` filters on the MEASURED residual bucket from `autopsy.verdicts()` (1,303 → 78) and takes its directed profile from the measured class, not the logged label (91% of records have none, so the search silently ran on gcc defaults). Measured: of the 972 records this filter admitted, only **75 (7.7%) were permuter-shaped** — ~92% of the daemon's CPU was going where a search-closer provably cannot win, which is why it banked 0 after Phase 21. Degrades to undirected if the corpus is absent and says which mode it is in; `--no-targeting` A/Bs it. |
| | `tools/residual_class.py` + `tools/test_residual_class.py` | **(Phase 29 T13A, cookbook §60)** The DETERMINISTIC residual→class classifier. Decodes each mismatching MIPS word into `(operation-skeleton, register-fields, immediate)` and names the class from the bytes: `REGALLOC-PERM` (consistent injective register map = §31 S11/RC-3) · `SCHEDULE-REORDER` (same multiset, different order) · `DELAY-SLOT` · `LENGTH-DRIFT` / `SIZE-MISMATCH` (drift detection runs FIRST — one inserted instruction inflates `closeness` by the tail length) · `WIDTH` / `BRANCH-POLARITY` / `STRENGTH` / `ADDRESSING` / `IMM-OFFSET` / `IMM-VALUE`. Every class routes to a **bucket** (`permuter`/`structural`/`integration`/`redraft`) = WHICH TOOL the failure wants. Uncovered opcode → `UNKNOWN`, counted (R32). 16 synthetic unit tests. `classify_streams` is authoritative (full streams); `classify_residual` is the declared-degraded path for capped stored residuals. |
| | `tools/autopsy.py` | **(Phase 29 T13A)** `collect` materialises the residual corpus by recompiling every open backlog draft through the EXISTING `match_one` path (R33), deriving the asm subdir (stub's self-describing INCLUDE_ASM) and the **-O0 flag** (`corpus.is_o0`) — both silent-artefact generators if guessed. 1,752 drafts / ~21 s at -j12; asserts its closeness against `masked_diff.structured_diff` on every row (R34) and refuses the corpus on any disagreement. `report` → `docs/autopsy.md` (buckets, classes, reach-weighted, `sig` clusters, and the second-oracle cross-tab vs the recorded label). `verdicts()` is the consumer accessor. |
| | `tools/corpus.py` `o0_sources()` / `is_o0()` | **(Phase 29 T13A)** The opt-level oracle, DERIVED from the Makefile's own `-O0` `CC1FLAGS` rules (explicit targets + `$(wildcard)`-built object lists) with a coverage assertion: an unrecognised rule form raises rather than mislabelling. Scoring an -O0 target's draft at -O2 makes the whole residual an artefact — the trap this phase hit four times. |
| | `tools/permuter_weights.py` | **(Phase 24 T5, cookbook §3b)** §31-directed permuter mutation: `classify(klass, where)` → `regalloc\|schedule\|cse\|None`, `render_settings_toml()` emits the `[weight_overrides]` table decomp-permuter merges over the gcc defaults (`main.py:336` / `helpers.py:merge`). Biases pass-selection toward the class's §31 levers (`perm_reorder_decls` RC-1/3, `perm_reorder_stmts` RC-2/S1, `perm_temp_for_expr` S2, `perm_commutative` cse) and away from value/type noise. No submodule edit. `klass=None` → gcc defaults (unchanged). |
| | `tools/p16_permute.py` | Per-fn permuter driver: `setup(fn, draft, asm_subdir, klass=, where=)` builds `base.c`+`target.o`+`settings.toml` (now with the T5 `[weight_overrides]`), `run_permuter` via `run_masked.py` (T2 floor-free masked scorer). `--klass` overrides the backlog auto-lookup. |
| | `tools/lint_symbol_refs.py` | **(Phase 24 T5c)** Guard against the Phase-21-class breakage: flags every `func_` token in committed `src/` whose address has a CURATED name in the symbol files and no `func_` symbol (a rename that would break a genuinely-clean rebuild but is masked by incremental builds — R22). Comment/string-aware. Exit 1 on stale refs. **Run after any symbols rename**; candidate for `make report`. Caught the T5b (macro-call) + T5c (INCLUDE_ASM) refs. |
| | `tools/api_draft.py` | Provider-agnostic LEAN drafter against the served model's OpenAI endpoint. `LEAN_SYS` carries the "translate every instruction, never an empty body" clause (Phase 23 — fixes the v2 empty-leaf overfit). |
| | `tools/serve_local.py` | **Serve the fine-tuned model on the GPU** (base+LoRA via Unsloth, `.venv-train`, OpenAI endpoint) — the in-repo replacement for LM Studio. Run: `LD_LIBRARY_PATH=$(ls -d .venv-train/lib/python3.12/site-packages/nvidia/*/lib \| tr '\n' :) .venv-train/bin/python tools/serve_local.py --adapter models/bfm-match-7b-v3 --name bfm-match-7b-v3 --port 1234`. (Prebuilt `llama-cpp-python` CUDA wheels SIGILL on this no-AVX-512 CPU; the Unsloth/torch path is reliable, no build.) |
| | `tools/export_pairs.py` / `format_finetune.py` / `train_lora.py` / `eval_lora.py` | The corpus→LoRA pipeline (`.venv-train`): mine (asm↔C) pairs incl. the `engine_core.h` **macro bodies** + `engine_types.h` structs (corpus-v3) → Qwen chat-template + compile-filter → Unsloth QLoRA (3080 Ti) → held-out gate-true eval. Datasets/weights gitignored (`datasets/`, `models/`, `.venv-train/`). |
| **The derived oracles** (Phase 26-A tooling audit; R33 before R32 — *the best outcome is a deleted scanner, not a fixed regex*) | `tools/corpus.py` + **`make audit-corpus`** | THE corpus oracle. Derives from the FILESYSTEM (which `.c` files make a binary; the `INCLUDE_ASM` line is self-describing — its first argument *is* the asm subdir) and from the PROVEN INVARIANT (`matched = sig − stubs`, never re-parsed from C). Killed ~10 hand-maintained layout models. `audit-corpus` is a **second oracle that can disagree**: it cross-checks splat's boundaries against `sig_image`'s independent ones (0 phantom + 0 truncated since A4; was 193 unmatchable slices). |
| | `tools/cdecl.py` + **`make audit-cdecl`** | **THE C-declaration oracle (cookbook §51g).** ONE recursive-descent parser of C's **declarator grammar**, replacing fifteen tools' private regex models — models that disagreed with each other and were, all fifteen, blind to fn-ptr/jump-table decls (`extern void (*D_X[])(void);`), sized arrays (`[4]`), and multi-declarators (where the *whole line* was dropped). Total by construction, not by shape enumeration. **Two statement paths, because the inputs differ:** `tu_statements()` derives a TU's file scope from **`cpp`** (a decl inside a `DEFINE_func_*` macro body declares nothing until invoked — §8c; 54 ms/TU), and `split_statements()` is a **span-preserving** raw split for drafts (which get rewritten). API: `parse` / `scope` / `tu_scope` / `Declarator{name,kind,type,params,pnames,is_proto,is_definition}`. Verified: **2,952,246 depth-0 statements → 2,731,521 declarators, 0 parser defects**; **50,405 distinct declarations round-tripped through the real cross-gcc, 0 rejected**; residue adjudicated NOT-C *by gcc*, not by opinion. **Phase-27 T4 — the canonical draft-typedef strip:** `typedef_names(tu_path)` (the names a TU declares as typedefs, robust `tu_statements`-based so a coverage gap can't crash the byte-gate) + `strip_provided_typedefs(draft, provided)` (drop a draft's self-contained typedefs the target already supplies, splitting multi-typedef lines and covering scalar AND struct typedefs). Replaced **six** copied scalar-name regexes with complementary holes: `harvest_verify` now strips per-TU (unblocks the 39 struct-typedef drafts `_TD` dropped) and **surfaces cc1 stderr** so a `redefinition`/`conflicting types` failure reports as **PLUMBING**, not a byte mismatch (`.run/harvest_failed.classified.txt`); `masked_diff.strip_scalar_typedefs()` (used by `match_one`/`p16_permute`) fixes the multi-typedef-line skip that discarded 42 masked-MATCH drafts over whitespace (`func_8015C030` → `MATCH (23 ins)` unedited). `canon_sig_reconcile`/`eval_lora`/`format_finetune` keep their own copies for now (migrate per-bank, byte-gated — the audit-prescribed cadence). |
| | Phase 26-A tool-hygiene close (A9d–A10) | **DELETED** (R33, dead Phase-17 chain): `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py` — `reconcile_tu`/`cdecl` answer their question from the build. **`overlay_src_split.py`**: `scan_construct` force_decl latch fixed (no longer swallows a def sharing a line with leading externs) + `hidden_definitions()` R32 coverage oracle wired into `selftest`. **`jr_isolate_all.py` `jr_inventory`**: `banked` DERIVED FROM THE IMAGE (`family_remap.reloc_targets` owns-a-carve) not a gitignored roster (R33) + curated-name via `addr_of` + 1:1 carve-ownership assert. **`family_remap.reloc_targets`**: optional `data=` param (read the image once, pass to N calls). **`backlog.py`**: `BACKLOG_NO_RENDER` env so parallel `gate_stage` workers skip the render race (append is atomic). `reconcile_tu` confirmed live on BOTH banking paths (`gate_stage` + `jtbl_family_bank.recover`→`bank_exemplar`). |

---

## Backup & private-repo posture (rules R20/R21)

This project lives in a **private** remote (rule H1, relaxed: ROM-derived material may be committed while the repo is private). Per-session checkpoint backups (R20) push all irreplaceable work; the lists below record what is and is not pushed as of 2026-06-15.

**Backed up to the private remote (2026-06-15):**

- The **Ghidra project** (`ghidra/`) — with `*.lock` / `tmp*.ps` transients excluded (regenerable / ext4-local lock files).
- **PsyQ SDK working artifacts** (`tools/psyq/`) **MINUS** the two >100 MB raw source archives — the `psyq40usa.zip` and the DTL-S2002 disc `.bin`/`.cue` (re-sourceable, over GitHub's file-size limit).
- **Old-gcc cc1 compiler tarballs** (`tools/bin/*.tar.gz`) — the extracted binaries are regenerable from these, so only the tarballs are kept.
- **Ghidra extension installers** — `tools/ghidra-ext/GhidrAssistMCP_2.8.0.zip` + `ghidra_psx_ldr_2026.06.04.zip` (hard to re-source at exact pinned versions).

**Deliberately NOT backed up** (regenerable, or >100 MB and re-sourceable):

- The disc dump (`disks/`).
- The `extracted/` bulk — regenerate via `make extract`.
- `build/`, `expected/`, `asm/` — all generated.
- `.venv/` — recreate from `tools/requirements-python.txt`.
- The two >100 MB raw PsyQ archives (the `psyq40usa.zip` + DTL-S2002 disc).
- The unused PCSX-Redux Linux AppImage — the real runtime oracle is the Windows-native build.

**Rules:**

- **R20** — back up all irreplaceable RE/decomp work plus gathered hard-to-re-source tooling at per-session checkpoints. This **loosens R8** (which mandated a single commit at phase end): checkpoint commits are now expected within a phase.
- **R21** — keep **THIS file** (`docs/SETUP.md`) current whenever tooling, the MCP setup, the session hooks, or the environment changes.

Sony **PsyQ libs and cc1 stay PRIVATE** — they are excluded from the future curated public mirror (the two-repo public-release plan; see `docs/gen2-roadmap.md`). The four submodules (asm-differ / m2c / maspsx / decomp-permuter) stay **gitlinks** on GitHub (the deliberate no-bloat choice over vendoring); residual risk = upstream deletion of a pinned commit.

---

## Disc provenance (redump — canonical integrity reference)

The `SLUS_007.26` executable's own hash is not published anywhere (redump hashes disc *tracks*, not inner files), so dump authenticity is proven at the track level. `tools/bfm_extract/extract_exe.py --verify-disc` checks the Track-1 BIN against these:

| Track 1 (data) | Value |
|---|---|
| SHA1 | `b44f0f0a19936f23b26188b658e13201a6a9c211` |
| CRC32 | `c238191b` |
| MD5 | `838e9ecf8b4e4810f3f82228b40e77d8` |
| EXE build date | 1998-08-25 |

A Track-1 match proves the dump is the canonical redump dump, which transitively validates the extracted EXE (our own SHA1 `143dbb89f34491258bbc27810d0a12ec8b43a8dd`, 413,696 bytes — stable, independently reproduced from raw sectors). Audio tracks 2–4 SHA1s are in the research archive if ever needed.

---

## Known-unverified ledger (recheck and update in place)

| # | Item | Status |
|---|---|---|
| 1 | GhidrAssistMCP/psx_ldr 12.1 zips on Ghidra 12.1.2 | **MOOT for us** — pinned Ghidra **12.1** exactly; both extensions load fine at `version=12.1` (confirmed 2026-06-13). The 12.1.2 question stays untested by design. |
| 2 | MCP `types`/`struct` resolution of attached-archive (.gdt) types | **RESOLVED 2026-06-13** — import (`resolve()`) the .gdt types into the program headlessly (`tools/ghidra_scripts/ImportPsyqGdt.java`); MCP `types` then resolves them fully (§2.5 step 5) |
| 3 | GhidrAssistMCP struct-tool ergonomics under matching-decomp load | **UNPROVEN** — psxrecomp never exercised heavy struct creation |
| 4 | PCSX-Redux web-server port config field (8080 collision) | **RESOLVED 2026-06-13** — `pcsx.json` → `emulator.Debug.WebServer=true` + `emulator.Debug.WebServerPort=8081`; dump at `GET http://127.0.0.1:8081/api/v1/cpu/ram/raw` (verified 2 MB; EXE-in-RAM byte-match) |
| 5 | WSL distro is Ubuntu 24.04 (the single all-in-WSL host) | **CONFIRMED 2026-06-14 (Phase 4)** — `/etc/os-release` = Ubuntu 24.04.4 LTS (VERSION_ID 24.04) |
| 6 | binutils ≥2.38 regression on Ubuntu 24.04's shipped binutils | **RESOLVED (Phase 5): 2.42 is byte-clean** — `make build` is SHA1-identical with our flags; no downgrade (§4.5) |
| 7 | sha256 hashes of old-gcc 0.17 tarballs | **RECORDED 2026-06-14 (Phase 4)** — psx `500a459b…`, cdk `42bb0df9…` in `tools/bin/CHECKSUMS.sha256` (§4.7) |
| 8 | `gp_value` in SLUS_007.26 header → -G0 vs -G8 | **RESOLVED (Phase 5): -G0** — zero $gp-relative addressing in the disasm (§5.3) |
| 9 | ASPSX tier for game code: 2.56 vs 2.67 | **OPEN** — Phase 6 empirical (§5.2 tell) |
| 10 | Cross-OS networking | **N/A under all-in-WSL** — MCP is local loopback (§4.2); no mirrored mode, firewall rule, or host-IP discovery |
| 11 | Canonical git remote URL (off-box push/pull backup) | **TBD** (§4.3) |
| 12 | Per-libnum stamp detail (raw-track scan reported 16 hits vs 12 genuine in extracted EXE — extracted-EXE scan is ground truth, see §5.1) | **RESOLVED 2026-06-13** — DetectPsyQ at headless import recorded `PsyQ Version = 4.0.0` (§2.5 step 3) |
| 13 | Overlay load addresses (resident 0x800CDF58 / location 0x80128508, EXE ptr table ~0x62620) | **JP-only — re-derive for US** (owned by docs/memory-map.md) |
| 14 | Greenfield claim: decomp.me scratch search is script-blocked (Cloudflare) | **TBD** — one-time manual browser check for BFM scratches |