BFM-decomp/phase-ends/PhaseEnd_Phase3.md

PhaseEnd — Phase 3: File-loader & overlay-map RE (+ PCSX-Redux harness)

Date: 2026-06-14 · Project Version: 1.3.0 · Phase Status: Complete

Build Log

Files created/changed and complete — do not recreate:

• tools/ghidra_mcp_start.sh / ghidra_mcp_stop.sh / ghidra_mcp_verify.sh — oracle lifecycle: start detached headless MCP server / clean save+stop (the ONLY way headless MCP writes persist) / read-only R9 verify. Used every RE session.
• tools/ghidra_scripts/BfmMcpServer.java — headless MCP server + commit-on-clean-shutdown; run by start.sh.
• tools/ghidra_scripts/GetSymbolAt.java — read-only symbol probe consumed by verify.sh (R9).
• tools/ghidra_scripts/ImportOverlay.java — headless: loads the proven resident blob (→0x800CEDF8) + one location overlay (→0x80128158) into the program (RAM block, convertToInitialized+setBytes); idempotent. (T7)
• tools/ghidra_scripts/VerifyOverlay.java — read-only R9 check that the imported blob bytes persisted. (T7)
• tools/ghidra_scripts/ExportSymbols.java + config/symbols.us.txt — headless export of all 967 USER_DEFINED symbols (PsyQ-signature names + manual RE) to committed text — the durable, diffable, re-importable record of the RE work (the .rep is gitignored); Phase-5 splat input. (phase-close)
• dumps/INDEX.md + dumps/*.bin — the 28-dump live RAM corpus, committed (H1 relaxed while private; static, not regenerable, off-machine backup); INDEX maps each dump → state. (phase-close)
• .gitignore — added /tools/pcsx-redux/ (AppImage), /.run/ (runtime scratch); dumps committed.
• .claude/settings.local.json — SessionStart hook that auto-starts the oracle. (T0)
• /tmp → project-local migration: ghidra_mcp_start.sh/stop.sh + BfmMcpServer.java now use ~/bfm-decomp/.run/{ghidra-mcp.log,mcp-stop.req} (no /tmp); full start/stop/save/R9 cycle re-certified. (R12)
• docs/memory-map.md — the phase's main written deliverable. Added: the T1 "imported into Ghidra" block; the T2 (CdRead chain) / T3 (LIST.CD cache+lookup) / T4 (LZSS staging) / T5 (US load addresses) / T6b (MILESTONE proof) blocks; §4.3 overlay-map table (13 area-states + the loader model); §3.4 player-state promoted to verified (gold/HP/BP/day/hour/tiredness-raw) + refutations; §2.3 saveHeaderTemplate; §5 debug-menu dispatch; Open Questions #1,#3,#4,#5,#6,#8,#9,#12 resolved; SLUS-90029 corrected (FF8 demo, not BFM).
• docs/SETUP.md — §3 PCSX-Redux as-built (Windows-native + WSL web-API bridge); ledger #4 RESOLVED (web server :8081, dump URL).
• phase-ends/CURRENT_PHASE.md — absorbed into this file and deleted (P8).
• phase-ends/PhaseEnd_Phase3.md — this file.

Local artifacts (gitignored / outside repo — present on this machine, do NOT recreate; a fresh clone redoes them):

• ~/bfm-decomp/ghidra/bfm.{gpr,rep} — the saved Ghidra project. SLUS_007.26 now carries the T1 seeds plus ~50 Phase-3 symbols: loader cluster (LoaderInitFileTable, CdReadRequest, CdReadStateMachine, CdReadSectorReadyCB, CdQueueBusy, cdReq_* control block), resource loader (ResourceLoadStateMachine, ResourceGetCdLoc, resourceIdMap, resLoad_*), cdFileLocTable, listCdBuffer, lzss_sectorStagingBuf, StreamLoadStateMachine, loadDestPtrTable, GameModeDispatch+gameModeHandlerTable, DebugMenuHandler, SaveLoadRoutine, saveHeaderTemplate, debugBinPresent, player-state fields — plus 2 imported memory blocks (resident blob @0x800CEDF8, tut_forest overlay @0x80128158). R9-verified; DB saved (clean shutdown).
• tools/pcsx-redux/PCSX-Redux-HEAD-x86_64.AppImage — Linux AppImage (downloaded; NOT used — WSLg display + WSL controller failed; the Windows-native build is the actual oracle). Gitignored.
• disks/ — 2 CRC-verified US prototypes (Aug-31-1998 "Musashi Demo" 1369DE07, Sep-8-1998 "Musashi Master" 5C24728E) + a "Squaresoft on PlayStation Vol.2 (FF8 Demo)" disc. Gitignored.

Tools/packages installed:

• PCSX-Redux (runtime oracle). The Linux AppImage was fetched to tools/pcsx-redux/ but is unused; the Windows-native build (distrib.app/.../dev-win-x64) is what runs — web server on :8081, reached from WSL at 172.17.208.1:8081 (NAT gateway). Ships OpenBIOS (no BIOS dump needed).
• usbipd-win (Windows, installed by Drew) — attempted DualSense→WSL passthrough; abandoned (WSL2 kernel has no CONFIG_HID_PLAYSTATION/joydev) in favour of the Windows-native emulator.

Verification results (literal):

• MILESTONE byte-proof (live PCSX-Redux RAM): resident MAIN.CD/FILE_010/1.1 @0x800CEDF8 = 18,788-byte contiguous exact prefix, sha1 dbc55765593d3c157965a22af5808990714dbae9 (RAM slice == extracted, EQUAL); overlay SC01/FILE_077/0.4.dec @0x80128158 = 389,400-byte contiguous exact prefix, sha1 8e40afab29200f00752573bfc28f9a8676000f85 EQUAL. EXE resident: RAM[0x10000:] == extracted SLUS_007.26[0x800:] (8192/8192).
• R9 persistence verifies — all PASS: T1 (CdPathTable, db.5); checkpoint#1 (StreamLoadStateMachine); #2 (loadDestPtrTable); T7 (VerifyOverlay — both blobs init=true, bytes correct); #3 (DebugMenuHandler).
• PCSX-Redux web API GET :8081/api/v1/cpu/ram/raw → exactly 2,097,152 B.
• Overlay map: 13 area-states identified by RAM-vs-extracted byte-match across SC01/SC02/SC03; 2 duplicate-overlay pairs (SC02/000≡003, SC01/005≡006).
• Player/time state live-verified (multi-datapoint / controlled diff): gold 0x80078E8C (÷10), hp_max/cur 0x80078EB2/EB4, bp_max/cur 0x80078EB6/EB8, dayCounter 0x80078EAC, hour 0x80078EB1, tiredness_raw 0x80078EA4, bossHp_SteamKnight 0x801E4398. Four 2-datapoint false positives caught and refuted (clock 0x8007CE82, two tiredness guesses, a dow byte) → see R10.
• git status: only docs + tooling/scripts tracked; all ROM-derived (ghidra/, disks/, extracted bulk, the PCSX AppImage) gitignored.

Milestone achieved: docs/memory-map.md §4.3 holds a loader/overlay table where the resident blob (0x800CEDF8) and a location overlay (0x80128158) have US load addresses proven byte-identical (sha1-equal contiguous prefixes) against a live PCSX-Redux RAM dump — the project's central "what loads where" unknown, resolved.

Next: Phase 4 = PROTOTYPE RESEARCH SPIKE (Drew's directive — supersedes the roadmap's "build toolchain" as the next phase; toolchain follows). Start with a symbol/debug-info hunt in the Aug-31 "Musashi Demo". See "Notes for Future Phases" below for the full brief.

Deviations

Item	Plan	Actual	Reason
Runtime-oracle location	PCSX-Redux under WSLg (all-in-WSL)	Windows-native PCSX-Redux; WSL fetches RAM via 172.17.208.1:8081	WSLg wouldn't paint the window; WSL2 kernel can't do the DualSense. Hybrid bridged by the web API (→ R11)
Overlay-mapping scope	≥1 overlay (milestone)	13 area-states across SC01/02/03 + full loader model	Drew-driven live tour; high value for Gen2 overlay configs
Player-state RE	not in plan	gold/HP/BP/day/hour/tiredness/boss-HP verified; save-header found	Opportunistic live triangulation (Drew fed ground truth)
Prototypes	parking lot	2 US protos acquired + CRC-verified; FF8 demo disc	Drew acquired them; seed the Phase-4 spike
Next phase	Phase 4 = build toolchain (roadmap)	Phase 4 = prototype research spike	Drew's directive — assess proto value before matching
T6b proof form	"sha1 of dump slice == sha1 of extracted blob" (whole blob)	sha1-equal large contiguous prefix (runtime data diverges past the code)	Runtime writes mutate the blob's data after load; the code prefix is byte-identical at the exact vaddr — conclusive for the load address (P9)
.DA/proto context	—	SLUS-90029 corrected to FF8 demo (not a BFM demo); no playable US BFM demo exists	Drew correction + web research

Commit Message

feat(phase-3): file-loader & overlay-map RE + PCSX-Redux runtime oracle

- RE'd the full file-loader/overlay subsystem in Ghidra via MCP: CdReadRequest ->
  CdReadStateMachine (hand-rolled CdControl SeekL/ReadN reader, NOT PsyQ CdRead);
  LIST.CD RAM cache cdFileLocTable + ResourceGetCdLoc/resourceIdMap lookup; LZSS
  staging buffer 0x80079A70; loadDestPtrTable; ~50 symbols named (T1-T5,T8)
- T6b MILESTONE: resident blob @0x800CEDF8 + location overlay @0x80128158 proven
  byte-identical (sha1-equal contiguous prefixes) vs live PCSX-Redux RAM; blobs
  imported into Ghidra (ImportOverlay.java); overlay-map table -> memory-map.md §4.3
- PCSX-Redux runtime oracle: Windows-native build, WSL reads RAM via the web API
  at 172.17.208.1:8081; ledger #4 resolved
- live-verified player/time state: gold(/10) HP BP day hour tiredness boss-HP;
  save-header template (Q#5) + debug-menu dispatch (Q#9: DEBUG.BIN path is dead)
- overlay map: 13 area-states across SC01/02/03, 2 duplicate pairs, location x
  chapter keying, free-roam vs menu-hub granularity
- resolves Open Q#1/#3/#4/#5/#6/#8/#9/#12; corrects SLUS-90029 (FF8 demo, not BFM)
- 2 US prototypes acquired + CRC-verified (Aug-31 1369DE07, Sep-8 5C24728E)
- rules R9 (verify persistence), R10 (>=3-datapoint live verification),
  R11 (runtime oracle Windows-native, bridged to WSL)
- bumps project version 1.2.0 -> 1.3.0

Rules Added This Phase

Rule	Reason
R9 — Verify Ghidra persistence after every save-shutdown. Read-only re-open (tools/ghidra_mcp_verify.sh <addr> <name>) and confirm a recent edit survived — never trust "Save succeeded" alone. Mandatory at phase end; recommended at each checkpoint.	Headless MCP writes persist only on clean shutdown; the live server holds an open transaction. Catches a lost save before it strands work. (Candidate since this phase; formalized.)
R10 — Multi-datapoint live-RAM verification. A RAM-address finding from PCSX-Redux is verified only with ≥3 consistent datapoints OR a controlled before/after diff (e.g. paused snapshots, sleep reset). Two-datapoint matches stay candidate (coincidence-prone).	Four 2-datapoint false positives this phase (clock 0x8007CE82, two tiredness guesses, a day-of-week byte) were each disproven by a 3rd datapoint. Aligns G1/P9 (bytes are truth).
R11 — Runtime oracle is Windows-native, bridged to WSL via the web API (http://172.17.208.1:8081). Build, Ghidra, splat, asm-differ, analysis all stay in WSL (R2/H2 unchanged for those); only the emulator GUI + controller live on Windows — the explicit all-in-WSL exception.	WSLg can't render PCSX-Redux and the WSL2 kernel can't expose a DualSense; the web API is a clean, low-friction WSL↔Windows boundary that preserves the rest of the architecture.
R12 — No /tmp; project-local data only. All project data — including transient runtime scratch (logs, sentinels, temp outputs) — lives under ~/bfm-decomp (runtime scratch → .run/, gitignored). Never /tmp.	Drew directive 2026-06-14: /tmp is volatile (28 non-regenerable RAM dumps nearly lost on a reboot). Keeps the clone self-contained and durable.

PhaseEnd Changelog

v1.2.0 → v1.3.0 — Phase 3 complete. The file-loader/overlay subsystem is fully reverse-engineered in Ghidra and the central "what loads where" unknown is byte-proven against live hardware behavior: the resident blob (0x800CEDF8) and location overlays (0x80128158) load position-locked and verbatim; the loader is a hand-rolled CdControl async reader (not PsyQ CdRead) driven by cdFileLocTable (the parsed LIST.CD TOC) + a two-level resource-ID map. A live overlay map of 13 area-states across SC01/SC02/SC03 was captured (duplicate overlays, location×chapter keying, free-roam-vs-menu-hub granularity). PCSX-Redux stood up as the runtime oracle (Windows-native, WSL web-API bridge). Bonus live-verified the player/time state block (gold÷10, HP/BP, day, hour, tiredness) and the save-header (Q#5) + debug-menu dispatch (Q#9: DEBUG.BIN dead). Open Questions #1/#3/#4/#5/#6/#8/#9/#12 resolved. Three rules added (R9 persistence, R10 multi-datapoint verification, R11 Windows-native oracle). Two US prototypes acquired + CRC-verified, seeding the Phase-4 spike.

Notes for Future Phases — Phase 4 RESEARCH SPIKE brief (Drew's directive)

Analyze the two CRC-verified US prototypes (disks/) before the build/matching phases:

1. Debug info? Hunt the Aug-31 "Musashi Demo" + Sep-8 "Musashi Master" for .SYM/symbol tables, debug strings, assertion/source-path text, less-stripped data. (Hidden Palace said "no symbols" — UNVERIFIED.)
2. Is the demo easier to decomp, and does it help the full game? Assess optimization level vs retail. Caveat: demo ≠ retail bytes → demo matches don't transfer to SLUS-00726; value is labels + a readable reference + shared tables, not matched bytes.
3. Free-label lead (already identified): the Aug-31 demo's title scene/area-select very likely shares code/tables with the retail debug menu mapped in T8 — GameModeDispatch (0x80010B40), gameModeHandlerTable (0x800629F4), DebugMenuHandler (0x80011144). Binary-diff to lift labels in.
4. Deliverable: a go/no-go on whether the protos accelerate Gen1 matching, plus any labels imported. This is a roadmap insertion before Phase 4 (build toolchain), which then follows. Also outstanding for later: the save-data block format (trace SaveLoadRoutine 0x8002B154, Q#5); the exact tiredness-% / minute / day-vs-dow encoding; importing more overlays as Ghidra overlay address spaces (Gen2).

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md absorbed. Drew commits AND pushes this PhaseEnd (R6/R8). Start a fresh session for the next phase. Keep this file forever.