BFM-decomp/docs/proto-correspondence.md

Prototype Research Spike — Correspondence & Go/No-Go (Phase 3.5)

Date: 2026-06-14 · Evolvable docs/ layer. The go/no-go deliverable for Phase 3.5. Question (Drew's directive): do the two CRC-verified US prototypes accelerate Gen1 matching? Bytes are the only truth (P9/G3): proto bytes ≠ retail, so proto matches never transfer as matches.

TL;DR — Verdict: SPLIT

Claim	Verdict	Why
Protos accelerate Gen1 matching via symbols / debug-info / labels / readability	NO-GO	Refuted: neither proto carries any symbols, source paths, or labels retail lacks (Finding 1). Both have fewer names than retail's 952 — the label flow is retail→proto, not proto→retail.
Protos are worth retaining as project assets	GO (modest, targeted)	Sep-8 is a near-duplicate of retail (99.6%) → a Phase-6 compiler-determinism corroborant. Aug-31 is the same engine rebuilt (862 1:1 correspondences, shared dispatch architecture) → a Gen2 overlay/engine asset (it covers the SC02 region) + a marginal differential-analysis aid.

Actions: (1) Do not build proto-based labeling into the Gen1 matching loop — there is no payoff. (2) Keep the imported Ghidra programs + the correspondence artifacts as committed assets. (3) The bounded lift (T6) is one-way, retail→proto (name the proto programs from retail to make them navigable for Phase-6/Gen2); nothing is pushed onto the retail DB (G6). (4) All proto-derived data is provenance/confidence-tagged (R13) so a later full-lift phase can verify/extend it without risk to config/symbols.us.txt.

The prototypes

Build	Main EXE	Size	Load addr	SHA1	Funcs	PsyQ	Source
Retail (SLUS-00726)	SLUS_007.26	413,696	0x80010000	143dbb89…	1726	4.0.0	redump
Sep-8 "Musashi Master"	SLUS_007.26 (root)	413,696	0x80010000	43006a31…	1726	4.0.0	Hidden Palace, CRC 5C24728E
Aug-31 "Musashi Demo"	MUSASHI/USA_DEMO.EXE	415,744	0x80018000	6150b0f8…	1737	4.0.0	Hidden Palace, CRC 1369DE07

• The Aug-31 demo loads 0x8000 higher; its root LAUNCH.EXE (24,576 B) is a bootloader owning the low 32 KB. The demo ships MAIN.CD (= retail size) + SC02.CD only → it is the SC02 region (tutorial/Allucaneet area). The Sep-8 master is structurally retail-identical (27 files, full MAIN+SC01–07).
• All three share PsyQ 4.0.0 (same compiler family — independent corroboration for the Phase-6 fingerprint).

Method (reproducible)

1. Extract proto main EXEs: tools/bfm_extract/extract_proto_exe.py (reuses the general iso9660 reader; --dir for the demo's MUSASHI/ subdir; magic-only validation) → extracted/proto/ (gitignored).
2. Import into the bfm Ghidra project: tools/ghidra_import.sh <exe> (PSX loader auto-detect → auto-analysis incl. PsyQ signatures + DetectPsyQ → PsyQ-4.0 types → DumpProgramInfo).
3. Fingerprint every function (read-only): tools/ghidra_scripts/DumpFunctionSignatures.java → .run/sig.<prog>.jsonl. Three tiers so the match count is bracketed:
   • h_exact — SHA1 of raw instruction bytes (strict).
   • h_norm — SHA1 of a MIPS-normalized stream: neutralize j/jal absolute targets, lui high-halves, %lo/gp-relative address immediates (detected via Ghidra's analyzed references), branch targets → PC-relative offsets; keep registers (regalloc) + true constants. The cross-build "same source, relocated" tier.
   • h_seq — SHA1 of the mnemonic-only sequence (loose).
4. Correspondence: tools/match_protos.py → per-tier counts + 1:1 unique h_norm pairs → .run/correspondence.<proto>-vs-retail.tsv.

Normalizer validated by anchors: LzssDecodeSector and CdReadRequest both h_norm-match across the Aug-31 address shift + rebuild (0x80018730→0x80020c20, 0x80019a24→0x80021ed0).

Finding 1 — No debug symbols in either prototype (T2)

Hidden Palace's "no symbols" is now verified, not assumed. Per-file byte comparison (the three main EXEs and the three disc images):

• Main EXEs are stripped identically. Each carries only two "debug" strings: \DEBUG.BIN;1 (the known dead loader path) and SetGraphDebug:level:%d… (a stock PsyQ format string, present in retail). No source paths, no symbol tables, no function names in any main EXE.
• Disc-wide, retail and Sep-8 are debug-string-identical. ^_dbg_RE1L@55; is in retail too (not Sep-8-only); C:\TIMPACK\OPDEMO0/1.PAT is in both (shipped opening-demo data); DEBUG-substring counts are retail 63 vs sep8 59 — both pure garbage (coincidental ASCII in compressed data).
• Correction: an early reconnaissance pass reported "Sep-8 is less-stripped (11× DEBUG + a dbg_ linker symbol)" as a strong go-signal. That was a false positive from naive disc-wide strings|grep counting coincidental X:\/DEBUG byte sequences. Per-file comparison shows parity. (→ rule candidate R14: verify sub-agent/recon summary counts against the bytes before relying on them.)

Finding 2 — Code correspondence (T3)

Retail real functions (≥1 instruction): 1569; substantial (≥8 ins): 1179.

Proto vs retail	h_exact (subst.)	h_norm (subst.)	h_seq (subst.)	1:1 unique norm pairs
Sep-8	99.4%	99.7%	99.7%	1072 (≥8 ins)
Aug-31	16.0%	65.9%	99.0%	690 (≥8 ins)

• Sep-8 is a near-duplicate of retail. Only 3 substantial functions differ at every tier — the pre-release diff: DebugMenuHandler (0x80011144, 30 ins), CdReadSectorReadyCB (0x8001a338, 424 ins — the inline-LZSS-decode CD-read callback), SaveLoadRoutine (0x8002b154, 60 ins) — plus 3 pure-relocation diffs. Everything else is byte-identical at the same address. Sep-8 therefore adds essentially no new matching information; its worth is corroboration (below).
• Aug-31 is the same engine, genuinely recompiled. Low h_exact (address-shifted + rebuilt) but 66% of substantial functions are normalized-identical and 99% share the opcode sequence — two-thirds of the engine is literally the same source→codegen, the rest minor codegen drift. 862 confident 1:1 correspondences (.run/correspondence.aug31_USA_DEMO.EXE-vs-retail.tsv).

Finding 3 — Shared dispatch architecture, diverged handlers (T4; resolves Open Q#10)

Anchor-bootstrap from the retail debug-menu anchors:

• The demo's game-mode dispatcher is FUN_80018b48 (matched by opcode sequence to retail GameModeDispatch 0x80010b40; relative offset +0xB48 vs retail +0xB40 — the demo mirrors retail's early-code layout, shifted +0x8000). It decompiles to the same pattern: (*(code*)(&handlerTable)[gameMode])().
• Both handler tables have exactly 18 entries, in the same order, with the same internal duplicate (index[10] == index[15] in both). retail gameModeHandlerTable @0x800629F4 ↔ demo table @0x8006AE8C.
• Slot [7] (the menu): retail DebugMenuHandler 0x80011144 ↔ demo 0x800194cc — same slot, but the handler code diverged (no norm/seq match in the demo or in Sep-8).

Q#10 answer: the demo's "press-Start scene-select" and retail's "L3 debug menu" are the same architectural feature (game mode #7 of an identical 18-mode dispatch table) — but the handler is reimplemented, and the demo is unnamed, so there are no labels to lift onto retail. The shared architecture is real; the "free labels" payoff is not.

Go/No-Go rationale

The spike's hypotheses, each tested against the bytes:

1. Protos carry symbols / debug info → refuted (Finding 1).
2. Protos are more readable / easier to decomp → no — equally stripped; no symbols (Finding 1).
3. The demo's title menu shares retail's debug-menu code → free labels → refuted — same architecture, diverged + unnamed handler (Finding 3).
4. A second build helps matching → only marginally — Sep-8 ≈ retail (no new info); Aug-31 gives a same-source-different-address differential that can disambiguate a stubborn function, but it is not a force-multiplier and carries no names (Finding 2).

So for the stated question — accelerate Gen1 matching — the answer is NO-GO. The protos do not shorten the matching loop.

What they are worth (hence retain, don't discard):

• Sep-8 → Phase 6. A second build that is 99.6% byte-identical to retail is strong evidence the toolchain is deterministic, and it pins exactly which 3 functions changed pre-release. When the compiler triple is pinned and retail is rebuilt, Sep-8 is a free second cross-check.
• Aug-31 → Gen2. A second, navigable build of the engine (once named from retail) covering the SC02 region, with 862 mapped functions and a confirmed shared dispatch architecture — a head-start for Gen2 overlay/engine RE. Its overlays (MAIN.CD = retail size; SC02.CD) are a second sample for the Gen2 overlay work.

Actionable outcomes

• Keep the imported Ghidra programs (sep8_SLUS_007.26, aug31_USA_DEMO.EXE) in the bfm project and the committed correspondence artifacts.
• T6 (bounded lift, one-way retail→proto): apply retail's names onto the proto programs via the 1:1 h_norm correspondences (proto-side only), export config/symbols.proto-sep8.txt / config/symbols.proto-demo.txt. This makes the protos navigable for Phase-6/Gen2 without touching config/symbols.us.txt.
• Do NOT push proto-derived names/comments onto the retail DB (the protos have nothing retail lacks; G6).
• Deferred (Drew): the exhaustive exact+norm+fuzzy adjudication (Version Tracking + manual near-miss review) is a later phase. Every proto-derived datum is tagged unverified unless independently confirmed against retail, so that phase can verify/extend safely (R13).

Provenance & confidence (R13)

Datum	Region	Confidence
Proto EXE sizes / SHA1s / load addrs / func counts / PsyQ version	PROTO	verified (read from the binaries)
Correspondence tier counts; 862/1072 1:1 pairs	PROTO↔US	verified (deterministic over the dumps)
Sep-8 pre-release diff = {DebugMenuHandler, CdReadSectorReadyCB, SaveLoadRoutine}	PROTO↔US	verified
Demo dispatch anchors: GameModeDispatch 0x80018b48, gameModeHandlerTable 0x8006ae8c (18), gameMode 0x800c1eb6, slot[7] 0x800194cc	DEMO	verified (decompiled + table read from the demo binary) — DEMO addresses, NOT valid for US retail
Individual retail↔proto function name assignments (T6 output)	PROTO	unverified until confirmed vs retail — quarantined to config/symbols.proto-*.txt