BFM-decomp/phase-ends/PhaseEnd_Phase3_5.md

# PhaseEnd — Phase 3.5: Prototype Research Spike
**Date:** 2026-06-14 · **Project Version:** 1.3.1 · **Phase Status:** Complete

> Inserted half-phase (Drew's directive at Phase-3 close): assess whether the two CRC-verified US
> prototypes accelerate Gen1 matching, **before** the build toolchain (which resumes as Phase 4).
> Filename uses an underscore (`Phase3_5`) so the glob load order sorts Phase3 → Phase3_5 → Phase4.

## Build Log

**Files created/changed and complete — do not recreate:**
- `tools/bfm_extract/extract_proto_exe.py` — extract a main EXE from a prototype disc; reuses the general
  `iso9660` reader, `--dir` for subdirectories (the demo's real EXE is in `MUSASHI/`), magic-only
  validation (proto sizes/SHA1s differ from retail). Companion to the retail-locked `extract_exe.py`.
- `tools/ghidra_import.sh` — reusable headless import of any PS-X EXE into the `bfm` project (PSX-loader
  auto-detect → auto-analysis incl. PsyQ signatures + DetectPsyQ → PsyQ-4.0 types via `ImportPsyqGdt.java`
  → `DumpProgramInfo.java`). Precondition: MCP server stopped (exclusive project lock).
- `tools/ghidra_scripts/DumpFunctionSignatures.java` — **read-only** per-function 3-tier fingerprint
  (`h_exact` raw bytes / `h_norm` MIPS-normalized / `h_seq` mnemonic-only) + ordered call targets →
  `.run/sig.<prog>.jsonl`. Consumed by `match_protos.py`.
- `tools/match_protos.py` — cross-binary correspondence: per-tier counts, 1:1 unique `h_norm` pairs,
  anchor sanity check; emits `config/symbols.proto-*.txt`. Consumes the `.run/sig.*.jsonl` dumps.
- `docs/proto-correspondence.md` — **the deliverable**: the evidence-backed go/no-go report.
- `config/symbols.proto-sep8.txt` (953) / `config/symbols.proto-demo.txt` (682) — retail names mapped onto
  the protos via 1:1 `h_norm` correspondence. **R13-tagged: proto-side, UNVERIFIED, never merge into
  `config/symbols.us.txt`.** Phase-6/Gen2 navigation aid.
- `docs/memory-map.md` — added the **Phase-3.5 block** (proto correspondence summary + DEMO dispatch
  anchors), resolved **Open Q#10**, upgraded §5 proto notes ("no symbols") to **VERIFIED**.
- `.gitignore` — nested allowlist for `extracted/retail/` (3 committed files re-included); `extracted/proto/`
  fully ignored.
- `tools/bfm_extract/extract.py` · `extract_exe.py` · `crosscheck.py` — DEFAULT output/extracted paths
  → `extracted/retail` (per-ROM subfolders).
- `tools/ghidra_mcp_start.sh` — `PROG` now `${1:-SLUS_007.26}` (serve a prototype by passing its name).
- **Renamed (git mv, history preserved):** `extracted/{SLUS_007.26,manifest.jsonl,manifest.sha1}` →
  `extracted/retail/` (the extracted/ reorg; the 34 gitignored bulk items moved alongside).

**Local artifacts (gitignored / outside the repo — present on this machine; a fresh clone redoes them):**
- `extracted/proto/sep8_SLUS_007.26` (413,696 B, sha1 `43006a314513d962528ddc90ee496c781867366a`) +
  `extracted/proto/aug31_USA_DEMO.EXE` (415,744 B, base 0x80018000, sha1
  `6150b0f839972ac52214e3510e7d7f0e2c194624`) — extracted proto main EXEs.
- Ghidra `bfm` project now holds **3 programs**: `SLUS_007.26` (retail) + `sep8_SLUS_007.26` (1726 funcs)
  + `aug31_USA_DEMO.EXE` (1737 funcs); all PsyQ 4.0.0; R9-verified. (The retail DB was NOT modified.)
- `extracted/retail/` bulk (the Phase-2 disc extraction, moved here); `.run/sig.*.jsonl` ×3,
  `.run/correspondence.*.tsv` ×2, `.run/proto-import.log` — scratch.

**Tools/packages installed:** None (used the existing Ghidra 12.1 + Python stack).

**Verification results (literal):**
- **Proto extraction:** sep8 `SLUS_007.26` 413,696 B, sha1 `43006a31…`; aug31 `USA_DEMO.EXE` 415,744 B,
  base 0x80018000, sha1 `6150b0f8…`; both `PS-X EXE` magic verified. `extracted/` reorg: `extract.py
  --verify --out extracted/retail` **PASS** (manifest paths are out_root-relative → move transparent);
  git shows 3 renames; **zero ROM bulk staged**.
- **Ghidra import:** sep8 **1726 funcs / PsyQ 4.0.0**, aug31 **1737 funcs / PsyQ 4.0.0**; both
  `Import succeeded` + **R9-verified** by read-only re-open.
- **T2 (debug-info):** main EXEs string-identical (retail=sep8=351, aug31=331; only `\DEBUG.BIN;1` +
  `SetGraphDebug`); disc-wide retail/sep8 debug-string-identical (`_dbg_RE1L@55`, `C:\TIMPACK\` in **both**;
  `DEBUG`-substrings retail 63 / sep8 59 = garbage). **No symbols in any proto — Hidden Palace claim
  VERIFIED; the recon "Sep-8 less-stripped" signal REFUTED.**
- **T3 (correspondence, substantial ≥8 ins):** sep8 h_exact **99.4%** / h_norm **99.7%** / h_seq 99.7%,
  1072 1:1 pairs; aug31 h_exact 16.0% / h_norm **65.9%** / h_seq 99.0%, 690 1:1 (862 all sizes). Normalizer
  validated by anchors (`LzssDecodeSector`, `CdReadRequest` matched across the demo's shift+rebuild).
  Sep-8→retail pre-release diff = exactly **3 substantial funcs** (`DebugMenuHandler`,
  `CdReadSectorReadyCB` 424 ins, `SaveLoadRoutine`).
- **T4 (anchor):** demo dispatch table @0x8006AE8C = **18 entries** (= retail's 18, same idx[10]==idx[15]
  duplicate); demo `GameModeDispatch` 0x80018b48, `gameMode` 0x800c1eb6; slot **[7]** 0x800194cc (= retail
  `DebugMenuHandler` slot, **code diverged**). **Q#10 resolved.**
- **T6:** `config/symbols.proto-sep8.txt` 953 names, `config/symbols.proto-demo.txt` 682 names, R13-tagged.

**Milestone achieved:** `docs/proto-correspondence.md` states an evidence-backed go/no-go — **SPLIT:
NO-GO** as a Gen1 label/symbol accelerator (no symbols in either proto, *verified* by byte compare; the
free-label lead refuted), **GO** to retain Sep-8 (Phase-6 compiler-determinism corroborant; 99.6%
byte-identical, 3-function pre-release diff) and both (Gen2 assets) — backed by a per-proto debug-info
inventory, quantified retail↔proto correspondence (Sep-8 99.6% / Aug-31 66% norm, 862 1:1), and the
Aug-31 anchor alignment (18-entry dispatch table located, slot[7] mapped) resolving Q#10; proto-side
labels imported (`config/symbols.proto-*.txt`).

**Next:** Phase 4 — Matching-build toolchain (WSL). Start with: apt toolchain + Python 3.12 venv +
splat64 installed; submodules (asm-differ, m2c, maspsx, decomp-permuter) added and pinned.

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Spike outcome | open (free-label lead hypothesized "GO") | NO-GO for Gen1 accel; GO only as asset | Bytes refuted the symbol/label hypothesis (T2); protos have FEWER names than retail's 952 |
| `extracted/` layout | single `extracted/` root | reorganized → `extracted/retail/` + `extracted/proto/` | Drew directive mid-phase (multiple ROMs now); inserted cleanup task |
| T6 lift direction | name protos **and** annotate retail w/ proto provenance | one-way retail→proto only; **retail untouched** | verdict: protos add nothing retail lacks → nothing to annotate; preserves crown-jewel DB (G6) |
| T6 application | apply names in the Ghidra proto programs + export | `config/symbols.proto-*.txt` only; **Ghidra-side naming deferred** | premature given NO-GO (protos are future assets); the text file is the durable record |
| T2 xref step | map debug strings → functions via MCP | moot — no new debug strings exist in any main EXE | negative finding; nothing to map |
| Recon debug signal | (4th explore agent: "Sep-8 = GO, strong") | refuted | naive disc-wide grep counted coincidental `X:\`/`DEBUG` byte sequences (→ R14) |
| `BfmVtExactMatch.java` (Version Tracking) | available for T6 on GO | not created/needed | correspondence computed in Python; the bounded lift needs no Ghidra writes |
| MCP for the spike | live MCP for xref/anchor | mostly standalone read-only `analyzeHeadless` | lock-serialized + avoids `/mcp` re-attach churn; only one decompile needed |

## Commit Message
```
feat(phase-3.5): prototype research spike — go/no-go (protos don't accelerate Gen1 matching)

- extract_proto_exe.py: subdir-aware proto main-EXE extraction (reuses iso9660); extracted
  sep8 SLUS_007.26 (413,696 B) + aug31 MUSASHI/USA_DEMO.EXE (415,744 B, base 0x80018000)
- ghidra_import.sh: reusable headless import; both protos imported into the bfm project
  (sep8 1726 funcs, aug31 1737 funcs, both PsyQ 4.0.0), R9-verified
- DumpFunctionSignatures.java + match_protos.py: 3-tier (exact/norm/seq) cross-binary
  correspondence; MIPS normalizer validated by anchors
- FINDINGS: (T2) no debug symbols in either proto — Hidden Palace "no symbols" VERIFIED;
  recon "Sep-8 less-stripped" REFUTED by per-file byte compare. (T3) Sep-8 99.6% byte-identical
  to retail (only 3 funcs differ: DebugMenuHandler, CdReadSectorReadyCB, SaveLoadRoutine);
  Aug-31 66% norm-identical, 862 1:1 correspondences. (T4) demo shares retail's 18-entry
  game-mode dispatch architecture but the handler code diverged — Q#10 resolved
- VERDICT (docs/proto-correspondence.md): NO-GO as a Gen1 label/symbol accelerator; GO to retain
  Sep-8 (Phase-6 compiler corroborant) + both (Gen2 assets); proto-side labels in
  config/symbols.proto-{sep8,demo}.txt (R13-tagged, never feed symbols.us.txt)
- reorg extracted/ into per-ROM subfolders (extracted/retail/, extracted/proto/); nested
  gitignore allowlist; extractor defaults updated; manifest --verify PASS
- memory-map.md: Phase-3.5 block + demo dispatch anchors; Q#10 RESOLVED; §5 proto notes VERIFIED
- ghidra_mcp_start.sh: PROG arg to serve a prototype
- rules R13 (proto-provenance/confidence tagging), R14 (verify recon counts vs bytes)
- bumps project version 1.3.0 -> 1.3.1
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **R13 — Proto-provenance & confidence tagging.** Every prototype/cross-build-derived datum carries source build (`sep8`/`demo`/…), match tier (`exact`/`norm`/`fuzzy`), and verification status (`verified-retail` only if independently confirmed in retail; else `unverified`). `config/symbols.us.txt` and the curated retail Ghidra DB receive ONLY `verified-retail` data; unverified proto-derived names live exclusively in `config/symbols.proto-*.txt` + reports (never merged into `symbols.us.txt`); retail comments carrying proto info are prefixed `[PROTO:… UNVERIFIED]`. | Drew directive 2026-06-14: keep the deferred full-lift safe and never corrupt the crown-jewel symbol list with uncertain cross-build data. Extends G5/G6. |
| **R14 — Verify recon/sub-agent summary counts against the bytes.** A sub-agent's or reconnaissance pass's summarized counts/signals are claims, not ground truth; confirm against the actual bytes before relying on them. | A recon "Sep-8 is less-stripped (strong GO signal)" was a false positive — coincidental `X:\`/`DEBUG` byte sequences in compressed data — refuted by per-file byte comparison. Extends G1/X2. |

## PhaseEnd Changelog
**v1.3.0 → v1.3.1** — Phase 3.5 (inserted prototype research spike) complete. Both US prototypes extracted,
imported (PsyQ 4.0.0), and diffed against retail via a new read-only 3-tier signature pipeline. **Go/no-go:
the prototypes do NOT accelerate Gen1 matching** — neither carries symbols/debug-info/labels retail lacks
(*verified*; the demo's scene-select shares retail's 18-mode dispatch *architecture* but its handler code
diverged and is unnamed, so the "free labels" lead is refuted). They are **retained** as a Phase-6
compiler-determinism corroborant (Sep-8 = 99.6% byte-identical to retail; the entire pre-release delta is 3
functions) and Gen2 assets (Aug-31 = the same engine rebuilt, SC02 region, 862 1:1 correspondences). Open
Q#10 resolved. `extracted/` reorganized into per-ROM subfolders. New reusable tooling (proto extractor,
headless importer, signature dumper + matcher). Rules R13 (proto-provenance tagging) and R14 (verify recon
vs bytes). **Deferred (Drew):** the exhaustive Version-Tracking + manual near-miss full-lift to a later phase.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` absorbed and removed. **Drew commits AND pushes** this PhaseEnd
(R6/R8). Start a fresh session for **Phase 4 — Matching-build toolchain**. Keep this file forever.