BFM-decomp/phase-ends/logs/Phase12.md

# CURRENT_PHASE — Phase 12: Key engine systems (script VM, save/Q#5, sound SQV)
**Generation:** Gen2 (5th phase, arc 8→9→10→11→**12**→{13}→14) · **Effort:** Max (Tier-1 phase) · **Status:** APPROVED 2026-06-16, in progress
**Plan gate (P3):** approved by Drew 2026-06-16. This is the per-task crash-recovery log (P4); archived to `phase-ends/logs/Phase12.md` at close (R19).

## Owner decisions locked (2026-06-16)
1. **Milestone = "Substantial + honest deferral."** Amend gen2-roadmap EXIT criterion #2 / Phase-12 milestone: "script-VM core matched" → **"script/event system *resolved* (core matched if a tractable interpreter exists, else its architecture documented)."** Honest documented/partial outcomes count for the VM + the entangled save remainder (P9). (gen2-roadmap.md is the evolvable docs layer — amendment in scope, done in T10.)
2. **Runtime = planned guided trace sessions.** T8 (VM) + T9 (Save) include scheduled PCSX-Redux trace sessions: Drew drives gameplay; Claude captures RAM via the web API `172.17.208.1:8081` (R11) + correlates against Ghidra.

## Verified starting state (byte-checked during planning — R14)
- `main` = EXE `143dbb89…` (PsyQ 4.0): 54 REAL / 959 LINKED / 7 NON_MATCHING / 50.3%.
- `resident` = `MAIN.CD/FILE_010/1.1` @ vram 0x800CEDF8 `8e17e02f…` (PsyQ **4.7**): **0 REAL / 0 LINKED / 143 stubs / 1.4%**. Code = ~17.5 KB only (143 funcs, code region 0x800CEDFC..0x800D3408); rest data tail. Harvest queue: 14 trivial + 50 non-jtbl leaves + 5 indirect-jump funcs.
- Resident **game-code triple UNKNOWN** (4.7 ⇒ maybe newer ASPSX/cc1 than the EXE's gcc-2.7.2/aspsx-2.56) — biggest fork (T2).
- 4.7 SDK at `tools/psyq/conv47/…/lib/` (55 `.o` + 23 `.a` + headers) — build to ELF, then `psyq_identify` for the real footprint.
- Save: `SaveLoadRoutine` @ 0x8002B154 in **EXE** (1139-ins multi-entry, Gen1-deferred). VM: **unlocated** (`VM_*` are libsnd, not a game VM; overlays may be compiled MIPS). SQV: resident `DsMix` @ 0x800D1BD8 (UNVERIFIED) + custom glue.
- Resident in **0 cross-binary dedup groups** (overlays call, don't embed) — match value = named symbol, not fleet credit. 6 h_exact / 9 h_norm intra-resident self-dup groups to collapse.
- EXE leftover `VM_*`/`EVENT_OBJ` stubs = Phase-8 scattered-`.bss` residue → **out of scope**.

## Task checklist (one at a time, byte-gated) — REVISED post-T1 (no resident library footprint; Drew-approved 2026-06-16)
> T1 found NIL library footprint → the resident is all game code. Library-integration tasks **REMOVED**. Harvest = **full substantial** (Drew). The old VM "investigation" reframes as compiled-MIPS state/mode dispatch, matched within the harvest + documented.

**Block A — Resident enablement**
- [x] **T1** — 4.7 lib footprint survey → **NIL** (resident is all game code; verified both oracles). Old T3/T4/T5 library-integration REMOVED.
- [x] **T2** — Resident compiler-fingerprint probe → **triple CONFIRMED = the EXE's** (`gcc-2.7.2-psx -O2 -G0 …` + `maspsx --aspsx-version=2.56 --expand-div`); opt level -O2 (0 frame-ptr sigs); 2 byte-exact matches across idiom classes. No ladder; STOP-cond didn't fire.

**Block B — Resident engine harvest (the phase body; full substantial)**
- [ ] **T3** — Harvest the resident engine in difficulty/cluster **batches** ← **CURRENT** (each byte-gated `make check BINARY=resident` + committed; bank don't strand): trivial accessors → leaves → mid-size orchestration/state-dispatch → larger/branch-heavy + the 5 indirect-jump funcs. Collapse the 6 h_exact / 9 h_norm intra-resident dup groups (dedup.us.yaml + src/shared/). Includes the sound cluster (DsMix region) + the mode/state-dispatch "script/event" functions. Names → symbols.resident.txt + Ghidra (R15/G6). `[xHigh; Max for harder clusters]`

**Block C — Documentation + EXE save system**
- [ ] **T4** — Document the SQV (sound) format + the script/event-dispatch architecture determination (compiled-MIPS dispatch, NOT a bytecode VM — P9) in docs/formats.md + memory-map.md. (Matching happens in T3; this is the written deliverable.) `[Max]`
- [ ] **T5** — Save/Q#5 (EXE `main`) RE + format doc + tractable matches (guided PCSX-Redux trace, Drew-operated). Gate: `make check BINARY=main` = `143dbb89`; honest NON_MATCHING for the entangled remainder (G4). `[Max]`

**Block D — Close-out**
- [ ] **T6** — Cookbook (R16) + finalize symbols.resident.txt mirrored to Ghidra + ghidra_mcp_stop/verify (R20/R23) + regenerate reports (both binaries) + amend gen2-roadmap criterion #2 + PhaseEnd_Phase12 (Tier-1). `[Max]`

## Current task pointer
**T3 — Resident engine harvest (full substantial; the phase body).** Triple confirmed (T2), harness proven. Harvest the ~141 remaining resident functions in difficulty/cluster batches (difficulty.resident.md order: 14 trivial → 50 leaves → mid-size orchestration/state-dispatch → larger/branch-heavy + 5 indirect-jump). Each function: write C (scaffold via decompile.py for the bigger ones; the small ones hand-write from asm) → `make build BINARY=resident` byte-gate. Commit per batch (R20). Collapse the 6 h_exact / 9 h_norm intra-resident dup groups (dedup.us.yaml + src/shared/). Name functions as understood → symbols.resident.txt + Ghidra (R15/G6). Includes the sound cluster (DsMix region) + the mode/state-dispatch "script/event" functions. Bank, don't exhaust. **R22:** clean-rebuild gate before each commit; both binaries green.

## Per-task progress log
*(append one entry per task as completed: what was done, the byte-gate result, the commit)*
- **T1 — 4.7 lib footprint survey (DONE; finding overturns the Block-A premise).** Built the 4.7 ELF lib set (the conv47 objects are already ELF mipsel R3000 → skipped psyq-obj-parser; `ar x` each `.a` → `.run/obj47/<lib>/`). Ran `psyq_identify --vram-base 0x800CEDF8 --exe …/FILE_010/1.1` over the resident for libsnd/libspu/libgte/libgpu/libcd/libetc/libapi/libc2/libmath/libmcrd/libds.
  - **Result: NIL library footprint.** Whole-file scan (code + data tail): 4.7 libsnd **1/226** (a 4-ins coincidence `ut_rev_2.o`), libspu **0/134**, libgte **0/509**, libgpu **0/61**; libcd/libmath/libds each 1 hit, all ≤8 ins (two alias the same 8-ins DsMix addr). 4.0 cross-check: libsnd 2/163 (`VM_DON.o` 5 ins / `VM_DOFF.o` 4 ins), libspu 0/129, libgte 0/381. All hits are the §9.5 short-object coincidental class.
  - **Tool sanity (confirms the negative is real):** 4.0 libsnd vs the EXE snd region = **35/163** placed (the tool works + is version-sensitive); 4.7 libsnd vs EXE = 7 (short coincidences only). So the resident genuinely embeds **no stock PsyQ library objects of either version**.
  - **Architectural conclusion:** the PsyQ SDK lives in the **EXE** (959 LINKED); the **resident is entirely custom engine code** that calls the EXE's resident SDK + engine fns via fixed addresses (no RAM-wasting SDK duplication in an always-loaded blob). R24's "resident is 4.7 → link its 4.7 libs" is **moot** — the DetectPsyQ 4.7 signal was one coincidental DsMix-region pattern, not a linked footprint. `DsMix` @0x800D1BD8 is custom sound code, not a stock libsnd object (R13 tag effectively refuted).
  - Resident still `8e17e02f` (nothing integrated). Footprint report to be committed with the revision (below).
  - **Ghidra verification (Drew-requested, G1 — CONFIRMS the pivot):** switched the headless MCP to the `resident` program (296 funcs); decompiled a sample. `FUN_800cf854` = game-global accessor (`return DAT_800ae6bf != 0`); `DsMix` = `{FUN_800d1bf8(); return 1;}` (custom wrapper ignoring its arg — NOT stock libsnd; R13 tag refuted); `FUN_800cee84` = engine init calling EXE REAL matches (func_80029254/func_8002AEF8) + GameModeDispatch-region fns; `FUN_800d2318` = entity-heading math calling the EXE's libgte `RATAN` @0x8004CFEC. Call-target scan: 61 distinct EXE-range `jal` vs 37 internal — the resident calls the EXE's SDK/engine. **Resident = game code, no linkable library. Verified by both oracles.**
- **T2 — Resident compiler-fingerprint (DONE; triple confirmed = EXE's).** Opt level = **-O2** (resident has 0 `21f0a003` frame-ptr sigs vs the EXE's 54; the EXE's `800` game module is also -O2). The resident uses the Makefile's default pinned triple (no override): `gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker` + `maspsx --aspsx-version=2.56 --expand-div`. **Two byte-exact matches** confirm it across idiom classes: `func_800CF854` (the `lbu`+`sltu $zero` unsigned-≠0 accessor) and `func_800CEE84` (8-call init with branch-store `D_80127084=0; if(D_800B9A00) =1` + a delay-slot-filled `func_80010AE0(0x3E0)` + early-scheduled load). **Clean rebuild (R22): resident `8e17e02f`, main `143dbb89` — both byte-identical.** `make report BINARY=resident` → REAL 0→**2**. The "resident is 4.7" was a coincidental DsMix signature; the actual compiler == the EXE's (R24 provenance: resident triple = main triple, byte-confirmed). No candidate ladder; STOP-cond did not fire.
- **T3 — Resident harvest (IN PROGRESS; ULTRACODE workflow harvest).**
  - **Batch 1** (hand-matched): 6 trivial accessors. REAL 2→8.
  - **Batch 2** (workflow): `resident-harvest-draft` (16 parallel agents) drafted matching C for all 135 unmatched fns (asm + cookbook + worked examples → `.run/drafts/<fn>.c`, no builds/no Ghidra; 14/16 agents ok, 2 hit API-500); `tools/harvest_verify.py` (new) byte-gated each draft vs `8e17e02f` (chunk+bisect, self-correcting). **62/127 verified byte-identical in one pass.** Cleaned 35 redundant `#include`. **REAL 8→70 (48.28%); byte-identical 49.66%.** Clean rebuild green both binaries. Commit `c37332fa1`.
  - **Batch 3** (`c37332fa1`→ this): two redraft workflow passes (`resident-harvest-redraft` + retry-resilient `resident-harvest-redraft3`, both seeded by prior drafts + a debugging checklist; the first hit transient server rate-limiting → 8 landed, the retry-wave second pass recovered all 13 batches → +9). Byte-gated via `tools/harvest_verify.py` (drafts2 then drafts3). **REAL 70→78→87 (60.0%); byte-identical 61.38%.** Both binaries clean-rebuild green.
  - **Batches 3b–5** (`8dbaa489f`→`393d34604`): two more workflow passes — retry-resilient redraft (+9→REAL 87), then `tools/match_one.py` (new: per-function isolated compile + relocation-masked diff vs the `.s`) enabling an **iteration loop** (agents compile→read diff→fix), +13→REAL 102. Plus a typedef-strip fix (some drafts inlined `typedef ... u8;` → C89 redefinition compile-fail, not a byte miss; +2).
  - **T3a — extern-conflict recovery (`a480d83be`): +20 → REAL 123.** The pass-4 functions that match_one matched standalone but the single-TU build rejected on conflicting shared-symbol extern types were recovered by unifying ~20 symbol types in `resident.c` (widening definitions' return types where byte-identical — `u16 f(){return u16g}` ≡ `s32 f(){…}`; asm-correct widths for globals; per-site `*(u16*)&g` where a neighbor pins the type). 20/21; `func_800D00E4` stays an honest stub (its switch needs the external `jtbl_80113ED8` relocation — analysis preserved under `#if 0`).
  - **HARVEST RESULT (this session): REAL 1 → 123 / 146 = 85.62% byte-identical**, all hand-matched engine C, via 5 ultracode workflow passes + a deterministic byte-gate + the extern recovery. Both binaries byte-identical (resident `8e17e02f`, main `143dbb89`). New reusable tooling: `harvest_verify.py`, `match_one.py` (→ Phase-13 overlays).
  - **T3b — REASSESSED & deferred (Max, 2026-06-16).** Intra-resident **dedup collapse**: all 12 members of the 6 h_exact (and the 9 h_norm) groups are now *individually matched byte-identical* → the collapse has **no recovery/count value**, only source-DRY elegance; the §11 share mechanism is already proven (Phase 11), and the real dedup leverage is cross-binary/overlay (Phase 13). Deferred as optional elegance. **Permuter residuals**: the ~21 remaining stubs are the documented compiler-internal tail (cross-jump/block-reorder/regalloc — `func_800D06E8` 257→118, `func_800D128C` cross-jump, etc.) — **honest P9 deferrals**, each documented in its draft header; decomp-permuter is deliberate per-function work, not a blind sweep. **T3 substantively COMPLETE at 85.62%** ("substantial not exhaustive", Drew's bar — crushed).
  - **Cookbook-worthy idioms found** (folded into cookbook §12 at the ultracode write-up; R16): struct-copy prologue (local = global table), void-dispatcher-is-actually-s32-returning, `*10`→`2x+8x` decomposition, masked-compare `andi` survives only when the load's range is unprovable, callee-return-type-forces-the-cast.
- **T4–T6 (revised):** pending.

## Blockers / decisions
- **RESOLVED (P5d — plan task removal, Drew-approved 2026-06-16).** T1's nil-footprint finding (verified by psyq_identify + Ghidra) vacated the library-integration tasks. Drew confirmed **full substantial harvest**. Revised plan locked above (T1 done; T2 fingerprint → T3 harvest → T4 docs → T5 EXE save → T6 close). Milestone measure shifts from "LINKED 0→N" to **REAL 0→N** (hand-matched engine C). The roadmap's Phase-12 milestone/criterion #2 amendment (T6) will also note the "link the 4.7 libs" opener was mooted by the bytes.
- _(no open blockers)_

## Stop-conditions (P5)
- T2 triple differs materially → consult Drew before a phase-consuming candidate ladder.
- Any system too entangled → honest NON_MATCHING/documented determination (P9), never forced/fake.
- R23: stop Ghidra MCP before any RE-checkpoint/phase-end commit; `db.*.gbf` rename churn = noise.