BFM-decomp/phase-ends/logs/Phase12.md

CURRENT_PHASE — Phase 12: Key engine systems (script VM, save/Q#5, sound SQV)

Generation: Gen2 (5th phase, arc 8→9→10→11→12→{13}→14) · Effort: Max (Tier-1 phase) · Status: APPROVED 2026-06-16, in progress Plan gate (P3): approved by Drew 2026-06-16. This is the per-task crash-recovery log (P4); archived to phase-ends/logs/Phase12.md at close (R19).

Owner decisions locked (2026-06-16)

1. Milestone = "Substantial + honest deferral." Amend gen2-roadmap EXIT criterion #2 / Phase-12 milestone: "script-VM core matched" → "script/event system resolved (core matched if a tractable interpreter exists, else its architecture documented)." Honest documented/partial outcomes count for the VM + the entangled save remainder (P9). (gen2-roadmap.md is the evolvable docs layer — amendment in scope, done in T10.)
2. Runtime = planned guided trace sessions. T8 (VM) + T9 (Save) include scheduled PCSX-Redux trace sessions: Drew drives gameplay; Claude captures RAM via the web API 172.17.208.1:8081 (R11) + correlates against Ghidra.

Verified starting state (byte-checked during planning — R14)

• main = EXE 143dbb89… (PsyQ 4.0): 54 REAL / 959 LINKED / 7 NON_MATCHING / 50.3%.
• resident = MAIN.CD/FILE_010/1.1 @ vram 0x800CEDF8 8e17e02f… (PsyQ 4.7): 0 REAL / 0 LINKED / 143 stubs / 1.4%. Code = ~17.5 KB only (143 funcs, code region 0x800CEDFC..0x800D3408); rest data tail. Harvest queue: 14 trivial + 50 non-jtbl leaves + 5 indirect-jump funcs.
• Resident game-code triple UNKNOWN (4.7 ⇒ maybe newer ASPSX/cc1 than the EXE's gcc-2.7.2/aspsx-2.56) — biggest fork (T2).
• 4.7 SDK at tools/psyq/conv47/…/lib/ (55 .o + 23 .a + headers) — build to ELF, then psyq_identify for the real footprint.
• Save: SaveLoadRoutine @ 0x8002B154 in EXE (1139-ins multi-entry, Gen1-deferred). VM: unlocated (VM_* are libsnd, not a game VM; overlays may be compiled MIPS). SQV: resident DsMix @ 0x800D1BD8 (UNVERIFIED) + custom glue.
• Resident in 0 cross-binary dedup groups (overlays call, don't embed) — match value = named symbol, not fleet credit. 6 h_exact / 9 h_norm intra-resident self-dup groups to collapse.
• EXE leftover VM_*/EVENT_OBJ stubs = Phase-8 scattered-.bss residue → out of scope.

Task checklist (one at a time, byte-gated) — REVISED post-T1 (no resident library footprint; Drew-approved 2026-06-16)

T1 found NIL library footprint → the resident is all game code. Library-integration tasks REMOVED. Harvest = full substantial (Drew). The old VM "investigation" reframes as compiled-MIPS state/mode dispatch, matched within the harvest + documented.

Block A — Resident enablement

• T1 — 4.7 lib footprint survey → NIL (resident is all game code; verified both oracles). Old T3/T4/T5 library-integration REMOVED.
• T2 — Resident compiler-fingerprint probe → triple CONFIRMED = the EXE's (gcc-2.7.2-psx -O2 -G0 … + maspsx --aspsx-version=2.56 --expand-div); opt level -O2 (0 frame-ptr sigs); 2 byte-exact matches across idiom classes. No ladder; STOP-cond didn't fire.

Block B — Resident engine harvest (the phase body; full substantial)

• T3 — Harvest the resident engine in difficulty/cluster batches ← CURRENT (each byte-gated make check BINARY=resident + committed; bank don't strand): trivial accessors → leaves → mid-size orchestration/state-dispatch → larger/branch-heavy + the 5 indirect-jump funcs. Collapse the 6 h_exact / 9 h_norm intra-resident dup groups (dedup.us.yaml + src/shared/). Includes the sound cluster (DsMix region) + the mode/state-dispatch "script/event" functions. Names → symbols.resident.txt + Ghidra (R15/G6). [xHigh; Max for harder clusters]

Block C — Documentation + EXE save system

• T4 — Document the SQV (sound) format + the script/event-dispatch architecture determination (compiled-MIPS dispatch, NOT a bytecode VM — P9) in docs/formats.md + memory-map.md. (Matching happens in T3; this is the written deliverable.) [Max]
• T5 — Save/Q#5 (EXE main) RE + format doc + tractable matches (guided PCSX-Redux trace, Drew-operated). Gate: make check BINARY=main = 143dbb89; honest NON_MATCHING for the entangled remainder (G4). [Max]

Block D — Close-out

• T6 — Cookbook (R16) + finalize symbols.resident.txt mirrored to Ghidra + ghidra_mcp_stop/verify (R20/R23) + regenerate reports (both binaries) + amend gen2-roadmap criterion #2 + PhaseEnd_Phase12 (Tier-1). [Max]

Current task pointer

T3 — Resident engine harvest (full substantial; the phase body). Triple confirmed (T2), harness proven. Harvest the ~141 remaining resident functions in difficulty/cluster batches (difficulty.resident.md order: 14 trivial → 50 leaves → mid-size orchestration/state-dispatch → larger/branch-heavy + 5 indirect-jump). Each function: write C (scaffold via decompile.py for the bigger ones; the small ones hand-write from asm) → make build BINARY=resident byte-gate. Commit per batch (R20). Collapse the 6 h_exact / 9 h_norm intra-resident dup groups (dedup.us.yaml + src/shared/). Name functions as understood → symbols.resident.txt + Ghidra (R15/G6). Includes the sound cluster (DsMix region) + the mode/state-dispatch "script/event" functions. Bank, don't exhaust. R22: clean-rebuild gate before each commit; both binaries green.

Per-task progress log

(append one entry per task as completed: what was done, the byte-gate result, the commit)

• T1 — 4.7 lib footprint survey (DONE; finding overturns the Block-A premise). Built the 4.7 ELF lib set (the conv47 objects are already ELF mipsel R3000 → skipped psyq-obj-parser; ar x each .a → .run/obj47/<lib>/). Ran psyq_identify --vram-base 0x800CEDF8 --exe …/FILE_010/1.1 over the resident for libsnd/libspu/libgte/libgpu/libcd/libetc/libapi/libc2/libmath/libmcrd/libds.
  • Result: NIL library footprint. Whole-file scan (code + data tail): 4.7 libsnd 1/226 (a 4-ins coincidence ut_rev_2.o), libspu 0/134, libgte 0/509, libgpu 0/61; libcd/libmath/libds each 1 hit, all ≤8 ins (two alias the same 8-ins DsMix addr). 4.0 cross-check: libsnd 2/163 (VM_DON.o 5 ins / VM_DOFF.o 4 ins), libspu 0/129, libgte 0/381. All hits are the §9.5 short-object coincidental class.
  • Tool sanity (confirms the negative is real): 4.0 libsnd vs the EXE snd region = 35/163 placed (the tool works + is version-sensitive); 4.7 libsnd vs EXE = 7 (short coincidences only). So the resident genuinely embeds no stock PsyQ library objects of either version.
  • Architectural conclusion: the PsyQ SDK lives in the EXE (959 LINKED); the resident is entirely custom engine code that calls the EXE's resident SDK + engine fns via fixed addresses (no RAM-wasting SDK duplication in an always-loaded blob). R24's "resident is 4.7 → link its 4.7 libs" is moot — the DetectPsyQ 4.7 signal was one coincidental DsMix-region pattern, not a linked footprint. DsMix @0x800D1BD8 is custom sound code, not a stock libsnd object (R13 tag effectively refuted).
  • Resident still 8e17e02f (nothing integrated). Footprint report to be committed with the revision (below).
  • Ghidra verification (Drew-requested, G1 — CONFIRMS the pivot): switched the headless MCP to the resident program (296 funcs); decompiled a sample. FUN_800cf854 = game-global accessor (return DAT_800ae6bf != 0); DsMix = {FUN_800d1bf8(); return 1;} (custom wrapper ignoring its arg — NOT stock libsnd; R13 tag refuted); FUN_800cee84 = engine init calling EXE REAL matches (func_80029254/func_8002AEF8) + GameModeDispatch-region fns; FUN_800d2318 = entity-heading math calling the EXE's libgte RATAN @0x8004CFEC. Call-target scan: 61 distinct EXE-range jal vs 37 internal — the resident calls the EXE's SDK/engine. Resident = game code, no linkable library. Verified by both oracles.
• T2 — Resident compiler-fingerprint (DONE; triple confirmed = EXE's). Opt level = -O2 (resident has 0 21f0a003 frame-ptr sigs vs the EXE's 54; the EXE's 800 game module is also -O2). The resident uses the Makefile's default pinned triple (no override): gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker + maspsx --aspsx-version=2.56 --expand-div. Two byte-exact matches confirm it across idiom classes: func_800CF854 (the lbu+sltu $zero unsigned-≠0 accessor) and func_800CEE84 (8-call init with branch-store D_80127084=0; if(D_800B9A00) =1 + a delay-slot-filled func_80010AE0(0x3E0) + early-scheduled load). Clean rebuild (R22): resident 8e17e02f, main 143dbb89 — both byte-identical. make report BINARY=resident → REAL 0→2. The "resident is 4.7" was a coincidental DsMix signature; the actual compiler == the EXE's (R24 provenance: resident triple = main triple, byte-confirmed). No candidate ladder; STOP-cond did not fire.
• T3 — Resident harvest (IN PROGRESS; ULTRACODE workflow harvest).
  • Batch 1 (hand-matched): 6 trivial accessors. REAL 2→8.
  • Batch 2 (workflow): resident-harvest-draft (16 parallel agents) drafted matching C for all 135 unmatched fns (asm + cookbook + worked examples → .run/drafts/<fn>.c, no builds/no Ghidra; 14/16 agents ok, 2 hit API-500); tools/harvest_verify.py (new) byte-gated each draft vs 8e17e02f (chunk+bisect, self-correcting). 62/127 verified byte-identical in one pass. Cleaned 35 redundant #include. REAL 8→70 (48.28%); byte-identical 49.66%. Clean rebuild green both binaries. Commit commit:0071.
  • Batch 3 (commit:0071→ this): two redraft workflow passes (resident-harvest-redraft + retry-resilient resident-harvest-redraft3, both seeded by prior drafts + a debugging checklist; the first hit transient server rate-limiting → 8 landed, the retry-wave second pass recovered all 13 batches → +9). Byte-gated via tools/harvest_verify.py (drafts2 then drafts3). REAL 70→78→87 (60.0%); byte-identical 61.38%. Both binaries clean-rebuild green.
  • Batches 3b–5 (commit:0073→commit:0075): two more workflow passes — retry-resilient redraft (+9→REAL 87), then tools/match_one.py (new: per-function isolated compile + relocation-masked diff vs the .s) enabling an iteration loop (agents compile→read diff→fix), +13→REAL 102. Plus a typedef-strip fix (some drafts inlined typedef ... u8; → C89 redefinition compile-fail, not a byte miss; +2).
  • T3a — extern-conflict recovery (commit:0078): +20 → REAL 123. The pass-4 functions that match_one matched standalone but the single-TU build rejected on conflicting shared-symbol extern types were recovered by unifying ~20 symbol types in resident.c (widening definitions' return types where byte-identical — u16 f(){return u16g} ≡ s32 f(){…}; asm-correct widths for globals; per-site *(u16*)&g where a neighbor pins the type). 20/21; func_800D00E4 stays an honest stub (its switch needs the external jtbl_80113ED8 relocation — analysis preserved under #if 0).
  • HARVEST RESULT (this session): REAL 1 → 123 / 146 = 85.62% byte-identical, all hand-matched engine C, via 5 ultracode workflow passes + a deterministic byte-gate + the extern recovery. Both binaries byte-identical (resident 8e17e02f, main 143dbb89). New reusable tooling: harvest_verify.py, match_one.py (→ Phase-13 overlays).
  • T3b — REASSESSED & deferred (Max, 2026-06-16). Intra-resident dedup collapse: all 12 members of the 6 h_exact (and the 9 h_norm) groups are now individually matched byte-identical → the collapse has no recovery/count value, only source-DRY elegance; the §11 share mechanism is already proven (Phase 11), and the real dedup leverage is cross-binary/overlay (Phase 13). Deferred as optional elegance. Permuter residuals: the ~21 remaining stubs are the documented compiler-internal tail (cross-jump/block-reorder/regalloc — func_800D06E8 257→118, func_800D128C cross-jump, etc.) — honest P9 deferrals, each documented in its draft header; decomp-permuter is deliberate per-function work, not a blind sweep. T3 substantively COMPLETE at 85.62% ("substantial not exhaustive", Drew's bar — crushed).
  • Cookbook-worthy idioms found (folded into cookbook §12 at the ultracode write-up; R16): struct-copy prologue (local = global table), void-dispatcher-is-actually-s32-returning, *10→2x+8x decomposition, masked-compare andi survives only when the load's range is unprovable, callee-return-type-forces-the-cast.
• T4–T6 (revised): pending.

Blockers / decisions

• RESOLVED (P5d — plan task removal, Drew-approved 2026-06-16). T1's nil-footprint finding (verified by psyq_identify + Ghidra) vacated the library-integration tasks. Drew confirmed full substantial harvest. Revised plan locked above (T1 done; T2 fingerprint → T3 harvest → T4 docs → T5 EXE save → T6 close). Milestone measure shifts from "LINKED 0→N" to REAL 0→N (hand-matched engine C). The roadmap's Phase-12 milestone/criterion #2 amendment (T6) will also note the "link the 4.7 libs" opener was mooted by the bytes.
• (no open blockers)

Stop-conditions (P5)

• T2 triple differs materially → consult Drew before a phase-consuming candidate ladder.
• Any system too entangled → honest NON_MATCHING/documented determination (P9), never forced/fake.
• R23: stop Ghidra MCP before any RE-checkpoint/phase-end commit; db.*.gbf rename churn = noise.