BFM-decomp/phase-ends/PhaseEnd_Phase12.md

# PhaseEnd — Phase 12: Key engine systems (resident harvest, script/event, save, SQV)
**Date:** 2026-06-16 · **Project Version:** 1.12.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (5th phase)

> Gen2 phase 5 of the arc (8→9→10→11→**12**→{13}→14). The granular per-task trail (the 5 workflow passes,
> the gate logs, the extern-conflict diagnostics, the save trace) is preserved on-demand at
> **`phase-ends/logs/Phase12.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism).
> Owner decisions (2026-06-16): milestone = **"substantial + honest deferral"** (criterion #2 amended); planned
> guided save trace; **full substantial resident harvest** (after T1 found no library to link); save deep-data =
> defer (Gen3); Ultracode used for the harvest (Drew enabled it).

## Build Log

**Files created/changed and complete — do not recreate:**

*New reusable tooling (committed — feeds Phase 13 directly):*
- `tools/harvest_verify.py` — **the byte-gate.** Substitutes each drafted `.c` for its `INCLUDE_ASM` stub,
  `make build BINARY=<bin>`, keeps it ONLY if the image stays byte-identical, else reverts (chunk + bisection).
  The build is the sole arbiter (G3/P9) — a wrong match cannot be accepted. Strips inline scalar-typedef
  redefinitions (C89 dup-error ≠ byte miss). Binary-parameterized (`--binary/--src/--asm-subdir/--good-sha`).
- `tools/match_one.py` — **per-function isolated compile + relocation-masked diff** vs the target `.s` (jal/HI16/
  LO16 mask, the `psyq_identify` logic). Own temp dir → parallel-safe → gives agents a real asm-differ loop
  without a shared build. gcc-2.7.2 -O2 compiles each global fn independently, so a standalone match == the
  whole-binary byte-gate for that fn (modulo TU-level extern conflicts — see Deviations).

*Config / source (the harvest):*
- `src/resident/resident.c` — **REAL 1 → 123** matched engine functions (was 100% INCLUDE_ASM at Phase-10 close).
  All hand-written C, each byte-gated. `func_800D00E4` is an honest `INCLUDE_ASM` stub (needs the external
  `jtbl_80113ED8` relocation; its analysis preserved under `#if 0`). 22 stubs remain (documented residuals).
- `config/symbols.resident.txt` — unchanged (count=1, `DsMix`): the harvest matched bytes; meaningful per-fn
  naming beyond `func_*` is a future refinement (the C bodies + the dispatch-table determination document behavior).

*Docs / knowledge base (the determinations + the flywheel):*
- `docs/matching-cookbook.md` **§12** — "Ultracode harvest — parallel-draft + byte-gate at scale" (the full reusable
  pipeline + the retry-wave resilience + the two TU gotchas + the idioms the swarm surfaced). R16 flywheel.
- `docs/memory-map.md` — **script/event system RESOLVED** (compiled-MIPS state/mode dispatch via fn-pointer tables;
  NO bytecode VM — byte-backed by the matched `D_800D3430[D_800B99F0]()`-class dispatchers mirroring the EXE's
  `GameModeDispatch`). Resolves the roadmap's "Script VM instruction set" open question.
- `docs/formats.md` — **§5 SQV driver = semi-custom** (Square wrappers/glue over stock PsyQ libsnd; `DsMix` is a
  custom 2-line wrapper, the 4.7 sig was coincidental) + **§6 Save/memcard format** (`saveHeaderTemplate` layout:
  filename `BASLUS-00726MUSASHI`, SJIS title, the `0x8002Bxxx` handler dispatch table; the `bu00:BASLUS…` libmcrd
  path; deep save-data + the `SaveLoadRoutine` code-match honestly deferred).
- `docs/psyq-worklist.md` — the resident-has-NO-PsyQ-footprint correction (T1, byte-evidence + regen command).
- `docs/effort-map.md` — **§ "Proactively prompt for Ultracode on breadth-heavy stretches"** (R26).
- `docs/gen2-roadmap.md` — **Phase-12 milestone + EXIT criterion #2 AMENDED** (script/event *resolved*, not a
  matched VM; save documented + deferred; 85.6% "substantial").
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase12.md`** (R19 archive). `phase-ends/PhaseEnd_Phase12.md` — this file.

**Local artifacts (gitignored / regenerable):** `.run/drafts{,2,3,4}/*.c` (the workflow drafts), `.run/match/*`
(match_one scratch), `.run/harvest_*.log`, `.run/ram_save*.bin` (the save trace), `build/`, `expected/`, `asm/`.

**Tools/packages installed:** None — the whole phase used the existing Phase-4/6 toolchain + the venv (rabbitizer).

**Verification results (literal) — the milestone proof (clean dual-binary rebuild, all green):**
- **`make clean && make extract BINARY=resident && make build BINARY=resident`** → `build/resident/resident` SHA1
  `8e17e02ff8954d07c979449198f7e1645046b353` (BYTE-IDENTICAL) — **with 123 hand-matched engine functions linked**.
- **`make extract BINARY=main && make build BINARY=main`** → `143dbb89f34491258bbc27810d0a12ec8b43a8dd`
  (BYTE-IDENTICAL) — no regression from any of the phase's work.
- **`make report BINARY=resident`** → **REAL 123 / 0 NON_MATCHING / 146 matchable = 85.62% byte-identical**
  (was 0 REAL / 1.4% at Phase-10 close). `make report BINARY=main` unchanged (54 REAL / 959 LINKED / 50.33%).
- **T1 (byte-survey + Ghidra):** the resident embeds **no PsyQ library object of either version** (4.7 libsnd 1/226
  / libgte 0/509 / libspu 0/134; 4.0 likewise ~0; the few hits are ≤8-ins coincidences); tool sanity 4.0 libsnd
  vs the EXE = 35/163. Ghidra-confirmed (DsMix = custom wrapper; 61 distinct EXE-range `jal` calls).
- **T2 (fingerprint):** resident game-code triple == the EXE's (`gcc-2.7.2-psx -O2 -G0 …` + `maspsx 2.56 --expand-div`),
  opt level -O2 (0 frame-ptr sigs), proven by 2 byte-exact matches across idiom classes.
- **G4 preserved:** 0 NON_MATCHING in either default build. `git status`: only `config/`/`tools/`/`src/`/`docs/`/
  `phase-ends/` tracked; zero ROM-derived/generated bulk staged. **No Ghidra DB change** (the harvest is in committed
  C/asm; pass-4 agents used read-only `get_code`) — R23 no-op (the `db.*.gbf` rename churn is not committed).

**Milestone achieved (confirmed by Drew, gate 2):** the always-resident engine blob is **matched substantively from
C source — REAL 1 → 123/146 (85.62% byte-identical)** with 0 NON_MATCHING and the EXE still `143dbb89…`; the
**script/event system is resolved** (compiled-MIPS dispatch, no bytecode VM — byte-backed), and the **SQV + save
formats are documented**. The amended "substantial + honest deferral" bar (criterion #2) is crushed.

**Next:** **Phase 13 — Representative location overlays (end-to-end + fleet runbook).** Start a fresh session
(effort **Max**, plan mode). The two tools built this phase (`harvest_verify.py` + `match_one.py`) + the cross-binary
dedup report (the 134-overlay queue, all sharing slot `0x80128158` + the now-mostly-matched resident engine they
call) feed directly into the overlay harvest — and **that 134-overlay harvest is the canonical Ultracode breadth
task** (R26): prompt to enable it.

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Opener (Phase-11 close + R24) | "link the resident's PsyQ **4.7** libs first" | **VACATED** — T1 byte-survey found **nil** library footprint | The resident is all custom engine code calling the EXE's resident SDK; nothing to link (verified by psyq_identify on 4.0 AND 4.7, + Ghidra). The DetectPsyQ 4.7 was one coincidental DsMix signature. |
| Library-integration tasks (T3/T4/T5 of the plan) | one byte-gated task per resident library | **removed** (P5d, Drew-confirmed) | No footprint → nothing to integrate; progress became REAL (hand-matched), not LINKED. |
| Script VM | "match the interpreter core + first handlers" | **resolved as a written determination — NO bytecode VM** | The `VM_*` symbols are PsyQ libsnd (T1); the matched dispatchers prove "scripting" is compiled-MIPS fn-ptr-table dispatch. P9-honest; criterion #2 amended (Drew-approved). |
| Save/Q#5 | "resolve `SaveLoadRoutine` + match" | **format documented; deep data + the 1139-ins code-match honestly deferred** | The save-data serialization is Gen3-repack territory (not needed for the byte-match); `SaveLoadRoutine` stays the Gen1-precedent hard-defer (P9). The guided save trace confirmed the libmcrd path + directory but the running game churns the data-buffer diff. |
| Harvest method | hand-match / decomp-permuter | **Ultracode: 5 parallel-draft + byte-gate workflow passes** (62+8+9+2+13) + a 20-fn extern-conflict recovery | Drew enabled Ultracode; the swarm + the incorruptible byte-gate produced 79 matches blind + 13 via the match_one iteration loop; the recovery unified ~25 shared-symbol extern types. Far faster than serial. → cookbook §12, R26. |
| Dedup collapse (T3 plan item) | collapse the 6 h_exact / 9 h_norm intra-resident groups | **reassessed → deferred** | All group members got individually matched byte-identical during the harvest → collapse has no recovery/count value (only source-DRY elegance); the §11 share mechanism is already proven; real leverage is Phase-13 overlays. |
| Effort | Max throughout | Max → **Ultracode** (harvest) → Max (the rest) | Breadth-shaped harvest under Ultracode; deep single-thread work (T1 pivot, T2 fingerprint, T4/T5 determinations, this PhaseEnd) at Max. Codified as R26. |

## Commit Message
```
(Phase 12 landed as ~17 per-task/per-batch checkpoint commits commit:0068..commit:0082 + this PhaseEnd.)

feat(phase-12): key engine systems — resident harvest 1.4%->85.6% + script/save/SQV resolved (v1.12.0)

- T1 (byte-survey + Ghidra): the resident has NO PsyQ library footprint -> all custom engine
  code; library-link tasks vacated, progress is REAL not LINKED (R24 premise mooted)
- T2: resident game-code triple == the EXE's (gcc-2.7.2 -O2 + maspsx 2.56), 2 byte-exact proofs
- T3: ULTRACODE harvest -> REAL 1->123/146 (85.62% byte-identical). 5 parallel-draft + byte-gate
  workflow passes (tools/harvest_verify.py) + a match_one.py iteration loop + a 20-fn extern-
  conflict recovery. New reusable tools: harvest_verify.py, match_one.py (-> Phase 13)
- T4: script/event system RESOLVED (compiled-MIPS fn-ptr-table dispatch, NO bytecode VM -
  byte-backed by the matched dispatchers); SQV driver = semi-custom (memory-map + formats.md)
- T5: save/memcard format documented (saveHeaderTemplate + handler table + bu00:BASLUS libmcrd
  path, runtime-confirmed); deep save-data + SaveLoadRoutine code-match honestly deferred (P9)
- both binaries byte-identical (resident 8e17e02f, main 143dbb89); 0 NON_MATCHING linked (G4)
- cookbook §12 (the ultracode harvest pattern), effort-map + R26 (prompt for ultracode on
  breadth), gen2-roadmap milestone/criterion-#2 amended (Drew-approved gate 2)
- rule R26 (ultracode-on-breadth prompting); bumps 1.11.0 -> 1.12.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **R26 — Prompt for Ultracode on breadth-heavy stretches during a Max session.** When a breadth-shaped, parallelizable sub-task appears (the *same analysis across many independent items* — bulk function matching / a whole-binary harvest, an EXE- or overlay-fleet-wide audit / survey / dedup), **proactively prompt Drew to enable `/effort ultracode`** rather than grinding it serially or settling for one one-off Workflow. Keep the deep single-thread tasks at **Max** (Ultracode caps depth at xHigh): phase planning, the compiler fingerprint, US-address derivation, non-obvious debugging, PhaseEnd synthesis. The "is this actually breadth, or mine-to-author-with-full-context?" call is itself a Max judgment. Governing doc: `docs/effort-map.md`; memory `effort-prompt-ultracode-on-breadth`. | Drew (2026-06-16): Ultracode + the parallel-draft/byte-gate harvest took the resident **1.4%→85.6% in one session** — Max-serial would have been ~10× slower, and the compounding across passes only happens if Ultracode is actually enabled. (The harvest *pattern* itself is captured as cookbook §12, per the Phase-8/11 precedent that techniques live in the cookbook, not as rules.) |

## PhaseEnd Changelog
**v1.11.0 → v1.12.0 — Phase 12 complete (Gen2 phase 5).** The always-resident engine blob is driven from 100%
INCLUDE_ASM to **REAL 123/146 = 85.62% byte-identical** — all hand-matched engine C, both binaries byte-identical,
0 NON_MATCHING. The phase opened with a byte-finding that reshaped it (Drew-approved): the resident embeds **no PsyQ
library** (T1) and uses **the EXE's compiler** (T2), so the planned 4.7-library-link tasks were vacated and the work
became a pure **REAL harvest**. That harvest was done under **Ultracode** as a reusable pattern (cookbook §12): a
swarm of agents drafts matching C and an **incorruptible byte-gate** (`tools/harvest_verify.py`) accepts only
byte-perfect output — 5 passes (62+8+9+2+13) plus a `tools/match_one.py` per-function iteration loop and a 20-function
extern-conflict recovery. Two long-standing questions were **resolved by the bytes**: there is **no bytecode script
VM** (it is compiled-MIPS state/mode dispatch via function-pointer tables, byte-backed by the matched dispatchers),
and the **save/memcard format** is documented (header + handler table + libmcrd path), with the deep save-data
serialization and `SaveLoadRoutine`'s 1139-ins code-match **honestly deferred** (Gen3 / Gen1-precedent, P9). The
milestone + EXIT criterion #2 were **amended** to "script/event system *resolved*; save documented + deferred"
(Drew-approved). One rule (**R26**, prompt-for-Ultracode-on-breadth); the harvest pattern + two reusable tools feed
directly into Phase 13's 134-overlay harvest.

## Plain-English Recap
The game's permanent "engine" went from essentially nothing decompiled to **86% rebuilt, bit-for-bit, from our own
source code** — in a single session. Two early checks (which Drew approved) saved a lot of wasted effort: the engine
contains none of Sony's prebuilt code, and it's built with the exact same compiler as the main program. With those
settled, we ran a swarm of AI agents that *draft* C, paired with an automatic bit-for-bit referee that rejects
anything wrong — so nothing incorrect can slip in — and looped it five times, each pass smarter than the last (the
final one let each agent test-compile and self-correct). Along the way we settled two long-standing mysteries with
hard evidence: the game has **no script "virtual machine"** — its scripting is ordinary compiled code steered by
lookup tables, which we proved by rebuilding the steering functions themselves; and we mapped the **save-file
header** (the memory-card name, the on-screen "BRAVE FENCER MUSASHI" title, and the table of save/load handlers).
The handful of functions left are the genuinely compiler-stubborn cases, documented exactly how to finish. The
biggest lasting win is the *method* — swarm-of-agents-with-a-bit-for-bit-referee — which we wrote down as a permanent
recipe and two reusable tools, because the next phase (the game's 134 levels) is the same shape and will benefit
enormously. Both the engine and the main program still rebuild perfectly.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase12.md` (R19). The Phase-12 work is committed
(~17 checkpoints `commit:0068`..`commit:0082` + this PhaseEnd commit, which also lands the gen2-roadmap amendment); **Drew
pushes** (R6). Gen2 continues — do **NOT** start Phase 13 here. Start a **fresh session** (effort **Max**, plan mode)
for **Phase 13 — Representative location overlays**; the 134-overlay harvest is the canonical Ultracode breadth task
(R26 — prompt to enable it). Keep this file forever.