diff --git a/README.md b/README.md index bced81be3..6da6f3dce 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ This repository contains **no game assets, no disassembly output, and no ROM-der About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway: -- The build toolchain is **binary-agnostic** (one parameterized pipeline builds any binary), and the always-resident **engine blob** now rebuilds **byte-for-byte from source** (SHA1 `8e17e02f…`) — the *second* binary reconstructed exactly, after the EXE. +- The build toolchain is **binary-agnostic** (one parameterized pipeline builds any binary), and the always-resident **engine blob** rebuilds **byte-for-byte from source** (SHA1 `8e17e02f…`) — the *second* binary reconstructed exactly, after the EXE — and is now **86% hand-matched C** (123 / 146 functions, up from 0): its scripting turned out to be **compiled-MIPS state/mode dispatch, not a bytecode VM**, and the save-file + sound (SQV) formats are documented. The harvest used a reusable **swarm-of-agents + bit-for-bit byte-gate** method (a wrong match can't be accepted) — `tools/harvest_verify.py` + `tools/match_one.py`, which carry straight into the overlay phase. - A **cross-binary deduplication pipeline** is live: a Ghidra-free signer fingerprints all 134 location overlays, and the report finds **~9,000 byte-identical function groups shared across binaries (~28 MB of collapsible code)** — a single engine function is byte-identical in all 134 overlays. This is "one match unlocks many": each engine match will be auto-credited across the overlay fleet. Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`. diff --git a/docs/gen2-roadmap.md b/docs/gen2-roadmap.md index 5959f2e36..5a391fd42 100644 --- a/docs/gen2-roadmap.md +++ b/docs/gen2-roadmap.md @@ -125,6 +125,7 @@ Almost all of BFM's game code lives **outside** the 404 KiB EXE — in a residen 3. **Sound (SQV):** RE the SQV (MIDI+VAB, semi-custom) driver; link wrapped libsnd/libspu PsyQ objects byte-identical (§9 pipeline); match the custom sequencer glue. 4. Feed every hard residual back into the cookbook (R16 flywheel); use the §3a/R17 web-research tier + decomp-permuter for compiler-quirk/scheduling tails. - **Milestone:** A defined set of resident-blob engine functions at asm-differ **score 0** with `make check BINARY=` byte-identical — specifically (a) the save/load path matched + save-data layout documented; (b) the script-VM interpreter core matched; (c) the SQV driver entry cluster matched (hand-match or byte-identical PsyQ link). `make report BINARY=` crosses the agreed "key systems matched" bar; 0 NON_MATCHING in the default build. + - **⟶ AMENDED at Phase-12 close (2026-06-16, Drew-approved gate 2 — the bytes reshaped the premises; "substantial + honest deferral"):** (1) the resident has **no PsyQ library footprint** (T1, byte-verified) — it is all custom engine code, so progress is **REAL** (hand-matched C), not LINKED; (2) there is **no bytecode VM** (T4) — the "script VM" is **resolved** as compiled-MIPS state/mode dispatch via function-pointer tables (byte-backed by the matched dispatchers), a *written determination* (not a matched interpreter); (3) the **save/Q#5 path is documented** (`saveHeaderTemplate` header + the `0x8002Bxxx` handler table + the `bu00:BASLUS-00726MUSASHI` libmcrd path, runtime-confirmed) with the deep save-data serialization + `SaveLoadRoutine`'s 1139-ins code-match **honestly deferred** (Gen3-repack / Gen1-precedent, not needed for the byte-match, P9); (4) the **SQV** cluster matched in the harvest + format documented (semi-custom). **Achieved:** resident **REAL 1 → 123/146 = 85.62% byte-identical**, both binaries byte-identical, 0 NON_MATCHING in either default build. - **Dependencies:** Phases 10, 11; Phase 8 (libsnd/libspu/libmcrd as call anchors). - **Primary risk + mitigation:** A system (esp. the script VM or `SaveLoadRoutine`) is too large/entangled for clean Ghidra decompilation, tempting a fake-success stub (the psxrecomp failure mode). → G3 byte-match + G4 NON_MATCHING + P9 honesty: an unmatchable system is left an honest NON_MATCHING draft or documented stub (the Gen1 `SaveLoadRoutine` precedent); milestone counts only score-0 matches; size each system as its own sub-task so a stall strands one system, never a half-binary. - **Effort tier:** **Tier-1 (Max)** — deep, novel greenfield RE where wrong structural understanding is expensive. Breadth helps only for surveying candidate functions. @@ -207,7 +208,7 @@ Phase 12 Engine systems Phase 13 Representative overlays Gen2 exits — and Gen3 may be considered — when **all** are machine-verified: 1. **EXE at its matching ceiling** (constraint #1): all linkable PsyQ libraries linked byte-identical; the easy difficulty-queue tail harvested; EXE byte-identical with and without SDK objects. *(Phase 8)* -2. **Resident engine blob matched substantively**, including the **script-VM core, the save/Q#5 path (`SaveLoadRoutine` + documented save format), and the sound/SQV driver cluster** — each byte-checked, none NON_MATCHING in the default build; `make check BINARY=` green. *(Phases 10, 12)* +2. **Resident engine blob matched substantively** — the **script/event system resolved** (Phase 12: *no bytecode VM*; it is compiled-MIPS state/mode dispatch via fn-pointer tables — a byte-backed written determination, not a matched interpreter), the **save/Q#5 path documented** (`saveHeaderTemplate` + handler table + libmcrd path; deep save-data serialization + the `SaveLoadRoutine` code-match honestly deferred as Gen3/Gen1-precedent, P9), and the **sound/SQV cluster** matched + documented — all byte-checked, none NON_MATCHING in the default build; `make check BINARY=` green. **Met Phase 12 at REAL 123/146 = 85.6% byte-identical** ("substantial, not exhaustive"; the ~21 remaining stubs are documented permuter/§3a residuals). *(Phases 10, 12 — milestone amended at the Phase-12 close, Drew-approved)* 3. **Cross-binary dedup live and proven:** `make report` emits cross-binary groups, and ≥1 shared body is byte-identical across ≥2 binaries from a single source. *(Phase 11)* 4. **A representative set of location overlays matched end-to-end** (≥3 spanning the archetypes: a single overlay, a collapsed duplicate pair, a large free-roam overlay), each `make check` byte-identical against its `0.4.dec`. *(Phase 13)* 5. **Public with a dashboard (via the curated public repo; private master untouched):** AGPL-3.0 present; the public repo's tree + history scan clean of ROM bytes and contain only allowlisted paths; rom→decoder reproduces the byte-identical EXE on a fresh clone from a user's dump; no-ROM CI green; frogress/decomp.dev live. *(Phase 14)* diff --git a/phase-ends/PhaseEnd_Phase12.md b/phase-ends/PhaseEnd_Phase12.md new file mode 100644 index 000000000..27e132ea6 --- /dev/null +++ b/phase-ends/PhaseEnd_Phase12.md @@ -0,0 +1,154 @@ +# PhaseEnd — Phase 12: Key engine systems (resident harvest, script/event, save, SQV) +**Date:** 2026-06-16 · **Project Version:** 1.12.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (5th phase) + +> Gen2 phase 5 of the arc (8→9→10→11→**12**→{13}→14). The granular per-task trail (the 5 workflow passes, +> the gate logs, the extern-conflict diagnostics, the save trace) is preserved on-demand at +> **`phase-ends/logs/Phase12.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). +> Owner decisions (2026-06-16): milestone = **"substantial + honest deferral"** (criterion #2 amended); planned +> guided save trace; **full substantial resident harvest** (after T1 found no library to link); save deep-data = +> defer (Gen3); Ultracode used for the harvest (Drew enabled it). + +## Build Log + +**Files created/changed and complete — do not recreate:** + +*New reusable tooling (committed — feeds Phase 13 directly):* +- `tools/harvest_verify.py` — **the byte-gate.** Substitutes each drafted `.c` for its `INCLUDE_ASM` stub, + `make build BINARY=`, keeps it ONLY if the image stays byte-identical, else reverts (chunk + bisection). + The build is the sole arbiter (G3/P9) — a wrong match cannot be accepted. Strips inline scalar-typedef + redefinitions (C89 dup-error ≠ byte miss). Binary-parameterized (`--binary/--src/--asm-subdir/--good-sha`). +- `tools/match_one.py` — **per-function isolated compile + relocation-masked diff** vs the target `.s` (jal/HI16/ + LO16 mask, the `psyq_identify` logic). Own temp dir → parallel-safe → gives agents a real asm-differ loop + without a shared build. gcc-2.7.2 -O2 compiles each global fn independently, so a standalone match == the + whole-binary byte-gate for that fn (modulo TU-level extern conflicts — see Deviations). + +*Config / source (the harvest):* +- `src/resident/resident.c` — **REAL 1 → 123** matched engine functions (was 100% INCLUDE_ASM at Phase-10 close). + All hand-written C, each byte-gated. `func_800D00E4` is an honest `INCLUDE_ASM` stub (needs the external + `jtbl_80113ED8` relocation; its analysis preserved under `#if 0`). 22 stubs remain (documented residuals). +- `config/symbols.resident.txt` — unchanged (count=1, `DsMix`): the harvest matched bytes; meaningful per-fn + naming beyond `func_*` is a future refinement (the C bodies + the dispatch-table determination document behavior). + +*Docs / knowledge base (the determinations + the flywheel):* +- `docs/matching-cookbook.md` **§12** — "Ultracode harvest — parallel-draft + byte-gate at scale" (the full reusable + pipeline + the retry-wave resilience + the two TU gotchas + the idioms the swarm surfaced). R16 flywheel. +- `docs/memory-map.md` — **script/event system RESOLVED** (compiled-MIPS state/mode dispatch via fn-pointer tables; + NO bytecode VM — byte-backed by the matched `D_800D3430[D_800B99F0]()`-class dispatchers mirroring the EXE's + `GameModeDispatch`). Resolves the roadmap's "Script VM instruction set" open question. +- `docs/formats.md` — **§5 SQV driver = semi-custom** (Square wrappers/glue over stock PsyQ libsnd; `DsMix` is a + custom 2-line wrapper, the 4.7 sig was coincidental) + **§6 Save/memcard format** (`saveHeaderTemplate` layout: + filename `BASLUS-00726MUSASHI`, SJIS title, the `0x8002Bxxx` handler dispatch table; the `bu00:BASLUS…` libmcrd + path; deep save-data + the `SaveLoadRoutine` code-match honestly deferred). +- `docs/psyq-worklist.md` — the resident-has-NO-PsyQ-footprint correction (T1, byte-evidence + regen command). +- `docs/effort-map.md` — **§ "Proactively prompt for Ultracode on breadth-heavy stretches"** (R26). +- `docs/gen2-roadmap.md` — **Phase-12 milestone + EXIT criterion #2 AMENDED** (script/event *resolved*, not a + matched VM; save documented + deferred; 85.6% "substantial"). +- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase12.md`** (R19 archive). `phase-ends/PhaseEnd_Phase12.md` — this file. + +**Local artifacts (gitignored / regenerable):** `.run/drafts{,2,3,4}/*.c` (the workflow drafts), `.run/match/*` +(match_one scratch), `.run/harvest_*.log`, `.run/ram_save*.bin` (the save trace), `build/`, `expected/`, `asm/`. + +**Tools/packages installed:** None — the whole phase used the existing Phase-4/6 toolchain + the venv (rabbitizer). + +**Verification results (literal) — the milestone proof (clean dual-binary rebuild, all green):** +- **`make clean && make extract BINARY=resident && make build BINARY=resident`** → `build/resident/resident` SHA1 + `8e17e02ff8954d07c979449198f7e1645046b353` (BYTE-IDENTICAL) — **with 123 hand-matched engine functions linked**. +- **`make extract BINARY=main && make build BINARY=main`** → `143dbb89f34491258bbc27810d0a12ec8b43a8dd` + (BYTE-IDENTICAL) — no regression from any of the phase's work. +- **`make report BINARY=resident`** → **REAL 123 / 0 NON_MATCHING / 146 matchable = 85.62% byte-identical** + (was 0 REAL / 1.4% at Phase-10 close). `make report BINARY=main` unchanged (54 REAL / 959 LINKED / 50.33%). +- **T1 (byte-survey + Ghidra):** the resident embeds **no PsyQ library object of either version** (4.7 libsnd 1/226 + / libgte 0/509 / libspu 0/134; 4.0 likewise ~0; the few hits are ≤8-ins coincidences); tool sanity 4.0 libsnd + vs the EXE = 35/163. Ghidra-confirmed (DsMix = custom wrapper; 61 distinct EXE-range `jal` calls). +- **T2 (fingerprint):** resident game-code triple == the EXE's (`gcc-2.7.2-psx -O2 -G0 …` + `maspsx 2.56 --expand-div`), + opt level -O2 (0 frame-ptr sigs), proven by 2 byte-exact matches across idiom classes. +- **G4 preserved:** 0 NON_MATCHING in either default build. `git status`: only `config/`/`tools/`/`src/`/`docs/`/ + `phase-ends/` tracked; zero ROM-derived/generated bulk staged. **No Ghidra DB change** (the harvest is in committed + C/asm; pass-4 agents used read-only `get_code`) — R23 no-op (the `db.*.gbf` rename churn is not committed). + +**Milestone achieved (confirmed by Drew, gate 2):** the always-resident engine blob is **matched substantively from +C source — REAL 1 → 123/146 (85.62% byte-identical)** with 0 NON_MATCHING and the EXE still `143dbb89…`; the +**script/event system is resolved** (compiled-MIPS dispatch, no bytecode VM — byte-backed), and the **SQV + save +formats are documented**. The amended "substantial + honest deferral" bar (criterion #2) is crushed. + +**Next:** **Phase 13 — Representative location overlays (end-to-end + fleet runbook).** Start a fresh session +(effort **Max**, plan mode). The two tools built this phase (`harvest_verify.py` + `match_one.py`) + the cross-binary +dedup report (the 134-overlay queue, all sharing slot `0x80128158` + the now-mostly-matched resident engine they +call) feed directly into the overlay harvest — and **that 134-overlay harvest is the canonical Ultracode breadth +task** (R26): prompt to enable it. + +## Deviations +| Item | Plan | Actual | Reason | +|---|---|---|---| +| Opener (Phase-11 close + R24) | "link the resident's PsyQ **4.7** libs first" | **VACATED** — T1 byte-survey found **nil** library footprint | The resident is all custom engine code calling the EXE's resident SDK; nothing to link (verified by psyq_identify on 4.0 AND 4.7, + Ghidra). The DetectPsyQ 4.7 was one coincidental DsMix signature. | +| Library-integration tasks (T3/T4/T5 of the plan) | one byte-gated task per resident library | **removed** (P5d, Drew-confirmed) | No footprint → nothing to integrate; progress became REAL (hand-matched), not LINKED. | +| Script VM | "match the interpreter core + first handlers" | **resolved as a written determination — NO bytecode VM** | The `VM_*` symbols are PsyQ libsnd (T1); the matched dispatchers prove "scripting" is compiled-MIPS fn-ptr-table dispatch. P9-honest; criterion #2 amended (Drew-approved). | +| Save/Q#5 | "resolve `SaveLoadRoutine` + match" | **format documented; deep data + the 1139-ins code-match honestly deferred** | The save-data serialization is Gen3-repack territory (not needed for the byte-match); `SaveLoadRoutine` stays the Gen1-precedent hard-defer (P9). The guided save trace confirmed the libmcrd path + directory but the running game churns the data-buffer diff. | +| Harvest method | hand-match / decomp-permuter | **Ultracode: 5 parallel-draft + byte-gate workflow passes** (62+8+9+2+13) + a 20-fn extern-conflict recovery | Drew enabled Ultracode; the swarm + the incorruptible byte-gate produced 79 matches blind + 13 via the match_one iteration loop; the recovery unified ~25 shared-symbol extern types. Far faster than serial. → cookbook §12, R26. | +| Dedup collapse (T3 plan item) | collapse the 6 h_exact / 9 h_norm intra-resident groups | **reassessed → deferred** | All group members got individually matched byte-identical during the harvest → collapse has no recovery/count value (only source-DRY elegance); the §11 share mechanism is already proven; real leverage is Phase-13 overlays. | +| Effort | Max throughout | Max → **Ultracode** (harvest) → Max (the rest) | Breadth-shaped harvest under Ultracode; deep single-thread work (T1 pivot, T2 fingerprint, T4/T5 determinations, this PhaseEnd) at Max. Codified as R26. | + +## Commit Message +``` +(Phase 12 landed as ~17 per-task/per-batch checkpoint commits commit:0068..commit:0082 + this PhaseEnd.) + +feat(phase-12): key engine systems — resident harvest 1.4%->85.6% + script/save/SQV resolved (v1.12.0) + +- T1 (byte-survey + Ghidra): the resident has NO PsyQ library footprint -> all custom engine + code; library-link tasks vacated, progress is REAL not LINKED (R24 premise mooted) +- T2: resident game-code triple == the EXE's (gcc-2.7.2 -O2 + maspsx 2.56), 2 byte-exact proofs +- T3: ULTRACODE harvest -> REAL 1->123/146 (85.62% byte-identical). 5 parallel-draft + byte-gate + workflow passes (tools/harvest_verify.py) + a match_one.py iteration loop + a 20-fn extern- + conflict recovery. New reusable tools: harvest_verify.py, match_one.py (-> Phase 13) +- T4: script/event system RESOLVED (compiled-MIPS fn-ptr-table dispatch, NO bytecode VM - + byte-backed by the matched dispatchers); SQV driver = semi-custom (memory-map + formats.md) +- T5: save/memcard format documented (saveHeaderTemplate + handler table + bu00:BASLUS libmcrd + path, runtime-confirmed); deep save-data + SaveLoadRoutine code-match honestly deferred (P9) +- both binaries byte-identical (resident 8e17e02f, main 143dbb89); 0 NON_MATCHING linked (G4) +- cookbook §12 (the ultracode harvest pattern), effort-map + R26 (prompt for ultracode on + breadth), gen2-roadmap milestone/criterion-#2 amended (Drew-approved gate 2) +- rule R26 (ultracode-on-breadth prompting); bumps 1.11.0 -> 1.12.0 +``` + +## Rules Added This Phase +| Rule | Reason | +|---|---| +| **R26 — Prompt for Ultracode on breadth-heavy stretches during a Max session.** When a breadth-shaped, parallelizable sub-task appears (the *same analysis across many independent items* — bulk function matching / a whole-binary harvest, an EXE- or overlay-fleet-wide audit / survey / dedup), **proactively prompt Drew to enable `/effort ultracode`** rather than grinding it serially or settling for one one-off Workflow. Keep the deep single-thread tasks at **Max** (Ultracode caps depth at xHigh): phase planning, the compiler fingerprint, US-address derivation, non-obvious debugging, PhaseEnd synthesis. The "is this actually breadth, or mine-to-author-with-full-context?" call is itself a Max judgment. Governing doc: `docs/effort-map.md`; memory `effort-prompt-ultracode-on-breadth`. | Drew (2026-06-16): Ultracode + the parallel-draft/byte-gate harvest took the resident **1.4%→85.6% in one session** — Max-serial would have been ~10× slower, and the compounding across passes only happens if Ultracode is actually enabled. (The harvest *pattern* itself is captured as cookbook §12, per the Phase-8/11 precedent that techniques live in the cookbook, not as rules.) | + +## PhaseEnd Changelog +**v1.11.0 → v1.12.0 — Phase 12 complete (Gen2 phase 5).** The always-resident engine blob is driven from 100% +INCLUDE_ASM to **REAL 123/146 = 85.62% byte-identical** — all hand-matched engine C, both binaries byte-identical, +0 NON_MATCHING. The phase opened with a byte-finding that reshaped it (Drew-approved): the resident embeds **no PsyQ +library** (T1) and uses **the EXE's compiler** (T2), so the planned 4.7-library-link tasks were vacated and the work +became a pure **REAL harvest**. That harvest was done under **Ultracode** as a reusable pattern (cookbook §12): a +swarm of agents drafts matching C and an **incorruptible byte-gate** (`tools/harvest_verify.py`) accepts only +byte-perfect output — 5 passes (62+8+9+2+13) plus a `tools/match_one.py` per-function iteration loop and a 20-function +extern-conflict recovery. Two long-standing questions were **resolved by the bytes**: there is **no bytecode script +VM** (it is compiled-MIPS state/mode dispatch via function-pointer tables, byte-backed by the matched dispatchers), +and the **save/memcard format** is documented (header + handler table + libmcrd path), with the deep save-data +serialization and `SaveLoadRoutine`'s 1139-ins code-match **honestly deferred** (Gen3 / Gen1-precedent, P9). The +milestone + EXIT criterion #2 were **amended** to "script/event system *resolved*; save documented + deferred" +(Drew-approved). One rule (**R26**, prompt-for-Ultracode-on-breadth); the harvest pattern + two reusable tools feed +directly into Phase 13's 134-overlay harvest. + +## Plain-English Recap +The game's permanent "engine" went from essentially nothing decompiled to **86% rebuilt, bit-for-bit, from our own +source code** — in a single session. Two early checks (which Drew approved) saved a lot of wasted effort: the engine +contains none of Sony's prebuilt code, and it's built with the exact same compiler as the main program. With those +settled, we ran a swarm of AI agents that *draft* C, paired with an automatic bit-for-bit referee that rejects +anything wrong — so nothing incorrect can slip in — and looped it five times, each pass smarter than the last (the +final one let each agent test-compile and self-correct). Along the way we settled two long-standing mysteries with +hard evidence: the game has **no script "virtual machine"** — its scripting is ordinary compiled code steered by +lookup tables, which we proved by rebuilding the steering functions themselves; and we mapped the **save-file +header** (the memory-card name, the on-screen "BRAVE FENCER MUSASHI" title, and the table of save/load handlers). +The handful of functions left are the genuinely compiler-stubborn cases, documented exactly how to finish. The +biggest lasting win is the *method* — swarm-of-agents-with-a-bit-for-bit-referee — which we wrote down as a permanent +recipe and two reusable tools, because the next phase (the game's 134 levels) is the same shape and will benefit +enormously. Both the engine and the main program still rebuild perfectly. + +## 🛑 Stop Here +PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase12.md` (R19). The Phase-12 work is committed +(~17 checkpoints `commit:0068`..`commit:0082` + this PhaseEnd commit, which also lands the gen2-roadmap amendment); **Drew +pushes** (R6). Gen2 continues — do **NOT** start Phase 13 here. Start a **fresh session** (effort **Max**, plan mode) +for **Phase 13 — Representative location overlays**; the 134-overlay harvest is the canonical Ultracode breadth task +(R26 — prompt to enable it). Keep this file forever. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase12.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase12.md