diff --git a/docs/SETUP.md b/docs/SETUP.md index 9955c581d..a86d217bf 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -238,6 +238,10 @@ The single clone lives at `~/bfm-decomp` (ext4). Builds, splat, asm-differ, Ghid ### §4.4 Copy the disc dump into the clone +> **As-built (P33 B1/B8):** after the copy, **`make disc-extract`** regenerates `extracted/` from `disks/` and verifies every +> file against the committed manifest (`disc-extract: OK`, 15.7 s; `PARTIAL` for a Track-1-only dump; the "P33 B1" section +> below has the flags and controls). `make extract-all` runs it once first; `make check-env` warns when the EXE is absent. + One-shot copy onto ext4 is fine (and required once): ```bash @@ -269,6 +273,11 @@ sudo apt-get update && sudo apt-get install -y \ ### §4.6 Python venv + splat + submodules +> **As-built (P33 B3/B8): `make bootstrap`** (`tools/bootstrap.sh`) does all of this idempotently on a fresh clone — apt +> presence check (prints the install line), the venv from `requirements-python.txt`, the submodules, the two cc1 tarballs +> sha256-checked and extracted, then `make check-env` — proven fresh-clone → 218/218 in 4 m 18 s (the "P33 B3" section). +> The manual steps below remain the reference for what it does. + ```bash cd ~/bfm-decomp python3 -m venv .venv # Python >= 3.12 required (24.04 ships 3.12; older = f-string SyntaxError mid-build) @@ -314,6 +323,11 @@ tar xzf gcc-2.7.2-cdk.tar.gz -C gcc-2.7.2-cdk ### §4.8 Optional: PsyQ 4.0/4.1 binaries for arbitration (via Wine) +> **As-built (P33 B4/B8):** the OPTIONAL Sony SDK *objects* that let `make check BINARY=main` link the real PsyQ libraries +> are obtained, sha256-verified and built by **`tools/fetch_psyq.sh`** (user-supplied 4.0 LIBs from the DTL-S2002 disc or +> `--from DIR`; the RTL 4.2 archive; `psyq-obj-parser`) — see the "P33 B4" section. Byte-identity never needs them +> (`make sdk-dual`). The Wine arbitration path below is the Phase-6 fingerprinting tool, unrelated to linking. + For byte-exact arbitration when maspsx output is in doubt, the **real** PsyQ Win32 tools can be driven from WSL under Wine (`sudo apt-get install -y wine`): - `https://github.com/mkst/esa/releases/download/psyq-binaries/psyq4.0.tar.gz` @@ -437,6 +451,11 @@ Modern cpp preprocesses → **vintage cc1** compiles to asm → **maspsx** emula ### §6.3 Planned make targets (Phase 5 builds these; names fixed now) +> **As-built at P33 (B8): `make help` is the live list.** Added since this table: `bootstrap`, `disc-extract`, `extract-all`, +> `check-all` (the R22 contract proof: `make clean && make extract-all && make check-all` → `check-all: 218 passed, 0 +> failed of 218`), `sdk-dual`, `report`, `tools-health`, the `sig-*` and `audit-*` families, `print-`. The public +> recipe with every expected last line is **`docs/verification.md`**. + | Target | Does | |---|---| | `make extract` | splat split per `config/splat.us.*.yaml` → `asm/`, linker scripts | @@ -895,6 +914,14 @@ This project lives in a **private** remote (rule H1, relaxed: ROM-derived materi - The two >100 MB raw PsyQ archives (the `psyq40usa.zip` + DTL-S2002 disc). - The unused PCSX-Redux Linux AppImage — the real runtime oracle is the Windows-native build. +**P33 update (B5/B8, 2026-09-06) — R20's new home after the public flip.** The Ghidra project, the RAM dumps, the PsyQ +SDK, the session archive and the extension zips leave git at C3 (`tools/public_rewrite/purge_set.txt`). What R20 backs up +INSTEAD: the RE work as text — **`config/ghidra/.jsonl` + `ROSTER.md`**, proven regenerable by +`tools/ghidra_rebuild.sh --proof` (all six PASS); the dumps' identity in `dumps/CHECKSUMS.sha1`; the SDK's +identity in `tools/psyq_CHECKSUMS.sha256` (+ `tools/fetch_psyq.sh`); the zips' sha256s (§2.3/§2.4). The one-time snapshot +of the binaries is the private archive repo `Druthulu/BFM-decomp-archive` (C4). **Never `git clean -x` in this tree** — +the purged paths become ignored files and a `-x` clean deletes the RE database (R20 amendment proposed at PhaseEnd_Phase33). + **Rules:** - **R20** — back up all irreplaceable RE/decomp work plus gathered hard-to-re-source tooling at per-session checkpoints. This **loosens R8** (which mandated a single commit at phase end): checkpoint commits are now expected within a phase. diff --git a/docs/verification.md b/docs/verification.md new file mode 100644 index 000000000..fc1377130 --- /dev/null +++ b/docs/verification.md @@ -0,0 +1,93 @@ +# Verify it yourself — the byte-identity contract, and how to check it with your own disc + +> BFM-decomp's claim is narrow and machine-checkable: **218 binaries** (the main executable `SLUS_007.26`, the resident +> engine blob, 138 location overlays and 78 code modules) **rebuild byte-for-byte from the C in `src/`**, compared against +> the originals extracted from a redump-verified USA disc. This page is the recipe, the expected output of every step, +> and the record of the last full run. Nothing ROM-derived is in the repository (`tools/audit_public.py` guards that), so +> every check below that touches game bytes needs YOUR disc dump. What CI proves without a disc is at the end. + +## 0. What you need + +| Item | Detail | +|---|---| +| The disc | *Brave Fencer Musashi (USA)* (SLUS-00726), redump layout: 4-track BIN/CUE. Track 1 SHA1 `b44f0f0a19936f23b26188b658e13201a6a9c211`, CRC32 `c238191b`. A Track-1-only dump also works (the 3 CD-audio tracks are then reported as unverified, never silently passed). | +| Linux (WSL2 Ubuntu 24.04 is what the project uses) | `binutils-mipsel-linux-gnu`, `cpp-mipsel-linux-gnu`, `python3.12`, `git`, `make` — `make bootstrap` prints the exact apt line and installs the rest (venv, submodules, the tracked gcc-2.7.2 `cc1` tarballs, sha256-checked). | +| Disk / time | ≈3 GB for the build tree; the full clean rebuild takes ≈5 min on 16 cores (measured 4 m 18 s on a fresh clone, single user 45 CPU-min). | + +The PsyQ SDK objects are **optional** and never needed for byte-identity: without them the main executable links the +project's own `INCLUDE_ASM` fallback tiles for Sony's library regions and is still byte-identical (this is the WITHOUT leg +of `make sdk-dual`, and it is what every fresh clone builds). See `docs/SETUP.md` §P33 B4 if you own the SDK. + +## 1. The recipe (R22: a CLEAN rebuild — never an incremental one) + +```bash +git clone --recurse-submodules https://github.com/Druthulu/BFM-decomp.git && cd BFM-decomp +make bootstrap # -> "check-env: OK" (one WARN: run make disc-extract) +mkdir -p disks && cp '/path/to/Brave Fencer Musashi (USA)'*.{bin,cue} disks/ +make disc-extract # -> "disc-extract: OK" (extracts the disc, compares 1,801 files to the committed manifest) +make clean && make -j"$(nproc)" extract-all && make -j"$(nproc)" check-all + # -> "extract-all: 217 extracted, 0 failed of 217 (+ main, serial)" + # -> "check-all: 218 passed, 0 failed of 218" <- THE contract line +make sdk-dual # optional, only with the SDK objects -> "sdk-dual: OK — main 143dbb89… byte-identical WITH and WITHOUT the PsyQ objects" +make tools-health # -> "tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green." +make report # -> the three 100% lines below, "INCLUDE_ASM stubs : 0" +``` + +Every `make` target exits non-zero on any failure; read the exit code, not the presence of an output file (a failed build +leaves the previous binary in place — rule R53). `make help` lists every target with one line each. + +### What each step proves + +| Step | Proves | Expected last line | +|---|---|---| +| `make bootstrap` | the toolchain is present and pinned (cc1 tarballs sha256 `500a459b…` / `42bb0df9…`, maspsx submodule at the pinned commit, binutils) | `check-env: OK` | +| `make disc-extract` | your disc is the redump image and extracts to the SAME 1,801 files the project was built against (`extracted/retail/manifest.jsonl` + `.sha1`, committed) | `disc-extract: OK` (or `PARTIAL: 1,798/1,798 code+data verified; 3 .DA unverified` for Track-1-only) | +| `make extract-all` | splat splits every binary from the extracted payloads (asm/ and the linker scripts are regenerated, never edited) | `extract-all: 217 extracted, 0 failed of 217 (+ main, serial)` | +| `make check-all` | **every binary's build SHA1 equals `config/check..sha`** — main `143dbb89f34491258bbc27810d0a12ec8b43a8dd` | `check-all: 218 passed, 0 failed of 218` | +| `make sdk-dual` | main is byte-identical both with Sony's real objects linked and with the fallback tiles | `sdk-dual: OK — …` | +| `make tools-health` | the two independent function-boundary oracles agree (0 phantom / 0 truncated / 0 pad-tail), the dedup registry validates (2,220 groups / 0 failures), the audits and derived indexes are fresh | `tools-health: OK — …` | +| `make report` | the three progress metrics, source-derived | `FLEET fn-count byte-ident: 363214 / 363214 = 100.00%` · `FLEET instr-weighted : 13492113 / 13492113 = 100.0%` · `FLEET distinct-code(uniq): 5820205 / 5820205 = 100.0%` · `MAIN game-code weighted : 45150 / 45150 = 100.0%` | + +### What is NOT our C (stated plainly) + +- **1,256 functions of the main executable are Sony's PsyQ library objects**, linked byte-identical from the SDK when you + have it and represented by assembly fallback tiles when you do not (`src/lib*.c`, `src/apicard*.c` — `INCLUDE_ASM` + stubs whose bytes come from the disc). Reimplementing Sony's libraries from source is out of scope. +- **5 functions fleet-wide are hand-written assembly in the original** (`config/verbatim_manifest.json`, PERMANENT rows); + they are kept as verbatim `__asm__` bodies, audited by `tools/verbatim_check.py --strict`. +- Everything else — every game-code function in all 218 binaries — is C. + +## 2. The last recorded run + +The project's own full run is recorded, log by log, under `.run/P33/verify/` (tracked: `NN_.log`, each ending in +`EXIT=` and a timestamp, plus `SUMMARY.md`), produced by `tools/verify_contract.sh` on the committed tree. The table +below is copied from that `SUMMARY.md`; regenerate it with the script, never by hand. + +| # | Step | Result | Log | +|---|---|---|---| +| — | *(pending: filled by the Phase-33 A5 run — see `phase-ends/CURRENT_PHASE.md`)* | | | + +The same run was performed after every banked batch of Phases 30–32 (218/218 at every one; the P32 close run is +`.run/P32/t4e/r22_check.log`) and the history-rewrite of the public flip is followed by one more (C8) on the adopted tree — +a content-preserving rewrite changes no tracked byte, and that run proves it. + +## 3. What CI proves without the disc + +`.github/workflows/no-rom.yml` runs on every push and pull request: + +- **audits** — no ROM-derived bytes, purge paths or >50 MiB files among the tracked files (`tools/audit_public.py`, whose + ROM-hash set is derived from the committed manifest + every `config/check.*.sha`); every source is text with portable + includes; the verbatim manifest has no drift; the derived indexes (cookbook, Ghidra roster) are fresh; the LZSS decoder's + unit tests; no stale symbol references. +- **compile-only** — every eligible translation unit (4,170 of 4,287; the rest are the library tiles that `.include` + disc-derived assembly) goes through cpp → gcc-2.7.2 `cc1` → maspsx → GNU `as` with the Makefile's exact flags. A PR + compiles four representative binaries (≈1 min); the whole fleet runs weekly and on demand. + +CI cannot compare bytes to the originals — that is your step 1. The two together are the full claim. + +## 4. Reverse-engineering artifacts you can also regenerate + +The Ghidra project is not in the repository (its database embeds the game's bytes). Its hand-authored content is tracked +as text under `config/ghidra/` (`ROSTER.md` lists the programs) and `tools/ghidra_rebuild.sh --proof` rebuilds a +program from your disc + the symbol files + that text and proves the result equals it (`PROOF PASS`). The 28 RAM images +the memory map was derived from are local-only; `dumps/CHECKSUMS.sha1` records their identity. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 69e1457b0..3e9c49304 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -58,7 +58,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo resident; roster; ExportSymbols R15 fix; path hardcodes; hooks) — Max (finished at medium, Drew's call) — see Log 2026-09-06 B5 - [x] **B6** `dumps/CHECKSUMS.sha1` + INDEX.md rewrite + memory-map Source-index row — Low/xHigh — see Log 2026-09-06 B6 - [x] **B7** No-ROM CI (`no-rom.yml`, `audit_public.py`, `compile_only.py`) — xHigh — see Log 2026-09-06 B7 -- [ ] **B8** SETUP.md rows/sections (R21) + `docs/verification.md` — xHigh +- [x] **B8** SETUP.md rows/sections (R21) + `docs/verification.md` — xHigh — see Log 2026-09-06 B8 - [ ] **A5** THE RECORDED RUN (`tools/verify_contract.sh` → `.run/P33/verify/`, SUMMARY all EXIT=0) — run Low, read Max - [ ] **B9/C3** The preparatory commit (`git rm --cached` purge set; psyq CHECKSUMS moved; zip sha256s; runbook; decision-log entry) — Max (P5c-class) @@ -210,21 +210,30 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact Gotcha, recorded: `pkill -f ''` from a shell whose own command line contains the pattern kills that shell (exit 144) — match on the child's distinctive argv or use `pgrep -f … | grep -v $$`. SETUP: P33 B7 section + 4 inventory rows (R21). Commit: see below. +- **2026-09-06 (S87) — B8 (SETUP cross-references + `docs/verification.md`).** Every P33 tool already had its SETUP section + + inventory row (written with its task, R21); B8 added the as-built pointers the plan named — §4.4 → `make disc-extract`, + §4.6 → `make bootstrap`, §4.8 → `tools/fetch_psyq.sh`, §6.3 → `make help` + the new targets + the R22 line — and the + "Backup & private-repo posture" P33 update (R20's new home: `config/ghidra/` + `--proof`, `dumps/CHECKSUMS.sha1`, + `tools/psyq_CHECKSUMS.sha256`, the archive repo; "never `git clean -x`"). NEW `docs/verification.md`: what you need + (redump SHA1/CRC, apt, disk/time), the R22 recipe with every expected last line (from `make help`, the P32/B3 logs and the + current digest: 363,214 / 13,492,113 / 5,820,205 / main 45,150), what is NOT our C (1,256 LINKED + 5 verbatim), the + last-recorded-run table (placeholder until A5 fills it from `SUMMARY.md`), what CI proves without the disc, the + regenerable RE artifacts. `tools/verify_contract.sh` is A5's (next). Commit: see below. -## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B7 ✓; NEXT = B8 (2026-09-06 ~22:00 MDT, written by session fa49faf3 "S87" after the B7 commit; SUPERSEDES the earlier blocks) +## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B8 ✓; NEXT = P6 rules check, then A5 THE RECORDED RUN (2026-09-06 ~22:20 MDT, written by session fa49faf3 "S87" after the B8 commit; SUPERSEDES the earlier blocks) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest -(R1–R73), then WAIT for Drew. **NEXT = B8** (xHigh), then the P6 rules check (12 tasks done), then A5. The harness task list must be REBUILT (Drew wants to monitor it — -one TaskCreate per plan item A1…G2, 40 items, mark A1–A4 + B1–B7 completed; R28). The SessionStart hook restarts the headless +(R1–R73), then WAIT for Drew. **NEXT = the P6 rules check (12 tasks done), then A5 THE RECORDED RUN** (run Low, read Max). The harness task list must be REBUILT (Drew wants to monitor it — +one TaskCreate per plan item A1…G2, 40 items, mark A1–A4 + B1–B8 completed; R28). The SessionStart hook restarts the headless MCP server when `ghidra/bfm.rep` exists (it did not stay up in S87 — `ss -tln` showed nothing on :8080; harmless): B6/B7/B8 need no Ghidra; run `tools/ghidra_mcp_stop.sh` before any headless step (R23). ### 1. Where we are **Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). R65–R73 ratified. **Done: A1 (`commit:4012`), A2 (`commit:4013`), A3 (`commit:4014`), A4 (`commit:4015`), B1 (`commit:4016`), B2 (`commit:4017`), B3 -(`commit:4018`), B4 (`commit:4019`), B5 (`commit:4022`), B6 (`commit:4023`), B7 (the S87 `feat(phase-33): B7 …` commit).** The approved plan is +(`commit:4018`), B4 (`commit:4019`), B5 (`commit:4022`), B6 (`commit:4023`), B7 (`commit:4024`), B8 (the S87 `docs(phase-33): B8 …` commit).** The approved plan is VERBATIM at the end of this file — Blocks A–G give every task's files, commands and verification; "Execution order and why" is the sequence. Effort: Drew ran S87 at **medium** by explicit choice (the plan says Max for B5); the plan's annotations still stand for the tasks ahead — restate them, Drew decides (R7/R27). @@ -250,22 +259,20 @@ still stand for the tasks ahead — restate them, Drew decides (R7/R27). ### 3. NEXT — in order 0. **Preflight:** `git status --short | grep -v ghidra/` (empty) · `git log -1 --format='%h %s'` · `df -h ~`. -1. **B8 — SETUP.md rows/sections (R21) + `docs/verification.md`** (xHigh): SETUP already carries P33 A2/A3/A4/B1/B2/B3/B4/B5/B7 - sections and inventory rows (written in the same change as each task); B8 = the plan's §4.4 disc-extract / §4.6 bootstrap / - §4.8 fetch_psyq / §6.3 new-targets cross-references (check each exists; add what is missing), the "Backup & private-repo - posture" paragraph pointing at `config/ghidra/` + `ghidra_rebuild.sh --proof` as R20's new home, and NEW - `docs/verification.md` — the public "verify it yourself" page: the commands (`make bootstrap` → `make disc-extract` → - `make extract-all` → `make check-all` → `make sdk-dual` [optional] → `make tools-health` → `make report`), each with its - expected last line, the R22 shape, the SHA1s (EXE `143dbb89…`, redump Track-1 `b44f0f0a…`), what CI proves vs what needs - the disc, and a placeholder table that A5's SUMMARY fills (A5 runs `tools/verify_contract.sh`, which B8 may write now or - A5 may — the plan puts the script under A5). Log, tick, refresh, commit. -2. **P6 rules check** (12 tasks done) → **A5 THE RECORDED RUN** (the plan's A5 paragraph: `tools/verify_contract.sh` → - `.run/P33/verify/NN_.log` + `SUMMARY.md`; allowlist `.run/P33/verify/` in `.gitignore` like `.run/P32/t4e/`; ≈1 h - for the clean fleet + sdk-dual + tools-health + audit-disc + report) → **B9/C3** (Max, P5c-class: the `git rm --cached` - commit — `git mv tools/psyq/CHECKSUMS.sha256` is ALREADY done (B4); the zip sha256s go into SETUP §2.3/§2.4; - `docs/public-flip-runbook.md`; decision-log entry). +1. **P6 rules check** (12 tasks done): re-read CLAUDE.md's rules + DIGEST §3, state "Rules check — re-read complete. Continuing + with A5." +2. **A5 — THE RECORDED RUN** (run Low, read Max; ≈1 h wall): write `tools/verify_contract.sh` per the plan's A5 paragraph — + steps 00 `git rev-parse HEAD` + porcelain (only the R23 `ghidra/` churn allowed) · 01 `make check-env` · 02 family_hseq + regen · 03 `make clean && make extract-all JOBS=16 && make check-all JOBS=16` · 04 `make sdk-dual` (keep both `.map`s; the + two from A3 are already in `.run/P33/verify/`) · 05 `make tools-health` (must contain `0 PHANTOM, 0 TRUNCATED, 0 PAD-TAIL`, + zero `[warn]`) · 06 `make audit-frontier` · 07 `make audit-disc` (4-track disc present in `disks/`; residue 0) · 08 + `make report` (three 100 lines, `INCLUDE_ASM stubs : 0`, `Open near-misses: 0`) · 09 SUMMARY.md; each log ends `EXIT=` + + timestamp; abort on the first non-zero (R53); `.gitignore` allowlists `.run/P33/verify/` (`*.log *.map *.md *.txt`) + exactly like `.run/P32/t4e/`; then paste SUMMARY's table into `docs/verification.md` §2 (replace the placeholder row). + Runs on the COMMITTED tree (commit B8 first — done). Log, tick, refresh, commit (the logs are tracked). +3. **B9/C3** (Max, P5c-class) → C1 → C2 → … per the task list. ### 4. Files S87 touched -B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh` +B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh` (`.proof` markers; dies unless `failed=0`), `tools/ghidra_annotations_delta.py` (the three drift classes), new `tools/ghidra_roster.py`, `tools/ghidra_mcp_start.sh` (silent no-op guard), `.claude/settings.json` (relative hooks), `Makefile` (roster check in tools-health), `docs/SETUP.md` (P33 B5 section, 5 inventory rows, §2.8), `config/ghidra/*`