docs(phase-33): B8 — docs/verification.md (the public verify-it-yourself page: redump SHA1s, the R22 recipe with every expected last line, what is not our C, the recorded-run table pending A5, what CI proves) + SETUP as-built pointers (§4.4 disc-extract, §4.6 bootstrap, §4.8 fetch_psyq, §6.3 make help + the R22 line) + the backup-posture P33 update (R20's new home: config/ghidra + --proof; never git clean -x); checkpoint -> NEXT = P6 check, A5

This commit is contained in:
Drew T
2026-09-06 20:52:07 -06:00
parent 6732771fe0
commit b2fb392ada
3 changed files with 147 additions and 20 deletions
+27
View File
@@ -238,6 +238,10 @@ The single clone lives at `~/bfm-decomp` (ext4). Builds, splat, asm-differ, Ghid
### §4.4 Copy the disc dump into the clone
> **As-built (P33 B1/B8):** after the copy, **`make disc-extract`** regenerates `extracted/` from `disks/` and verifies every
> file against the committed manifest (`disc-extract: OK`, 15.7 s; `PARTIAL` for a Track-1-only dump; the "P33 B1" section
> below has the flags and controls). `make extract-all` runs it once first; `make check-env` warns when the EXE is absent.
One-shot copy onto ext4 is fine (and required once):
```bash
@@ -269,6 +273,11 @@ sudo apt-get update && sudo apt-get install -y \
### §4.6 Python venv + splat + submodules
> **As-built (P33 B3/B8): `make bootstrap`** (`tools/bootstrap.sh`) does all of this idempotently on a fresh clone — apt
> presence check (prints the install line), the venv from `requirements-python.txt`, the submodules, the two cc1 tarballs
> sha256-checked and extracted, then `make check-env` — proven fresh-clone → 218/218 in 4 m 18 s (the "P33 B3" section).
> The manual steps below remain the reference for what it does.
```bash
cd ~/bfm-decomp
python3 -m venv .venv # Python >= 3.12 required (24.04 ships 3.12; older = f-string SyntaxError mid-build)
@@ -314,6 +323,11 @@ tar xzf gcc-2.7.2-cdk.tar.gz -C gcc-2.7.2-cdk
### §4.8 Optional: PsyQ 4.0/4.1 binaries for arbitration (via Wine)
> **As-built (P33 B4/B8):** the OPTIONAL Sony SDK *objects* that let `make check BINARY=main` link the real PsyQ libraries
> are obtained, sha256-verified and built by **`tools/fetch_psyq.sh`** (user-supplied 4.0 LIBs from the DTL-S2002 disc or
> `--from DIR`; the RTL 4.2 archive; `psyq-obj-parser`) — see the "P33 B4" section. Byte-identity never needs them
> (`make sdk-dual`). The Wine arbitration path below is the Phase-6 fingerprinting tool, unrelated to linking.
For byte-exact arbitration when maspsx output is in doubt, the **real** PsyQ Win32 tools can be driven from WSL under Wine (`sudo apt-get install -y wine`):
- `https://github.com/mkst/esa/releases/download/psyq-binaries/psyq4.0.tar.gz`
@@ -437,6 +451,11 @@ Modern cpp preprocesses → **vintage cc1** compiles to asm → **maspsx** emula
### §6.3 Planned make targets (Phase 5 builds these; names fixed now)
> **As-built at P33 (B8): `make help` is the live list.** Added since this table: `bootstrap`, `disc-extract`, `extract-all`,
> `check-all` (the R22 contract proof: `make clean && make extract-all && make check-all` → `check-all: 218 passed, 0
> failed of 218`), `sdk-dual`, `report`, `tools-health`, the `sig-*` and `audit-*` families, `print-<VAR>`. The public
> recipe with every expected last line is **`docs/verification.md`**.
| Target | Does |
|---|---|
| `make extract` | splat split per `config/splat.us.*.yaml` → `asm/`, linker scripts |
@@ -895,6 +914,14 @@ This project lives in a **private** remote (rule H1, relaxed: ROM-derived materi
- The two >100 MB raw PsyQ archives (the `psyq40usa.zip` + DTL-S2002 disc).
- The unused PCSX-Redux Linux AppImage — the real runtime oracle is the Windows-native build.
**P33 update (B5/B8, 2026-09-06) — R20's new home after the public flip.** The Ghidra project, the RAM dumps, the PsyQ
SDK, the session archive and the extension zips leave git at C3 (`tools/public_rewrite/purge_set.txt`). What R20 backs up
INSTEAD: the RE work as text — **`config/ghidra/<program>.jsonl` + `ROSTER.md`**, proven regenerable by
`tools/ghidra_rebuild.sh <program> --proof` (all six PASS); the dumps' identity in `dumps/CHECKSUMS.sha1`; the SDK's
identity in `tools/psyq_CHECKSUMS.sha256` (+ `tools/fetch_psyq.sh`); the zips' sha256s (§2.3/§2.4). The one-time snapshot
of the binaries is the private archive repo `Druthulu/BFM-decomp-archive` (C4). **Never `git clean -x` in this tree** —
the purged paths become ignored files and a `-x` clean deletes the RE database (R20 amendment proposed at PhaseEnd_Phase33).
**Rules:**
- **R20** — back up all irreplaceable RE/decomp work plus gathered hard-to-re-source tooling at per-session checkpoints. This **loosens R8** (which mandated a single commit at phase end): checkpoint commits are now expected within a phase.
+93
View File
@@ -0,0 +1,93 @@
# Verify it yourself — the byte-identity contract, and how to check it with your own disc
> BFM-decomp's claim is narrow and machine-checkable: **218 binaries** (the main executable `SLUS_007.26`, the resident
> engine blob, 138 location overlays and 78 code modules) **rebuild byte-for-byte from the C in `src/`**, compared against
> the originals extracted from a redump-verified USA disc. This page is the recipe, the expected output of every step,
> and the record of the last full run. Nothing ROM-derived is in the repository (`tools/audit_public.py` guards that), so
> every check below that touches game bytes needs YOUR disc dump. What CI proves without a disc is at the end.
## 0. What you need
| Item | Detail |
|---|---|
| The disc | *Brave Fencer Musashi (USA)* (SLUS-00726), redump layout: 4-track BIN/CUE. Track 1 SHA1 `b44f0f0a19936f23b26188b658e13201a6a9c211`, CRC32 `c238191b`. A Track-1-only dump also works (the 3 CD-audio tracks are then reported as unverified, never silently passed). |
| Linux (WSL2 Ubuntu 24.04 is what the project uses) | `binutils-mipsel-linux-gnu`, `cpp-mipsel-linux-gnu`, `python3.12`, `git`, `make` — `make bootstrap` prints the exact apt line and installs the rest (venv, submodules, the tracked gcc-2.7.2 `cc1` tarballs, sha256-checked). |
| Disk / time | ≈3 GB for the build tree; the full clean rebuild takes ≈5 min on 16 cores (measured 4 m 18 s on a fresh clone, single user 45 CPU-min). |
The PsyQ SDK objects are **optional** and never needed for byte-identity: without them the main executable links the
project's own `INCLUDE_ASM` fallback tiles for Sony's library regions and is still byte-identical (this is the WITHOUT leg
of `make sdk-dual`, and it is what every fresh clone builds). See `docs/SETUP.md` §P33 B4 if you own the SDK.
## 1. The recipe (R22: a CLEAN rebuild — never an incremental one)
```bash
git clone --recurse-submodules https://github.com/Druthulu/BFM-decomp.git && cd BFM-decomp
make bootstrap # -> "check-env: OK" (one WARN: run make disc-extract)
mkdir -p disks && cp '/path/to/Brave Fencer Musashi (USA)'*.{bin,cue} disks/
make disc-extract # -> "disc-extract: OK" (extracts the disc, compares 1,801 files to the committed manifest)
make clean && make -j"$(nproc)" extract-all && make -j"$(nproc)" check-all
# -> "extract-all: 217 extracted, 0 failed of 217 (+ main, serial)"
# -> "check-all: 218 passed, 0 failed of 218" <- THE contract line
make sdk-dual # optional, only with the SDK objects -> "sdk-dual: OK — main 143dbb89… byte-identical WITH and WITHOUT the PsyQ objects"
make tools-health # -> "tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green."
make report # -> the three 100% lines below, "INCLUDE_ASM stubs : 0"
```
Every `make` target exits non-zero on any failure; read the exit code, not the presence of an output file (a failed build
leaves the previous binary in place — rule R53). `make help` lists every target with one line each.
### What each step proves
| Step | Proves | Expected last line |
|---|---|---|
| `make bootstrap` | the toolchain is present and pinned (cc1 tarballs sha256 `500a459b…` / `42bb0df9…`, maspsx submodule at the pinned commit, binutils) | `check-env: OK` |
| `make disc-extract` | your disc is the redump image and extracts to the SAME 1,801 files the project was built against (`extracted/retail/manifest.jsonl` + `.sha1`, committed) | `disc-extract: OK` (or `PARTIAL: 1,798/1,798 code+data verified; 3 .DA unverified` for Track-1-only) |
| `make extract-all` | splat splits every binary from the extracted payloads (asm/ and the linker scripts are regenerated, never edited) | `extract-all: 217 extracted, 0 failed of 217 (+ main, serial)` |
| `make check-all` | **every binary's build SHA1 equals `config/check.<alias>.sha`** — main `143dbb89f34491258bbc27810d0a12ec8b43a8dd` | `check-all: 218 passed, 0 failed of 218` |
| `make sdk-dual` | main is byte-identical both with Sony's real objects linked and with the fallback tiles | `sdk-dual: OK — …` |
| `make tools-health` | the two independent function-boundary oracles agree (0 phantom / 0 truncated / 0 pad-tail), the dedup registry validates (2,220 groups / 0 failures), the audits and derived indexes are fresh | `tools-health: OK — …` |
| `make report` | the three progress metrics, source-derived | `FLEET fn-count byte-ident: 363214 / 363214 = 100.00%` · `FLEET instr-weighted : 13492113 / 13492113 = 100.0%` · `FLEET distinct-code(uniq): 5820205 / 5820205 = 100.0%` · `MAIN game-code weighted : 45150 / 45150 = 100.0%` |
### What is NOT our C (stated plainly)
- **1,256 functions of the main executable are Sony's PsyQ library objects**, linked byte-identical from the SDK when you
have it and represented by assembly fallback tiles when you do not (`src/lib*.c`, `src/apicard*.c` — `INCLUDE_ASM`
stubs whose bytes come from the disc). Reimplementing Sony's libraries from source is out of scope.
- **5 functions fleet-wide are hand-written assembly in the original** (`config/verbatim_manifest.json`, PERMANENT rows);
they are kept as verbatim `__asm__` bodies, audited by `tools/verbatim_check.py --strict`.
- Everything else — every game-code function in all 218 binaries — is C.
## 2. The last recorded run
The project's own full run is recorded, log by log, under `.run/P33/verify/` (tracked: `NN_<step>.log`, each ending in
`EXIT=<rc>` and a timestamp, plus `SUMMARY.md`), produced by `tools/verify_contract.sh` on the committed tree. The table
below is copied from that `SUMMARY.md`; regenerate it with the script, never by hand.
| # | Step | Result | Log |
|---|---|---|---|
| — | *(pending: filled by the Phase-33 A5 run — see `phase-ends/CURRENT_PHASE.md`)* | | |
The same run was performed after every banked batch of Phases 30–32 (218/218 at every one; the P32 close run is
`.run/P32/t4e/r22_check.log`) and the history-rewrite of the public flip is followed by one more (C8) on the adopted tree —
a content-preserving rewrite changes no tracked byte, and that run proves it.
## 3. What CI proves without the disc
`.github/workflows/no-rom.yml` runs on every push and pull request:
- **audits** — no ROM-derived bytes, purge paths or >50 MiB files among the tracked files (`tools/audit_public.py`, whose
ROM-hash set is derived from the committed manifest + every `config/check.*.sha`); every source is text with portable
includes; the verbatim manifest has no drift; the derived indexes (cookbook, Ghidra roster) are fresh; the LZSS decoder's
unit tests; no stale symbol references.
- **compile-only** — every eligible translation unit (4,170 of 4,287; the rest are the library tiles that `.include`
disc-derived assembly) goes through cpp → gcc-2.7.2 `cc1` → maspsx → GNU `as` with the Makefile's exact flags. A PR
compiles four representative binaries (≈1 min); the whole fleet runs weekly and on demand.
CI cannot compare bytes to the originals — that is your step 1. The two together are the full claim.
## 4. Reverse-engineering artifacts you can also regenerate
The Ghidra project is not in the repository (its database embeds the game's bytes). Its hand-authored content is tracked
as text under `config/ghidra/` (`ROSTER.md` lists the programs) and `tools/ghidra_rebuild.sh <program> --proof` rebuilds a
program from your disc + the symbol files + that text and proves the result equals it (`PROOF PASS`). The 28 RAM images
the memory map was derived from are local-only; `dumps/CHECKSUMS.sha1` records their identity.
+27 -20
View File
@@ -58,7 +58,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo
resident; roster; ExportSymbols R15 fix; path hardcodes; hooks) — Max (finished at medium, Drew's call) — see Log 2026-09-06 B5
- [x] **B6** `dumps/CHECKSUMS.sha1` + INDEX.md rewrite + memory-map Source-index row — Low/xHigh — see Log 2026-09-06 B6
- [x] **B7** No-ROM CI (`no-rom.yml`, `audit_public.py`, `compile_only.py`) — xHigh — see Log 2026-09-06 B7
- [ ] **B8** SETUP.md rows/sections (R21) + `docs/verification.md` — xHigh
- [x] **B8** SETUP.md rows/sections (R21) + `docs/verification.md` — xHigh — see Log 2026-09-06 B8
- [ ] **A5** THE RECORDED RUN (`tools/verify_contract.sh` → `.run/P33/verify/`, SUMMARY all EXIT=0) — run Low, read Max
- [ ] **B9/C3** The preparatory commit (`git rm --cached` purge set; psyq CHECKSUMS moved; zip sha256s; runbook;
decision-log entry) — Max (P5c-class)
@@ -210,21 +210,30 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact
Gotcha, recorded: `pkill -f '<pattern>'` from a shell whose own command line contains the pattern kills that shell
(exit 144) — match on the child's distinctive argv or use `pgrep -f … | grep -v $$`. SETUP: P33 B7 section + 4
inventory rows (R21). Commit: see below.
- **2026-09-06 (S87) — B8 (SETUP cross-references + `docs/verification.md`).** Every P33 tool already had its SETUP section +
inventory row (written with its task, R21); B8 added the as-built pointers the plan named — §4.4 → `make disc-extract`,
§4.6 → `make bootstrap`, §4.8 → `tools/fetch_psyq.sh`, §6.3 → `make help` + the new targets + the R22 line — and the
"Backup & private-repo posture" P33 update (R20's new home: `config/ghidra/` + `--proof`, `dumps/CHECKSUMS.sha1`,
`tools/psyq_CHECKSUMS.sha256`, the archive repo; "never `git clean -x`"). NEW `docs/verification.md`: what you need
(redump SHA1/CRC, apt, disk/time), the R22 recipe with every expected last line (from `make help`, the P32/B3 logs and the
current digest: 363,214 / 13,492,113 / 5,820,205 / main 45,150), what is NOT our C (1,256 LINKED + 5 verbatim), the
last-recorded-run table (placeholder until A5 fills it from `SUMMARY.md`), what CI proves without the disc, the
regenerable RE artifacts. `tools/verify_contract.sh` is A5's (next). Commit: see below.
## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B7 ✓; NEXT = B8 (2026-09-06 ~22:00 MDT, written by session fa49faf3 "S87" after the B7 commit; SUPERSEDES the earlier blocks)
## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B8 ✓; NEXT = P6 rules check, then A5 THE RECORDED RUN (2026-09-06 ~22:20 MDT, written by session fa49faf3 "S87" after the B8 commit; SUPERSEDES the earlier blocks)
### 0. How to use this block
You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file,
and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest
(R1–R73), then WAIT for Drew. **NEXT = B8** (xHigh), then the P6 rules check (12 tasks done), then A5. The harness task list must be REBUILT (Drew wants to monitor it —
one TaskCreate per plan item A1…G2, 40 items, mark A1–A4 + B1–B7 completed; R28). The SessionStart hook restarts the headless
(R1–R73), then WAIT for Drew. **NEXT = the P6 rules check (12 tasks done), then A5 THE RECORDED RUN** (run Low, read Max). The harness task list must be REBUILT (Drew wants to monitor it —
one TaskCreate per plan item A1…G2, 40 items, mark A1–A4 + B1–B8 completed; R28). The SessionStart hook restarts the headless
MCP server when `ghidra/bfm.rep` exists (it did not stay up in S87 — `ss -tln` showed nothing on :8080; harmless): B6/B7/B8
need no Ghidra; run `tools/ghidra_mcp_stop.sh` before any headless step (R23).
### 1. Where we are
**Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). R65–R73 ratified.
**Done: A1 (`commit:4012`), A2 (`commit:4013`), A3 (`commit:4014`), A4 (`commit:4015`), B1 (`commit:4016`), B2 (`commit:4017`), B3
(`commit:4018`), B4 (`commit:4019`), B5 (`commit:4022`), B6 (`commit:4023`), B7 (the S87 `feat(phase-33): B7 …` commit).** The approved plan is
(`commit:4018`), B4 (`commit:4019`), B5 (`commit:4022`), B6 (`commit:4023`), B7 (`commit:4024`), B8 (the S87 `docs(phase-33): B8 …` commit).** The approved plan is
VERBATIM at the end of this file — Blocks A–G give every task's files, commands and verification; "Execution order and why"
is the sequence. Effort: Drew ran S87 at **medium** by explicit choice (the plan says Max for B5); the plan's annotations
still stand for the tasks ahead — restate them, Drew decides (R7/R27).
@@ -250,22 +259,20 @@ still stand for the tasks ahead — restate them, Drew decides (R7/R27).
### 3. NEXT — in order
0. **Preflight:** `git status --short | grep -v ghidra/` (empty) · `git log -1 --format='%h %s'` · `df -h ~`.
1. **B8 — SETUP.md rows/sections (R21) + `docs/verification.md`** (xHigh): SETUP already carries P33 A2/A3/A4/B1/B2/B3/B4/B5/B7
sections and inventory rows (written in the same change as each task); B8 = the plan's §4.4 disc-extract / §4.6 bootstrap /
§4.8 fetch_psyq / §6.3 new-targets cross-references (check each exists; add what is missing), the "Backup & private-repo
posture" paragraph pointing at `config/ghidra/` + `ghidra_rebuild.sh --proof` as R20's new home, and NEW
`docs/verification.md` — the public "verify it yourself" page: the commands (`make bootstrap` → `make disc-extract` →
`make extract-all` → `make check-all` → `make sdk-dual` [optional] → `make tools-health` → `make report`), each with its
expected last line, the R22 shape, the SHA1s (EXE `143dbb89…`, redump Track-1 `b44f0f0a…`), what CI proves vs what needs
the disc, and a placeholder table that A5's SUMMARY fills (A5 runs `tools/verify_contract.sh`, which B8 may write now or
A5 may — the plan puts the script under A5). Log, tick, refresh, commit.
2. **P6 rules check** (12 tasks done) → **A5 THE RECORDED RUN** (the plan's A5 paragraph: `tools/verify_contract.sh` →
`.run/P33/verify/NN_<step>.log` + `SUMMARY.md`; allowlist `.run/P33/verify/` in `.gitignore` like `.run/P32/t4e/`; ≈1 h
for the clean fleet + sdk-dual + tools-health + audit-disc + report) → **B9/C3** (Max, P5c-class: the `git rm --cached`
commit — `git mv tools/psyq/CHECKSUMS.sha256` is ALREADY done (B4); the zip sha256s go into SETUP §2.3/§2.4;
`docs/public-flip-runbook.md`; decision-log entry).
1. **P6 rules check** (12 tasks done): re-read CLAUDE.md's rules + DIGEST §3, state "Rules check — re-read complete. Continuing
with A5."
2. **A5 — THE RECORDED RUN** (run Low, read Max; ≈1 h wall): write `tools/verify_contract.sh` per the plan's A5 paragraph —
steps 00 `git rev-parse HEAD` + porcelain (only the R23 `ghidra/` churn allowed) · 01 `make check-env` · 02 family_hseq
regen · 03 `make clean && make extract-all JOBS=16 && make check-all JOBS=16` · 04 `make sdk-dual` (keep both `.map`s; the
two from A3 are already in `.run/P33/verify/`) · 05 `make tools-health` (must contain `0 PHANTOM, 0 TRUNCATED, 0 PAD-TAIL`,
zero `[warn]`) · 06 `make audit-frontier` · 07 `make audit-disc` (4-track disc present in `disks/`; residue 0) · 08
`make report` (three 100 lines, `INCLUDE_ASM stubs : 0`, `Open near-misses: 0`) · 09 SUMMARY.md; each log ends `EXIT=<rc>`
+ timestamp; abort on the first non-zero (R53); `.gitignore` allowlists `.run/P33/verify/` (`*.log *.map *.md *.txt`)
exactly like `.run/P32/t4e/`; then paste SUMMARY's table into `docs/verification.md` §2 (replace the placeholder row).
Runs on the COMMITTED tree (commit B8 first — done). Log, tick, refresh, commit (the logs are tracked).
3. **B9/C3** (Max, P5c-class) → C1 → C2 → … per the task list.
### 4. Files S87 touched
B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
(`.proof` markers; dies unless `failed=0`), `tools/ghidra_annotations_delta.py` (the three drift classes), new
`tools/ghidra_roster.py`, `tools/ghidra_mcp_start.sh` (silent no-op guard), `.claude/settings.json` (relative hooks),
`Makefile` (roster check in tools-health), `docs/SETUP.md` (P33 B5 section, 5 inventory rows, §2.8), `config/ghidra/*`