mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-06 09:06:04 -04:00
docs(phase-33): D1 the README rewrite — the claim and the contract, a GENERATED numbers block (tools/progress.py --json -> docs/progress.json, --readme rewrites the marked block, --check asserts freshness; wired into make report), what is not our C, build-from-your-own-disc, layout, how it was made (ProjectArchitect), about the history (tokens + commit-map), license split, no-ROM policy, special thanks; forward references to D2/D3/F1/F2 files recorded in the checkpoint
This commit is contained in:
Drew T
@@ -320,6 +320,8 @@ ifeq ($(BINARY),main)
|
||||
$(VENV_PY) tools/dup_report.py --cross
|
||||
# Fleet roll-up (Phase 15): deterministic per-binary table + fleet totals -> docs/progress.fleet.md.
|
||||
$(VENV_PY) tools/progress.py --fleet
|
||||
# P33 D1/D3: the same numbers as DATA — docs/progress.json + the README block (never typed by hand)
|
||||
$(VENV_PY) tools/progress.py --json --readme
|
||||
# Backlog compaction (Phase 29): the near-miss log is append-only, so it fills with already-banked
|
||||
# noise (measured 6,867 rows, 98% banked). prune rewrites .run/backlog.jsonl to the open near-misses
|
||||
# (drop-now-matched P9 + best-per-addr) so the ledger tracks reality instead of drifting stale.
|
||||
|
||||
@@ -1,35 +1,144 @@
|
||||
# BFM-decomp
|
||||
# BFM-decomp — Brave Fencer Musashi, decompiled
|
||||
|
||||
A matching decompilation of **Brave Fencer Musashi** (PlayStation, SLUS-00726, USA 1998) — the first public decompilation effort for this game.
|
||||
|
||||
|
||||
|
||||
|
||||
## What "matching" means
|
||||
A **matching decompilation** of *Brave Fencer Musashi* (PlayStation, Square, 1998 — the USA release, SLUS-00726): C source
|
||||
that, compiled with the game's own 1990s toolchain, rebuilds every shipped code binary **byte for byte**. It is the first
|
||||
public decompilation of this game, and the whole of its game code is matched: the main executable, the always-resident
|
||||
engine, every location overlay and every code module streamed from the disc — **218 binaries**, verified against a redump
|
||||
image of the original disc on every build.
|
||||
|
||||
The goal is C source code that, compiled with the original-era toolchain (PsyQ 4.x / GCC 2.7.2-family + ASPSX via maspsx), produces a byte-for-byte identical `SLUS_007.26` and, eventually, byte-identical overlay binaries. SHA1 checksums are the ground truth; "functionally equivalent" does not count.
|
||||
The contract is narrow and machine-checkable. "Matching" means byte-identical output, checked by SHA1 against the
|
||||
originals; nothing "functionally equivalent" counts. What the repository claims is exactly what `make check-all` proves.
|
||||
|
||||
## No ROM content
|
||||
## Where it stands
|
||||
|
||||
This repository contains **no game assets, no disassembly output, and no ROM-derived data** — only source code, build configuration, symbol names/addresses, hashes, and documentation. To build or contribute you must provide your own dump of the game disc (4-track BIN/CUE, redump layout). See `.gitignore` for the firewall.
|
||||
<!-- progress:begin -->
|
||||
| Metric | Matched | Of | % |
|
||||
|---|---:|---:|---:|
|
||||
| Functions byte-identical (all binaries) | 363,214 | 363,214 | 100.00% |
|
||||
| Instructions byte-identical (instruction-weighted) | 13,492,113 | 13,492,113 | 100.0% |
|
||||
| Distinct code byte-identical (each body once) | 5,820,205 | 5,820,205 | 100.0% |
|
||||
| Main executable game code (excl. Sony objects) | 45,150 | 45,150 | 100.0% |
|
||||
|
||||
## Project status
|
||||
218 binaries rebuild byte-identical from source · 360,737 functions in C (255,632 of them shared bodies via 2,220 dedup groups) · 1,256 Sony PsyQ library functions linked from the SDK objects, not our C · 5 hand-written-assembly bodies kept verbatim · 0 assembly stubs left · 0 non-matching functions.
|
||||
|
||||
> **Latest (Phase 19, 2026-06-20):** the project builds **136 binaries byte-identical from a clean tree** (the EXE + the resident engine + all **134 location overlays**); **`make check-all` → 136/136**. Fleet byte-identical-from-source is **58.0%** (function-instance-weighted; see the PhaseEnds for the byte-weighted ~30% figure and what it includes). Shared engine functions are matched once in `ov_SC01_077` and propagated **×134** via `tools/dedup_propagate.py`. *(The narrative below is Phase-11/12-era; a full refresh is part of the public-flip prep.)*
|
||||
_Generated by `tools/progress.py --readme` from `docs/progress.json` — numbers are never typed by hand._
|
||||
<!-- progress:end -->
|
||||
|
||||
**Gen1 (foundation) complete — the matching pipeline is proven end-to-end.** `make extract && make build && make check` rebuilds `SLUS_007.26` **byte-for-byte identical** (SHA1 `143dbb89…`) from C + assembly, reproducibly across many sessions.
|
||||
Two things in those binaries are deliberately **not** our C, and are stated as such:
|
||||
|
||||
- **Compiler pinned by evidence:** `gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000` + `maspsx --aspsx-version=2.56 --expand-div`.
|
||||
- **52 functions hand-matched** to byte-identical machine code — including the LZSS streaming decompressor — with a decomp-permuter + matching-cookbook "flywheel" to accelerate the next.
|
||||
- **959 PsyQ SDK functions linked byte-identical** (libcd, libgs, libgte, libspu/libsnd, libgpu, libc2, libmcrd, libapi/libcard, libetc) straight from the real PsyQ 4.0 libraries instead of re-decompiling them — bringing byte-identical-from-source coverage of the EXE to **~50%**.
|
||||
- **File-loader / overlay system reverse-engineered**, with the resident engine blob + location overlays' load addresses **proven byte-identical against a live PCSX-Redux RAM dump**.
|
||||
- **1,256 functions of the main executable are Sony's PsyQ library code** (libcd, libgs, libgte, libgpu, libspu/libsnd,
|
||||
libetc, libapi, libcard…). They link byte-identical from the SDK's own objects when you have the SDK, and are carried as
|
||||
assembly tiles taken from the disc when you do not — either way the executable is byte-identical. Reimplementing Sony's
|
||||
libraries is out of scope.
|
||||
- **Five functions across the fleet are hand-written assembly in the original** and are kept as verbatim `__asm__`
|
||||
bodies (`config/verbatim_manifest.json`, audited by `tools/verbatim_check.py --strict`).
|
||||
|
||||
About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway:
|
||||
Everything else — every game-code function in all 218 binaries — is C that the pinned compiler turns into the original
|
||||
bytes. Progress numbers are generated from the sources (`tools/progress.py`), never typed by hand; the full recipe to
|
||||
reproduce them is in [`docs/verification.md`](docs/verification.md).
|
||||
|
||||
- The build toolchain is **binary-agnostic** (one parameterized pipeline builds any binary), and the always-resident **engine blob** rebuilds **byte-for-byte from source** (SHA1 `8e17e02f…`) — the *second* binary reconstructed exactly, after the EXE — and is now **86% hand-matched C** (123 / 146 functions, up from 0): its scripting turned out to be **compiled-MIPS state/mode dispatch, not a bytecode VM**, and the save-file + sound (SQV) formats are documented. The harvest used a reusable **swarm-of-agents + bit-for-bit byte-gate** method (a wrong match can't be accepted) — `tools/harvest_verify.py` + `tools/match_one.py`, which carry straight into the overlay phase.
|
||||
- A **cross-binary deduplication pipeline** is live: a Ghidra-free signer fingerprints all 134 location overlays, and the report finds **~9,000 byte-identical function groups shared across binaries (~28 MB of collapsible code)** — a single engine function is byte-identical in all 134 overlays. This is "one match unlocks many": each engine match will be auto-credited across the overlay fleet.
|
||||
## Build it from your own disc
|
||||
|
||||
Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`.
|
||||
The repository contains **no game data**: no executable, no disassembly, no assets, no memory dumps — only source, build
|
||||
configuration, symbol names and hashes. You need your own redump-layout dump of the USA disc (Track 1 SHA1
|
||||
`b44f0f0a19936f23b26188b658e13201a6a9c211`, CRC32 `c238191b`; the four-track BIN/CUE is ideal, Track 1 alone works).
|
||||
|
||||
This project is developed primarily by Claude Code driving Ghidra through an MCP server; see `CLAUDE.md`.
|
||||
```bash
|
||||
git clone --recurse-submodules https://github.com/Druthulu/BFM-decomp.git && cd BFM-decomp
|
||||
make bootstrap # toolchain preflight: venv, submodules, the pinned gcc-2.7.2 cc1 (sha256-checked)
|
||||
mkdir -p disks && cp '/path/to/Brave Fencer Musashi (USA)'*.{bin,cue} disks/
|
||||
make disc-extract # extracts the disc and checks all 1,801 files against the committed manifest
|
||||
make clean && make -j"$(nproc)" extract-all && make -j"$(nproc)" check-all
|
||||
# -> "check-all: 218 passed, 0 failed of 218"
|
||||
```
|
||||
|
||||
Linux (WSL2 Ubuntu 24.04 is what the project uses); `binutils-mipsel-linux-gnu`, `cpp-mipsel-linux-gnu`, Python 3.12.
|
||||
The main executable's SHA1 is `143dbb89f34491258bbc27810d0a12ec8b43a8dd`. `make help` lists every target;
|
||||
[`docs/verification.md`](docs/verification.md) gives each step's expected last line and the record of the last full run.
|
||||
|
||||
**Toolchain (pinned by fingerprint evidence):** `gcc-2.7.2-psx cc1 -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker`
|
||||
→ `maspsx --aspsx-version=2.56 --expand-div` → `mipsel-linux-gnu-as -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0`,
|
||||
with per-file `-O0` modules where the bytes demand it. The Sony PsyQ SDK objects are **optional** and user-supplied
|
||||
(`tools/fetch_psyq.sh`); they are never redistributed.
|
||||
|
||||
## What is in the repository
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `src/` | The C — one directory per binary (`src/` itself is the main executable, `src/resident/`, `src/ov_*/`, `src/md_*/`), `src/shared/` for the code shared across binaries via the dedup registry |
|
||||
| `config/` | splat configs, symbol files, per-binary SHA1 contracts (`check.*.sha`), the dedup registry, the Ghidra annotations as text (`config/ghidra/`) |
|
||||
| `tools/` | Everything the project built to do this: the byte gate, the extractor, the matching harness, the campaign machinery, the CI audits, the public-flip rewrite tooling; pinned submodules (maspsx, asm-differ, decomp-permuter, m2c) |
|
||||
| `docs/` | The matching cookbook (500+ compiler idioms), the gcc-2.7.2 codegen map, the memory map, the decision log, the accelerators, the runbooks, the story and the retrospective |
|
||||
| `phase-ends/` | The project's own governance record: one PhaseEnd per phase, the digest, the per-phase logs |
|
||||
| `include/`, `extracted/retail/manifest.*` | Headers; the sha1 manifest your disc extraction is checked against |
|
||||
|
||||
## How it was made
|
||||
|
||||
This project was carried out end to end by an AI coding agent — Claude Code — working inside a governance framework:
|
||||
a static constitution ([`PROJECT_CONTEXT.md`](PROJECT_CONTEXT.md)), a session protocol ([`CLAUDE.md`](CLAUDE.md)),
|
||||
two human gates per phase, an append-only record of every phase ([`phase-ends/`](phase-ends/)), and a growing rule set
|
||||
distilled from what went wrong. The framework is published separately as
|
||||
[**ProjectArchitect**](https://github.com/Druthulu/ProjectArchitect); this repository is its largest worked example.
|
||||
|
||||
The reverse-engineering side ran on Ghidra driven over MCP, with a PCSX-Redux runtime oracle; the matching side is a
|
||||
byte gate over the whole binary, a permuter, a compiler-idiom cookbook, and — decisively — reading the gcc 2.7.2 source
|
||||
to explain the bytes the compiler emits. Everything that was learned is written down where it can be reused:
|
||||
|
||||
- [`docs/matching-cookbook.md`](docs/matching-cookbook.md) — the idioms, each byte-proven, and the compiler-internals laws
|
||||
- [`docs/gcc-2.7.2-map/`](docs/gcc-2.7.2-map/) — the codegen map of the pinned compiler, by pass
|
||||
- [`docs/decision-log.md`](docs/decision-log.md) — every strategic pivot with its measurement and its hindsight
|
||||
- [`docs/accelerators.md`](docs/accelerators.md) — what would have made earlier phases faster
|
||||
- [`docs/story.md`](docs/story.md) and [`docs/retrospective.md`](docs/retrospective.md) — the timeline and the retrospective
|
||||
- [`docs/public-flip-runbook.md`](docs/public-flip-runbook.md) — how a private repository with game bytes in its history
|
||||
became this public one
|
||||
|
||||
**About the history.** The full development history is here — every commit, date and message — but it was rewritten
|
||||
before publication to remove game-derived files from every revision. Old commit hashes cited inside historical documents
|
||||
were replaced by inert tokens (`commit:NNNN`, the commit's ordinal); [`docs/commit-map.tsv`](docs/commit-map.tsv) maps
|
||||
them to the commits you see, and documents at the tip cite the current hashes directly.
|
||||
|
||||
## License
|
||||
|
||||
Private repository for now. **AGPL-3.0 is planned at public release**, modeled on [sotn-decomp](https://github.com/Xeeynamo/sotn-decomp). `tools/brave-CUE/` is CUE's BRAVE extractor (GPL, source included) and retains its own license.
|
||||
- `tools/` and `docs/` (the project's own work): **AGPL-3.0** — see [`LICENSE`](LICENSE).
|
||||
- `src/`: a reimplementation of copyrighted game code, provided for study and preservation; **no license is asserted** —
|
||||
see [`src/NOTICE.md`](src/NOTICE.md).
|
||||
- Third-party components keep their own licenses — see [`THIRD_PARTY.md`](THIRD_PARTY.md). `tools/brave-CUE/` (CUE's
|
||||
reference extractor) is GPL-3.0; the Sony PsyQ SDK is never distributed.
|
||||
|
||||
## Contributing, and the no-ROM policy
|
||||
|
||||
Nothing derived from the game may enter the repository: no executable, no disassembly, no assets, no memory image,
|
||||
no SDK. CI enforces the tracked tree (`tools/audit_public.py`: no ROM-derived content by hash, no purge path, nothing
|
||||
over 50 MiB), keeps every source text and portable, and compiles every translation unit with the pinned compiler. Byte
|
||||
identity itself is verified locally with the disc, and the record of the last full run is committed.
|
||||
|
||||
The matching frontier is empty, so contributions that help most are not new matches: readability (structs and names
|
||||
in place of raw addresses), shiftability, tooling, and documentation. Open an issue first if you are unsure whether
|
||||
something is ROM-derived.
|
||||
|
||||
## Special thanks
|
||||
|
||||
In the tradition of [sotn-decomp](https://github.com/Xeeynamo/sotn-decomp), whose method this project followed from
|
||||
day one — this work stands on:
|
||||
|
||||
- [splat](https://github.com/ethteck/splat), [spimdisasm](https://github.com/Decompollaborate/spimdisasm) and
|
||||
[rabbitizer](https://github.com/Decompollaborate/rabbitizer) — the split and the disassembly
|
||||
- [maspsx](https://github.com/mkst/maspsx) (mkst) — ASPSX's quirks, reproduced
|
||||
- [asm-differ](https://github.com/simonlindholm/asm-differ) and [decomp-permuter](https://github.com/simonlindholm/decomp-permuter) (simonlindholm)
|
||||
- [m2c](https://github.com/matt-kempster/m2c) (matt-kempster) — the scaffolds
|
||||
- [old-gcc](https://github.com/decompals/old-gcc) (decompals) — the compilers, and the gcc 2.7.2 source itself
|
||||
- [Ghidra](https://github.com/NationalSecurityAgency/ghidra), [ghidra_psx_ldr](https://github.com/lab313ru/ghidra_psx_ldr) (lab313ru)
|
||||
and [GhidrAssistMCP](https://github.com/symgraph/GhidrAssistMCP) — the static oracle and its hands
|
||||
- [psyq-obj-parser](https://github.com/decompme/compilers) (decomp.me) — Sony's objects, readable
|
||||
- [PCSX-Redux](https://github.com/grumpycoders/pcsx-redux) — the runtime oracle
|
||||
- [brave-CUE](tools/brave-CUE/) (CUE) — the reference extractor ours was validated against
|
||||
- The Xenogears, Vagrant Story and Tomba! decompilations — sibling PsyQ idioms
|
||||
- [redump](http://redump.org/), [TCRF](https://tcrf.net/) and [gamehacking.org](https://gamehacking.org/) — the ground truth on the disc and in memory
|
||||
- The [Brave Fencer Musashi Archipelago world](https://github.com/AegeusEvander/Brave-Fencer-Musashi-AP-World) (AegeusEvander) — the living RAM map
|
||||
- [ProjectArchitect](https://github.com/Druthulu/ProjectArchitect) and [Claude Code](https://claude.ai/code) — the method and the hands
|
||||
|
||||
*Brave Fencer Musashi is © 1998 Square. This project is not affiliated with or endorsed by Square Enix.*
|
||||
|
||||
+1790
File diff suppressed because it is too large
Load Diff
+24
-11
@@ -71,7 +71,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo
|
||||
- [x] **C8** R22 clean rebuild on the adopted tree — xHigh — see Log 2026-09-07 C8
|
||||
- [x] **C9** Final gate + Drew's force-push + local gc — xHigh — see Log 2026-09-07 C9
|
||||
- [ ] **C10** Support purge → probe gate → FLIP (Drew) — decision Max
|
||||
- [ ] **D1** README rewrite — Max
|
||||
- [x] **D1** README rewrite — Max — see Log 2026-09-07 D1
|
||||
- [ ] **D2** LICENSE + `src/NOTICE.md` + `THIRD_PARTY.md` — Max wording
|
||||
- [ ] **D3** Progress publishing (progress.json, badges, frogress_upload, objdiff_report + progress.yml) — xHigh
|
||||
- [ ] **D4** SETUP.md public-clean pass — xHigh
|
||||
@@ -348,14 +348,28 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact
|
||||
commit found") as gone (it had counted the vanished tag tip as alive). The rewritten history is the only one reachable
|
||||
locally and the only one reachable on `origin`; the old objects remain SERVABLE on GitHub until Support purges — hence C10.
|
||||
Runbook §10/§12 updated with the measured procedure. Commit: see below.
|
||||
- **2026-09-07 (S87, Max) — D1 the README rewrite.** The old README was Phase-19 era (136 binaries, 52 functions, "about
|
||||
half the EXE is stubs") — every number stale, so a full rewrite. **The numbers are GENERATED:** `tools/progress.py --json`
|
||||
→ `docs/progress.json` (schema 1: the four metrics with numerator/denominator/pct + a one-line "what", the counts — real
|
||||
360,737 / shared 255,632 / linked 1,256 / verbatim 5 / stubs 0 / non-matching 0 / matchable 363,214 / dedup 2,220 groups,
|
||||
255,708 instances — and 218 per-binary rows; no run date, so it never churns) and `--readme` rewrites the block between
|
||||
`<!-- progress:begin/end -->` (refuses a README without the markers — control run; `--check` asserts both are fresh);
|
||||
wired into `make report BINARY=main` after `--fleet`. README (144 lines): the claim and the contract · the generated block
|
||||
· what is NOT our C (1,256 Sony objects + 5 verbatim) · build from your own disc (the verification.md recipe, redump SHA1/
|
||||
CRC32, the EXE SHA1, the pinned triple, the optional SDK) · repository layout · how it was made (ProjectArchitect link,
|
||||
the governance files, the knowledge base) · **About the history** (the rewrite, the `commit:NNNN` tokens, `docs/commit-map.tsv`)
|
||||
· the license split · contributing + the no-ROM policy · **Special thanks** (the plan's list, sotn-decomp style) · the
|
||||
Square trademark line · 3 badges. **Forward references the README makes to files later tasks create** (D5's link checker
|
||||
must see them by then): `LICENSE`, `src/NOTICE.md`, `THIRD_PARTY.md` (D2); `docs/badges/fleet_instr.json`,
|
||||
`docs/badges/binaries.json` (D3 — these exact names); `docs/story.md` (F1), `docs/retrospective.md` (F2). Commit: see below.
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓ — THE REWRITTEN HISTORY IS ON origin (e821833986…), THE OLD ONE IS GONE LOCALLY; NEXT = C10 (Drew: the Support ticket + the daily probe) IN PARALLEL WITH D1 (Claude: the README, Max) (2026-09-07 ~06:45 UTC, written by session fa49faf3 "S87" after the C9 commit; SUPERSEDES the earlier blocks)
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1 ✓; C10 IN PROGRESS ON DREW'S SIDE (the Support ticket + the daily probe); NEXT = D2 (Max wording) (2026-09-07 ~07:30 UTC, written by session fa49faf3 "S87" after the D1 commit; SUPERSEDES the earlier blocks)
|
||||
|
||||
### 0. How to use this block
|
||||
You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file,
|
||||
and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest
|
||||
(R1–R73), then WAIT for Drew. **NEXT = two tracks.** **Drew (C10):** open the GitHub Support ticket (text in `docs/public-flip-runbook.md` §11), then run `tools/public_rewrite/probe_github.sh` daily (gh is authenticated in WSL; 31 of 33 sampled old hashes were alive at S87's baseline) until it prints PASS — only then the visibility flip, and only after Blocks D/E/F have landed. **Claude (D1, Max — prompt per R7/R27):** the README rewrite (the plan's D1 paragraph), then D2 → D3 → D4 → D5 → F1 → F2 → F3 → E3 → E4 → E5 → E6 on the still-private repo; E1/E2 and D3's outward actions after the flip. **Every commit cites NEW hashes only; `.run/public_rewrite/` (scratch: dict, mailmap, old-to-new, bundle 556 MB, the rewritten bare clone) holds the old ones — keep until the probe passes, then delete (C11).**
|
||||
one TaskCreate per plan item A1…G2, 40 items, mark A1–A5 + B1–B9/C3 + C1–C9 completed; R28). The SessionStart hook restarts the headless
|
||||
(R1–R73), then WAIT for Drew. **NEXT = two tracks.** **Drew (C10):** open the GitHub Support ticket (text in `docs/public-flip-runbook.md` §11), then run `tools/public_rewrite/probe_github.sh` daily (gh is authenticated in WSL; 31 of 33 sampled old hashes were alive at S87's baseline) until it prints PASS — only then the visibility flip, and only after Blocks D/E/F have landed. **Claude (NEXT = D2, Max wording):** root `LICENSE` = AGPL-3.0 VERBATIM (fetch `https://www.gnu.org/licenses/agpl-3.0.txt` — X2: data; never retype), `src/NOTICE.md` (reimplementation-for-study wording, no license asserted, © Square 1998 acknowledged, NEVER the phrase "clean-room"), `THIRD_PARTY.md` (the plan's D2 rows: brave-CUE GPL-3.0 source only; submodules maspsx MIT / decomp-permuter MIT / asm-differ Unlicense / m2c GPL-3.0 — VERIFY each upstream license file; old-gcc 0.17 GPL + sha256; GhidrAssistMCP MIT download; ghidra_psx_ldr — re-check upstream for a license; Ghidra Apache-2.0; binutils; splat/spimdisasm/rabbitizer pip; PsyQ SDK user-supplied never distributed; gcc-2.7.2 source GPL-2.0 sha256 `7cd8bce5…`; ProjectArchitect — Drew decides keep-or-link for `docs/history/project_architect_v1.3.0.md`); one statement in `tools/README.md` (no per-file SPDX). Then D3 → D4 → D5 → F1 → F2 → F3 → E3 → E4 → E5 → E6 on the still-private repo; E1/E2 and D3's outward actions after the flip. **Every commit cites NEW hashes only; `.run/public_rewrite/` (scratch: dict, mailmap, old-to-new, bundle 556 MB, the rewritten bare clone) holds the old ones — keep until the probe passes, then delete (C11).**
|
||||
one TaskCreate per plan item A1…G2, 40 items, mark A1–A5 + B1–B9/C3 + C1–C9 + D1 completed; R28). The SessionStart hook restarts the headless
|
||||
MCP server when `ghidra/bfm.rep` exists (it did not stay up in S87 — `ss -tln` showed nothing on :8080; harmless): B6/B7/B8
|
||||
need no Ghidra; run `tools/ghidra_mcp_stop.sh` before any headless step (R23).
|
||||
|
||||
@@ -391,16 +405,15 @@ still stand for the tasks ahead — restate them, Drew decides (R7/R27).
|
||||
1. **C10** (Drew; decision Max): the Support ticket (text in `docs/public-flip-runbook.md` §11), then `tools/public_rewrite/
|
||||
probe_github.sh` daily until PASS (gh auth done in S87; `old-to-new.tsv` + `old_tag_tip.txt` in scratch); the flip only
|
||||
after the probe passes AND Blocks D/E/F have landed.
|
||||
2. **D1 — README rewrite** (Max; the plan's D1 paragraph: what/why · the numbers table WITH denominators generated by
|
||||
`tools/progress.py --readme` from `docs/progress.json` (D3 provides progress.json — D1 may stub the block and D3 wires the
|
||||
generator, or do D3's progress.json first; decide at D1) · "what is not our C" · build-from-your-own-disc · verification
|
||||
evidence (`docs/verification.md`) · method (ProjectArchitect link, CLAUDE.md, phase-ends, decision-log, accelerators) ·
|
||||
story/retrospective/wiki links · license split · contributing / no-ROM policy · **Special thanks** (the plan's list) ·
|
||||
badges). Then D2 (LICENSE AGPL-3.0 verbatim, `src/NOTICE.md`, `THIRD_PARTY.md` — never the phrase "clean-room"), D3, D4, D5.
|
||||
2. **D2 — LICENSE + `src/NOTICE.md` + `THIRD_PARTY.md`** (Max wording, xHigh to place): see the NEXT line above. Verify
|
||||
every third-party license from the upstream file (submodule LICENSE files are on disk under `tools/<submodule>/`).
|
||||
Log, tick, refresh, commit. Then **D3** (progress.json exists since D1; add `docs/badges/{fleet_instr,binaries}.json`
|
||||
in shields endpoint format written by `--json`, `make audit-digest` extended to `progress.py --json --readme --check`,
|
||||
`tools/frogress_upload.py --dry-run`, `tools/objdiff_report.py` + `.github/workflows/progress.yml`).
|
||||
3. Aftercare reminders (C11): `gh auth logout` in WSL after the flip; `git remote remove archive`; other clones reset, never
|
||||
pull; `.run/public_rewrite/` deleted after the probe passes (keep the bundle off-machine if wanted).
|
||||
### 4. Files S87 touched
|
||||
C9: `tools/public_rewrite/probe_github.sh` (422 = gone), `docs/public-flip-runbook.md` §10/§12; `.git` (the archive remote removed, 12 worktrees removed, repacked to one 80 MB pack). C8: `.run/P33/verify/*` (the recorded run, tracked), `docs/verification.md` §2, `docs/family-hseq.md`, `tools/verify_contract.sh` (the step-00 fix). C4–C7: `docs/commit-map.tsv` (new), the 98 token-resolved files (see the C7 entry), `tools/public_rewrite/resolve_tokens.py` (the skip rule), `.git/config` (repo-local noreply identity). Scratch `.run/public_rewrite/`: `pre-rewrite.bundle`, `repo.git` (the rewritten bare clone, keep until C9), `c4c5.{sh,log}`, `old-to-new.tsv`, `unchanged_commits.txt`, `old_tag_tip.txt`, `c5_commit-map.tsv`, `ids_after.txt`, `refs_*.txt`. C1: `tools/public_rewrite/{common,hash_dict,scrub,gate_scan,run_filter,verify_rewrite,build_commit_map,resolve_tokens,absent_scan}.py`, `probe_github.sh`, `expected_offenders.txt` (all new), `requirements-python.txt`, `docs/SETUP.md`, `docs/public-flip-runbook.md`. Scratch `.run/public_rewrite/`: `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old_tag_tip.txt`, `trial*.{sh,log}`, `trial_commit-map.tsv`, `trial_old-to-new.tsv`, `unchanged_commits.txt`, `nonpurge_blob_ids.txt`, `filter.log`. B9/C3: 251 index removals (files on disk), `docs/public-flip-runbook.md` (new), `docs/SETUP.md` §2.3/§2.4, `docs/verification.md`, `CLAUDE.md`, `docs/decision-log.md`. A5: `tools/verify_contract.sh` (new), `tools/audit_frontier.py` (derived denominator lines), `.gitignore` (the `.run/P33/verify/` allowlist), `.run/P33/verify/*` (tracked evidence), `docs/verification.md` §2, regenerated `docs/family-hseq.md` + `docs/progress*.md`/`duplicates*.md`. B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
|
||||
D1: `README.md` (rewritten), `docs/progress.json` (new, generated), `tools/progress.py` (`--json`/`--readme`/`--check`), `Makefile` (report wiring). C9: `tools/public_rewrite/probe_github.sh` (422 = gone), `docs/public-flip-runbook.md` §10/§12; `.git` (the archive remote removed, 12 worktrees removed, repacked to one 80 MB pack). C8: `.run/P33/verify/*` (the recorded run, tracked), `docs/verification.md` §2, `docs/family-hseq.md`, `tools/verify_contract.sh` (the step-00 fix). C4–C7: `docs/commit-map.tsv` (new), the 98 token-resolved files (see the C7 entry), `tools/public_rewrite/resolve_tokens.py` (the skip rule), `.git/config` (repo-local noreply identity). Scratch `.run/public_rewrite/`: `pre-rewrite.bundle`, `repo.git` (the rewritten bare clone, keep until C9), `c4c5.{sh,log}`, `old-to-new.tsv`, `unchanged_commits.txt`, `old_tag_tip.txt`, `c5_commit-map.tsv`, `ids_after.txt`, `refs_*.txt`. C1: `tools/public_rewrite/{common,hash_dict,scrub,gate_scan,run_filter,verify_rewrite,build_commit_map,resolve_tokens,absent_scan}.py`, `probe_github.sh`, `expected_offenders.txt` (all new), `requirements-python.txt`, `docs/SETUP.md`, `docs/public-flip-runbook.md`. Scratch `.run/public_rewrite/`: `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old_tag_tip.txt`, `trial*.{sh,log}`, `trial_commit-map.tsv`, `trial_old-to-new.tsv`, `unchanged_commits.txt`, `nonpurge_blob_ids.txt`, `filter.log`. B9/C3: 251 index removals (files on disk), `docs/public-flip-runbook.md` (new), `docs/SETUP.md` §2.3/§2.4, `docs/verification.md`, `CLAUDE.md`, `docs/decision-log.md`. A5: `tools/verify_contract.sh` (new), `tools/audit_frontier.py` (derived denominator lines), `.gitignore` (the `.run/P33/verify/` allowlist), `.run/P33/verify/*` (tracked evidence), `docs/verification.md` §2, regenerated `docs/family-hseq.md` + `docs/progress*.md`/`duplicates*.md`. B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
|
||||
(`.proof` markers; dies unless `failed=0`), `tools/ghidra_annotations_delta.py` (the three drift classes), new
|
||||
`tools/ghidra_roster.py`, `tools/ghidra_mcp_start.sh` (silent no-op guard), `.claude/settings.json` (relative hooks),
|
||||
`Makefile` (roster check in tools-health), `docs/SETUP.md` (P33 B5 section, 5 inventory rows, §2.8), `config/ghidra/*`
|
||||
|
||||
@@ -1072,9 +1072,97 @@ def fleet():
|
||||
text = "\n".join(head + table) + "\n"
|
||||
print("\n".join(head[:-1])) # the summary block (drop the trailing blank)
|
||||
(ROOT / "docs/progress.fleet.md").write_text(text)
|
||||
# P33 D1/D3: the same numbers as DATA (docs/progress.json, README block, badges, frogress/objdiff feeds)
|
||||
nverb = 0
|
||||
vp = ROOT / "config/verbatim_manifest.json"
|
||||
if vp.exists():
|
||||
try:
|
||||
import json as _json
|
||||
vm = _json.loads(vp.read_text())
|
||||
nverb = len(vm.get("rows", vm) if isinstance(vm, dict) else vm)
|
||||
except Exception:
|
||||
nverb = 0
|
||||
return {
|
||||
"schema": 1,
|
||||
"note": "generated by tools/progress.py --fleet/--json from the committed sources — every number carries its denominator; regenerate, never edit",
|
||||
"binaries": len(rows),
|
||||
"fleet": {
|
||||
"fn_count": {"matched": BYTE, "total": MATCH, "pct": round(100 * BYTE / MATCH, 2) if MATCH else 0.0,
|
||||
"what": "functions byte-identical / matchable, every binary (one shared body counts once per overlay it is instantiated in)"},
|
||||
"instr_weighted": ({"matched": wm['fleet_m'], "total": wm['fleet_t'], "pct": round(wm['fleet_pct'], 1),
|
||||
"what": "shipped .text instructions across all binaries (the decomp.dev-style display number)"} if wm else None),
|
||||
"distinct_code": ({"matched": wm['dedup_m'], "total": wm['dedup_t'], "pct": round(wm['dedup_pct'], 1),
|
||||
"unique_fns_matched": wm['dedup_fns'], "unique_fns_total": wm['dedup_total_fns'],
|
||||
"what": "each distinct function body counted once, instruction-weighted (the distinct reverse-engineering number)"} if wm else None),
|
||||
"main_game_code": ({"matched": wm['main_m'], "total": wm['main_t'], "pct": round(wm['main_pct'], 1),
|
||||
"what": "SLUS_007.26 game-code instructions (excludes the linked Sony PsyQ objects); build-derived boundaries"} if wm and wm.get('main_t') else None),
|
||||
},
|
||||
"counts": {"real_c_functions": REAL, "dedup_shared_of_real": SHARED, "linked_psyq_objects": LINKED,
|
||||
"verbatim_asm_bodies": nverb, "include_asm_stubs": STUBS, "non_matching": NM, "matchable": MATCH,
|
||||
"dedup_groups": ngroups, "dedup_instances": nmembers},
|
||||
"per_binary": [{"binary": r['binary'], "real": r['real'], "shared": r['shared'], "linked": r['linked'],
|
||||
"byte_identical": r['byteident'], "matchable": r['matchable']} for r in rows],
|
||||
}
|
||||
|
||||
|
||||
README_BEGIN, README_END = "<!-- progress:begin -->", "<!-- progress:end -->"
|
||||
|
||||
|
||||
def readme_block(d):
|
||||
"""The README's generated numbers block, from progress.json (D1): a table WITH denominators (R41)."""
|
||||
f, c = d["fleet"], d["counts"]
|
||||
rows = [("Functions byte-identical (all binaries)", f["fn_count"]),
|
||||
("Instructions byte-identical (instruction-weighted)", f["instr_weighted"]),
|
||||
("Distinct code byte-identical (each body once)", f["distinct_code"]),
|
||||
("Main executable game code (excl. Sony objects)", f["main_game_code"])]
|
||||
out = [README_BEGIN,
|
||||
"| Metric | Matched | Of | % |", "|---|---:|---:|---:|"]
|
||||
for name, m in rows:
|
||||
if m:
|
||||
out.append(f"| {name} | {m['matched']:,} | {m['total']:,} | {m['pct']:.1f}% |" if name.startswith("Functions") is False
|
||||
else f"| {name} | {m['matched']:,} | {m['total']:,} | {m['pct']:.2f}% |")
|
||||
out += ["",
|
||||
f"{d['binaries']} binaries rebuild byte-identical from source · {c['real_c_functions']:,} functions in C "
|
||||
f"({c['dedup_shared_of_real']:,} of them shared bodies via {c['dedup_groups']:,} dedup groups) · "
|
||||
f"{c['linked_psyq_objects']:,} Sony PsyQ library functions linked from the SDK objects, not our C · "
|
||||
f"{c['verbatim_asm_bodies']} hand-written-assembly bodies kept verbatim · {c['include_asm_stubs']} assembly stubs left · "
|
||||
f"{c['non_matching']} non-matching functions.",
|
||||
"", f"_Generated by `tools/progress.py --readme` from `docs/progress.json` — numbers are never typed by hand._",
|
||||
README_END]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def write_json_and_readme(check=False):
|
||||
import json as _json
|
||||
d = fleet()
|
||||
jp = ROOT / "docs/progress.json"
|
||||
text = _json.dumps(d, indent=1) + "\n"
|
||||
if check:
|
||||
if not jp.exists() or jp.read_text() != text:
|
||||
sys.exit("progress.py --json --check: docs/progress.json is STALE — run `make report BINARY=main`")
|
||||
else:
|
||||
jp.write_text(text)
|
||||
rp = ROOT / "README.md"
|
||||
if not rp.exists():
|
||||
sys.exit("progress.py --readme: README.md missing")
|
||||
rt = rp.read_text()
|
||||
if README_BEGIN not in rt or README_END not in rt:
|
||||
sys.exit(f"progress.py --readme: README.md lacks the {README_BEGIN} … {README_END} markers — refusing (R43)")
|
||||
a, b = rt.index(README_BEGIN), rt.index(README_END) + len(README_END)
|
||||
new = rt[:a] + readme_block(d) + rt[b:]
|
||||
if check:
|
||||
if new != rt:
|
||||
sys.exit("progress.py --readme --check: README.md's progress block is STALE — run `make report BINARY=main`")
|
||||
print("progress.py --check: docs/progress.json + README block are fresh")
|
||||
return
|
||||
if new != rt:
|
||||
rp.write_text(new)
|
||||
print(f"progress.py: docs/progress.json written; README block {'updated' if new != rt else 'unchanged'}")
|
||||
|
||||
|
||||
def main():
|
||||
if '--json' in sys.argv or '--readme' in sys.argv:
|
||||
write_json_and_readme(check='--check' in sys.argv); return
|
||||
if '--fleet' in sys.argv:
|
||||
fleet(); return
|
||||
if '--weighted' in sys.argv:
|
||||
|
||||
Reference in New Issue
Block a user