mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 15:18:24 -04:00
docs(phase-11): T7 close-out — 4.7 sha-record, cookbook §11, SETUP §6.8, README/worklist
- tools/psyq/CHECKSUMS.sha256: integrity record for psyq-4.7-converted.zip (R20) + Phase-12 provenance (the resident is 4.7 -> link from conv47/, not the EXE's 4.0 libs; R24) - cookbook §11: the cross-binary dedup & code-share workflow — source-level macro share (NOT an object swap; game fns are interior to one object/binary, D1), the byte-gate, sig_image notes (h_exact workhorse / self-consistent h_norm D2 / linear-partition+code-end overlay boundaries), the cross-report as the Phase-12/13 work queue, per-binary SDK provenance (R24) - SETUP §6.8 + tooling inventory: sig_image.py, dedup_integrate.py, dup_report --cross, make sig-overlays, config/dedup.us.yaml + src/shared/ (R21) - docs/psyq-worklist.md: resident PsyQ is 4.7 (Phase-12 linking note) - README: Gen2 status — resident is the 2nd byte-identical binary; cross-binary dedup live (~9000 cross-binary groups / ~28 MB collapsible; one engine fn byte-identical in all 134 overlays)
This commit is contained in:
Drew T
@@ -19,7 +19,12 @@ This repository contains **no game assets, no disassembly output, and no ROM-der
|
||||
- **959 PsyQ SDK functions linked byte-identical** (libcd, libgs, libgte, libspu/libsnd, libgpu, libc2, libmcrd, libapi/libcard, libetc) straight from the real PsyQ 4.0 libraries instead of re-decompiling them — bringing byte-identical-from-source coverage of the EXE to **~50%**.
|
||||
- **File-loader / overlay system reverse-engineered**, with the resident engine blob + location overlays' load addresses **proven byte-identical against a live PCSX-Redux RAM dump**.
|
||||
|
||||
About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway: the build toolchain is now **binary-agnostic** (one parameterized pipeline builds any binary, proven a byte-exact no-op on the EXE), ready to stand up the overlays. Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`.
|
||||
About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway:
|
||||
|
||||
- The build toolchain is **binary-agnostic** (one parameterized pipeline builds any binary), and the always-resident **engine blob** now rebuilds **byte-for-byte from source** (SHA1 `8e17e02f…`) — the *second* binary reconstructed exactly, after the EXE.
|
||||
- A **cross-binary deduplication pipeline** is live: a Ghidra-free signer fingerprints all 134 location overlays, and the report finds **~9,000 byte-identical function groups shared across binaries (~28 MB of collapsible code)** — a single engine function is byte-identical in all 134 overlays. This is "one match unlocks many": each engine match will be auto-credited across the overlay fleet.
|
||||
|
||||
Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`.
|
||||
|
||||
This project is developed primarily by Claude Code driving Ghidra through an MCP server; see `CLAUDE.md`.
|
||||
|
||||
|
||||
+26
-3
@@ -543,6 +543,26 @@ The reusable **flat-blob recipe** (every Gen2 overlay follows it):
|
||||
- Per-binary `<bin>_GHIDRA_PROG` → `make sig-refresh BINARY=<bin>`; `diff_settings.py` + the three
|
||||
report scripts gain a `<bin>` entry; `make expected` is per-binary-safe (merge-copy, no sibling clobber).
|
||||
|
||||
### §6.8 Cross-binary dedup & code-sharing (Phase 11) — "one match unlocks many"
|
||||
Full how-to in `docs/matching-cookbook.md` §11. Command crib:
|
||||
- **`make sig-overlays`** — Ghidra-FREE sign all 134 location overlays (`SCxx 0.4.dec`) at the shared overlay
|
||||
vram `0x80128158` via `tools/sig_image.py` → `.run/sig.ov_<SCxx>_<nnn>.jsonl` (gitignored; ~27 s). Re-run when
|
||||
overlays change. (`make sig-refresh` still does the Ghidra-imported EXE/resident sigs.)
|
||||
- **`make report`** (gated `BINARY=main`) runs **`tools/dup_report.py --cross`** → `docs/duplicates.cross.md`:
|
||||
cross-binary duplicate groups across main + resident + all overlay sigs, ranked by collapsible bytes (the
|
||||
Phase-12/13 work queue), + **`tools/dedup_integrate.py --check`** (the byte-honesty gate — fail-closed if a
|
||||
registered share's sig hash drifts).
|
||||
- **Share a matched fn across binaries**: author the body ONCE as a macro in `src/shared/<fn>.h`, instantiate at
|
||||
each site in each binary's `.c`, register the group in **`config/dedup.us.yaml`** (`{id, tier, hash, source,
|
||||
func, members:[{binary, vram, name}]}`). Byte-gate = per-binary `make check`. `h_exact` = risk-free; `h_norm`
|
||||
= candidate (accept only if every claiming binary stays byte-identical). NOT an object swap — game-code fns are
|
||||
interior to one object per binary (cookbook §11 / deviation D1).
|
||||
- `tools/sig_image.py`: `h_exact` byte-matches the Ghidra dumper (validated 100% on the resident contiguous set);
|
||||
`h_norm` is self-consistent within the overlay fleet (not Ghidra-byte-exact — D2); overlay boundaries via linear
|
||||
partition + `detect_code_end` (BFS fails — overlays dispatch via function-pointer tables, not `jal`).
|
||||
- **PsyQ provenance (R24)**: the resident is PsyQ **4.7** (`tools/psyq/conv47/`, sha-recorded in
|
||||
`tools/psyq/CHECKSUMS.sha256`) — Phase 12 links its embedded SDK code from 4.7, not the EXE's 4.0 libs.
|
||||
|
||||
## §7 Session-start ritual
|
||||
|
||||
Order is load-bearing — MCP tools fail (sometimes silently) without an open program.
|
||||
@@ -626,10 +646,13 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/make_apicard_used.py` | **(Phase 8)** Build the combined libapi+libcard curated dir (§9.6). |
|
||||
| | `tools/ld_interleave.py` | Interleave linker inputs to match original section ordering. |
|
||||
| | `tools/split_src_region.py` | Split a `src/` region file at object boundaries. |
|
||||
| **Reports** | `tools/progress.py` | Decomp progress report (`make report`). |
|
||||
| **Reports** | `tools/progress.py` | Decomp progress report (`make report`); counts dedup-shared fns as REAL via the registry (Phase 11). |
|
||||
| | `tools/difficulty.py` | Per-function difficulty scoring. |
|
||||
| | `tools/dup_report.py` | Duplicate-function report. |
|
||||
| | `make report` / `make sig-refresh` | Convenience targets wrapping the report / signature-dump scripts. |
|
||||
| | `tools/dup_report.py` | Duplicate-function report; `--cross` (Phase 11) buckets all binaries → `docs/duplicates.cross.md`. |
|
||||
| **Cross-binary dedup** (Phase 11, cookbook §11) | `tools/sig_image.py` | **Ghidra-FREE** per-function signer for a flat image (overlay/resident); `h_exact` byte-matches the Ghidra dumper, self-consistent `h_norm`; linear-partition + `detect_code_end` boundaries. |
|
||||
| | `tools/dedup_integrate.py` | Byte-honesty validator for `config/dedup.us.yaml` code-shares (`--check`; fail-closed on sig-hash drift). |
|
||||
| | `config/dedup.us.yaml` / `src/shared/*.h` | The code-share registry + the shared bodies (one macro → N sites, byte-gated). |
|
||||
| | `make report` / `make sig-refresh` / `make sig-overlays` | Convenience targets: reports (+`--cross`) / Ghidra signature-dump / Ghidra-free sign all 134 overlays. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -578,3 +578,59 @@ residual is a specific compiler-internal placement rather than a randomizable C
|
||||
express the lever in C. Here a research agent reading `reorg.c`/`jump.c`/`local-alloc.c` produced all four
|
||||
levers directly; hand-iteration with the clean `.text` metric closed it in a few compiles. Pin every such
|
||||
construct with a `LOAD-BEARING` comment naming the pass — a future reader WILL try to "simplify" them.
|
||||
|
||||
## §11 Cross-binary dedup & code-sharing (Phase 11 — "one match unlocks many")
|
||||
|
||||
BFM is overlay-heavy: 134 location overlays all load to the SAME vram `0x80128158` and run on the same engine,
|
||||
so they share enormous amounts of code (a 770-instruction engine fn is byte-identical in **all 134**). Match a
|
||||
shared fn ONCE, credit every binary it lives in. The pipeline (all Ghidra-free except the EXE/resident sigs):
|
||||
|
||||
**1. Sign every binary → `.run/sig.<bin>.jsonl`.** `make sig-refresh` (Ghidra, EXE/resident) + `make
|
||||
sig-overlays` (the 134 `0.4.dec` via `tools/sig_image.py`, no Ghidra). Each fn gets `h_exact` (SHA1 of raw
|
||||
instruction bytes), `h_norm` (structural), `h_seq`, `nins`, `calls`.
|
||||
|
||||
**2. Group across binaries → `docs/duplicates.cross.md`.** `tools/dup_report.py --cross` (run by `make report`,
|
||||
gated `BINARY=main`) buckets ALL sigs by `h_exact` then `h_norm`, splits cross-binary (members in >1 binary —
|
||||
the **Phase-12/13 work queue**) vs intra-binary, ranks by collapsible bytes `(count−1)×nins×4`, top-200 capped.
|
||||
|
||||
**3. Register a share → `config/dedup.us.yaml`.** `group → {id, tier, hash, source, func, members:[{binary,
|
||||
vram, name}]}`. `tools/dedup_integrate.py --check` is the **byte-honesty gate** (fail-closed if a member's live
|
||||
sig hash drifts from the recorded `hash`); wired into `make report` so a stale share fails the report (P9).
|
||||
|
||||
### The mechanism: game-code dedup is SOURCE-LEVEL, not an object swap (R-D1, the key lesson)
|
||||
`psyq_integrate`'s stub-object swap works only for separate library **subsegments**. Game-code functions are
|
||||
**interior to one compiled object per binary** (`build/src/800.o`, `build/resident/resident.o`, each overlay's
|
||||
one object) — the linker can't excise interior bytes. So you share at the SOURCE level: author the matched body
|
||||
ONCE as a macro in `src/shared/<fn>.h` and instantiate it at each member site in each binary's `.c`:
|
||||
|
||||
```c
|
||||
// src/shared/clearTbl40.h
|
||||
#define CLEAR_TBL40(name) void name(void) { s32 i; for (i=0x40; i>=0; i-=0x10) (&D_80076251)[i]=0; }
|
||||
// src/800.c: CLEAR_TBL40(func_80037004) ... CLEAR_TBL40(func_80037334)
|
||||
```
|
||||
Same bytes land at each vram. The **byte-gate is the existing per-binary `make check`** — the image is identical
|
||||
or it is not. `h_exact` shares are risk-free; `h_norm` shares are CANDIDATES, accepted only if every claiming
|
||||
binary stays byte-identical (a wrong `h_norm` group wastes a build, never poisons an image). A shared `.h` is
|
||||
skipped by the `find src -name '*.c'` OBJS glob automatically (no exclusion needed). `progress.py` counts dedup
|
||||
members as REAL via the registry (the macro form isn't a parseable function def).
|
||||
|
||||
### `sig_image.py` (Ghidra-free signer) — notes for reuse on overlays
|
||||
- **`h_exact` is the workhorse**: SHA1 of raw bytes → format-independent → byte-matches the Ghidra dumper with
|
||||
no normalization. Validated 100% on the resident's contiguous/non-GTE functions. Use it as the cross-tool tier.
|
||||
- **`h_norm` is self-consistent, NOT Ghidra-byte-exact** (R-D2): masks j/jal targets, lui highs, hi/lo-paired
|
||||
address-los (a consistent lui→reg tracker); keeps registers / true constants / PC-relative branch offsets.
|
||||
Uniform within the overlay fleet (catches different-offset structural dups); does not cross-compare with the
|
||||
Ghidra-signed EXE/resident `h_norm` (low value — overlays *call*, don't embed, the resident). Full normToken
|
||||
byte-match is a deferred refinement.
|
||||
- **Boundary detection**: (a) seeded (pass `--seeds <sig.jsonl>` when boundaries are known, e.g. the resident);
|
||||
(b) `--bootstrap` for overlays = **linear partition** (split contiguous code at the first `jr $ra`(+delay)
|
||||
that lies at/after all forward branch targets — handles early-return + double-epilogue) bounded by
|
||||
`detect_code_end` (first run of ≥3 invalid instrs = the code→data transition; overlay code decodes ~100%
|
||||
valid). Call-graph BFS FAILS on overlays (they dispatch via function-pointer tables, not `jal`). Residual:
|
||||
jump-table-only fns + non-contiguous Ghidra bodies (D5) are missed — conservative, fixed when splat configs
|
||||
land (Phase 13).
|
||||
|
||||
### Per-binary toolchain provenance (R24)
|
||||
Verify the toolchain per binary before linking its library code: the EXE is PsyQ 4.0, the **resident is 4.7**
|
||||
(`tools/psyq/conv47/`, sha-recorded in `tools/psyq/CHECKSUMS.sha256`). Never assume one binary's SDK applies to
|
||||
another — the 4.0 libs won't byte-match the resident's 4.7 objects.
|
||||
|
||||
@@ -72,3 +72,9 @@ references resolve to >1 base in the EXE. Curate the library's `_used` dir to dr
|
||||
| libapi 800c3 cluster (~22 objs) | libapi | C57..L10/L02/L03 @0x5CE18.. in the 800c3 region (separate from the 800c2 apicard region) | **deferred**; ~22 4-ins BIOS syscall stubs; lowest value; another region resegmentation |
|
||||
|
||||
*If scattered-`.bss` proves prevalent across libgte/libspu/libsnd, escalate to a Max general fix (split each object's `.bss` into per-common NOLOAD sections at their EXE-resolved addresses); otherwise excluding the few affected objects is the GS_001-precedent decision.*
|
||||
|
||||
---
|
||||
|
||||
## The RESIDENT's PsyQ code is **4.7**, not the EXE's 4.0 (Phase 12, R24)
|
||||
|
||||
The map above is the **EXE** (PsyQ 4.0). The resident engine blob detects as **PsyQ 4.7.0** (DetectPsyQ + the `DsMix`/libsnd hit). Phase 12 links the resident's embedded PsyQ code (libsnd / libgte / libspu) from the **4.7** objects at `tools/psyq/conv47/` (sha-recorded in `tools/psyq/CHECKSUMS.sha256`), reusing the same `psyq_identify → psyq_link_region → psyq_integrate` pipeline pointed at the resident (Phase-9 `--vram-base`/`--exe` make it binary-agnostic). The 4.0 `.LIB`s will **not** byte-match the 4.7 objects — verify the version per binary before linking (R24).
|
||||
|
||||
Reference in New Issue
Block a user