From 190afe10f05015a4899ec0c9936435a27df2deef Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 16 Jun 2026 01:31:50 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-11):=20T7=20close-out=20=E2=80=94=20?= =?UTF-8?q?4.7=20sha-record,=20cookbook=20=C2=A711,=20SETUP=20=C2=A76.8,?= =?UTF-8?q?=20README/worklist?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - tools/psyq/CHECKSUMS.sha256: integrity record for psyq-4.7-converted.zip (R20) + Phase-12 provenance (the resident is 4.7 -> link from conv47/, not the EXE's 4.0 libs; R24) - cookbook §11: the cross-binary dedup & code-share workflow — source-level macro share (NOT an object swap; game fns are interior to one object/binary, D1), the byte-gate, sig_image notes (h_exact workhorse / self-consistent h_norm D2 / linear-partition+code-end overlay boundaries), the cross-report as the Phase-12/13 work queue, per-binary SDK provenance (R24) - SETUP §6.8 + tooling inventory: sig_image.py, dedup_integrate.py, dup_report --cross, make sig-overlays, config/dedup.us.yaml + src/shared/ (R21) - docs/psyq-worklist.md: resident PsyQ is 4.7 (Phase-12 linking note) - README: Gen2 status — resident is the 2nd byte-identical binary; cross-binary dedup live (~9000 cross-binary groups / ~28 MB collapsible; one engine fn byte-identical in all 134 overlays) --- README.md | 7 ++++- docs/SETUP.md | 29 +++++++++++++++++--- docs/matching-cookbook.md | 56 +++++++++++++++++++++++++++++++++++++++ docs/psyq-worklist.md | 6 +++++ 4 files changed, 94 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 6dca81467..bced81be3 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,12 @@ This repository contains **no game assets, no disassembly output, and no ROM-der - **959 PsyQ SDK functions linked byte-identical** (libcd, libgs, libgte, libspu/libsnd, libgpu, libc2, libmcrd, libapi/libcard, libetc) straight from the real PsyQ 4.0 libraries instead of re-decompiling them — bringing byte-identical-from-source coverage of the EXE to **~50%**. - **File-loader / overlay system reverse-engineered**, with the resident engine blob + location overlays' load addresses **proven byte-identical against a live PCSX-Redux RAM dump**. -About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway: the build toolchain is now **binary-agnostic** (one parameterized pipeline builds any binary, proven a byte-exact no-op on the EXE), ready to stand up the overlays. Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`. +About half the EXE is still `INCLUDE_ASM` stubs (correct bytes, not yet C), and the bulk of the game lives in compressed overlays inside the `.CD` archives — **Gen2** (overlays & engine at scale) is underway: + +- The build toolchain is **binary-agnostic** (one parameterized pipeline builds any binary), and the always-resident **engine blob** now rebuilds **byte-for-byte from source** (SHA1 `8e17e02f…`) — the *second* binary reconstructed exactly, after the EXE. +- A **cross-binary deduplication pipeline** is live: a Ghidra-free signer fingerprints all 134 location overlays, and the report finds **~9,000 byte-identical function groups shared across binaries (~28 MB of collapsible code)** — a single engine function is byte-identical in all 134 overlays. This is "one match unlocks many": each engine match will be auto-credited across the overlay fleet. + +Current phase and detailed progress live in `phase-ends/` (newest `PhaseEnd_*.md` = current state); methodology, rules, and the full roadmap are in `PROJECT_CONTEXT.md`; environment setup in `docs/SETUP.md`. This project is developed primarily by Claude Code driving Ghidra through an MCP server; see `CLAUDE.md`. diff --git a/docs/SETUP.md b/docs/SETUP.md index a736042e4..6c1890a0b 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -543,6 +543,26 @@ The reusable **flat-blob recipe** (every Gen2 overlay follows it): - Per-binary `_GHIDRA_PROG` → `make sig-refresh BINARY=`; `diff_settings.py` + the three report scripts gain a `` entry; `make expected` is per-binary-safe (merge-copy, no sibling clobber). +### §6.8 Cross-binary dedup & code-sharing (Phase 11) — "one match unlocks many" +Full how-to in `docs/matching-cookbook.md` §11. Command crib: +- **`make sig-overlays`** — Ghidra-FREE sign all 134 location overlays (`SCxx 0.4.dec`) at the shared overlay + vram `0x80128158` via `tools/sig_image.py` → `.run/sig.ov__.jsonl` (gitignored; ~27 s). Re-run when + overlays change. (`make sig-refresh` still does the Ghidra-imported EXE/resident sigs.) +- **`make report`** (gated `BINARY=main`) runs **`tools/dup_report.py --cross`** → `docs/duplicates.cross.md`: + cross-binary duplicate groups across main + resident + all overlay sigs, ranked by collapsible bytes (the + Phase-12/13 work queue), + **`tools/dedup_integrate.py --check`** (the byte-honesty gate — fail-closed if a + registered share's sig hash drifts). +- **Share a matched fn across binaries**: author the body ONCE as a macro in `src/shared/.h`, instantiate at + each site in each binary's `.c`, register the group in **`config/dedup.us.yaml`** (`{id, tier, hash, source, + func, members:[{binary, vram, name}]}`). Byte-gate = per-binary `make check`. `h_exact` = risk-free; `h_norm` + = candidate (accept only if every claiming binary stays byte-identical). NOT an object swap — game-code fns are + interior to one object per binary (cookbook §11 / deviation D1). +- `tools/sig_image.py`: `h_exact` byte-matches the Ghidra dumper (validated 100% on the resident contiguous set); + `h_norm` is self-consistent within the overlay fleet (not Ghidra-byte-exact — D2); overlay boundaries via linear + partition + `detect_code_end` (BFS fails — overlays dispatch via function-pointer tables, not `jal`). +- **PsyQ provenance (R24)**: the resident is PsyQ **4.7** (`tools/psyq/conv47/`, sha-recorded in + `tools/psyq/CHECKSUMS.sha256`) — Phase 12 links its embedded SDK code from 4.7, not the EXE's 4.0 libs. + ## §7 Session-start ritual Order is load-bearing — MCP tools fail (sometimes silently) without an open program. @@ -626,10 +646,13 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/make_apicard_used.py` | **(Phase 8)** Build the combined libapi+libcard curated dir (§9.6). | | | `tools/ld_interleave.py` | Interleave linker inputs to match original section ordering. | | | `tools/split_src_region.py` | Split a `src/` region file at object boundaries. | -| **Reports** | `tools/progress.py` | Decomp progress report (`make report`). | +| **Reports** | `tools/progress.py` | Decomp progress report (`make report`); counts dedup-shared fns as REAL via the registry (Phase 11). | | | `tools/difficulty.py` | Per-function difficulty scoring. | -| | `tools/dup_report.py` | Duplicate-function report. | -| | `make report` / `make sig-refresh` | Convenience targets wrapping the report / signature-dump scripts. | +| | `tools/dup_report.py` | Duplicate-function report; `--cross` (Phase 11) buckets all binaries → `docs/duplicates.cross.md`. | +| **Cross-binary dedup** (Phase 11, cookbook §11) | `tools/sig_image.py` | **Ghidra-FREE** per-function signer for a flat image (overlay/resident); `h_exact` byte-matches the Ghidra dumper, self-consistent `h_norm`; linear-partition + `detect_code_end` boundaries. | +| | `tools/dedup_integrate.py` | Byte-honesty validator for `config/dedup.us.yaml` code-shares (`--check`; fail-closed on sig-hash drift). | +| | `config/dedup.us.yaml` / `src/shared/*.h` | The code-share registry + the shared bodies (one macro → N sites, byte-gated). | +| | `make report` / `make sig-refresh` / `make sig-overlays` | Convenience targets: reports (+`--cross`) / Ghidra signature-dump / Ghidra-free sign all 134 overlays. | --- diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index c736956ba..ea0355b05 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -578,3 +578,59 @@ residual is a specific compiler-internal placement rather than a randomizable C express the lever in C. Here a research agent reading `reorg.c`/`jump.c`/`local-alloc.c` produced all four levers directly; hand-iteration with the clean `.text` metric closed it in a few compiles. Pin every such construct with a `LOAD-BEARING` comment naming the pass — a future reader WILL try to "simplify" them. + +## §11 Cross-binary dedup & code-sharing (Phase 11 — "one match unlocks many") + +BFM is overlay-heavy: 134 location overlays all load to the SAME vram `0x80128158` and run on the same engine, +so they share enormous amounts of code (a 770-instruction engine fn is byte-identical in **all 134**). Match a +shared fn ONCE, credit every binary it lives in. The pipeline (all Ghidra-free except the EXE/resident sigs): + +**1. Sign every binary → `.run/sig..jsonl`.** `make sig-refresh` (Ghidra, EXE/resident) + `make +sig-overlays` (the 134 `0.4.dec` via `tools/sig_image.py`, no Ghidra). Each fn gets `h_exact` (SHA1 of raw +instruction bytes), `h_norm` (structural), `h_seq`, `nins`, `calls`. + +**2. Group across binaries → `docs/duplicates.cross.md`.** `tools/dup_report.py --cross` (run by `make report`, +gated `BINARY=main`) buckets ALL sigs by `h_exact` then `h_norm`, splits cross-binary (members in >1 binary — +the **Phase-12/13 work queue**) vs intra-binary, ranks by collapsible bytes `(count−1)×nins×4`, top-200 capped. + +**3. Register a share → `config/dedup.us.yaml`.** `group → {id, tier, hash, source, func, members:[{binary, +vram, name}]}`. `tools/dedup_integrate.py --check` is the **byte-honesty gate** (fail-closed if a member's live +sig hash drifts from the recorded `hash`); wired into `make report` so a stale share fails the report (P9). + +### The mechanism: game-code dedup is SOURCE-LEVEL, not an object swap (R-D1, the key lesson) +`psyq_integrate`'s stub-object swap works only for separate library **subsegments**. Game-code functions are +**interior to one compiled object per binary** (`build/src/800.o`, `build/resident/resident.o`, each overlay's +one object) — the linker can't excise interior bytes. So you share at the SOURCE level: author the matched body +ONCE as a macro in `src/shared/.h` and instantiate it at each member site in each binary's `.c`: + +```c +// src/shared/clearTbl40.h +#define CLEAR_TBL40(name) void name(void) { s32 i; for (i=0x40; i>=0; i-=0x10) (&D_80076251)[i]=0; } +// src/800.c: CLEAR_TBL40(func_80037004) ... CLEAR_TBL40(func_80037334) +``` +Same bytes land at each vram. The **byte-gate is the existing per-binary `make check`** — the image is identical +or it is not. `h_exact` shares are risk-free; `h_norm` shares are CANDIDATES, accepted only if every claiming +binary stays byte-identical (a wrong `h_norm` group wastes a build, never poisons an image). A shared `.h` is +skipped by the `find src -name '*.c'` OBJS glob automatically (no exclusion needed). `progress.py` counts dedup +members as REAL via the registry (the macro form isn't a parseable function def). + +### `sig_image.py` (Ghidra-free signer) — notes for reuse on overlays +- **`h_exact` is the workhorse**: SHA1 of raw bytes → format-independent → byte-matches the Ghidra dumper with + no normalization. Validated 100% on the resident's contiguous/non-GTE functions. Use it as the cross-tool tier. +- **`h_norm` is self-consistent, NOT Ghidra-byte-exact** (R-D2): masks j/jal targets, lui highs, hi/lo-paired + address-los (a consistent lui→reg tracker); keeps registers / true constants / PC-relative branch offsets. + Uniform within the overlay fleet (catches different-offset structural dups); does not cross-compare with the + Ghidra-signed EXE/resident `h_norm` (low value — overlays *call*, don't embed, the resident). Full normToken + byte-match is a deferred refinement. +- **Boundary detection**: (a) seeded (pass `--seeds ` when boundaries are known, e.g. the resident); + (b) `--bootstrap` for overlays = **linear partition** (split contiguous code at the first `jr $ra`(+delay) + that lies at/after all forward branch targets — handles early-return + double-epilogue) bounded by + `detect_code_end` (first run of ≥3 invalid instrs = the code→data transition; overlay code decodes ~100% + valid). Call-graph BFS FAILS on overlays (they dispatch via function-pointer tables, not `jal`). Residual: + jump-table-only fns + non-contiguous Ghidra bodies (D5) are missed — conservative, fixed when splat configs + land (Phase 13). + +### Per-binary toolchain provenance (R24) +Verify the toolchain per binary before linking its library code: the EXE is PsyQ 4.0, the **resident is 4.7** +(`tools/psyq/conv47/`, sha-recorded in `tools/psyq/CHECKSUMS.sha256`). Never assume one binary's SDK applies to +another — the 4.0 libs won't byte-match the resident's 4.7 objects. diff --git a/docs/psyq-worklist.md b/docs/psyq-worklist.md index 88e6be4e5..7a1272529 100644 --- a/docs/psyq-worklist.md +++ b/docs/psyq-worklist.md @@ -72,3 +72,9 @@ references resolve to >1 base in the EXE. Curate the library's `_used` dir to dr | libapi 800c3 cluster (~22 objs) | libapi | C57..L10/L02/L03 @0x5CE18.. in the 800c3 region (separate from the 800c2 apicard region) | **deferred**; ~22 4-ins BIOS syscall stubs; lowest value; another region resegmentation | *If scattered-`.bss` proves prevalent across libgte/libspu/libsnd, escalate to a Max general fix (split each object's `.bss` into per-common NOLOAD sections at their EXE-resolved addresses); otherwise excluding the few affected objects is the GS_001-precedent decision.* + +--- + +## The RESIDENT's PsyQ code is **4.7**, not the EXE's 4.0 (Phase 12, R24) + +The map above is the **EXE** (PsyQ 4.0). The resident engine blob detects as **PsyQ 4.7.0** (DetectPsyQ + the `DsMix`/libsnd hit). Phase 12 links the resident's embedded PsyQ code (libsnd / libgte / libspu) from the **4.7** objects at `tools/psyq/conv47/` (sha-recorded in `tools/psyq/CHECKSUMS.sha256`), reusing the same `psyq_identify → psyq_link_region → psyq_integrate` pipeline pointed at the resident (Phase-9 `--vram-base`/`--exe` make it binary-agnostic). The 4.0 `.LIB`s will **not** byte-match the 4.7 objects — verify the version per binary before linking (R24).