BFM-decomp/phase-ends/PhaseEnd_Phase11.md

PhaseEnd — Phase 11: Cross-binary dedup pipeline

Date: 2026-06-16 · Project Version: 1.11.0 · Phase Status: Complete · Generation: Gen2 (4th phase)

Gen2 phase 4 of the arc (8→9→10→11→{12,13}→14). The granular per-task trail (the byte-finding that reshaped the milestone, the sig_image boundary/normalization iterations, the overlay-discovery dead-ends) is preserved on-demand at phase-ends/logs/Phase11.md (R19 — NOT auto-loaded; consult only when researching a mechanism). Owner decisions (2026-06-16): proving ground = "report + machinery; defer collapse"; PsyQ 4.7 = sha-record only (linking is Phase 12).

Build Log

Files created/changed and complete — do not recreate:

Cross-binary dedup pipeline (new tooling):

• tools/sig_image.py — new. Ghidra-FREE per-function signer for a flat image (overlay 0.4.dec / resident 1.1), using rabbitizer (already in the venv). Emits JSONL field-identical to DumpFunctionSignatures.java. h_exact = SHA1 of raw instruction bytes (format-independent — the cross-tool workhorse). h_norm/h_seq = self-consistent (D2). Boundary detection: seeded (--seeds), or --bootstrap = linear partition (split code at the first jr $ra+delay at/after all forward targets) bounded by detect_code_end (first run of ≥3 invalid instrs = code→data transition). Consumed by make sig-overlays + dup_report --cross.
• tools/dedup_integrate.py — new. Byte-honesty VALIDATOR for config/dedup.us.yaml (--check): fail-closed if a registered share's member sig-hash drifts from the recorded hash. NOT a psyq_integrate fork (D1).
• config/dedup.us.yaml — new, committed. Cross-binary code-share registry (group → {id, tier, hash, source, func, members:[{binary, vram, name}]}). Holds the Phase-11 proof group I0_clearTbl40.
• src/shared/clearTbl40.h — new, committed. The shared body (CLEAR_TBL40 macro) for the byte-identical pair func_80037004/func_80037334, instantiated at both sites in src/800.c (one source → two vrams).
• tools/dup_report.py — modified. Added --cross: ingest all BINARIES sigs + every .run/sig.ov_*.jsonl, bucket by h_exact then h_norm across binaries, split cross-binary (the Phase-12/13 work queue) vs intra, rank by collapsible bytes (count−1)×nins×4, top-200 capped + totals → docs/duplicates.cross.md. Per-binary path byte-stable.
• tools/progress.py — modified. Counts dedup-shared members (from dedup.us.yaml) as REAL (the macro form isn't a parseable function def) — honest measurement (P9).
• Makefile — modified. make report (gated BINARY=main) runs dup_report --cross + dedup_integrate --check (the byte-honesty gate, last line → fail-closed). New make sig-overlays (sign all 134 overlays).
• src/800.c — modified. The two INCLUDE_ASM stubs → CLEAR_TBL40(...) + the #include "shared/clearTbl40.h".

Reports / docs (committed):

• docs/duplicates.cross.md — new, committed. The cross-binary work queue: 9366 h_exact + 8957 h_norm cross-binary groups (28.5 MB / 38.3 MB collapsible), ranked by leverage.
• docs/progress.md / docs/difficulty.md — regenerated (REAL 52→54; the 2 dedup-shared fns left the queue).
• tools/psyq/CHECKSUMS.sha256 — new, committed. Integrity record for psyq-4.7-converted.zip (R20) + Phase-12 provenance (the resident is 4.7 → link from conv47/).
• docs/matching-cookbook.md — §11 (the cross-binary dedup & code-share workflow; R16 flywheel).
• docs/SETUP.md — §6.8 (dedup command crib) + tooling-inventory rows (R21).
• docs/psyq-worklist.md — resident-is-4.7 Phase-12 linking note (R24).
• README.md — Gen2 status (resident = 2nd byte-identical binary; cross-binary dedup live).
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase11.md (R19 archive). phase-ends/PhaseEnd_Phase11.md — this file.

Local artifacts (gitignored / regenerable — a fresh clone redoes them): .run/sig.ov_*.jsonl (134 overlay sigs — make sig-overlays, ~27 s), .run/sig.resident_image.jsonl (the T4/T5 validation sig), build/, expected/, asm/, generated include/ macros.

Tools/packages installed: None — rabbitizer (1.16.2) and PyYAML (6.0.3) were already in the venv as splat64 dependencies (Phase 4); the whole phase used the existing Phase-4/6 toolchain.

Verification results (literal) — the milestone proof (clean dual-binary rebuild, all green):

• (1) make clean && make extract BINARY=main && make build BINARY=main → build/us/SLUS_007.26 SHA1 143dbb89f34491258bbc27810d0a12ec8b43a8dd (BYTE-IDENTICAL) — WITH the I0 shared-C body linked.
• (2) make extract BINARY=resident && make build BINARY=resident → build/resident/resident SHA1 8e17e02ff8954d07c979449198f7e1645046b353 (BYTE-IDENTICAL) — no regression.
• (3) T3 dual invariant: the EXE is 143dbb89… WITH the shared-C match AND WITH the two sites reverted to INCLUDE_ASM stubs (both clean rebuilds) — the share is byte-honest and degrades gracefully.
• (4) make report → docs/duplicates.cross.md: h_exact 9366 cross-binary groups / 28,577,800 collapsible bytes; h_norm 8957 / 38,288,368; top group = a 770-instruction fn byte-identical in all 134 overlays. dedup-check: 1 validated, 0 failed; negative tests (corrupt hash / wrong vram) fail-closed (exit 1).
• (5) sig_image validated vs the resident Ghidra oracle: h_exact 140/140 = 100% on the contiguous/non-GTE subset, zero UNEXPLAINED (98.6% overall; 2 misses = non-contiguous bodies, D5). h_exact equivalence classes 6/6 == Ghidra; h_norm reproduces 7/9 Ghidra structural groups (disagreements benign — masked-field only).
• (6) SC01/005 ≡ 006 (byte-identical overlays) sign identically (same function set, all h_exact equal).
• (7) REAL 52→54 (2 dedup-shared); 7 NON_MATCHING, all #ifdef-guarded (0 linked in the default build, G4).
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged (.run/, build/, asm/ gitignored). No Ghidra DB change this phase (pure tooling — R23 no-op).

Milestone achieved: make report emits cross-binary duplicate groups spanning ≥2 binaries (9366 h_exact groups across 136 binaries, up to all 134 overlays per group), AND a single shared C body fills two addresses byte-gated with the EXE (143dbb89…) byte-identical with and without it — the cross-binary dedup pipeline is built and proven on the byte-verified EXE↔resident pair, with the 134-overlay fleet signed and ranked as the Phase-12/13 work queue. Confirmed by Drew (gate 2).

Next: Phase 12 — Key engine systems (script VM, save/Q#5, sound SQV). Start by linking the resident's PsyQ 4.7 library code (libsnd/libgte/libspu from tools/psyq/conv47/, sha-recorded; R24) via the proven psyq_identify → psyq_link_region → psyq_integrate pipeline pointed at the resident (the Phase-9 --vram-base/ --exe params make it binary-agnostic), then RE + match the script-VM core, the save/Q#5 path (SaveLoadRoutine), and the SQV driver — each engine match auto-credited across the 134 overlays via docs/duplicates.cross.md.

Deviations

Item	Plan	Actual	Reason
Milestone shape	roadmap: "one C body → both EXE and resident green"	report + machinery; defer collapse (owner decision, plan time)	The bytes showed EXE↔resident share NOTHING collapsible (h_exact=0; 4.0 vs 4.7 SDK + loader-vs-engine roles). Real cross-binary sharing is overlay↔overlay (proven: 9366 groups). The literal EXE↔resident collapse is infeasible; first true cross-binary collapse → Phase 13 (built overlay).
D1 dedup mechanism	"fork psyq_integrate.py — .ld interpose"	source-level macro share + per-binary make check gate	psyq_integrate's object-swap works only for separate library SUBSEGMENT stubs; game-code fns are interior to one object per binary → the linker can't excise them. Source-level share (macro in src/shared/) is the byte-correct mechanism. dedup_integrate.py is a byte-honesty validator, not an interpose engine. Simpler. (cookbook §11)
D2 sig_image h_norm	"Ghidra-byte-exact normToken replica, ≥98% match"	self-consistent normalization (scope-guard, pre-approved)	Byte-reality made byte-exact low-value (EXE↔resident share nothing; overlays share via h_exact at the same vram, and CALL not embed the resident). h_exact is the format-independent cross-tool workhorse; every h_norm candidate is byte-gated. Full replica = documented future refinement.
Overlay function discovery	(not specified)	linear partition + validity-based detect_code_end (call-graph BFS dead-ended)	Overlays dispatch most code via function-pointer tables, not jal, so BFS found ~2 fns; linear partition recovers the whole contiguous-code prefix. Jump-table-only fns + non-contiguous bodies (D5) are a documented coverage gap until splat configs (Phase 13).
4.7 .LIB fetch	"fetch + sha-verify from archive.org" (Phase-10 carried)	already present + git-tracked → reduced to a sha-record + provenance note	The converted 4.7 SDK (conv47/ 55 objs + the zip) was already committed (R20 backup satisfied). Fetch was moot; integrity recorded in tools/psyq/CHECKSUMS.sha256.
progress.py accounting	(not anticipated)	taught to count dedup-shared members as REAL (registry source)	The CLEAR_TBL40(name) macro form isn't a parseable function def → the 2 matched fns would have vanished from the count; the registry restores honest measurement (P9).

Commit Message

(Phase 11 landed as 7 per-task checkpoint commits 7cc291145..190afe10f + this PhaseEnd.)

feat(phase-11): cross-binary dedup pipeline — one match unlocks many (v1.11.0)

- T1 (7cc291145): dup_report --cross -> docs/duplicates.cross.md (multi-binary bucketing,
  cross/intra split, collapsible-bytes ranking); per-binary reports byte-stable
- T2 (b06d74c1b): config/dedup.us.yaml registry + tools/dedup_integrate.py byte-honesty
  validator (fail-closed on sig-hash drift); make report gate. DEVIATION D1: game-code
  dedup is SOURCE-LEVEL (shared macro body), not a psyq_integrate object-swap (fns are
  interior to one object/binary); the byte-gate is per-binary make check
- T3 (a8ae190fa): intra-binary collapse PROOF — src/shared/clearTbl40.h CLEAR_TBL40 fills
  func_80037004 + func_80037334 from one body; EXE 143dbb89 WITH the share AND WITHOUT
  (stub fallback); negative gate tests fail-closed; REAL 52->54
- T4 (685e35fdc): tools/sig_image.py Ghidra-free h_exact signer (raw-byte SHA1) + boundary
  detection; validated 100% on the resident contiguous/non-GTE set, 0 UNEXPLAINED
- T5 (81e33c7a1): sig_image self-consistent h_norm + h_seq (DEVIATION D2 — scope-guard, not
  a Ghidra-byte-exact normToken replica; byte-gate is the truth)
- T6 (f672c709c): make sig-overlays signs all 134 overlays (linear partition + code-end
  detection; BFS dead-ends on function-pointer dispatch). Cross-report spans 136 binaries:
  9366 h_exact groups / 28.5 MB collapsible; one 770-ins fn byte-identical in all 134
- T7 (190afe10f): tools/psyq/CHECKSUMS.sha256 (4.7, R20/R24); cookbook §11; SETUP §6.8 +
  inventory; psyq-worklist + README
- MILESTONE: clean dual-binary rebuild green (main 143dbb89, resident 8e17e02f);
  cross-report spans >=2 binaries; one shared body byte-gated in both states; 0 NON_MATCHING
  linked (G4). No new rules (deviations captured as cookbook §11 techniques)
- worklog archived -> phase-ends/logs/Phase11.md (R19); bumps 1.10.0 -> 1.11.0

Rules Added This Phase

Rule

Reason

None

Phase 11's findings are techniques, captured in docs/matching-cookbook.md §11 (the source-level share mechanism / byte-gate / sig_image notes — D1, D2) — the Phase-8 §9.6 precedent of recording techniques in the cookbook, not as governance rules. The existing G3/P9 byte-gate discipline (h_exact risk-free, h_norm candidate-only / link-confirmed) and R24 per-binary toolchain provenance already cover the new ground.

PhaseEnd Changelog

v1.10.0 → v1.11.0 — Phase 11 complete (Gen2 phase 4). The cross-binary deduplication + code-share pipeline is built and proven. New Ghidra-free tooling — sig_image.py (rabbitizer signer; h_exact byte-matches the Ghidra dumper, validated 100% on the resident contiguous set), dup_report --cross (docs/duplicates.cross.md), dedup_integrate.py (the byte-honesty validator), config/dedup.us.yaml + src/shared/ (the source-level share registry + bodies), make sig-overlays. A byte-finding reshaped the milestone (owner-approved): the EXE (4.0) and resident (4.7) share no collapsible code, so the machinery was proven on a real intra-binary h_exact collapse (func_80037004≡func_80037334, one body → two addresses, byte-gated, dual invariant) while the cross-binary report was proven by signing all 134 overlays — which share an enormous amount of code: 9366 h_exact cross-binary groups / 28.5 MB collapsible, one 770-instruction engine function byte-identical in all 134 overlays. This is the Phase-12/13 work queue ("one match unlocks many", quantified). Two deviations (D1 source-level share not an object-swap; D2 self-consistent h_norm not a Ghidra-byte-exact replica), both byte-correct and captured as cookbook §11 techniques — no new rules. The resident's PsyQ 4.7 provenance is sha-recorded (R24) for Phase 12. Both binaries rebuild byte-identical; 0 NON_MATCHING linked (G4).

Plain-English Recap

We taught our build system a new trick: spotting when the same piece of game code shows up in more than one part of the game, so we only have to figure it out once and reuse it everywhere. We proved the idea by rebuilding two copies of one function from a single shared source — and both the main program and the engine still rebuild perfectly, bit-for-bit. Then we fingerprinted all 134 location "levels" (without needing the slow analysis tool) and discovered they share a huge amount of identical code — about 28 megabytes, with one core engine function appearing byte-for-byte in all 134 levels. That means most functions we reverse-engineer from here on will complete dozens of levels at once — exactly the leverage the next phases need. One surprise the bytes settled early: the main program and the engine chunk share essentially nothing (they were built with different toolkits for different jobs), so the real payoff is in how much the levels share with each other. Nothing we already had broke, and we now have a ranked to-do list of the highest-value functions to match next.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase11.md (R19). The Phase-11 work is committed (7 checkpoints 7cc291145..190afe10f + this PhaseEnd commit); Drew pushes (R6). Gen2 continues — do NOT start Phase 12 here. Start a fresh session (effort Max, plan mode) for Phase 12 — Key engine systems, opening with the resident's PsyQ 4.7 library linking. Keep this file forever.