BFM-decomp/docs/wiki/The-dedup-engine.md

The dedup engine — match once, share everywhere

The fact it is built on

The game's code is not in the executable. It is in the resident engine blob and in the location overlays streamed from the disc, and the overlays are position-locked: every one loads verbatim to the same slot, 0x80128158, with no relocation (proven in Phase 3 against a live RAM image — a 389,400-byte contiguous exact match at the exact address). So a shared engine function has the same address and the same bytes in every overlay that contains it, and func_80144B9C in ov_SC01_077 is the same function as func_80144B9C in ov_SC06_018. Match it once and it is matched everywhere — up to 138 copies for one crack. This is the project's economic engine: the fleet went from 3.8% to 54% of functions in the single phase that built it, and every match made before it existed was worth ×1.

Signatures

tools/sig_image.py hashes every function in every binary at three tiers:

tier	keyed on	answers
h_exact	the raw bytes	"is there a byte-identical copy?" — the free tier
h_norm	the bytes with relocations normalized out	"is there a same-skeleton copy that differs only in symbols/immediates?" — a structural family; one exemplar, then a mechanical remap of the per-location symbols
h_seq	the mnemonic sequence	"is there a same-shape function?" — a cousin; useful as a card's worked example, rarely a free bank

.run/sig.<alias>.jsonl is regenerated by make sig-*; tools/dup_report.py --cross builds the cross-binary groups. Two lessons about the join itself: a similarity join must be a band, not a point (an edit-distance band to d ≤ 25 found 3.4× more reachable twins than the exact hash on a corpus believed fully mined — tools/seed_ref.py --near), and a hash tuned to under-match for dedup silently under-matches as a frontier join — one hash cannot serve both error directions.

Propagation

tools/dedup_propagate.py takes a function matched in one binary and:

1. extracts the matched body from that binary's .c;
2. authors it once as a DEFINE_func_<ADDR>() macro in a shared header under src/shared/ (the engine cluster is engine_core.h);
3. replaces the function's INCLUDE_ASM stub in every member binary with the macro instantiation, in address order;
4. byte-gates every touched binary (make build BINARY=<member> == its check.sha); on any miss it restores every file from an in-memory snapshot and aborts — nothing wrong can land;
5. registers the group in config/dedup.us.yaml.

The share is source-level, not an object swap: game-code functions are interior to one compiled object per binary and the linker cannot excise interior bytes, so the same C is instantiated at each member's site and the same bytes land at each member's address. (The object-swap mechanism exists too, for Sony's library subsegments in the main executable — that is how the real PsyQ objects are linked in when the SDK is present.)

The registry, and why it fail-closes

Each group in config/dedup.us.yaml binds one shared body to ≥ 2 member instances with the tier hash they must all share. tools/dedup_integrate.py --check (in make tools-health and make report) validates that every member's current signature still equals the recorded hash — a drifted share is caught before it can mislead. At the Phase-33 close: 2,220 groups, 255,708 instances, 0 failures.

Families, twins, and the free tiers

• Structural families (tools/family_sweep.py, family_remap): the same skeleton across overlays with per-location relocations and immediates — crack one exemplar, remap the rest mechanically, byte-gate each. Measured in Phase 25/26 at 88% of exact-hash and 80% of normalized families remapping cleanly; the ceiling is translation-unit type collisions, not the bytes. The right unit of scheduling was the family, ranked by reach × size, not the function ranked by difficulty.
• Twins (tools/twin_rescan.py): an open stub whose banked twin sits at the same address elsewhere. The twin graph changes after every bank — an open-open cluster becomes free remaps the moment one member lands — so the rescan runs after every gate that banked. Learned at a cost of ~250k tokens: agents once redrafted functions whose answer was already banked two words away.
• Constant flips and pass-throughs: bodies that differ from a banked one by a literal or a wrapper — a remap with an edit, still zero drafting tokens.

Honesty rules the engine imposes

• Propagation writes N binaries after verifying one; the fleet run from clean follows every propagating gate (R22).
• A group is keyed by integer address, never by name (0x80144b9c in the signature, func_80144B9C in splat's output — comparing the strings is the bug; R48 generalizes it: never key by bare function name).
• A dedup backlog of already-matched duplicate copies is orthogonal to completion — gates run with --no-propagate by default and propagate deliberately.
• make audit-binaries asserts every binary is a full citizen of every consumer — signature set, family map, dedup registry, shared-header include, reports (R36) — because a binary the engine does not know about is invisible to it.