feat(phase-28 T4): dedup_extend — wire newly-onboarded binaries in; ov_SC07_006 1543/1614 (95.6%)

The 4 SC07 overlays P27 onboarded were byte-clean but NOT citizens: their .c included only common.h (never ../shared/engine_core.h), so no shared body could reach them, and they appeared in ZERO dedup groups (1689 groups read "134 binaries", never 138). Each sat at ~80 matched / ~2400 stubs while its siblings were ~2150 matched. - NEW tools/dedup_extend.py — the missing mode. dedup_propagate is built for CRACK -> AUTHOR MACRO -> INSTANTIATE: --auto-from scans INLINE DEFS (planned only 11 here; the ~1600 shared bodies are ALREADY DEFINE_func_* macros in engine_core.h) and --addr dies "no source overlay has it matched" because no overlay holds an inline def. Extending an existing MACRO-BACKED group to a newly-onboarded binary is a different operation and nothing implemented it. - SAFETY (explicit — this feeds the byte-gate): h_exact is the SHA1 of RAW INSTRUCTION BYTES, so two instances sharing one are identical INCLUDING their jal/lui/%lo reloc immediates — same callees, same data addresses, same symbols. The body that compiles byte-identically at one member does so at the other with NO remap. (Exactly why dup_report calls h_exact "guaranteed byte-match" and h_norm "candidate-only".) A bug here can only FAIL TO BANK, never falsely bank. - REUSE, DON'T REBUILD (R33): owns only the set computation + the registry edit. The splice and the gate are harvest_verify verbatim (it already derives each stub's home TU from the corpus oracle, chunks + bisects, reverts on failure). h_exact members are byte-identical by construction -> the happy path is ~1 build per binary, not one per function. - RESULT ov_SC07_006: 1543 / 1614 banked = 95.6%, ~0 agent tokens. Stubs 2374 -> 831. The 71 non-banks are ALL PLUMBING, ZERO DIFF, in two named classes with existing tools: * func_80144B9C "undefined reference" — the whale's body lives in src/shared/func_80144B9C.h (the -O0 shared header), not engine_core.h, so no DEFINE macro exists to expand. * "conflicting types for D_800A5E60 / func_8012C750 / func_8012C0EC" — the loose-typing conflict class (cast_call_sites / canon_sig_reconcile / reconcile_tu already exist for it). - GATES: R22 make clean && extract-all && check-all -> 140 passed, 0 failed of 140, 0 FAIL lines. dedup-check 1840 validated / 0 failed; groups now read "135 members [135 binaries]" (was 134); C1 coverage 227211 -> 228754 = exactly +1543. The second oracle accepts the extension. - Mechanism had been proven by hand first (probe-before-investing): +include + ONE stub -> DEFINE_func_80128158() -> ov_SC07_006 built 7ca772be BYTE-IDENTICAL, then reverted.
2026-10-03 16:16:50 -04:00 · 2026-07-15 23:14:19 -06:00
parent 515d003dbe
commit c0486fe5f8
4 changed files with 16243 additions and 14556 deletions
@@ -4,15 +4,15 @@
 # cross-binary collapsible-byte leverage: docs/duplicates.cross.md.

 # THREE progress metrics (all matter — see the labels):
-FLEET fn-count byte-ident: 290728 / 353723 = 82.19%   (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
-FLEET instr-weighted     : 8857366 / 13081451 = 67.7%   (shipped .text across resident+138 overlays; the decomp.dev-DISPLAY number)
-FLEET distinct-code(uniq): 2755482 / 5574674 = 49.4%   (53524/87459 unique fns; the DISTINCT-RE number)
+FLEET fn-count byte-ident: 292275 / 353723 = 82.63%   (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
+FLEET instr-weighted     : 8898477 / 13081451 = 68.0%   (shipped .text across resident+138 overlays; the decomp.dev-DISPLAY number)
+FLEET distinct-code(uniq): 2757078 / 5574674 = 49.5%   (53528/87459 unique fns; the DISTINCT-RE number)
 MAIN game-code weighted  :     436 / 60201 = 0.7%   (Phase-27 T10; SEPARATE — LINKED-excluding Ghidra sig dated 2026-06-14, PROVISIONAL until a fresh/complete main sig; NOT folded into the fleet number)

-FLEET REAL substantive   : 288873   (of which dedup-shared 227167 via 1840 groups / 227211 instances)
+FLEET REAL substantive   : 290420   (of which dedup-shared 228710 via 1840 groups / 228754 instances)
 FLEET LINKED PsyQ objs   :    959
 FLEET NON_MATCHING       :      7   (0 in any default build — G4)
-FLEET INCLUDE_ASM stubs  :  62988
+FLEET INCLUDE_ASM stubs  :  61441
 FLEET matchable          : 353723

 | binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
@@ -151,9 +151,9 @@ FLEET matchable          : 353723
 | ov_SC07_000 | 2147 | 1697 | 0 | 2149 | 2521 | 85.2% |
 | ov_SC07_001 | 2133 | 1693 | 0 | 2135 | 2454 | 87.0% |
 | ov_SC07_002 | 2160 | 1693 | 0 | 2164 | 2579 | 83.9% |
-| ov_SC07_006 | 1 | 0 | 0 | 81 | 2456 | 3.3% |
-| ov_SC07_007 | 1 | 0 | 0 | 85 | 2614 | 3.3% |
+| ov_SC07_006 | 1545 | 1543 | 0 | 1625 | 2456 | 66.2% |
+| ov_SC07_007 | 2 | 0 | 0 | 86 | 2614 | 3.3% |
 | ov_SC07_008 | 2117 | 1693 | 0 | 2117 | 2386 | 88.7% |
 | ov_SC07_009 | 2127 | 1693 | 0 | 2129 | 2430 | 87.6% |
-| ov_SC07_010 | 0 | 0 | 0 | 83 | 2526 | 3.3% |
-| ov_SC07_011 | 1 | 0 | 0 | 81 | 2450 | 3.3% |
+| ov_SC07_010 | 1 | 0 | 0 | 84 | 2526 | 3.3% |
+| ov_SC07_011 | 2 | 0 | 0 | 82 | 2450 | 3.3% |
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""Phase-28 T4: EXTEND the existing dedup registry to newly-onboarded binaries.
+
+THE GAP THIS FILLS (byte-measured, P28 T3-A). `tools/new_overlay.sh` produces a byte-clean binary
+that is NOT a citizen of the shared-body ecosystem: its `.c` includes only `common.h` (never
+`../shared/engine_core.h`), so not one shared engine body can reach it, and it appears in ZERO of
+`config/dedup.us.yaml`'s groups. The 4 SC07 overlays the Phase-27 disc audit onboarded therefore sat
+at ~80 matched / ~2,400 stubs while their siblings were ~2,150 matched — and 1,689 registry groups
+still read "134 binaries", never 138.
+
+WHY THE EXISTING TOOL CANNOT DO IT. `dedup_propagate` is built for CRACK -> AUTHOR MACRO ->
+INSTANTIATE: `--auto-from` scans a source overlay's INLINE DEFS (it planned only 11 fns here,
+skipping the ~1,600 whose bodies are ALREADY `DEFINE_func_*` macros in engine_core.h), and `--addr`
+dies with "no source overlay has it matched" because no overlay holds an inline def — every one of
+the 134 uses the macro. Extending an existing MACRO-BACKED group to a new binary is a different
+operation, and nothing implemented it.
+
+WHY IT IS SAFE (the correctness argument — this is a byte-gate feeder, so it must be explicit).
+`h_exact` is the SHA1 of RAW INSTRUCTION BYTES (tools/sig_image.py). Two instances sharing an
+h_exact are therefore identical *including* their `jal`/`lui`/`%lo` reloc immediates — same callee
+addresses, same data addresses, same symbols. So the body that compiles byte-identically at one
+member compiles byte-identically at the other, with no remap at all. (This is exactly why
+dup_report calls h_exact "guaranteed byte-match" and h_norm "candidate-only".) The whole-binary
+byte-gate remains the sole arbiter (G3/P9): a wrong instantiation changes the bytes and fails SHA1.
+A bug here can make this tool FAIL TO BANK; it cannot make it falsely bank.
+
+REUSE, DON'T REBUILD (R33). This owns only the SET COMPUTATION and the REGISTRY EDIT. The splice and
+the gate are `tools/harvest_verify.py` verbatim — which already derives each stub's home TU from the
+corpus oracle, chunks + bisects, and reverts on failure. h_exact members are byte-identical by
+construction, so the happy path is ONE build per binary (not one per function).
+
+  tools/dedup_extend.py --binaries ov_SC07_006,ov_SC07_007,ov_SC07_010,ov_SC07_011 [--check-only]
+                        [--chunk N] [--limit N]
+"""
+import argparse
+import glob
+import json
+import os
+import re
+import subprocess
+import sys
+
+import yaml
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import corpus
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+PY = sys.executable
+REGISTRY = "config/dedup.us.yaml"
+SHARED_INCLUDE = '#include "../shared/engine_core.h"'
+EXTEND_DIR = ".run/extend"
+
+
+def sig_hashes(binary):
+    """{vram_int: h_exact} from the binary's sig — the ORIGINAL bytes, independent of splat."""
+    out = {}
+    p = os.path.join(REPO, f".run/sig.{binary}.jsonl")
+    for line in open(p):
+        d = json.loads(line)
+        out[int(d["addr"], 16)] = d["h_exact"]
+    return out
+
+
+def load_groups():
+    doc = yaml.safe_load(open(os.path.join(REPO, REGISTRY)))
+    groups = doc if isinstance(doc, list) else doc.get("groups", doc)
+    if not isinstance(groups, list):
+        sys.exit(f"dedup_extend: unexpected {REGISTRY} shape: {type(groups).__name__}")
+    return doc, groups
+
+
+def _addr(v):
+    return int(v, 16) if isinstance(v, str) else int(v)
+
+
+def plan_for(binary, groups):
+    """[(group, vram, macro)] for every h_exact group this binary could join.
+
+    A group is extendable into `binary` iff: it is h_exact tier, the binary is not already a member,
+    the binary has a LIVE STUB at the group's vram, and that stub's original bytes hash to the
+    group's recorded h_exact (the C1 equivalence the registry itself asserts).
+    """
+    sig = sig_hashes(binary)
+    stubs = corpus.stubs(binary)
+    out = []
+    for g in groups:
+        if g.get("tier") != "h_exact":
+            continue
+        if "vram" not in g or "binaries" not in g:
+            continue                                   # verbose-form group: not position-locked, skip
+        if binary in g["binaries"]:
+            continue
+        vram = _addr(g["vram"])
+        if vram not in stubs:
+            continue                                   # not a live stub here (matched already, or absent)
+        if sig.get(vram) != g["hash"]:
+            continue                                   # DIFFERENT CODE at the same vram — the whole point of the check
+        out.append((g, vram, g["func"]))
+    return out
+
+
+def ensure_include(binary, apply=True):
+    """Add the shared-header include if absent. Byte-neutral: engine_core.h is only `#define`s, so
+    including it without instantiating emits no code (proven: ov_SC07_006 built 7ca772be with the
+    include + one macro). Returns True if the file was changed."""
+    p = os.path.join(REPO, f"src/{binary}/{binary}.c")
+    t = open(p).read()
+    if SHARED_INCLUDE in t:
+        return False
+    if apply:
+        t = t.replace('#include "common.h"', '#include "common.h"\n' + SHARED_INCLUDE, 1)
+        open(p, "w").write(t)
+    return True
+
+
+def write_drafts(binary, plan):
+    d = os.path.join(REPO, EXTEND_DIR, binary)
+    subprocess.run(["rm", "-rf", d], check=False)
+    os.makedirs(d, exist_ok=True)
+    for g, vram, macro in plan:
+        fn = f"func_{vram:08X}"
+        body = f"{macro}()  /* dedup: shared engine-core @0x{vram:08x} (src/shared) */\n"
+        open(os.path.join(d, fn + ".c"), "w").write(body)
+    return d
+
+
+def gate(binary, drafts_dir, chunk):
+    """The EXISTING whole-binary byte-gate is the sole arbiter (G3/P9). Returns the banked fn set."""
+    good = open(os.path.join(REPO, f"config/check.{binary}.sha")).read().split()[0]
+    vout = f".run/extend_verified.{binary}.txt"
+    fout = f".run/extend_failed.{binary}.txt"
+    cmd = [PY, "tools/harvest_verify.py", "--binary", binary,
+           "--out", f"build/{binary}/{binary}", "--good-sha", good,
+           "--drafts", os.path.relpath(drafts_dir, REPO), "--chunk", str(chunk),
+           "--verified-out", vout, "--failed-out", fout]
+    subprocess.run(cmd, cwd=REPO, timeout=7200)
+    p = os.path.join(REPO, vout)
+    return set(open(p).read().split()) if os.path.exists(p) else set()
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--binaries", required=True, help="comma list of newly-onboarded binaries to wire in")
+    ap.add_argument("--chunk", type=int, default=512,
+                    help="drafts per build (h_exact is byte-identical by construction, so the happy "
+                         "path is ONE build; harvest_verify bisects on failure). default 512")
+    ap.add_argument("--limit", type=int, default=0, help="cap #groups per binary (0 = all)")
+    ap.add_argument("--check-only", action="store_true", help="print the plan, touch nothing")
+    a = ap.parse_args()
+    os.chdir(REPO)
+
+    doc, groups = load_groups()
+    targets = [b.strip() for b in a.binaries.split(",") if b.strip()]
+
+    dirty = subprocess.run("git status --porcelain -- config/ src/", shell=True,
+                           capture_output=True, text=True).stdout.strip()
+    if dirty and not a.check_only:
+        sys.exit("dedup_extend: config/ or src/ is dirty — commit first so a failed gate reverts "
+                 "cleanly.\n" + dirty[:400])
+
+    total_banked, total_planned = 0, 0
+    for b in targets:
+        plan = plan_for(b, groups)
+        if a.limit:
+            plan = plan[:a.limit]
+        total_planned += len(plan)
+        print(f"[{b}] {len(plan)} extendable h_exact group(s); "
+              f"include {'ABSENT -> add' if ensure_include(b, apply=False) else 'present'}")
+        if a.check_only or not plan:
+            continue
+
+        ensure_include(b)
+        d = write_drafts(b, plan)
+        banked = gate(b, d, a.chunk)
+        print(f"[{b}] BANKED {len(banked)} / {len(plan)}")
+        total_banked += len(banked)
+
+        if not banked:
+            ensure_include_revert(b)
+            continue
+        for g, vram, _ in plan:                        # registry: only what the GATE accepted (P9)
+            if f"func_{vram:08X}" in banked and b not in g["binaries"]:
+                g["binaries"].append(b)
+
+    if not a.check_only and total_banked:
+        yaml.safe_dump(doc, open(REGISTRY, "w"), sort_keys=False, width=10**6, default_flow_style=None)
+        print(f"registry: {REGISTRY} updated")
+    print(f"\n=== dedup_extend: banked {total_banked} / {total_planned} planned "
+          f"across {len(targets)} binaries ===")
+    if a.check_only:
+        print("(--check-only: no files touched)")
+
+
+def ensure_include_revert(binary):
+    p = os.path.join(REPO, f"src/{binary}/{binary}.c")
+    t = open(p).read().replace('#include "common.h"\n' + SHARED_INCLUDE, '#include "common.h"', 1)
+    open(p, "w").write(t)
+
+
+if __name__ == "__main__":
+    main()