fix(phase-28 T4): wire all 4 SC07 overlays (6174/6457, 95.6%) + REPAIR the registry I destroyed

Completes T4 and corrects two defects I introduced, both landed in commit:0649. - WIRED: 006 1543/1614 · 007 1544/1615 · 010 1544/1614 · 011 1543/1614 = 6174/6457 = 95.6%, ~0 agent tokens. Stubs/overlay ~2400 -> 831/984/898/825. Fleet instr 67.0 -> 68.9%, fn-count 82.16 -> 83.94%. dedup-check 1840 validated / 0 failed; groups now read "138 members [138 binaries]" (was 134); C1 coverage 227211 -> 233385 = exactly +6174. R22 make clean && extract-all && check-all -> 140 passed, 0 failed of 140 at every stage. - FIX #1 — I DESTROYED THE REGISTRY'S DOCUMENTATION, AND EVERY GATE CALLED IT GREEN (H5). The first cut wrote config/dedup.us.yaml with yaml.safe_dump, round-tripping the whole file: 47 comment lines -> 0 (including the curated Phase-11 header explaining WHY the share is source-level) and 1832 `vram: 0x80162FF4` -> `vram: 2148937716` (PyYAML parses YAML-1.1 hex to int; dumps int as decimal). 25,948 lines rewritten. It passed dedup-check 1840/0 AND check-all 140/140 because _addr() accepts both forms: THE DATA WAS CORRECT AND THE DOCUMENT WAS RUINED. Fixed forward (R6, no history rewrite): restored from commit:0649~1 and re-applied the 6174 memberships via a surgical text edit (add_members_surgical). Verified: 1545 insertions / 1545 deletions, 0 non-`binaries:` lines changed, 47 comments + 1908 hex fields intact, and the rebuilt fleet is byte-identical to the destructive version (140/140). THE LESSON: every oracle this project owns measures BYTES, so a formatting-destructive write is invisible to all of them by construction. R34 says the byte-gate is a null COVERAGE oracle; this is the same hole one layer out — it is a null DOCUMENT oracle too. - FIX #2 — I MIS-REPORTED THE DIFFs, TWICE (R14). (a) commit:0649 claims ov_SC07_006's 71 non-banks were "ALL PLUMBING, ZERO DIFF". FALSE — I read head -6 of the classified file and generalized. It has the same 4 DIFFs as the others. (b) I then built the jr guard assuming those 4 were the §53 jr class BECAUSE ov_SC01_077 hosts them in _jr_8017A4AC.c / _jr_80182268.c. has_mid_jr is FALSE for all four (33-52 ins, no jump table): they merely live in a carved jr-REGION split, which sweeps in every function in its address range. HOSTING FILE != FUNCTION CLASS. The guard is KEPT (preventive, §53-correct, currently skips 0 — no jr fn is in the extendable set) with its docstring corrected to record what it is NOT. The 12 DIFFs (0.19%) are UNDIAGNOSED and logged, correctly left as stubs by the gate — not dressed in a story. - The 283 non-banks: 271 PLUMBING (the loose-typing conflict class + the whale, whose body lives in src/shared/func_80144B9C.h so no DEFINE macro exists to expand) + 12 DIFF. Existing tools cover the plumbing (cast_call_sites / canon_sig_reconcile / reconcile_tu).
2026-09-26 13:33:34 -04:00 · 2026-07-16 00:10:12 -06:00
parent c0486fe5f8
commit fda9eebb42
7 changed files with 22361 additions and 17598 deletions
@@ -4,15 +4,15 @@
 # cross-binary collapsible-byte leverage: docs/duplicates.cross.md.

 # THREE progress metrics (all matter — see the labels):
-FLEET fn-count byte-ident: 292275 / 353723 = 82.63%   (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
-FLEET instr-weighted     : 8898477 / 13081451 = 68.0%   (shipped .text across resident+138 overlays; the decomp.dev-DISPLAY number)
+FLEET fn-count byte-ident: 296906 / 353723 = 83.94%   (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
+FLEET instr-weighted     : 9017152 / 13081451 = 68.9%   (shipped .text across resident+138 overlays; the decomp.dev-DISPLAY number)
 FLEET distinct-code(uniq): 2757078 / 5574674 = 49.5%   (53528/87459 unique fns; the DISTINCT-RE number)
 MAIN game-code weighted  :     436 / 60201 = 0.7%   (Phase-27 T10; SEPARATE — LINKED-excluding Ghidra sig dated 2026-06-14, PROVISIONAL until a fresh/complete main sig; NOT folded into the fleet number)

-FLEET REAL substantive   : 290420   (of which dedup-shared 228710 via 1840 groups / 228754 instances)
+FLEET REAL substantive   : 295051   (of which dedup-shared 233341 via 1840 groups / 233385 instances)
 FLEET LINKED PsyQ objs   :    959
 FLEET NON_MATCHING       :      7   (0 in any default build — G4)
-FLEET INCLUDE_ASM stubs  :  61441
+FLEET INCLUDE_ASM stubs  :  56810
 FLEET matchable          : 353723

 | binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
@@ -152,8 +152,8 @@ FLEET matchable          : 353723
 | ov_SC07_001 | 2133 | 1693 | 0 | 2135 | 2454 | 87.0% |
 | ov_SC07_002 | 2160 | 1693 | 0 | 2164 | 2579 | 83.9% |
 | ov_SC07_006 | 1545 | 1543 | 0 | 1625 | 2456 | 66.2% |
-| ov_SC07_007 | 2 | 0 | 0 | 86 | 2614 | 3.3% |
+| ov_SC07_007 | 1546 | 1544 | 0 | 1630 | 2614 | 62.4% |
 | ov_SC07_008 | 2117 | 1693 | 0 | 2117 | 2386 | 88.7% |
 | ov_SC07_009 | 2127 | 1693 | 0 | 2129 | 2430 | 87.6% |
-| ov_SC07_010 | 1 | 0 | 0 | 84 | 2526 | 3.3% |
-| ov_SC07_011 | 2 | 0 | 0 | 82 | 2450 | 3.3% |
+| ov_SC07_010 | 1545 | 1544 | 0 | 1628 | 2526 | 64.4% |
+| ov_SC07_011 | 1545 | 1543 | 0 | 1625 | 2450 | 66.3% |
@@ -147,5 +147,13 @@ Whole-binary byte-gate = sole arbiter (G3/P9) · R22 clean-fleet per banked batc
  - **R22: `make clean` + extract-all + `check-all` → 140/140**, 0 FAIL (stratum A's 4 banks).
  - **Carried → T3b:** strata **B** (legacy PURE non-jr, 95 fam / 7,993 members) and **C** (legacy IMM, 36 fam / 6,644) — the *legacy* h_seq rate, still genuinely unmeasured. The SC07 pool answered a different (and cheaper) question than the one T3 set out to ask.

+- **2026-07-16 — T4 ✅ `dedup_extend`: 6,174 / 6,457 (95.6%) wired across the 4 new overlays — and TWO self-inflicted defects, both mine.**
+  - **New `tools/dedup_extend.py`** — the missing mode. `dedup_propagate` is CRACK→AUTHOR MACRO→INSTANTIATE: `--auto-from` scans *inline defs* (planned only **11** here — the ~1,600 shared bodies are ALREADY `DEFINE_func_*` macros in `engine_core.h`), and `--addr` dies *"no source overlay has it matched"* because no overlay holds an inline def. **Extending an existing macro-backed group to a newly-onboarded binary had no implementation.**
+  - **Safety (explicit — it feeds the byte-gate):** `h_exact` = SHA1 of RAW instruction bytes ⇒ identical h_exact means identical bytes **including** `jal`/`lui`/`%lo` reloc immediates ⇒ same C, no remap. A bug can only FAIL TO BANK, never falsely bank. **Reuse (R33):** owns only the set computation + registry edit; the splice and gate are `harvest_verify` verbatim → ~1 build/binary, not 1/function.
+  - **Results:** 006 **1543/1614** · 007 **1544/1615** · 010 **1544/1614** · 011 **1543/1614** = **6,174/6,457 = 95.6%**, ~0 agent tokens. Stubs/overlay ~2,400 → 831/984/898/825. **R22 140/140** after each stage; **dedup-check 1840 validated / 0 failed**, groups now read **"138 members [138 binaries]"** (was 134); **C1 coverage 227,211 → 233,385 = exactly +6,174**. Fleet **instr 67.7 → 68.0%** on the first overlay alone.
+  - **The 283 non-banks: 271 PLUMBING + 12 DIFF, ZERO unexplained-at-scale.** Plumbing = the loose-typing conflict class (`conflicting types for D_800A5E60 / func_8012C750`) + the whale (`undefined reference` — its body is in `src/shared/func_80144B9C.h`, the -O0 header, so no `DEFINE` macro exists to expand). Existing tools cover both (`cast_call_sites` / `canon_sig_reconcile` / `reconcile_tu`).
+  - **❌ SELF-INFLICTED #1 — I DESTROYED the registry's documentation (H5), and every gate called it green.** My first cut wrote the registry with `yaml.safe_dump`, which round-tripped the whole file: **47 comment lines → 0** (including the curated Phase-11 header explaining *why* the share is source-level) and **1,832 `vram: 0x80162FF4` → `vram: 2148937716`** (PyYAML parses YAML-1.1 hex to int, dumps int as decimal). 25,948 lines rewritten. **It passed dedup-check 1840/0 AND check-all 140/140** — because `_addr()` accepts both forms: *the data was correct and the document was ruined*. Landed in `commit:0649`; **fixed forward** (R6 — no history rewrite): registry restored from `commit:0649~1`, memberships re-applied by a **surgical text edit** (`add_members_surgical`), verified **1545 insertions / 1545 deletions, 0 non-`binaries:` lines changed, 47 comments + 1908 hex fields intact**. *The lesson is the sharpest of the phase: every oracle we own measures BYTES, so a formatting-destructive write is invisible to all of them.*
+  - **❌ SELF-INFLICTED #2 — I mis-reported the DIFFs, twice (R14).** (a) I claimed ov_SC07_006's 71 non-banks were *"ALL PLUMBING, ZERO DIFF"* — from reading `head -6` of the classified file and generalizing. It has the same 4 DIFFs; the claim is **false and is in commit `commit:0649`**. (b) I then built a jr guard on the assumption those 4 were the §53 jr class *because ov_SC01_077 hosts them in `_jr_8017A4AC.c`* — **`has_mid_jr` is False for all four** (33-52 ins, no jump table). They merely live in a carved jr-**region** split (the carve sweeps in every fn in its address range). **Hosting file ≠ function class.** The guard is kept (preventive, §53-correct, currently skips 0) with its docstring corrected; **the 12 DIFFs (0.19%) are UNDIAGNOSED and logged** — correctly left as stubs by the gate.
+
  - **T3 strata (honest, from the fixed map):** SC07-only **1,255 fam / 6,268 mem / 230,612 ins** · legacy PURE non-jr **95 / 7,993 / 478,379** · legacy IMM **36 / 6,644 / 212,707** · legacy MIXED **30 / 968 / 10,462** · legacy PURE w/ jr **2 / 16 / 5,088** (T1's jtbl territory — note B2's family is far smaller here than the roadmap's "×112 ≈ 106k ins"; T1 re-derives it). **Total addressable = 937,248 ins = 21.7% of all remaining weight = 7.16pp of fleet instr if it all banked.** This is the prize the roadmap declared dead. It stays a prediction until T3's gate.
 </content>
@@ -44,6 +44,8 @@ import yaml

 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 import corpus
+import family_hseq          # has_mid_jr — the ONE jr oracle (R33)
+import family_remap as FR   # stream_words/nins_of

 REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 PY = sys.executable
@@ -70,34 +72,95 @@ def load_groups():
    return doc, groups


+def add_members_surgical(additions):
+    """Append binaries to each group's `binaries: [...]` list by TEXT EDIT, in place.
+
+    NEVER `yaml.safe_dump` this file. The first cut of this tool round-tripped it through
+    safe_dump and silently destroyed BOTH of the things a human needs from it (H5 — "never
+    silently drop comments on a rewrite"):
+      * all 47 comment lines — including the curated Phase-11 header explaining WHY the share is
+        source-level (the linker cannot excise bytes interior to an object) — dumped to nothing;
+      * every `vram: 0x80162FF4` re-serialized as `vram: 2148937716` (PyYAML parses YAML-1.1 hex
+        to int, and dumps int as decimal), making 1,832 entries unreadable.
+    It was invisible to every gate: dedup-check passed 1840/0 and check-all stayed 140/140,
+    because `_addr()` accepts both forms — the data was fine and the DOCUMENT was ruined. A
+    formatting-destructive write that all your oracles call green is exactly the class this
+    project keeps re-learning: the gate measures bytes, not intent.
+
+    `additions` = {group_id: [binary, ...]}. Idempotent: a binary already listed is skipped.
+    """
+    p = os.path.join(REPO, REGISTRY)
+    lines = open(p).read().splitlines(keepends=True)
+    cur, n = None, 0
+    for i, ln in enumerate(lines):
+        m = re.match(r"^\s*-?\s*id:\s*(\S+)\s*$", ln)
+        if m:
+            cur = m.group(1)
+            continue
+        if cur and cur in additions and re.match(r"^\s*binaries:\s*\[", ln):
+            add = [b for b in additions[cur] if re.search(rf"\b{re.escape(b)}\b", ln) is None]
+            if add:
+                lines[i] = ln.rstrip("\n").rstrip()[:-1].rstrip() + ", " + ", ".join(add) + "]\n"
+                n += len(add)
+            cur = None
+    open(p, "w").write("".join(lines))
+    return n
+
+
 def _addr(v):
    return int(v, 16) if isinstance(v, str) else int(v)


 def plan_for(binary, groups):
-    """[(group, vram, macro)] for every h_exact group this binary could join.
+    """[(group, vram, macro)] for every h_exact group this binary could join, plus the skip tally.

    A group is extendable into `binary` iff: it is h_exact tier, the binary is not already a member,
-    the binary has a LIVE STUB at the group's vram, and that stub's original bytes hash to the
-    group's recorded h_exact (the C1 equivalence the registry itself asserts).
+    the binary has a LIVE STUB at the group's vram, that stub's original bytes hash to the group's
+    recorded h_exact (the C1 equivalence the registry itself asserts), and it is NOT a jr/switch
+    function (see below).
+
+    THE jr GUARD (cookbook §53). A function with a mid-body `jr` on a non-$ra register dispatches
+    through a compiler-generated jump table the LINKER must place at the sibling's exact address — a
+    per-sibling `jtbl_carve` + interleave (`jtbl_family_bank.py`). This tool has no carve step and a
+    newly-onboarded overlay has no `_jr_*` split at all, so a jr body instantiated here would leave
+    its table unplaced and the gate would (correctly) reject it. The guard is PREVENTIVE: it skips
+    and REPORTS rather than feeding the gate drafts that cannot pass (R32 — a skip is only honest if
+    it is counted and named). On the SC07 set it currently skips ZERO: no jr function is in the
+    extendable set.
+
+    ⚠️ R14 — WHAT THIS GUARD IS *NOT*. It does NOT explain the 12 DIFFs the first run produced
+    (`func_80162FF4/801630C4/80163194/8016325C`, the same 4 in every overlay, 12 of 6457 = 0.19%).
+    I first assumed those were the jr class because ov_SC01_077 hosts them in `_jr_8017A4AC.c` /
+    `_jr_80182268.c` — a natural read, and WRONG: `has_mid_jr` is **False** for all four (33-52 ins,
+    no jump table). They merely LIVE in a carved jr-REGION split (the carve region for
+    `func_8017A4AC` sweeps in every function in its address range), which says nothing about them.
+    Their DIFF cause is UNDIAGNOSED and logged for follow-up; they are correctly left as stubs by the
+    gate. Hosting file != function class — do not infer one from the other.
+
+    Detector reused verbatim from `family_hseq.has_mid_jr` (R33 — one oracle, not two).
    """
    sig = sig_hashes(binary)
    stubs = corpus.stubs(binary)
-    out = []
+    out, skips = [], {}
    for g in groups:
        if g.get("tier") != "h_exact":
            continue
        if "vram" not in g or "binaries" not in g:
-            continue                                   # verbose-form group: not position-locked, skip
+            skips["verbose-form"] = skips.get("verbose-form", 0) + 1
+            continue                                   # not position-locked
        if binary in g["binaries"]:
            continue
        vram = _addr(g["vram"])
        if vram not in stubs:
            continue                                   # not a live stub here (matched already, or absent)
        if sig.get(vram) != g["hash"]:
-            continue                                   # DIFFERENT CODE at the same vram — the whole point of the check
+            continue                                   # DIFFERENT CODE at the same vram — the point of the check
+        words = FR.stream_words(binary, vram, FR.nins_of(binary, vram))
+        if words and family_hseq.has_mid_jr(words):
+            skips["jr-needs-carve"] = skips.get("jr-needs-carve", 0) + 1
+            continue                                   # §53: route via jtbl_family_bank, not here
        out.append((g, vram, g["func"]))
-    return out
+    return out, skips


 def ensure_include(binary, apply=True):
@@ -162,12 +225,13 @@ def main():

    total_banked, total_planned = 0, 0
    for b in targets:
-        plan = plan_for(b, groups)
+        plan, skips = plan_for(b, groups)
        if a.limit:
            plan = plan[:a.limit]
        total_planned += len(plan)
        print(f"[{b}] {len(plan)} extendable h_exact group(s); "
-              f"include {'ABSENT -> add' if ensure_include(b, apply=False) else 'present'}")
+              f"include {'ABSENT -> add' if ensure_include(b, apply=False) else 'present'}"
+              + (f"; skipped {skips}" if skips else ""))
        if a.check_only or not plan:
            continue

@@ -180,13 +244,10 @@ def main():
        if not banked:
            ensure_include_revert(b)
            continue
-        for g, vram, _ in plan:                        # registry: only what the GATE accepted (P9)
-            if f"func_{vram:08X}" in banked and b not in g["binaries"]:
-                g["binaries"].append(b)
-
-    if not a.check_only and total_banked:
-        yaml.safe_dump(doc, open(REGISTRY, "w"), sort_keys=False, width=10**6, default_flow_style=None)
-        print(f"registry: {REGISTRY} updated")
+        adds = {g["id"]: [b] for g, vram, _ in plan   # registry: only what the GATE accepted (P9)
+                if f"func_{vram:08X}" in banked}
+        n = add_members_surgical(adds)                 # TEXT edit — never safe_dump (see the docstring)
+        print(f"[{b}] registry: +{n} membership(s)")
    print(f"\n=== dedup_extend: banked {total_banked} / {total_planned} planned "
          f"across {len(targets)} binaries ===")
    if a.check_only: