feat(phase-29): tools/symcheck.py — the pre-gate symbol-set guard (§67a)

Builds the guard SESSION-17 left as a TODO after the func_801463A0 `_s`-alias trap, where a draft invented extern aliases no symbol table defines, read MATCH under rtu_match, and could never bank. - diffs the symbols a draft's object references (reloc records) against the target .s's %hi/%lo/jal set; reports MISSING (invented-alias signature) and INVENTED separately. - fills a real hole: match_one/masked_diff compare relocation-MASKED words (object-vs-.s is symbol-agnostic BY CONSTRUCTION) and rtu_match COMPILES WITHOUT LINKING -- so both are structurally blind to this class. R34: a second oracle that can disagree with the first. - NEGATIVE-CONTROL PROVEN: with one data extern renamed to an invented alias, match_one reports the SAME 14 mismatched as the correct draft; symcheck exits 1 naming both symbols. - --c compiles via match_one so the pinned triple/flags can never drift (R33); or --obj. - applied to the live func_8014D820 close=14 draft: 12/12 symbols agree, so a match there will link cleanly -- the §65c class is ruled out for it in advance. - cookbook §67a + SETUP tooling-inventory row (R21). Necessary condition, NOT a match oracle: still finish on the whole-binary byte-gate (G3/P9).
2026-10-01 15:43:06 -04:00 · 2026-07-24 21:30:08 -06:00
parent 34417179df
commit b8e7250c78
3 changed files with 153 additions and 0 deletions
@@ -700,6 +700,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
 | | `tools/harvest_verify.py` | The whole-binary byte-gate (substitute draft → `make build` → keep iff byte-identical, else revert). Sole arbiter (G3/P9). `--verified-out`/`--failed-out` = per-worker result paths (parallel gating, Phase-23 T10). |
 | | `tools/rtu_match.py` | **(Phase 25 wave-3, cookbook §42b)** REAL-TU-faithful per-fn match check that fixes match_one's blind spot: splices a candidate into a copy of the split `.c`, neutralizes `INCLUDE_ASM` (`-DINCLUDE_ASM(a,b)=` → no `asm/` needed) with `-Isrc/<source>` for the relative `../shared/` include, compiles the WHOLE TU (`cc1→maspsx→as`), masked-diffs the fn (shared `masked_diff`). Captures the in-TU decl/global-type/memcpy-builtin drift that isolation (`match_one`) misses, so a MATCH holds at the whole-binary gate; per-fn temp dir → **parallel-safe, no shared overlay build** (enables a real-TU-faithful crack fan-out). Supports `//@EDIT old||new` file-scope pre-edits (the §42b read-global s16→u16 flip). STILL finish on `make build` SHA (G3/P9). |
 | | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. |
+| | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). |
 | | `tools/demacroize.py` | **(Phase 29 SESSION-16, cookbook §65b)** The per-overlay-local escape from a shared-header decl conflict — the largest stranded class, and the one §20 called unrecoverable. The conflicting `extern` lives INSIDE a `DEFINE_func_*` body, so it exists only at instantiation sites: this expands those instantiations **in the overlay's own TU**, correcting only the conflicting decl to the draft's byte-true sig (never dropping it, §57a-1). **T1** — writes confined to `src/<binary>/**`, so the per-binary gate suffices and no R22 risk is created by construction (contrast `fix_header_decl`, fleet-blind, §63 UPDATE). `--emit-edits` (read-only, feeds `rtu_match`) / `--apply`. **Price: the function can no longer propagate ×138 — such a bank is ×1** (full distinct-code credit, ~1/138 of instr). |
 | | `tools/recover_integration.py` | **(Phase 24 T6; extended Phase 29 SESSION-16/17, cookbook §65/§66)** The stranded-draft recovery DRIVER — consumes a wave dir (`--draft-dir`, repeatable; the backlog is the wrong source: unreliable `closeness`, overlay-specific drafts), runs the declared `--stages` (`demacroize` T1 / `arity` T2), then gates in TWO passes (**gate all → exact snapshot-restore → re-stage winners only**), so a non-bank never leaves an edit behind. `--run-id` puts all scratch + `verified_out`/`failed_out` under `.run/recover/<id>/` (closes §55b trap 4); bank truth is `banked_from_source()` (the stub is GONE from src), never a gate report; `stub_map` derives from `corpus.stubs` (R33). **Blast-radius tiers are ENFORCED** (`--max-tier`, write-set assertion), and **propagation is itself fleet-tier**: it needs `--max-tier fleet` AND `--r22`, and is refused outright after `demacroize` (those banks are ×1 by construction; `--auto-from` would re-macroize and undo them) — both refusals negative-control-tested, exit 1. `--probe-only` / `--report`. Success path verified end-to-end by the §66 free re-bank test. |
 | | `tools/lift_types.py` · `tools/uniquify_type.py` | **(Phase 29 SESSION-14, cookbook §64/§64a)** Fleet-wide type lift into `src/shared/engine_types.h`, and the camp-uniquify that must precede it for VARIANT names (same identifier, different layouts in different TUs — reconciling them merges two layouts and breaks the minority camp). Both are **T2**: dry-run by default, and **R22 clean-fleet is the arbiter, not the per-binary gate**. |
@@ -5371,3 +5371,36 @@ and the launder is how you set it.
 **Where this applies.** Any draft that is `+1 ins with an unfilled load-delay nop` and shows mirrored
 `sw $sN` / `move $sN,$aX` prologue pairs. That signature is the tell — count the instructions first: a
 draft one *over* with a `nop` the target fills is a placement bug, not a regalloc wall.
+
+## §67a — Run the symbol-set guard BEFORE you pay for a gate (`tools/symcheck.py`, Phase 29 SESSION-18)
+
+SESSION-17 ended `func_801463A0` with an open TODO: *"a cheap guard worth building: diff a draft's
+referenced symbol set against the target `.s`'s `%hi/%lo/jal` set before gating."* Built, negative-control
+proven, and it belongs in front of every gate cycle.
+
+**Why nothing we already had can do this.** All three of our fast oracles are blind to it *by
+construction*, not by accident:
+
+| oracle | blindness |
+|---|---|
+| `match_one` / `masked_diff` | compares relocation-**masked** words; object-vs-`.s` mode is symbol-agnostic, so a reloc against the *wrong* symbol matches at that position |
+| `rtu_match` | **compiles but never links** — an extern no symbol table defines is invisible to it |
+| the whole-binary byte-gate | correct, but it tells you *rejected*, not *why*, and costs a full cycle |
+
+**The measurement that proves the point** (`func_8014D820`, one data extern renamed to an invented
+`_s` alias):
+
+| | correct draft | draft with an invented symbol |
+|---|---|---|
+| `match_one` | 14 mismatched | **14 mismatched — identical** |
+| `symcheck` | `SYMS-OK  12 symbols agree` | `SYMS-DIFF` + names MISSING and INVENTED, exit 1 |
+
+Two directions, both reported: **MISSING** (target references it, draft does not — the invented-alias
+signature, since the alias silently stands in for the real symbol) and **INVENTED** (draft references
+something the original never touched).
+
+**Where it goes:** immediately before `harvest_verify`, and immediately *after* any transform that
+rewrites declarations (`sig_unify`, `canon_resident_calls`, `cast_call_sites`, `canon_sig_reconcile`) —
+those rewrite extern names, which is precisely how an alias gets invented. It is a **necessary
+condition, not a match oracle**: `SYMS-OK` means "no link-level defect of this class", nothing more.
+Finish on the byte-gate (G3/P9).
@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""symcheck.py — the pre-gate SYMBOL-SET guard (BFM Phase 29 SESSION-18).
+
+Diff the set of symbols a draft's object REFERENCES against the set the target `.s` references
+(`%hi`/`%lo`/`jal`). A mismatch is a link-level defect that every masked-diff oracle we own is
+STRUCTURALLY BLIND TO:
+
+  * `tools/match_one.py` / `tools/masked_diff.py` compare relocation-MASKED words, so a reloc against
+    the wrong symbol scores as a match at that position (object-vs-.s mode is symbol-agnostic by
+    construction — see masked_diff's docstring).
+  * `tools/rtu_match.py` COMPILES but never LINKS, so an extern that no symbol table can resolve is
+    invisible to it *by construction*, not merely masked.
+
+That combination produced the SESSION-17 `func_801463A0` trap: the draft invented `_s`-suffixed alias
+externs (`D_80126BE0_s`) that no symbol table defines, read MATCH under rtu_match, and could never
+bank. The whole-binary byte-gate caught it (G3/P9, as always) — but only after a full gate cycle, and
+the *cause* took a symbol-set comparison to find. This makes that comparison a one-command check you
+run BEFORE paying for a gate.
+
+Two failure directions, both reported:
+  MISSING  — in the target, absent from the draft: a dropped reference, or a symbol renamed/aliased
+             into something that resolves elsewhere. This is the invented-alias signature.
+  INVENTED — in the draft, absent from the target: a symbol the original function never touched.
+
+Exit 0 iff the sets are equal. Not a match oracle and not a substitute for the byte-gate — a cheap
+necessary condition (R34: a second oracle that can DISAGREE with the masked ones).
+
+  python3 tools/symcheck.py func_8014D820 --c .run/giants/s18_func_8014D820_close14.c \
+      --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077_after
+  python3 tools/symcheck.py func_8014D820 --obj build/.../foo.o --asm-subdir <dir>
+"""
+import argparse
+import glob
+import os
+import re
+import subprocess
+import sys
+
+OBJDUMP = "mipsel-linux-gnu-objdump"
+PY = ".venv/bin/python"
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+# `%hi(sym)` / `%lo(sym)` / `jal sym` in a splat .s. The target .s is RESOLVED (no reloc records), so
+# the symbol names only survive in this operand syntax — which is exactly why the masked oracles,
+# which work on the encoded word, cannot see them.
+_S_SYM = re.compile(r"%(?:hi|lo)\(([A-Za-z_][A-Za-z0-9_]*)\)|\bjal\s+([A-Za-z_][A-Za-z0-9_]*)")
+_O_SYM = re.compile(r"R_MIPS_\S+\s+([A-Za-z_][A-Za-z0-9_]*)")
+
+
+def target_syms(s_path):
+    out = set()
+    with open(s_path) as fh:
+        for line in fh:
+            for a, b in _S_SYM.findall(line):
+                out.add(a or b)
+    return out
+
+
+def object_syms(obj, fn=None):
+    """Symbols referenced by relocations in `obj`. With `fn`, only relocs inside that function's
+    section slice — objects here are one-function-per-file, so the whole-object set is the same."""
+    txt = subprocess.run([OBJDUMP, "-drz", "-r", obj], capture_output=True, text=True).stdout
+    return set(m.group(1) for m in _O_SYM.finditer(txt))
+
+
+def compile_draft(fn, c_path, asm_subdir, o0=False):
+    """Compile via match_one so the triple/flags can never drift from the real gate path (R33:
+    derive, don't re-derive). Returns the object it left behind."""
+    work = os.path.join(".run/symcheck", f"{fn}.{os.getpid()}")
+    cmd = [PY, os.path.join(HERE, "match_one.py"), fn, "--c", c_path,
+           "--asm-subdir", asm_subdir, "--work", work]
+    if o0:
+        cmd.append("--o0")
+    subprocess.run(cmd, capture_output=True, text=True)
+    hits = glob.glob(os.path.join(work, "**", "*.o"), recursive=True)
+    if not hits:
+        sys.exit(f"symcheck: match_one produced no object for {fn} (does the draft compile?)")
+    return hits[0]
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("fn")
+    ap.add_argument("--c", help="draft C (externs + def); compiled via match_one")
+    ap.add_argument("--obj", help="pre-compiled object instead of --c")
+    ap.add_argument("--asm-subdir", required=True, help="dir holding <fn>.s")
+    ap.add_argument("--o0", action="store_true", help="compile at -O0 (the _o0 split subsegments)")
+    a = ap.parse_args()
+
+    if not (a.c or a.obj):
+        ap.error("one of --c or --obj is required")
+
+    s_path = os.path.join(a.asm_subdir, f"{a.fn}.s")
+    if not os.path.exists(s_path):
+        sys.exit(f"symcheck: no target .s at {s_path}")
+
+    obj = a.obj or compile_draft(a.fn, a.c, a.asm_subdir, a.o0)
+    tgt, mine = target_syms(s_path), object_syms(obj, a.fn)
+
+    missing = sorted(tgt - mine)
+    invented = sorted(mine - tgt)
+
+    if not missing and not invented:
+        print(f"SYMS-OK  {a.fn}   {len(tgt)} symbols agree")
+        return 0
+
+    print(f"SYMS-DIFF  {a.fn}   target={len(tgt)} draft={len(mine)}")
+    for s in missing:
+        print(f"  MISSING   {s}   (target references it; draft does not -- dropped ref, or an "
+              f"invented alias standing in for it)")
+    for s in invented:
+        print(f"  INVENTED  {s}   (draft references it; target does not -- if no symbol table "
+              f"defines it, rtu_match will still read MATCH and the gate will always reject)")
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())