diff --git a/docs/SETUP.md b/docs/SETUP.md index bae5cf8eba..e20f8202f2 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -700,6 +700,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/harvest_verify.py` | The whole-binary byte-gate (substitute draft → `make build` → keep iff byte-identical, else revert). Sole arbiter (G3/P9). `--verified-out`/`--failed-out` = per-worker result paths (parallel gating, Phase-23 T10). | | | `tools/rtu_match.py` | **(Phase 25 wave-3, cookbook §42b)** REAL-TU-faithful per-fn match check that fixes match_one's blind spot: splices a candidate into a copy of the split `.c`, neutralizes `INCLUDE_ASM` (`-DINCLUDE_ASM(a,b)=` → no `asm/` needed) with `-Isrc/` for the relative `../shared/` include, compiles the WHOLE TU (`cc1→maspsx→as`), masked-diffs the fn (shared `masked_diff`). Captures the in-TU decl/global-type/memcpy-builtin drift that isolation (`match_one`) misses, so a MATCH holds at the whole-binary gate; per-fn temp dir → **parallel-safe, no shared overlay build** (enables a real-TU-faithful crack fan-out). Supports `//@EDIT old||new` file-scope pre-edits (the §42b read-global s16→u16 flip). STILL finish on `make build` SHA (G3/P9). | | | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. | +| | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). | | | `tools/demacroize.py` | **(Phase 29 SESSION-16, cookbook §65b)** The per-overlay-local escape from a shared-header decl conflict — the largest stranded class, and the one §20 called unrecoverable. The conflicting `extern` lives INSIDE a `DEFINE_func_*` body, so it exists only at instantiation sites: this expands those instantiations **in the overlay's own TU**, correcting only the conflicting decl to the draft's byte-true sig (never dropping it, §57a-1). **T1** — writes confined to `src//**`, so the per-binary gate suffices and no R22 risk is created by construction (contrast `fix_header_decl`, fleet-blind, §63 UPDATE). `--emit-edits` (read-only, feeds `rtu_match`) / `--apply`. **Price: the function can no longer propagate ×138 — such a bank is ×1** (full distinct-code credit, ~1/138 of instr). | | | `tools/recover_integration.py` | **(Phase 24 T6; extended Phase 29 SESSION-16/17, cookbook §65/§66)** The stranded-draft recovery DRIVER — consumes a wave dir (`--draft-dir`, repeatable; the backlog is the wrong source: unreliable `closeness`, overlay-specific drafts), runs the declared `--stages` (`demacroize` T1 / `arity` T2), then gates in TWO passes (**gate all → exact snapshot-restore → re-stage winners only**), so a non-bank never leaves an edit behind. `--run-id` puts all scratch + `verified_out`/`failed_out` under `.run/recover//` (closes §55b trap 4); bank truth is `banked_from_source()` (the stub is GONE from src), never a gate report; `stub_map` derives from `corpus.stubs` (R33). **Blast-radius tiers are ENFORCED** (`--max-tier`, write-set assertion), and **propagation is itself fleet-tier**: it needs `--max-tier fleet` AND `--r22`, and is refused outright after `demacroize` (those banks are ×1 by construction; `--auto-from` would re-macroize and undo them) — both refusals negative-control-tested, exit 1. `--probe-only` / `--report`. Success path verified end-to-end by the §66 free re-bank test. | | | `tools/lift_types.py` · `tools/uniquify_type.py` | **(Phase 29 SESSION-14, cookbook §64/§64a)** Fleet-wide type lift into `src/shared/engine_types.h`, and the camp-uniquify that must precede it for VARIANT names (same identifier, different layouts in different TUs — reconciling them merges two layouts and breaks the minority camp). Both are **T2**: dry-run by default, and **R22 clean-fleet is the arbiter, not the per-binary gate**. | diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 7471d4dc07..026a40240b 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -5371,3 +5371,36 @@ and the launder is how you set it. **Where this applies.** Any draft that is `+1 ins with an unfilled load-delay nop` and shows mirrored `sw $sN` / `move $sN,$aX` prologue pairs. That signature is the tell — count the instructions first: a draft one *over* with a `nop` the target fills is a placement bug, not a regalloc wall. + +## §67a — Run the symbol-set guard BEFORE you pay for a gate (`tools/symcheck.py`, Phase 29 SESSION-18) + +SESSION-17 ended `func_801463A0` with an open TODO: *"a cheap guard worth building: diff a draft's +referenced symbol set against the target `.s`'s `%hi/%lo/jal` set before gating."* Built, negative-control +proven, and it belongs in front of every gate cycle. + +**Why nothing we already had can do this.** All three of our fast oracles are blind to it *by +construction*, not by accident: + +| oracle | blindness | +|---|---| +| `match_one` / `masked_diff` | compares relocation-**masked** words; object-vs-`.s` mode is symbol-agnostic, so a reloc against the *wrong* symbol matches at that position | +| `rtu_match` | **compiles but never links** — an extern no symbol table defines is invisible to it | +| the whole-binary byte-gate | correct, but it tells you *rejected*, not *why*, and costs a full cycle | + +**The measurement that proves the point** (`func_8014D820`, one data extern renamed to an invented +`_s` alias): + +| | correct draft | draft with an invented symbol | +|---|---|---| +| `match_one` | 14 mismatched | **14 mismatched — identical** | +| `symcheck` | `SYMS-OK 12 symbols agree` | `SYMS-DIFF` + names MISSING and INVENTED, exit 1 | + +Two directions, both reported: **MISSING** (target references it, draft does not — the invented-alias +signature, since the alias silently stands in for the real symbol) and **INVENTED** (draft references +something the original never touched). + +**Where it goes:** immediately before `harvest_verify`, and immediately *after* any transform that +rewrites declarations (`sig_unify`, `canon_resident_calls`, `cast_call_sites`, `canon_sig_reconcile`) — +those rewrite extern names, which is precisely how an alias gets invented. It is a **necessary +condition, not a match oracle**: `SYMS-OK` means "no link-level defect of this class", nothing more. +Finish on the byte-gate (G3/P9). diff --git a/tools/symcheck.py b/tools/symcheck.py new file mode 100644 index 0000000000..dcfbf31ecf --- /dev/null +++ b/tools/symcheck.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""symcheck.py — the pre-gate SYMBOL-SET guard (BFM Phase 29 SESSION-18). + +Diff the set of symbols a draft's object REFERENCES against the set the target `.s` references +(`%hi`/`%lo`/`jal`). A mismatch is a link-level defect that every masked-diff oracle we own is +STRUCTURALLY BLIND TO: + + * `tools/match_one.py` / `tools/masked_diff.py` compare relocation-MASKED words, so a reloc against + the wrong symbol scores as a match at that position (object-vs-.s mode is symbol-agnostic by + construction — see masked_diff's docstring). + * `tools/rtu_match.py` COMPILES but never LINKS, so an extern that no symbol table can resolve is + invisible to it *by construction*, not merely masked. + +That combination produced the SESSION-17 `func_801463A0` trap: the draft invented `_s`-suffixed alias +externs (`D_80126BE0_s`) that no symbol table defines, read MATCH under rtu_match, and could never +bank. The whole-binary byte-gate caught it (G3/P9, as always) — but only after a full gate cycle, and +the *cause* took a symbol-set comparison to find. This makes that comparison a one-command check you +run BEFORE paying for a gate. + +Two failure directions, both reported: + MISSING — in the target, absent from the draft: a dropped reference, or a symbol renamed/aliased + into something that resolves elsewhere. This is the invented-alias signature. + INVENTED — in the draft, absent from the target: a symbol the original function never touched. + +Exit 0 iff the sets are equal. Not a match oracle and not a substitute for the byte-gate — a cheap +necessary condition (R34: a second oracle that can DISAGREE with the masked ones). + + python3 tools/symcheck.py func_8014D820 --c .run/giants/s18_func_8014D820_close14.c \ + --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077_after + python3 tools/symcheck.py func_8014D820 --obj build/.../foo.o --asm-subdir +""" +import argparse +import glob +import os +import re +import subprocess +import sys + +OBJDUMP = "mipsel-linux-gnu-objdump" +PY = ".venv/bin/python" +HERE = os.path.dirname(os.path.abspath(__file__)) + +# `%hi(sym)` / `%lo(sym)` / `jal sym` in a splat .s. The target .s is RESOLVED (no reloc records), so +# the symbol names only survive in this operand syntax — which is exactly why the masked oracles, +# which work on the encoded word, cannot see them. +_S_SYM = re.compile(r"%(?:hi|lo)\(([A-Za-z_][A-Za-z0-9_]*)\)|\bjal\s+([A-Za-z_][A-Za-z0-9_]*)") +_O_SYM = re.compile(r"R_MIPS_\S+\s+([A-Za-z_][A-Za-z0-9_]*)") + + +def target_syms(s_path): + out = set() + with open(s_path) as fh: + for line in fh: + for a, b in _S_SYM.findall(line): + out.add(a or b) + return out + + +def object_syms(obj, fn=None): + """Symbols referenced by relocations in `obj`. With `fn`, only relocs inside that function's + section slice — objects here are one-function-per-file, so the whole-object set is the same.""" + txt = subprocess.run([OBJDUMP, "-drz", "-r", obj], capture_output=True, text=True).stdout + return set(m.group(1) for m in _O_SYM.finditer(txt)) + + +def compile_draft(fn, c_path, asm_subdir, o0=False): + """Compile via match_one so the triple/flags can never drift from the real gate path (R33: + derive, don't re-derive). Returns the object it left behind.""" + work = os.path.join(".run/symcheck", f"{fn}.{os.getpid()}") + cmd = [PY, os.path.join(HERE, "match_one.py"), fn, "--c", c_path, + "--asm-subdir", asm_subdir, "--work", work] + if o0: + cmd.append("--o0") + subprocess.run(cmd, capture_output=True, text=True) + hits = glob.glob(os.path.join(work, "**", "*.o"), recursive=True) + if not hits: + sys.exit(f"symcheck: match_one produced no object for {fn} (does the draft compile?)") + return hits[0] + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("fn") + ap.add_argument("--c", help="draft C (externs + def); compiled via match_one") + ap.add_argument("--obj", help="pre-compiled object instead of --c") + ap.add_argument("--asm-subdir", required=True, help="dir holding .s") + ap.add_argument("--o0", action="store_true", help="compile at -O0 (the _o0 split subsegments)") + a = ap.parse_args() + + if not (a.c or a.obj): + ap.error("one of --c or --obj is required") + + s_path = os.path.join(a.asm_subdir, f"{a.fn}.s") + if not os.path.exists(s_path): + sys.exit(f"symcheck: no target .s at {s_path}") + + obj = a.obj or compile_draft(a.fn, a.c, a.asm_subdir, a.o0) + tgt, mine = target_syms(s_path), object_syms(obj, a.fn) + + missing = sorted(tgt - mine) + invented = sorted(mine - tgt) + + if not missing and not invented: + print(f"SYMS-OK {a.fn} {len(tgt)} symbols agree") + return 0 + + print(f"SYMS-DIFF {a.fn} target={len(tgt)} draft={len(mine)}") + for s in missing: + print(f" MISSING {s} (target references it; draft does not -- dropped ref, or an " + f"invented alias standing in for it)") + for s in invented: + print(f" INVENTED {s} (draft references it; target does not -- if no symbol table " + f"defines it, rtu_match will still read MATCH and the gate will always reject)") + return 1 + + +if __name__ == "__main__": + sys.exit(main())