From 86caa7dfdbba38eadcffe24cb02bdc76b9a43006 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 28 Jul 2026 13:20:27 -0600 Subject: [PATCH] =?UTF-8?q?fix(phase-29):=20T36/T37=20=E2=80=94=20my=20che?= =?UTF-8?q?cker=20had=20a=20FORM-FEED=20bug;=20the=20"12=20fabricated"=20w?= =?UTF-8?q?ere=20mine,=20not=20the=20agents'?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit T36 CORRECTION (the important half). Building the cookbook sweep tool surfaced a defect in tools/verify_map_findings.py, which I had already used to validate BOTH map audits: - GNU C sources use FORM FEED (\f) page separators — loop.c 47, cse.c 36, reload1.c 27, local-alloc.c 21. Python's splitlines() splits on \f; grep/sed do not. Every line number computed after the first \f was shifted (up to 47 in loop.c), which is LARGER than the checker's own +/-40 window — precisely how a real quote gets reported FABRICATED. - Re-run after the fix: T34 regalloc 27 OK/153 NEAR/0 FAB -> 180 OK/0/0. T35 four-file 47 OK/240 NEAR/12 FAB -> 299 OK/0/0. THE AGENTS' LINE NUMBERS WERE EXACT ALL ALONG. I had even written the false "off by +2..+19" claim into the T35 agent prompt. - MY DIAGNOSIS OF THE 12 WAS ALSO WRONG. I said agents pasted map prose into source_quote and "verified" it by grepping — but I grepped claim_excerpt (which IS map prose) instead of source_quote. The real source_quote was ` record_jump_equiv (insn, 0);` at cse.c:7511, a correctly-located C line. Two stacked errors: a broken tool, then a check of the wrong field that appeared to confirm it. - Fixed: both tools use split("\n"); verify_map_findings.py documents the trap so it cannot return; loop.md's "12 unverified" note is WITHDRAWN in place. Nothing was deleted on this basis (all 12 were CONFIRMED-status, none underpinned a refutation), and the T34/T35 upheld/overturned splits are unaffected — those came from adversarial agents, not the checker. T37 THE COOKBOOK SWEEP (what was asked for). New tools/sweep_citations.py puts the mechanical half of a citation audit into zero-token tooling (offline-tooling-first): symbol-form cites are compared to the real 2.7.2 definition line; file-form cites are localised to their enclosing function. - matching-cookbook.md: 57 resolvable citations, all localisable. MIXED provenance but mostly sound — materially better than the map files. loop.c:5556, local-alloc.c:1765/1795/1825, global.c:906/917/ 924/1000, local-alloc.c:1021/1064, global.c:588/594, sched.c:820, expmed.c:556, jump.c:2131 all land where the prose says. GENUINE MISS: expr.c:5535 is MIN/MAX optab code; the /s grant sites are 4577 and 4904. - STATED LIMITATION: "lands in the right function" is weak for giants (expand_expr 4026->~6300, jump_optimize 139->~2200). This is a CITATION sweep, not a claim audit — proportionate because the cookbook's idioms are byte-proven and its cites are explanation. No idiom re-litigated. Docs+tools only: no src/ or config/ touched; R22 not re-run and not claimed. --- docs/gcc-2.7.2-map/loop.md | 6 ++ docs/matching-cookbook.md | 12 ++++ phase-ends/CURRENT_PHASE.md | 61 +++++++++++++++++ tools/sweep_citations.py | 126 +++++++++++++++++++++++++++++++++++ tools/verify_map_findings.py | 8 ++- 5 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 tools/sweep_citations.py diff --git a/docs/gcc-2.7.2-map/loop.md b/docs/gcc-2.7.2-map/loop.md index a83b51fed..574cd8a6c 100644 --- a/docs/gcc-2.7.2-map/loop.md +++ b/docs/gcc-2.7.2-map/loop.md @@ -30,6 +30,12 @@ differences are BEHAVIORAL and bit us immediately: > `check_dbra_loop` area returned evidence that quoted THIS FILE rather than the compiler source, so > they are **unverified, not confirmed** — the caveat table's own rows are the ones affected. Re-derive > them before leaning on a biv-elimination claim. +> **[A23-CORRECTION, same day]** The sentence above is WITHDRAWN. Those 12 findings are **VERIFIED, +> not unverified** — the failure was in MY checker, not the agents' evidence: it used Python +> `splitlines()`, which splits on the FORM FEED (`\f`) page separators GNU C sources are full of +> (loop.c has 47), so every computed line number after the first `\f` was shifted by up to 47 — +> beyond the checker's own ±40 window, which turned real quotes into "FABRICATED". Re-run after the +> fix: **299/299 exact, 0 fabricated.** The agents' line numbers were right all along. A vanilla 2.7.2 extraction is at `.run/gccmap/gcc-2.7.2-vanilla-src/` (loop.c, unroll.c, sched.c, cse.c, rtl.h, config/mips). **Recommend promoting it to `tools/reference/` — diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 0572ac8d3..7d9438253 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -1,5 +1,17 @@ # Matching Cookbook — reusable compiler idioms & C-writing techniques +> **⚠️ SOURCE-CITATION PROVENANCE (swept 2026-07-28, `tools/sweep_citations.py`).** This file's ~57 +> gcc-source citations are **MIXED provenance** — some are exact for our gcc 2.7.2 +> (`tools/reference/gcc-2.7.2/`), others came from `gcc-papermario`, which is gcc **2.8.1**. Spot- +> verified exact: `loop.c:5556` (`emit_iv_add_mult`), `local-alloc.c:1765/1795/1825` (`combine_regs`), +> `global.c:906/917/924/1000` (`find_reg`), `local-alloc.c:1021/1064` (`update_equiv_regs`). +> **Known miss: `expr.c:5535`** — that line is `MIN_EXPR`/`MAX_EXPR` optab code; the `/s` grant sites +> are **4577** and **4904**. +> **Why this is lower-stakes here than in `docs/gcc-2.7.2-map/*`:** those files are source-derived +> *reasoning*, so a bad read means bad advice. This file's idioms are **byte-proven**, with citations +> attached as *explanation* — a drifted cite corrupts the why, not the lever. Re-derive a citation +> (`grep -n '^sym (' tools/reference/gcc-2.7.2/*.c`) before building new reasoning on it. + > **Evolvable reference (docs/ layer). Created Phase 6, 2026-06-14.** Append an entry every > time a reusable nuance is found — these recur across nearly every function, so capturing > them once accelerates all future matching. Companion to `SETUP.md` §5 (the pinned triple) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index ed3f38a4d..6cab54b21 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -6905,3 +6905,64 @@ as explanation.** A drifted cookbook citation corrupts the *explanation* while t works*. Recommended: a **targeted citation sweep**, not a claim-by-claim audit. NOT yet done. **Docs-only — no `src/`/`config/` touched, R22 not re-run and not claimed.** + +## ⚠️ T36 — CORRECTION: the "12 FABRICATED" were MY tool's bug, and so was my diagnosis of it + +Drew asked for the cookbook sweep. Building the tool for it surfaced a defect in the tool I had +already used to validate **both** map audits — so this entry corrects the record before it reports +the sweep. + +### The bug: `splitlines()` vs FORM FEEDS +GNU C sources use **form-feed (`\f`) page separators** — `loop.c` has 47, `cse.c` 36, `reload1.c` 27, +`local-alloc.c` 21. **Python's `str.splitlines()` splits on `\f`; `grep`/`sed`/editors do not.** So +every line number my checker computed after the first `\f` was shifted (by 20 in local-alloc.c, up to +47 in loop.c) — and 47 is **larger than the checker's own ±40 search window**, which is exactly how a +real quote gets reported as FABRICATED. + +### What that invalidates — all in the agents' favour +| audit | as I reported it | **after the fix** | +|---|---|---| +| T34 regalloc | 27 OK · 153 NEAR · 0 FAB | **180 OK · 0 NEAR · 0 FAB** | +| T35 four files | 47 OK · 240 NEAR · **12 FAB** | **299 OK · 0 NEAR · 0 FAB** | + +**The agents' line numbers were exact all along.** Both the "153 NEAR, off by +2..+19" and the "12 +FABRICATED" were my instrument. I even wrote the false NEAR claim into the T35 agent prompt, telling +agents to be careful about an error that was mine. + +### And my DIAGNOSIS of the 12 was also wrong +I claimed the agents had pasted map prose into the `source_quote` field, and "verified" it by +grepping — **but I grepped `claim_excerpt` (which IS map prose, so of course it is absent from the +`.c`) instead of `source_quote`.** The actual `source_quote` for the exemplar I cited was +` record_jump_equiv (insn, 0);` at `cse.c:7511` — a real C line, correctly located. Two errors +stacked: a broken tool, then a check of the wrong field that appeared to confirm it. + +**Corrected:** `loop.md`'s "12 unverified" note is withdrawn in place; both tools now use +`split("\n")`; `verify_map_findings.py` carries a comment naming the trap so it cannot return. +**Nothing was deleted on this basis** — the 12 were CONFIRMED-status and never underpinned a +refutation — so no map content needs revisiting. The T34/T35 upheld/overturned splits are unaffected +(they came from the adversarial agents, not the checker). + +## ✅ T37 — the cookbook citation sweep (the task Drew asked for) +New tool **`tools/sweep_citations.py`** — the mechanical half of a citation audit as zero-token +tooling (offline-tooling-first), leaving only judgement to a human: symbol-form cites are compared to +the symbol's real 2.7.2 definition line (fully determined); file-form cites are localised to the +function that encloses them. + +**`docs/matching-cookbook.md`: 57 resolvable citations, all localisable, 0 out-of-file.** +Provenance is **MIXED but mostly sound** — far better than the map files. Verified by hand: +- `loop.c:5556` → `emit_iv_add_mult`, **exact** (the giv-init `emit_move_insn` the prose names). +- `local-alloc.c:1765/1795/1825` → `combine_regs`; `global.c:906/917/924/1000` → `find_reg`; + `local-alloc.c:1021/1064` → `update_equiv_regs`; `global.c:588/594` → `allocno_compare`; + `sched.c:820` → `true_dependence`; `expmed.c:556` → `store_fixed_bit_field`; + `jump.c:2131` → `duplicate_loop_exit_test` — all land where the prose says. +- **`expr.c:5535` is a genuine miss** (confirmed by grep, not the tool): that line is `MIN_EXPR`/ + `MAX_EXPR` optab code, while the `/s` grant sites are **4577** and **4904**. + +**Stated limitation, so nobody over-reads this:** "lands in the right function" is weak evidence for +the giant functions — `expand_expr` (4026→~6300), `jump_optimize` (139→~2200), `try_combine`, +`block_alloc`, `find_reg`. For those the sweep narrows nothing, and a claim-level check would be +needed. **This is a citation sweep, not a claim audit** — proportionate, because the cookbook's +idioms are byte-proven and its citations are explanation (a drifted cite corrupts the *why*, not the +*lever*). No cookbook idiom was re-litigated and none needs to be. + +**Docs+tools only — no `src/`/`config/`; R22 not re-run, not claimed.** diff --git a/tools/sweep_citations.py b/tools/sweep_citations.py new file mode 100644 index 000000000..f95ac1817 --- /dev/null +++ b/tools/sweep_citations.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""sweep_citations.py — deterministically localise every gcc-source citation in a markdown doc. + +WHY THIS IS A TOOL AND NOT AN AGENT WAVE. The codegen-map audits (T34/T35) established that our +docs cite `tools/reference/gcc-papermario/` = gcc **2.8.1**, while we build with **2.7.2** +(`tools/reference/gcc-2.7.2/`), and that the drift is large and NON-uniform (+27 in sched.c, +611 in +reload1.c) — large enough that a citation lands inside a DIFFERENT FUNCTION and every sentence built +on it still reads plausibly. Deciding *whether a claim is true* needs judgement. Deciding *where a +line number actually points* does not: it is a lookup, and a lookup belongs in zero-token tooling. + +So this tool answers the mechanical half exactly, and leaves only genuine judgement for a human/agent: + + symbol-form cite (`combine_regs:1825`) -> compare to the symbol's REAL 2.7.2 definition line. + Verdict is fully determined: OK / DRIFT(+N) / NO-SYMBOL. + file-form cite (`cse.c:8626`) -> report which function that line falls INSIDE in 2.7.2, + so a reader can see at a glance whether the doc's + surrounding prose is talking about that function at all. + +A cite whose enclosing function differs from the symbol the prose names is the exact signature of the +"landed in a neighbouring function" failure — that is what to grep the report for. + + python3 tools/sweep_citations.py docs/matching-cookbook.md [--src tools/reference/gcc-2.7.2] +""" +import argparse +import os +import re +import sys + +DEF_RE = re.compile(r"^([a-z_][a-z0-9_]*)\s*\(") # K&R definition: name on its own line +# `sym:1234` or `file.c:1234` (optionally a range `1234-1250`) +CITE_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_./-]*?)\s*:\s*(\d{2,5})(?:\s*-\s*\d{2,5})?") +SKIP_PREFIX = ("http", "https", "ftp", "Phase", "phase", "§") + + +def index_source(src): + """(defs, lines) — defs: symbol -> [(file, line)] ; lines: file -> [text].""" + defs, lines = {}, {} + for root, _dirs, files in os.walk(src): + for f in files: + if not f.endswith((".c", ".h", ".md")): + continue + p = os.path.join(root, f) + rel = os.path.relpath(p, src) + try: + L = open(p, errors="replace").read().split("\n") + except Exception: + continue + lines[rel] = L + lines.setdefault(os.path.basename(rel), L) # allow bare "cse.c" + for i, ln in enumerate(L, 1): + m = DEF_RE.match(ln) + if m: + defs.setdefault(m.group(1), []).append((rel, i)) + return defs, lines + + +def enclosing(lines, fname, line): + """Name of the function whose K&R definition most recently precedes `line`.""" + L = lines.get(fname) or lines.get(os.path.basename(fname)) + if not L or not (1 <= line <= len(L)): + return None, len(L) if L else 0 + best = None + for i in range(line - 1, -1, -1): + m = DEF_RE.match(L[i]) + if m: + best = (m.group(1), i + 1) + break + return best, len(L) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("doc") + ap.add_argument("--src", default="tools/reference/gcc-2.7.2") + ap.add_argument("--only-problems", action="store_true") + a = ap.parse_args() + + if not os.path.isdir(a.src): + print("no source tree at %s" % a.src) + sys.exit(2) + defs, lines = index_source(a.src) + + rows, seen = [], set() + for n, raw in enumerate(open(a.doc, errors="replace").read().split("\n"), 1): + for m in CITE_RE.finditer(raw): + who, ln = m.group(1), int(m.group(2)) + if who.startswith(SKIP_PREFIX) or who.isdigit(): + continue + key = (who, ln) + if key in seen: + continue + seen.add(key) + is_file = who.endswith((".c", ".h", ".md")) or "/" in who + if is_file: + base = os.path.basename(who) + if base not in lines and base.replace(".md", "") not in lines: + continue + enc, total = enclosing(lines, base, ln) + if enc is None: + rows.append((n, who, ln, "OUT-OF-RANGE", "file has %d lines" % total)) + else: + rows.append((n, who, ln, "IN-FN", "%s (defined :%d)" % (enc[0], enc[1]))) + else: + if who not in defs: + continue # not a gcc symbol we can resolve; skip quietly + cands = defs[who] + near = min(cands, key=lambda c: abs(c[1] - ln)) + d = ln - near[1] + verdict = "OK" if abs(d) <= 3 else "DRIFT %+d" % d + rows.append((n, who, ln, verdict, "%s real :%d" % (near[0], near[1]))) + + probs = [r for r in rows if r[3].startswith("DRIFT") or r[3] == "OUT-OF-RANGE"] + print("%s — %d resolvable citations" % (a.doc, len(rows))) + print(" symbol-form OK : %d" % len([r for r in rows if r[3] == "OK"])) + print(" symbol-form DRIFTED : %d" % len([r for r in rows if r[3].startswith("DRIFT")])) + print(" file-form localised : %d" % len([r for r in rows if r[3] == "IN-FN"])) + print(" out of range : %d" % len([r for r in rows if r[3] == "OUT-OF-RANGE"])) + print() + show = probs if a.only_problems else rows + print("%-6s %-30s %-7s %-12s %s" % ("mdline", "cite", "line", "verdict", "in real 2.7.2")) + for n, who, ln, v, note in show: + print("%-6d %-30s %-7d %-12s %s" % (n, who[:30], ln, v, note)) + + +if __name__ == "__main__": + main() diff --git a/tools/verify_map_findings.py b/tools/verify_map_findings.py index d201d7dcc..348938e78 100644 --- a/tools/verify_map_findings.py +++ b/tools/verify_map_findings.py @@ -29,6 +29,12 @@ SRC = os.path.join(REPO, "tools/reference/gcc-2.7.2") WINDOW = 40 +# NB — LINE NUMBERING: split("\n"), never splitlines(). GNU C sources use FORM FEED (\f) page +# separators (loop.c has 47, cse.c 36, reload1.c 27), and Python's splitlines() splits on \f while +# grep/sed/editors do not. Using splitlines() shifts every computed line number after the first \f +# — by up to 47 in loop.c, which is larger than this file's +/-40 search window, so a REAL quote can +# be reported FABRICATED. Self-inflicted and caught 2026-07-28 by cross-checking one known value. + def norm(s): return re.sub(r"\s+", " ", (s or "")).strip() @@ -75,7 +81,7 @@ def check(f): p = os.path.join(SRC, rf.replace("tools/reference/gcc-2.7.2/", "").lstrip("/")) if not os.path.exists(p): return "BAD-FILE", "no such file: %s" % rf - lines = open(p, errors="replace").read().splitlines() + lines = open(p, errors="replace").read().split("\n") if not (1 <= rl <= len(lines)): return "BAD-LINE", "%s has %d lines, cited %d" % (rf, len(lines), rl) nq = norm(q)