mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
fix(phase-29): T36/T37 — my checker had a FORM-FEED bug; the "12 fabricated" were mine, not the agents'
T36 CORRECTION (the important half). Building the cookbook sweep tool surfaced a defect in
tools/verify_map_findings.py, which I had already used to validate BOTH map audits:
- GNU C sources use FORM FEED (\f) page separators — loop.c 47, cse.c 36, reload1.c 27,
local-alloc.c 21. Python's splitlines() splits on \f; grep/sed do not. Every line number computed
after the first \f was shifted (up to 47 in loop.c), which is LARGER than the checker's own +/-40
window — precisely how a real quote gets reported FABRICATED.
- Re-run after the fix: T34 regalloc 27 OK/153 NEAR/0 FAB -> 180 OK/0/0. T35 four-file
47 OK/240 NEAR/12 FAB -> 299 OK/0/0. THE AGENTS' LINE NUMBERS WERE EXACT ALL ALONG. I had even
written the false "off by +2..+19" claim into the T35 agent prompt.
- MY DIAGNOSIS OF THE 12 WAS ALSO WRONG. I said agents pasted map prose into source_quote and
"verified" it by grepping — but I grepped claim_excerpt (which IS map prose) instead of
source_quote. The real source_quote was ` record_jump_equiv (insn, 0);` at cse.c:7511, a
correctly-located C line. Two stacked errors: a broken tool, then a check of the wrong field that
appeared to confirm it.
- Fixed: both tools use split("\n"); verify_map_findings.py documents the trap so it cannot return;
loop.md's "12 unverified" note is WITHDRAWN in place. Nothing was deleted on this basis (all 12
were CONFIRMED-status, none underpinned a refutation), and the T34/T35 upheld/overturned splits are
unaffected — those came from adversarial agents, not the checker.
T37 THE COOKBOOK SWEEP (what was asked for). New tools/sweep_citations.py puts the mechanical half of
a citation audit into zero-token tooling (offline-tooling-first): symbol-form cites are compared to
the real 2.7.2 definition line; file-form cites are localised to their enclosing function.
- matching-cookbook.md: 57 resolvable citations, all localisable. MIXED provenance but mostly sound —
materially better than the map files. loop.c:5556, local-alloc.c:1765/1795/1825, global.c:906/917/
924/1000, local-alloc.c:1021/1064, global.c:588/594, sched.c:820, expmed.c:556, jump.c:2131 all
land where the prose says. GENUINE MISS: expr.c:5535 is MIN/MAX optab code; the /s grant sites are
4577 and 4904.
- STATED LIMITATION: "lands in the right function" is weak for giants (expand_expr 4026->~6300,
jump_optimize 139->~2200). This is a CITATION sweep, not a claim audit — proportionate because the
cookbook's idioms are byte-proven and its cites are explanation. No idiom re-litigated.
Docs+tools only: no src/ or config/ touched; R22 not re-run and not claimed.
This commit is contained in:
Drew T
@@ -30,6 +30,12 @@ differences are BEHAVIORAL and bit us immediately:
|
||||
> `check_dbra_loop` area returned evidence that quoted THIS FILE rather than the compiler source, so
|
||||
> they are **unverified, not confirmed** — the caveat table's own rows are the ones affected. Re-derive
|
||||
> them before leaning on a biv-elimination claim.
|
||||
> **[A23-CORRECTION, same day]** The sentence above is WITHDRAWN. Those 12 findings are **VERIFIED,
|
||||
> not unverified** — the failure was in MY checker, not the agents' evidence: it used Python
|
||||
> `splitlines()`, which splits on the FORM FEED (`\f`) page separators GNU C sources are full of
|
||||
> (loop.c has 47), so every computed line number after the first `\f` was shifted by up to 47 —
|
||||
> beyond the checker's own ±40 window, which turned real quotes into "FABRICATED". Re-run after the
|
||||
> fix: **299/299 exact, 0 fabricated.** The agents' line numbers were right all along.
|
||||
|
||||
A vanilla 2.7.2 extraction is at `.run/gccmap/gcc-2.7.2-vanilla-src/` (loop.c, unroll.c,
|
||||
sched.c, cse.c, rtl.h, config/mips). **Recommend promoting it to `tools/reference/` —
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Matching Cookbook — reusable compiler idioms & C-writing techniques
|
||||
|
||||
> **⚠️ SOURCE-CITATION PROVENANCE (swept 2026-07-28, `tools/sweep_citations.py`).** This file's ~57
|
||||
> gcc-source citations are **MIXED provenance** — some are exact for our gcc 2.7.2
|
||||
> (`tools/reference/gcc-2.7.2/`), others came from `gcc-papermario`, which is gcc **2.8.1**. Spot-
|
||||
> verified exact: `loop.c:5556` (`emit_iv_add_mult`), `local-alloc.c:1765/1795/1825` (`combine_regs`),
|
||||
> `global.c:906/917/924/1000` (`find_reg`), `local-alloc.c:1021/1064` (`update_equiv_regs`).
|
||||
> **Known miss: `expr.c:5535`** — that line is `MIN_EXPR`/`MAX_EXPR` optab code; the `/s` grant sites
|
||||
> are **4577** and **4904**.
|
||||
> **Why this is lower-stakes here than in `docs/gcc-2.7.2-map/*`:** those files are source-derived
|
||||
> *reasoning*, so a bad read means bad advice. This file's idioms are **byte-proven**, with citations
|
||||
> attached as *explanation* — a drifted cite corrupts the why, not the lever. Re-derive a citation
|
||||
> (`grep -n '^sym (' tools/reference/gcc-2.7.2/*.c`) before building new reasoning on it.
|
||||
|
||||
> **Evolvable reference (docs/ layer). Created Phase 6, 2026-06-14.** Append an entry every
|
||||
> time a reusable nuance is found — these recur across nearly every function, so capturing
|
||||
> them once accelerates all future matching. Companion to `SETUP.md` §5 (the pinned triple)
|
||||
|
||||
@@ -6905,3 +6905,64 @@ as explanation.** A drifted cookbook citation corrupts the *explanation* while t
|
||||
works*. Recommended: a **targeted citation sweep**, not a claim-by-claim audit. NOT yet done.
|
||||
|
||||
**Docs-only — no `src/`/`config/` touched, R22 not re-run and not claimed.**
|
||||
|
||||
## ⚠️ T36 — CORRECTION: the "12 FABRICATED" were MY tool's bug, and so was my diagnosis of it
|
||||
|
||||
Drew asked for the cookbook sweep. Building the tool for it surfaced a defect in the tool I had
|
||||
already used to validate **both** map audits — so this entry corrects the record before it reports
|
||||
the sweep.
|
||||
|
||||
### The bug: `splitlines()` vs FORM FEEDS
|
||||
GNU C sources use **form-feed (`\f`) page separators** — `loop.c` has 47, `cse.c` 36, `reload1.c` 27,
|
||||
`local-alloc.c` 21. **Python's `str.splitlines()` splits on `\f`; `grep`/`sed`/editors do not.** So
|
||||
every line number my checker computed after the first `\f` was shifted (by 20 in local-alloc.c, up to
|
||||
47 in loop.c) — and 47 is **larger than the checker's own ±40 search window**, which is exactly how a
|
||||
real quote gets reported as FABRICATED.
|
||||
|
||||
### What that invalidates — all in the agents' favour
|
||||
| audit | as I reported it | **after the fix** |
|
||||
|---|---|---|
|
||||
| T34 regalloc | 27 OK · 153 NEAR · 0 FAB | **180 OK · 0 NEAR · 0 FAB** |
|
||||
| T35 four files | 47 OK · 240 NEAR · **12 FAB** | **299 OK · 0 NEAR · 0 FAB** |
|
||||
|
||||
**The agents' line numbers were exact all along.** Both the "153 NEAR, off by +2..+19" and the "12
|
||||
FABRICATED" were my instrument. I even wrote the false NEAR claim into the T35 agent prompt, telling
|
||||
agents to be careful about an error that was mine.
|
||||
|
||||
### And my DIAGNOSIS of the 12 was also wrong
|
||||
I claimed the agents had pasted map prose into the `source_quote` field, and "verified" it by
|
||||
grepping — **but I grepped `claim_excerpt` (which IS map prose, so of course it is absent from the
|
||||
`.c`) instead of `source_quote`.** The actual `source_quote` for the exemplar I cited was
|
||||
` record_jump_equiv (insn, 0);` at `cse.c:7511` — a real C line, correctly located. Two errors
|
||||
stacked: a broken tool, then a check of the wrong field that appeared to confirm it.
|
||||
|
||||
**Corrected:** `loop.md`'s "12 unverified" note is withdrawn in place; both tools now use
|
||||
`split("\n")`; `verify_map_findings.py` carries a comment naming the trap so it cannot return.
|
||||
**Nothing was deleted on this basis** — the 12 were CONFIRMED-status and never underpinned a
|
||||
refutation — so no map content needs revisiting. The T34/T35 upheld/overturned splits are unaffected
|
||||
(they came from the adversarial agents, not the checker).
|
||||
|
||||
## ✅ T37 — the cookbook citation sweep (the task Drew asked for)
|
||||
New tool **`tools/sweep_citations.py`** — the mechanical half of a citation audit as zero-token
|
||||
tooling (offline-tooling-first), leaving only judgement to a human: symbol-form cites are compared to
|
||||
the symbol's real 2.7.2 definition line (fully determined); file-form cites are localised to the
|
||||
function that encloses them.
|
||||
|
||||
**`docs/matching-cookbook.md`: 57 resolvable citations, all localisable, 0 out-of-file.**
|
||||
Provenance is **MIXED but mostly sound** — far better than the map files. Verified by hand:
|
||||
- `loop.c:5556` → `emit_iv_add_mult`, **exact** (the giv-init `emit_move_insn` the prose names).
|
||||
- `local-alloc.c:1765/1795/1825` → `combine_regs`; `global.c:906/917/924/1000` → `find_reg`;
|
||||
`local-alloc.c:1021/1064` → `update_equiv_regs`; `global.c:588/594` → `allocno_compare`;
|
||||
`sched.c:820` → `true_dependence`; `expmed.c:556` → `store_fixed_bit_field`;
|
||||
`jump.c:2131` → `duplicate_loop_exit_test` — all land where the prose says.
|
||||
- **`expr.c:5535` is a genuine miss** (confirmed by grep, not the tool): that line is `MIN_EXPR`/
|
||||
`MAX_EXPR` optab code, while the `/s` grant sites are **4577** and **4904**.
|
||||
|
||||
**Stated limitation, so nobody over-reads this:** "lands in the right function" is weak evidence for
|
||||
the giant functions — `expand_expr` (4026→~6300), `jump_optimize` (139→~2200), `try_combine`,
|
||||
`block_alloc`, `find_reg`. For those the sweep narrows nothing, and a claim-level check would be
|
||||
needed. **This is a citation sweep, not a claim audit** — proportionate, because the cookbook's
|
||||
idioms are byte-proven and its citations are explanation (a drifted cite corrupts the *why*, not the
|
||||
*lever*). No cookbook idiom was re-litigated and none needs to be.
|
||||
|
||||
**Docs+tools only — no `src/`/`config/`; R22 not re-run, not claimed.**
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""sweep_citations.py — deterministically localise every gcc-source citation in a markdown doc.
|
||||
|
||||
WHY THIS IS A TOOL AND NOT AN AGENT WAVE. The codegen-map audits (T34/T35) established that our
|
||||
docs cite `tools/reference/gcc-papermario/` = gcc **2.8.1**, while we build with **2.7.2**
|
||||
(`tools/reference/gcc-2.7.2/`), and that the drift is large and NON-uniform (+27 in sched.c, +611 in
|
||||
reload1.c) — large enough that a citation lands inside a DIFFERENT FUNCTION and every sentence built
|
||||
on it still reads plausibly. Deciding *whether a claim is true* needs judgement. Deciding *where a
|
||||
line number actually points* does not: it is a lookup, and a lookup belongs in zero-token tooling.
|
||||
|
||||
So this tool answers the mechanical half exactly, and leaves only genuine judgement for a human/agent:
|
||||
|
||||
symbol-form cite (`combine_regs:1825`) -> compare to the symbol's REAL 2.7.2 definition line.
|
||||
Verdict is fully determined: OK / DRIFT(+N) / NO-SYMBOL.
|
||||
file-form cite (`cse.c:8626`) -> report which function that line falls INSIDE in 2.7.2,
|
||||
so a reader can see at a glance whether the doc's
|
||||
surrounding prose is talking about that function at all.
|
||||
|
||||
A cite whose enclosing function differs from the symbol the prose names is the exact signature of the
|
||||
"landed in a neighbouring function" failure — that is what to grep the report for.
|
||||
|
||||
python3 tools/sweep_citations.py docs/matching-cookbook.md [--src tools/reference/gcc-2.7.2]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
DEF_RE = re.compile(r"^([a-z_][a-z0-9_]*)\s*\(") # K&R definition: name on its own line
|
||||
# `sym:1234` or `file.c:1234` (optionally a range `1234-1250`)
|
||||
CITE_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_./-]*?)\s*:\s*(\d{2,5})(?:\s*-\s*\d{2,5})?")
|
||||
SKIP_PREFIX = ("http", "https", "ftp", "Phase", "phase", "§")
|
||||
|
||||
|
||||
def index_source(src):
|
||||
"""(defs, lines) — defs: symbol -> [(file, line)] ; lines: file -> [text]."""
|
||||
defs, lines = {}, {}
|
||||
for root, _dirs, files in os.walk(src):
|
||||
for f in files:
|
||||
if not f.endswith((".c", ".h", ".md")):
|
||||
continue
|
||||
p = os.path.join(root, f)
|
||||
rel = os.path.relpath(p, src)
|
||||
try:
|
||||
L = open(p, errors="replace").read().split("\n")
|
||||
except Exception:
|
||||
continue
|
||||
lines[rel] = L
|
||||
lines.setdefault(os.path.basename(rel), L) # allow bare "cse.c"
|
||||
for i, ln in enumerate(L, 1):
|
||||
m = DEF_RE.match(ln)
|
||||
if m:
|
||||
defs.setdefault(m.group(1), []).append((rel, i))
|
||||
return defs, lines
|
||||
|
||||
|
||||
def enclosing(lines, fname, line):
|
||||
"""Name of the function whose K&R definition most recently precedes `line`."""
|
||||
L = lines.get(fname) or lines.get(os.path.basename(fname))
|
||||
if not L or not (1 <= line <= len(L)):
|
||||
return None, len(L) if L else 0
|
||||
best = None
|
||||
for i in range(line - 1, -1, -1):
|
||||
m = DEF_RE.match(L[i])
|
||||
if m:
|
||||
best = (m.group(1), i + 1)
|
||||
break
|
||||
return best, len(L)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("doc")
|
||||
ap.add_argument("--src", default="tools/reference/gcc-2.7.2")
|
||||
ap.add_argument("--only-problems", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
if not os.path.isdir(a.src):
|
||||
print("no source tree at %s" % a.src)
|
||||
sys.exit(2)
|
||||
defs, lines = index_source(a.src)
|
||||
|
||||
rows, seen = [], set()
|
||||
for n, raw in enumerate(open(a.doc, errors="replace").read().split("\n"), 1):
|
||||
for m in CITE_RE.finditer(raw):
|
||||
who, ln = m.group(1), int(m.group(2))
|
||||
if who.startswith(SKIP_PREFIX) or who.isdigit():
|
||||
continue
|
||||
key = (who, ln)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
is_file = who.endswith((".c", ".h", ".md")) or "/" in who
|
||||
if is_file:
|
||||
base = os.path.basename(who)
|
||||
if base not in lines and base.replace(".md", "") not in lines:
|
||||
continue
|
||||
enc, total = enclosing(lines, base, ln)
|
||||
if enc is None:
|
||||
rows.append((n, who, ln, "OUT-OF-RANGE", "file has %d lines" % total))
|
||||
else:
|
||||
rows.append((n, who, ln, "IN-FN", "%s (defined :%d)" % (enc[0], enc[1])))
|
||||
else:
|
||||
if who not in defs:
|
||||
continue # not a gcc symbol we can resolve; skip quietly
|
||||
cands = defs[who]
|
||||
near = min(cands, key=lambda c: abs(c[1] - ln))
|
||||
d = ln - near[1]
|
||||
verdict = "OK" if abs(d) <= 3 else "DRIFT %+d" % d
|
||||
rows.append((n, who, ln, verdict, "%s real :%d" % (near[0], near[1])))
|
||||
|
||||
probs = [r for r in rows if r[3].startswith("DRIFT") or r[3] == "OUT-OF-RANGE"]
|
||||
print("%s — %d resolvable citations" % (a.doc, len(rows)))
|
||||
print(" symbol-form OK : %d" % len([r for r in rows if r[3] == "OK"]))
|
||||
print(" symbol-form DRIFTED : %d" % len([r for r in rows if r[3].startswith("DRIFT")]))
|
||||
print(" file-form localised : %d" % len([r for r in rows if r[3] == "IN-FN"]))
|
||||
print(" out of range : %d" % len([r for r in rows if r[3] == "OUT-OF-RANGE"]))
|
||||
print()
|
||||
show = probs if a.only_problems else rows
|
||||
print("%-6s %-30s %-7s %-12s %s" % ("mdline", "cite", "line", "verdict", "in real 2.7.2"))
|
||||
for n, who, ln, v, note in show:
|
||||
print("%-6d %-30s %-7d %-12s %s" % (n, who[:30], ln, v, note))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -29,6 +29,12 @@ SRC = os.path.join(REPO, "tools/reference/gcc-2.7.2")
|
||||
WINDOW = 40
|
||||
|
||||
|
||||
# NB — LINE NUMBERING: split("\n"), never splitlines(). GNU C sources use FORM FEED (\f) page
|
||||
# separators (loop.c has 47, cse.c 36, reload1.c 27), and Python's splitlines() splits on \f while
|
||||
# grep/sed/editors do not. Using splitlines() shifts every computed line number after the first \f
|
||||
# — by up to 47 in loop.c, which is larger than this file's +/-40 search window, so a REAL quote can
|
||||
# be reported FABRICATED. Self-inflicted and caught 2026-07-28 by cross-checking one known value.
|
||||
|
||||
def norm(s):
|
||||
return re.sub(r"\s+", " ", (s or "")).strip()
|
||||
|
||||
@@ -75,7 +81,7 @@ def check(f):
|
||||
p = os.path.join(SRC, rf.replace("tools/reference/gcc-2.7.2/", "").lstrip("/"))
|
||||
if not os.path.exists(p):
|
||||
return "BAD-FILE", "no such file: %s" % rf
|
||||
lines = open(p, errors="replace").read().splitlines()
|
||||
lines = open(p, errors="replace").read().split("\n")
|
||||
if not (1 <= rl <= len(lines)):
|
||||
return "BAD-LINE", "%s has %d lines, cited %d" % (rf, len(lines), rl)
|
||||
nq = norm(q)
|
||||
|
||||
Reference in New Issue
Block a user