diff --git a/docs/decision-log.md b/docs/decision-log.md index 6e04b238e..95b966c2e 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -959,3 +959,41 @@ signature from garbage. explicitly warns that *making the parser see more ARMS dormant downstream transforms* — the moment `reconcile_decls` can parse a fn-ptr decl, its `data_access_subs` would happily mangle `D_1[i]()` into `((u8 *)D_1)[i]()`. Consumer migration is therefore one tool at a time, each byte-gated. + +--- + +## 2026-07-14 — Probe the compiler; and the adjudicator must BE the compiler + +**Context.** Building `cdecl.compatible()` — *"will cc1 accept these two declarations of one name?"*, the +question every recovery pass in this repo actually asks and four of them half-implement. I wrote the rules +from the C standard, then validated them against a compiler. + +**What happened.** The compiler contradicted me — and then the *right* compiler contradicted the first one. +Validating against modern `mipsel-linux-gnu-gcc` and against the real gcc-2.7.2 `cc1` gives **three +different answers** (with the standard as a third): typedef redefinition is an error in C89, accepted by +C11 gcc, and rejected by cc1; a qualifier mismatch is an error to modern gcc and **accepted** by cc1; the +no-prototype/narrow-param rule is an error to both — and **accepted by cc1 in one direction.** + +**The decision.** `--compat` adjudicates with `tools/bin/gcc-2.7.2-psx/cc1`, the front end that actually +arbitrates the build. Now 1,485/1,485 live corpus pairs agree. **Validating a compiler rule against a +compiler that is not the one compiling your code is not a shortcut — it is the same class of error as the +five phases we spent reading `gcc-papermario` believing it was 2.7.2. It was 2.8.1.** + +**The prize (→ A10).** Phase 15 closed the "159 arity/narrow-param conflicts" as *"no clean deterministic +fix — it is simply C's default-promotion rule."* **cc1 disagrees.** The rule is order-dependent: +`void X(s16); void X();` compiles; only `void X(); void X(s16);` fails. The wall's stated cause does not +hold. Four three-line probes, 90 seconds, zero tokens. + +**Hindsight / for the wiki.** *Probe the compiler for FACTS; read its source only for LEVERS; byte-validate +both.* Reading source is inference and can be wrong (it was, for five phases). Probing is ground truth, +because it IS the compiler — and it is orders of magnitude cheaper. We have the exact binary sitting in the +tree and spent 26 phases reasoning about it instead of asking it. + +**And the discipline that saved this from being an over-claim.** Fixing the wrong-TU bug (95.1% of drafts +canonicalized against a TU that would never compile them) took the callee-conflict repair from 8 to 58 of +196 drafts — 7× reach — and banked **exactly zero** functions, because the historical tail fails on codegen, +not plumbing. The real gain is narrower and still worth having: **52 drafts moved from "won't compile" to +"compiles, N instructions off"** — from an invisible failure that reads as a compiler wall into a scored +near-miss the permuter can act on. Three times in one session a confirmed mechanism produced a null +consequence. *"This tool is broken" and "this number will move" are different claims, needing different +evidence.* diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index ea63c832e..7677b6034 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -3681,3 +3681,46 @@ a compiler wall. **Gate:** `make audit-cdecl` (coverage + gcc). Standing, because a loud failure nobody counts is exactly as invisible as a silent one (LAW 2). + +**LAW 9 — THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE.** Not the C standard, and not +whatever `gcc` is on the PATH. Building `cdecl.compatible()` (*"will this declaration coexist with that +one?"* — the question every recovery pass actually asks) against **modern** `mipsel-linux-gnu-gcc` and +against the **real gcc-2.7.2 `cc1`** gives three different answers, and only one of them is the truth: + +| declarations in one TU | C standard | modern gcc (C11) | **gcc-2.7.2 `cc1`** | +|---|---|---|---| +| `typedef int X;` twice | error | **accepts** | **ERROR** — `redefinition of 'X'` | +| `extern u16 X;` + `extern volatile u16 X;` | error | error | **ACCEPTS** | +| `void X(s16);` then `void X();` | error | error | **ACCEPTS** | +| `void X();` then `void X(s16);` | error | error | **ERROR** | + +Validate against modern gcc and you encode rules cc1 rejects and miss rules cc1 accepts — a recovery pass +that "approves" declarations the real front end refuses is exactly the failure this whole audit is about. +`cdecl --compat` therefore adjudicates with `tools/bin/gcc-2.7.2-psx/cc1`, and now agrees with it on +**1,485/1,485 live corpus pairs**. Two rules in that table were *refuted* by the oracle after I had +already written them from the standard. + +> 🏆 **AND THE LAST ROW IS A WALL COMING DOWN.** The no-prototype rule is **ORDER-DEPENDENT**: only +> `()`-then-narrow-prototype fails; **prototype-then-`()` compiles fine.** Phase 15 wrote this class up as +> *"the 159 arity/narrow-param conflicts — no clean deterministic fix"* and closed it. **The stated cause +> does not hold.** Whether the resulting codegen matches is a separate question the byte-gate answers — +> but the door was never locked. It took four three-line probes and 90 seconds to find out. **Probe the +> compiler for FACTS; read its source only for LEVERS; byte-validate both** (we read `gcc-papermario` for +> five phases believing it was 2.7.2 — it was 2.8.1). + +**LAW 10 — DERIVE *WHICH* TU, TOO — not just what is in it.** `cast_call_sites` / `sig_unify` / +`reconcile_decls` take `--src-file`, an **optional, hand-passed** flag naming the TU to canonicalize +against; unset, it defaults to `src//.c`. No caller knows about the Phase-26 `_jr_` carves. +Measured on ov_SC01_077: **263 open stubs across 12 TUs — only 13 in the main `.c`. 95.1% of drafts were +being reconciled against a translation unit that would never compile them**, while `harvest_verify` +(fixed in A3) correctly spliced them into the right one. `corpus.stubs()` already knows the answer — the +`INCLUDE_ASM` line is self-describing. **Ask, don't assume.** Fixing it took the callee-conflict repair +from **8 → 58 of 196** drafts (7× reach). + +> **AND THE HONEST OTHER HALF (P9/R14):** those 58 produced **ZERO new banks.** The historical draft tail +> fails on *codegen*, not plumbing — `func_801387B8`, which the audit blamed on a single unparsed `[4]`, +> is really 67/100 instructions off with a `$s0`/`$s1` swap. What the fix *did* buy is real but narrower: +> **52 drafts moved from "won't compile" to "compiles, N instructions off."** That is not a bank — it is +> the difference between an **invisible failure that reads as a compiler wall** and a **scored near-miss +> the permuter and the §47/§48 dials can act on.** Which is the audit's thesis exactly. Do not sell it as +> more than it is; three times in one session a confirmed mechanism produced a null consequence. diff --git a/docs/tooling-audit.md b/docs/tooling-audit.md index 58615d4da..b782b7adb 100644 --- a/docs/tooling-audit.md +++ b/docs/tooling-audit.md @@ -726,6 +726,17 @@ scanners** — the "best outcome is a deleted scanner" rule (R33), applied at sc 3. **A recovery tool once wrote non-C into drafts:** `extern if ((func_80029178(0x119) & 0xFF) != 0);` appears in 33 drafts, *only* in `-canon`/`-recanon`/`-uni`/`-sigfix` output dirs (the pre-recovery draft has none), and gcc rejects it outright. **R14 blast-radius check: the source bug was already fixed in Phase 19** — today's oracle emits **0 garbage over 300 signatures** — so this is dead historical residue, not a live defect. *Mechanism confirmed, consequence nil.* But note what it cost at the time: a draft that cannot compile fails the byte-gate and reads, downstream, as **an intrinsic compiler wall**. 4. **`tu_ambient`'s function regex drops any callee with a fn-ptr parameter.** Its param class is `[^()]*`, so `extern void func_8012A568(void (*a0)(void));` — a real declaration in ov_SC01_077 — lands in **no bucket at all**: not funcs, not data, not typedefs. +### A3c — the first consumer migration (`cast_call_sites`), and what it measured + +| | | +|---|---| +| **[CRITICAL] the recovery passes canonicalize against the WRONG TU** | `--src-file` is an **optional hand-passed flag**; unset it defaults to `src//.c`, and no caller knows about the Phase-26 `_jr_` carves. Measured: ov_SC01_077 has **263 open stubs across 12 TUs — only 13 in the main `.c`**. So **95.1% of drafts were reconciled against a translation unit that would never compile them**, while `harvest_verify` (A3) correctly spliced them into the right one. Fixed by DERIVING the TU from `corpus.stubs()` (the `INCLUDE_ASM` line is self-describing). **Repair reach 8 → 58 of 196 drafts (7×).** | +| **[HIGH] the conflict predicate was wrong in both directions** | `norm_sig`/`_norm_type` collapse the int family to one token, so a **signedness** change reads as "already compatible" and gets no rewrite — while cc1 **rejects** that redeclaration. Replaced by `cdecl.compatible()`, validated against the real cc1 on **1,485/1,485** live corpus pairs. | +| **🏆 [FINDING] the Phase-15 narrow-param wall rests on a false premise** | The `()` no-prototype rule is **ORDER-DEPENDENT** on cc1: `void X(s16); void X();` **compiles**; only the reverse order fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as *"no clean deterministic fix"*. **The stated cause does not hold.** → **A10 re-test target.** | +| **[FINDING] cc1 ≠ modern gcc ≠ the C standard** | They give three different answers on typedef redefinition, qualifier mismatch, and the no-proto rule. Any compiler rule validated against modern gcc is validated against the wrong compiler. See cookbook §51g LAW 9 for the table. | + +**AND THE NULL RESULT, RECORDED HONESTLY (P9/R14).** Those 58 repaired drafts banked **ZERO** functions. The historical draft tail fails on **codegen**, not plumbing — `func_801387B8`, which finding F1 blames on a single unparsed `[4]`, is really **67/100 instructions off with a `$s0`/`$s1` swap** (that F1 claim does not reproduce on today's tree). What the fix genuinely buys: **52 drafts moved from "won't compile" to "compiles, N instructions off"** — from an *invisible failure that reads as a compiler wall* to a *scored near-miss the permuter can act on*. Real, but narrower than it first looked. **Three separate times in one session a confirmed mechanism produced a null consequence** — verify the blast radius, not just the defect. + ## Still open (round-2 findings not yet fixed) `masked_diff`/`match_one` (**155 provably-wrong closeness scores** — feeds false walls) · `harvest_verify._TD` diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index bea90f579..ea418dde4 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -659,6 +659,38 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-14 (session 10, A3c — first consumer migration: `cast_call_sites` onto the cdecl oracle; Max):** + Added **`cdecl.compatible()`** — *"will cc1 accept these two declarations of one name?"*, the predicate four + tools each half-implement and get wrong (`norm_sig`/`_norm_type` collapse the int family, so a **signedness** + change reads as "already compatible" and gets no rewrite — while cc1 **rejects** that redeclaration). Wrote the + rules from the C standard; then let a compiler judge. **It contradicted me — and then the RIGHT compiler + contradicted the first one.** Modern `mipsel-linux-gnu-gcc`, gcc-2.7.2 `cc1`, and the standard give **three + different answers** (typedef redefinition: C89 error / C11 accepts / **cc1 ERRORS**; qualifier mismatch: modern + gcc errors / **cc1 ACCEPTS**; no-proto + narrow param: both error / **cc1 accepts in ONE direction**). So + `--compat` now adjudicates with **`tools/bin/gcc-2.7.2-psx/cc1`, the front end that actually arbitrates the + build** → **1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.** 🏆 **THE PRIZE: the Phase-15 + narrow-param wall rests on a false premise.** The `()` rule is **ORDER-DEPENDENT**: `void X(s16); void X();` + **compiles**; only the reverse fails. "The 159 arity/narrow-param conflicts — no clean deterministic fix" was + closed on a rule cc1 does not enforce. Four three-line probes, 90 s, zero tokens → **A10 re-test target**. + **THEN the migration itself: `cast_call_sites` was canonicalizing 95.1% of drafts against the WRONG TU.** + `--src-file` is an *optional hand-passed flag* defaulting to `src//.c`, and no caller knows about the + Phase-26 `_jr_` carves — ov_SC01_077 has **263 open stubs across 12 TUs, only 13 in the main `.c`** — + while `harvest_verify` (A3) correctly splices into the real one. Now **DERIVED** from `corpus.stubs()` (the + `INCLUDE_ASM` line is self-describing) + the canonical map derived from `cdecl.tu_scope()` (cpp — so + macro-injected decls are finally visible). **Repair reach 8 → 58 of 196 drafts (7×).** + **THE NULL RESULT, REPORTED AS SUCH (P9/R14):** those 58 banked **ZERO**. The historical tail fails on + **codegen**, not plumbing — `func_801387B8`, which the audit blames on one unparsed `[4]`, is really **67/100 + instructions off with a `$s0`/`$s1` swap** (that claim does not reproduce). The real gain is narrower and still + worth having: **52 drafts moved from "won't compile" to "compiles, N instructions off"** — from an *invisible + failure that reads as a compiler wall* into a *scored near-miss the permuter can act on*. **Three times in one + session a confirmed mechanism produced a null consequence.** I also mis-diagnosed the callee oracle as + "returning nothing" (my probe was buggy — it did have the sig) — corrected. **And my own new audit printed + "ALL ORACLES GREEN" while silently skipping 100% of its corpus** (a missing `-Isrc`): the exact bug class, in + the tool written to hunt it. Fixed — *an unadjudicable check is not a passed check.* + **R22 clean-fleet 136/136 BYTE-IDENTICAL; src/ untouched (0 changes); `make audit-cdecl` green.** Knowledge + captured live (R30/R31): cookbook **§51g LAWS 9–10**, decision-log, tooling-audit **A3c**. + **NEXT: `sig_unify` + `reconcile_decls` have the SAME wrong-TU bug** (same `--src-file` flag) — migrate them, + then `lint_symbol_refs`, then **A10 (re-test the walls)** with the narrow-param finding as the first target. - **2026-07-14 (session 10, A3b — `tools/cdecl.py`, THE C-declaration oracle; Max):** Built the one parser that lets fifteen die. **Rejected the audit's own prescription** (a shape-aware alternation per tool, ~15 coordinated regex edits) on R33 grounds: fifteen hand-maintained models are precisely what diverged, and an diff --git a/tools/cast_call_sites.py b/tools/cast_call_sites.py index bf162ba05..392707d80 100644 --- a/tools/cast_call_sites.py +++ b/tools/cast_call_sites.py @@ -51,6 +51,30 @@ def _load(mod, rel): _ght = _load('ght', 'tools/gen_harvest_targets.py') _su = _load('su', 'tools/sig_unify.py') # reuse split_top_commas / param_type +_cdecl = _load('cdecl', 'tools/cdecl.py') # THE declaration oracle (Phase 26-A, §51g) +_corpus = _load('corpus', 'tools/corpus.py') # THE corpus oracle — which TU holds a stub + + +def tu_for(overlay, fn, override=None): + """The TU this draft will actually be SPLICED INTO — DERIVED from the corpus, not hand-passed. + + `--src-file` was an OPTIONAL flag defaulting to `src//.c`. Every caller that did not + know to set it — and none of them know about the Phase-26 `_jr_` carve files — was + therefore canonicalizing drafts against the MAIN .c while `harvest_verify` (fixed in A3) + correctly spliced them into the jr split. The recovery passes were reconciling against a + DIFFERENT TRANSLATION UNIT than the one that would compile the code, and the heavy Phase-26 + cores live in exactly those jr files. + + The INCLUDE_ASM line is self-describing, and corpus.stubs() reads it. Ask, don't assume.""" + if override: + return override + try: + st = _corpus.stubs(overlay).get(int(fn[5:], 16)) + if st: + return os.path.join(REPO, st.path) + except Exception: + pass + return os.path.join(REPO, f'src/{overlay}/{overlay}.c') # A function declaration line (prototype ending in `;`, NOT a definition): optional `extern`, # a type, func_X, a param list, `;`, optional trailing /* comment */. Matches both the @@ -91,16 +115,30 @@ def cast_type(dret, dptypes): def canonical_map(overlay, src_file=None): - """addr-int -> canonical sig string ' func_X()' for every func the TU declares - or defines (definitions/inline win over plain externs — the authoritative in-TU signature). - Mirrors sig_unify's precedence exactly. src_file overrides the .c (use the SPLIT file _a.c/_o0.c - for a draft that lands there — its TU sees that file's local decls, NOT the main .c's, so a - cross-file loose-typed callee (e.g. RotTransSV declared differently in main vs _a) resolves right).""" - ec = os.path.join(REPO, 'src/shared/engine_core.h') + """addr-int -> canonical sig string ' func_X()' — DERIVED from what cc1 actually + sees in the TU (`cdecl.tu_scope`, i.e. cpp), not scraped out of the raw text. + + WHY THIS CHANGED (Phase 26-A, R33; cookbook §51g LAW 7) + ------------------------------------------------------ + It used to union three raw-text scanners: + collect_extern_sigs([engine_core.h, the .c]) + collect_define_sigs(ec) + collect_inline_sigs(c) + and NONE of them can see a MACRO-INJECTED declaration — an `extern` inside a `DEFINE_func_*` + macro body, which becomes a genuine file-scope declaration of every TU that invokes the macro. + engine_core.h is 23,546 continuation lines inside 1,801 such macros, so this was not an edge case. + + The cost, measured: `func_801387B8` (a stub in 134 TUs) calls `func_80138DE0`, which its TU + declares — via a macro expansion — as `s32 (s32, s32, s32)`. The draft declares `s32 (s32)`. + The map returned NOTHING for that callee, so transform() took the + `if addr not in canon: continue # pure stub callee -> no conflict, leave it` + branch — on a premise that was simply false — and the draft died with `conflicting types`, + which reads downstream as an intrinsic compiler wall. cpp answers it exactly, in 54 ms. + """ c_path = src_file or os.path.join(REPO, f'src/{overlay}/{overlay}.c') - sigs = dict(_ght.collect_extern_sigs([ec, c_path])) - sigs.update(_ght.collect_define_sigs(ec)) - sigs.update(_ght.collect_inline_sigs(c_path)) + sigs = {} + for name, d in _cdecl.tu_scope(c_path).items(): + if d.kind != 'func' or not re.fullmatch(r'func_[0-9A-Fa-f]{8}', name): + continue + sigs[int(name[5:], 16)] = d.declaration(storage='').rstrip(';').strip() return sigs, c_path @@ -112,8 +150,33 @@ def split_sig_string(s): return m.group(1).strip(), m.group(2).strip() +_ABOVE = set() # names the TU declares ABOVE the splice point (set by main/gate_stage) + + +def _no_conflict(canon_sig, draft_line, fn): + """Will cc1 accept the TU's declaration of `fn` alongside the draft's? (cdecl.compatible, which + is validated against the REAL gcc-2.7.2 cc1 on 1,485 live corpus pairs — `cdecl.py --compat`.) + + This replaces `norm_sig(...) == norm_sig(...)`, which collapsed the int family (s32|u32|int| + unsigned|long) to ONE token and therefore called a SIGNEDNESS change "already compatible" and + emitted no rewrite — while cc1 rejects that redeclaration outright. It was right about codegen + (same width, same load) and wrong about the C FRONT END, which never reaches codegen. + + Order matters, and only cc1 could have told us so: a no-prototype decl followed by a + narrow-param prototype CONFLICTS, but the reverse order is ACCEPTED (§51g / the Phase-15 + narrow-param wall). So the TU's decl goes first iff it is declared above the splice point. + """ + try: + c = _cdecl.parse(f'extern {canon_sig};')[0] + d = _cdecl.parse(draft_line.strip())[0] + except (_cdecl.CDeclError, IndexError): + return False # unparseable -> be safe, reconcile it + a, b = (c, d) if fn in _ABOVE else (d, c) # TU order + return _cdecl.compatible(a, b) + + def transform(text, self_fn, canon): - """Return (new_text, n_callees_cast). For each callee whose canonical TU sig differs from the + """Return (new_text, n_callees_cast). For each callee whose canonical TU sig CONFLICTS with the draft's intended sig: rewrite the decl line to canonical + cast every call to the intended sig.""" lines = text.split('\n') @@ -136,8 +199,8 @@ def transform(text, self_fn, canon): if not csig: continue cret, cptypes = parse_sig(*csig) - if norm_sig(dret, dptypes) == norm_sig(cret, cptypes): - continue # already compatible -> no cast needed + if _no_conflict(canon[addr], ln, fn): + continue # cc1 accepts both -> no rewrite, no cast to_cast[fn] = cast_type(dret, dptypes) canon_decl[fn] = f'{indent}extern {canon[addr]};' decl_line_idx.setdefault(fn, set()).add(i) @@ -176,18 +239,30 @@ def main(): ap.add_argument('--out', dest='outdir', required=True) a = ap.parse_args() - canon, _c_path = canonical_map(a.overlay, a.src_file and os.path.join(REPO, a.src_file)) + override = a.src_file and os.path.join(REPO, a.src_file) os.makedirs(os.path.join(REPO, a.outdir), exist_ok=True) drafts = touched = total_callees = 0 + cache, ncanon = {}, 0 for p in sorted(glob.glob(os.path.join(REPO, a.indir, '*.c'))): fn = os.path.basename(p)[:-2] + # PER-DRAFT: canonicalize against the TU that will actually compile it (derived), not + # against whatever .c the caller happened to name. + tu = tu_for(a.overlay, fn, override) + if tu not in cache: + cache[tu] = canonical_map(a.overlay, tu)[0] + canon = cache[tu] + ncanon = max(ncanon, len(canon)) + # TU order for the no-prototype rule: which of the TU's decls precede this splice point? + # (cdecl caches the cpp run, so this is ~free per draft.) + _ABOVE.clear() + _ABOVE.update(_cdecl.tu_scope(tu, above=fn)) new, k = transform(open(p).read(), fn, canon) open(os.path.join(REPO, a.outdir, os.path.basename(p)), 'w').write(new) drafts += 1 if k: touched += 1 total_callees += k - print(f'canonical sigs: {len(canon)}; drafts: {drafts}; ' + print(f'canonical sigs: {ncanon} (per-TU, derived); TUs: {len(cache)}; drafts: {drafts}; ' f'cast-recovered: {touched} draft(s), {total_callees} callee(s)') diff --git a/tools/cdecl.py b/tools/cdecl.py index 1c94b4a30..a3653b064 100644 --- a/tools/cdecl.py +++ b/tools/cdecl.py @@ -733,10 +733,144 @@ def scope(statements, path=''): def tu_scope(tu_path, above=None): """{name: Declarator} the compiler sees in `tu_path` (optionally above a stub). THE oracle that - every 'what does this TU declare' question in this repo should be asking.""" + every 'what does this TU declare' question in this repo should be asking. + + ⚠️ `above` answers *"can I USE this symbol without declaring it"* — the visibility question, which + drives the block-scope-demotion branch. It is the WRONG question for *"will my declaration + CONFLICT"*: C requires every declaration of a name in a TU to be compatible **regardless of + order**, so a declaration BELOW the splice point conflicts just as hard as one above. Use the + full scope (`above=None`) for conflict detection. (`canon_sig_reconcile` drives both decisions + off one `visible` set — which is why `func_801387B8` hits `conflicting types` against a decl + 1,000 lines below its stub and the recovery pass reports nothing to fix.)""" return scope(tu_statements(tu_path, above), tu_path) +# --------------------------------------------------------------------------------------------- +# C TYPE COMPATIBILITY — the predicate four tools each half-implement, and get wrong +# --------------------------------------------------------------------------------------------- +# The question every recovery pass actually asks is "will gcc accept the draft's declaration +# alongside the TU's?" — and that is C's *compatible type* relation, which is NOT string equality: +# +# * `extern s32 D_x[];` IS compatible with `extern s32 D_x[4];` (incomplete vs complete array) +# * `extern s32 D_x;` is NOT compatible with `extern u32 D_x;` (signedness — gcc REJECTS it) +# +# Both existing implementations get this exactly backwards. `reconcile_decls._norm_type` collapses +# `s32|u32|int|unsigned|long` to ONE token, so it declares a signedness change "already compatible" +# and refuses to repair it — the compile then fails anyway, on a decl the tool looked at and +# approved. It is right about CODEGEN (same width, same load) and wrong about the C FRONT END, which +# rejects the redeclaration before codegen is ever reached. +# +# Validated against the real cross-gcc over every (TU, draft) declaration pair in the corpus: +# `tools/cdecl.py --audit --compat`. + +_PRIM = {'char': 'char', 'short': 'short', 'int': 'int', 'long': 'long', 'float': 'float', + 'double': 'double', 'void': 'void', 'signed': 'int', 'unsigned': 'unsigned int'} + + +@functools.lru_cache(1) +def _aliases(): + """typedef name -> underlying primitive, resolved transitively from the project's own prelude. + Derived by parsing common.h + engine_types.h with THIS parser — never a hand-kept table.""" + al = {} + for h in ('include/common.h', 'src/shared/engine_types.h'): + p = os.path.join(REPO, h) + if not os.path.exists(p): + continue + for st in split_statements(open(p).read()): + try: + for d in parse(st.text): + if d.storage == 'typedef' and not d.chain: + al[d.name] = d.base + except CDeclError: + pass + for _ in range(4): # resolve chains (u32 -> unsigned int -> …) + al = {k: al.get(v, v) for k, v in al.items()} + return al + + +def _prim(base): + """Resolve a base type to its underlying spelling, canonically ordered.""" + al = _aliases() + words = [al.get(w, w) for w in (base or '').split()] + words = ' '.join(words).split() + if any(w in TAGKW for w in words): + return ' '.join(words) # struct/union/enum: identity by tag + order = {'unsigned': 0, 'signed': 0, 'long': 1, 'short': 1, 'char': 2, 'int': 3, 'float': 2} + core = sorted((w for w in words if w not in QUALS), key=lambda w: order.get(w, 4)) + s = ' '.join(core) + return {'int unsigned': 'unsigned int', 'unsigned': 'unsigned int', 'signed': 'int', + 'signed int': 'int', 'long int': 'long', 'unsigned long int': 'unsigned long'}.get(s, s) + + +def compatible(a, b): + """CAN THESE TWO DECLARATIONS OF ONE NAME COEXIST IN ONE TU? — i.e. **will cc1 accept them.** + + Not "are these the same type", and NOT what the C standard says. This models **gcc-2.7.2's + actual behaviour**, because gcc-2.7.2 is what compiles this project, and it is measurably laxer + than both the standard and modern gcc. Every rule below was either confirmed or REFUTED by + running the real `cc1` over the live corpus (`tools/cdecl.py --compat`, 1,485 real pairs): + + * A TYPEDEF may not be redeclared at all — not even identically (`redefinition of 'X'`). Two + typedef declarations of one name can NEVER coexist, however equal their types. (Modern gcc + ALLOWS this — C11 relaxed it — which is exactly why the adjudicator must be cc1. This is + also why `_uniquify_draft_types` must strip-or-rename rather than compare.) + * QUALIFIERS DO NOT CONFLICT: cc1 accepts `extern u16 X;` beside `extern volatile u16 X;` + (modern gcc rejects it). REFUTED by the oracle; the rule was removed. + * THE NO-PROTOTYPE RULE IS **ORDER-DEPENDENT**, and that is the headline. Measured on cc1: + void X(s16); then void X(); -> ACCEPTS + void X(); then void X(s16); -> REJECTS (`conflicting types`) + void X(); then void X(s32)/X(void*) -> ACCEPTS (no default promotion) + i.e. a later PROTOTYPE must be compatible with the composite type the earlier `()` already + fixed (whose args are default-promoted) — but an earlier prototype simply wins. + ⚠️ **This is the wall Phase 15 wrote up as "the `()` no-prototype escape can never work"** — + the basis of the "159 arity/narrow-param conflicts — no clean deterministic fix" dead-end. + It is only true in ONE DIRECTION. Put the narrow-param prototype FIRST and cc1 accepts it. + Whether the resulting CODEGEN matches is a separate question the byte-gate answers — but the + wall's stated cause does not hold. **Re-test target for A10.** + + So `compatible(first, second)` takes them IN TU ORDER. Callers get the order from + `tu_scope(above=fn)`: a TU declaration above the splice point precedes the draft's; one below + follows it. + + Being permissive is also the SAFE direction: a decl cc1 accepts needs no rewrite, so the draft + keeps the types it intended (its `volatile`, its narrow params — all load-bearing for codegen), + and the whole-binary byte-gate remains the sole arbiter of the bytes (G3/P9). Every needless + rewrite is a perturbation that can only lose a match. + """ + if a is None or b is None: + return True # nothing to conflict with + if a.storage == 'typedef' or b.storage == 'typedef': + return False # C89/2.7.2 forbids redeclaring a typedef + ca, cb = [o[0] for o in a.chain], [o[0] for o in b.chain] + if ca != cb: + return False # array vs ptr vs fn: different types + if _prim(a.base) != _prim(b.base): + return False # includes the signedness case cc1 rejects + for x, y in zip(a.chain, b.chain): + if x[0] == ARR and x[1] and y[1] and x[1] != y[1]: + return False # T[4] vs T[8]; T[] vs T[4] IS compatible + if x[0] == FUN: + xt, yt, xe, ye = x[1], y[1], x[4], y[4] # `empty` == a no-prototype declaration + if xe and not ye: + # `()` FIRST, prototype SECOND: the prototype must be compatible with the composite + # type the `()` already fixed — so no parameter may be altered by default promotion. + return not any(_prim(p.rstrip('* ')) in _PROMOTES and '*' not in p for p in yt) + if ye: + continue # prototype first, `()` second: cc1 accepts + if len(xt) != len(yt): + return False # arity + if any(_prim(p.rstrip('* ')) != _prim(q.rstrip('* ')) or ('*' in p) != ('*' in q) + for p, q in zip(xt, yt)): + return False + return True + + +# The types C's default argument promotions alter. A later prototype naming one of these is +# incompatible with an earlier `()` — the one direction in which the "no-prototype escape" really +# does fail (measured on cc1, not assumed from the standard). +_PROMOTES = {'char', 'short', 'unsigned char', 'unsigned short', 'signed char', 'float'} + + # --------------------------------------------------------------------------------------------- # THE THREE ORACLES. The deliverable of this module is a MEASUREMENT, not a belief. # --------------------------------------------------------------------------------------------- @@ -937,6 +1071,95 @@ def audit_gcc(limit=None): return not bad +CC1 = os.path.join(REPO, 'tools/bin/gcc-2.7.2-psx/cc1') +CC1FLAGS = ['-quiet', '-O2', '-G0', '-mips1', '-mcpu=3000', '-mgas', '-msoft-float', '-fgnu-linker'] +_ERR = re.compile(r'error|redefinition|conflicting|incompatible|redeclar|parse error', re.I) + + +def _cc1_accepts(body): + """Does THE REAL BUILD FRONT END (gcc-2.7.2 `cc1`) accept this? — not modern gcc. + + They DISAGREE, and it matters: C11 permits redefining a typedef to the same type, C89/2.7.2 does + NOT (`redefinition of 'X'`). Validating a compatibility rule against a compiler that is not the + one compiling the code is *exactly* the failure this module exists to prevent — a recovery pass + that "approves" a declaration the real front end then refuses.""" + d = os.path.join(REPO, '.run/audit/cdecl') + os.makedirs(d, exist_ok=True) + c, i = os.path.join(d, 'cc1probe.c'), os.path.join(d, 'cc1probe.i') + open(c, 'w').write(body) + p = subprocess.run(CPP + ['-Isrc', c], capture_output=True, text=True, cwd=REPO) + if p.returncode: + return None # cannot adjudicate + open(i, 'w').write(p.stdout) + r = subprocess.run([CC1] + CC1FLAGS + [i, '-o', '/dev/null'], + capture_output=True, text=True, cwd=REPO) + return not (r.returncode or _ERR.search(r.stderr or '')) + + +def _synth_distinct(decls): + """Synthesize each unknown type as a DISTINCT struct. Emitting `typedef int Vec8;` would collapse + `extern Vec8 X;` and `extern int X;` into the same type and the probe would call them compatible + — the oracle destroying the very distinction it is meant to test.""" + known, out = _known_types(), set() + for d in decls: + for tok in re.findall(r'[A-Za-z_]\w*', (d.base or '') + ' ' + ' '.join(d.params or [])): + if tok not in known and tok not in ('void', 'struct', 'union', 'enum'): + out.add(tok) + return ''.join(f'typedef struct {{ int _{t}; }} {t};\n' for t in sorted(out)) + + +def audit_compat(limit=2500): + """ORACLE 4 — `compatible()` vs the REAL cc1, on every (TU-decl, draft-decl) pair the corpus + actually contains. My predicate says accept/reject; gcc-2.7.2 says accept/reject. Any + disagreement is MY bug, and it is the kind that silently caps the whole recovery pipeline.""" + tus = sorted(_glob.glob(os.path.join(REPO, 'src/ov_SC01_077/*.c'))) + scopes = {t: tu_scope(t) for t in tus} + pairs = {} + import random + for p in random.Random(11).sample(_drafts(), min(limit, len(_drafts()))): + try: + txt = open(p, errors='replace').read() + except Exception: + continue + for st in split_statements(txt): + try: + ds = parse(st.text) + except CDeclError: + continue + for d in ds: + if d.is_definition: + continue + for ns in scopes.values(): + tu = ns.get(d.name) + if tu is None: + continue + key = (tu.declaration(name='X'), d.declaration(name='X')) + pairs.setdefault(key, (compatible(tu, d), (tu, d))) + + print(f'[compat] {len(pairs)} distinct (TU-decl, draft-decl) pairs from the real corpus') + print(f'[compat] adjudicator: gcc-2.7.2 cc1 (THE BUILD FRONT END), not modern gcc') + bad, skip = [], 0 + for (tu_d, dr_d), (mine, (tu, d)) in pairs.items(): + body = _PROBE_HDR + _synth_distinct([tu, d]) + tu_d + '\n' + dr_d + '\n' + got = _cc1_accepts(body) + if got is None: + skip += 1 + elif got != mine: + bad.append(((tu_d, dr_d), mine, got)) + print(f'[compat] agree: {len(pairs) - len(bad) - skip} DISAGREE: {len(bad)} unadjudicable: {skip}') + for (k, mine, got) in bad[:10]: + print(f' TU : {k[0]}\n draft: {k[1]}') + print(f' cdecl says {"compatible" if mine else "CONFLICT"}, cc1 says ' + f'{"compatible" if got else "CONFLICT"}\n') + # An UNADJUDICABLE pair is a silent skip, and a gate that skips its whole corpus and prints GREEN + # is the exact bug this module was written to hunt. (It did that here, once: a missing `-Isrc` + # made cpp fail on all 1,485 probes, `bad` stayed empty, and the audit reported success. R32 is + # not "fail loud" — it is COUNT WHAT YOU SKIPPED.) + if skip: + print(f' !! {skip} pair(s) could not be adjudicated — a skipped check is NOT a passed one') + return not bad and not skip + + def audit_differential(): """ORACLE 3 — vs THE FIFTEEN INCUMBENTS. This parser must find a strict SUPERSET of every scanner it replaces: any symbol an incumbent sees and this one does not is a defect in THIS @@ -1018,6 +1241,8 @@ def main(): formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument('--audit', action='store_true', help='oracle 1 (coverage) + oracle 3 (differential)') ap.add_argument('--gcc', action='store_true', help='oracle 2: the C front end (slow, decisive)') + ap.add_argument('--compat', action='store_true', + help='oracle 4: compatible() vs the REAL gcc-2.7.2 cc1 on live corpus pairs') ap.add_argument('--limit', type=int, help='sample N TUs (a fast smoke run)') ap.add_argument('--tu', help='print the file-scope scope of a TU') ap.add_argument('--above', help='...truncated above this function\'s INCLUDE_ASM stub') @@ -1037,7 +1262,7 @@ def main(): print(f' {n:<24} {d.kind:<12} {d.type}') print(f'{len(ns)} file-scope names visible' + (f' above {a.above}' if a.above else '')) return - if a.audit or a.gcc: + if a.audit or a.gcc or a.compat: ok = True if a.audit: ok &= audit_coverage(a.limit, a.verbose) @@ -1046,6 +1271,9 @@ def main(): if a.gcc: print() ok &= audit_gcc(a.limit) + if a.compat: + print() + ok &= audit_compat() print('\n' + ('cdecl: ALL ORACLES GREEN' if ok else 'cdecl: DEFECTS FOUND (see above)')) sys.exit(0 if ok else 1) ap.print_help()