mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-30 15:36:55 -04:00
fix(phase-26a): A3c — the recovery passes were reconciling 95% of drafts against the WRONG TU
FIRST CONSUMER MIGRATION onto the cdecl oracle — and the compiler taught me two things I had
wrong, one of which reopens a wall that has been closed since Phase 15.
1. cdecl.compatible() — "will cc1 accept these two declarations of one name?"
The predicate four tools each half-implement and get wrong: norm_sig / _norm_type collapse the
int family to ONE token, so a SIGNEDNESS change reads as "already compatible" and gets no
rewrite -- while cc1 REJECTS that redeclaration. Right about codegen, wrong about the front end,
which never reaches codegen.
2. THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE (cookbook §51g LAW 9).
I wrote the rules from the C standard, then let a compiler judge. It contradicted me -- and then
the RIGHT compiler contradicted the first one. Three different answers:
declarations in one TU | standard | modern gcc | gcc-2.7.2 cc1
typedef int X; twice | error | ACCEPTS | ERROR
extern u16 X; + volatile u16 X| error | error | ACCEPTS
void X(s16); then void X(); | error | error | ACCEPTS
void X(); then void X(s16)| error | error | ERROR
--compat now adjudicates with tools/bin/gcc-2.7.2-psx/cc1, the front end that actually
arbitrates the build: 1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.
3. THE PRIZE: the Phase-15 narrow-param wall rests on a false premise.
The no-prototype rule is ORDER-DEPENDENT. `void X(s16); void X();` COMPILES; only the reverse
fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as "no clean deterministic fix --
it is simply C's default-promotion rule". cc1 does not enforce that rule in the direction the
wall assumed. Four three-line probes, 90 seconds, zero tokens. -> A10 RE-TEST TARGET.
Probe the compiler for FACTS; read its source only for LEVERS; byte-validate both. (We read
gcc-papermario for five phases believing it was 2.7.2. It was 2.8.1.)
4. THE MIGRATION: cast_call_sites canonicalized 95.1% of drafts against a TU that would never
compile them. `--src-file` is an OPTIONAL HAND-PASSED flag defaulting to src/<ov>/<ov>.c, and no
caller knows about the Phase-26 _jr_<ADDR> carves: ov_SC01_077 has 263 open stubs across 12 TUs
and only 13 are in the main .c -- while harvest_verify (A3) correctly splices into the real one.
Now DERIVED from corpus.stubs() (the INCLUDE_ASM line is self-describing), with the canonical map
derived from cdecl.tu_scope() (cpp -- so macro-injected DEFINE_func_* decls are finally visible).
Callee-conflict repair reach: 8 -> 58 of 196 drafts (7x).
5. AND THE NULL RESULT, REPORTED AS SUCH (P9/R14). Those 58 banked ZERO functions. The historical
draft tail fails on CODEGEN, not plumbing -- func_801387B8, which the audit blames on a single
unparsed `[4]`, is really 67/100 instructions off with a $s0/$s1 swap (that claim does not
reproduce on today's tree). The real gain is narrower and still worth having: 52 drafts moved
from "won't compile" to "compiles, N instructions off" -- from an INVISIBLE failure that reads as
a compiler wall into a SCORED near-miss the permuter and the §47/§48 dials can act on. That is
the audit's thesis, not a bank. THREE times in one session a confirmed mechanism produced a null
consequence.
Also: my own new audit printed "ALL ORACLES GREEN" while silently skipping 100% of its corpus (a
missing -Isrc). The exact bug class, in the tool written to hunt it. An unadjudicable check is not
a passed check.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
src/ untouched (0 changes) make audit-cdecl: green --compat: 1485/1485
NEXT: sig_unify + reconcile_decls carry the SAME wrong-TU bug (same --src-file flag).
This commit is contained in:
Drew T
@@ -959,3 +959,41 @@ signature from garbage.
|
||||
explicitly warns that *making the parser see more ARMS dormant downstream transforms* — the moment
|
||||
`reconcile_decls` can parse a fn-ptr decl, its `data_access_subs` would happily mangle `D_1[i]()` into
|
||||
`((u8 *)D_1)[i]()`. Consumer migration is therefore one tool at a time, each byte-gated.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-14 — Probe the compiler; and the adjudicator must BE the compiler
|
||||
|
||||
**Context.** Building `cdecl.compatible()` — *"will cc1 accept these two declarations of one name?"*, the
|
||||
question every recovery pass in this repo actually asks and four of them half-implement. I wrote the rules
|
||||
from the C standard, then validated them against a compiler.
|
||||
|
||||
**What happened.** The compiler contradicted me — and then the *right* compiler contradicted the first one.
|
||||
Validating against modern `mipsel-linux-gnu-gcc` and against the real gcc-2.7.2 `cc1` gives **three
|
||||
different answers** (with the standard as a third): typedef redefinition is an error in C89, accepted by
|
||||
C11 gcc, and rejected by cc1; a qualifier mismatch is an error to modern gcc and **accepted** by cc1; the
|
||||
no-prototype/narrow-param rule is an error to both — and **accepted by cc1 in one direction.**
|
||||
|
||||
**The decision.** `--compat` adjudicates with `tools/bin/gcc-2.7.2-psx/cc1`, the front end that actually
|
||||
arbitrates the build. Now 1,485/1,485 live corpus pairs agree. **Validating a compiler rule against a
|
||||
compiler that is not the one compiling your code is not a shortcut — it is the same class of error as the
|
||||
five phases we spent reading `gcc-papermario` believing it was 2.7.2. It was 2.8.1.**
|
||||
|
||||
**The prize (→ A10).** Phase 15 closed the "159 arity/narrow-param conflicts" as *"no clean deterministic
|
||||
fix — it is simply C's default-promotion rule."* **cc1 disagrees.** The rule is order-dependent:
|
||||
`void X(s16); void X();` compiles; only `void X(); void X(s16);` fails. The wall's stated cause does not
|
||||
hold. Four three-line probes, 90 seconds, zero tokens.
|
||||
|
||||
**Hindsight / for the wiki.** *Probe the compiler for FACTS; read its source only for LEVERS; byte-validate
|
||||
both.* Reading source is inference and can be wrong (it was, for five phases). Probing is ground truth,
|
||||
because it IS the compiler — and it is orders of magnitude cheaper. We have the exact binary sitting in the
|
||||
tree and spent 26 phases reasoning about it instead of asking it.
|
||||
|
||||
**And the discipline that saved this from being an over-claim.** Fixing the wrong-TU bug (95.1% of drafts
|
||||
canonicalized against a TU that would never compile them) took the callee-conflict repair from 8 to 58 of
|
||||
196 drafts — 7× reach — and banked **exactly zero** functions, because the historical tail fails on codegen,
|
||||
not plumbing. The real gain is narrower and still worth having: **52 drafts moved from "won't compile" to
|
||||
"compiles, N instructions off"** — from an invisible failure that reads as a compiler wall into a scored
|
||||
near-miss the permuter can act on. Three times in one session a confirmed mechanism produced a null
|
||||
consequence. *"This tool is broken" and "this number will move" are different claims, needing different
|
||||
evidence.*
|
||||
|
||||
@@ -3681,3 +3681,46 @@ a compiler wall.
|
||||
|
||||
**Gate:** `make audit-cdecl` (coverage + gcc). Standing, because a loud failure nobody counts is exactly
|
||||
as invisible as a silent one (LAW 2).
|
||||
|
||||
**LAW 9 — THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE.** Not the C standard, and not
|
||||
whatever `gcc` is on the PATH. Building `cdecl.compatible()` (*"will this declaration coexist with that
|
||||
one?"* — the question every recovery pass actually asks) against **modern** `mipsel-linux-gnu-gcc` and
|
||||
against the **real gcc-2.7.2 `cc1`** gives three different answers, and only one of them is the truth:
|
||||
|
||||
| declarations in one TU | C standard | modern gcc (C11) | **gcc-2.7.2 `cc1`** |
|
||||
|---|---|---|---|
|
||||
| `typedef int X;` twice | error | **accepts** | **ERROR** — `redefinition of 'X'` |
|
||||
| `extern u16 X;` + `extern volatile u16 X;` | error | error | **ACCEPTS** |
|
||||
| `void X(s16);` then `void X();` | error | error | **ACCEPTS** |
|
||||
| `void X();` then `void X(s16);` | error | error | **ERROR** |
|
||||
|
||||
Validate against modern gcc and you encode rules cc1 rejects and miss rules cc1 accepts — a recovery pass
|
||||
that "approves" declarations the real front end refuses is exactly the failure this whole audit is about.
|
||||
`cdecl --compat` therefore adjudicates with `tools/bin/gcc-2.7.2-psx/cc1`, and now agrees with it on
|
||||
**1,485/1,485 live corpus pairs**. Two rules in that table were *refuted* by the oracle after I had
|
||||
already written them from the standard.
|
||||
|
||||
> 🏆 **AND THE LAST ROW IS A WALL COMING DOWN.** The no-prototype rule is **ORDER-DEPENDENT**: only
|
||||
> `()`-then-narrow-prototype fails; **prototype-then-`()` compiles fine.** Phase 15 wrote this class up as
|
||||
> *"the 159 arity/narrow-param conflicts — no clean deterministic fix"* and closed it. **The stated cause
|
||||
> does not hold.** Whether the resulting codegen matches is a separate question the byte-gate answers —
|
||||
> but the door was never locked. It took four three-line probes and 90 seconds to find out. **Probe the
|
||||
> compiler for FACTS; read its source only for LEVERS; byte-validate both** (we read `gcc-papermario` for
|
||||
> five phases believing it was 2.7.2 — it was 2.8.1).
|
||||
|
||||
**LAW 10 — DERIVE *WHICH* TU, TOO — not just what is in it.** `cast_call_sites` / `sig_unify` /
|
||||
`reconcile_decls` take `--src-file`, an **optional, hand-passed** flag naming the TU to canonicalize
|
||||
against; unset, it defaults to `src/<ov>/<ov>.c`. No caller knows about the Phase-26 `_jr_<ADDR>` carves.
|
||||
Measured on ov_SC01_077: **263 open stubs across 12 TUs — only 13 in the main `.c`. 95.1% of drafts were
|
||||
being reconciled against a translation unit that would never compile them**, while `harvest_verify`
|
||||
(fixed in A3) correctly spliced them into the right one. `corpus.stubs()` already knows the answer — the
|
||||
`INCLUDE_ASM` line is self-describing. **Ask, don't assume.** Fixing it took the callee-conflict repair
|
||||
from **8 → 58 of 196** drafts (7× reach).
|
||||
|
||||
> **AND THE HONEST OTHER HALF (P9/R14):** those 58 produced **ZERO new banks.** The historical draft tail
|
||||
> fails on *codegen*, not plumbing — `func_801387B8`, which the audit blamed on a single unparsed `[4]`,
|
||||
> is really 67/100 instructions off with a `$s0`/`$s1` swap. What the fix *did* buy is real but narrower:
|
||||
> **52 drafts moved from "won't compile" to "compiles, N instructions off."** That is not a bank — it is
|
||||
> the difference between an **invisible failure that reads as a compiler wall** and a **scored near-miss
|
||||
> the permuter and the §47/§48 dials can act on.** Which is the audit's thesis exactly. Do not sell it as
|
||||
> more than it is; three times in one session a confirmed mechanism produced a null consequence.
|
||||
|
||||
@@ -726,6 +726,17 @@ scanners** — the "best outcome is a deleted scanner" rule (R33), applied at sc
|
||||
3. **A recovery tool once wrote non-C into drafts:** `extern if ((func_80029178(0x119) & 0xFF) != 0);` appears in 33 drafts, *only* in `-canon`/`-recanon`/`-uni`/`-sigfix` output dirs (the pre-recovery draft has none), and gcc rejects it outright. **R14 blast-radius check: the source bug was already fixed in Phase 19** — today's oracle emits **0 garbage over 300 signatures** — so this is dead historical residue, not a live defect. *Mechanism confirmed, consequence nil.* But note what it cost at the time: a draft that cannot compile fails the byte-gate and reads, downstream, as **an intrinsic compiler wall**.
|
||||
4. **`tu_ambient`'s function regex drops any callee with a fn-ptr parameter.** Its param class is `[^()]*`, so `extern void func_8012A568(void (*a0)(void));` — a real declaration in ov_SC01_077 — lands in **no bucket at all**: not funcs, not data, not typedefs.
|
||||
|
||||
### A3c — the first consumer migration (`cast_call_sites`), and what it measured
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **[CRITICAL] the recovery passes canonicalize against the WRONG TU** | `--src-file` is an **optional hand-passed flag**; unset it defaults to `src/<ov>/<ov>.c`, and no caller knows about the Phase-26 `_jr_<ADDR>` carves. Measured: ov_SC01_077 has **263 open stubs across 12 TUs — only 13 in the main `.c`**. So **95.1% of drafts were reconciled against a translation unit that would never compile them**, while `harvest_verify` (A3) correctly spliced them into the right one. Fixed by DERIVING the TU from `corpus.stubs()` (the `INCLUDE_ASM` line is self-describing). **Repair reach 8 → 58 of 196 drafts (7×).** |
|
||||
| **[HIGH] the conflict predicate was wrong in both directions** | `norm_sig`/`_norm_type` collapse the int family to one token, so a **signedness** change reads as "already compatible" and gets no rewrite — while cc1 **rejects** that redeclaration. Replaced by `cdecl.compatible()`, validated against the real cc1 on **1,485/1,485** live corpus pairs. |
|
||||
| **🏆 [FINDING] the Phase-15 narrow-param wall rests on a false premise** | The `()` no-prototype rule is **ORDER-DEPENDENT** on cc1: `void X(s16); void X();` **compiles**; only the reverse order fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as *"no clean deterministic fix"*. **The stated cause does not hold.** → **A10 re-test target.** |
|
||||
| **[FINDING] cc1 ≠ modern gcc ≠ the C standard** | They give three different answers on typedef redefinition, qualifier mismatch, and the no-proto rule. Any compiler rule validated against modern gcc is validated against the wrong compiler. See cookbook §51g LAW 9 for the table. |
|
||||
|
||||
**AND THE NULL RESULT, RECORDED HONESTLY (P9/R14).** Those 58 repaired drafts banked **ZERO** functions. The historical draft tail fails on **codegen**, not plumbing — `func_801387B8`, which finding F1 blames on a single unparsed `[4]`, is really **67/100 instructions off with a `$s0`/`$s1` swap** (that F1 claim does not reproduce on today's tree). What the fix genuinely buys: **52 drafts moved from "won't compile" to "compiles, N instructions off"** — from an *invisible failure that reads as a compiler wall* to a *scored near-miss the permuter can act on*. Real, but narrower than it first looked. **Three separate times in one session a confirmed mechanism produced a null consequence** — verify the blast radius, not just the defect.
|
||||
|
||||
## Still open (round-2 findings not yet fixed)
|
||||
|
||||
`masked_diff`/`match_one` (**155 provably-wrong closeness scores** — feeds false walls) · `harvest_verify._TD`
|
||||
|
||||
@@ -659,6 +659,38 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag
|
||||
|
||||
## Log
|
||||
|
||||
- **2026-07-14 (session 10, A3c — first consumer migration: `cast_call_sites` onto the cdecl oracle; Max):**
|
||||
Added **`cdecl.compatible()`** — *"will cc1 accept these two declarations of one name?"*, the predicate four
|
||||
tools each half-implement and get wrong (`norm_sig`/`_norm_type` collapse the int family, so a **signedness**
|
||||
change reads as "already compatible" and gets no rewrite — while cc1 **rejects** that redeclaration). Wrote the
|
||||
rules from the C standard; then let a compiler judge. **It contradicted me — and then the RIGHT compiler
|
||||
contradicted the first one.** Modern `mipsel-linux-gnu-gcc`, gcc-2.7.2 `cc1`, and the standard give **three
|
||||
different answers** (typedef redefinition: C89 error / C11 accepts / **cc1 ERRORS**; qualifier mismatch: modern
|
||||
gcc errors / **cc1 ACCEPTS**; no-proto + narrow param: both error / **cc1 accepts in ONE direction**). So
|
||||
`--compat` now adjudicates with **`tools/bin/gcc-2.7.2-psx/cc1`, the front end that actually arbitrates the
|
||||
build** → **1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.** 🏆 **THE PRIZE: the Phase-15
|
||||
narrow-param wall rests on a false premise.** The `()` rule is **ORDER-DEPENDENT**: `void X(s16); void X();`
|
||||
**compiles**; only the reverse fails. "The 159 arity/narrow-param conflicts — no clean deterministic fix" was
|
||||
closed on a rule cc1 does not enforce. Four three-line probes, 90 s, zero tokens → **A10 re-test target**.
|
||||
**THEN the migration itself: `cast_call_sites` was canonicalizing 95.1% of drafts against the WRONG TU.**
|
||||
`--src-file` is an *optional hand-passed flag* defaulting to `src/<ov>/<ov>.c`, and no caller knows about the
|
||||
Phase-26 `_jr_<ADDR>` carves — ov_SC01_077 has **263 open stubs across 12 TUs, only 13 in the main `.c`** —
|
||||
while `harvest_verify` (A3) correctly splices into the real one. Now **DERIVED** from `corpus.stubs()` (the
|
||||
`INCLUDE_ASM` line is self-describing) + the canonical map derived from `cdecl.tu_scope()` (cpp — so
|
||||
macro-injected decls are finally visible). **Repair reach 8 → 58 of 196 drafts (7×).**
|
||||
**THE NULL RESULT, REPORTED AS SUCH (P9/R14):** those 58 banked **ZERO**. The historical tail fails on
|
||||
**codegen**, not plumbing — `func_801387B8`, which the audit blames on one unparsed `[4]`, is really **67/100
|
||||
instructions off with a `$s0`/`$s1` swap** (that claim does not reproduce). The real gain is narrower and still
|
||||
worth having: **52 drafts moved from "won't compile" to "compiles, N instructions off"** — from an *invisible
|
||||
failure that reads as a compiler wall* into a *scored near-miss the permuter can act on*. **Three times in one
|
||||
session a confirmed mechanism produced a null consequence.** I also mis-diagnosed the callee oracle as
|
||||
"returning nothing" (my probe was buggy — it did have the sig) — corrected. **And my own new audit printed
|
||||
"ALL ORACLES GREEN" while silently skipping 100% of its corpus** (a missing `-Isrc`): the exact bug class, in
|
||||
the tool written to hunt it. Fixed — *an unadjudicable check is not a passed check.*
|
||||
**R22 clean-fleet 136/136 BYTE-IDENTICAL; src/ untouched (0 changes); `make audit-cdecl` green.** Knowledge
|
||||
captured live (R30/R31): cookbook **§51g LAWS 9–10**, decision-log, tooling-audit **A3c**.
|
||||
**NEXT: `sig_unify` + `reconcile_decls` have the SAME wrong-TU bug** (same `--src-file` flag) — migrate them,
|
||||
then `lint_symbol_refs`, then **A10 (re-test the walls)** with the narrow-param finding as the first target.
|
||||
- **2026-07-14 (session 10, A3b — `tools/cdecl.py`, THE C-declaration oracle; Max):** Built the one parser
|
||||
that lets fifteen die. **Rejected the audit's own prescription** (a shape-aware alternation per tool, ~15
|
||||
coordinated regex edits) on R33 grounds: fifteen hand-maintained models are precisely what diverged, and an
|
||||
|
||||
+89
-14
@@ -51,6 +51,30 @@ def _load(mod, rel):
|
||||
|
||||
_ght = _load('ght', 'tools/gen_harvest_targets.py')
|
||||
_su = _load('su', 'tools/sig_unify.py') # reuse split_top_commas / param_type
|
||||
_cdecl = _load('cdecl', 'tools/cdecl.py') # THE declaration oracle (Phase 26-A, §51g)
|
||||
_corpus = _load('corpus', 'tools/corpus.py') # THE corpus oracle — which TU holds a stub
|
||||
|
||||
|
||||
def tu_for(overlay, fn, override=None):
|
||||
"""The TU this draft will actually be SPLICED INTO — DERIVED from the corpus, not hand-passed.
|
||||
|
||||
`--src-file` was an OPTIONAL flag defaulting to `src/<ov>/<ov>.c`. Every caller that did not
|
||||
know to set it — and none of them know about the Phase-26 `_jr_<ADDR>` carve files — was
|
||||
therefore canonicalizing drafts against the MAIN .c while `harvest_verify` (fixed in A3)
|
||||
correctly spliced them into the jr split. The recovery passes were reconciling against a
|
||||
DIFFERENT TRANSLATION UNIT than the one that would compile the code, and the heavy Phase-26
|
||||
cores live in exactly those jr files.
|
||||
|
||||
The INCLUDE_ASM line is self-describing, and corpus.stubs() reads it. Ask, don't assume."""
|
||||
if override:
|
||||
return override
|
||||
try:
|
||||
st = _corpus.stubs(overlay).get(int(fn[5:], 16))
|
||||
if st:
|
||||
return os.path.join(REPO, st.path)
|
||||
except Exception:
|
||||
pass
|
||||
return os.path.join(REPO, f'src/{overlay}/{overlay}.c')
|
||||
|
||||
# A function declaration line (prototype ending in `;`, NOT a definition): optional `extern`,
|
||||
# a type, func_X, a param list, `;`, optional trailing /* comment */. Matches both the
|
||||
@@ -91,16 +115,30 @@ def cast_type(dret, dptypes):
|
||||
|
||||
|
||||
def canonical_map(overlay, src_file=None):
|
||||
"""addr-int -> canonical sig string '<ret> func_X(<params>)' for every func the TU declares
|
||||
or defines (definitions/inline win over plain externs — the authoritative in-TU signature).
|
||||
Mirrors sig_unify's precedence exactly. src_file overrides the .c (use the SPLIT file _a.c/_o0.c
|
||||
for a draft that lands there — its TU sees that file's local decls, NOT the main .c's, so a
|
||||
cross-file loose-typed callee (e.g. RotTransSV declared differently in main vs _a) resolves right)."""
|
||||
ec = os.path.join(REPO, 'src/shared/engine_core.h')
|
||||
"""addr-int -> canonical sig string '<ret> func_X(<params>)' — DERIVED from what cc1 actually
|
||||
sees in the TU (`cdecl.tu_scope`, i.e. cpp), not scraped out of the raw text.
|
||||
|
||||
WHY THIS CHANGED (Phase 26-A, R33; cookbook §51g LAW 7)
|
||||
------------------------------------------------------
|
||||
It used to union three raw-text scanners:
|
||||
collect_extern_sigs([engine_core.h, the .c]) + collect_define_sigs(ec) + collect_inline_sigs(c)
|
||||
and NONE of them can see a MACRO-INJECTED declaration — an `extern` inside a `DEFINE_func_*`
|
||||
macro body, which becomes a genuine file-scope declaration of every TU that invokes the macro.
|
||||
engine_core.h is 23,546 continuation lines inside 1,801 such macros, so this was not an edge case.
|
||||
|
||||
The cost, measured: `func_801387B8` (a stub in 134 TUs) calls `func_80138DE0`, which its TU
|
||||
declares — via a macro expansion — as `s32 (s32, s32, s32)`. The draft declares `s32 (s32)`.
|
||||
The map returned NOTHING for that callee, so transform() took the
|
||||
`if addr not in canon: continue # pure stub callee -> no conflict, leave it`
|
||||
branch — on a premise that was simply false — and the draft died with `conflicting types`,
|
||||
which reads downstream as an intrinsic compiler wall. cpp answers it exactly, in 54 ms.
|
||||
"""
|
||||
c_path = src_file or os.path.join(REPO, f'src/{overlay}/{overlay}.c')
|
||||
sigs = dict(_ght.collect_extern_sigs([ec, c_path]))
|
||||
sigs.update(_ght.collect_define_sigs(ec))
|
||||
sigs.update(_ght.collect_inline_sigs(c_path))
|
||||
sigs = {}
|
||||
for name, d in _cdecl.tu_scope(c_path).items():
|
||||
if d.kind != 'func' or not re.fullmatch(r'func_[0-9A-Fa-f]{8}', name):
|
||||
continue
|
||||
sigs[int(name[5:], 16)] = d.declaration(storage='').rstrip(';').strip()
|
||||
return sigs, c_path
|
||||
|
||||
|
||||
@@ -112,8 +150,33 @@ def split_sig_string(s):
|
||||
return m.group(1).strip(), m.group(2).strip()
|
||||
|
||||
|
||||
_ABOVE = set() # names the TU declares ABOVE the splice point (set by main/gate_stage)
|
||||
|
||||
|
||||
def _no_conflict(canon_sig, draft_line, fn):
|
||||
"""Will cc1 accept the TU's declaration of `fn` alongside the draft's? (cdecl.compatible, which
|
||||
is validated against the REAL gcc-2.7.2 cc1 on 1,485 live corpus pairs — `cdecl.py --compat`.)
|
||||
|
||||
This replaces `norm_sig(...) == norm_sig(...)`, which collapsed the int family (s32|u32|int|
|
||||
unsigned|long) to ONE token and therefore called a SIGNEDNESS change "already compatible" and
|
||||
emitted no rewrite — while cc1 rejects that redeclaration outright. It was right about codegen
|
||||
(same width, same load) and wrong about the C FRONT END, which never reaches codegen.
|
||||
|
||||
Order matters, and only cc1 could have told us so: a no-prototype decl followed by a
|
||||
narrow-param prototype CONFLICTS, but the reverse order is ACCEPTED (§51g / the Phase-15
|
||||
narrow-param wall). So the TU's decl goes first iff it is declared above the splice point.
|
||||
"""
|
||||
try:
|
||||
c = _cdecl.parse(f'extern {canon_sig};')[0]
|
||||
d = _cdecl.parse(draft_line.strip())[0]
|
||||
except (_cdecl.CDeclError, IndexError):
|
||||
return False # unparseable -> be safe, reconcile it
|
||||
a, b = (c, d) if fn in _ABOVE else (d, c) # TU order
|
||||
return _cdecl.compatible(a, b)
|
||||
|
||||
|
||||
def transform(text, self_fn, canon):
|
||||
"""Return (new_text, n_callees_cast). For each callee whose canonical TU sig differs from the
|
||||
"""Return (new_text, n_callees_cast). For each callee whose canonical TU sig CONFLICTS with the
|
||||
draft's intended sig: rewrite the decl line to canonical + cast every call to the intended sig."""
|
||||
lines = text.split('\n')
|
||||
|
||||
@@ -136,8 +199,8 @@ def transform(text, self_fn, canon):
|
||||
if not csig:
|
||||
continue
|
||||
cret, cptypes = parse_sig(*csig)
|
||||
if norm_sig(dret, dptypes) == norm_sig(cret, cptypes):
|
||||
continue # already compatible -> no cast needed
|
||||
if _no_conflict(canon[addr], ln, fn):
|
||||
continue # cc1 accepts both -> no rewrite, no cast
|
||||
to_cast[fn] = cast_type(dret, dptypes)
|
||||
canon_decl[fn] = f'{indent}extern {canon[addr]};'
|
||||
decl_line_idx.setdefault(fn, set()).add(i)
|
||||
@@ -176,18 +239,30 @@ def main():
|
||||
ap.add_argument('--out', dest='outdir', required=True)
|
||||
a = ap.parse_args()
|
||||
|
||||
canon, _c_path = canonical_map(a.overlay, a.src_file and os.path.join(REPO, a.src_file))
|
||||
override = a.src_file and os.path.join(REPO, a.src_file)
|
||||
os.makedirs(os.path.join(REPO, a.outdir), exist_ok=True)
|
||||
drafts = touched = total_callees = 0
|
||||
cache, ncanon = {}, 0
|
||||
for p in sorted(glob.glob(os.path.join(REPO, a.indir, '*.c'))):
|
||||
fn = os.path.basename(p)[:-2]
|
||||
# PER-DRAFT: canonicalize against the TU that will actually compile it (derived), not
|
||||
# against whatever .c the caller happened to name.
|
||||
tu = tu_for(a.overlay, fn, override)
|
||||
if tu not in cache:
|
||||
cache[tu] = canonical_map(a.overlay, tu)[0]
|
||||
canon = cache[tu]
|
||||
ncanon = max(ncanon, len(canon))
|
||||
# TU order for the no-prototype rule: which of the TU's decls precede this splice point?
|
||||
# (cdecl caches the cpp run, so this is ~free per draft.)
|
||||
_ABOVE.clear()
|
||||
_ABOVE.update(_cdecl.tu_scope(tu, above=fn))
|
||||
new, k = transform(open(p).read(), fn, canon)
|
||||
open(os.path.join(REPO, a.outdir, os.path.basename(p)), 'w').write(new)
|
||||
drafts += 1
|
||||
if k:
|
||||
touched += 1
|
||||
total_callees += k
|
||||
print(f'canonical sigs: {len(canon)}; drafts: {drafts}; '
|
||||
print(f'canonical sigs: {ncanon} (per-TU, derived); TUs: {len(cache)}; drafts: {drafts}; '
|
||||
f'cast-recovered: {touched} draft(s), {total_callees} callee(s)')
|
||||
|
||||
|
||||
|
||||
+230
-2
@@ -733,10 +733,144 @@ def scope(statements, path=''):
|
||||
|
||||
def tu_scope(tu_path, above=None):
|
||||
"""{name: Declarator} the compiler sees in `tu_path` (optionally above a stub). THE oracle that
|
||||
every 'what does this TU declare' question in this repo should be asking."""
|
||||
every 'what does this TU declare' question in this repo should be asking.
|
||||
|
||||
⚠️ `above` answers *"can I USE this symbol without declaring it"* — the visibility question, which
|
||||
drives the block-scope-demotion branch. It is the WRONG question for *"will my declaration
|
||||
CONFLICT"*: C requires every declaration of a name in a TU to be compatible **regardless of
|
||||
order**, so a declaration BELOW the splice point conflicts just as hard as one above. Use the
|
||||
full scope (`above=None`) for conflict detection. (`canon_sig_reconcile` drives both decisions
|
||||
off one `visible` set — which is why `func_801387B8` hits `conflicting types` against a decl
|
||||
1,000 lines below its stub and the recovery pass reports nothing to fix.)"""
|
||||
return scope(tu_statements(tu_path, above), tu_path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# C TYPE COMPATIBILITY — the predicate four tools each half-implement, and get wrong
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# The question every recovery pass actually asks is "will gcc accept the draft's declaration
|
||||
# alongside the TU's?" — and that is C's *compatible type* relation, which is NOT string equality:
|
||||
#
|
||||
# * `extern s32 D_x[];` IS compatible with `extern s32 D_x[4];` (incomplete vs complete array)
|
||||
# * `extern s32 D_x;` is NOT compatible with `extern u32 D_x;` (signedness — gcc REJECTS it)
|
||||
#
|
||||
# Both existing implementations get this exactly backwards. `reconcile_decls._norm_type` collapses
|
||||
# `s32|u32|int|unsigned|long` to ONE token, so it declares a signedness change "already compatible"
|
||||
# and refuses to repair it — the compile then fails anyway, on a decl the tool looked at and
|
||||
# approved. It is right about CODEGEN (same width, same load) and wrong about the C FRONT END, which
|
||||
# rejects the redeclaration before codegen is ever reached.
|
||||
#
|
||||
# Validated against the real cross-gcc over every (TU, draft) declaration pair in the corpus:
|
||||
# `tools/cdecl.py --audit --compat`.
|
||||
|
||||
_PRIM = {'char': 'char', 'short': 'short', 'int': 'int', 'long': 'long', 'float': 'float',
|
||||
'double': 'double', 'void': 'void', 'signed': 'int', 'unsigned': 'unsigned int'}
|
||||
|
||||
|
||||
@functools.lru_cache(1)
|
||||
def _aliases():
|
||||
"""typedef name -> underlying primitive, resolved transitively from the project's own prelude.
|
||||
Derived by parsing common.h + engine_types.h with THIS parser — never a hand-kept table."""
|
||||
al = {}
|
||||
for h in ('include/common.h', 'src/shared/engine_types.h'):
|
||||
p = os.path.join(REPO, h)
|
||||
if not os.path.exists(p):
|
||||
continue
|
||||
for st in split_statements(open(p).read()):
|
||||
try:
|
||||
for d in parse(st.text):
|
||||
if d.storage == 'typedef' and not d.chain:
|
||||
al[d.name] = d.base
|
||||
except CDeclError:
|
||||
pass
|
||||
for _ in range(4): # resolve chains (u32 -> unsigned int -> …)
|
||||
al = {k: al.get(v, v) for k, v in al.items()}
|
||||
return al
|
||||
|
||||
|
||||
def _prim(base):
|
||||
"""Resolve a base type to its underlying spelling, canonically ordered."""
|
||||
al = _aliases()
|
||||
words = [al.get(w, w) for w in (base or '').split()]
|
||||
words = ' '.join(words).split()
|
||||
if any(w in TAGKW for w in words):
|
||||
return ' '.join(words) # struct/union/enum: identity by tag
|
||||
order = {'unsigned': 0, 'signed': 0, 'long': 1, 'short': 1, 'char': 2, 'int': 3, 'float': 2}
|
||||
core = sorted((w for w in words if w not in QUALS), key=lambda w: order.get(w, 4))
|
||||
s = ' '.join(core)
|
||||
return {'int unsigned': 'unsigned int', 'unsigned': 'unsigned int', 'signed': 'int',
|
||||
'signed int': 'int', 'long int': 'long', 'unsigned long int': 'unsigned long'}.get(s, s)
|
||||
|
||||
|
||||
def compatible(a, b):
|
||||
"""CAN THESE TWO DECLARATIONS OF ONE NAME COEXIST IN ONE TU? — i.e. **will cc1 accept them.**
|
||||
|
||||
Not "are these the same type", and NOT what the C standard says. This models **gcc-2.7.2's
|
||||
actual behaviour**, because gcc-2.7.2 is what compiles this project, and it is measurably laxer
|
||||
than both the standard and modern gcc. Every rule below was either confirmed or REFUTED by
|
||||
running the real `cc1` over the live corpus (`tools/cdecl.py --compat`, 1,485 real pairs):
|
||||
|
||||
* A TYPEDEF may not be redeclared at all — not even identically (`redefinition of 'X'`). Two
|
||||
typedef declarations of one name can NEVER coexist, however equal their types. (Modern gcc
|
||||
ALLOWS this — C11 relaxed it — which is exactly why the adjudicator must be cc1. This is
|
||||
also why `_uniquify_draft_types` must strip-or-rename rather than compare.)
|
||||
* QUALIFIERS DO NOT CONFLICT: cc1 accepts `extern u16 X;` beside `extern volatile u16 X;`
|
||||
(modern gcc rejects it). REFUTED by the oracle; the rule was removed.
|
||||
* THE NO-PROTOTYPE RULE IS **ORDER-DEPENDENT**, and that is the headline. Measured on cc1:
|
||||
void X(s16); then void X(); -> ACCEPTS
|
||||
void X(); then void X(s16); -> REJECTS (`conflicting types`)
|
||||
void X(); then void X(s32)/X(void*) -> ACCEPTS (no default promotion)
|
||||
i.e. a later PROTOTYPE must be compatible with the composite type the earlier `()` already
|
||||
fixed (whose args are default-promoted) — but an earlier prototype simply wins.
|
||||
⚠️ **This is the wall Phase 15 wrote up as "the `()` no-prototype escape can never work"** —
|
||||
the basis of the "159 arity/narrow-param conflicts — no clean deterministic fix" dead-end.
|
||||
It is only true in ONE DIRECTION. Put the narrow-param prototype FIRST and cc1 accepts it.
|
||||
Whether the resulting CODEGEN matches is a separate question the byte-gate answers — but the
|
||||
wall's stated cause does not hold. **Re-test target for A10.**
|
||||
|
||||
So `compatible(first, second)` takes them IN TU ORDER. Callers get the order from
|
||||
`tu_scope(above=fn)`: a TU declaration above the splice point precedes the draft's; one below
|
||||
follows it.
|
||||
|
||||
Being permissive is also the SAFE direction: a decl cc1 accepts needs no rewrite, so the draft
|
||||
keeps the types it intended (its `volatile`, its narrow params — all load-bearing for codegen),
|
||||
and the whole-binary byte-gate remains the sole arbiter of the bytes (G3/P9). Every needless
|
||||
rewrite is a perturbation that can only lose a match.
|
||||
"""
|
||||
if a is None or b is None:
|
||||
return True # nothing to conflict with
|
||||
if a.storage == 'typedef' or b.storage == 'typedef':
|
||||
return False # C89/2.7.2 forbids redeclaring a typedef
|
||||
ca, cb = [o[0] for o in a.chain], [o[0] for o in b.chain]
|
||||
if ca != cb:
|
||||
return False # array vs ptr vs fn: different types
|
||||
if _prim(a.base) != _prim(b.base):
|
||||
return False # includes the signedness case cc1 rejects
|
||||
for x, y in zip(a.chain, b.chain):
|
||||
if x[0] == ARR and x[1] and y[1] and x[1] != y[1]:
|
||||
return False # T[4] vs T[8]; T[] vs T[4] IS compatible
|
||||
if x[0] == FUN:
|
||||
xt, yt, xe, ye = x[1], y[1], x[4], y[4] # `empty` == a no-prototype declaration
|
||||
if xe and not ye:
|
||||
# `()` FIRST, prototype SECOND: the prototype must be compatible with the composite
|
||||
# type the `()` already fixed — so no parameter may be altered by default promotion.
|
||||
return not any(_prim(p.rstrip('* ')) in _PROMOTES and '*' not in p for p in yt)
|
||||
if ye:
|
||||
continue # prototype first, `()` second: cc1 accepts
|
||||
if len(xt) != len(yt):
|
||||
return False # arity
|
||||
if any(_prim(p.rstrip('* ')) != _prim(q.rstrip('* ')) or ('*' in p) != ('*' in q)
|
||||
for p, q in zip(xt, yt)):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
# The types C's default argument promotions alter. A later prototype naming one of these is
|
||||
# incompatible with an earlier `()` — the one direction in which the "no-prototype escape" really
|
||||
# does fail (measured on cc1, not assumed from the standard).
|
||||
_PROMOTES = {'char', 'short', 'unsigned char', 'unsigned short', 'signed char', 'float'}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE THREE ORACLES. The deliverable of this module is a MEASUREMENT, not a belief.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
@@ -937,6 +1071,95 @@ def audit_gcc(limit=None):
|
||||
return not bad
|
||||
|
||||
|
||||
CC1 = os.path.join(REPO, 'tools/bin/gcc-2.7.2-psx/cc1')
|
||||
CC1FLAGS = ['-quiet', '-O2', '-G0', '-mips1', '-mcpu=3000', '-mgas', '-msoft-float', '-fgnu-linker']
|
||||
_ERR = re.compile(r'error|redefinition|conflicting|incompatible|redeclar|parse error', re.I)
|
||||
|
||||
|
||||
def _cc1_accepts(body):
|
||||
"""Does THE REAL BUILD FRONT END (gcc-2.7.2 `cc1`) accept this? — not modern gcc.
|
||||
|
||||
They DISAGREE, and it matters: C11 permits redefining a typedef to the same type, C89/2.7.2 does
|
||||
NOT (`redefinition of 'X'`). Validating a compatibility rule against a compiler that is not the
|
||||
one compiling the code is *exactly* the failure this module exists to prevent — a recovery pass
|
||||
that "approves" a declaration the real front end then refuses."""
|
||||
d = os.path.join(REPO, '.run/audit/cdecl')
|
||||
os.makedirs(d, exist_ok=True)
|
||||
c, i = os.path.join(d, 'cc1probe.c'), os.path.join(d, 'cc1probe.i')
|
||||
open(c, 'w').write(body)
|
||||
p = subprocess.run(CPP + ['-Isrc', c], capture_output=True, text=True, cwd=REPO)
|
||||
if p.returncode:
|
||||
return None # cannot adjudicate
|
||||
open(i, 'w').write(p.stdout)
|
||||
r = subprocess.run([CC1] + CC1FLAGS + [i, '-o', '/dev/null'],
|
||||
capture_output=True, text=True, cwd=REPO)
|
||||
return not (r.returncode or _ERR.search(r.stderr or ''))
|
||||
|
||||
|
||||
def _synth_distinct(decls):
|
||||
"""Synthesize each unknown type as a DISTINCT struct. Emitting `typedef int Vec8;` would collapse
|
||||
`extern Vec8 X;` and `extern int X;` into the same type and the probe would call them compatible
|
||||
— the oracle destroying the very distinction it is meant to test."""
|
||||
known, out = _known_types(), set()
|
||||
for d in decls:
|
||||
for tok in re.findall(r'[A-Za-z_]\w*', (d.base or '') + ' ' + ' '.join(d.params or [])):
|
||||
if tok not in known and tok not in ('void', 'struct', 'union', 'enum'):
|
||||
out.add(tok)
|
||||
return ''.join(f'typedef struct {{ int _{t}; }} {t};\n' for t in sorted(out))
|
||||
|
||||
|
||||
def audit_compat(limit=2500):
|
||||
"""ORACLE 4 — `compatible()` vs the REAL cc1, on every (TU-decl, draft-decl) pair the corpus
|
||||
actually contains. My predicate says accept/reject; gcc-2.7.2 says accept/reject. Any
|
||||
disagreement is MY bug, and it is the kind that silently caps the whole recovery pipeline."""
|
||||
tus = sorted(_glob.glob(os.path.join(REPO, 'src/ov_SC01_077/*.c')))
|
||||
scopes = {t: tu_scope(t) for t in tus}
|
||||
pairs = {}
|
||||
import random
|
||||
for p in random.Random(11).sample(_drafts(), min(limit, len(_drafts()))):
|
||||
try:
|
||||
txt = open(p, errors='replace').read()
|
||||
except Exception:
|
||||
continue
|
||||
for st in split_statements(txt):
|
||||
try:
|
||||
ds = parse(st.text)
|
||||
except CDeclError:
|
||||
continue
|
||||
for d in ds:
|
||||
if d.is_definition:
|
||||
continue
|
||||
for ns in scopes.values():
|
||||
tu = ns.get(d.name)
|
||||
if tu is None:
|
||||
continue
|
||||
key = (tu.declaration(name='X'), d.declaration(name='X'))
|
||||
pairs.setdefault(key, (compatible(tu, d), (tu, d)))
|
||||
|
||||
print(f'[compat] {len(pairs)} distinct (TU-decl, draft-decl) pairs from the real corpus')
|
||||
print(f'[compat] adjudicator: gcc-2.7.2 cc1 (THE BUILD FRONT END), not modern gcc')
|
||||
bad, skip = [], 0
|
||||
for (tu_d, dr_d), (mine, (tu, d)) in pairs.items():
|
||||
body = _PROBE_HDR + _synth_distinct([tu, d]) + tu_d + '\n' + dr_d + '\n'
|
||||
got = _cc1_accepts(body)
|
||||
if got is None:
|
||||
skip += 1
|
||||
elif got != mine:
|
||||
bad.append(((tu_d, dr_d), mine, got))
|
||||
print(f'[compat] agree: {len(pairs) - len(bad) - skip} DISAGREE: {len(bad)} unadjudicable: {skip}')
|
||||
for (k, mine, got) in bad[:10]:
|
||||
print(f' TU : {k[0]}\n draft: {k[1]}')
|
||||
print(f' cdecl says {"compatible" if mine else "CONFLICT"}, cc1 says '
|
||||
f'{"compatible" if got else "CONFLICT"}\n')
|
||||
# An UNADJUDICABLE pair is a silent skip, and a gate that skips its whole corpus and prints GREEN
|
||||
# is the exact bug this module was written to hunt. (It did that here, once: a missing `-Isrc`
|
||||
# made cpp fail on all 1,485 probes, `bad` stayed empty, and the audit reported success. R32 is
|
||||
# not "fail loud" — it is COUNT WHAT YOU SKIPPED.)
|
||||
if skip:
|
||||
print(f' !! {skip} pair(s) could not be adjudicated — a skipped check is NOT a passed one')
|
||||
return not bad and not skip
|
||||
|
||||
|
||||
def audit_differential():
|
||||
"""ORACLE 3 — vs THE FIFTEEN INCUMBENTS. This parser must find a strict SUPERSET of every
|
||||
scanner it replaces: any symbol an incumbent sees and this one does not is a defect in THIS
|
||||
@@ -1018,6 +1241,8 @@ def main():
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--audit', action='store_true', help='oracle 1 (coverage) + oracle 3 (differential)')
|
||||
ap.add_argument('--gcc', action='store_true', help='oracle 2: the C front end (slow, decisive)')
|
||||
ap.add_argument('--compat', action='store_true',
|
||||
help='oracle 4: compatible() vs the REAL gcc-2.7.2 cc1 on live corpus pairs')
|
||||
ap.add_argument('--limit', type=int, help='sample N TUs (a fast smoke run)')
|
||||
ap.add_argument('--tu', help='print the file-scope scope of a TU')
|
||||
ap.add_argument('--above', help='...truncated above this function\'s INCLUDE_ASM stub')
|
||||
@@ -1037,7 +1262,7 @@ def main():
|
||||
print(f' {n:<24} {d.kind:<12} {d.type}')
|
||||
print(f'{len(ns)} file-scope names visible' + (f' above {a.above}' if a.above else ''))
|
||||
return
|
||||
if a.audit or a.gcc:
|
||||
if a.audit or a.gcc or a.compat:
|
||||
ok = True
|
||||
if a.audit:
|
||||
ok &= audit_coverage(a.limit, a.verbose)
|
||||
@@ -1046,6 +1271,9 @@ def main():
|
||||
if a.gcc:
|
||||
print()
|
||||
ok &= audit_gcc(a.limit)
|
||||
if a.compat:
|
||||
print()
|
||||
ok &= audit_compat()
|
||||
print('\n' + ('cdecl: ALL ORACLES GREEN' if ok else 'cdecl: DEFECTS FOUND (see above)'))
|
||||
sys.exit(0 if ok else 1)
|
||||
ap.print_help()
|
||||
|
||||
Reference in New Issue
Block a user