From b89fcc2edc2b81059ff5b0ef90af6454719d2c27 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 14 Jul 2026 15:39:07 -0600 Subject: [PATCH] =?UTF-8?q?fix(phase-26a):=20A3e=20=E2=80=94=20gate=5Fstag?= =?UTF-8?q?e=20pinned=20the=20byte-gate=20back=20to=204.9%,=20OF=20A3'S=20?= =?UTF-8?q?OWN=20FIX?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit THE WORST DEFECT IN THE AUDIT IS NOT IN A SCANNER. It is one default argument in the CALLER of a scanner we had already fixed. # tools/gate_stage.py:315 summary = run_gate(a.drafts, binary=b, src=a.src or f"src/{b}/{b}.c", ...) # ALWAYS the main .c `src` RESTRICTS the byte-gate to ONE translation unit, and _gate1 does `if src: cmd += ["--src", src]` -- always truthy. A3 had just taught harvest_verify to DERIVE each draft's home TU *when --src is omitted*, lifting the byte-gate's reach from 4.9% to 100%. gate_stage NEVER OMITS IT. The fix was neutralised by its own caller's default, and the PRIMARY BANKING PATH -- every wave, the grinder, the orchestrator, bulk_harvest -- remained structurally unable to bank 250 of ov_SC01_077's 263 stubs. WHY IT SURVIVED 26 PHASES: harvest_verify cannot splice a draft whose stub is not in the TU it was pointed at, so the draft never verifies -- and is then logged as near/failed, i.e. AS A MATCHING PROBLEM. The wave reports a poor close-rate; the function goes to the backlog as a compiler residual. A tool that CANNOT bank a function is indistinguishable, in every log this project keeps, from a function that CANNOT BE banked. PROOF, same draft / same gate / same second: gate_stage rejected func_80129C40; harvest_verify run directly (no --src) VERIFIED it byte-identical and banked it. AND A COUNTING BUG THAT HID THE HIDING (gate_stage:261): when match_one says MATCH but the whole-binary gate rejects, the record is logged status="near" and THE COUNTER IS NEVER INCREMENTED. A 63-draft run printed `banked 0, near 0, failed 0` -- three zeros that do not sum to 63 -- for phases. Nobody ever added them up. (The number was not wrong. It was ABSENT.) ALSO FIXED, sig_unify (the same disease, one level down): it SILENTLY DROPPED 190 of 196 drafts (97%). `cur_stubs` was read from the main .c (13 of 263 stubs), so any draft whose stub lives in a _jr_ carve hit `if fn not in cur_stubs: continue` -- dropped BEFORE THE WRITE: never copied to --out, never gated, never logged, while the summary printed "drafts unified: 6" and read like success. THIS IS GATE_STAGE'S STAGE-2 RECOVERY -- the pass whose whole job is to rescue the stage-1 failures -- and it has been a no-op for nearly every draft it was meant to save. Now: TU derived per draft (corpus.stubs), canon derived from cdecl.tu_scope (cpp -- macro-injected decls finally visible), and _keep() so an already-acceptable decl is left alone (the §19 "sig_unify regresses canonical drafts" failure mode). Reach: 6 -> 196 drafts; callee-externs rewritten 2 -> 90; own def-sig 2 -> 86. MEASURED, all three consumers migrated (196 never-banked drafts): near 5 -> 116 failed 190 -> 17 => 173 of 190 "failures" were PLUMBING, not codegen: now compiling and SCORED instead of invisible. THE PRIZE (measured, not claimed): the backlog holds 1,588 entries at closeness==0 -- body byte-exact per match_one, whole-binary gate rejected. 1,215 have been banked since by other paths. 373 ARE STILL OPEN STUBS WHOSE BODIES ARE ALREADY BYTE-EXACT, sitting in a ledger that calls them unrecoverable. ⚠ THE HARVEST ITSELF IS NOT IN THIS COMMIT, AND IS NOT CLAIMED (P9). Gating the 63 ov_SC01_077 ones dragged `dedup_propagate --auto-from --recover` behind it; it ran >1h and hit its timeout -- its first-ever run over the FULL corpus (A6/A7 unblocked the 407 files it could never see). It MUTATES THE TREE BEFORE IT GATES, so the kill left 859 files + engine_core.h (+544 lines) written and UN-GATED with the registry never updated. R22 on that tree: 44 passed / 92 FAILED -> `git checkout -- src/ config/`, fleet restored to 136/136. Nothing lost (H4: the tree was clean, so the revert was one command). Two real lessons, recorded: dedup_propagate is NOT crash-safe and must never run under a timeout it can hit; and a 63-draft experiment must not drag an unbounded fleet-wide propagation behind it. R22 clean-fleet after revert: 136 passed, 0 failed of 136. src/ and config/ clean. cookbook §51g LAW 11: A FIX IS NOT LANDED UNTIL ITS CALLER STOPS OVERRIDING IT. After fixing a scanner, grep every call site and ask whether a caller's default re-disables it. An audit that stops at the callee is half an audit. --- docs/matching-cookbook.md | 31 ++++++++++++ docs/tooling-audit.md | 19 +++++++ phase-ends/CURRENT_PHASE.md | 41 +++++++++++++++ tools/gate_stage.py | 9 +++- tools/sig_unify.py | 99 ++++++++++++++++++++++++++++--------- 5 files changed, 174 insertions(+), 25 deletions(-) diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 7677b6034..f6e689791 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -3724,3 +3724,34 @@ from **8 → 58 of 196** drafts (7× reach). > the difference between an **invisible failure that reads as a compiler wall** and a **scored near-miss > the permuter and the §47/§48 dials can act on.** Which is the audit's thesis exactly. Do not sell it as > more than it is; three times in one session a confirmed mechanism produced a null consequence. + +**LAW 11 — A FIX IS NOT LANDED UNTIL ITS CALLER STOPS OVERRIDING IT.** The single worst defect in the +audit was not in a scanner. It was one default argument in the caller of a scanner we had *already fixed*: + +```python +# tools/gate_stage.py:315 +summary = run_gate(a.drafts, binary=b, src=a.src or f"src/{b}/{b}.c", ...) # <- ALWAYS the main .c +``` + +`src` **restricts the byte-gate to ONE translation unit.** A3 had just taught `harvest_verify` to derive +each draft's home TU *when `--src` is omitted*, lifting the byte-gate's reach from **4.9% to 100%** — and +**`gate_stage` never omits it.** So the primary banking path (every wave, the grinder, the orchestrator, +`bulk_harvest`) stayed structurally incapable of banking **250 of 263 stubs**, *after the fix*, *because of +its own caller's default*. + +> **AND HERE IS WHY IT SURVIVED 26 PHASES.** `harvest_verify` cannot splice a draft whose stub is not in +> the TU it was pointed at, so the draft simply never verifies — and is then logged as `near`/`failed`, +> i.e. **as a matching problem**. The wave reports a low close-rate and the function goes to the backlog +> as a residual. +> **A tool that CANNOT bank a function is indistinguishable, in every log this project keeps, from a +> function that CANNOT BE banked.** +> Proof, same draft, same gate, same second: `gate_stage` rejected `func_80129C40`; `harvest_verify` run +> directly (no `--src`) **verified it byte-identical and banked it.** + +**And a counting bug that hid the hiding:** when `match_one` says MATCH but the whole-binary gate rejects, +`gate_stage` logs `status="near"` and **never increments the counter**. A run of 63 such drafts printed +`banked 0, near 0, failed 0` — *three zeros that do not sum to 63* — for phases. **Nobody added them up.** +(LAW 2 again, and note the shape: the number was not wrong, it was ABSENT.) + +**Checklist item, promoted to the top:** after fixing a scanner, `grep` every call site and ask *does any +caller pass a default that re-disables this?* An audit that stops at the callee is half an audit. diff --git a/docs/tooling-audit.md b/docs/tooling-audit.md index b782b7adb..8dbbb53f1 100644 --- a/docs/tooling-audit.md +++ b/docs/tooling-audit.md @@ -726,6 +726,25 @@ scanners** — the "best outcome is a deleted scanner" rule (R33), applied at sc 3. **A recovery tool once wrote non-C into drafts:** `extern if ((func_80029178(0x119) & 0xFF) != 0);` appears in 33 drafts, *only* in `-canon`/`-recanon`/`-uni`/`-sigfix` output dirs (the pre-recovery draft has none), and gcc rejects it outright. **R14 blast-radius check: the source bug was already fixed in Phase 19** — today's oracle emits **0 garbage over 300 signatures** — so this is dead historical residue, not a live defect. *Mechanism confirmed, consequence nil.* But note what it cost at the time: a draft that cannot compile fails the byte-gate and reads, downstream, as **an intrinsic compiler wall**. 4. **`tu_ambient`'s function regex drops any callee with a fn-ptr parameter.** Its param class is `[^()]*`, so `extern void func_8012A568(void (*a0)(void));` — a real declaration in ov_SC01_077 — lands in **no bucket at all**: not funcs, not data, not typedefs. +### 🔴 A3e — THE ONE THAT MATTERS: `gate_stage` pinned the byte-gate back to 4.9% — *of its own fix* + +**`tools/gate_stage.py:315`** + +```python +summary = run_gate(a.drafts, binary=b, + src=a.src or f"src/{b}/{b}.c", # <- ALWAYS the main .c +``` + +`src` **restricts the byte-gate to ONE translation unit.** `_gate1` then does `if src: cmd += ["--src", src]` — and `src` is *always* truthy. + +**A3 fixed `harvest_verify` to derive each draft's home TU *when `--src` is omitted*, taking the byte-gate's reach from 4.9% to 100%. `gate_stage` never omits it.** The fix was silently neutralised **by its own caller**, and `gate_stage` — the *primary banking path*, the thing every wave, the grinder, the orchestrator and `bulk_harvest` call — remained structurally incapable of banking **250 of ov_SC01_077's 263 stubs (95%)**. + +**How it hid.** `harvest_verify` cannot splice a draft whose stub is not in the TU it was pointed at, so those drafts simply never verify. They are then logged as `near`/`failed` — i.e. *as matching problems* — and the wave reports a low close-rate. **A tool that cannot bank a function is indistinguishable, in every log this project keeps, from a function that cannot be banked.** Proof: `func_80129C40` — `gate_stage` rejected it; `harvest_verify` run directly (no `--src`) **VERIFIED it byte-identical and banked it.** Same draft, same gate, same second. + +**And a counting bug that hid the hiding** (`gate_stage:261`): when `match_one` says MATCH but the whole-binary gate rejects, the record is logged `status="near"` and **the counter is never incremented**. A run of 63 such drafts printed `banked 0, near 0, failed 0` — three zeros that do not sum to 63, printed for phases, **and nobody ever added them up.** + +**The backlog says how big this is.** 1,588 entries at `closeness == 0` (body byte-exact per `match_one`, whole-binary gate rejected). 1,215 have been banked since by other paths — leaving **373 still-open stubs whose bodies are already byte-exact**, sitting in a backlog that describes them as unrecoverable. + ### A3c — the first consumer migration (`cast_call_sites`), and what it measured | | | diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 1a73f2d45..8b94684b5 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -659,6 +659,47 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-14 (session 10, A3e — 🔴 THE BIG ONE: `gate_stage` pinned the byte-gate back to 4.9% — *of A3's own + fix*; Max):** Migrating `sig_unify` exposed a chain, and the last link is the worst defect in the whole audit. + **(1) `sig_unify` silently DROPPED 190 of 196 drafts (97%).** `cur_stubs` was read from the main `.c` (13 + stubs of 263), so any draft whose stub lives in a `_jr_*` carve hit `if fn not in cur_stubs: continue` — + dropped **before the write**: never copied to `--out`, never gated, never logged. The summary printed *"drafts + unified: 6"* and read like success. **This is gate_stage's STAGE-2 RECOVERY** — the pass whose entire job is + to rescue stage-1 failures — **and it has been a no-op for almost every draft it was meant to save.** Fixed by + deriving the TU per draft (`corpus.stubs`) + the canon from `cdecl.tu_scope` (cpp, so macro-injected decls are + visible) + `_keep()` (only rewrite when cc1 would actually reject — the §19 "sig_unify regresses canonical + drafts" failure mode). Reach: **6 → 196 drafts; callee-externs rewritten 2 → 90; own def-sig 2 → 86.** + With all three consumers migrated the 196-draft failure profile went **near 5 → 116, failed 190 → 17**: + **173 of 190 "failures" were PLUMBING, not codegen** — now compiling and *scored* instead of invisible. + **(2) 🔴 `gate_stage.py:315`: `src=a.src or f"src/{b}/{b}.c"`.** `src` **RESTRICTS the byte-gate to ONE TU**, + and `_gate1` does `if src: cmd += ["--src", src]` — always truthy. **A3 had just taught `harvest_verify` to + derive each draft's home TU when `--src` is OMITTED (reach 4.9% → 100%), and `gate_stage` never omits it.** + The fix was neutralised **by its own caller's default**, leaving the *primary banking path* (every wave, the + grinder, the orchestrator, `bulk_harvest`) structurally unable to bank **250 of 263 stubs**. + **WHY IT SURVIVED 26 PHASES:** a draft whose stub isn't in the pointed-at TU simply never verifies → it is + logged `near`/`failed`, i.e. **as a MATCHING problem** → the wave reports a poor close-rate → the function + goes to the backlog as a compiler residual. **A tool that CANNOT bank a function is indistinguishable, in + every log this project keeps, from a function that CANNOT BE banked.** Proof, same draft/gate/second: + `gate_stage` rejected `func_80129C40`; **`harvest_verify` run directly (no `--src`) VERIFIED it + byte-identical and banked it.** **(3)** And a counting bug hid the hiding (`gate_stage:261`): `match_one` + MATCH + gate-reject logs `status="near"` and **never increments the counter** — a 63-draft run printed + `banked 0, near 0, failed 0`, *three zeros that do not sum to 63*, for phases, and **nobody added them up**. + **THE PRIZE, MEASURED:** the backlog holds **1,588** `closeness==0` entries (body byte-exact per `match_one`, + gate rejected). 1,215 were banked since by other paths; **373 are STILL OPEN STUBS whose bodies are already + byte-exact**, sitting in a ledger that calls them unrecoverable. + **⚠ THE HARVEST ATTEMPT FAILED AND WAS REVERTED (P9).** Gating the 63 ov_SC01_077 ones dragged + `dedup_propagate --auto-from --recover` behind it, which ran **>1 h and hit its timeout** — its first-ever run + over the FULL corpus (A6/A7 unblocked the 407 files it could never see). It **mutates the tree BEFORE it + gates**, so the kill left **859 files + engine_core.h (+544 lines) written and UN-GATED, and the registry + never updated**. R22 clean-fleet on it: **44 passed / 92 FAILED** → **`git checkout -- src/ config/`**, fleet + restored. **Nothing was lost (H4: the tree was clean, so the revert was one command) — but two real lessons: + `dedup_propagate` is NOT crash-safe and must never run under a timeout it can hit; and a 63-draft experiment + must not drag an unbounded fleet-wide propagation behind it (gate with `propagate=False`, then propagate as + its own bounded job).** + **Committed: the TOOL FIXES only. The harvest is NOT banked and is not claimed.** Knowledge captured live + (R30/R31): cookbook **§51g LAW 11** (*a fix is not landed until its caller stops overriding it* — after + fixing a scanner, grep every call site and ask whether a caller's default re-disables it), tooling-audit + **A3e**. **NEXT: re-run the 373-stub harvest with `propagate=False`, then propagate separately.** - **2026-07-14 (session 10, A3d — RETIRE the fleet-majority oracle; both banking paths; Max):** R33 applied to the audit's worst finding: **not fixed, RETIRED.** `reconcile_decls` asks *"what does the FLEET call this symbol?"*; C asks *"what does THIS TU declare?"*. The engine is loosely typed, so one fleet-wide answer is diff --git a/tools/gate_stage.py b/tools/gate_stage.py index dcd44fd95..571a0f3db 100644 --- a/tools/gate_stage.py +++ b/tools/gate_stage.py @@ -259,6 +259,9 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_ note = sm.group(1).strip() if sm else None if kind == "match": # match_one says MATCH but the whole-binary gate rejected -> plumbing/TU conflict status, where = "near", note or "match_one MATCH but gate rejected (declaration/TU plumbing)" + near += 1 # …and COUNT it. It was logged as `near` and counted as NOTHING, so a + # run of 63 such drafts printed "banked 0, near 0, failed 0" — three + # zeros that do not sum to 63, and nobody ever added them up (R32). elif kind == "near": status, where = "near", note or f"{meta.get('lever') or meta.get('class') or 'residual'}: {close} mismatch" near += 1 @@ -311,8 +314,12 @@ def main(): ap.add_argument("--commit", action="store_true") a = ap.parse_args() b = a.binary + # `src` RESTRICTS the byte-gate to ONE TU. Defaulting it to the main .c silently pinned the gate + # to 13 of 263 stubs (4.9%): A3 taught harvest_verify to derive each draft's home TU *when --src + # is omitted* — and this caller never omitted it, so the fix was neutralised by its own caller. + # Now: pass --src ONLY when a human explicitly restricts the gate. Omitted => derive per draft. summary = run_gate(a.drafts, binary=b, - src=a.src or f"src/{b}/{b}.c", + src=a.src, asm=a.asm_subdir or f"asm/{b}/nonmatchings/{b}", out=a.out or f"build/{b}/{b}", good_sha=a.good_sha or DEF_SHA, diff --git a/tools/sig_unify.py b/tools/sig_unify.py index b935f91e9..4b7266fcc 100644 --- a/tools/sig_unify.py +++ b/tools/sig_unify.py @@ -145,6 +145,53 @@ def rewrite_def(txt, fn, canon): return txt[:m.start()] + new_hdr + txt[m.end():], True +_cdecl = _load('cdecl', 'tools/cdecl.py') if '_load' in dir() else None + + +def _tu_for(overlay, fn): + """The TU that holds this function's stub — DERIVED (corpus.stubs); None if it is not an open + stub anywhere (already banked, or not ours).""" + try: + import importlib.util + spec = importlib.util.spec_from_file_location('corpus', os.path.join(REPO, 'tools/corpus.py')) + c = importlib.util.module_from_spec(spec) + spec.loader.exec_module(c) + st = c.stubs(overlay).get(int(fn[5:], 16)) + return os.path.join(REPO, st.path) if st else None + except Exception: + return os.path.join(REPO, f'src/{overlay}/{overlay}.c') + + +def _canon_for(tu): + """{symbol: 'extern ;'} — DERIVED from what cc1 sees in THIS TU (cpp), so the + macro-injected `DEFINE_func_*` externs are finally visible. The old union of three raw-text + scanners (collect_define_sigs + collect_inline_sigs + collect_extern_sigs) could not see them, + and its DATA_DECL_RE was blind to fn-ptr / sized-array / multi-declarator decls (44 of 646 + symbols in this very corpus) — so the recovery pass silently did nothing for exactly the symbols + that were failing.""" + import cdecl as cd + out = {} + for name, d in cd.tu_scope(tu).items(): + if d.storage == 'typedef': + continue + out[name] = d.declaration(storage='extern') + return out + + +def _keep(draft_line, canon_decl): + """Leave the draft's decl alone when cc1 would accept it beside the TU's. sig_unify's known + failure mode is REGRESSING already-correct drafts (§19/§25: 'canon-first, sig_unify FALLBACK'), + and a rewrite that changes nothing semantic can still perturb codegen. Rewrite only when the + front end would actually reject the pair.""" + try: + import cdecl as cd + a = cd.parse(canon_decl)[0] + b = cd.parse(draft_line.strip())[0] + return cd.compatible(a, b) + except Exception: + return False + + def main(): ap = argparse.ArgumentParser() ap.add_argument('--overlay', default='ov_SC01_077') @@ -157,38 +204,41 @@ def main(): ap.add_argument('--out', dest='outdir', required=True) args = ap.parse_args() - c_path = os.path.join(REPO, args.src_file) if args.src_file else \ - os.path.join(REPO, f'src/{args.overlay}/{args.overlay}.c') - ec = os.path.join(REPO, 'src/shared/engine_core.h') - - fdef = _ght.collect_define_sigs(ec) - fdef.update(_ght.collect_inline_sigs(c_path)) - fdecl = _ght.collect_extern_sigs([ec, c_path]) - func_canon = {} - for a, s in fdecl.items(): - func_canon[f'func_{a:08X}'] = f'extern {s};' - for a, s in fdef.items(): - func_canon[f'func_{a:08X}'] = f'extern {s};' - data_canon = collect_data_decls([c_path, ec]) - canon = {**func_canon, **data_canon} - + override = os.path.join(REPO, args.src_file) if args.src_file else None os.makedirs(os.path.join(REPO, args.outdir), exist_ok=True) - stub_re = re.compile(r'INCLUDE_ASM\([^,]*,\s*(func_[0-9A-Fa-f]+)\)') - cur_stubs = set(stub_re.findall(open(c_path).read())) - n = ext_rw = def_rw = 0 + n = ext_rw = def_rw = passthru = 0 + _canon_cache = {} for p in sorted(glob.glob(os.path.join(REPO, args.indir, '*.c'))): fn = os.path.basename(p)[:-2] - if fn not in cur_stubs: - continue txt = open(p).read() + + # WHICH TU? DERIVED (Phase 26-A, §51g LAW 10). `--src-file` was an optional hand-passed flag + # defaulting to src//.c, and `cur_stubs` was read from THAT file — so a draft whose + # stub lives in a Phase-26 `_jr_` carve was not in the set and hit + # if fn not in cur_stubs: continue + # which dropped it BEFORE THE WRITE. It never reached --out, was never gated, was never + # logged; the summary just printed a smaller "drafts unified" and read like success. + # MEASURED: 190 of 196 drafts (97%) silently vanished — and this is gate_stage's STAGE-2 + # RECOVERY, the pass whose whole job is to rescue the stage-1 failures. It has been a no-op + # for almost every draft it was meant to save. + # corpus.stubs() knows which TU holds each stub (the INCLUDE_ASM line is self-describing), + # so the derivation IS the check and the silent drop is now structurally impossible. + tu = override or _tu_for(args.overlay, fn) + if tu is None: # not an open stub anywhere -> nothing to unify + open(os.path.join(REPO, args.outdir, os.path.basename(p)), 'w').write(txt) + passthru += 1 # PASSED THROUGH and COUNTED, never dropped + continue + if tu not in _canon_cache: + _canon_cache[tu] = _canon_for(tu) + canon = _canon_cache[tu] ext_changed = False def repl(m): nonlocal ext_changed line = m.group(0) s = sym_of(line) - if s and s in canon: + if s and s in canon and not _keep(line, canon[s]): new = canon[s] if re.sub(r'\s+', ' ', new).strip() != re.sub(r'\s+', ' ', line).strip(): ext_changed = True @@ -201,7 +251,7 @@ def main(): nonlocal ext_changed line = m.group(0) s = sym_of(line) - if s and s != fn and s in canon: + if s and s != fn and s in canon and not _keep(line, canon[s]): new = canon[s] if re.sub(r'\s+', ' ', new).strip() != re.sub(r'\s+', ' ', line).strip(): ext_changed = True @@ -215,8 +265,9 @@ def main(): ext_rw += ext_changed def_rw += def_changed - print(f'canonical decls: {len(func_canon)} func + {len(data_canon)} data') - print(f'drafts unified: {n} (callee-externs rewritten in {ext_rw}, OWN def-sig rewritten in {def_rw})') + print(f'drafts in: {n + passthru}; unified: {n}; passed through (not an open stub): {passthru}; ' + f'TUs: {len(_canon_cache)}') + print(f' callee-externs rewritten in {ext_rw}, OWN def-sig rewritten in {def_rw}') if __name__ == '__main__':