diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 555ece32f..97ef8e580 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -4583,10 +4583,29 @@ A 12-agent Ultracode wave over freshly-prefetched `ov_SC06_018` exemplars return This is §58's law at its sharpest — and splicing each class individually gave three *different* blockers, none of which the ladder currently clears: -1. **§8e-2 jtbl table-count drift — 10 of 12 drafts.** +1. **jtbl NON-CONTIGUOUS CARVE — 10 of 12 drafts.** *(Corrected: I first filed this as "§8e-2 + table-count drift". That is the SYMPTOM the filter reports; it is not the wall, and the fix is + NOT a jtbl_carve code change.)* `jtbl_rodata_pads: more rodata .align directives than pad specs (2) — table-count drift vs the carve`. - The draft introduces a switch/jump table into a TU whose jtbl carve has a FIXED pad spec, so the - fail-loud guard fires. This is the already-named wall that blocks `func_8014032C` and `func_8013BD74`. + The draft introduces a switch/jump table into a TU whose carve has a FIXED pad spec, so + `jtbl_rodata_pads` fires. But re-running `jtbl_carve --func ` to re-derive the spec REFUSES + with the real reason: *"subseg would host NON-CONTIGUOUS .rodata carves (0xaa810 and 0xaa920) — + a single object can't leave a gap for the unmatched jtbl between them."* The newly-banked + function's table is separated from the TU's existing carve by an UNMATCHED function's table, and + one object cannot straddle that gap. + + **THE RECIPE (byte-proven on `func_80135A4C`, 181 ins / 138 members):** + + tools/jr_isolate_all.py --only # give the fn its OWN code subseg + make extract BINARY= && make build # isolation is BYTE-NEUTRAL by construction — verify + + tools/jtbl_carve.py --func # now the table carves contiguously in its own object + make extract BINARY= && make build # -> BYTE-IDENTICAL + + The tool names its own remedy in the refusal message, and `jtbl_family_bank` already auto-isolates + on this class (Phase-29 Task-8) — but `gate_stage`/`harvest_verify` do NOT, which is why a wave + that banks through the ordinary gate reports a flat 0 and looks like a compiler wall. **A config + change needs `make extract`, not just `make build`** (the R22 corollary) — both steps above. **The structural finding: fresh crack fuel in a well-matched overlay CONCENTRATES in jtbl-carved TUs** (10 of 12 here), because the non-carved TUs were harvested first. So §8e-2 is not a rare straggler — it is the gate on the next tranche of substantial cracking. @@ -4608,3 +4627,34 @@ selection tool, and every selection tool in this project has eventually lied (R3 **Preserved:** all 12 drafts at `.run/giants/t5wave_*` (R20) — they are genuine cracks with per-function lever notes, recoverable the moment the three ladder stages exist. Do NOT re-draft them. + +### §61b — The jtbl gate stage: built, and the ORDERING law it exposed (Phase 29 Task-14 stage 4, 2026-07-21) + +`gate_stage` now carries a jtbl stage (`_jtbl_prepare`): for every draft whose function references a +`jtbl_`, carve its table into a contiguous object, auto-isolating (`jr_isolate_all --only `) on the +§8b walls — the logic lifted from `jtbl_family_bank` rather than re-implemented (R33). It is wired, it +runs, and it **does not yet bank**, for a reason worth writing down: + +> **THE CARVE MUST FOLLOW THE SPLICE.** The non-contiguity that requires isolation is only *detectable* +> once the function's body is in the object. While it is still `INCLUDE_ASM`, `jtbl_carve` reports +> SUCCESS and produces a spec that does not hold once the body lands. + +Byte-witnessed both ways on `func_80135A4C`: carving the **spliced** function → `NON-CONTIGUOUS … +0xaa810 and 0xaa920`; carving the **unspliced** one → `prepared 1/1`, no isolation, and the draft then +gates as a byte-DIFF. The MANUAL order banks it byte-identical: + + jr_isolate_all --only ; make extract ; ; jtbl_carve --func ; make extract ; build + +`gate_stage` runs the stage before `_gate1`, but `harvest_verify` owns the splice — so the fix is a +per-draft prep INSIDE the splice loop (harvest_verify), not a batch pre-pass in gate_stage. That is the +next increment; the stage's carve/isolate/undo machinery is correct and reusable as-is. + +**Two sub-findings, both paid for:** +* **A wholesale `git checkout -- config/…` undo is WRONG in a batch gate.** `jfb.revert` is right for + `jtbl_family_bank`'s one-function-at-a-time flow, but here it discarded a PREVIOUSLY-banked-but- + uncommitted carve in the same overlay, leaving that bank's source with no subseg → `undefined + reference to func_80136C90` at link. An inverse/wholesale undo cannot know what it did not do. + Now a SNAPSHOT-RESTORE of `config/splat..yaml` + `config/overlays.mk`, plus removal of only the + region files THIS run created (§61's constraint, applied where I had first ignored my own rule). +* **Being in a `_jr_*` TU ≠ having a table.** Only 4 of 8 wave drafts in jtbl-carved TUs actually + reference a `jtbl_`; the stage correctly prepares only those. The other 4 fail for other classes. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 309c0b78d..78f622c3f 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -828,6 +828,32 @@ conditional) · main-EXE/B9 + GLM/B6 + resident's 14 walls (P30) · behemoths B7 later stage than the warnings I was reading) — **check `rc`, read the tail unfiltered.** Tree reverted to clean; nothing banked, nothing committed to src. +- **✅ 2026-07-21 — TASK-14 STAGE 4 (jtbl isolate+carve) BUILT + the ORDERING LAW that blocks it.** + Diagnosed the wave's 10/12 blocker properly: **NOT "§8e-2 table-count drift"** (the symptom the filter + reports) but a **NON-CONTIGUOUS .rodata carve** — the new function's table is separated from the TU's + existing carve by an UNMATCHED function's table, and one object can't straddle that gap. `jtbl_carve` + names its own remedy in the refusal. **RECIPE BYTE-PROVEN on `func_80135A4C`** (181 ins, 138 members): + `jr_isolate_all --only ` → `make extract` → splice → `jtbl_carve --func ` → `make extract` + → **BYTE-IDENTICAL** (isolation is byte-neutral by construction; verified separately). + **BUILT:** `gate_stage._jtbl_prepare` — per-draft carve + auto-isolate on the §8b walls, logic LIFTED + from `jtbl_family_bank` (R33), `GATE_NO_ARITY` A/B guard, snapshot-restore undo. Ladder is now + canon → cast → reconcile_tu → **jtbl** → **arity** → gate → sig_unify → gate. + **⚠️ IT DOES NOT YET BANK, and the reason is the finding: THE CARVE MUST FOLLOW THE SPLICE.** The + non-contiguity is only DETECTABLE once the body is in the object; while still `INCLUDE_ASM`, + `jtbl_carve` reports SUCCESS and yields a spec that fails once the body lands. Byte-witnessed both + ways (spliced → `NON-CONTIGUOUS 0xaa810/0xaa920`; unspliced → `prepared 1/1` then byte-DIFF). + A/B'd out the innocent suspects: draft transforms leave the body **IDENTICAL** through canon/cast/ + reconcile_tu, and `GATE_NO_ARITY=1` changes nothing. **FIX (next increment): per-draft prep INSIDE + `harvest_verify`'s splice loop**, not a batch pre-pass in `gate_stage` — harvest_verify owns the + splice. The carve/isolate/undo machinery is correct and reusable as-is. + **Two sub-findings paid for:** (a) a wholesale `git checkout -- config/` undo (jfb.revert) is WRONG in + a batch gate — it discarded a previously-banked-but-UNCOMMITTED carve, leaving that bank's source with + no subseg (`undefined reference to func_80136C90`); now snapshot-restore + drop only this run's region + files (§61's own constraint, which I had ignored). (b) being in a `_jr_*` TU ≠ having a table — only + 4 of 8 wave drafts actually reference a `jtbl_`. Cookbook **§61a corrected + §61b**. + **Tree restored byte-identical; nothing banked this round; `func_80135A4C`'s manual bank was reverted + with it** (its draft is preserved at `.run/giants/t5wave_*`, re-bankable by the recipe in minutes). + > **🛑 SESSION-6 CHECKPOINT (2026-07-21) — safe to open a FRESH session here.** Tree clean (only the R23 > `db.*.gbf` churn + 4 preserved wave-4 `-O0` drafts). **R22 clean-fleet 140/140 byte-identical**; > `make tools-health` OK (dedup **1847/0**, C1 234343/234343); 0 NON_MATCHING (G4). HEAD `commit:0772` diff --git a/tools/gate_stage.py b/tools/gate_stage.py index 2e42e7816..3f739a821 100644 --- a/tools/gate_stage.py +++ b/tools/gate_stage.py @@ -73,6 +73,106 @@ def _xform(tool, ov, indir, suffix, extra=None): return out if _isdir(out) else indir +# -------------------------------------------------------------------------------------------- +# the jtbl stage (Phase-29 Task-14 stage 4) — isolate + re-carve so a table-bearing draft can link +# -------------------------------------------------------------------------------------------- +_JTBL_RE = re.compile(r"jtbl_[0-9A-Fa-f]{8}") + + +def _fn_has_jtbl(binary, fn): + """Does this function reference a jump table? (Then banking it needs a .rodata carve.)""" + try: + hit = next((s for s in corpus.stubs(binary).values() if s.symbol == fn), None) + except Exception: + return False + if hit is None: + return False + try: + return bool(_JTBL_RE.search(open(os.path.join(REPO, hit.asm_path)).read())) + except OSError: + return False + + +def _jtbl_prepare(binary, draft_fns): + """Carve every jtbl-bearing draft's table into its OWN contiguous object, auto-isolating on the + §8b same-subseg walls. Returns (prepared, keep_regions). + + WHY THIS STAGE EXISTS (byte-proven 2026-07-21, cookbook §61a). A 12-agent wave produced 11 + match_one MATCHes and the gate banked ZERO. 10 of the 12 drafts land in jtbl-CARVED TUs, and the + filter reported `more rodata .align directives than pad specs` — which reads like a compiler + wall and is not one. Re-running jtbl_carve gives the real reason: + + subseg would host NON-CONTIGUOUS .rodata carves (0xaa810 and 0xaa920) — a single object + can't leave a gap for the unmatched jtbl between them. + + The newly-banked function's table is separated from the TU's existing carve by an UNMATCHED + function's table. The remedy is §8b lazy isolation (give the fn its own code subseg, so its + carve spans only its own tables) — which `jtbl_family_bank` has done since Task 8 but + `gate_stage` did not, so every ordinary wave banking into a jtbl TU reported a phantom 0. + + STRUCTURAL NOTE: fresh crack fuel in a well-matched overlay CONCENTRATES in jtbl-carved TUs (the + non-carved ones get harvested first), so this is not a straggler path — it gates the next + tranche of substantial cracking. + + The isolate/revert primitives are IMPORTED from jtbl_family_bank rather than re-implemented + (R33: one implementation, two callers — the divergence between those two callers is exactly + what produced this bug). Config is shared-ish state, so the caller undoes by RESTORE + (jfb.revert), never an inverse transform, and verifies fleet-wide (R22) — cookbook §61.""" + import jtbl_family_bank as jfb + todo = [f for f in draft_fns if _fn_has_jtbl(binary, f)] + if not todo: + return [], None + # SNAPSHOT the config this stage may rewrite (§61 constraint). NOT jfb.revert(): that does a + # wholesale `git checkout -- config/…`, which is correct for jtbl_family_bank's one-function- + # at-a-time flow but WRONG here — it discards any PREVIOUSLY-banked-but-uncommitted carve in the + # same overlay, leaving that bank's source with no subseg to live in (byte-witnessed: it stripped + # func_80135A4C's carve while keeping its banked region file -> `undefined reference to + # func_80136C90` at link). An inverse/wholesale undo cannot know what it did not do. + keep = jfb.region_files(binary) + snap = {} + for f in (os.path.join(REPO, f"config/splat.{binary}.yaml"), + os.path.join(REPO, "config/overlays.mk")): + try: + snap[f] = open(f).read() + except OSError: + pass + prepared = [] + for fn in todo: + r = sh([PY, "tools/jtbl_carve.py", binary, "--func", fn], timeout=900) + out = (r.stdout or "") + (r.stderr or "") + if r.returncode and ("NON-CONTIGUOUS" in out or "do not fit the span" in out): + # §8b lazy isolation, then re-extract so the carve reads the new subseg layout, then retry. + # The DISTINCT "more rodata .align than pad specs" drift is NOT isolate-fixable and is + # deliberately not retried here (it falls through as a carve failure). + if sh([PY, "tools/jr_isolate_all.py", binary, "--only", fn], timeout=900).returncode: + print(f"[gate] jtbl: isolate failed for {fn}", file=sys.stderr); continue + if sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800).returncode: + print(f"[gate] jtbl: extract after isolate failed for {fn}", file=sys.stderr); continue + _corpus_reset() + r = sh([PY, "tools/jtbl_carve.py", binary, "--func", fn], timeout=900) + if r.returncode: + last = ((r.stdout or "") + (r.stderr or "")).strip().splitlines()[-1:] or [""] + print(f"[gate] jtbl: carve failed for {fn}: {last[0][:150]}", file=sys.stderr); continue + prepared.append(fn) + if prepared: + # a CONFIG change needs a re-extract, not just a rebuild (the R22 corollary) + sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800) + _corpus_reset() + print(f"[gate] jtbl: prepared {len(prepared)}/{len(todo)} table-bearing draft(s): " + f"{', '.join(prepared)}") + return prepared, (keep, snap) + + +def _corpus_reset(): + """Drop corpus caches after a config/extract change — the stub set and each stub's home TU + have moved, and every later stage derives from them.""" + for f in (getattr(corpus, "stubs", None), getattr(corpus, "sig", None), + getattr(corpus, "o0_sources", None), getattr(corpus, "symbols", None), + getattr(corpus, "src_files", None)): + if hasattr(f, "cache_clear"): + f.cache_clear() + + def _gate1(binary, src, asm, out, good_sha, d, verified_out=None, failed_out=None): """Whole-binary byte-gate (G3/P9, sole arbiter) on draft-dir d; return the verified func list. harvest_verify reads the CURRENT src as its baseline (so verified fns ACCUMULATE across calls — @@ -258,9 +358,17 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_ # raise on a non-zero exit, so the try/except below never saw it. Hence the explicit rc check: # a pre-pass that quietly does nothing is indistinguishable from one that found nothing to do, # which is the whole failure mode this ladder exists to remove (R32). + # jtbl stage FIRST: isolation rewrites which TU hosts the stub, and every later stage + # (arity scan, harvest_verify's TU derivation) reads that layout. + _jtbl_prepared, _jtbl_keep = [], None + try: + _jtbl_prepared, _jtbl_keep = _jtbl_prepare(binary, draft_fns) + except Exception as e: # never let the pre-pass sink the gate + print(f"[gate] jtbl stage skipped: {e}", file=sys.stderr) + _arity_rc = None _arity_snapshot = {} - if draft_fns: + if draft_fns and not os.environ.get("GATE_NO_ARITY"): # snapshot every file the pre-pass may touch, so the undo is a restore, not a re-derivation for _f in ([os.path.join(REPO, "src/shared/engine_core.h")] + glob.glob(os.path.join(REPO, f"src/{binary}/{binary}*.c"))): @@ -294,6 +402,30 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_ # Restoring the pre-pass snapshot and re-applying ONLY for the functions that banked is exact by # construction and cannot invent a signature. _unbanked = [f for f in draft_fns if f not in verified] + + # jtbl undo: if NOTHING banked, restore this overlay's config + drop the region files THIS run + # created (jfb.revert(keep_regions=...) keeps a previously-banked core's). If something DID bank, + # the carve/isolation is load-bearing for it and must stay — so we only revert the all-fail case, + # and a mixed batch keeps the config that the surviving banks need (the byte-gate already + # reverted the losers' source). + if _jtbl_prepared and not verified and _jtbl_keep: + try: + import jtbl_family_bank as jfb + _keep_regions, _snap = _jtbl_keep + for _f, _txt in _snap.items(): # RESTORE, never inverse-transform (§61) + with open(_f, "w") as _fh: + _fh.write(_txt) + for _rf in jfb.region_files(binary) - _keep_regions: # only what THIS run created + try: + os.remove(os.path.join(REPO, _rf)) + except OSError: + pass + sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800) + _corpus_reset() + print(f"[gate] jtbl: restored config for {len(_jtbl_prepared)} draft(s) (none banked)") + except Exception as e: + print(f"[gate] jtbl revert failed: {e}", file=sys.stderr) + if _unbanked and _arity_snapshot: try: for _f, _txt in _arity_snapshot.items():