mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 15:18:24 -04:00
feat(phase-29 T14 stage 4): jtbl isolate+carve stage built; the CARVE-MUST-FOLLOW-SPLICE law
DIAGNOSIS CORRECTED: the wave's 10/12 blocker is NOT "§8e-2 table-count drift" (the symptom the filter reports) but a NON-CONTIGUOUS .rodata carve — the new function's table is separated from the TU's existing carve by an UNMATCHED function's table, and one object cannot straddle that gap. jtbl_carve names its own remedy in the refusal message. RECIPE BYTE-PROVEN (func_80135A4C, 181 ins / 138 members): jr_isolate_all --only <fn> ; make extract ; <splice> ; jtbl_carve --func <fn> ; make extract ; make build -> BYTE-IDENTICAL (isolation verified byte-neutral on its own first; a config change needs extract, not just build.) BUILT: gate_stage._jtbl_prepare — per-draft carve + auto-isolate on the §8b walls, logic LIFTED from jtbl_family_bank (R33: one implementation, two callers — their divergence IS this bug), snapshot-restore undo, GATE_NO_ARITY A/B guard. Ladder: canon -> cast -> reconcile_tu -> jtbl -> arity -> gate -> sig_unify -> gate. IT DOES NOT YET BANK, and that is the finding: THE CARVE MUST FOLLOW THE SPLICE. The non-contiguity is only DETECTABLE once the body is in the object; while the fn is still INCLUDE_ASM, jtbl_carve reports SUCCESS and produces a spec that fails when the body lands. Byte-witnessed both ways (spliced -> NON-CONTIGUOUS 0xaa810/0xaa920; unspliced -> "prepared 1/1" then byte-DIFF). Innocent suspects A/B'd out: the draft is IDENTICAL through canon/cast/ reconcile_tu, and GATE_NO_ARITY=1 changes nothing. FIX = per-draft prep inside harvest_verify's splice loop (it owns the splice), not a batch pre-pass in gate_stage. SUB-FINDINGS: (a) a wholesale `git checkout -- config/` undo is WRONG in a batch gate — it discarded a previously-banked-but-UNCOMMITTED carve, leaving that bank's source with no subseg (undefined reference to func_80136C90 at link). Now snapshot-restore + drop only this run's region files (§61's constraint, which I had written and then not applied here). (b) being in a _jr_* TU != having a table: only 4 of 8 wave drafts reference a jtbl_. Tree restored byte-identical; nothing banked. cookbook §61a corrected + §61b.
This commit is contained in:
Drew T
@@ -4583,10 +4583,29 @@ A 12-agent Ultracode wave over freshly-prefetched `ov_SC06_018` exemplars return
|
||||
This is §58's law at its sharpest — and splicing each class individually gave three *different* blockers,
|
||||
none of which the ladder currently clears:
|
||||
|
||||
1. **§8e-2 jtbl table-count drift — 10 of 12 drafts.**
|
||||
1. **jtbl NON-CONTIGUOUS CARVE — 10 of 12 drafts.** *(Corrected: I first filed this as "§8e-2
|
||||
table-count drift". That is the SYMPTOM the filter reports; it is not the wall, and the fix is
|
||||
NOT a jtbl_carve code change.)*
|
||||
`jtbl_rodata_pads: more rodata .align directives than pad specs (2) — table-count drift vs the carve`.
|
||||
The draft introduces a switch/jump table into a TU whose jtbl carve has a FIXED pad spec, so the
|
||||
fail-loud guard fires. This is the already-named wall that blocks `func_8014032C` and `func_8013BD74`.
|
||||
The draft introduces a switch/jump table into a TU whose carve has a FIXED pad spec, so
|
||||
`jtbl_rodata_pads` fires. But re-running `jtbl_carve --func <fn>` to re-derive the spec REFUSES
|
||||
with the real reason: *"subseg would host NON-CONTIGUOUS .rodata carves (0xaa810 and 0xaa920) —
|
||||
a single object can't leave a gap for the unmatched jtbl between them."* The newly-banked
|
||||
function's table is separated from the TU's existing carve by an UNMATCHED function's table, and
|
||||
one object cannot straddle that gap.
|
||||
|
||||
**THE RECIPE (byte-proven on `func_80135A4C`, 181 ins / 138 members):**
|
||||
|
||||
tools/jr_isolate_all.py <ov> --only <fn> # give the fn its OWN code subseg
|
||||
make extract BINARY=<ov> && make build # isolation is BYTE-NEUTRAL by construction — verify
|
||||
<splice the draft>
|
||||
tools/jtbl_carve.py <ov> --func <fn> # now the table carves contiguously in its own object
|
||||
make extract BINARY=<ov> && make build # -> BYTE-IDENTICAL
|
||||
|
||||
The tool names its own remedy in the refusal message, and `jtbl_family_bank` already auto-isolates
|
||||
on this class (Phase-29 Task-8) — but `gate_stage`/`harvest_verify` do NOT, which is why a wave
|
||||
that banks through the ordinary gate reports a flat 0 and looks like a compiler wall. **A config
|
||||
change needs `make extract`, not just `make build`** (the R22 corollary) — both steps above.
|
||||
**The structural finding: fresh crack fuel in a well-matched overlay CONCENTRATES in jtbl-carved TUs**
|
||||
(10 of 12 here), because the non-carved TUs were harvested first. So §8e-2 is not a rare straggler —
|
||||
it is the gate on the next tranche of substantial cracking.
|
||||
@@ -4608,3 +4627,34 @@ selection tool, and every selection tool in this project has eventually lied (R3
|
||||
|
||||
**Preserved:** all 12 drafts at `.run/giants/t5wave_*` (R20) — they are genuine cracks with per-function
|
||||
lever notes, recoverable the moment the three ladder stages exist. Do NOT re-draft them.
|
||||
|
||||
### §61b — The jtbl gate stage: built, and the ORDERING law it exposed (Phase 29 Task-14 stage 4, 2026-07-21)
|
||||
|
||||
`gate_stage` now carries a jtbl stage (`_jtbl_prepare`): for every draft whose function references a
|
||||
`jtbl_`, carve its table into a contiguous object, auto-isolating (`jr_isolate_all --only <fn>`) on the
|
||||
§8b walls — the logic lifted from `jtbl_family_bank` rather than re-implemented (R33). It is wired, it
|
||||
runs, and it **does not yet bank**, for a reason worth writing down:
|
||||
|
||||
> **THE CARVE MUST FOLLOW THE SPLICE.** The non-contiguity that requires isolation is only *detectable*
|
||||
> once the function's body is in the object. While it is still `INCLUDE_ASM`, `jtbl_carve` reports
|
||||
> SUCCESS and produces a spec that does not hold once the body lands.
|
||||
|
||||
Byte-witnessed both ways on `func_80135A4C`: carving the **spliced** function → `NON-CONTIGUOUS …
|
||||
0xaa810 and 0xaa920`; carving the **unspliced** one → `prepared 1/1`, no isolation, and the draft then
|
||||
gates as a byte-DIFF. The MANUAL order banks it byte-identical:
|
||||
|
||||
jr_isolate_all --only <fn> ; make extract ; <splice> ; jtbl_carve --func <fn> ; make extract ; build
|
||||
|
||||
`gate_stage` runs the stage before `_gate1`, but `harvest_verify` owns the splice — so the fix is a
|
||||
per-draft prep INSIDE the splice loop (harvest_verify), not a batch pre-pass in gate_stage. That is the
|
||||
next increment; the stage's carve/isolate/undo machinery is correct and reusable as-is.
|
||||
|
||||
**Two sub-findings, both paid for:**
|
||||
* **A wholesale `git checkout -- config/…` undo is WRONG in a batch gate.** `jfb.revert` is right for
|
||||
`jtbl_family_bank`'s one-function-at-a-time flow, but here it discarded a PREVIOUSLY-banked-but-
|
||||
uncommitted carve in the same overlay, leaving that bank's source with no subseg → `undefined
|
||||
reference to func_80136C90` at link. An inverse/wholesale undo cannot know what it did not do.
|
||||
Now a SNAPSHOT-RESTORE of `config/splat.<ov>.yaml` + `config/overlays.mk`, plus removal of only the
|
||||
region files THIS run created (§61's constraint, applied where I had first ignored my own rule).
|
||||
* **Being in a `_jr_*` TU ≠ having a table.** Only 4 of 8 wave drafts in jtbl-carved TUs actually
|
||||
reference a `jtbl_`; the stage correctly prepares only those. The other 4 fail for other classes.
|
||||
|
||||
@@ -828,6 +828,32 @@ conditional) · main-EXE/B9 + GLM/B6 + resident's 14 walls (P30) · behemoths B7
|
||||
later stage than the warnings I was reading) — **check `rc`, read the tail unfiltered.**
|
||||
Tree reverted to clean; nothing banked, nothing committed to src.
|
||||
|
||||
- **✅ 2026-07-21 — TASK-14 STAGE 4 (jtbl isolate+carve) BUILT + the ORDERING LAW that blocks it.**
|
||||
Diagnosed the wave's 10/12 blocker properly: **NOT "§8e-2 table-count drift"** (the symptom the filter
|
||||
reports) but a **NON-CONTIGUOUS .rodata carve** — the new function's table is separated from the TU's
|
||||
existing carve by an UNMATCHED function's table, and one object can't straddle that gap. `jtbl_carve`
|
||||
names its own remedy in the refusal. **RECIPE BYTE-PROVEN on `func_80135A4C`** (181 ins, 138 members):
|
||||
`jr_isolate_all --only <fn>` → `make extract` → splice → `jtbl_carve --func <fn>` → `make extract`
|
||||
→ **BYTE-IDENTICAL** (isolation is byte-neutral by construction; verified separately).
|
||||
**BUILT:** `gate_stage._jtbl_prepare` — per-draft carve + auto-isolate on the §8b walls, logic LIFTED
|
||||
from `jtbl_family_bank` (R33), `GATE_NO_ARITY` A/B guard, snapshot-restore undo. Ladder is now
|
||||
canon → cast → reconcile_tu → **jtbl** → **arity** → gate → sig_unify → gate.
|
||||
**⚠️ IT DOES NOT YET BANK, and the reason is the finding: THE CARVE MUST FOLLOW THE SPLICE.** The
|
||||
non-contiguity is only DETECTABLE once the body is in the object; while still `INCLUDE_ASM`,
|
||||
`jtbl_carve` reports SUCCESS and yields a spec that fails once the body lands. Byte-witnessed both
|
||||
ways (spliced → `NON-CONTIGUOUS 0xaa810/0xaa920`; unspliced → `prepared 1/1` then byte-DIFF).
|
||||
A/B'd out the innocent suspects: draft transforms leave the body **IDENTICAL** through canon/cast/
|
||||
reconcile_tu, and `GATE_NO_ARITY=1` changes nothing. **FIX (next increment): per-draft prep INSIDE
|
||||
`harvest_verify`'s splice loop**, not a batch pre-pass in `gate_stage` — harvest_verify owns the
|
||||
splice. The carve/isolate/undo machinery is correct and reusable as-is.
|
||||
**Two sub-findings paid for:** (a) a wholesale `git checkout -- config/` undo (jfb.revert) is WRONG in
|
||||
a batch gate — it discarded a previously-banked-but-UNCOMMITTED carve, leaving that bank's source with
|
||||
no subseg (`undefined reference to func_80136C90`); now snapshot-restore + drop only this run's region
|
||||
files (§61's own constraint, which I had ignored). (b) being in a `_jr_*` TU ≠ having a table — only
|
||||
4 of 8 wave drafts actually reference a `jtbl_`. Cookbook **§61a corrected + §61b**.
|
||||
**Tree restored byte-identical; nothing banked this round; `func_80135A4C`'s manual bank was reverted
|
||||
with it** (its draft is preserved at `.run/giants/t5wave_*`, re-bankable by the recipe in minutes).
|
||||
|
||||
> **🛑 SESSION-6 CHECKPOINT (2026-07-21) — safe to open a FRESH session here.** Tree clean (only the R23
|
||||
> `db.*.gbf` churn + 4 preserved wave-4 `-O0` drafts). **R22 clean-fleet 140/140 byte-identical**;
|
||||
> `make tools-health` OK (dedup **1847/0**, C1 234343/234343); 0 NON_MATCHING (G4). HEAD `commit:0772`
|
||||
|
||||
+133
-1
@@ -73,6 +73,106 @@ def _xform(tool, ov, indir, suffix, extra=None):
|
||||
return out if _isdir(out) else indir
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# the jtbl stage (Phase-29 Task-14 stage 4) — isolate + re-carve so a table-bearing draft can link
|
||||
# --------------------------------------------------------------------------------------------
|
||||
_JTBL_RE = re.compile(r"jtbl_[0-9A-Fa-f]{8}")
|
||||
|
||||
|
||||
def _fn_has_jtbl(binary, fn):
|
||||
"""Does this function reference a jump table? (Then banking it needs a .rodata carve.)"""
|
||||
try:
|
||||
hit = next((s for s in corpus.stubs(binary).values() if s.symbol == fn), None)
|
||||
except Exception:
|
||||
return False
|
||||
if hit is None:
|
||||
return False
|
||||
try:
|
||||
return bool(_JTBL_RE.search(open(os.path.join(REPO, hit.asm_path)).read()))
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _jtbl_prepare(binary, draft_fns):
|
||||
"""Carve every jtbl-bearing draft's table into its OWN contiguous object, auto-isolating on the
|
||||
§8b same-subseg walls. Returns (prepared, keep_regions).
|
||||
|
||||
WHY THIS STAGE EXISTS (byte-proven 2026-07-21, cookbook §61a). A 12-agent wave produced 11
|
||||
match_one MATCHes and the gate banked ZERO. 10 of the 12 drafts land in jtbl-CARVED TUs, and the
|
||||
filter reported `more rodata .align directives than pad specs` — which reads like a compiler
|
||||
wall and is not one. Re-running jtbl_carve gives the real reason:
|
||||
|
||||
subseg would host NON-CONTIGUOUS .rodata carves (0xaa810 and 0xaa920) — a single object
|
||||
can't leave a gap for the unmatched jtbl between them.
|
||||
|
||||
The newly-banked function's table is separated from the TU's existing carve by an UNMATCHED
|
||||
function's table. The remedy is §8b lazy isolation (give the fn its own code subseg, so its
|
||||
carve spans only its own tables) — which `jtbl_family_bank` has done since Task 8 but
|
||||
`gate_stage` did not, so every ordinary wave banking into a jtbl TU reported a phantom 0.
|
||||
|
||||
STRUCTURAL NOTE: fresh crack fuel in a well-matched overlay CONCENTRATES in jtbl-carved TUs (the
|
||||
non-carved ones get harvested first), so this is not a straggler path — it gates the next
|
||||
tranche of substantial cracking.
|
||||
|
||||
The isolate/revert primitives are IMPORTED from jtbl_family_bank rather than re-implemented
|
||||
(R33: one implementation, two callers — the divergence between those two callers is exactly
|
||||
what produced this bug). Config is shared-ish state, so the caller undoes by RESTORE
|
||||
(jfb.revert), never an inverse transform, and verifies fleet-wide (R22) — cookbook §61."""
|
||||
import jtbl_family_bank as jfb
|
||||
todo = [f for f in draft_fns if _fn_has_jtbl(binary, f)]
|
||||
if not todo:
|
||||
return [], None
|
||||
# SNAPSHOT the config this stage may rewrite (§61 constraint). NOT jfb.revert(): that does a
|
||||
# wholesale `git checkout -- config/…`, which is correct for jtbl_family_bank's one-function-
|
||||
# at-a-time flow but WRONG here — it discards any PREVIOUSLY-banked-but-uncommitted carve in the
|
||||
# same overlay, leaving that bank's source with no subseg to live in (byte-witnessed: it stripped
|
||||
# func_80135A4C's carve while keeping its banked region file -> `undefined reference to
|
||||
# func_80136C90` at link). An inverse/wholesale undo cannot know what it did not do.
|
||||
keep = jfb.region_files(binary)
|
||||
snap = {}
|
||||
for f in (os.path.join(REPO, f"config/splat.{binary}.yaml"),
|
||||
os.path.join(REPO, "config/overlays.mk")):
|
||||
try:
|
||||
snap[f] = open(f).read()
|
||||
except OSError:
|
||||
pass
|
||||
prepared = []
|
||||
for fn in todo:
|
||||
r = sh([PY, "tools/jtbl_carve.py", binary, "--func", fn], timeout=900)
|
||||
out = (r.stdout or "") + (r.stderr or "")
|
||||
if r.returncode and ("NON-CONTIGUOUS" in out or "do not fit the span" in out):
|
||||
# §8b lazy isolation, then re-extract so the carve reads the new subseg layout, then retry.
|
||||
# The DISTINCT "more rodata .align than pad specs" drift is NOT isolate-fixable and is
|
||||
# deliberately not retried here (it falls through as a carve failure).
|
||||
if sh([PY, "tools/jr_isolate_all.py", binary, "--only", fn], timeout=900).returncode:
|
||||
print(f"[gate] jtbl: isolate failed for {fn}", file=sys.stderr); continue
|
||||
if sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800).returncode:
|
||||
print(f"[gate] jtbl: extract after isolate failed for {fn}", file=sys.stderr); continue
|
||||
_corpus_reset()
|
||||
r = sh([PY, "tools/jtbl_carve.py", binary, "--func", fn], timeout=900)
|
||||
if r.returncode:
|
||||
last = ((r.stdout or "") + (r.stderr or "")).strip().splitlines()[-1:] or [""]
|
||||
print(f"[gate] jtbl: carve failed for {fn}: {last[0][:150]}", file=sys.stderr); continue
|
||||
prepared.append(fn)
|
||||
if prepared:
|
||||
# a CONFIG change needs a re-extract, not just a rebuild (the R22 corollary)
|
||||
sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800)
|
||||
_corpus_reset()
|
||||
print(f"[gate] jtbl: prepared {len(prepared)}/{len(todo)} table-bearing draft(s): "
|
||||
f"{', '.join(prepared)}")
|
||||
return prepared, (keep, snap)
|
||||
|
||||
|
||||
def _corpus_reset():
|
||||
"""Drop corpus caches after a config/extract change — the stub set and each stub's home TU
|
||||
have moved, and every later stage derives from them."""
|
||||
for f in (getattr(corpus, "stubs", None), getattr(corpus, "sig", None),
|
||||
getattr(corpus, "o0_sources", None), getattr(corpus, "symbols", None),
|
||||
getattr(corpus, "src_files", None)):
|
||||
if hasattr(f, "cache_clear"):
|
||||
f.cache_clear()
|
||||
|
||||
|
||||
def _gate1(binary, src, asm, out, good_sha, d, verified_out=None, failed_out=None):
|
||||
"""Whole-binary byte-gate (G3/P9, sole arbiter) on draft-dir d; return the verified func list.
|
||||
harvest_verify reads the CURRENT src as its baseline (so verified fns ACCUMULATE across calls —
|
||||
@@ -258,9 +358,17 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_
|
||||
# raise on a non-zero exit, so the try/except below never saw it. Hence the explicit rc check:
|
||||
# a pre-pass that quietly does nothing is indistinguishable from one that found nothing to do,
|
||||
# which is the whole failure mode this ladder exists to remove (R32).
|
||||
# jtbl stage FIRST: isolation rewrites which TU hosts the stub, and every later stage
|
||||
# (arity scan, harvest_verify's TU derivation) reads that layout.
|
||||
_jtbl_prepared, _jtbl_keep = [], None
|
||||
try:
|
||||
_jtbl_prepared, _jtbl_keep = _jtbl_prepare(binary, draft_fns)
|
||||
except Exception as e: # never let the pre-pass sink the gate
|
||||
print(f"[gate] jtbl stage skipped: {e}", file=sys.stderr)
|
||||
|
||||
_arity_rc = None
|
||||
_arity_snapshot = {}
|
||||
if draft_fns:
|
||||
if draft_fns and not os.environ.get("GATE_NO_ARITY"):
|
||||
# snapshot every file the pre-pass may touch, so the undo is a restore, not a re-derivation
|
||||
for _f in ([os.path.join(REPO, "src/shared/engine_core.h")]
|
||||
+ glob.glob(os.path.join(REPO, f"src/{binary}/{binary}*.c"))):
|
||||
@@ -294,6 +402,30 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_
|
||||
# Restoring the pre-pass snapshot and re-applying ONLY for the functions that banked is exact by
|
||||
# construction and cannot invent a signature.
|
||||
_unbanked = [f for f in draft_fns if f not in verified]
|
||||
|
||||
# jtbl undo: if NOTHING banked, restore this overlay's config + drop the region files THIS run
|
||||
# created (jfb.revert(keep_regions=...) keeps a previously-banked core's). If something DID bank,
|
||||
# the carve/isolation is load-bearing for it and must stay — so we only revert the all-fail case,
|
||||
# and a mixed batch keeps the config that the surviving banks need (the byte-gate already
|
||||
# reverted the losers' source).
|
||||
if _jtbl_prepared and not verified and _jtbl_keep:
|
||||
try:
|
||||
import jtbl_family_bank as jfb
|
||||
_keep_regions, _snap = _jtbl_keep
|
||||
for _f, _txt in _snap.items(): # RESTORE, never inverse-transform (§61)
|
||||
with open(_f, "w") as _fh:
|
||||
_fh.write(_txt)
|
||||
for _rf in jfb.region_files(binary) - _keep_regions: # only what THIS run created
|
||||
try:
|
||||
os.remove(os.path.join(REPO, _rf))
|
||||
except OSError:
|
||||
pass
|
||||
sh(["make", "--no-print-directory", "extract", f"BINARY={binary}"], timeout=1800)
|
||||
_corpus_reset()
|
||||
print(f"[gate] jtbl: restored config for {len(_jtbl_prepared)} draft(s) (none banked)")
|
||||
except Exception as e:
|
||||
print(f"[gate] jtbl revert failed: {e}", file=sys.stderr)
|
||||
|
||||
if _unbanked and _arity_snapshot:
|
||||
try:
|
||||
for _f, _txt in _arity_snapshot.items():
|
||||
|
||||
Reference in New Issue
Block a user