From de9a550edbdf6cd5ce66b5d310af656a9f02f113 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Mon, 24 Aug 2026 01:00:59 -0600 Subject: [PATCH] =?UTF-8?q?fix(jtbl):=20refuse=20md=5F*/main=20in=20jtbl?= =?UTF-8?q?=5Fcarve=20=E2=80=94=20it=20corrupts=20those=20configs=20before?= =?UTF-8?q?=20erroring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independent reviews (ox design study + Fable validation) found parse_config does not implement its own documented contract on md_*/main: it can DELETE the c config line and corrupt the yaml on disk before failing. main also has no config/splat.main.yaml (it is splat.us.exe.yaml), which is the FileNotFoundError seen on every main jtbl target. Refuse loudly rather than corrupt quietly (R43). Lift only after parse_config is hardened and proven on two examples. Also: gate() stages into a per-run dir (a killed gate's 3,186 stale drafts survived into the next gate of the same tag and poisoned every group); reloc_filter no longer treats NOT-A-STUB as a pass (it means ALREADY BANKED — wave an staged 480 of them over source that already byte-matches). Designs recorded: docs/tool-designs/jtbl-island-split.md (ox) and -review.md (Fable): 14 CONFIRMED / 8 WRONG / 1 UNVERIFIABLE, verdict GO-WITH-CHANGES, and the correct fix is SMALLER than proposed — one inserted .rodata carve line + jr_isolate_all.py --only, no _pre piece and no ld_interleave leading mode. --- docs/tool-designs/jtbl-island-split-review.md | 138 ++++++++++++++ docs/tool-designs/jtbl-island-split.md | 176 ++++++++++++++++++ tools/jtbl_carve.py | 20 +- tools/ox_campaign.py | 98 ++++++++-- 4 files changed, 419 insertions(+), 13 deletions(-) create mode 100644 docs/tool-designs/jtbl-island-split-review.md create mode 100644 docs/tool-designs/jtbl-island-split.md diff --git a/docs/tool-designs/jtbl-island-split-review.md b/docs/tool-designs/jtbl-island-split-review.md new file mode 100644 index 000000000..801e56f58 --- /dev/null +++ b/docs/tool-designs/jtbl-island-split-review.md @@ -0,0 +1,138 @@ +# Review: jtbl leading-island split design (adversarial validation, 2026-08-24) + +**Reviewed:** `docs/tool-designs/jtbl-island-split.md` (ox-alpha, read-only agent). +**Method:** static verification only against the working tree — every tool source, the real +configs, the real Makefile, the installed splat 0.41.0 / spimdisasm 1.41.0 sources, the actual +generated `build/md_SC03_076/md_SC03_076.ld`, and the actual asm/src for md_SC03_076. No build was +run (gate/sweep processes were live: `pgrep -f 'sweep_parallel|gate_stage|gate_main'` non-empty). +The two pre-confirmed claims (`jtbl_carve.py:66`, `jr_isolate.py:83`) were not re-litigated. + +**Bottom line: GO-WITH-CHANGES — the carve idea is right and the byte layout works, but two of +the design's four steps are wrong (step 1 names a tool that provably cannot process the file; +step 2's `_pre` rename cannot build), and two are unnecessary (steps 3–4: splat's native linker +script already places `.rodata` ahead of `.text`). The correct implementation is *smaller* than +the design: one inserted config line + `jr_isolate_all.py --only`.** + +--- + +## 1. Claim-by-claim verdicts + +| # | Design claim | Verdict | Evidence | +|---|---|---|---| +| 1 | (a)1 `jr_isolate.py` contract "never edits a `.rodata` carve" (:16-17) | **CONFIRMED** | `tools/jr_isolate.py:16-17` | +| 2 | (a)1 isolation alone → "the island piece still binds to the OLD object name … the table bytes never move … reproduces the §8a duplicate-at-wrong-address disease" | **WRONG mechanism, right conclusion** | The island piece is a *dotted* `.rodata` paired by name to `md_SC03_076.c` (splat `segtypes/common/code.py:249-279` sibling pairing). Moving the fn to a jr c-file breaks migration (migration only targets the sibling file — `rodata.py:29-48` bails when `self.sibling is not None`), and the committed `md_SC03_076.c` has no `INCLUDE_RODATA` for the table — the bytes **vanish** from `md_SC03_076.o`; a matched-C table in `jr.o(.rodata)` has no ld entry and hits `/DISCARD/ { *(*); }`. Failure mode is *shrink/discard*, not duplicate. Either way isolation alone fails — the conclusion stands. | +| 3 | (a)2 `parse_config()` (:470-473) "defines the carvable region as the run of {data,.rodata} pieces AFTER the last `c` piece and exits 'no data-tail region' otherwise … structurally uncarvable" | **WRONG (and the truth is worse)** | "After the last c piece" is only the *comment* (`jtbl_carve.py:470`); the code takes **ALL** data/.rodata pieces (`:471`). The real config (which the design never saw) is `[0x0,.rodata]/[0x27C,c]/[0x35AC,data,tail]/[0xAAF0]` (`config/splat.md_SC03_076.yaml:96-99`) — so `parse_config` does **not** exit: it scopes the "region" 0x0..0xAAF0 *across the c line*, and `apply()`'s rewrite (`:749`, `lines[:lo]+region_lines+lines[hi:]`) would **delete `- [0x27C, c, md_SC03_076]`** — config corruption, written to disk (`:750`) *before* `set_overlays_var` dies on the missing overlays.mk anchor (`:766-768`; md vars live in `config/modules.mk:932+`). Reachable by any md fn whose table is NOT migrated (e.g. a jtbl in the 0x35AC tail): `migrated_tables` returns `[]` for it and `apply` proceeds. | +| 4 | (a)2 "apply() refuses in migrated_tables() before build_carve() is ever reached" | **CONFIRMED** (for migrated-table fns) | `jtbl_carve.py:727-746` | +| 5 | (a)3 `--order` branch (:101-115) hardcodes FILL/TEXT_START→text→empties→ordered→bss; nothing ahead of TEXT_START | **CONFIRMED** | `tools/ld_interleave.py:101-115` | +| 6 | Step 1: "`jr_isolate.py md_SC03_076 --func func_801F218C`, unchanged" | **WRONG — names a tool that cannot run on this file** | `jr_isolate.py:22-27` STATUS: scaffold "BLOCKED on split_src_region.py" (untouched since Phase 26, `git log`). Statically confirmed: `split_src_region.py` `trim` exits "cannot resolve address of item" (`:109-111`) on any top-level non-addressed item; `md_SC03_076.c:25-28` has exactly such a block (`extern void (*D_801F3468[])(void);` → `item_name` captures `void` → addr None). The production tool is **`jr_isolate_all.py --only func_801F218C`** (uses `overlay_src_split`; it is what `jtbl_family_bank.py:183` invokes). Its region model differs (cut *before* the jr fn; **no `_after` piece** — the jr object runs to the next boundary), and its name matches the design's: `md_SC03_076_jr_801F218C` (`jr_isolate_all.py:194-195`, `%08X`). | +| 7 | Step 2: replace `[0x0,.rodata,md_SC03_076]` with `[0x0,.rodata,md_SC03_076_pre]` + `[0x268,.rodata,md_SC03_076_jr_801F218C]` | **HALF-WRONG — the `_pre` rename cannot build** | The carve line (offset 0x268, name = jr c-subseg) is correct — splat pairs dotted `.rodata` to the same-named c subseg regardless of position (`code.py:249-279`; `readonly_before` exists precisely for rodata-first layouts, and md's leading island already migrates daily). But a dotted `.rodata` named `md_SC03_076_pre` has **no sibling c subseg** → splat resolves it to an "Implied C file" `src/md_SC03_076/md_SC03_076_pre.c` (`data.py:23-30`) that nothing creates → the ld references a never-built object. Worse, the rename **unbinds the island 0x0..0x268 from `md_SC03_076.c`**, breaking migration for the other five stub-owned jtbls (`jtbl_801EF5AC/5E4/61C/654/67C/6A4`) and orphaning the two committed `INCLUDE_RODATA` blobs (`md_SC03_076.c:131,133`) and compiled consts (`D_801EF6C0/6C8`, `:447,:502`). **Fix: keep the existing `- [0x0, .rodata, md_SC03_076]` line untouched and only INSERT the carve line** — its extent shrinks automatically (end = next piece's offset). The "split" is one inserted line. | +| 8 | Step 2 ground facts: carve at file 0x268; 5 words end at 0x27C == code start; `sltiu 0x5`; 0x801EF6D0 %8==0; single-table ⇒ no `JTBL_PADS`; no post piece (table is the island's last occupant) | **CONFIRMED** | `asm/md_SC03_076/nonmatchings/md_SC03_076/func_801F218C.s` (`/* 268 801EF6D0 */`, five `.word .L801F21C8/22C4/2238/2278/22A4`, `sltiu $v0,$v1,0x5`); `config/splat.md_SC03_076.yaml:97` (`[0x27C, c, …]`); `jtbl_carve.py:37` (single-table → no var) | +| 9 | Step 2: "read the migrated dlabels the way migrated_tables() does at :709-718" | **CONFIRMED** | `jtbl_carve.py:709-716` | +| 10 | Step 3: ld_interleave needs a `--leading` mode; "A leading island needs [.rodata pieces] BEFORE _TEXT_START; no existing mode does it" | **TRUE of ld_interleave, but the step is UNNECESSARY** | ld_interleave **never runs for md binaries** (no `_JTBL_INTERLEAVE` var; Makefile:648-649 gate) and never needs to: splat's native script is section-major with `.rodata` FIRST per `section_order` (verified in the actual `build/md_SC03_076/md_SC03_076.ld`: `RODATA_START` precedes `TEXT_START`), and within the `.rodata` group entries follow yaml/subsegment order (`linker_entry.py:238-270` buckets by section preserving order). Post-split the group is `md_SC03_076.o(.rodata)` then `md_SC03_076_jr_801F218C.o(.rodata)` back-to-back — exactly address order, tight (no intra-group ALIGN; `SUBALIGN(2)` caps input alignment). Auto-generated `.rodata` entries for carve-less c files (`auto_link_sections` default, `options.py:521-522`) insert *after* the declared ones (`code.py:84-140`) and are empty — byte-neutral. | +| 11 | Step 4: wire via `_JTBL_INTERLEAVE` / new `_JTBL_ISLAND`; "read the real Makefile + config/overlays.mk" | **PARTLY WRONG / unnecessary** | No interleave var is needed at all (see #10). And md per-binary vars live in **`config/modules.mk`** (`:932-…`), not overlays.mk — every mk-writing path in jtbl_carve hardcodes overlays.mk (`:114, :759, :782, :829-831`), so any future extension (e.g. a multi-table md span needing `JTBL_PADS`) must generalize the mk target first. The §8a trailing-comment/`$(strip)` gotcha is real (Makefile:122-125). | +| 12 | Ordering constraint (ii): same-subseg carves must abut (gap 0 / verified-zero gap 4) | **CONFIRMED as existing law** | `jtbl_carve.py:528-558` — but see §4 "pre-object contiguity" for the corollary the design missed. | +| 13 | Ordering constraint (iii): jtbl_rodata_pads must never see pre/post; refuses `.include` content | **CONFIRMED** | `tools/jtbl_rodata_pads.py:74-78`; pads armed per-object via target var (Makefile:689) | +| 14 | (b) crash chain `apply→migrated_tables→func_jtbls→func_subseg→code_pieces→overlay_vram_base→open(cfg_path)`; `config/splat.main.yaml` absent; per-binary naming a Phase-9/10 addition | **CONFIRMED** | Chain verified (`jtbl_carve.py:727→703-706→216→144-146→127→66`); `main_SPLAT_YAML := config/splat.us.exe.yaml` (Makefile:74); `config/splat.main.yaml` does not exist; Phase-9 binaries block (Makefile:39-47). The open is at `:66` (the design's verbatim text says :67-68; its own header says :66 — trivial drift). | +| 15 | (b) main roots are repo-level; the six fs helpers hardcode `asm/{ov}` (:162,:217,:233,:244,:260,:286) | **CONFIRMED** | `main_ASM_DIR := asm` (Makefile:85); `asm/main/` does not exist, main's tree is `asm/nonmatchings` at repo root; all six line numbers exact. | +| 16 | (b) "Whether main is then CARVEABLE … parse_config will answer that honestly once paths work" | **WRONG — the fix list is incomplete** | Main's region-line math happens to be safe (`splat.us.exe.yaml:206-209`: the data/rodata pieces are contiguous before the EOF line, no c lines between). But the **base/offset model breaks**: `overlay_vram_base` greps the first `vram:` = 0x80010000 (`:65`), while main's pieces are FILE offsets with `start: 0x800` — the correct flat base is 0x8000F800 (`main_VRAM_BASE`, Makefile:81). Every vram↔offset conversion is off by 0x800; `payload_word` **silently reads bytes 0x800 early** (zero-pad checks, gap checks, `spec_from_starts` all corrupted, no error raised). An `asm_root()`/`cfg_path()` fix alone ships a silently wrong tool for main. | +| 17 | (c) stub-state 43,760 bytes | **CONFIRMED** | `extracted/retail/SC03.CD.dir/FILE_076.dir/2.1` is 43,760 B (= EOF marker 0xAAF0), sha1 `9a165e…` == `config/check.md_SC03_076.sha` == yaml sha1. | +| 18 | (c) diseased baseline "43,768 (+8), first diff 0x144" for this experiment | **UNVERIFIABLE / misattributed** | Those numbers were byte-measured on **func_801F0F28** (`jtbl_carve.py:733`), not func_801F218C. Immaterial to the control's validity (pass = SHA-equality), but the design presents another function's disease signature as this experiment's baseline. (0x144 is the island offset of `jtbl_801EF5AC` — the first text-pointer word — consistent with a +8 text shift.) | +| 19 | (c) discriminators: jr object `.rodata sh_size == 0x14`; md object shrinks by 0x14; the five words at 0x801EF6D0; SHA gate the arbiter, never rtu_match | **CONFIRMED (one nit)** | Shrink 0x27C→0x268 ✓; the five words ✓ (from the .s); SHA-as-arbiter is project law. Nit: sh_size 0x14 holds because the table sits at *section offset 0* (`.align 3` is a no-op there), not because its final vram is 8-aligned — same number, misplaced reason. | +| 20 | (d)1 `migrated_tables()`'s "same address, by construction" premise (:684-692) refuted by apply()'s own error; stale doc must not survive | **CONFIRMED — real defect** | Docstring `:690-691` + `:699` ("a no-op here, NOT a refusal") vs the byte-measured refusal `apply():736-739`. | +| 21 | (d)2 hardcoded 0x80100000–0x801D0000 window (:306, :434) vacuous for md text | **CONFIRMED — and it is worse than stated** | Lines exact. md text starts 0x801EF6E4 (vram 0x801EF468 + 0x27C); every entry (0x801F21C8…) is outside the window. Consequences: `_continuation_words` can never absorb (under-carve risk, conservative), and the over-span clamp's *refusal* guard can never fire — for md the clamp would **silently drop real entries** (the dangerous direction). Also missed by the design: the window excludes **main's** text too (0x80010000..0x800629DC < 0x80100000). Derive from the binary's TEXT_LO/TEXT_HI (already in the Makefile per binary). | +| 22 | (d)3 `--order` docstring "overstated contract" | **PARTLY WRONG (minor)** | The same help text states "text -> [these] -> bss" explicitly (`ld_interleave.py:36`); only the clause "Generalises the … sandwich to N pieces" (`:40`) could mislead if read alone. A doc nit at most. | +| 23 | (d)4 parse_config's "after the last c piece" as the structural gap | **CONFIRMED as a gap, mischaracterized** | See #3: the comment is not implemented; the behavior is corrupt-on-rewrite, not refuse. Hardening it (implement the comment: anchor the region strictly after the last `c` piece, refuse anything else) is a **prerequisite**, not a nice-to-have. | +| 24 | (d)5 nested-root helpers + cfg_path wrong for main | **CONFIRMED** | Dup of #14/#15. | +| 25 | Config shape inferred as `[0x0,.rodata]+[0x27C,c]` (residual unknown, flagged) | **INCOMPLETE — honestly flagged, but it mattered** | The real config also has `[0x35AC, data, tail]` + EOF `[0xAAF0]`; that extra piece is what flips claim #3 from "refuses" to "corrupts". | +| 26 | Typo: "`md_SC03_077_after_`" | WRONG (trivial) | Should be `md_SC03_076_after_…`; and with `jr_isolate_all` no `_after` piece exists at all. | + +**Tally: 14 CONFIRMED, 8 WRONG/PARTLY-WRONG, 1 UNVERIFIABLE, 1 honestly-flagged-incomplete** (plus 1 trivial typo). + +--- + +## 2. Verdict on the mechanism: GO-WITH-CHANGES + +The core idea — give the matched jr function its own code subseg and bind its table span to a +same-named dotted `.rodata` piece — is sound, and the byte layout works. But **as written the +design does not build**: step 1's tool dies on the source file, step 2's `_pre` object has no +producer, and steps 3–4 solve a problem splat does not have. The working implementation is: + +1. `tools/jr_isolate_all.py md_SC03_076 --only func_801F218C` (production isolate; handles the + md .c structure via `overlay_src_split`; writes the region files itself; no mid-flow extract). +2. Insert **one line** into `config/splat.md_SC03_076.yaml`: + `- [0x268, .rodata, md_SC03_076_jr_801F218C]` between the existing `[0x0, .rodata, md_SC03_076]` + (which stays untouched — its extent shrinks automatically) and the c pieces. +3. `make extract BINARY=md_SC03_076` → `make build` (stage 1, stub state, must be SHA-green), + then splice the matched C into the jr file → `make build` (stage 2, must be SHA-green). + No ld_interleave, no make variable: splat's native section-major script (rodata group first, + entries in yaml order, `SUBALIGN(2)`) already produces the exact address-ordered layout. + +Why this is byte-plausible (verified statically): the generated `.ld` places +`md_SC03_076.o(.rodata)` then `md_SC03_076_jr_801F218C.o(.rodata)` back-to-back at 0x801EF468; +md.o's rodata = includes + consts + the five remaining stub tables = 0x268 bytes; jr.o's rodata = +the migrated table (stub state) or the compiled switch table (matched state) = 0x14 bytes; text +follows at 0x801EF6E4. Same-name sibling pairing for the carve is the proven §8a mechanism, and +the leading position is already exercised daily by the module's own 0x0 piece. + +## 3. The negative control (design §c) + +**Discriminating, with caveats.** The pass condition — whole-binary SHA1 equality with the +43,760-byte payload at both stages — is the project's own arbiter and cannot pass for a wrong +reason: any migration failure, duplicate, boundary drift, or discard changes bytes. The +plausible-failure list (table twice, boundary off, diff moved) all fail SHA. Caveats: + +- The "diseased baseline 43,768/+8/first diff 0x144" was measured on **func_801F0F28**, not the + control function (claim #18). Do not treat reproducing that exact signature as meaningful. +- Stage 2 conflates two hypotheses: a red stage 2 could mean the mechanism is broken **or** the + C match is wrong. Run stage 2 only with a candidate that already byte-matched elsewhere in the + family, or accept the ambiguity. +- With `jr_isolate_all` there is no `_after` object; the jr object spans 0x2D24..0x35AC (the jr fn + plus every trailing function). The objdump discriminator still holds — the census below shows + the trailing functions contribute zero rodata — but that census is part of the control's + validity and must be re-run per target. + +## 4. What the design missed (would break an implementation or the rollout) + +1. **The `_pre` object has no producer** (claim #7) — the single biggest defect; the design's + own config sketch cannot link. +2. **jtbl_carve corrupts md configs instead of refusing** when a table is not migrated (claim #3). + Harden `parse_config` (implement its own "after the last c piece" comment, refuse otherwise) + *before* any md work — and before pointing jtbl_carve at main. +3. **Migration re-homing precheck.** Isolation moves every function ≥ the cut into a different + c file; any moved function that owns a *leading-island* table, or matched C that emits const + rodata, breaks the island reconstruction. Safe for this target — verified census: all six + island jtbl owners (`func_801EFBB4/801F0210/801F0734/801F0A9C/801F0F28/801F218C`) are at or + before the cut, and no const/`INCLUDE_RODATA` exists after `md_SC03_076.c:634` — but this is a + per-target precondition, not a given. It belongs in the tool as a loud refusal. +4. **Pre-object contiguity wall.** `md_SC03_076.o` supplies ONE contiguous rodata run, so only + tables adjacent to the island END (or chains growing backwards from it) are carvable cheaply. + Example: carving `jtbl_801EF6A4` (@0x23C — func_801F0F28, the very function the §154-A refusal + was measured on) leaves md.o needing 0x0..0x23C *and* 0x258..0x268 (the consts) — impossible + for one object; the const-owning matched functions must be isolated too. The design's "a post + piece appears once a LATER table in the island is carved" glosses over this: the post piece + would have to be a *second* md_SC03_076 rodata entry, which GNU ld cannot split. The rollout + therefore unblocks fewer module jr functions than "every jr member of every family" — + plan for cascading isolations or accept end-adjacent-only. +5. **Pipeline integration.** Family waves reach carving through `jtbl_family_bank.py` → + `jtbl_carve.py` (`:222`), which refuses §154-A binaries. Unless the island split is reachable + from that path (and the refusal text updated), module jr functions stay wave-unbankable even + after the mechanism exists. +6. **mk-file targeting.** All of jtbl_carve's overlays.mk writers (`set_overlays_var:759`, + `set_pads_vars:782`, `revert:829`, `current_pads_specs:114`) and `jr_isolate_all`'s + `repoint_overlays_mk` hardcode `config/overlays.mk`; md vars live in `config/modules.mk`. + Dormant today (no md carve vars needed), a landmine for multi-table md spans later. +7. **Main's base/offset model** (claim #16): the path fix alone leaves `payload_word` silently + reading 0x800 early on main. Add a per-binary (vram_base, file_base) pair, not just paths. + +## 5. What could not be verified without a build + +- Stage-1 byte-neutrality end-to-end (splat's actual re-extract output for the split config, and + spimdisasm 1.41.0 writing the migrated table into the jr-subseg stub .s). Inferred from the + §8a overlay precedent (134× proven) plus the module's own daily leading-piece migration; the + linker-side ordering was verified against the real generated .ld and splat source, not a run. +- `overlay_src_split.partition()` on md-class .c files (static read says the model fits; the + design's chosen `split_src_region.py` path was statically *dis*proven, which is stronger). +- The exact disease signature (+8 / 0x144) for func_801F218C specifically (measured value exists + only for func_801F0F28). +- Whether any md_* binary actually has a tail-region jtbl (the trigger for hazard §4.2) — the + hazard is structural regardless. diff --git a/docs/tool-designs/jtbl-island-split.md b/docs/tool-designs/jtbl-island-split.md new file mode 100644 index 000000000..b3b730337 --- /dev/null +++ b/docs/tool-designs/jtbl-island-split.md @@ -0,0 +1,176 @@ +# jtbl leading-island split — design study (ox-alpha, P31 S58, 2026-08-24) + +**Provenance:** produced by `stealth/ox-alpha` under `tools/api_agent.py` with a read-only surface +widened to `config,tools,Makefile` (`EXTRA_READABLE`). It read the target `.s`, the full sources of +`jtbl_carve` / `jr_isolate` / `ld_interleave` / `jtbl_rodata_pads` / `jtbl_family_bank`, the Makefile +excerpt, and cookbook §8 / §8a / §8a-pad / §8e / §154. + +**Status: DESIGN, NOT VERIFIED.** ox cannot run `make`, so nothing here has been executed. Two of its +claims were independently checked against the source before this file was written, and both held: + +* `jtbl_carve.py:66` — `open(cfg_path(ov))` is unguarded and `cfg_path` hardwires + `config/splat.{ov}.yaml`; main's config is `config/splat.us.exe.yaml`, so main raises + `FileNotFoundError`. This is the crash observed on `SaveLoadRoutine`, `func_8003388C`, + `func_80035270` and `StreamLoadStateMachine`. +* `jr_isolate.py:83` — `line_re` matches only `- [0xADDR, c, SUB]` (code subsegs), so isolation + cannot move a `.rodata` island. `jtbl_carve`'s own error text proposed a fix its tools cannot + perform. + +The remainder is the agent's verbatim deliverable. + +--- + +// TOOLING PLAN — jtbl leading-island split (md_* modules) + main crash fix +// Ground truth read: asm/md_SC03_076/nonmatchings/md_SC03_076/func_801F218C.s, +// .run/bakeoff/toolwork/{jtbl_carve,jr_isolate,ld_interleave,jtbl_rodata_pads, +// jtbl_family_bank}.py, Makefile.excerpt, cookbook §8/§8a/§8a-pad/§8e/§154. +// config/*.yaml and the full Makefile are OUTSIDE my read surface; where that +// matters I name the file an engineer must read. + +// ───────────────────────────────────────────────────────────────────────────── +// (a) LEADING-ISLAND: evaluate the error-text proposal (jtbl_carve.py:741-743) +// ───────────────────────────────────────────────────────────────────────────── +// VERDICT: HALF-RIGHT. "Isolate the jr function into its own code subseg" works +// unchanged; "so its .rodata is a separate OBJECT" is FALSE as stated, and +// ld_interleave cannot do the ordering it promises. Evidence from source: +// +// 1. jr_isolate.py rewrites ONLY `- [off, c, SUB]` lines (line_re :83 demands +// `,\s*c,\s*`) and its contract (:16-17) is "never edits a `.rodata` carve". +// On md_SC03_076 (config shape quoted in apply()'s own error: `- [0x0, +// .rodata, md_SC03_076]` + `- [0x27C, c, md_SC03_076]`) it splits just the +// code piece into [c, md_SC03_076] + [c, md_SC03_076_jr_] + +// [c, md_SC03_077_after_]. The island piece still binds to the OLD +// object name; the new jr object has NO .rodata input; the table bytes never +// move. Isolation alone reproduces the §8a duplicate-at-wrong-address disease +// in a new costume. +// 2. jtbl_carve.py cannot supply the missing half: parse_config() (:470-473) +// defines the carvable region as the run of {data,.rodata} pieces AFTER the +// last `c` piece and exits "no data-tail region" otherwise. A module's island +// sits BEFORE its code — structurally uncarvable by this tool. (apply() +// refuses in migrated_tables() before build_carve() is ever reached.) +// 3. ld_interleave.py --order cannot express the placement: that branch +// (:101-115) hardcodes FILL/TEXT_START -> text -> empties -> ordered pieces +// -> bss. Nothing can be emitted ahead of {PREFIX}_TEXT_START. A leading +// island needs [.rodata pieces] BEFORE _TEXT_START; no existing mode does it. +// +// WHAT IS RIGHT (implementation): +// Step 1 — jr_isolate.py md_SC03_076 --func func_801F218C, unchanged. Gives +// the fn its own code object; that object's NAME is what the carve piece must +// carry (the §8a dotted-subseg law). +// Step 2 — NEW: split the LEADING `.rodata` piece around the fn's MIGRATED +// table(s). Replace +// - [0x0, .rodata, md_SC03_076] +// with +// - [0x0, .rodata, md_SC03_076_pre] +// - [0x268, .rodata, md_SC03_076_jr_801F218C] +// (no post piece HERE: the table is the island's LAST occupant — its .s header +// says file offset 0x268, 5 words end at 0x27C == the code piece start; a post +// piece appears once a LATER table in the island is carved). Because the middle +// piece's NAME equals the fn's code subseg, spimdisasm migrates the table into +// the jr object's .s — the exact §8a mechanism aimed at a leading piece. +// Span derivation reuses jtbl_range()'s oracles, but sourced from the FN'S OWN +// .s (the dlabels are migrated there — read them the way migrated_tables() +// already does at :709-718), bounded by the function's own `sltiu` (here +// `sltiu $v0,$v1,0x5` = 5 entries = 0x14 bytes; jtbl_801EF6D0 %8==0 so no +// trailing-pad trim and NO JTBL_PADS line — single-table carves get none, §8e). +// Step 3 — extend ld_interleave.py with a leading mode (new flag, e.g. +// `--leading `): emit the ordered .rodata piece lines FIRST, +// then _TEXT_START/text/_TEXT_END, then bss — keeping every START/END/SIZE +// symbol splat defines, with RODATA_START/END placed ahead of TEXT_START. +// Tight packing preserved (. = ALIGN(.,4) between pieces; SUBALIGN(2) in the +// splat script caps external alignment — §8e law 3). +// Step 4 — config: the splat yaml gains the two/three piece lines above; wire +// the interleave through the existing per-binary var mechanism (extend +// `_JTBL_INTERLEAVE` or add `_JTBL_ISLAND := leading`). Mind the §8a +// GOTCHA: no trailing #comment on the var line, $(strip) it. NOTE: the +// Makefile's extract-stage wiring of JTBL_INTERLEAVE is NOT in +// Makefile.excerpt — read the real Makefile + config/overlays.mk before +// editing (also confirms whether md_* binaries already have an output-section +// hook; ld_interleave takes `--section .`). +// ORDERING CONSTRAINTS: (i) all input-section lines must be emitted in +// ascending ADDRESS order — island pieces (0x0..0x27C) precede text (0x27C); +// (ii) multiple matched fns follow the SAME contiguity law as overlays — +// same-code-subseg carves must abut (gap 0, or gap 4 with a verifiably-zero +// payload word, else NON-CONTIGUOUS → isolate that fn too); (iii) +// jtbl_rodata_pads applies ONLY to the carve piece (pure $L/.word content) — +// it must NEVER see pre/post, it refuses mixed include-rodata content (the +// very 'unexpected rodata content .include "..."' error quoted in apply()). + +// ───────────────────────────────────────────────────────────────────────────── +// (b) main: the crash +// ───────────────────────────────────────────────────────────────────────────── +// `jtbl_carve.py main --func ` → apply() → migrated_tables() → func_jtbls() +// → func_subseg() → code_pieces() → overlay_vram_base() → +// jtbl_carve.py:67-68 `txt = open(cfg_path(ov)).read()` +// with cfg_path() (:60-61) hardwired to `config/splat.{ov}.yaml`. +// CAUSE: every other failure on the entry path is a CLEAN sys.exit (missing .s, +// missing jtbl, no data-tail region); a Python TRACEBACK requires an unguarded +// exception, and the only unguarded I/O on that path is this open(). main is the +// root binary and predates the per-binary config naming (Makefile.excerpt :6-9: +// per-binary aliases are a Phase-9/10 addition), so `config/splat.main.yaml` +// does not exist → FileNotFoundError traceback. CONFIRM BY READING (outside my +// surface): the Makefile "Binaries" block — what is main's SPLAT_YAML path? +// SECOND-ORDER (bites immediately after the path fix): Makefile.excerpt :106-108 +// says main's roots are REPO-LEVEL asm/ + src/ (siblings nest at asm//), +// but every fs-touching helper hardcodes the nested root — all_data_labels :233, +// jtbl_words :244, _label_words :260, _sltiu_bounds :286, overlay_jtbl_addrs +// :162, func_jtbls :217 all build `asm/{ov}/...` ⇒ for ov="main" they resolve to +// nonexistent asm/main/... and turn real data into "not found" exits. FIX: an +// asm_root(ov) helper beside cfg_path(ov); both return repo-level paths when +// ov == "main". (Whether main is then CARVEABLE at all depends on its config's +// piece layout — parse_config will answer that honestly once paths work.) + +// ───────────────────────────────────────────────────────────────────────────── +// (c) NEGATIVE CONTROL +// ───────────────────────────────────────────────────────────────────────────── +// Function: func_801F218C, binary md_SC03_076 (this task's target: 83 ins, +// jtbl_801EF6D0 @0x801EF6D0, sltiu bound 5, island span 0x268..0x27C). +// Two-stage control: +// Stage 1 (split-only, fn STILL A STUB): after jr_isolate + island split + +// re-extract, `make build BINARY=md_SC03_076` must be BYTE-IDENTICAL — size +// 43,760, SHA1 == payload. Proves the split itself byte-neutral. +// Stage 2 (splice the matched C, rebuild): STILL 43,760 / SHA-equal. The +// diseased baseline is the measured 43,768 (+8) with first diff at 0x144. +// Byte-level discriminators of success vs plausible-looking failure: +// - objdump -h build/src/md_SC03_076/md_SC03_076_jr_801F218C.o: .rodata +// sh_size == 0x14 (exactly 5 words, NO align pad — 0x...6D0 is 8-aligned); +// md_SC03_076.o's .rodata shrinks by exactly 0x14. If BOTH objects still +// carry table bytes, the migration did not happen (name mismatch). +// - Final image: bytes from 0x144 onward IDENTICAL to payload (the +8 +// signature GONE, not moved); the five words 801F21C8/801F22C4/801F2238/ +// 801F2278/801F22A4 sit at vram 0x801EF6D0. +// - A build that links at 43,760 but with the first diff merely MOVED (pre/ +// post boundary off by a pad word, or table present twice) is FAILURE. +// Arbiter = whole-binary SHA gate, NEVER rtu_match (§8a: masks relocs and +// excludes jtbl rodata — false-MATCH class). + +// ───────────────────────────────────────────────────────────────────────────── +// (d) WRONG rather than MISSING in the existing tools +// ───────────────────────────────────────────────────────────────────────────── +// 1. migrated_tables()'s premise (jtbl_carve.py:684-692: matching makes the C +// "emit the table into the same object's .rodata, AT THE SAME ADDRESS, by +// construction") is refuted by apply()'s own measured error text (+8 bytes, +// first diff 0x144): pruning the .s drops the table out of the include chain +// and cc1 re-emits it at the END of the object's .rodata. The REFUSAL is +// correct; the no-op rationale is byte-measured wrong and must not survive +// as documentation. +// 2. Hardcoded overlay VRAM window `0x80100000 <= w < 0x801D0000` in +// _continuation_words (:306) and the over-span clamp (:434): md_SC03_076's +// text lives at 0x801EF468+ — EVERY real entry (e.g. 0x801F21C8) falls +// OUTSIDE the window, so "surplus word looks like a code address" is vacuous +// and the continuation/clamp authorization gates misfire for exactly the +// module class this work enables. Derive the window from the binary's text +// bounds (config), not constants. +// 3. ld_interleave --order's docstring claims it "generalises the sandwich to N +// pieces"; it generalises only TAIL islands (text-first is structural, +// :101-115). Overstated contract — fix the doc or add the leading mode. +// 4. parse_config's "data region = after the last `c` piece" — the structural +// gap that makes (a) necessary rather than optional. +// 5. Nested-root path helpers + cfg_path for ov=="main" ((b)) — wrong default +// for the root binary, not merely absent. + +// Residual unknowns + the files that answer them: config/splat.md_SC03_076.yaml +// (exact piece list/indent/trailing-bin — shape above taken from apply()'s +// error text + the .s's own /* 268 */ offset), config/splat.main.yaml existence +// and the Makefile "Binaries"/extract-interleave wiring. All outside src/, +// asm/, docs/, include/, .run/bakeoff/. diff --git a/tools/jtbl_carve.py b/tools/jtbl_carve.py index 40403f117..a267251f3 100644 --- a/tools/jtbl_carve.py +++ b/tools/jtbl_carve.py @@ -58,7 +58,25 @@ EOF_RE = re.compile(r"^\s*- \[(0x[0-9A-Fa-f]+)\]\s*(?:#.*)?$") def cfg_path(ov): - return os.path.join(REPO, f"config/splat.{ov}.yaml") + # HARD REFUSAL for binary classes this tool cannot handle. Two independent reviews (ox design + # study + Fable validation, P31 S58) established that on md_* and main jtbl_carve does not + # merely fail — `parse_config` does not implement its own documented contract, so it can DELETE + # the `c` config line and CORRUPT the yaml on disk before it errors out. main additionally has + # no `config/splat.main.yaml` at all (it is `splat.us.exe.yaml`), so the unguarded open() below + # raised FileNotFoundError on every main jtbl target. + # + # Refusing loudly beats corrupting quietly (R43). Lift this only once parse_config is hardened + # and the fix is proven on two examples — see docs/tool-designs/jtbl-island-split-review.md, + # whose verdict is that the real change is ONE inserted `.rodata` carve line plus + # jr_isolate_all.py --only, NOT the _pre split or an ld_interleave leading mode. + if ov == "main" or ov.startswith("md_"): + sys.exit(f"jtbl_carve: REFUSING {ov} — md_*/main configs are corrupted by parse_config " + f"before it errors (see docs/tool-designs/jtbl-island-split-review.md). " + f"Overlays (ov_*) are supported.") + p = os.path.join(REPO, f"config/splat.{ov}.yaml") + if not os.path.exists(p): + sys.exit(f"jtbl_carve: no splat config at {p} for binary {ov!r}") + return p def overlay_vram_base(ov): diff --git a/tools/ox_campaign.py b/tools/ox_campaign.py index faec5899d..833905286 100644 --- a/tools/ox_campaign.py +++ b/tools/ox_campaign.py @@ -296,7 +296,15 @@ def reloc_filter(tag, drafts, cards_path): return batch, {"(reloc_identity produced no output — gating unfiltered)": len(batch)} status = {r["fn"]: r.get("status") for r in res} counts = collections.Counter(status.values()) - keep = [b for b in batch if status.get(b["fn"]) in ("AGREE", "NOT-A-STUB")] + # NOT-A-STUB IS NOT A PASS — it means reloc_identity found nothing to check because the + # function is ALREADY BANKED. Gating those re-stages a draft body over source that already + # byte-matches: pure waste at best, and at worst it perturbs a banked function inside a group + # and takes the group's genuinely-new drafts down with it. Wave `an` carried 480 NOT-A-STUB of + # 697 "gated" and banked 0. Only AGREE is a pass. (R43: refuse input the step cannot use.) + keep = [b for b in batch if status.get(b["fn"]) == "AGREE"] + n_banked_already = sum(1 for b in batch if status.get(b["fn"]) == "NOT-A-STUB") + if n_banked_already: + counts = dict(counts, _already_banked_excluded=n_banked_already) return keep, dict(counts) @@ -326,7 +334,7 @@ def gate_main_batch(tag, mains): return len(banked), banked -def gate(tag, keep, jobs): +def gate(tag, keep, jobs, run_id=None): # Any main drafts that slipped through (an older wave's cards) are PARKED for the periodic main # batch rather than gated inline — see draw_wave's note on why main is off the critical path. mains = [k for k in keep if k["binary"] == "main"] @@ -338,7 +346,12 @@ def gate(tag, keep, jobs): log(f" main: {len(mains)} drafts PARKED to .run/main_queue/{tag}.json " f"(gate them with tools/gate_main.py when no wave is waiting)") mains = [] - d = f".run/sweep_{tag}" + # PER-RUN staging dir. Reusing `.run/sweep_` let a KILLED gate's staged drafts survive + # into the next gate of the same tag: measured wave `an` staging 697 drafts into a directory + # that held 3,186, so every per-binary group carried stale bodies and failed as a group — + # 208 reloc-AGREE drafts banked 0. rmtree is not enough when a crashed or concurrent run may + # have written the same path; a unique path makes the contamination impossible, not unlikely. + d = f".run/sweep_{tag}" + (f".{run_id}" if run_id else "") shutil.rmtree(d, ignore_errors=True) for b in keep: bd = os.path.join(d, b["binary"]) @@ -448,6 +461,32 @@ def _drawlock(): return fh +def _drawlock_nb(): + """Non-blocking draw lock. Returns None if a gate holds it — the caller must NOT wait.""" + os.makedirs(".run/auto", exist_ok=True) + fh = open(DRAWLOCK, "w") + try: + fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB) + return fh + except BlockingIOError: + fh.close() + return None + + +def _predrawn(skip_tag): + """A wave whose cards exist but which has never been drafted — free work while a gate runs.""" + for p in sorted(glob.glob(".run/wave_??_cards.json"), key=os.path.getmtime): + t = os.path.basename(p)[5:7] + if t == skip_tag: + continue + if os.path.exists(f"{READY}/{t}.json"): # already drafted and queued + continue + if glob.glob(f".run/wave_{t}/shard*/*.c"): # already has drafts + continue + return t, p + return None + + def run_drafter(a): """THE LANE THAT MUST NEVER STOP. Draw -> shard -> draft -> publish a ready marker. Forever. @@ -474,11 +513,17 @@ def run_drafter(a): log(f"drafter: resuming rotation at index {w} -> lane " f"{lane_specs[w % len(lane_specs)]['name']}, band {bands[w % len(bands)]}") while not os.path.exists(STOP): - # Bound the run-ahead: drawing 40 waves of cards while the gater lags would strand them all - # as "already-waved" and starve later waves of candidates. - if len(glob.glob(f"{READY}/*.json")) >= a.queue_depth: - time.sleep(30) - continue + # Bound the run-ahead on DRAWING only. Drawing 40 waves while the gater lags would strand + # them all as "already-waved"; DRAFTING a wave whose cards already exist strands nothing. + # Conflating the two stopped the fleet dead: with queue_depth=2 and two waves queued, the + # drafter slept while `aa` and `ai` sat pre-drawn and undrafted (measured P31 S58). + backlogged = len(glob.glob(f"{READY}/*.json")) >= a.queue_depth + if backlogged: + pre = _predrawn(None) + if not pre: + log(f" gater backlogged ({a.queue_depth}+ waves) and nothing pre-drawn — waiting") + time.sleep(60) + continue cr = credits_left() if cr is not None and cr < a.credit_floor: log(f"drafter: credit floor (${cr:.2f}) — stopping"); break @@ -490,9 +535,38 @@ def run_drafter(a): w += 1 log(f"=== DRAFT {tag} · lane {lane['name']} · band {band[0]}-{band[1]} · " f"{a.workers} workers ===") - lk = _drawlock() - cards = draw_wave(tag, a.cards_per_wave, band, lane.get("levers")) - lk.close() + # NON-BLOCKING draw. The gater holds this lock for its WHOLE gate (substituted drafts make + # corpus.stubs() lie), so a blocking acquire here parks the entire drafting fleet for the + # duration of every gate — measured 7 minutes idle with 0 agents while wave `an` gated, and + # it would recur on every wave. Drafting is the clock-limited resource; it must never wait + # on gating. If the lock is busy we draft a wave that was PRE-DRAWN earlier instead, and + # only sleep when there is genuinely nothing drawn to work on. + lk = None if backlogged else _drawlock_nb() + if lk is None: + pre = _predrawn(tag) + if pre: + tag, cards = pre + log(f" gate in progress — drafting PRE-DRAWN wave {tag} instead of waiting") + else: + log(" gate holds the draw lock and nothing is pre-drawn — waiting 30s") + time.sleep(30); continue + else: + cards = draw_wave(tag, a.cards_per_wave, band, lane.get("levers")) + lk.close() + # Draw the NEXT wave's cards too while the lock is ours, so the next gate cannot idle + # the fleet. Cheap (a few minutes of CPU) and it buys a whole gate's worth of drafting. + nxt = next((t for t in tags + if t not in {os.path.basename(q)[5:7] + for q in glob.glob(".run/wave_??_cards.json")}), None) + if nxt: + nlane = lane_specs[(w) % len(lane_specs)] + nband = bands[(w) % len(bands)] + lk2 = _drawlock_nb() + if lk2 is not None: + log(f" pre-drawing {nxt} (lane {nlane['name']}, band {nband[0]}-{nband[1]}) " + f"so the next gate cannot idle the fleet") + draw_wave(nxt, a.cards_per_wave, nband, nlane.get("levers")) + lk2.close() if not cards: log(f" {tag}: draw failed — retrying next cycle"); time.sleep(30); continue targets = shard_targets(tag, cards, a.workers) @@ -524,7 +598,7 @@ def run_gater(a): log(f" reloc_identity: {counts} -> gating {len(keep)}") lk = _drawlock() try: - n, banked = gate(tag, keep, a.gate_jobs) + n, banked = gate(tag, keep, a.gate_jobs, run_id=str(os.getpid())) finally: lk.close() sha = commit(tag, n, banked)