mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
fix(jtbl): refuse md_*/main in jtbl_carve — it corrupts those configs before erroring
Two independent reviews (ox design study + Fable validation) found parse_config does not implement its own documented contract on md_*/main: it can DELETE the c config line and corrupt the yaml on disk before failing. main also has no config/splat.main.yaml (it is splat.us.exe.yaml), which is the FileNotFoundError seen on every main jtbl target. Refuse loudly rather than corrupt quietly (R43). Lift only after parse_config is hardened and proven on two examples. Also: gate() stages into a per-run dir (a killed gate's 3,186 stale drafts survived into the next gate of the same tag and poisoned every group); reloc_filter no longer treats NOT-A-STUB as a pass (it means ALREADY BANKED — wave an staged 480 of them over source that already byte-matches). Designs recorded: docs/tool-designs/jtbl-island-split.md (ox) and -review.md (Fable): 14 CONFIRMED / 8 WRONG / 1 UNVERIFIABLE, verdict GO-WITH-CHANGES, and the correct fix is SMALLER than proposed — one inserted .rodata carve line + jr_isolate_all.py --only, no _pre piece and no ld_interleave leading mode.
This commit is contained in:
Drew T
@@ -0,0 +1,138 @@
|
||||
# Review: jtbl leading-island split design (adversarial validation, 2026-08-24)
|
||||
|
||||
**Reviewed:** `docs/tool-designs/jtbl-island-split.md` (ox-alpha, read-only agent).
|
||||
**Method:** static verification only against the working tree — every tool source, the real
|
||||
configs, the real Makefile, the installed splat 0.41.0 / spimdisasm 1.41.0 sources, the actual
|
||||
generated `build/md_SC03_076/md_SC03_076.ld`, and the actual asm/src for md_SC03_076. No build was
|
||||
run (gate/sweep processes were live: `pgrep -f 'sweep_parallel|gate_stage|gate_main'` non-empty).
|
||||
The two pre-confirmed claims (`jtbl_carve.py:66`, `jr_isolate.py:83`) were not re-litigated.
|
||||
|
||||
**Bottom line: GO-WITH-CHANGES — the carve idea is right and the byte layout works, but two of
|
||||
the design's four steps are wrong (step 1 names a tool that provably cannot process the file;
|
||||
step 2's `_pre` rename cannot build), and two are unnecessary (steps 3–4: splat's native linker
|
||||
script already places `.rodata` ahead of `.text`). The correct implementation is *smaller* than
|
||||
the design: one inserted config line + `jr_isolate_all.py --only`.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Claim-by-claim verdicts
|
||||
|
||||
| # | Design claim | Verdict | Evidence |
|
||||
|---|---|---|---|
|
||||
| 1 | (a)1 `jr_isolate.py` contract "never edits a `.rodata` carve" (:16-17) | **CONFIRMED** | `tools/jr_isolate.py:16-17` |
|
||||
| 2 | (a)1 isolation alone → "the island piece still binds to the OLD object name … the table bytes never move … reproduces the §8a duplicate-at-wrong-address disease" | **WRONG mechanism, right conclusion** | The island piece is a *dotted* `.rodata` paired by name to `md_SC03_076.c` (splat `segtypes/common/code.py:249-279` sibling pairing). Moving the fn to a jr c-file breaks migration (migration only targets the sibling file — `rodata.py:29-48` bails when `self.sibling is not None`), and the committed `md_SC03_076.c` has no `INCLUDE_RODATA` for the table — the bytes **vanish** from `md_SC03_076.o`; a matched-C table in `jr.o(.rodata)` has no ld entry and hits `/DISCARD/ { *(*); }`. Failure mode is *shrink/discard*, not duplicate. Either way isolation alone fails — the conclusion stands. |
|
||||
| 3 | (a)2 `parse_config()` (:470-473) "defines the carvable region as the run of {data,.rodata} pieces AFTER the last `c` piece and exits 'no data-tail region' otherwise … structurally uncarvable" | **WRONG (and the truth is worse)** | "After the last c piece" is only the *comment* (`jtbl_carve.py:470`); the code takes **ALL** data/.rodata pieces (`:471`). The real config (which the design never saw) is `[0x0,.rodata]/[0x27C,c]/[0x35AC,data,tail]/[0xAAF0]` (`config/splat.md_SC03_076.yaml:96-99`) — so `parse_config` does **not** exit: it scopes the "region" 0x0..0xAAF0 *across the c line*, and `apply()`'s rewrite (`:749`, `lines[:lo]+region_lines+lines[hi:]`) would **delete `- [0x27C, c, md_SC03_076]`** — config corruption, written to disk (`:750`) *before* `set_overlays_var` dies on the missing overlays.mk anchor (`:766-768`; md vars live in `config/modules.mk:932+`). Reachable by any md fn whose table is NOT migrated (e.g. a jtbl in the 0x35AC tail): `migrated_tables` returns `[]` for it and `apply` proceeds. |
|
||||
| 4 | (a)2 "apply() refuses in migrated_tables() before build_carve() is ever reached" | **CONFIRMED** (for migrated-table fns) | `jtbl_carve.py:727-746` |
|
||||
| 5 | (a)3 `--order` branch (:101-115) hardcodes FILL/TEXT_START→text→empties→ordered→bss; nothing ahead of TEXT_START | **CONFIRMED** | `tools/ld_interleave.py:101-115` |
|
||||
| 6 | Step 1: "`jr_isolate.py md_SC03_076 --func func_801F218C`, unchanged" | **WRONG — names a tool that cannot run on this file** | `jr_isolate.py:22-27` STATUS: scaffold "BLOCKED on split_src_region.py" (untouched since Phase 26, `git log`). Statically confirmed: `split_src_region.py` `trim` exits "cannot resolve address of item" (`:109-111`) on any top-level non-addressed item; `md_SC03_076.c:25-28` has exactly such a block (`extern void (*D_801F3468[])(void);` → `item_name` captures `void` → addr None). The production tool is **`jr_isolate_all.py --only func_801F218C`** (uses `overlay_src_split`; it is what `jtbl_family_bank.py:183` invokes). Its region model differs (cut *before* the jr fn; **no `_after` piece** — the jr object runs to the next boundary), and its name matches the design's: `md_SC03_076_jr_801F218C` (`jr_isolate_all.py:194-195`, `%08X`). |
|
||||
| 7 | Step 2: replace `[0x0,.rodata,md_SC03_076]` with `[0x0,.rodata,md_SC03_076_pre]` + `[0x268,.rodata,md_SC03_076_jr_801F218C]` | **HALF-WRONG — the `_pre` rename cannot build** | The carve line (offset 0x268, name = jr c-subseg) is correct — splat pairs dotted `.rodata` to the same-named c subseg regardless of position (`code.py:249-279`; `readonly_before` exists precisely for rodata-first layouts, and md's leading island already migrates daily). But a dotted `.rodata` named `md_SC03_076_pre` has **no sibling c subseg** → splat resolves it to an "Implied C file" `src/md_SC03_076/md_SC03_076_pre.c` (`data.py:23-30`) that nothing creates → the ld references a never-built object. Worse, the rename **unbinds the island 0x0..0x268 from `md_SC03_076.c`**, breaking migration for the other five stub-owned jtbls (`jtbl_801EF5AC/5E4/61C/654/67C/6A4`) and orphaning the two committed `INCLUDE_RODATA` blobs (`md_SC03_076.c:131,133`) and compiled consts (`D_801EF6C0/6C8`, `:447,:502`). **Fix: keep the existing `- [0x0, .rodata, md_SC03_076]` line untouched and only INSERT the carve line** — its extent shrinks automatically (end = next piece's offset). The "split" is one inserted line. |
|
||||
| 8 | Step 2 ground facts: carve at file 0x268; 5 words end at 0x27C == code start; `sltiu 0x5`; 0x801EF6D0 %8==0; single-table ⇒ no `JTBL_PADS`; no post piece (table is the island's last occupant) | **CONFIRMED** | `asm/md_SC03_076/nonmatchings/md_SC03_076/func_801F218C.s` (`/* 268 801EF6D0 */`, five `.word .L801F21C8/22C4/2238/2278/22A4`, `sltiu $v0,$v1,0x5`); `config/splat.md_SC03_076.yaml:97` (`[0x27C, c, …]`); `jtbl_carve.py:37` (single-table → no var) |
|
||||
| 9 | Step 2: "read the migrated dlabels the way migrated_tables() does at :709-718" | **CONFIRMED** | `jtbl_carve.py:709-716` |
|
||||
| 10 | Step 3: ld_interleave needs a `--leading` mode; "A leading island needs [.rodata pieces] BEFORE _TEXT_START; no existing mode does it" | **TRUE of ld_interleave, but the step is UNNECESSARY** | ld_interleave **never runs for md binaries** (no `<ov>_JTBL_INTERLEAVE` var; Makefile:648-649 gate) and never needs to: splat's native script is section-major with `.rodata` FIRST per `section_order` (verified in the actual `build/md_SC03_076/md_SC03_076.ld`: `RODATA_START` precedes `TEXT_START`), and within the `.rodata` group entries follow yaml/subsegment order (`linker_entry.py:238-270` buckets by section preserving order). Post-split the group is `md_SC03_076.o(.rodata)` then `md_SC03_076_jr_801F218C.o(.rodata)` back-to-back — exactly address order, tight (no intra-group ALIGN; `SUBALIGN(2)` caps input alignment). Auto-generated `.rodata` entries for carve-less c files (`auto_link_sections` default, `options.py:521-522`) insert *after* the declared ones (`code.py:84-140`) and are empty — byte-neutral. |
|
||||
| 11 | Step 4: wire via `<ov>_JTBL_INTERLEAVE` / new `<ov>_JTBL_ISLAND`; "read the real Makefile + config/overlays.mk" | **PARTLY WRONG / unnecessary** | No interleave var is needed at all (see #10). And md per-binary vars live in **`config/modules.mk`** (`:932-…`), not overlays.mk — every mk-writing path in jtbl_carve hardcodes overlays.mk (`:114, :759, :782, :829-831`), so any future extension (e.g. a multi-table md span needing `JTBL_PADS`) must generalize the mk target first. The §8a trailing-comment/`$(strip)` gotcha is real (Makefile:122-125). |
|
||||
| 12 | Ordering constraint (ii): same-subseg carves must abut (gap 0 / verified-zero gap 4) | **CONFIRMED as existing law** | `jtbl_carve.py:528-558` — but see §4 "pre-object contiguity" for the corollary the design missed. |
|
||||
| 13 | Ordering constraint (iii): jtbl_rodata_pads must never see pre/post; refuses `.include` content | **CONFIRMED** | `tools/jtbl_rodata_pads.py:74-78`; pads armed per-object via target var (Makefile:689) |
|
||||
| 14 | (b) crash chain `apply→migrated_tables→func_jtbls→func_subseg→code_pieces→overlay_vram_base→open(cfg_path)`; `config/splat.main.yaml` absent; per-binary naming a Phase-9/10 addition | **CONFIRMED** | Chain verified (`jtbl_carve.py:727→703-706→216→144-146→127→66`); `main_SPLAT_YAML := config/splat.us.exe.yaml` (Makefile:74); `config/splat.main.yaml` does not exist; Phase-9 binaries block (Makefile:39-47). The open is at `:66` (the design's verbatim text says :67-68; its own header says :66 — trivial drift). |
|
||||
| 15 | (b) main roots are repo-level; the six fs helpers hardcode `asm/{ov}` (:162,:217,:233,:244,:260,:286) | **CONFIRMED** | `main_ASM_DIR := asm` (Makefile:85); `asm/main/` does not exist, main's tree is `asm/nonmatchings` at repo root; all six line numbers exact. |
|
||||
| 16 | (b) "Whether main is then CARVEABLE … parse_config will answer that honestly once paths work" | **WRONG — the fix list is incomplete** | Main's region-line math happens to be safe (`splat.us.exe.yaml:206-209`: the data/rodata pieces are contiguous before the EOF line, no c lines between). But the **base/offset model breaks**: `overlay_vram_base` greps the first `vram:` = 0x80010000 (`:65`), while main's pieces are FILE offsets with `start: 0x800` — the correct flat base is 0x8000F800 (`main_VRAM_BASE`, Makefile:81). Every vram↔offset conversion is off by 0x800; `payload_word` **silently reads bytes 0x800 early** (zero-pad checks, gap checks, `spec_from_starts` all corrupted, no error raised). An `asm_root()`/`cfg_path()` fix alone ships a silently wrong tool for main. |
|
||||
| 17 | (c) stub-state 43,760 bytes | **CONFIRMED** | `extracted/retail/SC03.CD.dir/FILE_076.dir/2.1` is 43,760 B (= EOF marker 0xAAF0), sha1 `9a165e…` == `config/check.md_SC03_076.sha` == yaml sha1. |
|
||||
| 18 | (c) diseased baseline "43,768 (+8), first diff 0x144" for this experiment | **UNVERIFIABLE / misattributed** | Those numbers were byte-measured on **func_801F0F28** (`jtbl_carve.py:733`), not func_801F218C. Immaterial to the control's validity (pass = SHA-equality), but the design presents another function's disease signature as this experiment's baseline. (0x144 is the island offset of `jtbl_801EF5AC` — the first text-pointer word — consistent with a +8 text shift.) |
|
||||
| 19 | (c) discriminators: jr object `.rodata sh_size == 0x14`; md object shrinks by 0x14; the five words at 0x801EF6D0; SHA gate the arbiter, never rtu_match | **CONFIRMED (one nit)** | Shrink 0x27C→0x268 ✓; the five words ✓ (from the .s); SHA-as-arbiter is project law. Nit: sh_size 0x14 holds because the table sits at *section offset 0* (`.align 3` is a no-op there), not because its final vram is 8-aligned — same number, misplaced reason. |
|
||||
| 20 | (d)1 `migrated_tables()`'s "same address, by construction" premise (:684-692) refuted by apply()'s own error; stale doc must not survive | **CONFIRMED — real defect** | Docstring `:690-691` + `:699` ("a no-op here, NOT a refusal") vs the byte-measured refusal `apply():736-739`. |
|
||||
| 21 | (d)2 hardcoded 0x80100000–0x801D0000 window (:306, :434) vacuous for md text | **CONFIRMED — and it is worse than stated** | Lines exact. md text starts 0x801EF6E4 (vram 0x801EF468 + 0x27C); every entry (0x801F21C8…) is outside the window. Consequences: `_continuation_words` can never absorb (under-carve risk, conservative), and the over-span clamp's *refusal* guard can never fire — for md the clamp would **silently drop real entries** (the dangerous direction). Also missed by the design: the window excludes **main's** text too (0x80010000..0x800629DC < 0x80100000). Derive from the binary's TEXT_LO/TEXT_HI (already in the Makefile per binary). |
|
||||
| 22 | (d)3 `--order` docstring "overstated contract" | **PARTLY WRONG (minor)** | The same help text states "text -> [these] -> bss" explicitly (`ld_interleave.py:36`); only the clause "Generalises the … sandwich to N pieces" (`:40`) could mislead if read alone. A doc nit at most. |
|
||||
| 23 | (d)4 parse_config's "after the last c piece" as the structural gap | **CONFIRMED as a gap, mischaracterized** | See #3: the comment is not implemented; the behavior is corrupt-on-rewrite, not refuse. Hardening it (implement the comment: anchor the region strictly after the last `c` piece, refuse anything else) is a **prerequisite**, not a nice-to-have. |
|
||||
| 24 | (d)5 nested-root helpers + cfg_path wrong for main | **CONFIRMED** | Dup of #14/#15. |
|
||||
| 25 | Config shape inferred as `[0x0,.rodata]+[0x27C,c]` (residual unknown, flagged) | **INCOMPLETE — honestly flagged, but it mattered** | The real config also has `[0x35AC, data, tail]` + EOF `[0xAAF0]`; that extra piece is what flips claim #3 from "refuses" to "corrupts". |
|
||||
| 26 | Typo: "`md_SC03_077_after_<addr>`" | WRONG (trivial) | Should be `md_SC03_076_after_…`; and with `jr_isolate_all` no `_after` piece exists at all. |
|
||||
|
||||
**Tally: 14 CONFIRMED, 8 WRONG/PARTLY-WRONG, 1 UNVERIFIABLE, 1 honestly-flagged-incomplete** (plus 1 trivial typo).
|
||||
|
||||
---
|
||||
|
||||
## 2. Verdict on the mechanism: GO-WITH-CHANGES
|
||||
|
||||
The core idea — give the matched jr function its own code subseg and bind its table span to a
|
||||
same-named dotted `.rodata` piece — is sound, and the byte layout works. But **as written the
|
||||
design does not build**: step 1's tool dies on the source file, step 2's `_pre` object has no
|
||||
producer, and steps 3–4 solve a problem splat does not have. The working implementation is:
|
||||
|
||||
1. `tools/jr_isolate_all.py md_SC03_076 --only func_801F218C` (production isolate; handles the
|
||||
md .c structure via `overlay_src_split`; writes the region files itself; no mid-flow extract).
|
||||
2. Insert **one line** into `config/splat.md_SC03_076.yaml`:
|
||||
`- [0x268, .rodata, md_SC03_076_jr_801F218C]` between the existing `[0x0, .rodata, md_SC03_076]`
|
||||
(which stays untouched — its extent shrinks automatically) and the c pieces.
|
||||
3. `make extract BINARY=md_SC03_076` → `make build` (stage 1, stub state, must be SHA-green),
|
||||
then splice the matched C into the jr file → `make build` (stage 2, must be SHA-green).
|
||||
No ld_interleave, no make variable: splat's native section-major script (rodata group first,
|
||||
entries in yaml order, `SUBALIGN(2)`) already produces the exact address-ordered layout.
|
||||
|
||||
Why this is byte-plausible (verified statically): the generated `.ld` places
|
||||
`md_SC03_076.o(.rodata)` then `md_SC03_076_jr_801F218C.o(.rodata)` back-to-back at 0x801EF468;
|
||||
md.o's rodata = includes + consts + the five remaining stub tables = 0x268 bytes; jr.o's rodata =
|
||||
the migrated table (stub state) or the compiled switch table (matched state) = 0x14 bytes; text
|
||||
follows at 0x801EF6E4. Same-name sibling pairing for the carve is the proven §8a mechanism, and
|
||||
the leading position is already exercised daily by the module's own 0x0 piece.
|
||||
|
||||
## 3. The negative control (design §c)
|
||||
|
||||
**Discriminating, with caveats.** The pass condition — whole-binary SHA1 equality with the
|
||||
43,760-byte payload at both stages — is the project's own arbiter and cannot pass for a wrong
|
||||
reason: any migration failure, duplicate, boundary drift, or discard changes bytes. The
|
||||
plausible-failure list (table twice, boundary off, diff moved) all fail SHA. Caveats:
|
||||
|
||||
- The "diseased baseline 43,768/+8/first diff 0x144" was measured on **func_801F0F28**, not the
|
||||
control function (claim #18). Do not treat reproducing that exact signature as meaningful.
|
||||
- Stage 2 conflates two hypotheses: a red stage 2 could mean the mechanism is broken **or** the
|
||||
C match is wrong. Run stage 2 only with a candidate that already byte-matched elsewhere in the
|
||||
family, or accept the ambiguity.
|
||||
- With `jr_isolate_all` there is no `_after` object; the jr object spans 0x2D24..0x35AC (the jr fn
|
||||
plus every trailing function). The objdump discriminator still holds — the census below shows
|
||||
the trailing functions contribute zero rodata — but that census is part of the control's
|
||||
validity and must be re-run per target.
|
||||
|
||||
## 4. What the design missed (would break an implementation or the rollout)
|
||||
|
||||
1. **The `_pre` object has no producer** (claim #7) — the single biggest defect; the design's
|
||||
own config sketch cannot link.
|
||||
2. **jtbl_carve corrupts md configs instead of refusing** when a table is not migrated (claim #3).
|
||||
Harden `parse_config` (implement its own "after the last c piece" comment, refuse otherwise)
|
||||
*before* any md work — and before pointing jtbl_carve at main.
|
||||
3. **Migration re-homing precheck.** Isolation moves every function ≥ the cut into a different
|
||||
c file; any moved function that owns a *leading-island* table, or matched C that emits const
|
||||
rodata, breaks the island reconstruction. Safe for this target — verified census: all six
|
||||
island jtbl owners (`func_801EFBB4/801F0210/801F0734/801F0A9C/801F0F28/801F218C`) are at or
|
||||
before the cut, and no const/`INCLUDE_RODATA` exists after `md_SC03_076.c:634` — but this is a
|
||||
per-target precondition, not a given. It belongs in the tool as a loud refusal.
|
||||
4. **Pre-object contiguity wall.** `md_SC03_076.o` supplies ONE contiguous rodata run, so only
|
||||
tables adjacent to the island END (or chains growing backwards from it) are carvable cheaply.
|
||||
Example: carving `jtbl_801EF6A4` (@0x23C — func_801F0F28, the very function the §154-A refusal
|
||||
was measured on) leaves md.o needing 0x0..0x23C *and* 0x258..0x268 (the consts) — impossible
|
||||
for one object; the const-owning matched functions must be isolated too. The design's "a post
|
||||
piece appears once a LATER table in the island is carved" glosses over this: the post piece
|
||||
would have to be a *second* md_SC03_076 rodata entry, which GNU ld cannot split. The rollout
|
||||
therefore unblocks fewer module jr functions than "every jr member of every family" —
|
||||
plan for cascading isolations or accept end-adjacent-only.
|
||||
5. **Pipeline integration.** Family waves reach carving through `jtbl_family_bank.py` →
|
||||
`jtbl_carve.py` (`:222`), which refuses §154-A binaries. Unless the island split is reachable
|
||||
from that path (and the refusal text updated), module jr functions stay wave-unbankable even
|
||||
after the mechanism exists.
|
||||
6. **mk-file targeting.** All of jtbl_carve's overlays.mk writers (`set_overlays_var:759`,
|
||||
`set_pads_vars:782`, `revert:829`, `current_pads_specs:114`) and `jr_isolate_all`'s
|
||||
`repoint_overlays_mk` hardcode `config/overlays.mk`; md vars live in `config/modules.mk`.
|
||||
Dormant today (no md carve vars needed), a landmine for multi-table md spans later.
|
||||
7. **Main's base/offset model** (claim #16): the path fix alone leaves `payload_word` silently
|
||||
reading 0x800 early on main. Add a per-binary (vram_base, file_base) pair, not just paths.
|
||||
|
||||
## 5. What could not be verified without a build
|
||||
|
||||
- Stage-1 byte-neutrality end-to-end (splat's actual re-extract output for the split config, and
|
||||
spimdisasm 1.41.0 writing the migrated table into the jr-subseg stub .s). Inferred from the
|
||||
§8a overlay precedent (134× proven) plus the module's own daily leading-piece migration; the
|
||||
linker-side ordering was verified against the real generated .ld and splat source, not a run.
|
||||
- `overlay_src_split.partition()` on md-class .c files (static read says the model fits; the
|
||||
design's chosen `split_src_region.py` path was statically *dis*proven, which is stronger).
|
||||
- The exact disease signature (+8 / 0x144) for func_801F218C specifically (measured value exists
|
||||
only for func_801F0F28).
|
||||
- Whether any md_* binary actually has a tail-region jtbl (the trigger for hazard §4.2) — the
|
||||
hazard is structural regardless.
|
||||
@@ -0,0 +1,176 @@
|
||||
# jtbl leading-island split — design study (ox-alpha, P31 S58, 2026-08-24)
|
||||
|
||||
**Provenance:** produced by `stealth/ox-alpha` under `tools/api_agent.py` with a read-only surface
|
||||
widened to `config,tools,Makefile` (`EXTRA_READABLE`). It read the target `.s`, the full sources of
|
||||
`jtbl_carve` / `jr_isolate` / `ld_interleave` / `jtbl_rodata_pads` / `jtbl_family_bank`, the Makefile
|
||||
excerpt, and cookbook §8 / §8a / §8a-pad / §8e / §154.
|
||||
|
||||
**Status: DESIGN, NOT VERIFIED.** ox cannot run `make`, so nothing here has been executed. Two of its
|
||||
claims were independently checked against the source before this file was written, and both held:
|
||||
|
||||
* `jtbl_carve.py:66` — `open(cfg_path(ov))` is unguarded and `cfg_path` hardwires
|
||||
`config/splat.{ov}.yaml`; main's config is `config/splat.us.exe.yaml`, so main raises
|
||||
`FileNotFoundError`. This is the crash observed on `SaveLoadRoutine`, `func_8003388C`,
|
||||
`func_80035270` and `StreamLoadStateMachine`.
|
||||
* `jr_isolate.py:83` — `line_re` matches only `- [0xADDR, c, SUB]` (code subsegs), so isolation
|
||||
cannot move a `.rodata` island. `jtbl_carve`'s own error text proposed a fix its tools cannot
|
||||
perform.
|
||||
|
||||
The remainder is the agent's verbatim deliverable.
|
||||
|
||||
---
|
||||
|
||||
// TOOLING PLAN — jtbl leading-island split (md_* modules) + main crash fix
|
||||
// Ground truth read: asm/md_SC03_076/nonmatchings/md_SC03_076/func_801F218C.s,
|
||||
// .run/bakeoff/toolwork/{jtbl_carve,jr_isolate,ld_interleave,jtbl_rodata_pads,
|
||||
// jtbl_family_bank}.py, Makefile.excerpt, cookbook §8/§8a/§8a-pad/§8e/§154.
|
||||
// config/*.yaml and the full Makefile are OUTSIDE my read surface; where that
|
||||
// matters I name the file an engineer must read.
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// (a) LEADING-ISLAND: evaluate the error-text proposal (jtbl_carve.py:741-743)
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// VERDICT: HALF-RIGHT. "Isolate the jr function into its own code subseg" works
|
||||
// unchanged; "so its .rodata is a separate OBJECT" is FALSE as stated, and
|
||||
// ld_interleave cannot do the ordering it promises. Evidence from source:
|
||||
//
|
||||
// 1. jr_isolate.py rewrites ONLY `- [off, c, SUB]` lines (line_re :83 demands
|
||||
// `,\s*c,\s*`) and its contract (:16-17) is "never edits a `.rodata` carve".
|
||||
// On md_SC03_076 (config shape quoted in apply()'s own error: `- [0x0,
|
||||
// .rodata, md_SC03_076]` + `- [0x27C, c, md_SC03_076]`) it splits just the
|
||||
// code piece into [c, md_SC03_076] + [c, md_SC03_076_jr_<addr>] +
|
||||
// [c, md_SC03_077_after_<addr>]. The island piece still binds to the OLD
|
||||
// object name; the new jr object has NO .rodata input; the table bytes never
|
||||
// move. Isolation alone reproduces the §8a duplicate-at-wrong-address disease
|
||||
// in a new costume.
|
||||
// 2. jtbl_carve.py cannot supply the missing half: parse_config() (:470-473)
|
||||
// defines the carvable region as the run of {data,.rodata} pieces AFTER the
|
||||
// last `c` piece and exits "no data-tail region" otherwise. A module's island
|
||||
// sits BEFORE its code — structurally uncarvable by this tool. (apply()
|
||||
// refuses in migrated_tables() before build_carve() is ever reached.)
|
||||
// 3. ld_interleave.py --order cannot express the placement: that branch
|
||||
// (:101-115) hardcodes FILL/TEXT_START -> text -> empties -> ordered pieces
|
||||
// -> bss. Nothing can be emitted ahead of {PREFIX}_TEXT_START. A leading
|
||||
// island needs [.rodata pieces] BEFORE _TEXT_START; no existing mode does it.
|
||||
//
|
||||
// WHAT IS RIGHT (implementation):
|
||||
// Step 1 — jr_isolate.py md_SC03_076 --func func_801F218C, unchanged. Gives
|
||||
// the fn its own code object; that object's NAME is what the carve piece must
|
||||
// carry (the §8a dotted-subseg law).
|
||||
// Step 2 — NEW: split the LEADING `.rodata` piece around the fn's MIGRATED
|
||||
// table(s). Replace
|
||||
// - [0x0, .rodata, md_SC03_076]
|
||||
// with
|
||||
// - [0x0, .rodata, md_SC03_076_pre]
|
||||
// - [0x268, .rodata, md_SC03_076_jr_801F218C]
|
||||
// (no post piece HERE: the table is the island's LAST occupant — its .s header
|
||||
// says file offset 0x268, 5 words end at 0x27C == the code piece start; a post
|
||||
// piece appears once a LATER table in the island is carved). Because the middle
|
||||
// piece's NAME equals the fn's code subseg, spimdisasm migrates the table into
|
||||
// the jr object's .s — the exact §8a mechanism aimed at a leading piece.
|
||||
// Span derivation reuses jtbl_range()'s oracles, but sourced from the FN'S OWN
|
||||
// .s (the dlabels are migrated there — read them the way migrated_tables()
|
||||
// already does at :709-718), bounded by the function's own `sltiu` (here
|
||||
// `sltiu $v0,$v1,0x5` = 5 entries = 0x14 bytes; jtbl_801EF6D0 %8==0 so no
|
||||
// trailing-pad trim and NO JTBL_PADS line — single-table carves get none, §8e).
|
||||
// Step 3 — extend ld_interleave.py with a leading mode (new flag, e.g.
|
||||
// `--leading <same comma list>`): emit the ordered .rodata piece lines FIRST,
|
||||
// then _TEXT_START/text/_TEXT_END, then bss — keeping every START/END/SIZE
|
||||
// symbol splat defines, with RODATA_START/END placed ahead of TEXT_START.
|
||||
// Tight packing preserved (. = ALIGN(.,4) between pieces; SUBALIGN(2) in the
|
||||
// splat script caps external alignment — §8e law 3).
|
||||
// Step 4 — config: the splat yaml gains the two/three piece lines above; wire
|
||||
// the interleave through the existing per-binary var mechanism (extend
|
||||
// `<ov>_JTBL_INTERLEAVE` or add `<ov>_JTBL_ISLAND := leading`). Mind the §8a
|
||||
// GOTCHA: no trailing #comment on the var line, $(strip) it. NOTE: the
|
||||
// Makefile's extract-stage wiring of JTBL_INTERLEAVE is NOT in
|
||||
// Makefile.excerpt — read the real Makefile + config/overlays.mk before
|
||||
// editing (also confirms whether md_* binaries already have an output-section
|
||||
// hook; ld_interleave takes `--section .<binary>`).
|
||||
// ORDERING CONSTRAINTS: (i) all input-section lines must be emitted in
|
||||
// ascending ADDRESS order — island pieces (0x0..0x27C) precede text (0x27C);
|
||||
// (ii) multiple matched fns follow the SAME contiguity law as overlays —
|
||||
// same-code-subseg carves must abut (gap 0, or gap 4 with a verifiably-zero
|
||||
// payload word, else NON-CONTIGUOUS → isolate that fn too); (iii)
|
||||
// jtbl_rodata_pads applies ONLY to the carve piece (pure $L/.word content) —
|
||||
// it must NEVER see pre/post, it refuses mixed include-rodata content (the
|
||||
// very 'unexpected rodata content .include "..."' error quoted in apply()).
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// (b) main: the crash
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// `jtbl_carve.py main --func <fn>` → apply() → migrated_tables() → func_jtbls()
|
||||
// → func_subseg() → code_pieces() → overlay_vram_base() →
|
||||
// jtbl_carve.py:67-68 `txt = open(cfg_path(ov)).read()`
|
||||
// with cfg_path() (:60-61) hardwired to `config/splat.{ov}.yaml`.
|
||||
// CAUSE: every other failure on the entry path is a CLEAN sys.exit (missing .s,
|
||||
// missing jtbl, no data-tail region); a Python TRACEBACK requires an unguarded
|
||||
// exception, and the only unguarded I/O on that path is this open(). main is the
|
||||
// root binary and predates the per-binary config naming (Makefile.excerpt :6-9:
|
||||
// per-binary aliases are a Phase-9/10 addition), so `config/splat.main.yaml`
|
||||
// does not exist → FileNotFoundError traceback. CONFIRM BY READING (outside my
|
||||
// surface): the Makefile "Binaries" block — what is main's SPLAT_YAML path?
|
||||
// SECOND-ORDER (bites immediately after the path fix): Makefile.excerpt :106-108
|
||||
// says main's roots are REPO-LEVEL asm/ + src/ (siblings nest at asm/<bin>/),
|
||||
// but every fs-touching helper hardcodes the nested root — all_data_labels :233,
|
||||
// jtbl_words :244, _label_words :260, _sltiu_bounds :286, overlay_jtbl_addrs
|
||||
// :162, func_jtbls :217 all build `asm/{ov}/...` ⇒ for ov="main" they resolve to
|
||||
// nonexistent asm/main/... and turn real data into "not found" exits. FIX: an
|
||||
// asm_root(ov) helper beside cfg_path(ov); both return repo-level paths when
|
||||
// ov == "main". (Whether main is then CARVEABLE at all depends on its config's
|
||||
// piece layout — parse_config will answer that honestly once paths work.)
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// (c) NEGATIVE CONTROL
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// Function: func_801F218C, binary md_SC03_076 (this task's target: 83 ins,
|
||||
// jtbl_801EF6D0 @0x801EF6D0, sltiu bound 5, island span 0x268..0x27C).
|
||||
// Two-stage control:
|
||||
// Stage 1 (split-only, fn STILL A STUB): after jr_isolate + island split +
|
||||
// re-extract, `make build BINARY=md_SC03_076` must be BYTE-IDENTICAL — size
|
||||
// 43,760, SHA1 == payload. Proves the split itself byte-neutral.
|
||||
// Stage 2 (splice the matched C, rebuild): STILL 43,760 / SHA-equal. The
|
||||
// diseased baseline is the measured 43,768 (+8) with first diff at 0x144.
|
||||
// Byte-level discriminators of success vs plausible-looking failure:
|
||||
// - objdump -h build/src/md_SC03_076/md_SC03_076_jr_801F218C.o: .rodata
|
||||
// sh_size == 0x14 (exactly 5 words, NO align pad — 0x...6D0 is 8-aligned);
|
||||
// md_SC03_076.o's .rodata shrinks by exactly 0x14. If BOTH objects still
|
||||
// carry table bytes, the migration did not happen (name mismatch).
|
||||
// - Final image: bytes from 0x144 onward IDENTICAL to payload (the +8
|
||||
// signature GONE, not moved); the five words 801F21C8/801F22C4/801F2238/
|
||||
// 801F2278/801F22A4 sit at vram 0x801EF6D0.
|
||||
// - A build that links at 43,760 but with the first diff merely MOVED (pre/
|
||||
// post boundary off by a pad word, or table present twice) is FAILURE.
|
||||
// Arbiter = whole-binary SHA gate, NEVER rtu_match (§8a: masks relocs and
|
||||
// excludes jtbl rodata — false-MATCH class).
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// (d) WRONG rather than MISSING in the existing tools
|
||||
// ─────────────────────────────────────────────────────────────────────────────
|
||||
// 1. migrated_tables()'s premise (jtbl_carve.py:684-692: matching makes the C
|
||||
// "emit the table into the same object's .rodata, AT THE SAME ADDRESS, by
|
||||
// construction") is refuted by apply()'s own measured error text (+8 bytes,
|
||||
// first diff 0x144): pruning the .s drops the table out of the include chain
|
||||
// and cc1 re-emits it at the END of the object's .rodata. The REFUSAL is
|
||||
// correct; the no-op rationale is byte-measured wrong and must not survive
|
||||
// as documentation.
|
||||
// 2. Hardcoded overlay VRAM window `0x80100000 <= w < 0x801D0000` in
|
||||
// _continuation_words (:306) and the over-span clamp (:434): md_SC03_076's
|
||||
// text lives at 0x801EF468+ — EVERY real entry (e.g. 0x801F21C8) falls
|
||||
// OUTSIDE the window, so "surplus word looks like a code address" is vacuous
|
||||
// and the continuation/clamp authorization gates misfire for exactly the
|
||||
// module class this work enables. Derive the window from the binary's text
|
||||
// bounds (config), not constants.
|
||||
// 3. ld_interleave --order's docstring claims it "generalises the sandwich to N
|
||||
// pieces"; it generalises only TAIL islands (text-first is structural,
|
||||
// :101-115). Overstated contract — fix the doc or add the leading mode.
|
||||
// 4. parse_config's "data region = after the last `c` piece" — the structural
|
||||
// gap that makes (a) necessary rather than optional.
|
||||
// 5. Nested-root path helpers + cfg_path for ov=="main" ((b)) — wrong default
|
||||
// for the root binary, not merely absent.
|
||||
|
||||
// Residual unknowns + the files that answer them: config/splat.md_SC03_076.yaml
|
||||
// (exact piece list/indent/trailing-bin — shape above taken from apply()'s
|
||||
// error text + the .s's own /* 268 */ offset), config/splat.main.yaml existence
|
||||
// and the Makefile "Binaries"/extract-interleave wiring. All outside src/,
|
||||
// asm/, docs/, include/, .run/bakeoff/.
|
||||
+19
-1
@@ -58,7 +58,25 @@ EOF_RE = re.compile(r"^\s*- \[(0x[0-9A-Fa-f]+)\]\s*(?:#.*)?$")
|
||||
|
||||
|
||||
def cfg_path(ov):
|
||||
return os.path.join(REPO, f"config/splat.{ov}.yaml")
|
||||
# HARD REFUSAL for binary classes this tool cannot handle. Two independent reviews (ox design
|
||||
# study + Fable validation, P31 S58) established that on md_* and main jtbl_carve does not
|
||||
# merely fail — `parse_config` does not implement its own documented contract, so it can DELETE
|
||||
# the `c` config line and CORRUPT the yaml on disk before it errors out. main additionally has
|
||||
# no `config/splat.main.yaml` at all (it is `splat.us.exe.yaml`), so the unguarded open() below
|
||||
# raised FileNotFoundError on every main jtbl target.
|
||||
#
|
||||
# Refusing loudly beats corrupting quietly (R43). Lift this only once parse_config is hardened
|
||||
# and the fix is proven on two examples — see docs/tool-designs/jtbl-island-split-review.md,
|
||||
# whose verdict is that the real change is ONE inserted `.rodata` carve line plus
|
||||
# jr_isolate_all.py --only, NOT the _pre split or an ld_interleave leading mode.
|
||||
if ov == "main" or ov.startswith("md_"):
|
||||
sys.exit(f"jtbl_carve: REFUSING {ov} — md_*/main configs are corrupted by parse_config "
|
||||
f"before it errors (see docs/tool-designs/jtbl-island-split-review.md). "
|
||||
f"Overlays (ov_*) are supported.")
|
||||
p = os.path.join(REPO, f"config/splat.{ov}.yaml")
|
||||
if not os.path.exists(p):
|
||||
sys.exit(f"jtbl_carve: no splat config at {p} for binary {ov!r}")
|
||||
return p
|
||||
|
||||
|
||||
def overlay_vram_base(ov):
|
||||
|
||||
+86
-12
@@ -296,7 +296,15 @@ def reloc_filter(tag, drafts, cards_path):
|
||||
return batch, {"(reloc_identity produced no output — gating unfiltered)": len(batch)}
|
||||
status = {r["fn"]: r.get("status") for r in res}
|
||||
counts = collections.Counter(status.values())
|
||||
keep = [b for b in batch if status.get(b["fn"]) in ("AGREE", "NOT-A-STUB")]
|
||||
# NOT-A-STUB IS NOT A PASS — it means reloc_identity found nothing to check because the
|
||||
# function is ALREADY BANKED. Gating those re-stages a draft body over source that already
|
||||
# byte-matches: pure waste at best, and at worst it perturbs a banked function inside a group
|
||||
# and takes the group's genuinely-new drafts down with it. Wave `an` carried 480 NOT-A-STUB of
|
||||
# 697 "gated" and banked 0. Only AGREE is a pass. (R43: refuse input the step cannot use.)
|
||||
keep = [b for b in batch if status.get(b["fn"]) == "AGREE"]
|
||||
n_banked_already = sum(1 for b in batch if status.get(b["fn"]) == "NOT-A-STUB")
|
||||
if n_banked_already:
|
||||
counts = dict(counts, _already_banked_excluded=n_banked_already)
|
||||
return keep, dict(counts)
|
||||
|
||||
|
||||
@@ -326,7 +334,7 @@ def gate_main_batch(tag, mains):
|
||||
return len(banked), banked
|
||||
|
||||
|
||||
def gate(tag, keep, jobs):
|
||||
def gate(tag, keep, jobs, run_id=None):
|
||||
# Any main drafts that slipped through (an older wave's cards) are PARKED for the periodic main
|
||||
# batch rather than gated inline — see draw_wave's note on why main is off the critical path.
|
||||
mains = [k for k in keep if k["binary"] == "main"]
|
||||
@@ -338,7 +346,12 @@ def gate(tag, keep, jobs):
|
||||
log(f" main: {len(mains)} drafts PARKED to .run/main_queue/{tag}.json "
|
||||
f"(gate them with tools/gate_main.py when no wave is waiting)")
|
||||
mains = []
|
||||
d = f".run/sweep_{tag}"
|
||||
# PER-RUN staging dir. Reusing `.run/sweep_<tag>` let a KILLED gate's staged drafts survive
|
||||
# into the next gate of the same tag: measured wave `an` staging 697 drafts into a directory
|
||||
# that held 3,186, so every per-binary group carried stale bodies and failed as a group —
|
||||
# 208 reloc-AGREE drafts banked 0. rmtree is not enough when a crashed or concurrent run may
|
||||
# have written the same path; a unique path makes the contamination impossible, not unlikely.
|
||||
d = f".run/sweep_{tag}" + (f".{run_id}" if run_id else "")
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
for b in keep:
|
||||
bd = os.path.join(d, b["binary"])
|
||||
@@ -448,6 +461,32 @@ def _drawlock():
|
||||
return fh
|
||||
|
||||
|
||||
def _drawlock_nb():
|
||||
"""Non-blocking draw lock. Returns None if a gate holds it — the caller must NOT wait."""
|
||||
os.makedirs(".run/auto", exist_ok=True)
|
||||
fh = open(DRAWLOCK, "w")
|
||||
try:
|
||||
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
return fh
|
||||
except BlockingIOError:
|
||||
fh.close()
|
||||
return None
|
||||
|
||||
|
||||
def _predrawn(skip_tag):
|
||||
"""A wave whose cards exist but which has never been drafted — free work while a gate runs."""
|
||||
for p in sorted(glob.glob(".run/wave_??_cards.json"), key=os.path.getmtime):
|
||||
t = os.path.basename(p)[5:7]
|
||||
if t == skip_tag:
|
||||
continue
|
||||
if os.path.exists(f"{READY}/{t}.json"): # already drafted and queued
|
||||
continue
|
||||
if glob.glob(f".run/wave_{t}/shard*/*.c"): # already has drafts
|
||||
continue
|
||||
return t, p
|
||||
return None
|
||||
|
||||
|
||||
def run_drafter(a):
|
||||
"""THE LANE THAT MUST NEVER STOP. Draw -> shard -> draft -> publish a ready marker. Forever.
|
||||
|
||||
@@ -474,11 +513,17 @@ def run_drafter(a):
|
||||
log(f"drafter: resuming rotation at index {w} -> lane "
|
||||
f"{lane_specs[w % len(lane_specs)]['name']}, band {bands[w % len(bands)]}")
|
||||
while not os.path.exists(STOP):
|
||||
# Bound the run-ahead: drawing 40 waves of cards while the gater lags would strand them all
|
||||
# as "already-waved" and starve later waves of candidates.
|
||||
if len(glob.glob(f"{READY}/*.json")) >= a.queue_depth:
|
||||
time.sleep(30)
|
||||
continue
|
||||
# Bound the run-ahead on DRAWING only. Drawing 40 waves while the gater lags would strand
|
||||
# them all as "already-waved"; DRAFTING a wave whose cards already exist strands nothing.
|
||||
# Conflating the two stopped the fleet dead: with queue_depth=2 and two waves queued, the
|
||||
# drafter slept while `aa` and `ai` sat pre-drawn and undrafted (measured P31 S58).
|
||||
backlogged = len(glob.glob(f"{READY}/*.json")) >= a.queue_depth
|
||||
if backlogged:
|
||||
pre = _predrawn(None)
|
||||
if not pre:
|
||||
log(f" gater backlogged ({a.queue_depth}+ waves) and nothing pre-drawn — waiting")
|
||||
time.sleep(60)
|
||||
continue
|
||||
cr = credits_left()
|
||||
if cr is not None and cr < a.credit_floor:
|
||||
log(f"drafter: credit floor (${cr:.2f}) — stopping"); break
|
||||
@@ -490,9 +535,38 @@ def run_drafter(a):
|
||||
w += 1
|
||||
log(f"=== DRAFT {tag} · lane {lane['name']} · band {band[0]}-{band[1]} · "
|
||||
f"{a.workers} workers ===")
|
||||
lk = _drawlock()
|
||||
cards = draw_wave(tag, a.cards_per_wave, band, lane.get("levers"))
|
||||
lk.close()
|
||||
# NON-BLOCKING draw. The gater holds this lock for its WHOLE gate (substituted drafts make
|
||||
# corpus.stubs() lie), so a blocking acquire here parks the entire drafting fleet for the
|
||||
# duration of every gate — measured 7 minutes idle with 0 agents while wave `an` gated, and
|
||||
# it would recur on every wave. Drafting is the clock-limited resource; it must never wait
|
||||
# on gating. If the lock is busy we draft a wave that was PRE-DRAWN earlier instead, and
|
||||
# only sleep when there is genuinely nothing drawn to work on.
|
||||
lk = None if backlogged else _drawlock_nb()
|
||||
if lk is None:
|
||||
pre = _predrawn(tag)
|
||||
if pre:
|
||||
tag, cards = pre
|
||||
log(f" gate in progress — drafting PRE-DRAWN wave {tag} instead of waiting")
|
||||
else:
|
||||
log(" gate holds the draw lock and nothing is pre-drawn — waiting 30s")
|
||||
time.sleep(30); continue
|
||||
else:
|
||||
cards = draw_wave(tag, a.cards_per_wave, band, lane.get("levers"))
|
||||
lk.close()
|
||||
# Draw the NEXT wave's cards too while the lock is ours, so the next gate cannot idle
|
||||
# the fleet. Cheap (a few minutes of CPU) and it buys a whole gate's worth of drafting.
|
||||
nxt = next((t for t in tags
|
||||
if t not in {os.path.basename(q)[5:7]
|
||||
for q in glob.glob(".run/wave_??_cards.json")}), None)
|
||||
if nxt:
|
||||
nlane = lane_specs[(w) % len(lane_specs)]
|
||||
nband = bands[(w) % len(bands)]
|
||||
lk2 = _drawlock_nb()
|
||||
if lk2 is not None:
|
||||
log(f" pre-drawing {nxt} (lane {nlane['name']}, band {nband[0]}-{nband[1]}) "
|
||||
f"so the next gate cannot idle the fleet")
|
||||
draw_wave(nxt, a.cards_per_wave, nband, nlane.get("levers"))
|
||||
lk2.close()
|
||||
if not cards:
|
||||
log(f" {tag}: draw failed — retrying next cycle"); time.sleep(30); continue
|
||||
targets = shard_targets(tag, cards, a.workers)
|
||||
@@ -524,7 +598,7 @@ def run_gater(a):
|
||||
log(f" reloc_identity: {counts} -> gating {len(keep)}")
|
||||
lk = _drawlock()
|
||||
try:
|
||||
n, banked = gate(tag, keep, a.gate_jobs)
|
||||
n, banked = gate(tag, keep, a.gate_jobs, run_id=str(os.getpid()))
|
||||
finally:
|
||||
lk.close()
|
||||
sha = commit(tag, n, banked)
|
||||
|
||||
Reference in New Issue
Block a user