docs: §371 the module-binary -O0 carve route + the spimdisasm rodata trap; SETUP.md S68 tooling rows (R21)

§371 ★★ carving a SINGLE-OBJECT module binary. One 'unaddressable content'
message was THREE stacked causes (interior-YAML-comment symbol-list truncation, a
trailing verbatim-asm chunk with no region, bare tag forward decls) -- fix one and
the message does not change, which is why it read as an impassable wall.

Then the reusable part: spimdisasm migrates single-referenced rodata into a
function's .s ONLY within the same subseg, so a carve that moves the function
silently DROPS it, and INCLUDE_RODATA cannot bring it back (splat marks it migrated
segment-wide and emits nothing). Rename the .rodata subseg to the object its
emitters moved to; the regenerated .s coming back byte-identical is the proof.

Also recorded: the Makefile -O0 glob hunk is PART of the carve, not a follow-up;
interleave_check's DRIFT on md_MAIN_003 is PRE-EXISTING and must not be 'fixed';
the still-open second-carve refusal (UNOWNED rodata 0x800cedf8); and the §126 plan
for the remaining 8 -O0 stubs (three are ADJACENT so one region covers them).

SETUP.md (R21): three tooling-inventory rows covering gater_lane/escalate_fable/
o0_boundary, the six overlay-layout fixes, and the module-binary carve route.
This commit is contained in:
Drew T
2026-08-31 18:34:38 -06:00
parent 650bb7285d
commit 54c0624e49
3 changed files with 59 additions and 3 deletions
+3
View File
@@ -776,6 +776,9 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
| | `tools/audit_digest.py` + **`make audit-digest`** | **(P30 S1e, cookbook §140)** The **scoreboard** oracle: recomputes the three headline metrics from the CURRENT tree and fails if the committed `docs/progress.fleet.md` disagrees. Wired into `tools-health` AFTER `report`. Exists because a digest generated from a working tree that later changed (work reverted before the commit landed) is **byte-invisible** — `check-all` stays 140/140 over it forever (R34: the byte-gate is a null oracle for DOCUMENTS) — and the next honest regeneration then reads as a REGRESSION that never happened. That is exactly what the `commit:1426` digest did: overstated **+7,879 ins / +130 unique fns**, which parked the phase's best lever on a phantom for a session. Compares **integers, not the printed percentages** (the staleness rendered as "94.4%" on both sides). Negative-control-proven against that stale digest. Same task hardened `progress.py stub_addrs`, which wrapped the fail-closed `corpus.stubs` in a bare `except` → empty stub set → `matched = sig − stubs` credited EVERY function: byte-witnessed reporting **instr 100.00% / distinct 100.00%** in a tree with no `asm/`. The identical swallow was fixed in `cast_call_sites.tu_for` + `reconcile_tu.tu_for`, where it silently reconciled drafts against the default `<ov>.c` instead of the jr/-O0 split TU — the very bug `cast_call_sites`' docstring exists to fix. |
| | `tools/cdecl.py` + **`make audit-cdecl`** | **THE C-declaration oracle (cookbook §51g).** ONE recursive-descent parser of C's **declarator grammar**, replacing fifteen tools' private regex models — models that disagreed with each other and were, all fifteen, blind to fn-ptr/jump-table decls (`extern void (*D_X[])(void);`), sized arrays (`[4]`), and multi-declarators (where the *whole line* was dropped). Total by construction, not by shape enumeration. **Two statement paths, because the inputs differ:** `tu_statements()` derives a TU's file scope from **`cpp`** (a decl inside a `DEFINE_func_*` macro body declares nothing until invoked — §8c; 54 ms/TU), and `split_statements()` is a **span-preserving** raw split for drafts (which get rewritten). API: `parse` / `scope` / `tu_scope` / `Declarator{name,kind,type,params,pnames,is_proto,is_definition}`. Verified: **2,952,246 depth-0 statements → 2,731,521 declarators, 0 parser defects**; **50,405 distinct declarations round-tripped through the real cross-gcc, 0 rejected**; residue adjudicated NOT-C *by gcc*, not by opinion. **Phase-27 T4 — the canonical draft-typedef strip:** `typedef_names(tu_path)` (the names a TU declares as typedefs, robust `tu_statements`-based so a coverage gap can't crash the byte-gate) + `strip_provided_typedefs(draft, provided)` (drop a draft's self-contained typedefs the target already supplies, splitting multi-typedef lines and covering scalar AND struct typedefs). Replaced **six** copied scalar-name regexes with complementary holes: `harvest_verify` now strips per-TU (unblocks the 39 struct-typedef drafts `_TD` dropped) and **surfaces cc1 stderr** so a `redefinition`/`conflicting types` failure reports as **PLUMBING**, not a byte mismatch (`.run/harvest_failed.classified.txt`); `masked_diff.strip_scalar_typedefs()` (used by `match_one`/`p16_permute`) fixes the multi-typedef-line skip that discarded 42 masked-MATCH drafts over whitespace (`func_8015C030` → `MATCH (23 ins)` unedited). `canon_sig_reconcile`/`eval_lora`/`format_finetune` keep their own copies for now (migrate per-bank, byte-gated — the audit-prescribed cadence). |
| | Phase 26-A tool-hygiene close (A9d–A10) | **DELETED** (R33, dead Phase-17 chain): `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py` — `reconcile_tu`/`cdecl` answer their question from the build. **`overlay_src_split.py`**: `scan_construct` force_decl latch fixed (no longer swallows a def sharing a line with leading externs) + `hidden_definitions()` R32 coverage oracle wired into `selftest`. **`jr_isolate_all.py` `jr_inventory`**: `banked` DERIVED FROM THE IMAGE (`family_remap.reloc_targets` owns-a-carve) not a gitignored roster (R33) + curated-name via `addr_of` + 1:1 carve-ownership assert. **`family_remap.reloc_targets`**: optional `data=` param (read the image once, pass to N calls). **`backlog.py`**: `BACKLOG_NO_RENDER` env so parallel `gate_stage` workers skip the render race (append is atomic). `reconcile_tu` confirmed live on BOTH banking paths (`gate_stage` + `jtbl_family_bank.recover`→`bank_exemplar`). |
| | **S68 tooling — the gater lane + the -O0 route** (P31 S68) | **`tools/gater_lane.py` (NEW)** — the continuous gater: drains a wave's finished drafts into `parallel_gate`, grouped by binary, `--r22` by default. Ledger AND verdicts keyed **`binary:fn:arm`** (R48 — and an escalation ALWAYS has a prior verdict, so arm-keying is what stops an in-flight fable draft riding the opus one). `--extra BINARY:PATH` for non-wave drafts, `--skip-binary` for lanes that may be writing, and **main is routed IN-TREE via `harvest_verify`** because `parallel_gate`'s worktree cannot stage main's psyq_integrate link inputs. **`tools/workflows/escalate_fable.js` (NEW)** — warm-started escalation: passes the prior draft + its measured closeness + its ruled-out levers, and demands a reusable `new_idiom`. **`tools/o0_boundary.py` (NEW)** — the stranded-boundary -O0 sweep (141 binaries / 288 boundaries / 0 candidates: the class is EXHAUSTED, and that null is negative-controlled). |
| | **S68 fixes — six instances of the overlay-layout assumption** (cookbook **§363**) | `dedup_propagate` could not even IMPORT (`os.` at module level in the one module that imports `os as _os`). `seed_ref` offered main's LINKED-subseg DEAD TEXT as bankable twins (43 of 82 hits — a draft there gates GREEN while wrong); now refuses and COUNTS the refusal. `parallel_gate.stage_generated` hard-coded `build/<b>/<b>.ld`; now asks the Makefile for `<b>_LD_SCRIPT`/`<b>_UNDEF_SYMS`/`<b>_UNDEF_FUNCS` and REFUSES when absent. `rtu_match` gained **`--tu`** (+ `blocker_probe` passes `stub.path` and `stub.asm_dir`) — it reconstructed `src/<source>/<split>.c`, which is the overlay layout; main's sources are LOOSE FILES in `src/`. `gate_stage` no longer synthesises `--out`/`--good-sha` — for main those resolved to a nonexistent path and then **ov_SC01_077's SHA** via `DEF_SHA`. **`psyq_integrate`**: the `*_externals.ld` map is now MONOTONIC — it was re-derived against the CURRENT `.ld`, so `firstfile = 0x80061FA8;` was DROPPED on every incremental relink and main was 2 bytes red before any draft was spliced (**the true identity of the 2026-08-15 'main link defect'**). |
| | **S68 — the module-binary -O0 carve route** (cookbook **§371**) | `jr_isolate_all` + `overlay_src_split` + the **Makefile -O0 glob widened to `src/md_*/md_*_o0?.c`** open carving for the single-object `md_*` binaries. Three stacked causes behind one `unaddressable content` message (interior-YAML-comment symbol-list truncation; a trailing verbatim-asm chunk with no region; bare tag forward decls), then the **spimdisasm rodata-migration trap**: migrated rodata follows its function ONLY within the same subseg, so a carve silently drops it and `INCLUDE_RODATA` cannot bring it back — rename the `.rodata` subseg to the object its emitters moved to. **The Makefile hunk MUST be committed with the carve** or a fresh clone loses -O0 on the region and every draft banked there mystery-fails. |
| | `tools/recover_rejects.py` | **(P31 S59)** Free recovery of PRE-GATE rejects, wired into the maintenance lane. Two paths exist for a draft that does not bank and only one was recorded: a gate failure gets a backlog row (closeness/class/best draft), while a draft the reloc pre-filter drops reached nothing — **569 of 1,261 drafts over eight waves, 45%**. Of the `MISMATCH?` rejects, **13% carry `shape: MATCH`** — right body, wrong symbol names, i.e. the §171 stale-seed class `aprop_symfix` rebases deterministically. Reads `.run/reloc_rejects.jsonl` (written by `ox_campaign.reloc_filter`), keeps shape-MATCH rows that are STILL open stubs, runs `aprop_symfix --fix`, and STAGES the rebased bodies into `.run/sweep_maint/<bin>/` for the lane's existing free gate. It never substitutes, gates or commits — a bad recovery can waste a build, never a bank. Tried-once is remembered in `.run/recover_rejects_seen.json`. Zero model tokens. |
| | `tools/lanes/restart_main_lane_when_idle.sh` | **(P31 S59)** Restart the main lane's SHELL at its one safe boundary — no main-lane agents alive AND no `gate_main` running, i.e. between its gate and its next draw — so an env/arg change (`MAXTOK`, `HTTP_TIMEOUT`) lands without discarding drafted work or aborting a batch. Companion to `relaunch_drafter_shell.sh` (wave boundary) and `restart_gater_when_idle.sh` (no sweep in flight); see `docs/accelerators.md` #5 for why a running lane never reads your edit. |
| | **campaign constants** (`MAXTOK` / `HTTP_TIMEOUT`) | **(P31 S59, probed against `stealth/ox-alpha`)** `--maxtok 16000` and `export HTTP_TIMEOUT=700` on both drafting lanes — **one setting, not two**. ox reports `reasoning_tokens=0` (its thinking is IN the content stream), so the output cap WAS the reasoning cap: measured over ALL turns, wave `bk` at 8k truncated **240 of 3,222 = 7.4%** and wave `bt` at 16k truncated **16 of 1,210 = 1.3%** (~6x better); an early count of **240 of 244** compared truncated turns against turns that printed a finish reason — against themselves — and wrongly read as ~100%. A truncated turn is a tax of one turn in 24, not a lost agent: the next turn emits the tool call. An uncapped hard prompt wanted **8,067** tokens. It generates at **~30 tok/s**, so 16k needs ~530 s and the old 420 s socket would have killed those turns (a timeout wastes the whole turn; truncation leaves a partial). Model ceiling is 1M context / **131,072** max completion, so 16k is our choice, not a limit. Ordering that must hold: generation < `HTTP_TIMEOUT` (700) < stallguard's wedged-agent kill (1200 s). `REASON_CAP` works on ox but shortens the ANSWER too (618-672 tokens) — a quality dial, not a truncation fix. Turn caps are NOT binding on the default lane (non-MATCH median 4 oracle calls, p90 12, of 24). |
+7 -3
View File
@@ -2,7 +2,7 @@
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
>
> `docs/matching-cookbook.md` is ~716 KB / 1021 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
> `docs/matching-cookbook.md` is ~716 KB / 1022 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
@@ -690,7 +690,7 @@
- **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) <sub>L31637</sub>
- **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) <sub>L31777</sub>
### jump tables & switches (52)
### jump tables & switches (53)
- **§8** — rodata island (compiler jump tables) — the `.data→.rodata→.data` sandwich (Phase 7) <sub>L339</sub>
- **§8a** — rodata island in a flat OVERLAY — the tail sandwich, per matched jr-function (Phase 26 — PoC PROVEN) <sub>L361</sub>
@@ -744,6 +744,7 @@
- **§338** — `jtbl_carve._sltiu_bounds` MISREADS A NON-SWITCH `sltiu` AS A BOUNDS CHECK, OVER-SPANNING THE TABLE (P31 S67; ov_SC06_022/func_80185B80, byte-diagnosed) <sub>L31171</sub>
- **§339** — A 2-CASE SWITCH OMITS THE LOW-BOUND RANGE TEST, SO THE PRESENCE OF `slti/bnez` BETWEEN THE `beq`s IS A **COUNT TELL** FOR A THIRD CASE NODE (P31 S67; byte-proven ov_SC02_005/func_80190538, 197 ins) <sub>L31181</sub>
- **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) <sub>L31682</sub>
- **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) <sub>L31890</sub>
### optimisation level (-O0/-O2) (21)
@@ -959,7 +960,7 @@
- **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) <sub>L31682</sub>
- **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) <sub>L31777</sub>
### build graph, splat & the harness (184)
### build graph, splat & the harness (185)
- **§4** — Flag/toolchain gotchas <sub>L190</sub>
- **Build** — mechanism — per-file opt override (splat resegmentation) <sub>L307</sub>
@@ -1145,6 +1146,7 @@
- **§353** — USE `-fno-thread-jumps` AS AN **ORACLE** TO PROVE A RESIDUAL IS thread_jumps, THEN LAUNDER THE *VALUE* TO KEEP A DEAD RE-TEST (P31 S67; byte-proven ov_SC01_008/func_8017EC68, 279 ins) <sub>L31519</sub>
- **§357** — ONE STRUCT POINTER, NOT TWO: A SECOND SOURCE VARIABLE BUILDS A THIRD IV (P31 S68; byte-proven ov_SC06_029/func_80181DF8, 335 ins, 330 → 13) <sub>L31589</sub>
- **§358** — (sharpens §333) — AN **UNREFERENCED** FIXED-SIZE AGGREGATE LOCAL IS LOAD-BEARING (P31 S68; same function) <sub>L31598</sub>
- **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) <sub>L31890</sub>
### process, measurement & doctrine (125)
@@ -2603,6 +2605,7 @@
- **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) <sub>L31792</sub>
- **§369** — REUSE THE **COMPARE CONSTANT'S OWN VARIABLE** FOR A MASK THAT KEEPS COALESCING (P31 S68; md_SC07_004/func_801AEC38, 365 ins) <sub>L31822</sub>
- **§370** — ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/func_8001BC6C, 69 ins, NOT closed — 33 → 28 over ~45 measured compiles) <sub>L31838</sub>
- **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) <sub>L31890</sub>
---
@@ -3636,3 +3639,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
| L31792 | §368 | ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_1 |
| L31822 | §369 | REUSE THE **COMPARE CONSTANT'S OWN VARIABLE** FOR A MASK THAT KEEPS COALESCING (P31 S68; m |
| L31838 | §370 | ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/fun |
| L31890 | §371 | ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP T |
+49
View File
@@ -31886,3 +31886,52 @@ union of one variant's load block and another's chain block — and no single sp
for the head (identical head residual unpinned) and load-bearing only for the tail.** That is the
§361 procedure applied correctly, and it is why this diagnosis can be trusted where the previous
one could not.
## §371 ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins)
The overlays arrive pre-split into many `jr_*` objects; the `md_*` module binaries are **ONE `c`
subseg**. Every attempt to carve an -O0 range out of one died at
`jr_isolate_all: unaddressable content`. That single message was **three stacked causes**, which is
exactly why it read as one impassable wall — fix one and the message does not change:
1. **`overlay_src_split.load_ov_syms` stopped at an interior YAML comment.** md_MAIN_003's yaml
annotates the body of its symbol-file list, so only `symbols.us.txt` loaded and `D_800D3200`
resolved to `None`.
2. **A trailing content chunk had nowhere to go.** The verbatim-asm pair after the last addressable
anchor now attaches to the LAST region when every symbol it defines resolves at/after the last
cut, instead of hard-refusing.
3. **Bare tag forward decls** (`struct S_D2394;`) tripped the `_file_scope_decls` dedupe refusal.
**THEN THE CARVE CAUSES A LINK FAILURE, and this is the reusable part.**
**spimdisasm migrates rodata referenced by exactly one function into that function's `.s` ONLY
within the same subseg.** A carve that moves the function to a new subseg while the `.rodata` island
stays behind makes splat **silently drop** those blocks:
undefined reference to `D_800CEE58' / `D_800CEE80' (from the post object's .text)
**`INCLUDE_RODATA` does NOT resurrect them** — splat marks them migrated segment-wide and emits
nothing (`can't open ... D_800CEE58.s`). **The fix is to rename the `.rodata` subseg to the object
where its emitters now live.** Verification that you got it right: the regenerated `.s` for the moved
function comes back **byte-identical to the pre-carve one**.
**THE MAKEFILE HUNK IS PART OF THE CARVE, NOT A FOLLOW-UP.** The -O0 wildcard covered only
`src/ov_*/ov_*_o0?.c`. A module region file compiles at **-O2** without widening it to
`src/md_*/md_*_o0?.c` — byte-neutral while the region holds only stubs (INCLUDE_ASM is verbatim
asm), but **every -O0 draft banked into it then mystery-fails the gate** (§362's trap class), and a
fresh clone reintroduces it. Commit the Makefile, the tool fixes and the carve TOGETHER.
**DO NOT chase `interleave_check` ALIGNED here.** md_MAIN_003 reports DRIFT on a CLEAN tree —
pre-existing, not carve-caused (verify before changing anything). It has no `_JTBL_INTERLEAVE` block
and must not get one: forcing ALIGNED moves the leading rodata island after `.text` and shifts every
address by 0xD8. The honest criterion is *"the carve does not change interleave_check's output"*.
**KNOWN, STILL OPEN:** a second carve on the same binary now refuses with
`jr_inventory(md_MAIN_003): committed .rodata carve ownership is not 1:1 (R32/R33) — a
stranded/duplicated carve: [('UNOWNED', '0x800cedf8')]`. The first carve's rodata rename left that
island unowned by the inventory's 1:1 check. Solve that before carving the remaining 7 -O0 stubs in
this binary's pre-TU.
**PLANNING THE REST (§126 applied):** of md_MAIN_003's 12 open stubs, **8 are -O0** (1,166 ins).
`func_800D0A7C` / `func_800D0B1C` / `func_800D0C50` are ADJACENT (gap 0) so ONE region covers all
three; the other five have matched bodies between them and need their own regions — K interleaved
matched bodies ⇒ K+1 regions, which `o0_subsplit` derives for you from a single `--lo/--hi` span.