From 3a886b9652ee2ef955eef8eed22163eab52635ab Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 2 Sep 2026 19:22:16 -0600 Subject: [PATCH] fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried decl layer: extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218(); CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)` closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a `def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype. A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated `/*` included, so the carried layer opened a comment that silently ate the next two declarations — a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip` tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom block comment and blanked everything to the next `*/`. FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`). `_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide. MEASURED, not assumed: * the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in 150 files — every one a binary whose next isolation would have carried a broken decl layer. * A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways; exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed implied prototypes 999 -> 984; 0 refusals. * negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae. * with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th (func_80184084) is the separate CARVE-REFUSED class. The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands in a dropped segment. Named in §437, deliberately not fixed here. Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise). The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools. --- docs/SETUP.md | 1 + docs/cookbook-index.md | 13 +++-- docs/matching-cookbook.md | 106 +++++++++++++++++++++++++++++++++++++ tools/jr_isolate_all.py | 20 ++++++- tools/overlay_src_split.py | 104 +++++++++++++++++++++++++++++++++--- tools/split_src_region.py | 12 ++++- 6 files changed, 244 insertions(+), 12 deletions(-) diff --git a/docs/SETUP.md b/docs/SETUP.md index 035304d36..7c1627093 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -769,6 +769,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `config/wave_exclude.txt` | **(P31 S72)** THE canonical wave exclude list — **tracked**, and **regenerated, never hand-edited**: `tools/exclude_audit.py config/wave_exclude.txt --write `. Consumed as `draw_waves --exclude-file config/wave_exclude.txt`, which AUDITS it as a PREREQUISITE and refuses to draw on a stale one (`--exclude-stale-ok` overrides, loudly). Two classes: **CARVE-BLOCKED** (derived from `split_indicator`; disappears when the subseg is split — **EMPTY since P31 S74**, all four overlays split) and **WALL** (curated, cannot be re-derived — the `# WALL:` annotation is a PIN that survives regeneration, and its note is the refutation list to beat before reopening the entry). It replaces the nine `.run/S*_exclude.txt` snapshots and seven walls ledgers, none of which was authoritative; measured on its predecessor, 88 of 107 entries were stale one day after it was written, 46 of them open drawable work totalling 12,750 instructions. | | | **main's TU layout (P31 S72)** | main's game code is no longer one file. `src/800.c` was split at the jtbl-span TU boundaries into **`src/800.c`** (vram 0x800123F0-0x8002B0B4, owns `.rodata` span A 0x80072A38-0x80072C70), **`src/800_b.c`** (0x8002B0B4-0x80035270, span B 0x80072E44-0x80073140) and **`src/800_c.c`** (0x80035270-0x8003A444, span C 0x800732A0-0x8007344C), plus **`src/800_shared.h`** for the 19 typedefs + 2 includes that cross. Reason: one code object contributes exactly ONE contiguous `.rodata` run, so each jump-table span needs its own object or every switch function in it double-emits its table (cookbook §426/§431). main's island is now a 7-piece sandwich and `ld_interleave` runs with `--order`, not `--front/--tail`. **Any tool that reads "main's source" must glob `corpus.src_files('main')`, never hardcode `src/800.c`** — `reconcile_slate` did, and saw 133 of 187 typedefs after the split. | | | `tools/split_indicator.py` | **(P31 S72, cookbook §426/§431)** Which code subsegs MUST be split before their switch functions can bank: a code object contributes exactly ONE contiguous `.rodata` run, so a subseg owning raw jump tables in ≥2 non-adjacent island spans makes every switch function outside the one carveable span unbankable at any effort. **Decidable from the raw image at 0% matched** — no attempt needed. `--self-test` fires on main's pre-S72 island, stays silent on main today, and does not over-fire on a one-span subseg; linked PsyQ subsegs are excluded on principle. **(P31 S74)** Runs in `make tools-health` as a **HARD GATE** — the four violations it waited on are split, the fleet is **213 OK of 213**, and a new one now fails the build. The owner of a table is derived from the CONFIG by address, never from the stub's directory path: `make extract` does not prune a re-homed subseg's `nonmatchings//` dir, and reading it made the tool assert NEEDS SPLIT for a split that was already correct and byte-green (cookbook §436). Stale debris is now named in a `note:` line, which prints on an OK verdict too. | +| | `tools/overlay_src_split.py` `comment_open_at()` | **(P31 S74, cookbook §437)** THE comment-state oracle for the TU-split chain: per line, does it BEGIN inside a block comment. Every peeler in the chain used to ask `line.strip().startswith("/*")`, which is blind to a comment a construct OPENS MID-LINE and WRAPS — 238 such lines across 193 tracked `.c` files. The construct ends at its `;` *before* the `/*`, so the caller resumed on comment PROSE and `scan_construct` read it as code (`(s32,s32)` in the prose closes a depth-0 paren -> `seen_header` latches -> every later `;` reads as a K&R param decl -> one 'construct' swallows the whole preamble). Result: `parse_overlay_c` anchored a `def` on a pure declaration run, and `def_proto` emitted `extern #define CALL_… extern void func_…();` into the §8b carried decl layer — **`parse error before '#'`, which blocked five independently-MATCHed jr bodies in `ov_SC06_029`**. Now consulted by `parse_overlay_c`, `def_proto`, `split_src_region.parse` and `jr_isolate_all._file_scope_decls` (which also truncates a hoisted col-0 decl at an unterminated `/*` — that comment used to run on and silently eat the next carried decls). `parse_overlay_c` may now RAISE `ValueError` (R43) when a wrapped comment CLOSES with code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide, so callers see it only if someone writes one. Building the guard exposed a third defect in the same model: **`_strip` tested for `/*` before stripping `//`**, so a line comment containing `/*` (`// … src/*/*.c`, 7 lines in 5 sources) opened a phantom block comment and blanked every following line until some later `*/`; `_strip` now lexes left to right. Negative-controlled: `ov_SC06_029` byte-identical before the change and after, with the five bodies banked (sha1 `b7b0d4ae…`). | | | `tools/exclude_audit.py` | **(P31 S72)** Classifies every wave-exclude entry by its CURRENT blocker — `BANKED` / `LINKED` / `RE-PROBE` (blocker since fixed) / `CARVE-BLOCKED` / `WALL` — and regenerates the list keeping only the still-valid classes. **`draw_waves --exclude-file` runs it as a PREREQUISITE and refuses a stale list** (`--exclude-stale-ok` overrides, loudly). Measured on `.run/S71_exclude.txt` one day after it was written: 88 of 107 entries stale, of which 46 were open drawable functions totalling 12,750 instructions — `main:SaveLoadRoutine` among them. | | | `tools/main_diff_locate.py` | **(P31 S72, cookbook §426/§427)** Turns a RED main gate — whose whole output is two SHA1s — into a NAMED list of divergent symbols: byte-diffs `build/us/SLUS_007.26` against `extracted/retail/SLUS_007.26`, coalesces the differing bytes into runs, and attributes them PER BYTE to symbols via `build/us/SLUS_007.26.map` (file-offset mapping DERIVED from each output section's `load address`, not the 0x800 header constant). `--focus ` prints the routing verdict, FOUR of them: **BODY REJECT** (divergence confined to the function) · **TABLE REJECT** (§405-A — `.text` byte-identical, all divergence in `.rodata`, i.e. its own jump table: fix the case VALUES/ORDER, do NOT respell the body and do NOT route to §376; checked FIRST because it was mislabelled PLUMBING until `func_800316F8` produced it) · **PLUMBING REJECT** (the function is byte-identical, everything differs elsewhere IN CODE → the §376/§378 chain) · **MIXED**. `--self-test ` is the negative control: flips one byte at a known address and asserts the containing symbol is named, plus the identical-pair-reports-zero direction. `gate_main` calls it automatically and preserves the red image + map under `.run/gate_main_fail/` BEFORE the R40 baseline control rebuilds over them. | | | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. | diff --git a/docs/cookbook-index.md b/docs/cookbook-index.md index 865dd430a..b7af7588f 100644 --- a/docs/cookbook-index.md +++ b/docs/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 1105 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 1106 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -618,7 +618,7 @@ - **§422** — ★★ — QImode ARITHMETIC VIA `(u8)(x - K)`, AND `flag ^ 1` NEEDS ITS OWN TEMP (P31 S71; byte-proven `resident/func_800D06E8`, 344 ins) L33760 - **§423** — ★★★ — "MATCH IN ISOLATION + GATE REJECTS + CAUSE NOT DETERMINED" ⇒ GREP THE TU FOR A FILE-SCOPE TYPEDEF THE DRAFT ALSO CARRIES (P31 S71; byte-proven `ov_SC03_092/func_8017FA74`) L33780 -### declarations, prototypes & K&R (114) +### declarations, prototypes & K&R (115) - **§3-T4** — Branch polarity: invert the source condition to flip gcc's chosen branch L90 - **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) L456 @@ -734,8 +734,9 @@ - **§378c** — ★★ — THE FIFTH DECL-BLOCKER VARIANT: THE DRAFT REDECLARES WHAT THE TU ALREADY OWNS (P31 S69) L32722 - **§398** — ★★★ — `family_remap` CARRIES THE **SOURCE** TU's DECL ENVIRONMENT INTO A DESTINATION THAT ALREADY OWNS THOSE NAMES (P31 S69; measured 3 banked of 22) L32835 - **§415** — ★★ — A FILE-SCOPE DECL MAKES gcc-2.7.2 MERGE THE TU'S LATER *BLOCK-SCOPE* EXTERNS INTO IT (P31 S71; byte-proven `ov_SC04_011/func_80180B24`, 215 ins) L33547 +- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) L34421 -### jump tables & switches (63) +### jump tables & switches (64) - **§8** — rodata island (compiler jump tables) — the `.data→.rodata→.data` sandwich (Phase 7) L339 - **§8a** — rodata island in a flat OVERLAY — the tail sandwich, per matched jr-function (Phase 26 — PoC PROVEN) L361 @@ -800,6 +801,7 @@ - **§431** — ★★★ — SPLITTING A 27,000-LINE TU AT ITS ORIGINAL BOUNDARIES: THE JTBL SPANS TELL YOU WHERE, AND THE COMPILER TELLS YOU WHAT CROSSES (P31 S72; `src/800.c` -> `800.c`/`800_b.c`/`800_c.c`, byte-identical with nothing banked) L34122 - **§433** — ★★★ — ON A SWITCH FUNCTION, CASE SOURCE ORDER IS THE DOMINANT RESIDUAL — AND `match_one` IS BLIND TO IT (P31 S73; 4 of 5 consecutive main MATCHes) L34185 - **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) L34307 +- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) L34421 ### optimisation level (-O0/-O2) (22) @@ -951,7 +953,7 @@ - **§420** — ★★★ — A MULTI-CLUSTER SYMBOL REBASE, AND THE BARE-NAME DEDUP THAT HID THREE QUARTERS OF IT (P31 S71; 4 banked in 57 s) L33692 - **§421** — ★★★ — A `la $tN` + `addiu` PAIR CAN BE A **RELOAD** ARTIFACT THAT NO C SPELLING REACHES (P31 S71; byte-proven `md_SC07_003/func_801A293C`, 313 ins, 6 → 0) L33729 -### integration / TU plumbing (79) +### integration / TU plumbing (80) - **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) L456 - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L502 @@ -1032,6 +1034,7 @@ - **§423** — ★★★ — "MATCH IN ISOLATION + GATE REJECTS + CAUSE NOT DETERMINED" ⇒ GREP THE TU FOR A FILE-SCOPE TYPEDEF THE DRAFT ALSO CARRIES (P31 S71; byte-proven `ov_SC03_092/func_8017FA74`) L33780 - **§431** — ★★★ — SPLITTING A 27,000-LINE TU AT ITS ORIGINAL BOUNDARIES: THE JTBL SPANS TELL YOU WHERE, AND THE COMPILER TELLS YOU WHAT CROSSES (P31 S72; `src/800.c` -> `800.c`/`800_b.c`/`800_c.c`, byte-identical with nothing banked) L34122 - **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) L34307 +- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) L34421 ### build graph, splat & the harness (196) @@ -2807,6 +2810,7 @@ - **§434** — ★★★ — TWO SYMBOLS, ONE FRAME: RUN THE FRAME CHECK BEFORE DRAFTING ANYTHING LARGE (P31 S73; `main/SaveLoadRoutine` + `func_8002B0B4`, byte-verified) L34259 - **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) L34307 - **§436** — ★★★ — TWO TOOLS THAT READ THE WRONG SOURCE OF TRUTH, AND THE SHAPE THEY SHARE (P31 S74; both fixed + negative-controlled) L34364 +- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) L34421 --- @@ -3924,3 +3928,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: ` | L34259 | §434 | ★★★ — TWO SYMBOLS, ONE FRAME: RUN THE FRAME CHECK BEFORE DRAFTING ANYTHING LARGE (P31 S73; | | L34307 | §435 | ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", | | L34364 | §436 | ★★★ — TWO TOOLS THAT READ THE WRONG SOURCE OF TRUTH, AND THE SHAPE THEY SHARE (P31 S74; bo | +| L34421 | §437 | ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern | diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 82f7a4888..607be7fed 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -34417,3 +34417,109 @@ refused to accept "the gate said no" as a fact about its own work. current config), then ask what it READS (an asm tree, or a whole file it only partly owns). Where those differ, the tool will one day report a true fact about a world that no longer exists — and it will be believed, because it is the tool whose job is to be trusted. + +## §437 ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) + +A parallel gate rejected **five** `ov_SC06_029` drafts with the identical, non-codegen verdict: + +``` +func_801867D0 PLUMBING: src/ov_SC06_029/ov_SC06_029_jr_801867D0.c:138: parse error before '#' +func_801898CC … :193 … func_801801D8 … :73 … func_80187660 … :141 … func_80180A70 … :73 … +``` + +Those `.c` files do not exist in the repo — `jr_isolate_all.py` writes them at gate time so each +jr's jump table can carve. Line 138 of the generated file was: + +```c +extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218(); +``` + +**The cause is a comment, not a compiler.** The TU carries this, at file scope: + +```c +extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU; + calls go through a (s32,s32) function-pointer cast, the TU's own idiom */ +#define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) +extern void func_8012C218(void *a0); +``` + +Every peeler in `overlay_src_split.py` decided "is this line a comment?" with +`line.strip().startswith("/*")` — which sees a comment that STARTS a line and is blind to one a +construct OPENS mid-line. The declaration ends at its `;`, which is BEFORE the `/*`, so +`scan_construct` returns and the caller resumes **on the comment's continuation line**, entered with +`in_block=False`. It then reads comment prose as code, and the prose is hostile: `(s32,s32)` closes a +depth-0 paren, so the scan latches `seen_header`, after which every `;` reads as a **K&R parameter +declaration** and one "construct" swallows 15 lines up to the next real definition. Two byte-relevant +consequences: + +* `parse_overlay_c` anchored a `def` item on a pure DECLARATION run, naming `func_8012C218` — a + resident function with no definition anywhere in this TU; +* `def_proto` → `_proto_from_lines` then rendered that whole run as the definition's "implied + prototype" and prefixed `extern`, producing the line above. `_proto_from_lines` already carried a + D1 backstop for a chunk that OPENS inside a comment (P30 S48) — proof the shape had been met + before — but the backstop trims the comment tail *after* the construct scan has already over-run, + so it could not help. + +A second, quieter defect rode along: `_file_scope_decls` hoisted the col-0 line **verbatim**, +unterminated `/*` included, so the region's carried decl layer opened a comment that ran on and +**silently ate the next carried declarations** (measured: `extern void func_8012C218(void *a0);` and +`extern u8 D_80190348[];` vanished into it). A dropped file-scope decl is a silent byte-changer. + +**The fix is one derived model of the file's comment state, not three line tests** (R33). +`overlay_src_split.comment_open_at(lines)` returns, per line, whether that line BEGINS inside a block +comment; `parse_overlay_c`, `def_proto`, `split_src_region.parse` and `jr_isolate_all._file_scope_decls` +all consult it, and a col-0 decl whose trailing `/*` never closes is truncated at the `/*` before it is +carried (comments emit no code, so that is byte-neutral). `_refuse_code_after_comment_close` refuses a +file where such a comment CLOSES with code after the `*/`, because that construct could never anchor +(R43; measured 0 occurrences across the 4,188 sources the parser is run on). + +**Building that guard immediately found a THIRD defect in the same model.** `_strip` tested for `/*` +BEFORE stripping `//`, so a LINE comment containing `/*` — + +```c +// src/shared/engine_core.h src/*/*.c (ov_SC07_006_jr_801457A4.c:3968) +``` + +— was read as opening a block comment. Everything after it stripped to `""` until some later `*/`, so +`scan_construct` saw blank lines where real declarations stood. `_strip` now lexes left to right: +whichever of `//` and `/*` comes first wins. (7 such lines in 5 sources.) + +**Scale, so nobody reads this as one overlay's quirk.** The wrapped-trailing-comment shape occurs +**238 times across 193 of our `.c` files**, and **153 of those are col-0 hoistable declarations in +150 files** — i.e. 150 binaries whose next isolation would have carried an unterminated comment into +its decl layer. This class was never specific to `ov_SC06_029`; that overlay is simply where a jr +isolation happened to cut next to one. + +**A/B over all 4,188 tracked sources, old parser vs new, so the blast radius is measured and not +asserted:** + +| | old | new | +|---|---|---| +| round-trip identity (header + items == file) | 4188/4188 | 4188/4188 | +| files whose ITEM LIST changed | — | **2** (`ov_SC02_031_jr_8017AE2C.c`, `ov_SC06_029_jr_8017C954.c`) | +| items gained | — | 0 | +| items lost | — | 1 per file, each a **phantom `def`** no definition backs (`func_8012C218`; an addr-less `block`) | +| malformed implied prototypes | 999 | 984 | +| R43 refusals | — | 0 | + +Nothing is gained and nothing real is lost: the only items that disappear are false anchors. **The +984 residual malformed prototypes are a DIFFERENT, pre-existing trigger** — a col-0 line that glues +several declarations and `DEFINE_func_*()` invocations together (`extern s32 aF…(…) __asm__(""); +DEFINE_func_8014C4AC() … s32 func_8014C5D0(…)`). `_file_scope_decls`'s final pass splits those on +`;` and drops every segment containing a `DEFINE_`, which is why they have never blocked a bank. +**Four of them still carry a `#`** (`… DEFINE_func_8014E5B4() #include "" extern void …`, in +`ov_SC01_077` ×3 and `ov_SC07_006`) and survive only because the `#` happens to land inside a +dropped `DEFINE_` segment — one edit away from being the same `parse error before '#'`. That pass +also discards the segment holding the DEFINITION's own implied prototype, and a dropped prototype is +a silent byte-changer, so the class deserves its own fix rather than its current luck. + +**The transferable law.** *A comment model that is right for the shape you had in mind is silently +wrong for the shape in front of you, and a parser that loses comment state at a chunk boundary will +emit that comment's PROSE as C.* When a gate rejects several independently-MATCHed bodies with the +SAME error, at line numbers in files that do not exist in the repo, the subject is the generator — +make the file appear on disk and read the line before touching a single draft. + +**Negative control (mandatory here, because the fix rewrites what the build compiles):** an +UNMODIFIED `make extract && make build -j$(nproc) && make check` of the binary FIRST, then the gate. +`ov_SC06_029` was byte-identical before, and byte-identical after with all five bodies banked — +`sha1 b7b0d4ae629fdc4f76c59fa146b4fcc78078c9d1`. diff --git a/tools/jr_isolate_all.py b/tools/jr_isolate_all.py index 9f4b38714..11fc484d7 100644 --- a/tools/jr_isolate_all.py +++ b/tools/jr_isolate_all.py @@ -638,7 +638,25 @@ def _file_scope_decls(items): out.append((block, True)) elif seen_types[key] != body: type_conflicts.append((key, seen_types[key], body)) - for line in text.split("\n"): + # THE CARRIED LINE MUST NOT CARRY AN UNTERMINATED COMMENT (P31 S74, byte-witnessed). + # A col-0 decl whose trailing `/*` note WRAPS — + # extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU; + # calls go through a (s32,s32) fn-ptr cast, the TU's own idiom */ + # — used to be hoisted VERBATIM, opening a comment in the region's decl layer that then + # ran on and SILENTLY ATE the next carried decls until the next `*/` (measured in + # ov_SC06_029_jr_801867D0.c: `extern void func_8012C218(void *a0);` and + # `extern u8 D_80190348[];` vanished into it). A dropped file-scope decl is a silent + # byte-changer, so mask on the SAME comment-state model the parser uses (oss.comment_open_at, + # R33): skip a line that BEGINS inside a comment, and truncate one that OPENS a comment it + # does not close. Comments emit no code, so dropping the note is byte-neutral. + _tlines = text.split("\n") + _opens = oss.comment_open_at(_tlines) + for _li, line in enumerate(_tlines): + if _opens[_li]: # interior of a wrapped block comment + continue + _code, _still_open = oss._strip(line, False) + if _still_open: # trailing `/*` that never closes on this line + line = line[:line.index("/*")].rstrip() if not line or line[0].isspace(): # col-0 only (block-scope stays put) continue if "{" in line or "}" in line: diff --git a/tools/overlay_src_split.py b/tools/overlay_src_split.py index 01dc6af0d..60c223df8 100644 --- a/tools/overlay_src_split.py +++ b/tools/overlay_src_split.py @@ -209,7 +209,19 @@ def def_name(construct_lines): def _strip(line, in_block): """Blank out // and /* */ comments (block state carried) + string/char literals, - for brace/paren/semicolon token counting. Returns (code, in_block).""" + for brace/paren/semicolon token counting. Returns (code, in_block). + + LEFT-TO-RIGHT, BECAUSE WHICHEVER MARKER COMES FIRST WINS (P31 S74). This used to test for + `/*` BEFORE stripping `//`, so a LINE comment containing `/*` — which our own commentary does + routinely, e.g. + + // src/shared/engine_core.h src/*/*.c (ov_SC07_006_jr_801457A4.c:3968) + + — was read as OPENING a block comment (`src/*` supplies the `/*`). Everything after it then + stripped to "" until some later `*/`, so `scan_construct` saw blank lines where real + declarations stood, and `comment_open_at` marked live code as comment interior. Measured: 7 + such lines in 5 tracked sources. Same class as the wrapped-comment defect above — a comment + model that is right for the common shape and silently wrong for the one in front of it.""" c = line if in_block: if "*/" in c: @@ -217,16 +229,80 @@ def _strip(line, in_block): in_block = False else: return "", True - c = re.sub(r'/\*.*?\*/', '', c) - if "/*" in c: - c = c.split("/*", 1)[0] - in_block = True - c = re.sub(r'//.*$', '', c) + out, i, n = [], 0, len(c) + while i < n: + two = c[i:i + 2] + if two == "//": # rest of the line is a comment + break + if two == "/*": + j = c.find("*/", i + 2) + if j == -1: # opens a block comment that wraps + in_block = True + break + i = j + 2 + continue + out.append(c[i]) + i += 1 + c = "".join(out) c = re.sub(r'"(?:\\.|[^"\\])*"', '""', c) c = re.sub(r"'(?:\\.|[^'\\])*'", "''", c) return c, in_block +def comment_open_at(lines, start_in_block=False): + """[bool] per line — does this line BEGIN inside an unterminated /* block comment? + + ONE derived model of the file's comment state (R33), consulted by every line-based peeler in + this module. They used to decide "is this line a comment?" from `line.strip().startswith("/*")`, + which is blind to the shape that actually occurs 238 times across 193 of our .c files: + + extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU; + calls go through a (s32,s32) function-pointer cast, the TU's own idiom */ + + The construct ENDS at its `;`, which is BEFORE the `/*`, so the peeler resumes on the comment's + CONTINUATION line — and `scan_construct`, entered with in_block=False, reads comment PROSE AS + CODE. Measured on ov_SC06_029 (P31 S74): `(s32,s32)` in the prose closed a depth-0 paren, so the + scan latched `seen_header`, every following `;` read as a K&R parameter declaration, and one + "construct" swallowed 15 lines of preamble up to the next real definition. Two consequences, + both byte-relevant: + * parse_overlay_c anchored a `def` item on a pure DECLARATION run (`func_8012C218`, whose + definition is in the resident, not this TU); + * def_proto/_proto_from_lines then rendered that run as the definition's "implied prototype", + emitting `extern #define CALL_80185C6C (...) extern void func_8012C218();` into the §8b + carried decl layer — `parse error before '#'`, which blocked five independently-MATCHed + jr bodies from banking in ov_SC06_029 alone. + + A line that begins inside a block comment is never an anchor and never starts a construct, so + the peelers skip it; the text itself is sliced by line and preserved verbatim either way. + + `start_in_block` is for a chunk that was CUT out of a larger file mid-comment (an item whose + preceding anchor line opened a wrapped comment): its first line already sits inside one.""" + out, in_block = [], start_in_block + for ln in lines: + out.append(in_block) + _c, in_block = _strip(ln, in_block) + return out + + +def _refuse_code_after_comment_close(lines, opens, where=""): + """R43: refuse a file where a wrapped block comment CLOSES with code after the `*/`. + + Such a line begins inside the comment (so the peelers skip it, above) yet carries a construct + that would then never anchor — a silent loss, which is exactly the failure mode this module + keeps paying for. Measured 0 occurrences across the 4,188 sources this parser is run on, so + refusing costs nothing today and can never become silent later. Split the line if it fires.""" + for k, ln in enumerate(lines): + if not opens[k] or "*/" not in ln: + continue + rest = ln.split("*/", 1)[1] + if re.sub(r'/\*.*?\*/|//.*$', '', rest).strip(): + raise ValueError( + "overlay_src_split: %sline %d closes a wrapped block comment and then carries " + "code on the SAME line (%r). That construct can never anchor — put it on its own " + "line. (R43: refusing rather than silently dropping it.)" + % (where, k + 1, ln.strip()[:120])) + + _DECL_KW = ("extern", "typedef", "struct", "union", "enum") @@ -326,6 +402,11 @@ def parse_overlay_c(src, syms): hdr_end = split_header(lines) header = "\n".join(lines[:hdr_end]) n = len(lines) + # THE FILE'S COMMENT STATE IS A DERIVED MODEL, NOT A PER-LINE GUESS (P31 S74, R33). See + # comment_open_at: the `s.startswith("/*")` test below only ever saw a comment that starts a + # LINE, so a construct whose TRAILING comment wraps left every peel branch resuming inside it. + opens = comment_open_at(lines) + _refuse_code_after_comment_close(lines, opens) items = [] i = hdr_end pre_start = i @@ -333,6 +414,9 @@ def parse_overlay_c(src, syms): raw = lines[i] s = raw.strip() # ---- peel non-anchor prefix lines into the accumulating preamble ---- + if opens[i]: # interior of a wrapped block comment + i += 1 + continue if s == "": i += 1 continue @@ -686,8 +770,16 @@ def def_proto(item_text): """The prototype implied by a `def`-kind item's function definition (None if none).""" lines = item_text.split("\n") n = len(lines) + # Same derived comment-state model as parse_overlay_c (R33). An ITEM can also OPEN inside a + # comment — its preceding anchor line wrapped one — so seed the state from the D1 test + # `_proto_from_lines` already uses: a `*/` with no `/*` before it. + _c, _o = item_text.find("*/"), item_text.find("/*") + opens = comment_open_at(lines, start_in_block=(_c != -1 and (_o == -1 or _o > _c))) i = 0 while i < n: # peel the preamble to reach the definition construct + if opens[i]: # interior of a wrapped block comment: never a construct + i += 1 + continue s = lines[i].strip() if s == "" or s.startswith("//"): i += 1 diff --git a/tools/split_src_region.py b/tools/split_src_region.py index 98deb78b0..5a722cfbd 100644 --- a/tools/split_src_region.py +++ b/tools/split_src_region.py @@ -20,8 +20,12 @@ Item vram comes from a func_XXXXXXXX name, else a name looked up in config/symbo Brace matching is naive (counts {}); the build's SHA1 check is the backstop if an item with string-literal braces is mis-split. """ +import os import re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from overlay_src_split import comment_open_at # ONE comment-state model for the whole split chain (R33) + SYMS_PATH = "config/symbols.us.txt" @@ -92,6 +96,11 @@ def parse(src): """Return (header, [item_text, ...]) splitting top-level items.""" lines = src.split("\n") n = len(lines) + # SAME BLINDNESS, SAME CURE as overlay_src_split (P31 S74, cookbook §437): the comment peel + # below only recognises a comment that STARTS a line, so a construct whose TRAILING `/*` note + # WRAPS (`extern void f(); /* note` + continuation) left this loop treating the comment's prose + # as an item. `comment_open_at` is the one derived model of the file's comment state (R33). + opens = comment_open_at(lines) i = 0 while i < n and (lines[i].startswith("#include") or lines[i].strip() == ""): i += 1 @@ -103,7 +112,8 @@ def parse(src): continue start = i # leading line/block comments belong to the following item - while i < n and (lines[i].lstrip().startswith("//") or lines[i].lstrip().startswith("/*") + while i < n and (opens[i] or lines[i].lstrip().startswith("//") + or lines[i].lstrip().startswith("/*") or lines[i].lstrip().startswith("*")): if lines[i].lstrip().startswith("/*") and "*/" not in lines[i]: while i < n and "*/" not in lines[i]: