mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-28 14:59:48 -04:00
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
This commit is contained in:
Drew T
@@ -769,6 +769,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `config/wave_exclude.txt` | **(P31 S72)** THE canonical wave exclude list — **tracked**, and **regenerated, never hand-edited**: `tools/exclude_audit.py config/wave_exclude.txt --write <new>`. Consumed as `draw_waves --exclude-file config/wave_exclude.txt`, which AUDITS it as a PREREQUISITE and refuses to draw on a stale one (`--exclude-stale-ok` overrides, loudly). Two classes: **CARVE-BLOCKED** (derived from `split_indicator`; disappears when the subseg is split — **EMPTY since P31 S74**, all four overlays split) and **WALL** (curated, cannot be re-derived — the `# WALL:` annotation is a PIN that survives regeneration, and its note is the refutation list to beat before reopening the entry). It replaces the nine `.run/S*_exclude.txt` snapshots and seven walls ledgers, none of which was authoritative; measured on its predecessor, 88 of 107 entries were stale one day after it was written, 46 of them open drawable work totalling 12,750 instructions. |
|
||||
| | **main's TU layout (P31 S72)** | main's game code is no longer one file. `src/800.c` was split at the jtbl-span TU boundaries into **`src/800.c`** (vram 0x800123F0-0x8002B0B4, owns `.rodata` span A 0x80072A38-0x80072C70), **`src/800_b.c`** (0x8002B0B4-0x80035270, span B 0x80072E44-0x80073140) and **`src/800_c.c`** (0x80035270-0x8003A444, span C 0x800732A0-0x8007344C), plus **`src/800_shared.h`** for the 19 typedefs + 2 includes that cross. Reason: one code object contributes exactly ONE contiguous `.rodata` run, so each jump-table span needs its own object or every switch function in it double-emits its table (cookbook §426/§431). main's island is now a 7-piece sandwich and `ld_interleave` runs with `--order`, not `--front/--tail`. **Any tool that reads "main's source" must glob `corpus.src_files('main')`, never hardcode `src/800.c`** — `reconcile_slate` did, and saw 133 of 187 typedefs after the split. |
|
||||
| | `tools/split_indicator.py` | **(P31 S72, cookbook §426/§431)** Which code subsegs MUST be split before their switch functions can bank: a code object contributes exactly ONE contiguous `.rodata` run, so a subseg owning raw jump tables in ≥2 non-adjacent island spans makes every switch function outside the one carveable span unbankable at any effort. **Decidable from the raw image at 0% matched** — no attempt needed. `--self-test` fires on main's pre-S72 island, stays silent on main today, and does not over-fire on a one-span subseg; linked PsyQ subsegs are excluded on principle. **(P31 S74)** Runs in `make tools-health` as a **HARD GATE** — the four violations it waited on are split, the fleet is **213 OK of 213**, and a new one now fails the build. The owner of a table is derived from the CONFIG by address, never from the stub's directory path: `make extract` does not prune a re-homed subseg's `nonmatchings/<old>/` dir, and reading it made the tool assert NEEDS SPLIT for a split that was already correct and byte-green (cookbook §436). Stale debris is now named in a `note:` line, which prints on an OK verdict too. |
|
||||
| | `tools/overlay_src_split.py` `comment_open_at()` | **(P31 S74, cookbook §437)** THE comment-state oracle for the TU-split chain: per line, does it BEGIN inside a block comment. Every peeler in the chain used to ask `line.strip().startswith("/*")`, which is blind to a comment a construct OPENS MID-LINE and WRAPS — 238 such lines across 193 tracked `.c` files. The construct ends at its `;` *before* the `/*`, so the caller resumed on comment PROSE and `scan_construct` read it as code (`(s32,s32)` in the prose closes a depth-0 paren -> `seen_header` latches -> every later `;` reads as a K&R param decl -> one 'construct' swallows the whole preamble). Result: `parse_overlay_c` anchored a `def` on a pure declaration run, and `def_proto` emitted `extern #define CALL_… extern void func_…();` into the §8b carried decl layer — **`parse error before '#'`, which blocked five independently-MATCHed jr bodies in `ov_SC06_029`**. Now consulted by `parse_overlay_c`, `def_proto`, `split_src_region.parse` and `jr_isolate_all._file_scope_decls` (which also truncates a hoisted col-0 decl at an unterminated `/*` — that comment used to run on and silently eat the next carried decls). `parse_overlay_c` may now RAISE `ValueError` (R43) when a wrapped comment CLOSES with code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide, so callers see it only if someone writes one. Building the guard exposed a third defect in the same model: **`_strip` tested for `/*` before stripping `//`**, so a line comment containing `/*` (`// … src/*/*.c`, 7 lines in 5 sources) opened a phantom block comment and blanked every following line until some later `*/`; `_strip` now lexes left to right. Negative-controlled: `ov_SC06_029` byte-identical before the change and after, with the five bodies banked (sha1 `b7b0d4ae…`). |
|
||||
| | `tools/exclude_audit.py` | **(P31 S72)** Classifies every wave-exclude entry by its CURRENT blocker — `BANKED` / `LINKED` / `RE-PROBE` (blocker since fixed) / `CARVE-BLOCKED` / `WALL` — and regenerates the list keeping only the still-valid classes. **`draw_waves --exclude-file` runs it as a PREREQUISITE and refuses a stale list** (`--exclude-stale-ok` overrides, loudly). Measured on `.run/S71_exclude.txt` one day after it was written: 88 of 107 entries stale, of which 46 were open drawable functions totalling 12,750 instructions — `main:SaveLoadRoutine` among them. |
|
||||
| | `tools/main_diff_locate.py` | **(P31 S72, cookbook §426/§427)** Turns a RED main gate — whose whole output is two SHA1s — into a NAMED list of divergent symbols: byte-diffs `build/us/SLUS_007.26` against `extracted/retail/SLUS_007.26`, coalesces the differing bytes into runs, and attributes them PER BYTE to symbols via `build/us/SLUS_007.26.map` (file-offset mapping DERIVED from each output section's `load address`, not the 0x800 header constant). `--focus <fn>` prints the routing verdict, FOUR of them: **BODY REJECT** (divergence confined to the function) · **TABLE REJECT** (§405-A — `.text` byte-identical, all divergence in `.rodata`, i.e. its own jump table: fix the case VALUES/ORDER, do NOT respell the body and do NOT route to §376; checked FIRST because it was mislabelled PLUMBING until `func_800316F8` produced it) · **PLUMBING REJECT** (the function is byte-identical, everything differs elsewhere IN CODE → the §376/§378 chain) · **MIXED**. `--self-test <addr>` is the negative control: flips one byte at a known address and asserts the containing symbol is named, plus the identical-pair-reports-zero direction. `gate_main` calls it automatically and preserves the red image + map under `.run/gate_main_fail/` BEFORE the R40 baseline control rebuilds over them. |
|
||||
| | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
|
||||
>
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1105 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1106 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
|
||||
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
|
||||
|
||||
@@ -618,7 +618,7 @@
|
||||
- **§422** — ★★ — QImode ARITHMETIC VIA `(u8)(x - K)`, AND `flag ^ 1` NEEDS ITS OWN TEMP (P31 S71; byte-proven `resident/func_800D06E8`, 344 ins) <sub>L33760</sub>
|
||||
- **§423** — ★★★ — "MATCH IN ISOLATION + GATE REJECTS + CAUSE NOT DETERMINED" ⇒ GREP THE TU FOR A FILE-SCOPE TYPEDEF THE DRAFT ALSO CARRIES (P31 S71; byte-proven `ov_SC03_092/func_8017FA74`) <sub>L33780</sub>
|
||||
|
||||
### declarations, prototypes & K&R (114)
|
||||
### declarations, prototypes & K&R (115)
|
||||
|
||||
- **§3-T4** — Branch polarity: invert the source condition to flip gcc's chosen branch <sub>L90</sub>
|
||||
- **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) <sub>L456</sub>
|
||||
@@ -734,8 +734,9 @@
|
||||
- **§378c** — ★★ — THE FIFTH DECL-BLOCKER VARIANT: THE DRAFT REDECLARES WHAT THE TU ALREADY OWNS (P31 S69) <sub>L32722</sub>
|
||||
- **§398** — ★★★ — `family_remap` CARRIES THE **SOURCE** TU's DECL ENVIRONMENT INTO A DESTINATION THAT ALREADY OWNS THOSE NAMES (P31 S69; measured 3 banked of 22) <sub>L32835</sub>
|
||||
- **§415** — ★★ — A FILE-SCOPE DECL MAKES gcc-2.7.2 MERGE THE TU'S LATER *BLOCK-SCOPE* EXTERNS INTO IT (P31 S71; byte-proven `ov_SC04_011/func_80180B24`, 215 ins) <sub>L33547</sub>
|
||||
- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) <sub>L34421</sub>
|
||||
|
||||
### jump tables & switches (63)
|
||||
### jump tables & switches (64)
|
||||
|
||||
- **§8** — rodata island (compiler jump tables) — the `.data→.rodata→.data` sandwich (Phase 7) <sub>L339</sub>
|
||||
- **§8a** — rodata island in a flat OVERLAY — the tail sandwich, per matched jr-function (Phase 26 — PoC PROVEN) <sub>L361</sub>
|
||||
@@ -800,6 +801,7 @@
|
||||
- **§431** — ★★★ — SPLITTING A 27,000-LINE TU AT ITS ORIGINAL BOUNDARIES: THE JTBL SPANS TELL YOU WHERE, AND THE COMPILER TELLS YOU WHAT CROSSES (P31 S72; `src/800.c` -> `800.c`/`800_b.c`/`800_c.c`, byte-identical with nothing banked) <sub>L34122</sub>
|
||||
- **§433** — ★★★ — ON A SWITCH FUNCTION, CASE SOURCE ORDER IS THE DOMINANT RESIDUAL — AND `match_one` IS BLIND TO IT (P31 S73; 4 of 5 consecutive main MATCHes) <sub>L34185</sub>
|
||||
- **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) <sub>L34307</sub>
|
||||
- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) <sub>L34421</sub>
|
||||
|
||||
### optimisation level (-O0/-O2) (22)
|
||||
|
||||
@@ -951,7 +953,7 @@
|
||||
- **§420** — ★★★ — A MULTI-CLUSTER SYMBOL REBASE, AND THE BARE-NAME DEDUP THAT HID THREE QUARTERS OF IT (P31 S71; 4 banked in 57 s) <sub>L33692</sub>
|
||||
- **§421** — ★★★ — A `la $tN` + `addiu` PAIR CAN BE A **RELOAD** ARTIFACT THAT NO C SPELLING REACHES (P31 S71; byte-proven `md_SC07_003/func_801A293C`, 313 ins, 6 → 0) <sub>L33729</sub>
|
||||
|
||||
### integration / TU plumbing (79)
|
||||
### integration / TU plumbing (80)
|
||||
|
||||
- **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) <sub>L456</sub>
|
||||
- **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) <sub>L502</sub>
|
||||
@@ -1032,6 +1034,7 @@
|
||||
- **§423** — ★★★ — "MATCH IN ISOLATION + GATE REJECTS + CAUSE NOT DETERMINED" ⇒ GREP THE TU FOR A FILE-SCOPE TYPEDEF THE DRAFT ALSO CARRIES (P31 S71; byte-proven `ov_SC03_092/func_8017FA74`) <sub>L33780</sub>
|
||||
- **§431** — ★★★ — SPLITTING A 27,000-LINE TU AT ITS ORIGINAL BOUNDARIES: THE JTBL SPANS TELL YOU WHERE, AND THE COMPILER TELLS YOU WHAT CROSSES (P31 S72; `src/800.c` -> `800.c`/`800_b.c`/`800_c.c`, byte-identical with nothing banked) <sub>L34122</sub>
|
||||
- **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) <sub>L34307</sub>
|
||||
- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) <sub>L34421</sub>
|
||||
|
||||
### build graph, splat & the harness (196)
|
||||
|
||||
@@ -2807,6 +2810,7 @@
|
||||
- **§434** — ★★★ — TWO SYMBOLS, ONE FRAME: RUN THE FRAME CHECK BEFORE DRAFTING ANYTHING LARGE (P31 S73; `main/SaveLoadRoutine` + `func_8002B0B4`, byte-verified) <sub>L34259</sub>
|
||||
- **§435** — ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", NOT "IS IT ZERO" (P31 S74; four overlays split, all byte-identical, CARVE-BLOCKED class emptied fleet-wide) <sub>L34307</sub>
|
||||
- **§436** — ★★★ — TWO TOOLS THAT READ THE WRONG SOURCE OF TRUTH, AND THE SHAPE THEY SHARE (P31 S74; both fixed + negative-controlled) <sub>L34364</sub>
|
||||
- **§437** — ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical) <sub>L34421</sub>
|
||||
|
||||
|
||||
---
|
||||
@@ -3924,3 +3928,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
|
||||
| L34259 | §434 | ★★★ — TWO SYMBOLS, ONE FRAME: RUN THE FRAME CHECK BEFORE DRAFTING ANYTHING LARGE (P31 S73; |
|
||||
| L34307 | §435 | ★★★ — AN OVERLAY TU SPLIT IS NEAR-FREE, AND THE GAP TEST IS "IS THIS WORD A CODE ADDRESS", |
|
||||
| L34364 | §436 | ★★★ — TWO TOOLS THAT READ THE WRONG SOURCE OF TRUTH, AND THE SHAPE THEY SHARE (P31 S74; bo |
|
||||
| L34421 | §437 | ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern |
|
||||
|
||||
@@ -34417,3 +34417,109 @@ refused to accept "the gate said no" as a fact about its own work.
|
||||
current config), then ask what it READS (an asm tree, or a whole file it only partly owns). Where
|
||||
those differ, the tool will one day report a true fact about a world that no longer exists — and it
|
||||
will be believed, because it is the tool whose job is to be trusted.
|
||||
|
||||
## §437 ★★★ — A WRAPPED TRAILING COMMENT BROKE THE TU PARSER, AND THE §8b DECL LAYER WROTE `extern #define` (P31 S74; five independently-MATCHed jr bodies unblocked, `ov_SC06_029` byte-identical)
|
||||
|
||||
A parallel gate rejected **five** `ov_SC06_029` drafts with the identical, non-codegen verdict:
|
||||
|
||||
```
|
||||
func_801867D0 PLUMBING: src/ov_SC06_029/ov_SC06_029_jr_801867D0.c:138: parse error before '#'
|
||||
func_801898CC … :193 … func_801801D8 … :73 … func_80187660 … :141 … func_80180A70 … :73 …
|
||||
```
|
||||
|
||||
Those `.c` files do not exist in the repo — `jr_isolate_all.py` writes them at gate time so each
|
||||
jr's jump table can carve. Line 138 of the generated file was:
|
||||
|
||||
```c
|
||||
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
|
||||
```
|
||||
|
||||
**The cause is a comment, not a compiler.** The TU carries this, at file scope:
|
||||
|
||||
```c
|
||||
extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU;
|
||||
calls go through a (s32,s32) function-pointer cast, the TU's own idiom */
|
||||
#define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C)
|
||||
extern void func_8012C218(void *a0);
|
||||
```
|
||||
|
||||
Every peeler in `overlay_src_split.py` decided "is this line a comment?" with
|
||||
`line.strip().startswith("/*")` — which sees a comment that STARTS a line and is blind to one a
|
||||
construct OPENS mid-line. The declaration ends at its `;`, which is BEFORE the `/*`, so
|
||||
`scan_construct` returns and the caller resumes **on the comment's continuation line**, entered with
|
||||
`in_block=False`. It then reads comment prose as code, and the prose is hostile: `(s32,s32)` closes a
|
||||
depth-0 paren, so the scan latches `seen_header`, after which every `;` reads as a **K&R parameter
|
||||
declaration** and one "construct" swallows 15 lines up to the next real definition. Two byte-relevant
|
||||
consequences:
|
||||
|
||||
* `parse_overlay_c` anchored a `def` item on a pure DECLARATION run, naming `func_8012C218` — a
|
||||
resident function with no definition anywhere in this TU;
|
||||
* `def_proto` → `_proto_from_lines` then rendered that whole run as the definition's "implied
|
||||
prototype" and prefixed `extern`, producing the line above. `_proto_from_lines` already carried a
|
||||
D1 backstop for a chunk that OPENS inside a comment (P30 S48) — proof the shape had been met
|
||||
before — but the backstop trims the comment tail *after* the construct scan has already over-run,
|
||||
so it could not help.
|
||||
|
||||
A second, quieter defect rode along: `_file_scope_decls` hoisted the col-0 line **verbatim**,
|
||||
unterminated `/*` included, so the region's carried decl layer opened a comment that ran on and
|
||||
**silently ate the next carried declarations** (measured: `extern void func_8012C218(void *a0);` and
|
||||
`extern u8 D_80190348[];` vanished into it). A dropped file-scope decl is a silent byte-changer.
|
||||
|
||||
**The fix is one derived model of the file's comment state, not three line tests** (R33).
|
||||
`overlay_src_split.comment_open_at(lines)` returns, per line, whether that line BEGINS inside a block
|
||||
comment; `parse_overlay_c`, `def_proto`, `split_src_region.parse` and `jr_isolate_all._file_scope_decls`
|
||||
all consult it, and a col-0 decl whose trailing `/*` never closes is truncated at the `/*` before it is
|
||||
carried (comments emit no code, so that is byte-neutral). `_refuse_code_after_comment_close` refuses a
|
||||
file where such a comment CLOSES with code after the `*/`, because that construct could never anchor
|
||||
(R43; measured 0 occurrences across the 4,188 sources the parser is run on).
|
||||
|
||||
**Building that guard immediately found a THIRD defect in the same model.** `_strip` tested for `/*`
|
||||
BEFORE stripping `//`, so a LINE comment containing `/*` —
|
||||
|
||||
```c
|
||||
// src/shared/engine_core.h src/*/*.c (ov_SC07_006_jr_801457A4.c:3968)
|
||||
```
|
||||
|
||||
— was read as opening a block comment. Everything after it stripped to `""` until some later `*/`, so
|
||||
`scan_construct` saw blank lines where real declarations stood. `_strip` now lexes left to right:
|
||||
whichever of `//` and `/*` comes first wins. (7 such lines in 5 sources.)
|
||||
|
||||
**Scale, so nobody reads this as one overlay's quirk.** The wrapped-trailing-comment shape occurs
|
||||
**238 times across 193 of our `.c` files**, and **153 of those are col-0 hoistable declarations in
|
||||
150 files** — i.e. 150 binaries whose next isolation would have carried an unterminated comment into
|
||||
its decl layer. This class was never specific to `ov_SC06_029`; that overlay is simply where a jr
|
||||
isolation happened to cut next to one.
|
||||
|
||||
**A/B over all 4,188 tracked sources, old parser vs new, so the blast radius is measured and not
|
||||
asserted:**
|
||||
|
||||
| | old | new |
|
||||
|---|---|---|
|
||||
| round-trip identity (header + items == file) | 4188/4188 | 4188/4188 |
|
||||
| files whose ITEM LIST changed | — | **2** (`ov_SC02_031_jr_8017AE2C.c`, `ov_SC06_029_jr_8017C954.c`) |
|
||||
| items gained | — | 0 |
|
||||
| items lost | — | 1 per file, each a **phantom `def`** no definition backs (`func_8012C218`; an addr-less `block`) |
|
||||
| malformed implied prototypes | 999 | 984 |
|
||||
| R43 refusals | — | 0 |
|
||||
|
||||
Nothing is gained and nothing real is lost: the only items that disappear are false anchors. **The
|
||||
984 residual malformed prototypes are a DIFFERENT, pre-existing trigger** — a col-0 line that glues
|
||||
several declarations and `DEFINE_func_*()` invocations together (`extern s32 aF…(…) __asm__("");
|
||||
DEFINE_func_8014C4AC() … s32 func_8014C5D0(…)`). `_file_scope_decls`'s final pass splits those on
|
||||
`;` and drops every segment containing a `DEFINE_`, which is why they have never blocked a bank.
|
||||
**Four of them still carry a `#`** (`… DEFINE_func_8014E5B4() #include "" extern void …`, in
|
||||
`ov_SC01_077` ×3 and `ov_SC07_006`) and survive only because the `#` happens to land inside a
|
||||
dropped `DEFINE_` segment — one edit away from being the same `parse error before '#'`. That pass
|
||||
also discards the segment holding the DEFINITION's own implied prototype, and a dropped prototype is
|
||||
a silent byte-changer, so the class deserves its own fix rather than its current luck.
|
||||
|
||||
**The transferable law.** *A comment model that is right for the shape you had in mind is silently
|
||||
wrong for the shape in front of you, and a parser that loses comment state at a chunk boundary will
|
||||
emit that comment's PROSE as C.* When a gate rejects several independently-MATCHed bodies with the
|
||||
SAME error, at line numbers in files that do not exist in the repo, the subject is the generator —
|
||||
make the file appear on disk and read the line before touching a single draft.
|
||||
|
||||
**Negative control (mandatory here, because the fix rewrites what the build compiles):** an
|
||||
UNMODIFIED `make extract && make build -j$(nproc) && make check` of the binary FIRST, then the gate.
|
||||
`ov_SC06_029` was byte-identical before, and byte-identical after with all five bodies banked —
|
||||
`sha1 b7b0d4ae629fdc4f76c59fa146b4fcc78078c9d1`.
|
||||
|
||||
+19
-1
@@ -638,7 +638,25 @@ def _file_scope_decls(items):
|
||||
out.append((block, True))
|
||||
elif seen_types[key] != body:
|
||||
type_conflicts.append((key, seen_types[key], body))
|
||||
for line in text.split("\n"):
|
||||
# THE CARRIED LINE MUST NOT CARRY AN UNTERMINATED COMMENT (P31 S74, byte-witnessed).
|
||||
# A col-0 decl whose trailing `/*` note WRAPS —
|
||||
# extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU;
|
||||
# calls go through a (s32,s32) fn-ptr cast, the TU's own idiom */
|
||||
# — used to be hoisted VERBATIM, opening a comment in the region's decl layer that then
|
||||
# ran on and SILENTLY ATE the next carried decls until the next `*/` (measured in
|
||||
# ov_SC06_029_jr_801867D0.c: `extern void func_8012C218(void *a0);` and
|
||||
# `extern u8 D_80190348[];` vanished into it). A dropped file-scope decl is a silent
|
||||
# byte-changer, so mask on the SAME comment-state model the parser uses (oss.comment_open_at,
|
||||
# R33): skip a line that BEGINS inside a comment, and truncate one that OPENS a comment it
|
||||
# does not close. Comments emit no code, so dropping the note is byte-neutral.
|
||||
_tlines = text.split("\n")
|
||||
_opens = oss.comment_open_at(_tlines)
|
||||
for _li, line in enumerate(_tlines):
|
||||
if _opens[_li]: # interior of a wrapped block comment
|
||||
continue
|
||||
_code, _still_open = oss._strip(line, False)
|
||||
if _still_open: # trailing `/*` that never closes on this line
|
||||
line = line[:line.index("/*")].rstrip()
|
||||
if not line or line[0].isspace(): # col-0 only (block-scope stays put)
|
||||
continue
|
||||
if "{" in line or "}" in line:
|
||||
|
||||
@@ -209,7 +209,19 @@ def def_name(construct_lines):
|
||||
|
||||
def _strip(line, in_block):
|
||||
"""Blank out // and /* */ comments (block state carried) + string/char literals,
|
||||
for brace/paren/semicolon token counting. Returns (code, in_block)."""
|
||||
for brace/paren/semicolon token counting. Returns (code, in_block).
|
||||
|
||||
LEFT-TO-RIGHT, BECAUSE WHICHEVER MARKER COMES FIRST WINS (P31 S74). This used to test for
|
||||
`/*` BEFORE stripping `//`, so a LINE comment containing `/*` — which our own commentary does
|
||||
routinely, e.g.
|
||||
|
||||
// src/shared/engine_core.h src/*/*.c (ov_SC07_006_jr_801457A4.c:3968)
|
||||
|
||||
— was read as OPENING a block comment (`src/*` supplies the `/*`). Everything after it then
|
||||
stripped to "" until some later `*/`, so `scan_construct` saw blank lines where real
|
||||
declarations stood, and `comment_open_at` marked live code as comment interior. Measured: 7
|
||||
such lines in 5 tracked sources. Same class as the wrapped-comment defect above — a comment
|
||||
model that is right for the common shape and silently wrong for the one in front of it."""
|
||||
c = line
|
||||
if in_block:
|
||||
if "*/" in c:
|
||||
@@ -217,16 +229,80 @@ def _strip(line, in_block):
|
||||
in_block = False
|
||||
else:
|
||||
return "", True
|
||||
c = re.sub(r'/\*.*?\*/', '', c)
|
||||
if "/*" in c:
|
||||
c = c.split("/*", 1)[0]
|
||||
in_block = True
|
||||
c = re.sub(r'//.*$', '', c)
|
||||
out, i, n = [], 0, len(c)
|
||||
while i < n:
|
||||
two = c[i:i + 2]
|
||||
if two == "//": # rest of the line is a comment
|
||||
break
|
||||
if two == "/*":
|
||||
j = c.find("*/", i + 2)
|
||||
if j == -1: # opens a block comment that wraps
|
||||
in_block = True
|
||||
break
|
||||
i = j + 2
|
||||
continue
|
||||
out.append(c[i])
|
||||
i += 1
|
||||
c = "".join(out)
|
||||
c = re.sub(r'"(?:\\.|[^"\\])*"', '""', c)
|
||||
c = re.sub(r"'(?:\\.|[^'\\])*'", "''", c)
|
||||
return c, in_block
|
||||
|
||||
|
||||
def comment_open_at(lines, start_in_block=False):
|
||||
"""[bool] per line — does this line BEGIN inside an unterminated /* block comment?
|
||||
|
||||
ONE derived model of the file's comment state (R33), consulted by every line-based peeler in
|
||||
this module. They used to decide "is this line a comment?" from `line.strip().startswith("/*")`,
|
||||
which is blind to the shape that actually occurs 238 times across 193 of our .c files:
|
||||
|
||||
extern void *func_80185C6C(); /* §183 SIGNATURE-adopted-TU;
|
||||
calls go through a (s32,s32) function-pointer cast, the TU's own idiom */
|
||||
|
||||
The construct ENDS at its `;`, which is BEFORE the `/*`, so the peeler resumes on the comment's
|
||||
CONTINUATION line — and `scan_construct`, entered with in_block=False, reads comment PROSE AS
|
||||
CODE. Measured on ov_SC06_029 (P31 S74): `(s32,s32)` in the prose closed a depth-0 paren, so the
|
||||
scan latched `seen_header`, every following `;` read as a K&R parameter declaration, and one
|
||||
"construct" swallowed 15 lines of preamble up to the next real definition. Two consequences,
|
||||
both byte-relevant:
|
||||
* parse_overlay_c anchored a `def` item on a pure DECLARATION run (`func_8012C218`, whose
|
||||
definition is in the resident, not this TU);
|
||||
* def_proto/_proto_from_lines then rendered that run as the definition's "implied prototype",
|
||||
emitting `extern #define CALL_80185C6C (...) extern void func_8012C218();` into the §8b
|
||||
carried decl layer — `parse error before '#'`, which blocked five independently-MATCHed
|
||||
jr bodies from banking in ov_SC06_029 alone.
|
||||
|
||||
A line that begins inside a block comment is never an anchor and never starts a construct, so
|
||||
the peelers skip it; the text itself is sliced by line and preserved verbatim either way.
|
||||
|
||||
`start_in_block` is for a chunk that was CUT out of a larger file mid-comment (an item whose
|
||||
preceding anchor line opened a wrapped comment): its first line already sits inside one."""
|
||||
out, in_block = [], start_in_block
|
||||
for ln in lines:
|
||||
out.append(in_block)
|
||||
_c, in_block = _strip(ln, in_block)
|
||||
return out
|
||||
|
||||
|
||||
def _refuse_code_after_comment_close(lines, opens, where=""):
|
||||
"""R43: refuse a file where a wrapped block comment CLOSES with code after the `*/`.
|
||||
|
||||
Such a line begins inside the comment (so the peelers skip it, above) yet carries a construct
|
||||
that would then never anchor — a silent loss, which is exactly the failure mode this module
|
||||
keeps paying for. Measured 0 occurrences across the 4,188 sources this parser is run on, so
|
||||
refusing costs nothing today and can never become silent later. Split the line if it fires."""
|
||||
for k, ln in enumerate(lines):
|
||||
if not opens[k] or "*/" not in ln:
|
||||
continue
|
||||
rest = ln.split("*/", 1)[1]
|
||||
if re.sub(r'/\*.*?\*/|//.*$', '', rest).strip():
|
||||
raise ValueError(
|
||||
"overlay_src_split: %sline %d closes a wrapped block comment and then carries "
|
||||
"code on the SAME line (%r). That construct can never anchor — put it on its own "
|
||||
"line. (R43: refusing rather than silently dropping it.)"
|
||||
% (where, k + 1, ln.strip()[:120]))
|
||||
|
||||
|
||||
_DECL_KW = ("extern", "typedef", "struct", "union", "enum")
|
||||
|
||||
|
||||
@@ -326,6 +402,11 @@ def parse_overlay_c(src, syms):
|
||||
hdr_end = split_header(lines)
|
||||
header = "\n".join(lines[:hdr_end])
|
||||
n = len(lines)
|
||||
# THE FILE'S COMMENT STATE IS A DERIVED MODEL, NOT A PER-LINE GUESS (P31 S74, R33). See
|
||||
# comment_open_at: the `s.startswith("/*")` test below only ever saw a comment that starts a
|
||||
# LINE, so a construct whose TRAILING comment wraps left every peel branch resuming inside it.
|
||||
opens = comment_open_at(lines)
|
||||
_refuse_code_after_comment_close(lines, opens)
|
||||
items = []
|
||||
i = hdr_end
|
||||
pre_start = i
|
||||
@@ -333,6 +414,9 @@ def parse_overlay_c(src, syms):
|
||||
raw = lines[i]
|
||||
s = raw.strip()
|
||||
# ---- peel non-anchor prefix lines into the accumulating preamble ----
|
||||
if opens[i]: # interior of a wrapped block comment
|
||||
i += 1
|
||||
continue
|
||||
if s == "":
|
||||
i += 1
|
||||
continue
|
||||
@@ -686,8 +770,16 @@ def def_proto(item_text):
|
||||
"""The prototype implied by a `def`-kind item's function definition (None if none)."""
|
||||
lines = item_text.split("\n")
|
||||
n = len(lines)
|
||||
# Same derived comment-state model as parse_overlay_c (R33). An ITEM can also OPEN inside a
|
||||
# comment — its preceding anchor line wrapped one — so seed the state from the D1 test
|
||||
# `_proto_from_lines` already uses: a `*/` with no `/*` before it.
|
||||
_c, _o = item_text.find("*/"), item_text.find("/*")
|
||||
opens = comment_open_at(lines, start_in_block=(_c != -1 and (_o == -1 or _o > _c)))
|
||||
i = 0
|
||||
while i < n: # peel the preamble to reach the definition construct
|
||||
if opens[i]: # interior of a wrapped block comment: never a construct
|
||||
i += 1
|
||||
continue
|
||||
s = lines[i].strip()
|
||||
if s == "" or s.startswith("//"):
|
||||
i += 1
|
||||
|
||||
@@ -20,8 +20,12 @@ Item vram comes from a func_XXXXXXXX name, else a name looked up in config/symbo
|
||||
Brace matching is naive (counts {}); the build's SHA1 check is the backstop if an item with
|
||||
string-literal braces is mis-split.
|
||||
"""
|
||||
import os
|
||||
import re, sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from overlay_src_split import comment_open_at # ONE comment-state model for the whole split chain (R33)
|
||||
|
||||
SYMS_PATH = "config/symbols.us.txt"
|
||||
|
||||
|
||||
@@ -92,6 +96,11 @@ def parse(src):
|
||||
"""Return (header, [item_text, ...]) splitting top-level items."""
|
||||
lines = src.split("\n")
|
||||
n = len(lines)
|
||||
# SAME BLINDNESS, SAME CURE as overlay_src_split (P31 S74, cookbook §437): the comment peel
|
||||
# below only recognises a comment that STARTS a line, so a construct whose TRAILING `/*` note
|
||||
# WRAPS (`extern void f(); /* note` + continuation) left this loop treating the comment's prose
|
||||
# as an item. `comment_open_at` is the one derived model of the file's comment state (R33).
|
||||
opens = comment_open_at(lines)
|
||||
i = 0
|
||||
while i < n and (lines[i].startswith("#include") or lines[i].strip() == ""):
|
||||
i += 1
|
||||
@@ -103,7 +112,8 @@ def parse(src):
|
||||
continue
|
||||
start = i
|
||||
# leading line/block comments belong to the following item
|
||||
while i < n and (lines[i].lstrip().startswith("//") or lines[i].lstrip().startswith("/*")
|
||||
while i < n and (opens[i] or lines[i].lstrip().startswith("//")
|
||||
or lines[i].lstrip().startswith("/*")
|
||||
or lines[i].lstrip().startswith("*")):
|
||||
if lines[i].lstrip().startswith("/*") and "*/" not in lines[i]:
|
||||
while i < n and "*/" not in lines[i]:
|
||||
|
||||
Reference in New Issue
Block a user