mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
ea75e0f231aa4a08a7e28131bebff5deb03ffe3c
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
6cb056c932 | tools(phase-35): T2 — the health chain learns the include-at-site and twin forms while the macro form still builds: one source-dir oracle (corpus.src_dirs/src_dir/twin_of from the Makefile), share_census.header_defs as the one reader of the include form, progress.classify counts include sites (fold keeps 255,632), overlay_src_split's include item kind + loud macro-table refusal, dedup_integrate C2a' (a header must DEFINE func), audit_binaries prelude + twin citizenship (CHECK 3b), cdecl differential not-applicable when the macro header is gone, recursive shared-header globs; the count correction 363,214 -> 363,221 (7 whale include sites never registered; ov_MAIN_012 2,401 -> 2,402) regenerated into progress.json + README | ||
|
Drew T
|
3a886b9652 |
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
|
||
|
Drew T
|
49c41094a6 |
feat(carve): jr_isolate_all places file-local statics — ov_SC03_010's carve refusal cleared
The CARVE-REFUSED class (10 of the frontier's gate failures) has one dominant cause: "subseg <ov>_jr_<addr> would host NON-CONTIGUOUS .rodata carves", whose named remedy is jr_isolate_all. The isolate itself then refused 4 of the 6 affected overlays over a file-local `static inline` helper (bandsetup, setup_80188D90) that has no address BY CONSTRUCTION — §82.1 helpers exist to shape their caller's code and emit no symbol. * jr_isolate_all now places such a definition with the ONE region that uses it, and refuses loudly if two regions do (two copies of a used static is a byte change, R43). * overlay_src_split._proto_from_lines no longer prefixes `extern` to a declaration that already has a storage class — `extern static inline void f(...)` is "multiple storage classes" to cc1. The two changes are inseparable: placing statics is what first made the tool emit a prototype for one. Byte-gated on ov_SC03_010: extract + build rc=0, sha1 cacaf7c2c08037e6934f9d02c0ae5d7c78cf2463 BYTE-IDENTICAL. jtbl_carve --probe then moves from `plan-refused` to `tail — standard §8a carve at gate time`. |
||
|
Drew T
|
7a969d1c61 |
feat(o0): md_MAIN_003 carve — the module-binary -O0 route opens, func_800D0D6C banked (345 ins)
The single-object module binaries could not be carved at all: o0_subsplit planned correctly and then jr_isolate_all refused with 'unaddressable content'. That blocked 9 of the 12 remaining -O0-in-an--O2-TU functions fleet-wide, including a byte-correct 345-instruction draft with nowhere to go. THREE ROOT CAUSES behind the refusal, all fixed here: * overlay_src_split.load_ov_syms: an interior YAML comment terminated the symbol-file list. md_MAIN_003's yaml annotates the list body, so only symbols.us.txt loaded and D_800D3200 resolved to None -> refusal. * jr_isolate_all._partition: a trailing content chunk (the verbatim-asm pair after the last addressable anchor) now attaches to the LAST region when every symbol it defines resolves at/after the last cut, instead of hard-refusing. * _file_scope_decls: bare tag forward decls (struct S_D2394;) exempted from the dedupe refusal; plus addr_of's D_<hex8> fallback. THEN A LINK FAILURE THE CARVE CAUSED, worth knowing: spimdisasm migrates rodata referenced by exactly one function into that function's .s ONLY within the same subseg. The carve moved func_800D30D0 into the jr subseg while the .rodata island stayed on md_MAIN_003, so three dlabel string blocks were SILENTLY DROPPED -> undefined reference to D_800CEE58/D_800CEE80. Adding INCLUDE_RODATA does not resurrect them (splat marks them migrated segment-wide and emits nothing). The fix is to rename the .rodata subseg to the jr object, where every island emitter lives. The regenerated func_800D30D0.s came back byte-identical to the pre-carve .s. Makefile: the -O0 glob widened to src/md_*/md_*_o0?.c. Without it the region file compiles -O2 -- byte-neutral while stub-only, but every -O0 draft banked into it would mystery-fail the gate (§362's trap class). This is why the Makefile and tool hunks MUST land with the carve: a fresh clone would otherwise lose the -O0 flag. VERIFIED INDEPENDENTLY of the agent that did it: sha1 dd1b32ecf1103c6f7cf1943d25546a3046e17b14 == config/check.md_MAIN_003.sha, from a rebuild I ran myself; md_MAIN_003 13 -> 12 stubs; func_800D0D6C absent from corpus.stubs. interleave_check's DRIFT on this binary is PRE-EXISTING (identical on a clean tree, verified before any change) -- md_MAIN_003 has no _JTBL_INTERLEAVE block and must not get one; forcing ALIGNED moves the leading rodata island after .text and shifts every address by 0xD8. config/overlays.mk untouched (R59/R60). 8 of the 9 md_MAIN_003 -O0 stubs remain: they need drafts and follow-on carves. |
||
|
Drew T
|
dba6defb10 |
fix(phase-30 S48-0b): a def item can OPEN inside a block comment
`_proto_from_lines` starts `_strip` with in_block=False, but item boundaries are `;`-terminated — so a declaration whose TRAILING comment wraps hands the comment's continuation to the NEXT item, and the implied prototype came out as extern * a prototyped (s32) decl is `conflicting types` … */ void func_80151664(void); It compiled only because the hoist emits the opening `/*` line immediately above it, so the garbage lands back inside a comment — but `_file_scope_decls` then meets a col-0 `extern …;` whose base type is `extern` and REFUSES (R32). That is the isolate-fail class: 23 of the 111 open 0b member-slots. - overlay_src_split._proto_from_lines: apply family_remap's D1 backstop — a `*/` with no `/*` before it means the chunk opened inside a comment; drop that residue before parsing the header. - Repaired the 16 already-emitted region files (the garbage line's live payload was a redundant `extern void func_80151664(void);`). Byte-gate after the repair: ov_MAIN_012 / ov_SC02_037 / ov_SC03_107 all BYTE-IDENTICAL. |
||
|
Drew T
|
713b093845 |
fix(phase-30 S47): three splitter defects blocking the monolithic-overlay carve
Found while scoping the jr carve for the 3 newly-onboarded binaries. My scoping said "one
unplaceable construct" — it was the first of four layers. Three are fixed here; the fourth is out
of this tool's scope and leaves the carve blocked.
1. asm_label_aliases: the scan could START inside a #define. `#define gte_SetRotMatrix(r0)
__asm__ volatile ("lw $12, 0( %0 );" ...)` is textually `ident(...) __asm__(...)`, and
cdecl._mask blanks string CONTENT — deleting the `;`s that would stop the greedy [^;{}]*.
The match ran 116 lines and swallowed the real `aF8012EFB8 ... __asm__("func_8012EFB8");`,
so the alias never entered the map and addr_of returned None. jr_isolate_all then refused to
carve (R32, correctly), which presented as 112 isolate-fails that looked like a per-binary wall.
Fixed: _mask_cpp_directives() — a preprocessor directive is the other place a match must not
start. Masking comments/strings fixed the comment case and left this one.
2. _split_macro_body returned a `static inline` internal HELPER as the macro's definition, so
_proto_from_lines hoisted `extern static inline void tail_8012F274(...);` into all 41 regions:
invalid C (multiple storage classes) AND the wrong function — the exported definition sits
below the helper and lost its implied declaration. Fixed: skip static definitions
brace-balanced on the masked body. A static helper needs no hoisted declaration at all.
3. A declaration that WRAPS across continuation lines was taken as one line, so half became a
`;`-less extern and the continuation was read as the definition header, producing
`extern __asm__(""); void aF801466F0(...);` in 22 regions. Fixed: accumulate until the
statement terminates, tested on the masked text. Same wrapped-declaration blindness
family_remap._alias_decl_for records fixing at S33 — never propagated here (§134/§139).
Not fixed, and why: jtbl_rodata_pads reports "consumed 0 rodata .align(s) but 4 pad spec(s) given
— table-count drift vs the carve". The carve moves jtbl-owning functions into _jr_ regions but
leaves the pad specs on the residual gap object. jr_isolate_all's docstring states this class is
NOT isolate-fixable; it needs JTBL_PADS repointing in overlays.mk. 122 jr member-slots stay blocked.
Regression-checked: 1,948 macros parse with 0 malformed externs; alias maps unchanged on three
already-carved overlays. No build impact (splitters run offline). Carve reverted, tree clean.
|
||
|
Drew T
|
3f7534c10a |
fix(tools): the alias scanner I added this morning had the §134 defect I documented this morning
asm_label_aliases scanned the RAW source with a greedy `[^;{}]*` that spans newlines. Byte-witnessed
on ov_SC07_006: a match STARTED inside a comment (`… -> MATCH (40 ins)`), ran through the `*/` and
two blank lines, and swallowed the real declaration below it — so the map recorded
`'MATCH': 'func_80146AFC'` while the actual alias `aF80146AFC` never appeared, and a
`register u8 *p __asm__("$6")` pin contributed `'void': '$2'`.
That is exactly the §134 multi-line-comment class whose project-wide answer is `cdecl._mask` (R33,
ONE masking oracle) — which I described in §139 this morning and then did not use. Sixth tool.
Two non-obvious things the fix had to get right:
- REJECTING a bad match after scanning the source does not work: finditer resumes at the END of the
match it yielded, so the greedy comment-spanning match CONSUMES the real declaration and
rejecting it loses that declaration entirely (verified: aF80146AFC stayed invisible with a reject
in place). The scan must run on the MASKED text so a match cannot start in a comment at all.
- `_mask` blanks string content AND its quotes, so the scan pattern cannot require them. Added
`_ALIAS_SCAN` (quotes optional) for the masked scan; the real symbol is read back from `src` at
the same offsets, which is legal precisely because the mask is length-preserving.
`$N` targets are excluded — a hard-register pin is not a symbol alias.
Found because the R32 partition guard added earlier today REFUSED to rewrite the file rather than
silently dropping the function — the guard working as intended, on its first real encounter.
|
||
|
Drew T
|
2483fc902a |
fix(tools): jr_isolate_all was SILENTLY DELETING asm-label-alias definitions during a repartition
ROOT CAUSE (byte-witnessed, P30 S38 — the fifth tool with this same blindness).
A function banked under the §37/§73 DEFINITION-SIDE ASM-LABEL ALIAS form is spelled with a private
C identifier and bound to its real symbol by a GNU asm label:
void aF8018A860(s32, s16 *, u8 *, u8 *) __asm__("func_80183AF8"); <- decl, stays in preamble
void aF8018A860(s32, s16 *, u8 *, u8 *) { ... } <- THIS emits func_80183AF8
overlay_src_split.addr_of() resolves `func_<hex>` arithmetically and everything else through `syms`.
`aF8018A860` matches NEITHER, so it returned None — and partition() keeps only items with a
resolved address, so the definition was dropped from EVERY region. The file was then rewritten
without it and nothing said so. One carve of ov_SC02_028 deleted the definitions emitting BOTH
func_80183AF8 and func_80184268; the overlay stopped linking with `undefined reference`, and six
wave-6 drafts were written off against that as a plumbing/compiler wall.
TWO FIXES:
- CAUSE: overlay_src_split now builds an asm-label alias map from the source and resolves a
definition through its EMITTED SYMBOL rather than its C name (verified: aF8018A860 -> 0x80183AF8,
aF8018AFD0 -> 0x80184268 — exactly the two symbols the link was missing).
- SILENCE: partition() and jr_isolate_all._partition() now REFUSE to rewrite a file when any
construct's address does not resolve (R32), instead of discarding it. That guard alone would
have surfaced this the first time it happened.
RESULT: 3 of the 6 alias-class wave-6 drafts bank immediately, for ZERO agent tokens —
func_801884D8 (137 ins) · func_80180B04 (251) · func_801380E0 (438). R22 clean-fleet 140/140.
The other 3 (the three LARGEST: 557/513/710 ins) have a second, size-correlated cause — open.
NOTE FOR THE FLYWHEEL: family_remap._alias_decl_for ALREADY handled this exact form, and its
docstring records the identical lesson ("that blindness was the WHOLE of the h_seq sweep's 137 'no
matched unit' skips. The tool, not the compiler (R35)"). The fix was never propagated. The alias
form needs ONE shared oracle, the way §134 comment-masking ended up on cdecl._mask — five tools
have now independently rediscovered it.
|
||
|
Drew T
|
356373efab |
fix(phase-30 S33b): the PAIR rule — my 42-decl relax did NOT unblock the lane; the mirror form did
HONEST CORRECTION to commit:1382. That commit's message implies the 42 `(void)` relaxes unblocked the PROPAGATE remainder. They did NOT: the re-run banked 0/1 in all 134 overlays with the same error, because DEFINE_func_8016BA68 declares func_80146C3C `(u8*)` — the MIRROR of the EXTEND-lane pair — and my relax only touched the `(void)` direction. Root cause is the R37 shape a third time: I bucketed by SYMBOL and stopped. The lever is set by the (macro-shape, TU-shape) PAIR, and the same symbol conflicts in BOTH directions across this fleet. One awk over the macro I was ACTUALLY fixing — which I ran for the EXTEND macros and not for this one — shows the pair before a 134-build run. §138 amended with the PAIR rule; correction logged in CURRENT_PHASE.md rather than rewritten out of history. The 42-decl relax still stands: byte-neutral, R22 140/140, removes a real conflict class. It just did not do what I predicted. THIS commit relaxes the 2 remaining `(u8*)` decls (uses are cast; `()` is compatible with the (void)/()/(u8*) forms the fleet carries and no decl of this symbol has a default-promotion param). R22 clean-fleet: 140 passed, 0 failed. ALSO: tools/overlay_src_split.py `_split_macro_body` — the §134 sweep's one real target, fixed. It carried the identical single-line-only comment test, and it decides where a macro body's file-scope externs END, so a multi-line comment truncated the extern set. SIZED FIRST: 38 live lines in engine_core.h macro bodies hit it today. Now decides on cdecl._mask (one oracle, R33) with the length-preservation invariant asserted (R32). Proven both directions by a control: pre-fix it stopped at `/* multi` carrying 1 of 2 externs and treated the comment as the definition head; post-fix both externs carry and the def head is correct. Not in the gate path (only o0_subsplit + jr_isolate_all import it). |
||
|
Drew T
|
96e025a324 |
fix(phase-26a): A9f — overlay_src_split swallowed 2 real definitions; the selftest was blind
scan_construct's force_decl latched from the FIRST token and returned at the
first depth-0 `;`, so a definition sharing a physical line with leading externs
(`extern A; extern B; void f(){...}`) was never anchored — absorbed into the
next anchor's preamble. The parser jr_isolate_all rewrites source from was short
two functions in the exemplar overlay. The round-trip selftest is a SERIALISATION
check (a miss lands in a preamble -> round-trip still exact BY CONSTRUCTION), so
it was structurally incapable of seeing this.
FIX (byte-safe): force_decl no longer survives a same-line `;` with trailing
code — re-classify from the remainder and keep scanning so the def anchors (its
leading externs stay in its whole-line item text -> round-trip byte-identical).
Rejected the audit's "split into 3 constructs": round-trip joins whole-line
chunks with `\n`, so sub-line splitting would insert a newline where a space was.
def_name now names the LAST top-level header before `{` (the definition, not the
first same-line extern; byte-identical on every single-def construct).
R32: hidden_definitions() coverage oracle wired into selftest — an independent
detector of `func_XXXX(...){` bodies not anchored. The selftest is now a coverage
check, not just serialisation.
Verified: 2 swallowed -> 0; regression over 1738 overlay .c = 0 round-trip fails,
0 non-monotonic, 0 non-additive changes, +2 anchored defs. Byte-safe: tool not in
the build path (R22-neutral); ov_SC01_077 rebuilds d19c9580; neither def straddles
a committed subseg boundary. Audit ledger line refs were stale (src rewritten);
real cases are ov_SC01_077_after.c:2020 + ov_SC01_077_jr_8015444C.c:1495.
|
||
|
Drew T
|
38ac5659aa |
feat(phase-26): §8b scoping wall BROKEN — decl-environment reconstruction + lazy per-core isolation
The full 54-jr isolate-all on ov_SC01_077 now builds d19c9580 BYTE-IDENTICAL
(R22 clean-fleet 136/136) — the configuration session 5 could not build. The
heavy-jr harvest (191 cores / 5.53M templatable ins) is unblocked.
- R14 CORRECTION: session-5's "gcc-2.7.2 block-scope-extern TU-persistence" root
cause was WRONG. There is no gcc quirk — DEFINE_func_* macros expand at FILE
scope, so their leading externs are genuine file-scope decls that merely live in
engine_core.h, invisible to any col-0 .c scan (1377 macros / 3929 lines / 1462 syms).
- REJECTED the approved "global symbol->type map + shadow set" design: the engine is
loosely typed (func_80173544 is DEFINED `s32 f(void*)` yet declared `extern void
f(void);` inside func_801734BC's body), so declaring every USED symbol hoists that
block-scope shadow to file scope and CREATES the conflict a shadow-set then dodges.
Instead reconstruct the original TU's file-scope decl environment and carry it
strictly FORWARD — conflict-free by construction (every carried decl already
coexisted with every definition in the one original TU; compatibility is
order-symmetric; shadows stay in bodies and travel with their item).
- The byte-gate found two MORE lost decl sources, not predicted: (a) a definition is
itself a declaration for everything below it in its TU (func_8012B2CC undeclared);
(b) file-local typedefs used by a carried prototype (parse error, Vec3s). K&R defs
must render `extern T f();` (unprototyped), never f(void).
- LAZY per-core isolation wired into jtbl_family_bank (Drew's call — upfront-x134 =
~7,200 region files): jtbl_carve NON-CONTIGUOUS fail-loud -> jr_isolate_all --only
<core> -> re-extract -> re-carve. Proven on func_80178D40 (890x134, heaviest core):
carve blocked -> isolated (byte-neutral d19c9580) -> carve in its own subseg.
- TWO LATENT BUGS fixed (both would have corrupted the heavy sweeps):
* jtbl_carve.func_subseg derived the owning subseg from the ASM TREE, which `make
extract` never prunes -> after an isolation it returned the STALE owner and
silently re-created the very collision the isolation removed. Now config-derived.
* jtbl_family_bank/jtbl_carve revert() DELETED the shared overlays.mk carve var
unconditionally -> would destroy a COMMITTED carve (all 134 overlays have one) on
any failed sibling. Now restored to its committed value; only region files created
by this attempt are removed; dirty-tree preflight refuses to start a sweep.
- docs: cookbook §8b RESOLVED + new §8c "splitting a TU means rebuilding its
DECLARATION ENVIRONMENT, not moving text"; decision-log 2026-07-13 (R30/R31).
- parser selftest 404/404; R22 clean-fleet 136/136; 0 NON_MATCHING (G4).
|
||
|
Drew T
|
234788dfc4 |
feat(phase-26): §8b overlay-src parser (404/404) + jr isolation tool + the gcc-scoping wall finding
- tools/overlay_src_split.py: overlay-.c-aware partition (header = includes + Phase-17 canonical-sig layer; per-address items = preamble + body; robust def/decl/K&R/DEFINE_func/ SETTER/RETCONST classification). Fleet-validated 404/404 overlay .c, 341,902 items — round-trip exact / 0 unresolved / 0 non-monotonic. The Stage-2 isolation unblock. - tools/jr_isolate_all.py: multi-cut jr resegment (config split at jr boundaries, source repartition + INCLUDE_ASM path repoint, banked-jr carve repoint, -O0 skip, ambient decl carry). SINGLE-cut isolation byte-identical (func_8013FFD8 -> d19c9580, R22). - FINDING (decision-log 2026-07-13): full 54-jr isolation of the dense _after object hits gcc-2.7.2 block-scope-extern TU-persistence (func_801734BC/D_80126B3E declared only in engine_core.h DEFINE_func macros); mechanical TU-split breaks it. Fix = declaration- completion from a global symbol->type map (Drew-approved next step; lazy per-core). - baseline intact (ov_SC01_077 rebuilds d19c9580); no config/src/binary change committed. CURRENT_PHASE session-5 checkpoint + decision-log R31. db.*.gbf = R23 noise, not staged. |