mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-03 00:05:11 -04:00
fa2ccea3ea3a2ebd9dbf4e8efbe40d38f52fbbbc
28 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
db63fcc673 | tools(phase-35): T6 part 2 — the dead macro branches dropped from 12 live tools; the macro-form guard in tool_census --check (ast scan of every LIVE tool, FROZEN/retired/6 whitelisted detectors excluded; 0 LIVE tools reference the retired form; --guard-root negative control: the pre-T4 tree flags exactly the 12 fixed + 3 retired); progress.py: an empty-bodied included member is EMPTY, not folded into REAL (the R32 bucket assertion caught the double count) — the fleet denominator 363,221 -> 363,680 (+459: SC03_015's 219 single-site macros, SC03_118 +3, the never-extended members of five groups) and REAL 360,744 -> 350,533 with EMPTY up the same; progress.json + README block + fleet table regenerated; SETUP; kit corpus | ||
|
Drew T
|
6cb056c932 | tools(phase-35): T2 — the health chain learns the include-at-site and twin forms while the macro form still builds: one source-dir oracle (corpus.src_dirs/src_dir/twin_of from the Makefile), share_census.header_defs as the one reader of the include form, progress.classify counts include sites (fold keeps 255,632), overlay_src_split's include item kind + loud macro-table refusal, dedup_integrate C2a' (a header must DEFINE func), audit_binaries prelude + twin citizenship (CHECK 3b), cdecl differential not-applicable when the macro header is gone, recursive shared-header globs; the count correction 363,214 -> 363,221 (7 whale include sites never registered; ov_MAIN_012 2,401 -> 2,402) regenerated into progress.json + README | ||
|
Drew T
|
c52190ca86 |
fix(phase-32): T1b (1) — jr_isolate_all keys a bodiless typedef struct Tag Alias; by the ALIAS (§497); ov_SC02_017 dry-run REFUSED -> CLEAN, no source rename
- _type_names returned the TAG for `typedef struct Rec801806C8_s Rec801806C8;`, so the typedef block and the tag's own packed struct definition collided under one key with different bodies and the R43 "CONFLICTING bodies — a rename is needed" refusal fired on legal C. Now keyed by the alias (_TYPEDEF_TAG_ALIAS); the `carried` set learns the alias; `typedef struct X X;` (alias == tag) keeps the old key so a second one still dedupes/refuses. Unit control on 7 block shapes PASS; ov_SC02_017 --only func_80186C64 --dry-run: 2 region files, no carve repoints. cookbook §497; SETUP row. |
||
|
Drew T
|
380ccdc843 |
feat(phase-32): T1a (1) — resident code subseg split (3 regions, byte-identical 8e17e02f) + jr_isolate_all include-derived provided types (§496)
- jr_isolate_all resident --only func_800D128C: [0x4 c resident] [0x12ec c resident_jr_800D00E4]
[0x2494 c resident_jr_800D128C]; the banked jr func_800D00E4's .rodata carve + JTBL_PADS + --order
repointed to resident_jr_800D00E4.o (config/overlays.mk resident block only, R60); make extract +
make build BINARY=resident -j8 rc 0, sha 8e17e02ff8954d07c979449198f7e1645046b353 == check (R53)
- TOOL FIX (R43/R33): the carried-type test consulted _engine_types() (engine_types.h + common.h) for
every TU, assuming each region includes engine_core.h; the resident includes only common.h, so its
file-local `typedef struct {...} CdFileLoc;` (a name engine_types.h also defines) was silently NOT
carried -> `parse error before cdFileLocTable` in both region TUs, build rc 2 while the stale binary
on disk read green. Now _provided_types(header) derives the set from the TU's own #include lines
(engine_core.h => engine_types.h + common.h, never engine_core's macro-internal typedefs; common.h
=> common.h) and _file_scope_decls(items, provided) uses it at both decision points. R39 controls:
overlay header == legacy set (1,197 names); resident set lacks CdFileLoc. cookbook §496; SETUP row
- rtu_match func_800D128C --split resident_jr_800D128C: MATCH (243 ins) on the stored S71 draft;
the gate is the next commit
|
||
|
Drew T
|
cdd535e45b |
fix(tools): reconcile_tu's cc1 premise, the &-cast arms, and the worktree sig gap
The S74 checkpoint's "one unfixed defect that is actively costing banks"
(reconcile_tu manufacturing declaration conflicts), run to ground — plus the
harness gap that produced a false carve-corruption verdict.
reconcile_tu.py — three defects, measured against the real gcc-2.7.2 front end
(cdecl._cc1_accepts, the oracle cdecl.compatible was validated with; R33):
* The premise "a decl BELOW still conflicts" is TRUE at file scope and FALSE
at block scope. cc1 ACCEPTS a block-scope extern against a TU decl below it
(pedwarn "type mismatch with previous external decl"); conforming it is
destructive, because the TU's decl names the TU's TYPE and a type declared
below the splice point is not in scope AT it -- the emitted result gets
"syntax error before 'D_x'". Byte-witnessed on resident:func_800D06E8 (344
ins), whose block-scoped `extern Blk80078E78` became `extern
Struct80078E78`, typedef 388 lines lower. That construct is what this
ladder's OWN scope_demote_drafts (§8d) rung emits on purpose, and three
already-banked functions in that TU use it: one rung undoing another.
* The cast pass rewrote COMMENT PROSE -- 8 rewrites inside one header comment,
including inside a quoted cc1 diagnostic. Now matches on cdecl._mask
(length-preserving, so a mask offset is a source offset) and splices into
the original.
* `&sym` emitted `&` applied to a cast: legal for the scalar arm, `invalid
lvalue in unary '&'` (measured) for the array/fnptr/fnptr_array arms. `&`
now selects a pointer form and consumes itself -- but ONLY with no trailing
subscript, because `&sym[i]` is the address of ELEMENT i and the old code
had that case right. That last clause exists because the R39 negative
control caught the fold as a regression in the first cut of this fix.
gate_stage.py — `--skip-stages` / `GATE_SKIP_STAGES` (loud when used). Stage 0
gates raw drafts first, so a broken rung can only cost a RECOVERY, which is
exactly what makes it invisible: the function it destroys was already failing,
so its DIFF reads as a fact about the function.
verify_worktree.py / jr_isolate_all.py / parallel_gate.py — provision() now
symlinks every .run/sig.*.jsonl (main clone 259, provisioned worktree 0), the
third member of the class holding extracted/ and .run/obj40. parallel_gate was
fixed for this identical bug in S69: two provisioners, no shared list, found
twice; they now cross-reference each other. jr_isolate_all no longer swallows
the resulting FileNotFoundError into `except: continue` -- that turned a missing
index into a confident carve-CORRUPTION verdict over 2,603 of 2,603 functions
(R54). Adds _assert_scan_covered: attempted == raised means the scan measured
nothing, so its zero is an artifact, not a finding (R32).
Verification:
* 4 cc1 probes (the table above), each run on the pinned front end.
* R39 negative control over the stored-draft corpus: 661 adjudicated, 652
IDENTICAL, 9 CHANGED and every one an intended class. 4,173 of 4,864 drafts
unadjudicable (filenames that are not func_<ADDR>) -- stated, not hidden.
* jr_isolate_all ov_SC03_105 --dry-run: unchanged in the main tree.
* make clean/extract/build BINARY=resident -> 8e17e02f... BYTE-IDENTICAL.
Docs ship with the change (R21): cookbook §442/§443, index regenerated (1,112
sections), 3 docs/SETUP.md rows, CURRENT_PHASE S75 log.
|
||
|
Drew T
|
5e10215269 |
feat(md): bank the 4 -O0-stranded functions — and the class is now essentially empty
md_MAIN_011:func_800CF28C · md_MAIN_003:func_800D0268/func_800D0740/func_800D0C50, all byte-verified
from a clean rebuild and counted from the SOURCE: md_MAIN_011 is now FULLY MATCHED (0 open stubs),
md_MAIN_003 is down to 1 (func_800CF3E8).
THE PREMISE I HANDED THE AGENT WAS WRONG, AND IT SAID SO. md_MAIN_011 is already a whole-object -O0
module — no carve was needed. Its real blocker was tools/jtbl_rodata_pads._s_rodata_span ignoring a
trailing `.align`, the SAME defect this session fixed for md_SC07_003 from the other direction: two
agents converged on it independently. Adopted this agent's stricter form (only a TRULY trailing
align rounds `hi`; an interior one is followed by data that sets `hi` higher anyway).
Note WHY it stayed latent: `derive`'s zero_gap self-corrects a 1-3 byte undershoot whenever the next
stream item is an anchor. A C jump table has NO anchor — so the bug can only fire the moment someone
banks a switch function into such an object, and when it fires it accuses the CARVE ("island layout
drift"), not itself.
md_MAIN_003 needed one new -O0 object, and the boundary I proposed (0x1f74 -> 0x1e58) was both too
narrow and off by 0x2B8. The carve made is `md_MAIN_003_o0e` at 0x1308 (vram 0x800D0100) running to
the existing o0c boundary: everything in that span is a §265 verbatim __asm__ body or an INCLUDE_ASM
stub — zero optimizable C — so the whole tail flips with one cut. Proved byte-identical with NOTHING
banked first (§431 discipline), then the three drafts gated one at a time.
TWO MORE GENERAL DEFECTS FIXED IN jr_isolate_all, both of which silently mis-place a boundary:
* an item-less CLOSING region emitted a duplicate `- [off, c, …]` line and the validator refused;
the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing
region look non-empty.
* A §265 VERBATIM __asm__ BODY IS PREAMBLE, AND PREAMBLE IS ASSUMED BYTE-NEUTRAL. It is not — it
emits bytes. `parse_overlay_c` has four addressed-anchor forms and a verbatim body is none of
them, so it attaches to the NEXT anchor: cutting at func_800D0268 would have moved 0x168 bytes of
other functions into the new object while the yaml claimed the region starts higher. New
`_region_emit_start()` derives the yaml offset from the region's CONTENT (item addresses + every
.globl/.ent the text names that resolves inside the object) and takes min(cut, emit), so a
boundary can only move DOWN. Where no verbatim asm is in play it equals the cut — every existing
isolate is unchanged.
BLAST RADIUS PROVEN, not argued: jtbl_rodata_pads is in the build path (`--derive` for md_*/main),
so the agent rebuilt main + all 70 md_* from scratch (71/71) and then ran the full fleet:
**make check-all 213/213 passed, 0 failed**, main 143dbb89 BYTE-IDENTICAL.
CENSUS, denominator asserted (1057 live stubs, 0 without a .s): exactly ONE -O0-prologue stub
remains stranded in an -O2 TU fleet-wide — main:func_8002C410 in src/800_b.c, 299 ins. Nothing more
should be built for this class; the general tool already existed and what was missing was
correctness, not coverage.
|
||
|
Drew T
|
3a886b9652 |
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
|
||
|
Drew T
|
93bfa45561 |
feat(carve): §323 blocker 2 cleared — jr-isolate ov_SC07_000 for func_8017F8B8, byte-identical
The type-name scan matched `}\s*(\w+)\s*;`, which reads `__attribute__` as the name and fails on the following `((` — so a packed file-local typedef never entered the carried set, every decl naming it read as an unknown type, and the isolate refused the whole overlay. Stripping attributes before the scan is the entire fix. |
||
|
Drew T
|
02e1b3a7e6 |
feat(waves): every pack now carries that function's own PAST-ATTEMPT history
tools/journal_notes.py mines the agent journals per (binary, fn) and appends a PAST ATTEMPTS section to the pack; claude_wave_packs.py calls it automatically, so it is the default rather than a step to remember. Idempotent, and R48-safe (a note stamped with a different binary is never served — §238 homonyms). Measured before adopting (S71 wave 1, 50 one-agent workflows over the 210-function real frontier where every target had already refused an earlier wave): * 38/39 MATCH at closeness 0 (97.4%) vs S70's 124/131 (94.7%) on an EASIER pool * 29/39 agents cite a prior attempt as what they used * 4/39 banked by RECOVERING a body that already matched, from a path a note named * 11/39 matched on the first compile The two costs it removes are re-testing a measured-inert lever (§406 lists twelve, §407 fifteen, §410 four — each paid for by an agent and never seen again) and re-deriving a body that already exists on disk. Also: jr_isolate_all places file-local `static` definitions with the region that uses them instead of refusing the whole file. A `static inline` helper (§82.1) has no address by construction, which is not a defect; the R32 guard was refusing these and blocking the isolate on 4 of the 6 overlays whose CARVE-REFUSED functions it is the named remedy for. Two regions using one static is still a hard refusal (duplicating a used static is a byte change, R43). docs: cookbook §411, wave-playbook step 3b, accelerators entry. |
||
|
Drew T
|
fc9b191806 |
fix(jr_inventory): ownership has three sources, not one — unblocks 8 binaries' carves
jr_inventory R32-aborted on 36 committed .rodata carves across 8 binaries with
"ownership is not 1:1 — a stranded/duplicated carve", blocking the whole §322b
carve route. Every one of those binaries is BYTE-GREEN (R22 213/213), so the config
was right and the MODEL was blind (R34). Measured, the two blind spots:
1. THE SUBSEG NAME IS THE OWNERSHIP RECORD -- 32 of 36 (89%). The isolate
convention writes the owner into the name: a carve in `<ov>_jr_<ADDR>` belongs
to func_<ADDR>. Several owners are RESIDENT-range (0x80135D20, 0x8015C32C)
instantiated through a shared macro, so they are not overlay-local definitions
and parse_overlay_c cannot see them at all. Reading the name is R33.
2. A CARVE FOR A STILL-STUBBED FUNCTION IS PENDING, NOT STRANDED -- the other 4.
ov_SC07_010's func_8016AB6C references its carve at 0x801A6460 from an
INCLUDE_ASM stub.
A carve with none of the three still aborts loudly -- that is the real corruption
the assertion exists to catch (§8b func_801734BC class).
jr_isolate_all --dry-run over the carve set: 7 PASS / 10 FAIL -> 15 PASS / 2 FAIL.
The 2 remaining are the §323 file-local-type class §322b already predicted
(ov_SC02_017 typedef, ov_SC03_029 "carry the naming type").
|
||
|
Drew T
|
e2f64a7c62 |
feat(o0): md_MAIN_003 second carve — func_800D12D0 (657 ins) banked as real -O0 C
MY HYPOTHESIS WAS WRONG AND THE AGENT SAID SO. I predicted the ownership oracle was blind to verbatim-asm owners. It is not. 0x800cedf8 is the §154-A LEADING RODATA ISLAND (the module-id header + jtbl/ptr table at segment offset 0), which rodata_carves already exempts via 'off == 0 and sub == ov'. The S68 first carve legitimately renamed that subseg to md_MAIN_003_jr_800D12D0 (§371: spimdisasm rodata migration is same-subseg-only), so the 'sub == ov' conjunct stopped firing and offset 0 leaked in as a 'carve'. The island has NO single owner BY DESIGN -- which is why the exemption exists -- so widening owner kinds could never have restored 1:1. The fix drops one conjunct: offset 0 alone is the honest structural key, because a carve is a table LIFTED OUT OF THE DATA TAIL and can never sit at the segment's own offset 0. Verified across all 213 configs: every offset-0 .rodata piece is an md_* leading island; ov_*/main have none. The R32 hard abort is UNTOUCHED -- this widens the recognised-island set, it does not soften the refusal. NEGATIVE CONTROL (R39) over all 184 binaries with .rodata pieces: OK 182 -> 183, ABORT 2 -> 1, and exactly ONE verdict moved (md_MAIN_003). The remaining us.exe abort (UNOWNED 0x80073238, the LZSS jtbl carve whose owner LzssDecodeSector does not live under src/us.exe/*.c) is byte-identical before and after -- PRE-EXISTING, not newly hidden, and logged rather than silently absorbed. Carve byte-neutral and bank byte-identical, both re-verified by my own rebuild: sha1 dd1b32ecf1103c6f7cf1943d25546a3046e17b14 == config/check.md_MAIN_003.sha. md_MAIN_003 12 -> 11 stubs. THREE o0_subsplit GAPS surfaced and hand-finished, and they must be fixed before the remaining 7 -O0 stubs here are carved: build_new_config drops a cut at the object start so region 0 kept the -O2 name while the tool PRINTED the _o0 name; parse_overlay_c folds pre-anchor text into the FOLLOWING anchor, so a verbatim body inside region 0 attached to region 1; and the island .rodata piece needs repointing to whichever TU ends up holding its emitters. |
||
|
Drew T
|
7a969d1c61 |
feat(o0): md_MAIN_003 carve — the module-binary -O0 route opens, func_800D0D6C banked (345 ins)
The single-object module binaries could not be carved at all: o0_subsplit planned correctly and then jr_isolate_all refused with 'unaddressable content'. That blocked 9 of the 12 remaining -O0-in-an--O2-TU functions fleet-wide, including a byte-correct 345-instruction draft with nowhere to go. THREE ROOT CAUSES behind the refusal, all fixed here: * overlay_src_split.load_ov_syms: an interior YAML comment terminated the symbol-file list. md_MAIN_003's yaml annotates the list body, so only symbols.us.txt loaded and D_800D3200 resolved to None -> refusal. * jr_isolate_all._partition: a trailing content chunk (the verbatim-asm pair after the last addressable anchor) now attaches to the LAST region when every symbol it defines resolves at/after the last cut, instead of hard-refusing. * _file_scope_decls: bare tag forward decls (struct S_D2394;) exempted from the dedupe refusal; plus addr_of's D_<hex8> fallback. THEN A LINK FAILURE THE CARVE CAUSED, worth knowing: spimdisasm migrates rodata referenced by exactly one function into that function's .s ONLY within the same subseg. The carve moved func_800D30D0 into the jr subseg while the .rodata island stayed on md_MAIN_003, so three dlabel string blocks were SILENTLY DROPPED -> undefined reference to D_800CEE58/D_800CEE80. Adding INCLUDE_RODATA does not resurrect them (splat marks them migrated segment-wide and emits nothing). The fix is to rename the .rodata subseg to the jr object, where every island emitter lives. The regenerated func_800D30D0.s came back byte-identical to the pre-carve .s. Makefile: the -O0 glob widened to src/md_*/md_*_o0?.c. Without it the region file compiles -O2 -- byte-neutral while stub-only, but every -O0 draft banked into it would mystery-fail the gate (§362's trap class). This is why the Makefile and tool hunks MUST land with the carve: a fresh clone would otherwise lose the -O0 flag. VERIFIED INDEPENDENTLY of the agent that did it: sha1 dd1b32ecf1103c6f7cf1943d25546a3046e17b14 == config/check.md_MAIN_003.sha, from a rebuild I ran myself; md_MAIN_003 13 -> 12 stubs; func_800D0D6C absent from corpus.stubs. interleave_check's DRIFT on this binary is PRE-EXISTING (identical on a clean tree, verified before any change) -- md_MAIN_003 has no _JTBL_INTERLEAVE block and must not get one; forcing ALIGNED moves the leading rodata island after .text and shifts every address by 0xD8. config/overlays.mk untouched (R59/R60). 8 of the 9 md_MAIN_003 -O0 stubs remain: they need drafts and follow-on carves. |
||
|
Drew T
|
c4380c19e4 |
feat(p31 s67): stranded-draft census + honest jtbl probe — the frontier's biggest class is carve plumbing
MEASURED (denominators in .run/S67_findings.md): * 193 of the 530 open functions ALREADY have a draft on disk (1,885 wave targets seen, 1,521 banked, 171 open-no-draft, 166 never drawn). Classified in their real TUs: 37 MATCH / 67 NEAR / 89 CC1-FAIL. * 159 open functions (30% of the frontier) reference a jump table; 96 are PLAN-REFUSED by build_carve (non-contiguous same-subseg .rodata), 75 non-main across 38 subsegs. Not a codegen wall and not a decl wall — carve plumbing. NEW * tools/strand_census.py — coverage-asserted census + rtu_match classifier + draft staging. Keys binary:fn (R48); classifies each pair once after merging every manifest's view. * tools/o0_detect.py — the -O0 prologue tell extracted from match_one (which parses argv at import and therefore cannot be imported). match_one re-exports it; ONE definition (R33). Wiring it into the classifier turned md_MAIN_003 from 8 NEAR (7 of them >20) into 6 MATCH. Negative-controlled both directions. * tools/scope_demote_drafts.py — §8d as an _xform-contract gate rung. NOT yet exercised. FIXED * jtbl_carve --probe now runs build_carve (a pure planner) and reports plan-refused. It previously called only island_probe, which answers a necessary-not-sufficient question — every blocked function probed "carveable", and S66 priced 32 of them as free on that. * blocker_probe.macro_scope selects the LAST #define per macro name, matching cpp. engine_core.h has 1,037 duplicate DEFINE_func_ names and 4 with DIFFERENT bodies. NOT VALIDATED — DO NOT SCALE * jr_isolate_all: two real defects fixed (carried types deduped by name; header-provided types no longer re-emitted) but ov_SC02_000 STILL fails the byte gate after them. Open lead: file_scope_types carries a block without its enclosing #if guard. 20 of 35 blocked overlays dry-run clean and that number means nothing until one round-trips. 0 functions banked this session. tools-health has ONE pre-existing cdecl defect (1 of 74,749 declarations, func_8017EE08_p55352/struct ZnRec) — cdecl.py and its inputs are byte-identical to HEAD, so it is not from this change. Knowledge banked: cookbook §322/§323/§323a/§323b, decision-log pivot, accelerators #13/#14. |
||
|
Drew T
|
cd1e38ee0b | fix(tools): jr_isolate_all recognises lifted "} __attribute__((...)) X;" typedefs as engine types (5 drafts refused on Block4 after the lift) and drops a cut equal to the object start (a jr function that leads its object made an empty region -> out of order); mk_write refuses to merge a block that moved on disk since the caller read it (R43) (P31 S62 T3) | ||
|
Drew T
|
fabde98078 | fix(tools): three carve-lane defects behind the last CARVE-REFUSED class — jr_isolate_all's decl carrier no longer carries glued DEFINE_…() invocations (it re-instantiated shared bodies as nameless functions in the new region; prototypes carried instead), jtbl_carve treats a full data-side overlap after isolation as the ordinary tail state, and mk_write.write_overlays_mk merges only the caller's changed blocks into the CURRENT file under the lock (base=; the whole-file rewrite raced between parallel carves — ov_SC03_029/105/MAIN_012 drifts) (P31 S62 T3) | ||
|
Drew T
|
d801b8f177 |
fix(config): the registry wiper found — four truncating writes, now atomic and collapse-refusing
ROOT CAUSE of both wipes today. config/overlays.mk was rewritten in four places with
open(mk, "w").write(txt)
(jr_isolate_all.py:593, jtbl_carve.py:1077/1118/1165) — which TRUNCATES to zero first and only
then writes. Three ways that loses the registry: the process dies between truncate and write
(empty file); another process reads inside that window (sees an empty registry); two writers
interleave (a partial line lands after the last good one — this morning's file ended in a stray
`uto.txt` fragment, exactly that fingerprint). The jtbl carve automation runs AT THE GATE, which
is when all three wipes happened, and ONE_PER_GID=0 made it far likelier by putting many more
carve members in every wave.
BLAST RADIUS, measured twice: with no binaries registered, main's object glob sweeps every
overlay's nonmatchings/*.s into MAIN's OBJS and assembles them standalone, so main cannot build,
the main lane correctly refuses against a RED baseline, and every overlay gate rejects every
draft. Waves dn/do banked 0/224 and 0/236; waves ei..em banked 2 of ~1,100 with 675 backlog rows
reading "match_one MATCH but the whole-binary gate rejected" — the local oracle proving the
drafts were byte-correct while the tree could not build them.
tools/mk_write.py is now the only writer: atomic (tmp + fsync + os.replace, so no reader ever
sees a partial file and a crash leaves the original intact), collapse-refusing (a rewrite below
80% of the current line count raises), and flock-serialized.
TWO HONEST LIMITS, recorded rather than papered over:
* Callers still READ outside the lock, so two concurrent carves can each read-edit-write and
the second drops the first's line. That is a LOST UPDATE — a missing line, not a wiped file —
caught downstream by the fleet check and jtbl_pads_fix. Closing it means holding the lock
across read-modify-write in every caller.
* The guard now also refuses when the CURRENT file is under 100 lines. That case cost me
directly: my own verification control overwrote a registry a carve had truncated seconds
earlier, because the collapse check was skipped when the old file was empty. A control must
assert its precondition; mine did not, and now the tool enforces it instead.
|
||
|
Drew T
|
63fcbeda98 |
fix(jr_isolate_all): the md_* leading island is not a jtbl carve
jr_inventory asserts every committed .rodata piece resolves to exactly one banked owner (R32) — true for ov_*, where every .rodata piece was created by jtbl_carve. On md_* the 154-A leading island - [0x0, .rodata, <alias>] is the module's own rodata blob (INCLUDE_RODATA data + every stub's still-migrated table), has no single owner, and made the check abort: md_SC03_076 reported UNOWNED 0x801ef468 and could not be isolated at all. rodata_carves() now takes the alias and skips that one piece. The discriminator was verified across all 213 splat configs: a .rodata piece at offset 0 whose subseg is the binary's own alias exists in exactly the 42 md_* configs and in none of the others, so this is a provable no-op for every ov_*/main config. md_SC03_076 --only func_801F218C now plans cleanly: 1 jr, 2 region .c files, no carve repoints. |
||
|
Drew T
|
ec749fa584 |
fix(phase-30 S48-0b): JTBL_PADS follows its span through jr isolation
The 0b blocker was not "the pads line is left behind" alone — it fails two different ways, and the second one is silent: * bare isolate + `make build`: the stale line arms the pads filter on the RESIDUAL object, which emits no jump table -> `jtbl_rodata_pads: consumed 0 rodata .align(s) but 4 pad spec(s) given` (S47). * isolate -> jtbl_carve (the jtbl_family_bank path): `set_pads_vars` regenerates the block keyed by the CURRENT subseg names, finds no prior spec under the new `_jr_<addr>` name, and DROPS the line. cc1's natural `.align 3` then pads the span's non-8-aligned interior tables and the image shifts — reported only as `built, bytes differ`. - jr_isolate_all.repoint_overlays_mk: repoint the `build/src/<ov>/<sub>.o: JTBL_PADS` target with the `--order` leaf whenever a carve moves; refuse loud if the old object still hosts a .rodata piece (R32). - jtbl_carve.set_pads_vars: second, disagreeing oracle (R34) — refuse when a spec would vanish for a subseg no longer in the carve set (rename/merge drift), instead of silently emitting a padless object. R37 probe: func_801789AC -> ov_SC02_037 went `built, bytes differ` -> BANKED on the whole-binary byte gate. ov_SC02_037's spec is 0,0,0,0 over tables +0x0,+0x14,+0x34,+0x4c — load-bearing (span start is 4 mod 8). |
||
|
Drew T
|
2483fc902a |
fix(tools): jr_isolate_all was SILENTLY DELETING asm-label-alias definitions during a repartition
ROOT CAUSE (byte-witnessed, P30 S38 — the fifth tool with this same blindness).
A function banked under the §37/§73 DEFINITION-SIDE ASM-LABEL ALIAS form is spelled with a private
C identifier and bound to its real symbol by a GNU asm label:
void aF8018A860(s32, s16 *, u8 *, u8 *) __asm__("func_80183AF8"); <- decl, stays in preamble
void aF8018A860(s32, s16 *, u8 *, u8 *) { ... } <- THIS emits func_80183AF8
overlay_src_split.addr_of() resolves `func_<hex>` arithmetically and everything else through `syms`.
`aF8018A860` matches NEITHER, so it returned None — and partition() keeps only items with a
resolved address, so the definition was dropped from EVERY region. The file was then rewritten
without it and nothing said so. One carve of ov_SC02_028 deleted the definitions emitting BOTH
func_80183AF8 and func_80184268; the overlay stopped linking with `undefined reference`, and six
wave-6 drafts were written off against that as a plumbing/compiler wall.
TWO FIXES:
- CAUSE: overlay_src_split now builds an asm-label alias map from the source and resolves a
definition through its EMITTED SYMBOL rather than its C name (verified: aF8018A860 -> 0x80183AF8,
aF8018AFD0 -> 0x80184268 — exactly the two symbols the link was missing).
- SILENCE: partition() and jr_isolate_all._partition() now REFUSE to rewrite a file when any
construct's address does not resolve (R32), instead of discarding it. That guard alone would
have surfaced this the first time it happened.
RESULT: 3 of the 6 alias-class wave-6 drafts bank immediately, for ZERO agent tokens —
func_801884D8 (137 ins) · func_80180B04 (251) · func_801380E0 (438). R22 clean-fleet 140/140.
The other 3 (the three LARGEST: 557/513/710 ins) have a second, size-correlated cause — open.
NOTE FOR THE FLYWHEEL: family_remap._alias_decl_for ALREADY handled this exact form, and its
docstring records the identical lesson ("that blindness was the WHOLE of the h_seq sweep's 137 'no
matched unit' skips. The tool, not the compiler (R35)"). The fix was never propagated. The alias
form needs ONE shared oracle, the way §134 comment-masking ended up on cdecl._mask — five tools
have now independently rediscovered it.
|
||
|
Drew T
|
faf4547345 |
feat(phase-29): func_8017C954 BANKED — jr carve chain cleared; a shared type was PRESENT but INVISIBLE
- BANKED (1,194 ins, ×1 distinct-code). Chain cleared, each step byte-gated before the next was
built on it: one-line fix to jr_isolate_all._engine_types() -> jr_isolate_all --only
func_8017C954 (2 fns / 1 object, NOT the bare 47-fn / 21-object resegment) -> BYTE-IDENTICAL
b7b0d4ae -> jtbl_carve --func func_8017C954 (44-piece carve set + interleave order) ->
BYTE-IDENTICAL -> harvest_verify VERIFIED BYTE-IDENTICAL -> R22 clean-fleet 140/140,
tools-health OK. instr 80.5 -> 80.6%; distinct-code 3,842,906 -> 3,844,100.
- THE DEFECT (tools/jr_isolate_all.py): _engine_types() harvested shared type names with four
patterns -- `typedef ... X;`, `} X;`, forward-decl `struct X;`, fn-ptr typedef -- and a TAGGED
DEFINITION WITH A BODY matches NONE of them. So `struct PW8017E6D8 { int w; }
__attribute__((packed));` at engine_types.h:658 was present in the shared header yet invisible
to the carried-type check, and `extern struct PW8017E6D8 D_801E1EC4;` could not be placed.
MEASURED BLAST RADIUS: 77 such tags in engine_types.h were invisible. One added pattern fixes
all 77.
- WHY THIS COST 20 MINUTES INSTEAD OF A MYSTERY BYTE-DIFF THREE PHASES LATER: the Phase-26 audit
had already turned this predicate's SILENT DROP into a LOUD REFUSAL. The original bug dropped
4,040 col-0 decls, 683 of them function PROTOTYPES -- and a dropped prototype is a SILENT
BYTE-CHANGER (C89 implicit `int f()`; return type drives delay-slot fill in this codebase). The
refusal named the exact symbols and the exact remedy. A loud "I cannot place this" is worth far
more than a green build -- the audit paying for itself, live.
- §81: the 3-step jr-carve chain + why match_one CANNOT see the problem (it masks jal/HI16/LO16,
so a jump-table function reports MATCH while the whole-binary gate reports DIFF, correctly).
Detect with `grep -cE 'jr \$(v0|v1|a0|t[0-9])'` on the target .s + a jtbl_ in asm/<ov>/data/.
ALWAYS use --only: bare would have resegmented 47 jr-functions across 21 objects.
|
||
|
Drew T
|
ea20bdf9f3 |
fix(phase-26a): A9g — jr_inventory: retire the ephemeral roster, derive banked from the image (R33)
jr_inventory's `banked` set was filtered by an EPHEMERAL, gitignored .run/banked_func_*.json roster: a `rm -rf .run` / fresh clone would blind ALL banked jr at once, cross-address siblings (roster named after the exemplar) were structurally invisible, and non-leader banked jr were missed. "The purest R33 case in the group" (audit). FIX (the audit's exact prescription): delete the roster glob + `cand` filter; `banked` is DERIVED FROM THE IMAGE — a real-C def/define fn is a banked jr iff family_remap.reloc_targets shows it references a committed .rodata carve offset (config + image, both durable; cross-address- and non-leader-immune). R32 assertion: every committed carve must resolve to EXACTLY ONE owner or abort (a stranded/duplicated carve is the §8b func_801734BC incident, never silent). Also fixed the adjacent finding: the asm_jr scan's func_-fullmatch dropped the curated-name listCdBuffer jr; now resolved via oss.addr_of(). (The --only path's own fullmatch is left — it parses user input, not the corpus.) Perf: read the overlay image ONCE and pass it to reloc_targets(..., data=) — a new backward-compatible param on family_remap (regression: 0/80 mismatch vs the re-read path). Verified: data-param behavior-identical; the R33 win — ov_SC02_000 now finds the cross-address sibling func_8017FCB0 the roster missed; full-fleet parallel run = 134/134 OK, 0 false aborts, 1336 banked jr == 1336 carves -> 1:1 ownership holds fleet-wide. Byte-safe: jr_isolate_all is not in the make build/extract path (R22-neutral); the change makes future isolations strictly more correct. |
||
|
Drew T
|
2086b15b48 |
fix(phase-26a): A8 — jr_isolate_all silently dropped 683 prototypes: a LATENT BYTE-CHANGER
_file_scope_decls() hoists a region's file-scope decls into the carried layer. Its _SAFE_TYPE guard
only ever whitelisted BUILTIN base types — so a decl naming a carried file-local type was recognised
by _HOIST_RE as hoistable and then SILENTLY DROPPED.
THE COMMENT ON _SAFE_TYPE DESCRIBED A FIX THAT WAS NEVER APPLIED TO THE CODE. Verbatim: "one naming a
FILE-LOCAL type is only safe once that type is carried too — which file_scope_types() now does, so such
decls ride along after their typedef." The predicate never implemented it.
MEASURED (audit: 4,040 fleet-wide; independently re-measured here over 4 overlays: 189 drops, 32 of
them function PROTOTYPES):
* The 3,357 dropped DATA externs are LOUD — an undeclared identifier is a compile error.
* The 683 dropped function PROTOTYPES are NOT. In C89 an undeclared function is implicitly `int f()`,
so the TU still COMPILES — with the wrong return type and lost pointer-ness. And this project has
BYTE-PROVEN that the return type drives codegen (cookbook: "schedule — delay-slot fill via void
return type"; ov_SC01_077_after.c carries an `extern int`->`extern void` flip described as
byte-neutral precisely because the return type moves the delay slot).
=> A DROPPED PROTOTYPE IS A SILENT BYTE-CHANGER, armed to fire on the next carve.
Today's split is green only because the source redundantly re-declares externs per fn-group, so most
items happen to carry their own decl. That is luck, not design.
Two of the rejected base types were not even file-local: `uint` (139 drops) and `code_fn` (21) are
DEFINED IN src/shared/engine_types.h, which engine_core.h pulls into every region — the predicate was
rejecting INCLUDE-PROVIDED types it had no reason to reject. `volatile` (3) fell off because the
qualifier group had `const` but not `volatile`.
FIX: implement what the comment promised. A decl is hoistable if its base type is a builtin, OR is
carried by this layer's own file_scope_types, OR is provided by the shared headers (108 type names
parsed from engine_types.h + common.h — including FN-PTR typedefs, whose name sits inside the parens
and which every name-before-';' pattern misses; those were exactly the 5 residual drops).
COVERAGE ASSERTION (R32): a line _HOIST_RE recognises as hoistable but that cannot be placed is now a
HARD FAILURE with the base-type histogram printed, not a silent no-op. Verified safe: 0 residual drops
across 5 overlays. This one check would have surfaced all 4,040 the day the first split shipped.
No build impact (the isolator runs only when carving); --dry-run clean; tree unchanged.
|
||
|
Drew T
|
07ebb5658d |
fix(phase-26): jr_isolate_all empty-region0 skip — cutting an already-isolated region's non-leader works
Cutting func_80178D40 out of ov_SC01_000_jr_801734BC adds the region's banked LEADER (0x801734BC) as a cut too (the one-carve-per-object rule), making region 0 EMPTY (the object's first item IS the first cut) — and region 1's derived name equals the object name, so emitting region 0 duplicated the line exactly -> splat "segments out of order". Skip an empty region 0; region 1 rightly claims the object's offset and name. First sibling then banks through the full chain (isolation validation green -> carve -> --raw remap -> stage ladder -> whole-binary gate): ov_SC01_000 BANKED, included here. The remaining 132 siblings sweep next. |
||
|
Drew T
|
7e4165676d |
fix(phase-26): isolation-residue corruption chain — config cleanup + revert() restores config + fail-loud validation + --raw sweep mode
Three-layer fix for the func_80178D40 ×133 sweep failures: - LAYER 1 (the residue): jtbl_family_bank.revert() restored carve pieces + src/ but NOT the isolation's CODE-subseg lines in the splat config. A failed bank attempt (BEBC's first try) left its isolation config in place; the successful retry re-isolated on top and a DUPLICATE `- [0x4b364, c, ov_SC01_000_jr_801734BC]` line rode into the commit (harmless to splat — zero-length — so R22 stayed green). revert() now also restores config/splat.<ov>.yaml. The committed duplicate is removed (ov_SC01_000 rebuilt BYTE-IDENTICAL 9052dc0e). - LAYER 2 (the detonation): jr_isolate_all walked the duplicated object TWICE -> two replacements -> a reversed duplicate block -> splat "segments out of order". It now VALIDATES the generated config (code subsegs strictly ascending, names unique) and refuses to write on violation, naming the likely cause — a corrupt input dies at the tool, not three tools later. - LAYER 3 (the sweep template): jtbl_family_bank gains --raw <crack.c> — template from the RAW crack via remap_hseq_body instead of the exemplar's banked source unit. REQUIRED when the exemplar banked at the `reconciled` stage: a reconciled body is TU-SPECIFIC (§41c — uniquified type names, TU-targeted casts), so extract_unit hands the sweep a polluted template and every sibling gate-fails (byte-proven: 178D40 banked reconciled -> sweep 0/4; 8015AE2C banked raw -> sweep 133/133). Same law as family_sweep --reconcile-raw. |
||
|
Drew T
|
9b93c254c2 |
feat(phase-26): func_8015AE2C (562 ins, x134) banked — Fable5 MATCH + 3 isolation bugs fixed
Exemplar banked byte-identical (d19c9580); R22 clean-fleet 136/136. Fable5 crack: MATCH 562/562, pin-free, jump table verified. THREE REAL BUGS the bank exposed in jr_isolate_all (each byte-proven; each would have silently corrupted every future heavy-core bank): 1. --only filtered `banked` as well as the cut set, so already-banked jr went untracked and their carves were never followed. --only selects what to CUT; it must not erase the record of what is already banked. 2. carve ownership was read from splat .s — but splat emits NO .s for a MATCHED function (its .c holds real C), so the lookup found nothing. Now resolved from the extracted IMAGE via family_remap.reloc_targets (byte-exact: func_801734BC -> 0x801d8c68 etc). 3. THE STRUCTURAL ONE: a region may host at most ONE .rodata carve, because an object's .rodata is a single CONTIGUOUS section. Cutting at func_8015AE2C (jtbl 0x801D8B54) left the banked func_801734BC (jtbl 0x801D8C68) inside the same region, so the object emitted a 0x34 .rodata spanning BOTH tables (image +33 B). Every already-banked jr in a cut object is now cut too -> exactly one carve per object. Cookbook 8b's "bank same-subseg families ASCENDING" note warned about this; it is now enforced by construction instead of left to discipline. Also required (per the crack's own analysis, all byte-verified): - engine_core.h: DEFINE_func_8015BEC4's zero-arg thunk returns func_8015AE2C(), so the extern must drop its (void) prototype and the def must stay K&R/unprototyped. Byte-neutral across all 136 (R22 green). - recovery chain: cast_call_sites (27 callees) + reconcile_decls (3 data syms). The raw body declares callees with types that conflict with their real engine_core.h defs; the original never redeclares them, it CASTS at the call site (cookbook 20). Layout now exact: .rodata 0x801d8b54/0x1c (7 entries, pad trimmed) + 0x801d8c68/0x14 + 0x801d92a0/0x20 — one table per object, each at its true address. |
||
|
Drew T
|
c66b530f71 |
fix(phase-26): bound the §8b carried decl layer + cross-address sibling naming
Two bugs the func_80182268 sibling sweep exposed (both would have silently capped every future jr family bank): - extract_unit walks BACKWARD from a definition absorbing preceding extern/comment lines as the fn's preamble. The §8b carried decl layer sits directly above the FIRST item of an isolated region, so the unit swallowed the whole layer -> the template dragged ~140 unrelated externs into each sibling (some naming types the sibling TU lacks) -> gate-fail. jr_isolate_all now emits an explicit end-marker and extract_unit stops at it (also guards the Phase-17 canonical-sig layer). - jtbl_family_bank passed the EXEMPLAR's name to the sibling's carve/isolate/stub lookup. Cross-address families (same engine fn at a different vram per overlay) therefore never resolved: ov_SC01_077 @0x80182268 -> ov_SC02_000/003 @0x8017FCB0. The sibling's name is now derived from to_addr. The first two banked jr families were same-address, so this had never surfaced. ov_SC01_077 d19c9580 byte-identical; R22 clean-fleet 136/136. |
||
|
Drew T
|
38ac5659aa |
feat(phase-26): §8b scoping wall BROKEN — decl-environment reconstruction + lazy per-core isolation
The full 54-jr isolate-all on ov_SC01_077 now builds d19c9580 BYTE-IDENTICAL
(R22 clean-fleet 136/136) — the configuration session 5 could not build. The
heavy-jr harvest (191 cores / 5.53M templatable ins) is unblocked.
- R14 CORRECTION: session-5's "gcc-2.7.2 block-scope-extern TU-persistence" root
cause was WRONG. There is no gcc quirk — DEFINE_func_* macros expand at FILE
scope, so their leading externs are genuine file-scope decls that merely live in
engine_core.h, invisible to any col-0 .c scan (1377 macros / 3929 lines / 1462 syms).
- REJECTED the approved "global symbol->type map + shadow set" design: the engine is
loosely typed (func_80173544 is DEFINED `s32 f(void*)` yet declared `extern void
f(void);` inside func_801734BC's body), so declaring every USED symbol hoists that
block-scope shadow to file scope and CREATES the conflict a shadow-set then dodges.
Instead reconstruct the original TU's file-scope decl environment and carry it
strictly FORWARD — conflict-free by construction (every carried decl already
coexisted with every definition in the one original TU; compatibility is
order-symmetric; shadows stay in bodies and travel with their item).
- The byte-gate found two MORE lost decl sources, not predicted: (a) a definition is
itself a declaration for everything below it in its TU (func_8012B2CC undeclared);
(b) file-local typedefs used by a carried prototype (parse error, Vec3s). K&R defs
must render `extern T f();` (unprototyped), never f(void).
- LAZY per-core isolation wired into jtbl_family_bank (Drew's call — upfront-x134 =
~7,200 region files): jtbl_carve NON-CONTIGUOUS fail-loud -> jr_isolate_all --only
<core> -> re-extract -> re-carve. Proven on func_80178D40 (890x134, heaviest core):
carve blocked -> isolated (byte-neutral d19c9580) -> carve in its own subseg.
- TWO LATENT BUGS fixed (both would have corrupted the heavy sweeps):
* jtbl_carve.func_subseg derived the owning subseg from the ASM TREE, which `make
extract` never prunes -> after an isolation it returned the STALE owner and
silently re-created the very collision the isolation removed. Now config-derived.
* jtbl_family_bank/jtbl_carve revert() DELETED the shared overlays.mk carve var
unconditionally -> would destroy a COMMITTED carve (all 134 overlays have one) on
any failed sibling. Now restored to its committed value; only region files created
by this attempt are removed; dirty-tree preflight refuses to start a sweep.
- docs: cookbook §8b RESOLVED + new §8c "splitting a TU means rebuilding its
DECLARATION ENVIRONMENT, not moving text"; decision-log 2026-07-13 (R30/R31).
- parser selftest 404/404; R22 clean-fleet 136/136; 0 NON_MATCHING (G4).
|
||
|
Drew T
|
234788dfc4 |
feat(phase-26): §8b overlay-src parser (404/404) + jr isolation tool + the gcc-scoping wall finding
- tools/overlay_src_split.py: overlay-.c-aware partition (header = includes + Phase-17 canonical-sig layer; per-address items = preamble + body; robust def/decl/K&R/DEFINE_func/ SETTER/RETCONST classification). Fleet-validated 404/404 overlay .c, 341,902 items — round-trip exact / 0 unresolved / 0 non-monotonic. The Stage-2 isolation unblock. - tools/jr_isolate_all.py: multi-cut jr resegment (config split at jr boundaries, source repartition + INCLUDE_ASM path repoint, banked-jr carve repoint, -O0 skip, ambient decl carry). SINGLE-cut isolation byte-identical (func_8013FFD8 -> d19c9580, R22). - FINDING (decision-log 2026-07-13): full 54-jr isolation of the dense _after object hits gcc-2.7.2 block-scope-extern TU-persistence (func_801734BC/D_80126B3E declared only in engine_core.h DEFINE_func macros); mechanical TU-split breaks it. Fix = declaration- completion from a global symbol->type map (Drew-approved next step; lazy per-core). - baseline intact (ov_SC01_077 rebuilds d19c9580); no config/src/binary change committed. CURRENT_PHASE session-5 checkpoint + decision-log R31. db.*.gbf = R23 noise, not staged. |