mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-28 14:59:48 -04:00
02f060f607b87e1964b78692b22ffa6197269607
725 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
02f060f607 |
feat(phase-31): S79 #5 — the libpad 4.2.1 + libapi 4.2 band and the apicard region LINKED from real objects: 13 stubs + 4 TUs + the reorder island gone; main 16 stubs, fleet 38
800c3 (0x8005CE18-0x8005FC68, one contiguous run of 33 interleaved Sony objects) is now four
stub rows — libapi1 (21 BIOS trampolines + COUNTER), libpad1 (PADENTRY + PADMAIN 760), libapi2
(L02/L03), libpad2 (PADCMD PADIF PADPORTD PADSEQD WAITRC2) — fed by two WINDOWED psyq_integrate
calls from the raw .run/obj42/{libapi42,libpad421} dirs (integrate tiles each stub with one
library; every boundary checked against .text SECTION sizes). The apicard region's three
"game code" rows were libapi 4.2's C objects to the byte: 800c2 = FIRST.o (firstfile + the
"no jump table wall" stub func_80062144), 800c2_2 = PAD.o, 800c2_3 = PATCH.o + CHCLRPAD.o ->
apicard5/6/7; make_apicard_used.py sources libapi from 4.2 (the EXE's real libapi; libcard
stays 4.0) into .run/obj42/apicard_used, 26 objects / 7 blocks, no game code left in
0x80061F38-0x80062888. src/800c3.c (129 hand-matched "C", 62 verbatim bodies, 19 stubs incl.
the four §332 %lo-in-a-delay-slot "walls"), src/800c2.c, src/800c2_2.c, src/800c2_3.c removed;
REORDER_TUS is empty (mechanism kept). Cookbook §490.
Two stale instruments fixed: exclude_audit let a pinned WALL outrank LINKED (PopMatrix/
PushMatrix had sat as walls since S68 while living in libgte3, linked since Phase 8) — LINKED
dominates now, config/wave_exclude.txt 13 -> 3; frontier_classify carried a hard-coded 49-name
LINKED set (R51) and reported 337 "stubs" — derived from the Makefile now.
Verified: main 143dbb89f34491258bbc27810d0a12ec8b43a8dd WITH all SDK dirs and WITHOUT them from
a fresh extract; make tools-health OK; R22 fleet extract-all 212/212 + check-all 213/213.
Metrics: main REAL 839->773, LINKED 1,150->1,256, VERBATIM 29->3, stubs 29->16, byte-identical
2,075/2,091 = 99.2%; game-code weighted 93.3% (38,748/41,534), remainder 2,786 = the open-stub
sum; fleet stubs 51->38 (frontier_classify: 39 rows incl. the data word). Verbatim manifest
33 -> 6. Docs: worklist rows + "S79 task #5", SETUP (fresh-clone obj42 commands, Makefile
blocks, exclude_audit), decision-log "S79 addendum 2", accelerators "S79 (2)", CURRENT_PHASE
S79 FINAL refreshed (census, metrics, the task #6 brief).
|
||
|
Drew T
|
757bd82a0f |
feat(phase-31): S79 #4 — scattered-.bss split at link-prepare (psyq_bss_split): SYS.o→libgpu2, VM_F.o→snd12, GS_001.o→libgs8 LINKED; libgpu_used retired
The §9.1 "scattered .bss commons" exclusion class (Phase 8 → P31) is closed 3/3. New tools/psyq_bss_split.py (own ELF32 REL reader/writer) cuts an object's packed .bss into per-base NOBITS pieces: bases derived from the game bytes per HI16/LO16 pair, references walked in offset order into single-base runs, cuts snapped to symbol starts (the linker scattered SYMBOLS), symbols moved, a LOCAL section symbol per piece inserted, relocs retargeted with the addend rewritten in the immediates, self-diffed. It runs inside the one prepare step shared by psyq_link.link_object / psyq_link_region.build_region / psyq_integrate.integrate (prepare_object before classify), re-derived every build. GS_001.o was certified "5 interleaved bases, NOT splittable" by the S77 probe, which grouped by BASE; by RUN it is six symbol-aligned pieces. All seven cuts across the three objects are confirmed by the other objects' by-name recoveries (_que 0x800C5510, _svm_sreg_buf 0x800B9B58, PSDBASEX/CLIP2/PSDBASEY/POSITION/GsDRAWENV). R39 negative control: 235 placed objects across 9 curated dirs, 0 refusals, exactly 3 splits (a libcd .bss+size end pointer refused the first build → reference problems are fatal only when a split is needed). Wiring: yaml 800c→libgpu2, sgap_6→sgap_6+snd12, gsgap3→libgs8 (comments rewritten); LIBGPU_ELF := .run/obj40/libgpu (curated libgpu_used retired); libgs 34 objs/8 blocks (make_libgs.sh +GS_001); snd 63/12 (make_snd_used.py exclusions 4→3). src/800c.c and src/gsgap3.c removed (Sony code hand-matched as REAL/verbatim), sgap_6.c keeps only func_8003FA54; splat-emitted libgpu2.c/libgs8.c/snd12.c stubs for the no-SDK fallback. Verified: main 143dbb89f34491258bbc27810d0a12ec8b43a8dd WITH the SDK objects and WITHOUT them from a fresh extract; make tools-health OK; R22 fleet clean extract-all 212/212 + check-all 213/213. Metrics: main REAL 886→839, LINKED 1,040→1,150, VERBATIM 85→29, stubs 29 (unchanged); game-code weighted 91.1% (40,895/44,870) — both terms lost the 3,667 SDK ins; the remainder is still exactly the 3,975-ins open-stub sum. Verbatim manifest --update 200→33 rows (subtractive). Docs: cookbook §489 (+index), psyq-worklist rows + "S78 task #4", SETUP S79 R21 table, decision-log S79 addendum, accelerators S79, CURRENT_PHASE S79 FINAL 🛑. |
||
|
Drew T
|
a85733a487 |
feat(phase-31): S78 #3 — 13 "game code" subsegs were PsyQ objects: wired LINKED (libgte 70/30, libgs 33/7, snd 62/11); main's game-code metric corrected to 91.8%
- exact tiles, 0 tokens: libgte23-26 (MSC01/02/05/09, SMP_00, FGO_01-06, PATCHGTE), libgte9 re-derived as SMP_05 NormalClip (SMP_06 NormalClipS = nested sub-pattern; psyq_integrate now drops nested placements), libgte27-30 (the libgs-gap MTX_05/07/11, REG03+REG11), libgs7 (2D_BG0+2D_BG1), snd10 (VM_NO1), snd11 (VM_NOWON carved off sgap_8). LINKED 959->1040, REAL 912->886 (SDK inline-asm wrappers re-provenanced), VERBATIM 146->85, 13 TUs deleted; splat re-emits the stub records. - main 143dbb89 WITH and WITHOUT the SDK objects. The no-SDK fallback had been red since S7x (CdReadyCallback called by its SDK name while the libcd stub carried func_800435B4) — curated CdReadyCallback = 0x800435B4, refs unified. R22 clean fleet 213/213; tools-health OK. - METRIC CORRECTION (R35): progress.py's "MAIN game-code weighted" sig never excluded the LINKED objects (its comment said it did) — ~31k linked-SDK ins sat in the denominator as unmatched game code. Exclusion now derived LIVE from the Makefile stub lists + yaml ranges: 91.8% (44,562/48,537), not 59.8%; the 3,975-ins remainder equals the open-stub sum exactly. - VM_F.o probed SPLITTABLE at .bss 0x50c (SYS.o's class -> task #4). cookbook §488; worklist S78 #3; decision-log + accelerators; SETUP rows. |
||
|
Drew T
|
a7394f44dc |
feat(phase-31): S78 #12 — the 800c3 "wall" band is LIBPAD 4.2.1 + LIBAPI 4.2: 46 names applied; integrate wired by subseg range; renames via ApplySymbols
- provenance: the psx loader's per-version PsyQ signature sets place PADENTRY/PADCMD/PADPORTD/ PADSEQD (4.2), WAITRC2 (4.3), COUNTER/C114/FIRST/PAD/PATCH/CHCLRPAD (libapi 4.2) byte-exact in 0x8005CE48-0x8005FC68 / 800c2 -> 12 of main's 29 stubs incl. all four §332 walls are Sony's DualShock library in reorder mode. 46 names -> symbols.us.txt (count 1081), band TUs, verbatim manifest, wave_exclude; firstfile/firstfile2 (4.2 naming); CdGetToc @0x800430B8 (was the Phase-21 xdedup mislabel DecDCToutCallback). SETUP §5.1 corrected; psyq-worklist S78; cookbook §487; decision-log + accelerators S78; CHECKSUMS +Psy-Q_46.zip +PSYQ_SDevTC_v4.5.zip. - psyq_integrate: --yaml maps stub<->objects by SUBSEG RANGE with an exact-tiling check and PRINTS the located-but-unwired residue (libgte: 13 objs / 1,264 ins) — main's LINKED build had been RED at HEAD since the S77 psyq_identify fix (22 libgte blocks merged to 3; gate worktrees take the stub fallback so it never showed); a library object's exported symbol whose recovered address the curated file names differently is --redefine-sym'd (R15; A66 firstfile->firstfile2). - Ghidra: 47 MCP renames did NOT persist through the sentinel stop (R9 caught it) -> NEW tools/ghidra_scripts/ApplySymbols.java + tools/ghidra_apply_symbols.sh mirror the curated file headless with a real save: 73 renamed, R9-verified x4. SETUP inventory rows (R21). - lint_symbol_refs: scans verbatim __asm__ bodies (`.ent\tfunc_X` is invisible to \b and to the string-masked scan); negative-controlled (red on the pre-fix TUs, green on the passing tree). - R22: clean extract-all 212/212 + check-all green on the final config; main rebuilt byte-identical 143dbb89 after the last src-only fix -> 213/213; tools-health OK. |
||
|
Drew T
|
375507834c |
docs(progress): main's R34 caveat is retired — its boundaries are independently verified now
The fleet report still printed 'caveat is R34: no independent second oracle for a PS-X EXE'. That was true until this session; make sig-main-oracle + audit-corpus now cover main at 0 phantom / 0 truncated / 1 explained pad-tail. A stale caveat is the same class of false statement as a stale wall verdict. |
||
|
Drew T
|
f030992c67 |
fix(psyq_identify): read .text bytes, not objdump's rendering — 10 more objects located
obj_text_pattern parsed ONE WORD PER DISASSEMBLY LINE, and objdump collapses a run of identical words into a single `...` line. Every collapsed word was silently missing from the pattern, so from the first run onward the pattern was MISALIGNED against the image and find() returned None -- printed as the confident, wrong sentence "not linked by EXE". Measured on 2D_BG0.o (libgs): 3 `...` lines, 520 words parsed for a 526-word object. It was listed as absent while 507 of its 507 non-relocated words match the EXE exactly at 0x8005080C. Any object whose .text holds a run of >=3 identical words was invisible -- to the map the entire library-linking pipeline consumes for placement. That is why 2D_BG0.o was never linked: not excluded by a reason, just invisible. It sits in config/splat.us.exe.yaml under a scattered-.bss exclusion that cannot apply to it, since the object has no .bss section at all. Reading the section bytes and taking relocation offsets from `objdump -r` removes the pretty-printer from the loop (R33). MEASURED: libgs goes from 36/201 to 46/201 objects located. |
||
|
Drew T
|
a13b2a5c38 |
carve(main): 3-way -O0 island split of 800_b for func_8002C410
func_8002C410 MATCHES 299/299 at -O0 and DIFFs 228-vs-299 at -O2 (verified
independently with match_one --o0 vs --no-auto-o0). gcc-2.7.2 has no
per-function optimize pragma, so opt level is per FILE, and the function needs
its own object. Main had no path to one: the Makefile's -O0 wildcard covered
src/ov_*/ and src/md_*/ but NOT top-level src/*.c, and o0_subsplit.py is
overlay-shaped -- it died on config/splat.main.yaml, which does not exist.
Measured the scope first (R37): the -O0 detector flags exactly TWO open main
stubs -- this one, and func_80011380, which already lives in -O0 boot.c and is
the proved floor. So this unblocks one function, not a class.
FIVE COUPLED PIECES, which is why the carve is worth recording:
1. splat code rows: 800_b cut 3 ways -- 800_b / 800_b_o0a / 800_b_2
2. splat .rodata: span B SPLIT, because the 3-way cut put its two jtbl owners
in different objects -- func_8002B0B4 into 800_b, func_800335B8 into
800_b_2 -- and one code object may contribute exactly ONE contiguous
.rodata run. The boundary is DERIVED, not guessed: 800_b.o's compiled
.rodata is 0xf8 bytes, so the front run ends at 0x80072E44+0xf8. The
build's own jtbl_rodata_pads caught the missing piece.
3. src/800_b.c split 3 ways -- 86-line prologue duplicated, 3 defs before the
island, 97 after
4. Makefile -O0 glob widened to top-level src/*_o0?.c
5. ld_interleave --order: 800_b_2.o inserted after 800_b.o. Missing this
floated the tail rodata and shifted every data symbol by exactly its size,
+0x204, across 704 two-byte runs -- which is how it was found.
o0_subsplit.py now REFUSES main loudly instead of dying on a missing file
(R43/R61a) and names the manual procedure.
VERIFIED BYTE-NEUTRAL BEFORE ANY BANKING: main builds
143dbb89f34491258bbc27810d0a12ec8b43a8dd with the split in place and
func_8002C410 still an INCLUDE_ASM stub.
|
||
|
Drew T
|
5399845172 |
feat(psyq_bss_probe): a Phase-8 link exclusion re-derived from the bytes — 3 of 4 objects are not blocked as recorded
The yaml has excluded SYS.o/GS_001.o/2D_BG0.o/VM_NO1.o from the LINKED build
since Phase 8 for 'scattered-.bss commons ... no single NOLOAD base reproduces
it'. Every word of that is true, and it does not imply unlinkable.
psyq_bss_probe derives each object's .bss bases FROM THE BYTES (for each
HI16/LO16 pair against the bare .bss section, the object's immediates give the
addend and the game's give the resolved address, so base = resolved - addend)
and then asks the unasked question: are the offset ranges DISJOINT?
SYS.o 3,109 ins 2 bases 0x0000-0x0044 @ 0x80078830
0x0148-0x0150 @ 0x800c53cc -> SPLITTABLE at 0x148
GS_001.o 384 ins 5 bases interleaved -> the genuine wall
2D_BG0.o 526 ins NO .bss -> reason cannot apply
VM_NO1.o 305 ins NO .bss -> reason cannot apply
§9.2's escape (weaken the .bss symbol, --defsym it) really cannot reach these —
a relocation against the bare SECTION has no name to defsym — and that is what
made 'unlinkable' look like the conclusion. But a section reference only needs
the section PLACED, and a section can be split.
Completeness checked before believing it (R32): the probe counts .bss refs from
EVERY section; SYS.o's .data has zero, so the two-way split covers every
reference. Placement is derived, not configured — the object is located by
masking relocated fields and requiring a UNIQUE match, which independently
reproduced SYS.o @ 0x80059234 / 3,109 ins, agreeing with both the yaml subseg
bounds and the manifest's psyq_identify count.
Incidental: src/800c.c is 100% SYS.o (its span is exactly the object's .text
size), despite the subseg comment calling it '-O2 game code'.
Cookbook §484; yaml comment corrected in the same change.
|
||
|
Drew T
|
867f09221c |
feat(oracle): main gets its independent second oracle — contract §1.3 closed
The roadmap's completion contract requires both audit oracles green before any
100% claim on main, and main had none: audit-corpus covered overlays and
resident only, and R34 is explicit that the byte gate is a perfect CORRECTNESS
oracle and a NULL COVERAGE oracle — green whether a function was sliced right
or invented, because the .s pieces paste back either way.
sig_image gains multi-range signing, closing all three blockers
docs/second-oracle.md scoped:
* the 0x800 PS-X EXE header -> --vram-base 0x8000F800 puts file offset 0 at
vram, so the header falls below the first range
* interleaved data + linked islands -> --segments derives 28 game-code ranges
from the splat yaml's SEGMENT rows
* one text range -> the signer loops ranges, bootstrapping INSIDE each, which
is what stops the linear partition running through a data island and minting
functions out of it (the detector manufacturing the class it detects)
INDEPENDENCE IS PRESERVED, NOT WORKED AROUND. Ranges come from segment TYPES,
never from splat's function boundaries; entries are still found by byte-derived
jal-closure. Seeding from splat's symbols would make every phantom look real —
the trap the design doc names. .run/sig.main.jsonl (the splat-SEEDED atlas sig)
is a different file and corpus.ORACLE_SIG keeps the audit off it.
RESULT: 986 functions signed. main audit = 0 PHANTOM, 0 TRUNCATED, 1 PAD-TAIL.
Fleet audit-corpus = 0 + 0, unchanged for resident and overlays.
NEW AUDIT CLASS, from the first real finding. func_80062144: splat .s 65 ins,
oracle 64 — the extra line is a nop one line BELOW endlabel. That is an
alignment pad the matching side already emits from C (§295; two S77 wave agents
did it on func_8005E13C and func_8005D538), not a mis-slice. Lumping it with
TRUNCATED would make the oracle's first finding look like a defect and bury the
class that is one.
COVERAGE ASSERTED both ways before trusting it (R32): all 30 game-code stubs
fall inside a range, and 0 of 199 addr-parseable LINKED stubs do.
|
||
|
Drew T
|
46097c2339 |
feat(permuter_sweep): hand a wave's NEARs to the permuter, and correct §479 a second time
THE GAP: a drafting agent is briefed to STOP at a plateaued permuter-class residual — right, since an agent grinding a register permutation burns tokens for nothing — so every SCHEDULE-REORDER/DELAY-SLOT/REGALLOC-PERM residual lands unattempted while the local permuter costs no tokens. In S77 the hand-off happened only when I remembered. THE CORRECTION THIS TOOL FORCED. §479 v2 claimed the predictor of a permuter win was 'prior-attempt history: all 3 winners were drafts nobody had worked'. Building the selector on that claim refuted it immediately: journal_notes reports prior attempts for ALL EIGHT known runs, winners included (2, 3, 3). What I had eyeballed was the DRAFT HEADER narrative, a different corpus — the winners came from a recovery pile whose files carry no header journal. That is provenance, not evidence. So the tool selects on the two NECESSARY conditions only (small residual, a match_one class the permuter can search), prints prior-attempt counts as information, and puts the unvalidated filter behind --skip-ground, off by default so it cannot silently discard good work (R39). AND A BUG IN THE NEW TOOL, caught by cross-checking against known-true numbers: wave_results globbed journals across EVERY session and did last-write-wins on a bare function name, so an older wave's row won and carried its stale draft_path — the sweep reported func_8002AC98 at closeness 73 and func_80015608 at 65 while both drafts measure 1 and 3. R48 inside a brand-new tool. Journals are now read newest-last and rows are kept only when the draft lives under this wave's directory. After the fix all seven cross-checkable residuals agree with what the agents independently reported (9, 8, 7, 3, 3, 1, 1). §479 now states the honest position: ~3 in 8 at <=4, no validated predictor, and a note that a yield table is evidence while a story about why is a hypothesis needing its own negative control before it goes in the cookbook. |
||
|
Drew T
|
ec258ff75a |
feat(recover_route): route a gate DROP to the tool that applies, and wire it into gate_main
gate_main printed ONE recovery chain for every dropped draft, and it was the
SELF chain (fix_arity_callers --any-proto + cast_self_callers) regardless of
what the clashing symbol actually was. Two of the three classes are not that
chain:
CALLEE — §378 does not transfer; cast_self_callers reads the return type off
the draft and cannot cast a callee, so --any-proto runs unprotected
over every call site. S69 measured 60 decls no-protoed, binary RED.
DATA — neither tool in the printed chain touches a data extern at all.
Measured cost of the wrong route THIS session: func_8006252C was dropped on a
clash with itself; following the shape of the printed chain I reached for
scope_demote_drafts first, which aliased D_80078D08 through __asm__ and BROKE
the build. The real blocker was one --sync-decls away. Three tools, wrong
order, one destructive — because the report named a chain instead of a route.
A route is an ORDERED LADDER, not a prediction: for a DATA clash the choice
between adopting the TU's spelling and demoting to block scope depends on
whether the draft can live with the TU's type, which no classifier can know.
The byte gate remains the sole arbiter (G3/P9). Refusals come first (R43/R61a):
a verbatim draft and a NEAR are not declaration problems.
NEGATIVE CONTROL (R39): all 7 S77 drops whose winning tool was already known
route correctly — 2 SELF (cast_self_callers), 1 CALLEE (sync_tu_decls via a
definition header), 4 DATA — and the DATA ladder's order matches which rung
actually won in each case (sync for D_80072978, demote for D_80072960 and
D_80074818). Verbatim draft refused; real-C draft not refused.
Playbook §4b and SETUP updated in the same change.
|
||
|
Drew T
|
3005fc1239 |
fix(sync_tu_decls): a no-op sync is not progress, and a repeated symbol is not a blocker
replace_decl returned True whenever the PATTERN matched, even when the substitution produced identical text. So a draft that already carries the TU's exact spelling looped until --rounds ran out, spending ONE CLEAN REBUILD PER ROUND, and then printed 'gave up after 6 rounds (6 synced)' — which reads as six useful syncs. Measured on func_8005FA94: 6 rounds, every one 'D_80072960 -> extern void (*D_80072960)(void *);', zero change to the draft, five wasted rebuilds and a misleading report. R61(a): a no-op must not be reported as work. Two guards: no text change ends the loop naming the already-correct spelling and saying the residual is elsewhere; and a symbol the gate names twice in one run ends it too, since re-syncing it cannot help. The comparison is LINE-NORMALISED because the pattern ends in \s*$ and the substitution eats the matched line's newline — a byte compare called that a change. Caught by a known-true check (identical/different/absent), not by reading the code. |
||
|
Drew T
|
39ac37a808 |
fix(gate_main): the in-TU clash pre-check must only compare FILE-SCOPE declarations
DECL is `^\s*extern`/MULTILINE, so it matched an INDENTED extern inside a function body, and the pre-check then compared that block-scope declaration against the TU's file-scope spelling — making the checker STRICTER THAN CC1. Per cookbook §481, gcc-2.7.2 raises `conflicting types' as an ERROR only in the SAME scope; across scopes it degrades to `type mismatch with previous external decl', a WARNING the build already emits elsewhere. So a block-scope extern cannot clash, and dropping on one refuses correct work. Measured in a single gate: func_8001FC08 (400 ins — a deliberately renamed MTX_8001FC08 at block scope, which is the ONLY legal fix there because two anonymous struct typedefs in one TU are never compatible in C89) and func_8002FF0C (166 ins — a deliberate block-scope scalar shadow of D_800A46D2). 566 instructions of byte-correct body refused by a rule the compiler does not apply. _depth0() blanks brace-nested regions, string literals and comments before DECL runs, on both the TU side and the draft side. NEGATIVE CONTROL (R39): on a synthetic TU it keeps both file-scope decls and excludes the block-scope, in-string and in-comment ones; on the two real drafts it removes EXACTLY the three disputed symbols (D_80074818, D_80075018, D_800A46D2) and leaves all 23 other declarations in each untouched. R39 governs the direction: a check that discards good work is worse than one that lets a failure through, and a real conflict still surfaces via the COMPILE-conflict path plus a byte gate that cannot be fooled. R61(b). |
||
|
Drew T
|
f9f446449e |
feat(claude_wave_packs): wire neighbor_ref into every pack, and resolve its names to the source spelling
playbook §2b has called neighbor_ref the biggest measured cost lever in the wave since S68 (~20x token swing) and documented it as a MANUAL per-card command wired into nothing — so it ran for approximately zero cards. Packs now carry an ALREADY-MATCHED NEIGHBOURS block, same additive never-fail contract as the past-attempt notes. First run: 30/30 targets had a matched neighbour. It also shipped with a defect that would have silently un-done it: neighbor_ref reports the SYMBOL-TABLE name, and for an unnamed function that is Ghidra's FUN_8003a0e4 — which appears nowhere in src/*.c, where the function is func_8003A0E4. An agent sent to read FUN_8003a0e4 finds nothing and concludes there is no neighbour. _src_name resolves against the destination TU's own text, falls back to the address, and shows the symbol-table spelling in parentheses. Measured: 150 of 150 neighbour names needed resolving; 0 primary names remain Ghidra-style. Checked against known-true cases first (resolves FUN_8003a0e4, leaves func_8003A0E4 alone, leaves an unknown name untouched). R61(b): the pack was asserting a name true of the symbol table and false of the world the agent works in. |
||
|
Drew T
|
408f826f29 |
fix(blocker_probe): a verbatim draft is not a decompile
A §265 verbatim draft — the target's own asm in a file-scope __asm__ — assembles to the bytes it was copied from, so BOTH of this tool's oracles emit the strongest possible signal: static `none`, real cc1 `MATCH`. The routing then reads "byte-correct body, nothing blocking it", the byte gate refuses it for free, and progress.py moves by exactly zero. Measured: of the 13 MATCH rows in the S77 overlay pool, SIX were verbatim (md_MAIN_003 x3, md_MAIN_020, ov_SC05_005, ov_SC06_010). The whole 13-draft cohort gated 0, and the probe had scoped it as the highest-value work available. S76 closed exactly this hole in gate_main, harvest_verify and api_agent.prior_draft. This is the fourth consumer — and the one that SCOPES the work, so it is the one whose blindness costs a session's plan. Uses the gate's own detector (DP.is_verbatim_asm_draft) so the two cannot drift (R33), and a VERBATIM row is excluded from the agreement arithmetic rather than counted as a match. NEGATIVE CONTROL (R39): md_MAIN_020's verbatim draft now reports `verbatim_asm / VERBATIM (not a decompile)`; ov_SC04_018's two real-C drafts still report `none / MATCH` exactly as before. |
||
|
Drew T
|
454d3878bc |
fix(sync_tu_decls): a definition is a declaration; a gate refusal is not a verdict
Two defects, both found by driving the last two self_decl_tu drafts to a bank.
1. tu_decl looked only for an `extern … sym …;` line, so when the clashing
symbol is a function the TU DEFINES it stopped with
stopping: func_8005E480 clashes with the TU itself but src/800c3.c has
no `extern` line to copy.
though the authoritative spelling was in the definition's own header at
src/800c3.c:916. This was the terminal blocker of BOTH remaining drafts
(func_8005E3AC on func_8005E480, func_8005E79C on func_8005E804). The
definition is now preferred over an extern when both exist — it is the one
cc1 checks every other declaration against. Banked func_8005E3AC in one
round. Checked against known-true cases before being trusted: definition
path on func_8005E480/func_8005E804, extern path still verbatim on
func_8005D734, absent symbol still None.
2. gate_main refuses outright on a dirty src/ or a red baseline and never
reaches a per-draft opinion. The round loop matched neither DROP_RE nor
COMPILE_RE in that output and fell through to "no declaration conflict
named; stopping after 0 sync(s)" — reporting a HARNESS refusal as a property
of the DRAFT (R40). Measured on func_8005E79C, whose gate was refused
because the bank one command earlier had left src/ uncommitted. The refusal
is now surfaced and exits 3.
|
||
|
Drew T
|
c04d5e0093 |
fix(cast_self_callers): --sync-decls must emit a declaration the TU can parse
`--sync-decls` copied the draft's parameter list verbatim into the TU. A draft
names types that are not in scope where the declaration sits, and both forms
of that broke the COMMITTED baseline build in one apply:
src/800.c:2631 extern void func_80015760(Obj_80015760 *obj, s32 *ot);
-> the type is draft-local; the TU has never heard of it
src/800c3.c:866 s32 func_8005E3AC(Ctx *s, s32 size);
-> `Ctx' is typedef'd at line 941, 75 lines BELOW the decl
src/800c3.c:866: parse error before `*'
src/800.c:2631: parse error before `*'
Caught by gate_main's BASELINE RED check with no draft substituted, so the
failure was attributed to the plumbing and not to seven innocent drafts.
THE FALLBACK FOLLOWS THE TOOL'S OWN DOCTRINE. Once the call sites are cast, the
declaration emits no code; it only has to be COMPATIBLE with the definition and
PARSE. `<ret> fn();` satisfies both without naming a type, and C89 6.5.4.3
makes it compatible with a prototyped definition exactly when no parameter is
affected by the default argument promotions. So the draft's own spelling is
still preferred — it is the byte-proven behaviour and it keeps the declaration
informative — and the no-proto form is used ONLY where that spelling cannot
parse at that line. Where it cannot parse AND a narrow parameter forbids
no-proto, the tool refuses loudly and names the type (R43).
NEGATIVE CONTROL (R39) over all 40 main recovery drafts: 68 edits before and
after, 65 byte-identical. The three that changed are exactly the declarations
naming an out-of-scope type — Obj_80015760, Ctx, and Slot54/Rec14 — and no
already-correct declaration is churned.
BASELINE PROOF: with all 14 plumbing edits applied and NO draft substituted,
main builds 143dbb89f34491258bbc27810d0a12ec8b43a8dd — byte-identical. The
casts move zero bytes, as the §20 fold predicts.
|
||
|
Drew T
|
2312c1f557 |
fix(cast_self_callers): a statement keyword is not a return type
`return func_X(a0, a1);` has the exact shape of a forward declaration —
leading identifier, name, parenthesised argument list, `;` — so every
permissive "<type> <fn>(...);" regex in this tool read that CALL as a
DECLARATION. One misclassification, three consumers, two opposite failures:
* `is_declaration` -> `cast_sites` SKIPPED the call site, leaving the
caller's bytes exposed to the synced (narrowed) prototype.
* `sync_decls` -> REWROTE the whole statement into a declaration,
silently deleting the function's `return`.
* `DEF_RE` -> read the same line as "the definition itself", and
in `draft_signature` could have handed back ret="return".
Witnessed on a dry run before anything touched src/:
src/800.c:713
- return func_80013154(a0, a1, a2);
+ s32 func_80013154(s16 x, s16 y, s16 step);
One shared `_kw_prefixed()` guard, called from all three sites (R33 — the
guard lives in one place, never duplicated into three regexes).
BLAST RADIUS: 524 `return func_X(...);` lines across 482 files fleet-wide.
AUDIT: no past journal records a keyword-prefixed `before`, so no committed
source was corrupted by this.
NEGATIVE CONTROL (R39), old vs new over all 40 main recovery drafts:
68 edits each, 67 byte-identical, zero false positives. The single
difference is the defect itself — the corrupting declaration-rewrite
replaced by the correct cast:
- return func_80013154(a0, a1, a2);
+ return ((s32 (*)())func_80013154)(a0, a1, a2);
|
||
|
Drew T
|
9bdd2e27f7 |
fix(sync_tu_decls): read the post-build conflict too, and refuse a NEAR up front
Two gaps found by running it over all 16 candidates. It only parsed the slate-load 'DROP … clashes with …' path, so five drafts whose conflict surfaced AFTER the build as 'COMPILE conflict on `SYM'' looked unrecoverable when they were the same class one symbol deeper. Both forms are read now. And a CC1-FAIL classification says the declaration blocked COMPILATION, never that the body underneath is right: five candidates compiled once synced and then failed the byte gate because they were NEARs (closeness 12-89) all along. It now scores the body first and refuses a NEAR, so a gate is not spent learning what match_one already knows (R37). That check had the §238 bug it exists to prevent — I called match_one without --asm-subdir, so it defaulted to asm/resident/nonmatchings/resident, judged a DIFFERENT function, returned no verdict, and let the NEAR through. The subdir now comes from the stub oracle. Caught only by controlling the guard against a case whose answer I already knew. Controls: closeness-12 draft REFUSED as a NEAR; func_80013154 still refused as self_decl_tu with the correct redirect. |
||
|
Drew T
|
14f0f91438 |
feat(tools): sync_tu_decls — bank a draft by copying the TU's own declarations
The dominant reason a byte-correct draft does not bank is not codegen: the draft and its destination TU spell a shared symbol differently and gcc-2.7.2 rejects the redeclaration. gate_main's pre-check already NAMES the symbol and which side it kept, and the TU holds the authoritative spelling — so the fix needs no judgement. Copy the TU's extern line verbatim into the draft, re-gate, repeat. Done by hand this session it banked func_8005EB28 in one round and func_8005EC00 in two, both stuck across multiple slates, both byte-identical after. The conflicts are typically a CASCADE: banking one function gives the TU a real definition that then contradicts the stale extern every later draft in that TU still carries. Refuses the self_decl_tu class loudly (the TU declares the function being banked, so the call SITES must change too — that is cast_self_callers --sync-decls), and refuses any binary but main, whose gate is the one that names the symbol (R43). Controls: on an already-banked function it reports no conflict rather than claiming a bank; on func_80013154 it refuses with the right reason. The byte gate remains the sole arbiter — every round ends in a real gate run. |
||
|
Drew T
|
dc4412b1de |
fix(verbatim_to_stub): refuse a §179-C epilogue-less fragment
A function with no `jr $ra` of its own falls into a sibling's shared
epilogue. gcc-2.7.2 has no sibcall/tail-merge pass and appends an epilogue to
every C function it compiles, so no C spelling can ever match — converting one
to an INCLUDE_ASM stub just puts an unbankable target into the drawable
frontier.
I did exactly that to six functions in src/800c.c, on a `rows == 1` filter
that meant "the manifest listed one row", not "this is an independent
function" — ignoring the DECOMPILE-AS-PARENT disposition whose entire meaning
is "this row is a FRAGMENT". Three drafting agents then rediscovered §179-C
from scratch, one citing the cookbook line that names its own target.
The symptom is one grep, so nobody should pay an agent to find it again.
TWO THINGS THIS COST, both caught only by testing a known-true case:
* the first version read the function's .s — but splat stops emitting <fn>.s
for a verbatim body, so it had nothing to read and returned False: inert
for precisely the case it guards. It now reads the verbatim block itself.
* my first negative control was CloseEvent, a libapi trampoline that
genuinely has no `jr $ra` — a "false positive" that was the correct
answer. Re-controlled on VectorNormal (verbatim, has jr $ra, guard stays
silent) vs func_80047E58 (verbatim, no jr $ra, guard fires).
Census of main's verbatim blocks: 37 have jr $ra, 100 do not.
|
||
|
Drew T
|
505a50a9b6 |
fix(draw_waves): --main was a no-op; every mixed draw saw ZERO main functions
bins is built from src/* DIRECTORIES, and main has no src/main/ — its TUs are top-level src/*.c. So "main" was never in the list, and the filter that keeps it could only ever preserve a "main" already present. --only-main worked solely because it overwrote the list; --main contributed nothing, in every mixed draw this project has ever run. The tool meanwhile printed "main: refusing 49 LINKED subseg(s)" whenever --main was passed, so it announced it was handling main while main was never iterated. A flag that changes nothing is worse than a missing flag: it answers the question you asked. Measured: 0 -> 55 main stubs reach the pool. This is why S76y's 47 main targets had to be assembled by hand from corpus.stubs — the draw could not see the actual frontier. Coverage is now ASSERTED (R32): --main with zero main stubs exits 4 and names itself a defect rather than reporting an empty population as a fact. |
||
|
Drew T
|
1778556b6d |
fix(draw_waves): a ledgered stub that is still OPEN is still work
After two S76 draws the tool reported 'population: 0 open stubs' with 51 open stubs on disk. True, and about a scope far narrower than the reader believes — the session's dominant defect class. The draw ledger records what was ATTEMPTED, not a property of the function, so a stub still open after being drawn (the draft was never gated, or the blocker has since been fixed) was filtered forever while the work remained. This is the S72 exclude-list lesson in a second place, and the fix is the same shape: --redraw-open includes them, and the population line now always names how many were filtered for that reason alone, saying explicitly when an empty pool means 'the ledger has seen them all', not 'the frontier is empty' (R41 — a number ships with its denominator). |
||
|
Drew T
|
63d9a36f8d |
fix(rtu_match): route the reorder-island TUs through as -O2, like match_one
The fourth copy of one defect. The Makefile pipes REORDER_TUS through reorder_passthrough.py into as -O2; rtu_match hardcoded maspsx + as -O1, so for those TUs it reported a phantom +1 epilogue instruction and recover_integration --probe-only booked it as a real DIFF. Found by a drafting agent on func_8005D4B8: the already-fixed match_one said MATCH 14/14 while rtu_match said 15/14, and the agent correctly identified its own oracle as the liar rather than the draft. Derived from the Makefile, never copied (R51). |
||
|
Drew T
|
9df4ae32f6 |
fix(api_agent): never warm-start a pack from the target's own assembly
Third door of one defect, and the one that mattered. A §265 verbatim body is stored as <fn>.c like any draft, so prior_draft offered it under 'a previous attempt left this body behind, keep what matches' — an invitation to resubmit it. match_one then says MATCH, the gate goes green, nothing is decompiled. Measured today: gate_main banked 9 such bodies with progress.py moving by exactly zero; harvest_verify had no guard at all; and with BOTH gates fixed, two relaunched agents (func_8005E79C, func_8005EAC8) STILL returned verbatim, because the pack handed it to them and they reasonably reported 'the prior draft is already MATCH closeness 0'. It is — that is the problem. Fixing the consumers is not the same as fixing the supply. Verified on func_8005EAC8: 2 verbatim candidates now rejected with a named reason (R32, never a silent drop) and the warm start falls back to a real C body from wave_m05/shard31. Shared by claude_wave_packs, so every future Claude wave gets it too. |
||
|
Drew T
|
186a8b1548 |
fix(match_one): model the reorder island, not maspsx, for its four TUs
REORDER_TUS := 800c2 800c2_2 800c2_3 800c3 are piped through reorder_passthrough.py into as -O2 by the Makefile — the mode that fills delay slots and emits the jr/addiu epilogue. That island landed 2026-09-01 and banked 20 functions. match_one, the oracle every drafting agent scores against, still compiled those TUs through maspsx + as -O1, so it reported a phantom LENGTH-DRIFT in the epilogue and an extra instruction. Measured on one plain-C draft of func_8005ECC0: maspsx + as -O1 closeness 5, 36 ins vs 35 'the §188 wall' reorder + as -O2 closeness 2, 35 ins vs 35 epilogue identical Cost, in the S76w wave alone: seven of eleven main agents produced correct C, saw the phantom tail, correctly identified the §182/§188 shape, consulted oracle_reorder.py — which told them 'file IMMOVABLE, no C-level work can ever close it' — and each submitted a §265 verbatim-asm body instead. They all reasoned correctly from a false premise the knowledge base gave them. The TU list is DERIVED from the Makefile, never a second copy (R51 — a derived property stored as config goes stale, which is this defect exactly). oracle_reorder.py's docstring is corrected and the cookbook carries the §182/§188 correction with the byte evidence. |
||
|
Drew T
|
dcbcb04bf1 |
fix(harvest_verify): refuse a verbatim-asm draft, same as gate_main
One defect, two doors. gate_main gained this refusal earlier today after 9 main functions round-tripped verbatim -> stub -> verbatim and 'banked' with progress.py moving by exactly zero. The S76w wave then produced verbatim submissions for md_MAIN_003 and ov_SC06_010 — which reach the tree through harvest_verify, not gate_main, so the guard I added would never have fired on them. This is the §442/S74 sibling-provisioner lesson again: a fix made in one of two paths is a fix in neither. Both gates now call the same draft_prechecks.is_verbatim_asm_draft, and harvest_verify SKIPs with a named reason rather than silently dropping (R32/R43). |
||
|
Drew T
|
9ab0d9eb67 |
fix(gate_main): refuse a verbatim-asm draft at slate load
I converted 9 main SDK functions from §265 verbatim bodies to INCLUDE_ASM stubs so they could be decompiled, then 'banked' all 9 from stored drafts that were those same verbatim asm blocks. match_one printed closeness 0 nine times and the whole-binary gate went BYTE-IDENTICAL — both truthfully, since a raw asm blob assembles to the bytes it was copied from. Nothing was decompiled. progress.py caught it by not moving: REAL 882, VERBATIM 164, INCLUDE_ASM 37, identical before and after. The banks are reverted. The cookbook's closing paragraph, written last session, describes this exact trap. I read it and hit it anyway ~4 hours later, because the rule was addressed to 'any burst over this class' and I was hand-picking stored drafts, and because 'no byte gate can catch it' reads as unpreventable. The byte CHECK cannot; a slate-load refusal can. draft_prechecks.is_verbatim_asm_draft: a file-scope __asm__ naming the fn via .ent/.globl/label AND no C definition of it. Both spellings of .ent handled (inside a C string it is a backslash-t, not a tab — five censuses of this class disagreed until that was fixed). gate_main refuses such a slate beside its existing INCLUDE_ASM no-op refusal (R43). Census of the draft store: 1,099 of 704,375 .c files are verbatim-asm drafts under ordinary <fn>.c names. Negative control: 0 false positives across 45,898 drafts carrying both a C definition and an inline __asm__ (R39). |
||
|
Drew T
|
9992cab319 |
refactor(main): convert the last 9 DECOMPILE-NOW verbatim bodies to stubs
The 9 SDK functions the verbatim manifest marks DECOMPILE-NOW in src/800c3.c and src/800c2_2.c were §265 verbatim __asm__ blocks: byte-identical by construction, undecompiled, and unreachable by every gate in the project, which splices a draft in place of an INCLUDE_ASM line these did not have. splat also stops emitting <fn>.s for them, so they had no target asm to match against either. Byte-neutral: main still builds 143dbb89. verbatim_to_stub refused three of them — src/800c2_2.c has no sibling INCLUDE_ASM to copy the subdir spelling from, and all three of its remaining functions are verbatim, so the file can never grow the sibling the rule wants. The tool that exists to reach unreachable functions could not reach them. It now DERIVES the spelling and proves it: the prefix from this binary's other TUs, the last component from the file stem, which must appear as a "c" segment in the binary's own splat config — the same file that decides where splat writes the .s. Still refuses when either half is unproven; --asm-subdir is the explicit override. The S75 checkpoint recorded this group as "20 of 21 banked, one bisection"; counted from src/, it is 9 outstanding, corroborated by an independent count from config/verbatim_manifest.json. |
||
|
Drew T
|
08d49c1715 |
feat(tools): gate main's slates in parallel worktrees, arbiter unchanged
gate_main is the only trustworthy EXE verifier and is strictly serial: one
flock, one tree, a full clean rebuild per bisect step. The serialization is
an artifact of the SHARED TREE, not of the verification, so this runs the
REAL gate_main inside N git worktrees and hands the union of what they prove
to ONE authoritative gate_main in the real tree. Workers discover; only the
final serial pass banks. Two chunks that each pass alone can still fail
together, which is exactly why that pass exists (G3 — the arbiter never moved).
The non-obvious hazard is asm/: parallel_gate symlinks all 442 MB because an
overlay gate only reads it, but main's verification RUNS make extract, which
writes it. main owns 6.2 MB of that, so this copies main's subtree per worker
and symlinks the other 214 binaries read-only.
Two defects the negative control caught, both mine:
* .run/obj40 (11 MB of SDK objects) was never provisioned. The Makefile says
a tree without them 'builds byte-identically via the stubs'; that is no
longer true for main, whose decompiled src/800_c.c CALLS CdReadyCallback —
the link failed outright with an empty build/psyq/.
* make_worktree reads parallel_gate's module-level WT_ROOT, so the first run
put its worktree in .run/pgate/wt0 — the slots parallel_gate force-removes.
Measured: one gate cycle is 16s in both trees, so MAX_STEPS=24 is ~6.4 min
serial and ~90s across four workers. The docstring's original '1-2 min per
step' was my assertion, not a measurement, and is corrected in the file.
|
||
|
Drew T
|
d564b4b4e7 |
feat(gate_main): persist every proven verdict the moment it exists
try_batch is stateless and the bisect loop held `good` only in memory, writing .run/gate_main_banked.json once at the very end. A 34-minute bisection killed by a timeout, a Ctrl-C or a supervisor therefore lost every match it had already PROVEN — and each of those proofs cost a full clean EXE rebuild. The S75 checkpoint named this the single highest-value gate improvement available. Adds an atomic .run/gate_main_progress.json written after every verdict, and a resume that reuses it. Three guards, each a way it could silently lie: the journal must belong to this slate; entries are re-keyed against `kept` so a draft dropped by resolve_conflicts cannot sneak back; and the draft's content hash must still match (R56 — a verdict measures those bytes). Resumed sets are re-verified as one batch anyway, so a wrong reuse costs one rebuild and can never bank anything unproven. --no-resume opts out. Negative-controlled on six cases incl. a changed draft, a foreign slate and a half-written journal. |
||
|
Drew T
|
fc7caf599b |
refactor(tools): retire asm_in_c.py — the taxonomy is DATA now, not a regex census
R33, "the best outcome is a DELETED SCANNER, not a fixed regex". asm_in_c.py
existed to DISCOVER the §265 verbatim class by parsing __asm__ blocks. That job
is done, and regex was the wrong instrument: five successive censuses returned
116 -> 112 -> 108 -> 178 -> 199, and the classification was worse than the count
-- it called 154 rows "game code" where the authoritative answer is 24.
The real answers came from evidence a regex cannot see:
* the <OBJ>_OBJ_<hex> naming key -- every one is placed_object.text_start +
hex, so those symbols are OFFSETS INTO LIBRARY OBJECTS, not functions;
* the PsyQ archive symbol tables in .run/obj40/, which keep statics as W
symbols, so for a byte-identical object the archive IS the function map
(checkRECT = SYS.o+0x52C = func_80059760, and NONE of the 44 SYS_OBJ_*
symbols in SYS.o is a function).
So:
config/verbatim_manifest.json (NEW, committed) -- the authoritative census.
200 rows, derived once from the ROM image + archives + naming key, each with a
class and a DISPOSITION:
PERMANENT-VERBATIM 69 rows / 57 units hand asm; never decompilable
DECOMPILE-AS-PARENT 57 rows / 23 units a FRAGMENT; decompile unit_entry,
never the fragment itself
DECOMPILE-NOW 41 rows / 41 units
DECOMPILE-LOW-VALUE 20 rows / 4 units
UNCERTAIN 5 / NOT-VERBATIM 7 / NOT-CODE 1
tools/verbatim_check.py (NEW) -- a GUARD, not a census. Detects verbatim bodies
(the cheap part, and the only part regex is good at), diffs the NAMES against the
manifest, and reports NEW / GONE / MOVED. A NEW row means someone banked assembly
and it is about to become invisible work; it is never allowed to inherit a
disposition by default. It deliberately does not classify or count units.
Compares case-insensitively on the hex, because an address is a NUMBER (R48).
tools/verbatim_target_s.py -- put on the MANIFEST LEASH. It used to enumerate
every verbatim SYMBOL, and 62 of those are not functions (fragments, bare
epilogue tails, padding, trampolines). Emitting per-symbol targets for them is
what sent two drafting bursts at things no C function can express. It now takes
only DRAFTABLE dispositions: 66 targets emitted, 134 skipped and SAID SO.
tools/verbatim_to_stub.py -- repointed to verbatim_check for detection, so there
is ONE detector in the tree rather than three copies.
tools/asm_in_c.py -- REMOVED.
|
||
|
Drew T
|
254feb8ee4 |
fix(gate_main): the uncommitted-work guard belonged OUTSIDE the bisection loop
I added the guard to try_batch() an hour ago. try_batch runs REPEATEDLY during
bisection, and its own first substitution makes main's TUs dirty -- so on
iteration two the guard could not tell the operator's unsaved work from the
gate's own in-flight edit, and aborted the run:
M src/800c3.c
gate_main: aborting with an UNVERIFIED substitution in main's TUs — reverting
It failed safely (reverted, no bank lost, and said so), but it made the gate
unusable for any batch larger than one.
Hoisted to assert_main_tus_clean(), called ONCE from main() before any
substitution. The lesson is worth the line it costs: A GUARD MUST BE ABLE TO
DISTINGUISH THE STATE IT PROTECTS FROM THE STATE IT CREATES. Placed inside the
loop it was checking its own footprints.
Negative-controlled both directions: a genuinely dirty src/800c3.c is refused by
name before anything is substituted, and a clean tree now proceeds into the
bisection (currently running 21 drafts).
|
||
|
Drew T
|
590fb37447 |
fix(gate_main): refuse to destroy uncommitted main work; name a tool refusal instead of hiding it
TWO DEFECTS, both found by the SaveLoadRoutine decompile and both of which made this gate unable to bank a whole class of function. 1. try_batch() opens with `git checkout -- <main TUs>`. Correct for the normal flow (restore stubs, re-extract, substitute drafts) and CATASTROPHIC for anything uncommitted. Measured twice today: the SaveLoadRoutine decompile (1,179 ins, byte-identical) sat uncommitted while a gate ran and survived only because it was committed first; and a §265 verbatim body converted to a stub is UNCOMMITTED BY CONSTRUCTION, so this line restored the __asm__ block NEXT TO the substituted C -- 9 jump tables instead of 5, jtbl_rodata_pads refused, and the gate REJECTED a byte-identical bank. gate_main could not bank anything in the verbatim class, by construction. Now refuses when main's TUs are dirty, printing the offending paths and telling the operator to commit (R42) or stash. --allow-dirty / GATE_MAIN_ALLOW_DIRTY=1 is the deliberate override. A destructive step that cannot be undone must ASK, not assume. 2. The failure analysis looks for error/undefined/conflict/..., and jtbl_rodata_pads aborts via sys.exit with a message containing none of them -- so a carve REFUSAL surfaced as "only warnings" and the real cause of a rejected bank was invisible. Refusals from jtbl_rodata_pads / jtbl_carve / corpus / jr_isolate_all are now named explicitly as the cause. Negative-controlled both directions: the guard fires on a dirty src/800_b.c naming the file, and --assert-baseline on a clean tree still reports BASELINE GREEN 143dbb89... BYTE-IDENTICAL. |
||
|
Drew T
|
8009f83b40 |
fix(tools): verbatim_target_s wrote targets into asm/, which the Makefile globs as build objects
`build/asm/%.o: asm/%.s` globs the ENTIRE asm/ tree, so the 146 regenerated
target .s files I emitted to asm/verbatim/ were picked up as BUILD OBJECTS and
main went red:
make: *** [Makefile:696: build/asm/verbatim/main/func_80052430.o] Error 1
Default --out moved to .run/verbatim_targets/, which is outside every build
glob. main verified green again: 143dbb89... BYTE-IDENTICAL.
The lesson belongs with the others from this session: a generator's OUTPUT
LOCATION is part of its contract, and asm/ is not a scratch directory. The
failure was invisible until a full build ran -- the tool itself succeeded, the
targets were correct, and nothing about them was wrong except where they sat.
|
||
|
Drew T
|
72fa482027 |
feat(tools): verbatim_to_stub.py — put the 147 asm-posing-as-C functions back where the gates can reach them
THE GAP. Every gate in this project substitutes a draft in place of an
INCLUDE_ASM line. A §265 verbatim __asm__ body has none, so:
gate_main.substitute() resolves each entry through the STUB map; a verbatim
function is reported "resolved to NO stub" and dropped
gate_stage/harvest_verify same splice, same gap
splat stops emitting <fn>.s once a function is not a stub
So all 147 were undecompilable AND ungateable -- not for want of information,
but because the information was in a form nothing consumes.
The fix is not a parallel gate (R33 -- one implementation). It is to put the
function back into the form every existing tool already understands: replace the
__asm__ block with INCLUDE_ASM. That is also the HONEST representation --
INCLUDE_ASM pastes the very same assembly the block transcribes, so the bytes
are identical either way, but a stub counts as OUTSTANDING WORK in progress.py
while a verbatim body counted as banked. The conversion moves a function from
"silently done" to "visibly to do".
Two refusals rather than guesses (R43), because both failure modes are silent
and destructive:
* the block is located by brace/paren MATCHING via asm_in_c.asm_blocks, never
regex-sliced -- these blocks are full of braces and parens inside string
literals and an approximate cut corrupts a file that currently builds;
* the asm subdir for the new INCLUDE_ASM is copied from a sibling stub IN THE
SAME FILE. Subsegs are per-file, so a neighbouring file's spelling names a
different subseg -- a stub with the wrong subdir compiles happily and
includes ANOTHER FUNCTION'S ASSEMBLY. With no sibling to copy, it refuses.
--gate rebuilds and asserts the SHA is unchanged, restoring the file if not:
byte-neutrality here is a claim, and this tool exists to enable a byte gate, so
it declines to be the one link that goes unchecked.
Dry-run verified on main:func_80047E58 -> src/800b.c, 9-line block, subdir
correctly derived as asm/nonmatchings/800b. The --gate proof is deferred only
because another agent is mid-build on main right now.
|
||
|
Drew T
|
70de5a8611 |
feat(tools): verbatim_target_s.py — the 147 asm-posing-as-C functions are workable again; bank func_8017DB98
THE BLOCKER. asm_in_c.py found 147 GAME functions that are §265 verbatim
__asm__ bodies. NONE of them could be worked on: splat emits
asm/nonmatchings/<subseg>/<fn>.s only for functions that are still INCLUDE_ASM
stubs, and a verbatim body is not a stub -- so splat stops emitting its .s,
while match_one and rtu_match BOTH consume one. Measured: 1 of 147 had a target
on disk. The class was unworkable because the information was in the wrong FORM,
not because it was missing.
verbatim_target_s.py regenerates a splat-format target .s from the EXTRACTED ROM
IMAGE -- never from the __asm__ block in our own source, because the block is
the thing under test and a target derived from it would agree with the candidate
by construction (R34). 146 of 147 emitted; the 1 refusal is REPORTED.
TWO DEFECTS CAUGHT BY CHECKING AGAINST A KNOWN-TRUE CASE, both of which would
have shipped ~147 silently-wrong targets:
* BYTE ORDER. splat writes the four bytes as they sit in the image
(`C8FFBD27` for instruction 0x27BDFFC8) and masked_diff.insns_from_s reads
the column with struct.unpack("<I", bytes.fromhex(...)). objdump prints the
VALUE, so reversing double-swaps: 91 of 1139 words agreed with splat's own
.s for the same function. The LENGTH matched perfectly, so nothing except a
word-level cross-check could have caught it.
* `-z` / --disassemble-zeroes. objdump ELIDES runs of zero bytes as `...`, and
a MIPS nop IS 0x00000000 -- so every nop vanished. func_80049610 (three
nops) disassembled to ZERO instructions; func_80047D3C 31 of 36. The length
assertion caught all of them, which is the only reason this was not shipped
as ~30 quietly-truncated targets.
Verification: regenerated SaveLoadRoutine target is 1139/1139 words IDENTICAL to
the .s splat itself emitted for the same function.
ALSO BANKED: ov_SC06_025:func_8017DB98 (122 ins). Its body was byte-exact on
disk since S71 and the blocker was one word: the TU declared
`extern void func_8017DB98(s32, s32)` where the epilogue is `addu $v0,$s3,$zero`
-- must be `extern s32`, and the caller discards the result so the change is
byte-neutral. That line number and fix were recorded in the agent journals the
whole time; frontier_classify only surfaced it once journal_notes was wired in
as a second oracle earlier this session.
|
||
|
Drew T
|
f5f4c2eeec |
feat(decomp): bank func_801806F8 + func_80180ABC (498 ins) + frontier_classify reads the journals
Two banks from the S75 redraft workflow (7 overlay functions, one agent each,
every claimed MATCH re-verified by an independent agent instructed to refute
it). Both were carried as F-FAR "a draft exists but is materially wrong":
func_801806F8 ov_SC03_105 241 ins (recorded closeness 235)
func_80180ABC ov_SC03_105 257 ins (recorded closeness 250)
Neither needed a better model. Both needed the recorded closeness not to be
believed -- see below.
frontier_classify.py, THREE fixes, each caught by testing against a case whose
answer was already known:
1. BEST closeness, not LAST. .run/backlog.jsonl is append-only, one row per
attempt across every lane and session, so the last row is evidence about
THAT LANE'S SEED, not about the function. Caught func_80180B3C (best 125,
last 287) and moved func_80181294 from "redraft" to "permuter" (best 19).
The draft that ACHIEVED the best score is kept, not the last one written.
2. journal_notes.py wired in as a SECOND, DISAGREEING oracle (R34). The backlog
does not have what the agent journals have. Measured on func_8017DB98:
backlog best == last == 115, so best-vs-last could not help, while the
journal holds "Attempt 2 (MATCH · closeness 0) ... MATCH 122/122 ... BANK
BLOCKER is TU plumbing, not the body (§376/§378)" WITH the draft path and the
exact declaration to change. Reclassified 37 functions; G-DRAFTED-UNKNOWN
fell 47 -> 10 and a new C-PLUMBING class holds 16 functions / 1,547 ins whose
BODIES ARE PROVEN and are blocked only by the TU.
3. A consuming-regex bug in my own extractor -- the session's signature defect,
committed a third time in the tool written to find it. The first cut used
`re.finditer(r'\*\*Attempt \d+\*\* \(([^)]*)\)(.{0,400})', ..., re.S)`, whose
400-char body window SWALLOWS THE NEXT ATTEMPT'S HEADER, so every record
following another was invisible. On func_8017DB98 it hid attempts 2 AND 6,
both `MATCH · closeness 0`, and returned attempt 1's NEAR (2) as the best --
exactly the records the oracle exists to find. Now splits on the marker
rather than consuming past it. A regex that consumes an unbounded body cannot
enumerate the items after the first.
Rows now carry attempts, closeness_last, journal_closeness, and a
!!WARMSTART-REGRESSION flag when a later attempt scored materially worse than
the best -- the shape a wave's warm-start regression makes, which from inside
the wave is indistinguishable from an unsolved function.
Gate ledger for the batch of 7: 2 banked, 3 near, 2 failed. func_800CB00C failed
despite being adversarially upheld -- it owns a jump table, and both matchers
compare .text only, so a verified .text MATCH proves nothing about table
placement (the agent's own write-up says so).
|
||
|
Drew T
|
c05ea15cbe |
feat(tools): asm_in_c.py — 154 game functions are assembly wearing a .c extension
A .c file in src/ looks decompiled. 199 functions are not: they are the target
assembly pasted into a C string literal (§265), byte-identical BY CONSTRUCTION
and completely unexplained. 45 are PsyQ/CRT routines where that is defensible;
154 are GAME CODE, 171 of the 199 in main, the largest being SaveLoadRoutine at
1,165 instructions.
They were invisible because progress.py's classify() matched INCLUDE_ASM,
INCLUDE_RODATA and C definitions, and a file-scope __asm__ block is none of
those -- so each landed in NO bucket, either swallowed by a surrounding
construct or surfacing as the single `UNPLACED (parse hole)` line the tool has
been printing all along.
progress.py gains a VERBATIM __asm__ bodies line: counted byte-identical (it is,
by construction) but NEVER as REAL. main's headline moves 45.88% -> 42.15%.
Nothing regressed and no work was lost -- the denominator was missing 173
functions that are real remaining work.
THE COUNTING LESSON IS THE REUSABLE PART. Counting these by hand went
116 -> 112 -> 108 -> 178 -> 199 across five attempts in one session, every
intermediate number reported confidently. All five errors were one shape, a
pattern narrower than the claim it supported:
* the sources use BOTH ".ent\tNAME\n" and ".ent NAME\n" -- anchoring on either
silently drops every instance of the other;
* a bare ".ent\t" fragment yields a phantom function literally named `t`, six
times, which is the only reason the error was noticed;
* __asm__ appears in 3,182 of 4,224 sources, almost all the §3a barrier, so
counting files or counting __asm__ measures nothing;
* `.globl NAME` + `NAME:` proves EXPORT, not CODE -- the first real run
reported jtbl_80072ED4/EEC/F0C/F24 as four "functions";
* a hand-written SDK name list reported 170 game functions because it did not
know VectorNormalSS / SquareRoot12 / OuterProduct12 are libgte.
So the tool does not trust one regex: THREE independent detectors that must
agree with disagreement reported as a defect (R34 -- that is what caught the
jump tables); SDK-ness DERIVED from the 14 shipped PsyQ archives via nm (2,227
symbols) rather than a list (R33); coverage asserted so a definition-shaped
block no detector claims fails loudly (R32/R43); and --selftest carrying a
known-true case of every spelling plus the phantom `t` and the jtbl regression.
Cookbook §448, SETUP row. Law: when a count comes from a text pattern, the
pattern has a denominator too -- validate it against one known-true case of
every FORM the corpus contains before quoting the number.
|
||
|
Drew T
|
1bac13b664 |
fix(jtbl): the pad walk cannot see a verbatim-asm rodata block — SaveLoadRoutine banks (bytes, not a decompile)
tools/jtbl_rodata_pads.py --derive walks a TU's rodata emission against the
retail island and validates only what it can SEE. A §265 verbatim-__asm__ body
emits its tables as `.section<TAB>.rodata` + `jtbl_xxxxxxxx:`, and the walk
missed BOTH spellings:
* the rodata directive was matched as the literal one-space string
".section .rodata" / ".rdata", so a tab-spelled directive never entered
rodata at all;
* inside rodata the anchor regex accepted only `D_xxxxxxxx` (the S74 dlabel
fix was one prefix short), so a `jtbl_xxxxxxxx:` label was invisible.
Consequence, traced: the walk skipped the block as if it were .text, every
later C table walked 104 bytes behind its retail address, EVERY WORD in that
range happens to be a valid code address so the entry guard never fired, and
the walk stopped short of the island's single zero word -- so the one trailing
pad was never emitted and the image linked 4 BYTES SHORT. That produced 3,989
differing bytes on a body the verdict layer had already called byte-identical.
Fixed: tokenised directive match (.rdata / .section .rodata, tabs and commas);
anchors keyed on the ADDRESS IN THE NAME for `D_` or `jtbl_`, with an
address-suffixed label of any other prefix now REFUSING loudly (R43) instead of
becoming a silent hole; and `.align N` modelled SECTION-RELATIVE from the walk
origin, as `as` does -- needed for `.align 3` when a section starts = 4 mod 8,
which span B (0x80072E44) does.
Negative control: old vs new derive over ALL 162 md_*/main derive-path TUs ->
160 byte-identical post-derive streams with identical exit codes, 0 DIFF; 2 SKIP
(800c2/800c3 are REORDER TUs with no derive stage).
main SHA1 143dbb89... BYTE-IDENTICAL, 413,696 bytes, cmp identical to retail.
WHAT THIS IS NOT. SaveLoadRoutine is banked as a §265 verbatim __asm__ block --
BYTES, NOT A DECOMPILE. Its 1,165 instructions are byte-correct and unexplained.
tools/progress.py correctly REFUSES to count it, reporting `UNPLACED (parse
hole)` rather than inflating REAL (which moved 880 -> 881 on func_8005DCA0
alone). Two such blocks already exist in this TU, documented as necessary
because those functions have no epilogue and fall into shared tails. A real C
decompile is now being attempted separately; this commit is the revertible
byte-green base for it.
|
||
|
Drew T
|
cf7f837231 |
fix(gate): main's TABLE REJECT verdict was unreachable — SaveLoadRoutine is a CARVE, not plumbing
SaveLoadRoutine (1,165 ins) is the largest open function in the project, 9.2%
of all remaining work, and has been carried as the §434 WALL. Gated alone
through gate_main, with the §376/§378 chain already applied, the verdict layer
says: "SaveLoadRoutine is BYTE-IDENTICAL; all 3989 differing bytes are
ELSEWHERE". The body has been correct the whole time.
What rejects it is where its FOUR jump tables (jtbl_80072ED4/EEC/F0C/F24) land:
.data/.rodata (jump tables) 3,787 bytes 94.9%
.text (perturbed code) 202 bytes 5.1%
and the built image is 4 bytes SHORTER than retail (413,692 vs 413,696) --
§446's first diagnostic, firing on a function §446 was not written about.
main_diff_locate.classify() already HAD a TABLE REJECT class, added in S72 under
a docstring reading "THE THIRD CLASS EXISTS BECAUSE THE FIRST TWO MISLABELLED
IT". It could not fire here for two independent reasons:
* it keyed on the literal string `(.rodata)`, but main's section_order is
[.rodata, .text, .data, .bss] -- its rodata sits BELOW .text and its jump
tables live in `.data` objects, so TABLE REJECT was UNREACHABLE BY
CONSTRUCTION on the binary with the most jump-table functions left. A
section NAME is not a section ROLE.
* it demanded purity (ro == outside), so 5% perturbed code defeated an
all-or-nothing test and dropped the verdict through to PLUMBING REJECT --
whose advice (fix_arity_callers -> cast_self_callers) addresses the 5% and
cannot touch the 95% that is data. That chain was run on this function
TWICE today and fixed nothing, exactly as the evidence predicts.
Now: table bytes counted in (.data) OR (.rodata), and the test is DOMINANCE
(>=60%) rather than purity, reporting the split and naming which part is the
carve problem and which the declaration problem.
Negative control over all five pre-existing verdict shapes (pure BODY, pure
PLUMBING, pure TABLE, MIXED, NOT FOUND) plus the S75 shape: 5 of 6 verdicts
UNCHANGED, only the SaveLoadRoutine shape flips PLUMBING REJECT -> TABLE
REJECT (MIXED).
Cookbook §447. The law: a class that cannot fire is worse than a class that does
not exist -- it converts "I don't know" into confident, specific, wrong advice.
When a verdict names a subsystem, check that subsystem owns the MAJORITY OF THE
BYTES before acting on it.
|
||
|
Drew T
|
abea9f0ac2 |
feat(decomp): bank func_8016AE5C (85 ins) + frontier_classify — the frontier is not a drafting problem
func_8016AE5C (ov_SC03_108) was logged "match_one MATCH but the whole-binary
gate rejected -- CAUSE NOT DETERMINED". Determined: the body is byte-perfect (0
differing words inside the function; all 1,168 diffs are uniform +0x20 shifts
outside it) and it emits an 8-entry jump table that was never carved. It banked
unchanged the moment the §446 jtbl_carve per-table bound landed.
tools/frontier_classify.py (NEW) — classify every open stub by its TRUE BLOCKER
from artifacts already on disk (R33/offline-tooling-first: zero tokens, no
agents, no builds). "69 functions left" is a stub count, not a difficulty
measure, and routing drafting agents at carve or plumbing problems wastes them.
A-TWIN-REMAP 3 302 a byte-identical copy is already banked elsewhere
B-CARVE 11 3,301 owns a switch jump table -> the §446 class
D-NEAR 2 106 closeness <=25 -> permuter fuel, not drafting
F-FAR 3 223 draft materially wrong -> redraft
G-DRAFTED-UNK 49 8,724 drafted before, no usable verdict on record
H-VIRGIN 1 1 never drafted (and it is a DATA BLOB, not a function)
68 of 69 remaining functions already have a draft on disk. The endgame is a
verification/integration problem, not a drafting one.
TWO SELF-INFLICTED DEFECTS FOUND BY CHECKING AGAINST KNOWN-TRUE CASES, both the
session's recurring shape (a scan narrower than the claim it supports, R32):
* The sig directory is NOT the fleet. Alongside the 213 real binaries `.run/`
holds `SLUS_007.26` (a STALE duplicate of main under the ROM filename),
`resident_image`, and two CROSS-BUILD binaries (`sep8_SLUS_007.26`,
`aug31_USA_DEMO.EXE`). Counting them as peers reported 38 fns / 7,516 ins of
free twin-remaps -- mostly main "already banked" in ITSELF, the rest proven
in a PROTOTYPE that R13 forbids as evidence. Now derives the fleet from the
Makefile and prints what it ignored. True figure: 3 fns / 302 ins.
* The draft scan globbed `.run/S7*` only, missing `.run/S69m2`, `.run/S68m1`,
`.run/s67m1`, `.run/wave_ds2`, `.run/gate_lane`, `.run/backlog_drafts`. All
32 drafted main functions read as "never drafted", which would have sent
agents to redraft 6,328 instructions that already have drafts. Now one
pruned os.walk of .run (worktrees excluded -- 7.4 GB of duplicate sources).
Honest negative result: resident:func_800D06E8 (344 ins) did NOT bank. I
predicted the carve fix would clear it; it did not. Its blocker is still open.
|
||
|
Drew T
|
cb948a6bbc |
feat(decomp): the ov_SC01 reloc-only cluster + its 5th latent victim — 5 fns, 1,301 ins
S74 handed this forward as "1,116 instructions behind one question": family_remap
on ov_SC01_004/005/006/008 gated DIFF 4/4 against the banked exemplar
ov_SC01_009:func_8017EB08, and the class had been carried as a codegen wall since
S70. The four bodies were byte-identical to the exemplar the entire time.
Word-level classification vs the exemplar, computed independently twice (a Fable
agent's script, then mine from scratch against the retail images), identical:
nins=279 EQ 213 · RELOC-HI16 23 · RELOC-LO16 24 · INTERNAL-J 19 · CODEGEN 0
Zero register-allocation, instruction-selection or scheduling differences.
ROOT CAUSE — tools/jtbl_carve.py reserved ONE WORD TOO MANY per table:
* spimdisasm runs an island's LAST `jtbl_` dlabel one word into the following
NON-ZERO data (string bytes 0x696F760A / 0x000013FF / 0x62647020), so the
zero-word trim cannot see it; and
* the over-span clamp that would have caught it was guarded by
`len(sltiu_bounds) == 1` -- but `sltiu` is ALSO how gcc emits an unsigned
range check ((u32)(x-lo) < n, I1). These four carry five distinct sltiu
immediates, so the guard silently disabled itself on precisely the functions
that needed it.
0x2C reserved for a 0x28 table => image 4 bytes short => ~850 %lo immediates
shift => whole-binary DIFF about a function whose own bytes are perfect.
Fixed with a PER-TABLE bound: gcc-2.7.2's dispatch is a fixed idiom, so the
`sltiu` nearest ABOVE that table's own %hi(jtbl_X) is unambiguous whatever else
the function tests. Second defect stacked behind it: a carve span whose
JTBL_PADS line lacks a `tables=` comment lost its existing table's start on
merge and refused "table starts do not fit the span" -- which harvest_verify
then "repaired" with a needless jr_isolate_all that walked back into the first.
THE NEGATIVE CONTROL IS THE STORY. Run over every other open table-bearing stub
fleet-wide, the fixed bound changed exactly one more table: ov_SC06_022/
func_80185B80 (185 ins), a FIFTH victim nobody had drafted against. A guard that
disables itself on a common idiom does not fail once -- it fails quietly across
the whole corpus.
Banked, each with its own byte-gate verdict (--no-propagate, clean re-gate):
func_8017EB30 ov_SC01_004 279
func_8017F2D4 ov_SC01_005 279
func_8017F2D4 ov_SC01_006 279
func_8017EC68 ov_SC01_008 279
func_80185B80 ov_SC06_022 185
Also here:
* dedup_propagate: memoize find_site's mask (lru_cache) -- 54 ms of masking
per call over the whole source, recomputed though it depends only on the
text. 2x on that loop (58.3 -> 33.0 ms/call), NC identical on 120 addrs.
Scoped honestly: that loop is ~2.4 min of a 30-min run; the profiler puts
43% in family_remap._alias_decl_for, which is NOT fixed here.
* Makefile: `clean` says out loud that BINARY= is ignored and it is fleet-wide
(cookbook §445) -- it silently deleted asm/ for all 213 binaries this session.
* Cookbook §446 (the carve law: when a standalone-MATCH jtbl draft gates DIFF,
diff the carve extent against 4 x sltiu before touching the body), §445, and
SETUP rows for both tools (R21).
* CURRENT_PHASE: the S75 log, incl. the measured fleet dedup-hygiene census
(~2,073 fns / ~12,116 items, all ALREADY MATCHED -- cleanup, not work) and
Drew's decision to leave it and gate --no-propagate from here.
|
||
|
Drew T
|
0c2b37a287 |
fix(gate): overlays.mk carve-state snapshot was singular, so the revert half-restored
Found by running one reject to ground. After a gate that REJECTED
resident:func_800D06E8, config/overlays.mk had a 4th JTBL_PADS entry and had
LOST `--pre hdr.rodata.o` (the §440 resident leading-rodata sandwich). The
binary then would not build at all -- "consumed 3 rodata jump table(s) but 4 pad
spec(s) given -- table-count drift vs the carve" -- while src/ was perfectly
clean, which is the only place anyone looks before building.
Root cause is a silent narrowing in the classic shape. harvest_verify snapshots
ONLY the gating binary's own overlays.mk block on purpose (the file is shared by
every parallel gate; a whole-file restore resurrects other binaries' lines --
the S62 defect). But _mk_block_span was SINGULAR: the first `# --- <binary>`
header through the next `# --- `. A binary whose carve state spans more than one
block was half-snapshotted and silently half-restored. It returned a TRUE span
for a scope smaller than the caller believed, and nothing compared the two (R32).
Blast radius measured before costing (R37): 1 of the 142 binaries that have a
block -- resident, which has exactly two (§8e pad spec, §8f leading-rodata
sandwich) and still holds 587 instructions of open stubs.
_mk_block_spans (plural) snapshots a LIST, restores tail-first so earlier spans
stay valid, collapses to the snapshot when the header count changed rather than
leaving half-state, and RE-READS and compares the result -- the defect it
replaces was a reported success. _mk_block returns None (not []) for the 71
binaries with no block, so the caller's guard keeps its meaning.
Negative control, three ways:
* snapshot -> restore is a NO-OP on 142/142 binaries with a block;
* the real S75 damage is fully undone;
* the OLD single-block restore provably does NOT undo it -- the positive
control that proves the fix is load-bearing, not decorative.
Cookbook §444 also records the two other findings from the same reject: the
classified ledger stores the LADDER'S FINAL verdict (the recorded CC1-FAIL came
from a late sig_unify rung; the raw draft compiles and fails on BYTES), and
match_one MATCH + rtu_match MATCH is still not bankable -- func_800D06E8's real
blocker is a jump table (built binary 20 bytes longer, 0x800CEDFC holds a table,
69,571 words shift), because neither matcher LINKS.
|
||
|
Drew T
|
cdd535e45b |
fix(tools): reconcile_tu's cc1 premise, the &-cast arms, and the worktree sig gap
The S74 checkpoint's "one unfixed defect that is actively costing banks"
(reconcile_tu manufacturing declaration conflicts), run to ground — plus the
harness gap that produced a false carve-corruption verdict.
reconcile_tu.py — three defects, measured against the real gcc-2.7.2 front end
(cdecl._cc1_accepts, the oracle cdecl.compatible was validated with; R33):
* The premise "a decl BELOW still conflicts" is TRUE at file scope and FALSE
at block scope. cc1 ACCEPTS a block-scope extern against a TU decl below it
(pedwarn "type mismatch with previous external decl"); conforming it is
destructive, because the TU's decl names the TU's TYPE and a type declared
below the splice point is not in scope AT it -- the emitted result gets
"syntax error before 'D_x'". Byte-witnessed on resident:func_800D06E8 (344
ins), whose block-scoped `extern Blk80078E78` became `extern
Struct80078E78`, typedef 388 lines lower. That construct is what this
ladder's OWN scope_demote_drafts (§8d) rung emits on purpose, and three
already-banked functions in that TU use it: one rung undoing another.
* The cast pass rewrote COMMENT PROSE -- 8 rewrites inside one header comment,
including inside a quoted cc1 diagnostic. Now matches on cdecl._mask
(length-preserving, so a mask offset is a source offset) and splices into
the original.
* `&sym` emitted `&` applied to a cast: legal for the scalar arm, `invalid
lvalue in unary '&'` (measured) for the array/fnptr/fnptr_array arms. `&`
now selects a pointer form and consumes itself -- but ONLY with no trailing
subscript, because `&sym[i]` is the address of ELEMENT i and the old code
had that case right. That last clause exists because the R39 negative
control caught the fold as a regression in the first cut of this fix.
gate_stage.py — `--skip-stages` / `GATE_SKIP_STAGES` (loud when used). Stage 0
gates raw drafts first, so a broken rung can only cost a RECOVERY, which is
exactly what makes it invisible: the function it destroys was already failing,
so its DIFF reads as a fact about the function.
verify_worktree.py / jr_isolate_all.py / parallel_gate.py — provision() now
symlinks every .run/sig.*.jsonl (main clone 259, provisioned worktree 0), the
third member of the class holding extracted/ and .run/obj40. parallel_gate was
fixed for this identical bug in S69: two provisioners, no shared list, found
twice; they now cross-reference each other. jr_isolate_all no longer swallows
the resulting FileNotFoundError into `except: continue` -- that turned a missing
index into a confident carve-CORRUPTION verdict over 2,603 of 2,603 functions
(R54). Adds _assert_scan_covered: attempted == raised means the scan measured
nothing, so its zero is an artifact, not a finding (R32).
Verification:
* 4 cc1 probes (the table above), each run on the pinned front end.
* R39 negative control over the stored-draft corpus: 661 adjudicated, 652
IDENTICAL, 9 CHANGED and every one an intended class. 4,173 of 4,864 drafts
unadjudicable (filenames that are not func_<ADDR>) -- stated, not hidden.
* jr_isolate_all ov_SC03_105 --dry-run: unchanged in the main tree.
* make clean/extract/build BINARY=resident -> 8e17e02f... BYTE-IDENTICAL.
Docs ship with the change (R21): cookbook §442/§443, index regenerated (1,112
sections), 3 docs/SETUP.md rows, CURRENT_PHASE S75 log.
|
||
|
Drew T
|
8edb918480 |
feat(cards): size-filter the same-address lead (§238 homonym) + bank the S74 lever set
THE CARD USED TO HAND AGENTS A WRONG TWIN ABOUT ONCE IN FIVE. `⭐ func X IS BANKED AT THIS ADDRESS`
never checked that the two functions were the same SIZE, and overlays share addresses between
unrelated functions as readily as they share code. Measured over this session's ~60 cards: about a
dozen agents reported discarding the lead themselves, and one card advertised a 72-instruction
namesake — with journal history claiming "already MATCH closeness 0" — to a 241-instruction target.
A confidently wrong lead costs more than no lead, because the agent believes it.
corpus.sig already carries `nins` and `h_seq`, so the fix is free: `_same_addr_banked` now returns
(binary, nins, h_seq); the card keeps a lead only at a MATCHING instruction count, marks it strong
when the mnemonic skeleton matches too, and prints an explicit `⚠ IGNORE` naming the binaries where
that address holds something else, with both sizes.
VERIFIED IN BOTH DIRECTIONS against known-true cases before being believed (never trust a filter you
have not tried to fool):
* the trap: ov_SC03_105:func_801806F8 (241) vs ov_SC03_013 (72) -> `⚠ IGNORE`.
* the positive: ov_SC02_003:func_80187B40 (158) -> strong lead to ov_SC02_000 (158, same h_seq,
banked this session) AND, in the same card, warned off ov_SC04_011's 138-ins homonym at that
same address. That is precisely the pair a wave agent sorted out by hand hours earlier.
Cookbook §438 (the law: a lead is fuel only if it carries the cheapest fact that can refute it —
size refutes a homonym for free and nobody had asked) and §439, the S74 lever set: MEM_IN_STRUCT_P
as a two-way alias-oracle dial (four agents converged on it independently); `goto`-into-a-shared-tail
vs longhand as a REGALLOC dial because gcc-2.7.2 cross-jumps after allocation; `for` -> do/while as a
length-changing scheduling dial; allocno PRIORITY via a non-volatile asm at a loop head, with the
measurement that register pins are actively harmful for that class; the -O0 global-RMW rule
(`x++` emits the copy-back quartet, `x = x+1` does not); why `sll 16; srl 16` survives only across a
CALL; `sltiu N` without `addiu -1` proving an empty `case 0` is mandatory; block-scoped temps in
duplicated bodies; two `register asm` vars cannot share a hard reg; and `x*32` vs `x<<5` emitting
lh vs lhu — which match_one's %lo mask HIDES, so it must be checked with objdump.
|
||
|
Drew T
|
6e840730a9 |
feat(carve): bank 4 more via the carve chain — and §8b's "non-adjacent => ISOLATE" is over-strict
resident:func_800D00E4/func_800D02D0/func_800D0488 + ov_SC07_002:func_80180248, all byte-verified
from clean rebuilds (resident 8e17e02f, ov_SC07_002 fad71342) and counted from the SOURCE.
ov_SC06_029's two are re-gated separately against HEAD — this agent's worktree predated five banks
there, so its numbers for that binary no longer apply.
TWO OF THE SIX NEEDED NO CARVE WORK AT ALL, AND CARVE-REFUSED WAS AN INSTRUMENT VERDICT.
ov_SC07_002:func_80180248's table is ALREADY inside a carve bound to its own subseg: in stub state
spimdisasm migrates the table into the fn's .s and the object fills the piece exactly, so banking
just swaps that block for cc1's identical one. `island_probe` classified it `tail` on the table's
ADDRESS, `apply()` routed it to build_carve, which resolves spans out of the RAW data asm where a
carved table no longer is -> "not found in the raw data asm" -> harvest_verify booked CARVE-REFUSED.
A verdict about the route we chose, not about the function (R43). jtbl_carve now has a `covered`
verdict (table inside an existing carve bound to the fn's OWN subseg) and a `covered-tpad` wall (the
retail copy carries a trailing §8a pad the matched body won't emit — bankable, needs a `0t<n>`
entry); a fully-covered batch is a no-op before either route.
THE RESIDENT CAN CARVE LIKE AN OVERLAY. Its three tables are adjacent and lead the island
(0x450e0..0x451ac, one span, all in subseg `resident`). The genuinely new part: the resident opens
with `- [0x0, rodata, hdr]`, a 1-word .rodata header BEFORE the code, so its layout is
rodata -> text -> data -> rodata(carve) -> data, which `ld_interleave --order` cannot express (every
listed piece lands after TEXT_START, and hdr.rodata.o would fall into the unchecked `empties` bucket
and be parked after the text, moving every byte). New `--pre` places a leading-rodata piece ahead of
the text; resident_JTBL_INTERLEAVE uses it.
NEW LAW, BYTE-PROVEN (§8b was over-strict — EXTEND the carve, do not isolate): a .rodata carve piece
binds to a code SUBSEG, not a function, and the object's .rodata is the address-ordered
concatenation of cc1's tables for BANKED functions and still-stubbed functions' MIGRATED tables. So
a span may legitimately hold a MIX, and extending a carve across an align-pad word and two unrelated
STILL-STUBBED tables was byte-identical with nothing banked — where the tooling demanded a
jr-isolation. Corollaries, all measured: migrated tables self-align (spimdisasm emits `.align 3` iff
the table's SPAN-RELATIVE offset is 8-aligned), so stubbed tables need no spec; JTBL_PADS counts cc1
tables only, so a mixed span's spec GROWS as each sibling banks; and the zero-word rule is INVALID
across a migrated boundary, because that zero is supplied by the preceding migrated block.
ALSO REPORTED, NOT FIXED (harness gap worth its own change): verify_worktree.provision omits
`.run/sig.<bin>.jsonl` — main clone 259 files, provisioned worktree 0 — and jr_isolate_all's
carve-ownership scan swallows the resulting FileNotFoundError in a bare `except: continue`. Measured:
2603 of 2603 functions raised, the scan found 0 owners, and the run aborted with a CONFIDENT FALSE
verdict ("committed .rodata carve ownership is not 1:1 — stranded/duplicated carve"). Both resolve
instantly once the sigs are present. Any worktree-run isolation before that is fixed reports a
corruption that is not there.
|
||
|
Drew T
|
089311e74d |
feat(md_SC07_004): 10/10 banked — md_SC07_004 is now ZERO stubs, and the "decl conflicts" were fake
Clean rebuild BYTE-IDENTICAL 87ac0de3; corpus.stubs('md_SC07_004') 10 -> 0, counted from the SOURCE.
THE §376/§378 CHAIN WAS NEVER NEEDED. Zero declaration edits: no fix_arity_callers, no
cast_self_callers, no --any-proto, no --sync-decls, no undo journal. `git diff -U0` on the TU removes
exactly the 10 INCLUDE_ASM stubs plus one hoisted typedef. Every recorded "declaration conflict" was
an INSTRUMENT defect. Four of them, all named, three patched here:
1. `CC1-FAIL(no-diagnostic)` was neither cc1 nor no-diagnostic. The failing stage was
`jtbl_rodata_pads --derive`, which prints to stderr AFTER cc1 exits 0 quietly. `_items` matched a
rodata anchor only as `D_xxxxxxxx:`, but a block written as inline `__asm__` in C arrives in the
labels.inc macro form `dlabel D_xxxxxxxx` — so an 8-byte hole opened in the walk and every C jump
table after it died with "island layout drift". The harness label was wrong twice: it said CC1
when the failure was a post-maspsx filter, and no-diagnostic when there was a precise one.
2. Same file, UNALIGNED ANCHOR: the ctable branch read `word(pos)` without first stepping the
sub-word zero gap, so a preceding `.asciz` ending at an odd address made it refuse a correct
layout. Now reuses the same zero_gap the anchor branches already use — a no-op wherever pos is
already aligned, i.e. everywhere that builds green today.
3. `harvest_verify` computed the typedef strip-set UNSCOPED: `cdecl.typedef_names(path)` without
`above=fn`, which that function supports for exactly this. A typedef declared BELOW the splice
point got stripped out of the draft that needed it -> `parse error`, logged as PLUMBING and
indistinguishable from a real conflict. One line.
4. NOT PATCHED, AND THE MOST IMPORTANT ONE: `reconcile_tu.py` (gate_stage's `-rc` stage) MANUFACTURED
both remaining "conflicts". The same drafts gate 9/9 byte-identical through harvest_verify and
7/9 through gate_stage. Isolated stage by stage, `-rc` (a) rewrites deliberately BLOCK-SCOPED
externs to a file-scope spelling whose typedef is declared ~2,800 lines lower — overwriting the
TU's own byte-proven house style, which three already-banked functions in that file use; and
(b) substitutes identifiers TEXTUALLY, including inside comments and inside `&`-expressions,
emitting `*(T *)&((s32 *)&D_800AE620)`. A correct draft using the block-scope-extern idiom
currently CANNOT survive gate_stage. Left for a deliberate fix: `--stages` should be able to skip
reconcile_tu, or a draft should be able to opt out.
The two surviving non-stub source edits are byte-neutral (proven by the SHA above): a §304
migrated-rodata re-emission (`D_801A01EC`, the exact form this TU already uses three times, needed
because banking the body deletes the .s that carried the island word), and one typedef moved up so a
function above it can see it (typedefs emit no bytes).
Also banked the 10th stub (func_801ADA10) that defect 3 had been silently blocking.
Regression-checked by the agent: main, md_MAIN_003, md_SC07_003, md_SC03_073, md_MAIN_011 all
rebuild BYTE-IDENTICAL. A full R22 follows before this session closes.
|
||
|
Drew T
|
5e10215269 |
feat(md): bank the 4 -O0-stranded functions — and the class is now essentially empty
md_MAIN_011:func_800CF28C · md_MAIN_003:func_800D0268/func_800D0740/func_800D0C50, all byte-verified
from a clean rebuild and counted from the SOURCE: md_MAIN_011 is now FULLY MATCHED (0 open stubs),
md_MAIN_003 is down to 1 (func_800CF3E8).
THE PREMISE I HANDED THE AGENT WAS WRONG, AND IT SAID SO. md_MAIN_011 is already a whole-object -O0
module — no carve was needed. Its real blocker was tools/jtbl_rodata_pads._s_rodata_span ignoring a
trailing `.align`, the SAME defect this session fixed for md_SC07_003 from the other direction: two
agents converged on it independently. Adopted this agent's stricter form (only a TRULY trailing
align rounds `hi`; an interior one is followed by data that sets `hi` higher anyway).
Note WHY it stayed latent: `derive`'s zero_gap self-corrects a 1-3 byte undershoot whenever the next
stream item is an anchor. A C jump table has NO anchor — so the bug can only fire the moment someone
banks a switch function into such an object, and when it fires it accuses the CARVE ("island layout
drift"), not itself.
md_MAIN_003 needed one new -O0 object, and the boundary I proposed (0x1f74 -> 0x1e58) was both too
narrow and off by 0x2B8. The carve made is `md_MAIN_003_o0e` at 0x1308 (vram 0x800D0100) running to
the existing o0c boundary: everything in that span is a §265 verbatim __asm__ body or an INCLUDE_ASM
stub — zero optimizable C — so the whole tail flips with one cut. Proved byte-identical with NOTHING
banked first (§431 discipline), then the three drafts gated one at a time.
TWO MORE GENERAL DEFECTS FIXED IN jr_isolate_all, both of which silently mis-place a boundary:
* an item-less CLOSING region emitted a duplicate `- [off, c, …]` line and the validator refused;
the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing
region look non-empty.
* A §265 VERBATIM __asm__ BODY IS PREAMBLE, AND PREAMBLE IS ASSUMED BYTE-NEUTRAL. It is not — it
emits bytes. `parse_overlay_c` has four addressed-anchor forms and a verbatim body is none of
them, so it attaches to the NEXT anchor: cutting at func_800D0268 would have moved 0x168 bytes of
other functions into the new object while the yaml claimed the region starts higher. New
`_region_emit_start()` derives the yaml offset from the region's CONTENT (item addresses + every
.globl/.ent the text names that resolves inside the object) and takes min(cut, emit), so a
boundary can only move DOWN. Where no verbatim asm is in play it equals the cut — every existing
isolate is unchanged.
BLAST RADIUS PROVEN, not argued: jtbl_rodata_pads is in the build path (`--derive` for md_*/main),
so the agent rebuilt main + all 70 md_* from scratch (71/71) and then ran the full fleet:
**make check-all 213/213 passed, 0 failed**, main 143dbb89 BYTE-IDENTICAL.
CENSUS, denominator asserted (1057 live stubs, 0 without a .s): exactly ONE -O0-prologue stub
remains stranded in an -O2 TU fleet-wide — main:func_8002C410 in src/800_b.c, 299 ins. Nothing more
should be built for this class; the general tool already existed and what was missing was
correctness, not coverage.
|
||
|
Drew T
|
3a886b9652 |
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
|