mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-08 09:43:04 -04:00
fa2ccea3ea3a2ebd9dbf4e8efbe40d38f52fbbbc
4842 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
72fa482027 |
feat(tools): verbatim_to_stub.py — put the 147 asm-posing-as-C functions back where the gates can reach them
THE GAP. Every gate in this project substitutes a draft in place of an
INCLUDE_ASM line. A §265 verbatim __asm__ body has none, so:
gate_main.substitute() resolves each entry through the STUB map; a verbatim
function is reported "resolved to NO stub" and dropped
gate_stage/harvest_verify same splice, same gap
splat stops emitting <fn>.s once a function is not a stub
So all 147 were undecompilable AND ungateable -- not for want of information,
but because the information was in a form nothing consumes.
The fix is not a parallel gate (R33 -- one implementation). It is to put the
function back into the form every existing tool already understands: replace the
__asm__ block with INCLUDE_ASM. That is also the HONEST representation --
INCLUDE_ASM pastes the very same assembly the block transcribes, so the bytes
are identical either way, but a stub counts as OUTSTANDING WORK in progress.py
while a verbatim body counted as banked. The conversion moves a function from
"silently done" to "visibly to do".
Two refusals rather than guesses (R43), because both failure modes are silent
and destructive:
* the block is located by brace/paren MATCHING via asm_in_c.asm_blocks, never
regex-sliced -- these blocks are full of braces and parens inside string
literals and an approximate cut corrupts a file that currently builds;
* the asm subdir for the new INCLUDE_ASM is copied from a sibling stub IN THE
SAME FILE. Subsegs are per-file, so a neighbouring file's spelling names a
different subseg -- a stub with the wrong subdir compiles happily and
includes ANOTHER FUNCTION'S ASSEMBLY. With no sibling to copy, it refuses.
--gate rebuilds and asserts the SHA is unchanged, restoring the file if not:
byte-neutrality here is a claim, and this tool exists to enable a byte gate, so
it declines to be the one link that goes unchecked.
Dry-run verified on main:func_80047E58 -> src/800b.c, 9-line block, subdir
correctly derived as asm/nonmatchings/800b. The --gate proof is deferred only
because another agent is mid-build on main right now.
|
||
|
|
70de5a8611 |
feat(tools): verbatim_target_s.py — the 147 asm-posing-as-C functions are workable again; bank func_8017DB98
THE BLOCKER. asm_in_c.py found 147 GAME functions that are §265 verbatim
__asm__ bodies. NONE of them could be worked on: splat emits
asm/nonmatchings/<subseg>/<fn>.s only for functions that are still INCLUDE_ASM
stubs, and a verbatim body is not a stub -- so splat stops emitting its .s,
while match_one and rtu_match BOTH consume one. Measured: 1 of 147 had a target
on disk. The class was unworkable because the information was in the wrong FORM,
not because it was missing.
verbatim_target_s.py regenerates a splat-format target .s from the EXTRACTED ROM
IMAGE -- never from the __asm__ block in our own source, because the block is
the thing under test and a target derived from it would agree with the candidate
by construction (R34). 146 of 147 emitted; the 1 refusal is REPORTED.
TWO DEFECTS CAUGHT BY CHECKING AGAINST A KNOWN-TRUE CASE, both of which would
have shipped ~147 silently-wrong targets:
* BYTE ORDER. splat writes the four bytes as they sit in the image
(`C8FFBD27` for instruction 0x27BDFFC8) and masked_diff.insns_from_s reads
the column with struct.unpack("<I", bytes.fromhex(...)). objdump prints the
VALUE, so reversing double-swaps: 91 of 1139 words agreed with splat's own
.s for the same function. The LENGTH matched perfectly, so nothing except a
word-level cross-check could have caught it.
* `-z` / --disassemble-zeroes. objdump ELIDES runs of zero bytes as `...`, and
a MIPS nop IS 0x00000000 -- so every nop vanished. func_80049610 (three
nops) disassembled to ZERO instructions; func_80047D3C 31 of 36. The length
assertion caught all of them, which is the only reason this was not shipped
as ~30 quietly-truncated targets.
Verification: regenerated SaveLoadRoutine target is 1139/1139 words IDENTICAL to
the .s splat itself emitted for the same function.
ALSO BANKED: ov_SC06_025:func_8017DB98 (122 ins). Its body was byte-exact on
disk since S71 and the blocker was one word: the TU declared
`extern void func_8017DB98(s32, s32)` where the epilogue is `addu $v0,$s3,$zero`
-- must be `extern s32`, and the caller discards the result so the change is
byte-neutral. That line number and fix were recorded in the agent journals the
whole time; frontier_classify only surfaced it once journal_notes was wired in
as a second oracle earlier this session.
|
||
|
|
f5f4c2eeec |
feat(decomp): bank func_801806F8 + func_80180ABC (498 ins) + frontier_classify reads the journals
Two banks from the S75 redraft workflow (7 overlay functions, one agent each,
every claimed MATCH re-verified by an independent agent instructed to refute
it). Both were carried as F-FAR "a draft exists but is materially wrong":
func_801806F8 ov_SC03_105 241 ins (recorded closeness 235)
func_80180ABC ov_SC03_105 257 ins (recorded closeness 250)
Neither needed a better model. Both needed the recorded closeness not to be
believed -- see below.
frontier_classify.py, THREE fixes, each caught by testing against a case whose
answer was already known:
1. BEST closeness, not LAST. .run/backlog.jsonl is append-only, one row per
attempt across every lane and session, so the last row is evidence about
THAT LANE'S SEED, not about the function. Caught func_80180B3C (best 125,
last 287) and moved func_80181294 from "redraft" to "permuter" (best 19).
The draft that ACHIEVED the best score is kept, not the last one written.
2. journal_notes.py wired in as a SECOND, DISAGREEING oracle (R34). The backlog
does not have what the agent journals have. Measured on func_8017DB98:
backlog best == last == 115, so best-vs-last could not help, while the
journal holds "Attempt 2 (MATCH · closeness 0) ... MATCH 122/122 ... BANK
BLOCKER is TU plumbing, not the body (§376/§378)" WITH the draft path and the
exact declaration to change. Reclassified 37 functions; G-DRAFTED-UNKNOWN
fell 47 -> 10 and a new C-PLUMBING class holds 16 functions / 1,547 ins whose
BODIES ARE PROVEN and are blocked only by the TU.
3. A consuming-regex bug in my own extractor -- the session's signature defect,
committed a third time in the tool written to find it. The first cut used
`re.finditer(r'\*\*Attempt \d+\*\* \(([^)]*)\)(.{0,400})', ..., re.S)`, whose
400-char body window SWALLOWS THE NEXT ATTEMPT'S HEADER, so every record
following another was invisible. On func_8017DB98 it hid attempts 2 AND 6,
both `MATCH · closeness 0`, and returned attempt 1's NEAR (2) as the best --
exactly the records the oracle exists to find. Now splits on the marker
rather than consuming past it. A regex that consumes an unbounded body cannot
enumerate the items after the first.
Rows now carry attempts, closeness_last, journal_closeness, and a
!!WARMSTART-REGRESSION flag when a later attempt scored materially worse than
the best -- the shape a wave's warm-start regression makes, which from inside
the wave is indistinguishable from an unsolved function.
Gate ledger for the batch of 7: 2 banked, 3 near, 2 failed. func_800CB00C failed
despite being adversarially upheld -- it owns a jump table, and both matchers
compare .text only, so a verified .text MATCH proves nothing about table
placement (the agent's own write-up says so).
|
||
|
|
8e3ce81a69 |
docs(phase-31): S75 addendum — the frontier is 154 game functions bigger than reported
tools/asm_in_c.py finds 199 functions that are assembly posing as C (154 game code, 171 in main). They were in no progress.py bucket, so main's REAL% was overstated: 45.88% -> 42.15% once counted. Nothing regressed; the denominator was missing 173 functions of real remaining work. Also records the counting cautionary tale (five hand counts, 116->112->108->178 ->199, each wrong the same way) and the tool design that answers it. Cookbook §448. |
||
|
|
c05ea15cbe |
feat(tools): asm_in_c.py — 154 game functions are assembly wearing a .c extension
A .c file in src/ looks decompiled. 199 functions are not: they are the target
assembly pasted into a C string literal (§265), byte-identical BY CONSTRUCTION
and completely unexplained. 45 are PsyQ/CRT routines where that is defensible;
154 are GAME CODE, 171 of the 199 in main, the largest being SaveLoadRoutine at
1,165 instructions.
They were invisible because progress.py's classify() matched INCLUDE_ASM,
INCLUDE_RODATA and C definitions, and a file-scope __asm__ block is none of
those -- so each landed in NO bucket, either swallowed by a surrounding
construct or surfacing as the single `UNPLACED (parse hole)` line the tool has
been printing all along.
progress.py gains a VERBATIM __asm__ bodies line: counted byte-identical (it is,
by construction) but NEVER as REAL. main's headline moves 45.88% -> 42.15%.
Nothing regressed and no work was lost -- the denominator was missing 173
functions that are real remaining work.
THE COUNTING LESSON IS THE REUSABLE PART. Counting these by hand went
116 -> 112 -> 108 -> 178 -> 199 across five attempts in one session, every
intermediate number reported confidently. All five errors were one shape, a
pattern narrower than the claim it supported:
* the sources use BOTH ".ent\tNAME\n" and ".ent NAME\n" -- anchoring on either
silently drops every instance of the other;
* a bare ".ent\t" fragment yields a phantom function literally named `t`, six
times, which is the only reason the error was noticed;
* __asm__ appears in 3,182 of 4,224 sources, almost all the §3a barrier, so
counting files or counting __asm__ measures nothing;
* `.globl NAME` + `NAME:` proves EXPORT, not CODE -- the first real run
reported jtbl_80072ED4/EEC/F0C/F24 as four "functions";
* a hand-written SDK name list reported 170 game functions because it did not
know VectorNormalSS / SquareRoot12 / OuterProduct12 are libgte.
So the tool does not trust one regex: THREE independent detectors that must
agree with disagreement reported as a defect (R34 -- that is what caught the
jump tables); SDK-ness DERIVED from the 14 shipped PsyQ archives via nm (2,227
symbols) rather than a list (R33); coverage asserted so a definition-shaped
block no detector claims fails loudly (R32/R43); and --selftest carrying a
known-true case of every spelling plus the phantom `t` and the jtbl regression.
Cookbook §448, SETUP row. Law: when a count comes from a text pattern, the
pattern has a denominator too -- validate it against one known-true case of
every FORM the corpus contains before quoting the number.
|
||
|
|
1bac13b664 |
fix(jtbl): the pad walk cannot see a verbatim-asm rodata block — SaveLoadRoutine banks (bytes, not a decompile)
tools/jtbl_rodata_pads.py --derive walks a TU's rodata emission against the
retail island and validates only what it can SEE. A §265 verbatim-__asm__ body
emits its tables as `.section<TAB>.rodata` + `jtbl_xxxxxxxx:`, and the walk
missed BOTH spellings:
* the rodata directive was matched as the literal one-space string
".section .rodata" / ".rdata", so a tab-spelled directive never entered
rodata at all;
* inside rodata the anchor regex accepted only `D_xxxxxxxx` (the S74 dlabel
fix was one prefix short), so a `jtbl_xxxxxxxx:` label was invisible.
Consequence, traced: the walk skipped the block as if it were .text, every
later C table walked 104 bytes behind its retail address, EVERY WORD in that
range happens to be a valid code address so the entry guard never fired, and
the walk stopped short of the island's single zero word -- so the one trailing
pad was never emitted and the image linked 4 BYTES SHORT. That produced 3,989
differing bytes on a body the verdict layer had already called byte-identical.
Fixed: tokenised directive match (.rdata / .section .rodata, tabs and commas);
anchors keyed on the ADDRESS IN THE NAME for `D_` or `jtbl_`, with an
address-suffixed label of any other prefix now REFUSING loudly (R43) instead of
becoming a silent hole; and `.align N` modelled SECTION-RELATIVE from the walk
origin, as `as` does -- needed for `.align 3` when a section starts = 4 mod 8,
which span B (0x80072E44) does.
Negative control: old vs new derive over ALL 162 md_*/main derive-path TUs ->
160 byte-identical post-derive streams with identical exit codes, 0 DIFF; 2 SKIP
(800c2/800c3 are REORDER TUs with no derive stage).
main SHA1 143dbb89... BYTE-IDENTICAL, 413,696 bytes, cmp identical to retail.
WHAT THIS IS NOT. SaveLoadRoutine is banked as a §265 verbatim __asm__ block --
BYTES, NOT A DECOMPILE. Its 1,165 instructions are byte-correct and unexplained.
tools/progress.py correctly REFUSES to count it, reporting `UNPLACED (parse
hole)` rather than inflating REAL (which moved 880 -> 881 on func_8005DCA0
alone). Two such blocks already exist in this TU, documented as necessary
because those functions have no epilogue and fall into shared tails. A real C
decompile is now being attempted separately; this commit is the revertible
byte-green base for it.
|
||
|
|
6d56972a4b |
docs(R31): S75 decision log — seven walls were seven instruments; leave the 12,000
Two entries. (1) The session's through-line: seven 'codegen walls' examined,
seven instrument defects, none the compiler -- and the reusable law that a
verdict class which CANNOT FIRE is worse than one that does not exist, because
it turns 'I don't know' into confident wrong advice. SaveLoadRoutine's 1,165
instructions are the price: a byte-identical body behind a verdict naming the
wrong subsystem, for a whole phase.
(2) Drew's decision not to convert the ~2,073 functions / ~12,116 duplicate
copies of dedup-hygiene backlog, on the sotn precedent our cookbook records
('sotn writes duplicate funcs explicitly'), with the caveat that the claim rests
on one parenthetical of ours rather than sotn's repo. Gate --no-propagate from
here; the backlog is orthogonal to completion %.
|
||
|
|
ac55da69d1 |
docs(phase-31): S75 checkpoint — 7 banked, seven instrument defects, SaveLoadRoutine is a carve
Full 🛑 block for a fresh session: state, the seven defects (§442-§447), the
measured frontier map, Drew's dedup decision, and the harness lessons.
The headline for whoever picks this up: SaveLoadRoutine (1,165 ins, the §434
wall, 9.2% of all remaining work) has a BYTE-IDENTICAL body and is blocked by a
jump-table carve -- which the verdict tool could not say because that verdict
class was unreachable by construction on main.
R22 clean-fleet is NOT yet run for S75 and the checkpoint says so.
|
||
|
|
cf7f837231 |
fix(gate): main's TABLE REJECT verdict was unreachable — SaveLoadRoutine is a CARVE, not plumbing
SaveLoadRoutine (1,165 ins) is the largest open function in the project, 9.2%
of all remaining work, and has been carried as the §434 WALL. Gated alone
through gate_main, with the §376/§378 chain already applied, the verdict layer
says: "SaveLoadRoutine is BYTE-IDENTICAL; all 3989 differing bytes are
ELSEWHERE". The body has been correct the whole time.
What rejects it is where its FOUR jump tables (jtbl_80072ED4/EEC/F0C/F24) land:
.data/.rodata (jump tables) 3,787 bytes 94.9%
.text (perturbed code) 202 bytes 5.1%
and the built image is 4 bytes SHORTER than retail (413,692 vs 413,696) --
§446's first diagnostic, firing on a function §446 was not written about.
main_diff_locate.classify() already HAD a TABLE REJECT class, added in S72 under
a docstring reading "THE THIRD CLASS EXISTS BECAUSE THE FIRST TWO MISLABELLED
IT". It could not fire here for two independent reasons:
* it keyed on the literal string `(.rodata)`, but main's section_order is
[.rodata, .text, .data, .bss] -- its rodata sits BELOW .text and its jump
tables live in `.data` objects, so TABLE REJECT was UNREACHABLE BY
CONSTRUCTION on the binary with the most jump-table functions left. A
section NAME is not a section ROLE.
* it demanded purity (ro == outside), so 5% perturbed code defeated an
all-or-nothing test and dropped the verdict through to PLUMBING REJECT --
whose advice (fix_arity_callers -> cast_self_callers) addresses the 5% and
cannot touch the 95% that is data. That chain was run on this function
TWICE today and fixed nothing, exactly as the evidence predicts.
Now: table bytes counted in (.data) OR (.rodata), and the test is DOMINANCE
(>=60%) rather than purity, reporting the split and naming which part is the
carve problem and which the declaration problem.
Negative control over all five pre-existing verdict shapes (pure BODY, pure
PLUMBING, pure TABLE, MIXED, NOT FOUND) plus the S75 shape: 5 of 6 verdicts
UNCHANGED, only the SaveLoadRoutine shape flips PLUMBING REJECT -> TABLE
REJECT (MIXED).
Cookbook §447. The law: a class that cannot fire is worse than a class that does
not exist -- it converts "I don't know" into confident, specific, wrong advice.
When a verdict names a subsystem, check that subsystem owns the MAJORITY OF THE
BYTES before acting on it.
|
||
|
|
23cd3a43a5 |
feat(decomp): bank main:func_8005DCA0 (118 ins) — and SaveLoadRoutine's body is BYTE-IDENTICAL
One clean-rebuild gate_main pass over main's stored drafts. 35 drafts on the slate -> 20 compatible after in-TU declaration resolution -> 1 byte-correct, found by bisection in 24 rebuilds. main SHA1 143dbb89... BYTE-IDENTICAL. banked: func_8005DCA0 src/800c3.c 118 ins THE HEADLINE IS NOT THE BANK. gate_main's per-function verdicts separate a BODY reject from a PLUMBING reject, and they say: SaveLoadRoutine: PLUMBING REJECT — SaveLoadRoutine is BYTE-IDENTICAL; all 3989 differing bytes are ELSEWHERE IN CODE. The substitution perturbed other functions (§376 — a stale forward declaration changes caller codegen). SaveLoadRoutine is 1,165 instructions -- the largest function left in the project, 9.2% of everything remaining, and carried as the §434 WALL. Its body is already correct. What rejects it is a forward declaration perturbing OTHER functions' codegen, which is exactly what fix_arity_callers -> cast_self_callers exist to repair. That is a plumbing job, not a matching job. Three genuine BODY rejects, correctly distinguished by the same verdict layer (divergence confined to the function itself): func_8001EFE0 (495 bytes), func_80015B6C (151), func_80011380 (8). Those drafts are really wrong. Honest read of the yield: 1 of 20 substituted drafts was byte-correct, so main's stored drafts are mostly NOT right. This tempers the "the endgame is integration, not drafting" line from the previous commit -- true for the overlays, only partly true for main, where several drafts need redrafting or permuter work. The distinction is now measured per function rather than assumed. Deferred to the reconcile chain, not discarded (11 dropped for in-TU decl conflict + 4 dropped across rounds to let the TU compile at all): func_800226C0 (670), func_800215F4 (465), func_8001FC08 (400), func_8005F290 (61) and the gate's own 11. All drafts remain on disk. |
||
|
|
abea9f0ac2 |
feat(decomp): bank func_8016AE5C (85 ins) + frontier_classify — the frontier is not a drafting problem
func_8016AE5C (ov_SC03_108) was logged "match_one MATCH but the whole-binary
gate rejected -- CAUSE NOT DETERMINED". Determined: the body is byte-perfect (0
differing words inside the function; all 1,168 diffs are uniform +0x20 shifts
outside it) and it emits an 8-entry jump table that was never carved. It banked
unchanged the moment the §446 jtbl_carve per-table bound landed.
tools/frontier_classify.py (NEW) — classify every open stub by its TRUE BLOCKER
from artifacts already on disk (R33/offline-tooling-first: zero tokens, no
agents, no builds). "69 functions left" is a stub count, not a difficulty
measure, and routing drafting agents at carve or plumbing problems wastes them.
A-TWIN-REMAP 3 302 a byte-identical copy is already banked elsewhere
B-CARVE 11 3,301 owns a switch jump table -> the §446 class
D-NEAR 2 106 closeness <=25 -> permuter fuel, not drafting
F-FAR 3 223 draft materially wrong -> redraft
G-DRAFTED-UNK 49 8,724 drafted before, no usable verdict on record
H-VIRGIN 1 1 never drafted (and it is a DATA BLOB, not a function)
68 of 69 remaining functions already have a draft on disk. The endgame is a
verification/integration problem, not a drafting one.
TWO SELF-INFLICTED DEFECTS FOUND BY CHECKING AGAINST KNOWN-TRUE CASES, both the
session's recurring shape (a scan narrower than the claim it supports, R32):
* The sig directory is NOT the fleet. Alongside the 213 real binaries `.run/`
holds `SLUS_007.26` (a STALE duplicate of main under the ROM filename),
`resident_image`, and two CROSS-BUILD binaries (`sep8_SLUS_007.26`,
`aug31_USA_DEMO.EXE`). Counting them as peers reported 38 fns / 7,516 ins of
free twin-remaps -- mostly main "already banked" in ITSELF, the rest proven
in a PROTOTYPE that R13 forbids as evidence. Now derives the fleet from the
Makefile and prints what it ignored. True figure: 3 fns / 302 ins.
* The draft scan globbed `.run/S7*` only, missing `.run/S69m2`, `.run/S68m1`,
`.run/s67m1`, `.run/wave_ds2`, `.run/gate_lane`, `.run/backlog_drafts`. All
32 drafted main functions read as "never drafted", which would have sent
agents to redraft 6,328 instructions that already have drafts. Now one
pruned os.walk of .run (worktrees excluded -- 7.4 GB of duplicate sources).
Honest negative result: resident:func_800D06E8 (344 ins) did NOT bank. I
predicted the carve fix would clear it; it did not. Its blocker is still open.
|
||
|
|
cb948a6bbc |
feat(decomp): the ov_SC01 reloc-only cluster + its 5th latent victim — 5 fns, 1,301 ins
S74 handed this forward as "1,116 instructions behind one question": family_remap
on ov_SC01_004/005/006/008 gated DIFF 4/4 against the banked exemplar
ov_SC01_009:func_8017EB08, and the class had been carried as a codegen wall since
S70. The four bodies were byte-identical to the exemplar the entire time.
Word-level classification vs the exemplar, computed independently twice (a Fable
agent's script, then mine from scratch against the retail images), identical:
nins=279 EQ 213 · RELOC-HI16 23 · RELOC-LO16 24 · INTERNAL-J 19 · CODEGEN 0
Zero register-allocation, instruction-selection or scheduling differences.
ROOT CAUSE — tools/jtbl_carve.py reserved ONE WORD TOO MANY per table:
* spimdisasm runs an island's LAST `jtbl_` dlabel one word into the following
NON-ZERO data (string bytes 0x696F760A / 0x000013FF / 0x62647020), so the
zero-word trim cannot see it; and
* the over-span clamp that would have caught it was guarded by
`len(sltiu_bounds) == 1` -- but `sltiu` is ALSO how gcc emits an unsigned
range check ((u32)(x-lo) < n, I1). These four carry five distinct sltiu
immediates, so the guard silently disabled itself on precisely the functions
that needed it.
0x2C reserved for a 0x28 table => image 4 bytes short => ~850 %lo immediates
shift => whole-binary DIFF about a function whose own bytes are perfect.
Fixed with a PER-TABLE bound: gcc-2.7.2's dispatch is a fixed idiom, so the
`sltiu` nearest ABOVE that table's own %hi(jtbl_X) is unambiguous whatever else
the function tests. Second defect stacked behind it: a carve span whose
JTBL_PADS line lacks a `tables=` comment lost its existing table's start on
merge and refused "table starts do not fit the span" -- which harvest_verify
then "repaired" with a needless jr_isolate_all that walked back into the first.
THE NEGATIVE CONTROL IS THE STORY. Run over every other open table-bearing stub
fleet-wide, the fixed bound changed exactly one more table: ov_SC06_022/
func_80185B80 (185 ins), a FIFTH victim nobody had drafted against. A guard that
disables itself on a common idiom does not fail once -- it fails quietly across
the whole corpus.
Banked, each with its own byte-gate verdict (--no-propagate, clean re-gate):
func_8017EB30 ov_SC01_004 279
func_8017F2D4 ov_SC01_005 279
func_8017F2D4 ov_SC01_006 279
func_8017EC68 ov_SC01_008 279
func_80185B80 ov_SC06_022 185
Also here:
* dedup_propagate: memoize find_site's mask (lru_cache) -- 54 ms of masking
per call over the whole source, recomputed though it depends only on the
text. 2x on that loop (58.3 -> 33.0 ms/call), NC identical on 120 addrs.
Scoped honestly: that loop is ~2.4 min of a 30-min run; the profiler puts
43% in family_remap._alias_decl_for, which is NOT fixed here.
* Makefile: `clean` says out loud that BINARY= is ignored and it is fleet-wide
(cookbook §445) -- it silently deleted asm/ for all 213 binaries this session.
* Cookbook §446 (the carve law: when a standalone-MATCH jtbl draft gates DIFF,
diff the carve extent against 4 x sltiu before touching the body), §445, and
SETUP rows for both tools (R21).
* CURRENT_PHASE: the S75 log, incl. the measured fleet dedup-hygiene census
(~2,073 fns / ~12,116 items, all ALREADY MATCHED -- cleanup, not work) and
Drew's decision to leave it and gate --no-propagate from here.
|
||
|
|
0c2b37a287 |
fix(gate): overlays.mk carve-state snapshot was singular, so the revert half-restored
Found by running one reject to ground. After a gate that REJECTED
resident:func_800D06E8, config/overlays.mk had a 4th JTBL_PADS entry and had
LOST `--pre hdr.rodata.o` (the §440 resident leading-rodata sandwich). The
binary then would not build at all -- "consumed 3 rodata jump table(s) but 4 pad
spec(s) given -- table-count drift vs the carve" -- while src/ was perfectly
clean, which is the only place anyone looks before building.
Root cause is a silent narrowing in the classic shape. harvest_verify snapshots
ONLY the gating binary's own overlays.mk block on purpose (the file is shared by
every parallel gate; a whole-file restore resurrects other binaries' lines --
the S62 defect). But _mk_block_span was SINGULAR: the first `# --- <binary>`
header through the next `# --- `. A binary whose carve state spans more than one
block was half-snapshotted and silently half-restored. It returned a TRUE span
for a scope smaller than the caller believed, and nothing compared the two (R32).
Blast radius measured before costing (R37): 1 of the 142 binaries that have a
block -- resident, which has exactly two (§8e pad spec, §8f leading-rodata
sandwich) and still holds 587 instructions of open stubs.
_mk_block_spans (plural) snapshots a LIST, restores tail-first so earlier spans
stay valid, collapses to the snapshot when the header count changed rather than
leaving half-state, and RE-READS and compares the result -- the defect it
replaces was a reported success. _mk_block returns None (not []) for the 71
binaries with no block, so the caller's guard keeps its meaning.
Negative control, three ways:
* snapshot -> restore is a NO-OP on 142/142 binaries with a block;
* the real S75 damage is fully undone;
* the OLD single-block restore provably does NOT undo it -- the positive
control that proves the fix is load-bearing, not decorative.
Cookbook §444 also records the two other findings from the same reject: the
classified ledger stores the LADDER'S FINAL verdict (the recorded CC1-FAIL came
from a late sig_unify rung; the raw draft compiles and fails on BYTES), and
match_one MATCH + rtu_match MATCH is still not bankable -- func_800D06E8's real
blocker is a jump table (built binary 20 bytes longer, 0x800CEDFC holds a table,
69,571 words shift), because neither matcher LINKS.
|
||
|
|
cdd535e45b |
fix(tools): reconcile_tu's cc1 premise, the &-cast arms, and the worktree sig gap
The S74 checkpoint's "one unfixed defect that is actively costing banks"
(reconcile_tu manufacturing declaration conflicts), run to ground — plus the
harness gap that produced a false carve-corruption verdict.
reconcile_tu.py — three defects, measured against the real gcc-2.7.2 front end
(cdecl._cc1_accepts, the oracle cdecl.compatible was validated with; R33):
* The premise "a decl BELOW still conflicts" is TRUE at file scope and FALSE
at block scope. cc1 ACCEPTS a block-scope extern against a TU decl below it
(pedwarn "type mismatch with previous external decl"); conforming it is
destructive, because the TU's decl names the TU's TYPE and a type declared
below the splice point is not in scope AT it -- the emitted result gets
"syntax error before 'D_x'". Byte-witnessed on resident:func_800D06E8 (344
ins), whose block-scoped `extern Blk80078E78` became `extern
Struct80078E78`, typedef 388 lines lower. That construct is what this
ladder's OWN scope_demote_drafts (§8d) rung emits on purpose, and three
already-banked functions in that TU use it: one rung undoing another.
* The cast pass rewrote COMMENT PROSE -- 8 rewrites inside one header comment,
including inside a quoted cc1 diagnostic. Now matches on cdecl._mask
(length-preserving, so a mask offset is a source offset) and splices into
the original.
* `&sym` emitted `&` applied to a cast: legal for the scalar arm, `invalid
lvalue in unary '&'` (measured) for the array/fnptr/fnptr_array arms. `&`
now selects a pointer form and consumes itself -- but ONLY with no trailing
subscript, because `&sym[i]` is the address of ELEMENT i and the old code
had that case right. That last clause exists because the R39 negative
control caught the fold as a regression in the first cut of this fix.
gate_stage.py — `--skip-stages` / `GATE_SKIP_STAGES` (loud when used). Stage 0
gates raw drafts first, so a broken rung can only cost a RECOVERY, which is
exactly what makes it invisible: the function it destroys was already failing,
so its DIFF reads as a fact about the function.
verify_worktree.py / jr_isolate_all.py / parallel_gate.py — provision() now
symlinks every .run/sig.*.jsonl (main clone 259, provisioned worktree 0), the
third member of the class holding extracted/ and .run/obj40. parallel_gate was
fixed for this identical bug in S69: two provisioners, no shared list, found
twice; they now cross-reference each other. jr_isolate_all no longer swallows
the resulting FileNotFoundError into `except: continue` -- that turned a missing
index into a confident carve-CORRUPTION verdict over 2,603 of 2,603 functions
(R54). Adds _assert_scan_covered: attempted == raised means the scan measured
nothing, so its zero is an artifact, not a finding (R32).
Verification:
* 4 cc1 probes (the table above), each run on the pinned front end.
* R39 negative control over the stored-draft corpus: 661 adjudicated, 652
IDENTICAL, 9 CHANGED and every one an intended class. 4,173 of 4,864 drafts
unadjudicable (filenames that are not func_<ADDR>) -- stated, not hidden.
* jr_isolate_all ov_SC03_105 --dry-run: unchanged in the main tree.
* make clean/extract/build BINARY=resident -> 8e17e02f... BYTE-IDENTICAL.
Docs ship with the change (R21): cookbook §442/§443, index regenerated (1,112
sections), 3 docs/SETUP.md rows, CURRENT_PHASE S75 log.
|
||
|
|
7f7e1c7d90 |
docs(phase-31): refresh the S74 checkpoint — §440/§441 and the doc-state line
The checkpoint listed §435-§439 and was written before the documentation catch-up; a stale checkpoint is worse than an absent one. Now records that every tool change this session carries a SETUP row and a cookbook entry, and names the rule the gap taught. |
||
|
|
eb0329b3a4 |
docs: the five agent-authored tool changes that shipped with a commit message and no docs
I answered Drew's yes/no honestly — NO — and this closes it. Every gap had the same shape: a tool
change that came from a SUBAGENT arrived as a report, I merged the code and wrote it up in the
commit message, and a commit message is not the knowledge base. The six changes I made myself were
documented inline; these five were not.
SETUP.md tooling ledger:
* `ld_interleave` — the row still said "interleave linker inputs" and predated BOTH --order
(S72, main's 7-piece island) and --pre (S74, the resident's leading-rodata header).
* `harvest_verify` — the typedef strip-set is computed SCOPED (`above=fn`) now, and why.
* `jtbl_rodata_pads` — a new row for the three S74 measurement corrections, each of which ACCUSES
THE CARVE when it fires, plus why the trailing-.align one stayed latent (zero_gap self-corrects
an undershoot when the next item is an anchor, and a C jump table has no anchor).
* NEW row `jtbl_carve` — the `covered` / `covered-tpad` verdicts.
* NEW row `jr_isolate_all` — `_region_emit_start` and the empty-closing-region skip.
Cookbook:
* §440 — a carve piece binds to a SUBSEG, not a function, so §8b's "non-adjacent => ISOLATE" is
over-strict: EXTEND the carve across still-stubbed material instead. Four byte-proven
corollaries (migrated tables self-align by SPAN-RELATIVE offset; JTBL_PADS counts cc1 tables
only so a mixed span's spec grows as siblings bank; the zero-word rule is invalid across a
migrated boundary; a covered table at 4-mod-8 gains 4 bytes when it banks). Plus the resident's
rodata->text->data->rodata->data layout and why it needed --pre.
* §441 — three more instrument defects that each produced a confident, precise, WRONG verdict
about a correct draft, with the habit they share: when a gate rejects a body you have
byte-verified standalone, the first suspect is the gate.
Playbook: new step 2a-0 — the same-address lead is size-filtered now; read the `⚠ IGNORE` line, and
regenerate any pack built before S74 rather than trusting a bare address lead.
|
||
|
|
b24394e732 |
docs(phase-31): S74 FINAL checkpoint — 50 banked, seven tool defects, R22 213/213
Every reject class this session was an instrument defect, not a codegen wall: 24 already-MATCHed bodies were sitting behind seven tool bugs, six of which are now fixed. The checkpoint names the seventh (reconcile_tu manufacturing declaration conflicts) as the first thing to fix next session, and the harness gap (verify_worktree omits the sig files, and the scan that needs them swallows the error and reports a false corruption) beside it. Fleet verified from a clean rebuild AFTER the last bank: check-all 213 passed, 0 failed of 213. split_indicator 213 OK and now a hard gate. INCLUDE_ASM lines in src/ 1,086 -> 1,036, measured at both commits — the same 50 the gates reported, counted independently from the source. |
||
|
|
8edb918480 |
feat(cards): size-filter the same-address lead (§238 homonym) + bank the S74 lever set
THE CARD USED TO HAND AGENTS A WRONG TWIN ABOUT ONCE IN FIVE. `⭐ func X IS BANKED AT THIS ADDRESS`
never checked that the two functions were the same SIZE, and overlays share addresses between
unrelated functions as readily as they share code. Measured over this session's ~60 cards: about a
dozen agents reported discarding the lead themselves, and one card advertised a 72-instruction
namesake — with journal history claiming "already MATCH closeness 0" — to a 241-instruction target.
A confidently wrong lead costs more than no lead, because the agent believes it.
corpus.sig already carries `nins` and `h_seq`, so the fix is free: `_same_addr_banked` now returns
(binary, nins, h_seq); the card keeps a lead only at a MATCHING instruction count, marks it strong
when the mnemonic skeleton matches too, and prints an explicit `⚠ IGNORE` naming the binaries where
that address holds something else, with both sizes.
VERIFIED IN BOTH DIRECTIONS against known-true cases before being believed (never trust a filter you
have not tried to fool):
* the trap: ov_SC03_105:func_801806F8 (241) vs ov_SC03_013 (72) -> `⚠ IGNORE`.
* the positive: ov_SC02_003:func_80187B40 (158) -> strong lead to ov_SC02_000 (158, same h_seq,
banked this session) AND, in the same card, warned off ov_SC04_011's 138-ins homonym at that
same address. That is precisely the pair a wave agent sorted out by hand hours earlier.
Cookbook §438 (the law: a lead is fuel only if it carries the cheapest fact that can refute it —
size refutes a homonym for free and nobody had asked) and §439, the S74 lever set: MEM_IN_STRUCT_P
as a two-way alias-oracle dial (four agents converged on it independently); `goto`-into-a-shared-tail
vs longhand as a REGALLOC dial because gcc-2.7.2 cross-jumps after allocation; `for` -> do/while as a
length-changing scheduling dial; allocno PRIORITY via a non-volatile asm at a loop head, with the
measurement that register pins are actively harmful for that class; the -O0 global-RMW rule
(`x++` emits the copy-back quartet, `x = x+1` does not); why `sll 16; srl 16` survives only across a
CALL; `sltiu N` without `addiu -1` proving an empty `case 0` is mandatory; block-scoped temps in
duplicated bodies; two `register asm` vars cannot share a hard reg; and `x*32` vs `x<<5` emitting
lh vs lhu — which match_one's %lo mask HIDES, so it must be checked with objdump.
|
||
|
|
5208f2279d |
feat(decomp): parallel gate — 2 fns across 1 binaries (1 workers)
ov_SC06_029 func_80182ED8 func_80184084 |
||
|
|
6e840730a9 |
feat(carve): bank 4 more via the carve chain — and §8b's "non-adjacent => ISOLATE" is over-strict
resident:func_800D00E4/func_800D02D0/func_800D0488 + ov_SC07_002:func_80180248, all byte-verified
from clean rebuilds (resident 8e17e02f, ov_SC07_002 fad71342) and counted from the SOURCE.
ov_SC06_029's two are re-gated separately against HEAD — this agent's worktree predated five banks
there, so its numbers for that binary no longer apply.
TWO OF THE SIX NEEDED NO CARVE WORK AT ALL, AND CARVE-REFUSED WAS AN INSTRUMENT VERDICT.
ov_SC07_002:func_80180248's table is ALREADY inside a carve bound to its own subseg: in stub state
spimdisasm migrates the table into the fn's .s and the object fills the piece exactly, so banking
just swaps that block for cc1's identical one. `island_probe` classified it `tail` on the table's
ADDRESS, `apply()` routed it to build_carve, which resolves spans out of the RAW data asm where a
carved table no longer is -> "not found in the raw data asm" -> harvest_verify booked CARVE-REFUSED.
A verdict about the route we chose, not about the function (R43). jtbl_carve now has a `covered`
verdict (table inside an existing carve bound to the fn's OWN subseg) and a `covered-tpad` wall (the
retail copy carries a trailing §8a pad the matched body won't emit — bankable, needs a `0t<n>`
entry); a fully-covered batch is a no-op before either route.
THE RESIDENT CAN CARVE LIKE AN OVERLAY. Its three tables are adjacent and lead the island
(0x450e0..0x451ac, one span, all in subseg `resident`). The genuinely new part: the resident opens
with `- [0x0, rodata, hdr]`, a 1-word .rodata header BEFORE the code, so its layout is
rodata -> text -> data -> rodata(carve) -> data, which `ld_interleave --order` cannot express (every
listed piece lands after TEXT_START, and hdr.rodata.o would fall into the unchecked `empties` bucket
and be parked after the text, moving every byte). New `--pre` places a leading-rodata piece ahead of
the text; resident_JTBL_INTERLEAVE uses it.
NEW LAW, BYTE-PROVEN (§8b was over-strict — EXTEND the carve, do not isolate): a .rodata carve piece
binds to a code SUBSEG, not a function, and the object's .rodata is the address-ordered
concatenation of cc1's tables for BANKED functions and still-stubbed functions' MIGRATED tables. So
a span may legitimately hold a MIX, and extending a carve across an align-pad word and two unrelated
STILL-STUBBED tables was byte-identical with nothing banked — where the tooling demanded a
jr-isolation. Corollaries, all measured: migrated tables self-align (spimdisasm emits `.align 3` iff
the table's SPAN-RELATIVE offset is 8-aligned), so stubbed tables need no spec; JTBL_PADS counts cc1
tables only, so a mixed span's spec GROWS as each sibling banks; and the zero-word rule is INVALID
across a migrated boundary, because that zero is supplied by the preceding migrated block.
ALSO REPORTED, NOT FIXED (harness gap worth its own change): verify_worktree.provision omits
`.run/sig.<bin>.jsonl` — main clone 259 files, provisioned worktree 0 — and jr_isolate_all's
carve-ownership scan swallows the resulting FileNotFoundError in a bare `except: continue`. Measured:
2603 of 2603 functions raised, the scan found 0 owners, and the run aborted with a CONFIDENT FALSE
verdict ("committed .rodata carve ownership is not 1:1 — stranded/duplicated carve"). Both resolve
instantly once the sigs are present. Any worktree-run isolation before that is fixed reports a
corruption that is not there.
|
||
|
|
089311e74d |
feat(md_SC07_004): 10/10 banked — md_SC07_004 is now ZERO stubs, and the "decl conflicts" were fake
Clean rebuild BYTE-IDENTICAL 87ac0de3; corpus.stubs('md_SC07_004') 10 -> 0, counted from the SOURCE.
THE §376/§378 CHAIN WAS NEVER NEEDED. Zero declaration edits: no fix_arity_callers, no
cast_self_callers, no --any-proto, no --sync-decls, no undo journal. `git diff -U0` on the TU removes
exactly the 10 INCLUDE_ASM stubs plus one hoisted typedef. Every recorded "declaration conflict" was
an INSTRUMENT defect. Four of them, all named, three patched here:
1. `CC1-FAIL(no-diagnostic)` was neither cc1 nor no-diagnostic. The failing stage was
`jtbl_rodata_pads --derive`, which prints to stderr AFTER cc1 exits 0 quietly. `_items` matched a
rodata anchor only as `D_xxxxxxxx:`, but a block written as inline `__asm__` in C arrives in the
labels.inc macro form `dlabel D_xxxxxxxx` — so an 8-byte hole opened in the walk and every C jump
table after it died with "island layout drift". The harness label was wrong twice: it said CC1
when the failure was a post-maspsx filter, and no-diagnostic when there was a precise one.
2. Same file, UNALIGNED ANCHOR: the ctable branch read `word(pos)` without first stepping the
sub-word zero gap, so a preceding `.asciz` ending at an odd address made it refuse a correct
layout. Now reuses the same zero_gap the anchor branches already use — a no-op wherever pos is
already aligned, i.e. everywhere that builds green today.
3. `harvest_verify` computed the typedef strip-set UNSCOPED: `cdecl.typedef_names(path)` without
`above=fn`, which that function supports for exactly this. A typedef declared BELOW the splice
point got stripped out of the draft that needed it -> `parse error`, logged as PLUMBING and
indistinguishable from a real conflict. One line.
4. NOT PATCHED, AND THE MOST IMPORTANT ONE: `reconcile_tu.py` (gate_stage's `-rc` stage) MANUFACTURED
both remaining "conflicts". The same drafts gate 9/9 byte-identical through harvest_verify and
7/9 through gate_stage. Isolated stage by stage, `-rc` (a) rewrites deliberately BLOCK-SCOPED
externs to a file-scope spelling whose typedef is declared ~2,800 lines lower — overwriting the
TU's own byte-proven house style, which three already-banked functions in that file use; and
(b) substitutes identifiers TEXTUALLY, including inside comments and inside `&`-expressions,
emitting `*(T *)&((s32 *)&D_800AE620)`. A correct draft using the block-scope-extern idiom
currently CANNOT survive gate_stage. Left for a deliberate fix: `--stages` should be able to skip
reconcile_tu, or a draft should be able to opt out.
The two surviving non-stub source edits are byte-neutral (proven by the SHA above): a §304
migrated-rodata re-emission (`D_801A01EC`, the exact form this TU already uses three times, needed
because banking the body deletes the .s that carried the island word), and one typedef moved up so a
function above it can see it (typedefs emit no bytes).
Also banked the 10th stub (func_801ADA10) that defect 3 had been silently blocking.
Regression-checked by the agent: main, md_MAIN_003, md_SC07_003, md_SC03_073, md_MAIN_011 all
rebuild BYTE-IDENTICAL. A full R22 follows before this session closes.
|
||
|
|
2954b250ec |
feat(decomp): parallel gate — 5 fns across 1 binaries (1 workers)
ov_SC06_029 func_801801D8 func_80180A70 func_801867D0 func_80187660 func_801898CC |
||
|
|
5e10215269 |
feat(md): bank the 4 -O0-stranded functions — and the class is now essentially empty
md_MAIN_011:func_800CF28C · md_MAIN_003:func_800D0268/func_800D0740/func_800D0C50, all byte-verified
from a clean rebuild and counted from the SOURCE: md_MAIN_011 is now FULLY MATCHED (0 open stubs),
md_MAIN_003 is down to 1 (func_800CF3E8).
THE PREMISE I HANDED THE AGENT WAS WRONG, AND IT SAID SO. md_MAIN_011 is already a whole-object -O0
module — no carve was needed. Its real blocker was tools/jtbl_rodata_pads._s_rodata_span ignoring a
trailing `.align`, the SAME defect this session fixed for md_SC07_003 from the other direction: two
agents converged on it independently. Adopted this agent's stricter form (only a TRULY trailing
align rounds `hi`; an interior one is followed by data that sets `hi` higher anyway).
Note WHY it stayed latent: `derive`'s zero_gap self-corrects a 1-3 byte undershoot whenever the next
stream item is an anchor. A C jump table has NO anchor — so the bug can only fire the moment someone
banks a switch function into such an object, and when it fires it accuses the CARVE ("island layout
drift"), not itself.
md_MAIN_003 needed one new -O0 object, and the boundary I proposed (0x1f74 -> 0x1e58) was both too
narrow and off by 0x2B8. The carve made is `md_MAIN_003_o0e` at 0x1308 (vram 0x800D0100) running to
the existing o0c boundary: everything in that span is a §265 verbatim __asm__ body or an INCLUDE_ASM
stub — zero optimizable C — so the whole tail flips with one cut. Proved byte-identical with NOTHING
banked first (§431 discipline), then the three drafts gated one at a time.
TWO MORE GENERAL DEFECTS FIXED IN jr_isolate_all, both of which silently mis-place a boundary:
* an item-less CLOSING region emitted a duplicate `- [off, c, …]` line and the validator refused;
the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing
region look non-empty.
* A §265 VERBATIM __asm__ BODY IS PREAMBLE, AND PREAMBLE IS ASSUMED BYTE-NEUTRAL. It is not — it
emits bytes. `parse_overlay_c` has four addressed-anchor forms and a verbatim body is none of
them, so it attaches to the NEXT anchor: cutting at func_800D0268 would have moved 0x168 bytes of
other functions into the new object while the yaml claimed the region starts higher. New
`_region_emit_start()` derives the yaml offset from the region's CONTENT (item addresses + every
.globl/.ent the text names that resolves inside the object) and takes min(cut, emit), so a
boundary can only move DOWN. Where no verbatim asm is in play it equals the cut — every existing
isolate is unchanged.
BLAST RADIUS PROVEN, not argued: jtbl_rodata_pads is in the build path (`--derive` for md_*/main),
so the agent rebuilt main + all 70 md_* from scratch (71/71) and then ran the full fleet:
**make check-all 213/213 passed, 0 failed**, main 143dbb89 BYTE-IDENTICAL.
CENSUS, denominator asserted (1057 live stubs, 0 without a .s): exactly ONE -O0-prologue stub
remains stranded in an -O2 TU fleet-wide — main:func_8002C410 in src/800_b.c, 299 ins. Nothing more
should be built for this class; the general tool already existed and what was missing was
correctness, not coverage.
|
||
|
|
3a886b9652 |
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
|
||
|
|
e9220d2d8a |
fix(pgate): a carve left asm/ stale, and the refusal that reported it named nothing
TWO DEFECTS, ONE INCIDENT. ov_SC03_105's own SUCCESSFUL gate committed an isolation's new TUs (src/ov_SC03_105/ov_SC03_105_jr_801813BC.c, _jr_80181C84.c) whose `INCLUDE_ASM` lines name .s files that do not exist until a re-extract. corpus.stubs then refused — correctly, "the tree and the source disagree" — so the NEXT gate on that binary died before doing any work, and a matched body (func_801818E8) sat unbankable behind it. 1. THE REASON NOW TRAVELS WITH THE REFUSAL. stubs_of() returned a bare None and the caller printed "corpus refused in worktree": true, and naming nothing. It took a hand-built worktree to see that corpus had said exactly what was wrong all along. It now returns the message and the result JSON carries it. Verified against a TRUE reproduction (delete one .s in a scratch worktree): verdict REFUSED + "1 stub(s) have NO .s on disk ... src/...:4214: asm/.../func_8017F018.s". 2. THE MERGE STEP REPAIRS WHAT IT BROKE. For every binary whose carve created a new source file, assert corpus.stubs is satisfiable in the MAIN tree; if not, `make extract BINARY=<b>` and re-assert; if it is STILL unreadable, say so loudly rather than leaving a tree no tool can read (R32/R43). This is the R22 corollary — a config change needs a make extract, not just a make check — firing inside a tool's own commit. Repaired the live instance by hand first: rm -rf asm/ov_SC03_105 + extract + build -j + check -> BYTE-IDENTICAL d305ff6d, corpus readable again, and func_801818E8 then banked (commit:3718). Cookbook §436-D; wave-playbook §6 carries the hand-gating version of the same warning. |
||
|
|
d0389352d7 |
feat(decomp): parallel gate — 1 fns across 1 binaries (1 workers)
ov_SC03_105 func_801818E8 |
||
|
|
0109f5bc92 |
feat(decomp): parallel gate — 1 fns across 1 binaries (2 workers)
md_SC07_003 func_801A09C8 |
||
|
|
ee2963514a |
fix(tools): jtbl_rodata_pads measured a .s rodata span without its trailing .align
`_s_rodata_span` summed only the DATA an included `.s` emits, so a file ending `.asciz "r"` + `.align 2` measured 0x801A00D8..DA instead of ..DC. The island walk then landed 2 bytes short and `--derive` aborted with `C table entry 0 at 0x801A00DA ... island layout drift` — a true statement about a span that was never the real one. Worse, the Makefile pipes md_*/main through `--derive` without `set -o pipefail`, so the failure could yield a short object rather than stopping the build. Three lines: round `hi` up to the trailing align, which is what the assembler actually emits. CONTROLS (both on UNMODIFIED sources, so this is proven byte-neutral, not argued): md_SC07_003 clean rm -rf + extract + build -j + check -> BYTE-IDENTICAL 46af79a1 gate_main --assert-baseline -> BASELINE GREEN 143dbb89 Found by a drafting agent (md_SC07_003/func_801A09C8) that ran its own gate reject to ground instead of respelling its body, and proved the patch in scratch first: with the fix its draft's .rodata is byte-identical to the green control and .text differs in exactly 1 of 6266 words — a %lo(jtbl) carrying a section-symbol reloc that three already-green C-jtbl functions in the same object already ship. Cookbook §436-C, with the habit that found it. |
||
|
|
7ce27c0d1f |
feat(decomp): parallel gate — 2 fns across 2 binaries (6 workers)
ov_SC05_010 func_8017F5B8 ov_SC03_024 func_801830A8 |
||
|
|
7c2342f87f |
feat(decomp): parallel gate — 5 fns across 3 binaries (4 workers)
ov_SC02_011 func_80183178 ov_SC01_084 func_80182328 ov_SC03_105 func_801813BC func_80181C84 func_8018624C |
||
|
|
797cb97cb7 |
feat(decomp): parallel gate — 9 fns across 6 binaries (8 workers)
ov_SC06_024 func_8017EC4C ov_SC06_029 func_8017F9C0 ov_SC03_105 func_801867D0 ov_SC05_010 func_80180F84 ov_SC02_005 func_80185060 func_80185E80 ov_SC02_011 func_80183630 func_8018418C func_80188E3C |
||
|
|
f2f3103691 |
feat(decomp): parallel gate — 7 fns across 7 binaries (10 workers)
ov_SC05_001 func_8017FE0C md_SC07_003 func_801A1120 ov_SC03_013 func_8017FAA8 ov_SC03_011 func_80180B68 ov_SC02_000 func_80187B40 md_SC07_004 func_801A94A0 ov_SC07_000 func_8017ECB4 |
||
|
|
95c7b7fe0f |
fix(tools): two tools read a source of truth describing a different world (+ hard-gate the third)
Three independent split agents hit both defects in one session, on the tools that CERTIFY and UNDO
the work they were doing. Each is fixed, negative-controlled against the exact failing case, wired
into its siblings, and documented in the same change (cookbook §436).
1. split_indicator attributed a jump table by the STUB'S DIRECTORY PATH. `make extract` does not
prune a re-homed subseg's `nonmatchings/<old>/` dir, so after a correct, byte-green §431 split
both the old and new dirs hold the moved stub — and the tool printed NEEDS SPLIT for a split that
was already correct. owners() now derives the owner from the CONFIG by address (R33), exactly as
jtbl_carve.func_subseg already does for the identical §8b hazard, and NAMES any leftover stub in
a `note:` line. Notes now print on an OK verdict too: hiding one behind `st != OK` is the same
defect in the other direction — a true verdict about a narrower world than the reader believes.
PROVEN by planting a stale stub for func_80182A00 under its old subseg: OK + the note, where the
old code would have seen one subseg owning two spans. --self-test still PASSes both directions.
2. jtbl_carve --revert did `git checkout --` on the WHOLE splat yaml. The carve owns only the
trailing data/.rodata region; the `c` pieces are source configuration it never writes. The blunt
form cannot tell "carve state I just added" from "the §431 split someone added to the same
uncommitted file", so --revert after a carve PROBE silently un-split the overlay — each agent
recovered only because they had backed the yaml up by hand. It now splices back only its own
region (parse_config gained an optional `lines=` so the SAME region derivation runs over the
committed text — one derivation, two callers), refuses loudly if the committed region carves onto
a subseg the current config no longer defines, and reports how many uncommitted `c` pieces it
preserved. PROVEN in the ov_SC01_084 worktree: carve → revert → the uncommitted split survived
("PRESERVED 30 uncommitted `c` piece(s)"), carve lines gone, diff back to the 6 split lines.
SIBLING: jtbl_family_bank.revert carried the same blunt checkout for the isolation's code pieces.
It now keeps whatever pre-dated the attempt (the `keep_regions` signal it already trusts for
src/) and NAMES anything it drops — an isolation region and a §431 split piece are both
`<ov>_jr_<addr>`, so no name test can tell them apart and only that signal can.
3. NOT A DEFECT, and recorded as such: a speculative carve fails the build with `jtbl_rodata_pads:
consumed 3 rodata jump table(s) but 9 pad spec(s) given`. That is R43 working — the pad spec is a
CONSEQUENCE of banking, not a prediction of it — and it reproduces identically on the pristine
unsplit config, so it is never evidence about a split.
make tools-health: split_indicator is a HARD GATE now, as its own comment promised it would become
once the last violation was split. 213 OK of 213; a new one fails the build instead of being echoed
past.
Cookbook §435 (an overlay TU split is near-free — 0/3,074, 1/2,679, 2/3,254 names crossed, because
the §8b carried decl layer re-emits externs per region so only typedefs can cross; and the gap test
between two rodata runs is "is this word a valid code address", not "is it zero") + §436 (the two
defects and the shape they share). Playbook + SETUP.md carry the emptied CARVE-BLOCKED class.
|
||
|
|
c7d67048a0 |
chore(waves): regenerate config/wave_exclude.txt — the CARVE-BLOCKED class is gone
16 of 27 entries dropped as STALE, all of them the CARVE-BLOCKED set the four §431 splits in commit:3709 just unblocked. 11 WALL entries kept (curated, pinned by their `# WALL:` annotation, which exclude_audit re-reads as a pin so a wall survives regeneration). An exclude list records what the TOOLING could not do, so it becomes a list of work you decided not to do the moment the tooling improves — regenerating it is part of the fix, not follow-up hygiene. draw_waves --exclude-file audits and REFUSES a stale list. |
||
|
|
33c292dea5 |
feat(overlays): split the 4 carve-blocked subsegs at their jtbl-span TU boundaries
BYTE-IDENTICAL on all four, with NOTHING banked (clean rm -rf asm/<bin> + extract + build -j + check), which is the whole point: the structure lands first and proves neutral, then drafts bank against it. split_indicator: 213 OK, 0 needing attention, of 213 — the CARVE-BLOCKED class is now EMPTY fleet-wide. One code object contributes exactly ONE contiguous .rodata run, so a subseg owning raw jump tables in two non-adjacent spans could carve only one of them and every switch function in the other span was unbankable at any effort (cookbook §426/§431). ov_SC01_084 2 pieces cut 0x80182A00 (0x5A8A8) ov_SC02_005 3 PIECES cuts 0x80185060 (0x5CF08) + 0x80185E80 (0x5DD28) ov_SC02_011 3 PIECES cuts 0x80183178 (0x5B020) + 0x80188E3C (0x60CE4) ov_SC03_105 2 pieces cut 0x8018624C (0x5E0F4) TWO OF THE FOUR NEEDED A CUT THE BRIEF DID NOT NAME, and the address evidence found it: each already had a carve run that could not merge with span 1, separated by rodata that is not padding — ov_SC02_005 by `0000F040 00000000` (8 bytes, twice the widest .align 3 pad the JTBL_PADS spec can emit), ov_SC02_011 by `FEBEF6AE 000002DC 0 0` (a TU's trailing const data). A gap detector keyed on zero words would have merged them and produced an unbuildable carve: the load-bearing test is "is this word a valid code address in this overlay's text range", not "is it zero". ov_SC01_084's divider is real data too (`0 FFFF0000 00080000 0 0`), while ITS span-1 gap word IS a zero .align 3 pad and merges. OVERLAY SPLITS ARE NEAR-FREE, AND THE REASON IS STRUCTURAL — the opposite of main. The Phase-26 §8b carried decl layer re-emits each region's externs locally, so only typedefs cross a cut: 1 name of 2,679 (ov_SC01_084, 0 typedefs) · 5 typedefs of 44 (ov_SC02_005) · 2 names of 3,254, 0 typedefs (ov_SC02_011) · 0 of 3,074 with zero compiler errors (ov_SC03_105). main's split moved 57 of 1,247. Every crossing typedef was MOVED to a <bin>_shared.h, never copied, and every list came from the compiler (R33). Carve probes (jtbl_carve --func, then reverted — carve state is added when a function banks, never speculatively): all four subsegs now accept a carve with no fail-loud, and jtbl_carve derived the §8e per-table pad specs the zero-word rule predicts. Unlocks 17 open switch functions: ov_SC01_084 func_80182A00 · ov_SC02_005 func_80185060, func_80185E80 · ov_SC02_011 func_80183178, func_80183630, func_8018418C, func_80188E3C · ov_SC03_105 func_801806F8, func_80180ABC, func_80180EC0, func_801813BC, func_801818E8, func_80181C84, func_8018624C, func_801867D0 (+2 more span-1 owners). CORRECTION, measured not assumed: config/wave_exclude.txt listed ov_SC01_084:func_80182328 as CARVE-BLOCKED and it never was — its table ABUTS the existing carve, so it always merged into one run. Proven by control on the PRISTINE unsplit config: --func func_80182A00 exits 1 "would host NON-CONTIGUOUS .rodata carves", --func func_80182328 succeeds. |
||
|
|
aa302aa050 |
docs(phase-31): S73 FINAL checkpoint — consolidated, verified, fresh-session ready
Supersedes S73 CLOSE and its addendum. Every number re-verified against the repo: REAL 880/1,918, MAIN 56.5%, frontier 124 (main 36), main jtbl fns 25 -> 2. |
||
|
|
a0ff94626a |
docs(phase-31): all 40 actively-misleading audit findings worked, not deferred
Replaces the 'known-remaining' TODO list with what was actually fixed. The four that were not doc typos: progress.py undercounting REAL by 7 (the #else half of a NON_MATCHING block is live code and classify() swallowed it), the silently deleted §429, the false §265 accusation in §434, and memory-map.md:309 claiming a 'verified' extent that overlaps the new span-B carve. |
||
|
|
78ef96f606 |
docs: cookbook §426/§434, memory-map extent, Makefile overlay comment
* §426 listed three localizer verdicts; there are four, and the missing TABLE REJECT is the dominant residual on main's switch functions (§433). Its span-B table also still advertised SaveLoadRoutine as an unlockable owner — it is the §434 frame pair. * §434 quoted SaveLoadRoutine at 1139 instructions; the .s has 1165. * docs/memory-map.md:309 recorded saveHeaderTemplate @0x80072DF0 with 'handler code ptrs @+0x54' at the ledger's HIGHEST confidence. 0x80072DF0+0x54 = 0x80072E44, which is jtbl_80072E44 — func_8002B0B4's dispatch table and the first 12 bytes of the S72 span-B carve. The row's extent is wrong past +0x54 and now says so; a 'verified' row that overlaps a carve boundary is how a future resegmentation gets talked out of itself. * Makefile's overlay --front/--tail comment sat directly under main's --order call with nothing distinguishing them; now says which is which. |
||
|
|
c43008a48b |
docs(playbook): seven stale spots — the copy-paste command was the worst
* Step-1's draw command still passed a superseded .run snapshot through the UN-AUDITED --exclude flag. Running it verbatim bypassed the freshness prerequisite built this session. Now --exclude-file config/wave_exclude.txt with a fresh --ledger. * Both exclude populations were wrong: '96 jtbl functions build_carve refuses' is 16 across 4 overlays (split_indicator derives it), and the seven .run/S6*_walls.txt ledgers are superseded by the WALL entries pinned in the canonical list. * Entry count said 19; it is 26. Replaced with 'trust exclude_audit, never a number written here' — a count in prose goes stale the moment anything is added. * Model routing still had a Sonnet band Drew abolished, and no mention that Fable is exhausted account-wide (three agents died on the limit in S73 at ~133k tokens each). * Section 1c's census was pre-session: 25 of 59 main jtbl functions, 'every one now drawable', and SaveLoadRoutine as the flagship drawable example. It is 2 of 36, and those two are the §434 frame pair, excluded from draws. * The gate step listed three gate_main verdicts; there are four, and the missing TABLE REJECT is the DOMINANT residual on main's switch functions (§433). * Triage still named jr_isolate_all as the usual CARVE unblock; it does not yet produce an assemblable object, and §431 is the cheaper route. |
||
|
|
57e5f970c8 |
docs(tools): four docstrings describing pre-session behaviour
* draw_waves Usage advertised [--no-main], which argparse never defined (the flags are --main / --only-main, and main is excluded by default), and omitted --exclude-file, which is now a PREREQUISITE that refuses a stale list. * jr_isolate's STATUS block still declared the tool BLOCKED on split_src_region with the blocker unbuilt. Five defects were fixed this session and it runs the full chain to completion; what remains is a duplicate-definition class at assembly. Says so, and points at §431 as the cheaper route than finishing the item model. * ld_interleave's layout diagram — the first thing anyone reads — showed the pre-S72 three-piece island with 6324C.data.o. main's island is SEVEN pieces driven by --order; --front/--tail is the overlay form now. * jtbl_rodata_pads described a stored-spec-only filter and advertised guards that no longer all exist; --derive serves main since S72. |
||
|
|
9f8242d63c |
fix(progress): the #else half of a NON_MATCHING block is LIVE — REAL was undercounting by 7
classify() consumed everything from '#ifdef NON_MATCHING' through '#endif', swallowing the #else half. But banking replaces the #else INCLUDE_ASM with the real body and leaves the old attempt in the dead half — so every function banked that way landed in NO bucket: not real, not a stub, invisible in both numerator and denominator. Measured: CdReadStateMachine, CdReadSectorReadyCB and StreamLoadStateMachine are byte-identical in the shipped build and counted as zero. REAL 873 -> 880, matchable 1911 -> 1918 (seven functions fleet-wide, not the three I first checked). Now consumes only the DEAD half, then decides from the LIVE half: an INCLUDE_ASM there still buckets as NON_MATCHING (accounting unchanged), anything else rewinds and is classified normally. THIRD coverage defect of this exact shape in this one function — the K&R-definition case (~190k instructions erased) and the '#if 0' case are both documented in its own comments, which is what pointed me at it. A scanner that walks preprocessor structure needs a test per branch, not per directive. Found by the S73 documentation audit, which I had written off as producing only doc typos. |
||
|
|
5ae43099e0 |
docs(phase-31): S73 CLOSE addendum — corrections, and the known-remaining list
Corrects three defects I introduced (deleted §429, a false accusation in §434, an over-strong SaveLoadRoutine verdict), two wrong numbers in the block above, and records the SETUP §6.6 gap that was the real answer to 'are the docs up to date'. Also lists what the audit found and I did NOT fix, with file:line, so a fresh session inherits the list instead of rediscovering it: jr_isolate/jtbl_rodata_pads/draw_waves docstrings, several stale playbook census numbers and its step-1 command, memory-map:309, and the pre-S72 --front/--tail descriptions in the Makefile and ld_interleave. |
||
|
|
aaed146bb2 |
fix(docs): restore §429 which I deleted, and retract a false accusation in §434
TWO REAL DEFECTS I INTRODUCED, both found by the audit:
1. §429 WAS SILENTLY DELETED. My §428a rewrite (commit:3659) wrote t[:start]+new instead of
t[:start]+new+t[end:], truncating everything below §428a. §429 ('every held pointer
needs its own local') was the casualty and had been gone for the rest of the session.
Restored verbatim from commit:3658, between §428a and §430. All of 426-434 now present;
index 1103 sections.
2. §434 ACCUSED AN AGENT OF INVENTING ITS CITATION OF §265. §265 exists and says exactly
what the agent said — 'THE VERBATIM-ASM BANK LANE: A FUNCTION NO -O2 C CAN EVER MATCH
BANKS AS A RAW __asm__ BODY' — with four named byte-banked precedents. I ran
cookbook_index --resolve 265, which resolves a LINE number not a section, and believed
it without opening §265. Retracted in the section itself.
The verdict also needed narrowing: gated, the §265 transcription of SaveLoadRoutine is
BYTE-IDENTICAL for the function itself and fails only because substituting one half of
the shared frame moves 3,989 bytes across 262 symbols. True statement: neither can bank
SEPARATELY; the route is to transcribe/resegment the PAIR together via §265. The
exclude entries now say 'excluded from DRAWS only' and name that route, instead of
reading as 'unmatchable'.
I also mis-read the draft as containing INCLUDE_ASM by grepping raw text — all three hits
were in comments. Sixth instance this session of reading prose as code.
|
||
|
|
0840c0991c |
docs(SETUP): fix four stale spots the tool-inventory update missed
I updated SETUP's tooling INVENTORY when each tool changed, but not the HUMAN-facing
procedure, and §6.6 is where a person learns the matching loop:
* :493 still said 'In src/800.c, replace the INCLUDE_ASM line with the C function body.'
main's game code is THREE TUs since S72, and WHICH one is load-bearing for any switch
function — one code object contributes exactly ONE contiguous .rodata run, so the TU
decides which jump-table span the body's table lands in. Following that line for a
span-B/C function re-creates the exact §426 double-emit this session existed to remove.
Replaced with the vram -> TU -> asm-path -> span table.
* :759 listed main_diff_locate's verdicts as an exhaustive three — BODY / PLUMBING /
MIXED. There are FOUR, and the missing TABLE REJECT is checked FIRST and covers
precisely the case the PLUMBING clause claimed ('byte-identical, everything differs
elsewhere'), routing the reader into the one chain the tool forbids for that class.
* config/wave_exclude.txt was named nowhere in SETUP despite being tracked config that
draw_waves now requires. Added, with both entry classes and the WALL pin.
* :537 described ld_interleave as --front/--tail only; main uses --order since S72.
|
||
|
|
891b240b5d | chore: regenerated burndown | ||
|
|
2c1d21bb9d |
docs(phase-31): S73 CLOSE — R22 213/213, 23 main banked this session, main 56.5%, frontier 124
The jump-table class on main is resolved: 25 -> 2, and both survivors are the §434 frame pair, provably unmatchable as separate C functions (resegmentation, not drafting). Wave S73m_1 banked 9 of 9 drafts (2,413 ins). Cookbook entries written this morning cracked functions this afternoon; two of mine were refuted by later MATCHes and rewritten. |
||
|
|
f06d81a7d0 |
feat(main): StreamLoadStateMachine banked — all 9 wave drafts in; gate_main is preprocessor-aware
StreamLoadStateMachine (459 ins) byte-identical. That completes wave S73m_1: 9 of 9 drafts banked, 2,413 instructions. gate_main defect this exposed, twice in one gate: resolve_conflicts scanned every 'extern' line with no notion of the preprocessor, so a declaration parked in the DEAD half of an '#ifdef NON_MATCHING / #else / #endif' pair read as a live constraint. It is never compiled and constrains nothing. func_80018714 and func_800377D8 each carry a stale '(void)' declaration in a dead branch while the LIVE definition takes a pointer and a u8 respectively; the first mis-blamed a draft, the second got a byte-verified draft DROPPED. live_text() now blanks those branches before the scan. The detour is instructive: I 'fixed' the draft twice to satisfy a constraint that did not exist, and each fix made it worse — the draft's original (u8) declaration was correct all along, because it matched the LIVE definition. Read which branch a declaration lives in before believing it. |
||
|
|
e1631eaacb |
feat(main): CdReadSectorReadyCB (424) + func_80035C4C (248) banked byte-identical
Both needed the §376 recovery in the DRAFT — adopt the TU's spelling for a symbol the
draft also declares:
* CdReadSectorReadyCB dropped its own 'extern void func_800599B8(void *rect, ...)';
the TU declares it (SpadRect_800184F0 *) at src/800.c:5480, above the insertion point.
* func_80035C4C adopted 'extern void func_8003D650(int,int,int)' — no caller anywhere
uses the return value, so the s32-vs-void difference was free to give up.
Also corrects src/800.c's dead-branch 'extern void func_80018714(void);' to '(void *)'.
That declaration lives inside #ifdef NON_MATCHING and is never compiled, but gate_main's
DECL scan has no notion of preprocessor guards and read it as a live conflict. The live
K&R definition at :5576 takes void *, so the correction makes the dead copy agree with
reality as well as clearing the false conflict.
|
||
|
|
35307803f7 |
feat(main): 6 more banked byte-identical from wave S73m_1
Six of the nine drafts, one clean rebuild, 10.5s. Verified from the SOURCE (the INCLUDE_ASM stub is gone), not from the tool's own report. |
||
|
|
383ff02d93 |
fix(main): align 12 forward declarations with their definitions (§376), byte-neutral
Prepares the S73 wave's 9 byte-verified drafts for gating. Five definitions have promotion-safe params so the declaration becomes K&R no-prototype — which also keeps func_8003388C's 'Ent388C *' typedef out of scope at the declaration site, where it is not yet defined. CdReadSectorReadyCB's u8 is NARROW so no-proto is unsafe (§17-stop); it gets the exact prototype, safe because that symbol is only ever passed BY ADDRESS. Verified BYTE-IDENTICAL with no draft substituted, via a DIRECT extract+build with the binary deleted first — NOT via gate_main --assert-baseline, whose first action is 'git checkout -- src/*.c'. I used that first and it silently reverted these very edits, then reported GREEN for a tree that no longer contained them: a verification of the wrong thing. Same hazard as the two banks lost this morning, from the other direction. Six of the twelve were found by checking every draft systematically rather than trusting the agents' notes; two were never reported. |