mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
d564b4b4e718febf18eb97ef985e7224fbc078d4
695 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
d564b4b4e7 |
feat(gate_main): persist every proven verdict the moment it exists
try_batch is stateless and the bisect loop held `good` only in memory, writing .run/gate_main_banked.json once at the very end. A 34-minute bisection killed by a timeout, a Ctrl-C or a supervisor therefore lost every match it had already PROVEN — and each of those proofs cost a full clean EXE rebuild. The S75 checkpoint named this the single highest-value gate improvement available. Adds an atomic .run/gate_main_progress.json written after every verdict, and a resume that reuses it. Three guards, each a way it could silently lie: the journal must belong to this slate; entries are re-keyed against `kept` so a draft dropped by resolve_conflicts cannot sneak back; and the draft's content hash must still match (R56 — a verdict measures those bytes). Resumed sets are re-verified as one batch anyway, so a wrong reuse costs one rebuild and can never bank anything unproven. --no-resume opts out. Negative-controlled on six cases incl. a changed draft, a foreign slate and a half-written journal. |
||
|
Drew T
|
fc7caf599b |
refactor(tools): retire asm_in_c.py — the taxonomy is DATA now, not a regex census
R33, "the best outcome is a DELETED SCANNER, not a fixed regex". asm_in_c.py
existed to DISCOVER the §265 verbatim class by parsing __asm__ blocks. That job
is done, and regex was the wrong instrument: five successive censuses returned
116 -> 112 -> 108 -> 178 -> 199, and the classification was worse than the count
-- it called 154 rows "game code" where the authoritative answer is 24.
The real answers came from evidence a regex cannot see:
* the <OBJ>_OBJ_<hex> naming key -- every one is placed_object.text_start +
hex, so those symbols are OFFSETS INTO LIBRARY OBJECTS, not functions;
* the PsyQ archive symbol tables in .run/obj40/, which keep statics as W
symbols, so for a byte-identical object the archive IS the function map
(checkRECT = SYS.o+0x52C = func_80059760, and NONE of the 44 SYS_OBJ_*
symbols in SYS.o is a function).
So:
config/verbatim_manifest.json (NEW, committed) -- the authoritative census.
200 rows, derived once from the ROM image + archives + naming key, each with a
class and a DISPOSITION:
PERMANENT-VERBATIM 69 rows / 57 units hand asm; never decompilable
DECOMPILE-AS-PARENT 57 rows / 23 units a FRAGMENT; decompile unit_entry,
never the fragment itself
DECOMPILE-NOW 41 rows / 41 units
DECOMPILE-LOW-VALUE 20 rows / 4 units
UNCERTAIN 5 / NOT-VERBATIM 7 / NOT-CODE 1
tools/verbatim_check.py (NEW) -- a GUARD, not a census. Detects verbatim bodies
(the cheap part, and the only part regex is good at), diffs the NAMES against the
manifest, and reports NEW / GONE / MOVED. A NEW row means someone banked assembly
and it is about to become invisible work; it is never allowed to inherit a
disposition by default. It deliberately does not classify or count units.
Compares case-insensitively on the hex, because an address is a NUMBER (R48).
tools/verbatim_target_s.py -- put on the MANIFEST LEASH. It used to enumerate
every verbatim SYMBOL, and 62 of those are not functions (fragments, bare
epilogue tails, padding, trampolines). Emitting per-symbol targets for them is
what sent two drafting bursts at things no C function can express. It now takes
only DRAFTABLE dispositions: 66 targets emitted, 134 skipped and SAID SO.
tools/verbatim_to_stub.py -- repointed to verbatim_check for detection, so there
is ONE detector in the tree rather than three copies.
tools/asm_in_c.py -- REMOVED.
|
||
|
Drew T
|
254feb8ee4 |
fix(gate_main): the uncommitted-work guard belonged OUTSIDE the bisection loop
I added the guard to try_batch() an hour ago. try_batch runs REPEATEDLY during
bisection, and its own first substitution makes main's TUs dirty -- so on
iteration two the guard could not tell the operator's unsaved work from the
gate's own in-flight edit, and aborted the run:
M src/800c3.c
gate_main: aborting with an UNVERIFIED substitution in main's TUs — reverting
It failed safely (reverted, no bank lost, and said so), but it made the gate
unusable for any batch larger than one.
Hoisted to assert_main_tus_clean(), called ONCE from main() before any
substitution. The lesson is worth the line it costs: A GUARD MUST BE ABLE TO
DISTINGUISH THE STATE IT PROTECTS FROM THE STATE IT CREATES. Placed inside the
loop it was checking its own footprints.
Negative-controlled both directions: a genuinely dirty src/800c3.c is refused by
name before anything is substituted, and a clean tree now proceeds into the
bisection (currently running 21 drafts).
|
||
|
Drew T
|
590fb37447 |
fix(gate_main): refuse to destroy uncommitted main work; name a tool refusal instead of hiding it
TWO DEFECTS, both found by the SaveLoadRoutine decompile and both of which made this gate unable to bank a whole class of function. 1. try_batch() opens with `git checkout -- <main TUs>`. Correct for the normal flow (restore stubs, re-extract, substitute drafts) and CATASTROPHIC for anything uncommitted. Measured twice today: the SaveLoadRoutine decompile (1,179 ins, byte-identical) sat uncommitted while a gate ran and survived only because it was committed first; and a §265 verbatim body converted to a stub is UNCOMMITTED BY CONSTRUCTION, so this line restored the __asm__ block NEXT TO the substituted C -- 9 jump tables instead of 5, jtbl_rodata_pads refused, and the gate REJECTED a byte-identical bank. gate_main could not bank anything in the verbatim class, by construction. Now refuses when main's TUs are dirty, printing the offending paths and telling the operator to commit (R42) or stash. --allow-dirty / GATE_MAIN_ALLOW_DIRTY=1 is the deliberate override. A destructive step that cannot be undone must ASK, not assume. 2. The failure analysis looks for error/undefined/conflict/..., and jtbl_rodata_pads aborts via sys.exit with a message containing none of them -- so a carve REFUSAL surfaced as "only warnings" and the real cause of a rejected bank was invisible. Refusals from jtbl_rodata_pads / jtbl_carve / corpus / jr_isolate_all are now named explicitly as the cause. Negative-controlled both directions: the guard fires on a dirty src/800_b.c naming the file, and --assert-baseline on a clean tree still reports BASELINE GREEN 143dbb89... BYTE-IDENTICAL. |
||
|
Drew T
|
8009f83b40 |
fix(tools): verbatim_target_s wrote targets into asm/, which the Makefile globs as build objects
`build/asm/%.o: asm/%.s` globs the ENTIRE asm/ tree, so the 146 regenerated
target .s files I emitted to asm/verbatim/ were picked up as BUILD OBJECTS and
main went red:
make: *** [Makefile:696: build/asm/verbatim/main/func_80052430.o] Error 1
Default --out moved to .run/verbatim_targets/, which is outside every build
glob. main verified green again: 143dbb89... BYTE-IDENTICAL.
The lesson belongs with the others from this session: a generator's OUTPUT
LOCATION is part of its contract, and asm/ is not a scratch directory. The
failure was invisible until a full build ran -- the tool itself succeeded, the
targets were correct, and nothing about them was wrong except where they sat.
|
||
|
Drew T
|
72fa482027 |
feat(tools): verbatim_to_stub.py — put the 147 asm-posing-as-C functions back where the gates can reach them
THE GAP. Every gate in this project substitutes a draft in place of an
INCLUDE_ASM line. A §265 verbatim __asm__ body has none, so:
gate_main.substitute() resolves each entry through the STUB map; a verbatim
function is reported "resolved to NO stub" and dropped
gate_stage/harvest_verify same splice, same gap
splat stops emitting <fn>.s once a function is not a stub
So all 147 were undecompilable AND ungateable -- not for want of information,
but because the information was in a form nothing consumes.
The fix is not a parallel gate (R33 -- one implementation). It is to put the
function back into the form every existing tool already understands: replace the
__asm__ block with INCLUDE_ASM. That is also the HONEST representation --
INCLUDE_ASM pastes the very same assembly the block transcribes, so the bytes
are identical either way, but a stub counts as OUTSTANDING WORK in progress.py
while a verbatim body counted as banked. The conversion moves a function from
"silently done" to "visibly to do".
Two refusals rather than guesses (R43), because both failure modes are silent
and destructive:
* the block is located by brace/paren MATCHING via asm_in_c.asm_blocks, never
regex-sliced -- these blocks are full of braces and parens inside string
literals and an approximate cut corrupts a file that currently builds;
* the asm subdir for the new INCLUDE_ASM is copied from a sibling stub IN THE
SAME FILE. Subsegs are per-file, so a neighbouring file's spelling names a
different subseg -- a stub with the wrong subdir compiles happily and
includes ANOTHER FUNCTION'S ASSEMBLY. With no sibling to copy, it refuses.
--gate rebuilds and asserts the SHA is unchanged, restoring the file if not:
byte-neutrality here is a claim, and this tool exists to enable a byte gate, so
it declines to be the one link that goes unchecked.
Dry-run verified on main:func_80047E58 -> src/800b.c, 9-line block, subdir
correctly derived as asm/nonmatchings/800b. The --gate proof is deferred only
because another agent is mid-build on main right now.
|
||
|
Drew T
|
70de5a8611 |
feat(tools): verbatim_target_s.py — the 147 asm-posing-as-C functions are workable again; bank func_8017DB98
THE BLOCKER. asm_in_c.py found 147 GAME functions that are §265 verbatim
__asm__ bodies. NONE of them could be worked on: splat emits
asm/nonmatchings/<subseg>/<fn>.s only for functions that are still INCLUDE_ASM
stubs, and a verbatim body is not a stub -- so splat stops emitting its .s,
while match_one and rtu_match BOTH consume one. Measured: 1 of 147 had a target
on disk. The class was unworkable because the information was in the wrong FORM,
not because it was missing.
verbatim_target_s.py regenerates a splat-format target .s from the EXTRACTED ROM
IMAGE -- never from the __asm__ block in our own source, because the block is
the thing under test and a target derived from it would agree with the candidate
by construction (R34). 146 of 147 emitted; the 1 refusal is REPORTED.
TWO DEFECTS CAUGHT BY CHECKING AGAINST A KNOWN-TRUE CASE, both of which would
have shipped ~147 silently-wrong targets:
* BYTE ORDER. splat writes the four bytes as they sit in the image
(`C8FFBD27` for instruction 0x27BDFFC8) and masked_diff.insns_from_s reads
the column with struct.unpack("<I", bytes.fromhex(...)). objdump prints the
VALUE, so reversing double-swaps: 91 of 1139 words agreed with splat's own
.s for the same function. The LENGTH matched perfectly, so nothing except a
word-level cross-check could have caught it.
* `-z` / --disassemble-zeroes. objdump ELIDES runs of zero bytes as `...`, and
a MIPS nop IS 0x00000000 -- so every nop vanished. func_80049610 (three
nops) disassembled to ZERO instructions; func_80047D3C 31 of 36. The length
assertion caught all of them, which is the only reason this was not shipped
as ~30 quietly-truncated targets.
Verification: regenerated SaveLoadRoutine target is 1139/1139 words IDENTICAL to
the .s splat itself emitted for the same function.
ALSO BANKED: ov_SC06_025:func_8017DB98 (122 ins). Its body was byte-exact on
disk since S71 and the blocker was one word: the TU declared
`extern void func_8017DB98(s32, s32)` where the epilogue is `addu $v0,$s3,$zero`
-- must be `extern s32`, and the caller discards the result so the change is
byte-neutral. That line number and fix were recorded in the agent journals the
whole time; frontier_classify only surfaced it once journal_notes was wired in
as a second oracle earlier this session.
|
||
|
Drew T
|
f5f4c2eeec |
feat(decomp): bank func_801806F8 + func_80180ABC (498 ins) + frontier_classify reads the journals
Two banks from the S75 redraft workflow (7 overlay functions, one agent each,
every claimed MATCH re-verified by an independent agent instructed to refute
it). Both were carried as F-FAR "a draft exists but is materially wrong":
func_801806F8 ov_SC03_105 241 ins (recorded closeness 235)
func_80180ABC ov_SC03_105 257 ins (recorded closeness 250)
Neither needed a better model. Both needed the recorded closeness not to be
believed -- see below.
frontier_classify.py, THREE fixes, each caught by testing against a case whose
answer was already known:
1. BEST closeness, not LAST. .run/backlog.jsonl is append-only, one row per
attempt across every lane and session, so the last row is evidence about
THAT LANE'S SEED, not about the function. Caught func_80180B3C (best 125,
last 287) and moved func_80181294 from "redraft" to "permuter" (best 19).
The draft that ACHIEVED the best score is kept, not the last one written.
2. journal_notes.py wired in as a SECOND, DISAGREEING oracle (R34). The backlog
does not have what the agent journals have. Measured on func_8017DB98:
backlog best == last == 115, so best-vs-last could not help, while the
journal holds "Attempt 2 (MATCH · closeness 0) ... MATCH 122/122 ... BANK
BLOCKER is TU plumbing, not the body (§376/§378)" WITH the draft path and the
exact declaration to change. Reclassified 37 functions; G-DRAFTED-UNKNOWN
fell 47 -> 10 and a new C-PLUMBING class holds 16 functions / 1,547 ins whose
BODIES ARE PROVEN and are blocked only by the TU.
3. A consuming-regex bug in my own extractor -- the session's signature defect,
committed a third time in the tool written to find it. The first cut used
`re.finditer(r'\*\*Attempt \d+\*\* \(([^)]*)\)(.{0,400})', ..., re.S)`, whose
400-char body window SWALLOWS THE NEXT ATTEMPT'S HEADER, so every record
following another was invisible. On func_8017DB98 it hid attempts 2 AND 6,
both `MATCH · closeness 0`, and returned attempt 1's NEAR (2) as the best --
exactly the records the oracle exists to find. Now splits on the marker
rather than consuming past it. A regex that consumes an unbounded body cannot
enumerate the items after the first.
Rows now carry attempts, closeness_last, journal_closeness, and a
!!WARMSTART-REGRESSION flag when a later attempt scored materially worse than
the best -- the shape a wave's warm-start regression makes, which from inside
the wave is indistinguishable from an unsolved function.
Gate ledger for the batch of 7: 2 banked, 3 near, 2 failed. func_800CB00C failed
despite being adversarially upheld -- it owns a jump table, and both matchers
compare .text only, so a verified .text MATCH proves nothing about table
placement (the agent's own write-up says so).
|
||
|
Drew T
|
c05ea15cbe |
feat(tools): asm_in_c.py — 154 game functions are assembly wearing a .c extension
A .c file in src/ looks decompiled. 199 functions are not: they are the target
assembly pasted into a C string literal (§265), byte-identical BY CONSTRUCTION
and completely unexplained. 45 are PsyQ/CRT routines where that is defensible;
154 are GAME CODE, 171 of the 199 in main, the largest being SaveLoadRoutine at
1,165 instructions.
They were invisible because progress.py's classify() matched INCLUDE_ASM,
INCLUDE_RODATA and C definitions, and a file-scope __asm__ block is none of
those -- so each landed in NO bucket, either swallowed by a surrounding
construct or surfacing as the single `UNPLACED (parse hole)` line the tool has
been printing all along.
progress.py gains a VERBATIM __asm__ bodies line: counted byte-identical (it is,
by construction) but NEVER as REAL. main's headline moves 45.88% -> 42.15%.
Nothing regressed and no work was lost -- the denominator was missing 173
functions that are real remaining work.
THE COUNTING LESSON IS THE REUSABLE PART. Counting these by hand went
116 -> 112 -> 108 -> 178 -> 199 across five attempts in one session, every
intermediate number reported confidently. All five errors were one shape, a
pattern narrower than the claim it supported:
* the sources use BOTH ".ent\tNAME\n" and ".ent NAME\n" -- anchoring on either
silently drops every instance of the other;
* a bare ".ent\t" fragment yields a phantom function literally named `t`, six
times, which is the only reason the error was noticed;
* __asm__ appears in 3,182 of 4,224 sources, almost all the §3a barrier, so
counting files or counting __asm__ measures nothing;
* `.globl NAME` + `NAME:` proves EXPORT, not CODE -- the first real run
reported jtbl_80072ED4/EEC/F0C/F24 as four "functions";
* a hand-written SDK name list reported 170 game functions because it did not
know VectorNormalSS / SquareRoot12 / OuterProduct12 are libgte.
So the tool does not trust one regex: THREE independent detectors that must
agree with disagreement reported as a defect (R34 -- that is what caught the
jump tables); SDK-ness DERIVED from the 14 shipped PsyQ archives via nm (2,227
symbols) rather than a list (R33); coverage asserted so a definition-shaped
block no detector claims fails loudly (R32/R43); and --selftest carrying a
known-true case of every spelling plus the phantom `t` and the jtbl regression.
Cookbook §448, SETUP row. Law: when a count comes from a text pattern, the
pattern has a denominator too -- validate it against one known-true case of
every FORM the corpus contains before quoting the number.
|
||
|
Drew T
|
1bac13b664 |
fix(jtbl): the pad walk cannot see a verbatim-asm rodata block — SaveLoadRoutine banks (bytes, not a decompile)
tools/jtbl_rodata_pads.py --derive walks a TU's rodata emission against the
retail island and validates only what it can SEE. A §265 verbatim-__asm__ body
emits its tables as `.section<TAB>.rodata` + `jtbl_xxxxxxxx:`, and the walk
missed BOTH spellings:
* the rodata directive was matched as the literal one-space string
".section .rodata" / ".rdata", so a tab-spelled directive never entered
rodata at all;
* inside rodata the anchor regex accepted only `D_xxxxxxxx` (the S74 dlabel
fix was one prefix short), so a `jtbl_xxxxxxxx:` label was invisible.
Consequence, traced: the walk skipped the block as if it were .text, every
later C table walked 104 bytes behind its retail address, EVERY WORD in that
range happens to be a valid code address so the entry guard never fired, and
the walk stopped short of the island's single zero word -- so the one trailing
pad was never emitted and the image linked 4 BYTES SHORT. That produced 3,989
differing bytes on a body the verdict layer had already called byte-identical.
Fixed: tokenised directive match (.rdata / .section .rodata, tabs and commas);
anchors keyed on the ADDRESS IN THE NAME for `D_` or `jtbl_`, with an
address-suffixed label of any other prefix now REFUSING loudly (R43) instead of
becoming a silent hole; and `.align N` modelled SECTION-RELATIVE from the walk
origin, as `as` does -- needed for `.align 3` when a section starts = 4 mod 8,
which span B (0x80072E44) does.
Negative control: old vs new derive over ALL 162 md_*/main derive-path TUs ->
160 byte-identical post-derive streams with identical exit codes, 0 DIFF; 2 SKIP
(800c2/800c3 are REORDER TUs with no derive stage).
main SHA1 143dbb89... BYTE-IDENTICAL, 413,696 bytes, cmp identical to retail.
WHAT THIS IS NOT. SaveLoadRoutine is banked as a §265 verbatim __asm__ block --
BYTES, NOT A DECOMPILE. Its 1,165 instructions are byte-correct and unexplained.
tools/progress.py correctly REFUSES to count it, reporting `UNPLACED (parse
hole)` rather than inflating REAL (which moved 880 -> 881 on func_8005DCA0
alone). Two such blocks already exist in this TU, documented as necessary
because those functions have no epilogue and fall into shared tails. A real C
decompile is now being attempted separately; this commit is the revertible
byte-green base for it.
|
||
|
Drew T
|
cf7f837231 |
fix(gate): main's TABLE REJECT verdict was unreachable — SaveLoadRoutine is a CARVE, not plumbing
SaveLoadRoutine (1,165 ins) is the largest open function in the project, 9.2%
of all remaining work, and has been carried as the §434 WALL. Gated alone
through gate_main, with the §376/§378 chain already applied, the verdict layer
says: "SaveLoadRoutine is BYTE-IDENTICAL; all 3989 differing bytes are
ELSEWHERE". The body has been correct the whole time.
What rejects it is where its FOUR jump tables (jtbl_80072ED4/EEC/F0C/F24) land:
.data/.rodata (jump tables) 3,787 bytes 94.9%
.text (perturbed code) 202 bytes 5.1%
and the built image is 4 bytes SHORTER than retail (413,692 vs 413,696) --
§446's first diagnostic, firing on a function §446 was not written about.
main_diff_locate.classify() already HAD a TABLE REJECT class, added in S72 under
a docstring reading "THE THIRD CLASS EXISTS BECAUSE THE FIRST TWO MISLABELLED
IT". It could not fire here for two independent reasons:
* it keyed on the literal string `(.rodata)`, but main's section_order is
[.rodata, .text, .data, .bss] -- its rodata sits BELOW .text and its jump
tables live in `.data` objects, so TABLE REJECT was UNREACHABLE BY
CONSTRUCTION on the binary with the most jump-table functions left. A
section NAME is not a section ROLE.
* it demanded purity (ro == outside), so 5% perturbed code defeated an
all-or-nothing test and dropped the verdict through to PLUMBING REJECT --
whose advice (fix_arity_callers -> cast_self_callers) addresses the 5% and
cannot touch the 95% that is data. That chain was run on this function
TWICE today and fixed nothing, exactly as the evidence predicts.
Now: table bytes counted in (.data) OR (.rodata), and the test is DOMINANCE
(>=60%) rather than purity, reporting the split and naming which part is the
carve problem and which the declaration problem.
Negative control over all five pre-existing verdict shapes (pure BODY, pure
PLUMBING, pure TABLE, MIXED, NOT FOUND) plus the S75 shape: 5 of 6 verdicts
UNCHANGED, only the SaveLoadRoutine shape flips PLUMBING REJECT -> TABLE
REJECT (MIXED).
Cookbook §447. The law: a class that cannot fire is worse than a class that does
not exist -- it converts "I don't know" into confident, specific, wrong advice.
When a verdict names a subsystem, check that subsystem owns the MAJORITY OF THE
BYTES before acting on it.
|
||
|
Drew T
|
abea9f0ac2 |
feat(decomp): bank func_8016AE5C (85 ins) + frontier_classify — the frontier is not a drafting problem
func_8016AE5C (ov_SC03_108) was logged "match_one MATCH but the whole-binary
gate rejected -- CAUSE NOT DETERMINED". Determined: the body is byte-perfect (0
differing words inside the function; all 1,168 diffs are uniform +0x20 shifts
outside it) and it emits an 8-entry jump table that was never carved. It banked
unchanged the moment the §446 jtbl_carve per-table bound landed.
tools/frontier_classify.py (NEW) — classify every open stub by its TRUE BLOCKER
from artifacts already on disk (R33/offline-tooling-first: zero tokens, no
agents, no builds). "69 functions left" is a stub count, not a difficulty
measure, and routing drafting agents at carve or plumbing problems wastes them.
A-TWIN-REMAP 3 302 a byte-identical copy is already banked elsewhere
B-CARVE 11 3,301 owns a switch jump table -> the §446 class
D-NEAR 2 106 closeness <=25 -> permuter fuel, not drafting
F-FAR 3 223 draft materially wrong -> redraft
G-DRAFTED-UNK 49 8,724 drafted before, no usable verdict on record
H-VIRGIN 1 1 never drafted (and it is a DATA BLOB, not a function)
68 of 69 remaining functions already have a draft on disk. The endgame is a
verification/integration problem, not a drafting one.
TWO SELF-INFLICTED DEFECTS FOUND BY CHECKING AGAINST KNOWN-TRUE CASES, both the
session's recurring shape (a scan narrower than the claim it supports, R32):
* The sig directory is NOT the fleet. Alongside the 213 real binaries `.run/`
holds `SLUS_007.26` (a STALE duplicate of main under the ROM filename),
`resident_image`, and two CROSS-BUILD binaries (`sep8_SLUS_007.26`,
`aug31_USA_DEMO.EXE`). Counting them as peers reported 38 fns / 7,516 ins of
free twin-remaps -- mostly main "already banked" in ITSELF, the rest proven
in a PROTOTYPE that R13 forbids as evidence. Now derives the fleet from the
Makefile and prints what it ignored. True figure: 3 fns / 302 ins.
* The draft scan globbed `.run/S7*` only, missing `.run/S69m2`, `.run/S68m1`,
`.run/s67m1`, `.run/wave_ds2`, `.run/gate_lane`, `.run/backlog_drafts`. All
32 drafted main functions read as "never drafted", which would have sent
agents to redraft 6,328 instructions that already have drafts. Now one
pruned os.walk of .run (worktrees excluded -- 7.4 GB of duplicate sources).
Honest negative result: resident:func_800D06E8 (344 ins) did NOT bank. I
predicted the carve fix would clear it; it did not. Its blocker is still open.
|
||
|
Drew T
|
cb948a6bbc |
feat(decomp): the ov_SC01 reloc-only cluster + its 5th latent victim — 5 fns, 1,301 ins
S74 handed this forward as "1,116 instructions behind one question": family_remap
on ov_SC01_004/005/006/008 gated DIFF 4/4 against the banked exemplar
ov_SC01_009:func_8017EB08, and the class had been carried as a codegen wall since
S70. The four bodies were byte-identical to the exemplar the entire time.
Word-level classification vs the exemplar, computed independently twice (a Fable
agent's script, then mine from scratch against the retail images), identical:
nins=279 EQ 213 · RELOC-HI16 23 · RELOC-LO16 24 · INTERNAL-J 19 · CODEGEN 0
Zero register-allocation, instruction-selection or scheduling differences.
ROOT CAUSE — tools/jtbl_carve.py reserved ONE WORD TOO MANY per table:
* spimdisasm runs an island's LAST `jtbl_` dlabel one word into the following
NON-ZERO data (string bytes 0x696F760A / 0x000013FF / 0x62647020), so the
zero-word trim cannot see it; and
* the over-span clamp that would have caught it was guarded by
`len(sltiu_bounds) == 1` -- but `sltiu` is ALSO how gcc emits an unsigned
range check ((u32)(x-lo) < n, I1). These four carry five distinct sltiu
immediates, so the guard silently disabled itself on precisely the functions
that needed it.
0x2C reserved for a 0x28 table => image 4 bytes short => ~850 %lo immediates
shift => whole-binary DIFF about a function whose own bytes are perfect.
Fixed with a PER-TABLE bound: gcc-2.7.2's dispatch is a fixed idiom, so the
`sltiu` nearest ABOVE that table's own %hi(jtbl_X) is unambiguous whatever else
the function tests. Second defect stacked behind it: a carve span whose
JTBL_PADS line lacks a `tables=` comment lost its existing table's start on
merge and refused "table starts do not fit the span" -- which harvest_verify
then "repaired" with a needless jr_isolate_all that walked back into the first.
THE NEGATIVE CONTROL IS THE STORY. Run over every other open table-bearing stub
fleet-wide, the fixed bound changed exactly one more table: ov_SC06_022/
func_80185B80 (185 ins), a FIFTH victim nobody had drafted against. A guard that
disables itself on a common idiom does not fail once -- it fails quietly across
the whole corpus.
Banked, each with its own byte-gate verdict (--no-propagate, clean re-gate):
func_8017EB30 ov_SC01_004 279
func_8017F2D4 ov_SC01_005 279
func_8017F2D4 ov_SC01_006 279
func_8017EC68 ov_SC01_008 279
func_80185B80 ov_SC06_022 185
Also here:
* dedup_propagate: memoize find_site's mask (lru_cache) -- 54 ms of masking
per call over the whole source, recomputed though it depends only on the
text. 2x on that loop (58.3 -> 33.0 ms/call), NC identical on 120 addrs.
Scoped honestly: that loop is ~2.4 min of a 30-min run; the profiler puts
43% in family_remap._alias_decl_for, which is NOT fixed here.
* Makefile: `clean` says out loud that BINARY= is ignored and it is fleet-wide
(cookbook §445) -- it silently deleted asm/ for all 213 binaries this session.
* Cookbook §446 (the carve law: when a standalone-MATCH jtbl draft gates DIFF,
diff the carve extent against 4 x sltiu before touching the body), §445, and
SETUP rows for both tools (R21).
* CURRENT_PHASE: the S75 log, incl. the measured fleet dedup-hygiene census
(~2,073 fns / ~12,116 items, all ALREADY MATCHED -- cleanup, not work) and
Drew's decision to leave it and gate --no-propagate from here.
|
||
|
Drew T
|
0c2b37a287 |
fix(gate): overlays.mk carve-state snapshot was singular, so the revert half-restored
Found by running one reject to ground. After a gate that REJECTED
resident:func_800D06E8, config/overlays.mk had a 4th JTBL_PADS entry and had
LOST `--pre hdr.rodata.o` (the §440 resident leading-rodata sandwich). The
binary then would not build at all -- "consumed 3 rodata jump table(s) but 4 pad
spec(s) given -- table-count drift vs the carve" -- while src/ was perfectly
clean, which is the only place anyone looks before building.
Root cause is a silent narrowing in the classic shape. harvest_verify snapshots
ONLY the gating binary's own overlays.mk block on purpose (the file is shared by
every parallel gate; a whole-file restore resurrects other binaries' lines --
the S62 defect). But _mk_block_span was SINGULAR: the first `# --- <binary>`
header through the next `# --- `. A binary whose carve state spans more than one
block was half-snapshotted and silently half-restored. It returned a TRUE span
for a scope smaller than the caller believed, and nothing compared the two (R32).
Blast radius measured before costing (R37): 1 of the 142 binaries that have a
block -- resident, which has exactly two (§8e pad spec, §8f leading-rodata
sandwich) and still holds 587 instructions of open stubs.
_mk_block_spans (plural) snapshots a LIST, restores tail-first so earlier spans
stay valid, collapses to the snapshot when the header count changed rather than
leaving half-state, and RE-READS and compares the result -- the defect it
replaces was a reported success. _mk_block returns None (not []) for the 71
binaries with no block, so the caller's guard keeps its meaning.
Negative control, three ways:
* snapshot -> restore is a NO-OP on 142/142 binaries with a block;
* the real S75 damage is fully undone;
* the OLD single-block restore provably does NOT undo it -- the positive
control that proves the fix is load-bearing, not decorative.
Cookbook §444 also records the two other findings from the same reject: the
classified ledger stores the LADDER'S FINAL verdict (the recorded CC1-FAIL came
from a late sig_unify rung; the raw draft compiles and fails on BYTES), and
match_one MATCH + rtu_match MATCH is still not bankable -- func_800D06E8's real
blocker is a jump table (built binary 20 bytes longer, 0x800CEDFC holds a table,
69,571 words shift), because neither matcher LINKS.
|
||
|
Drew T
|
cdd535e45b |
fix(tools): reconcile_tu's cc1 premise, the &-cast arms, and the worktree sig gap
The S74 checkpoint's "one unfixed defect that is actively costing banks"
(reconcile_tu manufacturing declaration conflicts), run to ground — plus the
harness gap that produced a false carve-corruption verdict.
reconcile_tu.py — three defects, measured against the real gcc-2.7.2 front end
(cdecl._cc1_accepts, the oracle cdecl.compatible was validated with; R33):
* The premise "a decl BELOW still conflicts" is TRUE at file scope and FALSE
at block scope. cc1 ACCEPTS a block-scope extern against a TU decl below it
(pedwarn "type mismatch with previous external decl"); conforming it is
destructive, because the TU's decl names the TU's TYPE and a type declared
below the splice point is not in scope AT it -- the emitted result gets
"syntax error before 'D_x'". Byte-witnessed on resident:func_800D06E8 (344
ins), whose block-scoped `extern Blk80078E78` became `extern
Struct80078E78`, typedef 388 lines lower. That construct is what this
ladder's OWN scope_demote_drafts (§8d) rung emits on purpose, and three
already-banked functions in that TU use it: one rung undoing another.
* The cast pass rewrote COMMENT PROSE -- 8 rewrites inside one header comment,
including inside a quoted cc1 diagnostic. Now matches on cdecl._mask
(length-preserving, so a mask offset is a source offset) and splices into
the original.
* `&sym` emitted `&` applied to a cast: legal for the scalar arm, `invalid
lvalue in unary '&'` (measured) for the array/fnptr/fnptr_array arms. `&`
now selects a pointer form and consumes itself -- but ONLY with no trailing
subscript, because `&sym[i]` is the address of ELEMENT i and the old code
had that case right. That last clause exists because the R39 negative
control caught the fold as a regression in the first cut of this fix.
gate_stage.py — `--skip-stages` / `GATE_SKIP_STAGES` (loud when used). Stage 0
gates raw drafts first, so a broken rung can only cost a RECOVERY, which is
exactly what makes it invisible: the function it destroys was already failing,
so its DIFF reads as a fact about the function.
verify_worktree.py / jr_isolate_all.py / parallel_gate.py — provision() now
symlinks every .run/sig.*.jsonl (main clone 259, provisioned worktree 0), the
third member of the class holding extracted/ and .run/obj40. parallel_gate was
fixed for this identical bug in S69: two provisioners, no shared list, found
twice; they now cross-reference each other. jr_isolate_all no longer swallows
the resulting FileNotFoundError into `except: continue` -- that turned a missing
index into a confident carve-CORRUPTION verdict over 2,603 of 2,603 functions
(R54). Adds _assert_scan_covered: attempted == raised means the scan measured
nothing, so its zero is an artifact, not a finding (R32).
Verification:
* 4 cc1 probes (the table above), each run on the pinned front end.
* R39 negative control over the stored-draft corpus: 661 adjudicated, 652
IDENTICAL, 9 CHANGED and every one an intended class. 4,173 of 4,864 drafts
unadjudicable (filenames that are not func_<ADDR>) -- stated, not hidden.
* jr_isolate_all ov_SC03_105 --dry-run: unchanged in the main tree.
* make clean/extract/build BINARY=resident -> 8e17e02f... BYTE-IDENTICAL.
Docs ship with the change (R21): cookbook §442/§443, index regenerated (1,112
sections), 3 docs/SETUP.md rows, CURRENT_PHASE S75 log.
|
||
|
Drew T
|
8edb918480 |
feat(cards): size-filter the same-address lead (§238 homonym) + bank the S74 lever set
THE CARD USED TO HAND AGENTS A WRONG TWIN ABOUT ONCE IN FIVE. `⭐ func X IS BANKED AT THIS ADDRESS`
never checked that the two functions were the same SIZE, and overlays share addresses between
unrelated functions as readily as they share code. Measured over this session's ~60 cards: about a
dozen agents reported discarding the lead themselves, and one card advertised a 72-instruction
namesake — with journal history claiming "already MATCH closeness 0" — to a 241-instruction target.
A confidently wrong lead costs more than no lead, because the agent believes it.
corpus.sig already carries `nins` and `h_seq`, so the fix is free: `_same_addr_banked` now returns
(binary, nins, h_seq); the card keeps a lead only at a MATCHING instruction count, marks it strong
when the mnemonic skeleton matches too, and prints an explicit `⚠ IGNORE` naming the binaries where
that address holds something else, with both sizes.
VERIFIED IN BOTH DIRECTIONS against known-true cases before being believed (never trust a filter you
have not tried to fool):
* the trap: ov_SC03_105:func_801806F8 (241) vs ov_SC03_013 (72) -> `⚠ IGNORE`.
* the positive: ov_SC02_003:func_80187B40 (158) -> strong lead to ov_SC02_000 (158, same h_seq,
banked this session) AND, in the same card, warned off ov_SC04_011's 138-ins homonym at that
same address. That is precisely the pair a wave agent sorted out by hand hours earlier.
Cookbook §438 (the law: a lead is fuel only if it carries the cheapest fact that can refute it —
size refutes a homonym for free and nobody had asked) and §439, the S74 lever set: MEM_IN_STRUCT_P
as a two-way alias-oracle dial (four agents converged on it independently); `goto`-into-a-shared-tail
vs longhand as a REGALLOC dial because gcc-2.7.2 cross-jumps after allocation; `for` -> do/while as a
length-changing scheduling dial; allocno PRIORITY via a non-volatile asm at a loop head, with the
measurement that register pins are actively harmful for that class; the -O0 global-RMW rule
(`x++` emits the copy-back quartet, `x = x+1` does not); why `sll 16; srl 16` survives only across a
CALL; `sltiu N` without `addiu -1` proving an empty `case 0` is mandatory; block-scoped temps in
duplicated bodies; two `register asm` vars cannot share a hard reg; and `x*32` vs `x<<5` emitting
lh vs lhu — which match_one's %lo mask HIDES, so it must be checked with objdump.
|
||
|
Drew T
|
6e840730a9 |
feat(carve): bank 4 more via the carve chain — and §8b's "non-adjacent => ISOLATE" is over-strict
resident:func_800D00E4/func_800D02D0/func_800D0488 + ov_SC07_002:func_80180248, all byte-verified
from clean rebuilds (resident 8e17e02f, ov_SC07_002 fad71342) and counted from the SOURCE.
ov_SC06_029's two are re-gated separately against HEAD — this agent's worktree predated five banks
there, so its numbers for that binary no longer apply.
TWO OF THE SIX NEEDED NO CARVE WORK AT ALL, AND CARVE-REFUSED WAS AN INSTRUMENT VERDICT.
ov_SC07_002:func_80180248's table is ALREADY inside a carve bound to its own subseg: in stub state
spimdisasm migrates the table into the fn's .s and the object fills the piece exactly, so banking
just swaps that block for cc1's identical one. `island_probe` classified it `tail` on the table's
ADDRESS, `apply()` routed it to build_carve, which resolves spans out of the RAW data asm where a
carved table no longer is -> "not found in the raw data asm" -> harvest_verify booked CARVE-REFUSED.
A verdict about the route we chose, not about the function (R43). jtbl_carve now has a `covered`
verdict (table inside an existing carve bound to the fn's OWN subseg) and a `covered-tpad` wall (the
retail copy carries a trailing §8a pad the matched body won't emit — bankable, needs a `0t<n>`
entry); a fully-covered batch is a no-op before either route.
THE RESIDENT CAN CARVE LIKE AN OVERLAY. Its three tables are adjacent and lead the island
(0x450e0..0x451ac, one span, all in subseg `resident`). The genuinely new part: the resident opens
with `- [0x0, rodata, hdr]`, a 1-word .rodata header BEFORE the code, so its layout is
rodata -> text -> data -> rodata(carve) -> data, which `ld_interleave --order` cannot express (every
listed piece lands after TEXT_START, and hdr.rodata.o would fall into the unchecked `empties` bucket
and be parked after the text, moving every byte). New `--pre` places a leading-rodata piece ahead of
the text; resident_JTBL_INTERLEAVE uses it.
NEW LAW, BYTE-PROVEN (§8b was over-strict — EXTEND the carve, do not isolate): a .rodata carve piece
binds to a code SUBSEG, not a function, and the object's .rodata is the address-ordered
concatenation of cc1's tables for BANKED functions and still-stubbed functions' MIGRATED tables. So
a span may legitimately hold a MIX, and extending a carve across an align-pad word and two unrelated
STILL-STUBBED tables was byte-identical with nothing banked — where the tooling demanded a
jr-isolation. Corollaries, all measured: migrated tables self-align (spimdisasm emits `.align 3` iff
the table's SPAN-RELATIVE offset is 8-aligned), so stubbed tables need no spec; JTBL_PADS counts cc1
tables only, so a mixed span's spec GROWS as each sibling banks; and the zero-word rule is INVALID
across a migrated boundary, because that zero is supplied by the preceding migrated block.
ALSO REPORTED, NOT FIXED (harness gap worth its own change): verify_worktree.provision omits
`.run/sig.<bin>.jsonl` — main clone 259 files, provisioned worktree 0 — and jr_isolate_all's
carve-ownership scan swallows the resulting FileNotFoundError in a bare `except: continue`. Measured:
2603 of 2603 functions raised, the scan found 0 owners, and the run aborted with a CONFIDENT FALSE
verdict ("committed .rodata carve ownership is not 1:1 — stranded/duplicated carve"). Both resolve
instantly once the sigs are present. Any worktree-run isolation before that is fixed reports a
corruption that is not there.
|
||
|
Drew T
|
089311e74d |
feat(md_SC07_004): 10/10 banked — md_SC07_004 is now ZERO stubs, and the "decl conflicts" were fake
Clean rebuild BYTE-IDENTICAL 87ac0de3; corpus.stubs('md_SC07_004') 10 -> 0, counted from the SOURCE.
THE §376/§378 CHAIN WAS NEVER NEEDED. Zero declaration edits: no fix_arity_callers, no
cast_self_callers, no --any-proto, no --sync-decls, no undo journal. `git diff -U0` on the TU removes
exactly the 10 INCLUDE_ASM stubs plus one hoisted typedef. Every recorded "declaration conflict" was
an INSTRUMENT defect. Four of them, all named, three patched here:
1. `CC1-FAIL(no-diagnostic)` was neither cc1 nor no-diagnostic. The failing stage was
`jtbl_rodata_pads --derive`, which prints to stderr AFTER cc1 exits 0 quietly. `_items` matched a
rodata anchor only as `D_xxxxxxxx:`, but a block written as inline `__asm__` in C arrives in the
labels.inc macro form `dlabel D_xxxxxxxx` — so an 8-byte hole opened in the walk and every C jump
table after it died with "island layout drift". The harness label was wrong twice: it said CC1
when the failure was a post-maspsx filter, and no-diagnostic when there was a precise one.
2. Same file, UNALIGNED ANCHOR: the ctable branch read `word(pos)` without first stepping the
sub-word zero gap, so a preceding `.asciz` ending at an odd address made it refuse a correct
layout. Now reuses the same zero_gap the anchor branches already use — a no-op wherever pos is
already aligned, i.e. everywhere that builds green today.
3. `harvest_verify` computed the typedef strip-set UNSCOPED: `cdecl.typedef_names(path)` without
`above=fn`, which that function supports for exactly this. A typedef declared BELOW the splice
point got stripped out of the draft that needed it -> `parse error`, logged as PLUMBING and
indistinguishable from a real conflict. One line.
4. NOT PATCHED, AND THE MOST IMPORTANT ONE: `reconcile_tu.py` (gate_stage's `-rc` stage) MANUFACTURED
both remaining "conflicts". The same drafts gate 9/9 byte-identical through harvest_verify and
7/9 through gate_stage. Isolated stage by stage, `-rc` (a) rewrites deliberately BLOCK-SCOPED
externs to a file-scope spelling whose typedef is declared ~2,800 lines lower — overwriting the
TU's own byte-proven house style, which three already-banked functions in that file use; and
(b) substitutes identifiers TEXTUALLY, including inside comments and inside `&`-expressions,
emitting `*(T *)&((s32 *)&D_800AE620)`. A correct draft using the block-scope-extern idiom
currently CANNOT survive gate_stage. Left for a deliberate fix: `--stages` should be able to skip
reconcile_tu, or a draft should be able to opt out.
The two surviving non-stub source edits are byte-neutral (proven by the SHA above): a §304
migrated-rodata re-emission (`D_801A01EC`, the exact form this TU already uses three times, needed
because banking the body deletes the .s that carried the island word), and one typedef moved up so a
function above it can see it (typedefs emit no bytes).
Also banked the 10th stub (func_801ADA10) that defect 3 had been silently blocking.
Regression-checked by the agent: main, md_MAIN_003, md_SC07_003, md_SC03_073, md_MAIN_011 all
rebuild BYTE-IDENTICAL. A full R22 follows before this session closes.
|
||
|
Drew T
|
5e10215269 |
feat(md): bank the 4 -O0-stranded functions — and the class is now essentially empty
md_MAIN_011:func_800CF28C · md_MAIN_003:func_800D0268/func_800D0740/func_800D0C50, all byte-verified
from a clean rebuild and counted from the SOURCE: md_MAIN_011 is now FULLY MATCHED (0 open stubs),
md_MAIN_003 is down to 1 (func_800CF3E8).
THE PREMISE I HANDED THE AGENT WAS WRONG, AND IT SAID SO. md_MAIN_011 is already a whole-object -O0
module — no carve was needed. Its real blocker was tools/jtbl_rodata_pads._s_rodata_span ignoring a
trailing `.align`, the SAME defect this session fixed for md_SC07_003 from the other direction: two
agents converged on it independently. Adopted this agent's stricter form (only a TRULY trailing
align rounds `hi`; an interior one is followed by data that sets `hi` higher anyway).
Note WHY it stayed latent: `derive`'s zero_gap self-corrects a 1-3 byte undershoot whenever the next
stream item is an anchor. A C jump table has NO anchor — so the bug can only fire the moment someone
banks a switch function into such an object, and when it fires it accuses the CARVE ("island layout
drift"), not itself.
md_MAIN_003 needed one new -O0 object, and the boundary I proposed (0x1f74 -> 0x1e58) was both too
narrow and off by 0x2B8. The carve made is `md_MAIN_003_o0e` at 0x1308 (vram 0x800D0100) running to
the existing o0c boundary: everything in that span is a §265 verbatim __asm__ body or an INCLUDE_ASM
stub — zero optimizable C — so the whole tail flips with one cut. Proved byte-identical with NOTHING
banked first (§431 discipline), then the three drafts gated one at a time.
TWO MORE GENERAL DEFECTS FIXED IN jr_isolate_all, both of which silently mis-place a boundary:
* an item-less CLOSING region emitted a duplicate `- [off, c, …]` line and the validator refused;
the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing
region look non-empty.
* A §265 VERBATIM __asm__ BODY IS PREAMBLE, AND PREAMBLE IS ASSUMED BYTE-NEUTRAL. It is not — it
emits bytes. `parse_overlay_c` has four addressed-anchor forms and a verbatim body is none of
them, so it attaches to the NEXT anchor: cutting at func_800D0268 would have moved 0x168 bytes of
other functions into the new object while the yaml claimed the region starts higher. New
`_region_emit_start()` derives the yaml offset from the region's CONTENT (item addresses + every
.globl/.ent the text names that resolves inside the object) and takes min(cut, emit), so a
boundary can only move DOWN. Where no verbatim asm is in play it equals the cut — every existing
isolate is unchanged.
BLAST RADIUS PROVEN, not argued: jtbl_rodata_pads is in the build path (`--derive` for md_*/main),
so the agent rebuilt main + all 70 md_* from scratch (71/71) and then ran the full fleet:
**make check-all 213/213 passed, 0 failed**, main 143dbb89 BYTE-IDENTICAL.
CENSUS, denominator asserted (1057 live stubs, 0 without a .s): exactly ONE -O0-prologue stub
remains stranded in an -O2 TU fleet-wide — main:func_8002C410 in src/800_b.c, 299 ins. Nothing more
should be built for this class; the general tool already existed and what was missing was
correctness, not coverage.
|
||
|
Drew T
|
3a886b9652 |
fix(tu-split): a block comment a construct OPENS MID-LINE and WRAPS defeated every peeler
FIVE independently-MATCHed ov_SC06_029 bodies were rejected by a `parse error before '#'` in a file
the GATE ITSELF generates, at a line no draft contains. The isolation emitted, into the §8b carried
decl layer:
extern #define CALL_80185C6C ((void *(*)(s32, s32))func_80185C6C) extern void func_8012C218();
CAUSE. Every peeler in the TU-split chain asked `line.strip().startswith("/*")`, which is blind to a
comment a construct opens MID-LINE and wraps. The declaration ends at its `;` BEFORE the `/*`, so
the caller resumed on the comment's PROSE with in_block=False — and the prose is hostile: `(s32,s32)`
closes a depth-0 paren, `seen_header` latches, and every later `;` reads as a K&R parameter
declaration, so one "construct" swallowed the whole preamble. `parse_overlay_c` then anchored a
`def` on a pure declaration run and `def_proto` rendered it as that definition's implied prototype.
A SECOND defect rode along: `_file_scope_decls` hoisted such a col-0 line VERBATIM, unterminated
`/*` included, so the carried layer opened a comment that silently ate the next two declarations —
a dropped file-scope decl is a silent byte-changer. Building the guard exposed a THIRD: `_strip`
tested for `/*` before stripping `//`, so `// … src/*/*.c` (7 lines in 5 sources) opened a phantom
block comment and blanked everything to the next `*/`.
FIX: one derived comment-state oracle, `comment_open_at()` (R33) — per line, does it BEGIN inside a
block comment — consulted by parse_overlay_c, def_proto, split_src_region.parse and
jr_isolate_all._file_scope_decls (which also truncates a hoisted decl at an unterminated `/*`).
`_strip` now lexes left to right. `parse_overlay_c` RAISES (R43) when a wrapped comment closes with
code after the `*/`, because that construct could never anchor — 0 occurrences fleet-wide.
MEASURED, not assumed:
* the shape occurs 238 times across 193 tracked .c files; 153 are col-0 hoistable declarations in
150 files — every one a binary whose next isolation would have carried a broken decl layer.
* A/B over all 4,188 tracked sources, old parser vs new: round-trip identity 4188/4188 both ways;
exactly 2 files' item lists change, each losing one PHANTOM def and gaining nothing; malformed
implied prototypes 999 -> 984; 0 refusals.
* negative control BEFORE any edit: ov_SC06_029 extract+build -j+check BYTE-IDENTICAL b7b0d4ae.
* with the fix, gate_stage banked 5 of 6 drafts, counted from the SOURCE; the 6th
(func_80184084) is the separate CARVE-REFUSED class.
The 984 residual malformed prototypes are a DIFFERENT pre-existing trigger (col-0 lines gluing
declarations to DEFINE_func_*() invocations); 4 still carry a `#` and survive only because it lands
in a dropped segment. Named in §437, deliberately not fixed here.
Cookbook §437 + a SETUP.md tooling-ledger row for comment_open_at (parse_overlay_c may now raise).
The banks themselves are NOT in this commit: the agent's worktree predated func_8017F9C0's bank, so
adopting its TU verbatim would have destroyed one. They get re-gated against HEAD with these tools.
|
||
|
Drew T
|
e9220d2d8a |
fix(pgate): a carve left asm/ stale, and the refusal that reported it named nothing
TWO DEFECTS, ONE INCIDENT. ov_SC03_105's own SUCCESSFUL gate committed an isolation's new TUs (src/ov_SC03_105/ov_SC03_105_jr_801813BC.c, _jr_80181C84.c) whose `INCLUDE_ASM` lines name .s files that do not exist until a re-extract. corpus.stubs then refused — correctly, "the tree and the source disagree" — so the NEXT gate on that binary died before doing any work, and a matched body (func_801818E8) sat unbankable behind it. 1. THE REASON NOW TRAVELS WITH THE REFUSAL. stubs_of() returned a bare None and the caller printed "corpus refused in worktree": true, and naming nothing. It took a hand-built worktree to see that corpus had said exactly what was wrong all along. It now returns the message and the result JSON carries it. Verified against a TRUE reproduction (delete one .s in a scratch worktree): verdict REFUSED + "1 stub(s) have NO .s on disk ... src/...:4214: asm/.../func_8017F018.s". 2. THE MERGE STEP REPAIRS WHAT IT BROKE. For every binary whose carve created a new source file, assert corpus.stubs is satisfiable in the MAIN tree; if not, `make extract BINARY=<b>` and re-assert; if it is STILL unreadable, say so loudly rather than leaving a tree no tool can read (R32/R43). This is the R22 corollary — a config change needs a make extract, not just a make check — firing inside a tool's own commit. Repaired the live instance by hand first: rm -rf asm/ov_SC03_105 + extract + build -j + check -> BYTE-IDENTICAL d305ff6d, corpus readable again, and func_801818E8 then banked (commit:3718). Cookbook §436-D; wave-playbook §6 carries the hand-gating version of the same warning. |
||
|
Drew T
|
ee2963514a |
fix(tools): jtbl_rodata_pads measured a .s rodata span without its trailing .align
`_s_rodata_span` summed only the DATA an included `.s` emits, so a file ending `.asciz "r"` + `.align 2` measured 0x801A00D8..DA instead of ..DC. The island walk then landed 2 bytes short and `--derive` aborted with `C table entry 0 at 0x801A00DA ... island layout drift` — a true statement about a span that was never the real one. Worse, the Makefile pipes md_*/main through `--derive` without `set -o pipefail`, so the failure could yield a short object rather than stopping the build. Three lines: round `hi` up to the trailing align, which is what the assembler actually emits. CONTROLS (both on UNMODIFIED sources, so this is proven byte-neutral, not argued): md_SC07_003 clean rm -rf + extract + build -j + check -> BYTE-IDENTICAL 46af79a1 gate_main --assert-baseline -> BASELINE GREEN 143dbb89 Found by a drafting agent (md_SC07_003/func_801A09C8) that ran its own gate reject to ground instead of respelling its body, and proved the patch in scratch first: with the fix its draft's .rodata is byte-identical to the green control and .text differs in exactly 1 of 6266 words — a %lo(jtbl) carrying a section-symbol reloc that three already-green C-jtbl functions in the same object already ship. Cookbook §436-C, with the habit that found it. |
||
|
Drew T
|
95c7b7fe0f |
fix(tools): two tools read a source of truth describing a different world (+ hard-gate the third)
Three independent split agents hit both defects in one session, on the tools that CERTIFY and UNDO
the work they were doing. Each is fixed, negative-controlled against the exact failing case, wired
into its siblings, and documented in the same change (cookbook §436).
1. split_indicator attributed a jump table by the STUB'S DIRECTORY PATH. `make extract` does not
prune a re-homed subseg's `nonmatchings/<old>/` dir, so after a correct, byte-green §431 split
both the old and new dirs hold the moved stub — and the tool printed NEEDS SPLIT for a split that
was already correct. owners() now derives the owner from the CONFIG by address (R33), exactly as
jtbl_carve.func_subseg already does for the identical §8b hazard, and NAMES any leftover stub in
a `note:` line. Notes now print on an OK verdict too: hiding one behind `st != OK` is the same
defect in the other direction — a true verdict about a narrower world than the reader believes.
PROVEN by planting a stale stub for func_80182A00 under its old subseg: OK + the note, where the
old code would have seen one subseg owning two spans. --self-test still PASSes both directions.
2. jtbl_carve --revert did `git checkout --` on the WHOLE splat yaml. The carve owns only the
trailing data/.rodata region; the `c` pieces are source configuration it never writes. The blunt
form cannot tell "carve state I just added" from "the §431 split someone added to the same
uncommitted file", so --revert after a carve PROBE silently un-split the overlay — each agent
recovered only because they had backed the yaml up by hand. It now splices back only its own
region (parse_config gained an optional `lines=` so the SAME region derivation runs over the
committed text — one derivation, two callers), refuses loudly if the committed region carves onto
a subseg the current config no longer defines, and reports how many uncommitted `c` pieces it
preserved. PROVEN in the ov_SC01_084 worktree: carve → revert → the uncommitted split survived
("PRESERVED 30 uncommitted `c` piece(s)"), carve lines gone, diff back to the 6 split lines.
SIBLING: jtbl_family_bank.revert carried the same blunt checkout for the isolation's code pieces.
It now keeps whatever pre-dated the attempt (the `keep_regions` signal it already trusts for
src/) and NAMES anything it drops — an isolation region and a §431 split piece are both
`<ov>_jr_<addr>`, so no name test can tell them apart and only that signal can.
3. NOT A DEFECT, and recorded as such: a speculative carve fails the build with `jtbl_rodata_pads:
consumed 3 rodata jump table(s) but 9 pad spec(s) given`. That is R43 working — the pad spec is a
CONSEQUENCE of banking, not a prediction of it — and it reproduces identically on the pristine
unsplit config, so it is never evidence about a split.
make tools-health: split_indicator is a HARD GATE now, as its own comment promised it would become
once the last violation was split. 213 OK of 213; a new one fails the build instead of being echoed
past.
Cookbook §435 (an overlay TU split is near-free — 0/3,074, 1/2,679, 2/3,254 names crossed, because
the §8b carried decl layer re-emits externs per region so only typedefs can cross; and the gap test
between two rodata runs is "is this word a valid code address", not "is it zero") + §436 (the two
defects and the shape they share). Playbook + SETUP.md carry the emptied CARVE-BLOCKED class.
|
||
|
Drew T
|
57e5f970c8 |
docs(tools): four docstrings describing pre-session behaviour
* draw_waves Usage advertised [--no-main], which argparse never defined (the flags are --main / --only-main, and main is excluded by default), and omitted --exclude-file, which is now a PREREQUISITE that refuses a stale list. * jr_isolate's STATUS block still declared the tool BLOCKED on split_src_region with the blocker unbuilt. Five defects were fixed this session and it runs the full chain to completion; what remains is a duplicate-definition class at assembly. Says so, and points at §431 as the cheaper route than finishing the item model. * ld_interleave's layout diagram — the first thing anyone reads — showed the pre-S72 three-piece island with 6324C.data.o. main's island is SEVEN pieces driven by --order; --front/--tail is the overlay form now. * jtbl_rodata_pads described a stored-spec-only filter and advertised guards that no longer all exist; --derive serves main since S72. |
||
|
Drew T
|
9f8242d63c |
fix(progress): the #else half of a NON_MATCHING block is LIVE — REAL was undercounting by 7
classify() consumed everything from '#ifdef NON_MATCHING' through '#endif', swallowing the #else half. But banking replaces the #else INCLUDE_ASM with the real body and leaves the old attempt in the dead half — so every function banked that way landed in NO bucket: not real, not a stub, invisible in both numerator and denominator. Measured: CdReadStateMachine, CdReadSectorReadyCB and StreamLoadStateMachine are byte-identical in the shipped build and counted as zero. REAL 873 -> 880, matchable 1911 -> 1918 (seven functions fleet-wide, not the three I first checked). Now consumes only the DEAD half, then decides from the LIVE half: an INCLUDE_ASM there still buckets as NON_MATCHING (accounting unchanged), anything else rewinds and is classified normally. THIRD coverage defect of this exact shape in this one function — the K&R-definition case (~190k instructions erased) and the '#if 0' case are both documented in its own comments, which is what pointed me at it. A scanner that walks preprocessor structure needs a test per branch, not per directive. Found by the S73 documentation audit, which I had written off as producing only doc typos. |
||
|
Drew T
|
f06d81a7d0 |
feat(main): StreamLoadStateMachine banked — all 9 wave drafts in; gate_main is preprocessor-aware
StreamLoadStateMachine (459 ins) byte-identical. That completes wave S73m_1: 9 of 9 drafts banked, 2,413 instructions. gate_main defect this exposed, twice in one gate: resolve_conflicts scanned every 'extern' line with no notion of the preprocessor, so a declaration parked in the DEAD half of an '#ifdef NON_MATCHING / #else / #endif' pair read as a live constraint. It is never compiled and constrains nothing. func_80018714 and func_800377D8 each carry a stale '(void)' declaration in a dead branch while the LIVE definition takes a pointer and a u8 respectively; the first mis-blamed a draft, the second got a byte-verified draft DROPPED. live_text() now blanks those branches before the scan. The detour is instructive: I 'fixed' the draft twice to satisfy a constraint that did not exist, and each fix made it worse — the draft's original (u8) declaration was correct all along, because it matched the LIVE definition. Read which branch a declaration lives in before believing it. |
||
|
Drew T
|
a5a78bc9cf |
fix(split_src_region, jr_isolate): five defects in the overlay TU-split path (blocked since Phase 26)
jr_isolate has been unusable since Phase 26 — its own docstring says "BLOCKED on split_src_region". Five distinct defects, each found only after fixing the one above it: 1. split_src_region demanded an address for EVERY top-level item, but an overlay .c is full of address-less constructs (hoisted typedef blocks, per-function extern runs, comment banners). coalesce() now merges an address-less run FORWARD into the item below it — they are a preamble belonging to that function, which is §431's model. 2. coalesce re-derived the name from the MERGED text, so item_name matched the preamble instead of the function. It now carries (addr, name, text) captured before the merge. 3. item_name scanned COMMENTS as if they were code: a comment containing any parenthesised token won over the real definition below it. 4. item_name matched a leading "extern void (*D_x[])(void);" and returned the name "void" — the §192 class, which gate_main.sym_of fixed for itself and this tool never got. A real function was then treated as a preamble and merged into its neighbour, leaving its body inside another item while its own stub survived: 26 duplicate symbols in one overlay. It now anchors on a DEFINITION (ends in an open brace, not a semicolon) and refuses type keywords as names. 5. That definition anchor required column 0, so an INDENTED top-level body was invisible. Also: jr_isolate's idempotency check keyed on the CONFIG, which it writes FIRST, so any failure in between left a half-applied tree the tool believed was finished. It now requires the source file too and refuses with recovery instructions. And inject accepts "already present and textually IDENTICAL" — splat emits an empty function as C, not as a stub — while still failing hard when the destination defines it DIFFERENTLY. Progress on ov_SC02_005: trim went 78 kept / 231 moved -> 89 / 255; duplicate symbols 26 -> 0; the chain now runs to completion (rc=0). NOT DONE: the object still fails to assemble on a remaining duplicate-definition class. Tree restored, ov_SC02_005 BYTE-IDENTICAL. |
||
|
Drew T
|
e830e63be5 |
fix(draw_waves): normalise exclude rows to (binary, fn) — the list was excluding NOTHING
My own regression from the same session: exclude_audit.parse now returns 4-tuples (it carries the WALL pin and each entry's note), and draw_waves built `skip` straight from them, so every membership test against a 2-tuple missed and the exclude list had no effect at all — while the run reported success. Caught by MEASURING the pool rather than trusting the run: it came back 88 non-main + 45 main = the full frontier, when a 25-entry list should have reduced it. Now 69 and 41, which reconciles exactly (88 - 16 carve-blocked - 3 non-main walls; 45 - 4 open main walls, PopMatrix/PushMatrix being linked and already refused). The silently-narrowed-scope shape again, and the third time this session that counting the RESULT rather than trusting the REPORT is what caught it. |
||
|
Drew T
|
ca7102e3b6 |
fix(exclude_audit): pin curated WALLs so a derived classifier cannot drop them; merge 7 walls ledgers
Found by checking readiness rather than asserting it: S71's two PROVEN walls (ov_SC03_105:func_801834A4, ov_SC06_022:func_8017DF28) were NOT in the canonical list — they lived in a separate .run/S71_walls_found.txt the regeneration never saw. Drawing would have spent agents re-proving them (playbook §1b: a full agent run each time). Merging them in exposed a second defect: a WALL has no jump table, so the DERIVED logic would classify it RE-PROBE and drop it. in the input is now read as a PIN that survives regeneration, and the entry's ORIGINAL note is carried through — a wall's value is its refutation list, and replacing that with boilerplate turns evidence into a bare 'do not try'. Round-trip verified idempotent: 9 walls survive a second pass unchanged. Merged 7 walls ledgers (S67/S68/S68_332/S69/S70/S71/S71_found) into config/wave_exclude.txt: 25 entries = 16 CARVE-BLOCKED (derived) + 9 WALL (curated). The audit found 8 of the merged walls already BANKED — a wall that got matched is no longer a wall. DELIBERATELY NOT merged: .run/t3wall_list.txt, 99 BARE function names with no binary. R48 — the same name is a different function in another overlay, so a bare-name exclude over-excludes silently fleet-wide. |
||
|
Drew T
|
983df054f2 |
feat(draw): audit the exclude list as a PREREQUISITE — a stale one is refused
An exclude list records what the TOOLING could not do, then gets treated as a property of the FUNCTIONS. Nothing re-examined it, so every tool fix left behind a population that is now tractable and still marked impossible — invisible, because the draw filters it out before anything measures it. MEASURED one day after .run/S71_exclude.txt was written: 88 of its 107 entries were stale — 28 already banked, 14 linked PsyQ symbols that were never targets, and 46 whose blocker had since been fixed. Those 46 are 12,750 instructions of open, drawable work including main:SaveLoadRoutine (1,165), the largest function left in main. * tools/exclude_audit.py (NEW) — classifies each entry by its CURRENT blocker (BANKED / LINKED / RE-PROBE / CARVE-BLOCKED / WALL), regenerates keeping only the still-valid classes, and --assert-fresh exits 3 on staleness. * draw_waves --exclude-file — runs that audit and REFUSES to draw on a stale list, naming the counts and the regenerate command. --exclude-stale-ok still draws but prints what it ignores: skipping is possible, never silent. Also fixes the old --exclude parsing, which could not survive a '#' comment. * .run/S72_exclude.txt — the regenerated list: 19 entries (16 CARVE-BLOCKED + 3 WALL), each carrying its reason, down from 107. Verified in all three directions: stale refuses rc=1, fresh proceeds rc=0, override proceeds and announces. The parser's own report-don't-drop design caught a bug I introduced in it (comma-splitting before comment-stripping). |
||
|
Drew T
|
b173d88676 |
feat(split_indicator): detect subsegs that MUST be split before their switch fns can bank
A code object contributes exactly ONE contiguous .rodata run, so a subseg owning raw jump tables in >=2 non-adjacent island spans makes every switch function outside the one carveable span unbankable at any effort. main sat in that state from Phase 7 to Phase 31 and eleven functions were written off as 'PROVEN gate-rejects' because of it. The evidence is derivable from the raw image on day one; nothing was comparing it. FIRST FLEET RUN: 209/213 OK, 4 overlays flagged — ov_SC01_084, ov_SC02_005, ov_SC02_011, ov_SC03_105 — holding 16 open functions / 3,613 instructions (18% of the non-main frontier). All 16 were already in the S71 exclude list, i.e. recorded as if unmatchable rather than as 'needs a subseg split'. 3.7s fleet-wide. Self-test covers all three directions: fires on main's pre-S72 island (fed synthetically, because the real tree no longer holds that state), stays silent on main today, and does not over-fire on a one-span subseg. Linked-library subsegs are excluded on principle — their code comes from a .a so cc1 emits no table for them; without that filter main reports NEEDS SPLIT on libgs6, which the self-test caught. Wired into make tools-health. accelerators #20 gains the when-to-split rule: split where the BUILD forces a boundary (decidable at 0% matched), at the span-owner boundaries and nowhere else, never on TU archaeology. |
||
|
Drew T
|
8e8521da22 |
fix+docs: make every consumer aware of main's new TU layout (R36)
The split created two new TUs and a shared header; four consumers still described main's
game code as one file:
* tools/reconcile_slate.py — HARDCODED open('src/800.c'), so after the split it saw a
THIRD of main's typedefs while reporting success (silently-narrowed scope, R32).
Measured: 133 visible before the fix, 187 after, 0 lost. Now globs
corpus.src_files('main') + src/800_shared.h, so a future split is already handled.
* docs/wave-playbook.md 1c — still said spans B/C/D were NOT drawable and that drawing
one is an R45 violation. That is now false and would have STOPPED a future session
from drawing the very targets this work unlocked.
* cookbook §426 — its 'the remaining spans need src/800.c split' paragraph now records
that it was done the same session, and points at §431 for the method.
* config/dedup.us.yaml + src/shared/clearTbl40.h — both said dedup group I0 is
instantiated 'at both sites in src/800.c'; both sites are above 0x80035270 and are now
in src/800_c.c.
Byte-neutral: dedup.us.yaml parses, gate_main --assert-baseline BYTE-IDENTICAL.
SETUP.md gains a row describing the layout and the rule it implies: never hardcode
src/800.c, glob corpus.src_files('main').
|
||
|
Drew T
|
00e5bfe57c |
feat(main_diff_locate): TABLE REJECT — the third verdict class, and the one this session is about
func_800316F8's .text was BYTE-IDENTICAL and all 18 differing bytes were its own jump table; the tool called it a PLUMBING REJECT and routed it to the §376 declaration chain, advice that would never have fixed it. classify() now separates a .rodata-only divergence and names it §405-A: match_one compares .text ONLY, so a draft sits at closeness 0 while emitting a wrong table — gcc emits case BODIES in source order while entry i points at case i, so case value and case order are independent and only the order is pinned by .text. Attribution reads one symbol LOW for a cc1-emitted table (once the function is C its table is a $L label, not a jtbl_ data symbol, so the bytes land in the PRECEDING table's extent) — the OBJECT name is what identifies it, not the symbol name. Shared by gate_main so both report the same four verdicts. Controls: self-test PASS, green build IDENTICAL, and the known func_800316F8 case now classifies TABLE REJECT. |
||
|
Drew T
|
483e2514a3 |
feat(main): 3 span-B functions banked; gate_main now sees header-provided typedefs
func_8002EED8 · func_8002F248 · func_80031988 — byte-identical, the first banks that span B's carve made possible. gate_main defect the split exposed: defs_above scans the destination .c ALONE, so a typedef the TU gets through #include is invisible to strip_dup_typedefs and every draft carrying its own copy dies with 'redefinition of X'. Latent until src/800.c's split moved 19 shared typedefs into src/800_shared.h, at which point func_80031988 — byte-correct, and one of the eleven — failed to compile for that reason alone. header_defs() now walks the destination file's quoted includes transitively and seeds defs_above with what they provide, so an identical copy is stripped and a different shape is renamed, exactly as for in-file definitions. func_80031988 had TWO stacked blockers: this one, and the struct-tag false conflict in typesig fixed earlier today. Neither was a property of the function. |
||
|
Drew T
|
ab5d1e3188 |
fix(gate_main): a tag keyword is not a type this model can see (R39 over-refusal)
typesig() keeps only tokens in TYPES, which DISCARDS every typedef name — so 'Ent30D80 *' and 'Rec14 *' both reduce to '*' and the model has always been blind to what a pointer points at. But 'struct' was in TYPES, so 'struct Ent30D80 *' reduced to 'struct*' and conflicted with 'Ent30D80 *', its own typedef. src/800.c declares func_80031988 BOTH ways and compiles today — gcc, the arbiter, agrees they are one type — yet the checker DROPPED the byte-verified draft, and S71 recorded it among the eleven 'PROVEN gate-rejects'. Dropping struct/union/enum from the param token set loosens nothing the typedef path had not already loosened. R39 control over the already-succeeded population (gate_main.typesig is imported by pregate_check and four other tools, so this reaches overlay slates too): 452 drafts across 54 binaries, 178 drops before / 177 after — ZERO new refusals, exactly one removed: main:func_80031988. |
||
|
Drew T
|
69f3f22112 |
fix(journal_notes): a pack with one old note could never receive a newer one
The idempotency check was 'heading present -> skip', and claude_wave_packs.py calls this tool at pack-build time, so EVERY pack that had any history was sealed against evidence recorded later. Measured on wave S72m_1: 4 of 5 targets took a freshly-recorded carve finding and the fifth silently did not, because it alone carried a prior note. Now idempotent by REPLACEMENT — the generated block is always the tail of the file, so truncate at the heading and re-render the current row set. Re-running is byte-identical (verified) and never duplicates the heading. Also records the S72 main carve unlock in .run/journal_notes_local.jsonl so it reaches every future main switch-function pack through the same one code path. |
||
|
Drew T
|
cbf5bae043 |
feat(main): unblock main's switch functions — the rodata span carve + derived jtbl pads
main's gate could only ever say "got X want Y". S71 read 7 such verdicts as body rejects and recorded 11 functions as "PROVEN gate-rejects, §376 in its purest form". They are not: all 11 are switch functions, and the blocker is that main has had exactly ONE rodata carve since Phase 7 (LZSS's jtbl_80072A38). Every other main jump table stayed raw in the tail data, so a drafted switch DOUBLE-EMITTED its table, the image grew (+28/+52/+76/+84 measured), and all 238 symbols above 0x80072A4C shifted. * tools/main_diff_locate.py (NEW) — turns a red image into a named list of divergent symbols via the linker map; per-byte attribution, self-test flips a byte at a known address and asserts the containing symbol (plus the identical-pair direction). * gate_main.py — PRESERVES the red image + map before the R40 baseline control rebuilds over it, and auto-localizes: BODY REJECT vs PLUMBING REJECT vs MIXED. Also -j on the build (was single-threaded) and the §376 drop list written to .run/gate_main_dropped.json with the reconciliation chain. * splat.us.exe.yaml — the .rodata carve extends from the LZSS table alone to the whole contiguous game-jtbl span 0x80072A38-0x80072C70 (12 tables, one 800.o run). Byte-neutral with no drafts substituted (probed first). * jtbl_rodata_pads.py — --derive now works for main: one file-0-vram expression makes both address->bytes and yaml-piece->address correct for the EXE's 0x800 header and leaves flat overlays unchanged. Makefile arms it for BINARY=main. Banked byte-identical: func_8001A114, func_8001AAD0, func_8001AF34 — three of the eleven. 25 of main's 59 frontier functions (6,215 of 12,912 instructions) are in this class; the remaining spans need src/800.c split at the TU boundaries the spans reveal. |
||
|
Drew T
|
38a039d121 |
fix(gate_main): refuse a draft that contains its own INCLUDE_ASM (R43)
Substituting such a draft puts the stub straight back: nothing changes, the clean build is trivially byte-identical, and the function is reported banked while its stub is still in src/. That is how func_8002B0B4 was counted in this morning's "BANKED 5 of 6" when only 4 had applied. Refused at slate load so it fires in every mode including dry run. Negative-controlled both directions: the no-op slate is refused by name, a real verbatim draft still passes. Scope measured before generalising: 5 of 2,749 stored drafts, all one of two functions - rare, but silent, which is why it is a refusal and not a warning. |
||
|
Drew T
|
ce671f7e14 |
fix(gate_main): count banks from the SOURCE, not from the slate
len(good) is "what we decided to keep", not "what was substituted". A draft whose stub pattern does not match is a SILENT NO-OP: nothing changes, the build is trivially byte-identical, the batch passes, and the function is reported banked while its INCLUDE_ASM is still in src/. Measured here: the bisect printed "BANKED 5 of 6" and func_8002B0B4's stub was still in src/800.c - four real banks. The stub's absence is the bank oracle everywhere else in this project; gate_main now uses it too, and names any accepted draft that never applied instead of counting it. |
||
|
Drew T
|
a2b0a6b98f |
tune(draw): opus up to 340 ins, not 150 - Fable for difficulty, not length (Drew)
Drew, 2026-09-02: "use opus mainly and only escalate the difficult ones to fable".
The S69 table that set the old 150 line actually puts opus's cliff at ~350:
m1 opus 191-347 ins 10/15 MATCH 1,291 tok/matched-ins <-- best measured
m2 opus 347-670 ins 1/9 MATCH 7,158 <-- the cliff
So 150 was handing opus's STRONGEST band to Fable. Fable is now reserved for >340
instructions and for arm_from_history's compiler-internal residual signal (§413) at any
size. On the current pool that moves three functions back to opus while two stay on
Fable because their history names a scheduling/regalloc residual - escalation by
difficulty rather than length, which is the point.
Retries default back to opus: one failure is not evidence of a wall, and the history
signal lifts a target on its own if the notes justify it.
|
||
|
Drew T
|
2cedd19aaa |
fix(symfix): key the slate by (binary, fn); §420 multi-cluster rebase banks 4 in 57s
aprop_symfix deduped its slate by BARE FUNCTION NAME, so a four-row slate for func_8016AB6C across ov_SC03_107/ov_SC07_007/010/011 reported "1 drafts audited" - and the three dropped rows each needed a DIFFERENT rebase, because each overlay has its own target symbols. Same root as reloc_filter's binof and gate_lane's homonym staging: three tools, one R48/§238 defect. With all four visible, the structure is two uniform delta clusters of two, identical in shape across all four overlays - one seed body's two data clusters each moving as a block. STALE-DELTA only admits ONE cluster, so it refused all four as AMBIGUOUS. §420 records the safe generalisation (runs of constant delta, every run >= 2 members, D_ symbols only) and the verification step. Rebased by hand under that rule, all four still MATCH at closeness 0, and the gate banked 4/4 in 57 seconds with no drafting (commit:3629). Frontier 165, 45 banked. |
||
|
Drew T
|
6fac2e0f35 |
feat(waves): refuse to launch an agent at an ALREADY-BANKED target (R43/R45)
wave_args asserts a target is open AT DRAW TIME, then the payload sits on disk while
gates run. S71 launched ov_SC01_006/func_8017F9F8 from a payload built before the gate
that banked it; the agent spent a full run to report "STALE CARD - already banked
today", with no .s left to score against. Filtering the wave-2 payload found 3 such
targets of 27.
launch_check.py re-asks the same oracle everything else uses (a bank REMOVES the
INCLUDE_ASM stub, so corpus.stubs not containing the symbol IS the bank), either for one
target or by filtering a {wave,targets} payload in place. An unreadable binary is treated
as OPEN - a tool fault is not a verdict about the subject (R40).
|
||
|
Drew T
|
2f72f8b20d |
fix(pgate): REFUSE main — an incremental main gate is a FALSE PASS, not just a false diff
S71 ran main through parallel_gate, got "11 banked", committed it, and the R22 clean-fleet verify came back 212/213. main did not compile from clean; once the two declaration conflicts were reconciled it built and was STILL not byte-identical. All 11 were then re-gated one at a time against a clean build — 11 of 11 REJECT. The rule was already written down in ox_campaign.gate_main_batch: "main is gated by ONE CLEAN REBUILD of the whole EXE, never incrementally … main's extract rewrites the linker script, so an incremental main gate returns a FALSE DIFF." parallel_gate's worker IS gate_stage, so it inherits that — and S58 recorded the false-DIFF direction while this is the false-PASS one, which is worse: a false diff wastes drafts, a false pass commits wrong bytes and reads green until the next clean fleet check (R53's signature — a failed build leaves the previous object on disk and the SHA check downstream reads it). Now a refusal naming tools/gate_main.py, not a docstring in the callee (R43). Cookbook §414, including the two instrument errors made while recovering. |
||
|
Drew T
|
2814d385ac |
feat(waves): journal_notes also reads a project-local note file
A stopped agent produces no journal row, so the next agent on that function learns nothing — including that a scratch directory full of compiled candidates and their match_one scores is sitting on disk. .run/journal_notes_local.jsonl is the same row shape read through the same code path, so hand-recorded evidence reaches the pack exactly as an agent's own note does. Used immediately: six S71 agents that ran past 36 minutes were stopped to free their slots; each now has a local note naming its scratch dir and stating that a stop is NOT evidence of difficulty. All six are back in the draw pool and will draw at the Fable tier per §413. |
||
|
Drew T
|
bfea4affd8 |
feat(draw): route the model tier off the prior RESIDUAL CLASS, not nins
Measured on S71's own wave: wall-clock tracks iteration count, and iteration count tracks the residual class, not size. A 26-instruction function took 18 min / 31 tool calls (regalloc, finished NEAR); a 122-instruction one took 80 s / 10. The 20-33 min runs were all compiler-internal residuals — scheduling ties, birthing boost, register colouring, LUID order — where every hypothesis costs a compile-and-measure cycle. arm_for keys on nins alone, so a 47-instruction regalloc wall could not be drawn at the higher tier and nothing escalates mid-run. arm_from_history() now reads the function's own journal notes at draw time and returns fable when they name one of those classes; it never downgrades the size ladder's choice. R39 control over 3,147 functions with history x 3 bands = 9,441 decisions: 4,020 upgrades (43%), 0 downgrades. The control's FIRST form passed over an empty set — it keyed on journal rows carrying a binary, and there are none: the agent verdict schema never had that field, so every historical note is name-keyed and the same name is a different function in another overlay (§238). claude_wave_draft.js's VERDICT now requires `binary`, so new rows are exact. Cookbook §413. |
||
|
Drew T
|
93bfa45561 |
feat(carve): §323 blocker 2 cleared — jr-isolate ov_SC07_000 for func_8017F8B8, byte-identical
The type-name scan matched `}\s*(\w+)\s*;`, which reads `__attribute__` as the name and fails on the following `((` — so a packed file-local typedef never entered the carried set, every decl naming it read as an unknown type, and the isolate refused the whole overlay. Stripping attributes before the scan is the entire fix. |
||
|
Drew T
|
49c41094a6 |
feat(carve): jr_isolate_all places file-local statics — ov_SC03_010's carve refusal cleared
The CARVE-REFUSED class (10 of the frontier's gate failures) has one dominant cause: "subseg <ov>_jr_<addr> would host NON-CONTIGUOUS .rodata carves", whose named remedy is jr_isolate_all. The isolate itself then refused 4 of the 6 affected overlays over a file-local `static inline` helper (bandsetup, setup_80188D90) that has no address BY CONSTRUCTION — §82.1 helpers exist to shape their caller's code and emit no symbol. * jr_isolate_all now places such a definition with the ONE region that uses it, and refuses loudly if two regions do (two copies of a used static is a byte change, R43). * overlay_src_split._proto_from_lines no longer prefixes `extern` to a declaration that already has a storage class — `extern static inline void f(...)` is "multiple storage classes" to cc1. The two changes are inseparable: placing statics is what first made the tool emit a prototype for one. Byte-gated on ov_SC03_010: extract + build rc=0, sha1 cacaf7c2c08037e6934f9d02c0ae5d7c78cf2463 BYTE-IDENTICAL. jtbl_carve --probe then moves from `plan-refused` to `tail — standard §8a carve at gate time`. |
||
|
Drew T
|
02e1b3a7e6 |
feat(waves): every pack now carries that function's own PAST-ATTEMPT history
tools/journal_notes.py mines the agent journals per (binary, fn) and appends a PAST ATTEMPTS section to the pack; claude_wave_packs.py calls it automatically, so it is the default rather than a step to remember. Idempotent, and R48-safe (a note stamped with a different binary is never served — §238 homonyms). Measured before adopting (S71 wave 1, 50 one-agent workflows over the 210-function real frontier where every target had already refused an earlier wave): * 38/39 MATCH at closeness 0 (97.4%) vs S70's 124/131 (94.7%) on an EASIER pool * 29/39 agents cite a prior attempt as what they used * 4/39 banked by RECOVERING a body that already matched, from a path a note named * 11/39 matched on the first compile The two costs it removes are re-testing a measured-inert lever (§406 lists twelve, §407 fifteen, §410 four — each paid for by an agent and never seen again) and re-deriving a body that already exists on disk. Also: jr_isolate_all places file-local `static` definitions with the region that uses them instead of refusing the whole file. A `static inline` helper (§82.1) has no address by construction, which is not a defect; the R32 guard was refusing these and blocking the isolate on 4 of the 6 overlays whose CARVE-REFUSED functions it is the named remedy for. Two regions using one static is still a hard refusal (duplicating a used static is a byte change, R43). docs: cookbook §411, wave-playbook step 3b, accelerators entry. |
||
|
Drew T
|
137c418bc7 |
docs(phase-31): S71 — the 64 standalone matches priced honestly; 12 banked, 52 in four named lanes
* gate 1 (all 64 across 33 binaries): 12 banked — main 11 + ov_SC07_006 1. * gate 2 tested "a bad draft kills its binary's good ones" by re-staging only the 25 that recover_integration --probe-only called MATCH in their real TU: 0 banked. An honest null — that probe compiles and diffs bytes but never LINKS or CARVES, so it is a third oracle with its own blind spot. * triage (25/25 accounted): CARVE-REFUSED 10, undefined-reference 4, DIFF 3, CC1-FAIL-no-diagnostic 2, PARSE 1; gate 1 adds 7 func-decl / 4 data-decl / 6 type-decl conflicts. * R37 probe of the carve class: 6 of 8 are one refusal — a subseg would host NON-CONTIGUOUS .rodata carves — whose named remedy is jr_isolate_all (§8b). tools/restage_matching.py — rebuild a gate plan from probe verdicts. tools/gate_triage.py — route a gate's verdicts to the lane each one names (R47). |
||
|
Drew T
|
8cf0104386 |
fix(cards): defect 5 — an expired BASELINE-RED claim, without discarding any measurement
The S70 patch was refused by its own adversarial review for sorting rows by recency: a pair's ledger rows are several PROBES about one draft, alternating between `closeness 4` and `won't compile standalone`, so max(ts) serves whichever probe ran last — often the least informative. This form keeps both. * the ts-newest verdict is still selected (file order made the per-binary bulk ledger always win regardless of age: 25 pairs mis-selected), * AND the best measurement ever taken on the pair rides alongside it, so a later uninformative probe can no longer erase an earlier residual: 981 of 2,605 pairs gain a line they were previously denied. * BASELINE-RED is a fact about a binary at a moment (R51), frozen into an append-only ledger and replayed forever — 2,676 rows all stamped 2026-08-26. gate_feedback now reads the same live red union gate_stage consults, so a pack and the next gate run cannot disagree: 173 expired claims retired, 0 binaries currently red. R39 control 3/3 (expired-when-green, harness-line-when-red, measurement-survives). |