mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
85fb289db582d842fc41dc059fa187bb992e76ea
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
b9033851cf |
fix(phase-30 S47-F1): classify diagnostics by position, not vocabulary — 93 unknowns all named
harvest_verify.classify_fail kept only stderr lines containing the word `error`. gcc-2.7.2 emits
no `error:` prefix on hard errors, so lines like
src/…/ov_SC02_037_jr_8013B83C.c:447: multiple storage classes in declaration of `tail_…'
src/…/ov_SC06_025_jr_8012ACE0.c:2217: `tbl_D_80187044' undeclared (first use this function)
never survived the filter, `errs` held nothing but make's `Error 33` wrapper, and every hard error
was labelled CC1-FAIL(no-diagnostic) — "the compiler failed and we cannot see why". Measured cost
this session: 132 siblings of func_80132018 classified that way by one missing declaration, which
reads as a codegen wall and gets a family deprioritised. rtu_match had the same blindness repaired
at T0(b); the fix was never propagated here.
Fix: a diagnostic is a POSITION, not a vocabulary — `<file>:<line>: <text>`, plus the assembler's
`{standard input}:<line>:` (_SRC_DIAG). Context lines carry no `:<line>:` and are skipped.
Five controls pass, including the two that guard against over-fixing: PLUMBING still wins on a
declaration conflict, and a warnings-only failure still returns no-diagnostic.
Re-swept: 0 no-diagnostic remain. The 93 resolve to 35 redefinition-note, 11 D_801202A0
undeclared, 10 too-many-arguments, 4 too-few-arguments, 4 func_8001534C undeclared — every one a
cheap declaration/arity class, not a wall.
Residue now fully named (737): 207 undefined-reference across 42 symbols (a link-stage REMAP gap,
now the largest class), 138 DIFF (real divergence, 19% — the honest floor), 44 data-symbol
conflicting-types, 35 redefinition-note, 26 memcpy, 24 redeclared, 15 undeclared, 14 arity.
|
||
|
Drew T
|
50c6f61056 |
fix(phase-30 S43): the CC1-FAIL label named MAKE, not the compiler (~3,000 content-free labels)
- ROOT CAUSE: classify_fail took errs[-1], and make prints its own summary `make: *** [Makefile:N: build/src/<ov>/<tu>.o] Error N` LAST, always — so the wrapper won every time and the label carried only the TU name the record already stores. Each CC1-FAIL therefore cost a manual splice-and-rebuild to learn what cc1 actually said (3x in S42 alone). - MEASURED (R37, over the committed .classified.txt corpus): ~3,000 of ~4,000 CC1-FAIL labels are that wrapper; a further 1,019 are bare CC1-FAIL with no message at all. - FIX: _MAKE_WRAP guard excludes make's summary lines; the FIRST real diagnostic wins (cc1 cascades — error #1 is the root cause); nothing-but-wrapper is now labelled CC1-FAIL(no-diagnostic) rather than disguised (R32). Same family as the §58 warning red-herring guard directly above it: a label identical for every input carries no information. - VERIFIED on the exact branch (exec'd the real source, see hazard below): "CC1-FAIL: make: *** [...] Error 33" -> "CC1-FAIL: src/…/tu.c:2240: error: too few arguments to function `gte_ldv3'". DIFF/SKIP/PLUMBING paths unchanged. - HAZARD DOCUMENTED (mine): harvest_verify.py has NO `if __name__ == '__main__'` guard — the whole gate is module-level, so `import harvest_verify` PARSES argv, RUNS A BUILD and overwrites .run/harvest_*.txt. Tripped it unit-testing classify_fail (resident stayed 8e17e02f, 0 banked, tree clean, no damage). Nothing imports it today, so it is flagged in the file header rather than fixed by a risky 500-line refactor of our most load-bearing gate. |
||
|
Drew T
|
6b21af6a14 |
fix(tools): classify_fail truncated its diagnostic from the LEFT, severing the symbol name
These diagnostics are "<long path>:<position>: <message>", and the MESSAGE names the symbol — which is the entire routing value of the PLUMBING/CC1-FAIL label. The old [:90] spent its budget on the path and cut the payload mid-token, so a real wave-6 failure was labelled: PLUMBING: src/ov_SC03_001/..._jr_8017AE2C.c:(.text+0xc090): undefined reference to `func_801 with the symbol severed at four hex digits — unroutable. New _squeeze() keeps a short head (the file is still identifiable) and the tail (the symbol), so the same line now ends "undefined reference to `func_80143C74'". Same family as the §58 red-herring guard directly below it and §136a: a label that cannot distinguish its inputs carries no information, and one that drops the payload is worse than none. |
||
|
Drew T
|
73321fe2c2 |
fix(phase-30 S38): the gate was booking CRASHES as silence — 10 of wave 6's 16 drafts vanished
THE DEFECT CHAIN (byte-witnessed, both ends fixed):
1. harvest_verify._reload_corpus re-applied the `--src` filter AFTER a jtbl carve. Following a
carved stub to its NEW TU is that function's entire documented purpose, and it was deleting the
very stub it had just followed. Then:
_stubs loses fn -> render() raises KeyError -> UNCAUGHT -> _jtbl_restore(snap) never runs
-> the carve is STRANDED in config/ + src/ -> every LATER group in the same gate run then
built against a tree the earlier crashes had mutated.
line 136 already calls --src "an optional filter, not a location oracle"; this was the one place
treating it as one. Fix: the filter never drops a draft under verification, wherever it now
lives, + an R32 loud report if a working stub vanishes across a reload (which also repairs
_touched/baseline — a carved fn missing from _stubs left its new TU unbaselined, so the revert
path could not have restored it either).
2. .run/s6f_gate.py never checked the child's returncode — it grepped stdout for VERIFIED:/FAILED:
and booked "neither" as NOTHING, printing a clean-looking tally over 10 missing verdicts. This
is the §136a defect I logged against my own capture tool last session, in the gate itself.
Fix: 1:1 accounting assertion (banked+failed+no-verdict == drafts), the child's rc + output tail
on anything unaccounted, and exit 1 — a crashed child may have stranded a carve, so it must
never look like success.
PROOF THE FIX IS NOT COSMETIC: func_8017EA84 (579 ins) now carves and banks BYTE-IDENTICAL. The old
tool reported it as nothing at all.
BANKED 7 (R22 clean-fleet 140/140 from `make clean` + extract-all + check-all):
func_8017EA84 ov_SC02_000 (579) · func_8017FEE0 ov_SC02_026 (299) · func_80181CE4 ov_SC03_111 (491)
func_80183AE0 ov_SC03_112 (240) · func_80184C74 ov_SC06_018 (288) · func_80180F98 ov_SC03_097 (263)
func_801841C8 ov_SC02_035 (44)
MY OWN ERROR, RECORDED (R37/R14): after reverting the stranded carves I re-extracted ONE overlay,
not all 16 — the Phase-20 R22 corollary (a reverted CONFIG needs a `make extract`, not just a
revert) which I know and skipped. Run 2 therefore read WORSE than run 1: three genuinely-banked
functions failed against stale asm. Re-extracting the 16 touched binaries produced the honest run.
A gate result measured against stale asm is not a measurement (R35).
.run/w6_diag.py: run the REAL gate path for one (ov,fn) with the child's full output. s36_capture.py
splices without the carve, which is the wrong path for a table-bearing fn (§61b: the carve must
follow the splice) and produces a failure that is an artifact of the diagnosis.
|
||
|
Drew T
|
97cd2739b5 |
fix(phase-29): the gate manufactured 3 false CC1-FAIL verdicts — carve-refusal, tree hygiene, R32 (§97)
A 15-draft harvest_verify batch reported CC1-FAIL=4 and `final SHA None`. Three of the four were the HARNESS, not the compiler. Checked the tree FIRST (the MISMATCH is a tree alarm, not a result), reverted to the committed baseline rather than reasoning about a half-applied state, rebuilt -> d19c9580 BYTE-IDENTICAL. No banked result was ever at risk: the byte-gate cannot manufacture a match, but it CAN manufacture a verdict — and verdicts are what the backlog and roadmap are built from. ORDERING PROVED THE CASCADE (R14): items 1-11 are real (9 PLUMBING, 2 DIFF), all before item 12 — jtbl_carve REFUSING func_8013B83C (§59(3) non-contiguous same-subseg table). Items 13-16 are four CC1-FAILs on the SAME ov_SC01_077_o0.o = one refused carve counted four times. THREE DEFECTS FIXED: 1. `_ok` was computed and IGNORED — a refused carve was spliced and built anyway into a guaranteed Error 33, filed as CC1-FAIL. Now a named CARVE-REFUSED class, skipped (one build cheaper). 2. attempt() never restored on failure, so the tree was dirty BETWEEN drafts — and _jtbl_snapshot() snapshots the tree AS IT FINDS IT, so a later carve captured an earlier FAILED draft's splice and its undo faithfully RE-APPLIED it, after the final _write(baseline). That is the entire `final SHA None` mechanism. Invariant restored: the tree is at baseline except while a draft is under test (atomic AND bisect branches). 3. The recovery's own `make extract` rc was unchecked (_sh does not raise — §93's sibling). Now loud. Plus an R32 assertion on the cleanup: at 0 verified a non-empty git status is residue, not a result; it names the files and the recovery command. It fired correctly on its first real run. MEASURED RECOVERY (same drafts, clean tree): func_8013B83C -> CARVE-REFUSED; func_801789AC -> PLUMBING (actionable); func_8017C974 -> DIFF (corroborates its agent's global_alloc spill diagnosis); func_80140958 -> CC1-FAIL (genuinely its own). final SHA None -> d19c9580; tracked diff empty. BLAST RADIUS OF §96, HONESTLY: the reconcile_tu span fix unblocked func_80176218 (banked, swept 133/137) and no other draft in the batch. 7 of the 9 PLUMBING are `conflicting types for <the function itself>` = the DEF-side self-decl axis conform_decls owns — the next lever, now a measured target list rather than a guess. cookbook §97. |
||
|
Drew T
|
3ebdd91fc6 |
feat(phase-29): jtbl post-carve reconcile (harvest_verify._jtbl_reconcile) + REFUTE the reach-138 jtbl families as plumbing wins
- ROOT CAUSE: the jtbl carve's s8b carried-decl layer conflicts with each draft's externs; the reconcile chain (cast_call_sites+reconcile_tu, --src-file-aware) exists but gate_stage runs it PRE-carve against the wrong TU (a jtbl fn's real TU is the split file, which doesn't exist until harvest_verify carves). - FIX (cookbook s62): harvest_verify._jtbl_reconcile runs the chain POST-carve against the CARVED TU, draft-only rewrite, guarded by _jsnap is not None. Validated: func_80135260 (callee) + func_80191C50 (data) both conflicting-types -> genuine DIFF. - FINDING (R14/R31 -> decision-log): dissolving the plumbing REVEALED all 4 jtbl drafts have a deeper issue -- func_80135260/80191C50 real %hi-share regalloc residual (agents' reloc-masked match_one MATCH over-claimed it); func_8012AAAC def-side-arity + FLEET- SHARED + still DIFFs after arity fix (def-side wall); func_80135EB0 isolate FAILED. The '+0.58pp from 3 reach-138 jtbl families' is REFUTED -- near-misses/walls, not plumbing. The fix banks any plumbing-ONLY jtbl fn + makes the jtbl gate honest. - s61 traps re-confirmed (s62): gate jtbl ONE-AT-A-TIME (mid-batch isolate-FAIL corrupts the batch); fix_arity on an engine_core.h fn leaks fleet-wide (restore src/shared/ too). - R22 clean-fleet 140/140; tool change only (no committed-byte change). |
||
|
Drew T
|
52e772628b |
feat(phase-29): isolate-with-body FIX + 3 more preserved cracks banked (incl. a 673-ins giant)
THE FIX (harvest_verify._jtbl_prep_one): isolate WITH THE BODY STILL SPLICED.
jr_isolate_all accumulates each object's file-scope decls as the new region's
`ambient` set, so partitioning around an INCLUDE_ASM stub hands the region a
DIFFERENT decl context than the draft's body needs — and the gate then produced a
byte-DIFF rather than a compile error, which is why a batch of these read as
"9 compile / 0 bank" and looked like a codegen wall. §61b's law extends one step:
THE CARVE MUST FOLLOW THE SPLICE — AND SO MUST THE ISOLATION.
The un-splice existed only because the tool is stub-centric (a spliced fn is no
longer in corpus.stubs). New _unsplice_body() handles that properly: it finds the
file that NOW holds the body (isolation may have MOVED it into a fresh region file)
and writes back the stub line for THAT subseg, so the gate re-splices identical text.
BANKED via the corrected path (each whole-binary byte-gated, one per invocation):
func_80135D20 (100 ins, reach 138)
func_801749C8 (105 ins, reach 136)
func_8019059C (673 ins, reach 3 — a giant)
Together with func_80135888 and func_8018F694 earlier: 5 of the 11 preserved
t5wave cracks are now banked.
THE REMAINING 6, honestly classed: 2 genuine DIFF (func_80135260, func_80191C50),
2 CC1-FAIL (func_801365B8, func_80165CA0), 2 §57 self-decl plumbing
(func_801299C8 `prototype declaration`, func_8012AAAC own-name conflict).
⚠️ RETRACTION: the earlier "the ladder converts 0/10, which prices Task 14 stages
2-3" is WRONG and is withdrawn. Three of those ten bank through harvest_verify
alone with the SAME transformed drafts that gate_stage rejected — so that 0/10 was
measuring gate_stage's own interference, not the residuals. Prime suspect is the
arity pre-pass (fix_arity_callers --apply --any-proto edits engine_core.h before
the gate; the §19 regression mode). GATE_NO_ARITY=1 is the ready A/B. Task 14
stages 2-3 remain UNPRICED until that runs.
- R22 clean-fleet 140/140 BYTE-IDENTICAL; tree clean before and after
|
||
|
Drew T
|
fcd83f705e |
fix(phase-29): the undo was EATING the tree — snapshot src too; kill the §58 label; giant func_8018F694 banked
THE DEFECT (byte-witnessed, and it is the §61c mechanism). `jr_isolate_all`
repartitions a code object by writing region 0 back over the ORIGINAL
src/<ov>/<nm>.c — TRUNCATED to just that region — and emitting the rest as new
_jr_<lo>.c files. `_jtbl_restore` undid only config/ + the new region files, so
every gate-REJECTED draft left the original TU permanently truncated and its
stubs gone. Nothing regenerates them (splat does not rewrite a committed overlay
.c). Measured across an 11-draft re-probe: live stubs 419 -> 414 -> 406 -> 395,
ending in `undefined reference to func_80191C50`.
It is INVISIBLE to the gate that causes it: the incremental build keeps linking
stale objects (§42b) so `make build` stays green while a CLEAN rebuild fails.
That is exactly the "139/140, twice" signature that became the §61c blocker —
the tree was being eaten by the undo meant to protect it. The §61c attribution
(batch _jtbl_prep residue) is now a demonstrated defect, not an inference.
FIXES (both negative-control-validated on a draft that fails):
- _jtbl_snapshot captures every src/<binary>/*.c; _jtbl_restore restores them and
removes exactly the files the attempt created (derived from the snapshot's file
set, not re-guessed from the _jr_* name shape, R33). Undo by SNAPSHOT-RESTORE,
never an inverse transform — §61's law one level deeper. Same failing draft that
previously broke the tree now leaves it byte-identical, git status clean.
- classify_fail ignores `warning:` lines. The benign `conflicting types for
built-in function 'memcpy'` warning was winning the match on 8 of 8 failures
across four different real causes — a label identical for every input, which the
cookbook had to work around by hand ("the gate label is useless here, splice
individually and read real cc1 stderr"). Now reports the real error, and falls
through to CC1-FAIL:<last error line> rather than guessing.
THE RE-PROBE THIS ENABLED (11 preserved t5wave cracks, one invocation each):
- BANKED: func_8018F694 (478 ins) — one of the wave's three giants, previously
recorded as part of "the gate banked ZERO".
- The other 10 now carry TEN DISTINCT diagnoses: 4 data-decl conflicts (D_80193B64
x2, D_8011D030, D_80126B5C), 3 callee-decl (func_80135480 x2, func_8012F14C),
3 self-decl/own-sig (§57). ZERO jtbl-drift, ZERO local-type redefinition, ZERO
codegen DIFF.
- So §61a's "§8e-2 jtbl table-count drift blocks 10 of 12" does NOT survive the
carve-follows-splice prep: the carve now succeeds and what is left is ordinary
decl plumbing the existing ladder already handles (cast_call_sites / reconcile_tu
/ normalize_self_decls / fix_arity_callers) = Task 14 stages 2-3, no new tooling.
- ov_SC06_018 BYTE-IDENTICAL cbbc4f44 with the giant banked; nothing committed broken
at any point (tree was restored from HEAD and re-verified before these fixes)
- cookbook: the `git add -u` complementary hole (an isolation's NEW region file is
untracked, so a carve/isolation bank needs `git add -A src/ config/`)
|
||
|
Drew T
|
41d65af73f |
fix(phase-29): per-function jtbl prep + stranded-carve undo; the clean-rebuild blocker named
THE BLOCKING FINDING (§61c): the jtbl carve+isolation path yields a state that is
INCREMENTALLY valid and CLEAN-INVALID. func_80135A4C banks every time through the automated
path (BYTE-IDENTICAL at harvest_verify's gate) and fails `make clean && extract-all &&
check-all` twice, identically (139/140, [FAIL] ov_SC06_018). The bank is therefore NOT
reproducible from committed config + source, and the gate that authorises it cannot see the
defect because the gate IS the incremental build (§42b, most expensive form).
=> NO jtbl core can be banked until that divergence is diagnosed. Next step is to diff the
incremental vs clean build/ov_SC06_018/** object set + generated .ld/asm for the carved
subseg — NOT to bank more. All 12 wave cracks stay preserved at .run/giants/t5wave_*.
TWO REAL DESIGN FAULTS FOUND AND FIXED IN harvest_verify:
1. A stranded carve poisons the overlay. _jtbl_prep carved a draft the gate then REJECTED;
the carve remained with NO owner (the fn is still INCLUDE_ASM), and jr_inventory's 1:1
ownership assertion then refused every later isolation in that overlay
([('UNOWNED','0x801d288c')] = func_801299C8's table). THE ASSERTION WAS RIGHT AND CAUGHT IT
— R32/R33 working as designed; the defect was mine for leaving the carve behind. Fixed:
per-function carve + exact snapshot-restore (config text + only this attempt's region files)
on gate rejection.
2. Per-function undo is UNSOUND IN A BATCH: isolation REPARTITIONS shared source, so restoring
one draft's snapshot deletes region files now hosting OTHER pending drafts, and their stubs
vanish (KeyError in render). jtbl drafts must run one per harvest_verify invocation, or the
undo must be region-aware.
MEASURED so it is not re-derived: of 11 preserved cracks exactly ONE (func_80135A4C) reaches
byte-identical through the carve path; the other 4 table-bearing ones fail one-at-a-time too,
on PLUMBING (§57 self-decl et al), not the carve.
Tree reverted; clean-fleet 140/140; nothing banked. cookbook §61c.
|
||
|
Drew T
|
aa1b8b3c1a |
feat(phase-29 T14 stage 4): jtbl prep moved into harvest_verify — it BANKS; new blocker named
THE LAW IMPLEMENTED: the carve must follow the splice. harvest_verify._jtbl_prep() splices each
table-bearing draft TEMPORARILY, asks jtbl_carve, isolates on the §8b walls, un-splices, re-extracts,
and RE-DERIVES the stub map + baseline (isolation MOVES a stub's TU, so both are keyed on stale
paths otherwise). gate_stage's batch pre-pass could not work: while a fn is still INCLUDE_ASM the
non-contiguity is undetectable, so jtbl_carve reports success and yields a spec that fails when the
body lands.
BYTE-PROVEN AUTOMATED: func_80135A4C (181 ins, 138 members) ->
[jtbl] carved 1/1 table-bearing draft(s): func_80135A4C
+ chunk(1): func_80135A4C verified 1 / failed 0 BYTE-IDENTICAL
NEW BLOCKER, PRECISELY NAMED — banking a jtbl core makes its own carve UNOWNED to jr_inventory,
which then refuses every subsequent isolation in that overlay:
"committed .rodata carve ownership is not 1:1 (R32/R33) — a stranded/duplicated carve
(§8b func_801734BC class): [('UNOWNED', '0x801d288c')]"
Byte-proven both ways: on the COMMITTED tree `jr_isolate_all --only func_80135260 --dry-run`
succeeds; with func_80135A4C banked it fails the assertion. The assertion is RIGHT (a banked fn's
stub .s is pruned, so the owner lookup finds nobody) but its conclusion is wrong — the carve IS
owned, by C rather than a stub. SO TODAY JTBL CORES BANK ONE PER OVERLAY. 10-draft batch: 6
table-bearing -> 1 carved, 4 isolate-FAILED on this assertion, 1 stale-asm carve failure.
NEXT INCREMENT (precise): resolve jr_inventory's carve owners from corpus.matched U stubs, not
stubs alone (R33 — the same derive-don't-reparse move that fixed the corpus oracle).
BANK NOT KEPT: R22 clean-fleet showed 139/140 (ov_SC06_018 FAILS from a clean tree) even though the
INCREMENTAL build read BYTE-IDENTICAL — the stale-incremental false pass R22 exists to catch (§42b).
Reverted; clean-fleet re-verified 140/140, tools-health OK (dedup 1848/0). All 12 wave cracks remain
preserved at .run/giants/t5wave_*. cookbook §61b updated.
|
||
|
Drew T
|
1dbf5477d0 |
perf(phase-28 T6): harvest_verify --chunk 1 no longer double-builds every failing draft
The gate's hot path is --chunk 1 (the prescribed default — chunked failures mis-attribute innocent neighbours, cookbook:1568). With an atomic chunk the old code fell into the bisect loop and re-ran attempt([fn]) on the SAME single element against the SAME baseline: a byte-identical DUPLICATE build. classify_fail reads _last_sha/_last_err, which the failed attempt(chunk) ALREADY set, so the re-attempt bought nothing but a second cc1+maspsx+as+ld. - FIX: an `elif len(chunk) == 1:` branch classifies + reports directly, skipping the bisect. Per-draft failure build cost 2 -> 1 in the loop; ~26% fewer builds overall at the measured 65% bank rate (successes were always 1 build; only failures doubled). Every wave, one branch. - CORRECTNESS unchanged: verified a guaranteed-failing draft still classifies DIFF and is reported, not silently dropped. Measured make-build calls for one failing draft = 2 (attempt + the single final confirming build), the redundant re-attempt gone. - Gate tooling only — no src/config/build change, cannot alter any binary's bytes (a bug here can only fail-to-bank, never falsely bank; G3/P9). No R22 owed. Found by the explore agent's gate-substrate survey (Phase-28 planning); the shard-farm half of T6 is deferred (conditional on P29 volume — the distinct-binary farm already exists in bulk_harvest, and the same-binary axis is a directory-shard build, documented in CURRENT_PHASE). |
||
|
Drew T
|
e0a0becfaa |
feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr
The plan named two defective regexes; the tree had SIX with complementary holes, each silently recording the resulting compile failure as "not a match" — a plumbing error wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32). - cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs; keeps draft-local types. lru_cached. - harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) -> unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1 stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… / CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer recorded byte-identically to a codegen miss. - masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33, cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one + p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over whitespace. Unblocks B4's func_8015C32C (redefinition of 's16'). - canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap). VERIFIED: - HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that was being discarded over whitespace). - unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped. - classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly. - all 5 edited tools import + AST-parse clean. - R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248 "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.) - SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the original asm; a wrong draft always changes bytes -> always fails SHA1). |
||
|
Drew T
|
70a99aa9f7 |
fix(phase-26a): A3 — the byte-gate could only see ONE translation unit (96.6% of stubs unreachable)
harvest_verify is the sole arbiter (G3/P9) and has never accepted a wrong match. It also could not
REACH most of the work: it scanned the single file the caller passed as --src and silently dropped
every draft whose stub lived elsewhere. An overlay's source spans up to 14 .c files, so:
open overlay stubs it could not see : 56,742 of 58,717 (96.6%)
ov_SC01_077 reachable by the gate : 13 of 264 (4.9%) -> 264 of 264 (100%)
THREE of the six callers passed no --src at all (orchestrator.py, grinder.py, idiom_hunt.py) and so
inherited gate_stage's `src = src or f"src/{binary}/{binary}.c"` default. For grinder.py that means
1,290 of its own 1,298 QUEUED FUNCTIONS COULD NEVER BANK, however good the permuter's output was.
=> Phase-22's "the permuter's fuel is exhausted" was never a safe conclusion. Re-test (A12).
gate_stage knew the right answer and then handed the gate the wrong file: its negative control
ALREADY globs every split .c to build bin_stubs. The default is now removed; --src is passed only
when a caller deliberately restricts the gate to one TU.
WHAT CHANGED, PRECISELY: only the SPLICE LOCATION. Each draft is now spliced into whichever TU
actually holds its stub, derived from tools/corpus.py. Every TU links into the same image, so ONE
`make build BINARY=<bin>` still gates them all — correct AND strictly fewer builds than the
per-split re-gate it replaces.
SAFETY (this is the byte-gate, so the argument is explicit): the VERDICT is untouched — `make build`
+ SHA1 == the locked hash. INCLUDE_ASM pastes the ORIGINAL assembly, so a wrong draft always changes
the bytes and always fails SHA1. A bug in the splice can therefore make the tool FAIL TO BANK; it
CANNOT make it falsely bank. The failure mode is conservative by construction.
VERIFIED end-to-end (2 real builds, tree clean before and after):
* discovery: 264 live stubs across 12 TUs (was: only those in the single --src file)
* IDENTITY known-answer test: 3 drafts whose stubs live in THREE DIFFERENT split TUs
(_jr_801734BC, _after, _jr_8012ACE0) — all discovered, spliced into their own files, built,
SHA-matched, committed, restored. Final SHA d19c9580 BYTE-IDENTICAL. Under the old code all
three were silently dropped as "not stubbed".
* `git checkout -- src/` recovers, exactly as the docstring promises.
Also derived rather than defaulted: --good-sha now reads config/check.<bin>.sha (a caller that
passed --binary but forgot --good-sha used to gate an overlay against RESIDENT's SHA), and
match_one_closeness resolves the asm subdir PER FUNCTION — one subdir for a whole batch is the same
single-TU bug, and pointing match_one at the wrong one scores a draft against a DIFFERENT function's
asm, producing a phantom non-zero closeness that lands in the backlog and feeds reserved_walls().
No committed source or config changed, so no build artifact can have moved; the byte-gate was
exercised twice and returned BYTE-IDENTICAL both times.
|
||
|
Drew T
|
c5cca1c6d4 |
feat(phase-23): T10 tooling — parallel-safe gate spine + bulk_harvest
- gate_stage.run_gate: optional lock_path (per-binary lock), verified_out/failed_out (per-worker harvest scratch), compute_fleet toggle — all default to today's behavior, so grinder/orchestrator/lora_grind are byte-unaffected - harvest_verify: --verified-out/--failed-out (per-worker result paths for parallel gating) - bulk_harvest.py (NEW): phase-separated saturation factory — bulk-draft (GPU) -> ProcessPool parallel-gate over distinct binaries (build/<bin>/** isolated) -> dedupe-once + one commit; round-robin fuel spread; whole-binary byte-gate stays the sole arbiter (G3/P9); on-demand/bounded (STOP sentinel); lora_grind kept as fallback |
||
|
Drew T
|
4c7cf48ebf |
feat(phase-12): T3 — +2 (typedef-strip recovery) + per-function match tool (REAL 87->89)
- tools/match_one.py (new): compile ONE function's C standalone (pinned triple), mask relocations (jal/HI16/LO16 like psyq_identify), compare to the target bytes in its .s. Isolated temp dir -> parallelizable; a real per-function asm-differ loop for agents to iterate against without a shared build. Validated: a matched fn -> MATCH, residuals -> informative diffs. - harvest_verify.py + match_one.py: strip inline scalar-typedef redefinitions (some drafts inlined `typedef unsigned char u8;` -> C89 redefinition error vs common.h = a COMPILE fail, not a byte mismatch). Re-gate recovered func_800CFBCC + func_800D1938. - REAL 89 (62.76% byte-identical); resident 8e17e02f, main 143dbb89 (R22 clean). - 54 stubs remain (scheduling/regalloc residuals + big GPU/dispatch fns) -> match_one-driven iteration pass next. |
||
|
Drew T
|
c37332fa19 |
feat(phase-12): T3 batch 2 — 62 resident functions matched via parallel-draft + byte-gate (REAL 8->70)
Ultracode harvest: a 16-agent workflow (resident-harvest-draft) drafted matching C for the 135 unmatched resident functions (each agent: asm + cookbook + the 8 worked examples, writing .run/drafts/<fn>.c, no builds/no Ghidra). Then tools/harvest_verify.py (new) byte- gated every draft against 8e17e02f one at a time (chunk+bisect), keeping only the byte- perfect ones and reverting the rest -- the build is the sole arbiter (G3/P9). - 62 / 127 drafts verified byte-identical in one pass; 65 near/hard misses reverted to stubs. - REAL 8 -> 70 (48.28%); byte-identical 49.66%. resident 8e17e02f, main 143dbb89 (R22 clean). - cleaned 35 redundant '#include "common.h"' lines the drafts inlined (still byte-identical). - remaining: 65 failed drafts (close-misses + the hard dispatch/GPU/big-fn tail) + 8 undrafted (2 agent batches hit transient API 500s) -> second-pass workflow + manual hard tail. |