The grinder/backlog pipeline was ov_SC01_077-hardcoded 5 layers deep (same class as the
T7 lora_grind bug). Fixed all so the permuter grinder can process a non-077 near-miss:
1. gate_stage.append_record stores the source "binary"
2. backlog.FIELDS keeps it (else append_record dropped it)
3. backlog.load_best/_open_stubs is fleet-aware: a fn matched in ov_SC01_077 but
propagation-stuck stays OPEN in its overlay, so it surfaces via that record instead
of being dropped as "matched" (the grinder must SEE it to grind it)
4. p16_permute.setup takes the target binary's asm-subdir (was hardcoded 077)
5. grinder resolves per-binary asm + gates grouped by binary + allows unknown nins
Backward-compatible: legacy records (no binary) default ov_SC01_077.
Validated end-to-end: the 3 fresh reach-134 close=1 ov_SC01_000 fns now surface, resolve
to ov_SC01_000's asm, and gate via ov_SC01_000.
TWO byte-evidenced findings (redirect the fuel strategy):
- the reach>=2 close=1 fuel is MODEL semantic-misses, not permuter fuel: func_8012E27C's
target is "return 1" but the 7B drafted an empty "void f(void){}" (corpus overfit
empty-leaf); func_8012BF4C/AD64 are trivial sw/sh setters drafted empty. A corrected
draft banks them (+3 byte-identical via the fixed gate, @commit:0326); the permuter cannot
add a missing return/store. Lever = corpus-v3 leaf variety, not the permuter.
- x reach is propagation-capped: the 3 are inline-matched in ov_SC01_077_a.c (the stuck-
local cap) -> dedup_propagate "nothing to propagate" -> banked x1. Lever = dedup-collapse.
check-all 136/136 throughout. docs/gen2-mips-matching-model.md + CURRENT_PHASE updated.
Add a sig-based reach oracle + --min-reach N to lora_grind so the mass-run can prefer
SHARED functions (one bank propagates x reach — the fleet-% multiplier). The oracle reads
the same .run/sig.ov_*.jsonl dedup_propagate uses (validated: 0 mismatch over 60 stubs +
the func_8017CE24=2 ground truth), so a reach>=N target is exactly one dedup_propagate
will stamp x reach after the bank.
Bounded reach>=2 mass-run (ov_SC01_000, 15 shared <=15-ins stubs): 0/15 banked, vs the
reach-1-heavy spot-run's 7/15. The model is weakest exactly on reach>=2 (shared) code:
(1) the corpus skipped the shared DEFINE_func macro bodies (export_pairs reads only src/
defs -> 96.6% of the corpus is overlay-unique), and (2) the shared engine fns are the
harder regalloc/schedule residuals. So reach>=2 model-only is NOT a fleet lever by itself.
BUT the reach>=2 drafts are high-value FUEL: 5/15 are close<=3 reach-134 near-misses
(3x close=1: func_8012E27C/BF4C/AD64) -> x134 each if closed. The real lever is
reach>=2 draft -> permuter-grinder close (x134), which needs the SAME per-binary fix T7
applied to lora_grind: grinder.py calls run_gate with no binary (-> 077) and the backlog
stores no binary field. That two-part fix is the next step. byte-neutral: check-all 136/136.
- docs/gen2-mips-matching-model.md: T9 RESULT
- phase-ends/CURRENT_PHASE.md: T9 done; next = grinder per-binary fix, then corpus-v3
The 500-fn calibration banked 0/222 across the binary rotation. Root-caused (R14, by
reading the code + the run's own backlog — resolving a flat contradiction between two
scout agents) to TWO independent bugs in lora_grind's use of gate_stage.run_gate, NOT
model quality:
- Bug A: good_sha() passed the sha1sum line "<sha> <name>" vs harvest_verify's bare
sha1() -> 0 banks for EVERY binary incl. 077 (so the "0/12" was a bug artifact, not
an exhausted tail)
- Bug B: the gate call left src/asm/out at the hardcoded ov_SC01_077 defaults -> non-077
drafts dropped at the 077 stub-filter, silently (and the asm mis-resolution contaminated
the backlog near-miss classification)
Fix (tools/gate_stage.py): run_gate resolves src/asm/out/good_sha from `binary` when unset
(binary-agnostic, no silent ov_SC01_077 default an overlay inherits; good_sha bare-hash
normalized) + a loud negative-control guard (0-overlap binary/src mismatch warns, so a 0
can never again masquerade as 'nothing matched'). tools/lora_grind.good_sha fixed at source.
Byte-neutral: make check-all 136/136.
Proof: ov_SC01_000 spot-run banked 7/15 (47%) byte-identical (@commit:0322); reach-2
func_8017CE24 propagated x2. ROI finding: 6/7 banks are reach-1 (overlay-unique) -> broad
rotation is high bank-RATE / low fleet-% ROI; the fleet lever is reach>=2 targeting (T9) +
corpus-v3 (T8). Backlog now correctly classified (4x close=1 = grinder fuel).
- docs/gen2-mips-matching-model.md: T7 RESULT section
- phase-ends/CURRENT_PHASE.md: T7 done; next = T8 corpus-v3 / T9 reach>=2 selection
The fine-tuned 7B (bfm-match-7b-v2) drafted real OPEN ov_SC01_077 stubs; whole-binary gate banked 4
(func_80160B34 func_8015CC74 func_8016084C func_801705C0). Sample: 9/22 match_one proxy -> 4/22
whole-binary banked (18%; the proxy->gate gap is the TU-plumbing wall). Model is format-robust (raw .s
== normalized). api_draft: NORMALIZE_ASM bridge (unused — model handles raw .s) + ghidra_c-empty fix.
7B QLoRA on 638 compile-filtered pairs: trivial ≤5-ins 39/41, but ≥6 ins 0/34, >15 ins 0/24 (=stock);
near-misses far (near≈nins). Root cause: compile-filter dropped the 536 harder global/struct fns →
no non-trivial signal. Fix = corpus-v2 (externs). Cheap-cloud (Haiku/GLM) remains the working tier.
Qwen3.6-35B-A3B stock drafter: fair harness fixed compiles but model stuck at fixed near-misses
(can't refine from diff); full cookbook worse+2.3x slower than curated (dilution). Gets structure,
misses gcc-2.7.2 precision — exactly what src-pair LoRA targets. Forward: fine-tune or permuter-seed.
docs/gen2-mips-matching-model.md: the BFM/gcc-2.7.2 matching-specialist idea (LoRA on our own
gate-verified pairs — the corpus off-the-shelf RE LLMs lack). export_pairs.py mines 1307 banked
(asm<->C) pairs from build objects (asm/ is gitignored, so disasm the ROM-identical build, splat-like
format) + src defs -> datasets/match_pairs/{pairs,train,test}.jsonl (gitignored, 1174/133 split).
api_draft.py: TEMP env-tunable. .gitignore: datasets/ models/ weights.
- from-scratch giant func_801372B0 (207): structurally cracked at xHigh (206/207 ins,
logic 100%%) but byte-blocked by the gcc list-scheduler/regalloc last-mile (§27/§17);
banked the structural draft to backlog as a future permuter head-start
- xHigh finding (Drew's test): giants reach structural match fast — the wall is gcc
determinism, not reasoning depth; xHigh sufficient for this work
- session total: 16 functions ×134 (2 giants type-lift + 14 close=0), fleet 63.22%->63.66%
- docs/{worklist,backlog,progress.fleet}.md regenerated after the func_8015126C x134 bank
- CURRENT_PHASE.md: RESUME HERE block (T2 giant queue + levers + op gotchas) for a fresh session
- grinder stopped (STOP sentinel set); tree clean except R23 db churn
- cookbook §28: the 4-way triage of a close=0 gate-rejected giant (pure-extern plumbing /
masked-residual / struct-walled / regressed-draft) + the canonical-extern recovery (the NEW
lever that banked func_8015126C, which Phase-21 left as a close=0 wall) + the coalescing pin
(register __asm__ + lazy in-&& assign). §28a: PS1-applicable decomp.wiki GCC patterns
(negative-offset loops, branch-duplication, load-coalescing, div-magic table, slti-0)
- tools/recover_giant.py: canonicalize a draft's engine_core.h-callee externs to def-sigs +
move all externs block-scope (validated func_8015126C -> MATCH)
- R14: close=0-giant fan-out NOT uniformly near-free — only func_8015126C was pure-extern;
the rest are struct-walled / masked-residual / regressed -> genuine per-giant hand-work
Drew chose deep re-RE of func_8014EA4C's real frame. Finding: the frame is a DEAD aggregate
copy (8 bytes unaligned-copied to sp+0x20, never read; gcc-2.7.2 keeps it, no DSE). The agent's
buf[16]+memcpy(buf+16) PRECISELY reproduces the exact gcc stack layout (local[0x38] -> frame
0x88, wrong). So the frame is already correct at close=6; the 6 residuals are NOT frame-caused
-- they're loop-body scheduler (global-store vs call-arg-load order) + regalloc (abs vs
in-place ), the irreducible §20/§25 wall. Deep frame-RE is a dead end for schedule/regalloc
residuals -- only the permuter explores that space. 5 levers (cheap-recovery, permuter,
giant-wave, hand-finish, deep-RE) byte-proven exhausted at fleet 63.17% for the reach-134 tail.
Hand-finish attempt on func_8014EA4C (close=6, Drew's choice): 2 byte-proven attempts both
cascaded — q0_arg reorder swapped p/q regalloc (6->123); abs-ternary dropped the load-bearing
dead buf[16] frame hack -> frame 0x60->0x58 -> offset cascade (6->23). Confirms the agent: the
residuals (store/load schedule + abs regalloc) are pressure-locked by the frame-forcing hack;
C-steering breaks the frame. The fix is the decomp-permuter (grinder), not C — it explores the
frame-preserving regalloc/schedule space. Grinder is now auto-permuting func_8014EA4C +
func_80144090 (re-logged close=6/25, token-free). cookbook §27.
- 6-giant worker_wave (1.16M tokens) BANKED 0. Agents got the SHAPE (§27 recipe works) but the
whole-binary byte-gate (G3/P9) rejected all 6. func_8014F74C close=0 match_one MATCH yet its RAW
draft FAILS the byte-gate (no plumbing/data conflict) -> match_one OVER-predicted (§25 masked
residual). The rest are pure regalloc/scheduling 'not C-steerable -> permuter' (close 6/25/53/91/164).
- Confirms cont.2/Phase-20/Phase-16: giants are HAND-FINISH/permuter fuel, NOT auto-bankable.
cookbook §27 updated. Do NOT scale giant auto-waves for % (1.16M tokens -> 0 banks).
- GATE BUG (R16, noted): canon/cast/sig_unify regressed near-misses (func_8014EA4C 6->128) and
poisoned the backlog. Re-logged the 6 with TRUE closeness + raw drafts (source=giant-raw);
func_8014EA4C(6)+func_80144090(25) now grinder-eligible (token-free).
- INFLECTION (P9): the automated reach-134 harvest is COMPLETE at 63.17% — cheap recovery (0 fresh),
permuter (~0), giant wave (0/6) all exhausted. Forward % = hand-finishing or close Phase 21.
- R14: sig_image h_exact = SHA1(raw IMAGE bytes), UNMASKED (sig_image.py:170-171) -> reach is
the ACCURATE shipped-byte reach; a resolved-reach probe just reproduces it. The real gap is
reach (shipped identical) >= realizable-x134 (shared-C reproduces all N), measurable ONLY by
dedup_propagate's byte-gate on a MATCHED fn. No cheap static probe -> cont.7 'build it' WITHDRAWN.
- R14: func_801775E0 backlog close=1 but saved draft compiles 31-off from source (prologue
save-scheduling/branch-sense/block-order, the S20 wall) -> close=1 was permuter-achieved.
RULE (S26): match_one a draft for SOURCE closeness before assuming a pin crack. The close-1..4
codegen tail is permuter-class (grinder territory), not cheap pin targets.
- INFLECTION (P9): the cached reach-134 automated harvest is COMPLETE; forward %-progress needs
a token-heavy lever (8 giants / per-fn pin-cracking) -> Drew's ROI greenlight. cookbook S26.
- THE FINDING (R14/P9): the _a close=0 recovery banks 0/20 (same def-side multi-way
loose-typing wall as MAIN 0/40, cont.6). cont.6's func_8012F568 was the lone simple-arity
win; the rest need a C type that conflicts caller-decl-vs-def with no single fitting sig.
The cheap close=0 ×134 recovery lever is EXHAUSTED.
- TOOLING (R16, idiom_loop.py --assess was DOUBLY inflated, now fixed):
(1) load_backlog drops banked-since-logged fns (intersect ledger w/ live INCLUDE_ASM stub
set, mirrors backlog._matched_now) -> killed the bogus 'unknown 14' class + ~10/class
stale-matched inflation (schedule 49->39 r134, regalloc 44->39).
(2) DETERMINISTIC-RECOVERY splits FRESH (never recovery-gated = genuine fuel) vs WALLED
(recovery-failed = def-side wall, do NOT re-run). Post-fix: 0 FRESH / 46 WALLED.
- schedule wave NOT fired (ROI-gate, S20/S26): ~50% x134 fraction on n=2, median 15 ins off,
over an h_exact-inflated count. No tokens spent speculatively.
- cont.6 option-3 CONFIRMED: cached reach-134 cheap fuel is dry; the genuine levers are
token-heavy fresh-session work (the 8 giants / per-fn pin-cracking / build a resolved-reach
probe). cookbook S26. 0 banks; fleet 63.17% unchanged; 136/136 byte-identical; 0 NON_MATCHING.
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just
the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation
into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure,
isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the
survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates);
byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical).
Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3
clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe.
- worker_wave.js: derive match_one self-check --asm-subdir per-target from t.asm
(was hardcoded main subdir -> every _a agent self-check crashed on missing .s ->
blind drafting). Enabled the 16/24 close rate on the _a measurement wave.
- CURRENT_PHASE cont.4: the _a vein is x134 (10/11 propagate byte-identical), not the
cont.3/§24 x1 wall. The gate's propagated:0 was 2 bugs: dedup_propagate all-or-nothing
batch-revert poisoned by 1 stale -O0 overlay-local straggler (func_8013C360), and
gate_stage sh() swallows dedup_propagate's non-zero exit. Realized +0.39% (commit:0267).
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0);
source-find scans all, member-loop edits the right file, structural-check spans all. Single-file
overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert.
- cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file
loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged.
- FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not
difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same
§16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a
RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe.
- no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted.
- SELF-CORRECTION: a per-fn objdump diff that didn't remove .o/.elf reported 10 FALSE giant
byte-MATCHes (the §20 stale-.o trap). The real gate (harvest_verify, removes output) banks 0/10:
the MAIN giants' prior-wave drafts are STALE+INCOMPLETE vs the grown engine_core.h (conflicting/
undeclared data decls, link errors). They need RE-DRAFTING fresh, not a recovery transform.
- fixed .run/diag_funcdiff.py to os.remove artifacts first (gitignored tool); the whole-binary SHA
gate is the sole arbiter (G3/P9) — it was right while the diag lied.
- cookbook §23 rewritten: func_80153E00 (cracked, scalar-data CAST) stands; the 'giants are
near-misses' generalization corrected. sig_unify trailing-/* */-comment extern skip documented
(fix reverted — unproven + data-cast is the right lever). 11 _a.c giants need split-aware tooling.
- no new banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22).
Drew T