The grinder/backlog pipeline was ov_SC01_077-hardcoded 5 layers deep (same class as the
T7 lora_grind bug). Fixed all so the permuter grinder can process a non-077 near-miss:
1. gate_stage.append_record stores the source "binary"
2. backlog.FIELDS keeps it (else append_record dropped it)
3. backlog.load_best/_open_stubs is fleet-aware: a fn matched in ov_SC01_077 but
propagation-stuck stays OPEN in its overlay, so it surfaces via that record instead
of being dropped as "matched" (the grinder must SEE it to grind it)
4. p16_permute.setup takes the target binary's asm-subdir (was hardcoded 077)
5. grinder resolves per-binary asm + gates grouped by binary + allows unknown nins
Backward-compatible: legacy records (no binary) default ov_SC01_077.
Validated end-to-end: the 3 fresh reach-134 close=1 ov_SC01_000 fns now surface, resolve
to ov_SC01_000's asm, and gate via ov_SC01_000.
TWO byte-evidenced findings (redirect the fuel strategy):
- the reach>=2 close=1 fuel is MODEL semantic-misses, not permuter fuel: func_8012E27C's
target is "return 1" but the 7B drafted an empty "void f(void){}" (corpus overfit
empty-leaf); func_8012BF4C/AD64 are trivial sw/sh setters drafted empty. A corrected
draft banks them (+3 byte-identical via the fixed gate, @commit:0326); the permuter cannot
add a missing return/store. Lever = corpus-v3 leaf variety, not the permuter.
- x reach is propagation-capped: the 3 are inline-matched in ov_SC01_077_a.c (the stuck-
local cap) -> dedup_propagate "nothing to propagate" -> banked x1. Lever = dedup-collapse.
check-all 136/136 throughout. docs/gen2-mips-matching-model.md + CURRENT_PHASE updated.
Add a sig-based reach oracle + --min-reach N to lora_grind so the mass-run can prefer
SHARED functions (one bank propagates x reach — the fleet-% multiplier). The oracle reads
the same .run/sig.ov_*.jsonl dedup_propagate uses (validated: 0 mismatch over 60 stubs +
the func_8017CE24=2 ground truth), so a reach>=N target is exactly one dedup_propagate
will stamp x reach after the bank.
Bounded reach>=2 mass-run (ov_SC01_000, 15 shared <=15-ins stubs): 0/15 banked, vs the
reach-1-heavy spot-run's 7/15. The model is weakest exactly on reach>=2 (shared) code:
(1) the corpus skipped the shared DEFINE_func macro bodies (export_pairs reads only src/
defs -> 96.6% of the corpus is overlay-unique), and (2) the shared engine fns are the
harder regalloc/schedule residuals. So reach>=2 model-only is NOT a fleet lever by itself.
BUT the reach>=2 drafts are high-value FUEL: 5/15 are close<=3 reach-134 near-misses
(3x close=1: func_8012E27C/BF4C/AD64) -> x134 each if closed. The real lever is
reach>=2 draft -> permuter-grinder close (x134), which needs the SAME per-binary fix T7
applied to lora_grind: grinder.py calls run_gate with no binary (-> 077) and the backlog
stores no binary field. That two-part fix is the next step. byte-neutral: check-all 136/136.
- docs/gen2-mips-matching-model.md: T9 RESULT
- phase-ends/CURRENT_PHASE.md: T9 done; next = grinder per-binary fix, then corpus-v3
The 500-fn calibration banked 0/222 across the binary rotation. Root-caused (R14, by
reading the code + the run's own backlog — resolving a flat contradiction between two
scout agents) to TWO independent bugs in lora_grind's use of gate_stage.run_gate, NOT
model quality:
- Bug A: good_sha() passed the sha1sum line "<sha> <name>" vs harvest_verify's bare
sha1() -> 0 banks for EVERY binary incl. 077 (so the "0/12" was a bug artifact, not
an exhausted tail)
- Bug B: the gate call left src/asm/out at the hardcoded ov_SC01_077 defaults -> non-077
drafts dropped at the 077 stub-filter, silently (and the asm mis-resolution contaminated
the backlog near-miss classification)
Fix (tools/gate_stage.py): run_gate resolves src/asm/out/good_sha from `binary` when unset
(binary-agnostic, no silent ov_SC01_077 default an overlay inherits; good_sha bare-hash
normalized) + a loud negative-control guard (0-overlap binary/src mismatch warns, so a 0
can never again masquerade as 'nothing matched'). tools/lora_grind.good_sha fixed at source.
Byte-neutral: make check-all 136/136.
Proof: ov_SC01_000 spot-run banked 7/15 (47%) byte-identical (@commit:0322); reach-2
func_8017CE24 propagated x2. ROI finding: 6/7 banks are reach-1 (overlay-unique) -> broad
rotation is high bank-RATE / low fleet-% ROI; the fleet lever is reach>=2 targeting (T9) +
corpus-v3 (T8). Backlog now correctly classified (4x close=1 = grinder fuel).
- docs/gen2-mips-matching-model.md: T7 RESULT section
- phase-ends/CURRENT_PHASE.md: T7 done; next = T8 corpus-v3 / T9 reach>=2 selection
- from-scratch giant func_801372B0 (207): structurally cracked at xHigh (206/207 ins,
logic 100%%) but byte-blocked by the gcc list-scheduler/regalloc last-mile (§27/§17);
banked the structural draft to backlog as a future permuter head-start
- xHigh finding (Drew's test): giants reach structural match fast — the wall is gcc
determinism, not reasoning depth; xHigh sufficient for this work
- session total: 16 functions ×134 (2 giants type-lift + 14 close=0), fleet 63.22%->63.66%
- docs/{worklist,backlog,progress.fleet}.md regenerated after the func_8015126C x134 bank
- CURRENT_PHASE.md: RESUME HERE block (T2 giant queue + levers + op gotchas) for a fresh session
- grinder stopped (STOP sentinel set); tree clean except R23 db churn
- cookbook §28: the 4-way triage of a close=0 gate-rejected giant (pure-extern plumbing /
masked-residual / struct-walled / regressed-draft) + the canonical-extern recovery (the NEW
lever that banked func_8015126C, which Phase-21 left as a close=0 wall) + the coalescing pin
(register __asm__ + lazy in-&& assign). §28a: PS1-applicable decomp.wiki GCC patterns
(negative-offset loops, branch-duplication, load-coalescing, div-magic table, slti-0)
- tools/recover_giant.py: canonicalize a draft's engine_core.h-callee externs to def-sigs +
move all externs block-scope (validated func_8015126C -> MATCH)
- R14: close=0-giant fan-out NOT uniformly near-free — only func_8015126C was pure-extern;
the rest are struct-walled / masked-residual / regressed -> genuine per-giant hand-work
Drew clarified the neverending loop's PURPOSE: learn idioms. Out of fuel -> analyze ALL
remaining, group by decomp issue, 1 idiom/group, cookbook, wave all applicable, bank, repeat.
ANSWER: this IS the idiom_loop flywheel (--assess groups by gcc-quirk class -> idiom -> distill
-> gen-fuel -> worker_wave -> gate -> repeat); no plan-mode build needed. ONE gap: --assess
clusters the BACKLOG (drafted near-misses), not ALL remaining stubs -> close it with TRIAGE waves
on un-drafted pools (reach-1 next, 281 fns). Effort already right (Max orchestration + Workflow
per-agent xHigh). Recommend a FRESH session (context hygiene); optional clean milestone = close
Phase 21 (reach-134 harvest byte-proven complete) -> Phase 22 idiom-loop at scale. Drew's call.
- 6-giant worker_wave (1.16M tokens) BANKED 0. Agents got the SHAPE (§27 recipe works) but the
whole-binary byte-gate (G3/P9) rejected all 6. func_8014F74C close=0 match_one MATCH yet its RAW
draft FAILS the byte-gate (no plumbing/data conflict) -> match_one OVER-predicted (§25 masked
residual). The rest are pure regalloc/scheduling 'not C-steerable -> permuter' (close 6/25/53/91/164).
- Confirms cont.2/Phase-20/Phase-16: giants are HAND-FINISH/permuter fuel, NOT auto-bankable.
cookbook §27 updated. Do NOT scale giant auto-waves for % (1.16M tokens -> 0 banks).
- GATE BUG (R16, noted): canon/cast/sig_unify regressed near-misses (func_8014EA4C 6->128) and
poisoned the backlog. Re-logged the 6 with TRUE closeness + raw drafts (source=giant-raw);
func_8014EA4C(6)+func_80144090(25) now grinder-eligible (token-free).
- INFLECTION (P9): the automated reach-134 harvest is COMPLETE at 63.17% — cheap recovery (0 fresh),
permuter (~0), giant wave (0/6) all exhausted. Forward % = hand-finishing or close Phase 21.
- R14: sig_image h_exact = SHA1(raw IMAGE bytes), UNMASKED (sig_image.py:170-171) -> reach is
the ACCURATE shipped-byte reach; a resolved-reach probe just reproduces it. The real gap is
reach (shipped identical) >= realizable-x134 (shared-C reproduces all N), measurable ONLY by
dedup_propagate's byte-gate on a MATCHED fn. No cheap static probe -> cont.7 'build it' WITHDRAWN.
- R14: func_801775E0 backlog close=1 but saved draft compiles 31-off from source (prologue
save-scheduling/branch-sense/block-order, the S20 wall) -> close=1 was permuter-achieved.
RULE (S26): match_one a draft for SOURCE closeness before assuming a pin crack. The close-1..4
codegen tail is permuter-class (grinder territory), not cheap pin targets.
- INFLECTION (P9): the cached reach-134 automated harvest is COMPLETE; forward %-progress needs
a token-heavy lever (8 giants / per-fn pin-cracking) -> Drew's ROI greenlight. cookbook S26.
- THE FINDING (R14/P9): the _a close=0 recovery banks 0/20 (same def-side multi-way
loose-typing wall as MAIN 0/40, cont.6). cont.6's func_8012F568 was the lone simple-arity
win; the rest need a C type that conflicts caller-decl-vs-def with no single fitting sig.
The cheap close=0 ×134 recovery lever is EXHAUSTED.
- TOOLING (R16, idiom_loop.py --assess was DOUBLY inflated, now fixed):
(1) load_backlog drops banked-since-logged fns (intersect ledger w/ live INCLUDE_ASM stub
set, mirrors backlog._matched_now) -> killed the bogus 'unknown 14' class + ~10/class
stale-matched inflation (schedule 49->39 r134, regalloc 44->39).
(2) DETERMINISTIC-RECOVERY splits FRESH (never recovery-gated = genuine fuel) vs WALLED
(recovery-failed = def-side wall, do NOT re-run). Post-fix: 0 FRESH / 46 WALLED.
- schedule wave NOT fired (ROI-gate, S20/S26): ~50% x134 fraction on n=2, median 15 ins off,
over an h_exact-inflated count. No tokens spent speculatively.
- cont.6 option-3 CONFIRMED: cached reach-134 cheap fuel is dry; the genuine levers are
token-heavy fresh-session work (the 8 giants / per-fn pin-cracking / build a resolved-reach
probe). cookbook S26. 0 banks; fleet 63.17% unchanged; 136/136 byte-identical; 0 NON_MATCHING.
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just
the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation
into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure,
isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the
survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates);
byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical).
Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3
clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe.
- worker_wave.js: derive match_one self-check --asm-subdir per-target from t.asm
(was hardcoded main subdir -> every _a agent self-check crashed on missing .s ->
blind drafting). Enabled the 16/24 close rate on the _a measurement wave.
- CURRENT_PHASE cont.4: the _a vein is x134 (10/11 propagate byte-identical), not the
cont.3/§24 x1 wall. The gate's propagated:0 was 2 bugs: dedup_propagate all-or-nothing
batch-revert poisoned by 1 stale -O0 overlay-local straggler (func_8013C360), and
gate_stage sh() swallows dedup_propagate's non-zero exit. Realized +0.39% (commit:0267).
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0);
source-find scans all, member-loop edits the right file, structural-check spans all. Single-file
overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert.
- cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file
loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged.
- FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not
difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same
§16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a
RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe.
- no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted.
- SELF-CORRECTION: a per-fn objdump diff that didn't remove .o/.elf reported 10 FALSE giant
byte-MATCHes (the §20 stale-.o trap). The real gate (harvest_verify, removes output) banks 0/10:
the MAIN giants' prior-wave drafts are STALE+INCOMPLETE vs the grown engine_core.h (conflicting/
undeclared data decls, link errors). They need RE-DRAFTING fresh, not a recovery transform.
- fixed .run/diag_funcdiff.py to os.remove artifacts first (gitignored tool); the whole-binary SHA
gate is the sole arbiter (G3/P9) — it was right while the diag lied.
- cookbook §23 rewritten: func_80153E00 (cracked, scalar-data CAST) stands; the 'giants are
near-misses' generalization corrected. sig_unify trailing-/* */-comment extern skip documented
(fix reverted — unproven + data-cast is the right lever). 11 _a.c giants need split-aware tooling.
- no new banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22).
- self-MATCH-but-gate-rejected fns can never bank via re-drafting (the drafter reproduces the
byte-correct body the gate rejects again) -> exclude from POOL selection so agents draft FRESH
targets; they stay in the backlog as recovery-tooling/hand-finish fuel
- CURRENT_PHASE: sustained-loop state + pool-mode override policy (compaction-recovery anchor)
- grinder.py: persistent blacklist for permuter-won/gate-rejected (plumbing-bound) fns;
stops the tried.clear()-after-idle churn (was 0 banks in 8h on func_8014F3E8/func_8014FE60)
- sig_unify.py rewrite_def: adopt the canonical param list on arity mismatch (DEF-side
loose-typing wall) — unused extras sit in $a0-$a3, free at -O2; banked func_8016EDEC/EE40
- cookbook §22 (DEF-side recovery + grinder blacklist); CURRENT_PHASE option-C log
- compounds: sig_unify is in the gate pipeline -> every future wave + the grinder auto-recover
- the %lo-folding indexed-global -O0 residual (§18, long "irreducible") is CRACKED: declare
the global as "extern Struct base[]" (sizeof == stride) and access "base[i].field" -> gcc
keeps sym a symbol_ref and folds %lo(sym+off) into the store, byte-matching. The failing
forms "*(T*)(&sym + i*stride)" materialize &sym as a value first, blocking the fold.
- func_8013B7AC byte-gated + banked (x1 — the -O0 cluster is overlay-local, R14 finding above);
the idiom is reusable fleet-wide for ANY indexed-global access (-O0 or -O2).
- cookbook §18 ("%lo-folding indexed global — CRACKED") + §20 placeholder closed.
- the other 9 -O0 stubs: idiom applies but need per-fn -O0 drafting (3 quick ones tried, failed
on -O0 codegen detail; 4 are large) -> backlog/autonomy fuel. clean rebuild d19c9580 (R22).
- byte-proven: only 1/134 overlays shares ov_SC01_077's -O0 cluster bytes; the 6 matched
-O0 fns reference per-overlay data addresses (func_8013B568 -> D_80187270 in ov_SC01_077
but 0x80182B04 in ov_SC01_005), so they're overlay-LOCAL, not ×134-shareable. The
Phase-18/19 'reach-134' label conflated function-present with byte-identical (R14).
- the §18 split infra (rollout tool + Makefile %_o0.o pattern + PROVIDE .ld) was built and
byte-validated to compile/link a 2nd overlay; the whole-binary gate then exposed the
per-overlay-data wall -> reverted (the finding is the deliverable, P9).
- -O0 rollout = ×1 per-overlay-unique re-matching (~0.2-0.3%), NOT +0.6% free -> SKIP.
cookbook §18 corrected; fleet unchanged 58.82%, 136/136 byte-identical.
Drew T