Commit Graph

108 Commits

Author SHA1 Message Date
Drew T 066a2e5c8f feat(phase-23): local GPU serving + prompt-fix + corpus-v3 (macro+struct mining)
LM Studio was ejected, so serving is ours now: tools/serve_local.py serves base+LoRA via
Unsloth (.venv-train cu128) as an OpenAI endpoint — no llama.cpp build (this CPU has no
AVX-512, which SIGILLs the prebuilt llama-cpp-python CUDA wheels). api_draft/lora_grind hit
it unchanged.

PROMPT FIX (api_draft.LEAN_SYS + format_finetune.SYS, kept in sync): 'translate EVERY
instruction, never an empty body' — a prompt test took the small-leaf band 0/3 -> 2/3 MATCH
(the v2 corpus overfit an empty void f(void){} leaf pattern). Validated end-to-end: a fresh
ov_SC01_001 batch banked 3 via the local server + better prompt.

CORPUS-V3 (export_pairs + format_finetune):
- export_pairs now ALSO mines the 1623 DEFINE_func macro bodies in engine_core.h (the shared
  setters/return-const/dispatchers extract_defs never saw -> 96.6% of v2 was overlay-unique,
  the root of the empty-leaf overfit). Corpus 1312 inline -> 2891 (1312 inline + 1579 macros).
- format_finetune inlines engine_types.h structs in the compile-filter so struct-using bodies
  are KEPT not dropped: train 2534/2591 (97.8%) compile standalone (v2 was 1111 total).
2026-06-30 01:46:54 -06:00
Drew T e91859fb4a feat(phase-23): grinder per-binary fix (5-layer) — unlock non-077 near-miss grinding
The grinder/backlog pipeline was ov_SC01_077-hardcoded 5 layers deep (same class as the
T7 lora_grind bug). Fixed all so the permuter grinder can process a non-077 near-miss:
1. gate_stage.append_record stores the source "binary"
2. backlog.FIELDS keeps it (else append_record dropped it)
3. backlog.load_best/_open_stubs is fleet-aware: a fn matched in ov_SC01_077 but
   propagation-stuck stays OPEN in its overlay, so it surfaces via that record instead
   of being dropped as "matched" (the grinder must SEE it to grind it)
4. p16_permute.setup takes the target binary's asm-subdir (was hardcoded 077)
5. grinder resolves per-binary asm + gates grouped by binary + allows unknown nins
Backward-compatible: legacy records (no binary) default ov_SC01_077.

Validated end-to-end: the 3 fresh reach-134 close=1 ov_SC01_000 fns now surface, resolve
to ov_SC01_000's asm, and gate via ov_SC01_000.

TWO byte-evidenced findings (redirect the fuel strategy):
- the reach>=2 close=1 fuel is MODEL semantic-misses, not permuter fuel: func_8012E27C's
  target is "return 1" but the 7B drafted an empty "void f(void){}" (corpus overfit
  empty-leaf); func_8012BF4C/AD64 are trivial sw/sh setters drafted empty. A corrected
  draft banks them (+3 byte-identical via the fixed gate, @commit:0326); the permuter cannot
  add a missing return/store. Lever = corpus-v3 leaf variety, not the permuter.
- x reach is propagation-capped: the 3 are inline-matched in ov_SC01_077_a.c (the stuck-
  local cap) -> dedup_propagate "nothing to propagate" -> banked x1. Lever = dedup-collapse.

check-all 136/136 throughout. docs/gen2-mips-matching-model.md + CURRENT_PHASE updated.
2026-06-30 00:58:13 -06:00
Drew T 366f7e4c7f feat(phase-23): T9 — reach>=2 targeting (lora_grind --min-reach); shared-code is the model's weak band
Add a sig-based reach oracle + --min-reach N to lora_grind so the mass-run can prefer
SHARED functions (one bank propagates x reach — the fleet-% multiplier). The oracle reads
the same .run/sig.ov_*.jsonl dedup_propagate uses (validated: 0 mismatch over 60 stubs +
the func_8017CE24=2 ground truth), so a reach>=N target is exactly one dedup_propagate
will stamp x reach after the bank.

Bounded reach>=2 mass-run (ov_SC01_000, 15 shared <=15-ins stubs): 0/15 banked, vs the
reach-1-heavy spot-run's 7/15. The model is weakest exactly on reach>=2 (shared) code:
(1) the corpus skipped the shared DEFINE_func macro bodies (export_pairs reads only src/
defs -> 96.6% of the corpus is overlay-unique), and (2) the shared engine fns are the
harder regalloc/schedule residuals. So reach>=2 model-only is NOT a fleet lever by itself.

BUT the reach>=2 drafts are high-value FUEL: 5/15 are close<=3 reach-134 near-misses
(3x close=1: func_8012E27C/BF4C/AD64) -> x134 each if closed. The real lever is
reach>=2 draft -> permuter-grinder close (x134), which needs the SAME per-binary fix T7
applied to lora_grind: grinder.py calls run_gate with no binary (-> 077) and the backlog
stores no binary field. That two-part fix is the next step. byte-neutral: check-all 136/136.

- docs/gen2-mips-matching-model.md: T9 RESULT
- phase-ends/CURRENT_PHASE.md: T9 done; next = grinder per-binary fix, then corpus-v3
2026-06-30 00:12:45 -06:00
Drew T bff03dbdb5 fix(phase-23): T7 — lora_grind broad-rotation banking (two gate bugs); ov_SC01_000 7/15
The 500-fn calibration banked 0/222 across the binary rotation. Root-caused (R14, by
reading the code + the run's own backlog — resolving a flat contradiction between two
scout agents) to TWO independent bugs in lora_grind's use of gate_stage.run_gate, NOT
model quality:
- Bug A: good_sha() passed the sha1sum line "<sha>  <name>" vs harvest_verify's bare
  sha1() -> 0 banks for EVERY binary incl. 077 (so the "0/12" was a bug artifact, not
  an exhausted tail)
- Bug B: the gate call left src/asm/out at the hardcoded ov_SC01_077 defaults -> non-077
  drafts dropped at the 077 stub-filter, silently (and the asm mis-resolution contaminated
  the backlog near-miss classification)

Fix (tools/gate_stage.py): run_gate resolves src/asm/out/good_sha from `binary` when unset
(binary-agnostic, no silent ov_SC01_077 default an overlay inherits; good_sha bare-hash
normalized) + a loud negative-control guard (0-overlap binary/src mismatch warns, so a 0
can never again masquerade as 'nothing matched'). tools/lora_grind.good_sha fixed at source.
Byte-neutral: make check-all 136/136.

Proof: ov_SC01_000 spot-run banked 7/15 (47%) byte-identical (@commit:0322); reach-2
func_8017CE24 propagated x2. ROI finding: 6/7 banks are reach-1 (overlay-unique) -> broad
rotation is high bank-RATE / low fleet-% ROI; the fleet lever is reach>=2 targeting (T9) +
corpus-v3 (T8). Backlog now correctly classified (4x close=1 = grinder fuel).

- docs/gen2-mips-matching-model.md: T7 RESULT section
- phase-ends/CURRENT_PHASE.md: T7 done; next = T8 corpus-v3 / T9 reach>=2 selection
2026-06-29 23:44:12 -06:00
Drew T edd62ef992 feat(phase-22): lora_grind.py — the free local-model mass-run (draft source for the grinder flywheel)
Rotates over every binary with open small stubs, drafts with the fine-tuned model (api_draft LEAN, GPU
via LM Studio), banks via gate_stage, propagates fleet-wide periodically. Near-misses -> backlog ->
grinder.py permuter closes regalloc/schedule residuals (synergy). Writes a classified near-miss
histogram (the 'missing idioms' signal -> corpus-v3 priorities). STOP/heartbeat/stats like grinder.py.
Smoke OK; histogram shows struct(49)+schedule/regalloc(94) dominate the backlog.
2026-06-29 22:10:05 -06:00
Drew T b4c312a30c feat(phase-22): FIRST free-local-model banks — 4 open stubs matched by the v2 LoRA
The fine-tuned 7B (bfm-match-7b-v2) drafted real OPEN ov_SC01_077 stubs; whole-binary gate banked 4
(func_80160B34 func_8015CC74 func_8016084C func_801705C0). Sample: 9/22 match_one proxy -> 4/22
whole-binary banked (18%; the proxy->gate gap is the TU-plumbing wall). Model is format-robust (raw .s
== normalized). api_draft: NORMALIZE_ASM bridge (unused — model handles raw .s) + ghidra_c-empty fix.
2026-06-29 21:58:09 -06:00
Drew T 748dece7fc feat(phase-22): train_lora auto-cleans GGUF intermediates (the ~30GB that crashed C:) 2026-06-29 20:07:48 -06:00
Drew T 8537706006 feat(phase-22): corpus-v2 — capture the extern block above each def (self-contained completions)
The pilot failed because the compile-filter dropped the 536 hard (global/struct) fns: bare defs
reference undeclared D_xxxx globals -> cc1 error. The src declares them as 'extern <type> D_xxx;'
immediately ABOVE each def with correct byte-verified types. Capturing that contiguous extern block
lifts standalone-compile 52%->92% (16/40 hard fns recovered) AND teaches the model to declare globals
with the right types (a real matching skill). Fixes the trivial-bias that capped the 7B pilot.
2026-06-29 20:03:53 -06:00
Drew T 803be6341c feat(phase-22): eval_lora held-out scorer (validated) + train_lora --max-steps
eval_lora.py: feeds a fine-tuned model the LEAN asm of held-out BANKED test fns, compiles its C,
compares to the corpus target reloc-masked (banked fns have no .s — corpus carries the asm). Validated:
known-good corpus C scores MATCH vs its own asm. train_lora --max-steps for smoke runs. GPU confirmed
(3080 Ti, torch 2.10+cu128); install + corpus-filter (638/75 clean pairs) done; smoke-train running.
2026-06-29 16:49:11 -06:00
Drew T dfd97226cc feat(phase-22): format_finetune compile-filter — keep only standalone-compilable pairs
Found 50% of bare-def corpus completions fail standalone compile (undeclared globals/structs — externs
lived in TU headers). Filtering to compile-verified pairs so the pilot LoRA learns to emit COMPILABLE
matching C, not uncompilable bodies. (Drops harder struct/global fns; corpus v2 = add externs to recover.)
2026-06-29 16:43:19 -06:00
Drew T 1c80b25ac6 feat(phase-22): LoRA specialist pilot tooling — format_finetune + train_lora + api_draft LEAN
format_finetune.py: corpus -> Qwen chat-template instruction JSONL (lean asm->C, 1174 train/133 test).
train_lora.py: Unsloth QLoRA recipe (Qwen2.5-Coder-7B 4-bit, fits 12GB; merges to GGUF for LM Studio).
api_draft.py LEAN=1: ~350-tok asm-only prompt matching the training shape (fine-tuned model needs no
inlined cookbook -> fast). Datasets gitignored. Eval: beat stock-local ~0 on the held-out frozen 20.
2026-06-29 16:35:06 -06:00
Drew T 4895843dda feat(phase-22): api_draft fair-harness — inline common.h + live cookbook + corpus examples
Gives the no-tool local model the context the agents read: common.h, live matching cookbook
(COOKBOOK_FULL toggle), 2 byte-matched corpus examples. Qwen3.6-35B-A3B result: harness fixes
compile-fails but model stays stuck at fixed near-misses; FULL cookbook worse+2.3x slower than
curated (dilution, gate-confirmed). Bottleneck is model refinement, not context.
2026-06-29 16:24:37 -06:00
Drew T ad535ad1ab feat(phase-22): gen2 matching-model design + export_pairs.py corpus miner
docs/gen2-mips-matching-model.md: the BFM/gcc-2.7.2 matching-specialist idea (LoRA on our own
gate-verified pairs — the corpus off-the-shelf RE LLMs lack). export_pairs.py mines 1307 banked
(asm<->C) pairs from build objects (asm/ is gitignored, so disasm the ROM-identical build, splat-like
format) + src defs -> datasets/match_pairs/{pairs,train,test}.jsonl (gitignored, 1174/133 split).
api_draft.py: TEMP env-tunable. .gitignore: datasets/ models/ weights.
2026-06-29 14:27:19 -06:00
Drew T 3549f756f7 feat(phase-22): api_draft.py — provider-agnostic draft worker (local/OpenRouter)
Script equivalent of one worker_wave drafter for any OpenAI-compatible endpoint (LM Studio /
llama.cpp / vLLM / OpenRouter). Inlines asm+ghidra_c+toolkit, calls /chat/completions, extracts C,
iterates against match_one (diff fed back, keep best). Output scores as a 'local'/'glm' arm via
ab_score.py. Env: API_BASE/API_KEY/MODEL. Logic self-tested; HTTP is standard OpenAI format.
2026-06-29 13:05:18 -06:00
Drew T f154822f7b feat(phase-22): cheap-tier A/B experiment harness + findings
ab_match.js (per-arm-model fan-out fork of worker_wave) + ab_score.py (disk-truth
match_one scorer, --measured for real $). Haiku drafters vs Opus on 20 reach1 fns:
proxy 10 vs 15 matches, Haiku ~4.8x more matches/$, parity on <=52-ins band. Whole-binary
gate banked 4 (proxy over-counts; TU-plumbing wall). backlog/fleet refreshed.
2026-06-29 12:52:39 -06:00
Drew T 0375cd529b docs(phase-22): T3 — distill §28 canonical-extern recovery + recover_giant.py + decomp.wiki patterns
- cookbook §28: the 4-way triage of a close=0 gate-rejected giant (pure-extern plumbing /
  masked-residual / struct-walled / regressed-draft) + the canonical-extern recovery (the NEW
  lever that banked func_8015126C, which Phase-21 left as a close=0 wall) + the coalescing pin
  (register __asm__ + lazy in-&& assign). §28a: PS1-applicable decomp.wiki GCC patterns
  (negative-offset loops, branch-duplication, load-coalescing, div-magic table, slti-0)
- tools/recover_giant.py: canonicalize a draft's engine_core.h-callee externs to def-sigs +
  move all externs block-scope (validated func_8015126C -> MATCH)
- R14: close=0-giant fan-out NOT uniformly near-free — only func_8015126C was pure-extern;
  the rest are struct-walled / masked-residual / regressed -> genuine per-giant hand-work
2026-06-26 03:06:54 -06:00
Drew T 013806b075 feat(phase-22): T0 — unified byte-weighted ranked worklist (the loop's decision spine)
- tools/worklist.py: join fuel_manifest (class/reach/nins/leverage/cached/region)
  + backlog near-misses (closeness/where_stuck/best_draft) -> docs/worklist.md +
  .run/worklist.json, ranked by byte-weighted gain (gain_ins = reach*nins),
  grouped by idiom class, with %rem + cum% columns; --refresh re-runs
  build_fuel_manifest + backlog.render for one-command loop use
- docs/worklist.md: 451 live stubs, 1.85M ins remaining gain; GIANT queue =
  23 fns = 37.2% of remaining gain; surfaces the idiom sub-groups (close=0
  plumbing-wall / won't-compile loose-typing / regalloc-coalescing / irreducible)
- phase-ends/CURRENT_PHASE.md: Phase-22 plan + per-task crash-recovery log
- no source touched -> byte-locked binaries unaffected (no build needed)
2026-06-26 02:07:03 -06:00
Drew T 62fa026f7e feat(phase-21): worker_wave — GIANT-guarded §27 hint (arg-arity decls, sibling templates, coalescing residual) 2026-06-25 16:46:35 -06:00
Drew T cf23808d5c feat(phase-21): cont.7 — cheap close=0 recovery EXHAUSTED (byte-proven); fix --assess double inflation
- THE FINDING (R14/P9): the _a close=0 recovery banks 0/20 (same def-side multi-way
  loose-typing wall as MAIN 0/40, cont.6). cont.6's func_8012F568 was the lone simple-arity
  win; the rest need a C type that conflicts caller-decl-vs-def with no single fitting sig.
  The cheap close=0 ×134 recovery lever is EXHAUSTED.
- TOOLING (R16, idiom_loop.py --assess was DOUBLY inflated, now fixed):
  (1) load_backlog drops banked-since-logged fns (intersect ledger w/ live INCLUDE_ASM stub
      set, mirrors backlog._matched_now) -> killed the bogus 'unknown 14' class + ~10/class
      stale-matched inflation (schedule 49->39 r134, regalloc 44->39).
  (2) DETERMINISTIC-RECOVERY splits FRESH (never recovery-gated = genuine fuel) vs WALLED
      (recovery-failed = def-side wall, do NOT re-run). Post-fix: 0 FRESH / 46 WALLED.
- schedule wave NOT fired (ROI-gate, S20/S26): ~50% x134 fraction on n=2, median 15 ins off,
  over an h_exact-inflated count. No tokens spent speculatively.
- cont.6 option-3 CONFIRMED: cached reach-134 cheap fuel is dry; the genuine levers are
  token-heavy fresh-session work (the 8 giants / per-fn pin-cracking / build a resolved-reach
  probe). cookbook S26. 0 banks; fleet 63.17% unchanged; 136/136 byte-identical; 0 NON_MATCHING.
2026-06-25 13:09:02 -06:00
Drew T 790326057a feat(phase-21): cont.6 — sig_unify --src-file + gate two-stage; schedule idiom = coalescing (§25)
- sig_unify.py --src-file: read cur_stubs/decls from the SPLIT .c (_a/_o0) so split-file
  drafts aren't dropped + get the def-side recovery (closes the §24 gap). gate_stage passes
  it to cast + sig_unify in --src-file mode. Banked func_8012F568 x134 earlier (commit:0277).
- gate_stage.py CANON-FIRST two-stage (§19 folded into the shared gate): stage1 canon+cast
  gate (already-correct/hand-pinned cracks bank), stage2 sig_unify ONLY the failures (def-side
  recovery, no regression). Fixes sig_unify regressing the func_80128ED8 crack.
- schedule idiom CRACKED (cookbook §25): func_80128ED8 = gcc copy-COALESCING, not scheduling;
  2 register pins (idc->$a3 forces the multiply onto $v1; idc+1->dead $v0 temp) -> MATCH,
  banked x1 (commit:0278). + the rank_for_schedule tie-break (priority->class->LUID source-order)
  + the genuine walls (§10 cross-jump/delay-slot, store-vs-load).
- FINDING (R14/P9): h_exact OVER-COUNTS x134 (relocation-masked) -> func_80128ED8 is x1 not
  x134; probe dedup_propagate --addr before a token-heavy wave. schedule is NOT a confirmed
  x134 vein -> the wave is ROI-deferred.
- check-all 136/136 byte-identical (R22); 0 NON_MATCHING (G4); fleet 63.13% -> 63.17%
2026-06-25 01:17:12 -06:00
Drew T a79aa410bd feat(phase-21): idiom_loop.py — the neverending idiom-learning meta-loop (wall->assess->fuel)
Drew's directive: Phase 21 is neverending (harvest easy fuel -> learn next gcc-quirk idiom ->
fresh session -> repeat, toward 100% decomp). idiom_loop.py: --fuel-status (wall), --assess
(cluster backlog by quirk class, rank by reach-134 leverage, name next idiom + cheap recovery
lever), --gen-fuel (research+harvest manifest). First assessment: next idiom = SCHEDULE (49
reach-134, sched.c tie-break); +64 close=0 reach-134 fns bankable x134 for ~0 tokens (def-side
recovery). CURRENT_PHASE cont.5 = the neverending cycle + the /loop prompt for the fresh session.
2026-06-24 22:59:04 -06:00
Drew T 5e5423e953 feat(phase-21): dedup_propagate drop-straggler retry — _a pipeline now hands-free
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just
the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation
into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure,
isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the
survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates);
byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical).

Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3
clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe.
2026-06-24 22:11:27 -06:00
Drew T 04c5c9b47f fix(phase-21): dedup_propagate skips -O0 overlay-local fns + gate_stage surfaces prop failures
The cont.4 root-cause fixes so future _a waves auto-realize ×134:
- dedup_propagate --auto-from now excludes *_o0.c (-O0) defs: -O0 codegen embeds per-overlay
  %lo data, so masked h_exact falsely reports reach-134 (§18/§20). One such straggler
  (func_8013C360) reverted 10 clean ×134 matches under the all-or-nothing batch gate (cont.4).
  Detected via find_site on the -O0 split file; --addr still forces them. (6 fns excluded.)
- gate_stage: capture dedup_propagate's exit; on a real byte-gate revert (not the benign
  'nothing to propagate' no-op) write .run/auto/last_propagate_error.log + add prop_error to
  the summary + warn. A swallowed revert previously hid the gain silently.
2026-06-24 21:32:05 -06:00
Drew T e30697a081 docs(phase-21): _a vein is a x134 lever (10/11) — worker_wave region-fix + cont.4 findings
- worker_wave.js: derive match_one self-check --asm-subdir per-target from t.asm
  (was hardcoded main subdir -> every _a agent self-check crashed on missing .s ->
  blind drafting). Enabled the 16/24 close rate on the _a measurement wave.
- CURRENT_PHASE cont.4: the _a vein is x134 (10/11 propagate byte-identical), not the
  cont.3/§24 x1 wall. The gate's propagated:0 was 2 bugs: dedup_propagate all-or-nothing
  batch-revert poisoned by 1 stale -O0 overlay-local straggler (func_8013C360), and
  gate_stage sh() swallows dedup_propagate's non-zero exit. Realized +0.39% (commit:0267).
2026-06-24 20:37:57 -06:00
Drew T 91430e9309 feat(phase-21): _a wave is turnkey — gate_stage --src-file + wave_targets region-aware asm
- gate_stage.py --src-file: cast canonicalizes vs the SPLIT .c's decls + sig_unify SKIPPED (it reads
  main .c only -> drops _a/_o0 drafts). Validated end-to-end: gate_stage _a-mode banked func_8012C098.
- wave_targets.py region-aware asm paths (REGION_SUB/asm_for): --region a serves the 68 fresh _a
  reach-134 fns with correct ..._a .s paths (pool + --class modes).
- the _a wave is now fully wired (wave_targets --region a -> worker_wave -> gate_stage --src-file ->
  dedup_propagate split-aware). Turnkey runbook in CURRENT_PHASE.md (cont.3) for a fresh session.
- no net banks; fleet 62.31%; ov_SC01_077 d19c9580 (R22).
2026-06-24 19:35:17 -06:00
Drew T cbf7e774ab feat(phase-21): split-aware dedup_propagate + cast_call_sites --src-file (unlock the _a.c vein)
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0);
  source-find scans all, member-loop edits the right file, structural-check spans all. Single-file
  overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert.
- cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file
  loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged.
- FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not
  difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same
  §16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a
  RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe.
- no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted.
2026-06-24 13:11:56 -06:00
Drew T 90b94971cd fix(phase-21): --class mode filters to still-unbanked genuine near-misses
--class selected ALL backlog near records of a class, incl. already-banked fns (stale
near records, no live_stubs check) and closeness==0 plumbing-blocked fns (re-draft cannot
bank) -> ~18 of 24 wasted slots. Now filters to: name in live_stubs() AND closeness>0.
Gives clean reach-134 wall fuel for worker --class re-attempts (grinder owns REGALLOC/SCHEDULE).
2026-06-23 12:46:39 -06:00
Drew T 39164b7b6b fix(phase-21): pool excludes re-drafted-twice near-miss walls
Smallest-first reach1 pool re-served close>0 near-misses at the front of every wave
(fixed small nins), but permuter-class walls (schedule LUID-tie / regalloc-coalesce /
iv-combine) never bank via blind re-draft -> func_80140E6C near-missed at close=4 in BOTH
wave 24 & 25, a slow leak that compounds as the band climbs.

reserved_walls(min_attempts=2): a near-miss re-drafted >=2x without banking is a wall (the
crackable ones crack on attempt 2 — func_801651B8/801549F8/80153D7C did). Skip from blind
waves; they stay in the backlog for the grinder (permuter) or a targeted --class re-attempt
(prior stuck-point fed in). 66 such walls now excluded. Byte-gate unaffected.
2026-06-23 03:08:30 -06:00
Drew T d09248361b fix(phase-21): pool excludes ALL self-MATCH-but-gate-rejected churners
plumbing_blocked() scanned backlog.load_best() (one record/fn), which could return an
early closeness>0 near-miss that MASKED a later 'none — MATCH' self-match-gate-reject
record (e.g. a sig_unify-regressed self-match logged at closeness 14). Result: fns like
func_80161CD0 self-MATCHed + gate-rejected every wave yet kept being re-served.

Fix: scan ALL raw .run/backlog.jsonl records. Now 83 plumbing-blocked fns are correctly
skipped from wave selection (they're recovery-tooling / hand-finish fuel; re-drafting just
reproduces the same gate-rejected body). Byte-gate unaffected.
2026-06-23 01:34:08 -06:00
Drew T 9d07712205 feat(phase-21): reach-1 harvest pivot — fuel built + handoff (×134 walls exhausted at 61.16%)
- ×134 wave fuel walled (waves 13-16: +6 banked incl 476-ins func_80141CA4 ×134, fleet
  60.93% -> 61.16%); grinder stopped (Drew). Pivot to reach-1 (overlay-unique) fuel.
- reach-1 fuel BUILT: 298 uncached reach-1 prefetched (headless DecompileFunctions.java,
  298 ok / 0 fail) -> all 420 reach-1 region-main draftable fns cached (median 47 ins).
- tools/wave_targets.py: new reach1 pool, SMALLEST-FIRST sort (idiom-mining order).
- tools/orchestrator.py: reach1 in POOLS; orch_state set to reach1 (prep --mode pool serves it).
- docs/automation-runbook.md + CURRENT_PHASE.md: the reach-1 smallest-first harvest cycle
  (distill EVERY wave) for the fresh session Drew drives. x1 leverage (ov_SC01_077 only).
- docs/matching-cookbook.md: marked the 2 floating §21 bullets CANDIDATE/unverified (near-miss
  func_801775E0, not byte-banked); drafter-cookbook-edit guard at commit:0219.
- NOT started: no waves launched (Drew's instruction). Byte-gate unchanged (G3/P9).
2026-06-22 21:22:35 -06:00
Drew T f137658171 fix(phase-21): drafters write ONLY their draft .c — distill owns cookbook (prevent unverified near-miss idiom pollution) 2026-06-22 09:36:47 -06:00
Drew T 7b477c82b9 feat(phase-21): wave_targets — exclude self-MATCH-but-gate-rejected via drafter verdict too
close=0 alone missed fns sig_unify regresses to close>0 post-transform (e.g. func_8014B944,
self-MATCH every wave, never banks). Also match the drafter's 'none — MATCH' where_stuck verdict.
Excluded set 7 -> 16; pool waves now draft more FRESH targets (wave-8 close-rate already up to 0.21).
2026-06-22 02:13:17 -06:00
Drew T 40fa76284e feat(phase-21): wave_targets skips close=0 plumbing-blocked near-misses from the pool
- self-MATCH-but-gate-rejected fns can never bank via re-drafting (the drafter reproduces the
  byte-correct body the gate rejects again) -> exclude from POOL selection so agents draft FRESH
  targets; they stay in the backlog as recovery-tooling/hand-finish fuel
- CURRENT_PHASE: sustained-loop state + pool-mode override policy (compaction-recovery anchor)
2026-06-22 00:59:06 -06:00
Drew T 4927f38c46 feat(phase-21): option C — grinder blacklist + sig_unify DEF-side arity-extend recovery
- grinder.py: persistent blacklist for permuter-won/gate-rejected (plumbing-bound) fns;
  stops the tried.clear()-after-idle churn (was 0 banks in 8h on func_8014F3E8/func_8014FE60)
- sig_unify.py rewrite_def: adopt the canonical param list on arity mismatch (DEF-side
  loose-typing wall) — unused extras sit in $a0-$a3, free at -O2; banked func_8016EDEC/EE40
- cookbook §22 (DEF-side recovery + grinder blacklist); CURRENT_PHASE option-C log
- compounds: sig_unify is in the gate pipeline -> every future wave + the grinder auto-recover
2026-06-21 22:55:40 -06:00
Drew T 5fc5d64445 feat(phase-21): R2/R3 — the learning flywheel (class-grouped waves + distill)
- residual class travels with the draft: drafter stamps // @class / // @stuck; gate_stage logs
  the worker's gcc-quirk class to the backlog (not the coarse manifest class).
- wave_targets.py --class <C> / --list-classes: class-grouped re-attempt waves over the backlog's
  classified near-misses (REGALLOC/SCHEDULE/REMAT/STRUCT/IV/LOOPGUARD/LOOSE/PLUMBING).
- orchestrator prep: auto-picks a class-focused wave when a class accrues >= threshold near-misses,
  else a fresh pool harvest; clears the draft dir per wave.
- tools/workflows/distill.js: post-wave agent extracts NEW byte-verified gcc idioms -> cookbook §21
  (R16, conservative); drafters now read the LIVE cookbook so distilled idioms feed forward.
- runbook: the full prep->worker->gate->distill cycle (the Phase-18 close-rate loop, automated).
2026-06-21 13:56:59 -06:00
Drew T 66d1af0e09 docs(phase-21): automation runbook + status tool (grinder running; worker = remote-driven)
- docs/automation-runbook.md: launch/monitor/stop for the grinder (live) + worker /loop;
  safety invariants (byte-gate sole arbiter, git crash-safe, auto_stop kill switch);
  remote-management note (Drew can remote in -> no bulletproof keep-alive needed).
- auto_status.sh: show grinder + worker heartbeats, backlog, phase-21 checkpoints, daemons.
2026-06-21 13:12:04 -06:00
Drew T 8de18e617d feat(phase-21): T4/T5/T6 — worker validated (func_80164530 x134), grinder, orchestrator
- T4 worker Workflow validated: 3-target wave, all self-MATCH; whole-binary gate banked
  func_80164530 x134 (5 register-pins + offset-fold + scheduling). fleet 58.90->58.98%.
  §20 holds (1/3 survived the whole-binary gate); the 2 near/failed -> backlog.
- T5 grinder: token-free permuter of backlog near-misses -> gate_stage; gate correctly
  rejected a permuter over-prediction (integrity intact). auto_supervisor DRIVER-param.
- T6 orchestrator.py: ROI pool rotation (prep/finish/status); gate_stage flock serializes
  grinder+orchestrator builds. Sandbox: detached daemons build fine, foreground needs override.
2026-06-21 13:06:34 -06:00
Drew T 2d7926e9f6 feat(phase-21): T4/T5 tooling — worker Workflow + wave selector + permuter grinder
- tools/workflows/worker_wave.js: parallel toolkit-aware drafter agents (xHigh) per target,
  each writes its best matching C to the drafts dir (gate run by the orchestrator after).
  args parsed robustly (harness may serialize to a JSON string — §20 gotcha).
- tools/wave_targets.py: select a wave batch from the fuel manifest (pool/region/N, skips
  backlog walls). tools/grinder.py: token-free permuter grinder — permute backlog near-misses
  -> gate_stage -> bank; STOP/heartbeat, idles for new worker near-misses, supervisor-friendly.
2026-06-21 12:41:14 -06:00
Drew T c2e93e50cd feat(phase-21): T3 — backlog ledger + shared gate stage (validated, +0.08%)
- tools/backlog.py: near-miss ledger (.run/backlog.jsonl + docs/backlog.md ranked by
  reach->closeness->size; drop-now-matched filter, P9). tools/gate_stage.py: the shared
  deterministic spine both worker+grinder call (canon_resident_calls -> cast_call_sites ->
  sig_unify -> harvest_verify --chunk 1 -> dedup_propagate --auto-from -> backlog -> summary).
- validated end-to-end on a real 30-draft m2c sample: banked func_801710DC + propagated x134
  (dedup 1506->1508), 29 near-misses logged+ranked. make check-all 136/136 byte-identical
  (R22); fleet 58.82%->58.90%; dedup-check 1508 validated/0 failed (G3/P9).
- capped-fns reframe (R14): they're matched-inline (propagation-blocked), not draftable ->
  deferred to the worker; not claiming the unbanked +0.3% (P9).
2026-06-21 12:32:21 -06:00
Drew T 5674587bcb feat(phase-21): T2 — fuel prefetch, Ghidra-C cache complete for the ROI pool
- headless DecompileFunctions over 263 uncached ROI-pool fns (MCP stopped, R23):
  263 ok / 0 fail / 0 no-func. cache 300 -> 563. ROI-pool fuel now 325/325 cached
  (28/28 giants, 310/310 reach-134, 7/7 capped) -> the unattended run never needs MCP.
- FIX DecompileFunctions.java: key output by entry address func_<UPPERHEX>.c (Ghidra's
  raw-import default is FUN_<lowerhex>, which every consumer + the existing 300 files
  miss). Renamed the 263; the tool fix makes re-prefetch reproducible.
- build_fuel_manifest.py --emit-prefetch: reproducible uncached-ROI-pool addr-list
  (for the T7 runbook). 349 reach-1 x1-leverage fns intentionally excluded (P9).
- ghidra/ db churn is the MCP-stop no-op save (R23 restart-noise, not staged).
2026-06-21 12:00:47 -06:00
Drew T 971f95f7c8 feat(phase-21): T1 — fuel manifest + R14 giants byte-verification
- tools/build_fuel_manifest.py -> .run/fuel_manifest.json: the automation target
  pool. reach RECOMPUTED from the current 134 overlay sigs (distinct-overlay
  h_exact), not trusted from cached fields (R14). Classes GIANT/WAVE/STRUCT/PINS/
  STUB/O0/O1 + the CAPPED recovery pool; cached? flags Ghidra-C gaps for T2.
- 803 live func_ stubs; 310 reach-134, 486 reach-1. R14: 28/28 giants verified
  genuinely byte-identical x134 (label honest, unlike the -O0 cluster). -O0
  nuance: 8/9 overlay-local but func_8013C08C is reach-134 (real exception).
- T2 workload scoped: 253 reach-134 uncached; 171/223 tractable reach-134 uncached.
- CURRENT_PHASE.md: T1 logged. No MCP, no build mutation.
2026-06-21 11:50:31 -06:00
Drew T 6c5bd1c4a9 feat(phase-20): cast_call_sites.py — the §17a-1 callee-cast cap tool; 6 recovered, fleet 58.63%->58.82%
- tools/cast_call_sites.py: per draft, callee whose canonical TU sig != draft's
  intended sig -> decl->canonical (kills in-TU conflicting types, keeps symbol in
  scope) + call-site cast ((ret(*)(args))func_X)(args) (codegen-neutral: gcc folds
  the cast of a known symbol -> direct jal). Pure --in/--out; whole-binary gate is
  the arbiter (G3/P9). Pipeline: canon_resident_calls -> cast_call_sites ->
  sig_unify -> harvest_verify --chunk 1
- recovered 6 of T6 batch-1's 33 gate-fails byte-identical (func_80153C44/8015CF58/
  801711FC/80161BE0/801683D8/8015F948); 5 propagated x134 + 1 local; check-all 136/136
- R14 CORRECTION of cookbook §20's '~33' projection (byte-proven): the batch is NOT
  mostly callee-conflicts — ~6 callee (cast-recovered), ~18 DEF-side loose-typing wall
  (def's matchable sig conflicts with the unchangeable caller-canonical; caller-side
  fix BLOCKED because INCLUDE_ASM declares no symbol -> a shared macro's internal
  extern is load-bearing in the 133 stub overlays), + a small DATA tail. The def-side
  needs re-drafting under the caller-canonical sig (future wave), not a text transform
- cookbook §20 RESOLUTION + SETUP tool inventory (R16/R21/R30); CURRENT_PHASE log (R28)
- fleet 58.63% -> 58.82%, dedup 1501 -> 1508, 0 NON_MATCHING (G4)
2026-06-21 02:04:39 -06:00
Drew T ba6cb87d30 feat(phase-20): T2 — exemplar_miner routes 835 residuals (WAVE 472 / PINS 45 reach-134 / STUB 90)
- tools/exemplar_miner.py: consume wall_taxonomy.json + per-overlay reach (dedup_propagate's
  computation) -> route each residual to a lever (WAVE/STRUCT/PINS/STUB); emit
  docs/exemplar_curriculum.md + .run/exemplar_routing.json (wave-gen input)
- routing: WAVE 472 (218 reach-134, T6 fuel) / STRUCT 159 / PINS 114 (45 reach-134) / STUB 90
  (74 ARITY_WALL loose-typing + 16 NONFAITHFUL, honest P9)
- KEY T3 input: 17 reach-134 fns at m2c-mismatch=1 = cleanest regalloc/schedule isolates
  (likely SS17-pins wins + the genuine class residuals); 3 named exemplars confirmed reach-134
- note: census is the Jun-19 snapshot (pre-Phase-19 waves); T5/T6 regen = authoritative WAVE list
2026-06-20 23:55:51 -06:00
Drew T 4f5f73c480 feat(phase-20): T1 — typedef lift closes the type-blocked propagation cap (9 fns ×134)
- build_engine_types.py: add find_typedefs() (brace-aware) + typedef_name() (anon-struct,
  fn-ptr, alias) + same-name/overlap collision guards; lift 14 typedefs to engine_types.h
  in source order after the named structs (A801593E4 -> S801593E4 dep preserved)
- engine_types.h: 48 named types + 14 typedefs; ov_SC01_077 byte-neutral (still d19c9580)
- dedup_propagate --auto-from: 9 reach-134 fns propagated x134 (8013E4B4 80148C9C 8014A238
  8014C118 80153978 801593E4 80168664 8016B964 8017AD0C); engine_core.h +9 DEFINE macros
- FINDING (R14/P9): cap was 16; only 9 were TYPE-blocked (all freed). The other 7 fail on
  undeclared callees/data (80142B2C 801535F4 80155E30 80157580 801576A8 80168F40 80170B48)
  -> the SS19 declaration-plumbing class, a recovery-tooling lever, NOT the type-lift
- check-all 136/136 byte-identical; fleet 58.00%% -> 58.35%%; dedup 1485 -> 1494 / 0 failed; 0 NON_MATCHING (G4)
2026-06-20 23:49:58 -06:00
Drew T 3e99d2836d feat(phase-19): T3 batch-2 wave — 25 matched, 8 propagated ×134, fleet 57.62%->57.93%
- Ultracode wave (38 targets, clean hints after the gen_harvest_targets fix): 35/38 match_one
  MATCH (92%, up from 88%); 2 of 3 garble-retries now match; 3 DIFF = genuine (delay-slot
  schedule, call-crossing hoist-vs-remat, narrow-param STOP)
- gen_harvest_targets.py: fixed INLINE_DEF_RE (was matching indented `if(func_X(...)){`
  call-exprs as defs -> garbled callee hints agents pasted as externs); 0 garbled of 749
- whole-binary gate: 25 banked. ALL 18 stage failures were the shared-caller ARITY class ->
  NEW tools/fix_arity_callers.py automates the no-proto caller-extern recovery (+8); 10 remain
  = genuine loose-typing conflicts (caller sig != def sig), Phase-16 documented wall
- pipeline finding folded in: gate canon-only FIRST, sig_unify FALLBACK (.run/t3_gate2.sh)
- propagated 8 ×134 (17 stayed ov_SC01_077-local: inline named structs need engine_types lifting)
- check-all 136/136 BYTE-IDENTICAL (R22); dedup 1475->1483 validated/0 failed; 0 NON_MATCHING (G4)
- fleet byte-identical 57.62% -> 57.93% (+0.31%)
2026-06-20 19:51:50 -06:00
Drew T 85afa0dc81 feat(phase-19): T2 recovery tooling — canon_resident_calls + no-proto caller fix (3 matches)
- R14 finding: the 3 Phase-18 "implicit-int / propagate-first" near-misses are NOT
  implicit-int (reproduced each through the byte-gate). Real classes:
  (a) resident-callee LINK-miss: draft calls func_<ADDR> but the addr has a curated
      name (0x8004CFEC = ratan2) -> "undefined reference to func_8004CFEC". NEW
      tools/canon_resident_calls.py rewrites func_<ADDR> -> curated name (pure draft
      text, body bytes unchanged) -> run FIRST in the recovery pipeline.
  (b) shared-caller ARITY conflict: an engine_core.h caller macro declares the callee
      file-scope `extern void func_X(void)` then calls func_X(); the real def takes an
      arg -> "conflicting types". FIX = caller extern -> no-proto `extern void func_X();`
      (byte-neutral for the caller; compatible with promotion-safe params int/long/ptr).
- 3 matches banked in ov_SC01_077: func_8017209C (ratan2 rewrite), func_80147514 (s32),
  func_80168F40 (void*); 2 no-proto lines in src/shared/engine_core.h
- recovery pipeline: draft -> canon_resident_calls -> sig_unify -> harvest_verify --chunk 1
- make check-all 136/136 BYTE-IDENTICAL (full extract, R22); dedup 1450 validated/0 failed;
  0 NON_MATCHING in any default build (G4); fleet 56.64% (the 3 are ov_SC01_077-local,
  x134 propagation rides T3/Phase-20)
- cookbook §17a-3 corrected (R14/R16); SETUP tool inventory (R21); CURRENT_PHASE T2 done
2026-06-20 17:12:00 -06:00
Drew T c4b0cef86a feat(phase-17): close — canonical-sig layer built; the wall is the compiler, not sigs; pivot to gcc research (v1.16.0)
- canonical-sig layer (session 4): tools/census_conflict_callees.py + derive_canonical_sigs.py
  -> a 20-extern byte-neutral block atop ov_SC01_077.c (LOCAL, not engine_core.h); census
  conflict callees 20->0, blocked targets 24->0; gate pipeline now draft -> sig_unify (MANDATORY)
  -> harvest_verify --chunk 1; fleet 136/136 byte-identical (R22), 55.51% (no regression)
- FINDING (R14/P9): the conflict wall is 7%-reach not ~2x; the 4 reach-134 circular targets are
  ALL gcc-quirk/regalloc/layout-bound (0 banked); the high-reach core IS the quirk tail; struct
  types are byte-neutral for matching (the wall is gcc codegen, not knowledge)
- leverage analysis: fleet % is function-count-weighted (size adds no %); "unblock many" = the
  layer (declaration, not matching); reach is the lever (already reach-sorted); 247 tractable
  reach-134 stubs ~ +3-4% projected
- GO/NO-GO: NO-GO on brute waves at the current ceiling; GO on a compiler-quirk research phase
  (read gcc-2.7.2 source + Xenogears + the §10/regalloc classes, R17) -> then resume the wave
- docs: cookbook §16 corrected + hand-matching-process.md §8 (the layer + the finding + handoff)
- worklog archived -> phase-ends/logs/Phase17.md (R19); bumps 1.15.0 -> 1.16.0
2026-06-20 09:35:18 -06:00
Drew T 0dd9bf6bac feat(phase-17): calibration harvest wave — fleet 55.04%->55.51% (+0.47%)
- Ultracode wave: 30 high-reach tractable residuals, parallel guided-hand-matching
  agents (m2c + Ghidra-C cache + asm + actor-struct + S1/S2/S3a playbook), self-
  validate match_one. 18/30 match_one MATCH (60%); 10/30 whole-binary verified (33%)
  after sig_unify recovered 2 -> all 10 propagated x134; 2 demo fns caught up
- 136/136 byte-identical (R22 clean check-all); REAL +1,610; 1421 dedup groups
- KEY FINDING: the match_one->whole-binary gap (60%->33%) is 100% SIG CONFLICTS
  (shared callees like func_80131CA8 declared inconsistently across parallel agents),
  NOT codegen. The canonical-sig layer (deferred as low-value) is the ESSENTIAL
  enabler -> would lift 33% toward 60%. 12/30 are the genuine gcc-quirk tail (DIFF).
- Ghidra pre-pass: tools/ghidra_scripts/DecompileFunctions.java (headless batch
  decompile -> .run/ghidra_c cache; no /mcp needed); 300 tractable targets sized
- harness: .run/harvest_wave_s3.js (embedded-targets Workflow; args channel doesn't
  transit arrays). cost ~1.89M tokens / 30 fns
2026-06-19 20:54:45 -06:00
Drew T 1431b523a4 feat(phase-17): T4+T5 — actor struct recovered (comprehension) but byte-neutral for matching
- recovered the engine actor/player struct cold via PCSX-Redux (R10/R11): base 0x80078E00,
  ~154 fields over 0x24C bytes, 32 pointer fields; HP/BP/gold/day/hour/position/BP-drain all
  live-verified and anchored to the Phase-3 player state + the walk diff. -> docs/actor-struct.md
- tools/ram_probe.py: PCSX-Redux RAM capture/diff/read/ptr-scan workhorse for field-typing
- FINDING (the matching verdict): feeding the struct to m2c as --context is BYTE-NEUTRAL.
  m2c adopts it (arg0->f_X) but structural-miss sample scores 0 better / 10 same / 2 worse vs
  m2c-alone (full code+RAM pointer set). The misses are 50-98% mismatched = m2c decompiler
  limits, not missing types. T4 fn-ptr context unblocks compilation but functions stay structural.
- conclusion: rich context (struct/type recovery) is a comprehension / decomp.me win, NOT a
  byte-match lever -> Phase-16 wall re-confirmed from the type angle. struct-core-pivot.md updated.
- emulator field-naming pass deferred (comprehension, off the matching path); NEXT: T6 permuter
  validation on the 146 genuine near-misses, then T7 go/no-go.
2026-06-19 13:03:14 -06:00
Drew T 070cf375c9 feat(phase-17): T3 — bank 42 sig_unify matches + propagate x134 (fleet 54.48%->55.00%); fix taxonomy classifier
- banked 42 gate-proven SIG_FIXABLE_KR matches in ov_SC01_077 (of 48 predicted; 6 honestly
  reverted by the whole-binary byte-gate, P9) + propagated 13 h_exact-shared x134 via
  dedup_propagate (one body -> 134 overlays, each byte-gated, fail-closed)
- fleet byte-identical-from-source 54.48% -> 55.00% (+0.52%); REAL +1,778; dedup groups
  1395 -> 1408; make check-all 136/136 byte-identical (R22); dedup-check 1408 validated/0 failed
- wall probe (R14): raw drafts 0/99 + sig_unify fails -> the K&R/sig_unify avenue is exhausted;
  the residual conflict-class is ARITY_WALL (the §14e param-count dead-end) or structural (T4/T5)
- FIX a T1 classifier bug surfaced by verifying the wall vs the bytes: tu_compile_error captured
  the baseline's pre-existing byte-NEUTRAL warnings ("makes integer from pointer", cc1 rc=0) and
  mislabeled 99 fns LOOSE_TYPING_WALL; now keeps only non-warning errors -> there is NO loose-typing
  wall, the real residual wall is ARITY_WALL=74. docs/wall-taxonomy.md amended with the correction.
- 134 overlay .c files carry the propagated DEFINE_func instantiations; a one-time full-fleet
  make extract was required for the per-overlay propagation gate + check-all
2026-06-19 12:13:38 -06:00