Found 50% of bare-def corpus completions fail standalone compile (undeclared globals/structs — externs
lived in TU headers). Filtering to compile-verified pairs so the pilot LoRA learns to emit COMPILABLE
matching C, not uncompilable bodies. (Drops harder struct/global fns; corpus v2 = add externs to recover.)
Gives the no-tool local model the context the agents read: common.h, live matching cookbook
(COOKBOOK_FULL toggle), 2 byte-matched corpus examples. Qwen3.6-35B-A3B result: harness fixes
compile-fails but model stays stuck at fixed near-misses; FULL cookbook worse+2.3x slower than
curated (dilution, gate-confirmed). Bottleneck is model refinement, not context.
docs/gen2-mips-matching-model.md: the BFM/gcc-2.7.2 matching-specialist idea (LoRA on our own
gate-verified pairs — the corpus off-the-shelf RE LLMs lack). export_pairs.py mines 1307 banked
(asm<->C) pairs from build objects (asm/ is gitignored, so disasm the ROM-identical build, splat-like
format) + src defs -> datasets/match_pairs/{pairs,train,test}.jsonl (gitignored, 1174/133 split).
api_draft.py: TEMP env-tunable. .gitignore: datasets/ models/ weights.
Script equivalent of one worker_wave drafter for any OpenAI-compatible endpoint (LM Studio /
llama.cpp / vLLM / OpenRouter). Inlines asm+ghidra_c+toolkit, calls /chat/completions, extracts C,
iterates against match_one (diff fed back, keep best). Output scores as a 'local'/'glm' arm via
ab_score.py. Env: API_BASE/API_KEY/MODEL. Logic self-tested; HTTP is standard OpenAI format.
- cookbook §28: the 4-way triage of a close=0 gate-rejected giant (pure-extern plumbing /
masked-residual / struct-walled / regressed-draft) + the canonical-extern recovery (the NEW
lever that banked func_8015126C, which Phase-21 left as a close=0 wall) + the coalescing pin
(register __asm__ + lazy in-&& assign). §28a: PS1-applicable decomp.wiki GCC patterns
(negative-offset loops, branch-duplication, load-coalescing, div-magic table, slti-0)
- tools/recover_giant.py: canonicalize a draft's engine_core.h-callee externs to def-sigs +
move all externs block-scope (validated func_8015126C -> MATCH)
- R14: close=0-giant fan-out NOT uniformly near-free — only func_8015126C was pure-extern;
the rest are struct-walled / masked-residual / regressed -> genuine per-giant hand-work
- THE FINDING (R14/P9): the _a close=0 recovery banks 0/20 (same def-side multi-way
loose-typing wall as MAIN 0/40, cont.6). cont.6's func_8012F568 was the lone simple-arity
win; the rest need a C type that conflicts caller-decl-vs-def with no single fitting sig.
The cheap close=0 ×134 recovery lever is EXHAUSTED.
- TOOLING (R16, idiom_loop.py --assess was DOUBLY inflated, now fixed):
(1) load_backlog drops banked-since-logged fns (intersect ledger w/ live INCLUDE_ASM stub
set, mirrors backlog._matched_now) -> killed the bogus 'unknown 14' class + ~10/class
stale-matched inflation (schedule 49->39 r134, regalloc 44->39).
(2) DETERMINISTIC-RECOVERY splits FRESH (never recovery-gated = genuine fuel) vs WALLED
(recovery-failed = def-side wall, do NOT re-run). Post-fix: 0 FRESH / 46 WALLED.
- schedule wave NOT fired (ROI-gate, S20/S26): ~50% x134 fraction on n=2, median 15 ins off,
over an h_exact-inflated count. No tokens spent speculatively.
- cont.6 option-3 CONFIRMED: cached reach-134 cheap fuel is dry; the genuine levers are
token-heavy fresh-session work (the 8 giants / per-fn pin-cracking / build a resolved-reach
probe). cookbook S26. 0 banks; fleet 63.17% unchanged; 136/136 byte-identical; 0 NON_MATCHING.
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just
the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation
into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure,
isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the
survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates);
byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical).
Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3
clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe.
The cont.4 root-cause fixes so future _a waves auto-realize ×134:
- dedup_propagate --auto-from now excludes *_o0.c (-O0) defs: -O0 codegen embeds per-overlay
%lo data, so masked h_exact falsely reports reach-134 (§18/§20). One such straggler
(func_8013C360) reverted 10 clean ×134 matches under the all-or-nothing batch gate (cont.4).
Detected via find_site on the -O0 split file; --addr still forces them. (6 fns excluded.)
- gate_stage: capture dedup_propagate's exit; on a real byte-gate revert (not the benign
'nothing to propagate' no-op) write .run/auto/last_propagate_error.log + add prop_error to
the summary + warn. A swallowed revert previously hid the gain silently.
- worker_wave.js: derive match_one self-check --asm-subdir per-target from t.asm
(was hardcoded main subdir -> every _a agent self-check crashed on missing .s ->
blind drafting). Enabled the 16/24 close rate on the _a measurement wave.
- CURRENT_PHASE cont.4: the _a vein is x134 (10/11 propagate byte-identical), not the
cont.3/§24 x1 wall. The gate's propagated:0 was 2 bugs: dedup_propagate all-or-nothing
batch-revert poisoned by 1 stale -O0 overlay-local straggler (func_8013C360), and
gate_stage sh() swallows dedup_propagate's non-zero exit. Realized +0.39% (commit:0267).
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0);
source-find scans all, member-loop edits the right file, structural-check spans all. Single-file
overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert.
- cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file
loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged.
- FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not
difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same
§16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a
RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe.
- no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted.
--class selected ALL backlog near records of a class, incl. already-banked fns (stale
near records, no live_stubs check) and closeness==0 plumbing-blocked fns (re-draft cannot
bank) -> ~18 of 24 wasted slots. Now filters to: name in live_stubs() AND closeness>0.
Gives clean reach-134 wall fuel for worker --class re-attempts (grinder owns REGALLOC/SCHEDULE).
Smallest-first reach1 pool re-served close>0 near-misses at the front of every wave
(fixed small nins), but permuter-class walls (schedule LUID-tie / regalloc-coalesce /
iv-combine) never bank via blind re-draft -> func_80140E6C near-missed at close=4 in BOTH
wave 24 & 25, a slow leak that compounds as the band climbs.
reserved_walls(min_attempts=2): a near-miss re-drafted >=2x without banking is a wall (the
crackable ones crack on attempt 2 — func_801651B8/801549F8/80153D7C did). Skip from blind
waves; they stay in the backlog for the grinder (permuter) or a targeted --class re-attempt
(prior stuck-point fed in). 66 such walls now excluded. Byte-gate unaffected.
plumbing_blocked() scanned backlog.load_best() (one record/fn), which could return an
early closeness>0 near-miss that MASKED a later 'none — MATCH' self-match-gate-reject
record (e.g. a sig_unify-regressed self-match logged at closeness 14). Result: fns like
func_80161CD0 self-MATCHed + gate-rejected every wave yet kept being re-served.
Fix: scan ALL raw .run/backlog.jsonl records. Now 83 plumbing-blocked fns are correctly
skipped from wave selection (they're recovery-tooling / hand-finish fuel; re-drafting just
reproduces the same gate-rejected body). Byte-gate unaffected.
close=0 alone missed fns sig_unify regresses to close>0 post-transform (e.g. func_8014B944,
self-MATCH every wave, never banks). Also match the drafter's 'none — MATCH' where_stuck verdict.
Excluded set 7 -> 16; pool waves now draft more FRESH targets (wave-8 close-rate already up to 0.21).
- self-MATCH-but-gate-rejected fns can never bank via re-drafting (the drafter reproduces the
byte-correct body the gate rejects again) -> exclude from POOL selection so agents draft FRESH
targets; they stay in the backlog as recovery-tooling/hand-finish fuel
- CURRENT_PHASE: sustained-loop state + pool-mode override policy (compaction-recovery anchor)
- grinder.py: persistent blacklist for permuter-won/gate-rejected (plumbing-bound) fns;
stops the tried.clear()-after-idle churn (was 0 banks in 8h on func_8014F3E8/func_8014FE60)
- sig_unify.py rewrite_def: adopt the canonical param list on arity mismatch (DEF-side
loose-typing wall) — unused extras sit in $a0-$a3, free at -O2; banked func_8016EDEC/EE40
- cookbook §22 (DEF-side recovery + grinder blacklist); CURRENT_PHASE option-C log
- compounds: sig_unify is in the gate pipeline -> every future wave + the grinder auto-recover
- residual class travels with the draft: drafter stamps // @class / // @stuck; gate_stage logs
the worker's gcc-quirk class to the backlog (not the coarse manifest class).
- wave_targets.py --class <C> / --list-classes: class-grouped re-attempt waves over the backlog's
classified near-misses (REGALLOC/SCHEDULE/REMAT/STRUCT/IV/LOOPGUARD/LOOSE/PLUMBING).
- orchestrator prep: auto-picks a class-focused wave when a class accrues >= threshold near-misses,
else a fresh pool harvest; clears the draft dir per wave.
- tools/workflows/distill.js: post-wave agent extracts NEW byte-verified gcc idioms -> cookbook §21
(R16, conservative); drafters now read the LIVE cookbook so distilled idioms feed forward.
- runbook: the full prep->worker->gate->distill cycle (the Phase-18 close-rate loop, automated).
- tools/workflows/worker_wave.js: parallel toolkit-aware drafter agents (xHigh) per target,
each writes its best matching C to the drafts dir (gate run by the orchestrator after).
args parsed robustly (harness may serialize to a JSON string — §20 gotcha).
- tools/wave_targets.py: select a wave batch from the fuel manifest (pool/region/N, skips
backlog walls). tools/grinder.py: token-free permuter grinder — permute backlog near-misses
-> gate_stage -> bank; STOP/heartbeat, idles for new worker near-misses, supervisor-friendly.
- tools/cast_call_sites.py: per draft, callee whose canonical TU sig != draft's
intended sig -> decl->canonical (kills in-TU conflicting types, keeps symbol in
scope) + call-site cast ((ret(*)(args))func_X)(args) (codegen-neutral: gcc folds
the cast of a known symbol -> direct jal). Pure --in/--out; whole-binary gate is
the arbiter (G3/P9). Pipeline: canon_resident_calls -> cast_call_sites ->
sig_unify -> harvest_verify --chunk 1
- recovered 6 of T6 batch-1's 33 gate-fails byte-identical (func_80153C44/8015CF58/
801711FC/80161BE0/801683D8/8015F948); 5 propagated x134 + 1 local; check-all 136/136
- R14 CORRECTION of cookbook §20's '~33' projection (byte-proven): the batch is NOT
mostly callee-conflicts — ~6 callee (cast-recovered), ~18 DEF-side loose-typing wall
(def's matchable sig conflicts with the unchangeable caller-canonical; caller-side
fix BLOCKED because INCLUDE_ASM declares no symbol -> a shared macro's internal
extern is load-bearing in the 133 stub overlays), + a small DATA tail. The def-side
needs re-drafting under the caller-canonical sig (future wave), not a text transform
- cookbook §20 RESOLUTION + SETUP tool inventory (R16/R21/R30); CURRENT_PHASE log (R28)
- fleet 58.63% -> 58.82%, dedup 1501 -> 1508, 0 NON_MATCHING (G4)
- R14 finding: the 3 Phase-18 "implicit-int / propagate-first" near-misses are NOT
implicit-int (reproduced each through the byte-gate). Real classes:
(a) resident-callee LINK-miss: draft calls func_<ADDR> but the addr has a curated
name (0x8004CFEC = ratan2) -> "undefined reference to func_8004CFEC". NEW
tools/canon_resident_calls.py rewrites func_<ADDR> -> curated name (pure draft
text, body bytes unchanged) -> run FIRST in the recovery pipeline.
(b) shared-caller ARITY conflict: an engine_core.h caller macro declares the callee
file-scope `extern void func_X(void)` then calls func_X(); the real def takes an
arg -> "conflicting types". FIX = caller extern -> no-proto `extern void func_X();`
(byte-neutral for the caller; compatible with promotion-safe params int/long/ptr).
- 3 matches banked in ov_SC01_077: func_8017209C (ratan2 rewrite), func_80147514 (s32),
func_80168F40 (void*); 2 no-proto lines in src/shared/engine_core.h
- recovery pipeline: draft -> canon_resident_calls -> sig_unify -> harvest_verify --chunk 1
- make check-all 136/136 BYTE-IDENTICAL (full extract, R22); dedup 1450 validated/0 failed;
0 NON_MATCHING in any default build (G4); fleet 56.64% (the 3 are ov_SC01_077-local,
x134 propagation rides T3/Phase-20)
- cookbook §17a-3 corrected (R14/R16); SETUP tool inventory (R21); CURRENT_PHASE T2 done
- canonical-sig layer (session 4): tools/census_conflict_callees.py + derive_canonical_sigs.py
-> a 20-extern byte-neutral block atop ov_SC01_077.c (LOCAL, not engine_core.h); census
conflict callees 20->0, blocked targets 24->0; gate pipeline now draft -> sig_unify (MANDATORY)
-> harvest_verify --chunk 1; fleet 136/136 byte-identical (R22), 55.51% (no regression)
- FINDING (R14/P9): the conflict wall is 7%-reach not ~2x; the 4 reach-134 circular targets are
ALL gcc-quirk/regalloc/layout-bound (0 banked); the high-reach core IS the quirk tail; struct
types are byte-neutral for matching (the wall is gcc codegen, not knowledge)
- leverage analysis: fleet % is function-count-weighted (size adds no %); "unblock many" = the
layer (declaration, not matching); reach is the lever (already reach-sorted); 247 tractable
reach-134 stubs ~ +3-4% projected
- GO/NO-GO: NO-GO on brute waves at the current ceiling; GO on a compiler-quirk research phase
(read gcc-2.7.2 source + Xenogears + the §10/regalloc classes, R17) -> then resume the wave
- docs: cookbook §16 corrected + hand-matching-process.md §8 (the layer + the finding + handoff)
- worklog archived -> phase-ends/logs/Phase17.md (R19); bumps 1.15.0 -> 1.16.0
- recovered the engine actor/player struct cold via PCSX-Redux (R10/R11): base 0x80078E00,
~154 fields over 0x24C bytes, 32 pointer fields; HP/BP/gold/day/hour/position/BP-drain all
live-verified and anchored to the Phase-3 player state + the walk diff. -> docs/actor-struct.md
- tools/ram_probe.py: PCSX-Redux RAM capture/diff/read/ptr-scan workhorse for field-typing
- FINDING (the matching verdict): feeding the struct to m2c as --context is BYTE-NEUTRAL.
m2c adopts it (arg0->f_X) but structural-miss sample scores 0 better / 10 same / 2 worse vs
m2c-alone (full code+RAM pointer set). The misses are 50-98% mismatched = m2c decompiler
limits, not missing types. T4 fn-ptr context unblocks compilation but functions stay structural.
- conclusion: rich context (struct/type recovery) is a comprehension / decomp.me win, NOT a
byte-match lever -> Phase-16 wall re-confirmed from the type angle. struct-core-pivot.md updated.
- emulator field-naming pass deferred (comprehension, off the matching path); NEXT: T6 permuter
validation on the 146 genuine near-misses, then T7 go/no-go.
- banked 42 gate-proven SIG_FIXABLE_KR matches in ov_SC01_077 (of 48 predicted; 6 honestly
reverted by the whole-binary byte-gate, P9) + propagated 13 h_exact-shared x134 via
dedup_propagate (one body -> 134 overlays, each byte-gated, fail-closed)
- fleet byte-identical-from-source 54.48% -> 55.00% (+0.52%); REAL +1,778; dedup groups
1395 -> 1408; make check-all 136/136 byte-identical (R22); dedup-check 1408 validated/0 failed
- wall probe (R14): raw drafts 0/99 + sig_unify fails -> the K&R/sig_unify avenue is exhausted;
the residual conflict-class is ARITY_WALL (the §14e param-count dead-end) or structural (T4/T5)
- FIX a T1 classifier bug surfaced by verifying the wall vs the bytes: tu_compile_error captured
the baseline's pre-existing byte-NEUTRAL warnings ("makes integer from pointer", cc1 rc=0) and
mislabeled 99 fns LOOSE_TYPING_WALL; now keeps only non-warning errors -> there is NO loose-typing
wall, the real residual wall is ARITY_WALL=74. docs/wall-taxonomy.md amended with the correction.
- 134 overlay .c files carry the propagated DEFINE_func instantiations; a one-time full-fleet
make extract was required for the per-overlay propagation gate + check-all
- docs/struct-core-pivot.md: findings + decision + new research directions. Root cause = the
original engine is LOOSELY TYPED (K&R; same fn called with int/ptr, arg/no-arg across sites),
so no single canonical signature exists -> m2c guesses inconsistently, permuter can't fix
semantics, byte-gate (correctly) rejects. Yields ~3%, not the crack. New plan: emulator-recover
the actor struct/types -> Ghidra global type propagation -> Ghidra-C -> permuter+gate.
- harness bug-fixes (REAL, kept): p16_permute output-0-only match (killed the false '42%'),
base.c keeps callee externs, winner_to_draft line-strip; sig_unify canonicalizes m2c's
no-extern prototypes; gen_engine_decls.py (documents why a global canonical header breaks
loose-typed matches).
- a few byte-gated leaf matches banked in ov_SC01_077.c.
- driver per-fn flow: m2c -> match_one prefilter -> if near-miss & --permute-secs>0, run
decomp-permuter (time-boxed, reuses p16_permute setup/run) -> use the score-0 winner as the
draft -> whole-binary byte-gate. The permuter is closing near-misses well in the overnight
batch, so it's the driver's core yield step for the unattended run.
- +--permute-j (RAM-bound on the 15GiB box).
- tools/sig_unify.py (NEW): unifies a draft's FULL signature set to the banked-canonical
decls — both callee externs AND the draft's OWN definition signature (return + param
types, body param-names preserved). Recovery for the standalone-MATCH residual: a probe
found 30/30 sampled failures were type conflicts, 0 false-positives. Gated 191 -> 32 verified.
- tools/dedup_propagate.py find_site FIX: the inline-def detector required the opening brace
ON THE SAME LINE as the signature, silently dropping every next-line-brace def from
propagation (sig_unify/permuter outputs + ~stragglers all session). Now accepts brace on
the same OR next line -> unlocked a 61-function propagation backlog.
- propagated 61 fns fleet-wide (incl. 32 sig-unify + the permuter win func_801508F8 +
earlier-dropped next-line-brace matches), all 134 byte-identical; dedup-check 1394/0
- R22 CLEAN fleet rebuild: 136/136 byte-identical. REAL 177501 -> 185646. Zero agent tokens.
- fix tools/build_engine_types.py to be ADDITIVE: merge new source types with the
types already in engine_types.h (a re-run after a prior --strip was destructive —
it dropped the ~34 already-migrated types and broke the build; caught + reverted)
- engine_types.h 34 -> 42 named types (the 8 from wave-1's struct-using matches);
inline defs stripped from ov_SC01_077.c (byte-neutral)
- dedup_propagate --auto-from: 8 of 15 struct-blocked matches now self-contained ->
propagated across all 134 overlays (byte-gated); 7 remain (anonymous/typedef'd types)
- R22 CLEAN fleet rebuild: 136/136 byte-identical; dedup-check 1333/0
- zero agent tokens (deterministic)
Drew T