BFM-decomp/phase-ends/PhaseEnd_Phase23.md

PhaseEnd — Phase 23: The offline LLM matching tier → the Fable5 wall-breaker + the gcc-2.7.2 codegen map

Date: 2026-07-02 · Project Version: 1.22.0 · Phase Status: Complete (milestone confirmed by Drew, gate 2) · Generation: Gen2 (15th phase of the arc; 14 deferred to Gen3+)

Gen2 phase 15 (8→…→22→23). The longest, highest-yield phase — it began as "stand up a free local LLM matching tier" and, mid-run, pivoted into the strategic breakthrough of the whole project: a frontier model reading the gcc-2.7.2 source is a wall-breaker, and reading the source once into a codegen map (cookbook §31 + docs/gcc-2.7.2-map/) converts per-function reverse-engineering into cheap lookup. The granular per-task trail (T1–T10.9 LLM-tier, then the giant campaign + §31 study + wave) is in phase-ends/logs/Phase23.md (R19 — NOT auto-loaded). This file is the synthesis. Owner decisions (Drew): build the LLM tier → exhaust it → GLM frontier test → pivot to hand-solving giants → Fable5 experiment → generalize via Opus + §30 → read the whole compiler source into §31 → prove it with a wave → close here (option 3); permuter+integration next phase.

Build Log

Files created/changed and complete — do not recreate:

The LLM matching tier (T1–T10, the phase's original substance):

• tools/serve_local.py (Unsloth GPU OpenAI endpoint — replaces the ejected LM Studio) · tools/api_draft.py (LEAN drafter + MAXTOK/REASON/OpenRouter env) · tools/lora_grind.py (mass-run) · tools/bulk_harvest.py (phase-separated parallel-gate harvester + --cycles) · tools/{export_pairs,format_finetune,train_lora,eval_lora}.py (the LoRA pipeline) · tools/glm_reconcile.py · tools/idiom_hunt.py · tools/inject_capped_externs.py (§28d). Models: models/bfm-match-7b-v3 (Qwen2.5-Coder-7B QLoRA on corpus-v3; v2 fallback). Corpus datasets/match_pairs/ (gitignored). Details in docs/gen2-mips-matching-model.md.

The giant campaign + the §31 codegen map (this session — the pivot):

• tools/glm_parallel.sh — K concurrent OpenRouter/GLM cloud drafters (key from .env at runtime, never echoed/persisted).
• docs/gcc-2.7.2-map/{sched,regalloc,loop,cse_expr}.md — the codegen map (935 lines). 4 Fable5 agents read the gcc-2.7.2 passes → source-cited, byte-proven residual → C-lever | intrinsic→permuter catalogs. cookbook §31 = the index + triage table.
• tools/reference/gcc-2.7.2/ (gitignored) — the REAL vanilla gcc-2.7.2 source (the loop agent proved gcc-papermario is gcc 2.8.1, a behavioral diff). SETUP §5.6 + §31 flag it.
• docs/matching-cookbook.md — §30 (Fable5 store-vs-load /s-flag + def-side macro-widen + birthing-boost), §30a (§30 generalizes via Opus agents + IV-combine + inline-limit + the mechanical-integration finding), §31 (the codegen-map index + triage + the gcc-version correction).
• src/shared/engine_core.h + src/shared/engine_types.h + 134 overlay .c + config/dedup.us.yaml — the banked matches + ×134 propagations.
• docs/SETUP.md §5.6 (gcc-version correction + the vanilla-2.7.2 reference).
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase23.md (R19). phase-ends/PhaseEnd_Phase23.md — this file.

Tools/packages installed: Unsloth + torch (cu128) in a separate .venv-train (LLM tier, prior sessions). OpenRouter (cloud, key in .env). No new apt. tools/reference/gcc-2.7.2/ staged (re-fetchable from GNU ftp).

Verification results (literal):

• make check-all → 136 passed, 0 failed of 136 (main 143dbb89…, resident 8e17e02f…, ov_SC01_077 d19c9580…, all 134 overlays) — reproduced clean after every bank (R22). (Milestone proof run at phase close.)
• Fleet: 63.66% (Phase-22 close) → 64.86% byte-identical-from-source (function-count); ~30% byte-weighted (decomp.dev-comparable). 0 NON_MATCHING in any default build (G4); dedup-check validated, 0 failed.
• LLM tier (T1–T10.9): local v3 saturated the ≤15-ins band (~1,300 banks, $0, T10.6 → 64.6%); frontier GLM-5.2 (OpenRouter, ~$4/$25) confirmed the def-side loose-typing wall is intrinsic (triple-confirmed) and the new-idiom well is DRY from both angles (failed-residual T10.8 + fresh-hand-solve T10.9). Conclusion: the free tier harvests the easy bulk; the paid frontier tier is a modest hard-band drafter + idiom teacher, not a wall-breaker at the leaf.
• The giant campaign (this session) — 5 giants/hard-fns banked ×134 (each byte-gated, propagated, check-all-verified): func_8014EE14 (248, Fable5), func_8014F2E0 (66), func_80150528 (53), func_80149374 (23, §31 study), func_80144090 (154, §31 wave). GLM parallel campaign: 2/102 on the walled reach-134 band (~$10; def-side wall confirmed at scale). Free v3 grind: +89 fns.
• The §31 wave (8 reach-134 near-misses, Opus agents applying the map): 3 leaf-MATCH (47–137k tokens each — vs 200–250k walls pre-map) + 5 substantially-advanced permuter seeds with precise class verdicts (func_80132784 240→4-off — the 400-ins Phase-22 "HARD-DEFER" giant; func_801412A8 198/198 structure-exact; func_8014E048 129→35; func_80176D94 56→52; func_80148094 front-half exact). Zero dead-end walls.
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged. No Ghidra DB change (R23 no-op — the db.*.gbf churn is SessionStart-restart noise, NOT staged).

Milestone achieved (confirmed by Drew, gate 2): the phase set out to build a free local LLM matching tier and did (v3, ≤15 saturated, $0). It then pivoted to the project's breakthrough: proving a frontier model (Fable5Max reading the gcc-2.7.2 source) is a wall-breaker — it matched func_8014EE14, a giant on the §20/§10 store-vs-load class 22 phases had "CONFIRMED unsteerable," refuting this phase's own "wall is intrinsic" verdict as model-relative. That generalized: standard Opus agents applying the documented idioms (§30/§30a) banked 2 more giants at 3–5× lower cost, and reading the whole compiler source once (4 Fable5 agents → §31 codegen map) broke multiple long-standing walls (hoist-vs-remat, delay-slot, coalescing) and was proven by an 8-function wave (3 leaf-MATCH by lookup + 5 tight permuter seeds, no walls). Fleet 63.66% → 64.86%, 136/136 byte-identical throughout. The map converts matching from expensive discovery into cheap lookup — the strategic unlock for the endgame.

Next: Phase 24 — Fix + EVOLVE the permuter (§31-class-aware), build the integration-recovery tool, then scale the §31 wave. The matching problem is solved by the map; the remaining bottlenecks are throughput plumbing (permuter for the register-permutation seeds; integration for auto-banking leaf-MATCHes). Full context in the Phase-24 Backlog below. Open a fresh session (plan mode, Max — Tier-1).

Deviations

Item	Plan	Actual	Reason
Phase scope	build the offline LLM matching tier	tier built + exhausted, then PIVOTED to the giant campaign + the gcc-2.7.2 codegen map	the LLM tier hit the def-side wall (intrinsic); Drew's "hand-solve giants + Fable5" opened the wall-breaker; the map is the far bigger prize
"The wall is intrinsic / idiom well dry" (T10.8/T10.9)	a firm conclusion	REFUTED (model-relative) — Fable5 reading gcc source cracked it	true for GLM/7B, false for a frontier model that reads the compiler internals (a clean R14 self-correction)
Giant matching	Fable5 per giant (~375k tokens)	generalized to Opus agents applying §30, then to §31 lookup (47–137k tokens)	once documented, the idioms are cheap to apply — Fable5 is only needed to discover
The compiler-source read	reactive, per-function	systematic: 4 Fable5 agents mapped the whole source once (§31)	Drew's insight — reading it per-function re-derives the same facts N times; one map serves all future matching + the cheap tier
Source reference	gcc-papermario = "our compiler" (since Phase 18)	it's gcc 2.8.1, not 2.7.2 — vanilla 2.7.2 staged	the loop agent caught a behavioral biv-elim difference; all byte-proven levers stand (match_one-validated vs the real cc1); §17/§30 citations were on 2.8.1
Giant banking	matches → auto-bank ×134	matching solved, but banking hit integration friction (macro-widen + TU-context + propagation stragglers) — 1 of 3 wave leaf-MATCHes banked	the bottleneck moved from matching to integration plumbing → Phase-24 tool
CURRENT_PHASE hygiene	kept current	the giant campaign initially ran under the stale "PhaseEnd next" stopping point → updated mid-session (Drew caught it earlier in the phase)	the session snowballed from "quick giant test" into the map breakthrough
Effort	Max	Max throughout (deep synthesis + orchestration; Fable5/Opus agents via the Agent tool)	the campaign + map are deep single-thread orchestration, not breadth

Commit Message

(Phase 23 landed across many sessions T1–T10.9 + the 07-02 giant/map session; commits 09e80573b..da4eb3ff6 + this PhaseEnd.)

feat(phase-23): LLM matching tier + the Fable5 wall-breaker + the gcc-2.7.2 codegen map (v1.22.0)

- LLM TIER (T1-T10.9): tools/serve_local + api_draft + lora_grind + bulk_harvest + the LoRA
  pipeline (v3 model). Local v3 saturated <=15 (~1300 banks, $0). Frontier GLM-5.2 (~$4)
  confirmed the def-side wall INTRINSIC + the idiom well DRY at the leaf (both angles).
- THE PIVOT (07-02): Fable5Max reading the gcc-2.7.2 SOURCE matched func_8014EE14, a §20/§10
  store-vs-load giant "CONFIRMED unsteerable" for 22 phases -> the "wall is intrinsic" verdict
  is MODEL-RELATIVE. Generalized to Opus agents applying §30 (2 more giants, 3-5x cheaper).
- §31 THE CODEGEN MAP: 4 Fable5 agents read the whole gcc-2.7.2 source -> docs/gcc-2.7.2-map/
  {sched,regalloc,loop,cse_expr}.md (935 lines, byte-proven residual->lever catalogs) +
  cookbook §31 (index+triage). Broke hoist-vs-remat/delay-slot/coalescing walls. Correction:
  gcc-papermario is 2.8.1 not 2.7.2 -> tools/reference/gcc-2.7.2/ (SETUP §5.6).
- THE WAVE (8 reach-134 near-misses, Opus + §31): 3 leaf-MATCH by lookup (47-137k tok vs
  200-250k walls) + 5 tight permuter seeds (func_80132784 240->4-off!), zero dead-ends.
- 5 giants/hard-fns banked x134 (func_8014EE14/F2E0/150528/149374/144090); fleet 63.66->64.86%,
  136/136 byte-identical, 0 NON_MATCHING. cookbook §30/§30a/§31. No new governance rules.
- FINDING: matching is SOLVED by the map; bottlenecks are now the permuter (S11 seeds) +
  whole-binary integration (auto-bank leaf-MATCHes) -> Phase 24. worklog -> logs/Phase23.md.

Rules Added This Phase

Rule

Reason

None (governance). Per the Phase-8+ precedent, the phase's lessons are findings + techniques, captured where they belong: the matching toolkit + the codegen map → cookbook §30/§30a/§31 + docs/gcc-2.7.2-map/; the LLM-tier discipline → docs/gen2-mips-matching-model.md + the cheap-tier-ab-validated memory; the "a wall verdict can be model-relative — a frontier model reading the compiler SOURCE breaks classes GLM/7B can't" meta-lesson is a direct extension of R14 (verify vs bytes) + R17 (web/read the compiler internals) + the dont-conclude-unsteerable-try-register-pins memory — now generalized to "read the source, build the map." The existing rules (G3/P9 byte-gate, R14, R16 flywheel, R17, R22, R26/R27 effort, R30 capture-during-session) covered all the new ground.

A build → pivot → breakthrough phase yields tools, a knowledge artifact (§31), and findings — not a new norm of conduct. Drew may later elevate the dont-block-loop-with-askuserquestion memory or a "map-before-grind" practice to a formal rule if he wants it binding.

PhaseEnd Changelog

v1.21.0 → v1.22.0 — Phase 23 complete (Gen2 phase 15; the breakthrough phase). Built the free local LLM matching tier (v3 QLoRA + serving + the harvest drivers), which saturated the ≤15-ins band for $0 (~1,300 banks) and then byte-proved its own ceiling — the def-side loose-typing wall is intrinsic and the idiom well is dry at the leaf, for a 7B/GLM. Drew pivoted to hand-solving giants, and the phase's breakthrough followed: Fable5Max, reading the actual gcc-2.7.2 source, matched a giant on a class 22 phases had "CONFIRMED unsteerable" — proving the "wall is intrinsic" verdict was model-relative. This generalized twice: standard Opus agents applying the now-documented idioms (§30/§30a) banked more giants at 3–5× lower cost; and reading the whole compiler source once (4 Fable5 agents → the §31 codegen map, docs/gcc-2.7.2-map/, 935 byte-proven lines) broke multiple long-standing walls (hoist-vs-remat, delay-slot, coalescing) and was proven by an 8-function wave (3 leaf-MATCH by cheap lookup + 5 tight permuter seeds, incl. the 400-ins func_80132784 driven to 4-off). Along the way it corrected a 5-phase-old error (the source reference was gcc 2.8.1, not 2.7.2). 5 giants/hard-fns banked ×134, fleet 63.66% → 64.86%, 136/136 byte-identical, 0 NON_MATCHING. No new governance rules (techniques → cookbook §30/§30a/§31 + the map). The decisive shift: matching is now solved by the map (cheap lookup, cheap-tier-usable); the remaining bottlenecks are throughput plumbing — the permuter (for the register-permutation seeds) and whole-binary integration (for auto-banking leaf-MATCHes) — which become Phase 24.

Plain-English Recap

This was the phase where the project cracked its hardest problem. We started by building a free, private AI (running on your own graphics card) that reconstructs the game's simple code for no cost — it finished off the small functions (~1,300 of them). But it hit a wall on the hard code, and we proved the wall was real for a small model. Then the breakthrough: we pointed the best AI model in the world at the actual source code of the 1990s compiler — and it solved functions we'd sworn were impossible, by understanding exactly why the old compiler arranged instructions the way it did. Even better, ordinary AI agents could then reuse those tricks cheaply. So we did the smart thing you suggested: instead of re-figuring-out the compiler for every function, we had four AI agents read the entire compiler source once and write a permanent "cheat sheet" — for every way the compiler scrambles our code, either the exact fix or an honest "this one needs the brute-forcer." We tested the cheat sheet on eight hard functions: three were solved instantly by looking up the answer, and the other five were driven to the brink (one giant went from "abandoned" to four instructions from perfect) — with zero of the expensive dead-ends we used to hit. The game is now ~65% rebuilt bit-for-bit, and — more importantly — we've turned "reverse-engineer the compiler" from a per-function slog into a cheat-sheet lookup that even the cheap AI can use. What's left isn't matching anymore; it's two pieces of plumbing — a smarter brute-forcer and an auto-installer for the finished code — which is next phase.

Phase-24 Backlog — "Fix + EVOLVE the permuter · build the integration tool · scale the wave" (the forward bridge — nothing here is lost on close; Drew emphasized storing full context)

Phase 24 opens in plan mode, Max (Tier-1). The matching problem is solved by §31; the two remaining bottlenecks are throughput plumbing. Prioritised:

1. [HIGHEST BYTE-WEIGHT] FIX + EVOLVE THE PERMUTER (Drew's "evolve it to help more" idea — a first-class design goal).

• Why: the §31 wave produced register-permutation / schedule seeds dominated by the S11 LUID⊗alloc coupling class — structure-exact drafts that no C lever moves (the map correctly routes them to the permuter). These are high byte-weight and right there: func_80132784 (4/400 — the 400-ins flagship, ×134 ≈ +1.6% byte-weight alone), func_801412A8 (198/198 structure-exact), func_8014E048 (35/143), func_80176D94 (52/152), func_80148094 (front-half exact), plus earlier func_801770E0/func_8014EA4C. All seeds are in .run/wave/ (+ .run/toolkit/, .run/backlog_drafts/).
• The current permuter is BLOCKED (tools/decomp-permuter + tools/permuter/ + tools/grinder.py; recipe cookbook §3/§3a): (a) pycparser can't parse the __asm__("$sN") register pins the toolkit drafts use → it strips/breaks them (both wave sched agents hit this); (b) it scores against a leaf compile with relocation noise (§5a floor), not the real bytes → it wanders (func_80176D94: base 225, best 210, never 0); (c) custom typedefs (GsOT/OTLINK) break its turnkey path. Phase-22 audit: grinder banked 7 all-time (all Phase 21), 0 since — the random permuter's tail is exhausted.
• FIX (baseline): (a) GNU-asm-declaration support — preprocess the pins to a permuter-safe placeholder + re-inject (or patch pycparser); (b) a relocation-masked scorer (mask jal/HI16/LO16 exactly like tools/match_one.py, which == the whole-binary byte-gate for a standalone reach-134 fn) so it scores true closeness; (c) custom-typedef dir setup.
• EVOLVE (the big idea — the §31 map GUIDES the search): instead of random mutations, make the permuter §31-class-aware — the agent already diagnosed each seed's residual class, so DIRECT the mutation set: S11 register-permutation → mutate the levers §31 says drive allocno_compare density (declaration order, def/use placement, pin scope — regalloc.md RC-1/RC-2/RC-3); S3 chain-priority → schedule-affecting statement reorders (sched.md); RC-6 pressure-lock → a stabilizing pin + targeted perturbation. This turns a random walk into a directed search over the §31 lever space — the map's second payoff (it guides not just agents but the permuter). Implement as PERM_* macros / a class-seeded mutation set.
• EVOLVE further (the ML wildcard — carried from Phase 15/16, now well-motivated): Drew has a 3080Ti/CUDA + LightGBM. Train a candidate-ranker on the banked corpus (asm/RTL features → likely-closeness) to prioritise the permuter's search, or an ML asm→C scorer. A real wildcard to evaluate once the directed permuter is running.

2. [UNLOCKS WAVE THROUGHPUT] THE INTEGRATION-RECOVERY TOOL (§30a "integration is mechanical → build the tool", now urgent).

• Why: §31 leaf-MATCHes do not auto-bank whole-binary — the wave banked only 1 of 3. The friction classes seen this phase: (a) def-side return-type macro-widen — a DEFINE_func_* caller macro declares the fn extern void but the matching def needs s32 (a void return DCEs a live local); widen the macro's extern void→s32, byte-neutral IFF the caller discards the return (§30#2; all 3 banked giants hit this — func_8014EE14/F2E0/150528, verified byte-neutral via check-all); (b) TU-context perturbation — a leaf-MATCH whose whole-binary bytes differ with no caller macro involved (func_80155800, func_8014F4C0 — drafts in .run/wave/; a different declaration/context effect, needs diagnosis); (c) propagation stragglers — a banked fn's dedup group has a cross-overlay member that isn't byte-identical (loose-typing/overlay-local) → the ×134 lift drops to ×1 (func_80144090 banked ×1; a pre-existing 0x80165CA0/0x8012A018 straggler); (d) data-extern type conflicts (func_8014EE14's D_80126720 u8[] vs s16).
• The tool: a gate_stage recovery pass that, on a leaf-MATCH-but-whole-binary-near fn, tries in order: macro-widen (a) → re-gate; TU-context diagnosis (b); straggler recovery (c) (bank ×134 where clean, ×1 where straggler-blocked, log the reason). Goal: every §31 leaf-MATCH auto-banks (with the reason logged when it can't reach ×134). Test cases: the unbanked wave leaf-MATCHes .run/wave/func_80155800.c, func_8014F4C0.c. Context: the recovery pipeline is gate_stage.py (canon_resident_calls → cast_call_sites → sig_unify → harvest_verify → dedup_propagate); the propagation cap/straggler is cookbook §19/§20; the macro-widen is §30#2 (manual now).

3. [THE HARVEST] SCALE THE §31 WAVE (after 1+2). With the permuter closing S11 seeds (1) and the integration tool auto-banking leaf-MATCHes (2), scale the §31-powered Opus-agent wave over the full near-miss backlog (~95 "schedule"/80 "regalloc"/31 "hoist" verdicts — most now steerable per §31). Each match ×134. Batch ~8; triage-first (bail to permuter on confirmed-intrinsic); gate winners together. The map + the two tools convert the backlog into a harvest.

4. [FREE-TIER FLYWHEEL, carried] v4 retrain on the idiom corpus. Once the wave banks many examples per idiom, retrain v4 on the banked-with-idioms corpus so the free local tier applies the common steerable idioms in straightforward cases (per the in-phase v4 discussion — a real but bounded gain; the 7B has a reasoning ceiling + does nothing for the intrinsic walls). Also worth a $0 experiment: §31 in v3's prompt on a few medium near-misses to gauge the ceiling. Post-wave.

5. [BYTE-WEIGHT, carried] The giants + the compile-blocked 8. The 20 giants are 35% of remaining byte-weight; the permuter (1) is the key to the S11 seeds. The 8 compile-blocked giants (region a) are mechanically unblocked (ghidra-type aliases + data externs — NOT struct-walled; type-map finding this phase) — a cheap pass makes them draftable, then §31 + Opus.

6. [STRATEGIC, carried] The public flip (Fable5 §6.3, deferred to Gen3+) remains the community-labor lever; the LLM tier + the map make the repo an attractive contributor on-ramp when it's time.

Carried LLM-tier state: v3 = models/bfm-match-7b-v3; serve via tools/serve_local.py (Unsloth GPU, NOT LM Studio); bulk_harvest.py for the ≤15 tail (~92% saturated, ~383 low-value untried left); corpus datasets/match_pairs/; OpenRouter key in .env (open_router_key, ~$21/$25 left). serve_local is DOWN, GPU free.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase23.md (R19). The Phase-23 work is committed across the session (09e80573b..da4eb3ff6); Drew commits AND pushes this PhaseEnd + the archived log + the regenerated digests (docs/progress.fleet.md) (R6/R8 — the milestone-close is the human-controlled gate). No Ghidra DB change (R23 no-op — the db.*.gbf churn is restart-noise; do NOT stage; the pre-session .gitignore/docs/backlog.md edits are unrelated). Keep this file forever. Gen2 continues — do NOT start Phase 24 here. Start a fresh session (effort Max, plan mode) for Phase 24 — Fix + evolve the permuter (§31-class-aware) · build the integration-recovery tool · scale the §31 wave (the Backlog above; permuter first — it's the highest byte-weight lever, and the flagship func_80132784 is 4 instructions from ×134).