BFM-decomp/phase-ends/PhaseEnd_Phase23.md

# PhaseEnd — Phase 23: The offline LLM matching tier → the Fable5 wall-breaker + the gcc-2.7.2 codegen map
**Date:** 2026-07-02 · **Project Version:** 1.22.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (15th phase of the arc; 14 deferred to Gen3+)

> Gen2 phase 15 (8→…→22→**23**). The **longest, highest-yield phase** — it began as "stand up a free local LLM matching tier" and, mid-run, **pivoted into the strategic breakthrough of the whole project**: a frontier model reading the gcc-2.7.2 source is a *wall-breaker*, and reading the source *once* into a **codegen map** (cookbook §31 + `docs/gcc-2.7.2-map/`) converts per-function reverse-engineering into cheap lookup. The granular per-task trail (T1–T10.9 LLM-tier, then the giant campaign + §31 study + wave) is in `phase-ends/logs/Phase23.md` (R19 — NOT auto-loaded). This file is the synthesis. Owner decisions (Drew): build the LLM tier → exhaust it → GLM frontier test → **pivot to hand-solving giants** → Fable5 experiment → generalize via Opus + §30 → **read the whole compiler source into §31** → prove it with a wave → **close here (option 3); permuter+integration next phase.**

## Build Log

**Files created/changed and complete — do not recreate:**

*The LLM matching tier (T1–T10, the phase's original substance):*
- `tools/serve_local.py` (Unsloth GPU OpenAI endpoint — replaces the ejected LM Studio) · `tools/api_draft.py` (LEAN drafter + `MAXTOK`/`REASON`/OpenRouter env) · `tools/lora_grind.py` (mass-run) · `tools/bulk_harvest.py` (phase-separated parallel-gate harvester + `--cycles`) · `tools/{export_pairs,format_finetune,train_lora,eval_lora}.py` (the LoRA pipeline) · `tools/glm_reconcile.py` · `tools/idiom_hunt.py` · `tools/inject_capped_externs.py` (§28d). Models: `models/bfm-match-7b-v3` (Qwen2.5-Coder-7B QLoRA on corpus-v3; v2 fallback). Corpus `datasets/match_pairs/` (gitignored). Details in `docs/gen2-mips-matching-model.md`.

*The giant campaign + the §31 codegen map (this session — the pivot):*
- **`tools/glm_parallel.sh`** — K concurrent OpenRouter/GLM cloud drafters (key from `.env` at runtime, never echoed/persisted).
- **`docs/gcc-2.7.2-map/{sched,regalloc,loop,cse_expr}.md`** — **the codegen map (935 lines).** 4 Fable5 agents read the gcc-2.7.2 passes → source-cited, byte-proven `residual → C-lever | intrinsic→permuter` catalogs. **cookbook §31 = the index + triage table.**
- **`tools/reference/gcc-2.7.2/`** (gitignored) — the **REAL vanilla gcc-2.7.2 source** (the loop agent proved `gcc-papermario` is gcc **2.8.1**, a behavioral diff). SETUP §5.6 + §31 flag it.
- `docs/matching-cookbook.md` — **§30** (Fable5 store-vs-load `/s`-flag + def-side macro-widen + birthing-boost), **§30a** (§30 generalizes via Opus agents + IV-combine + inline-limit + the mechanical-integration finding), **§31** (the codegen-map index + triage + the gcc-version correction).
- `src/shared/engine_core.h` + `src/shared/engine_types.h` + 134 overlay `.c` + `config/dedup.us.yaml` — the banked matches + ×134 propagations.
- `docs/SETUP.md` §5.6 (gcc-version correction + the vanilla-2.7.2 reference).
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase23.md`** (R19). `phase-ends/PhaseEnd_Phase23.md` — this file.

**Tools/packages installed:** Unsloth + torch (cu128) in a separate `.venv-train` (LLM tier, prior sessions). OpenRouter (cloud, key in `.env`). No new apt. `tools/reference/gcc-2.7.2/` staged (re-fetchable from GNU ftp).

**Verification results (literal):**
- **`make check-all` → 136 passed, 0 failed of 136** (main `143dbb89…`, resident `8e17e02f…`, ov_SC01_077 `d19c9580…`, all 134 overlays) — reproduced clean after every bank (R22). *(Milestone proof run at phase close.)*
- **Fleet: 63.66% (Phase-22 close) → 64.86% byte-identical-from-source** (function-count); ~30% byte-weighted (decomp.dev-comparable). **0 NON_MATCHING** in any default build (G4); `dedup-check` validated, 0 failed.
- **LLM tier (T1–T10.9):** local v3 saturated the ≤15-ins band (~1,300 banks, **$0**, T10.6 → 64.6%); frontier GLM-5.2 (OpenRouter, ~$4/$25) confirmed the **def-side loose-typing wall is intrinsic** (triple-confirmed) and the **new-idiom well is DRY from both angles** (failed-residual T10.8 + fresh-hand-solve T10.9). Conclusion: the free tier harvests the easy bulk; the paid frontier tier is a modest hard-band drafter + idiom teacher, not a wall-breaker **at the leaf**.
- **The giant campaign (this session) — 5 giants/hard-fns banked ×134** (each byte-gated, propagated, check-all-verified): `func_8014EE14` (248, Fable5), `func_8014F2E0` (66), `func_80150528` (53), `func_80149374` (23, §31 study), `func_80144090` (154, §31 wave). GLM parallel campaign: 2/102 on the walled reach-134 band (~$10; def-side wall confirmed at scale). Free v3 grind: +89 fns.
- **The §31 wave (8 reach-134 near-misses, Opus agents applying the map):** **3 leaf-MATCH** (47–137k tokens each — vs 200–250k *walls* pre-map) + **5 substantially-advanced permuter seeds** with precise class verdicts (**`func_80132784` 240→4-off** — the 400-ins Phase-22 "HARD-DEFER" giant; `func_801412A8` 198/198 structure-exact; `func_8014E048` 129→35; `func_80176D94` 56→52; `func_80148094` front-half exact). **Zero dead-end walls.**
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; zero ROM-derived/generated bulk staged. **No Ghidra DB change** (R23 no-op — the `db.*.gbf` churn is SessionStart-restart noise, NOT staged).

**Milestone achieved (confirmed by Drew, gate 2):** the phase set out to build a free local LLM matching tier and did (v3, ≤15 saturated, $0). It then **pivoted to the project's breakthrough**: proving a frontier model (**Fable5Max reading the gcc-2.7.2 source**) is a *wall-breaker* — it matched `func_8014EE14`, a giant on the §20/§10 store-vs-load class **22 phases had "CONFIRMED unsteerable,"** refuting this phase's own "wall is intrinsic" verdict as *model-relative*. That generalized: **standard Opus agents *applying* the documented idioms** (§30/§30a) banked 2 more giants at 3–5× lower cost, and reading the **whole compiler source once** (4 Fable5 agents → **§31 codegen map**) broke *multiple* long-standing walls (**hoist-vs-remat**, delay-slot, coalescing) and was **proven by an 8-function wave** (3 leaf-MATCH by lookup + 5 tight permuter seeds, no walls). Fleet 63.66% → 64.86%, 136/136 byte-identical throughout. The map converts matching from expensive discovery into cheap lookup — the strategic unlock for the endgame.

**Next:** **Phase 24 — Fix + EVOLVE the permuter (§31-class-aware), build the integration-recovery tool, then scale the §31 wave.** The matching problem is *solved* by the map; the remaining bottlenecks are throughput plumbing (permuter for the register-permutation seeds; integration for auto-banking leaf-MATCHes). Full context in the **Phase-24 Backlog** below. Open a fresh session (plan mode, Max — Tier-1).

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Phase scope | build the offline LLM matching tier | tier built + exhausted, then **PIVOTED to the giant campaign + the gcc-2.7.2 codegen map** | the LLM tier hit the def-side wall (intrinsic); Drew's "hand-solve giants + Fable5" opened the wall-breaker; the map is the far bigger prize |
| "The wall is intrinsic / idiom well dry" (T10.8/T10.9) | a firm conclusion | **REFUTED (model-relative)** — Fable5 reading gcc source cracked it | true for GLM/7B, false for a frontier model that reads the compiler internals (a clean R14 self-correction) |
| Giant matching | Fable5 per giant (~375k tokens) | **generalized to Opus agents applying §30, then to §31 lookup** (47–137k tokens) | once documented, the idioms are cheap to *apply* — Fable5 is only needed to *discover* |
| The compiler-source read | reactive, per-function | **systematic: 4 Fable5 agents mapped the whole source once (§31)** | Drew's insight — reading it per-function re-derives the same facts N times; one map serves all future matching + the cheap tier |
| Source reference | `gcc-papermario` = "our compiler" (since Phase 18) | **it's gcc 2.8.1, not 2.7.2** — vanilla 2.7.2 staged | the loop agent caught a behavioral biv-elim difference; all byte-proven levers stand (match_one-validated vs the real cc1); §17/§30 citations were on 2.8.1 |
| Giant banking | matches → auto-bank ×134 | **matching solved, but banking hit integration friction** (macro-widen + TU-context + propagation stragglers) — 1 of 3 wave leaf-MATCHes banked | the bottleneck moved from *matching* to *integration plumbing* → Phase-24 tool |
| CURRENT_PHASE hygiene | kept current | the giant campaign initially ran under the stale "PhaseEnd next" stopping point → updated mid-session (Drew caught it earlier in the phase) | the session snowballed from "quick giant test" into the map breakthrough |
| Effort | Max | Max throughout (deep synthesis + orchestration; Fable5/Opus agents via the Agent tool) | the campaign + map are deep single-thread orchestration, not breadth |

## Commit Message
```
(Phase 23 landed across many sessions T1–T10.9 + the 07-02 giant/map session; commits commit:0424..commit:0433 + this PhaseEnd.)

feat(phase-23): LLM matching tier + the Fable5 wall-breaker + the gcc-2.7.2 codegen map (v1.22.0)

- LLM TIER (T1-T10.9): tools/serve_local + api_draft + lora_grind + bulk_harvest + the LoRA
  pipeline (v3 model). Local v3 saturated <=15 (~1300 banks, $0). Frontier GLM-5.2 (~$4)
  confirmed the def-side wall INTRINSIC + the idiom well DRY at the leaf (both angles).
- THE PIVOT (07-02): Fable5Max reading the gcc-2.7.2 SOURCE matched func_8014EE14, a §20/§10
  store-vs-load giant "CONFIRMED unsteerable" for 22 phases -> the "wall is intrinsic" verdict
  is MODEL-RELATIVE. Generalized to Opus agents applying §30 (2 more giants, 3-5x cheaper).
- §31 THE CODEGEN MAP: 4 Fable5 agents read the whole gcc-2.7.2 source -> docs/gcc-2.7.2-map/
  {sched,regalloc,loop,cse_expr}.md (935 lines, byte-proven residual->lever catalogs) +
  cookbook §31 (index+triage). Broke hoist-vs-remat/delay-slot/coalescing walls. Correction:
  gcc-papermario is 2.8.1 not 2.7.2 -> tools/reference/gcc-2.7.2/ (SETUP §5.6).
- THE WAVE (8 reach-134 near-misses, Opus + §31): 3 leaf-MATCH by lookup (47-137k tok vs
  200-250k walls) + 5 tight permuter seeds (func_80132784 240->4-off!), zero dead-ends.
- 5 giants/hard-fns banked x134 (func_8014EE14/F2E0/150528/149374/144090); fleet 63.66->64.86%,
  136/136 byte-identical, 0 NON_MATCHING. cookbook §30/§30a/§31. No new governance rules.
- FINDING: matching is SOLVED by the map; bottlenecks are now the permuter (S11 seeds) +
  whole-binary integration (auto-bank leaf-MATCHes) -> Phase 24. worklog -> logs/Phase23.md.
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **None (governance).** Per the Phase-8+ precedent, the phase's lessons are *findings + techniques*, captured where they belong: the **matching toolkit + the codegen map** → cookbook §30/§30a/§31 + `docs/gcc-2.7.2-map/`; the **LLM-tier discipline** → `docs/gen2-mips-matching-model.md` + the `cheap-tier-ab-validated` memory; the **"a wall verdict can be model-relative — a frontier model reading the compiler SOURCE breaks classes GLM/7B can't"** meta-lesson is a direct extension of **R14** (verify vs bytes) + **R17** (web/read the compiler internals) + the `dont-conclude-unsteerable-try-register-pins` memory — now generalized to "read the source, build the map." The existing rules (G3/P9 byte-gate, R14, R16 flywheel, R17, R22, R26/R27 effort, R30 capture-during-session) covered all the new ground. | A build → pivot → breakthrough phase yields tools, a knowledge artifact (§31), and findings — not a new norm of conduct. Drew may later elevate the `dont-block-loop-with-askuserquestion` memory or a "map-before-grind" practice to a formal rule if he wants it binding. |

## PhaseEnd Changelog
**v1.21.0 → v1.22.0 — Phase 23 complete (Gen2 phase 15; the breakthrough phase).** Built the **free local LLM matching tier** (v3 QLoRA + serving + the harvest drivers), which **saturated the ≤15-ins band for $0 (~1,300 banks)** and then **byte-proved its own ceiling** — the def-side loose-typing wall is intrinsic and the idiom well is dry *at the leaf, for a 7B/GLM*. Drew pivoted to **hand-solving giants**, and the phase's breakthrough followed: **Fable5Max, reading the actual gcc-2.7.2 source, matched a giant on a class 22 phases had "CONFIRMED unsteerable"** — proving the "wall is intrinsic" verdict was **model-relative**. This generalized twice: **standard Opus agents applying the now-documented idioms** (§30/§30a) banked more giants at 3–5× lower cost; and reading the **whole compiler source once** (4 Fable5 agents → the **§31 codegen map**, `docs/gcc-2.7.2-map/`, 935 byte-proven lines) **broke multiple long-standing walls** (hoist-vs-remat, delay-slot, coalescing) and was **proven by an 8-function wave** (3 leaf-MATCH by cheap lookup + 5 tight permuter seeds, incl. the 400-ins `func_80132784` driven to **4-off**). Along the way it corrected a 5-phase-old error (the source reference was gcc **2.8.1**, not 2.7.2). **5 giants/hard-fns banked ×134**, fleet **63.66% → 64.86%**, 136/136 byte-identical, 0 NON_MATCHING. **No new governance rules** (techniques → cookbook §30/§30a/§31 + the map). The decisive shift: **matching is now solved by the map** (cheap lookup, cheap-tier-usable); the remaining bottlenecks are throughput plumbing — the **permuter** (for the register-permutation seeds) and **whole-binary integration** (for auto-banking leaf-MATCHes) — which become Phase 24.

## Plain-English Recap
This was the phase where the project cracked its hardest problem. We started by building a free, private AI (running on your own graphics card) that reconstructs the game's simple code for **no cost** — it finished off the small functions (~1,300 of them). But it hit a wall on the hard code, and we proved the wall was real *for a small model*. Then the breakthrough: we pointed **the best AI model in the world at the actual source code of the 1990s compiler** — and it *solved* functions we'd sworn were impossible, by understanding exactly *why* the old compiler arranged instructions the way it did. Even better, ordinary AI agents could then reuse those tricks cheaply. So we did the smart thing you suggested: instead of re-figuring-out the compiler for every function, we had four AI agents **read the entire compiler source once and write a permanent "cheat sheet"** — for every way the compiler scrambles our code, either the exact fix or an honest "this one needs the brute-forcer." We tested the cheat sheet on eight hard functions: three were solved instantly by *looking up the answer*, and the other five were driven to the brink (one giant went from "abandoned" to *four instructions* from perfect) — with **zero** of the expensive dead-ends we used to hit. The game is now **~65% rebuilt bit-for-bit**, and — more importantly — we've turned "reverse-engineer the compiler" from a per-function slog into a cheat-sheet lookup that even the cheap AI can use. What's left isn't *matching* anymore; it's two pieces of **plumbing** — a smarter brute-forcer and an auto-installer for the finished code — which is next phase.

## Phase-24 Backlog — "Fix + EVOLVE the permuter · build the integration tool · scale the wave" (the forward bridge — nothing here is lost on close; Drew emphasized storing full context)

> Phase 24 opens in **plan mode, Max** (Tier-1). The matching problem is solved by §31; **the two remaining bottlenecks are throughput plumbing.** Prioritised:

**1. [HIGHEST BYTE-WEIGHT] FIX + EVOLVE THE PERMUTER (Drew's "evolve it to help more" idea — a first-class design goal).**
- **Why:** the §31 wave produced **register-permutation / schedule seeds** dominated by the **S11 LUID⊗alloc coupling** class — structure-exact drafts that no C lever moves (the map correctly routes them to the permuter). These are high byte-weight and *right there*: **`func_80132784` (4/400 — the 400-ins flagship, ×134 ≈ +1.6% byte-weight alone)**, `func_801412A8` (198/198 structure-exact), `func_8014E048` (35/143), `func_80176D94` (52/152), `func_80148094` (front-half exact), plus earlier `func_801770E0`/`func_8014EA4C`. **All seeds are in `.run/wave/` (+ `.run/toolkit/`, `.run/backlog_drafts/`).**
- **The current permuter is BLOCKED** (tools/decomp-permuter + tools/permuter/ + tools/grinder.py; recipe cookbook §3/§3a): (a) **pycparser can't parse the `__asm__("$sN")` register pins** the toolkit drafts use → it strips/breaks them (both wave sched agents hit this); (b) it scores against a **leaf compile with relocation noise** (§5a floor), not the real bytes → it wanders (func_80176D94: base 225, best 210, never 0); (c) custom typedefs (GsOT/OTLINK) break its turnkey path. Phase-22 audit: grinder banked 7 all-time (all Phase 21), **0 since** — the *random* permuter's tail is exhausted.
- **FIX (baseline):** (a) GNU-asm-declaration support — preprocess the pins to a permuter-safe placeholder + re-inject (or patch pycparser); (b) a **relocation-masked scorer** (mask jal/HI16/LO16 exactly like `tools/match_one.py`, which == the whole-binary byte-gate for a standalone reach-134 fn) so it scores true closeness; (c) custom-typedef dir setup.
- **EVOLVE (the big idea — the §31 map GUIDES the search):** instead of random mutations, make the permuter **§31-class-aware** — the agent already diagnosed each seed's residual class, so DIRECT the mutation set: **S11 register-permutation** → mutate the levers §31 says drive `allocno_compare` density (**declaration order**, def/use placement, pin scope — regalloc.md RC-1/RC-2/RC-3); **S3 chain-priority** → schedule-affecting statement reorders (sched.md); **RC-6 pressure-lock** → a stabilizing pin + targeted perturbation. This turns a random walk into a **directed search over the §31 lever space** — the map's second payoff (it guides not just agents but the permuter). Implement as PERM_* macros / a class-seeded mutation set.
- **EVOLVE further (the ML wildcard — carried from Phase 15/16, now well-motivated):** Drew has a 3080Ti/CUDA + LightGBM. Train a **candidate-ranker** on the banked corpus (asm/RTL features → likely-closeness) to prioritise the permuter's search, or an ML asm→C scorer. A real wildcard to evaluate once the directed permuter is running.

**2. [UNLOCKS WAVE THROUGHPUT] THE INTEGRATION-RECOVERY TOOL (§30a "integration is mechanical → build the tool", now urgent).**
- **Why:** §31 leaf-MATCHes **do not auto-bank whole-binary** — the wave banked only 1 of 3. The friction classes seen this phase: (a) **def-side return-type macro-widen** — a `DEFINE_func_*` caller macro declares the fn `extern void` but the matching def needs `s32` (a void return DCEs a live local); widen the macro's extern `void→s32`, byte-neutral IFF the caller discards the return (§30#2; **all 3 banked giants** hit this — `func_8014EE14/F2E0/150528`, verified byte-neutral via check-all); (b) **TU-context perturbation** — a leaf-MATCH whose whole-binary bytes differ with **no caller macro involved** (`func_80155800`, `func_8014F4C0` — drafts in `.run/wave/`; a different declaration/context effect, needs diagnosis); (c) **propagation stragglers** — a banked fn's dedup group has a cross-overlay member that isn't byte-identical (loose-typing/overlay-local) → the ×134 lift drops to ×1 (`func_80144090` banked ×1; a pre-existing `0x80165CA0`/`0x8012A018` straggler); (d) **data-extern type conflicts** (`func_8014EE14`'s `D_80126720` `u8[]` vs `s16`).
- **The tool:** a `gate_stage` recovery pass that, on a leaf-MATCH-but-whole-binary-near fn, tries in order: **macro-widen (a)** → re-gate; **TU-context diagnosis (b)**; **straggler recovery (c)** (bank ×134 where clean, ×1 where straggler-blocked, log the reason). Goal: **every §31 leaf-MATCH auto-banks** (with the reason logged when it can't reach ×134). Test cases: the unbanked wave leaf-MATCHes `.run/wave/func_80155800.c`, `func_8014F4C0.c`. Context: the recovery pipeline is `gate_stage.py` (canon_resident_calls → cast_call_sites → sig_unify → harvest_verify → dedup_propagate); the propagation cap/straggler is cookbook §19/§20; the macro-widen is §30#2 (manual now).

**3. [THE HARVEST] SCALE THE §31 WAVE (after 1+2).** With the permuter closing S11 seeds (1) and the integration tool auto-banking leaf-MATCHes (2), scale the §31-powered Opus-agent wave over the **full near-miss backlog** (~95 "schedule"/80 "regalloc"/31 "hoist" verdicts — most now steerable per §31). Each match ×134. Batch ~8; triage-first (bail to permuter on confirmed-intrinsic); gate winners together. The map + the two tools convert the backlog into a harvest.

**4. [FREE-TIER FLYWHEEL, carried] v4 retrain on the idiom corpus.** Once the wave banks many examples per idiom, retrain v4 on the banked-with-idioms corpus so the **free** local tier applies the *common* steerable idioms in straightforward cases (per the in-phase v4 discussion — a real but bounded gain; the 7B has a reasoning ceiling + does nothing for the intrinsic walls). Also worth a $0 experiment: §31 *in v3's prompt* on a few medium near-misses to gauge the ceiling. Post-wave.

**5. [BYTE-WEIGHT, carried] The giants + the compile-blocked 8.** The 20 giants are 35% of remaining byte-weight; the permuter (1) is the key to the S11 seeds. The **8 compile-blocked giants** (region `a`) are **mechanically** unblocked (ghidra-type aliases + data externs — NOT struct-walled; type-map finding this phase) — a cheap pass makes them draftable, then §31 + Opus.

**6. [STRATEGIC, carried] The public flip** (Fable5 §6.3, deferred to Gen3+) remains the community-labor lever; the LLM tier + the map make the repo an attractive contributor on-ramp when it's time.

**Carried LLM-tier state:** v3 = `models/bfm-match-7b-v3`; serve via `tools/serve_local.py` (Unsloth GPU, NOT LM Studio); `bulk_harvest.py` for the ≤15 tail (~92% saturated, ~383 low-value untried left); corpus `datasets/match_pairs/`; OpenRouter key in `.env` (`open_router_key`, ~$21/$25 left). serve_local is DOWN, GPU free.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase23.md` (R19). The Phase-23 work is committed across the session (`commit:0424`..`commit:0433`); **Drew commits AND pushes** this PhaseEnd + the archived log + the regenerated digests (`docs/progress.fleet.md`) (R6/R8 — the milestone-close is the human-controlled gate). No Ghidra DB change (R23 no-op — the `db.*.gbf` churn is restart-noise; do NOT stage; the pre-session `.gitignore`/`docs/backlog.md` edits are unrelated). Keep this file forever. Gen2 continues — do **NOT** start Phase 24 here. Start a **fresh session** (effort **Max**, **plan mode**) for **Phase 24 — Fix + evolve the permuter (§31-class-aware) · build the integration-recovery tool · scale the §31 wave** (the Backlog above; permuter first — it's the highest byte-weight lever, and the flagship `func_80132784` is 4 instructions from ×134).