BFM-decomp/phase-ends/PhaseEnd_Phase18.md

# PhaseEnd — Phase 18: Compiler-quirk research → the matching TOOLKIT, taught to waves at scale
**Date:** 2026-06-20 · **Project Version:** 1.17.0 · **Phase Status:** Complete (milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (10th phase)

> Gen2 phase 10 of the arc (8→9→10→11→12→13→15→16→17→**18**; 14 deferred to Gen3+). Ran across **multiple
> sessions**; the granular per-task/per-session trail (T0–T7, the verdict reversal, the 5-avenue ceiling test,
> the Step-3b calibration, the Step-1 wave + recovery diagnostics) is preserved on-demand at
> **`phase-ends/logs/Phase18.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). This
> file is the synthesis. Owner decisions (Drew): pursue the research → on the "unsteerable" finding, **hand-match
> everything** (verdict reversal) → teach the toolkit to a calibration wave → **bounded ~40 Step-1 wave**
> (full tractable-247 sweep deferred to Phase 19) → close with a full Phase-19 backlog.

## Build Log

**Files created/changed and complete — do not recreate:**

*The research + the matching TOOLKIT (T0–T7 + verdict reversal):*
- `tools/reference/` (gitignored, SETUP §5.6) — cloned **gcc-papermario** (`a6afc2af`, the PSX gcc-2.7.2 lineage
  source) + **xenogears-decomp** (`f27c0768`, Square Oct-1998, our EXACT compiler). The primary-source corpus.
- `tools/wall_taxonomy.py` + `docs/wall-taxonomy.md` — census/size every residual class (carried from P17 T1).
- `tools/ram_probe.py` + `docs/actor-struct.md` — the actor/arg0 struct, recovered + live-verified (R10).
  **Re-confirmed byte-NEUTRAL for matching** (matching reads access width off the asm opcode, not a struct def).
- `tools/ghidra_scripts/DecompileFunctions.java` — headless Ghidra-C batch pre-pass (no /mcp); feeds the
  `.run/ghidra_c/` cache (300 fns) the waves draft from.
- `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py` — the **canonical-sig layer** builders
  (one byte-neutral `s32 func_X(s32…)` per conflict callee; arity from Ghidra-C + asm read-before-write $a0–$a3).
  The 20-extern block at the top of `src/ov_SC01_077/ov_SC01_077.c` (LOCAL; reach-1 names differ per overlay).
- `docs/hand-matching-process.md` — §1 loop / §2 idioms / §3a 5-move signature playbook / §7 wave / §8 the
  canonical-sig-layer finding / §8e Phase-18 outcome (corrected to the register-pinning toolkit).
- `docs/matching-cookbook.md` **§17** — **the matching TOOLKIT** (the phase's durable deliverable): the
  call-crossing register-ORDER class is **STEERABLE with `register __asm__` PINS + a scheduling barrier**
  (corrected from a wrong mid-phase "unsteerable" verdict); **array-decay** forces rematerialization;
  **for-loop / statement-order** control scheduling; the **`sig_unify`-MANDATORY** gate gotcha; the
  **narrow-param loose-typing** wall (stub). §16 reconciled (§10-vs-§16 tension resolved).
- `docs/matching-cookbook.md` **§17a (THIS SESSION)** — the toolkit **at wave scale** (measured close-rates) +
  the **pipeline-integration recovery** techniques (call-site casts not redeclaration + re-validate; stale
  sibling `M2C_UNK` decl reconcile; the implicit-int caller class) + **two NEW residual classes** (per-file
  `-O0`; gcc loop-guard operand-order).
- `src/shared/engine_core.h` — the worked-example macros `DEFINE_func_8012B8E4` (pins + barrier) and
  `DEFINE_func_801399A8` (for-loop/structural) + **~33 new Phase-18 reach-134 macros** (the wave harvest).
- `config/dedup.us.yaml` — **1450 validated cross-binary groups** (was ~1424 at phase start; +the Phase-18
  propagations). 0 failed.
- `src/ov_SC01_077/ov_SC01_077.c` + the **134 overlay `.c` files** — the harvest source + the ×134
  propagations; 2 recovery reconciles (func_8012A418 stale-sibling-decl; func_8012F68C parse-fix).

*The Step-3b/Step-1 wave harness (gitignored under `.run/`, regenerable; the recipe is cookbook §17/§17a):*
- `.run/harvest_wave_3b.js` (calibration, 16 targets) + `.run/harvest_wave_p18s1.js` (Step-1, 31 targets,
  +embedded canonical callee sigs +call-site-cast/re-validate). `tools/gen_harvest_targets.py` generates the
  per-target callee-sig/data-decl manifest the Step-1 prompt embeds.
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase18.md`** (R19 archive). `phase-ends/PhaseEnd_Phase18.md` — this file.

**Local artifacts (gitignored / regenerable):** `.run/ghidra_c/*` (300 cached Ghidra-C), `.run/drafts-*`
(wave drafts + gate-staging dirs), `.run/p18/*` (T1/T2-T3 analyses, exemplar scratch), `.run/harvest_*.js`,
`.run/*_targets*.json`, `build/`, `expected/`, `asm/`, `assets/`. `tools/reference/` (the cloned gcc/Xenogears
source). The 134 overlay Ghidra programs are script-reproducible (DB-commit deferred as regenerable).

**Tools/packages installed:** None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv.
(Wine deliberately NOT installed — W1 ruled build-divergence out by proxy; SETUP §4.8 unchanged.)

**Verification results (literal) — the milestone proof:**
- **`make check-all` → 136 passed, 0 failed of 136** (main `143dbb89…`, resident `8e17e02f…`, all 134 overlays
  byte-identical; ov_SC01_077 `d19c9580…`), reproduced from clean after every step (R22).
- **`make report`:** FLEET byte-identical **194,839 / 344,010 = 56.64%** (phase start 55.51%); **dedup-check
  1450 validated, 0 failed**; **0 NON_MATCHING** in any default build (G4).
- **The TOOLKIT, byte-gated per class:** register-ORDER → `func_8012B8E4` MATCH via pins+barrier (the flagship
  "unsteerable" fn); array-decay → `func_8012B4B8` 88→52, `func_80149544/584` MATCH; structural/for-loop →
  `func_801399A8` MATCH. Honest stubs: narrow-param (`func_80146A6C`), loop-guard order (`func_8012C2D0`),
  §10 sched (`func_8014F2E0`), per-file -O0 (`func_8013C360`+~17).
- **The toolkit at WAVE scale (the bounded demonstration):** Step-3b calibration **9/16 = 56% whole-binary**
  (vs the Phase-17 prompt's 33%); Step-1 (prompt v2: +embedded sigs +call-site-cast) **28/31 = 90% `match_one`,
  22 verified**. **Session fleet arc 55.58% → 56.64% (+1.06%, ~31 reach-134 matches; phase total 55.51→56.64 =
  +1.13%).**
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk
  staged**. **No Ghidra DB change** this phase (matching used cached Ghidra-C + asm; the `db.*.gbf` churn is
  SessionStart-restart noise — R23 no-op, do NOT stage).

**Milestone achieved (confirmed by Drew, gate 2):** the knowledge-gated milestone — a per-class byte-gated
verdict (validated C idiom OR honest "unsteerable" verdict naming the exact gcc pass), §10-vs-§16 reconciled,
≥1 quirk-tail function newly matched, 136 binaries byte-identical throughout — is **met and exceeded**. The
phase's headline reversal: the highest-reach **call-crossing register-ORDER class is STEERABLE** (`register
__asm__` pins + a scheduling barrier), not "unsteerable" as a mid-phase draft wrongly concluded. The toolkit
was then **taught to a parallel Ultracode wave** and demonstrated at scale (close-rate 33% → 56% → 90%
`match_one`), banking ~31 reach-134 matches (fleet +1.13%).

**Next:** **Phase 19 — Scale the toolkit (the wave-harvest at scale) + the cheap front-loaded levers.** A fresh
session, **plan mode** (Tier-1 Phase-Start). See the **Phase-19 Backlog** below for the full prioritized brief
(the -O0 split-file fix first, recovery-tooling, then 50-target waves scaling to 100+, with an optional bounded
gcc-research spike on the last residual classes). The infra is BUILT and staged.

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Register-ORDER verdict | T1–T5 concluded "UNSTEERABLE from C" | **REVERSED — STEERABLE via `register __asm__` pins** | I'd skipped the most direct lever; Drew's "we hand-match everything" pushed it; `func_8012B8E4` byte-gated (P9/R14 self-correction). The "brute waves won't help" point stood; the HAND pin lever does. |
| Phase scope | research + a bounded quirk-tail demo (gate-1) | research **+ a calibration wave + a bounded Step-1 wave** | the reversal made the toolkit teachable to waves; demonstrating it at scale (33%→56%→90%) is the real proof it raises the ceiling |
| Step-1 size | "scale the wave on the tail" | **bounded 31 targets** (Drew chose ~40; full tractable-247 → Phase 19) | gate-1 deferred the full sweep to Phase 19; Step 1 = a bounded demonstration of the taught wave |
| Wave close-rate gap | (not anticipated) | `match_one` MATCH ≫ whole-binary verified (12→9, 28→22) | the gap is **declaration plumbing** (call-site casts the agents skip + `sig_unify` narrowing the extern), NOT codegen — recovered deterministically (cookbook §17a) |
| W1 Wine native-compiler check | bounded Wine run | **ruled out by proxy, Wine dropped** | 700+ matches prove our cc1's regalloc is faithful; the real compiler emits the same bytes → confirms, doesn't crack; heavy install not worth it (plan escape clause) |
| Effort | Max planning, xHigh/Ultracode execution | Max/xHigh research + debugging; **Ultracode for the waves** (Drew toggled); xHigh for this close | breadth-shaped waves under Ultracode (R26/R27); deep synthesis at xHigh (Drew's saved default) |
| New residual classes | (not anticipated) | **per-file -O0** + **gcc loop-guard operand-order** surfaced at scale | the waves exposed classes the single-function research hadn't; the -O0 one is build-infra (high ROI), not a wall |

## Commit Message
```
(Phase 18 landed as 16 commits commit:0141..commit:0156 + this PhaseEnd.)

feat(phase-18): compiler-quirk research -> the matching TOOLKIT, taught to waves (v1.17.0)

- RESEARCH (T0-T7): read gcc-2.7.2 source (gcc-papermario) + mined Xenogears (our exact
  compiler) -> per-class byte-gated verdicts. VERDICT REVERSAL (Drew "hand-match everything"):
  the call-crossing register-ORDER class is STEERABLE via `register __asm__` pins + a scheduling
  barrier (func_8012B8E4, the flagship "unsteerable" fn, byte-gated) -- not unsteerable.
- TOOLKIT (cookbook §17): pins+barrier (register swap), array-decay (hoist-vs-remat),
  for-loop/statement-order (structural), sig_unify-mandatory; narrow-param = honest stub.
  §16 reconciled (§10-vs-§16).
- TAUGHT TO WAVES (§17a): calibration 9/16=56% whole-binary (vs 33%); Step-1 (embedded
  canonical callee sigs + call-site-cast/re-validate) 28/31=90% match_one, 22 verified.
  ~31 reach-134 matches banked, each propagated x134. fleet 55.51 -> 56.64% (+1.13%).
- FINDING: the match_one->gate gap is declaration plumbing (call-site casts not redeclaration;
  stale M2C_UNK sibling-decl reconcile; implicit-int caller class) -- recovered deterministically.
- NEW residual classes: per-file -O0 (~18 fns, needs an -O0 split file like src/boot.c -> high
  ROI) + gcc loop-guard operand-order (irreducible). W1: build-divergence ruled out by proxy.
- make check-all 136/136 byte-identical (R22); dedup 1450 validated/0 failed; 0 NON_MATCHING (G4).
- worklog -> phase-ends/logs/Phase18.md (R19); bumps 1.16.0 -> 1.17.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **None (governance).** Phase 18's lessons are *findings + techniques*, recorded where the Phase-8/11/13/15/16/17 precedent puts them (cookbook, not rules): the **register-pinning toolkit** + the **call-site-cast/re-validate** + the **sibling-decl/implicit-int recovery** + the **per-file -O0 and loop-guard residual classes** → cookbook §17/§17a + hand-matching §8e. The **verdict-reversal lesson** ("never conclude *unsteerable* before trying `register __asm__` pins; an intermediate metric is not the gate") is already governed by **G3/P9/R14** and the `dont-conclude-unsteerable-try-register-pins` memory. The **probe-before-investing / bounded-calibration-before-scale** discipline is already cookbook (Phase 15) + effort-map. No new norm of conduct emerged. | A research→toolkit→demonstrate phase produces knowledge, not governance. Existing rules covered the ground and the phase reinforced them (esp. R14: the "unsteerable" over-claim was an R14 failure mode, self-corrected). |

## PhaseEnd Changelog
**v1.16.0 → v1.17.0 — Phase 18 complete (Gen2 phase 10).** Set out to "understand gcc-2.7.2 to raise the
match-% ceiling," and delivered a **matching TOOLKIT** that does exactly that. Reading the real gcc-2.7.2 source
+ mining Xenogears (our exact compiler) produced per-class byte-gated verdicts — and a **decisive reversal**
(Drew's "hand-match everything"): the highest-reach **call-crossing register-allocation-ORDER class is
STEERABLE** with `register __asm__` pins + a scheduling barrier (the flagship `func_8012B8E4`, byte-gated),
*not* "unsteerable" as a mid-phase draft wrongly concluded (a clean R14 self-correction). The toolkit
(pins+barrier · array-decay · for-loop/statement-order · sig_unify-mandatory; narrow-param = honest stub) was
then **taught to a parallel Ultracode harvest wave** and demonstrated at scale: whole-binary close-rate **33% →
56%** (Step-3b calibration) **→ 90% `match_one`** (Step-1, after embedding each callee's canonical signature
and teaching call-site casts). **~31 reach-134 functions matched** (each propagated ×134), driving the fleet
**55.51% → 56.64%**, all 136 binaries byte-identical, 0 NON_MATCHING. The phase's reusable yield: the toolkit
itself (cookbook §17), and the **pipeline-integration recovery** insight (§17a) — the `match_one`→gate gap is
**declaration plumbing** (call-site casts, stale-sibling-decl reconcile, implicit-int callers), recoverable for
~0 agent tokens, not a codegen wall. Two new residual classes surfaced at scale: the **per-file -O0 class**
(~18 byte-exact-at-O0 functions needing their own -O0 split file — a high-ROI build-infra fix, not a wall) and
the **gcc loop-guard operand-order** residual (irreducible). W1 ruled build-divergence out by proxy (Wine
dropped). No new governance rules (techniques → cookbook). The harvest infra is BUILT and staged for Phase 19's
scale-up.

## Phase-19 Backlog — "Scale the toolkit" (Drew's research + wave recommendations, full context)

> Phase 19 = **scale the proven toolkit over the tractable reach-134 tail**, front-loaded with two cheaper
> levers, run as a flywheel. Detailed plan is a fresh-session Tier-1 Phase-Start task; this is the prioritized
> brief, by ROI. (Decided with Drew at the Phase-18 close.)

1. **[HIGHEST ROI — DO FIRST] The per-file `-O0` split-file fix (build-infra, no research, no agents).** ~18
   functions in ov_SC01_077 were built **-O0** (prologue sig `21F0A003` = `addu $fp,$sp,$zero`; args spilled to
   frame; load-delay nops; redundant `addu rd,rs,$zero`). Their correct C is **byte-exact at -O0** but the
   overlay TU compiles `-O2`, and **gcc-2.7.2 has no per-function optimize pragma** (opt is per-file, Makefile).
   Fix = give them their own `-O0`-compiled split file with a target-specific `CC1FLAGS := -O0 …` (the
   **`src/boot.c` precedent**, Phase 7). ~18 fns × reach-134 for an afternoon of Makefile work. Members:
   func_8013C360 + func_8013B568/B598/B6A0/B7AC/B7F4/B83C/BC7C/BCDC/BD34/BD74, func_8013C08C/C0F8/C414/C938/C964,
   func_80144B9C, func_801457A4. (One agent already produced verified-correct -O0 C for func_8013C360 — see logs.)
2. **[COMPOUNDING — DO SECOND] Recovery-tooling investment between batches.** This session's deterministic
   recoveries (call-site arity/void casts; stale `M2C_UNK` sibling-decl reconcile) banked matches for **~0 agent
   tokens**. Fold them into the gate so the close-rate rises mechanically: teach `sig_unify` (or a post-pass) to
   **auto-apply call-site arity/void casts** when a draft's call disagrees with the canonical extern, and to
   **auto-reconcile stale sibling `M2C_UNK func_X();` decls** to the real sig. Also solve the **implicit-int
   caller class** (3 deferred fns: func_80147514/8017209C/80168F40) — the clean path is **propagate-first**
   (the def lands in `engine_core.h`, included first, so no implicit-int forms), gate-after; build a small
   helper or a `dedup_propagate` mode for "matched-but-fails-in-place-gate" functions.
3. **[THE CORE — DO THIRD] Scale the toolkit waves over the tractable-247 reach-134 tail.** Setup: regenerate
   Ghidra-C (`DecompileFunctions.java`, headless) for the uncached tractable fns (the `.run/ghidra_c/` cache is
   only 300; this session's fresh cached+stub pool was ~27). Then run the **Step-1 wave prompt (`.run/
   harvest_wave_p18s1.js` recipe = §17 toolkit + embedded canonical callee sigs + call-site-cast/re-validate)**.
   **Batch size: 50, banking a prompt+tooling improvement each batch for the first 2–3, then scale to 100+ once
   the close-rate plateaus.** Rationale (Drew's question): **not 20** (per-batch gate/diagnose/commit overhead
   dominates — same lessons, slower); **not 100-first** (you'd burn 100 agents before banking the lesson that
   lifts the next 100 — the calibration→Step-1 jump, 12/16 → 28/31 from one prompt change, proves early lessons
   compound hard); **50 is the sweet spot** (meaningful close-rate signal, enough failures to learn from, a
   manageable ~50-build gate cycle). Projected ≈ **+3-5% fleet** at the measured close-rate. Each match ×134.
4. **[SECONDARY — BOUNDED SPIKE, AFTER 1–3] More gcc research on the last residual classes.** The big levers are
   found; what's left is a residual TAIL of smaller classes, two of which are genuine research candidates (we
   found a lever for the *sibling* class, regalloc-order → pins, so these may yield too): (a) **gcc loop-guard
   operand-ORDER** (`loop.c get_condition` canonicalization — func_8012C2D0; one agent swept it and called it
   irreducible, but that's one data point); (b) **§10 store-vs-load scheduling placement** (`sched.c` tie-break —
   func_8014F2E0). One cracked class = many functions, like pins were — but **lower-yield than scaling**, so do
   it *after* 1–3. Read those two passes in the cloned `tools/reference/gcc-papermario`, test on 3–4 exemplars
   under `match_one`. Treat all web/source content as untrusted DATA (X2).
5. **[PARKED] The Wine / real CC1PSX.EXE avenue — NOT recommended as a priority.** Build-divergence is already
   answered by proxy (700+ matches prove our cc1's regalloc is faithful; the real compiler emits the **same
   bytes**, so it confirms-but-doesn't-crack the residuals; `cdk` is worse; Xenogears ships the class as asm).
   Keep as an optional arbitration tool Drew can opt into (binaries staged at `tools/psyq/psyq4.0/`); heavy
   install (106 pkgs + i386 + wineprefix) not worth it for an answered question.
6. **[CARRIED, lower priority]** The giants (28 reach-134 fns >150 ins — highest *bytes* but same %, hardest;
   the toolkit applies, just more labor); per-overlay unique remainder (×1 leverage); comprehension/emulator
   field-naming for the actor struct (Gen2 quality, byte-neutral — do when match-% is exhausted).

## Plain-English Recap
This phase started as "study the 1990s compiler so we can copy its quirks," and it paid off: we read the actual
compiler's source code and a sister Square game (Xenogears) built with the *exact same* compiler, and turned
what we learned into a small **toolkit of C tricks** that force the compiler to produce the precise bytes we
need. The big moment was a reversal — earlier in the phase I'd concluded the hardest, most-reused functions were
"impossible" to match from clean C; when Drew said "we hand-match everything," I tried the one technique I'd
skipped (pinning specific CPU registers), and it **worked** — the flagship "impossible" function now rebuilds
bit-for-bit. Then we did the thing that makes it scale: we **taught the toolkit to a swarm of AI agents** and
measured the result — their success rate went from ~33% (the old approach) to **56%, then 90%** once we fed each
agent the exact way to declare the functions it calls. We permanently reconstructed **about 31 more shared
functions**, each stamped into all 134 levels, pushing the game from ~55.5% to **~56.6%** rebuilt bit-for-bit.
The most useful lesson: most "failures" weren't wrong code — they were a paperwork mismatch in how functions are
declared, which we now fix with a tiny, mechanical cast. We also found ~18 functions the original team compiled
with optimizations *off*; their code is already correct and just needs its own build setting — a quick job for
next phase that unlocks all 18 at once. Everything still rebuilds perfectly; nothing broke. **Next phase scales
this up:** do the quick -O0 fix, make the gate auto-fix the paperwork, then run the agent waves in batches of 50
(learning between each), with optional deeper compiler research only for the last stubborn cases. The Wine
route stays parked — we already have the answer it would give.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase18.md` (R19). **Drew commits AND pushes**
this PhaseEnd + any uncommitted Phase-18 work (R6/R8 — the matching commits `commit:0152`..`commit:0156` are already
landed; this PhaseEnd + the archived log are the close). No Ghidra DB change this phase (R23 no-op — the
`db.*.gbf` churn is restart-noise, do NOT stage it). Gen2 continues — do **NOT** start Phase 19 here. Start a
**fresh session** (plan mode) for **Phase 19 — Scale the toolkit** (the backlog above; -O0 fix first, then
recovery-tooling, then the 50-target waves). Keep this file forever.