BFM-decomp/phase-ends/PhaseEnd_Phase21.md

# PhaseEnd — Phase 21: The Automation Manager (built, run, and its ceiling byte-proven)
**Date:** 2026-06-26 · **Project Version:** 1.20.0 · **Phase Status:** Complete (open-ended phase; closed at Drew's gate-2 after the automated ceiling + a collaboration sweep proved no external shortcut) · **Generation:** Gen2 (13th phase)

> Gen2 phase 13 of the arc (8→9→10→11→12→13→15→16→17→18→19→20→**21**; 14 deferred to Gen3+). Ran across **~10 sessions (cont.1–8, 2026-06-21 → 06-26)** — the longest phase. The granular per-session trail (every wave, the `_a`-vein discovery, the gate-fix archaeology, the giant-wave results, and the full 5-scout collaboration-sweep reports) is preserved on-demand at **`phase-ends/logs/Phase21.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). This file is the synthesis. Owner decisions (Drew): build the supervised-autonomous grind → run it **neverending** (cont.5) → on the byte-proven automated ceiling + a sweep proving no external shortcut, **close the phase and pivot to human-guided hand-grinding** (the giants, ×134).

## Build Log

**Files created/changed and complete — do not recreate:**

*The automation manager (the phase's core deliverable, T1–T6):*
- `tools/build_fuel_manifest.py` — classify live stubs (GIANT/WAVE/STRUCT/PINS/STUB/O0) + reach buckets → `.run/fuel_manifest.json` (reach RECOMPUTED from current overlay sigs, R14).
- `tools/backlog.py` — the near-miss ledger (`.run/backlog.jsonl` + `docs/backlog.md` ranked render; drop-now-matched P9 filter; `log`/`render`/`show`).
- `tools/gate_stage.py` — the shared deterministic spine (worker + grinder): `canon_resident_calls → cast_call_sites → sig_unify → harvest_verify --chunk 1 → dedup_propagate`; **canon-first two-stage** (cont.6 — sig_unify is a FALLBACK; it regresses already-correct drafts); `--src-file` for the `_a`/`_o0` splits.
- `tools/wave_targets.py` — ROI pool selector (reach1/tractable/giants/any-reach134/o0/capped; `--region a`; `--class`; excludes self-MATCH-but-gate-rejected churners).
- `tools/orchestrator.py` — `prep`/`finish`/`status`; close-rate-gated pool rotation; flock-serialized gates.
- `tools/grinder.py` + `tools/auto_supervisor.sh` (DRIVER-param) + `tools/auto_stop.sh`/`auto_status.sh` — the token-free decomp-permuter daemon (blacklist of permuter-won-but-gate-rejected; STOP-sentinel safe-exit; self-supervising).
- `tools/workflows/worker_wave.js` — parallel xHigh drafter agents (one/target, toolkit-aware, §27 GIANT-guarded); returns a compact per-fn `{status, closeness, klass}`. `tools/workflows/distill.js` — post-wave byte-verified idiom extraction → cookbook (the flywheel).
- `tools/idiom_loop.py` — the NEVERENDING meta-loop: `--assess` (cluster the backlog by gcc-quirk class, rank by reach, name the next idiom + the deterministic-recovery lever), `--gen-fuel`, `--fuel-status` (cont.7: two assess-honesty fixes — drop banked-since-logged; split FRESH-vs-WALLED close=0).

*Recovery tooling (the deterministic levers, cont.2–7):*
- `tools/canon_resident_calls.py` (resident-callee link-miss), `tools/fix_arity_callers.py` (no-proto caller-extern arity), `tools/cast_call_sites.py` (§17a-1 per-site fn-ptr cast — the §20 cap tool), `tools/build_engine_types.py` extended (typedef/anon-struct lift — the §19 propagation cap), `tools/exemplar_miner.py` (residual router).

*The cross-project dedup tool (cont.8 — the novel byproduct):*
- `.run/xdedup/xsig.py` (+ drivers) — relocation-masked cross-project function signatures (find byte-identical code across decomp projects). Self-validated (self-match / relocation-invariance / register-sensitivity / discrimination). **Future work logged: genericize + open-source as a standalone MIT tool (2026-06-26).**

*Config / source (the harvest, banked across the phase):*
- `src/ov_SC01_077/ov_SC01_077.c` + `_a.c`/`_o0.c` (the Phase-19 3-object -O0 split) + `src/shared/engine_core.h`/`engine_types.h` + the 134 overlay `.c` + `config/dedup.us.yaml` — banked matches + ×134 propagations. `config/symbols.us.txt` — **+62 PsyQ names** (cont.8, xdedup-confirmed, byte-safe, R22-verified).

*Docs / knowledge base (the mature gcc-2.7.2 toolkit):*
- `docs/matching-cookbook.md` **§18** (-O0 split inside a blob) · **§19** (wave-at-scale recovery + the propagation CAP) · **§20** (gate cap + the cast tool + the DEF-side loose-typing wall) · **§25** (the "schedule" idiom is COALESCING not scheduling; gate two-stage; h_exact OVER-counts ×134) · **§26** (cheap close=0 recovery EXHAUSTED; assess-honesty fixes) · **§27** (the giant matching recipe).
- `docs/SETUP.md` (§5.3 -G stale-marker fix, R21); `docs/automation-runbook.md`; **`docs/backlog.md`** (the ranked hand-finish ledger — the phase's other durable deliverable, → Phase 22 fuel).
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase21.md`** (R19 archive). `phase-ends/PhaseEnd_Phase21.md` — this file.

**Tools/packages installed:** None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv (decomp-permuter, m2c, rabbitizer, the cc1 candidates).

**Verification results (literal):**
- **Fleet: 58.82% → 63.18% byte-identical** (+4.36%; `make report` → committed `docs/progress.fleet.md`); **136/136 binaries byte-identical maintained throughout** — every bank was byte-gated (G3/P9: the whole-binary byte-gate is the per-bank proof, and it **never accepted a wrong match**); **0 NON_MATCHING** in any default build (G4); `dedup-check` validated, 0 failed.
- **Main EXE re-verified `143dbb89f34491258bbc27810d0a12ec8b43a8dd` at close** (cont.8 — R22 clean-rebuild of `main` after the 62 PsyQ-name renames; the renames are byte-neutral).
- **The automated ceiling, byte-proven (cont.7/7d/8):** the cheap close=0 recovery lever is EXHAUSTED (0 fresh banks); giant auto-waves bank **0/6 twice** (giants are hand-finish / permuter fuel, not auto-bankable); the cached reach-134 automated harvest is COMPLETE at ~63.17%.
- **Collaboration sweep (cont.8) — 5 scouts, clean verdicts:** parity research (our harness IS the 2025-26 SOTA; no neural/GPU/diffusion 1:1 matcher exists); decomp.dev survey (4 gcc-2.7.2-psx twins, byte-verified our cc1 ≡ the psx lineage; **VS = CC0**); census (NO Square PS1 game shipped symbols); VS corpus (76 byte-verified pairs); **dedup probe = CLEAN NEGATIVE** (BFM shares byte-identical code with VS/Xenogears/Tomba ONLY in PsyQ library; AKAO refuted; 0 engine code) ⇒ **no external shortcut to the gcc-2.7.2 wall**.
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; zero ROM-derived/generated bulk staged; **no Ghidra DB change** (R23 — the `db.*.gbf` churn is SessionStart-restart noise, not staged).

**Milestone achieved (confirmed by Drew, gate 2):** the supervised-autonomous **"automation manager" was built, validated, and run** (parallel-LLM-drafter waves + a token-free permuter grinder + an ROI orchestrator + a whole-binary byte-gate as the sole arbiter + a ranked backlog ledger + the distillation flywheel), driving the fleet **58.82% → 63.18%** across ~10 sessions at **136/136 byte-identical / 0 NON_MATCHING** — and its **ceiling was then byte-proven** (the cheap automated levers exhausted; a 5-scout collaboration sweep established there is **no external shortcut** — no AI/GPU method, no shared sister-game code, no debug symbols — to the gcc-2.7.2 scheduler/regalloc/loose-typing wall). The phase closes with a complete **ranked hand-finish backlog** + the mature §17–§27 toolkit, handing Phase 22 a clear, high-leverage human-guided target.

**Next:** **Phase 22 — Hand-grind the high-value residuals (giants ×134) + idiom distillation.** Open a **fresh session, plan mode, Max** (Tier-1 Phase-Start). Center on the **giants** (reach-134 — the high-byte-weight lever that moves the decomp.dev-comparable ~30% number; the §27 recipe gets them *structurally* matched, then hand-finish the regalloc-coalescing last mile, each banking **×134**); **distill transferable giant-finishing idioms** (which DO transfer giant-to-giant + to the wave tail) — Drew's flywheel pointed at the highest-leverage targets. Secondary: the close-1..4 reach-134 permuter near-misses (`func_8015126C` close=2, `func_8014EE14` close=6). **Honest milestone shape:** *N giants banked ×134 + any idioms distilled* — not "learn X idioms" (the cookbook is mature; new-idiom yield is the secondary benefit). Ranked targets: `docs/backlog.md` + the giant list in the archived log.

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Phase shape | build + a multi-day unattended run (5 idle days) | built + RAN across ~10 sessions (driven by Workflow-completion, not the keep-alive) + **closed on the proven ceiling** | the cheap automated levers exhausted far faster than "5 days"; the value was the harvested % + the ranked backlog + the ceiling proof |
| reach-134 fuel source | "the tractable reach-134 tail" | the **`_a`/`_o0` split-file vein** (cont.3–5) held the fresh reach-134 fuel; built split-aware tooling | the Phase-19 -O0 3-way split held ~66 fresh cached reach-134 fns the main-file pipeline had never waved |
| Phase duration | a bounded phase | declared **NEVERENDING** (cont.5, Drew) → then **closed** (cont.8, Drew) | the idiom-loop was neverending until the ceiling + the collaboration sweep made closing the right strategic call |
| Giants | "high-byte-weight auto-harvest" | **0 auto-bank (×2)**; reframed as **hand-finish backlog fuel** (§27 gets them structurally close) | byte-proven: giant residuals are regalloc-coalescing / masked-residual — not auto-bankable (cont.7c/7d/8) |
| Collaboration sweep | (not in the original plan) | a **5-scout sweep** (Drew-driven Q&A) closing the "is there an external shortcut?" question with clean byte-verified verdicts | Drew's strategic questions (neural/GPU surrogates, 100-games census, debug symbols, sister-decomp code-sharing) — each answered decisively |
| AskUserQuestion mid-loop | used to steer | rejected once mid-loop → memory `dont-block-loop-with-askuserquestion` | Drew wants the opted-in `/loop` to run + adapt autonomously, not be re-asked once direction is set |
| reach-1 ROI | a harvest lever | banked real ×1 fns but byte-confirmed **poor fleet-ROI** (×1; band climbs, yield falls) | value is ov_SC01_077 completeness + idiom-mining, not fleet-% — re-confirmed cont.8 |

## Commit Message
```
(Phase 21 landed across ~10 sessions of checkpoint commits; this close = commit:0292..commit:0294 + this PhaseEnd.)

feat(phase-21): close — automation manager built + run + ceiling byte-proven; fleet 58.82%->63.18% (v1.20.0)

- THE AUTOMATION MANAGER (built + validated + run): worker_wave (parallel xHigh drafters) + grinder
  (token-free permuter daemon) + orchestrator (ROI pool rotation) + gate_stage (whole-binary byte-gate,
  sole arbiter) + backlog ledger + idiom_loop/distill flywheel. Drove fleet 58.82% -> 63.18% across ~10
  sessions, 136/136 byte-identical, 0 NON_MATCHING.
- THE CEILING, BYTE-PROVEN: cheap close=0 recovery exhausted; giant auto-waves 0/6 (x2 — giants don't
  auto-bank, they're hand-finish/permuter fuel); the cached reach-134 automated harvest COMPLETE at 63.17%.
- COLLABORATION SWEEP (5 scouts, clean verdicts): we're SOTA (no neural/GPU/diffusion shortcut); 4
  gcc-2.7.2-psx twins (VS=CC0, the clean pool partner); no Square symbols; dedup probe CLEAN NEGATIVE
  (BFM shares only PsyQ with sisters, AKAO refuted, 0 engine code) -> NO external shortcut to the gcc wall.
  +62 PsyQ names banked (byte-safe).
- TOOLKIT MATURED: cookbook §18 (-O0 split) / §19 (wave recovery + propagation cap) / §20 (cast tool +
  DEF-side wall) / §25 (coalescing + gate two-stage + h_exact over-count) / §26 (recovery exhausted) /
  §27 (giant recipe). Novel byproduct: xsig cross-project dedup tool (open-source TODO logged).
- DELIVERABLE for Phase 22: the ranked hand-finish backlog (docs/backlog.md) + the giants (×134).
- worklog -> phase-ends/logs/Phase21.md (R19); bumps project version 1.19.0 -> 1.20.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **None (governance).** Phase 21's lessons are *techniques + a working-preference*, recorded per the Phase-8+ precedent (techniques → cookbook, not rules): the matching toolkit → cookbook **§18–§27**; the strategic findings (the automated ceiling; no external shortcut; the dedup clean-negative) → this synthesis + the archived log; the **AskUserQuestion-during-loop discipline** → memory `dont-block-loop-with-askuserquestion` (extends R26/R27 + `drew-working-preferences`). The existing rules — **G3/P9** (byte-gate is truth), **R14** (verify vs bytes), **R16** (flywheel), **R22** (clean-rebuild), **R26/R27** (effort/loop), **R30** (capture-during-session) — covered all the new ground. Drew may elevate the AskUserQuestion memory to a formal rule (P10) if he wants it to bind every session. | A build → run → pivot phase produces tools, findings, and one process nuance — not a new governance norm. |

## PhaseEnd Changelog
**v1.19.0 → v1.20.0 — Phase 21 complete (Gen2 phase 13; the longest phase, ~10 sessions; closed at a byte-proven ceiling + a strategic pivot).** The phase **built, validated, and ran the supervised-autonomous "automation manager"** — parallel-LLM-drafter waves (`worker_wave`) + a token-free permuter **grinder** + an ROI **orchestrator** + a **whole-binary byte-gate** as the incorruptible sole arbiter (`gate_stage`/`harvest_verify`) + a ranked **backlog ledger** + the **distillation flywheel** (`idiom_loop`/`distill`) — driving the fleet **58.82% → 63.18%** across ~10 sessions, **136/136 byte-identical, 0 NON_MATCHING** throughout. The economic engine was **cross-binary ×134 propagation** (match once → stamp across all sharers); the harvest reached fresh fuel via the **`_a`/`_o0` split-file vein** and a hardened deterministic-recovery pipeline. The gcc-2.7.2 **toolkit matured** (cookbook §18–§27: the -O0 split, the wave-recovery + propagation cap, the cast tool + DEF-side wall, the coalescing/gate-two-stage/h_exact-overcount findings, the cheap-recovery-exhausted proof, and the giant recipe). **The decisive arc of the close:** the cheap automated levers were **byte-proven exhausted** (close=0 recovery dry; giant auto-waves 0/6 twice — giants are hand-finish/permuter fuel), and a **5-scout collaboration sweep** (parity research, decomp.dev survey, census, VS corpus, the cross-project dedup probe) closed the "is there an external shortcut?" question with **clean verdicts: there is none** — our harness is the SOTA, no neural/GPU surrogate is a workflow, no Square game shipped symbols, and BFM shares only PsyQ library code with its gcc-2.7.2 sisters (the AKAO theory refuted, 0 engine code). Durable byproducts: **+62 byte-confirmed PsyQ names**, the **VS matched-corpus** (low-priority asset), and the genuinely-novel **`xsig` cross-project shared-code tool** (open-source TODO logged). No tools installed; **no new governance rules** (techniques → cookbook; the loop-discipline lesson → memory). The phase hands Phase 22 a **complete ranked hand-finish backlog** and a clear conclusion: the wall is intrinsic to a 1997 compiler, and the only path forward is **human-guided hand-grinding** — with the **giants (×134)** as the highest-leverage target.

## Plain-English Recap
Over about ten sessions we built and ran an "automation factory" for reconstructing the game's code: swarms of AI agents draft the C, a token-free brute-forcer polishes the near-misses, and an automatic bit-for-bit referee accepts only perfect matches — nothing wrong can ever slip in. It pushed the game from **58.8% to 63.2%** rebuilt exactly, every piece still byte-perfect. Then it hit a ceiling, and — importantly — we **proved** it's a real ceiling, not a gap in our effort: the cheap automatic wins are used up, the biggest "giant" functions won't auto-finish (they need a human brain for the last stretch), and a thorough investigation of *every* outside shortcut you asked about — AI/GPU tricks, scanning a hundred other games, debug symbols, and borrowing code from sister Square games — came back with a clean, well-evidenced **"no shortcut exists."** That's genuinely valuable: it stops us chasing dead ends. Along the way we banked real bonuses — named 62 functions for free, built a reusable reference library, and invented a brand-new tool that finds shared code across *any* game decompilation (worth open-sourcing). So we're closing this phase having gotten everything the automation could give, and pivoting to the one path that's left and is actually high-value: **hand-finishing the giant functions** (each one, once cracked, copies into all 134 levels at once) and learning any reusable tricks they teach us. The wall is the 1990s compiler itself — and from here, beating it is careful human-guided craft, exactly where we're pointing next.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase21.md` (R19, via `git mv` — left uncommitted). **Drew commits AND pushes** this PhaseEnd + the archived log + tonight's session commits (`commit:0292`..`commit:0294`) (R6/R8 — the milestone-close commit is the gate-2 sign-off, under human control). No Ghidra DB change this phase (R23 no-op — the `db.*.gbf` churn is restart-noise; do NOT stage it). Gen2 continues — do **NOT** start Phase 22 here. Start a **fresh session** (effort **Max**, **plan mode**) for **Phase 22 — Hand-grind the high-value residuals (giants ×134) + idiom distillation**. Keep this file forever.