BFM-decomp/phase-ends/PhaseEnd_Phase19.md

PhaseEnd — Phase 19: Scale the toolkit (wave-harvest at scale + cheap front-loaded levers)

Date: 2026-06-20 · Project Version: 1.18.0 · Phase Status: Complete (milestone confirmed by Drew, gate 2 — Option A: close at a clean checkpoint) · Generation: Gen2 (11th phase)

Gen2 phase 11 of the arc (8→9→10→11→12→13→15→16→17→18→19; 14 deferred to Gen3+). Ran across 2 sessions; the granular per-task/per-batch trail (the T1 -O0 split diagnostics, the T2 reproduction logs, the two wave result sets, the gate-failure classifications, the propagation-skip breakdowns) is preserved on-demand at phase-ends/logs/Phase19.md (R19 — NOT auto-loaded; consult only when researching a mechanism). This file is the synthesis. Owner decisions (Drew): bounded flywheel (T1 + T2 + 2–3 wave batches); T1 re-scoped to "bank infra + 6, defer the rest" (R14); Ultracode for the wave batches; close after 2 batches at a clean checkpoint (Option A) with the propagation type-lift as the Phase-20 opener.

Build Log

Files created/changed and complete — do not recreate:

New reusable recovery tooling (the phase's durable output — feeds Phase 20 directly):

• tools/canon_resident_calls.py — (T2) link-miss recovery: rewrite each func_<ADDR> in a draft to the curated resident symbol name (0x8004CFEC→ratan2) when that addr has one in the stacked symbol files, so the linker resolves it. Pure draft-text (body bytes unchanged); runs FIRST in the recovery pipeline.
• tools/fix_arity_callers.py — (T3) the no-prototype recovery for the dominant gate-failure class: rewrite a banked shared caller's extern <ret> func_X(void); → extern <ret> func_X(); (byte-neutral; --apply/--revert; --drafts skips narrow-param defs). Automates the T2-proven hand fix; recovered 8/18 batch-2 failures.
• tools/gen_harvest_targets.py — FIXED INLINE_DEF_RE (was matching an indented if (func_X(...)) { call-expr as a "definition" → garbled callee hints agents pasted as externs → PARSE fails). Column-0 + type-only prefix → 0 garbled of 749; batch-2 match_one close-rate 88%→92%.

Config / source (the matches):

• src/ov_SC01_077/ov_SC01_077.c — T2's 3 matches + T3's banked matches (inline defs; the propagated ones spliced to DEFINE_func_*() macros) + the 4 byte-neutral no-proto caller edits' downstream. ~16 banked matches remain inline-LOCAL (typedef/local-type bodies — the propagation cap, Phase-20 lever).
• src/shared/engine_core.h — the no-proto caller-extern edits ((void)→() for func_80147514/80168F40/8014AB7C/ 8014AC10 + the batch-2 arity set) + +35 new DEFINE_func_* macros (the propagated matches).
• src/shared/engine_types.h — struct-lift (build_engine_types.py --strip): 48 named types (forward decls + defs) lifted from ov_SC01_077.c so struct-using shared bodies can propagate; byte-neutral.
• src/ov_*/*.c (134 overlays) — the ×134 DEFINE_func_*() instantiations from dedup_propagate.
• config/dedup.us.yaml — 1450 → 1485 validated cross-binary groups (0 failed).
• (T1, committed earlier this phase, prior session) the per-file -O0 split (src/ov_SC01_077_a.c / …_o0.c 3-object split + the Makefile -O0 target-specific CC1FLAGS) + 6 matched -O0 functions.

Docs / knowledge base (R16 flywheel, R21):

• docs/matching-cookbook.md — §17a-3 corrected (the Phase-18 "implicit-int / propagate-first" framing was WRONG, R14 — the real classes are resident-callee link-miss + shared-caller arity) + §18 (T1 -O0 split, prior session) + §19 (T3 wave-at-scale: the canon-first 2-stage gate, fix_arity_callers, the garbled-sig fix, and the propagation CAP).
• docs/SETUP.md — tool inventory rows for canon_resident_calls.py + fix_arity_callers.py.
• README.md — Phase-19 status line (136 binaries byte-identical, fleet 58.0%; the narrative body flagged as Phase-11/12-era pending a public-flip refresh).
• docs/{progress,difficulty,duplicates}.*.md — regenerated.
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase19.md (R19 archive). phase-ends/PhaseEnd_Phase19.md — this file.

Local artifacts (gitignored / regenerable): .run/harvest_wave_t3b{1,2}.js + .run/gen_wave.py (wave generator; recipe = cookbook §17/§19), .run/t3_gate2.sh (the canon-first 2-stage gate), .run/t3_{repro,diag, dump,postwave}.{py,sh} (diagnostics), .run/drafts-t3b* (wave drafts + recovery stages), .run/ghidra_c/* (300 cached Ghidra-C), .run/t3_b{1,2}_targets.json, build/, expected/, asm/, assets/. The 134 overlay Ghidra programs are script-reproducible (DB-commit deferred as regenerable).

Tools/packages installed: None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv.

Verification results (literal) — the milestone proof:

• make check-all → 136 passed, 0 failed of 136 (main 143dbb89…, resident 8e17e02f…, all 134 overlays; ov_SC01_077 d19c9580…), reproduced after every step (R22).
• make report: FLEET byte-identical 199,541 / 344,010 = 58.00% (phase start 56.64% / 194,839 → +1.36%); REAL 197,894; LINKED 959; dedup-check 1485 validated, 0 failed; 0 NON_MATCHING in any default build (G4).
• Batch 1 (50 targets): 44/50 match_one MATCH (88%); 6 DIFF = genuine gcc-quirk tail (correctly stubbed); whole-binary gate 30 banked; 25 propagated ×134; +0.98%.
• Batch 2 (38 targets, clean hints): 35/38 match_one MATCH (92%); 3 DIFF (genuine); gate + fix_arity_callers 25 banked (ALL 18 stage failures were the shared-caller arity class); 10 propagated ×134; +0.31%.
• Struct-lift top-up: +4 propagated; +0.07%.
• The decomp.dev-comparable (instruction/byte-weighted) estimate ≈ 30% (overlays 29.7%, resident 55.8%, main 46.4% hand-matched) — the 57.62%→~30% gap is the deferred giants + per-overlay unique code; recorded for honesty (P9) and the public-flip expectation.
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged. No Ghidra DB change this phase (matching used cached Ghidra-C + asm; the db.*.gbf churn is SessionStart-restart noise — R23 no-op, NOT staged).

Milestone achieved (confirmed by Drew, gate 2 — Option A): the bounded-flywheel milestone is met — the -O0 lever is banked (T1: infra + 6), recovery tooling is proven (T2: canon_resident_calls + the no-proto fix; T3: fix_arity_callers + the canon-first 2-stage gate + the struct-lift), the toolkit scaled across 2 wave batches (match_one 88%→92%), the fleet rose 56.64% → 58.00% with 136/136 byte-identical and 0 NON_MATCHING, and the phase closed at a clean checkpoint with a sharp Phase-20 backlog. The +3–5% target was not reached (+1.36%) — gated by the cache-limited target pool (~88 eligible, all run) and the phase's key finding: propagation, not matching, now caps the fleet-% yield (typedef/local-type bodies can't auto-propagate).

Next: Phase 20 — close the propagation cap, then scale. Open a fresh session (plan mode). First lever: extend the type-lift to typedefs/local types so the 16 banked-but-local matches propagate ×134 (+0.6% for ~0 agent tokens) and every future batch realizes its full yield; then more wave batches (Ghidra-C regen for fresh reach-134, R23/R29) and the deferred -O0 ×134 rollout. See the Phase-20 Backlog.

Deviations

Item	Plan	Actual	Reason
T1 -O0 rollout	full ×134, "an afternoon, free"	bank infra + 6, defer the 10 %lo-quirk fns + the ×134 rollout (R14)	the residual -O0 fns hit a %lo-folding codegen quirk (gcc-source research); ×134 needs per-overlay -O0 splits (engine_core.h is -O2) — two findings corrected the Phase-18 "free ×134" premise (prior session)
T2 framing	"implicit-int / propagate-first helper (3 fns)"	R14-corrected: NOT implicit-int — 3 real classes (resident-callee link-miss + shared-caller arity); canon_resident_calls + the no-proto fix; func_8012A418 isn't even a stub in ov_SC01_077	reproduced each through the gate — the Phase-18 backlog's premise was wrong (verify the framing against the bytes)
Recovery pipeline	cast_call_sites.py (call-site fn-ptr casts)	not built — the bytes never demanded it; instead the canon-first 2-stage gate + fix_arity_callers	the §17a "call-site cast = #1 miss" did not recur; the real recurring failures were sig_unify-regressions + the arity class
sig_unify in the pipeline	run unconditionally before the gate	gate canon-only FIRST, sig_unify FALLBACK	sig_unify REGRESSES already-canonical agent drafts (re-canonicalization perturbs codegen / mangles a line); canon-first recovered ~+5/batch
Fleet target	+3–5% (~56.6%→~60%)	+1.36% (→58.00%), closed at the checkpoint	cache-limited pool (~88 eligible) + the propagation cap (struct/typedef bodies stay local); Drew chose Option A (close) over batch-3/typedef-lift-now
T3 batch count	2–3 batches of 50	2 batches (50 + 38)	the cached reach-134 ≤90-ins pool was exhausted at ~88; a 3rd batch needs a Ghidra-C regen (R23/R29)
T4 gcc-research	conditional, on a residual-class plateau	NOT triggered	the waves never plateaued on a single gcc-quirk class (88–92% match_one); residuals were the loose-typing wall (Phase-16, genuine) + the propagation gap (tooling) — neither a gcc-research target
Effort	Max planning, Ultracode waves	Max/xHigh setup + recovery, Ultracode waves (Drew toggled, R27), xHigh close (Drew's saved default — PhaseEnd normally Max)	breadth waves under Ultracode; deterministic recovery/synthesis at xHigh

Commit Message

(Phase 19 landed as commits commit:0158..commit:0164 + this PhaseEnd.)

feat(phase-19): scale the toolkit — 2 waves (88%/92%), fleet 56.64%->58.00%, propagation is the cap (v1.18.0)

- T1 (prior session, commit:0158/commit:0159): per-file -O0 split infra + 6 -O0 matches; deferred the
  10 %lo-folding-quirk fns + the ×134 rollout (R14 corrected "free ×134")
- T2 (commit:0161): recovery tooling. R14: the 3 Phase-18 "implicit-int" near-misses are NOT
  implicit-int -> resident-callee link-miss (NEW canon_resident_calls.py) + shared-caller arity
  (no-proto caller-extern fix). 3 matches; cookbook §17a-3 corrected
- T3 (commit:0162/commit:0163/commit:0164): 2 Ultracode waves over reach-134 tractable residuals. batch-1
  44/50 match_one (30 banked, 25 propagated ×134); batch-2 35/38 (25 banked, 10 propagated).
  NEW fix_arity_callers.py (no-proto, ALL 18 batch-2 failures were the arity class); canon-first
  2-stage gate (sig_unify REGRESSES canonical drafts); gen_harvest_targets garbled-sig fix
  (0/749, 88%->92%); build_engine_types --strip lift (+4 propagated)
- FINDING: propagation (not matching) caps the fleet-% yield — typedef/local-type bodies can't
  auto-propagate (-> Phase-20 type-lift lever). decomp.dev-comparable byte-weighted est ~30%
- check-all 136/136 BYTE-IDENTICAL (R22); dedup 1450->1485/0 failed; 0 NON_MATCHING (G4);
  fleet 56.64% -> 58.00% (+1.36%). worklog -> phase-ends/logs/Phase19.md (R19)
- bumps 1.17.0 -> 1.18.0

Rules Added This Phase

Rule

Reason

None (governance). Phase 19's lessons are techniques + findings, recorded where the Phase-8/11/13/15/16/17/18 precedent puts them (cookbook, not rules): the two recovery tools (canon_resident_calls, fix_arity_callers) + the canon-first 2-stage gate + the garbled-sig fix + the propagation CAP → cookbook §17a/§19 + SETUP inventory. The R14 self-correction (the Phase-18 "implicit-int/propagate-first" framing was wrong — reproduce the failure against the bytes before trusting a prior framing) is a direct application of the existing R14/G3/P9. The bounded-flywheel + probe-before-investing discipline is already cookbook (Phase 15) + effort-map. No new norm of conduct emerged.

A scale-up + recovery-tooling phase produces knowledge, not governance; the existing rules (R14 verify-vs-bytes, R16 flywheel, R22 clean-rebuild, R26/R27 effort, G3/P9 byte-gate) covered the ground and the phase reinforced them.

PhaseEnd Changelog

v1.17.0 → v1.18.0 — Phase 19 complete (Gen2 phase 11). The proven §17/§18 toolkit was scaled over the tractable reach-134 tail as a bounded flywheel: two Ultracode parallel-draft/byte-gate waves matched at 88% then 92% match_one, and after a hardened deterministic recovery pipeline the fleet rose 56.64% → 58.00% (136/136 byte-identical throughout, 0 NON_MATCHING). The phase's reusable output is three recovery mechanisms — tools/canon_resident_calls.py (rewrite func_<ADDR> → the curated resident name for the link-miss class), tools/fix_arity_callers.py (the no-prototype caller-extern fix for the shared-caller arity class, which turned out to be ~100% of the residual gate failures once the other noise was removed), and the canon-first 2-stage gate (the finding that sig_unify regresses already-canonical agent drafts) — plus a root-cause fix to gen_harvest_targets' garbled callee hints (88%→92%) and the engine_types struct-lift. Two R14 self-corrections drove the work: the Phase-18 "implicit-int / propagate-first" framing for the 3 carried near-misses was wrong (the real classes are link-miss + arity, found by reproducing each through the gate), and the giants/per-overlay-unique deferral means the decomp.dev-comparable byte-weighted progress is ~30%, not 57.62% (recorded for honesty). The decisive finding: propagation — not matching — now caps the fleet-% yield. match_one is 88–92% and the gate banks 60–71%, but a matched function whose body uses a typedef'd / anonymous / sibling local type can't be lifted into the shared header, so ~16 banked matches stayed ov_SC01_077-local (no ×134). The +3–5% target was therefore not reached (+1.36%) — also gated by the cache-limited pool — and Drew chose to close at a clean checkpoint (Option A) with the propagation type-lift as the Phase-20 opener. No new governance rules (techniques → cookbook §17a/§19). All 136 binaries byte-identical; 0 NON_MATCHING linked.

Phase-20 Backlog — "Close the propagation cap, then scale" (the forward bridge; nothing here is lost on close)

Phase 20 opens in plan mode. The harvest's bottleneck has moved from matching (now 88–92%) to sharing. Prioritised by ROI:

1. [HIGHEST ROI — DO FIRST] Extend the type-lift to typedefs/local types (close the propagation cap). The named-struct-only build_engine_types.py leaves ~16 banked matches ov_SC01_077-LOCAL because their bodies use typedef'd / anonymous / sibling types dedup_propagate's compiles_standalone can't resolve. Extend the lifter to also lift typedef definitions (and detect collisions — the named-struct case had 0) into engine_types.h, then re-run dedup_propagate --auto-from. Recovers 16 already-matched fns ×134 for ~0 agent tokens (+0.6%) and raises every future batch's realized yield. This is the lever the whole phase pointed at.
2. [COMPOUNDING — automate the recovery further] fix_arity_callers generalization. It currently fixes (void)→(). The harder residual (batch-2's 10) is a caller sig ≠ def sig mismatch (e.g. caller declares void f(void*,s32), def needs s32 f(s32,u32); or banked siblings use s32 f(void) + call-site casts vs def void f(void*)). A generalized "reconcile the caller decl to <def-ret> func_X() (no-proto, def's return), byte-gated" pass would recover some — but some are the genuine Phase-16 loose-typing wall (no single C type). Worth a bounded probe (size the recoverable fraction first).
3. [THE CORE — more waves] Scale the toolkit over fresh reach-134 targets. Regen Ghidra-C (DecompileFunctions.java, headless — MCP stopped, R23/R29 /mcp reconnect) for the uncached tractable reach-134 stubs (the cache is 300; ~88 ≤90-ins eligible were exhausted in Phase 19), then run the §19 wave (canon-first 2-stage gate + fix_arity_callers folded in). With the type-lift (item 1) done, each match now actually propagates → batches realize ~full yield. Batch size 50; bank a prompt/tooling improvement each batch.
4. [THE BYTE-WEIGHTED LEVER] The giants (28 reach-134 fns >150 ins). Highest bytes (so the only lever that moves the byte-weighted ~30% number materially), same headline %, hardest. The §17 toolkit applies, just more labor per fn. Do after the cheap levers; ideal for a focused deep session.
5. [DEFERRED FROM PHASE 19] The -O0 ×134 rollout + the 10 %lo-quirk -O0 fns. Per-overlay -O0 splits (uniform offsets, scriptable) to bank the 6 matched -O0 fns ×134 (~+0.6%); and gcc-source research (R17) on the %lo-folding quirk blocking the other 10 -O0 fns (cookbook §18).
6. [CARRIED, lower priority] per-overlay unique remainder (×1 leverage); comprehension/emulator actor-field naming (Gen2 quality, byte-neutral); the public flip (Phase 14, Gen3+) incl. the README narrative refresh + the decomp.dev-comparable byte-weighted metric in progress.py.

Plain-English Recap

This phase was about scaling up the function-matching machine we built last phase. We pointed an AI agent swarm at batches of the game's shared engine functions, and they reconstructed them impressively well — 44 of 50, then 35 of 38 — each checked by an automatic bit-for-bit referee so nothing wrong can slip in. That pushed the game from ~56.6% to 58% rebuilt exactly, with all 136 pieces still byte-perfect. Along the way I built two small "fixer" tools that automatically clean up the most common reasons a correct function fails to slot in (a mismatched name, and a function-declaration quirk), and I fixed two bugs in our own tooling at the source. The big lesson: we're now great at reconstructing functions, but a separate step — copying one matched function into all 134 levels at once — chokes when the function uses certain local data-types, and that's what's actually holding the headline number back now (not the matching). That's a well-understood, cheap fix queued as the very first thing next phase. I also gave you an honest reality-check on the percentage: measured the way the public decompilation trackers measure it (by code size, where our deferred "giant" functions count heavily), we're around 30% — roughly level with the Xenogears project and a bit behind Symphony of the Night — not "ahead of both" as our internal 58% might suggest. We deliberately stopped at a clean checkpoint rather than grind for a higher number, with everything we learned written down. Nothing we already had broke.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase19.md (R19). The Phase-19 matching work is committed (commit:0158..commit:0164); Drew commits AND pushes this PhaseEnd + the archived log (R6/R8). No Ghidra DB change this phase (R23 no-op — the db.*.gbf churn is restart-noise, do NOT stage it). Gen2 continues — do NOT start Phase 20 here. Start a fresh session (plan mode) for Phase 20 — close the propagation cap, then scale (the backlog above; the typedef/local-type lift first). Keep this file forever.