BFM-decomp/phase-ends/PhaseEnd_Phase15.md

PhaseEnd — Phase 15: Overlay long-tail (fleet harvest + deterministic recovery)

Date: 2026-06-18 · Project Version: 1.14.0 · Phase Status: Complete (open-ended phase, milestone confirmed by Drew, gate 2) · Generation: Gen2 (7th phase)

Gen2 phase 7 of the arc (8→9→10→11→12→13→15→16). Phase 14 (public flip) is DEFERRED to Gen3+ (Drew, 2026-06-16). Phase 15 was open-ended, not a gate — "fleet matched-% rises monotonically, zero regressions." It ran across multiple sessions; the granular per-session trail (every harvest wave, the rate-limit consolidations, the per-tool diagnostics) is preserved on-demand at phase-ends/logs/Phase15.md (R19 — NOT auto-loaded; consult only when researching a mechanism). This file is the synthesis. Closed deliberately at a natural wall (the struct-heavy shared core), with a full Continuation Backlog (below) that becomes Phase 16.

Build Log

Files created/changed and complete — do not recreate:

The fleet (T1–T3):

• 134 location overlays onboarded → the project now builds 136 binaries (main + resident + 134 overlays). Each overlay: config/splat.ov_<SCxx>_<nnn>.yaml + config/check.ov_*.sha + config/symbols.ov_*.txt + src/ov_*/ov_*.c, all generated by tools/new_overlay.sh (Phase 13) and registered in the generated config/overlays.mk (-included before BINARIES :=; the Makefile body is never hand-edited to add one).
• Makefile — -MMD -MP header-dependency tracking (-include $(C_SRCS:%.c=build/%.d)) so edits to engine_core.h/engine_types.h/common.h correctly trigger recompiles (closed a stale-incremental gap that could false-pass a header-only change; R22-critical). build-all/check-all fleet targets (Phase 13).

The cross-binary dedup-propagation machinery (T2 — the economic engine of the phase):

• tools/dedup_propagate.py — match once in ov_SC01_077 → propagate fleet-wide. Extracts a matched body, authors it once as a #define DEFINE_func_<ADDR>() macro in src/shared/engine_core.h, instantiates it in every overlay that shares that function's h_exact, byte-gates each (fail-closed), registers the group in config/dedup.us.yaml. --auto-from <ov> propagates the whole shared set. Registry shorthand (vram + binaries:[...]) keeps dedup.us.yaml at ~4k lines for ~1400 groups. THIS SESSION: the find_site brace-placement fix (see Deviations) — the inline-def detector required the opening brace on the same line as the signature, silently dropping every next-line-brace def from propagation; now accepts brace on the same OR next line.
• src/shared/engine_core.h — the cumulative shared-body macro header (~15.5k lines, ~1395 DEFINE_func_* macros). Included once per overlay (via common.h path). The single source for every byte-identical engine fn.
• config/dedup.us.yaml — the cross-binary code-share registry (1395 validated groups, 0 failed).
• tools/dedup_integrate.py — byte-honesty validator (--check, fail-closed on sig-hash drift); the last line of make report.

Fleet reporting (T4):

• tools/progress.py --fleet → docs/progress.fleet.md (deterministic, source-derived, _DEDUP_CACHE-fast). tools/dup_report.py --cross → docs/duplicates.cross.md (the ranked cross-binary work queue; double-count fix).

The harvest + recovery toolchain (T5–T6 + this session):

• tools/harvest_verify.py (Phase 12) — the whole-binary byte-gate (substitute draft → make build → keep iff byte-identical, else revert; chunk+bisect). The incorruptible arbiter (G3/P9).
• tools/match_one.py (Phase 12) — per-function isolated compile + relocation-masked diff (leaf-exact; masks jal/HI16/LO16, so NOT a true gate for call/data — the whole-binary gate is).
• tools/gen_harvest_targets.py (T6a) — callee-signature-aware target manifest: resolves each callee/data symbol to its exact banked sig so drafts never guess a conflicting extern (the §14b wall-breaker).
• tools/canon_draft_decls.py (T6d) — rewrites a draft's callee externs to the banked-canonical set.
• tools/build_engine_types.py (T6 struct) — extracts inline-defined named struct/union types from ov_SC01_077.c into the shared src/shared/engine_types.h (so struct-using shared fns propagate). THIS SESSION: made ADDITIVE (merges with the existing header instead of overwriting — a re-run after a prior --strip was destructive and broke the build; caught + reverted + fixed).
• tools/sig_unify.py — NEW THIS SESSION. Unifies a draft's FULL signature set to the banked-canonical decls: both callee externs AND the draft's own definition signature (return + param types, body param-names kept). This is the recovery the existing canon_draft_decls was missing (it never touched the def's own signature). A probe found 30/30 sampled standalone-MATCH failures were type conflicts, 0 false-positives — i.e. those drafts are byte-correct C blocked only by signature unification. Gating the 191 → 32 verified byte-identical.

This session's matched-C (all byte-gated, propagated fleet-wide):

• src/ov_SC01_077/ov_SC01_077.c + src/shared/engine_core.h + config/dedup.us.yaml — the session added ~69 new shared functions (wave-1 27 + sig_unify 32 + permuter 1 + types-header 9) propagated across all 134 overlays. ov_SC01_077 is the canonical harvest source (965 stubs remain, all struct-heavy medium/large).

Docs:

• docs/gen2-roadmap.md (Phase-14 deferral; Phase-15 active), docs/matching-cookbook.md (the harvest + recovery techniques), docs/SETUP.md (tool inventory), README.md (status).
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase15.md (R19 archive). phase-ends/PhaseEnd_Phase15.md — this file.

Local artifacts (gitignored / regenerable): asm/<bin>/**, assets/<ov>/*.bin, build/, expected/, .run/sig.ov_*.jsonl (make sig-overlays), .run/drafts-* (harvest scratch), .run/permuter/<fn>/ (permuter scratch), .run/t6_*.json/.run/m2c_*.json (target manifests). The 134 overlay Ghidra programs are script-reproducible (ghidra_import_raw.sh + DefineFunctions.java) — DB-commit deferred as regenerable (no manual RE on them).

Tools/packages installed: None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv (rabbitizer, decomp-permuter, m2c, all already present).

Verification results (literal) — the milestone proof (this session's close):

• make clean && (extract all 136) && make check-all → 136 passed, 0 failed — every binary (main 143dbb89…, resident 8e17e02f…, all 134 overlays) byte-identical from a fully clean tree (R22). Reproduced repeatedly this session; negative control (corrupt one sha) → loud [FAIL].
• make report (deterministic): FLEET REAL 185,779 / LINKED 959 / byte-identical 187,426 / 344,010 = 54.48% / 0 NON_MATCHING in any default build (G4). dedup-check: 1395 validated, 0 failed.
• Fleet arc across Phase 15: 3.82% (T4 baseline) → 22.14% (T5a bulk) → 25.38% (T5b leaf) → 42.14% → 44.64% → 46.27% → 47.70% → 51.41% (prior sessions' T6 waves) → 54.48% (this session).
• This session's lever breakdown (the headline finding): the 50-agent Ultracode harvest added +0.36% for ~4.1M tokens; the deterministic recovery (types-header + permuter + sig_unify + the find_site bug-fix + types follow-up) added +2.67% for ~0 agent tokens — ~7× the agent wave's yield, free.
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged. Session committed as be0ceae55 → 4b0f4d100 → a10f7f8a1 → ab8b857cd (+ this PhaseEnd), each clean-rebuild-verified.

Milestone achieved (confirmed by Drew, gate 2): the fleet rebuilds 136/136 binaries byte-identical from a clean tree at 54.48% byte-identical-from-source (was 3.82% at phase start, 51.41% at this session's start); the cross-binary dedup-propagation engine (dedup_propagate --auto-from) and the deterministic signature-unification recovery (sig_unify) are both proven at fleet scale, zero regressions, 0 NON_MATCHING. The phase is closed at the natural wall — the remaining shared core is uniformly struct-heavy — with a complete Continuation Backlog handed to Phase 16.

Next: Phase 16 — Crack the struct-heavy shared core (research + the struct→m2c→permuter pipeline + the autonomous multi-day compute session). Open a fresh session in plan mode, Max effort (a Tier-1 research/ design phase). First task: validate the actor-struct + m2c-context + permuter pipeline on a 10-function sample before scaling. See the Continuation Backlog.

Deviations

Item	Plan	Actual	Reason
This session's opener	continue T6 agent harvest waves at 100+ agents	one 50-agent calibration wave, then pivot to deterministic recovery	the 50-agent wave yielded only +0.36% / ~9% gate yield on the hard residual (Drew flagged the ROI); the easy shared core was long banked — the tail is permuter-class
Recovery mechanism	(not planned)	sig_unify.py — unify the draft's OWN def signature, not just callee externs	a probe showed 30/30 standalone-MATCH failures were signature conflicts, byte-correct otherwise; canon only rewrote callee externs → the def-sig was the missing move (+32, +2.36% with the find_site fix)
find_site propagation	(assumed working)	silent brace-placement bug fixed — it required the brace on the same line, dropping every next-line-brace def (sig_unify/permuter output) from propagation	a green byte-gate masked under-propagation; the fix unlocked a 61-function backlog (this session's matches, not prior — verified)
build_engine_types.py	(re-runnable, assumed)	made additive — a 2nd --strip run was destructive (dropped already-migrated types, broke the build)	caught by the byte-check + reverted; fixed to merge with the existing header
The () no-prototype escape	(hypothesized as the arity-conflict fix)	rejected — gcc-2.7.2 refuses to match () against a prototype with a default-promotion param (s8/s16/u8/u16/float)	C's promotion rule; the 159 arity conflicts have no clean deterministic fix (documented dead-end)
m2c deterministic sweep	(planned as a free drafting path)	found non-viable on the remaining band — 20/20 sampled never-drafted fns are struct-heavy (actor->unkXXX); m2c output won't compile without the struct defined	the decompiler gives a scaffold, not byte-matching C — the struct is the real wall (→ Phase 16)
T7 giants	match via match_one	deferred to Phase 16 (untouched: 28 fns >150 ins, ~3.43 MB collapsible)	hand-iterating 770-ins struct-heavy fns without the struct is the worst token ROI; they are the prime struct→m2c→permuter compute target
Phase close	(open-ended grind)	closed at 54.48% with a Continuation Backlog	the clean deterministic vein is tapped; the rest is a distinct engine-struct sub-project better scoped fresh (Phase 16)

Commit Message

(Phase 15 landed across many session commits; this session: be0ceae55..ab8b857cd + this PhaseEnd.)

feat(phase-15): close — fleet 3.82% -> 54.48%, dedup-propagate + sig_unify proven (v1.14.0)

- 136 binaries (main + resident + 134 overlays) byte-identical from a clean tree (R22);
  fleet byte-identical-from-source 3.82% -> 54.48%, 1395 dedup groups, 0 NON_MATCHING (G4)
- the phase's engine: tools/dedup_propagate.py (match-once -> propagate fleet-wide via
  engine_core.h macros) + the callee-sig-aware harvest (gen_harvest_targets/harvest_verify/
  match_one/canon_draft_decls) + build_engine_types.py shared types header
- THIS SESSION (Max-effort deterministic recovery, ~0 agent tokens for +2.67%):
  - tools/sig_unify.py (NEW): unify a draft's OWN def-signature + callee externs to canonical;
    probe = 30/30 standalone-MATCH failures are conflicts, 0 false-positives; 191 -> 32 recovered
  - dedup_propagate find_site fix: accept brace on same OR next line (silent under-propagation
    bug) -> unlocked a 61-function backlog
  - build_engine_types additive merge; engine_types follow-up
  - one 50-agent Ultracode wave (+0.36%) vs deterministic recovery (+2.67%): the tail is
    permuter-class, not agent-draftable
- findings (-> Phase 16): the () no-proto escape fails on default-promotion params; m2c output
  is struct-heavy + won't compile (the actor struct is the wall); T7 giants deferred as the
  prime struct->m2c->permuter compute target
- worklog archived -> phase-ends/logs/Phase15.md (R19); bumps 1.13.0 -> 1.14.0

Rules Added This Phase

Rule

Reason

None (governance). Phase 15's lessons are techniques + a strategic finding, captured where they belong (the Phase-8/11/13 precedent: techniques live in docs/matching-cookbook.md, not as rules): the sig_unify signature-unification recovery + the probe-before-investing discipline (size a lever on a sample before building the full pass) + the find_site silent-under-propagation class (a green byte-gate proves correctness of what landed, NOT that everything that should propagate did — verify the OUTCOME metric, make report fleet %, not just the gate) + the ()/m2c dead-ends → cookbook. The strategic finding (on the hard tail, deterministic recovery beats agent waves ~7×/token; reach for sig_unify/permuter/propagation before mass-drafting) → docs/effort-map.md + cookbook. The existing R22 (clean-rebuild verify) + G3/P9 (byte-gate truth) + R26/R27 (Ultracode-on-breadth, effort transitions) already govern the new ground.

The new ground is mechanical/strategic, not a new norm of conduct. Elevate to a rule later only if a governance gap appears.

PhaseEnd Changelog

v1.13.0 → v1.14.0 — Phase 15 complete (Gen2 phase 7; open-ended, closed at a natural wall). The whole 134-overlay fleet is onboarded and the project rebuilds 136 binaries byte-identical from a clean tree, with fleet byte-identical-from-source driven 3.82% → 54.48% across the phase. The phase's economic engine is cross-binary dedup-propagation (tools/dedup_propagate.py + src/shared/engine_core.h): match a shared engine function once in ov_SC01_077, propagate it byte-identically across all 134 overlays — so one match is worth 134. The harvest was driven by a callee-signature-aware pipeline (gen_harvest_targets.py) past the extern-conflict wall, with an incorruptible whole-binary byte-gate (harvest_verify.py, G3/P9) as the sole arbiter. This (final) session's defining result: a single 50-agent Ultracode wave added +0.36% (~4.1M tokens), while a deterministic recovery — the new tools/sig_unify.py (unify each draft's own definition signature + callee externs to the banked-canonical set; a probe proved 30/30 standalone-MATCH failures were signature conflicts, not wrong C) plus a find_site brace-placement bug-fix that had been silently capping propagation — added +2.67% for ~0 agent tokens (~7× the wave, free). The phase closes at a clean wall: the remaining ~965 shared stubs (incl. 28 giants >150 ins, ~3.43 MB collapsible) are uniformly struct-heavy — m2c/Ghidra produce readable-but-uncompilable scaffolds (actor->unkXXX) and the () no-prototype escape fails on C's default-promotion rule. So the remaining decomp routes through one engine struct, handed forward as Phase 16's Continuation Backlog. No tools installed; no new governance rules (techniques → cookbook, the strategic finding → effort-map). Both byte-locked binaries and all 134 overlays rebuild identical; zero regressions; 0 NON_MATCHING linked.

Continuation Backlog → Phase 16 (the forward bridge; nothing here is lost on close)

Phase 16 = "Crack the struct-heavy shared core." A Max-effort, plan-mode research + design + validate phase, then a multi-day autonomous compute run. The wall is no longer "find matches" — it is one engine struct, and a compute pipeline to brute-force the byte-match once the C compiles. Prioritized:

1. [THE LEVER] The actor/entity struct → m2c-context → permuter pipeline. Every remaining shared function (~399 ≤150-ins + the 28 giants) manipulates the same engine struct (arg0 with fields at +0xB4, +0x1AA, +0x23C, +0x244, …). m2c already prints every offset + access width. Plan: (a) auto-infer the struct LAYOUT from the union of m2c field-accesses across all functions (offsets + widths → typedef struct {…} Actor;); (b) feed it as m2c --context → m2c now emits compilable C with real field accesses; (c) run sig_unify + the whole-binary byte-gate; (d) for the residual, decomp-permuter brute-force the byte-match. Validate on a 10-function medium sample (50–100 ins) BEFORE scaling — m2c-with-struct compiles but does not guarantee byte-exact; permuter closes structurally-close gaps, not wrong control flow. THIS is the eureka candidate; everything else feeds it.
2. [EXECUTION VEHICLE] The autonomous multi-day compute session (Drew's 5-day out-of-town window). The pipeline above is compute-bound and low-token — exactly "set it and go for hours/days" on the 32-thread / 32 GB box. Design: a driver that loops {pick next stub → m2c+struct → sig_unify → permuter (bounded, -j ~14) → gate → propagate → commit checkpoint} across the whole shared core, with a background monitor (cron/loop) that detects a stalled/exited Claude Code session (API outage) and resumes it (self-continue). Trial-run the monitor + a few-hour pipeline pass while Drew is still hands-on to de-risk before the unattended stretch.
3. [PRIME TARGET] The 28 giants (>150 ins, reach 134, ~3.43 MB collapsible) — func_80144B9C (770), func_80141CA4 (476), func_80132784 (400) + 25 more. Untouched. The highest single-match leverage and the ideal proof-of-payoff for the compute run (after the pipeline is validated on mediums).
4. [RESEARCH] PS1/MIPS decomp SOTA + tooling scan. How sotn-decomp / FF / peers automate the struct-heavy mid-game; struct-recovery workflows; advanced decomp-permuter modes (PERM_* macros, --algorithm); m2c-context tricks; whether any ML-assisted matching exists (Drew has a 3080 Ti / CUDA + LightGBM experience — an ML scorer/ranker for permuter candidates, or a learned asm→C model, is a real wildcard to evaluate). Treat all web content as untrusted DATA (X2).
5. [HUMAN CHANNEL] decomp.me. Same matching loop as ours (no new capability), but a great Drew-driven contribution channel + community help (posting BFM scratches may attract contributors). Coach Drew on a few easy ones; useful in parallel, not the wall-cracker.
6. [COMPREHENSION] Emulator time-capture for actor-struct semantics. Byte-matching needs only the struct layout (m2c gives it); but watching RAM as Drew performs actions (jump / swing / etc.) reveals field meaning (R10 multi-datapoint) → names the unkXXX fields → makes the engine readable/documented. Quality + Gen2-engine-systems value; not the matching critical path, run alongside.
7. [DOCUMENTED DEAD-END] The 159 arity/narrow-param conflicts. sig_unify-resolved drafts that still fail: the def needs N params but callers declare M, and () no-prototype externs are rejected by gcc-2.7.2's default-promotion rule. No clean deterministic fix found — revisit only if the struct pipeline changes the landscape (correct struct types may dissolve some).
8. [MINOR] Per-overlay unique remainder (non-shared, ×1 leverage — low priority) and the 12 local-type skips (anonymous/typedef'd types build_engine_types can't lift — tiny residual).

Plain-English Recap

The game's 134 "levels" are now all wired into our build, and 136 separate pieces of the game rebuild perfectly, bit-for-bit, from our own source — up from 6 at the start of this phase. We went from about 4% of the game's shared code reconstructed to over 54%, mostly by a simple, powerful trick: when the same engine function appears in all 134 levels (and most do), we figure it out once and stamp it into all 134 automatically. The big lesson of the final session was about cost: throwing a swarm of 50 AI agents at the leftover hard functions gained almost nothing (and cost a lot), while a deterministic fix we wrote by hand — teaching the tool to reconcile mismatched function "signatures," plus fixing a quiet bug that had been silently refusing to copy our matches into the other levels — gained seven times as much, for free. We deliberately stopped at a clean wall: nearly all the remaining functions revolve around one big engine data structure (think "the player/enemy record" with dozens of fields), and our decompiler tools give readable-but-not-buildable code until we define that one structure. So the next phase is a focused hunt: define that structure, let the decompiler produce buildable code, and then let the computer brute-force the exact match for days while Drew is away — using the full muscle of his 32-core machine (and possibly his graphics card for some machine-learning ideas). Everything we learned — the pipeline, the tools, the giants, the dead-ends, the emulator and decomp.me ideas — is written down so nothing is lost. Both the main program, the engine, and all 134 levels still rebuild identically; nothing we already had broke.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase15.md (R19). The Phase-15 matching work is committed (be0ceae55..ab8b857cd + this PhaseEnd commit, which lands the PhaseEnd + cookbook §14d/§14e + SETUP + gen2-roadmap); Drew pushes (R6). No Ghidra DB change this phase (pure build-tooling/matching — R23 no-op). Gen2 continues — do NOT start Phase 16 here. Start a fresh session (effort Max, plan mode) for Phase 16 — Crack the struct-heavy shared core, opening with the Continuation Backlog above (validate the struct→m2c→permuter pipeline on a medium sample first). Keep this file forever.