BFM-decomp/phase-ends/PhaseEnd_Phase15.md

# PhaseEnd — Phase 15: Overlay long-tail (fleet harvest + deterministic recovery)
**Date:** 2026-06-18 · **Project Version:** 1.14.0 · **Phase Status:** Complete (open-ended phase, milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (7th phase)

> Gen2 phase 7 of the arc (8→9→10→11→12→13→**15**→16). Phase 14 (public flip) is DEFERRED to Gen3+ (Drew,
> 2026-06-16). Phase 15 was **open-ended, not a gate** — "fleet matched-% rises monotonically, zero regressions."
> It ran across **multiple sessions**; the granular per-session trail (every harvest wave, the rate-limit
> consolidations, the per-tool diagnostics) is preserved on-demand at **`phase-ends/logs/Phase15.md`** (R19 — NOT
> auto-loaded; consult only when researching a mechanism). This file is the synthesis. **Closed deliberately at a
> natural wall** (the struct-heavy shared core), with a full **Continuation Backlog** (below) that becomes Phase 16.

## Build Log

**Files created/changed and complete — do not recreate:**

*The fleet (T1–T3):*
- **134 location overlays onboarded** → the project now builds **136 binaries** (main + resident + 134 overlays).
  Each overlay: `config/splat.ov_<SCxx>_<nnn>.yaml` + `config/check.ov_*.sha` + `config/symbols.ov_*.txt` +
  `src/ov_*/ov_*.c`, all generated by `tools/new_overlay.sh` (Phase 13) and registered in the generated
  **`config/overlays.mk`** (`-include`d before `BINARIES :=`; the Makefile body is never hand-edited to add one).
- `Makefile` — `-MMD -MP` header-dependency tracking (`-include $(C_SRCS:%.c=build/%.d)`) so edits to
  `engine_core.h`/`engine_types.h`/`common.h` correctly trigger recompiles (closed a stale-incremental gap that
  could false-pass a header-only change; R22-critical). `build-all`/`check-all` fleet targets (Phase 13).

*The cross-binary dedup-propagation machinery (T2 — the economic engine of the phase):*
- `tools/dedup_propagate.py` — **match once in `ov_SC01_077` → propagate fleet-wide.** Extracts a matched body,
  authors it once as a `#define DEFINE_func_<ADDR>()` macro in `src/shared/engine_core.h`, instantiates it in
  every overlay that shares that function's `h_exact`, byte-gates each (fail-closed), registers the group in
  `config/dedup.us.yaml`. **`--auto-from <ov>`** propagates the whole shared set. Registry shorthand
  (`vram + binaries:[...]`) keeps `dedup.us.yaml` at ~4k lines for ~1400 groups. **THIS SESSION: the `find_site`
  brace-placement fix** (see Deviations) — the inline-def detector required the opening brace on the *same line*
  as the signature, silently dropping every next-line-brace def from propagation; now accepts brace on the same
  OR next line.
- `src/shared/engine_core.h` — the cumulative shared-body macro header (~15.5k lines, ~1395 `DEFINE_func_*`
  macros). Included once per overlay (via `common.h` path). The single source for every byte-identical engine fn.
- `config/dedup.us.yaml` — the cross-binary code-share registry (1395 validated groups, 0 failed).
- `tools/dedup_integrate.py` — byte-honesty validator (`--check`, fail-closed on sig-hash drift); the last line of
  `make report`.

*Fleet reporting (T4):*
- `tools/progress.py --fleet` → **`docs/progress.fleet.md`** (deterministic, source-derived, `_DEDUP_CACHE`-fast).
  `tools/dup_report.py --cross` → `docs/duplicates.cross.md` (the ranked cross-binary work queue; double-count fix).

*The harvest + recovery toolchain (T5–T6 + this session):*
- `tools/harvest_verify.py` (Phase 12) — the whole-binary byte-gate (substitute draft → `make build` → keep iff
  byte-identical, else revert; chunk+bisect). The incorruptible arbiter (G3/P9).
- `tools/match_one.py` (Phase 12) — per-function isolated compile + relocation-masked diff (leaf-exact; masks
  jal/HI16/LO16, so NOT a true gate for call/data — the whole-binary gate is).
- `tools/gen_harvest_targets.py` (T6a) — **callee-signature-aware** target manifest: resolves each callee/data
  symbol to its exact banked sig so drafts never guess a conflicting extern (the §14b wall-breaker).
- `tools/canon_draft_decls.py` (T6d) — rewrites a draft's *callee* externs to the banked-canonical set.
- `tools/build_engine_types.py` (T6 struct) — extracts inline-defined named struct/union types from `ov_SC01_077.c`
  into the shared **`src/shared/engine_types.h`** (so struct-using shared fns propagate). **THIS SESSION: made
  ADDITIVE** (merges with the existing header instead of overwriting — a re-run after a prior `--strip` was
  destructive and broke the build; caught + reverted + fixed).
- **`tools/sig_unify.py` — NEW THIS SESSION.** Unifies a draft's FULL signature set to the banked-canonical decls:
  both callee externs AND **the draft's own definition signature** (return + param types, body param-names kept).
  This is the recovery the existing `canon_draft_decls` was missing (it never touched the def's own signature).
  A probe found **30/30 sampled standalone-MATCH failures were type conflicts, 0 false-positives** — i.e. those
  drafts are byte-correct C blocked only by signature unification. Gating the 191 → **32 verified** byte-identical.

*This session's matched-C (all byte-gated, propagated fleet-wide):*
- `src/ov_SC01_077/ov_SC01_077.c` + `src/shared/engine_core.h` + `config/dedup.us.yaml` — the session added
  **~69 new shared functions** (wave-1 27 + sig_unify 32 + permuter 1 + types-header 9) propagated across all 134
  overlays. `ov_SC01_077` is the canonical harvest source (965 stubs remain, all struct-heavy medium/large).

*Docs:*
- `docs/gen2-roadmap.md` (Phase-14 deferral; Phase-15 active), `docs/matching-cookbook.md` (the harvest +
  recovery techniques), `docs/SETUP.md` (tool inventory), `README.md` (status).
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase15.md`** (R19 archive). `phase-ends/PhaseEnd_Phase15.md` — this file.

**Local artifacts (gitignored / regenerable):** `asm/<bin>/**`, `assets/<ov>/*.bin`, `build/`, `expected/`,
`.run/sig.ov_*.jsonl` (`make sig-overlays`), `.run/drafts-*` (harvest scratch), `.run/permuter/<fn>/` (permuter
scratch), `.run/t6_*.json`/`.run/m2c_*.json` (target manifests). The 134 overlay Ghidra programs are
script-reproducible (`ghidra_import_raw.sh` + `DefineFunctions.java`) — DB-commit deferred as regenerable (no
manual RE on them).

**Tools/packages installed:** None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv
(rabbitizer, decomp-permuter, m2c, all already present).

**Verification results (literal) — the milestone proof (this session's close):**
- **`make clean && (extract all 136) && make check-all` → 136 passed, 0 failed** — every binary (main `143dbb89…`,
  resident `8e17e02f…`, all 134 overlays) byte-identical **from a fully clean tree** (R22). Reproduced repeatedly
  this session; negative control (corrupt one sha) → loud `[FAIL]`.
- **`make report` (deterministic):** FLEET **REAL 185,779** / LINKED 959 / **byte-identical 187,426 / 344,010 =
  54.48%** / 0 NON_MATCHING in any default build (G4). `dedup-check: 1395 validated, 0 failed`.
- **Fleet arc across Phase 15:** 3.82% (T4 baseline) → 22.14% (T5a bulk) → 25.38% (T5b leaf) → 42.14% → 44.64% →
  46.27% → 47.70% → **51.41%** (prior sessions' T6 waves) → **54.48%** (this session).
- **This session's lever breakdown** (the headline finding): the 50-agent Ultracode harvest added **+0.36%** for
  ~4.1M tokens; the **deterministic recovery** (types-header + permuter + `sig_unify` + the `find_site` bug-fix +
  types follow-up) added **+2.67% for ~0 agent tokens** — ~7× the agent wave's yield, free.
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk
  staged**. Session committed as `commit:0117 → commit:0118 → commit:0119 → commit:0120` (+ this PhaseEnd), each clean-rebuild-verified.

**Milestone achieved (confirmed by Drew, gate 2):** the fleet rebuilds **136/136 binaries byte-identical from a
clean tree** at **54.48% byte-identical-from-source** (was 3.82% at phase start, 51.41% at this session's start);
the **cross-binary dedup-propagation engine** (`dedup_propagate --auto-from`) and the **deterministic
signature-unification recovery** (`sig_unify`) are both proven at fleet scale, zero regressions, 0 NON_MATCHING.
The phase is closed at the natural wall — the remaining shared core is uniformly struct-heavy — with a complete
Continuation Backlog handed to Phase 16.

**Next:** **Phase 16 — Crack the struct-heavy shared core** (research + the struct→m2c→permuter pipeline +
the autonomous multi-day compute session). Open a fresh session in **plan mode, Max effort** (a Tier-1 research/
design phase). First task: validate the actor-struct + m2c-context + permuter pipeline on a 10-function sample
*before* scaling. See the Continuation Backlog.

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| This session's opener | continue T6 agent harvest waves at 100+ agents | **one 50-agent calibration wave, then pivot to deterministic recovery** | the 50-agent wave yielded only +0.36% / ~9% gate yield on the hard residual (Drew flagged the ROI); the easy shared core was long banked — the tail is permuter-class |
| Recovery mechanism | (not planned) | **`sig_unify.py` — unify the draft's OWN def signature**, not just callee externs | a probe showed 30/30 standalone-MATCH failures were *signature conflicts*, byte-correct otherwise; `canon` only rewrote callee externs → the def-sig was the missing move (+32, +2.36% with the find_site fix) |
| `find_site` propagation | (assumed working) | **silent brace-placement bug fixed** — it required the brace on the *same line*, dropping every next-line-brace def (sig_unify/permuter output) from propagation | a green byte-gate masked under-propagation; the fix unlocked a 61-function backlog (this session's matches, not prior — verified) |
| `build_engine_types.py` | (re-runnable, assumed) | **made additive** — a 2nd `--strip` run was destructive (dropped already-migrated types, broke the build) | caught by the byte-check + reverted; fixed to merge with the existing header |
| The `()` no-prototype escape | (hypothesized as the arity-conflict fix) | **rejected** — gcc-2.7.2 refuses to match `()` against a prototype with a default-promotion param (`s8/s16/u8/u16/float`) | C's promotion rule; the 159 arity conflicts have no clean deterministic fix (documented dead-end) |
| m2c deterministic sweep | (planned as a free drafting path) | **found non-viable on the remaining band** — 20/20 sampled never-drafted fns are struct-heavy (`actor->unkXXX`); m2c output won't compile without the struct defined | the decompiler gives a scaffold, not byte-matching C — the struct is the real wall (→ Phase 16) |
| T7 giants | match via match_one | **deferred to Phase 16** (untouched: 28 fns >150 ins, ~3.43 MB collapsible) | hand-iterating 770-ins struct-heavy fns without the struct is the worst token ROI; they are the prime struct→m2c→permuter compute target |
| Phase close | (open-ended grind) | **closed at 54.48% with a Continuation Backlog** | the clean deterministic vein is tapped; the rest is a distinct engine-struct sub-project better scoped fresh (Phase 16) |

## Commit Message
```
(Phase 15 landed across many session commits; this session: commit:0117..commit:0120 + this PhaseEnd.)

feat(phase-15): close — fleet 3.82% -> 54.48%, dedup-propagate + sig_unify proven (v1.14.0)

- 136 binaries (main + resident + 134 overlays) byte-identical from a clean tree (R22);
  fleet byte-identical-from-source 3.82% -> 54.48%, 1395 dedup groups, 0 NON_MATCHING (G4)
- the phase's engine: tools/dedup_propagate.py (match-once -> propagate fleet-wide via
  engine_core.h macros) + the callee-sig-aware harvest (gen_harvest_targets/harvest_verify/
  match_one/canon_draft_decls) + build_engine_types.py shared types header
- THIS SESSION (Max-effort deterministic recovery, ~0 agent tokens for +2.67%):
  - tools/sig_unify.py (NEW): unify a draft's OWN def-signature + callee externs to canonical;
    probe = 30/30 standalone-MATCH failures are conflicts, 0 false-positives; 191 -> 32 recovered
  - dedup_propagate find_site fix: accept brace on same OR next line (silent under-propagation
    bug) -> unlocked a 61-function backlog
  - build_engine_types additive merge; engine_types follow-up
  - one 50-agent Ultracode wave (+0.36%) vs deterministic recovery (+2.67%): the tail is
    permuter-class, not agent-draftable
- findings (-> Phase 16): the () no-proto escape fails on default-promotion params; m2c output
  is struct-heavy + won't compile (the actor struct is the wall); T7 giants deferred as the
  prime struct->m2c->permuter compute target
- worklog archived -> phase-ends/logs/Phase15.md (R19); bumps 1.13.0 -> 1.14.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **None (governance).** Phase 15's lessons are **techniques + a strategic finding**, captured where they belong (the Phase-8/11/13 precedent: techniques live in `docs/matching-cookbook.md`, not as rules): the **`sig_unify` signature-unification recovery** + the **probe-before-investing discipline** (size a lever on a sample before building the full pass) + the **`find_site` silent-under-propagation class** (a green byte-gate proves *correctness of what landed*, NOT that everything that should propagate did — verify the OUTCOME metric, `make report` fleet %, not just the gate) + the **`()`/m2c dead-ends** → cookbook. The **strategic finding** (on the hard tail, deterministic recovery beats agent waves ~7×/token; reach for `sig_unify`/permuter/propagation before mass-drafting) → `docs/effort-map.md` + cookbook. The existing **R22** (clean-rebuild verify) + **G3/P9** (byte-gate truth) + **R26/R27** (Ultracode-on-breadth, effort transitions) already govern the new ground. | The new ground is mechanical/strategic, not a new norm of conduct. Elevate to a rule later only if a *governance* gap appears. |

## PhaseEnd Changelog
**v1.13.0 → v1.14.0 — Phase 15 complete (Gen2 phase 7; open-ended, closed at a natural wall).** The whole
**134-overlay fleet** is onboarded and the project rebuilds **136 binaries byte-identical from a clean tree**,
with fleet byte-identical-from-source driven **3.82% → 54.48%** across the phase. The phase's economic engine is
**cross-binary dedup-propagation** (`tools/dedup_propagate.py` + `src/shared/engine_core.h`): match a shared
engine function once in `ov_SC01_077`, propagate it byte-identically across all 134 overlays — so one match is
worth 134. The harvest was driven by a **callee-signature-aware** pipeline (`gen_harvest_targets.py`) past the
extern-conflict wall, with an incorruptible whole-binary **byte-gate** (`harvest_verify.py`, G3/P9) as the sole
arbiter. **This (final) session's defining result:** a single 50-agent Ultracode wave added **+0.36%** (~4.1M
tokens), while a **deterministic recovery** — the new **`tools/sig_unify.py`** (unify each draft's own
definition signature + callee externs to the banked-canonical set; a probe proved 30/30 standalone-MATCH
failures were *signature conflicts*, not wrong C) plus a **`find_site` brace-placement bug-fix** that had been
silently capping propagation — added **+2.67% for ~0 agent tokens** (~7× the wave, free). The phase closes at a
**clean wall**: the remaining ~965 shared stubs (incl. 28 giants >150 ins, ~3.43 MB collapsible) are uniformly
**struct-heavy** — m2c/Ghidra produce readable-but-uncompilable scaffolds (`actor->unkXXX`) and the `()`
no-prototype escape fails on C's default-promotion rule. So the remaining decomp routes through **one engine
struct**, handed forward as Phase 16's Continuation Backlog. No tools installed; no new governance rules
(techniques → cookbook, the strategic finding → effort-map). Both byte-locked binaries and all 134 overlays
rebuild identical; zero regressions; 0 NON_MATCHING linked.

## Continuation Backlog → Phase 16 (the forward bridge; nothing here is lost on close)

> Phase 16 = **"Crack the struct-heavy shared core."** A Max-effort, plan-mode **research + design + validate**
> phase, then a **multi-day autonomous compute run**. The wall is no longer "find matches" — it is **one engine
> struct**, and a **compute pipeline** to brute-force the byte-match once the C compiles. Prioritized:

1. **[THE LEVER] The actor/entity struct → m2c-context → permuter pipeline.** Every remaining shared function
   (~399 ≤150-ins + the 28 giants) manipulates the *same* engine struct (`arg0` with fields at +0xB4, +0x1AA,
   +0x23C, +0x244, …). m2c already prints every offset + access width. **Plan:** (a) auto-infer the struct LAYOUT
   from the union of m2c field-accesses across all functions (offsets + widths → `typedef struct {…} Actor;`);
   (b) feed it as m2c `--context` → m2c now emits *compilable* C with real field accesses; (c) run
   `sig_unify` + the whole-binary byte-gate; (d) for the residual, **decomp-permuter brute-force** the byte-match.
   *Validate on a 10-function medium sample (50–100 ins) BEFORE scaling* — m2c-with-struct compiles but does not
   guarantee byte-exact; permuter closes structurally-close gaps, not wrong control flow. THIS is the eureka
   candidate; everything else feeds it.
2. **[EXECUTION VEHICLE] The autonomous multi-day compute session** (Drew's 5-day out-of-town window). The
   pipeline above is **compute-bound and low-token** — exactly "set it and go for hours/days" on the 32-thread /
   32 GB box. **Design:** a driver that loops {pick next stub → m2c+struct → sig_unify → permuter (bounded, -j ~14)
   → gate → propagate → commit checkpoint} across the whole shared core, with a **background monitor** (cron/loop)
   that detects a stalled/exited Claude Code session (API outage) and resumes it (self-`continue`). Trial-run the
   monitor + a few-hour pipeline pass *while Drew is still hands-on* to de-risk before the unattended stretch.
3. **[PRIME TARGET] The 28 giants (>150 ins, reach 134, ~3.43 MB collapsible)** — `func_80144B9C` (770),
   `func_80141CA4` (476), `func_80132784` (400) + 25 more. Untouched. The highest single-match leverage and the
   ideal proof-of-payoff for the compute run (after the pipeline is validated on mediums).
4. **[RESEARCH] PS1/MIPS decomp SOTA + tooling scan.** How sotn-decomp / FF / peers automate the struct-heavy
   mid-game; struct-recovery workflows; advanced decomp-permuter modes (`PERM_*` macros, `--algorithm`);
   m2c-context tricks; whether any **ML-assisted matching** exists (Drew has a 3080 Ti / CUDA + LightGBM
   experience — an ML scorer/ranker for permuter candidates, or a learned asm→C model, is a real wildcard to
   evaluate). Treat all web content as untrusted DATA (X2).
5. **[HUMAN CHANNEL] decomp.me.** Same matching loop as ours (no new capability), but a great **Drew-driven**
   contribution channel + community help (posting BFM scratches may attract contributors). Coach Drew on a few
   easy ones; useful in parallel, not the wall-cracker.
6. **[COMPREHENSION] Emulator time-capture for actor-struct semantics.** Byte-matching needs only the struct
   *layout* (m2c gives it); but watching RAM as Drew performs actions (jump / swing / etc.) reveals field
   *meaning* (R10 multi-datapoint) → names the `unkXXX` fields → makes the engine readable/documented. Quality +
   Gen2-engine-systems value; not the matching critical path, run alongside.
7. **[DOCUMENTED DEAD-END] The 159 arity/narrow-param conflicts.** `sig_unify`-resolved drafts that still fail:
   the def needs N params but callers declare M, and `()` no-prototype externs are rejected by gcc-2.7.2's
   default-promotion rule. No clean deterministic fix found — revisit only if the struct pipeline changes the
   landscape (correct struct types may dissolve some).
8. **[MINOR] Per-overlay unique remainder** (non-shared, ×1 leverage — low priority) and the **12 local-type
   skips** (anonymous/typedef'd types `build_engine_types` can't lift — tiny residual).

## Plain-English Recap
The game's 134 "levels" are now all wired into our build, and **136 separate pieces of the game rebuild perfectly,
bit-for-bit, from our own source** — up from 6 at the start of this phase. We went from about 4% of the game's
shared code reconstructed to **over 54%**, mostly by a simple, powerful trick: when the *same* engine function
appears in all 134 levels (and most do), we figure it out **once** and stamp it into all 134 automatically. The
big lesson of the final session was about *cost*: throwing a swarm of 50 AI agents at the leftover hard functions
gained almost nothing (and cost a lot), while a **deterministic fix we wrote by hand** — teaching the tool to
reconcile mismatched function "signatures," plus fixing a quiet bug that had been silently refusing to copy our
matches into the other levels — gained **seven times as much, for free**. We deliberately stopped at a clean wall:
nearly all the *remaining* functions revolve around one big engine data structure (think "the player/enemy record"
with dozens of fields), and our decompiler tools give readable-but-not-buildable code until we define that one
structure. So the next phase is a focused hunt: **define that structure, let the decompiler produce buildable
code, and then let the computer brute-force the exact match for days** while Drew is away — using the full muscle
of his 32-core machine (and possibly his graphics card for some machine-learning ideas). Everything we learned —
the pipeline, the tools, the giants, the dead-ends, the emulator and decomp.me ideas — is written down so nothing
is lost. Both the main program, the engine, and all 134 levels still rebuild identically; nothing we already had
broke.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase15.md` (R19). The Phase-15 matching work is
committed (`commit:0117`..`commit:0120` + this PhaseEnd commit, which lands the PhaseEnd + cookbook §14d/§14e + SETUP +
gen2-roadmap); **Drew pushes** (R6). No Ghidra DB change this phase (pure build-tooling/matching — R23 no-op).
Gen2 continues
— do **NOT** start Phase 16 here. Start a **fresh session** (effort **Max**, plan mode) for **Phase 16 — Crack the
struct-heavy shared core**, opening with the Continuation Backlog above (validate the struct→m2c→permuter pipeline
on a medium sample first). Keep this file forever.