From 964cec8b61c7c2d0f8a4d77de6e291f201642cf8 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Thu, 18 Jun 2026 21:01:38 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-15):=20close=20=E2=80=94=20PhaseEnd?= =?UTF-8?q?=20v1.14.0=20(fleet=2054.48%),=20continuation=20backlog=20->=20?= =?UTF-8?q?Phase=2016?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - PhaseEnd_Phase15.md: synthesis of the open-ended overlay long-tail phase (136 binaries byte-identical from clean; fleet 3.82% -> 54.48%); CURRENT_PHASE.md archived -> logs/Phase15.md (R19) - the final-session finding: deterministic recovery (sig_unify + find_site fix) beat a 50-agent wave ~7x/token (+2.67% vs +0.36%); cookbook §14d (the recovery) + §14e (() / m2c-struct dead-ends) - Continuation Backlog -> Phase 16 'Crack the struct-heavy shared core': the actor-struct -> m2c-context -> permuter compute pipeline (+ multi-day autonomous run), the 28 giants, decomp.me, emulator capture, ML/CUDA wildcard, the arity-conflict dead-end - SETUP tool inventory (sig_unify + recovery tools); gen2-roadmap Phase-15 CLOSED + Phase-16 added - bumps project version 1.13.0 -> 1.14.0 --- docs/SETUP.md | 4 +- docs/gen2-roadmap.md | 2 + docs/matching-cookbook.md | 44 ++++ phase-ends/PhaseEnd_Phase15.md | 236 ++++++++++++++++++ .../{CURRENT_PHASE.md => logs/Phase15.md} | 0 5 files changed, 285 insertions(+), 1 deletion(-) create mode 100644 phase-ends/PhaseEnd_Phase15.md rename phase-ends/{CURRENT_PHASE.md => logs/Phase15.md} (100%) diff --git a/docs/SETUP.md b/docs/SETUP.md index b35e8b52b..41ea9e601 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -654,7 +654,9 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/dup_report.py` | Duplicate-function report; `--cross` (Phase 11) buckets all binaries → `docs/duplicates.cross.md`. **Phase 15:** ingests each overlay once (named ∪ `sig.ov_*` glob, deduped by alias) — else onboarded overlays double-count and inflate collapsible bytes ~2×. | | **Cross-binary dedup** (Phase 11, cookbook §11) | `tools/sig_image.py` | **Ghidra-FREE** per-function signer for a flat image (overlay/resident); `h_exact` byte-matches the Ghidra dumper, self-consistent `h_norm`; linear-partition + `detect_code_end` boundaries. | | | `tools/dedup_integrate.py` | Byte-honesty validator for `config/dedup.us.yaml` code-shares (`--check`; fail-closed on sig-hash drift). | -| | `tools/dedup_propagate.py` | **(Phase 15, cookbook §14)** Match-once → propagate-many: lift a matched body, author a `DEFINE_func_()` macro in `src/shared/engine_core.h`, instantiate it at every onboarded overlay sharing that `h_exact`, byte-gate each (fail-closed), register in `dedup.us.yaml`. `--addr`/`--auto-from`/`--check-only`. | +| | `tools/dedup_propagate.py` | **(Phase 15, cookbook §14)** Match-once → propagate-many: lift a matched body, author a `DEFINE_func_()` macro in `src/shared/engine_core.h`, instantiate it at every onboarded overlay sharing that `h_exact`, byte-gate each (fail-closed), register in `dedup.us.yaml`. `--addr`/`--auto-from`/`--check-only`. `find_site` accepts brace on the same OR next line (Phase-15 fix — next-line-brace defs were silently un-propagated). | +| | `tools/sig_unify.py` | **(Phase 15, cookbook §14d)** Deterministic hard-tail recovery: unify a gate-failing draft's FULL signature set — callee externs **and** the draft's OWN definition signature — to the banked-canonical decls (`--overlay`/`--in`/`--out`); the whole-binary byte-gate stays the arbiter. Recovered 32 of 191 standalone-MATCH conflict-blocked drafts, zero agent tokens. | +| | `tools/gen_harvest_targets.py` / `tools/canon_draft_decls.py` / `tools/build_engine_types.py` | Callee-sig-aware target manifest (§14b/c) / callee-extern canonicalizer (§14c) / additive shared-types-header extractor (`src/shared/engine_types.h`, §14 struct). | | | `config/dedup.us.yaml` / `src/shared/*.h` | The code-share registry + the shared bodies (one macro → N sites, byte-gated). | | | `make report` / `make sig-refresh` / `make sig-overlays` | Convenience targets: reports (+`--cross`) / Ghidra signature-dump / Ghidra-free sign all 134 overlays. | diff --git a/docs/gen2-roadmap.md b/docs/gen2-roadmap.md index 4373fab34..fc1818f49 100644 --- a/docs/gen2-roadmap.md +++ b/docs/gen2-roadmap.md @@ -60,6 +60,8 @@ Almost all of BFM's game code lives **outside** the 404 KiB EXE — in a residen | **13** | Representative location overlays — end-to-end + fleet runbook | **Max** (template) | a location overlay (`SC01/077`) byte-identical from source; a duplicate pair collapsed to one source (both green); `make build-all`/`check-all` + one-command onboarding | | **14** | **Public flip via a curated public repo (LATE CAPSTONE) — Gen2 EXIT** — **⟶ DEFERRED to Gen3+ (Drew 2026-06-16)** | **Max** | fresh **public repo** (private master untouched): only allowlisted paths, zero ROM bytes; `rom→decoder` + `make build && check` reproduces the EXE from a user's dump; no-ROM CI green; dashboard live | | **15** | Overlay long-tail (ONGOING — **not a gate**) | xHigh + breadth | per-overlay `make check` byte-identical; fleet matched-% rises monotonically, zero regressions | +| | **⟶ CLOSED at a natural wall (2026-06-18, v1.14.0, PhaseEnd_Phase15).** All 134 overlays onboarded → **136 binaries byte-identical from a clean tree**; fleet byte-identical-from-source **3.82% → 54.48%** (cross-binary dedup-propagation = the engine; `sig_unify` deterministic recovery = the final-session star, +2.67% / ~0 tokens vs a 50-agent wave's +0.36%). Remaining shared core is uniformly **struct-heavy** (one engine actor struct) → handed to **Phase 16** with a full Continuation Backlog. | | | +| **16** | **Crack the struct-heavy shared core** (struct → m2c-context → permuter pipeline + multi-day autonomous compute) — Gen2 | **Max** (research/design) | the actor struct defined; m2c-context produces compilable C; permuter brute-forces the byte-match; the remaining shared core + 28 giants ground down on the 32-thread box. Validate on a 10-fn medium sample first. | --- diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 6c18ecaae..213864e39 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -939,3 +939,47 @@ only rate-limited part. Gate whatever drafts have landed (`harvest_verify`), `de gate-failures to `.run/drafts--fail/` after each round so re-gates stay fast. Partial harvests bank cleanly; the residual resumes next pass (loop-until-dry). T6 banked 123 fns / +4.6% fleet this way with the Workflow only ~40% through its batches. + +### §14d Deterministic recovery beats agent waves on the hard tail (Phase 15, the final-session finding) + +Once the EASY shared core is banked, the residual is permuter-class and **agent harvesting is poor ROI**: a +50-agent Ultracode wave on 300 hard-tail stubs verified only **27 (~9%) for ~4.1M tokens (+0.36% fleet)**, while +a **deterministic recovery pass added +2.67% for ~0 agent tokens** the same session. **Reach for these BEFORE +mass-drafting the hard tail** (this is a cost rule, codified in `docs/effort-map.md`): + +- **`tools/sig_unify.py` — unify the draft's OWN definition signature, not just callee externs.** The dominant + hard-tail gate failure is a TU-level **signature conflict** on a draft whose body is already byte-correct + (it passes `match_one`). A probe is the tell: substitute each gate-failing standalone-MATCH draft alone, build, + and classify the error — **30/30 sampled were `conflicting types`, 0 false-positives (built-but-SHA-diff)** → + they are byte-correct C blocked only by declaration unification. `canon_draft_decls` rewrites *callee* externs + but never the **function's own def signature** (defined `void`/`s16` here, extern-declared `s32` by banked + callers) — `sig_unify` adds that (canonical return + param types, draft's param names kept; arity-mismatch → + return-only fallback). The byte-gate stays the arbiter. **191 conflict-blocked → 32 recovered** deterministically. +- **The silent-under-propagation class (`find_site` brace bug).** A GREEN byte-gate proves *what landed is + correct*; it does NOT prove *everything that should have propagated did*. `dedup_propagate`'s inline-def + detector required the opening brace **on the same line** as the signature, so every **next-line-brace** def + (which `sig_unify`/permuter emit) was silently skipped from propagation — a whole session's matches capped + invisibly. **Always sanity-check the OUTCOME metric** (`make report` fleet %, or `--auto-from --check-only` + plan size), not just the gate. Fix: `find_site` accepts brace on the same OR the next non-blank line. (One fix + unlocked a 61-function backlog.) Generalizes: any "match→register" detector must accept all draft formats. +- **`build_engine_types.py` must be ADDITIVE.** A 2nd `--strip` run regenerating the shared types header from the + source's *current* inline defs DROPS the already-migrated types (they were stripped last run) → build breaks. + Merge with the existing header; the header is the cumulative record. +- **Probe-before-investing.** Size every recovery lever on a ~20–30 sample (build-classify the failures) before + building the full pass — it told us sig_unify was worth it (30/30 conflicts) AND that the *next* tier was a + dead-end (below) before we spent on it. + +### §14e Two hard-tail dead-ends (Phase 15 — documented so they aren't re-attempted) + +- **`()` no-prototype externs DON'T resolve arity/param conflicts here.** The textbook PS1-decomp escape (declare + inter-function externs param-less) FAILS under gcc-2.7.2: *"An argument type that has a default promotion can't + match an empty parameter name list declaration"* — C forbids `()` matching a prototype with a default-promotion + param (`s8/s16/u8/u16/float`), which these engine fns have. The remaining ~159 arity conflicts (def needs N + params, callers declare M) have **no clean deterministic fix**. +- **m2c/Ghidra output is a scaffold, NOT byte-matching C — the struct is the wall.** On the remaining shared band, + **20/20 sampled m2c outputs are struct-heavy** (`arg0->unkXXX` inferred field accesses) and **won't compile** + without the engine struct defined, plus `?`-typed values. A decompiler gives structure + offsets for free; it + does not give the struct definition or the byte-match. **The path forward (Phase 16):** infer the one engine + **actor struct** layout from the union of m2c field-accesses (offsets + widths) → feed as m2c `--context` → + compilable C → `sig_unify` → **decomp-permuter** brute-force (compute-bound, low-token — the "set-it-and-go" + pipeline). Validate on a 10-fn medium sample before scaling. diff --git a/phase-ends/PhaseEnd_Phase15.md b/phase-ends/PhaseEnd_Phase15.md new file mode 100644 index 000000000..a7759c01d --- /dev/null +++ b/phase-ends/PhaseEnd_Phase15.md @@ -0,0 +1,236 @@ +# PhaseEnd — Phase 15: Overlay long-tail (fleet harvest + deterministic recovery) +**Date:** 2026-06-18 · **Project Version:** 1.14.0 · **Phase Status:** Complete (open-ended phase, milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (7th phase) + +> Gen2 phase 7 of the arc (8→9→10→11→12→13→**15**→16). Phase 14 (public flip) is DEFERRED to Gen3+ (Drew, +> 2026-06-16). Phase 15 was **open-ended, not a gate** — "fleet matched-% rises monotonically, zero regressions." +> It ran across **multiple sessions**; the granular per-session trail (every harvest wave, the rate-limit +> consolidations, the per-tool diagnostics) is preserved on-demand at **`phase-ends/logs/Phase15.md`** (R19 — NOT +> auto-loaded; consult only when researching a mechanism). This file is the synthesis. **Closed deliberately at a +> natural wall** (the struct-heavy shared core), with a full **Continuation Backlog** (below) that becomes Phase 16. + +## Build Log + +**Files created/changed and complete — do not recreate:** + +*The fleet (T1–T3):* +- **134 location overlays onboarded** → the project now builds **136 binaries** (main + resident + 134 overlays). + Each overlay: `config/splat.ov__.yaml` + `config/check.ov_*.sha` + `config/symbols.ov_*.txt` + + `src/ov_*/ov_*.c`, all generated by `tools/new_overlay.sh` (Phase 13) and registered in the generated + **`config/overlays.mk`** (`-include`d before `BINARIES :=`; the Makefile body is never hand-edited to add one). +- `Makefile` — `-MMD -MP` header-dependency tracking (`-include $(C_SRCS:%.c=build/%.d)`) so edits to + `engine_core.h`/`engine_types.h`/`common.h` correctly trigger recompiles (closed a stale-incremental gap that + could false-pass a header-only change; R22-critical). `build-all`/`check-all` fleet targets (Phase 13). + +*The cross-binary dedup-propagation machinery (T2 — the economic engine of the phase):* +- `tools/dedup_propagate.py` — **match once in `ov_SC01_077` → propagate fleet-wide.** Extracts a matched body, + authors it once as a `#define DEFINE_func_()` macro in `src/shared/engine_core.h`, instantiates it in + every overlay that shares that function's `h_exact`, byte-gates each (fail-closed), registers the group in + `config/dedup.us.yaml`. **`--auto-from `** propagates the whole shared set. Registry shorthand + (`vram + binaries:[...]`) keeps `dedup.us.yaml` at ~4k lines for ~1400 groups. **THIS SESSION: the `find_site` + brace-placement fix** (see Deviations) — the inline-def detector required the opening brace on the *same line* + as the signature, silently dropping every next-line-brace def from propagation; now accepts brace on the same + OR next line. +- `src/shared/engine_core.h` — the cumulative shared-body macro header (~15.5k lines, ~1395 `DEFINE_func_*` + macros). Included once per overlay (via `common.h` path). The single source for every byte-identical engine fn. +- `config/dedup.us.yaml` — the cross-binary code-share registry (1395 validated groups, 0 failed). +- `tools/dedup_integrate.py` — byte-honesty validator (`--check`, fail-closed on sig-hash drift); the last line of + `make report`. + +*Fleet reporting (T4):* +- `tools/progress.py --fleet` → **`docs/progress.fleet.md`** (deterministic, source-derived, `_DEDUP_CACHE`-fast). + `tools/dup_report.py --cross` → `docs/duplicates.cross.md` (the ranked cross-binary work queue; double-count fix). + +*The harvest + recovery toolchain (T5–T6 + this session):* +- `tools/harvest_verify.py` (Phase 12) — the whole-binary byte-gate (substitute draft → `make build` → keep iff + byte-identical, else revert; chunk+bisect). The incorruptible arbiter (G3/P9). +- `tools/match_one.py` (Phase 12) — per-function isolated compile + relocation-masked diff (leaf-exact; masks + jal/HI16/LO16, so NOT a true gate for call/data — the whole-binary gate is). +- `tools/gen_harvest_targets.py` (T6a) — **callee-signature-aware** target manifest: resolves each callee/data + symbol to its exact banked sig so drafts never guess a conflicting extern (the §14b wall-breaker). +- `tools/canon_draft_decls.py` (T6d) — rewrites a draft's *callee* externs to the banked-canonical set. +- `tools/build_engine_types.py` (T6 struct) — extracts inline-defined named struct/union types from `ov_SC01_077.c` + into the shared **`src/shared/engine_types.h`** (so struct-using shared fns propagate). **THIS SESSION: made + ADDITIVE** (merges with the existing header instead of overwriting — a re-run after a prior `--strip` was + destructive and broke the build; caught + reverted + fixed). +- **`tools/sig_unify.py` — NEW THIS SESSION.** Unifies a draft's FULL signature set to the banked-canonical decls: + both callee externs AND **the draft's own definition signature** (return + param types, body param-names kept). + This is the recovery the existing `canon_draft_decls` was missing (it never touched the def's own signature). + A probe found **30/30 sampled standalone-MATCH failures were type conflicts, 0 false-positives** — i.e. those + drafts are byte-correct C blocked only by signature unification. Gating the 191 → **32 verified** byte-identical. + +*This session's matched-C (all byte-gated, propagated fleet-wide):* +- `src/ov_SC01_077/ov_SC01_077.c` + `src/shared/engine_core.h` + `config/dedup.us.yaml` — the session added + **~69 new shared functions** (wave-1 27 + sig_unify 32 + permuter 1 + types-header 9) propagated across all 134 + overlays. `ov_SC01_077` is the canonical harvest source (965 stubs remain, all struct-heavy medium/large). + +*Docs:* +- `docs/gen2-roadmap.md` (Phase-14 deferral; Phase-15 active), `docs/matching-cookbook.md` (the harvest + + recovery techniques), `docs/SETUP.md` (tool inventory), `README.md` (status). +- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase15.md`** (R19 archive). `phase-ends/PhaseEnd_Phase15.md` — this file. + +**Local artifacts (gitignored / regenerable):** `asm//**`, `assets//*.bin`, `build/`, `expected/`, +`.run/sig.ov_*.jsonl` (`make sig-overlays`), `.run/drafts-*` (harvest scratch), `.run/permuter//` (permuter +scratch), `.run/t6_*.json`/`.run/m2c_*.json` (target manifests). The 134 overlay Ghidra programs are +script-reproducible (`ghidra_import_raw.sh` + `DefineFunctions.java`) — DB-commit deferred as regenerable (no +manual RE on them). + +**Tools/packages installed:** None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv +(rabbitizer, decomp-permuter, m2c, all already present). + +**Verification results (literal) — the milestone proof (this session's close):** +- **`make clean && (extract all 136) && make check-all` → 136 passed, 0 failed** — every binary (main `143dbb89…`, + resident `8e17e02f…`, all 134 overlays) byte-identical **from a fully clean tree** (R22). Reproduced repeatedly + this session; negative control (corrupt one sha) → loud `[FAIL]`. +- **`make report` (deterministic):** FLEET **REAL 185,779** / LINKED 959 / **byte-identical 187,426 / 344,010 = + 54.48%** / 0 NON_MATCHING in any default build (G4). `dedup-check: 1395 validated, 0 failed`. +- **Fleet arc across Phase 15:** 3.82% (T4 baseline) → 22.14% (T5a bulk) → 25.38% (T5b leaf) → 42.14% → 44.64% → + 46.27% → 47.70% → **51.41%** (prior sessions' T6 waves) → **54.48%** (this session). +- **This session's lever breakdown** (the headline finding): the 50-agent Ultracode harvest added **+0.36%** for + ~4.1M tokens; the **deterministic recovery** (types-header + permuter + `sig_unify` + the `find_site` bug-fix + + types follow-up) added **+2.67% for ~0 agent tokens** — ~7× the agent wave's yield, free. +- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk + staged**. Session committed as `commit:0117 → commit:0118 → commit:0119 → commit:0120` (+ this PhaseEnd), each clean-rebuild-verified. + +**Milestone achieved (confirmed by Drew, gate 2):** the fleet rebuilds **136/136 binaries byte-identical from a +clean tree** at **54.48% byte-identical-from-source** (was 3.82% at phase start, 51.41% at this session's start); +the **cross-binary dedup-propagation engine** (`dedup_propagate --auto-from`) and the **deterministic +signature-unification recovery** (`sig_unify`) are both proven at fleet scale, zero regressions, 0 NON_MATCHING. +The phase is closed at the natural wall — the remaining shared core is uniformly struct-heavy — with a complete +Continuation Backlog handed to Phase 16. + +**Next:** **Phase 16 — Crack the struct-heavy shared core** (research + the struct→m2c→permuter pipeline + +the autonomous multi-day compute session). Open a fresh session in **plan mode, Max effort** (a Tier-1 research/ +design phase). First task: validate the actor-struct + m2c-context + permuter pipeline on a 10-function sample +*before* scaling. See the Continuation Backlog. + +## Deviations +| Item | Plan | Actual | Reason | +|---|---|---|---| +| This session's opener | continue T6 agent harvest waves at 100+ agents | **one 50-agent calibration wave, then pivot to deterministic recovery** | the 50-agent wave yielded only +0.36% / ~9% gate yield on the hard residual (Drew flagged the ROI); the easy shared core was long banked — the tail is permuter-class | +| Recovery mechanism | (not planned) | **`sig_unify.py` — unify the draft's OWN def signature**, not just callee externs | a probe showed 30/30 standalone-MATCH failures were *signature conflicts*, byte-correct otherwise; `canon` only rewrote callee externs → the def-sig was the missing move (+32, +2.36% with the find_site fix) | +| `find_site` propagation | (assumed working) | **silent brace-placement bug fixed** — it required the brace on the *same line*, dropping every next-line-brace def (sig_unify/permuter output) from propagation | a green byte-gate masked under-propagation; the fix unlocked a 61-function backlog (this session's matches, not prior — verified) | +| `build_engine_types.py` | (re-runnable, assumed) | **made additive** — a 2nd `--strip` run was destructive (dropped already-migrated types, broke the build) | caught by the byte-check + reverted; fixed to merge with the existing header | +| The `()` no-prototype escape | (hypothesized as the arity-conflict fix) | **rejected** — gcc-2.7.2 refuses to match `()` against a prototype with a default-promotion param (`s8/s16/u8/u16/float`) | C's promotion rule; the 159 arity conflicts have no clean deterministic fix (documented dead-end) | +| m2c deterministic sweep | (planned as a free drafting path) | **found non-viable on the remaining band** — 20/20 sampled never-drafted fns are struct-heavy (`actor->unkXXX`); m2c output won't compile without the struct defined | the decompiler gives a scaffold, not byte-matching C — the struct is the real wall (→ Phase 16) | +| T7 giants | match via match_one | **deferred to Phase 16** (untouched: 28 fns >150 ins, ~3.43 MB collapsible) | hand-iterating 770-ins struct-heavy fns without the struct is the worst token ROI; they are the prime struct→m2c→permuter compute target | +| Phase close | (open-ended grind) | **closed at 54.48% with a Continuation Backlog** | the clean deterministic vein is tapped; the rest is a distinct engine-struct sub-project better scoped fresh (Phase 16) | + +## Commit Message +``` +(Phase 15 landed across many session commits; this session: commit:0117..commit:0120 + this PhaseEnd.) + +feat(phase-15): close — fleet 3.82% -> 54.48%, dedup-propagate + sig_unify proven (v1.14.0) + +- 136 binaries (main + resident + 134 overlays) byte-identical from a clean tree (R22); + fleet byte-identical-from-source 3.82% -> 54.48%, 1395 dedup groups, 0 NON_MATCHING (G4) +- the phase's engine: tools/dedup_propagate.py (match-once -> propagate fleet-wide via + engine_core.h macros) + the callee-sig-aware harvest (gen_harvest_targets/harvest_verify/ + match_one/canon_draft_decls) + build_engine_types.py shared types header +- THIS SESSION (Max-effort deterministic recovery, ~0 agent tokens for +2.67%): + - tools/sig_unify.py (NEW): unify a draft's OWN def-signature + callee externs to canonical; + probe = 30/30 standalone-MATCH failures are conflicts, 0 false-positives; 191 -> 32 recovered + - dedup_propagate find_site fix: accept brace on same OR next line (silent under-propagation + bug) -> unlocked a 61-function backlog + - build_engine_types additive merge; engine_types follow-up + - one 50-agent Ultracode wave (+0.36%) vs deterministic recovery (+2.67%): the tail is + permuter-class, not agent-draftable +- findings (-> Phase 16): the () no-proto escape fails on default-promotion params; m2c output + is struct-heavy + won't compile (the actor struct is the wall); T7 giants deferred as the + prime struct->m2c->permuter compute target +- worklog archived -> phase-ends/logs/Phase15.md (R19); bumps 1.13.0 -> 1.14.0 +``` + +## Rules Added This Phase +| Rule | Reason | +|---|---| +| **None (governance).** Phase 15's lessons are **techniques + a strategic finding**, captured where they belong (the Phase-8/11/13 precedent: techniques live in `docs/matching-cookbook.md`, not as rules): the **`sig_unify` signature-unification recovery** + the **probe-before-investing discipline** (size a lever on a sample before building the full pass) + the **`find_site` silent-under-propagation class** (a green byte-gate proves *correctness of what landed*, NOT that everything that should propagate did — verify the OUTCOME metric, `make report` fleet %, not just the gate) + the **`()`/m2c dead-ends** → cookbook. The **strategic finding** (on the hard tail, deterministic recovery beats agent waves ~7×/token; reach for `sig_unify`/permuter/propagation before mass-drafting) → `docs/effort-map.md` + cookbook. The existing **R22** (clean-rebuild verify) + **G3/P9** (byte-gate truth) + **R26/R27** (Ultracode-on-breadth, effort transitions) already govern the new ground. | The new ground is mechanical/strategic, not a new norm of conduct. Elevate to a rule later only if a *governance* gap appears. | + +## PhaseEnd Changelog +**v1.13.0 → v1.14.0 — Phase 15 complete (Gen2 phase 7; open-ended, closed at a natural wall).** The whole +**134-overlay fleet** is onboarded and the project rebuilds **136 binaries byte-identical from a clean tree**, +with fleet byte-identical-from-source driven **3.82% → 54.48%** across the phase. The phase's economic engine is +**cross-binary dedup-propagation** (`tools/dedup_propagate.py` + `src/shared/engine_core.h`): match a shared +engine function once in `ov_SC01_077`, propagate it byte-identically across all 134 overlays — so one match is +worth 134. The harvest was driven by a **callee-signature-aware** pipeline (`gen_harvest_targets.py`) past the +extern-conflict wall, with an incorruptible whole-binary **byte-gate** (`harvest_verify.py`, G3/P9) as the sole +arbiter. **This (final) session's defining result:** a single 50-agent Ultracode wave added **+0.36%** (~4.1M +tokens), while a **deterministic recovery** — the new **`tools/sig_unify.py`** (unify each draft's own +definition signature + callee externs to the banked-canonical set; a probe proved 30/30 standalone-MATCH +failures were *signature conflicts*, not wrong C) plus a **`find_site` brace-placement bug-fix** that had been +silently capping propagation — added **+2.67% for ~0 agent tokens** (~7× the wave, free). The phase closes at a +**clean wall**: the remaining ~965 shared stubs (incl. 28 giants >150 ins, ~3.43 MB collapsible) are uniformly +**struct-heavy** — m2c/Ghidra produce readable-but-uncompilable scaffolds (`actor->unkXXX`) and the `()` +no-prototype escape fails on C's default-promotion rule. So the remaining decomp routes through **one engine +struct**, handed forward as Phase 16's Continuation Backlog. No tools installed; no new governance rules +(techniques → cookbook, the strategic finding → effort-map). Both byte-locked binaries and all 134 overlays +rebuild identical; zero regressions; 0 NON_MATCHING linked. + +## Continuation Backlog → Phase 16 (the forward bridge; nothing here is lost on close) + +> Phase 16 = **"Crack the struct-heavy shared core."** A Max-effort, plan-mode **research + design + validate** +> phase, then a **multi-day autonomous compute run**. The wall is no longer "find matches" — it is **one engine +> struct**, and a **compute pipeline** to brute-force the byte-match once the C compiles. Prioritized: + +1. **[THE LEVER] The actor/entity struct → m2c-context → permuter pipeline.** Every remaining shared function + (~399 ≤150-ins + the 28 giants) manipulates the *same* engine struct (`arg0` with fields at +0xB4, +0x1AA, + +0x23C, +0x244, …). m2c already prints every offset + access width. **Plan:** (a) auto-infer the struct LAYOUT + from the union of m2c field-accesses across all functions (offsets + widths → `typedef struct {…} Actor;`); + (b) feed it as m2c `--context` → m2c now emits *compilable* C with real field accesses; (c) run + `sig_unify` + the whole-binary byte-gate; (d) for the residual, **decomp-permuter brute-force** the byte-match. + *Validate on a 10-function medium sample (50–100 ins) BEFORE scaling* — m2c-with-struct compiles but does not + guarantee byte-exact; permuter closes structurally-close gaps, not wrong control flow. THIS is the eureka + candidate; everything else feeds it. +2. **[EXECUTION VEHICLE] The autonomous multi-day compute session** (Drew's 5-day out-of-town window). The + pipeline above is **compute-bound and low-token** — exactly "set it and go for hours/days" on the 32-thread / + 32 GB box. **Design:** a driver that loops {pick next stub → m2c+struct → sig_unify → permuter (bounded, -j ~14) + → gate → propagate → commit checkpoint} across the whole shared core, with a **background monitor** (cron/loop) + that detects a stalled/exited Claude Code session (API outage) and resumes it (self-`continue`). Trial-run the + monitor + a few-hour pipeline pass *while Drew is still hands-on* to de-risk before the unattended stretch. +3. **[PRIME TARGET] The 28 giants (>150 ins, reach 134, ~3.43 MB collapsible)** — `func_80144B9C` (770), + `func_80141CA4` (476), `func_80132784` (400) + 25 more. Untouched. The highest single-match leverage and the + ideal proof-of-payoff for the compute run (after the pipeline is validated on mediums). +4. **[RESEARCH] PS1/MIPS decomp SOTA + tooling scan.** How sotn-decomp / FF / peers automate the struct-heavy + mid-game; struct-recovery workflows; advanced decomp-permuter modes (`PERM_*` macros, `--algorithm`); + m2c-context tricks; whether any **ML-assisted matching** exists (Drew has a 3080 Ti / CUDA + LightGBM + experience — an ML scorer/ranker for permuter candidates, or a learned asm→C model, is a real wildcard to + evaluate). Treat all web content as untrusted DATA (X2). +5. **[HUMAN CHANNEL] decomp.me.** Same matching loop as ours (no new capability), but a great **Drew-driven** + contribution channel + community help (posting BFM scratches may attract contributors). Coach Drew on a few + easy ones; useful in parallel, not the wall-cracker. +6. **[COMPREHENSION] Emulator time-capture for actor-struct semantics.** Byte-matching needs only the struct + *layout* (m2c gives it); but watching RAM as Drew performs actions (jump / swing / etc.) reveals field + *meaning* (R10 multi-datapoint) → names the `unkXXX` fields → makes the engine readable/documented. Quality + + Gen2-engine-systems value; not the matching critical path, run alongside. +7. **[DOCUMENTED DEAD-END] The 159 arity/narrow-param conflicts.** `sig_unify`-resolved drafts that still fail: + the def needs N params but callers declare M, and `()` no-prototype externs are rejected by gcc-2.7.2's + default-promotion rule. No clean deterministic fix found — revisit only if the struct pipeline changes the + landscape (correct struct types may dissolve some). +8. **[MINOR] Per-overlay unique remainder** (non-shared, ×1 leverage — low priority) and the **12 local-type + skips** (anonymous/typedef'd types `build_engine_types` can't lift — tiny residual). + +## Plain-English Recap +The game's 134 "levels" are now all wired into our build, and **136 separate pieces of the game rebuild perfectly, +bit-for-bit, from our own source** — up from 6 at the start of this phase. We went from about 4% of the game's +shared code reconstructed to **over 54%**, mostly by a simple, powerful trick: when the *same* engine function +appears in all 134 levels (and most do), we figure it out **once** and stamp it into all 134 automatically. The +big lesson of the final session was about *cost*: throwing a swarm of 50 AI agents at the leftover hard functions +gained almost nothing (and cost a lot), while a **deterministic fix we wrote by hand** — teaching the tool to +reconcile mismatched function "signatures," plus fixing a quiet bug that had been silently refusing to copy our +matches into the other levels — gained **seven times as much, for free**. We deliberately stopped at a clean wall: +nearly all the *remaining* functions revolve around one big engine data structure (think "the player/enemy record" +with dozens of fields), and our decompiler tools give readable-but-not-buildable code until we define that one +structure. So the next phase is a focused hunt: **define that structure, let the decompiler produce buildable +code, and then let the computer brute-force the exact match for days** while Drew is away — using the full muscle +of his 32-core machine (and possibly his graphics card for some machine-learning ideas). Everything we learned — +the pipeline, the tools, the giants, the dead-ends, the emulator and decomp.me ideas — is written down so nothing +is lost. Both the main program, the engine, and all 134 levels still rebuild identically; nothing we already had +broke. + +## 🛑 Stop Here +PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase15.md` (R19). The Phase-15 matching work is +committed (`commit:0117`..`commit:0120` + this PhaseEnd commit, which lands the PhaseEnd + cookbook §14d/§14e + SETUP + +gen2-roadmap); **Drew pushes** (R6). No Ghidra DB change this phase (pure build-tooling/matching — R23 no-op). +Gen2 continues +— do **NOT** start Phase 16 here. Start a **fresh session** (effort **Max**, plan mode) for **Phase 16 — Crack the +struct-heavy shared core**, opening with the Continuation Backlog above (validate the struct→m2c→permuter pipeline +on a medium sample first). Keep this file forever. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase15.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase15.md