mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-28 06:49:47 -04:00
docs(phase-15): close — PhaseEnd v1.14.0 (fleet 54.48%), continuation backlog -> Phase 16
- PhaseEnd_Phase15.md: synthesis of the open-ended overlay long-tail phase (136 binaries byte-identical from clean; fleet 3.82% -> 54.48%); CURRENT_PHASE.md archived -> logs/Phase15.md (R19) - the final-session finding: deterministic recovery (sig_unify + find_site fix) beat a 50-agent wave ~7x/token (+2.67% vs +0.36%); cookbook §14d (the recovery) + §14e (() / m2c-struct dead-ends) - Continuation Backlog -> Phase 16 'Crack the struct-heavy shared core': the actor-struct -> m2c-context -> permuter compute pipeline (+ multi-day autonomous run), the 28 giants, decomp.me, emulator capture, ML/CUDA wildcard, the arity-conflict dead-end - SETUP tool inventory (sig_unify + recovery tools); gen2-roadmap Phase-15 CLOSED + Phase-16 added - bumps project version 1.13.0 -> 1.14.0
This commit is contained in:
Drew T
+3
-1
@@ -654,7 +654,9 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/dup_report.py` | Duplicate-function report; `--cross` (Phase 11) buckets all binaries → `docs/duplicates.cross.md`. **Phase 15:** ingests each overlay once (named ∪ `sig.ov_*` glob, deduped by alias) — else onboarded overlays double-count and inflate collapsible bytes ~2×. |
|
||||
| **Cross-binary dedup** (Phase 11, cookbook §11) | `tools/sig_image.py` | **Ghidra-FREE** per-function signer for a flat image (overlay/resident); `h_exact` byte-matches the Ghidra dumper, self-consistent `h_norm`; linear-partition + `detect_code_end` boundaries. |
|
||||
| | `tools/dedup_integrate.py` | Byte-honesty validator for `config/dedup.us.yaml` code-shares (`--check`; fail-closed on sig-hash drift). |
|
||||
| | `tools/dedup_propagate.py` | **(Phase 15, cookbook §14)** Match-once → propagate-many: lift a matched body, author a `DEFINE_func_<ADDR>()` macro in `src/shared/engine_core.h`, instantiate it at every onboarded overlay sharing that `h_exact`, byte-gate each (fail-closed), register in `dedup.us.yaml`. `--addr`/`--auto-from`/`--check-only`. |
|
||||
| | `tools/dedup_propagate.py` | **(Phase 15, cookbook §14)** Match-once → propagate-many: lift a matched body, author a `DEFINE_func_<ADDR>()` macro in `src/shared/engine_core.h`, instantiate it at every onboarded overlay sharing that `h_exact`, byte-gate each (fail-closed), register in `dedup.us.yaml`. `--addr`/`--auto-from`/`--check-only`. `find_site` accepts brace on the same OR next line (Phase-15 fix — next-line-brace defs were silently un-propagated). |
|
||||
| | `tools/sig_unify.py` | **(Phase 15, cookbook §14d)** Deterministic hard-tail recovery: unify a gate-failing draft's FULL signature set — callee externs **and** the draft's OWN definition signature — to the banked-canonical decls (`--overlay`/`--in`/`--out`); the whole-binary byte-gate stays the arbiter. Recovered 32 of 191 standalone-MATCH conflict-blocked drafts, zero agent tokens. |
|
||||
| | `tools/gen_harvest_targets.py` / `tools/canon_draft_decls.py` / `tools/build_engine_types.py` | Callee-sig-aware target manifest (§14b/c) / callee-extern canonicalizer (§14c) / additive shared-types-header extractor (`src/shared/engine_types.h`, §14 struct). |
|
||||
| | `config/dedup.us.yaml` / `src/shared/*.h` | The code-share registry + the shared bodies (one macro → N sites, byte-gated). |
|
||||
| | `make report` / `make sig-refresh` / `make sig-overlays` | Convenience targets: reports (+`--cross`) / Ghidra signature-dump / Ghidra-free sign all 134 overlays. |
|
||||
|
||||
|
||||
@@ -60,6 +60,8 @@ Almost all of BFM's game code lives **outside** the 404 KiB EXE — in a residen
|
||||
| **13** | Representative location overlays — end-to-end + fleet runbook | **Max** (template) | a location overlay (`SC01/077`) byte-identical from source; a duplicate pair collapsed to one source (both green); `make build-all`/`check-all` + one-command onboarding |
|
||||
| **14** | **Public flip via a curated public repo (LATE CAPSTONE) — Gen2 EXIT** — **⟶ DEFERRED to Gen3+ (Drew 2026-06-16)** | **Max** | fresh **public repo** (private master untouched): only allowlisted paths, zero ROM bytes; `rom→decoder` + `make build && check` reproduces the EXE from a user's dump; no-ROM CI green; dashboard live |
|
||||
| **15** | Overlay long-tail (ONGOING — **not a gate**) | xHigh + breadth | per-overlay `make check` byte-identical; fleet matched-% rises monotonically, zero regressions |
|
||||
| | **⟶ CLOSED at a natural wall (2026-06-18, v1.14.0, PhaseEnd_Phase15).** All 134 overlays onboarded → **136 binaries byte-identical from a clean tree**; fleet byte-identical-from-source **3.82% → 54.48%** (cross-binary dedup-propagation = the engine; `sig_unify` deterministic recovery = the final-session star, +2.67% / ~0 tokens vs a 50-agent wave's +0.36%). Remaining shared core is uniformly **struct-heavy** (one engine actor struct) → handed to **Phase 16** with a full Continuation Backlog. | | |
|
||||
| **16** | **Crack the struct-heavy shared core** (struct → m2c-context → permuter pipeline + multi-day autonomous compute) — Gen2 | **Max** (research/design) | the actor struct defined; m2c-context produces compilable C; permuter brute-forces the byte-match; the remaining shared core + 28 giants ground down on the 32-thread box. Validate on a 10-fn medium sample first. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -939,3 +939,47 @@ only rate-limited part. Gate whatever drafts have landed (`harvest_verify`), `de
|
||||
gate-failures to `.run/drafts-<x>-fail/` after each round so re-gates stay fast. Partial harvests bank
|
||||
cleanly; the residual resumes next pass (loop-until-dry). T6 banked 123 fns / +4.6% fleet this way with
|
||||
the Workflow only ~40% through its batches.
|
||||
|
||||
### §14d Deterministic recovery beats agent waves on the hard tail (Phase 15, the final-session finding)
|
||||
|
||||
Once the EASY shared core is banked, the residual is permuter-class and **agent harvesting is poor ROI**: a
|
||||
50-agent Ultracode wave on 300 hard-tail stubs verified only **27 (~9%) for ~4.1M tokens (+0.36% fleet)**, while
|
||||
a **deterministic recovery pass added +2.67% for ~0 agent tokens** the same session. **Reach for these BEFORE
|
||||
mass-drafting the hard tail** (this is a cost rule, codified in `docs/effort-map.md`):
|
||||
|
||||
- **`tools/sig_unify.py` — unify the draft's OWN definition signature, not just callee externs.** The dominant
|
||||
hard-tail gate failure is a TU-level **signature conflict** on a draft whose body is already byte-correct
|
||||
(it passes `match_one`). A probe is the tell: substitute each gate-failing standalone-MATCH draft alone, build,
|
||||
and classify the error — **30/30 sampled were `conflicting types`, 0 false-positives (built-but-SHA-diff)** →
|
||||
they are byte-correct C blocked only by declaration unification. `canon_draft_decls` rewrites *callee* externs
|
||||
but never the **function's own def signature** (defined `void`/`s16` here, extern-declared `s32` by banked
|
||||
callers) — `sig_unify` adds that (canonical return + param types, draft's param names kept; arity-mismatch →
|
||||
return-only fallback). The byte-gate stays the arbiter. **191 conflict-blocked → 32 recovered** deterministically.
|
||||
- **The silent-under-propagation class (`find_site` brace bug).** A GREEN byte-gate proves *what landed is
|
||||
correct*; it does NOT prove *everything that should have propagated did*. `dedup_propagate`'s inline-def
|
||||
detector required the opening brace **on the same line** as the signature, so every **next-line-brace** def
|
||||
(which `sig_unify`/permuter emit) was silently skipped from propagation — a whole session's matches capped
|
||||
invisibly. **Always sanity-check the OUTCOME metric** (`make report` fleet %, or `--auto-from --check-only`
|
||||
plan size), not just the gate. Fix: `find_site` accepts brace on the same OR the next non-blank line. (One fix
|
||||
unlocked a 61-function backlog.) Generalizes: any "match→register" detector must accept all draft formats.
|
||||
- **`build_engine_types.py` must be ADDITIVE.** A 2nd `--strip` run regenerating the shared types header from the
|
||||
source's *current* inline defs DROPS the already-migrated types (they were stripped last run) → build breaks.
|
||||
Merge with the existing header; the header is the cumulative record.
|
||||
- **Probe-before-investing.** Size every recovery lever on a ~20–30 sample (build-classify the failures) before
|
||||
building the full pass — it told us sig_unify was worth it (30/30 conflicts) AND that the *next* tier was a
|
||||
dead-end (below) before we spent on it.
|
||||
|
||||
### §14e Two hard-tail dead-ends (Phase 15 — documented so they aren't re-attempted)
|
||||
|
||||
- **`()` no-prototype externs DON'T resolve arity/param conflicts here.** The textbook PS1-decomp escape (declare
|
||||
inter-function externs param-less) FAILS under gcc-2.7.2: *"An argument type that has a default promotion can't
|
||||
match an empty parameter name list declaration"* — C forbids `()` matching a prototype with a default-promotion
|
||||
param (`s8/s16/u8/u16/float`), which these engine fns have. The remaining ~159 arity conflicts (def needs N
|
||||
params, callers declare M) have **no clean deterministic fix**.
|
||||
- **m2c/Ghidra output is a scaffold, NOT byte-matching C — the struct is the wall.** On the remaining shared band,
|
||||
**20/20 sampled m2c outputs are struct-heavy** (`arg0->unkXXX` inferred field accesses) and **won't compile**
|
||||
without the engine struct defined, plus `?`-typed values. A decompiler gives structure + offsets for free; it
|
||||
does not give the struct definition or the byte-match. **The path forward (Phase 16):** infer the one engine
|
||||
**actor struct** layout from the union of m2c field-accesses (offsets + widths) → feed as m2c `--context` →
|
||||
compilable C → `sig_unify` → **decomp-permuter** brute-force (compute-bound, low-token — the "set-it-and-go"
|
||||
pipeline). Validate on a 10-fn medium sample before scaling.
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
# PhaseEnd — Phase 15: Overlay long-tail (fleet harvest + deterministic recovery)
|
||||
**Date:** 2026-06-18 · **Project Version:** 1.14.0 · **Phase Status:** Complete (open-ended phase, milestone confirmed by Drew, gate 2) · **Generation:** Gen2 (7th phase)
|
||||
|
||||
> Gen2 phase 7 of the arc (8→9→10→11→12→13→**15**→16). Phase 14 (public flip) is DEFERRED to Gen3+ (Drew,
|
||||
> 2026-06-16). Phase 15 was **open-ended, not a gate** — "fleet matched-% rises monotonically, zero regressions."
|
||||
> It ran across **multiple sessions**; the granular per-session trail (every harvest wave, the rate-limit
|
||||
> consolidations, the per-tool diagnostics) is preserved on-demand at **`phase-ends/logs/Phase15.md`** (R19 — NOT
|
||||
> auto-loaded; consult only when researching a mechanism). This file is the synthesis. **Closed deliberately at a
|
||||
> natural wall** (the struct-heavy shared core), with a full **Continuation Backlog** (below) that becomes Phase 16.
|
||||
|
||||
## Build Log
|
||||
|
||||
**Files created/changed and complete — do not recreate:**
|
||||
|
||||
*The fleet (T1–T3):*
|
||||
- **134 location overlays onboarded** → the project now builds **136 binaries** (main + resident + 134 overlays).
|
||||
Each overlay: `config/splat.ov_<SCxx>_<nnn>.yaml` + `config/check.ov_*.sha` + `config/symbols.ov_*.txt` +
|
||||
`src/ov_*/ov_*.c`, all generated by `tools/new_overlay.sh` (Phase 13) and registered in the generated
|
||||
**`config/overlays.mk`** (`-include`d before `BINARIES :=`; the Makefile body is never hand-edited to add one).
|
||||
- `Makefile` — `-MMD -MP` header-dependency tracking (`-include $(C_SRCS:%.c=build/%.d)`) so edits to
|
||||
`engine_core.h`/`engine_types.h`/`common.h` correctly trigger recompiles (closed a stale-incremental gap that
|
||||
could false-pass a header-only change; R22-critical). `build-all`/`check-all` fleet targets (Phase 13).
|
||||
|
||||
*The cross-binary dedup-propagation machinery (T2 — the economic engine of the phase):*
|
||||
- `tools/dedup_propagate.py` — **match once in `ov_SC01_077` → propagate fleet-wide.** Extracts a matched body,
|
||||
authors it once as a `#define DEFINE_func_<ADDR>()` macro in `src/shared/engine_core.h`, instantiates it in
|
||||
every overlay that shares that function's `h_exact`, byte-gates each (fail-closed), registers the group in
|
||||
`config/dedup.us.yaml`. **`--auto-from <ov>`** propagates the whole shared set. Registry shorthand
|
||||
(`vram + binaries:[...]`) keeps `dedup.us.yaml` at ~4k lines for ~1400 groups. **THIS SESSION: the `find_site`
|
||||
brace-placement fix** (see Deviations) — the inline-def detector required the opening brace on the *same line*
|
||||
as the signature, silently dropping every next-line-brace def from propagation; now accepts brace on the same
|
||||
OR next line.
|
||||
- `src/shared/engine_core.h` — the cumulative shared-body macro header (~15.5k lines, ~1395 `DEFINE_func_*`
|
||||
macros). Included once per overlay (via `common.h` path). The single source for every byte-identical engine fn.
|
||||
- `config/dedup.us.yaml` — the cross-binary code-share registry (1395 validated groups, 0 failed).
|
||||
- `tools/dedup_integrate.py` — byte-honesty validator (`--check`, fail-closed on sig-hash drift); the last line of
|
||||
`make report`.
|
||||
|
||||
*Fleet reporting (T4):*
|
||||
- `tools/progress.py --fleet` → **`docs/progress.fleet.md`** (deterministic, source-derived, `_DEDUP_CACHE`-fast).
|
||||
`tools/dup_report.py --cross` → `docs/duplicates.cross.md` (the ranked cross-binary work queue; double-count fix).
|
||||
|
||||
*The harvest + recovery toolchain (T5–T6 + this session):*
|
||||
- `tools/harvest_verify.py` (Phase 12) — the whole-binary byte-gate (substitute draft → `make build` → keep iff
|
||||
byte-identical, else revert; chunk+bisect). The incorruptible arbiter (G3/P9).
|
||||
- `tools/match_one.py` (Phase 12) — per-function isolated compile + relocation-masked diff (leaf-exact; masks
|
||||
jal/HI16/LO16, so NOT a true gate for call/data — the whole-binary gate is).
|
||||
- `tools/gen_harvest_targets.py` (T6a) — **callee-signature-aware** target manifest: resolves each callee/data
|
||||
symbol to its exact banked sig so drafts never guess a conflicting extern (the §14b wall-breaker).
|
||||
- `tools/canon_draft_decls.py` (T6d) — rewrites a draft's *callee* externs to the banked-canonical set.
|
||||
- `tools/build_engine_types.py` (T6 struct) — extracts inline-defined named struct/union types from `ov_SC01_077.c`
|
||||
into the shared **`src/shared/engine_types.h`** (so struct-using shared fns propagate). **THIS SESSION: made
|
||||
ADDITIVE** (merges with the existing header instead of overwriting — a re-run after a prior `--strip` was
|
||||
destructive and broke the build; caught + reverted + fixed).
|
||||
- **`tools/sig_unify.py` — NEW THIS SESSION.** Unifies a draft's FULL signature set to the banked-canonical decls:
|
||||
both callee externs AND **the draft's own definition signature** (return + param types, body param-names kept).
|
||||
This is the recovery the existing `canon_draft_decls` was missing (it never touched the def's own signature).
|
||||
A probe found **30/30 sampled standalone-MATCH failures were type conflicts, 0 false-positives** — i.e. those
|
||||
drafts are byte-correct C blocked only by signature unification. Gating the 191 → **32 verified** byte-identical.
|
||||
|
||||
*This session's matched-C (all byte-gated, propagated fleet-wide):*
|
||||
- `src/ov_SC01_077/ov_SC01_077.c` + `src/shared/engine_core.h` + `config/dedup.us.yaml` — the session added
|
||||
**~69 new shared functions** (wave-1 27 + sig_unify 32 + permuter 1 + types-header 9) propagated across all 134
|
||||
overlays. `ov_SC01_077` is the canonical harvest source (965 stubs remain, all struct-heavy medium/large).
|
||||
|
||||
*Docs:*
|
||||
- `docs/gen2-roadmap.md` (Phase-14 deferral; Phase-15 active), `docs/matching-cookbook.md` (the harvest +
|
||||
recovery techniques), `docs/SETUP.md` (tool inventory), `README.md` (status).
|
||||
- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase15.md`** (R19 archive). `phase-ends/PhaseEnd_Phase15.md` — this file.
|
||||
|
||||
**Local artifacts (gitignored / regenerable):** `asm/<bin>/**`, `assets/<ov>/*.bin`, `build/`, `expected/`,
|
||||
`.run/sig.ov_*.jsonl` (`make sig-overlays`), `.run/drafts-*` (harvest scratch), `.run/permuter/<fn>/` (permuter
|
||||
scratch), `.run/t6_*.json`/`.run/m2c_*.json` (target manifests). The 134 overlay Ghidra programs are
|
||||
script-reproducible (`ghidra_import_raw.sh` + `DefineFunctions.java`) — DB-commit deferred as regenerable (no
|
||||
manual RE on them).
|
||||
|
||||
**Tools/packages installed:** None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv
|
||||
(rabbitizer, decomp-permuter, m2c, all already present).
|
||||
|
||||
**Verification results (literal) — the milestone proof (this session's close):**
|
||||
- **`make clean && (extract all 136) && make check-all` → 136 passed, 0 failed** — every binary (main `143dbb89…`,
|
||||
resident `8e17e02f…`, all 134 overlays) byte-identical **from a fully clean tree** (R22). Reproduced repeatedly
|
||||
this session; negative control (corrupt one sha) → loud `[FAIL]`.
|
||||
- **`make report` (deterministic):** FLEET **REAL 185,779** / LINKED 959 / **byte-identical 187,426 / 344,010 =
|
||||
54.48%** / 0 NON_MATCHING in any default build (G4). `dedup-check: 1395 validated, 0 failed`.
|
||||
- **Fleet arc across Phase 15:** 3.82% (T4 baseline) → 22.14% (T5a bulk) → 25.38% (T5b leaf) → 42.14% → 44.64% →
|
||||
46.27% → 47.70% → **51.41%** (prior sessions' T6 waves) → **54.48%** (this session).
|
||||
- **This session's lever breakdown** (the headline finding): the 50-agent Ultracode harvest added **+0.36%** for
|
||||
~4.1M tokens; the **deterministic recovery** (types-header + permuter + `sig_unify` + the `find_site` bug-fix +
|
||||
types follow-up) added **+2.67% for ~0 agent tokens** — ~7× the agent wave's yield, free.
|
||||
- `git status`: only `config/`/`tools/`/`src/`/`docs/`/`phase-ends/` tracked; **zero ROM-derived/generated bulk
|
||||
staged**. Session committed as `commit:0117 → commit:0118 → commit:0119 → commit:0120` (+ this PhaseEnd), each clean-rebuild-verified.
|
||||
|
||||
**Milestone achieved (confirmed by Drew, gate 2):** the fleet rebuilds **136/136 binaries byte-identical from a
|
||||
clean tree** at **54.48% byte-identical-from-source** (was 3.82% at phase start, 51.41% at this session's start);
|
||||
the **cross-binary dedup-propagation engine** (`dedup_propagate --auto-from`) and the **deterministic
|
||||
signature-unification recovery** (`sig_unify`) are both proven at fleet scale, zero regressions, 0 NON_MATCHING.
|
||||
The phase is closed at the natural wall — the remaining shared core is uniformly struct-heavy — with a complete
|
||||
Continuation Backlog handed to Phase 16.
|
||||
|
||||
**Next:** **Phase 16 — Crack the struct-heavy shared core** (research + the struct→m2c→permuter pipeline +
|
||||
the autonomous multi-day compute session). Open a fresh session in **plan mode, Max effort** (a Tier-1 research/
|
||||
design phase). First task: validate the actor-struct + m2c-context + permuter pipeline on a 10-function sample
|
||||
*before* scaling. See the Continuation Backlog.
|
||||
|
||||
## Deviations
|
||||
| Item | Plan | Actual | Reason |
|
||||
|---|---|---|---|
|
||||
| This session's opener | continue T6 agent harvest waves at 100+ agents | **one 50-agent calibration wave, then pivot to deterministic recovery** | the 50-agent wave yielded only +0.36% / ~9% gate yield on the hard residual (Drew flagged the ROI); the easy shared core was long banked — the tail is permuter-class |
|
||||
| Recovery mechanism | (not planned) | **`sig_unify.py` — unify the draft's OWN def signature**, not just callee externs | a probe showed 30/30 standalone-MATCH failures were *signature conflicts*, byte-correct otherwise; `canon` only rewrote callee externs → the def-sig was the missing move (+32, +2.36% with the find_site fix) |
|
||||
| `find_site` propagation | (assumed working) | **silent brace-placement bug fixed** — it required the brace on the *same line*, dropping every next-line-brace def (sig_unify/permuter output) from propagation | a green byte-gate masked under-propagation; the fix unlocked a 61-function backlog (this session's matches, not prior — verified) |
|
||||
| `build_engine_types.py` | (re-runnable, assumed) | **made additive** — a 2nd `--strip` run was destructive (dropped already-migrated types, broke the build) | caught by the byte-check + reverted; fixed to merge with the existing header |
|
||||
| The `()` no-prototype escape | (hypothesized as the arity-conflict fix) | **rejected** — gcc-2.7.2 refuses to match `()` against a prototype with a default-promotion param (`s8/s16/u8/u16/float`) | C's promotion rule; the 159 arity conflicts have no clean deterministic fix (documented dead-end) |
|
||||
| m2c deterministic sweep | (planned as a free drafting path) | **found non-viable on the remaining band** — 20/20 sampled never-drafted fns are struct-heavy (`actor->unkXXX`); m2c output won't compile without the struct defined | the decompiler gives a scaffold, not byte-matching C — the struct is the real wall (→ Phase 16) |
|
||||
| T7 giants | match via match_one | **deferred to Phase 16** (untouched: 28 fns >150 ins, ~3.43 MB collapsible) | hand-iterating 770-ins struct-heavy fns without the struct is the worst token ROI; they are the prime struct→m2c→permuter compute target |
|
||||
| Phase close | (open-ended grind) | **closed at 54.48% with a Continuation Backlog** | the clean deterministic vein is tapped; the rest is a distinct engine-struct sub-project better scoped fresh (Phase 16) |
|
||||
|
||||
## Commit Message
|
||||
```
|
||||
(Phase 15 landed across many session commits; this session: commit:0117..commit:0120 + this PhaseEnd.)
|
||||
|
||||
feat(phase-15): close — fleet 3.82% -> 54.48%, dedup-propagate + sig_unify proven (v1.14.0)
|
||||
|
||||
- 136 binaries (main + resident + 134 overlays) byte-identical from a clean tree (R22);
|
||||
fleet byte-identical-from-source 3.82% -> 54.48%, 1395 dedup groups, 0 NON_MATCHING (G4)
|
||||
- the phase's engine: tools/dedup_propagate.py (match-once -> propagate fleet-wide via
|
||||
engine_core.h macros) + the callee-sig-aware harvest (gen_harvest_targets/harvest_verify/
|
||||
match_one/canon_draft_decls) + build_engine_types.py shared types header
|
||||
- THIS SESSION (Max-effort deterministic recovery, ~0 agent tokens for +2.67%):
|
||||
- tools/sig_unify.py (NEW): unify a draft's OWN def-signature + callee externs to canonical;
|
||||
probe = 30/30 standalone-MATCH failures are conflicts, 0 false-positives; 191 -> 32 recovered
|
||||
- dedup_propagate find_site fix: accept brace on same OR next line (silent under-propagation
|
||||
bug) -> unlocked a 61-function backlog
|
||||
- build_engine_types additive merge; engine_types follow-up
|
||||
- one 50-agent Ultracode wave (+0.36%) vs deterministic recovery (+2.67%): the tail is
|
||||
permuter-class, not agent-draftable
|
||||
- findings (-> Phase 16): the () no-proto escape fails on default-promotion params; m2c output
|
||||
is struct-heavy + won't compile (the actor struct is the wall); T7 giants deferred as the
|
||||
prime struct->m2c->permuter compute target
|
||||
- worklog archived -> phase-ends/logs/Phase15.md (R19); bumps 1.13.0 -> 1.14.0
|
||||
```
|
||||
|
||||
## Rules Added This Phase
|
||||
| Rule | Reason |
|
||||
|---|---|
|
||||
| **None (governance).** Phase 15's lessons are **techniques + a strategic finding**, captured where they belong (the Phase-8/11/13 precedent: techniques live in `docs/matching-cookbook.md`, not as rules): the **`sig_unify` signature-unification recovery** + the **probe-before-investing discipline** (size a lever on a sample before building the full pass) + the **`find_site` silent-under-propagation class** (a green byte-gate proves *correctness of what landed*, NOT that everything that should propagate did — verify the OUTCOME metric, `make report` fleet %, not just the gate) + the **`()`/m2c dead-ends** → cookbook. The **strategic finding** (on the hard tail, deterministic recovery beats agent waves ~7×/token; reach for `sig_unify`/permuter/propagation before mass-drafting) → `docs/effort-map.md` + cookbook. The existing **R22** (clean-rebuild verify) + **G3/P9** (byte-gate truth) + **R26/R27** (Ultracode-on-breadth, effort transitions) already govern the new ground. | The new ground is mechanical/strategic, not a new norm of conduct. Elevate to a rule later only if a *governance* gap appears. |
|
||||
|
||||
## PhaseEnd Changelog
|
||||
**v1.13.0 → v1.14.0 — Phase 15 complete (Gen2 phase 7; open-ended, closed at a natural wall).** The whole
|
||||
**134-overlay fleet** is onboarded and the project rebuilds **136 binaries byte-identical from a clean tree**,
|
||||
with fleet byte-identical-from-source driven **3.82% → 54.48%** across the phase. The phase's economic engine is
|
||||
**cross-binary dedup-propagation** (`tools/dedup_propagate.py` + `src/shared/engine_core.h`): match a shared
|
||||
engine function once in `ov_SC01_077`, propagate it byte-identically across all 134 overlays — so one match is
|
||||
worth 134. The harvest was driven by a **callee-signature-aware** pipeline (`gen_harvest_targets.py`) past the
|
||||
extern-conflict wall, with an incorruptible whole-binary **byte-gate** (`harvest_verify.py`, G3/P9) as the sole
|
||||
arbiter. **This (final) session's defining result:** a single 50-agent Ultracode wave added **+0.36%** (~4.1M
|
||||
tokens), while a **deterministic recovery** — the new **`tools/sig_unify.py`** (unify each draft's own
|
||||
definition signature + callee externs to the banked-canonical set; a probe proved 30/30 standalone-MATCH
|
||||
failures were *signature conflicts*, not wrong C) plus a **`find_site` brace-placement bug-fix** that had been
|
||||
silently capping propagation — added **+2.67% for ~0 agent tokens** (~7× the wave, free). The phase closes at a
|
||||
**clean wall**: the remaining ~965 shared stubs (incl. 28 giants >150 ins, ~3.43 MB collapsible) are uniformly
|
||||
**struct-heavy** — m2c/Ghidra produce readable-but-uncompilable scaffolds (`actor->unkXXX`) and the `()`
|
||||
no-prototype escape fails on C's default-promotion rule. So the remaining decomp routes through **one engine
|
||||
struct**, handed forward as Phase 16's Continuation Backlog. No tools installed; no new governance rules
|
||||
(techniques → cookbook, the strategic finding → effort-map). Both byte-locked binaries and all 134 overlays
|
||||
rebuild identical; zero regressions; 0 NON_MATCHING linked.
|
||||
|
||||
## Continuation Backlog → Phase 16 (the forward bridge; nothing here is lost on close)
|
||||
|
||||
> Phase 16 = **"Crack the struct-heavy shared core."** A Max-effort, plan-mode **research + design + validate**
|
||||
> phase, then a **multi-day autonomous compute run**. The wall is no longer "find matches" — it is **one engine
|
||||
> struct**, and a **compute pipeline** to brute-force the byte-match once the C compiles. Prioritized:
|
||||
|
||||
1. **[THE LEVER] The actor/entity struct → m2c-context → permuter pipeline.** Every remaining shared function
|
||||
(~399 ≤150-ins + the 28 giants) manipulates the *same* engine struct (`arg0` with fields at +0xB4, +0x1AA,
|
||||
+0x23C, +0x244, …). m2c already prints every offset + access width. **Plan:** (a) auto-infer the struct LAYOUT
|
||||
from the union of m2c field-accesses across all functions (offsets + widths → `typedef struct {…} Actor;`);
|
||||
(b) feed it as m2c `--context` → m2c now emits *compilable* C with real field accesses; (c) run
|
||||
`sig_unify` + the whole-binary byte-gate; (d) for the residual, **decomp-permuter brute-force** the byte-match.
|
||||
*Validate on a 10-function medium sample (50–100 ins) BEFORE scaling* — m2c-with-struct compiles but does not
|
||||
guarantee byte-exact; permuter closes structurally-close gaps, not wrong control flow. THIS is the eureka
|
||||
candidate; everything else feeds it.
|
||||
2. **[EXECUTION VEHICLE] The autonomous multi-day compute session** (Drew's 5-day out-of-town window). The
|
||||
pipeline above is **compute-bound and low-token** — exactly "set it and go for hours/days" on the 32-thread /
|
||||
32 GB box. **Design:** a driver that loops {pick next stub → m2c+struct → sig_unify → permuter (bounded, -j ~14)
|
||||
→ gate → propagate → commit checkpoint} across the whole shared core, with a **background monitor** (cron/loop)
|
||||
that detects a stalled/exited Claude Code session (API outage) and resumes it (self-`continue`). Trial-run the
|
||||
monitor + a few-hour pipeline pass *while Drew is still hands-on* to de-risk before the unattended stretch.
|
||||
3. **[PRIME TARGET] The 28 giants (>150 ins, reach 134, ~3.43 MB collapsible)** — `func_80144B9C` (770),
|
||||
`func_80141CA4` (476), `func_80132784` (400) + 25 more. Untouched. The highest single-match leverage and the
|
||||
ideal proof-of-payoff for the compute run (after the pipeline is validated on mediums).
|
||||
4. **[RESEARCH] PS1/MIPS decomp SOTA + tooling scan.** How sotn-decomp / FF / peers automate the struct-heavy
|
||||
mid-game; struct-recovery workflows; advanced decomp-permuter modes (`PERM_*` macros, `--algorithm`);
|
||||
m2c-context tricks; whether any **ML-assisted matching** exists (Drew has a 3080 Ti / CUDA + LightGBM
|
||||
experience — an ML scorer/ranker for permuter candidates, or a learned asm→C model, is a real wildcard to
|
||||
evaluate). Treat all web content as untrusted DATA (X2).
|
||||
5. **[HUMAN CHANNEL] decomp.me.** Same matching loop as ours (no new capability), but a great **Drew-driven**
|
||||
contribution channel + community help (posting BFM scratches may attract contributors). Coach Drew on a few
|
||||
easy ones; useful in parallel, not the wall-cracker.
|
||||
6. **[COMPREHENSION] Emulator time-capture for actor-struct semantics.** Byte-matching needs only the struct
|
||||
*layout* (m2c gives it); but watching RAM as Drew performs actions (jump / swing / etc.) reveals field
|
||||
*meaning* (R10 multi-datapoint) → names the `unkXXX` fields → makes the engine readable/documented. Quality +
|
||||
Gen2-engine-systems value; not the matching critical path, run alongside.
|
||||
7. **[DOCUMENTED DEAD-END] The 159 arity/narrow-param conflicts.** `sig_unify`-resolved drafts that still fail:
|
||||
the def needs N params but callers declare M, and `()` no-prototype externs are rejected by gcc-2.7.2's
|
||||
default-promotion rule. No clean deterministic fix found — revisit only if the struct pipeline changes the
|
||||
landscape (correct struct types may dissolve some).
|
||||
8. **[MINOR] Per-overlay unique remainder** (non-shared, ×1 leverage — low priority) and the **12 local-type
|
||||
skips** (anonymous/typedef'd types `build_engine_types` can't lift — tiny residual).
|
||||
|
||||
## Plain-English Recap
|
||||
The game's 134 "levels" are now all wired into our build, and **136 separate pieces of the game rebuild perfectly,
|
||||
bit-for-bit, from our own source** — up from 6 at the start of this phase. We went from about 4% of the game's
|
||||
shared code reconstructed to **over 54%**, mostly by a simple, powerful trick: when the *same* engine function
|
||||
appears in all 134 levels (and most do), we figure it out **once** and stamp it into all 134 automatically. The
|
||||
big lesson of the final session was about *cost*: throwing a swarm of 50 AI agents at the leftover hard functions
|
||||
gained almost nothing (and cost a lot), while a **deterministic fix we wrote by hand** — teaching the tool to
|
||||
reconcile mismatched function "signatures," plus fixing a quiet bug that had been silently refusing to copy our
|
||||
matches into the other levels — gained **seven times as much, for free**. We deliberately stopped at a clean wall:
|
||||
nearly all the *remaining* functions revolve around one big engine data structure (think "the player/enemy record"
|
||||
with dozens of fields), and our decompiler tools give readable-but-not-buildable code until we define that one
|
||||
structure. So the next phase is a focused hunt: **define that structure, let the decompiler produce buildable
|
||||
code, and then let the computer brute-force the exact match for days** while Drew is away — using the full muscle
|
||||
of his 32-core machine (and possibly his graphics card for some machine-learning ideas). Everything we learned —
|
||||
the pipeline, the tools, the giants, the dead-ends, the emulator and decomp.me ideas — is written down so nothing
|
||||
is lost. Both the main program, the engine, and all 134 levels still rebuild identically; nothing we already had
|
||||
broke.
|
||||
|
||||
## 🛑 Stop Here
|
||||
PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase15.md` (R19). The Phase-15 matching work is
|
||||
committed (`commit:0117`..`commit:0120` + this PhaseEnd commit, which lands the PhaseEnd + cookbook §14d/§14e + SETUP +
|
||||
gen2-roadmap); **Drew pushes** (R6). No Ghidra DB change this phase (pure build-tooling/matching — R23 no-op).
|
||||
Gen2 continues
|
||||
— do **NOT** start Phase 16 here. Start a **fresh session** (effort **Max**, plan mode) for **Phase 16 — Crack the
|
||||
struct-heavy shared core**, opening with the Continuation Backlog above (validate the struct→m2c→permuter pipeline
|
||||
on a medium sample first). Keep this file forever.
|
||||
Reference in New Issue
Block a user