diff --git a/docs/struct-core-pivot.md b/docs/struct-core-pivot.md new file mode 100644 index 000000000..2110daca0 --- /dev/null +++ b/docs/struct-core-pivot.md @@ -0,0 +1,107 @@ +# Struct-Heavy Shared Core — Findings & Pivot (Phase 16) + +> **Status (2026-06-19):** the m2c + decomp-permuter brute-force approach **will not crack** the +> struct-heavy shared core. It yields a few percent, bounded by a fundamental wall (loose typing), +> not by a fixable bug. **Decision (Drew): pause this approach, document, and plan a new idea.** +> This file is the durable record + the seed of the next plan. The harness bug-fixes are real and kept. + +## Objective (unchanged) +Match the ~964 remaining **shared** engine functions in `ov_SC01_077` (struct-heavy; each propagates +×134 overlays via `dedup_propagate`). Cracking them would lift the fleet from 54.5% toward ~90%. + +## What we built and tried (the pipeline) +`m2c --valid-syntax` (+ `common.h` byte-faithful macros `M2C_FIELD`/`NULL`/`s64`) → `sig_unify` +(canonical signatures) → `match_one` (fast prefilter) → **decomp-permuter** (regalloc/schedule search) → +`harvest_verify` **whole-binary byte-gate** (the sole arbiter, G3) → `dedup_propagate` (fleet-wide). +Driver + supervisor + safe-exit (`tools/auto_*.{py,sh}`) built for an unattended 5-day run. + +## Harness bugs found and fixed (REAL improvements — kept regardless of direction) +The byte-gate caught a string of false positives; each fix is committed and correct: +1. **Permuter `base.c` stripped callee externs** → compiled with implicit-`int` callees → matched in the + wrong signature context. Fixed (`make_base_c` keeps the canonical externs). +2. **`run_permuter` counted `output-` dirs as matches** — only `output-0-*` is a true byte-match; + it was calling score-20/145 "best" results "MATCH" (this produced the illusory overnight "42%"). Fixed. +3. **`sig_unify` missed m2c's no-`extern` prototypes** (`M2C_UNK func_X(void *); /* extern */`), so callee + types stayed m2c's guess and the whole-binary build hit `conflicting types`. Fixed (`PROTO_DECL_RE`). +4. **`winner_to_draft` string-replace failed** (permuter reformats typedefs one-per-line) → typedef + redefinition. Fixed (line-filter strip). +These got the harness from "false 42%" to "honest ~1 in 8 on a controlled sample." + +## The fundamental wall: the original code is LOOSELY TYPED +The original BFM engine was written in **K&R-style / loosely-typed C** (PsyQ-era). Evidence (byte-checked): +- The *same* function is called with an **integer at one site and a pointer at another**, and with an + **argument at one site and no args at another** (e.g. `func_8012AAAC` is defined `void func(void)` but + called as `func_8012AAAC(arg0)` from a matched caller — and the caller is byte-correct). +- A **global canonical-decls header** (the planned "lever 1") forces ONE signature per function and + therefore **breaks the existing matches** (`passing arg makes integer from pointer without a cast` + across many already-matched functions). Confirmed and reverted. +- For many functions the **body is byte-correct in isolation** (`match_one` MATCH) but there is **no C + declaration** that simultaneously (a) matches the canonical definition and (b) supports the call site. + These are not fixable by any harness change — they'd need per-call-site type surgery or hand inline-asm. + +## Why this yields ~3%, not the crack +- The easy/consistently-typed functions were **already matched by Phase 15**. What remains is + disproportionately the **loose-typed hard class**. +- **m2c** is a per-function decompiler with weak type inference; it *guesses* a prototype per call, and on + loose-typed code those guesses are mutually inconsistent across the TU. +- The **permuter only fixes register allocation / scheduling** — it cannot fix a wrong callee type, a wrong + symbol, or a structural mis-decompilation, which is what the residual actually needs. +- The **byte-gate correctly rejects** the subtly-wrong drafts (this is good — zero false matches — but it + means the yield is genuinely low, not artificially low). +- Measured gate-verified rate on controlled samples: **~1–2 of 8–16**. A 5-day run would bank a modest + fraction (single-digit to low-double-digit percent of the 964), ×134 propagation ⇒ a few fleet points. + +## Decision +**Pause the m2c+permuter brute-force as the "crack."** Optionally still run it during the away window for +the cheap few percent (the byte-gate guarantees correctness) — but only after the harness is hardened, and +understood as a consolation, not the solution. **The real next step is a new approach + new research.** + +--- + +## New directions to research (the next plan) + +The root cause is **lost type information**. The compiled binary discarded the structs/signatures the +original C had; m2c can't recover them from one function at a time. The promising directions all attack +*that* — recover or model the real types — rather than brute-forcing past them. + +### Direction A — Ghidra's decompiler + GLOBAL type propagation (most promising) +We used m2c (lightweight, per-function). But the overlays are **already imported into Ghidra** (Phase 13), +and Ghidra's decompiler does **whole-program type propagation** — define the actor/entity struct(s) once and +Ghidra propagates field/param types across *all* functions, producing **consistent** signatures (exactly what +the loose-typing wall needs). Most mature decomps (sotn-decomp) start from Ghidra's C, not m2c. **Plan:** +define the actor struct in Ghidra (from analysis + Direction B), let it propagate, export typed C, then +permuter+byte-gate. Test on the known-answer ladder. + +### Direction B — Emulator-guided struct & type recovery (attacks the root cause) +PCSX-Redux (our runtime oracle, R11) can reveal the **true** struct layout and field semantics by watching the +running game (R10 multi-datapoint): which offsets are pointers vs scalars vs arrays, field widths, and the real +argument types at call sites. This recovers the type information the binary lost. Feed it into Ghidra +(Direction A). This is the backlog's "emulator time-capture," elevated from quality-nicety to **the enabling +step** — without real types, no decompiler (m2c or Ghidra) can produce consistent matchable C. + +### Direction C — Model the original's K&R / no-prototype declaration style +The original almost certainly used K&R declarations (`func_X();`, no arg types), which is *why* loose calls +compiled. The textbook `()` escape failed for **default-promotion params** (`s8/s16/u8/u16/float`) — but that's +a minority. **Research:** how do other PSX/PsyQ-era matching decomps (sotn, mednafen-era titles) handle +K&R-style engine code at scale? There may be a known declaration pattern (or a maspsx/compiler-flag trick) that +lets loose calls match without a global canonical signature. Treat web sources as untrusted (X2). + +### Direction D — LLM-in-the-loop WITH gate feedback (narrow, re-evaluated) +Earlier we (correctly) dismissed ML for *one-shot byte-exact generation*. A different, untried shape: an +agentic loop where the model proposes a draft, sees the **byte-gate diff**, and revises — gate-supervised, +not blind (§14d's poor ROI was measured on *blind* drafting). Possibly with the typed Ghidra C as the seed. +Low priority vs A/B, but a bounded experiment once types exist. + +### Research to run (deep-research skill, X2) +1. **Ghidra-decompiler matching workflow** — struct definition + type propagation + export, as used by + sotn-decomp / other PS1 decomps; tooling that bridges Ghidra C → compilable matching C. +2. **Loose-typed / K&R PSX engine matching** — community idioms for inconsistent call-site types. +3. **PS1/Square actor-struct recovery** — has any Square PS1 decomp recovered an entity struct, and how? + +### Recommendation +1. **Pause** the brute-force as the crack (this file). +2. **Optionally**, harden the harness and run the modest brute-force during the away window for the free + few percent (cheap compute, correctness gate-guaranteed) — a consolation, not the plan. +3. **Make Direction B → A the new plan**: emulator-recover the actor struct + field/arg types → define in + Ghidra → global type propagation → Ghidra-C → permuter + byte-gate. Run the deep-research first to ground + it in how the community does it. This attacks the root cause (lost types) instead of brute-forcing past it. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 9b748519e..b615518af 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -60,6 +60,19 @@ K1 type-graph won't compile (GATE-A <8/10) → STOP, fall back to hand-typing to ## Safety invariants Byte-gate (`make build` SHA1) is the only truth (G3/P9). R22 clean-rebuild for byte-checks. **NEVER `git checkout src/.c` during a harvest** (§14c — silently reverts banked matches). Commits on `phase-16-autodrive` branch, explicit pathspec `git add` (never `-A`), local only — owner pushes (R6). State/logs under `.run/auto/` (R12). 0 NON_MATCHING in default builds (G4). +## ⏸️ PIVOT (Fri 2026-06-19, Drew) — m2c+permuter will NOT crack the struct-heavy core +**Decision:** PAUSE the brute-force approach (yields ~3%, bounded by the loose-typing wall, not a fixable +bug). Full findings + new research directions in **`docs/struct-core-pivot.md`**. The harness bug-fixes are +real and kept (output-0 glob, base.c externs, sig_unify no-extern prototypes, winner_to_draft). New plan = +**emulator-recover the actor struct/types → Ghidra global type propagation → Ghidra-C → permuter+gate** +(attacks the root cause: lost types), grounded by deep-research on how PS1 decomps handle loose-typed engine +code. Optionally run the modest brute-force during the away window for the free few % (consolation, not the crack). + +## CRITICAL FINDING (Fri 2026-06-19) — the extern-context bug (byte-gate caught a false 42%) +- Overnight permuter "closed" 17/40 near-misses (42%) — BUT **0/17 whole-binary-gated.** Root cause: the permuter's `base.c` STRIPPED callee externs → compiled with implicit-`int` callees → matched the target in the WRONG signature context. The real whole-binary build declares those callees (engine_core.h) with true signatures → same body, different bytes → no match. **The overnight 42% was illusory.** (G3 working as designed: the byte-gate is the only truth; the permuter score-0 is object-level + context-dependent.) +- **FIX:** `p16_permute.make_base_c` now KEEPS the canonical externs (so the permuter matches in the same signature context as the whole-binary build); `winner_to_draft` strips only the TYPEDEFS block. Driver updated to match. **Re-validating now** (re-permute 6 prior winners with the fix → gate). The TRUE permuter yield is being re-measured. +- Lesson for the run: the permuter step is only valid if base.c's signature context == the whole-binary context. The whole-binary gate (harvest_verify) remains the sole arbiter; never trust the permuter score alone. + ## Timeline (Drew, 2026-06-18 Thu 10:26pm MDT — departs Sun 2026-06-21 afternoon) Must be verified + ready to launch unattended by Sun afternoon. Cadence (I own the launch/test/analyze/iterate loop): - **Thu night / Fri:** build pipeline (S2) + struct inference (S1); **first small known-answer test**. diff --git a/src/ov_SC01_077/ov_SC01_077.c b/src/ov_SC01_077/ov_SC01_077.c index 6601d2582..ed69f26bc 100644 --- a/src/ov_SC01_077/ov_SC01_077.c +++ b/src/ov_SC01_077/ov_SC01_077.c @@ -2612,7 +2612,18 @@ DEFINE_func_8015115C() /* dedup: shared engine-core @0x8015115C (src/shared) */ DEFINE_func_80151164() /* dedup: shared engine-core @0x80151164 (src/shared) */ -INCLUDE_ASM("asm/ov_SC01_077/nonmatchings/ov_SC01_077", func_80151184); + +s32 func_80151184(s32 arg0, s32 arg1, s32 arg2) +{ + u16 *new_var; + int new_var2; + *((s16 *) (((s8 *) arg0) + 0x3E)) = arg1; + *((s16 *) (((s8 *) arg0) + 0x40)) = arg2; + new_var = (u16 *) (((s8 *) arg0) + 0x42); + new_var2 = (u16) (*((u16 *) (((s8 *) (*((void **) (((s8 *) arg0) + 0x20)))) + 0x12))); + *((u16 *) (((s8 *) arg0) + 0x3C)) = (u16) ((*((u16 *) (((s8 *) arg0) + 0x3C))) | 1); + *new_var = new_var2; +} DEFINE_func_801511A8() /* dedup: shared engine-core @0x801511A8 (src/shared) */ diff --git a/tools/auto_driver.py b/tools/auto_driver.py index 188a6c40b..6d7688e6d 100644 --- a/tools/auto_driver.py +++ b/tools/auto_driver.py @@ -192,8 +192,8 @@ def main(): if pd: win = p16_permute.run_permuter(pd, a.permute_secs, a.permute_j) if win: - # draft = the function body only (common.h provides types/macros in the TU) - open(cpath, "w").write(p16_permute.strip_externs_and_includes(open(win).read())) + # draft = externs + permuted body (common.h provides scalars/macros in the TU) + open(cpath, "w").write(p16_permute.winner_to_draft(open(win).read())) cand += 1 if cand == 0: continue diff --git a/tools/gen_engine_decls.py b/tools/gen_engine_decls.py new file mode 100644 index 000000000..2ea5806a3 --- /dev/null +++ b/tools/gen_engine_decls.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""gen_engine_decls.py — the canonical-decls header (Phase 16, Drew's "lever 1", cookbook §14c-c). + +Declare EVERY engine function (from its definition signature) and EVERY data symbol ONCE, canonically, +in src/shared/engine_decls.h. The overlay .c includes it; harvest drafts then strip ALL their own +declarations -> a draft can NEVER conflict with engine_core.h's definitions or another draft's guess. +This dissolves the dominant whole-binary gate-failure (`conflicting types for func_X`) on call-heavy +functions, which m2c declares with a guessed signature (and without `extern`, so sig_unify misses them). + + python3 tools/gen_engine_decls.py # regenerate src/shared/engine_decls.h +""" +import os, re + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +ECORE = os.path.join(REPO, "src/shared/engine_core.h") +OVC = os.path.join(REPO, "src/ov_SC01_077/ov_SC01_077.c") +OUT = os.path.join(REPO, "src/shared/engine_decls.h") +# a function DEFINITION signature: ` func_XXXX() {` (in an engine_core.h macro or inline) +DEF_RE = re.compile(r"\b([A-Za-z_][\w \*]*\bfunc_[0-9A-Fa-f]+\s*\([^){;]*\))\s*\{") +DATA_RE = re.compile(r"extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;") + + +SCALARS = {"void", "u8", "u16", "u32", "s8", "s16", "s32", "u64", "s64", "f32", "f64", "char", + "int", "short", "long", "unsigned", "signed", "float", "double", + "M2C_UNK", "M2C_UNK8", "M2C_UNK16", "M2C_UNK32", "M2C_UNK64"} + + +def known_structs(): + """struct names declared in engine_types.h (forward decls + defs).""" + names = set() + p = os.path.join(REPO, "src/shared/engine_types.h") + if os.path.exists(p): + for m in re.finditer(r"\b(?:struct|union)\s+([A-Za-z_]\w*)", open(p).read()): + names.add(m.group(1)) + return names + + +def resolvable(sig, structs): + """True iff every type token in the signature is a scalar or a known shared struct/union.""" + s = re.sub(r"\bfunc_[0-9A-Fa-f]+\b", "", sig) # drop the function name + s = re.sub(r"\b[a-z_]\w*\s*(?=[,)])", "", s) # drop param names (lowercase, before , or )) + for m in re.finditer(r"\b(struct|union)\s+([A-Za-z_]\w*)", s): + if m.group(2) not in structs: + return False + s2 = re.sub(r"\b(struct|union)\s+\w+", "", s) # remove resolved struct refs + for tok in re.findall(r"\b([A-Za-z_]\w*)\b", s2): + if tok not in SCALARS: + return False + return True + + +def def_sigs(path, structs): + """addr -> canonical prototype string, from every RESOLVABLE function DEFINITION in `path`.""" + out = {} + if not os.path.exists(path): + return out + for m in DEF_RE.finditer(open(path).read()): + sig = re.sub(r"\s+", " ", m.group(1)).strip().rstrip("\\").strip() + if not resolvable(sig, structs): + continue + a = int(re.search(r"func_([0-9A-Fa-f]+)", sig).group(1), 16) + out.setdefault(a, sig) # first definition wins (engine_core.h is authoritative) + return out + + +def data_decls(paths): + out = {} + for p in paths: + if not os.path.exists(p): + continue + for m in DATA_RE.finditer(open(p).read()): + decl = re.sub(r"\s+", " ", m.group(1)).strip() + sym = re.search(r"D_[0-9A-Fa-f]+", decl).group(0) + out.setdefault(sym, decl) + return out + + +def main(): + structs = known_structs() + funcs = {} + funcs.update(def_sigs(ECORE, structs)) # engine_core.h macro defs (authoritative) + for a, s in def_sigs(OVC, structs).items(): # + inline 077-local defs + funcs.setdefault(a, s) + data = {k: v for k, v in data_decls([ECORE, OVC]).items() if resolvable(v, structs)} + + lines = ["#ifndef BFM_ENGINE_DECLS_H", "#define BFM_ENGINE_DECLS_H", + "/* src/shared/engine_decls.h — GENERATED by tools/gen_engine_decls.py (Phase 16 lever 1).", + " * Canonical declaration of every engine function + data symbol, so harvest drafts strip", + " * their own decls and can never `conflicting types`. Regenerate after new matches land. */", + '#include "common.h"', '#include "engine_types.h"', ""] + lines.append(f"/* {len(funcs)} function prototypes */") + for a in sorted(funcs): + lines.append(funcs[a] + ";") + lines.append("") + lines.append(f"/* {len(data)} data symbols */") + for sym in sorted(data): + lines.append(f"extern {data[sym]};") + lines += ["", "#endif"] + open(OUT, "w").write("\n".join(lines) + "\n") + print(f"wrote {OUT}: {len(funcs)} func protos + {len(data)} data decls") + + +if __name__ == "__main__": + main() diff --git a/tools/p16_permute.py b/tools/p16_permute.py index 5dce4fcd3..9c420d4ca 100644 --- a/tools/p16_permute.py +++ b/tools/p16_permute.py @@ -67,25 +67,37 @@ def expand_m2c_field(c): c = c[:i] + repl + c[j:] -def strip_externs_and_includes(c): - """drop #include / #define lines and m2c's leading extern/typedef decls — keep the function def.""" +def drop_preproc_and_scalar_typedefs(c): + """drop #include/#define and scalar/M2C typedef lines (TYPEDEFS/common.h provide them) — + but KEEP `extern` callee/data decls so the function compiles in its REAL signature context.""" lines = [] for ln in c.splitlines(): s = ln.strip() if s.startswith("#"): continue - if re.match(r"^(extern|typedef)\b", s): + if re.match(r"^typedef\b", s): continue lines.append(ln) return "\n".join(lines) def make_base_c(draft_c): + """permuter base.c = scalar typedefs + the draft's canonical externs + the M2C_FIELD-expanded body. + Keeping the externs is essential: without them callees fall back to implicit-int and the permuter + matches in a DIFFERENT context than the whole-binary build (-> winners don't byte-gate).""" body = expand_m2c_field(draft_c) - body = strip_externs_and_includes(body) + body = drop_preproc_and_scalar_typedefs(body) return TYPEDEFS + body + "\n" +def winner_to_draft(winner_c): + """permuter winner (typedefs + externs + body) -> gate-ready draft (externs + body). common.h + provides scalars/macros in the whole-binary TU. The permuter REFORMATS the typedefs (one per + line), so strip by line filter (drop #/typedef lines, keep externs + the function def), NOT a + string-replace of the TYPEDEFS block (that silently fails -> C89 typedef-redefinition error).""" + return drop_preproc_and_scalar_typedefs(winner_c) + + def setup(fn, draft_c): pd = os.path.join(REPO, ".run/permuter", fn) if os.path.exists(pd): @@ -116,7 +128,8 @@ def run_permuter(pd, secs, j): pass # kill stragglers subprocess.run(["pkill", "-f", "decomp-permuter/permuter.py"], capture_output=True) - win = glob.glob(f"{pd}/output-*/source.c") + # ONLY output-0-* is a true byte-match; output--* are intermediate bests (score N != 0) + win = glob.glob(f"{pd}/output-0-*/source.c") return win[0] if win else None diff --git a/tools/sig_unify.py b/tools/sig_unify.py index c7e49b382..86af438b7 100644 --- a/tools/sig_unify.py +++ b/tools/sig_unify.py @@ -36,6 +36,11 @@ _spec.loader.exec_module(_ght) DATA_DECL_RE = re.compile( r'extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;') DRAFT_EXTERN_LINE_RE = re.compile(r'^[ \t]*extern\b[^;]*;[ \t]*$', re.M) +# m2c emits callee prototypes WITHOUT `extern`: ` func_X(); /* extern */`. +# Canonicalize these too, else they keep m2c's GUESSED signature and `conflicting types` in the +# whole-binary build (the body is usually byte-correct; only the decl conflicts). Phase-16 fix. +PROTO_DECL_RE = re.compile( + r'^[ \t]*[A-Za-z_][\w \t\*]*\bfunc_[0-9A-Fa-f]+\s*\([^;{]*\)\s*;[ \t]*(?:/\*[^\n]*\*/)?[ \t]*$', re.M) def sym_of(decl): @@ -178,6 +183,19 @@ def main(): return line txt = DRAFT_EXTERN_LINE_RE.sub(repl, txt) + + def repl_proto(m): + nonlocal ext_changed + line = m.group(0) + s = sym_of(line) + if s and s != fn and s in canon: + new = canon[s] + if re.sub(r'\s+', ' ', new).strip() != re.sub(r'\s+', ' ', line).strip(): + ext_changed = True + return re.match(r'^[ \t]*', line).group(0) + new + return line + + txt = PROTO_DECL_RE.sub(repl_proto, txt) txt, def_changed = rewrite_def(txt, fn, canon) open(os.path.join(REPO, args.outdir, os.path.basename(p)), 'w').write(txt) n += 1