docs(phase-16): PIVOT — m2c+permuter won't crack the loose-typed core; new plan + harness fixes

- docs/struct-core-pivot.md: findings + decision + new research directions. Root cause = the
  original engine is LOOSELY TYPED (K&R; same fn called with int/ptr, arg/no-arg across sites),
  so no single canonical signature exists -> m2c guesses inconsistently, permuter can't fix
  semantics, byte-gate (correctly) rejects. Yields ~3%, not the crack. New plan: emulator-recover
  the actor struct/types -> Ghidra global type propagation -> Ghidra-C -> permuter+gate.
- harness bug-fixes (REAL, kept): p16_permute output-0-only match (killed the false '42%'),
  base.c keeps callee externs, winner_to_draft line-strip; sig_unify canonicalizes m2c's
  no-extern prototypes; gen_engine_decls.py (documents why a global canonical header breaks
  loose-typed matches).
- a few byte-gated leaf matches banked in ov_SC01_077.c.

docs/struct-core-pivot.md

@@ -0,0 +1,107 @@
+# Struct-Heavy Shared Core — Findings & Pivot (Phase 16)
+
+> **Status (2026-06-19):** the m2c + decomp-permuter brute-force approach **will not crack** the
+> struct-heavy shared core. It yields a few percent, bounded by a fundamental wall (loose typing),
+> not by a fixable bug. **Decision (Drew): pause this approach, document, and plan a new idea.**
+> This file is the durable record + the seed of the next plan. The harness bug-fixes are real and kept.
+
+## Objective (unchanged)
+Match the ~964 remaining **shared** engine functions in `ov_SC01_077` (struct-heavy; each propagates
+×134 overlays via `dedup_propagate`). Cracking them would lift the fleet from 54.5% toward ~90%.
+
+## What we built and tried (the pipeline)
+`m2c --valid-syntax` (+ `common.h` byte-faithful macros `M2C_FIELD`/`NULL`/`s64`) → `sig_unify`
+(canonical signatures) → `match_one` (fast prefilter) → **decomp-permuter** (regalloc/schedule search) →
+`harvest_verify` **whole-binary byte-gate** (the sole arbiter, G3) → `dedup_propagate` (fleet-wide).
+Driver + supervisor + safe-exit (`tools/auto_*.{py,sh}`) built for an unattended 5-day run.
+
+## Harness bugs found and fixed (REAL improvements — kept regardless of direction)
+The byte-gate caught a string of false positives; each fix is committed and correct:
+1. **Permuter `base.c` stripped callee externs** → compiled with implicit-`int` callees → matched in the
+   wrong signature context. Fixed (`make_base_c` keeps the canonical externs).
+2. **`run_permuter` counted `output-<score>` dirs as matches** — only `output-0-*` is a true byte-match;
+   it was calling score-20/145 "best" results "MATCH" (this produced the illusory overnight "42%"). Fixed.
+3. **`sig_unify` missed m2c's no-`extern` prototypes** (`M2C_UNK func_X(void *); /* extern */`), so callee
+   types stayed m2c's guess and the whole-binary build hit `conflicting types`. Fixed (`PROTO_DECL_RE`).
+4. **`winner_to_draft` string-replace failed** (permuter reformats typedefs one-per-line) → typedef
+   redefinition. Fixed (line-filter strip).
+These got the harness from "false 42%" to "honest ~1 in 8 on a controlled sample."
+
+## The fundamental wall: the original code is LOOSELY TYPED
+The original BFM engine was written in **K&R-style / loosely-typed C** (PsyQ-era). Evidence (byte-checked):
+- The *same* function is called with an **integer at one site and a pointer at another**, and with an
+  **argument at one site and no args at another** (e.g. `func_8012AAAC` is defined `void func(void)` but
+  called as `func_8012AAAC(arg0)` from a matched caller — and the caller is byte-correct).
+- A **global canonical-decls header** (the planned "lever 1") forces ONE signature per function and
+  therefore **breaks the existing matches** (`passing arg makes integer from pointer without a cast`
+  across many already-matched functions). Confirmed and reverted.
+- For many functions the **body is byte-correct in isolation** (`match_one` MATCH) but there is **no C
+  declaration** that simultaneously (a) matches the canonical definition and (b) supports the call site.
+  These are not fixable by any harness change — they'd need per-call-site type surgery or hand inline-asm.
+
+## Why this yields ~3%, not the crack
+- The easy/consistently-typed functions were **already matched by Phase 15**. What remains is
+  disproportionately the **loose-typed hard class**.
+- **m2c** is a per-function decompiler with weak type inference; it *guesses* a prototype per call, and on
+  loose-typed code those guesses are mutually inconsistent across the TU.
+- The **permuter only fixes register allocation / scheduling** — it cannot fix a wrong callee type, a wrong
+  symbol, or a structural mis-decompilation, which is what the residual actually needs.
+- The **byte-gate correctly rejects** the subtly-wrong drafts (this is good — zero false matches — but it
+  means the yield is genuinely low, not artificially low).
+- Measured gate-verified rate on controlled samples: **~1–2 of 8–16**. A 5-day run would bank a modest
+  fraction (single-digit to low-double-digit percent of the 964), ×134 propagation ⇒ a few fleet points.
+
+## Decision
+**Pause the m2c+permuter brute-force as the "crack."** Optionally still run it during the away window for
+the cheap few percent (the byte-gate guarantees correctness) — but only after the harness is hardened, and
+understood as a consolation, not the solution. **The real next step is a new approach + new research.**
+
+---
+
+## New directions to research (the next plan)
+
+The root cause is **lost type information**. The compiled binary discarded the structs/signatures the
+original C had; m2c can't recover them from one function at a time. The promising directions all attack
+*that* — recover or model the real types — rather than brute-forcing past them.
+
+### Direction A — Ghidra's decompiler + GLOBAL type propagation (most promising)
+We used m2c (lightweight, per-function). But the overlays are **already imported into Ghidra** (Phase 13),
+and Ghidra's decompiler does **whole-program type propagation** — define the actor/entity struct(s) once and
+Ghidra propagates field/param types across *all* functions, producing **consistent** signatures (exactly what
+the loose-typing wall needs). Most mature decomps (sotn-decomp) start from Ghidra's C, not m2c. **Plan:**
+define the actor struct in Ghidra (from analysis + Direction B), let it propagate, export typed C, then
+permuter+byte-gate. Test on the known-answer ladder.
+
+### Direction B — Emulator-guided struct & type recovery (attacks the root cause)
+PCSX-Redux (our runtime oracle, R11) can reveal the **true** struct layout and field semantics by watching the
+running game (R10 multi-datapoint): which offsets are pointers vs scalars vs arrays, field widths, and the real
+argument types at call sites. This recovers the type information the binary lost. Feed it into Ghidra
+(Direction A). This is the backlog's "emulator time-capture," elevated from quality-nicety to **the enabling
+step** — without real types, no decompiler (m2c or Ghidra) can produce consistent matchable C.
+
+### Direction C — Model the original's K&R / no-prototype declaration style
+The original almost certainly used K&R declarations (`func_X();`, no arg types), which is *why* loose calls
+compiled. The textbook `()` escape failed for **default-promotion params** (`s8/s16/u8/u16/float`) — but that's
+a minority. **Research:** how do other PSX/PsyQ-era matching decomps (sotn, mednafen-era titles) handle
+K&R-style engine code at scale? There may be a known declaration pattern (or a maspsx/compiler-flag trick) that
+lets loose calls match without a global canonical signature. Treat web sources as untrusted (X2).
+
+### Direction D — LLM-in-the-loop WITH gate feedback (narrow, re-evaluated)
+Earlier we (correctly) dismissed ML for *one-shot byte-exact generation*. A different, untried shape: an
+agentic loop where the model proposes a draft, sees the **byte-gate diff**, and revises — gate-supervised,
+not blind (§14d's poor ROI was measured on *blind* drafting). Possibly with the typed Ghidra C as the seed.
+Low priority vs A/B, but a bounded experiment once types exist.
+
+### Research to run (deep-research skill, X2)
+1. **Ghidra-decompiler matching workflow** — struct definition + type propagation + export, as used by
+   sotn-decomp / other PS1 decomps; tooling that bridges Ghidra C → compilable matching C.
+2. **Loose-typed / K&R PSX engine matching** — community idioms for inconsistent call-site types.
+3. **PS1/Square actor-struct recovery** — has any Square PS1 decomp recovered an entity struct, and how?
+
+### Recommendation
+1. **Pause** the brute-force as the crack (this file).
+2. **Optionally**, harden the harness and run the modest brute-force during the away window for the free
+   few percent (cheap compute, correctness gate-guaranteed) — a consolation, not the plan.
+3. **Make Direction B → A the new plan**: emulator-recover the actor struct + field/arg types → define in
+   Ghidra → global type propagation → Ghidra-C → permuter + byte-gate. Run the deep-research first to ground
+   it in how the community does it. This attacks the root cause (lost types) instead of brute-forcing past it.

phase-ends/CURRENT_PHASE.md

@@ -60,6 +60,19 @@ K1 type-graph won't compile (GATE-A <8/10) → STOP, fall back to hand-typing to
 ## Safety invariants
 Byte-gate (`make build` SHA1) is the only truth (G3/P9). R22 clean-rebuild for byte-checks. **NEVER `git checkout src/<overlay>.c` during a harvest** (§14c — silently reverts banked matches). Commits on `phase-16-autodrive` branch, explicit pathspec `git add` (never `-A`), local only — owner pushes (R6). State/logs under `.run/auto/` (R12). 0 NON_MATCHING in default builds (G4).
 
+## ⏸️ PIVOT (Fri 2026-06-19, Drew) — m2c+permuter will NOT crack the struct-heavy core
+**Decision:** PAUSE the brute-force approach (yields ~3%, bounded by the loose-typing wall, not a fixable
+bug). Full findings + new research directions in **`docs/struct-core-pivot.md`**. The harness bug-fixes are
+real and kept (output-0 glob, base.c externs, sig_unify no-extern prototypes, winner_to_draft). New plan =
+**emulator-recover the actor struct/types → Ghidra global type propagation → Ghidra-C → permuter+gate**
+(attacks the root cause: lost types), grounded by deep-research on how PS1 decomps handle loose-typed engine
+code. Optionally run the modest brute-force during the away window for the free few % (consolation, not the crack).
+
+## CRITICAL FINDING (Fri 2026-06-19) — the extern-context bug (byte-gate caught a false 42%)
+- Overnight permuter "closed" 17/40 near-misses (42%) — BUT **0/17 whole-binary-gated.** Root cause: the permuter's `base.c` STRIPPED callee externs → compiled with implicit-`int` callees → matched the target in the WRONG signature context. The real whole-binary build declares those callees (engine_core.h) with true signatures → same body, different bytes → no match. **The overnight 42% was illusory.** (G3 working as designed: the byte-gate is the only truth; the permuter score-0 is object-level + context-dependent.)
+- **FIX:** `p16_permute.make_base_c` now KEEPS the canonical externs (so the permuter matches in the same signature context as the whole-binary build); `winner_to_draft` strips only the TYPEDEFS block. Driver updated to match. **Re-validating now** (re-permute 6 prior winners with the fix → gate). The TRUE permuter yield is being re-measured.
+- Lesson for the run: the permuter step is only valid if base.c's signature context == the whole-binary context. The whole-binary gate (harvest_verify) remains the sole arbiter; never trust the permuter score alone.
+
 ## Timeline (Drew, 2026-06-18 Thu 10:26pm MDT — departs Sun 2026-06-21 afternoon)
 Must be verified + ready to launch unattended by Sun afternoon. Cadence (I own the launch/test/analyze/iterate loop):
 - **Thu night / Fri:** build pipeline (S2) + struct inference (S1); **first small known-answer test**.

src/ov_SC01_077/ov_SC01_077.c

@@ -2612,7 +2612,18 @@ DEFINE_func_8015115C()  /* dedup: shared engine-core @0x8015115C (src/shared) */
 
 DEFINE_func_80151164()  /* dedup: shared engine-core @0x80151164 (src/shared) */
 
-INCLUDE_ASM("asm/ov_SC01_077/nonmatchings/ov_SC01_077", func_80151184);
+
+s32 func_80151184(s32 arg0, s32 arg1, s32 arg2)
+{
+  u16 *new_var;
+  int new_var2;
+  *((s16 *) (((s8 *) arg0) + 0x3E)) = arg1;
+  *((s16 *) (((s8 *) arg0) + 0x40)) = arg2;
+  new_var = (u16 *) (((s8 *) arg0) + 0x42);
+  new_var2 = (u16) (*((u16 *) (((s8 *) (*((void **) (((s8 *) arg0) + 0x20)))) + 0x12)));
+  *((u16 *) (((s8 *) arg0) + 0x3C)) = (u16) ((*((u16 *) (((s8 *) arg0) + 0x3C))) | 1);
+  *new_var = new_var2;
+}
 
 DEFINE_func_801511A8()  /* dedup: shared engine-core @0x801511A8 (src/shared) */
 

tools/auto_driver.py

@@ -192,8 +192,8 @@ def main():
                     if pd:
                         win = p16_permute.run_permuter(pd, a.permute_secs, a.permute_j)
                         if win:
-                            # draft = the function body only (common.h provides types/macros in the TU)
-                            open(cpath, "w").write(p16_permute.strip_externs_and_includes(open(win).read()))
+                            # draft = externs + permuted body (common.h provides scalars/macros in the TU)
+                            open(cpath, "w").write(p16_permute.winner_to_draft(open(win).read()))
                 cand += 1
             if cand == 0:
                 continue

tools/gen_engine_decls.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""gen_engine_decls.py — the canonical-decls header (Phase 16, Drew's "lever 1", cookbook §14c-c).
+
+Declare EVERY engine function (from its definition signature) and EVERY data symbol ONCE, canonically,
+in src/shared/engine_decls.h. The overlay .c includes it; harvest drafts then strip ALL their own
+declarations -> a draft can NEVER conflict with engine_core.h's definitions or another draft's guess.
+This dissolves the dominant whole-binary gate-failure (`conflicting types for func_X`) on call-heavy
+functions, which m2c declares with a guessed signature (and without `extern`, so sig_unify misses them).
+
+  python3 tools/gen_engine_decls.py            # regenerate src/shared/engine_decls.h
+"""
+import os, re
+
+REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ECORE = os.path.join(REPO, "src/shared/engine_core.h")
+OVC = os.path.join(REPO, "src/ov_SC01_077/ov_SC01_077.c")
+OUT = os.path.join(REPO, "src/shared/engine_decls.h")
+# a function DEFINITION signature: `<ret> func_XXXX(<params>) {`  (in an engine_core.h macro or inline)
+DEF_RE = re.compile(r"\b([A-Za-z_][\w \*]*\bfunc_[0-9A-Fa-f]+\s*\([^){;]*\))\s*\{")
+DATA_RE = re.compile(r"extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;")
+
+
+SCALARS = {"void", "u8", "u16", "u32", "s8", "s16", "s32", "u64", "s64", "f32", "f64", "char",
+           "int", "short", "long", "unsigned", "signed", "float", "double",
+           "M2C_UNK", "M2C_UNK8", "M2C_UNK16", "M2C_UNK32", "M2C_UNK64"}
+
+
+def known_structs():
+    """struct names declared in engine_types.h (forward decls + defs)."""
+    names = set()
+    p = os.path.join(REPO, "src/shared/engine_types.h")
+    if os.path.exists(p):
+        for m in re.finditer(r"\b(?:struct|union)\s+([A-Za-z_]\w*)", open(p).read()):
+            names.add(m.group(1))
+    return names
+
+
+def resolvable(sig, structs):
+    """True iff every type token in the signature is a scalar or a known shared struct/union."""
+    s = re.sub(r"\bfunc_[0-9A-Fa-f]+\b", "", sig)         # drop the function name
+    s = re.sub(r"\b[a-z_]\w*\s*(?=[,)])", "", s)          # drop param names (lowercase, before , or ))
+    for m in re.finditer(r"\b(struct|union)\s+([A-Za-z_]\w*)", s):
+        if m.group(2) not in structs:
+            return False
+    s2 = re.sub(r"\b(struct|union)\s+\w+", "", s)          # remove resolved struct refs
+    for tok in re.findall(r"\b([A-Za-z_]\w*)\b", s2):
+        if tok not in SCALARS:
+            return False
+    return True
+
+
+def def_sigs(path, structs):
+    """addr -> canonical prototype string, from every RESOLVABLE function DEFINITION in `path`."""
+    out = {}
+    if not os.path.exists(path):
+        return out
+    for m in DEF_RE.finditer(open(path).read()):
+        sig = re.sub(r"\s+", " ", m.group(1)).strip().rstrip("\\").strip()
+        if not resolvable(sig, structs):
+            continue
+        a = int(re.search(r"func_([0-9A-Fa-f]+)", sig).group(1), 16)
+        out.setdefault(a, sig)   # first definition wins (engine_core.h is authoritative)
+    return out
+
+
+def data_decls(paths):
+    out = {}
+    for p in paths:
+        if not os.path.exists(p):
+            continue
+        for m in DATA_RE.finditer(open(p).read()):
+            decl = re.sub(r"\s+", " ", m.group(1)).strip()
+            sym = re.search(r"D_[0-9A-Fa-f]+", decl).group(0)
+            out.setdefault(sym, decl)
+    return out
+
+
+def main():
+    structs = known_structs()
+    funcs = {}
+    funcs.update(def_sigs(ECORE, structs))         # engine_core.h macro defs (authoritative)
+    for a, s in def_sigs(OVC, structs).items():    # + inline 077-local defs
+        funcs.setdefault(a, s)
+    data = {k: v for k, v in data_decls([ECORE, OVC]).items() if resolvable(v, structs)}
+
+    lines = ["#ifndef BFM_ENGINE_DECLS_H", "#define BFM_ENGINE_DECLS_H",
+             "/* src/shared/engine_decls.h — GENERATED by tools/gen_engine_decls.py (Phase 16 lever 1).",
+             " * Canonical declaration of every engine function + data symbol, so harvest drafts strip",
+             " * their own decls and can never `conflicting types`. Regenerate after new matches land. */",
+             '#include "common.h"', '#include "engine_types.h"', ""]
+    lines.append(f"/* {len(funcs)} function prototypes */")
+    for a in sorted(funcs):
+        lines.append(funcs[a] + ";")
+    lines.append("")
+    lines.append(f"/* {len(data)} data symbols */")
+    for sym in sorted(data):
+        lines.append(f"extern {data[sym]};")
+    lines += ["", "#endif"]
+    open(OUT, "w").write("\n".join(lines) + "\n")
+    print(f"wrote {OUT}: {len(funcs)} func protos + {len(data)} data decls")
+
+
+if __name__ == "__main__":
+    main()

tools/p16_permute.py

@@ -67,25 +67,37 @@ def expand_m2c_field(c):
         c = c[:i] + repl + c[j:]
 
 
-def strip_externs_and_includes(c):
-    """drop #include / #define lines and m2c's leading extern/typedef decls — keep the function def."""
+def drop_preproc_and_scalar_typedefs(c):
+    """drop #include/#define and scalar/M2C typedef lines (TYPEDEFS/common.h provide them) —
+    but KEEP `extern` callee/data decls so the function compiles in its REAL signature context."""
     lines = []
     for ln in c.splitlines():
         s = ln.strip()
         if s.startswith("#"):
             continue
-        if re.match(r"^(extern|typedef)\b", s):
+        if re.match(r"^typedef\b", s):
             continue
         lines.append(ln)
     return "\n".join(lines)
 
 
 def make_base_c(draft_c):
+    """permuter base.c = scalar typedefs + the draft's canonical externs + the M2C_FIELD-expanded body.
+    Keeping the externs is essential: without them callees fall back to implicit-int and the permuter
+    matches in a DIFFERENT context than the whole-binary build (-> winners don't byte-gate)."""
     body = expand_m2c_field(draft_c)
-    body = strip_externs_and_includes(body)
+    body = drop_preproc_and_scalar_typedefs(body)
     return TYPEDEFS + body + "\n"
 
 
+def winner_to_draft(winner_c):
+    """permuter winner (typedefs + externs + body) -> gate-ready draft (externs + body). common.h
+    provides scalars/macros in the whole-binary TU. The permuter REFORMATS the typedefs (one per
+    line), so strip by line filter (drop #/typedef lines, keep externs + the function def), NOT a
+    string-replace of the TYPEDEFS block (that silently fails -> C89 typedef-redefinition error)."""
+    return drop_preproc_and_scalar_typedefs(winner_c)
+
+
 def setup(fn, draft_c):
     pd = os.path.join(REPO, ".run/permuter", fn)
     if os.path.exists(pd):
@@ -116,7 +128,8 @@ def run_permuter(pd, secs, j):
         pass
     # kill stragglers
     subprocess.run(["pkill", "-f", "decomp-permuter/permuter.py"], capture_output=True)
-    win = glob.glob(f"{pd}/output-*/source.c")
+    # ONLY output-0-* is a true byte-match; output-<N>-* are intermediate bests (score N != 0)
+    win = glob.glob(f"{pd}/output-0-*/source.c")
     return win[0] if win else None
 
 

tools/sig_unify.py

@@ -36,6 +36,11 @@ _spec.loader.exec_module(_ght)
 DATA_DECL_RE = re.compile(
     r'extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;')
 DRAFT_EXTERN_LINE_RE = re.compile(r'^[ \t]*extern\b[^;]*;[ \t]*$', re.M)
+# m2c emits callee prototypes WITHOUT `extern`: `<type> func_X(<params>);  /* extern */`.
+# Canonicalize these too, else they keep m2c's GUESSED signature and `conflicting types` in the
+# whole-binary build (the body is usually byte-correct; only the decl conflicts). Phase-16 fix.
+PROTO_DECL_RE = re.compile(
+    r'^[ \t]*[A-Za-z_][\w \t\*]*\bfunc_[0-9A-Fa-f]+\s*\([^;{]*\)\s*;[ \t]*(?:/\*[^\n]*\*/)?[ \t]*$', re.M)
 
 
 def sym_of(decl):
@@ -178,6 +183,19 @@ def main():
             return line
 
         txt = DRAFT_EXTERN_LINE_RE.sub(repl, txt)
+
+        def repl_proto(m):
+            nonlocal ext_changed
+            line = m.group(0)
+            s = sym_of(line)
+            if s and s != fn and s in canon:
+                new = canon[s]
+                if re.sub(r'\s+', ' ', new).strip() != re.sub(r'\s+', ' ', line).strip():
+                    ext_changed = True
+                return re.match(r'^[ \t]*', line).group(0) + new
+            return line
+
+        txt = PROTO_DECL_RE.sub(repl_proto, txt)
         txt, def_changed = rewrite_def(txt, fn, canon)
         open(os.path.join(REPO, args.outdir, os.path.basename(p)), 'w').write(txt)
         n += 1