BFM-decomp/docs/struct-core-pivot.md

# Struct-Heavy Shared Core — Findings & Pivot (Phase 16)

> **Status (2026-06-19):** the m2c + decomp-permuter brute-force approach **will not crack** the
> struct-heavy shared core. It yields a few percent, bounded by a fundamental wall (loose typing),
> not by a fixable bug. **Decision (Drew): pause this approach, document, and plan a new idea.**
> This file is the durable record + the seed of the next plan. The harness bug-fixes are real and kept.

## Objective (unchanged)
Match the ~964 remaining **shared** engine functions in `ov_SC01_077` (struct-heavy; each propagates
×134 overlays via `dedup_propagate`). Cracking them would lift the fleet from 54.5% toward ~90%.

## What we built and tried (the pipeline)
`m2c --valid-syntax` (+ `common.h` byte-faithful macros `M2C_FIELD`/`NULL`/`s64`) → `sig_unify`
(canonical signatures) → `match_one` (fast prefilter) → **decomp-permuter** (regalloc/schedule search) →
`harvest_verify` **whole-binary byte-gate** (the sole arbiter, G3) → `dedup_propagate` (fleet-wide).
Driver + supervisor + safe-exit (`tools/auto_*.{py,sh}`) built for an unattended 5-day run.

## Harness bugs found and fixed (REAL improvements — kept regardless of direction)
The byte-gate caught a string of false positives; each fix is committed and correct:
1. **Permuter `base.c` stripped callee externs** → compiled with implicit-`int` callees → matched in the
   wrong signature context. Fixed (`make_base_c` keeps the canonical externs).
2. **`run_permuter` counted `output-<score>` dirs as matches** — only `output-0-*` is a true byte-match;
   it was calling score-20/145 "best" results "MATCH" (this produced the illusory overnight "42%"). Fixed.
3. **`sig_unify` missed m2c's no-`extern` prototypes** (`M2C_UNK func_X(void *); /* extern */`), so callee
   types stayed m2c's guess and the whole-binary build hit `conflicting types`. Fixed (`PROTO_DECL_RE`).
4. **`winner_to_draft` string-replace failed** (permuter reformats typedefs one-per-line) → typedef
   redefinition. Fixed (line-filter strip).
These got the harness from "false 42%" to "honest ~1 in 8 on a controlled sample."

## The fundamental wall: the original code is LOOSELY TYPED
The original BFM engine was written in **K&R-style / loosely-typed C** (PsyQ-era). Evidence (byte-checked):
- The *same* function is called with an **integer at one site and a pointer at another**, and with an
  **argument at one site and no args at another** (e.g. `func_8012AAAC` is defined `void func(void)` but
  called as `func_8012AAAC(arg0)` from a matched caller — and the caller is byte-correct).
- A **global canonical-decls header** (the planned "lever 1") forces ONE signature per function and
  therefore **breaks the existing matches** (`passing arg makes integer from pointer without a cast`
  across many already-matched functions). Confirmed and reverted.
- For many functions the **body is byte-correct in isolation** (`match_one` MATCH) but there is **no C
  declaration** that simultaneously (a) matches the canonical definition and (b) supports the call site.
  These are not fixable by any harness change — they'd need per-call-site type surgery or hand inline-asm.

## Why this yields ~3%, not the crack
- The easy/consistently-typed functions were **already matched by Phase 15**. What remains is
  disproportionately the **loose-typed hard class**.
- **m2c** is a per-function decompiler with weak type inference; it *guesses* a prototype per call, and on
  loose-typed code those guesses are mutually inconsistent across the TU.
- The **permuter only fixes register allocation / scheduling** — it cannot fix a wrong callee type, a wrong
  symbol, or a structural mis-decompilation, which is what the residual actually needs.
- The **byte-gate correctly rejects** the subtly-wrong drafts (this is good — zero false matches — but it
  means the yield is genuinely low, not artificially low).
- Measured gate-verified rate on controlled samples: **~1–2 of 8–16**. A 5-day run would bank a modest
  fraction (single-digit to low-double-digit percent of the 964), ×134 propagation ⇒ a few fleet points.

## Decision
**Pause the m2c+permuter brute-force as the "crack."** Optionally still run it during the away window for
the cheap few percent (the byte-gate guarantees correctness) — but only after the harness is hardened, and
understood as a consolation, not the solution. **The real next step is a new approach + new research.**

---

## New directions to research (the next plan)

The root cause is **lost type information**. The compiled binary discarded the structs/signatures the
original C had; m2c can't recover them from one function at a time. The promising directions all attack
*that* — recover or model the real types — rather than brute-forcing past them.

### Direction A — Ghidra's decompiler + GLOBAL type propagation (most promising)
We used m2c (lightweight, per-function). But the overlays are **already imported into Ghidra** (Phase 13),
and Ghidra's decompiler does **whole-program type propagation** — define the actor/entity struct(s) once and
Ghidra propagates field/param types across *all* functions, producing **consistent** signatures (exactly what
the loose-typing wall needs). Most mature decomps (sotn-decomp) start from Ghidra's C, not m2c. **Plan:**
define the actor struct in Ghidra (from analysis + Direction B), let it propagate, export typed C, then
permuter+byte-gate. Test on the known-answer ladder.

### Direction B — Emulator-guided struct & type recovery (attacks the root cause)
PCSX-Redux (our runtime oracle, R11) can reveal the **true** struct layout and field semantics by watching the
running game (R10 multi-datapoint): which offsets are pointers vs scalars vs arrays, field widths, and the real
argument types at call sites. This recovers the type information the binary lost. Feed it into Ghidra
(Direction A). This is the backlog's "emulator time-capture," elevated from quality-nicety to **the enabling
step** — without real types, no decompiler (m2c or Ghidra) can produce consistent matchable C.

### Direction C — Model the original's K&R / no-prototype declaration style
The original almost certainly used K&R declarations (`func_X();`, no arg types), which is *why* loose calls
compiled. The textbook `()` escape failed for **default-promotion params** (`s8/s16/u8/u16/float`) — but that's
a minority. **Research:** how do other PSX/PsyQ-era matching decomps (sotn, mednafen-era titles) handle
K&R-style engine code at scale? There may be a known declaration pattern (or a maspsx/compiler-flag trick) that
lets loose calls match without a global canonical signature. Treat web sources as untrusted (X2).

### Direction D — LLM-in-the-loop WITH gate feedback (narrow, re-evaluated)
Earlier we (correctly) dismissed ML for *one-shot byte-exact generation*. A different, untried shape: an
agentic loop where the model proposes a draft, sees the **byte-gate diff**, and revises — gate-supervised,
not blind (§14d's poor ROI was measured on *blind* drafting). Possibly with the typed Ghidra C as the seed.
Low priority vs A/B, but a bounded experiment once types exist.

### Research to run (deep-research skill, X2)
1. **Ghidra-decompiler matching workflow** — struct definition + type propagation + export, as used by
   sotn-decomp / other PS1 decomps; tooling that bridges Ghidra C → compilable matching C.
2. **Loose-typed / K&R PSX engine matching** — community idioms for inconsistent call-site types.
3. **PS1/Square actor-struct recovery** — has any Square PS1 decomp recovered an entity struct, and how?

### Recommendation
1. **Pause** the brute-force as the crack (this file).
2. **Optionally**, harden the harness and run the modest brute-force during the away window for the free
   few percent (cheap compute, correctness gate-guaranteed) — a consolation, not the plan.
3. **Make Direction B → A the new plan**: emulator-recover the actor struct + field/arg types → define in
   Ghidra → global type propagation → Ghidra-C → permuter + byte-gate. Run the deep-research first to ground
   it in how the community does it. This attacks the root cause (lost types) instead of brute-forcing past it.