BFM-decomp/docs/hindsight-study.md

Hindsight Study — the optimal order to run an AI-driven matching decompilation

Purpose. This document is a study substrate, not a status report. It distills what the Brave Fencer Musashi (BFM) decompilation taught us about how to sequence a matching decomp, separated into (a) BFM-specific facts and (b) platform/compiler-agnostic principles. It is written to be handed to a reasoning agent (a "Fable" study pass) that will read the full project history — PROJECT_CONTEXT.md, every phase-ends/PhaseEnd_*.md, docs/decision-log.md, docs/matching-cookbook.md, the docs/gcc-2.7.2-map/ — and refine this into a general roadmap for future decomps on any platform and compiler.

Written 2026-07-20, mid-Phase-29, at ~78% instr-weighted / ~66% distinct-code, from a project that ran ~29 phases almost entirely AI-driven under a human-gated phase system. The claims here are grounded in the decision-log (R31) and the phase-ends; where a lesson is uncertain or BFM-specific, it says so.

How to read this for generalization. Every section is tagged [UNIVERSAL] (transfers to any matching decomp), [COMPILER-CLASS] (transfers to same-compiler-family projects, e.g. other gcc-2.7.2 PS1 titles), or [BFM-SPECIFIC] (an artifact of this game, listed only to show the reasoning). A future roadmap should keep the UNIVERSAL spine, adapt the COMPILER-CLASS parts per toolchain, and discard the BFM-SPECIFIC details.

---

0. The one-sentence thesis

The bottleneck is never where you think it is — so the optimal order front-loads the two things that compound (structure-mapping + propagation tooling, and measurement integrity), builds the integration layer before you need it, and treats raw per-function cracking as the last, scarcest resource to spend, because everything else multiplies it.

Ideal order in one breath [UNIVERSAL]: byte-match baseline → pin the compiler by evidence → honest corpus census (with self-asserting tools) → build propagation + reconcile tooling → build the codegen-idiom map + start the permuter → then crack, leverage-first, letting each crack fan out across all its duplicates.

---

1. What our ordering got RIGHT (keep these first, in this order)

1.1 Deterministic extraction + a 100%-INCLUDE_ASM byte-match baseline before writing any C. [UNIVERSAL] SHA1 equality of the rebuilt binary is the only definition of "matched" that cannot be gamed (our G3/P9). Establish it first: a build that reassembles the original disassembly to a byte-identical binary, with the SHA check wired into the build so there is no "third state" between match and fail. Everything downstream — every claim of progress — is measured against this incorruptible oracle. Getting this first prevented the documented AI failure mode (stubs/HLE/redefined-success) that has killed other agentic RE attempts.

1.2 Pin the compiler/assembler triple by evidence, before bulk matching. [UNIVERSAL, method] / [COMPILER-CLASS, the answer] The single highest-stakes early decision. Copying a sibling project's toolchain (we were tempted by sotn's gcc-2.6.3) would have produced systematic near-miss diffs on every function — an invisible poison. We pinned gcc-2.7.2 + the ASPSX assembler version by fingerprint evidence (library stamps in the binary + idiom-revealing probe functions), not by assumption. Universal rule: identify the exact compiler, version, optimization default, and assembler by evidence from the binary itself, and expect per-module variation (mixed -O levels, mixed compilers within one image). Never inherit another project's triple without re-proving it.

These two were correctly first. Everything below is what we'd RE-ORDER.

---

2. The four things to re-order (each learned the hard way)

2.1 Characterize the corpus SHAPE before choosing a strategy — and never build strategy on an unverified measurement. [UNIVERSAL]

Our most expensive mistakes were not bad matches — they were strategic misdirections built on broken-tool readings. The worst: a "structural families template at ≈0%, so families can't be automated" doctrine that steered two entire phases of strategy. Its evidence was a single probe that silently omitted a required jump-table carve; run correctly, that same family matched 89%. The 0% and the 89% were the same number produced by a broken vs a fixed tool — opposite facts.

The re-order: the first move after the byte baseline is an honest census of the corpus:

• How much is byte-identical duplication (the free tier)?
• How many structural families (the same function recurring, differing only in per-location symbols/immediates)?
• What is the reach × size distribution (how many copies does each function have; how big is each)?
• How large is the genuinely-unique tail?

And — the meta-discipline that must accompany it — every scanner asserts its own coverage (compare what it found against an over-approximating candidate set and fail on the gap), and every measurement is checked against a second, disagreeing oracle before any strategy is built on it. (Our rules R32/R34/R35.) We adopted this at Phase 26 and immediately found seven silent-skip bugs and the collapse of the ≈0% doctrine. It belongs at the foundation, not as a late audit. A perfect correctness oracle (the byte-gate) is a null oracle for coverage — it is green at 0% decomp — so it cannot tell you what you failed to attempt. You need a second instrument that can.

Generalization: on any platform, before sequencing the work, measure the shape of the problem with tools you have verified against a disagreeing oracle. The cost of a wrong shape-measurement is phases of mis-directed effort, not hours.

2.2 Build the propagation / dedup infrastructure IMMEDIATELY after the first match — not incrementally over 15 phases. [UNIVERSAL where duplication exists; magnitude BFM-SPECIFIC]

The dominant economic fact of BFM: it is ~137 position-locked copies of one shared engine streamed as overlays, so one crack banks ~137 members for free. We knew the overlay structure early (Phase 3) but did not have the propagation tooling (dedup_propagate, family_sweep, jtbl_family_bank) mature until many phases later; we built it reactively. Front-loading it means every crack compounds from match #1. This is probably the largest leverage we left on the table through ordering alone.

Mechanics worth generalizing:

• h_exact duplicates (byte-identical, same address) → author once as a shared macro/body, propagate to all members, byte-gate each. Zero-crack.
• structural families (same skeleton, per-location relocations/immediates) → crack one exemplar, then mechanically remap the per-member symbols and byte-gate each. Near-zero-crack after the first.
• The right unit of work is the family, not the function. Rank and schedule by family leverage.

Generalization: any target with repeated/loaded code (overlays, statically-linked shared libs, templated code, copied engine across levels) has this multiplier. Detect the duplication structure and build the propagate-and-gate pipeline before bulk matching, so cracking is never the thing you repeat.

2.3 Order the matching by LEVERAGE, not by DIFFICULTY. [UNIVERSAL]

The intuitive ramp is "smallest, simplest leaf functions first." That is lowest-leverage. Fleet progress is function-weighted and reach-multiplied, so the correct schedule is highest (reach × size) first: crack the exemplar that unlocks the most members and the most instructions. We eventually built a leverage-ranked family manifest and it changed everything; doing it from the start would have front-loaded the percentage gains and surfaced the shape of the hard tail far earlier.

Generalization: build a difficulty/leverage inventory early, but sort execution by leverage, and re-sort as the frontier shrinks. "Easy first" optimizes for a feeling of progress; "leverage first" optimizes for actual progress.

2.4 Read the compiler SOURCE and build the codegen-idiom map early; run the permuter continuously from the first near-miss. [COMPILER-CLASS map / UNIVERSAL permuter]

Two compounding assets we built late:

• The codegen map (our docs/gcc-2.7.2-map/ + cookbook §31): a catalog of compiler pass → residual diff pattern → C-level lever that fixes it, built by reading the actual compiler source. Before it, cracking was per-function guesswork; after it, cracking became systematic pattern-application and every subsequent match got cheaper. A frontier model reading the compiler source broke wall-classes that trial-and-error could not. This is a "do it up front" investment: the map pays back on every crack that follows.
• The permuter track (decomp-permuter, automated by our grinder): LLM-free, CPU-bound, it takes a close draft (within a few instructions) and randomizes the C to close regalloc/scheduling residuals. It runs in parallel with everything else on a different budget (CPU, not tokens/reasoning). It should be grinding the near-miss backlog continuously from the first close draft. We ran it late; running it from the start is free background yield.

Generalization: whatever the compiler, invest early in a residual→lever map derived from its actual behavior (source if available, else systematic probes), and run a search-based closer (permuter or equivalent) continuously on near-misses. The map makes cracking systematic; the permuter harvests the tail for free.

---

3. The meta-principles (the transferable spine)

1. The bottleneck is never where you think. We optimized cracking (making a function match) while the real bottleneck was integration — banking a proven match past translation-unit / declaration / symbol conflicts (our §58/§59 reconcile classes). "Matching is solved; integration is the bottleneck" became true once cracking was systematic. Build the integration/reconcile layer before you need it, and budget for banking, not drafting. [UNIVERSAL — the specific conflict classes are COMPILER-CLASS]

2. Incorruptible oracles or nothing. The byte-gate (SHA1) is the only arbiter; intermediate metrics (closeness scores, standalone match, "looks right") are candidates, never matches. Every automated actor — AI agent or deterministic tool — reports to the byte-gate, so a wrong answer cannot land. This is what makes heavy automation safe: a weak model or a buggy tool is a throughput risk, never a correctness risk. [UNIVERSAL]

3. A correctness oracle is blind to coverage. The byte-gate is green at 0% decomp because the fallback pastes the original bytes. So it can prove what landed is correct, but never what you failed to attempt. You need a second, disagreeing oracle for coverage, and every scanner must assert it did not silently skip. Most of our worst multi-phase errors were invisible because nobody compared what a tool found to what it should have found. [UNIVERSAL]

4. Fix the instrument before trusting the measurement. A number from a broken tool and a number from a fixed tool are the same number and opposite facts. Tool-integrity is a precondition of a measurement, not a parallel concern. [UNIVERSAL]

5. Leverage compounds; effort doesn't. Prefer the move that makes the next move cheaper — the codegen map, the propagation pipeline, the reconcile tooling, the permuter — over the move that banks one more function. The whole project is a flywheel: each match should lower the cost of the average future match. [UNIVERSAL]

6. Capture the perishable reasoning while it's fresh. The terse "what changed" record survives; the why (the judgment behind a pivot, the byte-evidence behind a near-miss crack) evaporates across a context boundary. We keep a decision-log (R31) and a cookbook (R16/R30) written during the producing session. For an AI-driven project this is load-bearing: a fresh session inherits only compressed summaries. [UNIVERSAL for AI-driven]

7. Humans gate direction; automation fills between the gates. The phase system (two human gates per phase — plan approval and milestone confirmation — with autonomous execution between) is what let ~99% of the work be AI-driven without drifting. The gates are where a wrong direction gets caught; the byte-oracle is where a wrong result gets caught. Both are needed. [UNIVERSAL for AI-driven]

---

4. The tool inventory worth porting (what we built) [COMPILER-CLASS / re-implementable]

A future decomp on a similar stack could lift these directly; on a different stack, re-implement the roles:

• Corpus census & family map — cluster the frontier by byte-identity (h_exact), reloc-only skeleton (h_norm), and mnemonic skeleton (h_seq); rank families by leverage. (family_hseq.py, family_manifest.py)
• Propagation — author-once-share-many for duplicates and structural families, each member byte-gated. (dedup_propagate.py, family_sweep.py, jtbl_family_bank.py)
• Per-function gate — compile one function standalone, relocation-mask, compare to target; a fast local loop an agent iterates against. (match_one.py) + whole-binary gate (harvest_verify.py).
• The reconcile ladder — the integration layer that banks a proven match past TU-local conflicts: callee-decl casts, data-decl conformance, self-decl normalization, def-side canonical-sig reconciliation, symbol-name normalization (decompiler names → linker names), opt-level relocation. (cast_call_sites.py, reconcile_tu.py, normalize_self_decls.py, canon_sig_reconcile.py, …) — this is the part everyone under-budgets.
• The codegen map — compiler-pass → residual → C-lever catalog, from reading the compiler source.
• The permuter track — automated search-based near-miss closer, running on CPU budget in parallel. (grinder.py)
• The burn-down tracker — per-session velocity + remaining frontier + a yield-floor signal, so the ROI-gated close is data-driven, not a guess. (burndown.py)
• Self-asserting audits — every scanner compares found-vs-should-have-found and fails on the gap.

---

5. What is genuinely hard and stays hard (be honest in the roadmap)

• The novel unique tail. Functions with no duplicate, no structural sibling, and an m2c/decompiler scaffold too far from byte-exact for the permuter. These need reasoning (an LLM or a human) to produce the close seed. No deterministic normalizer we found covers all the context-dependent codegen idioms. Plan for this tail to be the last and most expensive fraction. [UNIVERSAL]
• Compiler-internal residuals (scheduling ties, register-allocation drift, boosted-load/delay-slot knife-edges): steerable only by zero-byte tricks, register pins, or the permuter — and a subset are genuine walls at a given model tier. Characterize them honestly; don't force a non-match. [COMPILER-CLASS]
• Build-integration walls (split/relocation infrastructure, jump-table carves, mixed opt-levels): these masquerade as "compiler walls" but are our tooling. Every single "wall" we deeply probed on BFM resolved to tooling, not the compiler. Suspect the instrument first. [UNIVERSAL lesson, BFM-SPECIFIC specifics]

---

6. For the future Fable study pass — how to turn this into a general roadmap

1. Read the primary sources, not just this summary: PROJECT_CONTEXT.md (the constitution + rules), every phase-ends/PhaseEnd_*.md in order (the true build history + each phase's "Rules Added"), docs/decision-log.md (the R31 pivots with byte-evidence), docs/matching-cookbook.md (the idiom/technique flywheel), docs/gcc-2.7.2-map/ (the codegen map), and docs/calibration.md (the measured close-rates).
2. Keep the UNIVERSAL spine (§0–§3, §5–§6). Re-derive the COMPILER-CLASS parts (§2.4 map, §4 tools, §5 walls) for the target's toolchain. Discard BFM-SPECIFIC facts (the overlay count, the exact idioms).
3. Produce a platform-parameterized roadmap: for a new target, the roadmap should first ask — what's the incorruptible build oracle? what's the exact toolchain (by evidence)? what's the duplication structure? what's the reach×size distribution? is the compiler source available for a codegen map? what's the search-closer for near-misses? — and then sequence per the UNIVERSAL order, adapting magnitudes.
4. Stress-test the thesis against counter-examples: platforms with little/no code duplication (where §2.2's multiplier is weak), closed-source or unknown compilers (where §2.4's map must come from probes not source), and hand-written assembly (where "matching a compiler" is the wrong frame entirely). The roadmap should say where the BFM playbook doesn't apply and what replaces it.

The deliverable is not "do what BFM did." It is: measure the shape, verify your instruments, build the multipliers before the grind, and spend reasoning last — parameterized by the target's oracle, toolchain, duplication structure, and available compiler knowledge.

---

---

7. The offline-automatic endgame — mine the permuter's FAILURES, not just its wins [UNIVERSAL]

Post-project goal (Drew, 2026-07-20): drive the offline tooling toward near-full automation, calling the LLM only to get over a genuine hump. The lever we are currently not pulling: systematically learning from what the search-closer (permuter/ILS) fails on.

The gap, stated precisely. Our flywheel (R16) evolves the permuter recipes/weights reactively from successes — when a human or LLM cracks a function, we distill the idiom back into the cookbook + the permuter weights. But the permuter's failures — every near-miss it plateaued on — are a richer, unused signal. Today the grinder re-logs a failure with only closeness + a coarse where_stuck label ("N mismatch") and, for the "won-standalone-but-gate-rejected" case, a blacklist entry. Nobody asks why the ILS plateaued or what would have bridged it. So the offline tool's coverage grows only as fast as we hand-feed it idioms.

The loop that closes the gap — the "permuter-failure autopsy":

1. Enrich the failure telemetry (the cheap prerequisite — start capturing this NOW so the corpus exists). On every plateau, log the structured residual, not a scalar: the exact mismatching instructions (post-mask), the register/schedule delta, the seed C used, the plateau closeness over the ILS run, and the mutation classes the permuter did try. Without this, no diagnosis is possible later; with it, the autopsy is a pure offline read.
2. LLM triage over the failure corpus (batched, not per-function) → a 3-way classification:
   • missing-transform — the byte-match is reachable by local mutation, but the permuter's mutation set / weights don't include the needed move (e.g., a specific commutative-operand swap, a delay-slot fill choice, a spill-slot sizing). → the LLM proposes a new permuter recipe/weight; add it to permuter_weights.py / permuter_ils.py; re-run the backlog. This is a PERMANENT offline win — that whole class now cracks with no LLM. This is the highest-value bucket and the whole point.
   • seed-structural — the C structure is wrong (wrong loop shape, wrong aggregate layout, a missing idiom the permuter can't introduce because it only does local edits). → the LLM does a one-shot seed rewrite (the "hump"), then hands it back to the permuter to finish. LLM used once, minimally.
   • genuine-wall — compiler-internal, unsteerable at this model tier (intrinsic scheduler knife-edge, global regalloc shift). → blacklist WITH A REASON + a tier tag (R35: the verdict expires when the gate or the model tier changes), and revisit only on a tier upgrade. Never silently give up; record why.
3. Track the offline-automatic fraction over time (extend burndown.py): what % of the frontier the permuter+wrapper close with zero LLM, and how that % moves as each autopsy adds transforms. The success metric of this whole endeavor is that number rising toward 1, with the LLM's role shrinking to the seed-structural humps + the wall-tier escalations.

Why this is the right shape. It makes the flywheel failure-driven and data-driven instead of success-driven and anecdotal. Each autopsy either (a) permanently widens the offline tool's reach (missing- transform → a new weight), (b) spends the LLM once on a real hump (seed-structural), or (c) honestly files a wall with an expiry condition. Over enough passes the offline pipeline (m2c/decompiler → corpus-NN seed → permuter+evolved-wrapper → reconcile-ladder bank → propagate) handles the bulk unattended, and the LLM is the scarce reserve it was always meant to be — pulled only for the genuine humps. This is the concrete mechanism behind the "finish a sibling game (e.g. Vagrant Story) mostly offline" ambition: the more failure-autopsy passes we run, the larger the offline fraction, on any same-compiler target.

Generalization [UNIVERSAL]. For any search-based match-closer (permuter or otherwise): treat its failure log as a first-class training corpus. Capture structured residuals, batch-diagnose with the strongest available model, and route each failure to extend the tool (permanent), spend one LLM seed (hump), or file a walled verdict with an expiry (honest). Measure the LLM-free fraction and make raising it the objective. The tool should get stronger every time it loses.

---

This document is evolvable (docs/ layer). It is the R31/R30 substrate for the project's "public how-to-AI-decomp" endgame deliverable. Corrections and extensions welcome; keep the UNIVERSAL/COMPILER-CLASS/BFM-SPECIFIC tagging so the general roadmap stays separable from the case study.