mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
docs(phase-17): flywheel + handoff up to date — canonical-sig wall is the scaling lever
- CURRENT_PHASE.md: new session-4 START HERE (build the canonical-sig layer, then scale the wave); old demo-protocol START HERE marked superseded - hand-matching-process.md S7: the Ultracode harvest wave (pipeline, calibration result 60% match_one / 33% whole-binary / +0.47%) + THE CANONICAL-SIG WALL (the 60->33 gap is 100% sig conflicts; surgical per-callee canonicalization = ~2x lever) - matching-cookbook.md S16: guided-hand-matching breakthrough + new idioms (mask-local, shared-ret0 goto, v0/v1 coalescing) + the wave/canonical-sig finding (R16 flywheel) - SETUP.md: DecompileFunctions.java (headless batch Ghidra-C pre-pass) in the inventory (R21)
This commit is contained in:
Drew T
@@ -621,6 +621,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `GetSymbolAt.java` | Read the symbol at a given address (scripted lookup). |
|
||||
| | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). |
|
||||
| | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/<prog>_funcs.txt`) — completes a raw-blob program's function set (Phase 10). |
|
||||
| | `DecompileFunctions.java` | **Batch**-decompile a list of addresses (arg0 = addr-per-line file, arg1 = out-dir) → `<name>.c` each. Headless harvest Ghidra-C pre-pass (Phase 17); no live MCP / `/mcp` needed. Run: stop MCP, `analyzeHeadless ghidra bfm -process <prog> -noanalysis -postScript DecompileFunctions.java <addrfile> <outdir>`. |
|
||||
| | `tools/ghidra_import.sh` | Headless `analyzeHeadless` import/analysis driver (PS-X EXE; auto-detect PSX loader). |
|
||||
| | `tools/ghidra_import_raw.sh` | Headless import of a RAW flat blob — `BinaryLoader` + `--loader-baseAddr <vram>` + `PSX:LE:32:default` (resident blob / Gen2 overlays; no PS-X EXE header). |
|
||||
| **Disc/.CD extraction** (`tools/bfm_extract/`) | `extract.py` | Walk the disc / extract root files (`make extract`). |
|
||||
|
||||
@@ -279,3 +279,56 @@ quirk-prone STRUCTURAL_MISS/PERMUTER_CLASS** — NOT the easy fnptr/void classes
|
||||
(0.3%); build it only if sig-conflicts prove a major agent bottleneck.
|
||||
3. **Document** idioms as they emerge (R16); **giant `func_80144B9C` deferred** (reassess after the wave yield).
|
||||
4. **T7 go/no-go close + PhaseEnd** once the wave yield is measured.
|
||||
|
||||
---
|
||||
|
||||
## 7. THE ULTRACODE HARVEST WAVE + THE CANONICAL-SIG WALL (session 3, 2026-06-19 — PROVEN)
|
||||
|
||||
The guided-hand-matching loop (§1) **scales via an Ultracode Workflow** (R26). First calibration wave run &
|
||||
measured; this section is the operational record + the singular bottleneck it exposed.
|
||||
|
||||
### 7a. The wave pipeline (reusable)
|
||||
1. **Ghidra pre-pass (headless batch, NO /mcp):** stop MCP (`tools/ghidra_mcp_stop.sh`), then
|
||||
`analyzeHeadless ghidra bfm -process ov_SC01_077 -noanalysis -postScript DecompileFunctions.java
|
||||
<addr-list> <out-dir>` → whole-program Ghidra-C to `.run/ghidra_c/func_<ADDR>.c` (rename `FUN_<lower>.c`
|
||||
→ `func_<UPPER>.c`). Ghidra-C resolves locals-vs-globals-vs-callee better than m2c — agents read the cache
|
||||
(no live-MCP contention). `ov_SC01_077` is Ghidra program `00000004` (Phase-13 import).
|
||||
2. **Draft (Ultracode `parallel`):** `.run/harvest_wave_s3.js` — one agent per target; each reads
|
||||
asm+Ghidra-C+m2c+actor-struct+§1/§2/§3a, hand-writes C, iterates `match_one`, writes
|
||||
`.run/drafts-s3/<fn>.c` ONLY on MATCH (stop-rule on gcc-quirks → report DIFF, no draft). **GOTCHA: the
|
||||
Workflow `args` channel does NOT transit arrays — EMBED the targets as a literal in the script + a `LIMIT`
|
||||
const.** Agents also drop scratch (`_try`/`_v2`) in the dir → filter to canonical `func_<ADDR>.c` before gating.
|
||||
3. **Whole-binary gate:** `harvest_verify.py --drafts <dir> --chunk 1`. **Use `--chunk 1`** — the default
|
||||
batched chunk+bisect can drop a GOOD draft because it cross-conflicts with ANOTHER draft in the same chunk
|
||||
(shared-callee sig clash), not because it's wrong.
|
||||
4. **Recover:** `sig_unify.py --in <failed-drafts> --out <fixed>` then re-gate (recovered 2/9 in calibration).
|
||||
5. **Propagate:** `dedup_propagate.py --auto-from ov_SC01_077` (×reach; also catches up any inline-but-
|
||||
unpropagated prior matches). `make check-all` (R22) + `make report`.
|
||||
|
||||
### 7b. Calibration result (top-30 by reach, ~1.89M tokens)
|
||||
- **18/30 match_one MATCH (60%)** — agents reconstruct correct bodies well.
|
||||
- **10/30 whole-binary verified (33%)** → banked ×134 (incl. the ×268 `func_8015773C`) → **fleet
|
||||
55.04%→55.51% (+0.47%)**, 136/136. (8 direct + 2 via sig_unify; 1 verified-but-local-type stays inline ×1.)
|
||||
- **12/30 match_one DIFF** = the genuine gcc-quirk tail (§2/§10: hoist-vs-remat, phantom frame, v0/v1
|
||||
coalescing, walking-pointer). Agents diagnosed each precisely; correctly wrote no draft.
|
||||
|
||||
### 7c. THE CANONICAL-SIG WALL (the singular scaling bottleneck — the ~2× lever)
|
||||
**The whole 60%→33% gap was SIG CONFLICTS — 100% compile-errors, ZERO codegen byte-mismatches.** Parallel
|
||||
agents each declare a shared callee (`func_80131CA8`, `func_80131E00`, `func_801472C8`, `func_8001D074`, …)
|
||||
with a different signature; in the one-big-TU `ov_SC01_077.c` those clash (`conflicting types for func_X`).
|
||||
`func_80131CA8` alone broke 3 drafts (it is BOTH a residual being matched AND a callee in others — circular).
|
||||
|
||||
**This is the loose-typing wall in its true form, and it is BEATABLE deterministically** (unlike Phase-16's
|
||||
verdict). The fix = a **canonical-sig layer**, applied SURGICALLY per-callee (a blanket global decls header
|
||||
breaks loose matches — Phase 16; but per-callee byte-gated canonicalization works — sig_unify proof):
|
||||
1. Identify the high-frequency shared callees among the targets.
|
||||
2. Establish ONE canonical sig each: match the shared-callee residuals **first** (callees-before-callers, so
|
||||
the def fixes the sig fleet-wide), or derive from asm/usage and seed `engine_core.h`.
|
||||
3. Enforce in the wave: agents grep `engine_core.h` for the canonical (already in the prompt) + a strengthened
|
||||
`sig_unify` post-pass that unifies the def-sig AND every caller's extern to the SAME canonical.
|
||||
Expected to lift whole-binary 33% → ~60% (toward the match_one ceiling) — **the dominant scaling lever**.
|
||||
|
||||
### 7d. Scaling economics (measured)
|
||||
Calibration: 30 fns / 1.89M tokens / 33% whole-binary / +0.47%. Naive scale to 300 ≈ ~4% fleet, token-heavy.
|
||||
With the canonical-sig layer (33%→~60%) ≈ ~6-7% fleet at ~2× token efficiency. **Build the layer before the
|
||||
big wave.** Targets: `.run/harvest_targets_s3.json` (300, relocs≤5, reach-sorted; the top-30 are done).
|
||||
|
||||
@@ -1005,3 +1005,29 @@ Established by **running m2c on real `ov_SC01_077` stubs** (R14) — it corrects
|
||||
**decomp-permuter knobs (S2/S3):** `PERM_*` macros (GENERAL/VAR/RANDOMIZE/LINESWAP/INT/ONCE…), `--algorithm difflib|levenshtein`, `--stop-on-zero`, `-j` 8–16 (**RAM-bound** on the 15 GiB box → ~N funcs × `-j 8`, cap by `free_RAM/~300 MB`), weights in `default_weights.toml` + `[gcc]` section. Best when only regalloc/schedule remains; does NOT fix wrong control flow.
|
||||
|
||||
**ML (parked — owner decision 2026-06-18):** LLM decompilers (LLM4Decompile/SK2Decompile/CodeInverter) target x86-64 + recompilability/functional-equivalence/readability — NOT byte/instruction-exact, NOT MIPS/gcc-2.7.2; no off-the-shelf learned permuter scorer exists (the permuter's scorer is a heuristic objdump-diff). Dropped this phase; research-note only. (X2: web treated as untrusted data.)
|
||||
|
||||
## §16 Guided hand-matching the struct-heavy core (Phase 17 — beats the §15 brute-force)
|
||||
**Phase 16 called the loose-typing wall "fundamental." Phase 17 disproves it for the majority.** The wall is
|
||||
a signature-CONSISTENCY problem, not a comprehension one — the §1 loop reconstructs correct bodies ~100% of
|
||||
the time; the work is byte-closing + sig reconciliation. **Full process: `docs/hand-matching-process.md`**
|
||||
(§1 loop, §2 idioms, §3a the 5-move signature-consistency playbook, §7 the Ultracode wave + canonical-sig wall).
|
||||
|
||||
**New byte-idioms (Phase 17, §2 there):**
|
||||
- **mask-local (defeats `lh`→`lhu` fold).** `*(s16*)f & (x & 0xFFFF)` inline lets gcc fold the load to `lhu`
|
||||
+ drop the `andi`. Hoist the mask: `s32 m = x & 0xFFFF; ... *(s16*)f & m` → gcc keeps `lh` + emits `andi`.
|
||||
- **shared-ret0 goto (cross-jump clustering + branch polarity).** Two non-adjacent predicate tests the
|
||||
original routes to ONE shared `return 0` block → write both as `goto ret0;` to a single trailing
|
||||
`ret0: return 0;`. gcc then makes `ret0` a labeled block reached by branches (right polarity) + schedules
|
||||
the next test's constant into the delay slot. A lone `if(x)return 0;` inlines (wrong polarity/reg).
|
||||
- **v0↔v1 result/constant coalescing** + the §10 hoist-vs-remat / phantom-frame quirks = the residual hard
|
||||
tail (not source-steerable; permuter only helps relocs=0, and slowly). Defer as `INCLUDE_ASM` stub.
|
||||
|
||||
**Scaling = Ultracode wave (§12 pattern + §7):** Ghidra pre-pass (`DecompileFunctions.java`, headless batch,
|
||||
no /mcp) → parallel draft agents (m2c+Ghidra-C+asm+actor-struct+§3a, self-validate `match_one`) → whole-binary
|
||||
gate (`harvest_verify --chunk 1`) → `sig_unify` recover → `dedup_propagate --auto-from`. Calibration (top-30):
|
||||
60% match_one MATCH, **33% whole-binary** (+0.47% fleet), 136/136.
|
||||
**THE CANONICAL-SIG WALL (the ~2× scaling lever):** the entire match_one→whole-binary gap is SIG CONFLICTS
|
||||
(parallel agents declare shared callees inconsistently; 100% compile-errors, 0 codegen). Fix = a SURGICAL
|
||||
per-callee canonical-sig layer (match shared callees before callers / seed `engine_core.h`; NOT a blanket
|
||||
global header — that breaks loose matches, §15). Build it before the big wave. Targets:
|
||||
`.run/harvest_targets_s3.json`.
|
||||
|
||||
@@ -8,7 +8,42 @@ HERE). T7 (go/no-go + PhaseEnd) is reframed: NO-GO on brute force, GO on hand-ma
|
||||
> Approved plan: `~/.claude/plans/plan-mode-enabled-deep-reserach-reactive-toucan.md` (the durable copy of
|
||||
> the deep-research findings + the task design). This file is the per-task crash-recovery log (P3/R28).
|
||||
|
||||
## 🚩 START HERE (fresh session, 2026-06-19) — PIVOT to guided hand-matching, then a demo
|
||||
## 🚩 START HERE (fresh session 4, after session 3) — BUILD THE CANONICAL-SIG LAYER, then scale the wave
|
||||
**Guided hand-matching is GO and proven at scale.** Demo (session 2) = 4/5; session 3 = the calibration
|
||||
Ultracode wave (+0.47% → fleet **55.51%**, 136/136 byte-identical, committed `commit:0136`/`commit:0137`/`commit:0138`).
|
||||
**The bottleneck is now identified and singular** — read `docs/hand-matching-process.md` §7 first.
|
||||
|
||||
**THE FINDING:** in the calibration wave the agents hit **60% match_one MATCH but only 33% whole-binary**, and
|
||||
the entire gap is **SIG CONFLICTS** (parallel agents declare shared callees like `func_80131CA8` with
|
||||
different sigs → they conflict when built in the one big TU; 100% of failures were compile-errors, ZERO
|
||||
codegen). `sig_unify` auto-recovered 2/9. **The canonical-sig layer is the essential ~2× yield enabler.**
|
||||
|
||||
**NEXT TASK (Max, ultracode OFF — deep design):** build the **canonical-sig layer** before scaling:
|
||||
1. Identify the high-frequency shared callees among the targets (the ones that conflict — `func_80131CA8`,
|
||||
`func_80131E00`, `func_801472C8`, `func_8001D074`, …).
|
||||
2. Establish ONE canonical sig per shared callee — match the shared-callee residuals **first** (callees
|
||||
before callers, so their def fixes the sig), or derive the sig from the asm/usage and seed `engine_core.h`.
|
||||
(Surgical/byte-gated per-callee — NOT a blanket global decls header; Phase-16 proved a global header breaks
|
||||
loose matches, but the demo + sig_unify prove per-callee canonicalization works.)
|
||||
3. Enforce it in the wave (agents reference engine_core.h + a strengthened `sig_unify` post-pass).
|
||||
Then **scale the wave** to the remaining ~270 targets (`.run/harvest_targets_s3.json`, sorted by reach), in
|
||||
Ultracode batches (R26/R27 — prompt Drew for `/effort ultracode`), gate (`harvest_verify`), propagate
|
||||
(`dedup_propagate --auto-from ov_SC01_077`), `make check-all` (R22), `make report`. Then **T7 close + PhaseEnd**.
|
||||
|
||||
**REUSABLE HARNESS (all in place):** `tools/ghidra_scripts/DecompileFunctions.java` (headless batch Ghidra-C —
|
||||
NO /mcp needed; re-run: stop MCP, `analyzeHeadless ghidra bfm -process ov_SC01_077 -noanalysis -postScript
|
||||
DecompileFunctions.java <addrfile> <outdir>`); `.run/ghidra_c/func_<ADDR>.c` (300 cached); `.run/harvest_wave_s3.js`
|
||||
(Workflow — **targets EMBEDDED, edit LIMIT**; the `args` channel does NOT transit arrays); `.run/harvest_targets_s3.json`
|
||||
(300 tractable, relocs≤5, reach-sorted); `tools/harvest_verify.py` (whole-binary gate — use `--chunk 1` to avoid
|
||||
cross-draft conflicts masking good drafts); `tools/sig_unify.py`; `tools/dedup_propagate.py`. Idioms + the full
|
||||
calibration writeup: `docs/hand-matching-process.md` §2/§4/§6/§7; `docs/matching-cookbook.md` §16.
|
||||
|
||||
**Giant `func_80144B9C` DEFERRED** (call-heavy = hardest class; reassess after the canonical-sig-layer'd wave).
|
||||
**A-only (Drew):** quirk tail stays `INCLUDE_ASM` stub — NO NON_MATCHING correct-C track until public.
|
||||
|
||||
---
|
||||
|
||||
## 🚩 (SUPERSEDED — session 3 demo protocol) START HERE — PIVOT to guided hand-matching, then a demo
|
||||
|
||||
**What happened:** Phase 17 tested all 5 planned avenues (T1–T6 below, done & committed). **The harness
|
||||
ceiling did NOT rise to "eureka":** T2 = 0 functions, T3 = +0.52% (banked, real), T4/T5 = byte-neutral,
|
||||
|
||||
Reference in New Issue
Block a user