diff --git a/docs/SETUP.md b/docs/SETUP.md index 41ea9e601..57a56d3d0 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -621,6 +621,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `GetSymbolAt.java` | Read the symbol at a given address (scripted lookup). | | | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). | | | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/_funcs.txt`) — completes a raw-blob program's function set (Phase 10). | +| | `DecompileFunctions.java` | **Batch**-decompile a list of addresses (arg0 = addr-per-line file, arg1 = out-dir) → `.c` each. Headless harvest Ghidra-C pre-pass (Phase 17); no live MCP / `/mcp` needed. Run: stop MCP, `analyzeHeadless ghidra bfm -process -noanalysis -postScript DecompileFunctions.java `. | | | `tools/ghidra_import.sh` | Headless `analyzeHeadless` import/analysis driver (PS-X EXE; auto-detect PSX loader). | | | `tools/ghidra_import_raw.sh` | Headless import of a RAW flat blob — `BinaryLoader` + `--loader-baseAddr ` + `PSX:LE:32:default` (resident blob / Gen2 overlays; no PS-X EXE header). | | **Disc/.CD extraction** (`tools/bfm_extract/`) | `extract.py` | Walk the disc / extract root files (`make extract`). | diff --git a/docs/hand-matching-process.md b/docs/hand-matching-process.md index 7d32527dc..b955950e0 100644 --- a/docs/hand-matching-process.md +++ b/docs/hand-matching-process.md @@ -279,3 +279,56 @@ quirk-prone STRUCTURAL_MISS/PERMUTER_CLASS** — NOT the easy fnptr/void classes (0.3%); build it only if sig-conflicts prove a major agent bottleneck. 3. **Document** idioms as they emerge (R16); **giant `func_80144B9C` deferred** (reassess after the wave yield). 4. **T7 go/no-go close + PhaseEnd** once the wave yield is measured. + +--- + +## 7. THE ULTRACODE HARVEST WAVE + THE CANONICAL-SIG WALL (session 3, 2026-06-19 — PROVEN) + +The guided-hand-matching loop (§1) **scales via an Ultracode Workflow** (R26). First calibration wave run & +measured; this section is the operational record + the singular bottleneck it exposed. + +### 7a. The wave pipeline (reusable) +1. **Ghidra pre-pass (headless batch, NO /mcp):** stop MCP (`tools/ghidra_mcp_stop.sh`), then + `analyzeHeadless ghidra bfm -process ov_SC01_077 -noanalysis -postScript DecompileFunctions.java + ` → whole-program Ghidra-C to `.run/ghidra_c/func_.c` (rename `FUN_.c` + → `func_.c`). Ghidra-C resolves locals-vs-globals-vs-callee better than m2c — agents read the cache + (no live-MCP contention). `ov_SC01_077` is Ghidra program `00000004` (Phase-13 import). +2. **Draft (Ultracode `parallel`):** `.run/harvest_wave_s3.js` — one agent per target; each reads + asm+Ghidra-C+m2c+actor-struct+§1/§2/§3a, hand-writes C, iterates `match_one`, writes + `.run/drafts-s3/.c` ONLY on MATCH (stop-rule on gcc-quirks → report DIFF, no draft). **GOTCHA: the + Workflow `args` channel does NOT transit arrays — EMBED the targets as a literal in the script + a `LIMIT` + const.** Agents also drop scratch (`_try`/`_v2`) in the dir → filter to canonical `func_.c` before gating. +3. **Whole-binary gate:** `harvest_verify.py --drafts --chunk 1`. **Use `--chunk 1`** — the default + batched chunk+bisect can drop a GOOD draft because it cross-conflicts with ANOTHER draft in the same chunk + (shared-callee sig clash), not because it's wrong. +4. **Recover:** `sig_unify.py --in --out ` then re-gate (recovered 2/9 in calibration). +5. **Propagate:** `dedup_propagate.py --auto-from ov_SC01_077` (×reach; also catches up any inline-but- + unpropagated prior matches). `make check-all` (R22) + `make report`. + +### 7b. Calibration result (top-30 by reach, ~1.89M tokens) +- **18/30 match_one MATCH (60%)** — agents reconstruct correct bodies well. +- **10/30 whole-binary verified (33%)** → banked ×134 (incl. the ×268 `func_8015773C`) → **fleet + 55.04%→55.51% (+0.47%)**, 136/136. (8 direct + 2 via sig_unify; 1 verified-but-local-type stays inline ×1.) +- **12/30 match_one DIFF** = the genuine gcc-quirk tail (§2/§10: hoist-vs-remat, phantom frame, v0/v1 + coalescing, walking-pointer). Agents diagnosed each precisely; correctly wrote no draft. + +### 7c. THE CANONICAL-SIG WALL (the singular scaling bottleneck — the ~2× lever) +**The whole 60%→33% gap was SIG CONFLICTS — 100% compile-errors, ZERO codegen byte-mismatches.** Parallel +agents each declare a shared callee (`func_80131CA8`, `func_80131E00`, `func_801472C8`, `func_8001D074`, …) +with a different signature; in the one-big-TU `ov_SC01_077.c` those clash (`conflicting types for func_X`). +`func_80131CA8` alone broke 3 drafts (it is BOTH a residual being matched AND a callee in others — circular). + +**This is the loose-typing wall in its true form, and it is BEATABLE deterministically** (unlike Phase-16's +verdict). The fix = a **canonical-sig layer**, applied SURGICALLY per-callee (a blanket global decls header +breaks loose matches — Phase 16; but per-callee byte-gated canonicalization works — sig_unify proof): +1. Identify the high-frequency shared callees among the targets. +2. Establish ONE canonical sig each: match the shared-callee residuals **first** (callees-before-callers, so + the def fixes the sig fleet-wide), or derive from asm/usage and seed `engine_core.h`. +3. Enforce in the wave: agents grep `engine_core.h` for the canonical (already in the prompt) + a strengthened + `sig_unify` post-pass that unifies the def-sig AND every caller's extern to the SAME canonical. +Expected to lift whole-binary 33% → ~60% (toward the match_one ceiling) — **the dominant scaling lever**. + +### 7d. Scaling economics (measured) +Calibration: 30 fns / 1.89M tokens / 33% whole-binary / +0.47%. Naive scale to 300 ≈ ~4% fleet, token-heavy. +With the canonical-sig layer (33%→~60%) ≈ ~6-7% fleet at ~2× token efficiency. **Build the layer before the +big wave.** Targets: `.run/harvest_targets_s3.json` (300, relocs≤5, reach-sorted; the top-30 are done). diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 2194cbced..3cde1d7da 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -1005,3 +1005,29 @@ Established by **running m2c on real `ov_SC01_077` stubs** (R14) — it corrects **decomp-permuter knobs (S2/S3):** `PERM_*` macros (GENERAL/VAR/RANDOMIZE/LINESWAP/INT/ONCE…), `--algorithm difflib|levenshtein`, `--stop-on-zero`, `-j` 8–16 (**RAM-bound** on the 15 GiB box → ~N funcs × `-j 8`, cap by `free_RAM/~300 MB`), weights in `default_weights.toml` + `[gcc]` section. Best when only regalloc/schedule remains; does NOT fix wrong control flow. **ML (parked — owner decision 2026-06-18):** LLM decompilers (LLM4Decompile/SK2Decompile/CodeInverter) target x86-64 + recompilability/functional-equivalence/readability — NOT byte/instruction-exact, NOT MIPS/gcc-2.7.2; no off-the-shelf learned permuter scorer exists (the permuter's scorer is a heuristic objdump-diff). Dropped this phase; research-note only. (X2: web treated as untrusted data.) + +## §16 Guided hand-matching the struct-heavy core (Phase 17 — beats the §15 brute-force) +**Phase 16 called the loose-typing wall "fundamental." Phase 17 disproves it for the majority.** The wall is +a signature-CONSISTENCY problem, not a comprehension one — the §1 loop reconstructs correct bodies ~100% of +the time; the work is byte-closing + sig reconciliation. **Full process: `docs/hand-matching-process.md`** +(§1 loop, §2 idioms, §3a the 5-move signature-consistency playbook, §7 the Ultracode wave + canonical-sig wall). + +**New byte-idioms (Phase 17, §2 there):** +- **mask-local (defeats `lh`→`lhu` fold).** `*(s16*)f & (x & 0xFFFF)` inline lets gcc fold the load to `lhu` + + drop the `andi`. Hoist the mask: `s32 m = x & 0xFFFF; ... *(s16*)f & m` → gcc keeps `lh` + emits `andi`. +- **shared-ret0 goto (cross-jump clustering + branch polarity).** Two non-adjacent predicate tests the + original routes to ONE shared `return 0` block → write both as `goto ret0;` to a single trailing + `ret0: return 0;`. gcc then makes `ret0` a labeled block reached by branches (right polarity) + schedules + the next test's constant into the delay slot. A lone `if(x)return 0;` inlines (wrong polarity/reg). +- **v0↔v1 result/constant coalescing** + the §10 hoist-vs-remat / phantom-frame quirks = the residual hard + tail (not source-steerable; permuter only helps relocs=0, and slowly). Defer as `INCLUDE_ASM` stub. + +**Scaling = Ultracode wave (§12 pattern + §7):** Ghidra pre-pass (`DecompileFunctions.java`, headless batch, +no /mcp) → parallel draft agents (m2c+Ghidra-C+asm+actor-struct+§3a, self-validate `match_one`) → whole-binary +gate (`harvest_verify --chunk 1`) → `sig_unify` recover → `dedup_propagate --auto-from`. Calibration (top-30): +60% match_one MATCH, **33% whole-binary** (+0.47% fleet), 136/136. +**THE CANONICAL-SIG WALL (the ~2× scaling lever):** the entire match_one→whole-binary gap is SIG CONFLICTS +(parallel agents declare shared callees inconsistently; 100% compile-errors, 0 codegen). Fix = a SURGICAL +per-callee canonical-sig layer (match shared callees before callers / seed `engine_core.h`; NOT a blanket +global header — that breaks loose matches, §15). Build it before the big wave. Targets: +`.run/harvest_targets_s3.json`. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index e35c4e48a..0c13bd2a2 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -8,7 +8,42 @@ HERE). T7 (go/no-go + PhaseEnd) is reframed: NO-GO on brute force, GO on hand-ma > Approved plan: `~/.claude/plans/plan-mode-enabled-deep-reserach-reactive-toucan.md` (the durable copy of > the deep-research findings + the task design). This file is the per-task crash-recovery log (P3/R28). -## 🚩 START HERE (fresh session, 2026-06-19) — PIVOT to guided hand-matching, then a demo +## 🚩 START HERE (fresh session 4, after session 3) — BUILD THE CANONICAL-SIG LAYER, then scale the wave +**Guided hand-matching is GO and proven at scale.** Demo (session 2) = 4/5; session 3 = the calibration +Ultracode wave (+0.47% → fleet **55.51%**, 136/136 byte-identical, committed `commit:0136`/`commit:0137`/`commit:0138`). +**The bottleneck is now identified and singular** — read `docs/hand-matching-process.md` §7 first. + +**THE FINDING:** in the calibration wave the agents hit **60% match_one MATCH but only 33% whole-binary**, and +the entire gap is **SIG CONFLICTS** (parallel agents declare shared callees like `func_80131CA8` with +different sigs → they conflict when built in the one big TU; 100% of failures were compile-errors, ZERO +codegen). `sig_unify` auto-recovered 2/9. **The canonical-sig layer is the essential ~2× yield enabler.** + +**NEXT TASK (Max, ultracode OFF — deep design):** build the **canonical-sig layer** before scaling: +1. Identify the high-frequency shared callees among the targets (the ones that conflict — `func_80131CA8`, + `func_80131E00`, `func_801472C8`, `func_8001D074`, …). +2. Establish ONE canonical sig per shared callee — match the shared-callee residuals **first** (callees + before callers, so their def fixes the sig), or derive the sig from the asm/usage and seed `engine_core.h`. + (Surgical/byte-gated per-callee — NOT a blanket global decls header; Phase-16 proved a global header breaks + loose matches, but the demo + sig_unify prove per-callee canonicalization works.) +3. Enforce it in the wave (agents reference engine_core.h + a strengthened `sig_unify` post-pass). +Then **scale the wave** to the remaining ~270 targets (`.run/harvest_targets_s3.json`, sorted by reach), in +Ultracode batches (R26/R27 — prompt Drew for `/effort ultracode`), gate (`harvest_verify`), propagate +(`dedup_propagate --auto-from ov_SC01_077`), `make check-all` (R22), `make report`. Then **T7 close + PhaseEnd**. + +**REUSABLE HARNESS (all in place):** `tools/ghidra_scripts/DecompileFunctions.java` (headless batch Ghidra-C — +NO /mcp needed; re-run: stop MCP, `analyzeHeadless ghidra bfm -process ov_SC01_077 -noanalysis -postScript +DecompileFunctions.java `); `.run/ghidra_c/func_.c` (300 cached); `.run/harvest_wave_s3.js` +(Workflow — **targets EMBEDDED, edit LIMIT**; the `args` channel does NOT transit arrays); `.run/harvest_targets_s3.json` +(300 tractable, relocs≤5, reach-sorted); `tools/harvest_verify.py` (whole-binary gate — use `--chunk 1` to avoid +cross-draft conflicts masking good drafts); `tools/sig_unify.py`; `tools/dedup_propagate.py`. Idioms + the full +calibration writeup: `docs/hand-matching-process.md` §2/§4/§6/§7; `docs/matching-cookbook.md` §16. + +**Giant `func_80144B9C` DEFERRED** (call-heavy = hardest class; reassess after the canonical-sig-layer'd wave). +**A-only (Drew):** quirk tail stays `INCLUDE_ASM` stub — NO NON_MATCHING correct-C track until public. + +--- + +## 🚩 (SUPERSEDED — session 3 demo protocol) START HERE — PIVOT to guided hand-matching, then a demo **What happened:** Phase 17 tested all 5 planned avenues (T1–T6 below, done & committed). **The harness ceiling did NOT rise to "eureka":** T2 = 0 functions, T3 = +0.52% (banked, real), T4/T5 = byte-neutral,