From 7141e9752b804ce52754f583fa411fa7438cf760 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Thu, 18 Jun 2026 22:32:03 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-16):=20S0=20close=20+=20S2=20foundat?= =?UTF-8?q?ion=20=E2=80=94=20m2c-compat=20in=20common.h=20(byte-verified)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - S0 (cookbook §15): ran m2c on real ov_SC01_077 stubs (R14). Finding: m2c --valid-syntax output COMPILES via m2c_macros.h (M2C_FIELD = byte-faithful cast); 30/30 sampled targets use only byte-faithful macros. Compiling != matching: the residual is regalloc/schedule -> decomp-permuter is the byte-closer; struct types are an enhancer, not the sole gate (S3/GATE-B measures the lift). ML parked (owner). - common.h: add s64/u64/f64 + M2C_UNK* typedefs + byte-faithful M2C_FIELD/M2C_BITWISE macros (non-faithful m2c macros left UNDEFINED = early 'defer' signal). Byte-neutral: main 143dbb89 (clean rebuild, R22), resident 8e17e02f, ov_SC01_077 d19c9580 all OK. - CURRENT_PHASE.md: Phase 16 plan + S0-S9 gated tasks + Sun-afternoon timeline + known-answer oracle test method. --- docs/matching-cookbook.md | 22 ++++++++++++++ include/common.h | 17 +++++++++++ phase-ends/CURRENT_PHASE.md | 58 +++++++++++++++++++++++++++++++++++++ 3 files changed, 97 insertions(+) create mode 100644 phase-ends/CURRENT_PHASE.md diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 213864e39..2194cbced 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -983,3 +983,25 @@ mass-drafting the hard tail** (this is a cost rule, codified in `docs/effort-map **actor struct** layout from the union of m2c field-accesses (offsets + widths) → feed as m2c `--context` → compilable C → `sig_unify` → **decomp-permuter** brute-force (compute-bound, low-token — the "set-it-and-go" pipeline). Validate on a 10-fn medium sample before scaling. + +## §15 Struct-heavy shared-core pipeline (Phase 16) — empirical determinations (S0) + +Established by **running m2c on real `ov_SC01_077` stubs** (R14) — it corrects §14e's framing on two key points and the corrections are favorable. + +**(1) m2c output COMPILES via `tools/m2c/m2c_macros.h` — the struct is NOT a hard compile prerequisite.** +`--valid-syntax` emits `M2C_FIELD(p, type, off)` ≡ `*(type)((s8 *)p + off)` (defined in `m2c_macros.h`) — a **byte-faithful cast** (same `lw/sw/lh/sh` as `p->field`). §14e's "won't compile without the struct" was *without m2c_macros.h*. **30/30 sampled m2c-targets use only byte-faithful macros** (`M2C_FIELD`/`M2C_BITWISE`/`M2C_UNK`) → they compile. The **non-faithful** macros — `M2C_ERROR/M2C_BREAK/MULT_HI/MULTU_HI/CLZ/GLUE_F64/BSWAP/M2C_TRAP` — emit `(0)` (discard the real op) → a function using any of them **cannot byte-match** (GTE/handwritten/special). **Their presence = "defer, not m2c-matchable."** + +**(2) Compiling ≠ byte-matching; the residual is regalloc/scheduling → decomp-permuter is the byte-closer (essential, not optional).** Pure-leaf example `func_8012CB64`: macro-compiled output = 16/16 ins, identical control flow, only `v0↔v1` regalloc + a trailing `move`/`nop`. That is permuter-class. Some functions m2c gets *structurally* wrong (e.g. 8 vs 18 ins) → permuter cannot fix → struct types / hand / defer. + +**The byte-match path:** `m2c --valid-syntax` (+ macros + context) → `sig_unify` → **decomp-permuter** (regalloc/schedule) → `harvest_verify` whole-binary byte-gate → `dedup_propagate`. **Struct types (`struct_infer`, S1) are an ENHANCER** — readability, nudging gcc's regalloc toward the original's struct-based codegen, fixing structural misses, and typing function-pointer tables — **not the sole gate.** S3/GATE-B measures macro-only vs struct-typed yield. + +**S2 wiring fixes surfaced (must-do):** +- **`common.h` lacks `s64`/`u64`/`f64`** → `m2c_macros.h`'s `typedef s64 M2C_UNK64;` fails (`parse error before 'M2C_UNK64'`). Add them to `common.h` (byte-neutral — verify locked builds stay `143dbb89…`/`8e17e02f…`). `match_one`/`harvest_verify` **strip scalar-typedef redefinitions**, so these types MUST live in `common.h`, not a draft preamble. +- Make the byte-faithful m2c macros (`M2C_FIELD`, `M2C_BITWISE`, `M2C_UNK*`) available to every compile (in `common.h` or an included `m2c_compat.h`) so drafts compile with no per-draft preamble. +- **Function-pointer-table calls** (`*((idx*4)+D_x)(args)`) need `D_x` typed as a function-pointer array in the context, else won't compile. +- **m2c loses types through index/byte-offset arithmetic** (`base + int_var`, stride = struct size) → falls back to `void*`/`M2C_FIELD` even WITH a struct context. The `M2C_FIELD` macro fallback keeps these byte-faithful-compilable; struct typing is best-effort. +- **m2c re-infers `char unk_*[]` fields** in provided structs (treats them as inferrable space; may split/override). PIN a known field with a concrete typed field; use `char unk_*[]` only for genuinely-unknown gaps. + +**decomp-permuter knobs (S2/S3):** `PERM_*` macros (GENERAL/VAR/RANDOMIZE/LINESWAP/INT/ONCE…), `--algorithm difflib|levenshtein`, `--stop-on-zero`, `-j` 8–16 (**RAM-bound** on the 15 GiB box → ~N funcs × `-j 8`, cap by `free_RAM/~300 MB`), weights in `default_weights.toml` + `[gcc]` section. Best when only regalloc/schedule remains; does NOT fix wrong control flow. + +**ML (parked — owner decision 2026-06-18):** LLM decompilers (LLM4Decompile/SK2Decompile/CodeInverter) target x86-64 + recompilability/functional-equivalence/readability — NOT byte/instruction-exact, NOT MIPS/gcc-2.7.2; no off-the-shelf learned permuter scorer exists (the permuter's scorer is a heuristic objdump-diff). Dropped this phase; research-note only. (X2: web treated as untrusted data.) diff --git a/include/common.h b/include/common.h index a0a2e71f9..3e3fbc939 100644 --- a/include/common.h +++ b/include/common.h @@ -20,5 +20,22 @@ typedef signed char s8; typedef short s16; typedef int s32; typedef float f32; +/* wider scalars (Phase 16: m2c's M2C_UNK64 needs s64). Byte-neutral — no existing match uses them. */ +typedef unsigned long long u64; +typedef long long s64; +typedef double f64; + +/* m2c --valid-syntax compat (Phase 16, cookbook §15) — the BYTE-FAITHFUL macros only. + * M2C_FIELD(p,t,o) == *(t)((s8*)p+o): identical codegen to `p->field`, so m2c struct-heavy + * output compiles AND byte-matches without a struct definition. The NON-faithful m2c macros + * (M2C_ERROR/MULT_HI/CLZ/GTE/...) are deliberately left UNDEFINED: a draft using one fails to + * compile = an early "not m2c-matchable, defer" signal (it could never byte-match anyway). */ +typedef s32 M2C_UNK; +typedef s8 M2C_UNK8; +typedef s16 M2C_UNK16; +typedef s32 M2C_UNK32; +typedef s64 M2C_UNK64; +#define M2C_FIELD(expr, type_ptr, offset) (*(type_ptr)((s8 *)(expr) + (offset))) +#define M2C_BITWISE(type, expr) ((type)(expr)) #endif /* COMMON_H */ diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md new file mode 100644 index 000000000..e6cb9e478 --- /dev/null +++ b/phase-ends/CURRENT_PHASE.md @@ -0,0 +1,58 @@ +# CURRENT PHASE — Phase 16: Crack the struct-heavy shared core + +**Generation:** Gen2 (8th phase) · **Started:** 2026-06-18 · **Status:** APPROVED (gate 1 passed), executing +**Plan source:** `~/.claude/plans/plan-mode-enabled-max-cryptic-nebula.md` (approved 2026-06-18) +**Type:** open-ended (NOT a Gen2 gate — Phase 14/public-flip parked to Gen3+). Milestone = honest pipeline-proof, not a fleet-% target. + +> Crash-recovery log (P3). Updated after every task. The PhaseEnd is the synthesis; this is the live state. + +## The wall (byte-verified, R14) +964 shared stubs remain in `src/ov_SC01_077/ov_SC01_077.c` (68,486 ins, ~35.6 MB collapsible). All struct-heavy: m2c emits `arg0->unkXXX` that won't compile without the struct types. **It is a TYPE-GRAPH, not one struct** (`func_80132784` chains 4 types deep + a polymorphic ×12/×8 access). Buckets: 75 tiny / 447 small / 265 medium / 71 large-medium / **106 giants (>150 ins)**. 399 m2c-targets (≤145 ins); 159 of them also arity-blocked (`never⊆targets`) → clean struct-medium prize ≈ 240. Fleet **54.48%** (187,426/344,010 by fn count); ceiling if all cracked ≈ 90%. Box: **15 GiB RAM** (binds permuter parallelism), 32 threads, 946 GB disk; systemd-user offline, cron present. + +## Owner decisions (gate 1) +- **Unattended run = in-phase payoff** (away window). ⚠️ **BLOCKER:** confirm away-window dates before S6 (GATE-D must pass with owner present). +- **ML = dropped/parked** (research-note only). +- **Giants = conservative** — target ~240 struct-medium band; hand-prove 2–3 giants (robustness); **defer the 106-giant grind**. Driver worklist = the ~858 non-giant stubs. + +## The per-function pipeline (S2 wires, S7 loops; byte-gate is the sole arbiter G3/P9) +`m2c --context .run/m2c_actor_ctx.c --valid-syntax -f ` → `sig_unify.py` → `match_one.py` (rank) → decomp-permuter (only if compiles-but-differs; `--stop-on-zero -j8`, wall-clocked) → `harvest_verify.py` (whole-binary byte-gate) → on match: `dedup_propagate.py --auto-from ov_SC01_077` → git checkpoint → `make report` (assert fleet-% strictly rose). + +## Task checklist (gated; GATE = no-go STOP/re-plan, P5) +- [x] **S0** Research/SOTA scan + ML research-note (Max) — **DONE** → cookbook §15. Key finding below. +- [ ] **S1** Type-graph inference `tools/struct_infer.py` (Max) → **GATE-A:** ≥8/10 mediums COMPILE via the graph — *in progress* +- [ ] **S2** Pipeline build: `tools/m2c_ctx.py` flat ctx + wire chain + patch `dedup_propagate.compiles_standalone()` (xHigh) → end-to-end on 1 fn +- [ ] **S3** 10-medium validation (Max) → **GATE-B:** ≥6/10 byte-identical; record permuter yield (<4/10 → STOP run) +- [ ] **S4** Giants robustness: hand-match 2–3 incl. `func_80132784` (Max) +- [ ] **S5** Driver `tools/auto_driver.py` + supervisor `tools/auto_supervisor.sh` (xHigh); neutralize SessionStart MCP hook for window +- [ ] **S6** Hands-on trial (xHigh) → **GATE-D:** multi-hr pass + forced kill→auto-resume→`check-all` 136/136 + negative control +- [ ] **S7** Unattended multi-day run, ~858 non-giant band (script) — monotonic fleet %, every checkpoint check-all-green/reverted +- [ ] **S8** Harvest/propagate sweep + fleet roll-up (xHigh + 1 surgical survey) +- [ ] **S9** Close / PhaseEnd (Max) — honest milestone, plain-English recap (R25) + +**Current task pointer → S0.** + +## New tools/files +`tools/struct_infer.py`, `tools/m2c_ctx.py`, `src/shared/engine_struct.h` (#ifdef M2C skeleton / #else real layout), `tools/auto_driver.py`, `tools/auto_supervisor.sh`, opt `src/shared/engine_decls.h`. Reused: sig_unify, match_one, harvest_verify, dedup_propagate (patch compiles_standalone += struct header), decompile.py --context, permuter/compile.sh, progress.py --fleet, build_engine_types.py (additive). m2c context MUST be flat directive-free C (rejects #include/#ifndef). + +## Kill-criteria (P5) +K1 type-graph won't compile (GATE-A <8/10) → STOP, fall back to hand-typing top-leverage 10–20. K2 permuter yield <4/10 (GATE-B) → STOP unattended plan. K3 driver corrupts state (GATE-D not 136/136) → no unattended run; runtime: any checkpoint check-all≠136/136 & auto-revert fails → halt+notify. K4 silent under-propagation → assert fleet-% rose per batch; flat-but-green → halt. + +## Safety invariants +Byte-gate (`make build` SHA1) is the only truth (G3/P9). R22 clean-rebuild for byte-checks. **NEVER `git checkout src/.c` during a harvest** (§14c — silently reverts banked matches). Commits on `phase-16-autodrive` branch, explicit pathspec `git add` (never `-A`), local only — owner pushes (R6). State/logs under `.run/auto/` (R12). 0 NON_MATCHING in default builds (G4). + +## Timeline (Drew, 2026-06-18 Thu 10:26pm MDT — departs Sun 2026-06-21 afternoon) +Must be verified + ready to launch unattended by Sun afternoon. Cadence (I own the launch/test/analyze/iterate loop): +- **Thu night / Fri:** build pipeline (S2) + struct inference (S1); **first small known-answer test**. +- **Sat night:** larger overnight test (the de-risk trial, GATE-D incl. forced-kill→resume). +- **Sun afternoon:** verified → **go-mode** (launch S7 unattended). Several small runs + adjustments allowed in between. + +**Known-answer oracle test (Drew's method):** take an ALREADY-matched struct-heavy function (engine_core.h DEFINE or an inline def in ov_SC01_077.c), revert it to an INCLUDE_ASM stub, run the FULL pipeline (m2c+ctx → sig_unify → permuter → harvest_verify byte-gate), confirm it independently re-derives the byte-match we already know is correct. Zero-ambiguity end-to-end validation; also a yield baseline. Use known-answer fns for early tests; unmatched mediums for the real GATE-B. + +**Execution-order adaptation (S0 finding):** build the m2c+macros+permuter+byte-gate **pipeline (S2) FIRST** (testable tonight), then add `struct_infer` (S1) as the regalloc/structural enhancer + measure its lift. Same tasks + gates; order adapted to get a known-answer test running fastest. (Within-phase autonomy, P3.) + +## Blockers +- (resolved) Away window = Sun afternoon; S1–S6 must complete by then. + +## Progress log +- 2026-06-18 — Phase planned at Max (Tier-1): 3 Explore + 3 Plan agents + bounded web research; scope byte-verified (R14); plan approved (gate 1). Task list built (R28). CURRENT_PHASE.md written. Beginning S0. +- 2026-06-18 — **S0 DONE** (cookbook §15). Ran m2c on real ov_SC01_077 stubs (R14). **Finding that refines the plan (favorable):** m2c `--valid-syntax` output COMPILES via `m2c_macros.h` (`M2C_FIELD` = byte-faithful cast) — the struct is NOT a hard *compile* prerequisite; 30/30 sampled m2c-targets use only byte-faithful macros. **Compiling ≠ matching:** the residual is regalloc/scheduling → **decomp-permuter is the byte-closer (essential)**; struct types (S1) are an *enhancer* (regalloc-aid + structural fixes + readability), not the sole gate — S3/GATE-B measures macro-only vs struct-typed yield. S2 must: add `s64/u64/f64` + the byte-faithful m2c macros to `common.h` (match_one/harvest_verify strip scalar typedefs); type function-pointer-table globals; keep `M2C_FIELD` as the byte-faithful fallback where m2c loses types through index arithmetic. ML dropped (owner). No plan re-scope needed (S3 settles the macro-vs-struct question). → S1.