diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index dd502898a9..35e8c2179e 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -26,22 +26,22 @@ idiom curriculum → sweep one exemplar per family (curriculum-ordered), each cr - [x] T0 — Refresh + housekeeping - [x] T1 — Integration-recovery pass (split-aware recover_integration fix; caller-arity cohort exhausted 0/16 — residuals backlogged) - [x] T2 — Exemplar target manifest (tools/family_manifest.py: 2,764 multi-member families → 127 draftable / 156 matched-free / 2481 absent) -- [ ] **T3 — Swing-question PROOF** (draft-free via the 156 matched-free families → `--tier h_norm` propagation → measure) ← CURRENT +- [x] T3 — Swing-question PROOF: free `--tier h_norm` REFUTED (0/133); **mechanical symbol-remap lever PROVEN** (tools/family_remap.py) ← endgame reshaped, see log - [ ] T4 — v4 retrain + A/B gate (cheap drafter prep; may run parallel to T2/T3) - [ ] T5 — Cheap-tier soften + measure wave → the complete class/closeness frontier map (Step A) + pre-advanced seeds *(Ultracode — prompt)* - [ ] T6 — Step B: Fable5Max curriculum authoring from the measured map *(Fable5Max — prompt)* - [ ] T7 — Sweep one exemplar per family, curriculum-ordered (validate top ~3-5 first, then scale) *(Ultracode harvests / Fable5 discovery — prompt)* - [ ] Close — clean-fleet verify · PhaseEnd synthesis · plain-English recap (R25) · Phase-26 backlog -## Current task — T3: Swing-question PROOF (decisive, draft-free) -The 156 **matched-free** families (2.3 MB) prove the reloc-only ×N economics WITHOUT drafting: ov_SC01_077's -member is already MATCHED, and the family has UNMATCHED h_norm-siblings in other overlays (byte-variants Phase-24 -propagated only via `--tier h_exact`). Steps: (1) pick a mid-size matched-free fn (find an already-matched -ov_SC01_077 addr whose h_norm has unmatched siblings); (2) `tools/dedup_propagate.py --addr --tier h_norm ---recover`; (3) measure how many siblings bank via the per-overlay byte-gate. A meaningful bank ⟹ the free -reloc-only propagation lever is proven (one C body matches h_norm-siblings when reloc targets resolve per-overlay -— the consistent-naming condition holds), banking free % now; then sweep the 156 (T3→partial-T7). Clean-fleet -verify (R22). If h_norm propagation banks ~0, fall back to draft-then-propagate on a `draftable` exemplar. +## Current task — reshape decision (report to Drew): the endgame is now MECHANICAL, not agent-drafted +T3 refuted the free lever but PROVED the mechanical remap (`tools/family_remap.py`). This collapses family MEMBER +work to ~0 agent tokens: crack ONE exemplar per family, then remap generates each member's C by positionally +substituting the exemplar's per-overlay symbols (from each member's image). So the endgame decomposes: +- **matched-free 156 / 2.3 MB** — the ov077 exemplar is ALREADY matched → remap+gate the members NOW (no drafting). +- **draftable 127 / 6.7 MB** — crack the ov077 exemplar (T5 wave / v4), then remap+gate members. +- The wave (T5) + v4 (T4) now crack **EXEMPLARS only** (127 + absent-family exemplars), NOT members. +Next: build the family-sweep driver (remap → decl-reconcile → `harvest_verify` → register/propagate) + harvest the +156 matched-free (free-ish, validates the sweep at scale), then reassess T4/T5. Pending Drew's steer on the reshape. ## Blockers None. @@ -97,3 +97,21 @@ None. - **Insight:** h_norm families are reloc-only by construction, so the matched-free bucket is likely bankable NOW by re-propagating already-matched ov077 fns via `--tier h_norm` (byte-gated). T3 tests decisively. - T3 candidates (mid-size draftable): 0x8012c890(149)/0x8016d1d8(148)/0x8013d9b0(141) — all inst=134, ~127 hexcls. + +### T3 — Swing-question PROOF ✅ (2026-07-08) — free lever refuted; **mechanical remap PROVEN** (the phase's pivot) +- **Free `--tier h_norm` dedup REFUTED (R14):** propagating ov_SC01_077's matched `0x80141100` across its 133 + h_norm-siblings banked **0/133** (all byte-diverge). h_norm families are byte-shattered *because each member + references per-overlay symbols* (`D_80187xxx` in ov077 vs `D_8017Fxxx` in ov000) — one C body can't name 134 + overlays' symbols. This also kills the plan's draft-then-propagate fallback (same mechanism). +- **Mechanical symbol-remap lever PROVEN → `tools/family_remap.py`:** two h_norm-identical members have identical + instruction streams except in the masked reloc fields, so disassemble both overlay images at ADDR, positionally + pair the resolved reloc targets, substitute the exemplar C's per-overlay symbol NAMES with the sibling's. Verified: + reloc decoder **22/22** vs .s; `0x80141100`→ov_SC01_000 **whole-binary BYTE-IDENTICAL + harvest_verify banked** + (9052dc0e); **12/12** siblings SC01-SC07 match_one; **7/9 families × 3 siblings = 21/27** match_one. +- **The 2 failures = decl/type plumbing** (custom `MatEntry` type + conflicting shared `ApplyMatrixSV` decl — a + COMPILE error, NOT a byte mismatch): the remapped draft must reconcile decls with the target's ambient + engine_core.h/engine_types.h (strip ambient externs). A known-solved T7 refinement, not a remap failure. +- **Gate path for remapped drafts = plain `harvest_verify`** (gate_stage's canon/cast/sig_unify transforms PERTURB + an already-correct remap → 0-bank; skip them for remaps). +- **Endgame reshaped:** the 11.1 MB h_norm families are cheaply, MECHANICALLY recoverable — crack ONE exemplar per + family → remap+gate members (~0 agent tokens/member). Members no longer need drafting; only exemplars do. diff --git a/tools/family_remap.py b/tools/family_remap.py new file mode 100644 index 0000000000..0a61290106 --- /dev/null +++ b/tools/family_remap.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Phase-25 family template-instantiation: mechanically remap a MATCHED exemplar's C to a byte-variant +h_norm SIBLING by positionally pairing their per-overlay symbol references. + +The Phase-24 reframe found the unmatched frontier clusters into h_norm STRUCTURAL families (same code +shape across overlays, byte-shattered because each member references PER-OVERLAY symbols). T3 proved these +are NOT free `--tier h_norm` dedup (one C body can't name 134 overlays' different symbols). But two +h_norm-identical members have IDENTICAL instruction streams except in the masked reloc fields (jal target, +lui/lo address) — so: disassemble both overlay images at ADDR, pair the resolved reloc targets in +instruction order, and substitute the exemplar C's per-overlay symbol NAMES (`D_`/`func_`) +with the sibling's. Shared EXE/resident symbols map to themselves. The result is the sibling's C — +generated mechanically from ONE crack, for ~0 agent tokens. The whole-binary byte-gate stays the sole +arbiter (G3/P9): a wrong remap is rejected. + + tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--out draft.c] +""" +import struct, json, glob, re, sys, argparse + +VRAM = 0x80128158 +# lo-type ops whose rs is a hi-base (loads/stores incl. unaligned, addiu, ori) +LO_OPS = {0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x28, 0x29, 0x2A, 0x2B, 0x2E, 0x09, 0x0D} + + +def img_path(ov): + _, sc, fn = ov.split("_") + p = f"extracted/retail/{sc}.CD.dir/FILE_{fn}.dir/0.4.dec" + return p if glob.glob(p) else None + + +def nins_of(ov, addr): + for line in open(f".run/sig.{ov}.jsonl"): + d = json.loads(line) + if int(d["addr"], 16) == addr: + return d["nins"] + return None + + +def reloc_targets(ov, addr): + """ordered [(kind, resolved_addr)] for jal targets + lui/lo address loads, in instruction order. + Verified against splat .s ground truth (22/22 on func_80141100).""" + data = open(img_path(ov), "rb").read() + n = nins_of(ov, addr) + off = addr - VRAM + out, pend = [], {} + for k in range(n): + pc = addr + k * 4 + w = struct.unpack_from("> 26 + if op in (2, 3): # j / jal + out.append(("call", ((pc + 4) & 0xF0000000) | ((w & 0x03FFFFFF) << 2))) + elif op == 0x0F: # lui -> rt holds the hi + pend[(w >> 16) & 0x1F] = (w & 0xFFFF) << 16 + elif op in LO_OPS: + rs = (w >> 21) & 0x1F + if rs in pend: + lo = w & 0xFFFF + if lo >= 0x8000: + lo -= 0x10000 + out.append(("data", pend[rs] + lo)) + del pend[rs] + pend.pop((w >> 16) & 0x1F, None) # rt overwritten + elif op == 0: # R-type: rd overwritten + pend.pop((w >> 11) & 0x1F, None) + return out + + +def symbol_map(addr, from_ov, to_ov): + """{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean.""" + ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, addr) + if len(ex) != len(tg): + return None, f"reloc-count mismatch {len(ex)}!={len(tg)} (not an h_norm-clean pair)" + m = {} + for (ke, ae), (kt, at) in zip(ex, tg): + if ke != kt: + return None, "reloc-kind mismatch (not h_norm-clean)" + if ae != at: + pfx = "func_" if ke == "call" else "D_" + m[f"{pfx}{ae:08X}"] = f"{pfx}{at:08X}" + return m, None + + +def extract_unit(ov, addr): + """the matched inline def + its contiguous preceding extern/blank/comment lines, from the overlay src. + func_ names are UPPERCASE-hex in src (func_8013DBE4); match case-insensitively to be safe.""" + pat = re.compile(rf'^\s*[A-Za-z_][\w \*]*\bfunc_{addr:08X}\s*\(', re.I) + for cf in sorted(glob.glob(f"src/{ov}/{ov}*.c")): + lines = open(cf).read().split("\n") + for i, ln in enumerate(lines): + if pat.search(ln) and "INCLUDE_ASM" not in ln and not ln.rstrip().endswith(";"): + j = i - 1 + while j >= 0 and (lines[j].strip() == "" or + lines[j].lstrip().startswith(("extern", "//", "/*", "*"))): + j -= 1 + start = j + 1 + depth, started, end = 0, False, i + for k in range(i, len(lines)): + depth += lines[k].count("{") - lines[k].count("}") + if "{" in lines[k]: + started = True + if started and depth <= 0: + end = k + break + return "\n".join(lines[start:end + 1]), cf + return None, None + + +def remap(addr, from_ov, to_ov): + """returns (draft_text, symbol_map) or (None, error_str).""" + m, err = symbol_map(addr, from_ov, to_ov) + if err: + return None, err + unit, cf = extract_unit(from_ov, addr) + if not unit: + return None, f"no matched unit for func_{addr:08x} in {from_ov}" + for src, dst in m.items(): + unit = re.sub(rf'\b{src}\b', dst, unit) + return unit, m + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--addr", required=True) + ap.add_argument("--from", dest="frm", required=True) + ap.add_argument("--to", required=True) + ap.add_argument("--out", default=None) + a = ap.parse_args() + addr = int(a.addr, 16) + draft, m = remap(addr, a.frm, a.to) + if draft is None: + print(f"REMAP FAIL (func_{addr:08x} {a.frm}->{a.to}): {m}") + sys.exit(1) + out = a.out or f".run/remap_{a.to}_{addr:08x}.c" + open(out, "w").write(draft + "\n") + print(f"remapped {len(m)} per-overlay symbol(s) {a.frm}->{a.to}: {m}") + print(f"-> {out}") + + +if __name__ == "__main__": + main()