mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-08 01:34:14 -04:00
feat(phase-25): T3 — mechanical symbol-remap lever PROVEN (tools/family_remap.py); free --tier h_norm refuted
Swing question resolved. h_norm families are byte-shattered by PER-OVERLAY symbols, so free --tier h_norm dedup fails (0/133 on 0x80141100). BUT the mechanical remap works: disassemble two h_norm-identical members, positionally pair their reloc targets, substitute the exemplar per-overlay symbol names -> the sibling C. Verified: reloc decoder 22/22 vs .s; 0x80141100-> ov_SC01_000 whole-binary byte-identical + harvest_verify banked; 12/12 siblings; 7/9 families x3 = 21/27 match_one. The 2 fails = decl/type plumbing (T7 refinement), not remap fails. Reshapes the endgame: crack ONE exemplar/family -> remap+gate members (~0 tokens/member); the 11.1 MB is cheaply mechanical. Members no longer need drafting; only exemplars do.
This commit is contained in:
Drew T
+28
-10
@@ -26,22 +26,22 @@ idiom curriculum → sweep one exemplar per family (curriculum-ordered), each cr
|
||||
- [x] T0 — Refresh + housekeeping
|
||||
- [x] T1 — Integration-recovery pass (split-aware recover_integration fix; caller-arity cohort exhausted 0/16 — residuals backlogged)
|
||||
- [x] T2 — Exemplar target manifest (tools/family_manifest.py: 2,764 multi-member families → 127 draftable / 156 matched-free / 2481 absent)
|
||||
- [ ] **T3 — Swing-question PROOF** (draft-free via the 156 matched-free families → `--tier h_norm` propagation → measure) ← CURRENT
|
||||
- [x] T3 — Swing-question PROOF: free `--tier h_norm` REFUTED (0/133); **mechanical symbol-remap lever PROVEN** (tools/family_remap.py) ← endgame reshaped, see log
|
||||
- [ ] T4 — v4 retrain + A/B gate (cheap drafter prep; may run parallel to T2/T3)
|
||||
- [ ] T5 — Cheap-tier soften + measure wave → the complete class/closeness frontier map (Step A) + pre-advanced seeds *(Ultracode — prompt)*
|
||||
- [ ] T6 — Step B: Fable5Max curriculum authoring from the measured map *(Fable5Max — prompt)*
|
||||
- [ ] T7 — Sweep one exemplar per family, curriculum-ordered (validate top ~3-5 first, then scale) *(Ultracode harvests / Fable5 discovery — prompt)*
|
||||
- [ ] Close — clean-fleet verify · PhaseEnd synthesis · plain-English recap (R25) · Phase-26 backlog
|
||||
|
||||
## Current task — T3: Swing-question PROOF (decisive, draft-free)
|
||||
The 156 **matched-free** families (2.3 MB) prove the reloc-only ×N economics WITHOUT drafting: ov_SC01_077's
|
||||
member is already MATCHED, and the family has UNMATCHED h_norm-siblings in other overlays (byte-variants Phase-24
|
||||
propagated only via `--tier h_exact`). Steps: (1) pick a mid-size matched-free fn (find an already-matched
|
||||
ov_SC01_077 addr whose h_norm has unmatched siblings); (2) `tools/dedup_propagate.py --addr <it> --tier h_norm
|
||||
--recover`; (3) measure how many siblings bank via the per-overlay byte-gate. A meaningful bank ⟹ the free
|
||||
reloc-only propagation lever is proven (one C body matches h_norm-siblings when reloc targets resolve per-overlay
|
||||
— the consistent-naming condition holds), banking free % now; then sweep the 156 (T3→partial-T7). Clean-fleet
|
||||
verify (R22). If h_norm propagation banks ~0, fall back to draft-then-propagate on a `draftable` exemplar.
|
||||
## Current task — reshape decision (report to Drew): the endgame is now MECHANICAL, not agent-drafted
|
||||
T3 refuted the free lever but PROVED the mechanical remap (`tools/family_remap.py`). This collapses family MEMBER
|
||||
work to ~0 agent tokens: crack ONE exemplar per family, then remap generates each member's C by positionally
|
||||
substituting the exemplar's per-overlay symbols (from each member's image). So the endgame decomposes:
|
||||
- **matched-free 156 / 2.3 MB** — the ov077 exemplar is ALREADY matched → remap+gate the members NOW (no drafting).
|
||||
- **draftable 127 / 6.7 MB** — crack the ov077 exemplar (T5 wave / v4), then remap+gate members.
|
||||
- The wave (T5) + v4 (T4) now crack **EXEMPLARS only** (127 + absent-family exemplars), NOT members.
|
||||
Next: build the family-sweep driver (remap → decl-reconcile → `harvest_verify` → register/propagate) + harvest the
|
||||
156 matched-free (free-ish, validates the sweep at scale), then reassess T4/T5. Pending Drew's steer on the reshape.
|
||||
|
||||
## Blockers
|
||||
None.
|
||||
@@ -97,3 +97,21 @@ None.
|
||||
- **Insight:** h_norm families are reloc-only by construction, so the matched-free bucket is likely bankable NOW by
|
||||
re-propagating already-matched ov077 fns via `--tier h_norm` (byte-gated). T3 tests decisively.
|
||||
- T3 candidates (mid-size draftable): 0x8012c890(149)/0x8016d1d8(148)/0x8013d9b0(141) — all inst=134, ~127 hexcls.
|
||||
|
||||
### T3 — Swing-question PROOF ✅ (2026-07-08) — free lever refuted; **mechanical remap PROVEN** (the phase's pivot)
|
||||
- **Free `--tier h_norm` dedup REFUTED (R14):** propagating ov_SC01_077's matched `0x80141100` across its 133
|
||||
h_norm-siblings banked **0/133** (all byte-diverge). h_norm families are byte-shattered *because each member
|
||||
references per-overlay symbols* (`D_80187xxx` in ov077 vs `D_8017Fxxx` in ov000) — one C body can't name 134
|
||||
overlays' symbols. This also kills the plan's draft-then-propagate fallback (same mechanism).
|
||||
- **Mechanical symbol-remap lever PROVEN → `tools/family_remap.py`:** two h_norm-identical members have identical
|
||||
instruction streams except in the masked reloc fields, so disassemble both overlay images at ADDR, positionally
|
||||
pair the resolved reloc targets, substitute the exemplar C's per-overlay symbol NAMES with the sibling's. Verified:
|
||||
reloc decoder **22/22** vs .s; `0x80141100`→ov_SC01_000 **whole-binary BYTE-IDENTICAL + harvest_verify banked**
|
||||
(9052dc0e); **12/12** siblings SC01-SC07 match_one; **7/9 families × 3 siblings = 21/27** match_one.
|
||||
- **The 2 failures = decl/type plumbing** (custom `MatEntry` type + conflicting shared `ApplyMatrixSV` decl — a
|
||||
COMPILE error, NOT a byte mismatch): the remapped draft must reconcile decls with the target's ambient
|
||||
engine_core.h/engine_types.h (strip ambient externs). A known-solved T7 refinement, not a remap failure.
|
||||
- **Gate path for remapped drafts = plain `harvest_verify`** (gate_stage's canon/cast/sig_unify transforms PERTURB
|
||||
an already-correct remap → 0-bank; skip them for remaps).
|
||||
- **Endgame reshaped:** the 11.1 MB h_norm families are cheaply, MECHANICALLY recoverable — crack ONE exemplar per
|
||||
family → remap+gate members (~0 agent tokens/member). Members no longer need drafting; only exemplars do.
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase-25 family template-instantiation: mechanically remap a MATCHED exemplar's C to a byte-variant
|
||||
h_norm SIBLING by positionally pairing their per-overlay symbol references.
|
||||
|
||||
The Phase-24 reframe found the unmatched frontier clusters into h_norm STRUCTURAL families (same code
|
||||
shape across overlays, byte-shattered because each member references PER-OVERLAY symbols). T3 proved these
|
||||
are NOT free `--tier h_norm` dedup (one C body can't name 134 overlays' different symbols). But two
|
||||
h_norm-identical members have IDENTICAL instruction streams except in the masked reloc fields (jal target,
|
||||
lui/lo address) — so: disassemble both overlay images at ADDR, pair the resolved reloc targets in
|
||||
instruction order, and substitute the exemplar C's per-overlay symbol NAMES (`D_<addr>`/`func_<addr>`)
|
||||
with the sibling's. Shared EXE/resident symbols map to themselves. The result is the sibling's C —
|
||||
generated mechanically from ONE crack, for ~0 agent tokens. The whole-binary byte-gate stays the sole
|
||||
arbiter (G3/P9): a wrong remap is rejected.
|
||||
|
||||
tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--out draft.c]
|
||||
"""
|
||||
import struct, json, glob, re, sys, argparse
|
||||
|
||||
VRAM = 0x80128158
|
||||
# lo-type ops whose rs is a hi-base (loads/stores incl. unaligned, addiu, ori)
|
||||
LO_OPS = {0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x28, 0x29, 0x2A, 0x2B, 0x2E, 0x09, 0x0D}
|
||||
|
||||
|
||||
def img_path(ov):
|
||||
_, sc, fn = ov.split("_")
|
||||
p = f"extracted/retail/{sc}.CD.dir/FILE_{fn}.dir/0.4.dec"
|
||||
return p if glob.glob(p) else None
|
||||
|
||||
|
||||
def nins_of(ov, addr):
|
||||
for line in open(f".run/sig.{ov}.jsonl"):
|
||||
d = json.loads(line)
|
||||
if int(d["addr"], 16) == addr:
|
||||
return d["nins"]
|
||||
return None
|
||||
|
||||
|
||||
def reloc_targets(ov, addr):
|
||||
"""ordered [(kind, resolved_addr)] for jal targets + lui/lo address loads, in instruction order.
|
||||
Verified against splat .s ground truth (22/22 on func_80141100)."""
|
||||
data = open(img_path(ov), "rb").read()
|
||||
n = nins_of(ov, addr)
|
||||
off = addr - VRAM
|
||||
out, pend = [], {}
|
||||
for k in range(n):
|
||||
pc = addr + k * 4
|
||||
w = struct.unpack_from("<I", data, off + k * 4)[0]
|
||||
op = w >> 26
|
||||
if op in (2, 3): # j / jal
|
||||
out.append(("call", ((pc + 4) & 0xF0000000) | ((w & 0x03FFFFFF) << 2)))
|
||||
elif op == 0x0F: # lui -> rt holds the hi
|
||||
pend[(w >> 16) & 0x1F] = (w & 0xFFFF) << 16
|
||||
elif op in LO_OPS:
|
||||
rs = (w >> 21) & 0x1F
|
||||
if rs in pend:
|
||||
lo = w & 0xFFFF
|
||||
if lo >= 0x8000:
|
||||
lo -= 0x10000
|
||||
out.append(("data", pend[rs] + lo))
|
||||
del pend[rs]
|
||||
pend.pop((w >> 16) & 0x1F, None) # rt overwritten
|
||||
elif op == 0: # R-type: rd overwritten
|
||||
pend.pop((w >> 11) & 0x1F, None)
|
||||
return out
|
||||
|
||||
|
||||
def symbol_map(addr, from_ov, to_ov):
|
||||
"""{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean."""
|
||||
ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, addr)
|
||||
if len(ex) != len(tg):
|
||||
return None, f"reloc-count mismatch {len(ex)}!={len(tg)} (not an h_norm-clean pair)"
|
||||
m = {}
|
||||
for (ke, ae), (kt, at) in zip(ex, tg):
|
||||
if ke != kt:
|
||||
return None, "reloc-kind mismatch (not h_norm-clean)"
|
||||
if ae != at:
|
||||
pfx = "func_" if ke == "call" else "D_"
|
||||
m[f"{pfx}{ae:08X}"] = f"{pfx}{at:08X}"
|
||||
return m, None
|
||||
|
||||
|
||||
def extract_unit(ov, addr):
|
||||
"""the matched inline def + its contiguous preceding extern/blank/comment lines, from the overlay src.
|
||||
func_<addr> names are UPPERCASE-hex in src (func_8013DBE4); match case-insensitively to be safe."""
|
||||
pat = re.compile(rf'^\s*[A-Za-z_][\w \*]*\bfunc_{addr:08X}\s*\(', re.I)
|
||||
for cf in sorted(glob.glob(f"src/{ov}/{ov}*.c")):
|
||||
lines = open(cf).read().split("\n")
|
||||
for i, ln in enumerate(lines):
|
||||
if pat.search(ln) and "INCLUDE_ASM" not in ln and not ln.rstrip().endswith(";"):
|
||||
j = i - 1
|
||||
while j >= 0 and (lines[j].strip() == "" or
|
||||
lines[j].lstrip().startswith(("extern", "//", "/*", "*"))):
|
||||
j -= 1
|
||||
start = j + 1
|
||||
depth, started, end = 0, False, i
|
||||
for k in range(i, len(lines)):
|
||||
depth += lines[k].count("{") - lines[k].count("}")
|
||||
if "{" in lines[k]:
|
||||
started = True
|
||||
if started and depth <= 0:
|
||||
end = k
|
||||
break
|
||||
return "\n".join(lines[start:end + 1]), cf
|
||||
return None, None
|
||||
|
||||
|
||||
def remap(addr, from_ov, to_ov):
|
||||
"""returns (draft_text, symbol_map) or (None, error_str)."""
|
||||
m, err = symbol_map(addr, from_ov, to_ov)
|
||||
if err:
|
||||
return None, err
|
||||
unit, cf = extract_unit(from_ov, addr)
|
||||
if not unit:
|
||||
return None, f"no matched unit for func_{addr:08x} in {from_ov}"
|
||||
for src, dst in m.items():
|
||||
unit = re.sub(rf'\b{src}\b', dst, unit)
|
||||
return unit, m
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--addr", required=True)
|
||||
ap.add_argument("--from", dest="frm", required=True)
|
||||
ap.add_argument("--to", required=True)
|
||||
ap.add_argument("--out", default=None)
|
||||
a = ap.parse_args()
|
||||
addr = int(a.addr, 16)
|
||||
draft, m = remap(addr, a.frm, a.to)
|
||||
if draft is None:
|
||||
print(f"REMAP FAIL (func_{addr:08x} {a.frm}->{a.to}): {m}")
|
||||
sys.exit(1)
|
||||
out = a.out or f".run/remap_{a.to}_{addr:08x}.c"
|
||||
open(out, "w").write(draft + "\n")
|
||||
print(f"remapped {len(m)} per-overlay symbol(s) {a.frm}->{a.to}: {m}")
|
||||
print(f"-> {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user