mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 23:24:32 -04:00
feat(phase-26): task 1 — extended reloc tracker (addu-hi) + single-pass remap; V0/V1 green
- reloc_targets: propagate lui-hi through add/addu index arithmetic (gcc-2.7.2 indexed-global idiom lui;addu $idx;lw %lo($at)). The pre-26 tracker dropped it -> D[i] functions mis-normalized per overlay (inflated the 'h_norm reach-1 tail') AND lost their indexed D_ symbols in remap. norm_stream/h_norm deliberately UNTOUCHED (fleet metrics + proven sweep depend on it). - remap/symbol_map: backward-compatible to_addr=None (cross-address T2b sibling + self-rename) + imm_map hook (T2a); sequential re.sub -> single-pass simultaneous substitution (fixes latent chained-rename/value-permutation corruption). All 5 family_sweep call sites unchanged. - V0 (.run/v0_reloc.py): func_80141100 22/22 NEW==OLD (zero regression); func_801407F4 15/15 vs splat .s (pre-fix 10), recovers indexed D_80187B88/90/B0; cross-addr symbol_map clean. - V1 (.run/v1_regression.py): 160 real h_norm sibling pairs, 96 SAME, 0 lost — differences are strict indexed-reloc improvements only. - docs: cookbook §40b (the technique, R30) + decision-log 2026-07-11 (the strategic why, R31).
This commit is contained in:
Drew T
@@ -260,3 +260,31 @@ sibling TU?". A hand crack that banked in ov077 by exotic register pins does not
|
||||
in ITS TU only; the ×134 claim needs a sibling-TU compile probe, because pins are TU-context-specific and cc1 *crashes*
|
||||
(not just drifts) on the ones that don't transfer. Corollary: rtu_match/match_one are blind here — their neutralized/
|
||||
isolation compiles crash too (harness artifact); only the real `make build` is the arbiter.
|
||||
|
||||
## 2026-07-11 · Phase 26 — the "reach-1 tail" is largely a reloc-tracker blind spot, not unique code (Task 1)
|
||||
|
||||
**Context + belief (from the Phase-25 close):** the h_seq reframe had already shown the "36k unique tail" collapses
|
||||
~90% into per-location families. The open question entering Phase 26 was HOW the families differ — the megaplan framed
|
||||
immediate-substitution as the central new problem (families "differ in immediates, so are NOT free dedup").
|
||||
|
||||
**What the design pressure-test found (byte-verified before any scaling — R14):** the dominant difference is NOT
|
||||
immediates — it is a **tracker blind spot**. `norm_stream`/`reloc_targets` dropped the lui-hi on every R-type write,
|
||||
but gcc-2.7.2's indexed-global idiom `lui;addu $idx;lw %lo($at)` preserves it. So `D[i]`-indexing functions were
|
||||
*mis-normalized per overlay* → they inflated the "h_norm reach-1 tail," and `family_remap` silently dropped their
|
||||
indexed `D_` symbols → those families couldn't bank even though they are pure per-location templates. On the
|
||||
substantial tail the classification is **PURE-same-addr 62 fams / 1.55M ins · PURE-cross-addr 103 / 0.10M · genuine
|
||||
IMM only 8 / 0.10M** — i.e. ~95% of the byte-weight is reloc-only, fixable by a ≤15-LOC tracker change, and the
|
||||
immediate engine shrinks to an escalation tier for ~8 families. A second latent bug surfaced alongside: `remap`'s
|
||||
sequential substitution corrupts chained/permuted maps (harmless on h_norm, breaks the imm engine).
|
||||
|
||||
**The pivot:** front-load the tracker fix (Task 1) as the load-bearing change, demote the immediate engine to a
|
||||
diff-driven 3-tier escalation (Task 3), and add a **free validation corpus** — 63 families / 0.31M ins already have a
|
||||
MATCHED exemplar and only failed earlier sweeps from this bug → they bank with zero cracking the moment the fix lands
|
||||
(Task 5 V2), simultaneously measuring the real template success rate before any Fable5 spend.
|
||||
|
||||
**Better path (hindsight):** the tracker's own design note already said "conservative: can miss a match, never forge
|
||||
one" — but a *missed* reloc in a REMAP tool isn't harmless the way a missed h_norm match is; it silently produces a
|
||||
wrong-but-compiling sibling body that only the byte-gate catches. When a normalization/remap tool is REUSED for code
|
||||
generation (not just clustering), its conservative-miss becomes a correctness bug. **LESSON (R14):** before treating a
|
||||
"unique/unmatchable" population as intrinsic, re-run the *grouping and the remap* under a corrected fingerprint —
|
||||
here the "reach-1 tail" and the "unremappable family" were the SAME artifact of one dropped register-tracking case.
|
||||
|
||||
@@ -2310,6 +2310,48 @@ includes (`src/shared/engine_types.h` via `engine_core.h`), then re-sweep. **+1,
|
||||
at this cheap ~30s gate, before any expensive sweep. Then the full R22 clean-fleet 136/136 confirms no fleet-wide
|
||||
header collision. Tools: `build_engine_types.py --file/--exclude`, `family_sweep.py --no-preclassify`.
|
||||
|
||||
### §40b — The reloc-tracker blind spot: the indexed-global idiom that hid the "reach-1 tail" (Phase 26 Task 1, 2026-07-11, byte-verified V0/V1)
|
||||
|
||||
**The discovery (why the "36k unique tail" was largely a measurement artifact).** Both `norm_stream`
|
||||
(`sig_image.py`) and `reloc_targets` (`family_remap.py`) tracked lui-hi/lo pairs but **popped the pending hi on
|
||||
ANY R-type write** (`pend.pop(rd)`). gcc-2.7.2's indexed-global access `D[i]` compiles to
|
||||
`lui $at,%hi(D); addu $at,$at,$idx; lw $v1,%lo(D)($at)` — the **`addu` PRESERVES the hi anchor** (the index shifts
|
||||
the runtime value, not the symbol). So every function that indexes a per-overlay global array left its `%lo` fields
|
||||
**raw** in `h_norm` → the function normalized DIFFERENTLY per overlay → it looked **fleet-unique (h_norm reach-1)**
|
||||
when it is actually a per-location family, AND `family_remap`'s symbol map dropped those indexed `D_` symbols →
|
||||
the sibling draft kept the exemplar's array name → **byte-gate fail** (an earlier "unremappable" wall).
|
||||
|
||||
**The fix (≤15 LOC, `reloc_targets` R-type branch only — NOT `norm_stream`).** On `add`/`addu` (funct 0x20/0x21),
|
||||
propagate the pending hi to `rd` when a source reg holds one, else pop:
|
||||
```python
|
||||
elif op == 0: # R-type
|
||||
funct = w & 0x3F; rd = (w >> 11) & 0x1F
|
||||
if funct in (0x20, 0x21): # add/addu: address arithmetic preserves the hi anchor
|
||||
rs, rt = (w>>21)&0x1F, (w>>16)&0x1F
|
||||
if rs in pend: pend[rd] = pend[rs]
|
||||
elif rt in pend: pend[rd] = pend[rt]
|
||||
else: pend.pop(rd, None)
|
||||
else: pend.pop(rd, None)
|
||||
```
|
||||
The `%lo` resolution is unchanged (`pend[rs] + signext(lo)` = the symbol; the index is a runtime reg). **Leave
|
||||
`sig_image.norm_stream` / `h_norm` UNTOUCHED** — the fleet metrics, the proven h_norm sweep, and the manifest all
|
||||
depend on its stable (blind) hashing; the h_seq family key (mnemonic skeleton) is unaffected by the tracker, so
|
||||
families still cluster correctly, and the fix only makes the SYMBOL PAIRING correct so the remapped body byte-matches.
|
||||
|
||||
**Verified (build-free, V0/V1 `2026-07-11`):** `func_801407F4` resolves **15/15 relocs vs splat `.s`** (pre-fix: 10),
|
||||
recovering the indexed arrays `D_80187B88/90/B0`; `func_80141100` (no idiom) stays **22/22 identical** (zero regression);
|
||||
across 160 real h_norm sibling pairs the new `remap` output is byte-identical to the committed pre-fix output (96 SAME,
|
||||
**0 lost**), differing only where it strictly recovers indexed relocs. Reproduce: `.run/v0_reloc.py`, `.run/v1_regression.py`.
|
||||
|
||||
**Companion fix — single-pass simultaneous substitution.** The old `remap` applied renames **sequentially**
|
||||
(`for src,dst: re.sub`), which corrupts a chained/permuted map (`D_A→D_B` then `D_B→D_C`, or an immediate value
|
||||
permutation `0x10→0xA & 0x4→0x10`). Never tripped on h_norm data (disjoint exemplar/sibling address spaces) but the
|
||||
h_seq imm engine (§46, T2a) needs it: build ONE `\b(alt|…)\b` regex over the full table (symbols ∪ self-rename ∪
|
||||
immediates), replace via a dict lookup on the match — each source token is matched once against the ORIGINAL text.
|
||||
This is also where the T2b **cross-address** self-rename (`func_<FROM>`→`func_<TO>`, definition + recursion) and the
|
||||
T2a immediate `imm_map` merge into one pass. `remap(addr, from_ov, to_ov, to_addr=None, imm_map=None)` — backward
|
||||
compatible (to_addr defaults to addr; the pre-26 same-address callers are byte-unchanged).
|
||||
|
||||
## §41 — The DEF-SIDE canonical-sig wall: mechanically banking a drafted giant past `conflicting types` (Phase 25 T5b batch-2, 2026-07-09; `tools/canon_sig_reconcile.py`, byte-proven on `func_8013B274`)
|
||||
|
||||
**The wall (dominant for GIANTS — ~universal, vs ~35% clean-bank for small fns):** a drafter writes an
|
||||
|
||||
@@ -11,7 +11,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families
|
||||
## Task checklist (effort per R7 · one commit per completed task, Drew pushes — R6)
|
||||
|
||||
- [x] **Task 0 — Bootstrap** `[xHigh]` — this file + harness task list (R28). *(completes with this commit)*
|
||||
- [ ] **Task 1 — Remap core: extended reloc tracker + single-pass substitution** `[xHigh]` — `family_remap.py`: hi propagates through addu-class index adds; `symbol_map(from_ov,from_addr,to_ov,to_addr)`; single-pass simultaneous substitution + self-rename + `imm_map` hook. norm_stream/h_norm UNTOUCHED. Record discovery → `docs/decision-log.md` (R31) + cookbook §40 addendum (R30). **V0:** 22/22 (`func_80141100`), 15/15 (`func_801407F4` vs splat .s), 49/49 kinds (952-family cross-addr pair). **V1:** 3 banked h_norm members re-derived via NEW path → replace-and-rebuild SHA unchanged → git checkout.
|
||||
- [x] **Task 1 — Remap core: extended reloc tracker + single-pass substitution** `[xHigh]` — `family_remap.py`: hi propagates through add/addu index adds; `symbol_map`/`remap` gained backward-compatible `to_addr=None` (cross-address) + `imm_map` hook; single-pass simultaneous substitution + self-rename. norm_stream/h_norm UNTOUCHED. Discovery → cookbook §40b (R30) + decision-log (R31). **V0 PASS** (`.run/v0_reloc.py`): 22/22 regression (`func_80141100` NEW==OLD), 15/15 fix (`func_801407F4` vs splat .s, recovers `D_80187B88/90/B0`), cross-addr symbol_map clean. **V1 PASS** (`.run/v1_regression.py`): 160 real h_norm pairs, 96 SAME, **0 lost**, differences are strict indexed-reloc improvements. *(committed)*
|
||||
- [ ] **Task 2 — `tools/family_hseq.py` + committed manifest** `[xHigh]` — full-frontier survey (tail + h_norm reach≥2); per family: members/nins/byte-weight/#addr/#ovs, tag {per-location|cross-address|scattered}, diff_class {PURE|IMM(+positions)|STRUCT-EXCLUDED} via shared classifier, matched-sibling count, `has_mid_jr` (§8 risk), exemplar pick (matched ▸ ov077 ▸ modal-addr), size band → `.run/family_hseq.json` + `docs/family-hseq.md`. Verify: 663/186/1.85M ±, 3 named families, classification totals, `progress.py --weighted` cross-check.
|
||||
- [ ] **Task 3 — Imm engine (T2a, 3 tiers) + cross-address delta (T2b)** `[xHigh]` — Tier 1 simultaneous value-replace (asm-side ambiguity check; C tokenizer; signed imm16/sa/shift-vs-multiply spellings); Tier 2 targeted probe (~1–6 values/family; match_one-pipeline recompile; nins+h_seq guards; FRAGILE); Tier 3 member-fail → agent queue. T2b: h_seq sibling discovery at any addr; self-rename; distinct "link-undef" logging (fallback: `config/symbols.ov_*.txt` append + re-extract, second pass). Verify: 25 matched-matched imm pairs A→B regression.
|
||||
- [ ] **Task 4 — `family_sweep --hseq` mode** `[xHigh]` — manifest-driven; member word-diff pre-filter (STRUCT skip); stage `.run/sweep/<ov>/`; existing two-phase stage→`harvest_verify` gate; `--only/--reconcile/--no-preclassify` carry over; 30-sec sibling-TU compile probe before ×N claims (pin-free, §42e).
|
||||
@@ -24,7 +24,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families
|
||||
- [ ] **Task 11 — Step-D residue map** `[xHigh]` — true singletons (~0.27M ins) + 5 behemoths → Phase-27 input doc. NO execution.
|
||||
- [ ] **Task 12 — PhaseEnd** `[Max — Tier 1; R27 prompt]` — P7 walk, milestone demo, gate 2, `PhaseEnd_Phase26.md`, worklog → `logs/Phase26.md` (R19), in-file recap (R25), decision-log current (R31).
|
||||
|
||||
## ▶ CURRENT TASK: Task 1 — Remap core (after Task-0 commit + the R27 toggle: Drew sets `/model opus` + `/effort xHigh`)
|
||||
## ▶ CURRENT TASK: Task 2 — `tools/family_hseq.py` + committed manifest (Opus/xHigh)
|
||||
|
||||
## Milestone (gate 2 — structural completion, per Drew)
|
||||
|
||||
@@ -47,4 +47,5 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag
|
||||
|
||||
## Log
|
||||
|
||||
- **2026-07-11 (session 1, Task 1):** Extended `reloc_targets` for the add/addu indexed-global idiom (the "reach-1 tail" was largely this tracker blind spot, not unique code — decision-log + cookbook §40b). `symbol_map`/`remap` gained backward-compatible `to_addr` (cross-address T2b) + `imm_map` hook; sequential→single-pass substitution (fixes the latent chained-rename bug). norm_stream/h_norm untouched. V0 (22/22 regression, 15/15 fix vs splat .s) + V1 (160 pairs, 0 regressions) both green. Committed.
|
||||
- **2026-07-11 (session 1, planning):** Phase Start (Tier 1, Fable5+Max+plan-mode). Megaplan analyzed; survey reproduced from sigs+src (R14) — fleet metrics match PhaseEnd_25 exactly; 186-not-986 substantial-family correction; #2 family is cross-address; 93 matched-sibling families found (0.51M ins). Tooling recon (Explore) + design pressure-test (Plan agent) → **the reloc-tracker blind-spot discovery** (addu-preserves-hi; 890×121 family is PURE reloc; 63-fam zero-crack corpus; sequential-substitution latent bug). Drew's gate-1 decisions: carried queue → end of phase; structural milestone. Plan approved; task list built (R28). Task 0 complete with this commit.
|
||||
|
||||
+66
-14
@@ -12,7 +12,15 @@ with the sibling's. Shared EXE/resident symbols map to themselves. The result is
|
||||
generated mechanically from ONE crack, for ~0 agent tokens. The whole-binary byte-gate stays the sole
|
||||
arbiter (G3/P9): a wrong remap is rejected.
|
||||
|
||||
tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--out draft.c]
|
||||
Phase-26 extends this to the looser h_seq family key (mnemonic skeleton, immediates may differ):
|
||||
* reloc_targets propagates a pending lui-hi through add/addu index arithmetic (the gcc-2.7.2
|
||||
indexed-global `lui;addu $idx;lw %lo($at)` idiom the pre-26 tracker dropped — the "reach-1 tail"
|
||||
was largely this blind spot, not unique code);
|
||||
* remap does a SINGLE-PASS simultaneous substitution (the old sequential re.sub corrupted chained /
|
||||
permuted maps) and merges an optional imm_map (T2a per-member immediate edits);
|
||||
* to_addr enables cross-address siblings (same engine fn at a different vram per overlay, T2b).
|
||||
|
||||
tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--to-addr 0xADDR2] [--out draft.c]
|
||||
"""
|
||||
import struct, json, glob, re, sys, argparse
|
||||
|
||||
@@ -37,7 +45,8 @@ def nins_of(ov, addr):
|
||||
|
||||
def reloc_targets(ov, addr):
|
||||
"""ordered [(kind, resolved_addr)] for jal targets + lui/lo address loads, in instruction order.
|
||||
Verified against splat .s ground truth (22/22 on func_80141100)."""
|
||||
Verified against splat .s ground truth (22/22 on func_80141100; 15/15 on func_801407F4 whose
|
||||
indexed-global D[i] accesses the pre-Phase-26 tracker missed — see the add/addu hi-propagation)."""
|
||||
data = open(img_path(ov), "rb").read()
|
||||
n = nins_of(ov, addr)
|
||||
off = addr - VRAM
|
||||
@@ -59,14 +68,32 @@ def reloc_targets(ov, addr):
|
||||
out.append(("data", pend[rs] + lo))
|
||||
del pend[rs]
|
||||
pend.pop((w >> 16) & 0x1F, None) # rt overwritten
|
||||
elif op == 0: # R-type: rd overwritten
|
||||
pend.pop((w >> 11) & 0x1F, None)
|
||||
elif op == 0: # R-type
|
||||
funct = w & 0x3F
|
||||
rd = (w >> 11) & 0x1F
|
||||
if funct in (0x20, 0x21): # add / addu: the indexed-global idiom
|
||||
# gcc-2.7.2 emits `lui $at,%hi(D); addu $at,$at,$idx; lw ..%lo(D)($at)` for D[i] —
|
||||
# the addu PRESERVES the hi anchor (index shifts the runtime value, not the symbol).
|
||||
# Propagate the pending hi through the add so the following %lo still resolves D.
|
||||
rs = (w >> 21) & 0x1F
|
||||
rt = (w >> 16) & 0x1F
|
||||
if rs in pend:
|
||||
pend[rd] = pend[rs]
|
||||
elif rt in pend:
|
||||
pend[rd] = pend[rt]
|
||||
else:
|
||||
pend.pop(rd, None)
|
||||
else:
|
||||
pend.pop(rd, None) # rd overwritten with a non-address value
|
||||
return out
|
||||
|
||||
|
||||
def symbol_map(addr, from_ov, to_ov):
|
||||
"""{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean."""
|
||||
ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, addr)
|
||||
def symbol_map(addr, from_ov, to_ov, to_addr=None):
|
||||
"""{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean.
|
||||
to_addr defaults to addr (same-address h_norm sibling); pass it for a cross-address (T2b) sibling."""
|
||||
if to_addr is None:
|
||||
to_addr = addr
|
||||
ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, to_addr)
|
||||
if len(ex) != len(tg):
|
||||
return None, f"reloc-count mismatch {len(ex)}!={len(tg)} (not an h_norm-clean pair)"
|
||||
m = {}
|
||||
@@ -104,16 +131,38 @@ def extract_unit(ov, addr):
|
||||
return None, None
|
||||
|
||||
|
||||
def remap(addr, from_ov, to_ov):
|
||||
"""returns (draft_text, symbol_map) or (None, error_str)."""
|
||||
m, err = symbol_map(addr, from_ov, to_ov)
|
||||
def apply_remap(unit, table):
|
||||
"""single-pass simultaneous substitution of {source_token: replacement} over unit — each source
|
||||
token is matched once against the ORIGINAL text, so chained/permuted maps (D_A->D_B, D_B->D_C, or
|
||||
an immediate value permutation 0x10->0xA & 0x4->0x10) are correct where the old sequential re.sub
|
||||
corrupted them. Tokens are word-bounded (`\\b`); longest-first avoids any prefix ambiguity."""
|
||||
if not table:
|
||||
return unit
|
||||
keys = sorted(table, key=len, reverse=True)
|
||||
rx = re.compile(r'\b(?:' + '|'.join(re.escape(k) for k in keys) + r')\b')
|
||||
return rx.sub(lambda mm: table[mm.group(0)], unit)
|
||||
|
||||
|
||||
def remap(addr, from_ov, to_ov, to_addr=None, imm_map=None):
|
||||
"""returns (draft_text, symbol_map) or (None, error_str).
|
||||
Same-overlay-address remap by default; pass to_addr for a cross-address (T2b) sibling — the self
|
||||
name func_<ADDR> is then remapped to func_<TO_ADDR> too. imm_map (T2a) merges per-member immediate
|
||||
/literal token substitutions into the SAME single pass. The returned map is the per-overlay symbol
|
||||
map only (self-rename + imm edits excluded), preserving the pre-Phase-26 contract."""
|
||||
if to_addr is None:
|
||||
to_addr = addr
|
||||
m, err = symbol_map(addr, from_ov, to_ov, to_addr)
|
||||
if err:
|
||||
return None, err
|
||||
unit, cf = extract_unit(from_ov, addr)
|
||||
if not unit:
|
||||
return None, f"no matched unit for func_{addr:08x} in {from_ov}"
|
||||
for src, dst in m.items():
|
||||
unit = re.sub(rf'\b{src}\b', dst, unit)
|
||||
table = dict(m)
|
||||
if addr != to_addr: # self-rename: definition + any recursion
|
||||
table[f"func_{addr:08X}"] = f"func_{to_addr:08X}"
|
||||
if imm_map:
|
||||
table.update(imm_map)
|
||||
unit = apply_remap(unit, table)
|
||||
return unit, m
|
||||
|
||||
|
||||
@@ -122,14 +171,17 @@ def main():
|
||||
ap.add_argument("--addr", required=True)
|
||||
ap.add_argument("--from", dest="frm", required=True)
|
||||
ap.add_argument("--to", required=True)
|
||||
ap.add_argument("--to-addr", dest="to_addr", default=None,
|
||||
help="sibling address if different from --addr (cross-address T2b family)")
|
||||
ap.add_argument("--out", default=None)
|
||||
a = ap.parse_args()
|
||||
addr = int(a.addr, 16)
|
||||
draft, m = remap(addr, a.frm, a.to)
|
||||
to_addr = int(a.to_addr, 16) if a.to_addr else None
|
||||
draft, m = remap(addr, a.frm, a.to, to_addr)
|
||||
if draft is None:
|
||||
print(f"REMAP FAIL (func_{addr:08x} {a.frm}->{a.to}): {m}")
|
||||
sys.exit(1)
|
||||
out = a.out or f".run/remap_{a.to}_{addr:08x}.c"
|
||||
out = a.out or f".run/remap_{a.to}_{(to_addr or addr):08x}.c"
|
||||
open(out, "w").write(draft + "\n")
|
||||
print(f"remapped {len(m)} per-overlay symbol(s) {a.frm}->{a.to}: {m}")
|
||||
print(f"-> {out}")
|
||||
|
||||
Reference in New Issue
Block a user