diff --git a/docs/decision-log.md b/docs/decision-log.md index f02752603..6340a942d 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -260,3 +260,31 @@ sibling TU?". A hand crack that banked in ov077 by exotic register pins does not in ITS TU only; the ×134 claim needs a sibling-TU compile probe, because pins are TU-context-specific and cc1 *crashes* (not just drifts) on the ones that don't transfer. Corollary: rtu_match/match_one are blind here — their neutralized/ isolation compiles crash too (harness artifact); only the real `make build` is the arbiter. + +## 2026-07-11 · Phase 26 — the "reach-1 tail" is largely a reloc-tracker blind spot, not unique code (Task 1) + +**Context + belief (from the Phase-25 close):** the h_seq reframe had already shown the "36k unique tail" collapses +~90% into per-location families. The open question entering Phase 26 was HOW the families differ — the megaplan framed +immediate-substitution as the central new problem (families "differ in immediates, so are NOT free dedup"). + +**What the design pressure-test found (byte-verified before any scaling — R14):** the dominant difference is NOT +immediates — it is a **tracker blind spot**. `norm_stream`/`reloc_targets` dropped the lui-hi on every R-type write, +but gcc-2.7.2's indexed-global idiom `lui;addu $idx;lw %lo($at)` preserves it. So `D[i]`-indexing functions were +*mis-normalized per overlay* → they inflated the "h_norm reach-1 tail," and `family_remap` silently dropped their +indexed `D_` symbols → those families couldn't bank even though they are pure per-location templates. On the +substantial tail the classification is **PURE-same-addr 62 fams / 1.55M ins · PURE-cross-addr 103 / 0.10M · genuine +IMM only 8 / 0.10M** — i.e. ~95% of the byte-weight is reloc-only, fixable by a ≤15-LOC tracker change, and the +immediate engine shrinks to an escalation tier for ~8 families. A second latent bug surfaced alongside: `remap`'s +sequential substitution corrupts chained/permuted maps (harmless on h_norm, breaks the imm engine). + +**The pivot:** front-load the tracker fix (Task 1) as the load-bearing change, demote the immediate engine to a +diff-driven 3-tier escalation (Task 3), and add a **free validation corpus** — 63 families / 0.31M ins already have a +MATCHED exemplar and only failed earlier sweeps from this bug → they bank with zero cracking the moment the fix lands +(Task 5 V2), simultaneously measuring the real template success rate before any Fable5 spend. + +**Better path (hindsight):** the tracker's own design note already said "conservative: can miss a match, never forge +one" — but a *missed* reloc in a REMAP tool isn't harmless the way a missed h_norm match is; it silently produces a +wrong-but-compiling sibling body that only the byte-gate catches. When a normalization/remap tool is REUSED for code +generation (not just clustering), its conservative-miss becomes a correctness bug. **LESSON (R14):** before treating a +"unique/unmatchable" population as intrinsic, re-run the *grouping and the remap* under a corrected fingerprint — +here the "reach-1 tail" and the "unremappable family" were the SAME artifact of one dropped register-tracking case. diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index b19a40747..ffd7f21af 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -2310,6 +2310,48 @@ includes (`src/shared/engine_types.h` via `engine_core.h`), then re-sweep. **+1, at this cheap ~30s gate, before any expensive sweep. Then the full R22 clean-fleet 136/136 confirms no fleet-wide header collision. Tools: `build_engine_types.py --file/--exclude`, `family_sweep.py --no-preclassify`. +### §40b — The reloc-tracker blind spot: the indexed-global idiom that hid the "reach-1 tail" (Phase 26 Task 1, 2026-07-11, byte-verified V0/V1) + +**The discovery (why the "36k unique tail" was largely a measurement artifact).** Both `norm_stream` +(`sig_image.py`) and `reloc_targets` (`family_remap.py`) tracked lui-hi/lo pairs but **popped the pending hi on +ANY R-type write** (`pend.pop(rd)`). gcc-2.7.2's indexed-global access `D[i]` compiles to +`lui $at,%hi(D); addu $at,$at,$idx; lw $v1,%lo(D)($at)` — the **`addu` PRESERVES the hi anchor** (the index shifts +the runtime value, not the symbol). So every function that indexes a per-overlay global array left its `%lo` fields +**raw** in `h_norm` → the function normalized DIFFERENTLY per overlay → it looked **fleet-unique (h_norm reach-1)** +when it is actually a per-location family, AND `family_remap`'s symbol map dropped those indexed `D_` symbols → +the sibling draft kept the exemplar's array name → **byte-gate fail** (an earlier "unremappable" wall). + +**The fix (≤15 LOC, `reloc_targets` R-type branch only — NOT `norm_stream`).** On `add`/`addu` (funct 0x20/0x21), +propagate the pending hi to `rd` when a source reg holds one, else pop: +```python +elif op == 0: # R-type + funct = w & 0x3F; rd = (w >> 11) & 0x1F + if funct in (0x20, 0x21): # add/addu: address arithmetic preserves the hi anchor + rs, rt = (w>>21)&0x1F, (w>>16)&0x1F + if rs in pend: pend[rd] = pend[rs] + elif rt in pend: pend[rd] = pend[rt] + else: pend.pop(rd, None) + else: pend.pop(rd, None) +``` +The `%lo` resolution is unchanged (`pend[rs] + signext(lo)` = the symbol; the index is a runtime reg). **Leave +`sig_image.norm_stream` / `h_norm` UNTOUCHED** — the fleet metrics, the proven h_norm sweep, and the manifest all +depend on its stable (blind) hashing; the h_seq family key (mnemonic skeleton) is unaffected by the tracker, so +families still cluster correctly, and the fix only makes the SYMBOL PAIRING correct so the remapped body byte-matches. + +**Verified (build-free, V0/V1 `2026-07-11`):** `func_801407F4` resolves **15/15 relocs vs splat `.s`** (pre-fix: 10), +recovering the indexed arrays `D_80187B88/90/B0`; `func_80141100` (no idiom) stays **22/22 identical** (zero regression); +across 160 real h_norm sibling pairs the new `remap` output is byte-identical to the committed pre-fix output (96 SAME, +**0 lost**), differing only where it strictly recovers indexed relocs. Reproduce: `.run/v0_reloc.py`, `.run/v1_regression.py`. + +**Companion fix — single-pass simultaneous substitution.** The old `remap` applied renames **sequentially** +(`for src,dst: re.sub`), which corrupts a chained/permuted map (`D_A→D_B` then `D_B→D_C`, or an immediate value +permutation `0x10→0xA & 0x4→0x10`). Never tripped on h_norm data (disjoint exemplar/sibling address spaces) but the +h_seq imm engine (§46, T2a) needs it: build ONE `\b(alt|…)\b` regex over the full table (symbols ∪ self-rename ∪ +immediates), replace via a dict lookup on the match — each source token is matched once against the ORIGINAL text. +This is also where the T2b **cross-address** self-rename (`func_`→`func_`, definition + recursion) and the +T2a immediate `imm_map` merge into one pass. `remap(addr, from_ov, to_ov, to_addr=None, imm_map=None)` — backward +compatible (to_addr defaults to addr; the pre-26 same-address callers are byte-unchanged). + ## §41 — The DEF-SIDE canonical-sig wall: mechanically banking a drafted giant past `conflicting types` (Phase 25 T5b batch-2, 2026-07-09; `tools/canon_sig_reconcile.py`, byte-proven on `func_8013B274`) **The wall (dominant for GIANTS — ~universal, vs ~35% clean-bank for small fns):** a drafter writes an diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 83fa1b700..8a038a8a1 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -11,7 +11,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families ## Task checklist (effort per R7 · one commit per completed task, Drew pushes — R6) - [x] **Task 0 — Bootstrap** `[xHigh]` — this file + harness task list (R28). *(completes with this commit)* -- [ ] **Task 1 — Remap core: extended reloc tracker + single-pass substitution** `[xHigh]` — `family_remap.py`: hi propagates through addu-class index adds; `symbol_map(from_ov,from_addr,to_ov,to_addr)`; single-pass simultaneous substitution + self-rename + `imm_map` hook. norm_stream/h_norm UNTOUCHED. Record discovery → `docs/decision-log.md` (R31) + cookbook §40 addendum (R30). **V0:** 22/22 (`func_80141100`), 15/15 (`func_801407F4` vs splat .s), 49/49 kinds (952-family cross-addr pair). **V1:** 3 banked h_norm members re-derived via NEW path → replace-and-rebuild SHA unchanged → git checkout. +- [x] **Task 1 — Remap core: extended reloc tracker + single-pass substitution** `[xHigh]` — `family_remap.py`: hi propagates through add/addu index adds; `symbol_map`/`remap` gained backward-compatible `to_addr=None` (cross-address) + `imm_map` hook; single-pass simultaneous substitution + self-rename. norm_stream/h_norm UNTOUCHED. Discovery → cookbook §40b (R30) + decision-log (R31). **V0 PASS** (`.run/v0_reloc.py`): 22/22 regression (`func_80141100` NEW==OLD), 15/15 fix (`func_801407F4` vs splat .s, recovers `D_80187B88/90/B0`), cross-addr symbol_map clean. **V1 PASS** (`.run/v1_regression.py`): 160 real h_norm pairs, 96 SAME, **0 lost**, differences are strict indexed-reloc improvements. *(committed)* - [ ] **Task 2 — `tools/family_hseq.py` + committed manifest** `[xHigh]` — full-frontier survey (tail + h_norm reach≥2); per family: members/nins/byte-weight/#addr/#ovs, tag {per-location|cross-address|scattered}, diff_class {PURE|IMM(+positions)|STRUCT-EXCLUDED} via shared classifier, matched-sibling count, `has_mid_jr` (§8 risk), exemplar pick (matched ▸ ov077 ▸ modal-addr), size band → `.run/family_hseq.json` + `docs/family-hseq.md`. Verify: 663/186/1.85M ±, 3 named families, classification totals, `progress.py --weighted` cross-check. - [ ] **Task 3 — Imm engine (T2a, 3 tiers) + cross-address delta (T2b)** `[xHigh]` — Tier 1 simultaneous value-replace (asm-side ambiguity check; C tokenizer; signed imm16/sa/shift-vs-multiply spellings); Tier 2 targeted probe (~1–6 values/family; match_one-pipeline recompile; nins+h_seq guards; FRAGILE); Tier 3 member-fail → agent queue. T2b: h_seq sibling discovery at any addr; self-rename; distinct "link-undef" logging (fallback: `config/symbols.ov_*.txt` append + re-extract, second pass). Verify: 25 matched-matched imm pairs A→B regression. - [ ] **Task 4 — `family_sweep --hseq` mode** `[xHigh]` — manifest-driven; member word-diff pre-filter (STRUCT skip); stage `.run/sweep//`; existing two-phase stage→`harvest_verify` gate; `--only/--reconcile/--no-preclassify` carry over; 30-sec sibling-TU compile probe before ×N claims (pin-free, §42e). @@ -24,7 +24,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families - [ ] **Task 11 — Step-D residue map** `[xHigh]` — true singletons (~0.27M ins) + 5 behemoths → Phase-27 input doc. NO execution. - [ ] **Task 12 — PhaseEnd** `[Max — Tier 1; R27 prompt]` — P7 walk, milestone demo, gate 2, `PhaseEnd_Phase26.md`, worklog → `logs/Phase26.md` (R19), in-file recap (R25), decision-log current (R31). -## ▶ CURRENT TASK: Task 1 — Remap core (after Task-0 commit + the R27 toggle: Drew sets `/model opus` + `/effort xHigh`) +## ▶ CURRENT TASK: Task 2 — `tools/family_hseq.py` + committed manifest (Opus/xHigh) ## Milestone (gate 2 — structural completion, per Drew) @@ -47,4 +47,5 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-11 (session 1, Task 1):** Extended `reloc_targets` for the add/addu indexed-global idiom (the "reach-1 tail" was largely this tracker blind spot, not unique code — decision-log + cookbook §40b). `symbol_map`/`remap` gained backward-compatible `to_addr` (cross-address T2b) + `imm_map` hook; sequential→single-pass substitution (fixes the latent chained-rename bug). norm_stream/h_norm untouched. V0 (22/22 regression, 15/15 fix vs splat .s) + V1 (160 pairs, 0 regressions) both green. Committed. - **2026-07-11 (session 1, planning):** Phase Start (Tier 1, Fable5+Max+plan-mode). Megaplan analyzed; survey reproduced from sigs+src (R14) — fleet metrics match PhaseEnd_25 exactly; 186-not-986 substantial-family correction; #2 family is cross-address; 93 matched-sibling families found (0.51M ins). Tooling recon (Explore) + design pressure-test (Plan agent) → **the reloc-tracker blind-spot discovery** (addu-preserves-hi; 890×121 family is PURE reloc; 63-fam zero-crack corpus; sequential-substitution latent bug). Drew's gate-1 decisions: carried queue → end of phase; structural milestone. Plan approved; task list built (R28). Task 0 complete with this commit. diff --git a/tools/family_remap.py b/tools/family_remap.py index 0a6129010..552dd8fb3 100644 --- a/tools/family_remap.py +++ b/tools/family_remap.py @@ -12,7 +12,15 @@ with the sibling's. Shared EXE/resident symbols map to themselves. The result is generated mechanically from ONE crack, for ~0 agent tokens. The whole-binary byte-gate stays the sole arbiter (G3/P9): a wrong remap is rejected. - tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--out draft.c] +Phase-26 extends this to the looser h_seq family key (mnemonic skeleton, immediates may differ): + * reloc_targets propagates a pending lui-hi through add/addu index arithmetic (the gcc-2.7.2 + indexed-global `lui;addu $idx;lw %lo($at)` idiom the pre-26 tracker dropped — the "reach-1 tail" + was largely this blind spot, not unique code); + * remap does a SINGLE-PASS simultaneous substitution (the old sequential re.sub corrupted chained / + permuted maps) and merges an optional imm_map (T2a per-member immediate edits); + * to_addr enables cross-address siblings (same engine fn at a different vram per overlay, T2b). + + tools/family_remap.py --addr 0xADDR --from ov_SC01_077 --to ov_SC01_000 [--to-addr 0xADDR2] [--out draft.c] """ import struct, json, glob, re, sys, argparse @@ -37,7 +45,8 @@ def nins_of(ov, addr): def reloc_targets(ov, addr): """ordered [(kind, resolved_addr)] for jal targets + lui/lo address loads, in instruction order. - Verified against splat .s ground truth (22/22 on func_80141100).""" + Verified against splat .s ground truth (22/22 on func_80141100; 15/15 on func_801407F4 whose + indexed-global D[i] accesses the pre-Phase-26 tracker missed — see the add/addu hi-propagation).""" data = open(img_path(ov), "rb").read() n = nins_of(ov, addr) off = addr - VRAM @@ -59,14 +68,32 @@ def reloc_targets(ov, addr): out.append(("data", pend[rs] + lo)) del pend[rs] pend.pop((w >> 16) & 0x1F, None) # rt overwritten - elif op == 0: # R-type: rd overwritten - pend.pop((w >> 11) & 0x1F, None) + elif op == 0: # R-type + funct = w & 0x3F + rd = (w >> 11) & 0x1F + if funct in (0x20, 0x21): # add / addu: the indexed-global idiom + # gcc-2.7.2 emits `lui $at,%hi(D); addu $at,$at,$idx; lw ..%lo(D)($at)` for D[i] — + # the addu PRESERVES the hi anchor (index shifts the runtime value, not the symbol). + # Propagate the pending hi through the add so the following %lo still resolves D. + rs = (w >> 21) & 0x1F + rt = (w >> 16) & 0x1F + if rs in pend: + pend[rd] = pend[rs] + elif rt in pend: + pend[rd] = pend[rt] + else: + pend.pop(rd, None) + else: + pend.pop(rd, None) # rd overwritten with a non-address value return out -def symbol_map(addr, from_ov, to_ov): - """{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean.""" - ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, addr) +def symbol_map(addr, from_ov, to_ov, to_addr=None): + """{exemplar_name: sibling_name} for the per-overlay symbols (positional zip). None,err if not clean. + to_addr defaults to addr (same-address h_norm sibling); pass it for a cross-address (T2b) sibling.""" + if to_addr is None: + to_addr = addr + ex, tg = reloc_targets(from_ov, addr), reloc_targets(to_ov, to_addr) if len(ex) != len(tg): return None, f"reloc-count mismatch {len(ex)}!={len(tg)} (not an h_norm-clean pair)" m = {} @@ -104,16 +131,38 @@ def extract_unit(ov, addr): return None, None -def remap(addr, from_ov, to_ov): - """returns (draft_text, symbol_map) or (None, error_str).""" - m, err = symbol_map(addr, from_ov, to_ov) +def apply_remap(unit, table): + """single-pass simultaneous substitution of {source_token: replacement} over unit — each source + token is matched once against the ORIGINAL text, so chained/permuted maps (D_A->D_B, D_B->D_C, or + an immediate value permutation 0x10->0xA & 0x4->0x10) are correct where the old sequential re.sub + corrupted them. Tokens are word-bounded (`\\b`); longest-first avoids any prefix ambiguity.""" + if not table: + return unit + keys = sorted(table, key=len, reverse=True) + rx = re.compile(r'\b(?:' + '|'.join(re.escape(k) for k in keys) + r')\b') + return rx.sub(lambda mm: table[mm.group(0)], unit) + + +def remap(addr, from_ov, to_ov, to_addr=None, imm_map=None): + """returns (draft_text, symbol_map) or (None, error_str). + Same-overlay-address remap by default; pass to_addr for a cross-address (T2b) sibling — the self + name func_ is then remapped to func_ too. imm_map (T2a) merges per-member immediate + /literal token substitutions into the SAME single pass. The returned map is the per-overlay symbol + map only (self-rename + imm edits excluded), preserving the pre-Phase-26 contract.""" + if to_addr is None: + to_addr = addr + m, err = symbol_map(addr, from_ov, to_ov, to_addr) if err: return None, err unit, cf = extract_unit(from_ov, addr) if not unit: return None, f"no matched unit for func_{addr:08x} in {from_ov}" - for src, dst in m.items(): - unit = re.sub(rf'\b{src}\b', dst, unit) + table = dict(m) + if addr != to_addr: # self-rename: definition + any recursion + table[f"func_{addr:08X}"] = f"func_{to_addr:08X}" + if imm_map: + table.update(imm_map) + unit = apply_remap(unit, table) return unit, m @@ -122,14 +171,17 @@ def main(): ap.add_argument("--addr", required=True) ap.add_argument("--from", dest="frm", required=True) ap.add_argument("--to", required=True) + ap.add_argument("--to-addr", dest="to_addr", default=None, + help="sibling address if different from --addr (cross-address T2b family)") ap.add_argument("--out", default=None) a = ap.parse_args() addr = int(a.addr, 16) - draft, m = remap(addr, a.frm, a.to) + to_addr = int(a.to_addr, 16) if a.to_addr else None + draft, m = remap(addr, a.frm, a.to, to_addr) if draft is None: print(f"REMAP FAIL (func_{addr:08x} {a.frm}->{a.to}): {m}") sys.exit(1) - out = a.out or f".run/remap_{a.to}_{addr:08x}.c" + out = a.out or f".run/remap_{a.to}_{(to_addr or addr):08x}.c" open(out, "w").write(draft + "\n") print(f"remapped {len(m)} per-overlay symbol(s) {a.frm}->{a.to}: {m}") print(f"-> {out}")