From 3ce81cbf56a112c8f75c91e094ec9d5c20a31633 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Sat, 11 Jul 2026 21:26:00 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-26):=20task=204=20=E2=80=94=20family?= =?UTF-8?q?=5Fsweep=20--hseq=20mode=20(cross-address=20+=20imm=20templatin?= =?UTF-8?q?g)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - additive hseq_sweep(): consumes .run/family_hseq.json, templates each matched-exemplar family's still-stubbed members via remap_hseq (reloc remap + imm subst + cross-address self-rename), stages by (overlay,split), gates each group once via harvest_verify. h_norm path byte-UNTOUCHED (new --hseq branch routes before it). Unique per-group verified-out fixes a latent multi-split overwrite. - member pre-filter = remap_hseq refusal (STRUCT regalloc-drift / unresolved immediates). - static pin guard: skip families whose matched exemplar carries a hard-reg pin __asm__("$N") — the x1-only cracks (decision-log 2026-07-11: func_8016DF5C/8013D9B0/80133AB0) that cc1-crash sibling TUs; they route to Task 7 pin-free re-crack instead of bisection-storming the gate. - flags: --hseq[=manifest] --band --min-members --stage-only --only. - VERIFIED (--stage-only): substantial band 29 families -> 1507 clean members staged / 267 groups, 1643 correctly skipped pinned (16/29 pinned); h_norm --only path intact; drafts byte-correct by construction (V3 0-DIFF), type-using -> whole-TU gate (Task 5). --- phase-ends/CURRENT_PHASE.md | 5 +- tools/family_sweep.py | 101 ++++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 2 deletions(-) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index fb9479274f..bc6e4a5372 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -14,7 +14,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families - [x] **Task 1 — Remap core: extended reloc tracker + single-pass substitution** `[xHigh]` — `family_remap.py`: hi propagates through add/addu index adds; `symbol_map`/`remap` gained backward-compatible `to_addr=None` (cross-address) + `imm_map` hook; single-pass simultaneous substitution + self-rename. norm_stream/h_norm UNTOUCHED. Discovery → cookbook §40b (R30) + decision-log (R31). **V0 PASS** (`.run/v0_reloc.py`): 22/22 regression (`func_80141100` NEW==OLD), 15/15 fix (`func_801407F4` vs splat .s, recovers `D_80187B88/90/B0`), cross-addr symbol_map clean. **V1 PASS** (`.run/v1_regression.py`): 160 real h_norm pairs, 96 SAME, **0 lost**, differences are strict indexed-reloc improvements. *(committed)* - [x] **Task 2 — `tools/family_hseq.py` + committed manifest** `[xHigh]` — full-frontier survey; shared word-diff classifier added to `family_remap.py` (`stream_words`/`reloc_indices`/`reg_fields`/`classify_member` — PURE/IMM/STRUCT, register-drift aware). → `.run/family_hseq.json` + `docs/family-hseq.md`. **VERIFIED:** fleet 74.8/58.2/30.3 (= PhaseEnd_25 & progress.py to the decimal); tail cross-check **663 families / 186 substantial / 1.847M ins** (exact); classification vs Plan-agent table **PURE-same 62, IMM 8 exact, PURE-cross 114≈103**; 890×134/562×134 PURE per-location + 952×113 #addr21 IMM confirmed; full frontier = 581 substantial families / 3.22M templatable ins, **345 matched-sibling PURE/IMM families / 1.14M ins = the V2/V3 zero-crack corpus**. *(committed)* - [x] **Task 3 — Imm engine (T2a Tier 1) + cross-address delta (T2b)** `[xHigh]` — `family_remap.imm_value`/`imm_map_tier1`/`remap_hseq`: diff-driven Tier-1 value-replace (asm-side ambiguity guard defers ambiguous values; C-literal swap preserving sign/hex-case) + T2b self-rename via the Task-1 `to_addr`. **Tier 2 (probe) DEFERRED** — build-if-needed per Task-5 measurement (byte-gate arbitrates; only ~8 low-weight IMM families). **VERIFIED** (`.run/v3_imm.py`, match_one reloc-masked): **0 DIFF** on every compilable pair — 3 IMM + 2 cross-address MATCH; 4 remap-fails were correctly-deferred asm-ambiguous values; 45 compile-fail(isolation) → whole-TU gate in Task 5. *(committed)* -- [ ] **Task 4 — `family_sweep --hseq` mode** `[xHigh]` — manifest-driven; member word-diff pre-filter (STRUCT skip); stage `.run/sweep//`; existing two-phase stage→`harvest_verify` gate; `--only/--reconcile/--no-preclassify` carry over; 30-sec sibling-TU compile probe before ×N claims (pin-free, §42e). +- [x] **Task 4 — `family_sweep --hseq` mode** `[xHigh]` — additive `hseq_sweep()` (h_norm path byte-untouched): consumes `family_hseq.json`, templates matched-exemplar members via `remap_hseq` (cross-address + imm), stages `.run/sweep//func_.c` grouped by (overlay,split), gates via `harvest_verify` (unique per-group verified-out — fixes a latent multi-split overwrite). Member pre-filter = `remap_hseq` refusal (STRUCT/unresolved). **Static pin guard** (`__asm__("$N"` → skip family; §42e — the ×1-only pinned cracks that cc1-crash sibling TUs). `--band`/`--min-members`/`--stage-only`/`--only`. **VERIFIED:** substantial → 29 families, 1507 clean members staged / 267 groups, 1643 correctly skipped as pinned (16/29 families pinned incl. the decision-log-named `func_8016DF5C`/`8013D9B0`/`80133AB0`); h_norm path intact; drafts byte-correct by construction (V3), type-using → whole-TU gate (Task 5). *(committed)* - [ ] **Task 5 — Zero-crack validation harvests (V2/V3) — GO/NO-GO** `[xHigh; UC if fan-out — R26 prompt]` — V2: the 63-fam/0.31M-ins tracker-miss corpus (incl. `0x801407f4`/`0x8015d5e8`/`0x8015f118` ×118) → real banks + measured success rate + failure taxonomy. V3: 12 imm families with matched exemplars (`0x8016b448` ×134, `0x8017dde8` ×108). Scale NOTHING until rates are on the table; poor → fix tooling or assisted-template fallback. - [ ] **Task 6 — Step-A triage of the 186 substantial exemplars** `[Ultracode — R26/R27 prompt + WAIT]` — ranked cheap-draft + `rtu_match` + §31/§44 classing → cheap-crackable / permuter / Fable5-necessary lists; 15-ins mega-family collision screen. Pilots: 102×4, 265×7, then the 952×100 cross-addr flagship. - [ ] **Task 7 — Family-core crack waves (the Fable5 window)** `[launch at Max — R27; agents model:fable, parallel-isolated, distill-between]` — biggest byte-weight first (890/562/536/493…); §45-templated prompts (read `.greg`/`.lreg` first; Lever-A merged-accumulator audit; 1-death in-out-asm; /s; gdb-on-cc1, `tools/reference/gcc-2.7.2/`); PIN-FREE (×134); `harvest_verify` arbitrates; distill each crack → cookbook §46+ (R30); cheap-Opus applies banked idioms between waves; cheap/permuter lists run in parallel `[UC]`. ~375k tok/crack guide; documented wall + levers = valid outcome (P9). @@ -24,7 +24,7 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families - [ ] **Task 11 — Step-D residue map** `[xHigh]` — true singletons (~0.27M ins) + 5 behemoths → Phase-27 input doc. NO execution. - [ ] **Task 12 — PhaseEnd** `[Max — Tier 1; R27 prompt]` — P7 walk, milestone demo, gate 2, `PhaseEnd_Phase26.md`, worklog → `logs/Phase26.md` (R19), in-file recap (R25), decision-log current (R31). -## ▶ CURRENT TASK: Task 4 — `family_sweep --hseq` mode (Opus/xHigh) +## ▶ CURRENT TASK: Task 5 — Zero-crack validation harvests (V2/V3) — the GO/NO-GO gate (Opus/xHigh; prompt Ultracode if the gate becomes a breadth wave — R27) ## Milestone (gate 2 — structural completion, per Drew) @@ -47,6 +47,7 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-11 (session 1, Task 4):** Added `family_sweep --hseq` (additive `hseq_sweep()`, h_norm path byte-untouched): manifest-driven templating via `remap_hseq`, unique per-group verified-out, static `__asm__("$N"` pin guard (skips the ×1-only pinned cracks → Task 7). Dry-run (`--stage-only`): substantial band stages 1507 clean / 267 groups, 1643 correctly skipped as pinned (16/29 families pinned, incl. the decision-log-named ones). Drafts byte-correct by construction; whole-TU gate = Task 5. Committed. - **2026-07-11 (session 1, Task 3):** Built the T2a imm engine (`imm_value`/`imm_map_tier1`/`remap_hseq`) — diff-driven Tier-1 literal swap with an asm-side ambiguity guard — + wired T2b cross-address self-rename. Tier-2 probe deferred (build-if-needed; byte-gate arbitrates). V3 (`.run/v3_imm.py`): 0 DIFF on all compilable pairs (5 MATCH), ambiguity guard + cross-address self-rename confirmed; comprehensive whole-TU gate validation is Task 5. Committed. - **2026-07-11 (session 1, Task 2):** Built `tools/family_hseq.py` (full-frontier h_seq survey) + the shared word-diff classifier in `family_remap.py` (PURE/IMM/STRUCT, register-drift aware). Reproduces the fleet metrics + the 663/186/1.85M tail cross-check exactly; classification matches the Plan-agent table; 581 substantial target families / 3.22M templatable ins, 345 matched-sibling families / 1.14M ins = the zero-crack corpus. `docs/family-hseq.md` committed. Committed. - **2026-07-11 (session 1, Task 1):** Extended `reloc_targets` for the add/addu indexed-global idiom (the "reach-1 tail" was largely this tracker blind spot, not unique code — decision-log + cookbook §40b). `symbol_map`/`remap` gained backward-compatible `to_addr` (cross-address T2b) + `imm_map` hook; sequential→single-pass substitution (fixes the latent chained-rename bug). norm_stream/h_norm untouched. V0 (22/22 regression, 15/15 fix vs splat .s) + V1 (160 pairs, 0 regressions) both green. Committed. diff --git a/tools/family_sweep.py b/tools/family_sweep.py index 1fce6a3757..3abbd7849c 100644 --- a/tools/family_sweep.py +++ b/tools/family_sweep.py @@ -165,6 +165,95 @@ def edit_remap_sweep(a, sig, src_sig, stubs): print(json.dumps({"banked": nb, "failed": nf, "skipped": dict(skipped)})) +def hseq_sweep(a): + """Phase-26 T3 --hseq: template h_seq FAMILIES (looser than h_norm — members may sit at DIFFERENT + addresses per overlay and differ in a few immediates). Per family in `.run/family_hseq.json` with a + MATCHED exemplar (its C exists), template each still-stubbed member via `family_remap.remap_hseq` + (reloc symbol remap §40b + immediate substitution T2a + cross-address self-rename T2b), stage grouped + by (overlay, split), and gate each group ONCE via plain harvest_verify (the sole arbiter, G3/P9). + remap_hseq REFUSES register-drift (STRUCT) and unresolved-immediate members — that IS the member + pre-filter, so a chunk never bisects on a hopeless draft. Pin-free templates only (§42e): a matched + exemplar carrying hard-reg pins would cc1-crash a sibling TU → that group fails+reverts at the gate.""" + manifest = json.load(open(os.path.join(REPO, a.hseq))) + stubs = {ov: stub_map(ov) for ov in + [os.path.basename(p).split("sig.")[1][:-6] for p in + sorted(glob.glob(os.path.join(REPO, ".run/sig.ov_*.jsonl")))]} + only = set(int(x, 16) for x in a.only.split(",")) if a.only else None + bands = None if a.band == "all" else set(a.band.split(",")) + + fams = [f for f in manifest["families"] if f["exemplar"]["kind"] in ("matched", "matched-ov077")] + if bands: + fams = [f for f in fams if f["band"] in bands] + if only is not None: + fams = [f for f in fams if int(f["exemplar"]["addr"], 16) in only] + if a.min_members > 1: + fams = [f for f in fams if f["n_members"] >= a.min_members] + fams.sort(key=lambda f: -f["byte_weight_templatable"]) + if a.limit: + fams = fams[:a.limit] + print(f"[hseq] {len(fams)} matched-exemplar families (band={a.band}); " + f"{sum(f['n_members'] for f in fams)} candidate members") + + # ---- phase 1: template every (exemplar -> still-stubbed member), stage by (overlay, split, subdir) + shutil.rmtree(os.path.join(REPO, SWEEP), ignore_errors=True) + groups = collections.defaultdict(list) # (ov, src_rel, subdir) -> [fn] + skip = collections.Counter() + for f in fams: + exov, exaddr = f["exemplar"]["ov"], int(f["exemplar"]["addr"], 16) + for ov, addr_s in f["members"]: + to_addr = int(addr_s, 16) + if (ov, to_addr) == (exov, exaddr): + continue + if to_addr not in stubs.get(ov, {}): # already matched / not a stub now + skip["not-stub"] += 1; continue + src_rel, subdir = stubs[ov][to_addr] + draft, info = FR.remap_hseq(exaddr, exov, ov, to_addr) + if draft is None: + r = info.split(":")[0] if isinstance(info, str) else "skip" + skip[r[:24]] += 1; continue + if re.search(r'__asm__\s*\(\s*"\$', draft): # hard-reg pin (§42e): ×1-only, cc1-crashes + skip["pinned-exemplar"] += 1; continue # sibling TUs → skip the family, don't bisect-storm + d = os.path.join(REPO, SWEEP, ov) + os.makedirs(d, exist_ok=True) + open(os.path.join(d, f"func_{to_addr:08X}.c"), "w").write(draft + "\n") + groups[(ov, src_rel, subdir)].append(f"func_{to_addr:08X}") + print(f"[hseq] staged {sum(len(v) for v in groups.values())} member drafts across {len(groups)} " + f"(overlay,split) groups; skipped {dict(skip)}") + if a.stage_only: + print(json.dumps({"families": len(fams), "staged": sum(len(v) for v in groups.values()), + "groups": len(groups), "skipped": dict(skip)})) + return + if not groups: + print(json.dumps({"families": len(fams), "banked": 0, "failed": 0, "skipped": dict(skip)})) + return + + # ---- phase 2: gate each group ONCE (unique verified-out per group so multi-split overlays sum right) + banked = collections.Counter() + failed = collections.Counter() + for gi, ((ov, src_rel, subdir), fns) in enumerate(sorted(groups.items())): + good_sha = open(os.path.join(REPO, f"config/check.{ov}.sha")).read().split()[0] + vout = f".run/hseq_verified.{ov}.{gi}.txt" + sh([PY, "tools/harvest_verify.py", "--binary", ov, "--src", src_rel, "--asm-subdir", subdir, + "--out", f"build/{ov}/{ov}", "--good-sha", good_sha, + "--drafts", os.path.join(SWEEP, ov), "--chunk", str(a.chunk), + "--verified-out", vout, "--failed-out", f".run/hseq_failed.{ov}.{gi}.txt"], timeout=3600) + vpath = os.path.join(REPO, vout) + nver = len([x for x in open(vpath).read().split() if x]) if os.path.exists(vpath) else 0 + banked[ov] += nver + failed[ov] += len(fns) - nver + print(f" {ov} [{os.path.basename(src_rel)}]: {nver}/{len(fns)} banked") + + nb, nf = sum(banked.values()), sum(failed.values()) + print(f"\n[hseq] BANKED {nb} member-matches / {nf} failed across {len(banked)} overlays; " + f"skipped {dict(skip)}") + if a.commit and nb: + sh(["git", "add", "-A", "src"]) + sh(["git", "commit", "-q", "-m", + f"feat(phase-26): h_seq family sweep — {nb} member-matches banked via remap_hseq"]) + print("[hseq] committed.") + print(json.dumps({"families": len(fams), "banked": nb, "failed": nf, "skipped": dict(skip)})) + + def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--source", default="ov_SC01_077") @@ -187,9 +276,21 @@ def main(): "so it false-negatives type-lifted families); route every remappable exemplar straight " "to the harvest_verify byte-gate (the real TU sees the shared types). Use after a " "build_engine_types type-lift for the decl-reconcile pass.") + ap.add_argument("--hseq", nargs="?", const=".run/family_hseq.json", default=None, metavar="MANIFEST", + help="Phase-26 T3: template h_seq FAMILIES from the family_hseq manifest (cross-address + " + "immediate-substituting via remap_hseq), instead of the same-address h_norm path.") + ap.add_argument("--band", default="substantial", + help="--hseq: family size band(s) to sweep — substantial|mid|tiny|all or a comma list") + ap.add_argument("--min-members", type=int, default=1, help="--hseq: skip families with fewer members") + ap.add_argument("--stage-only", action="store_true", + help="--hseq: stage the templated drafts to .run/sweep/ and stop before the byte-gate " + "(dry-run for inspection; no builds)") a = ap.parse_args() os.chdir(REPO) + if a.hseq: + return hseq_sweep(a) + sig = load_sigs() src_sig = sig[a.source] src_stubs = stub_map(a.source)