diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index cbc6bf26b..5f02ff6d9 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -9,7 +9,48 @@ --- -## ★★★ SESSION 2026-06-24 (cont. 4) — the `_a` vein IS a ×134 lever (10/11); cont.3 §24 pessimism CORRECTED (READ FIRST) +## ★★★ SESSION 2026-06-24 (cont. 5) — Phase 21 is now a NEVERENDING idiom-learning loop (READ FIRST) + +> **Drew's directive (2026-06-24): Phase 21 does NOT end until he says it does — assume neverending.** +> The model: harvest the EASY reach-134 fuel until ~80% consumed (a WALL), then **LEARN the next +> gcc-quirk IDIOM** that unlocks the highest-leverage residual, hand off to a **FRESH session** (clean +> context) with a `/loop` prompt to crack + harvest it, and **repeat forever → toward 100% decomp**. +> Each session ≈ one idiom. The vote is always: *target new gcc quirks, learn idioms, 100% the game.* + +**Fleet ~62.98%** (wave-4 in flight) · 136/136 byte-identical · 0 NON_MATCHING · grinder running (token-free). +The `_a` reach-134 EASY fuel is ~exhausted (waves 1–4 this session: 62.31 → 62.98%, +0.67%) — **we are AT +the first idiom wall.** + +**★ THE META-LOOP TOOL (new, committed): `tools/idiom_loop.py`** +- `--fuel-status` — rough wall indicator (counts ALL fresh cached incl. reach-1; the *precise* reach-134 + wall = `wave_targets --region a/main` returns a small / re-attempt-heavy batch, as now: 24→24→20→13). +- `--assess` — clusters the backlog near-misses by gcc-quirk class, ranks by reach-134 leverage, names + (a) the next IDIOM to learn + (b) the cheap deterministic recovery lever. +- `--gen-fuel [--out f]` — writes the research+harvest fuel manifest (closest-first) for a session. + +**★ FIRST ASSESSMENT (`--assess`, byte-backed from the 316-record backlog):** +- **NEXT IDIOM TO LEARN → `schedule`** (49 reach-134 fns, median 13 ins off): the gcc-2.7.2 + **instruction-scheduler (sched.c) load/store tie-break** — drafters consistently stub it "unsteerable" + after pins+barriers. Fuel: `.run/auto/idiom_fuel_schedule.json` (68 fns). Crack it on a **low-but-NONZERO** + closeness exemplar (e.g. func_80128ED8 / func_80131A34, close=3 — a single misplaced copy/load-order), + research sched.c (R17, `tools/reference/gcc-papermario`), distill to the cookbook, then re-wave ×134. +- **DETERMINISTIC RECOVERY LEVER → 64 close=0 reach-134 fns** (byte-correct, declaration-blocked = the `_a` + sig_unify-skip casualties + main plumbing). A def-side recovery (make `sig_unify` `--src-file`-aware so it + doesn't drop `_a` drafts, OR cast the def) banks them ×134 for ~0 agent tokens. **DO THIS FIRST each cycle.** + +**★ THE NEVERENDING CYCLE (every fresh session = the `/loop`):** +1. `idiom_loop.py --assess` → confirm the target idiom + the close=0 recovery count. +2. **Cheap harvest first:** run/extend the def-side recovery over the close=0 backlog → gate (drop-straggler-safe) → ×134. +3. **Learn the idiom:** research the gcc pass (R17) + crack 1 low-closeness exemplar (hand/agent) → distill to the cookbook (R16). +4. **Harvest the idiom:** `worker_wave` over the fuel manifest (seed agents with the new idiom) → `gate_stage --src-file --commit` → ×134. +5. At ~80% of this class's fuel → `--assess` next idiom → `--gen-fuel` → **request a fresh session + hand Drew a new `/loop` prompt.** Forever. + +**★ THE `/loop` PROMPT for the NEXT (fresh) session** (Drew pastes it to start): +> `/loop` Phase-21 idiom cycle. Read CURRENT_PHASE cont.5 + the cookbook. (1) `idiom_loop.py --assess`. (2) FIRST harvest the close=0 deterministic-recovery lever: make `sig_unify --src-file`-aware (don't drop `_a` drafts) or cast the def, re-gate the close=0 backlog → bank ×134. (3) LEARN the `schedule` gcc-2.7.2 idiom: research sched.c (R17), crack a close≈3 exemplar (func_80128ED8/func_80131A34) by hand, distill to the cookbook. (4) `worker_wave` the schedule fuel (`.run/auto/idiom_fuel_schedule.json`) seeded with the idiom → gate ×134. (5) At ~80% fuel → `--assess` the next idiom → `--gen-fuel` → request a fresh session + give Drew a new `/loop` prompt. Phase 21 is NEVERENDING; grinder runs token-free alongside (`tools/auto_status.sh`). + +--- + +## ★★★ SESSION 2026-06-24 (cont. 4) — the `_a` vein IS a ×134 lever (10/11); cont.3 §24 pessimism CORRECTED (superseded by cont. 5) > Authoritative. The first real `_a` measurement wave ran (Drew greenlit + `/effort Max`). CORRECTS cont.3/§24: > the `_a` vein is NOT ×1 loose-typing-walled — **10/11 banked fns propagate ×134 byte-identical**. The "×1" diff --git a/tools/idiom_loop.py b/tools/idiom_loop.py new file mode 100644 index 000000000..787bbf21f --- /dev/null +++ b/tools/idiom_loop.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""idiom_loop.py — Phase-21 NEVERENDING idiom-learning meta-loop (the wall->next-idiom->fuel engine). + +The harvest model: each session drains the EASY reach-134 wave fuel until ~80% is consumed (a WALL), +then learns the next gcc-quirk IDIOM that unlocks the highest-leverage residual, and hands off to a +fresh session (clean context) to crack + harvest it. Repeat forever (Phase 21 ends only when Drew says). + + --fuel-status : how much FRESH cached reach-134 wave fuel remains (the wall detector). + "WALL" when fresh ≤ --wall (default 12) -> run --assess. + --assess : cluster the backlog near-misses by gcc-quirk class, rank by reach-134 + leverage, and name (a) the next IDIOM to LEARN (top close>0 codegen class) + and (b) the cheap DETERMINISTIC recovery lever (close=0 plumbing/decl). + --gen-fuel : write the fuel manifest (.run/auto/idiom_fuel.json) for : each fn's + addr/nins/reach/closeness/where_stuck/best_draft/asm/ghidra_c — the + research+harvest fuel for the next idiom-learning session. + +A fresh session then: research the gcc pass (R17, tools/reference/gcc-papermario) -> crack the lowest- +closeness exemplar -> distill the idiom (cookbook) -> re-wave the class with it ×134 -> at ~80% of THAT +fuel, --assess again. Knowledge compounds toward 100% decomp. +""" +import argparse, collections, json, os + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +BACKLOG = os.path.join(ROOT, ".run/backlog.jsonl") +GHIDRA_C = ".run/ghidra_c" +# asm subdirs to probe per region (first existing wins) +ASM_DIRS = ["asm/ov_SC01_077/nonmatchings/ov_SC01_077_a", + "asm/ov_SC01_077/nonmatchings/ov_SC01_077", + "asm/ov_SC01_077/nonmatchings/ov_SC01_077_o0"] + + +def load_backlog(): + """last record per fn (the freshest near-miss state); only still-unbanked near/fail.""" + recs = {} + if os.path.exists(BACKLOG): + for line in open(BACKLOG): + line = line.strip() + if not line: + continue + try: + r = json.loads(line) + except ValueError: + continue + if r.get("name"): + recs[r["name"]] = r + return [r for r in recs.values() if r.get("status") in ("near", "failed")] + + +def klass_of(r): + k = r.get("klass") or "unknown" + return k.strip().split()[0] if k.strip() else "unknown" + + +def asm_for(name): + for d in ASM_DIRS: + p = os.path.join(ROOT, d, name + ".s") + if os.path.exists(p): + return os.path.join(d, name + ".s") + return None + + +def cluster(rows): + by = collections.defaultdict(lambda: {"n": 0, "r134": 0, "r1": 0, "close": []}) + for r in rows: + d = by[klass_of(r)] + d["n"] += 1 + rch = r.get("reach") or 0 + if rch and rch >= 134: + d["r134"] += 1 + elif rch == 1: + d["r1"] += 1 + c = r.get("closeness") + if isinstance(c, int): + d["close"].append(c) + return by + + +def med(xs): + xs = sorted(xs) + return xs[len(xs) // 2] if xs else None + + +def cmd_assess(rows): + by = cluster(rows) + print(f"{'class':<16}{'count':>6}{'r134':>6}{'r1':>5}{'med_close':>10}") + for k, d in sorted(by.items(), key=lambda kv: -kv[1]["r134"]): + print(f"{k:<16}{d['n']:>6}{d['r134']:>6}{d['r1']:>5}{str(med(d['close'])):>10}") + # the next IDIOM to LEARN = the top class whose residual is real codegen (median closeness > 0), + # excluding the deterministic/plumbing buckets (those are a recovery-tool job, not a gcc idiom). + RECOVERY = {"plumbing", "loose-typing"} + codegen = [(k, d) for k, d in by.items() + if k not in RECOVERY and k not in ("unknown", "GIANT") and (med(d["close"]) or 0) > 0] + codegen.sort(key=lambda kv: -kv[1]["r134"]) + # the cheap DETERMINISTIC lever = close=0 reach-134 fns (byte-correct, declaration-blocked). + det = sum(1 for r in rows if (r.get("reach") or 0) >= 134 and r.get("closeness") == 0) + print() + if codegen: + k, d = codegen[0] + print(f"NEXT IDIOM TO LEARN -> '{k}' ({d['r134']} reach-134 fns, median {med(d['close'])} ins off)") + print(f" gen its fuel: .venv/bin/python tools/idiom_loop.py --gen-fuel {k}") + print(f"DETERMINISTIC RECOVERY LEVER -> {det} close=0 reach-134 fns (byte-correct, decl-blocked) " + f"-> a sig_unify/cast recovery banks them ×134 for ~0 tokens (do this FIRST each cycle).") + + +def cmd_gen_fuel(rows, klass, out): + fuel = [] + for r in rows: + if klass_of(r) != klass: + continue + name = r["name"] + addr = r.get("addr") + fuel.append({ + "name": name, + "addr": (f"0x{addr:08x}" if isinstance(addr, int) else addr), + "nins": r.get("nins"), + "reach": r.get("reach"), + "closeness": r.get("closeness"), + "where_stuck": r.get("where_stuck"), + "best_draft": r.get("best_draft"), + "asm": asm_for(name), + "ghidra_c": (f"{GHIDRA_C}/{name}.c" if os.path.exists(os.path.join(ROOT, GHIDRA_C, name + ".c")) else None), + }) + # closest-first: the lowest-closeness exemplar is the one to crack the idiom on + fuel.sort(key=lambda f: (f["closeness"] if isinstance(f["closeness"], int) else 9999)) + os.makedirs(os.path.dirname(os.path.join(ROOT, out)), exist_ok=True) + open(os.path.join(ROOT, out), "w").write(json.dumps(fuel, indent=1)) + cached = sum(1 for f in fuel if f["ghidra_c"] and f["asm"]) + r134 = sum(1 for f in fuel if (f["reach"] or 0) >= 134) + print(f"wrote {len(fuel)} '{klass}' fns -> {out} ({r134} reach-134, {cached} cached/draftable)") + print("exemplars to crack the idiom on (closest first):") + for f in fuel[:5]: + print(f" {f['name']} close={f['closeness']} reach={f['reach']} nins={f['nins']}: " + f"{(f['where_stuck'] or '')[:90]}") + + +def cmd_fuel_status(wall): + """remaining FRESH cached reach-134 fns wave_targets would serve (not yet in the backlog).""" + seen = set() + if os.path.exists(BACKLOG): + for line in open(BACKLOG): + try: + seen.add(json.loads(line).get("name")) + except ValueError: + pass + # approximate fresh fuel = cached ghidra_c fns not yet attempted, with an _a/main asm. + cdir = os.path.join(ROOT, GHIDRA_C) + cached = {f[:-2] for f in os.listdir(cdir)} if os.path.isdir(cdir) else set() + fresh = [n for n in cached if n not in seen and asm_for(n)] + state = "WALL — run --assess" if len(fresh) <= wall else "fuel OK — keep waving" + print(f"fresh cached unbanked fuel ≈ {len(fresh)} (wall ≤ {wall}) -> {state}") + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--assess", action="store_true") + ap.add_argument("--gen-fuel", metavar="CLASS") + ap.add_argument("--fuel-status", action="store_true") + ap.add_argument("--out", default=".run/auto/idiom_fuel.json") + ap.add_argument("--wall", type=int, default=12) + a = ap.parse_args() + rows = load_backlog() + if a.fuel_status: + cmd_fuel_status(a.wall) + elif a.gen_fuel: + cmd_gen_fuel(rows, a.gen_fuel, a.out) + elif a.assess: + cmd_assess(rows) + else: + ap.print_help() + + +if __name__ == "__main__": + main()