feat(phase-21): R2/R3 — the learning flywheel (class-grouped waves + distill)

- residual class travels with the draft: drafter stamps // @class / // @stuck; gate_stage logs
  the worker's gcc-quirk class to the backlog (not the coarse manifest class).
- wave_targets.py --class <C> / --list-classes: class-grouped re-attempt waves over the backlog's
  classified near-misses (REGALLOC/SCHEDULE/REMAT/STRUCT/IV/LOOPGUARD/LOOSE/PLUMBING).
- orchestrator prep: auto-picks a class-focused wave when a class accrues >= threshold near-misses,
  else a fresh pool harvest; clears the draft dir per wave.
- tools/workflows/distill.js: post-wave agent extracts NEW byte-verified gcc idioms -> cookbook §21
  (R16, conservative); drafters now read the LIVE cookbook so distilled idioms feed forward.
- runbook: the full prep->worker->gate->distill cycle (the Phase-18 close-rate loop, automated).

docs/automation-runbook.md

@@ -37,14 +37,24 @@ The worker drafts matching C with LLM agents (the §17–20 toolkit: register pi
 (only a session can invoke the Workflow tool), so it runs as a self-paced **`/loop`**:
 
 1. In a Claude Code session in this repo, run **`/loop`** with this cycle as the prompt:
-   > Run one orchestrator cycle: `tools/orchestrator.py prep --n 24` → read `.run/auto/wave_batch.json`
-   > → launch the `tools/workflows/worker_wave.js` Workflow with `args={draftDir:".run/drafts-wave",
-   > targets:<the batch>}` → after it completes, `tools/orchestrator.py finish --drafts
-   > .run/drafts-wave --commit` → report the JSON summary. Stop if `.run/auto/STOP` exists.
-2. `/loop` self-paces (one wave per tick, ~15–20 min/wave, ~275k tokens/wave of ~24 agents).
-   The grinder runs alongside, draining the near-misses each wave produces (CPU vs token budget).
-3. Cost note: ~24 xHigh agents/wave. The ROI gate rotates pools (tractable → giants → o0 → capped)
-   when a pool's bank-rate drops, so it doesn't grind a wall. Stop anytime with `auto_stop.sh`.
+   > Run one Phase-21 orchestrator cycle, then stop if `.run/auto/STOP` exists:
+   > (a) `.venv/bin/python tools/orchestrator.py prep --n 24` — auto-picks a **class-focused** wave (re-attempt
+   > the backlog's biggest gcc-quirk class) or a **fresh pool** wave; it prints `{mode, sel, n, batch}`.
+   > (b) Read `.run/auto/wave_batch.json` and launch the **`tools/workflows/worker_wave.js`** Workflow with
+   > `args={draftDir:".run/drafts-wave", targets:<the batch array>}`. Wait for it.
+   > (c) `.venv/bin/python tools/orchestrator.py finish --drafts .run/drafts-wave --commit` — gates, banks,
+   > propagates ×134, logs near-misses; prints `{banked, propagated, near, verified, ...}`.
+   > (d) If `verified` is non-empty, launch the **`tools/workflows/distill.js`** Workflow with
+   > `args={draftsDir:".run/drafts-wave", verified:<the verified array>}` — it extracts any NEW byte-verified
+   > gcc idiom into the cookbook so the next wave inherits it (the learning flywheel).
+   > (e) Report the one-line result.
+2. **The learning flywheel:** fresh waves classify their near-misses by gcc-quirk (the drafter stamps
+   `// @class`/`// @stuck`); once a class accrues ≥6 near-misses, `prep` fires a **class-focused re-attempt
+   wave** for it (drafters get the prior stuck-point + the live cookbook); `distill` turns each wave's banked
+   techniques into new cookbook idioms. This is the Phase-18 close-rate-rising loop, automated.
+3. `/loop` self-paces (~15–20 min/wave, ~275k tokens/wave of ~24 agents). The grinder runs alongside,
+   draining near-misses. Cost-bounded by the ROI gate + `auto_stop.sh`. Inspect classes anytime:
+   `.venv/bin/python tools/wave_targets.py --list-classes`.
 
 **Remote management (Drew has laptop + can remote into the dev box):** you don't need a bulletproof
 keep-alive — if the worker `/loop` session dies, just remote in and re-run `/loop` (the grinder daemon

tools/gate_stage.py

@@ -120,23 +120,31 @@ def _run_gate_locked(drafts, binary, src, asm, out, good_sha, propagate, source_
         sh([PY, "tools/dedup_propagate.py", "--auto-from", binary, "--min-reach", "2"], timeout=3600)
         propagated = max(0, _dedup_group_count() - before)
 
-    # 6 log every non-match to the backlog (closeness + class + best draft for the human)
+    # 6 log every non-match to the backlog (closeness + RESIDUAL class + best draft for the human).
+    # The drafter stamps `// @class: <gcc-quirk class>` and `// @stuck: <note>` into the draft (so the
+    # residual class travels WITH the file) — use those for the learning flywheel; else fall back to the
+    # manifest class. This is what makes class-grouped waves (wave_targets.py --class) possible.
     near = failed = 0
     for fn in [f for f in draft_fns if f not in verified]:
         cpath = os.path.join(REPO, d, fn + ".c")
-        kind, close = match_one_closeness(fn, cpath, asm) if os.path.exists(cpath) else ("fail", None)
+        body = open(cpath).read() if os.path.exists(cpath) else ""
+        kind, close = match_one_closeness(fn, cpath, asm) if body else ("fail", None)
         meta = _manifest_class(fn)
+        cm = re.search(r"//\s*@class:\s*(.+)", body)
+        sm = re.search(r"//\s*@stuck:\s*(.+)", body)
+        rclass = cm.group(1).strip() if cm else None       # the worker's self-reported residual class
+        note = sm.group(1).strip() if sm else None
         if kind == "match":   # match_one says MATCH but the whole-binary gate rejected -> plumbing/TU conflict
-            status, where = "near", "match_one MATCH but gate rejected (declaration/TU plumbing)"
+            status, where = "near", note or "match_one MATCH but gate rejected (declaration/TU plumbing)"
         elif kind == "near":
-            status, where = "near", f"{meta.get('lever') or meta.get('class') or 'residual'}: {close} mismatch"
+            status, where = "near", note or f"{meta.get('lever') or meta.get('class') or 'residual'}: {close} mismatch"
             near += 1
         else:
-            status, where = "failed", "won't compile standalone (loose-typing / missing decl)"
+            status, where = "failed", note or "won't compile standalone (loose-typing / missing decl)"
             failed += 1
-        draft_path = backlog.save_draft(fn, open(cpath).read()) if os.path.exists(cpath) else None
+        draft_path = backlog.save_draft(fn, body) if body else None
         backlog.append_record({"addr": meta.get("addr"), "name": fn, "reach": meta.get("reach"),
-                               "klass": meta.get("class"), "nins": meta.get("nins"), "status": status,
+                               "klass": rclass or meta.get("class"), "nins": meta.get("nins"), "status": status,
                                "closeness": close, "where_stuck": where, "best_draft": draft_path,
                                "source": source_tag})
     backlog.render()

tools/orchestrator.py

@@ -20,7 +20,7 @@ Usage:
   orchestrator.py finish --drafts .run/drafts-wave [--commit]
   orchestrator.py status
 """
-import argparse, json, os, subprocess, sys, time
+import argparse, json, os, re, subprocess, sys, time
 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
 import gate_stage
 
@@ -46,20 +46,42 @@ def sh(cmd, timeout=None):
     return subprocess.run(cmd, capture_output=True, text=True, cwd=REPO, timeout=timeout)
 
 
+def _top_class(min_n):
+    """The backlog residual class with the most near-misses (>= min_n), for a class-focused wave."""
+    out = sh([PY, "tools/wave_targets.py", "--list-classes"], timeout=60).stdout
+    best, bestn = None, 0
+    for line in out.splitlines():
+        m = re.match(r"(\w+)\s+n=\s*(\d+)", line.strip())
+        if m and m.group(1) != "OTHER":
+            n = int(m.group(2))
+            if n > bestn:
+                best, bestn = m.group(1), n
+    return (best, bestn) if bestn >= min_n else (None, bestn)
+
+
 def cmd_prep(a):
     s = load_state()
-    pool = s["pool"]
-    # refresh the manifest so 'cached'/stub status is current (cheap)
-    sh([PY, "tools/build_fuel_manifest.py"], timeout=120)
-    region = a.region
-    r = sh([PY, "tools/wave_targets.py", "--pool", pool, "--n", str(a.n),
-            "--region", region, "--out", BATCH], timeout=120)
+    sh(["rm", "-rf", ".run/drafts-wave"])                   # fresh draft dir per wave
+    os.makedirs(os.path.join(REPO, ".run/drafts-wave"), exist_ok=True)
+    sh([PY, "tools/build_fuel_manifest.py"], timeout=120)   # refresh cached/stub status (cheap)
+    # FLYWHEEL: prefer a CLASS-FOCUSED re-attempt wave when the backlog has a worthwhile, distill-able
+    # class (the Phase-18 learning model); else harvest a FRESH pool (which classifies new near-misses).
+    mode, sel = "pool", s["pool"]
+    if a.mode in ("auto", "class"):
+        cls, cn = _top_class(a.class_threshold)
+        if cls:
+            mode, sel = "class", cls
+    if mode == "class":
+        sh([PY, "tools/wave_targets.py", "--class", sel, "--n", str(a.n), "--out", BATCH], timeout=120)
+    else:
+        sh([PY, "tools/wave_targets.py", "--pool", sel, "--n", str(a.n),
+            "--region", a.region, "--out", BATCH], timeout=120)
     n = 0
     try:
         n = len(json.load(open(os.path.join(REPO, BATCH))))
     except Exception:
         pass
-    print(json.dumps({"pool": pool, "n": n, "batch": BATCH, "wave": s["waves"] + 1}))
+    print(json.dumps({"mode": mode, "sel": sel, "n": n, "batch": BATCH, "wave": s["waves"] + 1}))
 
 
 def cmd_finish(a):
@@ -98,6 +120,10 @@ def main():
     ap = argparse.ArgumentParser()
     sub = ap.add_subparsers(dest="cmd", required=True)
     p = sub.add_parser("prep"); p.add_argument("--n", type=int, default=24); p.add_argument("--region", default="main")
+    p.add_argument("--mode", default="auto", choices=["auto", "class", "pool"],
+                   help="auto=class-focused wave when the backlog has a distill-able class, else pool harvest")
+    p.add_argument("--class-threshold", dest="class_threshold", type=int, default=6,
+                   help="min near-misses in a class before a class-focused wave fires")
     f = sub.add_parser("finish"); f.add_argument("--drafts", default=".run/drafts-wave")
     f.add_argument("--commit", action="store_true"); f.add_argument("--threshold", type=float, default=0.15)
     f.add_argument("--patience", type=int, default=2)

tools/wave_targets.py

@@ -11,11 +11,36 @@ o0 | capped | any-reach134.
 Usage: tools/wave_targets.py --pool tractable --n 24 [--region main|a|any] [--out -]
 """
 import argparse, glob, json, os, re, sys
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import backlog
 
 REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 STUB_RE = re.compile(r"INCLUDE_ASM\([^,]+,\s*(\w+)\)")
 ASM_SUBDIR = "asm/ov_SC01_077/nonmatchings/ov_SC01_077"
 
+# Canonical gcc-quirk residual classes (the cookbook §17–20 taxonomy). A wave studies ONE of these
+# at a time (the Phase-18 learning model): bank what the class's idiom reaches, distill the quirk,
+# feed it forward. Keyword-matched from the worker's self-reported klass + its where_stuck note.
+CLASS_KEYWORDS = [
+    ("REGALLOC", ("regalloc", "register", "$s", " pin", "reg-order", "reg order", "swap")),
+    ("SCHEDULE", ("schedule", "sched", "store-vs-load", "store vs load", "delay slot", "reorder", "operand-order", "operand order")),
+    ("REMAT", ("hoist", "remat", "rematerial", "array-decay", "array decay")),
+    ("STRUCT", ("struct", "field", " type", "layout", "%lo", "array-of-struct", "array of struct", "union")),
+    ("IV", ("iv-combine", "iv combine", "induction", "biv", "halfword rmw")),
+    ("LOOPGUARD", ("loop-guard", "loop guard", "get_condition", "strength-reduc")),
+    ("LOOSE", ("loose-typing", "loose typing", "arity", "conflicting types", "narrow-param")),
+    ("PLUMBING", ("plumbing", "declaration", "extern", "call-site cast", "callee", "no-proto", "sibling decl")),
+]
+
+
+def canon_class(rec):
+    """Normalize a backlog record (klass + where_stuck) to a canonical residual class."""
+    blob = ((rec.get("klass") or "") + " " + (rec.get("where_stuck") or "")).lower()
+    for name, kws in CLASS_KEYWORDS:
+        if any(k in blob for k in kws):
+            return name
+    return "OTHER"
+
 
 def live_stubs():
     s = set()
@@ -39,10 +64,24 @@ def backlog_walls():
     return walls
 
 
+def emit(batch, out):
+    s = json.dumps(batch, indent=0)
+    if out == "-":
+        sys.stdout.write(s + "\n")
+    else:
+        open(os.path.join(REPO, out), "w").write(s)
+        print(f"{len(batch)} targets -> {out}", file=sys.stderr)
+
+
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("--pool", default="tractable",
                     choices=["tractable", "giants", "o0", "capped", "any-reach134"])
+    ap.add_argument("--class", dest="rclass", default=None,
+                    help="CLASS-GROUPED wave: select backlog near-misses of this residual class "
+                         "(REGALLOC/SCHEDULE/REMAT/STRUCT/IV/LOOPGUARD/LOOSE/PLUMBING/OTHER) to re-attempt")
+    ap.add_argument("--list-classes", action="store_true",
+                    help="print the backlog residual-class histogram (ranked by leverage) and exit")
     ap.add_argument("--n", type=int, default=24)
     ap.add_argument("--region", default="main", choices=["main", "a", "any"])
     ap.add_argument("--max-nins", type=int, default=150)
@@ -50,6 +89,30 @@ def main():
     ap.add_argument("--out", default="-")
     a = ap.parse_args()
 
+    # --- CLASS-GROUPED modes (Phase-21 flywheel): operate on the backlog's classified near-misses ---
+    if a.list_classes:
+        import collections
+        recs = [r for r in backlog.load_best() if r.get("status") == "near"]
+        cnt = collections.Counter(canon_class(r) for r in recs)
+        lev = collections.Counter()
+        for r in recs:
+            lev[canon_class(r)] += (r.get("reach") or 1)
+        for cls, _ in lev.most_common():
+            print(f"{cls:10} n={cnt[cls]:3}  reach-weight={lev[cls]}")
+        return
+    if a.rclass:
+        rc = a.rclass.upper()
+        recs = [r for r in backlog.load_best()
+                if r.get("status") == "near" and canon_class(r) == rc and r.get("name")]
+        recs.sort(key=lambda r: (-(r.get("reach") or 1), r.get("closeness") if isinstance(r.get("closeness"), int) else 999))
+        batch = [{"name": r["name"], "addr": r.get("addr") or ("0x" + r["name"][5:].lower()),
+                  "nins": r.get("nins"), "class": rc, "asm": f"{ASM_SUBDIR}/{r['name']}.s",
+                  "ghidra_c": f".run/ghidra_c/{r['name']}.c", "prior_stuck": r.get("where_stuck"),
+                  "prior_closeness": r.get("closeness")}
+                 for r in recs[:a.n]]
+        emit(batch, a.out)
+        return
+
     m = json.load(open(os.path.join(REPO, ".run/fuel_manifest.json")))
     stubs = live_stubs()
     walls = set() if a.include_walls else backlog_walls()
@@ -81,12 +144,7 @@ def main():
     batch = [{"name": t["name"], "addr": t["addr"], "nins": t["nins"], "class": t["class"],
               "asm": f"{ASM_SUBDIR}/{t['name']}.s", "ghidra_c": f".run/ghidra_c/{t['name']}.c"}
              for t in pool]
-    out = json.dumps(batch, indent=0)
-    if a.out == "-":
-        sys.stdout.write(out + "\n")
-    else:
-        open(os.path.join(REPO, a.out), "w").write(out)
-        print(f"{len(batch)} targets -> {a.out} (pool={a.pool} region={a.region})", file=sys.stderr)
+    emit(batch, a.out)
 
 
 if __name__ == "__main__":

tools/workflows/distill.js

@@ -0,0 +1,54 @@
+export const meta = {
+  name: 'distill',
+  description: 'Phase-21 learning flywheel: after a wave gates, extract any NEW byte-verified gcc-quirk idiom from the wave\'s banked matches and append it to the cookbook, so the next wave inherits it.',
+  phases: [{ title: 'Distill', detail: 'one agent reads the banked drafts + cookbook, appends a new idiom' }],
+}
+
+// args = { draftsDir, verified: [fn,...] }  (verified = the fns the byte-gate banked this wave)
+// Returns { appended: bool, idiom: string|null, note: string }.
+const A = typeof args === 'string' ? JSON.parse(args) : (args || {})
+const draftsDir = A.draftsDir || '.run/drafts-wave'
+const verified = A.verified || []
+if (!verified.length) { log('distill: no banked matches this wave — nothing to distill'); return { appended: false, idiom: null, note: 'no banks' } }
+
+const SCHEMA = {
+  type: 'object', additionalProperties: false, required: ['appended'],
+  properties: {
+    appended: { type: 'boolean' },
+    idiom: { type: 'string', description: 'the one-line idiom appended (or empty)' },
+    note: { type: 'string', description: 'why appended / why not' },
+  },
+}
+
+phase('Distill')
+const prompt = `You are the LEARNING step of the Brave Fencer Musashi matching flywheel (R16). A worker wave just
+BYTE-MATCHED these functions (whole-binary gate-verified — they are byte-identical to the original):
+  ${verified.join(', ')}
+Their winning C is in: ${draftsDir}/<fn>.c (each starts with // @class and // @stuck comments naming the gcc
+quirk it overcame and the technique used).
+
+YOUR JOB: decide whether any of these wins teaches a NEW, GENERALIZABLE gcc-2.7.2 idiom that is NOT already in
+docs/matching-cookbook.md (§17–§20), and if so append it — so the NEXT wave's drafters (which read the live
+cookbook) inherit it. This is exactly how Phase 18 drove the close-rate 33%→56%→90%.
+
+PROCESS (you have Read, Bash, Edit):
+1. Read each banked draft in ${draftsDir} for the verified fns — note the // @class, // @stuck, and the actual
+   C technique (register pins, array-of-struct, statement reordering, casts, struct layout, etc.).
+2. Read docs/matching-cookbook.md §17–§20. Is the technique ALREADY documented there?
+3. DISTILL CONSERVATIVELY (R14/R16 — the cookbook is authoritative, byte-honest):
+   - Append ONLY a genuinely NEW + generalizable idiom (a C *shape* that triggers the wanted codegen for a
+     whole CLASS of functions), backed by these byte-verified wins. Cite the fn(s) as evidence.
+   - Do NOT append one-offs, restatements of existing §17–20 idioms, or anything you can't tie to a byte-match.
+   - If nothing is new/general, append NOTHING (that is the correct, common outcome — say so).
+4. If appending: add a tight bullet under a "### §21 — wave-distilled idioms (Phase 21)" heading at the END of
+   docs/matching-cookbook.md (create the heading once if absent). Format: **<class>:** <the C shape> — *fixes
+   <quirk>; evidence <fn>*. Preserve all existing content (H5). Keep it to 1–3 lines.
+5. Commit ONLY the cookbook if you changed it:
+   git add docs/matching-cookbook.md && git commit -q -m "docs(phase-21): distill — <class> idiom from <fn>"
+6. Return { appended, idiom, note }.
+
+Be rigorous: a wrong/over-broad idiom pollutes every future wave. When in doubt, append nothing.`
+
+const r = await agent(prompt, { label: 'distill', phase: 'Distill', schema: SCHEMA })
+log(`distill: ${r && r.appended ? 'APPENDED — ' + (r.idiom || '') : 'nothing new (' + ((r && r.note) || '') + ')'}`)
+return r || { appended: false, idiom: null, note: 'agent returned null' }

tools/workflows/worker_wave.js

@@ -27,10 +27,13 @@ const DRAFT_SCHEMA = {
 const ASM_SUBDIR = 'asm/ov_SC01_077/nonmatchings/ov_SC01_077'
 
 function drafterPrompt(t, draftDir) {
+  const prior = t.prior_stuck
+    ? `\nPRIOR ATTEMPT got stuck here (closeness ${t.prior_closeness}): "${t.prior_stuck}". This is a CLASS-FOCUSED re-attempt — concentrate on that residual; the cookbook may now have a newly-distilled idiom for it.\n`
+    : ''
   return `Match ONE MIPS function for the Brave Fencer Musashi PS1 matching decompilation (overlay ov_SC01_077).
 GOAL: write C that the pinned compiler (gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker + maspsx --aspsx-version=2.56 --expand-div) compiles to BYTE-IDENTICAL machine code.
 
-TARGET: ${t.name} @ ${t.addr} — ${t.nins} instructions, class hint "${t.class}".
+TARGET: ${t.name} @ ${t.addr} — ${t.nins} instructions, class hint "${t.class}".${prior}
 - Target asm (the ground truth): ${t.asm}
   (each line "/* off vaddr w0 w1 */ mnemonic ..." shows the exact encoded instructions.)
 - Ghidra-C reference (types/locals/callee names — NOT byte-accurate, a scaffold): ${t.ghidra_c}
@@ -49,8 +52,12 @@ THE TOOLKIT (docs/matching-cookbook.md §17–§20 — read those sections for d
   cast_call_sites + sig_unify fix most extern/arity mismatches. Focus on the BODY codegen.
 
 PROCESS (you have Bash + Read):
+0. Read the LIVE cookbook docs/matching-cookbook.md §17–20 FIRST — it accrues newly-distilled idioms between waves; a quirk you'd otherwise grind on may already be solved there.
 1. Read the target asm and the Ghidra-C.
 2. Write your best C (the function definition + any externs it needs) to: ${draftDir}/${t.name}.c
+   START the file with TWO comment lines so the residual class travels with the draft (the gate reads them for the learning flywheel + backlog):
+     // @class: <one of: regalloc-order | schedule | remat | struct | iv-combine | loop-guard | loose-typing | plumbing | other>
+     // @stuck: <one concrete line on the residual that remains, or "none — MATCH">
 3. Self-check (fast relocation-masked proxy for the byte-gate):
    .venv/bin/python tools/match_one.py ${t.name} --c ${draftDir}/${t.name}.c --asm-subdir ${ASM_SUBDIR}
    - "MATCH (N ins)"  => byte-identical (relocation-masked). You nailed it. Stop.