# Automation runbook — HISTORICAL: the OpenRouter / ox-alpha autonomous campaign > ## ⛔ THIS IS NOT THE CURRENT PIPELINE. See `docs/wave-playbook.md`. > > **Superseded 2026-08-31 (P31 S67).** This file documents the **OpenRouter (ox-alpha) era** > automation, built to sustain a free-window API quota. Its drafting toolchain (§2) is OpenRouter, > and its lanes — `drafter` / `gater` / `maintenance` / `stallguard` / `distill` / `main` — are > **all DEAD, deliberately.** `tools/campaign_status.py` will confirm that. > > The project returned to the **Claude agent wave** system on purpose. Running a wave today means > `draw_waves` → `t5_cards` → `claude_wave_packs` → `wave_args` → `claude_wave_draft.js` → > `gate_wave.py` → `twin_sweep` → harvest → R22, which is written up start-to-finish, with the > measured failure behind each guard, in **`docs/wave-playbook.md`**. > > Kept because its mechanism notes are still accurate for the parts that survived — the banking > toolchain by binary class (§4), the main lane's batch cadence (§5), the distill flywheel (§6), and > the recovery procedures (§8) — and because the rate-limit measurements (§7) are the record of why > that era ended. **Rewritten 2026-08-24 (P31 S59).** The previous version described the June reach-1 grinder pivot and was two months stale: it named no lane that exists today. If this file and the lane scripts ever disagree again, **the scripts are the truth** — `tools/lanes/*.sh` are the tracked copies of what runs. Companions: `docs/SETUP.md` (per-tool reference table, versions, install), `docs/accelerators.md` (ops traps that cost real time), `docs/matching-cookbook.md` (the matching knowledge the drafters grep). --- ## 1. What runs right now Six detached lanes, each `setsid`-launched from `.run/.sh` (a copy of `tools/lanes/.sh`): | lane | script | what it does | safe to restart? | |---|---|---|---| | **drafter** | `.run/drafter.sh` | draw → shard → draft → queue a ready marker, forever | **Never stop it to ship a change.** Doing that cost 139 of 162 idle minutes on 2026-08-23. Restart only at a wave boundary (see §3) | | **gater** | `.run/gater.sh` | reloc pre-filter → gate → commit → harvest → ledger | yes, freely — nothing is lost but the pause | | **maintenance** | `.run/maintenance.sh` | free zero-token work when the gater is idle: the A-prop sibling sweep **and** `recover_rejects.py` (rebase pre-gate rejects whose body already matches and only the symbols are wrong — §171; 45% of drafts never reach the gate and 13% of those are recoverable) | yes | | **stallguard** | `.run/stallguard.sh` | 60 s: revive a dead lane shell, kill agents silent >20 min, kill a gate >90 min, bounce an idle drafter | yes | | **distill** | `.run/distill.sh` | watch harvested candidates, raise a READY marker when a batch is worth review | yes | | **main** | `.run/main.sh` | the EXE's own draft→gate→commit cadence (§5) | yes — `gate_main` reverts its own aborts | ```bash # is everything alive? for l in drafter gater maintenance stallguard distill main; do printf "%-12s %s\n" "$l" "$(pgrep -f "bash .run/$l.sh" >/dev/null && echo alive || echo DEAD)"; done pgrep -cf '^\.venv/bin/python -u tools/api_agent' # drafting agents right now touch .run/ox_campaign.stop # THE KILL SWITCH — every lane exits at its next loop top rm .run/ox_campaign.stop # then relaunch the lanes you want ``` --- ## 2. The drafting toolchain (OpenRouter) `tools/ox_campaign.py --drafter` draws a wave, shards it, and spawns one `tools/api_agent.py` per card. **Shard count tracks CARDS, not the `--workers` cap**: a 71-card wave runs 71 agents however many workers are configured. Card supply, not throughput, is the binding constraint. * **Models.** `--models 'stealth/ox-alpha:2000'` — ox is free for this window. The paid deepseek lane was dropped 2026-08-24 at $2.22 remaining (§7). Restoring it is two edits, named in the drafter script's header. * **The card is the fuel.** `tools/build_wave_atlas.py` writes `.run/wave__cards.json`: `fn · binary · nins · lever · model · tu · seed_ref · tu_ref · decl_prior`. `api_agent._fuel` renders it, and `LEVER_CRIB` explains what each lever MEANS plus the cookbook § to grep — a bare lever label is a dead end (108 transcripts grepped `extend-tell` against a cookbook containing that string zero times). * **Budgets are per lane** (`ox_campaign.LANE_BUDGET`): tells gets 40 turns / $0.40, everything else 24 / $0.15. One global cap starved the large cards — 98 of 270 tells attempts ended AT the cap. * **Model routing inside a wave** is by function size (cookbook §157): Haiku ≤30 ins → Sonnet 50-120 → Opus ≥120. **Fable is for new wall classes only — never for idiom distillation or review.** * **Output budget and socket timeout are ONE setting** (P31 S59, probed directly against ox-alpha): | knob | value | why | |---|---|---| | `--maxtok` | **16000** | the model's thinking is IN the content stream (`reasoning_tokens=0`), so the output cap WAS the reasoning cap. Measured over ALL turns (the denominator matters — an early count compared truncated turns only against turns that printed a finish reason, i.e. against themselves, and read as ~100%): wave `bk` at 8k truncated **240 of 3,222 turns = 7.4%**; wave `bt` at 16k truncated **16 of 1,210 = 1.3%**, a ~6x reduction. A truncated turn is a TAX, not a death — the log shows the agent emitting a tool call on the very next turn — so it costs one turn of 24. An uncapped hard prompt wanted **8,067** tokens, finishing exactly where the old cap cut it | | `HTTP_TIMEOUT` | **700** | ox generates at **~30 tok/s**, so a full 16k generation needs ~530 s. At the old 420 s the socket would kill the very turns the bigger budget exists to allow — and a timeout wastes the whole turn where truncation leaves a partial | | model ceiling | 1M context / **131,072** max completion | so 16k is OUR choice, not a limit. The binding constraint is the timeout, not the model | **The ordering that must hold:** generation < `HTTP_TIMEOUT` (700) < stallguard's wedged-agent kill (1200 s). 420 was itself deliberate — 1800 once parked a hung agent for thirty minutes. A **reasoning cap** (`REASON_CAP` → `reasoning:{max_tokens}`) does work on ox, but it shortens the ANSWER too (618-672 tokens against 8,067 uncapped): a quality dial, not a truncation fix. * **Turn caps are NOT binding on the default lane.** Across 1,166 completions, non-MATCH runs used a median of 4 oracle calls and a p90 of 12 against 24 available; exactly 1 of 194 reached 20. Agents give up early after truncated turns — which is why the fix above is the budget, not the cap. (The tells lane WAS cap-bound, 98 of 270, hence its 40.) ### What the draw admits, and what it refuses (all counted in the skip census) | filter | why | |---|---| | `o0-in-an-O2-object` | an -O0 function in an -O2 object cannot bank however good the draft (§261). 11 such were drawn **79 times across 19 waves** before this existed | | `tells-oversize-serial-lane` | tell-lever members > `--tells-max-ins` (80). Bank rate 27-40% at 5-80, 10% at 81-120, 1% at 121-200, 0% above. Those 383 members are `idiom_serial`'s work | | `jtbl-*` (`main-manual`, `island-blocked`, `island-pads`) | the gate's carve cannot reach that table yet (`jtbl_carve.island_probe`) | | `jtbl-one-per-binary` | §61c: one table-bearing draft per gate invocation | | `already-waved` / `already-banked` / `out-of-band` | ordinary pool bookkeeping | **One lane, one band** (P31 S59): every wave is a full-range default draw. The tells slot became redundant when the quota landed, and the dedicated 120-2000 slot was the worst wave we ran — bank rate 3% at 120-200 and 6% above, against 57% under 50 ins. Large functions still appear: the full band contains them and the draw takes mass-first inside each gate group. **Quotas** reserve cards that the gate-group ranking would otherwise never pick, because they are spread thin across binaries: `--tells-quota 60`, `--jtbl-quota 6`. **A quota is a floor AND a ceiling** — tells enter through the quota or not at all (left free, they took 41% of a test wave). --- ## 3. Changing something while it runs (read this before editing a lane) bash parses a `while … done` body **in full** before executing it, so the loop's command line is fixed for the life of that shell (`docs/accelerators.md` #5): | change | how it takes effect | |---|---| | lane **code** (`tools/*.py`) | next python start — bounce the python, args are unchanged | | lane **args** (the `.sh` invocation) | needs a fresh **shell** — `tools/lanes/relaunch_drafter_shell.sh` waits for a wave to queue first, so no drafts are lost | | wave-draw **defaults** (`build_wave_atlas.py`) | next draw — it is a fresh subprocess per wave, no restart at all | | the **gater's** args | `tools/lanes/restart_gater_when_idle.sh` — restarts once no sweep is in flight | | the **main lane's** args | `tools/lanes/restart_main_lane_when_idle.sh` — restarts in the gap between its gate and its next draw | Verify from the PROCESS, never the file: `tr '\0' ' ' < /proc//cmdline`, or the startup banner. --- ## 4. The banking toolchain, by binary class **A bank is byte-identical instructions AND a green whole-binary SHA1. Nothing else counts (G3/P9).** | class | path | notes | |---|---|---| | overlays / md | `sweep_parallel` → `gate_stage` → `harvest_verify` | per-binary flock `.run/auto/gate..lock`; distinct binaries gate concurrently | | **main** | `gate_main.py` ONLY | incremental builds give a FALSE diff (main's extract rewrites the linker script). `sweep_parallel` REFUSES main (R43) after wave `ab` banked 0 of 105 | | jtbl functions | the carve happens **at the gate** (`harvest_verify._jtbl_prep_one`), never pre-draw | §61b's proven order; `jtbl_lane.py` is the serial route | | -O0 functions | ordinary gates, but only from an **-O0 object** | the Makefile decides per object; `corpus.o0_subseg()` is the oracle, not the subseg name | Standing verification: **R22 = a CLEAN rebuild** (`make clean && make extract-all && make check-all`, ~1 min parallel). Do not run it while gates are building; scope it to the binary you touched with `rm -rf build/ build/src/ && make extract BINARY= && make check BINARY=`. **R42 — never revert a dirty `src/` or `config/`.** Other lanes bank real, byte-proven functions with `commit=False`, and no tool can tell them from residue; a blind `git checkout` destroyed 61 banked functions once. Commit named paths, or leave the tree alone. --- ## 5. The main lane (P31 S59) main is excluded from every overlay wave draw (`--exclude-bins main`) because its gate is a clean whole-EXE rebuild that bisects — three measured stalls put it off the critical path. It now has its own lane instead of no cadence at all: ``` .run/main_queue/*.json (parked drafts, free) ─┐ build_wave_atlas --only-bins main ─┴→ draft → reloc filter → gate_main --apply → verify main byte-identical → commit (R42) ``` * Batches of 40: **one clean rebuild verifies the whole slate** (~15 s measured), which is what makes main affordable. The reloc pre-filter keeps the batch from bisecting. * On a `COMPILE conflict` the lane **halves the slate and retries** — `gate_main` deliberately refuses to bisect a decl clash (right for a human, wrong for an unattended lane: 40 innocent drafts died with one conflicting symbol that was in the TU and in no draft). * Failed drafts are parked with a try count (`.run/main_queue_failed.json`), capped at 2. * **Credit requires two oracles**: the `INCLUDE_ASM` line gone from the tree AND main re-checked byte-identical. The first version asked `corpus.stubs('main')` — which returns `{addr: Stub}`, keyed by INT — so a name-vs-int comparison reported "12 banked of 12" from a gate that banked nothing. * **Every cycle opens with `gate_main --assert-baseline`** (S59): one clean no-draft rebuild proving HEAD still builds byte-identical BEFORE any tokens are spent. On red the lane refuses (R43), writes `.run/main_lane.BASELINE_RED`, and re-checks every 30 min. `gate_main` runs the same control on any batch failure and exits 3 without bisecting when the TREE, not the slate, is broken (R40). Why: auto-commit `commit:2693` adopted a mid-flight substitution at 14:57 and the lane then banked 0 from four 200-card rounds until 18:43 — every rejection a false verdict. Full story: `docs/tool-designs/main-lane-fix-s59.md`. * **main's TUs (top-level `src/*.c`) have ONE writer (`gate_main`) and ONE committer (`main_lane`)** (S59). Every overlay/maintenance adopter (`ox_campaign` ×3, `maintenance.sh`, `gate_stage`, `gate_lane`, `idiom_serial`) unstages them before committing and never reverts them — the old revert-then-add carve-out was the losing half of the TOCTOU that produced `commit:2693`. --- ## 6. The distill lane and the flywheel The gater harvests every wave before the next draw (`idiom_harvest.py`) into `.run/idiom_candidates..md`. That is EXTRACTION. **Distillation into the cookbook is what changes the next wave**, because that is what the agents grep. * `.run/distill.sh` raises `.run/distill_ready/.json` at ≥30 novel candidates or ≥2 waves. * State is `{tag: novel-count-when-mined}` — never a done-list, because a re-gated wave rewrites its candidate file under the same tag. * **The reviewer is an Opus or Sonnet subagent, never Fable** (Drew, 2026-08-24). * Measured yield, twice: **~82-88% of candidates are already covered**, ~1 new law + ~10 addenda per 80 candidates, and **one in three credited levers is byte-inert** (§266 — strip it and recompile before writing the law). * Wave N's ore is distilled while wave N+1 drafts, so **wave N+2 is the first that can grep it**. That one-wave lag is the price of never pausing the fleet. --- ## 7. Rate limits, credits, and the measured ceiling | fact | number | |---|---| | lifetime requests / 429s | 121,166 / 7,667 = **6.3%** | | 429 attribution | 100% provider shared capacity (`openrouter_shared_capacity`, `upstream_provider_shared_pool`) — **zero platform refusals, ever** | | steady state | 50-95 req/min → **0-3.5%** 429s | | the knee | 195-227 req/min → **6.6-12.7%** 429s | | one-minute peak observed | 2,755 requests | | per-agent request rate | 0.31-0.78 req/min (mean ~0.6) — so ~350-380 agents sits at the knee | | retry policy | `MAX_429=10` with 20 s backoff (default 6). A 429 costs a retry; an idle agent costs a card | Telemetry: every request appends to `.run/api_rate.jsonl`; read it with `tools/api_rate.py`. **Bucket 429s over time before calling anything a ceiling** — launch bursts fake one. **Credits.** `credits_left()` reads the OpenRouter balance. `--credit-floor` does **not** pause the paid lane, it breaks the whole drafting loop — and the shell then restarts a python that breaks again. With a free model the balance is not a proxy for "can we draft", so the floor is 0.25. --- ## 8. Recovery 1. `git status --porcelain -- src/ config/` — if dirty, **COMMIT it, never revert** (R42). The one safe revert is `src/800.c`/`src/800c.c` alone when a `gate_main` was interrupted mid-substitution. 2. Salvage drafts: `find .run/wave_* -name '*.c' | wc -l` — drafts survive crashes and are worth re-gating via `ox_campaign.py --gate-only ` before drafting anything new. 3. Relaunch lanes: `setsid nohup .run/.sh >> .run/.log 2>&1