docs: gating is fully parallel — no serial lane; jtbl unlocked via isolate_asm; launch detached with setsid

This commit is contained in:
Drew T
2026-08-31 14:55:39 -06:00
parent c26ddf136d
commit 5bcb322283
2 changed files with 39 additions and 2 deletions
+7
View File
@@ -617,6 +617,13 @@ Full how-to in `docs/matching-cookbook.md` §11. Command crib:
## §6.9 Running a matching wave — see `docs/wave-playbook.md`
**GATING IS FULLY PARALLEL AS OF P31 S67 — there is no serial lane.** `tools/parallel_gate.py`
handles jtbl carves too (`isolate_asm()` gives a carving worker a writable copy of the ONE binary's
3.6-5 MB asm subtree instead of symlinking all 448 MB). Measured: 13 fns/13 binaries in 139 s and
19 fns/14 jtbl binaries in 188 s, against 58 minutes for a single jtbl binary serially.
**Always `make -j$(nproc) build`** — a per-binary build is ~35 objects and was single-threaded
(7.18 s -> 1.18 s, byte-identical); every tool that shells `make build` now passes `-j`.
The start-to-finish operational sequence (draw → cards → packs → validate → draft → split-gate →
twin_sweep → harvest → R22 → checkpoint), with the measured failure that justifies each guard.
The tooling-inventory rows below are the per-tool REFERENCE; the playbook is the PROCEDURE.
+32 -2
View File
@@ -100,7 +100,32 @@ still open **at draw time**.
Streaming **burns the 5-hour window faster** (it removes the idle gaps), so slots are the budget
dial. Model routing: ≤50 ins Sonnet · 51–120 Sonnet · >120 Opus. Never Haiku→Opus directly.
## 6. Gate — split on jtbl, run both lanes at once
## 6. Gate — EVERYTHING PARALLEL. There is no serial lane.
```
python3 tools/parallel_gate.py --plan plan.json --workers 12 --commit
# plan.json: [{"binary": "...", "drafts": "/abs/path"}, ...]
```
**MEASURED S67, and this is the bar:**
| lane | result |
|---|---|
| non-jtbl | **13 banked / 13 binaries / 139 s** (12 workers) |
| jtbl | **19 banked / 14 binaries / 188 s** (8 workers), 0 refusals |
| the same jtbl work, serially, earlier that day | **58 minutes for ONE binary** |
**jtbl used to be serial-only** because `harvest_verify`'s carve runs `make extract` and a worktree's
`asm/` is a symlink to the main tree — a carving worker would rewrite shared asm. `isolate_asm()`
fixes it: `asm/` is 448 MB but ONE binary's subtree is **3.6-5.0 MB**, so a carving job gets
per-binary symlinks plus a real copy of the single binary it carves (~5 MB/worker). It is applied
PER JOB because worker slots are reused, and gated by the same jtbl predicate `harvest_verify`
carves on.
**If you are writing `for b in binaries: gate_stage ...`, STOP.** That loop is the hour-long mistake
this section exists to prevent.
### 6b. (historical) split on jtbl, run both lanes at once
```
python3 tools/gate_wave.py --drafts <dir> --workers 8 --commit [--r22]
@@ -116,7 +141,8 @@ python3 tools/gate_wave.py --drafts <dir> --workers 8 --commit [--r22]
through the symlink and writes the MAIN tree while other workers read it. "Run everything parallel
and re-run the failures" can poison the whole batch.
* Measured cost of getting this wrong: I gated **16 binaries serially to protect ONE jtbl draft** —
about an hour for what should have taken minutes.
about an hour for what should have taken minutes. **`gate_wave.py`'s split is now an optimisation
(same-binary drafts share a build), NOT a safety requirement.**
## 7. After ANY bank
@@ -187,6 +213,10 @@ until ! pgrep -f "parallel_[g]ate.py" >/dev/null; do sleep 20; done # better,
The regex `gate_[s]tage.py` does not match the literal `gate_[s]tage.py`, but it matches the
`gate_stage.py` sitting in the launch half of the very same line.
**LAUNCH DETACHED:** `setsid nohup <cmd> > log 2>&1 < /dev/null & disown`. A plain `nohup` child
shares the launching shell's PROCESS GROUP, so the harness killing that shell on its timeout kills
the job too — measured S67: a `sleep 120` in the launching call discarded **8 completed jtbl carves**.
**THE RULE: launch and wait in SEPARATE shell invocations.** Launch in one call, return, then wait
in another whose command line never names the target unbracketed. Better still, wait on a
CONDITION the job itself produces — a completion marker in its log — rather than on process