fix(gate): log what the gate actually does; --gate-jobs 24 -> 32; TAIL_DONE_FRAC 0.80

THE GATE WAS A BLACK BOX. sweep_parallel's stdout was captured and dropped, so a gate logged
"reloc_identity -> gating 216" and then THIRTY MINUTES OF SILENCE before its bank line — no
worker count, no per-binary progress, no phase-A/phase-B split. Gate times went 31 -> 37 ->
50 -> 67 min across ej/ek/en/eo with nothing to diagnose from, and I twice asserted things
about phase B that the log could not support (its absence measured LOG CAPTURE, not
behaviour). A lane that must run unattended has to leave evidence.

MEASURED WHILE DIAGNOSING, and it rules out the obvious suspects: load average 2.6 on 32
cores with 1-3 concurrent builds during a gate — the gate is NOT CPU-bound and is not
saturating its own -j 24. Raising to 32 is cheap given ~8% utilisation, but the real answer
will come from the log this change adds.

TAIL_DONE_FRAC 0.85 -> 0.80. 0.85 overcorrected: the fleet fell to 15 agents / 11 req/min
because the drafter parks between waves while the gater drains a deep queue. 0.75 was too
deep (26% 429s, draft completion sliding 94->91->73->47% across eq/er/es/et). Neither number
is really the lever: the drafter cannot start a wave the gater has no room for, so the gate
throughput is what bounds the campaign now.

Generational tiering confirmed already correct: the top-off orders by generation at both
assembly levels (group ranking and within-group) without FILTERING any tier out, so every
generation stays eligible and the scarce never-drafted work simply goes first.
This commit is contained in:
Drew T
2026-08-25 19:00:02 -06:00
parent 348eb2808a
commit f78ea55b3e
3 changed files with 21 additions and 2 deletions
+12 -1
View File
@@ -104,7 +104,18 @@ export STRAGGLER_GRACE=700
# one wave draining and the next ramping. Handing off at 65% keeps 3-4 waves overlapping, so
# the fleet is always carrying a full ramp somewhere. Stragglers still keep the full 700s
# grace in the finisher thread — this changes WHEN THE NEXT WAVE STARTS, never what lands.
export TAIL_DONE_FRAC=0.75
# TAIL_DONE_FRAC 0.75 -> 0.85 (P31 S60, Drew). Deeper overlap bought concurrency and then
# started spending it on retries: eight simultaneous waves means near-continuous ramping, and
# ramps are where throttling bites. Measured at 0.75 with 877 agents: 429s at 26% over the
# hour and draft completion sliding 94% -> 91% -> 73% -> 47% across eq/er/es/et, against
# 97-99% completion earlier today at sub-10% 429s. Half of et's cards were being spent for
# nothing. Fewer waves in flight, same ~500-card uncollapsed draws: trade peak req/min for
# the number that actually converts.
# 0.85 overcorrected: the fleet fell to 15 agents / 11 req/min because the drafter parks
# between waves while the gater drains a deep queue. 0.75 was too deep (26% 429s, completion
# sliding to 47%), 0.85 too shallow. 0.80 splits it — and the real lever is the GATE, not the
# overlap: the drafter cannot start a wave the gater has no room for.
export TAIL_DONE_FRAC=0.80
# MAX_BINS 160 -> 50 (P31 S60). Wave size was the right lever at 50% conversion (wave dd:
# 217 banked of 422 gated in 39 min). It is dead weight at 5%: dq banked 8 of 167 gated and
# took 98 MINUTES of gate to do it, while four drafted waves queued behind it and the free-ox
+1 -1
View File
@@ -8,7 +8,7 @@ while [ ! -e .run/ox_campaign.stop ]; do
# idle. It matters now because the tells and jtbl QUOTAS deliberately pull cards from binaries
# outside the ranked gate groups: a measured draw went from ~24 groups to 63, i.e. 63 rebuilds per
# wave, and at 12 jobs that is five serial batches. Raise this with the quota sizes, not on its own.
.venv/bin/python tools/ox_campaign.py --gater --gate-jobs 24 2>&1
.venv/bin/python tools/ox_campaign.py --gater --gate-jobs 32 2>&1
echo "[$(date +%H:%M:%S)] [gater] exited; restarting in 20s"
sleep 20
done
+8
View File
@@ -665,6 +665,14 @@ def gate(tag, keep, jobs, run_id=None):
return 0, []
t0 = time.time()
r = sh(f"{PY} tools/sweep_parallel.py --drafts {d} -j {jobs}", timeout=28800, quiet=False)
# LOG WHAT THE GATE ACTUALLY DID (P31 S60). sweep_parallel's stdout was captured and dropped,
# so a gate showed `reloc_identity -> gating 216` and then THIRTY MINUTES OF SILENCE before its
# bank line — no per-binary progress, no phase-A/phase-B split, no worker count. When gates went
# from 32 to 67 minutes there was nothing to diagnose from, and I twice drew conclusions about
# phase B that the log could not support. A lane that runs unattended must leave evidence.
for _ln in (r.stdout or "").splitlines():
if _ln.strip() and not _ln.startswith(" ["): # skip the per-binary chatter
log(f" sweep: {_ln.strip()[:150]}")
banked = []
for f in glob.glob(".run/auto/bulk/*.verified.txt"):
if os.path.getmtime(f) >= t0: