- docs/commit-map.tsv: 4,032 rows (ordinal of the ORIGINAL main -> rewritten hash, author/committer dates, subject);
1 pruned row of zeros (ordinal 1712, "session archive update"); 0 old hashes asserted; ordinal 1 unchanged by the
rewrite (byte-identical)
- resolve_tokens: 1,238 commit:NNNN tokens -> shortest-unique new hashes in 98 files (docs, phase-ends, logs, tool
docstrings, 2 C comments, the A5 evidence logs); residue left as tokens: commit:1712 x4 (the pruned commit),
commit:orphan-24 x2, commit:orphan-26, commit:orphan-35 (cited commits that exist in no lineage)
- the rewrite (C4): filter-repo 2.47.0 on a bare clone of the C2 tip, 311 s, exactly 1 pruned, main 4,032 -> 4,031;
the pre-rewrite history is mirrored in the private archive repo and in the local bundle
- the proof (C5): verify_rewrite 4,031 pairs / 0 failures; absent_scan 0 offenders; gate_scan 0 offenders on the clone
- adoption (C6): 100 text files differ at the tip, 0 purge paths, 0 added/deleted; leftover refs dropped; no gc yet
- resolver skips tools/public_rewrite/ (its self-test fixtures are the token grammar, not citations); repo-local
identity is the GitHub noreply address from here on; CURRENT_PHASE: C4–C7 logged, checkpoint -> NEXT = C8
* `binof = {c["fn"]: c["binary"]}` was last-writer-wins, and `status`, `det` and `subof`
had the same shape — a draft of a name carried by two binaries was stamped with
whichever card came last and then reloc-checked against the OTHER binary's symbols.
* Resolve per draft instead: the shard's own target list first
(`.run/wave_<tag>_targets.<i>.json` = `targets[i::workers]`, each row carrying its
binary), a unique-name card second, a counted refusal when neither can answer (R43).
* R39 negative control over every historical wave: 42,655 drafts, 0 regressions,
2,317 (5.4%) previously mis-stamped; 2,107 homonym card names fleet-wide.
Intra-shard ambiguity: 0 of 50,684 (shard, name) pairs over 302,370 shard files.
docs: §408 — §406 refuted as a sweep (0 MATCH / 14 applied, 0 / 210). The 134-member
census counted main's 960 LINKED library stubs and matched a symmetric SHAPE; derived
from the mine-vs-target residual the addressable set is 15 / 210. Decision-log entry
records the pivot: 64 of 210 (30.5%) already match standalone, so the frontier's
largest lane is §376 integration, not codegen.
tools/weave_sweep.py — the derived-selector sweep (R32 coverage, R41 denominators,
--lever-all ablation control).
campaign_status: 'today: N banked' summed '— N banked' commit subjects and missed every bank that
rode in a chore/maint commit (S60: 2,185 reported vs 2,644 net stubs removed); it now derives the
number from INCLUDE_ASM stub counts at last-commit-before-midnight / HEAD / working tree (R33), and
alive() is anchored so pgrep no longer matches its own wrapper (every lane read ok with 0 processes).
ox_campaign gater: the ledger's wall_min counts drafting + queue wait since the ready marker's t0;
gate_min is the gate alone (the '30-67 min gates' picture was this conflation).
maintenance lane: the fleet R22 sweep skipped whenever any gate was in flight, i.e. always (last
real sweep 12:54 08-25); it now takes .run/auto/draw.lock and waits its turn, skipping only for gate_main.
THE GATE WAS A BLACK BOX. sweep_parallel's stdout was captured and dropped, so a gate logged
"reloc_identity -> gating 216" and then THIRTY MINUTES OF SILENCE before its bank line — no
worker count, no per-binary progress, no phase-A/phase-B split. Gate times went 31 -> 37 ->
50 -> 67 min across ej/ek/en/eo with nothing to diagnose from, and I twice asserted things
about phase B that the log could not support (its absence measured LOG CAPTURE, not
behaviour). A lane that must run unattended has to leave evidence.
MEASURED WHILE DIAGNOSING, and it rules out the obvious suspects: load average 2.6 on 32
cores with 1-3 concurrent builds during a gate — the gate is NOT CPU-bound and is not
saturating its own -j 24. Raising to 32 is cheap given ~8% utilisation, but the real answer
will come from the log this change adds.
TAIL_DONE_FRAC 0.85 -> 0.80. 0.85 overcorrected: the fleet fell to 15 agents / 11 req/min
because the drafter parks between waves while the gater drains a deep queue. 0.75 was too
deep (26% 429s, draft completion sliding 94->91->73->47% across eq/er/es/et). Neither number
is really the lever: the drafter cannot start a wave the gater has no room for, so the gate
throughput is what bounds the campaign now.
Generational tiering confirmed already correct: the top-off orders by generation at both
assembly levels (group ranking and within-group) without FILTERING any tier out, so every
generation stays eligible and the scarce never-drafted work simply goes first.
Two ways to spend a free drafting window on a tail that is 92% walls.
ONE_PER_GID=0 — the sibling collapse exists because a same-gid sibling banks by mechanical
remap once its exemplar cracks, so drafting it pays for what the remap does free. That prices
AGENT TOKENS as the scarce resource. On the free ox window they are not, and the collapse is
what makes 3,271 open crackable functions look like 334 drawable skeletons — of which 308 are
gen6+ walls whose exemplars have already refused six waves each. A sibling drafted directly
can crack on its OWN terms instead of waiting on an exemplar that never will.
Measured on a live draw rather than argued:
uncollapsed 627 cards / 44,403 ins / 160 binaries / 215 gate groups = 2.9 drafts per rebuild
collapsed 334 cards / 30,926 ins / 104 binaries / 127 gate groups = 2.6 drafts per rebuild
The gate cost is per (binary, TU) group and chunked, so siblings landing in binaries the wave
already touches are close to free at the gate — the card count nearly doubles and the gate gets
MORE efficient per build, not less.
ATTEMPTS=K — K independent shots at each card. The gate cost does not multiply: reloc_filter
keys by fn and staging writes <binary>/<fn>.c, so a function still gets exactly one whole-binary
build per wave; the attempts compete to BE that build, ranked by match_one, which is local and
needs no build. Alternates stay on disk for a later recovery pass. Default 1 (no-op).
A BUG I CAUGHT IN MY OWN SELECTOR before it shipped: it passed binof[fn] (a BINARY NAME) where
match_one wants --asm-subdir (an asm DIRECTORY). Every attempt would have scored identically at
infinity and the picker would have silently degraded to first-seen while appearing to rank —
the same "true number about the wrong thing" class as the day's other defects. Fixed with a
subof map, and a missing subdir now returns neutral instead of a fake score.
The gate's dirty-tree committer swept an empty config/overlays.mk into commit:2863 and took
the whole fleet down with it. R42 says commit a dirty tree rather than revert — true for
src/, where a per-binary gate leaves PROVEN banks uncommitted and reverting destroys them.
A config file is the opposite case: it holds no proven state that exists only in the
worktree, and a collapsed one is never intended.
config_sane() runs at all three commit sites: if config/overlays.mk or config/dedup.us.yaml
has fewer than 80% of HEAD's lines, it is restored from HEAD, NOT committed, and the refusal
is logged loudly. Controls both ways — positive (min_ratio=1.5 makes the healthy registry
trip the same branch: detected, restore path runs, file intact) and negative (normal
threshold: silent, returns True). P28's registry died this way too (H5); now it is enforced
rather than remembered.
MEASURED over 18 consecutive waves. Consecutive card sets: ck->cl 239/239 shared, co->cp
238/238, cv->cw 222/222, db->dc 208/209 — and the "different" pairs still shared 50-90%.
Yield alternated in lockstep: 47.6% / 3.8% / 35.3% / 3.6% / 29.9% / 3.7% / 43.4% / 14.6%,
because the duplicate wave gates AFTER the original banked its cards. Half of all drafting
went to work already in flight, and it read as campaign decay.
ROOT CAUSE: --retry-unbanked returns "previously waved but still an OPEN STUB" cards to the
pool — right in principle, unfinished work is not spent work. But the pre-draw for wave N+1
runs WHILE wave N drafts, when none of wave N's cards have been gated, so every one of them
is still an open stub and the filter hands the whole wave back. The ranking then rebuilds it
card for card. The filter knew about "banked" and "not banked" and had no notion of "in
flight".
FIX: a wave is finished when its GATE has run, and the gater already says so in its own log
(R33 — derive from the artifact that exists). Tags with no GATE line stay excluded; a tag
with no gate line whose cards are older than 6 h was killed, and is released so nothing is
locked out forever.
THROUGHPUT, same commit — the draw was setting the campaign's request rate:
* --max-bins 24 -> 160. Concentrating a wave into 24 gate groups was a CPU-economy choice
made when CPU was scarce. It is not: a live gate runs at load 2.7 of 32 cores (8%), one
harvest_verify at --chunk 1. Meanwhile the drafting fleet — the resource actually bounded
by the free-model clock — got 196 cards out of 699 available. Re-drawn with 160 bins:
644 drafts / 46,590 ins across 136 binaries, 3.3x the wave for the same gate economics.
* TAIL_DONE_FRAC 0.95 -> 0.80. Overlapping at 95% still left 25% of minutes under 20 req/min,
because a wave's last 5% is its SLOWEST 5% and 12 stragglers cannot fill a fleet. Handing
off at 80% starts the next ramp with ~40 agents still working. Stragglers keep their full
700s grace in the finisher thread; nothing is cut short.
* --queue-depth 2 -> 4, so a bigger wave's longer gate never parks the drafter.
Arithmetic this is aimed at: req/min = agents-in-flight x ~0.8 (a 16k-token turn at ~30
tok/s emits few requests). 644 cards x two overlapping waves puts the fleet where the
endpoint has already been measured to sustain it — 764 req/min for 15 min at 8% 429s, peak
2,755 in one minute.
One clean whole-EXE rebuild verified the batch (gate_main), and main re-checked
BYTE-IDENTICAL against config/check.us.sha before anything was credited.
SYS_OBJ_2DD8
_clr
func_8005EA68
Measured 23:20: the maintenance lane held .run/auto/draw.lock for a multi-minute
sweep, the pre-draw buffer happened to be empty, and the drafting fleet — the one
clock-limited resource — sat at 13 agents and 3 req/min printing 'gate holds the draw
lock and nothing is pre-drawn — waiting 30s' every thirty seconds.
That wait dates from when drawing during a gate was genuinely unsafe: corpus.stubs()
misreports for a binary whose sources carry a substituted draft (R35), so the draw
refused outright. Since 15:23 build_wave_atlas excludes exactly the binaries whose
per-binary gate lock is held and draws from the rest, so the hazard is handled at the
right granularity and the blanket wait now protects nothing.
Same lesson as the draw's own refusal earlier today: a guard scoped more broadly than
the hazard gets routed around or, worse, quietly starves the thing it sits in front of.
PROVEN: from 14:57:01 to 18:43:23 today HEAD built main to 307aa45d… against the
expected 143dbb89…, with NO draft substituted (measured under gate.main.lock, no
gate_main alive). Auto-commit commit:2693 had adopted a mid-flight gate_main
substitution — its carve-out reverted main's TUs, gate_main re-wrote them, and
`git add -A src/` swept the unverified bodies in (a TOCTOU race, 14 s after a
bisect chunk banked). Every main batch after it was doomed before its first
draft was judged: m00–m03 card cycles drafted ~737, slated 160, banked 0, and
burned ~50 clean rebuilds bisecting innocent slates. commit:2712 restored the
green content by accident (it swept this investigation's diagnostic checkout).
gate_main: on any batch failure, ONE try_batch([]) control runs first — if HEAD
itself is red it prints BASELINE RED, leaves the slate reusable, exits 3 (R40).
clean_build no longer reports a linked-but-mismatched build as "no binary" (the
build target embeds the SHA check), the compile-conflict shortcut fires only on
error-shaped lines naming a symbol some draft in the slate actually uses (the
baseline's own func_800143AC implicit-decl WARNING was matching — every m04
chunk died with "drafts declaring it: []"), reverts narrow to top-level src/*.c
(main_tus) so a main gate can never destroy overlay lanes' in-flight work, and
--assert-baseline is a first-class mode.
main_lane: every cycle opens with gate_main --assert-baseline and REFUSES to
draft or gate against a red baseline (R43) — BaselineRed parks nothing, burns
no tries, writes .run/main_lane.BASELINE_RED, re-checks every 30 min.
Adopters (ox_campaign ×3, maintenance.sh, gate_stage, gate_lane, idiom_serial):
main's TUs (top-level src/*.c) are never staged and never reverted by an
overlay/maintenance lane — one writer (gate_main), one committer (main_lane,
after the whole-EXE SHA re-checks green). Unstage-after-add is race-free where
the old revert-then-add was the losing half of the TOCTOU.
Diagnosis, evidence and the full timeline: docs/tool-designs/main-lane-fix-s59.md
Everything that reaches the GATE and fails gets a backlog row with closeness, class
and best draft. A draft the reloc pre-filter drops never reaches the gate, so it was
recorded nowhere and just sat on disk: 569 of 1,261 drafts across the last eight waves
— 45%.
They are not all garbage. 13% of the MISMATCH? rejects have a body that ALREADY
MATCHES and only the symbol names wrong, which is the deterministic aprop_symfix
stale-symbol class that banked 4 of 4 earlier this session. Roughly 6 recoverable
drafts per wave were being thrown away because no index existed to find them.
Now appended to .run/reloc_rejects.jsonl with the verdict, the shape (MATCH here means
right body, wrong symbols) and the first mismatches, so a recovery pass can work them
without re-drafting. Wrapped so telemetry can never break a gate.
Three measured harness defects, all fixed:
1. THE CARD NAMED A WORD THE COOKBOOK DOES NOT CONTAIN. api_agent stated the lever
as a bare label and nothing else; grep 'extend-tell' / 'swaprepeat' / 's16-div-tell'
over the 750-section cookbook returns ZERO. 108 failure transcripts grepped
extend-tell and 28 grepped swaprepeat against nothing while the knowledge sat at
172a/172b under different words. Fixed both ends: a LANE ALIASES grep-bait block at
172b, and LEVER_CRIB on the card — what the tell means, the section to grep, and
the byte-proven C spellings.
2. ONE GLOBAL AGENT BUDGET FOR CARDS OF VERY DIFFERENT SIZE. tells cards are 2.4x the
default lane's (median 89-95 ins vs 37-39) and stack 3-5 idioms; 98 of 270 final
attempts ended AT the 24-turn cap. LANE_BUDGET gives tells 40 turns / /bin/bash.40, and
logs the choice so it is auditable rather than invisible.
3. Two new SYS laws: grep the section your crib cites before drafting, and stop when
the residual class says [permuter]; plus 263 (an invented argument changes
scheduling — check arity before reaching for a fence the permuter cannot help with).
Cookbook 264 records the four recipes the tells agent drove to MATCH: the inline (s16)
in a call argument, the save-order/bb0 anti-dependence law (new), the opaque-bound
local assigned late (new), and 172b-1's multi-def mirror variable made concrete.
Closes the gap that made main red for nine hours. R42 ('commit a dirty tree rather than
revert it') is correct for a per-binary gate that leaves PROVEN banks uncommitted, and WRONG
for gate_main, whose substitution is unverified by construction until the SHA matches.
Two guards, defense in depth:
1. gate_main installs atexit + SIGTERM/SIGINT/SIGHUP handlers that revert its own substitution
unless a bank actually succeeded. Killed mid-run, it now cleans up after itself.
2. ox_campaign's dirty-tree commit REFUSES top-level src/*.c (main's sources), reverting those
and committing the rest. Verified: src/800c.c and src/800.c refused, src/ov_*/... and
src/shared/engine_core.h still commit.
Also versions the autonomous lane scripts under tools/lanes/ — they lived only in gitignored
.run/, so a fresh clone had no drafter, gater, maintenance or stallguard at all.
The main probe (8 drafts) ran 38 minutes without a verdict. Two defects, neither about the
drafts:
1. FIXED — the compile-error shortcut matched only 'previous declaration of', but gcc printed
'previous implicit declaration of func_80017930'. So a batch whose culprit gcc had already
named fell through to bisection, which costs a full clean EXE rebuild per step. The matcher
now accepts the implicit and conflicting-types forms too. (resolve_conflicts is separately
blind to this class: an implicit decl comes from a call site with no prototype.)
2. NOT FIXED, documented — the typedef-hoist repair is not idempotent. It emitted 'hoisted 2
typedef(s)' 150 times and left a duplicated marker comment; it re-hoists, rebuilds, fails
identically and repeats, so it cannot converge. Make it idempotent and bound the bisect
before gating main again.
Also: ox_campaign pre-draws the next wave AFTER launching shards (doing it before left the
fleet at 8 agents while a card job ran), collect_drafts grants stragglers a grace period
instead of letting 2 of 220 shards idle the fleet for 34 minutes, and drafter bands are now
mostly full-range (the 400-2000 band drew 9 cards for a 2,000-worker fleet).
Two independent reviews (ox design study + Fable validation) found parse_config does not
implement its own documented contract on md_*/main: it can DELETE the c config line and
corrupt the yaml on disk before failing. main also has no config/splat.main.yaml (it is
splat.us.exe.yaml), which is the FileNotFoundError seen on every main jtbl target.
Refuse loudly rather than corrupt quietly (R43). Lift only after parse_config is hardened
and proven on two examples.
Also: gate() stages into a per-run dir (a killed gate's 3,186 stale drafts survived into the
next gate of the same tag and poisoned every group); reloc_filter no longer treats
NOT-A-STUB as a pass (it means ALREADY BANKED — wave an staged 480 of them over source that
already byte-matches).
Designs recorded: docs/tool-designs/jtbl-island-split.md (ox) and -review.md (Fable):
14 CONFIRMED / 8 WRONG / 1 UNVERIFIABLE, verdict GO-WITH-CHANGES, and the correct fix is
SMALLER than proposed — one inserted .rodata carve line + jr_isolate_all.py --only, no _pre
piece and no ld_interleave leading mode.
CURRENT_PHASE.md gains a CRASH-RECOVERY checkpoint (not a fresh-session handoff): what is
running, restart order, the measured fleet/scaling facts, the fixes that must not regress,
and the ordered work queue.
Lanes: drafter (never stop it), gater (restartable), maintenance (free A-prop sibling lane),
stallguard (60s auto-repair). Drafting holds no lock; one narrow draw-vs-gate lock exists
because build_wave_atlas reads corpus.stubs and misreads substituted drafts mid-gate.
main is off the wave critical path — 157 drafts parked to .run/main_queue/ rather than
stalling the gater for another hour on a bisecting whole-EXE rebuild.
api_agent: 5xx retried like 429 (a 502 was abandoning functions at near-19), HTTP_TIMEOUT
420s not 1800 (a hung request parked an agent 30 min), EXTRA_READABLE for tooling briefs,
and bare-directory paths no longer refused against their own granted root.
R42: gate_main reverted 61 byte-proven overlay banks it could not distinguish from its own
substitution (sweep_parallel gates commit=False by design). Fixed by committing overlay banks
before the main batch, chunking main at 8 to bound bisect cost, and replacing every blind
'git checkout -- src/ config/' with commit-or-refuse in ox_campaign and idiom_serial.
R43: sweep_parallel had an explicit branch admitting main, which cannot be gated incrementally
— wave ab banked 0/105 main cards while its non-main cards banked 94/115 (82%), and the wave
read as a drafting failure. sweep_parallel now refuses main and names gate_main.py.
Also: validate_targets now prefers the card's own addr field (named symbols like SYS_OBJ_F00
were MALFORMED and discarded whole 220-card waves); ox_campaign deals model lanes by
smallest-ratio scheduling (a 73-card wave had put 73 shards on ox and 0 on deepseek);
docs/accelerators.md gains the four vacuous-check defects.
Drew T