PROVEN: from 14:57:01 to 18:43:23 today HEAD built main to 307aa45d… against the
expected 143dbb89…, with NO draft substituted (measured under gate.main.lock, no
gate_main alive). Auto-commit commit:2693 had adopted a mid-flight gate_main
substitution — its carve-out reverted main's TUs, gate_main re-wrote them, and
`git add -A src/` swept the unverified bodies in (a TOCTOU race, 14 s after a
bisect chunk banked). Every main batch after it was doomed before its first
draft was judged: m00–m03 card cycles drafted ~737, slated 160, banked 0, and
burned ~50 clean rebuilds bisecting innocent slates. commit:2712 restored the
green content by accident (it swept this investigation's diagnostic checkout).
gate_main: on any batch failure, ONE try_batch([]) control runs first — if HEAD
itself is red it prints BASELINE RED, leaves the slate reusable, exits 3 (R40).
clean_build no longer reports a linked-but-mismatched build as "no binary" (the
build target embeds the SHA check), the compile-conflict shortcut fires only on
error-shaped lines naming a symbol some draft in the slate actually uses (the
baseline's own func_800143AC implicit-decl WARNING was matching — every m04
chunk died with "drafts declaring it: []"), reverts narrow to top-level src/*.c
(main_tus) so a main gate can never destroy overlay lanes' in-flight work, and
--assert-baseline is a first-class mode.
main_lane: every cycle opens with gate_main --assert-baseline and REFUSES to
draft or gate against a red baseline (R43) — BaselineRed parks nothing, burns
no tries, writes .run/main_lane.BASELINE_RED, re-checks every 30 min.
Adopters (ox_campaign ×3, maintenance.sh, gate_stage, gate_lane, idiom_serial):
main's TUs (top-level src/*.c) are never staged and never reverted by an
overlay/maintenance lane — one writer (gate_main), one committer (main_lane,
after the whole-EXE SHA re-checks green). Unstage-after-add is race-free where
the old revert-then-add was the losing half of the TOCTOU.
Diagnosis, evidence and the full timeline: docs/tool-designs/main-lane-fix-s59.md
Closes the gap that made main red for nine hours. R42 ('commit a dirty tree rather than
revert it') is correct for a per-binary gate that leaves PROVEN banks uncommitted, and WRONG
for gate_main, whose substitution is unverified by construction until the SHA matches.
Two guards, defense in depth:
1. gate_main installs atexit + SIGTERM/SIGINT/SIGHUP handlers that revert its own substitution
unless a bank actually succeeded. Killed mid-run, it now cleans up after itself.
2. ox_campaign's dirty-tree commit REFUSES top-level src/*.c (main's sources), reverting those
and committing the rest. Verified: src/800c.c and src/800.c refused, src/ov_*/... and
src/shared/engine_core.h still commit.
Also versions the autonomous lane scripts under tools/lanes/ — they lived only in gitignored
.run/, so a fresh clone had no drafter, gater, maintenance or stallguard at all.
The P31 S58 main probe ran 38 minutes on 8 drafts and never produced a verdict. Two
independent non-termination bugs, both fixed and negative-controlled:
1. THE BISECT COULD NOT TERMINATE. On a failing multi-element chunk it did 'lo = head + lo',
restoring lo to exactly its prior value, so the next iteration recomputed the same head and
failed identically — forever. Replaced with an explicit-stack bisect that SPLITS a failing
chunk and pushes both halves, so work strictly decreases and termination is structural.
GATE_MAIN_MAX_STEPS (24) is a loud backstop, not the mechanism.
Verified: 8 drafts with one poisoned -> 7 banked, 1 rejected, 7 rebuilds.
2. THE TYPEDEF HOIST WAS NOT IDEMPOTENT. The block is inserted AT the anchor, so anything
hoisted previously still started after it and was re-hoisted every call, stacking a fresh
marker comment each time ('hoisted 2 typedef(s)' x150; the tree held a duplicated marker).
Now tracks the already-hoisted region and reuses the existing marker.
Verified: 3 consecutive passes hoist [Foo,Bar], [], [] with exactly 1 marker.
Together these unblock main: 170 parked drafts and ~1,041 open stubs.
The main probe (8 drafts) ran 38 minutes without a verdict. Two defects, neither about the
drafts:
1. FIXED — the compile-error shortcut matched only 'previous declaration of', but gcc printed
'previous implicit declaration of func_80017930'. So a batch whose culprit gcc had already
named fell through to bisection, which costs a full clean EXE rebuild per step. The matcher
now accepts the implicit and conflicting-types forms too. (resolve_conflicts is separately
blind to this class: an implicit decl comes from a call site with no prototype.)
2. NOT FIXED, documented — the typedef-hoist repair is not idempotent. It emitted 'hoisted 2
typedef(s)' 150 times and left a duplicated marker comment; it re-hoists, rebuilds, fails
identically and repeats, so it cannot converge. Make it idempotent and bound the bisect
before gating main again.
Also: ox_campaign pre-draws the next wave AFTER launching shards (doing it before left the
fleet at 8 agents while a card job ran), collect_drafts grants stragglers a grace period
instead of letting 2 of 220 shards idle the fleet for 34 minutes, and drafter bands are now
mostly full-range (the 400-2000 band drew 9 cards for a 2,000-worker fleet).
CURRENT_PHASE.md gains a CRASH-RECOVERY checkpoint (not a fresh-session handoff): what is
running, restart order, the measured fleet/scaling facts, the fixes that must not regress,
and the ordered work queue.
Lanes: drafter (never stop it), gater (restartable), maintenance (free A-prop sibling lane),
stallguard (60s auto-repair). Drafting holds no lock; one narrow draw-vs-gate lock exists
because build_wave_atlas reads corpus.stubs and misreads substituted drafts mid-gate.
main is off the wave critical path — 157 drafts parked to .run/main_queue/ rather than
stalling the gater for another hour on a bisecting whole-EXE rebuild.
api_agent: 5xx retried like 429 (a 502 was abandoning functions at near-19), HTTP_TIMEOUT
420s not 1800 (a hung request parked an agent 30 min), EXTRA_READABLE for tooling briefs,
and bare-directory paths no longer refused against their own granted root.
Second instance of the §192 defect class, found by wave Y's ov_SC02_017 slate. The project's own
§37/§124 idiom spells a renamed symbol as `extern s32 gVecX __asm__("D_80126B5C");`, and sym_of's
generic branch matched `__asm__` -- an identifier followed by '(' -- before reaching the real one.
Every aliased declaration therefore collided with every other one under the name `__asm__`.
Measured cost on one slate: 1 byte-verified draft DROPPED and 2 phantom CONFLICTING-EXTERN
failures, on an idiom this same session used to RECOVER work.
NC over src/ plus wave Y's drafts: 1,210 changed verdicts, every one `__asm__` -> the real alias
identifier (899 of them one symbol, aD800B9A02 -- the idiom is fleet-wide), 0 regressions.
Each one refused byte-verified work; each fix is probed, not reasoned:
* built-in redeclaration: a cc1 probe shows two conflicting "memcpy" declarations give
"warning: conflicting types for built-in function" + exit 0, while the same pair on a
non-builtin name errors. Every overlay TU in the fleet declares memcpy twice and compiles
today -> CONFLICTING-EXTERN on a builtin is now WARN.
* driver mismatch: overlays bank via gate_lane -> gate_stage -> harvest_verify, which strips
every typedef the target TU provides; pregate_check modelled gate_main's hoist/strip instead
and reported DUPLICATE-TYPEDEF for exactly the duplicates the real gate removes.
substitute() now takes an optional per-draft transform; pregate passes the overlay one.
* block-scope typedefs: two functions may each declare their own typedef inside their bodies
(that is how a draft stays self-contained for match_one). _typedefs now honours the brace
depth map the caller already computed.
* project scalar aliases: include/common.h's "typedef s32 M2C_UNK;" makes "extern s32 D_x" and
"extern M2C_UNK D_x" the same declaration; _ALIASES now DERIVES those from common.h (R33).
Measured on the 5 leftover slates: 28 drafts, all re-verified MATCH by match_one, went from
"0 kept / phantom FAILs" to main 2 clean, ov_SC04_011 15 clean, ov_SC03_028 1 clean,
ov_SC06_029 4 + 1 named TU edit, ov_SC02_005 2 real TYPEDEF-USED-ABOVE-DEFINITION.
build_wave_atlas: --one-per-gid collapses same-skeleton siblings to one card and defers
them to <out>.siblings.json for the post-bank family_sweep remap (R32 accounting asserted);
--rank total ranks gate groups by DELIVERED mass (card + deferred siblings). Measured on the
wave-T draw: 6,557 drafted ins carrying 12,709 sibling ins behind 69 of 71 gids = 19,266
instructions of potential for 71 agents, vs 9,985 behind 57 under --rank mass. R39 NC: the
flag is byte-inert on a pool whose gids are unique.
gate_main/pregate_check (§192): three defects that made the pre-gate ladder main-only while
reporting "clean" on overlay slates — (1) resolve_conflicts/substitute hardcoded
corpus.stubs('main') -> per-binary _stubs_for(); (2) sym_of returned the keyword `void` for
every `extern void (*D_x[])(...)`, manufacturing 192 phantom CONFLICTING-EXTERNs (NC over
5,526,100 declarations: 189,301 changed verdicts, 0 regressions); (3) `void f()` and
`void f(void)` were normalized together, costing 40 more phantoms — C89's unspecified-
parameter rule is now gate_main.sig_conflict. §192b: the tool refuses when it substituted 0
files, and prints the per-draft [DROP] reasons it used to compute and discard.
Same overlay slate now reports 2 failures, both real (duplicate typedef; memcpy declared two
ways). Cookbook §192/§192b + index regenerated (585 sections).
Wave P drafted at 97% and cost A DOZEN clean rebuilds to bank, and not one of those rebuilds
failed on a matching problem -- every one failed on a TEXTUAL property of the substituted file
that a grep could have reported instantly. This is that grep.
gate_main's resolve_conflicts cannot answer it, and not from carelessness: it inspects the DRAFTS
while the compiler sees the FILE THEY LAND IN -- after typedef stripping and renaming, at each
draft's own insertion offset, interleaved with declarations the file already had. Those
transformations run AFTER the conflict check passes. So substitute() gained write=False and this
tool checks the artifact itself.
Five checks, each earned by a rebuild lost this session (§176h): typedef used above its
definition; type never defined anywhere; duplicate typedef with different bodies; one symbol
declared two incompatible ways; definition contradicting a visible prototype.
CALIBRATED AGAINST THE COMPILER, NOT AGAINST C89 PEDANTRY -- and this mattered. The first version
reported 4 hard FAILUREs on the slate that had just built BYTE-IDENTICAL:
- it ignored SCOPE, but the project deliberately uses block-scope extern blocks, and a declaration
inside one function cannot conflict with a definition elsewhere. Now brace-depth aware.
- it split `void f()` from `void f(void)`, which gcc-2.7.2 accepts. Normalized.
- it called every def-vs-decl mismatch fatal, but gcc-2.7.2 accepted `void f(void*,s32)` against a
`void f(s8*,s32)` definition and even `G3P *f(...)` against `G4P *f(...)`. What it REJECTED was
a void/non-void RETURN split (func_8001ABBC). That split alone is FAIL; the rest are WARN.
Comments are masked via cdecl before any use-site scan (an unmasked scan reported 7 phantom hits).
R39 controls: the slate that banked is FAIL-free (exit 0, 3 informative warnings); four synthetic
defects each reported at FAIL; a clean text reports nothing; a block-scope extern does not
conflict; `short` vs `s16` does not conflict; array-vs-scalar does.
Wave P was the first full run of the 6k-ins doctrine: 60 cards / 6,589 ins in 2 gate groups,
59/60 claimed and 58/60 independently re-verified MATCH (6,372 ins), reloc_identity 58/58 AGREE
with ZERO symbol errors -- the second consecutive clean wave on symbol identity.
Banking cost a dozen rebuilds and exposed four more gate_main defects plus three regressions of
my own. The tool fixes, all NC'd:
- resolve_conflicts never read a draft's OWN DEFINITION, so the DEF-side wall (a draft defining
s32 func_X against a TU prototyping void func_X) reached the compiler. Now definition-aware:
it caught 13 conflicts up front where the build had been finding them one rebuild at a time.
- DECL and both typedef patterns anchored on end-of-line, so a TRAILING COMMENT hid a declaration
or typedef entirely -- and agents comment nearly everything they declare. Seventh instance of
one root cause: a scanner that looks green while reading less than it claims (R32).
- typedef handling is now BODY-AWARE and POSITION-AWARE, in a single pass:
* identical definition visible ABOVE the insertion point -> strip and reuse;
* same name, different shape -> rename (private to the draft);
* definition below the insertion point -> never reuse (it is not in scope there).
Three wrong strategies preceded this, each costing a rebuild: blanket strip (the file's copy can
sit BELOW the draft -> implicit-int, then a collision), blanket rename (breaks drafts that share
an IDENTICAL typedef, because their externs stop agreeing -- my regression, three drafts at
once), and a rescan loop that found the definition it had just renamed and stripped it as a
self-duplicate -> 'parse error before *'.
KNOWN LIMIT, recorded not fixed: the conflict check compares spelled type NAMES, so three drafts
each defining their own Slot54 with different layouts all declared func_80032A74(Slot54*) and
looked compatible. Comparing struct LAYOUTS for locally-defined types is the real fix.
13 + 4 verified-correct drafts are parked in two named buckets (competing local type models;
immovable TU declarations that gate_main reverts before every build).
36 fresh wave-O cracks + 10 recovered wave-J/K/L drafts, verified in ONE clean rebuild:
143dbb89f34491258bbc27810d0a12ec8b43a8dd BYTE-IDENTICAL. main stubs 1881 -> 1835.
Wave O was a 3-arm 49-card wave (6,266 ins): main head-crack, main UNKNOWN, overlay UNKNOWN.
47/49 standalone MATCH, independently re-verified by me (R14) at 47/49 -- exact agreement --
and reloc_identity reported 46 AGREE / 0 MISMATCH, the first wave of the campaign with zero
symbol errors. THE UNKNOWN LEVER DRAFTS LIKE ANY OTHER LANE, which matters strategically: it is
~138k ins fleet-wide (a quarter of everything open) and was routed as "needs its own lane".
FOUR gate_main defects fixed here, each of which had been silently costing prior waves drafts:
- typedef stripping walked drafts in SLATE order while substitution happens at ADDRESS order, so
the surviving typedef could land BELOW a draft using it -> "syntax error before D_800A651C".
Verified the two orders genuinely diverge for both destination files in this slate.
- a BUILD failure (sha None) fell through to a silent bisect -- a full clean rebuild per step to
rediscover what the compiler had already printed and discarded. Now the error lines are shown
and the offending drafts named for undefined-reference/redefinition/conflicting-types.
(My first version of that printer TAILED a stderr+stdout concatenation and faithfully showed 25
lines of make progress chatter instead of the error -- selecting by position, not by content.)
- typesig treated "short" and "s16" as different types (R39 over-refusal). Aliases now normalize;
11/11 NC cases pass, with signedness, volatile and array-vs-scalar still conflicting correctly.
- conflict detection ignored shared headers: engine_core.h's DEFINE_ macros declare symbols in
their own bodies, so a draft's file-scope array decl of D_800A651C was illegal. Block-scoping
the draft's extern fixes it byte-identically.
DECLARATION RECONCILIATION took the slate from 5 dropped to 0, and three of the four conflicts
were load-bearing CODEGEN, not style: the array form of D_80078D88 blocks a sched1 hoist (scalar
users adopt [0] for free); "volatile" on D_800B9A02 is required by one draft and fatal to two
others (plain u16 loses 1 bank, volatile loses 2); D_800A651C needs block scope. Cookbook §176f.
resolve_conflicts() had two defects, both found by the wave-J/K/L draft recovery:
(a) THE SYMBOL TABLE STARTED EMPTY -- only draft-vs-draft was compared, so a draft contradicting
a declaration ALREADY IN the .c reached the rebuild and surfaced only as a compile error and
a bisect. src/800.c carries 'extern void func_8001C9D0(void);' (from banked func_8001C2C4)
while three wave-J drafts declared it (s32)/(void *). The TU now seeds the table, and the
drop report names whether the clash is with the TU itself or an earlier draft.
(b) ONE NAMESPACE FOR ALL FILES -- 'seen' was global across the slate, so two drafts landing in
DIFFERENT .c files could not legally disagree about a symbol. Separate TUs are separate
namespaces; the table is now keyed per destination file (R39: over-refusal discards good work).
On the 11 recovered drafts the new check named 7 real TU conflicts that the old one missed
entirely. All 7 were repaired by adopting the TU's declaration verbatim and casting at the use
site -- including a NEW variant: when the TU's prototype takes no argument and the call must pass
one, cast through a function pointer, ((void (*)(s32))func_8001C9D0)(a0). All 11 re-verified
MATCH afterwards, so the cast is byte-identical in every case.
R39 NC: a synthetic draft re-contradicting the TU is still dropped; the repaired slate is 11/11.
- each draft compiles STANDALONE so it carries its own 'typedef struct {...} SVECTOR;'. Once
one such function banks, that typedef lives in src/800.c forever and every later draft
defining its own collides — a C89 duplicate-typedef error, not a byte miss. harvest_verify
already handles this; gate_main did not, and wave L lost a verified-correct draft to it.
- strip_dup_typedefs() drops typedefs the destination file (or an earlier body in the same
batch) already defines, and keeps novel ones.
- R39 NC: drops the duplicate, keeps the novel one, leaves the function body untouched, and is
a strict no-op when there are no duplicates.
- clean_build() ran 'make build' and then sha()'d build/us/SLUS_007.26 off disk. If the build
FAILED (compile error), the PREVIOUS successful binary was still there, so sha() returned the
good hash and the tool reported BYTE-IDENTICAL for a build that never ran.
- that is exactly how it claimed '43 banked' for wave K on a TU that did not compile; the
clean-fleet R22 caught it ([FAIL] main). A verifier that can pass without building is worse
than no verifier.
- fix: rm the output before building, and treat a non-zero make return as no-hash/never-pass.
- also: unbuffered print (a 16-min run looked hung with an empty log) and read the compile
error to name the culprit instead of bisecting at a full clean rebuild per step.
- gate_main reported wave K BYTE-IDENTICAL; the clean fleet R22 then failed [FAIL] main.
Three drafts declared D_80078D98 inconsistently (1 scalar, 2 array) and my conflict checker
could not see the difference: typesig() split on the symbol and kept only the prefix.
- fixed to retain the declarator suffix ('' vs '[]'); NC'd both directions — the wave-K
conflict is now caught, wave J's known answer (34/5) is unchanged.
- BOTH failure modes now documented in the tool: v1 too STRICT (compared parameter names,
discarded 2 good drafts), v2 too COARSE (ignored [], passed a real conflict). R22 caught
what the tool missed, which is exactly why the clean-rebuild rule exists.
- phase log: recorded the night's methodological lesson — every serious stall was an
instrument trusted without a control, never the compiler.
- main CANNOT be gated incrementally: its extract runs the EXE-only psyq_integrate +
ld_interleave steps that REWRITE the .ld, so gate_lane/gate_stage's incremental build
re-runs that on an already-rewritten script and yields a FALSE diff (R22's own rationale).
That cost a night: 4 byte-correct drafts gated 0/4 and I wrote up a nonexistent linker
defect before the null-draft control refuted it.
- gate_main substitutes the whole batch -> make extract BINARY=main -> make build -> compare
SHA. ONE clean build verifies the WHOLE batch (34 banked in one rebuild); bisects on failure
so a single bad draft can't sink the rest.
- handles both main-specific hazards: (1) in-TU cross-draft decl conflicts, resolved greedily
on TYPE SIGNATURES ONLY (comparing parameter names wrongly discards good drafts, R39 — my
own first version did exactly that); (2) stale .s after a revert (extract before resolving).
- NC'd against wave J's known answer: 34 compatible / 5 dropped, matching the hand result.
Drew T