From 85f6b79471e1b414098b8eff07b7d9a85f54aa97 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 25 Aug 2026 00:03:45 -0600 Subject: [PATCH] =?UTF-8?q?feat(maintenance):=20a=20periodic=20fleet=20R22?= =?UTF-8?q?=20=E2=80=94=20nothing=20was=20watching=20the=20binaries=20no?= =?UTF-8?q?=20lane=20touches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two binaries sat RED for hours today and nothing noticed: ov_SC07_010 from a maintenance commit whose final tree state was provably never built, and ov_SC07_002 from a stale 2-table jtbl pad spec written at wave bp. Every lane only ever checks the binary it is currently touching, so a byte-gate — a correctness oracle — was silent about everything it did not build. They were found by accident, by an agent's scoped R22 sweeping 141 binaries. Every 4th maintenance pass (~3h), skipped while any gate is in flight (check-all rebuilds stale objects and must not race a gate), it runs the fleet check and writes any REDs to .run/fleet_red.txt with a loud log line. It FIXES NOTHING: a wrong repair to a pad spec or a config is exactly how a silent byte shift gets committed, and the two we fixed today each needed a different, evidence-led remedy. --- tools/lanes/maintenance.sh | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tools/lanes/maintenance.sh b/tools/lanes/maintenance.sh index 5050892da6..b2ccd8a101 100644 --- a/tools/lanes/maintenance.sh +++ b/tools/lanes/maintenance.sh @@ -80,6 +80,28 @@ PY say "nothing banked this pass" fi fi + # PERIODIC FLEET CHECK (P31 S59). Two binaries sat RED for hours — ov_SC07_010 from a commit whose + # tree state was never built, ov_SC07_002 from a stale 2-table jtbl pad spec — and NOTHING noticed, + # because every lane only ever checks the binary it is currently touching. A byte-gate is a + # correctness oracle with a null coverage model: it is silent about everything it did not build. + # So sweep the whole fleet on a slow cadence, report REDs loudly, and FIX NOTHING automatically — + # a wrong repair to a pad spec or a config is exactly how a silent byte shift gets committed. + # Every 4th pass (~3 h). Skipped while any gate is in flight: check-all rebuilds stale objects and + # must not race a gate's build for the same binary. + FC=$(cat .run/maint_fleet_count 2>/dev/null || echo 0); FC=$((FC+1)); echo "$FC" > .run/maint_fleet_count + if [ $((FC % 4)) -eq 0 ] && ! pgrep -f 'tools/sweep_parallel|tools/gate_stage|tools/gate_main' >/dev/null; then + say "fleet R22 sweep (every 4th pass) — this checks binaries no lane has touched" + make check-all JOBS=12 >.run/fleet_check.log 2>&1 || true + grep -E "^\[FAIL\]" .run/check-all.txt 2>/dev/null | awk '{print $2}' > .run/fleet_red.txt || true + NRED=$(grep -c . .run/fleet_red.txt 2>/dev/null || echo 0) + if [ "$NRED" -gt 0 ]; then + say "*** $NRED BINARY/BINARIES ARE RED — see .run/fleet_red.txt (NOT auto-fixed, by design) ***" + head -8 .run/fleet_red.txt | sed 's/^/ RED: /' + else + say "fleet R22: all binaries byte-identical ($(grep -c '^\[ OK \]' .run/check-all.txt 2>/dev/null || echo 0) checked)" + fi + fi + say "pass complete; sleeping 45m" sleep 2700 done