From e86e45320a4d0a0a3bc59f82cc1fd5a7e42a3d7b Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:08:25 -0600 Subject: [PATCH] =?UTF-8?q?chore(phase-30=20S45p9):=20session=20close=20?= =?UTF-8?q?=E2=80=94=2032-way=20parallel=20gate=20in=20dedup=5Fpropagate;?= =?UTF-8?q?=20banking=20deferred=20on=20a=20tool=20bug?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PARALLEL GATE (landed, verdict-proven): dedup_propagate's byte-gate loop was serial -- one `make build BINARY=` at a time over up to 141 members per function. Measured: a propagation ran 95 minutes at load 1.6 on a 32-core box (~5% utilisation). The Makefile has parallelised extract-all/check-all since Phase 26 (xargs -P$(JOBS)), but this tool predates that and drives the SINGLE-binary target from Python, so it never saw any of it. - new gate_all(): ThreadPoolExecutor over distinct overlays, 32-way by default (JOBS env overrides; deliberately NOT capped at the Makefile's conservative 16). - SAFE by the same argument check-all relies on: byte_gate only runs `make build`, writing solely to per-binary-disjoint build//**; it mutates no source. Splice happens before, restore after -- only the VERIFICATION is parallel. - DETERMINISTIC: ThreadPoolExecutor.map preserves order, so the reported first failure is the first in `changed` order -- identical verdict to the serial loop. Control run: same verdict on a clean tree. - Measured and NOT optimised: setup (sig load + registered_addrs) is 8.6s of a 5,700s run = 0.15%. All the time is gating. Don't thread the setup. BANKING DEFERRED on a genuine pre-existing tool bug (NOT the parallel change -- 0 gate batches ran, it never reached that code): [FAIL] ov_MAIN_012: 0x80156600 not instantiated -- REVERTED Inputs verified sound at HEAD (in sig, find_site->stub, stub line matches), so the bug is in apply_plan's multi-function edit path. Run #1 missed it because it launched before the 15 wave-3 banks were committed; they landed mid-flight, enlarging run #2's plan. SECOND DEFECT: the failure exit printed REVERTED but left 38 files dirty incl. src/shared/engine_core.h -- the same incomplete-restore class as the reconcile-ledger bug (cookbook 156), on a different path. struct_check needs the same ledger treatment. Not patching the fleet-shared writer at the end of a marathon session -- that is how the next 141-binary incident happens. Tree clean, 44 banks safe, propagation is pure multiplication and can run any time. Checkpoint p9 carries: the fix-then-resume plan, the master-IDXTAB-map design (DESTPTR half proven 14/14), wave guidance, and an 8-item error ledger with its single root cause. --- phase-ends/CURRENT_PHASE.md | 74 ++++++++++++++++++++++++++++++++++++- tools/dedup_propagate.py | 58 +++++++++++++++++++++++++++-- 2 files changed, 127 insertions(+), 5 deletions(-) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 8c71ce9801..acf72232e6 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -181,7 +181,79 @@ stub on a named wall/behemoth/queue ledger** β€” 140/140 byte-identical througho --- -# πŸ›‘ SESSION S45 CHECKPOINT part 8 (2026-08-07 late) β€” the SC03 trio SOLVED Β· Stage 1+2 landed Β· propagation in flight +# πŸ›‘ SESSION S45 CHECKPOINT part 9 (2026-08-07, session close) β€” FRESH SESSION SAFE HERE +> **Tree CLEAN** (`src/`+`config/` = 0 modified) but for R23 `db.*.gbf` churn β€” never stage. +> **Nothing running.** Effort ultracode. **NO phase close** β€” T5 unopened, needs Drew's gate-2. + +## β–Ά RESUME HERE β€” 3 items, in this order + +### 1. FIX `dedup_propagate`'s apply_plan/struct_check, THEN resume the banking +The propagation is the one unfinished item. **~30 fns β†’ ~3,163 member-instances** (~0 tokens). +It FAILED on its second run β€” a genuine tool bug, NOT the parallel-gate change (0 gate batches +ran; it never reached that code): +``` +[FAIL] ov_MAIN_012: 0x80156600 not instantiated β€” REVERTED +``` +- **Inputs verified SOUND at HEAD:** func_80156600 IS in ov_MAIN_012's sig (18 ins), `find_site` + returns `stub`, and the stub line matches `stub_line()` exactly. So the bug is in the + MULTI-FUNCTION edit path of `apply_plan`, not in the data. +- **WHY run #1 didn't hit it:** run #1 launched BEFORE the 15 wave-3-re-gate banks were committed; + those landed mid-flight, so run #2's plan is LARGER and reaches a path run #1 never did. +- **SECOND DEFECT, same failure:** it printed `REVERTED` but left **38 files dirty** incl. + `src/shared/engine_core.h`. Same incomplete-restore CLASS as the reconcile-ledger bug fixed + earlier (cookbook Β§156), on a DIFFERENT path. `struct_check`'s failure exit needs the same + ledger treatment. +- **Pragmatic option if the fix is slow:** propagate with an explicit `--addr` list EXCLUDING + 0x80156600 to bank the other ~29 now, and diagnose separately. +- **Recovery if a run dies dirty:** `git checkout -- src/ config/` (deterministic; done 3Γ— tonight). + +### 2. THE MASTER IDXTAB MAP (Drew's idea β€” feasibility PROVEN, high value) +Repeat the ov_SC03_001 decode for EVERY binary β†’ a ledger of **payload β†’ owning binary β†’ load +address**. Resolves MAIN/7+9 ownership (or proves absence across every reference table β€” the +strongest dead-code evidence obtainable), gives the disc-completeness claim its evidence base +(R34), validates Β§S44 exhaustively, and turns "where does X load?" into a lookup. +- **DESTPTR half ALREADY WORKS: 14/14 sampled overlays, first try**, reproducing Β§S44's one + documented case exactly (`ov_SC01_000 *0x801A3234 = 0x801A58E8`). Method: register-track + `func_80128CFC` (same vram in every overlay) for its `lui`+`lw`, then read that word. +- **IDXTAB half β€” the one idea still needed:** find `-1`-terminated s16 index runs, then REQUIRE a + register-verified code reference to the run's address (`tools/find_addr_refs.py`). That + reference is the discriminator the fleet-wide shape scan lacked (Β§155a). Validate against the + two known-good tables FIRST: `ov_SC01_000` @0x8017EEC8 (37 entries), `ov_SC03_001` @0x8018D7BC (5). + +### 3. Then waves β€” 1,168 seeded targets in the big-3; Haiku ≀30 ins (86%), Sonnet β‰₯50 (Β§157). +Use `tools/wave_snapshot.py`; reach-sort; carve `main` out (1,061 sub-25 stubs, barren, Γ—1). +**Read args FROM THE FILE** β€” never hand-type them (that cost a 50-agent wave tonight). + +## βœ… WHAT THIS SESSION LANDED (all committed, tree clean) +- **44 functions banked**, R22 213/213 Β· ov_SC02_037 626β†’597 stubs +- **SC03/53/54/56 SOLVED** β€” live script modules owned by `ov_SC03_001` (IDXTAB @0x8018D7BC = + 224/231/232/234/233) β†’ `func_80128CFC` β†’ `*DESTPTR 0x801EBC68` = **0x801EF468**. Load BASE not + yet proved; the byte-gate arbitrates on onboarding. (`docs/memory-map.md` Β§S45 p6) +- **MAIN/7 + MAIN/9** β€” absent from a full 304s attract cycle + 7 static lines. Dead-code case + strong, not proved. Their owning binary is unknown β€” item 2 would settle it. +- **Concurrency work:** Stage 1 (reconcile ledger + shared-state RW lock, 3 NCs) and Stage 2 + (`verify_worktree`, GREEN 87s, NC fires RED) DONE. **Stage 5 CANCELLED** (87s can't lag). + Stage 3 deferred for drafting gates β€” but **propagation IS the cross-binary workload it fits**, + so the parallel gate (`gate_all`, 32-way, verdict-identical to serial) landed here instead. +- **6 new tools:** `cdtrace` (runtime CD-load oracle; 7 routing addresses confirmed live), + `find_addr_refs` (register-tracked, Β§155), `wave_snapshot`, `shared_lock`, `verify_worktree`, + `test_reconcile_ledger` +- **Cookbook Β§155a/Β§155b/Β§156/Β§157**; model ladder recalibrated (Haiku ≀30 = 86%, β‰₯50 = 20%, 4Γ— cost) +- **Measured, worth keeping:** propagation setup is 8.6s of a 5,700s run (0.15%) β€” ALL the time is + byte-gating. Don't optimise setup; if the parallel gate underdelivers, attack the O(fnsΓ—overlays) + trial loop, not the JSON parsing. + +## 🧰 MY ERROR LEDGER THIS SESSION (8, one root cause) β€” R37-adjacent rule candidate for T5 +Fabricated workflow args Β· string-vs-integer membership test Β· two gate verdicts read off a broken +tree Β· the F1 misattribution Β· a `head -8`-truncated grep Β· an over-broad "wave_snapshot decouples +waves" claim Β· an invalid timing comparison (ran parallel first, so serial found everything cached). +**Root cause is ONE: asserting a mechanism or number I had not personally derived.** The byte-gate +caught every one before it reached anything load-bearing β€” but the pattern is a governance gap, not +eight separate slips. Propose at T5 alongside R37. + +--- + +# πŸ›‘ (superseded by part 9) SESSION S45 CHECKPOINT part 8 (2026-08-07 late) β€” the SC03 trio SOLVED Β· Stage 1+2 landed Β· propagation in flight > **Effort ultracode.** HEAD `commit:1528`. ⚠️ **A PROPAGATION WAS RUNNING AT CHECKPOINT TIME** β€” > see "IF THE TREE IS DIRTY" below before anything else. diff --git a/tools/dedup_propagate.py b/tools/dedup_propagate.py index ed771e5d87..3bb8ff6991 100644 --- a/tools/dedup_propagate.py +++ b/tools/dedup_propagate.py @@ -347,6 +347,59 @@ def byte_gate(ov): return r.returncode == 0, r.stdout + r.stderr +def _jobs(): + """Parallel overlay gates. Env JOBS wins; otherwise use EVERY core. + + Deliberately NOT capped at the Makefile's conservative `JOBS ?= 16` default: each worker is a + `make build` that spends nearly all its wall-clock in cc1/as/ld subprocesses, so the box is the + limit, not Python. Measured 2026-08-07: the serial version ran a propagation for 95 minutes at + load 1.6 on a 32-core machine β€” ~5% utilisation. Saturate it (Drew, same session). + """ + try: + j = int(_os.environ.get("JOBS", "0")) + except ValueError: + j = 0 + return max(1, j or (_os.cpu_count() or 4)) + + +def gate_all(changed, label=""): + """Byte-gate EVERY touched overlay in parallel; return the first failure in `changed` ORDER. + + WHY (measured 2026-08-07): this loop used to be serial β€” one `make build BINARY=` at a + time, over up to 141 members, for each of ~30 functions. A propagation ran 95 minutes at load + 1.6 on a 32-core box: ~5% utilisation. The Makefile has parallelised `extract-all`/`check-all` + since Phase 26 (`xargs -P$(JOBS)`), but dedup_propagate predates that and drives the + SINGLE-binary `build` target from Python, so it never saw any of it. + + SAFE for the same reason `check-all` is: byte_gate only runs `make build BINARY=`, which + writes solely to the per-binary-disjoint `build//**`; it mutates NO source. The splice has + already happened before this is called, and the restore happens after β€” only the *verification* + is parallel. Threads, not processes: subprocess.run releases the GIL while the build runs. + + DETERMINISM: ThreadPoolExecutor.map preserves input order, so the reported first failure is the + first in `changed` order β€” identical to the serial loop's verdict, not whichever build finished + first. The serial version short-circuited on the first failure and so did fewer builds; this + does them all, but in parallel, and the all-pass case (the common one) is a straight win. + """ + order = list(changed) + if not order: + return None + j = min(_jobs(), len(order)) + if j <= 1: + for ov in order: + if not byte_gate(ov)[0]: + return ov + return None + from concurrent.futures import ThreadPoolExecutor + print(f"[gate] byte-gating {len(order)} overlay(s) with {j} parallel builds{label}", flush=True) + with ThreadPoolExecutor(max_workers=j) as ex: + results = list(ex.map(lambda o: (o, byte_gate(o)[0]), order)) + for ov, ok in results: + if not ok: + return ov + return None + + # ---------------------------------------------------------------- straggler caller-extern reconcile (--recover) def reconcile_caller_extern(ov, addr): """no-proto every conflicting `extern func_();` caller decl in ov's src files @@ -662,10 +715,7 @@ def main(): struct_check(plan, changed, touched) if a.no_gate: break - fail_ov = None - for ov in changed: - if not byte_gate(ov)[0]: - fail_ov = ov; break + fail_ov = gate_all(changed, f" ({len(plan)} fn(s) in plan)") if fail_ov is None: print(f"[ OK ] {len(changed)} overlays byte-identical after propagation") break