mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-01 07:40:42 -04:00
chore(phase-30 S45p9): session close — 32-way parallel gate in dedup_propagate; banking deferred on a tool bug
PARALLEL GATE (landed, verdict-proven): dedup_propagate's byte-gate loop was serial --
one `make build BINARY=<ov>` at a time over up to 141 members per function. Measured: a
propagation ran 95 minutes at load 1.6 on a 32-core box (~5% utilisation). The Makefile
has parallelised extract-all/check-all since Phase 26 (xargs -P$(JOBS)), but this tool
predates that and drives the SINGLE-binary target from Python, so it never saw any of it.
- new gate_all(): ThreadPoolExecutor over distinct overlays, 32-way by default (JOBS env
overrides; deliberately NOT capped at the Makefile's conservative 16).
- SAFE by the same argument check-all relies on: byte_gate only runs `make build`, writing
solely to per-binary-disjoint build/<bin>/**; it mutates no source. Splice happens before,
restore after -- only the VERIFICATION is parallel.
- DETERMINISTIC: ThreadPoolExecutor.map preserves order, so the reported first failure is
the first in `changed` order -- identical verdict to the serial loop. Control run: same
verdict on a clean tree.
- Measured and NOT optimised: setup (sig load + registered_addrs) is 8.6s of a 5,700s run
= 0.15%. All the time is gating. Don't thread the setup.
BANKING DEFERRED on a genuine pre-existing tool bug (NOT the parallel change -- 0 gate
batches ran, it never reached that code):
[FAIL] ov_MAIN_012: 0x80156600 not instantiated -- REVERTED
Inputs verified sound at HEAD (in sig, find_site->stub, stub line matches), so the bug is
in apply_plan's multi-function edit path. Run #1 missed it because it launched before the
15 wave-3 banks were committed; they landed mid-flight, enlarging run #2's plan.
SECOND DEFECT: the failure exit printed REVERTED but left 38 files dirty incl.
src/shared/engine_core.h -- the same incomplete-restore class as the reconcile-ledger bug
(cookbook 156), on a different path. struct_check needs the same ledger treatment.
Not patching the fleet-shared writer at the end of a marathon session -- that is how the
next 141-binary incident happens. Tree clean, 44 banks safe, propagation is pure
multiplication and can run any time.
Checkpoint p9 carries: the fix-then-resume plan, the master-IDXTAB-map design (DESTPTR half
proven 14/14), wave guidance, and an 8-item error ledger with its single root cause.
This commit is contained in:
@@ -181,7 +181,79 @@ stub on a named wall/behemoth/queue ledger** — 140/140 byte-identical througho
|
||||
|
||||
---
|
||||
|
||||
# 🛑 SESSION S45 CHECKPOINT part 8 (2026-08-07 late) — the SC03 trio SOLVED · Stage 1+2 landed · propagation in flight
|
||||
# 🛑 SESSION S45 CHECKPOINT part 9 (2026-08-07, session close) — FRESH SESSION SAFE HERE
|
||||
> **Tree CLEAN** (`src/`+`config/` = 0 modified) but for R23 `db.*.gbf` churn — never stage.
|
||||
> **Nothing running.** Effort ultracode. **NO phase close** — T5 unopened, needs Drew's gate-2.
|
||||
|
||||
## ▶ RESUME HERE — 3 items, in this order
|
||||
|
||||
### 1. FIX `dedup_propagate`'s apply_plan/struct_check, THEN resume the banking
|
||||
The propagation is the one unfinished item. **~30 fns → ~3,163 member-instances** (~0 tokens).
|
||||
It FAILED on its second run — a genuine tool bug, NOT the parallel-gate change (0 gate batches
|
||||
ran; it never reached that code):
|
||||
```
|
||||
[FAIL] ov_MAIN_012: 0x80156600 not instantiated — REVERTED
|
||||
```
|
||||
- **Inputs verified SOUND at HEAD:** func_80156600 IS in ov_MAIN_012's sig (18 ins), `find_site`
|
||||
returns `stub`, and the stub line matches `stub_line()` exactly. So the bug is in the
|
||||
MULTI-FUNCTION edit path of `apply_plan`, not in the data.
|
||||
- **WHY run #1 didn't hit it:** run #1 launched BEFORE the 15 wave-3-re-gate banks were committed;
|
||||
those landed mid-flight, so run #2's plan is LARGER and reaches a path run #1 never did.
|
||||
- **SECOND DEFECT, same failure:** it printed `REVERTED` but left **38 files dirty** incl.
|
||||
`src/shared/engine_core.h`. Same incomplete-restore CLASS as the reconcile-ledger bug fixed
|
||||
earlier (cookbook §156), on a DIFFERENT path. `struct_check`'s failure exit needs the same
|
||||
ledger treatment.
|
||||
- **Pragmatic option if the fix is slow:** propagate with an explicit `--addr` list EXCLUDING
|
||||
0x80156600 to bank the other ~29 now, and diagnose separately.
|
||||
- **Recovery if a run dies dirty:** `git checkout -- src/ config/` (deterministic; done 3× tonight).
|
||||
|
||||
### 2. THE MASTER IDXTAB MAP (Drew's idea — feasibility PROVEN, high value)
|
||||
Repeat the ov_SC03_001 decode for EVERY binary → a ledger of **payload → owning binary → load
|
||||
address**. Resolves MAIN/7+9 ownership (or proves absence across every reference table — the
|
||||
strongest dead-code evidence obtainable), gives the disc-completeness claim its evidence base
|
||||
(R34), validates §S44 exhaustively, and turns "where does X load?" into a lookup.
|
||||
- **DESTPTR half ALREADY WORKS: 14/14 sampled overlays, first try**, reproducing §S44's one
|
||||
documented case exactly (`ov_SC01_000 *0x801A3234 = 0x801A58E8`). Method: register-track
|
||||
`func_80128CFC` (same vram in every overlay) for its `lui`+`lw`, then read that word.
|
||||
- **IDXTAB half — the one idea still needed:** find `-1`-terminated s16 index runs, then REQUIRE a
|
||||
register-verified code reference to the run's address (`tools/find_addr_refs.py`). That
|
||||
reference is the discriminator the fleet-wide shape scan lacked (§155a). Validate against the
|
||||
two known-good tables FIRST: `ov_SC01_000` @0x8017EEC8 (37 entries), `ov_SC03_001` @0x8018D7BC (5).
|
||||
|
||||
### 3. Then waves — 1,168 seeded targets in the big-3; Haiku ≤30 ins (86%), Sonnet ≥50 (§157).
|
||||
Use `tools/wave_snapshot.py`; reach-sort; carve `main` out (1,061 sub-25 stubs, barren, ×1).
|
||||
**Read args FROM THE FILE** — never hand-type them (that cost a 50-agent wave tonight).
|
||||
|
||||
## ✅ WHAT THIS SESSION LANDED (all committed, tree clean)
|
||||
- **44 functions banked**, R22 213/213 · ov_SC02_037 626→597 stubs
|
||||
- **SC03/53/54/56 SOLVED** — live script modules owned by `ov_SC03_001` (IDXTAB @0x8018D7BC =
|
||||
224/231/232/234/233) → `func_80128CFC` → `*DESTPTR 0x801EBC68` = **0x801EF468**. Load BASE not
|
||||
yet proved; the byte-gate arbitrates on onboarding. (`docs/memory-map.md` §S45 p6)
|
||||
- **MAIN/7 + MAIN/9** — absent from a full 304s attract cycle + 7 static lines. Dead-code case
|
||||
strong, not proved. Their owning binary is unknown — item 2 would settle it.
|
||||
- **Concurrency work:** Stage 1 (reconcile ledger + shared-state RW lock, 3 NCs) and Stage 2
|
||||
(`verify_worktree`, GREEN 87s, NC fires RED) DONE. **Stage 5 CANCELLED** (87s can't lag).
|
||||
Stage 3 deferred for drafting gates — but **propagation IS the cross-binary workload it fits**,
|
||||
so the parallel gate (`gate_all`, 32-way, verdict-identical to serial) landed here instead.
|
||||
- **6 new tools:** `cdtrace` (runtime CD-load oracle; 7 routing addresses confirmed live),
|
||||
`find_addr_refs` (register-tracked, §155), `wave_snapshot`, `shared_lock`, `verify_worktree`,
|
||||
`test_reconcile_ledger`
|
||||
- **Cookbook §155a/§155b/§156/§157**; model ladder recalibrated (Haiku ≤30 = 86%, ≥50 = 20%, 4× cost)
|
||||
- **Measured, worth keeping:** propagation setup is 8.6s of a 5,700s run (0.15%) — ALL the time is
|
||||
byte-gating. Don't optimise setup; if the parallel gate underdelivers, attack the O(fns×overlays)
|
||||
trial loop, not the JSON parsing.
|
||||
|
||||
## 🧰 MY ERROR LEDGER THIS SESSION (8, one root cause) — R37-adjacent rule candidate for T5
|
||||
Fabricated workflow args · string-vs-integer membership test · two gate verdicts read off a broken
|
||||
tree · the F1 misattribution · a `head -8`-truncated grep · an over-broad "wave_snapshot decouples
|
||||
waves" claim · an invalid timing comparison (ran parallel first, so serial found everything cached).
|
||||
**Root cause is ONE: asserting a mechanism or number I had not personally derived.** The byte-gate
|
||||
caught every one before it reached anything load-bearing — but the pattern is a governance gap, not
|
||||
eight separate slips. Propose at T5 alongside R37.
|
||||
|
||||
---
|
||||
|
||||
# 🛑 (superseded by part 9) SESSION S45 CHECKPOINT part 8 (2026-08-07 late) — the SC03 trio SOLVED · Stage 1+2 landed · propagation in flight
|
||||
> **Effort ultracode.** HEAD `commit:1528`. ⚠️ **A PROPAGATION WAS RUNNING AT CHECKPOINT TIME** —
|
||||
> see "IF THE TREE IS DIRTY" below before anything else.
|
||||
|
||||
|
||||
@@ -347,6 +347,59 @@ def byte_gate(ov):
|
||||
return r.returncode == 0, r.stdout + r.stderr
|
||||
|
||||
|
||||
def _jobs():
|
||||
"""Parallel overlay gates. Env JOBS wins; otherwise use EVERY core.
|
||||
|
||||
Deliberately NOT capped at the Makefile's conservative `JOBS ?= 16` default: each worker is a
|
||||
`make build` that spends nearly all its wall-clock in cc1/as/ld subprocesses, so the box is the
|
||||
limit, not Python. Measured 2026-08-07: the serial version ran a propagation for 95 minutes at
|
||||
load 1.6 on a 32-core machine — ~5% utilisation. Saturate it (Drew, same session).
|
||||
"""
|
||||
try:
|
||||
j = int(_os.environ.get("JOBS", "0"))
|
||||
except ValueError:
|
||||
j = 0
|
||||
return max(1, j or (_os.cpu_count() or 4))
|
||||
|
||||
|
||||
def gate_all(changed, label=""):
|
||||
"""Byte-gate EVERY touched overlay in parallel; return the first failure in `changed` ORDER.
|
||||
|
||||
WHY (measured 2026-08-07): this loop used to be serial — one `make build BINARY=<ov>` at a
|
||||
time, over up to 141 members, for each of ~30 functions. A propagation ran 95 minutes at load
|
||||
1.6 on a 32-core box: ~5% utilisation. The Makefile has parallelised `extract-all`/`check-all`
|
||||
since Phase 26 (`xargs -P$(JOBS)`), but dedup_propagate predates that and drives the
|
||||
SINGLE-binary `build` target from Python, so it never saw any of it.
|
||||
|
||||
SAFE for the same reason `check-all` is: byte_gate only runs `make build BINARY=<ov>`, which
|
||||
writes solely to the per-binary-disjoint `build/<bin>/**`; it mutates NO source. The splice has
|
||||
already happened before this is called, and the restore happens after — only the *verification*
|
||||
is parallel. Threads, not processes: subprocess.run releases the GIL while the build runs.
|
||||
|
||||
DETERMINISM: ThreadPoolExecutor.map preserves input order, so the reported first failure is the
|
||||
first in `changed` order — identical to the serial loop's verdict, not whichever build finished
|
||||
first. The serial version short-circuited on the first failure and so did fewer builds; this
|
||||
does them all, but in parallel, and the all-pass case (the common one) is a straight win.
|
||||
"""
|
||||
order = list(changed)
|
||||
if not order:
|
||||
return None
|
||||
j = min(_jobs(), len(order))
|
||||
if j <= 1:
|
||||
for ov in order:
|
||||
if not byte_gate(ov)[0]:
|
||||
return ov
|
||||
return None
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
print(f"[gate] byte-gating {len(order)} overlay(s) with {j} parallel builds{label}", flush=True)
|
||||
with ThreadPoolExecutor(max_workers=j) as ex:
|
||||
results = list(ex.map(lambda o: (o, byte_gate(o)[0]), order))
|
||||
for ov, ok in results:
|
||||
if not ok:
|
||||
return ov
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- straggler caller-extern reconcile (--recover)
|
||||
def reconcile_caller_extern(ov, addr):
|
||||
"""no-proto every conflicting `extern <ret> func_<ADDR>(<params>);` caller decl in ov's src files
|
||||
@@ -662,10 +715,7 @@ def main():
|
||||
struct_check(plan, changed, touched)
|
||||
if a.no_gate:
|
||||
break
|
||||
fail_ov = None
|
||||
for ov in changed:
|
||||
if not byte_gate(ov)[0]:
|
||||
fail_ov = ov; break
|
||||
fail_ov = gate_all(changed, f" ({len(plan)} fn(s) in plan)")
|
||||
if fail_ov is None:
|
||||
print(f"[ OK ] {len(changed)} overlays byte-identical after propagation")
|
||||
break
|
||||
|
||||
Reference in New Issue
Block a user