CURRENT_PHASE — Phase 24: Fix + evolve the permuter (§31-class-aware) · integration-recovery tool · scale the §31 wave

Started: 2026-07-02 · Effort: Max · Gen2 phase 16 of the arc (8→…→23→24) · Open-ended matching phase (checkpoint-close, like 15/20/21/22). Plan: approved (Drew, sequencing = permuter-first, staged). Full plan: /home/musashi/.claude/plans/plan-mode-enabled-max-tender-bengio.md. Baseline at phase start: fleet 64.86% byte-identical-from-source (~30% byte-weighted); 136/136 binaries byte-identical; 0 NON_MATCHING; dedup groups per Phase-23 close. main 143dbb89…, resident 8e17e02f…, ov_SC01_077 d19c9580….

Invariant (every task)

Whole-binary byte-gate (tools/harvest_verify.py) is the SOLE arbiter (G3/P9). Tools only reshape declarations/plumbing or PROPOSE candidates; nothing banks except a byte-identical SHA1 rebuild, verified CLEAN (R22): make clean && (extract all 136) && make check-all → 136/136; tools/dedup_integrate.py --check 0-failed; tools/progress.py --fleet monotonic up. No Ghidra DB change (R23 restart-noise — do NOT stage db.*.gbf).

Task checklist

T1 — Re-log map-wave results + backlog hygiene (xHigh)
T2 — Permuter Stage-0: floor-free masked scorer (-drz) (Max)
T3 — Permuter setup fixes: pins · typedefs · -O0 (xHigh)
T4 — Probe the flagship func_80132784 (4/400) + count-exact seeds (Max)
T5 — Permuter evolve: §31-directed mutation + grinder fixes (Max)
T5b — Fable5 §31 map-extension spike: S11 CRACKED (func_8014E048 banked; S12/S13/RC-10 → §31) (Max; Fable5) — + found/fixed the Phase-21 overlay GetTPage breakage; main-side (62 refs) tracked open
T5c — Fix the Phase-21 main/library clean-rebuild breakage (62 dangling INCLUDE_ASM refs) (Max) — renamed to curated names; true clean 136/136 restored; added tools/lint_symbol_refs.py guard
T6 — Build the integration-recovery tool (Max) ← CURRENT
T6 — Build the integration-recovery tool (Max)
T7 — Unblock the 8 compile-blocked region-a giants (xHigh)
T8 — Scale the §31 wave (breadth → prompt for /effort toggle, R27)
T9 — Distill + PhaseEnd (Max, Tier-1)

Current task: T5b. Rules check ✓ after T4 (P6); next due after T8.

Fable5 policy (Drew, 2026-07-02): Fable5 (Agent(model:fable)) is the reserved discovery / wall-breaker tier (reads the gcc-2.7.2 source; ~375k tok/giant), NOT the workhorse. Cheap-tier-first: permuter + Opus applying §31 → Fable5 only when they stall. Two uses this phase: (1) on-demand escalation on any giant the fixed permuter can't close (T4/T8 — no task); (2) T5b proactive spike to test whether S11/RC-6 are truly intrinsic. A Fable5-cracked lever goes into §31 (R16) so Opus applies it thereafter.

R14 corrections carried in (do NOT re-adopt the stale Phase-23 backlog framings)

"split-file-blind glob" is refuted — the real defect is compile.sh/match_one.py are -O2 only (can't match _o0). → -O0 compile.sh variant (T3).
"churn-without-blacklist" is refuted — the real churn is floor-victim idle re-tries (grinder.py:120 tried.clear()). → scorer (T2) + input-changed gating (T5).
func_80144090 ×1 is a type-lift case (OtBlk not in engine_types.h), NOT a straggler. → build_engine_types.py (T6).
backlog.jsonl closeness is stale/regressed for the wave seeds (true values in .run/wave/*.c headers; func_8014E048 logged failed → wave_targets skips it). → re-log first (T1).
The "~95/80/31" class counts are aspirational; real canon SCHEDULE=30/REGALLOC=84/REMAT=3, steerable near-misses scattered in OTHER/None. → wave text-scans where_stuck (T8).
gcc-papermario is gcc 2.8.1, not 2.7.2; vanilla 2.7.2 at tools/reference/gcc-2.7.2/ (cite it). loop.md on vanilla; sched/regalloc/cse map files still cite 2.8.1 lines.
No decomp-permuter submodule edit for the overhaul (scorer rebind + #pragma _permuter b64literal pin carrier + weight_overrides, all in tools/). The pin-scope randomizer pass (only submodule edit) is deferred.

Progress log

(append one line per task on completion — the crash-recovery trail)

T6 (partial) ✓ (2026-07-03): 🎯 FLAGSHIP func_80132784 PROPAGATED ×134 (commit:0443) — the 400-ins S11 giant (cracked T4, banked ×1) now byte-identical in ALL 134 overlays, +1.6% byte-weighted (the single biggest lever; function-count fleet 64.86→64.90%). The T6 recovery pattern, proven end-to-end: the T4-deferred ×1 cap was one straggler overlay (ov_SC02_005) declaring a CONFLICTING caller extern void func_80132784(s32,s32,s32) vs the def (s32,s32,u32) → conflicting types → dedup_propagate's all-or-nothing single---addr plan dropped the whole fn. Fix: reconcile the straggler's caller extern to the def's canonical sig (s32→u32, byte-neutral — the call site casts anyway), then propagate → 134 overlays rebuilt byte-identical, group E_func_80132784 registered, dedup-check 1784/0. Clean R22 fleet check-all 136/136. REMAINING T6 (the tool): automate this — detect a straggler's conflicting caller decl → reconcile to the def's canonical sig → re-gate → propagate full (vs. all-or-nothing drop); + the other ×1 classes (func_8014E048 pin/asm self-containment — a DIFFERENT blocker: dedup_propagate's compiles_standalone rejects pin/asm bodies; func_80144090 OtBlk type-lift; func_8014F4C0/func_80155800 TU-context leaf-MATCHes in .run/wave/).
T5c ✓ (2026-07-03): 🟢 TRUE clean fleet 136/136 RESTORED (first genuinely-clean make clean+full-re-extract+check-all green since Phase 21). Fixed the pre-existing Phase-21 (commit:0292) rename-propagation breakage: renamed all 62 stale INCLUDE_ASM(func_<ADDR>) → curated names across 12 main/library src/*.c (800c.c/800c3.c×22/apicard1·2·4/boot.c/libcd1×8/libetc×6/libgpu×3/sgap.c/snd1×7/snd2.c) to match splat's canonical output (proven: splat's fresh stub-regen uses the curated names). main clean-builds 143dbb89 ✓; fleet 136/136 ✓ (all overlays byte-identical, resident 8e17e02f). Built tools/lint_symbol_refs.py (comment/string-aware, func_+D_) — flags any func_/D_<ADDR> ref whose addr was renamed (the R22-masked class); caught the T5b macro-call + this INCLUDE_ASM set; now 0 stale. SETUP inventory + CURRENT_PHASE finding updated. Commit: (checkpoint).
T5b ✓ (2026-07-03): 🎯 S11 CRACKED — the class verdict was map-incompleteness, not intrinsic codegen (Fable5 main-loop spike, Drew switched the session to Fable5 Max). func_8014E048 (143 ins, reach-134; "S11 LUID⊗alloc intrinsic / not source-steerable" since the map wave; 28-off even after T5's directed permuter) → MATCH (143/143) + whole-binary BANKED (harvest_verify 1/0, ov_SC01_077 d19c9580…). Derived by reading gcc-2.7.2 source + -dl -dg -dS RTL dumps (12 experiments, .run/gccmap/exp/e1a..e1k.c): S12 reused-s32-temp fence (output/anti deps forbid load batching; u16 temps DON'T work — unpromoted HI vars spawn per-use zext temps that combine folds away), S13 head-skip escape (body-local param copies keep hard arg regs live into the temp windows → conflict-steer the scratch contest; zero-byte volatile-asm dead-read as a wedge-slot fence; multi-input dead-read to rebalance K2 densities), cse-opaque asm-copy (addu %0,%1,$zero) for must-materialize pointer copies, RC-4b pinned store-temp, RC-10 preference-cascade mechanics (set_preference one-level expression unwrap; expand_preferences dying-into-set merge; find_reg pass-0 used-so-far/someone-prefers + pref override). Integration needed 2 hand-reconciles (both T6 classes): engine_core.h:17569 caller-decl s16*→u16* (ptr param type codegen-neutral) + canonical data decls with *(u16*) access casts (D_801152A8 u8[] / D_801152AC s16). ×134 lift blocked by dedup_propagate self-containment (pins/asm) → T6 target (joins func_80132784). Distilled (R30): sched.md §6/S12/S13 + regalloc.md §F/RC-10 + cookbook §31 triage-table S11 downgrade. Backlog re-logged (capped). R22 clean fleet check-all running at close. Commit: (checkpoint).
T5 ✓ (2026-07-03): §31-directed permuter mutation + grinder input-changed gating — built, validated, distilled. NEW tools/permuter_weights.py: classify(klass,where) → regalloc|schedule|cse|None + render_settings_toml emitting the [weight_overrides] table decomp-permuter merges over the gcc defaults (main.py:336 / helpers.py:merge — per-key REPLACE; no submodule edit, R3/R20). Biases pass-selection toward each class's §31 levers (perm_reorder_decls RC-1/3 · perm_reorder_stmts RC-2/S1 · perm_temp_for_expr S2 · perm_commutative cse) and pushes the value/type noise to ~0.1. Wired into p16_permute.setup(…, klass=, where=) (+--klass, backlog auto-lookup) and grinder.py (auto-threads klass/where_stuck; klass=None → gcc defaults = the pre-T5 undirected search, a safe superset). Grinder fix (R14): replaced the blind tried.clear() idle churn with input-changed gating (draft_sig=(best_draft mtime, closeness) — re-open a fn only when the worker improved it; the permuter is deterministic given base.c+target.o). VALIDATED: the regalloc profile drove func_8014E048 base masked-36 → 29 (match_one -drz 35→28) where the undirected search stalled — real directed progress; re-logged (closeness 28, improved draft promoted). The 4 flagship count-exact seeds are the project's WORST-CASE intrinsic S11/RC-6 walls ("C-space discontinuous" §31 RC-6) — directed mutation improves but doesn't gate them → T5b/backlog fuel (§31 "two probes, don't grind"). Distilled: cookbook §3b + SETUP inventory (R16/R30). No build-input changed → 136/136 fleet invariant untouched (tools/docs only). Commit: (checkpoint).
T4 ✓ (2026-07-02): 🎯 FLAGSHIP BANKED — func_80132784 (400 ins, "HARD-DEFER/irreducible" for 22 phases) matched whole-binary + committed (commit:0438); make check BINARY=ov_SC01_077 BYTE-IDENTICAL, dedup-check 1783/0. The permuter overhaul (T2+T3) is validated end-to-end by a real bank. Banked ×1 — the ×134 propagation is capped by per-overlay declaration plumbing (ov_SC02_005 cross-overlay straggler + an extern/type mismatch), NOT codegen → T6 target (the +1.6% byte-weight lands when T6's integration-recovery reconciles it). Hand-fix that unblocked the ×1: dropped the draft's redundant Blk16 typedef (already in engine_types.h:442) — T6 automates this. Count-exact seeds (func_8014E048 35 / func_80176D94 52 / func_80148094 72 / func_801412A8 110): permuter setup works (parse/compile/search clean, 0 internal failures), but random mutation didn't close them (func_80148094 only 77→74 in T2) — they're intrinsic residuals (RC-6/S11) → T5 directed mutation + T5b Fable5. Fleet 64.86% (×1 negligible). Commit: (checkpoint).
T3 ✓ (2026-07-02): permuter setup fixes (all in tools/, no submodule edit): p16_permute.hide_asm — a b64-pragma carrier for register pins AND GTE __asm__ blocks (pycparser parses the pragma; decomp-permuter's own process_pragmas decodes it back so cc1 sees the real asm); custom-typedef/#define preservation (drop only #include); f32 typedef gap fixed; compile_o0.sh -O0 variant auto-selected for _o0 targets; run_masked.py pin-var fallback (expr_type→int on an unknown/hidden-pin id → 0 "internal permuter failures", was >0). ROOT CAUSE of the GTE fail: compile.sh now prepends .include "macro.inc" so mvmva assembles (the real build gets it via include_asm.h, which base.c omits + -DPERMUTER disables) — byte-neutral for non-GTE (count-exact scores unchanged). ALL 5 seeds parse+compile (base 4/110/36/52/77). 🎯 The flagship func_80132784 (4/400) CLOSED to a masked-0 that match_one independently confirms MATCH (400 ins) — winner preserved .run/wave/func_80132784.win.c for the T4 whole-binary gate → ×134 (≈ +1.6% byte-weight). No build-input changed. Commit: (checkpoint).
T2 ✓ (2026-07-02): built the floor-free relocation-masked scorer, ENTIRELY in the tools/ layer (no decomp-permuter submodule edit): tools/masked_diff.py (shared objdump -drz oracle + masking), refactored tools/match_one.py to use it (-dr→-drz), tools/masked_scorer.py (MaskedScorer drop-in), tools/permuter/run_masked.py (rebinds src.main.Scorer), wired p16_permute.run_permuter. VALIDATED: masked_self=0; masked_cand≈match_one (77~~72/36~~35/52=52); STOCK floor 1750–2100 vs masked 36–77 (the wander cause, removed); live permuter base score = masked 77 (not stock 1930), descends to 74. -drz also fixed match_one's GTE under-count (regression-clean on count-exact seeds; func_80132784 now true 400 ins / 4-off). match_one output format unchanged → gate_stage/grinder compatible. No build-input changed. Commit: (checkpoint). T3 next handles the 2 compile FAILs surfaced (func_80132784 GTE mvmva/common.h; func_801412A8 custom-typedef drop).
T1 ✓ (2026-07-02): re-logged 7 map-wave seeds with BYTE-VERIFIED closeness (dropped 19 stale/unreproducible records from .run/backlog.jsonl, .bak kept; best_draft → .run/backlog_drafts/). Ground truth (match_one -dr): func_8014E048=35 (was mis-logged failed), func_80176D94=52, func_80148094=72 (best draft = vG2.c, NOT the named file), func_801412A8=110; T6 leaf-MATCHes func_8014F4C0/func_80155800=0. func_80144090 excluded (banked ×1, not a stub → T6 OtBlk type-lift). Fixed gate_stage sig_unify/--src-file doc-drift. wave_targets --class REGALLOC + grinder now surface all 4 count-exact permuter seeds with true closeness; func_80132784 correctly kept out of blind selection. No build-input changed → 136/136 invariant untouched. Commit: (checkpoint).

T6 targets (banked leaf-MATCH, ×N propagation capped — declaration plumbing)

func_80132784 — banked ×1 in ov_SC01_077_a.c; ×134 propagation blocked by ov_SC02_005 cross-overlay straggler + an extern/type mismatch. The auto-from batch is all-or-nothing (poisoned by pre-existing capped fns 0x8012A018/0x80165CA0); T6's tool must (a) drop-straggler per-function cleanly, (b) reconcile the extern/type conflict per overlay. The redundant-Blk16-typedef reconcile is the class-c type dedup T6 automates.
Carried from Phase-23 backlog: func_8014F4C0 (fwd-decl reconcile + Vec4u lift), func_80155800 (TU-context), func_80144090 (OtBlk type-lift). Test cases in .run/wave/.

Blockers / open findings

🔴🔴 MAJOR PRE-EXISTING (Phase-21) LATENT BREAKAGE — a truly-clean rebuild has been broken since Phase-21 close (commit:0292), masked by incremental builds. FOUND during T5b's R22 fleet verify (2026-07-03). NEEDS DREW'S CALL (P5 stop — touches the crown-jewel main EXE + the Phase-7/8/9 PsyQ library-build machinery + the project's central byte-identical-clean-rebuild invariant).
- Symptom: make clean + full re-extract + make check-all → main FAILS to build (can't open asm/nonmatchings/800c3/func_8005CE18.s … dozens); overlays failed 134× on undefined reference to func_80058B40. Only main's failure remains after the overlay fix (135/136).
- Root cause: Phase-21 xdedup (commit:0292, "+62 PsyQ names") renamed 62 PsyQ library functions to their proper names (InitHeap, FlushCache, GetTPage, SysEnqIntRP, SpuWrite, _SpuInit, CdMix, __main …) in symbols.us.txt, but did NOT propagate the rename to the committed source that references them by the OLD func_<ADDR> name: (a) 1 shared macro CALL in engine_core.h (func_80058B40, overlay-side) and (b) 62 INCLUDE_ASM(func_<ADDR>) stub lines across 12 main/library src/*.c (800c.c, 800c3.c×22, apicard1/2/4, boot.c, libcd1.c×8, libetc.c×6, libgpu.c×3, sgap.c, snd1.c×7, snd2.c). Splat now names/handles those addresses by the curated name (linked-not-stubbed via psyq_integrate, or emitted as <curated>.s), so the stale func_<ADDR> refs dangle. Incremental builds reused stale pre-rename .s/.o and masked it → every "check-all 136/136 / main 143dbb89" in Phases 21/22/23 was incrementally-stale, NEVER a genuinely-clean tree (the exact R22 failure mode).
- FIXED so far (overlay-side, byte-neutral, verified): the 1 shared-macro call func_80058B40 → GetTPage in engine_core.h (same addr 0x80058b40 → identical jal); took the fleet 135→ (134 overlays now pass); ov_SC07_009 byte-identical 2a6499b6. Static scan of engine_core.h+engine_types.h (func_ AND D_): this was the ONLY shared-header collision.
- FIXED (main/library-side, 62 refs — T5c, Drew: rename to curated names): the root mechanism, byte-proven: splat's FRESH regeneration of a stub .c uses the CURATED names (move src/800c3.c aside → splat writes INCLUDE_ASM(InitHeap) + emits InitHeap.s); the committed src/*.c were simply STALE (func_<ADDR>) — Phase-21 renamed the symbols but never regenerated/renamed the committed stub refs. Fix = rename the 62 INCLUDE_ASM(func_<ADDR>) → the curated name (InitHeap, FlushCache, GetTPage, SysEnqIntRP, SpuWrite, CdMix, __main …) across the 12 files (800c.c 1, 800c3.c 22, apicard1/2/4 4/6/2, boot.c 1, libcd1.c 8, libetc.c 6, libgpu.c 3, sgap.c 1, snd1.c 7, snd2.c 1). main clean-builds 143dbb89 ✓; full clean fleet check-all confirming 136/136. (NOT hex-case, NOT a proto-dup — only symbols.us.txt is stacked.)
- Lesson (→ cookbook, T9): a symbols.us.txt rename MUST be propagated to (a) shared-macro bodies AND (b) INCLUDE_ASM stub refs, AND verified by a genuinely-clean (make clean + full re-extract) check-all, never incremental. Add a lint: every func_<ADDR> referenced in committed src/ must have a matching emitted .s OR a defined symbol. Also: T5b's interim harvest_verify on func_8014E048 passed on a stale _a.o → re-verified clean (ov_SC01_077 was among the 135 that pass).
RESOLVED in T2 (R14 self-correction): the T1 finding that "func_80132784 is 204-off / not 4/400" was itself the -dr artifact. Once match_one uses -drz (T2), func_80132784 reads its true 400 ins and is 4/400 count-exact — the Phase-23 "4 instructions away" premise is CONFIRMED. Backlog re-logged closeness 4 (source T2-drz). The residual is a prologue save/init order swap (S7/S11) — a pure register/schedule swap, prime permuter/pin + T4 target. The realistic count-exact near-permuter seeds are: func_80132784 (4), func_8014E048 (35), func_80176D94 (52), func_80148094/vG2 (72), func_801412A8 (110). S11 blocks func_801412A8 + (partly) func_80132784 → T5b Fable5 spike still well-motivated. Lesson: the -drz fix (T2) is load-bearing for measurement honesty, not just the permuter's gradient.

19 KiB Raw Blame History Unescape Escape