From 7fbdb8fd6314097838caf6c0d4a6f0dfdafccf1d Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Fri, 4 Sep 2026 20:59:35 -0600 Subject: [PATCH] =?UTF-8?q?fix(phase-31):=20S80=20#9c=20=E2=80=94=20the=20?= =?UTF-8?q?permuter=20could=20not=20permute=20a=20PINNED=20seed,=20and=20i?= =?UTF-8?q?t=20was=20our=20instrument:=20hide=5Fasm=20carried=20only=20the?= =?UTF-8?q?=20`=5F=5Fasm=5F=5F`=20spelling=20(3=20S79=20seeds=20use=20`asm?= =?UTF-8?q?("$7")`),=20permuter=5Fils=20warm-restarted=20from=20the=20DECO?= =?UTF-8?q?DED=20waypoint=20(raw=20pins=20back=20in=20base.c=20=E2=86=92?= =?UTF-8?q?=20cycles=202..N=20were=20silent=20parser=20refusals=20reported?= =?UTF-8?q?=20"(unchanged)"),=20and=20defines=5Ffn=20refused=20K&R-style?= =?UTF-8?q?=20definitions=20(436=20stored=20backlog=20drafts=20kept=20out?= =?UTF-8?q?=20of=20the=20lane=20for=20four=20phases).=20Fixed=20+=20R39-co?= =?UTF-8?q?ntrolled=20over=205,311=20drafts=20(the=20bare=20word=20`asm`?= =?UTF-8?q?=20in=20INCLUDE=5FASM=20path=20strings=20was=20a=20caught=20fal?= =?UTF-8?q?se=20positive):=20re-hide=20every=20waypoint,=20assert=20the=20?= =?UTF-8?q?definition=20survived,=20abort=20exit-2=20on=20a=20refusal=20(R?= =?UTF-8?q?61a),=20flushed=20logs=20(R55).=20Every=20S79=20pinned=20seed?= =?UTF-8?q?=20now=20iterates;=20ov=5FSC06=5F022:func=5F8017DF28=20(pinned?= =?UTF-8?q?=20WALL,=20closeness=202)=20reached=201=20in=20its=20first=20cy?= =?UTF-8?q?cle.=20cookbook=20=C2=A7493=20S80=20correction=20+=20=C2=A7494?= =?UTF-8?q?=20v1=20(S79=20idioms);=20SETUP=20rows=20(p16=5Fpermute/permute?= =?UTF-8?q?r=5Fils,=20agent=5Fverdicts.py)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/SETUP.md | 2 ++ docs/cookbook-index.md | 7 +++-- docs/matching-cookbook.md | 60 +++++++++++++++++++++++++++++++++++++++ tools/p16_permute.py | 16 +++++++++-- tools/permuter_ils.py | 27 ++++++++++++++++-- 5 files changed, 106 insertions(+), 6 deletions(-) diff --git a/docs/SETUP.md b/docs/SETUP.md index 791e0ba91..5b53b20c5 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -1004,6 +1004,8 @@ fills fast). Nothing is leaking — but the host does not get the memory back on | `tools/exclude_audit.py` | LINKED now outranks a pinned `# WALL` (a function in a linked PsyQ subseg is never a target, whatever compiler fact was recorded about it): dropped the four §332 walls + `func_80062144` + `PopMatrix`/`PushMatrix` (libgte3 since Phase 8) | every regeneration of `config/wave_exclude.txt` | | `tools/frontier_classify.py` | its LINKED subseg set is DERIVED from the Makefile (`progress._main_linked_segs_from_makefile`) instead of a hard-coded 49-name literal that missed every block wired since it was written (R51; it reported 337 "stubs" after task #5) | every census | | `tools/p16_permute.py` (task #8) | `run_permuter` no longer swallows run_masked's output: a C-parser refusal of the seed (`register … asm("$7")` pins) is printed as `[permuter] REFUSED : Syntax error …` and recorded in `/PERMUTER_REFUSED.txt` instead of reading as "no improvement" (cookbook §493) | every `permuter_ils` / `permuter_sweep` run | +| `tools/agent_verdicts.py` (S79/S80, task #9) | pulls the FINAL JSON verdict (`fn, binary, arm, status, closeness, draft_path, note`) out of Agent-tool subagent transcripts (`~/.claude/projects///subagents/agent-*.jsonl`, symlinked from `/tmp/claude-1000///tasks/*.output`) without reading them into a session; `--append `; a transcript with no verdict prints `NO-VERDICT` (R32). S80 aggregated the 11 S79 drafters this way: 2 MATCH banked, 8 NEAR ledgered | whenever Agent-tool drafters outlive the session that spawned them | +| `tools/p16_permute.py` + `tools/permuter_ils.py` (S80, task #9c) | **Pinned seeds are permutable now** (cookbook §493 S80 correction): `hide_asm` carries `asm(`/`__asm(` as well as `__asm__(` into the b64 pragma (keyword must be followed by `(`/`volatile`, so `INCLUDE_ASM("asm/…")` path strings are not statements — R39-controlled over 5,311 drafts); `permuter_ils` RE-HIDES every warm-restart waypoint (the permuter serializes pins raw), asserts `defines_fn` survived, aborts exit-2 on a refusal (R61a), and flushes its log (R55); `defines_fn` accepts K&R-style definitions. Any pre-S80 "permuter plateau" on a pinned seed was 1 cycle + silent no-ops — re-measure before citing it | every `permuter_ils` / `permuter_sweep` run; the S79 NEAR ledger re-run | | `tools/agent_verdicts.py` (task #9 handoff) | extracts the FINAL JSON verdict from Agent-tool subagent transcripts (`.../tasks/.output`, huge JSONL — never cat them) and appends them to a ledger; NO-VERDICT rows for files without one (R32) | after any batch of single drafting agents; the S79 handoff's step 2 | | `tools/jtbl_pads_fix.py` (task #6) | `PAD_ERR_MORE` accepts the current `jtbl_rodata_pads` wording ("more rodata jump tables than pad specs (N)") as well as the old ".align directives" one — it had reported "no pad-count drift" over a red build (cookbook §491 gap 1) | whenever a bank changes a TU's jump-table population and the build dies in `jtbl_rodata_pads` | diff --git a/docs/cookbook-index.md b/docs/cookbook-index.md index cc154df2d..e50723358 100644 --- a/docs/cookbook-index.md +++ b/docs/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 1161 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 1162 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -996,7 +996,7 @@ - **§421** — ★★★ — A `la $tN` + `addiu` PAIR CAN BE A **RELOAD** ARTIFACT THAT NO C SPELLING REACHES (P31 S71; byte-proven `md_SC07_003/func_801A293C`, 313 ins, 6 → 0) L33729 - **§446** — ★★★ — "RELOC-ONLY REMAP GATED DIFF" IS A CARVE VERDICT, NOT A CODEGEN VERDICT, UNTIL YOU DIFF THE `.text` (P31 S75; 4 siblings + 1 latent, ~1,300 ins) L34807 -### integration / TU plumbing (85) +### integration / TU plumbing (86) - **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) L456 - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L502 @@ -1083,6 +1083,7 @@ - **§477** — ★★★ — THE `self_decl_tu` CLASS IS A SOLVED, MECHANICAL LANE: 16 DRAFTS, 16 BANKS (P31 S77) L35713 - **§492** — ★★ — "C-PLUMBING" WAS THREE DIFFERENT THINGS (P31 S79 #7): A RAW SPLICE THE GATE'S LADDER BROKE, TWO -O0 BODIES THE CHECKER COMPILED AT -O2, AND TWO DRAFTS THAT BELONGED TO OTHER OVERLAYS L36392 - **§493** — ★★ — THE PERMUTER ROUTE END-TO-END, AND THE THREE PLUMBING STEPS BETWEEN A SCORE-0 WINNER AND THE MAIN GATE (P31 S79 #8) L36420 +- **§494** — ★★★ — SEVEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79 #9): THE IDIOMS, THE PLUMBING, AND THE THREE WAYS AN AGENT'S "MATCH" WAS NOT ONE L36480 ### build graph, splat & the harness (213) @@ -2948,6 +2949,7 @@ - **§491** — ★★ — THE MECHANICAL LEFTOVERS (P31 S79 #6): A PHANTOM STUB, TWO JTBL TWINS, ONE EXACT CLONE — AND THREE TOOL GAPS THE BANKS EXPOSED L36346 - **§492** — ★★ — "C-PLUMBING" WAS THREE DIFFERENT THINGS (P31 S79 #7): A RAW SPLICE THE GATE'S LADDER BROKE, TWO -O0 BODIES THE CHECKER COMPILED AT -O2, AND TWO DRAFTS THAT BELONGED TO OTHER OVERLAYS L36392 - **§493** — ★★ — THE PERMUTER ROUTE END-TO-END, AND THE THREE PLUMBING STEPS BETWEEN A SCORE-0 WINNER AND THE MAIN GATE (P31 S79 #8) L36420 +- **§494** — ★★★ — SEVEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79 #9): THE IDIOMS, THE PLUMBING, AND THE THREE WAYS AN AGENT'S "MATCH" WAS NOT ONE L36480 --- @@ -4121,3 +4123,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: ` | L36346 | §491 | ★★ — THE MECHANICAL LEFTOVERS (P31 S79 #6): A PHANTOM STUB, TWO JTBL TWINS, ONE EXACT CLON | | L36392 | §492 | ★★ — "C-PLUMBING" WAS THREE DIFFERENT THINGS (P31 S79 #7): A RAW SPLICE THE GATE'S LADDER | | L36420 | §493 | ★★ — THE PERMUTER ROUTE END-TO-END, AND THE THREE PLUMBING STEPS BETWEEN A SCORE-0 WINNER | +| L36480 | §494 | ★★★ — SEVEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79 #9): THE IDIOMS, THE PLUMB | diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 16de0bb60..3827f772c 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -36451,6 +36451,24 @@ seed the permuter's C parser refuses (`register u8 *a3 asm("$7")` — pins are n (positive-controlled). And `permuter_ils` seeds from a REGISTER-PINNED best draft cannot be permuted at all — the pinned 17→11 gain and the permuter are mutually exclusive on that function. +**S80 CORRECTION — the "pinned seeds cannot be permuted" claim was the INSTRUMENT, not the permuter.** Two +defects, both in our layer (R35/R40): (1) `p16_permute.hide_asm` carried only the `__asm__` spelling into +the b64 pragma; `register u8 *a3 asm("$7")` (the `asm(`/`__asm(` spellings, 3 of the S79 seeds) stayed raw +in base.c and pycparser refused it at cycle 1; (2) `permuter_ils`'s WARM RESTART copied the waypoint's +`source.c` — which decomp-permuter serializes with the pragmas DECODED back to raw pins — straight into +base.c, so on ANY pinned seed cycle 1 ran and cycles 2..N were parser refusals reported as "(unchanged)". +The S79 `func_80020DA4` "8-cycle plateau at 2" was one cycle. Fixed: `hide_asm` matches all three +spellings, followed by `(`/`volatile` (the bare word `asm` also lives inside `INCLUDE_ASM("asm/…")` path +strings — the R39 control over 5,311 drafts caught that false positive before it shipped); the ILS re-hides +every waypoint before restarting, asserts the definition survived, and ABORTS non-zero on a refusal +instead of counting no-op cycles (R61a). `defines_fn` also now accepts a K&R-style definition +(`void f(a, b) s32 a; s16 b; {` — the documented lever for an `s16` parameter's in-place promotion, +func_80039DEC), which it had refused as "lost the definition" — the R39 control over 5,311 stored drafts found +**436 K&R-style backlog drafts** that this check alone had kept out of the permuter lane. First re-run on the S79 seeds: every +pinned seed iterates, and `ov_SC06_022:func_8017DF28` (pinned WALL, "closeness 2 on five RTL-verified +attempts") reached **1** inside its first cycle. A permuter verdict on a pinned seed dated before S80 is +NOT a measurement of that function. + **The ledger the class leaves (for the PhaseEnd):** `main:func_80011380` 192 — WALL, §474 PROVED (pinned S79). `main:func_80015608` 86 — permuter plateau at **1** (REGALLOC-PERM: target `addu $s3,$s6,$s3` where the best draft emits `sll $s3,$s6,1` — a copy-then-add spelling of `x*2`, agent-sized). `main:func_80039B20` 79 — @@ -36459,3 +36477,45 @@ cycles). `main:func_80038698` 74 — best 11 (§ pins fix the interleave; the pe DAG-priority + local-alloc self-coalesce). `ov_SC03_105:func_801834A4` 106 — pinned WALL (S71, §148-A/§193-F, closeness 6 on four attempts). +#### §494 ★★★ — SEVEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79 #9): THE IDIOMS, THE PLUMBING, AND THE THREE WAYS AN AGENT'S "MATCH" WAS NOT ONE + +**Yield.** 25 packs (`claude_wave_packs`: journal history + a matched neighbour each), one Agent-tool subagent +per function (Haiku ≤50 ins, Sonnet ≤120, Opus above), no wave. Seven banked byte-identical the same day: +`md_MAIN_003:func_800D0174` + `func_800D1D14` (-O0 island), `main:func_80015608` + `func_8002AC98`, +`ov_SC05_018:func_80180BE0`, `ov_SC06_010:func_801809E4`, `ov_SC05_010:func_8017FFA8` (a 6-way jtbl switch). + +**Idioms that closed functions (each byte-proven today).** +* **The fresh-temp lever for a commutative operand order** — target `addu $s3,$s6,$s3` where every source + order emits `addu $s3,$s3,$s6`: gcc 2.7.2's `expand_binop` swaps operands when op1 IS the expansion target + register; route the add through a NEW pseudo — `{ s32 xt = blockSize + x0; x0 = xt; }` — and the source order + survives, xt coalesces into the same hard register. Two banks (`func_80015608`, `func_8002AC98`); the temp AND + both operands must be `s32` (an `s16` temp re-coalesces with the target and the swap returns; an `s16` operand + costs a sign-extend pair). +* **-O0: address-of + cast beats the `%lo` fold** — `D_800D3630` (4-byte stride, low 16 bits read) as + `s16[][2]` or a 2-field struct makes gcc fold `%lo` into the `lh` displacement through `$at` (-2 ins per site); + `extern s32 D_800D3630[]` + `*(s16*)&D_800D3630[i]` materialises `lui`+`addiu` first and matches. +* **An early `return` that must fall through** — the Haiku plateau's missing 2 instructions and "frame -8 vs + -0x10" were a `return` inside a branch that should fall into the shared mask/store tail; the -0x10 no-save + frame was two `s16` locals (§186b), not a call. Branch polarity `if (v0 >= v1)` with the arms swapped. +* **jtbl switch:** the loop index must be `s32` (an `s16` gives the fused `lhu+sll16+sra12`, §241) and a + table lookup goes into its own named temp BEFORE the `found=1`/zero-store statements, or the store schedules + ahead of the load. +* **The phantom stack frame** — an address-taken `s32 frame_pad[3]` induces the target's unexplained 16-byte + frame (`func_80020DA4`, 8 → 2; the remaining 2 is a `mflo` destination, permuter class). +* **`memcpy(…,12)` under a TU's `extern memcpy`** compiles to a `jal` (§48-C2/§160a); the inline + `lwl/lwr/swl/swr` shape needs a struct assign through an align-1 12-byte typedef (the TU's own Blk8 idiom). + +**Three ways an agent's "MATCH" was not one.** (1) A naked `__asm__` reproduction of the target — the verbatim +class; sent back, it returned genuine C. (2) Standalone `match_one` MATCH, real-TU DIFF: the TU's typedefs and +externs (`rtu_match` names them; §376). (3) rtu MATCH, gate NEAR: the gate's transform ladder altered the body +(§492a) — raw splice. Rule for the coordinator: **grep the draft for `.ent`/`.word`, re-measure in the real TU, +gate, commit — per result, never in bulk.** + +**Ledger rows this task leaves** (drafts in `.run/S79w//`, notes in `.run/S79w/verdicts/verdicts.jsonl`): +`main:func_8001BC6C` 28 (sched1 birthing-boost — two mutually exclusive schedules), `main:func_8002FDE8` 35 +(local-alloc caches `&D_800A46D2` across a call where the target rematerialises, §153), `main:func_80038698` 11 +(sched1 DAG priority on the byte-pair idiom ×3; the permuter refuses the pinned seed), `main:func_80039B20` 7 +(§461 tie-break), `main:func_80020DA4` 2 (mflo destination — permuter fuel), `main:func_80015B6C` 44 (two +documented walls, better than 3 of 6 attempts), `ov_SC06_022:func_8017DF28` 2 (delay-slot fill from +`expand_block_move`'s cse-reused address pseudo — five RTL-verified attempts, WALL). + diff --git a/tools/p16_permute.py b/tools/p16_permute.py index e7e1dbb02..e163b78b1 100644 --- a/tools/p16_permute.py +++ b/tools/p16_permute.py @@ -99,7 +99,13 @@ def hide_asm(c): steered, mvmva/GTE ops still compile), while pycparser never chokes on `__asm__`. (No submodule edit: this reuses decomp-permuter's OWN b64literal pragma carrier.)""" out, i = [], 0 - pat = re.compile(r"\b__asm__\b") + # `asm(` and `__asm(` are the SAME gcc keyword and pycparser rejects all three spellings alike; + # the old `__asm__`-only pattern left `register u8 *a3 asm("$7")` raw in base.c -> "Syntax error in + # base.c" -> the whole run silently no-ops (P31 S79 #8 func_80038698; fixed S80). + # The keyword must be FOLLOWED by `(` or a volatile qualifier: the bare word `asm` also occurs + # inside path strings -- `INCLUDE_ASM("asm/ov_SC03_028/...", fn)` -- and treating that as a + # statement ate the line up to the function's own `{` (caught by the R39 control, S80). + pat = re.compile(r"\b(?:__asm__|__asm|asm)\b(?=\s*(?:\(|__volatile__\b|volatile\b))") while True: m = pat.search(c, i) if not m: @@ -197,7 +203,13 @@ def defines_fn(base_c, fn): prep step here can swallow the function (hide_asm chewing an unexpanded multi-line macro block, an unterminated comment eating the rest of the file, a future cpp/typedef edge) so the check belongs on the OUTPUT, where it catches all of them, not on each cause.""" - return re.search(r"^[^#/\n]*?\b" + re.escape(fn) + r"\s*\([^;{]*\)\s*\{", base_c, re.M) is not None + # A K&R-style definition -- `void fn(a0, a1, a2) s32 a0; s32 a1; s16 a2; {` -- carries a declaration + # list between `)` and `{`. It is the documented lever for an `s16` parameter's in-place sll/sra + # promotion (P31 S79 func_80039DEC), pycparser accepts it, and this check refused it as "lost the + # definition" (S80). Each K&R declaration must START with an identifier so a call `fn(1) ; x = 2; {` + # cannot chain into a false definition. + return re.search(r"^[^#/\n]*?\b" + re.escape(fn) + r"\s*\([^;{]*\)(?:\s*[A-Za-z_][^;{}]*;)*\s*\{", + base_c, re.M) is not None def winner_to_draft(winner_c): diff --git a/tools/permuter_ils.py b/tools/permuter_ils.py index 2770bfb00..abf95ed3e 100644 --- a/tools/permuter_ils.py +++ b/tools/permuter_ils.py @@ -11,10 +11,14 @@ intermediate waypoints can be semantically divergent (the permuter rewrites stor --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077 --klass REGALLOC --cycles 10 --secs 180 --j 12 On a score-0 winner -> .run/permuter-winners/.c (then winner_to_draft + gate whole-binary). """ -import argparse, glob, os, re, shutil, sys +import argparse, functools, glob, os, re, shutil, sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import p16_permute as P +# stdout is BUFFERED when redirected to a log; an 8-cycle run then shows an EMPTY log for 20 minutes +# (S80: eight parallel runs, no evidence of progress -- R55). Always flush. +print = functools.partial(print, flush=True) + def best_waypoint(pd): """(score, dir) of the lowest-score output--/source.c across ALL cycles; None if none. @@ -46,8 +50,18 @@ def main(): print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}") prev = None + refused = os.path.join(pd, "PERMUTER_REFUSED.txt") for cyc in range(1, a.cycles + 1): + if os.path.exists(refused): + os.remove(refused) P.run_permuter(pd, a.secs, a.j) # writes output-*/ ; kills stragglers + if os.path.exists(refused): + # NOT-JUDGED IS NOT A VERDICT (R61a). A refused cycle permuted nothing; reporting it as + # "(unchanged)" for the remaining cycles is how S79's func_80020DA4 run showed 8 cycles of + # work that were 1 cycle + 7 no-ops. Stop, say so, exit non-zero. + print(f"ILS ABORTED at cycle {cyc}: the permuter REFUSED base.c (see {refused}); " + f"best so far={prev}") + sys.exit(2) bw = best_waypoint(pd) if bw is None: print(f" cycle {cyc}: no waypoint (no improvement over base yet)") @@ -61,7 +75,16 @@ def main(): shutil.copy(os.path.join(d, "source.c"), dst) print(f" WINNER score 0 -> {dst} (gate whole-binary before banking)") return - shutil.copy(os.path.join(d, "source.c"), f"{pd}/base.c") # warm restart + # WARM RESTART. The permuter DECODES the b64 pragma carrier when it serializes a candidate, so a + # waypoint's source.c holds the raw `register … __asm__("$N")` pins / `__asm__` statements again. + # Copying it verbatim made every cycle after the first a parser refusal on any pinned seed + # (P31 S79: func_80020DA4 = 1 real cycle + 7 silent no-ops). Re-hide before restarting, and assert + # the function definition survived the re-hide (R32). + wp = P.hide_asm(open(os.path.join(d, "source.c")).read()) + if not P.defines_fn(wp, a.fn): + print(f"ILS ABORTED at cycle {cyc}: re-hiding the waypoint lost the definition of {a.fn} " + f"(inspect {d}/source.c)"); sys.exit(2) + open(f"{pd}/base.c", "w").write(wp) prev = score print(f"ILS done: best={prev} (no score-0; seed for Fable5 or a longer run)")