diff --git a/decomp-architect/corpus/cookbook/cookbook-index.md b/decomp-architect/corpus/cookbook/cookbook-index.md
index 3ee117dd2..3f058dc39 100644
--- a/decomp-architect/corpus/cookbook/cookbook-index.md
+++ b/decomp-architect/corpus/cookbook/cookbook-index.md
@@ -2,7 +2,7 @@
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
>
-> `docs/matching-cookbook.md` is ~716 KB / 1170 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
+> `docs/matching-cookbook.md` is ~716 KB / 1171 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
@@ -1461,7 +1461,7 @@
- **§479** — ★★★ — WHERE THE PERMUTER ACTUALLY PAYS: A MEASURED YIELD CURVE (P31 S77, 8 candidates) L35811
- **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) L37151
-### (unbucketed — title matched no symptom vocabulary) (337)
+### (unbucketed — title matched no symptom vocabulary) (338)
- **§3-How** — to use this L30
- **§1** — Idiom catalog (asm pattern → C that produces it) L39
@@ -1800,6 +1800,7 @@
- **§452** — ★★★ — NOT EVERY VERBATIM BODY IS UNDECOMPILED WORK, AND §448'S HEADLINE OVERSTATED IT (P31 S75; 10-function burst, 0 banks, and the negative result is the finding) L35104
- **§462** — FOUR LEVERS FROM `main:func_80024054` (91 ins, 74/53/32 → 4) L35644
- **§480** — 🔴 — A STATIC BLOCKER CLASS THAT THE REAL PIPELINE ALREADY REMOVES IS A PHANTOM L35890
+- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) L37604
## All sections, in order
@@ -2974,6 +2975,7 @@
- **§500** — ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MATCH / 9 NEAR / 2 FAIL, every closer, two NEW named gcc mechanisms, and the wave-process defects L36793
- **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) L37151
- **§453** — ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DEFECTS THAT SHAPED IT (P35 S94–S96) L37555
+- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) L37604
---
@@ -4156,3 +4158,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
| L36793 | §500 | ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MA |
| L37151 | §501 | ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABI |
| L37555 | §453 | ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DE |
+| L37604 | §454 | Rung D and rung R: taking a lever off a body that still has to compile to the same bytes ( |
diff --git a/decomp-architect/corpus/cookbook/matching-cookbook.md b/decomp-architect/corpus/cookbook/matching-cookbook.md
index 5959bea69..3c38acaf6 100644
--- a/decomp-architect/corpus/cookbook/matching-cookbook.md
+++ b/decomp-architect/corpus/cookbook/matching-cookbook.md
@@ -37600,3 +37600,70 @@ green. R34 in one sentence: build the oracle that can disagree, then measure the
the classifier became includes (ov_SC03_015's 219 single-site macros; the never-extended members of five under-listed groups). REAL
360,744 → 350,533 with EMPTY up the same: empty-bodied shared functions were folded into REAL under the macro form. Instruction-weighted
metrics unchanged. A number generated from a form the tooling cannot see is a number, not a count.
+
+## §454 — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6)
+
+The residue after the mechanical de-lever campaign (rungs A–C) is 12,970 bodies in 1,804 text classes, each with at least one site the
+oracle called NEEDED. Two instruments work on it: **rung D**, decomp-permuter seeded with the LEVER-FREE body, and **rung R**, the
+cookbook's byte-neutral shape recipes applied mechanically. What follows is what S99 measured, including the part where the measurement
+was wrong.
+
+**A DISASSEMBLY LISTING IS NOT A TARGET.** Two campaigns returned "0 of 16 exemplars" with a straight face. Both were the harness. The
+permuter's `target.o` had been assembled from `verbatim_target_s.py --gas`, a listing regenerated from the ROM image — and assembling a
+disassembly is a second toolchain with its own answers:
+
+* objdump prints the PSEUDO-instruction `move rX,rY` for `addu rX,rY,$zero` (0x…21). GAS assembles `move` as `or` (0x…25). In one
+ 234-instruction function that is **24 wrong words**, and nothing said so.
+* a listing's `%hi`/`%lo` pairs come back RESOLVED — `lui $a1,0x801f` / `addiu $a1,$a1,-11600`, no relocation — while every compiled
+ candidate carries `R_MIPS_HI16`/`LO16` against a symbol, and the masked scorer compares reloc operands.
+
+Result: the scorer reported **28 for a body that is byte-identical**, so score 0 was unreachable and every NO-MATCH was its own.
+
+The control that names this in one line: **seed the search with the body the tree already matches and read the base score.** It must be
+0. (`delever_permute.py --positive-control`.) Run it before believing any campaign's yield — a 0-yield run cannot otherwise be told from
+a broken scorer (R40).
+
+The fix is to stop assembling a listing: **the target is the tree's own body compiled by the build's own tail** into a one-function
+object, so it carries the candidates' relocations by construction. The ROM listing stays the INDEPENDENT oracle that proves it —
+`match_one` must call that body a MATCH before the search starts — which is what keeps the arrangement from being circular (R34/R56).
+`verbatim_target_s.py --gas` now verifies itself too (assemble → disassemble → compare word by word → `.word`-patch or REFUSE); it is
+the file pasted into decomp.me, and it was wrong for every function containing a `move`.
+
+**THE PROFILE FOLLOWS THE REGISTER, NOT THE SITE KIND.** A needed pin on a CALLEE-SAVED register ($16–$23) is an allocation-order
+residual and wants the regalloc profile. A pin on a CALLER-SAVED one ($2/$3/$4–$7) is not. `func_80163EC8` pins `$2`, and its residual
+is `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — the operand ORDER of one `&`. The regalloc
+profile weights `perm_commutative` 2.0; the cse profile weights it 40.0. Steering by "it was a pin, so regalloc" spends the whole budget
+20× away from the only lever that closes it.
+
+**WHAT RUNG D ACTUALLY FINDS, AND WHY RUNG R THEN GETS IT FOR FREE.** Every win is a small, nameable source shape:
+
+| function | start (mismatched ins) | time | the whole semantic change |
+|---|---|---|---|
+| `func_80135D20` | 12 | 24 s | `flag = 0;` → `do { flag = 0; } while (0);` (`perm_ins_block`, RC-5 scope) |
+| `func_80163EC8` | 8 | 314 s | `uVar5 = *(s32 *)(p+0x44); … uVar5 & ~0x20` → the temp inlined at its use (`perm_expand_expr`, §501-R's S2 kill) |
+
+The permuter's winner is machine-reprinted by pycparser (extra parens, `;` statements, its own brace style), so banking it verbatim
+trades a lever for a readability regression — which is exactly what this phase exists to prevent. Each win is therefore turned into a
+mechanical recipe that produces the SAME bytes from a one-line diff in the real source. Rung R's generators, in trial order (targeted
+first, blanket last), each one compile:
+
+* **R2** the formerly-pinned declarations permuted among their own lines · **R4** one of them moved through the whole declaration run
+ (§76/§501-R: the allocation order is the bank) · **R3** an initializer split off its declaration, the assignment placed after the
+ WHOLE run (C89 forbids a declaration after a statement — placing it after the last PINNED declaration compiles as K&R parameters and
+ fails in a way that reads like a compiler bug)
+* **R5** the operand order of one commutative operator (the caller-saved lever above; needs no pinned declaration)
+* **R6** a local assigned once and read once, inlined at its use with its dead declaration removed
+* **R7** one statement wrapped in a block — `{ stmt; }` tried before `do { stmt; } while (0);`, because the readable spelling should win
+ when both hold
+
+**THE CLASS, NOT THE BODY.** A win on an exemplar is worth its whole text class (the head of the draw is 80 classes of 134 copies), but a
+ledger REPLAY cannot deliver it: the ledger replays a SITE SET, and a reshaped body is not one. `delever.py --propagate TU FN` does it by
+address remap — the two old bodies are identical modulo `func_`/`D_` tokens (that IS the class hash), so their tokens correspond one for
+one and the map they define is applied to the new body, each sibling judged on its own objects. `--apply-body` records the before/after
+body text in its ledger row because after the write it exists nowhere else.
+
+**THE SHAPE OF THE POPULATION (first 6 of 16 exemplars, all ov_SC04_011, 134-copy classes).** Two closed, four did not, and the split is
+the starting distance, not the site count: the closers started 8 and 12 mismatched instructions from the target; the four that did not
+started at 70, 99, 105 and 131 — every one a body where removing a hand-placed `instruction` lever changed the instruction COUNT and
+shifted everything after it (`mine=103 ins, target=106`). They still improved a long way (131→35, 105→18, 99→37, 70→24), so they are
+seeds for a longer run or for T7's agents, not walls. `--max-start N` triages by that number instead of spending a search on it.
diff --git a/decomp-architect/corpus/record/docs/accelerators.md b/decomp-architect/corpus/record/docs/accelerators.md
index f1ef7ae2f..317ec257d 100644
--- a/decomp-architect/corpus/record/docs/accelerators.md
+++ b/decomp-architect/corpus/record/docs/accelerators.md
@@ -916,3 +916,33 @@ each a different action. (3) **A known-true control per join:** the registry rep
multi-batch tool's hardest bug (stale line numbers + a restore that wipes the previous batch) into a non-event; the driver commits
between batches. (5) **The census's per-instance forms are the registry check's C2d for free** (one cached scan, 36 s) — derive, don't
re-parse (R33).
+
+## P36 S99 (2026-09-09) — the search harness whose target was a disassembly, and the three cheap checks that would have caught it
+
+(1) **Before believing any search yield, seed the search with a body that already MATCHES and read the base score. It must be 0.**
+Two rung-D campaigns returned "0 of 16 exemplars" and both were the instrument: the permuter's `target.o` had been assembled from a
+regenerated disassembly listing, and assembling a disassembly is a second toolchain — objdump prints the pseudo-instruction `move` for
+`addu rX,rY,$zero` and GAS assembles `move` as `or` (24 wrong words in one 234-instruction function), while a listing's `%hi`/`%lo`
+pairs come back resolved, with no relocation, against candidates that all carry one. The scorer read 28 for a byte-identical body, so
+score 0 was unreachable. The control is one command and ~30 seconds (`delever_permute.py --positive-control`); without it, "the residue
+is hard" and "the scorer is broken" are the same observation. R40, and the first place this project has needed it against a SCORER
+rather than a model.
+
+(2) **The target of a byte search should be produced by the same toolchain as the candidates, with an independent oracle to prove it
+— not by a second toolchain that has to be argued with.** The fix was to compile the tree's own body into a one-function object (the
+candidates' relocations by construction) and require `match_one` against the ROM listing to call it a MATCH before the search starts.
+Faster to build than the listing-fixing it replaced, and it cannot drift.
+
+(3) **A generated artifact that nothing ever consumed is a claim.** `verbatim_target_s.py --gas` had been proven once, through
+decomp.me, on a function with no `move` in it — so it was wrong for a large fraction of the fleet and nobody knew (R98/DK-81 in a second
+place). It now assembles, disassembles and compares itself word by word against the image, `.word`-patches what does not reproduce, and
+REFUSES what still disagrees.
+
+(4) **Steer a directed search by what the residual IS, not by what produced it.** The weight profile was chosen from the site kind
+("a pin → regalloc"); a pin on a CALLER-saved register ($2/$3/$4–$7) is an operand-order residual, and the regalloc profile weights
+`perm_commutative` 2.0 where the cse profile weights it 40.0. One reading of one diff moved 8 of 16 exemplars to the right profile.
+
+(5) **Turn each search win into a mechanical recipe the same day.** Rung D's two wins were a statement wrapped in a block and a
+single-set temp inlined at its use — both one-line source changes that a recipe generator reproduces in ONE compile, and that keep the
+source readable, where banking the permuter's own pycparser-reprinted body would trade a lever for a readability regression in a phase
+whose whole purpose is readability.
diff --git a/decomp-architect/corpus/tools/INDEX.md b/decomp-architect/corpus/tools/INDEX.md
index 2cbbf3e5e..8498edf26 100644
--- a/decomp-architect/corpus/tools/INDEX.md
+++ b/decomp-architect/corpus/tools/INDEX.md
@@ -8,8 +8,8 @@
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
>
-> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
-> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
+> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows
+> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure).
## P1 — extraction + manifest
@@ -335,6 +335,7 @@
| `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout |
| `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout |
| `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule |
+| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger |
| `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas |
| `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name |
| `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing |
@@ -343,7 +344,7 @@
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
-| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
+| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
diff --git a/decomp-architect/corpus/tools/P10/delever.py b/decomp-architect/corpus/tools/P10/delever.py
index 54eff6096..bb22d443c 100644
--- a/decomp-architect/corpus/tools/P10/delever.py
+++ b/decomp-architect/corpus/tools/P10/delever.py
@@ -1254,6 +1254,512 @@ def scrub(a):
return 0 if not bad else 1
+# ----------------------------------------------------------------------------------------------------------------------
+# --recipes (rung R): the cookbook's byte-neutral shape recipes, tried mechanically on a RESIDUE body
+# ----------------------------------------------------------------------------------------------------------------------
+CTRL_KW = re.compile(r"^\s*(?:if|for|while|do|switch|else|return|goto|case|default|break|continue)\b")
+
+
+def pin_names(sites):
+ """the variable each pin site declares, in source order (the zero-register pins excluded: their declaration is deleted)."""
+ out = []
+ for s in sites:
+ if s["kind"] != "pin" or s.get("zero"):
+ continue
+ mm = re.search(r"\b([A-Za-z_]\w*)\s*(?:\[[^\]]*\])?\s*(?:__asm__|__asm|asm)\s*\(", s.get("text", ""))
+ if mm:
+ out.append(mm.group(1))
+ return out
+
+
+def is_decl_line(masked_line):
+ """a whole-statement DECLARATION on one line: `[\\[n\\]][ = init];`. The type and the name must be separated
+ (by space or `*`) — without that, `ret = f();` parses as the declaration `re t = …` and an initializer split lands its
+ assignment after the first statement, where C89 forbids the declarations that follow it."""
+ s = masked_line.strip()
+ return bool(s.endswith(";") and not CTRL_KW.match(s) and "(" not in s.split("=")[0]
+ and re.match(r"^[A-Za-z_][\w \t]*[\s*]\s*\*?\s*[A-Za-z_]\w*\s*(?:\[[^\]]*\])*\s*(?:=|;)", s))
+
+
+def decl_run_end(text, d):
+ """the 0-based index of the LAST line of the declaration run that opens fn's body — a C89 declaration may not follow a
+ statement, so an initializer split must put its assignment after the WHOLE run, not after the last pinned declaration."""
+ lines = text.split("\n")
+ last = d["line"] - 1
+ for i in range(d["line"], d["end"] - 1):
+ s = sc.mask_text(lines[i]).strip()
+ if not s or s.startswith("/*") or s.startswith("//") or s in ("{", "}"):
+ continue
+ if is_decl_line(sc.mask_text(lines[i])):
+ last = i
+ continue
+ break
+ return last
+
+
+def decl_lines(text, tu, fn, names):
+ """[(line index, text)] for the lines of fn's body that DECLARE one of `names` — one name per line, the line a whole
+ statement. The lever-free text is re-scanned for them (a stripped pin can delete its line, so the census's numbers have
+ moved); a name whose declaration is not found alone on one line makes the body ineligible (None)."""
+ recs = sc.scan_text(text, tu, shared_defs=None)
+ d = next((r for r in recs if r["form"] == "def" and r["name"] == fn), None)
+ if d is None:
+ return None
+ lines = text.split("\n")
+ found = {}
+ for i in range(d["line"], d["end"] - 1): # inside the body, never the header line
+ raw_line = lines[i]
+ s = sc.mask_text(raw_line).strip()
+ if not is_decl_line(s):
+ continue
+ hits = [n for n in names if re.search(r"(?!-/%":
+ continue
+ prev = expr[:i].rstrip()
+ if prev and (prev[-1].isalnum() or prev[-1] in "_)]"):
+ out.append(i)
+ return out
+
+
+def commutative_swaps(text, tu, fn, d_):
+ """[(description, candidate text)] — each line of fn's body that carries exactly ONE top-level commutative operator,
+ with its two operands swapped. THE lever for a caller-saved ($2/$3) residual: S99 read `and v0,v1,v0` against the
+ target's `and v0,v0,v1` on func_80163EC8, which is the operand order of one `&` in the source and nothing else."""
+ lines = text.split("\n")
+ out = []
+ for i in range(d_["line"], d_["end"] - 1):
+ raw_line, s = lines[i], sc.mask_text(lines[i])
+ body = s.strip()
+ if not body.endswith(";") or body.startswith("#"):
+ continue
+ # only an assignment's RHS or a `return` expression — an `if (…) stmt;` line would need the condition parsed out of
+ # the statement after it, and a wrong split is a candidate that cannot compile (wasted, and noisy in the ledger)
+ asg = re.search(r"(?+\-*/%&|^~])=(?!=)", s)
+ ret = re.match(r"^\s*return\b", s)
+ start = asg.end() if asg else (ret.end() if ret else -1)
+ if start < 0 or start >= len(s):
+ continue
+ expr_end = s.rstrip().rfind(";")
+ expr = s[start:expr_end]
+ ops = top_level_ops(expr)
+ if len(ops) != 1:
+ continue
+ o = start + ops[0]
+ left, right = raw_line[start:o], raw_line[o + 1:expr_end]
+ if not left.strip() or not right.strip():
+ continue
+ cand = list(lines)
+ cand[i] = raw_line[:start] + " " + right.strip() + " " + raw_line[o] + " " + left.strip() + raw_line[expr_end:]
+ out.append((f"swap {raw_line[o]} @{i + 1}", "\n".join(cand)))
+ return out
+
+
+IDENT = re.compile(r"(? 1:
+ continue
+ # ONE ASSIGNMENT AT A TIME, not one per variable. The single-set case is the easy half; the lever rung D actually
+ # found is narrower: `uVar5` is assigned in TWO branches of func_80163EC8, and the winning move inlined ONE of them.
+ # An assignment is inlinable when its value is read exactly once before the variable is written again — the classic
+ # def-with-one-use — so the assignment can go and the read can carry the expression.
+ order = sorted(places)
+ for ai, _, _ in asgs:
+ after = [p for p in order if p[0] > ai]
+ reads = [p for p in after if p not in [(x, y, z) for x, y, z in asgs] and not is_decl_line(masked[p[0]].strip())]
+ if not reads:
+ continue
+ ui, u0, u1 = reads[0]
+ nxt = [p for p in after if p[0] > ui]
+ if nxt and (nxt[0][0], nxt[0][1], nxt[0][2]) not in [(x, y, z) for x, y, z in asgs]:
+ continue # read again before it is rewritten: the assignment is not dead
+ mm = re.match(ASG % re.escape(v), masked[ai].strip())
+ if not mm:
+ continue
+ expr = lines[ai].strip()[mm.start(1):mm.end(1)]
+ if not expr.strip():
+ continue
+ cand = list(lines)
+ cand[ui] = lines[ui][:u0] + f"({expr})" + lines[ui][u1:]
+ cand[ai] = None # the assignment goes (with its declaration when they are one)
+ if decls and len(asgs) == 1: # the separate `T v;` is dead only when nothing else writes v
+ cand[decls[0][0]] = None
+ out.append((f"inline {v} @{ai + 1}", "\n".join(l for l in cand if l is not None)))
+ return out
+
+
+def block_wraps(text, tu, fn, d_):
+ """[(description, candidate text)] — one statement wrapped in a block. §501-R's RC-5 scope lever: a block changes the
+ statement's basic-block structure and with it the allocno live range, which is how rung D closed func_80135D20 in 24 s
+ (`flag = 0;` -> `do { flag = 0; } while (0);` and nothing else). The plain block is tried FIRST because it is the
+ readable spelling; the do-while is gcc's stronger form and is only reached when the plain one does not hold."""
+ lines = text.split("\n")
+ out = []
+ for i in range(d_["line"], d_["end"] - 1):
+ s = sc.mask_text(lines[i]).strip()
+ if not s.endswith(";") or CTRL_KW.match(s) or is_decl_line(s) or s.startswith("#") or "{" in s or "}" in s:
+ continue
+ raw_line = lines[i]
+ indent = raw_line[:len(raw_line) - len(raw_line.lstrip())]
+ stmt = raw_line.strip()
+ for tag, spelling in (("block", f"{indent}{{ {stmt} }}"), ("do-while", f"{indent}do {{ {stmt} }} while (0);")):
+ cand = list(lines)
+ cand[i] = spelling
+ out.append((f"{tag} @{i + 1}", "\n".join(cand)))
+ return out
+
+
+def recipe_candidates(text, tu, fn, names, limit=24, rng=None, cap=40, blocks=True):
+ """[(recipe, description, candidate text)] — the byte-neutral shape recipes of the cookbook, mechanically.
+ R2 (§76/§501-R, the allocation ORDER is the bank): the formerly-pinned declarations permuted among their own lines.
+ R4: one of them moved through the whole declaration run. R3 (§17a/§501-P): an initializer split off its declaration.
+ R5 (§137/§501-R, the caller-saved class): the operand order of one commutative operator — the ONLY recipe here that
+ needs no pinned declaration, so a residue of barriers and launders still has candidates."""
+ recs_ = sc.scan_text(text, tu, shared_defs=None)
+ d_ = next((r for r in recs_ if r["form"] == "def" and r["name"] == fn), None)
+ if d_ is None:
+ return []
+ dls = decl_lines(text, tu, fn, names) if names else None
+ lines = text.split("\n")
+ out = []
+ rng = rng or random.Random(0)
+ idx = [i for i, _ in dls] if dls else []
+ body = [t for _, t in dls] if dls else []
+ perms = []
+ if not dls:
+ perms = []
+ elif len(body) <= 4:
+ import itertools
+ perms = [p for p in itertools.permutations(range(len(body)))][1:] # the identity is the current text
+ else:
+ seen = set()
+ while len(perms) < limit and len(seen) < limit * 4:
+ p = tuple(rng.sample(range(len(body)), len(body)))
+ seen.add(p)
+ if p != tuple(range(len(body))) and p not in perms:
+ perms.append(p)
+ for p in perms[:limit]:
+ ls_ = list(lines)
+ for slot, src in zip(idx, p):
+ ls_[slot] = body[src]
+ out.append(("R2", "decl-order " + ",".join(str(x) for x in p), "\n".join(ls_)))
+ # R4 (§76/§501-R again, one variable at a time): a formerly-pinned declaration moved to every other slot of the body's
+ # declaration run — the allocno creation order the pin used to override. R2 permutes the pinned declarations among
+ # THEIR OWN slots; this reaches the orders that involve the untouched declarations too, at one compile each.
+ last = decl_run_end(text, d_)
+ run_idx = [i for i in range(d_["line"], last + 1) if is_decl_line(sc.mask_text(lines[i]))]
+ if dls and len(run_idx) > 1:
+ run_txt = [lines[i] for i in run_idx]
+ for i, t in dls:
+ if i not in run_idx:
+ continue
+ src = run_idx.index(i)
+ name = next((n for n in names if re.search(r"(?{dst}", "\n".join(ls_)))
+ for k, (i, t) in enumerate(dls or []):
+ head, _, init = t.partition("=")
+ if not init.strip().endswith(";") or "==" in t:
+ continue
+ name = [n for n in names if re.search(r"(? {b[2]} {b[3]}")
+ sys.exit("delever --recipes: the control did not pass — no verdict from this run is usable (R39)")
+ print(f"delever --recipes: {len(todo)} RESIDUE bodies", flush=True)
+ rows, won, tried, compiles = [], 0, 0, 0
+ t0 = time.time()
+ for n_body, ((tu, fn), r) in enumerate(todo, 1):
+ path = REPO / tu
+ raw = path.read_text(errors="surrogateescape")
+ names = pin_names(sites_by[(tu, fn)]) # may be empty: R5 needs no pinned declaration
+ try:
+ free = lever_free(tu, raw, fn)
+ except Refuse:
+ continue
+ cands = recipe_candidates(free, tu, fn, names, cap=a.cap)
+ if not cands:
+ continue
+ tried += 1
+ hit = None
+ for rec, desc, cand in cands:
+ v, dt, err = judge(tu, cand)
+ compiles += len(recs_for(tu))
+ if v == "IDENTICAL":
+ hit = (rec, desc, cand)
+ break
+ if n_body % 10 == 0 or hit: # R55: a lane that runs unattended leaves evidence
+ print(f" [{n_body}/{len(todo)}] {won} closed · {compiles} compiles · {(time.time() - t0) / 60:.1f} min",
+ flush=True)
+ row = dict(ts=time.strftime("%Y-%m-%d %H:%M:%S"), label=a.label, rung="R", calib=dict(head=oracle.head(), stamp=oracle.config_stamp()),
+ tu=tu, fn=fn, addr=fn_addr(fn, tu), aliases=r.get("aliases"), header=tu.endswith(".h"),
+ nhash_before=r.get("nhash_after"), nhash_after=None, candidates=len(cands), pins=len(names),
+ verdict=("LEVER-FREE" if hit else "RESIDUE"), recipe=(hit[0] if hit else None), how=(hit[1] if hit else None),
+ sites=([] if hit else r.get("sites", [])))
+ if hit:
+ path.write_text(hit[2], errors="surrogateescape")
+ # the body's own markers are now orphans (its levers are gone); every OTHER body's marker is still honest, so the
+ # scrub is scoped to this body's line span — a file-wide scrub would leave the census with UNMARKED sites elsewhere
+ recs_ = sc.scan_text(hit[2], tu, shared_defs=None)
+ d_ = next((x for x in recs_ if x["form"] == "def" and x["name"] == fn), None)
+ span = range(d_["line"], d_["end"] + 1) if d_ else range(0)
+ scrub = scrub_edits(hit[2], [i + 1 for i, l in enumerate(hit[2].split("\n")) if FAKE in l and i + 1 in span])
+ if scrub:
+ cand2 = apply_edits(hit[2], scrub)
+ v2, _, _ = judge(tu, cand2)
+ if v2 == "IDENTICAL":
+ path.write_text(cand2, errors="surrogateescape")
+ walk = lc.walk_file(path.read_text(errors="surrogateescape"), tu, tu.endswith(".h"))
+ row["nhash_after"] = next((x["nhash"] for x in walk["defs"] if x["name"] == fn), None)
+ won += 1
+ print(f" {tu}:{fn} — {hit[0]} {hit[1]} IDENTICAL ({len(names)} pin(s) gone)", flush=True)
+ rows.append(row)
+ ledger_append(rows)
+ needed = sum(len([s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]) for _, r in todo)
+ print(f"recipes: {won} of {tried} bodies closed lever-free ({needed} NEEDED sites in the {len(todo)} drawn), "
+ f"{compiles} compiles in {(time.time() - t0) / 60:.1f} min")
+ return 0
+
+
+def remap_body(ex_before, ex_after, sib_before):
+ """the exemplar's reshaped body, with its `func_/D_` addresses replaced by the sibling's — or (None, why).
+
+ A text class is "identical modulo addresses" (that IS the nhash), so the two old bodies' address tokens correspond
+ one for one in order; the map they define is applied to the new body. This is what turns one crack into a whole
+ class: the 134-copy classes are the reason the draw is ordered by copies. A ledger REPLAY cannot do it — the ledger
+ replays a SITE SET, and a reshaped body is not one."""
+ a, b = lc.NORM_SYM.findall(ex_before), lc.NORM_SYM.findall(sib_before)
+ if len(a) != len(b):
+ return None, f"{len(a)} address tokens in the exemplar, {len(b)} in the sibling"
+ m = {}
+ for x, y in zip(a, b):
+ if m.setdefault(x, y) != y:
+ return None, f"`{x}` maps to both `{m[x]}` and `{y}` — not one class"
+ return lc.NORM_SYM.sub(lambda mm: m.get(mm.group(0), mm.group(0)), ex_after), None
+
+
+def propagate(a):
+ """--propagate TU FN: the body TU:FN was reshaped and banked; give every RESIDUE sibling of its class the same
+ shape, with its own addresses, and judge each on its own objects."""
+ tu, fn = a.propagate
+ rows = load_ledger()
+ chain = [r for r in rows if r.get("tu") == tu and r.get("fn") == fn
+ and r.get("verdict") == "LEVER-FREE" and r.get("after_text")]
+ if not chain:
+ sys.exit(f"delever --propagate: no banked reshape of {tu}:{fn} in the ledger (its row must carry after_text)")
+ # THE CLASS is what the body looked like when the campaign found it, so the key and the "before" text come from the
+ # FIRST bank in this body's chain; the text to spread is the LAST one (a body reshaped, then tidied, has two rows, and
+ # taking the last row's before-hash would look for siblings of a text only this body ever had).
+ src_row = dict(chain[-1], nhash_before=chain[0]["nhash_before"], before_text=chain[0]["before_text"])
+ key = src_row["nhash_before"]
+ cur = {}
+ for r in rows:
+ if r.get("tu") and r.get("fn"):
+ cur[(r["tu"], r["fn"])] = r
+ sibs = [k for k, r in cur.items() if k != (tu, fn) and r.get("verdict") == "RESIDUE"
+ and (r.get("nhash_after") or r.get("nhash_before")) == key]
+ if a.only:
+ sibs = [k for k in sibs if any(o in k for o in a.only)]
+ sibs = sibs[:a.limit] if a.limit else sibs
+ print(f"delever --propagate: {tu}:{fn} -> {len(sibs)} sibling(s) of class {key[:12]}", flush=True)
+ if not sibs:
+ return 1 # R68: an empty work list is a refusal, not a success
+ ok = bad = 0
+ for stu, sfn in sibs:
+ path = REPO / stu
+ raw = path.read_text(errors="surrogateescape")
+ d = next((r for r in sc.scan_text(raw, stu, shared_defs=None) if r["form"] == "def" and r["name"] == sfn), None)
+ if d is None:
+ print(f" {stu}:{sfn}: not defined there — SKIPPED", flush=True)
+ bad += 1
+ continue
+ ls = line_starts(raw)
+ sib_before = raw[ls[d["line"] - 1]:ls[d["end"]]]
+ if lc.norm_hash(sc.mask_text(sib_before)) != key:
+ print(f" {stu}:{sfn}: its text is not this class any more — SKIPPED", flush=True)
+ bad += 1
+ continue
+ body, why = remap_body(src_row["before_text"], src_row["after_text"], sib_before)
+ if body is None:
+ print(f" {stu}:{sfn}: {why} — SKIPPED", flush=True)
+ bad += 1
+ continue
+ f = RUN / "propagate" / f"{stu.replace('/', '_')}__{sfn}.c"
+ f.parent.mkdir(parents=True, exist_ok=True)
+ f.write_text(body, errors="surrogateescape")
+ r = subprocess.run([sys.executable, str(REPO / "tools/delever.py"), "--apply-body", stu, sfn, str(f),
+ "--label", a.label, "--rung", src_row.get("rung") or "R", "--dirty-ok"],
+ cwd=REPO, capture_output=True, text=True)
+ line = ((r.stdout or r.stderr).strip().splitlines() or [""])[-1]
+ print(f" {line[:200]}", flush=True)
+ ok += r.returncode == 0
+ bad += r.returncode != 0
+ print(f"delever --propagate: {ok} of {len(sibs)} sibling(s) banked, {bad} refused")
+ return 0 if ok else 1
+
+
def status():
rows = load_ledger()
done, ex = ledger_index(rows)
@@ -1318,7 +1824,10 @@ def apply_body(a):
nhash_before=nh_before, nhash_after=nh_after, source=src, verdict=("LEVER-FREE" if v == "IDENTICAL" else f"BODY-{v}"),
sites=[dict(ord=i, kind=s["kind"], cls=s["cls"], detail=s["detail"], via=s.get("via", ""), line=s["line"], verdict="NEEDED",
why="left by the author", oracle="") for i, s in enumerate(levers)],
- compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_])
+ compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_],
+ # the body AS IT WAS: --propagate needs it to map this class's addresses onto a sibling's, and after the
+ # write it exists nowhere else (the tree has moved on and the ledger is the record)
+ before_text=before, after_text=new)
if v == "IDENTICAL":
path.write_text(cand, errors="surrogateescape")
ledger_append([row])
@@ -1569,6 +2078,47 @@ def selftest():
plan_, total_, _ = make_plan(fake_bodies, rows, False, 10, None)
if total_ != 1:
fail("a copy with a judged text but no row of its own must still be drawn")
+ # rung R's candidate generators, on a fixture whose every answer is known by hand
+ RFIX = ("void rfix(int p)\n{\n int a = p;\n int b;\n int c;\n"
+ " if (a == b) a = b & 3;\n c = a + 1;\n *(int *)(p + 4) = c;\n}\n")
+ rd = dict(line=1, end=9)
+ for line, want in [("s32 d = param_1;", True), ("ret = f();", False), ("u8 *p;", True), ("d = param_1;", False),
+ ("return x;", False), ("extern s32 D_1[];", True)]:
+ if is_decl_line(line) != want:
+ fail(f"is_decl_line({line!r}) != {want}")
+ if top_level_ops("a & b") != [2] or top_level_ops("a && b") or top_level_ops("(s32 *)p") \
+ or top_level_ops("*(s32 *)(p + 4) & 0xFF") != [16]:
+ fail("top_level_ops: a binary commutative operator at depth 0, and nothing else")
+ sw = commutative_swaps(RFIX, "src/x.c", "rfix", rd)
+ if [s for s, _ in sw] != ["swap & @6", "swap + @7"]:
+ fail(f"commutative_swaps found {[s for s, _ in sw]}")
+ if "a = 3 & b;" not in sw[0][1] or "c = 1 + a;" not in sw[1][1]:
+ fail("commutative_swaps must split at the assignment, never at `==`")
+ R6FIX = ("void r6(int p)\n{\n int v;\n int w;\n v = *(int *)(p + 4);\n"
+ " *(int *)(p + 4) = v & ~0x20;\n w = 3;\n *(int *)(p + 8) = w;\n}\n")
+ inl = inline_single_set_temps(R6FIX, "src/x.c", "r6", dict(line=1, end=9))
+ if [d for d, _ in inl] != ["inline v @5", "inline w @7"]:
+ fail(f"inline_single_set_temps found {[d for d, _ in inl]}")
+ if "= (*(int *)(p + 4)) & ~0x20;" not in inl[0][1] or "int v;" in inl[0][1]:
+ fail("R6 must inline the expression at the use AND drop the now-dead declaration")
+ bw = block_wraps(R6FIX, "src/x.c", "r6", dict(line=1, end=9))
+ if [d for d, _ in bw][:2] != ["block @5", "do-while @5"] or "{ v = *(int *)(p + 4); }" not in bw[0][1]:
+ fail(f"block_wraps: {[d for d, _ in bw][:3]} (the readable spelling first)")
+ cands = recipe_candidates(RFIX, "src/x.c", "rfix", ["a", "b"])
+ kinds_ = {r for r, _, _ in cands}
+ if not {"R2", "R3", "R5"} <= kinds_ or any(c == RFIX for _, _, c in cands):
+ fail(f"recipe_candidates: {kinds_} (the seed must never be a candidate)")
+ if any("a = p;" in c.split("\n")[7] for _, _, c in cands if _ == "R3"):
+ fail("R3 must place its assignment after the whole declaration run (C89)")
+ # the address remap that propagates a reshape to a class (R48-adjacent: one crack, 134 banks)
+ exb = "void func_80100000(void) { D_80200000 = func_80100004(); }"
+ exa = "void func_80100000(void) { s32 t = func_80100004(); D_80200000 = t; }"
+ sib = "void func_80300000(void) { D_80400000 = func_80300004(); }"
+ got, why = remap_body(exb, exa, sib)
+ if got != "void func_80300000(void) { s32 t = func_80300004(); D_80400000 = t; }":
+ fail(f"remap_body produced {got!r} ({why})")
+ if remap_body(exb, exa, "void func_80300000(void) { D_80400000 = 0; }")[0] is not None:
+ fail("remap_body must refuse a sibling with a different token count")
# the oracle's crash classification on its real message forms (R103)
if not oracle.SIGNAL_LINE.search("bash: line 1: 3845091 Done mipsel-linux-gnu-cpp ...\n 3845092 Aborted (core dumped) | tools/bin/gcc-2.7.2-psx/cc1 -quiet\n"):
fail("SIGNAL_LINE must match bash's job-status block")
@@ -1792,6 +2342,13 @@ def main():
ap.add_argument("--restore", action="store_true", help="restore every in-flight file from inflight.json")
ap.add_argument("--status", action="store_true")
ap.add_argument("--scrub", action="store_true", help="remove orphan !FAKE markers (a marker whose site is gone), byte-judged per file")
+ ap.add_argument("--propagate", nargs=2, metavar=("TU", "FN"),
+ help="give every RESIDUE sibling of this banked body's class the same shape, with its own addresses")
+ ap.add_argument("--recipes", action="store_true",
+ help="rung R: the cookbook's byte-neutral shape recipes tried mechanically on every RESIDUE body")
+ ap.add_argument("--cap", type=int, default=60, help="--recipes: candidates tried per body (each is one compile)")
+ ap.add_argument("--control", type=int, default=8, help="--recipes: how many LEVER-FREE bodies the control run reproduces (R39)")
+ ap.add_argument("--limit", type=int, help="--recipes: stop after this many RESIDUE bodies")
ap.add_argument("--apply-body", nargs=3, metavar=("TU", "FN", "FILE"))
ap.add_argument("--rung", default="E")
ap.add_argument("--allow-residue", action="store_true")
@@ -1810,6 +2367,14 @@ def main():
sys.exit(status())
if a.scrub:
sys.exit(scrub(a))
+ if a.propagate:
+ if not a.label:
+ sys.exit("delever --propagate: --label is required (R48)")
+ sys.exit(propagate(a))
+ if a.recipes:
+ if not a.label:
+ sys.exit("delever --recipes: --label is required (R48: the ledger rows are keyed by it)")
+ sys.exit(recipes(a))
if a.apply_body:
if not a.label:
sys.exit("delever --apply-body: --label is required")
diff --git a/decomp-architect/corpus/tools/P10/delever_permute.py b/decomp-architect/corpus/tools/P10/delever_permute.py
index 40495eec2..2e5a31141 100644
--- a/decomp-architect/corpus/tools/P10/delever_permute.py
+++ b/decomp-architect/corpus/tools/P10/delever_permute.py
@@ -23,10 +23,12 @@ THE PIPELINE PER EXEMPLAR (probe-proven at S98, the T6 log entry):
the permuter compiles must hold ONE .text function, because the masked scorer compares whole .text.
3. cpp -P with the BUILD's own CPPFLAGS + -I (common.h, the prelude, engine_types.h and the GTE header
expand here — p16_permute.make_base_c runs its own cpp WITHOUT includes and would lose them) -> draft.c.
- 4. the target .s — tools/verbatim_target_s.py regenerates it from the EXTRACTED ROM IMAGE (R34: an independent
- oracle; our own source is the thing under test), into the exemplar's scratch dir, where p16_permute.setup reads it.
- 5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds
- (pins -> regalloc, barriers/launders/keep-alives -> schedule, mixed -> regalloc).
+ 4. the target — the tree's OWN (levered) body compiled by the build's tail into a one-function object, so it
+ carries the candidates' relocations by construction; `tools/verbatim_target_s.py` regenerates the ROM listing beside it
+ and `match_one` must call that body a MATCH before the search starts (R34: the ROM image is the independent oracle that
+ PROVES the target; R56: check the baseline). S99 learned this the hard way — see THE INSTRUMENT below.
+ 5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds AND
+ the register each pin names (callee-saved -> regalloc, caller-saved -> cse, barriers/launders/keep-alives -> schedule).
6. a score-0 winner is a CANDIDATE, never a bank: --bank puts the function's definition back through
`delever.py --apply-body` (which refuses a body that still carries a class A/B lever and judges it on the object's bytes
through every recipe of the TU), then re-folds the cpp-expanded GTE asm with `gte_consolidate.py --apply --rejudge`.
@@ -36,6 +38,15 @@ SCRATCH IS KEYED BY ALIAS+FN (R48): `.run/P36/permuter/__/` — a fun
overlays overlap in RAM, so two different bodies can both be `func_8013B274`), and the permuter's own scratch/winner paths are
keyed by the bare name. Every attempt is recorded in `.run/P36/permuter/outcomes.jsonl`, which is also the skip list.
+THE INSTRUMENT, AND WHY IT IS CHECKED BEFORE IT IS BELIEVED (S99). The first two campaigns returned 0 of 16 with a straight
+face. Both were the harness: the target had been ASSEMBLED FROM A DISASSEMBLY LISTING, which is a second toolchain with its own
+answers — objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or` (24 wrong words in
+one 234-instruction function), and a listing's %hi/%lo pairs come back resolved with no relocation while every candidate carries
+one. The permuter scored 28 for a body that IS byte-identical, so score 0 was unreachable and every NO-MATCH was its own. The
+control that names this in one line is `--positive-control`, and the base score of the tree's own body is now 0. R40: exonerate
+the instrument before attributing a failure to its subject; the campaign that skips this control cannot tell a hard population
+from a broken scorer.
+
NEVER RUN THE CAMPAIGN AS A HARNESS BACKGROUND TASK (the low-memory guard kills it): `setsid nohup … &` + a Monitor on the log.
"""
import argparse
@@ -148,15 +159,32 @@ def exemplars(only=(), limit=None, include_done=False):
needed = [s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]
out.append(dict(nhash=nh, tu=tu, fn=fn, alias=(r.get("aliases") or [None])[0], copies=len(members),
needed=len(needed), kinds=sorted({s["kind"] for s in needed}),
+ regs=sorted({s.get("detail", "") for s in needed if s["kind"] == "pin"}),
members=[dict(tu=t, fn=f) for (t, f), _ in members], row=r))
out.sort(key=lambda e: (-e["copies"], e["needed"], e["tu"], e["fn"]))
out = [e for e in out if matches(e, only)]
return out[:limit] if limit else out
-def klass_for(kinds):
- """the permuter_weights profile for a residue's NEEDED-site mix (the profile NAME is accepted verbatim by classify())."""
+CALLEE_SAVED = {f"${n}" for n in range(16, 24)} | {f"$s{n}" for n in range(8)} | {"$fp", "$30"}
+
+
+def klass_for(kinds, regs=()):
+ """the permuter_weights profile for a residue, from the NEEDED sites' kinds AND the REGISTER each pin names (the profile
+ name is accepted verbatim by classify()).
+
+ The register is the part S99 learned by reading a residual instead of assuming one. A pin on a CALLEE-SAVED register
+ ($16-$23) is an allocation-ORDER residual — the regalloc profile's declaration/statement reordering is its lever. A pin
+ on a CALLER-SAVED one ($2/$3/$4-$7/…) is not: the S99 control on func_80163EC8 (`register … __asm__("$2")`) left exactly
+ a v0/v1 swap — `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — which is the
+ operand ORDER of one `&`, and the regalloc profile weights `perm_commutative` 2.0 while the cse profile weights it 40.0.
+ Steering that search by "it was a pin, so regalloc" spends the whole budget 20x away from the one lever that closes it."""
ks = set(kinds)
+ rs = {r for r in regs if r}
+ if "pin" in ks and rs and rs & CALLEE_SAVED:
+ return "regalloc"
+ if "pin" in ks and rs and not (rs & CALLEE_SAVED):
+ return "cse"
if ks and ks <= SCHED_KINDS:
return "schedule"
return "regalloc"
@@ -169,12 +197,17 @@ _sites_cache = None
def tu_sites(tu):
+ """the census's sites for one TU. Built ONCE, into a local, and published under the lock: the workers are threads, and
+ publishing the empty dict before filling it (2 s for 53,355 rows) let every other worker read "no site in this TU" and
+ report a whole batch UNSTRIPPABLE — a race that lies in the tool's own voice."""
global _sites_cache
- if _sites_cache is None:
- _sites_cache = collections.defaultdict(list)
- for s in dl.load_sites():
- _sites_cache[s["tu"]].append(s)
- return _sites_cache[tu]
+ with _LOCK:
+ if _sites_cache is None:
+ built = collections.defaultdict(list)
+ for s in dl.load_sites():
+ built[s["tu"]].append(s)
+ _sites_cache = built
+ return _sites_cache.get(tu, [])
def body_sites(tu, fn):
@@ -318,7 +351,7 @@ def scratch_of(alias, fn):
return RUN / f"{alias}__{fn}"
-def prepare(ex, quiet=False):
+def prepare(ex, quiet=False, require_sites=True):
"""(scratch dir, draft path, target path, klass) for one exemplar — the whole probe-2 pipeline."""
tu, fn = ex["tu"], ex["fn"]
alias = ex["alias"] or alias_of(tu, ex.get("row"))
@@ -327,7 +360,7 @@ def prepare(ex, quiet=False):
ex["alias"] = alias
raw = (REPO / tu).read_text(errors="surrogateescape")
sites = body_sites(tu, fn)
- if not sites:
+ if not sites and require_sites:
raise Unstrippable([("", 0, f"the census has no site in {tu}:{fn} (stale? rerun lever_census --sites)")])
d = scratch_of(alias, fn)
if d.exists():
@@ -357,10 +390,18 @@ def prepare(ex, quiet=False):
if r.returncode or not tgt.exists():
raise Unstrippable([("", 0, ((r.stderr or r.stdout).strip().splitlines() or ["no listing"])[-1][:160])])
shutil.copy(tgt, d / sub / f"{fn}.s") # p16_permute.setup / match_one.py read /.s
+ # THE TARGET OBJECT: the tree's own (levered) body through the build's own tail, so it carries the candidates'
+ # relocations by construction. Its fidelity is not assumed — `closeness(levered)` must be MATCH against the ROM
+ # listing above before any search runs (R34: the independent oracle proves the target; R56: check the baseline).
+ r = subprocess.run(["tools/permuter/compile.sh", str(d / "levered.c"), "-o", str(d / "target.o")],
+ capture_output=True, text=True, cwd=REPO)
+ if r.returncode or not (d / "target.o").exists():
+ raise Unstrippable([("", 0, ((r.stderr or r.stdout).strip().splitlines() or ["compile failed"])[-1][:160])])
if not quiet:
n = len((d / "draft.c").read_text(errors="surrogateescape").splitlines())
- print(f" prepared {alias}__{fn}: draft {n} lines, target {(r.stdout or '').strip().split()[-3:]}", flush=True)
- return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"])
+ print(f" prepared {alias}__{fn}: draft {n} lines, target.o {(d / 'target.o').stat().st_size} bytes "
+ f"+ the ROM listing", flush=True)
+ return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"], ex.get("regs", ()))
# ----------------------------------------------------------------------------------------------------------------------
@@ -382,6 +423,52 @@ def closeness(d, fn, c):
return (int(mm.group(1)) if mm else None), first
+def positive_control(a):
+ """Can rung D close a gap it is KNOWN to be able to close? Take a body the tree already matches, perturb it by ONE
+ reversible source change (the first commutative operand swap `delever` can generate), confirm the perturbed body no
+ longer matches, and give the permuter one cycle to find its way back to score 0.
+
+ Without this, "0 of 16" says nothing about the residue: it could equally be a harness that cannot reach ANY target
+ (R40 — exonerate the instrument before attributing the failure to its subject). With it, a PASS means the search
+ space, the target, the scorer and the winner path all work, and a 0-yield campaign is a fact about the population."""
+ tu, fn = a.positive_control
+ raw = (REPO / tu).read_text(errors="surrogateescape")
+ d_ = next((r for r in sc.scan_text(raw, tu, shared_defs=None) if r["form"] == "def" and r["name"] == fn), None)
+ if d_ is None:
+ sys.exit(f"delever_permute --positive-control: {fn} is not defined in {tu}")
+ swaps = dl.commutative_swaps(raw, tu, fn, d_)
+ if not swaps:
+ sys.exit(f"delever_permute --positive-control: {tu}:{fn} has no commutative operator to perturb — pick another body")
+ ex = dict(nhash="", tu=tu, fn=fn, alias=alias_of(tu), copies=1, needed=0, kinds=[], regs=[], members=[], row={})
+ d, draft, tgt, _ = prepare(ex, quiet=False, require_sites=False)
+ base, base_line = closeness(d, fn, draft)
+ desc, perturbed = swaps[a.which]
+ (d / "perturbed_tu.c").write_text(perturbed, errors="surrogateescape")
+ pert = drop_file_scope_asm(cpp_expand(tu, isolate(tu, perturbed, fn)), tu)
+ (d / "draft.c").write_text(pert, errors="surrogateescape") # the permuter's seed IS the perturbed body
+ after, after_line = closeness(d, fn, d / "draft.c")
+ print(f"positive control {alias_of(tu)}__{fn}: the tree's own body {base_line[:40]} · perturbed by `{desc}` {after_line[:60]}",
+ flush=True)
+ if base != 0:
+ sys.exit("delever_permute --positive-control: the UNPERTURBED body does not match — calibrate first (R56)")
+ if after == 0:
+ sys.exit(f"delever_permute --positive-control: `{desc}` is byte-neutral here — nothing to find; try --which {a.which + 1}")
+ rel = d.relative_to(REPO).as_posix()
+ log = d / "positive.log"
+ cmd = [PY, "tools/permuter_ils.py", fn, "--draft", rel + "/draft.c", "--asm-subdir", rel + "/gas",
+ "--klass", "cse", "--cycles", str(a.cycles), "--secs", str(a.secs), "--j", str(a.j),
+ "--winners", rel, "--pd", rel + "/pd"]
+ with open(log, "w") as f:
+ f.write(" ".join(cmd) + "\n\n")
+ f.flush()
+ subprocess.run(cmd, cwd=REPO, stdout=f, stderr=subprocess.STDOUT, timeout=a.cycles * (a.secs + 45) + 300)
+ out = log.read_text(errors="replace")
+ won = (d / f"{fn}.c").exists()
+ print(f"positive control: {'PASS — the permuter recovered score 0' if won else 'FAIL — it did not, in '
+ f'{a.cycles}x{a.secs}s'} ({[l for l in out.splitlines() if 'cycle' in l][-1:] or ['no cycle line']})")
+ return 0 if won else 1
+
+
def calibrate_one(ex, quiet=False):
"""the control: the body AS THE TREE HAS IT (levers and all) must be MATCH against the regenerated target. It proves the
whole chain — the target from the ROM image, the isolation, the cpp expansion, the pinned triple — is measuring THIS
@@ -405,7 +492,7 @@ BEST_RE = re.compile(r"best score = (\d+)")
def run_one(ex, secs, cycles, j, max_start=None):
t0 = time.time()
row = dict(kind="attempt", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=ex["nhash"], tu=ex["tu"], fn=ex["fn"],
- alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"],
+ alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"], regs=ex.get("regs", []),
secs=secs, cycles=cycles, j=j)
try:
d, draft, tgt, klass = prepare(ex)
@@ -437,7 +524,7 @@ def run_one(ex, secs, cycles, j, max_start=None):
log = d / "ils.log"
cmd = [PY, "tools/permuter_ils.py", ex["fn"], "--draft", (draft.relative_to(REPO)).as_posix(),
"--asm-subdir", rel + "/gas", "--klass", klass, "--cycles", str(cycles), "--secs", str(secs),
- "--j", str(j), "--winners", rel, "--pd", rel + "/pd"]
+ "--j", str(j), "--winners", rel, "--pd", rel + "/pd", "--target-o", rel + "/target.o"]
row["klass"] = klass
row["cmd"] = " ".join(cmd)
rc = None
@@ -485,10 +572,34 @@ def winner_body(winner_c, fn, tu):
return None
+def tidy_body(body):
+ """the permuter's winner, made to read like the tree it lands in — or None when there is nothing to tidy.
+
+ pycparser reprints a body it has parsed: two-space indent where this tree uses four, and an empty statement `;` left
+ where a statement was inlined away. Neither generates code, and the object oracle judges the tidied text before it is
+ kept, so this can only ever improve the source. What it deliberately does NOT touch is the parenthesisation
+ (`*((s32 *) (p + 4))`) or the brace style — that is the formatting phase's job, with clang-format, over the whole tree
+ at once (`decomp-architect/templates/.clang-format`); a per-bank reformat would make this phase's diffs unreadable."""
+ lines = [l for l in body.split("\n") if l.strip() != ";"]
+ indents = [len(l) - len(l.lstrip(" ")) for l in lines if l.strip() and l.startswith(" ")]
+ if indents and min(indents) == 2: # a two-space body: double every leading run
+ lines = [(" " * (2 * (len(l) - len(l.lstrip(" ")))) + l.lstrip(" ")) if l.startswith(" ") else l for l in lines]
+ out = "\n".join(lines)
+ return out if out != body else None
+
+
def bank(a):
outs = load_outcomes()
banked = {(o["alias"], o["fn"], o["nhash"]) for o in outs if o.get("kind") == "bank" and o.get("applied")}
+ # a body ANOTHER rung already closed is not banked again (S99: --bank re-applied the permuter's reprinted body over the
+ # one-line version rung R had banked for the same function, and the tree lost a clean diff for a byte-identical one)
+ cur = {}
+ for r in dl.load_ledger():
+ if r.get("tu") and r.get("fn"):
+ cur[(r["tu"], r["fn"])] = r
+ already = {(k[0], k[1]) for k, r in cur.items() if r.get("verdict") == "LEVER-FREE"}
todo = [o for o in outs if o.get("kind") == "attempt" and o.get("verdict") == "MATCH"
+ and (o["tu"], o["fn"]) not in already
and (o["alias"], o["fn"], o["nhash"]) not in banked
and (not a.only or any(x in (o["fn"], o["tu"], o["alias"]) or o["nhash"].startswith(x) for x in a.only))]
if not todo:
@@ -503,11 +614,17 @@ def bank(a):
bad += 1
continue
bf = d / "body.c"
- bf.write_text(body, errors="surrogateescape")
- cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(),
- "--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else [])
- r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
- print(" " + (r.stdout or r.stderr).strip().splitlines()[-1][:220], flush=True)
+ # the tidied body FIRST (the oracle judges it like any other candidate); the winner verbatim only if it is refused
+ forms = [(t_, "tidied") for t_ in [tidy_body(body)] if t_] + [(body, "verbatim")]
+ r = None
+ for text_, how in forms:
+ bf.write_text(text_, errors="surrogateescape")
+ cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(),
+ "--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else [])
+ r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
+ print(f" [{how}] " + (r.stdout or r.stderr).strip().splitlines()[-1][:210], flush=True)
+ if r.returncode == 0:
+ break
row = dict(kind="bank", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=o["nhash"], tu=o["tu"], fn=o["fn"],
alias=o["alias"], label=a.label, applied=(r.returncode == 0),
apply_out=(r.stdout or r.stderr).strip()[-400:])
@@ -535,7 +652,8 @@ def cmd_plan(a):
f"({sum(e['copies'] for e in ex):,} bodies behind them); NEEDED kinds {dict(kinds)}")
for e in ex[:a.show]:
print(f" {e['copies']:4d} copies · {e['needed']:2d} needed {','.join(e['kinds']):<28} "
- f"{klass_for(e['kinds']):8} {e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}")
+ f"{klass_for(e['kinds'], e.get('regs', ())):8} {','.join(e.get('regs', [])) or '-':<12} "
+ f"{e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}")
if len(ex) > a.show:
print(f" … {len(ex) - a.show} more")
return 0
@@ -667,6 +785,8 @@ def main():
g = ap.add_mutually_exclusive_group(required=True)
g.add_argument("--plan", action="store_true")
g.add_argument("--prepare", nargs=2, metavar=("TU", "FN"))
+ g.add_argument("--positive-control", nargs=2, metavar=("TU", "FN"),
+ help="perturb a MATCHING body by one commutative swap and require the permuter to find its way back")
g.add_argument("--calibrate", action="store_true",
help="the control: N exemplars' levered bodies must all be MATCH against their regenerated targets")
g.add_argument("--run", action="store_true")
@@ -682,6 +802,7 @@ def main():
ap.add_argument("-j", type=int, default=16)
ap.add_argument("--label", default="d1")
ap.add_argument("--dirty-ok", action="store_true")
+ ap.add_argument("--which", type=int, default=0, help="--positive-control: which commutative swap to perturb with")
ap.add_argument("--max-start", type=int,
help="skip (as FAR) an exemplar whose lever-free body is further than N instructions from the target")
ap.add_argument("--include-done", action="store_true", help="draw classes that already have an outcome row")
@@ -702,6 +823,8 @@ def main():
d, draft, tgt, klass = prepare(ex)
print(f"delever_permute --prepare: {d.relative_to(REPO)} (draft.c, {tgt.name}, klass={klass})")
return 0
+ if a.positive_control:
+ return positive_control(a)
if a.calibrate:
return cmd_calibrate(a)
if a.run:
diff --git a/decomp-architect/corpus/tools/P10/lever_progress.py b/decomp-architect/corpus/tools/P10/lever_progress.py
new file mode 100644
index 000000000..e73dba7c2
--- /dev/null
+++ b/decomp-architect/corpus/tools/P10/lever_progress.py
@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""lever_progress.py — the lever-removal series: what the compiler-forcing constructs cost, and how they came off.
+
+ tools/lever_progress.py --snapshot "T6 r1" # append today's census totals as a milestone row (evidence, with HEAD)
+ tools/lever_progress.py --render # regenerate the generated block of docs/levers.md from the data
+ tools/lever_progress.py --table # print both tables
+ tools/lever_progress.py --check # is the last snapshot this tree's? (a stale series is a wrong chart)
+
+WHY THIS EXISTS. The phase that takes the levers out is the only place the project ever counts them, and the count is worth
+more than the phase: it is the post-100% chart, the story's spine, and the evidence behind the day-one kit's advice on
+whether to prevent levers from the first bank or clean them up at the end. Numbers are DERIVED, never typed (R75).
+
+TWO TABLES, TWO SOURCES:
+ * MILESTONES — one appended row per census snapshot (`--snapshot`), because a census is a moment: it is not recoverable
+ later. Each row carries the tree's HEAD and the census's own totals, so a reader can re-derive it from that commit.
+ * THE CAMPAIGN — fully derived from `.run/P36/delever/ledger.jsonl` on every run: per batch label, the date, the rungs
+ used, the bodies first judged, the sites REMOVED / REWRITTEN, and — scored as a TRANSITION against what that body's
+ previous row left — the sites a later rung closed and the bodies it made lever-free. Rebuilt from scratch each render,
+ so it cannot drift.
+"""
+import argparse
+import collections
+import json
+import pathlib
+import subprocess
+import sys
+
+REPO = pathlib.Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO / "tools"))
+
+CENSUS = REPO / ".run" / "P36" / "census" / "lever_census.json"
+LEDGER = REPO / ".run" / "P36" / "delever" / "ledger.jsonl"
+SERIES = REPO / "docs" / "lever-progress.tsv"
+DOC = REPO / "docs" / "levers.md"
+BEGIN = ""
+END = ""
+COLS = ["date", "milestone", "head", "sites_AB", "pins", "asm", "bodies_AB", "distinct_AB", "marked", "unmarked",
+ "gte_levers", "per_tu_asm_macros", "class_C", "class_D", "class_E", "class_F", "class_G"]
+
+
+def census():
+ if not CENSUS.exists():
+ sys.exit("lever_progress: no census — run tools/lever_census.py --sites -j 16")
+ return json.loads(CENSUS.read_text())
+
+
+def ledger():
+ rows = []
+ if LEDGER.exists():
+ for l in LEDGER.read_text().splitlines():
+ if l.strip():
+ rows.append(json.loads(l))
+ return rows
+
+
+def snapshot_row(milestone):
+ d = census()
+ ab, cl = d["levers_AB"], d["classes"]
+ return dict(date=d.get("generated", ""), milestone=milestone, head=d.get("head", ""),
+ sites_AB=ab["sites"], pins=ab["pins"], asm=ab["asm"], bodies_AB=ab["bodies"],
+ distinct_AB=ab["distinct_bodies"], marked=ab["marked"], unmarked=ab["unmarked"],
+ gte_levers=d.get("gte_levers", {}).get("sites", 0),
+ per_tu_asm_macros=d.get("per_tu_asm_macro_definitions", {}).get("total", 0),
+ **{f"class_{k}": cl.get(k, {}).get("sites", 0) for k in "CDEFG"})
+
+
+def read_series():
+ if not SERIES.exists():
+ return []
+ lines = [l for l in SERIES.read_text().splitlines() if l.strip()]
+ if not lines:
+ return []
+ head = lines[0].split("\t")
+ return [dict(zip(head, l.split("\t"))) for l in lines[1:]]
+
+
+def write_series(rows):
+ SERIES.write_text("\t".join(COLS) + "\n" + "".join("\t".join(str(r.get(c, "")) for c in COLS) + "\n" for r in rows))
+
+
+def campaign():
+ """per batch label, derived from the ledger, as STATE TRANSITIONS — not as "the first row wins".
+
+ A body is judged more than once on purpose: rung A/B settles it with some sites left NEEDED, and a later rung (R, D or
+ an agent) comes back and closes those. Counting only a body's first row credits the whole campaign to T4 and reports
+ "0 removed" for the rung that actually finished the job (rung R's first batch closed 134 bodies and read as nothing).
+ So each row is scored against what that body's previous row left: `removed`/`rewritten` are the first pass's, and
+ `closed_later` is the fall in the NEEDED count afterwards, with `made_free` counting the bodies that reached zero."""
+ prev, per = {}, collections.OrderedDict()
+ for r in ledger():
+ lab = r.get("label") or "-"
+ key = (r.get("tu"), r.get("fn"))
+ ts = (r.get("ts") or "")[:10]
+ e = per.setdefault(lab, dict(label=lab, date=ts, rungs=set(), bodies=0, removed=0, rewritten=0,
+ closed_later=0, made_free=0, refused=0))
+ if ts and (not e["date"] or ts < e["date"]):
+ e["date"] = ts
+ if r.get("rung"):
+ e["rungs"].add(str(r["rung"]))
+ sites = r.get("sites", [])
+ now_needed = sum(1 for s in sites if s.get("verdict") == "NEEDED")
+ was = prev.get(key)
+ if was is None:
+ e["bodies"] += 1
+ e["removed"] += sum(1 for s in sites if s.get("verdict") == "REMOVED")
+ e["rewritten"] += sum(1 for s in sites if s.get("verdict") == "REWRITTEN")
+ e["refused"] += sum(1 for s in sites if s.get("verdict") == "REFUSED")
+ else:
+ e["closed_later"] += max(0, was - now_needed)
+ if was > 0 and now_needed == 0:
+ e["made_free"] += 1
+ prev[key] = now_needed
+ return list(per.values())
+
+
+def md_tables():
+ rows = read_series()
+ out = [BEGIN, "", "### Milestones — the count, at each moment a census ran", "",
+ "| date | milestone | pins | asm | **class A+B sites** | bodies | distinct | GTE levers | per-TU asm macros | C | D | E | F | G | HEAD |",
+ "|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|"]
+ for r in rows:
+ out.append("| {date} | {milestone} | {pins} | {asm} | **{sites_AB}** | {bodies_AB} | {distinct_AB} | {gte_levers} | "
+ "{per_tu_asm_macros} | {class_C} | {class_D} | {class_E} | {class_F} | {class_G} | `{head}` |".format(**r))
+ out += ["", "### The campaign — every batch, from the ledger (derived on every render)", "",
+ "| batch | date | rungs | bodies first judged | sites removed | rewritten | sites closed later | bodies made lever-free |",
+ "|---|---|---|---:|---:|---:|---:|---:|"]
+ tot = dict(bodies=0, removed=0, rewritten=0, closed_later=0, made_free=0)
+ for e in campaign():
+ if not any(e[k] for k in tot):
+ continue # a bookkeeping batch (a scrub, a re-mark) moved no site
+ out.append(f"| `{e['label']}` | {e['date']} | {','.join(sorted(e['rungs'])) or '-'} | {e['bodies']:,} | "
+ f"{e['removed']:,} | {e['rewritten']:,} | {e['closed_later']:,} | {e['made_free']:,} |")
+ for k in tot:
+ tot[k] += e[k]
+ out.append(f"| **total** | | | **{tot['bodies']:,}** | **{tot['removed']:,}** | **{tot['rewritten']:,}** | "
+ f"**{tot['closed_later']:,}** | **{tot['made_free']:,}** |")
+ out += ["", f"*Generated by `tools/lever_progress.py --render` from `.run/P36/census/lever_census.json` "
+ f"and `.run/P36/delever/ledger.jsonl`; the series lives in `docs/lever-progress.tsv` (R75: published "
+ f"numbers are generated, never typed).*", "", END]
+ return "\n".join(out)
+
+
+def render():
+ if not DOC.exists():
+ sys.exit(f"lever_progress: {DOC.relative_to(REPO)} does not exist yet")
+ t = DOC.read_text()
+ if BEGIN not in t or END not in t:
+ sys.exit(f"lever_progress: {DOC.relative_to(REPO)} has no generated block ({BEGIN} … {END})")
+ pre, rest = t.split(BEGIN, 1)
+ _, post = rest.split(END, 1)
+ DOC.write_text(pre + md_tables() + post)
+ print(f"lever_progress: rendered {len(read_series())} milestone row(s) + {len(campaign())} batch row(s) "
+ f"into {DOC.relative_to(REPO)}")
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--snapshot", metavar="MILESTONE", help="append today's census totals as a milestone row")
+ ap.add_argument("--render", action="store_true")
+ ap.add_argument("--table", action="store_true")
+ ap.add_argument("--check", action="store_true")
+ a = ap.parse_args()
+ if a.snapshot:
+ rows = read_series()
+ row = snapshot_row(a.snapshot)
+ rows = [r for r in rows if not (r.get("milestone") == row["milestone"] and r.get("head") == row["head"])]
+ rows.append(row)
+ write_series(rows)
+ print(f"lever_progress: {row['milestone']} — {row['sites_AB']:,} class A+B sites "
+ f"({row['pins']:,} pins + {row['asm']:,} asm) in {row['bodies_AB']:,} bodies, "
+ f"{row['unmarked']} unmarked, at {row['head']}")
+ if a.render or a.snapshot:
+ if DOC.exists():
+ render()
+ if a.table:
+ print(md_tables())
+ if a.check:
+ rows, d = read_series(), census()
+ if not rows:
+ sys.exit("lever_progress --check: the series is empty (run --snapshot)")
+ last = rows[-1]
+ if last.get("head") != d.get("head") or int(last.get("sites_AB", -1)) != d["levers_AB"]["sites"]:
+ sys.exit(f"lever_progress --check: the series' last row ({last.get('milestone')}, {last.get('sites_AB')} sites "
+ f"at {last.get('head')}) is not this tree ({d['levers_AB']['sites']} at {d.get('head')}) — "
+ f"run --snapshot after the census (R75)")
+ print(f"lever_progress --check: OK — {len(rows)} milestone(s), last {last['milestone']} "
+ f"({last['sites_AB']} class A+B sites)")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/decomp-architect/corpus/tools/P10/verbatim_target_s.py b/decomp-architect/corpus/tools/P10/verbatim_target_s.py
index adfc25a9c..1ae9536de 100644
--- a/decomp-architect/corpus/tools/P10/verbatim_target_s.py
+++ b/decomp-architect/corpus/tools/P10/verbatim_target_s.py
@@ -165,6 +165,43 @@ def to_gas(insns, vaddr, off, symtab):
return out
+def gas_roundtrip(rows, insns, fn):
+ """[(row index, ROM word)] for every emitted instruction whose RE-ASSEMBLY does not reproduce the ROM's word.
+
+ WHY (S99, and it invalidated a whole campaign before it was found): objdump prints the PSEUDO-instruction
+ `move s2,a0` for `addu s2,a0,$zero` (0x00809021), and gas assembles `move` as `or` (0x00809025). Every `move` in
+ the listing therefore came back a different word, so a target object built from this form differed from the ROM
+ in every one of them — and the permuter, scoring against it, reported 28 for a body that IS byte-identical. A
+ listing nothing ever assembled and compared is a claim, not an oracle (R98/DK-81 in a second place). Words that
+ carry a RELOCATION (jal/j, HI16/LO16) are excluded: the linker fills those, and both sides mask them."""
+ body = "\n".join([f".include \"macro.inc\"", ".set noat", ".set noreorder", ".section .text", f"glabel {fn}"] + rows)
+ with tempfile.NamedTemporaryFile("w", suffix=".s", delete=False) as fh:
+ fh.write(body + "\n")
+ spath = fh.name
+ opath = spath[:-2] + ".o"
+ try:
+ r = subprocess.run(["mipsel-linux-gnu-as", "-I", os.path.join(REPO, "include"), "-march=r3000", "-mtune=r3000",
+ "-no-pad-sections", "-O1", "-G0", spath, "-o", opath], capture_output=True, text=True)
+ if r.returncode:
+ return None, (r.stderr or "").strip().splitlines()[:3]
+ sys.path.insert(0, os.path.join(REPO, "tools"))
+ import masked_diff
+ got = masked_diff.insns_from_object(opath, None)
+ finally:
+ for p in (spath, opath):
+ if os.path.exists(p):
+ os.unlink(p)
+ if len(got) != len(insns):
+ return None, [f"re-assembly produced {len(got)} instructions, the image has {len(insns)}"]
+ bad = []
+ for k, (g, (word, _text)) in enumerate(zip(got, insns)):
+ if g["reloc_kind"]:
+ continue
+ if g["word"] != word:
+ bad.append((k, word))
+ return bad, None
+
+
def emit(binary, fn, outdir, quiet=False, gas=False):
fr = _fr()
vaddr, nins = func_extent(binary, fn)
@@ -195,12 +232,34 @@ def emit(binary, fn, outdir, quiet=False, gas=False):
fh.write(f' * thing under test. {nins} instructions at 0x{vaddr:08X}. $-registers, .L labels for\n')
fh.write(f' * in-function targets, symbol names for external jumps; the listing carries its delay-slot nops\n')
fh.write(f' * (hence .set noreorder). Paste WHOLE into decomp.me — its PS1 prelude defines glabel. */\n\n')
+ rows = to_gas(insns, vaddr, off, _symtab(binary))
+ # the listing must RE-ASSEMBLE to the image's own words, or it is not a target (S99). Each row that does
+ # not is replaced by its `.word`, with the mnemonic kept in the comment for the reader; then it is
+ # verified again, and a listing that still disagrees is REFUSED rather than emitted (R43).
+ patched = 0
+ for _ in range(3):
+ bad, err = gas_roundtrip(rows, insns, fn)
+ if bad is None:
+ return None, f'{binary}:{fn}: the gas form does not assemble ({err})'
+ if not bad:
+ break
+ # `rows` interleaves `.L:` label lines with instructions; `bad` counts INSTRUCTIONS. Indexing rows
+ # by an instruction index rewrote a label into a comment and the next round refused to assemble.
+ ins_rows = [i for i, r in enumerate(rows) if r.startswith('/*')]
+ for k, word in bad:
+ head, _, text = rows[ins_rows[k]].partition('*/')
+ rows[ins_rows[k]] = f'{head}*/ .word 0x{word:08X} /* {text.strip()} */'
+ patched += len(bad)
+ else:
+ return None, (f'{binary}:{fn}: {len(bad)} instruction(s) still do not re-assemble to the image '
+ f'(first at +{bad[0][0] * 4:#x}) — REFUSING to emit a target that is not the bytes')
fh.write('.set noat\n.set noreorder\n\n.section .text\n\n')
fh.write(f'glabel {fn}\n')
- for ln in to_gas(insns, vaddr, off, _symtab(binary)):
+ for ln in rows:
fh.write(ln + '\n')
if not quiet:
- print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form)')
+ print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form'
+ + (f', {patched} word-patched)' if patched else ')'))
return path, None
path = os.path.join(outdir, binary, f'{fn}.s')
with open(path, 'w') as fh:
diff --git a/decomp-architect/corpus/tools/P7/p16_permute.py b/decomp-architect/corpus/tools/P7/p16_permute.py
index 2f66ab554..205576a0f 100644
--- a/decomp-architect/corpus/tools/P7/p16_permute.py
+++ b/decomp-architect/corpus/tools/P7/p16_permute.py
@@ -233,7 +233,7 @@ def klass_for_fn(fn):
return "", ""
-def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None):
+def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None, target_o=None):
# `outdir` keys the scratch dir by the CALLER's identity instead of the bare function name (R48). A name is not
# unique across the fleet — the overlays overlap in RAM, so two different bodies are both `func_8013B274` — and two
# concurrent runs on one name would share (and corrupt) this directory. Default: the historical path, unchanged.
@@ -247,6 +247,22 @@ def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None):
f"'not found in base.c' and no-op in 0s. Inspect {pd}/base.c "
f"(prep order: comments -> cpp macros -> M2C_FIELD -> hide_asm -> typedefs).")
open(f"{pd}/base.c", "w").write(base)
+ if target_o:
+ # THE TARGET AS AN OBJECT, not a listing (S99). Assembling a disassembly listing is a second toolchain with its
+ # own answers: objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or`
+ # (24 words wrong in one 234-instruction function), and a listing's %hi/%lo pairs come back RESOLVED, with no
+ # relocation, while every candidate carries one — and the masked scorer compares reloc operands. Both made the
+ # scorer report a nonzero floor for a body that IS byte-identical, so score 0 was unreachable and every verdict
+ # was the instrument's. A target object compiled from the tree's own body by THIS pipeline has the candidate's
+ # relocations by construction; the ROM listing stays the independent oracle that PROVES it (match_one MATCH
+ # before the search starts, R34/R56) — the proof is what keeps this from being circular.
+ shutil.copy(target_o, f"{pd}/target.o")
+ body, prof = permuter_weights.render_settings_toml(fn, klass=klass, where=where)
+ open(f"{pd}/settings.toml", "w").write(body)
+ csh = "compile_o0.sh" if asm_subdir.rstrip("/").endswith("_o0") else "compile.sh"
+ open(f"{pd}/compile.sh", "w").write(f'#!/bin/bash\nexec {REPO}/tools/permuter/{csh} "$@"\n')
+ os.chmod(f"{pd}/compile.sh", 0o755)
+ return pd
s = os.path.join(REPO, asm_subdir, fn + ".s")
tgt = f"{pd}/target.s"
with open(tgt, "w") as f:
diff --git a/decomp-architect/corpus/tools/P7/permuter_ils.py b/decomp-architect/corpus/tools/P7/permuter_ils.py
index 106a01c23..fe05bed7c 100644
--- a/decomp-architect/corpus/tools/P7/permuter_ils.py
+++ b/decomp-architect/corpus/tools/P7/permuter_ils.py
@@ -41,12 +41,14 @@ def main():
ap.add_argument("--secs", type=int, default=180, help="per-cycle time box")
ap.add_argument("--j", type=int, default=12)
ap.add_argument("--winners", default=".run/permuter-winners")
+ ap.add_argument("--target-o", help="use this one-function object as the target instead of assembling "
+ "/.s (S99: a disassembly listing is a second toolchain)")
ap.add_argument("--pd", help="scratch dir for this run (default .run/permuter/) — key it by alias+fn when "
"several runs share a function NAME (R48)")
a = ap.parse_args()
draft = open(a.draft).read()
- pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd)
+ pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd, target_o=a.target_o)
if not pd:
print("ILS setup FAILED (target .s didn't assemble?)"); sys.exit(1)
print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}")
diff --git a/decomp-architect/tools/MANIFEST.md b/decomp-architect/tools/MANIFEST.md
index 1d3549ec0..a7c12c1f0 100644
--- a/decomp-architect/tools/MANIFEST.md
+++ b/decomp-architect/tools/MANIFEST.md
@@ -8,8 +8,8 @@
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
>
-> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
-> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
+> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows
+> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure).
## P1 — extraction + manifest
@@ -335,6 +335,7 @@
| `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout |
| `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout |
| `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule |
+| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger |
| `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas |
| `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name |
| `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing |
@@ -343,7 +344,7 @@
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
-| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
+| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
diff --git a/docs/tool-index.md b/docs/tool-index.md
index a8626b363..3085403d8 100644
--- a/docs/tool-index.md
+++ b/docs/tool-index.md
@@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on
dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool,
then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.*
-**Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 30 (a SETUP row only), ORPHAN 30 (neither) — of 301. Portability: PORTABLE 23, ADAPT 261, PROJECT-ONLY 17.
+**Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 31 (a SETUP row only), ORPHAN 30 (neither) — of 302. Portability: PORTABLE 23, ADAPT 262, PROJECT-ONLY 17.
## P1 — extraction + manifest
@@ -71,7 +71,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
| When you need to… | Tool | What it does | Proven by | Adapt | Class |
|---|---|---|---|---|---|
-| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+6) | MIPS relocation encodings | LIVE |
+| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+7) | MIPS relocation encodings | LIVE |
| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+23) | compiler triple, repo build flags | LIVE |
| dump every compiler pass file for a self-contained draft | `cc1_dumps.sh` | Dumps every compiler pass file for a self-contained draft into a private directory | cc1_dumps_tu.sh, ghost_census.py | compiler triple, repo scratch paths | LIVE |
| dump every compiler pass file for the spliced real translation unit | `cc1_dumps_tu.sh` | Same pass dumps for the spliced real translation unit, the faithful compile | alloc_table.py | absolute repo path, compiler triple | LIVE |
@@ -295,7 +295,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
| build a progress timeline from the repository's own committed digests | `timeline.py` | Builds a progress timeline from the repository's own committed digests, with a self-check | Makefile | repo doc paths | LIVE |
| check that every relative link in the public docs resolves | `doc_links.py` | Checks that every relative link in the public docs resolves and the link policy holds | Makefile, wiki_render.py, wiki_sync.sh | repo doc paths | LIVE |
| compile every eligible translation unit with the pinned toolchain, using no derived bytes | `compile_only.py` | Compiles every eligible translation unit with the pinned toolchain, without any game bytes | .github/workflows/no-rom.yml, delever_oracle.py, lever_census.py, macro_to_header.py (+1) | compiler triple, repo makefile parsing | LIVE |
-| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+33) | repo src/config layout | LIVE |
+| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+34) | repo src/config layout | LIVE |
| convert progress metrics into an external progress-report schema | `objdiff_report.py` | Converts the progress JSON into the external progress-report schema | .github/workflows/progress.yml | external report schema | LIVE |
| emit the public ordinal-to-hash map and the private old-to-new map | `public_rewrite/build_commit_map.py` | Emits the public ordinal-to-new-hash map and the private old-to-new map | probe_github.sh, resolve_tokens.py | repo doc/scratch paths | LIVE |
| gate a first push: prove no derived bytes among tracked files | `audit_public.py` | The first-push gate: no derived bytes among tracked files, four independent checks | .github/workflows/no-rom.yml, gate_scan.py | repo purge-set and path lists (the kit's template reads `config/firewall.txt` instead) | LIVE |
@@ -322,7 +322,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|---|---|---|---|---|---|
| add per-site function-pointer casts so a draft can call a differently typed callee | `cast_call_sites.py` | Adds per-site function-pointer casts so a draft can call a differently typed callee | canon_sig_reconcile.py, cast_self_callers.py, family_sweep.py, gate_stage.py (+4) | repo src layout | LIVE |
| apply curated names and signatures inside the analysis tool and save | `ghidra_scripts/ApplySymbols.java` | The in-tool half of that mirror: apply curated names and signatures, save on exit | ghidra_apply_symbols.sh, ghidra_export_annotations.sh, ghidra_rebuild.sh, ExportAnnotations.java (+1) | none | LIVE |
-| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+2) | repo paths, the shared-body scanner | LIVE |
+| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+3) | repo paths, the shared-body scanner | LIVE |
| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked _m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever.py, delever_cycle.sh, delever_permute.py, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE |
| convert a shared-body macro header into per-function plain-C headers included at each site | `macro_to_header.py` | The Phase-35 converter: every DEFINE_ macro body becomes a plain-C header under src/shared// included at its site, the legacy name-parameterized headers converted, the registry text-edited, the macro header deleted; --plan / --apply / --finalize / --verify | share_body.py, tool_census.py | the macro form is this project's; a kit-born project shares headers from its first bank | LIVE |
| emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE |
@@ -331,6 +331,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
| give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED |
| guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE |
| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE |
+| keep the lever count as a series a chart and a story can be drawn from | `lever_progress.py` | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | — | the census json, the delever ledger | REFERENCED |
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+10) | repo paths, the registry and signature schemas | LIVE |
| mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED |
| refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE |
@@ -339,7 +340,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
| run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | `share_body_cycle.sh` | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | — | repo paths, the phase-log format, the fleet count from config/check.*.sha | ORPHAN |
| search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED |
| share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE |
-| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
+| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
| turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED |
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md
index 3ddc5c739..9b4ac2999 100644
--- a/phase-ends/CURRENT_PHASE.md
+++ b/phase-ends/CURRENT_PHASE.md
@@ -62,7 +62,7 @@
- Rules check (P6) after T4.
- ☐ **T5** — GTE consolidation (one header, PsyQ's names; the `"memory"`-clobber variants are levers → residue, not a second spelling) +
the dead lever-macro sweep (`SHB` cross-file use counted). Verify: `lever_census` 0 per-TU asm macro definitions; R22 green.
-- ☐ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed,
+- ☑ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed,
16 workers; a score-0 applied through `delever --apply-body` and gated). Verify: `recipes: of NEEDED sites removed`;
`permuter: of exemplars matched lever-free in h`.
- ☐ **T7** (Ultracode, prompted) — The reshaping waves to zero: wave → harvest → toolify (cookbook + rung R) → sweep → next draw; wave 1
@@ -621,7 +621,83 @@ accumulate here as the phase produces them.**
refused more than a third of the population while the context was hot, and harvest→toolify applied to lever classes so a shape
sweeps the banked population instead of the lever propagating with every family remap (1,758 distinct bodies became 12,578).
-## 🛑 SESSION CHECKPOINT — S98 (2026-09-09): T0 ☑ T1 ☑ T1b ☑ T2 ☑ T3 ☑ T4 ☑ T5 ☑ — all committed (this close on top of `4bc987426`); NEXT = T6 BUILD: `tools/delever_permute.py` per §2 (the design is settled and its pipeline proven by two probes — the T6 log entry); S98 ended here at 90 % context | the number: 34,091 sites in 12,712 bodies (1,759 distinct) · marked 34,091 · UNMARKED 0 · orphans 0 · GTE levers 462 · per-TU asm macro definitions 314 (0 canonical duplicates) — `lever_census --check` OK
+- **S99 — T6 CLOSE: both yield lines measured, 664 lever sites gone, R22 green at every step. T6 ☑.**
+ **The two lines the plan asks for (R41):** `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of 2,131 bodies
+ behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in 6.0 min`.
+ **Banked this session:** rung R's batch `r1` (134 bodies, every one `R7 do-while @`, a ONE-LINE source change each) ·
+ rung D's four remaining exemplars (`d1`, tidied and judged) · the propagation `p1`/`p1b` (132 + 130 + 131 + 132 = **525
+ siblings**, 0 refused, each judged on its own objects). **34,091 → 33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies,
+ 33,427 marked, 0 UNMARKED, 0 orphans; `check-all: 218 passed, 0 failed of 218` after every batch.
+ **What T6 leaves for T7, stated rather than smoothed over:** rung R's recipe set does NOT reproduce the shape rung D found on
+ func_80163EC8 — R6 was generalised from "assigned once, read once" to "dead after one read" (the lever was one of two
+ assignments to `uVar5`) and still does not close it (154 candidates, 1.3 min, 0 hits), so that class was banked from the
+ permuter's own winner instead. The 11 exemplars rung D did not close are seeds, not walls: every one improved (78→9, 52→3,
+ 50→3, 37→2, 131→35) and the ones that stayed far all removed a hand-placed `instruction` whose C spelling changed the
+ instruction COUNT. A fleet-wide rung-R sweep is priced but not run: one body costs ~150 compiles / ~1.3 min at `--cap 200`
+ when nothing hits, so the 12,000-body residue needs the run parallelised across TUs before it is affordable.
+ **Instrument work banked with it:** `--recipes` is killable (its judge snapshots to inflight.json first — a killed run had
+ left a candidate in `src/`); `--bank` skips a body another rung already closed (it had re-applied a permuter body over rung
+ R's cleaner one-line version, restored through the oracle as `d1fix`); `--propagate` keys the class on the FIRST bank in a
+ body's chain (the later row's before-hash describes a text only that body ever had — func_80163EC8 found 0 siblings until
+ this was fixed, then 132); the permuter's winner is TIDIED before it is offered (pycparser's two-space indent and the corpse
+ `;` where a statement was inlined away), with the tidy judged like any other candidate.
+
+## 🛑 SESSION CHECKPOINT — S99 (2026-09-09): T0–T5 ☑ **T6 ☑** — all committed (this close on top of `1c2355054`); NEXT = **T7, the reshaping waves — and T7 STARTS ONLY ON DREW'S DIRECT APPROVAL IN THE SESSION THAT RUNS IT** (his words, twice: the `/effort ultracode` toggle is NOT approval); until he gives it, the drawable work is a PARALLELISED rung-R sweep (§2) | the number: **33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies · marked 33,427 · UNMARKED 0 · orphans 0 · GTE levers 462 — `lever_census --check` OK · `lever_progress --check` OK (3 milestones)
+
+### 0. How to use this block
+A fresh session reads CLAUDE.md's load order, replays this block verbatim, confirms the effort (T7's coordinator at Max, the waves at
+Ultracode — Drew toggles, R27) and executes §2. The tree is clean at HEAD = this close commit. **Never run a long job as a harness
+background task** (the low-memory guard kills them; the harness also backgrounds any foreground command over 120 s, which is fine for
+`make` but not for a campaign) — run campaigns DETACHED with `setsid nohup … &` and a `Monitor` on the log. **The calibration is keyed to
+HEAD: EVERY commit stales it** — `tools/delever_oracle.py --calibrate ov_SC04_011 ov_SC03_015 ov_SC03_014 main -j 16` (3 s) before any
+judging, and calibrate the FULL set: a single-alias calibration fails its own completeness check. A killed batch: `tools/delever.py
+--restore`.
+
+### 1. Where things stand
+- **Done: T0–T6.** T6 delivered both yield lines the plan asks for: `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of
+ 2,131 bodies behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in
+ 6.0 min`. Banked: rung R `r1` (134 bodies) · rung D `d1` (4 exemplars) · propagation `p1`/`p1b` (525 siblings, 0 refused).
+ **34,091 → 33,427 sites**; `check-all: 218 passed, 0 failed of 218` after every batch.
+- **THE INSTRUMENT LESSON THAT COST TWO CAMPAIGNS** (cookbook §454, accelerators P36 S99): the permuter's target had been ASSEMBLED
+ FROM A DISASSEMBLY LISTING and scored **28 for a byte-identical body**, so score 0 was unreachable and two campaigns reported "0 of
+ 16" about a healthy population. The target is now the tree's OWN body compiled by the build's tail (`--positive-control` reads base
+ score 0). **Run `tools/delever_permute.py --positive-control TU FN` before believing any future yield.**
+- **Instruments (SETUP + dictionary rows, kit corpus regenerated, `tool_census --check: OK`):** `tools/delever.py` (+ `--recipes`
+ rung R with R2/R3/R4/R5/R6/R7, `--propagate TU FN`, `--cap`, `--control`), `tools/delever_permute.py` (`--plan`, `--prepare`,
+ `--positive-control`, `--calibrate`, `--run`, `--bank`, `--status`, `--selftest`), `tools/lever_progress.py` (the series behind
+ `docs/levers.md`), `tools/delever_oracle.py`, `tools/lever_census.py`, `tools/gte_consolidate.py`, `tools/delever_cycle.sh`.
+- **Drew's directive, mid-session:** the lever count over time is a DELIVERABLE — `docs/levers.md` + `docs/lever-progress.tsv` +
+ `tools/lever_progress.py --snapshot ""` **after every task that changes the count** (the post-100% chart, the story, a wiki
+ page, and the kit's day-one rule). §5 of that document is the prevent-vs-defer argument, written from the generated numbers:
+ **38% of the class A/B population came off with no understanding at all**, so the rule is *ban the silence, not the lever*.
+- **Environment:** WSL2, `~/bfm-decomp`, `.venv`; R22 clean fleet ≈ 115 s; the census ≈ 35 s; `make kit-corpus` ≈ 25 s; 16 cores.
+
+### 2. What is drawable WITHOUT Drew's wave approval — the parallelised rung-R sweep
+Rung R is serial today: one body costs ~150 compiles / ~1.3 min at `--cap 200` when nothing hits, so the ~12,000-body residue is
+unaffordable as written. Make `recipes()` run TU-parallel the way `apply_batch` already does (a `ThreadPoolExecutor` over files, one
+worker owning a whole TU — two workers must never share a TU, and the oracle writes the candidate into the tree to compile it), then:
+1. `--cap 40` first over the whole residue (the targeted recipes R5/R6 and a few R7s) to measure the cheap yield with a denominator;
+2. `--cap 200` on what the cheap pass leaves, drawn by class size so a hit is worth 130 bodies;
+3. after each batch: R22 → `lever_census --sites --check` → `lever_progress --snapshot "