From aef1159488a6066eaed82b0d8af3ed2e9a0fff29 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 9 Sep 2026 13:02:51 -0600 Subject: [PATCH] =?UTF-8?q?phase-36:=20T6=20CLOSE=20=E2=80=94=20both=20yie?= =?UTF-8?q?ld=20lines=20measured=20(permuter=205=20of=2016=20exemplars=20/?= =?UTF-8?q?=20665=20of=202,131=20bodies;=20recipes=20134=20of=20134=20in?= =?UTF-8?q?=206.0=20min),=20664=20sites=20gone=20(34,091=20->=2033,427),?= =?UTF-8?q?=20R22=20218/218=20at=20every=20step;=20the=20S99=20checkpoint?= =?UTF-8?q?=20written=20for=20T7=20(which=20starts=20only=20on=20Drew's=20?= =?UTF-8?q?direct=20approval)=20with=20the=20parallelised=20rung-R=20sweep?= =?UTF-8?q?=20as=20the=20drawable=20work=20meanwhile?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../corpus/cookbook/cookbook-index.md | 7 +- .../corpus/cookbook/matching-cookbook.md | 67 +++ .../corpus/record/docs/accelerators.md | 30 + decomp-architect/corpus/tools/INDEX.md | 7 +- decomp-architect/corpus/tools/P10/delever.py | 567 +++++++++++++++++- .../corpus/tools/P10/delever_permute.py | 169 +++++- .../corpus/tools/P10/lever_progress.py | 192 ++++++ .../corpus/tools/P10/verbatim_target_s.py | 63 +- .../corpus/tools/P7/p16_permute.py | 18 +- .../corpus/tools/P7/permuter_ils.py | 4 +- decomp-architect/tools/MANIFEST.md | 7 +- docs/tool-index.md | 11 +- phase-ends/CURRENT_PHASE.md | 80 ++- tools/delever.py | 10 +- 14 files changed, 1186 insertions(+), 46 deletions(-) create mode 100644 decomp-architect/corpus/tools/P10/lever_progress.py diff --git a/decomp-architect/corpus/cookbook/cookbook-index.md b/decomp-architect/corpus/cookbook/cookbook-index.md index 3ee117dd2..3f058dc39 100644 --- a/decomp-architect/corpus/cookbook/cookbook-index.md +++ b/decomp-architect/corpus/cookbook/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 1170 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 1171 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -1461,7 +1461,7 @@ - **§479** — ★★★ — WHERE THE PERMUTER ACTUALLY PAYS: A MEASURED YIELD CURVE (P31 S77, 8 candidates) L35811 - **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) L37151 -### (unbucketed — title matched no symptom vocabulary) (337) +### (unbucketed — title matched no symptom vocabulary) (338) - **§3-How** — to use this L30 - **§1** — Idiom catalog (asm pattern → C that produces it) L39 @@ -1800,6 +1800,7 @@ - **§452** — ★★★ — NOT EVERY VERBATIM BODY IS UNDECOMPILED WORK, AND §448'S HEADLINE OVERSTATED IT (P31 S75; 10-function burst, 0 banks, and the negative result is the finding) L35104 - **§462** — FOUR LEVERS FROM `main:func_80024054` (91 ins, 74/53/32 → 4) L35644 - **§480** — 🔴 — A STATIC BLOCKER CLASS THAT THE REAL PIPELINE ALREADY REMOVES IS A PHANTOM L35890 +- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) L37604 ## All sections, in order @@ -2974,6 +2975,7 @@ - **§500** — ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MATCH / 9 NEAR / 2 FAIL, every closer, two NEW named gcc mechanisms, and the wave-process defects L36793 - **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) L37151 - **§453** — ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DEFECTS THAT SHAPED IT (P35 S94–S96) L37555 +- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) L37604 --- @@ -4156,3 +4158,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: ` | L36793 | §500 | ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MA | | L37151 | §501 | ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABI | | L37555 | §453 | ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DE | +| L37604 | §454 | Rung D and rung R: taking a lever off a body that still has to compile to the same bytes ( | diff --git a/decomp-architect/corpus/cookbook/matching-cookbook.md b/decomp-architect/corpus/cookbook/matching-cookbook.md index 5959bea69..3c38acaf6 100644 --- a/decomp-architect/corpus/cookbook/matching-cookbook.md +++ b/decomp-architect/corpus/cookbook/matching-cookbook.md @@ -37600,3 +37600,70 @@ green. R34 in one sentence: build the oracle that can disagree, then measure the the classifier became includes (ov_SC03_015's 219 single-site macros; the never-extended members of five under-listed groups). REAL 360,744 → 350,533 with EMPTY up the same: empty-bodied shared functions were folded into REAL under the macro form. Instruction-weighted metrics unchanged. A number generated from a form the tooling cannot see is a number, not a count. + +## §454 — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) + +The residue after the mechanical de-lever campaign (rungs A–C) is 12,970 bodies in 1,804 text classes, each with at least one site the +oracle called NEEDED. Two instruments work on it: **rung D**, decomp-permuter seeded with the LEVER-FREE body, and **rung R**, the +cookbook's byte-neutral shape recipes applied mechanically. What follows is what S99 measured, including the part where the measurement +was wrong. + +**A DISASSEMBLY LISTING IS NOT A TARGET.** Two campaigns returned "0 of 16 exemplars" with a straight face. Both were the harness. The +permuter's `target.o` had been assembled from `verbatim_target_s.py --gas`, a listing regenerated from the ROM image — and assembling a +disassembly is a second toolchain with its own answers: + +* objdump prints the PSEUDO-instruction `move rX,rY` for `addu rX,rY,$zero` (0x…21). GAS assembles `move` as `or` (0x…25). In one + 234-instruction function that is **24 wrong words**, and nothing said so. +* a listing's `%hi`/`%lo` pairs come back RESOLVED — `lui $a1,0x801f` / `addiu $a1,$a1,-11600`, no relocation — while every compiled + candidate carries `R_MIPS_HI16`/`LO16` against a symbol, and the masked scorer compares reloc operands. + +Result: the scorer reported **28 for a body that is byte-identical**, so score 0 was unreachable and every NO-MATCH was its own. + +The control that names this in one line: **seed the search with the body the tree already matches and read the base score.** It must be +0. (`delever_permute.py --positive-control`.) Run it before believing any campaign's yield — a 0-yield run cannot otherwise be told from +a broken scorer (R40). + +The fix is to stop assembling a listing: **the target is the tree's own body compiled by the build's own tail** into a one-function +object, so it carries the candidates' relocations by construction. The ROM listing stays the INDEPENDENT oracle that proves it — +`match_one` must call that body a MATCH before the search starts — which is what keeps the arrangement from being circular (R34/R56). +`verbatim_target_s.py --gas` now verifies itself too (assemble → disassemble → compare word by word → `.word`-patch or REFUSE); it is +the file pasted into decomp.me, and it was wrong for every function containing a `move`. + +**THE PROFILE FOLLOWS THE REGISTER, NOT THE SITE KIND.** A needed pin on a CALLEE-SAVED register ($16–$23) is an allocation-order +residual and wants the regalloc profile. A pin on a CALLER-SAVED one ($2/$3/$4–$7) is not. `func_80163EC8` pins `$2`, and its residual +is `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — the operand ORDER of one `&`. The regalloc +profile weights `perm_commutative` 2.0; the cse profile weights it 40.0. Steering by "it was a pin, so regalloc" spends the whole budget +20× away from the only lever that closes it. + +**WHAT RUNG D ACTUALLY FINDS, AND WHY RUNG R THEN GETS IT FOR FREE.** Every win is a small, nameable source shape: + +| function | start (mismatched ins) | time | the whole semantic change | +|---|---|---|---| +| `func_80135D20` | 12 | 24 s | `flag = 0;` → `do { flag = 0; } while (0);` (`perm_ins_block`, RC-5 scope) | +| `func_80163EC8` | 8 | 314 s | `uVar5 = *(s32 *)(p+0x44); … uVar5 & ~0x20` → the temp inlined at its use (`perm_expand_expr`, §501-R's S2 kill) | + +The permuter's winner is machine-reprinted by pycparser (extra parens, `;` statements, its own brace style), so banking it verbatim +trades a lever for a readability regression — which is exactly what this phase exists to prevent. Each win is therefore turned into a +mechanical recipe that produces the SAME bytes from a one-line diff in the real source. Rung R's generators, in trial order (targeted +first, blanket last), each one compile: + +* **R2** the formerly-pinned declarations permuted among their own lines · **R4** one of them moved through the whole declaration run + (§76/§501-R: the allocation order is the bank) · **R3** an initializer split off its declaration, the assignment placed after the + WHOLE run (C89 forbids a declaration after a statement — placing it after the last PINNED declaration compiles as K&R parameters and + fails in a way that reads like a compiler bug) +* **R5** the operand order of one commutative operator (the caller-saved lever above; needs no pinned declaration) +* **R6** a local assigned once and read once, inlined at its use with its dead declaration removed +* **R7** one statement wrapped in a block — `{ stmt; }` tried before `do { stmt; } while (0);`, because the readable spelling should win + when both hold + +**THE CLASS, NOT THE BODY.** A win on an exemplar is worth its whole text class (the head of the draw is 80 classes of 134 copies), but a +ledger REPLAY cannot deliver it: the ledger replays a SITE SET, and a reshaped body is not one. `delever.py --propagate TU FN` does it by +address remap — the two old bodies are identical modulo `func_`/`D_` tokens (that IS the class hash), so their tokens correspond one for +one and the map they define is applied to the new body, each sibling judged on its own objects. `--apply-body` records the before/after +body text in its ledger row because after the write it exists nowhere else. + +**THE SHAPE OF THE POPULATION (first 6 of 16 exemplars, all ov_SC04_011, 134-copy classes).** Two closed, four did not, and the split is +the starting distance, not the site count: the closers started 8 and 12 mismatched instructions from the target; the four that did not +started at 70, 99, 105 and 131 — every one a body where removing a hand-placed `instruction` lever changed the instruction COUNT and +shifted everything after it (`mine=103 ins, target=106`). They still improved a long way (131→35, 105→18, 99→37, 70→24), so they are +seeds for a longer run or for T7's agents, not walls. `--max-start N` triages by that number instead of spending a search on it. diff --git a/decomp-architect/corpus/record/docs/accelerators.md b/decomp-architect/corpus/record/docs/accelerators.md index f1ef7ae2f..317ec257d 100644 --- a/decomp-architect/corpus/record/docs/accelerators.md +++ b/decomp-architect/corpus/record/docs/accelerators.md @@ -916,3 +916,33 @@ each a different action. (3) **A known-true control per join:** the registry rep multi-batch tool's hardest bug (stale line numbers + a restore that wipes the previous batch) into a non-event; the driver commits between batches. (5) **The census's per-instance forms are the registry check's C2d for free** (one cached scan, 36 s) — derive, don't re-parse (R33). + +## P36 S99 (2026-09-09) — the search harness whose target was a disassembly, and the three cheap checks that would have caught it + +(1) **Before believing any search yield, seed the search with a body that already MATCHES and read the base score. It must be 0.** +Two rung-D campaigns returned "0 of 16 exemplars" and both were the instrument: the permuter's `target.o` had been assembled from a +regenerated disassembly listing, and assembling a disassembly is a second toolchain — objdump prints the pseudo-instruction `move` for +`addu rX,rY,$zero` and GAS assembles `move` as `or` (24 wrong words in one 234-instruction function), while a listing's `%hi`/`%lo` +pairs come back resolved, with no relocation, against candidates that all carry one. The scorer read 28 for a byte-identical body, so +score 0 was unreachable. The control is one command and ~30 seconds (`delever_permute.py --positive-control`); without it, "the residue +is hard" and "the scorer is broken" are the same observation. R40, and the first place this project has needed it against a SCORER +rather than a model. + +(2) **The target of a byte search should be produced by the same toolchain as the candidates, with an independent oracle to prove it +— not by a second toolchain that has to be argued with.** The fix was to compile the tree's own body into a one-function object (the +candidates' relocations by construction) and require `match_one` against the ROM listing to call it a MATCH before the search starts. +Faster to build than the listing-fixing it replaced, and it cannot drift. + +(3) **A generated artifact that nothing ever consumed is a claim.** `verbatim_target_s.py --gas` had been proven once, through +decomp.me, on a function with no `move` in it — so it was wrong for a large fraction of the fleet and nobody knew (R98/DK-81 in a second +place). It now assembles, disassembles and compares itself word by word against the image, `.word`-patches what does not reproduce, and +REFUSES what still disagrees. + +(4) **Steer a directed search by what the residual IS, not by what produced it.** The weight profile was chosen from the site kind +("a pin → regalloc"); a pin on a CALLER-saved register ($2/$3/$4–$7) is an operand-order residual, and the regalloc profile weights +`perm_commutative` 2.0 where the cse profile weights it 40.0. One reading of one diff moved 8 of 16 exemplars to the right profile. + +(5) **Turn each search win into a mechanical recipe the same day.** Rung D's two wins were a statement wrapped in a block and a +single-set temp inlined at its use — both one-line source changes that a recipe generator reproduces in ONE compile, and that keep the +source readable, where banking the permuter's own pycparser-reprinted body would trade a lever for a readability regression in a phase +whose whole purpose is readability. diff --git a/decomp-architect/corpus/tools/INDEX.md b/decomp-architect/corpus/tools/INDEX.md index 2cbbf3e5e..8498edf26 100644 --- a/decomp-architect/corpus/tools/INDEX.md +++ b/decomp-architect/corpus/tools/INDEX.md @@ -8,8 +8,8 @@ > its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task; > their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.* > -> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows -> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure). +> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows +> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure). ## P1 — extraction + manifest @@ -335,6 +335,7 @@ | `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout | | `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout | | `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule | +| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger | | `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas | | `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name | | `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing | @@ -343,7 +344,7 @@ | `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha | | `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator | | `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas | -| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | +| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | | `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout | ## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer) diff --git a/decomp-architect/corpus/tools/P10/delever.py b/decomp-architect/corpus/tools/P10/delever.py index 54eff6096..bb22d443c 100644 --- a/decomp-architect/corpus/tools/P10/delever.py +++ b/decomp-architect/corpus/tools/P10/delever.py @@ -1254,6 +1254,512 @@ def scrub(a): return 0 if not bad else 1 +# ---------------------------------------------------------------------------------------------------------------------- +# --recipes (rung R): the cookbook's byte-neutral shape recipes, tried mechanically on a RESIDUE body +# ---------------------------------------------------------------------------------------------------------------------- +CTRL_KW = re.compile(r"^\s*(?:if|for|while|do|switch|else|return|goto|case|default|break|continue)\b") + + +def pin_names(sites): + """the variable each pin site declares, in source order (the zero-register pins excluded: their declaration is deleted).""" + out = [] + for s in sites: + if s["kind"] != "pin" or s.get("zero"): + continue + mm = re.search(r"\b([A-Za-z_]\w*)\s*(?:\[[^\]]*\])?\s*(?:__asm__|__asm|asm)\s*\(", s.get("text", "")) + if mm: + out.append(mm.group(1)) + return out + + +def is_decl_line(masked_line): + """a whole-statement DECLARATION on one line: ` [\\[n\\]][ = init];`. The type and the name must be separated + (by space or `*`) — without that, `ret = f();` parses as the declaration `re t = …` and an initializer split lands its + assignment after the first statement, where C89 forbids the declarations that follow it.""" + s = masked_line.strip() + return bool(s.endswith(";") and not CTRL_KW.match(s) and "(" not in s.split("=")[0] + and re.match(r"^[A-Za-z_][\w \t]*[\s*]\s*\*?\s*[A-Za-z_]\w*\s*(?:\[[^\]]*\])*\s*(?:=|;)", s)) + + +def decl_run_end(text, d): + """the 0-based index of the LAST line of the declaration run that opens fn's body — a C89 declaration may not follow a + statement, so an initializer split must put its assignment after the WHOLE run, not after the last pinned declaration.""" + lines = text.split("\n") + last = d["line"] - 1 + for i in range(d["line"], d["end"] - 1): + s = sc.mask_text(lines[i]).strip() + if not s or s.startswith("/*") or s.startswith("//") or s in ("{", "}"): + continue + if is_decl_line(sc.mask_text(lines[i])): + last = i + continue + break + return last + + +def decl_lines(text, tu, fn, names): + """[(line index, text)] for the lines of fn's body that DECLARE one of `names` — one name per line, the line a whole + statement. The lever-free text is re-scanned for them (a stripped pin can delete its line, so the census's numbers have + moved); a name whose declaration is not found alone on one line makes the body ineligible (None).""" + recs = sc.scan_text(text, tu, shared_defs=None) + d = next((r for r in recs if r["form"] == "def" and r["name"] == fn), None) + if d is None: + return None + lines = text.split("\n") + found = {} + for i in range(d["line"], d["end"] - 1): # inside the body, never the header line + raw_line = lines[i] + s = sc.mask_text(raw_line).strip() + if not is_decl_line(s): + continue + hits = [n for n in names if re.search(r"(?!-/%": + continue + prev = expr[:i].rstrip() + if prev and (prev[-1].isalnum() or prev[-1] in "_)]"): + out.append(i) + return out + + +def commutative_swaps(text, tu, fn, d_): + """[(description, candidate text)] — each line of fn's body that carries exactly ONE top-level commutative operator, + with its two operands swapped. THE lever for a caller-saved ($2/$3) residual: S99 read `and v0,v1,v0` against the + target's `and v0,v0,v1` on func_80163EC8, which is the operand order of one `&` in the source and nothing else.""" + lines = text.split("\n") + out = [] + for i in range(d_["line"], d_["end"] - 1): + raw_line, s = lines[i], sc.mask_text(lines[i]) + body = s.strip() + if not body.endswith(";") or body.startswith("#"): + continue + # only an assignment's RHS or a `return` expression — an `if (…) stmt;` line would need the condition parsed out of + # the statement after it, and a wrong split is a candidate that cannot compile (wasted, and noisy in the ledger) + asg = re.search(r"(?+\-*/%&|^~])=(?!=)", s) + ret = re.match(r"^\s*return\b", s) + start = asg.end() if asg else (ret.end() if ret else -1) + if start < 0 or start >= len(s): + continue + expr_end = s.rstrip().rfind(";") + expr = s[start:expr_end] + ops = top_level_ops(expr) + if len(ops) != 1: + continue + o = start + ops[0] + left, right = raw_line[start:o], raw_line[o + 1:expr_end] + if not left.strip() or not right.strip(): + continue + cand = list(lines) + cand[i] = raw_line[:start] + " " + right.strip() + " " + raw_line[o] + " " + left.strip() + raw_line[expr_end:] + out.append((f"swap {raw_line[o]} @{i + 1}", "\n".join(cand))) + return out + + +IDENT = re.compile(r"(? 1: + continue + # ONE ASSIGNMENT AT A TIME, not one per variable. The single-set case is the easy half; the lever rung D actually + # found is narrower: `uVar5` is assigned in TWO branches of func_80163EC8, and the winning move inlined ONE of them. + # An assignment is inlinable when its value is read exactly once before the variable is written again — the classic + # def-with-one-use — so the assignment can go and the read can carry the expression. + order = sorted(places) + for ai, _, _ in asgs: + after = [p for p in order if p[0] > ai] + reads = [p for p in after if p not in [(x, y, z) for x, y, z in asgs] and not is_decl_line(masked[p[0]].strip())] + if not reads: + continue + ui, u0, u1 = reads[0] + nxt = [p for p in after if p[0] > ui] + if nxt and (nxt[0][0], nxt[0][1], nxt[0][2]) not in [(x, y, z) for x, y, z in asgs]: + continue # read again before it is rewritten: the assignment is not dead + mm = re.match(ASG % re.escape(v), masked[ai].strip()) + if not mm: + continue + expr = lines[ai].strip()[mm.start(1):mm.end(1)] + if not expr.strip(): + continue + cand = list(lines) + cand[ui] = lines[ui][:u0] + f"({expr})" + lines[ui][u1:] + cand[ai] = None # the assignment goes (with its declaration when they are one) + if decls and len(asgs) == 1: # the separate `T v;` is dead only when nothing else writes v + cand[decls[0][0]] = None + out.append((f"inline {v} @{ai + 1}", "\n".join(l for l in cand if l is not None))) + return out + + +def block_wraps(text, tu, fn, d_): + """[(description, candidate text)] — one statement wrapped in a block. §501-R's RC-5 scope lever: a block changes the + statement's basic-block structure and with it the allocno live range, which is how rung D closed func_80135D20 in 24 s + (`flag = 0;` -> `do { flag = 0; } while (0);` and nothing else). The plain block is tried FIRST because it is the + readable spelling; the do-while is gcc's stronger form and is only reached when the plain one does not hold.""" + lines = text.split("\n") + out = [] + for i in range(d_["line"], d_["end"] - 1): + s = sc.mask_text(lines[i]).strip() + if not s.endswith(";") or CTRL_KW.match(s) or is_decl_line(s) or s.startswith("#") or "{" in s or "}" in s: + continue + raw_line = lines[i] + indent = raw_line[:len(raw_line) - len(raw_line.lstrip())] + stmt = raw_line.strip() + for tag, spelling in (("block", f"{indent}{{ {stmt} }}"), ("do-while", f"{indent}do {{ {stmt} }} while (0);")): + cand = list(lines) + cand[i] = spelling + out.append((f"{tag} @{i + 1}", "\n".join(cand))) + return out + + +def recipe_candidates(text, tu, fn, names, limit=24, rng=None, cap=40, blocks=True): + """[(recipe, description, candidate text)] — the byte-neutral shape recipes of the cookbook, mechanically. + R2 (§76/§501-R, the allocation ORDER is the bank): the formerly-pinned declarations permuted among their own lines. + R4: one of them moved through the whole declaration run. R3 (§17a/§501-P): an initializer split off its declaration. + R5 (§137/§501-R, the caller-saved class): the operand order of one commutative operator — the ONLY recipe here that + needs no pinned declaration, so a residue of barriers and launders still has candidates.""" + recs_ = sc.scan_text(text, tu, shared_defs=None) + d_ = next((r for r in recs_ if r["form"] == "def" and r["name"] == fn), None) + if d_ is None: + return [] + dls = decl_lines(text, tu, fn, names) if names else None + lines = text.split("\n") + out = [] + rng = rng or random.Random(0) + idx = [i for i, _ in dls] if dls else [] + body = [t for _, t in dls] if dls else [] + perms = [] + if not dls: + perms = [] + elif len(body) <= 4: + import itertools + perms = [p for p in itertools.permutations(range(len(body)))][1:] # the identity is the current text + else: + seen = set() + while len(perms) < limit and len(seen) < limit * 4: + p = tuple(rng.sample(range(len(body)), len(body))) + seen.add(p) + if p != tuple(range(len(body))) and p not in perms: + perms.append(p) + for p in perms[:limit]: + ls_ = list(lines) + for slot, src in zip(idx, p): + ls_[slot] = body[src] + out.append(("R2", "decl-order " + ",".join(str(x) for x in p), "\n".join(ls_))) + # R4 (§76/§501-R again, one variable at a time): a formerly-pinned declaration moved to every other slot of the body's + # declaration run — the allocno creation order the pin used to override. R2 permutes the pinned declarations among + # THEIR OWN slots; this reaches the orders that involve the untouched declarations too, at one compile each. + last = decl_run_end(text, d_) + run_idx = [i for i in range(d_["line"], last + 1) if is_decl_line(sc.mask_text(lines[i]))] + if dls and len(run_idx) > 1: + run_txt = [lines[i] for i in run_idx] + for i, t in dls: + if i not in run_idx: + continue + src = run_idx.index(i) + name = next((n for n in names if re.search(r"(?{dst}", "\n".join(ls_))) + for k, (i, t) in enumerate(dls or []): + head, _, init = t.partition("=") + if not init.strip().endswith(";") or "==" in t: + continue + name = [n for n in names if re.search(r"(? {b[2]} {b[3]}") + sys.exit("delever --recipes: the control did not pass — no verdict from this run is usable (R39)") + print(f"delever --recipes: {len(todo)} RESIDUE bodies", flush=True) + rows, won, tried, compiles = [], 0, 0, 0 + t0 = time.time() + for n_body, ((tu, fn), r) in enumerate(todo, 1): + path = REPO / tu + raw = path.read_text(errors="surrogateescape") + names = pin_names(sites_by[(tu, fn)]) # may be empty: R5 needs no pinned declaration + try: + free = lever_free(tu, raw, fn) + except Refuse: + continue + cands = recipe_candidates(free, tu, fn, names, cap=a.cap) + if not cands: + continue + tried += 1 + hit = None + for rec, desc, cand in cands: + v, dt, err = judge(tu, cand) + compiles += len(recs_for(tu)) + if v == "IDENTICAL": + hit = (rec, desc, cand) + break + if n_body % 10 == 0 or hit: # R55: a lane that runs unattended leaves evidence + print(f" [{n_body}/{len(todo)}] {won} closed · {compiles} compiles · {(time.time() - t0) / 60:.1f} min", + flush=True) + row = dict(ts=time.strftime("%Y-%m-%d %H:%M:%S"), label=a.label, rung="R", calib=dict(head=oracle.head(), stamp=oracle.config_stamp()), + tu=tu, fn=fn, addr=fn_addr(fn, tu), aliases=r.get("aliases"), header=tu.endswith(".h"), + nhash_before=r.get("nhash_after"), nhash_after=None, candidates=len(cands), pins=len(names), + verdict=("LEVER-FREE" if hit else "RESIDUE"), recipe=(hit[0] if hit else None), how=(hit[1] if hit else None), + sites=([] if hit else r.get("sites", []))) + if hit: + path.write_text(hit[2], errors="surrogateescape") + # the body's own markers are now orphans (its levers are gone); every OTHER body's marker is still honest, so the + # scrub is scoped to this body's line span — a file-wide scrub would leave the census with UNMARKED sites elsewhere + recs_ = sc.scan_text(hit[2], tu, shared_defs=None) + d_ = next((x for x in recs_ if x["form"] == "def" and x["name"] == fn), None) + span = range(d_["line"], d_["end"] + 1) if d_ else range(0) + scrub = scrub_edits(hit[2], [i + 1 for i, l in enumerate(hit[2].split("\n")) if FAKE in l and i + 1 in span]) + if scrub: + cand2 = apply_edits(hit[2], scrub) + v2, _, _ = judge(tu, cand2) + if v2 == "IDENTICAL": + path.write_text(cand2, errors="surrogateescape") + walk = lc.walk_file(path.read_text(errors="surrogateescape"), tu, tu.endswith(".h")) + row["nhash_after"] = next((x["nhash"] for x in walk["defs"] if x["name"] == fn), None) + won += 1 + print(f" {tu}:{fn} — {hit[0]} {hit[1]} IDENTICAL ({len(names)} pin(s) gone)", flush=True) + rows.append(row) + ledger_append(rows) + needed = sum(len([s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]) for _, r in todo) + print(f"recipes: {won} of {tried} bodies closed lever-free ({needed} NEEDED sites in the {len(todo)} drawn), " + f"{compiles} compiles in {(time.time() - t0) / 60:.1f} min") + return 0 + + +def remap_body(ex_before, ex_after, sib_before): + """the exemplar's reshaped body, with its `func_/D_` addresses replaced by the sibling's — or (None, why). + + A text class is "identical modulo addresses" (that IS the nhash), so the two old bodies' address tokens correspond + one for one in order; the map they define is applied to the new body. This is what turns one crack into a whole + class: the 134-copy classes are the reason the draw is ordered by copies. A ledger REPLAY cannot do it — the ledger + replays a SITE SET, and a reshaped body is not one.""" + a, b = lc.NORM_SYM.findall(ex_before), lc.NORM_SYM.findall(sib_before) + if len(a) != len(b): + return None, f"{len(a)} address tokens in the exemplar, {len(b)} in the sibling" + m = {} + for x, y in zip(a, b): + if m.setdefault(x, y) != y: + return None, f"`{x}` maps to both `{m[x]}` and `{y}` — not one class" + return lc.NORM_SYM.sub(lambda mm: m.get(mm.group(0), mm.group(0)), ex_after), None + + +def propagate(a): + """--propagate TU FN: the body TU:FN was reshaped and banked; give every RESIDUE sibling of its class the same + shape, with its own addresses, and judge each on its own objects.""" + tu, fn = a.propagate + rows = load_ledger() + chain = [r for r in rows if r.get("tu") == tu and r.get("fn") == fn + and r.get("verdict") == "LEVER-FREE" and r.get("after_text")] + if not chain: + sys.exit(f"delever --propagate: no banked reshape of {tu}:{fn} in the ledger (its row must carry after_text)") + # THE CLASS is what the body looked like when the campaign found it, so the key and the "before" text come from the + # FIRST bank in this body's chain; the text to spread is the LAST one (a body reshaped, then tidied, has two rows, and + # taking the last row's before-hash would look for siblings of a text only this body ever had). + src_row = dict(chain[-1], nhash_before=chain[0]["nhash_before"], before_text=chain[0]["before_text"]) + key = src_row["nhash_before"] + cur = {} + for r in rows: + if r.get("tu") and r.get("fn"): + cur[(r["tu"], r["fn"])] = r + sibs = [k for k, r in cur.items() if k != (tu, fn) and r.get("verdict") == "RESIDUE" + and (r.get("nhash_after") or r.get("nhash_before")) == key] + if a.only: + sibs = [k for k in sibs if any(o in k for o in a.only)] + sibs = sibs[:a.limit] if a.limit else sibs + print(f"delever --propagate: {tu}:{fn} -> {len(sibs)} sibling(s) of class {key[:12]}", flush=True) + if not sibs: + return 1 # R68: an empty work list is a refusal, not a success + ok = bad = 0 + for stu, sfn in sibs: + path = REPO / stu + raw = path.read_text(errors="surrogateescape") + d = next((r for r in sc.scan_text(raw, stu, shared_defs=None) if r["form"] == "def" and r["name"] == sfn), None) + if d is None: + print(f" {stu}:{sfn}: not defined there — SKIPPED", flush=True) + bad += 1 + continue + ls = line_starts(raw) + sib_before = raw[ls[d["line"] - 1]:ls[d["end"]]] + if lc.norm_hash(sc.mask_text(sib_before)) != key: + print(f" {stu}:{sfn}: its text is not this class any more — SKIPPED", flush=True) + bad += 1 + continue + body, why = remap_body(src_row["before_text"], src_row["after_text"], sib_before) + if body is None: + print(f" {stu}:{sfn}: {why} — SKIPPED", flush=True) + bad += 1 + continue + f = RUN / "propagate" / f"{stu.replace('/', '_')}__{sfn}.c" + f.parent.mkdir(parents=True, exist_ok=True) + f.write_text(body, errors="surrogateescape") + r = subprocess.run([sys.executable, str(REPO / "tools/delever.py"), "--apply-body", stu, sfn, str(f), + "--label", a.label, "--rung", src_row.get("rung") or "R", "--dirty-ok"], + cwd=REPO, capture_output=True, text=True) + line = ((r.stdout or r.stderr).strip().splitlines() or [""])[-1] + print(f" {line[:200]}", flush=True) + ok += r.returncode == 0 + bad += r.returncode != 0 + print(f"delever --propagate: {ok} of {len(sibs)} sibling(s) banked, {bad} refused") + return 0 if ok else 1 + + def status(): rows = load_ledger() done, ex = ledger_index(rows) @@ -1318,7 +1824,10 @@ def apply_body(a): nhash_before=nh_before, nhash_after=nh_after, source=src, verdict=("LEVER-FREE" if v == "IDENTICAL" else f"BODY-{v}"), sites=[dict(ord=i, kind=s["kind"], cls=s["cls"], detail=s["detail"], via=s.get("via", ""), line=s["line"], verdict="NEEDED", why="left by the author", oracle="") for i, s in enumerate(levers)], - compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_]) + compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_], + # the body AS IT WAS: --propagate needs it to map this class's addresses onto a sibling's, and after the + # write it exists nowhere else (the tree has moved on and the ledger is the record) + before_text=before, after_text=new) if v == "IDENTICAL": path.write_text(cand, errors="surrogateescape") ledger_append([row]) @@ -1569,6 +2078,47 @@ def selftest(): plan_, total_, _ = make_plan(fake_bodies, rows, False, 10, None) if total_ != 1: fail("a copy with a judged text but no row of its own must still be drawn") + # rung R's candidate generators, on a fixture whose every answer is known by hand + RFIX = ("void rfix(int p)\n{\n int a = p;\n int b;\n int c;\n" + " if (a == b) a = b & 3;\n c = a + 1;\n *(int *)(p + 4) = c;\n}\n") + rd = dict(line=1, end=9) + for line, want in [("s32 d = param_1;", True), ("ret = f();", False), ("u8 *p;", True), ("d = param_1;", False), + ("return x;", False), ("extern s32 D_1[];", True)]: + if is_decl_line(line) != want: + fail(f"is_decl_line({line!r}) != {want}") + if top_level_ops("a & b") != [2] or top_level_ops("a && b") or top_level_ops("(s32 *)p") \ + or top_level_ops("*(s32 *)(p + 4) & 0xFF") != [16]: + fail("top_level_ops: a binary commutative operator at depth 0, and nothing else") + sw = commutative_swaps(RFIX, "src/x.c", "rfix", rd) + if [s for s, _ in sw] != ["swap & @6", "swap + @7"]: + fail(f"commutative_swaps found {[s for s, _ in sw]}") + if "a = 3 & b;" not in sw[0][1] or "c = 1 + a;" not in sw[1][1]: + fail("commutative_swaps must split at the assignment, never at `==`") + R6FIX = ("void r6(int p)\n{\n int v;\n int w;\n v = *(int *)(p + 4);\n" + " *(int *)(p + 4) = v & ~0x20;\n w = 3;\n *(int *)(p + 8) = w;\n}\n") + inl = inline_single_set_temps(R6FIX, "src/x.c", "r6", dict(line=1, end=9)) + if [d for d, _ in inl] != ["inline v @5", "inline w @7"]: + fail(f"inline_single_set_temps found {[d for d, _ in inl]}") + if "= (*(int *)(p + 4)) & ~0x20;" not in inl[0][1] or "int v;" in inl[0][1]: + fail("R6 must inline the expression at the use AND drop the now-dead declaration") + bw = block_wraps(R6FIX, "src/x.c", "r6", dict(line=1, end=9)) + if [d for d, _ in bw][:2] != ["block @5", "do-while @5"] or "{ v = *(int *)(p + 4); }" not in bw[0][1]: + fail(f"block_wraps: {[d for d, _ in bw][:3]} (the readable spelling first)") + cands = recipe_candidates(RFIX, "src/x.c", "rfix", ["a", "b"]) + kinds_ = {r for r, _, _ in cands} + if not {"R2", "R3", "R5"} <= kinds_ or any(c == RFIX for _, _, c in cands): + fail(f"recipe_candidates: {kinds_} (the seed must never be a candidate)") + if any("a = p;" in c.split("\n")[7] for _, _, c in cands if _ == "R3"): + fail("R3 must place its assignment after the whole declaration run (C89)") + # the address remap that propagates a reshape to a class (R48-adjacent: one crack, 134 banks) + exb = "void func_80100000(void) { D_80200000 = func_80100004(); }" + exa = "void func_80100000(void) { s32 t = func_80100004(); D_80200000 = t; }" + sib = "void func_80300000(void) { D_80400000 = func_80300004(); }" + got, why = remap_body(exb, exa, sib) + if got != "void func_80300000(void) { s32 t = func_80300004(); D_80400000 = t; }": + fail(f"remap_body produced {got!r} ({why})") + if remap_body(exb, exa, "void func_80300000(void) { D_80400000 = 0; }")[0] is not None: + fail("remap_body must refuse a sibling with a different token count") # the oracle's crash classification on its real message forms (R103) if not oracle.SIGNAL_LINE.search("bash: line 1: 3845091 Done mipsel-linux-gnu-cpp ...\n 3845092 Aborted (core dumped) | tools/bin/gcc-2.7.2-psx/cc1 -quiet\n"): fail("SIGNAL_LINE must match bash's job-status block") @@ -1792,6 +2342,13 @@ def main(): ap.add_argument("--restore", action="store_true", help="restore every in-flight file from inflight.json") ap.add_argument("--status", action="store_true") ap.add_argument("--scrub", action="store_true", help="remove orphan !FAKE markers (a marker whose site is gone), byte-judged per file") + ap.add_argument("--propagate", nargs=2, metavar=("TU", "FN"), + help="give every RESIDUE sibling of this banked body's class the same shape, with its own addresses") + ap.add_argument("--recipes", action="store_true", + help="rung R: the cookbook's byte-neutral shape recipes tried mechanically on every RESIDUE body") + ap.add_argument("--cap", type=int, default=60, help="--recipes: candidates tried per body (each is one compile)") + ap.add_argument("--control", type=int, default=8, help="--recipes: how many LEVER-FREE bodies the control run reproduces (R39)") + ap.add_argument("--limit", type=int, help="--recipes: stop after this many RESIDUE bodies") ap.add_argument("--apply-body", nargs=3, metavar=("TU", "FN", "FILE")) ap.add_argument("--rung", default="E") ap.add_argument("--allow-residue", action="store_true") @@ -1810,6 +2367,14 @@ def main(): sys.exit(status()) if a.scrub: sys.exit(scrub(a)) + if a.propagate: + if not a.label: + sys.exit("delever --propagate: --label is required (R48)") + sys.exit(propagate(a)) + if a.recipes: + if not a.label: + sys.exit("delever --recipes: --label is required (R48: the ledger rows are keyed by it)") + sys.exit(recipes(a)) if a.apply_body: if not a.label: sys.exit("delever --apply-body: --label is required") diff --git a/decomp-architect/corpus/tools/P10/delever_permute.py b/decomp-architect/corpus/tools/P10/delever_permute.py index 40495eec2..2e5a31141 100644 --- a/decomp-architect/corpus/tools/P10/delever_permute.py +++ b/decomp-architect/corpus/tools/P10/delever_permute.py @@ -23,10 +23,12 @@ THE PIPELINE PER EXEMPLAR (probe-proven at S98, the T6 log entry): the permuter compiles must hold ONE .text function, because the masked scorer compares whole .text. 3. cpp -P with the BUILD's own CPPFLAGS + -I (common.h, the prelude, engine_types.h and the GTE header expand here — p16_permute.make_base_c runs its own cpp WITHOUT includes and would lose them) -> draft.c. - 4. the target .s — tools/verbatim_target_s.py regenerates it from the EXTRACTED ROM IMAGE (R34: an independent - oracle; our own source is the thing under test), into the exemplar's scratch dir, where p16_permute.setup reads it. - 5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds - (pins -> regalloc, barriers/launders/keep-alives -> schedule, mixed -> regalloc). + 4. the target — the tree's OWN (levered) body compiled by the build's tail into a one-function object, so it + carries the candidates' relocations by construction; `tools/verbatim_target_s.py` regenerates the ROM listing beside it + and `match_one` must call that body a MATCH before the search starts (R34: the ROM image is the independent oracle that + PROVES the target; R56: check the baseline). S99 learned this the hard way — see THE INSTRUMENT below. + 5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds AND + the register each pin names (callee-saved -> regalloc, caller-saved -> cse, barriers/launders/keep-alives -> schedule). 6. a score-0 winner is a CANDIDATE, never a bank: --bank puts the function's definition back through `delever.py --apply-body` (which refuses a body that still carries a class A/B lever and judges it on the object's bytes through every recipe of the TU), then re-folds the cpp-expanded GTE asm with `gte_consolidate.py --apply --rejudge`. @@ -36,6 +38,15 @@ SCRATCH IS KEYED BY ALIAS+FN (R48): `.run/P36/permuter/__/` — a fun overlays overlap in RAM, so two different bodies can both be `func_8013B274`), and the permuter's own scratch/winner paths are keyed by the bare name. Every attempt is recorded in `.run/P36/permuter/outcomes.jsonl`, which is also the skip list. +THE INSTRUMENT, AND WHY IT IS CHECKED BEFORE IT IS BELIEVED (S99). The first two campaigns returned 0 of 16 with a straight +face. Both were the harness: the target had been ASSEMBLED FROM A DISASSEMBLY LISTING, which is a second toolchain with its own +answers — objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or` (24 wrong words in +one 234-instruction function), and a listing's %hi/%lo pairs come back resolved with no relocation while every candidate carries +one. The permuter scored 28 for a body that IS byte-identical, so score 0 was unreachable and every NO-MATCH was its own. The +control that names this in one line is `--positive-control`, and the base score of the tree's own body is now 0. R40: exonerate +the instrument before attributing a failure to its subject; the campaign that skips this control cannot tell a hard population +from a broken scorer. + NEVER RUN THE CAMPAIGN AS A HARNESS BACKGROUND TASK (the low-memory guard kills it): `setsid nohup … &` + a Monitor on the log. """ import argparse @@ -148,15 +159,32 @@ def exemplars(only=(), limit=None, include_done=False): needed = [s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"] out.append(dict(nhash=nh, tu=tu, fn=fn, alias=(r.get("aliases") or [None])[0], copies=len(members), needed=len(needed), kinds=sorted({s["kind"] for s in needed}), + regs=sorted({s.get("detail", "") for s in needed if s["kind"] == "pin"}), members=[dict(tu=t, fn=f) for (t, f), _ in members], row=r)) out.sort(key=lambda e: (-e["copies"], e["needed"], e["tu"], e["fn"])) out = [e for e in out if matches(e, only)] return out[:limit] if limit else out -def klass_for(kinds): - """the permuter_weights profile for a residue's NEEDED-site mix (the profile NAME is accepted verbatim by classify()).""" +CALLEE_SAVED = {f"${n}" for n in range(16, 24)} | {f"$s{n}" for n in range(8)} | {"$fp", "$30"} + + +def klass_for(kinds, regs=()): + """the permuter_weights profile for a residue, from the NEEDED sites' kinds AND the REGISTER each pin names (the profile + name is accepted verbatim by classify()). + + The register is the part S99 learned by reading a residual instead of assuming one. A pin on a CALLEE-SAVED register + ($16-$23) is an allocation-ORDER residual — the regalloc profile's declaration/statement reordering is its lever. A pin + on a CALLER-SAVED one ($2/$3/$4-$7/…) is not: the S99 control on func_80163EC8 (`register … __asm__("$2")`) left exactly + a v0/v1 swap — `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — which is the + operand ORDER of one `&`, and the regalloc profile weights `perm_commutative` 2.0 while the cse profile weights it 40.0. + Steering that search by "it was a pin, so regalloc" spends the whole budget 20x away from the one lever that closes it.""" ks = set(kinds) + rs = {r for r in regs if r} + if "pin" in ks and rs and rs & CALLEE_SAVED: + return "regalloc" + if "pin" in ks and rs and not (rs & CALLEE_SAVED): + return "cse" if ks and ks <= SCHED_KINDS: return "schedule" return "regalloc" @@ -169,12 +197,17 @@ _sites_cache = None def tu_sites(tu): + """the census's sites for one TU. Built ONCE, into a local, and published under the lock: the workers are threads, and + publishing the empty dict before filling it (2 s for 53,355 rows) let every other worker read "no site in this TU" and + report a whole batch UNSTRIPPABLE — a race that lies in the tool's own voice.""" global _sites_cache - if _sites_cache is None: - _sites_cache = collections.defaultdict(list) - for s in dl.load_sites(): - _sites_cache[s["tu"]].append(s) - return _sites_cache[tu] + with _LOCK: + if _sites_cache is None: + built = collections.defaultdict(list) + for s in dl.load_sites(): + built[s["tu"]].append(s) + _sites_cache = built + return _sites_cache.get(tu, []) def body_sites(tu, fn): @@ -318,7 +351,7 @@ def scratch_of(alias, fn): return RUN / f"{alias}__{fn}" -def prepare(ex, quiet=False): +def prepare(ex, quiet=False, require_sites=True): """(scratch dir, draft path, target path, klass) for one exemplar — the whole probe-2 pipeline.""" tu, fn = ex["tu"], ex["fn"] alias = ex["alias"] or alias_of(tu, ex.get("row")) @@ -327,7 +360,7 @@ def prepare(ex, quiet=False): ex["alias"] = alias raw = (REPO / tu).read_text(errors="surrogateescape") sites = body_sites(tu, fn) - if not sites: + if not sites and require_sites: raise Unstrippable([("", 0, f"the census has no site in {tu}:{fn} (stale? rerun lever_census --sites)")]) d = scratch_of(alias, fn) if d.exists(): @@ -357,10 +390,18 @@ def prepare(ex, quiet=False): if r.returncode or not tgt.exists(): raise Unstrippable([("", 0, ((r.stderr or r.stdout).strip().splitlines() or ["no listing"])[-1][:160])]) shutil.copy(tgt, d / sub / f"{fn}.s") # p16_permute.setup / match_one.py read /.s + # THE TARGET OBJECT: the tree's own (levered) body through the build's own tail, so it carries the candidates' + # relocations by construction. Its fidelity is not assumed — `closeness(levered)` must be MATCH against the ROM + # listing above before any search runs (R34: the independent oracle proves the target; R56: check the baseline). + r = subprocess.run(["tools/permuter/compile.sh", str(d / "levered.c"), "-o", str(d / "target.o")], + capture_output=True, text=True, cwd=REPO) + if r.returncode or not (d / "target.o").exists(): + raise Unstrippable([("", 0, ((r.stderr or r.stdout).strip().splitlines() or ["compile failed"])[-1][:160])]) if not quiet: n = len((d / "draft.c").read_text(errors="surrogateescape").splitlines()) - print(f" prepared {alias}__{fn}: draft {n} lines, target {(r.stdout or '').strip().split()[-3:]}", flush=True) - return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"]) + print(f" prepared {alias}__{fn}: draft {n} lines, target.o {(d / 'target.o').stat().st_size} bytes " + f"+ the ROM listing", flush=True) + return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"], ex.get("regs", ())) # ---------------------------------------------------------------------------------------------------------------------- @@ -382,6 +423,52 @@ def closeness(d, fn, c): return (int(mm.group(1)) if mm else None), first +def positive_control(a): + """Can rung D close a gap it is KNOWN to be able to close? Take a body the tree already matches, perturb it by ONE + reversible source change (the first commutative operand swap `delever` can generate), confirm the perturbed body no + longer matches, and give the permuter one cycle to find its way back to score 0. + + Without this, "0 of 16" says nothing about the residue: it could equally be a harness that cannot reach ANY target + (R40 — exonerate the instrument before attributing the failure to its subject). With it, a PASS means the search + space, the target, the scorer and the winner path all work, and a 0-yield campaign is a fact about the population.""" + tu, fn = a.positive_control + raw = (REPO / tu).read_text(errors="surrogateescape") + d_ = next((r for r in sc.scan_text(raw, tu, shared_defs=None) if r["form"] == "def" and r["name"] == fn), None) + if d_ is None: + sys.exit(f"delever_permute --positive-control: {fn} is not defined in {tu}") + swaps = dl.commutative_swaps(raw, tu, fn, d_) + if not swaps: + sys.exit(f"delever_permute --positive-control: {tu}:{fn} has no commutative operator to perturb — pick another body") + ex = dict(nhash="", tu=tu, fn=fn, alias=alias_of(tu), copies=1, needed=0, kinds=[], regs=[], members=[], row={}) + d, draft, tgt, _ = prepare(ex, quiet=False, require_sites=False) + base, base_line = closeness(d, fn, draft) + desc, perturbed = swaps[a.which] + (d / "perturbed_tu.c").write_text(perturbed, errors="surrogateescape") + pert = drop_file_scope_asm(cpp_expand(tu, isolate(tu, perturbed, fn)), tu) + (d / "draft.c").write_text(pert, errors="surrogateescape") # the permuter's seed IS the perturbed body + after, after_line = closeness(d, fn, d / "draft.c") + print(f"positive control {alias_of(tu)}__{fn}: the tree's own body {base_line[:40]} · perturbed by `{desc}` {after_line[:60]}", + flush=True) + if base != 0: + sys.exit("delever_permute --positive-control: the UNPERTURBED body does not match — calibrate first (R56)") + if after == 0: + sys.exit(f"delever_permute --positive-control: `{desc}` is byte-neutral here — nothing to find; try --which {a.which + 1}") + rel = d.relative_to(REPO).as_posix() + log = d / "positive.log" + cmd = [PY, "tools/permuter_ils.py", fn, "--draft", rel + "/draft.c", "--asm-subdir", rel + "/gas", + "--klass", "cse", "--cycles", str(a.cycles), "--secs", str(a.secs), "--j", str(a.j), + "--winners", rel, "--pd", rel + "/pd"] + with open(log, "w") as f: + f.write(" ".join(cmd) + "\n\n") + f.flush() + subprocess.run(cmd, cwd=REPO, stdout=f, stderr=subprocess.STDOUT, timeout=a.cycles * (a.secs + 45) + 300) + out = log.read_text(errors="replace") + won = (d / f"{fn}.c").exists() + print(f"positive control: {'PASS — the permuter recovered score 0' if won else 'FAIL — it did not, in ' + f'{a.cycles}x{a.secs}s'} ({[l for l in out.splitlines() if 'cycle' in l][-1:] or ['no cycle line']})") + return 0 if won else 1 + + def calibrate_one(ex, quiet=False): """the control: the body AS THE TREE HAS IT (levers and all) must be MATCH against the regenerated target. It proves the whole chain — the target from the ROM image, the isolation, the cpp expansion, the pinned triple — is measuring THIS @@ -405,7 +492,7 @@ BEST_RE = re.compile(r"best score = (\d+)") def run_one(ex, secs, cycles, j, max_start=None): t0 = time.time() row = dict(kind="attempt", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=ex["nhash"], tu=ex["tu"], fn=ex["fn"], - alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"], + alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"], regs=ex.get("regs", []), secs=secs, cycles=cycles, j=j) try: d, draft, tgt, klass = prepare(ex) @@ -437,7 +524,7 @@ def run_one(ex, secs, cycles, j, max_start=None): log = d / "ils.log" cmd = [PY, "tools/permuter_ils.py", ex["fn"], "--draft", (draft.relative_to(REPO)).as_posix(), "--asm-subdir", rel + "/gas", "--klass", klass, "--cycles", str(cycles), "--secs", str(secs), - "--j", str(j), "--winners", rel, "--pd", rel + "/pd"] + "--j", str(j), "--winners", rel, "--pd", rel + "/pd", "--target-o", rel + "/target.o"] row["klass"] = klass row["cmd"] = " ".join(cmd) rc = None @@ -485,10 +572,34 @@ def winner_body(winner_c, fn, tu): return None +def tidy_body(body): + """the permuter's winner, made to read like the tree it lands in — or None when there is nothing to tidy. + + pycparser reprints a body it has parsed: two-space indent where this tree uses four, and an empty statement `;` left + where a statement was inlined away. Neither generates code, and the object oracle judges the tidied text before it is + kept, so this can only ever improve the source. What it deliberately does NOT touch is the parenthesisation + (`*((s32 *) (p + 4))`) or the brace style — that is the formatting phase's job, with clang-format, over the whole tree + at once (`decomp-architect/templates/.clang-format`); a per-bank reformat would make this phase's diffs unreadable.""" + lines = [l for l in body.split("\n") if l.strip() != ";"] + indents = [len(l) - len(l.lstrip(" ")) for l in lines if l.strip() and l.startswith(" ")] + if indents and min(indents) == 2: # a two-space body: double every leading run + lines = [(" " * (2 * (len(l) - len(l.lstrip(" ")))) + l.lstrip(" ")) if l.startswith(" ") else l for l in lines] + out = "\n".join(lines) + return out if out != body else None + + def bank(a): outs = load_outcomes() banked = {(o["alias"], o["fn"], o["nhash"]) for o in outs if o.get("kind") == "bank" and o.get("applied")} + # a body ANOTHER rung already closed is not banked again (S99: --bank re-applied the permuter's reprinted body over the + # one-line version rung R had banked for the same function, and the tree lost a clean diff for a byte-identical one) + cur = {} + for r in dl.load_ledger(): + if r.get("tu") and r.get("fn"): + cur[(r["tu"], r["fn"])] = r + already = {(k[0], k[1]) for k, r in cur.items() if r.get("verdict") == "LEVER-FREE"} todo = [o for o in outs if o.get("kind") == "attempt" and o.get("verdict") == "MATCH" + and (o["tu"], o["fn"]) not in already and (o["alias"], o["fn"], o["nhash"]) not in banked and (not a.only or any(x in (o["fn"], o["tu"], o["alias"]) or o["nhash"].startswith(x) for x in a.only))] if not todo: @@ -503,11 +614,17 @@ def bank(a): bad += 1 continue bf = d / "body.c" - bf.write_text(body, errors="surrogateescape") - cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(), - "--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else []) - r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True) - print(" " + (r.stdout or r.stderr).strip().splitlines()[-1][:220], flush=True) + # the tidied body FIRST (the oracle judges it like any other candidate); the winner verbatim only if it is refused + forms = [(t_, "tidied") for t_ in [tidy_body(body)] if t_] + [(body, "verbatim")] + r = None + for text_, how in forms: + bf.write_text(text_, errors="surrogateescape") + cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(), + "--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else []) + r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True) + print(f" [{how}] " + (r.stdout or r.stderr).strip().splitlines()[-1][:210], flush=True) + if r.returncode == 0: + break row = dict(kind="bank", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=o["nhash"], tu=o["tu"], fn=o["fn"], alias=o["alias"], label=a.label, applied=(r.returncode == 0), apply_out=(r.stdout or r.stderr).strip()[-400:]) @@ -535,7 +652,8 @@ def cmd_plan(a): f"({sum(e['copies'] for e in ex):,} bodies behind them); NEEDED kinds {dict(kinds)}") for e in ex[:a.show]: print(f" {e['copies']:4d} copies · {e['needed']:2d} needed {','.join(e['kinds']):<28} " - f"{klass_for(e['kinds']):8} {e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}") + f"{klass_for(e['kinds'], e.get('regs', ())):8} {','.join(e.get('regs', [])) or '-':<12} " + f"{e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}") if len(ex) > a.show: print(f" … {len(ex) - a.show} more") return 0 @@ -667,6 +785,8 @@ def main(): g = ap.add_mutually_exclusive_group(required=True) g.add_argument("--plan", action="store_true") g.add_argument("--prepare", nargs=2, metavar=("TU", "FN")) + g.add_argument("--positive-control", nargs=2, metavar=("TU", "FN"), + help="perturb a MATCHING body by one commutative swap and require the permuter to find its way back") g.add_argument("--calibrate", action="store_true", help="the control: N exemplars' levered bodies must all be MATCH against their regenerated targets") g.add_argument("--run", action="store_true") @@ -682,6 +802,7 @@ def main(): ap.add_argument("-j", type=int, default=16) ap.add_argument("--label", default="d1") ap.add_argument("--dirty-ok", action="store_true") + ap.add_argument("--which", type=int, default=0, help="--positive-control: which commutative swap to perturb with") ap.add_argument("--max-start", type=int, help="skip (as FAR) an exemplar whose lever-free body is further than N instructions from the target") ap.add_argument("--include-done", action="store_true", help="draw classes that already have an outcome row") @@ -702,6 +823,8 @@ def main(): d, draft, tgt, klass = prepare(ex) print(f"delever_permute --prepare: {d.relative_to(REPO)} (draft.c, {tgt.name}, klass={klass})") return 0 + if a.positive_control: + return positive_control(a) if a.calibrate: return cmd_calibrate(a) if a.run: diff --git a/decomp-architect/corpus/tools/P10/lever_progress.py b/decomp-architect/corpus/tools/P10/lever_progress.py new file mode 100644 index 000000000..e73dba7c2 --- /dev/null +++ b/decomp-architect/corpus/tools/P10/lever_progress.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +"""lever_progress.py — the lever-removal series: what the compiler-forcing constructs cost, and how they came off. + + tools/lever_progress.py --snapshot "T6 r1" # append today's census totals as a milestone row (evidence, with HEAD) + tools/lever_progress.py --render # regenerate the generated block of docs/levers.md from the data + tools/lever_progress.py --table # print both tables + tools/lever_progress.py --check # is the last snapshot this tree's? (a stale series is a wrong chart) + +WHY THIS EXISTS. The phase that takes the levers out is the only place the project ever counts them, and the count is worth +more than the phase: it is the post-100% chart, the story's spine, and the evidence behind the day-one kit's advice on +whether to prevent levers from the first bank or clean them up at the end. Numbers are DERIVED, never typed (R75). + +TWO TABLES, TWO SOURCES: + * MILESTONES — one appended row per census snapshot (`--snapshot`), because a census is a moment: it is not recoverable + later. Each row carries the tree's HEAD and the census's own totals, so a reader can re-derive it from that commit. + * THE CAMPAIGN — fully derived from `.run/P36/delever/ledger.jsonl` on every run: per batch label, the date, the rungs + used, the bodies first judged, the sites REMOVED / REWRITTEN, and — scored as a TRANSITION against what that body's + previous row left — the sites a later rung closed and the bodies it made lever-free. Rebuilt from scratch each render, + so it cannot drift. +""" +import argparse +import collections +import json +import pathlib +import subprocess +import sys + +REPO = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO / "tools")) + +CENSUS = REPO / ".run" / "P36" / "census" / "lever_census.json" +LEDGER = REPO / ".run" / "P36" / "delever" / "ledger.jsonl" +SERIES = REPO / "docs" / "lever-progress.tsv" +DOC = REPO / "docs" / "levers.md" +BEGIN = "" +END = "" +COLS = ["date", "milestone", "head", "sites_AB", "pins", "asm", "bodies_AB", "distinct_AB", "marked", "unmarked", + "gte_levers", "per_tu_asm_macros", "class_C", "class_D", "class_E", "class_F", "class_G"] + + +def census(): + if not CENSUS.exists(): + sys.exit("lever_progress: no census — run tools/lever_census.py --sites -j 16") + return json.loads(CENSUS.read_text()) + + +def ledger(): + rows = [] + if LEDGER.exists(): + for l in LEDGER.read_text().splitlines(): + if l.strip(): + rows.append(json.loads(l)) + return rows + + +def snapshot_row(milestone): + d = census() + ab, cl = d["levers_AB"], d["classes"] + return dict(date=d.get("generated", ""), milestone=milestone, head=d.get("head", ""), + sites_AB=ab["sites"], pins=ab["pins"], asm=ab["asm"], bodies_AB=ab["bodies"], + distinct_AB=ab["distinct_bodies"], marked=ab["marked"], unmarked=ab["unmarked"], + gte_levers=d.get("gte_levers", {}).get("sites", 0), + per_tu_asm_macros=d.get("per_tu_asm_macro_definitions", {}).get("total", 0), + **{f"class_{k}": cl.get(k, {}).get("sites", 0) for k in "CDEFG"}) + + +def read_series(): + if not SERIES.exists(): + return [] + lines = [l for l in SERIES.read_text().splitlines() if l.strip()] + if not lines: + return [] + head = lines[0].split("\t") + return [dict(zip(head, l.split("\t"))) for l in lines[1:]] + + +def write_series(rows): + SERIES.write_text("\t".join(COLS) + "\n" + "".join("\t".join(str(r.get(c, "")) for c in COLS) + "\n" for r in rows)) + + +def campaign(): + """per batch label, derived from the ledger, as STATE TRANSITIONS — not as "the first row wins". + + A body is judged more than once on purpose: rung A/B settles it with some sites left NEEDED, and a later rung (R, D or + an agent) comes back and closes those. Counting only a body's first row credits the whole campaign to T4 and reports + "0 removed" for the rung that actually finished the job (rung R's first batch closed 134 bodies and read as nothing). + So each row is scored against what that body's previous row left: `removed`/`rewritten` are the first pass's, and + `closed_later` is the fall in the NEEDED count afterwards, with `made_free` counting the bodies that reached zero.""" + prev, per = {}, collections.OrderedDict() + for r in ledger(): + lab = r.get("label") or "-" + key = (r.get("tu"), r.get("fn")) + ts = (r.get("ts") or "")[:10] + e = per.setdefault(lab, dict(label=lab, date=ts, rungs=set(), bodies=0, removed=0, rewritten=0, + closed_later=0, made_free=0, refused=0)) + if ts and (not e["date"] or ts < e["date"]): + e["date"] = ts + if r.get("rung"): + e["rungs"].add(str(r["rung"])) + sites = r.get("sites", []) + now_needed = sum(1 for s in sites if s.get("verdict") == "NEEDED") + was = prev.get(key) + if was is None: + e["bodies"] += 1 + e["removed"] += sum(1 for s in sites if s.get("verdict") == "REMOVED") + e["rewritten"] += sum(1 for s in sites if s.get("verdict") == "REWRITTEN") + e["refused"] += sum(1 for s in sites if s.get("verdict") == "REFUSED") + else: + e["closed_later"] += max(0, was - now_needed) + if was > 0 and now_needed == 0: + e["made_free"] += 1 + prev[key] = now_needed + return list(per.values()) + + +def md_tables(): + rows = read_series() + out = [BEGIN, "", "### Milestones — the count, at each moment a census ran", "", + "| date | milestone | pins | asm | **class A+B sites** | bodies | distinct | GTE levers | per-TU asm macros | C | D | E | F | G | HEAD |", + "|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|"] + for r in rows: + out.append("| {date} | {milestone} | {pins} | {asm} | **{sites_AB}** | {bodies_AB} | {distinct_AB} | {gte_levers} | " + "{per_tu_asm_macros} | {class_C} | {class_D} | {class_E} | {class_F} | {class_G} | `{head}` |".format(**r)) + out += ["", "### The campaign — every batch, from the ledger (derived on every render)", "", + "| batch | date | rungs | bodies first judged | sites removed | rewritten | sites closed later | bodies made lever-free |", + "|---|---|---|---:|---:|---:|---:|---:|"] + tot = dict(bodies=0, removed=0, rewritten=0, closed_later=0, made_free=0) + for e in campaign(): + if not any(e[k] for k in tot): + continue # a bookkeeping batch (a scrub, a re-mark) moved no site + out.append(f"| `{e['label']}` | {e['date']} | {','.join(sorted(e['rungs'])) or '-'} | {e['bodies']:,} | " + f"{e['removed']:,} | {e['rewritten']:,} | {e['closed_later']:,} | {e['made_free']:,} |") + for k in tot: + tot[k] += e[k] + out.append(f"| **total** | | | **{tot['bodies']:,}** | **{tot['removed']:,}** | **{tot['rewritten']:,}** | " + f"**{tot['closed_later']:,}** | **{tot['made_free']:,}** |") + out += ["", f"*Generated by `tools/lever_progress.py --render` from `.run/P36/census/lever_census.json` " + f"and `.run/P36/delever/ledger.jsonl`; the series lives in `docs/lever-progress.tsv` (R75: published " + f"numbers are generated, never typed).*", "", END] + return "\n".join(out) + + +def render(): + if not DOC.exists(): + sys.exit(f"lever_progress: {DOC.relative_to(REPO)} does not exist yet") + t = DOC.read_text() + if BEGIN not in t or END not in t: + sys.exit(f"lever_progress: {DOC.relative_to(REPO)} has no generated block ({BEGIN} … {END})") + pre, rest = t.split(BEGIN, 1) + _, post = rest.split(END, 1) + DOC.write_text(pre + md_tables() + post) + print(f"lever_progress: rendered {len(read_series())} milestone row(s) + {len(campaign())} batch row(s) " + f"into {DOC.relative_to(REPO)}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--snapshot", metavar="MILESTONE", help="append today's census totals as a milestone row") + ap.add_argument("--render", action="store_true") + ap.add_argument("--table", action="store_true") + ap.add_argument("--check", action="store_true") + a = ap.parse_args() + if a.snapshot: + rows = read_series() + row = snapshot_row(a.snapshot) + rows = [r for r in rows if not (r.get("milestone") == row["milestone"] and r.get("head") == row["head"])] + rows.append(row) + write_series(rows) + print(f"lever_progress: {row['milestone']} — {row['sites_AB']:,} class A+B sites " + f"({row['pins']:,} pins + {row['asm']:,} asm) in {row['bodies_AB']:,} bodies, " + f"{row['unmarked']} unmarked, at {row['head']}") + if a.render or a.snapshot: + if DOC.exists(): + render() + if a.table: + print(md_tables()) + if a.check: + rows, d = read_series(), census() + if not rows: + sys.exit("lever_progress --check: the series is empty (run --snapshot)") + last = rows[-1] + if last.get("head") != d.get("head") or int(last.get("sites_AB", -1)) != d["levers_AB"]["sites"]: + sys.exit(f"lever_progress --check: the series' last row ({last.get('milestone')}, {last.get('sites_AB')} sites " + f"at {last.get('head')}) is not this tree ({d['levers_AB']['sites']} at {d.get('head')}) — " + f"run --snapshot after the census (R75)") + print(f"lever_progress --check: OK — {len(rows)} milestone(s), last {last['milestone']} " + f"({last['sites_AB']} class A+B sites)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/decomp-architect/corpus/tools/P10/verbatim_target_s.py b/decomp-architect/corpus/tools/P10/verbatim_target_s.py index adfc25a9c..1ae9536de 100644 --- a/decomp-architect/corpus/tools/P10/verbatim_target_s.py +++ b/decomp-architect/corpus/tools/P10/verbatim_target_s.py @@ -165,6 +165,43 @@ def to_gas(insns, vaddr, off, symtab): return out +def gas_roundtrip(rows, insns, fn): + """[(row index, ROM word)] for every emitted instruction whose RE-ASSEMBLY does not reproduce the ROM's word. + + WHY (S99, and it invalidated a whole campaign before it was found): objdump prints the PSEUDO-instruction + `move s2,a0` for `addu s2,a0,$zero` (0x00809021), and gas assembles `move` as `or` (0x00809025). Every `move` in + the listing therefore came back a different word, so a target object built from this form differed from the ROM + in every one of them — and the permuter, scoring against it, reported 28 for a body that IS byte-identical. A + listing nothing ever assembled and compared is a claim, not an oracle (R98/DK-81 in a second place). Words that + carry a RELOCATION (jal/j, HI16/LO16) are excluded: the linker fills those, and both sides mask them.""" + body = "\n".join([f".include \"macro.inc\"", ".set noat", ".set noreorder", ".section .text", f"glabel {fn}"] + rows) + with tempfile.NamedTemporaryFile("w", suffix=".s", delete=False) as fh: + fh.write(body + "\n") + spath = fh.name + opath = spath[:-2] + ".o" + try: + r = subprocess.run(["mipsel-linux-gnu-as", "-I", os.path.join(REPO, "include"), "-march=r3000", "-mtune=r3000", + "-no-pad-sections", "-O1", "-G0", spath, "-o", opath], capture_output=True, text=True) + if r.returncode: + return None, (r.stderr or "").strip().splitlines()[:3] + sys.path.insert(0, os.path.join(REPO, "tools")) + import masked_diff + got = masked_diff.insns_from_object(opath, None) + finally: + for p in (spath, opath): + if os.path.exists(p): + os.unlink(p) + if len(got) != len(insns): + return None, [f"re-assembly produced {len(got)} instructions, the image has {len(insns)}"] + bad = [] + for k, (g, (word, _text)) in enumerate(zip(got, insns)): + if g["reloc_kind"]: + continue + if g["word"] != word: + bad.append((k, word)) + return bad, None + + def emit(binary, fn, outdir, quiet=False, gas=False): fr = _fr() vaddr, nins = func_extent(binary, fn) @@ -195,12 +232,34 @@ def emit(binary, fn, outdir, quiet=False, gas=False): fh.write(f' * thing under test. {nins} instructions at 0x{vaddr:08X}. $-registers, .L labels for\n') fh.write(f' * in-function targets, symbol names for external jumps; the listing carries its delay-slot nops\n') fh.write(f' * (hence .set noreorder). Paste WHOLE into decomp.me — its PS1 prelude defines glabel. */\n\n') + rows = to_gas(insns, vaddr, off, _symtab(binary)) + # the listing must RE-ASSEMBLE to the image's own words, or it is not a target (S99). Each row that does + # not is replaced by its `.word`, with the mnemonic kept in the comment for the reader; then it is + # verified again, and a listing that still disagrees is REFUSED rather than emitted (R43). + patched = 0 + for _ in range(3): + bad, err = gas_roundtrip(rows, insns, fn) + if bad is None: + return None, f'{binary}:{fn}: the gas form does not assemble ({err})' + if not bad: + break + # `rows` interleaves `.L:` label lines with instructions; `bad` counts INSTRUCTIONS. Indexing rows + # by an instruction index rewrote a label into a comment and the next round refused to assemble. + ins_rows = [i for i, r in enumerate(rows) if r.startswith('/*')] + for k, word in bad: + head, _, text = rows[ins_rows[k]].partition('*/') + rows[ins_rows[k]] = f'{head}*/ .word 0x{word:08X} /* {text.strip()} */' + patched += len(bad) + else: + return None, (f'{binary}:{fn}: {len(bad)} instruction(s) still do not re-assemble to the image ' + f'(first at +{bad[0][0] * 4:#x}) — REFUSING to emit a target that is not the bytes') fh.write('.set noat\n.set noreorder\n\n.section .text\n\n') fh.write(f'glabel {fn}\n') - for ln in to_gas(insns, vaddr, off, _symtab(binary)): + for ln in rows: fh.write(ln + '\n') if not quiet: - print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form)') + print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form' + + (f', {patched} word-patched)' if patched else ')')) return path, None path = os.path.join(outdir, binary, f'{fn}.s') with open(path, 'w') as fh: diff --git a/decomp-architect/corpus/tools/P7/p16_permute.py b/decomp-architect/corpus/tools/P7/p16_permute.py index 2f66ab554..205576a0f 100644 --- a/decomp-architect/corpus/tools/P7/p16_permute.py +++ b/decomp-architect/corpus/tools/P7/p16_permute.py @@ -233,7 +233,7 @@ def klass_for_fn(fn): return "", "" -def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None): +def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None, target_o=None): # `outdir` keys the scratch dir by the CALLER's identity instead of the bare function name (R48). A name is not # unique across the fleet — the overlays overlap in RAM, so two different bodies are both `func_8013B274` — and two # concurrent runs on one name would share (and corrupt) this directory. Default: the historical path, unchanged. @@ -247,6 +247,22 @@ def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None): f"'not found in base.c' and no-op in 0s. Inspect {pd}/base.c " f"(prep order: comments -> cpp macros -> M2C_FIELD -> hide_asm -> typedefs).") open(f"{pd}/base.c", "w").write(base) + if target_o: + # THE TARGET AS AN OBJECT, not a listing (S99). Assembling a disassembly listing is a second toolchain with its + # own answers: objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or` + # (24 words wrong in one 234-instruction function), and a listing's %hi/%lo pairs come back RESOLVED, with no + # relocation, while every candidate carries one — and the masked scorer compares reloc operands. Both made the + # scorer report a nonzero floor for a body that IS byte-identical, so score 0 was unreachable and every verdict + # was the instrument's. A target object compiled from the tree's own body by THIS pipeline has the candidate's + # relocations by construction; the ROM listing stays the independent oracle that PROVES it (match_one MATCH + # before the search starts, R34/R56) — the proof is what keeps this from being circular. + shutil.copy(target_o, f"{pd}/target.o") + body, prof = permuter_weights.render_settings_toml(fn, klass=klass, where=where) + open(f"{pd}/settings.toml", "w").write(body) + csh = "compile_o0.sh" if asm_subdir.rstrip("/").endswith("_o0") else "compile.sh" + open(f"{pd}/compile.sh", "w").write(f'#!/bin/bash\nexec {REPO}/tools/permuter/{csh} "$@"\n') + os.chmod(f"{pd}/compile.sh", 0o755) + return pd s = os.path.join(REPO, asm_subdir, fn + ".s") tgt = f"{pd}/target.s" with open(tgt, "w") as f: diff --git a/decomp-architect/corpus/tools/P7/permuter_ils.py b/decomp-architect/corpus/tools/P7/permuter_ils.py index 106a01c23..fe05bed7c 100644 --- a/decomp-architect/corpus/tools/P7/permuter_ils.py +++ b/decomp-architect/corpus/tools/P7/permuter_ils.py @@ -41,12 +41,14 @@ def main(): ap.add_argument("--secs", type=int, default=180, help="per-cycle time box") ap.add_argument("--j", type=int, default=12) ap.add_argument("--winners", default=".run/permuter-winners") + ap.add_argument("--target-o", help="use this one-function object as the target instead of assembling " + "/.s (S99: a disassembly listing is a second toolchain)") ap.add_argument("--pd", help="scratch dir for this run (default .run/permuter/) — key it by alias+fn when " "several runs share a function NAME (R48)") a = ap.parse_args() draft = open(a.draft).read() - pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd) + pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd, target_o=a.target_o) if not pd: print("ILS setup FAILED (target .s didn't assemble?)"); sys.exit(1) print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}") diff --git a/decomp-architect/tools/MANIFEST.md b/decomp-architect/tools/MANIFEST.md index 1d3549ec0..a7c12c1f0 100644 --- a/decomp-architect/tools/MANIFEST.md +++ b/decomp-architect/tools/MANIFEST.md @@ -8,8 +8,8 @@ > its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task; > their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.* > -> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows -> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure). +> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows +> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure). ## P1 — extraction + manifest @@ -335,6 +335,7 @@ | `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout | | `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout | | `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule | +| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger | | `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas | | `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name | | `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing | @@ -343,7 +344,7 @@ | `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha | | `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator | | `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas | -| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | +| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | | `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout | ## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer) diff --git a/docs/tool-index.md b/docs/tool-index.md index a8626b363..3085403d8 100644 --- a/docs/tool-index.md +++ b/docs/tool-index.md @@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool, then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.* -**Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 30 (a SETUP row only), ORPHAN 30 (neither) — of 301. Portability: PORTABLE 23, ADAPT 261, PROJECT-ONLY 17. +**Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 31 (a SETUP row only), ORPHAN 30 (neither) — of 302. Portability: PORTABLE 23, ADAPT 262, PROJECT-ONLY 17. ## P1 — extraction + manifest @@ -71,7 +71,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | When you need to… | Tool | What it does | Proven by | Adapt | Class | |---|---|---|---|---|---| -| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+6) | MIPS relocation encodings | LIVE | +| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+7) | MIPS relocation encodings | LIVE | | compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+23) | compiler triple, repo build flags | LIVE | | dump every compiler pass file for a self-contained draft | `cc1_dumps.sh` | Dumps every compiler pass file for a self-contained draft into a private directory | cc1_dumps_tu.sh, ghost_census.py | compiler triple, repo scratch paths | LIVE | | dump every compiler pass file for the spliced real translation unit | `cc1_dumps_tu.sh` | Same pass dumps for the spliced real translation unit, the faithful compile | alloc_table.py | absolute repo path, compiler triple | LIVE | @@ -295,7 +295,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | build a progress timeline from the repository's own committed digests | `timeline.py` | Builds a progress timeline from the repository's own committed digests, with a self-check | Makefile | repo doc paths | LIVE | | check that every relative link in the public docs resolves | `doc_links.py` | Checks that every relative link in the public docs resolves and the link policy holds | Makefile, wiki_render.py, wiki_sync.sh | repo doc paths | LIVE | | compile every eligible translation unit with the pinned toolchain, using no derived bytes | `compile_only.py` | Compiles every eligible translation unit with the pinned toolchain, without any game bytes | .github/workflows/no-rom.yml, delever_oracle.py, lever_census.py, macro_to_header.py (+1) | compiler triple, repo makefile parsing | LIVE | -| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+33) | repo src/config layout | LIVE | +| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+34) | repo src/config layout | LIVE | | convert progress metrics into an external progress-report schema | `objdiff_report.py` | Converts the progress JSON into the external progress-report schema | .github/workflows/progress.yml | external report schema | LIVE | | emit the public ordinal-to-hash map and the private old-to-new map | `public_rewrite/build_commit_map.py` | Emits the public ordinal-to-new-hash map and the private old-to-new map | probe_github.sh, resolve_tokens.py | repo doc/scratch paths | LIVE | | gate a first push: prove no derived bytes among tracked files | `audit_public.py` | The first-push gate: no derived bytes among tracked files, four independent checks | .github/workflows/no-rom.yml, gate_scan.py | repo purge-set and path lists (the kit's template reads `config/firewall.txt` instead) | LIVE | @@ -322,7 +322,7 @@ then what proved it. The same data generates the day-one kit's manifest and its |---|---|---|---|---|---| | add per-site function-pointer casts so a draft can call a differently typed callee | `cast_call_sites.py` | Adds per-site function-pointer casts so a draft can call a differently typed callee | canon_sig_reconcile.py, cast_self_callers.py, family_sweep.py, gate_stage.py (+4) | repo src layout | LIVE | | apply curated names and signatures inside the analysis tool and save | `ghidra_scripts/ApplySymbols.java` | The in-tool half of that mirror: apply curated names and signatures, save on exit | ghidra_apply_symbols.sh, ghidra_export_annotations.sh, ghidra_rebuild.sh, ExportAnnotations.java (+1) | none | LIVE | -| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+2) | repo paths, the shared-body scanner | LIVE | +| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+3) | repo paths, the shared-body scanner | LIVE | | consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked _m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever.py, delever_cycle.sh, delever_permute.py, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE | | convert a shared-body macro header into per-function plain-C headers included at each site | `macro_to_header.py` | The Phase-35 converter: every DEFINE_ macro body becomes a plain-C header under src/shared// included at its site, the legacy name-parameterized headers converted, the registry text-edited, the macro header deleted; --plan / --apply / --finalize / --verify | share_body.py, tool_census.py | the macro form is this project's; a kit-born project shares headers from its first bank | LIVE | | emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE | @@ -331,6 +331,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED | | guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE | | judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE | +| keep the lever count as a series a chart and a story can be drawn from | `lever_progress.py` | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | — | the census json, the delever ledger | REFERENCED | | measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+10) | repo paths, the registry and signature schemas | LIVE | | mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED | | refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE | @@ -339,7 +340,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | `share_body_cycle.sh` | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | — | repo paths, the phase-log format, the fleet count from config/check.*.sha | ORPHAN | | search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED | | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE | -| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE | +| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE | | turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED | ## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 3ddc5c739..9b4ac2999 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -62,7 +62,7 @@ - Rules check (P6) after T4. - ☐ **T5** — GTE consolidation (one header, PsyQ's names; the `"memory"`-clobber variants are levers → residue, not a second spelling) + the dead lever-macro sweep (`SHB` cross-file use counted). Verify: `lever_census` 0 per-TU asm macro definitions; R22 green. -- ☐ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed, +- ☑ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed, 16 workers; a score-0 applied through `delever --apply-body` and gated). Verify: `recipes: of NEEDED sites removed`; `permuter: of exemplars matched lever-free in h`. - ☐ **T7** (Ultracode, prompted) — The reshaping waves to zero: wave → harvest → toolify (cookbook + rung R) → sweep → next draw; wave 1 @@ -621,7 +621,83 @@ accumulate here as the phase produces them.** refused more than a third of the population while the context was hot, and harvest→toolify applied to lever classes so a shape sweeps the banked population instead of the lever propagating with every family remap (1,758 distinct bodies became 12,578). -## 🛑 SESSION CHECKPOINT — S98 (2026-09-09): T0 ☑ T1 ☑ T1b ☑ T2 ☑ T3 ☑ T4 ☑ T5 ☑ — all committed (this close on top of `4bc987426`); NEXT = T6 BUILD: `tools/delever_permute.py` per §2 (the design is settled and its pipeline proven by two probes — the T6 log entry); S98 ended here at 90 % context | the number: 34,091 sites in 12,712 bodies (1,759 distinct) · marked 34,091 · UNMARKED 0 · orphans 0 · GTE levers 462 · per-TU asm macro definitions 314 (0 canonical duplicates) — `lever_census --check` OK +- **S99 — T6 CLOSE: both yield lines measured, 664 lever sites gone, R22 green at every step. T6 ☑.** + **The two lines the plan asks for (R41):** `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of 2,131 bodies + behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in 6.0 min`. + **Banked this session:** rung R's batch `r1` (134 bodies, every one `R7 do-while @`, a ONE-LINE source change each) · + rung D's four remaining exemplars (`d1`, tidied and judged) · the propagation `p1`/`p1b` (132 + 130 + 131 + 132 = **525 + siblings**, 0 refused, each judged on its own objects). **34,091 → 33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies, + 33,427 marked, 0 UNMARKED, 0 orphans; `check-all: 218 passed, 0 failed of 218` after every batch. + **What T6 leaves for T7, stated rather than smoothed over:** rung R's recipe set does NOT reproduce the shape rung D found on + func_80163EC8 — R6 was generalised from "assigned once, read once" to "dead after one read" (the lever was one of two + assignments to `uVar5`) and still does not close it (154 candidates, 1.3 min, 0 hits), so that class was banked from the + permuter's own winner instead. The 11 exemplars rung D did not close are seeds, not walls: every one improved (78→9, 52→3, + 50→3, 37→2, 131→35) and the ones that stayed far all removed a hand-placed `instruction` whose C spelling changed the + instruction COUNT. A fleet-wide rung-R sweep is priced but not run: one body costs ~150 compiles / ~1.3 min at `--cap 200` + when nothing hits, so the 12,000-body residue needs the run parallelised across TUs before it is affordable. + **Instrument work banked with it:** `--recipes` is killable (its judge snapshots to inflight.json first — a killed run had + left a candidate in `src/`); `--bank` skips a body another rung already closed (it had re-applied a permuter body over rung + R's cleaner one-line version, restored through the oracle as `d1fix`); `--propagate` keys the class on the FIRST bank in a + body's chain (the later row's before-hash describes a text only that body ever had — func_80163EC8 found 0 siblings until + this was fixed, then 132); the permuter's winner is TIDIED before it is offered (pycparser's two-space indent and the corpse + `;` where a statement was inlined away), with the tidy judged like any other candidate. + +## 🛑 SESSION CHECKPOINT — S99 (2026-09-09): T0–T5 ☑ **T6 ☑** — all committed (this close on top of `1c2355054`); NEXT = **T7, the reshaping waves — and T7 STARTS ONLY ON DREW'S DIRECT APPROVAL IN THE SESSION THAT RUNS IT** (his words, twice: the `/effort ultracode` toggle is NOT approval); until he gives it, the drawable work is a PARALLELISED rung-R sweep (§2) | the number: **33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies · marked 33,427 · UNMARKED 0 · orphans 0 · GTE levers 462 — `lever_census --check` OK · `lever_progress --check` OK (3 milestones) + +### 0. How to use this block +A fresh session reads CLAUDE.md's load order, replays this block verbatim, confirms the effort (T7's coordinator at Max, the waves at +Ultracode — Drew toggles, R27) and executes §2. The tree is clean at HEAD = this close commit. **Never run a long job as a harness +background task** (the low-memory guard kills them; the harness also backgrounds any foreground command over 120 s, which is fine for +`make` but not for a campaign) — run campaigns DETACHED with `setsid nohup … &` and a `Monitor` on the log. **The calibration is keyed to +HEAD: EVERY commit stales it** — `tools/delever_oracle.py --calibrate ov_SC04_011 ov_SC03_015 ov_SC03_014 main -j 16` (3 s) before any +judging, and calibrate the FULL set: a single-alias calibration fails its own completeness check. A killed batch: `tools/delever.py +--restore`. + +### 1. Where things stand +- **Done: T0–T6.** T6 delivered both yield lines the plan asks for: `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of + 2,131 bodies behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in + 6.0 min`. Banked: rung R `r1` (134 bodies) · rung D `d1` (4 exemplars) · propagation `p1`/`p1b` (525 siblings, 0 refused). + **34,091 → 33,427 sites**; `check-all: 218 passed, 0 failed of 218` after every batch. +- **THE INSTRUMENT LESSON THAT COST TWO CAMPAIGNS** (cookbook §454, accelerators P36 S99): the permuter's target had been ASSEMBLED + FROM A DISASSEMBLY LISTING and scored **28 for a byte-identical body**, so score 0 was unreachable and two campaigns reported "0 of + 16" about a healthy population. The target is now the tree's OWN body compiled by the build's tail (`--positive-control` reads base + score 0). **Run `tools/delever_permute.py --positive-control TU FN` before believing any future yield.** +- **Instruments (SETUP + dictionary rows, kit corpus regenerated, `tool_census --check: OK`):** `tools/delever.py` (+ `--recipes` + rung R with R2/R3/R4/R5/R6/R7, `--propagate TU FN`, `--cap`, `--control`), `tools/delever_permute.py` (`--plan`, `--prepare`, + `--positive-control`, `--calibrate`, `--run`, `--bank`, `--status`, `--selftest`), `tools/lever_progress.py` (the series behind + `docs/levers.md`), `tools/delever_oracle.py`, `tools/lever_census.py`, `tools/gte_consolidate.py`, `tools/delever_cycle.sh`. +- **Drew's directive, mid-session:** the lever count over time is a DELIVERABLE — `docs/levers.md` + `docs/lever-progress.tsv` + + `tools/lever_progress.py --snapshot ""` **after every task that changes the count** (the post-100% chart, the story, a wiki + page, and the kit's day-one rule). §5 of that document is the prevent-vs-defer argument, written from the generated numbers: + **38% of the class A/B population came off with no understanding at all**, so the rule is *ban the silence, not the lever*. +- **Environment:** WSL2, `~/bfm-decomp`, `.venv`; R22 clean fleet ≈ 115 s; the census ≈ 35 s; `make kit-corpus` ≈ 25 s; 16 cores. + +### 2. What is drawable WITHOUT Drew's wave approval — the parallelised rung-R sweep +Rung R is serial today: one body costs ~150 compiles / ~1.3 min at `--cap 200` when nothing hits, so the ~12,000-body residue is +unaffordable as written. Make `recipes()` run TU-parallel the way `apply_batch` already does (a `ThreadPoolExecutor` over files, one +worker owning a whole TU — two workers must never share a TU, and the oracle writes the candidate into the tree to compile it), then: +1. `--cap 40` first over the whole residue (the targeted recipes R5/R6 and a few R7s) to measure the cheap yield with a denominator; +2. `--cap 200` on what the cheap pass leaves, drawn by class size so a hit is worth 130 bodies; +3. after each batch: R22 → `lever_census --sites --check` → `lever_progress --snapshot "