From 22a31bc02561ae2ec882d2ddaa1f3daaa9096abf Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 9 Sep 2026 10:23:35 -0600 Subject: [PATCH] =?UTF-8?q?phase-36:=20T6=20=E2=80=94=20the=20kit=20corpus?= =?UTF-8?q?=20and=20the=20tool=20index=20regenerated=20for=20delever=5Fper?= =?UTF-8?q?mute.py=20(tool=5Fcensus=20--check:=200=20gaps)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- decomp-architect/corpus/tools/INDEX.md | 5 +- decomp-architect/corpus/tools/P10/delever.py | 83 ++ .../corpus/tools/P10/delever_permute.py | 714 ++++++++++++++++++ .../corpus/tools/P10/gte_consolidate.py | 48 +- .../corpus/tools/P7/p16_permute.py | 7 +- .../corpus/tools/P7/permuter_ils.py | 4 +- decomp-architect/tools/MANIFEST.md | 5 +- docs/tool-index.md | 23 +- 8 files changed, 853 insertions(+), 36 deletions(-) create mode 100644 decomp-architect/corpus/tools/P10/delever_permute.py diff --git a/decomp-architect/corpus/tools/INDEX.md b/decomp-architect/corpus/tools/INDEX.md index 62f9abc49..2cbbf3e5e 100644 --- a/decomp-architect/corpus/tools/INDEX.md +++ b/decomp-architect/corpus/tools/INDEX.md @@ -8,8 +8,8 @@ > its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task; > their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.* > -> **Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded), of which 300 live rows -> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 20 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 330 (the installer checks its copy against this figure). +> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows +> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure). ## P1 — extraction + manifest @@ -341,6 +341,7 @@ | `verbatim_target_s.py` | regenerate a splitter-format target disassembly for a function no longer stubbed | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | repo build/asm layout | | `delever_cycle.sh` | run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | the batch size, the aliases, the fleet count | | `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha | +| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator | | `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas | | `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | | `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout | diff --git a/decomp-architect/corpus/tools/P10/delever.py b/decomp-architect/corpus/tools/P10/delever.py index e501a4c0e..54eff6096 100644 --- a/decomp-architect/corpus/tools/P10/delever.py +++ b/decomp-architect/corpus/tools/P10/delever.py @@ -75,6 +75,7 @@ INFLIGHT = RUN / "inflight.json" PROBE = REPO / ".run" / "P36" / "probe" PRELUDE = "src/shared/engine_prelude.h" REMOVABLE = {("A", "pin"), ("B", "barrier"), ("B", "launder"), ("B", "keepalive"), ("B", "instruction"), + ("B", "gte-lever"), ("C", "cast"), ("C", "decl-body"), ("C", "param"), ("D", "register")} FILE_SCOPE_REMOVABLE = {("C", "decl-file"), ("B", "barrier"), ("B", "launder"), ("B", "keepalive"), ("B", "instruction")} # a file-scope asm statement is a TU-level site: a barrier/launder/keep-alive is judged like any other; a `.section` block is a rodata @@ -343,6 +344,52 @@ def macro_shape(raw, rel, name, use_line): return res +_GTE_CANON = None + + +_gte_variant_cache = {} + + +def gte_variant_target(raw, rel, name, use_line): + """The canonical macro name a lever-variant macro use should point at, from the SIGNATURE of the variant's own + governing `#define` (never from its spelling: T5 named `gte_rt_m` after Sony's `gte_rt` while that signature's + canonical name is `gte_rt_alt`). None when there is no single canonical macro for it.""" + key = (rel, name, use_line) + if key in _gte_variant_cache: + return _gte_variant_cache[key] + import gte_consolidate as gc # lazy: gte_consolidate imports THIS module + gte_canonical_clob("") # loads the table + body = None + for (l0, l1, n, b) in lc.define_blocks(raw): + if n == name and l1 < use_line: + body = b + inner = lc._macro_asm_inner(body) if body is not None else None + names = None + if inner is not None: + names, _ = gc.canonical_match(gc.signature(inner), _GTE_CANON) + res = names[0] if names and len(names) == 1 else None + _gte_variant_cache[key] = res + return res + + +def gte_canonical_clob(inner): + """The canonical clobber list for a GTE asm statement's inner text, or None when it has none to take + (no table, unsigned template, no canonical entry, or it already carries the canonical set).""" + global _GTE_CANON + if _GTE_CANON is None: + import gte_consolidate as gc # lazy: gte_consolidate imports THIS module + _GTE_CANON = json.loads(gc.CANON.read_text()) if gc.CANON.exists() else {} + canon = _GTE_CANON.get("canonical") or {} + if not canon: + return None + import gte_consolidate as gc + sg = gc.signature(inner) + names, clob = gc.canonical_match(sg, _GTE_CANON) + if names is None or tuple(sg["clob"]) == tuple(clob): + return None + return clob + + def site_edits(raw, m, ls, site, keep_register=False): """[(start, end, replacement)] for one site, or raise Refuse. `m` = same_len_mask(raw).""" pos = ls[site["line"] - 1] + site["col"] - 1 @@ -381,6 +428,42 @@ def site_edits(raw, m, ls, site, keep_register=False): return [(pos, consume_marker(raw, m, e), new)] if cls == "B" and kind in DEFERRED_KINDS: raise Refuse("asm-body: a whole routine in a C shell is T7's work (DEFERRED)") + if cls == "B" and kind == lc.GTE_LEVER_KIND: + # A GTE op whose clobber list exceeds its canonical signature's is a SCHEDULING STEER wearing Sony's + # coprocessor idiom (T5 named them and marked them; it judged only the macro DEFINITIONS, so a DIRECT + # statement's extra clobbers were never offered to the ladder). The rewrite is the canonical clobber + # set for that signature — not deletion: the op itself is real code. Refused when the tree has no + # canonical table, when the statement does not sign, when its signature is not canonical, or when it + # already carries the canonical set (then it is not a lever and the census is wrong about it, R43). + # The survivor keeps T5's own richer `// !FAKE:` text: gte-lever is deliberately NOT in MARK_KINDS. + name = site.get("via") + if name: + # a use of a LEVER VARIANT macro (`gte_x_m` / `gte_x_v`, kept per TU by T5): the lever is the variant's + # extra clobbers, so the rewrite points the use at the canonical macro of include/gte_inline.h. The variant's + # own `#define` is left dead for `gte_consolidate.py --sweep` (its marker for `--scrub`). + if not m.startswith(name, pos): + raise Refuse(f"token mismatch at {site['tu']}:{site['line']}: expected `{name}`") + base = gte_variant_target(raw, site["tu"], name, site["line"]) + if base is None: + raise Refuse(f"GTE lever `{name}`: no single canonical macro for its signature") + if base == name: + raise Refuse(f"GTE lever `{name}` already IS its canonical macro") + return [(pos, pos + len(name), base)] + if not ASM_HEAD.match(m, pos): + raise Refuse(f"token mismatch at {site['tu']}:{site['line']}: expected an asm statement") + o = m.find("(", pos) + c = lc._paren_span(m, o) + e = stmt_end(m, pos) + if o < 0 or c < 0 or e < 0: + raise Refuse("unterminated GTE asm statement") + clob = gte_canonical_clob(m[o + 1:c]) # the MASKED inner: T5's `// !FAKE:` sits INSIDE the parens + if clob is None: + raise Refuse(f"GTE lever `{site.get('detail', '')}`: no canonical signature to take the clobbers from") + import gte_consolidate as gc # lazy: gte_consolidate imports THIS module + # the statement without its comments (the marker included — a de-levered site owns no honesty marker); set_clobbers + # parses the sections positionally and a comment between `(` and the template would derail it + clean = "".join(raw[i] for i in range(pos, e) if m[i] == raw[i]) + return [(pos, consume_marker(raw, m, e), gc.set_clobbers(clean, clob))] if cls == "B" and site.get("via"): name = site["via"] if not m.startswith(name, pos): diff --git a/decomp-architect/corpus/tools/P10/delever_permute.py b/decomp-architect/corpus/tools/P10/delever_permute.py new file mode 100644 index 000000000..40495eec2 --- /dev/null +++ b/decomp-architect/corpus/tools/P10/delever_permute.py @@ -0,0 +1,714 @@ +#!/usr/bin/env python3 +"""delever_permute.py — rung D of the lever ladder: decomp-permuter on the RESIDUE exemplars (Phase 36 T6). + + tools/delever_permute.py --plan [--limit N] [--only X ...] # the exemplar list from the ledger (no writes) + tools/delever_permute.py --prepare TU FN # one exemplar's draft.c + target .s, no permuter + tools/delever_permute.py --run [--limit K] [--workers W] [--secs S] [--cycles C] [-j J] [--only X ...] + tools/delever_permute.py --bank [--label dN] [--only X ...] [--dirty-ok] + tools/delever_permute.py --status + tools/delever_permute.py --selftest + +WHY A PERMUTER RUNG AT ALL. Rungs A–C (tools/delever.py) asked "does the object stay identical without this site?" and left +12,970 bodies where the answer was no for at least one site. Those sites are not magic: each is a shape the compiler would have +produced anyway from DIFFERENT C. The permuter's randomizer moves exactly those shapes (temps, statement and declaration order, +scoping, operand order), so the residue is its natural population — and the seed is the LEVER-FREE body, which is what makes this +different from every earlier permuter run in this project: the search starts from C that has no pins and no asm at all, and any +score-0 it reaches is by construction a lever-free match. + +THE PIPELINE PER EXEMPLAR (probe-proven at S98, the T6 log entry): + 1. the lever-free TU text — delever's own rung-A rewrite of every REMOVABLE site of the body (a site it REFUSES makes the + exemplar UNSTRIPPABLE: recorded, skipped — never a silently narrowed scope, R43/silently-narrowed-tool-scope). + 2. the ISOLATED TU — every OTHER definition replaced by its prototype, every `#include "…/shared/…/func_*.h"` replaced + by the prototypes that header defines, INCLUDE_ASM/INCLUDE_RODATA lines and file-scope asm statements dropped: the object + the permuter compiles must hold ONE .text function, because the masked scorer compares whole .text. + 3. cpp -P with the BUILD's own CPPFLAGS + -I (common.h, the prelude, engine_types.h and the GTE header + expand here — p16_permute.make_base_c runs its own cpp WITHOUT includes and would lose them) -> draft.c. + 4. the target .s — tools/verbatim_target_s.py regenerates it from the EXTRACTED ROM IMAGE (R34: an independent + oracle; our own source is the thing under test), into the exemplar's scratch dir, where p16_permute.setup reads it. + 5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds + (pins -> regalloc, barriers/launders/keep-alives -> schedule, mixed -> regalloc). + 6. a score-0 winner is a CANDIDATE, never a bank: --bank puts the function's definition back through + `delever.py --apply-body` (which refuses a body that still carries a class A/B lever and judges it on the object's bytes + through every recipe of the TU), then re-folds the cpp-expanded GTE asm with `gte_consolidate.py --apply --rejudge`. + The clean fleet run (R22) gates the batch, as it gates every batch of this phase. + +SCRATCH IS KEYED BY ALIAS+FN (R48): `.run/P36/permuter/__/` — a function NAME is not unique across the fleet (the +overlays overlap in RAM, so two different bodies can both be `func_8013B274`), and the permuter's own scratch/winner paths are +keyed by the bare name. Every attempt is recorded in `.run/P36/permuter/outcomes.jsonl`, which is also the skip list. + +NEVER RUN THE CAMPAIGN AS A HARNESS BACKGROUND TASK (the low-memory guard kills it): `setsid nohup … &` + a Monitor on the log. +""" +import argparse +import collections +import json +import os +import pathlib +import re +import shutil +import subprocess +import sys +import threading +import time +from concurrent.futures import ThreadPoolExecutor + +REPO = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO / "tools")) +import delever as dl # noqa: E402 +import delever_oracle as oracle # noqa: E402 +import lever_census as lc # noqa: E402 +import p16_permute as p16 # noqa: E402 +import share_census as sc # noqa: E402 + +RUN = REPO / ".run" / "P36" / "permuter" +OUTCOMES = RUN / "outcomes.jsonl" +PY = str(REPO / ".venv" / "bin" / "python") +CPP = "mipsel-linux-gnu-cpp" +# the Makefile's own CPPFLAGS (line 1010) — the draft must see the same macro world the build sees +CPPFLAGS = ["-lang-c", "-Iinclude", "-undef", "-Wall", "-fno-builtin", "-Dmips", "-D__GNUC__=2", + "-D__OPTIMIZE__", "-Dpsx", "-D_PSYQ", "-D_MIPSEL", "-D_LANGUAGE_C"] +# decomp-permuter parses base.c with pycparser, which rejects `__attribute__((packed, aligned(1)))` outright +# ("Syntax error in base.c" -> the run REFUSES and permutes nothing). The draft therefore defines the keyword away. +# This is a fidelity LOSS, not a correctness one: a packed struct the exemplar actually uses will lay out differently +# in the search, so that exemplar simply cannot reach score 0 — and the bank gate is the real object oracle either +# way. The count is recorded per attempt (`attrs`) so the yield report can say whether it mattered (76 sites in the +# whole tree at T5; the class belongs to the types phase, R95). +CPP_ATTR = "-D__attribute__(x)=" +INCLUDE_ASM_LINE = re.compile(r"^[ \t]*INCLUDE_(?:ASM|RODATA)\b[^\n]*\n", re.M) +INCLUDE_LINE = re.compile(r'^[ \t]*#[ \t]*include[ \t]*"([^"]+)"[^\n]*\n', re.M) +SCHED_KINDS = {"barrier", "launder", "keepalive"} +_LOCK = threading.Lock() + + +class Unstrippable(Exception): + """the body cannot be made lever-free by the rewrite table — the exemplar is T7's, not rung D's.""" + + +# ---------------------------------------------------------------------------------------------------------------------- +# the population: the residue's classes, one exemplar each +# ---------------------------------------------------------------------------------------------------------------------- +def load_outcomes(): + rows = [] + if OUTCOMES.exists(): + for l in OUTCOMES.read_text().splitlines(): + if l.strip(): + rows.append(json.loads(l)) + return rows + + +def outcome_append(row): + RUN.mkdir(parents=True, exist_ok=True) + with _LOCK, open(OUTCOMES, "a") as f: + f.write(json.dumps(row) + "\n") + + +def alias_of(tu, row=None): + """the binary whose object holds this TU (a shared header: its first includer's). The ledger row's own aliases first.""" + al = (row or {}).get("aliases") + if al: + return al[0] + by_src = oracle.recipes_by_src(oracle.load_recipes()["recipes"]) + src = tu + if tu.endswith(".h"): + inc = dl.includers().get(tu) or [] + src = next((t for t in inc if t in by_src), None) + if src is None: + return None + recs = by_src.get(src) or [] + return recs[0]["alias"] if recs else None + + +def matches(ex, only): + if not only: + return True + return any(o == ex["fn"] or o == ex["tu"] or o in ex["tu"] or ex["nhash"].startswith(o) or o == ex["alias"] for o in only) + + +def exemplars(only=(), limit=None, include_done=False): + """One exemplar per RESIDUE text class, largest class first (a match banks every copy), then fewest NEEDED sites.""" + cur, order = {}, [] + for r in dl.load_ledger(): + k = (r.get("tu"), r.get("fn")) + if not k[0] or not k[1]: + continue + if k not in cur: + order.append(k) + cur[k] = r + groups = collections.OrderedDict() + for k in order: + r = cur[k] + if r.get("verdict") != "RESIDUE" or r["fn"] == dl.FILE_SCOPE_FN: + continue # a TU's file-scope `volatile` pseudo-body is not a function to permute + groups.setdefault(r.get("nhash_after") or r.get("nhash_before"), []).append((k, r)) + done = set() if include_done else {o["nhash"] for o in load_outcomes() if o.get("kind") == "attempt"} + out = [] + for nh, members in groups.items(): + if nh in done: + continue + (tu, fn), r = members[0] + needed = [s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"] + out.append(dict(nhash=nh, tu=tu, fn=fn, alias=(r.get("aliases") or [None])[0], copies=len(members), + needed=len(needed), kinds=sorted({s["kind"] for s in needed}), + members=[dict(tu=t, fn=f) for (t, f), _ in members], row=r)) + out.sort(key=lambda e: (-e["copies"], e["needed"], e["tu"], e["fn"])) + out = [e for e in out if matches(e, only)] + return out[:limit] if limit else out + + +def klass_for(kinds): + """the permuter_weights profile for a residue's NEEDED-site mix (the profile NAME is accepted verbatim by classify()).""" + ks = set(kinds) + if ks and ks <= SCHED_KINDS: + return "schedule" + return "regalloc" + + +# ---------------------------------------------------------------------------------------------------------------------- +# step 1-3: the lever-free, isolated, preprocessed draft +# ---------------------------------------------------------------------------------------------------------------------- +_sites_cache = None + + +def tu_sites(tu): + global _sites_cache + if _sites_cache is None: + _sites_cache = collections.defaultdict(list) + for s in dl.load_sites(): + _sites_cache[s["tu"]].append(s) + return _sites_cache[tu] + + +def body_sites(tu, fn): + return [s for s in tu_sites(tu) if s.get("fn") == fn] + + +def file_asm_sites(tu): + """the TU's file-scope asm STATEMENTS (13 in the fleet: 7 rodata carves, 6 barriers) — asked of the census, never + re-parsed: a bare `__asm__(` scan cannot tell a statement from the asm-LABEL clause of a declaration + (`extern void func_8005C324(…) __asm__("memcpy");` — 6,423 of those), and eating one leaves a headless K&R body.""" + return [s for s in tu_sites(tu) if s.get("scope") == "file" and s["cls"] == "B" + and s["kind"] not in lc.NON_LEVER_KINDS and s["kind"] != "asm-label"] + + +def lever_free_text(tu, raw, fn, sites, strip=True): + """the TU's text with every REMOVABLE site of THIS body rewritten away (delever's own table) and its file-scope asm + statements deleted (the draft compares .text only); Unstrippable if any body site refuses. + strip=False keeps every lever — that is the CALIBRATION seed: the body as the tree has it, which must score MATCH + against the regenerated target or the harness (target, isolation, cpp, compile) is not measuring this function.""" + m = dl.same_len_mask(raw) + ls = dl.line_starts(raw) + edits, refused = [], [] + for s in file_asm_sites(tu): + pos = ls[s["line"] - 1] + s["col"] - 1 + e = dl.stmt_end(m, pos) + if dl.ASM_HEAD.match(m, pos) and e > 0: + edits.append((pos, dl.consume_marker(raw, m, e), "")) + for s in (sites if strip else []): + if (s["cls"], s["kind"]) not in dl.REMOVABLE: + if s["kind"] in dl.DEFERRED_KINDS: + refused.append((s["kind"], s["line"], "asm-body: T7's work")) + continue + try: + edits += dl.site_edits(raw, m, ls, s) + except dl.Refuse as ex: + refused.append((s["kind"], s["line"], str(ex)[:120])) + if refused: + raise Unstrippable(refused) + if not edits: + return raw + try: + return dl.apply_edits(raw, edits) + except dl.Refuse as ex: + raise Unstrippable([("", 0, str(ex)[:120])]) + + +def prototype_at(text, m, start): + """`;` for the definition that starts at index `start` — the head up to the last `)` before its opening `{`.""" + o = m.find("{", start) + if o < 0: + return None + close = m.rfind(")", start, o) + if close < 0: + return None + head = " ".join(text[start:close + 1].split()) + return head + ";" + + +def header_prototypes(path): + """the prototypes of every function a shared header DEFINES (a static one keeps `static`).""" + p = REPO / path + if not p.exists(): + return [] + text = p.read_text(errors="surrogateescape") + m = dl.same_len_mask(text) + ls = dl.line_starts(text) + out = [] + for r in sc.scan_text(text, path, shared_defs=None): + if r["form"] != "def": + continue + pr = prototype_at(text, m, ls[r["line"] - 1]) + if pr: + out.append(pr) + return out + + +def isolate(tu, text, fn): + """every OTHER definition -> its prototype; shared-header includes -> their prototypes; INCLUDE_ASM/file-scope asm dropped.""" + m = dl.same_len_mask(text) + ls = dl.line_starts(text) + recs = sc.scan_text(text, tu, shared_defs=None) + defs = [r for r in recs if r["form"] == "def"] + if not any(r["name"] == fn for r in defs): + raise Unstrippable([("", 0, f"{fn} is not defined in {tu}")]) + edits = [] + for r in defs: + s, e = ls[r["line"] - 1], ls[r["end"]] + + if r["name"] == fn: + continue + pr = prototype_at(text, m, s) + if pr is None: + raise Unstrippable([("", r["line"], f"no prototype for {r['name']}")]) + edits.append((s, e, pr + "\n")) + for mm in INCLUDE_ASM_LINE.finditer(m): + edits.append((mm.start(), mm.end(), "")) + for mm in INCLUDE_LINE.finditer(m): + inc = mm.group(1) + rel = os.path.normpath(os.path.join(os.path.dirname(tu), inc)) + if not rel.startswith("src/shared/"): + continue + pr = header_prototypes(rel) + if pr: # a macro-only header (the prelude) has none: keep the include + edits.append((mm.start(), mm.end(), "\n".join(pr) + "\n")) + return dl.apply_edits(text, sorted(edits, key=lambda x: x[0])) + + +def drop_file_scope_asm(text, tu): + """delete every asm STATEMENT that is not inside a function — after cpp, `include/include_asm.h` has injected + `__asm__(".include \\"include/labels.inc\\"");` at the top of every draft, and the permuter's own compile.sh + already prepends `.include "macro.inc"`, so the assembler dies on `Macro 'glabel' was already defined`. + An asm-LABEL clause (`extern void f(…) __asm__("memcpy");`, 6,423 of them) is NOT a statement: it follows a + declarator, so the preceding non-space character is not `;` or `}`.""" + m = dl.same_len_mask(text) + ls = dl.line_starts(text) + spans = [(ls[r["line"] - 1], ls[r["end"]]) for r in sc.scan_text(text, tu, shared_defs=None) if r["form"] == "def"] + edits = [] + for mm in dl.ASM_HEAD.finditer(m): + p = mm.start() + if any(s <= p < e for s, e in spans): + continue + prev = m[:p].rstrip() + if prev and prev[-1] not in ";}": + continue # an asm-label clause on a declaration + e = dl.stmt_end(m, p) + if e > 0: + edits.append((p, e, "")) + return dl.apply_edits(text, edits) if edits else text + + +def cpp_expand(tu, text): + """the build's cpp over the isolated text (fed on stdin, so `-I` is what resolves its relative includes).""" + p = subprocess.run([CPP, "-P"] + CPPFLAGS + [CPP_ATTR, "-I" + os.path.dirname(tu), "-"], + input=text, capture_output=True, text=True, cwd=REPO) + if p.returncode or not p.stdout.strip(): + raise Unstrippable([("", 0, (p.stderr.strip().splitlines() or ["empty output"])[-1][:160])]) + return p.stdout + + +def scratch_of(alias, fn): + return RUN / f"{alias}__{fn}" + + +def prepare(ex, quiet=False): + """(scratch dir, draft path, target path, klass) for one exemplar — the whole probe-2 pipeline.""" + tu, fn = ex["tu"], ex["fn"] + alias = ex["alias"] or alias_of(tu, ex.get("row")) + if not alias: + raise Unstrippable([("", 0, f"no binary compiles {tu}")]) + ex["alias"] = alias + raw = (REPO / tu).read_text(errors="surrogateescape") + sites = body_sites(tu, fn) + if not sites: + raise Unstrippable([("", 0, f"the census has no site in {tu}:{fn} (stale? rerun lever_census --sites)")]) + d = scratch_of(alias, fn) + if d.exists(): + shutil.rmtree(d) + (d / "gas").mkdir(parents=True) + (d / "splat").mkdir() + for strip, name in ((True, "draft.c"), (False, "levered.c")): + free = lever_free_text(tu, raw, fn, sites, strip=strip) + iso = isolate(tu, free, fn) + text = drop_file_scope_asm(cpp_expand(tu, iso), tu) + if strip: + (d / "tu.c").write_text(free, errors="surrogateescape") # the lever-free TU (what rung A would have written) + (d / "iso.c").write_text(iso, errors="surrogateescape") # + one body, the rest prototypes (pre-cpp; for reading) + (d / name).write_text(text, errors="surrogateescape") + if strip: + base = p16.make_base_c(text) + if not p16.defines_fn(base, fn): + raise Unstrippable([("", 0, f"base.c would lose the definition of {fn} (inspect {d}/draft.c)")]) + # TWO target forms, for the two instruments. --gas is the ASSEMBLABLE one ($-registers, .L labels, symbol names) that + # `p16_permute.setup` turns into target.o; the default splat listing is the WORD oracle `match_one.py` reads. Nothing + # ever assembles the splat form — pasting it into decomp.me was P34's `invalid operands 'li a2,2'` (R98/DK-81), and + # `mipsel-as` refuses its bare `addiu sp,sp,-152` the same way. + for args, sub, name in (( ["--gas"], "gas", f"{fn}.gas.s"), ([], "splat", f"{fn}.s")): + r = subprocess.run([PY, "tools/verbatim_target_s.py", "--binary", alias, "--fn", fn] + args + + ["--out", str(d / "tgt")], capture_output=True, text=True, cwd=REPO) + tgt = d / "tgt" / alias / name + if r.returncode or not tgt.exists(): + raise Unstrippable([("", 0, ((r.stderr or r.stdout).strip().splitlines() or ["no listing"])[-1][:160])]) + shutil.copy(tgt, d / sub / f"{fn}.s") # p16_permute.setup / match_one.py read /.s + if not quiet: + n = len((d / "draft.c").read_text(errors="surrogateescape").splitlines()) + print(f" prepared {alias}__{fn}: draft {n} lines, target {(r.stdout or '').strip().split()[-3:]}", flush=True) + return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"]) + + +# ---------------------------------------------------------------------------------------------------------------------- +# the calibration (R39/R56): the harness must agree with the tree before it may judge a candidate +# ---------------------------------------------------------------------------------------------------------------------- +MISMATCH_RE = re.compile(r"(\d+) mismatched") + + +def closeness(d, fn, c): + """(mismatched instructions, first line) from tools/match_one.py against the SPLAT listing — 0 == byte-identical + (relocation-masked). None when the tool could not judge (R61: not judged is not a verdict).""" + r = subprocess.run([PY, "tools/match_one.py", fn, "--c", str(c), "--asm-subdir", (d / "splat").as_posix()], + capture_output=True, text=True, cwd=REPO) + first = ((r.stdout or "") + (r.stderr or "")).strip().splitlines() + first = first[0] if first else "(no output)" + if "MATCH" in first and "mismatched" not in first: + return 0, first + mm = MISMATCH_RE.search(first) + return (int(mm.group(1)) if mm else None), first + + +def calibrate_one(ex, quiet=False): + """the control: the body AS THE TREE HAS IT (levers and all) must be MATCH against the regenerated target. It proves the + whole chain — the target from the ROM image, the isolation, the cpp expansion, the pinned triple — is measuring THIS + function. Then the lever-free draft's distance is the search's real starting point, and it is worth recording (R37).""" + d, draft, tgt, klass = prepare(ex, quiet=quiet) + lev, lev_line = closeness(d, ex["fn"], d / "levered.c") + free, free_line = closeness(d, ex["fn"], draft) + ok = (lev == 0) + if not quiet: + print(f" {'OK ' if ok else 'FAIL'} {ex['alias']}__{ex['fn']}: levered {lev_line[:60]} · lever-free {free_line[:60]}", + flush=True) + return dict(ok=ok, levered=lev, free=free, levered_line=lev_line, free_line=free_line, klass=klass, d=d) + + +# ---------------------------------------------------------------------------------------------------------------------- +# step 5: the permuter +# ---------------------------------------------------------------------------------------------------------------------- +BEST_RE = re.compile(r"best score = (\d+)") + + +def run_one(ex, secs, cycles, j, max_start=None): + t0 = time.time() + row = dict(kind="attempt", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=ex["nhash"], tu=ex["tu"], fn=ex["fn"], + alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"], + secs=secs, cycles=cycles, j=j) + try: + d, draft, tgt, klass = prepare(ex) + except Unstrippable as ex_: + row.update(verdict="UNSTRIPPABLE", why=ex_.args[0][:6], seconds=round(time.time() - t0, 1)) + outcome_append(row) + print(f" {ex['alias'] or '?'}__{ex['fn']}: UNSTRIPPABLE — {ex_.args[0][:2]}", flush=True) + return row + # EVERY attempt calibrates itself first (R56): the body as the tree has it must be MATCH against the regenerated + # target, or this harness is not measuring this function and its budget would buy a meaningless number. ~15 s + # against ~12 min of search. + lev, lev_line = closeness(d, ex["fn"], d / "levered.c") + free, free_line = closeness(d, ex["fn"], draft) + row.update(levered=lev, free=free, levered_line=lev_line[:120], free_line=free_line[:120]) + if lev != 0: + row.update(verdict="UNCALIBRATED", seconds=round(time.time() - t0, 1)) + outcome_append(row) + print(f" {ex['alias']}__{ex['fn']}: UNCALIBRATED — the LEVERED body is not MATCH ({lev_line[:80]})", flush=True) + return row + if max_start is not None and (free is None or free > max_start): + # TRIAGE, not a verdict about the function: a lever-free body this far from the target (usually because removing a + # hand-placed instruction changed the instruction COUNT and shifted everything after it) will not close in a + # 12-minute search. Recorded with its distance; `--include-done` draws it again when the budget or the rung grows. + row.update(verdict="FAR", seconds=round(time.time() - t0, 1)) + outcome_append(row) + print(f" {ex['alias']}__{ex['fn']}: FAR start={free} > {max_start} — not searched ({ex['copies']} copies)", flush=True) + return row + rel = d.relative_to(REPO).as_posix() + log = d / "ils.log" + cmd = [PY, "tools/permuter_ils.py", ex["fn"], "--draft", (draft.relative_to(REPO)).as_posix(), + "--asm-subdir", rel + "/gas", "--klass", klass, "--cycles", str(cycles), "--secs", str(secs), + "--j", str(j), "--winners", rel, "--pd", rel + "/pd"] + row["klass"] = klass + row["cmd"] = " ".join(cmd) + rc = None + with open(log, "w") as f: + f.write(" ".join(cmd) + "\n\n") + f.flush() + try: + rc = subprocess.run(cmd, cwd=REPO, stdout=f, stderr=subprocess.STDOUT, + timeout=cycles * (secs + 45) + 300).returncode + except subprocess.TimeoutExpired: + f.write("\n[delever_permute] the ILS wrapper overran its own budget — killed\n") + out = log.read_text(errors="replace") + scores = [int(x) for x in BEST_RE.findall(out)] + win = d / f"{ex['fn']}.c" + # NOT JUDGED IS NOT A VERDICT (R61): a setup failure, a parser refusal or a wrapper that never iterated is its own + # outcome — reporting it as NO-MATCH is how a tool reports work it never did. + if win.exists(): + verdict = "MATCH" + elif "setup FAILED" in out: + verdict = "SETUP-FAILED" + elif "ABORTED" in out or "REFUSED" in out: + verdict = "ABORTED" + elif "cycle 1:" not in out: + verdict = "NOT-JUDGED" + else: + verdict = "NO-MATCH" + row.update(best=(min(scores) if scores else None), seconds=round(time.time() - t0, 1), rc=rc, + winner=(win.relative_to(REPO).as_posix() if win.exists() else None), verdict=verdict) + outcome_append(row) + print(f" {ex['alias']}__{ex['fn']}: {row['verdict']} start={row['free']} best={row['best']} " + f"({row['seconds']}s, {ex['copies']} copies, {ex['needed']} needed, {klass})", flush=True) + return row + + +# ---------------------------------------------------------------------------------------------------------------------- +# step 6: the bank (serial — every step here runs `make`) +# ---------------------------------------------------------------------------------------------------------------------- +def winner_body(winner_c, fn, tu): + """the winner's definition of fn, as the text `delever --apply-body` splices in.""" + draft = p16.winner_to_draft(pathlib.Path(winner_c).read_text(errors="surrogateescape")) + ls = dl.line_starts(draft) + for r in sc.scan_text(draft, tu, shared_defs=None): + if r["form"] == "def" and r["name"] == fn: + return draft[ls[r["line"] - 1]:ls[r["end"]]] + return None + + +def bank(a): + outs = load_outcomes() + banked = {(o["alias"], o["fn"], o["nhash"]) for o in outs if o.get("kind") == "bank" and o.get("applied")} + todo = [o for o in outs if o.get("kind") == "attempt" and o.get("verdict") == "MATCH" + and (o["alias"], o["fn"], o["nhash"]) not in banked + and (not a.only or any(x in (o["fn"], o["tu"], o["alias"]) or o["nhash"].startswith(x) for x in a.only))] + if not todo: + print("delever_permute --bank: no unbanked winner (R68: nothing to do, loudly)") + return 1 + ok, bad = 0, 0 + for o in todo: + d = scratch_of(o["alias"], o["fn"]) + body = winner_body(REPO / o["winner"], o["fn"], o["tu"]) if o.get("winner") else None + if not body: + print(f" {o['alias']}__{o['fn']}: the winner holds no definition of {o['fn']} — SKIPPED", flush=True) + bad += 1 + continue + bf = d / "body.c" + bf.write_text(body, errors="surrogateescape") + cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(), + "--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else []) + r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True) + print(" " + (r.stdout or r.stderr).strip().splitlines()[-1][:220], flush=True) + row = dict(kind="bank", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=o["nhash"], tu=o["tu"], fn=o["fn"], + alias=o["alias"], label=a.label, applied=(r.returncode == 0), + apply_out=(r.stdout or r.stderr).strip()[-400:]) + if r.returncode == 0: + g = subprocess.run([PY, "tools/gte_consolidate.py", "--apply", "--only", o["tu"], "--rejudge", + "--label", a.label + "g"], cwd=REPO, capture_output=True, text=True) + row["gte_refold"] = (g.stdout or g.stderr).strip()[-300:] + ok += 1 + else: + bad += 1 + outcome_append(row) + print(f"delever_permute --bank: {ok} applied, {bad} rejected of {len(todo)} — " + f"now the clean fleet run (R22) and one commit per batch (R42)") + return 0 if ok else 1 + + +# ---------------------------------------------------------------------------------------------------------------------- +# the commands +# ---------------------------------------------------------------------------------------------------------------------- +def cmd_plan(a): + ex = exemplars(a.only, a.limit, a.include_done) + tot = exemplars(include_done=True) + kinds = collections.Counter(k for e in ex for k in e["kinds"]) + print(f"delever_permute --plan: {len(ex)} exemplars drawable of {len(tot)} residue classes " + f"({sum(e['copies'] for e in ex):,} bodies behind them); NEEDED kinds {dict(kinds)}") + for e in ex[:a.show]: + print(f" {e['copies']:4d} copies · {e['needed']:2d} needed {','.join(e['kinds']):<28} " + f"{klass_for(e['kinds']):8} {e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}") + if len(ex) > a.show: + print(f" … {len(ex) - a.show} more") + return 0 + + +def cmd_run(a): + dl.ensure_census(a.j) + ex = exemplars(a.only, a.limit, a.include_done) + if not ex: + print("delever_permute --run: no drawable exemplar (every class attempted, or --only matched none)") + return 1 + per = max(1, a.j // max(1, a.workers)) + print(f"delever_permute --run: {len(ex)} exemplars, {a.workers} at a time, {a.cycles} cycles x {a.secs}s @ -j{per} each " + f"(budget ≈ {len(ex) / a.workers * a.cycles * a.secs / 3600:.1f} h)", flush=True) + t0 = time.time() + rows = [] + with ThreadPoolExecutor(max_workers=a.workers) as pool: + for r in pool.map(lambda e: run_one(e, a.secs, a.cycles, per, a.max_start), ex): + rows.append(r) + won = [r for r in rows if r["verdict"] == "MATCH"] + v = collections.Counter(r["verdict"] for r in rows) + print(f"\npermuter: {len(won)} of {len(rows)} exemplars matched lever-free in {(time.time() - t0) / 3600:.2f} h " + f"({sum(r['copies'] for r in won):,} of {sum(r['copies'] for r in rows):,} bodies behind them) — " + + " · ".join(f"{k} {n}" for k, n in v.most_common())) + return 0 + + +def cmd_calibrate(a): + ex = exemplars(a.only, a.limit or 6, include_done=True) + print(f"delever_permute --calibrate: {len(ex)} exemplars — the levered body must be MATCH, the lever-free one is the " + f"search's starting distance", flush=True) + rows, dist = [], [] + for e in ex: + try: + r = calibrate_one(e) + except Unstrippable as u: + print(f" SKIP {e['tu']}:{e['fn']} — {u.args[0][:2]}", flush=True) + continue + rows.append(r) + if r["ok"] and r["free"] is not None: + dist.append(r["free"]) + good = sum(1 for r in rows if r["ok"]) + print(f"\ncalibration: {good} of {len(rows)} levered bodies MATCH their regenerated target" + + (f" · lever-free distance min {min(dist)} / median {sorted(dist)[len(dist) // 2]} / max {max(dist)} " + f"over {len(dist)} exemplars" if dist else "")) + (RUN / "calibration.json").write_text(json.dumps( + [dict(tu=e["tu"], fn=e["fn"], alias=e["alias"], ok=r["ok"], levered=r["levered"], free=r["free"], + levered_line=r["levered_line"], free_line=r["free_line"]) + for e, r in zip(ex, rows)], indent=1)) + return 0 if good == len(rows) and rows else 1 + + +def cmd_status(a): + outs = load_outcomes() + att = [o for o in outs if o.get("kind") == "attempt"] + bk = [o for o in outs if o.get("kind") == "bank"] + left = exemplars() + v = collections.Counter(o.get("verdict") for o in att) + print(f"delever_permute --status: {len(att)} attempts {dict(v)} · {sum(1 for b in bk if b.get('applied'))} banked of " + f"{len(bk)} tried · {len(left)} residue classes not yet attempted " + f"({sum(e['copies'] for e in left):,} bodies)") + for o in att: + if o.get("verdict") == "MATCH": + print(f" MATCH {o['alias']}__{o['fn']} ({o['copies']} copies, best={o.get('best')}, {o.get('seconds')}s)") + return 0 + + +FIX_TU = """#include "common.h" +extern s32 D_800A0000; +s32 helper_one(s32 a0) +{ + return a0 + D_800A0000; +} +INCLUDE_ASM("asm/x/nonmatchings/x", func_dead); +s32 target_fn(s32 a0, s32 *a1) +{ + register s32 s __asm__("$16") = a0; + __asm__ __volatile__("" ::: "memory"); + *a1 = helper_one(s); + return s; +} +static void tail(void) { } +""" + + +def cmd_selftest(a): + ok = True + d = REPO / ".run" / "P36" / "permuter" / "_selftest" + if d.exists(): + shutil.rmtree(d) + d.mkdir(parents=True) + tu = "src/selftest/fix.c" + sites = [dict(tu=tu, fn="target_fn", cls="A", kind="pin", detail="$16", via="", line=10, col=5, + fn_line=9, fn_end=15, zero=False), + dict(tu=tu, fn="target_fn", cls="B", kind="barrier", detail="", via="", line=11, col=5, + fn_line=9, fn_end=15, zero=False)] + free = lever_free_text(tu, FIX_TU, "target_fn", sites) + if "__asm__" in free or "register" in free: + print(f"selftest: the lever-free text still carries a lever:\n{free}") + ok = False + iso = isolate(tu, free, "target_fn") + if "helper_one(s32 a0);" not in " ".join(iso.split()): + print(f"selftest: helper_one was not reduced to a prototype:\n{iso}") + ok = False + if "INCLUDE_ASM" in iso: + print("selftest: the INCLUDE_ASM line survived isolation") + ok = False + if iso.count("{") != 1: + print(f"selftest: {iso.count('{')} bodies left in the isolated TU (want 1)") + ok = False + if "static void tail(void);" not in " ".join(iso.split()): + print("selftest: a static definition lost its `static` (or its prototype)") + ok = False + if klass_for(["pin"]) != "regalloc" or klass_for(["barrier", "launder"]) != "schedule" \ + or klass_for(["pin", "barrier"]) != "regalloc" or klass_for([]) != "regalloc": + print("selftest: klass_for is wrong") + ok = False + base = p16.make_base_c(iso) + if not p16.defines_fn(base, "target_fn"): + print("selftest: base.c lost target_fn") + ok = False + shutil.rmtree(d) + print(f"delever_permute --selftest: {'OK' if ok else 'FAILED'} — lever-free + isolation + prototypes + klass + base.c") + return 0 if ok else 1 + + +def main(): + ap = argparse.ArgumentParser() + g = ap.add_mutually_exclusive_group(required=True) + g.add_argument("--plan", action="store_true") + g.add_argument("--prepare", nargs=2, metavar=("TU", "FN")) + g.add_argument("--calibrate", action="store_true", + help="the control: N exemplars' levered bodies must all be MATCH against their regenerated targets") + g.add_argument("--run", action="store_true") + g.add_argument("--bank", action="store_true") + g.add_argument("--status", action="store_true") + g.add_argument("--selftest", action="store_true") + ap.add_argument("--only", nargs="*", default=[], help="fn, TU (substring), alias or nhash prefix") + ap.add_argument("--limit", type=int) + ap.add_argument("--show", type=int, default=25) + ap.add_argument("--workers", type=int, default=4) + ap.add_argument("--secs", type=int, default=240) + ap.add_argument("--cycles", type=int, default=3) + ap.add_argument("-j", type=int, default=16) + ap.add_argument("--label", default="d1") + ap.add_argument("--dirty-ok", action="store_true") + ap.add_argument("--max-start", type=int, + help="skip (as FAR) an exemplar whose lever-free body is further than N instructions from the target") + ap.add_argument("--include-done", action="store_true", help="draw classes that already have an outcome row") + a = ap.parse_args() + os.chdir(REPO) + RUN.mkdir(parents=True, exist_ok=True) + if a.selftest: + return cmd_selftest(a) + if a.plan: + return cmd_plan(a) + if a.status: + return cmd_status(a) + if a.prepare: + tu, fn = a.prepare + ex = next((e for e in exemplars(include_done=True) if e["tu"] == tu and e["fn"] == fn), None) + if ex is None: + ex = dict(nhash="", tu=tu, fn=fn, alias=None, copies=1, needed=0, kinds=[], members=[], row={}) + d, draft, tgt, klass = prepare(ex) + print(f"delever_permute --prepare: {d.relative_to(REPO)} (draft.c, {tgt.name}, klass={klass})") + return 0 + if a.calibrate: + return cmd_calibrate(a) + if a.run: + return cmd_run(a) + if a.bank: + return bank(a) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/decomp-architect/corpus/tools/P10/gte_consolidate.py b/decomp-architect/corpus/tools/P10/gte_consolidate.py index 6d8765af9..d818cfa47 100644 --- a/decomp-architect/corpus/tools/P10/gte_consolidate.py +++ b/decomp-architect/corpus/tools/P10/gte_consolidate.py @@ -486,20 +486,16 @@ def plan_files(inv, canon, only=None, bound=None): return files -def direct_rewrite(s, canon): - """('call', text, name) | ('lever', variant_name, extra) | ('none', why, None) for a direct GTE statement.""" - sg = s["sig"] +def canonical_match(sg, canon): + """(names, clobbers) for a signed statement's signature — the canonical macro it IS (one entry), or the + concatenation of two canonical macros it is (a load then an op: bytes = b1 + b2, inputs the first's, + clobbers the union) — else (None, None). ONE reader of the canonical table (R33): `direct_rewrite` asks it + what the statement should look like, `delever.gte_canonical_clob` asks it what the canonical clobbers are.""" if not sg["ok"]: - return "none", "unsigned", None - key = sig_key(sg) - t = canon["canonical"].get(key) + return None, None + t = canon["canonical"].get(sig_key(sg)) if t and sg["nout"] == 0: - if tuple(sg["clob"]) == tuple(t["clob"]): - return "call", f"{t['name']}({', '.join(sg['ins'])});", t["name"] - extra = sorted(set(sg["clob"]) - set(t["clob"])) - fewer = sorted(set(t["clob"]) - set(sg["clob"])) - return "lever", (f"{t['name']}_m" if extra == ["memory"] and not fewer else f"{t['name']}_v"), extra or fewer - # a concatenation of two canonical macros (a load then an op): bytes = b1 + b2, inputs = the first's, clobbers = the union + return [t["name"]], list(t["clob"]) b = sg["bytes"] for k1, t1 in canon["canonical"].items(): b1 = k1.split("|")[0] @@ -509,12 +505,28 @@ def direct_rewrite(s, canon): t2 = canon["canonical"].get(k2) if not t2 or t1["nout"] or t1["nin"] != sg["nin"]: continue - union = sorted(set(t1["clob"]) | set(t2["clob"])) - if union == list(sg["clob"]): - return "call", f"{t1['name']}({', '.join(sg['ins'])}); {t2['name']}();", f"{t1['name']}+{t2['name']}" - extra = sorted(set(sg["clob"]) - set(union)) - return "lever", f"{t1['name']}+{t2['name']}_m" if extra == ["memory"] else f"{t1['name']}+{t2['name']}_v", extra or sorted(set(union) - set(sg["clob"])) - return "none", "no canonical signature", None + return [t1["name"], t2["name"]], sorted(set(t1["clob"]) | set(t2["clob"])) + return None, None + + +def direct_rewrite(s, canon): + """('call', text, name) | ('lever', variant_name, extra) | ('none', why, None) for a direct GTE statement.""" + sg = s["sig"] + if not sg["ok"]: + return "none", "unsigned", None + names, clob = canonical_match(sg, canon) + if names is None: + return "none", "no canonical signature", None + if len(names) == 1: + if tuple(sg["clob"]) == tuple(clob): + return "call", f"{names[0]}({', '.join(sg['ins'])});", names[0] + extra = sorted(set(sg["clob"]) - set(clob)) + fewer = sorted(set(clob) - set(sg["clob"])) + return "lever", (f"{names[0]}_m" if extra == ["memory"] and not fewer else f"{names[0]}_v"), extra or fewer + if clob == list(sg["clob"]): + return "call", f"{names[0]}({', '.join(sg['ins'])}); {names[1]}();", f"{names[0]}+{names[1]}" + extra = sorted(set(sg["clob"]) - set(clob)) + return "lever", f"{names[0]}+{names[1]}_m" if extra == ["memory"] else f"{names[0]}+{names[1]}_v", extra or sorted(set(clob) - set(sg["clob"])) def file_edits(tu, raw, p, canon, label): diff --git a/decomp-architect/corpus/tools/P7/p16_permute.py b/decomp-architect/corpus/tools/P7/p16_permute.py index e163b78b1..2f66ab554 100644 --- a/decomp-architect/corpus/tools/P7/p16_permute.py +++ b/decomp-architect/corpus/tools/P7/p16_permute.py @@ -233,8 +233,11 @@ def klass_for_fn(fn): return "", "" -def setup(fn, draft_c, asm_subdir=ASM, klass=None, where=""): - pd = os.path.join(REPO, ".run/permuter", fn) +def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None): + # `outdir` keys the scratch dir by the CALLER's identity instead of the bare function name (R48). A name is not + # unique across the fleet — the overlays overlap in RAM, so two different bodies are both `func_8013B274` — and two + # concurrent runs on one name would share (and corrupt) this directory. Default: the historical path, unchanged. + pd = os.path.join(REPO, outdir) if outdir else os.path.join(REPO, ".run/permuter", fn) if os.path.exists(pd): shutil.rmtree(pd) os.makedirs(pd) diff --git a/decomp-architect/corpus/tools/P7/permuter_ils.py b/decomp-architect/corpus/tools/P7/permuter_ils.py index abf95ed3e..106a01c23 100644 --- a/decomp-architect/corpus/tools/P7/permuter_ils.py +++ b/decomp-architect/corpus/tools/P7/permuter_ils.py @@ -41,10 +41,12 @@ def main(): ap.add_argument("--secs", type=int, default=180, help="per-cycle time box") ap.add_argument("--j", type=int, default=12) ap.add_argument("--winners", default=".run/permuter-winners") + ap.add_argument("--pd", help="scratch dir for this run (default .run/permuter/) — key it by alias+fn when " + "several runs share a function NAME (R48)") a = ap.parse_args() draft = open(a.draft).read() - pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass) + pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd) if not pd: print("ILS setup FAILED (target .s didn't assemble?)"); sys.exit(1) print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}") diff --git a/decomp-architect/tools/MANIFEST.md b/decomp-architect/tools/MANIFEST.md index 70cd897ad..1d3549ec0 100644 --- a/decomp-architect/tools/MANIFEST.md +++ b/decomp-architect/tools/MANIFEST.md @@ -8,8 +8,8 @@ > its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task; > their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.* > -> **Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded), of which 300 live rows -> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 20 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 330 (the installer checks its copy against this figure). +> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows +> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure). ## P1 — extraction + manifest @@ -341,6 +341,7 @@ | `verbatim_target_s.py` | regenerate a splitter-format target disassembly for a function no longer stubbed | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | repo build/asm layout | | `delever_cycle.sh` | run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | the batch size, the aliases, the fleet count | | `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha | +| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator | | `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas | | `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle | | `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout | diff --git a/docs/tool-index.md b/docs/tool-index.md index d998eb863..a8626b363 100644 --- a/docs/tool-index.md +++ b/docs/tool-index.md @@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool, then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.* -**Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 29 (a SETUP row only), ORPHAN 30 (neither) — of 300. Portability: PORTABLE 23, ADAPT 260, PROJECT-ONLY 17. +**Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 30 (a SETUP row only), ORPHAN 30 (neither) — of 301. Portability: PORTABLE 23, ADAPT 261, PROJECT-ONLY 17. ## P1 — extraction + manifest @@ -72,7 +72,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | When you need to… | Tool | What it does | Proven by | Adapt | Class | |---|---|---|---|---|---| | compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+6) | MIPS relocation encodings | LIVE | -| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+22) | compiler triple, repo build flags | LIVE | +| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+23) | compiler triple, repo build flags | LIVE | | dump every compiler pass file for a self-contained draft | `cc1_dumps.sh` | Dumps every compiler pass file for a self-contained draft into a private directory | cc1_dumps_tu.sh, ghost_census.py | compiler triple, repo scratch paths | LIVE | | dump every compiler pass file for the spliced real translation unit | `cc1_dumps_tu.sh` | Same pass dumps for the spliced real translation unit, the faithful compile | alloc_table.py | absolute repo path, compiler triple | LIVE | | re-judge standalone compile failures against their real translation unit before dropping | `rtu_second_chance.py` | Re-judges standalone compile-failures against their real translation unit before dropping them | maintenance.sh | repo scratch paths | LIVE | @@ -175,10 +175,10 @@ then what proved it. The same data generates the day-one kit's manifest and its | classify a length-drift near-miss against its own target and build its routing card | `len_tells.py` | Classifies a length-drift near-miss against its own target and builds the routing card | atlas_features.py | repo scratch paths | LIVE | | classify a residual's shape to its documented lever, as an independent implementation | `residual_rules_b.py` | Independent second implementation of residual-shape to rule classification, for head-to-head | triage_ladder.py | repo doc paths | LIVE | | classify where a remapped sibling's compiled bytes diverge from its target | `diff_regions.py` | Classifies where a remapped member's compiled bytes diverge from its target | — | repo build paths | ORPHAN | -| compile a permuter candidate exactly as the build rule does | `permuter/compile.sh` | The permuter's compile command, mirroring the build rule exactly so objects are build-faithful | p16_permute.py, compile_o0.sh | compiler triple, repo build flags | LIVE | +| compile a permuter candidate exactly as the build rule does | `permuter/compile.sh` | The permuter's compile command, mirroring the build rule exactly so objects are build-faithful | delever_permute.py, p16_permute.py, compile_o0.sh | compiler triple, repo build flags | LIVE | | compile a permuter candidate unoptimized, exactly as the build rule does | `permuter/compile_o0.sh` | Unoptimized variant of the same permuter compile command | p16_permute.py | compiler triple, repo build flags | LIVE | | decide whether a residual reorder is scheduling or a register-grant consequence | `oracle/reg_renumber_swap.sh` | Mechanized oracle separating a register-allocation swap from a scheduling reorder | — | compiler triple, repo scratch paths | ORPHAN | -| drive the permuter per function, building base, target and settings | `p16_permute.py` | Per-function permuter driver building base, target and settings, then reporting closeness | auto_driver.py, grinder.py, permuter_ils.py | repo scratch paths, permuter layout | LIVE | +| drive the permuter per function, building base, target and settings | `p16_permute.py` | Per-function permuter driver building base, target and settings, then reporting closeness | auto_driver.py, delever_permute.py, grinder.py, permuter_ils.py | repo scratch paths, permuter layout | LIVE | | find pseudo-registers with references but no remaining occurrences, and their frame slots | `ghost_census.py` | Frame-residue oracle: finds pseudos with references but no remaining occurrences, and their slots | cc1_dumps.sh | compiler pass-dump format | LIVE | | find which words diverge when the whole-binary gate says different but not where | `diff_autopsy.sh` | Reproduces exactly what the gate saw, then decodes the diverging words and restores the tree | — | repo src/build paths | REFERENCED | | grind the closest near-misses with the permuter and bank through the gate | `grinder.py` | Token-free permuter daemon grinding the closest near-misses and banking through the gate | auto_status.sh, auto_supervisor.sh, autopsy.py, gate_stage.py (+4) | repo scratch paths | LIVE | @@ -191,7 +191,7 @@ then what proved it. The same data generates the day-one kit's manifest and its | steer permuter mutation toward the lever family the residual class implies | `permuter_weights.py` | Directs permuter mutation toward the lever family the residual class implies | autopsy.py, p16_permute.py | permuter settings format | LIVE | | tag every compiler-source citation with the source tree it refers to | `gccmap_cites.py` | Tags every compiler-source citation in the codegen-map docs with the tree it refers to | .github/workflows/no-rom.yml, Makefile, tool_census.py | repo doc/reference paths | LIVE | | turn a red whole-binary result into a named list of divergent symbols | `main_diff_locate.py` | Turns a red whole-binary result into a named list of divergent symbols | gate_main.py | repo build/config layout | LIVE | -| warm-restart the permuter from its best waypoint each cycle | `permuter_ils.py` | Iterated-local-search wrapper warm-restarting the permuter from the best waypoint each cycle | grinder.py, permuter_sweep.py | permuter layout, repo scratch paths | LIVE | +| warm-restart the permuter from its best waypoint each cycle | `permuter_ils.py` | Iterated-local-search wrapper warm-restarting the permuter from the best waypoint each cycle | delever_permute.py, grinder.py, permuter_sweep.py | permuter layout, repo scratch paths | LIVE | ## P8 — the campaign: cards, lanes, gates, recovery, harvest @@ -322,23 +322,24 @@ then what proved it. The same data generates the day-one kit's manifest and its |---|---|---|---|---|---| | add per-site function-pointer casts so a draft can call a differently typed callee | `cast_call_sites.py` | Adds per-site function-pointer casts so a draft can call a differently typed callee | canon_sig_reconcile.py, cast_self_callers.py, family_sweep.py, gate_stage.py (+4) | repo src layout | LIVE | | apply curated names and signatures inside the analysis tool and save | `ghidra_scripts/ApplySymbols.java` | The in-tool half of that mirror: apply curated names and signatures, save on exit | ghidra_apply_symbols.sh, ghidra_export_annotations.sh, ghidra_rebuild.sh, ExportAnnotations.java (+1) | none | LIVE | -| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, gte_consolidate.py, progress.py (+1) | repo paths, the shared-body scanner | LIVE | -| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked _m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever_cycle.sh, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE | +| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+2) | repo paths, the shared-body scanner | LIVE | +| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked _m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever.py, delever_cycle.sh, delever_permute.py, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE | | convert a shared-body macro header into per-function plain-C headers included at each site | `macro_to_header.py` | The Phase-35 converter: every DEFINE_ macro body becomes a plain-C header under src/shared// included at its site, the legacy name-parameterized headers converted, the registry text-edited, the macro header deleted; --plan / --apply / --finalize / --verify | share_body.py, tool_census.py | the macro form is this project's; a kit-born project shares headers from its first bank | LIVE | | emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE | | extract inline-defined types and typedefs from a source file into a shared header | `build_engine_types.py` | Extracts inline-defined named types and typedefs from a source file into a shared header | inject_capped_externs.py, lift_types.py, uniquify_type.py | repo shared-header path | LIVE | | flag address-named references whose address now has a curated name | `lint_symbol_refs.py` | Flags address-named references in committed sources whose address now has a curated name | .github/workflows/no-rom.yml, Makefile | repo symbol/src paths | LIVE | | give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED | | guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE | -| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE | -| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+9) | repo paths, the registry and signature schemas | LIVE | +| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE | +| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+10) | repo paths, the registry and signature schemas | LIVE | | mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED | | refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE | -| regenerate a splitter-format target disassembly for a function no longer stubbed | `verbatim_target_s.py` | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | decompme_replica.sh | repo build/asm layout | LIVE | +| regenerate a splitter-format target disassembly for a function no longer stubbed | `verbatim_target_s.py` | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | decompme_replica.sh, delever_permute.py | repo build/asm layout | LIVE | | run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | `delever_cycle.sh` | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | delever.py, gte_consolidate.py | the batch size, the aliases, the fleet count | LIVE | | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | `share_body_cycle.sh` | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | — | repo paths, the phase-log format, the fleet count from config/check.*.sha | ORPHAN | +| search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED | | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE | -| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE | +| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE | | turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED | ## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)