mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 21:36:06 -04:00
phase-36: T6 — the kit corpus and the tool index regenerated for delever_permute.py (tool_census --check: 0 gaps)
This commit is contained in:
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded), of which 300 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 20 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 330 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -341,6 +341,7 @@
|
||||
| `verbatim_target_s.py` | regenerate a splitter-format target disassembly for a function no longer stubbed | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | repo build/asm layout |
|
||||
| `delever_cycle.sh` | run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | the batch size, the aliases, the fleet count |
|
||||
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
|
||||
@@ -75,6 +75,7 @@ INFLIGHT = RUN / "inflight.json"
|
||||
PROBE = REPO / ".run" / "P36" / "probe"
|
||||
PRELUDE = "src/shared/engine_prelude.h"
|
||||
REMOVABLE = {("A", "pin"), ("B", "barrier"), ("B", "launder"), ("B", "keepalive"), ("B", "instruction"),
|
||||
("B", "gte-lever"),
|
||||
("C", "cast"), ("C", "decl-body"), ("C", "param"), ("D", "register")}
|
||||
FILE_SCOPE_REMOVABLE = {("C", "decl-file"), ("B", "barrier"), ("B", "launder"), ("B", "keepalive"), ("B", "instruction")}
|
||||
# a file-scope asm statement is a TU-level site: a barrier/launder/keep-alive is judged like any other; a `.section` block is a rodata
|
||||
@@ -343,6 +344,52 @@ def macro_shape(raw, rel, name, use_line):
|
||||
return res
|
||||
|
||||
|
||||
_GTE_CANON = None
|
||||
|
||||
|
||||
_gte_variant_cache = {}
|
||||
|
||||
|
||||
def gte_variant_target(raw, rel, name, use_line):
|
||||
"""The canonical macro name a lever-variant macro use should point at, from the SIGNATURE of the variant's own
|
||||
governing `#define` (never from its spelling: T5 named `gte_rt_m` after Sony's `gte_rt` while that signature's
|
||||
canonical name is `gte_rt_alt`). None when there is no single canonical macro for it."""
|
||||
key = (rel, name, use_line)
|
||||
if key in _gte_variant_cache:
|
||||
return _gte_variant_cache[key]
|
||||
import gte_consolidate as gc # lazy: gte_consolidate imports THIS module
|
||||
gte_canonical_clob("") # loads the table
|
||||
body = None
|
||||
for (l0, l1, n, b) in lc.define_blocks(raw):
|
||||
if n == name and l1 < use_line:
|
||||
body = b
|
||||
inner = lc._macro_asm_inner(body) if body is not None else None
|
||||
names = None
|
||||
if inner is not None:
|
||||
names, _ = gc.canonical_match(gc.signature(inner), _GTE_CANON)
|
||||
res = names[0] if names and len(names) == 1 else None
|
||||
_gte_variant_cache[key] = res
|
||||
return res
|
||||
|
||||
|
||||
def gte_canonical_clob(inner):
|
||||
"""The canonical clobber list for a GTE asm statement's inner text, or None when it has none to take
|
||||
(no table, unsigned template, no canonical entry, or it already carries the canonical set)."""
|
||||
global _GTE_CANON
|
||||
if _GTE_CANON is None:
|
||||
import gte_consolidate as gc # lazy: gte_consolidate imports THIS module
|
||||
_GTE_CANON = json.loads(gc.CANON.read_text()) if gc.CANON.exists() else {}
|
||||
canon = _GTE_CANON.get("canonical") or {}
|
||||
if not canon:
|
||||
return None
|
||||
import gte_consolidate as gc
|
||||
sg = gc.signature(inner)
|
||||
names, clob = gc.canonical_match(sg, _GTE_CANON)
|
||||
if names is None or tuple(sg["clob"]) == tuple(clob):
|
||||
return None
|
||||
return clob
|
||||
|
||||
|
||||
def site_edits(raw, m, ls, site, keep_register=False):
|
||||
"""[(start, end, replacement)] for one site, or raise Refuse. `m` = same_len_mask(raw)."""
|
||||
pos = ls[site["line"] - 1] + site["col"] - 1
|
||||
@@ -381,6 +428,42 @@ def site_edits(raw, m, ls, site, keep_register=False):
|
||||
return [(pos, consume_marker(raw, m, e), new)]
|
||||
if cls == "B" and kind in DEFERRED_KINDS:
|
||||
raise Refuse("asm-body: a whole routine in a C shell is T7's work (DEFERRED)")
|
||||
if cls == "B" and kind == lc.GTE_LEVER_KIND:
|
||||
# A GTE op whose clobber list exceeds its canonical signature's is a SCHEDULING STEER wearing Sony's
|
||||
# coprocessor idiom (T5 named them and marked them; it judged only the macro DEFINITIONS, so a DIRECT
|
||||
# statement's extra clobbers were never offered to the ladder). The rewrite is the canonical clobber
|
||||
# set for that signature — not deletion: the op itself is real code. Refused when the tree has no
|
||||
# canonical table, when the statement does not sign, when its signature is not canonical, or when it
|
||||
# already carries the canonical set (then it is not a lever and the census is wrong about it, R43).
|
||||
# The survivor keeps T5's own richer `// !FAKE:` text: gte-lever is deliberately NOT in MARK_KINDS.
|
||||
name = site.get("via")
|
||||
if name:
|
||||
# a use of a LEVER VARIANT macro (`gte_x_m` / `gte_x_v<hash>`, kept per TU by T5): the lever is the variant's
|
||||
# extra clobbers, so the rewrite points the use at the canonical macro of include/gte_inline.h. The variant's
|
||||
# own `#define` is left dead for `gte_consolidate.py --sweep` (its marker for `--scrub`).
|
||||
if not m.startswith(name, pos):
|
||||
raise Refuse(f"token mismatch at {site['tu']}:{site['line']}: expected `{name}`")
|
||||
base = gte_variant_target(raw, site["tu"], name, site["line"])
|
||||
if base is None:
|
||||
raise Refuse(f"GTE lever `{name}`: no single canonical macro for its signature")
|
||||
if base == name:
|
||||
raise Refuse(f"GTE lever `{name}` already IS its canonical macro")
|
||||
return [(pos, pos + len(name), base)]
|
||||
if not ASM_HEAD.match(m, pos):
|
||||
raise Refuse(f"token mismatch at {site['tu']}:{site['line']}: expected an asm statement")
|
||||
o = m.find("(", pos)
|
||||
c = lc._paren_span(m, o)
|
||||
e = stmt_end(m, pos)
|
||||
if o < 0 or c < 0 or e < 0:
|
||||
raise Refuse("unterminated GTE asm statement")
|
||||
clob = gte_canonical_clob(m[o + 1:c]) # the MASKED inner: T5's `// !FAKE:` sits INSIDE the parens
|
||||
if clob is None:
|
||||
raise Refuse(f"GTE lever `{site.get('detail', '')}`: no canonical signature to take the clobbers from")
|
||||
import gte_consolidate as gc # lazy: gte_consolidate imports THIS module
|
||||
# the statement without its comments (the marker included — a de-levered site owns no honesty marker); set_clobbers
|
||||
# parses the sections positionally and a comment between `(` and the template would derail it
|
||||
clean = "".join(raw[i] for i in range(pos, e) if m[i] == raw[i])
|
||||
return [(pos, consume_marker(raw, m, e), gc.set_clobbers(clean, clob))]
|
||||
if cls == "B" and site.get("via"):
|
||||
name = site["via"]
|
||||
if not m.startswith(name, pos):
|
||||
|
||||
@@ -0,0 +1,714 @@
|
||||
#!/usr/bin/env python3
|
||||
"""delever_permute.py — rung D of the lever ladder: decomp-permuter on the RESIDUE exemplars (Phase 36 T6).
|
||||
|
||||
tools/delever_permute.py --plan [--limit N] [--only X ...] # the exemplar list from the ledger (no writes)
|
||||
tools/delever_permute.py --prepare TU FN # one exemplar's draft.c + target .s, no permuter
|
||||
tools/delever_permute.py --run [--limit K] [--workers W] [--secs S] [--cycles C] [-j J] [--only X ...]
|
||||
tools/delever_permute.py --bank [--label dN] [--only X ...] [--dirty-ok]
|
||||
tools/delever_permute.py --status
|
||||
tools/delever_permute.py --selftest
|
||||
|
||||
WHY A PERMUTER RUNG AT ALL. Rungs A–C (tools/delever.py) asked "does the object stay identical without this site?" and left
|
||||
12,970 bodies where the answer was no for at least one site. Those sites are not magic: each is a shape the compiler would have
|
||||
produced anyway from DIFFERENT C. The permuter's randomizer moves exactly those shapes (temps, statement and declaration order,
|
||||
scoping, operand order), so the residue is its natural population — and the seed is the LEVER-FREE body, which is what makes this
|
||||
different from every earlier permuter run in this project: the search starts from C that has no pins and no asm at all, and any
|
||||
score-0 it reaches is by construction a lever-free match.
|
||||
|
||||
THE PIPELINE PER EXEMPLAR (probe-proven at S98, the T6 log entry):
|
||||
1. the lever-free TU text — delever's own rung-A rewrite of every REMOVABLE site of the body (a site it REFUSES makes the
|
||||
exemplar UNSTRIPPABLE: recorded, skipped — never a silently narrowed scope, R43/silently-narrowed-tool-scope).
|
||||
2. the ISOLATED TU — every OTHER definition replaced by its prototype, every `#include "…/shared/…/func_*.h"` replaced
|
||||
by the prototypes that header defines, INCLUDE_ASM/INCLUDE_RODATA lines and file-scope asm statements dropped: the object
|
||||
the permuter compiles must hold ONE .text function, because the masked scorer compares whole .text.
|
||||
3. cpp -P with the BUILD's own CPPFLAGS + -I<the TU's directory> (common.h, the prelude, engine_types.h and the GTE header
|
||||
expand here — p16_permute.make_base_c runs its own cpp WITHOUT includes and would lose them) -> draft.c.
|
||||
4. the target .s — tools/verbatim_target_s.py regenerates it from the EXTRACTED ROM IMAGE (R34: an independent
|
||||
oracle; our own source is the thing under test), into the exemplar's scratch dir, where p16_permute.setup reads it.
|
||||
5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds
|
||||
(pins -> regalloc, barriers/launders/keep-alives -> schedule, mixed -> regalloc).
|
||||
6. a score-0 winner is a CANDIDATE, never a bank: --bank puts the function's definition back through
|
||||
`delever.py --apply-body` (which refuses a body that still carries a class A/B lever and judges it on the object's bytes
|
||||
through every recipe of the TU), then re-folds the cpp-expanded GTE asm with `gte_consolidate.py --apply --rejudge`.
|
||||
The clean fleet run (R22) gates the batch, as it gates every batch of this phase.
|
||||
|
||||
SCRATCH IS KEYED BY ALIAS+FN (R48): `.run/P36/permuter/<alias>__<fn>/` — a function NAME is not unique across the fleet (the
|
||||
overlays overlap in RAM, so two different bodies can both be `func_8013B274`), and the permuter's own scratch/winner paths are
|
||||
keyed by the bare name. Every attempt is recorded in `.run/P36/permuter/outcomes.jsonl`, which is also the skip list.
|
||||
|
||||
NEVER RUN THE CAMPAIGN AS A HARNESS BACKGROUND TASK (the low-memory guard kills it): `setsid nohup … &` + a Monitor on the log.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "tools"))
|
||||
import delever as dl # noqa: E402
|
||||
import delever_oracle as oracle # noqa: E402
|
||||
import lever_census as lc # noqa: E402
|
||||
import p16_permute as p16 # noqa: E402
|
||||
import share_census as sc # noqa: E402
|
||||
|
||||
RUN = REPO / ".run" / "P36" / "permuter"
|
||||
OUTCOMES = RUN / "outcomes.jsonl"
|
||||
PY = str(REPO / ".venv" / "bin" / "python")
|
||||
CPP = "mipsel-linux-gnu-cpp"
|
||||
# the Makefile's own CPPFLAGS (line 1010) — the draft must see the same macro world the build sees
|
||||
CPPFLAGS = ["-lang-c", "-Iinclude", "-undef", "-Wall", "-fno-builtin", "-Dmips", "-D__GNUC__=2",
|
||||
"-D__OPTIMIZE__", "-Dpsx", "-D_PSYQ", "-D_MIPSEL", "-D_LANGUAGE_C"]
|
||||
# decomp-permuter parses base.c with pycparser, which rejects `__attribute__((packed, aligned(1)))` outright
|
||||
# ("Syntax error in base.c" -> the run REFUSES and permutes nothing). The draft therefore defines the keyword away.
|
||||
# This is a fidelity LOSS, not a correctness one: a packed struct the exemplar actually uses will lay out differently
|
||||
# in the search, so that exemplar simply cannot reach score 0 — and the bank gate is the real object oracle either
|
||||
# way. The count is recorded per attempt (`attrs`) so the yield report can say whether it mattered (76 sites in the
|
||||
# whole tree at T5; the class belongs to the types phase, R95).
|
||||
CPP_ATTR = "-D__attribute__(x)="
|
||||
INCLUDE_ASM_LINE = re.compile(r"^[ \t]*INCLUDE_(?:ASM|RODATA)\b[^\n]*\n", re.M)
|
||||
INCLUDE_LINE = re.compile(r'^[ \t]*#[ \t]*include[ \t]*"([^"]+)"[^\n]*\n', re.M)
|
||||
SCHED_KINDS = {"barrier", "launder", "keepalive"}
|
||||
_LOCK = threading.Lock()
|
||||
|
||||
|
||||
class Unstrippable(Exception):
|
||||
"""the body cannot be made lever-free by the rewrite table — the exemplar is T7's, not rung D's."""
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# the population: the residue's classes, one exemplar each
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
def load_outcomes():
|
||||
rows = []
|
||||
if OUTCOMES.exists():
|
||||
for l in OUTCOMES.read_text().splitlines():
|
||||
if l.strip():
|
||||
rows.append(json.loads(l))
|
||||
return rows
|
||||
|
||||
|
||||
def outcome_append(row):
|
||||
RUN.mkdir(parents=True, exist_ok=True)
|
||||
with _LOCK, open(OUTCOMES, "a") as f:
|
||||
f.write(json.dumps(row) + "\n")
|
||||
|
||||
|
||||
def alias_of(tu, row=None):
|
||||
"""the binary whose object holds this TU (a shared header: its first includer's). The ledger row's own aliases first."""
|
||||
al = (row or {}).get("aliases")
|
||||
if al:
|
||||
return al[0]
|
||||
by_src = oracle.recipes_by_src(oracle.load_recipes()["recipes"])
|
||||
src = tu
|
||||
if tu.endswith(".h"):
|
||||
inc = dl.includers().get(tu) or []
|
||||
src = next((t for t in inc if t in by_src), None)
|
||||
if src is None:
|
||||
return None
|
||||
recs = by_src.get(src) or []
|
||||
return recs[0]["alias"] if recs else None
|
||||
|
||||
|
||||
def matches(ex, only):
|
||||
if not only:
|
||||
return True
|
||||
return any(o == ex["fn"] or o == ex["tu"] or o in ex["tu"] or ex["nhash"].startswith(o) or o == ex["alias"] for o in only)
|
||||
|
||||
|
||||
def exemplars(only=(), limit=None, include_done=False):
|
||||
"""One exemplar per RESIDUE text class, largest class first (a match banks every copy), then fewest NEEDED sites."""
|
||||
cur, order = {}, []
|
||||
for r in dl.load_ledger():
|
||||
k = (r.get("tu"), r.get("fn"))
|
||||
if not k[0] or not k[1]:
|
||||
continue
|
||||
if k not in cur:
|
||||
order.append(k)
|
||||
cur[k] = r
|
||||
groups = collections.OrderedDict()
|
||||
for k in order:
|
||||
r = cur[k]
|
||||
if r.get("verdict") != "RESIDUE" or r["fn"] == dl.FILE_SCOPE_FN:
|
||||
continue # a TU's file-scope `volatile` pseudo-body is not a function to permute
|
||||
groups.setdefault(r.get("nhash_after") or r.get("nhash_before"), []).append((k, r))
|
||||
done = set() if include_done else {o["nhash"] for o in load_outcomes() if o.get("kind") == "attempt"}
|
||||
out = []
|
||||
for nh, members in groups.items():
|
||||
if nh in done:
|
||||
continue
|
||||
(tu, fn), r = members[0]
|
||||
needed = [s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]
|
||||
out.append(dict(nhash=nh, tu=tu, fn=fn, alias=(r.get("aliases") or [None])[0], copies=len(members),
|
||||
needed=len(needed), kinds=sorted({s["kind"] for s in needed}),
|
||||
members=[dict(tu=t, fn=f) for (t, f), _ in members], row=r))
|
||||
out.sort(key=lambda e: (-e["copies"], e["needed"], e["tu"], e["fn"]))
|
||||
out = [e for e in out if matches(e, only)]
|
||||
return out[:limit] if limit else out
|
||||
|
||||
|
||||
def klass_for(kinds):
|
||||
"""the permuter_weights profile for a residue's NEEDED-site mix (the profile NAME is accepted verbatim by classify())."""
|
||||
ks = set(kinds)
|
||||
if ks and ks <= SCHED_KINDS:
|
||||
return "schedule"
|
||||
return "regalloc"
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# step 1-3: the lever-free, isolated, preprocessed draft
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
_sites_cache = None
|
||||
|
||||
|
||||
def tu_sites(tu):
|
||||
global _sites_cache
|
||||
if _sites_cache is None:
|
||||
_sites_cache = collections.defaultdict(list)
|
||||
for s in dl.load_sites():
|
||||
_sites_cache[s["tu"]].append(s)
|
||||
return _sites_cache[tu]
|
||||
|
||||
|
||||
def body_sites(tu, fn):
|
||||
return [s for s in tu_sites(tu) if s.get("fn") == fn]
|
||||
|
||||
|
||||
def file_asm_sites(tu):
|
||||
"""the TU's file-scope asm STATEMENTS (13 in the fleet: 7 rodata carves, 6 barriers) — asked of the census, never
|
||||
re-parsed: a bare `__asm__(` scan cannot tell a statement from the asm-LABEL clause of a declaration
|
||||
(`extern void func_8005C324(…) __asm__("memcpy");` — 6,423 of those), and eating one leaves a headless K&R body."""
|
||||
return [s for s in tu_sites(tu) if s.get("scope") == "file" and s["cls"] == "B"
|
||||
and s["kind"] not in lc.NON_LEVER_KINDS and s["kind"] != "asm-label"]
|
||||
|
||||
|
||||
def lever_free_text(tu, raw, fn, sites, strip=True):
|
||||
"""the TU's text with every REMOVABLE site of THIS body rewritten away (delever's own table) and its file-scope asm
|
||||
statements deleted (the draft compares .text only); Unstrippable if any body site refuses.
|
||||
strip=False keeps every lever — that is the CALIBRATION seed: the body as the tree has it, which must score MATCH
|
||||
against the regenerated target or the harness (target, isolation, cpp, compile) is not measuring this function."""
|
||||
m = dl.same_len_mask(raw)
|
||||
ls = dl.line_starts(raw)
|
||||
edits, refused = [], []
|
||||
for s in file_asm_sites(tu):
|
||||
pos = ls[s["line"] - 1] + s["col"] - 1
|
||||
e = dl.stmt_end(m, pos)
|
||||
if dl.ASM_HEAD.match(m, pos) and e > 0:
|
||||
edits.append((pos, dl.consume_marker(raw, m, e), ""))
|
||||
for s in (sites if strip else []):
|
||||
if (s["cls"], s["kind"]) not in dl.REMOVABLE:
|
||||
if s["kind"] in dl.DEFERRED_KINDS:
|
||||
refused.append((s["kind"], s["line"], "asm-body: T7's work"))
|
||||
continue
|
||||
try:
|
||||
edits += dl.site_edits(raw, m, ls, s)
|
||||
except dl.Refuse as ex:
|
||||
refused.append((s["kind"], s["line"], str(ex)[:120]))
|
||||
if refused:
|
||||
raise Unstrippable(refused)
|
||||
if not edits:
|
||||
return raw
|
||||
try:
|
||||
return dl.apply_edits(raw, edits)
|
||||
except dl.Refuse as ex:
|
||||
raise Unstrippable([("<combination>", 0, str(ex)[:120])])
|
||||
|
||||
|
||||
def prototype_at(text, m, start):
|
||||
"""`<head>;` for the definition that starts at index `start` — the head up to the last `)` before its opening `{`."""
|
||||
o = m.find("{", start)
|
||||
if o < 0:
|
||||
return None
|
||||
close = m.rfind(")", start, o)
|
||||
if close < 0:
|
||||
return None
|
||||
head = " ".join(text[start:close + 1].split())
|
||||
return head + ";"
|
||||
|
||||
|
||||
def header_prototypes(path):
|
||||
"""the prototypes of every function a shared header DEFINES (a static one keeps `static`)."""
|
||||
p = REPO / path
|
||||
if not p.exists():
|
||||
return []
|
||||
text = p.read_text(errors="surrogateescape")
|
||||
m = dl.same_len_mask(text)
|
||||
ls = dl.line_starts(text)
|
||||
out = []
|
||||
for r in sc.scan_text(text, path, shared_defs=None):
|
||||
if r["form"] != "def":
|
||||
continue
|
||||
pr = prototype_at(text, m, ls[r["line"] - 1])
|
||||
if pr:
|
||||
out.append(pr)
|
||||
return out
|
||||
|
||||
|
||||
def isolate(tu, text, fn):
|
||||
"""every OTHER definition -> its prototype; shared-header includes -> their prototypes; INCLUDE_ASM/file-scope asm dropped."""
|
||||
m = dl.same_len_mask(text)
|
||||
ls = dl.line_starts(text)
|
||||
recs = sc.scan_text(text, tu, shared_defs=None)
|
||||
defs = [r for r in recs if r["form"] == "def"]
|
||||
if not any(r["name"] == fn for r in defs):
|
||||
raise Unstrippable([("<isolate>", 0, f"{fn} is not defined in {tu}")])
|
||||
edits = []
|
||||
for r in defs:
|
||||
s, e = ls[r["line"] - 1], ls[r["end"]]
|
||||
|
||||
if r["name"] == fn:
|
||||
continue
|
||||
pr = prototype_at(text, m, s)
|
||||
if pr is None:
|
||||
raise Unstrippable([("<isolate>", r["line"], f"no prototype for {r['name']}")])
|
||||
edits.append((s, e, pr + "\n"))
|
||||
for mm in INCLUDE_ASM_LINE.finditer(m):
|
||||
edits.append((mm.start(), mm.end(), ""))
|
||||
for mm in INCLUDE_LINE.finditer(m):
|
||||
inc = mm.group(1)
|
||||
rel = os.path.normpath(os.path.join(os.path.dirname(tu), inc))
|
||||
if not rel.startswith("src/shared/"):
|
||||
continue
|
||||
pr = header_prototypes(rel)
|
||||
if pr: # a macro-only header (the prelude) has none: keep the include
|
||||
edits.append((mm.start(), mm.end(), "\n".join(pr) + "\n"))
|
||||
return dl.apply_edits(text, sorted(edits, key=lambda x: x[0]))
|
||||
|
||||
|
||||
def drop_file_scope_asm(text, tu):
|
||||
"""delete every asm STATEMENT that is not inside a function — after cpp, `include/include_asm.h` has injected
|
||||
`__asm__(".include \\"include/labels.inc\\"");` at the top of every draft, and the permuter's own compile.sh
|
||||
already prepends `.include "macro.inc"`, so the assembler dies on `Macro 'glabel' was already defined`.
|
||||
An asm-LABEL clause (`extern void f(…) __asm__("memcpy");`, 6,423 of them) is NOT a statement: it follows a
|
||||
declarator, so the preceding non-space character is not `;` or `}`."""
|
||||
m = dl.same_len_mask(text)
|
||||
ls = dl.line_starts(text)
|
||||
spans = [(ls[r["line"] - 1], ls[r["end"]]) for r in sc.scan_text(text, tu, shared_defs=None) if r["form"] == "def"]
|
||||
edits = []
|
||||
for mm in dl.ASM_HEAD.finditer(m):
|
||||
p = mm.start()
|
||||
if any(s <= p < e for s, e in spans):
|
||||
continue
|
||||
prev = m[:p].rstrip()
|
||||
if prev and prev[-1] not in ";}":
|
||||
continue # an asm-label clause on a declaration
|
||||
e = dl.stmt_end(m, p)
|
||||
if e > 0:
|
||||
edits.append((p, e, ""))
|
||||
return dl.apply_edits(text, edits) if edits else text
|
||||
|
||||
|
||||
def cpp_expand(tu, text):
|
||||
"""the build's cpp over the isolated text (fed on stdin, so `-I<the TU's directory>` is what resolves its relative includes)."""
|
||||
p = subprocess.run([CPP, "-P"] + CPPFLAGS + [CPP_ATTR, "-I" + os.path.dirname(tu), "-"],
|
||||
input=text, capture_output=True, text=True, cwd=REPO)
|
||||
if p.returncode or not p.stdout.strip():
|
||||
raise Unstrippable([("<cpp>", 0, (p.stderr.strip().splitlines() or ["empty output"])[-1][:160])])
|
||||
return p.stdout
|
||||
|
||||
|
||||
def scratch_of(alias, fn):
|
||||
return RUN / f"{alias}__{fn}"
|
||||
|
||||
|
||||
def prepare(ex, quiet=False):
|
||||
"""(scratch dir, draft path, target path, klass) for one exemplar — the whole probe-2 pipeline."""
|
||||
tu, fn = ex["tu"], ex["fn"]
|
||||
alias = ex["alias"] or alias_of(tu, ex.get("row"))
|
||||
if not alias:
|
||||
raise Unstrippable([("<alias>", 0, f"no binary compiles {tu}")])
|
||||
ex["alias"] = alias
|
||||
raw = (REPO / tu).read_text(errors="surrogateescape")
|
||||
sites = body_sites(tu, fn)
|
||||
if not sites:
|
||||
raise Unstrippable([("<census>", 0, f"the census has no site in {tu}:{fn} (stale? rerun lever_census --sites)")])
|
||||
d = scratch_of(alias, fn)
|
||||
if d.exists():
|
||||
shutil.rmtree(d)
|
||||
(d / "gas").mkdir(parents=True)
|
||||
(d / "splat").mkdir()
|
||||
for strip, name in ((True, "draft.c"), (False, "levered.c")):
|
||||
free = lever_free_text(tu, raw, fn, sites, strip=strip)
|
||||
iso = isolate(tu, free, fn)
|
||||
text = drop_file_scope_asm(cpp_expand(tu, iso), tu)
|
||||
if strip:
|
||||
(d / "tu.c").write_text(free, errors="surrogateescape") # the lever-free TU (what rung A would have written)
|
||||
(d / "iso.c").write_text(iso, errors="surrogateescape") # + one body, the rest prototypes (pre-cpp; for reading)
|
||||
(d / name).write_text(text, errors="surrogateescape")
|
||||
if strip:
|
||||
base = p16.make_base_c(text)
|
||||
if not p16.defines_fn(base, fn):
|
||||
raise Unstrippable([("<base>", 0, f"base.c would lose the definition of {fn} (inspect {d}/draft.c)")])
|
||||
# TWO target forms, for the two instruments. --gas is the ASSEMBLABLE one ($-registers, .L labels, symbol names) that
|
||||
# `p16_permute.setup` turns into target.o; the default splat listing is the WORD oracle `match_one.py` reads. Nothing
|
||||
# ever assembles the splat form — pasting it into decomp.me was P34's `invalid operands 'li a2,2'` (R98/DK-81), and
|
||||
# `mipsel-as` refuses its bare `addiu sp,sp,-152` the same way.
|
||||
for args, sub, name in (( ["--gas"], "gas", f"{fn}.gas.s"), ([], "splat", f"{fn}.s")):
|
||||
r = subprocess.run([PY, "tools/verbatim_target_s.py", "--binary", alias, "--fn", fn] + args
|
||||
+ ["--out", str(d / "tgt")], capture_output=True, text=True, cwd=REPO)
|
||||
tgt = d / "tgt" / alias / name
|
||||
if r.returncode or not tgt.exists():
|
||||
raise Unstrippable([("<target>", 0, ((r.stderr or r.stdout).strip().splitlines() or ["no listing"])[-1][:160])])
|
||||
shutil.copy(tgt, d / sub / f"{fn}.s") # p16_permute.setup / match_one.py read <dir>/<fn>.s
|
||||
if not quiet:
|
||||
n = len((d / "draft.c").read_text(errors="surrogateescape").splitlines())
|
||||
print(f" prepared {alias}__{fn}: draft {n} lines, target {(r.stdout or '').strip().split()[-3:]}", flush=True)
|
||||
return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"])
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# the calibration (R39/R56): the harness must agree with the tree before it may judge a candidate
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
MISMATCH_RE = re.compile(r"(\d+) mismatched")
|
||||
|
||||
|
||||
def closeness(d, fn, c):
|
||||
"""(mismatched instructions, first line) from tools/match_one.py against the SPLAT listing — 0 == byte-identical
|
||||
(relocation-masked). None when the tool could not judge (R61: not judged is not a verdict)."""
|
||||
r = subprocess.run([PY, "tools/match_one.py", fn, "--c", str(c), "--asm-subdir", (d / "splat").as_posix()],
|
||||
capture_output=True, text=True, cwd=REPO)
|
||||
first = ((r.stdout or "") + (r.stderr or "")).strip().splitlines()
|
||||
first = first[0] if first else "(no output)"
|
||||
if "MATCH" in first and "mismatched" not in first:
|
||||
return 0, first
|
||||
mm = MISMATCH_RE.search(first)
|
||||
return (int(mm.group(1)) if mm else None), first
|
||||
|
||||
|
||||
def calibrate_one(ex, quiet=False):
|
||||
"""the control: the body AS THE TREE HAS IT (levers and all) must be MATCH against the regenerated target. It proves the
|
||||
whole chain — the target from the ROM image, the isolation, the cpp expansion, the pinned triple — is measuring THIS
|
||||
function. Then the lever-free draft's distance is the search's real starting point, and it is worth recording (R37)."""
|
||||
d, draft, tgt, klass = prepare(ex, quiet=quiet)
|
||||
lev, lev_line = closeness(d, ex["fn"], d / "levered.c")
|
||||
free, free_line = closeness(d, ex["fn"], draft)
|
||||
ok = (lev == 0)
|
||||
if not quiet:
|
||||
print(f" {'OK ' if ok else 'FAIL'} {ex['alias']}__{ex['fn']}: levered {lev_line[:60]} · lever-free {free_line[:60]}",
|
||||
flush=True)
|
||||
return dict(ok=ok, levered=lev, free=free, levered_line=lev_line, free_line=free_line, klass=klass, d=d)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# step 5: the permuter
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
BEST_RE = re.compile(r"best score = (\d+)")
|
||||
|
||||
|
||||
def run_one(ex, secs, cycles, j, max_start=None):
|
||||
t0 = time.time()
|
||||
row = dict(kind="attempt", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=ex["nhash"], tu=ex["tu"], fn=ex["fn"],
|
||||
alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"],
|
||||
secs=secs, cycles=cycles, j=j)
|
||||
try:
|
||||
d, draft, tgt, klass = prepare(ex)
|
||||
except Unstrippable as ex_:
|
||||
row.update(verdict="UNSTRIPPABLE", why=ex_.args[0][:6], seconds=round(time.time() - t0, 1))
|
||||
outcome_append(row)
|
||||
print(f" {ex['alias'] or '?'}__{ex['fn']}: UNSTRIPPABLE — {ex_.args[0][:2]}", flush=True)
|
||||
return row
|
||||
# EVERY attempt calibrates itself first (R56): the body as the tree has it must be MATCH against the regenerated
|
||||
# target, or this harness is not measuring this function and its budget would buy a meaningless number. ~15 s
|
||||
# against ~12 min of search.
|
||||
lev, lev_line = closeness(d, ex["fn"], d / "levered.c")
|
||||
free, free_line = closeness(d, ex["fn"], draft)
|
||||
row.update(levered=lev, free=free, levered_line=lev_line[:120], free_line=free_line[:120])
|
||||
if lev != 0:
|
||||
row.update(verdict="UNCALIBRATED", seconds=round(time.time() - t0, 1))
|
||||
outcome_append(row)
|
||||
print(f" {ex['alias']}__{ex['fn']}: UNCALIBRATED — the LEVERED body is not MATCH ({lev_line[:80]})", flush=True)
|
||||
return row
|
||||
if max_start is not None and (free is None or free > max_start):
|
||||
# TRIAGE, not a verdict about the function: a lever-free body this far from the target (usually because removing a
|
||||
# hand-placed instruction changed the instruction COUNT and shifted everything after it) will not close in a
|
||||
# 12-minute search. Recorded with its distance; `--include-done` draws it again when the budget or the rung grows.
|
||||
row.update(verdict="FAR", seconds=round(time.time() - t0, 1))
|
||||
outcome_append(row)
|
||||
print(f" {ex['alias']}__{ex['fn']}: FAR start={free} > {max_start} — not searched ({ex['copies']} copies)", flush=True)
|
||||
return row
|
||||
rel = d.relative_to(REPO).as_posix()
|
||||
log = d / "ils.log"
|
||||
cmd = [PY, "tools/permuter_ils.py", ex["fn"], "--draft", (draft.relative_to(REPO)).as_posix(),
|
||||
"--asm-subdir", rel + "/gas", "--klass", klass, "--cycles", str(cycles), "--secs", str(secs),
|
||||
"--j", str(j), "--winners", rel, "--pd", rel + "/pd"]
|
||||
row["klass"] = klass
|
||||
row["cmd"] = " ".join(cmd)
|
||||
rc = None
|
||||
with open(log, "w") as f:
|
||||
f.write(" ".join(cmd) + "\n\n")
|
||||
f.flush()
|
||||
try:
|
||||
rc = subprocess.run(cmd, cwd=REPO, stdout=f, stderr=subprocess.STDOUT,
|
||||
timeout=cycles * (secs + 45) + 300).returncode
|
||||
except subprocess.TimeoutExpired:
|
||||
f.write("\n[delever_permute] the ILS wrapper overran its own budget — killed\n")
|
||||
out = log.read_text(errors="replace")
|
||||
scores = [int(x) for x in BEST_RE.findall(out)]
|
||||
win = d / f"{ex['fn']}.c"
|
||||
# NOT JUDGED IS NOT A VERDICT (R61): a setup failure, a parser refusal or a wrapper that never iterated is its own
|
||||
# outcome — reporting it as NO-MATCH is how a tool reports work it never did.
|
||||
if win.exists():
|
||||
verdict = "MATCH"
|
||||
elif "setup FAILED" in out:
|
||||
verdict = "SETUP-FAILED"
|
||||
elif "ABORTED" in out or "REFUSED" in out:
|
||||
verdict = "ABORTED"
|
||||
elif "cycle 1:" not in out:
|
||||
verdict = "NOT-JUDGED"
|
||||
else:
|
||||
verdict = "NO-MATCH"
|
||||
row.update(best=(min(scores) if scores else None), seconds=round(time.time() - t0, 1), rc=rc,
|
||||
winner=(win.relative_to(REPO).as_posix() if win.exists() else None), verdict=verdict)
|
||||
outcome_append(row)
|
||||
print(f" {ex['alias']}__{ex['fn']}: {row['verdict']} start={row['free']} best={row['best']} "
|
||||
f"({row['seconds']}s, {ex['copies']} copies, {ex['needed']} needed, {klass})", flush=True)
|
||||
return row
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# step 6: the bank (serial — every step here runs `make`)
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
def winner_body(winner_c, fn, tu):
|
||||
"""the winner's definition of fn, as the text `delever --apply-body` splices in."""
|
||||
draft = p16.winner_to_draft(pathlib.Path(winner_c).read_text(errors="surrogateescape"))
|
||||
ls = dl.line_starts(draft)
|
||||
for r in sc.scan_text(draft, tu, shared_defs=None):
|
||||
if r["form"] == "def" and r["name"] == fn:
|
||||
return draft[ls[r["line"] - 1]:ls[r["end"]]]
|
||||
return None
|
||||
|
||||
|
||||
def bank(a):
|
||||
outs = load_outcomes()
|
||||
banked = {(o["alias"], o["fn"], o["nhash"]) for o in outs if o.get("kind") == "bank" and o.get("applied")}
|
||||
todo = [o for o in outs if o.get("kind") == "attempt" and o.get("verdict") == "MATCH"
|
||||
and (o["alias"], o["fn"], o["nhash"]) not in banked
|
||||
and (not a.only or any(x in (o["fn"], o["tu"], o["alias"]) or o["nhash"].startswith(x) for x in a.only))]
|
||||
if not todo:
|
||||
print("delever_permute --bank: no unbanked winner (R68: nothing to do, loudly)")
|
||||
return 1
|
||||
ok, bad = 0, 0
|
||||
for o in todo:
|
||||
d = scratch_of(o["alias"], o["fn"])
|
||||
body = winner_body(REPO / o["winner"], o["fn"], o["tu"]) if o.get("winner") else None
|
||||
if not body:
|
||||
print(f" {o['alias']}__{o['fn']}: the winner holds no definition of {o['fn']} — SKIPPED", flush=True)
|
||||
bad += 1
|
||||
continue
|
||||
bf = d / "body.c"
|
||||
bf.write_text(body, errors="surrogateescape")
|
||||
cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(),
|
||||
"--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else [])
|
||||
r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
|
||||
print(" " + (r.stdout or r.stderr).strip().splitlines()[-1][:220], flush=True)
|
||||
row = dict(kind="bank", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=o["nhash"], tu=o["tu"], fn=o["fn"],
|
||||
alias=o["alias"], label=a.label, applied=(r.returncode == 0),
|
||||
apply_out=(r.stdout or r.stderr).strip()[-400:])
|
||||
if r.returncode == 0:
|
||||
g = subprocess.run([PY, "tools/gte_consolidate.py", "--apply", "--only", o["tu"], "--rejudge",
|
||||
"--label", a.label + "g"], cwd=REPO, capture_output=True, text=True)
|
||||
row["gte_refold"] = (g.stdout or g.stderr).strip()[-300:]
|
||||
ok += 1
|
||||
else:
|
||||
bad += 1
|
||||
outcome_append(row)
|
||||
print(f"delever_permute --bank: {ok} applied, {bad} rejected of {len(todo)} — "
|
||||
f"now the clean fleet run (R22) and one commit per batch (R42)")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# the commands
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
def cmd_plan(a):
|
||||
ex = exemplars(a.only, a.limit, a.include_done)
|
||||
tot = exemplars(include_done=True)
|
||||
kinds = collections.Counter(k for e in ex for k in e["kinds"])
|
||||
print(f"delever_permute --plan: {len(ex)} exemplars drawable of {len(tot)} residue classes "
|
||||
f"({sum(e['copies'] for e in ex):,} bodies behind them); NEEDED kinds {dict(kinds)}")
|
||||
for e in ex[:a.show]:
|
||||
print(f" {e['copies']:4d} copies · {e['needed']:2d} needed {','.join(e['kinds']):<28} "
|
||||
f"{klass_for(e['kinds']):8} {e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}")
|
||||
if len(ex) > a.show:
|
||||
print(f" … {len(ex) - a.show} more")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_run(a):
|
||||
dl.ensure_census(a.j)
|
||||
ex = exemplars(a.only, a.limit, a.include_done)
|
||||
if not ex:
|
||||
print("delever_permute --run: no drawable exemplar (every class attempted, or --only matched none)")
|
||||
return 1
|
||||
per = max(1, a.j // max(1, a.workers))
|
||||
print(f"delever_permute --run: {len(ex)} exemplars, {a.workers} at a time, {a.cycles} cycles x {a.secs}s @ -j{per} each "
|
||||
f"(budget ≈ {len(ex) / a.workers * a.cycles * a.secs / 3600:.1f} h)", flush=True)
|
||||
t0 = time.time()
|
||||
rows = []
|
||||
with ThreadPoolExecutor(max_workers=a.workers) as pool:
|
||||
for r in pool.map(lambda e: run_one(e, a.secs, a.cycles, per, a.max_start), ex):
|
||||
rows.append(r)
|
||||
won = [r for r in rows if r["verdict"] == "MATCH"]
|
||||
v = collections.Counter(r["verdict"] for r in rows)
|
||||
print(f"\npermuter: {len(won)} of {len(rows)} exemplars matched lever-free in {(time.time() - t0) / 3600:.2f} h "
|
||||
f"({sum(r['copies'] for r in won):,} of {sum(r['copies'] for r in rows):,} bodies behind them) — "
|
||||
+ " · ".join(f"{k} {n}" for k, n in v.most_common()))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_calibrate(a):
|
||||
ex = exemplars(a.only, a.limit or 6, include_done=True)
|
||||
print(f"delever_permute --calibrate: {len(ex)} exemplars — the levered body must be MATCH, the lever-free one is the "
|
||||
f"search's starting distance", flush=True)
|
||||
rows, dist = [], []
|
||||
for e in ex:
|
||||
try:
|
||||
r = calibrate_one(e)
|
||||
except Unstrippable as u:
|
||||
print(f" SKIP {e['tu']}:{e['fn']} — {u.args[0][:2]}", flush=True)
|
||||
continue
|
||||
rows.append(r)
|
||||
if r["ok"] and r["free"] is not None:
|
||||
dist.append(r["free"])
|
||||
good = sum(1 for r in rows if r["ok"])
|
||||
print(f"\ncalibration: {good} of {len(rows)} levered bodies MATCH their regenerated target"
|
||||
+ (f" · lever-free distance min {min(dist)} / median {sorted(dist)[len(dist) // 2]} / max {max(dist)} "
|
||||
f"over {len(dist)} exemplars" if dist else ""))
|
||||
(RUN / "calibration.json").write_text(json.dumps(
|
||||
[dict(tu=e["tu"], fn=e["fn"], alias=e["alias"], ok=r["ok"], levered=r["levered"], free=r["free"],
|
||||
levered_line=r["levered_line"], free_line=r["free_line"])
|
||||
for e, r in zip(ex, rows)], indent=1))
|
||||
return 0 if good == len(rows) and rows else 1
|
||||
|
||||
|
||||
def cmd_status(a):
|
||||
outs = load_outcomes()
|
||||
att = [o for o in outs if o.get("kind") == "attempt"]
|
||||
bk = [o for o in outs if o.get("kind") == "bank"]
|
||||
left = exemplars()
|
||||
v = collections.Counter(o.get("verdict") for o in att)
|
||||
print(f"delever_permute --status: {len(att)} attempts {dict(v)} · {sum(1 for b in bk if b.get('applied'))} banked of "
|
||||
f"{len(bk)} tried · {len(left)} residue classes not yet attempted "
|
||||
f"({sum(e['copies'] for e in left):,} bodies)")
|
||||
for o in att:
|
||||
if o.get("verdict") == "MATCH":
|
||||
print(f" MATCH {o['alias']}__{o['fn']} ({o['copies']} copies, best={o.get('best')}, {o.get('seconds')}s)")
|
||||
return 0
|
||||
|
||||
|
||||
FIX_TU = """#include "common.h"
|
||||
extern s32 D_800A0000;
|
||||
s32 helper_one(s32 a0)
|
||||
{
|
||||
return a0 + D_800A0000;
|
||||
}
|
||||
INCLUDE_ASM("asm/x/nonmatchings/x", func_dead);
|
||||
s32 target_fn(s32 a0, s32 *a1)
|
||||
{
|
||||
register s32 s __asm__("$16") = a0;
|
||||
__asm__ __volatile__("" ::: "memory");
|
||||
*a1 = helper_one(s);
|
||||
return s;
|
||||
}
|
||||
static void tail(void) { }
|
||||
"""
|
||||
|
||||
|
||||
def cmd_selftest(a):
|
||||
ok = True
|
||||
d = REPO / ".run" / "P36" / "permuter" / "_selftest"
|
||||
if d.exists():
|
||||
shutil.rmtree(d)
|
||||
d.mkdir(parents=True)
|
||||
tu = "src/selftest/fix.c"
|
||||
sites = [dict(tu=tu, fn="target_fn", cls="A", kind="pin", detail="$16", via="", line=10, col=5,
|
||||
fn_line=9, fn_end=15, zero=False),
|
||||
dict(tu=tu, fn="target_fn", cls="B", kind="barrier", detail="", via="", line=11, col=5,
|
||||
fn_line=9, fn_end=15, zero=False)]
|
||||
free = lever_free_text(tu, FIX_TU, "target_fn", sites)
|
||||
if "__asm__" in free or "register" in free:
|
||||
print(f"selftest: the lever-free text still carries a lever:\n{free}")
|
||||
ok = False
|
||||
iso = isolate(tu, free, "target_fn")
|
||||
if "helper_one(s32 a0);" not in " ".join(iso.split()):
|
||||
print(f"selftest: helper_one was not reduced to a prototype:\n{iso}")
|
||||
ok = False
|
||||
if "INCLUDE_ASM" in iso:
|
||||
print("selftest: the INCLUDE_ASM line survived isolation")
|
||||
ok = False
|
||||
if iso.count("{") != 1:
|
||||
print(f"selftest: {iso.count('{')} bodies left in the isolated TU (want 1)")
|
||||
ok = False
|
||||
if "static void tail(void);" not in " ".join(iso.split()):
|
||||
print("selftest: a static definition lost its `static` (or its prototype)")
|
||||
ok = False
|
||||
if klass_for(["pin"]) != "regalloc" or klass_for(["barrier", "launder"]) != "schedule" \
|
||||
or klass_for(["pin", "barrier"]) != "regalloc" or klass_for([]) != "regalloc":
|
||||
print("selftest: klass_for is wrong")
|
||||
ok = False
|
||||
base = p16.make_base_c(iso)
|
||||
if not p16.defines_fn(base, "target_fn"):
|
||||
print("selftest: base.c lost target_fn")
|
||||
ok = False
|
||||
shutil.rmtree(d)
|
||||
print(f"delever_permute --selftest: {'OK' if ok else 'FAILED'} — lever-free + isolation + prototypes + klass + base.c")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
g = ap.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--plan", action="store_true")
|
||||
g.add_argument("--prepare", nargs=2, metavar=("TU", "FN"))
|
||||
g.add_argument("--calibrate", action="store_true",
|
||||
help="the control: N exemplars' levered bodies must all be MATCH against their regenerated targets")
|
||||
g.add_argument("--run", action="store_true")
|
||||
g.add_argument("--bank", action="store_true")
|
||||
g.add_argument("--status", action="store_true")
|
||||
g.add_argument("--selftest", action="store_true")
|
||||
ap.add_argument("--only", nargs="*", default=[], help="fn, TU (substring), alias or nhash prefix")
|
||||
ap.add_argument("--limit", type=int)
|
||||
ap.add_argument("--show", type=int, default=25)
|
||||
ap.add_argument("--workers", type=int, default=4)
|
||||
ap.add_argument("--secs", type=int, default=240)
|
||||
ap.add_argument("--cycles", type=int, default=3)
|
||||
ap.add_argument("-j", type=int, default=16)
|
||||
ap.add_argument("--label", default="d1")
|
||||
ap.add_argument("--dirty-ok", action="store_true")
|
||||
ap.add_argument("--max-start", type=int,
|
||||
help="skip (as FAR) an exemplar whose lever-free body is further than N instructions from the target")
|
||||
ap.add_argument("--include-done", action="store_true", help="draw classes that already have an outcome row")
|
||||
a = ap.parse_args()
|
||||
os.chdir(REPO)
|
||||
RUN.mkdir(parents=True, exist_ok=True)
|
||||
if a.selftest:
|
||||
return cmd_selftest(a)
|
||||
if a.plan:
|
||||
return cmd_plan(a)
|
||||
if a.status:
|
||||
return cmd_status(a)
|
||||
if a.prepare:
|
||||
tu, fn = a.prepare
|
||||
ex = next((e for e in exemplars(include_done=True) if e["tu"] == tu and e["fn"] == fn), None)
|
||||
if ex is None:
|
||||
ex = dict(nhash="", tu=tu, fn=fn, alias=None, copies=1, needed=0, kinds=[], members=[], row={})
|
||||
d, draft, tgt, klass = prepare(ex)
|
||||
print(f"delever_permute --prepare: {d.relative_to(REPO)} (draft.c, {tgt.name}, klass={klass})")
|
||||
return 0
|
||||
if a.calibrate:
|
||||
return cmd_calibrate(a)
|
||||
if a.run:
|
||||
return cmd_run(a)
|
||||
if a.bank:
|
||||
return bank(a)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -486,20 +486,16 @@ def plan_files(inv, canon, only=None, bound=None):
|
||||
return files
|
||||
|
||||
|
||||
def direct_rewrite(s, canon):
|
||||
"""('call', text, name) | ('lever', variant_name, extra) | ('none', why, None) for a direct GTE statement."""
|
||||
sg = s["sig"]
|
||||
def canonical_match(sg, canon):
|
||||
"""(names, clobbers) for a signed statement's signature — the canonical macro it IS (one entry), or the
|
||||
concatenation of two canonical macros it is (a load then an op: bytes = b1 + b2, inputs the first's,
|
||||
clobbers the union) — else (None, None). ONE reader of the canonical table (R33): `direct_rewrite` asks it
|
||||
what the statement should look like, `delever.gte_canonical_clob` asks it what the canonical clobbers are."""
|
||||
if not sg["ok"]:
|
||||
return "none", "unsigned", None
|
||||
key = sig_key(sg)
|
||||
t = canon["canonical"].get(key)
|
||||
return None, None
|
||||
t = canon["canonical"].get(sig_key(sg))
|
||||
if t and sg["nout"] == 0:
|
||||
if tuple(sg["clob"]) == tuple(t["clob"]):
|
||||
return "call", f"{t['name']}({', '.join(sg['ins'])});", t["name"]
|
||||
extra = sorted(set(sg["clob"]) - set(t["clob"]))
|
||||
fewer = sorted(set(t["clob"]) - set(sg["clob"]))
|
||||
return "lever", (f"{t['name']}_m" if extra == ["memory"] and not fewer else f"{t['name']}_v"), extra or fewer
|
||||
# a concatenation of two canonical macros (a load then an op): bytes = b1 + b2, inputs = the first's, clobbers = the union
|
||||
return [t["name"]], list(t["clob"])
|
||||
b = sg["bytes"]
|
||||
for k1, t1 in canon["canonical"].items():
|
||||
b1 = k1.split("|")[0]
|
||||
@@ -509,12 +505,28 @@ def direct_rewrite(s, canon):
|
||||
t2 = canon["canonical"].get(k2)
|
||||
if not t2 or t1["nout"] or t1["nin"] != sg["nin"]:
|
||||
continue
|
||||
union = sorted(set(t1["clob"]) | set(t2["clob"]))
|
||||
if union == list(sg["clob"]):
|
||||
return "call", f"{t1['name']}({', '.join(sg['ins'])}); {t2['name']}();", f"{t1['name']}+{t2['name']}"
|
||||
extra = sorted(set(sg["clob"]) - set(union))
|
||||
return "lever", f"{t1['name']}+{t2['name']}_m" if extra == ["memory"] else f"{t1['name']}+{t2['name']}_v", extra or sorted(set(union) - set(sg["clob"]))
|
||||
return "none", "no canonical signature", None
|
||||
return [t1["name"], t2["name"]], sorted(set(t1["clob"]) | set(t2["clob"]))
|
||||
return None, None
|
||||
|
||||
|
||||
def direct_rewrite(s, canon):
|
||||
"""('call', text, name) | ('lever', variant_name, extra) | ('none', why, None) for a direct GTE statement."""
|
||||
sg = s["sig"]
|
||||
if not sg["ok"]:
|
||||
return "none", "unsigned", None
|
||||
names, clob = canonical_match(sg, canon)
|
||||
if names is None:
|
||||
return "none", "no canonical signature", None
|
||||
if len(names) == 1:
|
||||
if tuple(sg["clob"]) == tuple(clob):
|
||||
return "call", f"{names[0]}({', '.join(sg['ins'])});", names[0]
|
||||
extra = sorted(set(sg["clob"]) - set(clob))
|
||||
fewer = sorted(set(clob) - set(sg["clob"]))
|
||||
return "lever", (f"{names[0]}_m" if extra == ["memory"] and not fewer else f"{names[0]}_v"), extra or fewer
|
||||
if clob == list(sg["clob"]):
|
||||
return "call", f"{names[0]}({', '.join(sg['ins'])}); {names[1]}();", f"{names[0]}+{names[1]}"
|
||||
extra = sorted(set(sg["clob"]) - set(clob))
|
||||
return "lever", f"{names[0]}+{names[1]}_m" if extra == ["memory"] else f"{names[0]}+{names[1]}_v", extra or sorted(set(clob) - set(sg["clob"]))
|
||||
|
||||
|
||||
def file_edits(tu, raw, p, canon, label):
|
||||
|
||||
@@ -233,8 +233,11 @@ def klass_for_fn(fn):
|
||||
return "", ""
|
||||
|
||||
|
||||
def setup(fn, draft_c, asm_subdir=ASM, klass=None, where=""):
|
||||
pd = os.path.join(REPO, ".run/permuter", fn)
|
||||
def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None):
|
||||
# `outdir` keys the scratch dir by the CALLER's identity instead of the bare function name (R48). A name is not
|
||||
# unique across the fleet — the overlays overlap in RAM, so two different bodies are both `func_8013B274` — and two
|
||||
# concurrent runs on one name would share (and corrupt) this directory. Default: the historical path, unchanged.
|
||||
pd = os.path.join(REPO, outdir) if outdir else os.path.join(REPO, ".run/permuter", fn)
|
||||
if os.path.exists(pd):
|
||||
shutil.rmtree(pd)
|
||||
os.makedirs(pd)
|
||||
|
||||
@@ -41,10 +41,12 @@ def main():
|
||||
ap.add_argument("--secs", type=int, default=180, help="per-cycle time box")
|
||||
ap.add_argument("--j", type=int, default=12)
|
||||
ap.add_argument("--winners", default=".run/permuter-winners")
|
||||
ap.add_argument("--pd", help="scratch dir for this run (default .run/permuter/<fn>) — key it by alias+fn when "
|
||||
"several runs share a function NAME (R48)")
|
||||
a = ap.parse_args()
|
||||
|
||||
draft = open(a.draft).read()
|
||||
pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass)
|
||||
pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd)
|
||||
if not pd:
|
||||
print("ILS setup FAILED (target .s didn't assemble?)"); sys.exit(1)
|
||||
print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}")
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded), of which 300 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 20 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 330 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -341,6 +341,7 @@
|
||||
| `verbatim_target_s.py` | regenerate a splitter-format target disassembly for a function no longer stubbed | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | repo build/asm layout |
|
||||
| `delever_cycle.sh` | run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | the batch size, the aliases, the fleet count |
|
||||
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
|
||||
+12
-11
@@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on
|
||||
dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool,
|
||||
then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.*
|
||||
|
||||
**Coverage:** 300 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 29 (a SETUP row only), ORPHAN 30 (neither) — of 300. Portability: PORTABLE 23, ADAPT 260, PROJECT-ONLY 17.
|
||||
**Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 30 (a SETUP row only), ORPHAN 30 (neither) — of 301. Portability: PORTABLE 23, ADAPT 261, PROJECT-ONLY 17.
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -72,7 +72,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| When you need to… | Tool | What it does | Proven by | Adapt | Class |
|
||||
|---|---|---|---|---|---|
|
||||
| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+6) | MIPS relocation encodings | LIVE |
|
||||
| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+22) | compiler triple, repo build flags | LIVE |
|
||||
| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+23) | compiler triple, repo build flags | LIVE |
|
||||
| dump every compiler pass file for a self-contained draft | `cc1_dumps.sh` | Dumps every compiler pass file for a self-contained draft into a private directory | cc1_dumps_tu.sh, ghost_census.py | compiler triple, repo scratch paths | LIVE |
|
||||
| dump every compiler pass file for the spliced real translation unit | `cc1_dumps_tu.sh` | Same pass dumps for the spliced real translation unit, the faithful compile | alloc_table.py | absolute repo path, compiler triple | LIVE |
|
||||
| re-judge standalone compile failures against their real translation unit before dropping | `rtu_second_chance.py` | Re-judges standalone compile-failures against their real translation unit before dropping them | maintenance.sh | repo scratch paths | LIVE |
|
||||
@@ -175,10 +175,10 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| classify a length-drift near-miss against its own target and build its routing card | `len_tells.py` | Classifies a length-drift near-miss against its own target and builds the routing card | atlas_features.py | repo scratch paths | LIVE |
|
||||
| classify a residual's shape to its documented lever, as an independent implementation | `residual_rules_b.py` | Independent second implementation of residual-shape to rule classification, for head-to-head | triage_ladder.py | repo doc paths | LIVE |
|
||||
| classify where a remapped sibling's compiled bytes diverge from its target | `diff_regions.py` | Classifies where a remapped member's compiled bytes diverge from its target | — | repo build paths | ORPHAN |
|
||||
| compile a permuter candidate exactly as the build rule does | `permuter/compile.sh` | The permuter's compile command, mirroring the build rule exactly so objects are build-faithful | p16_permute.py, compile_o0.sh | compiler triple, repo build flags | LIVE |
|
||||
| compile a permuter candidate exactly as the build rule does | `permuter/compile.sh` | The permuter's compile command, mirroring the build rule exactly so objects are build-faithful | delever_permute.py, p16_permute.py, compile_o0.sh | compiler triple, repo build flags | LIVE |
|
||||
| compile a permuter candidate unoptimized, exactly as the build rule does | `permuter/compile_o0.sh` | Unoptimized variant of the same permuter compile command | p16_permute.py | compiler triple, repo build flags | LIVE |
|
||||
| decide whether a residual reorder is scheduling or a register-grant consequence | `oracle/reg_renumber_swap.sh` | Mechanized oracle separating a register-allocation swap from a scheduling reorder | — | compiler triple, repo scratch paths | ORPHAN |
|
||||
| drive the permuter per function, building base, target and settings | `p16_permute.py` | Per-function permuter driver building base, target and settings, then reporting closeness | auto_driver.py, grinder.py, permuter_ils.py | repo scratch paths, permuter layout | LIVE |
|
||||
| drive the permuter per function, building base, target and settings | `p16_permute.py` | Per-function permuter driver building base, target and settings, then reporting closeness | auto_driver.py, delever_permute.py, grinder.py, permuter_ils.py | repo scratch paths, permuter layout | LIVE |
|
||||
| find pseudo-registers with references but no remaining occurrences, and their frame slots | `ghost_census.py` | Frame-residue oracle: finds pseudos with references but no remaining occurrences, and their slots | cc1_dumps.sh | compiler pass-dump format | LIVE |
|
||||
| find which words diverge when the whole-binary gate says different but not where | `diff_autopsy.sh` | Reproduces exactly what the gate saw, then decodes the diverging words and restores the tree | — | repo src/build paths | REFERENCED |
|
||||
| grind the closest near-misses with the permuter and bank through the gate | `grinder.py` | Token-free permuter daemon grinding the closest near-misses and banking through the gate | auto_status.sh, auto_supervisor.sh, autopsy.py, gate_stage.py (+4) | repo scratch paths | LIVE |
|
||||
@@ -191,7 +191,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| steer permuter mutation toward the lever family the residual class implies | `permuter_weights.py` | Directs permuter mutation toward the lever family the residual class implies | autopsy.py, p16_permute.py | permuter settings format | LIVE |
|
||||
| tag every compiler-source citation with the source tree it refers to | `gccmap_cites.py` | Tags every compiler-source citation in the codegen-map docs with the tree it refers to | .github/workflows/no-rom.yml, Makefile, tool_census.py | repo doc/reference paths | LIVE |
|
||||
| turn a red whole-binary result into a named list of divergent symbols | `main_diff_locate.py` | Turns a red whole-binary result into a named list of divergent symbols | gate_main.py | repo build/config layout | LIVE |
|
||||
| warm-restart the permuter from its best waypoint each cycle | `permuter_ils.py` | Iterated-local-search wrapper warm-restarting the permuter from the best waypoint each cycle | grinder.py, permuter_sweep.py | permuter layout, repo scratch paths | LIVE |
|
||||
| warm-restart the permuter from its best waypoint each cycle | `permuter_ils.py` | Iterated-local-search wrapper warm-restarting the permuter from the best waypoint each cycle | delever_permute.py, grinder.py, permuter_sweep.py | permuter layout, repo scratch paths | LIVE |
|
||||
|
||||
## P8 — the campaign: cards, lanes, gates, recovery, harvest
|
||||
|
||||
@@ -322,23 +322,24 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
|---|---|---|---|---|---|
|
||||
| add per-site function-pointer casts so a draft can call a differently typed callee | `cast_call_sites.py` | Adds per-site function-pointer casts so a draft can call a differently typed callee | canon_sig_reconcile.py, cast_self_callers.py, family_sweep.py, gate_stage.py (+4) | repo src layout | LIVE |
|
||||
| apply curated names and signatures inside the analysis tool and save | `ghidra_scripts/ApplySymbols.java` | The in-tool half of that mirror: apply curated names and signatures, save on exit | ghidra_apply_symbols.sh, ghidra_export_annotations.sh, ghidra_rebuild.sh, ExportAnnotations.java (+1) | none | LIVE |
|
||||
| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, gte_consolidate.py, progress.py (+1) | repo paths, the shared-body scanner | LIVE |
|
||||
| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked <name>_m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever_cycle.sh, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE |
|
||||
| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+2) | repo paths, the shared-body scanner | LIVE |
|
||||
| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked <name>_m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever.py, delever_cycle.sh, delever_permute.py, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE |
|
||||
| convert a shared-body macro header into per-function plain-C headers included at each site | `macro_to_header.py` | The Phase-35 converter: every DEFINE_ macro body becomes a plain-C header under src/shared/<space>/ included at its site, the legacy name-parameterized headers converted, the registry text-edited, the macro header deleted; --plan / --apply / --finalize / --verify | share_body.py, tool_census.py | the macro form is this project's; a kit-born project shares headers from its first bank | LIVE |
|
||||
| emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE |
|
||||
| extract inline-defined types and typedefs from a source file into a shared header | `build_engine_types.py` | Extracts inline-defined named types and typedefs from a source file into a shared header | inject_capped_externs.py, lift_types.py, uniquify_type.py | repo shared-header path | LIVE |
|
||||
| flag address-named references whose address now has a curated name | `lint_symbol_refs.py` | Flags address-named references in committed sources whose address now has a curated name | .github/workflows/no-rom.yml, Makefile | repo symbol/src paths | LIVE |
|
||||
| give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED |
|
||||
| guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE |
|
||||
| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE |
|
||||
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+9) | repo paths, the registry and signature schemas | LIVE |
|
||||
| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE |
|
||||
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+10) | repo paths, the registry and signature schemas | LIVE |
|
||||
| mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED |
|
||||
| refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE |
|
||||
| regenerate a splitter-format target disassembly for a function no longer stubbed | `verbatim_target_s.py` | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | decompme_replica.sh | repo build/asm layout | LIVE |
|
||||
| regenerate a splitter-format target disassembly for a function no longer stubbed | `verbatim_target_s.py` | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | decompme_replica.sh, delever_permute.py | repo build/asm layout | LIVE |
|
||||
| run the de-lever batch cycle unattended: batch, verify by exit code, clean fleet gate, census, log, commit | `delever_cycle.sh` | The Phase-36 T3/T4 driver: per batch a clean-tree check, the oracle recalibrated when stale, delever --apply read by exit code and its final X/X line, the clean fleet run as the outer gate (218/218), the census rerun, the phase-log entry and the checkpoint headline, one commit per batch; stops on the first red with the files in place for --restore | delever.py, gte_consolidate.py | the batch size, the aliases, the fleet count | LIVE |
|
||||
| run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | `share_body_cycle.sh` | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | — | repo paths, the phase-log format, the fleet count from config/check.*.sha | ORPHAN |
|
||||
| search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED |
|
||||
| share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE |
|
||||
| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
|
||||
| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
|
||||
| turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
Reference in New Issue
Block a user