mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
phase-36: track readability_progress.py (the census's second oracle caught it untracked, R34)
tool_census's two enumerations disagreed on one file: the new tool was written but never added, so the dictionary row existed for a path git did not carry. That is exactly the disagreement the second oracle is there to find.
This commit is contained in:
Drew T
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 305 tool files in scope (submodules, vendored and downloaded code excluded), of which 305 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 25 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 335 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 307 tool files in scope (submodules, vendored and downloaded code excluded), of which 307 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 27 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 337 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -348,7 +348,9 @@
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `readability_progress.py` | track how honest and how readable the matched C is over time, as a generated series | The Gen3 companion to docs/levers.md: levers are only one way the source is untrue. Counts lying call declarations (a call site whose in-scope declaration names fewer parameters than the callee definition, split by the K&R-empty and (void) forms, and how many sit in a body that still holds an argument-register pin) and raw cast dereferences against struct member reads, which is the struct debt. Appends a dated row carrying its commit to docs/readability-progress.tsv and renders docs/readability.md; --check asserts the last row is this tree. | repo paths |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
| `decl_repair.py` | widen every call declaration that lies, and measure on the bytes which repairs are free | The free-correctness half of the missing-argument class: a declaration is repaired only when every call to that function in the unit ALREADY passes the arguments, so the code is right and only the promise is wrong. Each candidate is compiled from a scratch copy through the unit own recipe and compared on .text/.rodata/.data (never whole-object: the object records its own source path, which made the first fleet run return 0 free of 3,634 units). Every unit is negative-controlled by compiling it UNCHANGED first, and a unit whose control fails is reported as a harness failure rather than a result. --apply writes only the units this run judged identical. | repo paths; the recipe oracle |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
#!/usr/bin/env python3
|
||||
"""decl_repair — widen every call declaration that lies, and measure on the BYTES which repairs are free.
|
||||
|
||||
THE QUESTION (Drew, P36 S102). `argcheck` found 110,478 declarations narrower than the callee's real definition, over
|
||||
1,912 callees. Only a few hundred of those sit in a body that still holds a register pin; the rest are in functions that
|
||||
already match byte-for-byte. Those do not block anything — but they are wrong C: a call declared to take nothing that
|
||||
really takes a pointer compiles and matches through the calling convention, not because the declaration is true, and
|
||||
every one of them is a lie the names and types phases would inherit.
|
||||
|
||||
WHY THIS IS NOT HOUSEKEEPING. Repairing a declaration can CHANGE THE BYTES — that is exactly why it removes pins
|
||||
elsewhere. So the repair is not free by assumption, it is free or not by measurement, one translation unit at a time:
|
||||
|
||||
* IDENTICAL — the widened declarations produce the same object. The repair is FREE and can be applied with the gate as
|
||||
its proof.
|
||||
* DIFFERS — this unit's code depends on the false declaration. That is the interesting list, not the failure list:
|
||||
each one is a body quietly compensating for a wrong signature, and some of them are pins waiting to come off.
|
||||
* COMPILE-ERROR — the widened form conflicts with something else in the unit (a second declaration, a K&R definition).
|
||||
|
||||
Nothing is written into the tree: each candidate is compiled from a scratch copy of the unit through that unit's own
|
||||
recipe, and compared against the baseline snapshot the fleet gate cannot wipe. Run it while agents are working.
|
||||
"""
|
||||
import argparse, collections, concurrent.futures as cf, json, os, pathlib, re, subprocess, sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
import argcheck, delever_oracle as oracle, lever_census as lc
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
SCRATCH = REPO / ".run" / "P36" / "declrepair"
|
||||
DECL = re.compile(r"(\bextern\s+)([A-Za-z_][\w \t*]*?)\b([A-Za-z_]\w*)(\s*\()([^;)]*)(\)\s*;)")
|
||||
|
||||
|
||||
CALLSITE = re.compile(r"(?<![\w.>])([A-Za-z_]\w*)\s*\(")
|
||||
|
||||
|
||||
def call_arities(masked, callee):
|
||||
"""[n] — how many arguments each call to `callee` in this unit actually passes."""
|
||||
out = []
|
||||
for m in re.finditer(r"(?<![\w.>])" + re.escape(callee) + r"\s*\(", masked):
|
||||
depth, j = 0, m.end() - 1
|
||||
while j < len(masked):
|
||||
if masked[j] == "(":
|
||||
depth += 1
|
||||
elif masked[j] == ")":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
break
|
||||
j += 1
|
||||
if j >= len(masked):
|
||||
continue
|
||||
args = masked[m.end():j].strip()
|
||||
out.append(0 if args == "" else args.count(",") + 1)
|
||||
return out
|
||||
|
||||
|
||||
def widen(text, defs):
|
||||
"""(new text, [(callee, declared, real)]) — the declarations this unit can HONESTLY repair.
|
||||
|
||||
A declaration is only rewritten when EVERY call to that function in this unit ALREADY passes at least the real
|
||||
number of arguments. That is the free-correctness set: the code already does the right thing and only the promise is
|
||||
wrong, which is what m2c leaves behind when it emits `extern void f();` and then calls `f(x)`.
|
||||
|
||||
The first version of this tool widened every narrower declaration and left the call sites alone, which is not a
|
||||
repair at all — a prototype that demands an argument the call does not pass is a hard error, so 1,766 of the first
|
||||
3,250 units could not compile BY CONSTRUCTION and not one was IDENTICAL. A measurement that cannot come out any
|
||||
other way is not a measurement (R40: clear the instrument before believing its subject). Calls that pass too few
|
||||
arguments are the OTHER population — R19's, where the argument has to be chosen and the bytes decide.
|
||||
|
||||
The RETURN TYPE is never touched: this repairs arity, not results."""
|
||||
masked = lc.sc.mask_text(text)
|
||||
fixed = []
|
||||
|
||||
def sub(m):
|
||||
callee, params = m.group(3), m.group(5)
|
||||
real = defs.get(callee)
|
||||
if real is None:
|
||||
return m.group(0)
|
||||
dec = argcheck.arity(params)
|
||||
if dec >= real[0]:
|
||||
return m.group(0)
|
||||
calls = call_arities(masked, callee)
|
||||
if not calls or any(n < real[0] for n in calls):
|
||||
return m.group(0) # a call still passes too few: R19's job, not this one
|
||||
fixed.append((callee, dec, real[0]))
|
||||
return m.group(1) + m.group(2) + m.group(3) + m.group(4) + real[1] + m.group(6)
|
||||
|
||||
return DECL.sub(sub, text), fixed
|
||||
|
||||
|
||||
_SEC_CACHE = {}
|
||||
|
||||
|
||||
def sections(objpath, tag):
|
||||
"""The bytes of `.text`, `.rodata` and `.data` — the CODEGEN, without the object's embedded source path.
|
||||
|
||||
A whole-object comparison cannot work here: the candidate is compiled from a scratch copy, so its file name is
|
||||
recorded in the object and every candidate DIFFERS by construction. The first fleet run of this tool returned
|
||||
0 IDENTICAL of 3,634 units for exactly that reason and I believed it for one report — the negative control that
|
||||
would have caught it in a second (compile the UNCHANGED text through the same path and require IDENTICAL) is now
|
||||
part of every unit's judgement below (R39). With the path excluded, the control's sections match exactly."""
|
||||
key = (str(objpath), tag)
|
||||
if key in _SEC_CACHE:
|
||||
return _SEC_CACHE[key]
|
||||
out = b""
|
||||
for sec in (".text", ".rodata", ".data"):
|
||||
dst = SCRATCH / "_sec" / (re.sub(r"\W+", "_", str(objpath)) + tag + sec)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
r = subprocess.run(["mipsel-linux-gnu-objcopy", "-O", "binary", f"--only-section={sec}",
|
||||
str(objpath), str(dst)], capture_output=True)
|
||||
out += dst.read_bytes() if (r.returncode == 0 and dst.exists()) else b""
|
||||
_SEC_CACHE[key] = out
|
||||
return out
|
||||
|
||||
|
||||
def compile_to(rel, rec, text, tag):
|
||||
"""(object path | None, err) — `text` compiled from a scratch copy of the unit through its own recipe."""
|
||||
d = SCRATCH / re.sub(r"\W+", "_", rel)
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
scratch_src = d / os.path.basename(rec["src"])
|
||||
scratch_src.write_text(text if rel == rec["src"] else
|
||||
(REPO / rec["src"]).read_text(errors="surrogateescape"), errors="surrogateescape")
|
||||
pipeline = rec["pipeline"].replace(" " + rec["src"], " " + scratch_src.as_posix(), 1) \
|
||||
.replace("-Iinclude", f"-Iinclude -I{os.path.dirname(rec['src'])}", 1)
|
||||
data, dt, err = oracle.compile_obj(dict(rec, pipeline=pipeline, src=scratch_src.as_posix()), None,
|
||||
tag=tag + "_" + re.sub(r"\W+", "_", rel))
|
||||
if data is None:
|
||||
return None, err
|
||||
op = d / (tag + ".o")
|
||||
op.write_bytes(data)
|
||||
return op, ""
|
||||
|
||||
|
||||
def judge(rel, rec, defs):
|
||||
try:
|
||||
raw = (REPO / rel).read_text(errors="surrogateescape")
|
||||
except OSError as e:
|
||||
return dict(tu=rel, verdict="UNREADABLE", err=str(e)[:120], fixed=0)
|
||||
if "extern" not in raw:
|
||||
return None
|
||||
new, fixed = widen(raw, defs)
|
||||
if not fixed:
|
||||
return None
|
||||
basep = oracle.baseline_path(rec["obj"])
|
||||
if not pathlib.Path(basep).exists():
|
||||
return dict(tu=rel, verdict="NO-BASELINE", fixed=len(fixed), callees=[], err="")
|
||||
base = sections(basep, "base")
|
||||
# R39, in the tool: the UNCHANGED text must reproduce the baseline through this same path, or the verdict below is
|
||||
# about the harness rather than the repair.
|
||||
ctl, cerr = compile_to(rel, rec, raw, "ctl")
|
||||
if ctl is None or sections(ctl, "ctl") != base:
|
||||
return dict(tu=rel, verdict="CONTROL-FAILED", fixed=len(fixed), callees=[], err=cerr[:160])
|
||||
cand, err = compile_to(rel, rec, new, "cand")
|
||||
if cand is None:
|
||||
v = "COMPILE-CRASH" if err.startswith("CRASH:") else "COMPILE-ERROR"
|
||||
else:
|
||||
v = "IDENTICAL" if sections(cand, "cand") == base else "DIFFERS"
|
||||
return dict(tu=rel, verdict=v, fixed=len(fixed),
|
||||
callees=sorted({c for c, _, _ in fixed})[:8], err=err[:160] if cand is None else "")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("-j", "--jobs", type=int, default=8)
|
||||
ap.add_argument("--limit", type=int)
|
||||
ap.add_argument("--json", default=".run/P36/engine/decl_repair.json")
|
||||
ap.add_argument("--apply", action="store_true",
|
||||
help="write the repairs into the tree, ONLY for units this run judged IDENTICAL (each unit is "
|
||||
"re-judged here, never trusted from a previous run's json — the tree may have moved)")
|
||||
a = ap.parse_args()
|
||||
defs = argcheck.definitions()
|
||||
by_src = oracle.recipes_by_src(oracle.load_recipes()["recipes"])
|
||||
units = [(rel, recs[0]) for rel, recs in sorted(by_src.items()) if recs]
|
||||
if a.limit:
|
||||
units = units[:a.limit]
|
||||
print(f"decl_repair: {len(units)} translation unit(s) with a recipe; {len(defs)} definitions; -j{a.jobs}", flush=True)
|
||||
rows, n = [], 0
|
||||
with cf.ThreadPoolExecutor(max_workers=a.jobs) as ex:
|
||||
for r in ex.map(lambda u: judge(u[0], u[1], defs), units):
|
||||
n += 1
|
||||
if r:
|
||||
rows.append(r)
|
||||
if n % 250 == 0:
|
||||
c = collections.Counter(x["verdict"] for x in rows)
|
||||
print(f" {n}/{len(units)} judged; {dict(c)}", flush=True)
|
||||
if a.apply:
|
||||
# Only a unit THIS RUN judged IDENTICAL is written, and it is written from the same `widen` output the judgement
|
||||
# compiled — never re-derived, never taken from an earlier json (R42's cousin: the tree may have moved under it).
|
||||
n_w = n_d = 0
|
||||
for r in rows:
|
||||
if r["verdict"] != "IDENTICAL":
|
||||
continue
|
||||
path = REPO / r["tu"]
|
||||
raw = path.read_text(errors="surrogateescape")
|
||||
new, fixed = widen(raw, defs)
|
||||
if not fixed or new == raw:
|
||||
continue
|
||||
path.write_text(new, errors="surrogateescape")
|
||||
n_w += 1
|
||||
n_d += len(fixed)
|
||||
print(f"decl_repair --apply: {n_w} unit(s) rewritten, {n_d} declaration(s) repaired — now run the fleet gate")
|
||||
c = collections.Counter(x["verdict"] for x in rows)
|
||||
free = sum(x["fixed"] for x in rows if x["verdict"] == "IDENTICAL")
|
||||
held = sum(x["fixed"] for x in rows if x["verdict"] == "DIFFERS")
|
||||
print(f"decl_repair: {len(rows)} unit(s) carry a lying declaration; verdicts {dict(c)}")
|
||||
print(f" FREE to repair (the object is byte-identical with the declarations widened): "
|
||||
f"{c.get('IDENTICAL', 0)} unit(s), {free} declaration(s)")
|
||||
print(f" the object CHANGES (the code depends on the false declaration): "
|
||||
f"{c.get('DIFFERS', 0)} unit(s), {held} declaration(s)")
|
||||
pathlib.Path(a.json).write_text(json.dumps(rows, indent=1))
|
||||
print(f" rows -> {a.json}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env python3
|
||||
"""readability_progress — the Gen3 series: how honest and how readable the matched C is, over time.
|
||||
|
||||
WHY A SECOND SERIES. `docs/levers.md` (`tools/lever_progress.py`) counts LEVERS — the register pins and `asm`
|
||||
statements this project inserted to force a byte match — and it is the chart of Phase 36's own goal. But Gen3 is about
|
||||
the source being TRUE and readable, and the levers are only one of the ways it is not. Two more are measurable today and
|
||||
neither is a lever:
|
||||
|
||||
* LYING CALL DECLARATIONS. A file declares `extern void f(void);` while the function really takes an argument. It
|
||||
compiles, it matches, and it is false — and it was the largest single blocker in Phase 36's residue: six agents
|
||||
independently reached byte-identical output by restoring an argument the decompiler had dropped, and the register
|
||||
pin in each case existed only to fake the instruction the missing argument would have emitted.
|
||||
* RAW CAST DEREFERENCES. `*(u16 *)(p + 0x12)` where the original wrote a struct field. A struct is not recoverable
|
||||
from the binary — types are erased and a retail build carries no metadata — so this number falls only as the struct
|
||||
layers are inferred from base+offset+width evidence and applied.
|
||||
|
||||
Each row is dated and carries the commit it was taken at, so the chart is reproducible from the tree rather than typed
|
||||
(R75: a published number is generated). Append a row with `--snapshot "<label>"`; `--check` asserts the last row is
|
||||
this tree's.
|
||||
"""
|
||||
import argparse, datetime, json, pathlib, re, subprocess, sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
import argcheck
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
TSV = REPO / "docs" / "readability-progress.tsv"
|
||||
DOC = REPO / "docs" / "readability.md"
|
||||
CAST = re.compile(r"\*\s*\(\s*(?:struct\s+|union\s+)?[A-Za-z_]\w*\s*\*+\s*\)\s*\(")
|
||||
MEMBER = re.compile(r"->[A-Za-z_]\w*|\.[A-Za-z_]\w*\s*=")
|
||||
|
||||
|
||||
def counts():
|
||||
defs = argcheck.definitions()
|
||||
rows = argcheck.scan(defs)
|
||||
casts = members = 0
|
||||
for f in (REPO / "src").glob("**/*.c"):
|
||||
t = f.read_text(errors="surrogateescape")
|
||||
casts += len(CAST.findall(t))
|
||||
members += len(MEMBER.findall(t))
|
||||
for f in (REPO / "src" / "shared").glob("**/*.h"):
|
||||
t = f.read_text(errors="surrogateescape")
|
||||
casts += len(CAST.findall(t))
|
||||
members += len(MEMBER.findall(t))
|
||||
pinned = [r for r in rows if r["argpin"]]
|
||||
return dict(definitions=len(defs), narrow_decls=len(rows),
|
||||
callees=len({r["callee"] for r in rows}),
|
||||
kr_empty=sum(1 for r in rows if r["kind"] == "K&R-empty"),
|
||||
narrow=sum(1 for r in rows if r["kind"] == "narrow"),
|
||||
pinned_rows=len(pinned), pinned_bodies=len({(r["tu"], r["in_fn"]) for r in pinned}),
|
||||
raw_casts=casts, struct_members=members)
|
||||
|
||||
|
||||
HEAD = ("date\tlabel\tcommit\tdefinitions\tnarrow_decls\tcallees\tkr_empty\tnarrow\tpinned_rows\tpinned_bodies"
|
||||
"\traw_casts\tstruct_members\n")
|
||||
|
||||
|
||||
def render():
|
||||
rows = [l.rstrip("\n").split("\t") for l in TSV.read_text().splitlines()[1:] if l.strip()]
|
||||
out = ["# Readability of the matched C — the Gen3 series",
|
||||
"",
|
||||
"> **Generated by `tools/readability_progress.py --snapshot`; never typed (R75).** Companion to",
|
||||
"> `docs/levers.md`, which counts the compiler-forcing levers. This one counts the two things that are wrong",
|
||||
"> with the source even where no lever remains: call declarations that are FALSE, and memory read through raw",
|
||||
"> pointer casts where the original had a struct.",
|
||||
"",
|
||||
"| date | label | commit | lying declarations | of which `()` | of which `(void)` | in a pinned body | raw cast derefs | struct member reads |",
|
||||
"|---|---|---|---:|---:|---:|---:|---:|---:|"]
|
||||
for r in rows:
|
||||
out.append(f"| {r[0]} | {r[1]} | `{r[2][:9]}` | {r[4]} | {r[6]} | {r[7]} | {r[8]} | {r[10]} | {r[11]} |")
|
||||
out += ["",
|
||||
"**How to read it.** A *lying declaration* is a call site whose in-scope declaration names fewer parameters",
|
||||
"than the callee's own definition. It is not a style problem: on this processor an argument travels in a",
|
||||
"register, so a dropped argument removes an instruction, and Phase 36 found register pins inserted to fake",
|
||||
"exactly that instruction. *Raw cast derefs* is the struct debt — a struct cannot be recovered from the",
|
||||
"binary, only inferred from consistent base+offset+width evidence, so this figure falls as that work lands.",
|
||||
""]
|
||||
DOC.write_text("\n".join(out) + "\n")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--snapshot", metavar="LABEL")
|
||||
ap.add_argument("--check", action="store_true")
|
||||
a = ap.parse_args()
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=REPO, capture_output=True, text=True).stdout.strip()
|
||||
if a.check:
|
||||
if not TSV.exists():
|
||||
print("readability_progress --check: no series yet")
|
||||
return 1
|
||||
last = TSV.read_text().splitlines()[-1].split("\t")
|
||||
ok = last[2] == head
|
||||
print(f"readability_progress --check: {'OK' if ok else 'STALE'} — last row {last[1]!r} at {last[2][:9]}, "
|
||||
f"HEAD {head[:9]}")
|
||||
return 0 if ok else 1
|
||||
if not a.snapshot:
|
||||
print(__doc__)
|
||||
return 2
|
||||
c = counts()
|
||||
if not TSV.exists():
|
||||
TSV.write_text(HEAD)
|
||||
with TSV.open("a") as fh:
|
||||
fh.write("\t".join([datetime.date.today().isoformat(), a.snapshot, head, str(c["definitions"]),
|
||||
str(c["narrow_decls"]), str(c["callees"]), str(c["kr_empty"]), str(c["narrow"]),
|
||||
str(c["pinned_rows"]), str(c["pinned_bodies"]), str(c["raw_casts"]),
|
||||
str(c["struct_members"])]) + "\n")
|
||||
render()
|
||||
print(f"readability_progress: {a.snapshot} — {c['narrow_decls']} lying declaration(s) over {c['callees']} callee(s) "
|
||||
f"({c['kr_empty']} `()`, {c['narrow']} `(void)`), {c['pinned_rows']} in {c['pinned_bodies']} pinned bodies; "
|
||||
f"{c['raw_casts']} raw cast deref(s), {c['struct_members']} struct member read(s), at {head[:9]}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 305 tool files in scope (submodules, vendored and downloaded code excluded), of which 305 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 25 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 335 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 307 tool files in scope (submodules, vendored and downloaded code excluded), of which 307 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 27 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 337 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -348,7 +348,9 @@
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `readability_progress.py` | track how honest and how readable the matched C is over time, as a generated series | The Gen3 companion to docs/levers.md: levers are only one way the source is untrue. Counts lying call declarations (a call site whose in-scope declaration names fewer parameters than the callee definition, split by the K&R-empty and (void) forms, and how many sit in a body that still holds an argument-register pin) and raw cast dereferences against struct member reads, which is the struct debt. Appends a dated row carrying its commit to docs/readability-progress.tsv and renders docs/readability.md; --check asserts the last row is this tree. | repo paths |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
| `decl_repair.py` | widen every call declaration that lies, and measure on the bytes which repairs are free | The free-correctness half of the missing-argument class: a declaration is repaired only when every call to that function in the unit ALREADY passes the arguments, so the code is right and only the promise is wrong. Each candidate is compiled from a scratch copy through the unit own recipe and compared on .text/.rodata/.data (never whole-object: the object records its own source path, which made the first fleet run return 0 free of 3,634 units). Every unit is negative-controlled by compiling it UNCHANGED first, and a unit whose control fails is reported as a harness failure rather than a result. --apply writes only the units this run judged identical. | repo paths; the recipe oracle |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
|
||||
+6
-4
@@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on
|
||||
dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool,
|
||||
then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.*
|
||||
|
||||
**Coverage:** 305 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 243 (a runtime consumer), REFERENCED 32 (a SETUP row only), ORPHAN 30 (neither) — of 305. Portability: PORTABLE 23, ADAPT 265, PROJECT-ONLY 17.
|
||||
**Coverage:** 307 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 244 (a runtime consumer), REFERENCED 31 (a SETUP row only), ORPHAN 32 (neither) — of 307. Portability: PORTABLE 23, ADAPT 267, PROJECT-ONLY 17.
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -295,7 +295,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| build a progress timeline from the repository's own committed digests | `timeline.py` | Builds a progress timeline from the repository's own committed digests, with a self-check | Makefile | repo doc paths | LIVE |
|
||||
| check that every relative link in the public docs resolves | `doc_links.py` | Checks that every relative link in the public docs resolves and the link policy holds | Makefile, wiki_render.py, wiki_sync.sh | repo doc paths | LIVE |
|
||||
| compile every eligible translation unit with the pinned toolchain, using no derived bytes | `compile_only.py` | Compiles every eligible translation unit with the pinned toolchain, without any game bytes | .github/workflows/no-rom.yml, delever_oracle.py, lever_census.py, macro_to_header.py (+1) | compiler triple, repo makefile parsing | LIVE |
|
||||
| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+34) | repo src/config layout | LIVE |
|
||||
| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+35) | repo src/config layout | LIVE |
|
||||
| convert progress metrics into an external progress-report schema | `objdiff_report.py` | Converts the progress JSON into the external progress-report schema | .github/workflows/progress.yml | external report schema | LIVE |
|
||||
| emit the public ordinal-to-hash map and the private old-to-new map | `public_rewrite/build_commit_map.py` | Emits the public ordinal-to-new-hash map and the private old-to-new map | probe_github.sh, resolve_tokens.py | repo doc/scratch paths | LIVE |
|
||||
| gate a first push: prove no derived bytes among tracked files | `audit_public.py` | The first-push gate: no derived bytes among tracked files, four independent checks | .github/workflows/no-rom.yml, gate_scan.py | repo purge-set and path lists (the kit's template reads `config/firewall.txt` instead) | LIVE |
|
||||
@@ -329,12 +329,12 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE |
|
||||
| extract inline-defined types and typedefs from a source file into a shared header | `build_engine_types.py` | Extracts inline-defined named types and typedefs from a source file into a shared header | inject_capped_externs.py, lift_types.py, uniquify_type.py | repo shared-header path | LIVE |
|
||||
| find a lever-free spelling of a body by a guided search on the byte distance, not a blind test | `delever_search.py` | The Phase-36 rung G: the same shape generators as rung R (declaration order/move, initializer split, commutative swap, temp inlined/introduced/hoisted, block wrap/unwrap, adjacent swap) and the same per-TU oracle, but every candidate SCORED — the function's instructions read out of the scratch object and compared, reloc-masked, with the fleet run's baseline object, as an edit distance — the residual CLASSIFIED (REG caller/callee, COUNT, ORDER, MIXED) to choose the move families, and a beam search composing two and three moves under a compile budget; a score-0 verified on every recipe and banked through delever --apply-body, then --propagate to the class. Controls: the tree's own text must score 0 (per body), --positive-control perturbs a matching body and requires the way back, --selftest. | delever.py, delever_pack.py | repo scratch paths; the recipe oracle | LIVE |
|
||||
| find every call site whose in-scope declaration is narrower than the callee real definition | `argcheck.py` | The static filter for the missing-argument class: six T7 agents independently reached score 0 by restoring an argument the decompiled source had dropped (m2c drops them at unprototyped and indirect call sites), and no candidate generator can reach it because every family rewrites statements and declarations while this changes a call ARITY. Reads the real arity from the DEFINITION wherever it lives (a .c under src/ or a shared body header) and reports each narrower declaration with the scope that says which fix is available: the cast route is body-only at either scope, widening the declaration itself needs block scope. Known blind spot, stated in the docstring: it cannot see a definition that is uniformly wrong. | delever.py | repo paths; the census walk | LIVE |
|
||||
| find every call site whose in-scope declaration is narrower than the callee real definition | `argcheck.py` | The static filter for the missing-argument class: six T7 agents independently reached score 0 by restoring an argument the decompiled source had dropped (m2c drops them at unprototyped and indirect call sites), and no candidate generator can reach it because every family rewrites statements and declarations while this changes a call ARITY. Reads the real arity from the DEFINITION wherever it lives (a .c under src/ or a shared body header) and reports each narrower declaration with the scope that says which fix is available: the cast route is body-only at either scope, widening the declaration itself needs block scope. Known blind spot, stated in the docstring: it cannot see a definition that is uniformly wrong. | decl_repair.py, delever.py, readability_progress.py | repo paths; the census walk | LIVE |
|
||||
| flag address-named references whose address now has a curated name | `lint_symbol_refs.py` | Flags address-named references in committed sources whose address now has a curated name | .github/workflows/no-rom.yml, Makefile | repo symbol/src paths | LIVE |
|
||||
| give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED |
|
||||
| guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE |
|
||||
| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, delever_search.py (+1) | the Makefile's object rules, the twin rule | LIVE |
|
||||
| keep the lever count as a series a chart and a story can be drawn from | `lever_progress.py` | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | — | the census json, the delever ledger | REFERENCED |
|
||||
| keep the lever count as a series a chart and a story can be drawn from | `lever_progress.py` | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | readability_progress.py | the census json, the delever ledger | LIVE |
|
||||
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+11) | repo paths, the registry and signature schemas | LIVE |
|
||||
| mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED |
|
||||
| refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE |
|
||||
@@ -344,7 +344,9 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED |
|
||||
| share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE |
|
||||
| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_pack.py, delever_permute.py, delever_search.py (+2) | the census site records, the oracle | LIVE |
|
||||
| track how honest and how readable the matched C is over time, as a generated series | `readability_progress.py` | The Gen3 companion to docs/levers.md: levers are only one way the source is untrue. Counts lying call declarations (a call site whose in-scope declaration names fewer parameters than the callee definition, split by the K&R-empty and (void) forms, and how many sit in a body that still holds an argument-register pin) and raw cast dereferences against struct member reads, which is the struct debt. Appends a dated row carrying its commit to docs/readability-progress.tsv and renders docs/readability.md; --check asserts the last row is this tree. | — | repo paths | ORPHAN |
|
||||
| turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED |
|
||||
| widen every call declaration that lies, and measure on the bytes which repairs are free | `decl_repair.py` | The free-correctness half of the missing-argument class: a declaration is repaired only when every call to that function in the unit ALREADY passes the arguments, so the code is right and only the promise is wrong. Each candidate is compiled from a scratch copy through the unit own recipe and compared on .text/.rodata/.data (never whole-object: the object records its own source path, which made the first fleet run return 0 free of 3,634 units). Every unit is negative-controlled by compiling it UNCHANGED first, and a unit whose control fails is reported as a harness failure rather than a result. --apply writes only the units this run judged identical. | — | repo paths; the recipe oracle | ORPHAN |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user