mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 07:10:32 -04:00
phase-36: T6 CLOSE — both yield lines measured (permuter 5 of 16 exemplars / 665 of 2,131 bodies; recipes 134 of 134 in 6.0 min), 664 sites gone (34,091 -> 33,427), R22 218/218 at every step; the S99 checkpoint written for T7 (which starts only on Drew's direct approval) with the parallelised rung-R sweep as the drawable work meanwhile
This commit is contained in:
Drew T
@@ -2,7 +2,7 @@
|
||||
|
||||
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
|
||||
>
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1170 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1171 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
|
||||
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
|
||||
|
||||
@@ -1461,7 +1461,7 @@
|
||||
- **§479** — ★★★ — WHERE THE PERMUTER ACTUALLY PAYS: A MEASURED YIELD CURVE (P31 S77, 8 candidates) <sub>L35811</sub>
|
||||
- **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) <sub>L37151</sub>
|
||||
|
||||
### (unbucketed — title matched no symptom vocabulary) (337)
|
||||
### (unbucketed — title matched no symptom vocabulary) (338)
|
||||
|
||||
- **§3-How** — to use this <sub>L30</sub>
|
||||
- **§1** — Idiom catalog (asm pattern → C that produces it) <sub>L39</sub>
|
||||
@@ -1800,6 +1800,7 @@
|
||||
- **§452** — ★★★ — NOT EVERY VERBATIM BODY IS UNDECOMPILED WORK, AND §448'S HEADLINE OVERSTATED IT (P31 S75; 10-function burst, 0 banks, and the negative result is the finding) <sub>L35104</sub>
|
||||
- **§462** — FOUR LEVERS FROM `main:func_80024054` (91 ins, 74/53/32 → 4) <sub>L35644</sub>
|
||||
- **§480** — 🔴 — A STATIC BLOCKER CLASS THAT THE REAL PIPELINE ALREADY REMOVES IS A PHANTOM <sub>L35890</sub>
|
||||
- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) <sub>L37604</sub>
|
||||
|
||||
|
||||
## All sections, in order
|
||||
@@ -2974,6 +2975,7 @@
|
||||
- **§500** — ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MATCH / 9 NEAR / 2 FAIL, every closer, two NEW named gcc mechanisms, and the wave-process defects <sub>L36793</sub>
|
||||
- **§501** — ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABILITY FLIP BEFORE DISCARDING IT (P32 T4b, `main:func_800391D4`, a pinned wall banked by a Fable agent) <sub>L37151</sub>
|
||||
- **§453** — ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DEFECTS THAT SHAPED IT (P35 S94–S96) <sub>L37555</sub>
|
||||
- **§454** — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6) <sub>L37604</sub>
|
||||
|
||||
|
||||
---
|
||||
@@ -4156,3 +4158,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
|
||||
| L36793 | §500 | ★★★ — THE T3 WAVE HARVEST (P32 T3, 2026-09-05): 31 one-agent-per-function drafters → 20 MA |
|
||||
| L37151 | §501 | ★★★ — A LEVER THAT MEASURES WORSE MAY BE A CASCADE: READ THE `.loop` DUMP FOR THE DESIRABI |
|
||||
| L37555 | §453 | ★★★ — ONE SOURCE PER UNIQUE FUNCTION: THE INCLUDE-AT-SITE SHARE, ITS GATE, AND THE FOUR DE |
|
||||
| L37604 | §454 | Rung D and rung R: taking a lever off a body that still has to compile to the same bytes ( |
|
||||
|
||||
@@ -37600,3 +37600,70 @@ green. R34 in one sentence: build the oracle that can disagree, then measure the
|
||||
the classifier became includes (ov_SC03_015's 219 single-site macros; the never-extended members of five under-listed groups). REAL
|
||||
360,744 → 350,533 with EMPTY up the same: empty-bodied shared functions were folded into REAL under the macro form. Instruction-weighted
|
||||
metrics unchanged. A number generated from a form the tooling cannot see is a number, not a count.
|
||||
|
||||
## §454 — Rung D and rung R: taking a lever off a body that still has to compile to the same bytes (Phase 36 T6)
|
||||
|
||||
The residue after the mechanical de-lever campaign (rungs A–C) is 12,970 bodies in 1,804 text classes, each with at least one site the
|
||||
oracle called NEEDED. Two instruments work on it: **rung D**, decomp-permuter seeded with the LEVER-FREE body, and **rung R**, the
|
||||
cookbook's byte-neutral shape recipes applied mechanically. What follows is what S99 measured, including the part where the measurement
|
||||
was wrong.
|
||||
|
||||
**A DISASSEMBLY LISTING IS NOT A TARGET.** Two campaigns returned "0 of 16 exemplars" with a straight face. Both were the harness. The
|
||||
permuter's `target.o` had been assembled from `verbatim_target_s.py --gas`, a listing regenerated from the ROM image — and assembling a
|
||||
disassembly is a second toolchain with its own answers:
|
||||
|
||||
* objdump prints the PSEUDO-instruction `move rX,rY` for `addu rX,rY,$zero` (0x…21). GAS assembles `move` as `or` (0x…25). In one
|
||||
234-instruction function that is **24 wrong words**, and nothing said so.
|
||||
* a listing's `%hi`/`%lo` pairs come back RESOLVED — `lui $a1,0x801f` / `addiu $a1,$a1,-11600`, no relocation — while every compiled
|
||||
candidate carries `R_MIPS_HI16`/`LO16` against a symbol, and the masked scorer compares reloc operands.
|
||||
|
||||
Result: the scorer reported **28 for a body that is byte-identical**, so score 0 was unreachable and every NO-MATCH was its own.
|
||||
|
||||
The control that names this in one line: **seed the search with the body the tree already matches and read the base score.** It must be
|
||||
0. (`delever_permute.py --positive-control`.) Run it before believing any campaign's yield — a 0-yield run cannot otherwise be told from
|
||||
a broken scorer (R40).
|
||||
|
||||
The fix is to stop assembling a listing: **the target is the tree's own body compiled by the build's own tail** into a one-function
|
||||
object, so it carries the candidates' relocations by construction. The ROM listing stays the INDEPENDENT oracle that proves it —
|
||||
`match_one` must call that body a MATCH before the search starts — which is what keeps the arrangement from being circular (R34/R56).
|
||||
`verbatim_target_s.py --gas` now verifies itself too (assemble → disassemble → compare word by word → `.word`-patch or REFUSE); it is
|
||||
the file pasted into decomp.me, and it was wrong for every function containing a `move`.
|
||||
|
||||
**THE PROFILE FOLLOWS THE REGISTER, NOT THE SITE KIND.** A needed pin on a CALLEE-SAVED register ($16–$23) is an allocation-order
|
||||
residual and wants the regalloc profile. A pin on a CALLER-SAVED one ($2/$3/$4–$7) is not. `func_80163EC8` pins `$2`, and its residual
|
||||
is `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — the operand ORDER of one `&`. The regalloc
|
||||
profile weights `perm_commutative` 2.0; the cse profile weights it 40.0. Steering by "it was a pin, so regalloc" spends the whole budget
|
||||
20× away from the only lever that closes it.
|
||||
|
||||
**WHAT RUNG D ACTUALLY FINDS, AND WHY RUNG R THEN GETS IT FOR FREE.** Every win is a small, nameable source shape:
|
||||
|
||||
| function | start (mismatched ins) | time | the whole semantic change |
|
||||
|---|---|---|---|
|
||||
| `func_80135D20` | 12 | 24 s | `flag = 0;` → `do { flag = 0; } while (0);` (`perm_ins_block`, RC-5 scope) |
|
||||
| `func_80163EC8` | 8 | 314 s | `uVar5 = *(s32 *)(p+0x44); … uVar5 & ~0x20` → the temp inlined at its use (`perm_expand_expr`, §501-R's S2 kill) |
|
||||
|
||||
The permuter's winner is machine-reprinted by pycparser (extra parens, `;` statements, its own brace style), so banking it verbatim
|
||||
trades a lever for a readability regression — which is exactly what this phase exists to prevent. Each win is therefore turned into a
|
||||
mechanical recipe that produces the SAME bytes from a one-line diff in the real source. Rung R's generators, in trial order (targeted
|
||||
first, blanket last), each one compile:
|
||||
|
||||
* **R2** the formerly-pinned declarations permuted among their own lines · **R4** one of them moved through the whole declaration run
|
||||
(§76/§501-R: the allocation order is the bank) · **R3** an initializer split off its declaration, the assignment placed after the
|
||||
WHOLE run (C89 forbids a declaration after a statement — placing it after the last PINNED declaration compiles as K&R parameters and
|
||||
fails in a way that reads like a compiler bug)
|
||||
* **R5** the operand order of one commutative operator (the caller-saved lever above; needs no pinned declaration)
|
||||
* **R6** a local assigned once and read once, inlined at its use with its dead declaration removed
|
||||
* **R7** one statement wrapped in a block — `{ stmt; }` tried before `do { stmt; } while (0);`, because the readable spelling should win
|
||||
when both hold
|
||||
|
||||
**THE CLASS, NOT THE BODY.** A win on an exemplar is worth its whole text class (the head of the draw is 80 classes of 134 copies), but a
|
||||
ledger REPLAY cannot deliver it: the ledger replays a SITE SET, and a reshaped body is not one. `delever.py --propagate TU FN` does it by
|
||||
address remap — the two old bodies are identical modulo `func_`/`D_` tokens (that IS the class hash), so their tokens correspond one for
|
||||
one and the map they define is applied to the new body, each sibling judged on its own objects. `--apply-body` records the before/after
|
||||
body text in its ledger row because after the write it exists nowhere else.
|
||||
|
||||
**THE SHAPE OF THE POPULATION (first 6 of 16 exemplars, all ov_SC04_011, 134-copy classes).** Two closed, four did not, and the split is
|
||||
the starting distance, not the site count: the closers started 8 and 12 mismatched instructions from the target; the four that did not
|
||||
started at 70, 99, 105 and 131 — every one a body where removing a hand-placed `instruction` lever changed the instruction COUNT and
|
||||
shifted everything after it (`mine=103 ins, target=106`). They still improved a long way (131→35, 105→18, 99→37, 70→24), so they are
|
||||
seeds for a longer run or for T7's agents, not walls. `--max-start N` triages by that number instead of spending a search on it.
|
||||
|
||||
@@ -916,3 +916,33 @@ each a different action. (3) **A known-true control per join:** the registry rep
|
||||
multi-batch tool's hardest bug (stale line numbers + a restore that wipes the previous batch) into a non-event; the driver commits
|
||||
between batches. (5) **The census's per-instance forms are the registry check's C2d for free** (one cached scan, 36 s) — derive, don't
|
||||
re-parse (R33).
|
||||
|
||||
## P36 S99 (2026-09-09) — the search harness whose target was a disassembly, and the three cheap checks that would have caught it
|
||||
|
||||
(1) **Before believing any search yield, seed the search with a body that already MATCHES and read the base score. It must be 0.**
|
||||
Two rung-D campaigns returned "0 of 16 exemplars" and both were the instrument: the permuter's `target.o` had been assembled from a
|
||||
regenerated disassembly listing, and assembling a disassembly is a second toolchain — objdump prints the pseudo-instruction `move` for
|
||||
`addu rX,rY,$zero` and GAS assembles `move` as `or` (24 wrong words in one 234-instruction function), while a listing's `%hi`/`%lo`
|
||||
pairs come back resolved, with no relocation, against candidates that all carry one. The scorer read 28 for a byte-identical body, so
|
||||
score 0 was unreachable. The control is one command and ~30 seconds (`delever_permute.py --positive-control`); without it, "the residue
|
||||
is hard" and "the scorer is broken" are the same observation. R40, and the first place this project has needed it against a SCORER
|
||||
rather than a model.
|
||||
|
||||
(2) **The target of a byte search should be produced by the same toolchain as the candidates, with an independent oracle to prove it
|
||||
— not by a second toolchain that has to be argued with.** The fix was to compile the tree's own body into a one-function object (the
|
||||
candidates' relocations by construction) and require `match_one` against the ROM listing to call it a MATCH before the search starts.
|
||||
Faster to build than the listing-fixing it replaced, and it cannot drift.
|
||||
|
||||
(3) **A generated artifact that nothing ever consumed is a claim.** `verbatim_target_s.py --gas` had been proven once, through
|
||||
decomp.me, on a function with no `move` in it — so it was wrong for a large fraction of the fleet and nobody knew (R98/DK-81 in a second
|
||||
place). It now assembles, disassembles and compares itself word by word against the image, `.word`-patches what does not reproduce, and
|
||||
REFUSES what still disagrees.
|
||||
|
||||
(4) **Steer a directed search by what the residual IS, not by what produced it.** The weight profile was chosen from the site kind
|
||||
("a pin → regalloc"); a pin on a CALLER-saved register ($2/$3/$4–$7) is an operand-order residual, and the regalloc profile weights
|
||||
`perm_commutative` 2.0 where the cse profile weights it 40.0. One reading of one diff moved 8 of 16 exemplars to the right profile.
|
||||
|
||||
(5) **Turn each search win into a mechanical recipe the same day.** Rung D's two wins were a statement wrapped in a block and a
|
||||
single-set temp inlined at its use — both one-line source changes that a recipe generator reproduces in ONE compile, and that keep the
|
||||
source readable, where banking the permuter's own pycparser-reprinted body would trade a lever for a readability regression in a phase
|
||||
whose whole purpose is readability.
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -335,6 +335,7 @@
|
||||
| `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout |
|
||||
| `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout |
|
||||
| `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule |
|
||||
| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger |
|
||||
| `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas |
|
||||
| `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name |
|
||||
| `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing |
|
||||
@@ -343,7 +344,7 @@
|
||||
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
@@ -1254,6 +1254,512 @@ def scrub(a):
|
||||
return 0 if not bad else 1
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
# --recipes (rung R): the cookbook's byte-neutral shape recipes, tried mechanically on a RESIDUE body
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
CTRL_KW = re.compile(r"^\s*(?:if|for|while|do|switch|else|return|goto|case|default|break|continue)\b")
|
||||
|
||||
|
||||
def pin_names(sites):
|
||||
"""the variable each pin site declares, in source order (the zero-register pins excluded: their declaration is deleted)."""
|
||||
out = []
|
||||
for s in sites:
|
||||
if s["kind"] != "pin" or s.get("zero"):
|
||||
continue
|
||||
mm = re.search(r"\b([A-Za-z_]\w*)\s*(?:\[[^\]]*\])?\s*(?:__asm__|__asm|asm)\s*\(", s.get("text", ""))
|
||||
if mm:
|
||||
out.append(mm.group(1))
|
||||
return out
|
||||
|
||||
|
||||
def is_decl_line(masked_line):
|
||||
"""a whole-statement DECLARATION on one line: `<type> <name>[\\[n\\]][ = init];`. The type and the name must be separated
|
||||
(by space or `*`) — without that, `ret = f();` parses as the declaration `re t = …` and an initializer split lands its
|
||||
assignment after the first statement, where C89 forbids the declarations that follow it."""
|
||||
s = masked_line.strip()
|
||||
return bool(s.endswith(";") and not CTRL_KW.match(s) and "(" not in s.split("=")[0]
|
||||
and re.match(r"^[A-Za-z_][\w \t]*[\s*]\s*\*?\s*[A-Za-z_]\w*\s*(?:\[[^\]]*\])*\s*(?:=|;)", s))
|
||||
|
||||
|
||||
def decl_run_end(text, d):
|
||||
"""the 0-based index of the LAST line of the declaration run that opens fn's body — a C89 declaration may not follow a
|
||||
statement, so an initializer split must put its assignment after the WHOLE run, not after the last pinned declaration."""
|
||||
lines = text.split("\n")
|
||||
last = d["line"] - 1
|
||||
for i in range(d["line"], d["end"] - 1):
|
||||
s = sc.mask_text(lines[i]).strip()
|
||||
if not s or s.startswith("/*") or s.startswith("//") or s in ("{", "}"):
|
||||
continue
|
||||
if is_decl_line(sc.mask_text(lines[i])):
|
||||
last = i
|
||||
continue
|
||||
break
|
||||
return last
|
||||
|
||||
|
||||
def decl_lines(text, tu, fn, names):
|
||||
"""[(line index, text)] for the lines of fn's body that DECLARE one of `names` — one name per line, the line a whole
|
||||
statement. The lever-free text is re-scanned for them (a stripped pin can delete its line, so the census's numbers have
|
||||
moved); a name whose declaration is not found alone on one line makes the body ineligible (None)."""
|
||||
recs = sc.scan_text(text, tu, shared_defs=None)
|
||||
d = next((r for r in recs if r["form"] == "def" and r["name"] == fn), None)
|
||||
if d is None:
|
||||
return None
|
||||
lines = text.split("\n")
|
||||
found = {}
|
||||
for i in range(d["line"], d["end"] - 1): # inside the body, never the header line
|
||||
raw_line = lines[i]
|
||||
s = sc.mask_text(raw_line).strip()
|
||||
if not is_decl_line(s):
|
||||
continue
|
||||
hits = [n for n in names if re.search(r"(?<![\w])%s\b" % re.escape(n), s.split("=")[0])]
|
||||
if len(hits) != 1 or hits[0] in found:
|
||||
continue
|
||||
found[hits[0]] = (i, raw_line)
|
||||
if len(found) != len(set(names)):
|
||||
return None
|
||||
return sorted(found.values())
|
||||
|
||||
|
||||
COMMUTATIVE = "&|^+*"
|
||||
|
||||
|
||||
def top_level_ops(expr):
|
||||
"""positions of the BINARY commutative operators at paren/bracket depth 0 in `expr` (masked text). An operator is binary
|
||||
when the previous non-space character ends an operand (identifier, digit, `)`, `]`); that also excludes a unary `*`/`&`
|
||||
and a cast's `*`, which in any case sits inside parentheses."""
|
||||
out, depth = [], 0
|
||||
for i, c in enumerate(expr):
|
||||
if c in "([":
|
||||
depth += 1
|
||||
elif c in ")]":
|
||||
depth -= 1
|
||||
elif depth == 0 and c in COMMUTATIVE:
|
||||
if i + 1 < len(expr) and expr[i + 1] in "&|=+*": # && || &= += *= **
|
||||
continue
|
||||
if i and expr[i - 1] in "&|=+*<>!-/%":
|
||||
continue
|
||||
prev = expr[:i].rstrip()
|
||||
if prev and (prev[-1].isalnum() or prev[-1] in "_)]"):
|
||||
out.append(i)
|
||||
return out
|
||||
|
||||
|
||||
def commutative_swaps(text, tu, fn, d_):
|
||||
"""[(description, candidate text)] — each line of fn's body that carries exactly ONE top-level commutative operator,
|
||||
with its two operands swapped. THE lever for a caller-saved ($2/$3) residual: S99 read `and v0,v1,v0` against the
|
||||
target's `and v0,v0,v1` on func_80163EC8, which is the operand order of one `&` in the source and nothing else."""
|
||||
lines = text.split("\n")
|
||||
out = []
|
||||
for i in range(d_["line"], d_["end"] - 1):
|
||||
raw_line, s = lines[i], sc.mask_text(lines[i])
|
||||
body = s.strip()
|
||||
if not body.endswith(";") or body.startswith("#"):
|
||||
continue
|
||||
# only an assignment's RHS or a `return` expression — an `if (…) stmt;` line would need the condition parsed out of
|
||||
# the statement after it, and a wrong split is a candidate that cannot compile (wasted, and noisy in the ledger)
|
||||
asg = re.search(r"(?<![=!<>+\-*/%&|^~])=(?!=)", s)
|
||||
ret = re.match(r"^\s*return\b", s)
|
||||
start = asg.end() if asg else (ret.end() if ret else -1)
|
||||
if start < 0 or start >= len(s):
|
||||
continue
|
||||
expr_end = s.rstrip().rfind(";")
|
||||
expr = s[start:expr_end]
|
||||
ops = top_level_ops(expr)
|
||||
if len(ops) != 1:
|
||||
continue
|
||||
o = start + ops[0]
|
||||
left, right = raw_line[start:o], raw_line[o + 1:expr_end]
|
||||
if not left.strip() or not right.strip():
|
||||
continue
|
||||
cand = list(lines)
|
||||
cand[i] = raw_line[:start] + " " + right.strip() + " " + raw_line[o] + " " + left.strip() + raw_line[expr_end:]
|
||||
out.append((f"swap {raw_line[o]} @{i + 1}", "\n".join(cand)))
|
||||
return out
|
||||
|
||||
|
||||
IDENT = re.compile(r"(?<![\w.])([A-Za-z_]\w*)(?![\w])")
|
||||
|
||||
|
||||
def inline_single_set_temps(text, tu, fn, d_):
|
||||
"""[(description, candidate text)] — a local assigned ONCE and read ONCE, inlined at its use and its now-dead
|
||||
declaration removed. §501-R's S2 kill: a fresh single-set local gets a birthing boost in gcc 2.7.2's allocator, so
|
||||
creating or removing one moves the allocation — and this is the move rung D found first (S99, func_80163EC8:
|
||||
`uVar5 = *(s32 *)(psVar6 + 0x44); … = uVar5 & ~0x20;` became `… = *(s32 *)(psVar6 + 0x44) & ~0x20;`). Doing it here
|
||||
keeps the SOURCE readable: the permuter's own winner is machine-reprinted, and this phase is about readability."""
|
||||
lines = text.split("\n")
|
||||
lo, hi = d_["line"], d_["end"] - 1
|
||||
masked = [sc.mask_text(l) for l in lines]
|
||||
occ = collections.defaultdict(list)
|
||||
for i in range(lo, hi):
|
||||
for m in IDENT.finditer(masked[i]):
|
||||
occ[m.group(1)].append((i, m.start(), m.end()))
|
||||
out = []
|
||||
ASG = r"^(?:[A-Za-z_][\w \t]*[\s*]\s*\*?\s*)?%s\s*=(?!=)\s*(.+);\s*$"
|
||||
for v, places in occ.items():
|
||||
asgs, decls, uses = [], [], []
|
||||
for p in places:
|
||||
s = masked[p[0]].strip()
|
||||
if re.match(ASG % re.escape(v), s):
|
||||
asgs.append(p)
|
||||
elif is_decl_line(s) and "=" not in s.split(";")[0]:
|
||||
decls.append(p)
|
||||
else:
|
||||
uses.append(p)
|
||||
if not asgs or not uses or len(decls) > 1:
|
||||
continue
|
||||
# ONE ASSIGNMENT AT A TIME, not one per variable. The single-set case is the easy half; the lever rung D actually
|
||||
# found is narrower: `uVar5` is assigned in TWO branches of func_80163EC8, and the winning move inlined ONE of them.
|
||||
# An assignment is inlinable when its value is read exactly once before the variable is written again — the classic
|
||||
# def-with-one-use — so the assignment can go and the read can carry the expression.
|
||||
order = sorted(places)
|
||||
for ai, _, _ in asgs:
|
||||
after = [p for p in order if p[0] > ai]
|
||||
reads = [p for p in after if p not in [(x, y, z) for x, y, z in asgs] and not is_decl_line(masked[p[0]].strip())]
|
||||
if not reads:
|
||||
continue
|
||||
ui, u0, u1 = reads[0]
|
||||
nxt = [p for p in after if p[0] > ui]
|
||||
if nxt and (nxt[0][0], nxt[0][1], nxt[0][2]) not in [(x, y, z) for x, y, z in asgs]:
|
||||
continue # read again before it is rewritten: the assignment is not dead
|
||||
mm = re.match(ASG % re.escape(v), masked[ai].strip())
|
||||
if not mm:
|
||||
continue
|
||||
expr = lines[ai].strip()[mm.start(1):mm.end(1)]
|
||||
if not expr.strip():
|
||||
continue
|
||||
cand = list(lines)
|
||||
cand[ui] = lines[ui][:u0] + f"({expr})" + lines[ui][u1:]
|
||||
cand[ai] = None # the assignment goes (with its declaration when they are one)
|
||||
if decls and len(asgs) == 1: # the separate `T v;` is dead only when nothing else writes v
|
||||
cand[decls[0][0]] = None
|
||||
out.append((f"inline {v} @{ai + 1}", "\n".join(l for l in cand if l is not None)))
|
||||
return out
|
||||
|
||||
|
||||
def block_wraps(text, tu, fn, d_):
|
||||
"""[(description, candidate text)] — one statement wrapped in a block. §501-R's RC-5 scope lever: a block changes the
|
||||
statement's basic-block structure and with it the allocno live range, which is how rung D closed func_80135D20 in 24 s
|
||||
(`flag = 0;` -> `do { flag = 0; } while (0);` and nothing else). The plain block is tried FIRST because it is the
|
||||
readable spelling; the do-while is gcc's stronger form and is only reached when the plain one does not hold."""
|
||||
lines = text.split("\n")
|
||||
out = []
|
||||
for i in range(d_["line"], d_["end"] - 1):
|
||||
s = sc.mask_text(lines[i]).strip()
|
||||
if not s.endswith(";") or CTRL_KW.match(s) or is_decl_line(s) or s.startswith("#") or "{" in s or "}" in s:
|
||||
continue
|
||||
raw_line = lines[i]
|
||||
indent = raw_line[:len(raw_line) - len(raw_line.lstrip())]
|
||||
stmt = raw_line.strip()
|
||||
for tag, spelling in (("block", f"{indent}{{ {stmt} }}"), ("do-while", f"{indent}do {{ {stmt} }} while (0);")):
|
||||
cand = list(lines)
|
||||
cand[i] = spelling
|
||||
out.append((f"{tag} @{i + 1}", "\n".join(cand)))
|
||||
return out
|
||||
|
||||
|
||||
def recipe_candidates(text, tu, fn, names, limit=24, rng=None, cap=40, blocks=True):
|
||||
"""[(recipe, description, candidate text)] — the byte-neutral shape recipes of the cookbook, mechanically.
|
||||
R2 (§76/§501-R, the allocation ORDER is the bank): the formerly-pinned declarations permuted among their own lines.
|
||||
R4: one of them moved through the whole declaration run. R3 (§17a/§501-P): an initializer split off its declaration.
|
||||
R5 (§137/§501-R, the caller-saved class): the operand order of one commutative operator — the ONLY recipe here that
|
||||
needs no pinned declaration, so a residue of barriers and launders still has candidates."""
|
||||
recs_ = sc.scan_text(text, tu, shared_defs=None)
|
||||
d_ = next((r for r in recs_ if r["form"] == "def" and r["name"] == fn), None)
|
||||
if d_ is None:
|
||||
return []
|
||||
dls = decl_lines(text, tu, fn, names) if names else None
|
||||
lines = text.split("\n")
|
||||
out = []
|
||||
rng = rng or random.Random(0)
|
||||
idx = [i for i, _ in dls] if dls else []
|
||||
body = [t for _, t in dls] if dls else []
|
||||
perms = []
|
||||
if not dls:
|
||||
perms = []
|
||||
elif len(body) <= 4:
|
||||
import itertools
|
||||
perms = [p for p in itertools.permutations(range(len(body)))][1:] # the identity is the current text
|
||||
else:
|
||||
seen = set()
|
||||
while len(perms) < limit and len(seen) < limit * 4:
|
||||
p = tuple(rng.sample(range(len(body)), len(body)))
|
||||
seen.add(p)
|
||||
if p != tuple(range(len(body))) and p not in perms:
|
||||
perms.append(p)
|
||||
for p in perms[:limit]:
|
||||
ls_ = list(lines)
|
||||
for slot, src in zip(idx, p):
|
||||
ls_[slot] = body[src]
|
||||
out.append(("R2", "decl-order " + ",".join(str(x) for x in p), "\n".join(ls_)))
|
||||
# R4 (§76/§501-R again, one variable at a time): a formerly-pinned declaration moved to every other slot of the body's
|
||||
# declaration run — the allocno creation order the pin used to override. R2 permutes the pinned declarations among
|
||||
# THEIR OWN slots; this reaches the orders that involve the untouched declarations too, at one compile each.
|
||||
last = decl_run_end(text, d_)
|
||||
run_idx = [i for i in range(d_["line"], last + 1) if is_decl_line(sc.mask_text(lines[i]))]
|
||||
if dls and len(run_idx) > 1:
|
||||
run_txt = [lines[i] for i in run_idx]
|
||||
for i, t in dls:
|
||||
if i not in run_idx:
|
||||
continue
|
||||
src = run_idx.index(i)
|
||||
name = next((n for n in names if re.search(r"(?<![\w])%s\b" % re.escape(n), t.split("=")[0])), "?")
|
||||
for dst in range(len(run_txt)):
|
||||
if dst == src:
|
||||
continue
|
||||
seq = list(run_txt)
|
||||
seq.insert(dst, seq.pop(src))
|
||||
ls_ = list(lines)
|
||||
for slot, txt in zip(run_idx, seq):
|
||||
ls_[slot] = txt
|
||||
out.append(("R4", f"decl-move {name} {src}->{dst}", "\n".join(ls_)))
|
||||
for k, (i, t) in enumerate(dls or []):
|
||||
head, _, init = t.partition("=")
|
||||
if not init.strip().endswith(";") or "==" in t:
|
||||
continue
|
||||
name = [n for n in names if re.search(r"(?<![\w])%s\b" % re.escape(n), head)]
|
||||
if len(name) != 1:
|
||||
continue
|
||||
ls_ = list(lines)
|
||||
ls_[i] = head.rstrip() + ";"
|
||||
indent = t[:len(t) - len(t.lstrip())]
|
||||
ls_.insert(last + 1, f"{indent}{name[0]} ={init.rstrip()}")
|
||||
out.append(("R3", f"init-split {name[0]}", "\n".join(ls_)))
|
||||
for desc, cand in commutative_swaps(text, tu, fn, d_):
|
||||
out.append(("R5", desc, cand))
|
||||
for desc, cand in inline_single_set_temps(text, tu, fn, d_):
|
||||
out.append(("R6", desc, cand))
|
||||
if blocks: # last: one candidate per statement, so the targeted recipes go first
|
||||
for desc, cand in block_wraps(text, tu, fn, d_):
|
||||
out.append(("R7", desc, cand))
|
||||
seen, uniq = {text}, [] # never judge the seed twice, nor one candidate twice (R37)
|
||||
for rec, desc, cand in out:
|
||||
if cand in seen:
|
||||
continue
|
||||
seen.add(cand)
|
||||
uniq.append((rec, desc, cand))
|
||||
return uniq[:cap]
|
||||
|
||||
|
||||
def recipes(a):
|
||||
"""--recipes: rung R over the RESIDUE bodies. The seed is the body's LEVER-FREE text; the first candidate whose object is
|
||||
IDENTICAL replaces it and its markers are scrubbed with its levers. Scope, stated (R41): only a body with at least one
|
||||
formerly-PINNED declaration has candidates — the recipes are declaration-order and initializer-split levers — so a residue
|
||||
of barriers/launders/keep-alives alone is not drawable here and stays for rung D and T7. A hit banks the EXEMPLAR body;
|
||||
its copies are a separate step (their text must be remapped, not replayed: the ledger replays a SITE SET, and a reshaped
|
||||
body is not one)."""
|
||||
ensure_census(a.jobs)
|
||||
clean, dirty = src_clean()
|
||||
if not clean and not a.dirty_ok:
|
||||
sys.exit(f"delever --recipes: src/ is dirty (commit or --restore first):\n{dirty[:400]}")
|
||||
ok, why = oracle.calibration_current()
|
||||
if not ok:
|
||||
sys.exit(f"delever --recipes: calibration not current ({why})")
|
||||
by_src = oracle.recipes_by_src(oracle.load_recipes()["recipes"])
|
||||
inc = includers()
|
||||
sites_by = collections.defaultdict(list)
|
||||
for s in load_sites():
|
||||
if s.get("fn"):
|
||||
sites_by[(s["tu"], s["fn"])].append(s)
|
||||
cur = {}
|
||||
for r in load_ledger():
|
||||
if r.get("tu") and r.get("fn"):
|
||||
cur[(r["tu"], r["fn"])] = r
|
||||
|
||||
def recs_for(tu):
|
||||
return [r for t_ in inc.get(tu, []) for r in by_src.get(t_, [])] if tu.endswith(".h") else by_src.get(tu, [])
|
||||
|
||||
def judge(tu, cand):
|
||||
"""judge one candidate, and be killable: the oracle writes the candidate into the tree to compile it, so a SIGTERM
|
||||
between the write and the restore leaves a candidate in `src/`. The original text goes into inflight.json first, the
|
||||
same file `--restore` reads (P35's rule: a tool restores from its OWN snapshot, never `git checkout`) — S99 killed a
|
||||
run mid-judge and found exactly that leftover."""
|
||||
path = REPO / tu
|
||||
raw, st = path.read_text(errors="surrogateescape"), path.stat()
|
||||
RUN.mkdir(parents=True, exist_ok=True)
|
||||
INFLIGHT.write_text(json.dumps({tu: raw}))
|
||||
try:
|
||||
return oracle.judge_all(recs_for(tu), cand, tag="rec", write_path=(tu if tu.endswith(".h") else None))
|
||||
finally:
|
||||
restore_file(path, raw, st)
|
||||
INFLIGHT.unlink(missing_ok=True)
|
||||
|
||||
def lever_free(tu, raw, fn):
|
||||
m, ls = same_len_mask(raw), line_starts(raw)
|
||||
edits = []
|
||||
for s in sites_by[(tu, fn)]:
|
||||
if (s["cls"], s["kind"]) not in REMOVABLE:
|
||||
if s["kind"] in DEFERRED_KINDS:
|
||||
raise Refuse("asm-body")
|
||||
continue
|
||||
edits += site_edits(raw, m, ls, s)
|
||||
return apply_edits(raw, edits) if edits else raw
|
||||
|
||||
todo = [(k, r) for k, r in cur.items() if r.get("verdict") == "RESIDUE" and k[1] != FILE_SCOPE_FN]
|
||||
if a.only:
|
||||
todo = [(k, r) for k, r in todo if any(o in (k[0], k[1]) or o in k[0] for o in a.only)]
|
||||
csize = collections.Counter(r.get("nhash_after") or r.get("nhash_before") for _, r in todo)
|
||||
todo.sort(key=lambda x: (len([s for s in x[1].get("sites", []) if s.get("verdict") == "NEEDED"]),
|
||||
-csize[x[1].get("nhash_after") or x[1].get("nhash_before")], x[0]))
|
||||
todo = todo[:a.limit] if a.limit else todo
|
||||
# THE CONTROL, before any verdict is believed (R39), on the first bodies drawn and on BOTH instruments:
|
||||
# the splice machinery — the IDENTITY permutation through the same code must reproduce its input text exactly
|
||||
# (a text assertion, no compile: if the line surgery is not byte-neutral, every "DIFFERS" below is its own);
|
||||
# the oracle — the file exactly as the tree has it must still judge IDENTICAL (its baseline is live, R56).
|
||||
bad = []
|
||||
for (tu, fn), r in todo[:a.control]:
|
||||
raw = (REPO / tu).read_text(errors="surrogateescape")
|
||||
names = pin_names(sites_by[(tu, fn)])
|
||||
dls = decl_lines(raw, tu, fn, names) if names else None
|
||||
if dls:
|
||||
lines = raw.split("\n")
|
||||
ident = list(lines)
|
||||
for slot, src in zip([i for i, _ in dls], range(len(dls))):
|
||||
ident[slot] = [t for _, t in dls][src]
|
||||
if "\n".join(ident) != raw:
|
||||
bad.append((tu, fn, "SPLICE", "the identity permutation did not reproduce the text"))
|
||||
continue
|
||||
v, dt, err = judge(tu, raw)
|
||||
if v != "IDENTICAL":
|
||||
bad.append((tu, fn, v, err[:80]))
|
||||
print(f"delever --recipes: control {a.control - len(bad)}/{min(a.control, len(todo))} — the identity splice reproduces "
|
||||
f"its text and the untouched file its object", flush=True)
|
||||
if bad or not todo:
|
||||
for b in bad[:5]:
|
||||
print(f" CONTROL FAILED {b[0]}:{b[1]} -> {b[2]} {b[3]}")
|
||||
sys.exit("delever --recipes: the control did not pass — no verdict from this run is usable (R39)")
|
||||
print(f"delever --recipes: {len(todo)} RESIDUE bodies", flush=True)
|
||||
rows, won, tried, compiles = [], 0, 0, 0
|
||||
t0 = time.time()
|
||||
for n_body, ((tu, fn), r) in enumerate(todo, 1):
|
||||
path = REPO / tu
|
||||
raw = path.read_text(errors="surrogateescape")
|
||||
names = pin_names(sites_by[(tu, fn)]) # may be empty: R5 needs no pinned declaration
|
||||
try:
|
||||
free = lever_free(tu, raw, fn)
|
||||
except Refuse:
|
||||
continue
|
||||
cands = recipe_candidates(free, tu, fn, names, cap=a.cap)
|
||||
if not cands:
|
||||
continue
|
||||
tried += 1
|
||||
hit = None
|
||||
for rec, desc, cand in cands:
|
||||
v, dt, err = judge(tu, cand)
|
||||
compiles += len(recs_for(tu))
|
||||
if v == "IDENTICAL":
|
||||
hit = (rec, desc, cand)
|
||||
break
|
||||
if n_body % 10 == 0 or hit: # R55: a lane that runs unattended leaves evidence
|
||||
print(f" [{n_body}/{len(todo)}] {won} closed · {compiles} compiles · {(time.time() - t0) / 60:.1f} min",
|
||||
flush=True)
|
||||
row = dict(ts=time.strftime("%Y-%m-%d %H:%M:%S"), label=a.label, rung="R", calib=dict(head=oracle.head(), stamp=oracle.config_stamp()),
|
||||
tu=tu, fn=fn, addr=fn_addr(fn, tu), aliases=r.get("aliases"), header=tu.endswith(".h"),
|
||||
nhash_before=r.get("nhash_after"), nhash_after=None, candidates=len(cands), pins=len(names),
|
||||
verdict=("LEVER-FREE" if hit else "RESIDUE"), recipe=(hit[0] if hit else None), how=(hit[1] if hit else None),
|
||||
sites=([] if hit else r.get("sites", [])))
|
||||
if hit:
|
||||
path.write_text(hit[2], errors="surrogateescape")
|
||||
# the body's own markers are now orphans (its levers are gone); every OTHER body's marker is still honest, so the
|
||||
# scrub is scoped to this body's line span — a file-wide scrub would leave the census with UNMARKED sites elsewhere
|
||||
recs_ = sc.scan_text(hit[2], tu, shared_defs=None)
|
||||
d_ = next((x for x in recs_ if x["form"] == "def" and x["name"] == fn), None)
|
||||
span = range(d_["line"], d_["end"] + 1) if d_ else range(0)
|
||||
scrub = scrub_edits(hit[2], [i + 1 for i, l in enumerate(hit[2].split("\n")) if FAKE in l and i + 1 in span])
|
||||
if scrub:
|
||||
cand2 = apply_edits(hit[2], scrub)
|
||||
v2, _, _ = judge(tu, cand2)
|
||||
if v2 == "IDENTICAL":
|
||||
path.write_text(cand2, errors="surrogateescape")
|
||||
walk = lc.walk_file(path.read_text(errors="surrogateescape"), tu, tu.endswith(".h"))
|
||||
row["nhash_after"] = next((x["nhash"] for x in walk["defs"] if x["name"] == fn), None)
|
||||
won += 1
|
||||
print(f" {tu}:{fn} — {hit[0]} {hit[1]} IDENTICAL ({len(names)} pin(s) gone)", flush=True)
|
||||
rows.append(row)
|
||||
ledger_append(rows)
|
||||
needed = sum(len([s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]) for _, r in todo)
|
||||
print(f"recipes: {won} of {tried} bodies closed lever-free ({needed} NEEDED sites in the {len(todo)} drawn), "
|
||||
f"{compiles} compiles in {(time.time() - t0) / 60:.1f} min")
|
||||
return 0
|
||||
|
||||
|
||||
def remap_body(ex_before, ex_after, sib_before):
|
||||
"""the exemplar's reshaped body, with its `func_/D_` addresses replaced by the sibling's — or (None, why).
|
||||
|
||||
A text class is "identical modulo addresses" (that IS the nhash), so the two old bodies' address tokens correspond
|
||||
one for one in order; the map they define is applied to the new body. This is what turns one crack into a whole
|
||||
class: the 134-copy classes are the reason the draw is ordered by copies. A ledger REPLAY cannot do it — the ledger
|
||||
replays a SITE SET, and a reshaped body is not one."""
|
||||
a, b = lc.NORM_SYM.findall(ex_before), lc.NORM_SYM.findall(sib_before)
|
||||
if len(a) != len(b):
|
||||
return None, f"{len(a)} address tokens in the exemplar, {len(b)} in the sibling"
|
||||
m = {}
|
||||
for x, y in zip(a, b):
|
||||
if m.setdefault(x, y) != y:
|
||||
return None, f"`{x}` maps to both `{m[x]}` and `{y}` — not one class"
|
||||
return lc.NORM_SYM.sub(lambda mm: m.get(mm.group(0), mm.group(0)), ex_after), None
|
||||
|
||||
|
||||
def propagate(a):
|
||||
"""--propagate TU FN: the body TU:FN was reshaped and banked; give every RESIDUE sibling of its class the same
|
||||
shape, with its own addresses, and judge each on its own objects."""
|
||||
tu, fn = a.propagate
|
||||
rows = load_ledger()
|
||||
chain = [r for r in rows if r.get("tu") == tu and r.get("fn") == fn
|
||||
and r.get("verdict") == "LEVER-FREE" and r.get("after_text")]
|
||||
if not chain:
|
||||
sys.exit(f"delever --propagate: no banked reshape of {tu}:{fn} in the ledger (its row must carry after_text)")
|
||||
# THE CLASS is what the body looked like when the campaign found it, so the key and the "before" text come from the
|
||||
# FIRST bank in this body's chain; the text to spread is the LAST one (a body reshaped, then tidied, has two rows, and
|
||||
# taking the last row's before-hash would look for siblings of a text only this body ever had).
|
||||
src_row = dict(chain[-1], nhash_before=chain[0]["nhash_before"], before_text=chain[0]["before_text"])
|
||||
key = src_row["nhash_before"]
|
||||
cur = {}
|
||||
for r in rows:
|
||||
if r.get("tu") and r.get("fn"):
|
||||
cur[(r["tu"], r["fn"])] = r
|
||||
sibs = [k for k, r in cur.items() if k != (tu, fn) and r.get("verdict") == "RESIDUE"
|
||||
and (r.get("nhash_after") or r.get("nhash_before")) == key]
|
||||
if a.only:
|
||||
sibs = [k for k in sibs if any(o in k for o in a.only)]
|
||||
sibs = sibs[:a.limit] if a.limit else sibs
|
||||
print(f"delever --propagate: {tu}:{fn} -> {len(sibs)} sibling(s) of class {key[:12]}", flush=True)
|
||||
if not sibs:
|
||||
return 1 # R68: an empty work list is a refusal, not a success
|
||||
ok = bad = 0
|
||||
for stu, sfn in sibs:
|
||||
path = REPO / stu
|
||||
raw = path.read_text(errors="surrogateescape")
|
||||
d = next((r for r in sc.scan_text(raw, stu, shared_defs=None) if r["form"] == "def" and r["name"] == sfn), None)
|
||||
if d is None:
|
||||
print(f" {stu}:{sfn}: not defined there — SKIPPED", flush=True)
|
||||
bad += 1
|
||||
continue
|
||||
ls = line_starts(raw)
|
||||
sib_before = raw[ls[d["line"] - 1]:ls[d["end"]]]
|
||||
if lc.norm_hash(sc.mask_text(sib_before)) != key:
|
||||
print(f" {stu}:{sfn}: its text is not this class any more — SKIPPED", flush=True)
|
||||
bad += 1
|
||||
continue
|
||||
body, why = remap_body(src_row["before_text"], src_row["after_text"], sib_before)
|
||||
if body is None:
|
||||
print(f" {stu}:{sfn}: {why} — SKIPPED", flush=True)
|
||||
bad += 1
|
||||
continue
|
||||
f = RUN / "propagate" / f"{stu.replace('/', '_')}__{sfn}.c"
|
||||
f.parent.mkdir(parents=True, exist_ok=True)
|
||||
f.write_text(body, errors="surrogateescape")
|
||||
r = subprocess.run([sys.executable, str(REPO / "tools/delever.py"), "--apply-body", stu, sfn, str(f),
|
||||
"--label", a.label, "--rung", src_row.get("rung") or "R", "--dirty-ok"],
|
||||
cwd=REPO, capture_output=True, text=True)
|
||||
line = ((r.stdout or r.stderr).strip().splitlines() or [""])[-1]
|
||||
print(f" {line[:200]}", flush=True)
|
||||
ok += r.returncode == 0
|
||||
bad += r.returncode != 0
|
||||
print(f"delever --propagate: {ok} of {len(sibs)} sibling(s) banked, {bad} refused")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def status():
|
||||
rows = load_ledger()
|
||||
done, ex = ledger_index(rows)
|
||||
@@ -1318,7 +1824,10 @@ def apply_body(a):
|
||||
nhash_before=nh_before, nhash_after=nh_after, source=src, verdict=("LEVER-FREE" if v == "IDENTICAL" else f"BODY-{v}"),
|
||||
sites=[dict(ord=i, kind=s["kind"], cls=s["cls"], detail=s["detail"], via=s.get("via", ""), line=s["line"], verdict="NEEDED",
|
||||
why="left by the author", oracle="") for i, s in enumerate(levers)],
|
||||
compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_])
|
||||
compiles=len(recs_), seconds=round(dt, 3), objects=[r["obj"] for r in recs_],
|
||||
# the body AS IT WAS: --propagate needs it to map this class's addresses onto a sibling's, and after the
|
||||
# write it exists nowhere else (the tree has moved on and the ledger is the record)
|
||||
before_text=before, after_text=new)
|
||||
if v == "IDENTICAL":
|
||||
path.write_text(cand, errors="surrogateescape")
|
||||
ledger_append([row])
|
||||
@@ -1569,6 +2078,47 @@ def selftest():
|
||||
plan_, total_, _ = make_plan(fake_bodies, rows, False, 10, None)
|
||||
if total_ != 1:
|
||||
fail("a copy with a judged text but no row of its own must still be drawn")
|
||||
# rung R's candidate generators, on a fixture whose every answer is known by hand
|
||||
RFIX = ("void rfix(int p)\n{\n int a = p;\n int b;\n int c;\n"
|
||||
" if (a == b) a = b & 3;\n c = a + 1;\n *(int *)(p + 4) = c;\n}\n")
|
||||
rd = dict(line=1, end=9)
|
||||
for line, want in [("s32 d = param_1;", True), ("ret = f();", False), ("u8 *p;", True), ("d = param_1;", False),
|
||||
("return x;", False), ("extern s32 D_1[];", True)]:
|
||||
if is_decl_line(line) != want:
|
||||
fail(f"is_decl_line({line!r}) != {want}")
|
||||
if top_level_ops("a & b") != [2] or top_level_ops("a && b") or top_level_ops("(s32 *)p") \
|
||||
or top_level_ops("*(s32 *)(p + 4) & 0xFF") != [16]:
|
||||
fail("top_level_ops: a binary commutative operator at depth 0, and nothing else")
|
||||
sw = commutative_swaps(RFIX, "src/x.c", "rfix", rd)
|
||||
if [s for s, _ in sw] != ["swap & @6", "swap + @7"]:
|
||||
fail(f"commutative_swaps found {[s for s, _ in sw]}")
|
||||
if "a = 3 & b;" not in sw[0][1] or "c = 1 + a;" not in sw[1][1]:
|
||||
fail("commutative_swaps must split at the assignment, never at `==`")
|
||||
R6FIX = ("void r6(int p)\n{\n int v;\n int w;\n v = *(int *)(p + 4);\n"
|
||||
" *(int *)(p + 4) = v & ~0x20;\n w = 3;\n *(int *)(p + 8) = w;\n}\n")
|
||||
inl = inline_single_set_temps(R6FIX, "src/x.c", "r6", dict(line=1, end=9))
|
||||
if [d for d, _ in inl] != ["inline v @5", "inline w @7"]:
|
||||
fail(f"inline_single_set_temps found {[d for d, _ in inl]}")
|
||||
if "= (*(int *)(p + 4)) & ~0x20;" not in inl[0][1] or "int v;" in inl[0][1]:
|
||||
fail("R6 must inline the expression at the use AND drop the now-dead declaration")
|
||||
bw = block_wraps(R6FIX, "src/x.c", "r6", dict(line=1, end=9))
|
||||
if [d for d, _ in bw][:2] != ["block @5", "do-while @5"] or "{ v = *(int *)(p + 4); }" not in bw[0][1]:
|
||||
fail(f"block_wraps: {[d for d, _ in bw][:3]} (the readable spelling first)")
|
||||
cands = recipe_candidates(RFIX, "src/x.c", "rfix", ["a", "b"])
|
||||
kinds_ = {r for r, _, _ in cands}
|
||||
if not {"R2", "R3", "R5"} <= kinds_ or any(c == RFIX for _, _, c in cands):
|
||||
fail(f"recipe_candidates: {kinds_} (the seed must never be a candidate)")
|
||||
if any("a = p;" in c.split("\n")[7] for _, _, c in cands if _ == "R3"):
|
||||
fail("R3 must place its assignment after the whole declaration run (C89)")
|
||||
# the address remap that propagates a reshape to a class (R48-adjacent: one crack, 134 banks)
|
||||
exb = "void func_80100000(void) { D_80200000 = func_80100004(); }"
|
||||
exa = "void func_80100000(void) { s32 t = func_80100004(); D_80200000 = t; }"
|
||||
sib = "void func_80300000(void) { D_80400000 = func_80300004(); }"
|
||||
got, why = remap_body(exb, exa, sib)
|
||||
if got != "void func_80300000(void) { s32 t = func_80300004(); D_80400000 = t; }":
|
||||
fail(f"remap_body produced {got!r} ({why})")
|
||||
if remap_body(exb, exa, "void func_80300000(void) { D_80400000 = 0; }")[0] is not None:
|
||||
fail("remap_body must refuse a sibling with a different token count")
|
||||
# the oracle's crash classification on its real message forms (R103)
|
||||
if not oracle.SIGNAL_LINE.search("bash: line 1: 3845091 Done mipsel-linux-gnu-cpp ...\n 3845092 Aborted (core dumped) | tools/bin/gcc-2.7.2-psx/cc1 -quiet\n"):
|
||||
fail("SIGNAL_LINE must match bash's job-status block")
|
||||
@@ -1792,6 +2342,13 @@ def main():
|
||||
ap.add_argument("--restore", action="store_true", help="restore every in-flight file from inflight.json")
|
||||
ap.add_argument("--status", action="store_true")
|
||||
ap.add_argument("--scrub", action="store_true", help="remove orphan !FAKE markers (a marker whose site is gone), byte-judged per file")
|
||||
ap.add_argument("--propagate", nargs=2, metavar=("TU", "FN"),
|
||||
help="give every RESIDUE sibling of this banked body's class the same shape, with its own addresses")
|
||||
ap.add_argument("--recipes", action="store_true",
|
||||
help="rung R: the cookbook's byte-neutral shape recipes tried mechanically on every RESIDUE body")
|
||||
ap.add_argument("--cap", type=int, default=60, help="--recipes: candidates tried per body (each is one compile)")
|
||||
ap.add_argument("--control", type=int, default=8, help="--recipes: how many LEVER-FREE bodies the control run reproduces (R39)")
|
||||
ap.add_argument("--limit", type=int, help="--recipes: stop after this many RESIDUE bodies")
|
||||
ap.add_argument("--apply-body", nargs=3, metavar=("TU", "FN", "FILE"))
|
||||
ap.add_argument("--rung", default="E")
|
||||
ap.add_argument("--allow-residue", action="store_true")
|
||||
@@ -1810,6 +2367,14 @@ def main():
|
||||
sys.exit(status())
|
||||
if a.scrub:
|
||||
sys.exit(scrub(a))
|
||||
if a.propagate:
|
||||
if not a.label:
|
||||
sys.exit("delever --propagate: --label is required (R48)")
|
||||
sys.exit(propagate(a))
|
||||
if a.recipes:
|
||||
if not a.label:
|
||||
sys.exit("delever --recipes: --label is required (R48: the ledger rows are keyed by it)")
|
||||
sys.exit(recipes(a))
|
||||
if a.apply_body:
|
||||
if not a.label:
|
||||
sys.exit("delever --apply-body: --label is required")
|
||||
|
||||
@@ -23,10 +23,12 @@ THE PIPELINE PER EXEMPLAR (probe-proven at S98, the T6 log entry):
|
||||
the permuter compiles must hold ONE .text function, because the masked scorer compares whole .text.
|
||||
3. cpp -P with the BUILD's own CPPFLAGS + -I<the TU's directory> (common.h, the prelude, engine_types.h and the GTE header
|
||||
expand here — p16_permute.make_base_c runs its own cpp WITHOUT includes and would lose them) -> draft.c.
|
||||
4. the target .s — tools/verbatim_target_s.py regenerates it from the EXTRACTED ROM IMAGE (R34: an independent
|
||||
oracle; our own source is the thing under test), into the exemplar's scratch dir, where p16_permute.setup reads it.
|
||||
5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds
|
||||
(pins -> regalloc, barriers/launders/keep-alives -> schedule, mixed -> regalloc).
|
||||
4. the target — the tree's OWN (levered) body compiled by the build's tail into a one-function object, so it
|
||||
carries the candidates' relocations by construction; `tools/verbatim_target_s.py` regenerates the ROM listing beside it
|
||||
and `match_one` must call that body a MATCH before the search starts (R34: the ROM image is the independent oracle that
|
||||
PROVES the target; R56: check the baseline). S99 learned this the hard way — see THE INSTRUMENT below.
|
||||
5. tools/permuter_ils.py — warm-restarting decomp-permuter, the weight profile chosen from the NEEDED sites' kinds AND
|
||||
the register each pin names (callee-saved -> regalloc, caller-saved -> cse, barriers/launders/keep-alives -> schedule).
|
||||
6. a score-0 winner is a CANDIDATE, never a bank: --bank puts the function's definition back through
|
||||
`delever.py --apply-body` (which refuses a body that still carries a class A/B lever and judges it on the object's bytes
|
||||
through every recipe of the TU), then re-folds the cpp-expanded GTE asm with `gte_consolidate.py --apply --rejudge`.
|
||||
@@ -36,6 +38,15 @@ SCRATCH IS KEYED BY ALIAS+FN (R48): `.run/P36/permuter/<alias>__<fn>/` — a fun
|
||||
overlays overlap in RAM, so two different bodies can both be `func_8013B274`), and the permuter's own scratch/winner paths are
|
||||
keyed by the bare name. Every attempt is recorded in `.run/P36/permuter/outcomes.jsonl`, which is also the skip list.
|
||||
|
||||
THE INSTRUMENT, AND WHY IT IS CHECKED BEFORE IT IS BELIEVED (S99). The first two campaigns returned 0 of 16 with a straight
|
||||
face. Both were the harness: the target had been ASSEMBLED FROM A DISASSEMBLY LISTING, which is a second toolchain with its own
|
||||
answers — objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or` (24 wrong words in
|
||||
one 234-instruction function), and a listing's %hi/%lo pairs come back resolved with no relocation while every candidate carries
|
||||
one. The permuter scored 28 for a body that IS byte-identical, so score 0 was unreachable and every NO-MATCH was its own. The
|
||||
control that names this in one line is `--positive-control`, and the base score of the tree's own body is now 0. R40: exonerate
|
||||
the instrument before attributing a failure to its subject; the campaign that skips this control cannot tell a hard population
|
||||
from a broken scorer.
|
||||
|
||||
NEVER RUN THE CAMPAIGN AS A HARNESS BACKGROUND TASK (the low-memory guard kills it): `setsid nohup … &` + a Monitor on the log.
|
||||
"""
|
||||
import argparse
|
||||
@@ -148,15 +159,32 @@ def exemplars(only=(), limit=None, include_done=False):
|
||||
needed = [s for s in r.get("sites", []) if s.get("verdict") == "NEEDED"]
|
||||
out.append(dict(nhash=nh, tu=tu, fn=fn, alias=(r.get("aliases") or [None])[0], copies=len(members),
|
||||
needed=len(needed), kinds=sorted({s["kind"] for s in needed}),
|
||||
regs=sorted({s.get("detail", "") for s in needed if s["kind"] == "pin"}),
|
||||
members=[dict(tu=t, fn=f) for (t, f), _ in members], row=r))
|
||||
out.sort(key=lambda e: (-e["copies"], e["needed"], e["tu"], e["fn"]))
|
||||
out = [e for e in out if matches(e, only)]
|
||||
return out[:limit] if limit else out
|
||||
|
||||
|
||||
def klass_for(kinds):
|
||||
"""the permuter_weights profile for a residue's NEEDED-site mix (the profile NAME is accepted verbatim by classify())."""
|
||||
CALLEE_SAVED = {f"${n}" for n in range(16, 24)} | {f"$s{n}" for n in range(8)} | {"$fp", "$30"}
|
||||
|
||||
|
||||
def klass_for(kinds, regs=()):
|
||||
"""the permuter_weights profile for a residue, from the NEEDED sites' kinds AND the REGISTER each pin names (the profile
|
||||
name is accepted verbatim by classify()).
|
||||
|
||||
The register is the part S99 learned by reading a residual instead of assuming one. A pin on a CALLEE-SAVED register
|
||||
($16-$23) is an allocation-ORDER residual — the regalloc profile's declaration/statement reordering is its lever. A pin
|
||||
on a CALLER-SAVED one ($2/$3/$4-$7/…) is not: the S99 control on func_80163EC8 (`register … __asm__("$2")`) left exactly
|
||||
a v0/v1 swap — `lw v1,68(s1); li v0,-33; and v0,v1,v0` against `lw v0,68(s1); li v1,-33; and v0,v0,v1` — which is the
|
||||
operand ORDER of one `&`, and the regalloc profile weights `perm_commutative` 2.0 while the cse profile weights it 40.0.
|
||||
Steering that search by "it was a pin, so regalloc" spends the whole budget 20x away from the one lever that closes it."""
|
||||
ks = set(kinds)
|
||||
rs = {r for r in regs if r}
|
||||
if "pin" in ks and rs and rs & CALLEE_SAVED:
|
||||
return "regalloc"
|
||||
if "pin" in ks and rs and not (rs & CALLEE_SAVED):
|
||||
return "cse"
|
||||
if ks and ks <= SCHED_KINDS:
|
||||
return "schedule"
|
||||
return "regalloc"
|
||||
@@ -169,12 +197,17 @@ _sites_cache = None
|
||||
|
||||
|
||||
def tu_sites(tu):
|
||||
"""the census's sites for one TU. Built ONCE, into a local, and published under the lock: the workers are threads, and
|
||||
publishing the empty dict before filling it (2 s for 53,355 rows) let every other worker read "no site in this TU" and
|
||||
report a whole batch UNSTRIPPABLE — a race that lies in the tool's own voice."""
|
||||
global _sites_cache
|
||||
if _sites_cache is None:
|
||||
_sites_cache = collections.defaultdict(list)
|
||||
for s in dl.load_sites():
|
||||
_sites_cache[s["tu"]].append(s)
|
||||
return _sites_cache[tu]
|
||||
with _LOCK:
|
||||
if _sites_cache is None:
|
||||
built = collections.defaultdict(list)
|
||||
for s in dl.load_sites():
|
||||
built[s["tu"]].append(s)
|
||||
_sites_cache = built
|
||||
return _sites_cache.get(tu, [])
|
||||
|
||||
|
||||
def body_sites(tu, fn):
|
||||
@@ -318,7 +351,7 @@ def scratch_of(alias, fn):
|
||||
return RUN / f"{alias}__{fn}"
|
||||
|
||||
|
||||
def prepare(ex, quiet=False):
|
||||
def prepare(ex, quiet=False, require_sites=True):
|
||||
"""(scratch dir, draft path, target path, klass) for one exemplar — the whole probe-2 pipeline."""
|
||||
tu, fn = ex["tu"], ex["fn"]
|
||||
alias = ex["alias"] or alias_of(tu, ex.get("row"))
|
||||
@@ -327,7 +360,7 @@ def prepare(ex, quiet=False):
|
||||
ex["alias"] = alias
|
||||
raw = (REPO / tu).read_text(errors="surrogateescape")
|
||||
sites = body_sites(tu, fn)
|
||||
if not sites:
|
||||
if not sites and require_sites:
|
||||
raise Unstrippable([("<census>", 0, f"the census has no site in {tu}:{fn} (stale? rerun lever_census --sites)")])
|
||||
d = scratch_of(alias, fn)
|
||||
if d.exists():
|
||||
@@ -357,10 +390,18 @@ def prepare(ex, quiet=False):
|
||||
if r.returncode or not tgt.exists():
|
||||
raise Unstrippable([("<target>", 0, ((r.stderr or r.stdout).strip().splitlines() or ["no listing"])[-1][:160])])
|
||||
shutil.copy(tgt, d / sub / f"{fn}.s") # p16_permute.setup / match_one.py read <dir>/<fn>.s
|
||||
# THE TARGET OBJECT: the tree's own (levered) body through the build's own tail, so it carries the candidates'
|
||||
# relocations by construction. Its fidelity is not assumed — `closeness(levered)` must be MATCH against the ROM
|
||||
# listing above before any search runs (R34: the independent oracle proves the target; R56: check the baseline).
|
||||
r = subprocess.run(["tools/permuter/compile.sh", str(d / "levered.c"), "-o", str(d / "target.o")],
|
||||
capture_output=True, text=True, cwd=REPO)
|
||||
if r.returncode or not (d / "target.o").exists():
|
||||
raise Unstrippable([("<target.o>", 0, ((r.stderr or r.stdout).strip().splitlines() or ["compile failed"])[-1][:160])])
|
||||
if not quiet:
|
||||
n = len((d / "draft.c").read_text(errors="surrogateescape").splitlines())
|
||||
print(f" prepared {alias}__{fn}: draft {n} lines, target {(r.stdout or '').strip().split()[-3:]}", flush=True)
|
||||
return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"])
|
||||
print(f" prepared {alias}__{fn}: draft {n} lines, target.o {(d / 'target.o').stat().st_size} bytes "
|
||||
f"+ the ROM listing", flush=True)
|
||||
return d, d / "draft.c", d / "gas" / f"{fn}.s", klass_for(ex["kinds"], ex.get("regs", ()))
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
@@ -382,6 +423,52 @@ def closeness(d, fn, c):
|
||||
return (int(mm.group(1)) if mm else None), first
|
||||
|
||||
|
||||
def positive_control(a):
|
||||
"""Can rung D close a gap it is KNOWN to be able to close? Take a body the tree already matches, perturb it by ONE
|
||||
reversible source change (the first commutative operand swap `delever` can generate), confirm the perturbed body no
|
||||
longer matches, and give the permuter one cycle to find its way back to score 0.
|
||||
|
||||
Without this, "0 of 16" says nothing about the residue: it could equally be a harness that cannot reach ANY target
|
||||
(R40 — exonerate the instrument before attributing the failure to its subject). With it, a PASS means the search
|
||||
space, the target, the scorer and the winner path all work, and a 0-yield campaign is a fact about the population."""
|
||||
tu, fn = a.positive_control
|
||||
raw = (REPO / tu).read_text(errors="surrogateescape")
|
||||
d_ = next((r for r in sc.scan_text(raw, tu, shared_defs=None) if r["form"] == "def" and r["name"] == fn), None)
|
||||
if d_ is None:
|
||||
sys.exit(f"delever_permute --positive-control: {fn} is not defined in {tu}")
|
||||
swaps = dl.commutative_swaps(raw, tu, fn, d_)
|
||||
if not swaps:
|
||||
sys.exit(f"delever_permute --positive-control: {tu}:{fn} has no commutative operator to perturb — pick another body")
|
||||
ex = dict(nhash="", tu=tu, fn=fn, alias=alias_of(tu), copies=1, needed=0, kinds=[], regs=[], members=[], row={})
|
||||
d, draft, tgt, _ = prepare(ex, quiet=False, require_sites=False)
|
||||
base, base_line = closeness(d, fn, draft)
|
||||
desc, perturbed = swaps[a.which]
|
||||
(d / "perturbed_tu.c").write_text(perturbed, errors="surrogateescape")
|
||||
pert = drop_file_scope_asm(cpp_expand(tu, isolate(tu, perturbed, fn)), tu)
|
||||
(d / "draft.c").write_text(pert, errors="surrogateescape") # the permuter's seed IS the perturbed body
|
||||
after, after_line = closeness(d, fn, d / "draft.c")
|
||||
print(f"positive control {alias_of(tu)}__{fn}: the tree's own body {base_line[:40]} · perturbed by `{desc}` {after_line[:60]}",
|
||||
flush=True)
|
||||
if base != 0:
|
||||
sys.exit("delever_permute --positive-control: the UNPERTURBED body does not match — calibrate first (R56)")
|
||||
if after == 0:
|
||||
sys.exit(f"delever_permute --positive-control: `{desc}` is byte-neutral here — nothing to find; try --which {a.which + 1}")
|
||||
rel = d.relative_to(REPO).as_posix()
|
||||
log = d / "positive.log"
|
||||
cmd = [PY, "tools/permuter_ils.py", fn, "--draft", rel + "/draft.c", "--asm-subdir", rel + "/gas",
|
||||
"--klass", "cse", "--cycles", str(a.cycles), "--secs", str(a.secs), "--j", str(a.j),
|
||||
"--winners", rel, "--pd", rel + "/pd"]
|
||||
with open(log, "w") as f:
|
||||
f.write(" ".join(cmd) + "\n\n")
|
||||
f.flush()
|
||||
subprocess.run(cmd, cwd=REPO, stdout=f, stderr=subprocess.STDOUT, timeout=a.cycles * (a.secs + 45) + 300)
|
||||
out = log.read_text(errors="replace")
|
||||
won = (d / f"{fn}.c").exists()
|
||||
print(f"positive control: {'PASS — the permuter recovered score 0' if won else 'FAIL — it did not, in '
|
||||
f'{a.cycles}x{a.secs}s'} ({[l for l in out.splitlines() if 'cycle' in l][-1:] or ['no cycle line']})")
|
||||
return 0 if won else 1
|
||||
|
||||
|
||||
def calibrate_one(ex, quiet=False):
|
||||
"""the control: the body AS THE TREE HAS IT (levers and all) must be MATCH against the regenerated target. It proves the
|
||||
whole chain — the target from the ROM image, the isolation, the cpp expansion, the pinned triple — is measuring THIS
|
||||
@@ -405,7 +492,7 @@ BEST_RE = re.compile(r"best score = (\d+)")
|
||||
def run_one(ex, secs, cycles, j, max_start=None):
|
||||
t0 = time.time()
|
||||
row = dict(kind="attempt", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=ex["nhash"], tu=ex["tu"], fn=ex["fn"],
|
||||
alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"],
|
||||
alias=ex["alias"], copies=ex["copies"], needed=ex["needed"], kinds=ex["kinds"], regs=ex.get("regs", []),
|
||||
secs=secs, cycles=cycles, j=j)
|
||||
try:
|
||||
d, draft, tgt, klass = prepare(ex)
|
||||
@@ -437,7 +524,7 @@ def run_one(ex, secs, cycles, j, max_start=None):
|
||||
log = d / "ils.log"
|
||||
cmd = [PY, "tools/permuter_ils.py", ex["fn"], "--draft", (draft.relative_to(REPO)).as_posix(),
|
||||
"--asm-subdir", rel + "/gas", "--klass", klass, "--cycles", str(cycles), "--secs", str(secs),
|
||||
"--j", str(j), "--winners", rel, "--pd", rel + "/pd"]
|
||||
"--j", str(j), "--winners", rel, "--pd", rel + "/pd", "--target-o", rel + "/target.o"]
|
||||
row["klass"] = klass
|
||||
row["cmd"] = " ".join(cmd)
|
||||
rc = None
|
||||
@@ -485,10 +572,34 @@ def winner_body(winner_c, fn, tu):
|
||||
return None
|
||||
|
||||
|
||||
def tidy_body(body):
|
||||
"""the permuter's winner, made to read like the tree it lands in — or None when there is nothing to tidy.
|
||||
|
||||
pycparser reprints a body it has parsed: two-space indent where this tree uses four, and an empty statement `;` left
|
||||
where a statement was inlined away. Neither generates code, and the object oracle judges the tidied text before it is
|
||||
kept, so this can only ever improve the source. What it deliberately does NOT touch is the parenthesisation
|
||||
(`*((s32 *) (p + 4))`) or the brace style — that is the formatting phase's job, with clang-format, over the whole tree
|
||||
at once (`decomp-architect/templates/.clang-format`); a per-bank reformat would make this phase's diffs unreadable."""
|
||||
lines = [l for l in body.split("\n") if l.strip() != ";"]
|
||||
indents = [len(l) - len(l.lstrip(" ")) for l in lines if l.strip() and l.startswith(" ")]
|
||||
if indents and min(indents) == 2: # a two-space body: double every leading run
|
||||
lines = [(" " * (2 * (len(l) - len(l.lstrip(" ")))) + l.lstrip(" ")) if l.startswith(" ") else l for l in lines]
|
||||
out = "\n".join(lines)
|
||||
return out if out != body else None
|
||||
|
||||
|
||||
def bank(a):
|
||||
outs = load_outcomes()
|
||||
banked = {(o["alias"], o["fn"], o["nhash"]) for o in outs if o.get("kind") == "bank" and o.get("applied")}
|
||||
# a body ANOTHER rung already closed is not banked again (S99: --bank re-applied the permuter's reprinted body over the
|
||||
# one-line version rung R had banked for the same function, and the tree lost a clean diff for a byte-identical one)
|
||||
cur = {}
|
||||
for r in dl.load_ledger():
|
||||
if r.get("tu") and r.get("fn"):
|
||||
cur[(r["tu"], r["fn"])] = r
|
||||
already = {(k[0], k[1]) for k, r in cur.items() if r.get("verdict") == "LEVER-FREE"}
|
||||
todo = [o for o in outs if o.get("kind") == "attempt" and o.get("verdict") == "MATCH"
|
||||
and (o["tu"], o["fn"]) not in already
|
||||
and (o["alias"], o["fn"], o["nhash"]) not in banked
|
||||
and (not a.only or any(x in (o["fn"], o["tu"], o["alias"]) or o["nhash"].startswith(x) for x in a.only))]
|
||||
if not todo:
|
||||
@@ -503,11 +614,17 @@ def bank(a):
|
||||
bad += 1
|
||||
continue
|
||||
bf = d / "body.c"
|
||||
bf.write_text(body, errors="surrogateescape")
|
||||
cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(),
|
||||
"--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else [])
|
||||
r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
|
||||
print(" " + (r.stdout or r.stderr).strip().splitlines()[-1][:220], flush=True)
|
||||
# the tidied body FIRST (the oracle judges it like any other candidate); the winner verbatim only if it is refused
|
||||
forms = [(t_, "tidied") for t_ in [tidy_body(body)] if t_] + [(body, "verbatim")]
|
||||
r = None
|
||||
for text_, how in forms:
|
||||
bf.write_text(text_, errors="surrogateescape")
|
||||
cmd = [PY, "tools/delever.py", "--apply-body", o["tu"], o["fn"], bf.relative_to(REPO).as_posix(),
|
||||
"--label", a.label, "--rung", "D"] + (["--dirty-ok"] if a.dirty_ok else [])
|
||||
r = subprocess.run(cmd, cwd=REPO, capture_output=True, text=True)
|
||||
print(f" [{how}] " + (r.stdout or r.stderr).strip().splitlines()[-1][:210], flush=True)
|
||||
if r.returncode == 0:
|
||||
break
|
||||
row = dict(kind="bank", ts=time.strftime("%Y-%m-%d %H:%M:%S"), nhash=o["nhash"], tu=o["tu"], fn=o["fn"],
|
||||
alias=o["alias"], label=a.label, applied=(r.returncode == 0),
|
||||
apply_out=(r.stdout or r.stderr).strip()[-400:])
|
||||
@@ -535,7 +652,8 @@ def cmd_plan(a):
|
||||
f"({sum(e['copies'] for e in ex):,} bodies behind them); NEEDED kinds {dict(kinds)}")
|
||||
for e in ex[:a.show]:
|
||||
print(f" {e['copies']:4d} copies · {e['needed']:2d} needed {','.join(e['kinds']):<28} "
|
||||
f"{klass_for(e['kinds']):8} {e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}")
|
||||
f"{klass_for(e['kinds'], e.get('regs', ())):8} {','.join(e.get('regs', [])) or '-':<12} "
|
||||
f"{e['alias'] or alias_of(e['tu'], e['row'])} {e['tu']}:{e['fn']}")
|
||||
if len(ex) > a.show:
|
||||
print(f" … {len(ex) - a.show} more")
|
||||
return 0
|
||||
@@ -667,6 +785,8 @@ def main():
|
||||
g = ap.add_mutually_exclusive_group(required=True)
|
||||
g.add_argument("--plan", action="store_true")
|
||||
g.add_argument("--prepare", nargs=2, metavar=("TU", "FN"))
|
||||
g.add_argument("--positive-control", nargs=2, metavar=("TU", "FN"),
|
||||
help="perturb a MATCHING body by one commutative swap and require the permuter to find its way back")
|
||||
g.add_argument("--calibrate", action="store_true",
|
||||
help="the control: N exemplars' levered bodies must all be MATCH against their regenerated targets")
|
||||
g.add_argument("--run", action="store_true")
|
||||
@@ -682,6 +802,7 @@ def main():
|
||||
ap.add_argument("-j", type=int, default=16)
|
||||
ap.add_argument("--label", default="d1")
|
||||
ap.add_argument("--dirty-ok", action="store_true")
|
||||
ap.add_argument("--which", type=int, default=0, help="--positive-control: which commutative swap to perturb with")
|
||||
ap.add_argument("--max-start", type=int,
|
||||
help="skip (as FAR) an exemplar whose lever-free body is further than N instructions from the target")
|
||||
ap.add_argument("--include-done", action="store_true", help="draw classes that already have an outcome row")
|
||||
@@ -702,6 +823,8 @@ def main():
|
||||
d, draft, tgt, klass = prepare(ex)
|
||||
print(f"delever_permute --prepare: {d.relative_to(REPO)} (draft.c, {tgt.name}, klass={klass})")
|
||||
return 0
|
||||
if a.positive_control:
|
||||
return positive_control(a)
|
||||
if a.calibrate:
|
||||
return cmd_calibrate(a)
|
||||
if a.run:
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""lever_progress.py — the lever-removal series: what the compiler-forcing constructs cost, and how they came off.
|
||||
|
||||
tools/lever_progress.py --snapshot "T6 r1" # append today's census totals as a milestone row (evidence, with HEAD)
|
||||
tools/lever_progress.py --render # regenerate the generated block of docs/levers.md from the data
|
||||
tools/lever_progress.py --table # print both tables
|
||||
tools/lever_progress.py --check # is the last snapshot this tree's? (a stale series is a wrong chart)
|
||||
|
||||
WHY THIS EXISTS. The phase that takes the levers out is the only place the project ever counts them, and the count is worth
|
||||
more than the phase: it is the post-100% chart, the story's spine, and the evidence behind the day-one kit's advice on
|
||||
whether to prevent levers from the first bank or clean them up at the end. Numbers are DERIVED, never typed (R75).
|
||||
|
||||
TWO TABLES, TWO SOURCES:
|
||||
* MILESTONES — one appended row per census snapshot (`--snapshot`), because a census is a moment: it is not recoverable
|
||||
later. Each row carries the tree's HEAD and the census's own totals, so a reader can re-derive it from that commit.
|
||||
* THE CAMPAIGN — fully derived from `.run/P36/delever/ledger.jsonl` on every run: per batch label, the date, the rungs
|
||||
used, the bodies first judged, the sites REMOVED / REWRITTEN, and — scored as a TRANSITION against what that body's
|
||||
previous row left — the sites a later rung closed and the bodies it made lever-free. Rebuilt from scratch each render,
|
||||
so it cannot drift.
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(REPO / "tools"))
|
||||
|
||||
CENSUS = REPO / ".run" / "P36" / "census" / "lever_census.json"
|
||||
LEDGER = REPO / ".run" / "P36" / "delever" / "ledger.jsonl"
|
||||
SERIES = REPO / "docs" / "lever-progress.tsv"
|
||||
DOC = REPO / "docs" / "levers.md"
|
||||
BEGIN = "<!-- BEGIN GENERATED: lever_progress.py -->"
|
||||
END = "<!-- END GENERATED -->"
|
||||
COLS = ["date", "milestone", "head", "sites_AB", "pins", "asm", "bodies_AB", "distinct_AB", "marked", "unmarked",
|
||||
"gte_levers", "per_tu_asm_macros", "class_C", "class_D", "class_E", "class_F", "class_G"]
|
||||
|
||||
|
||||
def census():
|
||||
if not CENSUS.exists():
|
||||
sys.exit("lever_progress: no census — run tools/lever_census.py --sites -j 16")
|
||||
return json.loads(CENSUS.read_text())
|
||||
|
||||
|
||||
def ledger():
|
||||
rows = []
|
||||
if LEDGER.exists():
|
||||
for l in LEDGER.read_text().splitlines():
|
||||
if l.strip():
|
||||
rows.append(json.loads(l))
|
||||
return rows
|
||||
|
||||
|
||||
def snapshot_row(milestone):
|
||||
d = census()
|
||||
ab, cl = d["levers_AB"], d["classes"]
|
||||
return dict(date=d.get("generated", ""), milestone=milestone, head=d.get("head", ""),
|
||||
sites_AB=ab["sites"], pins=ab["pins"], asm=ab["asm"], bodies_AB=ab["bodies"],
|
||||
distinct_AB=ab["distinct_bodies"], marked=ab["marked"], unmarked=ab["unmarked"],
|
||||
gte_levers=d.get("gte_levers", {}).get("sites", 0),
|
||||
per_tu_asm_macros=d.get("per_tu_asm_macro_definitions", {}).get("total", 0),
|
||||
**{f"class_{k}": cl.get(k, {}).get("sites", 0) for k in "CDEFG"})
|
||||
|
||||
|
||||
def read_series():
|
||||
if not SERIES.exists():
|
||||
return []
|
||||
lines = [l for l in SERIES.read_text().splitlines() if l.strip()]
|
||||
if not lines:
|
||||
return []
|
||||
head = lines[0].split("\t")
|
||||
return [dict(zip(head, l.split("\t"))) for l in lines[1:]]
|
||||
|
||||
|
||||
def write_series(rows):
|
||||
SERIES.write_text("\t".join(COLS) + "\n" + "".join("\t".join(str(r.get(c, "")) for c in COLS) + "\n" for r in rows))
|
||||
|
||||
|
||||
def campaign():
|
||||
"""per batch label, derived from the ledger, as STATE TRANSITIONS — not as "the first row wins".
|
||||
|
||||
A body is judged more than once on purpose: rung A/B settles it with some sites left NEEDED, and a later rung (R, D or
|
||||
an agent) comes back and closes those. Counting only a body's first row credits the whole campaign to T4 and reports
|
||||
"0 removed" for the rung that actually finished the job (rung R's first batch closed 134 bodies and read as nothing).
|
||||
So each row is scored against what that body's previous row left: `removed`/`rewritten` are the first pass's, and
|
||||
`closed_later` is the fall in the NEEDED count afterwards, with `made_free` counting the bodies that reached zero."""
|
||||
prev, per = {}, collections.OrderedDict()
|
||||
for r in ledger():
|
||||
lab = r.get("label") or "-"
|
||||
key = (r.get("tu"), r.get("fn"))
|
||||
ts = (r.get("ts") or "")[:10]
|
||||
e = per.setdefault(lab, dict(label=lab, date=ts, rungs=set(), bodies=0, removed=0, rewritten=0,
|
||||
closed_later=0, made_free=0, refused=0))
|
||||
if ts and (not e["date"] or ts < e["date"]):
|
||||
e["date"] = ts
|
||||
if r.get("rung"):
|
||||
e["rungs"].add(str(r["rung"]))
|
||||
sites = r.get("sites", [])
|
||||
now_needed = sum(1 for s in sites if s.get("verdict") == "NEEDED")
|
||||
was = prev.get(key)
|
||||
if was is None:
|
||||
e["bodies"] += 1
|
||||
e["removed"] += sum(1 for s in sites if s.get("verdict") == "REMOVED")
|
||||
e["rewritten"] += sum(1 for s in sites if s.get("verdict") == "REWRITTEN")
|
||||
e["refused"] += sum(1 for s in sites if s.get("verdict") == "REFUSED")
|
||||
else:
|
||||
e["closed_later"] += max(0, was - now_needed)
|
||||
if was > 0 and now_needed == 0:
|
||||
e["made_free"] += 1
|
||||
prev[key] = now_needed
|
||||
return list(per.values())
|
||||
|
||||
|
||||
def md_tables():
|
||||
rows = read_series()
|
||||
out = [BEGIN, "", "### Milestones — the count, at each moment a census ran", "",
|
||||
"| date | milestone | pins | asm | **class A+B sites** | bodies | distinct | GTE levers | per-TU asm macros | C | D | E | F | G | HEAD |",
|
||||
"|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|"]
|
||||
for r in rows:
|
||||
out.append("| {date} | {milestone} | {pins} | {asm} | **{sites_AB}** | {bodies_AB} | {distinct_AB} | {gte_levers} | "
|
||||
"{per_tu_asm_macros} | {class_C} | {class_D} | {class_E} | {class_F} | {class_G} | `{head}` |".format(**r))
|
||||
out += ["", "### The campaign — every batch, from the ledger (derived on every render)", "",
|
||||
"| batch | date | rungs | bodies first judged | sites removed | rewritten | sites closed later | bodies made lever-free |",
|
||||
"|---|---|---|---:|---:|---:|---:|---:|"]
|
||||
tot = dict(bodies=0, removed=0, rewritten=0, closed_later=0, made_free=0)
|
||||
for e in campaign():
|
||||
if not any(e[k] for k in tot):
|
||||
continue # a bookkeeping batch (a scrub, a re-mark) moved no site
|
||||
out.append(f"| `{e['label']}` | {e['date']} | {','.join(sorted(e['rungs'])) or '-'} | {e['bodies']:,} | "
|
||||
f"{e['removed']:,} | {e['rewritten']:,} | {e['closed_later']:,} | {e['made_free']:,} |")
|
||||
for k in tot:
|
||||
tot[k] += e[k]
|
||||
out.append(f"| **total** | | | **{tot['bodies']:,}** | **{tot['removed']:,}** | **{tot['rewritten']:,}** | "
|
||||
f"**{tot['closed_later']:,}** | **{tot['made_free']:,}** |")
|
||||
out += ["", f"*Generated by `tools/lever_progress.py --render` from `.run/P36/census/lever_census.json` "
|
||||
f"and `.run/P36/delever/ledger.jsonl`; the series lives in `docs/lever-progress.tsv` (R75: published "
|
||||
f"numbers are generated, never typed).*", "", END]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def render():
|
||||
if not DOC.exists():
|
||||
sys.exit(f"lever_progress: {DOC.relative_to(REPO)} does not exist yet")
|
||||
t = DOC.read_text()
|
||||
if BEGIN not in t or END not in t:
|
||||
sys.exit(f"lever_progress: {DOC.relative_to(REPO)} has no generated block ({BEGIN} … {END})")
|
||||
pre, rest = t.split(BEGIN, 1)
|
||||
_, post = rest.split(END, 1)
|
||||
DOC.write_text(pre + md_tables() + post)
|
||||
print(f"lever_progress: rendered {len(read_series())} milestone row(s) + {len(campaign())} batch row(s) "
|
||||
f"into {DOC.relative_to(REPO)}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--snapshot", metavar="MILESTONE", help="append today's census totals as a milestone row")
|
||||
ap.add_argument("--render", action="store_true")
|
||||
ap.add_argument("--table", action="store_true")
|
||||
ap.add_argument("--check", action="store_true")
|
||||
a = ap.parse_args()
|
||||
if a.snapshot:
|
||||
rows = read_series()
|
||||
row = snapshot_row(a.snapshot)
|
||||
rows = [r for r in rows if not (r.get("milestone") == row["milestone"] and r.get("head") == row["head"])]
|
||||
rows.append(row)
|
||||
write_series(rows)
|
||||
print(f"lever_progress: {row['milestone']} — {row['sites_AB']:,} class A+B sites "
|
||||
f"({row['pins']:,} pins + {row['asm']:,} asm) in {row['bodies_AB']:,} bodies, "
|
||||
f"{row['unmarked']} unmarked, at {row['head']}")
|
||||
if a.render or a.snapshot:
|
||||
if DOC.exists():
|
||||
render()
|
||||
if a.table:
|
||||
print(md_tables())
|
||||
if a.check:
|
||||
rows, d = read_series(), census()
|
||||
if not rows:
|
||||
sys.exit("lever_progress --check: the series is empty (run --snapshot)")
|
||||
last = rows[-1]
|
||||
if last.get("head") != d.get("head") or int(last.get("sites_AB", -1)) != d["levers_AB"]["sites"]:
|
||||
sys.exit(f"lever_progress --check: the series' last row ({last.get('milestone')}, {last.get('sites_AB')} sites "
|
||||
f"at {last.get('head')}) is not this tree ({d['levers_AB']['sites']} at {d.get('head')}) — "
|
||||
f"run --snapshot after the census (R75)")
|
||||
print(f"lever_progress --check: OK — {len(rows)} milestone(s), last {last['milestone']} "
|
||||
f"({last['sites_AB']} class A+B sites)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -165,6 +165,43 @@ def to_gas(insns, vaddr, off, symtab):
|
||||
return out
|
||||
|
||||
|
||||
def gas_roundtrip(rows, insns, fn):
|
||||
"""[(row index, ROM word)] for every emitted instruction whose RE-ASSEMBLY does not reproduce the ROM's word.
|
||||
|
||||
WHY (S99, and it invalidated a whole campaign before it was found): objdump prints the PSEUDO-instruction
|
||||
`move s2,a0` for `addu s2,a0,$zero` (0x00809021), and gas assembles `move` as `or` (0x00809025). Every `move` in
|
||||
the listing therefore came back a different word, so a target object built from this form differed from the ROM
|
||||
in every one of them — and the permuter, scoring against it, reported 28 for a body that IS byte-identical. A
|
||||
listing nothing ever assembled and compared is a claim, not an oracle (R98/DK-81 in a second place). Words that
|
||||
carry a RELOCATION (jal/j, HI16/LO16) are excluded: the linker fills those, and both sides mask them."""
|
||||
body = "\n".join([f".include \"macro.inc\"", ".set noat", ".set noreorder", ".section .text", f"glabel {fn}"] + rows)
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".s", delete=False) as fh:
|
||||
fh.write(body + "\n")
|
||||
spath = fh.name
|
||||
opath = spath[:-2] + ".o"
|
||||
try:
|
||||
r = subprocess.run(["mipsel-linux-gnu-as", "-I", os.path.join(REPO, "include"), "-march=r3000", "-mtune=r3000",
|
||||
"-no-pad-sections", "-O1", "-G0", spath, "-o", opath], capture_output=True, text=True)
|
||||
if r.returncode:
|
||||
return None, (r.stderr or "").strip().splitlines()[:3]
|
||||
sys.path.insert(0, os.path.join(REPO, "tools"))
|
||||
import masked_diff
|
||||
got = masked_diff.insns_from_object(opath, None)
|
||||
finally:
|
||||
for p in (spath, opath):
|
||||
if os.path.exists(p):
|
||||
os.unlink(p)
|
||||
if len(got) != len(insns):
|
||||
return None, [f"re-assembly produced {len(got)} instructions, the image has {len(insns)}"]
|
||||
bad = []
|
||||
for k, (g, (word, _text)) in enumerate(zip(got, insns)):
|
||||
if g["reloc_kind"]:
|
||||
continue
|
||||
if g["word"] != word:
|
||||
bad.append((k, word))
|
||||
return bad, None
|
||||
|
||||
|
||||
def emit(binary, fn, outdir, quiet=False, gas=False):
|
||||
fr = _fr()
|
||||
vaddr, nins = func_extent(binary, fn)
|
||||
@@ -195,12 +232,34 @@ def emit(binary, fn, outdir, quiet=False, gas=False):
|
||||
fh.write(f' * thing under test. {nins} instructions at 0x{vaddr:08X}. $-registers, .L<addr> labels for\n')
|
||||
fh.write(f' * in-function targets, symbol names for external jumps; the listing carries its delay-slot nops\n')
|
||||
fh.write(f' * (hence .set noreorder). Paste WHOLE into decomp.me — its PS1 prelude defines glabel. */\n\n')
|
||||
rows = to_gas(insns, vaddr, off, _symtab(binary))
|
||||
# the listing must RE-ASSEMBLE to the image's own words, or it is not a target (S99). Each row that does
|
||||
# not is replaced by its `.word`, with the mnemonic kept in the comment for the reader; then it is
|
||||
# verified again, and a listing that still disagrees is REFUSED rather than emitted (R43).
|
||||
patched = 0
|
||||
for _ in range(3):
|
||||
bad, err = gas_roundtrip(rows, insns, fn)
|
||||
if bad is None:
|
||||
return None, f'{binary}:{fn}: the gas form does not assemble ({err})'
|
||||
if not bad:
|
||||
break
|
||||
# `rows` interleaves `.L<addr>:` label lines with instructions; `bad` counts INSTRUCTIONS. Indexing rows
|
||||
# by an instruction index rewrote a label into a comment and the next round refused to assemble.
|
||||
ins_rows = [i for i, r in enumerate(rows) if r.startswith('/*')]
|
||||
for k, word in bad:
|
||||
head, _, text = rows[ins_rows[k]].partition('*/')
|
||||
rows[ins_rows[k]] = f'{head}*/ .word 0x{word:08X} /* {text.strip()} */'
|
||||
patched += len(bad)
|
||||
else:
|
||||
return None, (f'{binary}:{fn}: {len(bad)} instruction(s) still do not re-assemble to the image '
|
||||
f'(first at +{bad[0][0] * 4:#x}) — REFUSING to emit a target that is not the bytes')
|
||||
fh.write('.set noat\n.set noreorder\n\n.section .text\n\n')
|
||||
fh.write(f'glabel {fn}\n')
|
||||
for ln in to_gas(insns, vaddr, off, _symtab(binary)):
|
||||
for ln in rows:
|
||||
fh.write(ln + '\n')
|
||||
if not quiet:
|
||||
print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form)')
|
||||
print(f' {binary:14s} {fn:26s} {nins:5d} ins @ 0x{vaddr:08X} -> {os.path.relpath(path, REPO)} (gas form'
|
||||
+ (f', {patched} word-patched)' if patched else ')'))
|
||||
return path, None
|
||||
path = os.path.join(outdir, binary, f'{fn}.s')
|
||||
with open(path, 'w') as fh:
|
||||
|
||||
@@ -233,7 +233,7 @@ def klass_for_fn(fn):
|
||||
return "", ""
|
||||
|
||||
|
||||
def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None):
|
||||
def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None, target_o=None):
|
||||
# `outdir` keys the scratch dir by the CALLER's identity instead of the bare function name (R48). A name is not
|
||||
# unique across the fleet — the overlays overlap in RAM, so two different bodies are both `func_8013B274` — and two
|
||||
# concurrent runs on one name would share (and corrupt) this directory. Default: the historical path, unchanged.
|
||||
@@ -247,6 +247,22 @@ def setup(fn, draft_c, asm_subdir=ASM, klass=None, where="", outdir=None):
|
||||
f"'not found in base.c' and no-op in 0s. Inspect {pd}/base.c "
|
||||
f"(prep order: comments -> cpp macros -> M2C_FIELD -> hide_asm -> typedefs).")
|
||||
open(f"{pd}/base.c", "w").write(base)
|
||||
if target_o:
|
||||
# THE TARGET AS AN OBJECT, not a listing (S99). Assembling a disassembly listing is a second toolchain with its
|
||||
# own answers: objdump prints the pseudo-instruction `move` for `addu rX,rY,$zero` and gas assembles it as `or`
|
||||
# (24 words wrong in one 234-instruction function), and a listing's %hi/%lo pairs come back RESOLVED, with no
|
||||
# relocation, while every candidate carries one — and the masked scorer compares reloc operands. Both made the
|
||||
# scorer report a nonzero floor for a body that IS byte-identical, so score 0 was unreachable and every verdict
|
||||
# was the instrument's. A target object compiled from the tree's own body by THIS pipeline has the candidate's
|
||||
# relocations by construction; the ROM listing stays the independent oracle that PROVES it (match_one MATCH
|
||||
# before the search starts, R34/R56) — the proof is what keeps this from being circular.
|
||||
shutil.copy(target_o, f"{pd}/target.o")
|
||||
body, prof = permuter_weights.render_settings_toml(fn, klass=klass, where=where)
|
||||
open(f"{pd}/settings.toml", "w").write(body)
|
||||
csh = "compile_o0.sh" if asm_subdir.rstrip("/").endswith("_o0") else "compile.sh"
|
||||
open(f"{pd}/compile.sh", "w").write(f'#!/bin/bash\nexec {REPO}/tools/permuter/{csh} "$@"\n')
|
||||
os.chmod(f"{pd}/compile.sh", 0o755)
|
||||
return pd
|
||||
s = os.path.join(REPO, asm_subdir, fn + ".s")
|
||||
tgt = f"{pd}/target.s"
|
||||
with open(tgt, "w") as f:
|
||||
|
||||
@@ -41,12 +41,14 @@ def main():
|
||||
ap.add_argument("--secs", type=int, default=180, help="per-cycle time box")
|
||||
ap.add_argument("--j", type=int, default=12)
|
||||
ap.add_argument("--winners", default=".run/permuter-winners")
|
||||
ap.add_argument("--target-o", help="use this one-function object as the target instead of assembling "
|
||||
"<asm-subdir>/<fn>.s (S99: a disassembly listing is a second toolchain)")
|
||||
ap.add_argument("--pd", help="scratch dir for this run (default .run/permuter/<fn>) — key it by alias+fn when "
|
||||
"several runs share a function NAME (R48)")
|
||||
a = ap.parse_args()
|
||||
|
||||
draft = open(a.draft).read()
|
||||
pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd)
|
||||
pd = P.setup(a.fn, draft, asm_subdir=a.asm_subdir, klass=a.klass, outdir=a.pd, target_o=a.target_o)
|
||||
if not pd:
|
||||
print("ILS setup FAILED (target .s didn't assemble?)"); sys.exit(1)
|
||||
print(f"ILS {a.fn}: {a.cycles} cycles x {a.secs}s @ -j{a.j}, klass={a.klass}")
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
> its platform SDK need the marked adaptation. The last table lists the tools that are project-only in code (their *shape* is a task;
|
||||
> their code does not transfer). *TODO(platform): the MIPS and PlayStation SDK hard-codes are the ones another platform replaces first.*
|
||||
>
|
||||
> **Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded), of which 301 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 21 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 331 (the installer checks its copy against this figure).
|
||||
> **Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded), of which 302 live rows
|
||||
> below; per phase: P1 2 · P2 26 · P3 17 · P4 9 · P5 26 · P6 51 · P7 20 · P8 86 · P9 27 · P10 22 · PROJECT-ONLY 16. Superseded tools appear only as pointers to their successor (30 pointer rows); one-offs are omitted. Table rows in all: 332 (the installer checks its copy against this figure).
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -335,6 +335,7 @@
|
||||
| `uniquify_type.py` | give each conflicting camp of a same-named type its own name | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | repo src layout |
|
||||
| `verbatim_check.py` | guard that every inline-assembly body still reproduces its target bytes | Regression guard that every inline-assembly body still reproduces its target bytes | repo src layout |
|
||||
| `delever_oracle.py` | judge one translation-unit edit by the bytes of its object in under a second | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | the Makefile's object rules, the twin rule |
|
||||
| `lever_progress.py` | keep the lever count as a series a chart and a story can be drawn from | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | the census json, the delever ledger |
|
||||
| `share_census.py` | measure duplicate function bodies across the fleet and assert one source per unique function | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | repo paths, the registry and signature schemas |
|
||||
| `ghidra_apply_symbols.sh` | mirror the curated symbol file into the analysis database with a real save | Mirrors the curated symbol file into the analysis program headlessly, with a real save | repo symbol path, project name |
|
||||
| `frozen.py` | refuse, from one place, the command line of a tool the project has frozen | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | nothing |
|
||||
@@ -343,7 +344,7 @@
|
||||
| `share_body_cycle.sh` | run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | repo paths, the phase-log format, the fleet count from config/check.*.sha |
|
||||
| `delever_permute.py` | search the compiler's own shape space for a lever-free body that keeps the bytes | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | the ledger, the permuter harness, the target regenerator |
|
||||
| `share_body.py` | share one byte-identical function class across its binaries through an include-at-site header, gated per binary | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | repo paths, the registry and signature schemas |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `delever.py` | take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | the census site records, the oracle |
|
||||
| `verbatim_to_stub.py` | turn an inline-assembly body back into a stub the toolchain can reach | Turns an inline-assembly body back into an include-assembly stub | repo src/asm layout |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
+6
-5
@@ -5,7 +5,7 @@ freshness). The derived columns come from the tree on every run; the authored on
|
||||
dictionary, whose coverage is asserted both ways. Read it by NEED: find the phrase that matches what you are trying to do, then the tool,
|
||||
then what proved it. The same data generates the day-one kit's manifest and its verbatim tool corpus.*
|
||||
|
||||
**Coverage:** 301 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 30 (a SETUP row only), ORPHAN 30 (neither) — of 301. Portability: PORTABLE 23, ADAPT 261, PROJECT-ONLY 17.
|
||||
**Coverage:** 302 tool files in scope (submodules, vendored and downloaded code excluded; `find` and `git ls-files` agree) + 38 retired under `tools/sunset/`. Classes: LIVE 241 (a runtime consumer), REFERENCED 31 (a SETUP row only), ORPHAN 30 (neither) — of 302. Portability: PORTABLE 23, ADAPT 262, PROJECT-ONLY 17.
|
||||
|
||||
## P1 — extraction + manifest
|
||||
|
||||
@@ -71,7 +71,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
|
||||
| When you need to… | Tool | What it does | Proven by | Adapt | Class |
|
||||
|---|---|---|---|---|---|
|
||||
| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+6) | MIPS relocation encodings | LIVE |
|
||||
| compare instructions with relocations masked, shared by matcher and scorer | `masked_diff.py` | Shared relocation-masked instruction comparison used by the matcher and the permuter scorer | diff_regions.py, masked_scorer.py, match_one.py, reg_renumber_swap.sh (+7) | MIPS relocation encodings | LIVE |
|
||||
| compile one function standalone and compare its masked bytes to the target | `match_one.py` | Compiles one function standalone with the pinned toolchain, masks relocations, compares to target bytes | ab_score.py, api_draft.py, aprop_autodraft.py, asm_verbatim.py (+23) | compiler triple, repo build flags | LIVE |
|
||||
| dump every compiler pass file for a self-contained draft | `cc1_dumps.sh` | Dumps every compiler pass file for a self-contained draft into a private directory | cc1_dumps_tu.sh, ghost_census.py | compiler triple, repo scratch paths | LIVE |
|
||||
| dump every compiler pass file for the spliced real translation unit | `cc1_dumps_tu.sh` | Same pass dumps for the spliced real translation unit, the faithful compile | alloc_table.py | absolute repo path, compiler triple | LIVE |
|
||||
@@ -295,7 +295,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| build a progress timeline from the repository's own committed digests | `timeline.py` | Builds a progress timeline from the repository's own committed digests, with a self-check | Makefile | repo doc paths | LIVE |
|
||||
| check that every relative link in the public docs resolves | `doc_links.py` | Checks that every relative link in the public docs resolves and the link policy holds | Makefile, wiki_render.py, wiki_sync.sh | repo doc paths | LIVE |
|
||||
| compile every eligible translation unit with the pinned toolchain, using no derived bytes | `compile_only.py` | Compiles every eligible translation unit with the pinned toolchain, without any game bytes | .github/workflows/no-rom.yml, delever_oracle.py, lever_census.py, macro_to_header.py (+1) | compiler triple, repo makefile parsing | LIVE |
|
||||
| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+33) | repo src/config layout | LIVE |
|
||||
| compute and publish progress metrics as report, machine data and per-binary breakdowns | `progress.py` | Computes and publishes the progress metrics as report, JSON and per-binary breakdowns | Makefile, docs/wave-playbook.md, api_agent.py, atlas.py (+34) | repo src/config layout | LIVE |
|
||||
| convert progress metrics into an external progress-report schema | `objdiff_report.py` | Converts the progress JSON into the external progress-report schema | .github/workflows/progress.yml | external report schema | LIVE |
|
||||
| emit the public ordinal-to-hash map and the private old-to-new map | `public_rewrite/build_commit_map.py` | Emits the public ordinal-to-new-hash map and the private old-to-new map | probe_github.sh, resolve_tokens.py | repo doc/scratch paths | LIVE |
|
||||
| gate a first push: prove no derived bytes among tracked files | `audit_public.py` | The first-push gate: no derived bytes among tracked files, four independent checks | .github/workflows/no-rom.yml, gate_scan.py | repo purge-set and path lists (the kit's template reads `config/firewall.txt` instead) | LIVE |
|
||||
@@ -322,7 +322,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
|---|---|---|---|---|---|
|
||||
| add per-site function-pointer casts so a draft can call a differently typed callee | `cast_call_sites.py` | Adds per-site function-pointer casts so a draft can call a differently typed callee | canon_sig_reconcile.py, cast_self_callers.py, family_sweep.py, gate_stage.py (+4) | repo src layout | LIVE |
|
||||
| apply curated names and signatures inside the analysis tool and save | `ghidra_scripts/ApplySymbols.java` | The in-tool half of that mirror: apply curated names and signatures, save on exit | ghidra_apply_symbols.sh, ghidra_export_annotations.sh, ghidra_rebuild.sh, ExportAnnotations.java (+1) | none | LIVE |
|
||||
| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+2) | repo paths, the shared-body scanner | LIVE |
|
||||
| census every compiler-forcing construct in the C and gate the levers-off phase | `lever_census.py` | The Phase-36 census of lever sites (register pins, asm statements by kind, volatile levers, bare register, asm-label aliases, builtins, attributes; the whole-body hand-asm routines and the GTE ops set apart) derived from the shared-body scanner with a per-token coverage assertion against the raw text, four known-true controls, the !FAKE marker split, --check (0 unmarked pins/asm, 0 orphan markers) and --strict (0 pins, 0 asm) gates, --sites for the delever ledger, a fixture self-test | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py (+3) | repo paths, the shared-body scanner | LIVE |
|
||||
| consolidate the GTE coprocessor inline-asm macros under Sony's names in one header and sweep dead lever macros | `gte_consolidate.py` | The Phase-36 T5 tool: every asm-bearing macro definition and direct GTE statement SIGNED by the build's own maspsx→as tail (template bytes with operands bound to fixed registers, operand counts, clobbers); one canonical text per signature named by PsyQ's inline_c.h convention (Sony's file supplies names and clobber lists, never opcode words) written to include/gte_inline.h; per file the canonical duplicates deleted, private names renamed, a variant with extra clobbers tried as canonical and kept as a marked <name>_m lever only when the object differs, direct statements rewritten into canonical calls (a concatenation of two included); every file judged through every recipe; --sweep deletes dead asm-bearing macros and drops a compound macro's inner asm when byte-neutral | delever.py, delever_cycle.sh, delever_permute.py, lever_census.py | Sony's inline_c.h path, the census sites, the oracle | LIVE |
|
||||
| convert a shared-body macro header into per-function plain-C headers included at each site | `macro_to_header.py` | The Phase-35 converter: every DEFINE_ macro body becomes a plain-C header under src/shared/<space>/ included at its site, the legacy name-parameterized headers converted, the registry text-edited, the macro header deleted; --plan / --apply / --finalize / --verify | share_body.py, tool_census.py | the macro form is this project's; a kit-born project shares headers from its first bank | LIVE |
|
||||
| emit a function's target assembly as an inline-assembly body | `asm_verbatim.py` | Emits the file-scope inline-assembly body form from a disassembly file | docs/wave-playbook.md, draft_prechecks.py, gate_main.py, recover_route.py | repo asm layout | LIVE |
|
||||
@@ -331,6 +331,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED |
|
||||
| guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, lever_census.py, progress.py, verbatim_target_s.py (+1) | repo src layout | LIVE |
|
||||
| judge one translation-unit edit by the bytes of its object in under a second | `delever_oracle.py` | The Phase-36 fast byte oracle: every object's exact build command captured once via make -n -W (the Makefile's own recipe, pad stage and -O0 overrides included), a candidate compiled in place with -o/-MF redirected to scratch and compared with the fleet run's object; --calibrate proves 100 % equality untouched, twin == primary and a positive control before any verdict is trusted | delever.py, delever_cycle.sh, delever_permute.py, gte_consolidate.py | the Makefile's object rules, the twin rule | LIVE |
|
||||
| keep the lever count as a series a chart and a story can be drawn from | `lever_progress.py` | The Phase-36 lever series: a milestone row appended per census snapshot (its totals by class with the tree's HEAD, since a census is a moment that cannot be recovered later) plus the campaign table derived from the de-lever ledger on every render — per batch the rungs used, the bodies first judged, the sites removed and rewritten, and, scored as a transition against what that body's previous row left, the sites a later rung closed and the bodies it made lever-free; renders both into docs/levers.md's generated block and --check refuses a series that is not this tree's | — | the census json, the delever ledger | REFERENCED |
|
||||
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, delever.py (+10) | repo paths, the registry and signature schemas | LIVE |
|
||||
| mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED |
|
||||
| refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE |
|
||||
@@ -339,7 +340,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
|
||||
| run the share-body batch cycle unattended: batch, verify by exit code, log, commit, periodic clean fleet check | `share_body_cycle.sh` | The Phase-35 T5 driver for bucket new: per batch share_body --apply --bucket new --batches 1 --label new<k>, the gated N/N line read, the phase log entry appended, the bank committed the moment it is green, the clean fleet run every N batches — stops on the first red | — | repo paths, the phase-log format, the fleet count from config/check.*.sha | ORPHAN |
|
||||
| search the compiler's own shape space for a lever-free body that keeps the bytes | `delever_permute.py` | The Phase-36 rung D: one exemplar per residue text class from the delever ledger (largest class first — a match banks every copy), each prepared as a single-function translation unit (delever's rung-A rewrite of every removable site, every other definition reduced to a prototype, shared-header includes to their prototypes, INCLUDE_ASM and file-scope asm dropped, the build's own CPPFLAGS through cpp -P) against a target regenerated from the ROM image in both the assemblable and the splat form; every attempt calibrates itself first (the LEVERED body must be MATCH, or the harness is not measuring that function) and records the lever-free body's starting distance; decomp-permuter through permuter_ils with the weight profile chosen from the NEEDED kinds; a score-0 winner is banked only through delever --apply-body and the GTE re-fold; scratch, winners and the outcome ledger keyed by alias+fn | — | the ledger, the permuter harness, the target regenerator | REFERENCED |
|
||||
| share one byte-identical function class across its binaries through an include-at-site header, gated per binary | `share_body.py` | The Phase-35 successor of dedup_propagate + dedup_extend for the include-at-site form: exemplar by majority text, the header written once, every private copy replaced by the include at its position, per-binary byte gate with object comparison, bisect on red, the registry appended or extended by text, the exception ledger on refusal; --plan / --apply --bucket extend|new | aprop_autodraft.py, audit_binaries.py, auto_driver.py, blocker_probe.py (+18) | repo paths, the registry and signature schemas | LIVE |
|
||||
| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
|
||||
| take register pins, asm statements, volatile and register levers out of matched C, byte-gated per site | `delever.py` | The Phase-36 de-lever engine and campaign tool: positional rewrites per lever class (pin -> plain declaration, barrier/keep-alive deleted, a launder deleted or turned into the assignment it is, a hand-placed instruction -> its C, the zero-register variable -> 0, a macro-carried site deleted/valued/refused by its macro's shape, volatile/register dropped); per body replay -> rung A strip-all -> rung B greedy through delever_oracle; the file is the write unit and its final compile through every recipe the proof; --plan/--apply batches (TUs parallel, exemplar files first; headers serial, includers parallel), the ledger keyed by tu+fn+addr with the body's nhash before/after, !FAKE markers on class A/B survivors, --redraw for bodies refused by an older tool, --restore from inflight.json, --scrub for orphan markers, --apply-body for T6/T7, --recipes (rung R: the byte-neutral shape recipes — the formerly-pinned declarations permuted, one moved through the declaration run, an initializer split — each judged by the oracle, with an identity control on both the splice and the oracle), --selftest with a stub oracle, --probe (T2) | delever_cycle.sh, delever_permute.py, gte_consolidate.py, lever_census.py | the census site records, the oracle | LIVE |
|
||||
| turn an inline-assembly body back into a stub the toolchain can reach | `verbatim_to_stub.py` | Turns an inline-assembly body back into an include-assembly stub | — | repo src/asm layout | REFERENCED |
|
||||
|
||||
## PROJECT-ONLY — project-only in code (the shape is a task; the code does not transfer)
|
||||
|
||||
@@ -62,7 +62,7 @@
|
||||
- Rules check (P6) after T4.
|
||||
- ☐ **T5** — GTE consolidation (one header, PsyQ's names; the `"memory"`-clobber variants are levers → residue, not a second spelling) +
|
||||
the dead lever-macro sweep (`SHB` cross-file use counted). Verify: `lever_census` 0 per-TU asm macro definitions; R22 green.
|
||||
- ☐ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed,
|
||||
- ☑ **T6** — Rung R (mechanical shape recipes, each negative-controlled) and rung D (the permuter on the residue exemplars, time-boxed,
|
||||
16 workers; a score-0 applied through `delever --apply-body` and gated). Verify: `recipes: <k> of <r> NEEDED sites removed`;
|
||||
`permuter: <k> of <r> exemplars matched lever-free in <t> h`.
|
||||
- ☐ **T7** (Ultracode, prompted) — The reshaping waves to zero: wave → harvest → toolify (cookbook + rung R) → sweep → next draw; wave 1
|
||||
@@ -621,7 +621,83 @@ accumulate here as the phase produces them.**
|
||||
refused more than a third of the population while the context was hot, and harvest→toolify applied to lever classes so a shape
|
||||
sweeps the banked population instead of the lever propagating with every family remap (1,758 distinct bodies became 12,578).
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — S98 (2026-09-09): T0 ☑ T1 ☑ T1b ☑ T2 ☑ T3 ☑ T4 ☑ T5 ☑ — all committed (this close on top of `4bc987426`); NEXT = T6 BUILD: `tools/delever_permute.py` per §2 (the design is settled and its pipeline proven by two probes — the T6 log entry); S98 ended here at 90 % context | the number: 34,091 sites in 12,712 bodies (1,759 distinct) · marked 34,091 · UNMARKED 0 · orphans 0 · GTE levers 462 · per-TU asm macro definitions 314 (0 canonical duplicates) — `lever_census --check` OK
|
||||
- **S99 — T6 CLOSE: both yield lines measured, 664 lever sites gone, R22 green at every step. T6 ☑.**
|
||||
**The two lines the plan asks for (R41):** `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of 2,131 bodies
|
||||
behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in 6.0 min`.
|
||||
**Banked this session:** rung R's batch `r1` (134 bodies, every one `R7 do-while @<line>`, a ONE-LINE source change each) ·
|
||||
rung D's four remaining exemplars (`d1`, tidied and judged) · the propagation `p1`/`p1b` (132 + 130 + 131 + 132 = **525
|
||||
siblings**, 0 refused, each judged on its own objects). **34,091 → 33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies,
|
||||
33,427 marked, 0 UNMARKED, 0 orphans; `check-all: 218 passed, 0 failed of 218` after every batch.
|
||||
**What T6 leaves for T7, stated rather than smoothed over:** rung R's recipe set does NOT reproduce the shape rung D found on
|
||||
func_80163EC8 — R6 was generalised from "assigned once, read once" to "dead after one read" (the lever was one of two
|
||||
assignments to `uVar5`) and still does not close it (154 candidates, 1.3 min, 0 hits), so that class was banked from the
|
||||
permuter's own winner instead. The 11 exemplars rung D did not close are seeds, not walls: every one improved (78→9, 52→3,
|
||||
50→3, 37→2, 131→35) and the ones that stayed far all removed a hand-placed `instruction` whose C spelling changed the
|
||||
instruction COUNT. A fleet-wide rung-R sweep is priced but not run: one body costs ~150 compiles / ~1.3 min at `--cap 200`
|
||||
when nothing hits, so the 12,000-body residue needs the run parallelised across TUs before it is affordable.
|
||||
**Instrument work banked with it:** `--recipes` is killable (its judge snapshots to inflight.json first — a killed run had
|
||||
left a candidate in `src/`); `--bank` skips a body another rung already closed (it had re-applied a permuter body over rung
|
||||
R's cleaner one-line version, restored through the oracle as `d1fix`); `--propagate` keys the class on the FIRST bank in a
|
||||
body's chain (the later row's before-hash describes a text only that body ever had — func_80163EC8 found 0 siblings until
|
||||
this was fixed, then 132); the permuter's winner is TIDIED before it is offered (pycparser's two-space indent and the corpse
|
||||
`;` where a statement was inlined away), with the tidy judged like any other candidate.
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — S99 (2026-09-09): T0–T5 ☑ **T6 ☑** — all committed (this close on top of `1c2355054`); NEXT = **T7, the reshaping waves — and T7 STARTS ONLY ON DREW'S DIRECT APPROVAL IN THE SESSION THAT RUNS IT** (his words, twice: the `/effort ultracode` toggle is NOT approval); until he gives it, the drawable work is a PARALLELISED rung-R sweep (§2) | the number: **33,427 sites** (19,982 pins + 13,445 asm) in 12,048 bodies · marked 33,427 · UNMARKED 0 · orphans 0 · GTE levers 462 — `lever_census --check` OK · `lever_progress --check` OK (3 milestones)
|
||||
|
||||
### 0. How to use this block
|
||||
A fresh session reads CLAUDE.md's load order, replays this block verbatim, confirms the effort (T7's coordinator at Max, the waves at
|
||||
Ultracode — Drew toggles, R27) and executes §2. The tree is clean at HEAD = this close commit. **Never run a long job as a harness
|
||||
background task** (the low-memory guard kills them; the harness also backgrounds any foreground command over 120 s, which is fine for
|
||||
`make` but not for a campaign) — run campaigns DETACHED with `setsid nohup … &` and a `Monitor` on the log. **The calibration is keyed to
|
||||
HEAD: EVERY commit stales it** — `tools/delever_oracle.py --calibrate ov_SC04_011 ov_SC03_015 ov_SC03_014 main -j 16` (3 s) before any
|
||||
judging, and calibrate the FULL set: a single-alias calibration fails its own completeness check. A killed batch: `tools/delever.py
|
||||
--restore`.
|
||||
|
||||
### 1. Where things stand
|
||||
- **Done: T0–T6.** T6 delivered both yield lines the plan asks for: `permuter: 5 of 16 exemplars matched lever-free in 0.69 h (665 of
|
||||
2,131 bodies behind them)` and `recipes: 134 of 134 bodies closed lever-free (134 NEEDED sites in the 134 drawn), 3243 compiles in
|
||||
6.0 min`. Banked: rung R `r1` (134 bodies) · rung D `d1` (4 exemplars) · propagation `p1`/`p1b` (525 siblings, 0 refused).
|
||||
**34,091 → 33,427 sites**; `check-all: 218 passed, 0 failed of 218` after every batch.
|
||||
- **THE INSTRUMENT LESSON THAT COST TWO CAMPAIGNS** (cookbook §454, accelerators P36 S99): the permuter's target had been ASSEMBLED
|
||||
FROM A DISASSEMBLY LISTING and scored **28 for a byte-identical body**, so score 0 was unreachable and two campaigns reported "0 of
|
||||
16" about a healthy population. The target is now the tree's OWN body compiled by the build's tail (`--positive-control` reads base
|
||||
score 0). **Run `tools/delever_permute.py --positive-control TU FN` before believing any future yield.**
|
||||
- **Instruments (SETUP + dictionary rows, kit corpus regenerated, `tool_census --check: OK`):** `tools/delever.py` (+ `--recipes`
|
||||
rung R with R2/R3/R4/R5/R6/R7, `--propagate TU FN`, `--cap`, `--control`), `tools/delever_permute.py` (`--plan`, `--prepare`,
|
||||
`--positive-control`, `--calibrate`, `--run`, `--bank`, `--status`, `--selftest`), `tools/lever_progress.py` (the series behind
|
||||
`docs/levers.md`), `tools/delever_oracle.py`, `tools/lever_census.py`, `tools/gte_consolidate.py`, `tools/delever_cycle.sh`.
|
||||
- **Drew's directive, mid-session:** the lever count over time is a DELIVERABLE — `docs/levers.md` + `docs/lever-progress.tsv` +
|
||||
`tools/lever_progress.py --snapshot "<task>"` **after every task that changes the count** (the post-100% chart, the story, a wiki
|
||||
page, and the kit's day-one rule). §5 of that document is the prevent-vs-defer argument, written from the generated numbers:
|
||||
**38% of the class A/B population came off with no understanding at all**, so the rule is *ban the silence, not the lever*.
|
||||
- **Environment:** WSL2, `~/bfm-decomp`, `.venv`; R22 clean fleet ≈ 115 s; the census ≈ 35 s; `make kit-corpus` ≈ 25 s; 16 cores.
|
||||
|
||||
### 2. What is drawable WITHOUT Drew's wave approval — the parallelised rung-R sweep
|
||||
Rung R is serial today: one body costs ~150 compiles / ~1.3 min at `--cap 200` when nothing hits, so the ~12,000-body residue is
|
||||
unaffordable as written. Make `recipes()` run TU-parallel the way `apply_batch` already does (a `ThreadPoolExecutor` over files, one
|
||||
worker owning a whole TU — two workers must never share a TU, and the oracle writes the candidate into the tree to compile it), then:
|
||||
1. `--cap 40` first over the whole residue (the targeted recipes R5/R6 and a few R7s) to measure the cheap yield with a denominator;
|
||||
2. `--cap 200` on what the cheap pass leaves, drawn by class size so a hit is worth 130 bodies;
|
||||
3. after each batch: R22 → `lever_census --sites --check` → `lever_progress --snapshot "<label>"` → commit (R42, R101).
|
||||
Also drawable: more rung-D exemplars (`delever_permute --run --limit N --workers 4 --secs 240 --cycles 3`), ordered by the STARTING
|
||||
DISTANCE the calibration prints — everything that closed started ≤ 29 mismatched instructions, everything ≥ 37 did not; `--max-start 35`
|
||||
triages the rest as FAR with their number.
|
||||
|
||||
### 3. T7, when Drew approves it
|
||||
One agent per residue exemplar with the pack (the body, the target listing, the ledger's NEEDED sites with the register each names,
|
||||
`alloc_table.py` on the real TU, §501-E/P/R + §17 + **§454**, `journal_notes`, "no lever of any class may remain", deliverable-first
|
||||
R67); `delever --apply-body` judges every draft on the bytes and `--propagate` spreads it to the class. The harvest → toolify gate
|
||||
between waves is not optional: every shape an agent finds becomes a rung-R recipe (R5/R6/R7 all came from rung D this way) and then
|
||||
sweeps the population for free.
|
||||
|
||||
### 4. Gotchas known before any work
|
||||
Everything in the T3/T4/T5 lists, plus: a disassembly listing is not a target (§454); `--only <fn>` in `--recipes` draws EVERY body of
|
||||
that name across the fleet, which is why `r1` banked a whole class in one run; the class key for `--propagate` is the FIRST bank in a
|
||||
body's chain; `pgrep -cf` matching your own command line counts itself (R79's cousin); the harness backgrounds any command over 120 s;
|
||||
`lever_progress --check` fails when the series is not this tree's.
|
||||
|
||||
|
||||
## (superseded) SESSION CHECKPOINT — S98 (2026-09-09): T0 ☑ T1 ☑ T1b ☑ T2 ☑ T3 ☑ T4 ☑ T5 ☑ — all committed (this close on top of `4bc987426`); NEXT = T6 BUILD: `tools/delever_permute.py` per §2 (the design is settled and its pipeline proven by two probes — the T6 log entry); S98 ended here at 90 % context | the number: 34,091 sites in 12,712 bodies (1,759 distinct) · marked 34,091 · UNMARKED 0 · orphans 0 · GTE levers 462 · per-TU asm macro definitions 314 (0 canonical duplicates) — `lever_census --check` OK
|
||||
|
||||
### 0. How to use this block
|
||||
A fresh session reads CLAUDE.md's load order, replays this block verbatim, confirms the effort (T6/T8/T9 xHigh per the plan; S98 ran at Max —
|
||||
|
||||
+7
-3
@@ -1705,10 +1705,14 @@ def propagate(a):
|
||||
shape, with its own addresses, and judge each on its own objects."""
|
||||
tu, fn = a.propagate
|
||||
rows = load_ledger()
|
||||
src_row = next((r for r in reversed(rows) if r.get("tu") == tu and r.get("fn") == fn
|
||||
and r.get("verdict") == "LEVER-FREE" and r.get("after_text")), None)
|
||||
if src_row is None:
|
||||
chain = [r for r in rows if r.get("tu") == tu and r.get("fn") == fn
|
||||
and r.get("verdict") == "LEVER-FREE" and r.get("after_text")]
|
||||
if not chain:
|
||||
sys.exit(f"delever --propagate: no banked reshape of {tu}:{fn} in the ledger (its row must carry after_text)")
|
||||
# THE CLASS is what the body looked like when the campaign found it, so the key and the "before" text come from the
|
||||
# FIRST bank in this body's chain; the text to spread is the LAST one (a body reshaped, then tidied, has two rows, and
|
||||
# taking the last row's before-hash would look for siblings of a text only this body ever had).
|
||||
src_row = dict(chain[-1], nhash_before=chain[0]["nhash_before"], before_text=chain[0]["before_text"])
|
||||
key = src_row["nhash_before"]
|
||||
cur = {}
|
||||
for r in rows:
|
||||
|
||||
Reference in New Issue
Block a user