diff --git a/docs/SETUP.md b/docs/SETUP.md index 4db43fed2..cdb61e556 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -766,6 +766,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/wiki_sync.sh [--push] [--wiki-url URL]` | **(P33 F3)** Render into `.run/wiki/render/` → clone or fast-forward `BFM-decomp.wiki.git` under `.run/wiki/` → REPLACE its pages with the rendered set (the repo is the source of truth; a page edited on GitHub is overwritten) → `git status --short`; **`--push` is Drew's (R6)**: commit + push. Before the wiki repo exists (after the flip AND the first page created in the GitHub UI) the dry run renders and lists (exit 0) and `--push` refuses (exit 2). | | | `tools/gccmap_cites.py [--dry-run \| --check \| --verify \| --controls \| --explain] [--retag]` | **(P33 E3)** Tag every `file.c:NNN` cite in `docs/gcc-2.7.2-map/*.md` with the source tree its line number belongs to — `[2.7.2]` (the vanilla subset), `[2.8.1 pm]` (gcc-papermario), `[repo]` — derived from the trees (quoted snippets, identifiers with function extents and nearest distance, the author's cues; a contradiction fails loudly; ties → `cite_overrides.tsv` → cue → 2.7.2); writes in place, idempotent, never silently changes an existing tag. `--check` is textual (every cite tagged, no stale override) and runs in `make tools-health` + CI; the others need both reference trees and refuse without them. | | | `tools/xsig/xsig.py sign-s \| sign-objdump \| cross \| verify \| selftest` | **(P33 E4; Phase-21 origin)** Relocation-masked per-function signatures for CROSS-PROJECT code identification: mask `j`/`jal` targets and HI16/LO16 immediates (from `%hi`/`%lo` operands or `objdump -dr` records), keep opcodes/registers/constants/branches; `cross` joins two JSONL sets on `sig` with a coverage line; `verify` prints the instruction-by-instruction diff (opcode/register/immediate/length). Self-contained (stdlib, MIT, own README + LICENSE + `tests/` from a game-free fixture compiled at two link addresses with `--emit-relocs`); `tests/test_xsig.py` (8) in `make tools-health` + CI. The standalone repo copy is prepared under `.run/P33/xsig-repo/` (Drew creates + pushes `Druthulu/xsig`). | +| | `tools/permuter/upstream/0001-reloc-masked-scorer.patch` | **(P33 E5)** The upstream PR as a `git format-patch` (one commit against `simonlindholm/decomp-permuter` main `41bd0bfc`, 2026-09-05): `src/reloc_scorer.py` (`RelocMaskedScorer`, a `Scorer` subclass), `--score-mode {mnemonic,reloc-masked}` + the `score_mode` settings key, docs, `test/test_reloc_scorer.py` (10, no cross toolchain). Regenerate the branch: clone upstream, `git am` the patch (proven clean). The scratch clone `.run/P33/permuter-upstream/` holds branch `reloc-masked-scorer` under the noreply identity; **Drew pushes it to his fork and opens the PR** (`docs/permuter-ils.md` §2). | | | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). | | | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data///` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. | | | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) | @@ -1145,6 +1146,35 @@ fills fast). Nothing is leaking — but the host does not get the memory back on `tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh [PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`. +### P33 E5 (S88, 2026-09-07) — the permuter upstream PR branch + `docs/permuter-ils.md` +- **The PR (PR-1).** `RelocMaskedScorer` in `src/reloc_scorer.py` — the in-tree `masked_diff.diff_object_object` rule + (mask from the TARGET's relocation records: `R_MIPS_26` → opcode only; the 16-bit-immediate relocs → opcode+rs+rt; a `j` + against `.text` compared relative to the function start; everything else the full word; symbol+addend equality at + masked slots; `score = mismatches + |Δlength|`, 0 iff link-identical) written for upstream `main` at `41bd0bfc` (39 + commits past our pin `b44b0622`; `Scorer.__init__` gained `ign_branch_targets` + `objdump_command`, which is why + `tools/permuter/run_masked.py`'s rebind targets the PIN, not HEAD). A `Scorer` subclass (no change to `Permuter` / + `Candidate`); `--score-mode {mnemonic,reloc-masked}` + `score_mode` in settings.toml (default unchanged); refused with + `-J` (`src/net/evaluator.py` builds a stock `Scorer`, `net/client.py` reads `scorer.stack_differences/algorithm` — the + subclass carries both); MIPS only (raises otherwise); USAGE/README/example_settings notes. +- **Verified at the commit:** `test/test_reloc_scorer.py` 10/10 on an embedded `objdump -drz` listing of the xsig + fixture object (real gcc 2.7.2 output, no game bytes; the end-to-end test feeds the scorer through + `objdump_command="cat"` with a 20-byte MIPS ELF header in front of the listing — `get_arch` reads ident + e_machine); + `black --check` clean on the touched files; `mypy` = upstream's baseline (5 pre-existing errors: `toml` stubs, + `Levenshtein`, `helpers.py` Any — none added; checked by stashing); `./run-tests.sh`: only `test_perm` fails (needs + `mips-linux-gnu-gcc`, absent here); **the real permuter end to end** on `.run/P33/permuter-e2e/` (our `compile.sh`, + `settings.toml` `objdump_command = "mipsel-linux-gnu-objdump -drz -m mips:4300"` — upstream looks for + `mips-linux-gnu-objdump`): base == target → base score 0 → `--stop-on-zero`; a swapped-operand `other()` (5 ins) → + reloc-masked base 4 → **score 0 at iteration 256** (<40 s, `-j4`) while the DEFAULT scorer read the same base as **3,585** + (3,420 after 20 iterations) — the floor in miniature. A dev venv (`.venv-dev/`, mypy 2.3.1 + black 26.5.1 + + toml + pycparser) lives in the scratch clone, never in the project venv. +- **PR-2 / the issue:** the MIPS symbol wildcard `"." in field` (`scorer.py:68/72`) — issue text in `docs/permuter-ils.md` + §3 (a configurable `symbol_regex`). **The recipe:** §4 of the same doc — the ILS warm restart and its five guards + (hidden pins re-hidden per cycle, refused-cycle abort, comment strip, flushed stdout, winner ≠ bank). +- **Drew:** fork upstream, `git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit + message is the description), then file the issue. The patch file is the tracked copy (R20); the scratch clone is + regenerable from it. + + ### P33 E4 (S88, 2026-09-07) — xsig packaged: `tools/xsig/` + the standalone repo, from the Phase-21 `.run/xdedup/xsig.py` - **What it is.** The Phase-21 cross-project dedup probe's signature library (`.run/xdedup/xsig.py`, 2026-06-25) plus the driver scripts around it (`sign_bfm.py`, `sign_xeno.py`, `cross.py`, `analyze.py`) folded into ONE stdlib file with a CLI diff --git a/docs/permuter-ils.md b/docs/permuter-ils.md new file mode 100644 index 000000000..2f90789cd --- /dev/null +++ b/docs/permuter-ils.md @@ -0,0 +1,122 @@ +# The permuter in BFM-decomp — the relocation-masked scorer, the ILS warm restart, and the upstream PR + +> **Status (P33 E5, 2026-09-07).** The recipe this project ran decomp-permuter with, written for two readers: a +> BFM maintainer (what `tools/permuter/` does and why) and the upstream project (the PR that carries the scorer, +> `tools/permuter/upstream/0001-reloc-masked-scorer.patch`, and the issue text for the symbol-wildcard defect). The +> permuter itself is the pinned submodule `tools/decomp-permuter` (commit `b44b0622`, 2024-12-30); the PR was written +> against upstream `main` at `41bd0bfc` (2026-09-05, 39 commits later). + +## 1. Why the stock scorer could not reach zero here + +decomp-permuter scores a candidate by diffing symbolised `objdump` rows against the target's and charging penalties +(register 5, reordering 60, insertion/deletion 100, …). On MIPS it recognises "this field is a relocation, ignore the +name" only when the field contains a `.` (`src/scorer.py`, `field_matches_any_symbol`: `return "." in field`) — true of +`.text+0x34`-style references, false of every symbol this project's splitter names (`func_80012345`, `D_80054C10`). +Every relocation slot that differed by *name* was therefore a register penalty the search could never reduce, and the +score floated on a floor: `func_80176D94` sat at base ≈225 / best ≈210 / never 0 while a byte-exact answer existed +(Phase 24 T2). A random walk with no gradient to zero diverges. + +## 2. The relocation-masked scorer (what the PR carries) + +Compare the two objects' instruction **words**, masking exactly the fields the linker fills in — read from the +**target's** relocation records, never guessed from a `lui` pattern — and require the relocation operands to agree there: + +| Target's relocation | Mask | Plus | +|---|---|---| +| `R_MIPS_26` (a `j`/`jal` to a symbol) | keep the 6-bit opcode, mask the 26-bit target | same symbol+addend | +| `HI16`/`LO16`/`PC16`/`LITERAL`/`GPREL16`/`GOT16`/`CALL16` | keep opcode + rs + rt, mask the low 16 bits | same symbol+addend | +| `R_MIPS_26` against `.text` (a `j` to a label inside the function) | as above | the target compared **relative to the function start** — two candidates that jump to different labels differ | +| none | the full 32-bit word | registers, constants and branch offsets all count | + +`score = mismatching positions + |length difference|` — one unit per instruction; **0 exactly when the candidate links +to the same bytes**. The in-tree form is [`tools/masked_diff.py`](../tools/masked_diff.py) (shared with `match_one` and +`rtu_match`, so the closeness an agent reads is the closeness the permuter optimises) rebound over the pinned permuter +by [`tools/masked_scorer.py`](../tools/masked_scorer.py) + [`tools/permuter/run_masked.py`](../tools/permuter/run_masked.py) +without editing the submodule. Four of its rules were bought with byte evidence and are in the PR too: + +- **`-drz`, not `-dr`.** objdump's default nop-elision under-counted GTE-heavy functions (`func_80132784` read as 384 + instructions instead of 400); `-z` keeps the runs. +- **The 26-bit field is masked only when the assembler left it to the linker.** A short-circuit on the *opcode* once + dropped every `j .L…` to a local label from the comparison — and which label a `j` takes is the difference between + `break` and `return` (`ov_SC03_118:func_801825EC`, one word). The internal-`j` target is compared relative to the + function start (P31 T1). +- **Keep the opcode even at a masked slot.** A mask of 0 at a `j`/`jal` position swallowed whatever the other side held + (a `j` vs a `bne` scored 0 one way and 1 the other — P31 S65). +- **`R_MIPS_PC16` is masked like `HI16`** (Phase 26-A): the object holds an unresolved placeholder in the branch + displacement, so a full-word compare could never succeed; masking it cured 151 of 155 false non-zeros across 60,740 + stubs and introduced none. + +**Upstream PR (PR-1).** `tools/permuter/upstream/0001-reloc-masked-scorer.patch` = one commit on the branch +`reloc-masked-scorer` of the scratch clone `.run/P33/permuter-upstream/` (regenerable: `git clone` upstream, `git am` +the patch — proven to apply onto `41bd0bfc`). It adds `src/reloc_scorer.py` (`RelocMaskedScorer`, a `Scorer` subclass +so `Permuter`/`Candidate` are untouched), `--score-mode {mnemonic,reloc-masked}` and a `score_mode` settings key +(default `mnemonic` = unchanged behaviour), a USAGE/README/example-settings note, a refusal with `-J` (remote evaluators +use the default scorer), and `test/test_reloc_scorer.py` — ten tests on an embedded `objdump -drz` listing of a small +gcc 2.7.2 object (link-time fields masked; symbol, register, opcode, branch and length differences each counted; +internal jump targets compared; the scorer end to end through `objdump_command`), needing no cross toolchain. +Verified at the commit: `black --check` clean on the touched files; `mypy` at upstream's own baseline (5 pre-existing +errors, none added); the suite's only failure is upstream's `test_perm`, which needs `mips-linux-gnu-gcc`; and the +real permuter run end to end with the mode on compiled objects: base == target → base score 0 → `--stop-on-zero`; +and **the floor, reproduced in miniature** — a 5-instruction function with two `xor` operands swapped in the base: +reloc-masked base score 4 (the debug diff names the four operand differences) and **score 0 at iteration 256**, under +40 s with `-j4`; the default scorer read the same base as **3,585** and sat at 3,420 twenty iterations later +(`.run/P33/permuter-e2e/`, `settings.toml` `objdump_command = "mipsel-linux-gnu-objdump -drz -m mips:4300"`). + +**What Drew does:** push the branch to his fork and open the PR against `simonlindholm/decomp-permuter`: + +```bash +git -C .run/P33/permuter-upstream remote add fork https://github.com/Druthulu/decomp-permuter.git # after forking +git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer +# then "Compare & pull request" on GitHub; the commit message is the PR description +``` + +## 3. The issue for upstream (PR-2, or an issue first): a configurable symbol regex + +Title: *MIPS scorer: `field_matches_any_symbol` recognises a relocation only when the field contains "."* + +> On MIPS, `Scorer.score`'s `field_matches_any_symbol` returns `"." in field` (`src/scorer.py`, the two branches +> for mips and arm32). Splitters such as splat name symbols `func_80012345` / `D_80054C10` — no dot — so a relocation +> slot whose symbol differs between target and candidate is charged as a register difference (`PENALTY_REGALLOC`) that +> no permutation can remove. The score then floats on a floor above zero even when a byte-exact candidate exists; we +> measured base ≈225 / best ≈210 / never 0 on a 225-instruction function that later matched. +> +> Proposal: a `symbol_regex` setting (and `--symbol-regex`), defaulting to the current behaviour (`\.`), that projects +> can set to their symbol shape (e.g. `^(func|D|jtbl)_[0-9A-Fa-f]{8}`), used wherever `field_matches_any_symbol` +> decides "this field is a symbol". Independent of, and complementary to, the relocation-masked scoring mode in PR-1 — +> that mode sidesteps the question by reading relocation records instead of symbol text, but the default scorer +> deserves the fix too. + +## 4. The ILS warm restart (a usage recipe, ours to keep) + +A cold permuter run on a pinned seed plateaus at the base score over ~10k iterations. **Warm-restarting `base.c` from +the best byte-waypoint each cycle**, with a fresh seed, descends where cold stalls: `func_80148094` 72 → 36 over ~8 +restarts, the big drops coming from fresh restarts, not from continuing a plateaued run (Phase 24 T7 §G). The wrapper +is [`tools/permuter_ils.py`](../tools/permuter_ils.py) over [`tools/p16_permute.py`](../tools/p16_permute.py): + +```bash +python3 tools/permuter_ils.py func_80148094 --draft .run/t7b/close/func_80148094.c \ + --asm-subdir asm/ov_SC01_077/nonmatchings/ov_SC01_077 --klass REGALLOC --cycles 10 --secs 180 --j 12 +``` + +Its guards, each bought with a session (P31 S79/S80, cookbook §493–§495): (1) **register pins are hidden** before the +permuter parses the seed — `register … __asm__("$16")` is carried in a base64 pragma so `cc1` still binds the register +while pycparser sees plain C — and a waypoint's `source.c` (which the permuter decodes back) is **re-hidden** before the +next cycle, or every cycle after the first is a parser refusal that reads as "unchanged" (`func_80020DA4`: 1 real cycle ++ 7 silent no-ops); (2) a **refused cycle aborts the run** with the reason — not-judged is not a verdict (R61); (3) the +seed's header comment is stripped before `cpp -P` (a `*/`-less comment made the permuter no-op silently on every +commented giant draft); (4) `stdout` is flushed — eight parallel runs once showed empty logs for twenty minutes (R55); +(5) **a score-0 winner is a candidate, not a bank**: intermediate waypoints can be semantically divergent (the permuter +rewrites stores for byte proximity), so the winner goes through `rtu_match` and the whole-binary gate. + +The measured limits, so the tool is used where it pays: the permuter's problem on this project was **targeting, not a +missing transform** (decision log 2026-07-21) — pointed at the right seed with the right scorer it closed +register-allocation and scheduling residuals; it never closed a structural residual (a wrong loop shape, a missing +idiom), which is the reader's job (`residual_class`, cookbook §66d: alternate a random search with a byte-verified idiom +and let the residual class decide whose turn it is). And a masked "1" is not a closeness until its diff is read (R63). + +## 5. Related + +- Cookbook §3 (the harness), §42/§45 (the F-band cracks where pins beat the permuter), §66d–§66d-5 (the permuter⇄reader + loop), §137 (REGALLOC-PERM as a two-compile arithmetic problem), §493–§495 (the S80 repairs). +- `docs/hindsight-study.md` §7 — mine the permuter's failures, not just its wins (the offline-automatic endgame). +- `docs/how-to-ai-decomp/07-compiler-source.md` — when the residual is the compiler's, read the pass instead. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 1cc4ce46a..585f9b806 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -81,7 +81,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo - [x] **F3** wiki + how-to-ai-decomp + `wiki_sync.sh` — Max (FULL, Drew 2026-09-07) — see Log 2026-09-07 F3 - [x] **E3** gcc-2.7.2 map README + `gccmap_cites.py` — xHigh — see Log 2026-09-07 E3 - [x] **E4** xsig packaging — xHigh — see Log 2026-09-07 E4 -- [ ] **E5** permuter upstream PR branch — Max +- [x] **E5** permuter upstream PR branch — Max (FULL, Drew 2026-09-07) — see Log 2026-09-07 E5 - [ ] **E6** drafter write-up — Max - [ ] **E1** decomp.me preset (after the flip) — xHigh - [ ] **E2** Archipelago outreach (after the flip) — xHigh @@ -534,22 +534,51 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact initial commit — **Drew creates `Druthulu/xsig` on GitHub (EMPTY) and pushes** (`git -C .run/P33/xsig-repo remote add origin … && git push -u origin main`). `make tools-health` re-run with the new line: **`tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` (EXIT=0)** (`.run/P33/e4_tools_health.log`). Commit: see below. +- **2026-09-07 (S88, Max) — E5 the permuter upstream PR branch + `docs/permuter-ils.md` — IN FULL (Drew's call).** + Scratch clone of `simonlindholm/decomp-permuter` at `main` `41bd0bfc` (2026-09-05; 39 commits past our pin `b44b0622`) → + branch **`reloc-masked-scorer`**, one commit under the noreply identity: `src/reloc_scorer.py` (`RelocMaskedScorer`, a + `Scorer` subclass — the in-tree `masked_diff` rule: mask from the TARGET's relocation records, `R_MIPS_26` → opcode only, + the 16-bit-immediate relocs → opcode+rs+rt, a `j` against `.text` compared relative to the function start, everything + else the full word, symbol+addend equality at masked slots, `score = mismatches + |Δlength|`, 0 iff link-identical; + MIPS only), `--score-mode {mnemonic,reloc-masked}` + the `score_mode` settings key (default unchanged), a refusal with + `-J` (remote evaluators build a stock `Scorer`), `--debug` per-instruction diff, USAGE/README/example_settings notes, + `test/test_reloc_scorer.py` (10 tests on an embedded `objdump -drz` listing of the xsig fixture object — real gcc 2.7.2 + output, no game bytes; the end-to-end test feeds `objdump_command="cat"` a file with a 20-byte MIPS ELF header ahead of + the listing, since `get_arch` reads ident + e_machine). **Verified at the commit:** 10/10; `black --check` clean; + `mypy` = upstream's baseline (5 pre-existing: `toml` stubs, `Levenshtein`, `helpers.py` Any — none added, checked by + stashing); `./run-tests.sh`: only upstream's `test_perm` fails (needs `mips-linux-gnu-gcc`); **the real permuter end to + end** (`.run/P33/permuter-e2e/`: our `compile.sh`, `settings.toml` with `objdump_command = "mipsel-linux-gnu-objdump + -drz -m mips:4300"` because upstream looks for `mips-linux-gnu-objdump`): base == target → base score 0 → + `--stop-on-zero`; **the floor in miniature** — `other()` (5 ins) with its `xor` operands swapped: reloc-masked base 4 → + **score 0 at iteration 256** (<40 s, `-j4`; the winner restored `(a * 3) ^ (b >> 2)`), while the DEFAULT scorer read the + same base as **3,585** and sat at 3,420 after 20 iterations. Two gotchas fixed on the way (both caught by the tests / + mypy before the commit): a `Match`-typed loop variable shadowed by the instruction loop (mypy union-attr ×3); the + stand-in object needing a real ELF header. **The tracked copy (R20):** `tools/permuter/upstream/ + 0001-reloc-masked-scorer.patch` (`git format-patch`; proven to `git am` cleanly onto upstream `main`; the one whitespace + warning — a doubled newline at USAGE.md's end — fixed and the commit amended). A dev venv (`.venv-dev/`: mypy 2.3.1, + black 26.5.1, toml, pycparser) lives in the scratch clone, never in the project venv. **`docs/permuter-ils.md`** (NEW; in + `doc_links`' default set, `--strict` PASS): §1 why the stock scorer floats (the `"." in field` wildcard), §2 the scorer + + the PR + the four byte-bought rules (`-drz`, the internal-`j` target, keep-the-opcode, PC16) + Drew's push/PR commands, + §3 the issue text for PR-2 (a configurable `symbol_regex`), §4 the ILS warm-restart recipe and its five guards + (`func_80148094` 72 → 36 over ~8 restarts; hidden pins re-hidden per cycle; refused-cycle abort R61; comment strip; + flushed stdout R55; winner ≠ bank), §5 related. SETUP: row + the P33 E5 section (R21). **Drew:** fork upstream, + `git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the + description), file the issue. Commit: see below. -## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓, E4 ✓ (31 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E5 (2026-09-07, written by session 4555f4e4 "S88" at the E4 close; SUPERSEDES the earlier blocks) +## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓, E4 ✓, E5 ✓ (32 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E6 (2026-09-07, written by session 4555f4e4 "S88" at the E5 close; SUPERSEDES the earlier blocks) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest -(R1–R73), then WAIT for Drew. **NEXT = E5** (Max, ≈1 session — the permuter upstream PR branch; it is the plan's last -named CUT CANDIDATE: ask Drew whether E5 runs in full, trimmed to the issue + `docs/permuter-ils.md`, or is cut — F3 and E3 -ran in full). Rebuild the harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3, E4 completed and -C10 in progress. **Every commit cites NEW +(R1–R73), then WAIT for Drew. **NEXT = E6** (Max, ≈0.5–1 session — the drafter write-up `docs/matching-drafter-pipeline.md`; +no cut candidates remain: F3, E3, E4, E5 all ran in full). Rebuild the harness task list (40 items, R28) marking A1–A5, +B1–B9/C3, C1–C9, D1–D5, F1–F3, E3, E4, E5 completed and C10 in progress. **Every commit cites NEW hashes only** (the history was rewritten; `docs/commit-map.tsv` maps ordinals → new hashes; the scratch `.run/public_rewrite/` holds the old ones and stays until the probe passes). **Never `git clean -x`** (CLAUDE.md fail-safe). ### 1. Where we are **Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). The approved plan -is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (31):** A1–A5, B1–B9/C3, +is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (32):** A1–A5, B1–B9/C3, C1–C9 (the rewrite, adopted, force-pushed by Drew, gc'd), D1–D5 (README, LICENSE/NOTICE/THIRD_PARTY, badges/objdiff/frogress, SETUP public-clean, governing docs + `doc_links`), F1 (timeline + story), F2 (retrospective), **F3 (S88: the wiki — 12 files under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `tools/wiki_render.py` + `tools/wiki_sync.sh`, @@ -557,18 +586,21 @@ under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `too + `cite_overrides.tsv`; all 135 map citations tagged `[2.7.2]`/`[2.8.1 pm]`/`[repo]` — 79/55/1 — verified, controls 6/6; `50c1b69e4d`), **E4 (S88: `tools/xsig/` — `xsig.py` library+CLI, README with the Phase-21 worked example, MIT LICENSE, `tests/` from a game-free fixture at two link addresses, 8/8; in tools-health + CI; the standalone copy at `.run/P33/xsig-repo/` -with one commit under the noreply identity — Drew creates `Druthulu/xsig` EMPTY and pushes; the commit that carries this -block).** **In progress (Drew, C10):** the GitHub +with one commit under the noreply identity — Drew creates `Druthulu/xsig` EMPTY and pushes; `9c4d32d651`), **E5 (S88: the +upstream PR branch `reloc-masked-scorer` in `.run/P33/permuter-upstream/` + its tracked copy +`tools/permuter/upstream/0001-reloc-masked-scorer.patch` + `docs/permuter-ils.md`; 10/10 tests, black clean, mypy at +upstream's baseline, the real permuter 4 → 0 in 256 iterations where the default scorer read 3,585 — Drew pushes the branch +to his fork and opens the PR + files the issue; the commit that carries this block).** **In progress (Drew, C10):** the GitHub Support ticket (text: `docs/public-flip-runbook.md` §11 — its filing was never confirmed to S88; ask) and the daily `tools/public_rewrite/probe_github.sh` until it prints PASS (S88's run: **31 of 33 old hashes still ALIVE = the S87 baseline; -no purge yet**). **Remaining (9):** E5 (permuter PR branch) · E6 (drafter write-up) — on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset +no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset — Drew), E2 (Archipelago — Drew), D3's outward actions (decomp.dev registration, frogress slug/key — Drew), **the wiki push (Drew: Wiki → "Create the first page" in the GitHub UI, then `tools/wiki_sync.sh --push`)**; then C11 (aftercare), G1 (`docs/gen3-handoff.md`), G2 (the PhaseEnd v2.0.0 + DIGEST + `v2.0.0` tag; Tier 1; WAIT for gate 2). ### 2. Facts the remaining tasks depend on (measured S88; verify if in doubt, R14) - **Repository state:** `main` = the rewritten history (4,031 commits) + the S87 tip commits (C7 → F2) + the S88 commits - (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, the E4 commit = HEAD); `origin/main` == the F2 commit + (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, the E5 commit = HEAD); `origin/main` == the F2 commit `5e57e88de2` — **Drew pushed the S87 tip on 2026-09-07 07:23Z; the S88 commits are NOT pushed** (a normal fast-forward push; R6). The first-ever GitHub runs of both workflows were GREEN on that push (`no-rom` 1 m 35 s, run 34095194524; `progress` 15 s, run 34095194479) — read the Actions tab again after the next push, fix red, never claim green unseen (P9). The repo is @@ -582,6 +614,10 @@ no purge yet**). **Remaining (9):** E5 (permuter PR branch) · E6 (drafter write old history). Expected after: one pack ≈ 80 MB, `.git` ≈ 93 MB, `git cat-file -e 296ff5551ba71269972e708031b52e9cf39bc782` fails. Until then the fixed probe's self-check prints "the WORKING repo's object store holds 31 of 33 sampled OLD commits" — that is the residue, not a new import (the fixed run left `git count-objects -v` byte-identical before/after). +- **Scratch that carries OUTWARD work for Drew (not regenerable from the repo without redoing it — keep until pushed):** + `.run/P33/xsig-repo/` (E4: one commit; push to `Druthulu/xsig`) and `.run/P33/permuter-upstream/` (E5: branch + `reloc-masked-scorer`, one commit; push to Drew's fork of decomp-permuter — regenerable from the tracked patch via + `git am`); `.run/P33/permuter-e2e/` is the demo dir (regenerable). - **Scratch to keep until the probe passes, then delete (C11):** `.run/public_rewrite/` (dict.json, mailmap, rom_blob_ids, old-to-new.tsv, unchanged_commits.txt, old_tag_tip.txt, the rewritten bare clone `repo.git`, the bundle `pre-rewrite.bundle` 556 MB, the trial logs; the probe's `probe_scratch.git` is created and deleted per run); `.run/objdiff/` (the objdiff-cli @@ -622,13 +658,11 @@ no purge yet**). **Remaining (9):** E5 (permuter PR branch) · E6 (drafter write Druthulu/BFM-decomp --limit 4`) · `git count-objects -v` (packs: 1 after Drew's gc; 30 = not yet run) · `df -h ~` · `.venv/bin/python tools/doc_links.py --strict` (PASS) · ask Drew: pushed? gc run? ticket filed? latest probe result? (`tools/public_rewrite/probe_github.sh` — ~1 min, gh-authenticated, safe to run from Claude since S88). -2. **E3 and E4 are DONE** (see the log; E4's outward action — Drew creates `Druthulu/xsig` EMPTY on GitHub, then +2. **E3, E4 and E5 are DONE** (see the log). Outward actions pending on Drew: E4 — create `Druthulu/xsig` EMPTY, then `git -C .run/P33/xsig-repo remote add origin https://github.com/Druthulu/xsig.git && git -C .run/P33/xsig-repo push -u origin - main` — is pending until Drew does it; the `.run/P33/xsig-repo/` copy is regenerable from `tools/xsig/`). **E5** (Max, 1): - the permuter upstream PR branch (`RelocMaskedScorer` behind `--score-mode reloc-masked`; upstream's `Scorer.__init__` gained - `ign_branch_targets, objdump_command`; the MIPS symbol wildcard `"." in field` at `scorer.py:66-67`; fixture tests, mypy, - black, `./run-tests.sh`; PR-2/issue: configurable `symbol_regex`; `docs/permuter-ils.md`); Drew opens the issue/PR. **E6** - (Max, 0.5–1): `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`, + main`; E5 — fork `simonlindholm/decomp-permuter`, `git -C .run/P33/permuter-upstream remote add fork && git -C + .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the description) and + file the issue from `docs/permuter-ils.md` §3. **E6** (Max, 0.5–1): `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`, `docs/gen2-mips-matching-model.md`, cookbook §12/§500, `docs/wave-playbook.md`; the ROM-derived pair dataset is NOT published. One commit per task after this file is updated; log; refresh this block. 3. **After the probe PASSES (Drew):** C10 the flip (Settings → Change visibility → Public, only with D/E/F landed) → E1 @@ -645,17 +679,19 @@ no purge yet**). **Remaining (9):** E5 (permuter PR branch) · E6 (drafter write the milestone evidence, WAIT for gate 2; then `PhaseEnd_Phase33.md` v2.0.0 with the rule candidates (a)–(h), `CURRENT_PHASE.md` → `phase-ends/logs/Phase33.md`, DIGEST §0/§2/§3 appended, the annotated `v2.0.0` tag; Drew pushes `main --tags`). -### 4. Files S88 touched (5 commits after the F2 tip) +### 4. Files S88 touched (6 commits after the F2 tip) Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`, `tools/xsig/` (xsig.py, README, LICENSE, -tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s); changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch + +tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s), +`tools/permuter/upstream/0001-reloc-masked-scorer.patch`; changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch + self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` + the xsig tests in tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites + xsig steps). Docs (new): `docs/wiki/*.md` (12), -`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`; changed: the five map files (135 +`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`; changed: the five map files (135 cites tagged in place), `docs/SETUP.md` (the probe clause in the public_rewrite row; 2 wiki rows; the P33 F3 section), `docs/public-flip-runbook.md` (§11: the R57 probe paragraph; the wiki push step), `docs/doc_links_pending.txt` (10 entries mid-task → EMPTY), `phase-ends/CURRENT_PHASE.md` (F3 ticked; the S88 preflight + F3 log entries; this block). Evidence: `.run/P33/f3_tools_health.log`, `.run/P33/e3_tools_health.log`, `.run/P33/e4_tools_health.log`, `.run/wiki/render/` -(regenerable), `.run/P33/xsig-repo/` (the standalone copy, one commit). The plan file: +(regenerable), `.run/P33/xsig-repo/` (the standalone copy, one commit), `.run/P33/permuter-upstream/` (the upstream clone + +branch + dev venv), `.run/P33/permuter-e2e/` (the demo dir). The plan file: `~/.claude/plans/max-effort-set-plan-twinkling-moonbeam.md` (copied below). --- diff --git a/tools/doc_links.py b/tools/doc_links.py index 0edb0872f..9e29bd679 100644 --- a/tools/doc_links.py +++ b/tools/doc_links.py @@ -20,7 +20,8 @@ REPO = pathlib.Path(__file__).resolve().parent.parent DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/README.md", "docs/SETUP.md", "docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md", "docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md", - "docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md"] + "docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md", + "docs/permuter-ils.md"] # whole directories in the default set (P33 F3): the wiki pages and the how-to chapters — every file, so a new page is # checked the moment it exists (the glob is expanded at run time; the count is printed with the rest, R41) DEFAULT_GLOBS = ["docs/wiki/*.md", "docs/how-to-ai-decomp/*.md"] diff --git a/tools/permuter/upstream/0001-reloc-masked-scorer.patch b/tools/permuter/upstream/0001-reloc-masked-scorer.patch new file mode 100644 index 000000000..8f24b31a3 --- /dev/null +++ b/tools/permuter/upstream/0001-reloc-masked-scorer.patch @@ -0,0 +1,630 @@ +From 09079705e56cc6ea535bd7456fc5c1216bc63288 Mon Sep 17 00:00:00 2001 +From: Drew T <50529377+Druthulu@users.noreply.github.com> +Date: Mon, 7 Sep 2026 11:24:21 -0600 +Subject: [PATCH] Add a relocation-masked scoring mode for MIPS (--score-mode + reloc-masked) +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +The default scorer diffs symbolised objdump rows and, on MIPS, recognises a +relocation slot only when the field contains a "." (scorer.py's +field_matches_any_symbol). For projects whose symbols carry no "." — e.g. +splat's func_80012345 / D_80054C10 — every relocation slot that differs by +symbol name is charged as a register difference the search can never reduce, +so the score floats on a floor above zero and the random walk has no gradient +toward a real match (measured on a 225-instruction function with a byte-exact +answer: base ~225, best ~210, never 0). + +RelocMaskedScorer (src/reloc_scorer.py) compares the two objects' instruction +words directly, masking exactly the fields the linker fills in — taken from +the TARGET's relocation records — and requiring the relocation operands to +agree there: + * R_MIPS_26: keep the opcode, mask the 26-bit target, same symbol+addend; + * HI16/LO16/PC16/LITERAL/GPREL16/GOT16/CALL16: keep opcode+rs+rt, mask the + 16-bit immediate, same symbol+addend; + * a `j` relocated against .text (a label inside the function): compared + relative to the function start; + * everything else, registers included: the full word. +score = mismatching positions + |length difference|; 0 iff the candidate links +to the same bytes. It subclasses Scorer so Permuter/Candidate need no change, +and is selected with --score-mode reloc-masked or score_mode = "reloc-masked" +in settings.toml (default "mnemonic" = unchanged behaviour). --debug prints +the per-instruction differences. Refused with -J: remote evaluators use the +default scorer. MIPS only; other architectures raise. + +Tests (test/test_reloc_scorer.py) run on an embedded objdump -drz listing of a +small gcc 2.7.2 object and need no cross toolchain: link-time fields masked, +symbol/register/opcode/branch/length differences each counted, internal jump +targets compared, and the scorer end to end through objdump_command. + +Written for the Brave Fencer Musashi decompilation, where this scorer (as an +out-of-tree drop-in) took the permuter from a non-zero floor to score-0 +matches on the register-allocation tail. +--- + README.md | 3 + + USAGE.md | 18 ++++ + example_settings.toml | 4 + + src/main.py | 51 ++++++++-- + src/reloc_scorer.py | 209 ++++++++++++++++++++++++++++++++++++++ + test/test_reloc_scorer.py | 209 ++++++++++++++++++++++++++++++++++++++ + 6 files changed, 487 insertions(+), 7 deletions(-) + create mode 100644 src/reloc_scorer.py + create mode 100644 test/test_reloc_scorer.py + +diff --git a/README.md b/README.md +index 6ffed6a..c7160b6 100644 +--- a/README.md ++++ b/README.md +@@ -101,6 +101,9 @@ permuter is currently quite bad at resolving stack differences). For more detail + see scorer.py. It's far from a perfect system, and should probably be tweaked to + look at e.g. the register diff graph. + ++For MIPS targets `--score-mode reloc-masked` scores relocation-masked instruction ++words instead (0 exactly when the candidate links to the same bytes) — see USAGE.md. ++ + **What sort of non-matchings are the permuter good at?** It's generally best towards + the end, when mostly regalloc changes remain. If there are reorderings or functional + changes, it's often easy to resolve those by hand, and neither the scorer nor the +diff --git a/USAGE.md b/USAGE.md +index 4f3e89d..5f28d69 100644 +--- a/USAGE.md ++++ b/USAGE.md +@@ -13,3 +13,21 @@ does all this for you. See README.md for more details. + - `/compile.sh /base.c -o /base.o` + - `./permuter.py --debug` + * `./permuter.py ` ++ ++## Scoring modes ++ ++By default candidates are scored by a penalty-weighted diff of the objdump'd rows ++(`src/scorer.py`). `--score-mode reloc-masked` (or `score_mode = "reloc-masked"` ++in `settings.toml`) switches MIPS targets to a relocation-masked word comparison ++(`src/reloc_scorer.py`): the fields the linker fills in (`j`/`jal` targets, ++`%hi`/`%lo` and other 16-bit relocations) are masked using the target object's ++relocation records, the relocation symbols must still agree, a `j` to a label ++inside the function is compared relative to the function start, and everything ++else — registers included — is compared as a full word. The score is the number ++of instructions that differ plus the length difference, so it is 0 exactly when ++the candidate links to the same bytes. Use it when the default score floats above ++0 for a function that has a byte-exact answer — typically a project whose symbols ++carry no `.` (splat's `func_80012345`), where the default scorer cannot recognise a ++relocation slot and charges every differing symbol as a register difference. ++`--debug` prints the per-instruction differences. Not available with the network ++(`-J`), whose evaluators use the default scorer. +diff --git a/example_settings.toml b/example_settings.toml +index dd8098a..152873f 100644 +--- a/example_settings.toml ++++ b/example_settings.toml +@@ -3,3 +3,7 @@ compiler_type = "ido" # examples: base, ido, mwcc, gcc + + [weight_overrides] + perm_temp_for_expr = 100 ++ ++# How candidates are scored: "mnemonic" (default) or "reloc-masked" (MIPS only; see ++# USAGE.md "Scoring modes"). The --score-mode flag overrides this. ++# score_mode = "reloc-masked" +diff --git a/src/main.py b/src/main.py +index 32965fc..457aedd 100644 +--- a/src/main.py ++++ b/src/main.py +@@ -51,6 +51,7 @@ from .preprocess import preprocess + from .printer import Printer + from .profiler import Profiler + from .randomizer import RANDOMIZATION_PASSES ++from .reloc_scorer import RelocMaskedScorer + from .scorer import Scorer + + MIN_PRIO = 0.01 +@@ -85,6 +86,7 @@ class Options: + debug_mode: bool = False + speed: int = 100 + ign_branch_targets: bool = False ++ score_mode: Optional[str] = None + + + def restricted_float(lo: float, hi: float) -> Callable[[str], float]: +@@ -360,14 +362,36 @@ def run_inner(options: Options, heartbeat: Callable[[], None]) -> List[int]: + + objdump_command = json_prop(settings, "objdump_command", str, "") or None + +- scorer = Scorer( +- target_o, +- stack_differences=options.stack_differences, +- algorithm=options.algorithm, +- debug_mode=options.debug_mode, +- ign_branch_targets=options.ign_branch_targets, +- objdump_command=objdump_command, ++ score_mode = options.score_mode or json_prop( ++ settings, "score_mode", str, "mnemonic" + ) ++ scorer: Scorer ++ if score_mode == "reloc-masked": ++ if options.use_network: ++ print( ++ "--score-mode reloc-masked cannot be combined with the network " ++ "(-J): remote evaluators score with the default scorer.", ++ file=sys.stderr, ++ ) ++ sys.exit(1) ++ scorer = RelocMaskedScorer( ++ target_o, ++ fn_name=fn_name, ++ debug_mode=options.debug_mode, ++ objdump_command=objdump_command, ++ ) ++ elif score_mode == "mnemonic": ++ scorer = Scorer( ++ target_o, ++ stack_differences=options.stack_differences, ++ algorithm=options.algorithm, ++ debug_mode=options.debug_mode, ++ ign_branch_targets=options.ign_branch_targets, ++ objdump_command=objdump_command, ++ ) ++ else: ++ print(f"Unknown score_mode {score_mode!r}", file=sys.stderr) ++ sys.exit(1) + c_source = preprocess(base_c) + + try: +@@ -701,6 +725,18 @@ def main() -> None: + choices=["difflib", "levenshtein"], + help="Diff algorithm to use", + ) ++ parser.add_argument( ++ "--score-mode", ++ dest="score_mode", ++ choices=["mnemonic", "reloc-masked"], ++ help="""How candidates are scored (default: the `score_mode` setting, else ++ "mnemonic"). "mnemonic" is the penalty-weighted diff of objdump'd rows. ++ "reloc-masked" (MIPS only) counts the instruction words that still differ ++ once the linker-filled fields are masked, and reaches 0 exactly when the ++ candidate links to the same bytes; use it when the default score floats ++ above 0 for a function that has a byte-exact answer (e.g. symbols without ++ a "." in their name).""", ++ ) + parser.add_argument( + "--keep-prob", + dest="keep_prob", +@@ -808,6 +844,7 @@ def main() -> None: + debug_mode=args.debug_mode, + speed=args.speed, + ign_branch_targets=args.ign_branch_targets, ++ score_mode=args.score_mode, + ) + + run(options) +diff --git a/src/reloc_scorer.py b/src/reloc_scorer.py +new file mode 100644 +index 0000000..7ccc14b +--- /dev/null ++++ b/src/reloc_scorer.py +@@ -0,0 +1,209 @@ ++"""Relocation-masked scoring for MIPS objects (`--score-mode reloc-masked`). ++ ++The default scorer diffs objdump'd mnemonic rows with relocations symbolised and ++counts register, stack, branch, reordering and insertion/deletion penalties. For a ++project whose symbols contain no "." (e.g. splat's `func_80012345` / `D_80054C10`), ++`field_matches_any_symbol` never fires, so every relocation slot that differs by ++symbol name is a REGALLOC penalty the search can never reduce: the score floats on ++a floor above zero and the random walk has no gradient toward a real match. That ++floor was measured on a 225-instruction function that had a byte-exact answer. ++ ++This scorer compares the two objects' instruction WORDS directly, masking exactly ++the fields the linker fills in and requiring the relocation operands to agree ++there. The mask is taken from the TARGET's relocation records: ++ ++* `R_MIPS_26` (a `j`/`jal` to a symbol): keep the opcode, mask the 26-bit target, ++ and require the same symbol+addend on both sides; ++* a 16-bit-immediate relocation (`R_MIPS_HI16`, `LO16`, `PC16`, `LITERAL`, ++ `GPREL16`, `GOT16`, `CALL16`): keep opcode + rs + rt, mask the low 16 bits, and ++ require the same symbol+addend; ++* a `j` whose relocation is against the section (a label inside the function): ++ the target is compared RELATIVE to the function start, so two candidates that ++ jump to different labels differ even though both words carry a section reloc; ++* everything else: the full 32-bit word — registers, constants, branch offsets. ++ ++ score = mismatching positions + |length difference| ++ ++It is 0 exactly when the two objects link to identical bytes for that function, ++and every unit is one instruction, so the number is a closeness, not a weighted ++guess. MIPS only; other architectures keep the default scorer. ++""" ++ ++import hashlib ++import re ++import shlex ++import subprocess ++from dataclasses import dataclass ++from typing import List, Optional, Sequence, Tuple ++ ++from .objdump import find_executable, get_arch ++from .scorer import Scorer ++ ++_HDR_RE = re.compile(r"^[0-9a-f]+ <([^>]+)>:") ++_INSN_RE = re.compile(r"^\s*([0-9a-f]+):\s+([0-9a-f]{8})\s+(.*)") ++_RELOC_RE = re.compile(r"R_MIPS_(\w+)\s+(\S+)") ++_IMM16_RELOCS = {"HI16", "LO16", "PC16", "LITERAL", "GPREL16", "GOT16", "CALL16"} ++ ++MASK_FULL = 0xFFFFFFFF ++MASK_OPCODE = 0xFC000000 # a relocated j/jal: the 26-bit target is the linker's ++MASK_NO_IMM16 = 0xFFFF0000 # a relocated 16-bit immediate: opcode + rs + rt stay ++ ++ ++@dataclass ++class Insn: ++ offset: int ++ word: int ++ text: str ++ reloc_kind: Optional[str] = None ++ reloc_op: Optional[str] = None ++ jrel: Optional[int] = ( ++ None # an internal `j`: its target relative to the function start ++ ) ++ ++ ++def mask_for(reloc_kind: Optional[str]) -> int: ++ if reloc_kind == "26": ++ return MASK_OPCODE ++ if reloc_kind in _IMM16_RELOCS: ++ return MASK_NO_IMM16 ++ return MASK_FULL ++ ++ ++def parse_listing(lines: Sequence[str], fn_name: Optional[str] = None) -> List[Insn]: ++ """Parse `objdump -drz` output into instructions with their relocation records. ++ ++ With `fn_name`, only that function's instructions are returned when the listing ++ has such a label; otherwise (or when the label is absent) the whole `.text` is ++ used, in order — the same scope the default scorer diffs. ++ """ ++ insns: List[Insn] = [] ++ fn_start: Optional[int] = None ++ in_fn = fn_name is None ++ for line in lines: ++ header = _HDR_RE.match(line) ++ if header: ++ in_fn = fn_name is None or header.group(1) == fn_name ++ continue ++ if not in_fn: ++ continue ++ m = _INSN_RE.match(line) ++ if m: ++ offset = int(m.group(1), 16) ++ if fn_start is None: ++ fn_start = offset ++ insns.append(Insn(offset, int(m.group(2), 16), m.group(3).strip())) ++ continue ++ reloc = _RELOC_RE.search(line) ++ if reloc and insns: ++ kind, op = reloc.group(1), reloc.group(2) ++ insns[-1].reloc_kind = kind ++ insns[-1].reloc_op = op ++ if fn_name is not None and not insns: ++ return parse_listing(lines, None) ++ start = fn_start or 0 ++ for insn in insns: ++ if ( ++ (insn.word >> 26) == 2 ++ and insn.reloc_kind == "26" ++ and insn.reloc_op == ".text" ++ ): ++ insn.jrel = ((insn.word & 0x3FFFFFF) << 2) - start ++ return insns ++ ++ ++def classify(cand: Insn, target: Insn) -> Optional[str]: ++ """None when the two instructions agree up to relocation, else why they differ.""" ++ mask = mask_for(target.reloc_kind) ++ if (cand.word & mask) != (target.word & mask): ++ if (cand.word >> 26) != (target.word >> 26): ++ return "opcode" ++ return "operand" ++ if cand.jrel is not None and target.jrel is not None and cand.jrel != target.jrel: ++ return "jump target" ++ if mask != MASK_FULL and (cand.reloc_op or "") != (target.reloc_op or ""): ++ return "relocation symbol" ++ return None ++ ++ ++def masked_diff( ++ cand: Sequence[Insn], target: Sequence[Insn] ++) -> Tuple[int, List[Tuple[int, Optional[Insn], Optional[Insn], str]]]: ++ """(score, [(index, cand_insn, target_insn, reason)]) — one unit per position.""" ++ diffs: List[Tuple[int, Optional[Insn], Optional[Insn], str]] = [] ++ for i in range(max(len(cand), len(target))): ++ if i >= len(cand) or i >= len(target): ++ diffs.append( ++ ( ++ i, ++ cand[i] if i < len(cand) else None, ++ target[i] if i < len(target) else None, ++ "length", ++ ) ++ ) ++ continue ++ reason = classify(cand[i], target[i]) ++ if reason is not None: ++ diffs.append((i, cand[i], target[i], reason)) ++ return len(diffs), diffs ++ ++ ++class RelocMaskedScorer(Scorer): ++ """A drop-in for `Scorer` that scores relocation-masked instruction words. ++ ++ `Scorer.__init__` is deliberately not called: this scorer neither symbolises nor ++ diffs mnemonic rows, so none of that state exists. The public contract is the ++ same — `score(cand_o) -> (int, hash)`, `PENALTY_INF`, `target_o`, `arch`, ++ `debug_mode`, `objdump_command`. ++ """ ++ ++ def __init__( ++ self, ++ target_o: str, ++ *, ++ fn_name: Optional[str] = None, ++ debug_mode: bool = False, ++ objdump_command: Optional[str] = None, ++ ): ++ self.target_o = target_o ++ self.arch = get_arch(target_o) ++ if self.arch.name != "mips": ++ raise ValueError( ++ f"reloc-masked scoring supports MIPS objects only, not {self.arch.name}" ++ ) ++ self.fn_name = fn_name ++ self.debug_mode = debug_mode ++ self.objdump_command = objdump_command or "" ++ self.stack_differences = False # read by the network client; unused here ++ self.algorithm = "reloc-masked" ++ self.target_insns = self._insns(target_o) ++ ++ def _insns(self, o_file: str) -> List[Insn]: ++ if self.objdump_command: ++ command = shlex.split(self.objdump_command) ++ else: ++ command = [find_executable(tuple(self.arch.executable), self.arch.name)] ++ command += ["-drz", "-j", ".text"] ++ output = subprocess.check_output(command + [o_file]).decode("utf-8", "replace") ++ return parse_listing(output.splitlines(), self.fn_name) ++ ++ def score(self, cand_o: Optional[str]) -> Tuple[int, str]: ++ if not cand_o: ++ return Scorer.PENALTY_INF, "" ++ cand = self._insns(cand_o) ++ if not cand: ++ return Scorer.PENALTY_INF, "" ++ value, diffs = masked_diff(cand, self.target_insns) ++ if self.debug_mode: ++ for index, c, t, reason in diffs: ++ left = f"{c.word:08x} {c.text}" if c else "--" ++ right = f"{t.word:08x} {t.text}" if t else "--" ++ print(f"#{index:<4} {left:40.40s} | {right:40.40s} {reason}") ++ print( ++ f"reloc-masked score: {value} ({len(cand)} vs {len(self.target_insns)} insns)" ++ ) ++ digest = hashlib.sha256( ++ "".join( ++ f"{i.word:08x}:{i.reloc_kind or ''}:{i.reloc_op or ''};" for i in cand ++ ).encode() ++ ).hexdigest() ++ return value, digest +diff --git a/test/test_reloc_scorer.py b/test/test_reloc_scorer.py +new file mode 100644 +index 0000000..8bb4e4c +--- /dev/null ++++ b/test/test_reloc_scorer.py +@@ -0,0 +1,209 @@ ++import os ++import tempfile ++import unittest ++from typing import Dict ++ ++from src.reloc_scorer import ( ++ MASK_FULL, ++ MASK_NO_IMM16, ++ MASK_OPCODE, ++ RelocMaskedScorer, ++ mask_for, ++ masked_diff, ++ parse_listing, ++) ++from src.scorer import Scorer ++ ++# `objdump -drz -j .text` of a small function compiled by gcc 2.7.2 (MIPS I, -O2 -G0): ++# a %hi/%lo pair (HI16/LO16 against `table`), a jal (R_MIPS_26 against `helper`), ++# two branches (PC-relative, encoded in the word), and a second function. ++LISTING = """ ++fixture.o: file format elf32-tradlittlemips ++ ++ ++Disassembly of section .text: ++ ++00000000 : ++ 0:\t27bdffd8 \taddiu\tsp,sp,-40 ++ 4:\tafb1001c \tsw\ts1,28(sp) ++ 8:\t00808821 \tmove\ts1,a0 ++ c:\tafb00018 \tsw\ts0,24(sp) ++ 10:\t00008021 \tmove\ts0,zero ++ 14:\t00002021 \tmove\ta0,zero ++ 18:\tafbf0024 \tsw\tra,36(sp) ++ 1c:\t1a200013 \tblez\ts1,6c ++ 20:\tafb20020 \tsw\ts2,32(sp) ++ 24:\t3c120000 \tlui\ts2,0x0 ++\t\t\t24: R_MIPS_HI16\ttable ++ 28:\t26520000 \taddiu\ts2,s2,0 ++\t\t\t28: R_MIPS_LO16\ttable ++ 2c:\t32020007 \tandi\tv0,s0,0x7 ++ 30:\t00021080 \tsll\tv0,v0,0x2 ++ 34:\t00521021 \taddu\tv0,v0,s2 ++ 38:\t8c420000 \tlw\tv0,0(v0) ++ 3c:\t00000000 \tnop ++ 40:\t00822021 \taddu\ta0,a0,v0 ++ 44:\t28820065 \tslti\tv0,a0,101 ++ 48:\t14400004 \tbnez\tv0,5c ++ 4c:\t00000000 \tnop ++ 50:\t0c000000 \tjal\t0 ++\t\t\t50: R_MIPS_26\thelper ++ 54:\t00000000 \tnop ++ 58:\t00402021 \tmove\ta0,v0 ++ 5c:\t26100001 \taddiu\ts0,s0,1 ++ 60:\t0211102a \tslt\tv0,s0,s1 ++ 64:\t1440fff2 \tbnez\tv0,30 ++ 68:\t32020007 \tandi\tv0,s0,0x7 ++ 6c:\t00801021 \tmove\tv0,a0 ++ 70:\t8fbf0024 \tlw\tra,36(sp) ++ 74:\t8fb20020 \tlw\ts2,32(sp) ++ 78:\t8fb1001c \tlw\ts1,28(sp) ++ 7c:\t8fb00018 \tlw\ts0,24(sp) ++ 80:\t27bd0028 \taddiu\tsp,sp,40 ++ 84:\t03e00008 \tjr\tra ++ 88:\t00000000 \tnop ++ ++0000008c : ++ 8c:\t00041040 \tsll\tv0,a0,0x1 ++ 90:\t00441021 \taddu\tv0,v0,a0 ++ 94:\t00052883 \tsra\ta1,a1,0x2 ++ 98:\t03e00008 \tjr\tra ++ 9c:\t00451026 \txor\tv0,v0,a1 ++""" ++ ++ ++def listing(edits: Dict[str, str]) -> str: ++ """LISTING with exact textual replacements applied (each key must occur).""" ++ text = LISTING ++ for old, new in edits.items(): ++ assert old in text, old ++ text = text.replace(old, new) ++ return text ++ ++ ++class TestParse(unittest.TestCase): ++ def test_whole_text_and_one_function(self) -> None: ++ whole = parse_listing(LISTING.splitlines()) ++ self.assertEqual(len(whole), 40) ++ only = parse_listing(LISTING.splitlines(), "other") ++ self.assertEqual( ++ [i.word for i in only], ++ [0x00041040, 0x00441021, 0x00052883, 0x03E00008, 0x00451026], ++ ) ++ # an absent label falls back to the whole section, like the default scorer's scope ++ self.assertEqual(len(parse_listing(LISTING.splitlines(), "no_such_fn")), 40) ++ ++ def test_relocations_are_attached(self) -> None: ++ insns = parse_listing(LISTING.splitlines(), "fixture") ++ by_off = {i.offset: i for i in insns} ++ self.assertEqual( ++ (by_off[0x24].reloc_kind, by_off[0x24].reloc_op), ("HI16", "table") ++ ) ++ self.assertEqual( ++ (by_off[0x28].reloc_kind, by_off[0x28].reloc_op), ("LO16", "table") ++ ) ++ self.assertEqual( ++ (by_off[0x50].reloc_kind, by_off[0x50].reloc_op), ("26", "helper") ++ ) ++ self.assertIsNone(by_off[0x48].reloc_kind) # a branch is encoded, not relocated ++ ++ def test_masks(self) -> None: ++ self.assertEqual(mask_for("26"), MASK_OPCODE) ++ self.assertEqual(mask_for("HI16"), MASK_NO_IMM16) ++ self.assertEqual(mask_for("LO16"), MASK_NO_IMM16) ++ self.assertEqual(mask_for(None), MASK_FULL) ++ ++ ++class TestMaskedDiff(unittest.TestCase): ++ def insns(self, text: str = LISTING) -> list: ++ return parse_listing(text.splitlines(), "fixture") ++ ++ def test_identical_is_zero(self) -> None: ++ self.assertEqual(masked_diff(self.insns(), self.insns())[0], 0) ++ ++ def test_link_time_fields_are_masked(self) -> None: ++ # the same code with the linker's fields filled differently (a jal target, a %hi/%lo pair) ++ cand = listing( ++ { ++ "0c000000 ": "0c004800 ", ++ "3c120000 ": "3c128002 ", ++ "26520000 ": "26522a40 ", ++ } ++ ) ++ self.assertEqual(masked_diff(self.insns(cand), self.insns())[0], 0) ++ ++ def test_other_symbol_at_a_relocated_slot_counts(self) -> None: ++ cand = listing({"R_MIPS_26\thelper": "R_MIPS_26\tother_helper"}) ++ score, diffs = masked_diff(self.insns(cand), self.insns()) ++ self.assertEqual((score, diffs[0][3]), (1, "relocation symbol")) ++ cand = listing( ++ { ++ "R_MIPS_HI16\ttable": "R_MIPS_HI16\ttable2", ++ "R_MIPS_LO16\ttable": "R_MIPS_LO16\ttable2", ++ } ++ ) ++ self.assertEqual(masked_diff(self.insns(cand), self.insns())[0], 2) ++ ++ def test_register_branch_and_opcode_differences_count(self) -> None: ++ cand = listing({"00521021 ": "00531021 "}) # addu v0,v0,s3 instead of s2 ++ score, diffs = masked_diff(self.insns(cand), self.insns()) ++ self.assertEqual((score, diffs[0][3]), (1, "operand")) ++ cand = listing({"14400004 ": "14400005 "}) # bnez to a different label ++ self.assertEqual(masked_diff(self.insns(cand), self.insns())[0], 1) ++ cand = listing({"8c420000 ": "94420000 "}) # lw -> lhu: a different opcode ++ score, diffs = masked_diff(self.insns(cand), self.insns()) ++ self.assertEqual((score, diffs[0][3]), (1, "opcode")) ++ ++ def test_length_difference_counts_per_instruction(self) -> None: ++ cand = LISTING.replace(" 88:\t00000000 \tnop\n", "") ++ score, diffs = masked_diff(self.insns(cand), self.insns()) ++ self.assertEqual((score, diffs[-1][3]), (1, "length")) ++ ++ def test_internal_jump_targets_are_compared(self) -> None: ++ # a `j` to a label inside the function carries `R_MIPS_26 .text`: its target is compared ++ # relative to the function start, so a jump to a different label is a mismatch ++ target = LISTING.replace( ++ " 4c:\t00000000 \tnop\n", ++ " 4c:\t08000018 \tj\t60 \n\t\t\t4c: R_MIPS_26\t.text\n", ++ ) ++ cand = LISTING.replace( ++ " 4c:\t00000000 \tnop\n", ++ " 4c:\t0800001b \tj\t6c \n\t\t\t4c: R_MIPS_26\t.text\n", ++ ) ++ t, c = self.insns(target), self.insns(cand) ++ self.assertEqual(masked_diff(t, t)[0], 0) ++ score, diffs = masked_diff(c, t) ++ self.assertEqual((score, diffs[0][3]), (1, "jump target")) ++ ++ ++class TestScorer(unittest.TestCase): ++ def test_scorer_end_to_end_with_a_stand_in_objdump(self) -> None: ++ # `objdump_command` may be any command that prints the listing: `cat` turns a listing file ++ # into an "object", so the whole scorer path runs without a cross toolchain ++ # get_arch() reads the ELF ident + e_machine (20 bytes): a little-endian MIPS header in ++ # front of the listing is all the scorer needs from the file itself ++ header = b"\x7fELF\x01\x01\x01" + bytes(9) + b"\x01\x00\x08\x00" ++ ++ def fake_object(path: str, text: str) -> str: ++ with open(path, "wb") as f: ++ f.write(header + b"\n" + text.encode()) ++ return path ++ ++ with tempfile.TemporaryDirectory() as d: ++ target = fake_object(os.path.join(d, "target.o"), LISTING) ++ same = fake_object( ++ os.path.join(d, "same.o"), listing({"0c000000 ": "0c004800 "}) ++ ) ++ worse = fake_object( ++ os.path.join(d, "worse.o"), listing({"00521021 ": "00531021 "}) ++ ) ++ scorer = RelocMaskedScorer(target, fn_name="fixture", objdump_command="cat") ++ self.assertIsInstance(scorer, Scorer) ++ self.assertEqual(scorer.score(same)[0], 0) ++ self.assertEqual(scorer.score(worse)[0], 1) ++ self.assertEqual(scorer.score(None)[0], Scorer.PENALTY_INF) ++ self.assertNotEqual(scorer.score(same)[1], scorer.score(worse)[1]) ++ ++ ++if __name__ == "__main__": ++ unittest.main() +-- +2.43.0 +