tools(phase-33): E4 — xsig packaged: tools/xsig/ (xsig.py library + CLI sign-s | sign-objdump | cross | verify | selftest, from the Phase-21 .run/xdedup probe; README with the recorded worked example — BFM × Xenogears + Vagrant Story 103 hits all PsyQ library/BIOS, BFM × Tomba 126 hits 124 library: the decision log's clean negative; MIT LICENSE; tests/ from a game-free fixture compiled with the pinned triple and linked at two addresses with --emit-relocs, relocation records merged into the listings, 8/8) in tools-health + CI + doc_links; SETUP row + P33 E4 section; the standalone copy at .run/P33/xsig-repo/ (one commit, noreply identity, system-python tests OK) for Drew to push as Druthulu/xsig; log + checkpoint (NEXT = E5, a cut candidate for Drew)

This commit is contained in:
Drew T
2026-09-07 02:51:02 -06:00
parent 50c1b69e4d
commit 9c4d32d651
14 changed files with 867 additions and 19 deletions
+2
View File
@@ -48,6 +48,8 @@ jobs:
run: python tools/cookbook_index.py --check
- name: gccmap_cites — every codegen-map citation carries its source-tree tag
run: python tools/gccmap_cites.py --check
- name: xsig — the cross-project signature tool's tests (fixtures carry no game bytes)
run: python tools/xsig/tests/test_xsig.py
- name: ghidra_roster — derived roster is fresh
run: python tools/ghidra_roster.py --check
- name: work_evidence self-test
+1
View File
@@ -298,6 +298,7 @@ tools-health:
# P33 D5: every relative link in the public-facing docs resolves (pending pages are listed, and must be gone by gate 2)
$(VENV_PY) tools/doc_links.py
$(VENV_PY) tools/wiki_render.py --selftest
$(VENV_PY) tools/xsig/tests/test_xsig.py 2>&1 | tail -1 | grep -q '^OK' && echo 'xsig tests: OK (8)' || { echo 'xsig tests: FAIL'; exit 1; }
# Behavioural guards (P31 S70): tools-health audits DATA integrity; these assert that a tool
# ACTUALLY DID the work it reports. A guard that is not running is not a guard (R54).
$(VENV_PY) tools/work_evidence.py --selftest
+23
View File
@@ -765,6 +765,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
| | `tools/wiki_render.py OUT_DIR \| --list \| --selftest` | **(P33 F3)** Render `docs/wiki/*.md` + `docs/how-to-ai-decomp/*.md` into GitHub-wiki page names with every relative link rewritten deterministically (wiki page → its name; a chapter → `How-to-AI-decomp-NN-name`; any other repo path → a `blob/main` / `tree/main` / raw URL; URLs, mailto and anchors untouched; **a dead link is an error**, R43). `--selftest` = the 12-case fixture incl. the dead-link negative control, in `make tools-health`. |
| | `tools/wiki_sync.sh [--push] [--wiki-url URL]` | **(P33 F3)** Render into `.run/wiki/render/` → clone or fast-forward `BFM-decomp.wiki.git` under `.run/wiki/` → REPLACE its pages with the rendered set (the repo is the source of truth; a page edited on GitHub is overwritten) → `git status --short`; **`--push` is Drew's (R6)**: commit + push. Before the wiki repo exists (after the flip AND the first page created in the GitHub UI) the dry run renders and lists (exit 0) and `--push` refuses (exit 2). |
| | `tools/gccmap_cites.py [--dry-run \| --check \| --verify \| --controls \| --explain] [--retag]` | **(P33 E3)** Tag every `file.c:NNN` cite in `docs/gcc-2.7.2-map/*.md` with the source tree its line number belongs to — `[2.7.2]` (the vanilla subset), `[2.8.1 pm]` (gcc-papermario), `[repo]` — derived from the trees (quoted snippets, identifiers with function extents and nearest distance, the author's cues; a contradiction fails loudly; ties → `cite_overrides.tsv` → cue → 2.7.2); writes in place, idempotent, never silently changes an existing tag. `--check` is textual (every cite tagged, no stale override) and runs in `make tools-health` + CI; the others need both reference trees and refuse without them. |
| | `tools/xsig/xsig.py sign-s \| sign-objdump \| cross \| verify \| selftest` | **(P33 E4; Phase-21 origin)** Relocation-masked per-function signatures for CROSS-PROJECT code identification: mask `j`/`jal` targets and HI16/LO16 immediates (from `%hi`/`%lo` operands or `objdump -dr` records), keep opcodes/registers/constants/branches; `cross` joins two JSONL sets on `sig` with a coverage line; `verify` prints the instruction-by-instruction diff (opcode/register/immediate/length). Self-contained (stdlib, MIT, own README + LICENSE + `tests/` from a game-free fixture compiled at two link addresses with `--emit-relocs`); `tests/test_xsig.py` (8) in `make tools-health` + CI. The standalone repo copy is prepared under `.run/P33/xsig-repo/` (Drew creates + pushes `Druthulu/xsig`). |
| | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). |
| | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data/<project>/<version>/` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. |
| | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) |
@@ -1144,6 +1145,28 @@ fills fast). Nothing is leaking — but the host does not get the memory back on
`tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh <addr> <name>
[PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`.
### P33 E4 (S88, 2026-09-07) — xsig packaged: `tools/xsig/` + the standalone repo, from the Phase-21 `.run/xdedup/xsig.py`
- **What it is.** The Phase-21 cross-project dedup probe's signature library (`.run/xdedup/xsig.py`, 2026-06-25) plus the
driver scripts around it (`sign_bfm.py`, `sign_xeno.py`, `cross.py`, `analyze.py`) folded into ONE stdlib file with a CLI
(`sign-s | sign-objdump | cross | verify | selftest`), a README that carries the recorded worked example (BFM × Xenogears +
Vagrant Story: 103 hits, all PsyQ library/BIOS — libapi 34, libcd 14, libetc 14, libspu 11, libcard 7 …, the largest
`_spu_setReverbAttr` 307 ins; BFM × Tomba: 126 hits, 124 library, one 19-ins non-library HIGH — the decision log's "clean
negative: zero engine code"), an MIT `LICENSE`, and `tests/`.
- **The fixture (no game bytes, R-candidate (a)):** `tests/fixture.c` (two functions of our own) compiled with the pinned
triple by `tests/make_fixtures.sh` and linked TWICE (`ld -q` = `--emit-relocs`, `-Ttext` 0x80010000 vs 0x80200000,
`--defsym` helper/table at different addresses). Gotcha: `objdump -dr` interleaves relocation records only for OBJECT
files; for a linked ELF they are listed separately (`-r`, section-relative offsets) — the generator merges them into the
listing in the object-listing format (`\t\t\t<vma>: R_MIPS_HI16\ttable`), which is what the front-end reads. A second gotcha:
a linked listing puts the address at column 0 (an object listing indents it) — the instruction regex is `^\s*` now.
`tests/test_xsig.py` (8 checks): same function at two link addresses signs identically while the raw words differ; the
relocated fields are present and the only masked ones (`verify`: IDENTICAL up to relocation, 3 fields); a different
function differs; a register flip changes the sig and is classified `register`; the `.s` and objdump front-ends agree;
`sign-objdump` + `cross` find exactly the two true pairs; `verify` accepts the pair and rejects the mismatch; `selftest`.
- **Wiring (R21):** the tests in `make tools-health` and the CI `audits` job; `tools/xsig/README.md` in `doc_links`' default
set; this row/section. The standalone repo: `.run/P33/xsig-repo/` = the same files, `git init`, the repo-local NOREPLY
identity (never the global one), one initial commit; Drew creates `Druthulu/xsig` on GitHub (empty) and pushes.
### P33 E3 (S88, 2026-09-07) — the codegen map's README + every citation tagged with its source tree
- **Why.** `docs/gcc-2.7.2-map/` was written (Phase 23) against `tools/reference/gcc-papermario`, which is gcc **2.8.1**, and
audited (P29 S23) against the vanilla 2.7.2 subset — the mechanisms held (119 confirmed · 40 line-drift · 7 refuted of 21
+50 -18
View File
@@ -80,7 +80,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo
- [x] **F2** retrospective — Max — see Log 2026-09-07 F2
- [x] **F3** wiki + how-to-ai-decomp + `wiki_sync.sh` — Max (FULL, Drew 2026-09-07) — see Log 2026-09-07 F3
- [x] **E3** gcc-2.7.2 map README + `gccmap_cites.py` — xHigh — see Log 2026-09-07 E3
- [ ] **E4** xsig packaging — xHigh
- [x] **E4** xsig packaging — xHigh — see Log 2026-09-07 E4
- [ ] **E5** permuter upstream PR branch — Max
- [ ] **E6** drafter write-up — Max
- [ ] **E1** decomp.me preset (after the flip) — xHigh
@@ -509,37 +509,66 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact
harness's low-memory guard during the report step (a transient spike; 29 GB available afterwards) — and the process table
held **8 orphaned `tools/permuter/run_masked.py` workers from a closed phase, 49 h old (parent PID 18)**, stopped by PID
(never `pkill -f` with a literal the calling shell carries); the foreground re-run passed. Commit: see below.
- **2026-09-07 (S88, Max) — E4 xsig packaged: `tools/xsig/` + the standalone repo.** The Phase-21 cross-project dedup
probe's library (`.run/xdedup/xsig.py`, 2026-06-25) and its four driver scripts folded into ONE stdlib file,
`tools/xsig/xsig.py` (library + CLI `sign-s | sign-objdump | cross | verify | selftest`; `compare()` = the
instruction-by-instruction diff a score cannot replace — opcode/register/immediate/length, R63), with `README.md` (the
recorded worked example: BFM × Xenogears + Vagrant Story = 103 hits, all PsyQ library/BIOS — libapi 34, libcd 14, libetc
14, libspu 11, libcard 7, largest `_spu_setReverbAttr` 307 ins, 37 already named; BFM × Tomba = 126 hits, 124 library, one
19-ins non-library HIGH; the decision log's "clean negative — zero engine code"), an MIT `LICENSE`, and `tests/`: a
game-free fixture — `tests/fixture.c` (two functions of our own) compiled with the pinned triple by
`tests/make_fixtures.sh` and linked TWICE (`ld -q` = `--emit-relocs`, `-Ttext` 0x80010000 vs 0x80200000, `--defsym`
helper/table at different addresses) → `fixture_a.txt` / `fixture_b.txt` (objdump listings) + `fixture_a.s` (the splat
form, `%hi`/`%lo` from the reloc records); `tests/test_xsig.py` 8 checks (same function at two link addresses signs
identically while the raw words differ; the 3 relocated fields are present and the only masked ones — `verify`: IDENTICAL
up to relocation; a different function differs; a register flip changes the sig and is classified `register`; the two
front-ends agree; `sign-objdump` + `cross` find exactly the two true pairs; `verify` rejects the mismatch; `selftest`) —
**8/8 OK**. Two gotchas, recorded: `objdump -dr` interleaves relocation records only for OBJECT files — a linked ELF lists
them separately (`-r`, section-relative offsets), so the generator merges them into the listing in the object-listing
format; and a linked listing puts the address at column 0 (an object listing indents it) — the instruction regex is
`^\s*`. Sibling-repo URLs in the README taken from the reference clones' `git remote get-url origin`, not from memory.
Wiring (R21): the tests in `make tools-health` (`xsig tests: OK (8)`) + the CI `audits` job; `tools/xsig/README.md` in
`doc_links`' default set (`--strict` PASS); SETUP row + the P33 E4 section. **The standalone repo:** `.run/P33/xsig-repo/`
= `xsig.py`, `README.md`, `LICENSE`, `tests/` (+ `.gitignore`), `git init`, the repo-local NOREPLY identity (the global
identity DIFFERS and was never used), the tests run there with the SYSTEM `python3` (self-contained, no venv), one
initial commit — **Drew creates `Druthulu/xsig` on GitHub (EMPTY) and pushes** (`git -C .run/P33/xsig-repo remote add
origin … && git push -u origin main`). `make tools-health` re-run with the new line: **`tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` (EXIT=0)**
(`.run/P33/e4_tools_health.log`). Commit: see below.
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓ (30 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E4 (2026-09-07, written by session 4555f4e4 "S88" at the E3 close; SUPERSEDES the earlier blocks)
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓, E4 ✓ (31 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E5 (2026-09-07, written by session 4555f4e4 "S88" at the E4 close; SUPERSEDES the earlier blocks)
### 0. How to use this block
You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file,
and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest
(R1–R73), then WAIT for Drew. **NEXT = E4** (xHigh, ≈0.5 session — the xsig packaging; the plan's remaining cut candidate is
E5 (the permuter PR, Max, 1 session) — Drew chose full scope for F3 and E3 ran in full, so ask only if he raises it). Rebuild
the harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3 completed and C10 in progress. **Every commit cites NEW
(R1–R73), then WAIT for Drew. **NEXT = E5** (Max, ≈1 session — the permuter upstream PR branch; it is the plan's last
named CUT CANDIDATE: ask Drew whether E5 runs in full, trimmed to the issue + `docs/permuter-ils.md`, or is cut — F3 and E3
ran in full). Rebuild the harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3, E4 completed and
C10 in progress. **Every commit cites NEW
hashes only** (the history was rewritten; `docs/commit-map.tsv` maps ordinals → new hashes; the scratch `.run/public_rewrite/`
holds the old ones and stays until the probe passes). **Never `git clean -x`** (CLAUDE.md fail-safe).
### 1. Where we are
**Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). The approved plan
is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (30):** A1–A5, B1–B9/C3,
is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (31):** A1–A5, B1–B9/C3,
C1–C9 (the rewrite, adopted, force-pushed by Drew, gc'd), D1–D5 (README, LICENSE/NOTICE/THIRD_PARTY, badges/objdiff/frogress,
SETUP public-clean, governing docs + `doc_links`), F1 (timeline + story), F2 (retrospective), **F3 (S88: the wiki — 12 files
under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `tools/wiki_render.py` + `tools/wiki_sync.sh`,
`doc_links` DEFAULT_GLOBS; `954362c81e` + `0cf971d1f4`), **E3 (S88: `docs/gcc-2.7.2-map/README.md` + `tools/gccmap_cites.py`
+ `cite_overrides.tsv`; all 135 map citations tagged `[2.7.2]`/`[2.8.1 pm]`/`[repo]` — 79/55/1 — verified, controls 6/6;
the commit that carries this block).** **In progress (Drew, C10):** the GitHub
`50c1b69e4d`), **E4 (S88: `tools/xsig/` — `xsig.py` library+CLI, README with the Phase-21 worked example, MIT LICENSE,
`tests/` from a game-free fixture at two link addresses, 8/8; in tools-health + CI; the standalone copy at `.run/P33/xsig-repo/`
with one commit under the noreply identity — Drew creates `Druthulu/xsig` EMPTY and pushes; the commit that carries this
block).** **In progress (Drew, C10):** the GitHub
Support ticket (text: `docs/public-flip-runbook.md` §11 — its filing was never confirmed to S88; ask) and the daily
`tools/public_rewrite/probe_github.sh` until it prints PASS (S88's run: **31 of 33 old hashes still ALIVE = the S87 baseline;
no purge yet**). **Remaining (10):** E4 (xsig) · E5 (permuter PR branch) · E6 (drafter write-up) — all on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset
no purge yet**). **Remaining (9):** E5 (permuter PR branch) · E6 (drafter write-up) — on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset
— Drew), E2 (Archipelago — Drew), D3's outward actions (decomp.dev registration, frogress slug/key — Drew), **the wiki push
(Drew: Wiki → "Create the first page" in the GitHub UI, then `tools/wiki_sync.sh --push`)**; then C11 (aftercare), G1
(`docs/gen3-handoff.md`), G2 (the PhaseEnd v2.0.0 + DIGEST + `v2.0.0` tag; Tier 1; WAIT for gate 2).
### 2. Facts the remaining tasks depend on (measured S88; verify if in doubt, R14)
- **Repository state:** `main` = the rewritten history (4,031 commits) + the S87 tip commits (C7 → F2) + the S88 commits
(`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, the E3 commit = HEAD); `origin/main` == the F2 commit
(`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, the E4 commit = HEAD); `origin/main` == the F2 commit
`5e57e88de2` — **Drew pushed the S87 tip on 2026-09-07 07:23Z; the S88 commits are NOT pushed** (a normal fast-forward push;
R6). The first-ever GitHub runs of both workflows were GREEN on that push (`no-rom` 1 m 35 s, run 34095194524; `progress`
15 s, run 34095194479) — read the Actions tab again after the next push, fix red, never claim green unseen (P9). The repo is
@@ -565,7 +594,8 @@ no purge yet**). **Remaining (10):** E4 (xsig) · E5 (permuter PR branch) · E6
any new forward link must be listed there with its creating task (gate 2 requires empty). `tools/wiki_render.py --selftest`
12/12; `tools/wiki_sync.sh` dry run: 25 pages, 264 links rewritten, "not clonable yet … dry run OK" (exit 0).
`tools/gccmap_cites.py --check` OK / `--verify` 0 disagreements / `--controls` 6/6 (needs both `tools/reference/` trees —
gitignored; `--check` alone is textual and is what CI and tools-health run); `make tools-health` at the E3 close:
gitignored; `--check` alone is textual and is what CI and tools-health run); `tools/xsig/tests/test_xsig.py` 8/8 (in
tools-health + CI); `make tools-health` at the E4 close: `tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` (EXIT=0) (`.run/P33/e4_tools_health.log`); at the E3 close:
`tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` (EXIT=0) (`.run/P33/e3_tools_health.log`).
- **Generated, never typed (R51):** `docs/progress.json`, the README block, `docs/badges/*.json`, `docs/story-timeline.md/.svg`,
`docs/commit-map.tsv`, `config/ghidra/ROSTER.md`, `docs/progress*.md`, `docs/cookbook-index.md` — regenerate with
@@ -592,9 +622,9 @@ no purge yet**). **Remaining (10):** E4 (xsig) · E5 (permuter PR branch) · E6
Druthulu/BFM-decomp --limit 4`) · `git count-objects -v` (packs: 1 after Drew's gc; 30 = not yet run) · `df -h ~` ·
`.venv/bin/python tools/doc_links.py --strict` (PASS) · ask Drew: pushed? gc run? ticket filed? latest probe result?
(`tools/public_rewrite/probe_github.sh` — ~1 min, gh-authenticated, safe to run from Claude since S88).
2. **E3 is DONE** (see the log). **E4** (xHigh, 0.5): `.run/xdedup/xsig.py` → `tools/xsig/` (library + CLI `sign-s | sign-objdump | cross | verify`, MIT
`LICENSE`, README with the Xenogears/Vagrant Story/Tomba hits, `tests/` from one 10-line C function compiled at two link
addresses — no game bytes); standalone repo prepared under `.run/P33/xsig-repo/` (Drew creates + pushes). **E5** (Max, 1):
2. **E3 and E4 are DONE** (see the log; E4's outward action — Drew creates `Druthulu/xsig` EMPTY on GitHub, then
`git -C .run/P33/xsig-repo remote add origin https://github.com/Druthulu/xsig.git && git -C .run/P33/xsig-repo push -u origin
main` — is pending until Drew does it; the `.run/P33/xsig-repo/` copy is regenerable from `tools/xsig/`). **E5** (Max, 1):
the permuter upstream PR branch (`RelocMaskedScorer` behind `--score-mode reloc-masked`; upstream's `Scorer.__init__` gained
`ign_branch_targets, objdump_command`; the MIPS symbol wildcard `"." in field` at `scorer.py:66-67`; fixture tests, mypy,
black, `./run-tests.sh`; PR-2/issue: configurable `symbol_regex`; `docs/permuter-ils.md`); Drew opens the issue/PR. **E6**
@@ -615,15 +645,17 @@ no purge yet**). **Remaining (10):** E4 (xsig) · E5 (permuter PR branch) · E6
the milestone evidence, WAIT for gate 2; then `PhaseEnd_Phase33.md` v2.0.0 with the rule candidates (a)–(h), `CURRENT_PHASE.md`
→ `phase-ends/logs/Phase33.md`, DIGEST §0/§2/§3 appended, the annotated `v2.0.0` tag; Drew pushes `main --tags`).
### 4. Files S88 touched (4 commits after the F2 tip)
Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`; changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch +
self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` in
tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites step). Docs (new): `docs/wiki/*.md` (12),
### 4. Files S88 touched (5 commits after the F2 tip)
Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`, `tools/xsig/` (xsig.py, README, LICENSE,
tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s); changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch +
self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` + the xsig tests in
tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites + xsig steps). Docs (new): `docs/wiki/*.md` (12),
`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`; changed: the five map files (135
cites tagged in place), `docs/SETUP.md` (the probe clause in the public_rewrite row;
2 wiki rows; the P33 F3 section), `docs/public-flip-runbook.md` (§11: the R57 probe paragraph; the wiki push step),
`docs/doc_links_pending.txt` (10 entries mid-task → EMPTY), `phase-ends/CURRENT_PHASE.md` (F3 ticked; the S88 preflight + F3 log
entries; this block). Evidence: `.run/P33/f3_tools_health.log`, `.run/P33/e3_tools_health.log`, `.run/wiki/render/` (regenerable). The plan file:
entries; this block). Evidence: `.run/P33/f3_tools_health.log`, `.run/P33/e3_tools_health.log`, `.run/P33/e4_tools_health.log`, `.run/wiki/render/`
(regenerable), `.run/P33/xsig-repo/` (the standalone copy, one commit). The plan file:
`~/.claude/plans/max-effort-set-plan-twinkling-moonbeam.md` (copied below).
---
+1 -1
View File
@@ -20,7 +20,7 @@ REPO = pathlib.Path(__file__).resolve().parent.parent
DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/README.md", "docs/SETUP.md",
"docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md",
"docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md",
"docs/gcc-2.7.2-map/README.md"]
"docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md"]
# whole directories in the default set (P33 F3): the wiki pages and the how-to chapters — every file, so a new page is
# checked the moment it exists (the glob is expanded at run time; the count is printed with the rest, R41)
DEFAULT_GLOBS = ["docs/wiki/*.md", "docs/how-to-ai-decomp/*.md"]
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Drew T (Druthulu)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+106
View File
@@ -0,0 +1,106 @@
# xsig — relocation-masked function signatures for cross-project code identification
Two projects that compile the **same source with the same compiler** emit byte-identical machine code except in the
operand fields the linker fills in — `j`/`jal` targets and `%hi`/`%lo` address halves — which differ by link address.
`xsig` hashes a per-function signature that masks exactly those fields and keeps everything else verbatim (opcodes,
registers, real constants, PC-relative branch offsets), so a shared signature means *the same function, linked
somewhere else*. It was written for the [Brave Fencer Musashi decompilation](https://github.com/Druthulu/BFM-decomp)
to ask one question of its unmatched functions: *does any sibling decompilation already have this body?*
MIPS (PlayStation-era) only. Single stdlib Python file, no dependencies. MIT.
## What it answers, and what it found
The question a byte-exact decompilation asks of its neighbours is not "is this code similar?" but "is this code the
same object?" — because if it is, the neighbour's C compiles to your bytes and the function costs nothing. In Phase 21
of BFM-decomp the unmatched functions of every binary were signed and joined against three sibling decompilations built
with the same gcc 2.7.2 toolchain — [Xenogears](https://github.com/ladysilverberg/xenogears-decomp),
[Vagrant Story](https://github.com/ser-pounce/rood-reverse) and [Tomba!](https://github.com/hansbonini/psx_tomba) —
each compiled to objects and listed with `objdump -dr`:
| Join | Hits | What they were |
|---|---|---|
| BFM × Xenogears + Vagrant Story | 103 (57 + 46) | every one a **PsyQ library or BIOS function** — `libapi` 34, `libcd` 14, `libetc` 14, `libspu` 11, `libcard` 7, the rest `libc2`/`libsn`/other SDK units; the largest `_spu_setReverbAttr` at 307 instructions; 37 of the 103 BFM functions were already named as library code |
| BFM × Tomba! | 126 | 124 library (77 high-confidence), **one** high-confidence non-library hit — a 19-instruction function — and one 2-instruction shape shared by 16 locations (a generic epilogue) |
The recorded verdict (BFM's decision log, 2026-07): *a clean negative — shared byte-identical code exists only in the
PsyQ library objects, zero engine code.* That closed a question with byte evidence in an afternoon, and the library
hits fed the project's other lever: link Sony's real objects instead of decompiling them. The tool is published because
the *question* transfers to any decompilation with same-compiler neighbours, even when the answer is "no".
## The signature
```
sig = sha1( little-endian masked word stream, trailing all-zero (nop) words stripped )
```
- `j` / `jal` (opcode 2/3): keep the 6-bit opcode, mask the 26-bit target — by opcode, no relocation record needed.
- An instruction carrying a **HI16/LO16 relocation**: keep opcode + `rs` + `rt`, mask the low 16 bits. The relocation is
read from the input, never guessed: a splat-style `.s` line whose operands contain `%hi(` / `%lo(`, or an
`objdump -dr` listing whose next line names `R_MIPS_HI16` / `R_MIPS_LO16`.
- Everything else is kept. **Registers are never masked** (masking them is too loose), a constant `lui` stays a
constant, a branch offset is position-independent and stays.
- Trailing nops are inter-function alignment; stripping them on both sides removes a boundary-dependent length
ambiguity and cannot forge a match.
Symmetric, explicit, and tighter than a heuristic that tracks `lui`/`addiu` pairs (which over-masks a genuine constant
`lui`). The price is the input: both sides need relocation information — a splitter's `.s` with `%hi`/`%lo`, or an object
listing — and a linked binary without emitted relocations cannot be signed faithfully.
## Usage
```bash
# sign a splitter's per-function .s files (one function per file), deduplicated by signature with a location list
python3 xsig.py sign-s asm/ --project bfm --dedup -o bfm.jsonl
# sign every function in objdump -dr listings of a sibling's objects
mipsel-linux-gnu-objdump -dr build/libcd.o > libcd.txt
python3 xsig.py sign-objdump libcd.txt other.txt --project sibling -o sibling.jsonl
# join: the same signature on both sides = the same function
python3 xsig.py cross bfm.jsonl sibling.jsonl --min-ins 8
# read the diff before believing any score: instruction-by-instruction comparison of two functions
python3 xsig.py verify asm/func_80044B14.s libcd.txt --name-b CD_flush
# the built-in property checks (no files)
python3 xsig.py selftest
```
Records are JSON Lines: `{project, tu, name, kind, sig, nins, nins_raw, mnem_head, path}` (`--dedup` adds `n_loc`
and `locs`). `cross` prints a TSV of hits sorted by instruction count, preceded by a coverage line (records and unique
signatures on each side — a join without its denominators is a number about an unknown world). `verify` exits 0 only
when the two functions are identical up to relocation and classifies every differing instruction as `opcode`,
`register`, `immediate` or `length`.
Library use: `import xsig` — `signature(pairs)`, `parse_splat_s(path)`, `parse_objdump_dr(text)`,
`compare(pairs_a, pairs_b)`, where `pairs` is a list of `(raw_word, is_address_immediate)`.
## Tests
`python3 tests/test_xsig.py` — eight checks on committed fixtures that contain **no game bytes**: one small function of
our own ([`tests/fixture.c`](tests/fixture.c)) compiled once with the gcc 2.7.2 toolchain and linked at two different
addresses with `--emit-relocs`, listed with `objdump` (the relocation records merged into the listing by
[`tests/make_fixtures.sh`](tests/make_fixtures.sh)), plus a splat-style `.s` rendering of the same function and a second,
different function. The tests prove: the same function at two link addresses signs identically while its raw words
differ; the relocated fields are present and are the only masked ones; a different function signs differently; a
one-bit register flip changes the signature and is classified as `register`; the two front-ends agree; the CLI's
`sign-objdump` + `cross` find exactly the two true pairs; `verify` accepts the pair and rejects the mismatch.
## Limits
- Same compiler, same flags, same conventions on both sides. Xenogears had to be signed under both its faithful `-G8`
and BFM's `-G0` convention; a different `-G` changes `gp`-relative addressing and the bytes.
- A function is identified, not understood: a hit says the *bytes* are shared. Small generic shapes (a two-instruction
epilogue) collide across unrelated functions — filter by instruction count, and read `verify`'s diff.
- MIPS I encodings; the relocation kinds handled are `R_MIPS_26`, `R_MIPS_HI16`, `R_MIPS_LO16` (PC16 is kept on purpose).
## Provenance
Written 2026-06-25 (BFM-decomp Phase 21, session "xdedup"), packaged 2026-09-07 (Phase 33). The method, the campaign it
served and the records it cites are in the BFM-decomp repository: the decision log, `docs/how-to-ai-decomp/`, and the
SETUP inventory row for `tools/xsig/`.
## License
MIT — see [`LICENSE`](LICENSE).
+20
View File
@@ -0,0 +1,20 @@
/* tests/fixture.c — the test fixture's source. NOT game code: two small functions written for the test, compiled with
* the toolchain xsig was built for (gcc-2.7.2-psx cc1 -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker →
* maspsx --aspsx-version=2.56 --expand-div → mipsel-linux-gnu-as) and linked TWICE at different addresses with
* --emit-relocs (see make_fixtures.sh). `fixture` exercises every field xsig masks — a jal (26-bit target), a %hi/%lo
* pair for a global (HI16/LO16) — and one it keeps (a PC-relative branch); `other` is the discrimination control. */
extern int table[8];
extern int helper(int);
int fixture(int n) {
int i, acc = 0;
for (i = 0; i < n; i++) {
acc += table[i & 7];
if (acc > 100) acc = helper(acc);
}
return acc;
}
int other(int a, int b) {
return (a * 3) ^ (b >> 2);
}
+36
View File
@@ -0,0 +1,36 @@
glabel fixture
/* 0 80010000 D8FFBD27 */ addiu sp,sp,-40
/* 4 80010004 1C00B1AF */ sw s1,28(sp)
/* 8 80010008 21888000 */ move s1,a0
/* C 8001000C 1800B0AF */ sw s0,24(sp)
/* 10 80010010 21800000 */ move s0,zero
/* 14 80010014 21200000 */ move a0,zero
/* 18 80010018 2400BFAF */ sw ra,36(sp)
/* 1C 8001001C 1300201A */ blez s1,8001006c <fixture+0x6c>
/* 20 80010020 2000B2AF */ sw s2,32(sp)
/* 24 80010024 0280123C */ lui s2,%hi(table)
/* 28 80010028 00005226 */ addiu s2,s2,%lo(table)
/* 2C 8001002C 07000232 */ andi v0,s0,0x7
/* 30 80010030 80100200 */ sll v0,v0,0x2
/* 34 80010034 21105200 */ addu v0,v0,s2
/* 38 80010038 0000428C */ lw v0,0(v0)
/* 3C 8001003C 00000000 */ nop
/* 40 80010040 21208200 */ addu a0,a0,v0
/* 44 80010044 65008228 */ slti v0,a0,101
/* 48 80010048 04004014 */ bnez v0,8001005c <fixture+0x5c>
/* 4C 8001004C 00000000 */ nop
/* 50 80010050 0048000C */ jal 80012000 <helper>
/* 54 80010054 00000000 */ nop
/* 58 80010058 21204000 */ move a0,v0
/* 5C 8001005C 01001026 */ addiu s0,s0,1
/* 60 80010060 2A101102 */ slt v0,s0,s1
/* 64 80010064 F2FF4014 */ bnez v0,80010030 <fixture+0x30>
/* 68 80010068 07000232 */ andi v0,s0,0x7
/* 6C 8001006C 21108000 */ move v0,a0
/* 70 80010070 2400BF8F */ lw ra,36(sp)
/* 74 80010074 2000B28F */ lw s2,32(sp)
/* 78 80010078 1C00B18F */ lw s1,28(sp)
/* 7C 8001007C 1800B08F */ lw s0,24(sp)
/* 80 80010080 2800BD27 */ addiu sp,sp,40
/* 84 80010084 0800E003 */ jr ra
/* 88 80010088 00000000 */ nop
+52
View File
@@ -0,0 +1,52 @@
/home/musashi/bfm-decomp/tools/xsig/tests/.fx.3YqO/a.elf: file format elf32-tradlittlemips
Disassembly of section .text:
80010000 <fixture>:
80010000: 27bdffd8 addiu sp,sp,-40
80010004: afb1001c sw s1,28(sp)
80010008: 00808821 move s1,a0
8001000c: afb00018 sw s0,24(sp)
80010010: 00008021 move s0,zero
80010014: 00002021 move a0,zero
80010018: afbf0024 sw ra,36(sp)
8001001c: 1a200013 blez s1,8001006c <fixture+0x6c>
80010020: afb20020 sw s2,32(sp)
80010024: 3c128002 lui s2,0x8002
80010024: R_MIPS_HI16 table
80010028: 26520000 addiu s2,s2,0
80010028: R_MIPS_LO16 table
8001002c: 32020007 andi v0,s0,0x7
80010030: 00021080 sll v0,v0,0x2
80010034: 00521021 addu v0,v0,s2
80010038: 8c420000 lw v0,0(v0)
8001003c: 00000000 nop
80010040: 00822021 addu a0,a0,v0
80010044: 28820065 slti v0,a0,101
80010048: 14400004 bnez v0,8001005c <fixture+0x5c>
8001004c: 00000000 nop
80010050: 0c004800 jal 80012000 <helper>
80010050: R_MIPS_26 helper
80010054: 00000000 nop
80010058: 00402021 move a0,v0
8001005c: 26100001 addiu s0,s0,1
80010060: 0211102a slt v0,s0,s1
80010064: 1440fff2 bnez v0,80010030 <fixture+0x30>
80010068: 32020007 andi v0,s0,0x7
8001006c: 00801021 move v0,a0
80010070: 8fbf0024 lw ra,36(sp)
80010074: 8fb20020 lw s2,32(sp)
80010078: 8fb1001c lw s1,28(sp)
8001007c: 8fb00018 lw s0,24(sp)
80010080: 27bd0028 addiu sp,sp,40
80010084: 03e00008 jr ra
80010088: 00000000 nop
8001008c <other>:
8001008c: 00041040 sll v0,a0,0x1
80010090: 00441021 addu v0,v0,a0
80010094: 00052883 sra a1,a1,0x2
80010098: 03e00008 jr ra
8001009c: 00451026 xor v0,v0,a1
+52
View File
@@ -0,0 +1,52 @@
/home/musashi/bfm-decomp/tools/xsig/tests/.fx.3YqO/b.elf: file format elf32-tradlittlemips
Disassembly of section .text:
80200000 <fixture>:
80200000: 27bdffd8 addiu sp,sp,-40
80200004: afb1001c sw s1,28(sp)
80200008: 00808821 move s1,a0
8020000c: afb00018 sw s0,24(sp)
80200010: 00008021 move s0,zero
80200014: 00002021 move a0,zero
80200018: afbf0024 sw ra,36(sp)
8020001c: 1a200013 blez s1,8020006c <fixture+0x6c>
80200020: afb20020 sw s2,32(sp)
80200024: 3c128023 lui s2,0x8023
80200024: R_MIPS_HI16 table
80200028: 26520000 addiu s2,s2,0
80200028: R_MIPS_LO16 table
8020002c: 32020007 andi v0,s0,0x7
80200030: 00021080 sll v0,v0,0x2
80200034: 00521021 addu v0,v0,s2
80200038: 8c420000 lw v0,0(v0)
8020003c: 00000000 nop
80200040: 00822021 addu a0,a0,v0
80200044: 28820065 slti v0,a0,101
80200048: 14400004 bnez v0,8020005c <fixture+0x5c>
8020004c: 00000000 nop
80200050: 0c084800 jal 80212000 <helper>
80200050: R_MIPS_26 helper
80200054: 00000000 nop
80200058: 00402021 move a0,v0
8020005c: 26100001 addiu s0,s0,1
80200060: 0211102a slt v0,s0,s1
80200064: 1440fff2 bnez v0,80200030 <fixture+0x30>
80200068: 32020007 andi v0,s0,0x7
8020006c: 00801021 move v0,a0
80200070: 8fbf0024 lw ra,36(sp)
80200074: 8fb20020 lw s2,32(sp)
80200078: 8fb1001c lw s1,28(sp)
8020007c: 8fb00018 lw s0,24(sp)
80200080: 27bd0028 addiu sp,sp,40
80200084: 03e00008 jr ra
80200088: 00000000 nop
8020008c <other>:
8020008c: 00041040 sll v0,a0,0x1
80200090: 00441021 addu v0,v0,a0
80200094: 00052883 sra a1,a1,0x2
80200098: 03e00008 jr ra
8020009c: 00451026 xor v0,v0,a1
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
# tests/make_fixtures.sh — regenerate the committed fixtures from tests/fixture.c with the pinned toolchain.
#
# Needs the BFM-decomp checkout's toolchain (run from that repository's root): the vintage cc1 under tools/bin/,
# maspsx under tools/maspsx/, and mipsel-linux-gnu binutils. The OUTPUTS are what the tests read — plain objdump
# listings and a splat-style .s — so the tests themselves need no compiler and contain no game bytes.
#
# fixture_a.txt objdump -dr of fixture.o linked at 0x80010000 (helper @0x80012000, table @0x80020000)
# fixture_b.txt the same object linked at 0x80200000 (helper @0x80212000, table @0x80230000)
# fixture_a.s fixture() from fixture_a.txt rewritten in splat's `/* off vaddr WORD */ mnemonic operands` form,
# with %hi()/%lo() operands where the listing carries R_MIPS_HI16/LO16
# The two listings differ ONLY in relocated fields (link addresses); xsig must sign them identically.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO="${BFM_REPO:-$(cd "$HERE/../../.." && pwd)}"
CC1="$REPO/tools/bin/gcc-2.7.2-psx/cc1"; MASPSX="$REPO/tools/maspsx/maspsx.py"; PY="${PY:-$REPO/.venv/bin/python}"
[ -x "$CC1" ] || { echo "make_fixtures: $CC1 missing (tools/bootstrap.sh in the BFM checkout)"; exit 2; }
W="$(mktemp -d "$HERE/.fx.XXXX")"; trap 'rm -rf "$W"' EXIT
mipsel-linux-gnu-cpp -lang-c -undef -Wall -fno-builtin -Dmips -D__GNUC__=2 -D__OPTIMIZE__ -Dpsx -D_PSYQ -D_MIPSEL -D_LANGUAGE_C "$HERE/fixture.c" \
| "$CC1" -quiet -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker \
| "$PY" "$MASPSX" --aspsx-version=2.56 --expand-div \
| mipsel-linux-gnu-as -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0 -o "$W/fixture.o"
link() { # $1 out.elf $2 text $3 helper $4 table
mipsel-linux-gnu-ld -q -e fixture -Ttext="$2" --defsym "helper=$3" --defsym "table=$4" -o "$1" "$W/fixture.o"
}
link "$W/a.elf" 0x80010000 0x80012000 0x80020000
link "$W/b.elf" 0x80200000 0x80212000 0x80230000
# objdump interleaves relocation records into the disassembly only for OBJECT files; for a linked ELF (--emit-relocs)
# it lists them separately (`-r`, section-relative offsets). Merge them in, exactly in the interleaved format an
# object listing has — the format xsig's front-end reads.
listing() { # $1 elf $2 text-vma $3 out
"$PY" - "$1" "$2" "$3" <<'PYEOF'
import re, subprocess, sys
elf, base, out = sys.argv[1], int(sys.argv[2], 16), sys.argv[3]
dis = subprocess.run(["mipsel-linux-gnu-objdump", "-d", elf], capture_output=True, text=True, check=True).stdout
rel = subprocess.run(["mipsel-linux-gnu-objdump", "-r", elf], capture_output=True, text=True, check=True).stdout
recs = {}
sect = None
for ln in rel.splitlines():
m = re.match(r"RELOCATION RECORDS FOR \[(\S+)\]:", ln)
if m:
sect = m.group(1); continue
m = re.match(r"^([0-9a-f]+)\s+(R_MIPS_\w+)\s+(\S+)", ln)
if m and sect == ".text":
recs.setdefault(base + int(m.group(1), 16), []).append((m.group(2), m.group(3)))
lines = []
for ln in dis.splitlines():
lines.append(ln)
m = re.match(r"^\s*([0-9a-f]+):\s+[0-9a-f]{8}\s", ln)
if m:
for typ, sym in recs.get(int(m.group(1), 16), []):
lines.append(f"\t\t\t{m.group(1)}: {typ}\t{sym}")
open(out, "w").write("\n".join(lines) + "\n")
print(f"{out}: {sum(len(v) for v in recs.values())} relocation records merged")
PYEOF
}
listing "$W/a.elf" 0x80010000 "$HERE/fixture_a.txt"
listing "$W/b.elf" 0x80200000 "$HERE/fixture_b.txt"
# the splat-style .s of fixture() from listing A: /* off vaddr LEHEX */ mnemonic operands, %hi/%lo from the reloc lines
"$PY" - "$HERE/fixture_a.txt" "$HERE/fixture_a.s" <<'EOF'
import re, struct, sys
src, dst = sys.argv[1], sys.argv[2]
lines = open(src).read().splitlines()
out, cur, i = [], None, 0
while i < len(lines):
ln = lines[i]
h = re.match(r"^([0-9a-f]+) <([^>]+)>:", ln)
if h:
cur = h.group(2); i += 1; continue
m = re.match(r"^\s*([0-9a-f]+):\s+([0-9a-f]{8})\s+(\S+)\s*(.*)", ln)
if m and cur == "fixture":
addr, word, mn, ops = int(m.group(1), 16), int(m.group(2), 16), m.group(3), m.group(4).strip()
reloc = lines[i + 1] if i + 1 < len(lines) and "R_MIPS" in lines[i + 1] else ""
if "_HI16" in reloc:
ops = re.sub(r"0x[0-9a-f]+|-?\d+$", "%hi(table)", ops) if ops else "%hi(table)"
elif "_LO16" in reloc:
ops = re.sub(r"(-?0x[0-9a-f]+|-?\d+)(\(\$\w+\))?$", r"%lo(table)\2", ops) if ops else "%lo(table)"
le = struct.pack("<I", word).hex().upper()
out.append(f"/* {addr - 0x80010000:X} {addr:X} {le} */ {mn} {ops}".rstrip())
i += 1
open(dst, "w").write("glabel fixture\n" + "\n".join(out) + "\n")
print(f"{dst}: {len(out)} instructions")
EOF
echo "fixtures regenerated: $(grep -c '^ *[0-9a-f]*:' "$HERE/fixture_a.txt") + $(grep -c '^ *[0-9a-f]*:' "$HERE/fixture_b.txt") listing lines"
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
"""tests/test_xsig.py — xsig's properties on the committed fixtures (no compiler, no game bytes needed).
python3 tests/test_xsig.py # from tools/xsig/ (or anywhere: paths are resolved from this file)
The fixtures (see make_fixtures.sh) are objdump -dr listings of one small function of our own, compiled once with the
gcc-2.7.2 toolchain and linked at two different addresses with --emit-relocs, plus a splat-style .s rendering of the
same function and a second, different function in the same listings.
"""
import io
import json
import os
import sys
import unittest
from contextlib import redirect_stdout, redirect_stderr
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
import xsig # noqa: E402
A = os.path.join(HERE, "fixture_a.txt")
B = os.path.join(HERE, "fixture_b.txt")
S = os.path.join(HERE, "fixture_a.s")
def fn(path, name):
for n, pairs, mnem in xsig.parse_objdump_dr(open(path).read()):
if n == name:
return pairs, mnem
raise AssertionError(f"{name} not in {path}")
class Fixtures(unittest.TestCase):
def test_same_function_two_link_addresses_same_signature(self):
pa, _ = fn(A, "fixture"); pb, _ = fn(B, "fixture")
self.assertNotEqual([w for w, _ in pa], [w for w, _ in pb], "the raw words must differ (different link addresses)")
self.assertEqual(xsig.signature(pa)["sig"], xsig.signature(pb)["sig"])
def test_relocated_fields_are_present_and_masked(self):
pa, _ = fn(A, "fixture")
self.assertTrue(any(a for _, a in pa), "the fixture carries HI16/LO16 relocations")
self.assertTrue(any((w >> 26) in xsig.J_OPS for w, _ in pa), "the fixture carries a jal")
c = xsig.compare(pa, fn(B, "fixture")[0])
self.assertTrue(c["same"]); self.assertEqual(c["diffs"], []); self.assertGreaterEqual(c["relocated"], 3)
def test_different_function_different_signature(self):
self.assertNotEqual(xsig.signature(fn(A, "fixture")[0])["sig"], xsig.signature(fn(A, "other")[0])["sig"])
def test_register_flip_changes_signature_and_is_classified(self):
pa, _ = fn(A, "fixture")
idx = next(i for i, (w, a) in enumerate(pa) if not a and (w >> 26) not in xsig.J_OPS and (w >> 26) != 0)
flipped = list(pa); w, a = flipped[idx]; flipped[idx] = (w ^ (1 << 16), a)
self.assertNotEqual(xsig.signature(flipped)["sig"], xsig.signature(pa)["sig"])
c = xsig.compare(pa, flipped)
self.assertEqual([d[3] for d in c["diffs"]], ["register"])
def test_splat_s_frontend_agrees_with_objdump(self):
ps, _ = xsig.parse_splat_s(S)
self.assertEqual(xsig.signature(ps)["sig"], xsig.signature(fn(A, "fixture")[0])["sig"])
def test_cli_sign_and_cross_find_the_pair(self):
out_a, out_b = os.path.join(HERE, ".a.jsonl"), os.path.join(HERE, ".b.jsonl")
try:
with redirect_stderr(io.StringIO()):
self.assertEqual(xsig.main(["sign-objdump", A, "--project", "left", "-o", out_a]), 0)
self.assertEqual(xsig.main(["sign-objdump", B, "--project", "right", "-o", out_b]), 0)
buf = io.StringIO()
with redirect_stdout(buf):
self.assertEqual(xsig.main(["cross", out_a, out_b, "--min-ins", "4"]), 0)
hits = [l for l in buf.getvalue().splitlines() if l and not l.startswith("#") and not l.startswith("nins")]
names = {(l.split("\t")[3], l.split("\t")[6]) for l in hits}
self.assertIn(("fixture", "fixture"), names); self.assertIn(("other", "other"), names)
self.assertNotIn(("fixture", "other"), names)
finally:
for p in (out_a, out_b):
if os.path.exists(p):
os.unlink(p)
def test_cli_verify(self):
with redirect_stdout(io.StringIO()):
self.assertEqual(xsig.main(["verify", A, B, "--name-a", "fixture", "--name-b", "fixture"]), 0)
self.assertEqual(xsig.main(["verify", A, B, "--name-a", "fixture", "--name-b", "other"]), 1)
def test_selftest(self):
with redirect_stdout(io.StringIO()):
self.assertEqual(xsig.selftest(), 0)
if __name__ == "__main__":
unittest.main(verbosity=1)
+329
View File
@@ -0,0 +1,329 @@
#!/usr/bin/env python3
"""xsig — relocation-masked per-function signatures for CROSS-PROJECT code identification (MIPS, PlayStation-era).
Two projects that compile the SAME source with the SAME compiler emit byte-identical machine code EXCEPT in the
relocated operand fields — `j`/`jal` 26-bit targets and `%hi`/`%lo` 16-bit immediates — which differ by link address.
xsig hashes a signature that MASKS exactly those fields and keeps everything else (opcodes, registers, true constants,
PC-relative branch offsets) verbatim, so a shared signature means "the same function, linked somewhere else".
sig = sha1( little-endian masked word stream, trailing all-zero (nop) words stripped )
Masking, driven by EXPLICIT relocation information on both sides:
* j / jal (opcode 2/3): keep the 6-bit opcode, mask the 26-bit target — by opcode, no reloc record needed;
* an instruction carrying a HI16/LO16 relocation: keep opcode+rs+rt, mask the low 16 bits. The relocation is read
from the input: a splat-style `.s` line whose operands contain `%hi(` / `%lo(`, or an `objdump -dr` listing whose
next line names R_MIPS_HI16 / R_MIPS_LO16;
* everything else is kept — registers are NEVER masked (that is too loose), a constant `lui` stays a constant.
Trailing nops are inter-function alignment padding; stripping them on both sides removes a boundary-dependent length
ambiguity and cannot forge a match. Symmetric and tighter than a heuristic that tracks `lui` pairs, which over-masks.
Library: signature(pairs) · parse_splat_s(path) · parse_objdump_dr(text) · compare(pairs_a, pairs_b)
CLI:
xsig.py sign-s PATH... [--project P] [--dedup] [-o out.jsonl] one function per splat-style .s file
xsig.py sign-objdump FILE... [--project P] [-o out.jsonl] every function in `objdump -dr` texts
xsig.py cross A.jsonl B.jsonl [--min-ins N] join two signature sets on `sig`
xsig.py verify A B [--name-a F] [--name-b G] compare two functions instruction by instruction
xsig.py selftest the built-in property checks (no files)
Input files may be splat-style `.s` (one function) or `objdump -dr` text (many functions, --name to pick one).
Written for the Brave Fencer Musashi decompilation (github.com/Druthulu/BFM-decomp), where it compared the unmatched
functions against three sibling decompilations built with the same compiler; see README.md. MIT license.
"""
import argparse
import glob
import hashlib
import json
import os
import re
import struct
import sys
__version__ = "1.0.0"
J_OPS = (2, 3) # j, jal
# ----------------------------------------------------------------------------- the signature
def mask_word(word, is_addr_imm):
op = word >> 26
if op in J_OPS:
return word & 0xFC000000 # keep opcode, mask the 26-bit target
if is_addr_imm:
return word & 0xFFFF0000 # keep opcode + rs + rt, mask the 16-bit immediate
return word & 0xFFFFFFFF # keep everything (registers, constants, branch offsets)
def _strip_trailing_nops(masked):
n = len(masked)
while n > 0 and masked[n - 1] == 0:
n -= 1
return masked[:n]
def signature(pairs):
"""pairs: [(raw_word, is_addr_imm)] → {sig, nins, nins_raw, masked}; `masked` is the trailing-nop-stripped
masked word list, kept for diffing near-misses."""
masked = [mask_word(w, a) for (w, a) in pairs]
core = _strip_trailing_nops(masked)
buf = b"".join(struct.pack("<I", w) for w in core)
return {"sig": hashlib.sha1(buf).hexdigest(), "nins": len(core), "nins_raw": len(pairs), "masked": core}
# ----------------------------------------------------------------------------- front-end: splat-style .s
# line form: /* <offset> <vaddr> <LE-HEX-WORD> */ <mnemonic> <operands>
_SLINE = re.compile(r"/\*\s*[0-9A-Fa-f]+\s+[0-9A-Fa-f]+\s+([0-9A-Fa-f]{8})\s*\*/\s+(\S.*)")
def parse_splat_s(path):
"""One function per file (splat's `nonmatchings/*.s`). Returns (pairs, mnemonics)."""
pairs, mnem = [], []
with open(path, "r", errors="replace") as fh:
for line in fh:
m = _SLINE.search(line)
if not m:
continue
word = struct.unpack("<I", bytes.fromhex(m.group(1)))[0]
ops = m.group(2)
pairs.append((word, ("%hi(" in ops) or ("%lo(" in ops)))
mnem.append(ops.split()[0] if ops.split() else "")
return pairs, mnem
# ----------------------------------------------------------------------------- front-end: objdump -dr text
_HDR = re.compile(r"^[0-9a-f]+ <([^>]+)>:")
_INS = re.compile(r"^\s*[0-9a-f]+:\s+([0-9a-f]{8})\s+(.*)") # an object file indents the address, a linked ELF does not
def parse_objdump_dr(text):
"""Every function in an `objdump -dr` listing → [(name, pairs, mnemonics)]. HI16/LO16 relocation lines flip the
preceding instruction's is_addr flag; R_MIPS_26 is handled by opcode; R_MIPS_PC16 (branches) is kept."""
out, cur, pairs, mnem = [], None, [], []
def flush():
if cur is not None and pairs:
out.append((cur, [(w, bool(a)) for (w, a) in pairs], list(mnem)))
for line in text.splitlines():
h = _HDR.match(line)
if h:
flush()
cur, pairs, mnem = h.group(1), [], []
continue
mi = _INS.match(line)
if mi:
pairs.append([int(mi.group(1), 16), False])
mnem.append(mi.group(2).split()[0] if mi.group(2).split() else "")
continue
if "R_MIPS" in line and pairs and (("_HI16" in line) or ("_LO16" in line)):
pairs[-1][1] = True
flush()
return out
def load_functions(path, name=None):
"""A .s file (one function, named after the file) or an objdump text (many). Returns [(name, pairs, mnem)]."""
if path.endswith(".s"):
pairs, mnem = parse_splat_s(path)
return [(os.path.splitext(os.path.basename(path))[0], pairs, mnem)]
fns = parse_objdump_dr(open(path, errors="replace").read())
if name is not None:
fns = [f for f in fns if f[0] == name]
if not fns:
sys.exit(f"xsig: no function named {name!r} in {path}")
return fns
# ----------------------------------------------------------------------------- compare (the diff a score cannot replace)
def compare(pairs_a, pairs_b):
"""Instruction-by-instruction comparison of two functions' masked streams. Returns a dict:
same (bool), nins_a/nins_b, relocated (fields masked on the A side), diffs = [(index, word_a, word_b, kind)]
with kind ∈ {'opcode', 'register', 'immediate', 'length'} — read it before believing any score."""
sa, sb = signature(pairs_a), signature(pairs_b)
diffs = []
n = max(len(sa["masked"]), len(sb["masked"]))
for i in range(n):
if i >= len(sa["masked"]) or i >= len(sb["masked"]):
diffs.append((i, sa["masked"][i] if i < len(sa["masked"]) else None,
sb["masked"][i] if i < len(sb["masked"]) else None, "length"))
continue
wa, wb = sa["masked"][i], sb["masked"][i]
if wa == wb:
continue
if (wa >> 26) != (wb >> 26) or ((wa >> 26) == 0 and (wa & 0x3F) != (wb & 0x3F)):
kind = "opcode"
elif (wa & 0x03E00000) != (wb & 0x03E00000) or (wa & 0x001F0000) != (wb & 0x001F0000) \
or ((wa >> 26) == 0 and (wa & 0x0000F800) != (wb & 0x0000F800)):
kind = "register"
else:
kind = "immediate"
diffs.append((i, wa, wb, kind))
relocated = sum(1 for (w, a) in pairs_a if a or (w >> 26) in J_OPS)
return {"same": sa["sig"] == sb["sig"], "sig_a": sa["sig"], "sig_b": sb["sig"], "nins_a": sa["nins"],
"nins_b": sb["nins"], "relocated": relocated, "diffs": diffs}
# ----------------------------------------------------------------------------- CLI
def _record(project, tu, name, pairs, mnem, path=""):
s = signature(pairs)
return {"project": project, "tu": tu, "name": name, "kind": "c_function", "sig": s["sig"], "nins": s["nins"],
"nins_raw": s["nins_raw"], "mnem_head": " ".join(mnem[:6]), "path": path}
def _write(records, out):
fh = open(out, "w") if out else sys.stdout
for r in records:
fh.write(json.dumps(r, separators=(",", ":")) + "\n")
if out:
fh.close()
def cmd_sign_s(a):
paths = []
for p in a.paths:
paths += sorted(glob.glob(os.path.join(p, "**", "*.s"), recursive=True)) if os.path.isdir(p) else [p]
recs, by_sig, empty = [], {}, 0
for p in paths:
pairs, mnem = parse_splat_s(p)
if not pairs:
empty += 1
continue
r = _record(a.project, os.path.dirname(p), os.path.splitext(os.path.basename(p))[0], pairs, mnem, p)
if a.dedup:
e = by_sig.get(r["sig"])
if e is None:
r["n_loc"], r["locs"] = 1, [{"tu": r["tu"], "name": r["name"]}]
by_sig[r["sig"]] = r
else:
e["n_loc"] += 1
if len(e["locs"]) < 40:
e["locs"].append({"tu": r["tu"], "name": r["name"]})
else:
recs.append(r)
recs = list(by_sig.values()) if a.dedup else recs
_write(recs, a.out)
print(f"xsig sign-s: {len(paths)} files ({empty} without instructions) -> {len(recs)} records"
+ (f" ({len(by_sig)} unique signatures)" if a.dedup else "") + (f" -> {a.out}" if a.out else ""), file=sys.stderr)
return 0 if paths else 2
def cmd_sign_objdump(a):
recs = []
for f in a.files:
tu = os.path.splitext(os.path.basename(f))[0]
for name, pairs, mnem in parse_objdump_dr(open(f, errors="replace").read()):
recs.append(_record(a.project, tu, name, pairs, mnem, f))
_write(recs, a.out)
print(f"xsig sign-objdump: {len(a.files)} listings -> {len(recs)} functions" + (f" -> {a.out}" if a.out else ""),
file=sys.stderr)
return 0 if recs else 2
def _load_jsonl(path):
return [json.loads(l) for l in open(path) if l.strip()]
def cmd_cross(a):
A, B = _load_jsonl(a.a), _load_jsonl(a.b)
by_sig = {}
for r in B:
by_sig.setdefault(r["sig"], []).append(r)
hits = [(ra, rb) for ra in A for rb in by_sig.get(ra["sig"], []) if ra["nins"] >= a.min_ins]
hits.sort(key=lambda h: -h[0]["nins"])
print(f"# xsig cross: {len(A)} records ({len(set(r['sig'] for r in A))} unique sigs) x {len(B)} records "
f"({len(by_sig)} unique sigs), min-ins {a.min_ins} -> {len(hits)} hits")
print("nins\tsig\tproject_a\tname_a\ttu_a\tproject_b\tname_b\ttu_b")
for ra, rb in hits:
print(f"{ra['nins']}\t{ra['sig'][:12]}\t{ra.get('project','')}\t{ra['name']}\t{ra.get('tu','')}\t"
f"{rb.get('project','')}\t{rb['name']}\t{rb.get('tu','')}")
return 0
def cmd_verify(a):
fa = load_functions(a.a, a.name_a)
fb = load_functions(a.b, a.name_b)
if len(fa) != 1 or len(fb) != 1:
sys.exit(f"xsig verify: pick one function per side (--name-a / --name-b): A has {len(fa)}, B has {len(fb)}")
(na, pa, ma), (nb, pb, mb) = fa[0], fb[0]
c = compare(pa, pb)
print(f"xsig verify: {na} ({c['nins_a']} ins) vs {nb} ({c['nins_b']} ins) — {c['relocated']} relocated fields masked on A")
for i, wa, wb, kind in c["diffs"]:
fa_ = f"{wa:08x}" if wa is not None else "--------"
fb_ = f"{wb:08x}" if wb is not None else "--------"
print(f" #{i:<4} {fa_} {fb_} {kind:9s} {ma[i] if i < len(ma) else ''}")
kinds = {}
for _, _, _, k in c["diffs"]:
kinds[k] = kinds.get(k, 0) + 1
print("xsig verify: " + ("IDENTICAL up to relocation" if c["same"] else
"DIFFERENT — " + ", ".join(f"{v} {k}" for k, v in sorted(kinds.items()))))
return 0 if c["same"] else 1
def selftest():
"""Property checks on a synthetic stream (no files, no game bytes): self-match, relocation invariance,
register sensitivity, discrimination, trailing-nop stripping, and the two front-ends agreeing on one function."""
import random
rng = random.Random(1)
# a synthetic 12-instruction function: lui/addiu %hi/%lo pair, a jal, a branch, ALU, loads, a nop tail
words = [(0x27BDFFE8, False), (0xAFBF0014, False), (0x3C028001, True), (0x24422A40, True), (0x8C430000, False),
(0x0C004A10, False), (0x00000000, False), (0x10400003, False), (0x24630001, False), (0x8FBF0014, False),
(0x03E00008, False), (0x27BD0018, False), (0x00000000, False), (0x00000000, False)]
base = signature(words)
assert signature(words)["sig"] == base["sig"], "self-match"
assert base["nins"] == 12 and base["nins_raw"] == 14, "trailing nops stripped, the delay-slot nop kept"
pert = []
for w, a in words:
if (w >> 26) in J_OPS:
pert.append(((w & 0xFC000000) | rng.randint(1, 0x3FFFFFF), a))
elif a:
pert.append(((w & 0xFFFF0000) | rng.randint(1, 0xFFFF), a))
else:
pert.append((w, a))
assert signature(pert)["sig"] == base["sig"], "relocation invariance"
reg = list(words); w, a = reg[4]; reg[4] = (w ^ (1 << 16), a) # flip rt of the lw
assert signature(reg)["sig"] != base["sig"], "register sensitivity"
other = [(0x00851021, False), (0x03E00008, False), (0x00000000, False)]
assert signature(other)["sig"] != base["sig"], "discrimination"
c = compare(words, reg)
assert not c["same"] and c["diffs"] and c["diffs"][0][3] == "register", "compare classifies a register flip"
# the two front-ends on the same function
s_lines, od_lines = [], ["00000000 <fn>:"]
for i, (w, a) in enumerate(words):
le = struct.pack("<I", w).hex().upper()
s_lines.append(f"/* {i*4:X} {0x80010000 + i*4:X} {le} */ ins {'%hi(sym)' if a else 'x'}")
od_lines.append(f" {i*4:x}:\t{w:08x} \tins\tx")
if a:
od_lines.append(f"\t\t\t{i*4:x}: R_MIPS_{'HI16' if i == 2 else 'LO16'}\tsym")
import tempfile
with tempfile.NamedTemporaryFile("w", suffix=".s", delete=False) as t:
t.write("\n".join(s_lines) + "\n"); sp = t.name
try:
pa, _ = parse_splat_s(sp)
finally:
os.unlink(sp)
fb = parse_objdump_dr("\n".join(od_lines))
assert len(fb) == 1 and signature(pa)["sig"] == signature(fb[0][1])["sig"] == base["sig"], "front-ends agree"
print("xsig selftest: 7 checks passed (self-match, nop stripping, relocation invariance, register sensitivity, "
"discrimination, compare classification, front-ends agree)")
return 0
def main(argv=None):
ap = argparse.ArgumentParser(prog="xsig", description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--version", action="version", version=f"xsig {__version__}")
sub = ap.add_subparsers(dest="cmd", required=True)
p = sub.add_parser("sign-s", help="sign splat-style .s files (one function each)")
p.add_argument("paths", nargs="+"); p.add_argument("--project", default="a"); p.add_argument("--dedup", action="store_true")
p.add_argument("-o", "--out"); p.set_defaults(fn=cmd_sign_s)
p = sub.add_parser("sign-objdump", help="sign every function in objdump -dr listings")
p.add_argument("files", nargs="+"); p.add_argument("--project", default="b"); p.add_argument("-o", "--out")
p.set_defaults(fn=cmd_sign_objdump)
p = sub.add_parser("cross", help="join two signature sets on sig")
p.add_argument("a"); p.add_argument("b"); p.add_argument("--min-ins", type=int, default=1); p.set_defaults(fn=cmd_cross)
p = sub.add_parser("verify", help="compare two functions instruction by instruction")
p.add_argument("a"); p.add_argument("b"); p.add_argument("--name-a"); p.add_argument("--name-b"); p.set_defaults(fn=cmd_verify)
p = sub.add_parser("selftest", help="the built-in property checks"); p.set_defaults(fn=lambda a: selftest())
a = ap.parse_args(argv)
return a.fn(a)
if __name__ == "__main__":
sys.exit(main())