From b29963eafbe625b0049b4d23de4bfab9a340815c Mon Sep 17 00:00:00 2001 From: Christopher Williams Date: Thu, 24 Sep 2026 11:47:41 -0400 Subject: [PATCH] phase11: promote sf3_cc + sf3_diff to tools/ (worker F) + cookbook 184-185 Worker F's judgement, which I agree with: these two together are the highest-value tooling built this phase, because the first removes the COST of a spelling and the second removes the GUESSWORK about what is wrong. tools/sf3_cc one-file cpp -> cc1 -> maspsx, printing the assembly. A spelling costs ~0.1s, which turns 'try a few variants' into 'grid the whole space'. tools/sf3_diff 0xS 0xE opcode-level diff of the candidate object against the original, difflib-aligned, so you see the SHAPE of the residual rather than a byte count. A differing-byte count tells you HOW WRONG a candidate is; it does not tell you WHICH DIMENSION the error lives in. sf3_diff marks differing instructions with '<<', so a run of identical mnemonics with differing register names is the allocator class -- a CLASSIFY signal, not a spelling signal. 184: a load's BASIC BLOCK is not source-movable -- sched2 cannot cross blocks, so a load in a different block than the candidate's is a SOURCE-ORDER fact. --- docs/MATCHING_COOKBOOK.md | 28 +++ tools/sf3_cc | 34 ++++ tools/sf3_diff | 406 +++++++------------------------------- 3 files changed, 131 insertions(+), 337 deletions(-) create mode 100755 tools/sf3_cc diff --git a/docs/MATCHING_COOKBOOK.md b/docs/MATCHING_COOKBOOK.md index 30214b7..a24a346 100644 --- a/docs/MATCHING_COOKBOOK.md +++ b/docs/MATCHING_COOKBOOK.md @@ -2987,3 +2987,31 @@ out to be **a transcription error**: it had written **two** dereference levels w direction for this case* — 149 is for bodies short by **missing nops around independent blocks**; this is short by a **missing load**. **Two different causes with the same 4–8 byte signature**, which is exactly why the byte count alone is not a diagnosis. + +### 184. A load's BASIC BLOCK is not source-movable (worker F) + +If the original has a global load in the **ENTRY block**, the source must load it **before the `if`** — +**sched2 cannot move a load across a basic block.** A single `d = D_80121B18 - x;` written *after* the +`if` leaves the load in the **merge** block; the original's is in the entry block, so the source must +be two statements (`int t = D_80121B18;` then `d = t - x;`). + +> **A load in a different basic block than your candidate's is a SOURCE-ORDER fact, not a scheduler +> fact** — the scheduler cannot cross blocks, so the ordering has to come from the source. + +### 185. Two tools promoted: `sf3_cc` and `sf3_diff` (worker F) + +Worker F's judgement, and I agree with it: **these two together are the highest-value tooling built +this phase**, because the first removes the *cost* of a spelling and the second removes the +*guesswork* about what is wrong. + +| tool | what it does | +|---|---| +| **`tools/sf3_cc `** | one-file cpp → cc1 → maspsx, printing the assembly. **A spelling costs ~0.1 s**, which is what turns "try a few variants" into "grid the whole space". | +| **`tools/sf3_diff 0xSTART 0xEND `** | opcode-level diff of the candidate object against the original bytes, aligned with difflib, so you see the **shape** of the residual rather than a byte count. | + +**A differing-byte count tells you HOW WRONG a candidate is; it does not tell you WHICH DIMENSION +the error lives in.** `sf3_diff` marks each aligned pair with `<<` when the instruction text differs, +so **a run of `<<` lines with identical mnemonics and differing register names is the allocator +class** — which is a CLASSIFY signal, not a spelling signal (finding 116). + +**Worker F's own example of why it matters:** *"this is what made every ordering failure legible."* diff --git a/tools/sf3_cc b/tools/sf3_cc new file mode 100755 index 0000000..65c5467 --- /dev/null +++ b/tools/sf3_cc @@ -0,0 +1,34 @@ +#!/bin/sh +# usage: sf3_cc -> prints cc1 assembly (maspsx applied) for one file +# +# WHY THIS EXISTS (worker F, Phase 11): a full `sf3_match range` run compiles, links, extracts +# and compares. When you are sweeping SPELLINGS you only need the assembly, and the sweep cost +# dominates everything else. This makes one spelling cost ~0.1 s, which is what turns "try a +# few variants" into "grid the whole space". +# +# Pair it with `sf3_diff`, which tells you WHICH DIMENSION a residual lives in -- worker F's +# judgement was that these two together are the highest-value tooling it built, because the +# first removes the cost of a spelling and the second removes the guesswork about what is wrong. +# +# Pipeline: cpp -> cc1 (the pinned 2.7.2-psx build) -> maspsx (the project's vendored patched +# version), which is exactly what sf3_match does internally. +set -e +REPO=$(cd "$(dirname "$0")/.." && pwd) +cd "$REPO" + +if [ $# -lt 1 ]; then + echo "usage: sf3_cc " >&2 + exit 2 +fi + +SRC=$1 +BASE=$(basename "$SRC" .c) +OUT=${SF3_CC_OUT:-.run/sf3_cc} +mkdir -p "$OUT" + +tools/old-gcc/gcc-2.7.2-psx/cpp -E -P -undef "$SRC" > "$OUT/$BASE.i" +tools/old-gcc/gcc-2.7.2-psx/cc1 -quiet -O2 -G0 "$OUT/$BASE.i" -o "$OUT/$BASE.s" +python3 tools/maspsx/maspsx.py --aspsx-version=2.56 "$OUT/$BASE.s" "$OUT/$BASE.ms.s" 2>/dev/null \ + || cp "$OUT/$BASE.s" "$OUT/$BASE.ms.s" + +cat "$OUT/$BASE.ms.s" diff --git a/tools/sf3_diff b/tools/sf3_diff index 4ea09e1..ac06994 100755 --- a/tools/sf3_diff +++ b/tools/sf3_diff @@ -1,365 +1,97 @@ #!/usr/bin/env python3 -"""Instruction-level diff and address resolver for one matching attempt. +"""Opcode-level diff of a CANDIDATE object against the ORIGINAL bytes. -Phase 8's workers diagnosed near-misses with a scratch `cmp.py` under the -ignored `.run/` tree. Phase 9 promotes it into the tracked toolchain, hardened -as its author specified: +Usage: + sf3_diff 0xSTART 0xEND - * the PS-X text address (LOAD) is read from the executable's header, never - hardcoded, and the payload file offset is derived from it; - * the objdump path is derived from the repository (`tools/.../binutils`); - * every temp file is written under a caller-supplied directory, because - three Phase 9 sessions share one worktree and a fixed `/tmp` name would - collide; - * the address resolver prints ready-to-paste symbol names - (`D_80122308`, `func_80020000`) rather than bare hex, so a worker copies - the name the harness will resolve. +`workdir` is a directory produced by `sf3_match range` (it contains `candidate.o`). The +original range is disassembled from the executable and the two instruction streams are +aligned with difflib, so the output shows the *shape* of the residual rather than a byte +count. -It has two subcommands: +WHY THIS EXISTS (worker F, Phase 11): a differing-byte count tells you HOW WRONG a candidate +is; it does not tell you WHICH DIMENSION the error lives in. Worker F's judgement was that +this tool is what made every ordering failure legible -- you can see at a glance whether the +residual is a register field, an operand order, a displaced block, or a missing instruction. - diff side-by-side instruction diff of the original payload range and one - candidate `.bin`, the same bytes `sf3_match range` links into - `/single.bin`. Prints the differing instruction count, the - first differing address, and a per-instruction table with each - resolved address annotated with a ready-to-paste symbol name. - resolve read an objdump-style disassembly listing and print, per line, the - resolved absolute address of every `off(gp)` operand and every - `lui r,HI` / `off(r)` pair, again as ready-to-paste names. +Pair it with `sf3_cc`, which makes one spelling cost ~0.1 s. -Neither subcommand matches anything: `sf3_match range` is the per-function -comparator and `sf3_match gate` the whole-binary authority. This tool exists so -a worker can see *which* instructions differ and *what address each operand -means*, which is what turns a `DIFF` into a decision. - -The tool never embeds game bytes in its source. Tests use self-authored -fixtures only and never read the extracted executable. - -Exit codes: 0 success, 1 mismatch (diff found), 2 usage or environment error. +The diff marks each aligned pair with `<<` when the instruction TEXT differs, so a run of +`<<` lines with identical mnemonics and differing register names is the allocator class -- +which is a CLASSIFY signal, not a spelling signal (finding 116). """ - from __future__ import annotations -import argparse +import difflib +import os import re -from dataclasses import dataclass -from pathlib import Path -import struct import subprocess import sys -from typing import Sequence +import tempfile +from pathlib import Path - -REPO_ROOT = Path(__file__).resolve().parent.parent -DEFAULT_OBJDUMP = ( - REPO_ROOT - / "tools/mipsel-none-elf-binutils/prefix/usr/bin/mipsel-none-elf-objdump" -) +REPO = Path(__file__).resolve().parent.parent +OBJDUMP = REPO / "tools/mipsel-none-elf-binutils/prefix/usr/bin/mipsel-none-elf-objdump" +EXE = REPO / "extracted/SCUS_946.40;1" +LOAD_ADDRESS = 0x80010000 HEADER_SIZE = 0x800 -PAYLOAD_LMA = 0x800 -EXE_MAGIC = b"PS-X EXE" -# The Phase 6 small-data census fixed gp at this address (cookbook finding 10). -DEFAULT_GP = 0x80121938 + +LINE = re.compile(r"^\s*([0-9a-f]+):\s+([0-9a-f]{8})\s+(.*)$") -class ToolError(Exception): - """A usage or environment problem; maps to exit code 2.""" +def parse_disassembly(text: str) -> list[tuple[str, str]]: + out = [] + for line in text.splitlines(): + match = LINE.match(line) + if match: + out.append((match.group(2), match.group(3).strip().replace("\t", " "))) + return out -# -------------------------------------------------------------------------- -# PS-X EXE parsing (aligned with sf3_match's parse_psx_exe) -# -------------------------------------------------------------------------- +def main(argv: list[str]) -> int: + if len(argv) < 3: + print(__doc__.strip()) + return 2 + start, end = int(argv[0], 16), int(argv[1], 16) + workdir = Path(argv[2]) + candidate = workdir / "candidate.o" + if not candidate.exists(): + print(f"no candidate.o in {workdir} -- run sf3_match range first", file=sys.stderr) + return 2 + payload = EXE.read_bytes() + offset = HEADER_SIZE + (start - LOAD_ADDRESS) + raw = payload[offset:offset + (end - start)] -@dataclass(frozen=True) -class PsxExe: - entry: int - text_address: int - text_size: int + with tempfile.TemporaryDirectory() as tmp: + binary = os.path.join(tmp, "range.bin") + with open(binary, "wb") as handle: + handle.write(raw) + original = subprocess.run( + [str(OBJDUMP), "-D", "-b", "binary", "-m", "mips:3000", + "--adjust-vma=%#x" % start, binary], + capture_output=True, text=True).stdout - @property - def payload_end(self) -> int: - return self.text_address + self.text_size + cand = subprocess.run( + [str(OBJDUMP), "-d", "-m", "mips:3000", str(candidate)], + capture_output=True, text=True).stdout - def file_offset(self, address: int) -> int: - """File offset of a payload address (the payload starts at 0x800).""" - return PAYLOAD_LMA + (address - self.text_address) + left, right = parse_disassembly(original), parse_disassembly(cand) + print(f"orig {len(left)} instructions cand {len(right)} instructions") - -def parse_psx_exe(data: bytes) -> PsxExe: - if len(data) < HEADER_SIZE: - raise ToolError("executable is smaller than a PS-X EXE header") - if data[:8] != EXE_MAGIC: - raise ToolError("executable does not carry the PS-X EXE magic") - entry, _gp, text_address, text_size = struct.unpack_from(" dict[int, str]: - """Disassemble raw bytes as MIPS at `vaddr` and return addr -> text rows. - - All temp files go under `work` (caller-supplied, so sessions never - collide); `stem` keeps several dumps from the same workdir apart. - """ - binary = work / f"{stem}.bin" - binary.write_bytes(data) - out = subprocess.run( - [ - str(objdump), "-D", "-b", "binary", "-m", "mips:3000", - f"--adjust-vma=%#x" % vaddr, str(binary), - ], - capture_output=True, text=True, check=True, - ).stdout - rows: dict[int, str] = {} - for line in out.splitlines(): - parts = line.split("\t") - if len(parts) >= 3 and parts[0].endswith(":"): - try: - addr = int(parts[0].rstrip(":"), 16) - except ValueError: - continue - rows[addr] = parts[2].split("#")[0].strip() - return rows - - -# -------------------------------------------------------------------------- -# address resolution -# -------------------------------------------------------------------------- - -GP_OPERAND = re.compile(r"(-?\w+)\(gp\)") -LUI = re.compile(r"lui\s+(\w+),(-?\w+)") -MEM = re.compile(r"\w+\s+\w+,(-?\w+)\((\w+)\)") - - -def symbol_name(address: int, registry: dict[int, str]) -> str: - """Ready-to-paste name for an address: registry name, else an - address-shaped placeholder. The harness resolves func_/D_/g_/lbl_ + 8 hex - digits implicitly, so the placeholder is directly usable in a source file. - """ - if address in registry: - return registry[address] - return f"D_{address:08X}" - - -def annotate_text( - text: str, registry: dict[int, str], gp: int -) -> tuple[str, list[str]]: - """Return (text, notes) with every resolvable operand annotated. - - Notes use ready-to-paste symbol names; for section-ambiguous addresses we - print the `D_` placeholder and the worker picks func_/g_/lbl_ if the bytes - say so. A registry row always wins over the placeholder. - """ - notes: list[str] = [] - mg = GP_OPERAND.search(text) - if mg: - try: - off = int(mg.group(1)) - except ValueError: - off = 0 - notes.append(f"-> {symbol_name(gp + off, registry)} (gp{off:+d})") - ml = LUI.match(text) - if ml: - try: - hi = int(ml.group(2), 16) << 16 - except ValueError: - hi = 0 - mo = MEM.search(text) - if mo and mo.group(2) == ml.group(1): - try: - lo = int(mo.group(1)) - except ValueError: - lo = 0 - if lo >= 0x8000: - lo -= 0x10000 - notes.append(f"-> {symbol_name(hi + lo, registry)} (abs)") - return text, notes - - -def load_registry(path: Path | None) -> dict[int, str]: - registry: dict[int, str] = {} - if path is None: - return registry - for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): - if not line.strip() or line.lstrip().startswith("#"): + matcher = difflib.SequenceMatcher(None, [x[1] for x in left], [x[1] for x in right], + autojunk=False) + for tag, i1, i2, j1, j2 in matcher.get_opcodes(): + if tag == "equal": continue - fields = line.split("\t") - if len(fields) < 2: - raise ToolError(f"symbols line {line_number}: expected NAMEaddress") - try: - address = int(fields[1], 16) - except ValueError as exc: - raise ToolError( - f"symbols line {line_number}: not a hex address: {fields[1]!r}" - ) from exc - registry.setdefault(address, fields[0]) - return registry - - -# -------------------------------------------------------------------------- -# subcommands -# -------------------------------------------------------------------------- - - -def command_diff(args: argparse.Namespace) -> int: - data = args.exe.read_bytes() - exe = parse_psx_exe(data) - if args.start < exe.text_address or args.end > exe.payload_end: - raise ToolError("requested range leaves the payload") - - if not args.work.is_dir(): - raise ToolError(f"work directory does not exist: {args.work}") - if not args.candidate.is_file(): - raise ToolError(f"candidate file does not exist: {args.candidate}") - registry = load_registry(args.symbols) - - original = data[exe.file_offset(args.start): exe.file_offset(args.end)] - candidate = args.candidate.read_bytes() - - da = disassemble(original, args.start, args.objdump, args.work, "orig") - db = disassemble(candidate, args.start, args.objdump, args.work, "cand") - - diff = [i for i in range(len(original)) if i >= len(candidate) - or original[i] != candidate[i]] - print(f"range=0x{args.start:08X}..0x{args.end:08X}") - print(f"orig_len={len(original)} cand_len={len(candidate)} " - f"differing_bytes={len(diff)}") - if not diff: - print("result=MATCH") - return 0 - first = args.start + (diff[0] // 4) * 4 - print(f"first_difference=0x{first:08X}") - print("result=DIFF") - print("%-10s %-22s | %-22s notes" % ("addr", "ORIGINAL", "CANDIDATE")) - for off in range(0, len(original), 4): - ea = original[off:off + 4] - eb = candidate[off:off + 4] if off < len(candidate) else b"" - if ea == eb: - continue - addr = args.start + off - text_a, notes_a = annotate_text( - da.get(addr, "?"), registry, args.gp) - text_b, notes_b = annotate_text( - db.get(addr, "?"), registry, args.gp) - notes = " ".join(dict.fromkeys(notes_a + notes_b)) - mark = " <== first" if addr == first else "" - print("%-10s %-22s | %-22s %s%s" - % (hex(addr), text_a, text_b, notes, mark)) - return 1 - - -def command_resolve(args: argparse.Namespace) -> int: - registry = load_registry(args.symbols) - line_re = re.compile(r"^\s*([0-9a-f]{8}):") - prev_hi: dict[str, int] = {} - for line in args.disasm.read_text(encoding="utf-8").splitlines(): - m = line_re.match(line) - addr = int(m.group(1), 16) if m else None - text = line.split("\t")[-1].split("<")[0].strip() if "\t" in line else line.strip() - note = "" - mg = GP_OPERAND.search(text) - if mg: - try: - off = int(mg.group(1)) - except ValueError: - off = 0 - note = f" -> {symbol_name(args.gp + off, registry)} (gp{off:+d})" - ml = LUI.match(text) - if ml: - try: - prev_hi[ml.group(1)] = int(ml.group(2), 16) << 16 - except ValueError: - pass - mo = MEM.search(text) - if mo and mo.group(2) == ml.group(1): - try: - lo = int(mo.group(1)) - except ValueError: - lo = 0 - if lo >= 0x8000: - lo -= 0x10000 - note = f" -> {symbol_name(prev_hi[ml.group(1)] + lo, registry)} (abs)" - mo = MEM.search(text) - if not note and mo and mo.group(2) in prev_hi: - try: - lo = int(mo.group(1)) - except ValueError: - lo = 0 - if lo >= 0x8000: - lo -= 0x10000 - note = f" -> {symbol_name(prev_hi[mo.group(2)] + lo, registry)} (abs)" - if note: - prefix = f"{addr:08x}: " if addr is not None else "" - print(f"{prefix}{text}{note}") + print(f"--- {tag} orig[{i1}:{i2}] cand[{j1}:{j2}]") + for k in range(max(i2 - i1, j2 - j1)): + a = left[i1 + k] if i1 + k < i2 else ("", "") + b = right[j1 + k] if j1 + k < j2 else ("", "") + mark = "<<" if a[1] != b[1] else " " + print(f" {mark} O {a[0]:<8} {a[1]:<32} | C {b[0]:<8} {b[1]}") return 0 -# -------------------------------------------------------------------------- -# CLI -# -------------------------------------------------------------------------- - - -def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - prog="sf3_diff", - description="Instruction-level diff and address resolver for one " - "matching attempt. Phase 8's worker scratch helper, " - "promoted and hardened in Phase 9.", - ) - sub = parser.add_subparsers(dest="command", required=True) - - p_diff = sub.add_parser("diff", help="diff one range against a candidate") - p_diff.add_argument("--exe", required=True, type=Path, - help="validated PS-X EXE (the ORIGINAL)") - p_diff.add_argument("--start", required=True, help="range start, hex") - p_diff.add_argument("--end", required=True, help="range end, hex") - p_diff.add_argument("--candidate", required=True, type=Path, - help="candidate .bin (sf3_match range writes " - "/single.bin)") - p_diff.add_argument("--work", required=True, type=Path, - help="caller-supplied directory for temp files " - "(sessions share the worktree; never /tmp)") - p_diff.add_argument("--objdump", type=Path, default=DEFAULT_OBJDUMP, - help="objdump binary (default: repository binutils)") - p_diff.add_argument("--symbols", type=Path, default=None, - help="tracked symbol registry (NAMEaddress)") - p_diff.add_argument("--gp", type=lambda s: int(s, 0), default=DEFAULT_GP, - help=f"gp base for gp-relative resolution " - f"(default: {DEFAULT_GP:#x})") - p_diff.set_defaults(handler=command_diff) - - p_res = sub.add_parser("resolve", help="resolve addresses in a disassembly") - p_res.add_argument("--disasm", required=True, type=Path, - help="objdump-style disassembly text") - p_res.add_argument("--symbols", type=Path, default=None, - help="tracked symbol registry (NAMEaddress)") - p_res.add_argument("--gp", type=lambda s: int(s, 0), default=DEFAULT_GP, - help=f"gp base (default: {DEFAULT_GP:#x})") - p_res.set_defaults(handler=command_resolve) - return parser - - -def main(argv: Sequence[str] | None = None) -> int: - parser = build_parser() - args = parser.parse_args(argv) - try: - if getattr(args, "start", None) is not None: - args.start = int(args.start, 0) - if getattr(args, "end", None) is not None: - args.end = int(args.end, 0) - return args.handler(args) - except ToolError as exc: - print(f"sf3_diff: {exc}", file=sys.stderr) - return 2 - - if __name__ == "__main__": - sys.exit(main()) \ No newline at end of file + raise SystemExit(main(sys.argv[1:]))