phase6: add the evidence-graded function-boundary inventory

2026-09-23 21:12:54 -04:00
parent d379b84ec3
commit d951e49b1d
5 changed files with 3336 additions and 1 deletions
@@ -0,0 +1,68 @@
+# Phase 6 — Evidence-Graded Function-Boundary Inventory
+
+**Scope:** P6-T5 — replace address-gap guessing with an inventory of candidate function starts graded
+by evidence, and state the limits of every grade.
+**Status:** complete. The inventory is tracked as `config/function_inventory.tsv`; no candidate is
+promoted to a match on the strength of the inventory alone.
+
+## What the inventory is
+
+`config/function_inventory.tsv` lists every candidate function start in the payload with the evidence
+that supports it. It contains **addresses and grade names only** — no instruction bytes and no
+disassembly.
+
+| Grade | Definition | Basis | Limit |
+|---|---|---|---|
+| `entry` | the PS-X EXE header's entry PC | header field | definite address, but it is CRT startup, not C |
+| `jal` | the target of a `jal` instruction in the payload | direct call site | strongest grade: a called address is a function entry; a shared tail reached by `jal` would be mislabeled, but no such case is known here |
+| `prologue` | first instruction `addiu sp,sp,-N` (N>0), second a `sw ...,off(sp)` | standard non-leaf prologue | medium: a function may contain such a sequence internally; leaf functions have no prologue |
+| `ghidra` | present in the Ghidra function list | Ghidra auto-analysis | medium: Ghidra both misses functions and can invent them; not necessary or sufficient |
+
+## Provenance and reproduction
+
+- Executable: the validated USA `SCUS_946.40;1` (SHA-1 `e173426c…`).
+- Ghidra list: `ghidra_read_function_list` from the ignored Phase 2 working program
+  `/P2-analysis-SCUS_946.40;1` — **1,721** defined functions. It is stored in the ignored path
+  `ghidra/function_starts.txt` (one hex address per line) and is not tracked.
+- Tool: tracked `tools/sf3_boundaries`, synthetic-tested (15 tests), standard library only.
+
+```bash
+./tools/sf3_boundaries scan \
+  --exe 'extracted/SCUS_946.40;1' \
+  --ghidra ghidra/function_starts.txt \
+  --out config/function_inventory.tsv
+# -> candidates=2875 grade_entry=1 grade_jal=2283 grade_prologue=1513 grade_ghidra=1721
+```
+
+## Agreement and disagreement
+
+| Measure | Count |
+|---|---|
+| Candidates (union of all grades) | 2,875 |
+| `entry` | 1 |
+| `jal` | 2,283 |
+| `prologue` | 1,513 |
+| `ghidra` | 1,721 |
+| Ghidra functions that are also `jal` targets | 1,708 |
+| **`jal` targets Ghidra did not define** | **575** |
+| `prologue` candidates Ghidra did not define | 691 |
+| Ghidra functions found by no scan grade | 6 |
+| Candidates whose only grade is `ghidra` | 6 |
+
+The important disagreement is the **575 called addresses Ghidra did not define**: every one is the
+target of a `jal`, so Ghidra's function set is incomplete for call targets. Conversely, only 6 Ghidra
+functions lack any scan grade, and prologue-only candidates are the least reliable (a repeated
+prologue pattern inside a larger function is a known false-positive source).
+
+## Library versus game code
+
+**Unresolved.** No grade distinguishes a PsyQ library routine from game code. The executable is a
+single linked image; there is no symbol table, no relocation table, and no section map that separates
+them. Any claim that a given address is library or game code requires evidence this phase does not
+have. This is recorded as a limit, not a conclusion.
+
+## Use
+
+The inventory is an input for choosing match targets (P6-T6): prefer `jal`-graded, unique, small
+bodies. It is **not** a match claim. A candidate becomes a match only after an instruction-identical
+`sf3_match range` comparison and a green clean full-binary `make gate`.
@@ -11,7 +11,7 @@
 - [x] **P6-T3 — `maspsx` integration and the ASPSX `la` verification** (complete)
 - [x] **P6-T4 — `-G` small-data threshold from byte evidence** (complete)
 - [x] **Rules check** — re-read `AGENTS.md` mandatory behavior after P6-T4 and stated the required continuation notice.
- [ ] P6-T5 — Evidence-graded function-boundary inventory
+- [x] **P6-T5 — Evidence-graded function-boundary inventory** (complete)
 - [ ] P6-T6 — First matching batch, with duplicate sharing
 - [ ] P6-T7 — Cookbook, conventions, verification record, and phase gate

@@ -163,3 +163,33 @@ the numeric `-G` is unresolved and unnecessary under this mechanism; the `la`-of
 is implemented but not yet exercised on a real function.

 **Rules check — re-read complete. Continuing with P6-T5.**
+
+## P6-T5 — Evidence-graded function-boundary inventory (2026-09-23)
+
+**Delivered:**
+
+- Tracked tool `tools/sf3_boundaries` (stdlib only, synthetic-tested) that scans the executable and
+  grades every candidate function start as `entry`, `jal`, `prologue`, and/or `ghidra`.
+- Tracked inventory `config/function_inventory.tsv` — 2,875 candidates, addresses and grades only
+  (no bytes, no disassembly).
+- Record `docs/PHASE6_BOUNDARIES.md` with the grade definitions, their limits, provenance, counts, and
+  the reproduction command.
+- The Ghidra list (1,721 functions) is kept in the ignored `ghidra/function_starts.txt`.
+
+**Key finding:** 575 `jal` targets are **not** in Ghidra's function set, so Ghidra alone is incomplete
+for call targets; the inventory is more complete for that grade. Prologue-only candidates are the
+least reliable (a repeated prologue inside a larger function is a false positive).
+
+**Library-versus-game-code is explicitly unresolved:** no grade separates PsyQ library code from game
+code in this single linked image.
+
+**Verification:**
+
+| Check | Result |
+|---|---|
+| `tools/sf3_boundaries scan` on the validated executable | 2,875 candidates; `jal` 2,283, `prologue` 1,513, `ghidra` 1,721 |
+| Synthetic suite | 86 tests pass (15 added for the boundary tool) |
+| ROM safety | inventory contains addresses and grade names only |
+
+**Limit:** no candidate is promoted to a match on the strength of the inventory alone; every match
+still needs `sf3_match range` and `make gate`.
@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""Evidence-graded function-boundary inventory for the USA executable.
+
+This replaces address-gap guessing with a deterministic scan that labels every
+candidate function start with the evidence that supports it:
+
+  entry     the PS-X EXE header's entry PC.
+  jal       the target of a `jal` instruction inside the payload. A direct call
+            target is a function entry; this is the strongest grade.
+  prologue  an address whose first instruction is `addiu sp,sp,-N` (N > 0) and
+            whose second is a `sw ...,off(sp)`. This is the standard non-leaf
+            prologue; it can be a false positive if such a sequence appears
+            inside another function.
+  ghidra    an address present in a Ghidra function list supplied by the caller
+            (`--ghidra FILE`, one hex address per line). Ghidra's auto-analysis
+            both misses functions and invents them, so this grade is neither
+            necessary nor sufficient.
+
+Output is a sorted TSV of `address<TAB>grades` with a short header. It contains
+addresses only -- never instruction bytes -- and the input executable is only
+read from a caller-supplied path. Nothing here promotes a candidate to a match:
+a match still requires an instruction-identical comparison and the full-binary
+gate.
+
+The library-versus-game-code question is **unresolved**; no grade distinguishes
+a PsyQ library routine from game code.
+
+Exit codes: 0 success, 2 usage or environment error.
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+import struct
+import sys
+from typing import Sequence
+
+
+EXE_MAGIC = b"PS-X EXE"
+HEADER_SIZE = 0x800
+PAYLOAD_LMA = 0x800
+
+GRADE_ORDER = ("entry", "jal", "prologue", "ghidra")
+
+
+class ToolError(Exception):
+    """A usage or environment problem; maps to exit code 2."""
+
+
+def parse_hex(text: str, label: str) -> int:
+    try:
+        return int(text, 16)
+    except ValueError as exc:
+        raise ToolError(f"{label}: not a hex address: {text!r}") from exc
+
+
+def require_file(path: Path, label: str) -> Path:
+    if not path.is_file():
+        raise ToolError(f"{label} is not a regular file: {path}")
+    return path
+
+
+def fresh_path(path: Path, label: str) -> Path:
+    if path.exists() or path.is_symlink():
+        raise ToolError(f"{label} already exists: {path}")
+    return path
+
+
+def parse_psx_exe(header: bytes) -> tuple[int, int, int]:
+    if len(header) < HEADER_SIZE:
+        raise ToolError("executable is smaller than a PS-X EXE header")
+    if header[:8] != EXE_MAGIC:
+        raise ToolError("executable does not carry the PS-X EXE magic")
+    entry, _gp, text_address, text_size = struct.unpack_from("<IIII", header, 0x10)
+    if text_size == 0:
+        raise ToolError("PS-X EXE header declares an empty payload")
+    return entry, text_address, text_size
+
+
+def load_ghidra(path: Path | None) -> set[int]:
+    if path is None:
+        return set()
+    addresses: set[int] = set()
+    for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
+        line = raw.split("#", 1)[0].strip()
+        if not line:
+            continue
+        addresses.add(parse_hex(line, f"ghidra line {number}"))
+    return addresses
+
+
+def scan(payload: bytes, text_address: int, entry: int, ghidra: set[int]) -> dict[int, set[str]]:
+    """Return address -> evidence grades for every candidate start."""
+    grades: dict[int, set[str]] = {}
+    end = text_address + len(payload)
+
+    def add(address: int, grade: str) -> None:
+        if text_address <= address < end and address % 4 == 0:
+            grades.setdefault(address, set()).add(grade)
+
+    add(entry, "entry")
+
+    for address in ghidra:
+        add(address, "ghidra")
+
+    words = len(payload) // 4
+    for index in range(words):
+        word = struct.unpack_from("<I", payload, index * 4)[0]
+        pc = text_address + index * 4
+        opcode = word >> 26
+        if opcode == 0x03:  # jal
+            target = ((pc + 4) & 0xF0000000) | ((word & 0x03FFFFFF) << 2)
+            add(target, "jal")
+        elif opcode == 0x09:  # addiu sp,sp,-N  followed by  sw ...,off(sp)
+            rs = (word >> 21) & 0x1F
+            rt = (word >> 16) & 0x1F
+            immediate = word & 0xFFFF
+            signed = immediate - 0x10000 if immediate & 0x8000 else immediate
+            if rs == 29 and rt == 29 and signed < 0 and index + 1 < words:
+                following = struct.unpack_from("<I", payload, (index + 1) * 4)[0]
+                if (following >> 26) == 0x2B and ((following >> 21) & 0x1F) == 29:
+                    add(pc, "prologue")
+
+    return grades
+
+
+def format_inventory(grades: dict[int, set[str]]) -> str:
+    lines = [
+        "# Syphon Filter 3 (USA) function-boundary inventory.",
+        "# Columns: address<TAB>comma-separated evidence grades.",
+        "# Grades: entry, jal, prologue, ghidra. Addresses only; no bytes.",
+        "# Library-versus-game-code is unresolved; no grade answers it.",
+    ]
+    for address in sorted(grades):
+        names = [grade for grade in GRADE_ORDER if grade in grades[address]]
+        lines.append(f"0x{address:08X}\t{','.join(names)}")
+    return "\n".join(lines) + "\n"
+
+
+def command_scan(args: argparse.Namespace) -> int:
+    exe_path = require_file(args.exe, "executable")
+    out = fresh_path(args.out, "output")
+    data = exe_path.read_bytes()
+    entry, text_address, text_size = parse_psx_exe(data)
+    payload = data[PAYLOAD_LMA:PAYLOAD_LMA + text_size]
+    ghidra = load_ghidra(args.ghidra)
+    grades = scan(payload, text_address, entry, ghidra)
+
+    counts = {grade: 0 for grade in GRADE_ORDER}
+    for names in grades.values():
+        for grade in names:
+            counts[grade] += 1
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(format_inventory(grades), encoding="ascii")
+
+    print(f"candidates={len(grades)}")
+    for grade in GRADE_ORDER:
+        print(f"grade_{grade}={counts[grade]}")
+    print(f"output={out}")
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+    scan_parser = subparsers.add_parser("scan", help="scan an executable")
+    scan_parser.add_argument("--exe", required=True, type=Path)
+    scan_parser.add_argument("--ghidra", type=Path, default=None,
+                             help="Ghidra function list, one hex address per line")
+    scan_parser.add_argument("--out", required=True, type=Path)
+    scan_parser.set_defaults(handler=command_scan)
+    return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    try:
+        return args.handler(args)
+    except ToolError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+    except OSError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -0,0 +1,166 @@
+"""Synthetic-only tests for the function-boundary inventory tool.
+
+Fixtures are self-authored bytes in temporary directories. They never read the
+local disc or the extracted game executable.
+"""
+
+from __future__ import annotations
+
+import importlib.machinery
+import importlib.util
+from pathlib import Path
+import struct
+import sys
+import tempfile
+import unittest
+
+
+TOOL_PATH = Path(__file__).resolve().parents[1] / "sf3_boundaries"
+
+
+def _load_tool() -> object:
+    loader = importlib.machinery.SourceFileLoader("sf3_boundaries_under_test", str(TOOL_PATH))
+    spec = importlib.util.spec_from_loader(loader.name, loader)
+    if spec is None:
+        raise RuntimeError("could not create an import specification")
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    loader.exec_module(module)
+    return module
+
+
+sf3_boundaries = _load_tool()
+
+PAYLOAD = 0x80010000
+
+
+def _r(opcode: int, rs: int, rt: int, immediate: int) -> int:
+    return (opcode << 26) | (rs << 21) | (rt << 16) | (immediate & 0xFFFF)
+
+
+def _jal(target: int) -> int:
+    return (0x03 << 26) | ((target >> 2) & 0x03FFFFFF)
+
+
+def _synthetic_exe(payload: bytes, entry: int) -> bytes:
+    header = bytearray(0x800)
+    header[:8] = b"PS-X EXE"
+    struct.pack_into("<IIII", header, 0x10, entry, 0, PAYLOAD, len(payload))
+    return bytes(header) + payload
+
+
+def _payload() -> bytes:
+    words = [0] * 16
+    words[0] = _jal(PAYLOAD + 0x20)                    # 0x80010000 calls 0x80010020
+    words[4] = 0x03E00008                             # 0x80010010: jr ra
+    words[8] = _r(0x09, 29, 29, 0xFFF0)               # 0x80010020: addiu sp,sp,-16
+    words[9] = _r(0x2B, 29, 31, 12)                   #             sw ra,12(sp)
+    return b"".join(struct.pack("<I", word) for word in words)
+
+
+class ParseTests(unittest.TestCase):
+    def test_rejects_missing_magic(self) -> None:
+        with self.assertRaises(sf3_boundaries.ToolError):
+            sf3_boundaries.parse_psx_exe(bytes(0x800))
+
+    def test_rejects_empty_payload(self) -> None:
+        header = bytearray(0x800)
+        header[:8] = b"PS-X EXE"
+        with self.assertRaises(sf3_boundaries.ToolError):
+            sf3_boundaries.parse_psx_exe(bytes(header))
+
+
+class GhidraTests(unittest.TestCase):
+    def test_parses_comments_and_hex(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "ghidra.txt"
+            path.write_text("# comment\n80010010\n\n0x80010020\n", encoding="ascii")
+            self.assertEqual(sf3_boundaries.load_ghidra(path),
+                             {0x80010010, 0x80010020})
+
+    def test_rejects_a_bad_address(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            path = Path(tmp) / "ghidra.txt"
+            path.write_text("nothex\n", encoding="ascii")
+            with self.assertRaises(sf3_boundaries.ToolError):
+                sf3_boundaries.load_ghidra(path)
+
+
+class ScanTests(unittest.TestCase):
+    def setUp(self) -> None:
+        self.payload = _payload()
+        self.grades = sf3_boundaries.scan(
+            self.payload, PAYLOAD, PAYLOAD, {PAYLOAD + 0x10}
+        )
+
+    def test_entry_is_graded(self) -> None:
+        self.assertIn("entry", self.grades[PAYLOAD])
+
+    def test_jal_target_is_graded(self) -> None:
+        self.assertIn("jal", self.grades[PAYLOAD + 0x20])
+
+    def test_prologue_is_graded(self) -> None:
+        self.assertIn("prologue", self.grades[PAYLOAD + 0x20])
+
+    def test_ghidra_only_function_is_graded(self) -> None:
+        self.assertEqual(self.grades[PAYLOAD + 0x10], {"ghidra"})
+
+    def test_an_address_with_no_evidence_is_absent(self) -> None:
+        self.assertNotIn(PAYLOAD + 0x04, self.grades)
+
+    def test_a_jal_outside_the_payload_is_not_graded(self) -> None:
+        payload = bytearray(self.payload)
+        struct.pack_into("<I", payload, 0, _jal(0x80020000))
+        grades = sf3_boundaries.scan(bytes(payload), PAYLOAD, PAYLOAD, set())
+        self.assertNotIn(0x80020000, grades)
+
+    def test_a_ghidra_address_outside_the_payload_is_not_graded(self) -> None:
+        grades = sf3_boundaries.scan(self.payload, PAYLOAD, PAYLOAD, {0x80020000})
+        self.assertNotIn(0x80020000, grades)
+
+
+class FormatTests(unittest.TestCase):
+    def test_sorted_with_grades_in_fixed_order(self) -> None:
+        text = sf3_boundaries.format_inventory(
+            {PAYLOAD + 8: {"ghidra", "jal"}, PAYLOAD: {"entry"}}
+        )
+        body = [line for line in text.splitlines() if not line.startswith("#")]
+        self.assertEqual(body, [
+            f"0x{PAYLOAD:08X}\tentry",
+            f"0x{PAYLOAD + 8:08X}\tjal,ghidra",
+        ])
+
+    def test_contains_no_bytes(self) -> None:
+        text = sf3_boundaries.format_inventory({PAYLOAD: {"entry"}})
+        self.assertNotIn("80010000:", text)
+
+
+class MainTests(unittest.TestCase):
+    def test_scan_writes_the_inventory(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            exe = root / "synthetic.exe"
+            exe.write_bytes(_synthetic_exe(_payload(), PAYLOAD))
+            ghidra = root / "ghidra.txt"
+            ghidra.write_text("80010010\n", encoding="ascii")
+            out = root / "inventory.tsv"
+            rc = sf3_boundaries.main(["scan", "--exe", str(exe),
+                                      "--ghidra", str(ghidra), "--out", str(out)])
+            self.assertEqual(rc, 0)
+            text = out.read_text(encoding="ascii")
+            self.assertIn(f"0x{PAYLOAD:08X}\tentry", text)
+            self.assertIn(f"0x{PAYLOAD + 0x20:08X}\tjal,prologue", text)
+
+    def test_refuses_an_existing_output(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            root = Path(tmp)
+            exe = root / "synthetic.exe"
+            exe.write_bytes(_synthetic_exe(_payload(), PAYLOAD))
+            out = root / "inventory.tsv"
+            out.write_text("", encoding="ascii")
+            rc = sf3_boundaries.main(["scan", "--exe", str(exe), "--out", str(out)])
+            self.assertEqual(rc, 2)
+
+
+if __name__ == "__main__":
+    unittest.main()