phase7: derive evidence-graded function extents from control flow
Phase 6 graded function starts and left every end to be derived by hand. This adds tools/sf3_extents, which explores all reachable control flow from each hard start (jal/entry) and reports an extent plus how far it can be trusted. Measured decisions, not stylistic ones: - Soft starts are not walk boundaries. A body's second instruction can satisfy the prologue grade exactly (0x800152AC is lw v1,8(gp) / addiu sp,sp,-176, so 0x800152B0 looks like a start). Enforcing soft boundaries stopped 155 of 416 walks inside a real body. - The walk is a full reachability computation, not a first-terminal search: a function whose paths return at different addresses must report the whole body. Grades: exact 1940 (1666 packed, gap=0), fallthrough 256, indirect 73, escape 15, contained 153, standalone 438; 63.8% of the payload covered. Verification: all 12 registered regions reproduce exactly (make extents-verify, now part of make check), 29 new synthetic tests (115 total), byte-identical across two runs, and Ghidra's independent body for FUN_80017ad4 agrees. Two defects were caught by writing the tests first and are recorded: reach had to be an exclusive end, and a terminal j's delay slot must not continue linearly. The table holds addresses, sizes, grades and site addresses only -- no bytes.
This commit is contained in:
Christopher Williams
@@ -25,11 +25,14 @@ INPUT_REPORT := .run/p3-exe-info.tsv
|
||||
# reproduces the all-payload data baseline exactly.
|
||||
REGIONS := config/regions.tsv
|
||||
SYMBOLS := config/symbols.tsv
|
||||
INVENTORY := config/function_inventory.tsv
|
||||
EXTENTS := config/function_extents.tsv
|
||||
MATCH := tools/sf3_match
|
||||
EXTENTS_TOOL := tools/sf3_extents
|
||||
CODE_OUT := build/code
|
||||
EXPECTED_SHA1 := e173426c157384ebf1b6caf8c6fea18a85a14af9
|
||||
|
||||
.PHONY: all validate split assemble link binary code gate test check clean
|
||||
.PHONY: all validate split assemble link binary code gate extents extents-verify test check clean
|
||||
|
||||
all: binary
|
||||
|
||||
@@ -84,12 +87,27 @@ gate: validate
|
||||
@"$(MATCH)" gate --exe "$(EXE)" --regions "$(REGIONS)" --symbols "$(SYMBOLS)" --out "$(CODE_OUT)" \
|
||||
--expect-sha1 "$(EXPECTED_SHA1)"
|
||||
|
||||
# Function extents (Phase 7). `extents` regenerates the tracked table from the
|
||||
# inventory; `extents-verify` re-derives it and requires every registered region
|
||||
# to agree, so a region can never silently disagree with the derived extent.
|
||||
extents: validate
|
||||
@test -f "$(EXTENTS_TOOL)"
|
||||
@test -f "$(INVENTORY)"
|
||||
@"$(EXTENTS_TOOL)" scan --exe "$(EXE)" --inventory "$(INVENTORY)" --out "$(EXTENTS)" --force
|
||||
|
||||
extents-verify: validate
|
||||
@test -f "$(EXTENTS_TOOL)"
|
||||
@test -f "$(INVENTORY)"
|
||||
@test -f "$(EXTENTS)"
|
||||
@"$(EXTENTS_TOOL)" verify --exe "$(EXE)" --inventory "$(INVENTORY)" \
|
||||
--extents "$(EXTENTS)" --regions "$(REGIONS)"
|
||||
|
||||
# Verification gates. `test` is synthetic-only and needs no game input;
|
||||
# `check` adds the full-binary byte gate (which does).
|
||||
# `check` adds the extents check and the full-binary byte gate (which do).
|
||||
test:
|
||||
@PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests
|
||||
|
||||
check: test gate
|
||||
check: test extents-verify gate
|
||||
|
||||
clean:
|
||||
@if test -e "$(BUILD)"; then \
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,135 @@
|
||||
# Phase 7 — Evidence-Graded Function Extents
|
||||
|
||||
**Scope:** P7-T2 — turn evidence-graded function **starts** into evidence-graded function
|
||||
`[start, end)` **extents**, so a match target no longer needs its end derived by hand.
|
||||
**Status:** complete. Tracked output: `config/function_extents.tsv`. Tool: `tools/sf3_extents`
|
||||
(29 synthetic tests).
|
||||
|
||||
## The problem this closes
|
||||
|
||||
Phase 6 produced `config/function_inventory.tsv`: 2,875 candidate function **starts**, each labelled
|
||||
with the evidence that supports it. It says nothing about where a function ends. Every end in
|
||||
`config/regions.tsv` was therefore derived by hand, and a wrong end is only ever discovered as a
|
||||
`LENGTH-MISMATCH` after the fact. That manual boundary step was the dominant per-function cost.
|
||||
|
||||
## The model
|
||||
|
||||
A **hard start** is a candidate graded `jal` or `entry` — a direct call target or the header entry.
|
||||
A **soft start** is a `prologue`/`ghidra`-only candidate.
|
||||
|
||||
Soft starts are **not** used as walk boundaries, and this is a measured decision, not a stylistic one.
|
||||
A body's second instruction can itself look exactly like a prologue: at `0x800152AC` the real body is
|
||||
`lw v1,8(gp)` / `addiu sp,sp,-176` / `sw ra,172(sp)`, and `0x800152B0` — the `addiu sp,sp,-176` —
|
||||
satisfies the `prologue` grade exactly. Treating soft starts as boundaries made 155 of 416 walks stop
|
||||
inside a real body. Soft starts are therefore recorded as candidates and classified, but never
|
||||
enforced.
|
||||
|
||||
From each hard start the tool explores **all** reachable paths (fall-through, branch and jump targets,
|
||||
delay slots, calls returning to their return address) and records the highest reachable instruction.
|
||||
It is a full reachability computation rather than a first-terminal search, because a function whose
|
||||
paths return at different addresses must report the whole body; a first-terminal walk under-reports it.
|
||||
|
||||
The walk stops at:
|
||||
|
||||
| Stop | Meaning |
|
||||
|---|---|
|
||||
| `jr ra` | return; the delay slot is part of the body |
|
||||
| `jr $rs` (rs ≠ ra) | register jump — a switch or jump table; targets cannot be followed statically |
|
||||
| `syscall` / `break` | no delay slot; the body ends here |
|
||||
| a tail `j` | target behind the start, or a known start: control leaves this function |
|
||||
| another hard start | a boundary the walk may not cross |
|
||||
|
||||
The extent end is the first byte after the last reachable instruction: for a call, the return address;
|
||||
for a return, the instruction after the delay slot.
|
||||
|
||||
## Grades
|
||||
|
||||
| Grade | Meaning | Basis | Limit |
|
||||
|---|---|---|---|
|
||||
| `exact` | The walk terminated at a return, `syscall`, `break` or tail jump and stayed inside its own region. `end` is the function's own last byte + 1. | 1,940 candidates; **all 12 registered regions reproduce exactly**; Ghidra's independent body for `FUN_80017ad4` is `[80017ad4, 80017ae7]`, i.e. exclusive end `0x80017AE8`, equal to the derived end. | It is the reachable extent of a called entry, not proof that the entry is a game function: a shared tail block reached by `jal` would also be graded `exact`. |
|
||||
| `fallthrough` | No terminal was reachable; the walk ran into the next hard start and the extent ends there. | 256 candidates. Typically a function that ends in a call to something that does not return. | The extent may include alignment padding between the body's last instruction and the next entry. |
|
||||
| `indirect` | The walk hit `jr $rs`, so its result is a **lower bound**: switch case bodies are not statically reachable. The recorded end is the next hard start. | 73 candidates; 44 of 45 measured earlier were within 4 bytes of the next start. | The recorded end is the best available bound and is **not verified**. |
|
||||
| `escape` | The walk crossed the next hard start, so control left this function's region. The extent is clipped to the next hard start and the crossing instruction is recorded. | 15 candidates; see the dossier below. | The cause is not determined per case (shared block, interleaved body, or bytes decoded as code). |
|
||||
| `outside` / `runaway` | The walk left the declared payload, or exceeded the 128 KB size cap. | 0 candidates. | — |
|
||||
| `contained` | A **soft** candidate lying inside an `exact` extent. | 153 candidates. Strong evidence the candidate is **not** a function start. | Only `exact` extents carry the inference; a candidate inside a `fallthrough`/`indirect`/`escape` extent is reported as `standalone` instead. |
|
||||
| `standalone` | A soft candidate covered by no `exact` extent. | 438 candidates. | Unresolved: it may be a function reached only indirectly, or a false positive in data. No evidence class separates them yet. |
|
||||
|
||||
`contained` and `standalone` candidates carry no extent: their `end`, `size`, `next` and `gap` fields
|
||||
are `-`.
|
||||
|
||||
## Results
|
||||
|
||||
| Measure | Count |
|
||||
|---|---|
|
||||
| Candidates | 2,875 |
|
||||
| Hard starts (`jal` + `entry`) | 2,284 |
|
||||
| `exact` | 1,940 |
|
||||
| of which packed against the next start (`gap=0`) | 1,666 |
|
||||
| `fallthrough` | 256 |
|
||||
| `indirect` | 73 |
|
||||
| `escape` | 15 |
|
||||
| `contained` (soft, provably not a start) | 153 |
|
||||
| `standalone` (soft, unresolved) | 438 |
|
||||
| Bytes covered by extents | 1,202,764 of 1,884,160 (63.8%) |
|
||||
|
||||
The remaining payload is data, padding, and the unresolved `standalone` candidates.
|
||||
|
||||
### The escape dossier
|
||||
|
||||
Fifteen hard starts have control flow that crosses the next hard start. In the nine cases measured in
|
||||
detail, the intervening start is a **self-contained function whose own walk ends at exactly the same
|
||||
`jr ra`** as the outer walk — so the outer walk escaped into another function's body rather than
|
||||
wrapping around it. The recorded crossing instructions are ordinary forward conditional branches
|
||||
(`escape=branch@0x80013510->0x800135D0`) and one `j` (`escape=j@0x8001347C->0x800135A4`). One case is
|
||||
sharper still: `0x80013524` is itself graded `escape` with a **backward** crossing
|
||||
(`j@0x8001347C->0x800135A4`), i.e. its body jumps back before its own start — the signature of a shared
|
||||
block that is called as if it were a function.
|
||||
|
||||
Extents for these addresses are clipped to the next hard start, which is the conservative and
|
||||
defensible bound; the underlying cause is recorded, not guessed.
|
||||
|
||||
## Verification
|
||||
|
||||
| Check | Command | Result |
|
||||
|---|---|---|
|
||||
| Registered regions agree | `make extents-verify` | `regions=12 disagreements=0`, `result=AGREE` |
|
||||
| Synthetic suite | `python3 -m unittest discover -s tools/tests` | 115 tests pass (29 added this task) |
|
||||
| Determinism | two `scan` runs, `cmp` | byte-identical |
|
||||
| Independent oracle | `ghidra_analyze_function FUN_80017ad4` | body `[80017ad4, 80017ae7]` → end `0x80017AE8` = derived end |
|
||||
| Firewall | table columns | addresses, sizes, grades and site addresses only — no instruction bytes |
|
||||
|
||||
`make extents-verify` is part of `make check`, so the registered regions can never silently disagree
|
||||
with the derived extents again.
|
||||
|
||||
Regenerate the table:
|
||||
|
||||
```bash
|
||||
make extents
|
||||
# or, explicitly:
|
||||
./tools/sf3_extents scan --exe 'extracted/SCUS_946.40;1' \
|
||||
--inventory config/function_inventory.tsv --out config/function_extents.tsv --force
|
||||
```
|
||||
|
||||
## What this does not establish
|
||||
|
||||
- **Library versus game code** remains unresolved. No grade distinguishes a PsyQ library routine from
|
||||
game code; `exact` means "the reachable extent of a called entry", which is equally true of both.
|
||||
- An `exact` extent is not a *function identity* claim. Two entries that share a tail are two entries.
|
||||
- `standalone` is not a verdict. 438 candidates are simply not covered by any `exact` extent.
|
||||
- A `gap` is padding, not proof of a missing candidate; and `gap=0` is packing, not proof that the
|
||||
neighbour is a separate function.
|
||||
- Nothing here promotes a candidate to a match. A match still requires an instruction-identical
|
||||
`sf3_match range` comparison and the clean full-binary gate.
|
||||
|
||||
## Tooling notes
|
||||
|
||||
Two defects were found and fixed while building this, both by writing the synthetic tests first:
|
||||
|
||||
1. **Delay-slot accounting.** `reach` must be an *exclusive* end. Treating it as an address made every
|
||||
extent 4 bytes short — which the registered-region check caught immediately (12/12 disagreements).
|
||||
2. **A tail jump's delay slot must not continue linearly.** After a terminal `j`, enqueuing the delay
|
||||
slot let the walk run past it into the next function, turning a clean tail call into a false
|
||||
`escape`.
|
||||
|
||||
Both are recorded because the same mistakes will otherwise be repeated when the walk is extended to
|
||||
handle jump tables.
|
||||
@@ -10,7 +10,7 @@ function bodies, bringing the project past **thirty** distinct byte-identical fu
|
||||
## Progress
|
||||
|
||||
- [x] **P7-T1 — Phase control records, baseline revalidation, and open-item triage** (complete)
|
||||
- [ ] **P7-T2 — Evidence-graded function extents**
|
||||
- [x] **P7-T2 — Evidence-graded function extents** (complete)
|
||||
- [ ] **P7-T3 — Duplicate-body census**
|
||||
- [ ] **P7-T4 — Candidate triage worklist**
|
||||
- [ ] **Rules check**
|
||||
@@ -67,6 +67,54 @@ this is a scope statement for P7-T2..T7.
|
||||
| CRT entry `[0x800FB368,0x800FB410)` | `docs/PHASE5_FIRST_MATCH.md`; Phase 4 entry dossier | CRT startup, not compiler output. Stays in the data fallback; excluded from the worklist. |
|
||||
| Library versus game code | `docs/PHASE6_BOUNDARIES.md` §Library versus game code | **Unresolved.** Does not block a byte-level match (a library routine that is byte-identical is as matched as any other), but it does affect what "finished" means. Re-flagged for P7-T2/T3/T4; if those produce evidence, it is recorded, not acted on silently. |
|
||||
|
||||
## P7-T2 — Evidence-graded function extents (2026-09-23)
|
||||
|
||||
**Delivered:**
|
||||
|
||||
- `tools/sf3_extents` — derives an `[start, end)` extent for every candidate start by exploring all
|
||||
reachable control flow from each **hard** start (`jal`/`entry`), with `scan` and `verify`
|
||||
subcommands. Standard library only; 29 synthetic tests.
|
||||
- `config/function_extents.tsv` — tracked table, `address<TAB>end<TAB>size<TAB>next<TAB>gap<TAB>grade<TAB>evidence`.
|
||||
Addresses, sizes, grades and site addresses only — no instruction bytes.
|
||||
- `make extents` (regenerate) and `make extents-verify` (re-derive and require agreement with
|
||||
`config/regions.tsv`); `extents-verify` is now part of `make check`.
|
||||
- `docs/PHASE7_EXTENTS.md` — the model, the grade table with basis and limits, the results census,
|
||||
and the escape dossier.
|
||||
|
||||
**Results:** 2,875 candidates → `exact` 1,940 (1,666 with `gap=0`), `fallthrough` 256, `indirect` 73,
|
||||
`escape` 15, `contained` 153, `standalone` 438. Extents cover 1,202,764 of 1,884,160 payload bytes
|
||||
(63.8%).
|
||||
|
||||
**Key measured decisions:**
|
||||
|
||||
- **Soft starts are not walk boundaries.** A body's second instruction can satisfy the `prologue`
|
||||
grade exactly (`0x800152AC` is `lw v1,8(gp)` / `addiu sp,sp,-176` / `sw ra,172(sp)`, so `0x800152B0`
|
||||
looks like a start). Enforcing soft boundaries stopped 155 of 416 walks inside a real body.
|
||||
- **The walk is a full reachability computation, not a first-terminal search.** A function whose paths
|
||||
return at different addresses must report the whole body.
|
||||
- **`contained` is an inference, not a label.** Only `exact` extents are trusted to declare a soft
|
||||
candidate a non-function.
|
||||
|
||||
**Verification:**
|
||||
|
||||
| Check | Command | Result |
|
||||
|---|---|---|
|
||||
| Registered regions agree | `make extents-verify` | `regions=12 disagreements=0`, `result=AGREE` |
|
||||
| Synthetic suite | `python3 -m unittest discover -s tools/tests` | **115 tests pass** (29 added) |
|
||||
| Determinism | two `scan` runs, `cmp` | byte-identical |
|
||||
| Independent oracle | `ghidra_analyze_function FUN_80017ad4` | Ghidra body `[80017ad4, 80017ae7]` → exclusive end `0x80017AE8`, equal to the derived end |
|
||||
| Integrated gate | `make check` | tests + `AGREE` + `c_regions=12`, 0 differing bytes, SHA-1 `e173426c…` |
|
||||
|
||||
**Two defects were found and fixed while building this**, both caught by writing the synthetic tests
|
||||
first: `reach` had to be an exclusive end (otherwise every extent is 4 bytes short — the
|
||||
registered-region check caught all 12), and a terminal `j`'s delay slot must not continue linearly
|
||||
(otherwise a clean tail call becomes a false `escape`). Both are recorded in the doc.
|
||||
|
||||
**Limits:** library-versus-game-code stays unresolved (an `exact` extent means "the reachable extent
|
||||
of a called entry", equally true of library and game code); `standalone` (438) is not a verdict; the
|
||||
15 `escape` cases are clipped conservatively with the crossing instruction recorded but the cause not
|
||||
determined per case; and nothing here promotes a candidate to a match.
|
||||
|
||||
## Notes and limits
|
||||
|
||||
- The four class items are the plan's P7-T2..T5 work; the single-function items are explicitly out of
|
||||
|
||||
Executable
+615
@@ -0,0 +1,615 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evidence-graded function **extents** for the USA executable.
|
||||
|
||||
Phase 6 produced evidence-graded function *starts* (`config/function_inventory.tsv`)
|
||||
and no evidence at all for where a function *ends*: every region end in
|
||||
`config/regions.tsv` was derived by hand. This tool derives the end from the
|
||||
payload's own control flow and states how much the result can be trusted.
|
||||
|
||||
## The model
|
||||
|
||||
A **hard start** is a candidate whose evidence is a direct call or the header
|
||||
entry (`jal` or `entry` grade). A **soft start** is a `prologue`/`ghidra`-only
|
||||
candidate. Phase 7 measurement showed soft starts are frequently *false
|
||||
positives inside a real function body* -- e.g. a body's second instruction can
|
||||
itself look like a prologue -- so soft starts are **not** used as walk
|
||||
boundaries, and the walk is never stopped by them.
|
||||
|
||||
From each hard start the tool walks control flow (linear fall-through, branch
|
||||
and jump targets, delay slots, calls falling through to the return address) and
|
||||
records the highest reachable instruction. The walk stops at:
|
||||
|
||||
* a return (`jr ra`) -- the delay slot is included;
|
||||
* a register jump (`jr $rs`, rs != ra), i.e. a switch/jump table, whose
|
||||
targets cannot be followed statically;
|
||||
* `syscall` / `break`;
|
||||
* a tail `j` whose target is outside this function (behind its start, or a
|
||||
known start);
|
||||
* another **hard** start, which is a boundary the walk may not cross.
|
||||
|
||||
The extent end is the first byte after the last reachable instruction. For a
|
||||
call this is the return address; for a return, the instruction after the delay
|
||||
slot.
|
||||
|
||||
## Grades
|
||||
|
||||
exact the walk terminated at a return / `syscall` / `break` / tail
|
||||
jump and stayed inside its own region. This is the grade a match
|
||||
candidate wants: the end is the function's own last byte + 1.
|
||||
All 12 regions registered in Phase 6 reproduce at this grade.
|
||||
fallthrough no terminal was reachable; the walk ran into the next hard
|
||||
start. The extent ends there. Typically a function that ends in
|
||||
a call to something that does not return.
|
||||
indirect the walk hit a register jump (`jr $rs`), so its result is a
|
||||
**lower bound**: the switch's case bodies are not reachable
|
||||
statically. The recorded end is the next hard start, which is
|
||||
the best available bound and is *not* verified.
|
||||
escape the walk crossed the next hard start, so control left this
|
||||
function's region (a shared tail block, an interleaved body, or
|
||||
bytes decoded as code). The extent is clipped to the next hard
|
||||
start and the crossing instruction is recorded as evidence.
|
||||
outside the walk left the declared payload.
|
||||
runaway the walk exceeded the size cap without terminating.
|
||||
contained a **soft** candidate lying inside an `exact` extent. Strong
|
||||
evidence that it is not an independent function.
|
||||
standalone a soft candidate not covered by any `exact` extent. Unresolved:
|
||||
it may be a function reached only indirectly, or a false
|
||||
positive in data.
|
||||
|
||||
`contained` and `standalone` candidates carry no extent: their `end`, `size`,
|
||||
`next` and `gap` fields are `-`.
|
||||
|
||||
## Output
|
||||
|
||||
A sorted TSV of `address<TAB>end<TAB>size<TAB>next<TAB>gap<TAB>grade<TAB>evidence`:
|
||||
|
||||
* `next` is the next hard start (the next row that has an extent);
|
||||
* `gap` is `next - end` -- 0 means the function is packed against the next
|
||||
one, which is what a registerable region needs; a positive gap is padding
|
||||
(alignment) that must stay out of the region;
|
||||
* `evidence` is a `;`-separated list of `key=value` tokens.
|
||||
|
||||
It contains **addresses, sizes and grade names only -- never instruction bytes**.
|
||||
The executable is read only from the caller-supplied path.
|
||||
|
||||
Nothing here promotes a candidate to a match: a match still requires an
|
||||
instruction-identical `sf3_match range` comparison and the clean full-binary
|
||||
gate.
|
||||
|
||||
Exit codes: 0 success, 1 verification disagreement, 2 usage or environment error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import bisect
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
from typing import Sequence
|
||||
|
||||
|
||||
EXE_MAGIC = b"PS-X EXE"
|
||||
HEADER_SIZE = 0x800
|
||||
PAYLOAD_LMA = 0x800
|
||||
MAX_FUNCTION_SIZE = 0x20000
|
||||
|
||||
HARD_GRADES = frozenset({"jal", "entry"})
|
||||
|
||||
# Grade order for reporting.
|
||||
GRADES = (
|
||||
"exact",
|
||||
"fallthrough",
|
||||
"indirect",
|
||||
"escape",
|
||||
"outside",
|
||||
"runaway",
|
||||
"contained",
|
||||
"standalone",
|
||||
)
|
||||
|
||||
HEADER_LINES = (
|
||||
"# Syphon Filter 3 (USA) function extents.",
|
||||
"# Columns: address<TAB>end<TAB>size<TAB>next<TAB>gap<TAB>grade<TAB>evidence.",
|
||||
"# Grades: exact, fallthrough, indirect, escape, outside, runaway,",
|
||||
"# contained, standalone. Addresses and sizes only; no bytes.",
|
||||
"# A dash means the field does not apply (a non-function candidate).",
|
||||
"# 'exact' is the only grade a match candidate should rely on;",
|
||||
"# 'contained' is evidence a soft candidate is NOT a function start.",
|
||||
"# Regenerate: ./tools/sf3_extents scan --exe '<exe>' \\",
|
||||
"# --inventory config/function_inventory.tsv --out config/function_extents.tsv --force",
|
||||
)
|
||||
|
||||
|
||||
class ToolError(Exception):
|
||||
"""A usage or environment problem; maps to exit code 2."""
|
||||
|
||||
|
||||
class Disagreement(Exception):
|
||||
"""A verification failure; maps to exit code 1."""
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# parsing helpers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_hex(text: str, label: str) -> int:
|
||||
try:
|
||||
return int(text, 16)
|
||||
except ValueError as exc:
|
||||
raise ToolError(f"{label}: not a hex address: {text!r}") from exc
|
||||
|
||||
|
||||
def require_file(path: Path, label: str) -> Path:
|
||||
if not path.is_file():
|
||||
raise ToolError(f"{label} is not a regular file: {path}")
|
||||
return path
|
||||
|
||||
|
||||
def resolve_output(path: Path, force: bool) -> Path:
|
||||
if path.exists() or path.is_symlink():
|
||||
if not force:
|
||||
raise ToolError(f"output already exists (use --force to overwrite): {path}")
|
||||
if not path.is_file() or path.is_symlink():
|
||||
raise ToolError(f"output is not a regular file: {path}")
|
||||
return path
|
||||
|
||||
|
||||
def parse_psx_exe(header: bytes) -> tuple[int, int, int]:
|
||||
if len(header) < HEADER_SIZE:
|
||||
raise ToolError("executable is smaller than a PS-X EXE header")
|
||||
if header[:8] != EXE_MAGIC:
|
||||
raise ToolError("executable does not carry the PS-X EXE magic")
|
||||
entry, _gp, text_address, text_size = struct.unpack_from("<IIII", header, 0x10)
|
||||
if text_size == 0:
|
||||
raise ToolError("PS-X EXE header declares an empty payload")
|
||||
return entry, text_address, text_size
|
||||
|
||||
|
||||
def load_inventory(path: Path) -> dict[int, set[str]]:
|
||||
"""Read `address<TAB>grades` rows, skipping comments and blank lines."""
|
||||
grades: dict[int, set[str]] = {}
|
||||
for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
continue
|
||||
fields = line.split("\t")
|
||||
if len(fields) != 2:
|
||||
raise ToolError(f"inventory line {number}: expected 'address<TAB>grades'")
|
||||
address = parse_hex(fields[0].strip(), f"inventory line {number}")
|
||||
names = {name.strip() for name in fields[1].split(",") if name.strip()}
|
||||
if not names:
|
||||
raise ToolError(f"inventory line {number}: no grades")
|
||||
if address in grades:
|
||||
raise ToolError(f"inventory line {number}: duplicate address {fields[0]!r}")
|
||||
grades[address] = names
|
||||
if not grades:
|
||||
raise ToolError("inventory is empty")
|
||||
return grades
|
||||
|
||||
|
||||
def load_regions(path: Path) -> list[tuple[int, int, str]]:
|
||||
"""Read `start<TAB>end<TAB>source[<TAB>overrides]` rows."""
|
||||
regions: list[tuple[int, int, str]] = []
|
||||
for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
continue
|
||||
fields = line.split("\t")
|
||||
if len(fields) < 3:
|
||||
raise ToolError(f"region line {number}: expected at least three fields")
|
||||
start = parse_hex(fields[0].strip(), f"region line {number}")
|
||||
end = parse_hex(fields[1].strip(), f"region line {number}")
|
||||
if end <= start:
|
||||
raise ToolError(f"region line {number}: end is not after start")
|
||||
regions.append((start, end, fields[2].strip()))
|
||||
return regions
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# instruction decoding and the control-flow walk
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Walk:
|
||||
"""The result of walking one candidate start.
|
||||
|
||||
`reach` is the highest reachable instruction address + 4: the whole
|
||||
reachable body is explored, not just the first path that terminates, so a
|
||||
function whose paths return at different addresses still reports its true
|
||||
extent. `boundary` is the lowest **hard** start the walk ran into, which is
|
||||
a boundary it may not cross. `escape_at` records the first branch or jump
|
||||
whose target left the region, which is what distinguishes a function that
|
||||
*falls through* into its neighbour from one that *jumps over* it.
|
||||
"""
|
||||
|
||||
__slots__ = ("reach", "boundary", "indirect_at", "terminals", "escape_at", "overflow")
|
||||
|
||||
def __init__(self, reach: int, boundary: int | None, indirect_at: int | None,
|
||||
terminals: list[tuple[str, int, int | None]], escape_at: tuple[int, int, str] | None,
|
||||
overflow: tuple[str, int] | None) -> None:
|
||||
self.reach = reach
|
||||
self.boundary = boundary
|
||||
self.indirect_at = indirect_at
|
||||
self.terminals = terminals
|
||||
self.escape_at = escape_at
|
||||
self.overflow = overflow
|
||||
|
||||
def terminal_evidence(self) -> str:
|
||||
if not self.terminals:
|
||||
return "term=none"
|
||||
kind, _pc, target = self.terminals[0]
|
||||
if kind == "tail_j" and target is not None:
|
||||
return f"term=tail_j:0x{target:08X}"
|
||||
return f"term={kind}"
|
||||
|
||||
def escape_evidence(self) -> str:
|
||||
assert self.escape_at is not None
|
||||
pc, target, kind = self.escape_at
|
||||
return f"escape={kind}@0x{pc:08X}->0x{target:08X}"
|
||||
|
||||
|
||||
def signed16(value: int) -> int:
|
||||
return value - 0x10000 if value & 0x8000 else value
|
||||
|
||||
|
||||
def jump_target(pc: int, word: int) -> int:
|
||||
return ((pc + 4) & 0xF0000000) | ((word & 0x03FFFFFF) << 2)
|
||||
|
||||
|
||||
def branch_target(pc: int, word: int) -> int:
|
||||
return pc + 4 + (signed16(word & 0xFFFF) << 2)
|
||||
|
||||
|
||||
def is_branch(word: int) -> bool:
|
||||
opcode = word >> 26
|
||||
if opcode in (0x04, 0x05, 0x06, 0x07):
|
||||
return True
|
||||
if opcode == 0x01: # REGIMM: bltz/bgez/bltzal/bgezal (+ likely)
|
||||
return ((word >> 16) & 0x1F) in (0, 1, 2, 3, 16, 17, 18, 19)
|
||||
if opcode == 0x10: # COP0: bc0f/bc0t
|
||||
return ((word >> 21) & 0x1F) == 8
|
||||
return False
|
||||
|
||||
|
||||
def walk(payload: bytes, text_address: int, start: int, hard: frozenset[int],
|
||||
hard_sorted: Sequence[int]) -> Walk:
|
||||
"""Explore every path from `start` and report its reachable extent."""
|
||||
limit = text_address + len(payload)
|
||||
pending = [start]
|
||||
visited: set[int] = set()
|
||||
reach = start - 4
|
||||
boundary: int | None = None
|
||||
indirect_at: int | None = None
|
||||
terminals: list[tuple[str, int, int | None]] = []
|
||||
escape_at: tuple[int, int, str] | None = None
|
||||
overflow: tuple[str, int] | None = None
|
||||
|
||||
def next_hard_after(pc: int) -> int | None:
|
||||
index = bisect.bisect_right(hard_sorted, pc)
|
||||
return hard_sorted[index] if index < len(hard_sorted) else None
|
||||
|
||||
def note_escape(pc: int, target: int, kind: str) -> None:
|
||||
nonlocal escape_at
|
||||
following = next_hard_after(pc)
|
||||
if escape_at is None and following is not None and target > following:
|
||||
escape_at = (pc, target, kind)
|
||||
|
||||
while pending:
|
||||
pc = pending.pop()
|
||||
while True:
|
||||
if pc in visited:
|
||||
break
|
||||
if pc < text_address or pc + 4 > limit:
|
||||
overflow = ("outside", pc)
|
||||
break
|
||||
if pc - start > MAX_FUNCTION_SIZE:
|
||||
overflow = ("runaway", pc)
|
||||
break
|
||||
if pc != start and pc in hard:
|
||||
if boundary is None or pc < boundary:
|
||||
boundary = pc
|
||||
break
|
||||
visited.add(pc)
|
||||
if pc + 4 > reach:
|
||||
reach = pc + 4
|
||||
word = struct.unpack_from("<I", payload, pc - text_address)[0]
|
||||
opcode = word >> 26
|
||||
funct = word & 0x3F
|
||||
|
||||
if opcode == 0 and funct == 0x08: # jr
|
||||
if pc + 8 > reach:
|
||||
reach = pc + 8 # the delay slot is part of the body
|
||||
rs = (word >> 21) & 0x1F
|
||||
if rs == 31:
|
||||
terminals.append(("jr_ra", pc, None))
|
||||
else:
|
||||
if indirect_at is None:
|
||||
indirect_at = pc
|
||||
break
|
||||
if opcode == 0 and funct == 0x09: # jalr
|
||||
pending.append(pc + 4)
|
||||
pc += 8
|
||||
continue
|
||||
if opcode == 0 and funct in (0x0C, 0x0D): # syscall / break
|
||||
terminals.append(("syscall" if funct == 0x0C else "break", pc, None))
|
||||
break
|
||||
if opcode == 0x02: # j
|
||||
target = jump_target(pc, word)
|
||||
# The delay slot executes and is part of the body, but control
|
||||
# then leaves this instruction; it must not continue linearly.
|
||||
if pc + 8 > reach:
|
||||
reach = pc + 8
|
||||
if target < start or target in hard:
|
||||
terminals.append(("tail_j", pc, target))
|
||||
break
|
||||
note_escape(pc, target, "j")
|
||||
pending.append(target)
|
||||
break
|
||||
if opcode == 0x03: # jal
|
||||
pending.append(pc + 4)
|
||||
pc += 8
|
||||
continue
|
||||
if is_branch(word):
|
||||
target = branch_target(pc, word)
|
||||
note_escape(pc, target, "branch")
|
||||
pending.append(pc + 4)
|
||||
pending.append(target)
|
||||
break
|
||||
pc += 4
|
||||
return Walk(reach, boundary, indirect_at, terminals, escape_at, overflow)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# extent derivation
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Extent:
|
||||
"""A derived extent (or a classified non-function candidate)."""
|
||||
|
||||
__slots__ = ("start", "end", "next", "grade", "evidence")
|
||||
|
||||
def __init__(self, start: int, end: int | None, next_start: int | None,
|
||||
grade: str, evidence: list[str]) -> None:
|
||||
self.start = start
|
||||
self.end = end
|
||||
self.next = next_start
|
||||
self.grade = grade
|
||||
self.evidence = evidence
|
||||
|
||||
@property
|
||||
def size(self) -> int | None:
|
||||
return None if self.end is None else self.end - self.start
|
||||
|
||||
@property
|
||||
def gap(self) -> int | None:
|
||||
if self.end is None or self.next is None:
|
||||
return None
|
||||
return self.next - self.end
|
||||
|
||||
|
||||
def derive_extents(payload: bytes, text_address: int,
|
||||
grades: dict[int, set[str]]) -> list[Extent]:
|
||||
"""Derive an extent for every candidate start, in address order."""
|
||||
hard = frozenset(a for a, names in grades.items() if names & HARD_GRADES)
|
||||
hard_sorted = sorted(hard)
|
||||
following: dict[int, int | None] = {}
|
||||
for index, address in enumerate(hard_sorted):
|
||||
following[address] = (hard_sorted[index + 1] if index + 1 < len(hard_sorted) else None)
|
||||
|
||||
extents: dict[int, Extent] = {}
|
||||
for start in hard_sorted:
|
||||
next_start = following[start]
|
||||
result = walk(payload, text_address, start, hard, hard_sorted)
|
||||
evidence: list[str] = []
|
||||
|
||||
# A hard start is another function's entry, so this function's extent
|
||||
# cannot extend past it; `clip` is the boundary the result is bounded by.
|
||||
clip = next_start if next_start is not None else result.boundary
|
||||
escaped = result.escape_at is not None or (clip is not None and result.reach > clip)
|
||||
|
||||
if result.overflow is not None:
|
||||
kind, pc = result.overflow
|
||||
grade = kind
|
||||
end = clip if clip is not None else result.reach
|
||||
evidence.append(f"pc=0x{pc:08X}")
|
||||
elif escaped:
|
||||
grade = "escape"
|
||||
end = clip if clip is not None else result.reach
|
||||
if result.escape_at is not None:
|
||||
evidence.append(result.escape_evidence())
|
||||
else:
|
||||
evidence.append(f"over=0x{result.reach:08X}")
|
||||
elif result.boundary is not None:
|
||||
grade = "fallthrough"
|
||||
end = result.boundary
|
||||
evidence.append(f"hit=0x{result.boundary:08X}")
|
||||
elif result.indirect_at is not None:
|
||||
grade = "indirect"
|
||||
end = next_start if next_start is not None else result.reach
|
||||
evidence.append(f"jr_reg=0x{result.indirect_at:08X}")
|
||||
evidence.append(f"lower_bound=0x{result.reach:08X}")
|
||||
else:
|
||||
grade = "exact"
|
||||
end = result.reach
|
||||
evidence.append(result.terminal_evidence())
|
||||
extents[start] = Extent(start, end, next_start, grade, evidence)
|
||||
|
||||
# Soft candidates: containment inside an `exact` extent is evidence that the
|
||||
# candidate is not an independent function. Extents of other grades are not
|
||||
# trustworthy enough to carry that inference.
|
||||
exact_spans = sorted(
|
||||
(extent.start, extent.end) for extent in extents.values()
|
||||
if extent.grade == "exact" and extent.end is not None
|
||||
)
|
||||
|
||||
def contained_by(address: int) -> int | None:
|
||||
for span_start, span_end in exact_spans:
|
||||
if span_start < address < span_end:
|
||||
return span_start
|
||||
if span_start > address:
|
||||
break
|
||||
return None
|
||||
|
||||
for address in sorted(set(grades) - set(hard)):
|
||||
names = ",".join(sorted(grades[address]))
|
||||
owner = contained_by(address)
|
||||
if owner is None:
|
||||
extents[address] = Extent(address, None, None, "standalone", [f"grades={names}"])
|
||||
else:
|
||||
extents[address] = Extent(address, None, None, "contained",
|
||||
[f"inside=0x{owner:08X}", f"grades={names}"])
|
||||
|
||||
return [extents[address] for address in sorted(extents)]
|
||||
|
||||
|
||||
def format_extents(extents: Sequence[Extent]) -> str:
|
||||
lines = list(HEADER_LINES)
|
||||
for extent in extents:
|
||||
end = "-" if extent.end is None else f"0x{extent.end:08X}"
|
||||
size = "-" if extent.size is None else str(extent.size)
|
||||
next_start = "-" if extent.next is None else f"0x{extent.next:08X}"
|
||||
gap = "-" if extent.gap is None else str(extent.gap)
|
||||
evidence = ";".join(extent.evidence)
|
||||
lines.append("\t".join((
|
||||
f"0x{extent.start:08X}", end, size, next_start, gap, extent.grade, evidence,
|
||||
)))
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# subcommands
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def load_payload(exe_path: Path) -> tuple[bytes, int]:
|
||||
data = exe_path.read_bytes()
|
||||
_entry, text_address, text_size = parse_psx_exe(data)
|
||||
payload = data[PAYLOAD_LMA:PAYLOAD_LMA + text_size]
|
||||
if len(payload) != text_size:
|
||||
raise ToolError("executable is truncated: payload is shorter than the header declares")
|
||||
return payload, text_address
|
||||
|
||||
|
||||
def command_scan(args: argparse.Namespace) -> int:
|
||||
exe_path = require_file(args.exe, "executable")
|
||||
inventory_path = require_file(args.inventory, "function inventory")
|
||||
out = resolve_output(args.out, args.force)
|
||||
payload, text_address = load_payload(exe_path)
|
||||
grades = load_inventory(inventory_path)
|
||||
extents = derive_extents(payload, text_address, grades)
|
||||
|
||||
counts = {grade: 0 for grade in GRADES}
|
||||
for extent in extents:
|
||||
counts[extent.grade] = counts.get(extent.grade, 0) + 1
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(format_extents(extents), encoding="ascii")
|
||||
|
||||
print(f"candidates={len(extents)}")
|
||||
for grade in GRADES:
|
||||
print(f"grade_{grade}={counts[grade]}")
|
||||
print(f"output={out}")
|
||||
return 0
|
||||
|
||||
|
||||
def command_verify(args: argparse.Namespace) -> int:
|
||||
exe_path = require_file(args.exe, "executable")
|
||||
inventory_path = require_file(args.inventory, "function inventory")
|
||||
extents_path = require_file(args.extents, "function extents")
|
||||
regions_path = require_file(args.regions, "region registry")
|
||||
|
||||
payload, text_address = load_payload(exe_path)
|
||||
grades = load_inventory(inventory_path)
|
||||
derived = {extent.start: extent for extent in derive_extents(payload, text_address, grades)}
|
||||
stored = load_inventory_extents(extents_path)
|
||||
regions = load_regions(regions_path)
|
||||
|
||||
failures: list[str] = []
|
||||
for start, end, source in regions:
|
||||
if start not in stored:
|
||||
failures.append(f"0x{start:08X} ({source}): absent from the extents table")
|
||||
continue
|
||||
if stored[start] != end:
|
||||
failures.append(
|
||||
f"0x{start:08X} ({source}): extents table says 0x{stored[start]:08X}, "
|
||||
f"registry says 0x{end:08X}")
|
||||
continue
|
||||
extent = derived.get(start)
|
||||
if extent is None:
|
||||
failures.append(f"0x{start:08X} ({source}): not a candidate in the inventory")
|
||||
continue
|
||||
if extent.end != end:
|
||||
failures.append(
|
||||
f"0x{start:08X} ({source}): derived 0x{extent.end:08X} (grade {extent.grade}), "
|
||||
f"registry says 0x{end:08X}")
|
||||
|
||||
for failure in failures:
|
||||
print(f"disagreement: {failure}", file=sys.stderr)
|
||||
if failures:
|
||||
print(f"regions={len(regions)} disagreements={len(failures)}")
|
||||
return 1
|
||||
print(f"regions={len(regions)} disagreements=0")
|
||||
print("result=AGREE")
|
||||
return 0
|
||||
|
||||
|
||||
def load_inventory_extents(path: Path) -> dict[int, int]:
|
||||
"""Read back a generated extents table: address -> end (function rows only)."""
|
||||
ends: dict[int, int] = {}
|
||||
for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
|
||||
line = raw.split("#", 1)[0].strip()
|
||||
if not line:
|
||||
continue
|
||||
fields = line.split("\t")
|
||||
if len(fields) != 7:
|
||||
raise ToolError(f"extents line {number}: expected seven fields")
|
||||
address = parse_hex(fields[0], f"extents line {number}")
|
||||
if fields[1] == "-":
|
||||
continue
|
||||
ends[address] = parse_hex(fields[1], f"extents line {number}")
|
||||
return ends
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
scan_parser = subparsers.add_parser("scan", help="derive extents for every candidate")
|
||||
scan_parser.add_argument("--exe", required=True, type=Path)
|
||||
scan_parser.add_argument("--inventory", required=True, type=Path,
|
||||
help="graded function-boundary inventory (address<TAB>grades)")
|
||||
scan_parser.add_argument("--out", required=True, type=Path)
|
||||
scan_parser.add_argument("--force", action="store_true",
|
||||
help="overwrite an existing output file")
|
||||
scan_parser.set_defaults(handler=command_scan)
|
||||
|
||||
verify_parser = subparsers.add_parser(
|
||||
"verify", help="check derived extents against the registered regions")
|
||||
verify_parser.add_argument("--exe", required=True, type=Path)
|
||||
verify_parser.add_argument("--inventory", required=True, type=Path)
|
||||
verify_parser.add_argument("--extents", required=True, type=Path)
|
||||
verify_parser.add_argument("--regions", required=True, type=Path)
|
||||
verify_parser.set_defaults(handler=command_verify)
|
||||
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
return args.handler(args)
|
||||
except ToolError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
except OSError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,411 @@
|
||||
"""Synthetic-only tests for the function-extent derivation tool.
|
||||
|
||||
Fixtures are self-authored bytes in temporary directories. They never read the
|
||||
local disc or the extracted game executable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
TOOL_PATH = Path(__file__).resolve().parents[1] / "sf3_extents"
|
||||
|
||||
|
||||
def _load_tool() -> object:
|
||||
loader = importlib.machinery.SourceFileLoader("sf3_extents_under_test", str(TOOL_PATH))
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
if spec is None:
|
||||
raise RuntimeError("could not create an import specification")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
sf3_extents = _load_tool()
|
||||
|
||||
PAYLOAD = 0x80010000
|
||||
|
||||
# Addresses (payload-relative) used by the fixture below.
|
||||
ENTRY = 0x00
|
||||
JAL_A = 0x20
|
||||
TWO_PATHS = 0x30
|
||||
TAIL_J = 0x48
|
||||
INDIRECT = 0x60
|
||||
FALLTHROUGH = 0x70
|
||||
PLAIN = 0x80
|
||||
SYSCALL_FN = 0x88
|
||||
ESCAPE = 0x98
|
||||
AFTER_ESCAPE = 0xA4
|
||||
FRAMED = 0xC0
|
||||
CONTAINS_SOFT = 0xE0
|
||||
SOFT_INSIDE = 0xE8
|
||||
SOFT_STANDALONE = 0x100
|
||||
LAST = 0x110
|
||||
|
||||
HARD_STARTS = (ENTRY, JAL_A, TWO_PATHS, TAIL_J, INDIRECT, FALLTHROUGH, PLAIN,
|
||||
SYSCALL_FN, ESCAPE, AFTER_ESCAPE, FRAMED, CONTAINS_SOFT, LAST)
|
||||
SOFT_STARTS = (SOFT_INSIDE, SOFT_STANDALONE)
|
||||
|
||||
|
||||
def _r(opcode: int, rs: int, rt: int, immediate: int) -> int:
|
||||
return (opcode << 26) | (rs << 21) | (rt << 16) | (immediate & 0xFFFF)
|
||||
|
||||
|
||||
def _jr(rs: int = 31) -> int:
|
||||
return (rs << 21) | 0x08
|
||||
|
||||
|
||||
def _jal(target: int) -> int:
|
||||
return (0x03 << 26) | ((target >> 2) & 0x03FFFFFF)
|
||||
|
||||
|
||||
def _j(target: int) -> int:
|
||||
return (0x02 << 26) | ((target >> 2) & 0x03FFFFFF)
|
||||
|
||||
|
||||
def _beq(offset_words: int) -> int:
|
||||
return _r(0x04, 0, 0, offset_words & 0xFFFF)
|
||||
|
||||
|
||||
def _addiu_sp(delta: int) -> int:
|
||||
return _r(0x09, 29, 29, delta & 0xFFFF)
|
||||
|
||||
|
||||
def _sw_sp(rt: int, offset: int) -> int:
|
||||
return _r(0x2B, 29, rt, offset)
|
||||
|
||||
|
||||
def _payload() -> bytes:
|
||||
"""A synthetic payload whose function layout is documented in the tests."""
|
||||
words: dict[int, int] = {}
|
||||
|
||||
def put(offset: int, word: int) -> None:
|
||||
words[offset] = word
|
||||
|
||||
put(0x00, _jal(PAYLOAD + JAL_A))
|
||||
put(0x08, _jr())
|
||||
put(0x10, _jal(PAYLOAD + TWO_PATHS))
|
||||
put(0x18, _jr())
|
||||
# A framed function: prologue, return, stack restore in the delay slot.
|
||||
put(JAL_A, _addiu_sp(-16))
|
||||
put(JAL_A + 4, _sw_sp(31, 12))
|
||||
put(JAL_A + 8, _jr())
|
||||
put(JAL_A + 12, _addiu_sp(16))
|
||||
# Two paths that return at different addresses: the extent must cover both.
|
||||
put(TWO_PATHS, _beq(2))
|
||||
put(TWO_PATHS + 4, _jr())
|
||||
put(TWO_PATHS + 12, 0)
|
||||
put(TWO_PATHS + 16, _jr())
|
||||
# A tail call to another function's entry.
|
||||
put(TAIL_J, _jal(PAYLOAD + INDIRECT))
|
||||
put(TAIL_J + 8, _j(PAYLOAD + PLAIN))
|
||||
# A register jump: the walk cannot follow it.
|
||||
put(INDIRECT, _jr(2))
|
||||
# No terminal: the body runs into the next hard start.
|
||||
put(FALLTHROUGH, _jal(PAYLOAD + PLAIN))
|
||||
put(PLAIN, _jr())
|
||||
put(SYSCALL_FN, 0x0C)
|
||||
# A branch whose target jumps over the next hard start.
|
||||
put(ESCAPE, _beq(6))
|
||||
put(ESCAPE + 4, _jr())
|
||||
put(ESCAPE + 28, 0)
|
||||
put(ESCAPE + 32, _jr())
|
||||
put(AFTER_ESCAPE, _jr())
|
||||
put(FRAMED, _jal(PAYLOAD + CONTAINS_SOFT))
|
||||
put(FRAMED + 8, _jr())
|
||||
# A framed function that contains a prologue-looking pair internally.
|
||||
put(CONTAINS_SOFT, _addiu_sp(-32))
|
||||
put(CONTAINS_SOFT + 4, _sw_sp(31, 28))
|
||||
put(CONTAINS_SOFT + 8, _addiu_sp(-8))
|
||||
put(CONTAINS_SOFT + 12, _sw_sp(8, 4))
|
||||
put(CONTAINS_SOFT + 16, _jr())
|
||||
put(CONTAINS_SOFT + 20, _addiu_sp(8))
|
||||
# A standalone prologue-looking pair that no extent covers.
|
||||
put(SOFT_STANDALONE, _addiu_sp(-16))
|
||||
put(SOFT_STANDALONE + 4, _sw_sp(31, 12))
|
||||
put(SOFT_STANDALONE + 8, _jr())
|
||||
put(LAST, _jr())
|
||||
|
||||
size = max(words) + 4
|
||||
return b"".join(struct.pack("<I", words.get(offset, 0))
|
||||
for offset in range(0, size, 4))
|
||||
|
||||
|
||||
def _synthetic_exe(payload: bytes, entry: int) -> bytes:
|
||||
header = bytearray(0x800)
|
||||
header[:8] = b"PS-X EXE"
|
||||
struct.pack_into("<IIII", header, 0x10, entry, 0, PAYLOAD, len(payload))
|
||||
return bytes(header) + payload
|
||||
|
||||
|
||||
def _inventory_text() -> str:
|
||||
lines = ["# synthetic inventory"]
|
||||
for address in HARD_STARTS:
|
||||
lines.append(f"0x{PAYLOAD + address:08X}\tjal")
|
||||
for address in SOFT_STARTS:
|
||||
lines.append(f"0x{PAYLOAD + address:08X}\tprologue")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def _derive() -> dict[int, object]:
|
||||
payload = _payload()
|
||||
grades: dict[int, set[str]] = {}
|
||||
for address in HARD_STARTS:
|
||||
grades[PAYLOAD + address] = {"jal"}
|
||||
grades[PAYLOAD] = {"entry", "jal"}
|
||||
for address in SOFT_STARTS:
|
||||
grades[PAYLOAD + address] = {"prologue"}
|
||||
return {extent.start: extent for extent in sf3_extents.derive_extents(payload, PAYLOAD, grades)}
|
||||
|
||||
|
||||
class ParseTests(unittest.TestCase):
|
||||
def test_rejects_missing_magic(self) -> None:
|
||||
with self.assertRaises(sf3_extents.ToolError):
|
||||
sf3_extents.parse_psx_exe(bytes(0x800))
|
||||
|
||||
def test_rejects_empty_payload(self) -> None:
|
||||
header = bytearray(0x800)
|
||||
header[:8] = b"PS-X EXE"
|
||||
with self.assertRaises(sf3_extents.ToolError):
|
||||
sf3_extents.parse_psx_exe(bytes(header))
|
||||
|
||||
|
||||
class InventoryTests(unittest.TestCase):
|
||||
def test_reads_grades_and_skips_comments(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "inventory.tsv"
|
||||
path.write_text("# c\n0x80010000\tentry,jal\n\n0x80010010\tprologue\n",
|
||||
encoding="ascii")
|
||||
grades = sf3_extents.load_inventory(path)
|
||||
self.assertEqual(grades[0x80010000], {"entry", "jal"})
|
||||
self.assertEqual(grades[0x80010010], {"prologue"})
|
||||
|
||||
def test_rejects_a_malformed_row(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "inventory.tsv"
|
||||
path.write_text("0x80010000\n", encoding="ascii")
|
||||
with self.assertRaises(sf3_extents.ToolError):
|
||||
sf3_extents.load_inventory(path)
|
||||
|
||||
def test_rejects_a_duplicate_address(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "inventory.tsv"
|
||||
path.write_text("0x80010000\tjal\n0x80010000\tentry\n", encoding="ascii")
|
||||
with self.assertRaises(sf3_extents.ToolError):
|
||||
sf3_extents.load_inventory(path)
|
||||
|
||||
def test_rejects_an_empty_inventory(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
path = Path(tmp) / "inventory.tsv"
|
||||
path.write_text("# nothing\n", encoding="ascii")
|
||||
with self.assertRaises(sf3_extents.ToolError):
|
||||
sf3_extents.load_inventory(path)
|
||||
|
||||
|
||||
class ExtentTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.extents = _derive()
|
||||
|
||||
def _extent(self, offset: int) -> object:
|
||||
return self.extents[PAYLOAD + offset]
|
||||
|
||||
def test_entry_extent_includes_its_return(self) -> None:
|
||||
extent = self._extent(ENTRY)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertEqual(extent.end, PAYLOAD + 0x10)
|
||||
self.assertEqual(extent.size, 0x10)
|
||||
|
||||
def test_framed_extent_includes_the_delay_slot(self) -> None:
|
||||
extent = self._extent(JAL_A)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertEqual(extent.end, PAYLOAD + JAL_A + 0x10)
|
||||
self.assertIn("term=jr_ra", extent.evidence)
|
||||
|
||||
def test_extent_covers_every_path_not_just_the_first_return(self) -> None:
|
||||
extent = self._extent(TWO_PATHS)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertEqual(extent.end, PAYLOAD + 0x48)
|
||||
|
||||
def test_tail_jump_is_a_terminal_and_names_its_target(self) -> None:
|
||||
extent = self._extent(TAIL_J)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertEqual(extent.end, PAYLOAD + TAIL_J + 0x10)
|
||||
self.assertIn(f"term=tail_j:0x{PAYLOAD + PLAIN:08X}", extent.evidence)
|
||||
|
||||
def test_register_jump_is_indirect_and_bounded_by_the_next_start(self) -> None:
|
||||
extent = self._extent(INDIRECT)
|
||||
self.assertEqual(extent.grade, "indirect")
|
||||
self.assertEqual(extent.end, PAYLOAD + FALLTHROUGH)
|
||||
self.assertTrue(any(item.startswith("jr_reg=") for item in extent.evidence))
|
||||
|
||||
def test_fallthrough_ends_at_the_next_hard_start(self) -> None:
|
||||
extent = self._extent(FALLTHROUGH)
|
||||
self.assertEqual(extent.grade, "fallthrough")
|
||||
self.assertEqual(extent.end, PAYLOAD + PLAIN)
|
||||
|
||||
def test_syscall_is_a_terminal(self) -> None:
|
||||
extent = self._extent(SYSCALL_FN)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertIn("term=syscall", extent.evidence)
|
||||
|
||||
def test_escape_is_clipped_and_records_the_crossing_instruction(self) -> None:
|
||||
extent = self._extent(ESCAPE)
|
||||
self.assertEqual(extent.grade, "escape")
|
||||
self.assertEqual(extent.end, PAYLOAD + AFTER_ESCAPE)
|
||||
self.assertTrue(any(item.startswith("escape=branch@") for item in extent.evidence))
|
||||
|
||||
def test_padding_shows_as_a_positive_gap(self) -> None:
|
||||
extent = self._extent(FRAMED)
|
||||
self.assertEqual(extent.grade, "exact")
|
||||
self.assertEqual(extent.end, PAYLOAD + 0xD0)
|
||||
self.assertEqual(extent.gap, CONTAINS_SOFT - 0xD0)
|
||||
|
||||
def test_last_start_has_no_next_and_no_gap(self) -> None:
|
||||
extent = self._extent(LAST)
|
||||
self.assertIsNone(extent.next)
|
||||
self.assertIsNone(extent.gap)
|
||||
|
||||
def test_soft_candidate_inside_an_exact_extent_is_contained(self) -> None:
|
||||
extent = self._extent(SOFT_INSIDE)
|
||||
self.assertEqual(extent.grade, "contained")
|
||||
self.assertIsNone(extent.end)
|
||||
self.assertIn(f"inside=0x{PAYLOAD + CONTAINS_SOFT:08X}", extent.evidence)
|
||||
|
||||
def test_soft_candidate_with_no_covering_extent_is_standalone(self) -> None:
|
||||
extent = self._extent(SOFT_STANDALONE)
|
||||
self.assertEqual(extent.grade, "standalone")
|
||||
self.assertIsNone(extent.end)
|
||||
|
||||
def test_hard_starts_are_never_contained(self) -> None:
|
||||
for address in HARD_STARTS:
|
||||
self.assertNotEqual(self.extents[PAYLOAD + address].grade, "contained")
|
||||
|
||||
def test_derivation_is_deterministic(self) -> None:
|
||||
self.assertEqual(
|
||||
sf3_extents.format_extents(list(_derive().values())),
|
||||
sf3_extents.format_extents(list(_derive().values())),
|
||||
)
|
||||
|
||||
|
||||
class FormatTests(unittest.TestCase):
|
||||
def test_header_and_column_count(self) -> None:
|
||||
text = sf3_extents.format_extents(list(_derive().values()))
|
||||
lines = text.splitlines()
|
||||
self.assertTrue(lines[0].startswith("#"))
|
||||
body = [line for line in lines if not line.startswith("#")]
|
||||
self.assertTrue(body)
|
||||
for line in body:
|
||||
self.assertEqual(len(line.split("\t")), 7)
|
||||
|
||||
def test_non_function_rows_use_dashes(self) -> None:
|
||||
text = sf3_extents.format_extents(list(_derive().values()))
|
||||
row = [line for line in text.splitlines()
|
||||
if line.startswith(f"0x{PAYLOAD + SOFT_INSIDE:08X}")][0]
|
||||
self.assertEqual(row.split("\t")[1:5], ["-", "-", "-", "-"])
|
||||
|
||||
def test_evidence_is_key_value_tokens_only(self) -> None:
|
||||
text = sf3_extents.format_extents(list(_derive().values()))
|
||||
for line in text.splitlines():
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
for token in line.split("\t")[6].split(";"):
|
||||
self.assertRegex(token, r"^[a-z_]+=")
|
||||
|
||||
def test_addresses_and_sizes_only(self) -> None:
|
||||
text = sf3_extents.format_extents(list(_derive().values()))
|
||||
for line in text.splitlines():
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
fields = line.split("\t")
|
||||
self.assertTrue(fields[0].startswith("0x"))
|
||||
self.assertIn(fields[5], sf3_extents.GRADES)
|
||||
if fields[2] != "-":
|
||||
self.assertTrue(fields[2].isdigit())
|
||||
|
||||
|
||||
class MainTests(unittest.TestCase):
|
||||
def _write_inputs(self, root: Path) -> tuple[Path, Path]:
|
||||
exe = root / "synthetic.exe"
|
||||
exe.write_bytes(_synthetic_exe(_payload(), PAYLOAD))
|
||||
inventory = root / "inventory.tsv"
|
||||
inventory.write_text(_inventory_text(), encoding="ascii")
|
||||
return exe, inventory
|
||||
|
||||
def test_scan_writes_the_extents_table(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, inventory = self._write_inputs(root)
|
||||
out = root / "extents.tsv"
|
||||
stdout = io.StringIO()
|
||||
with contextlib.redirect_stdout(stdout):
|
||||
rc = sf3_extents.main(["scan", "--exe", str(exe),
|
||||
"--inventory", str(inventory), "--out", str(out)])
|
||||
self.assertEqual(rc, 0)
|
||||
text = out.read_text(encoding="ascii")
|
||||
self.assertIn(f"0x{PAYLOAD + JAL_A:08X}\t0x{PAYLOAD + JAL_A + 0x10:08X}", text)
|
||||
self.assertIn("grade_exact=10", stdout.getvalue())
|
||||
|
||||
def test_scan_refuses_an_existing_output(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, inventory = self._write_inputs(root)
|
||||
out = root / "extents.tsv"
|
||||
out.write_text("", encoding="ascii")
|
||||
rc = sf3_extents.main(["scan", "--exe", str(exe),
|
||||
"--inventory", str(inventory), "--out", str(out)])
|
||||
self.assertEqual(rc, 2)
|
||||
|
||||
def test_scan_force_overwrites(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, inventory = self._write_inputs(root)
|
||||
out = root / "extents.tsv"
|
||||
out.write_text("stale\n", encoding="ascii")
|
||||
rc = sf3_extents.main(["scan", "--exe", str(exe), "--inventory", str(inventory),
|
||||
"--out", str(out), "--force"])
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertNotIn("stale", out.read_text(encoding="ascii"))
|
||||
|
||||
def test_verify_agrees_with_matching_regions(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, inventory = self._write_inputs(root)
|
||||
out = root / "extents.tsv"
|
||||
sf3_extents.main(["scan", "--exe", str(exe),
|
||||
"--inventory", str(inventory), "--out", str(out)])
|
||||
regions = root / "regions.tsv"
|
||||
regions.write_text(
|
||||
f"0x{PAYLOAD + JAL_A:08X}\t0x{PAYLOAD + JAL_A + 0x10:08X}\tsrc/f.c\n",
|
||||
encoding="ascii")
|
||||
rc = sf3_extents.main(["verify", "--exe", str(exe), "--inventory", str(inventory),
|
||||
"--extents", str(out), "--regions", str(regions)])
|
||||
self.assertEqual(rc, 0)
|
||||
|
||||
def test_verify_fails_on_a_disagreeing_region(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, inventory = self._write_inputs(root)
|
||||
out = root / "extents.tsv"
|
||||
sf3_extents.main(["scan", "--exe", str(exe),
|
||||
"--inventory", str(inventory), "--out", str(out)])
|
||||
regions = root / "regions.tsv"
|
||||
regions.write_text(
|
||||
f"0x{PAYLOAD + JAL_A:08X}\t0x{PAYLOAD + JAL_A + 0x14:08X}\tsrc/f.c\n",
|
||||
encoding="ascii")
|
||||
rc = sf3_extents.main(["verify", "--exe", str(exe), "--inventory", str(inventory),
|
||||
"--extents", str(out), "--regions", str(regions)])
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user