phase6: add the symbol registry and per-region flag overrides, and register 0x8002D2A0

This commit is contained in:
Christopher Williams
2026-09-23 20:54:10 -04:00
parent 73e8cd9a2e
commit 673edb2ded
8 changed files with 342 additions and 26 deletions
+5 -2
View File
@@ -24,6 +24,7 @@ INPUT_REPORT := .run/p3-exe-info.tsv
# data fallbacks taken from the original executable. An empty registry
# reproduces the all-payload data baseline exactly.
REGIONS := config/regions.tsv
SYMBOLS := config/symbols.tsv
MATCH := tools/sf3_match
CODE_OUT := build/code
EXPECTED_SHA1 := e173426c157384ebf1b6caf8c6fea18a85a14af9
@@ -71,14 +72,16 @@ binary: link
code: validate
@test -f "$(MATCH)"
@test -f "$(REGIONS)"
@test -f "$(SYMBOLS)"
@rm -rf "$(CODE_OUT)"
@"$(MATCH)" build --exe "$(EXE)" --regions "$(REGIONS)" --out "$(CODE_OUT)"
@"$(MATCH)" build --exe "$(EXE)" --regions "$(REGIONS)" --symbols "$(SYMBOLS)" --out "$(CODE_OUT)"
gate: validate
@test -f "$(MATCH)"
@test -f "$(REGIONS)"
@test -f "$(SYMBOLS)"
@rm -rf "$(CODE_OUT)"
@"$(MATCH)" gate --exe "$(EXE)" --regions "$(REGIONS)" --out "$(CODE_OUT)" \
@"$(MATCH)" gate --exe "$(EXE)" --regions "$(REGIONS)" --symbols "$(SYMBOLS)" --out "$(CODE_OUT)" \
--expect-sha1 "$(EXPECTED_SHA1)"
# Verification gates. `test` is synthetic-only and needs no game input;
+1
View File
@@ -7,3 +7,4 @@
#
# Matched so far:
0x80017AD4 0x80017AE8 src/func_80017AD4.c
0x8002D2A0 0x8002D2BC src/func_8002D2A0.c
1 # Code-region registry: one C region per matched function.
7 #
8 # Matched so far:
9 0x80017AD4
10 0x8002D2A0
+9
View File
@@ -0,0 +1,9 @@
# Symbol registry: absolute addresses for cross-references used by C regions.
#
# Columns: NAME<TAB>address (address is hex; 0x prefix optional)
#
# These name unmatched functions and globals so a C region can reference them
# by name. The harness passes each row to the assembler as
# `--defsym NAME=0xADDR`. A name here is a placeholder for an address, not a
# claim about the object's meaning; real names are earned from evidence.
g_80122354 0x80122354
1 # Symbol registry: absolute addresses for cross-references used by C regions.
2 #
3 # Columns: NAME<TAB>address (address is hex; 0x prefix optional)
4 #
5 # These name unmatched functions and globals so a C region can reference them
6 # by name. The harness passes each row to the assembler as
7 # `--defsym NAME=0xADDR`. A name here is a placeholder for an address, not a
8 # claim about the object's meaning; real names are earned from evidence.
9 g_80122354 0x80122354
+21 -4
View File
@@ -19,8 +19,18 @@ allocation" is **not** a match. If the bytes differ, the function stays fallback
Add one row to the tracked registry `config/regions.tsv`:
```
# start<TAB>end<TAB>source
# start<TAB>end<TAB>source[<TAB>overrides]
0x80017AD4 0x80017AE8 src/func_80017AD4.c
0x8002D2A0 0x8002D2BC src/func_8002D2A0.c
```
A row may carry an optional fourth field of **per-region flag overrides**: space-separated
`key=value` tokens with key `cc1` or `as`, each value a comma-separated flag list. The flags are
appended to that region's effective toolchain flags only, so a region can override a global default
(for example `cc1=-O0`). Example:
```
0x80030000 0x80030040 src/func_80030000.c cc1=-O0 as=-G8
```
Rules the harness enforces:
@@ -69,9 +79,16 @@ start before sharing.
## Symbols
Cross-references to functions and globals that are not themselves matched are supplied as absolute
assembler definitions (`--defsym NAME=0xADDR`). The harness currently accepts these **globally** for a
whole build; per-region symbols are not yet supported, which is why a symbol-referencing match such as
`0x8002D2A0` is verified but not yet registered.
assembler definitions (`--defsym NAME=0xADDR`). A **tracked symbol registry**, `config/symbols.tsv`,
holds these as `NAME<TAB>address` rows (hex, `0x` optional) and is loaded with `--symbols`:
```
g_80122354 0x80122354
```
`make code` and `make gate` pass `config/symbols.tsv` automatically; `range` accepts `--symbols`
(and `--defsym` for one-off names). A name in the registry is an address placeholder, not a claim
about the object's meaning. `0x8002D2A0` is registered and gated this way.
## Verification procedure
+26 -1
View File
@@ -7,7 +7,7 @@
## Progress
- [x] **P6-T1 — Phase control records, baseline revalidation, and open-item triage** (complete)
- [ ] P6-T2 — Registry symbol support and per-region flag overrides
- [x] **P6-T2 — Registry symbol support and per-region flag overrides** (complete)
- [ ] P6-T3 — `maspsx` integration and the ASPSX `la` verification
- [ ] P6-T4 — `-G` small-data threshold from byte evidence
- [ ] Rules check
@@ -34,6 +34,31 @@ carried context. Tracked file count is 62 (57 at Phase 5 closure plus the five `
archive records committed with the closure). No `CURRENT_PHASE.md` existed at phase start; this file
is that record.
## P6-T2 — Symbol registry and per-region flags (2026-09-23)
**Delivered:**
- `config/symbols.tsv` — tracked symbol registry, `NAME<TAB>address` rows (hex, `0x` optional).
- `tools/sf3_match --symbols FILE` — loads the registry and applies each row as `--defsym NAME=0xADDR`;
available on `range`, `build` and `gate`. `--defsym` still works for one-off names.
- `config/regions.tsv` optional fourth field — per-region `cc1=`/`as=` flag overrides, appended to
that region's effective flags. Append semantics is byte-justified: a probe showed `-O2 … -O0` and
`-O0` produce identical assembly for the same source.
- Registered `0x8002D2A0..0x8002D2BC` (`src/func_8002D2A0.c`) with the global `g_80122354`.
**Verification:**
| Check | Result |
|---|---|
| `sf3_match range --symbols config/symbols.tsv` for `0x8002D2A0` | 28 bytes, 0 differing, `result=MATCH` |
| `make gate` | `c_regions=2`, 0 differing, SHA-1 `e173426c…`, exit 0 |
| Synthetic suite | 68 tests pass (15 added: symbol/region-option parsing, failure modes, end-to-end symbol resolution, missing-symbol failure, per-region flag override) |
**Limits:** per-region overrides are append-only (a flag that must be *removed* cannot be expressed
yet); the override field is a single whitespace-delimited remainder, so flag values may not contain
spaces. Symbol names are validated assembler identifiers; addresses are not range-checked against the
payload because a symbol may name a value outside it.
## P6-T1 — Open-item triage
The Phase 5 open items are split by whether they block a **class** of functions or a **single**
+27
View File
@@ -0,0 +1,27 @@
/*
* func_8002D2A0 — 28 bytes at 0x8002D2A0..0x8002D2BC
*
* Byte-identical reconstruction of a leaf predicate. The original loads a
* global word, subtracts the first argument, and returns whether the unsigned
* difference is at least the second argument.
*
* The observed instructions are:
* lui v0,0x8012 \ v0 = g_80122354
* lw v0,9044(v0) /
* nop
* subu v0,v0,a0 v0 = g - a
* sltu v0,v0,a1 v0 = (g - a) < b
* jr ra
* xori v0,v0,0x1 return !((g - a) < b) (delay slot)
*
* LIMITS: the function name, the global's name and its type are hypotheses
* reconstructed from the disassembly. Only the compiled bytes are evidence.
* `g_80122354` is defined by the tracked `config/symbols.tsv`; it is an
* address placeholder, not a proven name.
*/
extern int g_80122354;
int func_8002D2A0(int a, int b) {
return (unsigned)(g_80122354 - a) >= (unsigned)b;
}
+103 -7
View File
@@ -11,6 +11,15 @@ This is the Phase 5 matching harness. It has four subcommands:
the original, and the registry's C regions compiled in address order.
gate build, then compare the whole rebuilt executable (cmp + SHA-1).
Cross-references to unmatched functions and globals are supplied as absolute
assembler definitions. `--defsym NAME=0xADDR` adds one on the command line;
`--symbols FILE` loads a tracked registry of `NAME<TAB>address` rows.
A region row may carry an optional fourth field of per-region flag overrides:
space-separated `key=value` tokens with key `cc1` or `as`, each value a
comma-separated flag list. The flags are appended to the effective toolchain
flags for that region only (so `cc1=-O0` overrides a global `-O2`).
The original executable and every generated artifact are caller-supplied or
written to a caller-selected fresh directory. This tool never embeds game bytes
in its own source or output and refuses to write into an existing directory.
@@ -27,8 +36,9 @@ from __future__ import annotations
import argparse
import hashlib
from dataclasses import dataclass
from dataclasses import dataclass, replace
from pathlib import Path
import re
import shutil
import struct
import subprocess
@@ -86,6 +96,17 @@ def parse_address(text: str) -> int:
raise ToolError(f"not an address: {text!r}") from exc
def parse_hex_address(text: str, line_number: int) -> int:
"""A symbol-registry address: hexadecimal, with or without a 0x prefix."""
try:
address = int(text, 16)
except ValueError as exc:
raise ToolError(f"symbols line {line_number}: not a hex address: {text!r}") from exc
if not 0 <= address <= 0xFFFFFFFF:
raise ToolError(f"symbols line {line_number}: address out of range")
return address
def require_file(path: Path, label: str) -> Path:
if not path.is_file():
raise ToolError(f"{label} is not a regular file: {path}")
@@ -130,6 +151,37 @@ class Region:
start: int
end: int
source: str
cc1_flags: tuple[str, ...] = ()
as_flags: tuple[str, ...] = ()
# A region's optional fourth field: space-separated `key=value` overrides.
# The value is a comma-separated flag list; the keys name the toolchain stage.
_REGION_OPTION_KEYS = ("cc1", "as")
def parse_region_options(text: str, line_number: int) -> tuple[tuple[str, ...], tuple[str, ...]]:
"""Parse the optional per-region flag override field."""
overrides: dict[str, tuple[str, ...]] = {}
for token in text.split():
if "=" not in token:
raise ToolError(
f"regions line {line_number}: expected 'key=value' override, got {token!r}"
)
key, _, value = token.partition("=")
if key not in _REGION_OPTION_KEYS:
raise ToolError(
f"regions line {line_number}: unknown override key {key!r}"
)
if key in overrides:
raise ToolError(
f"regions line {line_number}: duplicate override key {key!r}"
)
flags = tuple(part for part in value.split(",") if part)
if not flags:
raise ToolError(f"regions line {line_number}: empty override for {key!r}")
overrides[key] = flags
return overrides.get("cc1", ()), overrides.get("as", ())
def parse_regions(text: str) -> list[Region]:
@@ -138,14 +190,19 @@ def parse_regions(text: str) -> list[Region]:
line = raw.split("#", 1)[0].strip()
if not line:
continue
fields = line.split()
if len(fields) != 3:
raise ToolError(f"regions line {number}: expected 'start end source'")
fields = line.split(None, 3)
if len(fields) not in (3, 4):
raise ToolError(
f"regions line {number}: expected 'start end source [overrides]'"
)
start = parse_address(fields[0])
end = parse_address(fields[1])
if not 0 <= start < end:
raise ToolError(f"regions line {number}: invalid range")
regions.append(Region(start, end, fields[2]))
cc1_flags, as_flags = ((), ())
if len(fields) == 4:
cc1_flags, as_flags = parse_region_options(fields[3], number)
regions.append(Region(start, end, fields[2], cc1_flags, as_flags))
regions.sort(key=lambda region: region.start)
for left, right in zip(regions, regions[1:]):
if right.start < left.end:
@@ -153,6 +210,32 @@ def parse_regions(text: str) -> list[Region]:
return regions
# A tracked symbol registry: `NAME<TAB>address` rows naming absolute addresses.
_SYMBOL_NAME = re.compile(r"[A-Za-z_.][A-Za-z0-9_.]*\Z")
def parse_symbols(text: str) -> list[str]:
"""Parse a symbol registry into assembler `--defsym NAME=0xADDR` arguments."""
defsyms: list[str] = []
seen: set[str] = set()
for number, raw in enumerate(text.splitlines(), 1):
line = raw.split("#", 1)[0].strip()
if not line:
continue
fields = line.split()
if len(fields) != 2:
raise ToolError(f"symbols line {number}: expected 'NAME address'")
name = fields[0]
if not _SYMBOL_NAME.match(name):
raise ToolError(f"symbols line {number}: invalid symbol name {name!r}")
if name in seen:
raise ToolError(f"symbols line {number}: duplicate symbol {name!r}")
address = parse_hex_address(fields[1], number)
seen.add(name)
defsyms.append(f"{name}=0x{address:X}")
return defsyms
def validate_regions(regions: Sequence[Region], exe: PsxExe) -> None:
for region in regions:
if region.start < exe.text_address or region.end > exe.payload_end:
@@ -419,6 +502,7 @@ def _build(args: argparse.Namespace) -> tuple[Path, Path]:
)
tools = args.toolchain
by_start = {region.start: region for region in regions}
objects: list[str] = ["header.o"]
assemble_asm(header, out / "header.o", tools)
for item in items:
@@ -426,7 +510,13 @@ def _build(args: argparse.Namespace) -> tuple[Path, Path]:
if item.kind == "asm":
assemble_asm(item.source, object_path, tools)
else:
compile_c(item.source, object_path, work, tools)
region = by_start[item.start]
region_tools = replace(
tools,
cc1_flags=[*tools.cc1_flags, *region.cc1_flags],
as_flags=[*tools.as_flags, *region.as_flags],
)
compile_c(item.source, object_path, work, region_tools)
localize_symbols(object_path, tools)
objects.append(item.object_name)
@@ -504,6 +594,8 @@ def add_toolchain_arguments(parser: argparse.ArgumentParser) -> None:
help="extra assembler flag, replacing the defaults when given")
parser.add_argument("--defsym", action="append", default=[],
help="assembler --defsym NAME=0xADDR (repeatable)")
parser.add_argument("--symbols", type=Path, default=None,
help="tracked symbol registry file (NAME<TAB>address rows)")
def resolve_toolchain(args: argparse.Namespace) -> Toolchain:
@@ -513,6 +605,10 @@ def resolve_toolchain(args: argparse.Namespace) -> Toolchain:
if not located:
raise ToolError("no preprocessor found; pass --cpp")
cpp = Path(located)
defsyms = list(args.defsym)
if args.symbols is not None:
symbols_path = require_file(args.symbols, "symbol registry")
defsyms = parse_symbols(symbols_path.read_text(encoding="utf-8")) + defsyms
return Toolchain(
cpp=require_file(cpp, "preprocessor"),
cc1=require_file(args.cc1, "cc1"),
@@ -522,7 +618,7 @@ def resolve_toolchain(args: argparse.Namespace) -> Toolchain:
cpp_flags=args.cpp_flag or DEFAULT_CPP_FLAGS,
cc1_flags=args.cc1_flag or DEFAULT_CC1_FLAGS,
as_flags=args.as_flag or DEFAULT_AS_FLAGS,
defsyms=args.defsym,
defsyms=defsyms,
)
+150 -12
View File
@@ -161,6 +161,66 @@ class CompareTests(unittest.TestCase):
self.assertEqual(diff.differing_bytes, 2)
class RegionOptionTests(unittest.TestCase):
def test_parses_cc1_and_as_overrides(self) -> None:
regions = sf3_match.parse_regions(
"0x80010000 0x80010010 src/a.c\tcc1=-O0,-G8 as=-G8\n"
)
self.assertEqual(regions[0].cc1_flags, ("-O0", "-G8"))
self.assertEqual(regions[0].as_flags, ("-G8",))
def test_a_region_without_overrides_has_none(self) -> None:
regions = sf3_match.parse_regions("0x80010000 0x80010010 src/a.c\n")
self.assertEqual(regions[0].cc1_flags, ())
self.assertEqual(regions[0].as_flags, ())
def test_rejects_an_unknown_override_key(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_regions("0x80010000 0x80010010 src/a.c\tld=-O0\n")
def test_rejects_a_token_without_equals(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_regions("0x80010000 0x80010010 src/a.c\t-O0\n")
def test_rejects_an_empty_override_value(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_regions("0x80010000 0x80010010 src/a.c\tcc1=\n")
def test_rejects_a_duplicate_override_key(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_regions(
"0x80010000 0x80010010 src/a.c\tcc1=-O0 cc1=-O1\n"
)
def test_rejects_a_fifth_field(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_regions("0x80010000 0x80010010 src/a.c\tcc1=-O0 extra\n")
class SymbolTests(unittest.TestCase):
def test_parses_names_comments_and_hex(self) -> None:
defsyms = sf3_match.parse_symbols(
"# comment\n\ng_80122354\t0x80122354\nfunc_1\t80123456\n"
)
self.assertEqual(defsyms, ["g_80122354=0x80122354", "func_1=0x80123456"])
def test_rejects_a_malformed_line(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_symbols("g_1 0x80122354 0x80122358\n")
def test_rejects_an_invalid_name(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_symbols("1bad\t0x80122354\n")
def test_rejects_a_bad_address(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_symbols("g_1\tnot-a-number\n")
def test_rejects_a_duplicate_name(self) -> None:
with self.assertRaises(sf3_match.ToolError):
sf3_match.parse_symbols("g_1\t0x80122354\ng_1\t0x80122358\n")
def _toolchain_available() -> bool:
return all(
path.is_file()
@@ -179,11 +239,7 @@ int synthetic_get(Synthetic *p) { return p->value; }
class EndToEndTests(unittest.TestCase):
"""Build a synthetic executable whose payload starts with real compiler output."""
def _build_synthetic(self, root: Path, *, corrupt: bool) -> tuple[Path, int]:
source = root / "synthetic.c"
source.write_text(SOURCE, encoding="ascii")
work = root / "work"
obj = root / "synthetic.o"
def _toolchain(self, *, cc1_flags=(), as_flags=(), defsyms=()) -> object:
args = type("Args", (), {})()
args.cpp = None
args.cc1 = sf3_match.DEFAULT_CC1
@@ -191,13 +247,24 @@ class EndToEndTests(unittest.TestCase):
args.linker = sf3_match.DEFAULT_LD
args.objcopy = sf3_match.DEFAULT_OBJCOPY
args.cpp_flag = []
args.cc1_flag = []
args.as_flag = []
args.defsym = []
tools = sf3_match.resolve_toolchain(args)
args.cc1_flag = list(cc1_flags)
args.as_flag = list(as_flags)
args.defsym = list(defsyms)
args.symbols = None
return sf3_match.resolve_toolchain(args)
def _compile_payload(
self, root: Path, *, source_text: str = SOURCE, cc1_flags=(),
as_flags=(), defsyms=(), corrupt: bool = False,
) -> tuple[Path, int]:
source = root / "synthetic.c"
source.write_text(source_text, encoding="ascii")
work = root / "work"
obj = root / "synthetic.o"
tools = self._toolchain(cc1_flags=cc1_flags, as_flags=as_flags, defsyms=defsyms)
sf3_match.compile_c(source, obj, work, tools)
code = sf3_match.extract_text_bytes(obj, root / "code.bin", tools)
payload = bytearray(0x40)
payload = bytearray(max(0x40, len(code) + 0x10))
payload[:len(code)] = code
if corrupt:
payload[0] ^= 0xFF
@@ -205,14 +272,22 @@ class EndToEndTests(unittest.TestCase):
exe.write_bytes(_synthetic_exe(bytes(payload)))
return exe, len(code)
def _registry(self, root: Path, code_length: int) -> Path:
def _build_synthetic(self, root: Path, *, corrupt: bool) -> tuple[Path, int]:
return self._compile_payload(root, corrupt=corrupt)
def _registry(self, root: Path, code_length: int, overrides: str = "") -> Path:
regions = root / "regions.tsv"
regions.write_text(
f"0x{PAYLOAD:X} 0x{PAYLOAD + code_length:X} {root / 'synthetic.c'}\n",
f"0x{PAYLOAD:X}\t0x{PAYLOAD + code_length:X}\t{root / 'synthetic.c'}{overrides}\n",
encoding="ascii",
)
return regions
def _symbols(self, root: Path, rows: str) -> Path:
symbols = root / "symbols.tsv"
symbols.write_text(rows, encoding="ascii")
return symbols
def test_range_matches_its_own_compiler_output(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
@@ -287,6 +362,69 @@ class EndToEndTests(unittest.TestCase):
"--out", str(out)])
self.assertEqual(rc, 2)
def test_symbol_registry_resolves_an_extern_reference(self) -> None:
source = "extern int g_test;\nint synthetic_load(void) { return g_test; }\n"
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
exe, code_length = self._compile_payload(
root, source_text=source, defsyms=["g_test=0x80123456"]
)
regions = self._registry(root, code_length)
symbols = self._symbols(root, "g_test\t0x80123456\n")
out = root / "out"
rc = sf3_match.main(["gate", "--exe", str(exe), "--regions", str(regions),
"--symbols", str(symbols), "--out", str(out)])
self.assertEqual(rc, 0)
self.assertEqual((out / "scus_946_40.rebuilt").read_bytes(), exe.read_bytes())
def test_a_missing_symbol_fails_the_build(self) -> None:
source = "extern int g_test;\nint synthetic_load(void) { return g_test; }\n"
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
exe, code_length = self._compile_payload(
root, source_text=source, defsyms=["g_test=0x80123456"]
)
regions = self._registry(root, code_length)
symbols = self._symbols(root, "# empty registry\n")
rc = sf3_match.main(["gate", "--exe", str(exe), "--regions", str(regions),
"--symbols", str(symbols), "--out", str(root / "out")])
self.assertEqual(rc, 2)
def test_per_region_flag_override_changes_codegen(self) -> None:
source = "int synthetic_mul(int a, int b) { return (a + 1) * (b - 2); }\n"
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
src = root / "synthetic.c"
src.write_text(source, encoding="ascii")
# Reference: the source compiled directly with the full -O0 flag set.
ref_tools = self._toolchain(
cc1_flags=["-quiet", "-O0", "-G0", "-mno-split-addresses"]
)
ref_obj = root / "ref.o"
sf3_match.compile_c(src, ref_obj, root / "refwork", ref_tools)
ref_bytes = sf3_match.extract_text_bytes(ref_obj, root / "ref.bin", ref_tools)
exe = root / "synthetic.exe"
exe.write_bytes(_synthetic_exe(bytes(max(0x40, len(ref_bytes) + 0x10))))
def region_bytes(overrides: str, name: str) -> bytes:
regions = root / f"regions_{name}.tsv"
regions.write_text(
f"0x{PAYLOAD:X}\t0x{PAYLOAD + len(ref_bytes):X}\t{src}{overrides}\n",
encoding="ascii",
)
out = root / f"out_{name}"
rc = sf3_match.main(["build", "--exe", str(exe),
"--regions", str(regions), "--out", str(out)])
self.assertEqual(rc, 0)
return sf3_match.extract_text_bytes(
out / "region_000.o", root / f"region_{name}.bin", ref_tools
)
# The per-region override reaches the compiler: cc1=-O0 reproduces -O0.
self.assertEqual(region_bytes("\tcc1=-O0", "override"), ref_bytes)
# Without it, the global -O2 codegen differs.
self.assertNotEqual(region_bytes("", "plain"), ref_bytes)
if __name__ == "__main__":
unittest.main()