From 2f05e821fc61490b848edbdcf6923975df4f8d63 Mon Sep 17 00:00:00 2001 From: Christopher Williams Date: Wed, 23 Sep 2026 22:40:26 -0400 Subject: [PATCH] phase7: close the phase with the verification milestone Developer-confirmed milestone: 34 distinct byte-identical function bodies across 39 registered regions, up from 11 / 12 at Phase 6 close. Every region end is now derived from control flow and enforced by make check, duplicate bodies are censused, match targets come from a ranked worklist, and cross-references no longer need hand-written symbol rows. Adds PhaseEnd_Phase7.md and the digest entry, and archives CURRENT_PHASE.md as logs/Phase7.md. Final gates: 168 synthetic tests; make clean, make all, cmp and SHA-1 e173426c...; make check green at c_regions=39 with 0 differing bytes and regions=39 disagreements=0. 121 tracked files, none under a prohibited root. --- phase-ends/DIGEST.md | 37 +++++ phase-ends/PhaseEnd_Phase7.md | 132 ++++++++++++++++++ .../{CURRENT_PHASE.md => logs/Phase7.md} | 0 3 files changed, 169 insertions(+) create mode 100644 phase-ends/PhaseEnd_Phase7.md rename phase-ends/{CURRENT_PHASE.md => logs/Phase7.md} (100%) diff --git a/phase-ends/DIGEST.md b/phase-ends/DIGEST.md index b6791f6..f2eeaf2 100644 --- a/phase-ends/DIGEST.md +++ b/phase-ends/DIGEST.md @@ -27,3 +27,40 @@ Phase 5 identified the original build toolchain from byte evidence and produced ## Phase 6 — From One Match to a Matching Pipeline (2026-09-23) Phase 6 turned the single match into a pipeline and corrected the project's compiler identification. The harness gained a tracked symbol registry (`config/symbols.tsv`) resolved at **link** time, per-region compiler/assembler flag overrides in `config/regions.tsv`, a **maspsx** ASPSX-emulation stage between `cc1` and `as`, and a per-symbol `gp` marker that forces explicit `%gp_rel` for small-data accesses. A tracked, synthetic-tested `tools/sf3_boundaries` produces `config/function_inventory.tsv`: 2,875 candidate function starts graded `entry`/`jal`/`prologue`/`ghidra` (addresses only), of which 575 `jal` targets are absent from Ghidra's function set; library-versus-game-code is explicitly unresolved. Trying to match a framed function exposed that Phase 5's compiler (`egcs-2.91.66`, PsyQ 4.5) could not produce the game's epilogues (322 framed functions end `lw ra` / `addiu sp,sp,N` / `jr ra` / `nop`). The real PsyQ **4.0** and 4.1 SDKs were obtained (ignored `tools/psyq/`; checksums in `docs/SETUP.md`), and the original compiler is **PsyQ 4.0 `CC1PSX`** (`GNU C 2.7.2.SN32.3.7.0002`), for which the open `gcc-2.7.2-psx` is instruction-identical across **21/21** probe files (next best 6); the SDK 4.0 assembler is **ASPSX 2.56**, and Phase 5's `-mno-split-addresses` requirement was an artifact of the wrong compiler. The loader's `PsyQ Version = 4.5.0` is the runtime library version. The batch is **12 registered regions / 11 distinct functions** across three shapes — leaf getter/setter, `la`/`gp`-relative, and call with a frame — with duplicate sharing demonstrated on a real body that occurs exactly twice (`0x800262E0`/`0x800262EC`). The Ghidra-draft workflow (decompiler skeleton → first-draft C → byte gate) was adopted and documented; it reproduced the previously unexplained real `mult` by 68 in `0x8005DEF8` via a non-const local (`short k = 68;`), reaching a 5-byte near-match that is deliberately not registered. Final gates: 86 synthetic tests pass; `make clean`, `make all`, `cmp` and SHA-1 green; `make gate` `c_regions=12`, 0 differing bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`; 0 tracked paths under any prohibited root. Unresolved and recorded: `0x8005DEF8` (5-byte register tie-break), `0x800F3160` (store-in-delay-slot scheduling), the numeric `-G`, the CRT entry, library-versus-game-code, and broader segmentation. No rules were added. + +## Phase 7 — Function Extents, Duplicate Census, and a Scaled Matching Batch (2026-09-23) + +Phase 7 attacked the reason matching did not scale. Phase 6 graded function **starts** but left every +**end** to be derived by hand, and nothing knew how many bodies the executable shares with itself. Three +evidence artifacts closed that gap: `tools/sf3_extents` derives an `[start, end)` extent for all 2,875 +candidates from the payload's own control flow (grades `exact` 1,940, `fallthrough` 256, `indirect` 73, +`escape` 15, plus `contained` 153 and `standalone` 438 for soft candidates); `tools/sf3_dupes` censuses +byte-identical bodies (65 groups, of which **10 contain code** and all are `exact`-graded, including a +712-byte shared body); and `tools/sf3_triage` ranks a 1,916-entry worklist by `(tier, size, address)` +with every exclusion counted. All 39 registered regions reproduce at `exact` grade and +`make extents-verify` — part of `make check` — makes disagreement impossible to miss. + +Two measured corrections to the Phase 6 evidence model came out of this. Soft (`prologue`-only) starts +are frequently **false positives inside a real body** — at `0x800152AC` the body's second instruction +satisfies the prologue grade exactly — so only `jal`/`entry` are walk boundaries; enforcing soft +boundaries stopped 155 of 416 walks inside a real body. And the `jal` grade is **not sufficient in a +data region**: 252 extents have all-zero bodies, 245 of them inside the 160 KB zero band +`0x80147000`–`0x80170000`, because a data word with opcode `3` is graded as a call whose target lands in +the zeros. The census flags those groups rather than hiding them. + +Matching then scaled: **34 distinct bodies / 39 registered regions** (from 11 / 12), 23 new bodies, all +instruction-identical with the clean full-binary gate green at `c_regions=39`, 0 differing bytes, SHA-1 +`e173426c157384ebf1b6caf8c6fea18a85a14af9`. Three duplicate groups were matched once and registered N +times (`g0001` ×4, `g0002` ×2, `g0008` ×2). The per-function symbol cost was removed by resolving +address-named symbols (`func_XXXXXXXX`, `D_XXXXXXXX`, `g_XXXXXXXX`, `lbl_XXXXXXXX`) from the object's own +`nm -u` list, with a loud pre-link failure for anything unresolvable. New cookbook findings: a store-only +function leaves its constant in `v0` as scratch (writing `return 1` costs a delay-slot instruction), the +`gp`-relative offsets proved so far, and the shapes plain C cannot reach without SDK headers (GTE/COP2 +code, BIOS `syscall` wrappers, stack-pointer accessors). Recorded near-matches: `0x800F7FB4` (loop body +exact, an unexplained 8-byte frame after five C forms and an `-O` matrix) and `0x800F8AEC` +(`lui`/`jr ra`/`addiu` scheduling). Final gates: 168 synthetic tests pass; `make clean`, `make all`, +`cmp` and SHA-1 green; `make check` green; 121 tracked files, none under a prohibited root. The plan's one +deviation: the duplicate census is tracked rather than ignored, because it holds addresses, sizes and +grades only and the worklist must be reproducible from tracked inputs. Unresolved: `0x8005DEF8`, +`0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes, the numeric `-G`, the CRT entry, and +library-versus-game-code. No rules were added. diff --git a/phase-ends/PhaseEnd_Phase7.md b/phase-ends/PhaseEnd_Phase7.md new file mode 100644 index 0000000..0f3b4ea --- /dev/null +++ b/phase-ends/PhaseEnd_Phase7.md @@ -0,0 +1,132 @@ +# PhaseEnd — Phase 7: Function Extents, Duplicate Census, and a Scaled Matching Batch + +**Date:** 2026-09-23 +**Phase Status:** Complete +**Milestone confirmed by developer:** yes + +## Completed Checklist + +- P7-T1 — Created the phase control record, revalidated the clean baseline, and triaged the Phase 6 + open items into four class blockers and six deferred or unresolved items. +- P7-T2 — Derived evidence-graded function **extents** for all 2,875 candidates + (`tools/sf3_extents`, `config/function_extents.tsv`); all 12 then-registered regions reproduced + exactly, and `make extents-verify` became part of `make check`. +- P7-T3 — Built the **duplicate-body census** (`tools/sf3_dupes`, `config/duplicate_bodies.tsv`), + reproducing the hand-found Phase 6 pair and exposing a zero band of false positives. +- P7-T4 — Built the **ranked match worklist** (`tools/sf3_triage`, `config/match_worklist.tsv`) with + every exclusion counted. +- Rules check — Re-read `AGENTS.md` mandatory behavior after P7-T4 and stated the required notice. +- P7-T5 — Made **symbol rows scale**: implicit address-symbol resolution from the object's own + undefined-symbol list, and a loud pre-link failure for anything unresolvable. +- P7-T6 — Scaled batch A: matched 11 new bodies, taking the phase past fifteen. +- P7-T7 — Scaled batch B: matched 11 more, taking the project to **34 distinct bodies** — past the + milestone of thirty. +- P7-T8 — Folded the findings into the cookbook and conventions, wrote the verification record, ran + every clean gate, and obtained developer confirmation. + +## Verified Results + +### Matching batch + +**34 distinct bodies / 39 registered regions** (from 11 bodies / 12 regions at Phase 6 close; 23 new +bodies this phase). `make gate` → `c_regions=39`, 0 differing bytes, SHA-1 +`e173426c157384ebf1b6caf8c6fea18a85a14af9`. + +| Shape | Bodies | +|---|---| +| Empty function (registered against 4 duplicate addresses) | 1 | +| Leaf getter/setter (byte, halfword, word) | 12 | +| `gp`-relative getter/setter | 9 | +| Framed wrapper with a call | 1 | +| Word-buffer clearer | 1 | +| Phase 6 carry-in | 11 | + +Three duplicate groups were matched **once** and registered N times: `g0001` (4 addresses), +`g0002` (2), `g0008` (2). + +### The workflow that replaced hand-derivation + +| Before Phase 7 | After Phase 7 | +|---|---| +| A function's end was derived by hand from the disassembly | Derived from the payload's control flow; all 39 regions reproduce exactly and `make check` enforces it | +| Duplicates were found by searching for a body by hand | `config/duplicate_bodies.tsv` lists every group; the worklist ranks representatives first | +| Targets were chosen by eye from the inventory | `config/match_worklist.tsv` ranks 1,916 candidates by `(tier, size, address)` | +| Every cross-reference needed a hand-written registry row | Address-named symbols resolve implicitly; anything else fails loudly before the link | + +### New tracked artifacts + +| Artifact | Content | +|---|---| +| `config/function_extents.tsv` | 2,875 candidates: `exact` 1,940, `fallthrough` 256, `indirect` 73, `escape` 15, `contained` 153, `standalone` 438 | +| `config/duplicate_bodies.tsv` | 65 groups: 10 containing code (all `exact`-graded), 55 all-zero bodies flagged | +| `config/match_worklist.tsv` | 1,916 listed: tier 0 = 9, tier 1 = 509, tier 2 = 1,394, tier 3 = 4; 355 exclusions counted | +| `tools/sf3_extents`, `tools/sf3_dupes`, `tools/sf3_triage` | 74 synthetic tests; `tools/sf3_match` gained 8 more | + +### Measured corrections to the Phase 6 evidence model + +- **Soft starts are frequently false positives inside a real body.** At `0x800152AC` the real body is + `lw v1,8(gp)` / `addiu sp,sp,-176` / `sw ra,172(sp)`, so `0x800152B0` — the `addiu sp,sp,-176` — + satisfies the `prologue` grade exactly. Enforcing soft starts as walk boundaries stopped 155 of 416 + walks inside a real body; the extents tool therefore treats only `jal`/`entry` as boundaries. +- **The `jal` grade is not sufficient in a data region.** 252 extents have all-zero bodies, 245 of them + inside the 160 KB zero band `0x80147000`–`0x80170000`: a data word with opcode `3` is graded as a + call whose target lands in the zero band, and the walk then decodes zeros as `nop`s. The census flags + those groups `zero` rather than hiding them. +- **A walk must explore every path, not stop at the first terminal**, or a function whose paths return + at different addresses under-reports its extent. + +## Deviations and Bounded Results + +| Item | Plan | Actual | Reason / limit | +|---|---|---|---| +| Census artifact | ignored, only aggregate counts tracked | **tracked** (`config/duplicate_bodies.tsv`) | It holds addresses, sizes and grades only — the same class as the already-tracked inventory and extents tables — and the worklist must be reproducible from tracked inputs alone. The content hash is still computed and never written. | +| `0x800F7FB4` (36 bytes, ×3 addresses) | match it | **near-match** | The loop body reproduces exactly, but the original has no frame and every tested formulation allocates 8 bytes — five C forms plus an `-O1`/`-O2`/`-O3`/`-fomit-frame-pointer` matrix. Stopped after two distinct attempts. | +| `0x800F8AEC` (12 bytes) | match it | **near-match** | The original schedules `lui` / `jr ra` / `addiu` with the low half in the delay slot; the reconstruction emits `lui` / `addiu` / `jr ra` / `nop`. | +| `0x80010810` (60 bytes, ×2) | match it | deferred | GTE (COP2) code (`gte_ldVXY0`/`nRTPS`/`gte_stZ3`); needs the SDK's GTE macros, and the build has no headers. | +| `0x800FB5D4`, `0x80103FCC`, `0x80103FEC`, `0x8001EAFC` | match them | deferred | A stack-pointer accessor (`move v0,sp`), two BIOS `syscall` wrappers, and a shared jump block that is not a standalone function. | +| `0x8005DEF8`, `0x800F3160` | deferred by direction | unchanged | Recorded near-matches from Phase 6; still not registered. | +| Library vs game code | re-flag if evidence appears | **still unresolved** | An `exact` extent means "the reachable extent of a called entry", which is equally true of library and game code. | + +## Verification and Firewall + +- `make clean` exit 0; `make all` exit 0; `cmp` exit 0; both files SHA-1 + `e173426c157384ebf1b6caf8c6fea18a85a14af9`. +- `make check` exit 0: **168 tests pass** (82 added this phase); `regions=39 disagreements=0` + (`result=AGREE`); `c_regions=39`, `differing_bytes=0`, `result=MATCH`. +- Determinism: two `sf3_extents scan` runs are byte-identical; the worklist order is `(tier, size, + address)` from tracked inputs alone. +- Independent oracle: Ghidra's function bodies for `FUN_80017ad4` (`[80017ad4, 80017ae7]`) and + `FUN_800f8f9c` (`[800f8f9c, 800f8fbf]`) equal the derived extents. +- 121 tracked files, **0** under any prohibited root; `git diff --check` exit 0. +- No game bytes, disassembly, generated assembly, build outputs, expected binaries, Ghidra material, + dumps, or proprietary SDK material are tracked. The new tables hold addresses, sizes, grades, counts + and group labels only. + +## Rules Added This Phase + +None. + +## Next + +Phase 7 is closed. A future phase must begin in a fresh session with an approved task-by-task plan. It +should preserve the all-payload data baseline, the ordered-layout harness, the corrected toolchain, the +extents/census/worklist evidence tables, and the byte gate, and must treat the following as unresolved: +`0x8005DEF8`, `0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes (GTE, `syscall`, +`move v0,sp`), the numeric `-G`, the CRT entry, and library-versus-game-code. + +## Plain-English Recap + +Phase 7 attacked the reason matching did not scale. Phase 6 could say where a function *started* but +every function's *end* was worked out by hand, and nobody knew how many bodies the executable shares +with itself. The phase built three pieces of evidence — control-flow-derived extents, a duplicate +census, and a ranked worklist — and then used them to match twenty-three more functions, taking the +project from eleven to thirty-four and from twelve to thirty-nine registered regions. Along the way it +found two mistakes in the earlier evidence model (a "prologue" pattern that appears inside real bodies, +and a zero-filled region full of fake call targets), removed the need to hand-write a symbol row for +every cross-reference, and recorded two new near-misses with the exact evidence that stopped them. The +executable still rebuilds byte for byte. + +## 🛑 Stop Here + +`CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase7.md`; this PhaseEnd and the digest update +belong in the closure commit. Do not begin Phase 8 execution in this session. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase7.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase7.md