phase7: close the phase with the verification milestone
Developer-confirmed milestone: 34 distinct byte-identical function bodies across 39 registered regions, up from 11 / 12 at Phase 6 close. Every region end is now derived from control flow and enforced by make check, duplicate bodies are censused, match targets come from a ranked worklist, and cross-references no longer need hand-written symbol rows. Adds PhaseEnd_Phase7.md and the digest entry, and archives CURRENT_PHASE.md as logs/Phase7.md. Final gates: 168 synthetic tests; make clean, make all, cmp and SHA-1 e173426c...; make check green at c_regions=39 with 0 differing bytes and regions=39 disagreements=0. 121 tracked files, none under a prohibited root.
This commit is contained in:
Christopher Williams
@@ -27,3 +27,40 @@ Phase 5 identified the original build toolchain from byte evidence and produced
|
||||
## Phase 6 — From One Match to a Matching Pipeline (2026-09-23)
|
||||
|
||||
Phase 6 turned the single match into a pipeline and corrected the project's compiler identification. The harness gained a tracked symbol registry (`config/symbols.tsv`) resolved at **link** time, per-region compiler/assembler flag overrides in `config/regions.tsv`, a **maspsx** ASPSX-emulation stage between `cc1` and `as`, and a per-symbol `gp` marker that forces explicit `%gp_rel` for small-data accesses. A tracked, synthetic-tested `tools/sf3_boundaries` produces `config/function_inventory.tsv`: 2,875 candidate function starts graded `entry`/`jal`/`prologue`/`ghidra` (addresses only), of which 575 `jal` targets are absent from Ghidra's function set; library-versus-game-code is explicitly unresolved. Trying to match a framed function exposed that Phase 5's compiler (`egcs-2.91.66`, PsyQ 4.5) could not produce the game's epilogues (322 framed functions end `lw ra` / `addiu sp,sp,N` / `jr ra` / `nop`). The real PsyQ **4.0** and 4.1 SDKs were obtained (ignored `tools/psyq/`; checksums in `docs/SETUP.md`), and the original compiler is **PsyQ 4.0 `CC1PSX`** (`GNU C 2.7.2.SN32.3.7.0002`), for which the open `gcc-2.7.2-psx` is instruction-identical across **21/21** probe files (next best 6); the SDK 4.0 assembler is **ASPSX 2.56**, and Phase 5's `-mno-split-addresses` requirement was an artifact of the wrong compiler. The loader's `PsyQ Version = 4.5.0` is the runtime library version. The batch is **12 registered regions / 11 distinct functions** across three shapes — leaf getter/setter, `la`/`gp`-relative, and call with a frame — with duplicate sharing demonstrated on a real body that occurs exactly twice (`0x800262E0`/`0x800262EC`). The Ghidra-draft workflow (decompiler skeleton → first-draft C → byte gate) was adopted and documented; it reproduced the previously unexplained real `mult` by 68 in `0x8005DEF8` via a non-const local (`short k = 68;`), reaching a 5-byte near-match that is deliberately not registered. Final gates: 86 synthetic tests pass; `make clean`, `make all`, `cmp` and SHA-1 green; `make gate` `c_regions=12`, 0 differing bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`; 0 tracked paths under any prohibited root. Unresolved and recorded: `0x8005DEF8` (5-byte register tie-break), `0x800F3160` (store-in-delay-slot scheduling), the numeric `-G`, the CRT entry, library-versus-game-code, and broader segmentation. No rules were added.
|
||||
|
||||
## Phase 7 — Function Extents, Duplicate Census, and a Scaled Matching Batch (2026-09-23)
|
||||
|
||||
Phase 7 attacked the reason matching did not scale. Phase 6 graded function **starts** but left every
|
||||
**end** to be derived by hand, and nothing knew how many bodies the executable shares with itself. Three
|
||||
evidence artifacts closed that gap: `tools/sf3_extents` derives an `[start, end)` extent for all 2,875
|
||||
candidates from the payload's own control flow (grades `exact` 1,940, `fallthrough` 256, `indirect` 73,
|
||||
`escape` 15, plus `contained` 153 and `standalone` 438 for soft candidates); `tools/sf3_dupes` censuses
|
||||
byte-identical bodies (65 groups, of which **10 contain code** and all are `exact`-graded, including a
|
||||
712-byte shared body); and `tools/sf3_triage` ranks a 1,916-entry worklist by `(tier, size, address)`
|
||||
with every exclusion counted. All 39 registered regions reproduce at `exact` grade and
|
||||
`make extents-verify` — part of `make check` — makes disagreement impossible to miss.
|
||||
|
||||
Two measured corrections to the Phase 6 evidence model came out of this. Soft (`prologue`-only) starts
|
||||
are frequently **false positives inside a real body** — at `0x800152AC` the body's second instruction
|
||||
satisfies the prologue grade exactly — so only `jal`/`entry` are walk boundaries; enforcing soft
|
||||
boundaries stopped 155 of 416 walks inside a real body. And the `jal` grade is **not sufficient in a
|
||||
data region**: 252 extents have all-zero bodies, 245 of them inside the 160 KB zero band
|
||||
`0x80147000`–`0x80170000`, because a data word with opcode `3` is graded as a call whose target lands in
|
||||
the zeros. The census flags those groups rather than hiding them.
|
||||
|
||||
Matching then scaled: **34 distinct bodies / 39 registered regions** (from 11 / 12), 23 new bodies, all
|
||||
instruction-identical with the clean full-binary gate green at `c_regions=39`, 0 differing bytes, SHA-1
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`. Three duplicate groups were matched once and registered N
|
||||
times (`g0001` ×4, `g0002` ×2, `g0008` ×2). The per-function symbol cost was removed by resolving
|
||||
address-named symbols (`func_XXXXXXXX`, `D_XXXXXXXX`, `g_XXXXXXXX`, `lbl_XXXXXXXX`) from the object's own
|
||||
`nm -u` list, with a loud pre-link failure for anything unresolvable. New cookbook findings: a store-only
|
||||
function leaves its constant in `v0` as scratch (writing `return 1` costs a delay-slot instruction), the
|
||||
`gp`-relative offsets proved so far, and the shapes plain C cannot reach without SDK headers (GTE/COP2
|
||||
code, BIOS `syscall` wrappers, stack-pointer accessors). Recorded near-matches: `0x800F7FB4` (loop body
|
||||
exact, an unexplained 8-byte frame after five C forms and an `-O` matrix) and `0x800F8AEC`
|
||||
(`lui`/`jr ra`/`addiu` scheduling). Final gates: 168 synthetic tests pass; `make clean`, `make all`,
|
||||
`cmp` and SHA-1 green; `make check` green; 121 tracked files, none under a prohibited root. The plan's one
|
||||
deviation: the duplicate census is tracked rather than ignored, because it holds addresses, sizes and
|
||||
grades only and the worklist must be reproducible from tracked inputs. Unresolved: `0x8005DEF8`,
|
||||
`0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes, the numeric `-G`, the CRT entry, and
|
||||
library-versus-game-code. No rules were added.
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
# PhaseEnd — Phase 7: Function Extents, Duplicate Census, and a Scaled Matching Batch
|
||||
|
||||
**Date:** 2026-09-23
|
||||
**Phase Status:** Complete
|
||||
**Milestone confirmed by developer:** yes
|
||||
|
||||
## Completed Checklist
|
||||
|
||||
- P7-T1 — Created the phase control record, revalidated the clean baseline, and triaged the Phase 6
|
||||
open items into four class blockers and six deferred or unresolved items.
|
||||
- P7-T2 — Derived evidence-graded function **extents** for all 2,875 candidates
|
||||
(`tools/sf3_extents`, `config/function_extents.tsv`); all 12 then-registered regions reproduced
|
||||
exactly, and `make extents-verify` became part of `make check`.
|
||||
- P7-T3 — Built the **duplicate-body census** (`tools/sf3_dupes`, `config/duplicate_bodies.tsv`),
|
||||
reproducing the hand-found Phase 6 pair and exposing a zero band of false positives.
|
||||
- P7-T4 — Built the **ranked match worklist** (`tools/sf3_triage`, `config/match_worklist.tsv`) with
|
||||
every exclusion counted.
|
||||
- Rules check — Re-read `AGENTS.md` mandatory behavior after P7-T4 and stated the required notice.
|
||||
- P7-T5 — Made **symbol rows scale**: implicit address-symbol resolution from the object's own
|
||||
undefined-symbol list, and a loud pre-link failure for anything unresolvable.
|
||||
- P7-T6 — Scaled batch A: matched 11 new bodies, taking the phase past fifteen.
|
||||
- P7-T7 — Scaled batch B: matched 11 more, taking the project to **34 distinct bodies** — past the
|
||||
milestone of thirty.
|
||||
- P7-T8 — Folded the findings into the cookbook and conventions, wrote the verification record, ran
|
||||
every clean gate, and obtained developer confirmation.
|
||||
|
||||
## Verified Results
|
||||
|
||||
### Matching batch
|
||||
|
||||
**34 distinct bodies / 39 registered regions** (from 11 bodies / 12 regions at Phase 6 close; 23 new
|
||||
bodies this phase). `make gate` → `c_regions=39`, 0 differing bytes, SHA-1
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`.
|
||||
|
||||
| Shape | Bodies |
|
||||
|---|---|
|
||||
| Empty function (registered against 4 duplicate addresses) | 1 |
|
||||
| Leaf getter/setter (byte, halfword, word) | 12 |
|
||||
| `gp`-relative getter/setter | 9 |
|
||||
| Framed wrapper with a call | 1 |
|
||||
| Word-buffer clearer | 1 |
|
||||
| Phase 6 carry-in | 11 |
|
||||
|
||||
Three duplicate groups were matched **once** and registered N times: `g0001` (4 addresses),
|
||||
`g0002` (2), `g0008` (2).
|
||||
|
||||
### The workflow that replaced hand-derivation
|
||||
|
||||
| Before Phase 7 | After Phase 7 |
|
||||
|---|---|
|
||||
| A function's end was derived by hand from the disassembly | Derived from the payload's control flow; all 39 regions reproduce exactly and `make check` enforces it |
|
||||
| Duplicates were found by searching for a body by hand | `config/duplicate_bodies.tsv` lists every group; the worklist ranks representatives first |
|
||||
| Targets were chosen by eye from the inventory | `config/match_worklist.tsv` ranks 1,916 candidates by `(tier, size, address)` |
|
||||
| Every cross-reference needed a hand-written registry row | Address-named symbols resolve implicitly; anything else fails loudly before the link |
|
||||
|
||||
### New tracked artifacts
|
||||
|
||||
| Artifact | Content |
|
||||
|---|---|
|
||||
| `config/function_extents.tsv` | 2,875 candidates: `exact` 1,940, `fallthrough` 256, `indirect` 73, `escape` 15, `contained` 153, `standalone` 438 |
|
||||
| `config/duplicate_bodies.tsv` | 65 groups: 10 containing code (all `exact`-graded), 55 all-zero bodies flagged |
|
||||
| `config/match_worklist.tsv` | 1,916 listed: tier 0 = 9, tier 1 = 509, tier 2 = 1,394, tier 3 = 4; 355 exclusions counted |
|
||||
| `tools/sf3_extents`, `tools/sf3_dupes`, `tools/sf3_triage` | 74 synthetic tests; `tools/sf3_match` gained 8 more |
|
||||
|
||||
### Measured corrections to the Phase 6 evidence model
|
||||
|
||||
- **Soft starts are frequently false positives inside a real body.** At `0x800152AC` the real body is
|
||||
`lw v1,8(gp)` / `addiu sp,sp,-176` / `sw ra,172(sp)`, so `0x800152B0` — the `addiu sp,sp,-176` —
|
||||
satisfies the `prologue` grade exactly. Enforcing soft starts as walk boundaries stopped 155 of 416
|
||||
walks inside a real body; the extents tool therefore treats only `jal`/`entry` as boundaries.
|
||||
- **The `jal` grade is not sufficient in a data region.** 252 extents have all-zero bodies, 245 of them
|
||||
inside the 160 KB zero band `0x80147000`–`0x80170000`: a data word with opcode `3` is graded as a
|
||||
call whose target lands in the zero band, and the walk then decodes zeros as `nop`s. The census flags
|
||||
those groups `zero` rather than hiding them.
|
||||
- **A walk must explore every path, not stop at the first terminal**, or a function whose paths return
|
||||
at different addresses under-reports its extent.
|
||||
|
||||
## Deviations and Bounded Results
|
||||
|
||||
| Item | Plan | Actual | Reason / limit |
|
||||
|---|---|---|---|
|
||||
| Census artifact | ignored, only aggregate counts tracked | **tracked** (`config/duplicate_bodies.tsv`) | It holds addresses, sizes and grades only — the same class as the already-tracked inventory and extents tables — and the worklist must be reproducible from tracked inputs alone. The content hash is still computed and never written. |
|
||||
| `0x800F7FB4` (36 bytes, ×3 addresses) | match it | **near-match** | The loop body reproduces exactly, but the original has no frame and every tested formulation allocates 8 bytes — five C forms plus an `-O1`/`-O2`/`-O3`/`-fomit-frame-pointer` matrix. Stopped after two distinct attempts. |
|
||||
| `0x800F8AEC` (12 bytes) | match it | **near-match** | The original schedules `lui` / `jr ra` / `addiu` with the low half in the delay slot; the reconstruction emits `lui` / `addiu` / `jr ra` / `nop`. |
|
||||
| `0x80010810` (60 bytes, ×2) | match it | deferred | GTE (COP2) code (`gte_ldVXY0`/`nRTPS`/`gte_stZ3`); needs the SDK's GTE macros, and the build has no headers. |
|
||||
| `0x800FB5D4`, `0x80103FCC`, `0x80103FEC`, `0x8001EAFC` | match them | deferred | A stack-pointer accessor (`move v0,sp`), two BIOS `syscall` wrappers, and a shared jump block that is not a standalone function. |
|
||||
| `0x8005DEF8`, `0x800F3160` | deferred by direction | unchanged | Recorded near-matches from Phase 6; still not registered. |
|
||||
| Library vs game code | re-flag if evidence appears | **still unresolved** | An `exact` extent means "the reachable extent of a called entry", which is equally true of library and game code. |
|
||||
|
||||
## Verification and Firewall
|
||||
|
||||
- `make clean` exit 0; `make all` exit 0; `cmp` exit 0; both files SHA-1
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`.
|
||||
- `make check` exit 0: **168 tests pass** (82 added this phase); `regions=39 disagreements=0`
|
||||
(`result=AGREE`); `c_regions=39`, `differing_bytes=0`, `result=MATCH`.
|
||||
- Determinism: two `sf3_extents scan` runs are byte-identical; the worklist order is `(tier, size,
|
||||
address)` from tracked inputs alone.
|
||||
- Independent oracle: Ghidra's function bodies for `FUN_80017ad4` (`[80017ad4, 80017ae7]`) and
|
||||
`FUN_800f8f9c` (`[800f8f9c, 800f8fbf]`) equal the derived extents.
|
||||
- 121 tracked files, **0** under any prohibited root; `git diff --check` exit 0.
|
||||
- No game bytes, disassembly, generated assembly, build outputs, expected binaries, Ghidra material,
|
||||
dumps, or proprietary SDK material are tracked. The new tables hold addresses, sizes, grades, counts
|
||||
and group labels only.
|
||||
|
||||
## Rules Added This Phase
|
||||
|
||||
None.
|
||||
|
||||
## Next
|
||||
|
||||
Phase 7 is closed. A future phase must begin in a fresh session with an approved task-by-task plan. It
|
||||
should preserve the all-payload data baseline, the ordered-layout harness, the corrected toolchain, the
|
||||
extents/census/worklist evidence tables, and the byte gate, and must treat the following as unresolved:
|
||||
`0x8005DEF8`, `0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes (GTE, `syscall`,
|
||||
`move v0,sp`), the numeric `-G`, the CRT entry, and library-versus-game-code.
|
||||
|
||||
## Plain-English Recap
|
||||
|
||||
Phase 7 attacked the reason matching did not scale. Phase 6 could say where a function *started* but
|
||||
every function's *end* was worked out by hand, and nobody knew how many bodies the executable shares
|
||||
with itself. The phase built three pieces of evidence — control-flow-derived extents, a duplicate
|
||||
census, and a ranked worklist — and then used them to match twenty-three more functions, taking the
|
||||
project from eleven to thirty-four and from twelve to thirty-nine registered regions. Along the way it
|
||||
found two mistakes in the earlier evidence model (a "prologue" pattern that appears inside real bodies,
|
||||
and a zero-filled region full of fake call targets), removed the need to hand-write a symbol row for
|
||||
every cross-reference, and recorded two new near-misses with the exact evidence that stopped them. The
|
||||
executable still rebuilds byte for byte.
|
||||
|
||||
## 🛑 Stop Here
|
||||
|
||||
`CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase7.md`; this PhaseEnd and the digest update
|
||||
belong in the closure commit. Do not begin Phase 8 execution in this session.
|
||||
Reference in New Issue
Block a user