phase5: activate phase and record toolchain evidence inventory

P5-T1: revalidated the clean payload-data baseline (28 tests, make clean/all,
cmp, SHA-1 e173426c...), reviewed the firewall and Git state, and inventoried the
USA toolchain-relevant codegen signatures and SDK version-marker provenance.
No ROM-derived bytes, strings, or listings are tracked.
This commit is contained in:
Christopher Williams
2026-09-23 19:51:47 -04:00
parent b9543e5213
commit ed2f249397
3 changed files with 202 additions and 0 deletions
+74
View File
@@ -0,0 +1,74 @@
# Phase 5 Toolchain-Evidence Inventory
**Scope:** bounded, static inventory of USA evidence relevant to identifying the original build toolchain.
**Task:** P5-T1
**Status:** complete; this is an inventory, not a toolchain identification or a C-match record.
## Purpose and limits
This record consolidates the codegen signatures already observed in Phases 3–4 with a
bounded static re-check of the validated USA executable. It identifies **what the
original toolchain must reproduce**, not **which toolchain it was**. No compiler,
assembler, linker, optimization level, ABI, or library set is selected here.
## Provenance of sources
| Source | Identity | Basis |
|---|---|---|
| Validated USA executable | `SCUS_946.40;1`, 1,886,208 bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9` | Phase 1 header validation; re-hashed in P5-T1. |
| Entry range | `[0x800FB368, 0x800FB410)` | Header entry PC plus the `jal 0x800FB410` call target (Phase 4 dossier). |
| Local MIPS disassembler | `mipsel-none-elf-binutils` 2.46.0 `objdump`, `march=r3000` decode | Phase 3 local tool; raw output kept in ignored `.run/p5-t1/`. |
| Loader version detector | `ghidra_psx_ldr` source `src/main/java/psyq/DetectPsyQ.java` and `src/main/java/psx/PsxLoader.java` | Loader source tree at `/media/christopherwilliams/Data/Projects/decomp/ghidra_psx_ldr`. |
| Phase 3/4 records | `docs/PHASE3_BUILD.md`, `docs/PHASE4_*` | Prior phase evidence, carried with their stated limits. |
## Observed codegen signatures
All rows describe the validated USA main executable. They are expressed as mnemonic
idioms, not instruction listings; no instruction bytes are reproduced here.
| # | Signature | Basis | Limit |
|---|---|---|---|
| 1 | Absolute addresses are materialized as `lui` (high half) + `addiu` (low half). | Entry range; independently reported in Phase 4 (`PHASE4_C_REPRESENTATION_CHECK.md`). | The clear-loop address low halves have their high bit clear, so `addiu`/`ori` are both encodable; this is a code-generation style, not a correctness requirement. Consistent with an older GCC-family compiler; not proof. |
| 2 | Setting the cached-region bit uses `lui` of `0x8000` into a scratch register followed by `or` with the low part. | Entry range (cached stack pointer construction). | A single idiom instance; does not identify the compiler. |
| 3 | Clearing the top three bits of a KSEG0 pointer uses a paired `sll`/`srl` by 3. | Entry range (physical address derivation). | One instance; not compiler-specific by itself. |
| 4 | `gp` is initialized from an absolute address with `lui` + `addiu`. | Entry range. | One instance; consistent with `-mno-abicalls`-style non-PIC code, not proof of flags. |
| 5 | Return address is spilled to an absolute global via `lui at,imm` + `sw ra,off(at)` rather than to the stack frame. | Entry range. | One instance; may be a startup-routine artifact rather than a general convention. |
| 6 | Branch delay slots are filled with `nop` when no independent instruction is available; `jal` delay slots are filled with the following instruction (for example an argument adjustment). | Entry range and a bounded sample of two callees. | Small sample; delay-slot filling policy differs by optimization level, so this does not pin an `-O` level. |
| 7 | `jr ra` is followed by `nop` in the delay slot. | Bounded sample of two callees. | Small sample. |
| 8 | Function frames use `addiu sp,sp,-N`, save registers at positive offsets, and place `ra` at `N-4(sp)`, with a mirrored epilogue. | Bounded sample of two callees. | Two-function sample; not a proof of the original ABI or frame rules. |
| 9 | Signed division by a power of two uses a `bgez` guard plus an `addiu` rounding constant and an arithmetic shift. | One sampled callee. | Single instance; an idiom shared across GCC 2.x-era MIPS code generators. |
| 10 | 32-bit multiplication uses `mult` + `mflo`. | One sampled callee. | Single instance; not discriminating between revisions. |
Rows 1–5 are confirmed directly from the entry range in P5-T1. Rows 6–10 come from a
bounded two-function sample and are recorded as low-confidence observations only.
## SDK / compiler version markers
| Marker | Finding | Basis | Limit |
|---|---|---|---|
| `PsyQ Version = 4.5.0` | Loader-derived, **not** an ASCII string. `DetectPsyQ.getPsyqVersion` searches an 8-byte masked binary signature and decodes a numeric version from bytes at offsets `+6..+7`; `PsxLoader.addPsyqVerOption` formats the digits. | Loader source (see provenance table). | Identifies an SDK library signature set present in the executable. It does **not** select a compiler revision, optimization level, flag set, or ABI. |
| ASCII compiler/SDK markers | **Not found** in the main executable. A bounded search for a fixed self-authored marker list (including compiler-family, assembler-name, and SDK-name spellings) returned no match. | Bounded scan of the validated executable; raw hit log kept in ignored `.run/p5-t1/marker-hits.txt`. | The marker list is not exhaustive. Absence of these spellings does not exclude any compiler; it only removes one class of easy evidence. |
| Standard header copyright | The PS-X EXE header region contains the standard Sony copyright field; this is a fixed header field, not a compiler marker. | Header layout; the header region is not part of the loadable payload. | No toolchain information. A second occurrence of the vendor token exists in the payload; its meaning is unestablished and it is not treated as evidence. |
| Overlay/archive material | The extracted tree contains several archive candidates (for example `TITLE.HOG`, `XLOAD.HOG`, `FOG`, `FOG2`, `MINIGAME`, `MPLAYER`, `MPTITLE*`). | Phase 1 extraction manifest; Phase 2 investigation. | Phase 2 established **no** validated archive/overlay extraction model, so no overlay compiler strings are currently recoverable. These archives are not evidence about the toolchain. |
## What remains unestablished
- The compiler front end, its revision, and its patch level.
- The assembler (whether a GNU assembler or an SDK-specific assembler such as a
Maspsx-compatible one) and its accepted syntax.
- The linker and its layout behavior beyond the Phase 4 address-ordered reconstruction.
- Optimization level and the full flag set.
- ABI details, library set, and the exact SDK revision behind the `4.5.0` signature.
## Comparator requirements carried forward
Any future candidate toolchain must reproduce, at minimum, the observed signatures above
and must be judged only through an instruction-identical comparison plus a green clean
full-binary gate against `e173426c157384ebf1b6caf8c6fea18a85a14af9`.
## Firewall note
No game bytes, disassembly listings, ROM-derived strings, or ROM-derived fixtures were
added to tracked files. Raw disassembly and marker-scan output remain in ignored
`.run/p5-t1/`. The only tracked additions for P5-T1 are this record and
`phase-ends/CURRENT_PHASE.md`.
+44
View File
@@ -0,0 +1,44 @@
# Current Phase — Phase 5
**Phase:** Phase 5 — Original Toolchain Identification and First C Match
**Plan:** `phase-ends/Phase5_PLAN.md`
**Status:** active; developer instructed "Begin phase 5" (treated as plan approval — see note below)
**Active task:** P5-T1 complete — awaiting P5-T2 approval before toolchain acquisition
> Approval note: `Phase5_PLAN.md` was authored as a DRAFT requiring explicit developer
> approval. The developer's instruction to begin Phase 5 is being treated as that approval
> for control/validation/inventory work (P5-T1). P5-T2 acquires external toolchain
> material and therefore requires an explicit decision on which candidate path to take
> before any acquisition happens.
## Starting boundary (carried from Phase 4, verified not re-derived)
- Validated USA `SCUS_946.40;1`: 1,886,208 bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, payload `[0x80010000, 0x801DC000)`, entry PC `0x800FB368`.
- Tracked default build remains the Phase 3 all-payload **data** representation; it asserts no code/function/section model.
- Phase 4 established an ignored, address-ordered assembly recovery path that reproduces the executable exactly with the entry as a real MIPS `asm` subsegment: header `0x800`, payload prefix `0xEB368`, entry code `0xA8`, payload suffix `0xE0BF0`, non-payload metadata excluded during `objcopy`.
- Original compiler/assembler/linker/flags remain unidentified. No C match is claimed.
## P5-T1 completed evidence
### Baseline revalidation (clean state)
- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v` — exit 0; 28 synthetic-only tests passed.
- `make clean` — exit 0. `make all` — exit 0.
- `cmp -s build/scus_946_40.rebuilt "extracted/SCUS_946.40;1"` — exit 0; `cmp -l` reports 0 differing positions.
- Both files: 1,886,208 bytes and SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`.
- Git review: 46 tracked files, 0 tracked paths under prohibited roots (`disks/`, `extracted/`, `asm/`, `assets/`, `expected/`, `build/`, `ghidra/`, `dumps/`, `tools/psyq/`, `tools/splat/`, `tools/maspsx/`, `tools/mipsel-none-elf-binutils/`); `git diff --check` exit 0.
### Inventory
`docs/PHASE5_TOOLCHAIN_EVIDENCE.md` records the P5-T1 toolchain-evidence inventory: the
observed codegen signatures (each with basis and limit), the provenance of the
`PsyQ Version = 4.5.0` import value, the bounded negative ASCII marker search, and the
explicit list of what remains unestablished. No ROM-derived bytes, strings, or listings
were copied into tracked files.
## Next task
P5-T2 — candidate original toolchain identification and acquisition with provenance.
Blocked on an explicit developer decision between:
1. obtaining a PsyQ/GCC 4.5-era proprietary SDK locally (must stay ignored; never committed), or
2. the documented open GCC 2.x `mipsel-psx` alternative with its explicit limits.
+84
View File
@@ -0,0 +1,84 @@
# Phase 5 Plan — Original Toolchain Identification and First C Match
**Status:** ACTIVE — developer instructed "Begin phase 5", treated as approval of this plan for control/validation/inventory work. P5-T2 acquisition still requires an explicit candidate-path decision (proprietary PsyQ-era SDK vs. open GCC 2.x `mipsel-psx`).
**Planning effort:** Max
**Prepared:** 2026-09-23, immediately after Phase 4 closure.
## Session start (for the new session)
Read in order before doing anything: `AGENTS.md`, `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `phase-ends/PhaseEnd_Phase4.md`, this plan, then task-relevant `docs/` records (`docs/PHASE4_ORDERED_LINK.md`, `docs/PHASE4_ENTRY_DOSSIER.md`, `docs/PHASE4_VERIFICATION.md`, `docs/SETUP.md`). There is no `CURRENT_PHASE.md` at the start of this phase; Phase 5 must create one when activated.
## Goal
Identify and verify the original Syphon Filter 3 (USA) build toolchain — compiler, assembler, linker, flags, and ABI — using direct byte evidence, then use that verified toolchain to attempt the project's first instruction-identical C function match, gated by the clean full-binary hash check. If the toolchain cannot be reproduced, record the bounded blocker and keep all content as assembly/data fallback.
## Carried context (do not re-derive; verify before relying on)
- Validated USA `SCUS_946.40;1`: 1,886,208 bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, payload `[0x80010000, 0x801DC000)`, entry PC `0x800FB368`.
- Phase 3 baseline is a byte-identical all-payload **data** representation; it asserts no code/function/section model.
- Phase 4 established an address-ordered assembly recovery path that reproduces the binary exactly with the entry present as real MIPS code. Ordering matters: Splat's generated linker script emits `.text` before `.data` and adds 16 bytes of alignment padding; the fixed layout is header `0x800`, payload prefix `0xEB368`, entry code `0xA8`, payload suffix `0xE0BF0`, with non-payload metadata excluded during `objcopy`.
- Entry boundary evidence: start `0x800FB368` (header + loader entry); exclusive end `0x800FB410` (itself a `jal` target); callee `0x80029ED8` is a Ghidra-missed adjacent function start; no duplicate entry prologue found.
- Nine entry relocations have direct Ghidra provenance; the generated `D_80000004` label is unsupported and must not be defined.
- `PsyQ Version = 4.5.0` is an **SDK-level import string**, not a compiler revision. It must not be treated as compiler evidence.
- Clang 22.1.8 was only a deterministic **control**, never a candidate original compiler. No MIPS GCC is currently installed.
- Key fingerprint hint from Phase 4: the original materializes absolute addresses as `lui` + `addiu`, while Clang emits `lui` + `ori`. This is consistent with an older GCC-family compiler but is not proof.
## Preconditions
- Phase 3 payload-data baseline and the Phase 4 ordered assembly recovery path both remain reproducible.
- The preserved Ghidra imports (`/SCUS_946.40;1` and `/P2-analysis-SCUS_946.40;1`) remain available and unmodified unless a task explicitly says otherwise.
- All ROM-derived material stays in ignored paths.
## Scope and safeguards
- **ROM/SDK firewall:** disc dumps, extracted content, disassembly, generated assembly, build outputs, expected binaries, Ghidra DBs, RAM dumps, and **proprietary Sony SDK/PsyQ toolchains or libraries** must never be committed. Keep them ignored and record provenance and checksums in `docs/SETUP.md`.
- Do not present assumptions about the original compiler, ABI, flags, or behavior as facts. Ghidra is the static oracle; PCSX-Redux is the runtime oracle.
- Do not select a toolchain from a version label alone; selection requires byte/signature evidence.
- Preserve the payload-data fallback for every unresolved byte. Any C enters only behind an explicit `NON_MATCHING`/fallback guard and only after an instruction-identical comparison plus a green clean full-binary gate.
- Check for duplicates before matching; share a verified body only through a documented source/registry mechanism.
- One task at a time. Explain any non-trivial code or design change before writing it.
- Never use `git clean -x` or `git clean -fdx`. Use `make clean` for regenerable `build/` output only.
- Do not infer segmentation, symbols, or toolchain behavior from another project.
## Tasks
- [x] **P5-T1 — Phase control records, baseline revalidation, and toolchain-evidence inventory** *(xHigh)* — complete; see `phase-ends/CURRENT_PHASE.md` and `docs/PHASE5_TOOLCHAIN_EVIDENCE.md`.
- Create `phase-ends/CURRENT_PHASE.md` and re-run the documented clean baseline, synthetic suite, firewall checks, and Git review.
- Inventory existing USA evidence relevant to toolchain identification: static codegen signatures already observed (address materialization style, delay-slot use, frame conventions, call patterns), the Phase 3/4 records, and any SDK/compiler version strings recoverable statically from the executable or overlays.
- **Verify:** baseline gates pass (28 tests; `make clean`, `make all`, `cmp`, SHA-1 all green); every inventory fact has provenance, region/build source, and a stated limit; no ROM-derived strings, bytes, or listings are copied into tracked files.
- [ ] **P5-T2 — Candidate original toolchain identification and acquisition with provenance** *(xHigh)*
- Identify candidate PsyQ/GCC toolchain revision(s) consistent with `PsyQ Version = 4.5.0` and the P5-T1 evidence, including expected compiler front-end, assembler, linker, and library set.
- Acquire candidates into ignored `tools/` with recorded source, checksums, and provenance; never install system-wide; never commit proprietary SDK material.
- If a proprietary SDK cannot be safely obtained, document a matching open GCC 2.x `mipsel-psx` alternative and its explicit limits.
- **Verify:** provenance recorded in `docs/SETUP.md`; toolchain and all SDK files confirmed ignored; candidate assembler accepts the required PSX COP2/GTE syntax; no prohibited artifact is tracked.
- [ ] **P5-T3 — Byte-evidence compiler/assembler/linker fingerprint** *(xHigh)*
- Use small, self-authored probe functions to test candidate versions and flags against observed original signatures, including the `lui`+`addiu` low-half materialization, the BSS-clear loop idiom, branch/delay-slot filling, and frame layout.
- Prefer discriminating probes (e.g. low-half values with and without the high bit set) that separate compiler families and revisions.
- Record negative as well as positive results; do not select a toolchain from a version label.
- **Verify:** each result records exact commands, resolved tool versions, and comparison evidence; a candidate is only recorded as "selected" if it reproduces the observed signatures; the selected behavior is reproducible from a clean probe build.
- [ ] **P5-T4 — Reproducible instruction-range comparator harness** *(xHigh)*
- Add tracked, synthetic-tested tooling that compiles a C candidate with the selected toolchain/flags, extracts the exact instruction range, compares it byte-for-byte against the validated USA range, and then runs the clean full-binary gate through the ordered workflow.
- The harness must accept explicit input/output paths, refuse unsafe/existing destinations, keep all ROM-derived inputs ignored, and emit exit codes rather than relying on printed text.
- **Verify:** synthetic self-tests pass; the harness fails safely on a deliberately incorrect synthetic input; it contains no ROM bytes or listings; a documented invocation is recorded.
- [ ] **Rules check**
- Re-read `AGENTS.md` mandatory behavior after P5-T4 and state: `Rules check — re-read complete. Continuing with [next task].`
- [ ] **P5-T5 — Isolated first C-matching experiment for the entry** *(xHigh)*
- Write at most one C candidate for the entry routine behind an explicit `NON_MATCHING`/default-fallback guard; compile with the selected toolchain and flags; compare its exact `[0x800FB368, 0x800FB410)` instruction stream against the validated USA range.
- Inspect references and potential duplicates before matching; supply only the nine evidenced relocations; never define the rejected `0x80000004` label.
- Iterate only on comparison-supported root causes; stop after two distinct unexplained failures.
- **Verify:** the direct instruction-identical comparison AND the clean full-binary `cmp` + SHA-1 check both pass before C is enabled by default. On failure, keep fallback, record the bounded result, and do not claim a match.
- [ ] **P5-T6 — Cookbook, registry, verification record, and phase gate** *(xHigh)*
- If P5-T5 succeeds: record reusable compiler/codegen findings in a matching cookbook, add ROM-safe registration and verification conventions for matched functions, and share verified duplicate bodies through the documented mechanism.
- Otherwise: document the evidence-backed blocker.
- Complete the clean test, baseline/full-binary comparison, firewall, and Git-review gates, then request milestone confirmation.
- **Verify:** every match claim has direct instruction and clean full-binary evidence; unmatched material remains fallback; no prohibited artifact is tracked; developer confirmation is requested before any PhaseEnd.
## Milestone
The original build toolchain is identified with byte evidence and used to produce at least one instruction-identical C function match whose clean full-binary rebuild passes exact `cmp` and SHA-1, **or** Phase 5 records the evidence-backed blocker that prevents a reproducible toolchain and leaves all content as assembly/data fallback. Phase 5 closes only after the developer confirms the milestone; write a PhaseEnd record, update `phase-ends/DIGEST.md`, archive `CURRENT_PHASE.md` to `phase-ends/logs/Phase5.md`, commit, and stop without beginning Phase 6.