christopher/Syphon_Filter_3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(phase-1): close extraction and import milestone

Browse Source
This commit is contained in:
Christopher Williams
2026-09-23 15:03:12 -04:00
parent 04331bd5b4
commit 1dd720d998
6 changed files with 208 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PHASE1_REPRODUCIBILITY.md
+98
View File
@@ -0,0 +1,98 @@
# Phase 1 Reproducibility Record
**Scope:** deterministic extraction, PS-X EXE characterization, and first Ghidra import for the local USA input.
**Status:** task verification complete; Phase 1 remains open pending developer milestone confirmation.
This record intentionally contains commands and non-content metadata only. The disc image, extracted files, manifests, Ghidra project/database, and local reports remain ignored.
## Preconditions
- Use the local USA input identified in [disc-image.md](disc-image.md).
- Run from the repository root on the verified native writable filesystem.
- The tracked tool is `tools/sf3_extract`; it has no third-party Python dependency.
- Ghidra 12.1.2 and the locally built `ghidra_psx_ldr` extension are described in [SETUP.md](SETUP.md).
## Tool verification
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v
```
The final Phase 1 suite passed 19 synthetic-only tests. It covers ISO9660/MODE2 validation, deterministic extraction, transactional failure handling, multi-extent copying, manifest SHA-1 verification, and PS-X EXE magic/bounds/manifest checks. No test reads the local disc or extracted game data.
## Deterministic filesystem extraction
The output directory must not already exist; the extractor refuses to overwrite it.
```bash
./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted
```
The tool preserves exact ISO9660 identifiers, including `;1`, and writes an ignored `extracted/MANIFEST.tsv`. Manifest columns are `path`, first `lba`, complete `extents`, `size`, and `sha1`.
### Verified result
Two serial fresh runs produced byte-identical manifests with:
- 96 regular files;
- 601,671,468 total extracted bytes; and
- a successful independent SHA-1 rehash of all 96 final files against the final manifest.
For the verified same-destination rerun, the first `extracted/` directory was first confirmed to be a non-symlink, ignored, tool-generated directory containing its manifest. Its manifest was retained only under ignored `.run/`, then that known generated directory was removed before the second run. A safer future comparison can instead use two fresh ignored child destinations and compare only their manifests:
```bash
./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted/run-a
./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted/run-b
cmp -s extracted/run-a/MANIFEST.tsv extracted/run-b/MANIFEST.tsv
```
Do not stage either output tree or manifest.
## PS-X EXE characterization
```bash
./tools/sf3_extract psx-exe-info \
'extracted/SCUS_946.40;1' \
extracted/MANIFEST.tsv > .run/p1-t3-psx-exe-info.tsv
```
The command validates the PS-X EXE magic and 0x800-byte header, declared text bounds, extracted file size, and streaming SHA-1 against the manifest. The candidate passed with these non-content results:
| Property | Value |
|---|---|
| SHA-1 | `e173426c157384ebf1b6caf8c6fea18a85a14af9` |
| File size | 1,886,208 bytes |
| Text load address / size | `0x80010000` / `0x001CC000` |
| Entry PC | `0x800FB368` |
| Header GP | `0x00000000` |
| Header SP base / offset | `0x801FFFF0` / `0x00000000` |
These are header declarations only. They do not establish runtime register state or runtime address allocation. See [formats.md](formats.md) and [memory-map.md](memory-map.md) for the detailed provenance-aware record.
## First Ghidra import
The developer created an empty local project under ignored `ghidra/` and connected Ghidra MCP. The following MCP import request was issued against the extracted candidate:
- tool: `ghidra_import_file`;
- `file_path`: `extracted/SCUS_946.40;1`;
- `program_name`: `SCUS_946.40;1`;
- `open_after_import`: `true`;
- `suppress_analysis_prompt`: `true`; and
- no forced raw language, base address, compiler, or loader.
Ghidra automatically selected `PSX Executables Loader`, producing language `PSX:LE:32:default`, compiler spec `default`, image base `0x80000000`, 31 static loader-created blocks, and entry point `start` at `0x800FB368`. The developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze. The analyzer's displayed `4.7.0` fallback was not used as SDK evidence.
Auto Analyze remains unrun: the program is initialized, not analyzing, and has an analyzed flag of false. The ignored local import record is `ghidra/P1-T4-import.md`. Ghidra MCP's Java script provider could not read Program Information directly because of a `GhidraPlaceholderBundle`/`GhidraSourceBundle` class-cast failure; its temporary diagnostic script was removed. This limitation does not invalidate the developer's manual Program Information observation.
## Remaining unknowns
- Disc track count/boundaries outside the ISO filesystem.
- Archive, compression, audio, video, and overlay formats.
- Runtime loader behavior, overlay allocation, and all runtime memory observations.
- Whether `PsyQ 4.5.0` alone identifies the original compiler, compiler revision, flags, assembler, linker layout, or library set.
- Full matching toolchain and all-assembly rebuild baseline.
- Any function behavior or match status; no C matching has begun.
## Firewall check
Before closing this task, `git diff --cached --quiet` passed. `extracted/`, `.run/` reports, and `ghidra/` import records were confirmed ignored. The local `.mcp.json` configuration remains untracked and is intentionally excluded from project commits.
docs/formats.md
+6
View File
@@ -35,3 +35,9 @@ The exact ISO9660 identifier `SCUS_946.40;1` from the local USA image was extrac
| SP offset header field | 0x34 | `0x00000000` |
These are static header declarations from the USA file, not PCSX-Redux observations. In particular, the zero GP and SP-offset values do not establish post-loader register behavior, and the declared text range is not yet a runtime memory-map claim.
## Ghidra PSX-loader import (Phase 1 T4)
The extracted USA file was imported into the ignored local Ghidra project without forcing a raw language, base address, compiler, or loader selection. Ghidra selected `PSX Executables Loader` and reports language `PSX:LE:32:default`, compiler spec `default`, image base `0x80000000`, and entry point `0x800FB368` (`start`).
Before Auto Analyze was run, the developer manually read `PsyQ Version = 4.5.0` from the active program's Program Information. This is loader/import evidence for the local USA program, not a runtime observation or a final compiler identification. The registered `PsyQ Signatures` analyzer has a separate fallback option of `4.7.0`; that default was not used as evidence. The post-import analysis state is initialized but not analyzing, with the analyzed flag false. Local import settings and the MCP limitation record are preserved in ignored `ghidra/P1-T4-import.md`.
docs/memory-map.md
+18
View File
@@ -13,4 +13,22 @@ These entries are header-declared values from the local USA image's exact ISO966
| `0x00000000` | Initial GP header field | Local USA image | PS-X EXE header offset 0x14 | Header-valid; not a runtime GP observation |
| `0x801FFFF0` + `0x00000000` | SP base and SP offset header fields | Local USA image | PS-X EXE header offsets 0x30/0x34 | Header-valid; no effective/runtime SP inference made |
## Ghidra PSX-loader static layout (Phase 1 T4)
The following is the loader-created static program layout for the local USA import. It is not a PCSX-Redux memory observation and does not establish overlay/runtime allocation behavior.
| Address range / field | Purpose | Source build / region | Evidence | Verification status |
|---|---|---|---|---|
| `0x1F800000`–`0x1F801DBF` (22 discrete blocks) | PSX cache, controller, timer, DMA, CD-ROM, GPU, MDEC, and SPU mappings | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x80000000`–`0x8000FFFF` | Initial `RAM` block, RWX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x80010000`–`0x8010B50F` | `.text`, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x8010B510`–`0x8010EEFF` | `.rdata`, R | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x8010EF00`–`0x80121937` | `.data`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x80121938`–`0x801221D7` | `.sdata`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x801221D8`–`0x801227AB` | `.sbss`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x801227AC`–`0x8014694F` | `.bss`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x80146950`–`0x801DBFFF` | Second `.text` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x801DC000`–`0x801FFFFF` | Final `RAM` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
| `0x800FB368` | Entry point `start` | Local USA image | Ghidra entry-point report after PSX-loader import | Static loader entry; runtime execution unverified |
Record each future entry with: address range, purpose, source build/region, evidence source (Ghidra or runtime observation), and verification status. Do not transfer addresses from other Syphon Filter releases without explicit USA validation.
phase-ends/DIGEST.md
+4
View File
@@ -3,3 +3,7 @@
## Phase 0 — Governance and Disc Characterization (2026-09-23)
A ROM-safe Git baseline, permanent project context, setup/format/address documents, and phase-state workflow were created. Ghidra 12.1.2 with a locally built PSX loader and PCSX-Redux are ready. The local USA image is ignored and identified as a 691,530,336-byte MODE2/2352 image (`4abe30077c2b449ea68239083df7932d47ae0b69`) with 2048-byte user data at offset 24 and an ISO9660 PVD at sector 16 (`SCUS94640`, 294,018 blocks). Root entry `SCUS_946.40;1` at LBA 24 is the executable candidate for Phase 1 header validation. No CUE is required for filesystem extraction; it remains desirable for track metadata and later disc work. The approved Phase 1 plan is `phase-ends/Phase1_PLAN.md`. No rules were added.
## Phase 1 — Deterministic Extraction, Executable Discovery, and First Ghidra Import (2026-09-23)
A tracked synthetic-tested MODE2/2352 ISO9660 walker/extractor now produces ignored deterministic file trees and SHA-1 manifests. Two fresh runs produced 96 files and 601,671,468 bytes with byte-identical manifests; all final files re-hashed against the manifest. `SCUS_946.40;1` is header-validated as a PS-X EXE with SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. Ghidra imported it with `PSX Executables Loader` and language `PSX:LE:32:default`; the developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze. All memory-layout and SDK findings remain static import evidence, not runtime facts. The 19-test suite, firewall checks, and Phase 1 reproducibility record are complete. Phase 2 must begin in a fresh session with an approved plan; runtime loader/overlay/archive behavior remains unknown.
phase-ends/PhaseEnd_Phase1.md
+72
View File
@@ -0,0 +1,72 @@
# PhaseEnd — Phase 1: Deterministic Extraction, Executable Discovery, and First Ghidra Import
**Date:** 2026-09-23
**Phase Status:** Complete
**Milestone confirmed by developer:** yes
## Completed Checklist
- P1-T1 — Added executable, dependency-free `tools/sf3_extract` ISO9660/MODE2 walker with synthetic-only validation tests.
- P1-T2 — Added deterministic transactional extraction and ignored manifests; verified two fresh runs agree.
- P1-T3 — Validated the extracted executable candidate as a PS-X EXE and matched its streaming SHA-1 to the extraction manifest.
- P1-T4 — Imported the candidate into an ignored local Ghidra project using `PSX Executables Loader`; recorded static loader findings.
- Rules check — Re-read `AGENTS.md` mandatory behavior after P1-T4.
- P1-T5 — Wrote `docs/PHASE1_REPRODUCIBILITY.md` with commands, results, firewall checks, and remaining unknowns.
## Verified Results
### Extraction
- The tracked synthetic-only suite passes 19 tests.
- Two fresh extraction runs produced byte-identical manifests with 96 regular files and 601,671,468 total extracted bytes.
- All 96 final extracted files were independently re-hashed against the final manifest.
- Exact ISO9660 identifiers are preserved in ignored `extracted/`; manifests, temporary reports, and the disc remain untracked.
### Executable
- Exact ISO identifier: `SCUS_946.40;1`.
- PS-X EXE header and manifest validation passed.
- File SHA-1: `e173426c157384ebf1b6caf8c6fea18a85a14af9`.
- Declared text range: `[0x80010000, 0x801DC000)`.
- Declared entry PC: `0x800FB368`.
- Header GP/SP values are recorded in `docs/formats.md` as header facts only; they are not runtime claims.
### Ghidra
- Ghidra automatically selected `PSX Executables Loader` for the extracted candidate.
- Program language: `PSX:LE:32:default`; compiler spec: `default`; image base: `0x80000000`.
- The loader created 31 static memory blocks and entry point `start` at `0x800FB368`.
- Before Auto Analyze, the developer manually observed Program Information `PsyQ Version = 4.5.0` for the USA import. The analyzer fallback `4.7.0` was not used as evidence.
- Auto Analyze remains unrun. All layout and SDK findings are static import evidence, not PCSX-Redux runtime observations.
- Local project/import data is ignored under `ghidra/`; static layout and import provenance are recorded in `docs/memory-map.md`, `docs/formats.md`, and local `ghidra/P1-T4-import.md`.
## Verification and Firewall
- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v` passed all 19 tests at close.
- `git diff --check` passed.
- `git diff --cached --quiet` passed before closure staging.
- `extracted/`, `.run/`, and `ghidra/` records were explicitly confirmed ignored.
- No disc data, extraction output, manifest, Ghidra database, disassembly, or proprietary SDK file was staged.
## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Auto Analyze | Preserve first import results and record SDK evidence | Left unrun | Avoided replacing a loader-detected PsyQ value with the analyzer's `4.7.0` fallback before evidence was recorded. |
| Program Information query | Read PsyQ field through MCP | Developer read the field manually | The MCP Ghidra Java script provider failed with a `GhidraPlaceholderBundle`/`GhidraSourceBundle` class-cast error. The temporary diagnostic script was removed. |
## Rules Added This Phase
None.
## Next
Phase 2 — runtime loader/overlay and archive-format investigation. It requires a new task-by-task plan, verification criteria, and explicit developer approval in a fresh session. Existing Phase 1 static findings must not be promoted to runtime facts without PCSX-Redux observation.
## Plain-English Recap
The disc can now be extracted repeatably without leaking game material into Git, its executable has a validated PS-X EXE header, and Ghidra has imported it with the intended PSX loader. The project has static addresses and a PsyQ import clue, but no runtime behavior, overlay model, archive format, or matching toolchain has been established yet.
## 🛑 Stop Here
Phase 1 is closed. `CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase1.md`; this PhaseEnd and the digest update are ready for the closure commit. Do **not** begin Phase 2 in this session. Start a fresh session by drafting and approving a Phase 2 plan.
phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase1.md
+10 -4
View File
@@ -8,9 +8,9 @@
- [x] **P1-T1 — ISO walker:** added executable, dependency-free `tools/sf3_extract` and synthetic-only `unittest` coverage. The read-only `list` command validates MODE2/2352 sector geometry, ISO9660 descriptors and directory records, extents/bounds, output-path components, and recursive directory extents before returning deterministic paths.
- [x] **P1-T2 — Deterministic extraction:** extended `tools/sf3_extract` with transactional `extract`; it preserves exact ISO identifiers, copies every validated extent from the same open image handle, and writes ignored `extracted/MANIFEST.tsv` with path, first LBA, complete extents, size, and SHA-1. Two fresh direct runs produced identical manifests and file counts.
- [x] **P1-T3 — Executable characterization:** added synthetic-tested `psx-exe-info` validation to `tools/sf3_extract`. Exact ISO file `SCUS_946.40;1` passed PS-X EXE magic/header/text-bound checks and a streaming SHA-1 match against `extracted/MANIFEST.tsv`; permitted metadata is recorded in `docs/formats.md` and static-only header declarations in `docs/memory-map.md`.
- [ ] **P1-T4 — Ghidra import:** import the extracted executable with `ghidra_psx_ldr` into ignored `ghidra/` and record loader findings.
- [ ] **Rules check:** re-read `AGENTS.md` mandatory behavior after P1-T4.
- [ ] **P1-T5 — Reproducibility record:** document commands, results, and remaining unknowns; confirm no ROM-derived material is staged.
- [x] **P1-T4 — Ghidra import:** imported exact file `SCUS_946.40;1` into the ignored local project with automatic `PSX Executables Loader` selection. Ghidra reports `PSX:LE:32:default`, image base `0x80000000`, 31 loader-created blocks, and entry `start` at `0x800FB368`; loader/import evidence and static layout are recorded in `docs/formats.md`, `docs/memory-map.md`, and ignored `ghidra/P1-T4-import.md`.
- [x] **Rules check:** mandatory behavior in `AGENTS.md` re-read after P1-T4.
- [x] **P1-T5 — Reproducibility record:** added `docs/PHASE1_REPRODUCIBILITY.md` with safe commands, verification results, Ghidra import provenance, firewall behavior, and remaining unknowns. The final 19-test suite passed and no ROM-derived material is staged.
## Session checkpoint
@@ -20,4 +20,10 @@ P1-T1 is complete. `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s to
P1-T2 is complete. The host path is on native writable btrfs. The expanded synthetic-only suite passed 14 tests, including transactional extraction, multi-extent copying, refusal of a corrupt data sector and existing output, CLI extraction, and two fresh synthetic runs. Two serial fresh runs of `./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted` produced 96 files and 601,671,468 bytes each; their ignored manifests were byte-identical. All 96 final extracted files were re-hashed against the final manifest successfully. The first-run manifest is retained only in ignored `.run/p1-t2-manifest-run1.tsv`; the second direct run remains in ignored `extracted/`. No generated material is staged.
P1-T3 is complete. The 19-test synthetic-only suite covers PS-X EXE parsing plus bad magic, declared-text-bound, and manifest-hash rejection. `SCUS_946.40;1` passed the header and manifest checks with file SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. The GP/SP values in `docs/formats.md` are raw header fields only; no Ghidra import or runtime observation has occurred. The ignored metadata report is `.run/p1-t3-psx-exe-info.tsv`. P1-T4 has not started.
P1-T3 is complete. The 19-test synthetic-only suite covers PS-X EXE parsing plus bad magic, declared-text-bound, and manifest-hash rejection. `SCUS_946.40;1` passed the header and manifest checks with file SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. The GP/SP values in `docs/formats.md` are raw header fields only; no Ghidra import or runtime observation had occurred at T3 completion. The ignored metadata report is `.run/p1-t3-psx-exe-info.tsv`.
P1-T4 is complete. Ghidra automatically selected `PSX Executables Loader`; the imported USA program is open with language `PSX:LE:32:default`, default compiler spec, image base `0x80000000`, 31 static loader-created memory blocks, and entry point `start` at `0x800FB368`. The developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze; the analyzer fallback `4.7.0` was not used as evidence. Auto Analysis remains unrun. Local import settings/logs are in ignored `ghidra/P1-T4-import.md`. The MCP script provider failed to query Program Information directly and its temporary script was removed. No Ghidra database or ROM-derived output is staged.
Rules check — re-read complete. Continuing with P1-T5.
P1-T5 is complete. `docs/PHASE1_REPRODUCIBILITY.md` records the commands, extraction counts, executable metadata, Ghidra-loader results, firewall checks, and remaining unknowns without copying game data. The final 19-test suite passed; `git diff --cached --quiet` passed; and `extracted/`, `.run/`, and `ghidra/` records remain ignored. All Phase 1 checklist items are complete and ready for developer milestone confirmation. Do not write a PhaseEnd record, update the digest, or begin Phase 2 until the developer confirms the Phase 1 milestone.