diff --git a/docs/PHASE1_REPRODUCIBILITY.md b/docs/PHASE1_REPRODUCIBILITY.md new file mode 100644 index 0000000..cf30aa1 --- /dev/null +++ b/docs/PHASE1_REPRODUCIBILITY.md @@ -0,0 +1,98 @@ +# Phase 1 Reproducibility Record + +**Scope:** deterministic extraction, PS-X EXE characterization, and first Ghidra import for the local USA input. +**Status:** task verification complete; Phase 1 remains open pending developer milestone confirmation. + +This record intentionally contains commands and non-content metadata only. The disc image, extracted files, manifests, Ghidra project/database, and local reports remain ignored. + +## Preconditions + +- Use the local USA input identified in [disc-image.md](disc-image.md). +- Run from the repository root on the verified native writable filesystem. +- The tracked tool is `tools/sf3_extract`; it has no third-party Python dependency. +- Ghidra 12.1.2 and the locally built `ghidra_psx_ldr` extension are described in [SETUP.md](SETUP.md). + +## Tool verification + +```bash +PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v +``` + +The final Phase 1 suite passed 19 synthetic-only tests. It covers ISO9660/MODE2 validation, deterministic extraction, transactional failure handling, multi-extent copying, manifest SHA-1 verification, and PS-X EXE magic/bounds/manifest checks. No test reads the local disc or extracted game data. + +## Deterministic filesystem extraction + +The output directory must not already exist; the extractor refuses to overwrite it. + +```bash +./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted +``` + +The tool preserves exact ISO9660 identifiers, including `;1`, and writes an ignored `extracted/MANIFEST.tsv`. Manifest columns are `path`, first `lba`, complete `extents`, `size`, and `sha1`. + +### Verified result + +Two serial fresh runs produced byte-identical manifests with: + +- 96 regular files; +- 601,671,468 total extracted bytes; and +- a successful independent SHA-1 rehash of all 96 final files against the final manifest. + +For the verified same-destination rerun, the first `extracted/` directory was first confirmed to be a non-symlink, ignored, tool-generated directory containing its manifest. Its manifest was retained only under ignored `.run/`, then that known generated directory was removed before the second run. A safer future comparison can instead use two fresh ignored child destinations and compare only their manifests: + +```bash +./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted/run-a +./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted/run-b +cmp -s extracted/run-a/MANIFEST.tsv extracted/run-b/MANIFEST.tsv +``` + +Do not stage either output tree or manifest. + +## PS-X EXE characterization + +```bash +./tools/sf3_extract psx-exe-info \ + 'extracted/SCUS_946.40;1' \ + extracted/MANIFEST.tsv > .run/p1-t3-psx-exe-info.tsv +``` + +The command validates the PS-X EXE magic and 0x800-byte header, declared text bounds, extracted file size, and streaming SHA-1 against the manifest. The candidate passed with these non-content results: + +| Property | Value | +|---|---| +| SHA-1 | `e173426c157384ebf1b6caf8c6fea18a85a14af9` | +| File size | 1,886,208 bytes | +| Text load address / size | `0x80010000` / `0x001CC000` | +| Entry PC | `0x800FB368` | +| Header GP | `0x00000000` | +| Header SP base / offset | `0x801FFFF0` / `0x00000000` | + +These are header declarations only. They do not establish runtime register state or runtime address allocation. See [formats.md](formats.md) and [memory-map.md](memory-map.md) for the detailed provenance-aware record. + +## First Ghidra import + +The developer created an empty local project under ignored `ghidra/` and connected Ghidra MCP. The following MCP import request was issued against the extracted candidate: + +- tool: `ghidra_import_file`; +- `file_path`: `extracted/SCUS_946.40;1`; +- `program_name`: `SCUS_946.40;1`; +- `open_after_import`: `true`; +- `suppress_analysis_prompt`: `true`; and +- no forced raw language, base address, compiler, or loader. + +Ghidra automatically selected `PSX Executables Loader`, producing language `PSX:LE:32:default`, compiler spec `default`, image base `0x80000000`, 31 static loader-created blocks, and entry point `start` at `0x800FB368`. The developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze. The analyzer's displayed `4.7.0` fallback was not used as SDK evidence. + +Auto Analyze remains unrun: the program is initialized, not analyzing, and has an analyzed flag of false. The ignored local import record is `ghidra/P1-T4-import.md`. Ghidra MCP's Java script provider could not read Program Information directly because of a `GhidraPlaceholderBundle`/`GhidraSourceBundle` class-cast failure; its temporary diagnostic script was removed. This limitation does not invalidate the developer's manual Program Information observation. + +## Remaining unknowns + +- Disc track count/boundaries outside the ISO filesystem. +- Archive, compression, audio, video, and overlay formats. +- Runtime loader behavior, overlay allocation, and all runtime memory observations. +- Whether `PsyQ 4.5.0` alone identifies the original compiler, compiler revision, flags, assembler, linker layout, or library set. +- Full matching toolchain and all-assembly rebuild baseline. +- Any function behavior or match status; no C matching has begun. + +## Firewall check + +Before closing this task, `git diff --cached --quiet` passed. `extracted/`, `.run/` reports, and `ghidra/` import records were confirmed ignored. The local `.mcp.json` configuration remains untracked and is intentionally excluded from project commits. diff --git a/docs/formats.md b/docs/formats.md index acde95d..8fd7c24 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -35,3 +35,9 @@ The exact ISO9660 identifier `SCUS_946.40;1` from the local USA image was extrac | SP offset header field | 0x34 | `0x00000000` | These are static header declarations from the USA file, not PCSX-Redux observations. In particular, the zero GP and SP-offset values do not establish post-loader register behavior, and the declared text range is not yet a runtime memory-map claim. + +## Ghidra PSX-loader import (Phase 1 T4) + +The extracted USA file was imported into the ignored local Ghidra project without forcing a raw language, base address, compiler, or loader selection. Ghidra selected `PSX Executables Loader` and reports language `PSX:LE:32:default`, compiler spec `default`, image base `0x80000000`, and entry point `0x800FB368` (`start`). + +Before Auto Analyze was run, the developer manually read `PsyQ Version = 4.5.0` from the active program's Program Information. This is loader/import evidence for the local USA program, not a runtime observation or a final compiler identification. The registered `PsyQ Signatures` analyzer has a separate fallback option of `4.7.0`; that default was not used as evidence. The post-import analysis state is initialized but not analyzing, with the analyzed flag false. Local import settings and the MCP limitation record are preserved in ignored `ghidra/P1-T4-import.md`. diff --git a/docs/memory-map.md b/docs/memory-map.md index f6f6672..1dd9179 100644 --- a/docs/memory-map.md +++ b/docs/memory-map.md @@ -13,4 +13,22 @@ These entries are header-declared values from the local USA image's exact ISO966 | `0x00000000` | Initial GP header field | Local USA image | PS-X EXE header offset 0x14 | Header-valid; not a runtime GP observation | | `0x801FFFF0` + `0x00000000` | SP base and SP offset header fields | Local USA image | PS-X EXE header offsets 0x30/0x34 | Header-valid; no effective/runtime SP inference made | +## Ghidra PSX-loader static layout (Phase 1 T4) + +The following is the loader-created static program layout for the local USA import. It is not a PCSX-Redux memory observation and does not establish overlay/runtime allocation behavior. + +| Address range / field | Purpose | Source build / region | Evidence | Verification status | +|---|---|---|---|---| +| `0x1F800000`–`0x1F801DBF` (22 discrete blocks) | PSX cache, controller, timer, DMA, CD-ROM, GPU, MDEC, and SPU mappings | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x80000000`–`0x8000FFFF` | Initial `RAM` block, RWX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x80010000`–`0x8010B50F` | `.text`, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x8010B510`–`0x8010EEFF` | `.rdata`, R | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x8010EF00`–`0x80121937` | `.data`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x80121938`–`0x801221D7` | `.sdata`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x801221D8`–`0x801227AB` | `.sbss`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x801227AC`–`0x8014694F` | `.bss`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x80146950`–`0x801DBFFF` | Second `.text` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x801DC000`–`0x801FFFFF` | Final `RAM` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified | +| `0x800FB368` | Entry point `start` | Local USA image | Ghidra entry-point report after PSX-loader import | Static loader entry; runtime execution unverified | + Record each future entry with: address range, purpose, source build/region, evidence source (Ghidra or runtime observation), and verification status. Do not transfer addresses from other Syphon Filter releases without explicit USA validation. diff --git a/phase-ends/DIGEST.md b/phase-ends/DIGEST.md index ee94008..b9617d0 100644 --- a/phase-ends/DIGEST.md +++ b/phase-ends/DIGEST.md @@ -3,3 +3,7 @@ ## Phase 0 — Governance and Disc Characterization (2026-09-23) A ROM-safe Git baseline, permanent project context, setup/format/address documents, and phase-state workflow were created. Ghidra 12.1.2 with a locally built PSX loader and PCSX-Redux are ready. The local USA image is ignored and identified as a 691,530,336-byte MODE2/2352 image (`4abe30077c2b449ea68239083df7932d47ae0b69`) with 2048-byte user data at offset 24 and an ISO9660 PVD at sector 16 (`SCUS94640`, 294,018 blocks). Root entry `SCUS_946.40;1` at LBA 24 is the executable candidate for Phase 1 header validation. No CUE is required for filesystem extraction; it remains desirable for track metadata and later disc work. The approved Phase 1 plan is `phase-ends/Phase1_PLAN.md`. No rules were added. + +## Phase 1 — Deterministic Extraction, Executable Discovery, and First Ghidra Import (2026-09-23) + +A tracked synthetic-tested MODE2/2352 ISO9660 walker/extractor now produces ignored deterministic file trees and SHA-1 manifests. Two fresh runs produced 96 files and 601,671,468 bytes with byte-identical manifests; all final files re-hashed against the manifest. `SCUS_946.40;1` is header-validated as a PS-X EXE with SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. Ghidra imported it with `PSX Executables Loader` and language `PSX:LE:32:default`; the developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze. All memory-layout and SDK findings remain static import evidence, not runtime facts. The 19-test suite, firewall checks, and Phase 1 reproducibility record are complete. Phase 2 must begin in a fresh session with an approved plan; runtime loader/overlay/archive behavior remains unknown. diff --git a/phase-ends/PhaseEnd_Phase1.md b/phase-ends/PhaseEnd_Phase1.md new file mode 100644 index 0000000..036e255 --- /dev/null +++ b/phase-ends/PhaseEnd_Phase1.md @@ -0,0 +1,72 @@ +# PhaseEnd — Phase 1: Deterministic Extraction, Executable Discovery, and First Ghidra Import + +**Date:** 2026-09-23 +**Phase Status:** Complete +**Milestone confirmed by developer:** yes + +## Completed Checklist + +- P1-T1 — Added executable, dependency-free `tools/sf3_extract` ISO9660/MODE2 walker with synthetic-only validation tests. +- P1-T2 — Added deterministic transactional extraction and ignored manifests; verified two fresh runs agree. +- P1-T3 — Validated the extracted executable candidate as a PS-X EXE and matched its streaming SHA-1 to the extraction manifest. +- P1-T4 — Imported the candidate into an ignored local Ghidra project using `PSX Executables Loader`; recorded static loader findings. +- Rules check — Re-read `AGENTS.md` mandatory behavior after P1-T4. +- P1-T5 — Wrote `docs/PHASE1_REPRODUCIBILITY.md` with commands, results, firewall checks, and remaining unknowns. + +## Verified Results + +### Extraction + +- The tracked synthetic-only suite passes 19 tests. +- Two fresh extraction runs produced byte-identical manifests with 96 regular files and 601,671,468 total extracted bytes. +- All 96 final extracted files were independently re-hashed against the final manifest. +- Exact ISO9660 identifiers are preserved in ignored `extracted/`; manifests, temporary reports, and the disc remain untracked. + +### Executable + +- Exact ISO identifier: `SCUS_946.40;1`. +- PS-X EXE header and manifest validation passed. +- File SHA-1: `e173426c157384ebf1b6caf8c6fea18a85a14af9`. +- Declared text range: `[0x80010000, 0x801DC000)`. +- Declared entry PC: `0x800FB368`. +- Header GP/SP values are recorded in `docs/formats.md` as header facts only; they are not runtime claims. + +### Ghidra + +- Ghidra automatically selected `PSX Executables Loader` for the extracted candidate. +- Program language: `PSX:LE:32:default`; compiler spec: `default`; image base: `0x80000000`. +- The loader created 31 static memory blocks and entry point `start` at `0x800FB368`. +- Before Auto Analyze, the developer manually observed Program Information `PsyQ Version = 4.5.0` for the USA import. The analyzer fallback `4.7.0` was not used as evidence. +- Auto Analyze remains unrun. All layout and SDK findings are static import evidence, not PCSX-Redux runtime observations. +- Local project/import data is ignored under `ghidra/`; static layout and import provenance are recorded in `docs/memory-map.md`, `docs/formats.md`, and local `ghidra/P1-T4-import.md`. + +## Verification and Firewall + +- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v` passed all 19 tests at close. +- `git diff --check` passed. +- `git diff --cached --quiet` passed before closure staging. +- `extracted/`, `.run/`, and `ghidra/` records were explicitly confirmed ignored. +- No disc data, extraction output, manifest, Ghidra database, disassembly, or proprietary SDK file was staged. + +## Deviations + +| Item | Plan | Actual | Reason | +|---|---|---|---| +| Auto Analyze | Preserve first import results and record SDK evidence | Left unrun | Avoided replacing a loader-detected PsyQ value with the analyzer's `4.7.0` fallback before evidence was recorded. | +| Program Information query | Read PsyQ field through MCP | Developer read the field manually | The MCP Ghidra Java script provider failed with a `GhidraPlaceholderBundle`/`GhidraSourceBundle` class-cast error. The temporary diagnostic script was removed. | + +## Rules Added This Phase + +None. + +## Next + +Phase 2 — runtime loader/overlay and archive-format investigation. It requires a new task-by-task plan, verification criteria, and explicit developer approval in a fresh session. Existing Phase 1 static findings must not be promoted to runtime facts without PCSX-Redux observation. + +## Plain-English Recap + +The disc can now be extracted repeatably without leaking game material into Git, its executable has a validated PS-X EXE header, and Ghidra has imported it with the intended PSX loader. The project has static addresses and a PsyQ import clue, but no runtime behavior, overlay model, archive format, or matching toolchain has been established yet. + +## 🛑 Stop Here + +Phase 1 is closed. `CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase1.md`; this PhaseEnd and the digest update are ready for the closure commit. Do **not** begin Phase 2 in this session. Start a fresh session by drafting and approving a Phase 2 plan. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase1.md similarity index 57% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase1.md index 2934ea8..a8e788e 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/logs/Phase1.md @@ -8,9 +8,9 @@ - [x] **P1-T1 — ISO walker:** added executable, dependency-free `tools/sf3_extract` and synthetic-only `unittest` coverage. The read-only `list` command validates MODE2/2352 sector geometry, ISO9660 descriptors and directory records, extents/bounds, output-path components, and recursive directory extents before returning deterministic paths. - [x] **P1-T2 — Deterministic extraction:** extended `tools/sf3_extract` with transactional `extract`; it preserves exact ISO identifiers, copies every validated extent from the same open image handle, and writes ignored `extracted/MANIFEST.tsv` with path, first LBA, complete extents, size, and SHA-1. Two fresh direct runs produced identical manifests and file counts. - [x] **P1-T3 — Executable characterization:** added synthetic-tested `psx-exe-info` validation to `tools/sf3_extract`. Exact ISO file `SCUS_946.40;1` passed PS-X EXE magic/header/text-bound checks and a streaming SHA-1 match against `extracted/MANIFEST.tsv`; permitted metadata is recorded in `docs/formats.md` and static-only header declarations in `docs/memory-map.md`. -- [ ] **P1-T4 — Ghidra import:** import the extracted executable with `ghidra_psx_ldr` into ignored `ghidra/` and record loader findings. -- [ ] **Rules check:** re-read `AGENTS.md` mandatory behavior after P1-T4. -- [ ] **P1-T5 — Reproducibility record:** document commands, results, and remaining unknowns; confirm no ROM-derived material is staged. +- [x] **P1-T4 — Ghidra import:** imported exact file `SCUS_946.40;1` into the ignored local project with automatic `PSX Executables Loader` selection. Ghidra reports `PSX:LE:32:default`, image base `0x80000000`, 31 loader-created blocks, and entry `start` at `0x800FB368`; loader/import evidence and static layout are recorded in `docs/formats.md`, `docs/memory-map.md`, and ignored `ghidra/P1-T4-import.md`. +- [x] **Rules check:** mandatory behavior in `AGENTS.md` re-read after P1-T4. +- [x] **P1-T5 — Reproducibility record:** added `docs/PHASE1_REPRODUCIBILITY.md` with safe commands, verification results, Ghidra import provenance, firewall behavior, and remaining unknowns. The final 19-test suite passed and no ROM-derived material is staged. ## Session checkpoint @@ -20,4 +20,10 @@ P1-T1 is complete. `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s to P1-T2 is complete. The host path is on native writable btrfs. The expanded synthetic-only suite passed 14 tests, including transactional extraction, multi-extent copying, refusal of a corrupt data sector and existing output, CLI extraction, and two fresh synthetic runs. Two serial fresh runs of `./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted` produced 96 files and 601,671,468 bytes each; their ignored manifests were byte-identical. All 96 final extracted files were re-hashed against the final manifest successfully. The first-run manifest is retained only in ignored `.run/p1-t2-manifest-run1.tsv`; the second direct run remains in ignored `extracted/`. No generated material is staged. -P1-T3 is complete. The 19-test synthetic-only suite covers PS-X EXE parsing plus bad magic, declared-text-bound, and manifest-hash rejection. `SCUS_946.40;1` passed the header and manifest checks with file SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. The GP/SP values in `docs/formats.md` are raw header fields only; no Ghidra import or runtime observation has occurred. The ignored metadata report is `.run/p1-t3-psx-exe-info.tsv`. P1-T4 has not started. +P1-T3 is complete. The 19-test synthetic-only suite covers PS-X EXE parsing plus bad magic, declared-text-bound, and manifest-hash rejection. `SCUS_946.40;1` passed the header and manifest checks with file SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. The GP/SP values in `docs/formats.md` are raw header fields only; no Ghidra import or runtime observation had occurred at T3 completion. The ignored metadata report is `.run/p1-t3-psx-exe-info.tsv`. + +P1-T4 is complete. Ghidra automatically selected `PSX Executables Loader`; the imported USA program is open with language `PSX:LE:32:default`, default compiler spec, image base `0x80000000`, 31 static loader-created memory blocks, and entry point `start` at `0x800FB368`. The developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze; the analyzer fallback `4.7.0` was not used as evidence. Auto Analysis remains unrun. Local import settings/logs are in ignored `ghidra/P1-T4-import.md`. The MCP script provider failed to query Program Information directly and its temporary script was removed. No Ghidra database or ROM-derived output is staged. + +Rules check — re-read complete. Continuing with P1-T5. + +P1-T5 is complete. `docs/PHASE1_REPRODUCIBILITY.md` records the commands, extraction counts, executable metadata, Ghidra-loader results, firewall checks, and remaining unknowns without copying game data. The final 19-test suite passed; `git diff --cached --quiet` passed; and `extracted/`, `.run/`, and `ghidra/` records remain ignored. All Phase 1 checklist items are complete and ready for developer milestone confirmation. Do not write a PhaseEnd record, update the digest, or begin Phase 2 until the developer confirms the Phase 1 milestone.