feat(phase-1): add disc extraction and EXE validation
This commit is contained in:
Christopher Williams
@@ -2,25 +2,18 @@
|
||||
|
||||
Matching decompilation of **Syphon Filter 3 (USA)** for PlayStation. The goal is to rebuild original game binaries from C source with byte-for-byte identical output. A native PC port is out of scope until the matching decompilation is established.
|
||||
|
||||
## Current bootstrap state
|
||||
## Session-start load order
|
||||
|
||||
- The legally dumped disc image is local-only: `Syphon Filter 3 (USA).bin`.
|
||||
- Ghidra 12.1.2 is installed at `/opt/ghidra`.
|
||||
- The locally built `ghidra_psx_ldr` extension is installed for Ghidra 12.1.2.
|
||||
- PCSX-Redux is installed and launches normally.
|
||||
- Do not assume the game serial, executable name, compiler, SDK version, archive format, compression, overlay layout, or RAM load addresses. Establish each from the bytes or runtime observation.
|
||||
At the start of every session, read these files in order:
|
||||
|
||||
## Phase 0 — establish governance before reverse engineering
|
||||
1. `AGENTS.md` — standing agent rules.
|
||||
2. `PROJECT_CONTEXT.md` — permanent project constitution.
|
||||
3. `phase-ends/DIGEST.md` — completed-phase findings.
|
||||
4. `phase-ends/CURRENT_PHASE.md` — active in-phase state, when present.
|
||||
5. The active phase plan in `phase-ends/`.
|
||||
6. Task-relevant records under `docs/`.
|
||||
|
||||
The first task is to create the repository governance and state files:
|
||||
|
||||
- `.gitignore` and `.gitattributes`
|
||||
- `PROJECT_CONTEXT.md` (permanent constitution)
|
||||
- `phase-ends/DIGEST.md` and `phase-ends/CURRENT_PHASE.md`
|
||||
- `docs/SETUP.md`, `docs/formats.md`, and `docs/memory-map.md`
|
||||
- a task-level Phase 1 plan
|
||||
|
||||
Until those files exist, use this document as the bootstrap authority. Once they exist, update this file only to point to their session-start load order; do not duplicate their evolving state here.
|
||||
The active plan and current-phase record govern day-to-day execution; do not duplicate their evolving state in this file.
|
||||
|
||||
## Mandatory behavior
|
||||
|
||||
@@ -65,11 +58,3 @@ Until those files exist, use this document as the bootstrap authority. Once they
|
||||
- Keep current in-phase state in `phase-ends/CURRENT_PHASE.md`; write a checkpoint before ending a session.
|
||||
- After every four completed tasks, re-read these mandatory rules and state: `Rules check — re-read complete. Continuing with [next task].`
|
||||
- A phase closes only after every checklist item is explicitly verified and the developer confirms the milestone. Then write a PhaseEnd file, update the digest, and stop; do not begin the next phase in the same session.
|
||||
|
||||
## Initial roadmap
|
||||
|
||||
1. Phase 0: repository firewall, documentation/state structure, and disc-image characterization.
|
||||
2. Phase 1: deterministic extraction, PS-X EXE discovery, and first Ghidra import.
|
||||
3. Phase 2: runtime loader/overlay and archive-format investigation with PCSX-Redux.
|
||||
4. Phase 3: matching toolchain, all-assembly rebuild, and compiler fingerprint.
|
||||
5. Phase 4+: function matching, shared-body infrastructure, and scalable verification.
|
||||
|
||||
+19
-1
@@ -13,7 +13,25 @@ The current USA disc image is exactly 294,018 sectors of 2352 bytes. Phase 0 T1
|
||||
Filesystem/executable extraction may therefore read the BIN with a 2352-byte stride and offset 24 without a CUE. The following remain unverified:
|
||||
|
||||
- track count and track boundaries beyond the data filesystem;
|
||||
- executable filename and format;
|
||||
- archive, compression, audio, video, and overlay formats.
|
||||
|
||||
Do not infer later format details from other Syphon Filter games or other regions.
|
||||
|
||||
## PS-X EXE candidate (Phase 1 T3)
|
||||
|
||||
The exact ISO9660 identifier `SCUS_946.40;1` from the local USA image was extracted without renaming. `tools/sf3_extract psx-exe-info` validated its `PS-X EXE` magic, 0x800-byte header, declared text bounds, and streaming SHA-1 against ignored `extracted/MANIFEST.tsv`. The header-field offsets were cross-checked against the locally installed loader source, `ghidra_psx_ldr/src/main/java/psx/PsxExe.java`.
|
||||
|
||||
| Property | Header field / evidence | Verified value |
|
||||
|---|---|---|
|
||||
| File SHA-1 | Full extracted file; matches manifest | `e173426c157384ebf1b6caf8c6fea18a85a14af9` |
|
||||
| File size | Full extracted file | 1,886,208 bytes |
|
||||
| Header size | PS-X EXE header | 0x800 (2,048 bytes) |
|
||||
| Text load address | 0x18 | `0x80010000` |
|
||||
| Text size | 0x1C | 1,884,160 bytes (`0x001CC000`) |
|
||||
| Declared text range | load address + text size | `[0x80010000, 0x801DC000)` |
|
||||
| Entry PC | 0x10 | `0x800FB368` (within the declared text range) |
|
||||
| Initial GP header field | 0x14 | `0x00000000` |
|
||||
| SP base header field | 0x30 | `0x801FFFF0` |
|
||||
| SP offset header field | 0x34 | `0x00000000` |
|
||||
|
||||
These are static header declarations from the USA file, not PCSX-Redux observations. In particular, the zero GP and SP-offset values do not establish post-loader register behavior, and the declared text range is not yet a runtime memory-map claim.
|
||||
|
||||
@@ -2,4 +2,15 @@
|
||||
|
||||
No USA runtime addresses are verified yet.
|
||||
|
||||
## Static PS-X EXE header declarations (Phase 1 T3)
|
||||
|
||||
These entries are header-declared values from the local USA image's exact ISO9660 file `SCUS_946.40;1`. They are **not** Ghidra-import findings or PCSX-Redux runtime observations.
|
||||
|
||||
| Address / field | Purpose | Source build / region | Evidence | Verification status |
|
||||
|---|---|---|---|---|
|
||||
| `[0x80010000, 0x801DC000)` | Declared text range | Local USA image | PS-X EXE header offsets 0x18/0x1C, parsed by `tools/sf3_extract`; field layout cross-checked with local `ghidra_psx_ldr` source | Header-valid; runtime load status unverified |
|
||||
| `0x800FB368` | Declared entry PC | Local USA image | PS-X EXE header offset 0x10 | Header-valid and within declared text range; runtime execution unverified |
|
||||
| `0x00000000` | Initial GP header field | Local USA image | PS-X EXE header offset 0x14 | Header-valid; not a runtime GP observation |
|
||||
| `0x801FFFF0` + `0x00000000` | SP base and SP offset header fields | Local USA image | PS-X EXE header offsets 0x30/0x34 | Header-valid; no effective/runtime SP inference made |
|
||||
|
||||
Record each future entry with: address range, purpose, source build/region, evidence source (Ghidra or runtime observation), and verification status. Do not transfer addresses from other Syphon Filter releases without explicit USA validation.
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# CURRENT_PHASE — Phase 1: Deterministic Extraction, Executable Discovery, and First Ghidra Import
|
||||
|
||||
**Status:** in progress
|
||||
**Plan:** `phase-ends/Phase1_PLAN.md` (approved by developer)
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] **P1-T1 — ISO walker:** added executable, dependency-free `tools/sf3_extract` and synthetic-only `unittest` coverage. The read-only `list` command validates MODE2/2352 sector geometry, ISO9660 descriptors and directory records, extents/bounds, output-path components, and recursive directory extents before returning deterministic paths.
|
||||
- [x] **P1-T2 — Deterministic extraction:** extended `tools/sf3_extract` with transactional `extract`; it preserves exact ISO identifiers, copies every validated extent from the same open image handle, and writes ignored `extracted/MANIFEST.tsv` with path, first LBA, complete extents, size, and SHA-1. Two fresh direct runs produced identical manifests and file counts.
|
||||
- [x] **P1-T3 — Executable characterization:** added synthetic-tested `psx-exe-info` validation to `tools/sf3_extract`. Exact ISO file `SCUS_946.40;1` passed PS-X EXE magic/header/text-bound checks and a streaming SHA-1 match against `extracted/MANIFEST.tsv`; permitted metadata is recorded in `docs/formats.md` and static-only header declarations in `docs/memory-map.md`.
|
||||
- [ ] **P1-T4 — Ghidra import:** import the extracted executable with `ghidra_psx_ldr` into ignored `ghidra/` and record loader findings.
|
||||
- [ ] **Rules check:** re-read `AGENTS.md` mandatory behavior after P1-T4.
|
||||
- [ ] **P1-T5 — Reproducibility record:** document commands, results, and remaining unknowns; confirm no ROM-derived material is staged.
|
||||
|
||||
## Session checkpoint
|
||||
|
||||
Phase 1 opened in a fresh session after the completed Phase 0 close. The approved plan has been loaded, and the session-start load order and active checkpoint have been restored.
|
||||
|
||||
P1-T1 is complete. `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v` initially passed 8 synthetic-only walker tests, including malformed record, non-MODE2 sector, out-of-bounds extent, unsafe path, recursive extent, and CLI refusal cases. `./tools/sf3_extract --help` passed as a direct-command smoke check.
|
||||
|
||||
P1-T2 is complete. The host path is on native writable btrfs. The expanded synthetic-only suite passed 14 tests, including transactional extraction, multi-extent copying, refusal of a corrupt data sector and existing output, CLI extraction, and two fresh synthetic runs. Two serial fresh runs of `./tools/sf3_extract extract 'disks/Syphon Filter 3 (USA).bin' extracted` produced 96 files and 601,671,468 bytes each; their ignored manifests were byte-identical. All 96 final extracted files were re-hashed against the final manifest successfully. The first-run manifest is retained only in ignored `.run/p1-t2-manifest-run1.tsv`; the second direct run remains in ignored `extracted/`. No generated material is staged.
|
||||
|
||||
P1-T3 is complete. The 19-test synthetic-only suite covers PS-X EXE parsing plus bad magic, declared-text-bound, and manifest-hash rejection. `SCUS_946.40;1` passed the header and manifest checks with file SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. The GP/SP values in `docs/formats.md` are raw header fields only; no Ghidra import or runtime observation has occurred. The ignored metadata report is `.run/p1-t3-psx-exe-info.tsv`. P1-T4 has not started.
|
||||
Executable
+1146
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,656 @@
|
||||
"""Synthetic-only tests for ISO9660 extraction and PS-X EXE validation.
|
||||
|
||||
These fixtures are assembled in temporary directories. They never inspect the
|
||||
local disc image or any extracted game material.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
TOOL_PATH = Path(__file__).resolve().parents[1] / "sf3_extract"
|
||||
|
||||
|
||||
def _load_tool() -> object:
|
||||
loader = importlib.machinery.SourceFileLoader("sf3_extract_under_test", str(TOOL_PATH))
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
if spec is None:
|
||||
raise RuntimeError("could not create an import specification for sf3_extract")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
sf3_extract = _load_tool()
|
||||
|
||||
|
||||
def _set_both_endian_u16(data: bytearray, offset: int, value: int) -> None:
|
||||
data[offset : offset + 2] = value.to_bytes(2, "little")
|
||||
data[offset + 2 : offset + 4] = value.to_bytes(2, "big")
|
||||
|
||||
|
||||
def _set_both_endian_u32(data: bytearray, offset: int, value: int) -> None:
|
||||
data[offset : offset + 4] = value.to_bytes(4, "little")
|
||||
data[offset + 4 : offset + 8] = value.to_bytes(4, "big")
|
||||
|
||||
|
||||
def _directory_record(
|
||||
lba: int,
|
||||
size: int,
|
||||
identifier: bytes,
|
||||
*,
|
||||
flags: int = 0,
|
||||
extended_attribute_blocks: int = 0,
|
||||
) -> bytes:
|
||||
if not identifier:
|
||||
raise ValueError("synthetic records require a non-empty identifier")
|
||||
padding = 1 if len(identifier) % 2 == 0 else 0
|
||||
record = bytearray(33 + len(identifier) + padding)
|
||||
record[0] = len(record)
|
||||
record[1] = extended_attribute_blocks
|
||||
_set_both_endian_u32(record, 2, lba)
|
||||
_set_both_endian_u32(record, 10, size)
|
||||
record[25] = flags
|
||||
_set_both_endian_u16(record, 28, 1)
|
||||
record[32] = len(identifier)
|
||||
record[33 : 33 + len(identifier)] = identifier
|
||||
return bytes(record)
|
||||
|
||||
|
||||
def _directory_block(records: list[bytes]) -> bytes:
|
||||
block = bytearray(sf3_extract.LOGICAL_BLOCK_SIZE)
|
||||
cursor = 0
|
||||
for record in records:
|
||||
if cursor + len(record) > len(block):
|
||||
raise ValueError("synthetic directory records exceed one logical block")
|
||||
block[cursor : cursor + len(record)] = record
|
||||
cursor += len(record)
|
||||
return bytes(block)
|
||||
|
||||
|
||||
def _mode2_sector(user_data: bytes) -> bytes:
|
||||
if len(user_data) != sf3_extract.LOGICAL_BLOCK_SIZE:
|
||||
raise ValueError("synthetic user data must occupy one logical block")
|
||||
sector = bytearray(sf3_extract.RAW_SECTOR_SIZE)
|
||||
sector[: len(sf3_extract.MODE2_SYNC)] = sf3_extract.MODE2_SYNC
|
||||
sector[15] = 2
|
||||
sector[
|
||||
sf3_extract.USER_DATA_OFFSET : sf3_extract.USER_DATA_OFFSET
|
||||
+ sf3_extract.LOGICAL_BLOCK_SIZE
|
||||
] = user_data
|
||||
return bytes(sector)
|
||||
|
||||
|
||||
def _user_block(prefix: bytes) -> bytes:
|
||||
if len(prefix) > sf3_extract.LOGICAL_BLOCK_SIZE:
|
||||
raise ValueError("synthetic prefix exceeds one logical block")
|
||||
return prefix.ljust(sf3_extract.LOGICAL_BLOCK_SIZE, b"\x00")
|
||||
|
||||
|
||||
def _primary_volume_descriptor(volume_size: int, root_lba: int, root_size: int) -> bytes:
|
||||
pvd = bytearray(sf3_extract.LOGICAL_BLOCK_SIZE)
|
||||
pvd[0] = 1
|
||||
pvd[1:6] = b"CD001"
|
||||
pvd[6] = 1
|
||||
_set_both_endian_u32(pvd, 80, volume_size)
|
||||
_set_both_endian_u16(pvd, 120, 1)
|
||||
_set_both_endian_u16(pvd, 124, 1)
|
||||
_set_both_endian_u16(pvd, 128, sf3_extract.LOGICAL_BLOCK_SIZE)
|
||||
root = _directory_record(
|
||||
root_lba,
|
||||
root_size,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
)
|
||||
pvd[156 : 156 + len(root)] = root
|
||||
return bytes(pvd)
|
||||
|
||||
|
||||
def _volume_descriptor_terminator() -> bytes:
|
||||
descriptor = bytearray(sf3_extract.LOGICAL_BLOCK_SIZE)
|
||||
descriptor[0] = sf3_extract.VOLUME_DESCRIPTOR_TERMINATOR
|
||||
descriptor[1:6] = b"CD001"
|
||||
descriptor[6] = 1
|
||||
return bytes(descriptor)
|
||||
|
||||
|
||||
def _write_image(
|
||||
path: Path,
|
||||
*,
|
||||
root_block: bytes,
|
||||
directory_blocks: dict[int, bytes] | None = None,
|
||||
user_blocks: dict[int, bytes] | None = None,
|
||||
volume_size: int = 32,
|
||||
root_lba: int = 20,
|
||||
) -> None:
|
||||
if len(root_block) != sf3_extract.LOGICAL_BLOCK_SIZE:
|
||||
raise ValueError("synthetic root block must be one logical block")
|
||||
blocks = {root_lba: root_block, **(directory_blocks or {})}
|
||||
for lba, data in (user_blocks or {}).items():
|
||||
if lba in blocks:
|
||||
raise ValueError("synthetic user data collides with a directory block")
|
||||
blocks[lba] = data
|
||||
sectors = [_mode2_sector(bytes(sf3_extract.LOGICAL_BLOCK_SIZE)) for _ in range(volume_size)]
|
||||
sectors[sf3_extract.PVD_LBA] = _mode2_sector(
|
||||
_primary_volume_descriptor(volume_size, root_lba, sf3_extract.LOGICAL_BLOCK_SIZE)
|
||||
)
|
||||
sectors[sf3_extract.PVD_LBA + 1] = _mode2_sector(_volume_descriptor_terminator())
|
||||
for lba, data in blocks.items():
|
||||
if not 0 <= lba < volume_size:
|
||||
raise ValueError("synthetic directory LBA is outside the image")
|
||||
if len(data) != sf3_extract.LOGICAL_BLOCK_SIZE:
|
||||
raise ValueError("synthetic directory block must be one logical block")
|
||||
sectors[lba] = _mode2_sector(data)
|
||||
path.write_bytes(b"".join(sectors))
|
||||
|
||||
|
||||
def _valid_image(path: Path, *, user_blocks: dict[int, bytes] | None = None) -> None:
|
||||
root_lba = 20
|
||||
child_lba = 21
|
||||
root_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(22, 5, b"README.TXT;1"),
|
||||
_directory_record(
|
||||
child_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"DATA",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
]
|
||||
)
|
||||
child_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
child_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(23, 7, b"INNER.BIN;1"),
|
||||
]
|
||||
)
|
||||
_write_image(
|
||||
path,
|
||||
root_block=root_block,
|
||||
directory_blocks={child_lba: child_block},
|
||||
user_blocks=user_blocks,
|
||||
)
|
||||
|
||||
|
||||
def _set_u32_le(data: bytearray, offset: int, value: int) -> None:
|
||||
data[offset : offset + 4] = value.to_bytes(4, "little")
|
||||
|
||||
|
||||
def _write_psx_exe_fixture(
|
||||
root: Path,
|
||||
*,
|
||||
relative_path: str = "MAIN.EXE;1",
|
||||
payload: bytes = b"\x00" * 16,
|
||||
declared_text_size: int | None = None,
|
||||
text_load_address: int = 0x80010000,
|
||||
entry_pc: int | None = None,
|
||||
initial_gp: int = 0x80020000,
|
||||
sp_base: int = 0x801FFF00,
|
||||
sp_offset: int = 0,
|
||||
magic: bytes = sf3_extract.PSX_EXE_MAGIC,
|
||||
manifest_sha1: str | None = None,
|
||||
) -> tuple[Path, Path]:
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
header = bytearray(sf3_extract.PSX_EXE_HEADER_SIZE)
|
||||
header[: len(magic)] = magic
|
||||
_set_u32_le(
|
||||
header,
|
||||
sf3_extract.PSX_EXE_INIT_PC_OFFSET,
|
||||
text_load_address if entry_pc is None else entry_pc,
|
||||
)
|
||||
_set_u32_le(header, sf3_extract.PSX_EXE_INIT_GP_OFFSET, initial_gp)
|
||||
_set_u32_le(header, sf3_extract.PSX_EXE_TEXT_ADDRESS_OFFSET, text_load_address)
|
||||
_set_u32_le(
|
||||
header,
|
||||
sf3_extract.PSX_EXE_TEXT_SIZE_OFFSET,
|
||||
len(payload) if declared_text_size is None else declared_text_size,
|
||||
)
|
||||
_set_u32_le(header, sf3_extract.PSX_EXE_SP_BASE_OFFSET, sp_base)
|
||||
_set_u32_le(header, sf3_extract.PSX_EXE_SP_OFFSET_OFFSET, sp_offset)
|
||||
|
||||
executable = root / relative_path
|
||||
executable.parent.mkdir(parents=True, exist_ok=True)
|
||||
contents = bytes(header) + payload
|
||||
executable.write_bytes(contents)
|
||||
digest = manifest_sha1 if manifest_sha1 is not None else hashlib.sha1(contents).hexdigest()
|
||||
manifest = root / sf3_extract.MANIFEST_FILENAME
|
||||
manifest.write_text(
|
||||
sf3_extract.MANIFEST_HEADER
|
||||
+ f"{relative_path}\t24\t24:{len(contents)}\t{len(contents)}\t{digest}\n",
|
||||
encoding="ascii",
|
||||
)
|
||||
return executable, manifest
|
||||
|
||||
|
||||
class Sf3ExtractTests(unittest.TestCase):
|
||||
def synthetic_path(self) -> Path:
|
||||
temporary_directory = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temporary_directory.cleanup)
|
||||
return Path(temporary_directory.name) / "synthetic-mode2.bin"
|
||||
|
||||
def test_walks_nested_tree_in_deterministic_path_order(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
|
||||
entries = sf3_extract.walk_image(image)
|
||||
|
||||
self.assertEqual(
|
||||
[
|
||||
(
|
||||
entry.path,
|
||||
entry.is_directory,
|
||||
entry.size,
|
||||
[(extent.lba, extent.size) for extent in entry.extents],
|
||||
)
|
||||
for entry in entries
|
||||
],
|
||||
[
|
||||
("DATA", True, 2048, [(21, 2048)]),
|
||||
("DATA/INNER.BIN;1", False, 7, [(23, 7)]),
|
||||
("README.TXT;1", False, 5, [(22, 5)]),
|
||||
],
|
||||
)
|
||||
|
||||
def test_list_command_uses_validated_walker(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(TOOL_PATH), "list", str(image)],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(completed.returncode, 0, completed.stderr)
|
||||
self.assertEqual(
|
||||
completed.stdout.splitlines(),
|
||||
[
|
||||
"D\tDATA\t2048\t21:2048",
|
||||
"F\tDATA/INNER.BIN;1\t7\t23:7",
|
||||
"F\tREADME.TXT;1\t5\t22:5",
|
||||
],
|
||||
)
|
||||
self.assertEqual(completed.stderr, "")
|
||||
|
||||
def test_rejects_non_mode2_primary_volume_sector(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
with image.open("r+b") as handle:
|
||||
handle.seek((sf3_extract.PVD_LBA * sf3_extract.RAW_SECTOR_SIZE) + 15)
|
||||
handle.write(b"\x01")
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "not MODE2"):
|
||||
sf3_extract.walk_image(image)
|
||||
|
||||
def test_rejects_malformed_directory_record(self) -> None:
|
||||
root_lba = 20
|
||||
root_block = bytearray(
|
||||
_directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
]
|
||||
)
|
||||
)
|
||||
root_block[68] = 20
|
||||
image = self.synthetic_path()
|
||||
_write_image(image, root_block=bytes(root_block))
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "shorter than 34"):
|
||||
sf3_extract.walk_image(image)
|
||||
|
||||
def test_rejects_out_of_bounds_file_extent(self) -> None:
|
||||
root_lba = 20
|
||||
root_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(32, 1, b"OUT.BIN;1"),
|
||||
]
|
||||
)
|
||||
image = self.synthetic_path()
|
||||
_write_image(image, root_block=root_block)
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "outside volume size"):
|
||||
sf3_extract.walk_image(image)
|
||||
|
||||
def test_rejects_unsafe_path_component(self) -> None:
|
||||
root_lba = 20
|
||||
root_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(22, 1, b"../ESCAPE;1"),
|
||||
]
|
||||
)
|
||||
image = self.synthetic_path()
|
||||
_write_image(image, root_block=root_block)
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "unsafe path component"):
|
||||
sf3_extract.walk_image(image)
|
||||
|
||||
def test_rejects_recursive_directory_extent(self) -> None:
|
||||
root_lba = 20
|
||||
child_lba = 21
|
||||
root_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
child_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"DATA",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
]
|
||||
)
|
||||
child_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
child_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"LOOP",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
]
|
||||
)
|
||||
image = self.synthetic_path()
|
||||
_write_image(image, root_block=root_block, directory_blocks={child_lba: child_block})
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "cycle or alias refused"):
|
||||
sf3_extract.walk_image(image)
|
||||
|
||||
def test_cli_refuses_corrupt_input_loudly(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
with image.open("r+b") as handle:
|
||||
handle.seek((sf3_extract.PVD_LBA * sf3_extract.RAW_SECTOR_SIZE) + 15)
|
||||
handle.write(b"\x01")
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(TOOL_PATH), "list", str(image)],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(completed.returncode, 2)
|
||||
self.assertIn("sf3_extract: error:", completed.stderr)
|
||||
self.assertIn("not MODE2", completed.stderr)
|
||||
self.assertEqual(completed.stdout, "")
|
||||
|
||||
def test_extracts_files_and_writes_deterministic_manifest(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
readme = b"hello"
|
||||
inner = b"content"
|
||||
_valid_image(image, user_blocks={22: _user_block(readme), 23: _user_block(inner)})
|
||||
output = image.parent / "extracted"
|
||||
|
||||
result = sf3_extract.extract_image(image, output)
|
||||
|
||||
self.assertEqual(result.output_root, output.resolve())
|
||||
self.assertEqual(result.manifest_path, output.resolve() / sf3_extract.MANIFEST_FILENAME)
|
||||
self.assertEqual((result.file_count, result.byte_count), (2, len(readme) + len(inner)))
|
||||
self.assertEqual((output / "README.TXT;1").read_bytes(), readme)
|
||||
self.assertEqual((output / "DATA" / "INNER.BIN;1").read_bytes(), inner)
|
||||
self.assertEqual(
|
||||
result.manifest_path.read_text(encoding="ascii"),
|
||||
"".join(
|
||||
[
|
||||
sf3_extract.MANIFEST_HEADER,
|
||||
"DATA/INNER.BIN;1\t23\t23:7\t7\t"
|
||||
f"{hashlib.sha1(inner).hexdigest()}\n",
|
||||
"README.TXT;1\t22\t22:5\t5\t"
|
||||
f"{hashlib.sha1(readme).hexdigest()}\n",
|
||||
]
|
||||
),
|
||||
)
|
||||
|
||||
def test_two_fresh_extractions_have_identical_manifests(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(
|
||||
image,
|
||||
user_blocks={22: _user_block(b"hello"), 23: _user_block(b"content")},
|
||||
)
|
||||
first = image.parent / "first"
|
||||
second = image.parent / "second"
|
||||
|
||||
first_result = sf3_extract.extract_image(image, first)
|
||||
second_result = sf3_extract.extract_image(image, second)
|
||||
|
||||
self.assertEqual(first_result.file_count, second_result.file_count)
|
||||
self.assertEqual(
|
||||
first_result.manifest_path.read_bytes(), second_result.manifest_path.read_bytes()
|
||||
)
|
||||
self.assertEqual(
|
||||
(first / "README.TXT;1").read_bytes(), (second / "README.TXT;1").read_bytes()
|
||||
)
|
||||
self.assertEqual(
|
||||
(first / "DATA" / "INNER.BIN;1").read_bytes(),
|
||||
(second / "DATA" / "INNER.BIN;1").read_bytes(),
|
||||
)
|
||||
|
||||
def test_extract_copies_all_multi_extent_data(self) -> None:
|
||||
root_lba = 20
|
||||
root_block = _directory_block(
|
||||
[
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x00",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(
|
||||
root_lba,
|
||||
sf3_extract.LOGICAL_BLOCK_SIZE,
|
||||
b"\x01",
|
||||
flags=sf3_extract.DIRECTORY_FLAG,
|
||||
),
|
||||
_directory_record(22, 3, b"SPLIT.BIN;1", flags=sf3_extract.MULTI_EXTENT_FLAG),
|
||||
_directory_record(23, 2, b"SPLIT.BIN;1"),
|
||||
]
|
||||
)
|
||||
image = self.synthetic_path()
|
||||
_write_image(
|
||||
image,
|
||||
root_block=root_block,
|
||||
user_blocks={22: _user_block(b"abc"), 23: _user_block(b"de")},
|
||||
)
|
||||
output = image.parent / "multi"
|
||||
|
||||
result = sf3_extract.extract_image(image, output)
|
||||
|
||||
self.assertEqual((result.file_count, result.byte_count), (1, 5))
|
||||
self.assertEqual((output / "SPLIT.BIN;1").read_bytes(), b"abcde")
|
||||
self.assertIn("SPLIT.BIN;1\t22\t22:3,23:2\t5\t", result.manifest_path.read_text())
|
||||
|
||||
def test_extract_refuses_bad_file_sector_without_installing_output(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
with image.open("r+b") as handle:
|
||||
handle.seek((22 * sf3_extract.RAW_SECTOR_SIZE) + 15)
|
||||
handle.write(b"\x01")
|
||||
output = image.parent / "failed"
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "not MODE2"):
|
||||
sf3_extract.extract_image(image, output)
|
||||
|
||||
self.assertFalse(output.exists())
|
||||
self.assertEqual(list(image.parent.glob(".failed.tmp-*")), [])
|
||||
|
||||
def test_extract_refuses_existing_destination(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(image)
|
||||
output = image.parent / "existing"
|
||||
output.mkdir()
|
||||
marker = output / "keep"
|
||||
marker.write_text("do not overwrite", encoding="ascii")
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.IsoValidationError, "refusing to overwrite"):
|
||||
sf3_extract.extract_image(image, output)
|
||||
|
||||
self.assertEqual(marker.read_text(encoding="ascii"), "do not overwrite")
|
||||
|
||||
def test_extract_command_uses_transactional_extractor(self) -> None:
|
||||
image = self.synthetic_path()
|
||||
_valid_image(
|
||||
image,
|
||||
user_blocks={22: _user_block(b"hello"), 23: _user_block(b"content")},
|
||||
)
|
||||
output = image.parent / "cli-extracted"
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(TOOL_PATH), "extract", str(image), str(output)],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(completed.returncode, 0, completed.stderr)
|
||||
self.assertIn("extracted 2 files (12 bytes)", completed.stdout)
|
||||
self.assertTrue((output / sf3_extract.MANIFEST_FILENAME).is_file())
|
||||
self.assertEqual(completed.stderr, "")
|
||||
|
||||
def test_characterizes_manifest_tracked_psx_exe(self) -> None:
|
||||
root = self.synthetic_path().parent / "psx-exe"
|
||||
executable, manifest = _write_psx_exe_fixture(root)
|
||||
|
||||
info = sf3_extract.characterize_psx_exe(executable, manifest)
|
||||
|
||||
self.assertEqual(info.path, "MAIN.EXE;1")
|
||||
self.assertEqual(info.file_size, sf3_extract.PSX_EXE_HEADER_SIZE + 16)
|
||||
self.assertEqual(info.text_load_address, 0x80010000)
|
||||
self.assertEqual(info.text_size, 16)
|
||||
self.assertEqual(info.entry_pc, 0x80010000)
|
||||
self.assertEqual(info.initial_gp, 0x80020000)
|
||||
self.assertEqual(info.sp_base, 0x801FFF00)
|
||||
self.assertEqual(info.sp_offset, 0)
|
||||
self.assertEqual(info.sha1, hashlib.sha1(executable.read_bytes()).hexdigest())
|
||||
|
||||
def test_rejects_psx_exe_with_bad_magic(self) -> None:
|
||||
root = self.synthetic_path().parent / "bad-magic"
|
||||
executable, manifest = _write_psx_exe_fixture(root, magic=b"NOT EXE!")
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.PsxExeValidationError, "magic is missing"):
|
||||
sf3_extract.characterize_psx_exe(executable, manifest)
|
||||
|
||||
def test_rejects_psx_exe_text_beyond_file_payload(self) -> None:
|
||||
root = self.synthetic_path().parent / "bad-bounds"
|
||||
executable, manifest = _write_psx_exe_fixture(root, declared_text_size=32)
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.PsxExeValidationError, "exceeds payload size"):
|
||||
sf3_extract.characterize_psx_exe(executable, manifest)
|
||||
|
||||
def test_rejects_psx_exe_when_manifest_hash_differs(self) -> None:
|
||||
root = self.synthetic_path().parent / "bad-hash"
|
||||
executable, manifest = _write_psx_exe_fixture(root, manifest_sha1="0" * 40)
|
||||
|
||||
with self.assertRaisesRegex(sf3_extract.PsxExeValidationError, "SHA-1 does not match"):
|
||||
sf3_extract.characterize_psx_exe(executable, manifest)
|
||||
|
||||
def test_psx_exe_info_command_uses_header_and_manifest_validation(self) -> None:
|
||||
root = self.synthetic_path().parent / "psx-cli"
|
||||
executable, manifest = _write_psx_exe_fixture(root)
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, str(TOOL_PATH), "psx-exe-info", str(executable), str(manifest)],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(completed.returncode, 0, completed.stderr)
|
||||
self.assertIn("text_load_address\t0x80010000", completed.stdout)
|
||||
self.assertIn("entry_pc\t0x80010000", completed.stdout)
|
||||
self.assertIn("initial_gp\t0x80020000", completed.stdout)
|
||||
self.assertIn("sp_base\t0x801FFF00", completed.stdout)
|
||||
self.assertEqual(completed.stderr, "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user