chore(phase-2): close archive loader investigation
This commit is contained in:
Christopher Williams
@@ -0,0 +1,111 @@
|
||||
# Phase 2 Investigation Record
|
||||
|
||||
**Scope:** evidence-separated investigation of local USA disc-file access, executable entry state, candidate loader/overlay paths, and candidate archive structure.
|
||||
**Status:** task verification complete; Phase 2 remains open pending developer milestone confirmation.
|
||||
|
||||
This record contains reproducible non-content methods and interpreted findings only. The disc image, extraction tree, manifests, Ghidra programs, runtime logs, packet traces, memcards, and generated probe reports remain ignored. A static Ghidra observation and a PCSX-Redux observation are never treated as interchangeable evidence.
|
||||
|
||||
## Preconditions and preserved baseline
|
||||
|
||||
- The local exact USA input has the deterministic ignored extraction and manifest established in Phase 1.
|
||||
- The original Ghidra program `/SCUS_946.40;1` remains preserved and unanalyzed.
|
||||
- All Phase 2 static work used the separate ignored `/P2-analysis-SCUS_946.40;1` working program.
|
||||
- PCSX-Redux runtime work used isolated CLI/interpreter cold boots, fresh ignored artifacts, a regular configured BIOS file, and a locally readable disc input. No runtime result was inferred from emulator console text alone.
|
||||
|
||||
## Tool verification
|
||||
|
||||
Phase 2 added `tools/sf3_probe`, a dependency-free read-only structural feature probe. It validates manifest syntax, path safety, candidate regular-file status, size, and streaming SHA-1 before retaining a bounded prefix for opaque feature measurement. Its successful output contains deterministic opaque IDs and no source paths or source bytes.
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The final suite passed **28 synthetic-only tests**: 19 existing extractor tests and 9 probe tests. Probe coverage includes deterministic opaque output, extension selection, generic monotonic-word measurement, absent selection, malformed path, symbolic-link, size, digest, and CLI failure handling. No test reads the local disc or extracted game data.
|
||||
|
||||
## Evidence ledger
|
||||
|
||||
| ID | Provenance | Verified observation | Explicit limit |
|
||||
|---|---|---|---|
|
||||
| P2-T1 | Local USA manifest metadata | Two fresh aggregate-only censuses agreed; the existing manifest interface was sufficient. | Candidate classes are metadata only. |
|
||||
| P2-T2 | Separate Ghidra USA working program | Static candidate request, lookup/queue, and separate candidate-class transfer paths are reviewable at the recorded static addresses. | No physical transfer, completion, allocation, overlay, or file-format behavior is proved. |
|
||||
| P2-T3 | PCSX-Redux/GDB, two cold boots | Both runs halted at the executable entry breakpoint with identical PC/SP/GP captures. | Entry-state registers do not establish later loader or memory behavior. |
|
||||
| P2-T4 | PCSX-Redux/GDB, no-input cold-boot controls | The selected startup candidate was not reached; one predecessor invocation was observed; two conditioned controls repeated a no-match result. | This negative applies only to that input-free control window. |
|
||||
| P2-T5 | Local USA manifest-backed probe plus static revisit | Selected outer files passed full manifest SHA-1 verification in two identical reports; the separate static route is transfer setup only. | No inner container, table, compression, allocation, or overlay model is identified. |
|
||||
|
||||
## Static USA candidate paths
|
||||
|
||||
The following are Ghidra-generated candidate paths in the separate working program, not semantic function names or runtime facts.
|
||||
|
||||
| Static address / path | Bounded Ghidra basis | Status |
|
||||
|---|---|---|
|
||||
| `0x8002C22C` → `0x8002BC20` | Candidate-class request entry reaches request construction. | Static hypothesis; not observed in P2-T4's selected startup control. |
|
||||
| `0x8002BC20` → `0x800A6880` | Request construction reaches pathname lookup and queued-transfer setup. | Static hypothesis; its static destination is not a runtime allocation. |
|
||||
| `0x800A6880` → `0x800A6840` → `0x800A66B0` → `0x800A6FAC` | Lookup, length-check, registration, and conditional-dispatch candidates. | Static hypothesis; no CD/DMA/completion behavior claimed. |
|
||||
| `0x800B2B70` | Separate candidate-class routine constructs and registers a disc-file transfer through the candidate transfer subsystem. | Static transfer-route evidence only; no inner file parser/table semantics established. |
|
||||
|
||||
The ignored static records are `ghidra/P2-T2-static-survey.md` and `ghidra/P2-T5-archive-validation.md`. They contain interpreted findings without raw strings or disassembly.
|
||||
|
||||
## Runtime procedure and observations
|
||||
|
||||
The corrected isolated cold-boot configuration requires both `--gdb` and `--debugger`, plus CLI/interpreter/no-fastboot operation. Source inspection at the installed PCSX-Redux revision established that the remote server alone does not enable interpreter execution-breakpoint processing. A representative configuration is:
|
||||
|
||||
```bash
|
||||
pcsx-redux --cli --interpreter --debugger --gdb --gdb-port 3333 \
|
||||
--no-fastboot --no-webserver --bios "$REGULAR_BIOS_FILE" --loadiso "$LOCAL_DISC"
|
||||
```
|
||||
|
||||
Each accepted run required a packet-level `OK` for breakpoint insertion, a stopped target, register-only capture, and process/port cleanup. Two fresh runs halted at `0x800FB368` with identical PC `0x800FB368`, SP `0x801FFFF0`, and GP `0x00000000` observations. These are recorded as runtime entry-state facts in [memory-map.md](memory-map.md), not as a general runtime map.
|
||||
|
||||
For the selected no-input P2-T4 control:
|
||||
|
||||
- the direct static candidate at `0x8002C22C` did not stop during 75 seconds;
|
||||
- static predecessor `0x8002C3F8` stopped once with first argument register `0x00000001`; and
|
||||
- two independent 100-second controls conditioned on the branch value associated with the downstream candidate each showed six internal condition-evaluation stops and no matching stop.
|
||||
|
||||
No downstream breakpoint was armed after the conditioned no-match. These observations do not prove the path unreachable, do not name a state value, and do not establish a loader/overlay transition.
|
||||
|
||||
## Candidate-class structural validation
|
||||
|
||||
Only the census-/static-supported candidate class was measured. Two fresh runs of the probe used a 4096-byte prefix and streamed all selected candidates against the manifest SHA-1 before measurement:
|
||||
|
||||
```bash
|
||||
tools/sf3_probe prefix-features extracted/MANIFEST.tsv extracted \
|
||||
--extension "$CANDIDATE_EXTENSION" --prefix-size 4096 \
|
||||
> .run/p2-t5-probe.tsv
|
||||
```
|
||||
|
||||
The reports were byte-identical. They selected 46 manifest-backed outer files, had 46 distinct 64-byte prefix fingerprints, no class-wide shared leading byte or aligned word, and heterogeneous entropy, zero-byte, and generic monotonic-word feature counts. Some later fixed-prefix positions were shared, but their values were not recorded. These measurements neither identify a container boundary/table nor identify compression.
|
||||
|
||||
An initial ad hoc aggregate script used an invalid constant-position predicate. Its zero-shared-position output was explicitly superseded by the synthetic-tested probe and repeated positional-layout survey; no tracked conclusion uses the invalid value.
|
||||
|
||||
## Properly bounded conclusions
|
||||
|
||||
**Verified facts**
|
||||
|
||||
- Static USA evidence supports candidate request/transfer paths, including a separate candidate-class transfer route.
|
||||
- Two PCSX-Redux cold boots observed the same entry-register state.
|
||||
- The documented no-input startup control did not exercise the selected candidate branch/path.
|
||||
- The selected outer files passed manifest-integrity verification and have heterogeneous bounded-prefix measurements.
|
||||
|
||||
**Not established**
|
||||
|
||||
- Physical CD/DMA behavior, request completion, source-to-destination transfer, allocation range, or overlay execution.
|
||||
- Any inner archive/container boundary, table entry layout/meaning, compression format, or decompression behavior.
|
||||
- A runtime general memory map, runtime meaning of static destinations, or a semantic state associated with the observed predecessor argument.
|
||||
|
||||
## Firewall and phase gate
|
||||
|
||||
- Generated Ghidra records, probe reports, runtime traces, memcards, and `.run/` artifacts were checked as regular ignored files.
|
||||
- The Phase 1 Ghidra import remains present alongside the isolated P2 working program.
|
||||
- `.mcp.json` remains pre-existing, untracked, and untouched.
|
||||
- No disc-derived bytes, strings, file listings, disassembly, dumps, or proprietary SDK material were added to tracked files.
|
||||
|
||||
| Gate item | Status |
|
||||
|---|---|
|
||||
| P2-T1 through P2-T5 verification | Complete |
|
||||
| P2-T6 reproducibility record and source/confidence audit | Complete |
|
||||
| Clean synthetic-suite and documentation/firewall checks | Complete after final verification run |
|
||||
| Developer milestone confirmation | Required before Phase 2 closure |
|
||||
|
||||
Do not create a PhaseEnd record, update the digest, archive the current-phase record, or begin Phase 3 until the developer explicitly confirms the Phase 2 milestone.
|
||||
+14
-1
@@ -6,9 +6,22 @@
|
||||
- Ghidra: 12.1.2 at `/opt/ghidra`
|
||||
- PS1 loader: locally built `ghidra_psx_ldr`, installed in `~/.config/ghidra/ghidra_12.1.2_DEV/Extensions/ghidra_psx_ldr/`
|
||||
- Loader source: `/media/christopherwilliams/Data/Projects/decomp/ghidra_psx_ldr`
|
||||
- Runtime debugger/emulator: PCSX-Redux (`pcsx-redux-git`)
|
||||
- Runtime debugger/emulator: PCSX-Redux package `pcsx-redux-git r7541.b63d6bea-1` (installed package metadata)
|
||||
- GDB: package `gdb 17.2-1` / GNU gdb 17.2 (installed package metadata)
|
||||
- Python: package `python 3.14.7-1` / Python 3.14.7 (installed package metadata)
|
||||
- Java used to build the loader: `jdk21-openjdk` at `/usr/lib/jvm/java-21-openjdk`
|
||||
|
||||
## Project-local investigation tools
|
||||
|
||||
- `tools/sf3_extract`: tracked project Python source, standard library only; authored in this repository, with no external tool download. Its extraction/PS-X EXE commands are recorded in [PHASE1_REPRODUCIBILITY.md](PHASE1_REPRODUCIBILITY.md).
|
||||
- `tools/sf3_probe`: tracked project Python source, standard library only; authored in this repository, with no external tool download. Its Phase 2 structural-probe command and synthetic verification are recorded in [PHASE2_INVESTIGATION.md](PHASE2_INVESTIGATION.md).
|
||||
|
||||
Run the project synthetic suite with:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v
|
||||
```
|
||||
|
||||
## Loader rebuild
|
||||
|
||||
```bash
|
||||
|
||||
@@ -17,6 +17,12 @@ Filesystem/executable extraction may therefore read the BIN with a 2352-byte str
|
||||
|
||||
Do not infer later format details from other Syphon Filter games or other regions.
|
||||
|
||||
## Candidate-class structural validation (Phase 2 T5)
|
||||
|
||||
Phase 2 measured only a census-/static-supported candidate file class with the synthetic-tested, read-only `tools/sf3_probe`. Two ignored, byte-identical reports streamed every selected outer file against its extraction-manifest SHA-1 and retained only opaque fixed-prefix features. The class has no shared leading byte or aligned word in the measured 4096-byte prefix and has heterogeneous feature values.
|
||||
|
||||
These are bounded local-USA structural observations, not a container, table, compression, or overlay format identification. In particular, manifest-backed outer file boundaries do not establish inner boundaries, and generic monotonic-word measurements do not establish table semantics. See [PHASE2_INVESTIGATION.md](PHASE2_INVESTIGATION.md) for provenance and limits.
|
||||
|
||||
## PS-X EXE candidate (Phase 1 T3)
|
||||
|
||||
The exact ISO9660 identifier `SCUS_946.40;1` from the local USA image was extracted without renaming. `tools/sf3_extract psx-exe-info` validated its `PS-X EXE` magic, 0x800-byte header, declared text bounds, and streaming SHA-1 against ignored `extracted/MANIFEST.tsv`. The header-field offsets were cross-checked against the locally installed loader source, `ghidra_psx_ldr/src/main/java/psx/PsxExe.java`.
|
||||
|
||||
+14
-3
@@ -1,6 +1,17 @@
|
||||
# Memory Map
|
||||
|
||||
No USA runtime addresses are verified yet.
|
||||
Only the controlled PCSX-Redux register observations below are verified at runtime. They do not establish a general runtime memory map, loader completion, allocation, or overlay execution.
|
||||
|
||||
## PCSX-Redux runtime observations (Phase 2)
|
||||
|
||||
These are register observations from isolated PCSX-Redux/GDB controls using the local USA input. They are distinct from PS-X EXE header declarations and Ghidra loader-created blocks.
|
||||
|
||||
| Address / field | Purpose | Source build / region | Evidence | Verification status |
|
||||
|---|---|---|---|---|
|
||||
| `0x800FB368` | Program counter at the controlled executable entry breakpoint | Local USA input | Two fresh isolated cold boots; packet-level breakpoint acknowledgement and stopped-target register capture | Runtime-observed at entry only; no later execution/mapping inference |
|
||||
| `0x801FFFF0` | Stack-pointer register value at the controlled entry breakpoint | Local USA input | Same two PCSX-Redux/GDB captures | Runtime-observed register value at entry only; not a stack-region map |
|
||||
| `0x00000000` | Global-pointer register value at the controlled entry breakpoint | Local USA input | Same two PCSX-Redux/GDB captures | Runtime-observed register value at entry only; not a general GP claim |
|
||||
| `0x8002C3F8` | Program counter at one selected static-predecessor breakpoint | Local USA input | One isolated no-input cold-boot control; stopped-target capture also recorded first argument register `0x00000001` | Runtime execution observed for that invocation only; no semantic, loader, or memory-map conclusion |
|
||||
|
||||
## Static PS-X EXE header declarations (Phase 1 T3)
|
||||
|
||||
@@ -9,7 +20,7 @@ These entries are header-declared values from the local USA image's exact ISO966
|
||||
| Address / field | Purpose | Source build / region | Evidence | Verification status |
|
||||
|---|---|---|---|---|
|
||||
| `[0x80010000, 0x801DC000)` | Declared text range | Local USA image | PS-X EXE header offsets 0x18/0x1C, parsed by `tools/sf3_extract`; field layout cross-checked with local `ghidra_psx_ldr` source | Header-valid; runtime load status unverified |
|
||||
| `0x800FB368` | Declared entry PC | Local USA image | PS-X EXE header offset 0x10 | Header-valid and within declared text range; runtime execution unverified |
|
||||
| `0x800FB368` | Declared entry PC | Local USA image | PS-X EXE header offset 0x10 | Header-valid and within declared text range; Phase 2 separately observed runtime PC at entry, but the header does not establish runtime state |
|
||||
| `0x00000000` | Initial GP header field | Local USA image | PS-X EXE header offset 0x14 | Header-valid; not a runtime GP observation |
|
||||
| `0x801FFFF0` + `0x00000000` | SP base and SP offset header fields | Local USA image | PS-X EXE header offsets 0x30/0x34 | Header-valid; no effective/runtime SP inference made |
|
||||
|
||||
@@ -29,6 +40,6 @@ The following is the loader-created static program layout for the local USA impo
|
||||
| `0x801227AC`–`0x8014694F` | `.bss`, RW | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
|
||||
| `0x80146950`–`0x801DBFFF` | Second `.text` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
|
||||
| `0x801DC000`–`0x801FFFFF` | Final `RAM` block, RX | Local USA image | `PSX Executables Loader` memory-block report | Loader-created static map; runtime state unverified |
|
||||
| `0x800FB368` | Entry point `start` | Local USA image | Ghidra entry-point report after PSX-loader import | Static loader entry; runtime execution unverified |
|
||||
| `0x800FB368` | Entry point `start` | Local USA image | Ghidra entry-point report after PSX-loader import | Static loader entry; Phase 2 separately observed runtime PC at entry, but the loader-created map remains unverified |
|
||||
|
||||
Record each future entry with: address range, purpose, source build/region, evidence source (Ghidra or runtime observation), and verification status. Do not transfer addresses from other Syphon Filter releases without explicit USA validation.
|
||||
|
||||
@@ -7,3 +7,7 @@ A ROM-safe Git baseline, permanent project context, setup/format/address documen
|
||||
## Phase 1 — Deterministic Extraction, Executable Discovery, and First Ghidra Import (2026-09-23)
|
||||
|
||||
A tracked synthetic-tested MODE2/2352 ISO9660 walker/extractor now produces ignored deterministic file trees and SHA-1 manifests. Two fresh runs produced 96 files and 601,671,468 bytes with byte-identical manifests; all final files re-hashed against the manifest. `SCUS_946.40;1` is header-validated as a PS-X EXE with SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, declared text range `[0x80010000, 0x801DC000)`, and entry PC `0x800FB368`. Ghidra imported it with `PSX Executables Loader` and language `PSX:LE:32:default`; the developer manually observed Program Information `PsyQ Version = 4.5.0` before Auto Analyze. All memory-layout and SDK findings remain static import evidence, not runtime facts. The 19-test suite, firewall checks, and Phase 1 reproducibility record are complete. Phase 2 must begin in a fresh session with an approved plan; runtime loader/overlay/archive behavior remains unknown.
|
||||
|
||||
## Phase 2 — Archive, Loader, and Overlay Investigation (2026-09-23)
|
||||
|
||||
Phase 2 preserved the original Phase 1 Ghidra program and analyzed an isolated USA working copy, yielding static-only candidate request, lookup/queue, and separate candidate-class transfer paths. PCSX-Redux/GDB was configured reproducibly with both `--gdb` and `--debugger`; two isolated cold boots observed the same entry-state PC `0x800FB368`, SP `0x801FFFF0`, and GP `0x00000000`. A selected no-input startup control produced a repeated bounded negative: the direct candidate did not execute in the documented window, while a predecessor executed once without meeting the conditioned downstream branch. A new synthetic-tested read-only `tools/sf3_probe` safely measured the static/census-supported candidate class; two manifest-integrity-verified reports agreed but remained structurally inconclusive. The final suite passed 28 synthetic-only tests, generated artifacts remained ignored, and the Phase 2 investigation record documents all provenance and limits. No loader, allocation, overlay, archive, table, compression, or general runtime memory-map model was established. Future work requires fresh approval and must not promote these bounded findings beyond their evidence.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Phase 2 Plan — Archive, Loader, and Overlay Investigation
|
||||
|
||||
**Status:** approved by developer
|
||||
**Planning effort:** Max
|
||||
|
||||
## Goal
|
||||
|
||||
Establish an evidence-backed USA model of disc-backed archive access, executable loader behavior, and any observed overlay allocation/loading behavior. Static Ghidra findings and PCSX-Redux observations must remain separately provenance-labelled until they corroborate each other.
|
||||
|
||||
## Preconditions established in Phase 1
|
||||
|
||||
- The local USA image has a deterministic ignored extraction tree and manifest.
|
||||
- `SCUS_946.40;1` is header-validated and imported into an ignored Ghidra project with `PSX Executables Loader`.
|
||||
- The Phase 1 import remains unanalyzed; its loader/import facts are recorded separately from runtime facts.
|
||||
- Ghidra and PCSX-Redux are available as the static and runtime oracles, respectively.
|
||||
|
||||
## Scope and safeguards
|
||||
|
||||
- Keep all disc-derived reports, snapshots, emulator logs, Ghidra databases, and probe output in ignored locations.
|
||||
- Do not copy game bytes, disassembly listings, extracted directory listings, or ROM-derived fixtures into tracked sources or documentation.
|
||||
- New tracked probing code, if needed, must use synthetic-only test inputs and must not make a format claim solely from a filename, extension, or unverified signature.
|
||||
- Preserve the Phase 1 Ghidra import; perform any Auto Analyze or exploratory changes in a separate ignored working copy.
|
||||
- A negative result is valid evidence when its control sequence, instrumentation, and limits are documented.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **P2-T1 — Metadata-only asset census and probe foundation** *(xHigh)*
|
||||
- Produce a deterministic, ignored census of extracted-file metadata and safely identify possible archive/container candidates without placing source bytes or listings in tracked output.
|
||||
- Add a read-only probe only if existing tooling cannot provide the required evidence; any new probe receives synthetic-only tests.
|
||||
- **Verify:** relevant synthetic tests pass; two fresh census/probe runs agree; generated output is ignored; no format conclusion is promoted without further evidence.
|
||||
|
||||
- [ ] **P2-T2 — Static loader and archive-access survey** *(xHigh)*
|
||||
- Analyze a separate ignored copy of the USA Ghidra import to locate and characterize candidate CD/file/archive/overlay-loading paths and their references.
|
||||
- Record only interpreted, provenance-labelled evidence and uncertainty; retain the original Phase 1 import unchanged.
|
||||
- **Verify:** each candidate is reviewable in Ghidra through references or dataflow evidence; all addresses are labelled static USA evidence; no disassembly is tracked.
|
||||
|
||||
- [ ] **P2-T3 — PCSX-Redux runtime-observation preflight** *(xHigh)*
|
||||
- Establish a reproducible cold-boot/debugger procedure and observe the executable's runtime entry/load state without treating header or loader declarations as runtime facts.
|
||||
- **Verify:** two controlled cold boots produce consistent recorded observations; captures remain ignored; any unavailable debugger capability is recorded as a blocker rather than inferred around.
|
||||
|
||||
- [ ] **P2-T4 — Controlled loader/overlay transition trace** *(xHigh)*
|
||||
- Use static candidates and a documented controlled runtime sequence to observe candidate load transitions, relevant allocation ranges, and source/container relationships.
|
||||
- Correlate observations with Ghidra candidates while retaining conflicting or absent evidence as such.
|
||||
- **Verify:** repeat the selected transition and compare before/after observations, or record a bounded, reproducible negative result with its controls.
|
||||
|
||||
- [ ] **Rules check**
|
||||
- Re-read `AGENTS.md` mandatory behavior after P2-T4.
|
||||
|
||||
- [ ] **P2-T5 — Archive-structure validation** *(xHigh)*
|
||||
- Investigate only candidates supported by the census, static survey, or runtime trace; distinguish container boundaries, table semantics, and compression hypotheses from verified facts.
|
||||
- Extend any parser/probe only through synthetic fixtures and tests.
|
||||
- **Verify:** synthetic tests pass; claimed behavior has explicit USA static and/or runtime evidence; unresolved interpretations remain labelled hypotheses.
|
||||
|
||||
- [ ] **P2-T6 — Investigation record and phase gate** *(xHigh)*
|
||||
- Record reproducible non-content methods and findings in `docs/PHASE2_INVESTIGATION.md`, updating format and memory-map records only for properly sourced facts.
|
||||
- Prepare, but do not close, the phase until all checks are reviewed and the developer confirms the milestone.
|
||||
- **Verify:** documented clean checks pass, ignored-artifact/firewall review is clean, and every conclusion identifies its source and confidence.
|
||||
|
||||
## Milestone
|
||||
|
||||
A reproducible, provenance-aware record identifies the observed USA disc-loading path and its runtime behavior to the extent supported by evidence; it either validates an archive/overlay model through static and runtime correlation or documents bounded negative/inconclusive results. No matching-C claim or build-pipeline work is part of this phase.
|
||||
@@ -0,0 +1,72 @@
|
||||
# PhaseEnd — Phase 2: Archive, Loader, and Overlay Investigation
|
||||
|
||||
**Date:** 2026-09-23
|
||||
**Phase Status:** Complete
|
||||
**Milestone confirmed by developer:** yes
|
||||
|
||||
## Completed Checklist
|
||||
|
||||
- P2-T1 — Completed two deterministic metadata-only censuses using the existing manifest interface.
|
||||
- P2-T2 — Preserved the Phase 1 Ghidra program and surveyed candidate request/transfer paths in a separate ignored USA working program.
|
||||
- P2-T3 — Established an isolated PCSX-Redux/GDB cold-boot procedure and captured matching runtime entry-state registers twice.
|
||||
- P2-T4 — Completed a repeated, bounded no-input cold-boot negative for the selected static candidate chain.
|
||||
- Rules check — Re-read `AGENTS.md` mandatory behavior after P2-T4.
|
||||
- P2-T5 — Added the synthetic-tested, read-only `tools/sf3_probe` and completed structurally inconclusive candidate-class validation.
|
||||
- P2-T6 — Wrote the reproducible Phase 2 record, updated properly sourced format/memory-map/setup facts, and prepared the review gate.
|
||||
|
||||
## Verified Results
|
||||
|
||||
### Static USA investigation
|
||||
|
||||
- The preserved `/SCUS_946.40;1` Phase 1 Ghidra program remains unchanged; all Phase 2 analysis used ignored `/P2-analysis-SCUS_946.40;1`.
|
||||
- Static USA candidate paths identify request construction, pathname lookup/queued-transfer candidates, and a separate candidate-class transfer route.
|
||||
- The static route supports candidate transfer setup only. It does not establish physical transfer, completion, allocation, overlay execution, inner container boundaries, table semantics, or compression.
|
||||
|
||||
### PCSX-Redux runtime observations
|
||||
|
||||
- The corrected isolated procedure requires both `--gdb` and `--debugger` with CLI/interpreter/no-fastboot operation.
|
||||
- Two fresh cold boots acknowledged the entry breakpoint and captured identical runtime values: PC `0x800FB368`, SP `0x801FFFF0`, and GP `0x00000000`.
|
||||
- In the selected no-input P2-T4 control, static candidate `0x8002C22C` did not stop during 75 seconds. Its static predecessor `0x8002C3F8` was observed once with first argument register `0x00000001`.
|
||||
- Two independent 100-second conditioned predecessor controls each recorded six internal condition-evaluation stops and no matching branch stop. No downstream candidate was armed.
|
||||
- These are entry-state and bounded-control observations only. They do not establish a runtime loader, allocation, overlay, source/destination transfer, or general memory map.
|
||||
|
||||
### Candidate-class structural validation
|
||||
|
||||
- `tools/sf3_probe` is dependency-free tracked Python source with 9 synthetic-only tests. It safely validates manifest/path/file/size/SHA-1 conditions and emits opaque fixed-prefix features only.
|
||||
- The complete tools suite passed 28 synthetic-only tests.
|
||||
- Two ignored local-USA reports were byte-identical after streaming all selected candidates against their manifest SHA-1 values. The selected class had heterogeneous fixed-prefix features and no class-wide shared leading byte or aligned word.
|
||||
- These measurements do not identify a container, inner boundary, table, compression format, decompressor, or overlay format. An early ad hoc zero-shared-position aggregation error was corrected and explicitly superseded; it informed no tracked conclusion.
|
||||
|
||||
## Verification and Firewall
|
||||
|
||||
- `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests -v` passed all 28 synthetic-only tests.
|
||||
- Repeated ignored probe and positional-layout reports compared byte-identically.
|
||||
- `git diff --check` and unstaged-index checks passed before closure staging.
|
||||
- Generated Ghidra records, runtime logs, memcards, `.run/` reports, and local extraction material were confirmed regular ignored files.
|
||||
- PCSX-Redux and GDB port 3333 were absent after runtime work.
|
||||
- The preserved Phase 1 and isolated Phase 2 Ghidra programs remain open. No disc-derived bytes, strings, listings, disassembly, dumps, or proprietary SDK material were staged.
|
||||
- The pre-existing local `.mcp.json` remains untracked and untouched.
|
||||
|
||||
## Deviations and Bounded Results
|
||||
|
||||
| Item | Plan | Actual | Reason / limit |
|
||||
|---|---|---|---|
|
||||
| Initial runtime debugger control | Observe entry state through GDB | Initial trials exposed an incorrect BIOS path and an execution-breakpoint control gap | The corrected procedure used a regular BIOS file and `--debugger`; only the two matching corrected runs were accepted as evidence. |
|
||||
| P2-T4 transition trace | Observe a selected loader/overlay transition | Repeated bounded negative on the documented no-input cold boot | The selected static path was not reached in that window; this does not generalize to other states or inputs. |
|
||||
| P2-T5 structure validation | Validate a candidate archive/container model if supported | Structurally inconclusive result | Static transfer setup and fixed-prefix heuristics do not validate a format model. |
|
||||
|
||||
## Rules Added This Phase
|
||||
|
||||
None.
|
||||
|
||||
## Next
|
||||
|
||||
No next phase begins in this session. A future phase requires a fresh session, a task-by-task plan, explicit developer approval, and evidence that does not promote Phase 2 static or bounded runtime findings beyond their recorded limits.
|
||||
|
||||
## Plain-English Recap
|
||||
|
||||
Phase 2 established a reproducible static/runtime investigation method, verified the executable's controlled entry register state, and bounded one startup candidate path. It did not prove a loader, overlay, allocation, archive, table, or compression model. The project now has a tested safe structural probe and a provenance-aware record of what remains unknown.
|
||||
|
||||
## 🛑 Stop Here
|
||||
|
||||
Phase 2 is closed. `CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase2.md`; this PhaseEnd and the digest update belong in the closure commit. Do not begin Phase 3 in this session.
|
||||
@@ -0,0 +1,46 @@
|
||||
# CURRENT_PHASE — Phase 2: Archive, Loader, and Overlay Investigation
|
||||
|
||||
**Status:** awaiting developer milestone confirmation — Phase 2 gate prepared; do not close or begin Phase 3
|
||||
**Plan:** `phase-ends/Phase2_PLAN.md` (approved by developer)
|
||||
|
||||
## Tasks
|
||||
|
||||
- [x] **P2-T1 — Metadata-only asset census and probe foundation:** completed with the existing validated manifest interface; two fresh aggregate-only censuses agreed, and no new probe was needed or added.
|
||||
- [x] **P2-T2 — Static loader and archive-access survey:** preserved the Phase 1 import and used a separate ignored Ghidra working database to identify reviewable, static-only candidate request and transfer paths.
|
||||
- [x] **P2-T3 — PCSX-Redux runtime-observation preflight:** completed through approved P2-T3-R1; two isolated PCSX-GDB cold boots produced matching entry-state register captures.
|
||||
- [x] **P2-T4 — Controlled loader/overlay transition trace:** completed as a repeated bounded negative for the documented no-input cold-boot sequence; no runtime loader/overlay conclusion was made.
|
||||
- [x] **Rules check:** re-read `AGENTS.md` mandatory behavior after P2-T4.
|
||||
- [x] **P2-T5 — Archive-structure validation:** completed with a synthetic-tested, read-only probe and a static revisit; the supported candidate class remains structurally inconclusive and no format claim was made.
|
||||
- [x] **P2-T6 — Investigation record and phase gate:** completed; reproducible non-content documentation and the review gate are prepared, while Phase 2 remains open pending developer confirmation.
|
||||
|
||||
## Session checkpoint
|
||||
|
||||
Phase 2 was approved in a fresh session after the completed Phase 1 close. The Phase 1 extraction, static PS-X EXE facts, and unanalysed Ghidra import are baseline evidence only; none establishes runtime loader, archive, or overlay behavior.
|
||||
|
||||
P2-T1 is complete. The existing `MANIFEST.tsv` interface was sufficient for a deterministic metadata-only census, so no source tool or format probe was added. The existing synthetic-only suite passed all 19 tests. Two fresh ignored reports agreed; their schema contains only aggregate counts, numeric spans, generic extension aggregates, and size bins, with no paths or digests. The reports and baseline test log are regular ignored files. Candidate classes remain unverified metadata only; no archive/container format conclusion has been made.
|
||||
|
||||
P2-T2 is complete. The preserved import remains at one recovered function with its default PsyQ analyzer option; the separately imported ignored working database recovered 1,721 functions after full analysis with only `PsyQ Signatures` disabled. The static candidate request/lookup/queue path, a separate candidate-class access path, breakpoint candidates, and uncertainty are recorded without raw strings or disassembly in ignored `ghidra/P2-T2-static-survey.md`. The MCP adapter's post-analysis `Analyzed Flag: false` conflicts with its completed-task result and recovered-function count; this remains an adapter/state limitation. All P2-T2 findings are static USA hypotheses only.
|
||||
|
||||
P2-T3 initially blocked before approved recovery. Read-only preflight established the installed PCSX-Redux package, interpreter/GDB control flags from its upstream source at the installed revision, a readable local disc, a readable configured BIOS file, free GDB port 3333, and an isolated CLI/safe-mode launch design. Trial A mounted the disc and accepted a GDB connection but passed a configured BIOS browse directory rather than the regular BIOS file; it was terminated and corrected. Trial B used the regular BIOS file, mounted the disc, and reached the normal executable-handoff stage according to ignored emulator logging, but the pre-set GDB entry breakpoint did not halt execution and GDB timed out while the target was running. No PC/SP/GP observation was accepted from console output.
|
||||
|
||||
Both trial roots, fresh memcards, GDB command files, and logs are ignored under `.run/p2-t3-cold-*`; each process and port-3333 listener was confirmed absent after cleanup. These are two distinct root-cause attempts. The prior stop condition remains recorded.
|
||||
|
||||
The developer explicitly approved **P2-T3-R1** after source inspection at installed PCSX-Redux revision `b63d6bea` established that `--gdb` starts the remote server but execution-breakpoint processing in the interpreter is gated separately by `--debugger`. R1 completed with exactly two fresh isolated CLI/safe-mode cold boots using a regular BIOS file, interpreter, no fast boot, `--debugger`, and GDB on port 3333. Each ignored GDB packet trace confirmed `OK` for execution-breakpoint insertion, then halted at the header-declared entry address. The sanitized summaries are byte-identical: PC `0x800FB368`, SP `0x801FFFF0`, and GP `0x00000000`.
|
||||
|
||||
These are PCSX-Redux GDB runtime observations for the local USA input at the controlled entry breakpoint. They verify the observed entry state only; they do not establish later register values, loader completion, allocation, overlay execution, archive behavior, or a general runtime memory map. Both run roots, fresh memcards, logs, packet traces, and summaries remain ignored under `.run/p2-t3-r1-*`; process/listener cleanup and artifact firewall checks passed.
|
||||
|
||||
P2-T4 startup trace R2-A produced a bounded negative result. The isolated corrected debugger path acknowledged insertion of the first static candidate breakpoint at `0x8002C22C`, but no stop packet or hit marker occurred during the 75-second no-input cold-boot window. The trace therefore establishes only that this startup window did not reach that candidate; it does not reject the static path. Its root, GDB packet trace, and emulator log are ignored under `.run/p2-t4-r2-a.*`, and cleanup/firewall checks passed.
|
||||
|
||||
P2-T4 predecessor probe R3-A then established that the same cold boot does execute `0x8002C3F8`: breakpoint insertion was acknowledged and the first stopped invocation captured PC `0x8002C3F8` with first argument register `0x00000001`. This is a PCSX-GDB runtime register observation only; it does not assign a semantic state name or prove that the branch leading to `0x8002C22C` executed. The ignored root, packet trace, and sanitized register summary are under `.run/p2-t4-r3-a.*`, with cleanup/firewall checks passed.
|
||||
|
||||
P2-T4 conditional trace R4-A acknowledged a breakpoint at `0x8002C3F8` with a client-side condition for the static branch value associated with `0x8002C22C`. The bounded 100-second no-input cold boot produced internal GDB stop packets while the condition was evaluated, but no matching-condition hit marker; therefore no downstream candidate was armed. This is a bounded negative for that exact controlled condition, not proof that the path is globally unreachable. The ignored root, packet trace, and emulator log are under `.run/p2-t4-r4-a.*`, and cleanup/firewall checks passed.
|
||||
|
||||
P2-T4 is complete as a repeated bounded negative. Under the exact no-input, isolated cold-boot control, direct observation of `0x8002C22C` did not occur in 75 seconds; its immediate static predecessor was observed once with first argument register `0x00000001`; and two independent 100-second conditioned controls at that predecessor produced six internal GDB condition-evaluation stops each without a matching branch value. No downstream candidate was armed in either control. These results establish the limit of this control sequence only: they neither prove nor reject loader, archive, allocation, or overlay behavior in other runtime states. All traces, fresh memcards, logs, and summaries remain ignored; repeated cleanup/firewall checks passed.
|
||||
|
||||
Rules check — re-read complete. Continuing with P2-T5 — Archive-structure validation.
|
||||
|
||||
P2-T5 is complete as a provenance-separated, structurally inconclusive result. The new dependency-free `tools/sf3_probe` has synthetic-only coverage for deterministic opaque output and manifest/path/size/digest safety; its focused suite passed 9 tests and the complete tools suite passed 28 tests before local-input use. Two byte-identical ignored reports streamed all selected candidates against their manifest SHA-1 values and measured fixed-prefix properties without paths or source bytes. The static revisit found only a candidate-class transfer setup, not a validated inner parser. The selected class has no uniform leading shared byte/word across the measured prefix and heterogeneous generic features; this neither identifies a container/table nor compression. An earlier ad hoc zero-shared-position result had a faulty predicate, is explicitly superseded, and was not used for a tracked conclusion. The non-content ignored record is `ghidra/P2-T5-archive-validation.md`; report and firewall checks passed.
|
||||
|
||||
P2-T6 is complete. `docs/PHASE2_INVESTIGATION.md` records the provenance-separated methods, static candidates, runtime entry observation, bounded runtime negative, structural probe limit, firewall review, and explicit unknowns. `docs/formats.md` records only the structurally inconclusive candidate-class observation; `docs/memory-map.md` separately records the controlled runtime register observations; and `docs/SETUP.md` records the project-local probe and installed runtime-tool metadata. Final verification passed the 28-test synthetic suite, repeated ignored-report comparisons, documentation/source firewall checks, phase-closure guard, runtime cleanup, whitespace checks, and unstaged-index check. The preserved Phase 1 and isolated P2 Ghidra programs remain open.
|
||||
|
||||
Current task: **Phase 2 milestone gate — awaiting explicit developer confirmation.** Do not write a PhaseEnd record, update the digest, archive this checkpoint, close Phase 2, or begin Phase 3 until confirmation is received.
|
||||
Executable
+361
@@ -0,0 +1,361 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only, opaque structural feature probes for extracted disc files.
|
||||
|
||||
The probe validates a manifest-tracked extraction root before reading fixed-size
|
||||
prefixes. Successful reports use deterministic opaque candidate identifiers and
|
||||
never emit source paths or raw source bytes. It intentionally reports heuristics,
|
||||
not file-format conclusions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import math
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
from typing import Sequence
|
||||
|
||||
|
||||
MANIFEST_HEADER = "path\tlba\textents\tsize\tsha1"
|
||||
SAFE_COMPONENT = re.compile(r"[A-Za-z0-9._;-]+\Z")
|
||||
ISO_VERSION_SUFFIX = re.compile(r";[0-9]+\Z")
|
||||
SHA1_PATTERN = re.compile(r"[0-9a-f]{40}\Z")
|
||||
MAX_PREFIX_SIZE = 1024 * 1024
|
||||
|
||||
|
||||
class ProbeValidationError(ValueError):
|
||||
"""Raised when probe input cannot safely support a structural survey."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManifestEntry:
|
||||
"""One validated manifest row, retained only for local probe selection."""
|
||||
|
||||
path: str
|
||||
lba: int
|
||||
size: int
|
||||
sha1: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CandidateFeatures:
|
||||
"""Non-content structural measurements for one opaque candidate."""
|
||||
|
||||
identifier: str
|
||||
size: int
|
||||
prefix_size: int
|
||||
prefix_sha1: str
|
||||
entropy_milli: int
|
||||
zero_bytes: int
|
||||
monotonic_u32_run_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PrefixFeatureReport:
|
||||
"""A deterministic, opaque fixed-prefix feature report."""
|
||||
|
||||
extension: str
|
||||
candidate_count: int
|
||||
common_prefix_span: int
|
||||
constant_positions: int
|
||||
candidates: tuple[CandidateFeatures, ...]
|
||||
|
||||
|
||||
def _validate_manifest_path(path: str, context: str) -> None:
|
||||
if not path:
|
||||
raise ProbeValidationError(f"{context}: empty path")
|
||||
for component in path.split("/"):
|
||||
if not component:
|
||||
raise ProbeValidationError(f"{context}: empty path component")
|
||||
if component in {".", ".."}:
|
||||
raise ProbeValidationError(f"{context}: unsafe path component")
|
||||
try:
|
||||
component.encode("ascii")
|
||||
except UnicodeEncodeError as exc:
|
||||
raise ProbeValidationError(f"{context}: non-ASCII path component") from exc
|
||||
if SAFE_COMPONENT.fullmatch(component) is None:
|
||||
raise ProbeValidationError(f"{context}: unsafe path component")
|
||||
|
||||
|
||||
def _parse_extents(field: str, context: str) -> tuple[tuple[int, int], ...]:
|
||||
if not field:
|
||||
raise ProbeValidationError(f"{context}: empty extent field")
|
||||
extents: list[tuple[int, int]] = []
|
||||
for encoded_extent in field.split(","):
|
||||
lba_text, separator, size_text = encoded_extent.partition(":")
|
||||
if not separator or ":" in size_text:
|
||||
raise ProbeValidationError(f"{context}: malformed extent")
|
||||
if not lba_text.isdecimal() or not size_text.isdecimal():
|
||||
raise ProbeValidationError(f"{context}: non-decimal extent")
|
||||
extents.append((int(lba_text), int(size_text)))
|
||||
return tuple(extents)
|
||||
|
||||
|
||||
def read_manifest(manifest_path: str | Path) -> tuple[ManifestEntry, ...]:
|
||||
"""Read a strict extraction manifest without returning hashes or source data."""
|
||||
manifest = Path(manifest_path)
|
||||
if manifest.is_symlink():
|
||||
raise ProbeValidationError("manifest must not be a symbolic link")
|
||||
try:
|
||||
text = manifest.read_text(encoding="ascii")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ProbeValidationError("manifest is not ASCII") from exc
|
||||
except OSError as exc:
|
||||
raise ProbeValidationError(f"cannot read manifest: {exc}") from exc
|
||||
|
||||
lines = text.splitlines()
|
||||
if not lines or lines[0] != MANIFEST_HEADER:
|
||||
raise ProbeValidationError("manifest has an unexpected header")
|
||||
|
||||
entries: list[ManifestEntry] = []
|
||||
seen_paths: set[str] = set()
|
||||
for line_number, line in enumerate(lines[1:], start=2):
|
||||
fields = line.split("\t")
|
||||
if len(fields) != 5:
|
||||
raise ProbeValidationError(f"manifest line {line_number}: expected five fields")
|
||||
path, lba_text, extent_text, size_text, sha1 = fields
|
||||
context = f"manifest line {line_number}"
|
||||
_validate_manifest_path(path, context)
|
||||
if path in seen_paths:
|
||||
raise ProbeValidationError(f"{context}: duplicate path")
|
||||
seen_paths.add(path)
|
||||
if not lba_text.isdecimal() or not size_text.isdecimal():
|
||||
raise ProbeValidationError(f"{context}: non-decimal LBA or size")
|
||||
if SHA1_PATTERN.fullmatch(sha1) is None:
|
||||
raise ProbeValidationError(f"{context}: malformed SHA-1")
|
||||
extents = _parse_extents(extent_text, context)
|
||||
lba = int(lba_text)
|
||||
size = int(size_text)
|
||||
if extents[0][0] != lba:
|
||||
raise ProbeValidationError(f"{context}: first extent does not match LBA")
|
||||
if sum(extent_size for _extent_lba, extent_size in extents) != size:
|
||||
raise ProbeValidationError(f"{context}: extents do not sum to size")
|
||||
entries.append(ManifestEntry(path=path, lba=lba, size=size, sha1=sha1))
|
||||
return tuple(entries)
|
||||
|
||||
|
||||
def _normalize_extension(extension: str) -> str:
|
||||
normalized = extension.strip()
|
||||
if normalized.startswith("."):
|
||||
normalized = normalized[1:]
|
||||
if not normalized or re.fullmatch(r"[A-Za-z0-9]+", normalized) is None:
|
||||
raise ProbeValidationError("extension must contain only ASCII letters and digits")
|
||||
return normalized.upper()
|
||||
|
||||
|
||||
def _entry_extension(path: str) -> str | None:
|
||||
leaf = ISO_VERSION_SUFFIX.sub("", path.rsplit("/", 1)[-1])
|
||||
if "." not in leaf:
|
||||
return None
|
||||
return leaf.rsplit(".", 1)[1].upper()
|
||||
|
||||
|
||||
def _resolve_root(root_path: str | Path) -> Path:
|
||||
root = Path(root_path)
|
||||
if root.is_symlink():
|
||||
raise ProbeValidationError("extraction root must not be a symbolic link")
|
||||
try:
|
||||
resolved = root.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ProbeValidationError(f"cannot resolve extraction root: {exc}") from exc
|
||||
if not resolved.is_dir():
|
||||
raise ProbeValidationError("extraction root is not a directory")
|
||||
return resolved
|
||||
|
||||
|
||||
def _resolve_candidate(root: Path, entry: ManifestEntry) -> Path:
|
||||
candidate = root / entry.path
|
||||
if candidate.is_symlink():
|
||||
raise ProbeValidationError("candidate must not be a symbolic link")
|
||||
try:
|
||||
resolved = candidate.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ProbeValidationError(f"cannot resolve candidate: {exc}") from exc
|
||||
try:
|
||||
resolved.relative_to(root)
|
||||
except ValueError as exc:
|
||||
raise ProbeValidationError("candidate escapes extraction root") from exc
|
||||
if not resolved.is_file():
|
||||
raise ProbeValidationError("candidate is not a regular file")
|
||||
try:
|
||||
actual_size = resolved.stat().st_size
|
||||
except OSError as exc:
|
||||
raise ProbeValidationError(f"cannot stat candidate: {exc}") from exc
|
||||
if actual_size != entry.size:
|
||||
raise ProbeValidationError("candidate size does not match manifest")
|
||||
return resolved
|
||||
|
||||
|
||||
def _read_verified_prefix(candidate: Path, entry: ManifestEntry, prefix_size: int) -> bytes:
|
||||
"""Read a bounded prefix while streaming a complete manifest-integrity check."""
|
||||
digest = hashlib.sha1()
|
||||
prefix = bytearray()
|
||||
try:
|
||||
with candidate.open("rb") as source:
|
||||
while chunk := source.read(64 * 1024):
|
||||
digest.update(chunk)
|
||||
remaining = prefix_size - len(prefix)
|
||||
if remaining > 0:
|
||||
prefix.extend(chunk[:remaining])
|
||||
except OSError as exc:
|
||||
raise ProbeValidationError(f"cannot read candidate: {exc}") from exc
|
||||
if digest.hexdigest() != entry.sha1:
|
||||
raise ProbeValidationError("candidate SHA-1 does not match manifest")
|
||||
return bytes(prefix)
|
||||
|
||||
|
||||
def _entropy_milli(data: bytes) -> int:
|
||||
if not data:
|
||||
return 0
|
||||
counts = Counter(data)
|
||||
entropy = -sum(
|
||||
(count / len(data)) * math.log2(count / len(data)) for count in counts.values()
|
||||
)
|
||||
return round(entropy * 1000)
|
||||
|
||||
|
||||
def _monotonic_u32_run_count(prefix: bytes, file_size: int) -> int:
|
||||
"""Count an intentionally generic offset-table heuristic in one prefix."""
|
||||
count = 0
|
||||
for offset in range(0, len(prefix) - 15, 4):
|
||||
words = tuple(
|
||||
int.from_bytes(prefix[offset + index : offset + index + 4], "little")
|
||||
for index in range(0, 16, 4)
|
||||
)
|
||||
if all(word < file_size for word in words) and words == tuple(sorted(words)):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def probe_prefix_features(
|
||||
manifest_path: str | Path,
|
||||
root_path: str | Path,
|
||||
*,
|
||||
extension: str,
|
||||
prefix_size: int = 4096,
|
||||
) -> PrefixFeatureReport:
|
||||
"""Measure opaque fixed-prefix features for one manifest-selected extension."""
|
||||
if not 1 <= prefix_size <= MAX_PREFIX_SIZE:
|
||||
raise ProbeValidationError(
|
||||
f"prefix size must be between 1 and {MAX_PREFIX_SIZE} bytes"
|
||||
)
|
||||
normalized_extension = _normalize_extension(extension)
|
||||
root = _resolve_root(root_path)
|
||||
entries = sorted(
|
||||
(entry for entry in read_manifest(manifest_path) if _entry_extension(entry.path) == normalized_extension),
|
||||
key=lambda entry: entry.path,
|
||||
)
|
||||
if not entries:
|
||||
raise ProbeValidationError("manifest has no candidates for the requested extension")
|
||||
|
||||
prefixes: list[bytes] = []
|
||||
candidates: list[CandidateFeatures] = []
|
||||
for index, entry in enumerate(entries, start=1):
|
||||
candidate = _resolve_candidate(root, entry)
|
||||
prefix = _read_verified_prefix(candidate, entry, prefix_size)
|
||||
prefixes.append(prefix)
|
||||
candidates.append(
|
||||
CandidateFeatures(
|
||||
identifier=f"C{index:03d}",
|
||||
size=entry.size,
|
||||
prefix_size=len(prefix),
|
||||
prefix_sha1=hashlib.sha1(prefix).hexdigest(),
|
||||
entropy_milli=_entropy_milli(prefix),
|
||||
zero_bytes=prefix.count(0),
|
||||
monotonic_u32_run_count=_monotonic_u32_run_count(prefix, entry.size),
|
||||
)
|
||||
)
|
||||
|
||||
common_prefix_span = min(len(prefix) for prefix in prefixes)
|
||||
constant_positions = sum(
|
||||
all(prefix[position] == prefixes[0][position] for prefix in prefixes)
|
||||
for position in range(common_prefix_span)
|
||||
)
|
||||
return PrefixFeatureReport(
|
||||
extension=normalized_extension,
|
||||
candidate_count=len(candidates),
|
||||
common_prefix_span=common_prefix_span,
|
||||
constant_positions=constant_positions,
|
||||
candidates=tuple(candidates),
|
||||
)
|
||||
|
||||
|
||||
def format_prefix_features(report: PrefixFeatureReport) -> str:
|
||||
"""Format a deterministic report without candidate paths or raw bytes."""
|
||||
lines = [
|
||||
"format\tsf3_probe-prefix-features-v1",
|
||||
f"candidate_count\t{report.candidate_count}",
|
||||
f"common_prefix_span\t{report.common_prefix_span}",
|
||||
f"constant_positions_0_common_prefix_span\t{report.constant_positions}",
|
||||
]
|
||||
for candidate in report.candidates:
|
||||
lines.extend(
|
||||
[
|
||||
f"candidate\t{candidate.identifier}\tsize\t{candidate.size}",
|
||||
f"candidate\t{candidate.identifier}\tprefix_size\t{candidate.prefix_size}",
|
||||
f"candidate\t{candidate.identifier}\tprefix_sha1\t{candidate.prefix_sha1}",
|
||||
f"candidate\t{candidate.identifier}\tentropy_milli\t{candidate.entropy_milli}",
|
||||
f"candidate\t{candidate.identifier}\tzero_bytes\t{candidate.zero_bytes}",
|
||||
"candidate\t"
|
||||
f"{candidate.identifier}\tmonotonic_u32_run_count\t"
|
||||
f"{candidate.monotonic_u32_run_count}",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _positive_prefix_size(value: str) -> int:
|
||||
try:
|
||||
prefix_size = int(value, 10)
|
||||
except ValueError as exc:
|
||||
raise argparse.ArgumentTypeError("must be an integer") from exc
|
||||
if not 1 <= prefix_size <= MAX_PREFIX_SIZE:
|
||||
raise argparse.ArgumentTypeError(
|
||||
f"must be between 1 and {MAX_PREFIX_SIZE}"
|
||||
)
|
||||
return prefix_size
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Produce opaque, read-only structural feature reports for manifest-tracked files."
|
||||
)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
prefix_parser = commands.add_parser(
|
||||
"prefix-features",
|
||||
help="measure fixed-prefix heuristics for one manifest-selected extension",
|
||||
)
|
||||
prefix_parser.add_argument("manifest", type=Path, help="validated extraction manifest")
|
||||
prefix_parser.add_argument("root", type=Path, help="matching extraction root")
|
||||
prefix_parser.add_argument(
|
||||
"--extension",
|
||||
required=True,
|
||||
help="extension class to select, without an ISO version suffix",
|
||||
)
|
||||
prefix_parser.add_argument(
|
||||
"--prefix-size",
|
||||
type=_positive_prefix_size,
|
||||
default=4096,
|
||||
help="bytes read from each candidate (default: 4096)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
report = probe_prefix_features(
|
||||
args.manifest,
|
||||
args.root,
|
||||
extension=args.extension,
|
||||
prefix_size=args.prefix_size,
|
||||
)
|
||||
except ProbeValidationError as exc:
|
||||
print(f"sf3_probe: error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(format_prefix_features(report))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,253 @@
|
||||
"""Synthetic-only tests for opaque manifest-backed structural probing.
|
||||
|
||||
Fixtures are created in temporary directories. They never inspect the local disc,
|
||||
extracted game files, or any game-derived probe report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
TOOL_PATH = Path(__file__).resolve().parents[1] / "sf3_probe"
|
||||
|
||||
|
||||
def _load_tool() -> object:
|
||||
loader = importlib.machinery.SourceFileLoader("sf3_probe_under_test", str(TOOL_PATH))
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
if spec is None:
|
||||
raise RuntimeError("could not create an import specification for sf3_probe")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
sf3_probe = _load_tool()
|
||||
|
||||
|
||||
def _manifest_row(
|
||||
path: str,
|
||||
lba: int,
|
||||
contents: bytes,
|
||||
*,
|
||||
size: int | None = None,
|
||||
sha1: str | None = None,
|
||||
) -> str:
|
||||
declared_size = len(contents) if size is None else size
|
||||
digest = hashlib.sha1(contents).hexdigest() if sha1 is None else sha1
|
||||
return f"{path}\t{lba}\t{lba}:{declared_size}\t{declared_size}\t{digest}\n"
|
||||
|
||||
|
||||
def _write_manifest(root: Path, rows: list[tuple[str, bytes]]) -> Path:
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
manifest_rows = [sf3_probe.MANIFEST_HEADER + "\n"]
|
||||
for index, (relative_path, contents) in enumerate(rows, start=1):
|
||||
output = root / relative_path
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_bytes(contents)
|
||||
manifest_rows.append(_manifest_row(relative_path, 100 + index, contents))
|
||||
manifest = root / "MANIFEST.tsv"
|
||||
manifest.write_text("".join(manifest_rows), encoding="ascii")
|
||||
return manifest
|
||||
|
||||
|
||||
class Sf3ProbeTests(unittest.TestCase):
|
||||
def temporary_root(self) -> Path:
|
||||
temporary_directory = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(temporary_directory.cleanup)
|
||||
return Path(temporary_directory.name) / "extracted"
|
||||
|
||||
def test_prefix_features_are_deterministic_and_opaque(self) -> None:
|
||||
root = self.temporary_root()
|
||||
alpha = b"\x04\x00\x00\x00" + b"ALPHA-SYNTHETIC" + b"\x00" * 20
|
||||
beta = b"\x04\x00\x00\x00" + b"BETA-SYNTHETIC!" + b"\x01" * 20
|
||||
manifest = _write_manifest(
|
||||
root,
|
||||
[
|
||||
("NEST/ALPHA.HOG;1", alpha),
|
||||
("BETA.HOG;1", beta),
|
||||
("IGNORE.DAT;1", b"not-selected"),
|
||||
],
|
||||
)
|
||||
|
||||
first = sf3_probe.probe_prefix_features(
|
||||
manifest, root, extension="HOG", prefix_size=32
|
||||
)
|
||||
second = sf3_probe.probe_prefix_features(
|
||||
manifest, root, extension=".hog", prefix_size=32
|
||||
)
|
||||
first_text = sf3_probe.format_prefix_features(first)
|
||||
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(first.candidate_count, 2)
|
||||
self.assertEqual(first.common_prefix_span, 32)
|
||||
self.assertEqual(first.constant_positions, 4)
|
||||
self.assertEqual([candidate.identifier for candidate in first.candidates], ["C001", "C002"])
|
||||
self.assertEqual(first_text, sf3_probe.format_prefix_features(second))
|
||||
self.assertNotIn("NEST", first_text)
|
||||
self.assertNotIn("ALPHA", first_text)
|
||||
self.assertNotIn("BETA", first_text)
|
||||
self.assertNotIn("SYNTHETIC", first_text)
|
||||
self.assertNotIn("IGNORE", first_text)
|
||||
|
||||
def test_selects_extension_after_iso_version_suffix(self) -> None:
|
||||
root = self.temporary_root()
|
||||
manifest = _write_manifest(
|
||||
root,
|
||||
[
|
||||
("FIRST.HOG;1", b"A" * 32),
|
||||
("SECOND.hOg;99", b"B" * 32),
|
||||
("THIRD.HOGX;1", b"C" * 32),
|
||||
],
|
||||
)
|
||||
|
||||
report = sf3_probe.probe_prefix_features(
|
||||
manifest, root, extension="hog", prefix_size=16
|
||||
)
|
||||
|
||||
self.assertEqual(report.candidate_count, 2)
|
||||
self.assertEqual([candidate.prefix_size for candidate in report.candidates], [16, 16])
|
||||
|
||||
def test_counts_generic_monotonic_u32_runs_without_format_claim(self) -> None:
|
||||
root = self.temporary_root()
|
||||
words = b"".join(value.to_bytes(4, "little") for value in (4, 8, 12, 16, 20))
|
||||
manifest = _write_manifest(root, [("TABLE.HOG;1", words + b"x" * 64)])
|
||||
|
||||
report = sf3_probe.probe_prefix_features(
|
||||
manifest, root, extension="HOG", prefix_size=32
|
||||
)
|
||||
|
||||
self.assertGreaterEqual(report.candidates[0].monotonic_u32_run_count, 2)
|
||||
self.assertEqual(report.candidates[0].zero_bytes, 15)
|
||||
|
||||
def test_rejects_manifest_size_mismatch(self) -> None:
|
||||
root = self.temporary_root()
|
||||
contents = b"mismatch"
|
||||
output = root / "BAD.HOG;1"
|
||||
root.mkdir(parents=True)
|
||||
output.write_bytes(contents)
|
||||
manifest = root / "MANIFEST.tsv"
|
||||
manifest.write_text(
|
||||
sf3_probe.MANIFEST_HEADER
|
||||
+ "\n"
|
||||
+ _manifest_row("BAD.HOG;1", 101, contents, size=len(contents) + 1),
|
||||
encoding="ascii",
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "size does not match"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="HOG")
|
||||
|
||||
def test_rejects_manifest_sha1_mismatch(self) -> None:
|
||||
root = self.temporary_root()
|
||||
contents = b"digest-mismatch"
|
||||
root.mkdir(parents=True)
|
||||
(root / "BAD.HOG;1").write_bytes(contents)
|
||||
manifest = root / "MANIFEST.tsv"
|
||||
manifest.write_text(
|
||||
sf3_probe.MANIFEST_HEADER
|
||||
+ "\n"
|
||||
+ _manifest_row("BAD.HOG;1", 101, contents, sha1="0" * 40),
|
||||
encoding="ascii",
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "SHA-1 does not match"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="HOG")
|
||||
|
||||
def test_rejects_unsafe_manifest_path(self) -> None:
|
||||
root = self.temporary_root()
|
||||
root.mkdir(parents=True)
|
||||
contents = b"unsafe"
|
||||
manifest = root / "MANIFEST.tsv"
|
||||
manifest.write_text(
|
||||
sf3_probe.MANIFEST_HEADER
|
||||
+ "\n"
|
||||
+ _manifest_row("../ESCAPE.HOG;1", 101, contents),
|
||||
encoding="ascii",
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "unsafe path component"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="HOG")
|
||||
|
||||
def test_rejects_symbolic_link_candidate(self) -> None:
|
||||
root = self.temporary_root()
|
||||
root.mkdir(parents=True)
|
||||
target = root / "TARGET.BIN;1"
|
||||
contents = b"target-data"
|
||||
target.write_bytes(contents)
|
||||
link = root / "LINK.HOG;1"
|
||||
link.symlink_to(target.name)
|
||||
manifest = root / "MANIFEST.tsv"
|
||||
manifest.write_text(
|
||||
sf3_probe.MANIFEST_HEADER + "\n" + _manifest_row(link.name, 101, contents),
|
||||
encoding="ascii",
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "symbolic link"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="HOG")
|
||||
|
||||
def test_rejects_absent_extension_class_and_invalid_prefix_size(self) -> None:
|
||||
root = self.temporary_root()
|
||||
manifest = _write_manifest(root, [("ONLY.DAT;1", b"data")])
|
||||
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "no candidates"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="HOG")
|
||||
with self.assertRaisesRegex(sf3_probe.ProbeValidationError, "prefix size"):
|
||||
sf3_probe.probe_prefix_features(manifest, root, extension="DAT", prefix_size=0)
|
||||
|
||||
def test_cli_reports_opaque_features_and_refuses_invalid_input(self) -> None:
|
||||
root = self.temporary_root()
|
||||
manifest = _write_manifest(root, [("CLI.HOG;1", b"CLI-SYNTHETIC" + b"\x00" * 20)])
|
||||
|
||||
completed = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(TOOL_PATH),
|
||||
"prefix-features",
|
||||
str(manifest),
|
||||
str(root),
|
||||
"--extension",
|
||||
"HOG",
|
||||
"--prefix-size",
|
||||
"16",
|
||||
],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(completed.returncode, 0, completed.stderr)
|
||||
self.assertIn("format\tsf3_probe-prefix-features-v1", completed.stdout)
|
||||
self.assertNotIn("CLI.HOG", completed.stdout)
|
||||
self.assertNotIn("SYNTHETIC", completed.stdout)
|
||||
self.assertEqual(completed.stderr, "")
|
||||
|
||||
invalid = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(TOOL_PATH),
|
||||
"prefix-features",
|
||||
str(manifest),
|
||||
str(root),
|
||||
"--extension",
|
||||
"HOG!",
|
||||
],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
self.assertEqual(invalid.returncode, 2)
|
||||
self.assertIn("sf3_probe: error:", invalid.stderr)
|
||||
self.assertEqual(invalid.stdout, "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user