mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-26 02:11:36 -04:00
overlay: ABI v14 gte_precision_store_word shim + splitgen gating; AOT spike
Two provably-correct framework fixes (independent of the AOT work) plus the play-free overlay-extraction spike tooling and design doc. Framework fixes (accurate, safe, complete): - ABI v14: add gte_precision_store_word to OverlayCallbacks + preamble shim + runtime population. The emitter emits a direct call for every swc2 (GTE store-word); it is stateful runtime code (gte.cpp) so overlay DLLs must forward or fail to LINK (undefined reference). Play-captured shards rarely compiled a swc2; fuller static coverage surfaced the gap. Same pattern as the v13 ws_player_x_bound / v10 gte_read_ctrl shims. - Gate splitgen OFF in overlay mode (main_psx.cpp): the current recompiler's unconditional split-TU output deletes <stem>_full.c and emits only shards, which broke ALL overlay compilation (compile_overlays.py reads _full.c). Overlays are single small TUs with no parallel-compile benefit; emit the monolith. Restores overlay compile on the current tree. AOT spike (docs/AOT_OVERLAY_PLAN.md + tools/aot_overlay_spike/, WIP): - Play-free static overlay extraction from the disc image. Validated: Tomba 1 reproduces 90% of played functions + ~1964 unplayed, live-verified in-game (Dwarf Village runs native, zero runtime compile). Tomba 2 yields 661 functions byte-identical to the shipped/played vault (provably real), all from the self-describing MAIN.EXE producer; the header-table BIN producer needs loader-table RE (base recovery unreliable). Generality is producer-dependent. Spike scripts are Tomba-specific proofs of concept with hardcoded paths, not a finished tool. - Correctness is guaranteed by the existing per-function code_crc dispatch guard: a mis-positioned shard fails audit at compile or never fires at runtime -> coverage loss, never wrong execution. Autocompile self-heals gaps. CROSS-CUTTING: the ABI v14 change is part of the overlay codegen-hash source set, so it ROLLS the cg tag (1f2ad026 -> e8484213). On master + a fw bump, every title's overlay cache invalidates and needs regen+reshard. Branch only; NOT pushed; other games' caches untouched. Needs user validation before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
@@ -0,0 +1,335 @@
|
||||
# AOT Overlay Sharding — Spike Findings & Plan (Tomba-first)
|
||||
|
||||
Status: SPIKE / DESIGN (2026-07-17). Not yet executed.
|
||||
Author: investigation via 4 parallel agents + adversarial source verification.
|
||||
Scope: can we move overlay sharding from *runtime discovery* to *build-time (AOT)*,
|
||||
starting with Tomba (SCUS-94236)?
|
||||
|
||||
---
|
||||
|
||||
## ⭐ SESSION-2 BREAKTHROUGH (2026-07-17, VERIFIED FROM DISC + CORPUS)
|
||||
|
||||
Adversarial disc analysis overturned the pessimistic framing below. For Tomba's
|
||||
GAME-CODE overlays, **pure static analysis suffices — no playthrough, no headless
|
||||
execution, no decompressor.**
|
||||
|
||||
Verified facts (byte-checked against `Tomba! (USA).bin` + the 99-capture vault):
|
||||
- The disc carries a full ISO9660 directory: **1085 files, 20 `AREA` dirs.** Game code
|
||||
overlays are **17 distinct `X0N.BIN` files** (one per area; `X07→X01`, `X12→X04`,
|
||||
`X15` absent, `AREA19` dual) — each duplicated under `SYSTEM/`. All 17 enumerate in
|
||||
seconds via `extract_overlays.py`'s ISO9660 walker.
|
||||
- **`X0N.BIN` is uncompressed and POSITION-FIXED.** `AREA00/X00.BIN` (343,864 B) and
|
||||
`X09.BIN` (299,576 B) were found **100% byte-verbatim, contiguous** inside their
|
||||
captured `0x800E7000` RAM images (X00 at region offset 904; the 904-byte prefix is
|
||||
`0x11111111` fill). The file's export pointers are pre-relocated ON DISC to the exact
|
||||
load address (`0x800E922C` = `0x800E7388 + file-offset`), so **disc bytes == RAM
|
||||
bytes at a fixed address — zero runtime relocation, zero decompression.** This is
|
||||
Class A behavior; the earlier "scatter-load / must-capture-post-fixup" concern
|
||||
applies to the `.GAM` DATA packages and capture timing, NOT the code.
|
||||
- **`X0N.BIN` self-describes its entry points:** a plaintext `{count:u32,
|
||||
ptr[count]:u32}` header (X00: 29 pointers, all in-range). Function-entry SEEDS come
|
||||
free from the file — no execution needed to find them.
|
||||
- **Determinism is measured, not assumed:** across the corpus, 95–98% of overlay
|
||||
function entries have a single stable code-CRC every time the address loads (Tomba
|
||||
`0x800E7000`: 782/820 = 95%; `0x00000000`: 100%). Residual mismatches fall to the
|
||||
interpreter via the CRC dispatch guard — coverage loss, never correctness loss.
|
||||
- **The `0x80000000` region is BIOS-kernel code, not a game overlay** (entry offsets
|
||||
= A0/B0/C0 vectors `0xa0/0xb0/0xc0` + SIO stub `0xcf0`; identical structure across
|
||||
MMX5/MMX6/Vigilante8/Tomba — independent engines). AOT it ONCE from the BIOS,
|
||||
shared by every title.
|
||||
|
||||
⇒ **Revised project shape:** a build-time STATIC OVERLAY EXTRACTOR (walk disc →
|
||||
enumerate 17 `X0N.BIN` → parse header for load-addr + seeds → synthesize
|
||||
`overlay_captures.json` → existing `compile_overlays.py`). Headless materialization
|
||||
(§5) drops to a FALLBACK for any title whose overlays turn out compressed/relocated,
|
||||
not the primary path for Tomba.
|
||||
|
||||
Open verification before committing (do these first — see §6 revised):
|
||||
- Prove all 17 `X0N.BIN` (not just X00/X09) are 100% verbatim vs their own area's
|
||||
coherent capture, at a consistent offset.
|
||||
- Confirm no `X0N` has CPU-patched internal `jr` jump tables (disc≠RAM) — reconcile
|
||||
against `overlay_capture.c:187-196`. X00/X09 show none.
|
||||
- Confirm `.GAM`/`.000` packages contain no executable code needing shards (they look
|
||||
like models/palettes/scripts = pure data → not compiled).
|
||||
- Confirm Tomba code overlays live ONLY at `0x800E7000` (+ kernel `0x80000000`); no
|
||||
third code base.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
|
||||
- **The compile step is already AOT, AND the captured coverage already ships.**
|
||||
`tools/compile_overlays.py` runs fully offline; releases already bundle the
|
||||
precompiled gcc DLL cache (`package_release.ps1:223-235`,
|
||||
`COMPILING_OVERLAYS.md:23-33`). So "AOT the 99 captures" is the STATUS QUO, not a
|
||||
goal. The goal is coverage the dev playthrough never reached.
|
||||
- **"Runtime-applied fixups" are deterministic, not session-varying** — proven by the
|
||||
shipped cache CRC-matching on other machines/fresh boots (§T0.5). AOT is achievable
|
||||
because the post-fixup image is a fixed function of the disc; you just have to
|
||||
*evaluate* it (run the loader), not read it off the disc.
|
||||
- **The runtime-only input is the CAPTURE, not the compile.** The single artifact
|
||||
that today only exists after a played session is `overlay_captures.json` — the
|
||||
post-fixup RAM images + execution-verified entry seeds.
|
||||
- **Naive "scan the disc" AOT is DISPROVEN for Tomba** (two independent reasons,
|
||||
both verified in source — see §4). Do not pursue it as the Tomba path.
|
||||
- **The faithful, general AOT path is HEADLESS MATERIALIZATION:** run the game's
|
||||
*own* recompiled loader + decompressor + fixup code at build time, driven over
|
||||
every overlay, to produce the exact `overlay_captures.json` a playthrough would —
|
||||
then feed the existing offline compiler. This uses the game's real code (no
|
||||
per-game decompressor reimplementation → satisfies Rule -1), is general to every
|
||||
title, and is guarded by an existing correctness net.
|
||||
- **The correctness net makes AOT safe to be aggressive.** Shards are
|
||||
content-addressed `(entry_pc, code_crc)`; at dispatch the loader hashes live RAM
|
||||
and runs native *only* if it matches (`overlay_loader.c:2137`). A wrong/stale AOT
|
||||
shard never fires — it falls to the interpreter. AOT cannot cause divergence.
|
||||
|
||||
---
|
||||
|
||||
## 1. How overlays work today (verified)
|
||||
|
||||
### 1.1 Detection (runtime)
|
||||
- Trigger: **CD-ROM channel-3 DMA completion** into game RAM below the FMV floor.
|
||||
`dma.c:484-498` → `overlay_capture_on_dma(load_start, size, ram+load_start)`.
|
||||
- Gated on `[runtime] overlay_cache` and auto-armed on the first post-BIOS-handoff
|
||||
DMA (`overlay_capture.c:82-89`). Dedup by `load_addr` (`:91-95`).
|
||||
- A second signal — sustained dirty-RAM-interp pressure sampled ~every 2s while
|
||||
*not* mid-CD-load — auto-fires fresh captures for new *variants* at a reused
|
||||
address (`overlay_capture.c:299-532`).
|
||||
|
||||
### 1.2 Capture record (`overlay_captures.json`, schema "v2")
|
||||
Built from **live-RAM dirty pages at a coherent moment**, one record per contiguous
|
||||
dirty run (`overlay_capture.c:200-233`):
|
||||
```
|
||||
load_addr, size, bytes_b64 (LIVE post-fixup RAM),
|
||||
executed_pcs[], dispatch_entry_pcs[], function_entry_pcs[], seeds[]
|
||||
```
|
||||
`executed_pcs`/`dispatch_entry_pcs` come from always-on interpreter ring tables
|
||||
(`g_dirty_ram_pc_table`, `g_dirty_ram_exec_pc_table`) — **execution-verified**, not
|
||||
disassembly guesses.
|
||||
|
||||
### 1.3 Compile (offline, `tools/compile_overlays.py`)
|
||||
1. Python does function-boundary discovery from the seeds (`classify_overlay_seeds`,
|
||||
`_walk_overlay_function`) with a strict callable-prologue gate; interior/
|
||||
jump-table PCs become aliases, never walk roots.
|
||||
2. Captured bytes wrapped in a **synthetic PS-X EXE** (`make_psxexe`, `:266-274`) and
|
||||
handed to the *same* recompiler binary the main EXE uses:
|
||||
`psxrecomp-game <fake.psx> --seeds seeds.txt --overlay --ws-config game.toml`.
|
||||
3. Out-of-overlay calls rewritten to `call_by_address()` (resolved at runtime via the
|
||||
dispatch table); jump tables → native `switch` + `call_by_address` default.
|
||||
`audit_generated_c()` hard-rejects a shard if any direct call escapes or any
|
||||
instruction is unsupported.
|
||||
4. Per-function identity = CRC32 over its exact compiled code ranges. Output:
|
||||
`<phys>_<region_crc>.dll` + a `.ranges` manifest (`F <entry> <code_crc>` /
|
||||
`R <entry> <len>`).
|
||||
5. `--static` mode instead namespaces the C and folds variants into
|
||||
`overlays_static.c` linked directly into the binary, dispatched via a generated
|
||||
`psx_overlay_dispatch()` — **this is already an AOT delivery path**; it moves
|
||||
*link time* earlier, not *discovery time*.
|
||||
|
||||
### 1.4 Dispatch + cache identity (`overlay_loader.c`)
|
||||
- Two stacked cache tags: ABI (`PSX_OVERLAY_ABI_VERSION`=13 + flavor) and codegen
|
||||
(`cg<VER>_<HASH>` path segment, auto-invalidated on any emitter source change).
|
||||
Plus `verify_recompiler_matches_tag()` staleness guard.
|
||||
- **Content guard (the safety net):** each DLL export registers a `Candidate` with
|
||||
`crc_code`. On dispatch, `range_candidate_matches` re-hashes live RAM and runs
|
||||
native only if `live == crc_code` (`overlay_loader.c:2137`; registration gate at
|
||||
`:595`). Cache is **additive** — every variant ever seen for an address stays
|
||||
loaded; per-dispatch CRC picks the one that currently matches (`:1844`). This is
|
||||
how one address (`0x800E7xxx`) hosts village vs overworld safely.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tomba specifics (verified)
|
||||
|
||||
- Main EXE: `load_address=0x80010000`, `entry=0x8006B58C`, `text_size=0x88000`,
|
||||
loads contiguously, **no kernel-style relocation** (`game.toml:8, 276-285`).
|
||||
- **Only two overlay load addresses** across 99 captures in
|
||||
`_coverage_vault/SCUS-94236/`:
|
||||
- `0x80000000` — 45 caps, 52–60 KB. → 11 compiled DLL variants.
|
||||
- `0x800E7000` — 54 caps, 16 KB–624 KB (reused per scene). → 18 variants
|
||||
(4 ranges-only / audit-failed).
|
||||
- ~5.3 MB total DLL payload; largest DLL 879 KB.
|
||||
- Loader = **`FUN_80021340`** (asset/area loader AND live world-streamer; green
|
||||
thread, 1 state-step per vblank via `FUN_800171D4`→`ChangeTh`).
|
||||
- Decompressor = **`FUN_8003EF50`** — named in `game.toml:290` as "Tomba's real
|
||||
back-reference decompressor" (currently quarantined from the *data-shard replay*
|
||||
path; irrelevant to AOT, which runs it rather than replaying it). Exact algorithm
|
||||
not yet written up, but the function is identified and is already statically
|
||||
recompiled into the main EXE.
|
||||
- On-disc format = **scatter-load**: each sector is `(12-byte descriptor,
|
||||
2048-byte block)`, descriptor names the dest RAM address; content is *also*
|
||||
decompressed and *also* CPU-fixed-up after landing.
|
||||
|
||||
---
|
||||
|
||||
## 3. The four runtime-only gaps (what AOT must supply)
|
||||
|
||||
From the runtime-arch investigation, the inputs the runtime path has that a static
|
||||
pass lacks:
|
||||
1. **Enumeration** of every distinct overlay *image* per RAM window (keyed by
|
||||
content, not address).
|
||||
2. **Load address** each image lands at. *Partially static already:* the region
|
||||
floor is `(load_address+text_size)&0x1FFFFFFF` from the EXE header
|
||||
(`main.cpp:3229`); base addresses (`0x80000000`, `0x800E7000`) are known.
|
||||
3. **The post-fixup byte image** — decompressed AND relocated AND jump-table-patched
|
||||
by the game's own code. *This is the crux for Tomba.*
|
||||
4. **Execution-verified entry seeds** (`executed_pcs`/`dispatch_entry_pcs`).
|
||||
|
||||
Gaps 3 and 4 are why capture is runtime-only today.
|
||||
|
||||
---
|
||||
|
||||
## 4. Adversarial findings (what NOT to do, and why)
|
||||
|
||||
**F1 — Naive disc-scan AOT is dead for Tomba (two independent kills):**
|
||||
- (a) Scatter-load means the reconstituted RAM image is **not contiguous anywhere on
|
||||
disc** (`docs/overlay-discovery.md:36-45`). A CRC/byte scan can't find it.
|
||||
- (b) Even after de-scattering + decompressing, the executable bytes differ from the
|
||||
raw payload because the game applies **load-time fixups (relocated jump tables)**.
|
||||
Capturing pre-fixup DMA bytes was tried and caused the **village→overworld blue
|
||||
screen** — abandoned (`overlay_capture.c:187-196`). The correct image exists only
|
||||
*after the game's own code runs*.
|
||||
|
||||
**F2 — Two PSX overlay archetypes; Tomba is the hard one.**
|
||||
- *Class A (clean Psy-Q `over(group)`):* absolutely linked, position-fixed, **disc
|
||||
bytes == RAM bytes byte-for-byte, only `.bss` differs** (Psy-Q '96 training deck;
|
||||
Legaia RE static/dynamic byte-diff). Pure-static AOT is trivially correct here
|
||||
*once the load address is known*.
|
||||
- *Class B (scatter + decompress + runtime-relocate):* Tomba. Legaia's own RE team
|
||||
carved these out as **"ineligible — remain on the dynamic-capture path"**
|
||||
(`legend-of-legaia-re/docs/tooling/static-overlay-pipeline.md`). The clean-decomp
|
||||
world already concluded you must run the game for this class.
|
||||
|
||||
**F3 — There is no SDK-standard on-disc overlay directory.** The Psy-Q SDK ships no
|
||||
mandated `{id→LBA→size→addr}` table; the loader table is bespoke per game
|
||||
(1996 Sony seminar). So "enumerate overlays from a universal disc format" is a
|
||||
fiction — enumeration is per-game (loader-table RE) or must ride the game's loader.
|
||||
|
||||
**F4 — The CRC dispatch guard makes AOT risk-free to attempt.** Because a shard only
|
||||
runs on a live-RAM CRC match, AOT output is *self-validating*. Worst case of a bad
|
||||
AOT shard = it never fires. This means we can ship speculative AOT coverage without
|
||||
gating correctness on it.
|
||||
|
||||
**Confidence caveats:** the `DaCodeChick/Legaia-decomp` splat repo 404'd; PSX-splat
|
||||
`.yaml` field specifics are search-derived, not re-fetched. Core Class-A claims
|
||||
(position-fixed, no relocation, shared-address `over(group)`) are solidly sourced
|
||||
from the Psy-Q training PDF + Legaia RE docs. Tomba compression *algorithm* is
|
||||
un-characterized (function identified, bytes not decoded).
|
||||
|
||||
---
|
||||
|
||||
## 5. The strategy: HEADLESS MATERIALIZATION (the complete option)
|
||||
|
||||
Run the game's own recompiled loader/decompressor/fixup code at build time, over
|
||||
every overlay, to produce `overlay_captures.json` **without a human playing** — then
|
||||
feed the existing offline compiler. Rationale:
|
||||
- **Faithful:** reproduces post-fixup bytes by executing the real code, not a
|
||||
reimplemented decompressor. No per-game byte-format hack (Rule -1 compliant).
|
||||
- **General:** works for any title whose loader is recompiled (all of them),
|
||||
Class A and Class B alike.
|
||||
- **Reuses ~everything:** CD-DMA sim, dirty-RAM interp, decompressor-as-code, capture
|
||||
JSON writer, and the whole compile/dispatch pipeline already exist. The new code is
|
||||
a *driver*, not a new engine.
|
||||
- **Safe:** output is CRC-guarded at dispatch (F4).
|
||||
|
||||
Enumeration has two escalating sub-strategies (do both; (b) is the north star):
|
||||
- **(a) Route-replay bootstrap:** replay a recorded set of scene/LBA load requests
|
||||
(we already have `warm_cd_routes` + the B-2 `overlay_map.jsonl` LBA provenance) to
|
||||
re-trigger each load headlessly and deterministically.
|
||||
- **(b) Directory-driven completeness:** locate the loader's scene/overlay table
|
||||
statically (what `FUN_80021340` iterates), enumerate *all* scene IDs, and invoke
|
||||
the loader for each — covering scenes no playthrough visited.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tiered plan (deliverables + proof artifacts per Rule 10)
|
||||
|
||||
### T0 — Baseline: baking the captured coverage ALREADY SHIPS (not a new task)
|
||||
Correction (surfaced by user, verified): the release already bakes the captured
|
||||
coverage. `TombaRecomp/tools/package_release.ps1:223-235` bundles
|
||||
`build-stable/cache/SCUS-94236/…/*.dll + *.ranges` into the shipped `cache/`,
|
||||
cg-tag-namespaced, loaded from first launch (`docs/COMPILING_OVERLAYS.md:23-33`:
|
||||
"ship that `cache/` folder in the release"). A tcc fallback toolchain covers gaps a
|
||||
player reaches past the bundle. **So "AOT the 99" is the status quo, delivered as
|
||||
precompiled gcc DLLs — no user-machine recompile for covered areas.**
|
||||
- The only remaining T0-flavored option is a *delivery-form* change:
|
||||
`compile_overlays.py --static` → `generated/overlays_static.c` compiled INTO the
|
||||
exe (no `cache/` dir, no LoadLibrary). Same 99, single binary. Marginal; do only if
|
||||
a single-file exe is wanted. **Not a coverage win.**
|
||||
- **The real prize is coverage BEYOND the 99** — everything a dev's playthrough never
|
||||
reached. That is T1/T2, and it is the actual point of this spike.
|
||||
|
||||
### T0.5 — Empirical basis: post-fixup bytes are deterministic (already proven)
|
||||
The shipped DLL cache CRC-matches and runs natively on other machines / fresh boots.
|
||||
The dispatch guard runs a shard only if `live_RAM_CRC == compiled_code_CRC`
|
||||
(`overlay_loader.c:2137`). A reusable cross-machine cache is therefore *proof* the
|
||||
post-fixup RAM image is a fixed, reproducible function of the disc — "runtime-applied
|
||||
fixups" are deterministic, not session-varying. AOT = evaluate that function once at
|
||||
build time (run the game's own loader headless), not read raw disc bytes. Residual
|
||||
genuinely-variable-immediate overlays (if any) just fail the CRC and fall to interp —
|
||||
coverage loss, never correctness loss.
|
||||
|
||||
### T1 — Headless materialization harness (the core deliverable)
|
||||
A build-time mode of the runtime that boots (HLE boot-skip), then drives the loader
|
||||
directly instead of via SDL input, dumping `overlay_captures.json`.
|
||||
- **T1.1 Loader ABI RE:** determine `FUN_80021340`'s inputs (scene/area id, dest
|
||||
addr, disc LBA) and the "load complete" signal (`DAT_1f8001ce`). Artifact: an
|
||||
annotated calling-convention note + a single headless call that loads ONE known
|
||||
scene and dumps a capture **byte-identical** to the vault's capture for that scene.
|
||||
- **T1.2 Route-replay driver:** feed the known route (village, overworld, title, +
|
||||
`warm_cd_routes` LBAs); headlessly regenerate the current 99-capture set.
|
||||
Artifact: diff of headless captures vs `_coverage_vault` captures = empty (or only
|
||||
benign `.bss`/timing deltas). This *reproduces today's coverage with no human*.
|
||||
- **T1.3 Coherence + variant convergence:** reuse the existing "coherent moment"
|
||||
capture logic so multiplexed `0x800E7000` variants are each snapshotted cleanly.
|
||||
- **Proof:** `tomba-materialize.exe --captures out.json` produces a superset-or-equal
|
||||
of the vault; `compile_overlays.py` over it yields shards that all pass audit and
|
||||
all CRC-match at runtime.
|
||||
- **Risk:** medium. The loader is *also* the world-streamer (the instant-loads
|
||||
parking lesson) — but here we are NOT skipping yields; we let it run to completion
|
||||
headlessly. Watch for green-thread/IRQ dependencies; drive via the real scheduler.
|
||||
|
||||
### T2 — Static enumeration (directory discovery → completeness)
|
||||
Remove dependence on a recorded route.
|
||||
- RE the scene/overlay table `FUN_80021340` consumes (the loader's own directory).
|
||||
Enumerate every scene id + its disc LBA/dest/size statically.
|
||||
- Drive T1's harness over the *full* enumeration → capture overlays a playthrough
|
||||
never reached.
|
||||
- **Proof:** capture count and unique `code_crc` set strictly exceed the playthrough
|
||||
vault; spot-navigate the game to 3 previously-uncaptured scenes and confirm their
|
||||
shards CRC-match (native path taken on first entry, no compile).
|
||||
- **Artifact:** `game.toml [[overlays]]` populated automatically from the enumerated
|
||||
table (finally implementing the documented-but-empty schema; note the current
|
||||
discrepancy in `overlay-discovery.md` that `[[overlays]]` is unimplemented).
|
||||
|
||||
### T3 — Generalize the framework (Class A static fast-path + other titles)
|
||||
- For Class-A titles (position-fixed, disc==RAM), add a pure-static extractor
|
||||
(extend `extract_overlays.py`, seeded by the loader table) that skips headless
|
||||
execution entirely — validated by the same CRC guard.
|
||||
- Fold Legaia-style `jal`-call-graph base recovery for titles lacking a clean table.
|
||||
- Run the harness across the ecosystem (Ape, MMX, Tomba2, …); each title's shards
|
||||
become build artifacts, killing the runtime background-compile requirement fleet-
|
||||
wide.
|
||||
- **Proof:** ≥2 additional titles ship with fully-baked overlays, no runtime
|
||||
compile, playthrough-validated.
|
||||
|
||||
---
|
||||
|
||||
## 7. Immediate next actions (to power through)
|
||||
1. **T0 now:** `--static` bake of the Tomba vault; boot-validate; screenshot series.
|
||||
This de-risks delivery and gives a shippable artifact this session.
|
||||
2. **T1.1 next:** RE `FUN_80021340` inputs; write the one-scene headless load that
|
||||
reproduces a vault capture byte-for-byte. That single artifact proves the whole
|
||||
headless thesis.
|
||||
3. Then T1.2 route-replay, then T2 enumeration.
|
||||
|
||||
## 8. Open questions / to verify before committing effort
|
||||
- Does `FUN_80021340` run cleanly to completion headlessly without SDL/vblank pacing
|
||||
it doesn't get? (Instant-loads parking says don't skip yields — we won't; question
|
||||
is whether the harness supplies enough scheduler/IRQ progress.)
|
||||
- Is a scene/overlay directory table locatable statically, or must T2 fall back to
|
||||
route-replay only? (T1 works regardless; T2 completeness depends on this.)
|
||||
- `.bss` / uninitialized-tail handling: captures may include zero-fill runs; confirm
|
||||
the harness snapshots at the same coherent point the runtime does.
|
||||
- cg-tag/ABI: any AOT run must use a same-tree recompiler (`--codegen-hash` guard),
|
||||
and every fw bump re-rolls the tag → re-materialize. Bake this into the build.
|
||||
@@ -927,7 +927,22 @@ int main(int argc, char** argv) {
|
||||
// the on-disk representation changes. Function bodies are byte-identical
|
||||
// to what full_c_code would have contained; see
|
||||
// CodeGenerator::last_gen_funcs() / build_shared_decls_header().
|
||||
{
|
||||
if (overlay_mode) {
|
||||
// Overlays are single small TUs consumed by compile_overlays.py, which
|
||||
// reads <stem>_full.c directly. Emit the monolith; the split path below
|
||||
// would delete _full.c and leave only shards, breaking overlay compile
|
||||
// (no_output). Splitting a small overlay TU has no parallel-compile
|
||||
// benefit anyway.
|
||||
std::ofstream full_file(output_filename);
|
||||
if (full_file.is_open()) {
|
||||
full_file << full_c_code;
|
||||
full_file.close();
|
||||
fmt::print("✓ Saved overlay monolith to {}\n", output_filename.string());
|
||||
} else {
|
||||
fmt::print(stderr, "⚠ Failed to write overlay monolith {}\n\n",
|
||||
output_filename.string());
|
||||
}
|
||||
} else {
|
||||
// 1. Remove stale outputs: the old monolith and any previously
|
||||
// written shards (a shard count shrink must not leave orphans).
|
||||
std::error_code rm_ec;
|
||||
|
||||
@@ -62,7 +62,12 @@
|
||||
* site fails to LINK (undefined psx_ws_player_x_bound) and runs interpreted
|
||||
* forever — the rfe/gte_read_ctrl class again. Identity fallback (vanilla)
|
||||
* on a NULL host pointer keeps a v13 DLL correct (4:3) on an older host. */
|
||||
#define PSX_OVERLAY_ABI_VERSION 13
|
||||
/* v14: gte_precision_store_word forwarder — the emitter emits a direct call for
|
||||
* every swc2 (GTE store-word); stateful runtime code (gte.cpp) so the DLL
|
||||
* must forward or fail to LINK. Surfaced by full static overlay coverage
|
||||
* (play-captured shards rarely compiled a swc2). Appended last; ABI bump
|
||||
* to 14 rejects any pre-shim DLL. */
|
||||
#define PSX_OVERLAY_ABI_VERSION 14
|
||||
|
||||
/* Codegen flavor of the recompiled output the overlays + runtime were built
|
||||
* against. Overlays are keyed in the cache by guest-bytes CRC, which is
|
||||
@@ -221,6 +226,16 @@ typedef struct {
|
||||
* psx_ws_player_x_bound). Appended last; may be NULL on a host that
|
||||
* predates it — the DLL glue falls back to the vanilla constant (4:3). */
|
||||
int32_t (*ws_player_x_bound)(int32_t vanilla);
|
||||
/* Sub-pixel GTE precision-store tracker (ABI v14; see version-history note
|
||||
* above). The emitter emits a direct gte_precision_store_word() call for
|
||||
* every swc2 (GTE store-word); it is stateful runtime code (gte.cpp) so the
|
||||
* DLL must forward. Without the shim any shard containing a swc2 fails to
|
||||
* LINK (undefined gte_precision_store_word) and runs interpreted forever —
|
||||
* the rfe/gte_read_ctrl class again. Surfaced by full static overlay
|
||||
* coverage (play-captured shards rarely compiled a swc2). Appended last;
|
||||
* may be NULL on a host that predates it — the shim no-ops (precision
|
||||
* tracking is a widescreen enhancement, off by default). */
|
||||
void (*gte_precision_store_word)(uint32_t addr, uint8_t reg);
|
||||
} OverlayCallbacks;
|
||||
|
||||
#ifdef __cplusplus
|
||||
|
||||
@@ -223,5 +223,15 @@ uint32_t psx_ws_backdrop_value(uint32_t orig, int is_end, int window_cols) {
|
||||
int32_t psx_ws_player_x_bound(int32_t vanilla) {
|
||||
return g_cbs.ws_player_x_bound ? g_cbs.ws_player_x_bound(vanilla) : vanilla;
|
||||
}
|
||||
/* Sub-pixel GTE precision-store tracker (ABI v14). The emitter emits a direct
|
||||
* gte_precision_store_word() call for every swc2 (GTE store-word), including in
|
||||
* overlay-resident code, so the DLL must define it or the shard fails to link
|
||||
* (undefined reference — the rfe/gte_read_ctrl class). Forwards to the host's
|
||||
* stateful tracker (gte.cpp); no-op if the host predates the callback
|
||||
* (precision tracking is a widescreen enhancement, off by default — 4:3
|
||||
* behavior is unaffected). */
|
||||
void gte_precision_store_word(uint32_t addr, uint8_t reg) {
|
||||
if (g_cbs.gte_precision_store_word) g_cbs.gte_precision_store_word(addr, reg);
|
||||
}
|
||||
/* ----------------------------------------------------------------------- */
|
||||
|
||||
|
||||
@@ -1552,6 +1552,14 @@ static void init_callbacks(void) {
|
||||
extern int32_t psx_ws_player_x_bound(int32_t vanilla);
|
||||
s_callbacks.ws_player_x_bound = psx_ws_player_x_bound;
|
||||
}
|
||||
/* ABI v14: GTE precision-store tracker — the emitter emits a direct
|
||||
* gte_precision_store_word() call for every swc2 (GTE store-word),
|
||||
* including in overlay-resident code; it mutates the host's stateful
|
||||
* sub-pixel projection cache (gte.cpp) so the DLL must forward. */
|
||||
{
|
||||
extern void gte_precision_store_word(uint32_t addr, uint8_t reg);
|
||||
s_callbacks.gte_precision_store_word = gte_precision_store_word;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# AOT Overlay Sharding — Spike Tooling (WIP)
|
||||
|
||||
Play-free static overlay extraction: discover + compile overlay shards from the
|
||||
disc image at build time, so a fresh install ships with overlay coverage instead
|
||||
of relying entirely on runtime (tcc/gcc) autocompile. See `docs/AOT_OVERLAY_PLAN.md`
|
||||
for the full design, findings, and honest coverage numbers.
|
||||
|
||||
**Status: SPIKE. Provably-correct for clean producers, not exhaustive.** These
|
||||
scripts carry hardcoded paths (disc, scratchpad) and are Tomba-specific proofs of
|
||||
concept, not a finished general tool. They exist so a future session can improve
|
||||
discovery quality, not to be run as-is in a build.
|
||||
|
||||
## What each does
|
||||
|
||||
- `overlay_determinism.py` — corpus analysis across all games' captured overlays:
|
||||
base-address regularity, size/sector alignment, per-function code-CRC stability.
|
||||
Establishes that overlay code is deterministic (95-98% stable code-CRC).
|
||||
- `tomba1_extract.py` — Tomba 1 (SCUS-94236). Enumerates ALL code overlays from
|
||||
the disc by header signature ({count:u32, ptr[count]:u32}), NOT just X*.BIN
|
||||
(that filter missed INFO.BIN → Dwarf Village ran interpreted/slow). Emits a
|
||||
synthetic overlay_captures.json for compile_overlays.py. Position-fixed,
|
||||
verbatim, seeds via prologue scan @ region+904.
|
||||
- `tomba2_extract.py` — Tomba 2 (SCUS-94454). Two producers: MAIN.EXE (a PS-X EXE
|
||||
whose header self-describes the load address — ROBUST) and BIN/A*.BIN (header
|
||||
table; base recovery via prologue delta-sweep — UNRELIABLE, produces data-as-code).
|
||||
|
||||
## Validated result (see plan doc for detail)
|
||||
|
||||
- **Tomba 1:** 90% of played functions reproduced play-free + ~1964 unplayed
|
||||
extra; live-validated in-game (Dwarf Village went native, no runtime compile).
|
||||
- **Tomba 2:** 661 functions byte-identical (entry+code_crc) to the shipped/played
|
||||
vault — provably real — ALL from the MAIN.EXE producer; the BIN producer yields
|
||||
0 usable shards (broken base recovery). Generality is producer-dependent.
|
||||
|
||||
## Correctness guarantee (why partial coverage is safe)
|
||||
|
||||
Every shard is content-addressed by per-function `code_crc`; the runtime loader
|
||||
only executes a shard when live RAM byte-matches (`overlay_loader.c` lazy_man_matches).
|
||||
A mis-positioned / data-as-code shard either fails audit at compile time or never
|
||||
fires at runtime → coverage loss, never incorrect execution. Whatever static
|
||||
misses, production autocompile (tcc/gcc) self-heals on first visit.
|
||||
|
||||
## Next to raise coverage (future session)
|
||||
|
||||
1. Frameless-leaf + indirect-call-table seed discovery (helps every game; would
|
||||
lift MAIN.EXE past ~62% and Tomba 1 past 90%).
|
||||
2. Real loader-table RE for header-table (BIN) base recovery (per-game).
|
||||
3. Extract the kernel `0x80000000` region once from the BIOS (game-independent).
|
||||
4. Generalize into a real `tools/` producer (no hardcoded paths; disc + game.toml in).
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Corpus analysis: is there a DETERMINISTIC story across captured overlays?
|
||||
|
||||
Scans every overlay_captures.json + cache/*.ranges under the project tree,
|
||||
one representative per game, and quantifies:
|
||||
- base-address regularity (how many distinct load_addr per game; reuse depth)
|
||||
- relation of each base to the EXE text layout (load_address, text_size from game.toml)
|
||||
- size distribution + sector (2048) alignment
|
||||
- seed / dispatch-entry clustering (offset of entries from the region base)
|
||||
- cross-variant function-CRC stability (from .ranges: do (entry,crc) recur?)
|
||||
"""
|
||||
import json, os, re, glob, base64
|
||||
from collections import defaultdict, Counter
|
||||
|
||||
ROOT = r"F:\Projects\psxrecomp"
|
||||
|
||||
# ---- pick one representative captures.json per game id (prefer vault, else largest) ----
|
||||
def game_id_of(path):
|
||||
m = re.search(r'(SC[UE]S-\d{5}|SLP[SM]-\d{5})', path)
|
||||
if m: return m.group(1)
|
||||
# fall back to repo dir name
|
||||
parts = path.replace('\\','/').split('/')
|
||||
for p in parts:
|
||||
if p.endswith('Recomp'): return p
|
||||
return parts[-2]
|
||||
|
||||
def rec_count(p):
|
||||
try:
|
||||
with open(p, 'r', errors='ignore') as f: return f.read().count('"load_addr"')
|
||||
except: return 0
|
||||
|
||||
cands = defaultdict(list)
|
||||
for p in glob.glob(os.path.join(ROOT, '**', 'overlay_captures.json'), recursive=True):
|
||||
if any(x in p for x in ('_wt-', '_stale', '_backup', '_release_test')): continue
|
||||
cands[game_id_of(p)].append(p)
|
||||
|
||||
reps = {}
|
||||
for gid, ps in cands.items():
|
||||
ps.sort(key=lambda p: (0 if '_coverage_vault' in p else 1, -rec_count(p)))
|
||||
reps[gid] = ps[0]
|
||||
|
||||
# ---- load game.toml text layout per game ----
|
||||
def find_toml_layout(gid):
|
||||
for tp in glob.glob(os.path.join(ROOT, '*Recomp*', 'game.toml')):
|
||||
try:
|
||||
t = open(tp, errors='ignore').read()
|
||||
except: continue
|
||||
if gid in t or gid.replace('-','_') in t:
|
||||
la = re.search(r'load_address\s*=\s*"?(0x[0-9A-Fa-f]+)"?', t)
|
||||
ts = re.search(r'text_size\s*=\s*"?(0x[0-9A-Fa-f]+)"?', t)
|
||||
return (int(la.group(1),16) if la else None,
|
||||
int(ts.group(1),16) if ts else None, tp)
|
||||
return (None, None, None)
|
||||
|
||||
def norm(a): # KSEG-mask to physical, keep 0x80000000 base for display
|
||||
return a & 0x1FFFFFFF
|
||||
|
||||
print("="*78)
|
||||
print("OVERLAY DETERMINISM CORPUS ANALYSIS")
|
||||
print("="*78)
|
||||
|
||||
for gid in sorted(reps, key=lambda g: -rec_count(reps[g])):
|
||||
path = reps[gid]
|
||||
try:
|
||||
data = json.load(open(path, errors='ignore'))
|
||||
except Exception as e:
|
||||
print(f"\n## {gid}: FAILED to parse ({e})"); continue
|
||||
if not isinstance(data, list) or not data: continue
|
||||
la, ts, tp = find_toml_layout(gid)
|
||||
text_end = (la + ts) if (la and ts) else None
|
||||
|
||||
by_base = defaultdict(list)
|
||||
for r in data:
|
||||
try: base = int(r['load_addr'], 16)
|
||||
except: continue
|
||||
by_base[base].append(r)
|
||||
|
||||
print(f"\n{'#'*70}\n## {gid} ({len(data)} records, {len(by_base)} distinct load_addr)")
|
||||
print(f" repr: {os.path.relpath(path, ROOT)}")
|
||||
if la: print(f" EXE: load_address={hex(la)} text_size={hex(ts) if ts else '?'} "
|
||||
f"text_end={hex(text_end) if text_end else '?'}")
|
||||
|
||||
for base in sorted(by_base):
|
||||
recs = by_base[base]
|
||||
sizes = [r.get('size',0) for r in recs]
|
||||
uq = sorted(set(sizes))
|
||||
align = Counter('sector' if s % 2048 == 0 else ('1k' if s % 1024 == 0 else 'other') for s in sizes)
|
||||
# entry offsets: first seed/dispatch relative to base
|
||||
offs = []
|
||||
for r in recs:
|
||||
seeds = r.get('seeds') or r.get('dispatch_entry_pcs') or []
|
||||
for s in seeds:
|
||||
try: offs.append(int(s,16) - base)
|
||||
except: pass
|
||||
rel = ""
|
||||
if text_end is not None:
|
||||
if norm(base) == norm(text_end): rel = " <== EXACTLY text_end"
|
||||
elif abs(norm(base)-norm(text_end)) < 0x2000: rel = f" (~text_end {norm(base)-norm(text_end):+#x})"
|
||||
elif norm(base) < norm(la): rel = " (below load_address / low-RAM)"
|
||||
print(f" base {hex(base)}: {len(recs)} caps{rel}")
|
||||
print(f" sizes: {len(uq)} unique, min={min(sizes)} max={max(sizes)} "
|
||||
f"({min(sizes)//1024}K..{max(sizes)//1024}K), align={dict(align)}")
|
||||
if offs:
|
||||
offs_s = sorted(set(offs))
|
||||
head = [hex(o) for o in offs_s[:6]]
|
||||
print(f" entry offsets from base: {len(offs_s)} unique, "
|
||||
f"min={hex(min(offs))} max={hex(max(offs))}, first few={head}")
|
||||
|
||||
# ---- .ranges cross-variant function CRC stability ----
|
||||
print(f"\n{'='*78}\nFUNCTION-CRC STABILITY (from compiled .ranges shards)\n{'='*78}")
|
||||
for gid in sorted(reps):
|
||||
vault = os.path.join(ROOT, '_coverage_vault', gid, 'cache')
|
||||
ranges = glob.glob(os.path.join(vault, '**', '*.ranges'), recursive=True)
|
||||
if not ranges: continue
|
||||
# group by region (phys prefix in filename), collect (entry->set(crc))
|
||||
region_funcs = defaultdict(lambda: defaultdict(set))
|
||||
for rf in ranges:
|
||||
stem = os.path.basename(rf)
|
||||
region = stem.split('_')[0]
|
||||
for line in open(rf, errors='ignore'):
|
||||
m = re.match(r'F ([0-9A-Fa-f]+) ([0-9A-Fa-f]+)', line)
|
||||
if m: region_funcs[region][m.group(1)].add(m.group(2))
|
||||
print(f"\n## {gid}: {len(ranges)} .ranges files")
|
||||
for region, funcs in sorted(region_funcs.items()):
|
||||
stable = sum(1 for e,cs in funcs.items() if len(cs)==1)
|
||||
variable = sum(1 for e,cs in funcs.items() if len(cs)>1)
|
||||
print(f" region {region}: {len(funcs)} distinct entry PCs; "
|
||||
f"{stable} stable(1 crc), {variable} multi-crc "
|
||||
f"({100*stable//max(1,len(funcs))}% stable-address)")
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Broadened STATIC overlay extractor for Tomba 1 — ALL code overlays, not just X*.BIN.
|
||||
|
||||
A code overlay is any disc file whose head is a {count:u32, ptr[count]:u32} export
|
||||
table with the pointers landing in the 0x800E7000 overlay window (nulls allowed for
|
||||
unused slots). This catches X0N.BIN (17), INFO.BIN, DSPSUB*.BIN, A00*.000, etc.
|
||||
Each loads verbatim/position-fixed at region 0x800E7000 + offset 904; seeds come
|
||||
from a prologue scan at base 0x800E7388. Deduped by content (AREA vs SYSTEM copies).
|
||||
"""
|
||||
import sys, os, json, base64, struct, binascii
|
||||
sys.path.insert(0, r"F:\Projects\psxrecomp\psxrecomp\tools")
|
||||
import extract_overlays as eo
|
||||
|
||||
CUE = r"F:\Projects\psxrecomp\TombaRecomp\tomba\tomba.cue"
|
||||
OUT = r"C:\Users\Matthew\AppData\Local\Temp\claude\F--Projects-psxrecomp\58ec5aaf-6d7b-4e75-b72a-d255c28f5d96\scratchpad\overlay_captures_static2.json"
|
||||
REGION_BASE = 0x800E7000
|
||||
LOAD_OFFSET = 904
|
||||
FILL = b'\x11' * LOAD_OFFSET
|
||||
|
||||
bp, raw = eo.parse_cue(CUE); disc = eo.DiscReader(bp, raw=raw)
|
||||
files = list(eo.enumerate_files(disc))
|
||||
|
||||
def is_code_overlay(data):
|
||||
"""True if head is {count, ptr[count]} with ptrs in the 0x800E7000 window
|
||||
(nulls allowed). Returns count or None."""
|
||||
if len(data) < 8: return None
|
||||
cnt = struct.unpack_from('<I', data, 0)[0]
|
||||
if not (1 <= cnt <= 4096): return None
|
||||
if 4 + cnt * 4 > len(data): return None
|
||||
ptrs = struct.unpack_from(f'<{cnt}I', data, 4)
|
||||
inrange = sum(1 for p in ptrs if 0x800E7000 <= p < 0x80200000 and (p & 3) == 0)
|
||||
nulls = sum(1 for p in ptrs if p == 0)
|
||||
# accept if every non-null pointer is a valid in-window aligned address
|
||||
return cnt if (inrange + nulls == cnt and inrange >= 1) else None
|
||||
|
||||
def prologue_seeds(data, base):
|
||||
out = []
|
||||
for off in range(0, len(data) - 4, 4):
|
||||
w = struct.unpack_from('<I', data, off)[0]
|
||||
if (w >> 16) == 0x27BD and (w & 0x8000):
|
||||
out.append(base + off)
|
||||
return out
|
||||
|
||||
# detect + dedup by content
|
||||
seen = {}
|
||||
overlays = []
|
||||
for p, l, s in files:
|
||||
if s < 8 or s > 2_000_000: continue
|
||||
try: d = disc.read_file_bytes(l, s)
|
||||
except: continue
|
||||
cnt = is_code_overlay(d)
|
||||
if cnt is None: continue
|
||||
crc = binascii.crc32(d) & 0xFFFFFFFF
|
||||
if crc in seen: continue
|
||||
seen[crc] = 1
|
||||
overlays.append((p, d))
|
||||
|
||||
records = []
|
||||
for p, d in sorted(overlays, key=lambda x: x[0]):
|
||||
seeds = prologue_seeds(d, REGION_BASE + LOAD_OFFSET)
|
||||
region = FILL + d
|
||||
records.append({
|
||||
"schema": "psxrecomp overlay capture v2",
|
||||
"load_addr": f"0x{REGION_BASE:08X}",
|
||||
"size": len(region),
|
||||
"bytes_b64": base64.b64encode(region).decode('ascii'),
|
||||
"executed_pcs": [],
|
||||
"dispatch_entry_pcs": [f"0x{a:08X}" for a in seeds],
|
||||
"function_entry_pcs": [f"0x{a:08X}" for a in seeds],
|
||||
"seeds": [f"0x{a:08X}" for a in seeds],
|
||||
"_src": p,
|
||||
})
|
||||
print(f" {p:28s} {len(d):7d}B seeds={len(seeds)}")
|
||||
|
||||
json.dump(records, open(OUT, 'w'), indent=1)
|
||||
print(f"\nwrote {len(records)} code-overlay capture records -> {OUT}")
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env python3
|
||||
"""STATIC overlay extractor for Tomba 2 (SCUS-94454) — play-free.
|
||||
|
||||
Two producers, both verbatim/position-fixed (verified vs vault):
|
||||
1. MAIN.EXE — a standard PS-X EXE (load 0x80010000, 698K). Its body is the
|
||||
dominant overlay. The runtime keys by window-clamped region_start, splitting
|
||||
at the overlay floor (0x38000): below-floor -> region 0x80010000, above-floor
|
||||
-> region 0x80038000. We emit both.
|
||||
2. BIN/A*.BIN — {count:u32, ptr[count]:u32} header (like Tomba 1) loading in the
|
||||
0x80106000 region. Load base recovered play-free by delta-sweep: the base
|
||||
where the header's absolute pointers land on addiu-sp prologues.
|
||||
Seeds: prologue scan at each region's load address.
|
||||
"""
|
||||
import sys, os, json, base64, struct, binascii
|
||||
sys.path.insert(0, r"F:\Projects\psxrecomp\psxrecomp\tools")
|
||||
import extract_overlays as eo
|
||||
|
||||
CUE = r"F:\Projects\psxrecomp\Tomba2Recomp\tomba2\Tomba! 2 - The Evil Swine Return (USA).cue"
|
||||
OUT = r"C:\Users\Matthew\AppData\Local\Temp\claude\F--Projects-psxrecomp\58ec5aaf-6d7b-4e75-b72a-d255c28f5d96\scratchpad\t2_captures.json"
|
||||
FLOOR = 0x38000 # overlay floor offset (text_end 0x38800 -> page 0x38)
|
||||
MAIN_LOAD = 0x80010000
|
||||
|
||||
bp, raw = eo.parse_cue(CUE); disc = eo.DiscReader(bp, raw=raw)
|
||||
files = list(eo.enumerate_files(disc))
|
||||
def rdfull(name):
|
||||
for p,l,s in files:
|
||||
if p.upper()==name.upper() or p.upper().endswith('/'+name.upper()): return disc.read_file_bytes(l,s)
|
||||
return None
|
||||
|
||||
def prologues(data, base):
|
||||
return [base+off for off in range(0,len(data)-4,4)
|
||||
if (struct.unpack_from('<I',data,off)[0]>>16)==0x27BD
|
||||
and (struct.unpack_from('<I',data,off)[0]&0x8000)]
|
||||
|
||||
def rec(load_addr, data, seeds):
|
||||
return {"schema":"psxrecomp overlay capture v2","load_addr":f"0x{load_addr:08X}",
|
||||
"size":len(data),"bytes_b64":base64.b64encode(data).decode(),
|
||||
"executed_pcs":[],"dispatch_entry_pcs":[f"0x{a:08X}" for a in seeds],
|
||||
"function_entry_pcs":[f"0x{a:08X}" for a in seeds],
|
||||
"seeds":[f"0x{a:08X}" for a in seeds]}
|
||||
|
||||
records=[]
|
||||
|
||||
# --- Producer 1: MAIN.EXE, split at floor ---
|
||||
main=rdfull('MAIN.EXE'); body=main[2048:]
|
||||
# below-floor region 0x80010000 .. 0x80038000
|
||||
lo_len=FLOOR-(MAIN_LOAD & 0xFFFFF if False else 0x10000) # 0x38000-0x10000 = 0x28000
|
||||
lo=body[:0x28000]
|
||||
records.append(rec(0x80010000, lo, prologues(lo,0x80010000)))
|
||||
# above-floor region 0x80038000 ..
|
||||
hi=body[0x28000:]
|
||||
records.append(rec(0x80038000, hi, prologues(hi,0x80038000)))
|
||||
print(f"MAIN.EXE: below-floor {len(lo)}B @0x80010000 seeds={len(prologues(lo,0x80010000))}; "
|
||||
f"above-floor {len(hi)}B @0x80038000 seeds={len(prologues(hi,0x80038000))}")
|
||||
|
||||
# --- Producer 2: BIN/A*.BIN via delta-sweep base recovery ---
|
||||
def recover_base(data):
|
||||
cnt=struct.unpack_from('<I',data,0)[0]
|
||||
if not (1<=cnt<=4096) or 4+cnt*4>len(data): return None
|
||||
ptrs=[p for p in struct.unpack_from(f'<{cnt}I',data,4) if p]
|
||||
if not ptrs: return None
|
||||
# content starts after header; try candidate bases so header ptrs hit prologues
|
||||
hdr=4+cnt*4
|
||||
best=None
|
||||
lo=min(ptrs)&~0xFFF
|
||||
for base in range(lo-0x4000, lo+0x1000, 4):
|
||||
hits=0
|
||||
for p in ptrs:
|
||||
off=p-base
|
||||
if 0<=off<=len(data)-4:
|
||||
w=struct.unpack_from('<I',data,off)[0]
|
||||
if (w>>16)==0x27BD and (w&0x8000): hits+=1
|
||||
if best is None or hits>best[1]: best=(base,hits)
|
||||
return best # (base, prologue-hits)
|
||||
|
||||
nbin=0
|
||||
for p,l,s in sorted(files):
|
||||
if not (p.upper().startswith('BIN/') and p.upper().endswith('.BIN')): continue
|
||||
d=disc.read_file_bytes(l,s)
|
||||
r=recover_base(d)
|
||||
if not r or r[1]==0:
|
||||
print(f" {p}: no base recovered (hits=0) — skip"); continue
|
||||
base,hits=r
|
||||
seeds=prologues(d,base)
|
||||
records.append(rec(base,d,seeds))
|
||||
nbin+=1
|
||||
if nbin<=6: print(f" {p}: base=0x{base:08X} hdr_prologue_hits={hits} seeds={len(seeds)} size={s}")
|
||||
print(f"BIN overlays: {nbin} extracted")
|
||||
|
||||
json.dump(records, open(OUT,'w'))
|
||||
print(f"\nwrote {len(records)} records -> {OUT}")
|
||||
Reference in New Issue
Block a user