diff --git a/docs/AOT_OVERLAY_PLAN.md b/docs/AOT_OVERLAY_PLAN.md new file mode 100644 index 00000000..d9f278bc --- /dev/null +++ b/docs/AOT_OVERLAY_PLAN.md @@ -0,0 +1,335 @@ +# AOT Overlay Sharding — Spike Findings & Plan (Tomba-first) + +Status: SPIKE / DESIGN (2026-07-17). Not yet executed. +Author: investigation via 4 parallel agents + adversarial source verification. +Scope: can we move overlay sharding from *runtime discovery* to *build-time (AOT)*, +starting with Tomba (SCUS-94236)? + +--- + +## ⭐ SESSION-2 BREAKTHROUGH (2026-07-17, VERIFIED FROM DISC + CORPUS) + +Adversarial disc analysis overturned the pessimistic framing below. For Tomba's +GAME-CODE overlays, **pure static analysis suffices — no playthrough, no headless +execution, no decompressor.** + +Verified facts (byte-checked against `Tomba! (USA).bin` + the 99-capture vault): +- The disc carries a full ISO9660 directory: **1085 files, 20 `AREA` dirs.** Game code + overlays are **17 distinct `X0N.BIN` files** (one per area; `X07→X01`, `X12→X04`, + `X15` absent, `AREA19` dual) — each duplicated under `SYSTEM/`. All 17 enumerate in + seconds via `extract_overlays.py`'s ISO9660 walker. +- **`X0N.BIN` is uncompressed and POSITION-FIXED.** `AREA00/X00.BIN` (343,864 B) and + `X09.BIN` (299,576 B) were found **100% byte-verbatim, contiguous** inside their + captured `0x800E7000` RAM images (X00 at region offset 904; the 904-byte prefix is + `0x11111111` fill). The file's export pointers are pre-relocated ON DISC to the exact + load address (`0x800E922C` = `0x800E7388 + file-offset`), so **disc bytes == RAM + bytes at a fixed address — zero runtime relocation, zero decompression.** This is + Class A behavior; the earlier "scatter-load / must-capture-post-fixup" concern + applies to the `.GAM` DATA packages and capture timing, NOT the code. +- **`X0N.BIN` self-describes its entry points:** a plaintext `{count:u32, + ptr[count]:u32}` header (X00: 29 pointers, all in-range). Function-entry SEEDS come + free from the file — no execution needed to find them. +- **Determinism is measured, not assumed:** across the corpus, 95–98% of overlay + function entries have a single stable code-CRC every time the address loads (Tomba + `0x800E7000`: 782/820 = 95%; `0x00000000`: 100%). Residual mismatches fall to the + interpreter via the CRC dispatch guard — coverage loss, never correctness loss. +- **The `0x80000000` region is BIOS-kernel code, not a game overlay** (entry offsets + = A0/B0/C0 vectors `0xa0/0xb0/0xc0` + SIO stub `0xcf0`; identical structure across + MMX5/MMX6/Vigilante8/Tomba — independent engines). AOT it ONCE from the BIOS, + shared by every title. + +⇒ **Revised project shape:** a build-time STATIC OVERLAY EXTRACTOR (walk disc → +enumerate 17 `X0N.BIN` → parse header for load-addr + seeds → synthesize +`overlay_captures.json` → existing `compile_overlays.py`). Headless materialization +(§5) drops to a FALLBACK for any title whose overlays turn out compressed/relocated, +not the primary path for Tomba. + +Open verification before committing (do these first — see §6 revised): +- Prove all 17 `X0N.BIN` (not just X00/X09) are 100% verbatim vs their own area's + coherent capture, at a consistent offset. +- Confirm no `X0N` has CPU-patched internal `jr` jump tables (disc≠RAM) — reconcile + against `overlay_capture.c:187-196`. X00/X09 show none. +- Confirm `.GAM`/`.000` packages contain no executable code needing shards (they look + like models/palettes/scripts = pure data → not compiled). +- Confirm Tomba code overlays live ONLY at `0x800E7000` (+ kernel `0x80000000`); no + third code base. + +--- + +## 0. TL;DR + +- **The compile step is already AOT, AND the captured coverage already ships.** + `tools/compile_overlays.py` runs fully offline; releases already bundle the + precompiled gcc DLL cache (`package_release.ps1:223-235`, + `COMPILING_OVERLAYS.md:23-33`). So "AOT the 99 captures" is the STATUS QUO, not a + goal. The goal is coverage the dev playthrough never reached. +- **"Runtime-applied fixups" are deterministic, not session-varying** — proven by the + shipped cache CRC-matching on other machines/fresh boots (§T0.5). AOT is achievable + because the post-fixup image is a fixed function of the disc; you just have to + *evaluate* it (run the loader), not read it off the disc. +- **The runtime-only input is the CAPTURE, not the compile.** The single artifact + that today only exists after a played session is `overlay_captures.json` — the + post-fixup RAM images + execution-verified entry seeds. +- **Naive "scan the disc" AOT is DISPROVEN for Tomba** (two independent reasons, + both verified in source — see §4). Do not pursue it as the Tomba path. +- **The faithful, general AOT path is HEADLESS MATERIALIZATION:** run the game's + *own* recompiled loader + decompressor + fixup code at build time, driven over + every overlay, to produce the exact `overlay_captures.json` a playthrough would — + then feed the existing offline compiler. This uses the game's real code (no + per-game decompressor reimplementation → satisfies Rule -1), is general to every + title, and is guarded by an existing correctness net. +- **The correctness net makes AOT safe to be aggressive.** Shards are + content-addressed `(entry_pc, code_crc)`; at dispatch the loader hashes live RAM + and runs native *only* if it matches (`overlay_loader.c:2137`). A wrong/stale AOT + shard never fires — it falls to the interpreter. AOT cannot cause divergence. + +--- + +## 1. How overlays work today (verified) + +### 1.1 Detection (runtime) +- Trigger: **CD-ROM channel-3 DMA completion** into game RAM below the FMV floor. + `dma.c:484-498` → `overlay_capture_on_dma(load_start, size, ram+load_start)`. +- Gated on `[runtime] overlay_cache` and auto-armed on the first post-BIOS-handoff + DMA (`overlay_capture.c:82-89`). Dedup by `load_addr` (`:91-95`). +- A second signal — sustained dirty-RAM-interp pressure sampled ~every 2s while + *not* mid-CD-load — auto-fires fresh captures for new *variants* at a reused + address (`overlay_capture.c:299-532`). + +### 1.2 Capture record (`overlay_captures.json`, schema "v2") +Built from **live-RAM dirty pages at a coherent moment**, one record per contiguous +dirty run (`overlay_capture.c:200-233`): +``` +load_addr, size, bytes_b64 (LIVE post-fixup RAM), +executed_pcs[], dispatch_entry_pcs[], function_entry_pcs[], seeds[] +``` +`executed_pcs`/`dispatch_entry_pcs` come from always-on interpreter ring tables +(`g_dirty_ram_pc_table`, `g_dirty_ram_exec_pc_table`) — **execution-verified**, not +disassembly guesses. + +### 1.3 Compile (offline, `tools/compile_overlays.py`) +1. Python does function-boundary discovery from the seeds (`classify_overlay_seeds`, + `_walk_overlay_function`) with a strict callable-prologue gate; interior/ + jump-table PCs become aliases, never walk roots. +2. Captured bytes wrapped in a **synthetic PS-X EXE** (`make_psxexe`, `:266-274`) and + handed to the *same* recompiler binary the main EXE uses: + `psxrecomp-game --seeds seeds.txt --overlay --ws-config game.toml`. +3. Out-of-overlay calls rewritten to `call_by_address()` (resolved at runtime via the + dispatch table); jump tables → native `switch` + `call_by_address` default. + `audit_generated_c()` hard-rejects a shard if any direct call escapes or any + instruction is unsupported. +4. Per-function identity = CRC32 over its exact compiled code ranges. Output: + `_.dll` + a `.ranges` manifest (`F ` / + `R `). +5. `--static` mode instead namespaces the C and folds variants into + `overlays_static.c` linked directly into the binary, dispatched via a generated + `psx_overlay_dispatch()` — **this is already an AOT delivery path**; it moves + *link time* earlier, not *discovery time*. + +### 1.4 Dispatch + cache identity (`overlay_loader.c`) +- Two stacked cache tags: ABI (`PSX_OVERLAY_ABI_VERSION`=13 + flavor) and codegen + (`cg_` path segment, auto-invalidated on any emitter source change). + Plus `verify_recompiler_matches_tag()` staleness guard. +- **Content guard (the safety net):** each DLL export registers a `Candidate` with + `crc_code`. On dispatch, `range_candidate_matches` re-hashes live RAM and runs + native only if `live == crc_code` (`overlay_loader.c:2137`; registration gate at + `:595`). Cache is **additive** — every variant ever seen for an address stays + loaded; per-dispatch CRC picks the one that currently matches (`:1844`). This is + how one address (`0x800E7xxx`) hosts village vs overworld safely. + +--- + +## 2. Tomba specifics (verified) + +- Main EXE: `load_address=0x80010000`, `entry=0x8006B58C`, `text_size=0x88000`, + loads contiguously, **no kernel-style relocation** (`game.toml:8, 276-285`). +- **Only two overlay load addresses** across 99 captures in + `_coverage_vault/SCUS-94236/`: + - `0x80000000` — 45 caps, 52–60 KB. → 11 compiled DLL variants. + - `0x800E7000` — 54 caps, 16 KB–624 KB (reused per scene). → 18 variants + (4 ranges-only / audit-failed). + - ~5.3 MB total DLL payload; largest DLL 879 KB. +- Loader = **`FUN_80021340`** (asset/area loader AND live world-streamer; green + thread, 1 state-step per vblank via `FUN_800171D4`→`ChangeTh`). +- Decompressor = **`FUN_8003EF50`** — named in `game.toml:290` as "Tomba's real + back-reference decompressor" (currently quarantined from the *data-shard replay* + path; irrelevant to AOT, which runs it rather than replaying it). Exact algorithm + not yet written up, but the function is identified and is already statically + recompiled into the main EXE. +- On-disc format = **scatter-load**: each sector is `(12-byte descriptor, + 2048-byte block)`, descriptor names the dest RAM address; content is *also* + decompressed and *also* CPU-fixed-up after landing. + +--- + +## 3. The four runtime-only gaps (what AOT must supply) + +From the runtime-arch investigation, the inputs the runtime path has that a static +pass lacks: +1. **Enumeration** of every distinct overlay *image* per RAM window (keyed by + content, not address). +2. **Load address** each image lands at. *Partially static already:* the region + floor is `(load_address+text_size)&0x1FFFFFFF` from the EXE header + (`main.cpp:3229`); base addresses (`0x80000000`, `0x800E7000`) are known. +3. **The post-fixup byte image** — decompressed AND relocated AND jump-table-patched + by the game's own code. *This is the crux for Tomba.* +4. **Execution-verified entry seeds** (`executed_pcs`/`dispatch_entry_pcs`). + +Gaps 3 and 4 are why capture is runtime-only today. + +--- + +## 4. Adversarial findings (what NOT to do, and why) + +**F1 — Naive disc-scan AOT is dead for Tomba (two independent kills):** +- (a) Scatter-load means the reconstituted RAM image is **not contiguous anywhere on + disc** (`docs/overlay-discovery.md:36-45`). A CRC/byte scan can't find it. +- (b) Even after de-scattering + decompressing, the executable bytes differ from the + raw payload because the game applies **load-time fixups (relocated jump tables)**. + Capturing pre-fixup DMA bytes was tried and caused the **village→overworld blue + screen** — abandoned (`overlay_capture.c:187-196`). The correct image exists only + *after the game's own code runs*. + +**F2 — Two PSX overlay archetypes; Tomba is the hard one.** +- *Class A (clean Psy-Q `over(group)`):* absolutely linked, position-fixed, **disc + bytes == RAM bytes byte-for-byte, only `.bss` differs** (Psy-Q '96 training deck; + Legaia RE static/dynamic byte-diff). Pure-static AOT is trivially correct here + *once the load address is known*. +- *Class B (scatter + decompress + runtime-relocate):* Tomba. Legaia's own RE team + carved these out as **"ineligible — remain on the dynamic-capture path"** + (`legend-of-legaia-re/docs/tooling/static-overlay-pipeline.md`). The clean-decomp + world already concluded you must run the game for this class. + +**F3 — There is no SDK-standard on-disc overlay directory.** The Psy-Q SDK ships no +mandated `{id→LBA→size→addr}` table; the loader table is bespoke per game +(1996 Sony seminar). So "enumerate overlays from a universal disc format" is a +fiction — enumeration is per-game (loader-table RE) or must ride the game's loader. + +**F4 — The CRC dispatch guard makes AOT risk-free to attempt.** Because a shard only +runs on a live-RAM CRC match, AOT output is *self-validating*. Worst case of a bad +AOT shard = it never fires. This means we can ship speculative AOT coverage without +gating correctness on it. + +**Confidence caveats:** the `DaCodeChick/Legaia-decomp` splat repo 404'd; PSX-splat +`.yaml` field specifics are search-derived, not re-fetched. Core Class-A claims +(position-fixed, no relocation, shared-address `over(group)`) are solidly sourced +from the Psy-Q training PDF + Legaia RE docs. Tomba compression *algorithm* is +un-characterized (function identified, bytes not decoded). + +--- + +## 5. The strategy: HEADLESS MATERIALIZATION (the complete option) + +Run the game's own recompiled loader/decompressor/fixup code at build time, over +every overlay, to produce `overlay_captures.json` **without a human playing** — then +feed the existing offline compiler. Rationale: +- **Faithful:** reproduces post-fixup bytes by executing the real code, not a + reimplemented decompressor. No per-game byte-format hack (Rule -1 compliant). +- **General:** works for any title whose loader is recompiled (all of them), + Class A and Class B alike. +- **Reuses ~everything:** CD-DMA sim, dirty-RAM interp, decompressor-as-code, capture + JSON writer, and the whole compile/dispatch pipeline already exist. The new code is + a *driver*, not a new engine. +- **Safe:** output is CRC-guarded at dispatch (F4). + +Enumeration has two escalating sub-strategies (do both; (b) is the north star): +- **(a) Route-replay bootstrap:** replay a recorded set of scene/LBA load requests + (we already have `warm_cd_routes` + the B-2 `overlay_map.jsonl` LBA provenance) to + re-trigger each load headlessly and deterministically. +- **(b) Directory-driven completeness:** locate the loader's scene/overlay table + statically (what `FUN_80021340` iterates), enumerate *all* scene IDs, and invoke + the loader for each — covering scenes no playthrough visited. + +--- + +## 6. Tiered plan (deliverables + proof artifacts per Rule 10) + +### T0 — Baseline: baking the captured coverage ALREADY SHIPS (not a new task) +Correction (surfaced by user, verified): the release already bakes the captured +coverage. `TombaRecomp/tools/package_release.ps1:223-235` bundles +`build-stable/cache/SCUS-94236/…/*.dll + *.ranges` into the shipped `cache/`, +cg-tag-namespaced, loaded from first launch (`docs/COMPILING_OVERLAYS.md:23-33`: +"ship that `cache/` folder in the release"). A tcc fallback toolchain covers gaps a +player reaches past the bundle. **So "AOT the 99" is the status quo, delivered as +precompiled gcc DLLs — no user-machine recompile for covered areas.** +- The only remaining T0-flavored option is a *delivery-form* change: + `compile_overlays.py --static` → `generated/overlays_static.c` compiled INTO the + exe (no `cache/` dir, no LoadLibrary). Same 99, single binary. Marginal; do only if + a single-file exe is wanted. **Not a coverage win.** +- **The real prize is coverage BEYOND the 99** — everything a dev's playthrough never + reached. That is T1/T2, and it is the actual point of this spike. + +### T0.5 — Empirical basis: post-fixup bytes are deterministic (already proven) +The shipped DLL cache CRC-matches and runs natively on other machines / fresh boots. +The dispatch guard runs a shard only if `live_RAM_CRC == compiled_code_CRC` +(`overlay_loader.c:2137`). A reusable cross-machine cache is therefore *proof* the +post-fixup RAM image is a fixed, reproducible function of the disc — "runtime-applied +fixups" are deterministic, not session-varying. AOT = evaluate that function once at +build time (run the game's own loader headless), not read raw disc bytes. Residual +genuinely-variable-immediate overlays (if any) just fail the CRC and fall to interp — +coverage loss, never correctness loss. + +### T1 — Headless materialization harness (the core deliverable) +A build-time mode of the runtime that boots (HLE boot-skip), then drives the loader +directly instead of via SDL input, dumping `overlay_captures.json`. +- **T1.1 Loader ABI RE:** determine `FUN_80021340`'s inputs (scene/area id, dest + addr, disc LBA) and the "load complete" signal (`DAT_1f8001ce`). Artifact: an + annotated calling-convention note + a single headless call that loads ONE known + scene and dumps a capture **byte-identical** to the vault's capture for that scene. +- **T1.2 Route-replay driver:** feed the known route (village, overworld, title, + + `warm_cd_routes` LBAs); headlessly regenerate the current 99-capture set. + Artifact: diff of headless captures vs `_coverage_vault` captures = empty (or only + benign `.bss`/timing deltas). This *reproduces today's coverage with no human*. +- **T1.3 Coherence + variant convergence:** reuse the existing "coherent moment" + capture logic so multiplexed `0x800E7000` variants are each snapshotted cleanly. +- **Proof:** `tomba-materialize.exe --captures out.json` produces a superset-or-equal + of the vault; `compile_overlays.py` over it yields shards that all pass audit and + all CRC-match at runtime. +- **Risk:** medium. The loader is *also* the world-streamer (the instant-loads + parking lesson) — but here we are NOT skipping yields; we let it run to completion + headlessly. Watch for green-thread/IRQ dependencies; drive via the real scheduler. + +### T2 — Static enumeration (directory discovery → completeness) +Remove dependence on a recorded route. +- RE the scene/overlay table `FUN_80021340` consumes (the loader's own directory). + Enumerate every scene id + its disc LBA/dest/size statically. +- Drive T1's harness over the *full* enumeration → capture overlays a playthrough + never reached. +- **Proof:** capture count and unique `code_crc` set strictly exceed the playthrough + vault; spot-navigate the game to 3 previously-uncaptured scenes and confirm their + shards CRC-match (native path taken on first entry, no compile). +- **Artifact:** `game.toml [[overlays]]` populated automatically from the enumerated + table (finally implementing the documented-but-empty schema; note the current + discrepancy in `overlay-discovery.md` that `[[overlays]]` is unimplemented). + +### T3 — Generalize the framework (Class A static fast-path + other titles) +- For Class-A titles (position-fixed, disc==RAM), add a pure-static extractor + (extend `extract_overlays.py`, seeded by the loader table) that skips headless + execution entirely — validated by the same CRC guard. +- Fold Legaia-style `jal`-call-graph base recovery for titles lacking a clean table. +- Run the harness across the ecosystem (Ape, MMX, Tomba2, …); each title's shards + become build artifacts, killing the runtime background-compile requirement fleet- + wide. +- **Proof:** ≥2 additional titles ship with fully-baked overlays, no runtime + compile, playthrough-validated. + +--- + +## 7. Immediate next actions (to power through) +1. **T0 now:** `--static` bake of the Tomba vault; boot-validate; screenshot series. + This de-risks delivery and gives a shippable artifact this session. +2. **T1.1 next:** RE `FUN_80021340` inputs; write the one-scene headless load that + reproduces a vault capture byte-for-byte. That single artifact proves the whole + headless thesis. +3. Then T1.2 route-replay, then T2 enumeration. + +## 8. Open questions / to verify before committing effort +- Does `FUN_80021340` run cleanly to completion headlessly without SDL/vblank pacing + it doesn't get? (Instant-loads parking says don't skip yields — we won't; question + is whether the harness supplies enough scheduler/IRQ progress.) +- Is a scene/overlay directory table locatable statically, or must T2 fall back to + route-replay only? (T1 works regardless; T2 completeness depends on this.) +- `.bss` / uninitialized-tail handling: captures may include zero-fill runs; confirm + the harness snapshots at the same coherent point the runtime does. +- cg-tag/ABI: any AOT run must use a same-tree recompiler (`--codegen-hash` guard), + and every fw bump re-rolls the tag → re-materialize. Bake this into the build. diff --git a/recompiler/src/main_psx.cpp b/recompiler/src/main_psx.cpp index f36cc0ac..340a9287 100644 --- a/recompiler/src/main_psx.cpp +++ b/recompiler/src/main_psx.cpp @@ -927,7 +927,22 @@ int main(int argc, char** argv) { // the on-disk representation changes. Function bodies are byte-identical // to what full_c_code would have contained; see // CodeGenerator::last_gen_funcs() / build_shared_decls_header(). - { + if (overlay_mode) { + // Overlays are single small TUs consumed by compile_overlays.py, which + // reads _full.c directly. Emit the monolith; the split path below + // would delete _full.c and leave only shards, breaking overlay compile + // (no_output). Splitting a small overlay TU has no parallel-compile + // benefit anyway. + std::ofstream full_file(output_filename); + if (full_file.is_open()) { + full_file << full_c_code; + full_file.close(); + fmt::print("✓ Saved overlay monolith to {}\n", output_filename.string()); + } else { + fmt::print(stderr, "⚠ Failed to write overlay monolith {}\n\n", + output_filename.string()); + } + } else { // 1. Remove stale outputs: the old monolith and any previously // written shards (a shard count shrink must not leave orphans). std::error_code rm_ec; diff --git a/runtime/include/overlay_api.h b/runtime/include/overlay_api.h index a786ee8e..71be4085 100644 --- a/runtime/include/overlay_api.h +++ b/runtime/include/overlay_api.h @@ -62,7 +62,12 @@ * site fails to LINK (undefined psx_ws_player_x_bound) and runs interpreted * forever — the rfe/gte_read_ctrl class again. Identity fallback (vanilla) * on a NULL host pointer keeps a v13 DLL correct (4:3) on an older host. */ -#define PSX_OVERLAY_ABI_VERSION 13 +/* v14: gte_precision_store_word forwarder — the emitter emits a direct call for + * every swc2 (GTE store-word); stateful runtime code (gte.cpp) so the DLL + * must forward or fail to LINK. Surfaced by full static overlay coverage + * (play-captured shards rarely compiled a swc2). Appended last; ABI bump + * to 14 rejects any pre-shim DLL. */ +#define PSX_OVERLAY_ABI_VERSION 14 /* Codegen flavor of the recompiled output the overlays + runtime were built * against. Overlays are keyed in the cache by guest-bytes CRC, which is @@ -221,6 +226,16 @@ typedef struct { * psx_ws_player_x_bound). Appended last; may be NULL on a host that * predates it — the DLL glue falls back to the vanilla constant (4:3). */ int32_t (*ws_player_x_bound)(int32_t vanilla); + /* Sub-pixel GTE precision-store tracker (ABI v14; see version-history note + * above). The emitter emits a direct gte_precision_store_word() call for + * every swc2 (GTE store-word); it is stateful runtime code (gte.cpp) so the + * DLL must forward. Without the shim any shard containing a swc2 fails to + * LINK (undefined gte_precision_store_word) and runs interpreted forever — + * the rfe/gte_read_ctrl class again. Surfaced by full static overlay + * coverage (play-captured shards rarely compiled a swc2). Appended last; + * may be NULL on a host that predates it — the shim no-ops (precision + * tracking is a widescreen enhancement, off by default). */ + void (*gte_precision_store_word)(uint32_t addr, uint8_t reg); } OverlayCallbacks; #ifdef __cplusplus diff --git a/runtime/include/overlay_dispatch_preamble.c.inc b/runtime/include/overlay_dispatch_preamble.c.inc index 7c0c17d3..8e299972 100644 --- a/runtime/include/overlay_dispatch_preamble.c.inc +++ b/runtime/include/overlay_dispatch_preamble.c.inc @@ -223,5 +223,15 @@ uint32_t psx_ws_backdrop_value(uint32_t orig, int is_end, int window_cols) { int32_t psx_ws_player_x_bound(int32_t vanilla) { return g_cbs.ws_player_x_bound ? g_cbs.ws_player_x_bound(vanilla) : vanilla; } +/* Sub-pixel GTE precision-store tracker (ABI v14). The emitter emits a direct + * gte_precision_store_word() call for every swc2 (GTE store-word), including in + * overlay-resident code, so the DLL must define it or the shard fails to link + * (undefined reference — the rfe/gte_read_ctrl class). Forwards to the host's + * stateful tracker (gte.cpp); no-op if the host predates the callback + * (precision tracking is a widescreen enhancement, off by default — 4:3 + * behavior is unaffected). */ +void gte_precision_store_word(uint32_t addr, uint8_t reg) { + if (g_cbs.gte_precision_store_word) g_cbs.gte_precision_store_word(addr, reg); +} /* ----------------------------------------------------------------------- */ diff --git a/runtime/src/overlay_loader.c b/runtime/src/overlay_loader.c index b801d891..257c8c31 100644 --- a/runtime/src/overlay_loader.c +++ b/runtime/src/overlay_loader.c @@ -1552,6 +1552,14 @@ static void init_callbacks(void) { extern int32_t psx_ws_player_x_bound(int32_t vanilla); s_callbacks.ws_player_x_bound = psx_ws_player_x_bound; } + /* ABI v14: GTE precision-store tracker — the emitter emits a direct + * gte_precision_store_word() call for every swc2 (GTE store-word), + * including in overlay-resident code; it mutates the host's stateful + * sub-pixel projection cache (gte.cpp) so the DLL must forward. */ + { + extern void gte_precision_store_word(uint32_t addr, uint8_t reg); + s_callbacks.gte_precision_store_word = gte_precision_store_word; + } } } diff --git a/tools/aot_overlay_spike/README.md b/tools/aot_overlay_spike/README.md new file mode 100644 index 00000000..b6176d9b --- /dev/null +++ b/tools/aot_overlay_spike/README.md @@ -0,0 +1,49 @@ +# AOT Overlay Sharding — Spike Tooling (WIP) + +Play-free static overlay extraction: discover + compile overlay shards from the +disc image at build time, so a fresh install ships with overlay coverage instead +of relying entirely on runtime (tcc/gcc) autocompile. See `docs/AOT_OVERLAY_PLAN.md` +for the full design, findings, and honest coverage numbers. + +**Status: SPIKE. Provably-correct for clean producers, not exhaustive.** These +scripts carry hardcoded paths (disc, scratchpad) and are Tomba-specific proofs of +concept, not a finished general tool. They exist so a future session can improve +discovery quality, not to be run as-is in a build. + +## What each does + +- `overlay_determinism.py` — corpus analysis across all games' captured overlays: + base-address regularity, size/sector alignment, per-function code-CRC stability. + Establishes that overlay code is deterministic (95-98% stable code-CRC). +- `tomba1_extract.py` — Tomba 1 (SCUS-94236). Enumerates ALL code overlays from + the disc by header signature ({count:u32, ptr[count]:u32}), NOT just X*.BIN + (that filter missed INFO.BIN → Dwarf Village ran interpreted/slow). Emits a + synthetic overlay_captures.json for compile_overlays.py. Position-fixed, + verbatim, seeds via prologue scan @ region+904. +- `tomba2_extract.py` — Tomba 2 (SCUS-94454). Two producers: MAIN.EXE (a PS-X EXE + whose header self-describes the load address — ROBUST) and BIN/A*.BIN (header + table; base recovery via prologue delta-sweep — UNRELIABLE, produces data-as-code). + +## Validated result (see plan doc for detail) + +- **Tomba 1:** 90% of played functions reproduced play-free + ~1964 unplayed + extra; live-validated in-game (Dwarf Village went native, no runtime compile). +- **Tomba 2:** 661 functions byte-identical (entry+code_crc) to the shipped/played + vault — provably real — ALL from the MAIN.EXE producer; the BIN producer yields + 0 usable shards (broken base recovery). Generality is producer-dependent. + +## Correctness guarantee (why partial coverage is safe) + +Every shard is content-addressed by per-function `code_crc`; the runtime loader +only executes a shard when live RAM byte-matches (`overlay_loader.c` lazy_man_matches). +A mis-positioned / data-as-code shard either fails audit at compile time or never +fires at runtime → coverage loss, never incorrect execution. Whatever static +misses, production autocompile (tcc/gcc) self-heals on first visit. + +## Next to raise coverage (future session) + +1. Frameless-leaf + indirect-call-table seed discovery (helps every game; would + lift MAIN.EXE past ~62% and Tomba 1 past 90%). +2. Real loader-table RE for header-table (BIN) base recovery (per-game). +3. Extract the kernel `0x80000000` region once from the BIOS (game-independent). +4. Generalize into a real `tools/` producer (no hardcoded paths; disc + game.toml in). diff --git a/tools/aot_overlay_spike/overlay_determinism.py b/tools/aot_overlay_spike/overlay_determinism.py new file mode 100644 index 00000000..9dccf112 --- /dev/null +++ b/tools/aot_overlay_spike/overlay_determinism.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Corpus analysis: is there a DETERMINISTIC story across captured overlays? + +Scans every overlay_captures.json + cache/*.ranges under the project tree, +one representative per game, and quantifies: + - base-address regularity (how many distinct load_addr per game; reuse depth) + - relation of each base to the EXE text layout (load_address, text_size from game.toml) + - size distribution + sector (2048) alignment + - seed / dispatch-entry clustering (offset of entries from the region base) + - cross-variant function-CRC stability (from .ranges: do (entry,crc) recur?) +""" +import json, os, re, glob, base64 +from collections import defaultdict, Counter + +ROOT = r"F:\Projects\psxrecomp" + +# ---- pick one representative captures.json per game id (prefer vault, else largest) ---- +def game_id_of(path): + m = re.search(r'(SC[UE]S-\d{5}|SLP[SM]-\d{5})', path) + if m: return m.group(1) + # fall back to repo dir name + parts = path.replace('\\','/').split('/') + for p in parts: + if p.endswith('Recomp'): return p + return parts[-2] + +def rec_count(p): + try: + with open(p, 'r', errors='ignore') as f: return f.read().count('"load_addr"') + except: return 0 + +cands = defaultdict(list) +for p in glob.glob(os.path.join(ROOT, '**', 'overlay_captures.json'), recursive=True): + if any(x in p for x in ('_wt-', '_stale', '_backup', '_release_test')): continue + cands[game_id_of(p)].append(p) + +reps = {} +for gid, ps in cands.items(): + ps.sort(key=lambda p: (0 if '_coverage_vault' in p else 1, -rec_count(p))) + reps[gid] = ps[0] + +# ---- load game.toml text layout per game ---- +def find_toml_layout(gid): + for tp in glob.glob(os.path.join(ROOT, '*Recomp*', 'game.toml')): + try: + t = open(tp, errors='ignore').read() + except: continue + if gid in t or gid.replace('-','_') in t: + la = re.search(r'load_address\s*=\s*"?(0x[0-9A-Fa-f]+)"?', t) + ts = re.search(r'text_size\s*=\s*"?(0x[0-9A-Fa-f]+)"?', t) + return (int(la.group(1),16) if la else None, + int(ts.group(1),16) if ts else None, tp) + return (None, None, None) + +def norm(a): # KSEG-mask to physical, keep 0x80000000 base for display + return a & 0x1FFFFFFF + +print("="*78) +print("OVERLAY DETERMINISM CORPUS ANALYSIS") +print("="*78) + +for gid in sorted(reps, key=lambda g: -rec_count(reps[g])): + path = reps[gid] + try: + data = json.load(open(path, errors='ignore')) + except Exception as e: + print(f"\n## {gid}: FAILED to parse ({e})"); continue + if not isinstance(data, list) or not data: continue + la, ts, tp = find_toml_layout(gid) + text_end = (la + ts) if (la and ts) else None + + by_base = defaultdict(list) + for r in data: + try: base = int(r['load_addr'], 16) + except: continue + by_base[base].append(r) + + print(f"\n{'#'*70}\n## {gid} ({len(data)} records, {len(by_base)} distinct load_addr)") + print(f" repr: {os.path.relpath(path, ROOT)}") + if la: print(f" EXE: load_address={hex(la)} text_size={hex(ts) if ts else '?'} " + f"text_end={hex(text_end) if text_end else '?'}") + + for base in sorted(by_base): + recs = by_base[base] + sizes = [r.get('size',0) for r in recs] + uq = sorted(set(sizes)) + align = Counter('sector' if s % 2048 == 0 else ('1k' if s % 1024 == 0 else 'other') for s in sizes) + # entry offsets: first seed/dispatch relative to base + offs = [] + for r in recs: + seeds = r.get('seeds') or r.get('dispatch_entry_pcs') or [] + for s in seeds: + try: offs.append(int(s,16) - base) + except: pass + rel = "" + if text_end is not None: + if norm(base) == norm(text_end): rel = " <== EXACTLY text_end" + elif abs(norm(base)-norm(text_end)) < 0x2000: rel = f" (~text_end {norm(base)-norm(text_end):+#x})" + elif norm(base) < norm(la): rel = " (below load_address / low-RAM)" + print(f" base {hex(base)}: {len(recs)} caps{rel}") + print(f" sizes: {len(uq)} unique, min={min(sizes)} max={max(sizes)} " + f"({min(sizes)//1024}K..{max(sizes)//1024}K), align={dict(align)}") + if offs: + offs_s = sorted(set(offs)) + head = [hex(o) for o in offs_s[:6]] + print(f" entry offsets from base: {len(offs_s)} unique, " + f"min={hex(min(offs))} max={hex(max(offs))}, first few={head}") + +# ---- .ranges cross-variant function CRC stability ---- +print(f"\n{'='*78}\nFUNCTION-CRC STABILITY (from compiled .ranges shards)\n{'='*78}") +for gid in sorted(reps): + vault = os.path.join(ROOT, '_coverage_vault', gid, 'cache') + ranges = glob.glob(os.path.join(vault, '**', '*.ranges'), recursive=True) + if not ranges: continue + # group by region (phys prefix in filename), collect (entry->set(crc)) + region_funcs = defaultdict(lambda: defaultdict(set)) + for rf in ranges: + stem = os.path.basename(rf) + region = stem.split('_')[0] + for line in open(rf, errors='ignore'): + m = re.match(r'F ([0-9A-Fa-f]+) ([0-9A-Fa-f]+)', line) + if m: region_funcs[region][m.group(1)].add(m.group(2)) + print(f"\n## {gid}: {len(ranges)} .ranges files") + for region, funcs in sorted(region_funcs.items()): + stable = sum(1 for e,cs in funcs.items() if len(cs)==1) + variable = sum(1 for e,cs in funcs.items() if len(cs)>1) + print(f" region {region}: {len(funcs)} distinct entry PCs; " + f"{stable} stable(1 crc), {variable} multi-crc " + f"({100*stable//max(1,len(funcs))}% stable-address)") diff --git a/tools/aot_overlay_spike/tomba1_extract.py b/tools/aot_overlay_spike/tomba1_extract.py new file mode 100644 index 00000000..99d8a919 --- /dev/null +++ b/tools/aot_overlay_spike/tomba1_extract.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Broadened STATIC overlay extractor for Tomba 1 — ALL code overlays, not just X*.BIN. + +A code overlay is any disc file whose head is a {count:u32, ptr[count]:u32} export +table with the pointers landing in the 0x800E7000 overlay window (nulls allowed for +unused slots). This catches X0N.BIN (17), INFO.BIN, DSPSUB*.BIN, A00*.000, etc. +Each loads verbatim/position-fixed at region 0x800E7000 + offset 904; seeds come +from a prologue scan at base 0x800E7388. Deduped by content (AREA vs SYSTEM copies). +""" +import sys, os, json, base64, struct, binascii +sys.path.insert(0, r"F:\Projects\psxrecomp\psxrecomp\tools") +import extract_overlays as eo + +CUE = r"F:\Projects\psxrecomp\TombaRecomp\tomba\tomba.cue" +OUT = r"C:\Users\Matthew\AppData\Local\Temp\claude\F--Projects-psxrecomp\58ec5aaf-6d7b-4e75-b72a-d255c28f5d96\scratchpad\overlay_captures_static2.json" +REGION_BASE = 0x800E7000 +LOAD_OFFSET = 904 +FILL = b'\x11' * LOAD_OFFSET + +bp, raw = eo.parse_cue(CUE); disc = eo.DiscReader(bp, raw=raw) +files = list(eo.enumerate_files(disc)) + +def is_code_overlay(data): + """True if head is {count, ptr[count]} with ptrs in the 0x800E7000 window + (nulls allowed). Returns count or None.""" + if len(data) < 8: return None + cnt = struct.unpack_from(' len(data): return None + ptrs = struct.unpack_from(f'<{cnt}I', data, 4) + inrange = sum(1 for p in ptrs if 0x800E7000 <= p < 0x80200000 and (p & 3) == 0) + nulls = sum(1 for p in ptrs if p == 0) + # accept if every non-null pointer is a valid in-window aligned address + return cnt if (inrange + nulls == cnt and inrange >= 1) else None + +def prologue_seeds(data, base): + out = [] + for off in range(0, len(data) - 4, 4): + w = struct.unpack_from('> 16) == 0x27BD and (w & 0x8000): + out.append(base + off) + return out + +# detect + dedup by content +seen = {} +overlays = [] +for p, l, s in files: + if s < 8 or s > 2_000_000: continue + try: d = disc.read_file_bytes(l, s) + except: continue + cnt = is_code_overlay(d) + if cnt is None: continue + crc = binascii.crc32(d) & 0xFFFFFFFF + if crc in seen: continue + seen[crc] = 1 + overlays.append((p, d)) + +records = [] +for p, d in sorted(overlays, key=lambda x: x[0]): + seeds = prologue_seeds(d, REGION_BASE + LOAD_OFFSET) + region = FILL + d + records.append({ + "schema": "psxrecomp overlay capture v2", + "load_addr": f"0x{REGION_BASE:08X}", + "size": len(region), + "bytes_b64": base64.b64encode(region).decode('ascii'), + "executed_pcs": [], + "dispatch_entry_pcs": [f"0x{a:08X}" for a in seeds], + "function_entry_pcs": [f"0x{a:08X}" for a in seeds], + "seeds": [f"0x{a:08X}" for a in seeds], + "_src": p, + }) + print(f" {p:28s} {len(d):7d}B seeds={len(seeds)}") + +json.dump(records, open(OUT, 'w'), indent=1) +print(f"\nwrote {len(records)} code-overlay capture records -> {OUT}") diff --git a/tools/aot_overlay_spike/tomba2_extract.py b/tools/aot_overlay_spike/tomba2_extract.py new file mode 100644 index 00000000..1cffdcb8 --- /dev/null +++ b/tools/aot_overlay_spike/tomba2_extract.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""STATIC overlay extractor for Tomba 2 (SCUS-94454) — play-free. + +Two producers, both verbatim/position-fixed (verified vs vault): + 1. MAIN.EXE — a standard PS-X EXE (load 0x80010000, 698K). Its body is the + dominant overlay. The runtime keys by window-clamped region_start, splitting + at the overlay floor (0x38000): below-floor -> region 0x80010000, above-floor + -> region 0x80038000. We emit both. + 2. BIN/A*.BIN — {count:u32, ptr[count]:u32} header (like Tomba 1) loading in the + 0x80106000 region. Load base recovered play-free by delta-sweep: the base + where the header's absolute pointers land on addiu-sp prologues. +Seeds: prologue scan at each region's load address. +""" +import sys, os, json, base64, struct, binascii +sys.path.insert(0, r"F:\Projects\psxrecomp\psxrecomp\tools") +import extract_overlays as eo + +CUE = r"F:\Projects\psxrecomp\Tomba2Recomp\tomba2\Tomba! 2 - The Evil Swine Return (USA).cue" +OUT = r"C:\Users\Matthew\AppData\Local\Temp\claude\F--Projects-psxrecomp\58ec5aaf-6d7b-4e75-b72a-d255c28f5d96\scratchpad\t2_captures.json" +FLOOR = 0x38000 # overlay floor offset (text_end 0x38800 -> page 0x38) +MAIN_LOAD = 0x80010000 + +bp, raw = eo.parse_cue(CUE); disc = eo.DiscReader(bp, raw=raw) +files = list(eo.enumerate_files(disc)) +def rdfull(name): + for p,l,s in files: + if p.upper()==name.upper() or p.upper().endswith('/'+name.upper()): return disc.read_file_bytes(l,s) + return None + +def prologues(data, base): + return [base+off for off in range(0,len(data)-4,4) + if (struct.unpack_from('>16)==0x27BD + and (struct.unpack_from('len(data): return None + ptrs=[p for p in struct.unpack_from(f'<{cnt}I',data,4) if p] + if not ptrs: return None + # content starts after header; try candidate bases so header ptrs hit prologues + hdr=4+cnt*4 + best=None + lo=min(ptrs)&~0xFFF + for base in range(lo-0x4000, lo+0x1000, 4): + hits=0 + for p in ptrs: + off=p-base + if 0<=off<=len(data)-4: + w=struct.unpack_from('>16)==0x27BD and (w&0x8000): hits+=1 + if best is None or hits>best[1]: best=(base,hits) + return best # (base, prologue-hits) + +nbin=0 +for p,l,s in sorted(files): + if not (p.upper().startswith('BIN/') and p.upper().endswith('.BIN')): continue + d=disc.read_file_bytes(l,s) + r=recover_base(d) + if not r or r[1]==0: + print(f" {p}: no base recovered (hits=0) — skip"); continue + base,hits=r + seeds=prologues(d,base) + records.append(rec(base,d,seeds)) + nbin+=1 + if nbin<=6: print(f" {p}: base=0x{base:08X} hdr_prologue_hits={hits} seeds={len(seeds)} size={s}") +print(f"BIN overlays: {nbin} extracted") + +json.dump(records, open(OUT,'w')) +print(f"\nwrote {len(records)} records -> {OUT}")