diff --git a/docs/FEATURES.md b/docs/FEATURES.md new file mode 100644 index 00000000..94bc5afd --- /dev/null +++ b/docs/FEATURES.md @@ -0,0 +1,111 @@ +# PSXRecomp Overlay System — Feature Reference + +## Overview + +PSX games use **overlays**: regions of game code that are loaded from disc into a fixed +RAM window at runtime via CD DMA. The same physical addresses are reused by different +overlay images throughout the game. The static recompiler only sees the boot EXE; it +cannot see overlay code that does not exist at compile time. + +The overlay system has two layers, B and A, that together solve this. + +--- + +## Layer B — Capture and static compilation (IMPLEMENTED, B-1 done / B-2 in progress) + +### B-1: Overlay capture (runtime, done) + +`overlay_capture.c` hooks every CD DMA completion into game-code RAM +(`dest < 0x1C0000`). On each unique load address, it stores the DMA-time bytes +(unpatched copy, taken before the game can modify them) in an in-process set. + +On clean process exit, `overlay_capture_write_json()` writes +`build-dev/overlay_captures.json` alongside the executable. The JSON contains: + +- `load_addr` — virtual address where the overlay loaded (e.g. `0x800E7000`) +- `size` — byte count of the full dirty region +- `bytes_b64` — base64 of the DMA-time image (zero-filled for scatter-load gaps) +- `seeds` — per-PC interpreter hit addresses (execution-verified entry points) + +The `seeds` list is how we know which addresses inside the overlay are real function +entry points. They come from `dirty_ram_interp`'s PC table: addresses the interpreter +actually branched to during gameplay. + +### B-2: Static overlay compilation (build-time, in progress) + +`compile_overlays.py` reads `overlay_captures.json`, wraps the bytes in a fake PS-EXE +header, runs `psxrecomp-game` to recompile, post-processes the generated C for +standalone compilation, and produces a C file that is compiled into the binary. + +In **static mode** (B-2), the output is a `.c` file placed in `generated/overlays.c` +that the CMake build includes directly. No DLL, no runtime loading, no CRC matching. +The overlay functions join the static dispatch table exactly like game EXE functions. + +`game.toml` gains an `[[overlays]]` section that points to the captures file; CMake +includes `generated/overlays.c` when it exists. + +--- + +## Layer A — Dynamic DLL loading (runtime, SHELVED) + +### What it was supposed to do + +A-1 was the "live" complement to B: while B produces a binary rebuild, A-1 lets a +user contribute a new overlay by dropping a compiled DLL into a cache directory. +The runtime detects the overlay firing in `dirty_ram_dispatch`, finds the matching +DLL in the cache, loads it with `LoadLibrary`, and wires the overlay functions into +a dynamic dispatch hash table — all without a rebuild. + +The idea was that contributors could run `compile_overlays.py`, drop the DLL in +`build-dev/cache//`, and immediately have compiled overlay code. + +### What we built + +- `overlay_loader.c` — dynamic dispatch hash table (16 384 slots), DLL load + PE + export enumeration, `overlay_loader_dispatch` called from `dirty_ram_dispatch` +- `overlay_api.h` — shared ABI: `OverlayCallbacks` struct, `overlay_init` entry point +- `compile_overlays.py` DLL path — generates `{addr}_{crc}.dll` in the cache dir +- `overlay_loader_status` TCP debug command + +### Why it was shelved + +**Load Game regression.** When a DLL exists in the cache, loading a save triggers +the overlay dispatch, which loads the DLL, which runs the compiled overlay code. +This crashed consistently. Clearing the DLL cache restores stable behavior. We never +confirmed whether the compiled code itself is correct (the crash analysis was blocked +by the plumbing issues below). + +**CRC key instability.** We tried three different naming schemes before landing on +`{addr}_{dma_crc}.dll`: + +1. CRC of live RAM — wrong: scatter-load gaps contain old data in live RAM but zeros + in the DMA-assembled image, so the CRC never matched. +2. Address-only — no collision safety. +3. DMA-CRC (from `overlay_capture_get_region_crc`) — correct in theory, but the DMA + blocks are not in `s_entries` after a fast-boot restore (no DMA fired), so the + function hashed a zeroed image and produced the wrong CRC. + +**Final fix attempted:** scan the cache dir at init for `{addr}_*.dll` files and +build a `region_start → path` index. This avoids recomputing the CRC at dispatch +time. The strlen check in the scanner had an off-by-one (22 vs 21) that was fixed, +but the Load Game crash was not diagnosed before shelving. + +### What remains for A-1 to ship + +1. Confirm the generated DLL code is actually correct (same question as B-2). + Debug with `B476006F_patched.c` in `build-dev/cache/SCUS-94236/`. +2. Understand why Load Game crashes with the DLL active. +3. Possibly audit `patch_generated_c` in `compile_overlays.py` for incorrect + cross-call patching or missing extern declarations. + +The infrastructure is complete. The blocker is code correctness, not plumbing. + +--- + +## Tomba overlay map (known) + +| Region | Load addr | Size | Seeds | Status | +|----------------|--------------|---------|-------|---------------------| +| `000E7000` | `0x800E7000` | 364 544 | 66 | Captured, DLL built, shelved | + +More overlays will appear as gameplay progresses past Dwarf Village. diff --git a/docs/overlay-discovery.md b/docs/overlay-discovery.md index 465bc4e1..739d7800 100644 --- a/docs/overlay-discovery.md +++ b/docs/overlay-discovery.md @@ -1,7 +1,6 @@ # Overlay Discovery -**Date:** 2026-05-29 -**Branch:** `overlay-discovery` +**Updated:** 2026-05-30 **Phase gate:** Phase 5+ (game EXE loaded, dirty-RAM hits observed in game address space) --- @@ -24,133 +23,141 @@ interpretation of overlay code. BIOS dirty-RAM (Rule 18 in CLAUDE.md) is code the BIOS *assembles in-place* at boot — e.g. the 4-instruction SIO handler written to RAM 0xCF0. That code never existed on disc. -Game overlays are different: they are complete compiled MIPS modules on the disc filesystem, -loaded to fixed RAM addresses. Their bytes are known ahead of time; the recompiler just hasn't -seen them yet. This system fixes that. +Game overlays are different: they are compiled MIPS modules loaded to fixed RAM addresses. Their +bytes are known after they land in RAM; the recompiler just hasn't seen them yet. This system +fixes that. Missed heuristics (static-EXE functions the recompiler failed to identify) are orthogonal: they -exist in the main EXE at build time and will not match any disc file hash. The overlay system -ignores them and they fall through to dirty-RAM as today. Fix missed heuristics by improving -heuristics; fix overlays by this system. +exist in the main EXE at build time. Fix missed heuristics by improving heuristics; fix overlays +by this system. + +--- + +## Why disc extraction is not used + +The naive approach — scan the disc image for overlay bytes by CRC — does not work for Tomba and +many other PSX titles. Investigation confirmed that Tomba stores overlays in a **scatter-load +format**: each disc sector is a `(12-byte block descriptor, 2048-byte data block)` pair, where +the descriptor tells the loader which RAM address to write the data block to. The resulting RAM +content is not present anywhere on disc as a contiguous byte range, so no CRC or brute-force scan +can find it. + +The universal correct primitive is to capture bytes **at DMA time** — intercepting them as they +land in RAM, after all scatter-reassembly and decompression. This works for: + +- Scatter-load formats (Tomba, others) +- Game-specific LZ or RLE compression (bytes arrive decompressed via DMA) +- Direct file loads (trivially captured) --- ## Architecture Two layers, composing. As Layer B grows, Layer A fires less. Eventually Layer A goes silent. +Layer B is a manifest of Layer A's output: the same `(load_addr, bytes, seeds)` data, committed +to the repo and baked into the binary ahead of time. -### Layer B — Persistent map → static precompilation +### Layer B — Build-time pre-compilation -**Runtime side:** whenever CD DMA completes a load into RAM, record `(disc_file_hash, load_addr, -size)` to a per-game log file on disk. This is cheap — a hash computation and a file append. +**Runtime side (capture):** on every CD DMA completion into game-code RAM, the runtime appends +`(load_addr, size, bytes)` to a ring buffer. On process exit it writes +`overlay_captures.json` automatically — no user action required. The user discovers the file +exists and can attach it to a GitHub issue. -**Build-time side:** `recompiler/recomp.py` reads the log file. For each entry, it locates the -matching file on the disc image by hash, disassembles it at the recorded load address, and -emits a compiled C function set alongside the main EXE output. The overlay's functions are added -to the static dispatch table at their load addresses. +**Build-time side:** `psxrecomp-game` reads `game.toml [[overlays]]` entries, each pointing to +a captured `.bin` file and a list of seed entry points. It disassembles the MIPS bytes at +`load_addr`, emits C functions, and adds them to the static dispatch table. The shipped binary +has those overlays pre-compiled at zero runtime cost. -**Result:** any overlay logged in a previous run is statically compiled into the next build. No -runtime overhead on subsequent loads. +**Coverage grows** with every release as developer playthroughs and user contributions feed new +captures into the build. ### Layer A — Runtime background compilation with persistent cache -**On CD DMA completion:** -1. Hash the written bytes. -2. Check the on-disk compiled cache (`cache//.dll`). - - Cache hit → `LoadLibrary` the cached DLL, register its functions in the live dispatch table - for `[load_addr, load_addr + size)`. No interpretation ever needed. - - Cache miss → fall through to dirty-RAM interpretation. Queue a background thread to compile - this overlay (emit C for the bytes at `load_addr`, invoke the C compiler, write the DLL to - cache). On the *next* load of this overlay (next room visit, next level reload), the cache - hit path fires instead. +**On CD DMA completion into game-code RAM:** -**Result:** within a single playthrough, each overlay compiles once on first encounter. Every -subsequent encounter in that session, and in all future sessions, hits the cache. +1. Hash the written bytes. +2. Check `cache//.dll`. + - **Cache hit** → `LoadLibrary` the DLL, register its functions in the live dispatch table + for `[load_addr, load_addr + size)`. Interpreter never fires for this overlay. + - **Cache miss** → fall through to dirty-RAM interpretation. Enqueue a background thread to + compile this overlay: emit C for the bytes at `load_addr`, invoke the C compiler, write the + DLL to cache. On the *next* load of this overlay the cache hit fires. + +**Result:** within a single session, each new overlay is interpreted once and compiled once. +Every subsequent encounter — same session or future sessions — dispatches to native code. + +A user who plays through the entire game accumulates a complete cache. From that point forward +the game runs entirely as compiled native code, never touching the interpreter. ### How they compose ``` -Dispatch path (on branch to overlay address): - 1. Static dispatch table (Layer B) → precompiled function [zero runtime cost] - 2. Compiled cache (Layer A) → LoadLibrary'd function [one-time per session after first] - 3. Dirty-RAM interpreter → fallback [first-ever encounter only] +Dispatch (branch to overlay address): + 1. Static dispatch table [Layer B] → precompiled function zero runtime cost + 2. Compiled cache [Layer A] → LoadLibrary'd function one-time per session after first + 3. Dirty-RAM interpreter → fallback first-ever encounter only ``` Layer B coverage grows with every rebuild. Layer A covers the gap within a session before the -rebuild happens. Dirty-RAM is the last-resort fallback that fires exactly once per undiscovered -overlay. +rebuild happens. Dirty-RAM fires exactly once per undiscovered overlay. --- ## Game identification -Cache and log files are namespaced by SCUS/SLUS/SCES product code, read from `SYSTEM.CNF` on the -disc image at startup (`BOOT=cdrom:\SCUS_941.60;1` → key `SCUS-94160`). +Cache and capture files are namespaced by SCUS/SLUS/SCES product code, read from `SYSTEM.CNF`: ``` cache/ - SCUS-94160/ ← Tomba (NTSC-U) + SCUS-94236/ ← Tomba (NTSC-U) .dll - .dll - SCES-01234/ ← different region, separate cache - ... - logs/ - SCUS-94160/ - overlay_map.jsonl ← append-only: {hash, load_addr, size, disc_file} + SCUS-94236/ + overlay_map.jsonl +overlay_captures.json ← written at exit, next to the executable ``` -Different regional versions get separate caches automatically, which is correct since overlay -load addresses may differ between regions. - --- ## Cache invalidation -A cached DLL is keyed by the hash of the overlay bytes. The hash IS the identity. If a disc file -changes (different dump, different region), the hash changes and the old cache entry is simply -never matched — it sits unused. No explicit invalidation needed. +A cached DLL is keyed by the hash of the overlay bytes. The hash IS the identity. If a disc dump +differs between users or regions, the hash differs and the old cache entry is simply never +matched. No explicit invalidation needed. When Layer B statically compiles an overlay, Layer A's cached DLL for that hash becomes unreachable (static dispatch wins at step 1). It can be pruned post-build but is benign if left. --- -## Integration point - -The natural hook is the CD-ROM DMA completion callback in `runtime/src/cdrom.c`. This is where -bytes land in RAM. After the DMA write: -1. Compute hash of the destination range. -2. Log `(hash, load_addr, size, source_file)` for Layer B. -3. Check Layer A cache. If hit, register the DLL's functions in the live dispatch table. -4. If miss, mark the range dirty (existing dirty-RAM path) and enqueue background compilation. - ---- - ## Implementation phases -| Phase | Work | -|---|---| -| B-1 | CD DMA completion hook logs `(hash, load_addr, size)` to `logs//overlay_map.jsonl` | -| B-2 | `recomp.py` reads the log and emits overlay functions alongside the main EXE; adds entries to dispatch table | -| B-3 | Build-time validation: for each log entry, verify disc file hash matches before compiling | -| A-1 | On DMA completion, check `cache//.dll`; if present, `LoadLibrary` + register | -| A-2 | Background thread: emit C for newly-seen overlay, compile to DLL, write to cache | -| A-3 | On next load of same overlay, Layer A cache hit fires instead of dirty-RAM | +| Phase | Work | Status | +|---|---|---| +| — | Dirty-RAM interpreter (Layer A last-resort fallback) | Done | +| B-1 | CD DMA byte capture ring (always-on, retail-safe, `dest < 0x1C0000` filter) | To build | +| B-1 | Automatic `overlay_captures.json` written at process exit | To build | +| B-2 | `game.toml [[overlays]]` + `psxrecomp-game` overlay compilation | To build | +| B-3 | Contribution ingestion script: JSON → `.bin` + `game.toml` entries | To build | +| A-1 | On DMA completion, check cache DLL; if present, `LoadLibrary` + register | To build | +| A-2 | Background thread: emit C, compile to DLL, write to cache | To build | +| A-3 | On next overlay encounter, A-1 fires instead of dirty-RAM | Follows from A-1/A-2 | -B-1 through B-3 can ship before A-1. The system is useful with only Layer B (just requires a -rebuild after each new playthrough). Layer A is the quality-of-life layer that eliminates the -"rebuild required" friction. +B-1 through B-3 ship before A-1. The system is useful with only Layer B (requires a rebuild +after each new playthrough contribution). Layer A eliminates the "rebuild required" friction and +makes the game self-healing for any user. --- -## Open questions +## Scatter-load discovery (Tomba) -- Does Tomba use named disc files (ISO 9660) or raw LBA sector reads for overlay loads? Named - files make Layer B trivial (hash the file, record the name). Raw LBA reads require tracing the - sector number through to the ISO to recover the file identity. Determine by inspecting - dirty-RAM dispatch miss addresses: if they cluster at 2-3 fixed load addresses, it's classic - file-based overlays. If they scatter, it may be more ad-hoc. +Tomba's CD DMA log (captured 2026-05-30) confirmed the scatter-load format: -- `LoadLibrary` / `dlopen` infrastructure does not currently exist in the runtime. Layer A - requires it. Layer B does not. Implement B first; defer A until B has real coverage data. +- Every sector pair: `12-byte block descriptor` (always to a header buffer at `0x973C0`) + + `2048-byte data block` (scattered to the address the descriptor specifies). +- A single ISO file load produces DMA entries with non-contiguous destination addresses. +- The 12-byte descriptor encodes the block's destination; the loader dispatches accordingly. + +This format is game-specific but the DMA capture approach handles it transparently: the captured +bytes are the post-scatter RAM contents, which is exactly the execution form the recompiler needs. diff --git a/docs/overlay-plan.md b/docs/overlay-plan.md new file mode 100644 index 00000000..80b1db66 --- /dev/null +++ b/docs/overlay-plan.md @@ -0,0 +1,201 @@ +# Overlay System — Implementation Plan + +**Written:** 2026-05-30 +**Status:** Active + +--- + +## Goal + +Make PSX overlay code transparent to the user. Shipped builds run pre-compiled +overlays at native speed (Layer B). Unknown overlays fall back to the interpreter +once, compile in the background, and run as native code from the second encounter +onward (Layer A). Users contribute new coverage automatically by playing. + +--- + +## Data structures + +### Capture set + +A hash map keyed by `load_addr`. Write-once: a given load address is inserted +exactly once regardless of how many times that overlay loads during a session. + +```c +typedef enum { + OVERLAY_STATE_QUEUED = 0, /* in compilation queue, not yet started */ + OVERLAY_STATE_COMPILING = 1, /* background thread is working on it */ + OVERLAY_STATE_COMPILED = 2, /* DLL written to cache, dispatch patched */ +} OverlayState; + +typedef struct OverlayEntry { + uint32_t load_addr; + uint32_t size; + uint8_t *bytes; /* heap copy of the DMA'd bytes */ + OverlayState state; /* atomic, guards queue dedup */ + struct OverlayEntry *next; /* intrusive hash chain */ +} OverlayEntry; +``` + +### Compilation queue + +A simple FIFO of `OverlayEntry *` pointers into the capture set. Does not own +data. Background thread drains it. Never contains duplicates — the state machine +on the capture set entry prevents duplicate pushes. + +--- + +## State machine (per entry) + +``` +DMA completion (first time, load_addr not in set) + → alloc entry, copy bytes + → state = QUEUED + → push pointer onto queue + +DMA completion (load_addr already in set) + → state == QUEUED → no-op (already waiting to compile) + → state == COMPILING → no-op (being compiled now, interp handles it) + → state == COMPILED → no-op (dispatch table has it, interp won't fire) + +Background thread (dequeues entry) + → state = COMPILING + → emit C for bytes at load_addr + → compile to DLL → write cache//.dll + → LoadLibrary + register functions in live dispatch table + → state = COMPILED +``` + +The `NEW→QUEUED` transition and the queue push are done under a single mutex +lock so a concurrent DMA completion on a second thread cannot push the same +entry twice. + +--- + +## Implementation phases + +### B-1 — Capture + JSON dump *(prerequisite for everything)* + +**Files:** `runtime/src/overlay_capture.c` + `runtime/include/overlay_capture.h` + +- `overlay_capture_init(const char *game_id)` — called at game handoff +- `overlay_capture_on_dma(uint32_t load_addr, uint32_t size, const uint8_t *bytes)` — called from `dma.c` `execute_ch3_cdrom` when `load_start < 0x1C0000` +- `overlay_capture_write_json(const char *path)` — writes `overlay_captures.json` next to the executable; called from `main.cpp` at clean exit +- JSON schema per entry: + ```json + { "load_addr": "0xA3000", "size": 208896, "bytes_b64": "...", "seeds": [] } + ``` + (`seeds` is empty at capture time; filled in by the ingestion script from per_pc data) + +**Hook in `dma.c`:** existing `execute_ch3_cdrom` already has `load_start` and +`total_words`. Add one call after the existing cd_dma_log push: +```c +if (!step && load_start < 0x1C0000u) + overlay_capture_on_dma(load_start, total_words * 4u, ram + load_start); +``` + +**Hook in `main.cpp`:** at clean shutdown (SDL quit, window close) call +`overlay_capture_write_json`. Retail and dev builds both do this. + +--- + +### B-2 — psxrecomp-game overlay compilation + +**Files:** `recompiler/src/overlay_compiler.cpp`, extended `game.toml` schema + +`game.toml` gains an `[[overlays]]` array: +```toml +[[overlays]] +load_addr = "0xA3000" +capture = "captures/A3000.bin" +seeds = ["0xA3140", "0xA52C0"] +``` + +`psxrecomp-game` reads each entry, disassembles the MIPS bytes at `load_addr`, +traces from seeds, emits C functions, and appends to the game's dispatch table. +These functions are linked into the shipped binary at zero runtime cost. + +--- + +### B-3 — Ingestion script + +**File:** `tools/ingest_overlays.py` + +``` +python3 tools/ingest_overlays.py \ + --captures overlay_captures.json \ + --per-pc logs/SCUS-94236/per_pc.json \ + --out-dir captures/ \ + --toml game.toml +``` + +- Reads `overlay_captures.json` (from user submission or own playthrough) +- Reads `per_pc.json` for executed addresses within each overlay range +- Writes `captures/.bin` +- Appends new `[[overlays]]` blocks to `game.toml` (skips already-present entries) + +After running, `psxrecomp-game --config game.toml` re-emits the dispatch table +with the new overlays compiled in. + +--- + +### A-1 — Cache DLL check on DMA completion + +**Files:** `runtime/src/overlay_capture.c` (extended) + +On DMA completion, after the capture-set insert: +- Compute `hash = crc32(bytes, size)` +- Check `cache//.dll` exists +- If yes: `LoadLibrary` → enumerate exported functions → register each in the + live dispatch table for `[load_addr, load_addr + size)` → set state COMPILED +- If no: proceed to queue for background compilation (A-2) + +--- + +### A-2 — Background compiler thread + +**Files:** `runtime/src/overlay_compiler_thread.c` + +A single long-lived thread started at game handoff: + +``` +loop: + wait on queue semaphore + dequeue OverlayEntry* + set state = COMPILING + emit_c_for_overlay(entry) // calls into recompiler lib + compile_c_to_dll(entry) // invokes GCC/Clang subprocess + LoadLibrary + register dispatch + set state = COMPILED +``` + +The recompiler is embedded in the runtime as a static library for dev builds. +For retail builds, the compiler subprocess path uses the system C compiler if +present, or skips to "write JSON only" mode if not (Layer A gracefully degrades +to JSON-only on systems without a toolchain). + +--- + +## Build order + +1. **B-1** — capture set + JSON dump (unblocks everything) +2. **B-2** — `psxrecomp-game` overlay compilation (unblocks B-3) +3. **B-3** — ingestion script (completes the contribution loop) +4. **A-1** — cache DLL check (fast path for returning users) +5. **A-2** — background compiler thread (self-healing on first run) + +B-1 through B-3 are useful independently: they close the developer feedback +loop and give users a way to contribute without any runtime compilation. +A-1 and A-2 are the quality-of-life layer. + +--- + +## What already exists + +| Component | File | Status | +|---|---|---| +| Dirty-RAM interpreter (last-resort fallback) | `runtime/src/dirty_ram_interp.c` | Done | +| CD DMA ring log (`cd_read_log` TCP cmd) | `runtime/src/dma.c` | Done (filter + 65536 ring, 2026-05-30) | +| Dispatch table infrastructure | `runtime/src/full_function_emitter.cpp` | Done | +| Recompiler core (MIPS→C) | `recompiler/src/` | Done | +| `game.toml` config loader | `recompiler/src/config_loader.cpp` | Done | diff --git a/recompiler/include/code_generator.h b/recompiler/include/code_generator.h index 184dc3e7..78cf0836 100644 --- a/recompiler/include/code_generator.h +++ b/recompiler/include/code_generator.h @@ -34,6 +34,7 @@ struct CodeGenConfig { bool emit_line_numbers; // Include original address as comments bool optimize_zero_reg; // Optimize away $zero assignments bool use_switch_for_blocks; // Use switch instead of goto labels + bool split_mid_function_targets; // Split branch targets into funcs for legacy main-EXE analysis std::string indent; // Indentation string (default: " ") CodeGenConfig() @@ -41,6 +42,7 @@ struct CodeGenConfig { , emit_line_numbers(true) , optimize_zero_reg(true) , use_switch_for_blocks(false) + , split_mid_function_targets(true) , indent(" ") {} }; diff --git a/recompiler/include/function_analysis.h b/recompiler/include/function_analysis.h index c6f7fd49..1b7b4469 100644 --- a/recompiler/include/function_analysis.h +++ b/recompiler/include/function_analysis.h @@ -35,6 +35,11 @@ public: // Scan entire executable for function boundaries FunctionAnalysisResult analyze(); + // Analyze only explicit entry points and direct JAL targets reachable from + // them. This is for runtime-loaded overlays where scanning the whole blob + // treats data and basic-block targets as standalone functions. + FunctionAnalysisResult analyze_exact_entries(const std::vector& entries); + // Add a forced entry point address that is treated as a function start // even if it has no standard ADDIU $sp prologue. The function will be // included in the analysis result with has_prologue = false. diff --git a/recompiler/src/code_generator.cpp b/recompiler/src/code_generator.cpp index 96edab42..2f212505 100644 --- a/recompiler/src/code_generator.cpp +++ b/recompiler/src/code_generator.cpp @@ -1573,7 +1573,11 @@ std::string CodeGenerator::generate_file( // hit `psx_unknown_dispatch`. Surfaced by Phase B2 audit. The 16-pass // cap is a safety limit; in practice each pass strictly reduces the // remaining target set so convergence is fast. - { + // + // Overlay exact mode disables this: overlay branch targets are basic-block + // labels, not callable function entries, so splitting them into standalone + // functions produces broken dispatch — the mid-function-seed failure mode. + if (config_.split_mid_function_targets) { std::set cfgs_to_scan; // Which CFGs to scan (empty = all) uint32_t exe_start = exe_.header.load_address; uint32_t exe_end = exe_.end_address(); diff --git a/recompiler/src/config_loader.cpp b/recompiler/src/config_loader.cpp index 1c8b1288..a0e580d0 100644 --- a/recompiler/src/config_loader.cpp +++ b/recompiler/src/config_loader.cpp @@ -61,6 +61,9 @@ static RuntimeConfig parse_runtime_block(const toml::value& cfg, const fs::path& if (runtime.contains("fast_boot")) { rt.fast_boot = toml::find(runtime, "fast_boot"); } + if (runtime.contains("overlay_cache")) { + rt.overlay_cache = toml::find(runtime, "overlay_cache"); + } return rt; } diff --git a/recompiler/src/config_loader.h b/recompiler/src/config_loader.h index ec534b24..3dff63f6 100644 --- a/recompiler/src/config_loader.h +++ b/recompiler/src/config_loader.h @@ -47,6 +47,12 @@ struct RuntimeConfig { // subsequent launches, skipping BIOS execution entirely. Default off; // enable per-game in [runtime]. Snapshot is keyed on BIOS SHA256 + entry_pc. bool fast_boot = false; + + // overlay_cache: enable the overlay DLL cache + capture (Layer A). Off by + // default. When true the runtime scans cache// for precompiled + // overlay DLLs (loaded ahead of the dirty-RAM interpreter) and records + // overlay bytes to overlay_captures.json for offline compilation. + bool overlay_cache = false; }; // One entry from [[recompiler.bios_vectors]]. diff --git a/recompiler/src/function_analysis.cpp b/recompiler/src/function_analysis.cpp index 838c4022..859e2591 100644 --- a/recompiler/src/function_analysis.cpp +++ b/recompiler/src/function_analysis.cpp @@ -1,6 +1,7 @@ #include "function_analysis.h" #include #include +#include #include #include @@ -217,6 +218,381 @@ bool FunctionAnalyzer::is_likely_data_section(uint32_t start_addr, uint32_t end_ return false; } +namespace { + +enum class ExactCfKind { + Normal, + Branch, + Jump, + Jal, + JrRa, + JrOther, + Jalr, +}; + +struct ExactCf { + ExactCfKind kind = ExactCfKind::Normal; + uint32_t target = 0; +}; + +struct ExactWalkResult { + std::set visited; + std::set direct_jal_targets; + std::set jump_table_targets; + uint32_t jr_ra_count = 0; +}; + +static bool exact_is_jr_ra_word(uint32_t instr) { + return instr == 0x03E00008u; +} + +static bool exact_is_addiu_sp_neg(uint32_t instr) { + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + int16_t imm = static_cast(instr & 0xFFFFu); + return opcode == 0x09u && rs == 29u && rt == 29u && imm < 0; +} + +static bool exact_is_valid_mips_word(uint32_t instr) { + if (instr == 0xFFFFFFFFu || instr == 0xFFFFFFFDu) return false; + + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t funct = instr & 0x3Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + + if (opcode == 0x00u) { + switch (funct) { + case 0x00u: case 0x02u: case 0x03u: case 0x04u: + case 0x06u: case 0x07u: case 0x08u: case 0x09u: + case 0x0Cu: case 0x0Du: + case 0x10u: case 0x11u: case 0x12u: case 0x13u: + case 0x18u: case 0x19u: case 0x1Au: case 0x1Bu: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: case 0x27u: + case 0x2Au: case 0x2Bu: + return true; + default: + return false; + } + } + if (opcode == 0x01u) { + return rt == 0x00u || rt == 0x01u || rt == 0x10u || rt == 0x11u; + } + + switch (opcode) { + case 0x02u: case 0x03u: case 0x04u: case 0x05u: + case 0x06u: case 0x07u: + case 0x08u: case 0x09u: case 0x0Au: case 0x0Bu: + case 0x0Cu: case 0x0Du: case 0x0Eu: case 0x0Fu: + case 0x10u: case 0x12u: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: + case 0x28u: case 0x29u: case 0x2Au: case 0x2Bu: + case 0x2Eu: + case 0x30u: case 0x32u: case 0x38u: case 0x3Au: + return true; + default: + return false; + } +} + +static uint32_t exact_branch_target(uint32_t pc, uint32_t instr) { + int16_t imm = static_cast(instr & 0xFFFFu); + return pc + 4u + (static_cast(imm) << 2); +} + +static uint32_t exact_jump_target(uint32_t pc, uint32_t instr) { + return ((pc + 4u) & 0xF0000000u) | ((instr & 0x03FFFFFFu) << 2); +} + +static ExactCf exact_classify_cf(uint32_t pc, uint32_t instr) { + ExactCf cf; + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t funct = instr & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + + if (opcode == 0x00u && funct == 0x08u) { + cf.kind = (rs == 31u) ? ExactCfKind::JrRa : ExactCfKind::JrOther; + return cf; + } + if (opcode == 0x00u && funct == 0x09u) { + cf.kind = ExactCfKind::Jalr; + return cf; + } + if (opcode == 0x02u) { + cf.kind = ExactCfKind::Jump; + cf.target = exact_jump_target(pc, instr); + return cf; + } + if (opcode == 0x03u) { + cf.kind = ExactCfKind::Jal; + cf.target = exact_jump_target(pc, instr); + return cf; + } + if (opcode == 0x01u || (opcode >= 0x04u && opcode <= 0x07u) || + (opcode >= 0x14u && opcode <= 0x17u)) { + cf.kind = ExactCfKind::Branch; + cf.target = exact_branch_target(pc, instr); + return cf; + } + return cf; +} + +} // namespace + +FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(const std::vector& entries) { + FunctionAnalysisResult result; + result.total_instructions = 0; + result.jr_ra_count = 0; + result.prologue_count = 0; + result.call_discovered_count = 0; + result.state_continuation_count = 0; + + fmt::print("\n=== Exact Overlay Function Analysis ===\n\n"); + + auto in_exe = [&](uint32_t addr) { + return addr >= exe_.header.load_address && addr < exe_.end_address() && (addr & 3u) == 0; + }; + + auto callable_direct_jal_target = [&](uint32_t addr) { + auto word_opt = exe_.read_word(addr); + if (!word_opt.has_value() || !exact_is_valid_mips_word(*word_opt)) return false; + if (addr == exe_.header.load_address) return true; + + if (addr >= exe_.header.load_address + 8u) { + auto prev8_opt = exe_.read_word(addr - 8u); + if (prev8_opt.has_value() && exact_is_jr_ra_word(*prev8_opt)) return true; + } + + if (exact_is_addiu_sp_neg(*word_opt)) { + if (addr < exe_.header.load_address + 4u) return true; + auto prev_opt = exe_.read_word(addr - 4u); + if (!prev_opt.has_value()) return true; + return exact_classify_cf(addr - 4u, *prev_opt).kind == ExactCfKind::Normal; + } + + return false; + }; + + auto find_jump_table_targets = [&](uint32_t entry, uint32_t hard_cap, + uint32_t jr_pc, uint32_t jr_rs) { + std::vector targets; + uint32_t lw_base = 0xFFu; + int32_t lw_offset = 0; + uint32_t addu_cand[2] = {0xFFu, 0xFFu}; + uint32_t lui_val = 0; + int16_t addiu_val[2] = {0, 0}; + bool found_addiu[2] = {false, false}; + bool found_lui = false; + uint32_t table_count = 0; + + for (int back = 1; back <= 40; back++) { + uint32_t scan_addr = jr_pc - static_cast(back * 4); + if (scan_addr < entry) break; + auto scan_opt = exe_.read_word(scan_addr); + if (!scan_opt.has_value()) break; + + uint32_t instr = *scan_opt; + uint32_t op = (instr >> 26) & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + uint32_t rd = (instr >> 11) & 0x1Fu; + uint32_t fn = instr & 0x3Fu; + + if (op == 0x23u && rt == jr_rs && lw_base == 0xFFu) { + lw_base = rs; + lw_offset = static_cast(static_cast(instr & 0xFFFFu)); + continue; + } + if (op == 0x00u && fn == 0x21u && rd == lw_base && lw_base != 0xFFu && + addu_cand[0] == 0xFFu) { + addu_cand[0] = rs; + addu_cand[1] = rt; + continue; + } + if (op == 0x09u && addu_cand[0] != 0xFFu) { + for (int c = 0; c < 2; c++) { + if (!found_addiu[c] && addu_cand[c] != 0xFFu && + rs == addu_cand[c] && rt == addu_cand[c]) { + addiu_val[c] = static_cast(instr & 0xFFFFu); + found_addiu[c] = true; + break; + } + } + continue; + } + if (op == 0x0Fu && addu_cand[0] != 0xFFu && !found_lui) { + for (int c = 0; c < 2; c++) { + if (addu_cand[c] != 0xFFu && rt == addu_cand[c]) { + lui_val = (instr & 0xFFFFu) << 16; + if (!found_addiu[c]) addiu_val[c] = 0; + addiu_val[0] = addiu_val[c]; + found_addiu[0] = found_addiu[c]; + found_lui = true; + break; + } + } + continue; + } + if (op == 0x0Bu && table_count == 0) { + table_count = instr & 0xFFFFu; + continue; + } + if (found_lui && table_count != 0) break; + } + + if (!found_lui || table_count == 0 || table_count >= 512) return targets; + + uint32_t table_base = lui_val + + (found_addiu[0] ? static_cast(static_cast(addiu_val[0])) : 0u) + + static_cast(lw_offset); + for (uint32_t i = 0; i < table_count; i++) { + auto entry_opt = exe_.read_word(table_base + i * 4u); + if (!entry_opt.has_value()) continue; + uint32_t target = *entry_opt; + if (target >= entry && target < hard_cap && in_exe(target)) { + targets.push_back(target); + } + } + return targets; + }; + + auto walk = [&](uint32_t entry, uint32_t hard_cap) { + ExactWalkResult wr; + std::queue work; + work.push(entry); + + auto in_function = [&](uint32_t addr) { + return in_exe(addr) && addr >= entry && addr < hard_cap; + }; + + while (!work.empty()) { + uint32_t pc = work.front(); + work.pop(); + + if (wr.visited.count(pc) || !in_function(pc)) continue; + auto word_opt = exe_.read_word(pc); + if (!word_opt.has_value()) continue; + + uint32_t instr = *word_opt; + wr.visited.insert(pc); + ExactCf cf = exact_classify_cf(pc, instr); + uint32_t delay = pc + 4u; + + switch (cf.kind) { + case ExactCfKind::Normal: + if (in_function(pc + 4u)) work.push(pc + 4u); + break; + case ExactCfKind::Branch: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + if (in_function(cf.target)) work.push(cf.target); + break; + case ExactCfKind::Jump: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(cf.target)) work.push(cf.target); + break; + case ExactCfKind::Jal: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + if (in_exe(cf.target) && callable_direct_jal_target(cf.target)) { + wr.direct_jal_targets.insert(cf.target); + } + break; + case ExactCfKind::Jalr: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + break; + case ExactCfKind::JrRa: + if (in_function(delay)) wr.visited.insert(delay); + wr.jr_ra_count++; + break; + case ExactCfKind::JrOther: + if (in_function(delay)) wr.visited.insert(delay); + { + uint32_t jr_rs = (instr >> 21) & 0x1Fu; + for (uint32_t target : find_jump_table_targets(entry, hard_cap, pc, jr_rs)) { + wr.jump_table_targets.insert(target); + if (in_function(target)) work.push(target); + } + } + break; + } + } + return wr; + }; + + std::set known_entries; + std::queue pending; + for (uint32_t entry : entries) { + if (!in_exe(entry)) continue; + if (known_entries.insert(entry).second) pending.push(entry); + } + + size_t explicit_count = known_entries.size(); + fmt::print("Explicit entries: {}\n", explicit_count); + + std::set processed; + while (!pending.empty()) { + uint32_t entry = pending.front(); + pending.pop(); + if (processed.count(entry)) continue; + processed.insert(entry); + + auto next_it = known_entries.upper_bound(entry); + uint32_t hard_cap = (next_it != known_entries.end()) ? *next_it : exe_.end_address(); + ExactWalkResult wr = walk(entry, hard_cap); + + for (uint32_t target : wr.direct_jal_targets) { + if (known_entries.insert(target).second) pending.push(target); + } + } + + result.call_discovered_count = static_cast(known_entries.size() - explicit_count); + fmt::print("Direct-JAL entries: {}\n", result.call_discovered_count); + fmt::print("Total exact entries: {}\n\n", known_entries.size()); + + std::vector starts_vec(known_entries.begin(), known_entries.end()); + std::sort(starts_vec.begin(), starts_vec.end()); + + for (size_t i = 0; i < starts_vec.size(); i++) { + uint32_t entry = starts_vec[i]; + uint32_t hard_cap = (i + 1 < starts_vec.size()) ? starts_vec[i + 1] : exe_.end_address(); + ExactWalkResult wr = walk(entry, hard_cap); + if (wr.visited.empty()) continue; + + Function func; + func.start_addr = entry; + func.end_addr = *wr.visited.rbegin() + 4u; + if (func.end_addr > hard_cap) func.end_addr = hard_cap; + if (func.end_addr <= func.start_addr) func.end_addr = func.start_addr + 4u; + func.size = func.end_addr - func.start_addr; + func.name = fmt::format("func_{:08X}", func.start_addr); + + auto first_instr = exe_.read_word(func.start_addr); + if (first_instr.has_value()) { + int32_t stack_size; + func.has_prologue = is_prologue(*first_instr, stack_size); + func.stack_frame_size = func.has_prologue ? stack_size : 0; + if (func.has_prologue) result.prologue_count++; + } else { + func.has_prologue = false; + func.stack_frame_size = 0; + } + + func.has_epilogue = wr.jr_ra_count > 0; + func.is_data_section = false; + + result.jr_ra_count += static_cast(wr.jr_ra_count); + result.total_instructions += static_cast(wr.visited.size()); + result.functions.push_back(func); + } + + return result; +} + + FunctionAnalysisResult FunctionAnalyzer::analyze() { FunctionAnalysisResult result; result.total_instructions = 0; diff --git a/recompiler/src/main_psx.cpp b/recompiler/src/main_psx.cpp index dde3ae91..3326cf0e 100644 --- a/recompiler/src/main_psx.cpp +++ b/recompiler/src/main_psx.cpp @@ -40,6 +40,7 @@ int main(int argc, char** argv) { std::string extra_funcs_storage; // lifetime anchor for the .c_str() below const char* extra_funcs_path = nullptr; bool inspect_mode = false; + bool overlay_exact_mode = false; std::filesystem::path out_dir = "generated"; if (!config_path.empty()) { @@ -72,6 +73,8 @@ int main(int argc, char** argv) { * with psxrecomp-bios and project scripts. */ } else if (arg == "--inspect") { inspect_mode = true; + } else if (arg == "--overlay-exact") { + overlay_exact_mode = true; } } } @@ -197,7 +200,11 @@ int main(int argc, char** argv) { fmt::print("Performing function analysis...\n"); PSXRecomp::FunctionAnalyzer analyzer(*exe); - analyzer.add_forced_entry(exe->header.initial_pc); + std::vector exact_entries; + exact_entries.push_back(exe->header.initial_pc); + if (!overlay_exact_mode) { + analyzer.add_forced_entry(exe->header.initial_pc); + } // Forced entry points: functions called from the dispatch table but not // automatically detected by the heuristic-based function analysis. @@ -213,7 +220,10 @@ int main(int argc, char** argv) { if (line.empty() || line[0] == '#') continue; uint32_t addr = (uint32_t)std::strtoul(line.c_str(), nullptr, 16); if (addr >= 0x80010000u && addr < 0x80200000u) { - analyzer.add_forced_entry(addr); + exact_entries.push_back(addr); + if (!overlay_exact_mode) { + analyzer.add_forced_entry(addr); + } extra_count++; } } @@ -223,7 +233,9 @@ int main(int argc, char** argv) { } } - auto analysis_result = analyzer.analyze(); + auto analysis_result = overlay_exact_mode + ? analyzer.analyze_exact_entries(exact_entries) + : analyzer.analyze(); // Print summary statistics fmt::print("\n=== Function Analysis Summary ===\n\n"); @@ -344,6 +356,7 @@ int main(int argc, char** argv) { PSXRecomp::CodeGenConfig codegen_config; codegen_config.emit_comments = true; codegen_config.emit_line_numbers = true; + codegen_config.split_mid_function_targets = !overlay_exact_mode; // Load per-game annotations: annotations/_annotations.csv // Silently skipped if the file doesn't exist. diff --git a/runtime/include/cdrom.h b/runtime/include/cdrom.h index f35aa955..d773a00b 100644 --- a/runtime/include/cdrom.h +++ b/runtime/include/cdrom.h @@ -15,6 +15,9 @@ void cdrom_set_speed(int divisor); void cdrom_set_game_speed(int divisor); /* Called by fntrace on first game-range dispatch; switches to game speed. */ void cdrom_notify_game_started(void); +/* LBA of the most recent SetLoc command (-1 if none yet). Used by the DMA + * ring buffer to correlate each sector transfer with its disc position. */ +int cdrom_get_setloc_lba(void); /* MMIO read/write (0x1F801800-0x1F801803) */ uint32_t cdrom_read(uint32_t addr); diff --git a/runtime/include/dirty_ram_interp.h b/runtime/include/dirty_ram_interp.h index 95664fa6..271f99f0 100644 --- a/runtime/include/dirty_ram_interp.h +++ b/runtime/include/dirty_ram_interp.h @@ -56,13 +56,16 @@ extern const char *g_dirty_ram_last_unsupported_reason; * stubs actually fire — a single noisy spurious-dispatch site can mask * a legitimate handler that never runs. This open-addressed table keys * on the entry PC of each interpreted block. */ -#define DIRTY_RAM_PC_TABLE_SIZE 64 +#define DIRTY_RAM_PC_TABLE_SIZE 65536 typedef struct { uint32_t pc; /* entry PC, 0 = empty slot */ uint64_t hits; /* number of times dispatched here */ uint64_t insns; /* total instructions executed across hits */ } DirtyRamPcEntry; extern DirtyRamPcEntry g_dirty_ram_pc_table[DIRTY_RAM_PC_TABLE_SIZE]; +/* Companion table: every PC the interpreter actually executes (not just block + * entries). overlay_capture uses both to report execution-verified seeds. */ +extern DirtyRamPcEntry g_dirty_ram_exec_pc_table[DIRTY_RAM_PC_TABLE_SIZE]; /* Block-entry ring buffer. Records every dispatch into dirty RAM with the * caller's RA at entry, plus argument context — answers diff --git a/runtime/include/overlay_api.h b/runtime/include/overlay_api.h new file mode 100644 index 00000000..1e0c8073 --- /dev/null +++ b/runtime/include/overlay_api.h @@ -0,0 +1,44 @@ +#ifndef OVERLAY_API_H +#define OVERLAY_API_H + +/* Shared ABI between psx-runtime.exe and overlay DLLs. + * + * After LoadLibrary, the runtime calls overlay_init() with this struct. + * The DLL stores the pointers and routes all cross-module calls through them. + */ + +#include "cpu_state.h" +#include + +typedef struct { + /* Core dispatch: routes call_by_address() and out-of-overlay jal */ + void (*dispatch_call)(CPUState *cpu, uint32_t addr, uint32_t ra); + /* Interrupt check: called after every function return in overlay */ + void (*check_interrupts)(CPUState *cpu); + /* GTE coprocessor 2 execution */ + void (*gte_execute)(CPUState *cpu, uint32_t cmd); + /* MIPS syscall (break/syscall instructions) */ + void (*psx_syscall)(CPUState *cpu, uint32_t code); + /* Unresolved dispatch target */ + void (*psx_unknown_dispatch)(CPUState *cpu, uint32_t addr, uint32_t phys); + /* Debug instrumentation: called at every function entry (may be NULL) */ + void (*log_call_entry)(uint32_t func_addr); +} OverlayCallbacks; + +#ifdef __cplusplus +extern "C" { +#endif + +/* Exported by every overlay DLL. Call once after LoadLibrary. */ +#ifdef _WIN32 +__declspec(dllexport) +#else +__attribute__((visibility("default"))) +#endif +void overlay_init(const OverlayCallbacks *cbs); + +#ifdef __cplusplus +} +#endif + +#endif /* OVERLAY_API_H */ diff --git a/runtime/include/overlay_capture.h b/runtime/include/overlay_capture.h new file mode 100644 index 00000000..44391e44 --- /dev/null +++ b/runtime/include/overlay_capture.h @@ -0,0 +1,62 @@ +#ifndef OVERLAY_CAPTURE_H +#define OVERLAY_CAPTURE_H + +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/* overlay_capture — B-1 implementation of the overlay capture set. + * + * On every CD DMA completion into game-code RAM (dest < 0x1C0000), the runtime + * calls overlay_capture_on_dma(). Each unique load_addr is stored exactly once + * in a write-once set. At clean process exit, overlay_capture_write_json() + * writes overlay_captures.json next to the executable for user contribution. + * + * The set never overwrites: a second DMA to the same load_addr is a no-op. + * This is correct because PSX overlays always load the same bytes to the same + * address; recording once is sufficient. + */ + +/* Call once at startup (before the run loop) with the exe directory. + * Sets the output path for overlay_captures.json. Capture is not yet + * active — it activates automatically on the first post-game DMA transfer. */ +void overlay_capture_set_out_dir(const char *out_dir); + +/* Legacy init alias kept for compatibility; same as set_out_dir. */ +void overlay_capture_init(const char *out_dir); + +/* Enable/disable overlay capture. Off by default; main() enables it only when + * the overlay cache is turned on in [runtime] config. When disabled, + * overlay_capture_on_dma() and overlay_capture_write_json() are no-ops. */ +void overlay_capture_set_enabled(int enabled); + +/* Call from dma.c execute_ch3_cdrom() after every forward CH3 transfer with + * load_start < 0x1C0000 and fntrace_is_game_started(). + * load_addr is the physical RAM destination, size is byte count, + * bytes points to the RAM buffer at that address (already written by DMA). */ +void overlay_capture_on_dma(uint32_t load_addr, uint32_t size, + const uint8_t *bytes); + +/* Write overlay_captures.json to the directory supplied at init time. + * Safe to call even if no overlays were captured (writes nothing). + * Called from shutdown_runtime() in main.cpp. */ +void overlay_capture_write_json(void); + +/* Returns number of unique overlays captured so far. */ +int overlay_capture_count(void); + +/* Compute CRC32 of the DMA-time bytes for the region [region_start, + * region_start+region_size). Zero-filled for gaps between DMA blocks, + * exactly as overlay_captures.json bytes_b64 is assembled. Use this + * in overlay_loader to build a consistent DLL filename without reading + * live RAM (which has scatter-load gap contamination). */ +uint32_t overlay_capture_get_region_crc(uint32_t region_start, + uint32_t region_size); + +#ifdef __cplusplus +} +#endif + +#endif /* OVERLAY_CAPTURE_H */ diff --git a/runtime/include/overlay_loader.h b/runtime/include/overlay_loader.h new file mode 100644 index 00000000..2f9734d8 --- /dev/null +++ b/runtime/include/overlay_loader.h @@ -0,0 +1,58 @@ +#ifndef OVERLAY_LOADER_H +#define OVERLAY_LOADER_H + +/* overlay_loader — A-1 runtime overlay DLL cache and dynamic dispatch. + * + * On CD DMA completion (overlay_capture_on_dma), the loader checks whether + * a cached DLL exists for the overlay bytes. If it does, it LoadLibrary's + * the DLL, calls overlay_init() to wire callbacks, enumerates func_XXXXXXXX + * exports, and registers each in the dynamic dispatch table. + * + * dirty_ram_dispatch calls overlay_loader_dispatch() before the interpreter, + * so compiled overlay functions get priority over interpretation. + */ + +#include "cpu_state.h" +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/* Called at game handoff to set the cache root directory and game ID. + * cache_dir: absolute path to the cache root (e.g. "build-dev/cache") + * game_id: product code (e.g. "SCUS-94236") */ +void overlay_loader_init(const char *cache_dir, const char *game_id); + +/* Called from overlay_capture_on_dma after the capture-set insert. + * Computes CRC32 of bytes, checks cache, loads DLL if present. + * load_addr: physical RAM address, size/bytes: the transferred data. */ +void overlay_loader_check_cache(uint32_t load_addr, uint32_t size, + const uint8_t *bytes); + +/* Called from dirty_ram_dispatch before the interpreter. + * Returns 1 and calls the compiled function if addr is registered, + * 0 if not found (fall through to interpreter). */ +int overlay_loader_dispatch(CPUState *cpu, uint32_t addr); + +/* Returns number of functions currently registered in the dynamic table. */ +int overlay_loader_registered_count(void); + +/* Returns full loader state for TCP diagnostics. */ +void overlay_loader_get_status(int *active, int *registered, + int *regions_checked, + char *cache_dir_out, int cache_dir_len, + char *game_id_out, int game_id_len, + uint32_t *checked_out, int checked_max, + int *checked_written, + uint32_t *last_crc_out, int *last_file_found_out); + +/* Most recent loader event string (DLL load success/failure). Surfaced via + * the overlay_loader_status TCP command — no stderr logging (Rule 3). */ +const char *overlay_loader_last_msg(void); + +#ifdef __cplusplus +} +#endif + +#endif /* OVERLAY_LOADER_H */ diff --git a/runtime/runtime.cmake b/runtime/runtime.cmake index 32751371..ba12e4d8 100644 --- a/runtime/runtime.cmake +++ b/runtime/runtime.cmake @@ -92,6 +92,8 @@ set(PSXRECOMP_RUNTIME_SOURCES ${PSXRECOMP_ROOT}/runtime/src/starvation_ring.c ${PSXRECOMP_ROOT}/runtime/src/card_read_summary.c ${PSXRECOMP_ROOT}/runtime/src/card_data_writes.c + ${PSXRECOMP_ROOT}/runtime/src/overlay_capture.c + ${PSXRECOMP_ROOT}/runtime/src/overlay_loader.c ${PSXRECOMP_ROOT}/recompiler/src/config_loader.cpp ) @@ -112,6 +114,7 @@ function(psxrecomp_add_runtime_target target) set(oneValueArgs GAME_GENERATED_FULL_C GAME_GENERATED_DISPATCH_C + GAME_OVERLAY_STATIC_C DEBUG_PORT WINDOW_TITLE DEFAULT_BIOS_PATH @@ -147,6 +150,13 @@ function(psxrecomp_add_runtime_target target) list(APPEND generated_sources "${PSXRT_GAME_GENERATED_DISPATCH_C}") set(has_game_dispatch TRUE) endif() + # Layer B: statically-compiled overlay dispatch. Inert unless a game + # provides a generated overlays_static.c — no target sets this yet. + if(PSXRT_GAME_OVERLAY_STATIC_C AND EXISTS "${PSXRT_GAME_OVERLAY_STATIC_C}") + set_source_files_properties("${PSXRT_GAME_OVERLAY_STATIC_C}" PROPERTIES GENERATED TRUE) + list(APPEND generated_sources "${PSXRT_GAME_OVERLAY_STATIC_C}") + set(has_overlay_dispatch TRUE) + endif() if(PSXRT_ORACLE) set(mode_source ${PSXRECOMP_ROOT}/runtime/src/psx_interpreter.c) @@ -201,6 +211,9 @@ function(psxrecomp_add_runtime_target target) if(has_game_dispatch) target_compile_definitions(${target} PRIVATE PSX_HAS_GAME_DISPATCH=1) endif() + if(has_overlay_dispatch) + target_compile_definitions(${target} PRIVATE PSX_HAS_OVERLAY_DISPATCH=1) + endif() # PSX_DEBUG_TOOLS option declared at the top of runtime.cmake so it's # also visible to psx-beetle / non-runtime-helper targets. diff --git a/runtime/src/cdrom.c b/runtime/src/cdrom.c index 470a08c5..f0879d84 100644 --- a/runtime/src/cdrom.c +++ b/runtime/src/cdrom.c @@ -75,6 +75,7 @@ static uint64_t sector_history_seq; /* Seek target */ static uint8_t seek_min, seek_sec, seek_sect; +static int s_setloc_lba = -1; /* LBA captured at SetLoc time */ /* Read state */ static int reading; @@ -138,6 +139,8 @@ void cdrom_notify_game_started(void) { g_disc_speed_divisor = g_game_divisor; } +int cdrom_get_setloc_lba(void) { return s_setloc_lba; } + /* Minimum cycles between CD-ROM IRQs in fast modes. Must be enough for the * interrupt handler to save state, check the IRQ flag, process data, and * return. Too low → interrupt fires before the previous one is processed → @@ -659,6 +662,7 @@ static void exec_command(uint8_t cmd) { seek_min = bcd_to_bin(param_fifo[0]); seek_sec = bcd_to_bin(param_fifo[1]); seek_sect = bcd_to_bin(param_fifo[2]); + s_setloc_lba = msf_to_lba(seek_min, seek_sec, seek_sect); } response_push(stat_reg); set_irq(CDIRQ_ACK); diff --git a/runtime/src/debug_server.c b/runtime/src/debug_server.c index cfa5e716..0b6d3730 100644 --- a/runtime/src/debug_server.c +++ b/runtime/src/debug_server.c @@ -8,6 +8,7 @@ * so TCP.md and DEBUG.md are reusable across projects. */ #include "debug_server.h" +#include "overlay_loader.h" #include "cpu_state.h" #include "dma.h" #include "gpu.h" @@ -7214,6 +7215,88 @@ static void handle_fn_exit_dump(int id, const char *json) { free(buf); } +/* cd_read_log: dump the CD DMA transfer ring. Each entry records the + * setloc LBA, destination RAM address, and byte count for one forward + * channel-3 DMA transfer. Use with overlay_dump to map overlay regions + * back to their disc positions for the extract_overlays.py disc scanner. + * + * Parameters: "tail" (int, default 256) — how many recent entries to return. + */ +static void handle_cd_read_log(int id, const char *json) +{ + extern uint32_t cd_dma_log_get_total(void); + extern void cd_dma_log_get_entry(uint32_t idx, int *lba, + uint32_t *dest, uint32_t *size); + + int tail = json_get_int(json, "tail", 256); + uint32_t total = cd_dma_log_get_total(); + uint32_t cap = 65536; + uint32_t avail = total < cap ? total : cap; + if ((uint32_t)tail > avail) tail = (int)avail; + + uint32_t start_idx = total > (uint32_t)tail ? total - (uint32_t)tail : 0; + + send_fmt("{\"id\":%d,\"ok\":true,\"total\":%u,\"count\":%d,\"entries\":[", + id, total, tail); + int first = 1; + for (uint32_t i = start_idx; i < total; i++) { + int lba; uint32_t dest, size; + cd_dma_log_get_entry(i, &lba, &dest, &size); + if (lba < 0) continue; + send_fmt("%s{\"lba\":%d,\"dest\":\"0x%08X\",\"size\":%u}", + first ? "" : ",", lba, dest, size); + first = 0; + } + send_fmt("]}\n"); +} + +/* overlay_loader_status: report the dynamic overlay DLL loader state — + * whether active, how many functions are registered, which regions have + * been checked, and the most recent load event. Rule-3 inspection path for + * the loader (it does no stderr logging). */ +static void handle_overlay_loader_status(int id, const char *json) +{ + (void)json; + int active = 0, registered = 0, nchecked = 0, nwritten = 0; + int file_found = 0; + uint32_t last_crc = 0; + char cache_dir[512] = {0}, game_id[64] = {0}; + uint32_t checked[8] = {0}; + overlay_loader_get_status(&active, ®istered, &nchecked, + cache_dir, sizeof(cache_dir), + game_id, sizeof(game_id), + checked, 8, &nwritten, + &last_crc, &file_found); + /* Escape backslashes in cache_dir for JSON */ + char esc_dir[768] = {0}; + for (int si = 0, di = 0; cache_dir[si] && di < (int)sizeof(esc_dir)-2; si++) { + if (cache_dir[si] == '\\') esc_dir[di++] = '\\'; + esc_dir[di++] = cache_dir[si]; + } + const char *msg = overlay_loader_last_msg(); + char esc_msg[512] = {0}; + for (int si = 0, di = 0; msg[si] && di < (int)sizeof(esc_msg)-2; si++) { + if (msg[si] == '\\' || msg[si] == '"') esc_msg[di++] = '\\'; + esc_msg[di++] = msg[si]; + } + char buf[1536]; + int n = snprintf(buf, sizeof(buf), + "{\"id\":%d,\"ok\":true,\"active\":%d,\"registered\":%d," + "\"regions_checked\":%d,\"last_crc\":\"0x%08X\",\"file_found\":%d," + "\"cache_dir\":\"%s\",\"game_id\":\"%s\",\"last_msg\":\"%s\"", + id, active, registered, nchecked, last_crc, file_found, + esc_dir, game_id, esc_msg); + if (nwritten > 0) { + n += snprintf(buf + n, sizeof(buf) - n, ",\"checked\":["); + for (int i = 0; i < nwritten; i++) + n += snprintf(buf + n, sizeof(buf) - n, + "%s\"0x%08X\"", i ? "," : "", checked[i]); + n += snprintf(buf + n, sizeof(buf) - n, "]"); + } + snprintf(buf + n, sizeof(buf) - n, "}\n"); + send_fmt("%s", buf); +} + /* overlay_dump: extract RAM regions that dirty_ram has marked executable * above a threshold physical address. Writes .bin files to a * caller-supplied directory and returns a JSON manifest. @@ -7746,6 +7829,8 @@ static const CmdEntry s_commands[] = { { "fn_entry_tail", handle_fn_entry_tail }, { "fn_exit_dump", handle_fn_exit_dump }, { "overlay_dump", handle_overlay_dump }, + { "cd_read_log", handle_cd_read_log }, + { "overlay_loader_status", handle_overlay_loader_status }, { NULL, NULL } }; diff --git a/runtime/src/dirty_ram_interp.c b/runtime/src/dirty_ram_interp.c index 87ecdb76..90f732e5 100644 --- a/runtime/src/dirty_ram_interp.c +++ b/runtime/src/dirty_ram_interp.c @@ -52,6 +52,7 @@ uint32_t g_dirty_ram_last_unsupported_insn = 0; const char *g_dirty_ram_last_unsupported_reason = NULL; DirtyRamPcEntry g_dirty_ram_pc_table[DIRTY_RAM_PC_TABLE_SIZE] = {0}; +DirtyRamPcEntry g_dirty_ram_exec_pc_table[DIRTY_RAM_PC_TABLE_SIZE] = {0}; DirtyRamBlockLogEntry g_dirty_ram_block_log[DIRTY_RAM_BLOCK_LOG_CAP] = {0}; uint64_t g_dirty_ram_block_log_seq = 0; @@ -67,17 +68,29 @@ extern uint64_t s_frame_count; * the working set of install-stub PCs is tiny (handful), so this stays * O(1) in practice. Returns NULL if the table is full — caller treats * that as "stop tracking" rather than failing. */ -static DirtyRamPcEntry *pc_table_get_or_insert(uint32_t pc) { +static DirtyRamPcEntry *pc_table_get_or_insert_in(DirtyRamPcEntry *table, uint32_t pc) { uint32_t h = (pc * 2654435761u) & (DIRTY_RAM_PC_TABLE_SIZE - 1); for (uint32_t i = 0; i < DIRTY_RAM_PC_TABLE_SIZE; i++) { uint32_t idx = (h + i) & (DIRTY_RAM_PC_TABLE_SIZE - 1); - DirtyRamPcEntry *e = &g_dirty_ram_pc_table[idx]; + DirtyRamPcEntry *e = &table[idx]; if (e->pc == pc) return e; if (e->pc == 0) { e->pc = pc; return e; } } return NULL; } +static DirtyRamPcEntry *pc_table_get_or_insert(uint32_t pc) { + return pc_table_get_or_insert_in(g_dirty_ram_pc_table, pc); +} + +/* Record every PC the interpreter executes (not just block entries) so + * overlay_capture can report execution-verified seeds for the region. */ +static void exec_pc_table_record(uint32_t pc) { + DirtyRamPcEntry *e = pc_table_get_or_insert_in(g_dirty_ram_exec_pc_table, + pc & 0x1FFFFFFFu); + if (e) e->hits++; +} + /* From debug_server.c — keep our outer-frame attribution coherent. */ extern uint32_t g_debug_current_func_addr; extern uint32_t g_debug_last_store_pc; @@ -233,6 +246,54 @@ static int is_local_dirty_target(uint32_t target) { return phys >= 0x00098000u && dirty_ram_is_dirty(phys); } +/* A dirty-RAM target only deserves interpretation if its first word decodes + * as a plausible MIPS instruction. A scatter-loaded overlay leaves data + * bytes in dirty pages; jumping into those and interpreting them as code is + * how the original cache build produced black/blue screens. When the target + * word doesn't look like code, fall back to the normal dispatch path. */ +static int dirty_ram_word_looks_decodable(uint32_t insn) { + if (insn == 0xFFFFFFFFu || insn == 0xFFFFFFFDu) return 0; + + uint32_t op = op_field(insn); + uint32_t fn = funct_field(insn); + uint32_t rt = rt_field(insn); + + if (op == 0x00u) { + switch (fn) { + case 0x00u: case 0x02u: case 0x03u: case 0x04u: + case 0x06u: case 0x07u: case 0x08u: case 0x09u: + case 0x0Cu: case 0x0Du: + case 0x10u: case 0x11u: case 0x12u: case 0x13u: + case 0x18u: case 0x19u: case 0x1Au: case 0x1Bu: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: case 0x27u: + case 0x2Au: case 0x2Bu: + return 1; + default: + return 0; + } + } + if (op == 0x01u) { + return rt == 0x00u || rt == 0x01u || rt == 0x10u || rt == 0x11u; + } + + switch (op) { + case 0x02u: case 0x03u: case 0x04u: case 0x05u: + case 0x06u: case 0x07u: + case 0x08u: case 0x09u: case 0x0Au: case 0x0Bu: + case 0x0Cu: case 0x0Du: case 0x0Eu: case 0x0Fu: + case 0x10u: case 0x12u: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: + case 0x28u: case 0x29u: case 0x2Au: case 0x2Bu: + case 0x2Eu: + case 0x30u: case 0x32u: case 0x38u: case 0x3Au: + return 1; + default: + return 0; + } +} + static int dispatch_nonlocal_call(CPUState *cpu, uint32_t target, uint32_t return_pc, uint32_t *next_pc_out) { @@ -276,6 +337,7 @@ static void exec_delay_slot(CPUState *cpu, uint32_t pc) { } static int exec_one(CPUState *cpu, uint32_t pc, uint32_t *next_pc_out) { + exec_pc_table_record(pc); uint32_t phys = pc & 0x1FFFFFFFu; uint32_t insn = fetch_word(phys); uint32_t opc = op_field(insn); @@ -466,6 +528,9 @@ static int exec_one(CPUState *cpu, uint32_t pc, uint32_t *next_pc_out) { if (!is_local_dirty_target(target)) { return dispatch_nonlocal_call(cpu, target, return_pc, next_pc_out); } + if (!dirty_ram_word_looks_decodable(fetch_word(target & 0x1FFFFFFFu))) { + return dispatch_nonlocal_call(cpu, target, return_pc, next_pc_out); + } cpu->pc = target; return 1; } @@ -687,6 +752,23 @@ int dirty_ram_dispatch(CPUState* cpu, uint32_t addr) { if (psx_dispatch_game_compiled(cpu, addr)) return 1; #endif + /* B-2: statically-compiled overlay functions (generated/overlays_static.c). + * Inert unless a game provides an overlays_static.c at build time. */ +#ifdef PSX_HAS_OVERLAY_DISPATCH + { + extern int psx_overlay_dispatch(CPUState *cpu, uint32_t addr); + if (psx_overlay_dispatch(cpu, addr)) return 1; + } +#endif + + /* A-1: dynamically-loaded overlay DLL functions, checked before the + * interpreter. No-op (returns 0) until overlay_loader_init() runs, which + * only happens when the overlay cache is enabled in config. */ + { + extern int overlay_loader_dispatch(CPUState *cpu, uint32_t addr); + if (overlay_loader_dispatch(cpu, addr)) return 1; + } + if (!dirty_ram_is_dirty(phys)) return 0; /* Reset soft-fail state at block entry. */ diff --git a/runtime/src/dma.c b/runtime/src/dma.c index 3bb574ed..634487c3 100644 --- a/runtime/src/dma.c +++ b/runtime/src/dma.c @@ -17,6 +17,7 @@ #include "dirty_ram_interp.h" #include "gpu.h" #include "mdec.h" +#include "overlay_capture.h" #include "spu.h" #include #include @@ -49,11 +50,43 @@ typedef struct { uint32_t total_words; uint32_t remaining_words; uint32_t cycles_accum; + uint32_t start_addr; /* madr at transfer start (CD overlay capture) */ } DMAAsyncChannel; static DMAAsyncChannel mdec_async[2]; static DMAAsyncChannel cdrom_async; +/* ---- CD DMA transfer log ---- */ +/* Every forward CH3 DMA that lands below 0x1C0000 (game data region) records + * (setloc_lba, dest_addr, size). Transfers to 0x1C0000+ are FMV/streaming + * buffers and are excluded to keep the ring focused on overlay loads. + * Surfaced via the cd_read_log TCP command; pairs with overlay_dump to map + * overlay regions back to disc positions for extract_overlays.py. */ +#define CD_DMA_LOG_CAP 65536 +typedef struct { int lba; uint32_t dest; uint32_t size; } CdDmaEntry; +static CdDmaEntry cd_dma_log[CD_DMA_LOG_CAP]; +static uint32_t cd_dma_log_head = 0; +static uint32_t cd_dma_log_total = 0; + +static void cd_dma_log_push(int lba, uint32_t dest, uint32_t size) { + cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].lba = lba; + cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].dest = dest; + cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].size = size; + cd_dma_log_head = (cd_dma_log_head + 1) % CD_DMA_LOG_CAP; + cd_dma_log_total++; +} + +uint32_t cd_dma_log_get_total(void) { return cd_dma_log_total; } +void cd_dma_log_get_entry(uint32_t idx, int *lba, uint32_t *dest, uint32_t *size) { + uint32_t cap = CD_DMA_LOG_CAP; + uint32_t oldest = cd_dma_log_total > cap ? cd_dma_log_total - cap : 0; + if (idx < oldest || idx >= cd_dma_log_total) { *lba = -1; return; } + uint32_t slot = idx % cap; + *lba = cd_dma_log[slot].lba; + *dest = cd_dma_log[slot].dest; + *size = cd_dma_log[slot].size; +} + #define DMA_MDEC_IN_CYCLES_PER_WORD 1u #define DMA_MDEC_OUT_CYCLES_PER_WORD 14u #define DMA_GPU_CYCLES_PER_WORD 1u @@ -247,6 +280,7 @@ static void start_async_cdrom_transfer(void) { a->total_words = transfer_word_count(3); a->remaining_words = a->total_words; a->cycles_accum = 0; + a->start_addr = channels[3].madr & 0x1FFFFCu; if (a->total_words == 0) { cancel_async_transfer(3); @@ -255,6 +289,25 @@ static void start_async_cdrom_transfer(void) { } static void finish_async_cdrom_transfer(uint32_t final_addr) { + DMAAsyncChannel *a = &cdrom_async; + uint32_t step = (channels[3].chcr >> 1) & 1u; + uint32_t load_start = a->start_addr; + uint32_t size = a->total_words * 4u; + + /* Log and capture game-data transfers only. + * Transfers to 0x1C0000+ are FMV/streaming buffers; skip them. */ + if (!step && size > 0 && load_start < 0x1C0000u) { + int lba = cdrom_get_setloc_lba(); + if (lba >= 0) cd_dma_log_push(lba, load_start, size); + + /* B-1: capture overlay bytes into the write-once capture set. + * overlay_capture_on_dma auto-activates after game handoff and is a + * no-op unless the overlay cache is enabled in config. */ + extern uint8_t *memory_get_ram_ptr(void); + uint8_t *ram = memory_get_ram_ptr(); + overlay_capture_on_dma(load_start, size, ram + load_start); + } + channels[3].madr = final_addr; cancel_async_transfer(3); complete_transfer(3); diff --git a/runtime/src/main.cpp b/runtime/src/main.cpp index 38620603..58e69a34 100644 --- a/runtime/src/main.cpp +++ b/runtime/src/main.cpp @@ -10,6 +10,8 @@ #include "cdrom.h" #include "fntrace.h" #include "boot_state.h" +#include "overlay_capture.h" +#include "overlay_loader.h" #include "gpu.h" #include "sio.h" #include "spu.h" @@ -468,6 +470,7 @@ static void close_controller(void); static void shutdown_runtime(void) { memcard_flush_all(); + overlay_capture_write_json(); if (sdl_audio_device) { SDL_ClearQueuedAudio(sdl_audio_device); SDL_CloseAudioDevice(sdl_audio_device); @@ -1106,6 +1109,16 @@ int main(int argc, char** argv) { if (gc.runtime.has_disc_speed) disc_speed = gc.runtime.disc_speed; game_entry_pc = gc.entry_pc; fast_boot = gc.runtime.fast_boot; + /* Overlay DLL cache (Layer A). Off unless enabled in [runtime]; + * when on, capture overlay bytes and scan cache// for + * precompiled overlay DLLs. */ + if (gc.runtime.overlay_cache) { + std::filesystem::path exe_dir = exe_dir_from_argv(argv[0]); + std::string cache_dir = (exe_dir / "cache").string(); + overlay_capture_set_out_dir(exe_dir.string().c_str()); + overlay_capture_set_enabled(1); + overlay_loader_init(cache_dir.c_str(), game_id.c_str()); + } std::fprintf(stdout, "psxrecomp: loaded game config %s (%s, %s)\n", game_config_path, game_name.c_str(), game_id.c_str()); } catch (const std::exception& ex) { diff --git a/runtime/src/overlay_capture.c b/runtime/src/overlay_capture.c new file mode 100644 index 00000000..e3f553a1 --- /dev/null +++ b/runtime/src/overlay_capture.c @@ -0,0 +1,285 @@ +#include "overlay_capture.h" +#include "overlay_loader.h" +#include "dirty_ram_interp.h" +#include "crc32.h" +#include +#include +#include + +/* ---- Capture set -------------------------------------------------------- + * Tracks which physical addresses have received CD DMA loads since game + * handoff. Used for dedup (state machine) and A-layer compilation queue. + * The JSON output uses dirty_ram regions, not individual DMA blocks. + * -------------------------------------------------------------------------*/ + +typedef enum { + OV_QUEUED = 0, /* captured, not yet compiled */ + OV_COMPILING = 1, /* background thread working on it */ + OV_COMPILED = 2, /* DLL written, dispatch table patched */ +} OvState; + +typedef struct { + uint32_t load_addr; /* physical RAM address */ + uint32_t size; /* byte count of this DMA block */ + uint8_t *bytes; /* unpatched copy taken at DMA time */ + OvState state; +} OvEntry; + +#define MAX_OVERLAYS 4096 + +static OvEntry s_entries[MAX_OVERLAYS]; +static int s_count = 0; +static char s_out_dir[512]; +static int s_active = 0; +static int s_enabled = 0; /* config gate; off unless overlay cache enabled */ + +/* ---- Base64 encoder ----------------------------------------------------- */ + +static const char k_b64[] = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + +static void write_b64(FILE *f, const uint8_t *data, size_t len) +{ + size_t i; + for (i = 0; i < len; i += 3) { + uint32_t b = (uint32_t)data[i] << 16; + if (i + 1 < len) b |= (uint32_t)data[i + 1] << 8; + if (i + 2 < len) b |= (uint32_t)data[i + 2]; + fputc(k_b64[(b >> 18) & 0x3F], f); + fputc(k_b64[(b >> 12) & 0x3F], f); + fputc(i + 1 < len ? k_b64[(b >> 6) & 0x3F] : '=', f); + fputc(i + 2 < len ? k_b64[(b ) & 0x3F] : '=', f); + } +} + +/* ---- Public API --------------------------------------------------------- */ + +void overlay_capture_set_out_dir(const char *out_dir) +{ + strncpy(s_out_dir, out_dir, sizeof(s_out_dir) - 1); + s_out_dir[sizeof(s_out_dir) - 1] = '\0'; +} + +void overlay_capture_init(const char *out_dir) +{ + overlay_capture_set_out_dir(out_dir); +} + +void overlay_capture_set_enabled(int enabled) +{ + s_enabled = enabled ? 1 : 0; +} + +void overlay_capture_on_dma(uint32_t load_addr, uint32_t size, + const uint8_t *bytes) +{ + int i; + OvEntry *e; + extern int fntrace_is_game_started(void); + + if (!s_enabled) return; /* overlay cache disabled in config */ + if (size == 0) return; + + /* Auto-activate on the first post-game-handoff DMA. */ + if (!s_active) { + if (!fntrace_is_game_started()) return; + s_active = 1; + } + + /* Dedup by load_addr. */ + for (i = 0; i < s_count; i++) { + if (s_entries[i].load_addr == load_addr) + return; + } + + if (s_count >= MAX_OVERLAYS) return; + + e = &s_entries[s_count++]; + e->load_addr = load_addr; + e->size = size; + e->state = OV_QUEUED; + e->bytes = (uint8_t *)malloc(size); + if (e->bytes) memcpy(e->bytes, bytes, size); + + /* A-1: check if a compiled DLL exists in cache for these bytes. */ + overlay_loader_check_cache(load_addr, size, bytes); + /* A-2: push onto compilation queue (to be added in A-2). */ +} + +/* ---- JSON output — assembled dirty_ram regions -------------------------- */ + +void overlay_capture_write_json(void) +{ + extern uint32_t dirty_ram_get_bitmap_word(uint32_t word_index); + extern uint32_t dirty_ram_get_bitmap_word_count(void); + + char path[600]; + FILE *f; + uint32_t bw, page_sz, lo_page; + uint32_t page, run_start; + int in_run, first_region; + + if (!s_active) return; + + /* Use the same lo threshold as overlay_dump: only above 0x98000. */ + uint32_t lo_phys = 0x00098000u; + + bw = dirty_ram_get_bitmap_word_count(); + page_sz = 4096u; + lo_page = lo_phys / page_sz; + + snprintf(path, sizeof(path), "%s/overlay_captures.json", s_out_dir); + f = fopen(path, "w"); + if (!f) return; + + fprintf(f, "[\n"); + in_run = 0; + run_start = 0; + first_region = 1; + + for (page = lo_page; page <= bw * 32u; page++) { + int dirty = 0; + if (page < bw * 32u) { + uint32_t word = dirty_ram_get_bitmap_word(page >> 5); + dirty = (word >> (page & 31u)) & 1u; + } + + if (dirty && !in_run) { + in_run = 1; + run_start = page; + } else if (!dirty && in_run) { + uint32_t phys = run_start * page_sz; + uint32_t size = (page - run_start) * page_sz; + uint32_t virt = 0x80000000u | (phys & 0x1FFFFFu); + int j; + + in_run = 0; + + /* Seeds: only per-PC interpreter hits — execution-verified. */ +#define MAX_CAPTURE_PCS DIRTY_RAM_PC_TABLE_SIZE + uint32_t *executed = (uint32_t *)malloc( + MAX_CAPTURE_PCS * sizeof(uint32_t)); + uint32_t *dispatch = (uint32_t *)malloc( + MAX_CAPTURE_PCS * sizeof(uint32_t)); + int nexec = 0; + int ndisp = 0; + if (!executed || !dispatch) { + free(executed); + free(dispatch); + continue; + } + + for (j = 0; j < DIRTY_RAM_PC_TABLE_SIZE; j++) { + DirtyRamPcEntry *pe = &g_dirty_ram_pc_table[j]; + if (pe->pc == 0 || pe->hits == 0) continue; + if (pe->pc < phys || pe->pc >= phys + size) continue; + dispatch[ndisp++] = pe->pc; + } + for (j = 0; j < DIRTY_RAM_PC_TABLE_SIZE; j++) { + DirtyRamPcEntry *pe = &g_dirty_ram_exec_pc_table[j]; + if (pe->pc == 0 || pe->hits == 0) continue; + if (pe->pc < phys || pe->pc >= phys + size) continue; + executed[nexec++] = pe->pc; + } + + /* Skip pure-data regions — nothing to compile. */ + if (ndisp == 0 && nexec == 0) { + free(executed); + free(dispatch); + continue; + } + + /* Assemble unpatched bytes from DMA-time captures. + * Start with a zeroed image, then blit each stored DMA block + * into its correct offset. This gives pre-patch bytes, not + * the potentially-modified bytes now in live RAM. */ + uint8_t *image = (uint8_t *)calloc(size, 1); + if (!image) { + free(executed); + free(dispatch); + continue; + } + for (j = 0; j < s_count; j++) { + OvEntry *blk = &s_entries[j]; + if (!blk->bytes) continue; + if (blk->load_addr < phys || blk->load_addr >= phys + size) continue; + uint32_t off = blk->load_addr - phys; + uint32_t copy_sz = blk->size; + if (off + copy_sz > size) copy_sz = size - off; + memcpy(image + off, blk->bytes, copy_sz); + } + + if (!first_region) fprintf(f, ",\n"); + first_region = 0; + + fprintf(f, " {\n"); + fprintf(f, " \"schema\": \"psxrecomp overlay capture v2\",\n"); + fprintf(f, " \"load_addr\": \"0x%08X\",\n", virt); + fprintf(f, " \"size\": %u,\n", size); + fprintf(f, " \"bytes_b64\": \""); + write_b64(f, image, size); + free(image); + fprintf(f, "\",\n"); + + fprintf(f, " \"executed_pcs\": ["); + for (j = 0; j < nexec; j++) { + uint32_t seed_virt = 0x80000000u | (executed[j] & 0x1FFFFFu); + if (j > 0) fprintf(f, ", "); + fprintf(f, "\"0x%08X\"", seed_virt); + } + fprintf(f, "],\n"); + + fprintf(f, " \"dispatch_entry_pcs\": ["); + for (j = 0; j < ndisp; j++) { + uint32_t seed_virt = 0x80000000u | (dispatch[j] & 0x1FFFFFu); + if (j > 0) fprintf(f, ", "); + fprintf(f, "\"0x%08X\"", seed_virt); + } + fprintf(f, "],\n"); + + fprintf(f, " \"function_entry_pcs\": [],\n"); + + fprintf(f, " \"seeds\": ["); + for (j = 0; j < ndisp; j++) { + uint32_t seed_virt = 0x80000000u | (dispatch[j] & 0x1FFFFFu); + if (j > 0) fprintf(f, ", "); + fprintf(f, "\"0x%08X\"", seed_virt); + } + fprintf(f, "]\n"); + fprintf(f, " }"); + free(executed); + free(dispatch); + } + } + + fprintf(f, "\n]\n"); + fclose(f); +} + +int overlay_capture_count(void) +{ + return s_count; +} + +uint32_t overlay_capture_get_region_crc(uint32_t region_start, + uint32_t region_size) +{ + int j; + uint8_t *image = (uint8_t *)calloc(region_size, 1); + if (!image) return 0; + + for (j = 0; j < s_count; j++) { + OvEntry *blk = &s_entries[j]; + if (!blk->bytes) continue; + if (blk->load_addr < region_start || + blk->load_addr >= region_start + region_size) continue; + uint32_t off = blk->load_addr - region_start; + uint32_t copy_sz = blk->size; + if (off + copy_sz > region_size) copy_sz = region_size - off; + memcpy(image + off, blk->bytes, copy_sz); + } + + uint32_t crc = crc32_compute(image, region_size); + free(image); + return crc; +} diff --git a/runtime/src/overlay_loader.c b/runtime/src/overlay_loader.c new file mode 100644 index 00000000..c8e8901a --- /dev/null +++ b/runtime/src/overlay_loader.c @@ -0,0 +1,365 @@ +#include "overlay_loader.h" +#include "overlay_api.h" +#include "crc32.h" +#include "interrupts.h" +#include "debug_server.h" +#include +#include +#include +#include +#include + +#ifdef _WIN32 +# define WIN32_LEAN_AND_MEAN +# include +#else +# include +#endif + +/* ---- Dynamic dispatch hash table --------------------------------------- */ +/* Open-addressed hash map: physical addr → compiled function pointer. + * Sized at 16384 slots — enough for ~10 overlays × ~500 functions each + * at <50% load factor. */ + +#define DYNTAB_CAP 16384u +#define DYNTAB_MASK (DYNTAB_CAP - 1u) + +typedef void (*OverlayFn)(CPUState *); + +typedef struct { + uint32_t addr; /* physical address, 0 = empty */ + OverlayFn fn; +} DynEntry; + +static DynEntry s_table[DYNTAB_CAP]; +static int s_count = 0; + +static void dyntab_insert(uint32_t phys, OverlayFn fn) +{ + uint32_t h = (phys * 2654435761u) & DYNTAB_MASK; + uint32_t i; + for (i = 0; i < DYNTAB_CAP; i++) { + uint32_t idx = (h + i) & DYNTAB_MASK; + if (s_table[idx].addr == 0 || s_table[idx].addr == phys) { + if (s_table[idx].addr == 0) s_count++; + s_table[idx].addr = phys; + s_table[idx].fn = fn; + return; + } + } + /* Table full — shouldn't happen at <50% load */ +} + +static OverlayFn dyntab_lookup(uint32_t phys) +{ + uint32_t h = (phys * 2654435761u) & DYNTAB_MASK; + uint32_t i; + for (i = 0; i < DYNTAB_CAP; i++) { + uint32_t idx = (h + i) & DYNTAB_MASK; + if (s_table[idx].addr == 0) return NULL; + if (s_table[idx].addr == phys) return s_table[idx].fn; + } + return NULL; +} + +/* ---- Global state ------------------------------------------------------ */ + +static char s_cache_dir[512]; +static char s_game_id[64]; +static int s_active = 0; + +/* Rule 3: no stderr logging. The most recent loader event is recorded here + * and surfaced through the `overlay_loader_status` TCP command instead of + * being printed. */ +static char s_last_msg[256] = {0}; + +static void loader_log(const char *fmt, ...) +{ + va_list ap; + va_start(ap, fmt); + vsnprintf(s_last_msg, sizeof(s_last_msg), fmt, ap); + va_end(ap); +} + +const char *overlay_loader_last_msg(void) { return s_last_msg; } + +/* ---- Cache index: region_start → dll path ------------------------------- */ +/* Scanned once at init from {cache_dir}/{game_id}/{8hex}_{8hex}.dll files. + * Avoids recomputing the DMA CRC at dispatch time (which fails after + * fast_boot because DMA callbacks haven't fired to fill s_entries). */ + +#define CACHE_IDX_CAP 256 +typedef struct { uint32_t region_start; char path[768]; } CacheEntry; +static CacheEntry s_cache_idx[CACHE_IDX_CAP]; +static int s_cache_idx_count = 0; + +static void scan_cache_dir(void) +{ +#ifdef _WIN32 + char pattern[768]; + snprintf(pattern, sizeof(pattern), "%s/%s/*_*.dll", + s_cache_dir, s_game_id); + WIN32_FIND_DATAA fd; + HANDLE h = FindFirstFileA(pattern, &fd); + if (h == INVALID_HANDLE_VALUE) return; + do { + if (strlen(fd.cFileName) != 21) continue; /* 8+1+8+4 = 21 */ + uint32_t addr = (uint32_t)strtoul(fd.cFileName, NULL, 16); + if (addr == 0) continue; + if (s_cache_idx_count >= CACHE_IDX_CAP) break; + CacheEntry *e = &s_cache_idx[s_cache_idx_count++]; + e->region_start = addr; + snprintf(e->path, sizeof(e->path), "%s/%s/%s", + s_cache_dir, s_game_id, fd.cFileName); + } while (FindNextFileA(h, &fd)); + FindClose(h); +#endif +} + +/* ---- Runtime callbacks wired into overlay DLLs via overlay_init() ------ */ + +extern void psx_dispatch_call(CPUState *cpu, uint32_t addr, uint32_t ra); +extern void psx_check_interrupts(CPUState *cpu); +extern void gte_execute(CPUState *cpu, uint32_t cmd); +extern void psx_syscall(CPUState *cpu, uint32_t code); +extern void psx_unknown_dispatch(CPUState *cpu, uint32_t addr, uint32_t phys); +extern void debug_server_log_call_entry(uint32_t func_addr); + +static OverlayCallbacks s_callbacks; + +static void init_callbacks(void) +{ + s_callbacks.dispatch_call = psx_dispatch_call; + s_callbacks.check_interrupts = psx_check_interrupts; + s_callbacks.gte_execute = gte_execute; + s_callbacks.psx_syscall = psx_syscall; + s_callbacks.psx_unknown_dispatch = psx_unknown_dispatch; + s_callbacks.log_call_entry = debug_server_log_call_entry; +} + +/* ---- DLL loading and export enumeration -------------------------------- */ + +#ifdef _WIN32 +static int load_overlay_dll(const char *dll_path, uint32_t load_addr_virt) +{ + HMODULE dll = LoadLibraryA(dll_path); + if (!dll) { + loader_log("LoadLibrary(%s) failed: %lu", dll_path, GetLastError()); + return 0; + } + + /* Call overlay_init to wire the runtime callbacks. */ + typedef void (*InitFn)(const OverlayCallbacks *); + InitFn init_fn = (InitFn)GetProcAddress(dll, "overlay_init"); + if (!init_fn) { + loader_log("no overlay_init in %s", dll_path); + FreeLibrary(dll); + return 0; + } + init_fn(&s_callbacks); + + /* Enumerate PE export table for func_XXXXXXXX symbols. */ + BYTE *base = (BYTE *)dll; + IMAGE_DOS_HEADER *dos = (IMAGE_DOS_HEADER *)base; + IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)(base + dos->e_lfanew); + IMAGE_DATA_DIRECTORY *exp_dd = + &nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT]; + + if (!exp_dd->VirtualAddress) { + loader_log("no export dir in %s", dll_path); + FreeLibrary(dll); + return 0; + } + + IMAGE_EXPORT_DIRECTORY *exp = + (IMAGE_EXPORT_DIRECTORY *)(base + exp_dd->VirtualAddress); + DWORD *names = (DWORD *)(base + exp->AddressOfNames); + WORD *ordinals = (WORD *)(base + exp->AddressOfNameOrdinals); + DWORD *funcs = (DWORD *)(base + exp->AddressOfFunctions); + + int registered = 0; + for (DWORD i = 0; i < exp->NumberOfNames; i++) { + const char *name = (const char *)(base + names[i]); + /* Match "func_XXXXXXXX" — exactly 13 chars */ + if (strncmp(name, "func_", 5) != 0) continue; + if (strlen(name) != 13) continue; + uint32_t addr = (uint32_t)strtoul(name + 5, NULL, 16); + if (addr == 0) continue; + + WORD ord = ordinals[i]; + OverlayFn fn = (OverlayFn)(base + funcs[ord]); + + uint32_t phys = addr & 0x1FFFFFFFu; + dyntab_insert(phys, fn); + registered++; + } + + loader_log("loaded %s -> %d functions registered", dll_path, registered); + return registered; +} +#else +static int load_overlay_dll(const char *dll_path, uint32_t load_addr_virt) +{ + void *dll = dlopen(dll_path, RTLD_NOW | RTLD_LOCAL); + if (!dll) { + loader_log("dlopen(%s) failed: %s", dll_path, dlerror()); + return 0; + } + + typedef void (*InitFn)(const OverlayCallbacks *); + InitFn init_fn = (InitFn)dlsym(dll, "overlay_init"); + if (!init_fn) { + loader_log("no overlay_init in %s", dll_path); + dlclose(dll); + return 0; + } + init_fn(&s_callbacks); + + /* On non-Windows we can't easily enumerate exports without libelf/BFD. + * Fall back to scanning the known address range: probe GetProcAddress + * equivalents by constructing names from the seed list stored in the + * overlay capture. For now, register nothing — the interpreter handles + * it until proper export enumeration is implemented. */ + loader_log("%s loaded (posix export scan TODO)", dll_path); + return 0; +} +#endif + +/* ---- Public API -------------------------------------------------------- */ + +void overlay_loader_init(const char *cache_dir, const char *game_id) +{ + strncpy(s_cache_dir, cache_dir, sizeof(s_cache_dir) - 1); + strncpy(s_game_id, game_id, sizeof(s_game_id) - 1); + init_callbacks(); + scan_cache_dir(); + s_active = 1; +} + +void overlay_loader_check_cache(uint32_t load_addr, uint32_t size, + const uint8_t *bytes) +{ + /* Block-level check is not used — the DLL CRC is over the full assembled + * region, not individual DMA blocks. Cache loading is deferred to + * overlay_loader_dispatch on the first dispatch miss. */ + (void)load_addr; (void)size; (void)bytes; +} + +/* ---- Lazy region cache check ------------------------------------------- */ +/* On the first dispatch miss for any address in a dirty region, we compute + * the CRC32 of the full assembled region from RAM and look for a matching + * DLL. This fires AFTER the game has fully loaded the overlay and called + * into it — by that point all DMA blocks have landed in RAM. */ + +#define MAX_CHECKED 64 +static uint32_t s_checked[MAX_CHECKED]; +static int s_nchecked = 0; +static uint32_t s_last_crc = 0; +static int s_last_file_found = 0; + +static int already_checked(uint32_t region_start) { + int i; + for (i = 0; i < s_nchecked; i++) + if (s_checked[i] == region_start) return 1; + return 0; +} + +static void mark_checked(uint32_t region_start) { + if (s_nchecked < MAX_CHECKED) + s_checked[s_nchecked++] = region_start; +} + +static void try_load_region(uint32_t phys) +{ + extern uint32_t dirty_ram_get_bitmap_word(uint32_t word_index); + extern uint32_t dirty_ram_get_bitmap_word_count(void); + extern uint8_t *memory_get_ram_ptr(void); + + uint32_t page_sz = 4096u; + uint32_t bw = dirty_ram_get_bitmap_word_count(); + char dll_path[768]; + + /* Walk backward to find region start. */ + uint32_t pg = phys / page_sz; + while (pg > 0) { + uint32_t pp = pg - 1; + if (!((dirty_ram_get_bitmap_word(pp >> 5) >> (pp & 31u)) & 1u)) break; + pg = pp; + } + uint32_t region_start = pg * page_sz; + + if (already_checked(region_start)) return; + mark_checked(region_start); + + /* Walk forward to find region end. */ + uint32_t pg2 = phys / page_sz + 1; + while (pg2 < bw * 32u && + ((dirty_ram_get_bitmap_word(pg2 >> 5) >> (pg2 & 31u)) & 1u)) + pg2++; + uint32_t region_size = (pg2 - pg) * page_sz; + + /* Look up the compiled DLL by region_start in the cache index. + * The index was built at init by scanning for {addr}_{crc}.dll files, + * so we never need to recompute the CRC from RAM or DMA blocks. */ + s_last_crc = 0; + s_last_file_found = 0; + (void)region_size; + + int ci; + const char *found_path = NULL; + for (ci = 0; ci < s_cache_idx_count; ci++) { + if (s_cache_idx[ci].region_start == region_start) { + found_path = s_cache_idx[ci].path; + break; + } + } + if (!found_path) return; + + strncpy(dll_path, found_path, sizeof(dll_path) - 1); + s_last_file_found = 1; + load_overlay_dll(dll_path, 0x80000000u | (region_start & 0x1FFFFFFFu)); +} + +int overlay_loader_dispatch(CPUState *cpu, uint32_t addr) +{ + uint32_t phys = addr & 0x1FFFFFFFu; + OverlayFn fn = dyntab_lookup(phys); + if (fn) { fn(cpu); return 1; } + + /* First miss for this region: check if a DLL is cached. */ + if (s_active && phys >= 0x98000u) + try_load_region(phys); + + /* Retry after potential DLL load. */ + fn = dyntab_lookup(phys); + if (fn) { fn(cpu); return 1; } + + return 0; +} + +int overlay_loader_registered_count(void) +{ + return s_count; +} + +void overlay_loader_get_status(int *active, int *registered, + int *regions_checked, + char *cache_dir_out, int cache_dir_len, + char *game_id_out, int game_id_len, + uint32_t *checked_out, int checked_max, + int *checked_written, + uint32_t *last_crc_out, int *last_file_found_out) +{ + if (active) *active = s_active; + if (registered) *registered = s_count; + if (regions_checked) *regions_checked = s_nchecked; + if (cache_dir_out) strncpy(cache_dir_out, s_cache_dir, (size_t)cache_dir_len - 1); + if (game_id_out) strncpy(game_id_out, s_game_id, (size_t)game_id_len - 1); + if (checked_out && checked_written) { + int n = s_nchecked < checked_max ? s_nchecked : checked_max; + for (int i = 0; i < n; i++) checked_out[i] = s_checked[i]; + *checked_written = n; + } + if (last_crc_out) *last_crc_out = s_last_crc; + if (last_file_found_out) *last_file_found_out = s_last_file_found; +} diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py new file mode 100644 index 00000000..e2e4f907 --- /dev/null +++ b/tools/compile_overlays.py @@ -0,0 +1,1045 @@ +#!/usr/bin/env python3 +""" +compile_overlays.py — Offline overlay compilation primitive (B-2 / A-2 shared path). + +Reads overlay_captures.json, compiles each code overlay to a DLL, and writes +it to the cache directory. The runtime's A-1 LoadLibrary path reads from the +same cache. + +Usage: + python3 tools/compile_overlays.py \\ + --captures build-dev/overlay_captures.json \\ + --game-toml game.toml \\ + --recompiler psxrecomp/recompiler/build/psxrecomp-game.exe \\ + --runtime-include psxrecomp/runtime/include \\ + --out-dir build-dev/cache + +Each DLL is written to //.dll. +Each DLL exports: + overlay_init(dispatch_fn) — call once after LoadLibrary to wire dispatch + func_XXXXXXXX(CPUState*) — one export per compiled function entry point +""" + +import argparse +import base64 +import binascii +from collections import Counter, deque +import os +import re +import struct +import subprocess +import sys +import tempfile +import tomllib # Python 3.11+; fallback below + +try: + import tomllib +except ImportError: + try: + import tomli as tomllib + except ImportError: + print("ERROR: need tomllib (Python 3.11+) or 'pip install tomli'") + sys.exit(1) + +import json + + +# --------------------------------------------------------------------------- +# PS-EXE fake header +# --------------------------------------------------------------------------- + +def make_psxexe(load_addr: int, entry_pc: int, data: bytes) -> bytes: + """Wrap raw overlay bytes in a minimal PS-EXE header.""" + header = bytearray(2048) + header[0:8] = b'PS-X EXE' + struct.pack_into(' int: + if isinstance(value, int): + return value + return int(str(value), 16) + + +def _parse_addr_list(values) -> set[int]: + out = set() + for v in values or []: + try: + out.add(_parse_addr(v)) + except (TypeError, ValueError): + pass + return out + + +def _word_at(data: bytes, load_addr: int, addr: int): + off = addr - load_addr + if off < 0 or off + 4 > len(data): + return None + return struct.unpack_from(' bool: + return word == 0x03E00008 + + +def _is_addiu_sp_neg(word) -> bool: + if word is None: + return False + return ((word >> 26) & 0x3F) == 0x09 \ + and ((word >> 21) & 0x1F) == 29 \ + and ((word >> 16) & 0x1F) == 29 \ + and (word & 0x8000) != 0 + + +def _is_control_flow(word) -> bool: + if word is None: + return False + op = (word >> 26) & 0x3F + fn = word & 0x3F + return op in (0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x14, 0x15, 0x16, 0x17) or (op == 0 and fn in (0x08, 0x09)) + + +def _is_valid_mips_word(word) -> bool: + if word is None or word in (0xFFFFFFFF, 0xFFFFFFFD): + return False + + op = (word >> 26) & 0x3F + fn = word & 0x3F + rt = (word >> 16) & 0x1F + + if op == 0x00: + return fn in { + 0x00, 0x02, 0x03, 0x04, 0x06, 0x07, + 0x08, 0x09, 0x0C, 0x0D, + 0x10, 0x11, 0x12, 0x13, + 0x18, 0x19, 0x1A, 0x1B, + 0x20, 0x21, 0x22, 0x23, + 0x24, 0x25, 0x26, 0x27, + 0x2A, 0x2B, + } + if op == 0x01: + return rt in (0x00, 0x01, 0x10, 0x11) + + return op in { + 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x10, 0x12, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, + 0x28, 0x29, 0x2A, 0x2B, 0x2E, + 0x30, 0x32, 0x38, 0x3A, + } + + +def _jump_target(pc: int, word: int) -> int: + return ((pc + 4) & 0xF0000000) | ((word & 0x03FFFFFF) << 2) + + +def _branch_target(pc: int, word: int) -> int: + imm = word & 0xFFFF + if imm & 0x8000: + imm -= 0x10000 + return pc + 4 + (imm << 2) + + +def _classify_cf(pc: int, word: int) -> tuple[str, int]: + op = (word >> 26) & 0x3F + fn = word & 0x3F + rs = (word >> 21) & 0x1F + if op == 0 and fn == 0x08: + return ('jr_ra' if rs == 31 else 'jr', 0) + if op == 0 and fn == 0x09: + return ('jalr', 0) + if op == 0x02: + return ('j', _jump_target(pc, word)) + if op == 0x03: + return ('jal', _jump_target(pc, word)) + if op in (0x01, 0x04, 0x05, 0x06, 0x07, 0x14, 0x15, 0x16, 0x17): + return ('branch', _branch_target(pc, word)) + return ('normal', 0) + + +def _find_jump_table_targets(data: bytes, load_addr: int, size: int, + entry: int, hard_cap: int, + jr_pc: int, jr_rs: int) -> set[int]: + """Recognize the compiler's local jump-table idiom feeding `jr reg`.""" + lw_base = None + lw_offset = 0 + addu_cand = [None, None] + lui_val = None + addiu_val = [0, 0] + found_addiu = [False, False] + table_count = 0 + + for back in range(1, 41): + scan_addr = jr_pc - back * 4 + if scan_addr < entry: + break + word = _word_at(data, load_addr, scan_addr) + if word is None: + break + op = (word >> 26) & 0x3F + rs = (word >> 21) & 0x1F + rt = (word >> 16) & 0x1F + rd = (word >> 11) & 0x1F + fn = word & 0x3F + + if op == 0x23 and rt == jr_rs and lw_base is None: + lw_base = rs + lw_offset = word & 0xFFFF + if lw_offset & 0x8000: + lw_offset -= 0x10000 + continue + if op == 0x00 and fn == 0x21 and lw_base is not None and rd == lw_base \ + and addu_cand[0] is None: + addu_cand = [rs, rt] + continue + if op == 0x09 and addu_cand[0] is not None: + for i, cand in enumerate(addu_cand): + if cand is not None and not found_addiu[i] and rs == cand and rt == cand: + imm = word & 0xFFFF + if imm & 0x8000: + imm -= 0x10000 + addiu_val[i] = imm + found_addiu[i] = True + break + continue + if op == 0x0F and addu_cand[0] is not None and lui_val is None: + for i, cand in enumerate(addu_cand): + if cand is not None and rt == cand: + lui_val = (word & 0xFFFF) << 16 + addiu_val[0] = addiu_val[i] if found_addiu[i] else 0 + found_addiu[0] = found_addiu[i] + break + continue + if op == 0x0B and table_count == 0: + table_count = word & 0xFFFF + continue + if lui_val is not None and table_count: + break + + if lui_val is None or table_count == 0 or table_count >= 512: + return set() + + table_base = lui_val + (addiu_val[0] if found_addiu[0] else 0) + lw_offset + lo = load_addr + hi = load_addr + size + targets = set() + for i in range(table_count): + target = _word_at(data, load_addr, table_base + i * 4) + if target is None: + continue + if lo <= target < hi and entry <= target < hard_cap and (target & 3) == 0: + targets.add(target) + return targets + + +def _callable_legacy_seed(data: bytes, load_addr: int, addr: int) -> bool: + """Conservative fallback for old captures that only had a `seeds` list.""" + word = _word_at(data, load_addr, addr) + prev = _word_at(data, load_addr, addr - 4) + prev8 = _word_at(data, load_addr, addr - 8) + if not _is_valid_mips_word(word): + return False + if addr == load_addr: + return True + if _is_jr_ra(prev8): + return True + return _is_addiu_sp_neg(word) and not _is_control_flow(prev) + + +def _callable_direct_jal_target(data: bytes, load_addr: int, addr: int) -> bool: + """True only when a direct JAL target has independent boundary evidence.""" + return _callable_legacy_seed(data, load_addr, addr) + + +def _walk_overlay_function(data: bytes, load_addr: int, size: int, + entry: int, hard_cap: int) -> dict: + lo = load_addr + hi = load_addr + size + + def in_function(addr: int) -> bool: + return lo <= addr < hi and entry <= addr < hard_cap and (addr & 3) == 0 + + work = deque([entry]) + visited = set() + direct_jals = set() + branch_targets = set() + jump_table_targets = set() + + while work: + pc = work.popleft() + if pc in visited or not in_function(pc): + continue + word = _word_at(data, load_addr, pc) + if word is None: + continue + visited.add(pc) + kind, target = _classify_cf(pc, word) + delay = pc + 4 + + if kind == 'normal': + if in_function(pc + 4): + work.append(pc + 4) + elif kind == 'branch': + if in_function(delay): + visited.add(delay) + if in_function(pc + 8): + work.append(pc + 8) + branch_targets.add(pc + 8) + if in_function(target): + work.append(target) + branch_targets.add(target) + elif kind == 'j': + if in_function(delay): + visited.add(delay) + if in_function(target): + work.append(target) + branch_targets.add(target) + elif kind == 'jal': + if in_function(delay): + visited.add(delay) + if in_function(pc + 8): + work.append(pc + 8) + branch_targets.add(pc + 8) + if (lo <= target < hi and (target & 3) == 0 and + _callable_direct_jal_target(data, load_addr, target)): + direct_jals.add(target) + elif kind == 'jalr': + if in_function(delay): + visited.add(delay) + if in_function(pc + 8): + work.append(pc + 8) + branch_targets.add(pc + 8) + elif kind == 'jr': + if in_function(delay): + visited.add(delay) + jr_rs = (word >> 21) & 0x1F + for jt in _find_jump_table_targets(data, load_addr, size, + entry, hard_cap, pc, jr_rs): + jump_table_targets.add(jt) + branch_targets.add(jt) + if in_function(jt): + work.append(jt) + elif kind == 'jr_ra': + if in_function(delay): + visited.add(delay) + + return { + 'visited': visited, + 'direct_jals': direct_jals, + 'branch_targets': branch_targets, + 'jump_table_targets': jump_table_targets, + } + + +def _collect_toml_overlay_entries(toml_doc: dict, load_addr: int, crc32: int) -> set[int]: + entries = set() + for ov in toml_doc.get('overlays', []) or []: + if not isinstance(ov, dict): + continue + ov_load = ov.get('load_addr') or ov.get('load_address') + if ov_load is not None and _parse_addr(ov_load) != load_addr: + continue + ov_crc = ov.get('bytes_crc') or ov.get('crc32') or ov.get('crc') + if ov_crc is not None and _parse_addr(ov_crc) != crc32: + continue + for key in ('entry', 'entries', 'function_entry_pcs', 'function_entries'): + if key not in ov: + continue + val = ov[key] + if isinstance(val, list): + entries.update(_parse_addr_list(val)) + else: + entries.add(_parse_addr(val)) + return entries + + +def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, + crc32: int, toml_doc: dict) -> tuple[list[str], dict]: + lo = load_addr + hi = load_addr + size + region = lambda a: lo <= a < hi and (a & 3) == 0 + + schema_keys = {'executed_pcs', 'observed_pcs', 'dispatch_entry_pcs', + 'function_entry_pcs'} + has_split_schema = any(k in cap for k in schema_keys) + + legacy_seeds = _parse_addr_list(cap.get('seeds', [])) + executed_pcs = _parse_addr_list(cap.get('executed_pcs', + cap.get('observed_pcs', + legacy_seeds if not has_split_schema else []))) + dispatch_entry_pcs = _parse_addr_list(cap.get('dispatch_entry_pcs', [])) + captured_function_entries = _parse_addr_list(cap.get('function_entry_pcs', [])) + toml_entries = _collect_toml_overlay_entries(toml_doc, load_addr, crc32) + legacy_callable_seeds = {a for a in legacy_seeds + if region(a) and _callable_legacy_seed(data, load_addr, a)} + + included: dict[int, str] = {} + excluded: dict[int, str] = {} + game_text = _game_text_range(toml_doc) + + # A dispatch into dirty RAM can land on a jump-table case label. That + # proves coverage, not a callable function boundary, so discover those + # labels before promoting dispatch entries to seeds. + pre_roots = set(legacy_callable_seeds) | set(captured_function_entries) | set(toml_entries) + pre_roots.update(a for a in dispatch_entry_pcs + if region(a) and _callable_legacy_seed(data, load_addr, a)) + jump_table_targets = set() + pre_roots_sorted = sorted(a for a in pre_roots if region(a)) + for i, entry in enumerate(pre_roots_sorted): + hard_cap = pre_roots_sorted[i + 1] if i + 1 < len(pre_roots_sorted) else hi + walk = _walk_overlay_function(data, load_addr, size, entry, hard_cap) + jump_table_targets.update(walk['jump_table_targets']) + + def impossible_entry_start(addr: int) -> bool: + word = _word_at(data, load_addr, addr) + if word is None: + return True + if not _is_valid_mips_word(word): + return True + kind, target = _classify_cf(addr, word) + if kind == 'j': + return _classify_target(target, load_addr, size, game_text) == 'UNKNOWN_BAD' + if kind == 'jal' and region(target): + return not _callable_direct_jal_target(data, load_addr, target) + return False + + def include(addr: int, reason: str): + if not region(addr): + excluded[addr] = 'UNKNOWN' + return + if (reason != 'TOML_DECLARED_ENTRY' and addr in jump_table_targets and + not _callable_legacy_seed(data, load_addr, addr)): + excluded[addr] = 'BRANCH_TARGET_ONLY' + return + if reason == 'DISPATCH_ENTRY' and impossible_entry_start(addr): + excluded[addr] = 'UNKNOWN' + return + if reason in FATAL_SEED_REASONS: + raise RuntimeError(f'BUG: refusing to include 0x{addr:08X} as {reason}') + old = included.get(addr) + if old is None or old == 'FUNCTION_POINTER_TARGET': + included[addr] = reason + + for addr in dispatch_entry_pcs: + include(addr, 'DISPATCH_ENTRY') + for addr in captured_function_entries: + include(addr, 'FUNCTION_POINTER_TARGET') + for addr in toml_entries: + include(addr, 'TOML_DECLARED_ENTRY') + + legacy_seed_mode = bool(legacy_seeds) and not cap.get('schema') + if legacy_seed_mode: + # Old capture files used `seeds` for interpreter-observed PCs. Include + # only PCs whose surrounding bytes look callable; classify the rest + # after we know branch targets from accepted roots. + for addr in legacy_callable_seeds: + include(addr, 'FUNCTION_POINTER_TARGET') + + known = set(included) + pending = deque(sorted(known)) + processed = set() + all_branch_targets = set() + + while pending: + entry = pending.popleft() + if entry in processed: + continue + processed.add(entry) + sorted_known = sorted(known) + hard_cap = next((x for x in sorted_known if x > entry), hi) + walk = _walk_overlay_function(data, load_addr, size, entry, hard_cap) + all_branch_targets.update(walk['branch_targets']) + all_branch_targets.update(walk['jump_table_targets']) + for target in sorted(walk['direct_jals']): + if target not in known: + include(target, 'DIRECT_JAL_TARGET') + if target in included: + known.add(target) + pending.append(target) + + # Re-walk with the final function set so the branch-target exclusion count + # matches the actual compilation boundaries. + all_branch_targets.clear() + sorted_known = sorted(known) + for i, entry in enumerate(sorted_known): + hard_cap = sorted_known[i + 1] if i + 1 < len(sorted_known) else hi + walk = _walk_overlay_function(data, load_addr, size, entry, hard_cap) + all_branch_targets.update(walk['branch_targets']) + all_branch_targets.update(walk['jump_table_targets']) + + candidates = {a for a in (executed_pcs | dispatch_entry_pcs | + captured_function_entries | legacy_seeds | toml_entries) + if region(a)} + for addr in sorted(candidates - set(included)): + if addr in all_branch_targets or addr in jump_table_targets: + excluded[addr] = 'BRANCH_TARGET_ONLY' + elif addr in executed_pcs or addr in legacy_seeds: + excluded[addr] = 'OBSERVED_PC_ONLY' + else: + excluded[addr] = 'UNKNOWN' + + bad_included = [(a, r) for a, r in included.items() if r in FATAL_SEED_REASONS] + if bad_included: + details = ', '.join(f'0x{a:08X}:{r}' for a, r in bad_included) + raise RuntimeError(f'fatal seed classification: {details}') + + counts = Counter(included.values()) + excluded_counts = Counter(excluded.values()) + audit = { + 'load_addr': load_addr, + 'crc32': crc32, + 'lo': lo, + 'hi': hi, + 'executed_pcs': executed_pcs, + 'dispatch_entry_pcs': dispatch_entry_pcs, + 'function_entry_pcs': set(included), + 'included_reasons': included, + 'excluded_reasons': excluded, + 'branch_targets_excluded_count': len(all_branch_targets - set(included)), + 'counts': counts, + 'excluded_counts': excluded_counts, + } + seeds = [f'0x{addr:08X}' for addr in sorted(included)] + return seeds, audit + + +def print_seed_audit(audit: dict) -> None: + print(f'Overlay {audit["load_addr"]:08X}_{audit["crc32"]:08X}') + print(f'Region: {audit["lo"]:08X}..{audit["hi"] - 1:08X}') + print(f'executed_pcs: {len(audit["executed_pcs"])}') + print(f'dispatch_entry_pcs: {len(audit["dispatch_entry_pcs"])}') + print(f'function_entry_pcs: {len(audit["function_entry_pcs"])}') + print(f'direct_jal_targets_included: {audit["counts"].get("DIRECT_JAL_TARGET", 0)}') + print(f'function_pointer_targets_included: {audit["counts"].get("FUNCTION_POINTER_TARGET", 0)}') + print(f'toml_entries_included: {audit["counts"].get("TOML_DECLARED_ENTRY", 0)}') + print(f'branch_targets_excluded: {audit["branch_targets_excluded_count"]}') + print(f'observed_only_excluded: {audit["excluded_counts"].get("OBSERVED_PC_ONLY", 0)}') + print(f'unknown_excluded: {audit["excluded_counts"].get("UNKNOWN", 0)}') + for addr in sorted(audit['included_reasons']): + print(f' {addr:08X} {audit["included_reasons"][addr]}') + for addr in sorted(audit['excluded_reasons']): + reason = audit['excluded_reasons'][addr] + if reason in ('BRANCH_TARGET_ONLY', 'OBSERVED_PC_ONLY', 'UNKNOWN'): + print(f' {addr:08X} excluded: {reason}') + + +def _game_text_range(toml_doc: dict) -> tuple[int, int]: + game = toml_doc.get('game', {}) + load = game.get('load_address') + size = game.get('text_size') + if load is None or size is None: + return (0, 0) + lo = _parse_addr(load) & 0x1FFFFFFF + hi = lo + _parse_addr(size) + return lo, hi + + +def _classify_target(addr: int, load_addr: int, size: int, + game_text: tuple[int, int]) -> str: + phys = addr & 0x1FFFFFFF + ov_lo = load_addr & 0x1FFFFFFF + ov_hi = ov_lo + size + if ov_lo <= phys < ov_hi: + return 'INSIDE_OVERLAY' + game_lo, game_hi = game_text + if game_hi > game_lo and game_lo <= phys < game_hi: + return 'MAIN_EXE' + if 0x1FC00000 <= phys < 0x1FC80000 or 0xBFC00000 <= addr < 0xBFC80000: + return 'BIOS' + return 'UNKNOWN_BAD' + + +def audit_generated_c(src: str, load_addr: int, size: int, + crc32: int, toml_doc: dict) -> dict: + defs = {int(x, 16) for x in re.findall( + r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\)$', src, re.MULTILINE)} + decls = {int(x, 16) for x in re.findall( + r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\);$', src, re.MULTILINE)} + direct_calls = [int(x, 16) for x in re.findall( + r'\bfunc_([0-9A-Fa-f]{8})\(cpu\)', src)] + literal_cba = [int(x, 16) for x in re.findall( + r'\bcall_by_address\(cpu,\s*0x([0-9A-Fa-f]{8})u?\)', src)] + unsupported_todo_addrs = {int(x, 16) for x in re.findall( + r'TODO:[^\n]*?0x([0-9A-Fa-f]{8}):', src)} + + game_text = _game_text_range(toml_doc) + ov_lo = load_addr & 0x1FFFFFFF + ov_hi = ov_lo + size + + def in_overlay(addr: int) -> bool: + phys = addr & 0x1FFFFFFF + return ov_lo <= phys < ov_hi + + unknown_bad = set() + missing_direct = set() + direct_outside = set() + for addr in direct_calls: + if not in_overlay(addr): + direct_outside.add(addr) + unknown_bad.add(addr) + elif addr not in defs: + missing_direct.add(addr) + unknown_bad.add(addr) + + cba_classes = Counter() + for addr in literal_cba: + cls = _classify_target(addr, load_addr, size, game_text) + cba_classes[cls] += 1 + if cls == 'UNKNOWN_BAD': + unknown_bad.add(addr) + + decl_without_def = {a for a in decls if in_overlay(a) and a not in defs} + called_decl_without_def = decl_without_def & set(direct_calls) + if called_decl_without_def: + unknown_bad.update(called_decl_without_def) + + report = { + 'defs': defs, + 'decls': decls, + 'direct_calls': direct_calls, + 'literal_cba': literal_cba, + 'direct_inside': sum(1 for a in direct_calls if in_overlay(a)), + 'external_cba': sum(1 for a in literal_cba if not in_overlay(a)), + 'bios_calls': cba_classes.get('BIOS', 0), + 'syscall_uses': len(re.findall(r'\bpsx_syscall\s*\(', src)), + 'unknown_dispatch_uses': len(re.findall(r'\bpsx_unknown_dispatch\s*\(', src)), + 'unsupported_todo_addrs': unsupported_todo_addrs, + 'unknown_bad': unknown_bad, + 'direct_outside': direct_outside, + 'missing_direct': missing_direct, + 'decl_without_def': decl_without_def, + 'called_decl_without_def': called_decl_without_def, + 'cba_classes': cba_classes, + } + return report + + +def print_generated_c_audit(load_addr: int, size: int, crc32: int, + report: dict) -> None: + print(f'Overlay {load_addr:08X}_{crc32:08X}') + print(f'Region: {load_addr:08X}..{load_addr + size - 1:08X}') + print(f'Function definitions: {len(report["defs"])}') + print(f'Forward declarations: {len(report["decls"])}') + print(f'Direct calls inside overlay: {report["direct_inside"]}') + print(f'External calls via call_by_address: {report["external_cba"]}') + print(f'BIOS calls: {report["bios_calls"]}') + print(f'Syscall uses: {report["syscall_uses"]}') + print(f'Unsupported instruction TODOs: {len(report["unsupported_todo_addrs"])}') + print(f'Unknown/bad targets: {len(report["unknown_bad"])}') + for addr in sorted(report['unknown_bad']): + print(f' 0x{addr:08X} UNKNOWN_BAD') + for addr in sorted(report['unsupported_todo_addrs'])[:20]: + print(f' 0x{addr:08X} UNSUPPORTED_INSTRUCTION') + if len(report['unsupported_todo_addrs']) > 20: + print(f' ... {len(report["unsupported_todo_addrs"]) - 20} more unsupported instructions') + + +# --------------------------------------------------------------------------- +# Post-process generated C for DLL compilation +# --------------------------------------------------------------------------- + +DISPATCH_PREAMBLE = """\ +#include "overlay_api.h" + +/* ---- Overlay dispatch shim (inserted by compile_overlays.py) ----------- */ +static OverlayCallbacks g_cbs; + +#ifdef _WIN32 +__declspec(dllexport) +#else +__attribute__((visibility("default"))) +#endif +void overlay_init(const OverlayCallbacks *cbs) { g_cbs = *cbs; } + +void psx_dispatch_call(CPUState *cpu, uint32_t addr, uint32_t ra) { + (void)ra; + g_cbs.dispatch_call(cpu, addr, cpu->gpr[31]); +} +void psx_check_interrupts(CPUState *cpu) { + g_cbs.check_interrupts(cpu); +} +void gte_execute(CPUState *cpu, uint32_t cmd) { + g_cbs.gte_execute(cpu, cmd); +} +void psx_syscall(CPUState *cpu, uint32_t code) { + g_cbs.psx_syscall(cpu, code); +} +void psx_unknown_dispatch(CPUState *cpu, uint32_t addr, uint32_t phys) { + g_cbs.psx_unknown_dispatch(cpu, addr, phys); +} +void debug_server_log_call_entry(uint32_t func_addr) { + if (g_cbs.log_call_entry) g_cbs.log_call_entry(func_addr); +} +/* ----------------------------------------------------------------------- */ + +""" + +def patch_generated_c(src: str, load_addr: int, size: int) -> str: + """ + Post-process psxrecomp-game's _full.c output for standalone DLL compilation: + + 1. Prepend the dispatch function-pointer preamble (before any includes). + 2. Remove forward declarations for functions outside the overlay range — + they'd be unresolved externals in the DLL. + 3. Replace direct calls to out-of-range func_XXXXXXXX(cpu) with + call_by_address(cpu, 0xXXXXXXXXu) so they go through the dispatch ptr. + 4. Export all func_XXXXXXXX symbols so the runtime can enumerate them. + """ + ov_lo = load_addr & 0x1FFFFFFF + ov_hi = ov_lo + size + + def in_overlay(addr: int) -> bool: + phys = addr & 0x1FFFFFFF + return ov_lo <= phys < ov_hi + + # 1. Insert preamble AFTER the last #include line (so CPUState is complete) + last_inc = -1 + for m in re.finditer(r'^#include\s+[<"].*[>"]\s*$', src, re.MULTILINE): + last_inc = m.end() + if last_inc == -1: + src = DISPATCH_PREAMBLE + src + else: + src = src[:last_inc] + '\n' + DISPATCH_PREAMBLE + src[last_inc:] + + # 2. Remove out-of-range forward declarations + def drop_extern(m): + addr = int(m.group(1), 16) + return '' if not in_overlay(addr) else m.group(0) + src = re.sub(r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\);\n', + drop_extern, src, flags=re.MULTILINE) + + # 3. Replace out-of-range direct calls with call_by_address + def fix_call(m): + addr = int(m.group(1), 16) + if in_overlay(addr): + return m.group(0) + return f'call_by_address(cpu, 0x{addr:08X}u)' + src = re.sub(r'\bfunc_([0-9A-Fa-f]{8})\(cpu\)', + fix_call, src) + + # 4. Add dllexport to every in-overlay func_XXXXXXXX definition. + # psxrecomp-game emits "void func_XXXXXXXX(CPUState* cpu)" on one line + # with "{" on the NEXT line — match the signature line alone (no ";" = definition). + def add_export(m): + addr = int(m.group(1), 16) + if not in_overlay(addr): + return m.group(0) + return ( + '#ifdef _WIN32\n__declspec(dllexport)\n#else\n' + '__attribute__((visibility("default")))\n#endif\n' + + m.group(0) + ) + src = re.sub(r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\)$', + add_export, src, flags=re.MULTILINE) + + return src + + +# --------------------------------------------------------------------------- +# Static (B-2) post-processing +# --------------------------------------------------------------------------- + +STATIC_PREAMBLE = """\ +/* ---- Static overlay (B-2): psx_runtime.h already provides call_by_address. */ + +""" + +def patch_generated_c_static(src: str, load_addr: int, size: int) -> tuple: + """ + Post-process psxrecomp-game's _full.c output for static binary compilation. + + Returns (patched_src, sorted_list_of_in_overlay_virt_addrs). + + Differences from DLL path: + - No overlay_api.h / OverlayCallbacks / overlay_init + - Callbacks are direct extern calls, not function pointers + - No __declspec(dllexport) — all functions have normal external linkage + - Returns function address list so caller can build the switch dispatch + """ + ov_lo = load_addr & 0x1FFFFFFF + ov_hi = ov_lo + size + + def in_overlay(addr: int) -> bool: + return ov_lo <= (addr & 0x1FFFFFFF) < ov_hi + + # 1. Insert preamble after last #include + last_inc = -1 + for m in re.finditer(r'^#include\s+[<"].*[>"]\s*$', src, re.MULTILINE): + last_inc = m.end() + if last_inc == -1: + src = STATIC_PREAMBLE + src + else: + src = src[:last_inc] + '\n' + STATIC_PREAMBLE + src[last_inc:] + + # 2. Remove out-of-range forward declarations + def drop_extern(m): + addr = int(m.group(1), 16) + return '' if not in_overlay(addr) else m.group(0) + src = re.sub(r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\);\n', + drop_extern, src, flags=re.MULTILINE) + + # 3. Replace out-of-range direct calls with call_by_address + def fix_call(m): + addr = int(m.group(1), 16) + if in_overlay(addr): + return m.group(0) + return f'call_by_address(cpu, 0x{addr:08X}u)' + src = re.sub(r'\bfunc_([0-9A-Fa-f]{8})\(cpu\)', fix_call, src) + + # 4. Collect in-overlay function definition addresses (no export annotation needed) + func_virt_addrs = [] + def collect_fn(m): + addr = int(m.group(1), 16) + if in_overlay(addr): + func_virt_addrs.append(0x80000000 | (addr & 0x1FFFFFFF)) + return m.group(0) + src = re.sub(r'^void func_([0-9A-Fa-f]{8})\(CPUState\* cpu\)$', + collect_fn, src, flags=re.MULTILINE) + + return src, sorted(func_virt_addrs) + + +def generate_overlay_dispatch(all_virt_addrs: list) -> str: + """Generate psx_overlay_dispatch() covering all statically-compiled overlays.""" + lines = [ + '', + '/* Auto-generated overlay dispatch — do not edit. */', + 'int psx_overlay_dispatch(CPUState *cpu, uint32_t addr) {', + ' switch (addr) {', + ] + for va in sorted(all_virt_addrs): + lines.append(f' case 0x{va:08X}u: func_{va:08X}(cpu); return 1;') + lines += [ + ' default: return 0;', + ' }', + '}', + '', + ] + return '\n'.join(lines) + + +# --------------------------------------------------------------------------- +# DLL compilation +# --------------------------------------------------------------------------- + +def compile_dll(c_path: str, out_dll: str, include_dirs: list[str], + gcc: str = 'gcc') -> bool: + import platform + includes = [f'-I{d}' for d in include_dirs] + # On Windows, DLLs use PE relocations — -fPIC triggers GCC CRT init + # that conflicts with the host process. Use -shared without -fPIC. + pic_flag = [] if platform.system() == 'Windows' else ['-fPIC'] + cmd = [ + gcc, '-shared', *pic_flag, '-O2', + c_path, + '-o', out_dll, + *includes, + '-lm', + ] + print(f' compile: {" ".join(cmd)}') + r = subprocess.run(cmd, capture_output=True, text=True) + if r.returncode != 0: + print(f' COMPILE ERROR:\n{r.stderr}') + return False + return True + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--captures', required=True, + help='overlay_captures.json from the runtime') + ap.add_argument('--game-toml', required=True, + help='game.toml (for game_id)') + ap.add_argument('--recompiler', required=True, + help='path to psxrecomp-game.exe') + ap.add_argument('--runtime-include', required=True, + help='path to psxrecomp runtime/include dir') + ap.add_argument('--out-dir', default='build-dev/cache', + help='cache root dir (default: build-dev/cache)') + ap.add_argument('--gcc', default='gcc', + help='GCC binary (default: gcc)') + ap.add_argument('--force', action='store_true', + help='recompile even if output already exists') + ap.add_argument('--static', action='store_true', + help='B-2 mode: compile into binary (overlays_static.c) instead of DLL') + args = ap.parse_args() + + # Read game ID from game.toml (strip BOM if present) + with open(args.game_toml, 'rb') as f: + raw = f.read().lstrip(b'\xef\xbb\xbf') # UTF-8 BOM + toml = tomllib.loads(raw.decode('utf-8')) + game_id = toml.get('game', {}).get('id', 'UNKNOWN') + print(f'Game ID: {game_id}') + + if args.static: + static_out = os.path.join(args.out_dir, 'overlays_static.c') + if os.path.exists(static_out) and not args.force: + print(f'SKIP: {static_out} already exists (use --force to recompile)') + return + os.makedirs(args.out_dir, exist_ok=True) + else: + cache_dir = os.path.join(args.out_dir, game_id) + os.makedirs(cache_dir, exist_ok=True) + + with open(args.captures) as f: + captures = json.load(f) + + print(f'Captures: {len(captures)} overlay(s) to process\n') + + # B-2 static mode: accumulate all overlay C into one file + static_parts = [] # list of (patched_src, func_virt_addrs) + + for cap in captures: + load_addr = int(cap['load_addr'], 16) + size = int(cap['size']) + data = base64.b64decode(cap['bytes_b64']) + crc32 = binascii.crc32(data) & 0xFFFFFFFF + phys_addr = (load_addr & 0x1FFFFFFF) + seeds, seed_audit = classify_overlay_seeds(cap, data, load_addr, size, + crc32, toml) + + if not args.static: + dll_path = os.path.join(cache_dir, f'{phys_addr:08X}_{crc32:08X}.dll') + + print(f'Overlay load=0x{load_addr:08X} size={size} crc32=0x{crc32:08X}') + if args.static: + print(f' seeds: {len(seeds)} mode: static -> {static_out}') + else: + print(f' seeds: {len(seeds)} dll: {dll_path}') + print_seed_audit(seed_audit) + + if not seeds: + print(' SKIP: no seeds (data-only region)\n') + continue + + if not args.static and os.path.exists(dll_path) and not args.force: + print(' SKIP: DLL already exists (use --force to recompile)\n') + continue + + with tempfile.TemporaryDirectory() as tmp: + # Write fake PS-EXE + entry_pc = int(seeds[0], 16) + psx_path = os.path.join(tmp, f'overlay_{load_addr:08X}.psx') + with open(psx_path, 'wb') as f: + f.write(make_psxexe(load_addr, entry_pc, data)) + + # Write seeds file + seeds_path = os.path.join(tmp, 'seeds.txt') + with open(seeds_path, 'w') as f: + for s in seeds: + f.write(s + '\n') + + out_dir_tmp = os.path.join(tmp, 'out') + os.makedirs(out_dir_tmp) + + # Run psxrecomp-game + cmd = [args.recompiler, psx_path, + '--seeds', seeds_path, + '--out-dir', out_dir_tmp, + '--overlay-exact'] + print(f' recompile: {args.recompiler} ...') + toml_dir = os.path.dirname(os.path.abspath(args.game_toml)) + r = subprocess.run(cmd, capture_output=True, text=True, + cwd=toml_dir) + if r.returncode != 0: + print(f' RECOMPILER ERROR:\n{r.stderr or r.stdout}') + continue + + # Find the generated _full.c + stem = os.path.basename(psx_path) + full_c = os.path.join(out_dir_tmp, stem + '_full.c') + if not os.path.exists(full_c): + # psxrecomp-game uses filename() not stem() + candidates = [f for f in os.listdir(out_dir_tmp) if f.endswith('_full.c')] + if not candidates: + print(f' ERROR: no _full.c in {out_dir_tmp}') + continue + full_c = os.path.join(out_dir_tmp, candidates[0]) + + with open(full_c) as f: + src = f.read() + + # Post-process + if args.static: + src, func_addrs = patch_generated_c_static(src, load_addr, size) + c_audit = audit_generated_c(src, load_addr, size, crc32, toml) + print_generated_c_audit(load_addr, size, crc32, c_audit) + if c_audit['unknown_bad'] or c_audit['unsupported_todo_addrs']: + print(' GENERATED-C AUDIT FAILED\n') + continue + static_parts.append((src, func_addrs)) + print(f' recompiled: {len(func_addrs)} functions\n') + else: + src = patch_generated_c(src, load_addr, size) + c_audit = audit_generated_c(src, load_addr, size, crc32, toml) + print_generated_c_audit(load_addr, size, crc32, c_audit) + if c_audit['unknown_bad'] or c_audit['unsupported_todo_addrs']: + print(' GENERATED-C AUDIT FAILED\n') + continue + patched_c = os.path.join(tmp, 'overlay_patched.c') + with open(patched_c, 'w') as f: + f.write(src) + # Save debug copy for inspection + debug_c = os.path.join(os.path.dirname(dll_path), + f'{crc32:08X}_patched.c') + with open(debug_c, 'w') as f: + f.write(src) + + # Compile to DLL + include_dirs = [args.runtime_include] + recomp_root = os.path.dirname(os.path.dirname(args.recompiler)) + for lib_inc in ['lib/fmt/include']: + p = os.path.join(recomp_root, lib_inc) + if os.path.isdir(p): + include_dirs.append(p) + + success = compile_dll(patched_c, dll_path, include_dirs, + gcc=args.gcc) + if success: + print(f' OK -> {dll_path}\n') + else: + print(f' FAILED\n') + + # B-2: write combined static C file + if args.static and static_parts: + all_addrs = [] + combined = '/* Auto-generated overlay dispatch — do not edit.\n' + combined += ' * Rebuild: python3 psxrecomp/tools/compile_overlays.py --static ...\n' + combined += ' */\n' + for src, addrs in static_parts: + combined += src + all_addrs.extend(addrs) + combined += generate_overlay_dispatch(all_addrs) + with open(static_out, 'w') as f: + f.write(combined) + print(f'Static output: {static_out} ({len(all_addrs)} functions total)') + + print('Done.') + + +if __name__ == '__main__': + main() diff --git a/tools/dump_overlays.py b/tools/dump_overlays.py index c4301e7d..769fafb2 100644 --- a/tools/dump_overlays.py +++ b/tools/dump_overlays.py @@ -2,13 +2,15 @@ """ dump_overlays.py — Extract executed overlay regions from a live runtime. -Queries the overlay_dump TCP command which: - 1. Scans the dirty_ram bitmap for contiguous dirty pages above --lo - 2. Writes each region as .bin in --dir - 3. Returns a JSON manifest +Queries overlay_dump (dirty_ram bitmap scan) and cd_read_log (CD DMA +transfer ring) to produce overlay_map.jsonl entries with: + crc32 — CRC32 of the RAM bytes (for verification) + load_addr — physical RAM address where the overlay lives + size — byte count of the region + start_lba — disc LBA of the first sector of this overlay file -Then writes logs//overlay_map.jsonl from the manifest so the -build tool (B-2) has (load_addr, size, crc32, bytes_file) for each overlay. +The start_lba lets extract_overlays.py seek directly to the correct disc +position without brute-force scanning the whole disc image. Usage: python3 tools/dump_overlays.py [--port PORT] [--lo PHYS] [--dir DIR] [--out JSONL] @@ -41,32 +43,72 @@ def send_cmd(port, cmd, **kwargs): s.close() return json.loads(data.decode()) +def find_lba_for_region(cd_log, phys_addr, size): + """Return the disc start_lba for an overlay region if known, else -1. + + Groups consecutive CD DMA entries (same setloc_lba, dest advancing by + sector size) and returns the lba of the group whose dest range covers + phys_addr for at least `size` bytes. + """ + SECTOR = 2048 + entries = cd_log.get('entries', []) + if not entries: + return -1 + + # Build file-load groups: consecutive entries where dest advances by 2048 + # and lba is the same (all sectors of one file share the SetLoc LBA). + groups = [] + cur = None + for e in entries: + dest = int(e['dest'], 16) + lba = e['lba'] + sz = e['size'] + if cur and cur['lba'] == lba and dest == cur['end']: + cur['end'] += sz + cur['total'] += sz + else: + if cur: + groups.append(cur) + cur = {'lba': lba, 'start': dest, 'end': dest + sz, 'total': sz} + if cur: + groups.append(cur) + + # Find a group whose start <= phys_addr and end >= phys_addr + size + for g in groups: + if g['start'] <= phys_addr and g['end'] >= phys_addr + size: + # LBA offset for the exact start of the overlay within this file + offset_sectors = (phys_addr - g['start']) // SECTOR + return g['lba'] + offset_sectors + # Partial overlap — return the group start LBA as best guess + if g['start'] <= phys_addr < g['end']: + offset_sectors = (phys_addr - g['start']) // SECTOR + return g['lba'] + offset_sectors + return -1 + def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument('--port', type=int, default=4470) - ap.add_argument('--lo', default='0x98000', - help='Low physical address threshold (default: 0x98000)') - ap.add_argument('--dir', default='logs/SCUS-94236/overlays', - help='Directory for .bin files (created if missing)') - ap.add_argument('--out', default='logs/SCUS-94236/overlay_map.jsonl', - help='Output JSONL manifest path') + ap.add_argument('--lo', default='0x98000') + ap.add_argument('--dir', default='logs/SCUS-94236/overlays') + ap.add_argument('--out', default='logs/SCUS-94236/overlay_map.jsonl') args = ap.parse_args() os.makedirs(args.dir, exist_ok=True) os.makedirs(os.path.dirname(args.out) or '.', exist_ok=True) - print(f'Querying overlay_dump on port {args.port} (lo={args.lo}, dir={args.dir})...') + print('Querying overlay_dump...') result = send_cmd(args.port, 'overlay_dump', lo=args.lo, dir=args.dir) - if not result.get('ok'): - print(f'ERROR: {result.get("error")}') - sys.exit(1) - + print(f'ERROR: {result.get("error")}'); sys.exit(1) regions = result.get('regions', []) - print(f'Found {len(regions)} overlay region(s).') - # Read existing JSONL to avoid duplicate entries + print('Querying cd_read_log...') + cd_log = send_cmd(args.port, 'cd_read_log', tail=4096) + + print(f'Found {len(regions)} overlay region(s), {cd_log.get("total", 0)} CD DMA entries') + + # Load existing manifest to avoid duplicates existing = set() if os.path.exists(args.out): with open(args.out, encoding='utf-8') as f: @@ -81,19 +123,25 @@ def main(): with open(args.out, 'a', encoding='utf-8') as f: for r in regions: key = (r['crc32'], r['addr']) + phys_addr = int(r['addr'], 16) + + lba = find_lba_for_region(cd_log, phys_addr, r['size']) + lba_str = f'0x{lba:X}' if lba >= 0 else 'unknown' + if key in existing: - print(f' skip {r["addr"]} {r["size"]} bytes {r["crc32"]} (already logged)') + print(f' skip {r["addr"]} {r["size"]:>8} bytes lba={lba_str} (already logged)') continue + entry = { 'crc32': r['crc32'], 'load_addr': r['addr'], 'size': r['size'], - 'bytes_file': f'overlays/{r["file"]}', + 'start_lba': lba_str, } f.write(json.dumps(entry) + '\n') existing.add(key) new_count += 1 - print(f' new {r["addr"]} {r["size"]} bytes {r["crc32"]} -> {r["file"]}') + print(f' new {r["addr"]} {r["size"]:>8} bytes lba={lba_str} crc32={r["crc32"]}') print(f'\n{new_count} new overlay(s) appended to {args.out}') diff --git a/tools/extract_overlays.py b/tools/extract_overlays.py new file mode 100644 index 00000000..0ad3804a --- /dev/null +++ b/tools/extract_overlays.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python3 +""" +extract_overlays.py — Scan a PSX disc image for overlay byte regions. + +Reads overlay_map.jsonl (committed metadata: crc32, load_addr, size). +Scans the disc via ISO 9660 file enumeration, computing CRC32 of each +file's bytes. When a file matches an overlay entry, writes its bytes to +overlays/.bin for the recompiler (B-2). + +Falls back to sector-by-sector scan for any overlays not found via ISO. + +Supports: + BIN/CUE MODE2/2352 — standard PSX disc rip (2352 bytes/sector) + ISO 2048 bytes/sector — less common but handled + +Usage: + python3 tools/extract_overlays.py \\ + --disc \\ + --manifest \\ + --out + +Outputs: + /.bin for each matched overlay +""" + +import sys +import os +import re +import struct +import argparse +import binascii + +# --------------------------------------------------------------------------- +# Disc sector reader +# --------------------------------------------------------------------------- + +SECTOR_RAW = 2352 # MODE2/2352 +SECTOR_DATA = 2048 # user data per sector +DATA_OFFSET = 24 # offset of user data in raw MODE2 sector + +class DiscReader: + """Reads user-data sectors from a PSX BIN/CUE or ISO image.""" + + def __init__(self, bin_path, raw=True): + self.f = open(bin_path, 'rb') + self.raw = raw # True = MODE2/2352, False = 2048-byte ISO + self.sec_sz = SECTOR_RAW if raw else SECTOR_DATA + fsize = os.path.getsize(bin_path) + self.total = fsize // self.sec_sz + + def close(self): + self.f.close() + + def read_sector_data(self, lba): + """Return 2048 user-data bytes for the given LBA.""" + offset = lba * self.sec_sz + self.f.seek(offset) + raw = self.f.read(self.sec_sz) + if len(raw) < self.sec_sz: + return b'\x00' * SECTOR_DATA + if self.raw: + return raw[DATA_OFFSET: DATA_OFFSET + SECTOR_DATA] + return raw[:SECTOR_DATA] + + def read_file_bytes(self, lba, size): + """Read exactly `size` bytes starting at `lba`.""" + sectors_needed = (size + SECTOR_DATA - 1) // SECTOR_DATA + data = b'' + for i in range(sectors_needed): + data += self.read_sector_data(lba + i) + return data[:size] + +# --------------------------------------------------------------------------- +# CUE parser — extracts BIN path and track mode +# --------------------------------------------------------------------------- + +def parse_cue(cue_path): + """Return (bin_path, is_raw) from a .cue file.""" + cue_dir = os.path.dirname(os.path.abspath(cue_path)) + bin_path = None + is_raw = True # default: MODE2/2352 + with open(cue_path, encoding='utf-8', errors='replace') as f: + for line in f: + m = re.match(r'\s*FILE\s+"([^"]+)"\s+BINARY', line, re.IGNORECASE) + if m: + bin_path = os.path.join(cue_dir, m.group(1)) + m = re.match(r'\s*TRACK\s+\d+\s+(MODE\d/(\d+))', line, re.IGNORECASE) + if m: + is_raw = (int(m.group(2)) == SECTOR_RAW) + if bin_path is None: + raise ValueError(f'No BINARY FILE found in {cue_path}') + return bin_path, is_raw + +# --------------------------------------------------------------------------- +# ISO 9660 directory walker +# --------------------------------------------------------------------------- + +def _read_pvd(disc, lba=16): + """Return root dir LBA and size from the Primary Volume Descriptor.""" + data = disc.read_sector_data(lba) + if data[0] != 1 or data[1:6] != b'CD001': + raise ValueError('Sector 16 is not an ISO 9660 PVD') + root = data[156:190] + root_lba = struct.unpack_from(' len(raw): + break + rec = raw[pos: pos + rec_len] + flags = rec[25] + is_dir = bool(flags & 0x02) + elba = struct.unpack_from(' {out_name}') + else: + print(f' MISMATCH lba={lba} expected=0x{crc:08X}' + f' got=0x{actual_crc:08X} (will try ISO/brute-force)') + + # --- Pass 1: ISO 9660 file scan (fast) --- + print('Pass 1: ISO 9660 file enumeration...') + try: + for path, lba, size in enumerate_files(disc): + if not remaining: + break + # Try exact file size and also overlay-truncated size + for try_size in sorted({size, *[o['size'] for o in remaining.values()]}): + if try_size > size + SECTOR_DATA: + continue # file too small to cover this overlay + data = disc.read_file_bytes(lba, try_size) + crc = crc32_of(data[:try_size]) + if crc in remaining: + o = remaining.pop(crc) + out_name = f'{crc:08X}.bin' + out_path = os.path.join(args.out, out_name) + with open(out_path, 'wb') as f: + f.write(data[:o['size']]) + found[crc] = (path, lba, size) + print(f' FOUND {path:40s} load=0x{o["load_addr"]:08X}' + f' size={o["size"]} -> {out_name}') + break + except Exception as e: + print(f' ISO scan error: {e}') + + # --- Pass 2: brute-force sector scan for anything not found --- + if remaining: + print() + print(f'Pass 2: brute-force sector scan for {len(remaining)} unfound overlay(s)...') + for ov_crc, ov in list(remaining.items()): + sz = ov['size'] + nsec = (sz + SECTOR_DATA - 1) // SECTOR_DATA + print(f' Scanning for crc32=0x{ov_crc:08X} size={sz} ({nsec} sectors)...', + flush=True) + found_it = False + for lba in range(disc.total - nsec + 1): + data = disc.read_file_bytes(lba, sz) + if crc32_of(data) == ov_crc: + remaining.pop(ov_crc) + out_name = f'{ov_crc:08X}.bin' + out_path = os.path.join(args.out, out_name) + with open(out_path, 'wb') as f: + f.write(data) + found[ov_crc] = (f'lba:{lba}', lba, sz) + print(f' FOUND lba={lba} load=0x{ov["load_addr"]:08X}' + f' size={sz} -> {out_name}') + found_it = True + break + if not found_it: + print(f' NOT FOUND') + + disc.close() + + print() + print(f'Results: {len(found)} found {len(remaining)} not found') + if remaining: + print('Not found (play through more content and re-run dump_overlays.py):') + for crc, o in remaining.items(): + print(f' 0x{crc:08X} load=0x{o["load_addr"]:08X} size={o["size"]}') + else: + print('All overlays extracted successfully.') + +if __name__ == '__main__': + main()