mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-26 02:11:36 -04:00
overlay: restore overlay DLL cache + capture (faithful port of codex/cache-discovery)
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline overlay compilation pipeline that were built on the pruned codex/cache-discovery branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt rather than history-restored. Runtime (off by default, opt in via [runtime] overlay_cache = true): - overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(), enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter. - overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and writes overlay_captures.json (execution-verified seeds) at clean exit. - dma.c: capture + cd_dma_log hook at CD DMA completion (async model). - cdrom.c: cdrom_get_setloc_lba() for disc-position correlation. - dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks. - debug_server.c: cd_read_log + overlay_loader_status TCP commands. - config_loader: [runtime] overlay_cache flag (default false). Recompiler (compile_overlays.py / extract_overlays.py drive these): - main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL targets and set split_mid_function_targets=false, so overlay basic-block targets are NOT split into bogus standalone functions (the mid-function-seed failure mode that produced black-on-load / blue-on-new-game). - function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the mid-function split pre-pass behind split_mid_function_targets (default true, so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs). Deltas from the original: save states dropped (never worked); overlay enable is runtime-config-gated (off by default) instead of a build-time define; loader stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3). Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
@@ -0,0 +1,111 @@
|
||||
# PSXRecomp Overlay System — Feature Reference
|
||||
|
||||
## Overview
|
||||
|
||||
PSX games use **overlays**: regions of game code that are loaded from disc into a fixed
|
||||
RAM window at runtime via CD DMA. The same physical addresses are reused by different
|
||||
overlay images throughout the game. The static recompiler only sees the boot EXE; it
|
||||
cannot see overlay code that does not exist at compile time.
|
||||
|
||||
The overlay system has two layers, B and A, that together solve this.
|
||||
|
||||
---
|
||||
|
||||
## Layer B — Capture and static compilation (IMPLEMENTED, B-1 done / B-2 in progress)
|
||||
|
||||
### B-1: Overlay capture (runtime, done)
|
||||
|
||||
`overlay_capture.c` hooks every CD DMA completion into game-code RAM
|
||||
(`dest < 0x1C0000`). On each unique load address, it stores the DMA-time bytes
|
||||
(unpatched copy, taken before the game can modify them) in an in-process set.
|
||||
|
||||
On clean process exit, `overlay_capture_write_json()` writes
|
||||
`build-dev/overlay_captures.json` alongside the executable. The JSON contains:
|
||||
|
||||
- `load_addr` — virtual address where the overlay loaded (e.g. `0x800E7000`)
|
||||
- `size` — byte count of the full dirty region
|
||||
- `bytes_b64` — base64 of the DMA-time image (zero-filled for scatter-load gaps)
|
||||
- `seeds` — per-PC interpreter hit addresses (execution-verified entry points)
|
||||
|
||||
The `seeds` list is how we know which addresses inside the overlay are real function
|
||||
entry points. They come from `dirty_ram_interp`'s PC table: addresses the interpreter
|
||||
actually branched to during gameplay.
|
||||
|
||||
### B-2: Static overlay compilation (build-time, in progress)
|
||||
|
||||
`compile_overlays.py` reads `overlay_captures.json`, wraps the bytes in a fake PS-EXE
|
||||
header, runs `psxrecomp-game` to recompile, post-processes the generated C for
|
||||
standalone compilation, and produces a C file that is compiled into the binary.
|
||||
|
||||
In **static mode** (B-2), the output is a `.c` file placed in `generated/overlays.c`
|
||||
that the CMake build includes directly. No DLL, no runtime loading, no CRC matching.
|
||||
The overlay functions join the static dispatch table exactly like game EXE functions.
|
||||
|
||||
`game.toml` gains an `[[overlays]]` section that points to the captures file; CMake
|
||||
includes `generated/overlays.c` when it exists.
|
||||
|
||||
---
|
||||
|
||||
## Layer A — Dynamic DLL loading (runtime, SHELVED)
|
||||
|
||||
### What it was supposed to do
|
||||
|
||||
A-1 was the "live" complement to B: while B produces a binary rebuild, A-1 lets a
|
||||
user contribute a new overlay by dropping a compiled DLL into a cache directory.
|
||||
The runtime detects the overlay firing in `dirty_ram_dispatch`, finds the matching
|
||||
DLL in the cache, loads it with `LoadLibrary`, and wires the overlay functions into
|
||||
a dynamic dispatch hash table — all without a rebuild.
|
||||
|
||||
The idea was that contributors could run `compile_overlays.py`, drop the DLL in
|
||||
`build-dev/cache/<game-id>/`, and immediately have compiled overlay code.
|
||||
|
||||
### What we built
|
||||
|
||||
- `overlay_loader.c` — dynamic dispatch hash table (16 384 slots), DLL load + PE
|
||||
export enumeration, `overlay_loader_dispatch` called from `dirty_ram_dispatch`
|
||||
- `overlay_api.h` — shared ABI: `OverlayCallbacks` struct, `overlay_init` entry point
|
||||
- `compile_overlays.py` DLL path — generates `{addr}_{crc}.dll` in the cache dir
|
||||
- `overlay_loader_status` TCP debug command
|
||||
|
||||
### Why it was shelved
|
||||
|
||||
**Load Game regression.** When a DLL exists in the cache, loading a save triggers
|
||||
the overlay dispatch, which loads the DLL, which runs the compiled overlay code.
|
||||
This crashed consistently. Clearing the DLL cache restores stable behavior. We never
|
||||
confirmed whether the compiled code itself is correct (the crash analysis was blocked
|
||||
by the plumbing issues below).
|
||||
|
||||
**CRC key instability.** We tried three different naming schemes before landing on
|
||||
`{addr}_{dma_crc}.dll`:
|
||||
|
||||
1. CRC of live RAM — wrong: scatter-load gaps contain old data in live RAM but zeros
|
||||
in the DMA-assembled image, so the CRC never matched.
|
||||
2. Address-only — no collision safety.
|
||||
3. DMA-CRC (from `overlay_capture_get_region_crc`) — correct in theory, but the DMA
|
||||
blocks are not in `s_entries` after a fast-boot restore (no DMA fired), so the
|
||||
function hashed a zeroed image and produced the wrong CRC.
|
||||
|
||||
**Final fix attempted:** scan the cache dir at init for `{addr}_*.dll` files and
|
||||
build a `region_start → path` index. This avoids recomputing the CRC at dispatch
|
||||
time. The strlen check in the scanner had an off-by-one (22 vs 21) that was fixed,
|
||||
but the Load Game crash was not diagnosed before shelving.
|
||||
|
||||
### What remains for A-1 to ship
|
||||
|
||||
1. Confirm the generated DLL code is actually correct (same question as B-2).
|
||||
Debug with `B476006F_patched.c` in `build-dev/cache/SCUS-94236/`.
|
||||
2. Understand why Load Game crashes with the DLL active.
|
||||
3. Possibly audit `patch_generated_c` in `compile_overlays.py` for incorrect
|
||||
cross-call patching or missing extern declarations.
|
||||
|
||||
The infrastructure is complete. The blocker is code correctness, not plumbing.
|
||||
|
||||
---
|
||||
|
||||
## Tomba overlay map (known)
|
||||
|
||||
| Region | Load addr | Size | Seeds | Status |
|
||||
|----------------|--------------|---------|-------|---------------------|
|
||||
| `000E7000` | `0x800E7000` | 364 544 | 66 | Captured, DLL built, shelved |
|
||||
|
||||
More overlays will appear as gameplay progresses past Dwarf Village.
|
||||
+86
-79
@@ -1,7 +1,6 @@
|
||||
# Overlay Discovery
|
||||
|
||||
**Date:** 2026-05-29
|
||||
**Branch:** `overlay-discovery`
|
||||
**Updated:** 2026-05-30
|
||||
**Phase gate:** Phase 5+ (game EXE loaded, dirty-RAM hits observed in game address space)
|
||||
|
||||
---
|
||||
@@ -24,133 +23,141 @@ interpretation of overlay code.
|
||||
BIOS dirty-RAM (Rule 18 in CLAUDE.md) is code the BIOS *assembles in-place* at boot — e.g. the
|
||||
4-instruction SIO handler written to RAM 0xCF0. That code never existed on disc.
|
||||
|
||||
Game overlays are different: they are complete compiled MIPS modules on the disc filesystem,
|
||||
loaded to fixed RAM addresses. Their bytes are known ahead of time; the recompiler just hasn't
|
||||
seen them yet. This system fixes that.
|
||||
Game overlays are different: they are compiled MIPS modules loaded to fixed RAM addresses. Their
|
||||
bytes are known after they land in RAM; the recompiler just hasn't seen them yet. This system
|
||||
fixes that.
|
||||
|
||||
Missed heuristics (static-EXE functions the recompiler failed to identify) are orthogonal: they
|
||||
exist in the main EXE at build time and will not match any disc file hash. The overlay system
|
||||
ignores them and they fall through to dirty-RAM as today. Fix missed heuristics by improving
|
||||
heuristics; fix overlays by this system.
|
||||
exist in the main EXE at build time. Fix missed heuristics by improving heuristics; fix overlays
|
||||
by this system.
|
||||
|
||||
---
|
||||
|
||||
## Why disc extraction is not used
|
||||
|
||||
The naive approach — scan the disc image for overlay bytes by CRC — does not work for Tomba and
|
||||
many other PSX titles. Investigation confirmed that Tomba stores overlays in a **scatter-load
|
||||
format**: each disc sector is a `(12-byte block descriptor, 2048-byte data block)` pair, where
|
||||
the descriptor tells the loader which RAM address to write the data block to. The resulting RAM
|
||||
content is not present anywhere on disc as a contiguous byte range, so no CRC or brute-force scan
|
||||
can find it.
|
||||
|
||||
The universal correct primitive is to capture bytes **at DMA time** — intercepting them as they
|
||||
land in RAM, after all scatter-reassembly and decompression. This works for:
|
||||
|
||||
- Scatter-load formats (Tomba, others)
|
||||
- Game-specific LZ or RLE compression (bytes arrive decompressed via DMA)
|
||||
- Direct file loads (trivially captured)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
Two layers, composing. As Layer B grows, Layer A fires less. Eventually Layer A goes silent.
|
||||
Layer B is a manifest of Layer A's output: the same `(load_addr, bytes, seeds)` data, committed
|
||||
to the repo and baked into the binary ahead of time.
|
||||
|
||||
### Layer B — Persistent map → static precompilation
|
||||
### Layer B — Build-time pre-compilation
|
||||
|
||||
**Runtime side:** whenever CD DMA completes a load into RAM, record `(disc_file_hash, load_addr,
|
||||
size)` to a per-game log file on disk. This is cheap — a hash computation and a file append.
|
||||
**Runtime side (capture):** on every CD DMA completion into game-code RAM, the runtime appends
|
||||
`(load_addr, size, bytes)` to a ring buffer. On process exit it writes
|
||||
`overlay_captures.json` automatically — no user action required. The user discovers the file
|
||||
exists and can attach it to a GitHub issue.
|
||||
|
||||
**Build-time side:** `recompiler/recomp.py` reads the log file. For each entry, it locates the
|
||||
matching file on the disc image by hash, disassembles it at the recorded load address, and
|
||||
emits a compiled C function set alongside the main EXE output. The overlay's functions are added
|
||||
to the static dispatch table at their load addresses.
|
||||
**Build-time side:** `psxrecomp-game` reads `game.toml [[overlays]]` entries, each pointing to
|
||||
a captured `.bin` file and a list of seed entry points. It disassembles the MIPS bytes at
|
||||
`load_addr`, emits C functions, and adds them to the static dispatch table. The shipped binary
|
||||
has those overlays pre-compiled at zero runtime cost.
|
||||
|
||||
**Result:** any overlay logged in a previous run is statically compiled into the next build. No
|
||||
runtime overhead on subsequent loads.
|
||||
**Coverage grows** with every release as developer playthroughs and user contributions feed new
|
||||
captures into the build.
|
||||
|
||||
### Layer A — Runtime background compilation with persistent cache
|
||||
|
||||
**On CD DMA completion:**
|
||||
1. Hash the written bytes.
|
||||
2. Check the on-disk compiled cache (`cache/<SCUS_ID>/<hash>.dll`).
|
||||
- Cache hit → `LoadLibrary` the cached DLL, register its functions in the live dispatch table
|
||||
for `[load_addr, load_addr + size)`. No interpretation ever needed.
|
||||
- Cache miss → fall through to dirty-RAM interpretation. Queue a background thread to compile
|
||||
this overlay (emit C for the bytes at `load_addr`, invoke the C compiler, write the DLL to
|
||||
cache). On the *next* load of this overlay (next room visit, next level reload), the cache
|
||||
hit path fires instead.
|
||||
**On CD DMA completion into game-code RAM:**
|
||||
|
||||
**Result:** within a single playthrough, each overlay compiles once on first encounter. Every
|
||||
subsequent encounter in that session, and in all future sessions, hits the cache.
|
||||
1. Hash the written bytes.
|
||||
2. Check `cache/<SCUS_ID>/<hash>.dll`.
|
||||
- **Cache hit** → `LoadLibrary` the DLL, register its functions in the live dispatch table
|
||||
for `[load_addr, load_addr + size)`. Interpreter never fires for this overlay.
|
||||
- **Cache miss** → fall through to dirty-RAM interpretation. Enqueue a background thread to
|
||||
compile this overlay: emit C for the bytes at `load_addr`, invoke the C compiler, write the
|
||||
DLL to cache. On the *next* load of this overlay the cache hit fires.
|
||||
|
||||
**Result:** within a single session, each new overlay is interpreted once and compiled once.
|
||||
Every subsequent encounter — same session or future sessions — dispatches to native code.
|
||||
|
||||
A user who plays through the entire game accumulates a complete cache. From that point forward
|
||||
the game runs entirely as compiled native code, never touching the interpreter.
|
||||
|
||||
### How they compose
|
||||
|
||||
```
|
||||
Dispatch path (on branch to overlay address):
|
||||
1. Static dispatch table (Layer B) → precompiled function [zero runtime cost]
|
||||
2. Compiled cache (Layer A) → LoadLibrary'd function [one-time per session after first]
|
||||
3. Dirty-RAM interpreter → fallback [first-ever encounter only]
|
||||
Dispatch (branch to overlay address):
|
||||
1. Static dispatch table [Layer B] → precompiled function zero runtime cost
|
||||
2. Compiled cache [Layer A] → LoadLibrary'd function one-time per session after first
|
||||
3. Dirty-RAM interpreter → fallback first-ever encounter only
|
||||
```
|
||||
|
||||
Layer B coverage grows with every rebuild. Layer A covers the gap within a session before the
|
||||
rebuild happens. Dirty-RAM is the last-resort fallback that fires exactly once per undiscovered
|
||||
overlay.
|
||||
rebuild happens. Dirty-RAM fires exactly once per undiscovered overlay.
|
||||
|
||||
---
|
||||
|
||||
## Game identification
|
||||
|
||||
Cache and log files are namespaced by SCUS/SLUS/SCES product code, read from `SYSTEM.CNF` on the
|
||||
disc image at startup (`BOOT=cdrom:\SCUS_941.60;1` → key `SCUS-94160`).
|
||||
Cache and capture files are namespaced by SCUS/SLUS/SCES product code, read from `SYSTEM.CNF`:
|
||||
|
||||
```
|
||||
cache/
|
||||
SCUS-94160/ ← Tomba (NTSC-U)
|
||||
SCUS-94236/ ← Tomba (NTSC-U)
|
||||
<hash>.dll
|
||||
<hash>.dll
|
||||
SCES-01234/ ← different region, separate cache
|
||||
...
|
||||
|
||||
logs/
|
||||
SCUS-94160/
|
||||
overlay_map.jsonl ← append-only: {hash, load_addr, size, disc_file}
|
||||
SCUS-94236/
|
||||
overlay_map.jsonl
|
||||
overlay_captures.json ← written at exit, next to the executable
|
||||
```
|
||||
|
||||
Different regional versions get separate caches automatically, which is correct since overlay
|
||||
load addresses may differ between regions.
|
||||
|
||||
---
|
||||
|
||||
## Cache invalidation
|
||||
|
||||
A cached DLL is keyed by the hash of the overlay bytes. The hash IS the identity. If a disc file
|
||||
changes (different dump, different region), the hash changes and the old cache entry is simply
|
||||
never matched — it sits unused. No explicit invalidation needed.
|
||||
A cached DLL is keyed by the hash of the overlay bytes. The hash IS the identity. If a disc dump
|
||||
differs between users or regions, the hash differs and the old cache entry is simply never
|
||||
matched. No explicit invalidation needed.
|
||||
|
||||
When Layer B statically compiles an overlay, Layer A's cached DLL for that hash becomes
|
||||
unreachable (static dispatch wins at step 1). It can be pruned post-build but is benign if left.
|
||||
|
||||
---
|
||||
|
||||
## Integration point
|
||||
|
||||
The natural hook is the CD-ROM DMA completion callback in `runtime/src/cdrom.c`. This is where
|
||||
bytes land in RAM. After the DMA write:
|
||||
1. Compute hash of the destination range.
|
||||
2. Log `(hash, load_addr, size, source_file)` for Layer B.
|
||||
3. Check Layer A cache. If hit, register the DLL's functions in the live dispatch table.
|
||||
4. If miss, mark the range dirty (existing dirty-RAM path) and enqueue background compilation.
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
| Phase | Work |
|
||||
|---|---|
|
||||
| B-1 | CD DMA completion hook logs `(hash, load_addr, size)` to `logs/<SCUS_ID>/overlay_map.jsonl` |
|
||||
| B-2 | `recomp.py` reads the log and emits overlay functions alongside the main EXE; adds entries to dispatch table |
|
||||
| B-3 | Build-time validation: for each log entry, verify disc file hash matches before compiling |
|
||||
| A-1 | On DMA completion, check `cache/<SCUS_ID>/<hash>.dll`; if present, `LoadLibrary` + register |
|
||||
| A-2 | Background thread: emit C for newly-seen overlay, compile to DLL, write to cache |
|
||||
| A-3 | On next load of same overlay, Layer A cache hit fires instead of dirty-RAM |
|
||||
| Phase | Work | Status |
|
||||
|---|---|---|
|
||||
| — | Dirty-RAM interpreter (Layer A last-resort fallback) | Done |
|
||||
| B-1 | CD DMA byte capture ring (always-on, retail-safe, `dest < 0x1C0000` filter) | To build |
|
||||
| B-1 | Automatic `overlay_captures.json` written at process exit | To build |
|
||||
| B-2 | `game.toml [[overlays]]` + `psxrecomp-game` overlay compilation | To build |
|
||||
| B-3 | Contribution ingestion script: JSON → `.bin` + `game.toml` entries | To build |
|
||||
| A-1 | On DMA completion, check cache DLL; if present, `LoadLibrary` + register | To build |
|
||||
| A-2 | Background thread: emit C, compile to DLL, write to cache | To build |
|
||||
| A-3 | On next overlay encounter, A-1 fires instead of dirty-RAM | Follows from A-1/A-2 |
|
||||
|
||||
B-1 through B-3 can ship before A-1. The system is useful with only Layer B (just requires a
|
||||
rebuild after each new playthrough). Layer A is the quality-of-life layer that eliminates the
|
||||
"rebuild required" friction.
|
||||
B-1 through B-3 ship before A-1. The system is useful with only Layer B (requires a rebuild
|
||||
after each new playthrough contribution). Layer A eliminates the "rebuild required" friction and
|
||||
makes the game self-healing for any user.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
## Scatter-load discovery (Tomba)
|
||||
|
||||
- Does Tomba use named disc files (ISO 9660) or raw LBA sector reads for overlay loads? Named
|
||||
files make Layer B trivial (hash the file, record the name). Raw LBA reads require tracing the
|
||||
sector number through to the ISO to recover the file identity. Determine by inspecting
|
||||
dirty-RAM dispatch miss addresses: if they cluster at 2-3 fixed load addresses, it's classic
|
||||
file-based overlays. If they scatter, it may be more ad-hoc.
|
||||
Tomba's CD DMA log (captured 2026-05-30) confirmed the scatter-load format:
|
||||
|
||||
- `LoadLibrary` / `dlopen` infrastructure does not currently exist in the runtime. Layer A
|
||||
requires it. Layer B does not. Implement B first; defer A until B has real coverage data.
|
||||
- Every sector pair: `12-byte block descriptor` (always to a header buffer at `0x973C0`) +
|
||||
`2048-byte data block` (scattered to the address the descriptor specifies).
|
||||
- A single ISO file load produces DMA entries with non-contiguous destination addresses.
|
||||
- The 12-byte descriptor encodes the block's destination; the loader dispatches accordingly.
|
||||
|
||||
This format is game-specific but the DMA capture approach handles it transparently: the captured
|
||||
bytes are the post-scatter RAM contents, which is exactly the execution form the recompiler needs.
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
# Overlay System — Implementation Plan
|
||||
|
||||
**Written:** 2026-05-30
|
||||
**Status:** Active
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Make PSX overlay code transparent to the user. Shipped builds run pre-compiled
|
||||
overlays at native speed (Layer B). Unknown overlays fall back to the interpreter
|
||||
once, compile in the background, and run as native code from the second encounter
|
||||
onward (Layer A). Users contribute new coverage automatically by playing.
|
||||
|
||||
---
|
||||
|
||||
## Data structures
|
||||
|
||||
### Capture set
|
||||
|
||||
A hash map keyed by `load_addr`. Write-once: a given load address is inserted
|
||||
exactly once regardless of how many times that overlay loads during a session.
|
||||
|
||||
```c
|
||||
typedef enum {
|
||||
OVERLAY_STATE_QUEUED = 0, /* in compilation queue, not yet started */
|
||||
OVERLAY_STATE_COMPILING = 1, /* background thread is working on it */
|
||||
OVERLAY_STATE_COMPILED = 2, /* DLL written to cache, dispatch patched */
|
||||
} OverlayState;
|
||||
|
||||
typedef struct OverlayEntry {
|
||||
uint32_t load_addr;
|
||||
uint32_t size;
|
||||
uint8_t *bytes; /* heap copy of the DMA'd bytes */
|
||||
OverlayState state; /* atomic, guards queue dedup */
|
||||
struct OverlayEntry *next; /* intrusive hash chain */
|
||||
} OverlayEntry;
|
||||
```
|
||||
|
||||
### Compilation queue
|
||||
|
||||
A simple FIFO of `OverlayEntry *` pointers into the capture set. Does not own
|
||||
data. Background thread drains it. Never contains duplicates — the state machine
|
||||
on the capture set entry prevents duplicate pushes.
|
||||
|
||||
---
|
||||
|
||||
## State machine (per entry)
|
||||
|
||||
```
|
||||
DMA completion (first time, load_addr not in set)
|
||||
→ alloc entry, copy bytes
|
||||
→ state = QUEUED
|
||||
→ push pointer onto queue
|
||||
|
||||
DMA completion (load_addr already in set)
|
||||
→ state == QUEUED → no-op (already waiting to compile)
|
||||
→ state == COMPILING → no-op (being compiled now, interp handles it)
|
||||
→ state == COMPILED → no-op (dispatch table has it, interp won't fire)
|
||||
|
||||
Background thread (dequeues entry)
|
||||
→ state = COMPILING
|
||||
→ emit C for bytes at load_addr
|
||||
→ compile to DLL → write cache/<game_id>/<hash>.dll
|
||||
→ LoadLibrary + register functions in live dispatch table
|
||||
→ state = COMPILED
|
||||
```
|
||||
|
||||
The `NEW→QUEUED` transition and the queue push are done under a single mutex
|
||||
lock so a concurrent DMA completion on a second thread cannot push the same
|
||||
entry twice.
|
||||
|
||||
---
|
||||
|
||||
## Implementation phases
|
||||
|
||||
### B-1 — Capture + JSON dump *(prerequisite for everything)*
|
||||
|
||||
**Files:** `runtime/src/overlay_capture.c` + `runtime/include/overlay_capture.h`
|
||||
|
||||
- `overlay_capture_init(const char *game_id)` — called at game handoff
|
||||
- `overlay_capture_on_dma(uint32_t load_addr, uint32_t size, const uint8_t *bytes)` — called from `dma.c` `execute_ch3_cdrom` when `load_start < 0x1C0000`
|
||||
- `overlay_capture_write_json(const char *path)` — writes `overlay_captures.json` next to the executable; called from `main.cpp` at clean exit
|
||||
- JSON schema per entry:
|
||||
```json
|
||||
{ "load_addr": "0xA3000", "size": 208896, "bytes_b64": "...", "seeds": [] }
|
||||
```
|
||||
(`seeds` is empty at capture time; filled in by the ingestion script from per_pc data)
|
||||
|
||||
**Hook in `dma.c`:** existing `execute_ch3_cdrom` already has `load_start` and
|
||||
`total_words`. Add one call after the existing cd_dma_log push:
|
||||
```c
|
||||
if (!step && load_start < 0x1C0000u)
|
||||
overlay_capture_on_dma(load_start, total_words * 4u, ram + load_start);
|
||||
```
|
||||
|
||||
**Hook in `main.cpp`:** at clean shutdown (SDL quit, window close) call
|
||||
`overlay_capture_write_json`. Retail and dev builds both do this.
|
||||
|
||||
---
|
||||
|
||||
### B-2 — psxrecomp-game overlay compilation
|
||||
|
||||
**Files:** `recompiler/src/overlay_compiler.cpp`, extended `game.toml` schema
|
||||
|
||||
`game.toml` gains an `[[overlays]]` array:
|
||||
```toml
|
||||
[[overlays]]
|
||||
load_addr = "0xA3000"
|
||||
capture = "captures/A3000.bin"
|
||||
seeds = ["0xA3140", "0xA52C0"]
|
||||
```
|
||||
|
||||
`psxrecomp-game` reads each entry, disassembles the MIPS bytes at `load_addr`,
|
||||
traces from seeds, emits C functions, and appends to the game's dispatch table.
|
||||
These functions are linked into the shipped binary at zero runtime cost.
|
||||
|
||||
---
|
||||
|
||||
### B-3 — Ingestion script
|
||||
|
||||
**File:** `tools/ingest_overlays.py`
|
||||
|
||||
```
|
||||
python3 tools/ingest_overlays.py \
|
||||
--captures overlay_captures.json \
|
||||
--per-pc logs/SCUS-94236/per_pc.json \
|
||||
--out-dir captures/ \
|
||||
--toml game.toml
|
||||
```
|
||||
|
||||
- Reads `overlay_captures.json` (from user submission or own playthrough)
|
||||
- Reads `per_pc.json` for executed addresses within each overlay range
|
||||
- Writes `captures/<load_addr_hex>.bin`
|
||||
- Appends new `[[overlays]]` blocks to `game.toml` (skips already-present entries)
|
||||
|
||||
After running, `psxrecomp-game --config game.toml` re-emits the dispatch table
|
||||
with the new overlays compiled in.
|
||||
|
||||
---
|
||||
|
||||
### A-1 — Cache DLL check on DMA completion
|
||||
|
||||
**Files:** `runtime/src/overlay_capture.c` (extended)
|
||||
|
||||
On DMA completion, after the capture-set insert:
|
||||
- Compute `hash = crc32(bytes, size)`
|
||||
- Check `cache/<game_id>/<hash>.dll` exists
|
||||
- If yes: `LoadLibrary` → enumerate exported functions → register each in the
|
||||
live dispatch table for `[load_addr, load_addr + size)` → set state COMPILED
|
||||
- If no: proceed to queue for background compilation (A-2)
|
||||
|
||||
---
|
||||
|
||||
### A-2 — Background compiler thread
|
||||
|
||||
**Files:** `runtime/src/overlay_compiler_thread.c`
|
||||
|
||||
A single long-lived thread started at game handoff:
|
||||
|
||||
```
|
||||
loop:
|
||||
wait on queue semaphore
|
||||
dequeue OverlayEntry*
|
||||
set state = COMPILING
|
||||
emit_c_for_overlay(entry) // calls into recompiler lib
|
||||
compile_c_to_dll(entry) // invokes GCC/Clang subprocess
|
||||
LoadLibrary + register dispatch
|
||||
set state = COMPILED
|
||||
```
|
||||
|
||||
The recompiler is embedded in the runtime as a static library for dev builds.
|
||||
For retail builds, the compiler subprocess path uses the system C compiler if
|
||||
present, or skips to "write JSON only" mode if not (Layer A gracefully degrades
|
||||
to JSON-only on systems without a toolchain).
|
||||
|
||||
---
|
||||
|
||||
## Build order
|
||||
|
||||
1. **B-1** — capture set + JSON dump (unblocks everything)
|
||||
2. **B-2** — `psxrecomp-game` overlay compilation (unblocks B-3)
|
||||
3. **B-3** — ingestion script (completes the contribution loop)
|
||||
4. **A-1** — cache DLL check (fast path for returning users)
|
||||
5. **A-2** — background compiler thread (self-healing on first run)
|
||||
|
||||
B-1 through B-3 are useful independently: they close the developer feedback
|
||||
loop and give users a way to contribute without any runtime compilation.
|
||||
A-1 and A-2 are the quality-of-life layer.
|
||||
|
||||
---
|
||||
|
||||
## What already exists
|
||||
|
||||
| Component | File | Status |
|
||||
|---|---|---|
|
||||
| Dirty-RAM interpreter (last-resort fallback) | `runtime/src/dirty_ram_interp.c` | Done |
|
||||
| CD DMA ring log (`cd_read_log` TCP cmd) | `runtime/src/dma.c` | Done (filter + 65536 ring, 2026-05-30) |
|
||||
| Dispatch table infrastructure | `runtime/src/full_function_emitter.cpp` | Done |
|
||||
| Recompiler core (MIPS→C) | `recompiler/src/` | Done |
|
||||
| `game.toml` config loader | `recompiler/src/config_loader.cpp` | Done |
|
||||
@@ -34,6 +34,7 @@ struct CodeGenConfig {
|
||||
bool emit_line_numbers; // Include original address as comments
|
||||
bool optimize_zero_reg; // Optimize away $zero assignments
|
||||
bool use_switch_for_blocks; // Use switch instead of goto labels
|
||||
bool split_mid_function_targets; // Split branch targets into funcs for legacy main-EXE analysis
|
||||
std::string indent; // Indentation string (default: " ")
|
||||
|
||||
CodeGenConfig()
|
||||
@@ -41,6 +42,7 @@ struct CodeGenConfig {
|
||||
, emit_line_numbers(true)
|
||||
, optimize_zero_reg(true)
|
||||
, use_switch_for_blocks(false)
|
||||
, split_mid_function_targets(true)
|
||||
, indent(" ") {}
|
||||
};
|
||||
|
||||
|
||||
@@ -35,6 +35,11 @@ public:
|
||||
// Scan entire executable for function boundaries
|
||||
FunctionAnalysisResult analyze();
|
||||
|
||||
// Analyze only explicit entry points and direct JAL targets reachable from
|
||||
// them. This is for runtime-loaded overlays where scanning the whole blob
|
||||
// treats data and basic-block targets as standalone functions.
|
||||
FunctionAnalysisResult analyze_exact_entries(const std::vector<uint32_t>& entries);
|
||||
|
||||
// Add a forced entry point address that is treated as a function start
|
||||
// even if it has no standard ADDIU $sp prologue. The function will be
|
||||
// included in the analysis result with has_prologue = false.
|
||||
|
||||
@@ -1573,7 +1573,11 @@ std::string CodeGenerator::generate_file(
|
||||
// hit `psx_unknown_dispatch`. Surfaced by Phase B2 audit. The 16-pass
|
||||
// cap is a safety limit; in practice each pass strictly reduces the
|
||||
// remaining target set so convergence is fast.
|
||||
{
|
||||
//
|
||||
// Overlay exact mode disables this: overlay branch targets are basic-block
|
||||
// labels, not callable function entries, so splitting them into standalone
|
||||
// functions produces broken dispatch — the mid-function-seed failure mode.
|
||||
if (config_.split_mid_function_targets) {
|
||||
std::set<uint32_t> cfgs_to_scan; // Which CFGs to scan (empty = all)
|
||||
uint32_t exe_start = exe_.header.load_address;
|
||||
uint32_t exe_end = exe_.end_address();
|
||||
|
||||
@@ -61,6 +61,9 @@ static RuntimeConfig parse_runtime_block(const toml::value& cfg, const fs::path&
|
||||
if (runtime.contains("fast_boot")) {
|
||||
rt.fast_boot = toml::find<bool>(runtime, "fast_boot");
|
||||
}
|
||||
if (runtime.contains("overlay_cache")) {
|
||||
rt.overlay_cache = toml::find<bool>(runtime, "overlay_cache");
|
||||
}
|
||||
return rt;
|
||||
}
|
||||
|
||||
|
||||
@@ -47,6 +47,12 @@ struct RuntimeConfig {
|
||||
// subsequent launches, skipping BIOS execution entirely. Default off;
|
||||
// enable per-game in [runtime]. Snapshot is keyed on BIOS SHA256 + entry_pc.
|
||||
bool fast_boot = false;
|
||||
|
||||
// overlay_cache: enable the overlay DLL cache + capture (Layer A). Off by
|
||||
// default. When true the runtime scans cache/<game_id>/ for precompiled
|
||||
// overlay DLLs (loaded ahead of the dirty-RAM interpreter) and records
|
||||
// overlay bytes to overlay_captures.json for offline compilation.
|
||||
bool overlay_cache = false;
|
||||
};
|
||||
|
||||
// One entry from [[recompiler.bios_vectors]].
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#include "function_analysis.h"
|
||||
#include <algorithm>
|
||||
#include <map>
|
||||
#include <queue>
|
||||
#include <set>
|
||||
#include <fmt/format.h>
|
||||
|
||||
@@ -217,6 +218,381 @@ bool FunctionAnalyzer::is_likely_data_section(uint32_t start_addr, uint32_t end_
|
||||
return false;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
enum class ExactCfKind {
|
||||
Normal,
|
||||
Branch,
|
||||
Jump,
|
||||
Jal,
|
||||
JrRa,
|
||||
JrOther,
|
||||
Jalr,
|
||||
};
|
||||
|
||||
struct ExactCf {
|
||||
ExactCfKind kind = ExactCfKind::Normal;
|
||||
uint32_t target = 0;
|
||||
};
|
||||
|
||||
struct ExactWalkResult {
|
||||
std::set<uint32_t> visited;
|
||||
std::set<uint32_t> direct_jal_targets;
|
||||
std::set<uint32_t> jump_table_targets;
|
||||
uint32_t jr_ra_count = 0;
|
||||
};
|
||||
|
||||
static bool exact_is_jr_ra_word(uint32_t instr) {
|
||||
return instr == 0x03E00008u;
|
||||
}
|
||||
|
||||
static bool exact_is_addiu_sp_neg(uint32_t instr) {
|
||||
uint32_t opcode = (instr >> 26) & 0x3Fu;
|
||||
uint32_t rs = (instr >> 21) & 0x1Fu;
|
||||
uint32_t rt = (instr >> 16) & 0x1Fu;
|
||||
int16_t imm = static_cast<int16_t>(instr & 0xFFFFu);
|
||||
return opcode == 0x09u && rs == 29u && rt == 29u && imm < 0;
|
||||
}
|
||||
|
||||
static bool exact_is_valid_mips_word(uint32_t instr) {
|
||||
if (instr == 0xFFFFFFFFu || instr == 0xFFFFFFFDu) return false;
|
||||
|
||||
uint32_t opcode = (instr >> 26) & 0x3Fu;
|
||||
uint32_t funct = instr & 0x3Fu;
|
||||
uint32_t rt = (instr >> 16) & 0x1Fu;
|
||||
|
||||
if (opcode == 0x00u) {
|
||||
switch (funct) {
|
||||
case 0x00u: case 0x02u: case 0x03u: case 0x04u:
|
||||
case 0x06u: case 0x07u: case 0x08u: case 0x09u:
|
||||
case 0x0Cu: case 0x0Du:
|
||||
case 0x10u: case 0x11u: case 0x12u: case 0x13u:
|
||||
case 0x18u: case 0x19u: case 0x1Au: case 0x1Bu:
|
||||
case 0x20u: case 0x21u: case 0x22u: case 0x23u:
|
||||
case 0x24u: case 0x25u: case 0x26u: case 0x27u:
|
||||
case 0x2Au: case 0x2Bu:
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
if (opcode == 0x01u) {
|
||||
return rt == 0x00u || rt == 0x01u || rt == 0x10u || rt == 0x11u;
|
||||
}
|
||||
|
||||
switch (opcode) {
|
||||
case 0x02u: case 0x03u: case 0x04u: case 0x05u:
|
||||
case 0x06u: case 0x07u:
|
||||
case 0x08u: case 0x09u: case 0x0Au: case 0x0Bu:
|
||||
case 0x0Cu: case 0x0Du: case 0x0Eu: case 0x0Fu:
|
||||
case 0x10u: case 0x12u:
|
||||
case 0x20u: case 0x21u: case 0x22u: case 0x23u:
|
||||
case 0x24u: case 0x25u: case 0x26u:
|
||||
case 0x28u: case 0x29u: case 0x2Au: case 0x2Bu:
|
||||
case 0x2Eu:
|
||||
case 0x30u: case 0x32u: case 0x38u: case 0x3Au:
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
static uint32_t exact_branch_target(uint32_t pc, uint32_t instr) {
|
||||
int16_t imm = static_cast<int16_t>(instr & 0xFFFFu);
|
||||
return pc + 4u + (static_cast<int32_t>(imm) << 2);
|
||||
}
|
||||
|
||||
static uint32_t exact_jump_target(uint32_t pc, uint32_t instr) {
|
||||
return ((pc + 4u) & 0xF0000000u) | ((instr & 0x03FFFFFFu) << 2);
|
||||
}
|
||||
|
||||
static ExactCf exact_classify_cf(uint32_t pc, uint32_t instr) {
|
||||
ExactCf cf;
|
||||
uint32_t opcode = (instr >> 26) & 0x3Fu;
|
||||
uint32_t funct = instr & 0x3Fu;
|
||||
uint32_t rs = (instr >> 21) & 0x1Fu;
|
||||
|
||||
if (opcode == 0x00u && funct == 0x08u) {
|
||||
cf.kind = (rs == 31u) ? ExactCfKind::JrRa : ExactCfKind::JrOther;
|
||||
return cf;
|
||||
}
|
||||
if (opcode == 0x00u && funct == 0x09u) {
|
||||
cf.kind = ExactCfKind::Jalr;
|
||||
return cf;
|
||||
}
|
||||
if (opcode == 0x02u) {
|
||||
cf.kind = ExactCfKind::Jump;
|
||||
cf.target = exact_jump_target(pc, instr);
|
||||
return cf;
|
||||
}
|
||||
if (opcode == 0x03u) {
|
||||
cf.kind = ExactCfKind::Jal;
|
||||
cf.target = exact_jump_target(pc, instr);
|
||||
return cf;
|
||||
}
|
||||
if (opcode == 0x01u || (opcode >= 0x04u && opcode <= 0x07u) ||
|
||||
(opcode >= 0x14u && opcode <= 0x17u)) {
|
||||
cf.kind = ExactCfKind::Branch;
|
||||
cf.target = exact_branch_target(pc, instr);
|
||||
return cf;
|
||||
}
|
||||
return cf;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(const std::vector<uint32_t>& entries) {
|
||||
FunctionAnalysisResult result;
|
||||
result.total_instructions = 0;
|
||||
result.jr_ra_count = 0;
|
||||
result.prologue_count = 0;
|
||||
result.call_discovered_count = 0;
|
||||
result.state_continuation_count = 0;
|
||||
|
||||
fmt::print("\n=== Exact Overlay Function Analysis ===\n\n");
|
||||
|
||||
auto in_exe = [&](uint32_t addr) {
|
||||
return addr >= exe_.header.load_address && addr < exe_.end_address() && (addr & 3u) == 0;
|
||||
};
|
||||
|
||||
auto callable_direct_jal_target = [&](uint32_t addr) {
|
||||
auto word_opt = exe_.read_word(addr);
|
||||
if (!word_opt.has_value() || !exact_is_valid_mips_word(*word_opt)) return false;
|
||||
if (addr == exe_.header.load_address) return true;
|
||||
|
||||
if (addr >= exe_.header.load_address + 8u) {
|
||||
auto prev8_opt = exe_.read_word(addr - 8u);
|
||||
if (prev8_opt.has_value() && exact_is_jr_ra_word(*prev8_opt)) return true;
|
||||
}
|
||||
|
||||
if (exact_is_addiu_sp_neg(*word_opt)) {
|
||||
if (addr < exe_.header.load_address + 4u) return true;
|
||||
auto prev_opt = exe_.read_word(addr - 4u);
|
||||
if (!prev_opt.has_value()) return true;
|
||||
return exact_classify_cf(addr - 4u, *prev_opt).kind == ExactCfKind::Normal;
|
||||
}
|
||||
|
||||
return false;
|
||||
};
|
||||
|
||||
auto find_jump_table_targets = [&](uint32_t entry, uint32_t hard_cap,
|
||||
uint32_t jr_pc, uint32_t jr_rs) {
|
||||
std::vector<uint32_t> targets;
|
||||
uint32_t lw_base = 0xFFu;
|
||||
int32_t lw_offset = 0;
|
||||
uint32_t addu_cand[2] = {0xFFu, 0xFFu};
|
||||
uint32_t lui_val = 0;
|
||||
int16_t addiu_val[2] = {0, 0};
|
||||
bool found_addiu[2] = {false, false};
|
||||
bool found_lui = false;
|
||||
uint32_t table_count = 0;
|
||||
|
||||
for (int back = 1; back <= 40; back++) {
|
||||
uint32_t scan_addr = jr_pc - static_cast<uint32_t>(back * 4);
|
||||
if (scan_addr < entry) break;
|
||||
auto scan_opt = exe_.read_word(scan_addr);
|
||||
if (!scan_opt.has_value()) break;
|
||||
|
||||
uint32_t instr = *scan_opt;
|
||||
uint32_t op = (instr >> 26) & 0x3Fu;
|
||||
uint32_t rs = (instr >> 21) & 0x1Fu;
|
||||
uint32_t rt = (instr >> 16) & 0x1Fu;
|
||||
uint32_t rd = (instr >> 11) & 0x1Fu;
|
||||
uint32_t fn = instr & 0x3Fu;
|
||||
|
||||
if (op == 0x23u && rt == jr_rs && lw_base == 0xFFu) {
|
||||
lw_base = rs;
|
||||
lw_offset = static_cast<int32_t>(static_cast<int16_t>(instr & 0xFFFFu));
|
||||
continue;
|
||||
}
|
||||
if (op == 0x00u && fn == 0x21u && rd == lw_base && lw_base != 0xFFu &&
|
||||
addu_cand[0] == 0xFFu) {
|
||||
addu_cand[0] = rs;
|
||||
addu_cand[1] = rt;
|
||||
continue;
|
||||
}
|
||||
if (op == 0x09u && addu_cand[0] != 0xFFu) {
|
||||
for (int c = 0; c < 2; c++) {
|
||||
if (!found_addiu[c] && addu_cand[c] != 0xFFu &&
|
||||
rs == addu_cand[c] && rt == addu_cand[c]) {
|
||||
addiu_val[c] = static_cast<int16_t>(instr & 0xFFFFu);
|
||||
found_addiu[c] = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (op == 0x0Fu && addu_cand[0] != 0xFFu && !found_lui) {
|
||||
for (int c = 0; c < 2; c++) {
|
||||
if (addu_cand[c] != 0xFFu && rt == addu_cand[c]) {
|
||||
lui_val = (instr & 0xFFFFu) << 16;
|
||||
if (!found_addiu[c]) addiu_val[c] = 0;
|
||||
addiu_val[0] = addiu_val[c];
|
||||
found_addiu[0] = found_addiu[c];
|
||||
found_lui = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (op == 0x0Bu && table_count == 0) {
|
||||
table_count = instr & 0xFFFFu;
|
||||
continue;
|
||||
}
|
||||
if (found_lui && table_count != 0) break;
|
||||
}
|
||||
|
||||
if (!found_lui || table_count == 0 || table_count >= 512) return targets;
|
||||
|
||||
uint32_t table_base = lui_val +
|
||||
(found_addiu[0] ? static_cast<uint32_t>(static_cast<int32_t>(addiu_val[0])) : 0u) +
|
||||
static_cast<uint32_t>(lw_offset);
|
||||
for (uint32_t i = 0; i < table_count; i++) {
|
||||
auto entry_opt = exe_.read_word(table_base + i * 4u);
|
||||
if (!entry_opt.has_value()) continue;
|
||||
uint32_t target = *entry_opt;
|
||||
if (target >= entry && target < hard_cap && in_exe(target)) {
|
||||
targets.push_back(target);
|
||||
}
|
||||
}
|
||||
return targets;
|
||||
};
|
||||
|
||||
auto walk = [&](uint32_t entry, uint32_t hard_cap) {
|
||||
ExactWalkResult wr;
|
||||
std::queue<uint32_t> work;
|
||||
work.push(entry);
|
||||
|
||||
auto in_function = [&](uint32_t addr) {
|
||||
return in_exe(addr) && addr >= entry && addr < hard_cap;
|
||||
};
|
||||
|
||||
while (!work.empty()) {
|
||||
uint32_t pc = work.front();
|
||||
work.pop();
|
||||
|
||||
if (wr.visited.count(pc) || !in_function(pc)) continue;
|
||||
auto word_opt = exe_.read_word(pc);
|
||||
if (!word_opt.has_value()) continue;
|
||||
|
||||
uint32_t instr = *word_opt;
|
||||
wr.visited.insert(pc);
|
||||
ExactCf cf = exact_classify_cf(pc, instr);
|
||||
uint32_t delay = pc + 4u;
|
||||
|
||||
switch (cf.kind) {
|
||||
case ExactCfKind::Normal:
|
||||
if (in_function(pc + 4u)) work.push(pc + 4u);
|
||||
break;
|
||||
case ExactCfKind::Branch:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
if (in_function(pc + 8u)) work.push(pc + 8u);
|
||||
if (in_function(cf.target)) work.push(cf.target);
|
||||
break;
|
||||
case ExactCfKind::Jump:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
if (in_function(cf.target)) work.push(cf.target);
|
||||
break;
|
||||
case ExactCfKind::Jal:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
if (in_function(pc + 8u)) work.push(pc + 8u);
|
||||
if (in_exe(cf.target) && callable_direct_jal_target(cf.target)) {
|
||||
wr.direct_jal_targets.insert(cf.target);
|
||||
}
|
||||
break;
|
||||
case ExactCfKind::Jalr:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
if (in_function(pc + 8u)) work.push(pc + 8u);
|
||||
break;
|
||||
case ExactCfKind::JrRa:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
wr.jr_ra_count++;
|
||||
break;
|
||||
case ExactCfKind::JrOther:
|
||||
if (in_function(delay)) wr.visited.insert(delay);
|
||||
{
|
||||
uint32_t jr_rs = (instr >> 21) & 0x1Fu;
|
||||
for (uint32_t target : find_jump_table_targets(entry, hard_cap, pc, jr_rs)) {
|
||||
wr.jump_table_targets.insert(target);
|
||||
if (in_function(target)) work.push(target);
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
return wr;
|
||||
};
|
||||
|
||||
std::set<uint32_t> known_entries;
|
||||
std::queue<uint32_t> pending;
|
||||
for (uint32_t entry : entries) {
|
||||
if (!in_exe(entry)) continue;
|
||||
if (known_entries.insert(entry).second) pending.push(entry);
|
||||
}
|
||||
|
||||
size_t explicit_count = known_entries.size();
|
||||
fmt::print("Explicit entries: {}\n", explicit_count);
|
||||
|
||||
std::set<uint32_t> processed;
|
||||
while (!pending.empty()) {
|
||||
uint32_t entry = pending.front();
|
||||
pending.pop();
|
||||
if (processed.count(entry)) continue;
|
||||
processed.insert(entry);
|
||||
|
||||
auto next_it = known_entries.upper_bound(entry);
|
||||
uint32_t hard_cap = (next_it != known_entries.end()) ? *next_it : exe_.end_address();
|
||||
ExactWalkResult wr = walk(entry, hard_cap);
|
||||
|
||||
for (uint32_t target : wr.direct_jal_targets) {
|
||||
if (known_entries.insert(target).second) pending.push(target);
|
||||
}
|
||||
}
|
||||
|
||||
result.call_discovered_count = static_cast<int>(known_entries.size() - explicit_count);
|
||||
fmt::print("Direct-JAL entries: {}\n", result.call_discovered_count);
|
||||
fmt::print("Total exact entries: {}\n\n", known_entries.size());
|
||||
|
||||
std::vector<uint32_t> starts_vec(known_entries.begin(), known_entries.end());
|
||||
std::sort(starts_vec.begin(), starts_vec.end());
|
||||
|
||||
for (size_t i = 0; i < starts_vec.size(); i++) {
|
||||
uint32_t entry = starts_vec[i];
|
||||
uint32_t hard_cap = (i + 1 < starts_vec.size()) ? starts_vec[i + 1] : exe_.end_address();
|
||||
ExactWalkResult wr = walk(entry, hard_cap);
|
||||
if (wr.visited.empty()) continue;
|
||||
|
||||
Function func;
|
||||
func.start_addr = entry;
|
||||
func.end_addr = *wr.visited.rbegin() + 4u;
|
||||
if (func.end_addr > hard_cap) func.end_addr = hard_cap;
|
||||
if (func.end_addr <= func.start_addr) func.end_addr = func.start_addr + 4u;
|
||||
func.size = func.end_addr - func.start_addr;
|
||||
func.name = fmt::format("func_{:08X}", func.start_addr);
|
||||
|
||||
auto first_instr = exe_.read_word(func.start_addr);
|
||||
if (first_instr.has_value()) {
|
||||
int32_t stack_size;
|
||||
func.has_prologue = is_prologue(*first_instr, stack_size);
|
||||
func.stack_frame_size = func.has_prologue ? stack_size : 0;
|
||||
if (func.has_prologue) result.prologue_count++;
|
||||
} else {
|
||||
func.has_prologue = false;
|
||||
func.stack_frame_size = 0;
|
||||
}
|
||||
|
||||
func.has_epilogue = wr.jr_ra_count > 0;
|
||||
func.is_data_section = false;
|
||||
|
||||
result.jr_ra_count += static_cast<int>(wr.jr_ra_count);
|
||||
result.total_instructions += static_cast<int>(wr.visited.size());
|
||||
result.functions.push_back(func);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
FunctionAnalysisResult FunctionAnalyzer::analyze() {
|
||||
FunctionAnalysisResult result;
|
||||
result.total_instructions = 0;
|
||||
|
||||
@@ -40,6 +40,7 @@ int main(int argc, char** argv) {
|
||||
std::string extra_funcs_storage; // lifetime anchor for the .c_str() below
|
||||
const char* extra_funcs_path = nullptr;
|
||||
bool inspect_mode = false;
|
||||
bool overlay_exact_mode = false;
|
||||
std::filesystem::path out_dir = "generated";
|
||||
|
||||
if (!config_path.empty()) {
|
||||
@@ -72,6 +73,8 @@ int main(int argc, char** argv) {
|
||||
* with psxrecomp-bios and project scripts. */
|
||||
} else if (arg == "--inspect") {
|
||||
inspect_mode = true;
|
||||
} else if (arg == "--overlay-exact") {
|
||||
overlay_exact_mode = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -197,7 +200,11 @@ int main(int argc, char** argv) {
|
||||
fmt::print("Performing function analysis...\n");
|
||||
|
||||
PSXRecomp::FunctionAnalyzer analyzer(*exe);
|
||||
analyzer.add_forced_entry(exe->header.initial_pc);
|
||||
std::vector<uint32_t> exact_entries;
|
||||
exact_entries.push_back(exe->header.initial_pc);
|
||||
if (!overlay_exact_mode) {
|
||||
analyzer.add_forced_entry(exe->header.initial_pc);
|
||||
}
|
||||
|
||||
// Forced entry points: functions called from the dispatch table but not
|
||||
// automatically detected by the heuristic-based function analysis.
|
||||
@@ -213,7 +220,10 @@ int main(int argc, char** argv) {
|
||||
if (line.empty() || line[0] == '#') continue;
|
||||
uint32_t addr = (uint32_t)std::strtoul(line.c_str(), nullptr, 16);
|
||||
if (addr >= 0x80010000u && addr < 0x80200000u) {
|
||||
analyzer.add_forced_entry(addr);
|
||||
exact_entries.push_back(addr);
|
||||
if (!overlay_exact_mode) {
|
||||
analyzer.add_forced_entry(addr);
|
||||
}
|
||||
extra_count++;
|
||||
}
|
||||
}
|
||||
@@ -223,7 +233,9 @@ int main(int argc, char** argv) {
|
||||
}
|
||||
}
|
||||
|
||||
auto analysis_result = analyzer.analyze();
|
||||
auto analysis_result = overlay_exact_mode
|
||||
? analyzer.analyze_exact_entries(exact_entries)
|
||||
: analyzer.analyze();
|
||||
|
||||
// Print summary statistics
|
||||
fmt::print("\n=== Function Analysis Summary ===\n\n");
|
||||
@@ -344,6 +356,7 @@ int main(int argc, char** argv) {
|
||||
PSXRecomp::CodeGenConfig codegen_config;
|
||||
codegen_config.emit_comments = true;
|
||||
codegen_config.emit_line_numbers = true;
|
||||
codegen_config.split_mid_function_targets = !overlay_exact_mode;
|
||||
|
||||
// Load per-game annotations: annotations/<exe_stem>_annotations.csv
|
||||
// Silently skipped if the file doesn't exist.
|
||||
|
||||
@@ -15,6 +15,9 @@ void cdrom_set_speed(int divisor);
|
||||
void cdrom_set_game_speed(int divisor);
|
||||
/* Called by fntrace on first game-range dispatch; switches to game speed. */
|
||||
void cdrom_notify_game_started(void);
|
||||
/* LBA of the most recent SetLoc command (-1 if none yet). Used by the DMA
|
||||
* ring buffer to correlate each sector transfer with its disc position. */
|
||||
int cdrom_get_setloc_lba(void);
|
||||
|
||||
/* MMIO read/write (0x1F801800-0x1F801803) */
|
||||
uint32_t cdrom_read(uint32_t addr);
|
||||
|
||||
@@ -56,13 +56,16 @@ extern const char *g_dirty_ram_last_unsupported_reason;
|
||||
* stubs actually fire — a single noisy spurious-dispatch site can mask
|
||||
* a legitimate handler that never runs. This open-addressed table keys
|
||||
* on the entry PC of each interpreted block. */
|
||||
#define DIRTY_RAM_PC_TABLE_SIZE 64
|
||||
#define DIRTY_RAM_PC_TABLE_SIZE 65536
|
||||
typedef struct {
|
||||
uint32_t pc; /* entry PC, 0 = empty slot */
|
||||
uint64_t hits; /* number of times dispatched here */
|
||||
uint64_t insns; /* total instructions executed across hits */
|
||||
} DirtyRamPcEntry;
|
||||
extern DirtyRamPcEntry g_dirty_ram_pc_table[DIRTY_RAM_PC_TABLE_SIZE];
|
||||
/* Companion table: every PC the interpreter actually executes (not just block
|
||||
* entries). overlay_capture uses both to report execution-verified seeds. */
|
||||
extern DirtyRamPcEntry g_dirty_ram_exec_pc_table[DIRTY_RAM_PC_TABLE_SIZE];
|
||||
|
||||
/* Block-entry ring buffer. Records every dispatch into dirty RAM with the
|
||||
* caller's RA at entry, plus argument context — answers
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
#ifndef OVERLAY_API_H
|
||||
#define OVERLAY_API_H
|
||||
|
||||
/* Shared ABI between psx-runtime.exe and overlay DLLs.
|
||||
*
|
||||
* After LoadLibrary, the runtime calls overlay_init() with this struct.
|
||||
* The DLL stores the pointers and routes all cross-module calls through them.
|
||||
*/
|
||||
|
||||
#include "cpu_state.h"
|
||||
#include <stdint.h>
|
||||
|
||||
typedef struct {
|
||||
/* Core dispatch: routes call_by_address() and out-of-overlay jal */
|
||||
void (*dispatch_call)(CPUState *cpu, uint32_t addr, uint32_t ra);
|
||||
/* Interrupt check: called after every function return in overlay */
|
||||
void (*check_interrupts)(CPUState *cpu);
|
||||
/* GTE coprocessor 2 execution */
|
||||
void (*gte_execute)(CPUState *cpu, uint32_t cmd);
|
||||
/* MIPS syscall (break/syscall instructions) */
|
||||
void (*psx_syscall)(CPUState *cpu, uint32_t code);
|
||||
/* Unresolved dispatch target */
|
||||
void (*psx_unknown_dispatch)(CPUState *cpu, uint32_t addr, uint32_t phys);
|
||||
/* Debug instrumentation: called at every function entry (may be NULL) */
|
||||
void (*log_call_entry)(uint32_t func_addr);
|
||||
} OverlayCallbacks;
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* Exported by every overlay DLL. Call once after LoadLibrary. */
|
||||
#ifdef _WIN32
|
||||
__declspec(dllexport)
|
||||
#else
|
||||
__attribute__((visibility("default")))
|
||||
#endif
|
||||
void overlay_init(const OverlayCallbacks *cbs);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* OVERLAY_API_H */
|
||||
@@ -0,0 +1,62 @@
|
||||
#ifndef OVERLAY_CAPTURE_H
|
||||
#define OVERLAY_CAPTURE_H
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* overlay_capture — B-1 implementation of the overlay capture set.
|
||||
*
|
||||
* On every CD DMA completion into game-code RAM (dest < 0x1C0000), the runtime
|
||||
* calls overlay_capture_on_dma(). Each unique load_addr is stored exactly once
|
||||
* in a write-once set. At clean process exit, overlay_capture_write_json()
|
||||
* writes overlay_captures.json next to the executable for user contribution.
|
||||
*
|
||||
* The set never overwrites: a second DMA to the same load_addr is a no-op.
|
||||
* This is correct because PSX overlays always load the same bytes to the same
|
||||
* address; recording once is sufficient.
|
||||
*/
|
||||
|
||||
/* Call once at startup (before the run loop) with the exe directory.
|
||||
* Sets the output path for overlay_captures.json. Capture is not yet
|
||||
* active — it activates automatically on the first post-game DMA transfer. */
|
||||
void overlay_capture_set_out_dir(const char *out_dir);
|
||||
|
||||
/* Legacy init alias kept for compatibility; same as set_out_dir. */
|
||||
void overlay_capture_init(const char *out_dir);
|
||||
|
||||
/* Enable/disable overlay capture. Off by default; main() enables it only when
|
||||
* the overlay cache is turned on in [runtime] config. When disabled,
|
||||
* overlay_capture_on_dma() and overlay_capture_write_json() are no-ops. */
|
||||
void overlay_capture_set_enabled(int enabled);
|
||||
|
||||
/* Call from dma.c execute_ch3_cdrom() after every forward CH3 transfer with
|
||||
* load_start < 0x1C0000 and fntrace_is_game_started().
|
||||
* load_addr is the physical RAM destination, size is byte count,
|
||||
* bytes points to the RAM buffer at that address (already written by DMA). */
|
||||
void overlay_capture_on_dma(uint32_t load_addr, uint32_t size,
|
||||
const uint8_t *bytes);
|
||||
|
||||
/* Write overlay_captures.json to the directory supplied at init time.
|
||||
* Safe to call even if no overlays were captured (writes nothing).
|
||||
* Called from shutdown_runtime() in main.cpp. */
|
||||
void overlay_capture_write_json(void);
|
||||
|
||||
/* Returns number of unique overlays captured so far. */
|
||||
int overlay_capture_count(void);
|
||||
|
||||
/* Compute CRC32 of the DMA-time bytes for the region [region_start,
|
||||
* region_start+region_size). Zero-filled for gaps between DMA blocks,
|
||||
* exactly as overlay_captures.json bytes_b64 is assembled. Use this
|
||||
* in overlay_loader to build a consistent DLL filename without reading
|
||||
* live RAM (which has scatter-load gap contamination). */
|
||||
uint32_t overlay_capture_get_region_crc(uint32_t region_start,
|
||||
uint32_t region_size);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* OVERLAY_CAPTURE_H */
|
||||
@@ -0,0 +1,58 @@
|
||||
#ifndef OVERLAY_LOADER_H
|
||||
#define OVERLAY_LOADER_H
|
||||
|
||||
/* overlay_loader — A-1 runtime overlay DLL cache and dynamic dispatch.
|
||||
*
|
||||
* On CD DMA completion (overlay_capture_on_dma), the loader checks whether
|
||||
* a cached DLL exists for the overlay bytes. If it does, it LoadLibrary's
|
||||
* the DLL, calls overlay_init() to wire callbacks, enumerates func_XXXXXXXX
|
||||
* exports, and registers each in the dynamic dispatch table.
|
||||
*
|
||||
* dirty_ram_dispatch calls overlay_loader_dispatch() before the interpreter,
|
||||
* so compiled overlay functions get priority over interpretation.
|
||||
*/
|
||||
|
||||
#include "cpu_state.h"
|
||||
#include <stdint.h>
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
/* Called at game handoff to set the cache root directory and game ID.
|
||||
* cache_dir: absolute path to the cache root (e.g. "build-dev/cache")
|
||||
* game_id: product code (e.g. "SCUS-94236") */
|
||||
void overlay_loader_init(const char *cache_dir, const char *game_id);
|
||||
|
||||
/* Called from overlay_capture_on_dma after the capture-set insert.
|
||||
* Computes CRC32 of bytes, checks cache, loads DLL if present.
|
||||
* load_addr: physical RAM address, size/bytes: the transferred data. */
|
||||
void overlay_loader_check_cache(uint32_t load_addr, uint32_t size,
|
||||
const uint8_t *bytes);
|
||||
|
||||
/* Called from dirty_ram_dispatch before the interpreter.
|
||||
* Returns 1 and calls the compiled function if addr is registered,
|
||||
* 0 if not found (fall through to interpreter). */
|
||||
int overlay_loader_dispatch(CPUState *cpu, uint32_t addr);
|
||||
|
||||
/* Returns number of functions currently registered in the dynamic table. */
|
||||
int overlay_loader_registered_count(void);
|
||||
|
||||
/* Returns full loader state for TCP diagnostics. */
|
||||
void overlay_loader_get_status(int *active, int *registered,
|
||||
int *regions_checked,
|
||||
char *cache_dir_out, int cache_dir_len,
|
||||
char *game_id_out, int game_id_len,
|
||||
uint32_t *checked_out, int checked_max,
|
||||
int *checked_written,
|
||||
uint32_t *last_crc_out, int *last_file_found_out);
|
||||
|
||||
/* Most recent loader event string (DLL load success/failure). Surfaced via
|
||||
* the overlay_loader_status TCP command — no stderr logging (Rule 3). */
|
||||
const char *overlay_loader_last_msg(void);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif /* OVERLAY_LOADER_H */
|
||||
@@ -92,6 +92,8 @@ set(PSXRECOMP_RUNTIME_SOURCES
|
||||
${PSXRECOMP_ROOT}/runtime/src/starvation_ring.c
|
||||
${PSXRECOMP_ROOT}/runtime/src/card_read_summary.c
|
||||
${PSXRECOMP_ROOT}/runtime/src/card_data_writes.c
|
||||
${PSXRECOMP_ROOT}/runtime/src/overlay_capture.c
|
||||
${PSXRECOMP_ROOT}/runtime/src/overlay_loader.c
|
||||
${PSXRECOMP_ROOT}/recompiler/src/config_loader.cpp
|
||||
)
|
||||
|
||||
@@ -112,6 +114,7 @@ function(psxrecomp_add_runtime_target target)
|
||||
set(oneValueArgs
|
||||
GAME_GENERATED_FULL_C
|
||||
GAME_GENERATED_DISPATCH_C
|
||||
GAME_OVERLAY_STATIC_C
|
||||
DEBUG_PORT
|
||||
WINDOW_TITLE
|
||||
DEFAULT_BIOS_PATH
|
||||
@@ -147,6 +150,13 @@ function(psxrecomp_add_runtime_target target)
|
||||
list(APPEND generated_sources "${PSXRT_GAME_GENERATED_DISPATCH_C}")
|
||||
set(has_game_dispatch TRUE)
|
||||
endif()
|
||||
# Layer B: statically-compiled overlay dispatch. Inert unless a game
|
||||
# provides a generated overlays_static.c — no target sets this yet.
|
||||
if(PSXRT_GAME_OVERLAY_STATIC_C AND EXISTS "${PSXRT_GAME_OVERLAY_STATIC_C}")
|
||||
set_source_files_properties("${PSXRT_GAME_OVERLAY_STATIC_C}" PROPERTIES GENERATED TRUE)
|
||||
list(APPEND generated_sources "${PSXRT_GAME_OVERLAY_STATIC_C}")
|
||||
set(has_overlay_dispatch TRUE)
|
||||
endif()
|
||||
|
||||
if(PSXRT_ORACLE)
|
||||
set(mode_source ${PSXRECOMP_ROOT}/runtime/src/psx_interpreter.c)
|
||||
@@ -201,6 +211,9 @@ function(psxrecomp_add_runtime_target target)
|
||||
if(has_game_dispatch)
|
||||
target_compile_definitions(${target} PRIVATE PSX_HAS_GAME_DISPATCH=1)
|
||||
endif()
|
||||
if(has_overlay_dispatch)
|
||||
target_compile_definitions(${target} PRIVATE PSX_HAS_OVERLAY_DISPATCH=1)
|
||||
endif()
|
||||
|
||||
# PSX_DEBUG_TOOLS option declared at the top of runtime.cmake so it's
|
||||
# also visible to psx-beetle / non-runtime-helper targets.
|
||||
|
||||
@@ -75,6 +75,7 @@ static uint64_t sector_history_seq;
|
||||
|
||||
/* Seek target */
|
||||
static uint8_t seek_min, seek_sec, seek_sect;
|
||||
static int s_setloc_lba = -1; /* LBA captured at SetLoc time */
|
||||
|
||||
/* Read state */
|
||||
static int reading;
|
||||
@@ -138,6 +139,8 @@ void cdrom_notify_game_started(void) {
|
||||
g_disc_speed_divisor = g_game_divisor;
|
||||
}
|
||||
|
||||
int cdrom_get_setloc_lba(void) { return s_setloc_lba; }
|
||||
|
||||
/* Minimum cycles between CD-ROM IRQs in fast modes. Must be enough for the
|
||||
* interrupt handler to save state, check the IRQ flag, process data, and
|
||||
* return. Too low → interrupt fires before the previous one is processed →
|
||||
@@ -659,6 +662,7 @@ static void exec_command(uint8_t cmd) {
|
||||
seek_min = bcd_to_bin(param_fifo[0]);
|
||||
seek_sec = bcd_to_bin(param_fifo[1]);
|
||||
seek_sect = bcd_to_bin(param_fifo[2]);
|
||||
s_setloc_lba = msf_to_lba(seek_min, seek_sec, seek_sect);
|
||||
}
|
||||
response_push(stat_reg);
|
||||
set_irq(CDIRQ_ACK);
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
* so TCP.md and DEBUG.md are reusable across projects.
|
||||
*/
|
||||
#include "debug_server.h"
|
||||
#include "overlay_loader.h"
|
||||
#include "cpu_state.h"
|
||||
#include "dma.h"
|
||||
#include "gpu.h"
|
||||
@@ -7214,6 +7215,88 @@ static void handle_fn_exit_dump(int id, const char *json) {
|
||||
free(buf);
|
||||
}
|
||||
|
||||
/* cd_read_log: dump the CD DMA transfer ring. Each entry records the
|
||||
* setloc LBA, destination RAM address, and byte count for one forward
|
||||
* channel-3 DMA transfer. Use with overlay_dump to map overlay regions
|
||||
* back to their disc positions for the extract_overlays.py disc scanner.
|
||||
*
|
||||
* Parameters: "tail" (int, default 256) — how many recent entries to return.
|
||||
*/
|
||||
static void handle_cd_read_log(int id, const char *json)
|
||||
{
|
||||
extern uint32_t cd_dma_log_get_total(void);
|
||||
extern void cd_dma_log_get_entry(uint32_t idx, int *lba,
|
||||
uint32_t *dest, uint32_t *size);
|
||||
|
||||
int tail = json_get_int(json, "tail", 256);
|
||||
uint32_t total = cd_dma_log_get_total();
|
||||
uint32_t cap = 65536;
|
||||
uint32_t avail = total < cap ? total : cap;
|
||||
if ((uint32_t)tail > avail) tail = (int)avail;
|
||||
|
||||
uint32_t start_idx = total > (uint32_t)tail ? total - (uint32_t)tail : 0;
|
||||
|
||||
send_fmt("{\"id\":%d,\"ok\":true,\"total\":%u,\"count\":%d,\"entries\":[",
|
||||
id, total, tail);
|
||||
int first = 1;
|
||||
for (uint32_t i = start_idx; i < total; i++) {
|
||||
int lba; uint32_t dest, size;
|
||||
cd_dma_log_get_entry(i, &lba, &dest, &size);
|
||||
if (lba < 0) continue;
|
||||
send_fmt("%s{\"lba\":%d,\"dest\":\"0x%08X\",\"size\":%u}",
|
||||
first ? "" : ",", lba, dest, size);
|
||||
first = 0;
|
||||
}
|
||||
send_fmt("]}\n");
|
||||
}
|
||||
|
||||
/* overlay_loader_status: report the dynamic overlay DLL loader state —
|
||||
* whether active, how many functions are registered, which regions have
|
||||
* been checked, and the most recent load event. Rule-3 inspection path for
|
||||
* the loader (it does no stderr logging). */
|
||||
static void handle_overlay_loader_status(int id, const char *json)
|
||||
{
|
||||
(void)json;
|
||||
int active = 0, registered = 0, nchecked = 0, nwritten = 0;
|
||||
int file_found = 0;
|
||||
uint32_t last_crc = 0;
|
||||
char cache_dir[512] = {0}, game_id[64] = {0};
|
||||
uint32_t checked[8] = {0};
|
||||
overlay_loader_get_status(&active, ®istered, &nchecked,
|
||||
cache_dir, sizeof(cache_dir),
|
||||
game_id, sizeof(game_id),
|
||||
checked, 8, &nwritten,
|
||||
&last_crc, &file_found);
|
||||
/* Escape backslashes in cache_dir for JSON */
|
||||
char esc_dir[768] = {0};
|
||||
for (int si = 0, di = 0; cache_dir[si] && di < (int)sizeof(esc_dir)-2; si++) {
|
||||
if (cache_dir[si] == '\\') esc_dir[di++] = '\\';
|
||||
esc_dir[di++] = cache_dir[si];
|
||||
}
|
||||
const char *msg = overlay_loader_last_msg();
|
||||
char esc_msg[512] = {0};
|
||||
for (int si = 0, di = 0; msg[si] && di < (int)sizeof(esc_msg)-2; si++) {
|
||||
if (msg[si] == '\\' || msg[si] == '"') esc_msg[di++] = '\\';
|
||||
esc_msg[di++] = msg[si];
|
||||
}
|
||||
char buf[1536];
|
||||
int n = snprintf(buf, sizeof(buf),
|
||||
"{\"id\":%d,\"ok\":true,\"active\":%d,\"registered\":%d,"
|
||||
"\"regions_checked\":%d,\"last_crc\":\"0x%08X\",\"file_found\":%d,"
|
||||
"\"cache_dir\":\"%s\",\"game_id\":\"%s\",\"last_msg\":\"%s\"",
|
||||
id, active, registered, nchecked, last_crc, file_found,
|
||||
esc_dir, game_id, esc_msg);
|
||||
if (nwritten > 0) {
|
||||
n += snprintf(buf + n, sizeof(buf) - n, ",\"checked\":[");
|
||||
for (int i = 0; i < nwritten; i++)
|
||||
n += snprintf(buf + n, sizeof(buf) - n,
|
||||
"%s\"0x%08X\"", i ? "," : "", checked[i]);
|
||||
n += snprintf(buf + n, sizeof(buf) - n, "]");
|
||||
}
|
||||
snprintf(buf + n, sizeof(buf) - n, "}\n");
|
||||
send_fmt("%s", buf);
|
||||
}
|
||||
|
||||
/* overlay_dump: extract RAM regions that dirty_ram has marked executable
|
||||
* above a threshold physical address. Writes <crc32>.bin files to a
|
||||
* caller-supplied directory and returns a JSON manifest.
|
||||
@@ -7746,6 +7829,8 @@ static const CmdEntry s_commands[] = {
|
||||
{ "fn_entry_tail", handle_fn_entry_tail },
|
||||
{ "fn_exit_dump", handle_fn_exit_dump },
|
||||
{ "overlay_dump", handle_overlay_dump },
|
||||
{ "cd_read_log", handle_cd_read_log },
|
||||
{ "overlay_loader_status", handle_overlay_loader_status },
|
||||
{ NULL, NULL }
|
||||
};
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@ uint32_t g_dirty_ram_last_unsupported_insn = 0;
|
||||
const char *g_dirty_ram_last_unsupported_reason = NULL;
|
||||
|
||||
DirtyRamPcEntry g_dirty_ram_pc_table[DIRTY_RAM_PC_TABLE_SIZE] = {0};
|
||||
DirtyRamPcEntry g_dirty_ram_exec_pc_table[DIRTY_RAM_PC_TABLE_SIZE] = {0};
|
||||
|
||||
DirtyRamBlockLogEntry g_dirty_ram_block_log[DIRTY_RAM_BLOCK_LOG_CAP] = {0};
|
||||
uint64_t g_dirty_ram_block_log_seq = 0;
|
||||
@@ -67,17 +68,29 @@ extern uint64_t s_frame_count;
|
||||
* the working set of install-stub PCs is tiny (handful), so this stays
|
||||
* O(1) in practice. Returns NULL if the table is full — caller treats
|
||||
* that as "stop tracking" rather than failing. */
|
||||
static DirtyRamPcEntry *pc_table_get_or_insert(uint32_t pc) {
|
||||
static DirtyRamPcEntry *pc_table_get_or_insert_in(DirtyRamPcEntry *table, uint32_t pc) {
|
||||
uint32_t h = (pc * 2654435761u) & (DIRTY_RAM_PC_TABLE_SIZE - 1);
|
||||
for (uint32_t i = 0; i < DIRTY_RAM_PC_TABLE_SIZE; i++) {
|
||||
uint32_t idx = (h + i) & (DIRTY_RAM_PC_TABLE_SIZE - 1);
|
||||
DirtyRamPcEntry *e = &g_dirty_ram_pc_table[idx];
|
||||
DirtyRamPcEntry *e = &table[idx];
|
||||
if (e->pc == pc) return e;
|
||||
if (e->pc == 0) { e->pc = pc; return e; }
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static DirtyRamPcEntry *pc_table_get_or_insert(uint32_t pc) {
|
||||
return pc_table_get_or_insert_in(g_dirty_ram_pc_table, pc);
|
||||
}
|
||||
|
||||
/* Record every PC the interpreter executes (not just block entries) so
|
||||
* overlay_capture can report execution-verified seeds for the region. */
|
||||
static void exec_pc_table_record(uint32_t pc) {
|
||||
DirtyRamPcEntry *e = pc_table_get_or_insert_in(g_dirty_ram_exec_pc_table,
|
||||
pc & 0x1FFFFFFFu);
|
||||
if (e) e->hits++;
|
||||
}
|
||||
|
||||
/* From debug_server.c — keep our outer-frame attribution coherent. */
|
||||
extern uint32_t g_debug_current_func_addr;
|
||||
extern uint32_t g_debug_last_store_pc;
|
||||
@@ -233,6 +246,54 @@ static int is_local_dirty_target(uint32_t target) {
|
||||
return phys >= 0x00098000u && dirty_ram_is_dirty(phys);
|
||||
}
|
||||
|
||||
/* A dirty-RAM target only deserves interpretation if its first word decodes
|
||||
* as a plausible MIPS instruction. A scatter-loaded overlay leaves data
|
||||
* bytes in dirty pages; jumping into those and interpreting them as code is
|
||||
* how the original cache build produced black/blue screens. When the target
|
||||
* word doesn't look like code, fall back to the normal dispatch path. */
|
||||
static int dirty_ram_word_looks_decodable(uint32_t insn) {
|
||||
if (insn == 0xFFFFFFFFu || insn == 0xFFFFFFFDu) return 0;
|
||||
|
||||
uint32_t op = op_field(insn);
|
||||
uint32_t fn = funct_field(insn);
|
||||
uint32_t rt = rt_field(insn);
|
||||
|
||||
if (op == 0x00u) {
|
||||
switch (fn) {
|
||||
case 0x00u: case 0x02u: case 0x03u: case 0x04u:
|
||||
case 0x06u: case 0x07u: case 0x08u: case 0x09u:
|
||||
case 0x0Cu: case 0x0Du:
|
||||
case 0x10u: case 0x11u: case 0x12u: case 0x13u:
|
||||
case 0x18u: case 0x19u: case 0x1Au: case 0x1Bu:
|
||||
case 0x20u: case 0x21u: case 0x22u: case 0x23u:
|
||||
case 0x24u: case 0x25u: case 0x26u: case 0x27u:
|
||||
case 0x2Au: case 0x2Bu:
|
||||
return 1;
|
||||
default:
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
if (op == 0x01u) {
|
||||
return rt == 0x00u || rt == 0x01u || rt == 0x10u || rt == 0x11u;
|
||||
}
|
||||
|
||||
switch (op) {
|
||||
case 0x02u: case 0x03u: case 0x04u: case 0x05u:
|
||||
case 0x06u: case 0x07u:
|
||||
case 0x08u: case 0x09u: case 0x0Au: case 0x0Bu:
|
||||
case 0x0Cu: case 0x0Du: case 0x0Eu: case 0x0Fu:
|
||||
case 0x10u: case 0x12u:
|
||||
case 0x20u: case 0x21u: case 0x22u: case 0x23u:
|
||||
case 0x24u: case 0x25u: case 0x26u:
|
||||
case 0x28u: case 0x29u: case 0x2Au: case 0x2Bu:
|
||||
case 0x2Eu:
|
||||
case 0x30u: case 0x32u: case 0x38u: case 0x3Au:
|
||||
return 1;
|
||||
default:
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
static int dispatch_nonlocal_call(CPUState *cpu, uint32_t target,
|
||||
uint32_t return_pc,
|
||||
uint32_t *next_pc_out) {
|
||||
@@ -276,6 +337,7 @@ static void exec_delay_slot(CPUState *cpu, uint32_t pc) {
|
||||
}
|
||||
|
||||
static int exec_one(CPUState *cpu, uint32_t pc, uint32_t *next_pc_out) {
|
||||
exec_pc_table_record(pc);
|
||||
uint32_t phys = pc & 0x1FFFFFFFu;
|
||||
uint32_t insn = fetch_word(phys);
|
||||
uint32_t opc = op_field(insn);
|
||||
@@ -466,6 +528,9 @@ static int exec_one(CPUState *cpu, uint32_t pc, uint32_t *next_pc_out) {
|
||||
if (!is_local_dirty_target(target)) {
|
||||
return dispatch_nonlocal_call(cpu, target, return_pc, next_pc_out);
|
||||
}
|
||||
if (!dirty_ram_word_looks_decodable(fetch_word(target & 0x1FFFFFFFu))) {
|
||||
return dispatch_nonlocal_call(cpu, target, return_pc, next_pc_out);
|
||||
}
|
||||
cpu->pc = target;
|
||||
return 1;
|
||||
}
|
||||
@@ -687,6 +752,23 @@ int dirty_ram_dispatch(CPUState* cpu, uint32_t addr) {
|
||||
if (psx_dispatch_game_compiled(cpu, addr)) return 1;
|
||||
#endif
|
||||
|
||||
/* B-2: statically-compiled overlay functions (generated/overlays_static.c).
|
||||
* Inert unless a game provides an overlays_static.c at build time. */
|
||||
#ifdef PSX_HAS_OVERLAY_DISPATCH
|
||||
{
|
||||
extern int psx_overlay_dispatch(CPUState *cpu, uint32_t addr);
|
||||
if (psx_overlay_dispatch(cpu, addr)) return 1;
|
||||
}
|
||||
#endif
|
||||
|
||||
/* A-1: dynamically-loaded overlay DLL functions, checked before the
|
||||
* interpreter. No-op (returns 0) until overlay_loader_init() runs, which
|
||||
* only happens when the overlay cache is enabled in config. */
|
||||
{
|
||||
extern int overlay_loader_dispatch(CPUState *cpu, uint32_t addr);
|
||||
if (overlay_loader_dispatch(cpu, addr)) return 1;
|
||||
}
|
||||
|
||||
if (!dirty_ram_is_dirty(phys)) return 0;
|
||||
|
||||
/* Reset soft-fail state at block entry. */
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
#include "dirty_ram_interp.h"
|
||||
#include "gpu.h"
|
||||
#include "mdec.h"
|
||||
#include "overlay_capture.h"
|
||||
#include "spu.h"
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
@@ -49,11 +50,43 @@ typedef struct {
|
||||
uint32_t total_words;
|
||||
uint32_t remaining_words;
|
||||
uint32_t cycles_accum;
|
||||
uint32_t start_addr; /* madr at transfer start (CD overlay capture) */
|
||||
} DMAAsyncChannel;
|
||||
|
||||
static DMAAsyncChannel mdec_async[2];
|
||||
static DMAAsyncChannel cdrom_async;
|
||||
|
||||
/* ---- CD DMA transfer log ---- */
|
||||
/* Every forward CH3 DMA that lands below 0x1C0000 (game data region) records
|
||||
* (setloc_lba, dest_addr, size). Transfers to 0x1C0000+ are FMV/streaming
|
||||
* buffers and are excluded to keep the ring focused on overlay loads.
|
||||
* Surfaced via the cd_read_log TCP command; pairs with overlay_dump to map
|
||||
* overlay regions back to disc positions for extract_overlays.py. */
|
||||
#define CD_DMA_LOG_CAP 65536
|
||||
typedef struct { int lba; uint32_t dest; uint32_t size; } CdDmaEntry;
|
||||
static CdDmaEntry cd_dma_log[CD_DMA_LOG_CAP];
|
||||
static uint32_t cd_dma_log_head = 0;
|
||||
static uint32_t cd_dma_log_total = 0;
|
||||
|
||||
static void cd_dma_log_push(int lba, uint32_t dest, uint32_t size) {
|
||||
cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].lba = lba;
|
||||
cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].dest = dest;
|
||||
cd_dma_log[cd_dma_log_head % CD_DMA_LOG_CAP].size = size;
|
||||
cd_dma_log_head = (cd_dma_log_head + 1) % CD_DMA_LOG_CAP;
|
||||
cd_dma_log_total++;
|
||||
}
|
||||
|
||||
uint32_t cd_dma_log_get_total(void) { return cd_dma_log_total; }
|
||||
void cd_dma_log_get_entry(uint32_t idx, int *lba, uint32_t *dest, uint32_t *size) {
|
||||
uint32_t cap = CD_DMA_LOG_CAP;
|
||||
uint32_t oldest = cd_dma_log_total > cap ? cd_dma_log_total - cap : 0;
|
||||
if (idx < oldest || idx >= cd_dma_log_total) { *lba = -1; return; }
|
||||
uint32_t slot = idx % cap;
|
||||
*lba = cd_dma_log[slot].lba;
|
||||
*dest = cd_dma_log[slot].dest;
|
||||
*size = cd_dma_log[slot].size;
|
||||
}
|
||||
|
||||
#define DMA_MDEC_IN_CYCLES_PER_WORD 1u
|
||||
#define DMA_MDEC_OUT_CYCLES_PER_WORD 14u
|
||||
#define DMA_GPU_CYCLES_PER_WORD 1u
|
||||
@@ -247,6 +280,7 @@ static void start_async_cdrom_transfer(void) {
|
||||
a->total_words = transfer_word_count(3);
|
||||
a->remaining_words = a->total_words;
|
||||
a->cycles_accum = 0;
|
||||
a->start_addr = channels[3].madr & 0x1FFFFCu;
|
||||
|
||||
if (a->total_words == 0) {
|
||||
cancel_async_transfer(3);
|
||||
@@ -255,6 +289,25 @@ static void start_async_cdrom_transfer(void) {
|
||||
}
|
||||
|
||||
static void finish_async_cdrom_transfer(uint32_t final_addr) {
|
||||
DMAAsyncChannel *a = &cdrom_async;
|
||||
uint32_t step = (channels[3].chcr >> 1) & 1u;
|
||||
uint32_t load_start = a->start_addr;
|
||||
uint32_t size = a->total_words * 4u;
|
||||
|
||||
/* Log and capture game-data transfers only.
|
||||
* Transfers to 0x1C0000+ are FMV/streaming buffers; skip them. */
|
||||
if (!step && size > 0 && load_start < 0x1C0000u) {
|
||||
int lba = cdrom_get_setloc_lba();
|
||||
if (lba >= 0) cd_dma_log_push(lba, load_start, size);
|
||||
|
||||
/* B-1: capture overlay bytes into the write-once capture set.
|
||||
* overlay_capture_on_dma auto-activates after game handoff and is a
|
||||
* no-op unless the overlay cache is enabled in config. */
|
||||
extern uint8_t *memory_get_ram_ptr(void);
|
||||
uint8_t *ram = memory_get_ram_ptr();
|
||||
overlay_capture_on_dma(load_start, size, ram + load_start);
|
||||
}
|
||||
|
||||
channels[3].madr = final_addr;
|
||||
cancel_async_transfer(3);
|
||||
complete_transfer(3);
|
||||
|
||||
@@ -10,6 +10,8 @@
|
||||
#include "cdrom.h"
|
||||
#include "fntrace.h"
|
||||
#include "boot_state.h"
|
||||
#include "overlay_capture.h"
|
||||
#include "overlay_loader.h"
|
||||
#include "gpu.h"
|
||||
#include "sio.h"
|
||||
#include "spu.h"
|
||||
@@ -468,6 +470,7 @@ static void close_controller(void);
|
||||
|
||||
static void shutdown_runtime(void) {
|
||||
memcard_flush_all();
|
||||
overlay_capture_write_json();
|
||||
if (sdl_audio_device) {
|
||||
SDL_ClearQueuedAudio(sdl_audio_device);
|
||||
SDL_CloseAudioDevice(sdl_audio_device);
|
||||
@@ -1106,6 +1109,16 @@ int main(int argc, char** argv) {
|
||||
if (gc.runtime.has_disc_speed) disc_speed = gc.runtime.disc_speed;
|
||||
game_entry_pc = gc.entry_pc;
|
||||
fast_boot = gc.runtime.fast_boot;
|
||||
/* Overlay DLL cache (Layer A). Off unless enabled in [runtime];
|
||||
* when on, capture overlay bytes and scan cache/<game_id>/ for
|
||||
* precompiled overlay DLLs. */
|
||||
if (gc.runtime.overlay_cache) {
|
||||
std::filesystem::path exe_dir = exe_dir_from_argv(argv[0]);
|
||||
std::string cache_dir = (exe_dir / "cache").string();
|
||||
overlay_capture_set_out_dir(exe_dir.string().c_str());
|
||||
overlay_capture_set_enabled(1);
|
||||
overlay_loader_init(cache_dir.c_str(), game_id.c_str());
|
||||
}
|
||||
std::fprintf(stdout, "psxrecomp: loaded game config %s (%s, %s)\n",
|
||||
game_config_path, game_name.c_str(), game_id.c_str());
|
||||
} catch (const std::exception& ex) {
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
#include "overlay_capture.h"
|
||||
#include "overlay_loader.h"
|
||||
#include "dirty_ram_interp.h"
|
||||
#include "crc32.h"
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdio.h>
|
||||
|
||||
/* ---- Capture set --------------------------------------------------------
|
||||
* Tracks which physical addresses have received CD DMA loads since game
|
||||
* handoff. Used for dedup (state machine) and A-layer compilation queue.
|
||||
* The JSON output uses dirty_ram regions, not individual DMA blocks.
|
||||
* -------------------------------------------------------------------------*/
|
||||
|
||||
typedef enum {
|
||||
OV_QUEUED = 0, /* captured, not yet compiled */
|
||||
OV_COMPILING = 1, /* background thread working on it */
|
||||
OV_COMPILED = 2, /* DLL written, dispatch table patched */
|
||||
} OvState;
|
||||
|
||||
typedef struct {
|
||||
uint32_t load_addr; /* physical RAM address */
|
||||
uint32_t size; /* byte count of this DMA block */
|
||||
uint8_t *bytes; /* unpatched copy taken at DMA time */
|
||||
OvState state;
|
||||
} OvEntry;
|
||||
|
||||
#define MAX_OVERLAYS 4096
|
||||
|
||||
static OvEntry s_entries[MAX_OVERLAYS];
|
||||
static int s_count = 0;
|
||||
static char s_out_dir[512];
|
||||
static int s_active = 0;
|
||||
static int s_enabled = 0; /* config gate; off unless overlay cache enabled */
|
||||
|
||||
/* ---- Base64 encoder ----------------------------------------------------- */
|
||||
|
||||
static const char k_b64[] =
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
||||
|
||||
static void write_b64(FILE *f, const uint8_t *data, size_t len)
|
||||
{
|
||||
size_t i;
|
||||
for (i = 0; i < len; i += 3) {
|
||||
uint32_t b = (uint32_t)data[i] << 16;
|
||||
if (i + 1 < len) b |= (uint32_t)data[i + 1] << 8;
|
||||
if (i + 2 < len) b |= (uint32_t)data[i + 2];
|
||||
fputc(k_b64[(b >> 18) & 0x3F], f);
|
||||
fputc(k_b64[(b >> 12) & 0x3F], f);
|
||||
fputc(i + 1 < len ? k_b64[(b >> 6) & 0x3F] : '=', f);
|
||||
fputc(i + 2 < len ? k_b64[(b ) & 0x3F] : '=', f);
|
||||
}
|
||||
}
|
||||
|
||||
/* ---- Public API --------------------------------------------------------- */
|
||||
|
||||
void overlay_capture_set_out_dir(const char *out_dir)
|
||||
{
|
||||
strncpy(s_out_dir, out_dir, sizeof(s_out_dir) - 1);
|
||||
s_out_dir[sizeof(s_out_dir) - 1] = '\0';
|
||||
}
|
||||
|
||||
void overlay_capture_init(const char *out_dir)
|
||||
{
|
||||
overlay_capture_set_out_dir(out_dir);
|
||||
}
|
||||
|
||||
void overlay_capture_set_enabled(int enabled)
|
||||
{
|
||||
s_enabled = enabled ? 1 : 0;
|
||||
}
|
||||
|
||||
void overlay_capture_on_dma(uint32_t load_addr, uint32_t size,
|
||||
const uint8_t *bytes)
|
||||
{
|
||||
int i;
|
||||
OvEntry *e;
|
||||
extern int fntrace_is_game_started(void);
|
||||
|
||||
if (!s_enabled) return; /* overlay cache disabled in config */
|
||||
if (size == 0) return;
|
||||
|
||||
/* Auto-activate on the first post-game-handoff DMA. */
|
||||
if (!s_active) {
|
||||
if (!fntrace_is_game_started()) return;
|
||||
s_active = 1;
|
||||
}
|
||||
|
||||
/* Dedup by load_addr. */
|
||||
for (i = 0; i < s_count; i++) {
|
||||
if (s_entries[i].load_addr == load_addr)
|
||||
return;
|
||||
}
|
||||
|
||||
if (s_count >= MAX_OVERLAYS) return;
|
||||
|
||||
e = &s_entries[s_count++];
|
||||
e->load_addr = load_addr;
|
||||
e->size = size;
|
||||
e->state = OV_QUEUED;
|
||||
e->bytes = (uint8_t *)malloc(size);
|
||||
if (e->bytes) memcpy(e->bytes, bytes, size);
|
||||
|
||||
/* A-1: check if a compiled DLL exists in cache for these bytes. */
|
||||
overlay_loader_check_cache(load_addr, size, bytes);
|
||||
/* A-2: push onto compilation queue (to be added in A-2). */
|
||||
}
|
||||
|
||||
/* ---- JSON output — assembled dirty_ram regions -------------------------- */
|
||||
|
||||
void overlay_capture_write_json(void)
|
||||
{
|
||||
extern uint32_t dirty_ram_get_bitmap_word(uint32_t word_index);
|
||||
extern uint32_t dirty_ram_get_bitmap_word_count(void);
|
||||
|
||||
char path[600];
|
||||
FILE *f;
|
||||
uint32_t bw, page_sz, lo_page;
|
||||
uint32_t page, run_start;
|
||||
int in_run, first_region;
|
||||
|
||||
if (!s_active) return;
|
||||
|
||||
/* Use the same lo threshold as overlay_dump: only above 0x98000. */
|
||||
uint32_t lo_phys = 0x00098000u;
|
||||
|
||||
bw = dirty_ram_get_bitmap_word_count();
|
||||
page_sz = 4096u;
|
||||
lo_page = lo_phys / page_sz;
|
||||
|
||||
snprintf(path, sizeof(path), "%s/overlay_captures.json", s_out_dir);
|
||||
f = fopen(path, "w");
|
||||
if (!f) return;
|
||||
|
||||
fprintf(f, "[\n");
|
||||
in_run = 0;
|
||||
run_start = 0;
|
||||
first_region = 1;
|
||||
|
||||
for (page = lo_page; page <= bw * 32u; page++) {
|
||||
int dirty = 0;
|
||||
if (page < bw * 32u) {
|
||||
uint32_t word = dirty_ram_get_bitmap_word(page >> 5);
|
||||
dirty = (word >> (page & 31u)) & 1u;
|
||||
}
|
||||
|
||||
if (dirty && !in_run) {
|
||||
in_run = 1;
|
||||
run_start = page;
|
||||
} else if (!dirty && in_run) {
|
||||
uint32_t phys = run_start * page_sz;
|
||||
uint32_t size = (page - run_start) * page_sz;
|
||||
uint32_t virt = 0x80000000u | (phys & 0x1FFFFFu);
|
||||
int j;
|
||||
|
||||
in_run = 0;
|
||||
|
||||
/* Seeds: only per-PC interpreter hits — execution-verified. */
|
||||
#define MAX_CAPTURE_PCS DIRTY_RAM_PC_TABLE_SIZE
|
||||
uint32_t *executed = (uint32_t *)malloc(
|
||||
MAX_CAPTURE_PCS * sizeof(uint32_t));
|
||||
uint32_t *dispatch = (uint32_t *)malloc(
|
||||
MAX_CAPTURE_PCS * sizeof(uint32_t));
|
||||
int nexec = 0;
|
||||
int ndisp = 0;
|
||||
if (!executed || !dispatch) {
|
||||
free(executed);
|
||||
free(dispatch);
|
||||
continue;
|
||||
}
|
||||
|
||||
for (j = 0; j < DIRTY_RAM_PC_TABLE_SIZE; j++) {
|
||||
DirtyRamPcEntry *pe = &g_dirty_ram_pc_table[j];
|
||||
if (pe->pc == 0 || pe->hits == 0) continue;
|
||||
if (pe->pc < phys || pe->pc >= phys + size) continue;
|
||||
dispatch[ndisp++] = pe->pc;
|
||||
}
|
||||
for (j = 0; j < DIRTY_RAM_PC_TABLE_SIZE; j++) {
|
||||
DirtyRamPcEntry *pe = &g_dirty_ram_exec_pc_table[j];
|
||||
if (pe->pc == 0 || pe->hits == 0) continue;
|
||||
if (pe->pc < phys || pe->pc >= phys + size) continue;
|
||||
executed[nexec++] = pe->pc;
|
||||
}
|
||||
|
||||
/* Skip pure-data regions — nothing to compile. */
|
||||
if (ndisp == 0 && nexec == 0) {
|
||||
free(executed);
|
||||
free(dispatch);
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Assemble unpatched bytes from DMA-time captures.
|
||||
* Start with a zeroed image, then blit each stored DMA block
|
||||
* into its correct offset. This gives pre-patch bytes, not
|
||||
* the potentially-modified bytes now in live RAM. */
|
||||
uint8_t *image = (uint8_t *)calloc(size, 1);
|
||||
if (!image) {
|
||||
free(executed);
|
||||
free(dispatch);
|
||||
continue;
|
||||
}
|
||||
for (j = 0; j < s_count; j++) {
|
||||
OvEntry *blk = &s_entries[j];
|
||||
if (!blk->bytes) continue;
|
||||
if (blk->load_addr < phys || blk->load_addr >= phys + size) continue;
|
||||
uint32_t off = blk->load_addr - phys;
|
||||
uint32_t copy_sz = blk->size;
|
||||
if (off + copy_sz > size) copy_sz = size - off;
|
||||
memcpy(image + off, blk->bytes, copy_sz);
|
||||
}
|
||||
|
||||
if (!first_region) fprintf(f, ",\n");
|
||||
first_region = 0;
|
||||
|
||||
fprintf(f, " {\n");
|
||||
fprintf(f, " \"schema\": \"psxrecomp overlay capture v2\",\n");
|
||||
fprintf(f, " \"load_addr\": \"0x%08X\",\n", virt);
|
||||
fprintf(f, " \"size\": %u,\n", size);
|
||||
fprintf(f, " \"bytes_b64\": \"");
|
||||
write_b64(f, image, size);
|
||||
free(image);
|
||||
fprintf(f, "\",\n");
|
||||
|
||||
fprintf(f, " \"executed_pcs\": [");
|
||||
for (j = 0; j < nexec; j++) {
|
||||
uint32_t seed_virt = 0x80000000u | (executed[j] & 0x1FFFFFu);
|
||||
if (j > 0) fprintf(f, ", ");
|
||||
fprintf(f, "\"0x%08X\"", seed_virt);
|
||||
}
|
||||
fprintf(f, "],\n");
|
||||
|
||||
fprintf(f, " \"dispatch_entry_pcs\": [");
|
||||
for (j = 0; j < ndisp; j++) {
|
||||
uint32_t seed_virt = 0x80000000u | (dispatch[j] & 0x1FFFFFu);
|
||||
if (j > 0) fprintf(f, ", ");
|
||||
fprintf(f, "\"0x%08X\"", seed_virt);
|
||||
}
|
||||
fprintf(f, "],\n");
|
||||
|
||||
fprintf(f, " \"function_entry_pcs\": [],\n");
|
||||
|
||||
fprintf(f, " \"seeds\": [");
|
||||
for (j = 0; j < ndisp; j++) {
|
||||
uint32_t seed_virt = 0x80000000u | (dispatch[j] & 0x1FFFFFu);
|
||||
if (j > 0) fprintf(f, ", ");
|
||||
fprintf(f, "\"0x%08X\"", seed_virt);
|
||||
}
|
||||
fprintf(f, "]\n");
|
||||
fprintf(f, " }");
|
||||
free(executed);
|
||||
free(dispatch);
|
||||
}
|
||||
}
|
||||
|
||||
fprintf(f, "\n]\n");
|
||||
fclose(f);
|
||||
}
|
||||
|
||||
int overlay_capture_count(void)
|
||||
{
|
||||
return s_count;
|
||||
}
|
||||
|
||||
uint32_t overlay_capture_get_region_crc(uint32_t region_start,
|
||||
uint32_t region_size)
|
||||
{
|
||||
int j;
|
||||
uint8_t *image = (uint8_t *)calloc(region_size, 1);
|
||||
if (!image) return 0;
|
||||
|
||||
for (j = 0; j < s_count; j++) {
|
||||
OvEntry *blk = &s_entries[j];
|
||||
if (!blk->bytes) continue;
|
||||
if (blk->load_addr < region_start ||
|
||||
blk->load_addr >= region_start + region_size) continue;
|
||||
uint32_t off = blk->load_addr - region_start;
|
||||
uint32_t copy_sz = blk->size;
|
||||
if (off + copy_sz > region_size) copy_sz = region_size - off;
|
||||
memcpy(image + off, blk->bytes, copy_sz);
|
||||
}
|
||||
|
||||
uint32_t crc = crc32_compute(image, region_size);
|
||||
free(image);
|
||||
return crc;
|
||||
}
|
||||
@@ -0,0 +1,365 @@
|
||||
#include "overlay_loader.h"
|
||||
#include "overlay_api.h"
|
||||
#include "crc32.h"
|
||||
#include "interrupts.h"
|
||||
#include "debug_server.h"
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <stdio.h>
|
||||
#include <stdarg.h>
|
||||
|
||||
#ifdef _WIN32
|
||||
# define WIN32_LEAN_AND_MEAN
|
||||
# include <windows.h>
|
||||
#else
|
||||
# include <dlfcn.h>
|
||||
#endif
|
||||
|
||||
/* ---- Dynamic dispatch hash table --------------------------------------- */
|
||||
/* Open-addressed hash map: physical addr → compiled function pointer.
|
||||
* Sized at 16384 slots — enough for ~10 overlays × ~500 functions each
|
||||
* at <50% load factor. */
|
||||
|
||||
#define DYNTAB_CAP 16384u
|
||||
#define DYNTAB_MASK (DYNTAB_CAP - 1u)
|
||||
|
||||
typedef void (*OverlayFn)(CPUState *);
|
||||
|
||||
typedef struct {
|
||||
uint32_t addr; /* physical address, 0 = empty */
|
||||
OverlayFn fn;
|
||||
} DynEntry;
|
||||
|
||||
static DynEntry s_table[DYNTAB_CAP];
|
||||
static int s_count = 0;
|
||||
|
||||
static void dyntab_insert(uint32_t phys, OverlayFn fn)
|
||||
{
|
||||
uint32_t h = (phys * 2654435761u) & DYNTAB_MASK;
|
||||
uint32_t i;
|
||||
for (i = 0; i < DYNTAB_CAP; i++) {
|
||||
uint32_t idx = (h + i) & DYNTAB_MASK;
|
||||
if (s_table[idx].addr == 0 || s_table[idx].addr == phys) {
|
||||
if (s_table[idx].addr == 0) s_count++;
|
||||
s_table[idx].addr = phys;
|
||||
s_table[idx].fn = fn;
|
||||
return;
|
||||
}
|
||||
}
|
||||
/* Table full — shouldn't happen at <50% load */
|
||||
}
|
||||
|
||||
static OverlayFn dyntab_lookup(uint32_t phys)
|
||||
{
|
||||
uint32_t h = (phys * 2654435761u) & DYNTAB_MASK;
|
||||
uint32_t i;
|
||||
for (i = 0; i < DYNTAB_CAP; i++) {
|
||||
uint32_t idx = (h + i) & DYNTAB_MASK;
|
||||
if (s_table[idx].addr == 0) return NULL;
|
||||
if (s_table[idx].addr == phys) return s_table[idx].fn;
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* ---- Global state ------------------------------------------------------ */
|
||||
|
||||
static char s_cache_dir[512];
|
||||
static char s_game_id[64];
|
||||
static int s_active = 0;
|
||||
|
||||
/* Rule 3: no stderr logging. The most recent loader event is recorded here
|
||||
* and surfaced through the `overlay_loader_status` TCP command instead of
|
||||
* being printed. */
|
||||
static char s_last_msg[256] = {0};
|
||||
|
||||
static void loader_log(const char *fmt, ...)
|
||||
{
|
||||
va_list ap;
|
||||
va_start(ap, fmt);
|
||||
vsnprintf(s_last_msg, sizeof(s_last_msg), fmt, ap);
|
||||
va_end(ap);
|
||||
}
|
||||
|
||||
const char *overlay_loader_last_msg(void) { return s_last_msg; }
|
||||
|
||||
/* ---- Cache index: region_start → dll path ------------------------------- */
|
||||
/* Scanned once at init from {cache_dir}/{game_id}/{8hex}_{8hex}.dll files.
|
||||
* Avoids recomputing the DMA CRC at dispatch time (which fails after
|
||||
* fast_boot because DMA callbacks haven't fired to fill s_entries). */
|
||||
|
||||
#define CACHE_IDX_CAP 256
|
||||
typedef struct { uint32_t region_start; char path[768]; } CacheEntry;
|
||||
static CacheEntry s_cache_idx[CACHE_IDX_CAP];
|
||||
static int s_cache_idx_count = 0;
|
||||
|
||||
static void scan_cache_dir(void)
|
||||
{
|
||||
#ifdef _WIN32
|
||||
char pattern[768];
|
||||
snprintf(pattern, sizeof(pattern), "%s/%s/*_*.dll",
|
||||
s_cache_dir, s_game_id);
|
||||
WIN32_FIND_DATAA fd;
|
||||
HANDLE h = FindFirstFileA(pattern, &fd);
|
||||
if (h == INVALID_HANDLE_VALUE) return;
|
||||
do {
|
||||
if (strlen(fd.cFileName) != 21) continue; /* 8+1+8+4 = 21 */
|
||||
uint32_t addr = (uint32_t)strtoul(fd.cFileName, NULL, 16);
|
||||
if (addr == 0) continue;
|
||||
if (s_cache_idx_count >= CACHE_IDX_CAP) break;
|
||||
CacheEntry *e = &s_cache_idx[s_cache_idx_count++];
|
||||
e->region_start = addr;
|
||||
snprintf(e->path, sizeof(e->path), "%s/%s/%s",
|
||||
s_cache_dir, s_game_id, fd.cFileName);
|
||||
} while (FindNextFileA(h, &fd));
|
||||
FindClose(h);
|
||||
#endif
|
||||
}
|
||||
|
||||
/* ---- Runtime callbacks wired into overlay DLLs via overlay_init() ------ */
|
||||
|
||||
extern void psx_dispatch_call(CPUState *cpu, uint32_t addr, uint32_t ra);
|
||||
extern void psx_check_interrupts(CPUState *cpu);
|
||||
extern void gte_execute(CPUState *cpu, uint32_t cmd);
|
||||
extern void psx_syscall(CPUState *cpu, uint32_t code);
|
||||
extern void psx_unknown_dispatch(CPUState *cpu, uint32_t addr, uint32_t phys);
|
||||
extern void debug_server_log_call_entry(uint32_t func_addr);
|
||||
|
||||
static OverlayCallbacks s_callbacks;
|
||||
|
||||
static void init_callbacks(void)
|
||||
{
|
||||
s_callbacks.dispatch_call = psx_dispatch_call;
|
||||
s_callbacks.check_interrupts = psx_check_interrupts;
|
||||
s_callbacks.gte_execute = gte_execute;
|
||||
s_callbacks.psx_syscall = psx_syscall;
|
||||
s_callbacks.psx_unknown_dispatch = psx_unknown_dispatch;
|
||||
s_callbacks.log_call_entry = debug_server_log_call_entry;
|
||||
}
|
||||
|
||||
/* ---- DLL loading and export enumeration -------------------------------- */
|
||||
|
||||
#ifdef _WIN32
|
||||
static int load_overlay_dll(const char *dll_path, uint32_t load_addr_virt)
|
||||
{
|
||||
HMODULE dll = LoadLibraryA(dll_path);
|
||||
if (!dll) {
|
||||
loader_log("LoadLibrary(%s) failed: %lu", dll_path, GetLastError());
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Call overlay_init to wire the runtime callbacks. */
|
||||
typedef void (*InitFn)(const OverlayCallbacks *);
|
||||
InitFn init_fn = (InitFn)GetProcAddress(dll, "overlay_init");
|
||||
if (!init_fn) {
|
||||
loader_log("no overlay_init in %s", dll_path);
|
||||
FreeLibrary(dll);
|
||||
return 0;
|
||||
}
|
||||
init_fn(&s_callbacks);
|
||||
|
||||
/* Enumerate PE export table for func_XXXXXXXX symbols. */
|
||||
BYTE *base = (BYTE *)dll;
|
||||
IMAGE_DOS_HEADER *dos = (IMAGE_DOS_HEADER *)base;
|
||||
IMAGE_NT_HEADERS *nt = (IMAGE_NT_HEADERS *)(base + dos->e_lfanew);
|
||||
IMAGE_DATA_DIRECTORY *exp_dd =
|
||||
&nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT];
|
||||
|
||||
if (!exp_dd->VirtualAddress) {
|
||||
loader_log("no export dir in %s", dll_path);
|
||||
FreeLibrary(dll);
|
||||
return 0;
|
||||
}
|
||||
|
||||
IMAGE_EXPORT_DIRECTORY *exp =
|
||||
(IMAGE_EXPORT_DIRECTORY *)(base + exp_dd->VirtualAddress);
|
||||
DWORD *names = (DWORD *)(base + exp->AddressOfNames);
|
||||
WORD *ordinals = (WORD *)(base + exp->AddressOfNameOrdinals);
|
||||
DWORD *funcs = (DWORD *)(base + exp->AddressOfFunctions);
|
||||
|
||||
int registered = 0;
|
||||
for (DWORD i = 0; i < exp->NumberOfNames; i++) {
|
||||
const char *name = (const char *)(base + names[i]);
|
||||
/* Match "func_XXXXXXXX" — exactly 13 chars */
|
||||
if (strncmp(name, "func_", 5) != 0) continue;
|
||||
if (strlen(name) != 13) continue;
|
||||
uint32_t addr = (uint32_t)strtoul(name + 5, NULL, 16);
|
||||
if (addr == 0) continue;
|
||||
|
||||
WORD ord = ordinals[i];
|
||||
OverlayFn fn = (OverlayFn)(base + funcs[ord]);
|
||||
|
||||
uint32_t phys = addr & 0x1FFFFFFFu;
|
||||
dyntab_insert(phys, fn);
|
||||
registered++;
|
||||
}
|
||||
|
||||
loader_log("loaded %s -> %d functions registered", dll_path, registered);
|
||||
return registered;
|
||||
}
|
||||
#else
|
||||
static int load_overlay_dll(const char *dll_path, uint32_t load_addr_virt)
|
||||
{
|
||||
void *dll = dlopen(dll_path, RTLD_NOW | RTLD_LOCAL);
|
||||
if (!dll) {
|
||||
loader_log("dlopen(%s) failed: %s", dll_path, dlerror());
|
||||
return 0;
|
||||
}
|
||||
|
||||
typedef void (*InitFn)(const OverlayCallbacks *);
|
||||
InitFn init_fn = (InitFn)dlsym(dll, "overlay_init");
|
||||
if (!init_fn) {
|
||||
loader_log("no overlay_init in %s", dll_path);
|
||||
dlclose(dll);
|
||||
return 0;
|
||||
}
|
||||
init_fn(&s_callbacks);
|
||||
|
||||
/* On non-Windows we can't easily enumerate exports without libelf/BFD.
|
||||
* Fall back to scanning the known address range: probe GetProcAddress
|
||||
* equivalents by constructing names from the seed list stored in the
|
||||
* overlay capture. For now, register nothing — the interpreter handles
|
||||
* it until proper export enumeration is implemented. */
|
||||
loader_log("%s loaded (posix export scan TODO)", dll_path);
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
/* ---- Public API -------------------------------------------------------- */
|
||||
|
||||
void overlay_loader_init(const char *cache_dir, const char *game_id)
|
||||
{
|
||||
strncpy(s_cache_dir, cache_dir, sizeof(s_cache_dir) - 1);
|
||||
strncpy(s_game_id, game_id, sizeof(s_game_id) - 1);
|
||||
init_callbacks();
|
||||
scan_cache_dir();
|
||||
s_active = 1;
|
||||
}
|
||||
|
||||
void overlay_loader_check_cache(uint32_t load_addr, uint32_t size,
|
||||
const uint8_t *bytes)
|
||||
{
|
||||
/* Block-level check is not used — the DLL CRC is over the full assembled
|
||||
* region, not individual DMA blocks. Cache loading is deferred to
|
||||
* overlay_loader_dispatch on the first dispatch miss. */
|
||||
(void)load_addr; (void)size; (void)bytes;
|
||||
}
|
||||
|
||||
/* ---- Lazy region cache check ------------------------------------------- */
|
||||
/* On the first dispatch miss for any address in a dirty region, we compute
|
||||
* the CRC32 of the full assembled region from RAM and look for a matching
|
||||
* DLL. This fires AFTER the game has fully loaded the overlay and called
|
||||
* into it — by that point all DMA blocks have landed in RAM. */
|
||||
|
||||
#define MAX_CHECKED 64
|
||||
static uint32_t s_checked[MAX_CHECKED];
|
||||
static int s_nchecked = 0;
|
||||
static uint32_t s_last_crc = 0;
|
||||
static int s_last_file_found = 0;
|
||||
|
||||
static int already_checked(uint32_t region_start) {
|
||||
int i;
|
||||
for (i = 0; i < s_nchecked; i++)
|
||||
if (s_checked[i] == region_start) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void mark_checked(uint32_t region_start) {
|
||||
if (s_nchecked < MAX_CHECKED)
|
||||
s_checked[s_nchecked++] = region_start;
|
||||
}
|
||||
|
||||
static void try_load_region(uint32_t phys)
|
||||
{
|
||||
extern uint32_t dirty_ram_get_bitmap_word(uint32_t word_index);
|
||||
extern uint32_t dirty_ram_get_bitmap_word_count(void);
|
||||
extern uint8_t *memory_get_ram_ptr(void);
|
||||
|
||||
uint32_t page_sz = 4096u;
|
||||
uint32_t bw = dirty_ram_get_bitmap_word_count();
|
||||
char dll_path[768];
|
||||
|
||||
/* Walk backward to find region start. */
|
||||
uint32_t pg = phys / page_sz;
|
||||
while (pg > 0) {
|
||||
uint32_t pp = pg - 1;
|
||||
if (!((dirty_ram_get_bitmap_word(pp >> 5) >> (pp & 31u)) & 1u)) break;
|
||||
pg = pp;
|
||||
}
|
||||
uint32_t region_start = pg * page_sz;
|
||||
|
||||
if (already_checked(region_start)) return;
|
||||
mark_checked(region_start);
|
||||
|
||||
/* Walk forward to find region end. */
|
||||
uint32_t pg2 = phys / page_sz + 1;
|
||||
while (pg2 < bw * 32u &&
|
||||
((dirty_ram_get_bitmap_word(pg2 >> 5) >> (pg2 & 31u)) & 1u))
|
||||
pg2++;
|
||||
uint32_t region_size = (pg2 - pg) * page_sz;
|
||||
|
||||
/* Look up the compiled DLL by region_start in the cache index.
|
||||
* The index was built at init by scanning for {addr}_{crc}.dll files,
|
||||
* so we never need to recompute the CRC from RAM or DMA blocks. */
|
||||
s_last_crc = 0;
|
||||
s_last_file_found = 0;
|
||||
(void)region_size;
|
||||
|
||||
int ci;
|
||||
const char *found_path = NULL;
|
||||
for (ci = 0; ci < s_cache_idx_count; ci++) {
|
||||
if (s_cache_idx[ci].region_start == region_start) {
|
||||
found_path = s_cache_idx[ci].path;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!found_path) return;
|
||||
|
||||
strncpy(dll_path, found_path, sizeof(dll_path) - 1);
|
||||
s_last_file_found = 1;
|
||||
load_overlay_dll(dll_path, 0x80000000u | (region_start & 0x1FFFFFFFu));
|
||||
}
|
||||
|
||||
int overlay_loader_dispatch(CPUState *cpu, uint32_t addr)
|
||||
{
|
||||
uint32_t phys = addr & 0x1FFFFFFFu;
|
||||
OverlayFn fn = dyntab_lookup(phys);
|
||||
if (fn) { fn(cpu); return 1; }
|
||||
|
||||
/* First miss for this region: check if a DLL is cached. */
|
||||
if (s_active && phys >= 0x98000u)
|
||||
try_load_region(phys);
|
||||
|
||||
/* Retry after potential DLL load. */
|
||||
fn = dyntab_lookup(phys);
|
||||
if (fn) { fn(cpu); return 1; }
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int overlay_loader_registered_count(void)
|
||||
{
|
||||
return s_count;
|
||||
}
|
||||
|
||||
void overlay_loader_get_status(int *active, int *registered,
|
||||
int *regions_checked,
|
||||
char *cache_dir_out, int cache_dir_len,
|
||||
char *game_id_out, int game_id_len,
|
||||
uint32_t *checked_out, int checked_max,
|
||||
int *checked_written,
|
||||
uint32_t *last_crc_out, int *last_file_found_out)
|
||||
{
|
||||
if (active) *active = s_active;
|
||||
if (registered) *registered = s_count;
|
||||
if (regions_checked) *regions_checked = s_nchecked;
|
||||
if (cache_dir_out) strncpy(cache_dir_out, s_cache_dir, (size_t)cache_dir_len - 1);
|
||||
if (game_id_out) strncpy(game_id_out, s_game_id, (size_t)game_id_len - 1);
|
||||
if (checked_out && checked_written) {
|
||||
int n = s_nchecked < checked_max ? s_nchecked : checked_max;
|
||||
for (int i = 0; i < n; i++) checked_out[i] = s_checked[i];
|
||||
*checked_written = n;
|
||||
}
|
||||
if (last_crc_out) *last_crc_out = s_last_crc;
|
||||
if (last_file_found_out) *last_file_found_out = s_last_file_found;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
+70
-22
@@ -2,13 +2,15 @@
|
||||
"""
|
||||
dump_overlays.py — Extract executed overlay regions from a live runtime.
|
||||
|
||||
Queries the overlay_dump TCP command which:
|
||||
1. Scans the dirty_ram bitmap for contiguous dirty pages above --lo
|
||||
2. Writes each region as <crc32>.bin in --dir
|
||||
3. Returns a JSON manifest
|
||||
Queries overlay_dump (dirty_ram bitmap scan) and cd_read_log (CD DMA
|
||||
transfer ring) to produce overlay_map.jsonl entries with:
|
||||
crc32 — CRC32 of the RAM bytes (for verification)
|
||||
load_addr — physical RAM address where the overlay lives
|
||||
size — byte count of the region
|
||||
start_lba — disc LBA of the first sector of this overlay file
|
||||
|
||||
Then writes logs/<game_id>/overlay_map.jsonl from the manifest so the
|
||||
build tool (B-2) has (load_addr, size, crc32, bytes_file) for each overlay.
|
||||
The start_lba lets extract_overlays.py seek directly to the correct disc
|
||||
position without brute-force scanning the whole disc image.
|
||||
|
||||
Usage:
|
||||
python3 tools/dump_overlays.py [--port PORT] [--lo PHYS] [--dir DIR] [--out JSONL]
|
||||
@@ -41,32 +43,72 @@ def send_cmd(port, cmd, **kwargs):
|
||||
s.close()
|
||||
return json.loads(data.decode())
|
||||
|
||||
def find_lba_for_region(cd_log, phys_addr, size):
|
||||
"""Return the disc start_lba for an overlay region if known, else -1.
|
||||
|
||||
Groups consecutive CD DMA entries (same setloc_lba, dest advancing by
|
||||
sector size) and returns the lba of the group whose dest range covers
|
||||
phys_addr for at least `size` bytes.
|
||||
"""
|
||||
SECTOR = 2048
|
||||
entries = cd_log.get('entries', [])
|
||||
if not entries:
|
||||
return -1
|
||||
|
||||
# Build file-load groups: consecutive entries where dest advances by 2048
|
||||
# and lba is the same (all sectors of one file share the SetLoc LBA).
|
||||
groups = []
|
||||
cur = None
|
||||
for e in entries:
|
||||
dest = int(e['dest'], 16)
|
||||
lba = e['lba']
|
||||
sz = e['size']
|
||||
if cur and cur['lba'] == lba and dest == cur['end']:
|
||||
cur['end'] += sz
|
||||
cur['total'] += sz
|
||||
else:
|
||||
if cur:
|
||||
groups.append(cur)
|
||||
cur = {'lba': lba, 'start': dest, 'end': dest + sz, 'total': sz}
|
||||
if cur:
|
||||
groups.append(cur)
|
||||
|
||||
# Find a group whose start <= phys_addr and end >= phys_addr + size
|
||||
for g in groups:
|
||||
if g['start'] <= phys_addr and g['end'] >= phys_addr + size:
|
||||
# LBA offset for the exact start of the overlay within this file
|
||||
offset_sectors = (phys_addr - g['start']) // SECTOR
|
||||
return g['lba'] + offset_sectors
|
||||
# Partial overlap — return the group start LBA as best guess
|
||||
if g['start'] <= phys_addr < g['end']:
|
||||
offset_sectors = (phys_addr - g['start']) // SECTOR
|
||||
return g['lba'] + offset_sectors
|
||||
return -1
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--port', type=int, default=4470)
|
||||
ap.add_argument('--lo', default='0x98000',
|
||||
help='Low physical address threshold (default: 0x98000)')
|
||||
ap.add_argument('--dir', default='logs/SCUS-94236/overlays',
|
||||
help='Directory for .bin files (created if missing)')
|
||||
ap.add_argument('--out', default='logs/SCUS-94236/overlay_map.jsonl',
|
||||
help='Output JSONL manifest path')
|
||||
ap.add_argument('--lo', default='0x98000')
|
||||
ap.add_argument('--dir', default='logs/SCUS-94236/overlays')
|
||||
ap.add_argument('--out', default='logs/SCUS-94236/overlay_map.jsonl')
|
||||
args = ap.parse_args()
|
||||
|
||||
os.makedirs(args.dir, exist_ok=True)
|
||||
os.makedirs(os.path.dirname(args.out) or '.', exist_ok=True)
|
||||
|
||||
print(f'Querying overlay_dump on port {args.port} (lo={args.lo}, dir={args.dir})...')
|
||||
print('Querying overlay_dump...')
|
||||
result = send_cmd(args.port, 'overlay_dump', lo=args.lo, dir=args.dir)
|
||||
|
||||
if not result.get('ok'):
|
||||
print(f'ERROR: {result.get("error")}')
|
||||
sys.exit(1)
|
||||
|
||||
print(f'ERROR: {result.get("error")}'); sys.exit(1)
|
||||
regions = result.get('regions', [])
|
||||
print(f'Found {len(regions)} overlay region(s).')
|
||||
|
||||
# Read existing JSONL to avoid duplicate entries
|
||||
print('Querying cd_read_log...')
|
||||
cd_log = send_cmd(args.port, 'cd_read_log', tail=4096)
|
||||
|
||||
print(f'Found {len(regions)} overlay region(s), {cd_log.get("total", 0)} CD DMA entries')
|
||||
|
||||
# Load existing manifest to avoid duplicates
|
||||
existing = set()
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out, encoding='utf-8') as f:
|
||||
@@ -81,19 +123,25 @@ def main():
|
||||
with open(args.out, 'a', encoding='utf-8') as f:
|
||||
for r in regions:
|
||||
key = (r['crc32'], r['addr'])
|
||||
phys_addr = int(r['addr'], 16)
|
||||
|
||||
lba = find_lba_for_region(cd_log, phys_addr, r['size'])
|
||||
lba_str = f'0x{lba:X}' if lba >= 0 else 'unknown'
|
||||
|
||||
if key in existing:
|
||||
print(f' skip {r["addr"]} {r["size"]} bytes {r["crc32"]} (already logged)')
|
||||
print(f' skip {r["addr"]} {r["size"]:>8} bytes lba={lba_str} (already logged)')
|
||||
continue
|
||||
|
||||
entry = {
|
||||
'crc32': r['crc32'],
|
||||
'load_addr': r['addr'],
|
||||
'size': r['size'],
|
||||
'bytes_file': f'overlays/{r["file"]}',
|
||||
'start_lba': lba_str,
|
||||
}
|
||||
f.write(json.dumps(entry) + '\n')
|
||||
existing.add(key)
|
||||
new_count += 1
|
||||
print(f' new {r["addr"]} {r["size"]} bytes {r["crc32"]} -> {r["file"]}')
|
||||
print(f' new {r["addr"]} {r["size"]:>8} bytes lba={lba_str} crc32={r["crc32"]}')
|
||||
|
||||
print(f'\n{new_count} new overlay(s) appended to {args.out}')
|
||||
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
extract_overlays.py — Scan a PSX disc image for overlay byte regions.
|
||||
|
||||
Reads overlay_map.jsonl (committed metadata: crc32, load_addr, size).
|
||||
Scans the disc via ISO 9660 file enumeration, computing CRC32 of each
|
||||
file's bytes. When a file matches an overlay entry, writes its bytes to
|
||||
overlays/<crc32>.bin for the recompiler (B-2).
|
||||
|
||||
Falls back to sector-by-sector scan for any overlays not found via ISO.
|
||||
|
||||
Supports:
|
||||
BIN/CUE MODE2/2352 — standard PSX disc rip (2352 bytes/sector)
|
||||
ISO 2048 bytes/sector — less common but handled
|
||||
|
||||
Usage:
|
||||
python3 tools/extract_overlays.py \\
|
||||
--disc <path/to/game.cue | game.bin | game.iso> \\
|
||||
--manifest <overlay_map.jsonl> \\
|
||||
--out <output_dir>
|
||||
|
||||
Outputs:
|
||||
<out>/<CRC32HEX>.bin for each matched overlay
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import argparse
|
||||
import binascii
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Disc sector reader
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SECTOR_RAW = 2352 # MODE2/2352
|
||||
SECTOR_DATA = 2048 # user data per sector
|
||||
DATA_OFFSET = 24 # offset of user data in raw MODE2 sector
|
||||
|
||||
class DiscReader:
|
||||
"""Reads user-data sectors from a PSX BIN/CUE or ISO image."""
|
||||
|
||||
def __init__(self, bin_path, raw=True):
|
||||
self.f = open(bin_path, 'rb')
|
||||
self.raw = raw # True = MODE2/2352, False = 2048-byte ISO
|
||||
self.sec_sz = SECTOR_RAW if raw else SECTOR_DATA
|
||||
fsize = os.path.getsize(bin_path)
|
||||
self.total = fsize // self.sec_sz
|
||||
|
||||
def close(self):
|
||||
self.f.close()
|
||||
|
||||
def read_sector_data(self, lba):
|
||||
"""Return 2048 user-data bytes for the given LBA."""
|
||||
offset = lba * self.sec_sz
|
||||
self.f.seek(offset)
|
||||
raw = self.f.read(self.sec_sz)
|
||||
if len(raw) < self.sec_sz:
|
||||
return b'\x00' * SECTOR_DATA
|
||||
if self.raw:
|
||||
return raw[DATA_OFFSET: DATA_OFFSET + SECTOR_DATA]
|
||||
return raw[:SECTOR_DATA]
|
||||
|
||||
def read_file_bytes(self, lba, size):
|
||||
"""Read exactly `size` bytes starting at `lba`."""
|
||||
sectors_needed = (size + SECTOR_DATA - 1) // SECTOR_DATA
|
||||
data = b''
|
||||
for i in range(sectors_needed):
|
||||
data += self.read_sector_data(lba + i)
|
||||
return data[:size]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CUE parser — extracts BIN path and track mode
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_cue(cue_path):
|
||||
"""Return (bin_path, is_raw) from a .cue file."""
|
||||
cue_dir = os.path.dirname(os.path.abspath(cue_path))
|
||||
bin_path = None
|
||||
is_raw = True # default: MODE2/2352
|
||||
with open(cue_path, encoding='utf-8', errors='replace') as f:
|
||||
for line in f:
|
||||
m = re.match(r'\s*FILE\s+"([^"]+)"\s+BINARY', line, re.IGNORECASE)
|
||||
if m:
|
||||
bin_path = os.path.join(cue_dir, m.group(1))
|
||||
m = re.match(r'\s*TRACK\s+\d+\s+(MODE\d/(\d+))', line, re.IGNORECASE)
|
||||
if m:
|
||||
is_raw = (int(m.group(2)) == SECTOR_RAW)
|
||||
if bin_path is None:
|
||||
raise ValueError(f'No BINARY FILE found in {cue_path}')
|
||||
return bin_path, is_raw
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ISO 9660 directory walker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _read_pvd(disc, lba=16):
|
||||
"""Return root dir LBA and size from the Primary Volume Descriptor."""
|
||||
data = disc.read_sector_data(lba)
|
||||
if data[0] != 1 or data[1:6] != b'CD001':
|
||||
raise ValueError('Sector 16 is not an ISO 9660 PVD')
|
||||
root = data[156:190]
|
||||
root_lba = struct.unpack_from('<I', root, 2)[0]
|
||||
root_size = struct.unpack_from('<I', root, 10)[0]
|
||||
return root_lba, root_size
|
||||
|
||||
def _iter_dir(disc, lba, size):
|
||||
"""Yield (name, file_lba, file_size, is_dir) for each entry in a dir."""
|
||||
raw = disc.read_file_bytes(lba, size)
|
||||
pos = 0
|
||||
while pos < len(raw):
|
||||
rec_len = raw[pos]
|
||||
if rec_len == 0:
|
||||
# Skip to next sector boundary
|
||||
pos = ((pos // SECTOR_DATA) + 1) * SECTOR_DATA
|
||||
continue
|
||||
if pos + rec_len > len(raw):
|
||||
break
|
||||
rec = raw[pos: pos + rec_len]
|
||||
flags = rec[25]
|
||||
is_dir = bool(flags & 0x02)
|
||||
elba = struct.unpack_from('<I', rec, 2)[0]
|
||||
esize = struct.unpack_from('<I', rec, 10)[0]
|
||||
name_len = rec[32]
|
||||
name = rec[33: 33 + name_len].decode('ascii', errors='replace')
|
||||
name = name.split(';')[0] # strip version suffix
|
||||
if name not in ('', '\x00', '\x01'):
|
||||
yield name, elba, esize, is_dir
|
||||
pos += rec_len
|
||||
|
||||
def enumerate_files(disc, dir_lba=None, dir_size=None, prefix=''):
|
||||
"""Recursively yield (path, lba, size) for every file on the disc."""
|
||||
if dir_lba is None:
|
||||
dir_lba, dir_size = _read_pvd(disc)
|
||||
for name, lba, size, is_dir in _iter_dir(disc, dir_lba, dir_size):
|
||||
path = f'{prefix}/{name}' if prefix else name
|
||||
if is_dir:
|
||||
yield from enumerate_files(disc, lba, size, path)
|
||||
else:
|
||||
yield path, lba, size
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Overlay manifest
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_manifest(jsonl_path):
|
||||
import json
|
||||
entries = []
|
||||
with open(jsonl_path, encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
e = json.loads(line)
|
||||
lba_raw = e.get('start_lba', 'unknown')
|
||||
entries.append({
|
||||
'crc32': int(e['crc32'], 16),
|
||||
'load_addr': int(e['load_addr'], 16),
|
||||
'size': int(e['size']),
|
||||
'start_lba': int(lba_raw, 16) if lba_raw != 'unknown' else -1,
|
||||
})
|
||||
return entries
|
||||
|
||||
def crc32_of(data):
|
||||
return binascii.crc32(data) & 0xFFFFFFFF
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--disc', required=True,
|
||||
help='Path to .cue, .bin, or .iso disc image')
|
||||
ap.add_argument('--manifest', required=True,
|
||||
help='overlay_map.jsonl with (crc32, load_addr, size)')
|
||||
ap.add_argument('--out', default='overlays',
|
||||
help='Output directory for .bin files (default: overlays/)')
|
||||
args = ap.parse_args()
|
||||
|
||||
os.makedirs(args.out, exist_ok=True)
|
||||
|
||||
# Open disc
|
||||
disc_path = args.disc
|
||||
if disc_path.lower().endswith('.cue'):
|
||||
bin_path, is_raw = parse_cue(disc_path)
|
||||
else:
|
||||
bin_path = disc_path
|
||||
is_raw = not disc_path.lower().endswith('.iso')
|
||||
|
||||
print(f'Disc : {bin_path} ({"MODE2/2352" if is_raw else "ISO 2048"})')
|
||||
disc = DiscReader(bin_path, raw=is_raw)
|
||||
|
||||
overlays = load_manifest(args.manifest)
|
||||
print(f'Manifest: {len(overlays)} overlay(s) to find')
|
||||
print()
|
||||
|
||||
# Build lookup by crc32
|
||||
remaining = {o['crc32']: o for o in overlays}
|
||||
found = {}
|
||||
|
||||
# --- Pass 0: Direct LBA extraction (fastest — uses cd_read_log data) ---
|
||||
print('Pass 0: direct LBA extraction (from cd_read_log)...')
|
||||
for crc, o in list(remaining.items()):
|
||||
lba = o.get('start_lba', -1)
|
||||
if lba < 0:
|
||||
continue
|
||||
size = o['size']
|
||||
data = disc.read_file_bytes(lba, size)
|
||||
actual_crc = crc32_of(data)
|
||||
if actual_crc == crc:
|
||||
remaining.pop(crc)
|
||||
out_name = f'{crc:08X}.bin'
|
||||
out_path = os.path.join(args.out, out_name)
|
||||
with open(out_path, 'wb') as f:
|
||||
f.write(data)
|
||||
found[crc] = (f'lba:{lba}(direct)', lba, size)
|
||||
print(f' FOUND lba={lba} load=0x{o["load_addr"]:08X}'
|
||||
f' size={size} -> {out_name}')
|
||||
else:
|
||||
print(f' MISMATCH lba={lba} expected=0x{crc:08X}'
|
||||
f' got=0x{actual_crc:08X} (will try ISO/brute-force)')
|
||||
|
||||
# --- Pass 1: ISO 9660 file scan (fast) ---
|
||||
print('Pass 1: ISO 9660 file enumeration...')
|
||||
try:
|
||||
for path, lba, size in enumerate_files(disc):
|
||||
if not remaining:
|
||||
break
|
||||
# Try exact file size and also overlay-truncated size
|
||||
for try_size in sorted({size, *[o['size'] for o in remaining.values()]}):
|
||||
if try_size > size + SECTOR_DATA:
|
||||
continue # file too small to cover this overlay
|
||||
data = disc.read_file_bytes(lba, try_size)
|
||||
crc = crc32_of(data[:try_size])
|
||||
if crc in remaining:
|
||||
o = remaining.pop(crc)
|
||||
out_name = f'{crc:08X}.bin'
|
||||
out_path = os.path.join(args.out, out_name)
|
||||
with open(out_path, 'wb') as f:
|
||||
f.write(data[:o['size']])
|
||||
found[crc] = (path, lba, size)
|
||||
print(f' FOUND {path:40s} load=0x{o["load_addr"]:08X}'
|
||||
f' size={o["size"]} -> {out_name}')
|
||||
break
|
||||
except Exception as e:
|
||||
print(f' ISO scan error: {e}')
|
||||
|
||||
# --- Pass 2: brute-force sector scan for anything not found ---
|
||||
if remaining:
|
||||
print()
|
||||
print(f'Pass 2: brute-force sector scan for {len(remaining)} unfound overlay(s)...')
|
||||
for ov_crc, ov in list(remaining.items()):
|
||||
sz = ov['size']
|
||||
nsec = (sz + SECTOR_DATA - 1) // SECTOR_DATA
|
||||
print(f' Scanning for crc32=0x{ov_crc:08X} size={sz} ({nsec} sectors)...',
|
||||
flush=True)
|
||||
found_it = False
|
||||
for lba in range(disc.total - nsec + 1):
|
||||
data = disc.read_file_bytes(lba, sz)
|
||||
if crc32_of(data) == ov_crc:
|
||||
remaining.pop(ov_crc)
|
||||
out_name = f'{ov_crc:08X}.bin'
|
||||
out_path = os.path.join(args.out, out_name)
|
||||
with open(out_path, 'wb') as f:
|
||||
f.write(data)
|
||||
found[ov_crc] = (f'lba:{lba}', lba, sz)
|
||||
print(f' FOUND lba={lba} load=0x{ov["load_addr"]:08X}'
|
||||
f' size={sz} -> {out_name}')
|
||||
found_it = True
|
||||
break
|
||||
if not found_it:
|
||||
print(f' NOT FOUND')
|
||||
|
||||
disc.close()
|
||||
|
||||
print()
|
||||
print(f'Results: {len(found)} found {len(remaining)} not found')
|
||||
if remaining:
|
||||
print('Not found (play through more content and re-run dump_overlays.py):')
|
||||
for crc, o in remaining.items():
|
||||
print(f' 0x{crc:08X} load=0x{o["load_addr"]:08X} size={o["size"]}')
|
||||
else:
|
||||
print('All overlays extracted successfully.')
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user