mirror/psxrecomp
1
0
Fork 0
mirror of https://github.com/RetroPortingToolKit/psxrecomp synced 2026-10-25 18:05:12 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs: production-friendly contributor docs + declutter root

Browse Source 
Make the repo approachable for outside contributors and an IDE integrator:

- New contributor docs:
  - docs/EXECUTION_MODEL.md — semi-technical tour of the static / native-overlay
    / interpreter execution model and how coverage gaps get plugged.
  - docs/ARCHITECTURE.md — component-level view (recompiler, runtime, LLE+HLE
    BIOS, overlay backend static->gcc->tcc, renderers, oracle model).
  - docs/BUILDING.md — accurate dependency table, per-platform prereqs, build
    steps, and the submodule + local-junction framework-linking setup.
  - CONTRIBUTING.md — distilled core rules, verification expectations, PR flow.
- Declutter root: move ~14 internal dev-logs into docs/internal/ (history
  preserved via rename) with an index README; repoint CLAUDE.md references.
- README: correct build recipe, LLE+HLE + overlay-backend summary, links to the
  new docs.
- code_provider.c: update the stale backend comment to the real static->gcc->tcc
  resolution (dead SLJIT.md reference removed).
- config_schema.md: drop the obsolete "once Phase A2/A3 ships --config" note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
2026-07-07 18:25:16 -07:00
parent d1ce28efc2
commit 77d036bfb6
23 changed files with 689 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+7 -7
View File
@@ -43,11 +43,11 @@ loss), never as the starting point — see §0's amendment below and
recomp-template `PRINCIPLES.md` → "LLE Is the Baseline; HLE Is a Subsystem
Replacement, Not a Starting Point".
**The authoritative game plan is `FAITHFUL_TIMING_PLAN.md` — READ IT EACH
**The authoritative game plan is `docs/internal/FAITHFUL_TIMING_PLAN.md` — READ IT EACH
SESSION** (north star, phased plan P1–P6, current status/log). Update its
Status/Log section every session. The full-coverage accuracy burndown across ALL
axes (semantics, cycle, IRQ, MMIO, peripherals, static-vs-dynamic, determinism)
lives in `ACCURACY_BURNDOWN.md` — every item must be cross-referenced against an
lives in `docs/internal/ACCURACY_BURNDOWN.md` — every item must be cross-referenced against an
external comparative (psx-spx / in-tree Beetle source / DuckStation / HW test
ROMs), not asserted. Axis 5 (peripherals — esp. SIO/controller, the hybrid-pad
bug) is the suspected-weakest second front after the cycle axis.
@@ -149,14 +149,14 @@ first-class High-Level Emulation tier alongside LLE, modeled on
If you find yourself wanting to violate any of the above three
paragraphs **beyond the two amendments just above**,
**stop and re-read PLAN.md**. Every prior attempt failed by
**stop and re-read docs/internal/PLAN.md**. Every prior attempt failed by
violating exactly these rules under pressure.
---
## 1. The BIOS is recompilation target #1, the game is target #2
Phase 1-3 of PLAN.md exist to get the BIOS recompiled and booting on
Phase 1-3 of docs/internal/PLAN.md exist to get the BIOS recompiled and booting on
its own. The BIOS must reach the Sony logo and the BIOS shell, running
entirely as native C, before any game work begins. There is no path
that loads a game EXE before the BIOS is fully working in v4. **Do not
@@ -236,7 +236,7 @@ a fake event. The fake delivery was not progress, it was theater.
At the start of every session, before any code change:
1. Read this file (CLAUDE.md).
2. Read PLAN.md to confirm what phase we are in and what the next
2. Read docs/internal/PLAN.md to confirm what phase we are in and what the next
concrete milestone is.
3. Verify `docs/psx_bios_disasm.txt` exists (primary reference).
4. Verify Ghidra MCP is reachable. If not, stop and ask.
@@ -330,7 +330,7 @@ Code without proof is invalid.
Before any Phase 2 work:
- FIRST_MILESTONE.md must be complete
- docs/internal/FIRST_MILESTONE.md must be complete
- boot_slice must compile
- all instructions must be supported
@@ -462,7 +462,7 @@ the visibility you need.
## 17. Phase 5 gate — fix hardware stubs before loading a game
`STUBS_TO_FIX.md` lists every known stub in the runtime. Before any
`docs/internal/STUBS_TO_FIX.md` lists every known stub in the runtime. Before any
Phase 5 work (loading Tomba or any game EXE), every stub marked
"Phase 5+" in that file **must be implemented and oracle-verified**:
CONTRIBUTING.md
+93
View File
@@ -0,0 +1,93 @@
# Contributing to PSXRecomp
Thanks for your interest! PSXRecomp is a static recompiler, not an emulator, and
it holds itself to a high correctness bar. This guide covers how the project is
organized, the rules that keep it faithful, and how to get a change merged.
New to the codebase? Read [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) and
[`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md) first, then
[`docs/BUILDING.md`](docs/BUILDING.md) to get a build going.
## Repository layout
- **`mstan/psxrecomp`** (this repo) — the framework: the recompiler tool
(`recompiler/`), the runtime engine (`runtime/`), shared tools, and docs.
Framework changes go here.
- **Per-game repos** (`mstan/TombaRecomp`, `mstan/MegaManX6Recomp`, …) — one per
title. Each contains only that game's config, seeds, and build glue and links
this framework in as a submodule at `psxrecomp/`. Game-specific work goes in
the game repo; nothing game-specific belongs in the framework.
See [Linking the framework](docs/BUILDING.md#linking-the-framework) for the
submodule + local-junction dev setup.
## The core rules
These are non-negotiable because violating them is how past efforts turned into
un-debuggable emulators. The full engineering constitution is
[`CLAUDE.md`](CLAUDE.md); the debugging philosophy is
[`PRINCIPLES.md`](PRINCIPLES.md). The short version:
1. **Build the faithful core; no per-game hacks in the foundation.** The correct
fix is the general, hardware-accurate one, not a surgical workaround for one
title. (Per-game *enhancements* — widescreen, faster loads — are legitimate
once the faithful core is proven; see [`ENHANCEMENTS.md`](ENHANCEMENTS.md).)
2. **No stubs.** A function is fully implemented or it fails loudly. No
`return 0;` placeholders, no `// TODO` behavior, no hand-delivered fake
events. If execution reaches code we can't handle, we stop and fix
discovery/codegen — we don't fake the result.
3. **Never hand-edit generated code.** Files under `generated/` and the
recompiler's output are build artifacts. If the emitted C is wrong, fix the
recompiler (`recompiler/src/`), then regenerate.
4. **LLE is the baseline; HLE is a validated subsystem replacement, never a
starting point.** The recompiled BIOS is the reference and oracle. An HLE tier
is allowed only where the LLE path has a genuine landmine, must be general
(every game), must operate on the real guest structures, and must be
continuously checked against the Beetle oracle — never a "produce the answer
the BIOS would have" shim.
5. **No `printf`/log-file debugging.** Runtime inspection goes through the TCP
debug server ([`TCP_COMMANDS.md`](TCP_COMMANDS.md)) and always-on ring
buffers, not `fprintf`.
6. **Fix broken tooling immediately** — don't route around it with indirect
evidence, and don't infer correctness from two implementations sharing a bug.
## Verifying a change
Correctness is demonstrated, not asserted:
- **Oracle-check against Beetle.** Run `psx-beetle` alongside `psx-runtime` and
compare via the shared TCP protocol; for timing/codegen, use the co-sim build.
See [`docs/internal/COSIM_ORACLE.md`](docs/internal/COSIM_ORACLE.md).
- **Verify visually, end to end.** A milestone is "the pixels appear on screen"
— take a screenshot, don't trust a counter or a ring-buffer flag alone.
- **A decoder / codegen change requires a playthrough check**, not just a clean
build: cosim/lockstep first, then run the affected area.
## Adding a new game
At a high level: create a game repo from the template, add this framework as the
`psxrecomp/` submodule, seed function starts (typically from a Ghidra export in
`seeds/`), write `game.toml`, generate the game C with
`psxrecomp-game --config game.toml`, then build and bring the title up boot →
menu → gameplay, oracle-checking as you go. The existing game repos
(TombaRecomp, MegaManX6Recomp) are the working references.
## Pull requests
- Branch off `master`; keep framework changes and game changes in their
respective repos.
- **Do not commit** local artifacts: BIOS images, disc images, generated C
(`generated/`), memory cards, Ghidra databases, build outputs, or overlay
capture files. These are gitignored for good reasons (copyright + size).
- Explain *what hardware behavior* the change matches and *how you verified it*
(oracle diff, screenshot, playthrough). Changes that can't state their
verification are hard to accept.
- Keep the fix minimal and at the right layer (recompiler vs runtime vs game
config). If you find yourself editing generated code or adding a stub, step
back — the real fix is upstream of that.
## Questions / community
Development happens with the **R.A.I.D. (Retro AI Development)** community —
Discord invite in the [README](README.md). Open a GitHub issue for bugs and
build problems (include `gcc -v` / OS / generator for build failures).
README.md
+36 -19
View File
@@ -21,12 +21,19 @@ BIOS layer, no stubs, and no general-purpose interpreter fallback for static
code.
PSXRecomp is a framework. Game-specific projects live in their own
repositories and pull this one in to build a game binary. The active end-to-end
targets are:
repositories and link this one in as a **git submodule** to build a game binary.
The active end-to-end targets are:
- [TombaRecomp](https://github.com/mstan/TombaRecomp) — *Tomba!*
- [MegaManX6Recomp](https://github.com/mstan/MegaManX6Recomp) — *Mega Man X6*
**New here?** The fastest way in:
[`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md) (how a game actually
runs — static / native-overlay / interpreter), then
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md),
[`docs/BUILDING.md`](docs/BUILDING.md), and
[`CONTRIBUTING.md`](CONTRIBUTING.md).
## Philosophy — toward 100% static recompilation
The goal is simple and absolute: **a PS1 game should run as native code, not be
@@ -144,29 +151,30 @@ Windows, `ucontext` on POSIX (`runtime/src/psx_fiber.c`) — so the recompiled
BIOS's cooperative thread switching (the CD-boot handoff in particular) behaves
the same on every platform.
Requirements:
Requirements at a glance (full details, dependency table, and per-platform
prerequisites in [`docs/BUILDING.md`](docs/BUILDING.md)):
- A C/C++ toolchain: MSVC or MinGW (Windows), Apple Clang (macOS), Clang/GCC (Linux).
- CMake 3.20+. On macOS/Linux also `ninja` and `pkg-config`.
- SDL2: the bundled dev pack on Windows; `brew install sdl2 pkg-config ninja`
on macOS; `libsdl2-dev` (or distro equivalent) on Linux.
- A C/C++ toolchain: MSVC or MinGW/MSYS2 (Windows), Apple Clang (macOS),
Clang/GCC (Linux). CMake 3.20+; on macOS/Linux also `ninja` and `pkg-config`.
- SDL2 (system / bundled). RmlUi and FreeType come in as **git submodules** —
clone with `--recurse-submodules`.
- A legally obtained `SCPH1001.BIN` BIOS dump. Not included.
- For game projects, a legally obtained game disc/EXE dump. Not included.
Build the framework runtime:
Build the framework (recompiler tool + BIOS-only runtime):
```sh
# Windows (MSYS2/MinGW)
cmake -S recompiler -B recompiler/build -G "Unix Makefiles" && cmake --build recompiler/build
cmake -S runtime -B runtime/build -G "Unix Makefiles" && cmake --build runtime/build --target psx-runtime
git clone --recurse-submodules https://github.com/mstan/psxrecomp.git && cd psxrecomp
# macOS / Linux (Ninja)
cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C recompiler/build
cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C runtime/build psx-runtime
cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build recompiler/build
cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build runtime/build --target psx-runtime
```
Game projects generate their own `generated/<serial>_*.c` files and link this
runtime source tree through CMake.
On Windows swap `-G Ninja` for your generator if you prefer (e.g.
`-G "Unix Makefiles"`); always keep an explicit `-DCMAKE_BUILD_TYPE` so the
generated C is optimized. Game projects generate their own
`generated/<serial>_*.c` files and link this runtime through CMake — see
[`docs/BUILDING.md`](docs/BUILDING.md#build-and-run-a-game).
## Keyboard Map
@@ -209,10 +217,19 @@ The recompiler emits C functions and dispatch tables for BIOS and game code.
The runtime loads the BIOS/game assets into emulated PS1 memory, links the
generated C as native code, and simulates hardware through MMIO handlers for
GPU, DMA, timers, CD-ROM, MDEC, SIO0, memory cards, SPU, GTE, and interrupt
delivery. BIOS A0/B0/C0 vectors go through the recompiled BIOS, not HLE shims.
delivery. The recompiled `SCPH1001.BIN` is the low-level (LLE) kernel and the
correctness oracle; an optional HLE tier lays instant boot-skip and a few BIOS
services on top, always falling through to the recompiled BIOS.
See `CLAUDE.md`, `PLAN.md`, and `FAITHFUL_TIMING_PLAN.md` for the development
rules and current project context.
Code that can't be seen ahead of time (disc-streamed **overlays**) is captured
and compiled to native code the first time it appears (`static → gcc → tcc`
backend), with a small interpreter as the correctness fallback until it is. Full
story in [`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md); component-level
detail in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).
See [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`CLAUDE.md`](CLAUDE.md) for the
development rules, and [`docs/internal/`](docs/internal/) for the phased plans
and deep design notes (`PLAN.md`, `FAITHFUL_TIMING_PLAN.md`, …).
## Disc Speed
docs/ARCHITECTURE.md
+159
View File
@@ -0,0 +1,159 @@
# Architecture
How PSXRecomp is put together, end to end. For the higher-level "why three ways
of running code" story, read [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) first;
this doc is the component-level view.
## Two programs: the recompiler and the runtime
PSXRecomp is split into two CMake projects that are built and run separately:
```
recompiler/ runtime/
─────────── ────────
MIPS ──▶ C C + hardware simulation ──▶ native game binary
(build-time tool) (linked with the generated C)
```
- **`recompiler/`** (C++20) is an offline tool. It reads MIPS R3000A machine code
and emits C source files under `generated/`.
- **`runtime/`** (C99 + C++17) is the engine. It loads the game's assets into an
emulated PS1 address space, links the generated C in as native functions, and
simulates the console's hardware around them.
A **game repository** (e.g. TombaRecomp) contains no framework source — it links
this framework in (as a submodule) and provides the game's config, seeds, and
build glue. See [`BUILDING.md`](BUILDING.md#linking-the-framework).
## The recompiler (`recompiler/`)
Two entry points share the same translation core:
- `src/main_bios.cpp` — ingests the flat BIOS ROM (`SCPH1001.BIN`, loaded at
`0xBFC00000`) and emits `generated/SCPH1001_*.c`.
- `src/main_psx.cpp` — ingests a `PS-X EXE` extracted from a game disc and emits
`generated/<serial>_*.c`.
The translation pipeline:
| Stage | File | Job |
|---|---|---|
| Decode | `mips_decoder.cpp` (+ vendored `rabbitizer`) | Decode MIPS instructions |
| Control flow | `control_flow.cpp`, `basic_block.cpp` | Build basic blocks / CFG |
| Discovery | `function_discovery.cpp`, `function_analysis.cpp` | Find function entry points (seeded from Ghidra exports) |
| Codegen | `code_generator.cpp`, `full_function_emitter.cpp`, `strict_translator.cpp` | Emit C: one function per guest function, plus a dispatch table |
Output is **two files** per program: a `_full.c` (the function bodies) and a
`_dispatch.c` (the address→function dispatch table). Both are build artifacts and
are **never hand-edited** — if the C is wrong, the fix is in the recompiler.
GTE (the PS1 geometry coprocessor) instructions are emitted inline;
COP0 kernel-mode instructions the BIOS needs are handled in codegen.
## The runtime (`runtime/`)
The runtime is assembled by one CMake helper, `psxrecomp_add_runtime_target()`
(defined in `runtime/runtime.cmake`), which a game's `CMakeLists.txt` calls with
the paths to its generated C. The runtime provides:
- **Memory & address space** (`memory.c`) — the 2 MB main RAM, scratchpad, BIOS
ROM, and MMIO regions, with the dispatch that routes a guest PC to its native
function (static, overlay, or interpreter).
- **Hardware simulation via MMIO handlers** — GPU (`gpu.c`, `gpu_sw_renderer.c`,
`gpu_gl_renderer.c`), DMA (`dma.c`), timers (`timers.c`), CD-ROM
(`cdrom.c`, `iso_reader.cpp`), MDEC, SIO0 controllers/memory cards
(`sio.c`, `memcard.c`), SPU (`spu.c`), GTE (`gte.cpp`), and interrupt delivery
(`interrupts.c`).
- **Host services** — window/input/audio via SDL2, a cooperative-thread
scheduler on host fibers (`psx_fiber.c`: Win32 Fibers / POSIX `ucontext`), and
the optional debug TCP server (`debug_server.c`).
### BIOS: LLE baseline + a swappable HLE tier
The recompiled `SCPH1001.BIN` is the **low-level (LLE) baseline**: it *is* the
kernel, and it is the reference implementation and the correctness oracle. There
are no per-vector HLE shims replacing it.
On top of that, PSXRecomp carries an optional **HLE tier** (`bios_hle`, on by
default for player convenience) that can synthesize the post-boot kernel handoff
state to skip the BIOS boot sequence, and intercept a small set of BIOS services
— always falling through to the recompiled BIOS for anything it doesn't
implement. LLE stays fully linked and is what every accuracy check runs against.
Turn it off with `[runtime] bios_hle = false` or `PSX_BIOS_HLE=0`; with it off the
build behaves as pure LLE. (Design notes:
[`docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md`](internal/HLE_SCHEDULER_CARVEOUT_PLAN.md).)
### Static / overlay / interpreter dispatch
This is the heart of the system and has its own doc,
[`EXECUTION_MODEL.md`](EXECUTION_MODEL.md). In brief: a guest PC resolves to
statically-recompiled native code (BIOS + main EXE), a runtime-compiled **native
overlay** (`overlay_capture.c` → `tools/compile_overlays.py` →
`overlay_loader.c`), or the small **dirty-RAM interpreter**
(`dirty_ram_interp.c`) — in that priority order.
### Overlay compile backend
When an overlay needs compiling, the runtime spawns a C compiler on the
recompiler-emitted C and loads the resulting DLL — it does not JIT in-process.
The backend tier is resolved in `overlay_backend.c` (see `main.cpp` around the
`code_provider` setup):
```
static → gcc → tcc
```
- **static** — the overlay was baked into the binary at build time (best case).
- **gcc** — used when a system `gcc` is on `PATH` (the development default).
- **tcc** — TinyCC, the toolchain-free fallback **bundled beside shipped
executables** in `overlay_toolchain/` (an embedded Python + `tcc.exe`) so
players never need a compiler installed.
Compiled overlays are stored in a content-addressed cache namespaced by
compiler and target ABI (`<game>/gcc/<arch-abi>/…` vs `<game>/tcc/…`); a `gcc`
shard wins over a `tcc` shard for the same region. See
[`docs/FEATURES.md`](FEATURES.md), [`docs/OVERLAY_CACHE_V2.md`](OVERLAY_CACHE_V2.md),
and [`docs/ASYNC_OVERLAY_COMPILE.md`](ASYNC_OVERLAY_COMPILE.md).
## Renderers
Three GPU backends behind one interface:
- **Software rasterizer** — CPU, most portable, the reference look.
- **OpenGL** — GPU-authoritative VRAM/FBO renderer, the default; moves
rasterization and supersampling onto the GPU. Falls back to software if GL
init fails. (See [`docs/internal/GL_RENDERER_HANDOFF.md`](internal/GL_RENDERER_HANDOFF.md).)
- **Vulkan** — experimental, off by default (`PSX_ENABLE_VULKAN=OFF`).
Widescreen (a genuine wider GTE FOV, not a stretch) is opt-in and gen-time; see
[`WIDESCREEN.md`](../WIDESCREEN.md) and
[`docs/internal/NATIVE_WIDE_PLAN.md`](internal/NATIVE_WIDE_PLAN.md).
## The oracle model (how correctness is checked)
PSXRecomp validates itself against **Beetle PSX** (the mednafen-psx libretro
core), run as a **separate process** with an identical TCP JSON debug protocol:
- `psx-runtime` — the recompiled runtime (debug server on port 4370).
- `psx-beetle` — Beetle PSX (debug server on port 4380).
A tool written against one works against the other by switching ports; cross-
checking is done by querying both, never by sharing state in one process. There
is also a first-divergence **co-sim** build that cycle-locksteps the compiled
backend against the interpreter. See
[`docs/internal/COSIM_ORACLE.md`](internal/COSIM_ORACLE.md),
[`docs/config_schema.md`](config_schema.md), and [`TCP_COMMANDS.md`](../TCP_COMMANDS.md).
## Configuration
Every process has a **BIOS config**; a game adds a **game config** (`game.toml`),
and the two are merged (game wins on scalar keys; program blocks and generated
dispatch tables are additive). Full schema:
[`docs/config_schema.md`](config_schema.md).
## Where to go next
- [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) — static/native/interp in depth.
- [`BUILDING.md`](BUILDING.md) — dependencies + build steps.
- [`../CONTRIBUTING.md`](../CONTRIBUTING.md) — dev workflow and rules.
- [`../CLAUDE.md`](../CLAUDE.md) — the exhaustive engineering constitution.
docs/BUILDING.md
+181
View File
@@ -0,0 +1,181 @@
# Building PSXRecomp
PSXRecomp builds natively on **Windows** (MSVC or MinGW/MSYS2), **macOS** (Apple
Silicon & Intel), and **Linux**. There are two things you can build:
- the **framework** itself (this repo) — a BIOS-only runtime plus the recompiler
tool; useful for BIOS work and as the thing game repos link against;
- a **game** — done from that game's own repository, which links this framework
in as a submodule (see [Linking the framework](#linking-the-framework)).
You always supply your own **`SCPH1001.BIN` BIOS** and, for a game, your own
legally-obtained **disc image**. This project ships none of those.
## Toolchain requirements
- A C/C++ toolchain:
- **Windows:** MSYS2 MinGW-w64 (`mingw-w64-x86_64` toolchain) *or* MSVC.
- **macOS:** Apple Clang (Xcode command-line tools).
- **Linux:** GCC or Clang.
- **CMake ≥ 3.20**. On macOS/Linux also **Ninja** and **pkg-config**.
- Language standards used: recompiler is **C++20**; runtime is **C99 + C++17**.
- `ccache` is auto-detected and used if present (optional, speeds rebuilds).
## Dependencies
| Dependency | How it's provided | Used for |
|---|---|---|
| **SDL2** | System. Windows/MSVC: prebuilt pack auto-found at `../sdl2-msvc/SDL2-*`. Windows/MinGW + macOS + Linux: found via pkg-config. | Window, input, GL/Vulkan context, audio, threads |
| **RmlUi** | **git submodule** `lib/RmlUi` (static) | The launcher UI |
| **FreeType** | **git submodule** `lib/freetype` (static) | RmlUi's font engine |
| **fmt** 9.1.0 | vendored `recompiler/lib/fmt` | String formatting (runtime uses it header-only) |
| **toml11** | vendored `recompiler/lib/toml11` | Parsing `game.toml` / configs |
| **ELFIO** | vendored `recompiler/lib/ELFIO` | ELF parsing (recompiler only) |
| **rabbitizer** | vendored `recompiler/lib/rabbitizer` | MIPS instruction decoding (recompiler only) |
| **TinyCC (TCC) 0.9.27** | Not in this repo — downloaded at release-packaging time and bundled beside the game exe in `overlay_toolchain/`. | Toolchain-free overlay compilation for players (run as a subprocess) |
| **Python 3** | System (development) or an embedded copy bundled in releases | Runs `tools/compile_overlays.py` in the overlay pipeline |
| **OpenGL** | System (`opengl32` on Windows; `find_package(OpenGL)` elsewhere) | The GL renderer |
| **Vulkan** | Headers only, optional, **off by default** (`PSX_ENABLE_VULKAN=OFF`); loaded dynamically via SDL. Shader compilation needs `glslc` from the Vulkan SDK. | The experimental Vulkan renderer |
Developers building overlays locally just need `gcc` on `PATH` (the `gcc` tier);
the bundled `tcc` matters only for end-user release packages.
### Get the source (with submodules)
RmlUi and FreeType are submodules, so clone recursively (or init after the fact):
```sh
git clone --recurse-submodules https://github.com/mstan/psxrecomp.git
# or, in an existing clone:
git submodule update --init --recursive
```
## Per-platform prerequisites
**Windows (MSYS2/MinGW — recommended for release parity):**
```sh
# In an MSYS2 MinGW64 shell:
pacman -S --needed mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake \
mingw-w64-x86_64-ninja mingw-w64-x86_64-SDL2 mingw-w64-x86_64-ccache
```
**macOS:**
```sh
brew install sdl2 pkg-config ninja cmake
```
**Linux (Debian/Ubuntu):**
```sh
sudo apt install build-essential cmake ninja-build pkg-config libsdl2-dev
```
## Build the framework
Two CMake trees: the recompiler (a tool) and the runtime (the engine).
```sh
# 1. Recompiler tool → produces psxrecomp-bios and psxrecomp-game
cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build recompiler/build
# 2. (Optional) regenerate the BIOS C from your SCPH1001.BIN
# Requires bios/SCPH1001.BIN to be present.
bash tools/regen_bios.sh
# 3. Runtime → produces psx-runtime (BIOS-only for this repo)
cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build runtime/build --target psx-runtime
```
On Windows with MSVC or plain MinGW makefiles, swap `-G Ninja` for your generator
(e.g. `-G "Unix Makefiles"`); everything else is identical.
> **Build type matters.** With no `-DCMAKE_BUILD_TYPE`, the framework defaults to
> **Release** (optimized). The huge generated C compiles unusably slowly at `-O0`,
> so never build it debug-by-accident. Use `-DCMAKE_BUILD_TYPE=RelWithDebInfo`
> when you need symbols. Release turns the debug TCP server **off**; RelWithDebInfo
> and Debug turn it **on** (`PSX_DEBUG_TOOLS`).
### Useful CMake options
| Option | Default | Effect |
|---|---|---|
| `PSX_DEBUG_TOOLS` | ON for Debug/RelWithDebInfo, OFF for Release | TCP debug server + heartbeat + per-block recording |
| `PSX_STATIC_RUNTIME` | ON for MinGW Release | Self-contained exe (statically links SDL2 + libgcc/libstdc++) |
| `PSX_LAUNCHER` | ON | Build the RmlUi launcher UI |
| `PSX_ENABLE_VULKAN` | OFF | Build the experimental Vulkan renderer |
| `PSX_BUILD_COSIM` | OFF | Build the first-divergence co-sim oracle target |
## Build and run a game
From the **game's** repository (not this one). Each game repo links this
framework in and provides its own `game.toml`, seeds, and generated C. Example
for TombaRecomp:
```sh
# Extract the game's PS-X EXE from your disc (helper included in the game repo):
python3 tools/extract_psx_exe.py tomba/tomba.bin SCUS_942.36 tomba/SCUS_942.36
# Regenerate the game's C from the disc/EXE (game repos invoke the recompiler
# binary directly with their config):
../psxrecomp/recompiler/build/psxrecomp-game --config game.toml
# Configure + build the game runtime:
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --target psx-runtime
./build/psx-runtime --game game.toml --disc tomba/tomba.cue
```
## Linking the framework
Game repositories do **not** vendor the framework. They reference it as a git
**submodule** at path `psxrecomp/`, which the game's `CMakeLists.txt` points
`PSXRECOMP_ROOT` at:
```cmake
set(PSXRECOMP_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/psxrecomp" CACHE PATH "Path to psxrecomp")
include("${PSXRECOMP_ROOT}/runtime/runtime.cmake")
```
A fresh clone of a game repo gets the framework automatically:
```sh
git clone --recurse-submodules https://github.com/mstan/TombaRecomp.git
```
**Local dev tip — share one framework checkout across games.** If you work on
several game repos plus the framework at once, you don't want N multi-GB copies.
Replace each game's `psxrecomp/` submodule working directory with a **junction /
symlink** to a single shared framework checkout:
```sh
# Windows (junction):
cmd /c mklink /J psxrecomp F:\path\to\shared\psxrecomp
# macOS / Linux (symlink):
ln -s /path/to/shared/psxrecomp psxrecomp
```
Git treats the link's target as the submodule working tree, so
`git submodule status` stays clean **as long as the shared checkout is at the
commit the game pins**. Bump a game to a newer framework by checking out the new
commit in the shared repo and committing the updated submodule pointer in the
game repo. This is the setup the game repos are configured for.
## Troubleshooting
**MinGW: `Error: too many sections` / a `*.c.o` (or `.obj`) fails to assemble.**
Windows COFF object files have a 32,768-section limit; very large translation
units (the generated game C, and `debug_server.c` in debug-info builds) can
exceed it on older binutils. The fix is to assemble those units as *big objects*:
add `-Wa,-mbig-obj` to their compile options (recent binutils, ≥ ~2.40, handle
these files without it). If you hit this on a framework source file, please open
an issue with your `gcc -v` / `as --version` — the build should apply the flag
for you.
**`SDL2 MSVC dev package not found`.** The MSVC build expects the prebuilt SDL2
pack beside the repo at `../sdl2-msvc/SDL2-*`. Use the MSYS2/MinGW toolchain
(which finds SDL2 via pkg-config) or place the pack there.
**Overlays never compile / stay slow.** In development you need `gcc` on `PATH`
for the `gcc` tier; otherwise areas stay in the interpreter. See
[`EXECUTION_MODEL.md`](EXECUTION_MODEL.md).
docs/EXECUTION_MODEL.md
+160
View File
@@ -0,0 +1,160 @@
# The Execution Model — static, native-overlay, and interpreter
*A semi-technical tour of how PSXRecomp actually runs a PlayStation game. If you
only read one design doc, read this one.* For the low-level plumbing, see
[`ARCHITECTURE.md`](ARCHITECTURE.md).
## The one-sentence version
PSXRecomp turns a PS1 game into a **native program** by translating its MIPS code
to C ahead of time; the small amount of code it *can't* see ahead of time is
either compiled to native code the first time it appears, or run by a tiny
built-in interpreter until it is — so the game is always **correct**, and gets
**faster** the more it's played.
## Why a PS1 game can't just be "decompiled once"
A PlayStation game is not one fixed blob of code sitting in memory. The main
program (the `PS-X EXE` on the disc) is, but games are far bigger than the PS1's
2 MB of RAM, so they **stream code off the disc as you play**. A level's logic,
a boss's AI, a menu — these are loaded from the CD into a fixed region of RAM,
run, and then *overwritten* by the next chunk when you move on. These streamed
chunks are called **overlays**.
That creates a hard problem for any ahead-of-time recompiler:
- The main EXE exists on disc at build time → we can translate all of it.
- An overlay's code **does not exist anywhere at build time** in a form we can
find. It only appears in RAM at a specific moment during play, and the same
RAM addresses get reused by dozens of different overlays over a playthrough.
So "recompile the game" can never be a single up-front pass. PSXRecomp handles
this with **three tiers of execution**, in strict priority order.
```
┌─────────────────────────────────────────────────────────┐
│ Guest wants to run code at some address │
└─────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
1. STATIC (AOT) 2. NATIVE OVERLAY 3. INTERPRETER
Translated at Compiled to native Runs the guest's
build time from code the first time own instructions
the disc's EXE the overlay is seen, directly, as a
and the BIOS. then cached forever. correctness safety
The bulk of Fills the "streamed net for anything
execution. from disc" gap. not yet native.
── fastest ── ── fast ── ── slowest, always
correct ──
```
## Tier 1 — Static (ahead-of-time)
At build time the **recompiler** reads two things and translates their MIPS
machine code into C, function by function:
- the real PS1 **BIOS** (`SCPH1001.BIN`) — this becomes the kernel, and
- the game's **main executable** extracted from your disc.
That C is compiled and linked into the runtime as ordinary native functions, so
when the game calls one of them the CPU just runs native code — no emulation of
individual instructions. **This is the overwhelming majority of what executes**:
boot, the engine, the main loop, most gameplay systems. It is as fast as the C
compiler can make it.
Everything a pure AOT recompiler can see, it recompiles. The only things left
over are the two categories below.
## Tier 2 — Native overlays (compiled the moment they're discovered)
Overlays can't be seen at build time — but they *can* be seen the instant the
game loads one. PSXRecomp watches for that moment and turns each overlay into
native code so that, from then on, it runs at Tier-1 speed:
1. **Capture.** Every time the game DMAs code from the disc into the overlay RAM
window, the runtime records the exact bytes (before the game can modify them)
and the addresses that were actually executed. These go into a per-game file,
`overlay_captures.json`.
2. **Compile.** Those captured bytes are fed back through the *same recompiler*
used for the main EXE, producing C for that overlay. The C is compiled to a
small native library (a DLL) — using `gcc` in development, or a bundled
`tcc` (TinyCC) toolchain in shipped releases so players need no compiler
installed. Each library is keyed by the overlay's content hash.
3. **Cache & dispatch.** The compiled overlay lands in a content-addressed
cache. On this run and every future run, when that overlay loads, the runtime
finds the matching native library and dispatches into it **before** falling
back to the interpreter.
The practical effect: **the first time anyone visits an area, it may run in the
slow tier for a moment; after it's been captured and compiled, that area is
native for them — and, once the capture is shared into a release, for everyone.**
Coverage grows toward "100% native" the more the game is played.
## Tier 3 — The interpreter (the correctness safety net)
There is always a window between "an overlay just appeared in RAM" and "we have
native code for it." There's also a genuinely hard corner: code that is
**rewritten to different bytes on every load** (self-modifying / per-load
relocated), which by definition has no single ahead-of-time translation.
For both, PSXRecomp includes a **small MIPS interpreter** that runs the guest's
*own* instructions directly on the CPU register state. It is deliberately narrow:
- It runs **only** code that was installed into RAM at runtime (dirty pages) and
is not yet covered by a native overlay. It is **never** on the BIOS or
main-EXE path — those are always static.
- It is a **safety net and a coverage feeder**, never a substitute for
recompiling. If the interpreter is running something, that's a signal to
capture-and-compile it (Tier 2), not to leave it interpreted.
Crucially, this is **not** high-level emulation (HLE). The interpreter executes
the program's real instructions, exactly as the game's authors wrote them — the
*only* difference from static recompilation is where the instructions came from
(RAM-at-runtime vs disc-at-build-time). We never synthesize "the answer the code
would have produced."
## The guiding rule: precision over recall
The tiers are ordered so the system can only ever fail *toward being slow*, never
*toward being wrong*:
- Code we **haven't** compiled yet safely drops to the interpreter and gets
captured for next time — under-coverage **self-heals**.
- Code we might compile **wrong** would corrupt the machine — so native code is
dispatched *only* when its source RAM is provably unchanged, and a compiled
overlay is instantly revoked the moment the RAM it came from is overwritten.
> **The worst case is always performance, never correctness.** Anything not yet
> native simply runs interpreted, correctly.
## How the gaps actually get plugged
Two complementary loops close the gap toward full native coverage:
- **While you play (automatic).** The capture step runs continuously in the
background. New areas you reach are recorded and compiled to native code —
often within a minute, without interrupting play — and reused on every later
launch. Your discoveries live in `overlay_captures.json` next to the game.
- **In releases (shared head start).** A game release ships a `cache/` folder of
overlays already compiled from everywhere players have collectively visited,
so common areas are native from the very first moment you arrive.
The frontier of the project is pushing Tier-2 coverage up (so the interpreter
goes idle) and, eventually, handling the self-modifying corner by baking in the
relocation pattern at build time. Until then, that corner stays correctly
interpreted — a narrow, accepted cost, not a wall.
## Where this lives in the code
| Concern | Where |
|---|---|
| MIPS→C translation (static + overlay) | `recompiler/src/` (`code_generator.cpp`, `mips_decoder.cpp`, …) |
| Overlay capture | `runtime/src/overlay_capture.c` |
| Overlay compile pipeline | `tools/compile_overlays.py`, `runtime/src/overlay_backend.c`, `overlay_compile_worker.c` |
| Native overlay load/dispatch | `runtime/src/overlay_loader.c` |
| Runtime interpreter (dirty RAM) | `runtime/src/dirty_ram_interp.c` |
See [`ARCHITECTURE.md`](ARCHITECTURE.md) for how these connect to the hardware
simulation, and [`docs/FEATURES.md`](FEATURES.md) for the overlay-system feature
reference.
docs/config_schema.md
+1 -2
View File
@@ -1,8 +1,7 @@
# PSXRecomp v4 — config TOML schema
Consumed by:
- `psxrecomp-bios.exe` and `psxrecomp-game.exe` (once Phase A2/A3 ships
`--config <toml>` support)
- `psxrecomp-bios.exe` and `psxrecomp-game.exe` (via `--config <toml>`)
- `tools/audit_config.py` and the audit tooling that wraps it
- The runtime cmake macro (`runtime/runtime.cmake`)
ACCURACY_BURNDOWN.md → docs/internal/ACCURACY_BURNDOWN.md
View File
BACKDROP_PRELOAD.md → docs/internal/BACKDROP_PRELOAD.md
View File
COSIM_ORACLE.md → docs/internal/COSIM_ORACLE.md
View File
CYCLE_MODEL_BEETLE.md → docs/internal/CYCLE_MODEL_BEETLE.md
View File
FAITHFUL_TIMING_PLAN.md → docs/internal/FAITHFUL_TIMING_PLAN.md
View File
FIRST_MILESTONE.md → docs/internal/FIRST_MILESTONE.md
View File
GL_RENDERER_HANDOFF.md → docs/internal/GL_RENDERER_HANDOFF.md
View File
HLE_SCHEDULER_CARVEOUT_PLAN.md → docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md
View File
NATIVE_WIDE_PLAN.md → docs/internal/NATIVE_WIDE_PLAN.md
View File
PLAN.md → docs/internal/PLAN.md
View File
PRECISE_IRQ_SLICE.md → docs/internal/PRECISE_IRQ_SLICE.md
View File
docs/internal/README.md
+42
View File
@@ -0,0 +1,42 @@
# Internal dev-logs & design notes
These are the working design documents and burn-down logs the project was built
from. They are **historical/working notes**, not contributor onboarding docs —
they capture decisions, phased plans, and deep subsystem investigations in the
order they happened, and some sections are superseded by later ones.
If you're new here, start with the top-level [`README.md`](../../README.md),
then [`docs/ARCHITECTURE.md`](../ARCHITECTURE.md),
[`docs/EXECUTION_MODEL.md`](../EXECUTION_MODEL.md),
[`docs/BUILDING.md`](../BUILDING.md), and
[`CONTRIBUTING.md`](../../CONTRIBUTING.md). Come here when you need the *why*
behind a specific subsystem.
## Plans & roadmap
- [`PLAN.md`](PLAN.md) — the original phased build plan (P1–P6).
- [`FAITHFUL_TIMING_PLAN.md`](FAITHFUL_TIMING_PLAN.md) — the authoritative
cycle-accuracy north-star plan and status log.
- [`FIRST_MILESTONE.md`](FIRST_MILESTONE.md) — the first-boot milestone gate.
- [`STUBS_TO_FIX.md`](STUBS_TO_FIX.md) — hardware-stub burn-down (pre-game gate).
## Cycle accuracy & timing
- [`PRECISE_IRQ_SLICE.md`](PRECISE_IRQ_SLICE.md) — exact-instruction IRQ delivery.
- [`CYCLE_MODEL_BEETLE.md`](CYCLE_MODEL_BEETLE.md) — the Beetle-derived cycle model.
- [`ACCURACY_BURNDOWN.md`](ACCURACY_BURNDOWN.md) — full accuracy burn-down across
all axes (semantics, cycle, IRQ, MMIO, peripherals, determinism).
- [`COSIM_ORACLE.md`](COSIM_ORACLE.md) — the first-divergence co-sim oracle.
## BIOS / scheduler
- [`HLE_SCHEDULER_CARVEOUT_PLAN.md`](HLE_SCHEDULER_CARVEOUT_PLAN.md) — the
cooperative-thread scheduler HLE carve-out.
- [`WEDGE_load4_shell_rootcause.md`](WEDGE_load4_shell_rootcause.md) — a specific
boot-shell wedge root-cause writeup.
## Renderer / widescreen
- [`GL_RENDERER_HANDOFF.md`](GL_RENDERER_HANDOFF.md) — OpenGL renderer design.
- [`NATIVE_WIDE_PLAN.md`](NATIVE_WIDE_PLAN.md) — native-wide 16:9 (GTE FOV) plan.
- [`BACKDROP_PRELOAD.md`](BACKDROP_PRELOAD.md) — widescreen backdrop preload.
## Overlay cache internals
- [`SLJIT_PERSIST_CACHE.md`](SLJIT_PERSIST_CACHE.md) — persisted overlay-shard
cache design.
SLJIT_PERSIST_CACHE.md → docs/internal/SLJIT_PERSIST_CACHE.md
View File
STUBS_TO_FIX.md → docs/internal/STUBS_TO_FIX.md
View File
WEDGE_load4_shell_rootcause.md → docs/internal/WEDGE_load4_shell_rootcause.md
View File
runtime/src/code_provider.c
+10 -7
View File
@@ -1,11 +1,14 @@
/* code_provider.c — the two CodeProvider implementations + active selection.
* See code_provider.h for the abstraction and SLJIT.md §3/§6/§7 for the plan.
/* code_provider.c — the CodeProvider implementations + active selection.
* See code_provider.h for the abstraction.
*
* This is the step-2 seam: the gcc provider is a thin pass-through over the
* validated autocompile_* path (no behavior change), and the sljit provider
* wraps the in-process JIT. Which one is active is decided once at init by
* overlay_backend_resolve (env PSX_OVERLAY_BACKEND > [runtime] overlay_backend >
* auto; auto => gcc if a compile cmd is wired, else sljit). gcc is the default. */
* The "spawn compiler" provider is a thin pass-through over the validated
* autocompile_* path (compile emitted overlay C to a DLL and load it); it backs
* BOTH the gcc and tcc tiers, which differ only in which compiler the pipeline
* invokes. The in-process JIT provider (code_provider_sljit) is a separate,
* legacy path. Which provider is active is decided once at init by
* overlay_backend_resolve() in overlay_backend.c (env PSX_OVERLAY_BACKEND >
* [runtime] overlay_backend > auto). The auto tier order is:
* static -> gcc (if gcc on PATH) -> tcc (bundled toolchain-free fallback). */
#include <stddef.h>