diff --git a/CLAUDE.md b/CLAUDE.md index 313fe23d..5b4a19af 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,11 +43,11 @@ loss), never as the starting point — see §0's amendment below and recomp-template `PRINCIPLES.md` → "LLE Is the Baseline; HLE Is a Subsystem Replacement, Not a Starting Point". -**The authoritative game plan is `FAITHFUL_TIMING_PLAN.md` — READ IT EACH +**The authoritative game plan is `docs/internal/FAITHFUL_TIMING_PLAN.md` — READ IT EACH SESSION** (north star, phased plan P1–P6, current status/log). Update its Status/Log section every session. The full-coverage accuracy burndown across ALL axes (semantics, cycle, IRQ, MMIO, peripherals, static-vs-dynamic, determinism) -lives in `ACCURACY_BURNDOWN.md` — every item must be cross-referenced against an +lives in `docs/internal/ACCURACY_BURNDOWN.md` — every item must be cross-referenced against an external comparative (psx-spx / in-tree Beetle source / DuckStation / HW test ROMs), not asserted. Axis 5 (peripherals — esp. SIO/controller, the hybrid-pad bug) is the suspected-weakest second front after the cycle axis. @@ -149,14 +149,14 @@ first-class High-Level Emulation tier alongside LLE, modeled on If you find yourself wanting to violate any of the above three paragraphs **beyond the two amendments just above**, -**stop and re-read PLAN.md**. Every prior attempt failed by +**stop and re-read docs/internal/PLAN.md**. Every prior attempt failed by violating exactly these rules under pressure. --- ## 1. The BIOS is recompilation target #1, the game is target #2 -Phase 1-3 of PLAN.md exist to get the BIOS recompiled and booting on +Phase 1-3 of docs/internal/PLAN.md exist to get the BIOS recompiled and booting on its own. The BIOS must reach the Sony logo and the BIOS shell, running entirely as native C, before any game work begins. There is no path that loads a game EXE before the BIOS is fully working in v4. **Do not @@ -236,7 +236,7 @@ a fake event. The fake delivery was not progress, it was theater. At the start of every session, before any code change: 1. Read this file (CLAUDE.md). -2. Read PLAN.md to confirm what phase we are in and what the next +2. Read docs/internal/PLAN.md to confirm what phase we are in and what the next concrete milestone is. 3. Verify `docs/psx_bios_disasm.txt` exists (primary reference). 4. Verify Ghidra MCP is reachable. If not, stop and ask. @@ -330,7 +330,7 @@ Code without proof is invalid. Before any Phase 2 work: -- FIRST_MILESTONE.md must be complete +- docs/internal/FIRST_MILESTONE.md must be complete - boot_slice must compile - all instructions must be supported @@ -462,7 +462,7 @@ the visibility you need. ## 17. Phase 5 gate — fix hardware stubs before loading a game -`STUBS_TO_FIX.md` lists every known stub in the runtime. Before any +`docs/internal/STUBS_TO_FIX.md` lists every known stub in the runtime. Before any Phase 5 work (loading Tomba or any game EXE), every stub marked "Phase 5+" in that file **must be implemented and oracle-verified**: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..be8444fe --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,93 @@ +# Contributing to PSXRecomp + +Thanks for your interest! PSXRecomp is a static recompiler, not an emulator, and +it holds itself to a high correctness bar. This guide covers how the project is +organized, the rules that keep it faithful, and how to get a change merged. + +New to the codebase? Read [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) and +[`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md) first, then +[`docs/BUILDING.md`](docs/BUILDING.md) to get a build going. + +## Repository layout + +- **`mstan/psxrecomp`** (this repo) — the framework: the recompiler tool + (`recompiler/`), the runtime engine (`runtime/`), shared tools, and docs. + Framework changes go here. +- **Per-game repos** (`mstan/TombaRecomp`, `mstan/MegaManX6Recomp`, …) — one per + title. Each contains only that game's config, seeds, and build glue and links + this framework in as a submodule at `psxrecomp/`. Game-specific work goes in + the game repo; nothing game-specific belongs in the framework. + +See [Linking the framework](docs/BUILDING.md#linking-the-framework) for the +submodule + local-junction dev setup. + +## The core rules + +These are non-negotiable because violating them is how past efforts turned into +un-debuggable emulators. The full engineering constitution is +[`CLAUDE.md`](CLAUDE.md); the debugging philosophy is +[`PRINCIPLES.md`](PRINCIPLES.md). The short version: + +1. **Build the faithful core; no per-game hacks in the foundation.** The correct + fix is the general, hardware-accurate one, not a surgical workaround for one + title. (Per-game *enhancements* — widescreen, faster loads — are legitimate + once the faithful core is proven; see [`ENHANCEMENTS.md`](ENHANCEMENTS.md).) +2. **No stubs.** A function is fully implemented or it fails loudly. No + `return 0;` placeholders, no `// TODO` behavior, no hand-delivered fake + events. If execution reaches code we can't handle, we stop and fix + discovery/codegen — we don't fake the result. +3. **Never hand-edit generated code.** Files under `generated/` and the + recompiler's output are build artifacts. If the emitted C is wrong, fix the + recompiler (`recompiler/src/`), then regenerate. +4. **LLE is the baseline; HLE is a validated subsystem replacement, never a + starting point.** The recompiled BIOS is the reference and oracle. An HLE tier + is allowed only where the LLE path has a genuine landmine, must be general + (every game), must operate on the real guest structures, and must be + continuously checked against the Beetle oracle — never a "produce the answer + the BIOS would have" shim. +5. **No `printf`/log-file debugging.** Runtime inspection goes through the TCP + debug server ([`TCP_COMMANDS.md`](TCP_COMMANDS.md)) and always-on ring + buffers, not `fprintf`. +6. **Fix broken tooling immediately** — don't route around it with indirect + evidence, and don't infer correctness from two implementations sharing a bug. + +## Verifying a change + +Correctness is demonstrated, not asserted: + +- **Oracle-check against Beetle.** Run `psx-beetle` alongside `psx-runtime` and + compare via the shared TCP protocol; for timing/codegen, use the co-sim build. + See [`docs/internal/COSIM_ORACLE.md`](docs/internal/COSIM_ORACLE.md). +- **Verify visually, end to end.** A milestone is "the pixels appear on screen" + — take a screenshot, don't trust a counter or a ring-buffer flag alone. +- **A decoder / codegen change requires a playthrough check**, not just a clean + build: cosim/lockstep first, then run the affected area. + +## Adding a new game + +At a high level: create a game repo from the template, add this framework as the +`psxrecomp/` submodule, seed function starts (typically from a Ghidra export in +`seeds/`), write `game.toml`, generate the game C with +`psxrecomp-game --config game.toml`, then build and bring the title up boot → +menu → gameplay, oracle-checking as you go. The existing game repos +(TombaRecomp, MegaManX6Recomp) are the working references. + +## Pull requests + +- Branch off `master`; keep framework changes and game changes in their + respective repos. +- **Do not commit** local artifacts: BIOS images, disc images, generated C + (`generated/`), memory cards, Ghidra databases, build outputs, or overlay + capture files. These are gitignored for good reasons (copyright + size). +- Explain *what hardware behavior* the change matches and *how you verified it* + (oracle diff, screenshot, playthrough). Changes that can't state their + verification are hard to accept. +- Keep the fix minimal and at the right layer (recompiler vs runtime vs game + config). If you find yourself editing generated code or adding a stub, step + back — the real fix is upstream of that. + +## Questions / community + +Development happens with the **R.A.I.D. (Retro AI Development)** community — +Discord invite in the [README](README.md). Open a GitHub issue for bugs and +build problems (include `gcc -v` / OS / generator for build failures). diff --git a/README.md b/README.md index 937bd1cd..97aaa10a 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,19 @@ BIOS layer, no stubs, and no general-purpose interpreter fallback for static code. PSXRecomp is a framework. Game-specific projects live in their own -repositories and pull this one in to build a game binary. The active end-to-end -targets are: +repositories and link this one in as a **git submodule** to build a game binary. +The active end-to-end targets are: - [TombaRecomp](https://github.com/mstan/TombaRecomp) — *Tomba!* - [MegaManX6Recomp](https://github.com/mstan/MegaManX6Recomp) — *Mega Man X6* +**New here?** The fastest way in: +[`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md) (how a game actually +runs — static / native-overlay / interpreter), then +[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md), +[`docs/BUILDING.md`](docs/BUILDING.md), and +[`CONTRIBUTING.md`](CONTRIBUTING.md). + ## Philosophy — toward 100% static recompilation The goal is simple and absolute: **a PS1 game should run as native code, not be @@ -144,29 +151,30 @@ Windows, `ucontext` on POSIX (`runtime/src/psx_fiber.c`) — so the recompiled BIOS's cooperative thread switching (the CD-boot handoff in particular) behaves the same on every platform. -Requirements: +Requirements at a glance (full details, dependency table, and per-platform +prerequisites in [`docs/BUILDING.md`](docs/BUILDING.md)): -- A C/C++ toolchain: MSVC or MinGW (Windows), Apple Clang (macOS), Clang/GCC (Linux). -- CMake 3.20+. On macOS/Linux also `ninja` and `pkg-config`. -- SDL2: the bundled dev pack on Windows; `brew install sdl2 pkg-config ninja` - on macOS; `libsdl2-dev` (or distro equivalent) on Linux. +- A C/C++ toolchain: MSVC or MinGW/MSYS2 (Windows), Apple Clang (macOS), + Clang/GCC (Linux). CMake 3.20+; on macOS/Linux also `ninja` and `pkg-config`. +- SDL2 (system / bundled). RmlUi and FreeType come in as **git submodules** — + clone with `--recurse-submodules`. - A legally obtained `SCPH1001.BIN` BIOS dump. Not included. - For game projects, a legally obtained game disc/EXE dump. Not included. -Build the framework runtime: +Build the framework (recompiler tool + BIOS-only runtime): ```sh -# Windows (MSYS2/MinGW) -cmake -S recompiler -B recompiler/build -G "Unix Makefiles" && cmake --build recompiler/build -cmake -S runtime -B runtime/build -G "Unix Makefiles" && cmake --build runtime/build --target psx-runtime +git clone --recurse-submodules https://github.com/mstan/psxrecomp.git && cd psxrecomp -# macOS / Linux (Ninja) -cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C recompiler/build -cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release && ninja -C runtime/build psx-runtime +cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build recompiler/build +cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build runtime/build --target psx-runtime ``` -Game projects generate their own `generated/_*.c` files and link this -runtime source tree through CMake. +On Windows swap `-G Ninja` for your generator if you prefer (e.g. +`-G "Unix Makefiles"`); always keep an explicit `-DCMAKE_BUILD_TYPE` so the +generated C is optimized. Game projects generate their own +`generated/_*.c` files and link this runtime through CMake — see +[`docs/BUILDING.md`](docs/BUILDING.md#build-and-run-a-game). ## Keyboard Map @@ -209,10 +217,19 @@ The recompiler emits C functions and dispatch tables for BIOS and game code. The runtime loads the BIOS/game assets into emulated PS1 memory, links the generated C as native code, and simulates hardware through MMIO handlers for GPU, DMA, timers, CD-ROM, MDEC, SIO0, memory cards, SPU, GTE, and interrupt -delivery. BIOS A0/B0/C0 vectors go through the recompiled BIOS, not HLE shims. +delivery. The recompiled `SCPH1001.BIN` is the low-level (LLE) kernel and the +correctness oracle; an optional HLE tier lays instant boot-skip and a few BIOS +services on top, always falling through to the recompiled BIOS. -See `CLAUDE.md`, `PLAN.md`, and `FAITHFUL_TIMING_PLAN.md` for the development -rules and current project context. +Code that can't be seen ahead of time (disc-streamed **overlays**) is captured +and compiled to native code the first time it appears (`static → gcc → tcc` +backend), with a small interpreter as the correctness fallback until it is. Full +story in [`docs/EXECUTION_MODEL.md`](docs/EXECUTION_MODEL.md); component-level +detail in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). + +See [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`CLAUDE.md`](CLAUDE.md) for the +development rules, and [`docs/internal/`](docs/internal/) for the phased plans +and deep design notes (`PLAN.md`, `FAITHFUL_TIMING_PLAN.md`, …). ## Disc Speed diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..3293157c --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,159 @@ +# Architecture + +How PSXRecomp is put together, end to end. For the higher-level "why three ways +of running code" story, read [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) first; +this doc is the component-level view. + +## Two programs: the recompiler and the runtime + +PSXRecomp is split into two CMake projects that are built and run separately: + +``` + recompiler/ runtime/ + ─────────── ──────── + MIPS ──▶ C C + hardware simulation ──▶ native game binary + (build-time tool) (linked with the generated C) +``` + +- **`recompiler/`** (C++20) is an offline tool. It reads MIPS R3000A machine code + and emits C source files under `generated/`. +- **`runtime/`** (C99 + C++17) is the engine. It loads the game's assets into an + emulated PS1 address space, links the generated C in as native functions, and + simulates the console's hardware around them. + +A **game repository** (e.g. TombaRecomp) contains no framework source — it links +this framework in (as a submodule) and provides the game's config, seeds, and +build glue. See [`BUILDING.md`](BUILDING.md#linking-the-framework). + +## The recompiler (`recompiler/`) + +Two entry points share the same translation core: + +- `src/main_bios.cpp` — ingests the flat BIOS ROM (`SCPH1001.BIN`, loaded at + `0xBFC00000`) and emits `generated/SCPH1001_*.c`. +- `src/main_psx.cpp` — ingests a `PS-X EXE` extracted from a game disc and emits + `generated/_*.c`. + +The translation pipeline: + +| Stage | File | Job | +|---|---|---| +| Decode | `mips_decoder.cpp` (+ vendored `rabbitizer`) | Decode MIPS instructions | +| Control flow | `control_flow.cpp`, `basic_block.cpp` | Build basic blocks / CFG | +| Discovery | `function_discovery.cpp`, `function_analysis.cpp` | Find function entry points (seeded from Ghidra exports) | +| Codegen | `code_generator.cpp`, `full_function_emitter.cpp`, `strict_translator.cpp` | Emit C: one function per guest function, plus a dispatch table | + +Output is **two files** per program: a `_full.c` (the function bodies) and a +`_dispatch.c` (the address→function dispatch table). Both are build artifacts and +are **never hand-edited** — if the C is wrong, the fix is in the recompiler. + +GTE (the PS1 geometry coprocessor) instructions are emitted inline; +COP0 kernel-mode instructions the BIOS needs are handled in codegen. + +## The runtime (`runtime/`) + +The runtime is assembled by one CMake helper, `psxrecomp_add_runtime_target()` +(defined in `runtime/runtime.cmake`), which a game's `CMakeLists.txt` calls with +the paths to its generated C. The runtime provides: + +- **Memory & address space** (`memory.c`) — the 2 MB main RAM, scratchpad, BIOS + ROM, and MMIO regions, with the dispatch that routes a guest PC to its native + function (static, overlay, or interpreter). +- **Hardware simulation via MMIO handlers** — GPU (`gpu.c`, `gpu_sw_renderer.c`, + `gpu_gl_renderer.c`), DMA (`dma.c`), timers (`timers.c`), CD-ROM + (`cdrom.c`, `iso_reader.cpp`), MDEC, SIO0 controllers/memory cards + (`sio.c`, `memcard.c`), SPU (`spu.c`), GTE (`gte.cpp`), and interrupt delivery + (`interrupts.c`). +- **Host services** — window/input/audio via SDL2, a cooperative-thread + scheduler on host fibers (`psx_fiber.c`: Win32 Fibers / POSIX `ucontext`), and + the optional debug TCP server (`debug_server.c`). + +### BIOS: LLE baseline + a swappable HLE tier + +The recompiled `SCPH1001.BIN` is the **low-level (LLE) baseline**: it *is* the +kernel, and it is the reference implementation and the correctness oracle. There +are no per-vector HLE shims replacing it. + +On top of that, PSXRecomp carries an optional **HLE tier** (`bios_hle`, on by +default for player convenience) that can synthesize the post-boot kernel handoff +state to skip the BIOS boot sequence, and intercept a small set of BIOS services +— always falling through to the recompiled BIOS for anything it doesn't +implement. LLE stays fully linked and is what every accuracy check runs against. +Turn it off with `[runtime] bios_hle = false` or `PSX_BIOS_HLE=0`; with it off the +build behaves as pure LLE. (Design notes: +[`docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md`](internal/HLE_SCHEDULER_CARVEOUT_PLAN.md).) + +### Static / overlay / interpreter dispatch + +This is the heart of the system and has its own doc, +[`EXECUTION_MODEL.md`](EXECUTION_MODEL.md). In brief: a guest PC resolves to +statically-recompiled native code (BIOS + main EXE), a runtime-compiled **native +overlay** (`overlay_capture.c` → `tools/compile_overlays.py` → +`overlay_loader.c`), or the small **dirty-RAM interpreter** +(`dirty_ram_interp.c`) — in that priority order. + +### Overlay compile backend + +When an overlay needs compiling, the runtime spawns a C compiler on the +recompiler-emitted C and loads the resulting DLL — it does not JIT in-process. +The backend tier is resolved in `overlay_backend.c` (see `main.cpp` around the +`code_provider` setup): + +``` + static → gcc → tcc +``` + +- **static** — the overlay was baked into the binary at build time (best case). +- **gcc** — used when a system `gcc` is on `PATH` (the development default). +- **tcc** — TinyCC, the toolchain-free fallback **bundled beside shipped + executables** in `overlay_toolchain/` (an embedded Python + `tcc.exe`) so + players never need a compiler installed. + +Compiled overlays are stored in a content-addressed cache namespaced by +compiler and target ABI (`/gcc//…` vs `/tcc/…`); a `gcc` +shard wins over a `tcc` shard for the same region. See +[`docs/FEATURES.md`](FEATURES.md), [`docs/OVERLAY_CACHE_V2.md`](OVERLAY_CACHE_V2.md), +and [`docs/ASYNC_OVERLAY_COMPILE.md`](ASYNC_OVERLAY_COMPILE.md). + +## Renderers + +Three GPU backends behind one interface: + +- **Software rasterizer** — CPU, most portable, the reference look. +- **OpenGL** — GPU-authoritative VRAM/FBO renderer, the default; moves + rasterization and supersampling onto the GPU. Falls back to software if GL + init fails. (See [`docs/internal/GL_RENDERER_HANDOFF.md`](internal/GL_RENDERER_HANDOFF.md).) +- **Vulkan** — experimental, off by default (`PSX_ENABLE_VULKAN=OFF`). + +Widescreen (a genuine wider GTE FOV, not a stretch) is opt-in and gen-time; see +[`WIDESCREEN.md`](../WIDESCREEN.md) and +[`docs/internal/NATIVE_WIDE_PLAN.md`](internal/NATIVE_WIDE_PLAN.md). + +## The oracle model (how correctness is checked) + +PSXRecomp validates itself against **Beetle PSX** (the mednafen-psx libretro +core), run as a **separate process** with an identical TCP JSON debug protocol: + +- `psx-runtime` — the recompiled runtime (debug server on port 4370). +- `psx-beetle` — Beetle PSX (debug server on port 4380). + +A tool written against one works against the other by switching ports; cross- +checking is done by querying both, never by sharing state in one process. There +is also a first-divergence **co-sim** build that cycle-locksteps the compiled +backend against the interpreter. See +[`docs/internal/COSIM_ORACLE.md`](internal/COSIM_ORACLE.md), +[`docs/config_schema.md`](config_schema.md), and [`TCP_COMMANDS.md`](../TCP_COMMANDS.md). + +## Configuration + +Every process has a **BIOS config**; a game adds a **game config** (`game.toml`), +and the two are merged (game wins on scalar keys; program blocks and generated +dispatch tables are additive). Full schema: +[`docs/config_schema.md`](config_schema.md). + +## Where to go next + +- [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) — static/native/interp in depth. +- [`BUILDING.md`](BUILDING.md) — dependencies + build steps. +- [`../CONTRIBUTING.md`](../CONTRIBUTING.md) — dev workflow and rules. +- [`../CLAUDE.md`](../CLAUDE.md) — the exhaustive engineering constitution. diff --git a/docs/BUILDING.md b/docs/BUILDING.md new file mode 100644 index 00000000..ce2423b7 --- /dev/null +++ b/docs/BUILDING.md @@ -0,0 +1,181 @@ +# Building PSXRecomp + +PSXRecomp builds natively on **Windows** (MSVC or MinGW/MSYS2), **macOS** (Apple +Silicon & Intel), and **Linux**. There are two things you can build: + +- the **framework** itself (this repo) — a BIOS-only runtime plus the recompiler + tool; useful for BIOS work and as the thing game repos link against; +- a **game** — done from that game's own repository, which links this framework + in as a submodule (see [Linking the framework](#linking-the-framework)). + +You always supply your own **`SCPH1001.BIN` BIOS** and, for a game, your own +legally-obtained **disc image**. This project ships none of those. + +## Toolchain requirements + +- A C/C++ toolchain: + - **Windows:** MSYS2 MinGW-w64 (`mingw-w64-x86_64` toolchain) *or* MSVC. + - **macOS:** Apple Clang (Xcode command-line tools). + - **Linux:** GCC or Clang. +- **CMake ≥ 3.20**. On macOS/Linux also **Ninja** and **pkg-config**. +- Language standards used: recompiler is **C++20**; runtime is **C99 + C++17**. +- `ccache` is auto-detected and used if present (optional, speeds rebuilds). + +## Dependencies + +| Dependency | How it's provided | Used for | +|---|---|---| +| **SDL2** | System. Windows/MSVC: prebuilt pack auto-found at `../sdl2-msvc/SDL2-*`. Windows/MinGW + macOS + Linux: found via pkg-config. | Window, input, GL/Vulkan context, audio, threads | +| **RmlUi** | **git submodule** `lib/RmlUi` (static) | The launcher UI | +| **FreeType** | **git submodule** `lib/freetype` (static) | RmlUi's font engine | +| **fmt** 9.1.0 | vendored `recompiler/lib/fmt` | String formatting (runtime uses it header-only) | +| **toml11** | vendored `recompiler/lib/toml11` | Parsing `game.toml` / configs | +| **ELFIO** | vendored `recompiler/lib/ELFIO` | ELF parsing (recompiler only) | +| **rabbitizer** | vendored `recompiler/lib/rabbitizer` | MIPS instruction decoding (recompiler only) | +| **TinyCC (TCC) 0.9.27** | Not in this repo — downloaded at release-packaging time and bundled beside the game exe in `overlay_toolchain/`. | Toolchain-free overlay compilation for players (run as a subprocess) | +| **Python 3** | System (development) or an embedded copy bundled in releases | Runs `tools/compile_overlays.py` in the overlay pipeline | +| **OpenGL** | System (`opengl32` on Windows; `find_package(OpenGL)` elsewhere) | The GL renderer | +| **Vulkan** | Headers only, optional, **off by default** (`PSX_ENABLE_VULKAN=OFF`); loaded dynamically via SDL. Shader compilation needs `glslc` from the Vulkan SDK. | The experimental Vulkan renderer | + +Developers building overlays locally just need `gcc` on `PATH` (the `gcc` tier); +the bundled `tcc` matters only for end-user release packages. + +### Get the source (with submodules) + +RmlUi and FreeType are submodules, so clone recursively (or init after the fact): + +```sh +git clone --recurse-submodules https://github.com/mstan/psxrecomp.git +# or, in an existing clone: +git submodule update --init --recursive +``` + +## Per-platform prerequisites + +**Windows (MSYS2/MinGW — recommended for release parity):** +```sh +# In an MSYS2 MinGW64 shell: +pacman -S --needed mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake \ + mingw-w64-x86_64-ninja mingw-w64-x86_64-SDL2 mingw-w64-x86_64-ccache +``` + +**macOS:** +```sh +brew install sdl2 pkg-config ninja cmake +``` + +**Linux (Debian/Ubuntu):** +```sh +sudo apt install build-essential cmake ninja-build pkg-config libsdl2-dev +``` + +## Build the framework + +Two CMake trees: the recompiler (a tool) and the runtime (the engine). + +```sh +# 1. Recompiler tool → produces psxrecomp-bios and psxrecomp-game +cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release +cmake --build recompiler/build + +# 2. (Optional) regenerate the BIOS C from your SCPH1001.BIN +# Requires bios/SCPH1001.BIN to be present. +bash tools/regen_bios.sh + +# 3. Runtime → produces psx-runtime (BIOS-only for this repo) +cmake -S runtime -B runtime/build -G Ninja -DCMAKE_BUILD_TYPE=Release +cmake --build runtime/build --target psx-runtime +``` + +On Windows with MSVC or plain MinGW makefiles, swap `-G Ninja` for your generator +(e.g. `-G "Unix Makefiles"`); everything else is identical. + +> **Build type matters.** With no `-DCMAKE_BUILD_TYPE`, the framework defaults to +> **Release** (optimized). The huge generated C compiles unusably slowly at `-O0`, +> so never build it debug-by-accident. Use `-DCMAKE_BUILD_TYPE=RelWithDebInfo` +> when you need symbols. Release turns the debug TCP server **off**; RelWithDebInfo +> and Debug turn it **on** (`PSX_DEBUG_TOOLS`). + +### Useful CMake options + +| Option | Default | Effect | +|---|---|---| +| `PSX_DEBUG_TOOLS` | ON for Debug/RelWithDebInfo, OFF for Release | TCP debug server + heartbeat + per-block recording | +| `PSX_STATIC_RUNTIME` | ON for MinGW Release | Self-contained exe (statically links SDL2 + libgcc/libstdc++) | +| `PSX_LAUNCHER` | ON | Build the RmlUi launcher UI | +| `PSX_ENABLE_VULKAN` | OFF | Build the experimental Vulkan renderer | +| `PSX_BUILD_COSIM` | OFF | Build the first-divergence co-sim oracle target | + +## Build and run a game + +From the **game's** repository (not this one). Each game repo links this +framework in and provides its own `game.toml`, seeds, and generated C. Example +for TombaRecomp: + +```sh +# Extract the game's PS-X EXE from your disc (helper included in the game repo): +python3 tools/extract_psx_exe.py tomba/tomba.bin SCUS_942.36 tomba/SCUS_942.36 + +# Regenerate the game's C from the disc/EXE (game repos invoke the recompiler +# binary directly with their config): +../psxrecomp/recompiler/build/psxrecomp-game --config game.toml + +# Configure + build the game runtime: +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release +cmake --build build --target psx-runtime +./build/psx-runtime --game game.toml --disc tomba/tomba.cue +``` + +## Linking the framework + +Game repositories do **not** vendor the framework. They reference it as a git +**submodule** at path `psxrecomp/`, which the game's `CMakeLists.txt` points +`PSXRECOMP_ROOT` at: + +```cmake +set(PSXRECOMP_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/psxrecomp" CACHE PATH "Path to psxrecomp") +include("${PSXRECOMP_ROOT}/runtime/runtime.cmake") +``` + +A fresh clone of a game repo gets the framework automatically: + +```sh +git clone --recurse-submodules https://github.com/mstan/TombaRecomp.git +``` + +**Local dev tip — share one framework checkout across games.** If you work on +several game repos plus the framework at once, you don't want N multi-GB copies. +Replace each game's `psxrecomp/` submodule working directory with a **junction / +symlink** to a single shared framework checkout: + +```sh +# Windows (junction): +cmd /c mklink /J psxrecomp F:\path\to\shared\psxrecomp +# macOS / Linux (symlink): +ln -s /path/to/shared/psxrecomp psxrecomp +``` + +Git treats the link's target as the submodule working tree, so +`git submodule status` stays clean **as long as the shared checkout is at the +commit the game pins**. Bump a game to a newer framework by checking out the new +commit in the shared repo and committing the updated submodule pointer in the +game repo. This is the setup the game repos are configured for. + +## Troubleshooting + +**MinGW: `Error: too many sections` / a `*.c.o` (or `.obj`) fails to assemble.** +Windows COFF object files have a 32,768-section limit; very large translation +units (the generated game C, and `debug_server.c` in debug-info builds) can +exceed it on older binutils. The fix is to assemble those units as *big objects*: +add `-Wa,-mbig-obj` to their compile options (recent binutils, ≥ ~2.40, handle +these files without it). If you hit this on a framework source file, please open +an issue with your `gcc -v` / `as --version` — the build should apply the flag +for you. + +**`SDL2 MSVC dev package not found`.** The MSVC build expects the prebuilt SDL2 +pack beside the repo at `../sdl2-msvc/SDL2-*`. Use the MSYS2/MinGW toolchain +(which finds SDL2 via pkg-config) or place the pack there. + +**Overlays never compile / stay slow.** In development you need `gcc` on `PATH` +for the `gcc` tier; otherwise areas stay in the interpreter. See +[`EXECUTION_MODEL.md`](EXECUTION_MODEL.md). diff --git a/docs/EXECUTION_MODEL.md b/docs/EXECUTION_MODEL.md new file mode 100644 index 00000000..abc68745 --- /dev/null +++ b/docs/EXECUTION_MODEL.md @@ -0,0 +1,160 @@ +# The Execution Model — static, native-overlay, and interpreter + +*A semi-technical tour of how PSXRecomp actually runs a PlayStation game. If you +only read one design doc, read this one.* For the low-level plumbing, see +[`ARCHITECTURE.md`](ARCHITECTURE.md). + +## The one-sentence version + +PSXRecomp turns a PS1 game into a **native program** by translating its MIPS code +to C ahead of time; the small amount of code it *can't* see ahead of time is +either compiled to native code the first time it appears, or run by a tiny +built-in interpreter until it is — so the game is always **correct**, and gets +**faster** the more it's played. + +## Why a PS1 game can't just be "decompiled once" + +A PlayStation game is not one fixed blob of code sitting in memory. The main +program (the `PS-X EXE` on the disc) is, but games are far bigger than the PS1's +2 MB of RAM, so they **stream code off the disc as you play**. A level's logic, +a boss's AI, a menu — these are loaded from the CD into a fixed region of RAM, +run, and then *overwritten* by the next chunk when you move on. These streamed +chunks are called **overlays**. + +That creates a hard problem for any ahead-of-time recompiler: + +- The main EXE exists on disc at build time → we can translate all of it. +- An overlay's code **does not exist anywhere at build time** in a form we can + find. It only appears in RAM at a specific moment during play, and the same + RAM addresses get reused by dozens of different overlays over a playthrough. + +So "recompile the game" can never be a single up-front pass. PSXRecomp handles +this with **three tiers of execution**, in strict priority order. + +``` + ┌─────────────────────────────────────────────────────────┐ + │ Guest wants to run code at some address │ + └─────────────────────────────────────────────────────────┘ + │ + ┌─────────────────────┼─────────────────────┐ + ▼ ▼ ▼ + 1. STATIC (AOT) 2. NATIVE OVERLAY 3. INTERPRETER + Translated at Compiled to native Runs the guest's + build time from code the first time own instructions + the disc's EXE the overlay is seen, directly, as a + and the BIOS. then cached forever. correctness safety + The bulk of Fills the "streamed net for anything + execution. from disc" gap. not yet native. + ── fastest ── ── fast ── ── slowest, always + correct ── +``` + +## Tier 1 — Static (ahead-of-time) + +At build time the **recompiler** reads two things and translates their MIPS +machine code into C, function by function: + +- the real PS1 **BIOS** (`SCPH1001.BIN`) — this becomes the kernel, and +- the game's **main executable** extracted from your disc. + +That C is compiled and linked into the runtime as ordinary native functions, so +when the game calls one of them the CPU just runs native code — no emulation of +individual instructions. **This is the overwhelming majority of what executes**: +boot, the engine, the main loop, most gameplay systems. It is as fast as the C +compiler can make it. + +Everything a pure AOT recompiler can see, it recompiles. The only things left +over are the two categories below. + +## Tier 2 — Native overlays (compiled the moment they're discovered) + +Overlays can't be seen at build time — but they *can* be seen the instant the +game loads one. PSXRecomp watches for that moment and turns each overlay into +native code so that, from then on, it runs at Tier-1 speed: + +1. **Capture.** Every time the game DMAs code from the disc into the overlay RAM + window, the runtime records the exact bytes (before the game can modify them) + and the addresses that were actually executed. These go into a per-game file, + `overlay_captures.json`. +2. **Compile.** Those captured bytes are fed back through the *same recompiler* + used for the main EXE, producing C for that overlay. The C is compiled to a + small native library (a DLL) — using `gcc` in development, or a bundled + `tcc` (TinyCC) toolchain in shipped releases so players need no compiler + installed. Each library is keyed by the overlay's content hash. +3. **Cache & dispatch.** The compiled overlay lands in a content-addressed + cache. On this run and every future run, when that overlay loads, the runtime + finds the matching native library and dispatches into it **before** falling + back to the interpreter. + +The practical effect: **the first time anyone visits an area, it may run in the +slow tier for a moment; after it's been captured and compiled, that area is +native for them — and, once the capture is shared into a release, for everyone.** +Coverage grows toward "100% native" the more the game is played. + +## Tier 3 — The interpreter (the correctness safety net) + +There is always a window between "an overlay just appeared in RAM" and "we have +native code for it." There's also a genuinely hard corner: code that is +**rewritten to different bytes on every load** (self-modifying / per-load +relocated), which by definition has no single ahead-of-time translation. + +For both, PSXRecomp includes a **small MIPS interpreter** that runs the guest's +*own* instructions directly on the CPU register state. It is deliberately narrow: + +- It runs **only** code that was installed into RAM at runtime (dirty pages) and + is not yet covered by a native overlay. It is **never** on the BIOS or + main-EXE path — those are always static. +- It is a **safety net and a coverage feeder**, never a substitute for + recompiling. If the interpreter is running something, that's a signal to + capture-and-compile it (Tier 2), not to leave it interpreted. + +Crucially, this is **not** high-level emulation (HLE). The interpreter executes +the program's real instructions, exactly as the game's authors wrote them — the +*only* difference from static recompilation is where the instructions came from +(RAM-at-runtime vs disc-at-build-time). We never synthesize "the answer the code +would have produced." + +## The guiding rule: precision over recall + +The tiers are ordered so the system can only ever fail *toward being slow*, never +*toward being wrong*: + +- Code we **haven't** compiled yet safely drops to the interpreter and gets + captured for next time — under-coverage **self-heals**. +- Code we might compile **wrong** would corrupt the machine — so native code is + dispatched *only* when its source RAM is provably unchanged, and a compiled + overlay is instantly revoked the moment the RAM it came from is overwritten. + +> **The worst case is always performance, never correctness.** Anything not yet +> native simply runs interpreted, correctly. + +## How the gaps actually get plugged + +Two complementary loops close the gap toward full native coverage: + +- **While you play (automatic).** The capture step runs continuously in the + background. New areas you reach are recorded and compiled to native code — + often within a minute, without interrupting play — and reused on every later + launch. Your discoveries live in `overlay_captures.json` next to the game. +- **In releases (shared head start).** A game release ships a `cache/` folder of + overlays already compiled from everywhere players have collectively visited, + so common areas are native from the very first moment you arrive. + +The frontier of the project is pushing Tier-2 coverage up (so the interpreter +goes idle) and, eventually, handling the self-modifying corner by baking in the +relocation pattern at build time. Until then, that corner stays correctly +interpreted — a narrow, accepted cost, not a wall. + +## Where this lives in the code + +| Concern | Where | +|---|---| +| MIPS→C translation (static + overlay) | `recompiler/src/` (`code_generator.cpp`, `mips_decoder.cpp`, …) | +| Overlay capture | `runtime/src/overlay_capture.c` | +| Overlay compile pipeline | `tools/compile_overlays.py`, `runtime/src/overlay_backend.c`, `overlay_compile_worker.c` | +| Native overlay load/dispatch | `runtime/src/overlay_loader.c` | +| Runtime interpreter (dirty RAM) | `runtime/src/dirty_ram_interp.c` | + +See [`ARCHITECTURE.md`](ARCHITECTURE.md) for how these connect to the hardware +simulation, and [`docs/FEATURES.md`](FEATURES.md) for the overlay-system feature +reference. diff --git a/docs/config_schema.md b/docs/config_schema.md index 22f85529..c7eab4c1 100644 --- a/docs/config_schema.md +++ b/docs/config_schema.md @@ -1,8 +1,7 @@ # PSXRecomp v4 — config TOML schema Consumed by: -- `psxrecomp-bios.exe` and `psxrecomp-game.exe` (once Phase A2/A3 ships - `--config ` support) +- `psxrecomp-bios.exe` and `psxrecomp-game.exe` (via `--config `) - `tools/audit_config.py` and the audit tooling that wraps it - The runtime cmake macro (`runtime/runtime.cmake`) diff --git a/ACCURACY_BURNDOWN.md b/docs/internal/ACCURACY_BURNDOWN.md similarity index 100% rename from ACCURACY_BURNDOWN.md rename to docs/internal/ACCURACY_BURNDOWN.md diff --git a/BACKDROP_PRELOAD.md b/docs/internal/BACKDROP_PRELOAD.md similarity index 100% rename from BACKDROP_PRELOAD.md rename to docs/internal/BACKDROP_PRELOAD.md diff --git a/COSIM_ORACLE.md b/docs/internal/COSIM_ORACLE.md similarity index 100% rename from COSIM_ORACLE.md rename to docs/internal/COSIM_ORACLE.md diff --git a/CYCLE_MODEL_BEETLE.md b/docs/internal/CYCLE_MODEL_BEETLE.md similarity index 100% rename from CYCLE_MODEL_BEETLE.md rename to docs/internal/CYCLE_MODEL_BEETLE.md diff --git a/FAITHFUL_TIMING_PLAN.md b/docs/internal/FAITHFUL_TIMING_PLAN.md similarity index 100% rename from FAITHFUL_TIMING_PLAN.md rename to docs/internal/FAITHFUL_TIMING_PLAN.md diff --git a/FIRST_MILESTONE.md b/docs/internal/FIRST_MILESTONE.md similarity index 100% rename from FIRST_MILESTONE.md rename to docs/internal/FIRST_MILESTONE.md diff --git a/GL_RENDERER_HANDOFF.md b/docs/internal/GL_RENDERER_HANDOFF.md similarity index 100% rename from GL_RENDERER_HANDOFF.md rename to docs/internal/GL_RENDERER_HANDOFF.md diff --git a/HLE_SCHEDULER_CARVEOUT_PLAN.md b/docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md similarity index 100% rename from HLE_SCHEDULER_CARVEOUT_PLAN.md rename to docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md diff --git a/NATIVE_WIDE_PLAN.md b/docs/internal/NATIVE_WIDE_PLAN.md similarity index 100% rename from NATIVE_WIDE_PLAN.md rename to docs/internal/NATIVE_WIDE_PLAN.md diff --git a/PLAN.md b/docs/internal/PLAN.md similarity index 100% rename from PLAN.md rename to docs/internal/PLAN.md diff --git a/PRECISE_IRQ_SLICE.md b/docs/internal/PRECISE_IRQ_SLICE.md similarity index 100% rename from PRECISE_IRQ_SLICE.md rename to docs/internal/PRECISE_IRQ_SLICE.md diff --git a/docs/internal/README.md b/docs/internal/README.md new file mode 100644 index 00000000..5ab8d0de --- /dev/null +++ b/docs/internal/README.md @@ -0,0 +1,42 @@ +# Internal dev-logs & design notes + +These are the working design documents and burn-down logs the project was built +from. They are **historical/working notes**, not contributor onboarding docs — +they capture decisions, phased plans, and deep subsystem investigations in the +order they happened, and some sections are superseded by later ones. + +If you're new here, start with the top-level [`README.md`](../../README.md), +then [`docs/ARCHITECTURE.md`](../ARCHITECTURE.md), +[`docs/EXECUTION_MODEL.md`](../EXECUTION_MODEL.md), +[`docs/BUILDING.md`](../BUILDING.md), and +[`CONTRIBUTING.md`](../../CONTRIBUTING.md). Come here when you need the *why* +behind a specific subsystem. + +## Plans & roadmap +- [`PLAN.md`](PLAN.md) — the original phased build plan (P1–P6). +- [`FAITHFUL_TIMING_PLAN.md`](FAITHFUL_TIMING_PLAN.md) — the authoritative + cycle-accuracy north-star plan and status log. +- [`FIRST_MILESTONE.md`](FIRST_MILESTONE.md) — the first-boot milestone gate. +- [`STUBS_TO_FIX.md`](STUBS_TO_FIX.md) — hardware-stub burn-down (pre-game gate). + +## Cycle accuracy & timing +- [`PRECISE_IRQ_SLICE.md`](PRECISE_IRQ_SLICE.md) — exact-instruction IRQ delivery. +- [`CYCLE_MODEL_BEETLE.md`](CYCLE_MODEL_BEETLE.md) — the Beetle-derived cycle model. +- [`ACCURACY_BURNDOWN.md`](ACCURACY_BURNDOWN.md) — full accuracy burn-down across + all axes (semantics, cycle, IRQ, MMIO, peripherals, determinism). +- [`COSIM_ORACLE.md`](COSIM_ORACLE.md) — the first-divergence co-sim oracle. + +## BIOS / scheduler +- [`HLE_SCHEDULER_CARVEOUT_PLAN.md`](HLE_SCHEDULER_CARVEOUT_PLAN.md) — the + cooperative-thread scheduler HLE carve-out. +- [`WEDGE_load4_shell_rootcause.md`](WEDGE_load4_shell_rootcause.md) — a specific + boot-shell wedge root-cause writeup. + +## Renderer / widescreen +- [`GL_RENDERER_HANDOFF.md`](GL_RENDERER_HANDOFF.md) — OpenGL renderer design. +- [`NATIVE_WIDE_PLAN.md`](NATIVE_WIDE_PLAN.md) — native-wide 16:9 (GTE FOV) plan. +- [`BACKDROP_PRELOAD.md`](BACKDROP_PRELOAD.md) — widescreen backdrop preload. + +## Overlay cache internals +- [`SLJIT_PERSIST_CACHE.md`](SLJIT_PERSIST_CACHE.md) — persisted overlay-shard + cache design. diff --git a/SLJIT_PERSIST_CACHE.md b/docs/internal/SLJIT_PERSIST_CACHE.md similarity index 100% rename from SLJIT_PERSIST_CACHE.md rename to docs/internal/SLJIT_PERSIST_CACHE.md diff --git a/STUBS_TO_FIX.md b/docs/internal/STUBS_TO_FIX.md similarity index 100% rename from STUBS_TO_FIX.md rename to docs/internal/STUBS_TO_FIX.md diff --git a/WEDGE_load4_shell_rootcause.md b/docs/internal/WEDGE_load4_shell_rootcause.md similarity index 100% rename from WEDGE_load4_shell_rootcause.md rename to docs/internal/WEDGE_load4_shell_rootcause.md diff --git a/runtime/src/code_provider.c b/runtime/src/code_provider.c index 18930588..cbae5926 100644 --- a/runtime/src/code_provider.c +++ b/runtime/src/code_provider.c @@ -1,11 +1,14 @@ -/* code_provider.c — the two CodeProvider implementations + active selection. - * See code_provider.h for the abstraction and SLJIT.md §3/§6/§7 for the plan. +/* code_provider.c — the CodeProvider implementations + active selection. + * See code_provider.h for the abstraction. * - * This is the step-2 seam: the gcc provider is a thin pass-through over the - * validated autocompile_* path (no behavior change), and the sljit provider - * wraps the in-process JIT. Which one is active is decided once at init by - * overlay_backend_resolve (env PSX_OVERLAY_BACKEND > [runtime] overlay_backend > - * auto; auto => gcc if a compile cmd is wired, else sljit). gcc is the default. */ + * The "spawn compiler" provider is a thin pass-through over the validated + * autocompile_* path (compile emitted overlay C to a DLL and load it); it backs + * BOTH the gcc and tcc tiers, which differ only in which compiler the pipeline + * invokes. The in-process JIT provider (code_provider_sljit) is a separate, + * legacy path. Which provider is active is decided once at init by + * overlay_backend_resolve() in overlay_backend.c (env PSX_OVERLAY_BACKEND > + * [runtime] overlay_backend > auto). The auto tier order is: + * static -> gcc (if gcc on PATH) -> tcc (bundled toolchain-free fallback). */ #include