mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-28 19:07:47 -04:00
Matthew Stanley
e6ce5006b4
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
186 lines
9.1 KiB
Markdown
186 lines
9.1 KiB
Markdown
# Architecture
|
|
|
|
How PSXRecomp is put together, end to end. For the higher-level "why three ways
|
|
of running code" story, read [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) first;
|
|
this doc is the component-level view.
|
|
|
|
## Two programs: the recompiler and the runtime
|
|
|
|
PSXRecomp is split into two CMake projects that are built and run separately:
|
|
|
|
```
|
|
recompiler/ runtime/
|
|
─────────── ────────
|
|
MIPS ──▶ C C + hardware simulation ──▶ native game binary
|
|
(build-time tool) (linked with the generated C)
|
|
```
|
|
|
|
- **`recompiler/`** (C++20) is an offline tool. It reads MIPS R3000A machine code
|
|
and emits C source files under `generated/`.
|
|
- **`runtime/`** (C99 + C++17) is the engine. It loads the game's assets into an
|
|
emulated PS1 address space, links the generated C in as native functions, and
|
|
simulates the console's hardware around them.
|
|
|
|
A **game repository** (e.g. TombaRecomp) contains no framework source — it links
|
|
this framework in (as a submodule) and provides the game's config, seeds, and
|
|
build glue. See [`BUILDING.md`](BUILDING.md#linking-the-framework).
|
|
|
|
## The recompiler (`recompiler/`)
|
|
|
|
Two entry points share the same translation core:
|
|
|
|
- `src/main_bios.cpp` — ingests the flat BIOS ROM (`SCPH1001.BIN`, loaded at
|
|
`0xBFC00000`) and emits `generated/SCPH1001_*.c`.
|
|
- `src/main_psx.cpp` — ingests a `PS-X EXE` extracted from a game disc and emits
|
|
`generated/<serial>_*.c`.
|
|
|
|
The translation pipeline:
|
|
|
|
| Stage | File | Job |
|
|
|---|---|---|
|
|
| Decode | `mips_decoder.cpp` (+ vendored `rabbitizer`) | Decode MIPS instructions |
|
|
| Control flow | `control_flow.cpp`, `basic_block.cpp` | Build basic blocks / CFG |
|
|
| Discovery | `function_discovery.cpp`, `function_analysis.cpp` | Find function entry points (seeded from Ghidra exports) |
|
|
| Codegen | `code_generator.cpp`, `full_function_emitter.cpp`, `strict_translator.cpp` | Emit C: one function per guest function, plus a dispatch table |
|
|
|
|
Output is **two files** per program: a `_full.c` (the function bodies) and a
|
|
`_dispatch.c` (the address→function dispatch table). Both are build artifacts and
|
|
are **never hand-edited** — if the C is wrong, the fix is in the recompiler.
|
|
|
|
GTE (the PS1 geometry coprocessor) instructions are emitted inline;
|
|
COP0 kernel-mode instructions the BIOS needs are handled in codegen.
|
|
|
|
## The runtime (`runtime/`)
|
|
|
|
The runtime is assembled by one CMake helper, `psxrecomp_add_runtime_target()`
|
|
(defined in `runtime/runtime.cmake`), which a game's `CMakeLists.txt` calls with
|
|
the paths to its generated C. The runtime provides:
|
|
|
|
- **Memory & address space** (`memory.c`) — the 2 MB main RAM, scratchpad, BIOS
|
|
ROM, and MMIO regions, with the dispatch that routes a guest PC to its native
|
|
function (static, overlay, or interpreter).
|
|
- **Hardware simulation via MMIO handlers** — GPU (`gpu.c`, `gpu_sw_renderer.c`,
|
|
`gpu_gl_renderer.c`), DMA (`dma.c`), timers (`timers.c`), CD-ROM
|
|
(`cdrom.c`, `iso_reader.cpp`), MDEC, SIO0 controllers/memory cards
|
|
(`sio.c`, `memcard.c`), SPU (`spu.c`), GTE (`gte.cpp`), and interrupt delivery
|
|
(`interrupts.c`).
|
|
- **Host services** — window/input/audio via SDL3 by default (SDL2 is an
|
|
explicit compatibility backend), a cooperative-thread
|
|
scheduler on host fibers (`psx_fiber.c`: Win32 Fibers / POSIX `ucontext`), and
|
|
the optional debug TCP server (`debug_server.c`).
|
|
|
|
### BIOS: LLE baseline + a swappable HLE tier
|
|
|
|
The recompiled `SCPH1001.BIN` is the **low-level (LLE) baseline**: it *is* the
|
|
kernel, and it is the reference implementation and the correctness oracle. There
|
|
are no per-vector HLE shims replacing it.
|
|
|
|
On top of that, PSXRecomp carries an optional **HLE tier** (`bios_hle`, on by
|
|
default for player convenience) that skips the BIOS boot sequence and intercepts
|
|
a small set of BIOS services — always falling through to the recompiled BIOS for
|
|
anything it doesn't implement. LLE stays fully linked and is what every accuracy
|
|
check runs against. Turn it off with `[runtime] bios_hle = false` or
|
|
`PSX_BIOS_HLE=0`; with it off the build behaves as pure LLE. (Design notes:
|
|
[`docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md`](internal/HLE_SCHEDULER_CARVEOUT_PLAN.md).)
|
|
|
|
Those are **two independent axes**, and the distinction matters because a build
|
|
links more than one BIOS ([`BIOS_SELECTION.md`](BIOS_SELECTION.md)):
|
|
|
|
| axis | what it needs from the image | on retail SCPH-1001 | on bundled OpenBIOS |
|
|
|---|---|---|---|
|
|
| boot-skip | `shell_entry_phys` — works under pure LLE | yes | yes |
|
|
| kernel-call HLE | `deliver_event_ret` — the kernel's own DeliverEvent `$ra` | yes | refused, loudly |
|
|
|
|
So **"skip the BIOS and go straight to the game" means the same thing on every
|
|
BIOS**: the boot-skip is not synthesis, it just returns immediately from the
|
|
shell call, so it needs nothing BIOS-specific beyond knowing where the shell is
|
|
entered. Whether the *kernel-call* tier is additionally available is a separate,
|
|
per-image question, and refusing it must never cancel the boot-skip — deriving
|
|
one from the other is the bug fixed in 2026-07. Both axes are decided in one pure
|
|
place, `psx_bios_hle_plan()`
|
|
([`runtime/include/bios_hle_plan.h`](../runtime/include/bios_hle_plan.h)), which
|
|
is unit-tested over the whole matrix.
|
|
|
|
### Static / overlay / interpreter dispatch
|
|
|
|
This is the heart of the system and has its own doc,
|
|
[`EXECUTION_MODEL.md`](EXECUTION_MODEL.md). In brief: a guest PC resolves to
|
|
statically-recompiled native code (BIOS + main EXE), a runtime-compiled **native
|
|
overlay** (`overlay_capture.c` → `tools/compile_overlays.py` →
|
|
`overlay_loader.c`), or the small **dirty-RAM interpreter**
|
|
(`dirty_ram_interp.c`) — in that priority order.
|
|
|
|
### Overlay compile backend
|
|
|
|
When an overlay needs compiling, the runtime spawns a C compiler on the
|
|
recompiler-emitted C and loads the resulting DLL — it does not JIT in-process.
|
|
The backend tier is resolved in `overlay_backend.c` (see `main.cpp` around the
|
|
`code_provider` setup):
|
|
|
|
```
|
|
static → gcc → tcc
|
|
```
|
|
|
|
- **static** — the overlay was baked into the binary at build time (best case).
|
|
- **gcc** — used when a system `gcc` is on `PATH` (the development default).
|
|
- **tcc** — TinyCC, the toolchain-free fallback **bundled beside shipped
|
|
executables** in `overlay_toolchain/` (an embedded Python + `tcc.exe`) so
|
|
players never need a compiler installed.
|
|
|
|
Compiled overlays are stored in a content-addressed cache namespaced by
|
|
compiler and target ABI (`<game>/gcc/<arch-abi>/…` vs `<game>/tcc/…`); a `gcc`
|
|
shard wins over a `tcc` shard for the same region. See
|
|
[`docs/FEATURES.md`](FEATURES.md), [`docs/OVERLAY_CACHE_V2.md`](OVERLAY_CACHE_V2.md),
|
|
and [`docs/ASYNC_OVERLAY_COMPILE.md`](ASYNC_OVERLAY_COMPILE.md).
|
|
|
|
## Renderers
|
|
|
|
Three GPU backends behind one interface:
|
|
|
|
- **Software rasterizer** — CPU, most portable, the reference look.
|
|
- **OpenGL** — GPU-authoritative VRAM/FBO renderer, the default; moves
|
|
rasterization and supersampling onto the GPU. Falls back to software if GL
|
|
init fails. (See [`docs/internal/GL_RENDERER_HANDOFF.md`](internal/GL_RENDERER_HANDOFF.md).)
|
|
- **Vulkan** — experimental. The build option `PSX_ENABLE_VULKAN` defaults **ON**
|
|
(compiled when the SDK tools are present), but it is not the runtime default
|
|
renderer: selecting it also requires the game to offer Vulkan and the user to
|
|
request it, otherwise the runtime falls back to OpenGL.
|
|
|
|
Widescreen (a genuine wider GTE FOV, not a stretch) is opt-in and gen-time; see
|
|
[`WIDESCREEN.md`](WIDESCREEN.md) and
|
|
[`docs/internal/NATIVE_WIDE_PLAN.md`](internal/NATIVE_WIDE_PLAN.md).
|
|
|
|
## The oracle model (how correctness is checked)
|
|
|
|
PSXRecomp validates itself against **Beetle PSX** (the mednafen-psx libretro
|
|
core), run as a **separate process** with an identical TCP JSON debug protocol:
|
|
|
|
- `psx-runtime` — the recompiled runtime (debug server on port 4370).
|
|
- `psx-beetle` — Beetle PSX (debug server on port 4380).
|
|
|
|
A tool written against one works against the other by switching ports; cross-
|
|
checking is done by querying both, never by sharing state in one process. There
|
|
is also a first-divergence **co-sim** build that cycle-locksteps the compiled
|
|
backend against the interpreter. See
|
|
[`docs/internal/COSIM_ORACLE.md`](internal/COSIM_ORACLE.md),
|
|
[`docs/config_schema.md`](config_schema.md), and [`TCP_COMMANDS.md`](TCP_COMMANDS.md).
|
|
|
|
## Configuration
|
|
|
|
A game is configured by its **game config** (`game.toml`). A **BIOS config**
|
|
(`bios/*.toml`) describes a BIOS image's identity and address model.
|
|
|
|
The two are **not merged.** The BIOS config is consumed only by the recompiler
|
|
at build time (`psxrecomp-bios`, and `psxrecomp-game` for the address model);
|
|
the shipping runtime never loads it — `runtime/src/main.cpp` contains no call to
|
|
`load_bios_config`. `[runtime]` keys therefore take effect only from `game.toml`,
|
|
`settings.toml`, the CLI, and the environment. Full schema:
|
|
[`docs/config_schema.md`](config_schema.md).
|
|
|
|
## Where to go next
|
|
|
|
- [`EXECUTION_MODEL.md`](EXECUTION_MODEL.md) — static/native/interp in depth.
|
|
- [`BUILDING.md`](BUILDING.md) — dependencies + build steps.
|
|
- [`../CONTRIBUTING.md`](../CONTRIBUTING.md) — dev workflow and rules.
|
|
- [`../CLAUDE.md`](../CLAUDE.md) — the exhaustive engineering constitution.
|