From 60919f5a87682ff1b5d37f23afa6caffbb233d03 Mon Sep 17 00:00:00 2001 From: Alex Vanderveen Date: Wed, 30 Sep 2026 19:20:59 -0400 Subject: [PATCH] bios: build a retail backend from the player's own dump as a loadable module Bundled releases link OpenBIOS only; retail-BIOS-derived C never ships. "Bring your own BIOS" now works without it: when a dump matching a shipped profile (psx_bios_known_images.h) is selected, the runtime builds that backend on the player's machine with the overlay toolchain the release already carries, caches it, and loads it as one more registered backend. - runtime/include/psx_bios_module.h: the module contract. An overlay-style self-contained shared library (no imports from the executable) exporting psx_bios_module_{abi,codegen_hash,init,backend}; reaches the runtime only through OverlayCallbacks plus a new PsxBiosModuleCallbacks table (traps, dirty-RAM/kernel-bless probes, fntrace, log_probe) and pointer aliases of the globals the generated dispatcher touches. Its #defines rewrite the emitter's own extern lines, so the generated C compiles unchanged. - runtime/include/psx_bios_module_glue.c.inc: module-side forwarders and the four exports, compiled after overlay_dispatch_preamble.c.inc. - runtime/src/psx_bios_module.c: host side. Cache path /cache/bios//bm__f/_, ABI + codegen-hash gated load, callback fill, and a blocking build via overlay_toolchain/bios_module_build.py (gcc via compiler.txt or PATH, else tcc), then psx_bios_register(). - psx_bios_backend.c: the registry is now a mutable table seeded from the CMake-generated psx_bios_builtin_registry by a constructor, with psx_bios_register(); every existing walker keeps working. - main.cpp: validate_bios_for_launch acquires a module (announcing a first build) before declaring a mismatch; discovery accepts a buildable dump; the launcher verify reports it usable ("ready" / "compiled on first launch") with no Generate & rebuild; the BIOS row shows whenever the toolchain is present; bundled_only respects module support. - emitter: the CPS-marker constructor is not emitted in a module build. - overlay_api.h: OverlayCallbacks gains a struct tag; overlay_loader_callbacks() exposes the host table before loader init. - tools/bios_module_build.py (ships in the toolchain): identity + profile SHA-256 gate, staged emitter tree, emit, three-TU compile, ctypes self-check, atomic publish. release_stage.py stages psxrecomp-bios, every bios/*.toml, the seeds and the builder into overlay_toolchain/. - Tests: runtime/tests/test_bios_module_host.c against three fake modules (good / wrong ABI / wrong hash) on real dlopen; recompiler/tests/ test_bios_module_build.py builds the bundled OpenBIOS end to end and reads the descriptor back; guards extended. Verified on Twisted Metal 4 (bundled build + staged toolchain, headless): first launch with --bios SCPH1001.BIN builds and loads the module and boots on image=SCPH-1001; the second launch loads it from cache. Co-Authored-By: Claude Fable 5.1 --- docs/BIOS_SELECTION.md | 47 +- docs/ci/BUNDLED_RELEASES.md | 7 +- docs/ci/README.md | 3 +- recompiler/CMakeLists.txt | 11 + recompiler/src/full_function_emitter.cpp | 5 + recompiler/tests/test_bios_module_build.py | 152 ++++++ runtime/CMakeLists.txt | 31 ++ runtime/include/overlay_api.h | 5 +- runtime/include/overlay_loader.h | 9 + runtime/include/psx_bios_backend.h | 19 +- runtime/include/psx_bios_module.h | 181 +++++++ runtime/include/psx_bios_module_glue.c.inc | 127 +++++ runtime/runtime.cmake | 5 +- runtime/src/main.cpp | 97 +++- runtime/src/overlay_loader.c | 10 + runtime/src/psx_bios_backend.c | 58 +++ runtime/src/psx_bios_module.c | 527 ++++++++++++++++++++ runtime/tests/bios_module_fake.c | 74 +++ runtime/tests/test_bios_module_host.c | 148 ++++++ runtime/tests/test_bios_selection_guards.py | 19 +- tools/bios_module_build.py | 318 ++++++++++++ tools/release_stage.py | 36 +- 22 files changed, 1852 insertions(+), 37 deletions(-) create mode 100644 recompiler/tests/test_bios_module_build.py create mode 100644 runtime/include/psx_bios_module.h create mode 100644 runtime/include/psx_bios_module_glue.c.inc create mode 100644 runtime/src/psx_bios_module.c create mode 100644 runtime/tests/bios_module_fake.c create mode 100644 runtime/tests/test_bios_module_host.c create mode 100644 tools/bios_module_build.py diff --git a/docs/BIOS_SELECTION.md b/docs/BIOS_SELECTION.md index b5bdb076..8e948c12 100644 --- a/docs/BIOS_SELECTION.md +++ b/docs/BIOS_SELECTION.md @@ -15,19 +15,42 @@ is decided when the game launches, not when it is built. translation of the Sony ROM's code, so it is never committed or built in CI (`tools/ci/check_generated.sh` refuses it; see `docs/ci/BUNDLED_RELEASES.md`). Requiring the player's dump at runtime would not change what the zip -distributes. In such a build the launcher hides the BIOS row, an explicit -`--bios` or stale `bios.cfg` is ignored, and a verify of a retail image -reports "runs its bundled OpenBIOS only" rather than offering -Generate & rebuild, which has no CLI, sources or toolchain to run there. +distributes. -Planned, not built (2026-09-30): a **player-side backend build**. The bundle -already ships `overlay_toolchain/` and compiles game code from the player's -own disc at runtime; the same mechanism can validate a retail dump, run -`psxrecomp-bios` on it, compile the resulting C into a loadable module and -register it, so "bring your own BIOS" works without any Sony-derived code in -the release. It needs the backend registry to accept a dynamically loaded -`PsxBiosBackend`, `psxrecomp-bios` in the toolchain, and a compiler on Linux -and macOS where the toolchain relies on the system one. +**Bring your own BIOS: the backend is built on the player's machine.** The +release already ships `overlay_toolchain/` and compiles game code from the +player's own disc at runtime; the same mechanism builds a retail backend from +the player's own dump (`runtime/include/psx_bios_module.h`, +`runtime/src/psx_bios_module.c`, `tools/bios_module_build.py`). When the +player selects a dump whose size and CRC match a shipped profile +(`psx_bios_known_images.h`: SCPH-1001, SCPH-5552), the runtime: + +1. looks for `/cache/bios//bm__f/_.{so,dll}` + and loads it if present (ABI tag and codegen hash gated, like an overlay shard); +2. otherwise announces a one-time build, runs `bios_module_build.py` from the + toolchain (the bundled `psxrecomp-bios` recompiles the dump — the profile's + SHA-256 pin refuses any other image — and tcc or the system compiler turns + the C plus the module glue into a self-contained shared library), then + loads it; +3. registers the descriptor with `psx_bios_register()` and activates it. Every + existing selection path then sees it as one more linked backend: `bios.cfg` + remembers it, netplay settle sees it, and it is hot-swapped like OpenBIOS. + +The module imports nothing from the executable. Everything it calls goes +through `OverlayCallbacks` plus a BIOS-specific `PsxBiosModuleCallbacks` +table, and the runtime globals the generated dispatcher reads are aliased by +pointer (the `#define`s in `psx_bios_module.h` rewrite the emitter's own +`extern` lines). The generated C is untouched apart from the emitter no longer +writing its CPS-marker constructor in a module build. A build takes about a +minute with gcc for the 32 MB SCPH-1001 output, seconds with tcc; it happens +once per dump and per framework codegen hash. + +What the launcher shows in a bundled build: the BIOS row is present when the +toolchain is beside the executable (it is, in every release zip). A matching +dump verifies as "ready" or "compiled on first launch"; an image no profile +covers is refused with the accepted list. Without the toolchain the row is +hidden and a stray `--bios` is ignored, and no build ever offers +Generate & rebuild for a BIOS. ## The rule diff --git a/docs/ci/BUNDLED_RELEASES.md b/docs/ci/BUNDLED_RELEASES.md index 8b153f26..b35fa5c5 100644 --- a/docs/ci/BUNDLED_RELEASES.md +++ b/docs/ci/BUNDLED_RELEASES.md @@ -63,8 +63,9 @@ A release relies on two things instead: - the **static AOT shard** committed under `generated/` and linked into the executable (titles with an `aot/overlays.json` `static_output`); - the bundled **`overlay_toolchain/`** (pinned relocatable Python + - `psxrecomp-game` + headers, tcc on Windows), which lets the runtime compile - any other overlay it streams from the player's disc. + `psxrecomp-game` + `psxrecomp-bios` + headers + BIOS profiles and seeds, tcc + on Windows), which lets the runtime compile any other overlay it streams + from the player's disc, and a retail BIOS backend from the player's dump. A developer packaging locally with a cache can ship it: `PSX_OVERLAY_CACHE_ROOT= scripts/package_release.sh …`. The packager @@ -90,7 +91,7 @@ link-only rebuild of the tens of MB of generated C. |------|---------| | Install | Extract the zip. Run the game. Pick the disc on first run. | | Update | Extract the new zip over the old one (raw zip extract: this is a prebuilt Play binary). Saves and settings live beside the exe and are preserved. | -| Retail BIOS | Not needed and not shipped; the release runs on the bundled OpenBIOS. | +| Retail BIOS | Not shipped. A player who owns a dump picks it in the launcher; the release builds that backend from the dump on their machine, once (`docs/BIOS_SELECTION.md`). | ## Title checklist (short) diff --git a/docs/ci/README.md b/docs/ci/README.md index 025b9fab..516badcf 100644 --- a/docs/ci/README.md +++ b/docs/ci/README.md @@ -60,7 +60,8 @@ the ICD dynamically via SDL; CI only needs headers and the shader compiler. | `ci/sign_windows.sh` | Authenticode-sign the staged Windows binaries (no-op without a certificate) | | `fetch_toolchain.sh` | Download/unpack cmake-clang-v1 (Windows emitter builds on the CI machine only) | | `package_game_release.sh` | Bundled zip: compiled game + runtime data + OpenBIOS + mods + overlay toolchain | -| `release_stage.py` | Shared staging surface (mod catalog, overlay toolchain, overlay cache tag/shards) | +| `release_stage.py` | Shared staging surface (mod catalog, overlay toolchain incl. the BIOS emitter/profiles/seeds, overlay cache tag/shards) | +| `bios_module_build.py` | Ships inside the toolchain; builds a retail BIOS backend from the player's dump at runtime | | `bundle_mingw_dlls.sh` | Copy imported non-system DLLs next to Windows PEs | | `templates/game.gitignore` | Suggested gitignore for title repos (`generated/` tracked) | diff --git a/recompiler/CMakeLists.txt b/recompiler/CMakeLists.txt index 12865c49..56e8c7aa 100644 --- a/recompiler/CMakeLists.txt +++ b/recompiler/CMakeLists.txt @@ -730,6 +730,17 @@ if(BUILD_TESTING) ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_cli_project_packaging.py WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) + # tools/bios_module_build.py end to end: recompile the bundled OpenBIOS + # image with the psxrecomp-bios built here, compile it as a loadable + # BIOS module, and read the descriptor back (psx_bios_module.h). + # Skips itself without a C compiler on PATH. + add_test( + NAME bios_module_build + COMMAND ${Python3_EXECUTABLE} + ${CMAKE_CURRENT_SOURCE_DIR}/tests/test_bios_module_build.py + --emitter $ + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) + # tools/coverage_vault.py's history compaction and streaming JSON # reader. Pure unit test — loads the module by path, needs no built # target and no generated BIOS. It sat in tools/ with no add_test() diff --git a/recompiler/src/full_function_emitter.cpp b/recompiler/src/full_function_emitter.cpp index 6fd212d6..43cf7d08 100644 --- a/recompiler/src/full_function_emitter.cpp +++ b/recompiler/src/full_function_emitter.cpp @@ -2498,6 +2498,10 @@ void FullFunctionEmitter::emit_dispatch( // RECURSION_BUG.md §25 — mark CPS mode at startup for runtime code that // routes CPS continuations (overlay_loader.c). out += "\n/* CPS runtime-mode marker (the overlay loader reads g_psx_cps_mode). */\n"; + out += "/* Not in a loadable BIOS module (psx_bios_module.h): the host that\n"; + out += " * loads one already runs in CPS mode, and a constructor firing at\n"; + out += " * dlopen would write a runtime global the module cannot reach. */\n"; + out += "#ifndef PSX_BIOS_MODULE_BUILD\n"; out += "static void psx_cps_mark_bios(void) {\n"; out += " extern int g_psx_cps_mode; g_psx_cps_mode = 1;\n"; out += "}\n"; @@ -2509,6 +2513,7 @@ void FullFunctionEmitter::emit_dispatch( out += "#else\n"; out += "__attribute__((constructor)) static void psx_cps_mark_bios_ctor(void) { psx_cps_mark_bios(); }\n"; out += "#endif\n"; + out += "#endif /* !PSX_BIOS_MODULE_BUILD */\n"; } // --- Backend descriptor (runtime/include/psx_bios_backend.h) --- diff --git a/recompiler/tests/test_bios_module_build.py b/recompiler/tests/test_bios_module_build.py new file mode 100644 index 00000000..33be7c31 --- /dev/null +++ b/recompiler/tests/test_bios_module_build.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""tools/bios_module_build.py end to end against the real emitter. + +Builds a loadable backend from the bundled, redistributable OpenBIOS image +(the only image this tree can legally test with), then loads the result in +this interpreter and reads the descriptor back through ctypes: + + * the module is self-contained (RTLD_NOW resolves everything or fails), + * it exports exactly the psx_bios_module.h entry points, + * its ABI tag carries the requested flavor and its codegen hash is the one + runtime/include/overlay_codegen_hash.h (or overlay_api.h's default) names, + * the descriptor's image identity is OpenBIOS's real size and CRC-32, + * a dump that does not match the profile's SHA-256 pin is refused before + anything is compiled. + +Needs the built psxrecomp-bios (--emitter, default recompiler/build) and a C +compiler on PATH; skips (exit 0 with a notice) when either is absent, since a +BIOS-free tree cannot prove anything here. + + python3 recompiler/tests/test_bios_module_build.py [--emitter PATH] +""" + +import argparse +import ctypes +import os +import re +import shutil +import subprocess +import sys +import tempfile +import zlib + +ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..')) +TOOLS = os.path.join(ROOT, 'tools') + + +def codegen_hash_expected(): + p = os.path.join(ROOT, 'runtime', 'include', 'overlay_codegen_hash.h') + if os.path.isfile(p): + m = re.search(r'PSX_OVERLAY_CODEGEN_HASH\s+(0x[0-9a-fA-F]+)', open(p).read()) + if m: + return int(m.group(1), 16) + return None # header absent: overlay_api.h defaults to 0u + + +class PsxBiosImageInfo(ctypes.Structure): + _fields_ = [('kbless_ram_lo', ctypes.c_uint32), ('kbless_ram_hi', ctypes.c_uint32), + ('kbless_rom_off', ctypes.c_uint32), ('shell_entry_phys', ctypes.c_uint32), + ('deliver_event_ret', ctypes.c_uint32), ('image_size', ctypes.c_uint32), + ('image_crc32', ctypes.c_uint32), ('image_wordsum', ctypes.c_uint32), + ('image_sha256', ctypes.c_char_p), ('image_id', ctypes.c_char_p), + ('image_bundled', ctypes.c_int)] + + +class PsxBiosBackend(ctypes.Structure): + _fields_ = [('image', ctypes.POINTER(PsxBiosImageInfo)), + ('dispatch', ctypes.c_void_p), ('dispatch_call', ctypes.c_void_p), + ('kernel_bodies', ctypes.c_void_p), ('kernel_body_count', ctypes.c_uint32), + ('kernel_patch_ranges', ctypes.c_void_p), ('kernel_patch_range_count', ctypes.c_uint32)] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('--emitter', default=None) + args = ap.parse_args() + emitter = args.emitter + if not emitter: + for c in ('recompiler/build/psxrecomp-bios', 'recompiler/build/psxrecomp-bios.exe', + 'recompiler/build-t2/psxrecomp-bios'): + if os.path.isfile(os.path.join(ROOT, c)): + emitter = os.path.join(ROOT, c) + break + if not emitter or not os.path.isfile(emitter): + print('SKIP: psxrecomp-bios not built (pass --emitter)') + return 0 + cc = shutil.which('gcc') or shutil.which('cc') or shutil.which('clang') + if not cc: + print('SKIP: no C compiler on PATH') + return 0 + if sys.platform.startswith('win'): + print('SKIP: ctypes descriptor read is validated on POSIX; the host test covers Windows loading') + return 0 + + dump = os.path.join(ROOT, 'bios', 'openbios.bin') + with open(dump, 'rb') as f: + data = f.read() + want_crc = zlib.crc32(data) & 0xFFFFFFFF + + work = tempfile.mkdtemp(prefix='bios_module_test_') + try: + out = os.path.join(work, 'OpenBIOS_test.so') + base = [sys.executable, os.path.join(TOOLS, 'bios_module_build.py'), + '--toolchain', work, # no staged layout: every piece overridden + '--emitter', emitter, + '--include', os.path.join(ROOT, 'runtime', 'include'), + '--profiles', os.path.join(ROOT, 'bios'), + '--seeds', os.path.join(ROOT, 'recompiler', 'seeds'), + '--stem', 'OpenBIOS', '--compiler', 'gcc', '--gcc', cc] + + # A wrong image is refused by the profile's SHA-256 pin, before compiling. + wrong = os.path.join(work, 'wrong.bin') + with open(wrong, 'wb') as f: + f.write(bytes(len(data))) + r = subprocess.run(base + ['--dump', wrong, '--out', out, '--flavor', '0'], + capture_output=True, text=True) + assert r.returncode != 0, 'a zero-filled image must be refused' + assert 'not OPENBIOS' in r.stdout or 'is not' in r.stdout, r.stdout[-600:] + assert not os.path.exists(out), 'refusal must leave no module' + + # The real image builds, with flavor 2 so the tag's flavor bits are exercised. + r = subprocess.run(base + ['--dump', dump, '--out', out, '--flavor', '2'], + capture_output=True, text=True) + assert r.returncode == 0, r.stdout[-3000:] + r.stderr[-1000:] + assert f'PSX_BIOS_MODULE_PUBLISHED {out}' in r.stdout, r.stdout[-600:] + assert os.path.isfile(out) + + lib = ctypes.CDLL(out) # RTLD_NOW: an unresolved symbol fails here + lib.psx_bios_module_abi.restype = ctypes.c_int + lib.psx_bios_module_codegen_hash.restype = ctypes.c_uint32 + lib.psx_bios_module_backend.restype = ctypes.POINTER(PsxBiosBackend) + abi = lib.psx_bios_module_abi() + assert (abi & 0xFFFF) == 1, f'ABI version {abi & 0xFFFF}' + assert ((abi >> 16) & 0xFFFF) == 2, f'flavor bits {(abi >> 16) & 0xFFFF}' + h = lib.psx_bios_module_codegen_hash() + want_h = codegen_hash_expected() + if want_h is not None: + assert h == want_h, f'codegen hash {h:#x} != header {want_h:#x}' + b = lib.psx_bios_module_backend().contents + img = b.image.contents + assert img.image_id == b'OPENBIOS', img.image_id + assert img.image_size == len(data), img.image_size + assert img.image_crc32 == want_crc, f'{img.image_crc32:#x} != {want_crc:#x}' + assert img.image_bundled == 1 + assert b.dispatch and b.dispatch_call, 'dispatch entry points missing' + assert b.kernel_body_count > 0, 'no kernel bodies published' + + # The CPS constructor must not be in a module (it would write a host global). + nm = shutil.which('nm') + if nm: + syms = subprocess.run([nm, '-D', '--undefined-only', out], capture_output=True, text=True).stdout + assert 'g_psx_cps_mode' not in syms, 'module still references g_psx_cps_mode' + names = [ln.split()[-1] for ln in syms.splitlines() if ln.strip()] + bad = [n for n in names if not n.startswith('_') and '@' not in n] + assert not bad, f'module imports non-libc symbols: {bad}' + print(f'OK: OpenBIOS module built and loaded (abi={abi:#x}, hash={h:#x}, crc={img.image_crc32:08X})') + return 0 + finally: + shutil.rmtree(work, ignore_errors=True) + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/runtime/CMakeLists.txt b/runtime/CMakeLists.txt index 49905e35..c8144547 100644 --- a/runtime/CMakeLists.txt +++ b/runtime/CMakeLists.txt @@ -264,6 +264,37 @@ if(BUILD_TESTING) add_test(NAME isc_store_test COMMAND isc_store_test) endif() + # Loadable BIOS backends (psx_bios_module.h): the host loader against + # three fake modules built here from one source -- good, wrong ABI tag, + # wrong codegen hash. Real dlopen/LoadLibrary, real registry, stubbed core. + if(NOT MSVC) + foreach(_fake IN ITEMS good bad_abi bad_hash) + add_library(bios_module_fake_${_fake} SHARED tests/bios_module_fake.c) + target_include_directories(bios_module_fake_${_fake} PRIVATE include) + target_compile_definitions(bios_module_fake_${_fake} PRIVATE + PSX_OVERLAY_DLL_BUILD PSX_NO_DEBUG_TOOLS) + set_target_properties(bios_module_fake_${_fake} PROPERTIES PREFIX "") + endforeach() + target_compile_definitions(bios_module_fake_bad_abi PRIVATE FAKE_ABI_TAG=0x7fff) + target_compile_definitions(bios_module_fake_bad_hash PRIVATE FAKE_CODEGEN_HASH=0x1u) + add_executable(bios_module_host_test + tests/test_bios_module_host.c + src/psx_bios_module.c + src/psx_bios_backend.c + src/overlay_posix.c + src/crc32.c) + target_include_directories(bios_module_host_test PRIVATE include src) + target_compile_definitions(bios_module_host_test PRIVATE + PSX_NO_DEBUG_TOOLS + FAKE_GOOD="$" + FAKE_BAD_ABI="$" + FAKE_BAD_HASH="$") + target_link_libraries(bios_module_host_test PRIVATE ${CMAKE_DL_LIBS}) + add_dependencies(bios_module_host_test + bios_module_fake_good bios_module_fake_bad_abi bios_module_fake_bad_hash) + add_test(NAME bios_module_host_test COMMAND bios_module_host_test) + endif() + add_executable(host_keymap_test tests/test_host_keymap.c src/host_keymap.c) diff --git a/runtime/include/overlay_api.h b/runtime/include/overlay_api.h index 540159e4..99cffe93 100644 --- a/runtime/include/overlay_api.h +++ b/runtime/include/overlay_api.h @@ -198,7 +198,10 @@ # define PSX_OVERLAY_CODEGEN_HASH 0u #endif -typedef struct { +/* Named so consumers that only forward the table (overlay_loader.h, + * psx_bios_module.h) can declare `struct OverlayCallbacks` without pulling + * this header in. Layout unchanged. */ +typedef struct OverlayCallbacks { /* Core dispatch: routes call_by_address() and out-of-overlay jal */ void (*dispatch_call)(CPUState *cpu, uint32_t addr, uint32_t ra); /* Interrupt check: called after every function return in overlay */ diff --git a/runtime/include/overlay_loader.h b/runtime/include/overlay_loader.h index d77e170f..ed954d4c 100644 --- a/runtime/include/overlay_loader.h +++ b/runtime/include/overlay_loader.h @@ -40,6 +40,15 @@ static inline int psx_overlay_code_segment_pc(uint32_t pc) { /* Called at game handoff to set the cache root directory and game ID. * cache_dir: absolute path to the cache root (e.g. "build-dev/cache") * game_id: product code (e.g. "SCUS-94236") */ +/* Host callback table handed to every loaded module (overlay shards and BIOS + * modules alike). Safe before overlay_loader_init. */ +struct OverlayCallbacks; +const struct OverlayCallbacks *overlay_loader_callbacks(void); + +/* "-" of this build (PSX_OVERLAY_ARCH_ABI), the cache layout's + * arch-abi segment. Also keyed into the BIOS module cache. */ +const char *overlay_loader_arch_abi(void); + void overlay_loader_init(const char *cache_dir, const char *game_id, uint32_t config_hash); diff --git a/runtime/include/psx_bios_backend.h b/runtime/include/psx_bios_backend.h index 62398197..8d525d1f 100644 --- a/runtime/include/psx_bios_backend.h +++ b/runtime/include/psx_bios_backend.h @@ -58,14 +58,29 @@ typedef struct PsxBiosBackend { uint32_t kernel_patch_range_count; } PsxBiosBackend; +/* Dispatch nesting depth, shared by every backend (psx_bios_backend.c). */ +extern int g_psx_dispatch_depth; + /* The backend in use. Null before psx_bios_select() runs; every forwarder and * every consumer of psx_bios_image depends on it being set first. */ extern const PsxBiosBackend *psx_bios_active; /* Backends compiled into this binary, in preference order (bundled OpenBIOS * first). Emitted by the build as psx_bios_registry.c. */ -extern const PsxBiosBackend *const psx_bios_registry[]; -extern const uint32_t psx_bios_registry_count; +extern const PsxBiosBackend *const psx_bios_builtin_registry[]; +extern const uint32_t psx_bios_builtin_registry_count; + +/* The live registry: the builtins plus any backend registered at run time + * (a BIOS module built from the player's dump, psx_bios_module.h). Walk it + * only after psx_bios_registry_seeded_count() or any lookup below has run; + * the count is 0 until then in a process that never looked. */ +extern const PsxBiosBackend *psx_bios_registry[]; +extern uint32_t psx_bios_registry_count; +uint32_t psx_bios_registry_seeded_count(void); + +/* Add a backend loaded at run time. Idempotent for a pointer already present; + * 0 when the table is full or the descriptor has no image. */ +int psx_bios_register(const PsxBiosBackend *backend); /* Look up a compiled-in backend by its profile id ("SCPH-1001", "OPENBIOS"). * Null if this build does not carry it. */ diff --git a/runtime/include/psx_bios_module.h b/runtime/include/psx_bios_module.h new file mode 100644 index 00000000..a3b71d26 --- /dev/null +++ b/runtime/include/psx_bios_module.h @@ -0,0 +1,181 @@ +/* psx_bios_module.h — a recompiled BIOS backend as a separately loaded module. + * + * WHY. A bundled release links only the redistributable OpenBIOS backend: the + * C a retail image recompiles to is a translation of that image's code, so it + * never ships (docs/BIOS_SELECTION.md, docs/ci/BUNDLED_RELEASES.md). For a + * player who owns a retail dump, the release therefore builds that backend on + * THEIR machine, from THEIR dump, with the overlay toolchain the release + * already carries — the same mechanism that turns overlays streamed from the + * player's disc into native code — and loads it here as a shared library. + * + * THE CONTRACT. A BIOS module is an overlay-style self-contained library. It + * imports nothing from the executable (the executable exports no symbols on + * any platform); every call back into the runtime goes through two callback + * tables handed over at init: + * + * OverlayCallbacks (overlay_api.h) — the part a BIOS shares with + * overlay shards: cycles, loads, + * GTE, syscall, dispatch_call… + * PsxBiosModuleCallbacks (this header) — what only a BIOS dispatcher + * needs: traps, the dirty-RAM + * and kernel-bless probes, and + * the runtime globals the + * generated dispatch reads and + * writes (through pointers). + * + * The generated C is compiled UNCHANGED with -DPSX_OVERLAY_DLL_BUILD (so the + * inline RAM-load and cycle fast paths in cpu_state.h / psx_cyc.h / + * psx_cycles.h become out-of-line calls the glue forwards) plus + * -DPSX_BIOS_MODULE_BUILD, with this header prepended: the mapping block + * below renames the handful of runtime globals the emitter declares `extern` + * to pointer dereferences, exactly as cpu_state.h does for the bail state. + * A macro'd `extern T g_x;` becomes `extern T (*g_x_p);`, which is the + * pointer's own declaration — no per-symbol shim is needed in the C. + * + * The module exports four functions (psx_bios_module_glue.c.inc): + * int psx_bios_module_abi(void) == PSX_BIOS_MODULE_ABI_TAG + * uint32_t psx_bios_module_codegen_hash(void) == PSX_OVERLAY_CODEGEN_HASH + * int psx_bios_module_init(const OverlayCallbacks*, const PsxBiosModuleCallbacks*) + * const PsxBiosBackend *psx_bios_module_backend(void) + * and is otherwise opaque. The host (runtime/src/psx_bios_module.c) gates a + * load on the first two, calls init, then registers the descriptor with + * psx_bios_register() so every existing selection path sees it as one more + * linked backend. + * + * ABI. Bump PSX_BIOS_MODULE_ABI_VERSION whenever PsxBiosModuleCallbacks + * changes shape, a mapping is added, or the glue's forwarding semantics + * change. Append-only growth of the table is still a bump: a module built + * against a shorter table must not be handed a longer one it never checks. + * The cache directory name carries this version, the codegen hash, and the + * flavor, so a stale module is never even opened. + */ +#ifndef PSX_BIOS_MODULE_H +#define PSX_BIOS_MODULE_H + +#include +#include + +/* ---- Name mappings (module build only) ------------------------------------- + * Pure #defines, deliberately with no declarations: each generated `extern` + * line, and psx_memory.h's own `extern uint32_t g_psx_ram_mask;`, declares + * the pointer with the right type once the name is rewritten. Prepended by + * tools/bios_module_build.py before the first #include, so psx_memory.h sees + * the g_psx_ram_mask mapping too (the emitted normalize() masks with it). */ +#if defined(PSX_BIOS_MODULE_BUILD) +/* The one global the emitter uses without its own extern line (cpu_state.h + * declares it only outside PSX_OVERLAY_DLL_BUILD), so declare the pointer. */ +extern uint64_t *g_psx_bm_bail_flattened_p; +# define g_psx_bios_hle_hook (*g_psx_bm_hle_hook_p) +# define g_psx_dispatch_depth (*g_psx_bm_dispatch_depth_p) +# define g_dispatch_static_hits (*g_psx_bm_static_hits_p) +# define g_debug_current_func_addr (*g_psx_bm_current_func_addr_p) +# define g_psx_bail_flattened (*g_psx_bm_bail_flattened_p) +# define g_psx_ram_mask (*g_psx_bm_ram_mask_p) +# define s_frame_count (*g_psx_bm_frame_count_p) +#endif + +#ifdef __cplusplus +extern "C" { +#endif + +struct CPUState; +struct OverlayCallbacks; +struct PsxBiosBackend; + +#define PSX_BIOS_MODULE_ABI_VERSION 1 + +/* Combined tag: ABI version (low 16) | codegen flavor (high 16), like + * PSX_OVERLAY_ABI_TAG. The flavor comes from the same -DPSX_OVERLAY_FLAVOR + * the executable was built with; the glue and the host both evaluate it. */ +#ifndef PSX_OVERLAY_FLAVOR +#define PSX_OVERLAY_FLAVOR 0 +#endif +#define PSX_BIOS_MODULE_ABI_TAG \ + ((int)((PSX_BIOS_MODULE_ABI_VERSION & 0xFFFF) | ((PSX_OVERLAY_FLAVOR & 0xFFFF) << 16))) + +/* What the BIOS dispatcher needs beyond OverlayCallbacks. Function members + * are the runtime's real implementations; pointer members alias the + * runtime's globals so a module read/write is the runtime's read/write. */ +typedef struct PsxBiosModuleCallbacks { + uint32_t size; /* sizeof(PsxBiosModuleCallbacks) on the host side */ + + /* Traps (runtime/src/traps.c; strict_translator emits these). */ + void (*arith_overflow)(struct CPUState *cpu); + void (*brk)(struct CPUState *cpu, uint32_t code, uint32_t pc); + void (*unaligned_access)(struct CPUState *cpu, uint32_t addr, uint32_t pc); + void (*rfe_escape_check)(struct CPUState *cpu); + + /* Dispatch probes the generated dispatcher consults per iteration. */ + int (*dirty_ram_dispatch)(struct CPUState *cpu, uint32_t addr, uint32_t stop_addr); + int (*dirty_ram_is_dirty)(uint32_t phys); + int (*dirty_ram_text_native_ok)(uint32_t phys); + int (*kernel_bless_dispatchable)(uint32_t phys); + int (*game_address_in_text)(uint32_t addr); + int (*fntrace_is_game_started)(void); + void (*fntrace_record)(struct CPUState *cpu, uint32_t target); + void (*trace_dispatch)(uint32_t func_addr); + /* Retail images emit probe sites (SCPH-1001 does; OpenBIOS does not). */ + void (*log_probe)(uint32_t pc, struct CPUState *cpu); + /* Segment-aware dispatch: a PC served by another segment's home body is + * recorded in the runtime's segment-miss ring (psx_segment_miss.h). */ + void (*segment_miss_record_kind)(uint32_t addr, uint32_t home, uint32_t ra, + uint32_t sp, uint32_t frame, uint32_t kind); + + /* Runtime-owned state the dispatcher reads/writes. Never NULL from a + * host at this ABI; the glue keeps local dummies for a pre-init call. */ + int (**hle_hook)(struct CPUState *cpu, uint32_t phys); /* bios_hle.c */ + int *dispatch_depth; /* psx_bios_backend.c */ + uint64_t *static_hits; /* generated game dispatch counter */ + uint32_t *current_func_addr; /* debug_server.c */ + uint64_t *bail_flattened; /* traps.c */ + uint32_t *ram_mask; /* psx_ram_geometry.c (live 2/8 MiB) */ + uint64_t *frame_count; /* debug_server.c s_frame_count (miss records) */ +} PsxBiosModuleCallbacks; + +/* ---- Host side (runtime/src/psx_bios_module.c) ----------------------------- */ +#if !defined(PSX_BIOS_MODULE_BUILD) + +/* Can this process turn a retail dump into a loaded backend? True when a + * bundled backend is linked (this is a product build, not a setup host) and + * the overlay toolchain is present beside the executable. */ +int psx_bios_module_supported(const char *exe_dir); + +/* The profile identity a dump of this size/CRC would build with, or NULL + * when no shipped profile matches (then no module can be built). */ +const char *psx_bios_module_known_id(uint32_t crc32, uint32_t size); + +/* Load a previously built module for this dump if one is cached, else build + * it with the toolchain (blocking; seconds with tcc, minutes with gcc on a + * large image), then load and register it. Returns the registered backend + * or NULL with `err` filled. `exe_dir` is the executable's directory; the + * cache lives under /cache/bios/. */ +const struct PsxBiosBackend *psx_bios_module_acquire(const char *dump_path, + const char *exe_dir, + int allow_build, + char *err, size_t err_cap); + +/* 1 when a module for this dump is already built and cached (no build + * needed to select it), 0 otherwise or when the dump is not a known image. */ +int psx_bios_module_is_cached(const char *dump_path, const char *exe_dir); + +/* Cached module path for a dump identity, whether or not it exists. */ +int psx_bios_module_cache_path(const char *exe_dir, const char *stem, + uint32_t crc32, char *out, size_t cap); + +/* Load one module file: ABI + codegen-hash gate, init, register. */ +const struct PsxBiosBackend *psx_bios_module_load(const char *path, + char *err, size_t err_cap); + +/* Callback tables the host hands to a module (also used by the test host). */ +void psx_bios_module_fill_callbacks(PsxBiosModuleCallbacks *out); + +/* Last build/load diagnostic line, for logs and the launcher. */ +const char *psx_bios_module_last_message(void); + +#endif /* !PSX_BIOS_MODULE_BUILD */ + +#ifdef __cplusplus +} +#endif + +#endif /* PSX_BIOS_MODULE_H */ diff --git a/runtime/include/psx_bios_module_glue.c.inc b/runtime/include/psx_bios_module_glue.c.inc new file mode 100644 index 00000000..e43e57cd --- /dev/null +++ b/runtime/include/psx_bios_module_glue.c.inc @@ -0,0 +1,127 @@ +/* psx_bios_module_glue.c.inc — the module side of a loadable BIOS backend. + * + * Compiled ONCE per module as its own translation unit by + * tools/bios_module_build.py, after overlay_dispatch_preamble.c.inc (which + * defines g_cbs, overlay_flush_cycles and every forwarder a BIOS shares with + * overlay shards). This file adds what only a BIOS dispatcher needs — see + * psx_bios_module.h — and the four exported entry points. + * + * Contract: nothing here may reach a runtime symbol directly. Every call and + * every global goes through g_bcbs / g_cbs, or through the pointers the + * generated C's macro-rewritten externs declare. PSX_BIOS_MODULE_STEM is the + * image stem (-DPSX_BIOS_MODULE_STEM=SCPH1001) and names the descriptor the + * generated dispatch exports. + */ +#include "psx_bios_backend.h" +#include "psx_bios_module.h" + +#ifndef PSX_BIOS_MODULE_STEM +#error "PSX_BIOS_MODULE_STEM must name the image stem (e.g. -DPSX_BIOS_MODULE_STEM=SCPH1001)" +#endif +#define PSX_BM_CAT2(a, b) a##b +#define PSX_BM_CAT(a, b) PSX_BM_CAT2(a, b) +#define PSX_BM_DESCRIPTOR PSX_BM_CAT(PSX_BIOS_MODULE_STEM, _psx_bios_backend) + +extern const PsxBiosBackend PSX_BM_DESCRIPTOR; + +static PsxBiosModuleCallbacks g_bcbs; + +/* Local dummies so a call before init cannot fault; init repoints them. */ +static int (*s_hle_hook_dummy)(CPUState *cpu, uint32_t phys); +static int s_depth_dummy; +static uint64_t s_hits_dummy; +static uint32_t s_func_addr_dummy; +static uint64_t s_flattened_dummy; +static uint32_t s_ram_mask_dummy = 0x1FFFFFu; +static uint64_t s_frame_count_dummy; + +/* The pointers the generated C declares (via the psx_bios_module.h name + * mappings): `extern int g_psx_dispatch_depth;` became + * `extern int (*g_psx_bm_dispatch_depth_p);`, and so on. Defined here. */ +int (**g_psx_bm_hle_hook_p)(CPUState *cpu, uint32_t phys) = &s_hle_hook_dummy; +int *g_psx_bm_dispatch_depth_p = &s_depth_dummy; +uint64_t *g_psx_bm_static_hits_p = &s_hits_dummy; +uint32_t *g_psx_bm_current_func_addr_p = &s_func_addr_dummy; +uint64_t *g_psx_bm_bail_flattened_p = &s_flattened_dummy; +uint32_t *g_psx_bm_ram_mask_p = &s_ram_mask_dummy; +uint64_t *g_psx_bm_frame_count_p = &s_frame_count_dummy; + +/* ---- Forwarders for what OverlayCallbacks does not carry ----------------- */ +void psx_arith_overflow(CPUState *cpu) { + overlay_flush_cycles(); + if (g_bcbs.arith_overflow) g_bcbs.arith_overflow(cpu); +} +void psx_break(CPUState *cpu, uint32_t code, uint32_t pc) { + overlay_flush_cycles(); + if (g_bcbs.brk) g_bcbs.brk(cpu, code, pc); +} +void psx_unaligned_access(CPUState *cpu, uint32_t addr, uint32_t pc) { + overlay_flush_cycles(); + if (g_bcbs.unaligned_access) g_bcbs.unaligned_access(cpu, addr, pc); +} +void psx_rfe_escape_check(CPUState *cpu) { + overlay_flush_cycles(); + if (g_bcbs.rfe_escape_check) g_bcbs.rfe_escape_check(cpu); +} +int dirty_ram_dispatch(CPUState *cpu, uint32_t addr, uint32_t stop_addr) { + overlay_flush_cycles(); + return g_bcbs.dirty_ram_dispatch ? g_bcbs.dirty_ram_dispatch(cpu, addr, stop_addr) : 0; +} +int dirty_ram_is_dirty(uint32_t phys) { + return g_bcbs.dirty_ram_is_dirty ? g_bcbs.dirty_ram_is_dirty(phys) : 0; +} +int dirty_ram_text_native_ok(uint32_t phys) { + return g_bcbs.dirty_ram_text_native_ok ? g_bcbs.dirty_ram_text_native_ok(phys) : 0; +} +int psx_kernel_bless_dispatchable(uint32_t phys) { + return g_bcbs.kernel_bless_dispatchable ? g_bcbs.kernel_bless_dispatchable(phys) : 0; +} +int psx_game_address_in_text(uint32_t addr) { + return g_bcbs.game_address_in_text ? g_bcbs.game_address_in_text(addr) : 0; +} +int fntrace_is_game_started(void) { + return g_bcbs.fntrace_is_game_started ? g_bcbs.fntrace_is_game_started() : 0; +} +void fntrace_record(CPUState *cpu, uint32_t target) { + if (g_bcbs.fntrace_record) g_bcbs.fntrace_record(cpu, target); +} +void debug_server_trace_dispatch(uint32_t func_addr) { + if (g_bcbs.trace_dispatch) g_bcbs.trace_dispatch(func_addr); +} +void debug_server_log_probe(uint32_t pc, CPUState *cpu) { + if (g_bcbs.log_probe) g_bcbs.log_probe(pc, cpu); +} +void psx_segment_miss_record_kind(uint32_t addr, uint32_t home, uint32_t ra, + uint32_t sp, uint32_t frame, uint32_t kind) { + if (g_bcbs.segment_miss_record_kind) + g_bcbs.segment_miss_record_kind(addr, home, ra, sp, frame, kind); +} + +/* ---- Exports ------------------------------------------------------------- */ +PSX_OVERLAY_EXPORT int psx_bios_module_abi(void) { + return PSX_BIOS_MODULE_ABI_TAG; +} +PSX_OVERLAY_EXPORT uint32_t psx_bios_module_codegen_hash(void) { + return (uint32_t)PSX_OVERLAY_CODEGEN_HASH; +} +PSX_OVERLAY_EXPORT int psx_bios_module_init(const OverlayCallbacks *cbs, + const PsxBiosModuleCallbacks *bcbs) { + if (!cbs || !bcbs) return 0; + /* The host's table must be at least as long as ours: a shorter one would + * leave our tail members reading past its end. The ABI tag already pins + * the version; this is the belt to that suspender. */ + if (bcbs->size < sizeof(PsxBiosModuleCallbacks)) return 0; + overlay_init(cbs); + g_bcbs = *bcbs; + if (g_bcbs.hle_hook) g_psx_bm_hle_hook_p = g_bcbs.hle_hook; + if (g_bcbs.dispatch_depth) g_psx_bm_dispatch_depth_p = g_bcbs.dispatch_depth; + if (g_bcbs.static_hits) g_psx_bm_static_hits_p = g_bcbs.static_hits; + if (g_bcbs.current_func_addr) g_psx_bm_current_func_addr_p = g_bcbs.current_func_addr; + if (g_bcbs.bail_flattened) g_psx_bm_bail_flattened_p = g_bcbs.bail_flattened; + if (g_bcbs.ram_mask) g_psx_bm_ram_mask_p = g_bcbs.ram_mask; + if (g_bcbs.frame_count) g_psx_bm_frame_count_p = g_bcbs.frame_count; + return 1; +} +PSX_OVERLAY_EXPORT const PsxBiosBackend *psx_bios_module_backend(void) { + return &PSX_BM_DESCRIPTOR; +} diff --git a/runtime/runtime.cmake b/runtime/runtime.cmake index 61530fd8..e2794015 100644 --- a/runtime/runtime.cmake +++ b/runtime/runtime.cmake @@ -428,6 +428,7 @@ set(PSXRECOMP_RUNTIME_SOURCES ${PSXRECOMP_ROOT}/runtime/src/mod_texture_banks.c ${PSXRECOMP_ROOT}/runtime/src/psx_keybinds.c ${PSXRECOMP_ROOT}/runtime/src/psx_bios_backend.c + ${PSXRECOMP_ROOT}/runtime/src/psx_bios_module.c ${PSXRECOMP_ROOT}/runtime/src/psx_netplay.c ${PSXRECOMP_ROOT}/runtime/src/psx_lobby_client.c ${PSXRECOMP_ROOT}/runtime/src/netplay_bios_settle.c @@ -741,12 +742,12 @@ file(WRITE "${_psxrt_registry_c}" " "${_psxrt_registry_externs}" " -const PsxBiosBackend *const psx_bios_registry[] = { +const PsxBiosBackend *const psx_bios_builtin_registry[] = { " "${_psxrt_registry_entries}" "}; " -"const uint32_t psx_bios_registry_count = ${_psxrt_bios_count}u; +"const uint32_t psx_bios_builtin_registry_count = ${_psxrt_bios_count}u; ") list(APPEND PSXRECOMP_BIOS_GENERATED "${_psxrt_registry_c}") diff --git a/runtime/src/main.cpp b/runtime/src/main.cpp index 0fff114a..45d51168 100644 --- a/runtime/src/main.cpp +++ b/runtime/src/main.cpp @@ -20,6 +20,7 @@ #include "bios_hle_plan.h" #include "psx_bios_known_images.h" #include "psx_bios_backend.h" +#include "psx_bios_module.h" #include "psx_cycles.h" #include "starvation_ring.h" #include "load_accel.h" @@ -2696,6 +2697,16 @@ static const PsxBiosBackend* bios_backend_for_file(const std::filesystem::path& /* What a player may supply, for mismatch and picker copy. The bundled image is * excluded: it is never something to go and find. */ +/* Directory of the running executable, for the BIOS module cache and + * toolchain (psx_bios_module.h). Set by resolve_bios_for_runtime before any + * selection; empty means "unknown", which disables module building. */ +static std::string s_bios_exe_dir; + +static bool bios_module_supported_here() { + return !s_bios_exe_dir.empty() && + psx_bios_module_supported(s_bios_exe_dir.c_str()) != 0; +} + static std::string bios_accepted_images() { std::string s; for (uint32_t i = 0; i < psx_bios_registry_count; i++) { @@ -2705,6 +2716,16 @@ static std::string bios_accepted_images() { s += b->image->image_id; s += " (" + std::to_string(b->image->image_size / 1024u) + " KB)"; } + /* A bundled build can also BUILD a backend for any image it ships a + * profile for (psx_bios_module.h), so those count as accepted too. */ + if (bios_module_supported_here()) { + for (const PsxKnownBiosImage& k : psx_known_bios_images) { + if (s.find(k.id) != std::string::npos) continue; + if (!s.empty()) s += ", "; + s += k.id; + s += " (" + std::to_string(k.size / 1024u) + " KB, built on first use)"; + } + } return s.empty() ? std::string("(this build ships its own BIOS)") : s; } @@ -2714,6 +2735,32 @@ static bool validate_bios_for_launch(const std::filesystem::path& path) { const PsxBiosBackend* b = bios_backend_for_file(path, &crc, &size); if (b) return psx_bios_activate(b) != 0; + /* Not linked, but this build can build it from the player's dump + * (docs/BIOS_SELECTION.md, "player-side backend build"). A cached + * module loads silently; a first build is announced, since it can take + * a minute with gcc, and its failure explained before falling through + * to the ordinary mismatch/fallback path. */ + if (bios_module_supported_here() && + psx_bios_module_known_id(crc, (uint32_t)size)) { + const std::string dump = path.string(); + char err[256] = {0}; + if (!psx_bios_module_is_cached(dump.c_str(), s_bios_exe_dir.c_str())) { + launcher_info("Preparing your BIOS", + std::string(psx_bios_module_known_id(crc, (uint32_t)size)) + + " is not part of this build, so it will be compiled from your " + "own image now. This happens once and can take a minute; the " + "game starts when it finishes."); + } + const PsxBiosBackend* m = psx_bios_module_acquire( + dump.c_str(), s_bios_exe_dir.c_str(), /*allow_build=*/1, err, sizeof(err)); + if (m) return psx_bios_activate(m) != 0; + launcher_warning("BIOS Not Prepared", + std::string("Could not build a backend for this BIOS: ") + err + + "\n\nSee the log beside the executable. Clear the BIOS field to " + "play on the bundled OpenBIOS."); + return false; + } + char buf[512]; std::snprintf(buf, sizeof(buf), "That BIOS (%llu bytes, CRC32 %08X) is not an image this build was " @@ -2743,7 +2790,9 @@ static std::filesystem::path resolve_bios_for_runtime(const char* requested, bool requested_is_explicit) { const bool openbios_allowed = s_openbios_allowed; const PsxBiosBackend* bundled = psx_bios_bundled(); - const bool player_bios_selectable = psx_bios_has_selectable() != 0; + s_bios_exe_dir = exe_dir_from_argv(argv0).string(); + const bool player_bios_selectable = + psx_bios_has_selectable() != 0 || bios_module_supported_here(); const bool bundled_only = openbios_allowed && bundled && !player_bios_selectable; @@ -2938,8 +2987,12 @@ static bool retail_bios_file_ok(const std::filesystem::path& path) { std::error_code ec; if (!std::filesystem::is_regular_file(path, ec)) return false; if (psx_bios_registry_count > 0) { - const PsxBiosBackend* b = bios_backend_for_file(path, nullptr, nullptr); - return b && b->image && !b->image->image_bundled; + uint32_t crc = 0; uint64_t size = 0; + const PsxBiosBackend* b = bios_backend_for_file(path, &crc, &size); + if (b) return b->image && !b->image->image_bundled; + /* Bundled build: a dump this build can compile a backend from. */ + return bios_module_supported_here() && + psx_bios_module_known_id(crc, (uint32_t)size) != nullptr; } /* Setup host (no backends linked yet): accept the retail image THIS build * pins, from psx_bios_known_images.h. This used to hardcode SCPH-1001, so @@ -8062,8 +8115,13 @@ namespace { /* Bundled build with only its shipped backend and no way to compile * another: a retail image can never be used here, whatever its CRC. * Say so before the identity checks, whose "this build expects - * SCPH-1001" wording describes the pinned stem, not a linked backend. */ - if (!g_lnch_can_regen && psx_bios_registry_count > 0 && + * SCPH-1001" wording describes the pinned stem, not a linked backend. + * (With the toolchain present a module CAN be built; that case falls + * through to the identity check and the module branch below.) */ + const bool lnch_module_ok = + psx_bios_module_supported(exe_dir_from_argv( + g_lnch_argv0 ? g_lnch_argv0 : "").string().c_str()) != 0; + if (!g_lnch_can_regen && !lnch_module_ok && psx_bios_registry_count > 0 && !psx_bios_has_selectable()) { out->ok = 0; out->needs_regen = 0; @@ -8135,7 +8193,22 @@ namespace { return 1; } out->ok = 0; - if (g_lnch_can_regen) { + if (lnch_module_ok && psx_bios_module_known_id(crc, (uint32_t)size)) { + /* Buildable from this dump: usable, no Generate & rebuild. + * The build itself runs at launch (validate_bios_for_launch). */ + const std::string exe_dir = + exe_dir_from_argv(g_lnch_argv0 ? g_lnch_argv0 : "").string(); + const bool cached = + psx_bios_module_is_cached(open_path.c_str(), exe_dir.c_str()) != 0; + out->ok = 1; + out->warn = 0; + out->needs_regen = 0; + std::snprintf(out->detail, sizeof(out->detail), + cached ? "%s (CRC OK, ready)." + : "%s (CRC OK). Compiled from your image on first " + "launch — one-time, about a minute.", + psx_bios_module_known_id(crc, (uint32_t)size)); + } else if (g_lnch_can_regen) { out->needs_regen = 1; std::snprintf(out->detail, sizeof(out->detail), "This BIOS is not compiled into the current build. " @@ -14198,7 +14271,9 @@ int main(int argc, char** argv) { recomp_launcher_set_preserve_sdl(1); int lr = 2; /* 0 = launch, 1 = quit, 2 = unavailable */ const bool bios_choice_supported = - psx_bios_has_selectable() != 0 || psx_bios_registry_count == 0; + psx_bios_has_selectable() != 0 || psx_bios_registry_count == 0 || + psx_bios_module_supported( + exe_dir_from_argv(argv[0]).string().c_str()) != 0; PSXRecompV4::UserSettings seed; /* Netplay session BIOS is match-only; never overwrite seed/bios.cfg. */ std::filesystem::path match_session_bios_path; @@ -14619,7 +14694,9 @@ int main(int argc, char** argv) { * offered "Generate & rebuild" with no CLI or toolchain to run it. */ g_lnch_can_regen = gi.prepare_with_progress != nullptr || psx_bios_registry_count == 0; - gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen) ? 1 : 0; + gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen || + psx_bios_module_supported( + exe_dir_from_argv(argv[0]).string().c_str())) ? 1 : 0; #endif #endif /* PSX_HAS_SETUP_WIZARD */ launcher_boot_timing_mark("host:setup_checks_done"); @@ -16745,7 +16822,9 @@ soft_return_lobby: psx_game_codegen_setup_apply(&gi); g_lnch_can_regen = gi.prepare_with_progress != nullptr || psx_bios_registry_count == 0; - gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen) ? 1 : 0; + gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen || + psx_bios_module_supported( + exe_dir_from_argv(argv[0]).string().c_str())) ? 1 : 0; #endif char rui_out_disc[1024] = {0}; diff --git a/runtime/src/overlay_loader.c b/runtime/src/overlay_loader.c index 80efe260..1ce4d94a 100644 --- a/runtime/src/overlay_loader.c +++ b/runtime/src/overlay_loader.c @@ -2951,6 +2951,16 @@ static int load_bios_resident_shards(void) { return functions; } +/* The host callback table, for a consumer that loads overlay-style modules + * before (or without) overlay_loader_init: the BIOS module host activates a + * player's retail backend during BIOS selection, ahead of the deferred + * overlay init thread. init_callbacks() is pure assignment, so filling it + * here and again in overlay_loader_init is harmless. */ +const OverlayCallbacks *overlay_loader_callbacks(void) { + init_callbacks(); + return &s_callbacks; +} + void overlay_loader_init(const char *cache_dir, const char *game_id, uint32_t config_hash) { { diff --git a/runtime/src/psx_bios_backend.c b/runtime/src/psx_bios_backend.c index d00340be..d7890aa2 100644 --- a/runtime/src/psx_bios_backend.c +++ b/runtime/src/psx_bios_backend.c @@ -39,6 +39,61 @@ uint32_t psx_bios_kernel_patch_range_count = 0; * could not be linked together. */ int g_psx_dispatch_depth = 0; +/* ── Registry ─────────────────────────────────────────────────────────────── + * + * The build lists the backends it LINKS in psx_bios_builtin_registry (written + * by runtime.cmake as psx_bios_registry.c). The registry every consumer walks + * is this mutable table: it starts as a copy of the builtins and grows when a + * BIOS module built on the player's machine registers itself + * (psx_bios_module.c). A backend is never removed: the published image and + * kernel tables may point into it for the life of the process. */ +#define PSX_BIOS_REGISTRY_CAP 8 + +const PsxBiosBackend *psx_bios_registry[PSX_BIOS_REGISTRY_CAP]; +uint32_t psx_bios_registry_count = 0; +static int s_registry_seeded = 0; + +static void registry_seed(void) +{ + uint32_t i; + if (s_registry_seeded) return; + s_registry_seeded = 1; + for (i = 0; i < psx_bios_builtin_registry_count && i < PSX_BIOS_REGISTRY_CAP; i++) { + const PsxBiosBackend *b = psx_bios_builtin_registry[i]; + if (b) psx_bios_registry[psx_bios_registry_count++] = b; + } +} + +int psx_bios_register(const PsxBiosBackend *backend) +{ + uint32_t i; + registry_seed(); + if (!backend || !backend->image) return 0; + for (i = 0; i < psx_bios_registry_count; i++) + if (psx_bios_registry[i] == backend) return 1; + if (psx_bios_registry_count >= PSX_BIOS_REGISTRY_CAP) return 0; + psx_bios_registry[psx_bios_registry_count++] = backend; + return 1; +} + +uint32_t psx_bios_registry_seeded_count(void) +{ + registry_seed(); + return psx_bios_registry_count; +} + +/* main.cpp walks psx_bios_registry[] directly in a dozen places, some before + * any lookup here has run, and "count == 0" there means "setup host". Seed + * before main() so those walks are never wrong; the same constructor idiom + * the generated BIOS uses for its CPS marker. */ +#if defined(_MSC_VER) +static void psx_bios_registry_ctor(void) { registry_seed(); } +#pragma section(".CRT$XCU", read) +__declspec(allocate(".CRT$XCU")) static void (*psx_bios_registry_ctor_p)(void) = psx_bios_registry_ctor; +#else +__attribute__((constructor)) static void psx_bios_registry_ctor(void) { registry_seed(); } +#endif + /* ── Forwarders ───────────────────────────────────────────────────────────── * * The generated BIOS C defines _psx_dispatch; these keep the unprefixed @@ -60,6 +115,7 @@ void psx_dispatch_call(CPUState *cpu, uint32_t addr, uint32_t return_addr) const PsxBiosBackend *psx_bios_find(const char *image_id) { + registry_seed(); if (!image_id) return 0; for (uint32_t i = 0; i < psx_bios_registry_count; i++) { const PsxBiosBackend *b = psx_bios_registry[i]; @@ -72,6 +128,7 @@ const PsxBiosBackend *psx_bios_find(const char *image_id) const PsxBiosBackend *psx_bios_bundled(void) { + registry_seed(); for (uint32_t i = 0; i < psx_bios_registry_count; i++) { const PsxBiosBackend *b = psx_bios_registry[i]; if (b && b->image && b->image->image_bundled) return b; @@ -102,6 +159,7 @@ int psx_bios_activate(const PsxBiosBackend *backend) * the retail one (and clear back). */ int psx_bios_has_selectable(void) { + registry_seed(); for (uint32_t i = 0; i < psx_bios_registry_count; i++) { const PsxBiosBackend *b = psx_bios_registry[i]; if (b && b->image && !b->image->image_bundled) return 1; diff --git a/runtime/src/psx_bios_module.c b/runtime/src/psx_bios_module.c new file mode 100644 index 00000000..79365b89 --- /dev/null +++ b/runtime/src/psx_bios_module.c @@ -0,0 +1,527 @@ +/* psx_bios_module.c — host side of loadable BIOS backends (psx_bios_module.h). + * + * A bundled release links OpenBIOS only. When the player picks a retail dump + * whose identity matches a shipped profile, this file (1) looks for a module + * already built for that dump under /cache/bios, (2) otherwise runs the + * bundled overlay toolchain's bios_module_build.py to emit and compile one + * from the dump, then (3) loads it overlay-style — ABI and codegen-hash + * gated, callback tables handed over at init — and registers its descriptor + * so the ordinary selection code (main.cpp) treats it as one more linked + * backend. + * + * Nothing here formats the overlay cache tag; the BIOS module directory has + * its own shape (bm__f), written here and read + * back by bios_module_build.py --print-cache-dir, which are the only two + * places that know it. + */ +#include +#include +#include +#include +#include + +#if defined(_WIN32) +# define WIN32_LEAN_AND_MEAN +# include +# include +#else +# include +# include "overlay_posix.h" +#endif + +#include "autocompile.h" +#include "cpu_state.h" +#include "crc32.h" +#include "debug_server.h" +#include "dirty_ram_interp.h" +#include "fntrace.h" +#include "overlay_api.h" +#include "overlay_loader.h" +#include "psx_bios_backend.h" +#include "psx_bios_known_images.h" +#include "psx_bios_module.h" +#include "psx_memory.h" +#include "psx_segment_miss.h" + +#if defined(_WIN32) +# define PSX_BM_EXT "dll" +#else +# define PSX_BM_EXT "so" +#endif + +static char s_last_msg[512]; + +static void note(const char *fmt, ...) { + va_list ap; + va_start(ap, fmt); + vsnprintf(s_last_msg, sizeof(s_last_msg), fmt, ap); + va_end(ap); + fprintf(stdout, "psx_bios_module: %s\n", s_last_msg); + fflush(stdout); +} + +const char *psx_bios_module_last_message(void) { return s_last_msg; } + +static void set_err(char *err, size_t cap, const char *msg) { + if (err && cap) { + strncpy(err, msg, cap - 1); + err[cap - 1] = '\0'; + } +} + +static int file_exists(const char *path) { + struct stat st; + return path && path[0] && stat(path, &st) == 0 && (st.st_mode & S_IFREG); +} + +static int dir_exists(const char *path) { + struct stat st; + return path && path[0] && stat(path, &st) == 0 && (st.st_mode & S_IFDIR); +} + +/* ---- Toolchain beside the executable ------------------------------------- */ +typedef struct { + char dir[1024]; + char python[1100]; + char builder[1100]; + char emitter[1100]; + char tcc[1100]; + char gcc[1100]; /* compiler.txt, or "" */ + int has_tcc; + int has_gcc; /* compiler.txt or a cc on PATH */ +} BmToolchain; + +static int probe_toolchain(const char *exe_dir, BmToolchain *tk) { + memset(tk, 0, sizeof(*tk)); + if (!exe_dir || !exe_dir[0]) return 0; + snprintf(tk->dir, sizeof(tk->dir), "%s/overlay_toolchain", exe_dir); +#if defined(_WIN32) + snprintf(tk->python, sizeof(tk->python), "%s/python/python.exe", tk->dir); + snprintf(tk->emitter, sizeof(tk->emitter), "%s/psxrecomp-bios.exe", tk->dir); + snprintf(tk->tcc, sizeof(tk->tcc), "%s/tcc/tcc.exe", tk->dir); +#else + snprintf(tk->python, sizeof(tk->python), "%s/python/bin/python3", tk->dir); + snprintf(tk->emitter, sizeof(tk->emitter), "%s/psxrecomp-bios", tk->dir); + snprintf(tk->tcc, sizeof(tk->tcc), "%s/tcc/tcc", tk->dir); +#endif + snprintf(tk->builder, sizeof(tk->builder), "%s/bios_module_build.py", tk->dir); + if (!file_exists(tk->python) || !file_exists(tk->builder) || !file_exists(tk->emitter)) + return 0; + tk->has_tcc = file_exists(tk->tcc); + { + char cf[1100]; + FILE *f; + snprintf(cf, sizeof(cf), "%s/compiler.txt", tk->dir); + f = fopen(cf, "r"); + if (f) { + if (fgets(tk->gcc, sizeof(tk->gcc), f)) { + size_t n = strlen(tk->gcc); + while (n && (tk->gcc[n - 1] == '\n' || tk->gcc[n - 1] == '\r' || + tk->gcc[n - 1] == ' ')) + tk->gcc[--n] = '\0'; + if (!file_exists(tk->gcc)) tk->gcc[0] = '\0'; + } + fclose(f); + } + } + tk->has_gcc = tk->gcc[0] || autocompile_toolchain_available(); + return tk->has_gcc || tk->has_tcc; +} + +int psx_bios_module_supported(const char *exe_dir) { + BmToolchain tk; + if (!psx_bios_bundled()) return 0; /* setup host: nothing to add to */ + return probe_toolchain(exe_dir, &tk); +} + +const char *psx_bios_module_known_id(uint32_t crc32, uint32_t size) { + size_t i; + for (i = 0; i < sizeof(psx_known_bios_images) / sizeof(psx_known_bios_images[0]); ++i) { + if (psx_known_bios_images[i].crc32 == crc32 && psx_known_bios_images[i].size == size) + return psx_known_bios_images[i].id; + } + return 0; +} + +static const PsxKnownBiosImage *known_for(uint32_t crc32, uint32_t size) { + size_t i; + for (i = 0; i < sizeof(psx_known_bios_images) / sizeof(psx_known_bios_images[0]); ++i) { + if (psx_known_bios_images[i].crc32 == crc32 && psx_known_bios_images[i].size == size) + return &psx_known_bios_images[i]; + } + return 0; +} + +/* ---- Cache layout --------------------------------------------------------- */ +int psx_bios_module_cache_path(const char *exe_dir, const char *stem, + uint32_t crc32, char *out, size_t cap) { + int n; + if (!exe_dir || !stem || !out || !cap) return 0; + n = snprintf(out, cap, "%s/cache/bios/%s/bm%d_%08x_f%d/%s_%08X." PSX_BM_EXT, + exe_dir, overlay_loader_arch_abi(), + PSX_BIOS_MODULE_ABI_VERSION, (unsigned)PSX_OVERLAY_CODEGEN_HASH, + (int)PSX_OVERLAY_FLAVOR, stem, crc32); + return n > 0 && (size_t)n < cap; +} + +/* ---- Callback tables ------------------------------------------------------ */ +void psx_bios_module_fill_callbacks(PsxBiosModuleCallbacks *out) { + extern int (*g_psx_bios_hle_hook)(struct CPUState *cpu, uint32_t phys); + extern uint64_t g_dispatch_static_hits; + extern uint64_t g_psx_bail_flattened; + extern int psx_game_address_in_text(uint32_t addr); + extern uint64_t s_frame_count; + memset(out, 0, sizeof(*out)); + out->size = (uint32_t)sizeof(*out); + out->arith_overflow = psx_arith_overflow; + out->brk = psx_break; + out->unaligned_access = psx_unaligned_access; + out->rfe_escape_check = psx_rfe_escape_check; + out->dirty_ram_dispatch = dirty_ram_dispatch; + out->dirty_ram_is_dirty = dirty_ram_is_dirty; + out->dirty_ram_text_native_ok = dirty_ram_text_native_ok; + out->kernel_bless_dispatchable = psx_kernel_bless_dispatchable; + out->game_address_in_text = psx_game_address_in_text; + out->fntrace_is_game_started = fntrace_is_game_started; + out->fntrace_record = fntrace_record; + out->trace_dispatch = debug_server_trace_dispatch; + out->log_probe = debug_server_log_probe; + out->segment_miss_record_kind = psx_segment_miss_record_kind; + out->hle_hook = &g_psx_bios_hle_hook; + out->dispatch_depth = &g_psx_dispatch_depth; + out->static_hits = &g_dispatch_static_hits; + out->current_func_addr = &g_debug_current_func_addr; + out->bail_flattened = &g_psx_bail_flattened; + out->ram_mask = &g_psx_ram_mask; + out->frame_count = &s_frame_count; +} + +/* ---- Loading -------------------------------------------------------------- */ +typedef int (*BmAbiFn)(void); +typedef uint32_t (*BmHashFn)(void); +typedef int (*BmInitFn)(const OverlayCallbacks *, const PsxBiosModuleCallbacks *); +typedef const PsxBiosBackend *(*BmBackendFn)(void); + +#if defined(_WIN32) +typedef HMODULE BmHandle; +static BmHandle bm_open(const char *path, char *err, size_t cap) { + HMODULE h = LoadLibraryA(path); + if (!h) { + char b[256]; + snprintf(b, sizeof(b), "LoadLibrary failed (%lu)", (unsigned long)GetLastError()); + set_err(err, cap, b); + } + return h; +} +static void *bm_sym(BmHandle h, const char *name) { return (void *)GetProcAddress(h, name); } +static void bm_close(BmHandle h) { FreeLibrary(h); } +#else +typedef void *BmHandle; +static BmHandle bm_open(const char *path, char *err, size_t cap) { + char e[256]; + void *h = psx_overlay_posix_library_open(path, e, sizeof(e)); + if (!h) set_err(err, cap, e); + return h; +} +static void *bm_sym(BmHandle h, const char *name) { return psx_overlay_posix_library_symbol(h, name); } +static void bm_close(BmHandle h) { psx_overlay_posix_library_close(h); } +#endif + +const PsxBiosBackend *psx_bios_module_load(const char *path, char *err, size_t err_cap) { + BmHandle h; + BmAbiFn abi_fn; + BmHashFn hash_fn; + BmInitFn init_fn; + BmBackendFn backend_fn; + const PsxBiosBackend *b; + PsxBiosModuleCallbacks bcbs; + char e[256] = {0}; + + if (!file_exists(path)) { set_err(err, err_cap, "module file missing"); return 0; } + h = bm_open(path, e, sizeof(e)); + if (!h) { set_err(err, err_cap, e); return 0; } + + abi_fn = (BmAbiFn)bm_sym(h, "psx_bios_module_abi"); + hash_fn = (BmHashFn)bm_sym(h, "psx_bios_module_codegen_hash"); + init_fn = (BmInitFn)bm_sym(h, "psx_bios_module_init"); + backend_fn = (BmBackendFn)bm_sym(h, "psx_bios_module_backend"); + if (!abi_fn || !hash_fn || !init_fn || !backend_fn) { + bm_close(h); + set_err(err, err_cap, "not a BIOS module (exports missing)"); + return 0; + } + if (abi_fn() != PSX_BIOS_MODULE_ABI_TAG) { + char b[160]; + snprintf(b, sizeof(b), "ABI tag %08x != host %08x", (unsigned)abi_fn(), + (unsigned)PSX_BIOS_MODULE_ABI_TAG); + bm_close(h); + set_err(err, err_cap, b); + return 0; + } + if (hash_fn() != (uint32_t)PSX_OVERLAY_CODEGEN_HASH) { + char b[160]; + snprintf(b, sizeof(b), "codegen hash %08x != host %08x", (unsigned)hash_fn(), + (unsigned)PSX_OVERLAY_CODEGEN_HASH); + bm_close(h); + set_err(err, err_cap, b); + return 0; + } + psx_bios_module_fill_callbacks(&bcbs); + if (!init_fn(overlay_loader_callbacks(), &bcbs)) { + bm_close(h); + set_err(err, err_cap, "module init refused the callback tables"); + return 0; + } + b = backend_fn(); + if (!b || !b->image || !b->dispatch || !b->dispatch_call) { + bm_close(h); + set_err(err, err_cap, "module descriptor incomplete"); + return 0; + } + if (!psx_bios_register(b)) { + bm_close(h); + set_err(err, err_cap, "backend registry full"); + return 0; + } + /* The handle is never closed: the registry and, once selected, the + * published psx_bios_image/kernel tables point into the module. */ + note("loaded %s (%s, CRC %08X)", path, b->image->image_id ? b->image->image_id : "?", + (unsigned)b->image->image_crc32); + return b; +} + +/* ---- Building -------------------------------------------------------------- */ +static int mkdirs(const char *path) { + char buf[1200]; + size_t i, n = strlen(path); + if (n >= sizeof(buf)) return 0; + memcpy(buf, path, n + 1); + for (i = 1; i < n; ++i) { + if (buf[i] == '/' || buf[i] == '\\') { + char c = buf[i]; + buf[i] = '\0'; +#if defined(_WIN32) + _mkdir(buf); +#else + mkdir(buf, 0755); +#endif + buf[i] = c; + } + } + return 1; +} + +/* Run a command line, streaming its output to our stdout, and return its + * exit status (or -1 when it could not be started). Windows: no console + * window; the executable is a GUI-subsystem process. */ +static int run_command(const char *cmd) { +#if defined(_WIN32) + STARTUPINFOA si; + PROCESS_INFORMATION pi; + SECURITY_ATTRIBUTES sa; + HANDLE rd = NULL, wr = NULL; + char *mutable_cmd; + DWORD code = (DWORD)-1; + size_t n = strlen(cmd); + memset(&sa, 0, sizeof(sa)); + sa.nLength = sizeof(sa); + sa.bInheritHandle = TRUE; + if (!CreatePipe(&rd, &wr, &sa, 0)) return -1; + SetHandleInformation(rd, HANDLE_FLAG_INHERIT, 0); + memset(&si, 0, sizeof(si)); + si.cb = sizeof(si); + si.dwFlags = STARTF_USESTDHANDLES; + si.hStdOutput = wr; + si.hStdError = wr; + si.hStdInput = GetStdHandle(STD_INPUT_HANDLE); + memset(&pi, 0, sizeof(pi)); + mutable_cmd = (char *)malloc(n + 1); + if (!mutable_cmd) { CloseHandle(rd); CloseHandle(wr); return -1; } + memcpy(mutable_cmd, cmd, n + 1); + if (!CreateProcessA(NULL, mutable_cmd, NULL, NULL, TRUE, + CREATE_NO_WINDOW | BELOW_NORMAL_PRIORITY_CLASS, + NULL, NULL, &si, &pi)) { + free(mutable_cmd); + CloseHandle(rd); CloseHandle(wr); + return -1; + } + free(mutable_cmd); + CloseHandle(wr); + { + char buf[512]; + DWORD got; + while (ReadFile(rd, buf, sizeof(buf) - 1, &got, NULL) && got) { + buf[got] = '\0'; + fputs(buf, stdout); + } + fflush(stdout); + } + WaitForSingleObject(pi.hProcess, INFINITE); + GetExitCodeProcess(pi.hProcess, &code); + CloseHandle(pi.hThread); + CloseHandle(pi.hProcess); + CloseHandle(rd); + return (int)code; +#else + FILE *p = popen(cmd, "r"); + char buf[512]; + int status; + if (!p) return -1; + while (fgets(buf, sizeof(buf), p)) fputs(buf, stdout); + fflush(stdout); + status = pclose(p); + if (status == -1) return -1; +# if defined(WIFEXITED) + if (WIFEXITED(status)) return WEXITSTATUS(status); + return -1; +# else + return status; +# endif +#endif +} + +static int append_q(char *buf, size_t cap, const char *s) { + size_t n = strlen(buf), m = strlen(s); + if (n + m + 3 >= cap) return 0; + buf[n] = '"'; + memcpy(buf + n + 1, s, m); + buf[n + 1 + m] = '"'; + buf[n + 2 + m] = '\0'; + return 1; +} +static int append_s(char *buf, size_t cap, const char *s) { + size_t n = strlen(buf), m = strlen(s); + if (n + m + 1 >= cap) return 0; + memcpy(buf + n, s, m + 1); + return 1; +} + +static int build_module(const BmToolchain *tk, const char *dump_path, + const char *stem, const char *out_path, + char *err, size_t err_cap) { + char cmd[6144] = {0}; + char flavor[32]; + int rc; + const char *compiler = tk->has_gcc ? "gcc" : "tcc"; + snprintf(flavor, sizeof(flavor), "%d", (int)PSX_OVERLAY_FLAVOR); + if (!append_q(cmd, sizeof(cmd), tk->python) || + !append_s(cmd, sizeof(cmd), " ") || !append_q(cmd, sizeof(cmd), tk->builder) || + !append_s(cmd, sizeof(cmd), " --dump ") || !append_q(cmd, sizeof(cmd), dump_path) || + !append_s(cmd, sizeof(cmd), " --toolchain ") || !append_q(cmd, sizeof(cmd), tk->dir) || + !append_s(cmd, sizeof(cmd), " --stem ") || !append_s(cmd, sizeof(cmd), stem) || + !append_s(cmd, sizeof(cmd), " --out ") || !append_q(cmd, sizeof(cmd), out_path) || + !append_s(cmd, sizeof(cmd), " --flavor ") || !append_s(cmd, sizeof(cmd), flavor) || + !append_s(cmd, sizeof(cmd), " --compiler ") || !append_s(cmd, sizeof(cmd), compiler)) { + set_err(err, err_cap, "build command too long"); + return 0; + } + if (tk->has_gcc && tk->gcc[0]) { + append_s(cmd, sizeof(cmd), " --gcc "); + append_q(cmd, sizeof(cmd), tk->gcc); + } + if (!tk->has_gcc) { + append_s(cmd, sizeof(cmd), " --tcc "); + append_q(cmd, sizeof(cmd), tk->tcc); + } + note("building %s backend from %s with %s (one-time; this can take a while)", + stem, dump_path, compiler); + fprintf(stdout, "psx_bios_module: %s\n", cmd); + fflush(stdout); + rc = run_command(cmd); + if (rc != 0) { + char b[200]; + snprintf(b, sizeof(b), "bios_module_build.py exited %d (see log above)", rc); + set_err(err, err_cap, b); + return 0; + } + if (!file_exists(out_path)) { + set_err(err, err_cap, "builder reported success but wrote no module"); + return 0; + } + return 1; +} + +/* ---- Entry point ------------------------------------------------------------ */ +static int read_identity(const char *path, uint32_t *crc, uint32_t *size); + +int psx_bios_module_is_cached(const char *dump_path, const char *exe_dir) { + uint32_t crc = 0, size = 0; + const PsxKnownBiosImage *img; + char path[1400]; + if (!read_identity(dump_path, &crc, &size)) return 0; + img = known_for(crc, size); + if (!img) return 0; + if (!psx_bios_module_cache_path(exe_dir, img->stem, crc, path, sizeof(path))) return 0; + return file_exists(path); +} + +static int read_identity(const char *path, uint32_t *crc, uint32_t *size) { + FILE *f = fopen(path, "rb"); + unsigned char *data; + long n; + if (!f) return 0; + if (fseek(f, 0, SEEK_END) != 0) { fclose(f); return 0; } + n = ftell(f); + if (n <= 0 || n > (8 << 20)) { fclose(f); return 0; } + rewind(f); + data = (unsigned char *)malloc((size_t)n); + if (!data) { fclose(f); return 0; } + if (fread(data, 1, (size_t)n, f) != (size_t)n) { free(data); fclose(f); return 0; } + fclose(f); + *crc = crc32_compute(data, (size_t)n); + *size = (uint32_t)n; + free(data); + return 1; +} + +const PsxBiosBackend *psx_bios_module_acquire(const char *dump_path, const char *exe_dir, + int allow_build, char *err, size_t err_cap) { + uint32_t crc = 0, size = 0; + const PsxKnownBiosImage *img; + char path[1400]; + char e[256] = {0}; + const PsxBiosBackend *b; + BmToolchain tk; + + if (!psx_bios_bundled()) { set_err(err, err_cap, "no bundled backend (setup host)"); return 0; } + if (!read_identity(dump_path, &crc, &size)) { set_err(err, err_cap, "cannot read the BIOS file"); return 0; } + img = known_for(crc, size); + if (!img) { + set_err(err, err_cap, "no shipped profile matches this image (size/CRC)"); + return 0; + } + /* Already registered (a prior acquire in this process)? */ + { + uint32_t i; + for (i = 0; i < psx_bios_registry_count; i++) { + const PsxBiosBackend *r = psx_bios_registry[i]; + if (r && r->image && r->image->image_crc32 == crc && r->image->image_size == size) + return r; + } + } + if (!psx_bios_module_cache_path(exe_dir, img->stem, crc, path, sizeof(path))) { + set_err(err, err_cap, "cache path too long"); + return 0; + } + if (file_exists(path)) { + b = psx_bios_module_load(path, e, sizeof(e)); + if (b) return b; + note("cached module %s rejected: %s (rebuilding)", path, e); + remove(path); + } + if (!allow_build) { set_err(err, err_cap, "not built yet"); return 0; } + if (!probe_toolchain(exe_dir, &tk)) { + set_err(err, err_cap, "no overlay toolchain beside the executable (or no compiler)"); + return 0; + } + mkdirs(path); + if (!build_module(&tk, dump_path, img->stem, path, err, err_cap)) return 0; + b = psx_bios_module_load(path, e, sizeof(e)); + if (!b) { + char buf[320]; + snprintf(buf, sizeof(buf), "freshly built module rejected: %s", e); + set_err(err, err_cap, buf); + return 0; + } + return b; +} diff --git a/runtime/tests/bios_module_fake.c b/runtime/tests/bios_module_fake.c new file mode 100644 index 00000000..1ded92f2 --- /dev/null +++ b/runtime/tests/bios_module_fake.c @@ -0,0 +1,74 @@ +/* A fake BIOS module for test_bios_module_host.c: exports the four entry + * points psx_bios_module.h requires, with a descriptor whose identity the + * test can assert, and records what init was handed. Built as a shared + * library by runtime/CMakeLists.txt; never shipped. + * + * Which ABI/hash it reports is chosen at compile time so the same source + * builds the good module and the two bad ones the host must reject. */ +#include +#include + +#include "cpu_state.h" +#include "overlay_api.h" +#include "psx_bios_backend.h" +#include "psx_bios_module.h" + +#ifndef FAKE_ABI_TAG +#define FAKE_ABI_TAG PSX_BIOS_MODULE_ABI_TAG +#endif +#ifndef FAKE_CODEGEN_HASH +#define FAKE_CODEGEN_HASH PSX_OVERLAY_CODEGEN_HASH +#endif + +#if defined(_WIN32) +# define FAKE_EXPORT __declspec(dllexport) +#else +# define FAKE_EXPORT __attribute__((visibility("default"))) +#endif + +static int s_dispatch_calls; +static void fake_dispatch(struct CPUState *cpu, uint32_t addr) { + (void)cpu; (void)addr; + s_dispatch_calls++; +} +static void fake_dispatch_call(struct CPUState *cpu, uint32_t addr, uint32_t ra) { + (void)cpu; (void)addr; (void)ra; + s_dispatch_calls++; +} + +static const PsxBiosImageInfo s_image = { + 0, 0, 0, + 0, 0, + 524288u, 0xDEADBEEFu, 0u, + "0000000000000000000000000000000000000000000000000000000000000000", + "FAKE-0001", + 0, +}; + +static const PsxBiosBackend s_backend = { + &s_image, fake_dispatch, fake_dispatch_call, 0, 0, 0, 0, +}; + +/* What init received, for the test to inspect through psx_bios_module_fake_probe. */ +static uint32_t s_init_bcbs_size; +static int s_init_ok; +static int *s_seen_depth; + +FAKE_EXPORT int psx_bios_module_abi(void) { return FAKE_ABI_TAG; } +FAKE_EXPORT uint32_t psx_bios_module_codegen_hash(void) { return (uint32_t)FAKE_CODEGEN_HASH; } +FAKE_EXPORT int psx_bios_module_init(const OverlayCallbacks *cbs, + const PsxBiosModuleCallbacks *bcbs) { + if (!cbs || !bcbs || bcbs->size < sizeof(PsxBiosModuleCallbacks)) return 0; + s_init_bcbs_size = bcbs->size; + s_seen_depth = bcbs->dispatch_depth; + s_init_ok = 1; + return 1; +} +FAKE_EXPORT const PsxBiosBackend *psx_bios_module_backend(void) { return &s_backend; } + +/* Test-only probe: 1 if init ran with a full table and the depth pointer + * aliases the host's (the host passes &g_psx_dispatch_depth). */ +FAKE_EXPORT int psx_bios_module_fake_probe(int *host_depth) { + return s_init_ok && s_init_bcbs_size == sizeof(PsxBiosModuleCallbacks) && + s_seen_depth == host_depth; +} diff --git a/runtime/tests/test_bios_module_host.c b/runtime/tests/test_bios_module_host.c new file mode 100644 index 00000000..65fb5a39 --- /dev/null +++ b/runtime/tests/test_bios_module_host.c @@ -0,0 +1,148 @@ +/* test_bios_module_host.c — the host side of loadable BIOS backends + * (runtime/src/psx_bios_module.c) against fake modules built by + * runtime/CMakeLists.txt from tests/bios_module_fake.c. + * + * What it proves, with the real host code and a real dlopen/LoadLibrary: + * 1. A module with the right ABI tag and codegen hash loads, is handed a + * full callback table whose pointers alias the host's globals, and is + * registered AFTER the builtins (which the constructor seeded). + * 2. A module with the wrong ABI tag is refused, and so is one with the + * wrong codegen hash; neither reaches the registry. + * 3. Loading the same module twice registers it once. + * 4. The registry seeded from psx_bios_builtin_registry before main(). + * + * Every runtime symbol psx_bios_module.c forwards is stubbed here; nothing + * about the emulation core is exercised. Paths to the three fake modules come + * from compile definitions set by CMake. + */ +#include +#include +#include + +#include "cpu_state.h" +#include "overlay_api.h" +#include "psx_bios_backend.h" +#include "psx_bios_module.h" + +/* ---- stubs for what the host forwards -------------------------------------- */ +void psx_arith_overflow(CPUState *cpu) { (void)cpu; } +void psx_break(CPUState *cpu, uint32_t code, uint32_t pc) { (void)cpu; (void)code; (void)pc; } +void psx_unaligned_access(CPUState *cpu, uint32_t addr, uint32_t pc) { (void)cpu; (void)addr; (void)pc; } +void psx_rfe_escape_check(CPUState *cpu) { (void)cpu; } +int dirty_ram_dispatch(CPUState *cpu, uint32_t addr, uint32_t stop) { (void)cpu; (void)addr; (void)stop; return 0; } +int dirty_ram_is_dirty(uint32_t phys) { (void)phys; return 0; } +int dirty_ram_text_native_ok(uint32_t phys) { (void)phys; return 0; } +int psx_kernel_bless_dispatchable(uint32_t phys) { (void)phys; return 0; } +int psx_game_address_in_text(uint32_t addr) { (void)addr; return 0; } +int fntrace_is_game_started(void) { return 0; } +void fntrace_record(CPUState *cpu, uint32_t target) { (void)cpu; (void)target; } +void debug_server_trace_dispatch(uint32_t f) { (void)f; } +void debug_server_log_probe(uint32_t pc, CPUState *cpu) { (void)pc; (void)cpu; } +void psx_segment_miss_record_kind(uint32_t a, uint32_t h, uint32_t r, uint32_t s, uint32_t f, uint32_t k) { (void)a; (void)h; (void)r; (void)s; (void)f; (void)k; } +uint64_t s_frame_count = 0; +int (*g_psx_bios_hle_hook)(struct CPUState *cpu, uint32_t phys) = 0; +uint64_t g_dispatch_static_hits = 0; +uint32_t g_debug_current_func_addr = 0; +uint64_t g_psx_bail_flattened = 0; +uint32_t g_psx_ram_mask = 0x1FFFFFu; +void psx_bios_hle_configure(uint32_t a, uint32_t b) { (void)a; (void)b; } +int autocompile_toolchain_available(void) { return 0; } +const char *overlay_loader_arch_abi(void) { return "test-arch"; } +static OverlayCallbacks s_cbs; +const OverlayCallbacks *overlay_loader_callbacks(void) { return &s_cbs; } + +/* One builtin, standing in for the linked OpenBIOS. */ +static const PsxBiosImageInfo s_builtin_image = { + 0, 0, 0, 0, 0, 524288u, 0x11111111u, 0u, "", "OPENBIOS", 1, +}; +static void builtin_dispatch(struct CPUState *c, uint32_t a) { (void)c; (void)a; } +static void builtin_dispatch_call(struct CPUState *c, uint32_t a, uint32_t r) { (void)c; (void)a; (void)r; } +static const PsxBiosBackend s_builtin = { + &s_builtin_image, builtin_dispatch, builtin_dispatch_call, 0, 0, 0, 0, +}; +const PsxBiosBackend *const psx_bios_builtin_registry[] = { &s_builtin }; +const uint32_t psx_bios_builtin_registry_count = 1u; + +static int s_fail; +#define CHECK(cond, ...) do { if (!(cond)) { s_fail++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0) + +#ifndef FAKE_GOOD +#error "FAKE_GOOD / FAKE_BAD_ABI / FAKE_BAD_HASH must name the fake module paths" +#endif + +int main(void) { + char err[256]; + const PsxBiosBackend *b; + + /* 4. Seeded before main() by the constructor. */ + CHECK(psx_bios_registry_count == 1, "registry seeded count %u", (unsigned)psx_bios_registry_count); + CHECK(psx_bios_registry[0] == &s_builtin, "registry[0] is the builtin"); + CHECK(psx_bios_bundled() == &s_builtin, "bundled() finds the builtin"); + CHECK(!psx_bios_has_selectable(), "nothing selectable before a module loads"); + + /* 2. Wrong ABI tag: refused, registry untouched. */ + err[0] = 0; + b = psx_bios_module_load(FAKE_BAD_ABI, err, sizeof(err)); + CHECK(b == 0, "bad-ABI module must be refused"); + CHECK(strstr(err, "ABI tag") != 0, "bad-ABI reason names the tag: %s", err); + CHECK(psx_bios_registry_count == 1, "bad-ABI module not registered"); + + err[0] = 0; + b = psx_bios_module_load(FAKE_BAD_HASH, err, sizeof(err)); + CHECK(b == 0, "bad-hash module must be refused"); + CHECK(strstr(err, "codegen hash") != 0, "bad-hash reason names the hash: %s", err); + CHECK(psx_bios_registry_count == 1, "bad-hash module not registered"); + + /* 1. The good one loads, inits with the host tables, registers after the builtin. */ + err[0] = 0; + b = psx_bios_module_load(FAKE_GOOD, err, sizeof(err)); + CHECK(b != 0, "good module loads: %s", err); + if (b) { + CHECK(b->image && strcmp(b->image->image_id, "FAKE-0001") == 0, "descriptor identity"); + CHECK(b->image->image_crc32 == 0xDEADBEEFu, "descriptor crc"); + CHECK(psx_bios_registry_count == 2, "registered: count %u", (unsigned)psx_bios_registry_count); + CHECK(psx_bios_registry[1] == b, "registered after the builtin"); + CHECK(psx_bios_has_selectable(), "a non-bundled backend is now selectable"); + CHECK(psx_bios_find("FAKE-0001") == b, "find() by id"); + CHECK(psx_bios_activate(b) == 1, "activate"); + CHECK(psx_bios_active == b, "active pointer"); + CHECK(psx_bios_image.image_crc32 == 0xDEADBEEFu, "published image copied"); + /* The module saw the host's own depth counter (aliasing, not a copy). */ + { + PsxBiosModuleCallbacks probe; + psx_bios_module_fill_callbacks(&probe); + CHECK(probe.size == sizeof(probe), "table size stamped"); + CHECK(probe.dispatch_depth == &g_psx_dispatch_depth, "depth pointer aliases host"); + CHECK(probe.hle_hook == &g_psx_bios_hle_hook, "hle hook pointer aliases host"); + CHECK(probe.ram_mask == &g_psx_ram_mask, "ram mask pointer aliases host"); + } + } + + /* 3. Same file again: the descriptor is the same object, registered once. */ + { + const PsxBiosBackend *again = psx_bios_module_load(FAKE_GOOD, err, sizeof(err)); + CHECK(again == b, "second load returns the same descriptor"); + CHECK(psx_bios_registry_count == 2, "second load does not duplicate: %u", + (unsigned)psx_bios_registry_count); + } + + /* Cache path shape: one place formats it; assert the pieces are present. */ + { + char path[512]; + CHECK(psx_bios_module_cache_path("/x", "SCPH1001", 0x37157331u, path, sizeof(path)), + "cache path fits"); + CHECK(strstr(path, "/x/cache/bios/test-arch/bm1_") == path, "cache path root: %s", path); + CHECK(strstr(path, "/SCPH1001_37157331.") != 0, "cache path stem+crc: %s", path); + } + + /* Identity table: the shipped profiles' images are known; junk is not. */ + CHECK(psx_bios_module_known_id(0x37157331u, 524288u) && + strcmp(psx_bios_module_known_id(0x37157331u, 524288u), "SCPH-1001") == 0, + "SCPH-1001 known"); + CHECK(psx_bios_module_known_id(0x12345678u, 524288u) == 0, "unknown crc rejected"); + CHECK(psx_bios_module_known_id(0x37157331u, 1u) == 0, "wrong size rejected"); + + if (s_fail) { printf("%d failure(s)\n", s_fail); return 1; } + printf("bios module host: ok\n"); + return 0; +} diff --git a/runtime/tests/test_bios_selection_guards.py b/runtime/tests/test_bios_selection_guards.py index 57011b6e..4938814a 100644 --- a/runtime/tests/test_bios_selection_guards.py +++ b/runtime/tests/test_bios_selection_guards.py @@ -29,10 +29,27 @@ HOST = (ROOT / "host" / "psxrecomp_codegen_host.c").read_text(encoding="utf-8") assert "gi->has_bios = 1;" not in HOST assert MAIN.count("gi.has_bios = 1;") == 0 assert MAIN.count("g_lnch_can_regen = gi.prepare_with_progress != nullptr") == 2 -assert "gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen) ? 1 : 0;" in MAIN +assert MAIN.count("gi.has_bios = (psx_bios_has_selectable() || g_lnch_can_regen ||") == 2 assert "if (g_lnch_can_regen) {" in MAIN assert MAIN.index("if (g_lnch_can_regen) {") < MAIN.index( '"Generate & rebuild to switch (or use OpenBIOS)."') assert "cannot be selected. Clear the BIOS field to play." in MAIN +# Loadable BIOS backends (psx_bios_module.h): a bundled build with the +# toolchain beside it can BUILD a retail backend from the player's dump, so a +# player choice is meaningful there, discovery accepts a known dump, and the +# launcher verify reports the dump usable without offering Generate & rebuild. +assert "psx_bios_module_acquire(" in MAIN +assert "psx_bios_module_is_cached(" in MAIN +assert "psx_bios_has_selectable() != 0 || bios_module_supported_here();" in MAIN +assert MAIN.count("psx_bios_module_supported(") >= 4 +assert "Preparing your BIOS" in MAIN +# The registry is mutable and seeded from the builtins before main(). +BACKEND = (ROOT / "runtime" / "src" / "psx_bios_backend.c").read_text(encoding="utf-8") +assert "int psx_bios_register(const PsxBiosBackend *backend)" in BACKEND +assert "psx_bios_registry_ctor" in BACKEND +CMAKE = (ROOT / "runtime" / "runtime.cmake").read_text(encoding="utf-8") +assert "psx_bios_builtin_registry[]" in CMAKE +assert "psx_bios_module.c" in CMAKE + print("BIOS selection capability guards passed") diff --git a/tools/bios_module_build.py b/tools/bios_module_build.py new file mode 100644 index 00000000..1b1769dd --- /dev/null +++ b/tools/bios_module_build.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +"""bios_module_build.py — build a loadable BIOS backend from the player's own +retail dump, on the player's machine. + +WHY +=== +A bundled release links only the redistributable OpenBIOS backend: the C a +retail image recompiles to is a translation of that image's code and never +ships (docs/BIOS_SELECTION.md). This script is what the runtime runs, from the +overlay toolchain beside the executable, when the player selects a retail dump: +it recompiles THAT dump with the bundled psxrecomp-bios and compiles the result +into a shared library the runtime loads as one more backend +(runtime/src/psx_bios_module.c, runtime/include/psx_bios_module.h). The same +interpreter and compiler already turn overlays streamed from the player's disc +into native code; this is that mechanism applied to the BIOS. + +CONTRACT +======== +The module is a self-contained shared library (no imports from the executable +on any platform), exporting psx_bios_module_abi / _codegen_hash / _init / +_backend (see psx_bios_module_glue.c.inc). It is compiled with the same flags +the executable's own BIOS backend uses (no debug tools, block cycles, game +dispatch present) plus the two module defines, from THREE translation units: + + _full.c generated function bodies (+ psx_bios_module.h prepended) + _dispatch.c generated dispatcher/descriptor (+ psx_bios_module.h prepended) + glue.c overlay preamble + BIOS glue (the only exports) + +The identity gate is the emitter's own: a profile pins the image's SHA-256 and +psxrecomp-bios refuses any other image, so a module can only ever be built from +the exact dump its code describes. + +USAGE (as the runtime invokes it) + python3 bios_module_build.py --dump --toolchain \ + --stem SCPH1001 --out /SCPH1001_.so --flavor 0 \ + --compiler gcc|tcc [--gcc ] [--tcc ] + +The toolchain dir is overlay_toolchain/ as tools/release_stage.py stages it +(python, psxrecomp-bios, include/, bios/*.toml, recompiler/seeds/). Each +piece can be overridden (--emitter, --include, --profiles, --seeds) so a +developer tree can run it without staging. + +Prints `PSX_BIOS_MODULE_PUBLISHED ` on success; exit 0. Any failure +leaves no file at --out and exits non-zero with the reason on stdout. +""" + +from __future__ import annotations + +import argparse +import ctypes +import hashlib +import os +import re +import shutil +import subprocess +import sys +import tempfile +import zlib + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) + +import compile_overlays as co # noqa: E402 (shared compiler driver helpers) + + +def die(msg: str) -> "NoReturn": + print(f"bios_module_build: ERROR: {msg}") + sys.exit(1) + + +def read_profile(path: str) -> dict: + """The few keys this needs from a bios/.toml (simple `key = "value"` + lines inside [program] / [program.image] / [recompiler]). No TOML library: + the pinned interpreter is the stdlib-only embeddable Python.""" + out = {"rom": "", "sha256": "", "seeds": "", "out_stem": "", "id": ""} + section = "" + with open(path, encoding="utf-8", errors="replace") as f: + for line in f: + s = line.split("#", 1)[0].strip() + if not s: + continue + m = re.match(r"^\[([^\]]+)\]$", s) + if m: + section = m.group(1).strip() + continue + m = re.match(r'^([A-Za-z_][A-Za-z0-9_]*)\s*=\s*"([^"]*)"', s) + if not m: + continue + k, v = m.group(1), m.group(2) + if section == "program" and k == "rom": + out["rom"] = v + elif section == "program" and k == "id": + out["id"] = v + elif section == "program.image" and k == "sha256": + out["sha256"] = v.lower() + elif section == "recompiler" and k == "seeds": + out["seeds"] = v + elif section == "recompiler" and k == "out_stem": + out["out_stem"] = v + return out + + +def identity(path: str) -> tuple[int, int, str]: + with open(path, "rb") as f: + data = f.read() + return len(data), zlib.crc32(data) & 0xFFFFFFFF, hashlib.sha256(data).hexdigest() + + +MODULE_DEFINES = [ + "-DPSX_OVERLAY_DLL_BUILD", + "-DPSX_BIOS_MODULE_BUILD", + # Mirror the executable's own BIOS objects (runtime.cmake): no debug tools, + # faithful block cycles, and the game-dispatch-aware dispatcher body. + "-DPSX_NO_DEBUG_TOOLS", + "-DPSX_ENABLE_BLOCK_CYCLES=1", + "-DPSX_HAS_GAME_DISPATCH", +] + + +def compile_module(sources: list[str], out: str, include_dirs: list[str], + stem: str, flavor: int, compiler: str, gcc: str, tcc: str) -> bool: + defines = MODULE_DEFINES + [f"-DPSX_OVERLAY_FLAVOR={int(flavor)}", + f"-DPSX_BIOS_MODULE_STEM={stem}"] + if int(flavor) & 2: + defines.append("-DPSX_PGXP=1") + if compiler == "tcc": + # Same treatment compile_overlays gives a shard: strip any BOM, add the + # __builtin_ctz shim, and hand tcc BOM-free copies of the headers. + for src in sources: + with open(src, "rb") as f: + data = f.read() + if data[:3] == b"\xef\xbb\xbf": + data = data[3:] + if not data.startswith(co._TCC_COMPAT_PREFIX): + data = co._TCC_COMPAT_PREFIX + data + with open(src, "wb") as f: + f.write(data) + cmd = [tcc, "-shared", *defines, *[co.native_path(s) for s in sources], + "-o", co.native_path(out)] + for d in include_dirs: + cmd.append("-I" + co.native_path(co._bom_free_incdir(d))) + env = None + else: + env, tc_dir = co._toolchain_env(gcc) + pic = [] if co.is_windows() else ["-fPIC"] + cmd = [gcc, "-shared", *pic, "-O2", *defines, + *[co.native_path(s) for s in sources], + "-o", co.native_path(out), + *[f"-I{co.native_path(d)}" for d in include_dirs], "-lm"] + if tc_dir is None: + print(" WARNING: no gcc toolchain dir found; compile may fail to launch") + print(" compile: " + " ".join(cmd)) + r = subprocess.run(cmd, capture_output=True, text=True, env=env) + if r.returncode != 0: + msg = (r.stderr or r.stdout or "").strip() or "(no compiler output)" + print(f" COMPILE ERROR (exit {r.returncode}):\n{msg}") + return False + return True + + +def self_check(path: str, flavor: int) -> bool: + """Load the fresh module in this interpreter: proves it is self-contained + (RTLD_NOW resolves every symbol or fails) and that its ABI tag names the + flavor asked for. The host repeats the gate with its own tag.""" + try: + lib = ctypes.CDLL(co.native_path(path)) + except OSError as exc: + print(f" self-check: module does not load: {exc}") + return False + try: + abi = ctypes.c_int(lib.psx_bios_module_abi()).value + h = ctypes.c_uint32(lib.psx_bios_module_codegen_hash()).value + except AttributeError as exc: + print(f" self-check: export missing: {exc}") + return False + if (abi >> 16) & 0xFFFF != int(flavor) & 0xFFFF: + print(f" self-check: ABI flavor {(abi >> 16) & 0xFFFF} != requested {flavor}") + return False + print(f" self-check: abi=0x{abi:08x} codegen_hash=0x{h:08x} ok") + return True + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--dump", required=True, help="the player's BIOS image") + ap.add_argument("--toolchain", default=HERE, help="overlay_toolchain/ dir (default: this script's dir)") + ap.add_argument("--stem", required=True, help="profile stem, e.g. SCPH1001 (bios/.toml)") + ap.add_argument("--out", required=True, help="module path to publish (.so/.dll)") + ap.add_argument("--flavor", type=int, default=0, help="PSX_OVERLAY_FLAVOR of the host build") + ap.add_argument("--compiler", choices=("gcc", "tcc"), default="gcc") + ap.add_argument("--gcc", default="gcc") + ap.add_argument("--tcc", default="tcc") + ap.add_argument("--emitter", default=None, help="psxrecomp-bios (default: /psxrecomp-bios[.exe])") + ap.add_argument("--include", default=None, help="runtime include dir (default: /include)") + ap.add_argument("--profiles", default=None, help="bios/ profiles dir (default: /bios)") + ap.add_argument("--seeds", default=None, help="seeds dir (default: /recompiler/seeds)") + ap.add_argument("--keep-work", action="store_true", help="keep the temporary emit tree (debugging)") + args = ap.parse_args(argv) + + tk = os.path.abspath(args.toolchain) + # Everything absolute: the emitter runs with cwd inside the staged work + # tree, and the compiler is handed paths from more than one directory. + emitter = os.path.abspath(args.emitter or os.path.join( + tk, "psxrecomp-bios.exe" if co.is_windows() else "psxrecomp-bios")) + include = os.path.abspath(args.include or os.path.join(tk, "include")) + profiles = os.path.abspath(args.profiles or os.path.join(tk, "bios")) + seeds_dir = os.path.abspath(args.seeds or os.path.join(tk, "recompiler", "seeds")) + args.dump = os.path.abspath(args.dump) + args.out = os.path.abspath(args.out) + for label, p in (("emitter", emitter), ("include dir", include), ("profiles dir", profiles), + ("seeds dir", seeds_dir), ("dump", args.dump)): + if not os.path.exists(p): + die(f"{label} not found: {p}") + if not re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", args.stem): + die(f"bad stem {args.stem!r}") + profile_path = os.path.join(profiles, f"{args.stem}.toml") + if not os.path.isfile(profile_path): + die(f"no profile for stem {args.stem}: {profile_path}") + prof = read_profile(profile_path) + if not prof["rom"] or not prof["seeds"]: + die(f"profile {profile_path} lacks rom/seeds") + out_stem = prof["out_stem"] or args.stem + if out_stem != args.stem: + die(f"profile out_stem {out_stem!r} != --stem {args.stem!r}; the descriptor symbol must match") + + size, crc, sha = identity(args.dump) + print(f"dump: {args.dump} ({size} bytes, CRC32 {crc:08X}, SHA-256 {sha[:16]}…)") + if size != 512 * 1024: + die(f"a PlayStation BIOS image is exactly 524288 bytes (got {size})") + if prof["sha256"] and prof["sha256"] != sha: + die(f"this image is not {prof['id'] or args.stem}: profile pins SHA-256 " + f"{prof['sha256'][:16]}…, dump is {sha[:16]}…") + + seeds_src = os.path.join(seeds_dir, os.path.basename(prof["seeds"])) + if not os.path.isfile(seeds_src): + die(f"seeds file the profile names is not shipped: {seeds_src}") + + # Stage an emitter project root: the profile's rom/seeds paths are relative + # to the directory holding a .gitignore/.git/CMakeLists.txt marker + # (config_loader find_project_root), so recreate exactly that shape. + work = tempfile.mkdtemp(prefix="psx_bios_module_") + try: + with open(os.path.join(work, ".gitignore"), "w") as f: + f.write("*\n") + rom_dst = os.path.join(work, prof["rom"]) + os.makedirs(os.path.dirname(rom_dst), exist_ok=True) + shutil.copy2(args.dump, rom_dst) + seeds_dst = os.path.join(work, prof["seeds"]) + os.makedirs(os.path.dirname(seeds_dst), exist_ok=True) + shutil.copy2(seeds_src, seeds_dst) + prof_dst = os.path.join(work, "bios", f"{args.stem}.toml") + os.makedirs(os.path.dirname(prof_dst), exist_ok=True) + shutil.copy2(profile_path, prof_dst) + gen = os.path.join(work, "out") + os.makedirs(gen, exist_ok=True) + + env = os.environ.copy() + env.setdefault("PSX_CPS", "1") + cmd = [emitter, "--config", os.path.join("bios", f"{args.stem}.toml"), "--out-dir", gen] + print(" emit: " + " ".join(cmd)) + r = subprocess.run(cmd, cwd=work, capture_output=True, text=True, env=env) + if r.returncode != 0: + tail = "\n".join((r.stderr or r.stdout or "").splitlines()[-25:]) + die(f"psxrecomp-bios exited {r.returncode}:\n{tail}") + full_c = os.path.join(gen, f"{args.stem}_full.c") + disp_c = os.path.join(gen, f"{args.stem}_dispatch.c") + for p in (full_c, disp_c): + if not os.path.isfile(p): + die(f"emitter produced no {p}") + with open(disp_c, encoding="utf-8", errors="replace") as f: + if f"{args.stem}_psx_bios_backend" not in f.read(): + die("emitted dispatch defines no backend descriptor (emitter too old)") + + # Module translation units: the name mappings go before the first + # #include so psx_memory.h's g_psx_ram_mask is rewritten too. + prelude = '#include "psx_bios_module.h"\n' + for p in (full_c, disp_c): + with open(p, "rb") as f: + data = f.read() + if data[:3] == b"\xef\xbb\xbf": + data = data[3:] + with open(p, "wb") as f: + f.write(prelude.encode("utf-8") + data) + glue_c = os.path.join(gen, "glue.c") + with open(glue_c, "w", encoding="utf-8") as f: + f.write(prelude) + f.write('#include "overlay_dispatch_preamble.c.inc"\n') + f.write('#include "psx_bios_module_glue.c.inc"\n') + + out_dir = os.path.dirname(os.path.abspath(args.out)) or "." + os.makedirs(out_dir, exist_ok=True) + tmp_out = os.path.join(out_dir, "." + os.path.basename(args.out) + ".tmp") + if os.path.exists(tmp_out): + os.remove(tmp_out) + include_dirs = [include] + fmt_inc = os.path.join(tk, "lib", "fmt", "include") # dev-tree layout only + if os.path.isdir(fmt_inc): + include_dirs.append(fmt_inc) + if not compile_module([full_c, disp_c, glue_c], tmp_out, include_dirs, + args.stem, args.flavor, args.compiler, args.gcc, args.tcc): + if os.path.exists(tmp_out): + os.remove(tmp_out) + die("compile failed") + if not self_check(tmp_out, args.flavor): + os.remove(tmp_out) + die("self-check failed") + os.replace(tmp_out, args.out) + print(f"PSX_BIOS_MODULE_PUBLISHED {args.out}") + return 0 + finally: + if args.keep_work: + print(f" work tree kept: {work}") + else: + shutil.rmtree(work, ignore_errors=True) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/release_stage.py b/tools/release_stage.py index 89bc9d5c..47ece446 100644 --- a/tools/release_stage.py +++ b/tools/release_stage.py @@ -619,6 +619,8 @@ TOOLCHAIN_PY_REL = {'win': os.path.join('python', 'python.exe'), 'macos-arm64': os.path.join('python', 'bin', 'python3')} TOOLCHAIN_RECOMPILER = {'win': 'psxrecomp-game.exe', 'linux': 'psxrecomp-game', 'macos-x64': 'psxrecomp-game', 'macos-arm64': 'psxrecomp-game'} +TOOLCHAIN_BIOS_EMITTER = {'win': 'psxrecomp-bios.exe', 'linux': 'psxrecomp-bios', + 'macos-x64': 'psxrecomp-bios', 'macos-arm64': 'psxrecomp-bios'} def stage_toolchain(stage, recomp_dir, recomp_tools, recomp_include, dl_cache, @@ -691,6 +693,18 @@ def stage_toolchain(stage, recomp_dir, recomp_tools, recomp_include, dl_cache, % (copied, dll_src, '' if copied else ' -- assuming a static recompiler')) shutil.copy2(os.path.join(recomp_tools, 'compile_overlays.py'), toolchain) + # The BIOS module builder (docs/BIOS_SELECTION.md, "player-side backend + # build"): turns the player's own retail dump into a loadable backend with + # the same interpreter and compiler the overlay path uses. It needs the + # BIOS emitter, every shipped profile, and the seeds the profiles name. + shutil.copy2(os.path.join(recomp_tools, 'bios_module_build.py'), toolchain) + bios_emitter = TOOLCHAIN_BIOS_EMITTER[platform_tag] + src_bios_emitter = os.path.join(recomp_dir, bios_emitter) + if not os.path.isfile(src_bios_emitter): + _die('overlay toolchain needs the BIOS emitter at %s; build the ' + 'psxrecomp-bios target first' % src_bios_emitter) + shutil.copy2(src_bios_emitter, os.path.join(toolchain, bios_emitter)) + os.chmod(os.path.join(toolchain, bios_emitter), 0o755) tool_inc = _mkdirs(os.path.join(toolchain, 'include')) for h in os.listdir(recomp_include): if h.endswith(('.h', '.c.inc')): @@ -698,10 +712,16 @@ def stage_toolchain(stage, recomp_dir, recomp_tools, recomp_include, dl_cache, recomp_root = os.path.abspath(os.path.join(recomp_include, '..', '..')) bios_src = os.path.join(recomp_root, 'bios') bios_dest = _mkdirs(os.path.join(toolchain, 'bios')) - for profile in ('SCPH1001.toml', 'OpenBIOS.toml'): - src = os.path.join(bios_src, profile) - if os.path.isfile(src): - shutil.copy2(src, bios_dest) + # Profiles only (TOML): never an image. A retail .BIN beside them in a + # developer's bios/ must not ride along, so the copy is by extension. + for profile in sorted(os.listdir(bios_src)): + if profile.endswith('.toml'): + shutil.copy2(os.path.join(bios_src, profile), bios_dest) + seeds_src = os.path.join(recomp_root, 'recompiler', 'seeds') + seeds_dest = _mkdirs(os.path.join(toolchain, 'recompiler', 'seeds')) + for seed in sorted(os.listdir(seeds_src)): + if seed.endswith('.json'): + shutil.copy2(os.path.join(seeds_src, seed), seeds_dest) # The runtime gates autocompile on this exact file. If the layout ever # changes, fail here rather than shipping a toolchain the runtime ignores. @@ -713,7 +733,11 @@ def stage_toolchain(stage, recomp_dir, recomp_tools, recomp_include, dl_cache, % os.path.relpath(probe, toolchain)) os.chmod(probe, 0o755) for required in (os.path.join('include', 'overlay_dispatch_preamble.c.inc'), - os.path.join('bios', 'SCPH1001.toml')): + os.path.join('include', 'psx_bios_module_glue.c.inc'), + os.path.join('include', 'psx_bios_module.h'), + 'bios_module_build.py', + os.path.join('bios', 'SCPH1001.toml'), + os.path.join('recompiler', 'seeds', 'phase2_ghidra_seeds.json')): path = os.path.join(toolchain, required) if not os.path.isfile(path): _die('staged overlay_toolchain is missing %s; bundled shard ' @@ -737,7 +761,7 @@ def stage_toolchain(stage, recomp_dir, recomp_tools, recomp_include, dl_cache, links += 1 continue total += st.st_size - log('Bundled overlay toolchain (pinned python%s + recompiler + headers): ' + log('Bundled overlay toolchain (pinned python%s + recompiler + BIOS emitter + headers): ' '~%d MB in %d file(s) + %d symlink(s)' % (' + tcc' if pins['tcc_url'] else '', total // (1 << 20), sum(len(f) for _r, _d, f in os.walk(toolchain)) - links, links))