An x86_64 macOS runtime under Rosetta compiled every overlay shard for
arm64. The runtime's shell and the bundled Python inherit x86_64, so
compile_overlays.py named the cache macos-x64, but it ran the compiler
without -arch and Xcode's clang is an arm64-only binary that defaults to
arm64. The arm64 shards landed in the macos-x64 cache, the runtime could
not load any of them, and every overlay ran interpreted. The BIOS module
builder had the same compile line.
- overlay_loader.h now defines PSX_OVERLAY_ARCH_ABI (moved from
overlay_loader.c), and autocompile.c exports it to every compile it
spawns, next to PSX_OVERLAY_FLAVOR. It is the running slice of a
universal binary.
- compile_overlays.py adopts it (env wins over the new --arch-abi), names
the cache with it, and passes -arch on macOS. Without a pin it builds
for its own interpreter, still with an explicit -arch.
- Shards are validated by loading them into the Python that built them,
so a Python that cannot run as the runtime's architecture now refuses
up front instead of writing shards nothing loads. The Python's
architecture is its process's: platform.machine() on macOS and Linux,
the interpreter's build (sysconfig.get_platform()) on Windows, where
platform.machine() reports the machine. So an x64 Python emulated on
Windows on ARM still builds the x64 runtime's shards.
- psx_bios_module.c passes --arch-abi to bios_module_build.py, which
applies the same rules.
- Windows and Linux compile lines are unchanged, and so are their cache
directories, except that the x64 build on Windows on ARM now uses the
runtime's win-x64 instead of the machine's win-arm64.
overlay_compile_arch_test (CTest) pins the handoff, the flags, the refusal,
the Windows process architecture and the unchanged Windows/Linux commands;
with a real cc it builds a shard for each slice the Mac can run and loads
it in a process of that architecture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bundled releases link OpenBIOS only; retail-BIOS-derived C never ships.
"Bring your own BIOS" now works without it: when a dump matching a shipped
profile (psx_bios_known_images.h) is selected, the runtime builds that
backend on the player's machine with the overlay toolchain the release
already carries, caches it, and loads it as one more registered backend.
- runtime/include/psx_bios_module.h: the module contract. An overlay-style
self-contained shared library (no imports from the executable) exporting
psx_bios_module_{abi,codegen_hash,init,backend}; reaches the runtime only
through OverlayCallbacks plus a new PsxBiosModuleCallbacks table (traps,
dirty-RAM/kernel-bless probes, fntrace, log_probe) and pointer aliases of
the globals the generated dispatcher touches. Its #defines rewrite the
emitter's own extern lines, so the generated C compiles unchanged.
- runtime/include/psx_bios_module_glue.c.inc: module-side forwarders and the
four exports, compiled after overlay_dispatch_preamble.c.inc.
- runtime/src/psx_bios_module.c: host side. Cache path
<exe>/cache/bios/<os-arch>/bm<abi>_<codegen hash>_f<flavor>/<STEM>_<crc>,
ABI + codegen-hash gated load, callback fill, and a blocking build via
overlay_toolchain/bios_module_build.py (gcc via compiler.txt or PATH,
else tcc), then psx_bios_register().
- psx_bios_backend.c: the registry is now a mutable table seeded from the
CMake-generated psx_bios_builtin_registry by a constructor, with
psx_bios_register(); every existing walker keeps working.
- main.cpp: validate_bios_for_launch acquires a module (announcing a first
build) before declaring a mismatch; discovery accepts a buildable dump;
the launcher verify reports it usable ("ready" / "compiled on first
launch") with no Generate & rebuild; the BIOS row shows whenever the
toolchain is present; bundled_only respects module support.
- emitter: the CPS-marker constructor is not emitted in a module build.
- overlay_api.h: OverlayCallbacks gains a struct tag;
overlay_loader_callbacks() exposes the host table before loader init.
- tools/bios_module_build.py (ships in the toolchain): identity + profile
SHA-256 gate, staged emitter tree, emit, three-TU compile, ctypes
self-check, atomic publish. release_stage.py stages psxrecomp-bios, every
bios/*.toml, the seeds and the builder into overlay_toolchain/.
- Tests: runtime/tests/test_bios_module_host.c against three fake modules
(good / wrong ABI / wrong hash) on real dlopen; recompiler/tests/
test_bios_module_build.py builds the bundled OpenBIOS end to end and reads
the descriptor back; guards extended.
Verified on Twisted Metal 4 (bundled build + staged toolchain, headless):
first launch with --bios SCPH1001.BIN builds and loads the module and boots
on image=SCPH-1001; the second launch loads it from cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/SEGMENT_AWARE_CODE.md §5.7 (decision 4). #417 made every KUSEG and
KSEG1 overlay dispatch fall to the interpreter, because every shard was
compiled for KSEG0. The loader now keys shards by segment, so a shard
compiled for KUSEG or KSEG1 runs for PCs of that segment.
Cache layout. KSEG0 shards keep their directory and {phys}_{crc} names, so
no existing cache is invalidated. KUSEG and KSEG1 shards live in the cache
tag's seg-kuseg/ and seg-kseg1/ subdirectories, under the same names; both
tiers scan them, and a shard's segment is its directory's
(cache_seg_from_path).
Manifest. A KUSEG or KSEG1 manifest starts with `S <segment>` (00000000 or
A0000000) before its first F. F entries are full VAs in that segment; the
parser no longer forces them to KSEG0. R records stay KSEG0-spelled byte
extents. Without S a manifest is KSEG0, as before. Refused, when indexing,
when loading and through live publication alike:
- a manifest whose segment differs from its directory's;
- an F entry outside its segment;
- S after an F, S twice, or S that is not a RAM segment base.
Dispatch. Each candidate records its segment. The physical indexes stay (the
bytes are one identity), and every lookup matches the PC's segment:
- the entry chain (idx_head_seg);
- CPS continuation ownership;
- the lazy manifest index and try_load_region;
- the interpreter's native-call contract.
The lazy-miss and range-owner memos are keyed by the full PC. A PC in a
segment that maps no RAM never runs a shard. Exports are resolved by their
full VA.
Counters. segment_alias_interp (from #417) now counts KUSEG/KSEG1 dispatches
the loader leaves to the interpreter because no shard of their segment is
cached or valid yet. The new segment_native counts those a shard of their
own segment runs. Both are in the TCP overlay_loader_status reply. A warm
cache takes the first to 0.
Full-VA compares:
- the overlay idle note's return test compares the resume PC with $ra
exactly, and counts it as internal only for a running shard of its segment;
- the shadow diff's two "returned to the caller" tests compare the exact PC;
- the diff mode's kernel-window exclusion tests the physical address, so
KSEG0 and KUSEG kernel candidates are treated alike. Before, it tested the
full PC, so only a KUSEG PC ever matched, and #417's gate never let one reach
it. This is opt-in (PSX_OVERLAY_DIFF).
Tests (loader harness, overlay_segment_gate):
- a KUSEG shard runs natively for its KUSEG PCs, entry and continuation,
while KSEG0/KSEG1 aliases are interpreted;
- one shard per segment for the same word, preloaded and lazy;
- a stale own-segment shard is interpreted rather than handed to another
segment's;
- a KUSEG continuation where KSEG0 has an entry at that word, and a lazy one
whose competing KSEG0 bundle is newer;
- refused manifests, including through live publication;
- the native-call contract;
- a source check that the interpreter records the segment from the full PC,
and that no site clears dispatch evidence without its segment bits.
The harness gains a segment-run scenario that drives real compiled shards
through a CPS dispatch loop. overlay_init_guard and overlay_retry_c11 follow
the per-segment lookup, and so do the disabled interpreter_perf_guards'
negative-miss and try_load_region checks (it still stops where it did on D).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The watchdog leaves a runaway pass by longjmp from inside guest code, which
skips the exits of every frame it leaves: generated functions' bb_defer
cleanup, overlay_loader_call_native's unit depth, the native-shard stack and
cycle-flush hook, the DMA kick's exec depth, and the dirty-RAM interpreter's
active/phase/precise flags, resume latch and pending load. The checkpoint
now saves all of it and the restore puts it back (bb_defer with the cycle
freeze), as the scheduler and exception landings do for theirs. An overlay
shard's unpublished pass cycles are flushed into the frozen clock at the
landing so they never reach the live one.
PSX_RENDER_PASS_VERIFY also checks that a pass which returned normally left
the nesting balanced, since the restore would hide that. render_pass_stats
gains nesting_repairs and the fault log names what the landing undid.
render_pass_abort_test aborts a pass from five nested frames (real
render_pass.c and psx_cycles.c, mock devices and presenter) and checks every
value, the clock and RAM; it fails without this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlay shards are compiled at the KSEG0 address of their bytes: captures
record 0x80000000|offset and the manifest parser keys entries there. The
emitter bakes that segment into every PC a shard produces: the $ra a
jal/jalr writes, jump and exception-resume PCs, and I-cache fetch tags.
Dispatch matched candidates by the segment-stripped address, so a KUSEG or
KSEG1 alias of the same bytes ran the KSEG0 body at the wrong
architectural PC.
Found in R4 (SLUS-00797) with OpenBIOS: the exception handler enters its
RAM patch slots at KUSEG (0x0000281C, 0x0000357C). The kernel-page shard
linked $ra=0x8000282C where the interpreter (and hardware) gives
0x0000282C, and fetched 0x8000281C/0x8000357C, missing lines the kernel
had just filled with KUSEG tags: +4 cycles per slot, so the next VBlank
IRQ landed at a different instruction and warm (native) vs cold
(interpreted) overlay-cache runs saved different registers from frame
194 on. A KSEG1 entry would also be charged as cached.
PSX_OVERLAY_CODE_SEGMENT (overlay_loader.h) names the rule for capture,
the manifest parser, overlay_loader_dispatch and the generated static
overlay table: a PC outside it falls to the interpreter, which runs at
the PC it was handed. CPS continuations already failed closed this way
(their entry switch only knows KSEG0 PCs); function entries now match.
overlay_loader_status reports segment_alias_interp.
Static game code has the same shape for KUSEG-linked EXEs (the parser
normalizes their headers to KSEG0); that needs segment-relative emission
and is not changed here. Shard cache tags are unchanged.
Tests: test_overlay_segment_gate.py drives the real loader (pair-dedup
harness, which also gains the RAM-geometry globals it no longer linked
without) with a KSEG0 fixture shard: KUSEG and KSEG1 dispatches stay
interpreted and never call it, KSEG0 still runs native. It fails with the
gate removed. test_compile_overlays_static_split.py covers the generated
static table and rejects a non-KSEG0 entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
it stops checking the entire PSX disc for full hash check every time the launcher runs, assuming the user's hash checked it when selecting the disc the first time and saving it to settings, so it just validates the file name and size instead, then it shares lifecycle with the launcher and the game to load the game faster, and several optimizations were made to loadstate paths which I think in the end relied on reanchoring host timings to the restored clock that came with the state
The tier mis-compiled and has been hard-gated off since 2026-06-25; overlay
gaps fall to gcc/tcc shards and the dirty-RAM interpreter, unchanged.
Re-do on master of the removal validated on the stale branch (6da8c02),
reconciled with everything master grew since the fork:
- Delete overlay_sljit.{c,h}, overlay_compile_worker.{c,h}, lib/sljit,
runtime/tests/sljit_*.c, and the runtime.cmake sources/include-dirs.
- Strip the sljit provider/backend from code_provider.{c,h},
overlay_backend.{c,h} ("sljit" in overlay_backend/PSX_OVERLAY_BACKEND now
degrades to auto so stale configs still boot), main.cpp, debug_server.c
(sljit_status/sljit_async/sljit_try TCP commands), overlay_loader.{c,h}
(live policy, JIT-on-miss, persisted-shard cache, async worker glue,
psx_sljit_call/cop2/memx host helpers — reachable only via the removed
cpu->sljit_helpers table), and cpu_state.h (SLJIT_HLP_* enum + the
sljit_helpers[] member; CPUState layout change reshards via the cg tag).
- Master-only entanglement resolved before deletion: 28165d6's Einhander WS
player-X bound emit existed in three paths (gcc code_generator, sljit,
dirty_ram_interp). The gcc shard link contract now forwards
psx_ws_player_x_bound (ABI v13, previous commit), so deleting the sljit
emit path loses no widescreen behavior; ffc6688's game-dispatch block
inside the dead psx_sljit_call helper goes with it.
- Reword comments that described sljit as a live backend (gpu.{c,h},
ws_*_detect.h, cosim_state.c, dirty_ram_interp.c, autocompile.h,
boot_state.h, config_loader.h, compile_overlays.py, and the CPS marker
comments the recompiler emits).
recompiler psxrecomp-game builds clean; game runtimes rebuilt + boot-verified
separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In-progress investigation work snapshotted before rebasing onto the
tcc-refactored psxrecomp master. Includes overlay ABI v8
(check_interrupts_at resume-PC callback for async RFE recovery) and
code_generator/full_function_emitter delay-slot cycle accounting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dispatch path re-CRC32'd a native overlay function's ENTIRE code body on
every single call (cand_crc, unconditional) before running it -- a real tax on
hot per-frame code that blunted the native win. The page-generation counter that
could make this a cheap skip existed but was demoted to a diagnostic.
Promote it: a watched-page write is the ONLY way an entry's code can change, and
every CPU/DMA store funnels through the store chokepoint which bumps
overlay_page_gen for watched pages. So if the page-generation sum is unchanged
since this entry was last confirmed VALID, the bytes are provably unchanged ->
run native WITHOUT re-hashing. Fall back to the full crc32 only when the gen
moved or the entry isn't known-valid (this also re-confirms reload-on-return,
since write-away-then-back bumps the gen). Applied to both the main candidate
chain and the CPS continuation re-entry path. Correctness identical to the old
unconditional hash; only redundant hashing is cut. New gen_fastpath counter
exposed in the overlay status.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root-cause class: a toolchain-less machine never self-improved via sljit on the
normal play path. Two blocks: (1) `auto` resolved to gcc on EVERY machine because
the discriminator was autocompile_configured() — "is a command STRING set" — and
the shipped game.toml always carries overlay_autocompile_cmd; (2) live sljit mode
was reachable only via a hand-set env var (dead public setter, no policy).
Fix:
- autocompile.c: autocompile_toolchain_available() probes PATH for a real C
compiler (gcc/cc/clang). main.cpp passes
`autocompile_configured() && autocompile_toolchain_available()` to
code_provider_init, so `auto` picks gcc only when one is actually reachable,
else sljit.
- overlay_loader.c: overlay_loader_apply_live_policy() (called from main.cpp
AFTER backend resolution) defaults live ON when the resolved backend is sljit
— so a toolchain-less box runs VALIDATED shards live (self-improves on the
normal path); PSX_OVERLAY_SLJIT_LIVE still overrides. The env-only block in
overlay_loader_init is removed (the policy fn subsumes it).
Runtime-only (no regen). Refs SLJIT.md §11 Phase C.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Runtime detects an uncovered overlay/kernel variant by sustained dirty-RAM
interp pressure inside the capture windows (g_dirty_window_dispatches,
sampled every 600 vblanks; fires only at a coherent non-loading moment via
cdrom_load_in_progress, 60s cooldown, 16/session cap), writes the capture
JSON, and spawns the game.toml-configured compile command
([runtime] overlay_autocompile_cmd, cmd.exe /C, cwd=project root, child
output kept in an in-memory ring — no log files). On child exit 0 the emu
thread rescans the cache (overlay_loader_rescan: idempotent dir re-scan +
checked-regions reset) so the new DLL loads on the next dispatch and the
variant goes native in-session. Dedup is tool-side (existing-CRC skip);
pressure drops to zero once code is native, so the loop self-limits.
New TCP: autocompile_status / autocompile_run / overlay_rescan.
compile_dll now uses absolute native paths and reports exit codes.
VERIFIED live (user-played route): 2 auto-triggers fired, captures written,
child ran recompiler+audit (new variants 1E7DDE2C, A62D0A7C discovered and
audit-clean), rescan picked the compiled DLL up in-session. OPEN ISSUE: gcc
exits 1 with no output inside the spawned cmd.exe child (same absolute-path
command succeeds from an interactive shell) — compile step currently
completed manually; suspect child-env difference, to be isolated next
session.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Accumulated working-tree state for the overlay-JIT-cache phase, committed as a
checkpoint before the Bug A fix (next commit):
- event_ring.{c,h}: always-on unified event-timeline ring (IRQ deliver/gate/
raise, I_STAT changes, DMA kick/sched/done, ENQ/DEQ per source, exec-mode
tagged). Edge-recorded, eviction-bounded, survives a blue screen; dump via
event_ring_dump/_tail/_clear.
- cdrom.c: disc_speed="instant" no longer collapses every sector read to the
500cy floor (which flooded ~1100 DMA3 completions/frame and starved main
code). CDROM_INSTANT_PERIOD bounds sector IRQs to CDROM_INSTANT_MAX_PER_FRAME
(32)/frame. Known caveat: a transient ~1-2s reentry storm at boot frame ~210
still trips the watchdog once, then recovers.
- interrupts.c: claimed-IRQ cooldown is now per-source — SIO stays 0 (card
reads need back-to-back IRQs); DMA/VBLANK/timer get CLAIMED_PROGRESS_QUANTUM
(8) blocks of guaranteed main-code progress between deliveries. Root cause of
the instant-disc freeze: cooldown=0 let the block leader re-take the
exception before its body ran (pinned at the 0x8005A1D8 field loop).
- gpu/dma/memory/sio/timers/psx_cycles: event-ring instrumentation at raise/
kick/complete/schedule sites; g_event_step_conservative (single-cycle event
stepping A/B used to refute the event-granularity hypothesis for Bug A).
- overlay_capture.c, overlay_loader.h, crc32: capture/manifest plumbing for
per-entry validity (Inc3).
- recompiler: overlay-mode emission updates (code_generator, main_psx).
- tools: compile_overlays.py audit/patch updates; tomba_loadtest.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reload-on-return does not fire against the current merged-capture DLL. Added
code-span watch/hash (vs whole-region) + diagnostics, and the counters pinpoint
why:
r0_fn_lo=0x000E73C0 r0_fn_hi=0x0013B184 -> 343 KB function span
r0_writes_since_invalid=107222 -> below span/2 threshold (171 KB)
reval_attempts=0 -> never triggered
The DLL was compiled from the merged 1.1 MB roaming capture, so its functions
are scattered across the whole region with mutable data interleaved. There is no
coherent reload unit to threshold or hash: the re-validate trigger never fires,
and a 343 KB hash including data would never match anyway. OV-1 (correctness) is
unaffected — invalidation still fires (no blue screen); only the native speedup
on return is blocked.
Conclusion: reload-on-return requires coherent per-overlay segmentation (split
the merged region into real load units). The reload machinery stays in as inert
scaffolding until segmentation lands. Diagnostics exposed via
overlay_loader_status (r0_*, reval_attempts, reval_crc_miss, last_reval_crc).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two additions:
1. overlay_capture_dump TCP command — flush overlay_captures.json on demand,
so a freeze/kill can't lose a capture session (no clean window-close needed).
2. Reload-on-return: when an invalidated overlay region is loaded back with the
SAME content, re-enable its functions natively instead of staying interpreted.
- At register, hash the live region (crc_live).
- On invalidation, keep watching and count writes into the region (instead of
clearing the watch).
- When the region is substantially rewritten (>= size/2 bytes — a real reload,
not incidental data writes), the next dispatch re-hashes once; on a match to
crc_live, flip the region valid again. O(1) — dyntab entries never left, no
DLL reload. Cheap-gated so steady-state churn doesn't pay per-dispatch hashes.
- New counter: revalidations (in overlay_loader_status).
Ghidra note: searched the Tomba main EXE for the overlay load address (0x800E7000
virt / 0x000E7000 phys) — no flat overlay table; the address is computed and the
overlay is copied/decompressed to the code region by a BIOS routine. So overlay
identity must be a runtime content hash (this commit), not a static table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A registered overlay function is only callable while the RAM it was compiled
from is unchanged. Adds coarse, region-level write-invalidation:
- overlay_loader: each loaded DLL gets a region record {base,size,generation,
valid}; dyntab entries carry their region index. dyntab_lookup returns a
function only if its region is still valid (else counts stale_blocked and
falls through). Invalidation is O(1) (flip valid=0) — no open-addressing
deletion. DLL stays loaded; only callability is revoked
(Compiled != Registered != Callable).
- memory.c: a watched-overlay-pages bitmap, set when a region registers; one
bitmap test added to the RAM-store branch of psx_write_{word,half,byte}
(the single audited store chokepoint — all CPU + DMA writes funnel here).
A write into a watched page calls overlay_loader_invalidate_at(), which
flips the region invalid and stops watching it.
- Counters via overlay_loader_status: loads, invalidations, unregistered_funcs,
dispatch_native, dispatch_interp_fallback, stale_blocked, last_write_{pc,
addr,size}, regions.
Coarse by design: ANY write into a registered region invalidates the WHOLE
region (over-invalidation is safe — it falls back to the interpreter, which
self-heals). Per-function/dependency-range narrowing waits for the Inc2
manifest. After invalidation the region is not auto-reloaded this session
(re-registering a CRC-matched cache is also Inc2); the new content runs
interpreted, which is correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline
overlay compilation pipeline that were built on the pruned codex/cache-discovery
branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt
rather than history-restored.
Runtime (off by default, opt in via [runtime] overlay_cache = true):
- overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at
cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader
LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(),
enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter.
- overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and
writes overlay_captures.json (execution-verified seeds) at clean exit.
- dma.c: capture + cd_dma_log hook at CD DMA completion (async model).
- cdrom.c: cdrom_get_setloc_lba() for disc-position correlation.
- dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a
decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks.
- debug_server.c: cd_read_log + overlay_loader_status TCP commands.
- config_loader: [runtime] overlay_cache flag (default false).
Recompiler (compile_overlays.py / extract_overlays.py drive these):
- main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL
targets and set split_mid_function_targets=false, so overlay basic-block
targets are NOT split into bogus standalone functions (the mid-function-seed
failure mode that produced black-on-load / blue-on-new-game).
- function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the
mid-function split pre-pass behind split_mid_function_targets (default true,
so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs).
Deltas from the original: save states dropped (never worked); overlay enable is
runtime-config-gated (off by default) instead of a build-time define; loader
stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3).
Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shokunin