Fixes Ape Escape terrain seams: CPU-built subdivision midpoints and lwl/lwr-copied
grid corners drew native next to precise neighbours. Adds pgxp_tri_ring/pgxp_store_ring.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Preserve current segmented dispatch, active-plan mod hooks and software scale limits. Extend function filters through the active plan, retain the last-store pointer with overlay ABI 25, and keep default config fingerprints stable. Refresh BIOS emitter stamps after byte-identical regeneration and reconcile regression fixtures.
Bundled releases link OpenBIOS only; retail-BIOS-derived C never ships.
"Bring your own BIOS" now works without it: when a dump matching a shipped
profile (psx_bios_known_images.h) is selected, the runtime builds that
backend on the player's machine with the overlay toolchain the release
already carries, caches it, and loads it as one more registered backend.
- runtime/include/psx_bios_module.h: the module contract. An overlay-style
self-contained shared library (no imports from the executable) exporting
psx_bios_module_{abi,codegen_hash,init,backend}; reaches the runtime only
through OverlayCallbacks plus a new PsxBiosModuleCallbacks table (traps,
dirty-RAM/kernel-bless probes, fntrace, log_probe) and pointer aliases of
the globals the generated dispatcher touches. Its #defines rewrite the
emitter's own extern lines, so the generated C compiles unchanged.
- runtime/include/psx_bios_module_glue.c.inc: module-side forwarders and the
four exports, compiled after overlay_dispatch_preamble.c.inc.
- runtime/src/psx_bios_module.c: host side. Cache path
<exe>/cache/bios/<os-arch>/bm<abi>_<codegen hash>_f<flavor>/<STEM>_<crc>,
ABI + codegen-hash gated load, callback fill, and a blocking build via
overlay_toolchain/bios_module_build.py (gcc via compiler.txt or PATH,
else tcc), then psx_bios_register().
- psx_bios_backend.c: the registry is now a mutable table seeded from the
CMake-generated psx_bios_builtin_registry by a constructor, with
psx_bios_register(); every existing walker keeps working.
- main.cpp: validate_bios_for_launch acquires a module (announcing a first
build) before declaring a mismatch; discovery accepts a buildable dump;
the launcher verify reports it usable ("ready" / "compiled on first
launch") with no Generate & rebuild; the BIOS row shows whenever the
toolchain is present; bundled_only respects module support.
- emitter: the CPS-marker constructor is not emitted in a module build.
- overlay_api.h: OverlayCallbacks gains a struct tag;
overlay_loader_callbacks() exposes the host table before loader init.
- tools/bios_module_build.py (ships in the toolchain): identity + profile
SHA-256 gate, staged emitter tree, emit, three-TU compile, ctypes
self-check, atomic publish. release_stage.py stages psxrecomp-bios, every
bios/*.toml, the seeds and the builder into overlay_toolchain/.
- Tests: runtime/tests/test_bios_module_host.c against three fake modules
(good / wrong ABI / wrong hash) on real dlopen; recompiler/tests/
test_bios_module_build.py builds the bundled OpenBIOS end to end and reads
the descriptor back; guards extended.
Verified on Twisted Metal 4 (bundled build + staged toolchain, headless):
first launch with --bios SCPH1001.BIN builds and loads the module and boots
on image=SCPH-1001; the second launch loads it from cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The emitter writes g_debug_last_store_pc before every guest store. The
overlay preamble gave each DLL a private copy on the theory that only debug
tooling reads it. The runtime actually reads it in every build:
- memory.c psx_write_word_raw keys its RAM 0x0-0xF store filters on it;
- the fingerprint pc/mmio/sp columns use it, and so do wtrace,
record_frame, the MMIO/GP1 rings, parity provenance and the freeze
heartbeat.
So after a native overlay store the host still named an older store, and
those consumers saw an instruction the interpreter would not have.
OverlayCallbacks gains last_store_pc, which the loader always points at
the runtime's copy. In PSX_OVERLAY_DLL_BUILD, cpu_state.h maps the
emitted name onto that pointer, the same pattern as the bail-state
pointers. Generated code is unchanged. A NULL host pointer keeps the old
private copy.
The ABI goes to 24, so the loader rejects and regenerates older shards.
The preamble change already moves the codegen-hash cache path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The emitter emits a direct psx_game_text_native_ok() stale-static guard at block
entry (code_generator.cpp:115) and before every direct jal (:2402). That function
is DEFINED IN THE GENERATED DISPATCH SHARD (main_psx.cpp:1620), which links into
the executable. An overlay DLL is a separate module and cannot resolve it, and
overlay_dispatch_preamble.c.inc had no shim for it.
Found by bumping Tomba 2 from a 254-commit-old pin to master and regenerating its
shard cache against the identical vault:
before bump : built OK 2799, FAILED 1
after bump : built OK 2233, FAILED 109
107 of the 109 were the same error:
ld.exe: undefined reference to `psx_game_text_native_ok'
collect2.exe: error: ld returned 1 exit status
This is the fifth instance of one class, and overlay_api.h's own version history
records the previous four:
v9 faithful-timing callbacks overlay code failed to link
v10 gte_read_ctrl GTE-heavy overlays failed to LINK without shims,
silently blocking new coverage
v12 rfe_mark_escape without the shim any rfe-containing overlay
fails to LINK
v14 gte_precision_store_word the rfe / gte_read_ctrl class again
Every time the shape is identical: an emitter gains a runtime call, the overlay
ABI does not gain the shim, and the only symptom is shards that quietly fail to
build. The affected code then runs interpreted forever and nothing at runtime
says so. Severity is coverage, not correctness -- but that is exactly the defect
that let Ape Escape ship 100% interpreted without anyone noticing.
Fix follows the v10/v12/v14 pattern exactly:
- overlay_api.h: append game_text_native_ok to OverlayCallbacks, bump
PSX_OVERLAY_ABI_VERSION 21 -> 22, add the version-history note
- overlay_dispatch_preamble.c.inc: shim forwarding to g_cbs
- overlay_loader.c init_callbacks: host populates it from the dispatch shard
The NULL guard deliberately returns 0, not 1. Returning 0 routes the guest
through the dispatcher -- slower, never wrong. Returning 1 would let an overlay
run native code the guard was explicitly trying to reject.
Follow-up worth doing, tracked as beads-eio.3.97: five instances of one class
means the process is the bug, not the symbol. The emitter knows every helper name
it can emit and the preamble enumerates every shim; a test comparing those two
sets would have caught v9, v10, v12, v14 and this one at authoring time instead
of at some future title's link step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Emitter side of the value-propagation engine (ENHANCEMENTS.md G1.10):
- recompiler/src/pgxp_hook_emitter.{h,cpp} (new, in the codegen-hash
surface): ONE hook grammar shared by BOTH emitters, wrapping loads/
stores/COP2 transfers and precision-carrying ALU ops (moves, add/sub,
or-merge, shifts, LUI, HI/LO, muldiv) with PGXP_*() macros. Clobberable
sources are captured into locals before the statement. Precision-
destroying ops (AND/XOR/SLT/...) are deliberately unhooked - validate-
on-read drops their stale shadows for free.
- code_generator.cpp: central append in translate_instruction; LWC2/SWC2
emissions capture the RAW loaded/stored word (a masked GTE register
write must validate against the word as loaded).
- strict_translator.cpp: translate() wraps translate_impl()'s c_code with
the shared grammar; deferred load-delay variants hook the psx_ldd temp.
- BIOS regenerated, fingerprints verified CHANGED (SCPH1001 + OpenBIOS;
42.6k/12.2k macro sites). Base runtime + 33/33 tests green against it.
- Gating: macros expand only under -DPSX_PGXP=1. runtime.cmake grows a
PGXP option on psxrecomp_add_runtime_target (defines PSX_PGXP=1 +
PSX_OVERLAY_FLAVOR=2, _pgxp exe suffix) and PSX_PGXP_VARIANT to build
the sibling target from the same generated tree (framework runtime +
game wrapper). No ABI version bump needed: the appended PGXPHooks
table is NULL-tolerant and the flavor half of the ABI tag rejects any
host/DLL flavor mix.
- Shard caches: flavor folded into the cache tag (cg<V>_<hash>_gc<hash>
_f<flavor>) in overlay_loader.c + compile_overlays.py so base and pgxp
caches never share a directory; compile_overlays.py passes
-DPSX_PGXP=1 when flavor bit 2 is set; overlay preamble forwards
psx_pgxp_* through the table. PSX_OVERLAY_CODEGEN_VER 9 -> 10.
- Env overrides PSX_GEOMETRY_CORRECTION / PSX_PERSPECTIVE_TEXTURING /
PSX_PGXP_CPU_MODE (validation path: free-running headless boots arm
from the first vertex); census grew produced/swc2_stores and the pgxp
verb reports suppress/active gate state.
Validated: PSXRecomp_pgxp.exe boots the recompiled BIOS with hooks armed
(engine active, per-vertex lookups flowing; the BIOS shell has no RTPS
content so produced=0 there is correct - coverage measurement moves to a
3D title in phase 3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add opt-in [widescreen.cull] xclip_load_sites: a configured `lw rt,off(rs)`
loading a per-primitive X-reject bound (sltu-compared against ANDI-masked
u16 screen X) routes through psx_ws_xclip_bound() - INT32_MAX while the
margins are revealed (reject disabled; the wide-surface scissor clips the
overflow, wrapped off-left u16 coords pass), vanilla value at 4:3. Matched
native codegen emit and interpreter hook; overlay ABI v18 forwarder (NULL
preserves the vanilla loaded value); codegen ver 8. Parser, codegen and
overlay-mismatch regression coverage.
Split from #49: replaces the xclip_globals RAM poke with a recompile-time
instruction transform so native shards and the interpreter behave identically.
Add opt-in [widescreen.cull] plane_nx_sites: a configured `lw rt,off(rs)`
loading the X component of a side frustum-plane normal (sign-tested per
corner, dot = nx*px + nz*pz) routes through psx_ws_plane_nx() — scaled by
the inverse aspect factor (4*den)/(3*num) while the margins are revealed,
widening the plane cone by exactly atan((3*num)/(4*den)*tan(theta));
identity at 4:3. Matched native codegen emit and interpreter hook; overlay
ABI v17 forwarder (NULL preserves the vanilla normal); codegen ver 7.
Parser, codegen and overlay-mismatch regression coverage.
Split from #49: replaces the plane sign-test RAM pokes with a
recompile-time instruction transform so native shards and the interpreter
behave identically.
Add opt-in slti/sltiu far-bound scaling with matched native and interpreter semantics, correct immediate sign extension, and an overlay ABI forwarder for live aspect state.
Co-authored-by: Opok <novelasevilliousesp@gmail.com>
Add an opt-in horizontal low-edge transform for subu rd,zero,rt classifiers, with matching native and dirty-RAM interpreter behavior and guarded overlay variants.
Co-authored-by: Opok <novelasevilliousesp@gmail.com>
Two provably-correct framework fixes (independent of the AOT work) plus the
play-free overlay-extraction spike tooling and design doc.
Framework fixes (accurate, safe, complete):
- ABI v14: add gte_precision_store_word to OverlayCallbacks + preamble shim +
runtime population. The emitter emits a direct call for every swc2 (GTE
store-word); it is stateful runtime code (gte.cpp) so overlay DLLs must
forward or fail to LINK (undefined reference). Play-captured shards rarely
compiled a swc2; fuller static coverage surfaced the gap. Same pattern as
the v13 ws_player_x_bound / v10 gte_read_ctrl shims.
- Gate splitgen OFF in overlay mode (main_psx.cpp): the current recompiler's
unconditional split-TU output deletes <stem>_full.c and emits only shards,
which broke ALL overlay compilation (compile_overlays.py reads _full.c).
Overlays are single small TUs with no parallel-compile benefit; emit the
monolith. Restores overlay compile on the current tree.
AOT spike (docs/AOT_OVERLAY_PLAN.md + tools/aot_overlay_spike/, WIP):
- Play-free static overlay extraction from the disc image. Validated:
Tomba 1 reproduces 90% of played functions + ~1964 unplayed, live-verified
in-game (Dwarf Village runs native, zero runtime compile). Tomba 2 yields
661 functions byte-identical to the shipped/played vault (provably real),
all from the self-describing MAIN.EXE producer; the header-table BIN
producer needs loader-table RE (base recovery unreliable). Generality is
producer-dependent. Spike scripts are Tomba-specific proofs of concept
with hardcoded paths, not a finished tool.
- Correctness is guaranteed by the existing per-function code_crc dispatch
guard: a mis-positioned shard fails audit at compile or never fires at
runtime -> coverage loss, never wrong execution. Autocompile self-heals gaps.
CROSS-CUTTING: the ABI v14 change is part of the overlay codegen-hash source
set, so it ROLLS the cg tag (1f2ad026 -> e8484213). On master + a fw bump,
every title's overlay cache invalidates and needs regen+reshard. Branch only;
NOT pushed; other games' caches untouched. Needs user validation before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
28165d6 (Einhander native-wide player bounds) rewrites configured
[widescreen] signed_x_bound LUI sites into psx_ws_player_x_bound(vanilla)
calls in code_generator.cpp — and that site loop runs in overlay_mode too, so
the call lands in overlay-resident code. But the shard link contract never
forwarded the symbol: any shard containing a bound site fails to LINK
(undefined psx_ws_player_x_bound) and that variant runs interpreted forever —
the same silent class as v10 gte_read_ctrl and v12 psx_rfe_mark_escape (now
loud under the shard observability suite, but still a coverage hole).
The clamp computes against the host's live widescreen state (gpu.c), so a
local copy can't work: add the ws_player_x_bound callback (appended last),
the preamble forwarder (identity fallback => correct 4:3 on a host that
predates it), and the loader init wiring. Bump the ABI to 13 so stale DLL
mixes are rejected and regenerate.
This also unblocks removing the sljit tier: sljit's emit_ws_player_x_bound
was the only other native path carrying the clamp; with the gcc shard path
forwarding it (and dirty_ram_interp already handling the site), the feature
no longer depends on sljit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The emitter emits psx_rfe_mark_escape() at every rfe — including in
overlay-resident exception-context code — but the dispatch preamble
never provided it, so any rfe-containing overlay failed to LINK
(undefined reference) and ran interpreted forever: the gte_read_ctrl
silent-coverage-blocker class again (11 of Tomba's vault shards hit
it). It mutates shared RFE-pending escape state (interrupts.c), so the
shim forwards unconditionally through OverlayCallbacks; ABI bump to 12
rejects pre-shim caches.
Found live during LOAD_TIME_ZERO E0 bring-up: the user's first pig
load triggered autocapture and the compile died on this link error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The emitter routes mfc2/cfc2/mtc2/ctc2 on flag/IR/derived GTE registers
through direct gte_read_data/gte_read_ctrl/gte_write_data/gte_write_ctrl
calls (code_generator.cpp COP2 case), but the overlay DLL preamble had no
shims for them — any GTE-heavy overlay failed to LINK (undefined
gte_read_ctrl) and its region ran interpreted forever. That silently
blocked all new native coverage for the lava/Trolley attract-demo regions
after the caches rotated.
GTE state is shared runtime state, so the shims forward unconditionally
through OverlayCallbacks (a local copy would fork the flag/IR timeline).
ABI bumped 9 -> 10; the loader gate rejects+deletes pre-shim DLLs and
autocompile self-heals; plain gte_data[]/gte_ctrl[] accesses stay direct
CPUState reads in emitted code, matching the emitter.
Smoke-validated: previously-failing GTE-heavy captures now compile clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cycle-accuracy emitter emits calls to 12 timing runtime functions
(psx_cyc_load_*, psx_icache_fetch, psx_muldiv_*, psx_gte_*, psx_slice_block,
...) + debug_server_cyc_observe + g_debug_last_store_pc. Overlay DLLs never
got these symbols, so every overlay failed to link -> 0 native shards ->
100% interpreter (unplayable). Fix (option A): expose the 12 via the overlay
callback ABI.
- overlay_api.h: +12 timing callbacks in OverlayCallbacks, ABI 8 -> 9.
- overlay_loader.c: populate the 12; add warn_on_cgtag_mismatch() (loud
warning when shards exist under the wrong cg-hash).
- compile_overlays.py: 12 preamble forwarders + local g_debug_last_store_pc
+ -DPSX_NO_DEBUG_TOOLS.
- cosim.c/main.cpp: input-cosim setpad + held pad (deterministic input).
- cosim.py: env-overridable COSIM_EXE/CWD/GAME.
KNOWN REGRESSION (patched on a follow-up branch): Tomba pause menu (triangle)
wedges. Root-caused: an IRQ taken mid-block resumes at the function entry
leader instead of the interrupted PC, re-zeroing the menu's VSync wait
counter (scratchpad 0x1F8001E8) every frame so it never reaches its target.
Not the scheduler (LLE reproduces) and not timing math -- interrupt-resume
re-entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In-progress investigation work snapshotted before rebasing onto the
tcc-refactored psxrecomp master. Includes overlay ABI v8
(check_interrupts_at resume-PC callback for async RFE recovery) and
code_generator/full_function_emitter delay-slot cycle accounting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root-caused the Tomba2 native logo-stuck bug: it is a guest-cycle-timeline
divergence between the compiled-overlay backend and the dirty-RAM interpreter,
surfacing through a Timer1 (0x1F801110) debounce loop at game pc 0x8008592C that
reads 1 (native) vs 2 (interp), forking logo->FMV. The handoff's "jalr target
divergence" was a benign store-PC artifact (read-watch proves a0=0xB870 and
t3=[a0]=0x800896E0 are identical in both backends); the scratchpad blind spot
was ruled out (separate sp fingerprint forks at the same frame).
First-divergence observability (debug_server.c, memory.c):
- frame_fingerprint: added separate sp (scratchpad) cumulative hash column +
per-entry cyc, so a scratchpad or cycle fork is classified, not hidden.
- Unified ordered access recorder: one buffer, kind = ramw/spw/mmiow/mmior/ramr,
array index = true execution order, per-entry cyc (psx_get_cycle_count).
record_frame_dump emits the unified stream; record_reads_dump filters reads.
- Targeted main-RAM read-watch (PSX_READ_WATCH="lo,hi" + g_ram_read_watch_active
guard in psx_read_word) — proves a load's value/address per backend.
- PSX_RECORD_FRAME env seed: arm the recorder from boot (race-free), never
arm-then-time.
Fix A (cycle-model unification, part 1): compiled-overlay code was built WITHOUT
PSX_ENABLE_BLOCK_CYCLES, so every psx_advance_cycles() in overlay code was
compiled out -> a function run as a native overlay charged ~0 cycles while the
same function via the interp charged per-instruction (~32x), the entire drift.
- overlay_api.h: OverlayCallbacks gains advance_cycles; ABI v6->v7 (rejects
stale zero-cycle DLLs).
- compile_overlays.py: build overlays with -DPSX_ENABLE_BLOCK_CYCLES=1 and
emit a psx_advance_cycles shim forwarding to the shared host timeline.
- overlay_loader.c: wire s_callbacks.advance_cycles = psx_advance_cycles.
Result: the first debounce now reads Timer1=2 in native (was 1); overlay cycle
charge went 9.5k -> 380k (interp 306k). Native still logo-stuck: a residual,
perfectly linear ~1 cycle/write over-charge remains (block.instruction_count
counts branch + delay-slot as 2 while the interp charges the pair as 1 step),
which flips a later debounce 3 vs 2. Fix A pt2 (cycle-exact block totals) is the
next step. wt/tomba2 only; no pin bump, no master merge, no other-game regen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings the fail-closed native entry guard from wt/tomba2 (b5a6940 / bd60381) to
master, where it had never been merged — which is why Tomba (and any master-built
title) still wedged. This is the long-standing "blue screen" / shared-RAM-
divergence-between-interp-and-native class.
Root cause: an overlay function dispatched at a FOREIGN INTERIOR PC (a range-
ownership mismatch routed a PC the function does not own into it) hit the CPS
entry-switch `default: break`, which fell through and ran the function FROM ITS
TOP — re-running its prologue and corrupting shared CPU/RAM state, so the guest's
main loop then waits forever on a condition that never gets set (the 0xCF0 SIO
spin / blue screen). Reproduced on Tomba's Village of All Beginnings: a native
0x000E7000-region shard function entered at a foreign PC wedged; the interpreter
ran the same bytes correctly.
Fix: the generated CPS entry-switch for OVERLAY functions (config_.overlay_mode)
now FAILS CLOSED on an unknown non-zero continuation PC — it restores the PC and
calls psx_native_bad_entry instead of running from the top. overlay_loader_dispatch
consumes g_native_bad_entry and routes the PC to the sanctioned dirty-RAM
interpreter (the live bytes run correctly — NOT a stub/HLE). True-prologue entry
still runs from the top. Static (BIOS/boot-EXE) functions keep the legacy fall-
through. ABI bumped 5->6 so stale ABI-5 caches are rejected and rebuilt with the
guard.
Minimal port: kept only the fail-closed mechanism + its ABI plumbing. The Tomba2
host_aliases interior-routing and the native-block / CPS-probe diagnostics (which
depend on the overlay-cache-v2 multi-range Candidate fields not on master) were
intentionally omitted — the fail-closed default alone is the safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The manual PSX_OVERLAY_CODEGEN_VER must be hand-bumped on every emitter change;
when it isn't (e.g. between CPS emitter iterations), a stale-but-cgN DLL is reused
-> the v0.3.0 black screen. Now hash_codegen.cmake hashes the recompiler codegen
sources into a (gitignored) overlay_codegen_hash.h at build time; overlay_api.h
exposes PSX_OVERLAY_CODEGEN_HASH and the cache path becomes gcc/<arch-abi>/cg<N>_<hash>/.
The loader and compile_overlays.py read the same value, so any emitter change
auto-invalidates the cache (old dirs coexist, no stale load). Falls back to hash 0
when the header is absent (fresh checkout) — identical to today's single-cgN path.
Verified: build emits hash 0x347751c1; autocompile + loader both use cg4_347751c1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).
Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.
sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)
gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
inside a registered candidate's code range (a tail-transferred call's continuation),
overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.
Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.
VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).
KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.
Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
tail-transfer (jalr captures rs before the link, alias-safe); each call's
return point registered as a dispatchable continuation; entry-switch (keyed on
cpu->pc) at the top of every function AND every overlapping-alias body. 0
psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
(case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).
Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
(un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).
Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
[widescreen.cull] auto_backdrop — generic far-backdrop column preload for
16:9. A shared detector (runtime/include/ws_backdrop_detect.h) finds each
scrolling-backdrop column-window generator by its /96-magic + 0x176-camX +
sra,5 divide-tail invariant and records the window's START/END bound
instructions + width (window_cols). All four execution paths route those
bounds through psx_ws_backdrop_value(orig, is_end, window_cols), which widens
the camera-tracked window by exactly the 16:9 reveal (margin scales with
window_cols; identity at 4:3 so the build stays byte-identical with widescreen
off). gcc emit + sljit + dirty-RAM interp + overlay ABI v4 / codegen ver 3.
Also: CodeGenConfig::overlay_mode — explicit [widescreen.*] per-address site
shape checks now SKIP (apply only where bytes match) instead of hard-exit in
overlay mode, because overlay regions reuse the same address for different
code across scene variants. Main-EXE mode keeps the hard error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three coupled landings on top of the sljit merge:
1. Shared widescreen render-cull helper. auto_screen_x now emits a call to one
runtime helper psx_ws_cull_sltiu(sx,imm) (gpu.c) instead of an inline expr, and
the sljit JIT (overlay_sljit.c) + dirty-RAM interpreter (dirty_ram_interp.c)
route flagged render-funnel sltiu 0x140/0x141 through the SAME helper. So an
overlay's GTE render funnel widens identically whether it runs as a gcc DLL, a
sljit shard, or interpreted. Helper is byte-identical to vanilla at 4:3
(margin 0), so widescreen-off stays bit-exact. psx_ws_func_has_screen_cull
gates the widen to real render funnels (sltiu 0x140 + sltiu 0xE0 signature).
2. Versioned overlay cache path. gcc cache moves to <game_id>/gcc/<arch-abi>/cg<N>/
where N = PSX_OVERLAY_CODEGEN_VER (overlay_api.h), parsed identically by the
runtime (overlay_loader.c) and compile_overlays.py so they can't drift. A build
whose emitter output changed (e.g. this auto_screen_x change) reads+writes a
FRESH dir and never reuses a stale DLL; old versions coexist (no auto-delete).
Pre-1.0: legacy flat/unversioned fallbacks dropped.
3. sljit default-on (validated). gcc and sljit are complementary, not exclusive:
gcc fills the cache via prebuilt DLLs + async batch (compile_fragment NULL),
sljit JITs gcc-absent regions synchronously on-miss. New code_provider_sljit()
exposes sljit for the JIT-on-miss gap-fill independent of the "active" backend;
the dispatch gate fires whenever sljit is available (head<0 == gcc-absent), and
apply_live_policy defaults live ON when sljit is available. Priority gcc > sljit
> interp; shards still route through the same-state diff gate (device-touch +
diverging shards pinned to interp) so default-on does NOT reintroduce the
save-load wedge; a declined/failing shard falls to interp (never a crash).
Verified live: live=1, 62 shards compiled, 2 declines->interp, native:interp
ratio ~31%->~87%, gcc still primary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 16:9 edge pop-in is the parallax 2D BACKDROP layer (ocean/cloud/mountain/
grass), drawn by overlay actor handlers (FUN_801216bc family) that compute
screenX = (worldX - camX) >> parallax in pure integer math and store it to the
object's +0x12 field WITHOUT the GTE — so the GTE X-squash that gives 3D the
wider 16:9 FOV never reaches them and far pieces sit past the 320px edge,
clipped (the blue void / half-rectangles at the edges). See TombaRecomp
ISSUES.md #8C for the full root-cause trail.
Fix = psx_ws_backdrop_x() (gpu.c): squash screenX around screen-centre by the
same ws factor as the GTE, identity at 4:3. Applied on BOTH execution paths
because overlay code runs interpreted when no cache DLL is loaded (the common
case in dev — see #8G):
- Native: recompiler emits write_half(.., psx_ws_backdrop_x(rt)) at the
[widescreen.backdrop] x_sites store (config_loader + code_generator +
main_psx --ws-config so the overlay compile gets the sites). Forwarded into
overlay DLLs via OverlayCallbacks {ws_backdrop_x, ws_x_margin, ws_sprite_tag}
(overlay ABI v2->v3 as the cache discriminator); compile_overlays.py glue +
--ws-config passthrough.
- Interp: dirty_ram_interp SH hook keyed by psx_ws_is_backdrop_site(pc); the
runtime registers the x_sites from game.toml (main.cpp).
Verified in data: backdrop slot screenX squashes 371 -> 318 (pulled on-screen).
PARTIAL: visible culling persists -> remaining cause is the backdrop spawn/
despawn window (separate from screenX clip); see #8C "REMAINING CAUSE".
Also added (earlier this session): ws_census draw ring, ws_margin override,
gpu_state ws fields.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The overlay cache keys DLLs by guest-bytes CRC, which is flavor-blind — the
same guest overlay yields the same filename whether compiled by the base
recompiler or a variant (e.g. widescreen, which emits psx_ws_*/GTE-adjust
symbols). Two flavors sharing a cache dir would collide: a base runtime could
pick up a widescreen DLL (or vice versa) with no guard, since both report ABI
v2.
Fold a codegen-flavor id into overlay_abi() (high 16 bits) and check the
combined tag in the loader's existing ABI gate. Base/master flavor is 0, so the
tag stays == PSX_OVERLAY_ABI_VERSION — fully backward compatible with every
existing base DLL (verified: an 8-DLL base cache loads with zero rejections).
A variant build sets PSX_OVERLAY_FLAVOR (e.g. widescreen: -DPSX_OVERLAY_FLAVOR=1
for the runtime, --flavor 1 for compile_overlays.py); its overlays then report a
distinct tag, so the base runtime rejects+regenerates them and vice versa. The
caches stay separate automatically, even in a shared directory — no new paths.
overlay_api.h: PSX_OVERLAY_FLAVOR (default 0) + PSX_OVERLAY_ABI_TAG.
overlay_loader.c: both ABI gates check the tag.
compile_overlays.py: --flavor arg -> -DPSX_OVERLAY_FLAVOR, baked into overlay_abi().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bug D root cause (Tomba chest freeze, also QA issue #1): the game's own
dispatcher func_80056E08 has a dead jumptable case (type 0x21 ->
jal 0x80120B3C) whose target is mid-function in EVERY overlay variant on
the disc (only the boot EXE references it; the real handler entry is
0x80120AC8). Entering mid-function skips the prologue, runs an epilogue
for a frame that was never pushed, and returns through garbage. Our C
model then resumed the suspended generated continuations on the moved
guest stack -> double-executed tails -> infinite zombie loop (~3K wild
calls/frame) -> MMIO fatal. Family root of Bugs A/C/D.
The contract: a generated C continuation may only run if the guest
actually arrived at it ($ra == call site return address, $sp == caller
sp at the call). Violations begin a "bail" unwind: cpu->pc := the wild
jr's true target, every generated frame returns immediately, and the
unwind resolves at the first enclosing site whose contract matches (or
flattens at the outermost dispatch loop, re-dispatching the wild target
on a clean host stack -- matching real-HW one-stack semantics).
- cpu_state.h: bail state + psx_call_contract() helper (+ overlay-DLL
pointer indirection under PSX_OVERLAY_DLL_BUILD)
- emitters: (ra, sp) checks at every BIOS and game jal/jalr site;
psx_dispatch_impl validates pc==0 and stop-hits against (stop, sp),
handles bail resolve/propagate/flatten
- dirty_ram_interp: surfaced calls validate the contract on return;
bail aborts the interp run; tail transfers into compiled code now run
with pc cleared (closes a latent double-run on plain C returns)
- overlay cache ABI v2: DLLs export overlay_abi(); loader rejects and
deletes stale DLLs (autocompile regenerates with contract checks)
- telemetry: bail_first/resolved/flattened/anomaly in heartbeat + dump
headers; anomaly guard at exception entry; TCP_COMMANDS.md docs
Verified: BIOS+Tomba regen, boot to intro at locked 60.0 fps, all bail
counters 0 through boot/title/attract (no false positives). Chest-area
gameplay verification pending (requires user play).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matthew Stanley