Native-wide perf attribution instrumentation (permanent):
- Each wide-mirror pass (tex-batch mirror, flat-geometry mirror, overlay
full-width rect, wide_clear) is bracketed with a GL_TIMESTAMP pair
(glQueryCounter; TIME_ELAPSED cannot nest inside the frame scene query).
Pairs pool per buffered frame (1024/frame), summed at readback into
mirror_gpu_ms + mirror_passes per sample; frame_perf now reports
mirror_gpu_ms_avg/max, canon_gpu_ms_avg (scene minus mirror),
mirror_passes_avg and mirror_pass_us for wide frames.
- New debug cmd gl_ws_ablate mode=<0..3>: 0 normal, 1 skip mirror passes
entirely, 2 mirror state churn without draws, 3 mirror draws stay on the
hr FBO (no per-pass FBO rebind; content-corrupting perf probe).
First MMX6 16:9 measurements with the split: mirror passes cost 20-125us
EACH (FBO ping-pong penalty per pass) but MMX6 2D batches to ~3-15
passes/frame so 16:9 attract holds ~60fps on this branch; the per-pass
cost x ~900 immediate flat prims is exactly the Tomba2 3D 60ms scene.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause (proven via the new present ring + gl_coh_ring correlation):
flush_cpu_upload() merged all pending CPU->VRAM writes into ONE union
bounding box. A frame with two disjoint uploads (sprite column + tile row —
routine MMX6 texture streaming) produced a union spanning the display
framebuffers, which the flush painted from the STALE CPU VRAM mirror (the
FBO is authoritative under GL) — stomping live frames with black for
exactly two presents (one per double-buffer parity). Scene-dependence of
the uploads made it look config/timing dependent; the software renderer is
immune because its CPU array is authoritative.
Fix: exact pending-rect list (16 rects; merges only when zero uncovered
pixels are added — containment or same-band extension; wrap-aware GP0(A0)
transfers split into up to 4 exact rects instead of the old "wrapped: take
all" full-VRAM union; overflow -> order-preserving flush-all; pre-context
union retained where the CPU array is authoritative). Merge rule verified
by a 20k-randomized-rect host test (0 stale / 0 missing painted pixels).
Observability (always-on): gl_present_ring records every SwapWindow (path,
src/letterbox rects, drained glGetError, wall-ms, backbuffer + blit-source
pixel samples) — the instrument that gave the 1:1 black-capture
correlation. debug_server_send_fmt now heap-formats responses >64KB (they
previously truncated into unparseable JSON, breaking large ring dumps).
(debug_server.c also carries the Vulkan screenshot_file sync hunk from the
concurrent VK bring-up; see the next commit.)
Validated: ~18-min MMX6 GL attract soak, ~1600 window captures across 3+
attract cycles — zero isolated black frames, zero GL errors (pre-fix:
19/40 captures black at the intro). Tomba1 build-gl boots clean; its full
attract soak follows. ENHANCEMENTS.md added (framework enhancement ledger,
R1 OpenGL / R2 Vulkan state).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tomba2's Trolley attract demo interprets a working set past 64K distinct
PCs. The per-PC tracking tables (g_dirty_ram_pc_table / exec_pc_table) used
an UNBOUNDED linear probe, so once a table filled, every lookup for an
absent PC scanned all 65536 slots before returning NULL — per executed
instruction. Host thread profile showed ~99% of emu-thread time inside
pc_table_get_or_insert_in; guest crawled at 0.3 fps (1M guest cycles per
3s wall) while telemetry showed the window "LIVE" on residual motion.
Fix: hard-bound the probe to 128 slots (miss => NULL => "stop tracking",
the documented degradation) and grow the tables 65536 -> 262144 so the
Trolley working set actually fits — the same table feeds overlay-capture
seed lists (MAX_CAPTURE_PCS aliases it), so the old size also silently
truncated capture manifests for large regions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The emitter routes mfc2/cfc2/mtc2/ctc2 on flag/IR/derived GTE registers
through direct gte_read_data/gte_read_ctrl/gte_write_data/gte_write_ctrl
calls (code_generator.cpp COP2 case), but the overlay DLL preamble had no
shims for them — any GTE-heavy overlay failed to LINK (undefined
gte_read_ctrl) and its region ran interpreted forever. That silently
blocked all new native coverage for the lava/Trolley attract-demo regions
after the caches rotated.
GTE state is shared runtime state, so the shims forward unconditionally
through OverlayCallbacks (a local copy would fork the flag/IR timeline).
ABI bumped 9 -> 10; the loader gate rejects+deletes pre-shim DLLs and
autocompile self-heals; plain gte_data[]/gte_ctrl[] accesses stay direct
CPUState reads in emitted code, matching the emitter.
Smoke-validated: previously-failing GTE-heavy captures now compile clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of MMX6 "A game data could not be found" (weeks-old, cross-backend):
the game's save-scan calls memset through the kernel A0 jump table; entry
A0:2B points into BIOS ROM at 0xBFC02B8C (one of the entries the kernel never
patches into RAM). Transitive discovery can never find it (reachable only via
the data table, no jal), so the dispatch missed and psx_unknown_dispatch
silently no-op'd — killing the scan before firstfile ever ran. Every card
layer (SIO, IRQs, SwCARD events, sector data, kernel directory parse) was
verified correct along the way; the sole defect was the undiscovered ROM
function plus the silent miss.
- recompiler/seeds: enumerate the post-init A0/B0/C0 tables (kernel RAM
0x200/0x874/0x674) and seed every ROM-pointing entry with no dispatch
coverage — 30 functions (memset, memcmp, memmove, strncat, rand-family,
card/dev services...). memset was merely the one MMX6 called; the rest were
identical latent bugs. BIOS regen: 4406 -> 4439 dispatch entries.
MMX6 CONTINUE now lists and loads saves (user-validated + scripted run:
full FILE SELECT, 1093 card txns, unknown_dispatch_log total=0).
- traps.c: an unresolvable dispatch target now dumps a crash report and halts
BY DEFAULT (PSX_FAIL_FAST_UNKNOWN_DISPATCH=0 opts out for diagnostic
sessions; the miss is ring-recorded either way). The old silent-no-op
default left stale registers and a severed call chain, and masked this bug
for weeks. Deliberately NO interp fallback: the dirty-RAM interpreter is
for runtime-written code (Rule 18); interpreting around a static-discovery
hole would hide it exactly like the no-op did.
- runtime.cmake: EXE_NAME option — per-game process names (mmx6-runtime.exe)
so one game's `taskkill /IM psx-runtime.exe` dev sweep no longer kills
every other game's instance.
- debug rings (the tooling that found this): HLE call ring gains
consecutive-duplicate collapse w/ repeat counts (a 4-way TestEvent poll at
175K calls/s no longer flushes the ring in 0.1s), per-entry TCB + in_exc
(thread attribution), a "since" paging param and 8MB dump buffer;
card-trace EvCB watch widened to all 7 card event slots (slot 0 =
SwCARD-IOE, the success event, was previously invisible).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER PIVOT 2026-07-02 (CLAUDE.md §0 amendment included): psxrecomp now
carries an opt-in High-Level Emulation backend alongside LLE. LLE (the
recompiled BIOS) stays the default, the reference implementation, and the
correctness/timing oracle; every service HLE does not implement falls
through to LLE, so HLE is never load-bearing beyond what it covers.
Mechanism: full_function_emitter.cpp emits a null-by-default hook
g_psx_bios_hle_hook consulted at the top of every psx_dispatch_impl
iteration, on the pre-normalize physical address, BEFORE any backend
(game image / dirty-RAM / static table) claims the target. A handled
call resumes at $ra through the trampoline's existing return/tail
contract. NULL (default) = pure LLE, dispatch-identical. Regenerating
BIOS+game images is required (emitter change).
v1 call-HLE (runtime/src/bios_hle.c): the B0 event family — DeliverEvent
(incl. callback dispatch with the kernel-true $ra 0x1720), OpenEvent,
CloseEvent, TestEvent, EnableEvent, DisableEvent — each ground-truthed
instruction-by-instruction against the SCPH1001 kernel disassembly
(0xBFC11644..0xBFC11A84) and operating on the REAL guest EvCB table
([0x120]/[0x124], stride 0x1C). WaitEvent/threads/pads/card/A0/C0 fall
through to LLE. Cycle charges are documented approximations; LLE remains
the timing oracle. No-stubs stands.
Boot HLE: with the tier on (and no keep-intro opt-out) the hook one-shot
intercepts the first dispatch of the BIOS shell entry (RAM 0x30000 —
LoadRunShell's indirect call always routes through the dispatcher), so
BIOS Main() proceeds straight to SYSTEM.CNF + game EXE load with kernel
state built entirely by the real recompiled kernel init, unpaced (boot
turbo) until game handoff. This REPLACES the fast_boot snapshot
capture/restore; fast_boot=true now aliases the boot-skip alone.
Selection: [runtime] bios_hle / bios_hle_keep_intro (+ settings.toml
bios_hle mirror), PSX_BIOS_HLE / PSX_BIOS_HLE_KEEP_INTRO env overrides,
startup banner bios_backend=/bios_boot=. The PSX_HLE_SCHEDULER spike is
folded in as the tier's standing subsystem replacement: default from
[runtime] hle_scheduler via psx_hle_scheduler_set_default, env wins.
Observability: always-on 16K-entry HLE ring (route LLE-fallthrough /
HLE / boot-skip per record) + hle_dump TCP command (TCP_COMMANDS.md).
VALIDATED on Tomba (build-wt): LLE pass byte-identical behavior (real
intro, card check, save/load OK); HLE pass boots shell-less, TestEvent/
DeliverEvent route HLE, ChangeTh/ReturnFromException/OpenTh/CloseTh and
the card stack fall through to LLE, save/load OK (user-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cycle-accuracy emitter emits calls to 12 timing runtime functions
(psx_cyc_load_*, psx_icache_fetch, psx_muldiv_*, psx_gte_*, psx_slice_block,
...) + debug_server_cyc_observe + g_debug_last_store_pc. Overlay DLLs never
got these symbols, so every overlay failed to link -> 0 native shards ->
100% interpreter (unplayable). Fix (option A): expose the 12 via the overlay
callback ABI.
- overlay_api.h: +12 timing callbacks in OverlayCallbacks, ABI 8 -> 9.
- overlay_loader.c: populate the 12; add warn_on_cgtag_mismatch() (loud
warning when shards exist under the wrong cg-hash).
- compile_overlays.py: 12 preamble forwarders + local g_debug_last_store_pc
+ -DPSX_NO_DEBUG_TOOLS.
- cosim.c/main.cpp: input-cosim setpad + held pad (deterministic input).
- cosim.py: env-overridable COSIM_EXE/CWD/GAME.
KNOWN REGRESSION (patched on a follow-up branch): Tomba pause menu (triangle)
wedges. Root-caused: an IRQ taken mid-block resumes at the function entry
leader instead of the interrupted PC, re-zeroing the menu's VSync wait
counter (scratchpad 0x1F8001E8) every frame so it never reaches its target.
Not the scheduler (LLE reproduces) and not timing math -- interrupt-resume
re-entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New always-on ring (device_trace.{c,h}, compiles into both binaries like
parity_trace) recording every hardware IRQ-raise edge with its guest cycle,
source (I_STAT bit) and detail. All native device sources funnel their raise
through one choke point psx_irq_raise(bit,detail) in interrupts.c (VBLANK/GPU/
CDROM/DMA/timers/SIO routed; DMA passes DICR channel flags, timers pass index),
so the device-timing audit sees every event from one place. devtrace_dump/ctl
debug commands (cycle-window + source filters) + tools/devtrace_diff.py align
the two device-event timelines by source ordinal and flag where cycles drift.
The Beetle oracle side (mednafen IRQ_Assert rising-edge hook) lives in the main
checkout.
This is the level below parity_trace: parity names the divergent control-flow /
gate input; device_trace names the divergent HARDWARE EVENT behind it.
First result (MMX6 cutscene asset-load over-fast wedge): the load's CD+DMA
activity is compressed into ~232M cycles on native vs ~384M on Beetle (~1.65x),
completing the func_8001CB3C scene setup ~100M cycles / ~176 frames early. The
producer is CD-read / DMA-completion timing (DMA channels 2/3/4 = GPU/CD/SPU
asset streaming) -- confirming the device-completion-latency hypothesis and
ruling in the CDROM/DMA cycle model as the over-fast subsystem. (VBlank verified
correct: native VBlank edges are exactly 564480 cyc apart.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends the two-process parity ring so each row carries, per watch word, the
exact store PC + guest cycle + frame + (approx) writing TCB of the most recent
write to that word. parity_trace_note_write() maintains the table from BOTH
processes' write paths (native: memory.c psx_write_* with g_debug_last_store_pc;
Beetle: beetle_libretro.cpp wtrace_callback with the store pc). Both parity_dump
serializers emit the new wwpc/wwcy/wwf/wwt arrays (identical wire format), and
parity_diff.py prints a side-by-side WATCH PROVENANCE block at the divergence.
This turns "the gate-input value diverged" into "who/when produced it" -- the
instrument ChatGPT recommended for the MMX6 over-fast wedge.
First result (MMX6 cutscene->gameplay, watch 0x80071088/0x800CD3F8/0x8006D9AC/
0x800CD3FC/0x800CD404/0x800200F4): the cutscene-setup func_8001CB3C (writer pc
0x8001CB7C of countdown 0x800CD3FC) runs ~99.7M guest cycles / ~176 frames
EARLIER on native (cyc 998,597,937 frame 1769) than Beetle (cyc 1,098,296,890
frame 1942) -- same writer PC, fired early. On Beetle the state struct
0x800CD3F8 is never touched by game code (stays BIOS-init 0 -> setter path); on
native it advances. The over-fast wait is localized one level above the state
machine, at whatever schedules func_8001CB3C.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add absolute guest-CPU-cycle (since boot) to every parity row via a per-process
parity_host_cycle() accessor (native psx_get_cycle_count; Beetle
beetle_core_get_guest_cycles — directly comparable). Widen ParityEntry.cycle to
64-bit; both dump handlers emit it. This turns the cutscene divergence from "273
frames early" into a hard cycle delta: Beetle runs 566k cycles/frame (≈ real PSX
VBlank 564480); native reaches the cutscene state-advance ~150-200M guest-cycles
early — an over-fast cycle deficit (DMA-cycle-steal signature) in the faithful-
timing stack. Beetle producer side mirrored in the main checkout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
End the "somewhere upstream" blindness with a reusable first-divergence
instrument (PRINCIPLES.md "find the first divergence; is the value WRONG or the
behavior MISSING"). A TCB-scoped, freeze-on-trigger control-flow ring with an
identical struct + JSON wire format on both processes, so a generic sequence-diff
(tools/parity_diff.py) locates the first logical divergence between recomp and
oracle — not a frame diff.
- parity_trace.{h,c}: armed-from-boot ring, gated on current_tcb==watched_tcb,
latches on a configured trigger target so the pre-divergence window survives
the wedge loop. Up to 6 watch-word snapshots/row. Self-contained TU shared by
both binaries; frame source decoupled via parity_host_frame().
- fntrace.c: per-dispatch producer hook (cheap armed gate on the hot path).
- traps.c: YIELD control-event hook in the scheduler switch.
- main.cpp: env-armed from boot (PSX_PARITY_TRACE=1; TCB/trigger/watch tunable),
defaults target the MMX6 cutscene wedge.
- debug_server.c: parity_dump + parity_ctl commands.
- tools/parity_diff.py: pulls both rings, sequence-diffs by (kind,pc,target).
First use (MMX6 freeze) confirmed MISSING behavior, not a wrong value: native
thread1 is permanently parked at func_8002000C resume (TCB EPC 0x800200F4),
handshake flag 0x8006D9AC never set (0 across 16384 rows), setter func_8001344C
never runs; thread1 re-resumes a dead func_8002000C frame whose saved-ra slot was
reused -> jr 0x800CD3F8 wedge. Upstream context-staleness/scheduling root.
Beetle mirror producer + the cross-process diff are the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the non-deterministic host-fiber thread-switch bridge with a deterministic
TCB scheduler (HLE_SCHEDULER_CARVEOUT_PLAN.md steps 2-8, minimal "A+" scope):
- psx_scheduler_run (traps.c): outer trampoline wrapping the top-level dispatch
(main.cpp calls it instead of bare psx_dispatch). setjmp(g_scheduler_jmpbuf);
restore current TCB / dispatch / on-escape reset g_psx_dispatch_depth +
g_psx_call_bail; boot/pre-TCB runs raw cpu->pc; one-level switch-back safety
net; fail-closed psx_is_dispatchable on every restored EPC.
- psx_request_thread_switch: save yielder ctx (resume=$ra), repoint
dword_108->entry, longjmp to the scheduler. No fibers, no cpu->pc=0 signal.
Fires only with in_exception==0, so it never unwinds a live exception_jmpbuf
frame (the in-exception RFE path stays LLE/intact).
- sched_escape_ring (step 8) for always-on observability.
- Hidden toggle PSX_HLE_SCHEDULER (default 1=HLE, 0=legacy fibers), read once;
psx_change_thread branches on it; startup banner prints the mode. The
trampoline is transparent in LLE mode. Fiber funcs kept compiled-but-dead.
Validated live (HLE): thread1 trace shows ONLY the new TCB-switch events and
ZERO fiber kinds 10/11/13 — the per-frame fiber recreate is eliminated. Builds
clean. Freeze NOT resolved: same 0x800CD3F8 wedge persists (separate upstream
root, per ChatGPT Q5). Keep HLE default until the upstream cause is fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First step of the deterministic TCB cooperative-thread scheduler carve-out
(HLE_SCHEDULER_CARVEOUT_PLAN.md §9, ChatGPT-approved "A+"). Replaces the
non-deterministic host-fiber bridge (confirmed root of the MMX6 cutscene freeze:
thread1's fiber recreate-per-frame, pc=0 silent exit).
- runtime/include/psx_scheduler.h: psx_run_reason_t (CONTINUE / YIELD_TO_TCB /
RESUME_CURRENT / GUEST_EXIT / FATAL) + psx_sched_escape_t, the design rationale
(pure-TCB BIOS scheduler, no stack switch; guest state lives only in
CPUState/RAM/TCB), and the load-bearing invariant.
- traps.c: g_scheduler_jmpbuf + g_sched_escape definitions + psx_is_dispatchable
(fail-closed on pc==0 and the exception sentinel — the known-bad resume PCs the
old pc=0 pathology produced).
INERT: nothing calls these yet; psx_change_thread_fiber (host-fiber bridge) is
still authoritative. Builds clean (traps.c only, no game-C regen). No behavior
change. Next: outer scheduler loop + route ChangeThread/RFE through structured
escapes, then remove cpu->pc=0 + fiber recreation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recompiler/code_generator.cpp: stamp g_debug_last_store_pc before EVERY game
store (SB/SH/SWL/SW/SWR, SWC2, + the ws-backdrop / game-option special paths)
and declare the extern in the game-C preamble. Previously only the BIOS emitter
(strict_translator.cpp) stamped it, so game stores inherited the last BIOS store
pc -> wtrace/readtrace mis-attributed game writes to the BIOS. VERIFIED LIVE:
MMX6's producer trace for the 0x800CD3F8 write now reports the real game store
pc 0x8001A708 (matches Beetle), not the stale 0xBFC11BF0.
runtime: g_pc0_reason slice-exit provenance, fail-closed exception-sentinel
asserts, named thread-event kinds, freeze-heartbeat RVA emission (diagnostics,
default-on, additive).
Carries the in-progress faithful-timing / MDEC work (cpu_state.h, memory.c,
dma.c, interrupts.c, dirty_ram_interp.c, full_function_emitter.cpp) as an
uncommitted-WIP checkpoint. tools/: regen_bios.sh, bios_emitter_fingerprint.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New runtime/src/psx_icache.c: direct-mapped (4 KB / 256-line) instruction-
cache fetch cost, transcribed from Beetle PS_CPU::ReadInstruction:
HIT (TV == addr) -> +0, no give-back clear
MISS, KSEG1/uncached -> +4 (BIOS ROM)
MISS, cached (KSEG0/KUSEG) -> +3, then +1 per word refilled from the
missing word to the line end (earlier words
stay invalid); also clears the load give-back.
Mirrors only the per-word TV tag array (cost + tag state; the value is already
in hand). Wired into the dirty-RAM interpreter (exec_one) per instruction,
charged BEFORE the §1 base — Beetle order.
New ruler #2 loop `icache_miss`: loop top + a victim 0x1000 apart alias the
same direct-mapped line with different tags, forcing a refill miss every
iteration (other loops are all-hits after warm-up).
VALIDATED interp vs Beetle (PSX_FORCE_INTERP=1 PSX_ICACHE=1): icache_miss
native +14 == Beetle +14 (per-iter 17); the other 12 loops unchanged at +0
fetch. So the I-cache hit AND refill-miss costs are MEASURED equal to the
oracle on the interp path.
Opt-in via PSX_ICACHE=1 (default OFF) — charging fetch cost in only one
backend would fork mixed compiled/interp timing. Production activation +
the compiled-path emit (per cache-line leader) + ruler #1 cold-spike
validation are Stage 2. Default-off => no production change (the interp's
fetch call is a no-op without the env).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the last steady-state cycle divergences; native now == Beetle on
the complete 12-loop ruler #2 suite.
- MFC2/CFC2 (GTE reg -> GPR read): new psx_gte_read stalls to gte_ts_done
AND hands the stall amount to following instructions as a load-delay
give-back (Beetle cpu.cpp:1332-1341: LDAbsorb=gte_ts_done-ts,
LDWhich=rt). MTC2/CTC2 (writes) keep stall-only psx_gte_stall.
- MFC0: arms ld_absorb=0 / ld_which_t=rt (Beetle delayed load) — sets
ReadFudge=rt so a load in the next slot gets no fudge.
- psx_muldiv_stall (MFLO/MFHI): now CONSUMES the pending load give-back
while stalling to muldiv_ts_done, plus the muldiv_ts_done-1 off-by-one
shortcut (Beetle cpu.cpp:1723-1736). No-load code is unaffected
(read_absorb==0 -> plain advance).
Wired in both static emitters (code_generator, strict_translator) and
the dirty interp.
METHODOLOGY (Rule 15): Beetle cyc_watch must be sampled at STEADY STATE
— its warm-up window reports the no-give-back value (the source of the
prior "gte_rtps +15 EXACT" mis-record; true steady Beetle = +11).
Validated at steady state, native == Beetle on ALL 12 ruler #2 loops:
the 4 divergences closed (gte_rtps 18->14, gte_nclip 11->7,
gte_read_use 19->14, ld_div 45->49); the 8 CPU-load/alu/div/mult loops
held. Ruler #1 delta 0. Tomba 2 FMV plays (no regression).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the last unmodeled load-path cycle component — the R3000A
load-delay pipeline interlock (Beetle ReadAbsorb/ReadFudge/LDAbsorb/
LDWhich) — as ONE shared per-instruction model across the dirty interp
and BOTH static emitters, replacing the flat per-instruction +1 base and
the flat +4 main-RAM read wait-state.
Model (transcribed from beetle-psx cpu.cpp + libretro.cpp, verified Δ):
- §1 base: a pending load's per-register give-back (read_absorb) is
consumed INSTEAD of charging +1 (pipeline write-back overlap).
- GPR_DEPRES: each opcode's src/dst regs end the give-back on use.
- DO_LDS: commits the prior load's give-back + ReadFudge.
- load ReadMemory: +2 fudge iff the predecessor committed no load;
main-RAM region +3 + completion +2 (LWC2 +1) become the LDAbsorb
give-back; scratchpad +0.
New runtime/include/psx_cyc.h (inline §1/deps/DO_LDS + step) over new
CPUState timing fields; psx_cyc_load_*/psx_cyc_lwc2_read in memory.c
(region-aware, own the whole load interlock, value via uncharged
psx_read_*); psx_cyc_dep_res_mask in psx_instr_cost.h (per-opcode
GPR_DEP/RES, gen-time literal). cpu->read_* rewired to uncharged in
main.cpp; charge_main_ram_read removed. Wired before each instruction
body in code_generator.cpp (game) and full_function_emitter.cpp +
strict_translator.cpp (BIOS); loads route through psx_cyc_load_*.
Validated against the Beetle oracle:
- ruler #2 load2: native +10 -> +11 == Beetle (all other components
still exact: alu+1/load+5/div+38/mult+15/gte_rtps+15/gte_nclip+8).
- ruler #1 [0x80001C5C->0x80001CA4]: 54 -> 56 == Beetle steady-state
(residual 84/77 spikes = I-cache cold refill, the separate P2 axis).
- Tomba 2 boots to the intro FMV (pixels on screen, no regression).
Follow-up (separate commit): GTE-read (MFC2/CFC2) + MFC0 LDAbsorb
give-back, and muldiv-stall give-back consumption — needed for mixed
game/FMV faithfulness but do not affect the rulers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Models the GTE command latency + stall-on-COP2-access, the next Stage-2 cycle
component. Mirrors the mult/div completion-stall mechanism:
- cpu->gte_ts_done (new CPUState field, appended): a GTE command sets it to
psx_cycle_count + (cost-1); back-to-back commands serialize (psx_gte_set
stalls to the prior deadline first). Any COP2 register access
(MFC2/CFC2/MTC2/CTC2/LWC2/SWC2) calls psx_gte_stall, advancing guest cycles
to the deadline if the op is still in flight.
- Cost table (psx_cycles.c, psx_gte_cmd_latency): per-op cost-1, transcribed +
verified from beetle-psx gte.cpp op returns (RTPS15 NCLIP8 MVMVA8 NCDS19
NCDT44 RTPT23 NCCT39 ... AVSZ4=5 not 6). cost-1 because the COP2 instruction's
own +1 base is charged separately (Beetle GTE_Instruction returns ret-1).
- Set armed in the shared gte_execute() (one site for both backends); stall
emitted before every COP2 reg access in both emitters (code_generator,
strict_translator) and the dirty interp. The compiled "+1 at top" vs interp
"base after exec_one" offset cancels (same as muldiv).
Validated EXACT on ruler #2 (new gte_rtps + gte_nclip loops): native +15/+8 ==
Beetle +15/+8. All other components unchanged (no regression); Tomba2 boots to
FMV. Requires regen (emitters changed); runtime+BIOS+game regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2's biggest cost gap closed. MULT/MULTU/DIV/DIVU now set a completion
deadline; MFLO/MFHI stall guest cycles until it — exactly like Beetle's
muldiv_ts_done. Latencies transcribed from cpu.cpp: DIV/DIVU=37; MULT via
MULT_Tab24 (14/10/7) on the first-operand magnitude. Native previously charged
ZERO for this.
- Per-instruction cycle charging is now the DEFAULT on this audit branch
(PSX_CODEGEN_CYCLE_PER_INSN), required so the stall absorbs correctly (the
running cycle count must be accurate mid-block). Validated cycle-neutral for the
identity base model; Tomba 2 still reaches the FMV.
- runtime: CPUState.muldiv_ts_done + psx_muldiv_set/stall + psx_mult_latency_s/u
in psx_cycles.c. Emitted at the mult/div/mflo/mfhi sites in the GAME emitter
(gated PSX_ENABLE_BLOCK_CYCLES). Stall fires even for mflo/mfhi to $zero.
- ruler #2 native half: tools/cycle_testrom/CMakeLists.txt builds psx-cyctest
(links the recompiled test EXE), boots disc/cyctest.cue (serial CYCT-00101 so
disc-identity matches, no warning dialog). measure.py --port 4600.
VALIDATION (native vs Beetle oracle, per-iteration component cost):
div +38 vs +38 EXACT | div_spaced +38 vs +38 EXACT (absorb works) |
mult +15 vs +15 EXACT | baseline/alu match.
Remaining: load native +7 vs Beetle +5 (memory.c flat +6 wait-state needs
calibration to ~4) — a smaller refinement. BIOS emitter still block-mode (ruler
#1 unchanged at 30) — applying per-insn + muldiv there is the next step to close
ruler #1's div-stall gap.
No FMV regression (i_stat 0x8D, screenshot verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build the game-independent cycle-attribution ruler the plan called for, and
fix two real bugs it exposed.
Tooling (debug_server.c, full_function_emitter.cpp):
- cyc_watch double-fire dedupe: observe fired from BOTH the dispatcher and the
function prologue at the same cycle, double-recording every dispatched entry.
Key on (phys,cycle); a genuine re-entry is always >=1 cycle later.
- Per-block-leader cycle observe (debug_server_cyc_observe), emitted by the BIOS
emitter under #ifndef PSX_NO_DEBUG_TOOLS (prod = zero overhead). Native now
samples at EVERY compiled block leader (matching Beetle's before-instruction
sample), so cyc_watch can anchor any block-leader PC, not just fn entries.
Emitted at normalize_address() so relocated-kernel anchors match the RAM space.
Bug fix (psx_instr_cost.h):
- Loads were double-counted. Stage-2 #1a (2ef47bd) returned 3 for loads (+2
data-access) in psx_instr_base_cycles WHILE memory.c charge_main_ram_read
already charged +6 per main-RAM read. The ruler region [0x80001C5C->0x80001CA4]
decomposed native=34 as 16(base) + 2 + 12(mem) with loads counted twice. The
opaque 0x80017FC4 window hid it because the load over-charge masked the entirely
unmodeled divu->mflo stall. Reverted the opcode fn to pure execute base (loads=1)
per its own stated contract; memory.c is the single address-keyed data-access
owner (like Beetle ReadMemory). Ruler: native 34->30 (exact), FMV unaffected.
Docs: CYCLE_MODEL_BEETLE.md (full R3000A model transcribed from in-tree Beetle
cpu.cpp, cited), plan + burndown updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Transcribed from Beetle PS_CPU::ReadMemory: a CPU load adds +2 cycles (LWC2 +1);
stores posted (~0). psx_instr_base_cycles now returns 3 for LB/LH/LWL/LW/LBU/LHU/LWR,
2 for LWC2, 1 else. Flows to BOTH backends via the single-source seam.
Δ GATE (cyc_watch @0x80017FC4, native vs Beetle per-iteration): 46 -> 56 (toward
Beetle 91) — load cost closed ~10 cyc; remaining ~35 = later components (I-cache
fetch, mult/div, GTE, region/absorb refinements). FMV still streams (no regression).
Approximation noted in the header: no scratchpad-free / load-delay-absorb / region
yet (Δ-gated next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
P3 step 1 (FAITHFUL_TIMING_PLAN.md): both backends now account guest CPU cycles
through ONE function, runtime/include/psx_instr_cost.h::psx_instr_base_cycles(insn),
so they can never disagree (the -8 drift class). Interp routes its 3 per-insn
charges (exec_delay_slot, dirty dispatch loop, precise slice) through it; the
recompiler sums it over each block (+ the outside delay-slot clone) to fold into
the compile-time block charge. Stage 1 = identity (1 cycle/insn) = the proven
baseline, so the seam is BEHAVIOR-PRESERVING: regen is byte-identical (verified,
full.c + dispatch.c) and Tomba 2 still reaches the intro FMV (i_stat 0x8D
streaming). Stage 2 edits ONLY psx_instr_base_cycles (transcribe real R3000A
costs from in-tree Beetle cpu.cpp/gte.cpp + psx-spx, verify each vs Beetle) and
both backends update together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The multi-week Whoopee-Camp logo->FMV stall was a TIMING-FAITHFULNESS bug, not
game logic: the static recompiler charged cycles block-granular and a branch
whose delay slot is a SEPARATE block leader (branch at end_addr, delay slot
outside the block) undercharged by one cycle on every execution -- the
always-executed delay-slot clone was emitted but counted by neither the branch
block nor the (unentered) delay-slot block. ~8 such sites on the boot init path
= the measured -8 cycle drift that made native run behind the interpreter, so
the frame-1824 elapsed-Timer1 logo-delay loop exited after ~42 iterations
(native) instead of ~1557 (interp/HW) and never reached the FMV.
Fix (code_generator.cpp translate_basic_block, the faithful class fix -- every
dynamically executed instruction charged exactly once):
block_exec_cycles = instruction_count
+ (exit branch at end_addr with delay slot outside ? 1 : 0)
applied to both the block cycle charge and the slice budget. R3000 is MIPS-I so
the delay slot is unconditional; the not-taken path already falls through to
branch_pc+8 (no double-count). Result, screenshot-verified: native passes the
logo-delay loop, screen animates, CDROM+DMA stream active, the jungle intro FMV
plays.
Also in this checkpoint (all on wt/tomba2, not merged/pinned):
- PARKED precise interrupt-slicing (PRECISE_IRQ_SLICE.md): root-caused its pc=0
boot exit to a mid-function clean-text resume PC the dispatcher can't re-enter;
ChatGPT-validated fix is "every block leader = re-enterable CPS continuation"
(psx_game_is_function_entry predicate added; precise_pc_dispatchable +
irq_taken-once restructure in psx_run_precise). Default OFF (PSX_PRECISE_SLICE=1
to resume); inert on the baseline. Slice-trace diagnostics added to freeze_check.
- FAITHFUL_TIMING_PLAN.md: authoritative game plan (P1-P6) + standing directive
(build the faithful core, never hacks; ecosystem is being built, not preserved);
referenced from CLAUDE.md Rule -1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 1 of the cycle-budgeted precise interrupt-take-point fix (root cause:
compiled blocks take IRQs only at block edges; HW + the dirty-RAM interpreter
take them at the exact instruction — oracle-confirmed at the Tomba 2 logo
Timer1 debounce). Design validated with ChatGPT; full spec in PRECISE_IRQ_SLICE.md.
This commit lands the additive, no-behavior-change foundation (Task #2) plus the
prior session's diagnostic take-PC capture, and a build-hygiene dedup:
- cycles_to_next_event() (interrupts.c): single source of truth — min guest-cycle
distance to the next DELIVERABLE interrupt (source raises its I_STAT bit AND
that bit is unmasked in i_mask). Aggregates VBLANK + per-peripheral queries.
- timers/cdrom/dma/sio _cycles_to_irq(i_mask): each peripheral owns its own
next-IRQ-distance query; all conservative under-estimates (smaller => slice
more => never run past an event).
- debug_server: cycles_to_next_event command for live validation.
- interrupts.c: take-PC capture into the IRQ_DELIVER event-ring aux (prior
session's diagnostic; records the architectural take-PC for cross-backend diff).
- overlay_loader.c: remove duplicate psx_native_bad_entry definition (HEAD had it
twice; would not link cleanly).
Live-validated at the logo: returns the VBLANK distance, correctly excludes the
three free-running (unarmed/masked) timers and idle CD/DMA/SIO. Runtime-only,
no regen. The two-tier executor (Task #3) that consumes this is next.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In-progress investigation work snapshotted before rebasing onto the
tcc-refactored psxrecomp master. Includes overlay ABI v8
(check_interrupts_at resume-PC callback for async RFE recovery) and
code_generator/full_function_emitter delay-slot cycle accounting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root-caused the Tomba2 native logo-stuck bug: it is a guest-cycle-timeline
divergence between the compiled-overlay backend and the dirty-RAM interpreter,
surfacing through a Timer1 (0x1F801110) debounce loop at game pc 0x8008592C that
reads 1 (native) vs 2 (interp), forking logo->FMV. The handoff's "jalr target
divergence" was a benign store-PC artifact (read-watch proves a0=0xB870 and
t3=[a0]=0x800896E0 are identical in both backends); the scratchpad blind spot
was ruled out (separate sp fingerprint forks at the same frame).
First-divergence observability (debug_server.c, memory.c):
- frame_fingerprint: added separate sp (scratchpad) cumulative hash column +
per-entry cyc, so a scratchpad or cycle fork is classified, not hidden.
- Unified ordered access recorder: one buffer, kind = ramw/spw/mmiow/mmior/ramr,
array index = true execution order, per-entry cyc (psx_get_cycle_count).
record_frame_dump emits the unified stream; record_reads_dump filters reads.
- Targeted main-RAM read-watch (PSX_READ_WATCH="lo,hi" + g_ram_read_watch_active
guard in psx_read_word) — proves a load's value/address per backend.
- PSX_RECORD_FRAME env seed: arm the recorder from boot (race-free), never
arm-then-time.
Fix A (cycle-model unification, part 1): compiled-overlay code was built WITHOUT
PSX_ENABLE_BLOCK_CYCLES, so every psx_advance_cycles() in overlay code was
compiled out -> a function run as a native overlay charged ~0 cycles while the
same function via the interp charged per-instruction (~32x), the entire drift.
- overlay_api.h: OverlayCallbacks gains advance_cycles; ABI v6->v7 (rejects
stale zero-cycle DLLs).
- compile_overlays.py: build overlays with -DPSX_ENABLE_BLOCK_CYCLES=1 and
emit a psx_advance_cycles shim forwarding to the shared host timeline.
- overlay_loader.c: wire s_callbacks.advance_cycles = psx_advance_cycles.
Result: the first debounce now reads Timer1=2 in native (was 1); overlay cycle
charge went 9.5k -> 380k (interp 306k). Native still logo-stuck: a residual,
perfectly linear ~1 cycle/write over-charge remains (block.instruction_count
counts branch + delay-slot as 2 while the interp charges the pair as 1 step),
which flips a later debounce 3 vs 2. Fix A pt2 (cycle-exact block totals) is the
next step. wt/tomba2 only; no pin bump, no master merge, no other-game regen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dispatch path re-CRC32'd a native overlay function's ENTIRE code body on
every single call (cand_crc, unconditional) before running it -- a real tax on
hot per-frame code that blunted the native win. The page-generation counter that
could make this a cheap skip existed but was demoted to a diagnostic.
Promote it: a watched-page write is the ONLY way an entry's code can change, and
every CPU/DMA store funnels through the store chokepoint which bumps
overlay_page_gen for watched pages. So if the page-generation sum is unchanged
since this entry was last confirmed VALID, the bytes are provably unchanged ->
run native WITHOUT re-hashing. Fall back to the full crc32 only when the gen
moved or the entry isn't known-valid (this also re-confirms reload-on-return,
since write-away-then-back bumps the gen). Applied to both the main candidate
chain and the CPS continuation re-entry path. Correctness identical to the old
unconditional hash; only redundant hashing is cut. New gen_fastpath counter
exposed in the overlay status.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tomba 2's title menu was unresponsive and the idle attract demo never
started. Root cause (proven against the Beetle oracle): the SIO pad
state machine answered the DualShock config-mode commands (0x43 enter-
config, 0x4D rumble-map, etc.) for EVERY pad, including a plain digital
controller. Tomba 2's pad driver probes the controller with 0x43 every
frame to detect a DualShock; on real hardware a digital pad (SCPH-1080,
id 0x41) ignores 0x43 (hi-z, no ACK), so the driver classifies it as
digital-only and just polls with 0x42. Our SM answered 0x43, so the
driver went down the DualShock config path and read the 0x00 config-
response data bytes as button data -> phantom "all buttons pressed"
(pad word 0x0000). That made the menu's edge-detected navigation see
nothing newly-pressed (unresponsive) AND reset the idle/attract timer
every frame (no attract).
Fix: model a plain digital controller. New per-slot pad_supports_config
flag (sio_set_pad_config_capable), set from the player pad mode
(PAD_MODE_DIGITAL => 0, ANALOG/HYBRID => 1, default 1). When 0, all
config-command branches (0x43 / legacy / modern 0x44-0x4F) fall through
to the hi-z "no response" path, so a digital pad answers ONLY the 0x42
poll -- byte-for-byte matching the oracle.
Validated live: SIO now shows only 0x42 polls (id 0x41, buttons 0xFFFF),
idle pad word is clean (no 0x0000 phantom), every injected button reaches
the menu var, and START navigates the title menu and starts the game.
Gated so only DIGITAL-mode pads change behaviour; HYBRID/ANALOG pads
(MMX6, Tomba 1, Ape) are unaffected (config-capable as before) -- but a
T1/MMX6/Ape regression pass is still required before any master merge.
NEW failure point (next bug in the chain, now reachable): with input
fixed, the idle attract demo triggers and loads a gameplay level, which
FREEZES -- guest spins in compiled kernel/dispatch code, dirty-interp
idle (insns_run flat), vblank stalled, level textures corrupted. This is
the untested gameplay-overlay stage, a separate issue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OVERLAY_REGION_FLOOR was hardcoded to 0x98000 = Tomba 1's main-EXE text end
(0x10000 + 0x88000). Tomba 2's boot EXE is only 0x28800 (text ends 0x38800),
so its disc overlays load at 0x80085000+ — inside [0x38800, 0x98000), which the
hardcoded floor wrongly classified as statically-recompiled main-EXE text. That
forced every Tomba 2 overlay onto (a) slow block-by-block dispatch and (b) the
bail-prone dispatch_nonlocal_call contract path; when the per-frame interrupt
pump fired mid-contract it cascaded into the $ra=$sp=1 bail storm = the
Whoopee-Camp splash freeze.
Make the floor a runtime value (g_overlay_region_floor), pinned at game load to
(load_address + text_size) & 0x1FFFFFFF. Defaults to 0x98000 for BIOS-only;
evaluates to the same 0x98000 for Tomba 1 (no behavior change), and is
correct-by-construction for every game.
Resolves: bail storm gone (bail_first=0); splash advances at 60 fps past the
old frame-1823 wedge. NEW failure point exposed next: dispatch_depth desync on
the now-local-flow overlay path (fixed in the following commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Always-on, heartbeat-resident diagnostics for the Whoopee-Camp splash spin:
- VBLANK raise/deliver/ack counters (interrupts.c + memory.c) + irq_deliver.
- Deduped bail-source ledger keyed by (call-site $ra, wild target) (traps.c,
hooked at psx_call_contract in cpu_state.h) + heartbeat top-key/first-bail.
Result: REFUTES the exception-reentry-storm hypothesis. VBLANK is raised,
delivered AND acked exactly 1/frame (textbook-healthy); irq_deliver is 1/frame,
not a storm. So the freeze is NOT an interrupt/timer/CD problem -- the hardware
sim is healthy. The ~68k bails/frame are the GAME splash loop busy-polling a
kernel event (TestEvent/WaitEvent) that never fires -- the DeliverEvent/WaitEvent
discriminator is the next probe.
(The dominant dispatch-trampoline bails are emitted in generated BIOS code, so
the precise bail call-site needs a recompiler-emit change + BIOS regen; the
direct-call ledger here is partial. Deferred -- not needed to localize to the
event wait.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The .cue parser only extracted the FILE line and the CD model hardcoded a
single data track: GetTN always returned 01..01 and GetTD returned LBA 0 for
any track. Multi-track discs with a CD-DA audio track (Tomba 2: TRACK 02 AUDIO
@ 38:48:74) therefore reported a bogus TOC.
- iso_reader: parse TRACK/INDEX lines into a CDTrack table; synthesize a single
data track for bare .bin/.iso. New TrackCount/TrackStartLBA/TrackIsAudio +
iso_track_* C wrappers.
- cdrom.c GetTN returns first=1/last=track_count (BCD); GetTD returns each
track's real start MSF (lead-out via 0xAA/track 0 unchanged).
Framework fix for every CD-DA title. NOTE: does not by itself unblock Tomba 2's
boot — that still stalls in a GetStat/GetID/Init disc-init loop (display toggles,
reading=0) which does NOT call GetTN/GetTD; next is an oracle CD command/response
diff at that loop (and the hardcoded GetID region "SCEI" vs USA "SCEA" is a
candidate). cdrom_trace_dump floods with DMA 'D' events — fix that (Rule 15)
before the diff.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings the fail-closed native entry guard from wt/tomba2 (b5a6940 / bd60381) to
master, where it had never been merged — which is why Tomba (and any master-built
title) still wedged. This is the long-standing "blue screen" / shared-RAM-
divergence-between-interp-and-native class.
Root cause: an overlay function dispatched at a FOREIGN INTERIOR PC (a range-
ownership mismatch routed a PC the function does not own into it) hit the CPS
entry-switch `default: break`, which fell through and ran the function FROM ITS
TOP — re-running its prologue and corrupting shared CPU/RAM state, so the guest's
main loop then waits forever on a condition that never gets set (the 0xCF0 SIO
spin / blue screen). Reproduced on Tomba's Village of All Beginnings: a native
0x000E7000-region shard function entered at a foreign PC wedged; the interpreter
ran the same bytes correctly.
Fix: the generated CPS entry-switch for OVERLAY functions (config_.overlay_mode)
now FAILS CLOSED on an unknown non-zero continuation PC — it restores the PC and
calls psx_native_bad_entry instead of running from the top. overlay_loader_dispatch
consumes g_native_bad_entry and routes the PC to the sanctioned dirty-RAM
interpreter (the live bytes run correctly — NOT a stub/HLE). True-prologue entry
still runs from the top. Static (BIOS/boot-EXE) functions keep the legacy fall-
through. ABI bumped 5->6 so stale ABI-5 caches are rejected and rebuilt with the
guard.
Minimal port: kept only the fail-closed mechanism + its ABI plumbing. The Tomba2
host_aliases interior-routing and the native-block / CPS-probe diagnostics (which
depend on the overlay-cache-v2 multi-range Candidate fields not on master) were
intentionally omitted — the fail-closed default alone is the safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two framework bugs that made overlay variants run interpreted forever (the
"agonizing lag" in overlay-heavy areas like Tomba's dwarf village):
1. Cache read/write divergence. The loader reads <exe>/cache and the runtime
writes captures to <exe>/overlay_captures.json, but the gcc autocompile path
used the game.toml command verbatim, whose hard-coded --out-dir/--captures
pointed elsewhere (build-stable/build-modern). So compiled shards landed where
the loader never looked and captures were read stale -> the variant never went
native. Fix: the runtime now injects PSX_OVERLAY_CACHE_DIR / PSX_OVERLAY_CAPTURES
(the canonical loader paths) into every autocompile spawn; compile_overlays.py
honors them over any CLI flag (--captures now optional). One source of truth for
the cache location, uniform across all games, dev or prod. gcc/tcc stay separated
by the existing <compiler> namespace.
2. Overlay DLL preamble missing psx_ws_cull_sltiu. auto_screen_x emits this helper
at every width-cull site into EVERY build's C (identity at 4:3), and the main EXE
links it from gpu.c, but the overlay DLL preamble never defined it -> any overlay
variant with cull sites failed to link -> no shard -> interpreter. Fix: provide
the self-contained helper (depends only on the already-forwarded psx_ws_x_margin)
in the DLL preamble.
Proven on Tomba dwarf village: dispatch_native 29/s -> 11454/s (was ~97% interp,
now ~97% native).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compiler-neutral tier-selection policy extracted from the mis-named
overlay_sljit.{c,h} into NEW overlay_backend.{c,h} (OverlayBackend enum +
resolve/name/active). overlay_sljit.{c,h} is now just the deprecated emitter.
New tcc tier. gcc and tcc run the IDENTICAL pipeline (recompiler -> C ->
compiler -> DLL -> loader); only the compiler binary differs, so both map to the
autocompile-spawn provider (code_provider.c) and the COMMAND is chosen by the
resolved backend (main.cpp). compile_overlays.py gains --compiler gcc|tcc: tcc
writes a separate tcc/ cache namespace, BOM-strips the C + a copy of the include
headers (tcc 0.9.27 can't skip a UTF-8 BOM), and uses tcc's built-in linker.
Backends (overlay_backend): auto (gcc if a gcc TOOLCHAIN is present, else tcc --
was sljit), gcc, tcc, auto-no-gcc (dev/test: force tcc even with gcc present, to
exercise the toolchain-less user path), sljit (DEPRECATED, gated off, lowest).
Priority: static > gcc shard > tcc shard > sljit(off) > interp. The loader scans
gcc/ then tcc/ and dedups by region+crc (filename) so a gcc shard always wins a
tie.
Self-contained tcc: with no explicit overlay_autocompile_cmd_tcc, the runtime
CONSTRUCTS the tcc command from a toolchain bundled beside the exe
(<exe>/overlay_toolchain/: embedded python + tcc + recompiler + compile_overlays.py
+ runtime headers) -- no system python or gcc needed. config_loader gains
overlay_autocompile_cmd_tcc (a dev override). autocompile cmd buffers enlarged for
the constructed path-heavy command.
Compilation stays OFF the dispatch/render thread (off-process autocompile
subprocess, same model as gcc); the main thread only spawns it and dlopens the
finished DLL.
Validated: MMX6 with overlay_backend=auto-no-gcc fills the tcc/ namespace live
from the bundled toolchain and plays through save-load + gameplay -- the exact
path the sljit tier softlocked on. The gcc path is untouched.
Runtime-only, no regen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Off-thread sljit compile worker (overlay_compile_worker.c/.h): on a dispatch
gap-fill miss the dispatch thread snapshots the region bytes and ENQUEUES; a
background worker JITs from the snapshot (never live RAM) and publishes the shard
to the on-disk cache via an atomic temp+rename; dispatch picks it up on a later
miss via an idempotent rescan, so the candidate table stays single-threaded. This
removes the synchronous JIT frame hitch. Validated on Tomba + MMX6: compiles
off-thread (one MMX6 compile was 26ms, paid entirely off the dispatch thread with
no frame-pacing spike), shards persist + reload, queue never backs up.
Observability (ring-buffer doctrine): always-on recent-compile ring + per-compile
timing in the worker, the resolved on-disk cache dir, and a tier-enabled flag, all
surfaced via a new sljit_async TCP debug command (documented in TCP_COMMANDS.md).
DISABLE the sljit tier by default (g_sljit_tier_enabled = 0): the sljit EMITTER
mistranslates some MMX6 overlay instruction(s) -- sljit-only save-load softlocks/
loopbacks while PURE INTERP works (discriminator), and it reproduces with the
worker OFF in the synchronous path (PSX_SLJIT_SYNC=1), so it is the emitter, not
the async worker. Until the emitter bug is found + fixed: no shard READ
(scan_sljit_cache_dir skipped), no GENERATE (worker not started + sljit_live
forced 0 + dispatch gap-fill gated), every overlay miss falls to the interpreter.
gcc (DLL cache + autocompile) and overlay capture are untouched, so priority is
gcc > interp and dispatch misses are still recorded for future gcc coverage. gcc
shards are bundled in release zips, so shipped builds are unaffected.
PSX_SLJIT_ENABLE=1 re-enables the tier for emitter-fix testing.
Runtime-only, no regen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the old incomplete RAM-only fast_boot snapshot with a COMPLETE
hardware save-state captured at the BIOS->game handoff. First launch (and the
first launch after any update) boots the real BIOS normally and captures the
full machine state at the game's PS-EXE entry_pc; every subsequent same-build
launch restores it and presents the game's first frame instantly.
This is not HLE: it persists the real hardware state a real BIOS run produced.
Format (boot_state.h v2): a sectioned file with an integrity key
(bios_checksum + entry_pc + PSX_OVERLAY_CODEGEN_HASH + ABI tag + codegen ver).
Any mismatch -> reject -> normal boot + recapture, so a user/app update can
never silently load a stale snapshot. Missing/short/unknown sections also reject
(no partial restore).
Complete capture (one section per subsystem), the fix for the old snapshot's
real bug (it left GPU/SPU/CD/DMA/SIO at reset while CPU/RAM assumed configured):
- CPU regs, 2 MB RAM, 1 KB scratchpad, IRQ (i_stat/i_mask), root timers, guest
cycle counter, dirty-RAM page bitmap (guest-written code pages).
- GPU: display/draw-area/offset/mask/texpage + GPUSTAT bits + GP0 command
machine + VRAM-transfer state; VRAM (1 MB) via the renderer-agnostic
gr_vram_transfer_out/in hooks (so it is correct on the GPU-authoritative GL
backend, where the FBO -- not gpu.c vram[] -- is truth, and the FBO re-upload
on restore happens for free).
- SPU: register file + 24 voice decode/ADSR state + latches + 512 KB SPU RAM.
- CD-ROM: full controller FSM (regs/IRQ-latch/FIFOs/seek/read/pending/queued +
XA decoder history + disc-speed model).
- DMA: channels[7] + dpcr/dicr + async-transfer state.
- SIO: registers + pad-config FSM + memcard FSM (incl. per-slot mc_slots[]) +
cycle-paced shifter/ACK scheduler.
Host-side/recompiler-derived state (overlay/sljit tables, code-provider) is NOT
serialized -- it re-initializes fresh on the normal startup path.
Per-subsystem accessors use an X-macro field list so bytes/write/read can never
drift. Runtime-only; no recompiler/codegen change, no regen. Reuses the
[runtime] fast_boot key (semantics upgraded; type unchanged).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comprehensive 3-way merge of the dev/ape-escape branch into master for
the isolated wt/ape-escape worktree. Brought in every complementary ape
change automatically (dma.c, memory.c interrupt byte/half lanes,
function_analysis discovery, main_bios vector seeds, iso_reader
iso_sector_count, config_loader, crash-trace fields, headers). Resolved
the 13 conflicts by rule:
- CPS dispatch / recompiler (code_generator, full_function_emitter,
dirty_ram_interp, traps, main_psx) -> KEEP MASTER (evolved system).
- CD model (cdrom.c) -> take ape (the model that booted Ape).
- interrupts.c/.h -> reverted to MASTER (ape's interrupt-delivery is
dispatch-coupled; the merged Frankenstein stormed).
- sio/debug_server/freeze_heartbeat/main.cpp -> keep master (evolved).
- crash_trace.c -> master (ape freeze-dump fns omitted to avoid the
debug-ring cascade).
STATUS: does NOT boot yet. ape-cdrom + master-interrupts derails to
PC=0 at frame ~2092 (insta-close); ape-cdrom + ape-interrupts storms
(297M exception re-entries) at frame 2077. Root finding: ape's CD
anti-storm is coupled to ape's interrupt-delivery + call-depth + EPC
model, which master replaced with CPS continuation-passing, so the CD
subsystem does not forward-port as pure hardware simulation. Preserved
as the reference state for a design consult on bridging the two models.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The launcher Hybrid controller mode (built first, for Tomba, to mimic Tomba:
Special Edition) flips the emulated pad's reported type 0x41<->0x73 on
d-pad<->stick. After the real DualShock config state machine was added for MMX6
(98aa688), that flip regressed every game's libpad re-detect into a 1-frame
"pad unplugged": Tomba unpauses its menu / drops input, MMX6 phantom-dashes.
The modern config SM is correct and required by MMX6, so instead of weakening it
globally, restore the pre-98aa688 "config commands always answer 0xF3" behaviour
as a per-game opt-in: [controller] legacy_pad_config (default OFF). Only Tomba
sets it; with the flag off the modern pad path is byte-for-byte unchanged, so no
other title is affected. A/B verified live: legacy = clean Tomba flip / MMX6
input; modern = Tomba disconnect + MMX6 dash.
Explicitly LEGACY: a correct findpad/re-detect refactor would make the flip
benign for every game with no per-game branch (real hardware tolerates unlimited
analog-button presses); when that lands this whole feature set is to be removed.
Full rationale in sio.c (g_pad_legacy_cfg).
Also includes the deferred coherent pad-type apply (pad_type_req, applied only at
an idle/non-config bus boundary so a flip can't split a poll/handshake) and a
`pad_cfg` debug command + tools/padcfg.py + tools/sio_flip_trace.py used to A/B
the config SM live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 16:9 BG widen draws LEFT extra tile columns each side, but the engine's
streamer (FUN_800273e4 -> FUN_800274a0) only refreshes one column per side per
frame at the leading edge. On scene entry / fast scroll the inner reveal columns
held stale tiles from a prior scrollY -> real-but-misplaced tiles spilling
up-and-left (the "staircase"). Census confirmed this is NOT the BG packet-budget
overflow (674-816 prims/frame, well under the 1000-tile cap).
Fix (render window == refresh window): clone FUN_800274a0's inner metatile->ring
fill into mmx6_fill_column() and re-stream every column of the widened window for
all 3 layers each frame (psx_ws_mmx6_bg_refill_all), triggered once/frame from the
existing count_site hook (mmx6_bg_refill_tick, frame-gated). Uses each layer's own
scroll, omits the engine's scrollX wrap-bookkeeping writes (those drive the engine's
own incremental streamer), skips map-loop seams; off-map columns fill tile 0 (the
renderer skips them). Validated byte-exact against the engine's own streaming
(0 mismatches / 1113 cells x 3 frames; debug cmd mmx6_freshfix {validate:1}). Gated
native-wide <=384w so 4:3 and the 512 hi-res title stay byte-identical.
Also removes the 2-col reveal cap (WS_FULL2D_MAX_REVEAL): it never addressed the
staircase and cost screen real-estate; the full 53px/4-col reveal is restored.
Parked, gated-off experiments ride along (inert, default off): host-side reveal
(Attempt A: gpu_ws_mmx6_emit_reveal + sw_wide_emit_tile) and BG packet-buffer
relocation (Attempt B: [widescreen.bg2d] bufbase/cap recompiler hooks). These target
the separate dense-stage budget-overflow concern, not this fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The BG renderer widen (a8f37c1) draws extra tile columns on both sides, but the
engine's 64-column tile ring is only refreshed at its leading edge: the streamer
(FUN_800273e4) re-streams ONE column per side per frame at world-X scrollX-16
(left) and scrollX+16+width (right). So the columns the widened renderer reaches
were never populated — they showed empty (black margin) or stale (old tiles as
the camera scrolled), the "stale margins" the user saw in motion.
Add two more [widescreen.bg2d] sites (stream_left_site / stream_right_site) that
push the streamer's left/right leading-edge world-X out by LEFT*16 via gpu.c
psx_ws_mmx6_bg_stream_{left,right}, so the incremental streamer keeps the ring
valid across the widened window as the camera scrolls. Same number of streams
(the single leading-edge target just moves out LEFT columns) — no extra cost.
Identity at 4:3 / 512 hi-res.
Verified: the previously-black left reveal margin now fills with coherent
adjacent stage; blackL/blackR = 0 across scenes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure-2D sprite games (Mega Man X6) render a true 4:3 field of view with no
overscan to reveal, so 16:9 requires making the engine draw more. Two parts:
- [widescreen] full_2d opts a pure-2D game into the native-wide present path
(it never emits the 3D sprite-tag hook the gameplay detector keys on, so
every in-game frame was misclassified as a 4:3 menu and pillarboxed).
- [widescreen.bg2d] is a gen-time hook on the per-layer background tile
renderer (MMX6 FUN_800270d0) at its column-count / start-tile-column /
start-screen-x instructions, routed through gpu.c psx_ws_mmx6_bg_{cols,
startcol,startx}. It draws LEFT=ceil(nw_offset/16) extra 16px columns on
BOTH sides so the loop fills the 16:9 reveal margins with real adjacent
stage, existing columns staying pixel-aligned with objects. Identity at
4:3 and in the engine's own 512 hi-res mode (so the title is untouched).
Tooling: wide_full debug command + sw_wide_dump_full dump the entire
native-wide compositor surface (both vertical-double-buffer y-bands and both
reveal margins) for diagnosis.
Known follow-ups: object screen-X cull widen (actors in the margins);
top/bottom flicker + stale-margin refresh on the vertical double-buffer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The manual PSX_OVERLAY_CODEGEN_VER must be hand-bumped on every emitter change;
when it isn't (e.g. between CPS emitter iterations), a stale-but-cgN DLL is reused
-> the v0.3.0 black screen. Now hash_codegen.cmake hashes the recompiler codegen
sources into a (gitignored) overlay_codegen_hash.h at build time; overlay_api.h
exposes PSX_OVERLAY_CODEGEN_HASH and the cache path becomes gcc/<arch-abi>/cg<N>_<hash>/.
The loader and compile_overlays.py read the same value, so any emitter change
auto-invalidates the cache (old dirs coexist, no stale load). Falls back to hash 0
when the header is absent (fresh checkout) — identical to today's single-cgN path.
Verified: build emits hash 0x347751c1; autocompile + loader both use cg4_347751c1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A persisted option value outside its declared [min,max] is now REJECTED at load
(the game's own default is used instead), so a corrupt/stale options file can no
longer inject an out-of-range value into a config global the game may use as a
jump-table index (wild jump = softlock). Ranges declared per option in
game_options.toml (min/max; absent => no validation, e.g. signed screen offsets).
Adds a format_version header (newer-than-known files are rejected wholesale).
Restore stays name-keyed (offset-drift-safe). Runtime-only, no regen.
Validated: corrupt values rejected (defaults applied); valid file restores intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vulkan backend alongside software + OpenGL: geometry/textured-prim
pipelines, GPU-authoritative VRAM image, deferred-submit + drain-at-present
perf model, GPU-direct present (vk_renderer_present_vram), and a
full-VRAM-pack readback path (ensure_cpu rect_adds the whole VRAM rect
before pack_flush so screenshots/guest VRAM reads reflect live content,
not just the incremental pack-dirty rect — readback-only, no per-frame cost).
Runtime plumbing: --debug-port / --renderer / --window-title CLI overrides
(multi-instance fleet), latency_ring, SPIR-V shaders + embed_spirv.py,
config_loader renderer selection, debug_server screenshot wiring.
All on feat/vulkan-renderer; snapshot in backup/vulkan-wip-20260617.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).
Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.
sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)
gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
inside a registered candidate's code range (a tail-transferred call's continuation),
overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.
Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.
VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).
KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.
Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
tail-transfer (jalr captures rs before the link, alias-safe); each call's
return point registered as a dispatchable continuation; entry-switch (keyed on
cpu->pc) at the top of every function AND every overlapping-alias body. 0
psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
(case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).
Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
(un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).
Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oracle disambiguation (RECURSION_BUG.md §23–§24). Traced one func_8001A954
invocation on real HW (psx-beetle) vs the recomp in the IDENTICAL Tomba idle
state (New Game village dialogue; object 0x801FD950, state/scene/jumptable=0).
VERDICT: real HW's per-frame chain is BALANCED — func_8001A954 is entered once
per frame from main_loop (JAL 0x8001A630, sp=0x801FE3E0) and RETURNS once per
frame (jr ra @0x8001A9E8 -> 0x8001A638, SP fully restored); no re-entry, the
overlay 0x80063864 never calls back (refutes §21/§22's "call-cycle"). The recomp
runs the same guest flow and the guest SP unwinds identically, but the HOST stack
leaks +1.17 KB/frame (ce_profile, live-reconfirmed: 20099->22215 KB / 1807 frames)
because the per-frame compiled chain (func_8001A954->AC00->B2B4->B5A8->func_80046264,
emitted as DIRECT func_X(cpu) host calls) re-enters itself via func_80046264's
CD-DMA-dirty hop and never unwinds.
native_stack early-trip (PSX_STACK_GUARD_KB=6144, frame 7592, 6 MB) pins the
leaking host cycle. Cross-checked: xprobe mixed_depth bounded at 6, dirty-insn/
dirty-block show NO literal func_8001A954 transfer, and a954=1/frame is a SURFACED
guest tail-transfer (j/jr) to func_8001A954 that the oracle proves real HW never
makes — i.e. there is both (A) a correctness divergence (spurious per-frame
re-entry) and (B) the architectural host-mirroring limitation that amplifies it.
Fix scope CORRECTED from §23's "quick contained runtime patch": the UNIVERSAL fix
is §11 — route in-range guest calls through the depth-managed psx_dispatch_impl
trampoline instead of mirroring them as direct host calls, so the host stack stays
bounded regardless of guest re-entrancy (real or fabricated). §12 (CD-DMA dirty
precision) and fixing the specific divergence are surgical mitigations, not
universal. Performance loss accepted for now (static recomp: right before fast).
Instruments added (runtime-only, no regen):
- SP capture in the fntrace control-flow ring on BOTH backends (GPR[29] read live):
psx-beetle (beetle_libretro.cpp + beetle_debug_server.c) and the recomp
(fntrace.h/.c + debug_server.c) — enables the func_8001A954 SP/RA-lifecycle oracle.
- psx-beetle default debug port 4380 -> 4382 (+ --port N override): 4380 collided
with psxref and cdirecomp/CdiRuntime.exe; two LISTENers on one port silently
misroute connections (the phantom "oracle keeps dying" early in the session).
Oracle tooling: the control-flow oracle is psx-beetle (Beetle interprets -> records
every J/JAL/JR/JALR), NOT psxref (no PC trace; the assumed emu_step/emu_trace_addr
never existed). Loads Tomba via: psx-beetle.exe --disc tomba.cue <bios> --port 4382.
Next: implement the §11 universal fix (codegen + full regen).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Matthew Stanley