Triage of Tomba2Recomp GH issue #1 exposed three ways a release exe
behaves differently on the build machine than on every user machine:
- DEFAULT_BIOS_PATH baked an ABSOLUTE build-tree path into the binary,
so promoted/release exes silently load the builder's BIOS wherever
that path exists and the clean-install picker flow is never
exercised where releases are validated. runtime.cmake now refuses
absolute values (loud configure warning, converts to the relative
default); dev checkouts keep prompt-free boots via a new
resolve_bios_path rung that walks exe-dir ancestors for
psxrecomp-v4/<relative>, a layout only dev trees have.
- A settings.toml with a TOML syntax error was silently discarded
wholesale (user's renderer choice, BIOS/disc paths - everything),
and the next launcher save then overwrote the file with defaults.
load_user_settings now reports parse_error; the runtime preserves
the broken file as settings.toml.bad and warns via message box +
stderr.
- package_release.ps1 gains guards: fail packaging if the staged exe
contains a drive-letter absolute SCPH1001.BIN path, if the stage
carries BIOS/disc/save/sidecar files (SCPH*.BIN, *.cue, *.iso,
*.mcd, bios.cfg, disc.cfg, settings.toml, keybinds.ini,
overlay_captures.json), or if saves/ is non-empty.
Verified: rebuilt Tomba2 against this tree - configure warning fires
on the absolute arg, strings scan of the exe finds no drive-letter
BIOS path, and a boot with no [bios] setting and no bios.cfg resolves
the BIOS through the psxrecomp-v4 dev rung.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The offline cache build ran one capture at a time - classify, recompiler
subprocess, patch/audit, gcc - so a full coverage-vault rebuild (Tomba 2:
1293 captures) took hours. The wall clock is subprocess-dominated and the
subprocesses release the GIL, so a ThreadPoolExecutor across captures
fills the machine: measured ~24x on the capture-reprocess phase
(6/min -> ~143/min, 24 concurrent recompiler/gcc children).
Concurrency contract:
- cov_lock guards the per-region dedup coverage sets. The
check->build->update window is deliberately unlocked: concurrent
captures may build overlapping DLLs - redundancy, never corruption
(the loader content-matches every function by (entry, code_crc)
across all DLLs at a region).
- A per-(region, crc32) key lock serializes captures of IDENTICAL bytes,
which share one dll/ranges filename: the second sees the first build
(dll-exists skip / prior-ranges merge) exactly as it would sequentially.
- The interior-fragment pass still runs after all captures, reading the
final on-disk entry coverage (unchanged failure-domain decoupling).
- Per-thread stdout capture prints each capture log as one atomic block.
- --jobs 1 and --static take the exact sequential path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The controller map had no l3/r3 rows at all, so pad bits 1/2 could never
be produced by a gamepad — DualShock-only games (Ape Escape) were
unplayable. Keyboard slots existed but shipped unbound.
- main.cpp: PAD_L3/PAD_R3 bits; controller_map 14->16 entries; defaults
l3=leftstick, r3=rightstick; input.ini template + doc comment updated.
- psx_keybinds.c: keyboard defaults L3=T, R3=Y (continue the Q/W/E/R
shoulder row). Launcher Controls page picks them up automatically.
- beetle_main.cpp: same T/Y keys on the oracle so psx-beetle stays
drivable for L3/R3 games.
- tools/package_release.ps1: shipped input.ini template gains l3/r3.
- README: keyboard + controller map tables.
Wire is unchanged: sio.c transmits the button word verbatim, matching
Beetle's dualshock.cpp. Validated in-game on Ape Escape (user-confirmed
R3/L3 inputs on a real pad).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two hardware-faithfulness fixes found by the always-on tap rings + Beetle
oracle A/B on fingerprint-matched attract music (tools/audio_pair_segments):
the recomp played the identical track at -3.8 dBFS vs the oracle's
-14.1 dBFS — +10.3 dB hot, clipping continuously.
1. spu.c: the voice repeat/loop address register (voice reg 7) is LIVE
state on real hardware — writing it after KEYON retargets where the
next END+REPEAT block jumps (Beetle spu.cpp:1150/333), and the block
auto-latch (flag bit 2) updates the guest-visible register (386/1302).
The recomp cached repeat_addr at KEYON only, so X5's driver idiom —
KEYON, then point the loop register at a silent tail block; no KEYOFF
ever — left voices looping the LOUD sample body forever. Measured: 3
stuck voices re-looping ~1400x/s, 160k SPU events/min vs the oracle's
633, zero game KEYOFF writes (the game never needs one), +11 dB.
2. cdrom.c: implement the CD controller's decode volume matrix
([src][port], 0x80 unity, pending banks + apply latch at 1F801803.3
bit 5 — mapping copied from Beetle cdc.cpp:1483-1501, applied after
the XA resample per PS1 hardware tests). X5 drives it constantly:
0x73-0x7E steady, ramped to 0x00 at scene changes (measured via the
new beetle `cdc_volume` command + PS_CDC::PSXRecomp_GetDecodeVolume
peek). Without it music fades never happened and steady level sat
~0.6 dB hot.
Also: AUDIO_TAP_VOICES (T0 voice-sum tap) for source-by-source mix
decomposition, per-tap WAV rates, bridge stats in audio_stats.
Measured after (MMX5 attract, 20-25 s windows):
spu_out RMS: -3.8 dBFS (clipping) -> -15.5 dBFS [oracle -14.1]
music-only chain already matched the oracle at -14.8 dBFS, confirming
decoder + mixer gain equivalence; the entire excess was stuck voices.
underruns: 2 events / 98 s (delivery fix holding).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measurement infrastructure for the audio-quality campaign (MMX5 complaint),
ported from snesrecomp's audio_trace/oracle methodology and the
_audio_round2 tap model. No behavior change to the audio path itself.
- audio_trace.{c,h}: three always-on 95 s PCM rings — T1 spu_out (canonical
spu_render output), T2 cd_in (XA/CD-DA PCM entering the SPU CD bus),
T3 host (exact post-fade bytes handed to SDL_QueueAudio) — plus a 512k
event ring (every SPU reg write stamped with the SPU sample clock;
RENDER/PUMP_SKIP/UNDERRUN/MUTE/UNMUTE/CD_PUSH/SPU-DMA), audible-vs-
silent counters (-42 dBFS gate), queue watermarks, and a server-side
44100 Hz WAV dumper (rate exactness matters: snesrecomp da33c06).
- psx-beetle: the libretro audio callbacks stop discarding Beetle's fully-
mixed reference SPU PCM and record it into the same tap ring. Port 4380
gains audio_stats/audio_wav/audio_events with the identical wire protocol
as the runtime (rule 16 symmetric harness) — the oracle WAV capture.
- debug_server: audio_stats / audio_wav / audio_events on the runtime port.
- tools: audio_ab_diff.py + audio_spectral_ab.py (drift-tolerant offline
A/B: drift ppm, onsets, cents, timbre, click rate) and audio_analyze.py
(first-dirty-tap cross-tap decision tree) ported from snesrecomp/
_audio_round2; audio_capture_ab.py dumps both processes symmetrically.
Probes query history; nothing is armed at probe time.
Games that discard their boot/init code load gameplay overlays OVER the
main-EXE text window (Tomba 2: hot code at 0x1DA88/0x24548/0x3116C, below
its 0x38800 floor). Post-baseline, a DIRTY page in [KERNEL_WINDOW_END,
FLOOR) is by definition live overlay code the compiled image is stale for,
so make overlay_cache_window_contains() admit dirty boot-text pages as a
capture/candidate window instead of forcing ~300ns/insn interpretation.
Auto-capture convergence: replace the fixed trigger gate with
insn-pressure sizing and exponential futility backoff — retry only when
the manifest signature or loader candidate count actually changes, so
coverage stops churning once it has converged. compile_overlays.py moves
to entry-based coverage; autocapture.c escapes >=0x80 cp1252 bytes from
child-compile output so the status JSON stays valid UTF-8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds full 16:9/21:9 native-wide support for fully-3D titles (Ape Escape,
SCUS-94423) as a generic, opt-in, 4:3-byte-identical framework capability, plus
a native-wide GL compositor perf fix and a cutscene present-mode flicker fix.
Widescreen (generic, per-game opt-in via [widescreen*]):
- Shared screen-extent cull detector (runtime/include/ws_cull_detect.h) used by
all three backends (recompiler, dirty-RAM interp, sljit). Per-game signature
immediates ([widescreen.cull] screen_w_imms/screen_h_imms) — Ape's funnel
tests 0x181/0xF1 on its 368px display, not Tomba's 0x140/0xE0.
- Three funnel idioms widened: unsigned per-vertex sltiu (psx_ws_cull_sltiu),
signed min/max slti right edge + bare-bltz left edge (psx_ws_cull_slti/bltz,
new helpers + SLJIT_HLP appends), and billboard center+/-halfwidth. Explicit
[widescreen.cull] slti_sites for X-only funnels the auto-detector can't
qualify.
- gte_game_mode: GTE-activity gameplay detector for 3D titles with no sprite-tag
helper (gte.cpp notes every RTPS/RTPT). STICKY (>=3 verts, 45-frame hysteresis)
so sparse cutscene frames never flip to a 4:3 pillarbox = fixes intro flicker.
- nw_hud_corners: push outer-third screen-space HUD sprites to the true wide
corners in native-wide (else inset by the reveal).
- nw_backdrop: stretch a full-frame axis-aligned 2D backdrop quad to fill the
wide frame (for titles with a flat sky plane).
- Pattern-scanned interp/sljit widen hooks now gated per-game (auto_screen_x /
auto_backdrop): fixes an ungated auto_backdrop false-positive that rewrote a
live GPR = the ~24s native-wide crash on Ape.
Native-wide GL perf (fixes 16:9 frame drops):
- The wide surface centre equals the canonical 4:3 framebuffer, so wide_blit_center
copies canonical->wide-centre at present (both present paths) and the per-prim
mirror SKIPS prims fully inside the 4:3 frame (mirror_x/geo/batch_center_only).
Only reveal-margin prims re-rasterize. Peak frame time 38.8ms -> 17.5ms, no
vsync misses. Toggle gl_wide_fast; 4:3 untouched (all gated on g_wide_cur).
Runtime-only except the recompiler cull emit (regen needed for cull widening).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace absolute F:/C: paths with repo-relative (../TombaRecomp, bios/, beetle-psx/)
or placeholders (<desktop>/<psxref>/<nesrecomp>/<psxrecomp-v4>) across accuracy docs,
WIDESCREEN/SHADOW/RECURSION_BUG/config_schema/ISSUES/README, and tools/*.py|.ps1.
Removed a personal C:\Users\Matthew\Desktop path (ISSUES.md). Left CLAUDE.md/PLAN.md
sibling-project + version-history paths intact (intentional layout documentation).
PR #9 makes our CD controller more faithful than Beetle (SCPH-1001 version
94/09/19/C0 vs Beetle's hardcoded 97, ADPBUSY idle=0, disc-derived GetID region).
devtrace_diff.py now LABELS a cdrom-source native/beetle delta as KNOWN intentional
(more faithful) instead of red-flagging it; --strict shows raw. Beetle stays the
independent oracle. Documented in ACCURACY_BURNDOWN.md Axis 5.
merge_cache used a flat listdir() against a nested cache
(<compiler>/<arch-abi>/cg<N>_<hash>/) so it matched nothing — the vault's
DLL mirror has been empty since the tool shipped. Walk the tree and keep
relative paths: the same content-keyed filename exists under different cg
dirs with different compiled bytes, so flattening would mix emitter
generations. stats() walks recursively to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The emitter routes mfc2/cfc2/mtc2/ctc2 on flag/IR/derived GTE registers
through direct gte_read_data/gte_read_ctrl/gte_write_data/gte_write_ctrl
calls (code_generator.cpp COP2 case), but the overlay DLL preamble had no
shims for them — any GTE-heavy overlay failed to LINK (undefined
gte_read_ctrl) and its region ran interpreted forever. That silently
blocked all new native coverage for the lava/Trolley attract-demo regions
after the caches rotated.
GTE state is shared runtime state, so the shims forward unconditionally
through OverlayCallbacks (a local copy would fork the flag/IR timeline).
ABI bumped 9 -> 10; the loader gate rejects+deletes pre-shim DLLs and
autocompile self-heals; plain gte_data[]/gte_ctrl[] accesses stay direct
CPUState reads in emitted code, matching the emitter.
Smoke-validated: previously-failing GTE-heavy captures now compile clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the fresh-launch logo freeze (present at every commit in the
stack; window-capture-verified 13/13 repros, never a regression): the BIOS
dispatch normalize() maps RAM phys 0x30000-0x5AFFF to the shell ROM alias.
The Class-A fix routed post-game-start shell-window targets to the game/dirty
path first, but its gate also required psx_game_address_in_text() — which
exempts runtime-loaded OVERLAY code above the boot-EXE text end. Tomba2's
text ends at 0x38800; its post-logo main-thread resume PC 0x50B80 fell
through to normalize() and executed func_1FC38B80, a stale shell-image
data-as-code self-loop, starving the main thread forever (one VBlank
delivery per frame at the same EPC, zero main-thread instructions retired;
the game's ISR ran fine, so TCP frame counters kept ticking — the freeze
was invisible to every telemetry-only validation). Even PSX_FORCE_INTERP
missed it: normalize() shadowed the address before the interp path saw it.
Fix: post-game-start, the WHOLE shell-overlap window routes to
dirty_ram_dispatch before normalize() — real hardware executes the RAM
bytes regardless of who loaded them (boot EXE, overlay loader, DMA stream).
Pre-start shell addresses still normalize to the shell as before.
Also:
- interrupts/debug_server: nestgate_* per-guard veto-attribution counters on
the nested-delivery gate (freeze_check), used to exonerate the IEc/SR
guards and redirect the hunt from interrupts.c to dispatch.
- tools/window_capture.py: PrintWindow-based SDL-window capture with
LIVE/FROZEN series verdict — window-liveness ground truth; TCP/VRAM
screenshots provably lie about presentation (this bug's entire history).
Validated: fresh launch now runs logo -> intro FMV -> title -> attract with
sustained full-frame motion (window-level capture series).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
platform.system() under an MSYS/Cygwin python returns 'MSYS_NT-...', not
'Windows', so cache_arch_abi() filed every compiled overlay DLL under
gcc/linux-x64/ while the Windows loader reads gcc/win-x64/. Result: the
loader found no native overlays (interp forever), while autocompile kept
checking its own linux-x64 dir and reporting 'already covered — no new
native code to build'. Measured on Tomba2 attract: ~14M interp insns/s,
dirty dispatches 534-695/s, 55-57fps / 91-95%% guest speed. After fixing
the tag (+ migrating the misfiled DLLs): 58-60fps, 33.8M cycles/s = 100%%
speed, dirty dispatches down 3x. Also fixes the -fPIC-on-Windows flag
picked by the same broken check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- [video] fmv_skip_no_xa: broaden FMV detection to MDEC activity alone.
Silent RAM-preloaded movies never stream XA, so the MDEC+XA detector
missed them and no skip mechanism was even attempted. Per-game opt-in.
- tools/host_thread_probe.py: suspend/sample RIP+RSP of every host
thread of a live or wedged runtime, stack-scan return addresses into
the main exe, addr2line-symbolize (RelWithDebInfo). Diagnosed the
fatal-halt-serve loop in minutes; use whenever a runtime 'freezes'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the Tomba pause-menu wedge (loaded saves only): autocompiled
overlay shards were emitted by a recompiler binary OLDER than the last emitter
change, yet stamped with the CURRENT cg tag (compile_overlays derives the tag
from --runtime-include; nothing tied it to the emitting binary). The stale
shards corrupted the display-list task queue during load-game. NOT the
IRQ-resume class the prior handoff claimed - disproven with the new ring.
- recompiler: bake the emitter-source hash (shared canonical list
runtime/codegen_hash_sources.cmake + hash_codegen.cmake) into psxrecomp-game;
new --codegen-hash flag prints it (machine-parseable, pre-banner).
- compile_overlays.py: HARD-FAIL when the recompiler binary's baked hash
differs from the cg tag hash it stamps (or the binary lacks the flag).
Applies to both cache and --static modes. Closes the read==write-tag /
stale-content drift class structurally.
- runtime.cmake: codegen-hash source list moved to the shared canonical file.
- interrupts.c: always-on exception-EXIT half added to irqctx_ring (take_pc,
real_epc, exit_pc, escape reason, same-thread/restore flags, v1/ra
saved-vs-exit, RestoreState redirect count) - proves per delivery which
escape path fired and whether the interrupted GPRs were restored. Kept the
same_thread_resume GPR-restore refinement (ring-verified equal-value in
practice; faithful: hardware restores all GPRs on every exception return).
- debug_server.c: irqctx_ring dump extended with the exit-half fields.
Validated live: Tomba menu opens on loaded save (Stormy Mountain) with the
cache fully enabled after regenerating shards with a rebuilt recompiler
(build-cosim + build-prod caches). Guard verified positive and negative.
MMX6 gate passed: boot -> title -> opening cutscene -> gameplay handoff OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cycle-accuracy emitter emits calls to 12 timing runtime functions
(psx_cyc_load_*, psx_icache_fetch, psx_muldiv_*, psx_gte_*, psx_slice_block,
...) + debug_server_cyc_observe + g_debug_last_store_pc. Overlay DLLs never
got these symbols, so every overlay failed to link -> 0 native shards ->
100% interpreter (unplayable). Fix (option A): expose the 12 via the overlay
callback ABI.
- overlay_api.h: +12 timing callbacks in OverlayCallbacks, ABI 8 -> 9.
- overlay_loader.c: populate the 12; add warn_on_cgtag_mismatch() (loud
warning when shards exist under the wrong cg-hash).
- compile_overlays.py: 12 preamble forwarders + local g_debug_last_store_pc
+ -DPSX_NO_DEBUG_TOOLS.
- cosim.c/main.cpp: input-cosim setpad + held pad (deterministic input).
- cosim.py: env-overridable COSIM_EXE/CWD/GAME.
KNOWN REGRESSION (patched on a follow-up branch): Tomba pause menu (triangle)
wedges. Root-caused: an IRQ taken mid-block resumes at the function entry
leader instead of the interrupted PC, re-zeroing the menu's VSync wait
counter (scratchpad 0x1F8001E8) every frame so it never reaches its target.
Not the scheduler (LLE reproduces) and not timing math -- interrupt-resume
re-entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New always-on ring (device_trace.{c,h}, compiles into both binaries like
parity_trace) recording every hardware IRQ-raise edge with its guest cycle,
source (I_STAT bit) and detail. All native device sources funnel their raise
through one choke point psx_irq_raise(bit,detail) in interrupts.c (VBLANK/GPU/
CDROM/DMA/timers/SIO routed; DMA passes DICR channel flags, timers pass index),
so the device-timing audit sees every event from one place. devtrace_dump/ctl
debug commands (cycle-window + source filters) + tools/devtrace_diff.py align
the two device-event timelines by source ordinal and flag where cycles drift.
The Beetle oracle side (mednafen IRQ_Assert rising-edge hook) lives in the main
checkout.
This is the level below parity_trace: parity names the divergent control-flow /
gate input; device_trace names the divergent HARDWARE EVENT behind it.
First result (MMX6 cutscene asset-load over-fast wedge): the load's CD+DMA
activity is compressed into ~232M cycles on native vs ~384M on Beetle (~1.65x),
completing the func_8001CB3C scene setup ~100M cycles / ~176 frames early. The
producer is CD-read / DMA-completion timing (DMA channels 2/3/4 = GPU/CD/SPU
asset streaming) -- confirming the device-completion-latency hypothesis and
ruling in the CDROM/DMA cycle model as the over-fast subsystem. (VBlank verified
correct: native VBlank edges are exactly 564480 cyc apart.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends the two-process parity ring so each row carries, per watch word, the
exact store PC + guest cycle + frame + (approx) writing TCB of the most recent
write to that word. parity_trace_note_write() maintains the table from BOTH
processes' write paths (native: memory.c psx_write_* with g_debug_last_store_pc;
Beetle: beetle_libretro.cpp wtrace_callback with the store pc). Both parity_dump
serializers emit the new wwpc/wwcy/wwf/wwt arrays (identical wire format), and
parity_diff.py prints a side-by-side WATCH PROVENANCE block at the divergence.
This turns "the gate-input value diverged" into "who/when produced it" -- the
instrument ChatGPT recommended for the MMX6 over-fast wedge.
First result (MMX6 cutscene->gameplay, watch 0x80071088/0x800CD3F8/0x8006D9AC/
0x800CD3FC/0x800CD404/0x800200F4): the cutscene-setup func_8001CB3C (writer pc
0x8001CB7C of countdown 0x800CD3FC) runs ~99.7M guest cycles / ~176 frames
EARLIER on native (cyc 998,597,937 frame 1769) than Beetle (cyc 1,098,296,890
frame 1942) -- same writer PC, fired early. On Beetle the state struct
0x800CD3F8 is never touched by game code (stays BIOS-init 0 -> setter path); on
native it advances. The over-fast wait is localized one level above the state
machine, at whatever schedules func_8001CB3C.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
End the "somewhere upstream" blindness with a reusable first-divergence
instrument (PRINCIPLES.md "find the first divergence; is the value WRONG or the
behavior MISSING"). A TCB-scoped, freeze-on-trigger control-flow ring with an
identical struct + JSON wire format on both processes, so a generic sequence-diff
(tools/parity_diff.py) locates the first logical divergence between recomp and
oracle — not a frame diff.
- parity_trace.{h,c}: armed-from-boot ring, gated on current_tcb==watched_tcb,
latches on a configured trigger target so the pre-divergence window survives
the wedge loop. Up to 6 watch-word snapshots/row. Self-contained TU shared by
both binaries; frame source decoupled via parity_host_frame().
- fntrace.c: per-dispatch producer hook (cheap armed gate on the hot path).
- traps.c: YIELD control-event hook in the scheduler switch.
- main.cpp: env-armed from boot (PSX_PARITY_TRACE=1; TCB/trigger/watch tunable),
defaults target the MMX6 cutscene wedge.
- debug_server.c: parity_dump + parity_ctl commands.
- tools/parity_diff.py: pulls both rings, sequence-diffs by (kind,pc,target).
First use (MMX6 freeze) confirmed MISSING behavior, not a wrong value: native
thread1 is permanently parked at func_8002000C resume (TCB EPC 0x800200F4),
handshake flag 0x8006D9AC never set (0 across 16384 rows), setter func_8001344C
never runs; thread1 re-resumes a dead func_8002000C frame whose saved-ra slot was
reused -> jr 0x800CD3F8 wedge. Upstream context-staleness/scheduling root.
Beetle mirror producer + the cross-process diff are the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
recompiler/code_generator.cpp: stamp g_debug_last_store_pc before EVERY game
store (SB/SH/SWL/SW/SWR, SWC2, + the ws-backdrop / game-option special paths)
and declare the extern in the game-C preamble. Previously only the BIOS emitter
(strict_translator.cpp) stamped it, so game stores inherited the last BIOS store
pc -> wtrace/readtrace mis-attributed game writes to the BIOS. VERIFIED LIVE:
MMX6's producer trace for the 0x800CD3F8 write now reports the real game store
pc 0x8001A708 (matches Beetle), not the stale 0xBFC11BF0.
runtime: g_pc0_reason slice-exit provenance, fail-closed exception-sentinel
asserts, named thread-event kinds, freeze-heartbeat RVA emission (diagnostics,
default-on, additive).
Carries the in-progress faithful-timing / MDEC work (cpu_state.h, memory.c,
dma.c, interrupts.c, dirty_ram_interp.c, full_function_emitter.cpp) as an
uncommitted-WIP checkpoint. tools/: regen_bios.sh, bios_emitter_fingerprint.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The faithful dequant/IDCT/YUV transcription was correct, but the 15bpp pack did
(c+4)>>3 on `r ^ 0x80` as a SIGNED int. For negative Mask9ClampS8 results that
yields a negative int (r=-1 → -129) → garbage 5-bit channel → B saturated, R/G
doubled = the rainbow fringing on real FMV. Beetle's RGB_to_RGB555 takes uint8
params (truncates to 0..255); the 24bpp path already cast to uint8_t. Fix:
rgb_to_555_chan(uint8_t) so the caller's int truncates on the call.
Caught only by comparing our VRAM to the REAL Beetle process (new vram_peek on
the Beetle side, held-FMV diff) — both standalone pins shared the bug in their
self-transcribed reference and falsely passed. After the fix the per-channel VRAM
delta vs Beetle collapsed from mean (12,16,27) to (0.7,0.9,1.3) and the FMV renders
natural colours. Full-path pin (mdec_e2e, corrected uint8 oracle) is byte-exact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The isolated dequant/IDCT/YUV pin matches Beetle byte-exact on all stages, but the
WIP numeric-only change REGRESSES the live FMV (rainbow fringing; baseline is clean)
because the block-assembly (append_color_macroblock) was not transcribed to the new
block layout. Proper fix needs Beetle's EncodeImage assembly + a full-path pin. See
tools/mdec_pin/FINDING.md. DO NOT MERGE the WIP until the full-path pin + live FMV pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the placeholder region cost (RAM +3 / everything-else +0) in
psx_cyc_readmem with the full Beetle MemRW device-region read-wait table
(libretro.cpp), size-aware:
main RAM (<0x800000) +3
SPU 0x1F801C00-1FFF +36 (32-bit) / +16 (8/16-bit)
CDC 0x1F801800-180F +6 * size
GPU/MDEC/SysCtrl/FIO/SIO/IRQ/DMA/Timers (0x1F801000-113F) +1
BIOS ROM / Expansion-PIO / unmatched / scratchpad +0
Thread the access size (1/2/4) from psx_cyc_load_word/half/byte and
psx_cyc_lwc2_read into psx_cyc_readmem (SPU/CDC waits are width-dependent).
The device wait combines with the existing +2 completion (+1 LWC2) and fudge
exactly as Beetle ReadMemory (LDAbsorb = region + completion). Runtime-only:
psx_cyc_load_* signatures unchanged, no emitter regen.
New ruler #2 loops mmio_timer (Timer0 read -> +3) and mmio_spu (32-bit SPU
read -> +38) added to gen_testrom.py. Validated: cyctest compiled (4600) ==
Beetle (4382) EXACT on all 15 loops incl. mmio_timer +3 / mmio_spu +38; the 13
prior loops unchanged; Tomba 2 boots past the BIOS to its SCEA-Presents intro
splash, no freeze.
Documented residual: DMACycleSteal (dynamic per-read DMA bus-steal) remains
unmodeled — needs the live steal count from the DMA controller; not statically
rulerable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New runtime/src/psx_icache.c: direct-mapped (4 KB / 256-line) instruction-
cache fetch cost, transcribed from Beetle PS_CPU::ReadInstruction:
HIT (TV == addr) -> +0, no give-back clear
MISS, KSEG1/uncached -> +4 (BIOS ROM)
MISS, cached (KSEG0/KUSEG) -> +3, then +1 per word refilled from the
missing word to the line end (earlier words
stay invalid); also clears the load give-back.
Mirrors only the per-word TV tag array (cost + tag state; the value is already
in hand). Wired into the dirty-RAM interpreter (exec_one) per instruction,
charged BEFORE the §1 base — Beetle order.
New ruler #2 loop `icache_miss`: loop top + a victim 0x1000 apart alias the
same direct-mapped line with different tags, forcing a refill miss every
iteration (other loops are all-hits after warm-up).
VALIDATED interp vs Beetle (PSX_FORCE_INTERP=1 PSX_ICACHE=1): icache_miss
native +14 == Beetle +14 (per-iter 17); the other 12 loops unchanged at +0
fetch. So the I-cache hit AND refill-miss costs are MEASURED equal to the
oracle on the interp path.
Opt-in via PSX_ICACHE=1 (default OFF) — charging fetch cost in only one
backend would fork mixed compiled/interp timing. Production activation +
the compiled-path emit (per cache-line leader) + ruler #1 cold-spike
validation are Stage 2. Default-off => no production change (the interp's
fetch call is a no-op without the env).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a force-interp knob (dirty_ram_is_dirty reports all RAM above the
kernel window dirty when PSX_FORCE_INTERP=1) so the dispatcher routes
clean compiled game text through the dirty-RAM interpreter — the SAME
path overlays take — with no emitter/dispatch change. This is the
interp-path Δ-ruler enabler: it lets the cyctest isolation loops be
measured native-INTERP vs Beetle.
RESULT — the dirty-RAM interpreter == Beetle EXACT on all 12 ruler#2
components (baseline/alu/load/load2/load_use/div/div_spaced/mult/
gte_rtps/gte_nclip/gte_read_use/ld_div). The interpreter's per-instruction
load-delay + GTE/muldiv interlock is now MEASURED equal to the oracle and
to the compiled backend, not merely shared-by-construction.
Process notes (cost real time, now documented in the cyctest README):
- launch psx-cyctest via PowerShell Start-Process; a bash '&' launch
fails to boot (pc=0).
- sample cyc_watch / freeze_check at STEADY STATE: an early query (before
the BIOS boots to the EXE entry) reports dirty_ram_insns=0 / warm-up
values. This premature-sampling artifact was the whole "dispatch
paradox" — the interp engages only after the EXE entry is reached.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds three cycle-isolation loops that exercise the load-delay give-back
TERMINATION and the two not-yet-modeled give-back interactions, each
measured against the Beetle oracle AT STEADY STATE (Beetle's warm-up
window reports the no-give-back value — the source of the handoff's
mis-recorded "gte_rtps +15 EXACT"; true steady Beetle = +11):
- load_use (lw; addu deps the loaded reg): native +5 == Beetle +5 —
Δ-validates the GPR_DEP give-back termination shipped in the prior
commit (load/load2 load into dead regs and never exercise it).
- gte_read_use / gte_rtps / gte_nclip: native OVER-counts by 4-5 — the
MFC2/CFC2 LDAbsorb give-back (gte_ts_done-now) is unmodeled.
- ld_div (lw; divu; mflo): native UNDER-counts by 4 — the muldiv stall
must CONSUME the pending lw give-back (Beetle decrements ReadAbsorb
while stalling MFLO to muldiv_ts_done).
The 8 CPU-load/alu/div/mult components all MATCH at steady state, so the
prior load-interlock commit is solid. The 4 confirmed divergences are
the Δ-gates for the GTE-read/MFC0/muldiv give-back follow-up; spec +
targets in accuracy/load_readfudge_ldabsorb.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Models the GTE command latency + stall-on-COP2-access, the next Stage-2 cycle
component. Mirrors the mult/div completion-stall mechanism:
- cpu->gte_ts_done (new CPUState field, appended): a GTE command sets it to
psx_cycle_count + (cost-1); back-to-back commands serialize (psx_gte_set
stalls to the prior deadline first). Any COP2 register access
(MFC2/CFC2/MTC2/CTC2/LWC2/SWC2) calls psx_gte_stall, advancing guest cycles
to the deadline if the op is still in flight.
- Cost table (psx_cycles.c, psx_gte_cmd_latency): per-op cost-1, transcribed +
verified from beetle-psx gte.cpp op returns (RTPS15 NCLIP8 MVMVA8 NCDS19
NCDT44 RTPT23 NCCT39 ... AVSZ4=5 not 6). cost-1 because the COP2 instruction's
own +1 base is charged separately (Beetle GTE_Instruction returns ret-1).
- Set armed in the shared gte_execute() (one site for both backends); stall
emitted before every COP2 reg access in both emitters (code_generator,
strict_translator) and the dirty interp. The compiled "+1 at top" vs interp
"base after exec_one" offset cancels (same as muldiv).
Validated EXACT on ruler #2 (new gte_rtps + gte_nclip loops): native +15/+8 ==
Beetle +15/+8. All other components unchanged (no regression); Tomba2 boots to
FMV. Requires regen (emitters changed); runtime+BIOS+game regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage-2's biggest cost gap closed. MULT/MULTU/DIV/DIVU now set a completion
deadline; MFLO/MFHI stall guest cycles until it — exactly like Beetle's
muldiv_ts_done. Latencies transcribed from cpu.cpp: DIV/DIVU=37; MULT via
MULT_Tab24 (14/10/7) on the first-operand magnitude. Native previously charged
ZERO for this.
- Per-instruction cycle charging is now the DEFAULT on this audit branch
(PSX_CODEGEN_CYCLE_PER_INSN), required so the stall absorbs correctly (the
running cycle count must be accurate mid-block). Validated cycle-neutral for the
identity base model; Tomba 2 still reaches the FMV.
- runtime: CPUState.muldiv_ts_done + psx_muldiv_set/stall + psx_mult_latency_s/u
in psx_cycles.c. Emitted at the mult/div/mflo/mfhi sites in the GAME emitter
(gated PSX_ENABLE_BLOCK_CYCLES). Stall fires even for mflo/mfhi to $zero.
- ruler #2 native half: tools/cycle_testrom/CMakeLists.txt builds psx-cyctest
(links the recompiled test EXE), boots disc/cyctest.cue (serial CYCT-00101 so
disc-identity matches, no warning dialog). measure.py --port 4600.
VALIDATION (native vs Beetle oracle, per-iteration component cost):
div +38 vs +38 EXACT | div_spaced +38 vs +38 EXACT (absorb works) |
mult +15 vs +15 EXACT | baseline/alu match.
Remaining: load native +7 vs Beetle +5 (memory.c flat +6 wait-state needs
calibration to ~4) — a smaller refinement. BIOS emitter still block-mode (ruler
#1 unchanged at 30) — applying per-insn + muldiv there is the next step to close
ruler #1's div-stall gap.
No FMV regression (i_stat 0x8D, screenshot verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The test ROM now boots on a real BIOS. The license region (required or the BIOS
drops to its shell) is extracted from a disc the developer OWNS via the in-tree
dumpsxiso and embedded by mkpsxiso — LOCAL ONLY, gitignored, never redistributed.
Both backends boot disc/cyctest.cue identically (real BIOS -> SYSTEM.CNF -> EXE),
so CPU/cache state matches.
- disc/cyctest.xml + disc/SYSTEM.CNF: mkpsxiso project (tracked). license_data.dat
+ built .bin/.cue gitignored (copyrighted Sony data, local only).
- measure.py: per-anchor consecutive-hit Δ = per-iteration cycle cost; reports
per-component cost via baseline subtraction.
- README: boot-disc build + license-extraction recipe + the oracle results.
Beetle ORACLE component costs (the HW targets for the cost model):
baseline 3 | alu +1 | load +5 | load2 +11 (2nd load +1 = ReadFudge)
| div +38 (~36 stall) | div_spaced +38 (fillers ABSORBED) | mult +15 (~13 stall)
Native currently charges 0 for div/mult stall — the dominant gaps. div_spaced==div
proves the stall must be a completion-timestamp + stall-on-mflo-read (absorbable),
NOT a flat charge on the divu.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ruler #2 (the in-tree cycle-model integration-test ecosystem):
- tools/cycle_testrom/gen_testrom.py: hand-encoded MIPS PS-X EXE of cycle-
isolation loops (baseline/alu/load/load2/div/div_spaced/mult). Each loop top is
a block leader, so a cyc_watch single-anchor Δ = one iteration; baseline
subtraction isolates each component (overhead + fetch cancel). Encoding
validated by a clean recompile (divu -> gpr[10]/gpr[11], etc.).
- game.toml/seeds.txt/README.md; .gitignore for the regenerable EXE/json/generated.
Universal per-block-leader observe:
- Added debug_server_cyc_observe at every block leader in the GAME emitter
(code_generator.cpp) to match the BIOS emitter, so ANY recompiled code (game,
BIOS, overlay) is anchorable on both backends — not just function entries.
#ifndef PSX_NO_DEBUG_TOOLS so prod = zero overhead.
Regression-checked: regen Tomba 2 + rebuild, FMV still streams (i_stat 0x8D), no
visual regression. Builds clean.
Remaining for ruler #2: native test-runtime delivery (boot->entry handoff) +
Beetle .exe sideload + the per-component compare harness (see README TODO).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cyc_watch + cycle_compare.py now take an optional END anchor: each recorded value
is Δcycles of one START->END pass (cost of the known code path between two dispatch
points). [A,A] = Δ between consecutive A hits (call-to-call cost). Mirrored EXACTLY
on both backends (native debug_server.c + Beetle libretro.cpp/beetle_debug_server.c,
the latter uncommitted in main checkout, additive). VALIDATED: region [0x80017FC4,
0x80017FC4] = native Δ 56 vs Beetle Δ 91, gap -35, ZERO variance across 8 passes —
a stable, offset-independent per-region gate to drive Stage-2 components to
convergence (faithful components shrink the gap monotonically; can't overshoot).
Note: [A,B] with distinct B needs B to reliably follow A (a dispatch point on the
deterministic path); B=0x8001EFFC failed (func has multiple exits) — pick wrapper/
single-exit pairs or use [A,A].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Native cyc_watch (ported from agent branch) + the missing universal hook:
debug_server_log_call_entry now samples cyc_watch_observe at every compiled
function entry (game AND BIOS; the agent's debug_server_trace_dispatch hook only
covered the BIOS trampoline, so game-text anchors fired 0). tools/cycle_compare.py
arms the same anchor on native(4500)+Beetle(4382) and diffs per-hit cycles.
Beetle side (libretro.cpp cyc_watch state/observe + cpu.cpp RunReal loop hook +
beetle_debug_server.c commands) is in the MAIN checkout (additive diagnostic,
uncommitted) and built into psx-beetle.
VALIDATED end-to-end: both backends capture 16 hits at game-text anchor 0x80017FC4.
FINDING: native is ~121M cycles BEHIND Beetle at that anchor = a gross boot-cycle-
path divergence (turbo-loads/fast-CD + overlay/dirty charge far fewer cycles than
Beetle's real-time CD boot), NOT the -8 fine drift. => absolute-cycle comparison
through boot is dominated by load-model differences; the -8 needs a controlled
anchor (early/pre-divergence or an interp-convergence point). Relocated BIOS-shell
funcs (phys 0x30000-0x5AFFF, e.g. 0x80050B0C) dispatch at a different native phys —
anchor on game-text or BIOS-ROM funcs, or the relocated phys.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In-progress investigation work snapshotted before rebasing onto the
tcc-refactored psxrecomp master. Includes overlay ABI v8
(check_interrupts_at resume-PC callback for async RFE recovery) and
code_generator/full_function_emitter delay-slot cycle accounting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dev tooling only (never shipped: tools/ is not bundled in releases, and the
debug-server commands it drives are all PSX_NO_DEBUG_TOOLS-stripped in release
builds). Saved so the first-divergence workflow that root-caused the Tomba2
logo->FMV stall is never re-invented.
- t2probe.py layer 1 fingerprint cap/diff; layer 2 unified-recorder dump +
recdiff/recdiff_state (state fork, addr/val only); resolves the
debug_client import relative to tools/.
- waitf.py retry-connect + poll-until-frame (survives socket aborts).
- shot.py screenshot to PNG for VISUAL end-state verification.
- README.md the 3-layer method (always-on fingerprint ring -> divergent
frame; frame-gated recorder -> divergent access; read-watch +
per-entry cyc -> value/pointer/timing), env seeds, and the Tomba2
worked example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root-caused the Tomba2 native logo-stuck bug: it is a guest-cycle-timeline
divergence between the compiled-overlay backend and the dirty-RAM interpreter,
surfacing through a Timer1 (0x1F801110) debounce loop at game pc 0x8008592C that
reads 1 (native) vs 2 (interp), forking logo->FMV. The handoff's "jalr target
divergence" was a benign store-PC artifact (read-watch proves a0=0xB870 and
t3=[a0]=0x800896E0 are identical in both backends); the scratchpad blind spot
was ruled out (separate sp fingerprint forks at the same frame).
First-divergence observability (debug_server.c, memory.c):
- frame_fingerprint: added separate sp (scratchpad) cumulative hash column +
per-entry cyc, so a scratchpad or cycle fork is classified, not hidden.
- Unified ordered access recorder: one buffer, kind = ramw/spw/mmiow/mmior/ramr,
array index = true execution order, per-entry cyc (psx_get_cycle_count).
record_frame_dump emits the unified stream; record_reads_dump filters reads.
- Targeted main-RAM read-watch (PSX_READ_WATCH="lo,hi" + g_ram_read_watch_active
guard in psx_read_word) — proves a load's value/address per backend.
- PSX_RECORD_FRAME env seed: arm the recorder from boot (race-free), never
arm-then-time.
Fix A (cycle-model unification, part 1): compiled-overlay code was built WITHOUT
PSX_ENABLE_BLOCK_CYCLES, so every psx_advance_cycles() in overlay code was
compiled out -> a function run as a native overlay charged ~0 cycles while the
same function via the interp charged per-instruction (~32x), the entire drift.
- overlay_api.h: OverlayCallbacks gains advance_cycles; ABI v6->v7 (rejects
stale zero-cycle DLLs).
- compile_overlays.py: build overlays with -DPSX_ENABLE_BLOCK_CYCLES=1 and
emit a psx_advance_cycles shim forwarding to the shared host timeline.
- overlay_loader.c: wire s_callbacks.advance_cycles = psx_advance_cycles.
Result: the first debounce now reads Timer1=2 in native (was 1); overlay cycle
charge went 9.5k -> 380k (interp 306k). Native still logo-stuck: a residual,
perfectly linear ~1 cycle/write over-charge remains (block.instruction_count
counts branch + delay-slot as 2 while the interp charges the pair as 1 step),
which flips a later debounce 3 vs 2. Fix A pt2 (cycle-exact block totals) is the
next step. wt/tomba2 only; no pin bump, no master merge, no other-game regen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compiles the FMV-driver class of hot code. After the region compiles, a DECOUPLED
pass finds DISPATCH_INTERIOR PCs that ACTUALLY EXECUTED this session but that no
built DLL covers (orphan interiors whose host static analysis never discovered, so
the region can't alias them) and compiles each as its OWN isolated island shard:
a single dispatch_root seed entered AT the interior PC, covering the recompiler's
worklist-discovered reachable CFG (out-of-island branches become dispatcher
exits). No host recovery.
Per ChatGPT architecture review: keep speculative/interior units in a SEPARATE
failure domain from the trusted region DLL. So each fragment is its own
<region>_<key>.dll; a fragment that fails the generated-C audit is dropped ALONE
(the region's DLL is untouched), and the pass runs even when the region's own
compile audit-failed. Safe because dispatch_root is exempt from the boundary
re-check that would reject a mid-function start, the recompiler translates
mid-function code literally (no synthesized frame; the caller set it up), and the
loader's per-function live-RAM CRC gate rejects any non-matching fragment. Gated
to EXECUTED orphans (resident code, not cross-variant data). Validated: probe
shows a clean mid-entry island for 0x80083094 (audit-clean); full run builds 34/47
executed orphans at 0x38000 as isolated shards and 0x80083094 is now covered,
even though that region's normal compile audit-failed on a bad capture variant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The build artifact was keyed by a CRC32 of the WHOLE captured region, so any
volatile byte (animation counters, data tables) changed the key and minted a
brand-new DLL even when every compiled function was byte-identical -> piles of
redundant DLLs, skip-if-exists never fired.
The loader already content-matches each function by (entry, code_crc) across ALL
DLLs sharing a region_start, so a new DLL is only needed when it adds a NEW
function identity. Gate the gcc compile on exactly that: compute the capture's
(entry, code_crc) set from the recompiler manifest, and skip the build if every
function is already covered by an existing .ranges at that region_start.
- Refactor write_overlay_ranges -> parse_overlay_func_ids (identity list) +
write_overlay_ranges_from (emit). Add load_region_coverage.
- main(): warm per-region coverage set; skip gcc when this capture's function
set is a subset of what's already built.
No loader/runtime change, no filename change. Logic unit-tested: identical code
+ different data dedups; a new function still builds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Brings the fail-closed native entry guard from wt/tomba2 (b5a6940 / bd60381) to
master, where it had never been merged — which is why Tomba (and any master-built
title) still wedged. This is the long-standing "blue screen" / shared-RAM-
divergence-between-interp-and-native class.
Root cause: an overlay function dispatched at a FOREIGN INTERIOR PC (a range-
ownership mismatch routed a PC the function does not own into it) hit the CPS
entry-switch `default: break`, which fell through and ran the function FROM ITS
TOP — re-running its prologue and corrupting shared CPU/RAM state, so the guest's
main loop then waits forever on a condition that never gets set (the 0xCF0 SIO
spin / blue screen). Reproduced on Tomba's Village of All Beginnings: a native
0x000E7000-region shard function entered at a foreign PC wedged; the interpreter
ran the same bytes correctly.
Fix: the generated CPS entry-switch for OVERLAY functions (config_.overlay_mode)
now FAILS CLOSED on an unknown non-zero continuation PC — it restores the PC and
calls psx_native_bad_entry instead of running from the top. overlay_loader_dispatch
consumes g_native_bad_entry and routes the PC to the sanctioned dirty-RAM
interpreter (the live bytes run correctly — NOT a stub/HLE). True-prologue entry
still runs from the top. Static (BIOS/boot-EXE) functions keep the legacy fall-
through. ABI bumped 5->6 so stale ABI-5 caches are rejected and rebuilt with the
guard.
Minimal port: kept only the fail-closed mechanism + its ABI plumbing. The Tomba2
host_aliases interior-routing and the native-block / CPS-probe diagnostics (which
depend on the overlay-cache-v2 multi-range Candidate fields not on master) were
intentionally omitted — the fail-closed default alone is the safety net.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Matthew Stanley