With the delay-slot fix the Beetle deltas were sane but not comparable: the
three unrolled call sequences sat at different I-cache line offsets, and
probe_run shared a line with seeded, so the KUSEG run found its first line
already filled. Beetle read 59/59/89 (KUSEG/KSEG0/KSEG1), where the fetch
model predicts a 23-cycle KSEG1 surcharge.
Pad each call sequence so its first T2 read starts a line, and start
probe_run on its own line. Beetle (disc boot, OpenBIOS and SCPH-1001 alike)
now reads 56/56/82: KUSEG equals KSEG0, and KSEG1 exceeds them by 26 cycles,
exactly the Beetle fetch-model difference for the 15 KSEG1 fetches (60 vs a
cold cached 34).
probe_run moves to 0x000100F0. Ledger ids are unchanged; fetch-tag-segment
counts 30 tags and alias-fetch-coherence reads 22 vs Beetle 11, because leaf
now straddles two lines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Timer 2 subtraction sat in the second lw's load delay slot, so it read
the stale register, and the difference was not masked to T2's 16 bits. The
Beetle disc-boot run of the probe returned 0xFFFFFFF3-style values for
every delta. Add a nop after the load and an andi 0xFFFF.
The layout shifts by six words: probe_run moves from 0x000100C0 to
0x000100D8. The ledger ids are unchanged; fetch-tag-segment now counts 27
tags and alias-fetch-coherence reads 10 vs Beetle 5 for the moved leaf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compiled code bakes KSEG0 into link values, CPS/IRQ resume PCs and I-cache
tags. KUSEG-linked EXEs run KUSEG PCs through those bodies (and their KUSEG
seeds are dropped), and overlay code executed at KUSEG/KSEG1 stays
interpreted after #417. Until #429, both emitters also charged the uncached
KSEG1 fetch only at cache-line leaders, so compiled BIOS ROM code was 4
cycles short on 5,477 of OpenBIOS's 9,592 KSEG1 instruction sites (Beetle
and the interp charge every fetch).
docs/SEGMENT_AWARE_CODE.md recommends per-segment compiled variants (code
identity = full VA, emitter segment as a per-compile constant, exact-segment
dispatch, per-instruction fetch in uncached bodies, segment-aware capture
and cache keys) over segment-relative emission, which measures +3.2 % /
+27.2 % __TEXT on R4 shards.
tools/segment_testrom/gen_segment_exe.py emits a synthetic KUSEG-linked
PS-X EXE that also works as an oracle probe. The new ctest
segment_aware_codegen runs the real recompilers on it and on OpenBIOS and
checks emitted PCs, the emitted dispatch lookup and fetch cycles (through
runtime/src/psx_icache.c) against a Beetle ReadInstruction transcription.
It passes only while the observed gaps equal its ledger (9 today).
#429 (rollout A) fixed the BIOS undercharge on master before this landed,
so KNOWN_GAPS does not list bios-kseg1-fetch-charge: the OpenBIOS check
finds no uncharged KSEG1 site and stays as a regression guard that reports
the id as a new gap. The last commit of this series brings the doc up to
date with #429.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Normal-mode discovery ran psxrecomp-game with no --project-root and ignored its exit code, so from a cwd without bios/ every producer silently lost its discovered entries (Tomba 2 shipped 54 native pairs instead of 81).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Boot anchors (reset, the kernel copy, the first A0/B0/C0 calls, the shell)
pass within the first frames, before a TCP cyc_watch arm can land, so a
native-vs-Beetle boot comparison could not be run from the tree.
psx-runtime (debug_server.c) and psx-beetle (beetle_main.cpp) now read
PSX_CYC_WATCH ("<pc>" or "<pc>-<end>" for region mode) and PSX_CYC_WATCH_N
(default 16) at start-up and arm the watch before the first guest
instruction. The native arm shares the TCP handler's code path
(cyc_watch_arm); psx-beetle calls beetle_cyc_watch_arm. With neither
variable set nothing changes. tools/cycle_compare.py --no-arm then reads
both rings.
PSX_BEETLE_UNPACED=1 skips psx-beetle's 59.94 Hz wall-clock pacing for
headless oracle runs. Guest timing is unaffected.
docs/beetle-macos.md step 7 documents the variables and the anchor-key
difference for BIOS copy windows declared dispatch_key = "rom": the shell
is keyed natively at its ROM address (OpenBIOS 0xBFC0A500, SCPH-1001
0xBFC18000) while Beetle sees 0x80030000. Checked on macOS: the documented
example records the OpenBIOS shell entry at 5,015,670 in Beetle, the value
of the #429 land gate, and the native watch arms from boot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ruler #2 no longer built from master:
- game.toml's text_size 0x800 is now rejected by the EXE parser ("a
truncated text_size must be 4 KiB page-aligned"). Use 0x2000, the
page-rounded size of the 0x1800-byte text. The icache_miss victim block
at +0x1000 is now inside the analysed range and compiled too.
- psxrecomp-game writes the text as shards (<exe>_full_NN.c), but
CMakeLists.txt still named a single <exe>_full.c. Glob the shards and
fail with the regeneration hint when there are none.
Add mkdisc.py, a pure-Python stand-in for mkpsxiso (Windows-only here):
it writes the ISO9660 boot disc as MODE2/2352 sectors with a correct
EDC/ECC and copies the license area from a disc the developer owns. The
built image stays gitignored.
README: the macOS/Linux flow (headless on both backends), and why
icache_miss reads +16 on the current ROM but +14 in the 2026-06-27 log:
the mmio loops moved the loop top from word 1 to word 0 of its cache
line, so each of its two misses refills one more word. Records the
Beetle values re-measured on macOS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- run: a crash, hang-up or socket timeout on the final frame_fingerprint /
dispatch_stats queries is a RunError (exit 3, INCOMPLETE), not a
JSONDecodeError traceback with exit 1 (MISMATCH). Seeding, log and dump
I/O errors and a runtime that survives SIGKILL also exit 3. A missing
output directory is a usage error. The dump is written to OUT.tmp and
renamed.
- compare: with different FMV-quiet settings, once a quiet frame occurred
(qc > 0, a different frame set, or no qc column) the runs are not
comparable. Report INCOMPLETE and skip the DIVERGE, missing-frame and
locator lines, which were artefacts of the setting. Mixed settings with no
quiet frame are judged as usual, with a note.
- --launch on Windows keeps backslashes (shlex with no escape character),
so C:\psx\build\game-runtime.exe is not mangled into C:psxbuild...
Self-test: 59 cases (11 new), ctest fp_identity passes. R4 dumps re-judged
and fresh warm/cold runs repeat the IDENTICAL results with 497 tolerated
straddles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the warm/cold A/B tool out of R4 so every title can use it. `run`
launches the runtime from a command template ({build} {port} {headless}
{state_dir}) or from --runtime/--game/--disc, optionally installs a saved
overlay state (--seed), turbos to frame N and saves the frame_fingerprint
ring plus the dispatch-miss count. Runs default to PSX_DEBUG_FMV_QUIET=0 and
PSX_OVERLAY_AUTOCOMPILE_OFF=1, and the dump records what they used.
`compare` judges on the guest-fact columns (cyc mmio mc sp sc wc ws qc) and
reports wr/pc only as the frame where the runs part. It tolerates the
one-write VBlank straddle from docs/TCP_COMMANDS.md (ws/wc off for one frame,
equal again at the next) and, when FMV-quiet was on, the wc+qc shift with ws
judged against the carried offset. Every tolerated frame is counted and
listed; --strict turns them off. A straddle on the last frame, a judge column
an older runtime does not report, or different FMV-quiet settings make the
result INCOMPLETE. Exit codes are unchanged from the R4 tool: IDENTICAL 0,
MISMATCH 1, usage 2, INCOMPLETE 3, MISSES 4.
The self-test (ctest fp_identity) covers column judging, both tolerances,
truncated and mismatched frame sets, dumps without ws/qc, and drives `run`
against a fake debug server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The emitter writes both helpers at a listed bgez_sites or
clip_edge_x_load_sites address whose instruction matches in overlay code
too, but overlay_dispatch_preamble.c.inc did not define them, so such a
shard failed to link (macOS, Windows) or to load (Linux dlopen RTLD_NOW)
and its overlay stayed interpreted.
The preamble now defines both over psx_ws_x_margin(), with the same pure
math gpu.c uses (ws_cull_edge.h, stdint-only, staged with the other
runtime headers). ws_cull_edge.h joins the shard compile surface in
codegen_hash_sources.cmake. overlay_shim_compile_contract links a shim
that calls both with the emitter's extern declarations, and
overlay_widescreen_callbacks runs them in an executable: identity with no
margin callback, widened at margin 120, interior clip edges unchanged.
Both fail without the definitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlay shards are compiled at the KSEG0 address of their bytes: captures
record 0x80000000|offset and the manifest parser keys entries there. The
emitter bakes that segment into every PC a shard produces: the $ra a
jal/jalr writes, jump and exception-resume PCs, and I-cache fetch tags.
Dispatch matched candidates by the segment-stripped address, so a KUSEG or
KSEG1 alias of the same bytes ran the KSEG0 body at the wrong
architectural PC.
Found in R4 (SLUS-00797) with OpenBIOS: the exception handler enters its
RAM patch slots at KUSEG (0x0000281C, 0x0000357C). The kernel-page shard
linked $ra=0x8000282C where the interpreter (and hardware) gives
0x0000282C, and fetched 0x8000281C/0x8000357C, missing lines the kernel
had just filled with KUSEG tags: +4 cycles per slot, so the next VBlank
IRQ landed at a different instruction and warm (native) vs cold
(interpreted) overlay-cache runs saved different registers from frame
194 on. A KSEG1 entry would also be charged as cached.
PSX_OVERLAY_CODE_SEGMENT (overlay_loader.h) names the rule for capture,
the manifest parser, overlay_loader_dispatch and the generated static
overlay table: a PC outside it falls to the interpreter, which runs at
the PC it was handed. CPS continuations already failed closed this way
(their entry switch only knows KSEG0 PCs); function entries now match.
overlay_loader_status reports segment_alias_interp.
Static game code has the same shape for KUSEG-linked EXEs (the parser
normalizes their headers to KSEG0); that needs segment-relative emission
and is not changed here. Shard cache tags are unchanged.
Tests: test_overlay_segment_gate.py drives the real loader (pair-dedup
harness, which also gains the RAM-geometry globals it no longer linked
without) with a KSEG0 fixture shard: KUSEG and KSEG1 dispatches stay
interpreted and never call it, KSEG0 still runs native. It fails with the
gate removed. test_compile_overlays_static_split.py covers the generated
static table and rejects a non-KSEG0 entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two failures hit while migrating a title on Windows 11:
- `audit` crashed with UnicodeEncodeError on the "->" mark (U+2192) when the
console code page can't encode it (cp1250 here). main() now reconfigures
stdout/stderr with errors="replace", so reports degrade those characters
instead of aborting.
- record_framework_pins failed. subprocess.run(["bash", ...]) lets
CreateProcess search System32 first, which starts WSL's bash; it can't see
D:/ paths, and a CRLF checkout of record_pins.sh breaks any bash anyway.
It now resolves bash via shutil.which, passes POSIX-style paths, and falls
back to the existing git rev-parse snapshot when the script can't run.
Most of this was done by Claude Opus 5.5; I reviewed and tested it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New projects describe themselves to community lists (recomp.board et al.)
via .recomp.json at the repo root: game, system PS1, type recomp,
toolchain psxrecomp, status exploring, and original region + serial.
write_recomp_json.py takes the serial from disc_probe.json and derives
region from the Sony serial prefix, falling back to --region (which
defaults to USA and would mislabel PAL discs). It only fills missing
fields, so re-running it never clobbers a maintainer's edits.
Both gitignore templates ignored /*.json at the root, which would have
silently dropped the file from the initial commit; un-ignore it.
Closes#406
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlay bands are shared address windows, so every observed interpreted PC
is demanded for every occupant of the band. classify_overlay_seeds promotes
such a demand to a linear walk root whenever _callable_legacy_seed accepts
it, and that predicate accepts three unequal grades of evidence: the image
entry, an `addiu sp, sp, -imm` prologue, or a `jr $ra` two words back.
The third grade fires exactly where it is least trustworthy. The first word
after a function's return is also the first word of whatever the linker laid
down next, and in an overlay image that is routinely a pointer table, a
packed record array or zero fill. The walk then starts at the table head,
never reaches a return, and runs to the image end; the generated-C audit
reports a column of UNSUPPORTED_INSTRUCTION and the shard fails.
Require the bounded CFG probe the discovery roots already use
(plausible_callable_target: reachable return, no fallthrough past the probe)
before a prologue-less dispatch entry may be a walk root. A failure demotes
it to DISPATCH_INTERIOR, which keeps the dispatch evidence and the isolated
fragment demand and only declines to start a walk there; the image entry and
the prologue keep their own evidence and are not probed.
Measured on Breath of Fire III (SLPS-00990), 406 overlay captures:
dispatch entries already accepted as walk roots 5,339
with a prologue 2,066 unchanged
on the preceding `jr $ra` alone 3,273
rejected by the probe 41
DISPATCH_ENTRY roots 1,141 -> 1,100 (-41)
DISPATCH_INTERIOR 81,474 -> 80,851 (-623, interiors of the
fabricated walks)
STATIC_DISCOVERY_ROOT 10,741 -> 10,741
STATIC_DISPATCH_ENTRY 194 -> 194
FUNCTION_POINTER_TARGET 20 -> 20
isolated fragment demands 105,891 -> 105,891 (nothing lost)
41 of 406 captures change at all. All 41 rejected addresses are data by
inspection: in-image code-pointer tables, packed 16-bit record arrays,
signed delta pairs, zero fill. 27 of them are precisely the 27 shards whose
all-bands compile failed the audit -- one bad root per failing shard. The
other 14 are the same defect in shards whose data happened to decode all the
way to the image end without meeting an unsupported opcode.
New CTest: frameless_dispatch_root_cfg_proof.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_compile_overlays_static_split.test_data_only_capture_reports_skip_with_captured_log
has been red on master since 4683e923 (2026-09-12). That commit reworded both
emit sites deliberately -- "no walk-root seeds (data-only region)" became
"no shared walk-root seeds; checking fragments separately", because the region
is no longer declared data-only: its executed dispatch demands are still
served by the isolated-fragment pass. The assertion still names the old text,
so the test fails on a clean checkout:
AssertionError: 'SKIP: no walk-root seeds' not found in '... SKIP: no
shared walk-root seeds; checking fragments separately ...'
The literal was duplicated at the two emit sites and copied a third time into
the test, so rewording two of the three left the third stale with nothing to
catch it. Give it a name -- NO_SHARED_WALK_ROOT_SEEDS_SKIP -- use it at both
emit sites, and have the test assert the constant rather than a copy of its
text. A future reword then updates all three at once or fails loudly.
The printed output is unchanged, verified byte for byte against the pre-change
text including both leading spaces and the trailing blank line.
tools/test_compile_overlays_static_split.py: 10 tests, was 9 passed 1 failed,
now 10 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TOOLCHAIN_PINS had entries for win and linux only, and stage_toolchain
picked 'linux' for every non-Windows host, so a Mac either died with "no
toolchain pins" or staged a Linux interpreter it cannot run. No Mac player
ever got a bundled toolchain. The stock /usr/bin/python3 on macOS 13/14 is
3.9, which cannot run compile_overlays.py (tomllib, PEP 604 unions), so the
system interpreter is no fallback either.
Add macos-x64 and macos-arm64 pins: the same python-build-standalone release
and CPython 3.13.1 as the Linux pin, install_only_stripped assets, SHA256
measured by downloading both and matched against the publisher's .sha256
sidecars. host_platform_tag() picks the key from sys.platform and the
machine architecture. No prebuilt tcc on macOS: the bundled interpreter
drives the system clang.
Verified on an Intel Mac (macOS 13, 8 GB): toolchain staged beside the
Parasite Eve diagnostic product (45 MB), and with the POSIX spawner the
runtime compiled 9 shards in the first 24 s of a headless boot.
(cherry picked from commit 5c49e4d3cf835ed026964cd24998907c41a8be4c)
78 of 105 published releases lack the key, and the runtime never initialises
the overlay loader without it. The packager now refuses a staged game.toml
without it unless --ship-without-overlay-cache-because states why, and the
setup host's self-check reports overlay_cache_configured so CI can assert it.
(cherry picked from commit a030f4843d8e55eafed4cb42eafc6d74bb89c086)
No setup package ever staged one, so overlay_cache = true alone left every
streamed overlay on the interpreter. rebuild now stages the pinned Python,
tcc, this pin's recompiler, headers and the build's codegen hash beside the
normal and diagnostic products, and writes compiler.txt naming the compiler
the wizard already has; the runtime treats that as a gcc-tier compiler, so a
player with nothing on PATH gets -O2 shards. Verified on the NCII kit: gcc
tier reported and four shards built with no compiler on PATH.
(cherry picked from commit 03865e10f6ee198d58c47c74526df5db46bb35a9)
The runtime mounts a .chd natively and the launcher lists it in the disc
picker, so a player could select one and see a valid disc identity, then have
first-run setup fail: verify-disc, prepare_disc.py and probe_disc.py hash and
parse raw 2352-byte track data, which a CHD stores compressed. Nothing on the
Python side could read it, and telling players to install MAME for chdman is
not an answer.
libchdr is already vendored and compiled for the runtime and the native media
CLI. Build it once more as a shared library in recompiler/ (CMake target chdr,
output next to the emitters, static CRT on Windows so a plain python.exe can
load it) and read the CHD from Python through ctypes. tools/psx_chd.py mirrors
the layout the runtime's iso_reader.cpp uses -- 2448-byte frames, 4-frame
track padding, stored versus virtual pregaps, big-endian audio -- and
reproduces the .bin files a Redump dump of the same disc contains, in either
of Redump's two layouts (one bin per track, or one bin for the disc).
verify-disc hashes those reconstructed track bytes against [prepare_disc]
instead of the container. prepare_disc.py writes the tracks into out_dir in
whichever layout the recorded digests match and then continues down its
normal cue path, so the data track is written once, SBI companions are looked
up next to the CHD, and the receipt names the CHD the player chose.
probe_disc.py extracts to a scratch directory and probes the cue. The CLI
builds target chdr on demand for a kit whose emitters predate it, and falls
back to the extract-it-first message when it cannot.
Verified against two discs from a Redump library converted with chdman: a
single-track MODE2 disc and a four-track disc with stored audio pregaps both
reproduce their catalogued track-1 size, MD5 and SHA-1 exactly; verify-disc
and prepare_disc.py accept the four-track one end to end. Tests build a
synthetic uncompressed v5 CHD around a cooked ISO image, so no retail data is
needed; the decode tests skip with a reason when no libchdr is built.
3a174fab added reconcile_empty_primary_scans() at the tail of main() and
called it unconditionally with cache_dir. cache_dir is bound only on the
DLL path; --static never assigns it. So every --static run now ends in
UnboundLocalError: cannot access local variable 'cache_dir' where it
is not associated with a value
after generated/overlays_static.c and all its translation units have
already been written. The output is complete, but the process exits 1,
and callers that gate on the compiler's exit (axis_b_loop.sh expects 0
or 2) abort before the runtime is rebuilt. Found regenerating Breath of
Fire III against master 193a60b8.
The reconciliation has nothing to do in static mode: _do_capture hands
each capture to static_capture_job and returns before the only
pending_empty_primary.append(), and the check it performs reads the
per-game DLL cache, which static mode does not have. Guard the call on
the non-static path. In static mode, raise instead of silently skipping
if anything was ever queued, so a future change that routes static
captures through the DLL path fails loudly rather than dropping a root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shokunin