Normal-mode discovery ran psxrecomp-game with no --project-root and ignored its exit code, so from a cwd without bios/ every producer silently lost its discovered entries (Tomba 2 shipped 54 native pairs instead of 81).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- run: a crash, hang-up or socket timeout on the final frame_fingerprint /
dispatch_stats queries is a RunError (exit 3, INCOMPLETE), not a
JSONDecodeError traceback with exit 1 (MISMATCH). Seeding, log and dump
I/O errors and a runtime that survives SIGKILL also exit 3. A missing
output directory is a usage error. The dump is written to OUT.tmp and
renamed.
- compare: with different FMV-quiet settings, once a quiet frame occurred
(qc > 0, a different frame set, or no qc column) the runs are not
comparable. Report INCOMPLETE and skip the DIVERGE, missing-frame and
locator lines, which were artefacts of the setting. Mixed settings with no
quiet frame are judged as usual, with a note.
- --launch on Windows keeps backslashes (shlex with no escape character),
so C:\psx\build\game-runtime.exe is not mangled into C:psxbuild...
Self-test: 59 cases (11 new), ctest fp_identity passes. R4 dumps re-judged
and fresh warm/cold runs repeat the IDENTICAL results with 497 tolerated
straddles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves the warm/cold A/B tool out of R4 so every title can use it. `run`
launches the runtime from a command template ({build} {port} {headless}
{state_dir}) or from --runtime/--game/--disc, optionally installs a saved
overlay state (--seed), turbos to frame N and saves the frame_fingerprint
ring plus the dispatch-miss count. Runs default to PSX_DEBUG_FMV_QUIET=0 and
PSX_OVERLAY_AUTOCOMPILE_OFF=1, and the dump records what they used.
`compare` judges on the guest-fact columns (cyc mmio mc sp sc wc ws qc) and
reports wr/pc only as the frame where the runs part. It tolerates the
one-write VBlank straddle from docs/TCP_COMMANDS.md (ws/wc off for one frame,
equal again at the next) and, when FMV-quiet was on, the wc+qc shift with ws
judged against the carried offset. Every tolerated frame is counted and
listed; --strict turns them off. A straddle on the last frame, a judge column
an older runtime does not report, or different FMV-quiet settings make the
result INCOMPLETE. Exit codes are unchanged from the R4 tool: IDENTICAL 0,
MISMATCH 1, usage 2, INCOMPLETE 3, MISSES 4.
The self-test (ctest fp_identity) covers column judging, both tolerances,
truncated and mismatched frame sets, dumps without ws/qc, and drives `run`
against a fake debug server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The emitter writes both helpers at a listed bgez_sites or
clip_edge_x_load_sites address whose instruction matches in overlay code
too, but overlay_dispatch_preamble.c.inc did not define them, so such a
shard failed to link (macOS, Windows) or to load (Linux dlopen RTLD_NOW)
and its overlay stayed interpreted.
The preamble now defines both over psx_ws_x_margin(), with the same pure
math gpu.c uses (ws_cull_edge.h, stdint-only, staged with the other
runtime headers). ws_cull_edge.h joins the shard compile surface in
codegen_hash_sources.cmake. overlay_shim_compile_contract links a shim
that calls both with the emitter's extern declarations, and
overlay_widescreen_callbacks runs them in an executable: identity with no
margin callback, widened at margin 120, interior clip edges unchanged.
Both fail without the definitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlay bands are shared address windows, so every observed interpreted PC
is demanded for every occupant of the band. classify_overlay_seeds promotes
such a demand to a linear walk root whenever _callable_legacy_seed accepts
it, and that predicate accepts three unequal grades of evidence: the image
entry, an `addiu sp, sp, -imm` prologue, or a `jr $ra` two words back.
The third grade fires exactly where it is least trustworthy. The first word
after a function's return is also the first word of whatever the linker laid
down next, and in an overlay image that is routinely a pointer table, a
packed record array or zero fill. The walk then starts at the table head,
never reaches a return, and runs to the image end; the generated-C audit
reports a column of UNSUPPORTED_INSTRUCTION and the shard fails.
Require the bounded CFG probe the discovery roots already use
(plausible_callable_target: reachable return, no fallthrough past the probe)
before a prologue-less dispatch entry may be a walk root. A failure demotes
it to DISPATCH_INTERIOR, which keeps the dispatch evidence and the isolated
fragment demand and only declines to start a walk there; the image entry and
the prologue keep their own evidence and are not probed.
Measured on Breath of Fire III (SLPS-00990), 406 overlay captures:
dispatch entries already accepted as walk roots 5,339
with a prologue 2,066 unchanged
on the preceding `jr $ra` alone 3,273
rejected by the probe 41
DISPATCH_ENTRY roots 1,141 -> 1,100 (-41)
DISPATCH_INTERIOR 81,474 -> 80,851 (-623, interiors of the
fabricated walks)
STATIC_DISCOVERY_ROOT 10,741 -> 10,741
STATIC_DISPATCH_ENTRY 194 -> 194
FUNCTION_POINTER_TARGET 20 -> 20
isolated fragment demands 105,891 -> 105,891 (nothing lost)
41 of 406 captures change at all. All 41 rejected addresses are data by
inspection: in-image code-pointer tables, packed 16-bit record arrays,
signed delta pairs, zero fill. 27 of them are precisely the 27 shards whose
all-bands compile failed the audit -- one bad root per failing shard. The
other 14 are the same defect in shards whose data happened to decode all the
way to the image end without meeting an unsupported opcode.
New CTest: frameless_dispatch_root_cfg_proof.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime mounts a .chd natively and the launcher lists it in the disc
picker, so a player could select one and see a valid disc identity, then have
first-run setup fail: verify-disc, prepare_disc.py and probe_disc.py hash and
parse raw 2352-byte track data, which a CHD stores compressed. Nothing on the
Python side could read it, and telling players to install MAME for chdman is
not an answer.
libchdr is already vendored and compiled for the runtime and the native media
CLI. Build it once more as a shared library in recompiler/ (CMake target chdr,
output next to the emitters, static CRT on Windows so a plain python.exe can
load it) and read the CHD from Python through ctypes. tools/psx_chd.py mirrors
the layout the runtime's iso_reader.cpp uses -- 2448-byte frames, 4-frame
track padding, stored versus virtual pregaps, big-endian audio -- and
reproduces the .bin files a Redump dump of the same disc contains, in either
of Redump's two layouts (one bin per track, or one bin for the disc).
verify-disc hashes those reconstructed track bytes against [prepare_disc]
instead of the container. prepare_disc.py writes the tracks into out_dir in
whichever layout the recorded digests match and then continues down its
normal cue path, so the data track is written once, SBI companions are looked
up next to the CHD, and the receipt names the CHD the player chose.
probe_disc.py extracts to a scratch directory and probes the cue. The CLI
builds target chdr on demand for a kit whose emitters predate it, and falls
back to the extract-it-first message when it cannot.
Verified against two discs from a Redump library converted with chdman: a
single-track MODE2 disc and a four-track disc with stored audio pregaps both
reproduce their catalogued track-1 size, MD5 and SHA-1 exactly; verify-disc
and prepare_disc.py accept the four-track one end to end. Tests build a
synthetic uncompressed v5 CHD around a cooked ISO image, so no retail data is
needed; the decode tests skip with a reason when no libchdr is built.
3a174fab added reconcile_empty_primary_scans() at the tail of main() and
called it unconditionally with cache_dir. cache_dir is bound only on the
DLL path; --static never assigns it. So every --static run now ends in
UnboundLocalError: cannot access local variable 'cache_dir' where it
is not associated with a value
after generated/overlays_static.c and all its translation units have
already been written. The output is complete, but the process exits 1,
and callers that gate on the compiler's exit (axis_b_loop.sh expects 0
or 2) abort before the runtime is rebuilt. Found regenerating Breath of
Fire III against master 193a60b8.
The reconciliation has nothing to do in static mode: _do_capture hands
each capture to static_capture_job and returns before the only
pending_empty_primary.append(), and the check it performs reads the
per-game DLL cache, which static mode does not have. Guard the call on
the non-static path. In static mode, raise instead of silently skipping
if anything was ever queued, so a future change that routes static
captures through the DLL path fails loudly rather than dropping a root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The launcher and studio ship as Retro Launcher and Retro Studio now,
from the RetroPortingToolKit org, so the old product name is gone from
display text, documentation and release asset filenames.
The split is by case, and it is not cosmetic. CamelCase RetComM was
always the product; lowercase retcomm and RETCOMM_ are identifiers --
the C++ namespace, the binaries, the data directories, the CMake and
environment variables, the Cloudflare service names, the readme markers.
Renaming those would orphan existing installs for no benefit, so they
are untouched: 1101 lowercase retcomm and 205 RETCOMM_ still stand in
the launcher alone.
Four things are deliberately still spelled RetComM, and three of them
would have been real breakage:
- RetComM-Data is CamelCase but it is a directory on disk --
<exe_dir>\RetComM-Data\{runtime,config,data} for a portable install.
Renaming it strands every portable user's config, toolchains and
catalog cache.
- TechnicallyComputers/RetComM-Launcher in readme_has_launcher() is the
historical half of a both-slugs test. Renaming it makes it a duplicate
of the line above and sends every port built before the transfer back
to reporting its launcher section missing.
- _LEGACY_LAUNCHER_RE matched "^## RetComM Launcher". A sweep renamed it,
which would have silently stopped it matching every README already out
there; it now accepts both headings.
- The portable-stub lookup gained the pre-rename filenames back as
fallbacks, because old zips stay on the releases page and a rollback
downloads one.
Release assets become Retro-Launcher-* / Retro-Studio-*. That is safe
for installed clients: pick_launcher_asset() gates on *linux*/*windows*/
*macos* plus extension and arch, not on the product name, and the
Windows portable stub is identified by its RCM1 trailer rather than its
filename.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013DpaQqVguHFpBuCmdoFYWY
The static isolated-fragment pass resolved demands by bare entry address across
every capture: once any part had a variant at an address, that address counted
as served for every other capture of the band, and static_entry_sources kept
only the last capture's bytes. The dispatcher validates each variant against
the exact bytes it was compiled from, so in a band with N occupants exactly one
occupant ever received fragments (BoF3: BOSS055 35/35, PLP678 19/19, SCENA19
20/20; the 128-occupant and 181-occupant bands got none). 23,728 (entry, image)
demands were unserved; at the title screen the resident SCENA16 ran every
observed entry on the interpreter while every spanning piece came from
SCENA19/04/06.
- Demands are keyed (entry, image crc); parts carry image_crc;
select_static_fragment_demands / served_static_variant_keys are the pure
helpers (unit-tested).
- Fragment compiles run on the --jobs pool; the image table is sent once per
worker (initializer) instead of once per demand; results merge in demand
order and duplicates already served by an earlier-merged part are dropped,
so the output equals the sequential loop's.
- generate_interior_fragment_static returns (part, reason). Deterministic
verdicts on the bytes (generated-C audit, requested-entry audit, and a walk
that runs off the image, which the recompiler reports by throwing) are
memoized in <out-dir>/interior_fail_memo.txt and counted as skipped, never
as shard failures; --force-interior retries an entry. Only a
non-deterministic recompiler failure is a shard failure.
- Fragment parts are grouped one translation unit per image (BoF3: 779 units
instead of 20,635 files).
BoF3 full run: 20,250 fragments built, 3,570 memoized rejections, 238 s cold,
nothing lost from the previous dispatcher (99,778 -> 138,148 served keys).
Title screen: 12 interpreted SCENARIO-band entries -> 2, both unharvested PCs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The overlay capture format DELIBERATELY appends one coherent guard instruction
past the end of a dirty-page run (overlay_capture.c write_json_window, `size
+= 4u`, pinned by runtime/tests/test_interpreter_perf_guards.py:47): a MIPS
branch sitting at the run's final word (...FFC) always executes its delay slot
in the next page (...000), and the shard must be able to emit and hash that
word. So a capture whose size is a page multiple PLUS 4 is a CORRECT capture --
N bytes of overlay plus one guard word. Measured: 36 of 36 Ape Escape
(SCUS-94423) vault records and 846 of 946 Tomba 2 (SCUS-94454) records are that
shape.
The recompiler was never taught the asymmetry: the guard word is a legal
DELAY-SLOT SOURCE but an ILLEGAL BLOCK LEADER, because the word after it does
not exist in the image. It had exactly one bound -- end_address() -- used both
to read words and to decide what is code.
MEASURED on Ape Escape, its own single-source vault at framework
v0.3.2-alpha-139-g1bf70960, 36 variants:
PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0
! [recompiler] overlay 0x80136000 crc 4BAB23C9: what(): cannot emit control
flow at 0x80138000: mandatory delay slot 0x80138004 is outside the input
image
+ 16 x [fragment] interior 0x801360.. @region 0x00136000, identical message
One root cause, seventeen shards. The whole region then falls back to the
dirty-RAM interpreter forever, and nothing at runtime says so.
Causal chain, every step read in source:
1. overlay_capture.c:349-361 appends the guard word; region 0x80136000 gets
size 0x2004, so the image is [0x80136000, 0x80138004).
2. compile_overlays.py make_psxexe wraps those bytes verbatim with text size
= len(data), so PS1Executable::end_address() = 0x80138004.
3. function_analysis.cpp in_exe / in_function admit 0x80138000 as ordinary
code, and the walk reaches it by fall-through from 0x80137FFC
(0x24022000, addiu -- not control flow), so func.end_addr becomes the
image end.
4. control_flow.cpp makes the final block end_addr = func.end_addr - 4 =
0x80138000: the block LEADER is the guard word.
5. control_flow.cpp's "Also check last instruction itself" promotion finds
0x80137FFC is not control flow and therefore promotes read_word(end_addr)
-- the guard word 0x1062005A, a beq -- to block.exit_instr, with
has_delay_slot = true and address == end_addr. THIS IS THE DEFECT.
6. code_generator.cpp computes delay_slot_addr = 0x80138004, read_word
returns nullopt, and it throws.
THE THROW AT STEP 6 IS CORRECT AND IS PRESERVED. Emitting a control transfer
without its architectural delay slot silently corrupts guest state, which is
what the code did before that throw was added. The gap is step 5.
Fix: give the image view two distinct bounds.
end_address() READ bound -- every byte the image supplies, so a
mandatory delay slot living at a guard word resolves.
analysis_end_address() ANALYSIS bound -- one past the last byte that may be
discovered, made a block leader, or made a block's
exit instruction.
read_word() keeps the READ bound. Discovery and block construction move to the
ANALYSIS bound: function_analysis.cpp in_exe / in_exe_p3 / add_forced_entry /
every hard_cap that used to be the image end / the jr-table producer_hi
defaults / the static jr-ra and JAL sweeps; control_flow.cpp gains
analysis_walk_hi(func) and uses it for cfg.function_end, alias entries,
add_boundary, the scan loop, the final block's end_addr and in_func;
code_generator.cpp's mid-function split pre-pass; main_psx.cpp seed admission
and the data-scan pointer promotion; analysis_db.cpp, main_toml.cpp,
main_cli.cpp target/seed admission. The clamp is applied only when it leaves a
non-empty walk, so a degenerate function starting at or past the bound keeps
failing closed rather than silently producing an empty CFG.
HOW THE BOUND IS DERIVED -- the producer STATES it; the recompiler is told.
The alternative was for the recompiler to infer the count from `size % 4096 ==
4`. That fits every capture the current writer has ever produced, but it is a
consumer guessing at a producer's intent, and this tree already has the
counter-example: 52 of the 946 SCUS-94454 records have residues from 112 to
3996 bytes (one not even 4-byte aligned) because they come from the STATIC
extractor, which records an exact file extent and appends no guard word. A rule
keyed to the page-run shape is a rule that shifts the moment a producer
changes.
So the count travels with the image, in a magic-tagged field in the otherwise
zero-filled tail of the 2048-byte PS-EXE header (exe_tag in ps1_exe_parser.h:
"PSXRGRD1" at 0x7E0, uint32 count at 0x7E8). Putting it in the header rather
than on the command line means every consumer gets it automatically and no
future invocation path can forget to pass it -- compile_overlays has three
make_psxexe sites and five recompiler spawn sites. An absent or malformed tag
means ZERO guard bytes, which is exactly what every genuine PS-X EXE is: the
BIOS and every retail main executable have no guard word, their analysis bound
equals their read bound, and their analysis is bit-for-bit unchanged.
Producer side:
overlay_capture.c writes "guard_bytes": 4|0 per region -- the writer
stating its own intent, additive to the v2 schema and
preserved through coverage_vault.py compaction.
compile_overlays.py capture_guard_bytes(): prefer that field; for captures
recorded before it existed, reconstruct the single
shape write_json_window can produce (page multiple + 4)
and return 0 for everything else, which is the
fail-closed answer -- an unrecognised guard word is
analysed as code and the step-6 throw still fires
loudly at generation time.
make_psxexe guard_bytes is keyword-only with NO default, so a new
call site is a TypeError, not a silently mis-analysed
shard. The count is threaded through
compile_fragment_batch / compile_interior_fragment /
generate_interior_fragment_static and into the fragment
fail-memo key (bumped to v3, so a memo recorded under
the old analysis cannot suppress a retry under the
corrected one).
SECOND-ORDER DEFECT FIXED IN THE SAME PASS. generate_ranges_manifest extends a
range by +4 for any block whose exit instruction IS its last word, so the
cloned delay slot joins the candidate CRC and the page-generation watch. That
extension was unclamped. Reachable TODAY, independent of the guard word: a
branch-likely opcode (0x14-0x17, RESERVED on the R3000A) as the image's very
last word is short-circuited by translate_basic_block into an inline RI raise
BEFORE any delay slot is read, so generation succeeds and the manifest claims 4
bytes past the image end. Measured on origin/master with a 28-byte synthetic
image: `R 80010000 20` for an image ending at 0x8001001C. The cache would then
validate the shard against bytes the shard never saw. Now clamped to the READ
bound -- not the analysis bound, because a guard word genuinely IS compiled in
when it serves as a delay slot and must participate in the CRC.
DEAD CODE, documented rather than trusted: basic_block.cpp's BasicBlockAnalyzer
bounds its walk correctly but nothing in the tree constructs it. It is still
compiled and still hashed into the codegen tag, so a change there invalidates
caches without moving an emitted byte. Its bounds move to the analysis bound
too -- wiring it up must not reintroduce this defect -- and a header comment now
says where the live walker actually is.
TESTS. Both registered, so ctest runs them and the configure-time
registration guard is satisfied.
recompiler/tests/test_overlay_guard_word_analysis_bound.py drives
psxrecomp-game over five synthetic images:
A tagged, 2 pages + guard word, last real word not control flow, guard word
IS a beq -- the measured Ape shape. Must generate; the guard word must not
be emitted as an instruction; the block must publish 0x80012000 for the
runtime to dispatch; no range may claim it.
B byte-for-byte the same payload with NO tag. Must still fail closed with
the mandatory-delay-slot diagnostic naming 0x80012000. A and B differ only
in the header tag, which proves the tag is what changed behaviour and that
the throw was not weakened.
B2 a misaligned declaration (6). Must be rejected and behave like B.
C tagged, with a real branch at the last analysable word. The guard word
must be emitted AS the delay slot and must be covered by the manifest.
D branch-likely as the final word. Generation must succeed and no range may
exceed the image end.
Against origin/master's psxrecomp-game the same file reports exactly the two
things this commit changes, and passes B, B2 and C:
FAIL: tagged guard word still fails generation: what(): cannot emit control
flow at 0x80012000: mandatory delay slot 0x80012004 is outside the
input image
FAIL: range [0x80010000,0x80010020) claims 4 byte(s) past the image end
0x8001001C
tools/tests/test_overlay_guard_bytes_declaration.py pins the producer half:
field precedence, the reconstruction table, rejection of nonsense
declarations, the tag's byte offsets, that make_psxexe with guard_bytes=0 is
byte-identical to the pre-fix wrapper, and that overlay_capture.c both appends
the guard word and now declares it.
MEASURED OUTCOME, Ape Escape, same vault, same flags, fresh cache:
before PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0
after PSX_SHARD_RESULT ok=31 failed=0 skipped=57 capacity_fastpath=0
ok falls because 19 isolated island fragment shards at 0x00136000 are no longer
needed once the region variant compiles as one shard. Coverage is what matters,
and it is measured from the DLL exports: 349 -> 361 unique func_XXXXXXXX
entries, ZERO lost, 12 gained -- 0x80136140, 0x8013614C, 0x80136154,
0x80136238, 0x801362D4, 0x801362DC, 0x801362F0, 0x80136324, 0x80136370,
0x80136400, 0x80136480, 0x80137950, all from the failure list. The other four
failing PCs were already served by a sibling variant's shard.
INERTNESS.
BIOS (SCPH1001, psxrecomp-bios --config bios/SCPH1001.toml): all four
generated artifacts BYTE-IDENTICAL, verified by sha256 rather than by exit
code, because tools/regen_bios.sh exits 0 on failure.
Tomba 2 main EXE (psxrecomp-game --config game.toml): all nine generated
artifacts BYTE-IDENTICAL, and the console log is identical too.
Tomba 2 overlays, 82 variants over 6 regions from its own vault: identical
result line (ok=98 failed=0 skipped=103) and identical shard identity set,
4901 unique func exports on both sides -- zero gained, zero lost. 97 of 98
range manifests are byte-identical. The one that differs is region
0x8010A000 variant A32DB188, size 0x2004, whose last function's range goes
from 0x218 to 0x214 bytes: its guard word (0x24840008, addiu) had been
emitted as an ordinary instruction and hashed, and now is not. That is
precisely the intended change and nothing else moved.
Bead beads-eio.3.100.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Supersedes the global HYBRID controller-mode shape from PR #233 with a trusted plugin callback that can choose analog or digital presentation per input sample.
Keeps analog mode hardware-faithful by no longer folding physical D-pad input onto the left stick, while leaving debug injection able to steer stick-only paths.
Includes the pad/cursor probe tooling from #233 for diagnosing input doubling.
Keep dispatch counters as raw event counts, fail closed on malformed or reset evidence, label rolling and censored scopes, retain bounded-ring provenance, and write hash-bound diagnostic artifacts atomically. Add focused protocol tests and register them with CTest.
Matthew Stanley