Commit Graph

69 Commits

Author SHA1 Message Date
Matthew Stanley 4c19677b9e docs: CLEAR!! already English + HUD render model (GP0-ring evidence)
Corrects and refines the Bucket-1/3 findings with live GPU evidence:

- CLEAR!! banner: ALREADY renders in stylized Latin ("CLEAR!!") — verified
  on-screen in a gameplay stage. Not Japanese; no translation needed. The
  earlier "resister/baked-sprite" assumption was wrong.

- In-game HUD (ステージ/バッテリー): pinned the render model via the always-on
  GP0 ring (gpu_frame_dump). The game composites — ~6 draw prims/frame,
  rebuilding only the animated foreground (character/blocks, ~8 textured 0x2C
  quads at screen y>=31, via BIOS DMA from an OT at ~0x138xxx). The HUD (screen
  y~8) is composed ONCE at stage-load, not per-frame, and has no static SJIS
  source (RAM-scan proof). Remapping it needs catching that stage-load compose
  (a fresh stage load while observing) and a scoped blit/tile intercept — not a
  source-patch. NOT shipped: no clean pixel-verified remap this pass, and a
  blanket VRAM font swap would corrupt katakana elsewhere (Rule 5, no hacks).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 18:07:42 -07:00
Matthew Stanley 938fc26f17 docs: per-glyph coverage pass — shipped labels + composed-HUD/CLEAR resisters
Records the outcome of the per-glyph translation pass (Buckets 1-3):

- SHIPPED: stage/level/tutorial names (source-patch, f00631e/31ab3c2);
  name-entry prompt -> "YOUR NAME?" (source-patch, 31ba92d); name-entry
  Latin input confirmed NATIVE (grid already draws the font's A-Z/0-9).

- RESISTERS (not shipped, no fake): the in-game HUD (ステージ/バッテリー)
  and the CLEAR!! banner are composed per-glyph at runtime with NO static
  source address. Proven: a live 2 MB RAM scan finds ステージ/バッテリー
  only inside static message sentences (never a standalone HUD label,
  no ステージ: template); クリア/CLEAR/CLEAR exist nowhere as a
  standalone banner; and every disc graphic TIM was extracted+rendered —
  the only baked text is the already-English TSUMU light logo. So the
  source-patch (which needs a static address) cannot reach them; they
  require the scoped per-glyph sprite-draw index-remap hook (Appendix Y),
  a separate larger effort. Includes the full disc asset map.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:49:17 -07:00
Matthew Stanley b0f8a8518c docs: per-glyph text investigation (font index map, label storage, hook design)
Records the concrete findings for the deferred per-glyph HUD/menu/title
localization: the font glyph-index map (Latin A-Z=73-98, 0-9=105-114;
fullwidth Latin glyphs present, no injection needed), the static LE-SJIS
label storage in fixed-width space-padded slots (level names 0x80070755+,
tutorial/mode labels 0x80070300+), and the cleanest hook design (load-time
source patch: rewrite a label's katakana/hiragana SJIS to fullwidth-Latin
so the existing per-glyph routine draws English).

Not shipped: the verification bar is a rendered-English screenshot, and the
target screens (mode/level-select, in-game HUD) are unreachable in the
headless build — the attract never auto-plays the demo, and reaching them
needs completing the name-entry glyph grid (blind grid nav to the "End"
cell did not land). Committing a RAM-patch without that screenshot would be
unverified coverage. Remaining resisters (HUD runtime counter; title/CLEAR
sprites) documented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 15:29:56 -07:00
Matthew Stanley d24d660774 docs: coverage status - 138-entry message table + reader limitation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 13:52:32 -07:00
Matthew Stanley c72e36d10e docs: record Phase-0 empirical findings for string translation
Fold the hard-won runtime findings into STRING_TRANSLATION.md: the
message-table text is little-endian 16-bit Shift-JIS (EXE menu labels are
big-endian); FE-FF/FF-FF/30-FC control framing; message text is
string-pointer-drawn (interceptable) but HUD/title/menu are per-glyph
sprites (need a separate glyph-index hook); struct-embedded NUL records
crash if replaced (apply gated to 0xFFFF-framed messages by default);
capture quality gate; verification status; open items for full coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 12:19:53 -07:00
Matthew Stanley bf4482b4c8 docs: spec on-the-fly string translation / localization framework
Implementation-ready design for capturing PSX source strings from memory
and substituting translated bytes on the fly, so JP-only titles (Tsumu
Light SLPS-02253 first) can be played in English or any language.

Adapts the author's proven n64recomp prior art (Pocket Monsters Stadium
JP): always-on capture ring keyed by FNV-1a64 content hash, a per-title
multilingual table, and a draw-time apply hook that repoints the string
arg at guest scratch so the original renderer draws the replacement (no
regen, unbounded length, dynamic args preserved).

Grounds the PSX design in real findings: Tsumu text is Shift-JIS (2-byte
fullwidth + 1-byte halfwidth kana) with 0xFFFF-framed message records;
in-EXE text block ~0x80070900-0x80072000 via pointer table at 0x80071474;
menu labels at 0x80010C5C/0x80010C94; font already has fullwidth Latin
(English renders with zero glyph injection). Hooks onto the psx_dispatch
chokepoint (fntrace_record census + g_psx_bios_hle_hook precedent), so no
per-function codegen and works in Release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 11:04:47 -07:00
Matthew Stanley 77a1f8bd6b fix(recompiler): overlay in-function jal targets become alias entries (B/C)
Completeness layer for the Tomba 2 FMV-freeze fix (the discovery/ownership side
of "containment is not ownership, ownership is not entry legality"). In overlay
discovery (main_psx.cpp --overlay), after walking the seed roots, every DIRECT
CALL (jal) target that lands strictly INSIDE an already-discovered function is
registered as an ALIAS ENTRY of that host via the existing alias machinery
(materialize_alias_groups). So when a host's discovered range absorbs a separate
routine (e.g. the crt0 `jal 0x50B08; break` falling through into the init-array
at 0x89788), the host's entry-switch now routes a dispatch of that PC to the
correct absorbed interior block and runs it NATIVE — instead of depending on the
PC having been captured, or falling closed to the interpreter.

Derived purely from static jal evidence (jal encodes a code target, so it never
mints code from data); does not depend on execution capture. Preferred over an
explicit range-clip because the absorbed code IS the right code — it just needed
a routable entry. The static two-phase FunctionDiscovery already enforces
hard_cap = next-entry, so only the overlay path needed this.

VERIFIED: regenerated cache (codegen cg4_42de29a8) emits `case 0x80089788u` in
func_800896E0; cps_probe for 0x89788 drops 1021 -> 0 (now main-chain native, not
the broken CPS find-by-range default); freeze stays gone, 60fps sustained. Pairs
with the fail-closed entry guard (bd60381), which remains the safety net for any
genuinely-unrouted interior PC.

Framework-wide: regression-test Tomba 1 / MMX6 / Ape before any master merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:23:15 -07:00
Matthew Stanley 2b5c32d564 docs(tomba2): FMV freeze FIXED + verified (fail-closed entry guard)
Record the implemented fix and verification at the top of the freeze writeup:
overlay CPS entry-switch fails closed on a foreign interior PC -> sanctioned
dirty-RAM interpreter (ABI v6). Verified native passes the freeze (frame
1823 -> 18001+, 60fps, i_mask=0x0D, reval_crc_miss=0). Notes the optional B/C
completeness layer (clip overlay ownership range in the capture-driven overlay
discovery path) and that the static two-phase discovery already enforces
hard_cap=next-entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:23:15 -07:00
Matthew Stanley ff714d1ee1 diag(overlay): confirm Tomba 2 FMV-freeze root cause (break range pollution)
Adds overlay_cps_probe (debug cmd) to record what the CPS interior-continuation
dispatch decides for a watched PC. Used it to CONFIRM the root cause:

func_80050B08 calls jal 0x89788 (init-array). 0x89788 isn't a registered entry,
so overlay_find_by_range(0x89788) runs -- 6 loaded candidates have a range
containing it -- and returns func_800896E0 (the crt0 bootstrap), whose range
wrongly spans 0x89788. 0x89788 is not a case in its entry-switch, so default
re-runs the bootstrap from the top: InitHeap/A0 spin, VBLANK-masked deadlock.

func_800896E0's range covers 0x89788 because function_discovery.cpp:493-499
treats BREAK as fall-through; the bootstrap's "jal 0x50B08; break" (break guards
the main-returned case) falls through at 0x89784 into the separate function at
0x89788, absorbing it. So the early break-no-op finding WAS the root -- damage is
at discovery/range time, not runtime break execution. Interp is immune (executes
0x89788 bytes directly).

Fix options + recommendation (clip function range at next entry / don't cross a
function boundary on break fall-through) documented in
docs/TOMBA2_NATIVE_FMV_FREEZE.md. Diagnostic only; no fix applied yet (needs
regression on Tomba1/MMX6/Ape). Workaround: overlay_native_block 0x80050B08.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:22:37 -07:00
Matthew Stanley 0acc6f2dbb feat(overlay): per-function native-disable for divergence bisection
Adds a phys-keyed blocklist in overlay_loader.c that forces a named overlay
function through the sanctioned dirty-RAM interpreter (it still RUNS — not
skip/stub/HLE), as if s_native_exec were off but for that one function only.
Exposed as the `overlay_native_block` debug command (add/clear/report). This is
the localization tool for native<->interp divergences (e.g. the Tomba 2 FMV
freeze): binary-search which compiled overlay function's native execution flips
the bug, without rebuilding or losing candidate visibility. Runtime/tool-only.

Used it to localize the Tomba 2 end-of-Whoopee-Camp FMV freeze to
func_80050B08's OWN native body (blocking it passes the freeze with callees
still native; all 33 jal targets + continuations verified clean). Narrowed to
the main-loop region with a strong interior-entry suspect at 0x80050CE8.
Findings in docs/TOMBA2_NATIVE_FMV_FREEZE.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:22:36 -07:00
Matthew Stanley e8ded2f306 docs(tomba2): root-cause the native-overlay FMV freeze (frame ~1823)
Whoopee-Camp-end freeze with the warm overlay cache is a DETERMINISTIC
codegen divergence, not a timing race -- proven via the sandboxed
native<->interp shadow-compare (overlay_diff/shadow_dump): both engines
run from identical CPU+RAM state, no IRQ/timing, and native still diverges.

First divergence (overlay_shadow_detail) is fn 0x800896E0: native t1=0xE4
(=0x39<<2, the A0 dispatcher`s sll t1,t1,2) / t0=0xBFC01E24 (A0_table[0x39],
InitHeap) vs interp t1=0x39 / t0=0x80000000. Native wrongly routes through
the A0:0x39 dispatch path interp doesn`t take -> re-inits every spin-loop
iteration -> endless A0-vector dispatch with VBLANK masked -> deadlock.

Also documents a separate latent CLASS bug: the recompiler does not model
the R3000A load-delay slot (strict_translator.cpp:830) while the dirty
interp does; 50 load-delay hazards found in this game`s overlays (0
control-flow) -- not this freeze, but a real divergence class to class-fix.

Same ~frame-1828 hang a prior session believed fixed by 47f3f15; it was
masked by interp fallback (audit-failed region), the clean warm cache
re-exposes it. Exact diverging instruction + recompiler fix still pending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:21:50 -07:00
Matthew Stanley 8e60a419a9 docs(overlay-cache v2): interior-entry fragments IMPLEMENTED + runtime-stable
P1+P2 done (c5efd70). Decoupled post-loop fragment pass; validated by probe
(clean mid-entry island for 0x80083094), full run (34/47 executed orphans at
0x38000 incl 0x80083094 as isolated shards, despite that region's normal compile
audit-failing), and a fresh runtime run (20 DLLs / 264 native / crc_miss=0 — no
divergence). Open: clean good-variant warm for end-to-end FMV/title speed (messy
accumulated captures audit-fail the region, losing hot NORMAL funcs); optionally
extend isolated fragments to executed normal funcs; P3 host recovery optional.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:21:50 -07:00
Matthew Stanley a8a0b0da88 docs(overlay-cache v2): ChatGPT-validated plan — interior-entry "island" fragment
Architecture review verdict for the FMV-driver coverage gap: DON'T recover the
host. The regression was mixing speculative recovered-host roots into the trusted
all-roots region DLL (shared failure domain). Fix = three isolated build classes
(REGION_TRUSTED / RECOVERED_HOST_SPECULATIVE / INTERIOR_ENTRY_FRAGMENT); for the
FMV, compile an interior-entry ISLAND that enters at the executed PC 0x80083094
over a worklist-discovered CFG (not linear-to-boundary), its own DLL, keyed by
covered ranges + covered-byte hash. Mid-function start is safe iff codegen makes
no function-entry assumptions (no synthesized frame; out-of-island branches are
dispatcher exits, not audit failures). Phased plan recorded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:21:49 -07:00
Matthew Stanley 17caf25dd5 docs(overlay-cache v2): host-recovery attempt reverted — needs compile isolation
Attempted orphan-interior host recovery (walk back to the host start, root it so
the FMV driver 0x80083094 compiles). It REGRESSES: adding the recovered host as a
root to the region's shared seed list (even gated to executed/resident orphans)
makes the recompiler emit C that fails the strict generated-C audit, and because
all roots in a region compile into ONE DLL, one failure skips the whole region ->
the previously-good 0x38000 region stopped compiling. Reverted to clean
checkpoint. Robust path = isolate each recovered host (+ its interior aliases)
into its OWN compile unit / DLL so a bad recovery fails alone (loader already
content-matches per function across DLLs), or audit-validate each host before
including it. Dedicated effort; recorded for next session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:21:49 -07:00
Matthew Stanley e6bcdb57a3 docs(overlay-cache v2): session outcome — native stabilized; FMV-driver discovery wall
Records the three fixes that made native overlay execution stable through boot
(region_start load fix bb62b20, alias-rollup 47f3f15, timer write8 6004ced) and
the remaining FMV-speed blocker: the hot FMV driver 0x80083094 is a
DISPATCH_INTERIOR orphan (no discovered host -> not compilable -> interp-only).
Plus the next-step plan: recover the host by walking backward from the interior
PC to a callable-start signature, root it, and let the interior become an
alias-entry into it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:21:49 -07:00
Matthew Stanley aab66e6699 docs(overlay-cache v2): record region_start fix + alias-keying divergence finding
The slowness root cause (region_start walkback overshoot) is fixed (bb62b20):
native funcs 14 -> 790. Remaining blocker = a native-vs-interp divergence at
0x80051FA4 that hangs the boot when native runs.

0x80051FA4 is a CPS CONTINUATION point inside host 0x80051F80, and the recompiler
emits BOTH a full host func_80051F80 AND a separate epilogue-only alias body
func_80051FA4 -- with the SAME .ranges range (R 80051F80 34) and SAME code_crc
(5CE0B05E). So two candidates cover identical bytes and a continuation to 0x51FA4
can resolve to the epilogue-only alias instead of the full host. Hypothesis:
aliases/interior entries should roll up to (dedup into) the parent host shard.
Recorded for the recompiler fix + ChatGPT architecture review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:18 -07:00
Matthew Stanley f89dadc22e docs: overlay cache v2 rearchitecture spec
Audit of the current overlay-cache subsystem (loader, capture, autocompile,
compile tool) plus the design target. Key finding: the correctness model is
sound (dispatch re-validates live RAM vs per-function code_crc every call, so
no stale execution) -- the problems are efficiency: whole-region-CRC build
keying causes redundant DLLs, a per-dispatch full CRC32 taxes warm code, and
capture/dlopen/JIT run on the main thread. Phased plan P1-P5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:17 -07:00
Matthew Stanley 1a65b5c92a docs: CORRECTION — CD streaming is NOT the immediate blocker (directory traversal works)
The fixed CD command trace shows single-sector data reads (PVD16->root22->subdir373) all work; the
game stalls AFTER the directory read at the FILE-LOAD DECISION, issuing no further CD command. The
CD-audio/XA streaming is still missing (oracle DMA3-streams, recomp doesn't) but the game never
reaches it, so it's downstream, not the gate. Corrected blocker: a state the game tests after the
dir read diverges (no SetLoc+ReadN for the next file). Supersedes the premature 'CD streaming root
cause' conclusion.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:15 -07:00
Matthew Stanley 7ea75991a8 docs: CONFIRMED root cause — incomplete CD STREAMING path (Tomba 2 is first to exercise it)
The splash softlock = the CD streaming/XA/CD-DA + DMA-3 audio path being incomplete. Confirmed: the
game's CD irq handler at pc=0x8008B6F4 spins on [0x800AC2C4]=0x1F8010B8 (DMA3/CDROM CHCR) bit
0x01000000, waiting for the CD-streaming DMA to complete; the recomp gets empty reads (no stream to
0x1Axxxxx like the oracle's ~1.1M DMA3 transfers via game routine 0x8008DB88). Directory read is
perfect (full file list) — not a CD-fidelity/game-code bug; the game just never loads the next file
(0x107xxx advance task) because the streaming sequence stalls. cdrom.c Play(0x03) comment confirms
'Red Book sample output is not yet decoded — silent for now'. Tomba 1/MMX6/Ape never streamed CD audio
(SPU-voice/SEQ, single-track data discs) so this path was scaffolded-not-built. FIX = build the
Architecture-A CD streaming hardware-sim: DMA3 delivers requested sectors continuously (data->game buf,
XA->SPU decode, CD-DA Red Book output + position/end), so the irq DMA-wait completes and the splash
advances. STUBS_TO_FIX S3/S4/S5 family, now actually exercised.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:15 -07:00
Matthew Stanley b6fbf24c14 docs: directory parsed perfectly + NEW clue — recomp's CD-DA/XA splash-audio streaming fails (empty reads)
Recomp's 0x104368 subdir (LBA 373) decodes to the full file list (A00-A0L.BIN, CRD/DEMO/GAME/OPN/
SOP/START.BIN, valid extents) — dir parse is perfect; recomp just never LOADS the next file. The
LBA-24/0x41800 512KB load holds the splash code; the 0x107xxx advance task is a separate file that
never loads. NEW CLUE: recomp CD froze ~frame 1987 (stuck 4379), cdrom_trace 100% flooded with dma
empty-reads (val=0, func=0x00000E10 exception handler, pc=0x8008B6F4); recomp CD dests only
0x41800/0x104368 while ORACLE hammers DMA3 to 0x1Axxxxx (~1.1M, routine 0x8008DB88). => recomp is NOT
streaming the splash CD-DA/XA audio (Whoopee jingle, multi-track TRACK02/XA) — getting EMPTY reads in
the CD interrupt handler. Hypothesis: splash->attract is gated on the audio stream; empty audio ->
gate never fires -> next-file load (0x107xxx advance task) never issued -> state stuck 0. Ties to the
multi-track CD-DA/XA streaming path in the CD/VBlank handler (same neighborhood as frame-1997). BLOCKING
TOOLING (Rule15): cdrom_trace ring evicted by dma empty-reads — edge-suppress 'D' to see CD commands.
Next: fix the flood; investigate multi-track CD-DA/XA audio read (empty from pc=0x8008B6F4); confirm the
splash gate on audio.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:14 -07:00
Matthew Stanley a88959c1e0 docs: sub-case (b) confirmed — directory read is VALID, it's a file-search/load-decision divergence
Decoded recomp 0x104368: a well-formed ISO9660 directory (rec0 {len48, extent_LBA 373, size 2048,
flags 0x02 dir, name '.'}, rec1 extent 22 '..', more). So the recomp read+parsed the LBA-373 directory
CORRECTLY — CD fidelity + data are fine. The divergence is AFTER the directory read: the code that
walks the dir to find the target file and issue its load (->0x107xxx) takes a different path in the
recomp and never issues the ReadN. Next: find the routine that issued the LBA-373 read (wtrace CD cmd
reg from boot -> PC/ra), disasm live, compare its post-read branch + tested values vs oracle (likely a
filename compare / dir-walk early-exit / 'already loaded' or event-flag check / differing return value).
DMA3 trace also showed the oracle hammers CD-DA/XA audio DMA (~1.1M MADR writes to 0x1Axxxxx via game
routine 0x8008DB88) during the splash; the recomp's CD dests are only 0x41800/0x104368 (may also not be
streaming the splash audio — possibly related).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:14 -07:00
Matthew Stanley 588bb679fc docs: recomp NEVER issues the 0x107xxx CD read — code-path/task divergence in the load decision
Restarted recomp, dumped cd_read_log: total=2651, dest range 0x41800..0x104368, LBAs {16,18,24,373},
covers_0x107xxx=False. Recomp does the directory lookup (LBA16/18/373->0x104368) but never issues the
file load to 0x107xxx, then goes quiet (stuck). NOT a DMA-targeting/CD-fidelity bug (the 0x41800 load
proved raw reads land correct advancing code) — it's a code-path/TASK divergence in the file-load
DECISION. The state-0 handler 0x80050D44 only renders+loops, so the load is issued by another
task/callback during the splash that the recomp doesn't run. Likely candidates: (a) per-frame
scheduler/event chain off the game VBlank handler (same neighborhood as frame-1997 HookEntryInt) so
the scene-loader task never runs; (b) CD-complete-event callback; (c) a thread/TCB never resumed.
Next: trace the oracle's CD cmd stream (@0x1F801801) ~frame 2289-3960 to find the SetLoc+ReadN for
0x107xxx + the issuing PC, then check if the recomp executes it / how its event-callback chain differs.
Tooling debt: beetle lacks cd_read_log + cdrom cmd trace; raw-socket cd_read_log parse fails (use client).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:14 -07:00
Matthew Stanley bc7faba43d docs: loader hunt result — splash-advance code is a LATE CD DMA load the recomp never gets
wtrace 0x80107000-0x80108000 on a fresh oracle: only BIOS zeroing (0xBFC0D864) hits via CPU
stores; the real 0x80107268 code is ZERO at oracle frame 2289 but present by 3960 with ZERO
CPU stores carrying it => installed by CD DMA (ch3 CD->RAM), LATE (mid-splash ~frame 2289-3960),
not at initial boot. So CD is NOT fully ruled out: the early 0x41800 load works but this later
0x80107xxx overlay DMA is the one the recomp never receives (region zero even at frame 78k).
Refined model: the state-0 splash loop initiates/waits on a 2nd CD load that brings the advance
task; on HW the DMA completes and drives state 2<->1; recomp's never lands so state stuck 0.
Next: fix cd_read_log cross-tooling (Rule15), restart recomp w/ from-boot CD logging to see if it
issues the 0x107xxx read at all, and trace the state-0 loop's CD cmd+DeliverEvent right before the
load on both. cd_read_log cross-compare currently blocked (beetle lacks it; recomp dump empty once).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:14 -07:00
Matthew Stanley d388767901 docs: ROOT CAUSE of splash softlock — missing splash-advance code at 0x80107xxx
Oracle reverse-writer-hunt (wtrace on the state byte from boot): [0x1F80019C] ping-pongs
every frame on HW — 0x80107268 writes state=2 (advance trigger), 0x80050DA8 writes state=1.
The recomp NEVER advances because 0x80107200-0x8010728C is ALL ZEROS in the recomp vs the
real splash-advance function on the oracle. High-RAM 0x80100000+ is partially-loaded and
divergent (recomp has 0x104368/0x10A000 with different content; 0x107000/0x110000 are zero
vs real code). So the softlock is a LOAD/RELOCATION divergence during boot: the recomp
failed to bring the 0x80107xxx overlay into RAM. JIT/cache ruled out (pure interp:
dispatch_native=0, sljit live=0, autocompile off). Next: loader hunt — wtrace
0x80107000-0x80108000 installs from boot on both + early CD-read log, to find the load
mechanism and why the recomp skips/relocates it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:14 -07:00
Matthew Stanley bc0737aec9 docs: splash->attract plan (Recomp GPT reverse-writer-hunt) + execution findings
ChatGPT (fresh Recomp thread) gave a reverse-writer-hunt plan: catch the oracle AT the
transition, dump the pre-transition scratchpad write history, find the writer PC + predicate.
Top hypothesis: a GPU/DrawSync/fade-completion or CD/event-callback flag (state stuck while
VBlank pacing + CD load are healthy). Traps: verify committed-PC in the trace + scratchpad
byte/halfword endian.

Execution so far: confirmed the splash state-0 handler (0x80050D44) only renders+loops and
NEVER advances [0x1F80019C]; scanned all loaded code 0x40000-0x92000 and found NO writer that
sets the state byte to 2 (only init->0 and the state-2 handler->1) => the 0->2 advance is
external/event-driven. Confirmed 0x80050CE4 is a common VSync-wait loop both scenes hit
(oracle in attract also samples it). Decisive next step: oracle write-watch on [0x1F80019C]
from boot to catch the transition writer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:13 -07:00
Matthew Stanley 28d5719ed7 docs: splash->attract is first-divergence (CD RULED OUT) + ~17k crash note
Investigated the splash softlock: the attract data loads CORRECTLY (consecutive CD-DMA
dests hold different valid MIPS code; read_msf reached LBA 374). The cd_read_log 'lba'
field misleadingly logs the SetLoc target (24) not the per-sector position — a tool bug
(Rule 15) that cost a wrong 'CD stuck at LBA 24' hypothesis. So the splash->attract
divergence is a true first-divergence in the scene-transition state machine ([0x1F8001A4]
stuck at 0), upstream of the common 0x80050CE8 loop both recomp and oracle reach. Next:
oracle PC-trace from the splash. Secondary: ~frame-17k silent crash (~8e9 dirty insns)
- possible host-stack growth from the frame-1997 fix's re-dispatch on nested RFEs; also
a regression-risk signal for the shared path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:13 -07:00
Matthew Stanley 09398645b9 docs: frame-1997 FIXED status + new frontier (splash->attract divergence)
Tomba 2 runs past the frame-1997 exit (splash renders, frame 16k+, aborts=0). New
cascade frontier: recomp lingers on splash with CD idle while the Beetle oracle on the
same disc reached the attract/intro gameplay; both at idle PC 0x80050CE8, so the recomp
tracks the oracle's code but diverged in game state (oracle triggered the attract CD
load, recomp did not). Pending gates documented: T1/MMX6/Ape regression + scaffolding
cleanup before master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:13 -07:00
Matthew Stanley 9940ab2c0f wip(tomba2 exc): observability + corrected frame-1997 diagnosis (longjmp-to-outermost, NOT async)
Implemented + instrumented the consult's Option-A async-resume idea; the data REFUTED
the in_exception=0 async model:
- freeze_check now reports async_rfe_set/fire, reach_dirty/traps, reach_async,
  dirty_safe_resume_pc, async_rfe_resume_pc.
- Measured: reach_dirty=19921 (~1/exception), dirty_in_exc0=0, async_rfe_fire=0 ->
  the sentinel 0x80000048 ALWAYS arrives at the dirty gate with in_exception=1 and
  takes the longjmp. There is no in_exception=0 dispatch of the sentinel.

Corrected mechanism (docs/TOMBA2_WHOOPEE_BAIL.md): the game's ReturnFromException
longjmp lands in psx_check_interrupts (jmp_val==1), restores GPRs and returns with
cpu->pc=0; for ~19920 exceptions it lands at a NESTED dispatch (game continues), but
for the failing one it unwinds to the OUTERMOST trampoline where cpu->pc=0 reads as a
clean exit. It's a longjmp-landing-depth bug in the 'resume via host return' model,
not an async-resume bug.

Latches added (g_dirty_safe_resume_pc around the 3 dirty pumps, g_async_rfe_resume_pc
at dirty-safe exception entry) + the sentinel-gate redirect scaffolding are INERT
under the corrected model (never fire) — kept as tooling + scaffolding for the
candidate fix. Builds clean; behaves identically to pre-fix (exit at frame 1997, no
regression). wt/tomba2 only; not on master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:13 -07:00
Matthew Stanley 845b39489a docs: Recomp GPT consult response — Option A dual-mode EPC design + safety rails
ChatGPT converged on Option A as a scoped dual-mode exception system:
- dirty-interp-interrupted -> EPC=real guest PC, architectural RFE resume, no host
  GPR restore (trust guest TCB save/restore)
- compiled-native-interrupted -> sentinel EPC + host GPR restore + longjmp (UNCHANGED)
Key rails: make the sentinel LESS permissive (latch RFE_SENTINEL_ESCAPED, never
silent pc=0); block nested dirty-pump entry; delay-slot precondition (MMX6 BD/EPC);
dispatch_depth save/restore both modes; paired EXC_ENTRY/RFE_EXIT confirmation ledger.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:12 -07:00
Matthew Stanley 8037fe5c40 docs: confirm frame-1997 root cause with live data (EPC sentinel in TCB) + fix design + consult package
Turned the prior session's inference into proof via PSX_EXIT_HALT live state:
- dispatch-tail ring shows game VBlank handler 0x80085D8C -> B0:0x17
  ReturnFromException -> pc=sentinel 0x80000048 -> exit.
- in_exception=0 at the failing RFE (freeze_check + GATE_COOLDOWN, two ways).
- TCB save_area off128 (EPC)=0x80000048 sentinel, while ra/sp are the correct
  interrupted ctx -> the sentinel-in-TCB is the smoking gun.
- handler is HookEntryInt-installed (B0:0x19 @0x80085D3C) and self-drives RFE.
- dirty-interp pump commits cpu->pc before psx_check_interrupts -> a precise
  guest resume PC exists but is discarded for the sentinel.

Fix design = scoped EPC de-overload (Option A) in docs; consult package in
docs/TOMBA2_EXC_CONSULT.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:12 -07:00
Matthew Stanley e95371527c docs: pin Tomba 2 frame-1997 frontier (game HookEntryInt + ReturnFromException vs EPC-sentinel model)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:12 -07:00
Matthew Stanley 90bb5f95f7 interrupts: preserve dispatch-depth across the async exception handler
With Tomba 2's overlays now on the local-flow path (prior commit), the
per-4096-insn interrupt pump delivers the exception handler from a NESTED
dispatch (g_psx_dispatch_depth > 0). psx_check_interrupts reset the counter to 0
around the handler but never restored the interrupted code's nesting, and a
ReturnFromException longjmp skips the handler frames' --depth decrements. The
outer frames then unwound below zero (dispatch_depth -> -1), the `outermost`
test misfired, and the top-level dispatch returned to PC=0 ("execution
completed" abnormal exit at frame 1997).

Save the interrupted code's depth before the handler, run the handler as its own
outermost context (0), and restore the saved depth afterward. Depth-transparent
for callers that were already at depth 0 (Tomba 1 / MMX6 / Ape), so no behavior
change there.

Resolves: dispatch_depth stays correct (0, not -1) across delivered interrupts.
NEW failure point (next in cascade, NOT addressed here): a genuine guest-state
divergence at frame ~1997 — overlay chain 0x80089784 -> 0x80080FC4 -> 0x800499E8
-> 0x80050CEC does a jr to a null register (final_ra 0x80050CC8, PC=0). Needs an
oracle first-divergence pass. See docs/TOMBA2_WHOOPEE_BAIL.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:11 -07:00
Matthew Stanley bb73e4810c docs: CORRECTION - tripwire refutes exception theory (it's bail/flatten stack)
The $ra->1 tripwire + first-bail in_exc latch (d965d6e) refuted the
exception/EPC model: in_exc=0 everywhere; $ra->1 is at a normal epilogue
'lw $ra,0x34($sp)' (0x8008B018) loading a corrupt saved-$ra because $sp is
wrong after the first bail's non-local B0 return. Real bug = control-flow/
stack-contract in the bail/flatten of a B0 non-local (tail-call) return
(recomp-debug Class 3), not exception/RFE/EPC. EPC-separation fix marked refuted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:10 -07:00
Matthew Stanley cc544fdae9 docs: ground Tomba 2 freeze root cause (overlay/async-interrupt + EPC overload)
Refined via binary truth + actual code + recomp-debug methodology + ChatGPT.
0x80085900/0x8008AE50 are NOT in the boot EXE (PS-EXE t_size=0x28800) -> disc
OVERLAY run by the dirty-RAM interp. The bug is the dirty-interp/overlay<->
compiled-BIOS boundary when an async interrupt (now delivered by the pump fix
0a6920a) lands mid-call. interrupts.c:355-384 overloads guest EPC as a host
longjmp sentinel (0x80000048), colliding with the BIOS's architectural EPC/TCB
use -> the interrupted continuation's $ra is clobbered to 1 -> wild-return bail
storm. NOT a broken BIOS exception handler (recomp-debug 'platform infra is
correct' trap). Fix design: separate host exception-return metadata from guest
EPC; restore host call-contract on RFE; or defer async IRQs to a safe point.
Confirm-first: $ra->1 tripwire.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:10 -07:00
Matthew Stanley 90585eafab docs: Tomba 2 Whoopee-Camp freeze root cause (wild-call-contract bail storm)
Clear writeup of the splash freeze: what's ruled out (all hardware-sim healthy
- interrupts/timer/CD/events), the root cause (wild-call/return-contract
corruption at the exception<->B0 boundary -  clobbered to 1 on RFE resume of
an interrupted wild/dirty call -> infinite bail loop = the 68k/frame spin), the
captured evidence (trigger 0xB0->0x8008AE50 @frame 1823, register state), open
questions, and fix direction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 23:16:09 -07:00
Matthew Stanley 01b4d91e8a WIP: forward-port Ape Escape framework (dev/ape-escape) onto master
Comprehensive 3-way merge of the dev/ape-escape branch into master for
the isolated wt/ape-escape worktree. Brought in every complementary ape
change automatically (dma.c, memory.c interrupt byte/half lanes,
function_analysis discovery, main_bios vector seeds, iso_reader
iso_sector_count, config_loader, crash-trace fields, headers). Resolved
the 13 conflicts by rule:

  - CPS dispatch / recompiler (code_generator, full_function_emitter,
    dirty_ram_interp, traps, main_psx) -> KEEP MASTER (evolved system).
  - CD model (cdrom.c) -> take ape (the model that booted Ape).
  - interrupts.c/.h -> reverted to MASTER (ape's interrupt-delivery is
    dispatch-coupled; the merged Frankenstein stormed).
  - sio/debug_server/freeze_heartbeat/main.cpp -> keep master (evolved).
  - crash_trace.c -> master (ape freeze-dump fns omitted to avoid the
    debug-ring cascade).

STATUS: does NOT boot yet. ape-cdrom + master-interrupts derails to
PC=0 at frame ~2092 (insta-close); ape-cdrom + ape-interrupts storms
(297M exception re-entries) at frame 2077. Root finding: ape's CD
anti-storm is coupled to ape's interrupt-delivery + call-depth + EPC
model, which master replaced with CPS continuation-passing, so the CD
subsystem does not forward-port as pure hardware simulation. Preserved
as the reference state for a design consult on bridging the two models.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 15:55:23 -07:00
Matthew Stanley 7281319005 recursion: make CPS the DEFAULT (opt-out via PSX_CPS=0)
The continuation-passing leak fix (RECURSION_BUG.md §25) is validated (Tomba boots+plays,
ce_profile flat through a 56.9k-frame soak, full overlay CPS). Flip the gate in all three
emitter sites so a normal regen emits CPS; PSX_CPS=0 is the legacy escape hatch. Verified:
default regen -> CPS markers present; PSX_CPS=0 -> none; default-CPS Tomba boots clean, leak flat.
Doc §25.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 14:10:44 -07:00
Matthew Stanley 5c893d472a overlay(async): gcc-only on gcc boxes (mutually exclusive with sljit) + async-compile is built
The user-requested async overlay tiering (interp-now / compile-async / native-next, gcc-if-present
else sljit, always async, file-based handoff) is LARGELY ALREADY IMPLEMENTED — overlay_capture.c
(autocapture on interp pressure) + autocompile.c (background compile via a configured command,
toolchain-probed, watcher thread, file-based) + the code_provider abstraction. It just needed
CPS wiring.

This commit:
- overlay_loader_apply_live_policy: when the active backend is gcc, force sljit-live OFF so a gcc
  box is gcc>interp (NOT the previous complementary gcc+sljit). Toolchain-less boxes stay
  sljit>interp. PSX_OVERLAY_SLJIT_LIVE still overrides. ("mutually exclusive" per user.)
- ASYNC_OVERLAY_COMPILE.md: re-headed to reflect the feature is built+validated under CPS, with
  the deltas this session added (compile_overlays --cps from §25.5, the gcc-only policy, the
  dev-config overlay_autocompile_cmd --cps).

VALIDATED under CPS: autocapture fires -> background compile_overlays --cps produces CPS overlay
DLLs -> loader loads them -> dispatch_native rises; gcc box shows 0 sljit shards; ce_profile flat.
The native overlay cache auto-builds as you play (each scene compiles on its first sustained
visit). PRODUCTION game.toml overlay_autocompile_cmd still needs --cps appended when CPS ships.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:54:57 -07:00
Matthew Stanley d99323c72b docs: async background overlay compilation — implementation plan
Build plan for the next session: interp-now / compile-async / native-next overlay tiering
(gcc-if-toolchain-present else sljit, mutually exclusive, always async). FILE-BASED handoff —
the worker writes .dll/.sljit artifacts; the dispatch fiber loads them via the existing
try_load_region / deserialize paths, so the candidate table stays single-threaded (no race).
Motivated by the CPS title screen being interp-bound (12.2M interp vs 14.8K native) because its
overlays aren't gcc-compiled + the sync sljit JIT-on-miss hitches the dispatch thread.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:40:45 -07:00
Matthew Stanley 2d468ee5b8 recursion(stage3): overlay (sljit + gcc-DLL) CPS parity — full native overlay coverage under CPS
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).

Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.

sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
  JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)

gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
  per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
  psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
  inside a registered candidate's code range (a tail-transferred call's continuation),
  overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
  to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
  This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.

Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:35:31 -07:00
Matthew Stanley 94a1966444 recursion(fix): universal continuation-passing — FIXES the idle-freeze host-stack leak (BIOS+game)
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.

VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).

KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.

Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
  tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
  tail-transfer (jalr captures rs before the link, alias-safe); each call's
  return point registered as a dispatchable continuation; entry-switch (keyed on
  cpu->pc) at the top of every function AND every overlapping-alias body. 0
  psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
  (case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
  syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).

Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
  Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
  fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
  ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
  (un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
  on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).

Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 12:45:02 -07:00
Matthew Stanley e28c2a0ff9 recursion(fix): WIP universal continuation-passing call/return (gated PSX_CPS) — Stage 1 emitter
The universal fix for the idle-freeze host-stack leak (RECURSION_BUG.md §25): make
the recompiler emit guest calls as TAIL-TRANSFERS driven by the trampoline + the
guest's own $ra/$sp, instead of nesting one psx_dispatch_impl per guest call frame.
This makes the host stack incapable of mirroring unbounded guest re-entrancy — the
architectural root of the leak — for ANY game. Perf cost accepted ("right before
fast").

Stage 1 (full_function_emitter.cpp), GATED behind gen-time env PSX_CPS:
- jal/jalr (orphaned-delay + pending-delay paths): emit `cpu->pc=target; return;`
  (tail-transfer) + register the return address as a dispatchable continuation,
  instead of nested psx_dispatch + psx_call_contract. JALR captures the target reg
  BEFORE the link write (rd==rs alias-safe).
- jr $ra (all 3 sites): publish `cpu->pc = cpu->gpr[31]; return;` under CPS so the
  trampoline dispatches the caller's continuation (legacy `return;` relied on the
  nested-dispatch C-return). Required — without it the BIOS exits at frame 0.

GATING PROVEN SAFE: regen+build WITHOUT PSX_CPS boots Tomba normally — default
codegen byte-unaffected; CPS fully opt-in.

OPEN (CPS path only): with PSX_CPS=1 the BIOS boots main() then exits at frame 0
(execution completed, PC=0). Root cause (high confidence): the trampoline must
dispatch mid-function continuation addresses, but the dispatch table holds only
function entries → a return to a continuation misses → psx_unknown_dispatch →
pc=0 → exit. NEXT: add registered continuations to the dispatch table (map
continuation_addr → containing func), then re-regen PSX_CPS=1 and run the §25 gate
(ce_profile flat + oracle-identical + 15-min soak + dwarf->overworld). Stages 2
(interp interp_enter_compiled surface) + 3 (sljit/overlay parity) follow.

Safe revert point for the whole feature: commit ae47e69.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 08:25:39 -07:00
Matthew Stanley ae47e69902 recursion(oracle): oracle disambiguation DONE + SP-fntrace instruments; verdict = §11 universal fix (docs §23–§24)
Oracle disambiguation (RECURSION_BUG.md §23–§24). Traced one func_8001A954
invocation on real HW (psx-beetle) vs the recomp in the IDENTICAL Tomba idle
state (New Game village dialogue; object 0x801FD950, state/scene/jumptable=0).

VERDICT: real HW's per-frame chain is BALANCED — func_8001A954 is entered once
per frame from main_loop (JAL 0x8001A630, sp=0x801FE3E0) and RETURNS once per
frame (jr ra @0x8001A9E8 -> 0x8001A638, SP fully restored); no re-entry, the
overlay 0x80063864 never calls back (refutes §21/§22's "call-cycle"). The recomp
runs the same guest flow and the guest SP unwinds identically, but the HOST stack
leaks +1.17 KB/frame (ce_profile, live-reconfirmed: 20099->22215 KB / 1807 frames)
because the per-frame compiled chain (func_8001A954->AC00->B2B4->B5A8->func_80046264,
emitted as DIRECT func_X(cpu) host calls) re-enters itself via func_80046264's
CD-DMA-dirty hop and never unwinds.

native_stack early-trip (PSX_STACK_GUARD_KB=6144, frame 7592, 6 MB) pins the
leaking host cycle. Cross-checked: xprobe mixed_depth bounded at 6, dirty-insn/
dirty-block show NO literal func_8001A954 transfer, and a954=1/frame is a SURFACED
guest tail-transfer (j/jr) to func_8001A954 that the oracle proves real HW never
makes — i.e. there is both (A) a correctness divergence (spurious per-frame
re-entry) and (B) the architectural host-mirroring limitation that amplifies it.

Fix scope CORRECTED from §23's "quick contained runtime patch": the UNIVERSAL fix
is §11 — route in-range guest calls through the depth-managed psx_dispatch_impl
trampoline instead of mirroring them as direct host calls, so the host stack stays
bounded regardless of guest re-entrancy (real or fabricated). §12 (CD-DMA dirty
precision) and fixing the specific divergence are surgical mitigations, not
universal. Performance loss accepted for now (static recomp: right before fast).

Instruments added (runtime-only, no regen):
- SP capture in the fntrace control-flow ring on BOTH backends (GPR[29] read live):
  psx-beetle (beetle_libretro.cpp + beetle_debug_server.c) and the recomp
  (fntrace.h/.c + debug_server.c) — enables the func_8001A954 SP/RA-lifecycle oracle.
- psx-beetle default debug port 4380 -> 4382 (+ --port N override): 4380 collided
  with psxref and cdirecomp/CdiRuntime.exe; two LISTENers on one port silently
  misroute connections (the phantom "oracle keeps dying" early in the session).

Oracle tooling: the control-flow oracle is psx-beetle (Beetle interprets -> records
every J/JAL/JR/JALR), NOT psxref (no PC trace; the assumed emu_step/emu_trace_addr
never existed). Loads Tomba via: psx-beetle.exe --disc tomba.cue <bios> --port 4382.

Next: implement the §11 universal fix (codegen + full regen).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 08:03:04 -07:00
Matthew Stanley fd2570ff13 recursion(doc): §22 — fix (A) interp-call surface validated INSUFFICIENT, reverted
Pinned the exact leaking crossing (xprobe detail): entry to func_8001A954 is a single
JAL, src 0x8001A630 (the main loop's only call site), once/frame. Implemented (A):
surface interp JAL/JALR to game-text targets to the owner trampoline (gated by
psx_game_address_in_text so overlay-native keeps the nesting contract). §19 result: leak
STILL CLIMBS (~0.875 KB/frame, down from 1.17) -> reduced, not eliminated -> REVERTED.

It is a guest CALL-CYCLE (main_loop 0x8001A51C <-> func_8001A954): surfacing the interp
side just relocates the nest to the compiled-side jal back to the main loop. Can't fix a
call-cycle by surfacing one crossing. Leak is ~1 host frame PER FRAME (not per cycle
iteration) -> the loop mostly returns; exactly one call/frame fails to unwind. Real HW
idles forever (balanced), so either we mishandle one specific return/frame (Hyp C,
contained) or it needs the §11 continuation-passing redesign. DECISIVE NEXT = oracle
(psxref): does real HW return from func_8001A954 to 0x8001A634 each frame, or never?
Stop surfacing single crossings (block-loop §20 + JAL/JALR §22 both dead ends).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 00:57:46 -07:00
Matthew Stanley add57f5560 recursion: SOLVE root (gradual per-frame host-stack leak) + instruments + pin exact crossing
The ~14-min Tomba idle freeze is a gradual per-frame host-stack LEAK (not a
within-one-frame spiral, not the gradual-marking theory). Confirmed via a new
compiled-entry depth profile (§19): per-frame MAX guest-stack-used climbs dead-linear
+1.17 KB/frame from gameplay start to 48 MB on the 64 MB main-thread stack -> native
stack guard -> freeze. Once per frame an interpreted overlay (reached via
func_80046264's dirty hop jal 0x80063864) does jalr func_8001A954, nesting a host call
that never unwinds (the per-frame loop never returns). Full writeup RECURSION_BUG.md
§16-§21.

Instruments (always-on rings, runtime-only, no regen):
- §17 stack_profile: per-vblank host-stack sample (main.cpp; stack_profile TCP cmd).
- §18 xprobe / xprobe_arm: interp<->compiled boundary crossing flight recorder
  (dirty_ram_interp.c; per-frame summary + per-crossing detail).
- §19 ce_profile: per-compiled-entry depth profile + raw TEB at the trip
  (debug_server.c; ce_profile TCP cmd) — the trustworthy instrument; the regression
  check for the fix (max_kb must stay FLAT).
- PSX_STACK_GUARD_KB early-trip + PSX_FIBER_STACK_KB (debug_server.c / traps.c).

Fix progress:
- Block-loop tail-transfer surface (dirty_ram_interp.c ~1514): surface compiled
  tail-transfers to the trampoline instead of nesting. Correct for tail-transfers but
  RULED OUT as the leak (§19 climb unchanged) — the leak is the CALL path (jalr/jal).
- Real fix (next): surface compiled-game CALLS to the owner trampoline, gated to
  compiled-game targets only (a naive all-targets surface reintroduces the
  dwarf->overworld lost-continuation bug). Needs a non-nesting membership test
  (generated psx_dispatch_game_has) or §12 CD-DMA dirty-marking precision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 00:15:15 -07:00
Matthew Stanley 08484afc68 recursion: flight-recorder runs #1/#2 findings + fix buffer overflow (doc §16)
Run #1: interp->0x8001A954 edge is ORDINARY (1/frame). Run #2: dispatch_8001A954=0
(dirty_ram_dispatch never called with 0x8001A954) AND recursion_func VARIES
(0x8004DEE0/0x8004DFA0/0x8005FF60) -> address-specific counters are the wrong
instrument. dirty_block_tail shows the real cycle: dirty_ram_dispatch(overlay
0x8011-0x8013) called from main-EXE 0x8002-0x8005 = the per-frame MESSAGE/RENDER
overlay system recursing. SITE_CAP 256->32 fixes the report-JSON overflow. Pivot:
use data-driven rings (native_stack/dirty_block_tail/dispatch_tail) from a clean
report, not guessed addresses; then last-writer slice on the loop termination var.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 21:12:34 -07:00
Matthew Stanley 2475ee0550 recursion: re-entry flight recorder + strategy reset (doc §15)
Per-frame count of the re-entry edge interp->func_8001A954 (the per-frame loop
head) + cycle counter, in a 256-frame ring the crash report dumps
(dirty_ram_re954_json). Answers ChatGPT's step-1 question: is the re-entry
ORDINARY bounded per-frame behavior that stops terminating at ~50k, or a new edge
— and whether cycles advance (it's within ONE frame; 23k laps ~= 23k cycles <<
565k/frame, so no VBlank before overflow => control-flow cycle, not starvation).

docs/RECURSION_BUG.md §15 captures the full strategy reset (stop RAM-diffing;
triggered control-flow recorder + backward last-writer slice; narrow oracle use
only on the resulting small address set). The separately-navigated oracle RAM-diff
(§14) was a dead end (alignment noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 20:42:06 -07:00
Matthew Stanley afd843cdc9 docs(recursion): CORRECT the dialogue-trigger error — it is the ~14-min IDLE freeze (frame ~50k, location-independent); the boundary fix is symptom-only; NEXT = oracle idle-diff (turbo both, find first divergence vs real HW). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> 2026-06-17 20:01:22 -07:00
Matthew Stanley 0d3a1623e6 docs(recursion): full root-cause + fix design + oracle plan
Capture everything for recall in a future session:
- Resolved the overworld-load regression (stale build artifact, not tooling/master).
- 3 organic captures (deterministic: v0=0x8004DEE0, ra=v0+8, pc=0; both New-Game
  and Dwarf-Forest reproduce).
- The contract mechanism (S7): psx_dispatch_impl tail-call trampoline + psx_call_contract
  bail keep COMPILED tail-loops flat; the dirty interpreter bypasses them
  (exec_one JAL/JALR + dirty_ram_dispatch_inner:~1179 nest psx_dispatch_game_compiled).
  Proof it's host-only: guest sp flat (~7KB) across ~23k laps.
- Ghidra map of the guest state machine (S8): func_8001A954 state[0x4a]==1 ->
  func_8001AC00 chain -> func_80046264 jumptable (selector @0x8009BCDD, table
  @0x80013C60) -> func_8004DFA0 (a render handler that advances no selector).
- External-review verdict (S9): Hypothesis A overwhelmingly likely; B has no
  positive evidence; C (interp control-transfer bug fabricating the re-entry) must
  be excluded by an oracle PC-trace.
- Oracle disambiguation plan (S10), the single mixed-dispatch-owner fix design
  (S11, runtime-only / no regen), CD-DMA dirty-marking critique (S12), and the next
  disassembly target around 0x80063864 (S13).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 18:12:17 -07:00
Matthew Stanley d9b6ae5e16 recursion: capture native_stack tooling + working doc
Snapshot of the uncommitted native_stack capture tool (crash_trace.c,
append_native_stack) that recovers the true compiled<->dirty-interp
recursion cycle the recent_fn ring can't show, plus docs/RECURSION_BUG.md
documenting the freeze root cause, the 4-instance soak reproduction
harness, and the overworld-load regression now blocking repro.

This is the build that currently insta-crashes soaks on overworld load;
preserved as-is before hardening so the regression stays reproducible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:02:14 -07:00