mirror/psxrecomp
1
0
Fork 0
mirror of https://github.com/RetroPortingToolKit/psxrecomp synced 2026-10-28 02:56:36 -04:00
Code Issues Packages Projects Releases Wiki Activity
327 Commits 437 Branches 8 Tags
2e7b751bfcb1637b79ca38bcd3bc06be3cff7698
Commit Graph

10 Commits

Author SHA1 Message Date
Matthew Stanley 1f6713bfc0 overlay cache: auto codegen-hash in the cache path (cg<N>_<hash>) 
The manual PSX_OVERLAY_CODEGEN_VER must be hand-bumped on every emitter change;
when it isn't (e.g. between CPS emitter iterations), a stale-but-cgN DLL is reused
-> the v0.3.0 black screen. Now hash_codegen.cmake hashes the recompiler codegen
sources into a (gitignored) overlay_codegen_hash.h at build time; overlay_api.h
exposes PSX_OVERLAY_CODEGEN_HASH and the cache path becomes gcc/<arch-abi>/cg<N>_<hash>/.
The loader and compile_overlays.py read the same value, so any emitter change
auto-invalidates the cache (old dirs coexist, no stale load). Falls back to hash 0
when the header is absent (fresh checkout) — identical to today's single-cgN path.
Verified: build emits hash 0x347751c1; autocompile + loader both use cg4_347751c1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-18 18:57:52 -07:00
Matthew Stanley 2d468ee5b8 recursion(stage3): overlay (sljit + gcc-DLL) CPS parity — full native overlay coverage under CPS 
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).

Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.

sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
  JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)

gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
  per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
  psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
  inside a registered candidate's code range (a tail-transferred call's continuation),
  overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
  to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
  This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.

Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-18 13:35:31 -07:00
Matthew Stanley 94a1966444 recursion(fix): universal continuation-passing — FIXES the idle-freeze host-stack leak (BIOS+game) 
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.

VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).

KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.

Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
  tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
  tail-transfer (jalr captures rs before the link, alias-safe); each call's
  return point registered as a dispatchable continuation; entry-switch (keyed on
  cpu->pc) at the top of every function AND every overlapping-alias body. 0
  psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
  (case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
  syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).

Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
  Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
  fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
  ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
  (un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
  on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).

Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-18 12:45:02 -07:00
Matthew Stanley c983ef521f ws backdrop preload: camera-tracked window widening (auto_backdrop) 
[widescreen.cull] auto_backdrop — generic far-backdrop column preload for
16:9. A shared detector (runtime/include/ws_backdrop_detect.h) finds each
scrolling-backdrop column-window generator by its /96-magic + 0x176-camX +
sra,5 divide-tail invariant and records the window's START/END bound
instructions + width (window_cols). All four execution paths route those
bounds through psx_ws_backdrop_value(orig, is_end, window_cols), which widens
the camera-tracked window by exactly the 16:9 reveal (margin scales with
window_cols; identity at 4:3 so the build stays byte-identical with widescreen
off). gcc emit + sljit + dirty-RAM interp + overlay ABI v4 / codegen ver 3.

Also: CodeGenConfig::overlay_mode — explicit [widescreen.*] per-address site
shape checks now SKIP (apply only where bytes match) instead of hard-exit in
overlay mode, because overlay regions reuse the same address for different
code across scene variants. Main-EXE mode keeps the hard error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 14:48:15 -07:00
Matthew Stanley 0c1f005c35 feat(widescreen,sljit): shared cull helper across backends, cache versioning, sljit default-on 
Three coupled landings on top of the sljit merge:

1. Shared widescreen render-cull helper. auto_screen_x now emits a call to one
   runtime helper psx_ws_cull_sltiu(sx,imm) (gpu.c) instead of an inline expr, and
   the sljit JIT (overlay_sljit.c) + dirty-RAM interpreter (dirty_ram_interp.c)
   route flagged render-funnel sltiu 0x140/0x141 through the SAME helper. So an
   overlay's GTE render funnel widens identically whether it runs as a gcc DLL, a
   sljit shard, or interpreted. Helper is byte-identical to vanilla at 4:3
   (margin 0), so widescreen-off stays bit-exact. psx_ws_func_has_screen_cull
   gates the widen to real render funnels (sltiu 0x140 + sltiu 0xE0 signature).

2. Versioned overlay cache path. gcc cache moves to <game_id>/gcc/<arch-abi>/cg<N>/
   where N = PSX_OVERLAY_CODEGEN_VER (overlay_api.h), parsed identically by the
   runtime (overlay_loader.c) and compile_overlays.py so they can't drift. A build
   whose emitter output changed (e.g. this auto_screen_x change) reads+writes a
   FRESH dir and never reuses a stale DLL; old versions coexist (no auto-delete).
   Pre-1.0: legacy flat/unversioned fallbacks dropped.

3. sljit default-on (validated). gcc and sljit are complementary, not exclusive:
   gcc fills the cache via prebuilt DLLs + async batch (compile_fragment NULL),
   sljit JITs gcc-absent regions synchronously on-miss. New code_provider_sljit()
   exposes sljit for the JIT-on-miss gap-fill independent of the "active" backend;
   the dispatch gate fires whenever sljit is available (head<0 == gcc-absent), and
   apply_live_policy defaults live ON when sljit is available. Priority gcc > sljit
   > interp; shards still route through the same-state diff gate (device-touch +
   diverging shards pinned to interp) so default-on does NOT reintroduce the
   save-load wedge; a declined/failing shard falls to interp (never a crash).
   Verified live: live=1, 62 shards compiled, 2 declines->interp, native:interp
   ratio ~31%->~87%, gcc still primary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-15 19:36:43 -07:00
Matthew Stanley 748e29f1b9 widescreen(8C): backdrop screenX squash on native + interp paths 
The 16:9 edge pop-in is the parallax 2D BACKDROP layer (ocean/cloud/mountain/
grass), drawn by overlay actor handlers (FUN_801216bc family) that compute
screenX = (worldX - camX) >> parallax in pure integer math and store it to the
object's +0x12 field WITHOUT the GTE — so the GTE X-squash that gives 3D the
wider 16:9 FOV never reaches them and far pieces sit past the 320px edge,
clipped (the blue void / half-rectangles at the edges). See TombaRecomp
ISSUES.md #8C for the full root-cause trail.

Fix = psx_ws_backdrop_x() (gpu.c): squash screenX around screen-centre by the
same ws factor as the GTE, identity at 4:3. Applied on BOTH execution paths
because overlay code runs interpreted when no cache DLL is loaded (the common
case in dev — see #8G):
- Native: recompiler emits write_half(.., psx_ws_backdrop_x(rt)) at the
  [widescreen.backdrop] x_sites store (config_loader + code_generator +
  main_psx --ws-config so the overlay compile gets the sites). Forwarded into
  overlay DLLs via OverlayCallbacks {ws_backdrop_x, ws_x_margin, ws_sprite_tag}
  (overlay ABI v2->v3 as the cache discriminator); compile_overlays.py glue +
  --ws-config passthrough.
- Interp: dirty_ram_interp SH hook keyed by psx_ws_is_backdrop_site(pc); the
  runtime registers the x_sites from game.toml (main.cpp).

Verified in data: backdrop slot screenX squashes 371 -> 318 (pulled on-screen).
PARTIAL: visible culling persists -> remaining cause is the backdrop spawn/
despawn window (separate from screenX clip); see #8C "REMAINING CAUSE".

Also added (earlier this session): ws_census draw ring, ws_margin override,
gpu_state ws fields.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-15 12:52:17 -07:00
Matthew Stanley 1010c58c41 overlay: flavor-tag the cache so variant builds can't cross-contaminate 
The overlay cache keys DLLs by guest-bytes CRC, which is flavor-blind — the
same guest overlay yields the same filename whether compiled by the base
recompiler or a variant (e.g. widescreen, which emits psx_ws_*/GTE-adjust
symbols). Two flavors sharing a cache dir would collide: a base runtime could
pick up a widescreen DLL (or vice versa) with no guard, since both report ABI
v2.

Fold a codegen-flavor id into overlay_abi() (high 16 bits) and check the
combined tag in the loader's existing ABI gate. Base/master flavor is 0, so the
tag stays == PSX_OVERLAY_ABI_VERSION — fully backward compatible with every
existing base DLL (verified: an 8-DLL base cache loads with zero rejections).
A variant build sets PSX_OVERLAY_FLAVOR (e.g. widescreen: -DPSX_OVERLAY_FLAVOR=1
for the runtime, --flavor 1 for compile_overlays.py); its overlays then report a
distinct tag, so the base runtime rejects+regenerates them and vice versa. The
caches stay separate automatically, even in a shared directory — no new paths.

overlay_api.h: PSX_OVERLAY_FLAVOR (default 0) + PSX_OVERLAY_ABI_TAG.
overlay_loader.c: both ABI gates check the tag.
compile_overlays.py: --flavor arg -> -DPSX_OVERLAY_FLAVOR, baked into overlay_abi().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-14 16:04:21 -07:00
Matthew Stanley 2439d4dd44 dispatch call contract: continuations only run when the guest really returns 
Bug D root cause (Tomba chest freeze, also QA issue #1): the game's own
dispatcher func_80056E08 has a dead jumptable case (type 0x21 ->
jal 0x80120B3C) whose target is mid-function in EVERY overlay variant on
the disc (only the boot EXE references it; the real handler entry is
0x80120AC8). Entering mid-function skips the prologue, runs an epilogue
for a frame that was never pushed, and returns through garbage. Our C
model then resumed the suspended generated continuations on the moved
guest stack -> double-executed tails -> infinite zombie loop (~3K wild
calls/frame) -> MMIO fatal. Family root of Bugs A/C/D.

The contract: a generated C continuation may only run if the guest
actually arrived at it ($ra == call site return address, $sp == caller
sp at the call). Violations begin a "bail" unwind: cpu->pc := the wild
jr's true target, every generated frame returns immediately, and the
unwind resolves at the first enclosing site whose contract matches (or
flattens at the outermost dispatch loop, re-dispatching the wild target
on a clean host stack -- matching real-HW one-stack semantics).

- cpu_state.h: bail state + psx_call_contract() helper (+ overlay-DLL
  pointer indirection under PSX_OVERLAY_DLL_BUILD)
- emitters: (ra, sp) checks at every BIOS and game jal/jalr site;
  psx_dispatch_impl validates pc==0 and stop-hits against (stop, sp),
  handles bail resolve/propagate/flatten
- dirty_ram_interp: surfaced calls validate the contract on return;
  bail aborts the interp run; tail transfers into compiled code now run
  with pc cleared (closes a latent double-run on plain C returns)
- overlay cache ABI v2: DLLs export overlay_abi(); loader rejects and
  deletes stale DLLs (autocompile regenerates with contract checks)
- telemetry: bail_first/resolved/flattened/anomaly in heartbeat + dump
  headers; anomaly guard at exception entry; TCP_COMMANDS.md docs

Verified: BIOS+Tomba regen, boot to intro at locked 60.0 fps, all bail
counters 0 through boot/title/attract (no false positives). Chest-area
gameplay verification pending (requires user play).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-10 22:30:41 -07:00
Matthew Stanley 11799d4189 Step 2.7: kernel-window native coverage for install-at-runtime stubs 
Extend the overlay JIT cache below the 0x98000 floor to kernel RAM
[0, 0x10000) — the BIOS part-2 image + install-at-runtime stubs
(Rule 18), dominated by the SIO data-byte slot at 0xCF0 whose
install-slot hook tail-dispatches the whole exception-handler tail
into the interpreter on EVERY exception once installed.

Window model (dirty_ram_interp.h): two capture/candidate windows —
kernel [0, 0x10000) and overlay [0x98000, RAM); main-EXE text between
them is statically recompiled, never captured. Interp local-flow gates
intentionally stay overlay-only (kernel runs per-block in exception
context; named constant documents this).

- overlay_capture: per-window dirty-run enumeration, runs clamped at
  window edges (kernel CPU-write pages and main-EXE DMA pages can be
  adjacent; never merge them).
- overlay_loader: dispatch lazy-load gate covers both windows;
  scan_cache_dir validates the <addr8>_<crc8>.dll shape explicitly
  (kernel region_start 0 is legal, zero-parse is no longer a sentinel).
- classifier (compile_overlays.py): kernel-window-only DISPATCH_ROOT
  promotion — an orphan dispatch interior no rooted walk covers has no
  host to truncate and is a real execution root by construction (the
  static install-slot hooks tail-dispatch into it). One orphan per
  iteration so remaining orphans alias into the new host. Overlay
  regions stay ineligible (cross-variant per-PC evidence: 0xE889C class).
- main_psx --overlay: 'dispatch_root' seed lines are trusted walk roots
  (exempt from the boundary re-check); seed range is now the loaded
  image's own bounds (the old hardcoded [0x80010000,0x80200000) window
  silently dropped kernel seeds).
- overlay_api/loader/DLL preamble: route psx_restore_state_escape
  through OverlayCallbacks (appended last, by-value copy keeps old
  DLLs compatible) — exception-context kernel code longjmp-returns.

Tomba verified (user-played: title -> LOAD list renders -> save loads
-> overworld laps): kernel interp per-PC rows 3 -> 0 (was 0xCF0 =
61,775 dispatches / 2.71M insns at the LOAD screen alone, ~4.6M
insns/session); host func [0xCF0,0xEA0) + 0xDF8/0xE08 aliases, audit
clean; dispatch_native 357,763; dispatch_miss_total 0; interp aborts 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-06-09 15:24:30 -07:00
Matthew Stanley b2929aa504 overlay: restore overlay DLL cache + capture (faithful port of codex/cache-discovery) 
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline
overlay compilation pipeline that were built on the pruned codex/cache-discovery
branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt
rather than history-restored.

Runtime (off by default, opt in via [runtime] overlay_cache = true):
- overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at
  cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader
  LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(),
  enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter.
- overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and
  writes overlay_captures.json (execution-verified seeds) at clean exit.
- dma.c: capture + cd_dma_log hook at CD DMA completion (async model).
- cdrom.c: cdrom_get_setloc_lba() for disc-position correlation.
- dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a
  decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks.
- debug_server.c: cd_read_log + overlay_loader_status TCP commands.
- config_loader: [runtime] overlay_cache flag (default false).

Recompiler (compile_overlays.py / extract_overlays.py drive these):
- main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL
  targets and set split_mid_function_targets=false, so overlay basic-block
  targets are NOT split into bogus standalone functions (the mid-function-seed
  failure mode that produced black-on-load / blue-on-new-game).
- function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the
  mid-function split pre-pass behind split_mid_function_targets (default true,
  so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs).

Deltas from the original: save states dropped (never worked); overlay enable is
runtime-config-gated (off by default) instead of a build-time define; loader
stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3).

Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-07 12:32:19 -07:00