Commit Graph

184 Commits

Author SHA1 Message Date
Alex Vanderveen 0aa50e514f Update bios_emitter_fingerprint.sh 2026-07-21 21:38:05 -07:00
Alex Vanderveen 4eb1794abb Update regen_bios.sh 2026-07-21 21:38:05 -07:00
Matthew Stanley e03c743e87 tools: add Linux macOS dev setup helper
Add a conservative source-checkout setup helper based on the Linux quick-start intent from PR #54.

The script builds the CLI/recompiler tools, refreshes BIOS C when a BIOS is present, and builds the BIOS-only runtime only when BIOS/generated sources are available. It deliberately leaves game project creation to the existing CLI generator.

Co-authored-by: Keeganator <keegan.pan@gmail.com>
2026-07-21 17:02:05 -07:00
Matthew Stanley 9228f32c00 runtime: keep Vulkan opt-in and remove old launcher
Remove the in-tree launcher implementation and its submodules now that game projects use recomp-ui for launcher support.

Keep Vulkan hidden from the launcher unless a game explicitly sets [video] offer_vulkan = true, and clamp stale Vulkan settings back to OpenGL when it is not offered.

Co-authored-by: Martin Penkava <mpenkava1337@gmail.com>
2026-07-21 16:31:36 -07:00
Matthew Stanley 8f78ddbd89 integrate interpreter hotpath and AOT sharding 2026-07-20 15:58:43 -07:00
Matthew Stanley ae3acc7db6 aot: deduplicate complete shard pairs safely 2026-07-20 14:13:46 -07:00
Matthew Stanley a59cf8a88d aot: make saturated caches a deterministic fixed point 2026-07-20 14:06:26 -07:00
Matthew Stanley c2f4c8bf4b aot: close cross-variant coverage deterministically 2026-07-20 14:06:26 -07:00
Matthew Stanley 71257f6424 aot: persist exact candidate coverage gaps 2026-07-20 14:06:26 -07:00
Matthew Stanley fd0795cd65 aot: batch supplemental exact-root shards 2026-07-20 14:06:25 -07:00
Matthew Stanley 09d5818efd aot: recover play-free continuation fragments 2026-07-20 14:06:25 -07:00
Matthew Stanley 5ded80e61e Merge master (AOT overlay generation) into perf/interpreter-hotpath
Integration branch: the interpreter-perf series (interpreter fallback +
additive shard autocompile + always-on production observability + the
fail-closed alias-body emitter fix) rebased-by-merge onto master's AOT
overlay generation (play-free extraction, bounded indirect discovery,
AOT recall, shadow-diff redesign).

Semantic-union highlights:
- gte.cpp: HEAD's direct-field marshaling fast paths kept; master's
  s_gte_replay_sandbox guards grafted onto every affected site.
- psx_icache.c: HEAD's global tag array + fetch_miss split kept;
  master's transactional shadow record/replay retargeted onto it.
- overlay_capture.c: HEAD's commit/retry/wait-pending pipeline kept;
  master's durable history + autocapture APIs wired into it
  (persist_history_snapshot fed from the immutable .d contribution).
- overlay_loader.c: master's interp-record/native-replay shadow-diff
  redesign kept as base with HEAD's scheduler-bail hardening grafted
  (new SHDIFF_SCHED divergence kind); strict manifest v2 parsing keeps
  master's P pair-id lines; always-on native-call ring preserved.
- main.cpp shutdown calls BOTH overlay_autocapture_shutdown() and
  overlay_capture_wait_pending() (the latter drains the preserve-write
  FIFO the former does not).
- recompiler: master's generate_alias_group refactor retains the
  fail-closed alias-body re-entry defaults from c85f1d1.

Validated on this tree: recompiler + runtime + tests build; all six
runtime unit tests PASS; interpreter perf guards PASS; recompiler
patch/structural tests PASS (l1 decoder oracle noise = 376 pre-existing
on both parents).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 00:43:41 -07:00
Matthew Stanley 9446038dbb autocompile: survive quoted interpreter paths + pin Windows Python in launcher
Two independent failures made every live reshard fail SILENTLY (runs
counted, shard_ok stayed 0, the game just never got shards):

- autocompile.c: the child line is executed via 'cmd.exe /C <line>'.
  When the line BEGINS with a quoted path and contains further quotes,
  cmd.exe strips the first and last quote characters and mangles the
  command ('The filename, directory name, or volume label syntax is
  incorrect'). Wrap the whole line in one extra pair of quotes -- cmd
  then strips exactly those and executes the inner command verbatim.
- launch_tomba2_interp_perf.ps1: the autocompile command used an
  unqualified 'python'; with an MSYS python first on the inherited
  PATH, it cannot open the Windows-style script path ('can't open
  file'). Use the canonical Windows Python312 explicitly and fail the
  launch loudly if it is absent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 00:07:33 -07:00
Matthew Stanley 156d9b9b34 Add self-contained media-to-source CLI release 2026-07-20 00:00:22 -07:00
Matthew Stanley b3bfdd9330 Fix in-handler RFE contract gaps that livelock the boot splash (Tomba 2 frame 1385)
Three-layer faithful-core fix for the exception-return contract, root-caused
live over TCP on a deterministic dbg-build repro (vblank authority vetoed
forever while the guest spun in the kernel idle loop, in_exception stuck 1):

1. Interpreter RFE backend parity: dirty-RAM RFE now calls
   psx_rfe_mark_escape() exactly like the recompiled opcode and every
   overlay shard (ABI v12), and dirty_ram_dispatch_inner takes a pending
   escape at the committed transfer before local dirty flow can keep
   interpreting the resumed code inside psx_check_interrupts' window.
2. Legacy-sentinel deliveries arm the escape too: the old blanket decline
   assumed a sentinel delivery always resumes AT the sentinel, but guest
   TCBs never hold the sentinel (Fix B anti-leak), so a kernel
   ReturnFromException restore resumes at an approximate real PC with no
   exit path armed. The take site preserves the legacy contract: a resume
   landing ON the sentinel still exits through the sentinel detector.
3. Same-thread completion of a sentinel delivery via explicit RFE /
   ReturnFromException converts to the legacy exit (restore interrupted
   GPRs, pc=0, continue the live host chain) keyed on the structural
   entry_tcb == exit_tcb test - otherwise the approximate TCB PC surfaces
   up the trampoline, severs the live chain, and the run dies one hop
   later on an un-re-enterable mid-function PC ("execution completed").

Validated: production build passes the previously-fatal splash transition
(frames 4717+ at 60fps, was deterministic death at 1385 x3 runs); dbg build
advances past the wedge (was frozen at exactly 1385); all runtime tests +
structural guards pass (guards extended to pin the interp RFE contract).
Known follow-up: the PSX_DEBUG_TOOLS build still loses one kernel CD event
through a third timing path at the same site - tracked as an open
foundation item; production config unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 19:00:29 -07:00
Matthew Stanley 5a4b74f5d3 fix: report AOT recall with exact semantics 2026-07-19 17:39:55 -07:00
Matthew Stanley 504db26445 feat: add bounded indirect overlay discovery 2026-07-19 17:15:17 -07:00
Matthew Stanley 76257fcf48 tools: point interp-perf launcher at the isolated game worktree/build
Two changes to finish the interpreter-perf / AOT-investigation split:

1. Build dir switches from runtime/build-tomba2-perf-rel (configured
   against the SHARED Tomba2Recomp checkout, CMAKE_HOME_DIRECTORY=
   F:/Projects/psxrecomp/Tomba2Recomp) to the new isolated
   runtime/build-tomba2-interp-iso-rel, configured against
   _wt-tomba2-interp-perf (branch interp/game-isolated, forked from
   Tomba2Recomp's master). That build passes
   -DPSXRECOMP_GAME_EXE_NAME_OVERRIDE=Tomba2InterpPerf, so it emits
   Tomba2InterpPerf.exe natively. The old build-tomba2-perf-rel is left
   untouched on disk as evidence of the original measurement; only this
   script's target moves.

2. Drops the Copy-Item alias hack: since the isolated build already
   produces the isolated exe name directly, there is nothing left to
   alias.

3. GameRoot now defaults to the isolated game worktree instead of the
   shared Tomba2Recomp checkout, so this script no longer reads
   game.toml from a tree whose HEAD (feat/aot-overlay-spike) belongs to
   a different investigation and has already drifted from master
   (overlay_capture_history).

4. The pre-launch guard now refuses to start if EITHER Tomba2InterpPerf
   OR Tomba2Recomp is already running (previously only checked its own
   process name) — the two builds can no longer collide on disk, but
   still share the BIOS file, debug-port range, and taskkill-by-name
   cleanup surface.

State paths (memcards/captures/logs/overlay cache) were already scoped
under _wt-interpreter-perf and are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:54:48 -07:00
Matthew Stanley 02f29b979d aot: harden exact interior fragment recovery 2026-07-19 14:29:57 -07:00
Matthew Stanley 05f6ef3e02 Optimize GTE fallback and isolate interpreter perf runs 2026-07-19 14:23:36 -07:00
Matthew Stanley 7376e6b042 Harden additive overlay shards and speed interpreter fetch 2026-07-19 12:11:56 -07:00
Matthew Stanley 91c9aea91f aot: make discovery and shard publication fail closed 2026-07-18 14:08:45 -07:00
Matthew Stanley aac755b0ae tools: accelerate cumulative coverage scoring 2026-07-18 13:14:57 -07:00
Matthew Stanley 205438eb98 tools: clean TinyCC staging sidecars 2026-07-18 11:15:13 -07:00
Matthew Stanley 4af125cf07 tools: align TinyCC overlay build contract 2026-07-18 11:08:16 -07:00
Matthew Stanley dbe14dfadb aot: recover safe frameless and interior overlay coverage 2026-07-18 10:13:18 -07:00
Matthew Stanley 6af5437058 tools: publish overlay shards atomically 2026-07-18 09:05:41 -07:00
Matthew Stanley ae1f1e7f19 tools: report true AOT coverage gaps 2026-07-18 08:59:26 -07:00
Matthew Stanley 3949854981 Optimize interpreter fallback and additive overlay autocompile 2026-07-18 08:32:42 -07:00
Matthew Stanley 32cc2e81f4 aot: bound header dispatch fallback discovery 2026-07-18 08:27:09 -07:00
Matthew Stanley 5e8cc5b8e5 aot: enrich archive discovery with safe fallback 2026-07-18 07:49:29 -07:00
Matthew Stanley 41f498fd09 aot: recover linked archives and static aliases 2026-07-18 07:25:49 -07:00
Matthew Stanley 731f43235e tools: make large disc reads linear 2026-07-18 06:30:37 -07:00
Matthew Stanley 8f90df9b43 aot: preload exact-hash BIOS resident shards 2026-07-18 05:31:03 -07:00
Matthew Stanley baf8b5928c aot: recover cross-boundary branch roots 2026-07-18 04:40:07 -07:00
Matthew Stanley 4fb144a4fd aot: preserve additive static discovery 2026-07-18 04:16:51 -07:00
Matthew Stanley e2a1037dad aot: score verified capture history 2026-07-18 04:16:51 -07:00
Matthew Stanley 9ab17d084b aot: close guarded BIOS vector coverage gaps 2026-07-18 03:25:48 -07:00
Matthew Stanley 65d89a06d7 runtime: compact persisted capture addenda 2026-07-18 02:56:53 -07:00
Matthew Stanley 495bf2de53 aot: constrain calls across composite producers 2026-07-18 02:56:42 -07:00
Matthew Stanley 23de1d5bcd aot: seed dense function pointer tables 2026-07-18 01:55:05 -07:00
Matthew Stanley f53569dfea aot: account for native BIOS coverage 2026-07-18 01:32:49 -07:00
Matthew Stanley 6c67f81a33 docs: record Tomba 2 static attract recovery 2026-07-18 01:21:05 -07:00
Matthew Stanley 88e471274f aot: separate code-range recall from entry parity 2026-07-18 01:18:48 -07:00
Matthew Stanley bc24934048 aot: recover raw and adjacent overlay producers 2026-07-18 00:38:26 -07:00
Matthew Stanley a544809edf aot: persist overlay capture history atomically 2026-07-17 23:56:58 -07:00
Matthew Stanley 2834f0468d aot: recover call-sparse overlay bases 2026-07-17 23:29:38 -07:00
Matthew Stanley 43cd37624a aot: discover frameless static call targets 2026-07-17 22:58:09 -07:00
Matthew Stanley 9092d15dc8 aot_overlay_spike: export-table dispatch seeds + coverage recall report
extract_generic: supplement prologue seeds with the overlay's OWN export/dispatch
table (the header pointers) as dispatch entries — the game's authoritative
entry-point manifest, on the disc, play-free. compile_overlays classifies mid-fn
targets DISPATCH_INTERIOR aliases. Honest note: no-op for Tomba 2 (skipped=23) —
the recompiler's control-flow walk already reaches those entries; kept because it
helps engines where the walk cannot.

coverage_report.py (new): measures play-free static-extraction RECALL against a
played vault (most complete needed-set) and/or a live-session captures json, and
persists a durable gap manifest (misses grouped by region). Tomba 2: 60.0%
entry-level recall vs vault (1113/1856), 743 misses dominated by MAIN.EXE
frameless/indirect functions + the kernel region.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 21:51:25 -07:00
Matthew Stanley 414314d412 aot_overlay_spike: robust header-table base recovery via jal self-consistency
The old delta-sweep matched export-table pointers against 0x27BD prologues,
but those pointers point at mid-function DISPATCH entries, not prologues
(measured: 0/29 hit a prologue at the true base). It scattered bases by
2-16KB and decoded data as code.

Fix: j/jal targets are BASE-INDEPENDENT (0x80000000 | (imm26<<2), since all
game code lives in 0x800xxxxx). At the true base, the maximum number of
intra-overlay jal targets land on a real prologue. recover_base() votes every
(jal-target, prologue-offset) pair for base t-o within the pointer-containment
window; the peak is unique and sharp (~2x margin). Records are emitted
page-aligned with a FILL prefix, because the runtime keys shards by a
page-aligned region_start and compile_overlays takes load_addr verbatim.
Weak-signal files are skipped (score<4 or <1.5x runner-up) -- safe coverage
loss under the per-function code_crc dispatch guard. Also dedup overlays by
content CRC (position-fixed BINs are reused across AREA folders).

Validation: generic output is byte-identical (base, region envelope, seed set)
to the live-proven tomba1_extract.py hand tool for ALL 22 Tomba 1 overlays,
with no hardcoded constants. Tomba 2's 21 A*.BIN overlays converge on a single
consistent base 0x80108F9C (region 0x80108000 +3996).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-17 20:19:03 -07:00