From f7652e2b82cdd474c40d6d27a05ab4ccbf47795d Mon Sep 17 00:00:00 2001 From: Alexandros Mandravillis Date: Mon, 14 Sep 2026 17:36:05 +0300 Subject: [PATCH] Record the TAS campaign logs and upstream integration notes Materializes the running campaign documents, workflow triggers and the Bio Hazard replay records deferred from the subsystem branches. --- .github/workflows/tasreplays.yml | 4 +- docs/NYMASHOCK_SOURCE_REPLAY.md | 157 ++++++++++++++++++++++++ docs/TAS_ACCURACY.md | 69 +++++++++++ docs/TAS_UPSTREAM_INTEGRATION.md | 75 ++++++++++++ docs/internal/ACCURACY_BURNDOWN.md | 5 +- docs/internal/FAITHFUL_TIMING_PLAN.md | 26 ++++ docs/playbooks/AUTOMATED_TESTING.md | 165 ++++++++++++++++++++++++++ tools/tasreplays/README.md | 19 +-- tools/tasreplays/investigation.md | 2 +- 9 files changed, 510 insertions(+), 12 deletions(-) create mode 100644 docs/NYMASHOCK_SOURCE_REPLAY.md create mode 100644 docs/TAS_ACCURACY.md create mode 100644 docs/TAS_UPSTREAM_INTEGRATION.md create mode 100644 docs/playbooks/AUTOMATED_TESTING.md diff --git a/.github/workflows/tasreplays.yml b/.github/workflows/tasreplays.yml index 804188d7..a8fa9261 100644 --- a/.github/workflows/tasreplays.yml +++ b/.github/workflows/tasreplays.yml @@ -1,9 +1,9 @@ name: TAS replay source checks on: pull_request: - branches: [tasreplays] + branches: [master, tasreplays, "codex/tas-upstream-integration-20260913"] push: - branches: [tasreplays] + branches: [master, tasreplays, "codex/tas-upstream-integration-20260913"] workflow_dispatch: permissions: contents: read diff --git a/docs/NYMASHOCK_SOURCE_REPLAY.md b/docs/NYMASHOCK_SOURCE_REPLAY.md new file mode 100644 index 00000000..2a9c0987 --- /dev/null +++ b/docs/NYMASHOCK_SOURCE_REPLAY.md @@ -0,0 +1,157 @@ +# Bio Hazard independent source admission + +`tools/tasreplays/nymashock_admission.py` admits the exact Japanese Director's +Cut movie after a completed stock replay and a separate complete observer +replay. It does not consume a native run as an expected result. + +Both source runs must consume all 227,202 unchanged original inputs, use the +same declared neutral tail, exit cleanly, and retain a witnessed ending image. +The role, original movie, BIOS, complete stock host closure, loaded core, +reviewed read-only helper, frozen launcher/Lua and run-specific command are +verified. The closure is rechecked against the pinned release archive. Empty +sync maps are resolved through the source: one P1 DualShock, only card1 enabled, +fixed RTC, and the admitted Nymashock defaults. The initial card bytes and +isolated writable directory are part of that identity. + +The stock and observer must match at every full-RAM SHA-256, accumulated +master clock and lag observation, and every required screenshot. The observer +adds 512 page hashes per completed return and one terminal raw 2 MiB RAM dump. +Page clocks must match the independent stock record; the terminal dump must +match all 512 page hashes and the independent full-RAM SHA-256. Missing rows, +gaps, invalid clocks, mismatched endings/settings or existing output reject +reference creation. Both cleanly exited source runs must also retain identical +persisted128KiB card1 files; a missing, malformed or differing card rejects +admission even when every RAM/clock row matches. + +```text +python tools/tasreplays/nymashock_admission.py STOCK_RUN OBSERVER_RUN NEW_REFERENCE.json +``` + +The output is `biohazard-independent-source-v1`. It binds the complete +source artifacts, terminal RAM, page index, initial and persisted terminal card, +and ending evidence. +Native input, clocks, RAM, repeated playback and progression-save gates remain +separate; a completed source replay is not a native gameplay pass. + +The source's fixed cold PRNG sequence is separately reproducible with +`tools/tasreplays/external/collect_nymashock_random.py`. That tool compiles +the exact generator type from clean pinned Mednafen source at O0/O2 and +compares all 65,536 raw words to the existing external generator. Its output +tape is identical, but CD range rejection, command ordering, random-word +consumption and deadlines still need independent source/native comparison. +The external generator remains separate from the generic native tape reader. + +Progression saves require fresh per-run raw-card copies, in-game loads and +save/reload checks on both source and native. Emulator states are not raw +cards and cannot satisfy those gates. + +`biohazard.py setup` requires that admitted reference plus the exact owned +disc, BIOS, movie and Nymashock raw-generator qualification receipt. It creates +a fresh candidate from clean source, freezes the original input and initial +card, and records generated-code/build identities. No installed old title +binary is used. `biohazard.py run` compares every declared source/native RAM +page and clock, and full playback additionally compares terminal RAM bytes and +the candidate's persisted card after clean exit. Native normal initialization +registers `memcard_flush_all` with `atexit`, so the observation-end exit follows +the existing persistence path. Terminal evidence errors produce a failed +comparison receipt; they cannot leave an apparent successful playback. +An optional diagnostic cutoff retains only an unchanged original prefix; +it cannot qualify full playback or replace the original movie. + +The candidate explicitly selects available runtime comparison models, including +the separately qualified Nymashock controller/card models. Older profile names +on other devices describe implementations being compared; they do not assert +automatic compatibility with Nymashock. Native timing remains unqualified +until the independent comparison passes. Source endpoint N needs one extra +neutral native input boundary because native completion precedes its final +return observer. Full runs keep the original input count intact and record +that declared tail. The per-run storage bound is3GiB for full page/CPU checks. + + +## Experimental Bio Hazard drive comparison + +`tools/tasreplays/biohazard.py` selects `--cd-drive-model nymashock-1.29.0`, +`--mdec-source-model nymashock-1.29.0` and `--syscall-model guest-exception`. +The drive model uses both logical-seek pipeline slots, tracks the physical +head during Pause/standby and ReadTOC, consumes the active-Pause random draw, +and separates Reset travel/header phases from its command-reply deadline. +During an established read, GetlocP reports the last physical sector's sub-Q +position, one sector ahead of the next data delivery. +The source clock tape is required. The MDEC option uses 512 clocks per block; +the older Octoshock option keeps 474. Guest syscalls enter the installed BIOS +exception handler, including thread switches. In this source profile, +interpreted calls return their target to the dispatcher rather than retaining +native call frames across guest thread switches. + +Source-profile CD register reads sample stored device state before servicing +events crossed by their width-dependent bus wait. The load still consumes its +full duration. `runtime/tests/test_cd_read_sample_order.py` checks event ordering, +byte/half/word reads, LWC2 and address aliases at O0/O2 without retail assets. + +These are cold diagnostic options. This integration exposes no TAS checkpoint +launcher; the source memory-card/DualShock profile also remains +unsupported for checkpoints. Device state encoding does not qualify TAS +continuation. Existing default model options retain their prior behavior. + +On historical replay revision `7a0d0ff4`, the measured Bio Hazard RAM-page/clock prefix advanced from return 463 +to 233,567, with the first state/clock difference at 233,568 in the neutral +ending tail. That difference is closed by timing an explicit seek from the +physical read head: an established read keeps the source's CurSector two +sectors ahead of the sector handed to the guest, so a mid-read SeekL timed +from the delivery cursor travelled two sectors too far and completed 209 +cycles late, pushing the driver's following ReadS past the return boundary. +A cold full-route replay carrying that correction matches the admitted stock +reference on all 239,202 returns with no divergence, and terminal RAM and the +persisted memory card both match. +All forty-six raw RAM snapshots at 5,000-return intervals match the admitted stock +SHA-256. The flat-line and shaded-line stops previously encountered before +returns 20,372 and 20,387 are passed by this replay. These progress captures +are not resumable emulator states. +A prior transient mismatch at return 1645 is removed by the CD read-order +correction; paired raw snapshots at 1644, 1645 and 1646 also match. +The flat-call correction removes the prior recursion failure, and Reset +during an established data read is included in this bounded comparison. +GPU upload history now stops capturing when its 128 retained entries are full. +Previously later uploads kept incrementing the last entry until its signed +counter overflowed and indexed outside the history buffer. The regression +fixture exercises the production capture code at capacity and the maximum +legal upload size at O0/O2. The full replay passes the former crash at 214,410. +Full Bio Hazard playback remains unqualified. No release or accepted pin +changes. Logical header retry/error behavior and Reset during audio or an +unfinished data seek remain outside this bounded drive comparison. + +The source GPU comparison path now handles flat and shaded two-vertex lines (GP0 +0x40-0x47 and 0x50-0x57), including degenerate single-pixel lines, dithering, blending, +mask evaluation, clipping and interlaced row skipping. Timing charges two +dispatch clocks, sixteen setup clocks, and twice the unclipped major-axis +length; oversize lines pay only dispatch and setup. Shaded lines interpolate each endpoint color with twelve fractional bits, +including endpoint reversal and half-unit color bias. Polylines remain +unsupported. The new path has 512 authored component cases comparing +full VRAM SHA-256 and timing with extracted, unedited Nymashock line functions. +`tools/tasreplays/source_line_oracle.py` records source and harness identities; +`runtime/tests/test_source_gpu_line_contracts.py` checks the retained fixtures +at O0/O2. This is component evidence, not an unmodified-core replay pass. + +The source sprite path also admits raw semi-transparent rectangles (GP0 0x67), +using the existing raw texture and blend implementation. Like 0x66, this is a +four-word packet with three-word FIFO feedback; raw colour bypasses modulation +and leaves sprite timing unchanged. The sprite fixture checks both opcodes at +O0/O2. The stock Octoshock2.3 pixel oracle compares 192 full VRAM images across +texture depths, blend modes, mask modes and command colours. Nymashock +`gpu_sprite.cpp` and `SPR_HELPER_SUB` select the same raw/blend semantics. +The 150,000-return replay passes the former 0x67 stop at return 122,977. +The launcher and RAM probe accept up to 64 selected raw snapshots, enough +for the full Biohazard route at 5,000-return intervals plus terminal diagnostics. +The observer tests cover the 64-snapshot boundary and reject a 65th entry. + +Source basis: BizHawk 2.9.1's Mednafen 1.29.0 `psx/cdc.cpp` (`HandlePlayRead`, +`CalcSeekTime`, `Command_SeekL`, `Command_Pause`, `Command_Reset`, +`Command_ReadTOC`), `psx/mdec.cpp` (`WriteImageData`, `MDEC_Run`), and guest +CPU exception entry. Passive observers reading the exact stock Waterbox +image reproduced all 6,000 reference return RAM hashes, clocks and lag counts +before their CD/CPU state was used to diagnose divergences. Raw CPU register +snapshots are diagnostic data, not a claim of CPU-register equality. + +Source admission binds the exact bytes of its admission tool. The repository +keeps that Python file in LF form through .gitattributes so Windows checkout +conversion does not invalidate an otherwise unchanged reference. diff --git a/docs/TAS_ACCURACY.md b/docs/TAS_ACCURACY.md new file mode 100644 index 00000000..09a85a69 --- /dev/null +++ b/docs/TAS_ACCURACY.md @@ -0,0 +1,69 @@ +# TAS accuracy profile and Tekken 3 validation + +This submission ports the TAS comparison profile onto upstream master +`85cd26f0`. See [integration status](TAS_UPSTREAM_INTEGRATION.md) for the +current checks, source mapping and exclusions. The results below were measured +on the earlier campaign builds, whose base was `f23c5ba1`; they do not qualify +this newly integrated runtime. Its overlay callback ABI is 24, preserving the +upstream v23 callbacks before appending TAS instruction-boundary callbacks. + +The unchanged Spikestuff movie contains7,974 inputs. The integrated native +build wins Arcade at8.80 seconds. Every original-input return clock and all512 +4KiB RAM-page hashes match the independent Octoshock2.2.2 reference;28 full +2MiB snapshots are also byte-identical. This is stronger evidence for this +route than a clean process exit or simply consuming the input file. It is not +a proof of complete hardware or cross-title compatibility. + +## Corrections integrated + +Related corrections are grouped here by their shared mechanism. Snapshot +numbers in the original research included diagnostics and experiments; they +are not bug counts. + +| Area | Corrected mechanisms | +|---|---| +| CPU/code generation | Uncached instruction fetch costs; copied BIOS/RAM alias admission and syscall EPC; exception transfer and post-SYS continuation; deferred multiply deadlines; isolated cache-tag stores; ROM/RAM precision guards and conservative block bounds; pending-load ownership across compiled/interpreted boundaries; signed arithmetic exceptions; GTE load cancellation, delayed register values, and COP2 interrupt ordering. | +| Interrupts | Execute the actual vector instructions; interrupted fetch and branch-delay context; EPC/Cause/BD/BT/TAR ownership; IRQ reentry/cooldown and repeated delivery after RFE; Cause CE bits; register read image and sampling; recognize IRQs after DMA halt; VBlank latching during a guest IRQ handler. | +| DMA | Live request/linked-list GPU transfer; upload/readback readiness and word ownership; cancellation at packet boundaries; initial credit and register-write order; OTC/CD CPU stalls and instruction fetch overlap; completion deadline invalidation; fractional-service/frontend ordering; SPU DMA initial/word/block timing. | +| GPU | One FIFO/service owner for command admission, dispatch and visible effects; phase-correct triangle rendering and queued environment state; A0/C0 transfers; polygons, textures/CLUT/cache, shading, masks and blending; fill/copy/rectangles; clipping and VRAM wrapping; field-aware row work; raster/status timing and automatic frame-return ownership. | +| CD-ROM | Implicit and explicit seek transitions; asynchronous sector IRQ presentation; firmware/cold tray state; TOC/seek/read pipeline and command clocks; reset; Pause ACK and head rewind/resume; trigger/read-head state. | +| Timers and SIO | Source raster field clock; timer1 HBlank and read sample; timer2 divider/IRQ/deadline; digital-pad ACK pulse; opt-in handling of an unrelated legacy card repair. | +| SPU | Delayed control/status visibility with independent sample service; key-on/envelope phase; ADPCM decode queue, END flags, pitch/filter state and guest register readback. | +| Overlay integration | ABI24 forwards live instruction-boundary callbacks and pending cycle ownership through generated DLLs, preserving replay exclusion and rejecting stale ABI caches. | + +Most detailed timing changes are enabled by explicit `octoshock-2.2.2` profile +options; they model the exact emulator on which this TAS was authored. +General CPU/controller defects also have source-level corrections. The profile +does not contain a title-name switch, a precomputed game-state stream, or +delays chosen to force an opponent or victory. + +The reference is the immutable +[BizHawk source commit](https://github.com/TASEmulators/BizHawk/tree/519e14aa1ad7a9d6df2edc7808c5ed687dfee046), +observed using ordinary authored guest programs and original movie playback. +The native CPU, devices and renderer execute independently. The isolated +external raw PRNG utility has a separate GPL license and file interface; +its arithmetic is not imported into the framework runtime. + +## Reproduction and validation boundaries + +The recipe generates the BIOS and title from owned assets using the repaired +emitters. All 57 resulting C files must match the retained historical +fingerprints. Until regenerated code and a fresh retail replay are qualified, +a fingerprint mismatch correctly prevents this integration from being advertised +as the earlier verified build. It builds dependencies from pinned archives and uses +software rendering in both visible and headless modes. Recompiler and authored +O0/O2 tests run without retail assets. The final replay verifier rejects an +input mismatch, a missing or reordered checkpoint, a different RAM-page hash, +a return-clock difference, or incomplete termination. + +Historical C fixtures that require retained original-core captures are still +identified as fixtures by the registration checker. Their old private driver +names describe provenance, not a claim that those drivers ship in this branch. +Portable CTest registrations and the complete retail replay are the repeatable +checks available from the clone. + +The ending is explicitly bounded at8,400 frames:7,974 original inputs plus426 +neutral inputs, through the visible victory. Later CDDA Play seek remains +unqualified and stopped the research tail after return10,704. PAL, save-state +restoration, other input devices, other titles, complete audio/pixel equality, +and unrestricted post-victory playback are outside this qualification. diff --git a/docs/TAS_UPSTREAM_INTEGRATION.md b/docs/TAS_UPSTREAM_INTEGRATION.md new file mode 100644 index 00000000..10a46d32 --- /dev/null +++ b/docs/TAS_UPSTREAM_INTEGRATION.md @@ -0,0 +1,75 @@ +# TAS upstream integration status + +This draft ports the Tekken 3, Pepsiman and Biohazard replay work onto upstream +master `85cd26f0`. It is a source submission, not a qualification of the new +combined runtime. Release pins and existing campaign builds remain unchanged. + +## Review map + +| Area | Entry points | +|---|---| +| Profile and historical Tekken evidence | [TAS accuracy](TAS_ACCURACY.md), [replay tools](../tools/tasreplays/README.md) | +| CPU, exception and overlay integration | `recompiler/src`, `runtime/src/traps.c`, `runtime/include/overlay_api.h` | +| GPU and DMA service | `runtime/src/source_gpu_runtime.c`, `runtime/include/source_gpu_service_clock.h`, `runtime/src/dma.c`, `runtime/src/gpu.c` | +| CD-ROM, MDEC, SIO, SPU and timers | Corresponding modules under `runtime/src` and focused fixtures under `runtime/tests` | +| Biohazard source profile and limitations | [Nymashock source replay](NYMASHOCK_SOURCE_REPLAY.md) | +| Replay investigation and input identity | [Investigation guide](../tools/tasreplays/investigation.md) | +| Asset-free checks | `tools/tasreplays/tests/CMakeLists.txt`, `recompiler/CMakeLists.txt`, `.github/workflows/tasreplays.yml` | + +The foundation was published in fork PRs Alexbeav/psxrecomp#28, #29 and #30, +but had not reached upstream master. This branch starts from upstream and +selectively ports that foundation, subsequent Pepsiman fixes through +`39484acb`, controller/Biohazard work through `b73c16f3`, and cold-run, +investigation and GPU corrections through `7a0d0ff4`. It also ports the ordinary +scheduler precise-owner and pending GPU return corrections from `b1413e71`. + +Integration preserves upstream asynchronous GPU linked-list DMA, CFG metadata, +UNC handling and widescreen/mod callbacks. Overlay ABI 24 appends TAS callbacks +to upstream ABI 23. The generated-code identity checks remain enforced; old +fingerprints do not qualify a freshly generated upstream build. + +The bounded GPU upload-history fix is independently submitted as +[upstream PR #360](https://github.com/RetroPortingToolKit/psxrecomp/pull/360). +This integration contains the same fix and its regression test. + +## Evidence boundaries + +| Historical source build | Result | Limit | +|---|---|---| +| Tekken `0ff6e765` | 8,399 return clocks and RAM-page hashes matched; the original 7,974 inputs plus 426 neutral inputs reached YOU WIN, FINAL 8.80. | This is the original campaign build, not this rebased integration. Speed settings preserved identity but measured about 0.45x throughput. | +| Controller `fd4edc81` / equivalent Biohazard cherry-pick `85cf6437` | Cold Tekken 8,399/8,399 and Pepsiman 75,406/75,406 matched; Pepsiman terminal RAM matched. | The first Pepsiman attempt hit its watchdog at 60,863. An unchanged executable/input/profile retry on another local volume passed; the host cause remains unknown. | +| Biohazard `7a0d0ff4` | 239,202 returns and clean exit; exact prefix through 233,567; persisted card matched. | First clock mismatch at 233,568 was one cycle early; terminal RAM differed by nine bytes. This is not a fidelity pass. The historical native tail also differs from stock by one neutral input. | + +The controller acceptance record is the 2026-09-13 corpus snapshot with SHA-256 +`dab598ca4673896c86daa8dceb06181637c9d47642c171840fc1bfde8d0d1904`. +Private replay traces and machine-local evidence are not part of this submission. +The compressed Tekken reference contains comparison hashes, not RAM contents. +The large GPU JSON fixtures are synthetic command/oracle cases. + +## Validation on this integration + +The recompiler and asset-free TAS targets build with GCC 16.1 on Windows UCRT, +CMake/Ninja, and Python 3.12. All 160 enabled CTest tests pass; the three +pre-existing disabled tests are `interpreter_perf_guards`, +`runtime_perf_diag_guards` and `overlay_pair_dedup_runtime`. +All 28 changed Markdown files pass the relative-link check. +Focused scheduler and pending-GPU-return fixtures execute production modules +at both `-O0` and `-O2`; unrelated link seams abort if reached. + +The actual `psx-runtime` target also links with the no-BIOS configuration, +debug tools enabled, and UI, netplay, rewind, setup wizard and Vulkan disabled. +This build check does not exercise a retail game. No new retail replay has +qualified this combined upstream base. + +## Deferred work + +The T52 boot-state wire/checkpoint stack and the newer `155003cb` checkpoint +implementation are excluded. Their dependencies and full/cross-build restore +qualification are separate work. This branch exposes no checkpoint launcher +and makes no arbitrary-resume or cross-build restore claim. Source-profile CD +snapshots remain rejected when the required state cannot be represented. + +Before this draft is marked ready, regenerate and qualify the integrated retail +builds with their exact input, source and output identities. Resolve or explicitly +retain the Biohazard mismatch as a documented profile limitation. Do not infer +release acceptance from the historical results or asset-free tests. diff --git a/docs/internal/ACCURACY_BURNDOWN.md b/docs/internal/ACCURACY_BURNDOWN.md index 6e8aaa07..d0df91fd 100644 --- a/docs/internal/ACCURACY_BURNDOWN.md +++ b/docs/internal/ACCURACY_BURNDOWN.md @@ -185,7 +185,10 @@ bug**, not timing. soak PASSED (2026-07-06). Trivially revertable (4 bytes, `cdrom.c` Test 0x20). - **Status reg `0x1F801800` bit 2 (ADPBUSY)**: idles at 0 (was wrongly pinned set under an "ADPCM empty" label). psx-spx: bit2 = XA playback → 0 when idle. - - **GetID license region**: derived from disc serial (`SCEA`/`SCEE`/`SCEI`) + - **GetID license region**: derived from disc serial (`SCEA`/`SCEE`/`SCEI`). + Disc identification reads the actual ISO9660 `SYSTEM.CNF` before falling + back to an early-image scan, so a late boot file is not missed or replaced + by an unrelated `cdrom:` string in executable data. instead of hardcoded `SCEI`; NTSC-U (MMX6/Tomba) now correctly report `SCEA`. - Method reminder (top-of-doc LESSON): these were HYPOTHESES until OUTPUT validation — the MMX6/Tomba playtest soak WAS that validation. Revert any diff --git a/docs/internal/FAITHFUL_TIMING_PLAN.md b/docs/internal/FAITHFUL_TIMING_PLAN.md index 3a036f59..0bd5ba5a 100644 --- a/docs/internal/FAITHFUL_TIMING_PLAN.md +++ b/docs/internal/FAITHFUL_TIMING_PLAN.md @@ -232,6 +232,25 @@ on a fixed region -> next. checkpoint; refresh onto upstream 7025bb5f (rewind-key aliases only) and recheck before merging. Game pins and other PRs remain untouched. See `SIO_CARD_NO_HACKS_AUDIT.md`; central issue beads-eio.3.154. +- **2026-09-13 (PR361 Python test discovery):** Discover Python before the + GNU TAS contract-test registrations expand their commands. A fresh original + CHD-enabled configure produced 16 script-as-executable commands; the patched + configure discovers Python without a cache override and wraps all 16 + correctly. Python is required only for the GNU test suite; testing disabled + still configures without Python, and Clang retains optional Python tests. + Fresh GNU 16.1 / CHD-enabled build and all 160 enabled CTests pass; three + pre-existing disabled tests remain disabled. The local mixed-MinGW overlay + fixture required matching compiler runtime DLLs staged beside the test-build + emitter (not a source change or fingerprint bypass). + This is a test-configuration fix, not retail replay qualification or a + resolution of the runtime conflicts with the landed no-hack SIO removal. + Tracking: central issue `beads-eio.3.157`. + +- **2026-09-13 (TAS upstream submission):** Port the authored replay profile, + source-oracle component fixtures and cold launchers onto upstream `85cd26f0`. + Preserve current upstream DMA, GPU polyline, CUE mount and overlay callbacks. + Retail acceptance remains tied to the earlier campaign builds; see + [integration status](../TAS_UPSTREAM_INTEGRATION.md). No release pin promotion. - **2026-09-12 (FMV brief follow-ups, correctness separated from experiments):** `fix/cfg-metadata-integrity` repairs missing live reverse edges and replaces @@ -299,6 +318,13 @@ on a fixed region -> next. builds also pass their 600-frame headless gates. Visible software-renderer routes and another fork review are still required before the public branch can change. +- **2026-09-09 (TAS host-speed validation):** The published Tekken 3 cold-boot + route reproduces the 8.80 victory with 7,974 unchanged inputs and 8,399 exact + RAM/clock returns. Added replay `--speed` control using the existing host + fast-forward path, including 32x/64x caps and an uncapped mode. Guest timing, + code generation and controller routing are unchanged. Full visible 4x, 8x, + 32x and uncapped checks use the same fingerprint gate; results belong in the + speed-validation PR. Requested caps and measured throughput are separate. - **2026-07-28 (per-game host audio cushion — implemented, parser validated):** Added `[audio] buffer_ms` as a runtime-only developer setting with a guarded diff --git a/docs/playbooks/AUTOMATED_TESTING.md b/docs/playbooks/AUTOMATED_TESTING.md new file mode 100644 index 00000000..36cb8129 --- /dev/null +++ b/docs/playbooks/AUTOMATED_TESTING.md @@ -0,0 +1,165 @@ +# Automatic accuracy testing + +Every new compiled title should produce a test receipt alongside its build receipt. +Use tool-assisted speedruns (TAS), memory-card routes, and focused component tests +together. A passed TAS proves a specific route under specific conditions. +It does not establish a percentage of whole-game or hardware accuracy. + +## Current implementation + +This repository provides the [Tekken setup and replay tools](../../tools/tasreplays/README.md) +and the [qualified TAS comparison](../TAS_ACCURACY.md). +The [unattended investigation controller](../../tools/tasreplays/investigation.md) +adds explicit build/test/replay recipes, bounded diagnostics, durable recovery, +and strict per-return evidence reports. It does not lift operator replay holds. +The native runner validates original inputs, complete RAM/clock observations, +and a bounded ending. The visible run --speed 32 or --speed max option sets a cap; run --headless is already uncapped. Measure actual elapsed time separately. + +A separate private portfolio harness currently performs input admission, +repeat checks, coverage reporting, and sequential speed trials. +That harness is outside this source checkout. Save-route adapters and automatic +integration with every title build are not yet supplied by this repository. +The remaining sections define the expansion plan and acceptance rules. + +## Per-build gate + +1. Record the source commit, compiler, generated-code identity, and executable hash. +2. Run source-owned component regressions for the changed subsystems. +3. Bind the title revision, BIOS, emulator core/version, input, cards, and settings. +4. Boot each case from its declared reset or fresh card load. +5. Check progression and behavior at declared observation points. +6. Repeat each case from a clean start and compare its deterministic outputs. +7. Report failed checks and missing requirements separately from build success. + +A shared runtime change should test every registered affected title. +Changes to CPU, scheduling, memory, or BIOS dispatch should default to all +registered titles. Use narrower selection only with recorded dependency evidence. +A missing title suite is an explicit gap; it must not become an implicit pass. +This integration policy is the target for build entry points, not a claim that +all existing build scripts already implement it. + +## Test layers + +| Layer | Required observation | Typical coverage | +|---|---|---| +| Component programs | Hardware-qualified result or documented reference trace | CPU/GTE instructions, exceptions, interrupts, DMA, timers, GPU, SPU, MDEC, CD and serial devices | +| Original full TAS | Unchanged input plus qualified state boundaries and ending | Connected progression and accumulated timing differences | +| Progression cards | Fresh boot/load, explicit markers, then controlled gameplay | Later areas and normal persistence | +| Short alternate routes | Specific action and expected effect | Death, retry, pause, menus, optional paths, unusual resources | +| Media samples | Pixel/VRAM and audio evidence at aligned points | Missing layers, movies, music, dialogue and timing | +| Lifecycle and endurance | Repeated transitions plus terminal state | Long sessions, save/reload, disc changes and resource growth | + +Component tests must exercise the relevant path through the recompiler and +runtime, not merely a standalone helper with a similar implementation. +For generated and fallback execution paths, retain separate evidence where the +path matters. Reduce title failures into small authored programs when possible. +Keep the original title route as an integration regression. + +## Reference selection + +Replay each TAS in the exact source core and version first. +Pin firmware, disc topology, controller mode, card state, reset mode, and sync settings. +Confirm that the source replay reaches the intended endpoint before admitting it. +Tekken's Octoshock 2.2.2 compatibility work does not qualify another core version. +The selected [Pepsiman movie](https://tasvideos.org/6044S) uses Octoshock 2.3.0; +the selected [RE1 movie](https://tasvideos.org/8957S) uses Nymashock 2.9.1. + +Native replay can work when the necessary timing is reproduced, as the Tekken +route demonstrates. A desynchronization is useful evidence, but it is not by +itself proof of a specific hardware bug. Find the first divergent input poll, +clock, state, or device event before assigning ownership. + +A second independent emulator can help isolate a disagreement. +Agreement among emulators does not overrule measured hardware behavior. +Track source-emulator compatibility separately from hardware accuracy. + +## Memory cards + +Start with four to eight progression saves where the game supports them. +Choose saves by system coverage as well as distance through the story. +Include early gameplay, a transition-heavy area, a boss, and late-game state. +Add alternate characters, inventories, or routes when they change behavior. + +Keep raw 128 KiB card images and record the original file and conversion recipe. +Verify the card structure and the title's actual ability to load it. +Region, revision, and save identifiers matter; renaming a file or identifier +does not establish compatibility. Give every run an independent writable copy. + +A card contains persistent saves, not the complete machine state. +To make reference markers, boot the oracle and load that card through the game. +Observe the resulting area, flags, inventory, health, and player control. +Use the same procedure in the candidate. Arbitrary emulator save states are +not portable runtime checkpoints. + +A TAS can produce useful cards only when it performs saves or when a separately +recorded save route is added. Many speedruns skip saving entirely. +Keep the original TAS unchanged; qualify the save-producing route separately. +Keep BIOS/card component checks in parallel with route acquisition, so one +shared card failure does not waste a full title campaign. + +## Comparison rules + +Use exact clocks and RAM when equivalent execution boundaries are established. +Use declared semantic markers and bounded waits for fresh save loads whose +complete initial machine state is not identical. A bounded wait is part of +that test; it must not silently retime an original TAS. + +Keep timing-sensitive reads, instrumentation overhead, and capture settings explicit. +Read-only observers should not patch guest memory to manufacture a milestone. +Function-level comparisons need all relevant memory, device state, and side +effects in their contract; register inputs alone are insufficient for I/O code. + +Keep correctness tests at the original presentation settings. +Exact software-renderer comparisons are useful where the reference permits them. +Enhanced rendering needs separate expectations and declared tolerances. +Audio deserves real sample or feature checks, including channel activity, +clipping, duration, transitions, and synchronization; silence detection alone +cannot qualify sound correctness. + +## Speed and test budgets + +Alex deferred performance work on September 9, 2026. +The active campaign expands TAS and save coverage at the current execution speed. +The speed rules below apply only to a later performance campaign. + + +First prove the route twice and measure its unbounded throughput. Test 1x, 8x, 16x, and 32x only while measured throughput supports the next target. Stop after an unattained target and improve actual throughput before testing higher caps. +Change host pacing only; retain every guest operation and original input boundary. +Record requested and achieved speed, guest cycles, elapsed host time, affinity, +priority, diagnostic settings, and completion. Repeat each speed setting. +If throughput misses its target, report it as unattained even if state matches. + +Keep cheap component and short route tests in the per-build gate. +Run full registered routes for qualification; use longer scheduled endurance +campaigns only after the routes are stable. A schedule must identify its source +and fixtures. This document does not create a background schedule or CI job. +Keep retail data and private state captures on authorized private test hosts. +Source-owned tests can run in public CI. + +## Failure handling and coverage + +Retain the first divergent observation, exact inputs, source/build identity, +terminal state, and the shortest known reproducer. Distinguish a setup failure, +reference desync, candidate mismatch, timeout, and missing coverage. +Consult the shared findings before adding instrumentation or changing a model. +Make each verified fix a reusable regression and rerun affected title routes. + +Track systems, content branches, persistence, presentation, and lifecycle as +separate requirements. Count executed code only as supporting coverage evidence. +A single speedrun can bypass most characters, weapons, saves, deaths, and menus. +User reports should become deterministic regressions where practical. +The size of the remaining human-only coverage is unknown until this matrix is populated. + +## Implementation order + +1. Retain the qualified Tekken replay as a regression for shared runtime changes. +2. Qualify Pepsiman's original source replay, then its native route. +3. Add RE1's exact Nymashock replay and the first fresh-load save adapter. +4. Add alternate paths, card lifecycle, and media checks to those title matrices. +5. Attach the shared admission/report contract to each supported build entry point. +6. Expand long-session and multi-disc coverage before admitting an FF7 full run. + +The [TAS accuracy record](../TAS_ACCURACY.md) owns the current replay claim. +The [runner guide](../../tools/tasreplays/README.md) owns supported commands. +Private portfolio contracts retain campaign receipts, coverage gaps, and host +performance admission; this plan does not change a title qualification state. diff --git a/tools/tasreplays/README.md b/tools/tasreplays/README.md index 3fe4b5af..2293762e 100644 --- a/tools/tasreplays/README.md +++ b/tools/tasreplays/README.md @@ -5,8 +5,11 @@ see the [unattended investigation controller guide](investigation.md). Build and play [Spikestuff's tool-assisted speedrun](https://tasvideos.org/4164M) from your own Tekken 3 USA disc and SCPH1001 BIOS. The original 7,974 inputs -finish Arcade with Yoshimitsu at **8.80 seconds**. This branch reproduces the -qualified Octoshock 2.2.2 timing configuration in the native psxrecomp runtime. +finish Arcade with Yoshimitsu at **8.80 seconds**. The original campaign +qualified the Octoshock 2.2.2 comparison configuration. This upstream integration +is pending fresh retail qualification; see [current status](../../docs/TAS_UPSTREAM_INTEGRATION.md). +Setup retains the historical generated-code fingerprint gate and can refuse +changed output from the updated upstream emitter. ## Quick start: Windows x64 @@ -18,7 +21,7 @@ Allow several GB of disk space and a few minutes for the initial build. Internet access is needed for the original TAS and the pinned SDL3/zlib sources. ```powershell -git clone --branch tasreplays https://github.com/Alexbeav/psxrecomp.git +git clone --branch codex/tas-upstream-integration-20260913 https://github.com/Alexbeav/psxrecomp.git cd psxrecomp python tools/tasreplays/tekken3.py setup --disc "D:/Games/Tekken 3 (USA).cue" --bios "D:/BIOS/SCPH1001.BIN" python tools/tasreplays/tekken3.py run @@ -119,13 +122,13 @@ See [accuracy changes](../../docs/TAS_ACCURACY.md) for the repair scope. The replay deliberately ends at the observed victory boundary, matching the declared426 neutral inputs after the original movie. A longer exploratory tail in the research build stopped at the explicit **unqualified CDDA Play seek** -guard after return10,704. That later behavior is still unresolved. This branch +guard after return10,704. That statement describes the original Tekken revision; later optional CDDA +component work is documented in [the source CDDA profile](../../docs/source-cdda-profile.md). This branch does not claim unrestricted post-victory playback, complete audio/pixel parity, save-state support, PAL support, or other-game TAS success. -The source starts from the exact validated framework base -`f23c5ba1a220fe1ca8818cc48c026d6c2f7f2c64`; it is a dedicated branch in the -Alexbeav fork. It does not update that fork's main branch or upstream master. +The historical campaign started from `f23c5ba1`. This submission starts from +upstream `85cd26f0`. It does not change release pins or merge itself. ## Tests without retail assets @@ -135,7 +138,7 @@ cmake --build build/tests --parallel 8 ctest --test-dir build/tests --output-on-failure ``` -The suite includes the existing recompiler tests, the ABI22 overlay boundary +The suite includes the existing recompiler tests, the ABI24 overlay boundary tests, O0/O2 TAS input/clock/controller regressions, BK2 validation, and failures for incorrect media topology and replay evidence. Three pre-existing disabled tests remain visible in CTest. Historical fixtures that need the separately diff --git a/tools/tasreplays/investigation.md b/tools/tasreplays/investigation.md index efe49c26..86859aef 100644 --- a/tools/tasreplays/investigation.md +++ b/tools/tasreplays/investigation.md @@ -91,7 +91,7 @@ with existing launcher options such as `--ram-snapshot-frame`. Configure bounded CPU/device windows explicitly with the launcher's supported options. The controller does not invent clock conversions, patch a source observer, or bypass checkpoint guards. In particular, the full Biohazard profile cannot capture/resume checkpoints, -and supported checkpoint profiles remain bound to the same executable. +and this upstream integration does not include the separate TAS state format. Top-level `snapshots: {"source": "...", "native": "..."}` optionally selects the diagnostic directories; otherwise the original comparison pair is used. The first