mirror of https://github.com/RetroPortingToolKit/psxrecomp synced 2026-10-26 10:19:47 -04:00

Files

T

Matthew Stanley 5aced6c306 test: register 21 written-but-unwired tests; document how to run them

Correction to an earlier claim of mine: ctest was NOT absent. recompiler/ and
runtime/ already had 21 add_test() calls between them and `ctest` ran 8 tests
green in the recompiler tree. My original survey used a quoted git-grep
pathspec that the shell passed literally, matched nothing, and I read the empty
result as "no test wiring anywhere". The real problem was narrower: partial
registration and zero documentation.

Of 61 first-party test sources (excluding 3 fixtures/harnesses), 37 were
referenced by no CMakeLists at all. Each was run by hand; 21 pass and are now
registered, taking the recompiler tree from 8 to 29 tests -- still under five
seconds, and still requiring no BIOS dump, no disc image, and no generated
code.

The 15 runtime source-invariant guards are registered from
recompiler/CMakeLists.txt rather than runtime/CMakeLists.txt, following the
existing dirty_text_continuation_guards precedent. They only read runtime
source, and the runtime tree cannot configure until a BIOS has been generated,
so registering them there would have made them unreachable from the only tree a
newcomer can build.

Three tests are deliberately NOT registered because they fail today. Wiring a
known-red test would recreate exactly what took CI off pull requests. Detail is
in docs/TESTING.md; briefly:

  test_interpreter_perf_guards.py    asserts psx_devices_mmio_sync invalidates
      the inline cycle limit. It does not. The function delegates either to
      psx_devices_service_to_now() (which clears g_psx_cycle_fast_limit,
      psx_cycles.c:161) or to psx_devices_recompute_deadline() (:153-157),
      and that second branch never clears it. Needs a timing owner: the guard
      may have found a real hole, or the invariant may have moved. The guard is
      itself partly stale -- it still names s_next_service_cycle, since renamed
      to psx_next_service_cycle.
  test_runtime_perf_diag_guards.py   asserts a substring no longer present in
      the runtime source.
  test_overlay_pair_dedup_runtime.py needs its companion harness built; not a
      source-only guard.

New docs/TESTING.md covers running the suite, running one test, what each group
needs, the known-failing three, and why fixtures must not be registered. It
also records two traps that cost time here: parallel builds can crash cc1plus
on config_loader.cpp under memory pressure (retry with -j 1, it is not a code
error), and the codegen tests need an ABSOLUTE recompiler path -- a relative one
passes their os.path.isfile check and then fails inside subprocess.run on
Windows with WinError 2, which reads as a broken test.

Linked from CONTRIBUTING.md (above the existing manual regression checklist,
noting the suite does not tell you whether a game still runs) and from
docs/BUILDING.md.

Verified: clean configure + build, ctest 29/29 pass in 4.08s.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-27 11:49:31 -07:00

4.9 KiB

Raw Blame History

Testing

Running the tests

cmake -S recompiler -B recompiler/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build recompiler/build
cd recompiler/build && ctest --output-on-failure

That is the whole thing. 29 tests, under 5 seconds, and it needs no BIOS dump, no disc image, and no generated code — a plain recompiler build is enough. This is the check to run before opening a PR.

Until 2026-07-27 no document in this repository mentioned ctest, pytest, or how to run a test at all, so the suite was effectively invisible. If you add a test, add it to ctest in the same commit — an unregistered test cannot fail, and a test that cannot fail is not a test.

On a memory-constrained machine, parallel builds of this tree can crash cc1plus while compiling the toml11-heavy config_loader.cpp. If cmake --build dies with no diagnostic, retry with -j 2 or -j 1; the failure is resource exhaustion, not a code error.

Running one test

ctest -R overlay_guard_codegen --output-on-failure   # by name (regex)
ctest -N                                             # list without running

Every test is also a plain executable or script, so you can run it directly:

./recompiler/build/bios_address_model_test
python recompiler/tests/test_overlay_guard_codegen.py \
       --recompiler "$(pwd)/recompiler/build/psxrecomp-game.exe"

Pass the recompiler as an absolute path. A relative path satisfies the test's own os.path.isfile check but then fails inside subprocess.run on Windows with WinError 2, which looks like a broken test rather than a bad argument.

What the suite covers

Group	Where	Needs
C/C++ unit tests	`recompiler/tests/`, `runtime/tests/`	recompiler build
Codegen contract tests	`recompiler/tests/test_*.py`	`psxrecomp-game`
Runtime source-invariant guards	`runtime/tests/test_*.py`	nothing — they read source

The source-invariant guards are the cheapest and most useful class here. They assert structural properties of the runtime (an ordering holds, a fast path is invalidated, a fallback exists) by reading the source, so they cost milliseconds and catch whole regression classes without running a game. Registered from recompiler/CMakeLists.txt rather than runtime/CMakeLists.txt, because the runtime tree cannot configure until a BIOS has been generated, and these need neither.

Known-failing tests (not registered)

Three tests exist and are deliberately left out of ctest because they fail today. They are not registered because a suite with a known-red test is a suite people stop believing — the exact failure mode that took CI off pull requests in the first place (see .github/workflows/cli-release.yml). Fix or retire them, then wire them in.

Test	Status
`runtime/tests/test_interpreter_perf_guards.py`	Asserts `psx_devices_mmio_sync` invalidates the inline cycle limit. It does not: the function delegates to `psx_devices_service_to_now()` (which clears `g_psx_cycle_fast_limit`, `psx_cycles.c:161`) or to `psx_devices_recompute_deadline()` (`:153-157`), and that second branch never clears it. Needs a timing owner to decide whether the guard found a real hole or the invariant moved. The guard is also partly stale — it still names `s_next_service_cycle`, since renamed to `psx_next_service_cycle`.
`runtime/tests/test_runtime_perf_diag_guards.py`	Asserts a substring that is no longer present in the runtime source. Either the diagnostic was removed or it was renamed; the guard has not been updated either way.
`runtime/tests/test_overlay_pair_dedup_runtime.py`	Needs its companion harness (`overlay_pair_dedup_harness.c`) built. Unlike the other Python tests it is not source-only, so it needs a build target before it can be registered.

Tests that are not in `ctest` and should not be

runtime/tests/ also holds fixtures and harnesses (*_fixture.c, *_harness.c) that are inputs to other tests, not tests themselves. Do not register them.

Several C tests under runtime/tests/ require a built runtime, which requires a generated BIOS (see BUILDING.md). Those are wired into runtime/CMakeLists.txt and run from runtime/build:

cd runtime/build && ctest --output-on-failure

CI

.github/workflows/cli-release.yml runs on workflow_dispatch and published releases only. Its header explains why per-PR triggers were removed on 2026-07-25: the Windows job failed often enough that a red check stopped carrying information, and a check nobody trusts costs attention without buying confidence.

That reasoning still holds. The gap it left was that no fast, trustworthy alternative existed. The ctest suite above is a candidate: it is hermetic (no BIOS, no disc, no network), takes under five seconds, and is currently green. Restoring a per-PR check on top of it is a smaller decision than restoring the old one.

4.9 KiB Raw Blame History