Record real renderer activity and explicit playtest verification coverage

This commit is contained in:
Matthew Stanley
2026-10-18 20:55:22 -07:00
parent b3a82a2a6b
commit db596520ab
16 changed files with 635 additions and 16 deletions
+85 -3
View File
@@ -36,9 +36,12 @@ No texture feature,
texture package or other enhancement is enabled or exposed by this helper.
The implementation reads no title-specific addresses. `telemetry.csv` stores
host intervals, incident markers, CD activity and render-pass counters. These
are passive evidence and never feed replay or simulation. Press L3 + R3 together
to mark an incident; holding the chord produces one marker. Game-specific
host intervals, incident markers, CD activity, ordinary GP0 draw commands,
host presentation calls, effective video configuration and optional render-pass
state-verification counters. These are passive evidence and never feed replay
or simulation. Press L3 + R3 together
to mark an incident and queue a composed screenshot on supported presenters;
holding the chord produces one marker/request. Game-specific
gameplay samplers remain separate, validated adapters.
## Feedback bundle
@@ -55,6 +58,9 @@ Quit normally after reproducing a problem and zip the complete session folder:
- `inputs.csv` and `session.json` describe inputs and execution/build identity.
- `telemetry.csv` contains timing, incident, CD and renderer evidence.
- `incident-<number>.png`, when the presenter completed an incident capture,
shows composed output rather than only native VRAM. OpenGL and software
support this path; headless and Vulkan currently reject it explicitly.
- `mods-state.toml`, optional `settings.toml`, and starting cards or explicit
absence markers preserve the original choices and save data.
- A slot-state restore starts a new segment with its complete `baseline.pst`.
@@ -65,6 +71,78 @@ contain the player's saves. Recording and input display are independently
controlled by the title's mod features. Netplay frames are skipped; this
offline recorder does not claim to reproduce network sessions.
## Coverage and effective video state
`psxrecomp-telemetry-v2` preserves the original thirteen CSV columns and appends
the following evidence. Input-v2 and strict replay checks are unchanged.
- `gpu_draw_commands`: cumulative ordinary GP0 drawing commands observed by the
GPU front end. This is command activity, not a count of correct rendered pixels.
- `host_presents` / `host_present_errors`: cumulative returned/failed calls
from the actual host present API. OpenGL's direct and threaded paths, Vulkan,
and software presentation publish these without requiring debug tools.
`host_present_status_available` says whether the latest host present API
supplied a result. SDL2 has no return status, so its activity counter only
proves the call returned; zero errors under SDL2 is missing error coverage. Vulkan
success includes `VK_SUBOPTIMAL_KHR`. Neither API proves monitor scanout.
Counts include repeated images, pause/overlay presents and generated frames;
do not label them unique game frames. GP0 commands and presents need not be 1:1.
- `video_renderer`: the active backend, after fallback (`0` none/headless,
`1` software, `2` OpenGL, `3` Vulkan). `video_resolution_preset`,
`video_aspect_num/den` and `video_widescreen` sample the emulation owner's
settings after mod overrides, including changes during the session.
- `video_snapshot_valid`: whether a coherent passive backend snapshot was
available. `composed_*` fields describe the last composed image's backend,
actual internal scale, native source dimensions, drawable dimensions and
fitted viewport dimensions. Dimensions unavailable on a path are zero.
A threaded renderer can compose ahead of the swap counter; these fields do
not identify the exact image a particular swap displayed. FMV/menus may use
a narrower viewport than the configured widescreen mode.
- `render_verify_enabled`, existing `render_verify_checks/mismatch`, and
`render_verify_ever_executed`: enablement, cumulative checks/failures and
whether any optional extra render-pass state-preservation check has run.
These do not monitor ordinary rendering or texture correctness. An enabled
check that never executes supplies no coverage. To measure a recording
interval, compare the counter increase, not just a preexisting nonzero value.
- `visual_verify_available` / `visual_verify_checks`: currently zero because
this recorder has no independent pixel oracle. This is explicit missing
coverage, never a successful texture test. Screenshots/clips and matched
original-game comparisons remain necessary for visible defects.
- `incident_capture_request`: `-1` means no request on this row, `0` rejected,
and `1` queued on the existing composed-output capture path. The next
presented image is written as `incident-<number>.png` if capture succeeds;
an accepted request alone is not a completed screenshot. The analyzer lists
accepted/rejected requests separately from image files present in the bundle.
Capturing on an explicit incident may cost a readback/PNG write; ordinary
frames do not request pixel capture. A pending diagnostic request can be
replaced by a later explicit screenshot request before it is fulfilled.
The backend observer does not read pixels, query GL errors, synchronize the
render thread or change guest state. Its snapshot makes at most three attempts
and reports unavailable if a coherent publication cannot be sampled.
Analyze either v1 or v2 without launching a game:
```text
python replay-diagnostics.py <session-dir> --analyze
```
The JSON report separates activity, optional state checks and visual coverage.
Old v1 bundles explicitly report absent ordinary-renderer/effective-mode data.
Missing probes, zero executed checks, sequence discontinuities and counter
resets are visible; zero mismatches alone is never presented as a pass. A torn
final row is ignored, while a malformed complete row is rejected.
A diagnostic recording is helpful for exact input/state reproduction, but is
not a prerequisite for investigating a visual report. Reproduce known menu,
attract-demo, transition and resize cases directly with a fresh private save
directory and muted automated output. Capture the composed presentation, since
raw/native VRAM screenshots can omit the very widescreen/HD transformation at
issue. Preserve location, camera direction, effective mode and before/after
images. Escalate to an incident-marked recording when the trigger is unclear
or depends on intermittent timing; do not classify an untried case as
improbable merely because the reporter supplied no log.
## Replay
Use Python 3.8+ (install `tomli` before Python 3.11) with the recording's original
@@ -114,3 +192,7 @@ debouncing, first divergence, identity mismatches, save-state baselines, readonl
output fallback, release-build input-only rejection and isolated replay setup.
They establish the shared engine-boundary contract. Actual title/disc playtests
are still needed to qualify each game's original-build replay.
The video tests additionally cover concurrent passive snapshots, composition
versus completed presentation, failed presents, live scale/aspect changes,
ordinary activity with no optional checks, legacy coverage reporting and the
analysis-only CLI.
@@ -4,8 +4,14 @@
namespace psx_playtest_diagnostic {
// Host evidence only. No game-specific addresses or classification assumptions.
using TelemetryRow = std::array<int64_t,13>;
using TelemetryRow = std::array<int64_t,36>;
constexpr const char *TelemetryHeader =
"sequence,host_elapsed_us,host_interval_us,incident,cd_reading,cd_pending,"
"cd_lba,cd_data_read,cd_xa,render_passes,render_aborted,render_verify_checks,render_verify_mismatch\n";
"cd_lba,cd_data_read,cd_xa,render_passes,render_aborted,render_verify_checks,render_verify_mismatch,"
"gpu_draw_commands,host_presents,host_present_errors,video_snapshot_valid,video_renderer,"
"video_resolution_preset,video_aspect_num,video_aspect_den,video_widescreen,"
"composed_renderer,composed_internal_scale,composed_source_width,composed_source_height,"
"composed_drawable_width,composed_drawable_height,composed_viewport_width,composed_viewport_height,"
"render_verify_enabled,render_verify_ever_executed,visual_verify_available,visual_verify_checks,"
"incident_capture_request,host_present_status_available\n";
}
+3
View File
@@ -75,6 +75,9 @@ typedef struct RenderPassStats {
} RenderPassStats;
void render_pass_get_stats(RenderPassStats *out);
/* Whether optional state-preservation verification is enabled. Zero executed
* checks still means no coverage even when this is enabled. Not a pixel test. */
int render_pass_verification_enabled(void);
/* Clears counters and the sticky fault latch (new mod session). */
void render_pass_reset_session(void);
typedef struct RenderStereoStats {
+47
View File
@@ -0,0 +1,47 @@
#ifndef PSX_VIDEO_TELEMETRY_H
#define PSX_VIDEO_TELEMETRY_H
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
enum { PSX_VIDEO_NONE = 0, PSX_VIDEO_SOFTWARE = 1,
PSX_VIDEO_OPENGL = 2, PSX_VIDEO_VULKAN = 3 };
typedef struct PsxVideoTelemetry {
uint64_t presents, present_errors;
int valid, renderer, internal_scale, present_status_available;
int source_width, source_height;
int drawable_width, drawable_height;
int viewport_width, viewport_height;
} PsxVideoTelemetry;
/* Render-owner publication, before submission. These describe the latest
* composed image, which can lead actual swaps when presentation is threaded.
* Zero dimensions mean unavailable, not a zero-sized successful image.
* A backend has one composing owner at a time. No readbacks or GL syncs. */
void psx_video_telemetry_compose(int renderer, int scale, int source_w, int source_h,
int drawable_w, int drawable_h, int viewport_w, int viewport_h);
/* Call after the backend's swap/present API returns. A success means the host
* API accepted/completed the call, not that a monitor displayed correct pixels.
* Counts include repeats, overlays and generated frames. */
/* result: 1 accepted, 0 failed, -1 returned but API supplies no status (SDL2). */
void psx_video_telemetry_present(int result);
/* Bounded passive read: valid=0 if a coherent composition was unavailable.
* Completion/error counters remain available in that case. */
void psx_video_telemetry_snapshot(PsxVideoTelemetry *out);
/* Emulation-owner settings after runtime mod overrides, sampled every VBlank.
* Actual allocated scale/dimensions come from the separate backend snapshot. */
typedef struct PsxVideoConfigTelemetry {
int renderer, resolution_preset, aspect_num, aspect_den, widescreen;
} PsxVideoConfigTelemetry;
void psx_video_config_telemetry(PsxVideoConfigTelemetry *out);
/* Queue one composed-output PNG on the existing presenter capture path.
* Zero means rejected/unavailable. Acceptance is not proof the file was written.
* Called only for an explicit owner incident, never for every ordinary frame. */
int psx_video_telemetry_capture(const char *path);
#ifdef __cplusplus
}
#endif
#endif
+1
View File
@@ -440,6 +440,7 @@ set(PSXRECOMP_RUNTIME_SOURCES
${PSXRECOMP_ROOT}/runtime/src/bios_hle_plan.c
${PSXRECOMP_ROOT}/runtime/src/savestate.c
${PSXRECOMP_ROOT}/runtime/src/video_texture_filter.c
${PSXRECOMP_ROOT}/runtime/src/video_telemetry.cpp
${PSXRECOMP_ROOT}/runtime/src/psx_savestate_menu.c
${PSXRECOMP_ROOT}/runtime/src/psx_rewind.c
${PSXRECOMP_ROOT}/runtime/src/host_osd.c
+13 -1
View File
@@ -79,6 +79,7 @@
#include "gl_scale_limits.h"
#include "gpu_sw_renderer.h"
#include "gpu_gl_renderer.h"
#include "video_telemetry.h"
#include "gpu_hd_textures.h"
#include "mod_texture_banks.h"
#include "gl_batch_policy.h"
@@ -1401,6 +1402,10 @@ static int postfx_active(void);
static void postfx_present(int fresh, int with_aa, int lx, int ly, int lw, int lh);
static void pres_record(int path, int dx, int dy, int w, int h,
int lx, int ly, int lw, int lh) {
int drawable_w = 0, drawable_h = 0;
SDL_GL_GetDrawableSize(s_win, &drawable_w, &drawable_h);
psx_video_telemetry_compose(PSX_VIDEO_OPENGL, s_out_scale, w, h,
drawable_w, drawable_h, lw, lh);
/* The composed game image, before hold-last capture and the OSD: the
* post-effect chains (bloom, [video] post_shader, AA; gl_postfx.inc). */
if ((path == GL_PRES_VRAM || path == GL_PRES_WIDE || path == GL_PRES_INTERP) && postfx_active())
@@ -9606,7 +9611,14 @@ static int pt_cb_ctx(void *user, int current) {
/* The one owned swap call: the composing thread's (direct) or the present
* thread's. */
static void gl_window_swap(void) { SDL_GL_SwapWindow(s_win); }
static void gl_window_swap(void) {
#if defined(PSX_SDL3)
psx_video_telemetry_present(SDL_GL_SwapWindow(s_win) ? 1 : 0);
#else
SDL_GL_SwapWindow(s_win);
psx_video_telemetry_present(-1); // SDL2 exposes no swap result.
#endif
}
static void *pt_cb_present(void *user, int slot, void *ready) {
(void)user;
+5 -1
View File
@@ -20,6 +20,7 @@
#include "gpu.h"
#include "gpu_render.h"
#include "gpu_vk_renderer.h"
#include "video_telemetry.h"
#include "gpu_sw_renderer.h"
#include "host_osd.h"
#include "psx_savestate_menu.h"
@@ -1675,7 +1676,10 @@ static void submit_present(VkCommandBuffer cb, uint32_t img_idx, uint32_t fr) {
pi.swapchainCount = 1; pi.pSwapchains = &s_swapchain;
pi.pImageIndices = &img_idx;
uint64_t present_start = SDL_GetPerformanceCounter();
p_vkQueuePresentKHR(s_queue, &pi);
psx_video_telemetry_compose(PSX_VIDEO_VULKAN, s_scale, 0, 0,
(int)s_sc_extent.width, (int)s_sc_extent.height, 0, 0);
const VkResult present_result = p_vkQueuePresentKHR(s_queue, &pi);
psx_video_telemetry_present(present_result == VK_SUCCESS || present_result == VK_SUBOPTIMAL_KHR);
s_perf_cur.present_us += perf_elapsed_us(present_start);
s_frame_idx++;
}
+42 -4
View File
@@ -123,6 +123,7 @@ extern "C" uint32_t g_guest_last_store_addr, g_guest_last_store_val;
#include "mod_session_baseline.h"
#include "mod_runtime.h"
#include "video_texture_filter.h"
#include "video_telemetry.h"
#include "host_launch_timing.h"
#include "mod_packages.h"
#include "present_image_ring.h"
@@ -2553,6 +2554,38 @@ extern "C" void psx_video_resolution_info(int *preset, int *ref_lines, int *requ
if (px_h) *px_h = ph;
}
extern "C" void psx_video_config_telemetry(PsxVideoConfigTelemetry *out) {
if (!out) return;
*out = {};
out->renderer = g_gl_active ? PSX_VIDEO_OPENGL : g_vk_active ? PSX_VIDEO_VULKAN
: g_headless ? PSX_VIDEO_NONE : PSX_VIDEO_SOFTWARE;
out->resolution_preset = effective_internal_resolution();
out->aspect_num = g_video_aspect_num;
out->aspect_den = g_video_aspect_den;
out->widescreen = g_ws_engaged && g_video_aspect_num * 3 != g_video_aspect_den * 4;
}
#ifndef PSX_SDL_NO_RENDER
/* All software swaps, including display-disabled and held frames. Query only
* the owning SDL renderer; never synchronize another render thread. */
static void software_present_with_telemetry(SDL_Renderer *renderer) {
int width = 0, height = 0;
#if defined(PSX_SDL3)
SDL_GetRenderOutputSize(renderer, &width, &height);
#else
SDL_GetRendererOutputSize(renderer, &width, &height);
#endif
psx_video_telemetry_compose(PSX_VIDEO_SOFTWARE, gr_scale(), 0, 0,
width, height, 0, 0);
#if defined(PSX_SDL3)
psx_video_telemetry_present(SDL_RenderPresent(renderer) ? 1 : 0);
#else
SDL_RenderPresent(renderer);
psx_video_telemetry_present(-1); // SDL2's API has no return status.
#endif
}
#endif
extern "C" int present_shot_request(const char *path)
{
if (!path || !*path) return 0;
@@ -2572,6 +2605,11 @@ extern "C" int present_shot_request(const char *path)
/* Backend hook: atomically claim a staged shot (returns 1 and fills `out`),
* then call present_shot_done(ok) once the PNG is written — or not written. */
extern "C" int psx_video_telemetry_capture(const char *path) {
if (!path || std::strlen(path) >= sizeof(s_present_shot_path)) return 0;
return present_shot_request(path);
}
extern "C" int present_shot_take(char *out, int n)
{
if (!out || n <= 0) return 0;
@@ -7548,7 +7586,7 @@ static void netplay_hold_last_present_tick(void) {
SDL_RenderClear(sdl_renderer);
SDL_RenderCopy(sdl_renderer, sdl_texture, &s_sw_hold_src, &s_sw_hold_dst);
host_osd_draw_sdl(sdl_renderer);
SDL_RenderPresent(sdl_renderer);
software_present_with_telemetry(sdl_renderer);
did = 1;
}
#endif
@@ -8253,7 +8291,7 @@ static void rewind_pause_present(void) {
SDL_RenderCopy(sdl_renderer, sdl_texture, &s_sw_hold_src,
&s_sw_hold_dst);
host_osd_draw_sdl(sdl_renderer);
SDL_RenderPresent(sdl_renderer);
software_present_with_telemetry(sdl_renderer);
}
#endif
}
@@ -9394,7 +9432,7 @@ static NetplayVblankEpilogue sdl_vblank_present_body(void) {
SDL_SetRenderDrawColor(sdl_renderer, 0, 0, 0, 255);
SDL_RenderClear(sdl_renderer);
host_osd_draw_sdl(sdl_renderer);
SDL_RenderPresent(sdl_renderer);
software_present_with_telemetry(sdl_renderer);
}
}
}
@@ -9874,7 +9912,7 @@ static NetplayVblankEpilogue sdl_vblank_present_body(void) {
{
latency_ring_mark(LAT_SWAP_BEGIN);
const Uint64 t0 = SDL_GetPerformanceCounter();
SDL_RenderPresent(sdl_renderer);
software_present_with_telemetry(sdl_renderer);
const Uint64 t1 = SDL_GetPerformanceCounter();
latency_ring_mark(LAT_SWAP_END);
netplay_note_present();
+24 -1
View File
@@ -22,6 +22,8 @@
#include "playtest_diagnostic_telemetry.h"
#include "cdrom.h"
#include "render_pass.h"
#include "gpu.h"
#include "video_telemetry.h"
#include "psx_sha256_file.h"
#include <array>
#include <algorithm>
@@ -61,6 +63,7 @@ psx_playtest_diagnostic::Writer<4096, psx_playtest_diagnostic::TelemetryRow> tel
std::chrono::steady_clock::time_point telemetry_start, telemetry_last;
unsigned incident;
bool incident_held, telemetry_failed;
std::filesystem::path recording_folder;
std::ifstream replay;
Row pending{}; // replay row the next VBlank verifies
bool pending_valid;
@@ -285,7 +288,8 @@ void seed_first_row() {
std::string session_json(bool netplay) {
std::ostringstream out;
out << "{\n \"format_version\": 1,\n \"input_format\": \"" PSX_DIAGNOSTIC_INPUT_FORMAT "\",\n"
<< " \"telemetry_format\": \"psxrecomp-telemetry-v1\",\n"
<< " \"telemetry_format\": \"psxrecomp-telemetry-v2\",\n"
<< " \"visual_verification\": \"unavailable: no independent pixel comparison\",\n"
<< " \"game_id\": \"" PSX_DIAGNOSTIC_GAME_ID "\",\n"
<< " \"boundary\": \"guest-vblank-sio-pad-state\",\n"
<< " \"digest\": \"fnv-mix over " << DigestWords << " main-RAM words per VBlank, stride "
@@ -309,6 +313,7 @@ void begin_recording() {
std::chrono::system_clock::now().time_since_epoch()).count();
const auto folder = psx_playtest_diagnostic_session_directory(
application_directory(), PSX_DIAGNOSTIC_GAME_ID, "session-" + std::to_string(stamp));
recording_folder = folder;
recording.open(folder / "inputs.csv", std::string("# " PSX_DIAGNOSTIC_INPUT_FORMAT "\n# execution ") +
PSX_EXECUTION_ID + "\n# plan " + plan_identity() + "\n# build " + build_identity() + "\n");
if (!recording) throw std::runtime_error(recording.error_text());
@@ -384,8 +389,11 @@ void capture_telemetry(const Row& input) {
row[2] = sequence ? std::chrono::duration_cast<std::chrono::microseconds>(now - telemetry_last).count() : 0;
telemetry_last = now;
const bool held = (input[ColButtons] & 6) == 0;
row[34] = -1; // No capture requested on this sample.
if (held && !incident_held) {
++incident;
const auto image = recording_folder / ("incident-" + std::to_string(incident) + ".png");
row[34] = psx_video_telemetry_capture(psx_playtest_diagnostic_utf8(image).c_str()) ? 1 : 0;
host_osd_push(("Playtest incident " + std::to_string(incident) + " marked").c_str(), 2000);
}
incident_held = held; row[3] = incident;
@@ -396,6 +404,21 @@ void capture_telemetry(const Row& input) {
RenderPassStats render{}; render_pass_get_stats(&render);
row[9] = render.passes; row[10] = render.aborted;
row[11] = render.verify_checks; row[12] = render.verify_mismatch;
uint64_t nop = 0, fill = 0, draw_commands = 0, environment = 0, copy = 0;
gpu_get_gp0_stats(&nop, &fill, &draw_commands, &environment, &copy);
PsxVideoTelemetry video{}; psx_video_telemetry_snapshot(&video);
PsxVideoConfigTelemetry config{}; psx_video_config_telemetry(&config);
row[13] = draw_commands; row[14] = video.presents; row[15] = video.present_errors;
row[16] = video.valid; row[17] = config.renderer;
row[18] = config.resolution_preset;
row[19] = config.aspect_num; row[20] = config.aspect_den; row[21] = config.widescreen;
row[22] = video.renderer; row[23] = video.internal_scale;
row[24] = video.source_width; row[25] = video.source_height;
row[26] = video.drawable_width; row[27] = video.drawable_height;
row[28] = video.viewport_width; row[29] = video.viewport_height;
row[30] = render_pass_verification_enabled(); row[31] = render.verify_checks != 0;
row[32] = 0; row[33] = 0; // No independent visual oracle is wired here.
row[35] = video.present_status_available;
telemetry.native(row);
if (!telemetry) {
telemetry_failed = true;
+2
View File
@@ -324,6 +324,8 @@ void render_pass_get_stats(RenderPassStats *out) {
out->dropped[i] = g_render_pass_dropped_writes[i];
}
int render_pass_verification_enabled(void) { return verify_on(); }
void render_pass_reset_session(void) {
s_netplay_allowed = 0;
memset(&s_stats, 0, sizeof s_stats);
+46
View File
@@ -0,0 +1,46 @@
#include "video_telemetry.h"
#include <atomic>
namespace {
std::atomic<uint64_t> presents{0}, errors{0};
std::atomic<int> status_available{0};
std::atomic<unsigned> generation{0};
std::atomic<int> fields[8]{};
}
extern "C" void psx_video_telemetry_compose(int renderer, int scale, int sw, int sh,
int dw, int dh, int vw, int vh) {
const int values[] = {renderer, scale, sw, sh, dw, dh, vw, vh};
// Single composing owner. Atomic fields avoid data races even when the
// bounded reader abandons an in-progress publication. This observer never
// waits for the renderer, queries GL, hashes pixels or changes guest state.
generation.fetch_add(1);
for (unsigned i = 0; i < 8; ++i) fields[i].store(values[i]);
generation.fetch_add(1);
}
extern "C" void psx_video_telemetry_present(int result) {
status_available.store(result >= 0, std::memory_order_relaxed);
(result != 0 ? presents : errors).fetch_add(1, std::memory_order_relaxed);
}
extern "C" void psx_video_telemetry_snapshot(PsxVideoTelemetry *out) {
if (!out) return;
*out = {};
out->presents = presents.load(std::memory_order_relaxed);
out->present_errors = errors.load(std::memory_order_relaxed);
out->present_status_available = status_available.load(std::memory_order_relaxed);
for (unsigned attempt = 0; attempt < 3; ++attempt) {
const unsigned before = generation.load();
if (!before || (before & 1u)) continue;
int values[8];
for (unsigned i = 0; i < 8; ++i) values[i] = fields[i].load();
if (before != generation.load()) continue;
out->valid = 1;
out->renderer = values[0]; out->internal_scale = values[1];
out->source_width = values[2]; out->source_height = values[3];
out->drawable_width = values[4]; out->drawable_height = values[5];
out->viewport_width = values[6]; out->viewport_height = values[7];
return;
}
}
@@ -13,7 +13,7 @@ foreach(_test writer boundary player)
target_include_directories(playtest_${_test}_test PRIVATE ../../include)
target_link_libraries(playtest_${_test}_test PRIVATE Threads::Threads)
if(NOT _test STREQUAL "writer")
target_sources(playtest_${_test}_test PRIVATE ../../src/psx_sha256_file.cpp ../../src/psx_sha256.c)
target_sources(playtest_${_test}_test PRIVATE ../../src/psx_sha256_file.cpp ../../src/psx_sha256.c ../../src/video_telemetry.cpp)
target_compile_definitions(playtest_${_test}_test PRIVATE PSX_SHA256_FILE_PORTABLE=1)
endif()
if(_test STREQUAL "player")
@@ -23,9 +23,17 @@ foreach(_test writer boundary player)
"${CMAKE_CURRENT_BINARY_DIR}/${_test}-evidence")
set_tests_properties(playtest_${_test}_test PROPERTIES TIMEOUT 20)
endforeach()
add_executable(video_telemetry_test ../test_video_telemetry.cpp ../../src/video_telemetry.cpp)
target_include_directories(video_telemetry_test PRIVATE ../../include)
target_compile_features(video_telemetry_test PRIVATE cxx_std_17)
target_link_libraries(video_telemetry_test PRIVATE Threads::Threads)
add_test(NAME video_telemetry_test COMMAND video_telemetry_test)
find_package(Python3 COMPONENTS Interpreter)
if(Python3_FOUND)
add_test(NAME replay_diagnostics_prepare_test
COMMAND "${Python3_EXECUTABLE}" -I
"${CMAKE_CURRENT_SOURCE_DIR}/../../../tools/tests/test_replay_diagnostics.py")
add_test(NAME playtest_telemetry_analysis_test
COMMAND "${Python3_EXECUTABLE}" -I
"${CMAKE_CURRENT_SOURCE_DIR}/../../../tools/tests/test_playtest_telemetry_analysis.py")
endif()
@@ -12,6 +12,8 @@
#include "sio.h"
#include "cdrom.h"
#include "render_pass.h"
#include "gpu.h"
#include "video_telemetry.h"
#include <array>
#include <cstdio>
#include <cstring>
@@ -40,6 +42,9 @@ static std::string plan_text="fixture-plan";
static void (*activation)(),(*vblank)(),(*savestate)(),(*osd_activation)(),(*osd_vblank)(),(*session_reset)();
static void (*recording_activation)();
static uint64_t ram_reads;
static uint64_t ordinary_draws;
static PsxVideoConfigTelemetry video_config{PSX_VIDEO_OPENGL, 1, 4, 3, 0};
static std::vector<std::string> incident_images;
static void activate_recording_plan() {session_reset();recording_activation();}
@@ -78,6 +83,13 @@ void cdrom_debug_snapshot(CDROMDebugState *out) {*out={};}
int cdrom_data_read_active(void) {return 0;}
int cdrom_xa_stream_active(void) {return 0;}
void render_pass_get_stats(RenderPassStats *out) {*out={};}
int render_pass_verification_enabled(void) {return 0;}
void gpu_get_gp0_stats(uint64_t *nop, uint64_t *fill, uint64_t *draw,
uint64_t *environment, uint64_t *copy) {
*nop = *fill = *environment = *copy = 0; *draw = ordinary_draws;
}
void psx_video_config_telemetry(PsxVideoConfigTelemetry *out) {*out=video_config;}
int psx_video_telemetry_capture(const char *path) {incident_images.emplace_back(path);return 1;}
void savestate_status_json(char *buf,size_t cap) {
snprintf(buf,cap,"\"generation\":%u,\"pending\":%u,\"last_ok\":%d,\"last_op\":\"%s\",\"last_slot\":%d",
status_generation,status_pending,status_ok,status_op,status_slot);
@@ -153,6 +165,13 @@ static void guest_frame(unsigned t) {
// VBlank of that frame to model a diverging guest.
static void run_frames(unsigned from,unsigned to,int tamper=-1) {
for(unsigned t=from;t<to;++t) {
ordinary_draws += 17;
if (t == 50) video_config = {PSX_VIDEO_OPENGL, 1080, 16, 9, 1};
psx_video_telemetry_compose(PSX_VIDEO_OPENGL, t < 50 ? 1 : 5,
t < 50 ? 320 : 426, 240,
t < 50 ? 640 : 1920, t < 50 ? 480 : 1080,
t < 50 ? 640 : 1920, t < 50 ? 480 : 1080);
psx_video_telemetry_present(1);
vblank();
if(osd_enabled)osd_vblank();
if((int)t==tamper)memory[0x80000]^=1;
@@ -229,6 +248,26 @@ int main(int argc,char **argv) {
auto timings=slurp(session/"telemetry.csv");
CHECK(timings.rfind("sequence,host_elapsed_us,host_interval_us,",0)==0);
CHECK(std::count(timings.begin(),timings.end(),'\n')==201);
CHECK(timings.find("gpu_draw_commands,host_presents,host_present_errors")!=std::string::npos);
CHECK(timings.find("visual_verify_available,visual_verify_checks")!=std::string::npos);
// Ordinary frames are observable with no optional render passes/checks.
// Runtime changes must be sampled, rather than copied from saved settings.
{std::istringstream csv(timings);std::string line;std::getline(csv,line);unsigned capture_requests=0;
for(unsigned n=0;n<200;++n) {
CHECK(bool(std::getline(csv,line)));std::replace(line.begin(),line.end(),',',' ');
std::istringstream values(line);psx_playtest_diagnostic::TelemetryRow row{};
for(auto& value:row)CHECK(bool(values>>value));
std::string extra;CHECK(!(values>>extra));
CHECK(row[9]==0 && row[11]==0 && row[12]==0);
CHECK(row[13]>0 && row[14]>0 && row[16]==1);
CHECK(row[18]==(n<50?1:1080) && row[19]==(n<50?4:16));
CHECK(row[23]==(n<50?1:5) && row[26]==(n<50?640:1920));
CHECK(row[30]==0 && row[31]==0 && row[32]==0 && row[33]==0);
CHECK(row[34]==-1 || row[34]==1);capture_requests+=row[34]==1;
CHECK(row[35]==1);
}
CHECK(capture_requests==1 && incident_images.size()==1);
CHECK(std::filesystem::u8path(incident_images[0])==(session/"incident-1.png"));}
CHECK(incident==1); // holding the chord across five frames marks once
// Row 0 is the initial SIO state; row 11 shows Cross, which the model's
// script first sampled at the boundary after frame 10... recorded one
@@ -239,6 +278,7 @@ int main(int argc,char **argv) {
CHECK(std::filesystem::exists(session/"card2.absent"));
CHECK(std::filesystem::exists(session/"mods-state.toml") && std::filesystem::exists(session/"settings.toml"));
CHECK(slurp(session/"session.json").find("\"netplay\": false")!=std::string::npos);
CHECK(slurp(session/"session.json").find("psxrecomp-telemetry-v2")!=std::string::npos);
CHECK(slurp(session/"session.json").find("baseline")==std::string::npos);
{auto location=slurp(std::filesystem::path(save_root)/"diagnostics-location.txt");
CHECK(location.find(psx_playtest_diagnostic_utf8(std::filesystem::absolute(session).make_preferred()))!=std::string::npos &&
+57
View File
@@ -0,0 +1,57 @@
#include "video_telemetry.h"
#include <atomic>
#include <cstdio>
#include <thread>
#define CHECK(value) do { if (!(value)) { std::fprintf(stderr, "line %d: %s\n", __LINE__, #value); return 1; } } while (0)
int main() {
PsxVideoTelemetry sample{};
psx_video_telemetry_snapshot(&sample);
CHECK(!sample.valid && !sample.presents && !sample.present_errors);
// A composition is not a swap. A failed present is not a completed one.
psx_video_telemetry_compose(PSX_VIDEO_SOFTWARE, 1, 320, 240, 640, 480, 640, 480);
psx_video_telemetry_snapshot(&sample);
CHECK(sample.valid && sample.renderer == PSX_VIDEO_SOFTWARE && !sample.presents);
psx_video_telemetry_present(0);
psx_video_telemetry_snapshot(&sample);
CHECK(sample.present_errors == 1 && sample.presents == 0);
CHECK(sample.present_status_available);
psx_video_telemetry_present(1);
psx_video_telemetry_snapshot(&sample);
CHECK(sample.presents == 1 && sample.present_errors == 1);
psx_video_telemetry_present(-1); // SDL2: the call returned, no status exists.
psx_video_telemetry_snapshot(&sample);
CHECK(!sample.present_status_available && sample.presents == 2);
// A live resize and scale change publish together. A busy writer permits
// an unavailable snapshot, never a fabricated/mixed coherent result.
std::atomic<bool> done{false};
std::thread producer([&] {
for (int scale = 1; scale <= 10000; ++scale) {
psx_video_telemetry_compose(PSX_VIDEO_OPENGL, scale,
320 * scale, 240 * scale,
640 * scale, 480 * scale,
640 * scale, 480 * scale);
psx_video_telemetry_present(1);
}
done.store(true);
});
bool consistent = true;
unsigned observations = 0;
do {
psx_video_telemetry_snapshot(&sample);
if (sample.valid && sample.renderer == PSX_VIDEO_OPENGL) {
consistent &= sample.source_width == sample.internal_scale * 320 &&
sample.drawable_width == sample.internal_scale * 640 &&
sample.viewport_height == sample.internal_scale * 480;
++observations;
}
} while (!done.load());
producer.join();
psx_video_telemetry_snapshot(&sample);
CHECK(consistent && observations > 0);
CHECK(sample.valid && sample.internal_scale == 10000 && sample.presents == 10002);
CHECK(sample.present_status_available);
CHECK(sample.present_errors == 1);
psx_video_telemetry_snapshot(nullptr);
return 0;
}
+112 -3
View File
@@ -22,6 +22,8 @@ does not verify build/mod identity or RAM. V1 routes require an explicit
An input-only completion is never an exact replay, and player builds reject it.
"""
import argparse
import csv
import io
import json
import os
from pathlib import Path
@@ -39,6 +41,107 @@ ENV_PREFIXES = ('PSX_', 'RNET_', 'LNG_', 'PSX_DIAGNOSTIC_', 'RECOMP_')
LOG_PREFIX = 'psxrecomp.diagnostics: '
def analyze_telemetry(recording):
"""Read passive evidence without launching a game or certifying pixels.
Old bundles remain readable, with missing coverage stated explicitly.
Counters may reset across state changes; report those resets separately.
"""
folder = recording if recording.is_dir() else recording.parent
metadata_path = folder / 'session.json'
metadata = json.loads(metadata_path.read_text(encoding='utf-8')) if metadata_path.exists() else {}
version = metadata.get('telemetry_format')
if version not in (None, 'psxrecomp-telemetry-v1', 'psxrecomp-telemetry-v2'):
raise ValueError('Unsupported telemetry format: ' + str(version))
raw = (folder / 'telemetry.csv').read_text(encoding='utf-8')
lines = raw.splitlines(keepends=True)
torn = bool(lines and not lines[-1].endswith('\n'))
if torn:
lines.pop()
reader = csv.DictReader(io.StringIO(''.join(lines)))
columns = set(reader.fieldnames or [])
required = {'sequence', 'host_elapsed_us', 'host_interval_us', 'render_verify_checks',
'render_verify_mismatch'}
if not required <= columns:
raise ValueError('Incomplete telemetry header')
rows = []
for row in reader:
if None in row or any(value is None for value in row.values()):
raise ValueError('Incomplete telemetry row')
rows.append({key: int(value) for key, value in row.items()})
if not rows:
raise ValueError('No complete telemetry samples')
def counter(name):
if name not in columns:
return {'available': False}
values = [row[name] for row in rows]
return {'available': True, 'first': values[0], 'last': values[-1],
'observed_increase': sum(max(b - a, 0) for a, b in zip(values, values[1:])),
'resets': sum(b < a for a, b in zip(values, values[1:]))}
warnings = []
draws, presents = counter('gpu_draw_commands'), counter('host_presents')
checks, mismatches = counter('render_verify_checks'), counter('render_verify_mismatch')
covered = checks['observed_increase'] > 0
if not draws['available'] or not presents['available']:
warnings.append('Ordinary drawing/presentation coverage was not recorded by this bundle.')
status_samples = [row['host_present_status_available'] for previous, row in zip(rows, rows[1:])
if row.get('host_presents') != previous.get('host_presents')
or row.get('host_present_errors') != previous.get('host_present_errors')] if 'host_present_status_available' in columns else []
present_status = all(status_samples) if 'host_present_status_available' in columns and status_samples else None
if not present_status:
warnings.append('Host present error status was not available for all samples; SDL2 only proves a present call returned.')
if not covered:
warnings.append('No render-pass state-preservation checks executed in the observed interval; zero mismatches is not a pass.')
warnings.append('Activity counters and state-preservation checks do not verify texture correctness; independent visual comparison is required.')
mode_fields = ('video_renderer', 'video_resolution_preset', 'video_aspect_num',
'video_aspect_den', 'video_widescreen')
modes = sorted({tuple(row[key] for key in mode_fields) for row in rows}) if set(mode_fields) <= columns else []
composed_fields = ('composed_renderer', 'composed_internal_scale', 'composed_source_width',
'composed_source_height', 'composed_drawable_width', 'composed_drawable_height',
'composed_viewport_width', 'composed_viewport_height')
composed = sorted({tuple(row[key] for key in composed_fields) for row in rows
if row.get('video_snapshot_valid')}) if set(composed_fields) <= columns else []
if not modes:
warnings.append('Effective runtime video configuration was not recorded; saved settings and mod state are requests, not proof of the live mode.')
gaps = sum(b['sequence'] != a['sequence'] + 1 for a, b in zip(rows, rows[1:]))
if gaps:
warnings.append('Telemetry sequence gaps or resets are present; do not treat the capture as contiguous.')
enabled = any(row.get('render_verify_enabled', 0) for row in rows) if 'render_verify_enabled' in columns else None
return {
'telemetry_format': version or 'inferred from header',
'samples': len(rows), 'sequence_discontinuities': gaps,
'ignored_torn_final_row': torn,
'host_elapsed_seconds': (rows[-1]['host_elapsed_us'] - rows[0]['host_elapsed_us']) / 1e6,
'max_vblank_interval_ms': max(row['host_interval_us'] for row in rows) / 1000,
'timing_scope': 'Host intervals between guest VBlanks; not unique game frames or presentation latency.',
'ordinary_draw_commands': draws, 'host_present_calls': presents,
'host_present_errors': counter('host_present_errors'),
'host_present_status_available_all_observed_calls': present_status,
'render_state_verification': {'enabled': enabled, 'covered_in_observed_interval': covered,
'checks': checks, 'mismatches': mismatches,
'scope': 'Optional extra render-pass state preservation, not normal rendering or pixels.'},
'visual_verification': {'available': any(row.get('visual_verify_available', 0) for row in rows),
'checks': counter('visual_verify_checks'), 'result': 'not evaluated'},
'effective_video_modes': [dict(zip(mode_fields, values)) for values in modes[:32]],
'effective_video_mode_count': len(modes),
'composed_video_modes': [dict(zip(composed_fields, values)) for values in composed[:32]],
'composed_video_mode_count': len(composed),
'composed_video_samples': sum(row.get('video_snapshot_valid', 0) != 0 for row in rows),
'incident_captures': {
'requests_recorded': 'incident_capture_request' in columns,
'accepted_requests': sum(row.get('incident_capture_request', -1) == 1 for row in rows),
'rejected_requests': sum(row.get('incident_capture_request', -1) == 0 for row in rows),
'files_present': sorted(path.name for path in folder.glob('incident-*.png')
if re.fullmatch(r'incident-\d+\.png', path.name) and path.is_file()),
'scope': 'Queued requests are not completed captures; files still need visual inspection.',
},
'composition_scope': 'Latest composed image may lead actual swaps with threaded presentation; zero dimensions mean unavailable.',
'warnings': warnings,
}
def native_executable(path):
executable = path.resolve(strict=True)
with executable.open('rb') as source:
@@ -215,17 +318,23 @@ def prepare(args):
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('recording', type=Path, help='Session directory or inputs.csv')
parser.add_argument('--exe', type=Path, required=True, help='Original native PSX playtest build')
parser.add_argument('--analyze', action='store_true', help='Summarize telemetry and coverage without launching a game')
parser.add_argument('--exe', type=Path, help='Original native PSX playtest build')
parser.add_argument('--bios', type=Path, help='BIOS image (default: bundled OpenBIOS)')
parser.add_argument('--disc', type=Path, required=True)
parser.add_argument('--disc', type=Path)
parser.add_argument('--game', type=Path)
parser.add_argument('--settings', type=Path, help='Settings used when recording')
parser.add_argument('--output', type=Path, required=True, help='New isolated directory')
parser.add_argument('--output', type=Path, help='New isolated replay directory')
parser.add_argument('--timeout', type=float, default=900)
parser.add_argument('--debug-port', type=int, help='Developer build TCP inspection port')
parser.add_argument('--input-only', action='store_true', help='Developer build only: apply inputs across fixes without claiming build/mod/RAM equivalence')
parser.add_argument('--legacy-connected', choices=('00','10','01','11'), help='Explicit P1/P2 presence assumption for an input-only v1 route')
args = parser.parse_args()
if args.analyze:
print(json.dumps(analyze_telemetry(args.recording), indent=2))
return
if not args.exe or not args.disc or not args.output:
parser.error('replay requires --exe, --disc and --output; --analyze only needs a session')
disc = args.disc.resolve(strict=True)
bios = args.bios.resolve(strict=True) if args.bios else None
output, target, trace = prepare(args)
@@ -0,0 +1,141 @@
"""Coverage claims must distinguish missing probes, inactive checks and activity."""
import csv
import importlib.util
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
SCRIPT = Path(__file__).resolve().parents[1] / 'replay_diagnostics.py'
spec = importlib.util.spec_from_file_location('replay', SCRIPT)
replay = importlib.util.module_from_spec(spec)
spec.loader.exec_module(replay)
class Coverage(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.folder = Path(self.temp.name)
def write(self, rows, version='v2'):
(self.folder / 'session.json').write_text(json.dumps({'telemetry_format': 'psxrecomp-telemetry-' + version}))
with (self.folder / 'telemetry.csv').open('w', newline='') as output:
writer = csv.DictWriter(output, fieldnames=rows[0].keys())
writer.writeheader()
writer.writerows(rows)
def samples(self, modern=False):
rows = [dict(sequence=i, host_elapsed_us=i*16667, host_interval_us=16667,
render_verify_checks=0, render_verify_mismatch=0) for i in range(3)]
if modern:
for i, row in enumerate(rows):
row.update(gpu_draw_commands=200*i, host_presents=i, host_present_errors=0,
host_present_status_available=1,
video_snapshot_valid=1, video_renderer=2,
video_resolution_preset=1 if i == 0 else 1080,
video_aspect_num=4 if i == 0 else 16,
video_aspect_den=3 if i == 0 else 9, video_widescreen=int(i > 0),
render_verify_enabled=0, visual_verify_available=0, visual_verify_checks=0)
row.update(composed_renderer=2, composed_internal_scale=1 if i == 0 else 5,
composed_source_width=320 if i == 0 else 426, composed_source_height=240,
composed_drawable_width=640 if i == 0 else 1920,
composed_drawable_height=480 if i == 0 else 1080,
composed_viewport_width=640 if i == 0 else 1920,
composed_viewport_height=480 if i == 0 else 1080)
return rows
def test_legacy_zero_checks_are_unknown_coverage_not_success(self):
self.write(self.samples(), 'v1')
result = replay.analyze_telemetry(self.folder)
self.assertFalse(result['ordinary_draw_commands']['available'])
self.assertFalse(result['render_state_verification']['covered_in_observed_interval'])
self.assertIsNone(result['render_state_verification']['enabled'])
self.assertFalse(result['visual_verification']['available'])
self.assertTrue(any('not recorded' in text for text in result['warnings']))
def test_normal_rendering_can_advance_with_no_extra_render_passes(self):
self.write(self.samples(True))
result = replay.analyze_telemetry(self.folder)
self.assertEqual(result['ordinary_draw_commands']['observed_increase'], 400)
self.assertEqual(result['host_present_calls']['observed_increase'], 2)
self.assertFalse(result['render_state_verification']['covered_in_observed_interval'])
self.assertEqual(result['visual_verification']['result'], 'not evaluated')
self.assertEqual(result['effective_video_mode_count'], 2)
self.assertEqual(result['effective_video_modes'][1]['video_resolution_preset'], 1080)
self.assertEqual(result['composed_video_modes'][1]['composed_internal_scale'], 5)
def test_enabled_but_never_exercised_and_preexisting_checks_are_uncovered(self):
rows = self.samples(True)
for row in rows:
row['render_verify_enabled'] = 1
row['render_verify_checks'] = 9
self.write(rows)
result = replay.analyze_telemetry(self.folder)['render_state_verification']
self.assertTrue(result['enabled'])
self.assertFalse(result['covered_in_observed_interval'])
def test_checks_do_not_certify_pixels_and_counter_resets_remain_visible(self):
rows = self.samples(True)
for row, checks in zip(rows, (0, 3, 1)):
row['render_verify_checks'] = checks
row['render_verify_enabled'] = 1
rows[-1]['host_present_errors'] = 1
self.write(rows)
result = replay.analyze_telemetry(self.folder)
self.assertTrue(result['render_state_verification']['covered_in_observed_interval'])
self.assertEqual(result['render_state_verification']['checks']['resets'], 1)
self.assertEqual(result['host_present_errors']['observed_increase'], 1)
self.assertFalse(result['visual_verification']['available'])
def test_torn_final_record_is_ignored_and_internal_malformed_rows_fail(self):
self.write(self.samples(True))
path = self.folder / 'telemetry.csv'
with path.open('a') as output:
output.write('3,partial')
self.assertTrue(replay.analyze_telemetry(self.folder)['ignored_torn_final_row'])
with path.open('a') as output:
output.write('\n')
with self.assertRaises(ValueError):
replay.analyze_telemetry(self.folder)
def test_analysis_cli_needs_no_build_disc_or_replay_output(self):
self.write(self.samples(True))
completed = subprocess.run([sys.executable, '-I', str(SCRIPT), str(self.folder), '--analyze'],
check=True, capture_output=True, text=True)
self.assertEqual(json.loads(completed.stdout)['samples'], 3)
self.assertFalse((self.folder / 'replay').exists())
def test_incident_request_acceptance_does_not_invent_a_written_image(self):
rows = self.samples(True)
for row, state in zip(rows, (-1, 1, 0)):
row['incident_capture_request'] = state
self.write(rows)
result = replay.analyze_telemetry(self.folder)['incident_captures']
self.assertEqual(result['accepted_requests'], 1)
self.assertEqual(result['rejected_requests'], 1)
self.assertEqual(result['files_present'], [])
(self.folder / 'incident-1.png').write_bytes(b'fixture')
self.assertEqual(replay.analyze_telemetry(self.folder)['incident_captures']['files_present'], ['incident-1.png'])
def test_sdl2_void_present_is_activity_with_unknown_failure_coverage(self):
rows = self.samples(True)
for row in rows:
row['host_present_status_available'] = 0
self.write(rows)
result = replay.analyze_telemetry(self.folder)
self.assertEqual(result['host_present_calls']['observed_increase'], 2)
self.assertFalse(result['host_present_status_available_all_observed_calls'])
self.assertTrue(any('SDL2' in warning for warning in result['warnings']))
def test_startup_before_first_present_is_not_a_missing_result_for_a_call(self):
rows = self.samples(True)
rows[0]['host_present_status_available'] = 0
self.write(rows)
self.assertTrue(replay.analyze_telemetry(self.folder)['host_present_status_available_all_observed_calls'])
if __name__ == '__main__':
unittest.main()