psxrecomp/docs/FRAME_GENERATION.md

Smooth motion (frame generation, [video] frame_generation)

Opt-in, OpenGL with the render thread only, off by default (PSX_FRAME_GEN=0/1 overrides for one run). Off, nothing in this document runs and the renderer is the render thread of docs/RENDER_THREAD.md.

[video]
render_thread = true
frame_generation = true
# How in-between frames are made: "redraw" (default) or "reprojection".
frame_generation_method = "redraw"

frame_generation_method (PSX_FRAME_GEN_METHOD=redraw|reprojection overrides it for one run):

• redraw (the default for every title): each in-between frame draws the recorded list again from an in-between camera (How, below).
• reprojection (opt-in per title, in its game.toml): each in-between frame warps the newer real frame's finished image (Reprojection, below). It is cheaper (about a millisecond of GPU per frame) but its object rules were tuned on R4 and are assumptions about how a game draws; on other titles it has shown flicker (THPS2), so no title gets it unless it opts in.

With Smooth motion off, or in redraw, nothing of reprojection runs or is allocated: no snapshots, depth image, warp programs, HUD matching, per-frame getenv or uniform lookups. Switching to redraw (or Smooth motion off) frees what reprojection held.

Rule

Real frames at the guest's rate come first and resolution is the only quality lever for them (dynamic resolution). Frames above that are a bonus, drawn by the render thread from recorded data only: no guest code runs again, nothing is snapshotted or rolled back, and only when the render thread has measured time to spare.

How

• Lists. The render thread keeps a copy of every replayed state and drawing record (not transfers, copies, presents, steps). Display buffers are learned from presents; a list is one game frame's drawing into one buffer and closes when drawing moves to another buffer, or when the game flips to the buffer being drawn. (R4 starts drawing the next frame a VBlank before it flips, so a flip alone is not a frame boundary.)

• Cadence. A flip (a present of a different display address) is a new game frame; a present of the same buffer is the same frame again. R4 updates every other VBlank, so generation interpolates between flips (30 Hz game frames), not between VBlanks.

• Sources. While generation is on, every RTPS/RTPT output is noted (gte.cpp, gte_fg_source_*) with an identity (the issuing function ra and the model-space vertex), the camera-space position the GTE divided and H, indexed by the packed screen word. The same point projected again in a frame (a corner shared by separate meshes, another function) is one vertex (the smaller identity); up to two different points on one pixel are kept, and gpu.c picks the one projected by the same function as the triangle's other vertices, else the one nearest their depth. gpu.c looks each polygon vertex up by its packet word and records the sources ahead of the triangle (RTH_FG_SRC, never drawn). CPU-built vertices have none. Guest-visible GTE results are untouched.

• Camera (frame_gen.c, fg_cam_fit). No primitive is paired by draw order. Per view (draw area), vertices of the two frames with the same identity are paired in camera space (instances: the clearly nearest) and a rigid motion is fitted (RANSAC + least squares, inliers within 1 screen px at their depth). The static world agrees with it; paired vertices that disagree are objects (cars, the followed car). Verdict: no projections, a view with few pairs or under half inliers, or a turn/travel no camera makes between two frames rejects the generated frame (the real frame shows).

• Drawing (fg_cam_place). One real frame is redrawn from an in-between camera: world vertices re-projected through the fraction of the fitted motion, object vertices lerped in camera space, CPU-built vertices of placed triangles moved with their neighbours, everything else (HUD, 2D, sprites, the mirror's own view if unprojected) unchanged. Moving forward the older frame is redrawn (seen from further on it spreads past the screen edges); otherwise the newer. Drawing starts on the newer real frame's image and skips the clears before the first draw, and strips along a view's edges the picture moved away from are copied from that image, so what the in-between camera uncovers shows the newer frame instead of a hole. Every placed vertex at one screen position moves alike (a weld: the same corner reaches the GPU from several projections), and a vertex without a projection lying on an edge between placed vertices moves as that point of the edge (closes the T-junctions the game's polygon splits leave). A game frame whose in-between camera would pass through geometry (a vertex behind it: tunnel ceilings, overhead bridges) is not generated: without near-plane clipping its image would be wrong. The raw texture mirror is not packed from the generated surfaces; all bookkeeping the draws touch is restored, so the real stream is unaffected.

• Schedule. With n in-between frames per game frame, at the flip the real frame is composed and kept (its buffer may be drawn over before it is shown), frames t = k/(n+1) are presented flip/(n+1) apart, then the kept real frame. Timed work runs from the render-thread core's tick hook between records and while idle.

• Latency. Interpolation shows a real frame after the in-between frames that lead to it: up to n/(n+1) of a game frame later than without generation (R4 at 30 Hz on 120 Hz: 25 ms at n = 3, 17 ms at n = 1). Extrapolation (no delay, guessed motion) is not offered.

• Plan. n = fg_plan(...): slots = fg_slots, the display intervals the clock actually fills (in-between frames at phase k / (flip × refresh) below 0.985, plus the real one: R4 on 120 Hz: 4; 59.94 Hz content on a 60 Hz panel, 2.002 intervals: 2, one in-between frame), limited to what fits in 85 % of the interval after the real frames' render-thread CPU time and one swap, at the measured cost of a generated frame (CPU, GPU via GL_TIME_ELAPSED, plus a swap). On macOS a real frame's TIME_ELAPSED span reads close to the whole interval (it counts the GPU waiting for records), so it is not used here; GPU overload shows as backpressure. The real frames' measured cost excludes generation (its query is paused, segments summed); dynamic resolution adds the generated frames actually drawn, each at the mean measured cost of the timed ones, to its load. So it steps up while the render thread has room for both, and the in-between frames count against the budget once they crowd the real ones (no reserve assumed for frames not drawn). With the present thread ([video] present_thread, docs/RENDER_THREAD.md) a swap here is the hand-off to it, and its cost is the time spent waiting for a free slot, not the compositor's round trip.

• Breaker. Off for 3 s (doubling to 24 s on repeats within 10 s) after the queue backed up (the guest waited), the render thread fell two frames behind, or a guest frame took over 1.5 intervals. While dynamic resolution is over budget on the real frames alone (the last window's load less the generated frames' part) or stepping down, generation pauses (no escalation). One frame behind just shows the waiting real frame at once.

• Sync points show the waiting real frame and restart the lists at the next flip.

• Variable refresh. [video] vsync = "vrr" (game.toml, or the player's settings.toml, which a launcher save writes back as "vrr"): for G-Sync / FreeSync / ProMotion panels. Each frame is presented when it is ready (swap interval 0); Smooth motion targets the panel's maximum refresh at the current size and never exceeds it. The other modes are "on" (default), "immediate" ("off") and "adaptive". PSX_HOST_REFRESH_HZ / PSX_HOST_REFRESH_MAX_HZ simulate a display in headless tests.

Reprojection (frame_generation_method = "reprojection")

• Source. At each flip the newer list's finished image is copied (rp_snapshot_list): the game draws on into its buffers while the in-between frames are shown. The camera is fitted for the newer frame (fg_cam_fit, a view that fails leaves its vertices unchanged; a third of the pairs is enough, since whatever moves on its own is redrawn).
• Depth image (once per game frame, 2 texels per native pixel): the newer list's recorded triangles in painter order, each pixel one of world (every corner placed by the camera: its 1/z and view), object, keep, or HUD needle.
• Warp (per in-between frame, one draw): a grid with a vertex per native pixel; a world vertex is unprojected with its depth, moved by its view's in-between camera (fg_view_affine) and projected again (nearest wins); cells over a depth edge, a view boundary or a non-world pixel are not drawn, so the snapshot underneath shows there (disocclusion is filled with the newer frame). Keep pixels (2D, HUD, the mirror) are then put back from the snapshot; needle pixels from the older snapshot. The warp's depth is cleared before anything else is drawn (with PGXP depth the cars would otherwise test LEQUAL against it).
• Objects. Cars are drawn again from the recorded list at their own in-between position; HUD needles are moved as fg_hud_lerp moves them.
• Breaker. Each method has its own breaker settings, set again when the method changes: reprojection holds 0.1 s (at most 1 s; redraw 0.5 s to 8 s). Dynamic resolution over budget does not pause reprojection (its frames cost little; a step down still pauses it 0.05 s), and a late guest frame trips it only when its in-between frames cost over 3 ms of CPU. The generated-frame cost estimate moves at most 4x per sample.
• Dynamic resolution budgets the real frames as their cost against their share of the frame (redraw: the share of the period).

Object rules (R4-tuned assumptions)

These rules decide what the warp moves and what is drawn again. They were tuned on R4; a title that opts in gets them as they are:

• Cars are triangles the fit calls objects (FG_PLACE_OBJECT) whose corners really moved against the world (more than 4 % of their depth from where the camera alone puts them; near road subdivided past the fit's tolerance is world), textured (an untextured gradient is backdrop), and small (at most 96 x 64 native px). Anything else the fit moves on its own (a sky dome) is backdrop the warp keeps where it is.
• Backdrop. A view's triangles drawn before its first world triangle are backdrop and are not drawn again.
• HUD needles are 2D triangles at most 48 px on a side that moved at most 24 px per corner between the two frames and drew the same thing (fg_hud_match); they are drawn at their in-between position over the older frame's dial.
• HUD protection. 2D pieces up to 160 x 120 px (rects and triangles) of the newer and the older list are keep pixels: the newer frame's pixels are shown there, never warped. Big 2D (a backdrop drawn first) is not.
• Small views (a view whose area is under 15 % of the biggest, R4's mirror) are keep: shown as the newer frame drew them.
• Split screen. Each view is fitted on its own; one view's failed fit leaves the others reprojected.

Diagnostics (read once when the programs are built): PSX_RP_DEBUG=1 shows the warp grid, =2 the pixel kinds; PSX_RP_NOPREV=1 draws needles over the newer frame's dial. frame_gen reports reproject, reprojected, reproject_fallback, rp_ms and rp_zn.

Limits

• Only GTE-projected triangles move; sprites, lines and fills are the redrawn frame's. Areas uncovered by the in-between camera show the newer frame. Vertices without a projection that are neither on a placed edge nor shared take their triangle's mean motion (guessed counts those over a 1 px spread; 0 in R4 races once shared corners resolve).
• A texture the frame drew earlier into its own displayed buffer is sampled as the raw mirror holds it when the in-between frame is drawn.
• Single-buffered games never flip, so nothing is generated.
• Windowed high-resolution mode and depth24 frames are not generated.
• HD texture replacements and dumping: generated frames draw with the same replacements as real frames (HUD/2D included); they are presentation only, so native VRAM and dumps see only real frames (generated draws suppress dump queries).

Debug

{"cmd":"frame_gen"} (not a sync point): generated / real presents, flips, dups, plan (last_n, slots), costs (real_ms, gen_ms, swap_ms), last camera fit (views, pairs, inliers, place_*, cam_angle_deg, cam_shift), verdict (verdict_ok, rejected, reject_why), breaker, ceiling, last hold, total swaps.

PSX_PRESENT_SHOT_GENERATED=1 (diagnostic): a staged present_shot is taken at the next generated frame instead of the next present, to inspect in-between images in game. PSX_PRESENT_SHOT_BURST=N: a staged present_shot takes the next N composed frames (written after the burst as <path>_NN_{r|g}_f<flip>.png, in composition order: a real frame precedes the in-between frames that lead up to it).

Tests

• frame_gen_test: the camera fit on a synthetic grandstand of one repeated texture, redrawn in another order with a row culled and one added (the case the old draw-order matcher paired wrongly: 225 of 226 pairs) - phase 0 / 0.5 / 1 land where the older / halfway / newer camera sees each vertex; a followed car, a HUD triangle, a CPU-built vertex; the verdict (no projections, a cut, an impossible turn, inconsistent motion); the plan, breaker, cost, pace, ceiling.
• render_thread_test: the tick hook runs on the render thread with the context, never early, and wakes an idle thread.
• gl_frame_gen_test (real GL, gpu;opengl;hardware): a double-buffered 30 Hz scene, VRAM and native-wide presents, flipping right after drawing and a VBlank late: the distinct real presented images are identical with generation off and on, generated frames are presented, and an in-between frame composed at t = 1 / t = 0 equals the newer / older real frame. In redraw (the default) the fixture checks that nothing of reprojection ran or was allocated and prints the phase 0.5 image's digest (p05=), which matches master's. With FG_METHOD=reprojection the warp's pixels are checked: with a parallax pan (a projected backdrop twice as far as the triangles) phase 0.5 matches the redraw of the same pair (under 5 % of the pixels differ, under half of what the unwarped newer frame differs), every pixel is a real frame's or the redraw's (no holes), the HUD is exactly the newer frame's, the real frames are those of generation off, and with PGXP depth the warp leaves no depth behind for the cars.