diff --git a/docs/PLAYTEST_DIAGNOSTICS.md b/docs/PLAYTEST_DIAGNOSTICS.md index 2b3b5c4c..dd3649ac 100644 --- a/docs/PLAYTEST_DIAGNOSTICS.md +++ b/docs/PLAYTEST_DIAGNOSTICS.md @@ -44,9 +44,12 @@ default them on. - The source manifest carries the production default: `default_enabled = false` on both the `recording` and `osd` features. -- `PSX_PUBLIC_RELEASE` (CMake option, default `OFF`) is the one build-level - notion of a public release. Release packagers configure with - `-DPSX_PUBLIC_RELEASE=ON`; nothing else should. +- `PSX_PUBLIC_RELEASE` is the one build-level notion of a public release. It + is derived from `PSX_RELEASE_CHANNEL` (`public` => ON; `playtest` or unset + => OFF; `runtime/psx_release_channel.cmake`). Releases are cut only by + `tools/package_title_release.py --channel public|playtest` + ([RELEASING_A_TITLE.md](RELEASING_A_TITLE.md)); `-DPSX_PUBLIC_RELEASE=ON` + without a channel fails configure. - After the framework stages the catalog into `/mods/bundled`, `runtime/psx_playtest_diagnostics_defaults.cmake` edits only the staged copy: with `PSX_PUBLIC_RELEASE=OFF` it sets both features to `default_enabled = diff --git a/docs/RELEASING_A_TITLE.md b/docs/RELEASING_A_TITLE.md new file mode 100644 index 00000000..90df4d8e --- /dev/null +++ b/docs/RELEASING_A_TITLE.md @@ -0,0 +1,197 @@ +# Releasing a title + +One framework command cuts every player release, for every title, on Windows +and Linux. A title keeps only its release config, `packaging/release.toml`. +Release fixes land here once instead of in a forked packager per title. + +## Commands + +From the title repository root, with submodules checked out at their pins: + +```powershell +# Windows host: Windows ZIP, Linux AppImage (through WSL), or both +psxrecomp/tools/package_title_release.ps1 --channel public --platform all +psxrecomp/tools/package_title_release.ps1 --channel playtest --platform all +``` + +```sh +# Native Linux host: the AppImage +psxrecomp/tools/package_title_release.sh --channel public +psxrecomp/tools/package_title_release.sh --channel playtest +``` + +`--channel` is required and has no default. Without it the command fails at +once with `PACKAGING FAILED`. Artifacts land in `dist/`: + +| Channel | Diagnostics default | Windows | Linux | +|---|---|---|---| +| `public` | off | `-v-windows-x64.zip` | `-v-linux-x86_64.AppImage` | +| `playtest` | on | `-v-playtest-windows-x64.zip` | `-v-playtest-linux-x86_64.AppImage` | + +Each artifact has a `.sha256` file beside it. `RELEASE_MANIFEST.json` inside +the package records the channel, version, title/framework/recomp-ui commits, +mod list, imports or glibc, and a hash for every file. The command never tags, +pushes or uploads. + +Other options: `--jobs N` (default `min(6, cores-2)`), `--out-dir`, +`--work-dir` (default `build-package/`), `--toolchain-bin` +(Windows MinGW, default `C:\msys64\mingw64\bin`), and `--allow-dirty`. +Without `--allow-dirty`, uncommitted tracked changes, including a submodule +moved off its pin, are refused. With it they are recorded in the manifest, so +that package must not be published. + +`verify --channel C ARTIFACT...` extracts finished artifacts and re-runs every +check. The command already does this after each package. + +## What the command does + +Any failure ends with `PACKAGING FAILED: ` and a non-zero exit code. +No step warns and then continues. + +1. **Provenance.** Reads the title, framework and recomp-ui commits and checks + for tracked changes. +2. **Emitters.** Builds `psxrecomp-game` and `psxrecomp-bios` before any slow + step. The overlay toolchain needs both. +3. **Sources.** Regenerates the recompiled OpenBIOS if it is missing, then + regenerates the game C from the disc. For AOT mode `static`, it also builds + the static AOT units from the original disc. On Windows, `--platform linux` + or `all` prepares the sources once there. The WSL build then compiles those + same audited sources. +4. **Build.** Release, `PSX_DEBUG_TOOLS=OFF`, OpenBIOS-only backend + (`PSXRECOMP_BIOS_STEMS=OpenBIOS`), `PSX_RELEASE_CHANNEL=`, and + `PSX_GAME_VERSION` from the title's version file. Every flag the command + owns is read back from `CMakeCache.txt`. A title's `cmake_args` may not set + any of them. +5. **Stage.** Stages these items: + - the executable and its version stamp + - launcher assets + - OpenBIOS and its MIT notice + - `game.toml` and `game_options.toml` + - the mod catalog, checked against the build's own catalog list. + Developer-channel features are removed from public packages. + - for AOT mode `cache`, the original-disc AOT cache with + `AOT_CACHE_AUDIT.json` + - the overlay toolchain, if the title ships it + - framework, recomp-ui and third-party notices + - the title's files + - `START_HERE.txt` with `@VERSION@` filled in. A template without the + token is an error. + - the execution contract +6. **Gates.** Refuses the package if it contains any of these: disc images, + retail BIOS images, `.bin` files other than `bios/openbios.bin`, saves, + memory cards, save states, personal settings or mod state, generated or + other C sources outside `overlay_toolchain/`, or build-kit trees. It also + checks that the Playtest Diagnostics defaults match the channel, and runs + `tools/ci/audit_playtest_diagnostics_package.py --mode `. On + Windows the executable must import only system DLLs. For AOT mode `static`, + every audited unit must match its receipt and be linked into the executable. + A diagnostics package that the config does not declare is an error. +7. **Package and verify.** Writes a Windows ZIP with portable `/` names, or + builds a Linux AppImage with `tools/package_appimage.sh`. The AppImage uses + fixed linuxdeploy `1-alpha-20251107-1` and appimagetool `1.9.1`, checked by + SHA-256. The command writes the `.sha256` file, then extracts the artifact + and verifies it. + +Linux needs `cmake`, `ninja` (or make), `gcc`, Python 3.11+, ImageMagick, +`curl`, `file` and the SDL3/OpenGL development packages. On a Windows host, +the Linux half runs inside the default WSL distribution, using the same +checkout through `/mnt/`. The build tree stays on that drive. Only the +payload, AppDir and verification extraction use `/tmp`, and they are removed +afterwards. + +## The backstop: no package without a channel + +`runtime/psx_release_channel.cmake` makes `PSX_RELEASE_CHANNEL` +(`public` | `playtest`) the only release input. `PSX_PUBLIC_RELEASE` is now +derived from it. Configure fails in these cases: + +- a channel other than `public` or `playtest` +- `-DPSX_PUBLIC_RELEASE=ON` without a channel (the old way to ask for a public + build) +- a `PSX_PUBLIC_RELEASE` value that contradicts the channel + +A tree configured with no channel is a dev build. The packager configures its +own tree with the channel, then reads it back before staging. + +## `packaging/release.toml` + +Unknown keys are errors, so a typo cannot silently drop a step. + +```toml +schema = 1 + +[product] +name = "JerseyDevilRecomp" # artifact name prefix (and AppImage data dir token) +display_name = "Jersey Devil Recompiled" +exe_name = "JerseyDevil_Recompiled" # staged executable; default = CMake OUTPUT_NAME +version_file = "packaging/release/VERSION" # raw text = PSX_GAME_VERSION and @VERSION@ +# game_toml = "game.toml" # codegen config (default) +# player_game_toml = "game.toml" # shipped config (default = game_toml) +# runtime_target = "psx-runtime" +# zip_folder = "MediEvilIIRecomp" # optional single top-level ZIP folder +launcher_boxart = "launcher_assets/img/boxart.tga" # staged copy must match + +[build] +cmake_args = ["-DPSX_SDL_BACKEND=SDL3", "-DPSX_PGXP_VARIANT=OFF"] +# windows_cmake_args / linux_cmake_args: per-platform extras +# regenerate = true # psxrecomp-game --config game.toml first +# binary_identity = "{version}+{commit12}" # bytes the executable must contain + +[aot] +mode = "cache" # cache: ship the original-disc AOT cache + # static: compile audited units into the exe + # none +profile = "aot/overlays.json" +# cps = true # default: true for static, false for cache + +[overlay_toolchain] +ship = true # bundle the runtime overlay compiler + +[diagnostics] # required if the catalog has a *.diagnostics package +game_id = "SCUS-94907" +package_id = "jerseydevil.diagnostics" +# replay_helper = false # title records with its own tool (no replay-diagnostics.py) + +[payload] +start_here = "packaging/release/START_HERE.txt" # must contain @VERSION@ +files = ["README.md", { from = "packaging/release/input.ini", to = "input.ini" }] +# build_files = ["..."] # extra files from beside the built executable + +[linux] +app_run = "packaging/linux/AppRun" # @VERSION@ @APP_NAME@ @EXE_NAME@ @PAYLOAD_DIR@ + # @ENV_PREFIX@ @ARTIFACT_NAME@ are filled in +desktop_file = "packaging/linux/io.github.mstan.JerseyDevilRecomp.desktop" +icon = "launcher_assets/img/boxart.tga" # PNG, or converted with ImageMagick +payload_dir = "jerseydevilrecomp" # usr/share/ +env_prefix = "JERSEYDEVIL_RECOMP" +# prune_unreferenced_libs = true # drop usr/lib entries the exe does not load + +# Optional title steps, run at prepare | post_build | post_stage. Tokens: +# {python} {root} {framework} {channel} {platform} {version} {build_dir} +# {work_dir}, plus {exe} (post_build) and {stage} (post_stage). A non-zero +# exit fails the release. +# [[hooks.post_stage]] +# command = ["{python}", "tools/check_my_payload.py", "{stage}"] +# platforms = ["windows", "linux"] +``` + +`aot.mode = "static"` checks `generated/AOT_STATIC_AUDIT.json` in these ways: +every pair is valid, the recipe count matches `expected_records`, the +`game.toml` and profile hashes match, each unit's hash matches its receipt, +and each unit appears in the executable's link line. + +## Migrating a title + +Rules for every migrated title: + +- Delete the forked packagers (`tools/package_release.ps1`, + `tools/package_appimage.sh`, `tools/package_release.py`, prepare scripts, + `scripts/package_setup_release.sh`) and their tests. +- Write `packaging/release.toml`. +- Put `@VERSION@` in `START_HERE.txt`. +- Remove any title CMake that forces `PSX_PUBLIC_RELEASE`. +- Pin the latest psxrecomp and recomp-ui. + +Prove the migration with both channels on both platforms, then verify the +artifacts. Titles migrated so far: Jersey Devil and MediEvil II. The backlog +is tracked in bead `beads-eio.3.369`. diff --git a/docs/ci/BUNDLED_RELEASES.md b/docs/ci/BUNDLED_RELEASES.md index 2ce0b22e..706d3742 100644 --- a/docs/ci/BUNDLED_RELEASES.md +++ b/docs/ci/BUNDLED_RELEASES.md @@ -1,5 +1,10 @@ # Bundled releases +> Local title releases (Windows ZIP + Linux AppImage, public or playtest) are +> cut with one framework command, `tools/package_title_release.py`, from a +> title's `packaging/release.toml`: see [../RELEASING_A_TITLE.md](../RELEASING_A_TITLE.md). +> This page describes the CI template built on `package_game_release.sh`. + Public CI builds and ships the **compiled game**. The title commits its recompiled game C under `generated/`; CI links it, packages the executable with its runtime data, and attaches one zip per platform. Players download, point diff --git a/docs/ci/templates/game-release.yml b/docs/ci/templates/game-release.yml index 16154c10..a6f81a50 100644 --- a/docs/ci/templates/game-release.yml +++ b/docs/ci/templates/game-release.yml @@ -65,6 +65,13 @@ on: required: true type: boolean default: true + channel: + description: 'Release channel: public (diagnostics off) or playtest (diagnostics on, -playtest names)' + required: true + type: choice + options: + - public + - playtest reuse_cached_emitters: description: 'Skip emitter rebuild when build-recompiler cache hits (psxrecomp SHA)' required: false @@ -502,6 +509,7 @@ jobs: -DPSXRECOMP_ALLOW_NO_BIOS=OFF \ -DPSX_SETUP_WIZARD=ON \ -DPSX_GAME_VERSION="${RELEASE_VERSION}" \ + -DPSX_RELEASE_CHANNEL="${{ github.event_name == 'push' && 'public' || github.event.inputs.channel }}" \ "${EXTRA[@]}" 2>&1 | tee build-ci-configure.log # Assert the configure took the FULL path, not the setup-host one. if ! grep -q 'linking generated game C (full runtime)' build-ci-configure.log; then diff --git a/recompiler/CMakeLists.txt b/recompiler/CMakeLists.txt index 806513f2..41abcd0b 100644 --- a/recompiler/CMakeLists.txt +++ b/recompiler/CMakeLists.txt @@ -1214,6 +1214,7 @@ if(BUILD_TESTING) aligned_lzss_banks bpe_container audit_playtest_diagnostics_package + package_title_release bundled_release_layout collect_game_misses cursor_writers @@ -1244,6 +1245,14 @@ if(BUILD_TESTING) ${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_playtest_diagnostics_defaults.py --cmake ${CMAKE_COMMAND} WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) + # Configures throwaway projects against runtime/psx_release_channel.cmake: + # a release-package build must name its channel. + add_test( + NAME release_channel_cmake + COMMAND ${Python3_EXECUTABLE} + ${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_release_channel_cmake.py + --cmake ${CMAKE_COMMAND} + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) add_test( NAME compile_overlays_static_split COMMAND ${Python3_EXECUTABLE} diff --git a/runtime/psx_release_channel.cmake b/runtime/psx_release_channel.cmake new file mode 100644 index 00000000..d0968bbc --- /dev/null +++ b/runtime/psx_release_channel.cmake @@ -0,0 +1,83 @@ +include_guard(GLOBAL) + +# Release channel: the one input that says what a release-package build is. +# +# PSX_RELEASE_CHANNEL=public public production release +# PSX_RELEASE_CHANNEL=playtest non-public playtest / pre-release package +# unset (empty) a dev or test build; never packaged +# +# tools/package_title_release.py is the only supported way to cut a release. +# It requires --channel public|playtest (no default), configures with +# -DPSX_RELEASE_CHANNEL=, and refuses to stage a build tree whose +# channel is empty or different. PSX_PUBLIC_RELEASE is DERIVED from the +# channel here (public => ON); it is no longer an input. Before this module, +# a packaging configure that forgot -DPSX_PUBLIC_RELEASE=ON silently produced +# a playtest build (diagnostics on) under a public name. +# +# Backstop rules (FATAL_ERROR): +# * a channel other than public/playtest; +# * -DPSX_PUBLIC_RELEASE=ON without a channel -- the pre-channel way to ask +# for a public package; name the channel instead; +# * a PSX_PUBLIC_RELEASE value that contradicts the channel. +# An explicit or cached PSX_PUBLIC_RELEASE=OFF without a channel is the old +# option's default (every dev tree configured before this module has it) and +# stays a dev build. +set(PSX_RELEASE_CHANNEL "" CACHE STRING + "Release package channel: public or playtest (empty = dev build, not packageable)") +set_property(CACHE PSX_RELEASE_CHANNEL PROPERTY STRINGS ";public;playtest") +if(NOT PSX_RELEASE_CHANNEL STREQUAL "" AND + NOT PSX_RELEASE_CHANNEL MATCHES "^(public|playtest)$") + message(FATAL_ERROR + "PSX_RELEASE_CHANNEL must be 'public' or 'playtest' (got '${PSX_RELEASE_CHANNEL}'). " + "Cut releases with psxrecomp/tools/package_title_release.py --channel public|playtest.") +endif() + +if(DEFINED CACHE{PSX_PUBLIC_RELEASE}) + get_property(_psx_public_type CACHE PSX_PUBLIC_RELEASE PROPERTY TYPE) + if(NOT _psx_public_type STREQUAL "INTERNAL") + # Set by the caller (-D) or left by a configure that predates channels. + if(PSX_PUBLIC_RELEASE AND PSX_RELEASE_CHANNEL STREQUAL "") + message(FATAL_ERROR + "PSX_PUBLIC_RELEASE=ON without a release channel. A packaging build must " + "name its channel: -DPSX_RELEASE_CHANNEL=public|playtest (PSX_PUBLIC_RELEASE " + "is derived from it). Releases are cut with " + "psxrecomp/tools/package_title_release.py --channel public|playtest. " + "To reuse this tree as a dev build, reconfigure with -UPSX_PUBLIC_RELEASE.") + endif() + if((PSX_PUBLIC_RELEASE AND PSX_RELEASE_CHANNEL STREQUAL "playtest") OR + (NOT PSX_PUBLIC_RELEASE AND PSX_RELEASE_CHANNEL STREQUAL "public")) + message(FATAL_ERROR + "PSX_PUBLIC_RELEASE=${PSX_PUBLIC_RELEASE} contradicts " + "PSX_RELEASE_CHANNEL=${PSX_RELEASE_CHANNEL}. Pass only the channel " + "(reconfigure with -UPSX_PUBLIC_RELEASE).") + endif() + endif() + unset(_psx_public_type) +endif() + +if(PSX_RELEASE_CHANNEL STREQUAL "public") + set(_psx_public_release ON) +else() + set(_psx_public_release OFF) +endif() +set(PSX_PUBLIC_RELEASE ${_psx_public_release} CACHE INTERNAL + "Derived from PSX_RELEASE_CHANNEL (public => ON); not an input") +unset(_psx_public_release) +if(PSX_RELEASE_CHANNEL STREQUAL "") + message(STATUS "psxrecomp: release channel: none (dev build; not packageable)") +else() + message(STATUS "psxrecomp: release channel: ${PSX_RELEASE_CHANNEL} " + "(PSX_PUBLIC_RELEASE=${PSX_PUBLIC_RELEASE})") +endif() + +# Optional exact source commit for build identities. The packager passes the +# title's HEAD; a tree configured from WSL cannot always ask git itself (a +# Windows worktree's .git names an F:/ path). Empty = the title decides. +set(PSX_SOURCE_COMMIT "" CACHE STRING + "Exact title source commit for build identities (set by the release packager)") +string(LENGTH "${PSX_SOURCE_COMMIT}" _psx_source_commit_length) +if(NOT PSX_SOURCE_COMMIT STREQUAL "" AND + (NOT PSX_SOURCE_COMMIT MATCHES "^[0-9a-f]+$" OR NOT _psx_source_commit_length EQUAL 40)) + message(FATAL_ERROR "PSX_SOURCE_COMMIT must be a full 40-hex commit (got '${PSX_SOURCE_COMMIT}')") +endif() +unset(_psx_source_commit_length) diff --git a/runtime/runtime.cmake b/runtime/runtime.cmake index d979a3c7..f84eab05 100644 --- a/runtime/runtime.cmake +++ b/runtime/runtime.cmake @@ -122,11 +122,12 @@ endif() # Dev-only: the lobby browser lists every version of this title instead of # pinning to our exact game_version. Joining still requires an exact match. option(PSX_LOBBY_DEV_LIST "Lobby browser ignores game_version (dev testing only)" OFF) -# The one build-level notion of a public production release. Release packagers -# set it ON; dev trees and playtest/pre-release packages leave it OFF. Today it -# selects Playtest Diagnostics defaults (runtime/playtest_diagnostics.cmake): -# OFF stages them default-on, ON keeps the source manifests' default-off. -option(PSX_PUBLIC_RELEASE "Public production release (Playtest Diagnostics default off)" OFF) +# Release channel (public | playtest | empty = dev) and the PSX_PUBLIC_RELEASE +# it derives: the one build-level notion of a public production release. Today +# PSX_PUBLIC_RELEASE selects Playtest Diagnostics defaults +# (runtime/playtest_diagnostics.cmake): OFF stages them default-on, ON keeps +# the source manifests' default-off. See runtime/psx_release_channel.cmake. +include("${CMAKE_CURRENT_LIST_DIR}/psx_release_channel.cmake") set(PSX_SDL_BACKEND "SDL3" CACHE STRING "SDL backend (SDL3 or SDL2)") set_property(CACHE PSX_SDL_BACKEND PROPERTY STRINGS SDL3 SDL2) string(TOUPPER "${PSX_SDL_BACKEND}" _psx_sdl_backend) diff --git a/tools/ci/audit_playtest_diagnostics_package.py b/tools/ci/audit_playtest_diagnostics_package.py index ddeba084..d7cc8cbf 100644 --- a/tools/ci/audit_playtest_diagnostics_package.py +++ b/tools/ci/audit_playtest_diagnostics_package.py @@ -1,7 +1,13 @@ #!/usr/bin/env python3 -"""Audit a flat game/setup ZIP without extracting or launching its contents.""" +"""Audit a flat game/setup ZIP (or its staged directory) without launching it. + +The argument is a release ZIP or a staged payload directory. A directory is +read through the same checks, so a Linux payload is audited before it is +wrapped into an AppImage (an AppImage has no ZIP to audit). +""" import argparse -from pathlib import PurePosixPath +import os +from pathlib import Path, PurePosixPath import re import sys import zipfile @@ -28,11 +34,56 @@ PERSONAL_NAMES = { } -def audit(path, game_id, diagnostics_id, public): +class _DirEntry: + def __init__(self, root, name): + self.filename = name + self._path = os.path.join(root, *name.split("/")) + self.file_size = os.path.getsize(self._path) + + def is_dir(self): + return False + + +class _DirArchive: + """The read-only subset of zipfile.ZipFile the audit uses, over a directory.""" + + def __init__(self, root): + self._root = str(root) + self._entries = [] + for base, dirs, files in os.walk(self._root): + dirs.sort() + for leaf in sorted(files): + rel = os.path.relpath(os.path.join(base, leaf), self._root) + self._entries.append(_DirEntry(self._root, rel.replace(os.sep, "/"))) + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + def infolist(self): + return list(self._entries) + + def read(self, entry): + with open(entry._path, "rb") as stream: + return stream.read() + + def open(self, entry): + return open(entry._path, "rb") + + +def _open_package(path): + if os.path.isdir(path): + return _DirArchive(path) + return zipfile.ZipFile(path) + + +def audit(path, game_id, diagnostics_id, public, replay_helper=True): """public=True: a public production release (Playtest Diagnostics ship off). public=False: a playtest/pre-release package (they ship on). Either way the source manifests keep the production default, off.""" - with zipfile.ZipFile(path) as archive: + with _open_package(path) as archive: files = {} for entry in archive.infolist(): name = entry.filename.replace("\\", "/") @@ -79,9 +130,13 @@ def audit(path, game_id, diagnostics_id, public): recipe = tomllib.loads(read("game.toml").decode("utf-8-sig")) if recipe.get("game", {}).get("id") != game_id: raise ValueError("game.toml game ID does not match " + game_id) - read("replay-diagnostics.py") + # Titles on psxrecomp_add_playtest_diagnostics ship the shared replay + # launcher beside the executable; a title with its own recorder does not. + if replay_helper: + read("replay-diagnostics.py") if not any(executable(name, entry) for name, entry in files.items()): - raise ValueError("no executable adjacent to replay-diagnostics.py") + raise ValueError("no executable adjacent to replay-diagnostics.py" if replay_helper + else "no executable at the package root") runtime = [] for name in sorted(files): @@ -145,15 +200,19 @@ def validate_diagnostics(manifest, name, game_id, diagnostics_id, default_enable def main(): parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument("zip") + parser.add_argument("zip", help="release ZIP or staged payload directory") parser.add_argument("--game-id", required=True) parser.add_argument("--diagnostics-id", required=True) parser.add_argument("--mode", required=True, choices=("public", "playtest"), help="public: production release, diagnostics default off; " "playtest: non-public package, diagnostics default on") + parser.add_argument("--no-replay-helper", action="store_true", + help="the title records with its own diagnostics package and ships " + "no replay-diagnostics.py") args = parser.parse_args() try: - count = audit(args.zip, args.game_id, args.diagnostics_id, args.mode == "public") + count = audit(args.zip, args.game_id, args.diagnostics_id, args.mode == "public", + replay_helper=not args.no_replay_helper) except (ValueError, OSError, zipfile.BadZipFile, KeyError, UnicodeError) as error: sys.exit("diagnostics package audit failed: " + str(error)) print("Diagnostics package audit passed: {} ({} runtime manifest(s), {} mode)".format( diff --git a/tools/package_appimage.sh b/tools/package_appimage.sh index afcf3ef7..2c9bc641 100755 --- a/tools/package_appimage.sh +++ b/tools/package_appimage.sh @@ -5,13 +5,17 @@ # and usr/share/, preserving user settings/cards in writable data. # Usage: package_appimage.sh --payload DIR --exe-name NAME --payload-name NAME # --app-run FILE --desktop-file FILE.desktop --icon FILE.png --output FILE.AppImage -# [--framework DIR] [--tool-cache DIR] [--jobs N] +# [--framework DIR] [--tool-cache DIR] [--jobs N] [--prune-unreferenced-libs] +# --prune-unreferenced-libs drops usr/lib entries the executable does not load +# (keeps a host file picker's LD_LIBRARY_PATH clean). Every run fails if the +# deployed executable has an unresolved library. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" FRAMEWORK="$(cd "${SCRIPT_DIR}/.." && pwd)" PAYLOAD=""; EXE_NAME=""; PAYLOAD_NAME=""; APP_RUN=""; DESKTOP=""; ICON=""; OUTPUT="" TOOLS="${XDG_CACHE_HOME:-${HOME:?}/.cache}/recomp-appimage-tools" JOBS=2 +PRUNE_LIBS=0 while [[ $# -gt 0 ]]; do case "$1" in --payload) PAYLOAD="${2:?}"; shift 2;; @@ -24,7 +28,8 @@ while [[ $# -gt 0 ]]; do --framework) FRAMEWORK="${2:?}"; shift 2;; --tool-cache) TOOLS="${2:?}"; shift 2;; --jobs) JOBS="${2:?}"; shift 2;; - -h|--help) sed -n '2,8p' "$0"; exit 0;; + --prune-unreferenced-libs) PRUNE_LIBS=1; shift;; + -h|--help) sed -n '2,11p' "$0"; exit 0;; *) echo "unknown argument: $1" >&2; exit 2;; esac done @@ -102,6 +107,20 @@ export NO_STRIP=1 "$LINUXDEPLOY" --appimage-extract-and-run --appdir "$APPDIR" \ --executable "$APPDIR/usr/bin/$EXE_NAME" --desktop-file "$APPDIR/$DESKTOP_ID.desktop" \ --icon-file "$APPDIR/$DESKTOP_ID.png" +if [[ "$PRUNE_LIBS" == 1 ]]; then + KEEP="$(mktemp "${TMPDIR:-/tmp}/psx-appimage-keep.XXXXXXXX")" + env -u LD_LIBRARY_PATH ldd "$APPDIR/usr/bin/$EXE_NAME" | + awk '{ for (i = 1; i <= NF; i++) if ($i ~ /^\//) print $i }' | + while read -r lib; do readlink -f "$lib"; done | sort -u > "$KEEP" + for lib in "$APPDIR"/usr/lib/*; do + [[ -e "$lib" ]] || continue + grep -qxF "$(readlink -f "$lib")" "$KEEP" || { echo "pruned unreferenced $(basename "$lib")"; rm -f -- "$lib"; } + done + rm -f -- "$KEEP" +fi +if env -u LD_LIBRARY_PATH ldd "$APPDIR/usr/bin/$EXE_NAME" | grep 'not found'; then + echo "unresolved libraries in the deployed executable" >&2; exit 1 +fi # Deployment may rewrite RPATH. The original sidecar supplies the contract, # while the helper verifies its embedded identity and hashes the FINAL ELF; # a prior payload binary_sha256 is never accepted as the final-byte receipt. diff --git a/tools/package_game_release.sh b/tools/package_game_release.sh index b3b09543..a463b45f 100755 --- a/tools/package_game_release.sh +++ b/tools/package_game_release.sh @@ -39,6 +39,14 @@ # the same deliberately unwieldy switch release_stage.py exposes, because a # packager that quietly staged nothing once looked like a good release twice. # +# RELEASE CHANNEL. The build tree must have been configured with +# -DPSX_RELEASE_CHANNEL=public|playtest (runtime/psx_release_channel.cmake); +# a tree without one is a dev build and is refused. playtest packages are +# named --playtest-.zip, public packages always +# drop developer-channel mods, and the staged Playtest Diagnostics defaults +# must match the channel. Local title releases use +# tools/package_title_release.py instead (docs/RELEASING_A_TITLE.md). +# # DEVELOPER CHANNEL. mods with channel = "developer" are pruned from public # releases (EXCLUDE_DEV_MODS=1, which CI sets via $CI) and kept in local # exports; --include-dev-mods / --exclude-dev-mods override. Pruning is by @@ -198,6 +206,18 @@ if ! grep -qs 'PSX_HAS_GAME_DISPATCH' "${BUILD_DIR}/build.ninja" "${BUILD_DIR}/c exit 1 fi +# --- Release channel: from the build, never defaulted ------------------------- +CHANNEL="$(sed -n 's/^PSX_RELEASE_CHANNEL:[A-Z]*=//p' "${BUILD_DIR}/CMakeCache.txt" 2>/dev/null | head -n1 || true)" +case "${CHANNEL}" in + public) CHANNEL_SUFFIX=""; EXCLUDE_DEV_MODS=1 ;; + playtest) CHANNEL_SUFFIX="-playtest" ;; + *) + echo "error: ${BUILD_DIR} has no release channel (PSX_RELEASE_CHANNEL='${CHANNEL}')." >&2 + echo " Configure the release build with -DPSX_RELEASE_CHANNEL=public or =playtest." >&2 + exit 1 ;; +esac +echo "release channel: ${CHANNEL}" + normalize_ver() { local v; v="$(printf '%s' "${1:-}" | tr -d '[:space:]')"; printf '%s' "${v#v}"; } BUILT="" for stamp in "${EXE_DIR}/psx_game_version.txt" "${BUILD_DIR}/psx_game_version.txt" "${BUILD_DIR}/Release/psx_game_version.txt"; do @@ -222,7 +242,7 @@ printf '%s\n' "${VERSION}" >"${ROOT}/VERSION" DIST="${ROOT:?}/dist" STAGE="${DIST:?}/stage-game-${ARTIFACT:?}" -ZIP_NAME="${ZIP_PREFIX:?}-${VERSION:?}-${ARTIFACT:?}.zip" +ZIP_NAME="${ZIP_PREFIX:?}-${VERSION:?}${CHANNEL_SUFFIX}-${ARTIFACT:?}.zip" rm -rf "${STAGE:?}" mkdir -p "${STAGE}" "${DIST}" if [[ "${STAGE_ONLY}" != 1 ]]; then rm -f "${DIST:?}/${ZIP_NAME:?}"; fi @@ -338,6 +358,11 @@ else [[ "${n}" -eq 0 ]] || echo "note: package includes ${n} developer-channel manifest(s); pass --exclude-dev-mods for a public release" fi +if ! "${PY}" "${SCRIPT_DIR}/package_title_release.py" check-diagnostics --channel "${CHANNEL}" "${STAGE}/mods/bundled"; then + echo "error: Playtest Diagnostics defaults do not match the ${CHANNEL} channel" >&2 + exit 1 +fi + # --- Overlay shard cache (developer-local only) and toolchain ----------------- GAME_BIN="" for cand in "${RECOMPILER_BUILD}/psxrecomp-game" "${RECOMPILER_BUILD}/psxrecomp-game.exe" "${RECOMPILER_BUILD}/Release/psxrecomp-game.exe"; do diff --git a/tools/package_title_release.ps1 b/tools/package_title_release.ps1 new file mode 100644 index 00000000..a106e00f --- /dev/null +++ b/tools/package_title_release.ps1 @@ -0,0 +1,19 @@ +# Windows launcher for tools/package_title_release.py (the one release command). +# Run from the title repository root: +# psxrecomp/tools/package_title_release.ps1 --channel public|playtest [--platform windows|linux|all] +# Arguments pass through unchanged; --channel is required (no default). +# Exit code is the packager's; failures end with "PACKAGING FAILED: ". +$ErrorActionPreference = 'Stop' +$script = Join-Path $PSScriptRoot 'package_title_release.py' +$python = $env:PSX_RELEASE_PYTHON +$prefix = @() +if (-not $python) { + if (Get-Command py -ErrorAction SilentlyContinue) { $python = 'py'; $prefix = @('-3') } + elseif (Get-Command python -ErrorAction SilentlyContinue) { $python = 'python' } + else { + Write-Host 'PACKAGING FAILED: Python 3.11+ not found (install it or set PSX_RELEASE_PYTHON)' -ForegroundColor Red + exit 1 + } +} +& $python @prefix $script @args +exit $LASTEXITCODE diff --git a/tools/package_title_release.py b/tools/package_title_release.py new file mode 100755 index 00000000..ed3edb45 --- /dev/null +++ b/tools/package_title_release.py @@ -0,0 +1,1391 @@ +#!/usr/bin/env python3 +"""Cut a PSXRecomp title release: one command, every platform, both channels. + + psxrecomp/tools/package_title_release.py --channel public|playtest + [--config packaging/release.toml] [--platform windows|linux|all] + +Run from the title repository root (or pass --root). The title keeps only its +release config (packaging/release.toml, see docs/RELEASING_A_TITLE.md); every +step below is the framework's, so a release fix lands once for every title. + +--channel is REQUIRED and has no default: + public public production release. -DPSX_RELEASE_CHANNEL=public, so + PSX_PUBLIC_RELEASE=ON and Playtest Diagnostics ship default off. + playtest non-public playtest package. Diagnostics ship default on and every + artifact name carries "-playtest". +Every audit runs in the matching mode. + +What one run owns, in order (any failure ends with one line +"PACKAGING FAILED: " and a non-zero exit; nothing warns and continues): + 1. source provenance: title/framework/recomp-ui commits; refuses a dirty + tracked tree unless --allow-dirty (recorded in the manifest). + 2. emitters: psxrecomp-game AND psxrecomp-bios, always, up front. + 3. source preparation: recompiled OpenBIOS present (regen if missing), + game C regenerated from the disc, static AOT units when the title's AOT + mode is "static". On Windows, --platform linux/all prepares once here and + the Linux build compiles the same audited sources. + 4. configure + build: Release, PSX_DEBUG_TOOLS=OFF, OpenBIOS-only backend + (PSXRECOMP_BIOS_STEMS=OpenBIOS), PSX_RELEASE_CHANNEL=, + PSX_GAME_VERSION from the title's version file; every owned value is + read back from CMakeCache.txt. + 5. stage: executable, assets, OpenBIOS + notice, configs, mod catalog + (verified against the build's own catalog list; developer-channel + features pruned from public packages), overlay cache via the original- + disc AOT pipeline (AOT mode "cache") and/or overlay toolchain, notices, + title docs, START_HERE.txt with @VERSION@ stamped, execution contract. + 6. gates: no disc/BIOS dumps/saves/personal state/generated C/build kit; + OpenBIOS only; diagnostics defaults per channel; Windows executable + imports only system DLLs; static AOT receipt matches the linked sources. + 7. artifact: Windows ZIP (portable POSIX names) or Linux AppImage + (tools/package_appimage.sh, fixed linuxdeploy/appimagetool pins), its + .sha256, RELEASE_MANIFEST.json inside, then the artifact is re-opened / + extracted and verified (`verify` below) before success is printed. + +Linux from Windows: --platform linux (or all) runs the Linux half inside WSL +(default distribution) on the same checkout, from /mnt//... + + package_title_release.py verify --channel C [--config ...] ARTIFACT +re-checks a finished ZIP or AppImage by extracting it. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +from pathlib import Path, PurePosixPath +import platform as _platform +import re +import shlex +import shutil +import subprocess +import sys +import tempfile +import zipfile + +try: + import tomllib +except ModuleNotFoundError: # Python < 3.11 + import tomli as tomllib # type: ignore + +TOOLS = Path(__file__).resolve().parent +FRAMEWORK = TOOLS.parent +sys.path.insert(0, str(TOOLS)) + +CHANNELS = ('public', 'playtest') +PLATFORMS = ('windows', 'linux') +ARTIFACT_SUFFIX = {'windows': 'windows-x64.zip', 'linux': 'linux-x86_64.AppImage'} +TOOLCHAIN_PLATFORM = {'windows': 'win', 'linux': 'linux'} +AOT_MODES = ('cache', 'static', 'none') +OPENBIOS_SIZE = 524288 +CREATE_NO_WINDOW = 0x08000000 + +# Flags the packager owns. A title's [build].cmake_args may not set them. +OWNED_CMAKE = ('CMAKE_BUILD_TYPE', 'PSX_DEBUG_TOOLS', 'PSX_RELEASE_CHANNEL', + 'PSX_PUBLIC_RELEASE', 'PSXRECOMP_BIOS_STEMS', 'PSX_GAME_VERSION', + 'PSX_SOURCE_COMMIT', 'PSXRECOMP_ROOT') + +# Windows system DLLs a self-contained release executable may import. +SYSTEM_DLLS = frozenset(''' +advapi32 bcrypt cfgmgr32 comdlg32 crypt32 d2d1 d3d11 d3d12 dbghelp dinput8 dwmapi +dwrite dxgi gdi32 hid imm32 iphlpapi kernel32 msvcrt ntdll ole32 oleaut32 +opengl32 rpcrt4 setupapi shell32 shlwapi ucrtbase user32 userenv uxtheme +version winhttp winmm wintrust ws2_32 +'''.split()) + +# Never in a player package (beyond bios/openbios.bin). +DISC_SUFFIXES = {'.cue', '.iso', '.img', '.chd', '.ccd', '.sub', '.mds', '.mdf', + '.pbp', '.ecm', '.mcd', '.mcr', '.pst', '.pack', '.srm', '.gme'} +PERSONAL_NAMES = {'settings.toml', 'disc.cfg', 'bios.cfg', 'rom.cfg', 'state.toml', + 'state.toml.tmp', 'keybinds.ini', 'overlay_captures.json', + 'diagnostics-location.txt', 'psx_last_run_report.json', + 'psx_crash.txt', 'psx_freeze_heartbeat.json'} +BUILD_KIT = ('psxrecomp', 'recomp-ui', 'generated', 'seeds', 'src', 'disc', 'iso', + 'saves', 'toolchain', 'CMakeLists.txt', 'codegen_setup.c', + 'codegen_setup.h', 'psxrecomp_cli.py', 'psxrecomp-game', + 'psxrecomp-game.exe', 'psxrecomp-bios', 'psxrecomp-bios.exe') + + +class PackagingError(Exception): + pass + + +def fail(message): + raise PackagingError(message) + + +def log(message=''): + print(message, flush=True) + + +def sha256(path): + digest = hashlib.sha256() + with open(path, 'rb') as stream: + for block in iter(lambda: stream.read(1 << 20), b''): + digest.update(block) + return digest.hexdigest() + + +def host_platform(): + if os.name == 'nt': + return 'windows' + if sys.platform.startswith('linux'): + return 'linux' + fail('unsupported host %s: packaging runs on Windows or Linux' % sys.platform) + + +def run(command, what, cwd=None, env=None, capture=False): + """Run a child, stream its output, fail loudly on a non-zero exit. + + Output is piped through this process so CREATE_NO_WINDOW never loses it: + no child ever opens a console window on the desktop.""" + command = [str(c) for c in command] + log('+ ' + ' '.join(shlex.quote(c) for c in command)) + kwargs = dict(cwd=str(cwd) if cwd else None, env=env, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if os.name == 'nt': + kwargs['creationflags'] = CREATE_NO_WINDOW + try: + process = subprocess.Popen(command, **kwargs) + except OSError as error: + fail('%s: cannot start %s (%s)' % (what, command[0], error)) + captured = [] + for raw in process.stdout: + line = raw.decode('utf-8', 'replace') + if capture: + captured.append(line) + else: + sys.stdout.write(line) + sys.stdout.flush() + code = process.wait() + if code != 0: + if capture: + sys.stdout.write(''.join(captured)) + fail('%s failed (exit %d)' % (what, code)) + return ''.join(captured) + + +def read_text(path): + return Path(path).read_text(encoding='utf-8-sig') + + +# -------------------------------------------------------------------------- +# Config +# -------------------------------------------------------------------------- +class Config: + """packaging/release.toml, validated. Unknown keys are errors (a typo in a + release config must not silently drop a step).""" + + SCHEMA = { + '': {'schema', 'product', 'build', 'aot', 'overlay_toolchain', 'diagnostics', + 'payload', 'linux', 'hooks'}, + 'product': {'name', 'display_name', 'exe_name', 'version_file', 'game_toml', + 'player_game_toml', 'runtime_target', 'zip_folder', 'launcher_boxart'}, + 'build': {'cmake_args', 'linux_cmake_args', 'windows_cmake_args', 'regenerate', + 'binary_identity'}, + 'aot': {'mode', 'profile', 'cps'}, + 'overlay_toolchain': {'ship'}, + 'diagnostics': {'game_id', 'package_id', 'replay_helper'}, + 'payload': {'start_here', 'files', 'build_files'}, + 'linux': {'app_run', 'desktop_file', 'icon', 'payload_dir', 'env_prefix', + 'prune_unreferenced_libs'}, + 'hooks': {'prepare', 'post_build', 'post_stage'}, + } + + def __init__(self, path, root): + self.path = Path(path) + self.root = Path(root) + if not self.path.is_file(): + fail('release config not found: %s' % self.path) + try: + data = tomllib.loads(read_text(self.path)) + except (tomllib.TOMLDecodeError, UnicodeError) as error: + fail('%s: %s' % (self.path, error)) + self._check_keys(data) + if data.get('schema') != 1: + fail('%s: schema must be 1' % self.path) + product = self._table(data, 'product', required=True) + self.name = self._name(product, 'name') + self.display_name = self._str(product, 'display_name', self.name) + self.exe_name = product.get('exe_name') + if self.exe_name is not None: + self._name(product, 'exe_name') + self.version_file = self._file(product, 'version_file', required=True) + self.game_toml = self._file(product, 'game_toml', default='game.toml') + self.player_game_toml = self._file(product, 'player_game_toml', + default=product.get('game_toml', 'game.toml')) + self.runtime_target = self._str(product, 'runtime_target', 'psx-runtime') + self.zip_folder = self._str(product, 'zip_folder', '') + if self.zip_folder: + self._name(product, 'zip_folder') + self.launcher_boxart = self._file(product, 'launcher_boxart') + + build = self._table(data, 'build') + self.cmake_args = self._args(build, 'cmake_args') + self.platform_cmake_args = {'linux': self._args(build, 'linux_cmake_args'), + 'windows': self._args(build, 'windows_cmake_args')} + self.regenerate = self._bool(build, 'regenerate', True) + self.binary_identity = self._str(build, 'binary_identity', '') + + aot = self._table(data, 'aot') + self.aot_mode = self._str(aot, 'mode', 'none') + if self.aot_mode not in AOT_MODES: + fail('%s: [aot] mode must be one of %s' % (self.path, ', '.join(AOT_MODES))) + self.aot_profile = self._file(aot, 'profile', required=self.aot_mode != 'none') + self.aot_cps = self._bool(aot, 'cps', self.aot_mode == 'static') + + self.ship_toolchain = self._bool(self._table(data, 'overlay_toolchain'), 'ship', False) + + diag = self._table(data, 'diagnostics') + self.diagnostics = None + if diag: + self.diagnostics = dict(game_id=self._str(diag, 'game_id', required=True), + package_id=self._str(diag, 'package_id', required=True), + replay_helper=self._bool(diag, 'replay_helper', True)) + + payload = self._table(data, 'payload', required=True) + self.start_here = self._file(payload, 'start_here', required=True) + self.files = [] + for item in payload.get('files', []): + if isinstance(item, str): + item = {'from': item, 'to': item} + if not isinstance(item, dict) or set(item) - {'from', 'to'} or 'from' not in item: + fail('%s: [payload] files entries are "path" or {from=, to=}' % self.path) + source = self.root / item['from'] + if not source.is_file(): + fail('%s: payload file not found: %s' % (self.path, item['from'])) + self.files.append((source, self._relpath(item.get('to', item['from'])))) + self.build_files = [self._relpath(p) for p in payload.get('build_files', [])] + + linux = self._table(data, 'linux') + self.linux = None + if linux: + self.linux = dict(app_run=self._file(linux, 'app_run', required=True), + desktop_file=self._file(linux, 'desktop_file', required=True), + icon=self._file(linux, 'icon', required=True), + payload_dir=self._name(linux, 'payload_dir'), + env_prefix=self._str(linux, 'env_prefix', ''), + prune_libs=self._bool(linux, 'prune_unreferenced_libs', False)) + + hooks = self._table(data, 'hooks') + self.hooks = {} + for phase in ('prepare', 'post_build', 'post_stage'): + entries = hooks.get(phase, []) + if not isinstance(entries, list): + fail('%s: [[hooks.%s]] must be an array of tables' % (self.path, phase)) + self.hooks[phase] = [] + for entry in entries: + if not isinstance(entry, dict) or set(entry) - {'command', 'platforms'}: + fail('%s: hooks.%s entries take command = [...] and optional platforms' + % (self.path, phase)) + command = entry.get('command') + if (not isinstance(command, list) or not command or + not all(isinstance(c, str) for c in command)): + fail('%s: hooks.%s command must be a non-empty list of strings' + % (self.path, phase)) + platforms = entry.get('platforms', list(PLATFORMS)) + if not set(platforms) <= set(PLATFORMS): + fail('%s: hooks.%s platforms must be within %s' % (self.path, phase, PLATFORMS)) + self.hooks[phase].append((command, platforms)) + + for args in [self.cmake_args] + list(self.platform_cmake_args.values()): + for arg in args: + match = re.match(r'^-D([A-Za-z0-9_]+)', arg) + if match and match.group(1) in OWNED_CMAKE: + fail('%s: [build] cmake args may not set %s; the packager owns it ' + '(channel via --channel)' % (self.path, match.group(1))) + + def _check_keys(self, data): + unknown = set(data) - self.SCHEMA[''] + for table, allowed in self.SCHEMA.items(): + if table and isinstance(data.get(table), dict): + unknown |= {'%s.%s' % (table, k) for k in set(data[table]) - allowed} + if unknown: + fail('%s: unknown key(s): %s' % (self.path, ', '.join(sorted(unknown)))) + + def _table(self, data, key, required=False): + value = data.get(key) + if value is None: + if required: + fail('%s: missing [%s]' % (self.path, key)) + return {} + if not isinstance(value, dict): + fail('%s: %s must be a table' % (self.path, key)) + return value + + def _str(self, table, key, default=None, required=False): + value = table.get(key, default) + if value is None or (required and value == ''): + fail('%s: missing %s' % (self.path, key)) + if not isinstance(value, str): + fail('%s: %s must be a string' % (self.path, key)) + return value + + def _bool(self, table, key, default): + value = table.get(key, default) + if not isinstance(value, bool): + fail('%s: %s must be true or false' % (self.path, key)) + return value + + def _name(self, table, key): + value = self._str(table, key, required=True) + if not re.fullmatch(r'[A-Za-z0-9_][A-Za-z0-9_.-]*', value): + fail('%s: %s %r must be a plain file name' % (self.path, key, value)) + return value + + def _args(self, table, key): + value = table.get(key, []) + if not isinstance(value, list) or not all(isinstance(a, str) for a in value): + fail('%s: %s must be a list of strings' % (self.path, key)) + return value + + def _relpath(self, value): + path = PurePosixPath(value) + if path.is_absolute() or '..' in path.parts or not path.parts or ':' in value: + fail('%s: %r must be a relative path inside the package' % (self.path, value)) + return path.as_posix() + + def _file(self, table, key, default=None, required=False): + value = table.get(key, default) + if value is None: + if required: + fail('%s: missing %s' % (self.path, key)) + return None + if not isinstance(value, str): + fail('%s: %s must be a string' % (self.path, key)) + path = self.root / value + if not path.is_file(): + fail('%s: %s not found: %s' % (self.path, key, value)) + return path + + def version(self): + raw = read_text(self.version_file).strip() + if not raw or not re.fullmatch(r'v?[0-9][0-9A-Za-z.+-]*', raw): + fail('%s: invalid version %r' % (self.version_file, raw)) + return raw + + +def artifact_name(config, version, channel, plat): + bare = version[1:] if version.startswith('v') else version + suffix = '-playtest' if channel == 'playtest' else '' + return '%s-v%s%s-%s' % (config.name, bare, suffix, ARTIFACT_SUFFIX[plat]) + + +# -------------------------------------------------------------------------- +# Provenance +# -------------------------------------------------------------------------- +def git_executable(): + explicit = os.environ.get('PSX_RELEASE_GIT') + if explicit: + return explicit + if os.name == 'nt': + # PowerShell's `git` is often devkitPro MSYS git, which cannot read a + # worktree whose .git names an F:/ path. + for candidate in (r'C:\Program Files\Git\cmd\git.exe',): + if os.path.isfile(candidate): + return candidate + found = shutil.which('git') + if not found: + fail('git not found (set PSX_RELEASE_GIT)') + return found + + +def git(repo, *args): + return run([git_executable(), '-C', repo] + list(args), 'git %s' % ' '.join(args), + capture=True).strip() + + +def provenance(root, ui_root): + """Commits and cleanliness, from git or (inside WSL) from the Windows parent.""" + env = os.environ + if env.get('PSX_RELEASE_SOURCE_COMMIT'): + return dict(source_commit=env['PSX_RELEASE_SOURCE_COMMIT'], + source_date_epoch=int(env.get('PSX_RELEASE_SOURCE_DATE_EPOCH') or 0), + source_dirty=env.get('PSX_RELEASE_SOURCE_DIRTY') == '1', + dirty_paths=json.loads(env.get('PSX_RELEASE_DIRTY_PATHS', '[]')), + framework_commit=env['PSX_RELEASE_FRAMEWORK_COMMIT'], + recomp_ui_commit=env.get('PSX_RELEASE_UI_COMMIT') or None) + status = git(root, 'status', '--porcelain', '--untracked-files=no', '--ignore-submodules=none') + dirty = [line[3:] for line in status.splitlines() if line.strip()] + fw_status = git(FRAMEWORK, 'status', '--porcelain', '--untracked-files=no') + dirty += ['psxrecomp/' + line[3:] for line in fw_status.splitlines() if line.strip()] + ui_commit = None + if ui_root and (Path(ui_root) / '.git').exists(): + ui_commit = git(ui_root, 'rev-parse', 'HEAD') + return dict(source_commit=git(root, 'rev-parse', 'HEAD'), + source_date_epoch=int(git(root, 'log', '-1', '--format=%ct')), + source_dirty=bool(dirty), + dirty_paths=dirty, framework_commit=git(FRAMEWORK, 'rev-parse', 'HEAD'), + recomp_ui_commit=ui_commit) + + +# -------------------------------------------------------------------------- +# Toolchain +# -------------------------------------------------------------------------- +class Toolchain: + def __init__(self, plat, toolchain_bin): + self.platform = plat + self.env = dict(os.environ) + if plat == 'windows': + self.bin = Path(toolchain_bin) + for tool in ('cmake.exe', 'ninja.exe', 'gcc.exe', 'objdump.exe'): + if not (self.bin / tool).is_file(): + fail('%s missing from %s (pass --toolchain-bin)' % (tool, self.bin)) + self.env['PATH'] = str(self.bin) + os.pathsep + self.env.get('PATH', '') + self.cmake = str(self.bin / 'cmake.exe') + self.gcc = str(self.bin / 'gcc.exe') + self.objdump = str(self.bin / 'objdump.exe') + self.exe_suffix = '.exe' + else: + for tool in ('cmake', 'gcc'): + if not shutil.which(tool): + fail('%s not found on PATH' % tool) + self.cmake, self.gcc, self.objdump = 'cmake', 'gcc', None + self.exe_suffix = '' + self.generator = 'Ninja' if (plat == 'windows' or shutil.which('ninja')) else 'Unix Makefiles' + + +def hook_tokens(ctx, **extra): + tokens = dict(python=sys.executable, root=str(ctx.root), framework=str(FRAMEWORK), + channel=ctx.channel, platform=ctx.platform, version=ctx.version, + build_dir=str(ctx.runtime_build), work_dir=str(ctx.work)) + tokens.update(extra) + return tokens + + +def run_hooks(ctx, phase, **extra): + for command, platforms in ctx.config.hooks[phase]: + if ctx.platform not in platforms: + continue + tokens = hook_tokens(ctx, **extra) + try: + argv = [part.format(**tokens) for part in command] + except KeyError as error: + fail('hooks.%s: unknown token %s' % (phase, error)) + run(argv, 'hooks.%s %s' % (phase, ' '.join(command[:2])), cwd=ctx.root, env=ctx.tc.env) + + +# -------------------------------------------------------------------------- +# Build steps +# -------------------------------------------------------------------------- +class Context: + pass + + +def build_emitters(ctx): + build = ctx.work / 'recompiler' + run([ctx.tc.cmake, '-S', FRAMEWORK / 'recompiler', '-B', build, '-G', ctx.tc.generator, + '-DCMAKE_BUILD_TYPE=Release'], 'emitter configure', env=ctx.tc.env) + run([ctx.tc.cmake, '--build', build, '--target', 'psxrecomp-game', 'psxrecomp-bios', + '-j', ctx.jobs], 'emitter build', env=ctx.tc.env) + for name in ('psxrecomp-game', 'psxrecomp-bios'): + if not (build / (name + ctx.tc.exe_suffix)).is_file(): + fail('emitter build did not produce %s' % (build / (name + ctx.tc.exe_suffix))) + ctx.recompiler_dir = build + ctx.recompiler = build / ('psxrecomp-game' + ctx.tc.exe_suffix) + + +def ensure_openbios(ctx): + if (FRAMEWORK / 'generated/OpenBIOS_dispatch.c').is_file(): + return + log('Recompiled OpenBIOS backend missing; generating it') + if not getattr(ctx, 'recompiler_dir', None): + build_emitters(ctx) + rel_build = os.path.relpath(ctx.recompiler_dir, FRAMEWORK).replace(os.sep, '/') + if ctx.platform == 'windows': + bash = Path(r'C:\msys64\usr\bin\bash.exe') + if not bash.is_file(): + fail('OpenBIOS C is missing and MSYS2 bash is not installed to regenerate it') + posix = lambda p: run([str(bash.parent / 'cygpath.exe'), '-u', str(p)], 'cygpath', + capture=True).strip() + script = ("export PATH='%s':/usr/bin:$PATH; cd '%s' && PSXRECOMP_BIOS_BUILD='%s' " + "tools/regen_bios.sh --config bios/OpenBIOS.toml" + % (posix(ctx.tc.bin), posix(FRAMEWORK), rel_build)) + run([bash, '-c', script], 'regen_bios OpenBIOS', env=ctx.tc.env) + else: + env = dict(ctx.tc.env, PSXRECOMP_BIOS_BUILD=rel_build) + run(['bash', 'tools/regen_bios.sh', '--config', 'bios/OpenBIOS.toml'], + 'regen_bios OpenBIOS', cwd=FRAMEWORK, env=env) + # regen_bios.sh has exited 0 on failure before; check the product. + if not (FRAMEWORK / 'generated/OpenBIOS_dispatch.c').is_file(): + fail('regen_bios produced no generated/OpenBIOS_dispatch.c') + + +def prepare_sources(ctx): + """Platform-independent source generation: game C and static AOT units.""" + config = ctx.config + if config.regenerate: + run([ctx.recompiler, '--config', config.game_toml], 'game C regeneration', + cwd=ctx.root, env=ctx.tc.env) + if config.aot_mode == 'static': + run([sys.executable, FRAMEWORK / 'tools/aot_overlay_pipeline.py', 'static', + '--profile', config.aot_profile, '--game-toml', config.game_toml, + '--recompiler', ctx.recompiler, '--work-dir', ctx.work / 'aot-static', + '--gcc', ctx.tc.gcc, '--cmake', ctx.tc.cmake, '--workers', ctx.jobs, '--reuse'] + + (['--cps'] if config.aot_cps else []), + 'static AOT generation', cwd=ctx.root, env=ctx.tc.env) + run_hooks(ctx, 'prepare') + + +def cmake_cache(build): + path = Path(build) / 'CMakeCache.txt' + if not path.is_file(): + fail('no CMakeCache.txt in %s' % build) + values = {} + for line in path.read_text(encoding='utf-8', errors='replace').splitlines(): + match = re.match(r'^([A-Za-z0-9_.+-]+):[A-Z_]+=(.*)$', line) + if match: + values[match.group(1)] = match.group(2) + return values + + +def cmake_bool(value): + return str(value).upper() in ('ON', '1', 'TRUE', 'YES', 'Y') + + +def configure_and_build(ctx): + owned = { + 'CMAKE_BUILD_TYPE': 'Release', + 'PSX_DEBUG_TOOLS': 'OFF', + 'PSX_RELEASE_CHANNEL': ctx.channel, + 'PSXRECOMP_BIOS_STEMS': 'OpenBIOS', + 'PSX_GAME_VERSION': ctx.version, + 'PSX_SOURCE_COMMIT': ctx.prov['source_commit'], + # The framework running this script is the framework that gets built. + 'PSXRECOMP_ROOT': FRAMEWORK.as_posix(), + } + args = ['-D%s=%s' % kv for kv in owned.items()] + args += ctx.config.cmake_args + ctx.config.platform_cmake_args[ctx.platform] + if ctx.platform == 'linux': + # The interpreter running this script; WSL otherwise finds the Windows + # pyenv shim first (beads-eio.3.299). --build-id=none keeps the ELF a + # function of its sources. + args += ['-DPSX_PYTHON=%s' % sys.executable, + '-DCMAKE_EXE_LINKER_FLAGS=-Wl,--build-id=none', + '-DCMAKE_C_COMPILER_LAUNCHER=', '-DCMAKE_CXX_COMPILER_LAUNCHER='] + run([ctx.tc.cmake, '-S', ctx.root, '-B', ctx.runtime_build, '-G', ctx.tc.generator] + args, + 'runtime configure', env=ctx.tc.env) + cache = cmake_cache(ctx.runtime_build) + expected = dict(owned, PSX_PUBLIC_RELEASE='ON' if ctx.channel == 'public' else 'OFF') + for arg in ctx.config.cmake_args + ctx.config.platform_cmake_args[ctx.platform]: + match = re.match(r'^-D([A-Za-z0-9_]+)(?::[A-Z]+)?=(.*)$', arg) + if match: + expected.setdefault(match.group(1), match.group(2)) + for key, want in expected.items(): + got = cache.get(key) + if key == 'PSXRECOMP_ROOT': + same = got is not None and Path(got).resolve() == FRAMEWORK + else: + same = (got == want or + (want.upper() in ('ON', 'OFF') and got is not None and + cmake_bool(got) == cmake_bool(want))) + if not same: + fail('configured %s=%r but CMakeCache.txt has %r (the title CMake overrode it?)' + % (key, want, got)) + if cache.get('PSX_RECOMP_UI', 'ON').upper() == 'OFF': + fail('PSX_RECOMP_UI=OFF: a player release needs the recomp-ui launcher') + run([ctx.tc.cmake, '--build', ctx.runtime_build, '--target', ctx.config.runtime_target, + '-j', ctx.jobs], 'runtime build', env=ctx.tc.env) + ctx.cache = cmake_cache(ctx.runtime_build) + + +def locate_executable(ctx): + marker = ctx.runtime_build / ('psxrecomp_exe_name-%s.txt' % ctx.config.runtime_target) + candidates = [] + if marker.is_file(): + candidates.append(read_text(marker).strip()) + candidates.append(ctx.config.runtime_target) + for name in candidates: + for directory in (ctx.runtime_build, ctx.runtime_build / 'Release'): + path = directory / (name + ctx.tc.exe_suffix) + if path.is_file(): + return path + fail('built executable not found under %s (marker %s)' % (ctx.runtime_build, marker)) + + +def check_build_outputs(ctx): + exe = locate_executable(ctx) + ctx.built_exe = exe + ctx.exe_dir = exe.parent + stamp = ctx.exe_dir / 'psx_game_version.txt' + if not stamp.is_file(): + fail('missing %s (rebuild with a current runtime.cmake)' % stamp) + built = read_text(stamp).strip() + if built != ctx.version: + fail('binary version stamp %r != %s %r' % (built, ctx.config.version_file.name, ctx.version)) + if ctx.config.binary_identity: + token = ctx.config.binary_identity.format( + version=ctx.version, commit=ctx.prov['source_commit'], + commit12=ctx.prov['source_commit'][:12]) + data = exe.read_bytes() + if token.encode('utf-8') not in data: + fail('executable lacks build identity %r: stale binary or wrong source commit' % token) + log('build identity present: %s' % token) + if ctx.config.aot_mode == 'static': + ctx.static_audit = check_static_audit(ctx) + run_hooks(ctx, 'post_build', exe=str(exe)) + + +def check_static_audit(ctx): + """The linked static AOT units are exactly the audited original-disc output.""" + config = ctx.config + profile = json.loads(read_text(config.aot_profile)) + output = profile.get('static_output') + if not output: + fail('%s: AOT mode "static" needs a profile static_output' % config.aot_profile) + out_dir = (config.game_toml.parent / output).parent + receipt_path = out_dir / 'AOT_STATIC_AUDIT.json' + if not receipt_path.is_file(): + fail('missing %s; the static AOT step did not publish' % receipt_path) + audit = json.loads(read_text(receipt_path)) + if not audit.get('all_pairs_valid') or not audit.get('all_guards_match_known_input_bytes'): + fail('original-disc static AOT audit failed (%s)' % receipt_path) + expected = profile.get('expected_records') + if expected is not None and audit.get('recipe_count') != expected: + fail('static AOT recipe_count %s != profile expected_records %s' + % (audit.get('recipe_count'), expected)) + inputs = audit.get('generation_inputs', {}) + if inputs.get('game_toml_sha256') != sha256(config.game_toml): + fail('%s changed since static AOT generation' % config.game_toml.name) + if inputs.get('profile_sha256') != sha256(config.aot_profile): + fail('AOT profile changed since static AOT generation') + files = audit.get('files') or {} + if not files: + fail('static AOT receipt lists no files') + for name, digest in files.items(): + if sha256(out_dir / name) != digest: + fail('static AOT unit %s differs from its audit receipt' % name) + ninja = ctx.runtime_build / 'build.ninja' + if ninja.is_file(): + exe = ctx.built_exe.name + links = [line for line in read_text(ninja).splitlines() + if line.startswith('build ') and 'EXECUTABLE_LINKER' in line and exe in line.split(':')[0]] + if not links: + fail('no link rule for %s in build.ninja' % exe) + for name in files: + if name.endswith('.c') and not any(name in line for line in links): + fail('static AOT unit %s is not linked into %s' % (name, exe)) + log('static AOT receipt verified: %d recipes, %d files' % (audit.get('recipe_count', 0), len(files))) + return dict(recipe_count=audit.get('recipe_count'), + published_variants=audit.get('published_variants'), + full_static_coverage_proven=audit.get('full_static_coverage_proven'), + profile_sha256=audit.get('profile_sha256'), + generation_inputs_sha256=audit.get('generation_inputs_sha256')) + + +# -------------------------------------------------------------------------- +# Staging +# -------------------------------------------------------------------------- +def copy(src, dst): + dst = Path(dst) + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + + +def stage_payload(ctx): + config, stage, exe_dir = ctx.config, ctx.stage, ctx.exe_dir + if stage.exists(): + shutil.rmtree(stage) + stage.mkdir(parents=True) + exe_name = (config.exe_name or ctx.built_exe.stem) + ctx.tc.exe_suffix + ctx.exe_name = exe_name + copy(ctx.built_exe, stage / exe_name) + if ctx.platform == 'linux': + os.chmod(stage / exe_name, 0o755) + copy(ctx.exe_dir / 'psx_game_version.txt', stage / 'psx_game_version.txt') + + for tree in ('assets/fonts', 'assets/img'): + if not (exe_dir / tree).is_dir(): + fail('%s missing: the recomp-ui launcher assets were not staged by the build' + % (exe_dir / tree)) + shutil.copytree(exe_dir / 'assets', stage / 'assets') + if config.launcher_boxart is not None: + staged = stage / 'assets/img/boxart.tga' + if not staged.is_file() or sha256(staged) != sha256(config.launcher_boxart): + fail('staged launcher boxart does not match %s' % config.launcher_boxart) + + bios, notice = exe_dir / 'bios/openbios.bin', exe_dir / 'bios/OpenBIOS.LICENSE' + if not bios.is_file() or bios.stat().st_size != OPENBIOS_SIZE or not notice.is_file(): + fail('the build did not stage OpenBIOS (%d bytes) and its MIT notice' % OPENBIOS_SIZE) + copy(bios, stage / 'bios/openbios.bin') + copy(notice, stage / 'bios/OpenBIOS.LICENSE') + + copy(config.player_game_toml, stage / 'game.toml') + options = config.root / 'game_options.toml' + if options.is_file(): + copy(options, stage / 'game_options.toml') + game = tomllib.loads(read_text(stage / 'game.toml')) + ctx.game_id = game.get('game', {}).get('id') + if not ctx.game_id: + fail('[game] id missing from %s' % config.player_game_toml) + + # Mod catalog: the build's own catalog list is the invariant. + run([sys.executable, TOOLS / 'release_stage.py', 'stage-mods', '--build-path', exe_dir, + '--stage', stage, '--runtime-target', config.runtime_target], 'mod catalog staging') + if ctx.channel == 'public': + run([sys.executable, TOOLS / 'mod_channel_filter.py', stage / 'mods/bundled'], + 'developer-channel pruning') + ctx.mod_packages = staged_mod_ids(stage) + if not ctx.mod_packages: + fail('staged mod catalog is empty') + check_diagnostics(ctx, stage) + + for rel in config.build_files: + if not (exe_dir / rel).is_file(): + fail('build file %s missing beside the executable' % rel) + copy(exe_dir / rel, stage / rel) + if config.diagnostics and config.diagnostics['replay_helper']: + helper = exe_dir / 'replay-diagnostics.py' + if not helper.is_file(): + fail('diagnostics replay helper missing beside the executable (%s)' % helper) + copy(helper, stage / 'replay-diagnostics.py') + + if config.aot_mode == 'cache': + run([sys.executable, TOOLS / 'aot_overlay_pipeline.py', 'release', + '--profile', config.aot_profile, '--game-toml', config.game_toml, + '--runtime-config', stage / 'game.toml', '--runtime-build-dir', ctx.runtime_build, + '--runtime-target', config.runtime_target, '--recompiler', ctx.recompiler, + '--work-dir', ctx.work / 'aot-cache', '--stage', stage, '--gcc', ctx.tc.gcc, + '--workers', ctx.jobs] + (['--cps'] if config.aot_cps else []), + 'original-disc AOT cache', cwd=ctx.root, env=ctx.tc.env) + if not (stage / 'AOT_CACHE_AUDIT.json').is_file(): + fail('AOT release staged no AOT_CACHE_AUDIT.json') + if config.ship_toolchain: + command = [sys.executable, TOOLS / 'release_stage.py', 'stage-toolchain', '--stage', stage, + '--recomp-dir', ctx.recompiler_dir, '--recomp-tools', TOOLS, + '--recomp-include', FRAMEWORK / 'runtime/include', + '--dl-cache', ctx.dl_cache, '--platform', TOOLCHAIN_PLATFORM[ctx.platform]] + if ctx.platform == 'windows': + command += ['--mingw-bin', ctx.tc.bin] + run(command, 'overlay toolchain staging', env=ctx.tc.env) + + licenses = stage / 'licenses' + licenses.mkdir(exist_ok=True) + for notice_file in sorted((FRAMEWORK / 'runtime/licenses').iterdir()): + if notice_file.is_file(): + copy(notice_file, licenses / notice_file.name) + copy(FRAMEWORK / 'LICENSE', licenses / 'psxrecomp-LICENSE') + if (FRAMEWORK / 'THIRD_PARTY_ATTRIBUTION.md').is_file(): + copy(FRAMEWORK / 'THIRD_PARTY_ATTRIBUTION.md', licenses / 'psxrecomp-THIRD_PARTY_ATTRIBUTION.md') + ui_root = Path(ctx.cache.get('RECOMP_UI_ROOT') or (config.root / 'recomp-ui')) + if not (ui_root / 'LICENSE').is_file(): + fail('recomp-ui LICENSE not found under %s' % ui_root) + copy(ui_root / 'LICENSE', licenses / 'recomp-ui-LICENSE') + for sub in ('fonts', 'img'): + notice_md = ui_root / 'assets/common' / sub / 'NOTICE.md' + if notice_md.is_file(): + copy(notice_md, stage / 'assets' / sub / 'NOTICE.md') + + for source, rel in config.files: + copy(source, stage / rel) + start_here = read_text(config.start_here) + if '@VERSION@' not in start_here: + fail('%s has no @VERSION@ token; the shipped header must be stamped, not hand-edited' + % config.start_here) + start_here = start_here.replace('@VERSION@', ctx.version) + (stage / 'START_HERE.txt').write_text(start_here, encoding='utf-8', newline='\n') + run_hooks(ctx, 'post_stage', stage=str(stage)) + + +def staged_mod_ids(stage): + bundled = Path(stage) / 'mods/bundled' + ids = [] + for manifest in sorted(bundled.glob('*/*/manifest.toml')): + ids.append(tomllib.loads(read_text(manifest)).get('id', manifest.parent.parent.name)) + return sorted(set(ids)) + + +def diagnostics_manifests(mods_root): + found = {} + for manifest in sorted(Path(mods_root).glob('*/*/manifest.toml')): + data = tomllib.loads(read_text(manifest)) + features = {f.get('id') for f in data.get('feature', [])} + if {'recording', 'osd'} <= features and str(data.get('id', '')).endswith('.diagnostics'): + found.setdefault(data['id'], []).append((manifest, data)) + return found + + +def check_diagnostics(ctx, stage): + """Diagnostics defaults follow the channel; a diagnostics package the + config does not declare is a config error, not something to skip.""" + found = diagnostics_manifests(Path(stage) / 'mods/bundled') + declared = ctx.config.diagnostics + if not declared: + if found: + fail('staged catalog carries %s but the release config has no [diagnostics] table' + % ', '.join(sorted(found))) + return + entries = found.get(declared['package_id']) + if not entries: + fail('staged catalog is missing %s' % declared['package_id']) + want = ctx.channel == 'playtest' + for manifest, data in entries: + if data.get('channel') == 'developer': + fail('%s must stay player-selectable' % manifest) + features = {f.get('id'): f for f in data.get('feature', [])} + for feature in ('recording', 'osd'): + got = features[feature].get('default_enabled') + if got is not want or features[feature].get('channel') == 'developer': + fail('%s: %s default_enabled=%r, a %s package needs %r' + % (manifest, feature, got, ctx.channel, want)) + log('Playtest Diagnostics default %s (%s)' % ('on' if want else 'off', ctx.channel)) + + +def payload_gates(stage, plat, exe_name): + """Content that must never ship, whatever the title config says.""" + stage = Path(stage) + present = [name for name in BUILD_KIT if (stage / name).exists()] + if present: + fail('package contains build-kit/user content: %s' % ', '.join(present)) + for path in sorted(stage.rglob('*')): + if path.is_symlink(): + target = (path.parent / os.readlink(path)).resolve() + if stage.resolve() not in target.parents: + fail('symlink leaving the package: %s' % path.relative_to(stage)) + continue + if not path.is_file(): + continue + rel = path.relative_to(stage).as_posix() + lower, leaf = rel.lower(), path.name.lower() + suffix = path.suffix.lower() + if suffix in DISC_SUFFIXES: + fail('disc/save data in package: %s' % rel) + if suffix in ('.bin', '.rom') and lower != 'bios/openbios.bin': + fail('binary image in package (only bios/openbios.bin may ship): %s' % rel) + if re.match(r'scph.*\.(bin|rom)', leaf): + fail('retail BIOS image in package: %s' % rel) + if leaf in PERSONAL_NAMES or lower.startswith(('mods/installed/', 'saves/', 'diagnostics/')): + fail('personal/user state in package: %s' % rel) + if suffix in ('.c', '.cpp', '.cc') and not lower.startswith('overlay_toolchain/'): + fail('source code in package: %s' % rel) + if plat == 'linux' and suffix in ('.dll', '.exe'): + fail('Windows binary in a Linux package: %s' % rel) + bios = stage / 'bios/openbios.bin' + if not bios.is_file() or bios.stat().st_size != OPENBIOS_SIZE: + fail('package lacks the bundled OpenBIOS') + if not (stage / exe_name).is_file(): + fail('package lacks its executable %s' % exe_name) + + +def windows_imports(exe, objdump): + text = run([objdump, '-p', exe], 'objdump imports', capture=True) + imports = re.findall(r'DLL Name:\s*(\S+)', text) + if not imports: + fail('objdump found no imports in %s' % exe) + foreign = [d for d in imports if d.lower().removesuffix('.dll') not in SYSTEM_DLLS + and not re.match(r'api-ms-win-[a-z0-9-]+\.dll$', d.lower())] + if foreign: + fail('%s is not self-contained: imports %s' % (Path(exe).name, ', '.join(foreign))) + return imports + + +def sign_windows(ctx): + if not (os.environ.get('WINDOWS_SIGN_PFX') or os.environ.get('WINDOWS_SIGN_PFX_BASE64')): + log('Windows binaries ship unsigned (no WINDOWS_SIGN_PFX certificate configured)') + ctx.signed = False + return + bash = shutil.which('bash') + if not bash: + fail('a signing certificate is configured but bash is not available to run sign_windows.sh') + run([bash, TOOLS / 'ci/sign_windows.sh', ctx.stage], 'Authenticode signing') + ctx.signed = True + + +def bind_execution(ctx, binary, output): + sidecar = ctx.built_exe.with_name(ctx.built_exe.stem + '.execution.json') + if not sidecar.is_file(): + fail('build published no execution contract %s' % sidecar) + run([sys.executable, TOOLS / 'release_stage.py', 'stage-execution', '--binary', binary, + '--manifest', sidecar, '--output', output], 'execution contract binding') + + +def write_manifest(ctx, extra): + stage = ctx.stage + manifest = dict( + schema=1, product=ctx.config.name, display_name=ctx.config.display_name, + version=ctx.version, channel=ctx.channel, public_release=ctx.channel == 'public', + platform=ctx.platform, game_id=ctx.game_id, executable=ctx.exe_name, + source_commit=ctx.prov['source_commit'], source_dirty=ctx.prov['source_dirty'], + dirty_paths=ctx.prov['dirty_paths'], framework_commit=ctx.prov['framework_commit'], + recomp_ui_commit=ctx.prov['recomp_ui_commit'], debug_tools=False, + bios_backends=['OpenBIOS'], setup_wizard=cmake_bool(ctx.cache.get('PSX_SETUP_WIZARD', 'OFF')), + playtest_diagnostics_default_enabled=(ctx.channel == 'playtest') if ctx.config.diagnostics else None, + mod_packages=ctx.mod_packages, aot_mode=ctx.config.aot_mode, + overlay_toolchain=ctx.config.ship_toolchain, signed=getattr(ctx, 'signed', False), + packager='psxrecomp/tools/package_title_release.py') + if getattr(ctx, 'static_audit', None): + manifest['static_aot'] = ctx.static_audit + manifest.update(extra) + files = {} + for path in sorted(stage.rglob('*')): + if path.is_file() and path.name != 'RELEASE_MANIFEST.json': + files[path.relative_to(stage).as_posix()] = sha256(path) + manifest['files'] = files + (stage / 'RELEASE_MANIFEST.json').write_text(json.dumps(manifest, indent=2) + '\n', + encoding='utf-8', newline='\n') + return manifest + + +def write_sha256(artifact): + sidecar = Path(str(artifact) + '.sha256') + sidecar.write_text('%s %s\n' % (sha256(artifact), Path(artifact).name), encoding='ascii', + newline='\n') + return sidecar + + +def diagnostics_audit(ctx, target): + diag = ctx.config.diagnostics + if not diag: + return + command = [sys.executable, TOOLS / 'ci/audit_playtest_diagnostics_package.py', target, + '--game-id', diag['game_id'], '--diagnostics-id', diag['package_id'], + '--mode', ctx.channel] + if not diag['replay_helper']: + command.append('--no-replay-helper') + run(command, 'Playtest Diagnostics package audit (%s)' % ctx.channel) + + +def package_windows(ctx): + payload_gates(ctx.stage, 'windows', ctx.exe_name) + imports = windows_imports(ctx.stage / ctx.exe_name, ctx.tc.objdump) + log('Verified self-contained: %d imports, all system DLLs' % len(imports)) + sign_windows(ctx) + bind_execution(ctx, ctx.stage / ctx.exe_name, + ctx.stage / (Path(ctx.exe_name).stem + '.execution.json')) + write_manifest(ctx, dict(system_dll_imports=imports)) + artifact = ctx.out_dir / artifact_name(ctx.config, ctx.version, ctx.channel, 'windows') + for stale in (artifact, Path(str(artifact) + '.sha256')): + if stale.exists(): + stale.unlink() + zip_source = ctx.stage + if ctx.config.zip_folder: + zip_source = ctx.work / 'zip-root' + if zip_source.exists(): + shutil.rmtree(zip_source) + zip_source.mkdir(parents=True) + shutil.copytree(ctx.stage, zip_source / ctx.config.zip_folder) + run([sys.executable, TOOLS / 'create_release_zip.py', '--source', zip_source, + '--output', artifact], 'ZIP creation') + diagnostics_audit(ctx, artifact) + write_sha256(artifact) + return artifact + + +def render_template(path, tokens): + text = read_text(path) + for key, value in tokens.items(): + text = text.replace('@%s@' % key, value) + left = sorted(set(re.findall(r'@[A-Z][A-Z0-9_]*@', text))) + if left: + fail('%s has unstamped token(s) %s' % (path, ', '.join(left))) + return text + + +def package_linux(ctx): + linux = ctx.config.linux + if not linux: + fail('%s has no [linux] table (AppRun, desktop file, icon, payload_dir)' % ctx.config.path) + payload_gates(ctx.stage, 'linux', ctx.exe_name) + diagnostics_audit(ctx, ctx.stage) + bind_execution(ctx, ctx.stage / ctx.exe_name, ctx.stage / (ctx.exe_name + '.execution.json')) + libc = run(['getconf', 'GNU_LIBC_VERSION'], 'glibc version', capture=True).strip() + write_manifest(ctx, dict(linux_build_libc=libc)) + meta = ctx.work / 'appimage-meta' + if meta.exists(): + shutil.rmtree(meta) + meta.mkdir(parents=True) + tokens = dict(VERSION=ctx.version, APP_NAME=ctx.config.display_name, EXE_NAME=ctx.exe_name, + PAYLOAD_DIR=linux['payload_dir'], ENV_PREFIX=linux['env_prefix'], + ARTIFACT_NAME=ctx.config.name) + app_run = meta / 'AppRun' + app_run.write_text(render_template(linux['app_run'], tokens), encoding='utf-8', newline='\n') + desktop = meta / linux['desktop_file'].name + desktop.write_text(render_template(linux['desktop_file'], tokens), encoding='utf-8', newline='\n') + icon = meta / (desktop.stem + '.png') + convert = shutil.which('magick') or shutil.which('convert') + if linux['icon'].suffix.lower() == '.png': + copy(linux['icon'], icon) + elif convert: + run([convert, linux['icon'], '-resize', '240x240', '-background', 'transparent', + '-gravity', 'center', '-extent', '256x256', icon], 'icon conversion') + else: + fail('ImageMagick is required to convert %s to the AppImage PNG icon' % linux['icon']) + artifact = ctx.out_dir / artifact_name(ctx.config, ctx.version, ctx.channel, 'linux') + for stale in (artifact, Path(str(artifact) + '.sha256')): + if stale.exists() or stale.is_symlink(): + stale.unlink() + env = dict(ctx.tc.env, SOURCE_DATE_EPOCH=str(ctx.source_date_epoch), + PSX_RELEASE_STAGE_PYTHON=sys.executable) + command = ['bash', TOOLS / 'package_appimage.sh', '--payload', ctx.stage, + '--exe-name', ctx.exe_name, '--payload-name', linux['payload_dir'], + '--app-run', app_run, '--desktop-file', desktop, '--icon', icon, + '--output', artifact, '--jobs', ctx.jobs] + if linux['prune_libs']: + command.append('--prune-unreferenced-libs') + run(command, 'AppImage assembly', env=env) + write_sha256(artifact) + return artifact + + +# -------------------------------------------------------------------------- +# Verification of a finished artifact (also the `verify` subcommand) +# -------------------------------------------------------------------------- +def verify_payload(payload, exe_path, config, channel, plat, version): + payload = Path(payload) + manifest_path = payload / 'RELEASE_MANIFEST.json' + if not manifest_path.is_file(): + fail('artifact has no RELEASE_MANIFEST.json') + manifest = json.loads(read_text(manifest_path)) + problems = [] + if manifest.get('channel') != channel: + problems.append('manifest channel %r != %s' % (manifest.get('channel'), channel)) + if manifest.get('version') != version: + problems.append('manifest version %r != %s' % (manifest.get('version'), version)) + if read_text(payload / 'psx_game_version.txt').strip() != version: + problems.append('psx_game_version.txt does not say %s' % version) + start = read_text(payload / 'START_HERE.txt') + if version not in start or '@VERSION@' in start: + problems.append('START_HERE.txt is not stamped with %s' % version) + staged = staged_mod_ids(payload) + if staged != manifest.get('mod_packages'): + problems.append('mod catalog %s != manifest %s' % (staged, manifest.get('mod_packages'))) + found = diagnostics_manifests(payload / 'mods/bundled') + if config.diagnostics: + want = channel == 'playtest' + entries = found.get(config.diagnostics['package_id'], []) + if not entries: + problems.append('diagnostics package missing') + for manifest_file, data in entries: + features = {f.get('id'): f for f in data.get('feature', [])} + for feature in ('recording', 'osd'): + if features[feature].get('default_enabled') is not want: + problems.append('%s %s default_enabled is not %r' % (manifest_file, feature, want)) + elif found: + problems.append('undeclared diagnostics package(s) %s' % sorted(found)) + files = manifest.get('files', {}) + for rel, digest in files.items(): + path = payload / rel + if plat == 'linux' and rel in (manifest.get('executable'), + '%s.execution.json' % manifest.get('executable')): + continue # moved to usr/bin, possibly re-linked; re-bound by its execution contract + if not path.is_file(): + problems.append('manifest file missing: %s' % rel) + elif sha256(path) != digest: + problems.append('manifest hash mismatch: %s' % rel) + if problems: + fail('verification failed:\n ' + '\n '.join(problems)) + exe_name = manifest['executable'] + # Re-run the content gates over exactly what a player receives (the + # payload is a private extraction, so the AppImage's ELF can join it). + if plat == 'linux': + copy(exe_path, payload / exe_name) + payload_gates(payload, plat, exe_name) + return manifest + + +def verify_artifact(artifact, config, channel, version, objdump=None, scratch=None): + """Extract into a private directory under `scratch` (default: beside the + artifact; never the system temp drive on Windows) and check everything.""" + artifact = Path(artifact) + plat = 'windows' if artifact.name.endswith('.zip') else 'linux' + expected = artifact_name(config, version, channel, plat) + if artifact.name != expected: + fail('artifact name %s != expected %s' % (artifact.name, expected)) + sidecar = Path(str(artifact) + '.sha256') + if not sidecar.is_file() or read_text(sidecar).split()[0] != sha256(artifact): + fail('%s missing or does not match' % sidecar.name) + if scratch is None: + scratch = tempfile.gettempdir() if host_platform() == 'linux' else artifact.parent + work = Path(tempfile.mkdtemp(prefix='.psx-release-verify-', dir=str(scratch))) + try: + if plat == 'windows': + with zipfile.ZipFile(artifact) as archive: + for info in archive.infolist(): + name = info.filename + if '\\' in name or name.startswith('/') or '..' in PurePosixPath(name).parts: + fail('unsafe/non-portable ZIP entry %r' % name) + archive.extractall(work) + payload = work / config.zip_folder if config.zip_folder else work + if config.zip_folder and sorted(p.name for p in work.iterdir()) != [config.zip_folder]: + fail('ZIP must hold exactly one top-level folder %s' % config.zip_folder) + manifest = verify_payload(payload, None, config, channel, 'windows', version) + exe = payload / manifest['executable'] + if objdump: + windows_imports(exe, objdump) + layout = 'zip' + else: + if host_platform() != 'linux': + fail('verify an AppImage on Linux (or through WSL)') + os.chmod(artifact, 0o755) + run([artifact, '--appimage-extract'], 'AppImage extraction', cwd=work, capture=True) + root = work / 'squashfs-root' + linux = config.linux + payload = root / 'usr/share' / linux['payload_dir'] + manifest = json.loads(read_text(payload / 'RELEASE_MANIFEST.json')) + exe = root / 'usr/bin' / manifest['executable'] + for need, what in ((root / 'AppRun', 'AppRun'), (exe, 'usr/bin executable'), + (payload / 'game.toml', 'payload game.toml')): + if not need.is_file(): + fail('AppImage layout: %s missing' % what) + if not os.access(root / 'AppRun', os.X_OK) or not os.access(exe, os.X_OK): + fail('AppImage layout: AppRun/executable not executable') + if exe.read_bytes()[:4] != b'\x7fELF': + fail('AppImage executable is not ELF') + if not list(root.glob('*.desktop')) or not (root / '.DirIcon').exists(): + fail('AppImage layout: desktop entry or .DirIcon missing') + if not (root / 'usr/bin/assets').exists(): + fail('AppImage layout: usr/bin/assets link missing') + missing = run(['env', '-u', 'LD_LIBRARY_PATH', 'ldd', exe], 'ldd', capture=True) + if 'not found' in missing: + fail('AppImage executable has unresolved libraries:\n' + missing) + verify_payload(payload, exe, config, channel, 'linux', version) + layout = 'appimage' + log('VERIFIED %s: %s %s, %d mod packages, diagnostics %s, OpenBIOS only (%s)' + % (artifact.name, manifest['product'], manifest['version'], len(manifest['mod_packages']), + {True: 'on', False: 'off', None: 'n/a'}[manifest['playtest_diagnostics_default_enabled']], + layout)) + return manifest + finally: + shutil.rmtree(work, ignore_errors=True) + + +# -------------------------------------------------------------------------- +# Driver +# -------------------------------------------------------------------------- +def to_wsl_path(path): + path = str(Path(path).resolve()) + match = re.match(r'^([A-Za-z]):[\\/](.*)$', path) + if not match: + fail('cannot map %s into WSL (only drive paths are supported)' % path) + return '/mnt/%s/%s' % (match.group(1).lower(), match.group(2).replace('\\', '/')) + + +def run_in_wsl(args, ctx): + """Run this script's Linux half inside WSL on the same checkout.""" + wsl = shutil.which('wsl.exe') or shutil.which('wsl') + if not wsl: + fail('--platform linux on Windows needs WSL (wsl.exe not found)') + argv = ['python3', to_wsl_path(Path(__file__)), 'package', '--channel', args.channel, + '--platform', 'linux', '--root', to_wsl_path(ctx.root), + '--config', to_wsl_path(ctx.config.path), '--jobs', str(ctx.jobs), + '--out-dir', to_wsl_path(ctx.out_dir), '--sources-prepared'] + if args.work_dir: + argv += ['--work-dir', to_wsl_path(Path(args.work_dir))] + if args.allow_dirty: + argv.append('--allow-dirty') + env = {'PSX_RELEASE_SOURCE_COMMIT': ctx.prov['source_commit'], + 'PSX_RELEASE_SOURCE_DIRTY': '1' if ctx.prov['source_dirty'] else '0', + 'PSX_RELEASE_DIRTY_PATHS': json.dumps(ctx.prov['dirty_paths']), + 'PSX_RELEASE_FRAMEWORK_COMMIT': ctx.prov['framework_commit'], + 'PSX_RELEASE_UI_COMMIT': ctx.prov['recomp_ui_commit'] or '', + 'PSX_RELEASE_SOURCE_DATE_EPOCH': str(ctx.prov['source_date_epoch'])} + script = 'set -e; ' + ' '.join('export %s=%s;' % (k, shlex.quote(v)) for k, v in env.items()) + script += ' exec ' + ' '.join(shlex.quote(a) for a in argv) + run([wsl, '-e', 'bash', '-c', script], 'Linux packaging in WSL') + + +def make_context(args, plat): + ctx = Context() + ctx.root = Path(args.root).resolve() + ctx.config = Config(Path(args.config) if Path(args.config).is_absolute() + else ctx.root / args.config, ctx.root) + ctx.channel = args.channel + ctx.platform = plat + ctx.version = ctx.config.version() + ctx.jobs = args.jobs + ctx.out_dir = Path(args.out_dir).resolve() if args.out_dir else ctx.root / 'dist' + ctx.out_dir.mkdir(parents=True, exist_ok=True) + base = Path(args.work_dir).resolve() if args.work_dir else ctx.root / 'build-package' + ctx.work = base / plat + ctx.work.mkdir(parents=True, exist_ok=True) + ctx.runtime_build = ctx.work / 'runtime' + ctx.stage = ctx.work / 'stage' + ctx.dl_cache = base / 'dl-cache' + return ctx + + +def package(args): + host = host_platform() + platforms = list(PLATFORMS) if args.platform == 'all' else [args.platform or host] + if host == 'linux' and 'windows' in platforms: + fail('Windows packages are built on a Windows host') + artifacts = [] + for plat in platforms: + ctx = make_context(args, plat) + ui_root = ctx.root / 'recomp-ui' + ctx.prov = provenance(ctx.root, ui_root) + if ctx.prov['source_dirty'] and not args.allow_dirty: + fail('tracked changes in the release source (%s); commit them, or pass --allow-dirty ' + 'for a non-publishable test package' % ', '.join(ctx.prov['dirty_paths'][:8])) + log('== %s %s | channel %s | platform %s | source %s%s' + % (ctx.config.name, ctx.version, ctx.channel, plat, ctx.prov['source_commit'][:12], + ' (DIRTY)' if ctx.prov['source_dirty'] else '')) + if plat == 'linux' and host == 'windows': + # Prepare once with this host's emitters; Linux compiles the same sources. + ctx.tc = Toolchain('windows', args.toolchain_bin) + ctx.work = ctx.work.parent / 'windows' + ctx.runtime_build = ctx.work / 'runtime' + build_emitters(ctx) + ensure_openbios(ctx) + prepare_sources(ctx) + run_in_wsl(args, ctx) + artifacts.append(ctx.out_dir / artifact_name(ctx.config, ctx.version, ctx.channel, 'linux')) + continue + ctx.tc = Toolchain(plat, args.toolchain_bin) + scratch = None + if plat == 'linux': + if hasattr(os, 'nice'): + try: + os.nice(10) # packaging must not make the machine unusable + except OSError: + pass + # The payload, AppDir and verification extraction need a native + # Linux filesystem (symlinks, exec bits); a /mnt/ DrvFs tree + # cannot hold them. Removed when this platform finishes. + scratch = Path(tempfile.mkdtemp(prefix='psx-title-release.', dir='/tmp')) + fs = run(['stat', '-f', '-c', '%T', scratch], 'stat', capture=True).strip() + if fs in ('9p', 'drvfs') or fs.startswith('ntfs'): + fail('/tmp is not a native Linux filesystem (%s)' % fs) + ctx.stage = scratch / 'stage' + ctx.tc.env['TMPDIR'] = str(scratch) + tempfile.tempdir = str(scratch) + try: + artifacts.append(package_platform(ctx, args)) + finally: + if scratch is not None: + tempfile.tempdir = None + shutil.rmtree(scratch, ignore_errors=True) + for artifact in artifacts: + if not artifact.is_file(): + fail('expected artifact was not produced: %s' % artifact) + log('PACKAGED %s' % artifact) + return 0 + + +def package_platform(ctx, args): + plat = ctx.platform + epoch = os.environ.get('SOURCE_DATE_EPOCH') + ctx.source_date_epoch = (int(epoch) if epoch and epoch.isdigit() + else ctx.prov['source_date_epoch']) + needs_emitters = (not args.sources_prepared or ctx.config.ship_toolchain or + ctx.config.aot_mode == 'cache') + if needs_emitters: + build_emitters(ctx) + ensure_openbios(ctx) + if not args.sources_prepared: + prepare_sources(ctx) + configure_and_build(ctx) + check_build_outputs(ctx) + stage_payload(ctx) + artifact = package_windows(ctx) if plat == 'windows' else package_linux(ctx) + verify_artifact(artifact, ctx.config, ctx.channel, ctx.version, + objdump=ctx.tc.objdump if plat == 'windows' else None, + scratch=ctx.work if plat == 'windows' else None) + return artifact + + +def verify(args): + root = Path(args.root).resolve() + config = Config(Path(args.config) if Path(args.config).is_absolute() else root / args.config, root) + objdump = None + if os.name == 'nt': + candidate = Path(args.toolchain_bin) / 'objdump.exe' + objdump = str(candidate) if candidate.is_file() else shutil.which('objdump') + for artifact in args.artifacts: + verify_artifact(artifact, config, args.channel, config.version(), objdump=objdump) + return 0 + + +def check_diagnostics_dir(args): + """Every *.diagnostics package in a staged mods/bundled follows the channel. + For packagers that have no release config (package_game_release.sh).""" + found = diagnostics_manifests(args.bundled) + want = args.channel == 'playtest' + for package, entries in sorted(found.items()): + for manifest, data in entries: + features = {f.get('id'): f for f in data.get('feature', [])} + for feature in ('recording', 'osd'): + got = features[feature].get('default_enabled') + if got is not want or features[feature].get('channel') == 'developer': + fail('%s: %s default_enabled=%r, a %s package needs %r' + % (manifest, feature, got, args.channel, want)) + log('%s: Playtest Diagnostics default %s (%s)' % (package, 'on' if want else 'off', + args.channel)) + if not found: + log('no Playtest Diagnostics package in %s' % args.bundled) + return 0 + + +def parse_args(argv): + parser = argparse.ArgumentParser( + prog='package_title_release.py', description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest='command') + + def common(p): + p.add_argument('--channel', required=True, choices=CHANNELS, + help='REQUIRED, no default: public or playtest') + p.add_argument('--config', default='packaging/release.toml', + help='title release config, relative to --root') + p.add_argument('--root', default='.', help='title repository root (default: cwd)') + p.add_argument('--toolchain-bin', default=r'C:\msys64\mingw64\bin', + help='Windows: MinGW-w64 bin with cmake/ninja/gcc/objdump') + + pkg = sub.add_parser('package', help='build, stage, package and verify (default)') + common(pkg) + pkg.add_argument('--platform', choices=PLATFORMS + ('all',), + help='default: this host; on Windows "linux"/"all" use WSL') + cores = os.cpu_count() or 4 + pkg.add_argument('--jobs', type=int, default=max(1, min(6, cores - 2)), + help='parallel jobs (default min(6, cores-2))') + pkg.add_argument('--out-dir', help='artifact directory (default /dist)') + pkg.add_argument('--work-dir', help='build/stage directory (default /build-package)') + pkg.add_argument('--allow-dirty', action='store_true', + help='package uncommitted tracked changes (recorded; not for publication)') + pkg.add_argument('--sources-prepared', action='store_true', help=argparse.SUPPRESS) + + ver = sub.add_parser('verify', help='extract and verify finished artifacts') + common(ver) + ver.add_argument('artifacts', nargs='+') + + chk = sub.add_parser('check-diagnostics', + help='check a staged mods/bundled against the channel (no config)') + chk.add_argument('--channel', required=True, choices=CHANNELS) + chk.add_argument('bundled', type=Path) + + if not argv or argv[0] not in ('package', 'verify', 'check-diagnostics', '-h', '--help'): + argv = ['package'] + list(argv) + args = parser.parse_args(argv) + if args.command == 'package' and args.jobs < 1: + parser.error('--jobs must be positive') + return args + + +def main(argv=None): + argv = sys.argv[1:] if argv is None else argv + try: + args = parse_args(argv) + except SystemExit as exit_: + if exit_.code not in (0, None): + print('\nPACKAGING FAILED: invalid arguments (--channel public|playtest is required)', + file=sys.stderr, flush=True) + raise + try: + if args.command == 'check-diagnostics': + return check_diagnostics_dir(args) + return package(args) if args.command == 'package' else verify(args) + except PackagingError as error: + print('\nPACKAGING FAILED: %s' % error, file=sys.stderr, flush=True) + return 1 + except KeyboardInterrupt: + print('\nPACKAGING FAILED: interrupted', file=sys.stderr, flush=True) + return 130 + except Exception as error: # an unexpected bug is still a loud failure + import traceback + traceback.print_exc() + print('\nPACKAGING FAILED: %s: %s' % (type(error).__name__, error), file=sys.stderr, flush=True) + return 1 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/package_title_release.sh b/tools/package_title_release.sh new file mode 100755 index 00000000..9eba7e9d --- /dev/null +++ b/tools/package_title_release.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Linux/macOS-shell launcher for tools/package_title_release.py (the one release +# command). Run from the title repository root: +# psxrecomp/tools/package_title_release.sh --channel public|playtest +# Arguments pass through unchanged; --channel is required (no default). +set -euo pipefail +python="${PSX_RELEASE_PYTHON:-python3}" +command -v "$python" >/dev/null 2>&1 || { + printf '\nPACKAGING FAILED: %s not found (Python 3.11+ required)\n' "$python" >&2; exit 1; +} +exec "$python" "$(cd "$(dirname "$0")" && pwd)/package_title_release.py" "$@" diff --git a/tools/tests/test_audit_playtest_diagnostics_package.py b/tools/tests/test_audit_playtest_diagnostics_package.py index 6df148f4..c3ff7235 100644 --- a/tools/tests/test_audit_playtest_diagnostics_package.py +++ b/tools/tests/test_audit_playtest_diagnostics_package.py @@ -186,6 +186,34 @@ class ArchiveTests(unittest.TestCase): self.check() self.files[name] = content + def test_title_recorder_without_replay_helper(self): + # A title with its own recorder ships no replay-diagnostics.py. + del self.files["replay-diagnostics.py"] + self.write() + with self.assertRaises(ValueError): + audit.audit(self.zip, GAME, DIAG, True) + self.assertEqual(audit.audit(self.zip, GAME, DIAG, True, replay_helper=False), 1) + del self.files["Game.exe"] + self.write() + with self.assertRaisesRegex(ValueError, "no executable at the package root"): + audit.audit(self.zip, GAME, DIAG, True, replay_helper=False) + + def test_staged_directory_is_audited_like_a_zip(self): + stage = Path(self.tmp.name) / "stage" + for name, content in self.files.items(): + path = stage / name + path.parent.mkdir(parents=True, exist_ok=True) + if isinstance(content, str): + content = content.encode() + path.write_bytes(content) + self.assertEqual(audit.audit(stage, GAME, DIAG, True), 1) + with self.assertRaises(ValueError): + audit.audit(stage, GAME, DIAG, False) + (stage / "saves").mkdir() + (stage / "saves/card1.mcd").write_bytes(b"card") + with self.assertRaisesRegex(ValueError, "personal state"): + audit.audit(stage, GAME, DIAG, True) + def test_wrong_game_recipe_and_manifest_target(self): self.files["game.toml"] = '[game]\nid="WRONG"\n' with self.assertRaisesRegex(ValueError, "game ID"): diff --git a/tools/tests/test_package_title_release.py b/tools/tests/test_package_title_release.py new file mode 100644 index 00000000..a39c2b8f --- /dev/null +++ b/tools/tests/test_package_title_release.py @@ -0,0 +1,333 @@ +#!/usr/bin/env python3 +"""Unit tests for tools/package_title_release.py (no build, no disc, no game). + +Pins the release contract every title now shares: --channel is required with +no default, owned CMake flags cannot be overridden by a title, artifact names +follow the channel, forbidden content fails the payload gates, diagnostics +defaults follow the channel, and a finished ZIP is re-verified by extraction. +""" +import contextlib +import hashlib +import io +import json +from pathlib import Path +import shutil +import sys +import tempfile +import unittest +import zipfile + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import package_title_release as ptr # noqa: E402 + +DIAG = '''format_version = 5 +id = "fixture.diagnostics" +version = "1.0.0" +name = "Playtest Diagnostics" +channel = "stable" + +[[target]] +game_id = "SLUS-00000" + +[[feature]] +id = "recording" +default_enabled = {on} + +[[feature]] +id = "osd" +default_enabled = {on} + +[[plugin]] +feature = "recording" +id = "fixture.diagnostics" + +[[plugin]] +feature = "osd" +id = "fixture.diagnostics.osd" +''' + +CONFIG = '''schema = 1 + +[product] +name = "FixtureRecomp" +display_name = "Fixture Recompiled" +exe_name = "Fixture" +version_file = "VERSION" + +[build] +cmake_args = ["-DPSX_PGXP_VARIANT=OFF"] + +[diagnostics] +game_id = "SLUS-00000" +package_id = "fixture.diagnostics" + +[payload] +start_here = "START_HERE.txt" +files = ["README.md", {{ from = "notes/input.ini", to = "input.ini" }}] +{extra} +''' + + +class Fixture: + def __init__(self, extra=''): + self.tmp = tempfile.TemporaryDirectory() + self.root = Path(self.tmp.name) + (self.root / 'VERSION').write_text('v1.2.3\n') + (self.root / 'START_HERE.txt').write_text('Fixture @VERSION@\n') + (self.root / 'README.md').write_text('readme\n') + (self.root / 'game.toml').write_text('[game]\nid = "SLUS-00000"\n') + (self.root / 'notes').mkdir() + (self.root / 'notes/input.ini').write_text('[controller]\n') + (self.root / 'release.toml').write_text(CONFIG.format(extra=extra)) + + def config(self): + return ptr.Config(self.root / 'release.toml', self.root) + + def close(self): + self.tmp.cleanup() + + +def write_stage(stage, channel, version='v1.2.3', exe='Fixture.exe'): + stage.mkdir(parents=True) + (stage / exe).write_bytes(b'MZ fixture') + (stage / 'psx_game_version.txt').write_text(version + '\n') + (stage / 'START_HERE.txt').write_text('Fixture %s\n' % version) + (stage / 'game.toml').write_text('[game]\nid = "SLUS-00000"\n') + (stage / 'replay-diagnostics.py').write_text('# helper\n') + (stage / 'bios').mkdir() + (stage / 'bios/openbios.bin').write_bytes(b'\0' * ptr.OPENBIOS_SIZE) + package = stage / 'mods/bundled/fixture.diagnostics/1.0.0' + package.mkdir(parents=True) + (package / 'manifest.toml').write_text( + DIAG.format(on='true' if channel == 'playtest' else 'false')) + files = {p.relative_to(stage).as_posix(): ptr.sha256(p) + for p in sorted(stage.rglob('*')) if p.is_file()} + manifest = dict(channel=channel, version=version, product='FixtureRecomp', executable=exe, + mod_packages=['fixture.diagnostics'], + playtest_diagnostics_default_enabled=channel == 'playtest', files=files) + (stage / 'RELEASE_MANIFEST.json').write_text(json.dumps(manifest)) + + +def zip_stage(stage, artifact): + with zipfile.ZipFile(artifact, 'w') as archive: + for path in sorted(stage.rglob('*')): + if path.is_file(): + archive.write(path, path.relative_to(stage).as_posix()) + Path(str(artifact) + '.sha256').write_text( + '%s %s\n' % (hashlib.sha256(artifact.read_bytes()).hexdigest(), artifact.name)) + + +class ChannelIsRequired(unittest.TestCase): + def run_main(self, argv): + err = io.StringIO() + with contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as raised: + ptr.main(argv) + return raised.exception.code, err.getvalue() + + def test_missing_channel_fails_loudly(self): + code, err = self.run_main(['--platform', 'windows']) + self.assertNotEqual(code, 0) + self.assertIn('PACKAGING FAILED', err) + self.assertIn('--channel', err) + + def test_unknown_channel_fails(self): + code, err = self.run_main(['--channel', 'release']) + self.assertNotEqual(code, 0) + self.assertIn('PACKAGING FAILED', err) + + def test_verify_also_requires_channel(self): + code, _ = self.run_main(['verify', 'x.zip']) + self.assertNotEqual(code, 0) + + def test_default_subcommand_is_package(self): + args = ptr.parse_args(['--channel', 'playtest']) + self.assertEqual((args.command, args.channel), ('package', 'playtest')) + + +class ConfigTests(unittest.TestCase): + def setUp(self): + self.fixture = Fixture() + self.addCleanup(self.fixture.close) + + def test_loads(self): + config = self.fixture.config() + self.assertEqual(config.version(), 'v1.2.3') + self.assertEqual([rel for _, rel in config.files], ['README.md', 'input.ini']) + self.assertEqual(config.aot_mode, 'none') + self.assertTrue(config.diagnostics['replay_helper']) + + def test_owned_cmake_flags_are_refused(self): + for flag in ('PSX_PUBLIC_RELEASE', 'PSX_RELEASE_CHANNEL', 'PSX_DEBUG_TOOLS', + 'PSXRECOMP_BIOS_STEMS', 'CMAKE_BUILD_TYPE'): + text = CONFIG.format(extra='').replace( + '"-DPSX_PGXP_VARIANT=OFF"', '"-D%s=ON"' % flag) + (self.fixture.root / 'release.toml').write_text(text) + with self.assertRaisesRegex(ptr.PackagingError, 'packager owns'): + self.fixture.config() + + def test_unknown_key_is_an_error(self): + (self.fixture.root / 'release.toml').write_text( + CONFIG.format(extra='').replace('[build]', '[build]\nskip_audit = true')) + with self.assertRaisesRegex(ptr.PackagingError, 'unknown key'): + self.fixture.config() + + def test_missing_payload_file_is_an_error(self): + (self.fixture.root / 'README.md').unlink() + with self.assertRaisesRegex(ptr.PackagingError, 'not found'): + self.fixture.config() + + def test_package_path_cannot_escape(self): + (self.fixture.root / 'release.toml').write_text( + CONFIG.format(extra='').replace('to = "input.ini"', 'to = "../input.ini"')) + with self.assertRaisesRegex(ptr.PackagingError, 'relative path'): + self.fixture.config() + + +class NamingTests(unittest.TestCase): + def test_artifact_names_follow_channel(self): + fixture = Fixture() + self.addCleanup(fixture.close) + config = fixture.config() + self.assertEqual(ptr.artifact_name(config, 'v1.2.3', 'public', 'windows'), + 'FixtureRecomp-v1.2.3-windows-x64.zip') + self.assertEqual(ptr.artifact_name(config, '1.2.3', 'playtest', 'windows'), + 'FixtureRecomp-v1.2.3-playtest-windows-x64.zip') + self.assertEqual(ptr.artifact_name(config, '0.2.4-alpha', 'playtest', 'linux'), + 'FixtureRecomp-v0.2.4-alpha-playtest-linux-x86_64.AppImage') + + +class GateTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.stage = Path(self.tmp.name) / 'stage' + write_stage(self.stage, 'public') + + def test_clean_stage_passes(self): + ptr.payload_gates(self.stage, 'windows', 'Fixture.exe') + + def assert_refused(self, rel, data=b'x', message=None): + path = self.stage / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + with self.assertRaises(ptr.PackagingError) as raised: + ptr.payload_gates(self.stage, 'windows', 'Fixture.exe') + if message: + self.assertIn(message, str(raised.exception)) + top = self.stage / Path(rel).parts[0] + if top.is_dir() and top.name not in ('bios', 'mods'): + shutil.rmtree(top) + else: + path.unlink() + + def test_forbidden_content(self): + self.assert_refused('Game (USA).cue', message='disc') + self.assert_refused('bios/SCPH1001.BIN', message='image') + self.assert_refused('saves/card1.mcd') + self.assert_refused('settings.toml', message='personal') + self.assert_refused('mods/installed/x.psxmod', message='personal') + self.assert_refused('state.pst', message='disc') + self.assert_refused('lib/game_full_1.c', message='source') + self.assert_refused('generated/x.h', message='build-kit') + + def test_toolchain_sources_are_allowed(self): + (self.stage / 'overlay_toolchain/tcc/examples').mkdir(parents=True) + (self.stage / 'overlay_toolchain/tcc/examples/fib.c').write_text('int x;') + ptr.payload_gates(self.stage, 'windows', 'Fixture.exe') + + def test_openbios_required(self): + (self.stage / 'bios/openbios.bin').write_bytes(b'short') + with self.assertRaisesRegex(ptr.PackagingError, 'OpenBIOS'): + ptr.payload_gates(self.stage, 'windows', 'Fixture.exe') + + def test_windows_binary_in_linux_package(self): + (self.stage / 'Fixture').write_bytes(b'\x7fELF') + (self.stage / 'extra.dll').write_bytes(b'MZ') + with self.assertRaisesRegex(ptr.PackagingError, 'Windows binary'): + ptr.payload_gates(self.stage, 'linux', 'Fixture') + + def test_template_tokens_must_all_be_stamped(self): + template = Path(self.tmp.name) / 'AppRun' + template.write_text('echo @VERSION@ @UNKNOWN_TOKEN@\n') + with self.assertRaisesRegex(ptr.PackagingError, 'UNKNOWN_TOKEN'): + ptr.render_template(template, {'VERSION': '1'}) + template.write_text('echo @VERSION@\n') + self.assertEqual(ptr.render_template(template, {'VERSION': '1'}), 'echo 1\n') + + +class DiagnosticsTests(unittest.TestCase): + def setUp(self): + self.fixture = Fixture() + self.addCleanup(self.fixture.close) + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + + def ctx(self, channel): + ctx = ptr.Context() + ctx.config, ctx.channel = self.fixture.config(), channel + return ctx + + def test_defaults_follow_channel(self): + for staged, channel, ok in (('public', 'public', True), ('playtest', 'playtest', True), + ('playtest', 'public', False), ('public', 'playtest', False)): + stage = Path(self.tmp.name) / ('%s-%s' % (staged, channel)) + write_stage(stage, staged) + if ok: + ptr.check_diagnostics(self.ctx(channel), stage) + else: + with self.assertRaisesRegex(ptr.PackagingError, 'default_enabled'): + ptr.check_diagnostics(self.ctx(channel), stage) + + def test_undeclared_diagnostics_package_fails(self): + (self.fixture.root / 'release.toml').write_text( + CONFIG.format(extra='').replace( + '[diagnostics]\ngame_id = "SLUS-00000"\npackage_id = "fixture.diagnostics"\n', '')) + stage = Path(self.tmp.name) / 'stage' + write_stage(stage, 'public') + with self.assertRaisesRegex(ptr.PackagingError, 'no \\[diagnostics\\]'): + ptr.check_diagnostics(self.ctx('public'), stage) + + +class VerifyZipTests(unittest.TestCase): + def setUp(self): + self.fixture = Fixture() + self.addCleanup(self.fixture.close) + self.config = self.fixture.config() + self.out = Path(self.fixture.root) / 'dist' + self.out.mkdir() + + def build(self, channel, staged_channel=None): + stage = Path(self.fixture.root) / ('stage-' + channel) + write_stage(stage, staged_channel or channel) + artifact = self.out / ptr.artifact_name(self.config, 'v1.2.3', channel, 'windows') + zip_stage(stage, artifact) + return artifact + + def test_public_and_playtest_verify(self): + for channel in ptr.CHANNELS: + manifest = ptr.verify_artifact(self.build(channel), self.config, channel, 'v1.2.3') + self.assertEqual(manifest['channel'], channel) + + def test_wrong_channel_fails(self): + artifact = self.build('public') + with self.assertRaises(ptr.PackagingError): + ptr.verify_artifact(artifact, self.config, 'playtest', 'v1.2.3') + + def test_tampered_payload_fails(self): + artifact = self.build('public') + with zipfile.ZipFile(artifact, 'a') as archive: + archive.writestr('saves/card.mcd', b'card') + Path(str(artifact) + '.sha256').write_text( + '%s %s\n' % (ptr.sha256(artifact), artifact.name)) + with self.assertRaises(ptr.PackagingError): + ptr.verify_artifact(artifact, self.config, 'public', 'v1.2.3') + + def test_stale_checksum_fails(self): + artifact = self.build('public') + Path(str(artifact) + '.sha256').write_text('0' * 64 + ' x\n') + with self.assertRaisesRegex(ptr.PackagingError, 'sha256'): + ptr.verify_artifact(artifact, self.config, 'public', 'v1.2.3') + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/tests/test_release_channel_cmake.py b/tools/tests/test_release_channel_cmake.py new file mode 100644 index 00000000..44d2dc78 --- /dev/null +++ b/tools/tests/test_release_channel_cmake.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""Configure-time backstop of runtime/psx_release_channel.cmake. + +Usage: test_release_channel_cmake.py [--cmake PATH] + +A release-package build must name its channel; PSX_PUBLIC_RELEASE is derived +from it. Each case configures a throwaway project that includes the module. +""" +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +MODULE = ROOT / "runtime/psx_release_channel.cmake" +CMAKE = os.environ.get("PSX_TEST_CMAKE") or shutil.which("cmake") + + +class ReleaseChannelTests(unittest.TestCase): + def setUp(self): + if not CMAKE: + self.skipTest("cmake not found") + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.src = Path(self.tmp.name) / "src" + self.src.mkdir() + (self.src / "CMakeLists.txt").write_text( + "cmake_minimum_required(VERSION 3.20)\nproject(channel_fixture NONE)\n" + 'include("%s")\n' + 'file(WRITE "${CMAKE_BINARY_DIR}/result.txt" ' + '"${PSX_RELEASE_CHANNEL}|${PSX_PUBLIC_RELEASE}")\n' % MODULE.as_posix()) + self.build = Path(self.tmp.name) / "build" + + def configure(self, *defs): + result = subprocess.run([CMAKE, "-S", str(self.src), "-B", str(self.build)] + list(defs), + capture_output=True, text=True) + return result + + def result(self): + return (self.build / "result.txt").read_text() + + def test_dev_build_has_no_channel(self): + r = self.configure() + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "|OFF") + self.assertIn("not packageable", r.stdout) + + def test_public_channel_derives_public_release(self): + r = self.configure("-DPSX_RELEASE_CHANNEL=public") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "public|ON") + + def test_playtest_channel(self): + r = self.configure("-DPSX_RELEASE_CHANNEL=playtest") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "playtest|OFF") + + def test_switching_channel_on_the_same_tree(self): + self.assertEqual(self.configure("-DPSX_RELEASE_CHANNEL=public").returncode, 0) + r = self.configure("-DPSX_RELEASE_CHANNEL=playtest") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "playtest|OFF") + + def test_unknown_channel_fails(self): + r = self.configure("-DPSX_RELEASE_CHANNEL=release") + self.assertNotEqual(r.returncode, 0) + self.assertIn("must be 'public' or 'playtest'", r.stderr) + + def test_public_release_without_channel_fails(self): + # The pre-channel way to ask for a public package: forgetting the + # channel must not configure. + r = self.configure("-DPSX_PUBLIC_RELEASE=ON") + self.assertNotEqual(r.returncode, 0) + self.assertIn("without a release channel", r.stderr) + + def test_contradiction_fails(self): + r = self.configure("-DPSX_RELEASE_CHANNEL=playtest", "-DPSX_PUBLIC_RELEASE=ON") + self.assertNotEqual(r.returncode, 0) + self.assertIn("contradicts", r.stderr) + r = self.configure("-UPSX_PUBLIC_RELEASE", "-DPSX_RELEASE_CHANNEL=public", + "-DPSX_PUBLIC_RELEASE=OFF") + self.assertNotEqual(r.returncode, 0) + self.assertIn("contradicts", r.stderr) + + def test_consistent_explicit_value_is_accepted(self): + r = self.configure("-DPSX_RELEASE_CHANNEL=public", "-DPSX_PUBLIC_RELEASE=ON") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "public|ON") + + def test_legacy_off_without_channel_stays_a_dev_build(self): + # Every dev tree configured by the old option() carries OFF. + r = self.configure("-DPSX_PUBLIC_RELEASE:BOOL=OFF") + self.assertEqual(r.returncode, 0, r.stderr) + self.assertEqual(self.result(), "|OFF") + + def test_source_commit_must_be_full(self): + r = self.configure("-DPSX_SOURCE_COMMIT=abc123") + self.assertNotEqual(r.returncode, 0) + r = self.configure("-DPSX_SOURCE_COMMIT=" + "a" * 40) + self.assertEqual(r.returncode, 0, r.stderr) + + +if __name__ == "__main__": + if "--cmake" in sys.argv: + index = sys.argv.index("--cmake") + CMAKE = sys.argv[index + 1] + del sys.argv[index:index + 2] + unittest.main()