diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c95c27eb..c0871f53 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,9 +27,9 @@ record; the submodule pointer is the source of truth). The framework evolves on its own cadence; a game only moves to a newer framework when someone deliberately bumps that submodule pointer. -**Starting a new title or shipping a setup-host zip?** Follow +**Starting a new title or shipping a release?** Follow [`docs/GAME_PROJECT_SETUP.md`](docs/GAME_PROJECT_SETUP.md) (layout, CI template -under `docs/ci/templates/setup-release.yml`, release checklist). Also see +under `docs/ci/templates/game-release.yml`, release checklist). Also see [Linking the framework](docs/BUILDING.md#linking-the-framework) and [Framework changes and the pin](#framework-changes-and-the-pin) below. diff --git a/README.md b/README.md index ee828301..3c21a953 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ see [Release Package](#release-package) below. Bringing up a title of your own? Start with [`docs/GAME_PROJECT_SETUP.md`](docs/GAME_PROJECT_SETUP.md) (submodules, -setup-host CI template, release checklist), then +bundled-release CI template, release checklist), then [`CONTRIBUTING.md`](CONTRIBUTING.md). Community projects are listed here alongside the rest. @@ -230,7 +230,7 @@ These are the 3 most important folders to be aware of: | Build the framework | [`docs/BUILDING.md`](docs/BUILDING.md) | | **Ship a game repo** (submodules + CI + release checklist) | [`docs/GAME_PROJECT_SETUP.md`](docs/GAME_PROJECT_SETUP.md) | | **Netplay** (rollback, SFU/ICE, dual-raster, disc gates) | [`docs/NETPLAY.md`](docs/NETPLAY.md) | -| Setup-host CI template | [`docs/ci/templates/setup-release.yml`](docs/ci/templates/setup-release.yml) | +| Bundled-release CI template | [`docs/ci/templates/game-release.yml`](docs/ci/templates/game-release.yml) | | Local Generate & rebuild CLI | [`docs/LOCAL_CODEGEN_SDK.md`](docs/LOCAL_CODEGEN_SDK.md) | | Mods | [`docs/MOD_PACKAGES.md`](docs/MOD_PACKAGES.md) | | Widescreen / native-wide | [`docs/WIDESCREEN.md`](docs/WIDESCREEN.md) | @@ -813,7 +813,7 @@ regenerate), and a change proves itself against the Beetle oracle / on screen rather than by assertion. Game-specific work lives in the game repos, which pin exact framework and UI commits as root-level submodules. -- New title / setup-host release: +- New title / bundled release: [`docs/GAME_PROJECT_SETUP.md`](docs/GAME_PROJECT_SETUP.md) - Framework PRs: [`CONTRIBUTING.md`](CONTRIBUTING.md) (rules, verification, regression checklist, how a fix reaches a game through its pin) diff --git a/docs/AOT_SHARDING.md b/docs/AOT_SHARDING.md index c489aec5..aa880300 100644 --- a/docs/AOT_SHARDING.md +++ b/docs/AOT_SHARDING.md @@ -191,9 +191,11 @@ with exactly the audited files plus `AOT_STATIC_AUDIT.json`. Point the game's runtime flavor the DLL release cannot target (for example a PGXP build); pass `--cps` when the runtime is continuation-passing, as the compiler requires. -**Players get it from Generate.** Titles ship without game bytes, so the shard -has to be built on the player's machine. A profile opts in by declaring where -the build links it: +**It ships compiled.** The shard is generated on the developer's machine, +committed with the rest of `generated/`, and linked by release CI into the +shipped executable (`docs/ci/BUNDLED_RELEASES.md`); the overlay *cache* is +never built in CI, so this shard is the native overlay coverage a release +carries. A profile opts in by declaring where the build links it: ```json "static_output": "generated/overlays_static.c" diff --git a/docs/GAME_PROJECT_SETUP.md b/docs/GAME_PROJECT_SETUP.md index 31567e75..03e11e01 100644 --- a/docs/GAME_PROJECT_SETUP.md +++ b/docs/GAME_PROJECT_SETUP.md @@ -2,8 +2,10 @@ This guide is for **title developers**: you keep game code and config in your own repository, pin this framework and `recomp-ui` as **root-level submodules**, -and (optionally) ship a **setup-host** zip that users generate from a legal disc -locally — the same model as Bomberman Party Edition / Masters of Teräs Käsi. +commit the recompiled game C under `generated/`, and ship a **bundled** zip: +the compiled game, built by CI from that committed C. Players bring their own +disc and play; nothing is generated or built on their machine +(`docs/ci/BUNDLED_RELEASES.md`). Related docs: @@ -79,7 +81,7 @@ Non-interactive / CI (`--yes` / `-Yes` or `PSXRECOMP_SETUP_YES=1`): requires | Netplay lobby URL? | Only if netplay = Y (players ≥ 2); default host `netplay.retcomm.net` → `ws://…:8765` | | GitHub Actions release workflow? | Y → `.github/workflows/release.yml` (logs submodule SHAs via `record_pins.sh`) | | Fetch libretro boxart? | Needs network | -| Run Generate now? | Emitters + OpenBIOS + game C (`generated/` gitignored) | +| Run Generate now? | Emitters + OpenBIOS + game C; `generated/` is **committed** (release CI builds from it) | | Configure & build after Generate? | Only if Generate = Y | | Create GitHub repo (`gh`)? | Opt-in; needs `gh` auth | @@ -196,8 +198,8 @@ disc identity, digests and netplay gates. Older titles (e.g. `psxrecomp-v4` submodule, `psxrecomp_add_runtime_target`, `packaging/` prebuilt zips) can be audited and migrated onto this layout with -**Project Studio**. Releases remain **setup-host only** — the tool does not -create prebuilt generated-C packages. +**Project Studio**. Releases are **bundled**: the tool un-ignores and expects +committed `generated/`, and emits the packager + CI that ship the compiled game. | Entry | Role | | ----- | ---- | @@ -214,13 +216,13 @@ python3 tools/new_project_layout/migrate_project.py gui ``` Typical apply order: rename `psxrecomp-v4` → emit `codegen_setup` → rewrite -CMake (`psxrecomp_add_game_runtime` + wizard) → setup-host packager + CI → +CMake (`psxrecomp_add_game_runtime` + wizard) → bundled packager + CI → optional `probe_disc` / pins. CMake rewrites keep `CMakeLists.txt.pre_migrate.bak` plus `CMakeLists.migrate_extras.txt` when the old file had tests / `EXTRAS_SOURCES` / mod `POST_BUILD` hooks. -`apply` always enables **recomp-ui + setup wizard** (required for -`PSXRECOMP_FORCE_SETUP_HOST`). Netplay stays opt-in (`--enable-netplay` / GUI). +`apply` always enables **recomp-ui + setup wizard** (the shipped game's +first-run disc picker). Netplay stays opt-in (`--enable-netplay` / GUI). ### After scaffold (still not automatic) @@ -230,9 +232,9 @@ playable tree (OpenBIOS / optional retail BIOS C). You still must: 1. **Boot / soak** — fix missing seeds, overlays, FMV/runtime quirks in `game.toml` 2. **Netplay QA** — LAN then lobby; confirm digests + TOC fp; pin `VERSION` 3. **Polish** — more symbols in `symbols.toml`, boxart name mismatches -4. **Ship** — scaffold already creates the repo and pushes once at the end when - you opt in; otherwise push manually. Enable Actions, tag `vX.Y.Z` (CI ships - setup-host **without** `generated/` — end users run Generate locally / via wizard) +4. **Ship** — commit `generated/` (the scaffold does this after a successful + Generate), push, enable Actions, tag `vX.Y.Z`. CI builds the committed C and + attaches the compiled game for every platform. **Legacy layout** (CLI `psxrecomp build`, nested UI under older trees, `tools/setup_dev.sh` for framework-only boots) remains supported. Prefer the @@ -332,20 +334,20 @@ YourGameRecomp/ # your git repo │ # your own POST_BUILD command — see │ # docs/MOD_PACKAGES.md ("How bundled/ gets staged"). ├── scripts/ -│ └── package_setup_release.sh # scaffold fills from package_setup_release.sh.in +│ └── package_release.sh # scaffold fills from package_release.sh.in ├── .github/workflows/ -│ └── release.yml # scaffold fills from docs/ci/templates/setup-release.yml +│ └── release.yml # scaffold fills from docs/ci/templates/game-release.yml ├── psxrecomp/ # submodule → framework + CLI + codegen host + CI tools │ ├── host/psxrecomp_codegen_host.* │ └── lib/recomp-net/ # nested submodule (netplay) ├── recomp-ui/ # submodule → launcher UI -├── generated/ # local only — gitignore (not in CI setup zip) +├── generated/ # COMMITTED recompiled game C — release CI builds it └── disc/ # local disc working tree — gitignore ``` -Minimal `CMakeLists.txt` shape for a **setup-host** title (CI zip / first-run -Generate & rebuild). Wizard is required whenever you ship -`PSXRECOMP_FORCE_SETUP_HOST=ON` — without it the launcher never opens first-run. +Minimal `CMakeLists.txt` shape. The wizard is the shipped game's first-run disc +picker; `GEN_MARKER` / `GEN_FULL_GLOB` name the committed `generated/` C, which +release CI requires (`-DPSXRECOMP_REQUIRE_GAME_C=ON`). ```cmake set(PSXRECOMP_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/psxrecomp") @@ -407,12 +409,16 @@ After scaffold (or a manual clone): python3 psxrecomp/psxrecomp_cli.py generate \ --config game.toml --project-root . --disc disc/game.cue ``` -5. **Build the playable runtime**: +5. **Commit the generated C** — it is the release's input: + ```bash + git add generated && git commit -m "Regenerate game C" + ``` +6. **Build the playable runtime**: ```bash cmake -S . -B build-release -G Ninja -DCMAKE_BUILD_TYPE=Release cmake --build build-release --target psx-runtime -j"$(nproc)" ``` -6. **Soak** — offline boot → LAN netplay → lobby. Tune `[runtime]` quirks only +7. **Soak** — offline boot → LAN netplay → lobby. Tune `[runtime]` quirks only after you have evidence (see MotK `game.toml` comments for the pattern). Details: `[LOCAL_CODEGEN_SDK.md](LOCAL_CODEGEN_SDK.md)`, `[BUILDING.md](BUILDING.md)`, @@ -420,34 +426,24 @@ Details: `[LOCAL_CODEGEN_SDK.md](LOCAL_CODEGEN_SDK.md)`, `[BUILDING.md](BUILDING --- -## Setup-host releases (self-build zip) +## Bundled releases (compiled game zip) -CI ships a zip **without** `generated/` game C, retail BIOS dumps, or an -embedded `toolchain/`. The zip includes: +CI builds the committed `generated/` C and ships a zip that is the game: -- Setup host exe (`recomp-ui` + generate/rebuild wizard) -- Your game sources (`game.toml`, seeds, CMake, host glue, …) -- `psxrecomp/` (runtime + CLI + OpenBIOS profiles + emitters) -- `recomp-ui/` sources (needed to rebuild) -- On Windows: MinGW runtime DLLs beside the host and emitters +- The game executable (signed on Windows when a certificate is configured) +- `assets/`, `mods/bundled/` (catalog verified against the build's manifest), + `game.toml` / `game_options.toml` +- `bios/openbios.bin` + its MIT notice — the release runs on OpenBIOS +- `overlay_toolchain/` so overlays outside the compiled static shard can be + turned native from the player's disc at runtime +- `licenses/` (third-party notices), `README.txt` -Players (or [Retro](https://github.com/RetroPortingToolKit/Retro-Launcher)) -run **Generate once** (wizard or Retro Build & Install) with a legal disc. -Retro / the wizard download `cmake-clang-v1` from -[retcomm-toolchains](https://github.com/RetroPortingToolKit/RetroPorting-Toolchains) -(or accept an offline zip / `RETCOMM_TOOLCHAIN_DIR`). Pass -`--embed-toolchain` to `package_setup_host.sh` only for special offline-first -packs. - -### Player updates (after first Generate) - -| Action | Meaning | -|--------|---------| -| **Update** (Retro) | New setup-host zip → refresh source → cmake rebuild. Skips disc→C when `codegen-cache` fingerprints (ROM/BIOS/emitters) still match. | -| **Generate & Rebuild** | Force regenerate game C from the disc, then rebuild. Use when emit inputs change or cache is wrong. | - -Ordinary host/UI releases do **not** require Generate & Rebuild. Details: -[`ci/HOST_ONLY_RELEASES.md`](ci/HOST_ONLY_RELEASES.md). CI tip: leave the +It contains **no** sources, emitters at the root, CLI, `psxrecomp/` or +`recomp-ui/` tree, generated C, disc, or BIOS dump. Players (or +[Retro](https://github.com/RetroPortingToolKit/Retro-Launcher)) extract it, +pick their legal disc on first run, and play; an update is a raw zip extract +over the previous one. Details and the exact CI steps: +[`ci/BUNDLED_RELEASES.md`](ci/BUNDLED_RELEASES.md). CI tip: leave the `psxrecomp` gitlink pinned and use `reuse_cached_emitters` so release jobs skip rebuilding emitters when only host sources moved. @@ -461,13 +457,14 @@ Do **not** set `PSX_PGO` in CI. PGO stays user-local when `[pgo] enabled = true` | Tool | Role | | ------------------------------- | -------------------------------------------------------- | | `tools/ci/normalize_version.sh` | `vX.Y.Z` → `VERSION` / `TAG` | -| `tools/ci/clear_generated.sh` | Wipe `generated/` for setup-host CI | +| `tools/ci/check_generated.sh` | Committed game C present + tracked; no retail BIOS C | +| `tools/ci/generate_openbios.sh` | Emit the OpenBIOS backend in CI from the bundled image | | `tools/ci/record_pins.sh` | Log submodule SHAs | | `tools/ci/build_emitters.sh` | Build `psxrecomp-game` + `psxrecomp-bios` | -| `tools/fetch_toolchain.sh` | Optional download/unpack (embed packs only) | -| `tools/stage_setup_sdk.sh` | Emitters + OpenBIOS + optional `toolchain/` + MinGW DLLs | +| `tools/fetch_toolchain.sh` | Windows emitter build on the CI machine (never shipped) | | `tools/bundle_mingw_dlls.sh` | Windows runtime DLL copy | -| `tools/package_setup_host.sh` | Full setup-host zip (title args) | +| `tools/package_game_release.sh` | Bundled zip: compiled game + runtime data (title args) | +| `tools/release_stage.py` | Mod catalog / overlay toolchain / overlay cache staging | Composite actions (from the game repo after checkout): @@ -502,16 +499,16 @@ powershell -File psxrecomp\tools\generate_ci.ps1 It is the single-purpose face of Project Studio's `emit_ci_workflow`: the template comes from the project's own psxrecomp submodule, the zip prefix from -`scripts/package_setup_release.sh` so CI uploads the zips the packager built, +`scripts/package_release.sh` so CI uploads the zips the packager built, and an installed workflow is compared by step name -- `--check` exits 1 when it is missing or stale, and only `--force` overwrites one. The New Project Layout scaffold **copies and fills** this for you after submodules land: -- `.github/workflows/release.yml` ← `docs/ci/templates/setup-release.yml` +- `.github/workflows/release.yml` ← `docs/ci/templates/game-release.yml` (Linux x64, Windows x64, macOS arm64 + Intel) -- `scripts/package_setup_release.sh` ← zip prefix / exe / display name +- `scripts/package_release.sh` ← zip prefix / exe / display name `--exe-name` / `codegen_setup.exe_basename` should match CMake `OUTPUT_NAME` (`MAKE_C_IDENTIFIER(WINDOW_TITLE)`, e.g. `TwistedMetal4_Recompiled`), not the @@ -538,16 +535,16 @@ Manual install (existing repos): ```bash mkdir -p .github/workflows scripts -cp psxrecomp/docs/ci/templates/setup-release.yml .github/workflows/release.yml +cp psxrecomp/docs/ci/templates/game-release.yml .github/workflows/release.yml # replace YOUR_ZIP_PREFIX / YOUR_GAME_TITLE / yourgame-release -# add scripts/package_setup_release.sh (see MotK / BPE) +# add scripts/package_release.sh (fill tools/new_project_layout/templates/package_release.sh.in) ``` Or re-run token fill against the submodule template: ```bash python3 psxrecomp/tools/new_project_layout/fill_tokens.py \ - psxrecomp/docs/ci/templates/setup-release.yml \ + psxrecomp/docs/ci/templates/game-release.yml \ .github/workflows/release.yml \ --ci-placeholders \ --set ZIP_PREFIX=mygame \ @@ -560,8 +557,7 @@ Full action reference: `[ci/README.md](ci/README.md)`. ## Bundled release checklist -Use this before tagging a setup-host release that matches other titles -(BPE / MotK / Retro). +Use this before tagging a release. ### Repository @@ -584,73 +580,52 @@ Use this before tagging a setup-host release that matches other titles ``` `VERSION` matches the release you will tag ``` -- [ ] Disc images, `generated/`, BIOS, logs, and runtime dumps are gitignored - ``` - (scaffold writes a rich `.gitignore`; see also `docs/ci/templates/game.gitignore`) - ``` +- [ ] Disc images, BIOS, logs, and runtime dumps are gitignored; `generated/` + is **not** (scaffold writes a rich `.gitignore`; see also + `docs/ci/templates/game.gitignore`) +- [ ] `generated/_dispatch.c` + `_full_*.c` (+ `overlays_static*.c` when + the AOT profile declares it) are committed and current for the pinned + framework (`tools/ci/check_generated.sh` passes locally) +- [ ] No `generated/SCPH*` anywhere (retail-BIOS-derived C never ships) - [ ] Submodule gitlinks (`psxrecomp` / `recomp-ui` / nested `recomp-net`) are ``` the SHAs you intend to ship (CI builds those; `record_pins.sh` only logs) ``` -- [ ] Setup-host CMake path builds with **no** game C and **no** BIOS backends - ``` - CI: `-DPSXRECOMP_FORCE_SETUP_HOST=ON -DPSXRECOMP_ALLOW_NO_BIOS=ON - -DPSX_SETUP_WIZARD=ON` (CMakeLists must also set `PSX_SETUP_WIZARD` / - `ENABLE_SETUP_WIZARD` — FORCE_SETUP_HOST alone does not open the wizard; - `psxrecomp_add_game_runtime` FATAL_ERRORs if FORCE_SETUP_HOST is on without - the wizard). clear_generated.sh wipes OpenBIOS C; users Generate via wizard. - ``` +- [ ] Full build configures from the committed C: CI passes + `-DPSXRECOMP_REQUIRE_GAME_C=ON -DPSX_SETUP_WIZARD=ON` and asserts the + configure log says `linking generated game C (full runtime)` and + `BIOS backends linked: OpenBIOS` +- [ ] `[runtime] overlay_cache = true` in `game.toml` (the packager refuses + otherwise) - [ ] Thin `codegen_setup.c` + `psxrecomp_add_game_runtime` (codegen host is in - ``` `psxrecomp/host/`). Must export `psx_game_codegen_forward_if_built` (see - `codegen_setup.c.in`) — setup-host CI links `main.cpp`, which always calls it. - ``` + `codegen_setup.c.in`) — `main.cpp` always calls it; in a full build it is a + no-op ### Packaging (shared helpers) -- [ ] CI uses `./psxrecomp/.github/actions/build-emitters` -- [ ] Packager calls `psxrecomp/tools/package_setup_host.sh` (lean zip by - ``` - default; optional `--embed-toolchain`) or `stage_setup_sdk.sh` after a - custom stage - ``` -- [ ] Staged tree includes `psxrecomp/psxrecomp_cli.py` -- [ ] Staged `psxrecomp/bios/` has `OpenBIOS.toml`, `openbios.bin`, - ``` - `OpenBIOS.LICENSE`, `SCPH1001.toml` — and **no** retail `.BIN` - ``` -- [ ] Windows zip has `libstdc++-6.dll` + `libgcc_s_seh-1.dll` next to both - ``` - emitters (and host deps such as `zlib1.dll` next to the exe) - ``` -- [ ] Zip mtimes are normalized (packager `touch`) so Ninja does not see - ``` - “future” files after extract - ``` +- [ ] CI uses `./psxrecomp/.github/actions/build-emitters` (OpenBIOS emission + and the shipped overlay toolchain need them) +- [ ] `scripts/package_release.sh` calls `psxrecomp/tools/package_game_release.sh` +- [ ] Staged tree has the game exe, `psx_game_version.txt`, `assets/`, + `mods/bundled/`, `game.toml`, `bios/openbios.bin` + `OpenBIOS.LICENSE`, + `overlay_toolchain/`, `licenses/` — and **no** `psxrecomp/`, `recomp-ui/`, + CLI, root-level emitters, sources, generated C, retail `.BIN` +- [ ] Windows zip has the runtime DLLs the exe imports beside it + (`bundle_mingw_dlls.sh`), signed when a certificate is configured ### Zip contents smoke test - [ ] Linux / macOS / Windows artifacts named consistently - ``` (`YOUR_PREFIX--.zip`) - ``` -- [ ] Extract → run host → Generate with a legal disc succeeds end-to-end -- [ ] After rebuild, game launches; saves/settings land beside the exe +- [ ] Extract → run → pick a legal disc → the game boots; saves/settings land + beside the exe - [ ] Retro install/update (if you publish a catalog entry) uses this **same** - ``` - zip — no separate tools pack required; Update rebuilds via codegen-cache - (not raw zip extract over a Play binary) - ``` -- [ ] Release notes: first Generate once; later Updates are host/UI rebuilds - ``` - unless emitters/ROM/BIOS fingerprints change (see ci/HOST_ONLY_RELEASES.md) - ``` + zip as a prebuilt Play binary (raw zip extract; no build step) ### Publish - [ ] Tag `vX.Y.Z`; GitHub Release attaches all four platform zips -- [ ] Release notes say the zip is a setup host (no disc / no retail BIOS / - ``` - no pre-generated game C) - ``` +- [ ] Release notes say the zip is the compiled game: bring your own disc, + no BIOS, no build - [ ] Catalog / Retro entry points at the release assets when ready \ No newline at end of file diff --git a/docs/LOCAL_CODEGEN_SDK.md b/docs/LOCAL_CODEGEN_SDK.md index 42a67a59..dcca01a7 100644 --- a/docs/LOCAL_CODEGEN_SDK.md +++ b/docs/LOCAL_CODEGEN_SDK.md @@ -7,17 +7,21 @@ automation. This does **not** redistribute disc images. PGO (optional) runs **only on the user’s machine** during local rebuild when `game.toml` has `[pgo] enabled = true`. CI must not set `PSX_PGO`. -## Setup host (CI without game/BIOS generated C) +## Where this sits now (bundled releases) -Games can ship a **setup host**: `psx-runtime` linked **without** game C and -**without** BIOS backends (`-DPSXRECOMP_FORCE_SETUP_HOST=ON`, or legacy -`-DBPE_FORCE_SETUP_HOST=ON`) via `psxrecomp_add_game_runtime(...)`, and with -`PSX_SETUP_WIZARD=ON` / `ENABLE_SETUP_WIZARD` so first-run Generate & rebuild -actually appears. CI never -needs BIOS dumps or private assets. First-run Generate emits OpenBIOS (from -bundled `openbios.bin`) and optional SCPH1001 (player dump), then game C, then -rebuild links everything into `build-release/` (or the title’s -`build_dir_name`). +Public releases are **bundled**: the title commits `generated/`, CI builds the +compiled game and ships it (`docs/ci/BUNDLED_RELEASES.md`). Players never run +Generate. The contract below is the **developer** and Retro-automation path +for producing that committed C, and the runtime's own Generate & rebuild +wizard for a locally built setup host (`-DPSXRECOMP_FORCE_SETUP_HOST=ON`, +a developer convenience — not what CI ships). + +A setup host is `psx-runtime` linked **without** game C and **without** BIOS +backends via `psxrecomp_add_game_runtime(...)`, with `PSX_SETUP_WIZARD=ON` / +`ENABLE_SETUP_WIZARD` so first-run Generate & rebuild appears. First-run +Generate emits OpenBIOS (from bundled `openbios.bin`) and optional SCPH1001 +(local dump), then game C, then rebuild links everything into `build-release/` +(or the title’s `build_dir_name`). **Layout after Generate & rebuild:** the zip-root exe stays the setup host; the playable product (exe + `bios/` + `mods/` + `assets/` + `settings.toml`) @@ -31,15 +35,16 @@ product binary (`psxrecomp_codegen_host_forward_if_built`). Opt out with | Product exe | `build-release/` — Play, bios/mods/assets/settings | | Game zip `psxrecomp/` | submodule tree: CLI, tools, emitters, OpenBIOS profiles | | `psxrecomp_cli.py` | Generate / rebuild / verify-disc (in the submodule) | -| `tools/package_setup_host.sh` | Universal setup-host zip packager | -| `tools/ci/*.sh` | normalize version, clear generated, record pins, build emitters | +| `tools/package_game_release.sh` | Bundled zip packager (what CI ships) | +| `tools/package_setup_host.sh` | Retired setup-host zip packager (not used by release CI) | +| `tools/ci/*.sh` | normalize version, check generated, generate OpenBIOS, record pins, build emitters | | `tools/fetch_toolchain.sh` | Optional: download/unpack `cmake-clang-v1` (CI embed or local) | | `tools/toolchain_pack.py` | CLI: resolve / download / unpack into `toolchain/` + shared cache | | `tools/stage_setup_sdk.sh` | Pack: emitters, OpenBIOS checks, optional `toolchain/`, MinGW DLLs | | `tools/bundle_mingw_dlls.sh` | Windows: copy MinGW runtime DLLs next to host + emitters | | Project `toolchain/` | Stamp file `.psxrecomp-bin` pointing at the shared pack `bin/` | | Shared toolchain cache | `%LOCALAPPDATA%/retcomm/toolchains/cmake-clang-v1/` (Windows) or `~/.local/share/retcomm/toolchains/cmake-clang-v1/` — same tree Retro uses | -| `docs/ci/` | Composite actions + [`templates/setup-release.yml`](ci/templates/setup-release.yml) | +| `docs/ci/` | Composite actions + [`templates/game-release.yml`](ci/templates/game-release.yml) | | `docs/GAME_PROJECT_SETUP.md` | Submodules, CI template usage, bundled-release checklist | | Game sources | `game.toml`, seeds, `CMakeLists.txt` at repo root; `psxrecomp/`, `recomp-ui/` submodules | diff --git a/docs/ci/BUNDLED_RELEASES.md b/docs/ci/BUNDLED_RELEASES.md new file mode 100644 index 00000000..8b153f26 --- /dev/null +++ b/docs/ci/BUNDLED_RELEASES.md @@ -0,0 +1,105 @@ +# Bundled releases + +Public CI builds and ships the **compiled game**. The title commits its +recompiled game C under `generated/`; CI links it, packages the executable with +its runtime data, and attaches one zip per platform. Players download, point +the game at their own disc, and play. Nothing is generated or compiled on a +player's machine. + +This replaces the setup-host model (2026-09-30), under which CI wiped +`generated/`, shipped emitters plus sources plus a wizard, and every player +regenerated and rebuilt the title locally. That shipped uncompiled work, made +first-run depend on a toolchain download, and put every generate/build failure +on the player's machine. + +## What the title commits + +| Path | Status | +|------|--------| +| `generated/_dispatch.c`, `generated/_full_*.c`, `generated/_decls.h`, `.ranges` | **Committed.** `psxrecomp_cli.py generate` writes them; commit after every regenerate. Shards are ~1.4 MB each, far under GitHub's blob limit. | +| `generated/overlays_static*.c`, `AOT_STATIC_AUDIT.json` | **Committed** when the AOT profile declares `static_output` (`docs/AOT_SHARDING.md`). This is the overlay coverage a release links. | +| `psxrecomp/generated/` (BIOS backends) | Not committable: it is inside the framework submodule. CI regenerates OpenBIOS from the bundled image. | +| `generated/SCPH*`, `psxrecomp/generated/SCPH*` | **Never.** Retail-BIOS-derived C. `tools/ci/check_generated.sh` fails the release if it is present. | +| Disc images, BIOS dumps | Never. | + +`.gitignore` must not carry `/generated/`; the scaffold (`gitignore.in`, +`docs/ci/templates/game.gitignore`) no longer writes it, and Project Studio's +`merge_gitignore` removes it from an existing file. + +## What CI does + +Template: [`templates/game-release.yml`](templates/game-release.yml). + +1. `check_boot_exe.sh` — the three copies of the boot-EXE name agree, so the + build looks for the C the checkout actually has. +2. `check_generated.sh` — `generated/_dispatch.c` and its shards exist + **and are tracked**; no retail BIOS C; no tracked BIOS dump. +3. Build `psxrecomp-game` / `psxrecomp-bios` (or restore them from the Actions + cache when the `psxrecomp` submodule SHA matches a prior run). +4. `generate_openbios.sh` — `psxrecomp-bios --config bios/OpenBIOS.toml` into + `psxrecomp/generated/`, plus the emitter fingerprint runtime.cmake checks. +5. Configure with `-DPSXRECOMP_REQUIRE_GAME_C=ON` (a checkout without game C + fails configure instead of quietly producing a setup host), assert the + configure log says `linking generated game C (full runtime)` and + `BIOS backends linked: OpenBIOS`, build `psx-runtime`. +6. `scripts/package_release.sh` → `tools/package_game_release.sh` — stage the + executable, `assets/`, `bios/openbios.bin` + notice, `game.toml`, + `game_options.toml`, the mod catalog (verified against the manifest the + build published, developer-channel packages pruned), `overlay_toolchain/`, + third-party notices; refuse if anything kit-shaped is in the stage; sign on + Windows; zip. +7. Verify the zip: executable + OpenBIOS + catalog present; no `psxrecomp/`, + `recomp-ui/`, CLI, emitters at the root, sources, generated C, dumps, + per-machine mod state. + +`PSX_SETUP_WIZARD` stays on. In a full build it is the first-run disc picker; +it never offers Generate & rebuild because the binary already carries the game. + +## Overlays + +The overlay **shard cache** is compiled from the disc, so CI cannot build one. +A release relies on two things instead: + +- the **static AOT shard** committed under `generated/` and linked into the + executable (titles with an `aot/overlays.json` `static_output`); +- the bundled **`overlay_toolchain/`** (pinned relocatable Python + + `psxrecomp-game` + headers, tcc on Windows), which lets the runtime compile + any other overlay it streams from the player's disc. + +A developer packaging locally with a cache can ship it: +`PSX_OVERLAY_CACHE_ROOT= scripts/package_release.sh …`. The packager +routes it through `tools/release_stage.py`, the only place that knows the cache +tag and layout. + +## Faster host/UI bumps + +`workflow_dispatch` input **`reuse_cached_emitters`** (default `true`): + +| Situation | Emitters | +|-----------|----------| +| Cache miss (new `psxrecomp` SHA) | Always rebuild | +| Cache hit + reuse on | Skip rebuild; use cached binaries | +| Cache hit + reuse off | Rebuild anyway | + +Linux/macOS game compiles use **ccache**, so a host/UI-only bump is close to a +link-only rebuild of the tens of MB of generated C. + +## What players / Retro do + +| Step | Meaning | +|------|---------| +| Install | Extract the zip. Run the game. Pick the disc on first run. | +| Update | Extract the new zip over the old one (raw zip extract: this is a prebuilt Play binary). Saves and settings live beside the exe and are preserved. | +| Retail BIOS | Not needed and not shipped; the release runs on the bundled OpenBIOS. | + +## Title checklist (short) + +- `generated/` committed and regenerated whenever seeds, `game.toml` codegen + settings, or the framework pin change. +- `.gitignore` does not ignore `generated/`. +- `scripts/package_release.sh` (scaffold template `package_release.sh.in`). +- `.github/workflows/release.yml` from `templates/game-release.yml` + (`tools/generate_ci` writes and checks it). +- `[runtime] overlay_cache = true` in `game.toml`. +- Release notes say the zip is the compiled game: bring your own disc, no + BIOS, no build. diff --git a/docs/ci/HOST_ONLY_RELEASES.md b/docs/ci/HOST_ONLY_RELEASES.md deleted file mode 100644 index 31d56b7f..00000000 --- a/docs/ci/HOST_ONLY_RELEASES.md +++ /dev/null @@ -1,64 +0,0 @@ -# Host-only / setup-host releases - -Public CI builds a **setup-host** zip: wizard + game sources + emitters + -framework pins. It never ships retail discs, BIOS dumps, or `generated/` game C. - -## What CI does - -1. `clear_generated.sh` — empty `generated/` so CMake takes the setup-host path. -2. Build `psxrecomp-game` / `psxrecomp-bios` (or restore them from Actions cache - when the `psxrecomp` submodule SHA matches a prior run). -3. Configure with `-DPSXRECOMP_FORCE_SETUP_HOST=ON` + wizard flags; build the - host binary only. -4. `package_setup_host.sh` — lean zip (no embedded toolchain by default). - Requires `psx_game_version.txt` beside the host exe (stamped at compile - time). Refuses to ship if `RELEASE_VERSION` / `VERSION` disagree with that - stamp — a mismatched pin makes netplay lobby browsers filter each other out. - -Template: [`templates/setup-release.yml`](templates/setup-release.yml). - -### Faster host/UI bumps - -`workflow_dispatch` input **`reuse_cached_emitters`** (default `true`): - -| Situation | Emitters | -|-----------|----------| -| Cache miss (new `psxrecomp` SHA) | Always rebuild | -| Cache hit + reuse on | Skip rebuild; package cached binaries | -| Cache hit + reuse off | Rebuild anyway | - -Linux/macOS setup-host compiles also use **ccache**. Windows emitters still use -the portable llvm-mingw fetch; host compile stays on MSYS2 g++. - -Bump the `psxrecomp` gitlink when emitter/CLI behavior must change; leave it -pinned for pure host/UI/`recomp-ui` releases so CI stays cmake-time. - -## What players / Retro do - -| Step | When | -|------|------| -| **Generate** (wizard or Retro Build & Install) | First install, or after ROM/BIOS/emitter fingerprint changes | -| **Update** (Retro) | New setup-host zip → refresh source → cmake rebuild; **codegen-cache** skips regenerate when fingerprints match | -| **Generate & Rebuild** | Force disc→C again (`force_generate`) | - -Players do **not** need Generate & Rebuild for ordinary host/UI updates after the -first successful generate. Retro keeps `apps//codegen-cache/` keyed by -ROM/BIOS + emitter fingerprints (see Retro `docs/BUILD_PACKS.md`). - -Generate also writes the static overlay shard when the title's -`aot/overlays.json` declares `static_output` (`generated/overlays_static*.c` -plus `AOT_STATIC_AUDIT.json`; see `docs/AOT_SHARDING.md`). A cache that -restores `generated/` without regenerating has to carry those files and -include the profile in its key. Otherwise the restored build links a stale -shard or none. Generate itself reuses an unchanged shard, so re-running it -costs little. - -Raw zip extract is for true prebuilt Play binaries only. Setup-host catalog -entries use local generate+cmake (`install_title_auto` / `update_title_auto`). - -## Title checklist (short) - -- Ship wizard (`PSX_SETUP_WIZARD` / `ENABLE_SETUP_WIZARD`). -- Thin `codegen_setup` forward for setup-host link. -- Catalog: `build.enabled` + release asset globs pointing at the setup zip. -- Document that first-run Generate is expected; later Updates are host rebuilds. diff --git a/docs/ci/README.md b/docs/ci/README.md index 54802c99..025b9fab 100644 --- a/docs/ci/README.md +++ b/docs/ci/README.md @@ -1,50 +1,47 @@ -# CI helpers for setup-host releases +# CI helpers for bundled releases Shared scripts and GitHub Actions so every PSXRecomp title uses one -standardized release flow. +standardized release flow: CI builds the title's **committed `generated/` +game C** and ships the **compiled game**. -**Start here:** [`../GAME_PROJECT_SETUP.md`](../GAME_PROJECT_SETUP.md) +**Start here:** [`../GAME_PROJECT_SETUP.md`](../GAME_PROJECT_SETUP.md), then +[`BUNDLED_RELEASES.md`](BUNDLED_RELEASES.md) for what ships and why. ## Workflow template **New Project Layout** (`tools/new_project_layout/setup_project.{sh,ps1}`) copies this template into the title repo and replaces `YOUR_*` / `yourgame-release` using `--zip-prefix` (or a derived acronym) plus the window title. It also -writes `scripts/package_setup_release.sh`, `README.md`, and an optional +writes `scripts/package_release.sh`, `README.md`, and an optional `framework_pins.txt` snapshot, and can opt in wizard/netplay via `--enable-wizard` / `--enable-netplay`. Release CI pins via submodule gitlinks and only logs SHAs with `record_pins.sh` (no `verify_pins` gate). -**Migrating an older title** onto this CI shape: Project Studio -(`tools/new_project_layout/migrate_project.py apply` / `gui`) emits the same -packager + filled `release.yml`. Setup-host only — no prebuilt game-C zips. +**Existing titles:** `tools/generate_ci` writes (and `--check`s) the filled +workflow; Project Studio (`tools/new_project_layout/migrate_project.py apply` / +`gui`) emits the same packager wrapper + `release.yml` and un-ignores +`generated/`. Manual: ```bash mkdir -p .github/workflows -cp psxrecomp/docs/ci/templates/setup-release.yml .github/workflows/release.yml +cp psxrecomp/docs/ci/templates/game-release.yml .github/workflows/release.yml # edit YOUR_* placeholders — or use fill_tokens.py --ci-placeholders # Release name is YAML-quoted so titles with ':' (e.g. Marvel vs. Capcom: …) parse. ``` -Template: [`templates/setup-release.yml`](templates/setup-release.yml) +Template: [`templates/game-release.yml`](templates/game-release.yml) **Versioning:** `workflow_dispatch` with an empty `version` auto-bumps the next `X.Y.Z` from the latest `v*.*.*` git tag (`bump`: patch / minor / major; default patch). Pass an explicit version to override. Shared helper: `tools/ci/normalize_version.sh --next [patch|minor|major]`. - -**Host-only model / player updates:** [`HOST_ONLY_RELEASES.md`](HOST_ONLY_RELEASES.md) -(CI never ships game C; Retro Update reuses codegen-cache for host/UI bumps; -`reuse_cached_emitters` + ccache speed up release jobs when the `psxrecomp` pin -is unchanged). - Matrix: `ubuntu-24.04` (linux-x64), `windows-2022` (windows-x64), `macos-15` (macos-arm64), `macos-15-intel` (macos-x64). -Release hosts always configure with Vulkan headers + `glslc` (MSYS2 / +Release builds always configure with Vulkan headers + `glslc` (MSYS2 / apt / Homebrew packages), verify `vulkan.h` + `glslc` before configure, and fail the job if CMake does not select `PSX_HAVE_VULKAN`. Runtime still loads the ICD dynamically via SDL; CI only needs headers and the shader compiler. @@ -54,58 +51,50 @@ the ICD dynamically via SDL; CI only needs headers and the shader compiler. | Script | Role | |--------|------| | `ci/normalize_version.sh` | Normalize / write `VERSION` + `TAG`; `--next` auto-bumps from latest tag | -| `ci/clear_generated.sh` | Clear `generated/` for setup-host CI | +| `ci/check_boot_exe.sh` | The three copies of the boot-EXE name agree | +| `ci/check_generated.sh` | Committed game C present + tracked; no retail BIOS C; no BIOS dump | +| `ci/generate_openbios.sh` | Emit the OpenBIOS backend C in CI from the bundled MIT image | | `ci/record_pins.sh` | Log `psxrecomp` / `recomp-ui` / `recomp-net` SHAs (CI + scaffold) | | `ci/verify_pins.sh` | Optional local check vs `framework_pins.txt` (not used by release CI) | | `ci/build_emitters.sh` | Build `psxrecomp-game` + `psxrecomp-bios` | -| `fetch_toolchain.sh` | Download/unpack cmake-clang-v1 (Windows emitter builds; optional embed) | -| `stage_setup_sdk.sh` | Emitters, OpenBIOS, optional `toolchain/`, MinGW DLLs | +| `ci/sign_windows.sh` | Authenticode-sign the staged Windows binaries (no-op without a certificate) | +| `fetch_toolchain.sh` | Download/unpack cmake-clang-v1 (Windows emitter builds on the CI machine only) | +| `package_game_release.sh` | Bundled zip: compiled game + runtime data + OpenBIOS + mods + overlay toolchain | +| `release_stage.py` | Shared staging surface (mod catalog, overlay toolchain, overlay cache tag/shards) | | `bundle_mingw_dlls.sh` | Copy imported non-system DLLs next to Windows PEs | -| `package_setup_host.sh` | Lean setup-host zip (optional `--embed-toolchain`) | -| `../cmake/toolchain-mingw-w64.cmake` | Linux→Windows MinGW cross toolchain | -| `../host/psxrecomp_codegen_host.*` | Portable Generate & rebuild host (via CMake helper) | -| `templates/game.gitignore` | Suggested gitignore for title repos | +| `templates/game.gitignore` | Suggested gitignore for title repos (`generated/` tracked) | -Also use `psxrecomp_add_game_runtime(...)` in `runtime/runtime.cmake` for -setup-host / full-game wiring. The CLI (`psxrecomp_cli.py`) lives in this -submodule — there is no separate `psxrecomp-sdk/` overlay. +`package_setup_host.sh` / `stage_setup_sdk.sh` are the retired setup-host +packagers; they are not part of the release flow. ## Composite actions ```yaml - uses: ./psxrecomp/.github/actions/build-emitters - -# Prefer package_setup_host.sh (allow-no-toolchain by default). -# fetch-toolchain + --embed-toolchain only for offline-first packs. -- uses: ./psxrecomp/.github/actions/stage-setup-sdk - with: - stage: dist/stage-setup-${{ matrix.artifact }} - recompiler-build: build-recompiler - allow-no-toolchain: 'true' - runtime-bin: /mingw64/bin # Windows / MSYS2 +- uses: ./psxrecomp/.github/actions/fetch-toolchain # Windows emitter build only ``` ## Title responsibilities Keep only this in the game repo: -- Setup-host CMake flags and exe basename +- `generated/` committed, regenerated (and re-committed) whenever seeds, + codegen settings, or the framework pin change +- CMake via `psxrecomp_add_game_runtime` with `GEN_MARKER` / `GEN_FULL_GLOB` + naming the committed C, and the wizard on (first-run disc picker) - Thin `codegen_setup.c` / `.h` with `psx_game_codegen_forward_if_built` - (scaffold: `tools/new_project_layout/templates/codegen_setup.c.in`) + (scaffold: `tools/new_project_layout/templates/codegen_setup.c.in`) — + `main.cpp` always calls it; in a full build it is a no-op - For netplay titles: `PSX_NETPLAY ON` **before** `include(runtime.cmake)` (`ENABLE_NETPLAY_IF_PRESENT` alone is too late), plus `-DPSX_NETPLAY=ON` in the release workflow configure step -- Thin `scripts/package_setup_release.sh` wrapping `package_setup_host.sh` +- Thin `scripts/package_release.sh` wrapping `package_game_release.sh` - Zip prefix / display name / disc hint in that wrapper - Release notes / GitHub Release job naming -Release CI configures with `-DPSXRECOMP_FORCE_SETUP_HOST=ON`, -`-DPSXRECOMP_ALLOW_NO_BIOS=ON`, and `-DPSX_SETUP_WIZARD=ON` after -`clear_generated.sh`. Titles must also set `PSX_SETUP_WIZARD` / -`ENABLE_SETUP_WIZARD` in `CMakeLists.txt` — without the wizard flag the -setup-host zip never opens first-run / Generate & rebuild. The zip ships -emitters + OpenBIOS.toml; end users Generate (OpenBIOS always; SCPH1001 if -they have a dump) via the wizard, then rebuild. +Release CI configures with `-DPSXRECOMP_REQUIRE_GAME_C=ON` after +`check_generated.sh` and `generate_openbios.sh`, and asserts from the configure +log that the generated game C and the OpenBIOS backend were linked. ## Windows code signing @@ -114,9 +103,9 @@ executable only if it carries a valid Authenticode signature or its exact hash already has reputation. Every CI build is a new hash, so an unsigned release is blocked outright, with no "run anyway". -`package_setup_host.sh` therefore runs `tools/ci/sign_windows.sh` over the -staged tree before zipping. It signs the host exe, every DLL, and the -emitters (an embedded `toolchain/` is left alone) with SHA-256 and an RFC 3161 +`package_game_release.sh` therefore runs `tools/ci/sign_windows.sh` over the +staged tree before zipping. It signs the game exe and every DLL (the +`overlay_toolchain/` bundle is left alone) with SHA-256 and an RFC 3161 timestamp, then verifies each file. | Secret / variable | Meaning | @@ -128,7 +117,7 @@ timestamp, then verifies each file. **No certificate = no signing, and the build still succeeds** (a notice is printed). A certificate that is present but fails to sign is fatal. The -workflow template passes the two secrets to the *Package setup zip* step; +workflow template passes the two secrets to the *Package game zip* step; forks without them package unsigned exactly as before. Certificate choice: an OV certificate satisfies Smart App Control at once, @@ -137,12 +126,10 @@ or Azure Trusted Signing is trusted by both immediately. Trusted Signing does not hand out a .pfx; wiring it means a different signing step, not this script. -What signing cannot cover: a setup-host title is **rebuilt on the player's -machine** after Generate. That locally built game exe is a new, unsigned -binary and Smart App Control judges it on its own. Signing covers the -shipped host/wizard, emitters and DLLs; a machine with Smart App Control -*On* will still refuse the locally built game until the policy is turned -off (Windows Security → App & browser control), which is permanent. +Because the shipped executable **is** the game, the signature now covers +what the player runs. The setup-host model could not offer that: the binary +players ran was built on their own machine after Generate and carried no +signature. ## Release checklist diff --git a/docs/ci/setup-release.yml.example b/docs/ci/setup-release.yml.example deleted file mode 100644 index 169e5121..00000000 --- a/docs/ci/setup-release.yml.example +++ /dev/null @@ -1,23 +0,0 @@ -# Deprecated stub — use the full template: -# -# docs/ci/templates/setup-release.yml -# -# Install: -# cp psxrecomp/docs/ci/templates/setup-release.yml .github/workflows/release.yml -# -# Guide: docs/GAME_PROJECT_SETUP.md -# Actions: docs/ci/README.md - -name: Release builds (see docs/ci/templates/setup-release.yml) - -on: - workflow_dispatch: - -jobs: - redirect: - runs-on: ubuntu-24.04 - steps: - - run: | - echo "Copy psxrecomp/docs/ci/templates/setup-release.yml to .github/workflows/release.yml" - echo "See psxrecomp/docs/GAME_PROJECT_SETUP.md" - exit 1 diff --git a/docs/ci/templates/setup-release.yml b/docs/ci/templates/game-release.yml similarity index 69% rename from docs/ci/templates/setup-release.yml rename to docs/ci/templates/game-release.yml index 41bedd46..88c4349f 100644 --- a/docs/ci/templates/setup-release.yml +++ b/docs/ci/templates/game-release.yml @@ -1,30 +1,41 @@ -# Setup-host multi-platform release template for a PSXRecomp *game* repository. +# Bundled multi-platform release template for a PSXRecomp *game* repository. # # Install: # mkdir -p .github/workflows -# cp psxrecomp/docs/ci/templates/setup-release.yml .github/workflows/release.yml -# Then replace every YOUR_* token (or let setup_project / fill_tokens.py do it). -# See docs/GAME_PROJECT_SETUP.md. +# cp psxrecomp/docs/ci/templates/game-release.yml .github/workflows/release.yml +# Then replace every YOUR_* token (or let setup_project / fill_tokens.py / +# tools/generate_ci do it). See docs/GAME_PROJECT_SETUP.md. +# +# WHAT THIS SHIPS. A compiled game. The title COMMITS its generated/ game C +# (psxrecomp_cli.py generate, run once by the developer from a legal disc); +# CI builds the emitters, emits the OpenBIOS backend from the bundled MIT +# image, links the committed game C into the runtime, and packages the +# executable with its runtime data. Players download, point it at their disc, +# and play. Nothing is generated or compiled on a player's machine, and no +# emitters, sources, or CLI ship at the zip root. +# +# WHAT NEVER SHIPS. Disc images. Retail BIOS dumps and any C derived from one +# (tools/ci/check_generated.sh fails the job if generated/SCPH* exists). The +# overlay shard cache: it is compiled FROM THE DISC, which CI does not have, +# so the committed static AOT shard (generated/overlays_static*.c) is linked +# and the bundled overlay_toolchain/ compiles anything else from the player's +# disc at runtime. Do not set PSX_PGO in CI. # # Shared (do not fork these into the title): -# ./psxrecomp/.github/actions/{build-emitters,fetch-toolchain (optional embed),stage-setup-sdk} -# psxrecomp/tools/ci/*.sh -# psxrecomp/tools/{fetch_toolchain,stage_setup_sdk,bundle_mingw_dlls,package_setup_host}.sh -# psxrecomp/psxrecomp_cli.py (CLI lives in the submodule — no psxrecomp-sdk/) +# ./psxrecomp/.github/actions/{build-emitters,fetch-toolchain} +# psxrecomp/tools/ci/{check_boot_exe,check_generated,generate_openbios,record_pins}.sh +# psxrecomp/tools/{package_game_release,bundle_mingw_dlls}.sh +# psxrecomp/tools/release_stage.py (mod catalog + overlay toolchain staging) # # Title-owned: -# CMake setup-host flags, thin scripts/package_setup_release.sh wrapper +# CMakeLists.txt flags, the thin scripts/package_release.sh wrapper # -# No private CI assets. No BIOS dumps. No generated/ required. -# Do not set PSX_PGO in CI (PGO stays user-local after Generate). -# Release hosts always build with PSX_HAVE_VULKAN (headers + glslc); runtime -# still loads the ICD dynamically via SDL. +# Release hosts always build with PSX_HAVE_VULKAN (headers + glslc); the +# runtime still loads the ICD dynamically via SDL. # # workflow_dispatch: leave version empty to auto-bump patch (or choose bump). -# Host-focused releases: CI never ships game C. Players Generate once; Retro -# Update reuses codegen-cache when ROM/BIOS/emitters match (cmake-only). -# See docs/ci/HOST_ONLY_RELEASES.md. reuse_cached_emitters skips rebuilding -# psxrecomp-game/bios when the psxrecomp SHA cache hits (host/UI bumps). +# reuse_cached_emitters skips rebuilding psxrecomp-game/bios when the +# psxrecomp submodule SHA cache hits. See docs/ci/BUNDLED_RELEASES.md. name: Release builds @@ -195,17 +206,19 @@ jobs: shell: bash run: bash psxrecomp/tools/ci/record_pins.sh - # A skew between game.toml, CMakeLists GEN_MARKER and codegen_setup.c - # makes every player's self-build loop generate+build forever. The CLI - # also refuses the skew at generate time, but failing the release is - # cheaper than failing the player. + # game.toml, CMakeLists GEN_MARKER and codegen_setup.c each carry the + # boot-EXE name; the committed generated/ files are named after it, so a + # skew means the build looks for C the checkout does not have. - name: Lint boot-EXE name consistency shell: bash run: bash psxrecomp/tools/ci/check_boot_exe.sh . - - name: Setup host mode (no game C / BIOS backends in CI) + # The release is built from COMMITTED game C. Fail here, at the top, + # when it is absent or untracked, and when any retail-BIOS-derived C + # or BIOS dump is in the tree. + - name: Verify committed game C shell: bash - run: bash psxrecomp/tools/ci/clear_generated.sh + run: bash psxrecomp/tools/ci/check_generated.sh --root . - name: Setup MSYS2 (Windows) if: runner.os == 'Windows' @@ -223,6 +236,7 @@ jobs: mingw-w64-x86_64-shaderc curl zip + unzip - name: Install deps (Linux) if: runner.os == 'Linux' @@ -234,7 +248,7 @@ jobs: libsdl2-dev libgl1-mesa-dev libxtst-dev \ libx11-dev libxext-dev libxrandr-dev libxcursor-dev \ libxfixes-dev libxi-dev libxss-dev libxkbcommon-dev \ - libasound2-dev libpulse-dev libudev-dev zip \ + libasound2-dev libpulse-dev libudev-dev zip unzip \ libvulkan-dev glslc - name: Install deps (macOS) @@ -247,7 +261,7 @@ jobs: # Headers + glslc are enough to compile PSX_HAVE_VULKAN (runtime still # loads the ICD dynamically via SDL). Fail early if the stub path would - # be selected so release hosts always ship the real Vulkan backend. + # be selected so releases always ship the real Vulkan backend. - name: Verify Vulkan build tools run: | set -euo pipefail @@ -276,11 +290,11 @@ jobs: fi echo "Found ${HDR}" - # Ubuntu 24.04 has no libsdl3-dev; Windows MSYS ships SDL2. Setup-host - # therefore FetchContents the pinned SDL3 tarball. CMake's HTTP/2 download - # from github.com release assets intermittently dies with REFUSED_STREAM - # (curl status 56). Prefetch with curl --http1.1 and point FetchContent - # at the extracted tree (macOS uses brew sdl3 and skips this). + # Ubuntu 24.04 has no libsdl3-dev; Windows MSYS ships SDL2. The runtime + # therefore FetchContents the pinned SDL3 tarball. CMake's HTTP/2 + # download from github.com release assets intermittently dies with + # REFUSED_STREAM (curl status 56). Prefetch with curl --http1.1 and point + # FetchContent at the extracted tree (macOS uses brew sdl3 and skips this). - name: Cache SDL3 source if: runner.os != 'macOS' uses: actions/cache@v4 @@ -340,16 +354,19 @@ jobs: rm -rf "${OUT}.extracting" sdl3_env - # Windows: build emitters with llvm-mingw from cmake-clang-v1 (static CLI). - # Zip still omits toolchain/ — fetch is for the build machine only. + # Windows: build emitters with llvm-mingw from cmake-clang-v1 (static + # CLI). The pack is for the build machine only; it never ships. - name: Fetch portable toolchain (Windows emitters) if: runner.os == 'Windows' uses: ./psxrecomp/.github/actions/fetch-toolchain with: artifact: ${{ matrix.artifact }} - # Host/UI-only bumps keep the same psxrecomp gitlink → cache hit skips - # rebuilding emitters (see reuse_cached_emitters / HOST_ONLY_RELEASES.md). + # Emitters are needed for two things in a bundled release: psxrecomp-bios + # emits the OpenBIOS backend C, and psxrecomp-game ships inside + # overlay_toolchain/ so the runtime can compile overlays from the + # player's disc. Host/UI-only bumps keep the same psxrecomp gitlink → + # cache hit skips rebuilding them (see reuse_cached_emitters). - name: Emitter cache key id: emitter_cache shell: bash @@ -377,7 +394,7 @@ jobs: shell: bash run: | set -euo pipefail - # Restored cache or fresh build must leave packager inputs in place. + # Restored cache or fresh build must leave both emitters in place. test -d build-recompiler find build-recompiler -type f \( -name 'psxrecomp-game*' -o -name 'psxrecomp-bios*' \) | head count="$(find build-recompiler -type f \( -name 'psxrecomp-game*' -o -name 'psxrecomp-bios*' \) | wc -l | tr -d ' ')" @@ -386,14 +403,23 @@ jobs: exit 1 fi - - name: Setup ccache (Linux/macOS host build) + # BIOS C lives in the framework submodule's generated/, which a title + # cannot commit. OpenBIOS is MIT and its image is bundled, so CI emits + # that one backend here. No retail backend is ever produced in CI. + - name: Generate OpenBIOS backend C + shell: bash + run: bash psxrecomp/tools/ci/generate_openbios.sh --framework psxrecomp --recompiler-build build-recompiler + + # The generated game C is tens of MB per title; ccache turns a host/UI + # bump into a link-only rebuild on every platform. + - name: Setup ccache (Linux/macOS) if: runner.os != 'Windows' uses: hendrikmuhs/ccache-action@v1.2 with: - key: setup-host-${{ matrix.artifact }} - max-size: 800M + key: game-${{ matrix.artifact }} + max-size: 2G - - name: Configure & build setup host + - name: Configure & build the game env: CMAKE_OSX_DEPLOYMENT_TARGET: ${{ matrix.deployment_target || '' }} run: | @@ -445,39 +471,21 @@ jobs: fi JOBS="$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)" # Emitters may have set PSXRECOMP_TOOLCHAIN_DIR (llvm-mingw) via - # GITHUB_ENV. Clear it before the MSYS2 g++ setup-host configure so - # pack lib/ cannot win -L for -static-libstdc++ (codecvt link errors). + # GITHUB_ENV. Clear it before the MSYS2 g++ configure so the pack's + # lib/ cannot win -L for -static-libstdc++ (codecvt link errors). if [[ "$(uname -s)" == MINGW* || "$(uname -s)" == MSYS* ]]; then unset PSXRECOMP_TOOLCHAIN_DIR TOOLCHAIN_DIR BPE_TOOLCHAIN_DIR RETCOMM_TOOLCHAIN_DIR || true unset CMAKE_PREFIX_PATH SDL3_DIR ZLIB_ROOT || true fi - # Setup host: no game C, no BIOS backends linked. clear_generated.sh - # already wiped generated/; end users Generate OpenBIOS (+ optional - # SCPH1001) locally via the wizard, then rebuild. - # Prefer -DPSXRECOMP_FORCE_SETUP_HOST=ON (psxrecomp_add_game_runtime). - # Legacy -DYOURGAME_FORCE_SETUP_HOST=ON still works if the title wires it. - # PSX_SETUP_WIZARD is required with FORCE_SETUP_HOST (CMakeLists - # ENABLE_SETUP_WIZARD + -D here). Without it the zip never opens - # first-run / Generate & rebuild. - # Title root codegen_setup.c must define - # psx_game_codegen_forward_if_built (see codegen_setup.c.in) or link - # fails — main.cpp always calls it, including setup-host CI builds. + # Full game build: the committed generated/ C is REQUIRED, so a + # checkout without it fails configure instead of silently producing + # a setup host. OpenBIOS was emitted above; runtime.cmake links every + # backend whose C is present, and only OpenBIOS can be. + # PSX_SETUP_WIZARD stays on: it is the first-run disc picker. # Netplay titles: set PSX_NETPLAY ON in CMakeLists *before* - # include(runtime.cmake). ENABLE_NETPLAY_IF_PRESENT alone is too late - # (option() already cached OFF) and psx_netplay_sched.c still needs - # recomp_net/session.h. Also add -DPSX_NETPLAY=ON to this cmake - # invocation for CI belt-and-suspenders (do not leave it commented - # mid-continuation — that breaks the shell line). - # With PSX_NETPLAY, runtime.cmake defaults RNET_ENABLE_ICE=ON. - # recomp-net FetchContents libjuice via a pinned URL tarball (not - # git clone) so AppImage / mismatched libcurl hosts still configure. - # Offline: vendor under psxrecomp/lib/recomp-net/third_party/libjuice - # or pass -DRNET_LIBJUICE_ROOT=… / -DRNET_ENABLE_ICE=OFF. + # include(runtime.cmake) (ENABLE_NETPLAY_IF_PRESENT alone is too + # late) and add -DPSX_NETPLAY=ON below as belt-and-suspenders. # Add title-specific -D flags here if needed (e.g. -DMOTK_NATIVE=OFF). - # Netplay titles: also pass -DPSX_NETPLAY=ON here (belt-and-suspenders - # when CMakeLists already FORCE-caches it before include(runtime.cmake)). - # Prefer pre-fetched SDL3 (see Prefetch SDL3 source) so configure does - # not call CMake file(DOWNLOAD) against GitHub release assets. if [[ -n "${PSX_SDL3_SOURCE_DIR:-}" && -f "${PSX_SDL3_SOURCE_DIR}/CMakeLists.txt" ]]; then EXTRA+=("-DFETCHCONTENT_SOURCE_DIR_SDL3=${PSX_SDL3_SOURCE_DIR}") elif [[ -f "${GITHUB_WORKSPACE}/.cache/sdl3-src/CMakeLists.txt" ]]; then @@ -485,34 +493,51 @@ jobs: fi cmake -S . -B build-ci -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ - -DPSXRECOMP_FORCE_SETUP_HOST=ON \ - -DPSXRECOMP_ALLOW_NO_BIOS=ON \ + -DPSXRECOMP_REQUIRE_GAME_C=ON \ + -DPSXRECOMP_FORCE_SETUP_HOST=OFF \ + -DPSXRECOMP_ALLOW_NO_BIOS=OFF \ -DPSX_SETUP_WIZARD=ON \ -DPSX_GAME_VERSION="${RELEASE_VERSION}" \ "${EXTRA[@]}" 2>&1 | tee build-ci-configure.log + # Assert the configure took the FULL path, not the setup-host one. + if ! grep -q 'linking generated game C (full runtime)' build-ci-configure.log; then + echo "::error::CMake did not link the generated game C (see configure log)" + exit 1 + fi + if ! grep -qE 'BIOS backends linked: .*OpenBIOS' build-ci-configure.log; then + echo "::error::CMake did not link the OpenBIOS backend (generate_openbios.sh output missing?)" + exit 1 + fi + if grep -qE 'BIOS backends linked: .*SCPH' build-ci-configure.log; then + echo "::error::a retail BIOS backend was linked — that C must never be in a CI checkout" + exit 1 + fi if ! grep -q 'Vulkan backend: headers' build-ci-configure.log; then echo "::error::CMake did not enable PSX_HAVE_VULKAN (see configure log)" exit 1 fi # Ninja does not emit flags.make (Makefiles generator only). Probe - # whichever compile-flag artifact exists for PSX_HAVE_VULKAN. - FOUND_VK_DEFINE=0 - for f in \ - build-ci/build.ninja \ - build-ci/compile_commands.json \ - build-ci/CMakeFiles/psx-runtime.dir/flags.make; do - if [[ -f "${f}" ]] && grep -q 'PSX_HAVE_VULKAN' "${f}"; then - FOUND_VK_DEFINE=1 - echo "Found PSX_HAVE_VULKAN in ${f}" - break + # whichever compile-flag artifact exists for the defines the release + # depends on. + for define in PSX_HAVE_VULKAN PSX_HAS_GAME_DISPATCH; do + FOUND=0 + for f in \ + build-ci/build.ninja \ + build-ci/compile_commands.json \ + build-ci/CMakeFiles/psx-runtime.dir/flags.make; do + if [[ -f "${f}" ]] && grep -q "${define}" "${f}"; then + FOUND=1 + echo "Found ${define} in ${f}" + break + fi + done + if [[ "${FOUND}" -ne 1 ]]; then + echo "::error::${define} missing from psx-runtime compile flags" + exit 1 fi done - if [[ "${FOUND_VK_DEFINE}" -ne 1 ]]; then - echo "::error::PSX_HAVE_VULKAN missing from psx-runtime compile flags" - exit 1 - fi cmake --build build-ci --target psx-runtime -j"${JOBS}" - # Lobby list pin must match RELEASE_VERSION (package_setup_host enforces). + # Lobby list pin must match RELEASE_VERSION (the packager enforces). STAMP="" for cand in \ build-ci/psx_game_version.txt \ @@ -522,7 +547,6 @@ jobs: if [[ -f "${cand}" ]]; then STAMP="${cand}"; break; fi done if [[ -z "${STAMP}" ]]; then - # Fallback: stamp next to the host exe STAMP="$(find build-ci -name psx_game_version.txt -type f 2>/dev/null | head -n1 || true)" fi if [[ -z "${STAMP}" || ! -f "${STAMP}" ]]; then @@ -537,70 +561,95 @@ jobs: fi echo "lobby pin ok: ${BUILT} (${STAMP})" - - name: Package setup zip + - name: Package game zip # Windows code signing (Smart App Control / SmartScreen): the packager - # signs the host exe, DLLs and emitters when these secrets exist and - # skips signing, with a notice, when they do not. Put the .pfx in the + # signs the game exe and its DLLs when these secrets exist and skips + # signing, with a notice, when they do not. Put the .pfx in the # repo's Actions secrets base64-encoded (`base64 -w0 cert.pfx`). - # See psxrecomp/docs/ci/README.md, "Windows code signing". + # See psxrecomp/docs/ci/README.md, "Windows code signing". Because the + # shipped exe IS the game, the signature now covers what the player + # runs — there is no unsigned locally built binary any more. env: WINDOWS_SIGN_PFX_BASE64: ${{ secrets.WINDOWS_SIGN_PFX_BASE64 }} WINDOWS_SIGN_PFX_PASSWORD: ${{ secrets.WINDOWS_SIGN_PFX_PASSWORD }} run: | set -euo pipefail - chmod +x scripts/package_setup_release.sh + chmod +x scripts/package_release.sh if [[ "${{ runner.os }}" == "Windows" ]]; then export PSXRECOMP_RUNTIME_BIN_DIR=/mingw64/bin export BPE_RUNTIME_BIN_DIR=/mingw64/bin fi export RELEASE_VERSION - # Lean zip (no embedded toolchain/). Retro / wizard download - # cmake-clang-v1; pass --embed-toolchain only for offline-first packs. - scripts/package_setup_release.sh build-ci "${{ matrix.artifact }}" build-recompiler + scripts/package_release.sh build-ci "${{ matrix.artifact }}" build-recompiler ls -la dist/ - - name: Verify zip ships the mod catalog + - name: Verify zip contents shell: bash run: | set -euo pipefail - # Gate the release on the catalog actually being inside the zip. - # The packager stages mods/ by default, but this shipped broken for - # a long time precisely because nothing checked: the Mods page was - # empty in every release and the build logs looked fine. Assert the - # artifact, not the intent. + # Assert the artifact, not the intent: every rule below was once + # "the packager does that by default" and shipped broken anyway. ZIP=$(ls dist/YOUR_ZIP_PREFIX-*-"${{ matrix.artifact }}".zip | head -1) echo "checking ${ZIP}" - MANIFESTS=$(unzip -l "${ZIP}" | grep -c 'mods/bundled/.*/manifest\.toml' || true) + LIST="$(unzip -Z1 "${ZIP}")" + # 1. It is a game: the executable, the bundled OpenBIOS, and the + # runtime data sit at the root. + for must in 'bios/openbios.bin' 'bios/OpenBIOS.LICENSE' 'game.toml' 'psx_game_version.txt' 'assets/fonts/' 'mods/bundled/'; do + if ! grep -q "^${must}" <<<"${LIST}"; then + echo "::error::release zip lacks ${must}" >&2 + printf '%s\n' "${LIST}" | head -40 >&2 + exit 1 + fi + done + MANIFESTS=$(grep -c 'mods/bundled/.*/manifest\.toml' <<<"${LIST}" || true) echo "mod manifests in zip: ${MANIFESTS}" if [[ "${MANIFESTS}" -lt 1 ]]; then echo "::error::release zip ships no mod catalog (mods/bundled/*/manifest.toml)" >&2 - unzip -l "${ZIP}" | head -40 >&2 exit 1 fi - # Per-machine mod state must never ship: preloaded catalogs are - # default-disabled, so a leaked state.toml makes one machine's - # selections everybody's defaults. - if unzip -l "${ZIP}" | grep -q 'state\.toml'; then + # 2. It is NOT a build kit: no framework tree, no CLI, no emitters at + # the root, no sources, no generated C. + for never in '^psxrecomp/' '^recomp-ui/' '^psxrecomp_cli.py' '^psxrecomp-game' '^psxrecomp-bios' \ + '^generated/' '^seeds/' '^CMakeLists.txt' '^codegen_setup\.' '^toolchain/' '^src/'; do + if grep -qE "${never}" <<<"${LIST}"; then + echo "::error::release zip contains build-kit content matching ${never}" >&2 + grep -E "${never}" <<<"${LIST}" | head -20 >&2 + exit 1 + fi + done + if grep -E '\.c$' <<<"${LIST}" | grep -qv '^overlay_toolchain/'; then + echo "::error::release zip contains C sources outside overlay_toolchain/" >&2 + grep -E '\.c$' <<<"${LIST}" | grep -v '^overlay_toolchain/' | head -20 >&2 + exit 1 + fi + # 3. No BIOS dump, disc, or memory card; the only image is OpenBIOS. + if grep -iE '(^|/)SCPH|\.(cue|iso|img|chd|mcd|mcr)$' <<<"${LIST}" | grep -qv '\.toml$'; then + echo "::error::release zip contains a BIOS dump, disc, or memory card" >&2 + grep -iE '(^|/)SCPH|\.(cue|iso|img|chd|mcd|mcr)$' <<<"${LIST}" | grep -v '\.toml$' >&2 + exit 1 + fi + if grep -iE '\.bin$' <<<"${LIST}" | grep -qv '^bios/openbios.bin$'; then + echo "::error::release zip contains a .bin other than bios/openbios.bin" >&2 + grep -iE '\.bin$' <<<"${LIST}" | grep -v '^bios/openbios.bin$' >&2 + exit 1 + fi + # 4. Per-machine mod state never ships: state.toml is one machine's + # selections, mods/installed/ is one developer's installs, and + # developer-channel packages are unfinished work. + if grep -q 'state\.toml' <<<"${LIST}"; then echo "::error::release zip contains mods/state.toml (per-machine state)" >&2 exit 1 fi - # Same rule, other half: mods/installed/ is the launcher-owned tree of - # .psxmod archives a developer installed as a player would. Only - # mods/bundled/ is build output, and only build output ships. - if unzip -l "${ZIP}" | grep -q 'mods/installed/'; then + if grep -q 'mods/installed/' <<<"${LIST}"; then echo "::error::release zip contains mods/installed (per-machine installs)" >&2 exit 1 fi - # Developer-channel packages are unfinished work that ships with - # local builds only. The packager drops them under CI automatically - # (it keys off $CI); assert the outcome so a future change to that - # default cannot quietly publish in-progress mods. if unzip -p "${ZIP}" 'mods/**/manifest.toml' 2>/dev/null \ | grep -Eq '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"'; then echo "::error::release zip contains developer-channel mod packages" >&2 exit 1 fi - echo "mod catalog verified in ${ZIP} (no developer-channel packages)" + echo "zip verified: compiled game, bundled OpenBIOS, mod catalog, no kit / dumps / dev mods" - name: Upload artifact uses: actions/upload-artifact@v4 diff --git a/docs/ci/templates/game.gitignore b/docs/ci/templates/game.gitignore index c0b3ad98..05383109 100644 --- a/docs/ci/templates/game.gitignore +++ b/docs/ci/templates/game.gitignore @@ -8,9 +8,10 @@ /out/ /.happypath/ -# Setup-host: leave generated/ gitignored (users generate locally). -# Titles that still CI-build from committed game C may track generated/ instead. -/generated/ +# generated/ is COMMITTED: releases ship the compiled game, and CI builds it +# from the game C in this repo (psxrecomp_cli.py generate, then git add +# generated). Never ignore it. Only the framework submodule's own generated/ +# (BIOS backends) stays untracked, and that is the submodule's rule. # psxrecomp_cli.py work areas (overlay toolchain download, static overlay # shard evidence -- the latter holds bytes read from the disc). /.cache/ diff --git a/runtime/tests/test_packagers_never_format_cache_tag.py b/runtime/tests/test_packagers_never_format_cache_tag.py index ced7a634..e93d8873 100644 --- a/runtime/tests/test_packagers_never_format_cache_tag.py +++ b/runtime/tests/test_packagers_never_format_cache_tag.py @@ -117,8 +117,14 @@ SKIP_DIRS = {'.git', 'build', 'generated', 'node_modules', '__pycache__', # but that is a separate change with a separate proof (macOS in particular # cannot be built or verified here), so this test does not pretend to cover it. PACKAGER_RE = re.compile(r'^package_.*\.(sh|ps1)$') +# `overlay_cache` followed by `=` (in any grep/sed spelling of whitespace) is +# the game.toml KEY `[runtime] overlay_cache = true`, which packagers gate on +# (Wave-5 F4) without touching the cache itself, and `overlay_cache_...` is an +# identifier continuing (SHIP_WITHOUT_OVERLAY_CACHE_BECAUSE, the reason flag). +# Neither is tag consumption. TAG_CONSUMER_RE = re.compile( - r'cg_?tag|CgTag|cache_tag|overlay_cache|overlay_toolchain', re.I) + r'cg_?tag|CgTag|cache_tag|overlay_cache(?!\w|(\[\[:space:\]\]|\\s|\s)*\*?=)|overlay_toolchain', + re.I) SHARED_SURFACE_RE = re.compile(r'release_overlay_stage|release_stage\.py') diff --git a/tools/ci/check_generated.sh b/tools/ci/check_generated.sh new file mode 100755 index 00000000..5575ec64 --- /dev/null +++ b/tools/ci/check_generated.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Gate a bundled release on the committed game C being present and clean. +# +# Releases ship a compiled game: CI builds from the generated/ tree the title +# COMMITS, so a checkout missing it (or carrying the wrong marker name) must +# fail here, at the top of the job, not twenty minutes later at link -- and +# never on a player's machine, which is where the old setup-host model put +# every failure of this kind. +# +# Asserts, from the game repo root: +# 1. generated/<boot>_dispatch.c exists, where <boot> is the boot-EXE name +# game.toml declares (the same name check_boot_exe.sh cross-checks), and +# at least one generated/<boot>_full*.c shard sits beside it. +# 2. Those files are tracked by git. An untracked generated/ builds fine on +# the developer's machine and is absent from every CI checkout, which is +# the exact gap between "works for me" and a red release. +# 3. No retail-BIOS-derived C is present anywhere it could be linked: +# generated/SCPH*_{dispatch,full}.c and psxrecomp/generated/SCPH*. Those +# are derivatives of a copyrighted Sony image and must not reach a public +# artifact. OpenBIOS is the only backend a release links, and CI emits it +# from the bundled MIT image (tools/ci/generate_openbios.sh). +# 4. No BIOS dump is tracked (SCPH*.BIN / *.bin under bios/). +# +# Usage: check_generated.sh [--root DIR] (default: cwd) +# Runs on bash 3.2+ (macOS runners). +set -euo pipefail + +ROOT="${PWD}" +while [[ $# -gt 0 ]]; do + case "$1" in + --root) ROOT="${2:?}"; shift 2 ;; + -h|--help) + sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//' + exit 0 + ;; + *) + echo "error: unknown arg: $1" >&2 + exit 2 + ;; + esac +done +ROOT="$(cd "${ROOT}" && pwd)" + +toml_get() { # <section> <key> -- first quoted value of key inside [section] + awk -v sec="$1" -v key="$2" ' + /^[[:space:]]*\[/ { in_sec = ($0 ~ ("^[[:space:]]*\\[" sec "\\]")) } + in_sec && $0 ~ ("^[[:space:]]*" key "[[:space:]]*=") { + if (match($0, /"[^"]*"/)) { print substr($0, RSTART + 1, RLENGTH - 2) } + exit + }' "${ROOT}/game.toml" +} + +fail() { echo "error: $*" >&2; exit 1; } + +[[ -f "${ROOT}/game.toml" ]] || fail "${ROOT}/game.toml missing -- not a game repo root" + +boot="$(toml_get prepare_disc boot_exe || true)" +if [[ -z "${boot}" ]]; then + exe="$(toml_get game exe || true)" + boot="${exe##*/}" +fi +if [[ -z "${boot}" && -f "${ROOT}/CMakeLists.txt" ]]; then + boot="$(sed -n 's/.*GEN_MARKER[[:space:]]*"generated\/\(.*\)_dispatch\.c".*/\1/p' \ + "${ROOT}/CMakeLists.txt" | head -n1)" +fi +[[ -n "${boot}" ]] || fail "cannot determine the boot-EXE name (game.toml [prepare_disc] boot_exe / [game] exe, or CMakeLists GEN_MARKER)" + +gen="${ROOT}/generated" +dispatch="${gen}/${boot}_dispatch.c" + +if [[ ! -f "${dispatch}" ]]; then + echo "error: ${dispatch#"${ROOT}"/} is missing." >&2 + echo " Bundled releases build from COMMITTED game C. Generate it once from a" >&2 + echo " legal disc and commit generated/:" >&2 + echo " python3 psxrecomp/psxrecomp_cli.py generate --config game.toml \\" >&2 + echo " --project-root . --disc disc/<game>.cue" >&2 + echo " git add generated && git commit -m 'Add generated game C'" >&2 + if grep -qE '^/?generated/?[[:space:]]*$' "${ROOT}/.gitignore" 2>/dev/null; then + echo " .gitignore still ignores generated/ (the old setup-host rule) -- remove" >&2 + echo " that line; see psxrecomp/docs/ci/templates/game.gitignore." >&2 + fi + exit 1 +fi + +shards=0 +for f in "${gen}/${boot}_full"*.c; do + [[ -f "${f}" ]] && shards=$((shards + 1)) +done +[[ "${shards}" -gt 0 ]] || fail "no generated/${boot}_full*.c beside ${boot}_dispatch.c -- regenerate" + +# 2. Tracked, not merely present. +if git -C "${ROOT}" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + untracked="$(git -C "${ROOT}" ls-files --others --exclude-standard -- "generated/${boot}_dispatch.c" "generated/${boot}_full"*.c 2>/dev/null || true)" + ignored="$(git -C "${ROOT}" ls-files --others --ignored --exclude-standard -- "generated/${boot}_dispatch.c" "generated/${boot}_full"*.c 2>/dev/null || true)" + if [[ -n "${untracked}${ignored}" ]]; then + echo "error: generated game C is present but NOT tracked by git:" >&2 + printf '%s\n' ${untracked} ${ignored} | sed 's/^/ /' >&2 + echo " A release builds from the checkout, so these files never reach CI." >&2 + echo " Remove any /generated/ rule from .gitignore, then: git add generated" >&2 + exit 1 + fi +fi + +# 3. No retail-BIOS-derived C anywhere it could link. +retail=() +for d in "${gen}" "${ROOT}/psxrecomp/generated"; do + [[ -d "${d}" ]] || continue + for f in "${d}"/SCPH*_dispatch.c "${d}"/SCPH*_full.c "${d}"/SCPH*_full_*.c; do + [[ -f "${f}" ]] && retail+=("${f#"${ROOT}"/}") + done +done +if [[ "${#retail[@]}" -gt 0 ]]; then + echo "error: retail-BIOS-derived C is present and would be linked into the release:" >&2 + printf ' %s\n' "${retail[@]}" >&2 + echo " Only the OpenBIOS backend ships (CI emits it from the bundled MIT image)." >&2 + echo " Delete these from the checkout; they must never be committed." >&2 + exit 1 +fi + +# 4. No BIOS dump tracked. +if git -C "${ROOT}" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + dumps="$(git -C "${ROOT}" ls-files -- 'bios/*' '*.BIN' 'SCPH*' 2>/dev/null \ + | grep -Ei '\.(bin|rom)$' | grep -v -i 'openbios' || true)" + if [[ -n "${dumps}" ]]; then + echo "error: BIOS image(s) tracked in the repo:" >&2 + printf '%s\n' "${dumps}" | sed 's/^/ /' >&2 + exit 1 + fi +fi + +echo "generated game C ok: generated/${boot}_dispatch.c + ${shards} full shard(s), tracked, no retail BIOS C" diff --git a/tools/ci/clear_generated.sh b/tools/ci/clear_generated.sh deleted file mode 100755 index 7dc555cd..00000000 --- a/tools/ci/clear_generated.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/usr/bin/env bash -# Clear generated game/BIOS C so CI can build a BIOS-free setup host. -# -# Usage (from game repo root): -# psxrecomp/tools/ci/clear_generated.sh [--root DIR] -set -euo pipefail - -ROOT="${PWD}" -while [[ $# -gt 0 ]]; do - case "$1" in - --root) ROOT="${2:?}"; shift 2 ;; - -h|--help) - sed -n '2,7p' "$0" | sed 's/^# \{0,1\}//' - exit 0 - ;; - *) - echo "error: unknown arg: $1" >&2 - exit 2 - ;; - esac -done - -ROOT="$(cd "${ROOT}" && pwd)" -rm -rf "${ROOT}/generated" "${ROOT}/psxrecomp/generated" -echo "cleared generated/ under ${ROOT} (setup-host mode)" diff --git a/tools/ci/generate_openbios.sh b/tools/ci/generate_openbios.sh new file mode 100755 index 00000000..7967ff33 --- /dev/null +++ b/tools/ci/generate_openbios.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Emit the OpenBIOS backend C in CI so a bundled release links a BIOS. +# +# A bundled release compiles the title's COMMITTED game C, but BIOS C lives in +# the framework submodule's generated/ (psxrecomp/generated/), which a title +# cannot commit. OpenBIOS is MIT and its image ships in psxrecomp/bios/, so CI +# regenerates that one backend from the emitter it just built. Retail BIOS +# backends are never produced here: their C derives from a Sony image and must +# not reach a public artifact (tools/ci/check_generated.sh refuses them). +# +# This is the same emission tools/regen_bios.sh performs locally, minus that +# script's "rebuild the emitter first" step (CI has just built or restored it) +# and its search for a recompiler build dir under the framework root (CI +# builds emitters out of tree, at <game root>/build-recompiler). +# +# Usage (from game repo root): +# psxrecomp/tools/ci/generate_openbios.sh [--framework psxrecomp] \ +# [--recompiler-build build-recompiler] +set -euo pipefail + +FRAMEWORK="psxrecomp" +RECOMPILER_BUILD="build-recompiler" +while [[ $# -gt 0 ]]; do + case "$1" in + --framework) FRAMEWORK="${2:?}"; shift 2 ;; + --recompiler-build) RECOMPILER_BUILD="${2:?}"; shift 2 ;; + -h|--help) + sed -n '2,18p' "$0" | sed 's/^# \{0,1\}//' + exit 0 + ;; + *) + echo "error: unknown arg: $1" >&2 + exit 2 + ;; + esac +done + +[[ -d "${FRAMEWORK}" ]] || { echo "error: framework dir missing: ${FRAMEWORK}" >&2; exit 1; } +FRAMEWORK="$(cd "${FRAMEWORK}" && pwd)" +[[ -d "${RECOMPILER_BUILD}" ]] || { echo "error: recompiler build dir missing: ${RECOMPILER_BUILD}" >&2; exit 1; } +RECOMPILER_BUILD="$(cd "${RECOMPILER_BUILD}" && pwd)" + +EMITTER="" +for cand in "${RECOMPILER_BUILD}/psxrecomp-bios" "${RECOMPILER_BUILD}/psxrecomp-bios.exe" \ + "${RECOMPILER_BUILD}/Release/psxrecomp-bios.exe"; do + if [[ -f "${cand}" ]]; then EMITTER="${cand}"; break; fi +done +[[ -n "${EMITTER}" ]] || { echo "error: psxrecomp-bios not found under ${RECOMPILER_BUILD} (run build_emitters.sh)" >&2; exit 1; } + +PROFILE="bios/OpenBIOS.toml" +for req in "${PROFILE}" bios/openbios.bin bios/OpenBIOS.LICENSE tools/bios_emitter_fingerprint.sh; do + [[ -f "${FRAMEWORK}/${req}" ]] || { echo "error: ${FRAMEWORK}/${req} missing" >&2; exit 1; } +done + +toml_value() { + sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*\"\([^\"]*\)\".*/\1/p" "${FRAMEWORK}/${PROFILE}" | head -1 +} +OUT="$(toml_value out_dir)"; OUT="${OUT:-generated}" +STEM="$(toml_value out_stem)" +[[ -n "${STEM}" ]] || { echo "error: no out_stem in ${PROFILE}" >&2; exit 1; } + +# The profile's rom / seeds paths are framework-relative, so run from there. +mkdir -p "${FRAMEWORK}/${OUT}" +echo "generate_openbios: ${EMITTER} --config ${PROFILE} (stem ${STEM}) -> ${FRAMEWORK}/${OUT}" +( cd "${FRAMEWORK}" && "${EMITTER}" --config "${PROFILE}" ) + +# Same fingerprint regen_bios.sh records, so runtime.cmake's staleness check +# sees a matching stamp instead of warning on every CI configure. +( cd "${FRAMEWORK}" && bash tools/bios_emitter_fingerprint.sh "${PROFILE}" > "${OUT}/${STEM}.emitter.sha" ) + +dispatch="${FRAMEWORK}/${OUT}/${STEM}_dispatch.c" +full="${FRAMEWORK}/${OUT}/${STEM}_full.c" +for f in "${dispatch}" "${full}"; do + [[ -f "${f}" ]] || { echo "error: emitter produced no ${f}" >&2; exit 1; } +done +# runtime.cmake links a stem only when its descriptor is present; probe the +# same thing so a stale emitter fails here rather than as a silent skip. +if ! grep -q "${STEM}_psx_bios_backend" "${dispatch}"; then + echo "error: ${dispatch} defines no ${STEM}_psx_bios_backend descriptor -- emitter predates the backend mechanism" >&2 + exit 1 +fi +echo "generate_openbios: ok ($(grep -m1 -o 'Dispatch entries: [0-9]*' "${dispatch}" || echo 'descriptor present'))" diff --git a/tools/generate_ci.ps1 b/tools/generate_ci.ps1 index 68016d99..1783dc82 100644 --- a/tools/generate_ci.ps1 +++ b/tools/generate_ci.ps1 @@ -1,6 +1,6 @@ <# .SYNOPSIS -Write the setup-host release workflow into an existing PSX project (Windows). +Write the bundled release workflow into an existing PSX project (Windows). .DESCRIPTION The logic is tools/generate_ci.py; this only finds Python and passes every diff --git a/tools/generate_ci.py b/tools/generate_ci.py index 90580d46..3d9249b7 100755 --- a/tools/generate_ci.py +++ b/tools/generate_ci.py @@ -1,6 +1,9 @@ #!/usr/bin/env python3 -"""generate_ci.py -- write the setup-host release workflow into an EXISTING -PSX project. Nothing else: no commit, no push, no migration plan. +"""generate_ci.py -- write the bundled release workflow into an EXISTING PSX +project. Nothing else: no commit, no push, no migration plan. + +The workflow builds the title's COMMITTED generated/ game C and ships the +compiled game (docs/ci/BUNDLED_RELEASES.md). python3 psxrecomp/tools/generate_ci.py # cwd is the project python3 psxrecomp/tools/generate_ci.py ~/src/MyGameRecomp @@ -10,14 +13,14 @@ PSX project. Nothing else: no commit, no push, no migration plan. This is the thin, single-purpose face of Project Studio's emit_ci_workflow operation (tools/new_project_layout/project_studio/ops.py), which is also what `migrate_project.py apply` and `git install-ci` run. The template is -docs/ci/templates/setup-release.yml from the project's own psxrecomp +docs/ci/templates/game-release.yml from the project's own psxrecomp submodule when it has one, so the workflow matches the framework the project pins; this checkout is the fallback. Where the values come from: - ZIP_PREFIX scripts/package_setup_release.sh --zip-prefix <x> (the - packager's prefix, so the zips CI uploads are the zips it - built), else derived from the project name, else --zip-prefix + ZIP_PREFIX scripts/package_release.sh --zip-prefix <x> (the packager's + prefix, so the zips CI uploads are the zips it built), else + derived from the project name, else --zip-prefix GAME_TITLE CMakeLists.txt / game.toml window_title, else --title An existing, filled release.yml is not overwritten silently: its step names @@ -40,13 +43,17 @@ WORKFLOW_REL = pathlib.Path(".github") / "workflows" / "release.yml" def zip_prefix_from_packager(root: pathlib.Path) -> str: - try: - text = (root / "scripts" / "package_setup_release.sh").read_text( - encoding="utf-8", errors="replace") - except OSError: - return "" - m = re.search(r"--zip-prefix\s+([^\s\\'\"]+)", text) - return m.group(1) if m else "" + # package_setup_release.sh is the retired setup-host wrapper; its prefix is + # still the right one for a title that has not re-emitted the packager. + for name in ("package_release.sh", "package_setup_release.sh"): + try: + text = (root / "scripts" / name).read_text(encoding="utf-8", errors="replace") + except OSError: + continue + m = re.search(r"--zip-prefix\s+([^\s\\'\"]+)", text) + if m: + return m.group(1) + return "" def main() -> int: @@ -70,16 +77,17 @@ def main() -> int: try: from project_studio.models import MigrateOptions from project_studio.ops import op_emit_ci_workflow, _ci_steps_missing_from - from project_studio.paths import ci_setup_release_template + from project_studio.paths import ci_release_template except ImportError as exc: print(f"generate_ci: cannot import Project Studio from {HERE / 'new_project_layout'}: {exc}", file=sys.stderr) return 2 - template = ci_setup_release_template(root) + template = ci_release_template(root) if template is None: - print("generate_ci: no docs/ci/templates/setup-release.yml found (neither the " - "project's psxrecomp submodule nor this checkout)", file=sys.stderr) + print("generate_ci: no docs/ci/templates/game-release.yml found (neither the " + "project's psxrecomp submodule nor this checkout -- a submodule pinned " + "before bundled releases must be bumped first)", file=sys.stderr) return 2 dst = root / WORKFLOW_REL diff --git a/tools/generate_ci.sh b/tools/generate_ci.sh index 9afce222..d464f823 100755 --- a/tools/generate_ci.sh +++ b/tools/generate_ci.sh @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# Write the setup-host release workflow into an existing PSX project (Linux / +# Write the bundled release workflow into an existing PSX project (Linux / # macOS / WSL / Git Bash). The logic is tools/generate_ci.py; this finds Python. # # sh psxrecomp/tools/generate_ci.sh # cwd is the project diff --git a/tools/new_project_layout/README.md b/tools/new_project_layout/README.md index 98574f96..1e35bb80 100644 --- a/tools/new_project_layout/README.md +++ b/tools/new_project_layout/README.md @@ -1,8 +1,9 @@ # New Project Layout + Project Studio -Scaffold **new** titles and **migrate** older ones onto the setup-host layout. +Scaffold **new** titles and **migrate** older ones onto the New Project Layout. -**Policy:** public releases are **setup-host only** (no prebuilt generated game C). +**Policy:** public releases are **bundled** — `generated/` is committed and CI +builds and ships the compiled game (`docs/ci/BUNDLED_RELEASES.md`). ## New project @@ -85,11 +86,13 @@ For **bulk ops across many titles / platforms**, use the sibling tool | `ensure_recomp_ui_submodule` | Add `recomp-ui` | | `emit_codegen_setup` | Thin `codegen_setup.c/.h` | | `rewrite_cmake_setup_host` | `psxrecomp_add_game_runtime` + wizard | -| `emit_packager` | `scripts/package_setup_release.sh` | -| `emit_ci_workflow` | Setup-host `release.yml` | +| `merge_gitignore` | Disc/BIOS/dist rules; **un-ignores** `generated/` | +| `emit_packager` | `scripts/package_release.sh` (bundled) | +| `emit_ci_workflow` | Bundled `release.yml` | | `probe_disc_refresh` | TOC / catalog / seeds (needs `--disc`) | -| `annotate_legacy_packaging` | Mark old prebuilt packagers obsolete | +| `annotate_legacy_packaging` | Mark hand-forked packagers obsolete | -Wizard + `recomp-ui` are forced on for `apply` (setup-host requirement). +Wizard + `recomp-ui` are forced on for `apply` (the shipped game's first-run +disc picker). Helpers reused: `probe_disc.py`, `fill_tokens.py`, `sync_symbols.py`, `write_recomp_json.py` (`.recomp.json`, [recomp.fyi spec](https://recomp.fyi/spec)), `templates/*`. diff --git a/tools/new_project_layout/project_studio/__init__.py b/tools/new_project_layout/project_studio/__init__.py index ea335ef4..a024f591 100644 --- a/tools/new_project_layout/project_studio/__init__.py +++ b/tools/new_project_layout/project_studio/__init__.py @@ -1,6 +1,6 @@ """PSXRecomp Project Studio — audit / plan / apply for New Project Layout. -Setup-host releases only (no prebuilt game-C packaging path). +Bundled releases: generated/ is committed and CI ships the compiled game. """ from __future__ import annotations diff --git a/tools/new_project_layout/project_studio/buildops.py b/tools/new_project_layout/project_studio/buildops.py index eb15a437..abcf8a49 100644 --- a/tools/new_project_layout/project_studio/buildops.py +++ b/tools/new_project_layout/project_studio/buildops.py @@ -848,7 +848,7 @@ def launch_status() -> str: # a broken package (an empty Mods page) survived so long -- nobody produced one # locally to look inside. # -# This runs the SAME wrapper CI runs (scripts/package_setup_release.sh), so a +# This runs the SAME wrapper CI runs (scripts/package_release.sh), so a # local export and a CI release are the same artifact by construction rather # than by convention. # --------------------------------------------------------------------------- @@ -891,11 +891,13 @@ def export_release( packages dropped. A local export keeps them by default, because the point of exporting locally is to test the work in progress. """ - wrapper = root / "scripts" / "package_setup_release.sh" - if not wrapper.is_file(): + from .detect import packager_wrapper + + wrapper = packager_wrapper(root) + if wrapper is None: return CmdResult( False, - "No scripts/package_setup_release.sh", + "No scripts/package_release.sh", "Emit it first (Migrate -> packager), or install release CI.", ) build_path = root / build_dir diff --git a/tools/new_project_layout/project_studio/bulkops.py b/tools/new_project_layout/project_studio/bulkops.py index a91ee709..f1dfd3a6 100644 --- a/tools/new_project_layout/project_studio/bulkops.py +++ b/tools/new_project_layout/project_studio/bulkops.py @@ -907,7 +907,7 @@ def bulk_install_ci( jobs: int = 1, on_repo: OnRepoResults | None = None, ) -> list[CmdResult]: - """Install/push setup-host ``release.yml`` on each selected game repo.""" + """Install/push the bundled ``release.yml`` on each selected game repo.""" from fill_tokens import derive_zip_prefix def one(label: str, root: Path) -> list[CmdResult]: diff --git a/tools/new_project_layout/project_studio/cli.py b/tools/new_project_layout/project_studio/cli.py index 21e016fd..9354025c 100644 --- a/tools/new_project_layout/project_studio/cli.py +++ b/tools/new_project_layout/project_studio/cli.py @@ -49,7 +49,7 @@ def _print_audit(report, *, as_json: bool) -> int: fails = sum(1 for c in report.checks if c.status.value == "fail") warns = sum(1 for c in report.checks if c.status.value == "warn") print() - print(f"Summary: {fails} fail, {warns} warn (setup-host releases only)") + print(f"Summary: {fails} fail, {warns} warn (bundled releases: committed generated/ C)") return 1 if fails else 0 @@ -116,7 +116,7 @@ def cmd_plan(args: argparse.Namespace) -> int: def cmd_apply(args: argparse.Namespace) -> int: root = Path(args.root).expanduser().resolve() opts = _options_from_args(args) - # Setup-host exclusive hard rules + # Hard rules: recomp-ui + wizard are the shipped game's first-run disc picker opts.enable_wizard = True opts.enable_recomp_ui = True if opts.players < 2: @@ -1600,7 +1600,7 @@ def build_parser() -> argparse.ArgumentParser: prog="migrate_project", description=( "PSXRecomp Project Studio — migrate / update title repos to the " - "New Project Layout (setup-host releases only)." + "New Project Layout (bundled releases: committed generated/ C, compiled game shipped)." ), ) ap.add_argument("--version", action="version", version=f"%(prog)s {__version__}") @@ -1625,9 +1625,9 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument("--enable-netplay", action="store_true") p.add_argument("--lobby-url", default="ws://netplay.retcomm.net:8765") p.add_argument("--no-recomp-ui", action="store_true", - help="Ignored for setup-host apply (forced on)") + help="Ignored for apply (forced on)") p.add_argument("--no-wizard", action="store_true", - help="Ignored for setup-host apply (forced on)") + help="Ignored for apply (forced on)") p.add_argument("--no-ci", action="store_true") p.add_argument("--no-boxart", action="store_true") p.add_argument("--no-rewrite-cmake", action="store_true") @@ -2206,7 +2206,7 @@ def build_parser() -> argparse.ArgumentParser: p_gbci = git_sub.add_parser( "bulk-install-ci", - help="Install/push setup-host release.yml on selected indexed repos", + help="Install/push bundled release.yml on selected indexed repos", ) add_bulk_select(p_gbci) p_gbci.add_argument( @@ -2233,7 +2233,7 @@ def build_parser() -> argparse.ArgumentParser: p_gci = git_sub.add_parser( "install-ci", - help="Write psxrecomp setup-release.yml, commit, and push", + help="Write psxrecomp game-release.yml, commit, and push", ) add_git_root(p_gci) p_gci.add_argument("--zip-prefix", default="", help="CI asset zip prefix") @@ -2281,12 +2281,12 @@ def build_parser() -> argparse.ArgumentParser: p_bm.add_argument( "--setup-host", action="store_true", - help="CI-parity setup-host configure (FORCE_SETUP_HOST + wizard)", + help="Retired: setup-host configure parity (releases build the committed game C)", ) p_bm.add_argument( "--package", action="store_true", - help="Run scripts/package_setup_release.sh after build", + help="Run scripts/package_release.sh after build", ) p_bm.add_argument( "--package-only", diff --git a/tools/new_project_layout/project_studio/detect.py b/tools/new_project_layout/project_studio/detect.py index e4d4063d..6e158896 100644 --- a/tools/new_project_layout/project_studio/detect.py +++ b/tools/new_project_layout/project_studio/detect.py @@ -1,4 +1,4 @@ -"""Audit an existing title repo against the New Project Layout (setup-host).""" +"""Audit an existing title repo against the New Project Layout (bundled releases).""" from __future__ import annotations @@ -144,6 +144,26 @@ def _cmake_source_refs(cmake: str) -> tuple[set[str], set[str]]: return refs, guarded +# The bundled-release wrapper and the retired setup-host one. Audits accept +# either as "a packager is present" but only the first as current. +PACKAGER_WRAPPER = "package_release.sh" +LEGACY_PACKAGER_WRAPPER = "package_setup_release.sh" + + +def packager_wrapper(root: Path) -> Path | None: + """The title's packager wrapper: the bundled one, else the retired one.""" + for name in (PACKAGER_WRAPPER, LEGACY_PACKAGER_WRAPPER): + p = root / "scripts" / name + if p.is_file(): + return p + return None + + +def packager_is_bundled(packager: Path) -> bool: + """True when the wrapper calls the shared bundled packager (no source allowlist).""" + return "package_game_release.sh" in _read(packager) + + def packager_staged_paths(packager_text: str) -> set[str]: """Top-level paths the packager passes as --project-file / --project-dir.""" out: set[str] = set() @@ -165,8 +185,8 @@ def packager_missing_paths(root: Path) -> list[str]: own pre-zip gate applies, so the two cannot disagree. """ cmake_path = root / "CMakeLists.txt" - packager = root / "scripts" / "package_setup_release.sh" - if not cmake_path.is_file() or not packager.is_file(): + packager = packager_wrapper(root) + if not cmake_path.is_file() or packager is None or packager_is_bundled(packager): return [] refs, guarded = _cmake_source_refs(_read(cmake_path)) staged = packager_staged_paths(_read(packager)) @@ -190,8 +210,8 @@ def packager_missing_optional_paths(root: Path) -> list[str]: select, so the mod builds and never enables. """ cmake_path = root / "CMakeLists.txt" - packager = root / "scripts" / "package_setup_release.sh" - if not cmake_path.is_file() or not packager.is_file(): + packager = packager_wrapper(root) + if not cmake_path.is_file() or packager is None or packager_is_bundled(packager): return [] refs, guarded = _cmake_source_refs(_read(cmake_path)) staged = packager_staged_paths(_read(packager)) @@ -457,15 +477,29 @@ def audit_project(root: Path) -> AuditReport: ) # Packager - packager = root / "scripts" / "package_setup_release.sh" - if packager.is_file(): + packager = packager_wrapper(root) + if packager is not None and packager_is_bundled(packager): checks.append( CheckResult( id="setup_host_packager", - title="scripts/package_setup_release.sh", + title=f"scripts/{PACKAGER_WRAPPER}", status=CheckStatus.PASS, severity=Severity.REQUIRED, - detail="Setup-host packager wrapper present.", + detail="Bundled-release packager wrapper present (ships the compiled game).", + ) + ) + elif packager is not None: + checks.append( + CheckResult( + id="setup_host_packager", + title=f"scripts/{PACKAGER_WRAPPER}", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + f"Only the retired setup-host wrapper scripts/{packager.name} is present; " + "releases now ship the compiled game. Re-emit the packager (force)." + ), + fix_op="emit_packager", ) ) # Present is not the same as current: the allowlist inside it has to @@ -521,10 +555,10 @@ def audit_project(root: Path) -> AuditReport: checks.append( CheckResult( id="setup_host_packager", - title="scripts/package_setup_release.sh", + title=f"scripts/{PACKAGER_WRAPPER}", status=CheckStatus.FAIL, severity=Severity.REQUIRED, - detail="Missing setup-host packager (required; prebuilt releases are not used).", + detail="Missing bundled-release packager wrapper (required).", fix_op="emit_packager", ) ) @@ -537,18 +571,32 @@ def audit_project(root: Path) -> AuditReport: checks.append( CheckResult( id="ci_release", - title="Setup-host release CI", + title="Bundled release CI", status=CheckStatus.WARN, severity=Severity.REQUIRED, detail="release.yml still has YOUR_* placeholders.", fix_op="emit_ci_workflow", ) ) + elif "clear_generated.sh" in wtext or "FORCE_SETUP_HOST=ON" in wtext: + checks.append( + CheckResult( + id="ci_release", + title="Bundled release CI", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + "release.yml is the retired setup-host workflow (wipes generated/ and " + "ships an uncompiled kit). Re-emit it (force) to ship the compiled game." + ), + fix_op="emit_ci_workflow", + ) + ) else: checks.append( CheckResult( id="ci_release", - title="Setup-host release CI", + title="Bundled release CI", status=CheckStatus.PASS, severity=Severity.REQUIRED, detail=".github/workflows/release.yml present.", @@ -558,28 +606,31 @@ def audit_project(root: Path) -> AuditReport: checks.append( CheckResult( id="ci_release", - title="Setup-host release CI", + title="Bundled release CI", status=CheckStatus.FAIL, severity=Severity.REQUIRED, - detail="Missing setup-host release.yml workflow.", + detail="Missing release.yml workflow.", fix_op="emit_ci_workflow", ) ) - # Legacy prebuilt packaging — warn only (setup-host exclusive policy) + # Hand-forked per-title packagers — warn only. The shared bundled packager + # (scripts/package_release.sh -> psxrecomp/tools/package_game_release.sh) + # supersedes them; forks are where the cache-tag drift lived. legacy_pack = False - if (root / "packaging").is_dir() or (root / "tools" / "package_release.ps1").is_file(): + if (root / "packaging").is_dir() or (root / "tools" / "package_release.ps1").is_file() \ + or (root / "tools" / "package_appimage.sh").is_file(): legacy_pack = True checks.append( CheckResult( id="legacy_prebuilt", - title="Legacy prebuilt packaging", + title="Hand-forked packaging scripts", status=CheckStatus.WARN, severity=Severity.RECOMMENDED, detail=( - "Found packaging/ or tools/package_release.ps1. " - "This studio only ships setup-host zips — leave prebuilt scripts unused " - "or remove them after CI is green." + "Found packaging/, tools/package_release.ps1 or tools/package_appimage.sh. " + "Releases go through scripts/package_release.sh + release.yml; " + "remove the forks once CI is green." ), fix_op="annotate_legacy_packaging", ) @@ -588,10 +639,10 @@ def audit_project(root: Path) -> AuditReport: checks.append( CheckResult( id="legacy_prebuilt", - title="Legacy prebuilt packaging", + title="Hand-forked packaging scripts", status=CheckStatus.PASS, severity=Severity.INFO, - detail="No legacy prebuilt packager detected.", + detail="No hand-forked packager detected.", ) ) @@ -813,25 +864,42 @@ def audit_project(root: Path) -> AuditReport: gi = _read(root / ".gitignore") missing_gi = [ pat - for pat in ("/generated/", "/disc/", "/bios/", "/dist/", "/analysis/") + for pat in ("/disc/", "/bios/", "/dist/", "/analysis/") if pat not in gi and pat.rstrip("/") not in gi ] + ignores_generated = any( + re.match(r"^\s*/?generated/?\s*$", line) for line in gi.splitlines() + ) if not (root / ".gitignore").is_file(): checks.append( CheckResult( id="gitignore", - title=".gitignore setup-host rules", + title=".gitignore rules", status=CheckStatus.FAIL, severity=Severity.RECOMMENDED, detail="Missing .gitignore.", fix_op="merge_gitignore", ) ) + elif ignores_generated: + checks.append( + CheckResult( + id="gitignore", + title=".gitignore rules", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + ".gitignore ignores generated/ (the retired setup-host rule). Releases " + "build the committed game C, so CI would find nothing to compile." + ), + fix_op="merge_gitignore", + ) + ) elif missing_gi: checks.append( CheckResult( id="gitignore", - title=".gitignore setup-host rules", + title=".gitignore rules", status=CheckStatus.WARN, severity=Severity.RECOMMENDED, detail="Missing patterns: " + ", ".join(missing_gi), @@ -842,10 +910,82 @@ def audit_project(root: Path) -> AuditReport: checks.append( CheckResult( id="gitignore", - title=".gitignore setup-host rules", + title=".gitignore rules", status=CheckStatus.PASS, severity=Severity.RECOMMENDED, - detail="Essential setup-host ignore rules present.", + detail="Disc / BIOS / dist / analysis ignored; generated/ tracked.", + ) + ) + + # Committed game C: the release's input. Present and tracked, or the + # workflow has nothing to build. + if boot: + marker = root / "generated" / f"{boot}_dispatch.c" + tracked = False + if marker.is_file(): + try: + r = subprocess.run( + ["git", "-C", str(root), "ls-files", "--error-unmatch", + f"generated/{boot}_dispatch.c"], + capture_output=True, text=True, check=False, + ) + tracked = r.returncode == 0 + except OSError: + tracked = False + if not marker.is_file(): + checks.append( + CheckResult( + id="generated_committed", + title="Committed game C", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + f"generated/{boot}_dispatch.c is missing. Generate once from a legal " + "disc (psxrecomp_cli.py generate) and commit generated/." + ), + ) + ) + elif not tracked: + checks.append( + CheckResult( + id="generated_committed", + title="Committed game C", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + f"generated/{boot}_dispatch.c exists but is not tracked by git; " + "release CI checks out the repo and would find nothing to build. " + "git add generated." + ), + ) + ) + else: + checks.append( + CheckResult( + id="generated_committed", + title="Committed game C", + status=CheckStatus.PASS, + severity=Severity.REQUIRED, + detail=f"generated/{boot}_dispatch.c tracked.", + ) + ) + retail_c = sorted( + p.relative_to(root).as_posix() + for d in (root / "generated", root / "psxrecomp" / "generated") + if d.is_dir() + for p in d.glob("SCPH*_*.c") + ) + if retail_c: + checks.append( + CheckResult( + id="retail_bios_c", + title="Retail BIOS C absent", + status=CheckStatus.FAIL, + severity=Severity.REQUIRED, + detail=( + "Retail-BIOS-derived C present: " + ", ".join(retail_c[:4]) + + ". It must never be committed or shipped; only OpenBIOS links in a release." + ), ) ) @@ -946,7 +1086,7 @@ def audit_project(root: Path) -> AuditReport: if legacy_pack or "psxrecomp_add_runtime_target" in cmake or has_v4: layout = LayoutClass.LEGACY_PACKAGING elif fails: - if packager.is_file() or wizard_ok or (root / "codegen_setup.c").is_file(): + if packager is not None or wizard_ok or (root / "codegen_setup.c").is_file(): layout = LayoutClass.SETUP_HOST_PARTIAL else: layout = LayoutClass.UNKNOWN @@ -958,7 +1098,7 @@ def audit_project(root: Path) -> AuditReport: if boot is None: notes.append("Could not parse [game].exe from game.toml — set --boot-exe when applying.") notes.append( - "Policy: setup-host releases only (no prebuilt generated-C zips)." + "Policy: bundled releases -- generated/ is committed and CI ships the compiled game." ) return AuditReport( diff --git a/tools/new_project_layout/project_studio/gitops.py b/tools/new_project_layout/project_studio/gitops.py index 8e31f29b..13745050 100644 --- a/tools/new_project_layout/project_studio/gitops.py +++ b/tools/new_project_layout/project_studio/gitops.py @@ -1463,7 +1463,7 @@ def install_and_push_release_ci( push_remote: bool = True, dry_run: bool = False, ) -> CmdResult: - """Write psxrecomp setup-release.yml (+ packager), commit, and push. + """Write psxrecomp game-release.yml (+ packager), commit, and push. Uses the same template as ``setup_project --enable-ci`` / ``op_emit_ci_workflow``. After push, best-effort checks that Actions has @@ -1493,7 +1493,7 @@ def install_and_push_release_ci( if not paths and not dry_run: # Already present — still allow push of any uncommitted workflow edits. wf = root / ".github" / "workflows" / "release.yml" - pkg = root / "scripts" / "package_setup_release.sh" + pkg = root / "scripts" / "package_release.sh" for p in (wf, pkg): if p.is_file(): rel = str(p.relative_to(root)).replace("\\", "/") @@ -1538,7 +1538,7 @@ def install_and_push_release_ci( root, "commit", "-m", - "ci: add setup-host release.yml (psxrecomp template)", + "ci: add bundled release.yml (psxrecomp template)", ) if code != 0: return CmdResult(False, "git commit failed", err or out) diff --git a/tools/new_project_layout/project_studio/gui.py b/tools/new_project_layout/project_studio/gui.py index ce3d266e..67d412a3 100644 --- a/tools/new_project_layout/project_studio/gui.py +++ b/tools/new_project_layout/project_studio/gui.py @@ -523,7 +523,7 @@ class ProjectStudioApp: ).pack(side="left") ctk.CTkLabel( header, - text="Migrate, audit, and GitHub ops for setup-host game repos", + text="Migrate, audit, and GitHub ops for PSXRecomp game repos", text_color=("gray40", "gray65"), font=ctk.CTkFont(size=13), ).pack(side="left", padx=(12, 0), pady=(6, 0)) @@ -657,7 +657,7 @@ class ProjectStudioApp: opts.pack(fill="x", padx=4, pady=4) ctk.CTkLabel( opts, - text="Options · setup-host only", + text="Options · bundled releases", font=ctk.CTkFont(size=13, weight="bold"), ).pack(anchor="w", padx=12, pady=(10, 4)) @@ -707,7 +707,7 @@ class ProjectStudioApp: ctk.CTkLabel( opts, - text="Wizard + recomp-ui are always enabled. Releases are setup-host only (no prebuilt game C).", + text="Wizard + recomp-ui are always enabled. Releases build the committed generated/ C and ship the compiled game.", text_color=("gray40", "gray60"), font=ctk.CTkFont(size=12), wraplength=900, @@ -1935,7 +1935,7 @@ class ProjectStudioApp: ).pack(side="left") ctk.CTkLabel( rel, - text="Install & push CI writes psxrecomp setup-release.yml → " + text="Install & push CI writes psxrecomp game-release.yml → " ".github/workflows/release.yml, commits, pushes, and registers Actions.", text_color=("gray30", "gray70"), anchor="w", @@ -5000,7 +5000,7 @@ class ProjectStudioApp: elif not self._git_dry(): if not messagebox.askyesno( "Project Studio", - "Install psxrecomp setup-release.yml as .github/workflows/release.yml,\n" + "Install psxrecomp game-release.yml as .github/workflows/release.yml,\n" "commit, push to origin, and register Actions?\n\n" f"{root}", parent=self.root, diff --git a/tools/new_project_layout/project_studio/models.py b/tools/new_project_layout/project_studio/models.py index 94ad8eb1..49af40fa 100644 --- a/tools/new_project_layout/project_studio/models.py +++ b/tools/new_project_layout/project_studio/models.py @@ -76,7 +76,7 @@ class AuditReport: @dataclass class MigrateOptions: - """User choices for apply (setup-host exclusively).""" + """User choices for apply (bundled releases: committed generated/ C).""" disc: str | None = None project_name: str | None = None diff --git a/tools/new_project_layout/project_studio/naming.py b/tools/new_project_layout/project_studio/naming.py index 82c9c638..24201e4a 100644 --- a/tools/new_project_layout/project_studio/naming.py +++ b/tools/new_project_layout/project_studio/naming.py @@ -170,7 +170,7 @@ def netplay_configured(root: Path) -> bool: def ci_workflow_present(root: Path) -> bool: - """True if setup-host ``.github/workflows/release.yml`` exists.""" + """True if ``.github/workflows/release.yml`` exists.""" root = root.expanduser().resolve() return (root / ".github" / "workflows" / "release.yml").is_file() diff --git a/tools/new_project_layout/project_studio/ops.py b/tools/new_project_layout/project_studio/ops.py index b8e98b32..4bc2f93b 100644 --- a/tools/new_project_layout/project_studio/ops.py +++ b/tools/new_project_layout/project_studio/ops.py @@ -1,4 +1,4 @@ -"""Apply migration plan steps (setup-host exclusively).""" +"""Apply migration plan steps (bundled releases: committed generated/ C, compiled game shipped).""" from __future__ import annotations @@ -23,7 +23,14 @@ from .naming import ( window_title_from_cmake, window_title_from_name, ) -from .paths import ci_setup_release_template, psxrecomp_root_from_toolkit, templates_dir, toolkit_dir +from .paths import ci_release_template, psxrecomp_root_from_toolkit, templates_dir, toolkit_dir + +# The title's packager wrapper (bundled release) and the retired setup-host one. +PACKAGER_WRAPPER = "package_release.sh" +LEGACY_PACKAGER_WRAPPER = "package_setup_release.sh" +# A .gitignore line that hides the release's inputs: `/generated/`, `generated/`, +# `generated`, with optional surrounding whitespace. +_GENERATED_IGNORE_RE = re.compile(r"^\s*/?generated/?\s*$") def _run(cmd: list[str], cwd: Path, dry_run: bool) -> tuple[bool, str]: @@ -660,8 +667,9 @@ def op_merge_gitignore(root: Path, options: MigrateOptions) -> ApplyResult: dst = root / ".gitignore" existing = dst.read_text(encoding="utf-8") if dst.is_file() else "" + # generated/ is deliberately NOT here: releases build from the committed + # game C, so an ignore rule for it is removed below, never added. required = [ - "/generated/", "/disc/", "/bios/", "/dist/", @@ -684,21 +692,35 @@ def op_merge_gitignore(root: Path, options: MigrateOptions) -> ApplyResult: _write(dst, template if not template.startswith("@") else "\n".join(required) + "\n", options.dry_run) return ApplyResult("merge_gitignore", True, "Wrote .gitignore from template", [".gitignore"]) - if not additions: + # The setup-host layout ignored generated/; a bundled release is built + # from it, so that rule hides the release's inputs from CI. Drop it. + kept_lines = [] + dropped = 0 + for line in existing.splitlines(): + if _GENERATED_IGNORE_RE.match(line): + dropped += 1 + continue + kept_lines.append(line) + if dropped: + existing = "\n".join(kept_lines) + ("\n" if existing.endswith("\n") else "") + + if not additions and not dropped: return ApplyResult("merge_gitignore", True, "No gitignore changes needed", []) - block = ( - "\n# --- PSXRecomp setup-host (project studio) ---\n" - + "\n".join(additions) - + "\n" - ) - _write(dst, existing.rstrip() + "\n" + block, options.dry_run) - return ApplyResult( - "merge_gitignore", - True, - f"Appended {len(additions)} ignore rules", - [".gitignore"], - ) + text = existing.rstrip() + "\n" + if additions: + text += ( + "\n# --- PSXRecomp (project studio) ---\n" + + "\n".join(additions) + + "\n" + ) + _write(dst, text, options.dry_run) + parts = [] + if additions: + parts.append(f"appended {len(additions)} ignore rule(s)") + if dropped: + parts.append("un-ignored generated/ (releases build from the committed game C)") + return ApplyResult("merge_gitignore", True, "; ".join(parts).capitalize(), [".gitignore"]) def op_emit_mods_preloaded(root: Path, options: MigrateOptions) -> ApplyResult: @@ -851,9 +873,8 @@ def op_ensure_app_icon(root: Path, options: MigrateOptions) -> ApplyResult: def op_rewrite_cmake_setup_host(root: Path, options: MigrateOptions) -> ApplyResult: tokens = _resolve_tokens(root, options) - # Force wizard for setup-host policy + # The wizard is the shipped game's first-run disc picker; keep it on. if not options.enable_wizard and options.enable_recomp_ui: - # Still rewrite with wizard ON — exclusive setup-host options = MigrateOptions(**{**options.to_dict(), "enable_wizard": True}) tokens = _resolve_tokens(root, options) @@ -886,7 +907,7 @@ def op_rewrite_cmake_setup_host(root: Path, options: MigrateOptions) -> ApplyRes if extras_bits and not options.dry_run: extras_note.write_text( - "Preserved notes after setup-host CMake rewrite:\n\n" + "Preserved notes after the CMake rewrite:\n\n" + "\n".join(f"- {b}" for b in extras_bits) + "\n\nFull previous file: CMakeLists.txt.pre_migrate.bak\n", encoding="utf-8", @@ -896,25 +917,25 @@ def op_rewrite_cmake_setup_host(root: Path, options: MigrateOptions) -> ApplyRes return ApplyResult( "rewrite_cmake_setup_host", True, - "Wrote setup-host CMakeLists.txt (backup .pre_migrate.bak)", + "Wrote CMakeLists.txt (psxrecomp_add_game_runtime; backup .pre_migrate.bak)", changed, ) def op_emit_packager(root: Path, options: MigrateOptions) -> ApplyResult: tokens = _resolve_tokens(root, options) - dst = root / "scripts" / "package_setup_release.sh" + dst = root / "scripts" / PACKAGER_WRAPPER if dst.is_file() and not options.force: return ApplyResult("emit_packager", True, "Packager already present", []) - _fill(templates_dir() / "package_setup_release.sh.in", dst, tokens, options.dry_run) + _fill(templates_dir() / (PACKAGER_WRAPPER + ".in"), dst, tokens, options.dry_run) if not options.dry_run: dst.chmod(dst.stat().st_mode | 0o111) - return ApplyResult( - "emit_packager", - True, - "Wrote scripts/package_setup_release.sh", - ["scripts/package_setup_release.sh"], - ) + msg = f"Wrote scripts/{PACKAGER_WRAPPER} (bundled release: compiled game)" + legacy = root / "scripts" / LEGACY_PACKAGER_WRAPPER + if legacy.is_file(): + msg += (f"; scripts/{LEGACY_PACKAGER_WRAPPER} is the retired setup-host " + "wrapper and can be deleted") + return ApplyResult("emit_packager", True, msg, [f"scripts/{PACKAGER_WRAPPER}"]) _PACKAGER_BLURB = { @@ -937,13 +958,24 @@ def op_sync_packager_project_dirs(root: Path, options: MigrateOptions) -> ApplyR routinely hand-extended with extra --project-file lines, and rewriting from the template would silently drop them. """ - from .detect import packager_missing_optional_paths, packager_missing_paths + from .detect import ( + packager_missing_optional_paths, + packager_missing_paths, + packager_wrapper, + packager_is_bundled, + ) - dst = root / "scripts" / "package_setup_release.sh" - if not dst.is_file(): + dst = packager_wrapper(root) + if dst is None: return ApplyResult( - "sync_packager_project_dirs", False, "No scripts/package_setup_release.sh", [] + "sync_packager_project_dirs", False, f"No scripts/{PACKAGER_WRAPPER}", [] ) + if packager_is_bundled(dst): + # A bundled release ships the compiled game and no sources, so there + # is no source allowlist to keep in step with CMakeLists.txt. + return ApplyResult( + "sync_packager_project_dirs", True, + "Bundled packager ships no sources; nothing to stage", []) missing = sorted(set(packager_missing_paths(root)) | set(packager_missing_optional_paths(root))) if not missing: @@ -977,11 +1009,12 @@ def op_sync_packager_project_dirs(root: Path, options: MigrateOptions) -> ApplyR text = text.replace(anchor, "\n" + block + anchor, 1) _write(dst, text, options.dry_run) + rel = dst.relative_to(root).as_posix() return ApplyResult( "sync_packager_project_dirs", True, - "Staged " + ", ".join(added) + " in scripts/package_setup_release.sh", - ["scripts/package_setup_release.sh"], + "Staged " + ", ".join(added) + f" in {rel}", + [rel], ) @@ -1015,12 +1048,13 @@ def _ci_steps_missing_from(installed: Path, template: Path) -> list[str]: def op_emit_ci_workflow(root: Path, options: MigrateOptions) -> ApplyResult: tokens = _resolve_tokens(root, options) - src = ci_setup_release_template(root) + src = ci_release_template(root) if src is None: return ApplyResult( "emit_ci_workflow", False, - "Cannot find docs/ci/templates/setup-release.yml (need psxrecomp submodule)", + "Cannot find docs/ci/templates/game-release.yml (need a psxrecomp " + "submodule pinned at or after bundled releases)", [], ) dst = root / ".github" / "workflows" / "release.yml" @@ -1054,36 +1088,37 @@ def op_emit_ci_workflow(root: Path, options: MigrateOptions) -> ApplyResult: return ApplyResult( "emit_ci_workflow", True, - "Wrote .github/workflows/release.yml (setup-host)", + "Wrote .github/workflows/release.yml (bundled release: builds the committed generated/ C)", [".github/workflows/release.yml"], ) def op_annotate_legacy_packaging(root: Path, options: MigrateOptions) -> ApplyResult: - note = root / "packaging" / "SETUP_HOST_MIGRATION.txt" + note = root / "packaging" / "PACKAGING_MIGRATION.txt" text = ( - "This title is migrating to setup-host releases only.\n" - "Do not ship prebuilt generated game C.\n" - "Use scripts/package_setup_release.sh + .github/workflows/release.yml\n" - "(see psxrecomp/docs/GAME_PROJECT_SETUP.md and docs/ci/HOST_ONLY_RELEASES.md).\n" - "Legacy package_release scripts in this folder are obsolete for public releases.\n" + "This title releases through the shared bundled-release flow:\n" + f"scripts/{PACKAGER_WRAPPER} + .github/workflows/release.yml build the\n" + "COMMITTED generated/ game C and ship the compiled game\n" + "(see psxrecomp/docs/GAME_PROJECT_SETUP.md and docs/ci/BUNDLED_RELEASES.md).\n" + "Hand-forked packagers in this folder (package_release.ps1,\n" + "package_appimage.sh, ...) are superseded and can be deleted once CI is green.\n" ) if (root / "packaging").is_dir(): _write(note, text, options.dry_run) return ApplyResult( "annotate_legacy_packaging", True, - "Wrote packaging/SETUP_HOST_MIGRATION.txt", - ["packaging/SETUP_HOST_MIGRATION.txt"], + "Wrote packaging/PACKAGING_MIGRATION.txt", + ["packaging/PACKAGING_MIGRATION.txt"], ) # tools-only legacy - tools_note = root / "tools" / "SETUP_HOST_MIGRATION.txt" + tools_note = root / "tools" / "PACKAGING_MIGRATION.txt" _write(tools_note, text, options.dry_run) return ApplyResult( "annotate_legacy_packaging", True, - "Wrote tools/SETUP_HOST_MIGRATION.txt", - ["tools/SETUP_HOST_MIGRATION.txt"], + "Wrote tools/PACKAGING_MIGRATION.txt", + ["tools/PACKAGING_MIGRATION.txt"], ) diff --git a/tools/new_project_layout/project_studio/paths.py b/tools/new_project_layout/project_studio/paths.py index 0838ead3..94bd146b 100644 --- a/tools/new_project_layout/project_studio/paths.py +++ b/tools/new_project_layout/project_studio/paths.py @@ -23,17 +23,28 @@ def psxrecomp_root_from_toolkit() -> Path | None: return None -def ci_setup_release_template(game_root: Path | None = None) -> Path | None: - """Prefer game submodule copy; fall back to toolkit's parent psxrecomp.""" +CI_RELEASE_TEMPLATE = "game-release.yml" + + +def ci_release_template(game_root: Path | None = None) -> Path | None: + """The bundled-release workflow template (docs/ci/templates/game-release.yml). + + Prefer the game's own psxrecomp submodule so the workflow matches the + framework the project pins; fall back to the toolkit's parent psxrecomp. + A submodule pinned before bundled releases has no such template, and the + caller says so rather than silently installing the retired setup-host one. + """ if game_root is not None: - p = game_root / "psxrecomp" / "docs" / "ci" / "templates" / "setup-release.yml" - if p.is_file(): - return p - p = game_root / "psxrecomp-v4" / "docs" / "ci" / "templates" / "setup-release.yml" - if p.is_file(): - return p + for sub in ("psxrecomp", "psxrecomp-v4"): + p = game_root / sub / "docs" / "ci" / "templates" / CI_RELEASE_TEMPLATE + if p.is_file(): + return p root = psxrecomp_root_from_toolkit() if root is None: return None - p = root / "docs" / "ci" / "templates" / "setup-release.yml" + p = root / "docs" / "ci" / "templates" / CI_RELEASE_TEMPLATE return p if p.is_file() else None + + +# Retired name, kept so a caller pinned to it still resolves. +ci_setup_release_template = ci_release_template diff --git a/tools/new_project_layout/project_studio/plan.py b/tools/new_project_layout/project_studio/plan.py index 000da746..cc8b71ff 100644 --- a/tools/new_project_layout/project_studio/plan.py +++ b/tools/new_project_layout/project_studio/plan.py @@ -42,16 +42,16 @@ _OP_TITLES = { "emit_version": "Emit VERSION", "emit_symbols_toml": "Emit symbols.toml stub", "emit_sync_symbols": "Install tools/sync_symbols.py", - "merge_gitignore": "Merge setup-host .gitignore rules", + "merge_gitignore": "Merge .gitignore rules (un-ignore generated/)", "emit_mods_preloaded": "Stub mods/preloaded catalog", "relocate_boxart": "Relocate boxart → launcher_assets/", "emit_boxart_stub": "Create launcher_assets stub dir", "ensure_app_icon": "Install assets/psxrecomp app icon", - "rewrite_cmake_setup_host": "Rewrite CMakeLists.txt (setup-host)", - "emit_packager": "Emit scripts/package_setup_release.sh", + "rewrite_cmake_setup_host": "Rewrite CMakeLists.txt (psxrecomp_add_game_runtime)", + "emit_packager": "Emit scripts/package_release.sh (bundled release)", "sync_packager_project_dirs": "Stage src/ + mods/ the release zip is missing", - "emit_ci_workflow": "Emit setup-host release.yml", - "annotate_legacy_packaging": "Annotate legacy prebuilt packaging", + "emit_ci_workflow": "Emit bundled release.yml", + "annotate_legacy_packaging": "Annotate hand-forked packaging scripts", "probe_disc_refresh": "Refresh disc identity via probe_disc.py", "record_framework_pins": "Write framework_pins.txt", "patch_readme_metrics": "Patch README badges, Retro Launcher, and R.A.I.D. footer", @@ -101,7 +101,7 @@ def build_plan( if options.enable_ci: wanted.add("emit_ci_workflow") - # Always ensure wizard/codegen for setup-host policy when rewriting + # Always ensure wizard/codegen (first-run disc picker) when rewriting if "rewrite_cmake_setup_host" in wanted: wanted.add("emit_codegen_setup") diff --git a/tools/new_project_layout/setup_project.ps1 b/tools/new_project_layout/setup_project.ps1 index bd1e6875..51db2ee6 100644 --- a/tools/new_project_layout/setup_project.ps1 +++ b/tools/new_project_layout/setup_project.ps1 @@ -196,7 +196,7 @@ if (-not $useRecompUi) { } Write-Host " (recomp-ui declined -- skipping wizard/netplay; PSX_RECOMP_UI=OFF)" } else { - # Default ON (interactive + non-interactive): setup-host CI needs the wizard. + # Default ON (interactive + non-interactive): the wizard is the first-run disc picker. $useWizard = Resolve-BoolOpt -Enable:$EnableWizard -No:$NoWizard -PromptDefault:$true ` -Question "Enable first-run setup wizard + Generate & rebuild?" if (-not $interactive -and -not $NoWizard) { $useWizard = $true } @@ -230,7 +230,7 @@ $useCi = Resolve-BoolOpt -Enable:$EnableCi -No:$NoCi -PromptDefault:$true ` $doBoxart = Resolve-BoolOpt -Enable:$FetchBoxart -No:$NoFetchBoxart -PromptDefault:$true ` -Question "Fetch libretro boxart now? (needs network)" $doGenerate = Resolve-BoolOpt -Enable:$Generate -No:$NoGenerate -PromptDefault:$false ` - -Question "Run Generate now (emitters + OpenBIOS + game C)?" + -Question "Run Generate now (emitters + OpenBIOS + game C; generated/ is committed for release CI)?" $doBuild = $false if ($doGenerate) { @@ -476,11 +476,11 @@ if ($useNetplay -and -not (Test-Path "psxrecomp\lib\recomp-net\CMakeLists.txt")) } Write-Host "== Packager stub ==" -Fill-Template (Join-Path $TemplateDir "package_setup_release.sh.in") (Join-Path $Root "scripts\package_setup_release.sh") +Fill-Template (Join-Path $TemplateDir "package_release.sh.in") (Join-Path $Root "scripts\package_release.sh") if ($useCi) { Write-Host "== CI release workflow ==" - $WfSrc = Join-Path $Root "psxrecomp\docs\ci\templates\setup-release.yml" + $WfSrc = Join-Path $Root "psxrecomp\docs\ci\templates\game-release.yml" $WfDst = Join-Path $Root ".github\workflows\release.yml" if (Test-Path -LiteralPath $WfSrc) { New-Item -ItemType Directory -Force -Path (Split-Path -Parent $WfDst) | Out-Null @@ -731,7 +731,15 @@ if ($doGenerate) { } python @genArgs $GeneratedOk = $true - Write-Host " generate OK (generated/ is gitignored -- not committed)" + # Release CI builds the compiled game from this tree, so it is part + # of the repo, not a local by-product. + git add generated 2>$null + git -c user.email=setup@localhost -c user.name=setup commit -q -m "Add generated game C" 2>$null + if ($LASTEXITCODE -eq 0) { + Write-Host " generate OK (generated/ committed -- release CI builds from it)" + } else { + Write-Host " generate OK -- commit generated/ before tagging a release: git add generated" + } } catch { Write-Warning "Generate failed -- fix seeds/disc and re-run generate by hand." $doBuild = $false @@ -845,16 +853,17 @@ Build playable runtime: "@ } else { @" -Build emitters, Generate, then playable runtime: +Build emitters, Generate (and commit it -- release CI builds from generated/), then playable runtime: bash psxrecomp/tools/ci/build_emitters.sh python psxrecomp\psxrecomp_cli.py generate --config game.toml --project-root . --disc $GenDiscHint + git add generated; git commit -m "Add generated game C" cmake -S . -B build-release -G Ninja -DCMAKE_BUILD_TYPE=Release cmake --build build-release --target psx-runtime "@ } $CiNote = if ($useCi) { - $n = "CI: .github/workflows/release.yml ready (zip prefix=$ZipPrefix; submodule gitlinks pin the build)." + $n = "CI: .github/workflows/release.yml ready (zip prefix=$ZipPrefix; builds the committed generated/ C into the shipped game; submodule gitlinks pin the build)." if ($githubPushed) { $n += " Pushed -- open Actions -> Release builds (workflow_dispatch)." } $n } else { diff --git a/tools/new_project_layout/setup_project.sh b/tools/new_project_layout/setup_project.sh index 7d15f2ae..6cc6337b 100755 --- a/tools/new_project_layout/setup_project.sh +++ b/tools/new_project_layout/setup_project.sh @@ -342,8 +342,8 @@ if [ "$ENABLE_RECOMP_UI" -eq 0 ]; then echo " (recomp-ui declined — skipping wizard/netplay; PSX_RECOMP_UI=OFF)" else if [ "$SET_WIZARD" -eq 0 ]; then - # Default ON: setup-host CI zips need the wizard (FORCE_SETUP_HOST - # without PSX_SETUP_WIZARD opens no first-run UI). + # Default ON: the wizard is the first-run disc picker of the shipped + # (compiled) game. if [ "$YES_MODE" -eq 1 ] || ! is_tty; then ENABLE_WIZARD=1 else @@ -388,14 +388,6 @@ if [ "$SET_CI" -eq 0 ]; then fi fi -# Setup-host release CI needs PSX_SETUP_WIZARD (FORCE_SETUP_HOST alone is not enough). -if [ "${ENABLE_CI:-0}" -eq 1 ] && [ "${ENABLE_RECOMP_UI:-0}" -eq 1 ] && - [ "${ENABLE_WIZARD:-0}" -eq 0 ]; then - echo "error: release CI requires the first-run setup wizard." >&2 - echo " Pass --enable-wizard (or omit --no-wizard)." >&2 - exit 1 -fi - if [ "$SET_BOXART" -eq 0 ]; then if [ "$YES_MODE" -eq 1 ] || ! is_tty; then FETCH_BOXART_FLAG=0 @@ -408,7 +400,7 @@ if [ "$SET_GENERATE" -eq 0 ]; then if [ "$YES_MODE" -eq 1 ] || ! is_tty; then DO_GENERATE=0 else - prompt_yn "Run Generate now (emitters + OpenBIOS + game C)?" DO_GENERATE 0 + prompt_yn "Run Generate now (emitters + OpenBIOS + game C; generated/ is committed for release CI)?" DO_GENERATE 0 fi fi @@ -706,14 +698,14 @@ if [ "$ENABLE_NETPLAY" -eq 1 ] && [ ! -f psxrecomp/lib/recomp-net/CMakeLists.txt fi echo "== Packager stub ==" -fill_template "$TEMPLATE_DIR/package_setup_release.sh.in" \ - "$ROOT/scripts/package_setup_release.sh" -chmod +x "$ROOT/scripts/package_setup_release.sh" +fill_template "$TEMPLATE_DIR/package_release.sh.in" \ + "$ROOT/scripts/package_release.sh" +chmod +x "$ROOT/scripts/package_release.sh" CI_WORKFLOW_OK=0 if [ "$ENABLE_CI" -eq 1 ]; then echo "== CI release workflow ==" - WF_SRC="$ROOT/psxrecomp/docs/ci/templates/setup-release.yml" + WF_SRC="$ROOT/psxrecomp/docs/ci/templates/game-release.yml" if [ -f "$WF_SRC" ]; then mkdir -p "$ROOT/.github/workflows" python3 "$FILL_TOKENS" "$WF_SRC" "$ROOT/.github/workflows/release.yml" \ @@ -964,7 +956,15 @@ if [ "$DO_GENERATE" -eq 1 ]; then python3 psxrecomp/psxrecomp_cli.py generate "$@" ) && GENERATED_OK=1 if [ "$GENERATED_OK" -eq 1 ]; then - echo " generate OK (generated/ is gitignored — not committed)" + # Release CI builds the compiled game from this tree, so it is part + # of the repo, not a local by-product. + if git add generated 2>/dev/null && \ + git -c user.email=setup@localhost -c user.name=setup \ + commit -q -m "Add generated game C" 2>/dev/null; then + echo " generate OK (generated/ committed — release CI builds from it)" + else + echo " generate OK — commit generated/ before tagging a release: git add generated" + fi else echo "warning: generate failed — fix seeds/disc and re-run generate by hand." >&2 DO_BUILD=0 @@ -1086,17 +1086,18 @@ elif [ "$GENERATED_OK" -eq 1 ]; then $BUILD_HINT" else - STEP2="Build emitters, Generate, then playable runtime: + STEP2="Build emitters, Generate (and commit it — release CI builds from generated/), then playable runtime: ./psxrecomp/tools/ci/build_emitters.sh python3 psxrecomp/psxrecomp_cli.py generate \\ --config game.toml --project-root . --disc \"$GEN_DISC_HINT\" + git add generated && git commit -m 'Add generated game C' $BUILD_HINT" fi CI_NOTE="CI workflow not installed (declined)." if [ "$ENABLE_CI" -eq 1 ] && [ "$CI_WORKFLOW_OK" -eq 1 ]; then - CI_NOTE="CI: .github/workflows/release.yml ready (zip prefix=$ZIP_PREFIX; submodule gitlinks pin the build)." + CI_NOTE="CI: .github/workflows/release.yml ready (zip prefix=$ZIP_PREFIX; builds the committed generated/ C into the shipped game; submodule gitlinks pin the build)." if [ "$GITHUB_PUSHED" -eq 1 ]; then CI_NOTE="$CI_NOTE Pushed — open Actions → Release builds (workflow_dispatch)." fi diff --git a/tools/new_project_layout/templates/CMakeLists.txt.in b/tools/new_project_layout/templates/CMakeLists.txt.in index 2b43b0fb..f984ea95 100644 --- a/tools/new_project_layout/templates/CMakeLists.txt.in +++ b/tools/new_project_layout/templates/CMakeLists.txt.in @@ -13,8 +13,9 @@ project(@PROJECT_CMAKE_NAME@ LANGUAGES C CXX) # --- Opt-in product surfaces (OFF by default in the framework) ------------- # Scaffold prompts / flags fill these. Without recomp-ui, PSX_RECOMP_UI is OFF. -# Setup-host CI requires WIZARD_CMAKE_BLOCK / ENABLE_SETUP_WIZARD (FORCE_SETUP_HOST -# alone does not open first-run / Generate & rebuild). +# The wizard (WIZARD_CMAKE_BLOCK / ENABLE_SETUP_WIZARD) is the first-run disc +# picker; releases are compiled from the committed generated/ C, so it never +# needs to generate or rebuild anything on a player's machine. @RECOMP_UI_CMAKE_BLOCK@ @NETPLAY_CMAKE_BLOCK@ @WIZARD_CMAKE_BLOCK@ @@ -40,6 +41,9 @@ psxrecomp_add_game_runtime(psx-runtime MAX_PLAYERS @PLAYERS@ @BOXART_CMAKE_ARG@ @APP_ICON_CMAKE_ARG@ + # Committed game C (psxrecomp_cli.py generate; git add generated). Release + # CI configures with -DPSXRECOMP_REQUIRE_GAME_C=ON so a checkout without + # it fails instead of building a setup host. GEN_MARKER "generated/@BOOT_EXE@_dispatch.c" GEN_FULL_GLOB "generated/@BOOT_EXE@_full_*.c" @CODEGEN_SETUP_ARG@ diff --git a/tools/new_project_layout/templates/README.md.in b/tools/new_project_layout/templates/README.md.in index 1a4a7902..5c5c3233 100644 --- a/tools/new_project_layout/templates/README.md.in +++ b/tools/new_project_layout/templates/README.md.in @@ -25,9 +25,8 @@ Scaffolded with the New Project Layout. See <!-- retcomm-readme-launcher --> ## Retro Launcher -You can run this title **standalone** (release zip + the built-in recomp-ui -Generate & Build flow), or manage installs, updates, ROM/BIOS wiring, and queued -builds more intuitively with +You can run this title **standalone** (download the release zip, point it at +your disc, play), or manage installs, updates, and disc/BIOS wiring with **[Retro Launcher](https://github.com/RetroPortingToolKit/Retro-Launcher)** — the Retro Compilation Manager hub for self-compiling recomps. @@ -42,16 +41,19 @@ the Retro Compilation Manager hub for self-compiling recomps. <img src="https://raw.githubusercontent.com/RetroPortingToolKit/Retro-Launcher/main/docs/screenshots/queue-and-background-build.png" alt="Background cmake build with titles queued" width="720"> </p> -Retro checks for updates, rebuilds with existing build data when possible, -shares the portable toolchain used by per-title launchers, and automates -BIOS/ROM/save plumbing so you are not stuck repeating each game’s wizard by hand. +Retro checks for updates, installs the prebuilt release zips, and automates +BIOS/ROM/save plumbing so you are not stuck repeating each game’s first run by hand. <!-- /retcomm-readme-launcher --> ## Legal You must own the original game. Disc images under `disc/` are gitignored and -must never be committed. Retail BIOS dumps are not redistributed; OpenBIOS is -used for Generate unless you supply your own SCPH locally. +must never be committed. Retail BIOS dumps are not redistributed and no C +derived from one may be committed; releases run on the bundled MIT OpenBIOS. + +`generated/` (the recompiled game C) **is committed**: releases ship the +compiled game, built by CI from that tree. Regenerate and commit it whenever +seeds or the framework pin change. Default app icon: `assets/psxrecomp.ico` (and `.png` / `.svg`) — Retro-themed controller mark from `psxrecomp/assets/`. Windows builds embed it via `APP_ICON`. @@ -66,11 +68,15 @@ git submodule update --init --recursive ./psxrecomp/tools/ci/build_emitters.sh python3 psxrecomp/psxrecomp_cli.py generate \ --config game.toml --project-root . --disc disc/<your>.cue +git add generated && git commit -m "Regenerate game C" cmake -S . -B build-release -G Ninja -DCMAKE_BUILD_TYPE=Release cmake --build build-release --target psx-runtime ``` -Zip prefix for CI artifacts: `@ZIP_PREFIX@`. +Releases: tag `vX.Y.Z` (or run the *Release builds* workflow). CI builds the +committed `generated/` C on Linux, Windows and macOS and attaches +`@ZIP_PREFIX@-<version>-<platform>.zip`, the compiled game. Locally: +`scripts/package_release.sh build-release linux-x64`. ## Symbols diff --git a/tools/new_project_layout/templates/gitignore.in b/tools/new_project_layout/templates/gitignore.in index 39906cdf..bb5ba543 100644 --- a/tools/new_project_layout/templates/gitignore.in +++ b/tools/new_project_layout/templates/gitignore.in @@ -7,8 +7,10 @@ /out/ /.happypath/ -# Setup-host: players Generate game C locally (CI clears generated/). -/generated/ +# generated/ is COMMITTED: releases ship the compiled game, and CI builds it +# from the game C in this repo (psxrecomp_cli.py generate, then git add +# generated). Never ignore it. Only the framework submodule's own generated/ +# (BIOS backends) stays untracked, and that is the submodule's rule. # psxrecomp_cli.py work areas (overlay toolchain download, static overlay # shard evidence -- the latter holds bytes read from the disc). /.cache/ diff --git a/tools/new_project_layout/templates/package_release.sh.in b/tools/new_project_layout/templates/package_release.sh.in new file mode 100644 index 00000000..805cd904 --- /dev/null +++ b/tools/new_project_layout/templates/package_release.sh.in @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Thin wrapper around the shared psxrecomp bundled-release packager. +# Autofilled by tools/new_project_layout/setup_project.{sh,ps1}. +# +# Ships the COMPILED game built from this repo's committed generated/ C: +# executable, runtime data, bundled OpenBIOS, mod catalog, overlay toolchain. +# No sources, emitters, CLI, generated C, or BIOS dumps. +# +# Usage: +# scripts/package_release.sh <build-dir> <artifact-tag> [recompiler-build-dir] +# +# Writes: dist/@ZIP_PREFIX@-<VERSION>-<artifact-tag>.zip +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +BUILD_DIR="${1:-}" +ARTIFACT_TAG="${2:-}" +RECOMPILER_BUILD="${3:-build-recompiler}" + +if [[ -z "${BUILD_DIR}" || -z "${ARTIFACT_TAG}" ]]; then + echo "usage: $0 <build-dir> <artifact-tag> [recompiler-build-dir]" >&2 + exit 2 +fi + +PACKAGER="${ROOT}/psxrecomp/tools/package_game_release.sh" +if [[ ! -f "${PACKAGER}" ]]; then + echo "error: missing ${PACKAGER} (psxrecomp submodule predates bundled releases -- bump it)" >&2 + exit 1 +fi +chmod +x "${PACKAGER}" 2>/dev/null || true + +EXTRA=() +# Overlay shard cache. It is compiled FROM THE DISC, so CI (no disc) cannot +# build one: the committed static AOT shard is linked into the executable and +# the bundled overlay_toolchain/ compiles the rest from the player's disc at +# runtime. A developer packaging locally with a cache can ship it instead: +# PSX_OVERLAY_CACHE_ROOT=/path/to/cache scripts/package_release.sh ... +if [[ -n "${PSX_OVERLAY_CACHE_ROOT:-}" ]]; then + EXTRA+=(--overlay-cache-root "${PSX_OVERLAY_CACHE_ROOT}") +else + EXTRA+=(--ship-without-overlay-cache-because \ + "no disc at package time: the committed static AOT shard is compiled in and overlay_toolchain/ autocompiles the rest on the player's machine") +fi +# Extra docs shipped at the zip root (DISC.md tells players which dump works). +for _doc in DISC.md; do + if [[ -f "${ROOT}/${_doc}" ]]; then + EXTRA+=(--doc "${_doc}") + fi +done + +cd "${ROOT}" +exec bash "${PACKAGER}" \ + --root "${ROOT}" \ + --build-dir "${BUILD_DIR}" \ + --artifact "${ARTIFACT_TAG}" \ + --zip-prefix @ZIP_PREFIX@ \ + --exe-name @EXE_BASENAME@ \ + --display-name "@WINDOW_TITLE@" \ + --recompiler-build "${RECOMPILER_BUILD}" \ + --version-env RELEASE_VERSION \ + --disc-hint "@DISC_HINT@" \ + "${EXTRA[@]}" diff --git a/tools/new_project_layout/templates/package_setup_release.sh.in b/tools/new_project_layout/templates/package_setup_release.sh.in deleted file mode 100644 index abe7083a..00000000 --- a/tools/new_project_layout/templates/package_setup_release.sh.in +++ /dev/null @@ -1,94 +0,0 @@ -#!/usr/bin/env bash -# Thin wrapper around the shared psxrecomp setup-host packager. -# Autofilled by tools/new_project_layout/setup_project.{sh,ps1}. -# -# Usage: -# scripts/package_setup_release.sh <build-dir> <artifact-tag> [recompiler-build-dir] -# -# Writes: dist/@ZIP_PREFIX@-<VERSION>-<artifact-tag>.zip -set -euo pipefail - -ROOT="$(cd "$(dirname "$0")/.." && pwd)" -BUILD_DIR="${1:-}" -ARTIFACT_TAG="${2:-}" -RECOMPILER_BUILD="${3:-build-recompiler}" - -if [[ -z "${BUILD_DIR}" || -z "${ARTIFACT_TAG}" ]]; then - echo "usage: $0 <build-dir> <artifact-tag> [recompiler-build-dir]" >&2 - exit 2 -fi - -# mods/ is staged by the framework packager by default (catalog next to the -# exe, plus the source catalog for the setup-host rebuild) -- no flag needed. -# --project-dir src ships game-owned trusted-plugin sources; CMakeLists.txt -# references them unconditionally, so the rebuild cannot configure without it. -# copy_proj skips a path that does not exist, so it is inert for titles with -# no src/. -PACKAGER="${ROOT}/psxrecomp/tools/package_setup_host.sh" -if [[ ! -f "${PACKAGER}" ]]; then - echo "error: missing ${PACKAGER} (psxrecomp submodule)" >&2 - exit 1 -fi -chmod +x "${PACKAGER}" 2>/dev/null || true - -EXTRA_PROJECT=() -if [[ -f "${ROOT}/catalog_identity.json" ]]; then - EXTRA_PROJECT+=(--project-file catalog_identity.json) -fi -if [[ -f "${ROOT}/framework_pins.txt" ]]; then - EXTRA_PROJECT+=(--project-file framework_pins.txt) -fi -# Optional in-game options defaults (titles that POST_BUILD-copy this file). -if [[ -f "${ROOT}/game_options.toml" ]]; then - EXTRA_PROJECT+=(--project-file game_options.toml) -fi -# Game-owned C that CMakeLists.txt compiles into the runtime — mod activation -# plugins live here (CODEGEN_SETUP_SOURCES "src/<game>_mods.c"). Leaving this -# out ships a CMakeLists that references a file the zip does not contain, so -# the build dies at configure with "Cannot find source file". -if [[ -d "${ROOT}/src" ]]; then - EXTRA_PROJECT+=(--project-dir src) -fi -# Preloaded mod packages (mods/preloaded/packages/<id>). The activation plugin -# above is compiled in, but selects itself through these marker files; without -# them the mod is built and never enabled. -if [[ -d "${ROOT}/mods" ]]; then - EXTRA_PROJECT+=(--project-dir mods) -fi -# Desktop/app icon sources. CMakeLists.txt passes APP_ICON -# "${CMAKE_CURRENT_SOURCE_DIR}/assets/psxrecomp.ico" unguarded, so a zip -# without assets/ cannot configure. runtime.cmake also stages the sibling -# psxrecomp.png from this directory as the SDL window icon, so the rebuilt exe -# would otherwise come out unbranded even if configure had survived. -if [[ -d "${ROOT}/assets" ]]; then - EXTRA_PROJECT+=(--project-dir assets) -fi -# Generated symbol headers. Game-owned C in src/ includes these by relative -# path ("../psx_symbols.h"), so they are compile inputs of the shipped tree even -# though nothing in CMakeLists.txt names them. -for _sym in psx_symbols.h psx_symbols_overlays.h; do - if [[ -f "${ROOT}/${_sym}" ]]; then - EXTRA_PROJECT+=(--project-file "${_sym}") - fi -done - -cd "${ROOT}" -exec bash "${PACKAGER}" \ - --root "${ROOT}" \ - --build-dir "${BUILD_DIR}" \ - --artifact "${ARTIFACT_TAG}" \ - --zip-prefix @ZIP_PREFIX@ \ - --exe-name @EXE_BASENAME@ \ - --display-name "@WINDOW_TITLE@" \ - --recompiler-build "${RECOMPILER_BUILD}" \ - --version-env RELEASE_VERSION \ - --disc-hint "@DISC_HINT@" \ - --project-file CMakeLists.txt \ - --project-file game.toml \ - --project-file VERSION \ - --project-file codegen_setup.c \ - --project-file codegen_setup.h \ - --project-file README.md \ - --project-dir seeds \ - --project-dir launcher_assets \ - "${EXTRA_PROJECT[@]}" diff --git a/tools/package_game_release.sh b/tools/package_game_release.sh new file mode 100755 index 00000000..a61d345f --- /dev/null +++ b/tools/package_game_release.sh @@ -0,0 +1,466 @@ +#!/usr/bin/env bash +# Bundled-release zip packager for PSXRecomp game repos: ships the COMPILED +# game, not a kit that compiles it. +# +# A release built from the title's committed generated/ C is a finished +# product. The zip carries the game executable and everything the runtime +# reads beside it -- assets, the mod catalog, the bundled OpenBIOS image and +# its notice, game.toml / game_options.toml, third-party notices -- plus the +# overlay toolchain so overlays outside the compiled static shard can still be +# turned native on the player's machine. It carries NO sources, NO emitters at +# the root, NO psxrecomp/ or recomp-ui/ tree, NO CLI, NO generated C and NO +# BIOS dump. The player supplies a legal disc and plays. +# +# Usage (from game repo root): +# psxrecomp/tools/package_game_release.sh \ +# --build-dir build-ci \ +# --artifact linux-x64 \ +# --zip-prefix bpe \ +# --exe-name Bomberman_Party_Edition_Recompiled \ +# --display-name "Bomberman Party Edition Recompiled" \ +# --recompiler-build build-recompiler \ +# [--runtime-file REL]... # extra exe-relative files (keybinds.ini, ...) +# [--runtime-dir NAME]... # extra exe-relative dirs (bezels, ...) +# [--doc REL]... # extra docs from the repo (DISC.md, ...) +# [--overlay-cache-root DIR] # ship a prebuilt overlay shard cache +# [--ship-without-overlay-cache-because REASON] +# [--no-overlay-toolchain] # do not bundle overlay_toolchain/ +# [--disc-hint "your legally owned disc"] [--version-env RELEASE_VERSION] +# +# Overlay cache. The runtime's overlay shard cache is compiled FROM THE DISC, +# so a CI job -- which has no disc -- cannot build one; the committed static +# AOT shard (generated/overlays_static*.c) is compiled into the executable +# instead, and the bundled overlay_toolchain/ compiles the rest from the +# player's own disc at runtime. A developer packaging locally with a cache +# passes --overlay-cache-root and it is staged through release_stage.py +# (which owns the cache tag and layout; nothing here knows either). Without a +# cache the caller must say why, in words, and the reason is printed: this is +# the same deliberately unwieldy switch release_stage.py exposes, because a +# packager that quietly staged nothing once looked like a good release twice. +# +# DEVELOPER CHANNEL. mods with channel = "developer" are pruned from public +# releases (EXCLUDE_DEV_MODS=1, which CI sets via $CI) and kept in local +# exports; --include-dev-mods / --exclude-dev-mods override. Pruning is by +# package directory via tools/mod_channel_filter.py, never by editing a +# manifest. +# +# Env: +# RELEASE_VERSION / <version-env> / VERSION file (must match binary stamp) +# PSXRECOMP_RUNTIME_BIN_DIR | BPE_RUNTIME_BIN_DIR (Windows MinGW DLL search) +# PSX_RELEASE_STAGE_PYTHON (python3 override) +# +# Lobby pin: the exe must have been built with current runtime.cmake so +# $<TARGET_FILE_DIR>/psx_game_version.txt exists. This script refuses to ship +# a VERSION that disagrees with that stamp (netplay list filter bug). +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +FRAMEWORK="$(cd "${SCRIPT_DIR}/.." && pwd)" +ROOT="$(pwd)" + +BUILD_DIR="" +ARTIFACT="" +ZIP_PREFIX="" +EXE_NAME="" +DISPLAY_NAME="" +RECOMPILER_BUILD="build-recompiler" +VERSION_ENV="RELEASE_VERSION" +DISC_HINT="your legally owned game disc" +RUNTIME_TARGET="psx-runtime" +RUNTIME_FILES=() +RUNTIME_DIRS=() +DOCS=() +OVERLAY_CACHE_ROOT="" +SHIP_WITHOUT_OVERLAY_CACHE_BECAUSE="" +SHIP_WITHOUT_OVERLAY_CACHE_KEY_BECAUSE="" +STAGE_OVERLAY_TOOLCHAIN=1 +if [[ -z "${EXCLUDE_DEV_MODS:-}" ]]; then + if [[ -n "${CI:-}" ]]; then EXCLUDE_DEV_MODS=1; else EXCLUDE_DEV_MODS=0; fi +fi +RUNTIME_BIN_DIR="${PSXRECOMP_RUNTIME_BIN_DIR:-${BPE_RUNTIME_BIN_DIR:-/usr/x86_64-w64-mingw32/bin}}" + +usage() { + sed -n '2,32p' "$0" | sed 's/^# \{0,1\}//' + exit "${1:-2}" +} + +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage 0 ;; + --build-dir) BUILD_DIR="${2:?}"; shift 2 ;; + --artifact) ARTIFACT="${2:?}"; shift 2 ;; + --zip-prefix) ZIP_PREFIX="${2:?}"; shift 2 ;; + --exe-name) EXE_NAME="${2:?}"; shift 2 ;; + --display-name) DISPLAY_NAME="${2:?}"; shift 2 ;; + --recompiler-build) RECOMPILER_BUILD="${2:?}"; shift 2 ;; + --runtime-target) RUNTIME_TARGET="${2:?}"; shift 2 ;; + --version-env) VERSION_ENV="${2:?}"; shift 2 ;; + --disc-hint) DISC_HINT="${2:?}"; shift 2 ;; + --runtime-file) RUNTIME_FILES+=("${2:?}"); shift 2 ;; + --runtime-dir) RUNTIME_DIRS+=("${2:?}"); shift 2 ;; + --doc) DOCS+=("${2:?}"); shift 2 ;; + --overlay-cache-root) OVERLAY_CACHE_ROOT="${2:?}"; shift 2 ;; + --ship-without-overlay-cache-because) SHIP_WITHOUT_OVERLAY_CACHE_BECAUSE="${2:?}"; shift 2 ;; + --ship-without-overlay-cache-key-because) SHIP_WITHOUT_OVERLAY_CACHE_KEY_BECAUSE="${2:?}"; shift 2 ;; + --no-overlay-toolchain) STAGE_OVERLAY_TOOLCHAIN=0; shift ;; + --exclude-dev-mods) EXCLUDE_DEV_MODS=1; shift ;; + --include-dev-mods) EXCLUDE_DEV_MODS=0; shift ;; + --runtime-bin) RUNTIME_BIN_DIR="${2:?}"; shift 2 ;; + --root) ROOT="${2:?}"; shift 2 ;; + *) + echo "error: unknown arg: $1" >&2 + usage 2 + ;; + esac +done + +if [[ -z "${BUILD_DIR}" || -z "${ARTIFACT}" || -z "${ZIP_PREFIX}" || -z "${EXE_NAME}" ]]; then + echo "error: --build-dir, --artifact, --zip-prefix, and --exe-name are required" >&2 + usage 2 +fi +[[ -n "${DISPLAY_NAME}" ]] || DISPLAY_NAME="${EXE_NAME}" + +ROOT="$(cd "${ROOT}" && pwd)" +if [[ -d "${ROOT}/${BUILD_DIR}" ]]; then + BUILD_DIR="$(cd "${ROOT}/${BUILD_DIR}" && pwd)" +elif [[ -d "${BUILD_DIR}" ]]; then + BUILD_DIR="$(cd "${BUILD_DIR}" && pwd)" +else + echo "error: build dir not found: ${BUILD_DIR}" >&2 + exit 1 +fi +if [[ -d "${ROOT}/${RECOMPILER_BUILD}" ]]; then + RECOMPILER_BUILD="$(cd "${ROOT}/${RECOMPILER_BUILD}" && pwd)" +elif [[ -d "${RECOMPILER_BUILD}" ]]; then + RECOMPILER_BUILD="$(cd "${RECOMPILER_BUILD}" && pwd)" +fi + +# The shared staging surface: cache tag / shards / toolchain / mod catalog are +# release_stage.py's, and only its. This script marshals arguments. +# shellcheck source=release_overlay_stage.sh +. "${SCRIPT_DIR}/release_overlay_stage.sh" +psx_release_stage_init "${FRAMEWORK}" +PY="${_psx_py}" + +# --- Version: the compile-time stamp is authoritative ------------------------ +REQUESTED="" +if [[ -n "${VERSION_ENV}" ]]; then REQUESTED="${!VERSION_ENV:-}"; fi +if [[ -z "${REQUESTED}" && -n "${RELEASE_VERSION:-}" ]]; then REQUESTED="${RELEASE_VERSION}"; fi +if [[ -z "${REQUESTED}" && -f "${ROOT}/VERSION" ]]; then REQUESTED="$(tr -d '[:space:]' <"${ROOT}/VERSION")"; fi +REQUESTED="$(printf '%s' "${REQUESTED}" | tr -d '[:space:]')" +REQUESTED="${REQUESTED#v}" + +EXE="" +for cand in "${BUILD_DIR}/${EXE_NAME}" "${BUILD_DIR}/${EXE_NAME}.exe" "${BUILD_DIR}/Release/${EXE_NAME}.exe"; do + if [[ -f "${cand}" ]]; then EXE="${cand}"; break; fi +done +if [[ -z "${EXE}" ]]; then + # CMake records the OUTPUT_NAME it chose; prefer it over the wrapper's copy + # so a renamed title does not read as "binary missing". + marker="${BUILD_DIR}/psxrecomp_exe_name-${RUNTIME_TARGET}.txt" + if [[ -f "${marker}" ]]; then + chosen="$(tr -d '[:space:]' <"${marker}")" + for cand in "${BUILD_DIR}/${chosen}" "${BUILD_DIR}/${chosen}.exe"; do + if [[ -f "${cand}" ]]; then EXE="${cand}"; break; fi + done + if [[ -n "${EXE}" ]]; then + echo "note: --exe-name ${EXE_NAME} does not match CMake OUTPUT_NAME ${chosen}; using the marker" >&2 + EXE_NAME="${chosen}" + fi + fi +fi +if [[ -z "${EXE}" ]]; then + echo "error: game executable '${EXE_NAME}' not found under ${BUILD_DIR}" >&2 + ls -la "${BUILD_DIR}" >&2 || true + exit 1 +fi +EXE_BASENAME="$(basename "${EXE}")" +EXE_DIR="$(dirname "${EXE}")" + +# A bundled release is a FULL build. The build publishes which it is: the +# PSX_HAS_GAME_DISPATCH define only exists when generated C was linked. +# Refuse to package a setup host as a game. +if [[ -f "${BUILD_DIR}/CMakeCache.txt" ]] && + grep -q '^PSXRECOMP_FORCE_SETUP_HOST:BOOL=ON' "${BUILD_DIR}/CMakeCache.txt"; then + echo "error: ${BUILD_DIR} was configured with PSXRECOMP_FORCE_SETUP_HOST=ON -- that is a setup host, not a game." >&2 + echo " Reconfigure with -DPSXRECOMP_REQUIRE_GAME_C=ON (generated/ committed) and rebuild." >&2 + exit 1 +fi +if ! grep -qs 'PSX_HAS_GAME_DISPATCH' "${BUILD_DIR}/build.ninja" "${BUILD_DIR}/compile_commands.json" \ + "${BUILD_DIR}/CMakeFiles/${RUNTIME_TARGET}.dir/flags.make" 2>/dev/null; then + echo "error: ${BUILD_DIR} did not compile generated game C (no PSX_HAS_GAME_DISPATCH in the build flags)." >&2 + echo " A bundled release links the committed generated/ tree; configure with -DPSXRECOMP_REQUIRE_GAME_C=ON." >&2 + exit 1 +fi + +normalize_ver() { local v; v="$(printf '%s' "${1:-}" | tr -d '[:space:]')"; printf '%s' "${v#v}"; } +BUILT="" +for stamp in "${EXE_DIR}/psx_game_version.txt" "${BUILD_DIR}/psx_game_version.txt" "${BUILD_DIR}/Release/psx_game_version.txt"; do + if [[ -f "${stamp}" ]]; then + BUILT="$(normalize_ver "$(cat "${stamp}")")" + echo "lobby pin stamp: ${stamp} -> ${BUILT}" + break + fi +done +if [[ -z "${BUILT}" ]]; then + echo "error: missing psx_game_version.txt next to ${EXE}" >&2 + echo " Rebuild with current psxrecomp runtime.cmake so the lobby pin is stamped." >&2 + exit 1 +fi +if [[ -n "${REQUESTED}" && "${REQUESTED}" != "${BUILT}" ]]; then + echo "error: RELEASE_VERSION/VERSION=${REQUESTED} but binary stamp=${BUILT}" >&2 + echo " Rebuild with -DPSX_GAME_VERSION=${REQUESTED} (or pin VERSION then reconfigure)." >&2 + exit 1 +fi +VERSION="${BUILT}" +printf '%s\n' "${VERSION}" >"${ROOT}/VERSION" + +DIST="${ROOT:?}/dist" +STAGE="${DIST:?}/stage-game-${ARTIFACT:?}" +ZIP_NAME="${ZIP_PREFIX:?}-${VERSION:?}-${ARTIFACT:?}.zip" +rm -rf "${STAGE:?}" +mkdir -p "${STAGE}" "${DIST}" +rm -f "${DIST:?}/${ZIP_NAME:?}" + +# --- The executable and what CMake staged beside it -------------------------- +cp -a "${EXE}" "${STAGE}/" +if [[ -f "${EXE_DIR}/psx_game_version.txt" ]]; then + cp -a "${EXE_DIR}/psx_game_version.txt" "${STAGE}/psx_game_version.txt" +else + printf '%s\n' "${VERSION}" >"${STAGE}/psx_game_version.txt" +fi + +if [[ "${EXE_BASENAME}" == *.exe ]]; then + shopt -s nullglob + for dll in "${EXE_DIR}"/*.dll "${EXE_DIR}"/*.DLL; do + cp -a "${dll}" "${STAGE}/" + echo "staged sibling DLL $(basename "${dll}")" + done + shopt -u nullglob +fi + +if [[ ! -d "${EXE_DIR}/assets/fonts" || ! -d "${EXE_DIR}/assets/img" ]]; then + echo "error: ${EXE_DIR}/assets/{fonts,img} missing -- rebuild ${RUNTIME_TARGET}" >&2 + exit 1 +fi +mkdir -p "${STAGE}/assets" +cp -a "${EXE_DIR}/assets/fonts" "${STAGE}/assets/" +cp -a "${EXE_DIR}/assets/img" "${STAGE}/assets/" +if [[ ! -f "${STAGE}/assets/img/boxart.tga" && -f "${ROOT}/launcher_assets/img/boxart.tga" ]]; then + cp -a "${ROOT}/launcher_assets/img/boxart.tga" "${STAGE}/assets/img/boxart.tga" +fi +# Window icon staged by runtime.cmake beside the exe (see APP_ICON). +shopt -s nullglob +for icon in "${EXE_DIR}"/psxrecomp.png "${EXE_DIR}"/*.png; do + cp -a "${icon}" "${STAGE}/" + break +done +shopt -u nullglob + +# Bundled OpenBIOS: runtime.cmake stages the exact image the compiled backend +# consumed plus its MIT notice at <exe>/bios/. The runtime resolves +# "bios/openbios.bin" relative to the executable, so ship the directory as a +# unit. Nothing else from bios/ may ship. +if [[ ! -f "${EXE_DIR}/bios/openbios.bin" || ! -f "${EXE_DIR}/bios/OpenBIOS.LICENSE" ]]; then + echo "error: ${EXE_DIR}/bios/{openbios.bin,OpenBIOS.LICENSE} missing -- the build did not link the OpenBIOS backend" >&2 + exit 1 +fi +mkdir -p "${STAGE}/bios" +cp -a "${EXE_DIR}/bios/openbios.bin" "${EXE_DIR}/bios/OpenBIOS.LICENSE" "${STAGE}/bios/" + +# Runtime configs: game.toml is what the runtime reads beside the exe; the +# build stages it there, so take the staged copy (it is the packaged config +# the cache tag derives from). game_options.toml is optional. +for cfg in game.toml game_options.toml; do + if [[ -f "${EXE_DIR}/${cfg}" ]]; then + cp -a "${EXE_DIR}/${cfg}" "${STAGE}/${cfg}" + elif [[ -f "${ROOT}/${cfg}" ]]; then + cp -a "${ROOT}/${cfg}" "${STAGE}/${cfg}" + fi +done +[[ -f "${STAGE}/game.toml" ]] || { echo "error: no game.toml to ship (build tree or repo root)" >&2; exit 1; } + +# Wave-5 F4: without overlay_cache = true the runtime never initialises the +# overlay loader and every streamed overlay runs interpreted for every player. +if ! grep -qE '^[[:space:]]*overlay_cache[[:space:]]*=[[:space:]]*true' "${STAGE}/game.toml"; then + if [[ -z "${SHIP_WITHOUT_OVERLAY_CACHE_KEY_BECAUSE}" ]]; then + echo "error: REFUSING TO PACKAGE: ${STAGE}/game.toml has no '[runtime] overlay_cache = true'." >&2 + echo " Add the key, or pass --ship-without-overlay-cache-key-because '<reason>'." >&2 + exit 1 + fi + echo "warning: packaging without overlay_cache = true (reason: ${SHIP_WITHOUT_OVERLAY_CACHE_KEY_BECAUSE})" >&2 +fi + +for rel in "${RUNTIME_FILES[@]+"${RUNTIME_FILES[@]}"}"; do + [[ -n "${rel}" ]] || continue + src="" + for c in "${EXE_DIR}/${rel}" "${ROOT}/${rel}"; do [[ -f "${c}" ]] && { src="${c}"; break; }; done + [[ -n "${src}" ]] || { echo "error: --runtime-file ${rel} not found (build tree or repo)" >&2; exit 1; } + mkdir -p "$(dirname "${STAGE}/${rel}")" + cp -a "${src}" "${STAGE}/${rel}" +done +for name in "${RUNTIME_DIRS[@]+"${RUNTIME_DIRS[@]}"}"; do + [[ -n "${name}" ]] || continue + [[ -d "${EXE_DIR}/${name}" ]] || { echo "error: --runtime-dir ${name} missing under ${EXE_DIR}" >&2; exit 1; } + mkdir -p "${STAGE}/${name}" + cp -a "${EXE_DIR}/${name}/." "${STAGE}/${name}/" + echo "staged runtime dir ${name}/" +done + +# --- Mod catalog, verified against what THIS build declared ------------------- +MANIFEST="$(find "${BUILD_DIR}" -maxdepth 3 -name "psx_mod_catalog_${RUNTIME_TARGET}.txt" -type f 2>/dev/null | head -n1 || true)" +mods_args=(--build-path "${EXE_DIR}" --stage "${STAGE}" --runtime-target "${RUNTIME_TARGET}") +[[ -n "${MANIFEST}" ]] && mods_args+=(--catalog-manifest "${MANIFEST}") +psx_add_mod_catalog "${mods_args[@]}" + +prune_dev_mods() { + if ! "${PY}" "${SCRIPT_DIR}/mod_channel_filter.py" "$@"; then + echo "error: developer-channel filtering failed" >&2 + exit 1 + fi +} +if [[ "${EXCLUDE_DEV_MODS}" -eq 1 ]]; then + echo "excluding developer-channel mods from this package" + prune_dev_mods "${STAGE}/mods/bundled" + left=$( { grep -rlE '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' \ + "${STAGE}/mods" --include=manifest.toml 2>/dev/null || true; } | wc -l) + if [[ "${left}" -ne 0 ]]; then + echo "error: ${left} developer manifest(s) survived pruning" >&2 + exit 1 + fi +else + n=$( { grep -rlE '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' \ + "${STAGE}/mods" --include=manifest.toml 2>/dev/null || true; } | wc -l) + [[ "${n}" -eq 0 ]] || echo "note: package includes ${n} developer-channel manifest(s); pass --exclude-dev-mods for a public release" +fi + +# --- Overlay shard cache (developer-local only) and toolchain ----------------- +GAME_BIN="" +for cand in "${RECOMPILER_BUILD}/psxrecomp-game" "${RECOMPILER_BUILD}/psxrecomp-game.exe" "${RECOMPILER_BUILD}/Release/psxrecomp-game.exe"; do + if [[ -f "${cand}" ]]; then GAME_BIN="${cand}"; break; fi +done + +if [[ -n "${OVERLAY_CACHE_ROOT}" ]]; then + [[ -n "${GAME_BIN}" ]] || { echo "error: --overlay-cache-root needs psxrecomp-game under ${RECOMPILER_BUILD} (cache tag derivation)" >&2; exit 1; } + game_id="$(sed -n 's/^[[:space:]]*id[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' "${STAGE}/game.toml" | head -1)" + [[ -n "${game_id}" ]] || { echo "error: [game] id missing from the staged game.toml (cache namespace)" >&2; exit 1; } + cg_tag="$(psx_overlay_cg_tag --runtime-include "${FRAMEWORK}/runtime/include" \ + --recompiler "${GAME_BIN}" --game-toml "${STAGE}/game.toml" \ + --flavor-from-build "${BUILD_DIR}" --runtime-target "${RUNTIME_TARGET}")" + [[ -n "${cg_tag}" ]] || { echo "error: could not derive the overlay cache tag" >&2; exit 1; } + psx_add_overlay_cache --game-id "${game_id}" --cache-src-root "${OVERLAY_CACHE_ROOT}" \ + --stage "${STAGE}" --cg-tag "${cg_tag}" +else + if [[ -z "${SHIP_WITHOUT_OVERLAY_CACHE_BECAUSE}" ]]; then + echo "error: no --overlay-cache-root given. A shard cache is compiled from the disc, so a" >&2 + echo " release without one relies on the compiled static shard plus runtime autocompile." >&2 + echo " State that in words: --ship-without-overlay-cache-because '<reason>'." >&2 + exit 1 + fi + echo "overlay cache: NONE shipped -- ${SHIP_WITHOUT_OVERLAY_CACHE_BECAUSE}" +fi + +if [[ "${STAGE_OVERLAY_TOOLCHAIN}" -eq 1 ]]; then + [[ -n "${GAME_BIN}" ]] || { echo "error: overlay toolchain needs psxrecomp-game under ${RECOMPILER_BUILD} (pass --recompiler-build, or --no-overlay-toolchain)" >&2; exit 1; } + case "${ARTIFACT}" in + windows-*) platform=win ;; + linux-*) platform=linux ;; + macos-arm64|macos-aarch64) platform=macos-arm64 ;; + macos-x64|macos-x86_64) platform=macos-x64 ;; + *) echo "error: cannot map artifact '${ARTIFACT}' to an overlay toolchain platform" >&2; exit 1 ;; + esac + tc_args=(--stage "${STAGE}" --recomp-dir "$(dirname "${GAME_BIN}")" \ + --recomp-tools "${FRAMEWORK}/tools" --recomp-include "${FRAMEWORK}/runtime/include" \ + --dl-cache "${DIST}/.dl-cache" --platform "${platform}") + [[ "${platform}" == win ]] && tc_args+=(--mingw-bin "${RUNTIME_BIN_DIR}") + psx_add_overlay_toolchain "${tc_args[@]}" +fi + +# --- Notices and docs --------------------------------------------------------- +mkdir -p "${STAGE}/licenses" +if [[ -d "${FRAMEWORK}/runtime/licenses" ]]; then + cp -a "${FRAMEWORK}/runtime/licenses/." "${STAGE}/licenses/" +fi +[[ -f "${FRAMEWORK}/THIRD_PARTY_ATTRIBUTION.md" ]] && cp -a "${FRAMEWORK}/THIRD_PARTY_ATTRIBUTION.md" "${STAGE}/licenses/" +[[ -f "${FRAMEWORK}/LICENSE" ]] && cp -a "${FRAMEWORK}/LICENSE" "${STAGE}/licenses/psxrecomp-LICENSE" +if [[ -z "$(ls -A "${STAGE}/licenses" 2>/dev/null)" ]]; then + echo "error: no third-party notices staged (${FRAMEWORK}/runtime/licenses empty?)" >&2 + exit 1 +fi +for d in "${DOCS[@]+"${DOCS[@]}"}"; do + [[ -n "${d}" ]] || continue + [[ -f "${ROOT}/${d}" ]] || { echo "error: --doc ${d} not found" >&2; exit 1; } + cp -a "${ROOT}/${d}" "${STAGE}/$(basename "${d}")" +done +mkdir -p "${STAGE}/saves" + +cat >"${STAGE}/README.txt" <<EOF +${DISPLAY_NAME} ${VERSION} +Platform: ${ARTIFACT} + +This is the compiled game. Nothing needs to be generated or built. + +1. Run ${EXE_BASENAME}. +2. On first run, point it at ${DISC_HINT}. The disc is never copied here. +3. Play. Saves land in saves/; settings beside the executable. + +The game runs on the bundled OpenBIOS (bios/, MIT licensed). No retail BIOS +is included or required. Third-party notices are in licenses/. + +overlay_toolchain/ lets the game turn code it streams from your disc into +native code on this machine; it is not a build kit and needs no action. +EOF + +# --- Gates: what must NOT be in a bundled release ------------------------------- +forbidden=() +for p in psxrecomp recomp-ui generated seeds CMakeLists.txt codegen_setup.c codegen_setup.h \ + psxrecomp_cli.py psxrecomp-game psxrecomp-game.exe psxrecomp-bios psxrecomp-bios.exe \ + toolchain disc src; do + [[ -e "${STAGE}/${p}" ]] && forbidden+=("${p}") +done +if [[ "${#forbidden[@]}" -gt 0 ]]; then + echo "error: bundled release stage contains build-kit content:" >&2 + printf ' %s\n' "${forbidden[@]}" >&2 + exit 1 +fi +bad_bios="$(find "${STAGE}" -type f \( -iname 'SCPH*' -o -iname '*.BIN' -o -iname '*.bin' \) \ + ! -path "${STAGE}/bios/openbios.bin" 2>/dev/null || true)" +if [[ -n "${bad_bios}" ]]; then + echo "error: BIOS image(s) other than the bundled OpenBIOS in the stage:" >&2 + printf '%s\n' "${bad_bios}" | sed 's/^/ /' >&2 + exit 1 +fi +bad_disc="$(find "${STAGE}" -type f \( -iname '*.cue' -o -iname '*.iso' -o -iname '*.img' -o -iname '*.chd' -o -iname '*.mcd' -o -iname '*.mcr' \) 2>/dev/null || true)" +if [[ -n "${bad_disc}" ]]; then + echo "error: disc / memory-card files in the stage:" >&2 + printf '%s\n' "${bad_disc}" | sed 's/^/ /' >&2 + exit 1 +fi +if find "${STAGE}" -type f -name '*.c' ! -path "${STAGE}/overlay_toolchain/*" | grep -q .; then + echo "error: C sources in the stage outside overlay_toolchain/:" >&2 + find "${STAGE}" -type f -name '*.c' ! -path "${STAGE}/overlay_toolchain/*" | sed 's/^/ /' >&2 + exit 1 +fi + +# --- Windows: DLLs beside the exe, then Authenticode ------------------------- +if [[ "${EXE_BASENAME}" == *.exe ]]; then + BUNDLE="${SCRIPT_DIR}/bundle_mingw_dlls.sh" + [[ -f "${BUNDLE}" ]] || { echo "error: missing ${BUNDLE}" >&2; exit 1; } + bash "${BUNDLE}" \ + --runtime-bin "${RUNTIME_BIN_DIR}" --search-dir "${EXE_DIR}" --search-dir "${BUILD_DIR}" \ + --exe "${STAGE}/${EXE_BASENAME}" --dest "${STAGE}" --label "${EXE_BASENAME}" \ + --require libgcc_s_seh-1.dll --require libstdc++-6.dll --require libwinpthread-1.dll \ + --require libssp-0.dll --require zlib1.dll --require z.dll + SIGN_SH="${SCRIPT_DIR}/ci/sign_windows.sh" + if [[ -f "${SIGN_SH}" ]]; then + bash "${SIGN_SH}" "${STAGE}" + else + echo "note: ${SIGN_SH} missing; Windows binaries ship unsigned" >&2 + fi +fi + +find "${STAGE}" -exec touch -c {} + 2>/dev/null || find "${STAGE}" -exec touch {} + + +"${PY}" "${SCRIPT_DIR}/create_release_zip.py" --source "${STAGE}" --output "${DIST}/${ZIP_NAME}" >/dev/null +echo "Wrote ${DIST}/${ZIP_NAME}" +du -h "${DIST}/${ZIP_NAME}" diff --git a/tools/package_setup_host.sh b/tools/package_setup_host.sh index c82d42bf..7dac9564 100755 --- a/tools/package_setup_host.sh +++ b/tools/package_setup_host.sh @@ -1,4 +1,9 @@ #!/usr/bin/env bash +# RETIRED (2026-09-30): setup-host zip packager. Public releases are bundled +# (tools/package_game_release.sh ships the compiled game built from the +# title's committed generated/ C; docs/ci/BUNDLED_RELEASES.md). Kept only for +# a locally built self-compile kit; no release workflow calls it. +# # Universal setup-host zip packager for PSXRecomp game repos. # # Stages the host exe, title sources, filtered psxrecomp/ + recomp-ui/, then @@ -553,7 +558,7 @@ if [[ -f "${STAGE}/CMakeLists.txt" ]]; then if (( ${#missing_refs[@]} )); then echo "error: CMakeLists.txt references paths that are not staged in the zip:" >&2 for r in "${missing_refs[@]}"; do echo " - ${r}" >&2; done - echo " add them via --project-file / --project-dir in scripts/package_setup_release.sh" >&2 + echo " add them via --project-file / --project-dir in the title's setup-host wrapper" >&2 exit 1 fi fi @@ -597,7 +602,7 @@ done if (( ${#missing_incs[@]} )); then echo "error: staged project sources include files that are not in the zip:" >&2 printf ' - %s\n' "${missing_incs[@]}" >&2 - echo " add them via --project-file / --project-dir in scripts/package_setup_release.sh" >&2 + echo " add them via --project-file / --project-dir in the title's setup-host wrapper" >&2 exit 1 fi diff --git a/tools/tests/test_bundled_release_layout.py b/tools/tests/test_bundled_release_layout.py new file mode 100644 index 00000000..30e0b419 --- /dev/null +++ b/tools/tests/test_bundled_release_layout.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python3 +"""Releases ship the compiled game built from COMMITTED generated/ C. + +WHY THIS EXISTS +=============== +Until 2026-09-30 the release flow was "setup-host": CI wiped generated/, +shipped emitters + sources + a wizard, and every player regenerated and +rebuilt the title on their own machine. That shipped uncompiled work. The +pivot to bundled releases touches four places that can drift back +independently -- the CI template, the scaffold's .gitignore, the scaffold's +packager wrapper, and Project Studio's audit/apply -- so this test pins each +one to the bundled contract rather than trusting the docs. + +Every check here is against the ARTIFACT (the template text, the wrapper it +emits, the script's exit status on a fixture tree), never the intent. + +Run: python3 tools/tests/test_bundled_release_layout.py +""" + +import os +import re +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +FW = Path(__file__).resolve().parents[2] +TOOLKIT = FW / "tools" / "new_project_layout" +sys.path.insert(0, str(TOOLKIT)) + +TEMPLATE = FW / "docs" / "ci" / "templates" / "game-release.yml" +CHECK_GENERATED = FW / "tools" / "ci" / "check_generated.sh" +BOOT = "SLUS_012.34" + + +def _git(root, *args): + return subprocess.run(["git", "-C", str(root), *args], capture_output=True, + text=True, check=False) + + +def _fixture_title(root: Path, *, ignore_generated=False, track=True, + retail=False): + """A minimal game repo with committed generated C.""" + root.mkdir(parents=True, exist_ok=True) + _git(root, "init", "-q") + _git(root, "config", "user.email", "t@example.invalid") + _git(root, "config", "user.name", "t") + (root / "game.toml").write_text( + f'[game]\nname = "Fixture"\nexe = "disc/{BOOT}"\n' + '[runtime]\noverlay_cache = true\n', encoding="utf-8") + (root / "CMakeLists.txt").write_text( + 'psxrecomp_add_game_runtime(psx-runtime\n' + f' GEN_MARKER "generated/{BOOT}_dispatch.c"\n' + f' GEN_FULL_GLOB "generated/{BOOT}_full_*.c"\n)\n', encoding="utf-8") + gen = root / "generated" + gen.mkdir(exist_ok=True) + (gen / f"{BOOT}_dispatch.c").write_text("/* dispatch */\n", encoding="utf-8") + (gen / f"{BOOT}_full_00.c").write_text("/* full */\n", encoding="utf-8") + if retail: + (gen / "SCPH1001_dispatch.c").write_text("/* retail */\n", encoding="utf-8") + (gen / "SCPH1001_full.c").write_text("/* retail */\n", encoding="utf-8") + (root / ".gitignore").write_text( + ("/generated/\n" if ignore_generated else "") + "/disc/\n/bios/\n/dist/\n/analysis/\n", + encoding="utf-8") + _git(root, "add", "game.toml", "CMakeLists.txt", ".gitignore") + if track and not ignore_generated: + _git(root, "add", "generated") + _git(root, "commit", "-q", "-m", "fixture") + return root + + +class TemplateContract(unittest.TestCase): + def setUp(self): + self.text = TEMPLATE.read_text(encoding="utf-8") + + def test_template_exists_under_its_new_name(self): + self.assertTrue(TEMPLATE.is_file(), TEMPLATE) + self.assertFalse((TEMPLATE.parent / "setup-release.yml").exists(), + "the setup-host template must not linger beside the bundled one") + + def test_never_clears_generated(self): + self.assertNotIn("clear_generated", self.text) + self.assertFalse((FW / "tools" / "ci" / "clear_generated.sh").exists(), + "clear_generated.sh wipes the release's inputs; it must be gone") + + def test_gates_on_committed_game_c_before_building(self): + self.assertIn("tools/ci/check_generated.sh", self.text) + self.assertLess(self.text.index("check_generated.sh"), + self.text.index("Configure & build the game")) + + def test_emits_openbios_in_ci(self): + self.assertIn("tools/ci/generate_openbios.sh", self.text) + + def test_configures_a_full_build(self): + self.assertIn("-DPSXRECOMP_REQUIRE_GAME_C=ON", self.text) + self.assertNotIn("-DPSXRECOMP_FORCE_SETUP_HOST=ON", self.text) + self.assertNotIn("-DPSXRECOMP_ALLOW_NO_BIOS=ON", self.text) + self.assertIn("linking generated game C (full runtime)", self.text) + self.assertIn("BIOS backends linked: .*OpenBIOS", self.text) + + def test_packages_through_the_bundled_wrapper(self): + self.assertIn("scripts/package_release.sh build-ci", self.text) + self.assertNotIn("package_setup_release.sh", self.text) + self.assertNotIn("package_setup_host.sh", self.text) + + def test_zip_verification_rejects_kit_content(self): + for pat in ("'^psxrecomp/'", "'^psxrecomp_cli.py'", "'^generated/'", "'^psxrecomp-game'"): + self.assertIn(pat, self.text, f"zip gate must reject {pat}") + self.assertIn("bios/openbios.bin", self.text) + + def test_parses_as_yaml_when_pyyaml_is_available(self): + try: + import yaml # noqa: F401 + except ImportError: + self.skipTest("pyyaml not installed") + import yaml + doc = yaml.safe_load(self.text) + self.assertEqual(set(doc["jobs"]), {"prepare", "build", "release"}) + names = [s.get("name", "") for s in doc["jobs"]["build"]["steps"]] + self.assertIn("Verify committed game C", names) + self.assertIn("Generate OpenBIOS backend C", names) + self.assertIn("Package game zip", names) + + +class ScaffoldContract(unittest.TestCase): + def test_gitignore_templates_track_generated(self): + for p in (TOOLKIT / "templates" / "gitignore.in", + FW / "docs" / "ci" / "templates" / "game.gitignore"): + lines = p.read_text(encoding="utf-8").splitlines() + ignored = [ln for ln in lines if re.match(r"^\s*/?generated/?\s*$", ln)] + self.assertEqual(ignored, [], f"{p} ignores generated/") + + def test_packager_wrapper_template_is_bundled(self): + tpl = TOOLKIT / "templates" / "package_release.sh.in" + self.assertTrue(tpl.is_file(), tpl) + self.assertFalse((TOOLKIT / "templates" / "package_setup_release.sh.in").exists()) + text = tpl.read_text(encoding="utf-8") + self.assertIn("package_game_release.sh", text) + self.assertNotIn("package_setup_host.sh", text) + self.assertNotIn("--project-dir", text, "a bundled release ships no sources") + + def test_scaffold_scripts_use_the_new_template_and_wrapper(self): + for script in (TOOLKIT / "setup_project.sh", TOOLKIT / "setup_project.ps1"): + text = script.read_text(encoding="utf-8") + self.assertIn("game-release.yml", text, script) + self.assertIn("package_release.sh.in", text, script) + self.assertNotIn("setup-release.yml", text, script) + self.assertNotIn("package_setup_release", text, script) + self.assertIn("git add generated", text, script) + + def test_fill_tokens_fills_the_template(self): + with tempfile.TemporaryDirectory() as td: + out = Path(td) / "release.yml" + r = subprocess.run([sys.executable, str(TOOLKIT / "fill_tokens.py"), + str(TEMPLATE), str(out), "--ci-placeholders", + "--set", "ZIP_PREFIX=fx", "--set", "GAME_TITLE=Fixture: Recompiled"], + capture_output=True, text=True, check=False) + self.assertEqual(r.returncode, 0, r.stderr) + text = out.read_text(encoding="utf-8") + self.assertNotIn("YOUR_ZIP_PREFIX", text) + self.assertNotIn("YOUR_GAME_TITLE", text) + self.assertIn("dist/fx-*-${{ matrix.artifact }}.zip", text) + self.assertIn('name: "Fixture: Recompiled ${{ needs.prepare.outputs.tag }}"', text) + + +class CheckGeneratedScript(unittest.TestCase): + def run_check(self, root): + return subprocess.run(["bash", str(CHECK_GENERATED), "--root", str(root)], + capture_output=True, text=True, check=False) + + def test_passes_on_committed_game_c(self): + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t") + r = self.run_check(root) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + self.assertIn("generated game C ok", r.stdout) + + def test_fails_when_marker_is_missing(self): + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t") + (root / "generated" / f"{BOOT}_dispatch.c").unlink() + r = self.run_check(root) + self.assertNotEqual(r.returncode, 0) + self.assertIn("is missing", r.stderr) + + def test_fails_when_generated_is_ignored(self): + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t", ignore_generated=True) + r = self.run_check(root) + self.assertNotEqual(r.returncode, 0) + self.assertIn("NOT tracked", r.stderr) + + def test_fails_when_generated_is_untracked(self): + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t", track=False) + r = self.run_check(root) + self.assertNotEqual(r.returncode, 0) + self.assertIn("NOT tracked", r.stderr) + + def test_fails_on_retail_bios_c(self): + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t", retail=True) + r = self.run_check(root) + self.assertNotEqual(r.returncode, 0) + self.assertIn("retail-BIOS-derived", r.stderr) + + +class ProjectStudioContract(unittest.TestCase): + def test_audit_flags_ignored_generated_and_missing_c(self): + from project_studio.detect import audit_project + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t", ignore_generated=True) + report = audit_project(root) + by_id = {c.id: c for c in report.checks} + self.assertEqual(by_id["gitignore"].status.value, "fail") + self.assertEqual(by_id["generated_committed"].status.value, "fail") + + def test_audit_passes_committed_generated(self): + from project_studio.detect import audit_project + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t") + report = audit_project(root) + by_id = {c.id: c for c in report.checks} + self.assertEqual(by_id["generated_committed"].status.value, "pass") + self.assertNotIn("retail_bios_c", by_id) + + def test_merge_gitignore_unignores_generated(self): + from project_studio.models import MigrateOptions + from project_studio.ops import op_merge_gitignore + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t", ignore_generated=True) + res = op_merge_gitignore(root, MigrateOptions()) + self.assertTrue(res.ok, res.message) + lines = (root / ".gitignore").read_text(encoding="utf-8").splitlines() + self.assertFalse(any(re.match(r"^\s*/?generated/?\s*$", ln) for ln in lines)) + self.assertIn("/disc/", lines) + + def test_emit_packager_and_ci_workflow(self): + from project_studio.models import MigrateOptions + from project_studio.ops import op_emit_ci_workflow, op_emit_packager + with tempfile.TemporaryDirectory() as td: + root = _fixture_title(Path(td) / "t") + opts = MigrateOptions(zip_prefix="fx", window_title="Fixture Recompiled", + project_name="FixtureRecomp") + pk = op_emit_packager(root, opts) + self.assertTrue(pk.ok, pk.message) + wrapper = root / "scripts" / "package_release.sh" + self.assertTrue(wrapper.is_file()) + self.assertIn("package_game_release.sh", wrapper.read_text(encoding="utf-8")) + ci = op_emit_ci_workflow(root, opts) + self.assertTrue(ci.ok, ci.message) + wf = (root / ".github" / "workflows" / "release.yml").read_text(encoding="utf-8") + self.assertIn("check_generated.sh", wf) + self.assertIn("dist/fx-*", wf) + # generate_ci --check agrees the installed workflow is current. + r = subprocess.run([sys.executable, str(FW / "tools" / "generate_ci.py"), + str(root), "--check"], capture_output=True, text=True, + check=False) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + + +class PackagerScriptContract(unittest.TestCase): + def test_bundled_packager_routes_through_release_stage(self): + text = (FW / "tools" / "package_game_release.sh").read_text(encoding="utf-8") + self.assertIn("release_overlay_stage.sh", text) + self.assertIn("psx_add_mod_catalog", text) + self.assertIn("psx_add_overlay_toolchain", text) + self.assertIn("PSXRECOMP_FORCE_SETUP_HOST:BOOL=ON", text, + "must refuse to package a setup host as a game") + # No source allowlist of any kind: nothing project-owned is shipped. + self.assertNotIn("--project-file", text) + self.assertNotIn("--project-dir", text) + + def test_scripts_are_executable_and_parse(self): + for rel in ("tools/package_game_release.sh", "tools/ci/check_generated.sh", + "tools/ci/generate_openbios.sh"): + p = FW / rel + self.assertTrue(os.access(p, os.X_OK), f"{rel} not executable") + r = subprocess.run(["bash", "-n", str(p)], capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + + +if __name__ == "__main__": + if shutil.which("git") is None or shutil.which("bash") is None: + print("git and bash are required", file=sys.stderr) + sys.exit(2) + unittest.main(verbosity=1) diff --git a/tools/tests/test_overlay_static_sources_cmake.py b/tools/tests/test_overlay_static_sources_cmake.py index 0526030f..f83a9e27 100644 --- a/tools/tests/test_overlay_static_sources_cmake.py +++ b/tools/tests/test_overlay_static_sources_cmake.py @@ -108,11 +108,13 @@ message(STATUS "PROBE_SOURCES=${{srcs}}") self.profile(static_output='generated/overlays_static.c') code, out, _ = self.configure(None) self.assertNotEqual(code, 0) - self.assertIn('passes no GAME_OVERLAY_STATIC_C', out) + # CMake re-wraps message() text at ~72 columns, so this phrase can + # straddle a line break; compare with whitespace collapsed. + self.assertIn('passes no GAME_OVERLAY_STATIC_C', ' '.join(out.split())) other = self.project / 'elsewhere' / 'overlays_static.c' code, out, _ = self.configure(other) self.assertNotEqual(code, 0) - self.assertIn('Make them name the same file', out) + self.assertIn('Make them name the same file', ' '.join(out.split())) def test_undeclared_profiles_and_capture_built_shards_are_unaffected(self): # A release/DLL-cache title: profile without static_output, no static C.