diff --git a/docs/MOD_PACKAGES.md b/docs/MOD_PACKAGES.md index cec1a1b6..2235e1c7 100644 --- a/docs/MOD_PACKAGES.md +++ b/docs/MOD_PACKAGES.md @@ -12,10 +12,49 @@ The player selects a verified stock BIN/CUE. Resolution produces guarded native operations and sparse disc overlays without rewriting or replacing that stock image. +## Where packages live + +Two catalog roots sit beside the executable, split by who owns the files: + +```text +/mods/ + bundled/ build output — the framework's mods/builtin/packages plus the + title's mods/preloaded/packages. Every build WIPES and re-stages + this tree, so nothing a player owns may live here. + installed/ launcher-owned — .psxmod archives installed through the Mods + manager. No build ever touches this tree. + state.toml user selection state (enabled features, option values). +``` + +Both roots use the same `//manifest.toml` layout and are +scanned into one catalog, bundled first. An installed package with the same id +as a bundled one deliberately shadows it and records that it did so, so an +override is visible rather than decided by directory-iteration order. + +A bundled package is not removable from the Mods page: deleting build output +would succeed and then be undone by the next build. Remove it from the title's +`mods/preloaded/packages` instead. + +Only `bundled/` ships in a release. `installed/` and `state.toml` are the local +machine's and are excluded by every packager. + +**Migration.** A pre-split install has one `mods/packages/` tree holding both. +The first scan moves each package the current build did not also stage into +`installed/`, drops the rest as redundant build output, and removes the old +tree — but only once every version directory has been dealt with. Anything that +could not be moved is left exactly where it is and reported. + +A manifest that fails to parse is never skipped silently: it is reported by +`scan_errors()` and logged, naming the path and the reason. + ## Feature manifest +Write new manifests at the current format version, which is **6**. Older +versions stay readable so installed packages survive an update, and each +section below notes the version a field first required. + ```toml -format_version = 1 +format_version = 6 id = "example.localization" version = "1.2.0" name = "Example Localization Pack" @@ -261,31 +300,63 @@ Sparse fields and integer predicates are still pre-boot plan construction. They do not provide a general expression evaluator, masks, arithmetic beyond the checked field addend, package code execution, or per-frame dispatch. -## Developer channel +## Channels -A package may declare itself unfinished: +A feature declares how finished it is. Format 6 puts `channel` on the +**feature**, not the package: ```toml -format_version = 5 +format_version = 6 id = "example.enhancements" -channel = "developer" + +[[feature]] +id = "widescreen" +channel = "experimental" # ships, badged, default off + +[[feature]] +id = "hook-trace" +channel = "developer" # absent from any release build ``` -`channel = "developer"` marks work in progress. Such a package is staged next -to the executable by an ordinary build and ships in a locally exported -package, so it can be tested exactly as a player would see it — but release -packaging drops it, so unfinished work is never published. +| Channel | Ships | In the launcher | +|---|---|---| +| `stable` (default) | yes | no tag — the absence is the stable case | +| `experimental` | yes | amber `EXP` tag, and a line saying it is unvalidated | +| `developer` | **no** | secondary-accent `DEV` tag; only ever visible on a local build | -Packaging defaults the exclusion from `$CI`: on under any CI provider, off +An absent `channel` means `stable`. A package may still declare one, which its +features inherit unless they state their own — which is how a format-5 manifest +carrying a package-level `channel` keeps working unchanged. + +**Why the feature and not the package.** A package is the installation and +**trust** boundary; how finished one of its features is has nothing to do with +that. When the marker sat on the package, a catalog holding one player-ready +feature and one developer instrument had to declare itself entirely +developer — so neither shipped, and the documented workaround was to split the +catalog into two packages. Channels per feature remove that trade. + +**"Developer does not ship" means absent, not hidden.** Two mechanisms, one per +catalog root: + +- **`bundled/` is filtered when it is staged.** `tools/mod_channel_filter.py` + emits a manifest without the developer features and without the `[[option]]`, + `[[patch]]`, `[[overlay]]`, `[[plugin]]`, `[[resource]]` and `[[constraint]]` + entries that only served them; a package whose every feature is developer has + its directory removed. This is generation, not rewriting: the staged catalog + is build output and the author's manifest in the repo is never touched. +- **`installed/` is refused at load.** A third-party archive is never modified, + so the runtime declines to surface developer features from one instead. + +The runtime gate is the build definition `PSX_MOD_DEVELOPER_CHANNEL`, which +`runtime.cmake` sets for a local build and clears under `$CI`. A contributor +reaches developer features by cloning the repo and building; a release build +carries neither the features nor their operations, so a stale `state.toml` +naming one cannot reach them either. + +Packaging defaults the exclusion from `$CI` — on under any CI provider, off locally. `--exclude-dev-mods` / `--include-dev-mods` (or `EXCLUDE_DEV_MODS=0|1`) override it, and `project_studio build export --exclude-dev-mods` reproduces -what a release would contain. Any absent `channel` means stable. - -Pruning removes the whole package directory. A manifest is never rewritten -during packaging — a package's content stays exactly what its author shipped, -so the maturity marker is package-level. Split a player-ready feature into its -own stable package rather than promoting a catalog that still holds -in-progress work. +what a release would contain. ## Trusted static plugins diff --git a/docs/ci/templates/setup-release.yml b/docs/ci/templates/setup-release.yml index b074e2c2..177f1639 100644 --- a/docs/ci/templates/setup-release.yml +++ b/docs/ci/templates/setup-release.yml @@ -562,10 +562,10 @@ jobs: # artifact, not the intent. ZIP=$(ls dist/YOUR_ZIP_PREFIX-*-"${{ matrix.artifact }}".zip | head -1) echo "checking ${ZIP}" - MANIFESTS=$(unzip -l "${ZIP}" | grep -c 'mods/packages/.*/manifest\.toml' || true) + MANIFESTS=$(unzip -l "${ZIP}" | grep -c 'mods/bundled/.*/manifest\.toml' || true) echo "mod manifests in zip: ${MANIFESTS}" if [[ "${MANIFESTS}" -lt 1 ]]; then - echo "::error::release zip ships no mod catalog (mods/packages/*/manifest.toml)" >&2 + echo "::error::release zip ships no mod catalog (mods/bundled/*/manifest.toml)" >&2 unzip -l "${ZIP}" | head -40 >&2 exit 1 fi @@ -576,6 +576,13 @@ jobs: echo "::error::release zip contains mods/state.toml (per-machine state)" >&2 exit 1 fi + # Same rule, other half: mods/installed/ is the launcher-owned tree of + # .psxmod archives a developer installed as a player would. Only + # mods/bundled/ is build output, and only build output ships. + if unzip -l "${ZIP}" | grep -q 'mods/installed/'; then + echo "::error::release zip contains mods/installed (per-machine installs)" >&2 + exit 1 + fi # Developer-channel packages are unfinished work that ships with # local builds only. The packager drops them under CI automatically # (it keys off $CI); assert the outcome so a future change to that diff --git a/runtime/include/mod_packages.h b/runtime/include/mod_packages.h index 65f4510c..b4860c92 100644 --- a/runtime/include/mod_packages.h +++ b/runtime/include/mod_packages.h @@ -9,6 +9,36 @@ namespace PSXRecompV4 { +/* How finished a feature is, and therefore who is allowed to see it. + * + * Stable ships and is offered to everyone. Experimental ships, badged, and is + * default-off: it works, but has not been validated on this title. Developer + * does not ship at all -- contributors reach those features by cloning the + * repo and building locally. + * + * This lives on the FEATURE, not the package. A package is the trust and + * installation boundary; how finished one of its features is has nothing to do + * with that, and tying the two forced a whole catalog down to the maturity of + * its least finished feature. A package may still declare a channel, which its + * features inherit unless they state their own. */ +enum class ModChannel { + Stable = 0, + Experimental = 1, + Developer = 2, +}; + +/* Whether this build may see developer-channel features. A local build may; a + * CI/release build may not, which is what "developer does not ship" means in + * practice. runtime.cmake defines PSX_MOD_DEVELOPER_CHANNEL for local builds. */ +#if defined(PSX_MOD_DEVELOPER_CHANNEL) && PSX_MOD_DEVELOPER_CHANNEL +inline constexpr bool kDeveloperChannelDefault = true; +#else +inline constexpr bool kDeveloperChannelDefault = false; +#endif + +const char* mod_channel_name(ModChannel channel); +bool mod_channel_from_name(const std::string& name, ModChannel& out); + enum class ModOptionType { Boolean, Choice, @@ -45,6 +75,7 @@ struct ModFeature { std::string author; std::string description; std::string group = "General"; + ModChannel channel = ModChannel::Stable; bool default_enabled = false; bool hidden = false; bool legacy = false; @@ -190,6 +221,20 @@ struct ModAuthorLink { std::string url; }; +/* Where a package's files came from, and therefore who owns them on disk. + * + * Bundled packages are build output: the framework and the title stage them + * into /mods/bundled, which every build wipes and re-stages. Nothing the + * player owns may live there. Installed packages are written by the launcher + * into /mods/installed and are never touched by a build. + * + * Before this split there was one /mods/packages tree serving both roles, + * so a rebuild deleted every mod the player had installed. */ +enum class ModPackageOrigin { + Bundled, + Installed, +}; + struct ModPackage { uint32_t format_version = 0; std::string id; @@ -203,6 +248,11 @@ struct ModPackage { std::string source_url; std::string resolver = "declarative"; std::string save_compatibility = "shared"; + /* Default channel for features that do not declare their own. */ + ModChannel channel = ModChannel::Stable; + ModPackageOrigin origin = ModPackageOrigin::Installed; + /* Set when an installed package of the same id shadows a bundled one. */ + bool shadows_bundled = false; std::filesystem::path root; std::vector targets; std::vector dependencies; @@ -312,6 +362,18 @@ public: void set_root(std::filesystem::path mods_root); const std::filesystem::path& root() const { return root_; } + /* Developer-channel features are stripped at scan and install time unless + * this is set. Defaults to kDeveloperChannelDefault; tests set it + * explicitly so they exercise both a local and a release build. */ + void set_developer_channel_visible(bool visible) { + developer_channel_ = visible; + } + bool developer_channel_visible() const { return developer_channel_; } + + /* Build-owned catalog: wiped and re-staged by every build. */ + std::filesystem::path bundled_root() const { return root_ / "bundled"; } + /* Launcher-owned catalog: a build never touches this tree. */ + std::filesystem::path installed_root() const { return root_ / "installed"; } bool scan(std::string* error = nullptr); bool load_state(std::string* error = nullptr); @@ -367,8 +429,23 @@ public: static bool read_manifest(const std::filesystem::path& path, ModPackage& out, std::string* error = nullptr); + /* Manifests that failed to parse during the last scan(), as + * ": ". A package that cannot be read must never vanish + * silently -- the launcher surfaces these so a mod author sees why their + * package is absent from the list. */ + const std::vector& scan_errors() const { return scan_errors_; } + private: + /* One-time move of a pre-split /mods/packages tree into the two + * owned roots. Anything the build also staged into bundled/ is dropped; + * everything else is the player's and moves to installed/. */ + void migrate_legacy_root(); + bool scan_root(const std::filesystem::path& packages_root, + ModPackageOrigin origin, std::string* error); + std::filesystem::path root_; + bool developer_channel_ = kDeveloperChannelDefault; + std::vector scan_errors_; std::map> packages_; std::map selections_; }; diff --git a/runtime/runtime.cmake b/runtime/runtime.cmake index 65cc6cee..f6112229 100644 --- a/runtime/runtime.cmake +++ b/runtime/runtime.cmake @@ -1377,8 +1377,14 @@ function(psxrecomp_add_runtime_target target) # Clear first: copy_directory MERGES, so a mod deleted from source # would otherwise survive in the build output forever and keep # appearing on the Mods page (and inflate release catalog assertions). - # Scoped to mods/packages, not mods/: state.toml is launcher-owned user - # state and must survive a rebuild. + # + # Scoped to mods/bundled, which is build output and nothing else. + # mods/installed/ is the launcher's (player-installed .psxmod archives) + # and mods/state.toml is user selection state; a build must never touch + # either. Before that split this wipe was scoped to mods/packages, + # which ALSO held everything the player had installed -- so a rebuild, + # including the one a self-compiling setup release runs on the player's + # own machine, deleted their mods without a word. set(_psx_builtin_mod_copy_commands "") if(DEFINED PSX_BUILTIN_MOD_ALLOWLIST AND NOT "${PSX_BUILTIN_MOD_ALLOWLIST}" STREQUAL "") @@ -1390,17 +1396,17 @@ function(psxrecomp_add_runtime_target target) list(APPEND _psx_builtin_mod_copy_commands COMMAND ${CMAKE_COMMAND} -E copy_directory "${PSXRECOMP_ROOT}/mods/builtin/packages/${_psx_builtin_mod}" - "$/mods/packages/${_psx_builtin_mod}") + "$/mods/bundled/${_psx_builtin_mod}") endforeach() else() list(APPEND _psx_builtin_mod_copy_commands COMMAND ${CMAKE_COMMAND} -E copy_directory - "${PSXRECOMP_ROOT}/mods/builtin" - "$/mods") + "${PSXRECOMP_ROOT}/mods/builtin/packages" + "$/mods/bundled") endif() add_custom_command(TARGET ${target} POST_BUILD COMMAND ${CMAKE_COMMAND} -E rm -rf - "$/mods/packages" + "$/mods/bundled" ${_psx_builtin_mod_copy_commands} COMMENT "Staging framework-owned mod catalog" VERBATIM) @@ -1419,6 +1425,22 @@ function(psxrecomp_add_runtime_target target) if(PSX_SHELLWIN_INTERP) target_compile_definitions(${target} PRIVATE PSX_SHELLWIN_INTERP_DEFAULT=1) endif() + + # Developer-channel mod features do not ship. A contributor reaches them by + # cloning the repo and building locally; a release build must not carry + # them at all -- not hidden behind a toggle, absent. Keying the default off + # $CI matches what the packagers already do for EXCLUDE_DEV_MODS, so one + # rule covers the catalog on disk and the catalog in the binary. + if(NOT DEFINED PSX_MOD_DEVELOPER_CHANNEL) + if(DEFINED ENV{CI} AND NOT "$ENV{CI}" STREQUAL "") + set(PSX_MOD_DEVELOPER_CHANNEL OFF) + else() + set(PSX_MOD_DEVELOPER_CHANNEL ON) + endif() + endif() + if(PSX_MOD_DEVELOPER_CHANNEL) + target_compile_definitions(${target} PRIVATE PSX_MOD_DEVELOPER_CHANNEL=1) + endif() if(has_game_dispatch) target_compile_definitions(${target} PRIVATE PSX_HAS_GAME_DISPATCH=1) endif() diff --git a/runtime/src/mod_packages.cpp b/runtime/src/mod_packages.cpp index e9292e2e..150a5a3d 100644 --- a/runtime/src/mod_packages.cpp +++ b/runtime/src/mod_packages.cpp @@ -22,7 +22,7 @@ namespace PSXRecompV4 { namespace { constexpr uint32_t kMinFormatVersion = 1; -constexpr uint32_t kMaxFormatVersion = 5; +constexpr uint32_t kMaxFormatVersion = 6; constexpr uint64_t kMaxArchiveBytes = 256ull * 1024ull * 1024ull; constexpr uint32_t kMaxArchiveFiles = 4096; @@ -1471,6 +1471,65 @@ void mod_clear_plugins_for_tests() { registered_plugins().clear(); } +const char* mod_channel_name(ModChannel channel) { + switch (channel) { + case ModChannel::Experimental: return "experimental"; + case ModChannel::Developer: return "developer"; + case ModChannel::Stable: break; + } + return "stable"; +} + +bool mod_channel_from_name(const std::string& name, ModChannel& out) { + if (name == "stable") { out = ModChannel::Stable; return true; } + if (name == "experimental") { out = ModChannel::Experimental; return true; } + if (name == "developer") { out = ModChannel::Developer; return true; } + return false; +} + +namespace { + +/* Remove every developer-channel feature and everything that only existed to + * serve one. "Developer does not ship" means absent, not hidden: a release + * build must not carry the operations either, or a stale state.toml naming a + * dropped feature could still reach them. + * + * Returns false when nothing is left, so the caller can drop the package + * rather than list one with no features. */ +bool strip_developer_features(ModPackage& package) { + std::set dropped; + for (const ModFeature& feature : package.features) + if (feature.channel == ModChannel::Developer) + dropped.insert(feature.id); + if (dropped.empty()) return !package.features.empty(); + + const auto orphaned = [&dropped](const std::string& feature_id) { + return dropped.count(feature_id) != 0; + }; + const auto prune = [&orphaned](auto& entries) { + entries.erase(std::remove_if(entries.begin(), entries.end(), + [&orphaned](const auto& entry) { + return orphaned(entry.feature_id); + }), + entries.end()); + }; + package.features.erase( + std::remove_if(package.features.begin(), package.features.end(), + [&orphaned](const ModFeature& feature) { + return orphaned(feature.id); + }), + package.features.end()); + prune(package.options); + prune(package.constraints); + prune(package.patches); + prune(package.overlays); + prune(package.plugins); + prune(package.resources); + return !package.features.empty(); +} + +} // namespace + ModPackageManager::ModPackageManager(fs::path mods_root) : root_(std::move(mods_root)) {} void ModPackageManager::set_root(fs::path mods_root) { @@ -1517,6 +1576,16 @@ bool ModPackageManager::read_manifest(const fs::path& path, ModPackage& out, cfg.contains("resolver") ? toml::find(cfg, "resolver") : "declarative"; out.save_compatibility = cfg.contains("save_compatibility") ? toml::find(cfg, "save_compatibility") : "shared"; + /* Accepted at every format version: v5 manifests already carried a + * package-level channel, enforced then only by a packaging grep over + * the manifest text. Now it parses, and it seeds the per-feature + * default rather than quarantining the whole package. */ + if (cfg.contains("channel")) { + const std::string name = toml::find(cfg, "channel"); + if (!mod_channel_from_name(name, out.channel)) + throw std::runtime_error( + "channel must be stable, experimental or developer"); + } out.root = path.parent_path(); if (out.format_version < kMinFormatVersion || out.format_version > kMaxFormatVersion) @@ -1573,6 +1642,16 @@ bool ModPackageManager::read_manifest(const fs::path& path, ModPackage& out, feature.default_enabled = toml::find_or(v, "default_enabled", false); feature.hidden = toml::find_or(v, "hidden", false); + feature.channel = out.channel; + if (v.contains("channel")) { + if (out.format_version < 6) + throw std::runtime_error( + "per-feature channel requires format_version 6"); + const std::string name = toml::find(v, "channel"); + if (!mod_channel_from_name(name, feature.channel)) + throw std::runtime_error( + "channel must be stable, experimental or developer"); + } if (!valid_id(feature.id) || !feature_ids.insert(feature.id).second) throw std::runtime_error("invalid or duplicate feature id"); @@ -1588,6 +1667,7 @@ bool ModPackageManager::read_manifest(const fs::path& path, ModPackage& out, feature.name = out.name; feature.author = out.author; feature.description = out.description; + feature.channel = out.channel; feature.legacy = true; out.features.push_back(std::move(feature)); } @@ -2305,10 +2385,68 @@ bool ModPackageManager::read_manifest(const fs::path& path, ModPackage& out, } } -bool ModPackageManager::scan(std::string* error) { - packages_.clear(); +void ModPackageManager::migrate_legacy_root() { + std::error_code ec; + const fs::path legacy = root_ / "packages"; + if (!fs::exists(legacy, ec)) return; + const fs::path installed = installed_root(); + const fs::path bundled = bundled_root(); + size_t stranded = 0; + for (const fs::directory_entry& id_dir : fs::directory_iterator(legacy, ec)) { + if (ec) break; + if (!id_dir.is_directory()) continue; + const std::string id = id_dir.path().filename().string(); + for (const fs::directory_entry& version_dir : + fs::directory_iterator(id_dir.path(), ec)) { + if (ec) break; + if (!version_dir.is_directory()) continue; + const std::string version = version_dir.path().filename().string(); + std::error_code move_ec; + /* Already staged by the build: this copy is redundant output, not + * anything the player owns. */ + if (fs::exists(bundled / id / version, move_ec)) { + fs::remove_all(version_dir.path(), move_ec); + continue; + } + const fs::path destination = installed / id / version; + if (fs::exists(destination, move_ec)) { + fs::remove_all(version_dir.path(), move_ec); + continue; + } + fs::create_directories(destination.parent_path(), move_ec); + if (move_ec) { ++stranded; continue; } + fs::rename(version_dir.path(), destination, move_ec); + if (move_ec) { + /* Cross-device or a locked file: copy rather than lose it. */ + move_ec.clear(); + fs::copy(version_dir.path(), destination, + fs::copy_options::recursive, move_ec); + if (move_ec) { + ++stranded; + continue; + } + fs::remove_all(version_dir.path(), move_ec); + } + } + } + /* Drop the legacy tree only when every version directory was dealt with. + * A blanket remove_all here would be the very data loss this migration + * exists to end, so anything that could not be moved stays exactly where + * it is and is reported instead. */ + if (stranded != 0 || ec) { + scan_errors_.push_back( + legacy.string() + ": could not migrate " + + std::to_string(stranded) + " package version(s) to " + + installed.string() + "; the old catalog was left in place"); + return; + } + ec.clear(); + fs::remove_all(legacy, ec); +} + +bool ModPackageManager::scan_root(const fs::path& packages_root, + ModPackageOrigin origin, std::string* error) { std::error_code ec; - const fs::path packages_root = root_ / "packages"; if (!fs::exists(packages_root, ec)) return true; for (const fs::directory_entry& id_dir : fs::directory_iterator(packages_root, ec)) { if (ec) break; @@ -2321,13 +2459,34 @@ bool ModPackageManager::scan(std::string* error) { ModPackage package; std::string parse_error; if (!read_manifest(manifest, package, &parse_error)) { + /* A package that cannot be parsed must say so. Silently + * skipping it leaves a mod author with a mod that simply does + * not exist and nothing anywhere explaining why. */ + scan_errors_.push_back(manifest.string() + ": " + parse_error); continue; } if (package.id != id_dir.path().filename().string() || package.version != version_dir.path().filename().string()) { - set_error(error, "package path does not match manifest id/version: " + - manifest.string()); - return false; + scan_errors_.push_back( + manifest.string() + + ": package path does not match manifest id/version"); + continue; + } + if (!developer_channel_ && !strip_developer_features(package)) { + /* Every feature was developer-channel, so on this build the + * package has nothing to offer. Not an error: it is the + * intended outcome of shipping without them. */ + continue; + } + package.origin = origin; + if (origin == ModPackageOrigin::Installed) { + const auto existing = packages_.find(package.id); + package.shadows_bundled = + existing != packages_.end() && + std::any_of(existing->second.begin(), existing->second.end(), + [](const std::pair& e) { + return e.second.origin == ModPackageOrigin::Bundled; + }); } packages_[package.id][package.version] = std::move(package); } @@ -2339,6 +2498,18 @@ bool ModPackageManager::scan(std::string* error) { return true; } +bool ModPackageManager::scan(std::string* error) { + packages_.clear(); + scan_errors_.clear(); + migrate_legacy_root(); + /* Bundled first, then installed: an installed package of the same id and + * version deliberately shadows the build-staged one, and records that it + * did so, rather than the two racing on directory-iteration order. */ + if (!scan_root(bundled_root(), ModPackageOrigin::Bundled, error)) return false; + if (!scan_root(installed_root(), ModPackageOrigin::Installed, error)) return false; + return true; +} + bool ModPackageManager::load_state(std::string* error) { selections_.clear(); const fs::path path = root_ / "state.toml"; @@ -2536,7 +2707,16 @@ bool ModPackageManager::install_archive(const fs::path& archive, fs::remove_all(staging, ec); return false; } - const fs::path destination = root_ / "packages" / package.id / package.version; + if (!developer_channel_ && !strip_developer_features(package)) { + fs::remove_all(staging, ec); + set_error(error, + "this package contains only developer-channel features, " + "which are not available in this build"); + return false; + } + /* Player installs land in the launcher-owned root. They must never share + * a tree with the build-staged catalog, which every build wipes. */ + const fs::path destination = installed_root() / package.id / package.version; if (fs::exists(destination)) { fs::remove_all(staging, ec); set_error(error, "package version is already installed"); @@ -2555,6 +2735,7 @@ bool ModPackageManager::install_archive(const fs::path& archive, return false; } package.root = destination; + package.origin = ModPackageOrigin::Installed; packages_[package.id][package.version] = package; if (installed_id) *installed_id = package.id; if (installed_version) *installed_version = package.version; @@ -2595,6 +2776,12 @@ bool ModPackageManager::remove_version(const std::string& id, const std::string& set_error(error, "package version is not installed"); return false; } + if (pit->second.at(version).origin == ModPackageOrigin::Bundled) { + /* Deleting build output would succeed and then be undone by the next + * build, which is not a model any player can reason about. */ + set_error(error, "cannot remove a bundled package; it ships with the game"); + return false; + } const fs::path path = pit->second.at(version).root; std::error_code ec; fs::remove_all(path, ec); diff --git a/runtime/src/mod_runtime.cpp b/runtime/src/mod_runtime.cpp index e481779f..3f8b5765 100644 --- a/runtime/src/mod_runtime.cpp +++ b/runtime/src/mod_runtime.cpp @@ -626,7 +626,10 @@ int provider_package_get(void*, int index, RecompLauncherCModPackage* out) { copy_text(out->source_url, sizeof(out->source_url), package->source_url); out->enabled = package_has_enabled_feature(*package); out->option_count = (int)package->options.size(); - out->removable = !out->enabled; + /* A bundled package is build output. Offering to remove it would succeed + * and then be silently undone by the next build. */ + out->removable = !out->enabled && + package->origin == ModPackageOrigin::Installed; return 1; } @@ -742,6 +745,19 @@ int provider_feature_get(void*, int index, RecompLauncherCModFeature* out) { copy_text(out->source_url, sizeof(out->source_url), package->source_url); copy_text(out->group, sizeof(out->group), feature->group); out->hidden = feature->hidden ? 1 : 0; + switch (feature->channel) { + case ModChannel::Experimental: + out->channel = RECOMP_MOD_CHANNEL_EXPERIMENTAL; + break; + case ModChannel::Developer: + /* Only reachable on a local developer build: a shipped catalog + * carries no developer-channel feature to report. */ + out->channel = RECOMP_MOD_CHANNEL_DEVELOPER; + break; + case ModChannel::Stable: + out->channel = RECOMP_MOD_CHANNEL_STABLE; + break; + } out->enabled = state().manager.feature_enabled(package->id, feature->id) ? 1 : 0; out->option_count = @@ -957,8 +973,9 @@ int provider_version_get(void*, const char* package_id, int index, copy_text(out->version, sizeof(out->version), version->first); const ModPackage* selected = selected_package(package_id); out->selected = selected && selected->version == version->first; - out->removable = !out->selected || - !selected || !package_has_enabled_feature(*selected); + out->removable = (!out->selected || !selected || + !package_has_enabled_feature(*selected)) && + version->second.origin == ModPackageOrigin::Installed; return 1; } @@ -1107,6 +1124,11 @@ bool mod_runtime_initialize(const std::filesystem::path& root, if (error) *error = s.error; return false; } + /* A manifest that fails to parse used to be skipped in silence, so a mod + * author's typo produced a mod that simply did not exist. Name every one. */ + for (const std::string& scan_error : s.manager.scan_errors()) + std::fprintf(stderr, "psxrecomp: mod manifest ignored: %s\n", + scan_error.c_str()); if (!sha256_file(exe_path, s.exe_sha256, &s.error)) { /* Release installs commonly do not carry a loose PS-X EXE; game-id and * expected-byte guards remain available in that case. */ diff --git a/runtime/tests/test_mod_packages.cpp b/runtime/tests/test_mod_packages.cpp index a72ace43..b86b15a3 100644 --- a/runtime/tests/test_mod_packages.cpp +++ b/runtime/tests/test_mod_packages.cpp @@ -1356,6 +1356,200 @@ int main() { check(!ModPackageManager::read_manifest(root / "bad.toml", invalid, &error), "unsafe package id must be rejected"); + /* Catalog roots. mods/bundled is build output that every build wipes and + * re-stages; mods/installed belongs to the launcher and a build never + * touches it. Before the split both lived in mods/packages, so a rebuild + * deleted every mod the player had installed. */ + { + const fs::path split_root = root / "split"; + /* Pre-split layout: a build-staged package and a player install, + * indistinguishable in the same tree. */ + write_text(split_root / "packages/psx.builtin/1.0.0/manifest.toml", + manifest("psx.builtin", "1.0.0")); + write_text(split_root / "packages/player.mod/1.0.0/manifest.toml", + manifest("player.mod", "1.0.0")); + /* The current build has already staged its own copy of the builtin. */ + write_text(split_root / "bundled/psx.builtin/1.0.0/manifest.toml", + manifest("psx.builtin", "1.0.0")); + + ModPackageManager split(split_root); + check(split.scan(&error), error.c_str()); + check(!fs::exists(split_root / "packages"), + "the legacy mods/packages tree must be migrated away"); + check(fs::exists(split_root / "installed/player.mod/1.0.0/manifest.toml"), + "a player-installed package must migrate into mods/installed"); + check(!fs::exists(split_root / "installed/psx.builtin"), + "a package the build also staged must not migrate to installed"); + check(split.packages().at("psx.builtin").at("1.0.0").origin == + ModPackageOrigin::Bundled, + "a package under mods/bundled must be Bundled"); + check(split.packages().at("player.mod").at("1.0.0").origin == + ModPackageOrigin::Installed, + "a migrated player package must be Installed"); + + check(!split.remove_version("psx.builtin", "1.0.0", &error), + "a bundled package must not be removable"); + + /* What a rebuild does: wipe bundled/ and re-stage it. The player's + * installed catalog must survive untouched. */ + fs::remove_all(split_root / "bundled", ec); + write_text(split_root / "bundled/psx.builtin/1.0.0/manifest.toml", + manifest("psx.builtin", "1.0.0")); + ModPackageManager rebuilt(split_root); + check(rebuilt.scan(&error), error.c_str()); + check(rebuilt.packages().count("player.mod") == 1, + "a rebuild must not delete a player-installed package"); + check(rebuilt.packages().count("psx.builtin") == 1, + "a rebuild must re-stage the bundled catalog"); + + /* An installed package of the same id shadows the bundled one, and + * says so rather than winning silently. */ + write_text(split_root / "installed/psx.builtin/1.0.0/manifest.toml", + manifest("psx.builtin", "1.0.0")); + ModPackageManager shadowed(split_root); + check(shadowed.scan(&error), error.c_str()); + const ModPackage& shadow = + shadowed.packages().at("psx.builtin").at("1.0.0"); + check(shadow.origin == ModPackageOrigin::Installed && + shadow.shadows_bundled, + "an installed package must shadow the bundled one and record it"); + + /* A manifest that cannot be parsed must be named, never skipped in + * silence -- an author's typo used to produce a mod that simply did + * not exist, with nothing anywhere explaining why. */ + write_text(split_root / "installed/broken.mod/1.0.0/manifest.toml", + "format_version = 5\nid = \"broken.mod\"\n"); + ModPackageManager broken(split_root); + check(broken.scan(&error), error.c_str()); + check(broken.packages().count("broken.mod") == 0, + "an unparseable package must not load"); + check(std::any_of(broken.scan_errors().begin(), + broken.scan_errors().end(), + [](const std::string& e) { + return e.find("broken.mod") != std::string::npos; + }), + "an unparseable manifest must be reported by scan_errors()"); + } + + /* Channels. A feature carries its own maturity: a package is the trust and + * installation boundary, and tying the two forced a whole catalog down to + * the maturity of its least finished feature. */ + { + const fs::path chan_root = root / "channel"; + const std::string mixed = + "format_version = 6\n" + "id = \"chan.mod\"\n" + "version = \"1.0.0\"\n" + "name = \"Channel Mod\"\n" + "[[target]]\n" + "game_id = \"SLUS-TEST\"\n" + "[[feature]]\n" + "id = \"solid\"\n" + "name = \"Solid\"\n" + "[[feature]]\n" + "id = \"rough\"\n" + "name = \"Rough\"\n" + "channel = \"experimental\"\n" + "[[feature]]\n" + "id = \"probe\"\n" + "name = \"Probe\"\n" + "channel = \"developer\"\n" + "[[option]]\n" + "feature = \"probe\"\n" + "id = \"depth\"\n" + "label = \"Depth\"\n" + "type = \"boolean\"\n" + "default = \"false\"\n"; + write_text(chan_root / "bundled/chan.mod/1.0.0/manifest.toml", mixed); + + /* A local build sees all three. */ + ModPackageManager local(chan_root); + local.set_developer_channel_visible(true); + check(local.scan(&error), error.c_str()); + const ModPackage& all = local.packages().at("chan.mod").at("1.0.0"); + check(all.features.size() == 3, "a local build must see every channel"); + check(all.features[0].channel == ModChannel::Stable && + all.features[1].channel == ModChannel::Experimental && + all.features[2].channel == ModChannel::Developer, + "each feature must carry its own channel"); + + /* A release build must not carry the developer feature at all -- + * absent, not hidden -- nor the options that only served it. */ + ModPackageManager shipped(chan_root); + shipped.set_developer_channel_visible(false); + check(shipped.scan(&error), error.c_str()); + const ModPackage& published = shipped.packages().at("chan.mod").at("1.0.0"); + check(published.features.size() == 2, + "a release build must drop developer-channel features"); + check(std::none_of(published.features.begin(), published.features.end(), + [](const ModFeature& f) { return f.id == "probe"; }), + "the developer feature must be absent from a release build"); + check(std::none_of(published.options.begin(), published.options.end(), + [](const ModOption& o) { return o.feature_id == "probe"; }), + "options serving a dropped feature must go with it"); + check(published.features[1].channel == ModChannel::Experimental, + "an experimental feature still ships"); + + /* A package whose every feature is developer-channel simply is not + * there on a release build, rather than listing with nothing in it. */ + write_text(chan_root / "bundled/dev.only/1.0.0/manifest.toml", + "format_version = 6\n" + "id = \"dev.only\"\n" + "version = \"1.0.0\"\n" + "name = \"Dev Only\"\n" + "channel = \"developer\"\n" + "[[target]]\n" + "game_id = \"SLUS-TEST\"\n" + "[[feature]]\n" + "id = \"instrument\"\n" + "name = \"Instrument\"\n"); + ModPackageManager dev_only(chan_root); + dev_only.set_developer_channel_visible(false); + check(dev_only.scan(&error), error.c_str()); + check(dev_only.packages().count("dev.only") == 0, + "a developer-only package must not load in a release build"); + ModPackageManager dev_seen(chan_root); + dev_seen.set_developer_channel_visible(true); + check(dev_seen.scan(&error), error.c_str()); + check(dev_seen.packages().count("dev.only") == 1, + "a developer-only package must load in a local build"); + check(dev_seen.packages().at("dev.only").at("1.0.0") + .features[0].channel == ModChannel::Developer, + "a feature must inherit its package's channel by default"); + + /* The key is version-gated like every other format addition, and only + * the three names are accepted. */ + ModPackage rejected; + write_text(chan_root / "v5-feature-channel.toml", + "format_version = 5\n" + "id = \"old.mod\"\n" + "version = \"1.0.0\"\n" + "name = \"Old\"\n" + "[[target]]\n" + "game_id = \"SLUS-TEST\"\n" + "[[feature]]\n" + "id = \"f\"\n" + "name = \"F\"\n" + "channel = \"experimental\"\n"); + check(!ModPackageManager::read_manifest( + chan_root / "v5-feature-channel.toml", rejected, &error), + "a per-feature channel must require format_version 6"); + write_text(chan_root / "bad-channel.toml", + "format_version = 6\n" + "id = \"bad.mod\"\n" + "version = \"1.0.0\"\n" + "name = \"Bad\"\n" + "channel = \"beta\"\n" + "[[target]]\n" + "game_id = \"SLUS-TEST\"\n" + "[[feature]]\n" + "id = \"f\"\n" + "name = \"F\"\n"); + check(!ModPackageManager::read_manifest( + chan_root / "bad-channel.toml", rejected, &error), + "an unknown channel name must be rejected"); + } + fs::remove_all(root, ec); if (failures) { std::cerr << failures << " mod package test(s) failed\n"; diff --git a/tools/mod_channel_filter.py b/tools/mod_channel_filter.py new file mode 100644 index 00000000..1ff601a6 --- /dev/null +++ b/tools/mod_channel_filter.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""Drop developer-channel mod features from a staged catalog tree. + +Developer-channel work does not ship. A contributor reaches it by cloning the +repo and building locally; a release must not carry it at all -- absent, not +hidden behind a toggle. + +Channels are per FEATURE, so pruning by package directory is no longer right: +a catalog may hold a stable feature and an instrument side by side, and dropping +the directory would take the player-facing feature with it. That is exactly what +a line-anchored `grep '^channel = "developer"'` does, because it cannot tell a +package-level key from one inside a [[feature]] block. + +What this does instead: + + * package-level channel = "developer", or every feature developer + -> remove the version directory (nothing in it ships) + * some features developer + -> emit a manifest without those features, and without the + [[option]] / [[patch]] / [[overlay]] / [[plugin]] / [[resource]] / + [[constraint]] entries that only served them + +Emitting is not rewriting someone's archive. A staged catalog is build output +generated from the title's mods/preloaded/ source (or the framework's +mods/builtin/); the author's manifest is the one in the repo and is never +touched. Installed .psxmod archives are a different matter -- those are never +modified, and the runtime declines to surface developer features from them +instead (see PSX_MOD_DEVELOPER_CHANNEL). + +Usage: + mod_channel_filter.py [ ...] [--quiet] + +A catalog dir directly contains //manifest.toml. Missing +directories are skipped. Exits non-zero if any manifest could not be parsed or +if a filtered manifest failed to re-parse, so packaging fails closed rather +than publishing unfinished work. +""" + +from __future__ import annotations + +import argparse +import re +import shutil +import sys +from pathlib import Path + +try: + import tomllib +except ModuleNotFoundError: # pragma: no cover - Python < 3.11 + tomllib = None + try: + import tomli as tomllib # type: ignore[no-redef] + except ModuleNotFoundError: + pass + +DEVELOPER = "developer" + +# A top-level array-of-tables header starts a block. A dotted header +# ([[option.choice]]) belongs to the block above it and rides along with it. +_TABLE = re.compile(r"^\s*\[\[\s*([A-Za-z0-9_.]+)\s*\]\]\s*(?:#.*)?$") +# Only used on lines inside one block, so it cannot pick up a package-level key. +_FEATURE_KEY = re.compile(r"""^\s*feature\s*=\s*["']([^"']*)["']\s*(?:#.*)?$""") +_ID_KEY = re.compile(r"""^\s*id\s*=\s*["']([^"']*)["']\s*(?:#.*)?$""") + +# Entries that name the feature they serve and are meaningless without it. +DEPENDENT_TABLES = { + "option", "patch", "overlay", "plugin", "resource", "constraint", +} + + +class ManifestError(Exception): + pass + + +def _split_blocks(text: str) -> tuple[str, list[tuple[str, str]]]: + """Return (preamble, [(table-name, block-text), ...]). + + The preamble is everything before the first top-level [[table]] -- the + package keys. Each block keeps its own trailing blank lines and any comment + lines that precede its header, so removing one leaves the rest untouched. + """ + lines = text.splitlines(keepends=True) + preamble: list[str] = [] + blocks: list[tuple[str, list[str]]] = [] + pending: list[str] = [] # comments/blanks that belong to the NEXT header + + for line in lines: + match = _TABLE.match(line) + name = match.group(1) if match else None + if name is not None and "." not in name: + if blocks: + blocks[-1][1].extend([]) + blocks.append((name, pending + [line])) + pending = [] + continue + if not blocks: + if line.strip().startswith("#") or not line.strip(): + pending.append(line) + else: + preamble.extend(pending) + pending = [] + preamble.append(line) + continue + if line.strip().startswith("#") or not line.strip(): + pending.append(line) + continue + blocks[-1][1].extend(pending) + pending = [] + blocks[-1][1].append(line) + + if blocks: + blocks[-1][1].extend(pending) + else: + preamble.extend(pending) + return "".join(preamble), [(n, "".join(b)) for n, b in blocks] + + +def _block_value(block: str, pattern: re.Pattern[str]) -> str | None: + for line in block.splitlines(): + if _TABLE.match(line): + continue + match = pattern.match(line) + if match: + return match.group(1) + return None + + +def _block_channel(block: str) -> str | None: + for line in block.splitlines(): + if _TABLE.match(line): + continue + match = re.match(r"""^\s*channel\s*=\s*["']([^"']*)["']\s*(?:#.*)?$""", line) + if match: + return match.group(1) + return None + + +def filter_manifest(text: str) -> tuple[str | None, list[str]]: + """Return (filtered text or None to drop the package, dropped feature ids). + + None means nothing in this package ships: the package itself is + developer-channel, or every feature it defines is. + """ + if tomllib is None: + raise ManifestError( + "no TOML parser available (need Python 3.11+, or tomli installed)") + try: + parsed = tomllib.loads(text) + except Exception as exc: # tomllib.TOMLDecodeError, and anything it raises + raise ManifestError(f"unparseable manifest: {exc}") from exc + + package_channel = parsed.get("channel") + features = parsed.get("feature") or [] + if package_channel == DEVELOPER: + return None, [str(f.get("id", "")) for f in features] + + dropped = [ + str(f.get("id", "")) + for f in features + if f.get("channel", package_channel) == DEVELOPER + ] + if not dropped: + return text, [] + if len(dropped) == len(features): + return None, dropped + + preamble, blocks = _split_blocks(text) + kept: list[str] = [] + dropped_option_ids: set[tuple[str, str]] = set() + for name, block in blocks: + if name == "feature": + if _block_value(block, _ID_KEY) in dropped: + continue + elif name in DEPENDENT_TABLES: + owner = _block_value(block, _FEATURE_KEY) + if owner in dropped: + if name == "option": + dropped_option_ids.add((owner, _block_value(block, _ID_KEY) or "")) + continue + kept.append(block) + + filtered = preamble + "".join(kept) + try: + reparsed = tomllib.loads(filtered) + except Exception as exc: + raise ManifestError(f"filtered manifest does not parse: {exc}") from exc + surviving = {str(f.get("id", "")) for f in (reparsed.get("feature") or [])} + if surviving & set(dropped): + raise ManifestError( + "developer feature survived filtering: " + + ", ".join(sorted(surviving & set(dropped)))) + if not surviving: + raise ManifestError("filtering left a package with no features") + return filtered, dropped + + +def filter_catalog(catalog: Path, quiet: bool = False) -> tuple[int, int]: + """Filter every package under one catalog dir. Returns (removed, edited).""" + removed = edited = 0 + if not catalog.is_dir(): + return 0, 0 + for manifest in sorted(catalog.glob("*/*/manifest.toml")): + version_dir = manifest.parent + package_dir = version_dir.parent + try: + filtered, dropped = filter_manifest( + manifest.read_text(encoding="utf-8")) + except ManifestError as exc: + raise ManifestError(f"{manifest}: {exc}") from exc + if filtered is None: + shutil.rmtree(version_dir) + try: + package_dir.rmdir() + except OSError: + pass + removed += 1 + if not quiet: + print(f" excluded developer package: " + f"{package_dir.name}/{version_dir.name}") + continue + if dropped: + manifest.write_text(filtered, encoding="utf-8") + edited += 1 + if not quiet: + print(f" excluded developer feature(s) from " + f"{package_dir.name}/{version_dir.name}: " + f"{', '.join(sorted(dropped))}") + return removed, edited + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Drop developer-channel mod features from staged catalogs.") + parser.add_argument("catalog", type=Path, nargs="+", + help="directory holding //manifest.toml") + parser.add_argument("--quiet", action="store_true") + args = parser.parse_args() + + removed = edited = 0 + try: + for catalog in args.catalog: + r, e = filter_catalog(catalog, args.quiet) + removed += r + edited += e + except ManifestError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + if not args.quiet: + print(f"developer-channel filter: {removed} package(s) removed, " + f"{edited} manifest(s) filtered") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/new_project_layout/setup_project.ps1 b/tools/new_project_layout/setup_project.ps1 index 061a43b0..051cf695 100644 --- a/tools/new_project_layout/setup_project.ps1 +++ b/tools/new_project_layout/setup_project.ps1 @@ -398,7 +398,7 @@ New-Item -ItemType Directory -Force -Path (Join-Path $Root "scripts") | Out-Null New-Item -ItemType Directory -Force -Path (Join-Path $Root "tools") | Out-Null New-Item -ItemType Directory -Force -Path (Join-Path $Root "mods\preloaded\packages") | Out-Null New-Item -ItemType Directory -Force -Path (Join-Path $Root "assets") | Out-Null -# Empty mod catalog tree (runtime copies mods/preloaded -> beside the exe as mods/). +# Empty mod catalog tree (build stages mods/preloaded/packages -> /mods/bundled). @' # Preloaded mods @@ -410,9 +410,15 @@ packages/// ... ``` -Build wiring copies `mods/preloaded` next to the game executable as `mods/`. -Install player `.psxmod` archives through the launcher Mods manager instead of -committing them here. See `psxrecomp/docs/MOD_PACKAGES.md`. +Build wiring copies `mods/preloaded/packages` next to the game executable as +`mods/bundled/`. That tree is build output: every build wipes and re-stages it, +so nothing you place there by hand survives. + +Player-installed `.psxmod` archives live in `mods/installed/`, which the +launcher owns and no build ever touches. Install them through the launcher Mods +manager rather than committing them here. + +See `psxrecomp/docs/MOD_PACKAGES.md`. '@ | Set-Content -Encoding utf8 (Join-Path $Root "mods\preloaded\README.md") New-Item -ItemType File -Force -Path (Join-Path $Root "mods\preloaded\packages\.gitkeep") | Out-Null Copy-Item (Join-Path $ScriptDir "sync_symbols.py") (Join-Path $Root "tools\sync_symbols.py") diff --git a/tools/new_project_layout/setup_project.sh b/tools/new_project_layout/setup_project.sh index 6f7a658f..8011a311 100755 --- a/tools/new_project_layout/setup_project.sh +++ b/tools/new_project_layout/setup_project.sh @@ -620,7 +620,7 @@ chmod +x "$ROOT/tools/sync_symbols.py" if [ -f "$TEMPLATE_DIR/raid-discord.png" ]; then cp "$TEMPLATE_DIR/raid-discord.png" "$ROOT/.github/raid-discord.png" fi -# Empty mod catalog tree (runtime copies mods/preloaded → beside the exe as mods/). +# Empty mod catalog tree (build stages mods/preloaded/packages → /mods/bundled). cat > "$ROOT/mods/preloaded/README.md" <<'EOF' # Preloaded mods @@ -632,9 +632,15 @@ packages/// … ``` -Build wiring copies `mods/preloaded` next to the game executable as `mods/`. -Install player `.psxmod` archives through the launcher Mods manager instead of -committing them here. See `psxrecomp/docs/MOD_PACKAGES.md`. +Build wiring copies `mods/preloaded/packages` next to the game executable as +`mods/bundled/`. That tree is build output: every build wipes and re-stages it, +so nothing you place there by hand survives. + +Player-installed `.psxmod` archives live in `mods/installed/`, which the +launcher owns and no build ever touches. Install them through the launcher Mods +manager rather than committing them here. + +See `psxrecomp/docs/MOD_PACKAGES.md`. EOF : > "$ROOT/mods/preloaded/packages/.gitkeep" diff --git a/tools/package_release_macos.sh b/tools/package_release_macos.sh index 48e15c36..84727bac 100755 --- a/tools/package_release_macos.sh +++ b/tools/package_release_macos.sh @@ -40,7 +40,7 @@ # --mods-src DIR override catalog source (default: the build tree's # mods/, which carries framework builtins AND the title's # own packages; per-machine state.toml is stripped) -# --require-mod PKG path under mods/packages that must exist (repeatable) +# --require-mod PKG path under mods/bundled that must exist (repeatable) # --data FILE file the runtime reads BESIDE the executable, relative to # repo, e.g. keybinds.ini (repeatable) # --doc FILE extra doc to ship, relative to repo (repeatable) @@ -142,7 +142,7 @@ cp "$BIN" "$APPDIR/Contents/MacOS/$BIN_NAME" # they must be reachable from Contents/MacOS. They cannot literally live there: # codesign treats every non-executable under Contents/MacOS as a nested-code # candidate and rejects the whole bundle ("bundle format unrecognized, invalid, -# or unsuitable") — mods/packages/// parses as a versioned bundle, +# or unsuitable") — mods/bundled/// parses as a versioned bundle, # assets/img/*.tga as stray nested code. So data lives in Contents/Resources and # Contents/MacOS carries symlinks: codesign seals a symlink as a symlink instead # of descending into it, and runtime path resolution is unchanged. @@ -189,19 +189,22 @@ _mod_manifests=$(find "$APPDIR/Contents/Resources/mods" -name manifest.toml 2>/d || die "staged mods/ contains no manifest.toml; the catalog would ship empty" [ -z "$(find "$APPDIR/Contents/Resources/mods" -name 'state.toml*' 2>/dev/null)" ] \ || die "per-machine mods/state.toml survived staging" -# Developer-channel packages (channel = "developer") are unfinished work that -# ships with local builds but must not be published. Prune by PACKAGE -# directory; never rewrite a manifest during packaging. +# Developer-channel work is unfinished: it ships with local builds and must +# never be published. Channels are per FEATURE, so this cannot be a grep -- a +# line-anchored match cannot tell a package-level key from one inside a +# [[feature]] block, and would drop a whole catalog over one instrument. +# mod_channel_filter.py parses instead, dropping the version directory when +# nothing in it ships and otherwise emitting a manifest without the developer +# features. The staged catalog is build output, so emitting it filtered is +# generation; the author's manifest in the repo is never touched. if [ "${EXCLUDE_DEV_MODS:-0}" = "1" ]; then note "excluding developer-channel mods" - find "$APPDIR/Contents/Resources/mods/packages" -mindepth 3 -maxdepth 3 \ - -name manifest.toml 2>/dev/null | while read -r m; do - if grep -Eq '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' "$m"; then - v=$(dirname "$m"); p=$(dirname "$v") - rm -rf "$v"; rmdir "$p" 2>/dev/null || true - note " excluded developer package: $(basename "$p")/$(basename "$v")" - fi - done + command -v python3 >/dev/null 2>&1 \ + || die "no python3 on PATH; cannot filter developer-channel mods" + python3 "$(dirname "$0")/mod_channel_filter.py" \ + "$APPDIR/Contents/Resources/mods/bundled" \ + "$APPDIR/Contents/Resources/mods/packages" \ + || die "developer-channel filtering failed" _dev_left=$( { grep -rlE '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' \ "$APPDIR" --include=manifest.toml 2>/dev/null || true; } | wc -l) [ "$_dev_left" -eq 0 ] || die "developer manifest(s) survived pruning: $_dev_left" @@ -334,8 +337,8 @@ note "bundled OpenBIOS (512 KiB) + MIT notice" # 4. Required mod packages. A silently mod-less launcher looks fine but has no # Mods page at all, which is indistinguishable from the feature being cut. for m in "${REQUIRE_MODS[@]}"; do - [ -f "$APPDIR/Contents/Resources/mods/packages/$m" ] \ - || die "required mod package missing: mods/packages/$m" + [ -f "$APPDIR/Contents/Resources/mods/bundled/$m" ] \ + || die "required mod package missing: mods/bundled/$m" done [ ${#REQUIRE_MODS[@]} -eq 0 ] || note "mod catalog: ${#REQUIRE_MODS[@]} required package(s) present" diff --git a/tools/package_setup_host.sh b/tools/package_setup_host.sh index af9c629c..977802e3 100755 --- a/tools/package_setup_host.sh +++ b/tools/package_setup_host.sh @@ -298,9 +298,11 @@ fi # Exe-relative runtime data (mods catalog, bezels, ...): the runtime resolves # these from the exe's own directory, so ship them at the stage root exactly -# as the build tree staged them. mods/state.toml is the LOCAL machine's mod -# enable/disable state -- preloaded catalogs ship default-disabled, so a dev -# machine's selections must never leak into a release. +# as the build tree staged them. Two things under mods/ belong to the local +# machine and must never leak into a release: state.toml is this machine's mod +# enable/disable state (preloaded catalogs ship default-disabled), and +# installed/ holds .psxmod archives the developer installed as a player would. +# Only mods/bundled/ is build output, and only build output ships. copy_runtime_dir() { local name="$1" optional="$2" if [[ ! -d "${EXE_DIR}/${name}" ]]; then @@ -315,36 +317,40 @@ copy_runtime_dir() { mkdir -p "${STAGE}/${name}" if command -v rsync >/dev/null 2>&1; then rsync -a --exclude 'state.toml' --exclude 'state.toml.tmp' \ + --exclude 'installed' --exclude 'installed/***' \ "${EXE_DIR}/${name}/" "${STAGE}/${name}/" else cp -a "${EXE_DIR}/${name}/." "${STAGE}/${name}/" rm -f "${STAGE}/${name}/state.toml" "${STAGE}/${name}/state.toml.tmp" + rm -rf "${STAGE}/${name}/installed" fi echo "staged runtime dir ${name}/" } -# Remove every channel = "developer" package from a staged catalog tree. -# /packages///manifest.toml is the layout; drop the VERSION -# directory that declares the channel, then the package directory once it has -# no versions left. Matching is on the manifest's own declaration so a package -# carries its own status -- nothing here maintains a list of names to exclude. -prune_dev_mods() { # prune_dev_mods - local tree="$1" manifest pkgdir verdir dropped=0 - [[ -d "${tree}/packages" ]] || return 0 - while IFS= read -r manifest; do - # Tolerate spacing/quoting variants; anchor to a top-level key so a - # feature description mentioning the word cannot trigger a drop. - if grep -Eq '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' \ - "${manifest}"; then - verdir="$(dirname "${manifest}")" - pkgdir="$(dirname "${verdir}")" - rm -rf "${verdir}" - rmdir "${pkgdir}" 2>/dev/null || true - echo " excluded developer package: $(basename "${pkgdir}")/$(basename "${verdir}")" - dropped=$((dropped + 1)) - fi - done < <(find "${tree}/packages" -mindepth 3 -maxdepth 3 -name manifest.toml 2>/dev/null) - return 0 +# Drop developer-channel work from a staged catalog tree. +# +# Channels are per FEATURE, so this cannot be a grep: a line-anchored match on +# `channel = "developer"` cannot tell a package-level key from one inside a +# [[feature]] block, and would drop a whole catalog because one instrument in +# it is unfinished. mod_channel_filter.py parses the manifest, removes the +# version directory when nothing in it ships, and otherwise emits a manifest +# without the developer features and the entries that only served them. +# +# Fails closed: if the filter cannot run, packaging stops rather than +# publishing unfinished work. +prune_dev_mods() { # prune_dev_mods //manifest.toml>... + local python="" + for candidate in python3 python; do + if command -v "${candidate}" >/dev/null 2>&1; then python="${candidate}"; break; fi + done + if [[ -z "${python}" ]]; then + echo "error: no python3 on PATH; cannot filter developer-channel mods" >&2 + exit 1 + fi + if ! "${python}" "${SCRIPT_DIR}/mod_channel_filter.py" "$@"; then + echo "error: developer-channel filtering failed" >&2 + exit 1 + fi } if ((${#RUNTIME_DIRS[@]})); then @@ -384,14 +390,16 @@ for d in "${PROJECT_DIRS[@]}"; do done # Developer-channel pruning runs AFTER every catalog is staged, and covers BOTH -# copies: the runtime catalog under mods/packages that the launcher lists, and +# copies: the runtime catalog under mods/bundled that the launcher lists, and # the source catalog under mods/preloaded that the setup-host rebuild re-stages # from. Pruning only the first would publish the package in source form and let -# the player's own rebuild put it straight back. +# the player's own rebuild put it straight back. mods/packages is the pre-split +# layout, kept here so an older staged tree is still pruned rather than shipped. if [[ "${EXCLUDE_DEV_MODS}" -eq 1 ]]; then echo "excluding developer-channel mods from this package" - prune_dev_mods "${STAGE}/mods" - prune_dev_mods "${STAGE}/mods/preloaded" + prune_dev_mods "${STAGE}/mods/bundled" \ + "${STAGE}/mods/preloaded/packages" \ + "${STAGE}/mods/packages" _dev_left=$( { grep -rlE '^[[:space:]]*channel[[:space:]]*=[[:space:]]*"developer"[[:space:]]*$' \ "${STAGE}" --include=manifest.toml 2>/dev/null || true; } | wc -l) if [[ "${_dev_left}" -ne 0 ]]; then diff --git a/tools/release_overlay_stage.ps1 b/tools/release_overlay_stage.ps1 index 274bdf07..07e6f7e7 100644 --- a/tools/release_overlay_stage.ps1 +++ b/tools/release_overlay_stage.ps1 @@ -298,17 +298,25 @@ silently drops them (pattern and rationale from MegaManX6). #> function Add-ModCatalog { param( - [Parameter(Mandatory)][string]$BuildPath, # build dir (has mods/packages) + [Parameter(Mandatory)][string]$BuildPath, # build dir (has mods/bundled) [Parameter(Mandatory)][string]$Stage, [string]$GameModSource = "", # /mods/preloaded [string]$FrameworkModSource = "" # /mods/builtin ) $ModsSrc = Join-Path $BuildPath "mods" - if (-not (Test-Path (Join-Path $ModsSrc "packages"))) { + if (-not (Test-Path (Join-Path $ModsSrc "bundled"))) { throw "No mod catalog staged at $ModsSrc - build the runtime first (the framework stages its builtin packages there)" } Copy-Item -Recurse -Force $ModsSrc (Join-Path $Stage "mods") - $StagedPkgDir = Join-Path $Stage "mods\packages" + # mods/bundled is build output and ships. Two things under mods/ belong to + # this machine and must never reach a release: installed/ holds .psxmod + # archives the developer installed as a player would, and state.toml is + # their own enable/disable selection over a catalog that ships default-off. + $StagedMods = Join-Path $Stage "mods" + Remove-Item -Recurse -Force (Join-Path $StagedMods "installed") -ErrorAction SilentlyContinue + Remove-Item -Force (Join-Path $StagedMods "state.toml") -ErrorAction SilentlyContinue + Remove-Item -Force (Join-Path $StagedMods "state.toml.tmp") -ErrorAction SilentlyContinue + $StagedPkgDir = Join-Path $Stage "mods\bundled" $stagedIds = @(Get-ChildItem $StagedPkgDir -Directory -ErrorAction SilentlyContinue | ForEach-Object { $_.Name })