From e6495195b9a0a9b2cdb7983ef3cdc50cf0df6255 Mon Sep 17 00:00:00 2001 From: briaguya <70942617+briaguya0@users.noreply.github.com> Date: Fri, 24 Jul 2026 21:18:00 -0400 Subject: [PATCH] Add Torch migration plan Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset pipeline, following HarbourMasters/Torch#219 ("oot support"). Co-Authored-By: Claude --- torch-migration/PLAN.md | 458 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 458 insertions(+) create mode 100644 torch-migration/PLAN.md diff --git a/torch-migration/PLAN.md b/torch-migration/PLAN.md new file mode 100644 index 0000000000..bde3b1ba27 --- /dev/null +++ b/torch-migration/PLAN.md @@ -0,0 +1,458 @@ +# Migrate Shipwright from ZAPDTR/OTRExporter to Torch + +## Context + +SoH extracts OoT assets from the user's ROM with **ZAPDTR** (a ZAPD fork) driven by +**OTRExporter**, described by 7,680 ZAPD XMLs under `soh/assets/xml/`. Both are git +submodules, and ZAPD is additionally linked *into the game binary* so users can drag a +ROM onto SoH and get an `oot.o2r` with a progress bar. + +[HarbourMasters/Torch#219](https://github.com/HarbourMasters/Torch/pull/219) ("oot support", +merged 2026-07-24) landed the OoT factories in Torch. `briaguya0/zapd-to-torch-test-harness` +proves the swap is safe: **14 of 14 ROM targets extract byte-for-byte identical** to an +OTRExporter reference O2R from Shipwright `95d8f7e` (35,386–39,066 assets per ROM, +0 failed / 0 missing / 0 extra). + +**Goal:** delete ZAPDTR and OTRExporter entirely; Torch becomes the only asset pipeline. +**Shape:** one big PR — `develop` is never half-migrated. + +Verified compatibility that makes this viable: Torch's 0x40-byte resource header +(`src/factories/BaseFactory.cpp:3`) already matches `libultraship/src/ship/resource/ResourceLoader.cpp:246-259`, +and Torch's OoT FourCC `ResourceType`s are byte-identical to `soh/soh/resource/type/SohResourceType.h`. + +--- + +## Decisions + +| Question | Decision | +|---|---| +| ZAPDTR / OTRExporter | Full deletion | +| Torch consumption | `FetchContent`, pinned to `HarbourMasters/Torch` **`4cae44160693e1beb562e39dc301bd42278be1f9`** | +| Asset YAML home | Shipwright, but a **separate repo consumed as a submodule** for PR reviewability; flatten into the tree post-merge | +| In-game extraction | Keep it — link Torch as a **static lib** (`USE_STANDALONE=OFF`) and drive `Companion` directly | +| `soh.o2r` | Small **in-tree C++ packer** (no Python — Windows devs). Long term: Torch PR adding PNG support to `Companion::Pack` | +| `ExtractAssetHeaders` | **Dropped.** Freeze the 1,085 checked-in `.h` files; follow-up Torch PR (see Blocker below) | +| Release payload | Ship the raw 111 MB YAML tree; zip it (~15 MB) in a follow-up | +| Dep collisions | Try `FetchContent` as-is on Linux + Windows first, fix only what actually breaks | + +**Pin note:** `git diff --stat 9422bf4 upstream/main` = 0 lines; the harness-pinned fork +commit and `HarbourMasters/Torch` main have identical trees (`5d247f78…`). Nothing to upstream +for parity. Re-check `git diff 4cae4416 ` right before opening the PR in case main moves. + +--- + +## Blocker resolved up front: headers + +`torch header` **cannot** replace `ExtractAssetHeaders`, for two independent reasons: + +1. **No OoT factory registers a Header exporter.** Every `REGISTER(` in `src/factories/oot/*.h` + is `REGISTER(Binary, ...)`. `Companion::ParseNode` skips assets whose factory lacks an + exporter for the active `ExportType` (`Companion.cpp:882-884`), so ~10k assets — skeletons, + limbs, animations, collision, arrays, paths, cutscenes, text, scenes, rooms — would be + silently omitted. `gLinkAdultSkel` simply would not appear. +2. **No `#define d` line.** Torch emits only + `static const ALIGN_ASSET(2) char sym[] = "__OTR__path";`. SoH's headers emit a pair, and + soh source references the `d`-prefixed macros **24,698 times**. Also `#pragma once` vs + `#ifndef OBJECTS_..._H`, no `#include "align_asset_macro.h"`, and a flat + `headers/objects/foo.h` layout vs SoH's `objects/foo/foo.h`. + +The load-bearing part — the `__OTR__objects/…/gSym` path string — already matches byte-for-byte +(that's what the harness proves). What's missing is presentation plus ~14 mechanical Header +exporters. **This PR deletes the target and keeps the `.h` files as ordinary checked-in source** +(they already are, and contain no ZAPD-derived includes beyond `align_asset_macro.h`). +`soh/assets/soh_assets.h` is hand-written and was never generated. Document loudly in the PR +that header regeneration is temporarily unavailable; file the Torch PR in parallel. + +--- + +## Phase 0 — Verification gates (no Shipwright changes) + +All run in `zapd-to-torch-test-harness` against the committed `manifests/*.json`. + +**Gate A — `USE_STANDALONE=OFF` is byte-identical.** +The static-lib build has no CLI, so add a throwaway driver *in the harness* +(`torchlib-driver/`: `add_subdirectory(../torch)` with `USE_STANDALONE=OFF`, ~40-line `main.cpp` +constructing `Companion` and calling `Init(ExportType::Binary)`). Build both ways from the same +tree with `-DPORT_VERSION_ENDIANNESS=ON`, run all 14 versions, compare via `check.sh`. +Expected identical: the only reachable `#ifdef STANDALONE` difference is `enums:` parsing +(`Companion.cpp:1524`) and the Code exporter — and `assets/yml/config.yml` has no `enums:` key. +Smoke-test `pal_gc` first. + +**Gate B — OoT-only build is byte-identical.** +`-DBUILD_{SM64,MK64,SF64,PM64,FZERO,BK64,MARIO_ARTIST,NAUDIO}=OFF`. Run the 14-target matrix +(`ROM_VERSION= python3 tools/test_assets.py roms/.z64`). The 14/14 parity was measured +with defaults ON, so this is a genuine change. Two real effects: `BUILD_BK64=OFF` drops the +`BK64::TrySynthesizeRomConfig` fallback (irrelevant — all 17 hashes are in `config.yml`), and +`BUILD_NAUDIO=OFF` removes Torch's only tinyxml2 consumers. + +**Gate A′ — the shipping configuration.** Re-run Gate A *with* the Gate B flags. This exact +combination is what ships. + +**Gate E — `soh.o2r` baseline.** From current `develop`, build `GenerateSohOtr` and record a +sha256 manifest of all 1,041 entries. This is the acceptance test for the new packer. All 31 +convertible PNGs are 8-bit / colour-type 6, so decode risk is low; the risk is quantisation +(see Phase 4). + +**Gate D — upstream pin.** Already satisfied (trees equal). Just record the SHA. + +**Nothing in Shipwright changes until A, B, A′, E are green.** + +--- + +## Phase 1 — The asset YAML repo + +New repo (e.g. `HarbourMasters/soh-assets`), submoduled at **`assets/`** (top level, sibling of +`soh/`) so the eventual flatten is one `git rm` + `git add` of a single directory. + +``` +config.yml 17 ROM SHA1 → 14 version dirs (gbi: F3DEX2_OoT, sort: OFFSET, + primary_virtual_segment: 0x80, strict_declarations: true) +<14 version dirs>/ 20,353 .yml, 111 MB +xml/ SNAPSHOT of soh/assets/xml before deletion ← most important item here +filelists/ symbols/ moved from soh/assets/extractor (generator inputs) +dma/ supplemental/ 17 JSON each (generator inputs) +tools/ zapd_to_torch.py, extract_dma.py, generate_supplemental.py +manifests/ 14 sha256 manifests from the OTRExporter reference +README.md regeneration recipe + which SoH commit the reference came from +``` + +Python lives **here**, not in Shipwright — SoH devs never need it; regeneration is a maintainer +task. Regeneration: `extract_dma.py` → `generate_supplemental.py` → `zapd_to_torch.py` → +`manifest.sh` → re-run the 14-target check. + +**Snapshotting `soh/assets/xml/` into this repo before deleting it is non-negotiable** — it is +the only input from which the YAML can be regenerated. + +--- + +## Phase 2 — CMake + +### Root `CMakeLists.txt` + +Add **before** `add_subdirectory(libultraship)` (~L196) — ordering matters, Torch declares +`tinyxml2` and `zlib` with `OVERRIDE_FIND_PACKAGE`: + +```cmake +include(FetchContent) +set(USE_STANDALONE OFF CACHE BOOL "" FORCE) # static lib, linkable into soh +set(PORT_VERSION_ENDIANNESS ON CACHE BOOL "" FORCE) # 7-byte portVersion, OTRExporter parity +set(ROM_CRC_BSWAP OFF CACHE BOOL "" FORCE) +set(BUILD_UI OFF CACHE BOOL "" FORCE) # would FetchContent a 2nd libultraship +set(BUILD_STORMLIB OFF CACHE BOOL "" FORCE) # MANDATORY: target-name clash with LUS's `storm` +set(BUILD_OOT ON CACHE BOOL "" FORCE) +foreach(g SM64 MK64 SF64 PM64 FZERO BK64 MARIO_ARTIST NAUDIO) + set(BUILD_${g} OFF CACHE BOOL "" FORCE) +endforeach() +FetchContent_Declare(torch + GIT_REPOSITORY https://github.com/HarbourMasters/Torch.git + GIT_TAG 4cae44160693e1beb562e39dc301bd42278be1f9) +FetchContent_MakeAvailable(torch) +``` + +Remove: `add_subdirectory(ZAPDTR/ZAPD …)` (L199), `add_subdirectory(OTRExporter)` (L200), both +`install(TARGETS ZAPD …)` (L209, L292), the `soh/assets/extractor` + `soh/assets/xml` install +rules (L210-211, 215-216, 296-297), `find_package(Python3 …)` (L219), and the three custom +targets (L222-261). Fix the stale L178 comment ("to use libgfxd from ZAPDTR" — LUS FetchContents +libgfxd itself, so the Gfx debugger is unaffected). + +### Two new in-tree tools + +Torch can't be added twice, and the static-lib build has no `main` (`int main` is inside +`#if defined(STANDALONE)`, while `Companion* Companion::Instance;` at `src/main.cpp:15` is +deliberately *outside* it — so the lib gets the singleton storage and no entry point). + +- **`tools/torch-cli/`** → target `soh-torch`. ~40 lines: parse `--src/--dest/--version/--rom`, + construct `Companion`, `SetVersion`, `Init(ExportType::Binary)`. Links `torch`. One Torch build + serves both the game link and the build-time targets; CI never needs the upstream CLI. +- **`tools/soh-o2r-packer/`** → see Phase 4. + +### Replacement targets + +```cmake +add_custom_target(ExtractAssets + COMMAND ${CMAKE_COMMAND} -E rm -f oot.o2r oot-mq.o2r + COMMAND $ --src ${CMAKE_SOURCE_DIR}/assets + --dest ${CMAKE_BINARY_DIR}/soh --version ${CMAKE_PROJECT_VERSION} --rom ${SOH_ROM_PATH} + DEPENDS soh-torch) + +add_custom_target(GenerateSohOtr + COMMAND ${CMAKE_COMMAND} -E rm -r -f ${CMAKE_SOURCE_DIR}/soh/assets/custom/shaders/ + COMMAND ${CMAKE_COMMAND} -E copy_directory + ${CMAKE_SOURCE_DIR}/libultraship/src/fast/shaders/ ${CMAKE_SOURCE_DIR}/soh/assets/custom/shaders/ + COMMAND $ ${CMAKE_SOURCE_DIR}/soh/assets/custom + ${CMAKE_BINARY_DIR}/soh/soh.o2r ${CMAKE_PROJECT_VERSION} + DEPENDS soh-o2r-packer) +# ExtractAssetHeaders: DELETED +``` + +The LUS-shader copy must be preserved verbatim — it's why `soh/assets/custom/shaders/` exists and +it currently lives inside both deleted targets. + +### `soh/CMakeLists.txt` + +- L107-109: drop the `ZAPDLib` `add_subdirectory` guard (Torch comes from the root). +- L325: delete the dead `../ZAPDTR/ZAPD/resource/type` include — **that directory doesn't exist**. +- L599-612: swap the `soh/assets/extractor` + `soh/assets/xml` POST_BUILD copies for a copy of + `${CMAKE_SOURCE_DIR}/assets`; drop the ZAPD-only `assets/symbols` mkdir. +- L624/642/662/704: `ZAPDLib` → `torch`, keeping the existing + `NOT CMAKE_SYSTEM_NAME MATCHES "NintendoSwitch|CafeOS"` guard — **Switch and Wii U don't link + the extractor today and must not link Torch either.** +- Add `soh/soh/Extractor/TorchExtract.cpp`. + +### Known collision landmines (fix only if they fire) + +| Dep | Risk | +|---|---| +| **zlib** | Torch `FetchContent_Declare(zlib … OVERRIDE_FIND_PACKAGE)` unconditionally and links `zlibstatic` — but `grep zlib.h torch/src` = **0 hits**; it's only reachable via StormLib, which we disable. Redirects every later `find_package(ZLIB)` (libpng, libzip, SDL) and defines `zlibstatic` but not `ZLIB::ZLIB`. **Highest risk.** Best fix is a one-line Torch PR gating the fetch on `BUILD_STORMLIB OR BUILD_UI`. | +| **stb** | `lib/n64graphics/CMakeLists.txt:13-31` renames `stbi_write_*` → `n64g_stbi_write_*` **only `if(BUILD_UI)`**. `stbi_load*` is never renamed. LUS links its own `stb`. | +| **StringHelper / StrHash64** | Torch excludes its copies **only `if(BUILD_UI)`** (`CMakeLists.txt:146-152`), citing exactly this clash. SoH has the same files in `libultraship/include/ship/utils/`. | +| **tinyxml2** | `OVERRIDE_FIND_PACKAGE` hijacks LUS's `find_package(tinyxml2 REQUIRED)`. Torch fetches 10.0.0 — the same version `.github/actions/install-tinyxml2` installs, so declaring Torch first is likely *beneficial* (one copy instead of two). All Torch tinyxml2 consumers are `naudio`, which we disable. | +| **spdlog** | `find_package(spdlog QUIET)` runs first and SoH always has vcpkg spdlog, so no ABI clash. But Torch calls `spdlog::set_level`/`set_pattern` on the **global default logger** from the worker thread (`Companion.cpp:193-194, 1810-1814`), clobbering SoH's config at `OTRGlobals.cpp:808-815`. Save/restore around the call. | +| **storm** | Hard CMake error if `BUILD_STORMLIB=ON` (LUS already creates a `storm` target, and `INCLUDE_MPQ_SUPPORT` is ON at root L186). Avoided by keeping it OFF; `.o2r` uses vendored miniz anyway. | + +--- + +## Phase 3 — In-game extractor rewrite + +`Extractor::CallZapd` (`soh/soh/Extractor/Extract.cpp:641`) currently chdirs into a temp dir with +`assets/` symlinked and hand-builds 22 argv entries for +`extern "C" zapd_report(argc, argv, extractCount, totalExtract)`. Everything that needed the chdir +becomes constructor arguments — `grep current_path torch/src` hits only `main.cpp` (STANDALONE) and +`LusBackend.cpp` (BUILD_UI), so **Torch has no CWD dependence** when both `srcdir` and `destdir` +are passed. + +**Isolate Torch in its own TU.** `Companion.h` pulls in yaml-cpp, `n64/Cartridge.h`, a bare +`CONTAINS` macro, and the `torch` target exports Torch's `src/` **and** `lib/` (CLI11, TinySHA1, +`hj/`, `nposix/`) as PUBLIC includes. Don't mix that into `Extract.cpp` alongside game headers. + +- `soh/soh/Extractor/TorchExtract.h` — declarations only, no Torch types: + `SohTorch::CountAssetFiles(ymlDir)` and + `SohTorch::Extract(rom, srcDir, destDir, portVersion, std::atomic* progress)`. +- `soh/soh/Extractor/TorchExtract.cpp` — the only file that includes `Companion.h`. + +**Flow:** `srcDir = /assets`; `destDir = Mkdtemp()` (reuse `Extract.cpp:619`); +`totalExtract` = recursive count of `*.yml` under `srcDir/` minus `config.yml`; +run; then `rename(destDir/oot.o2r → exportdir/(IsMasterQuest() ? "oot-mq.o2r" : "oot.o2r"))`; +`remove_all(destDir)`. + +**Driving Companion:** +```cpp +auto* c = new Companion(fs::path(rom), ArchiveType::O2R, /*debug*/false, srcDir, destDir); +Companion::Instance = c; // bare singleton, no getter — factories deref it unconditionally +c->SetVersion(portVersion); +c->SetPhaseCallback([progress](int){ if (progress) ++*progress; }); +c->Init(ExportType::Binary); // Init IS the whole run; it calls Process() internally +``` + +- **Must `try/catch`.** 127 `throw std::runtime_error` across `torch/src`; nothing catches at the + top level, so an escape means `std::terminate`. The existing `extractionTask->get()` try/catch + at `OTRGlobals.cpp:730-734` turns a rethrow into the "Extraction Crashed" popup. +- **Must also stat the output.** `Process()` returns `void` and several fatal conditions just log + and `return` — no `config.yml` (`Companion.cpp:1304`), no config entry for the ROM hash (`1334`), + no `config:` node (`1390`), bad GBI (`1484`). A ROM dump outside the 17-hash table produces no + archive and no exception. +- **Fresh `Companion` per call.** `gProcessedFiles` (`Companion.h:300`) is never cleared — a second + `Process()` on the same instance skips *every* file. `gAddrMap`/`gParseResults`/`gWriteMap` also + accumulate. `RunExtract` can extract more than once per process (`PS_FIRST`→`PS_SECOND`, + `ES_EXTRACT_ARGS` loops over CLI args), so allocate anew and reassign `Companion::Instance`. + Watch: `AliasManager::Instance` is not cleared at end of `Process` (inert for OoT — no OoT factory + calls `Register` — but unproven), and `AudioManager::Instance` is leaked per run (`Companion.cpp:1584`). + +**Progress: use `SetPhaseCallback`, not `Init`'s atomic overload.** `ProcessFile` +(`Companion.cpp:1282-1293`) resets `*gAssetCounter = 0` and recomputes the total **once per YAML +file**, so with ~1,450 YAMLs a naive bar restarts 1,450 times. The phase callback has exactly one +call site (`gPhaseCallback(2)` at `Companion.cpp:1282-1283`), fired once per YAML at the +parse→export transition, after `NodeHasChanges` has already let the file through — a 1:1 match for +a denominator computed by counting files on disk. Sub-files from `AddSubFileAsset` are virtual names +never present on disk, so the mapping is exact. `NodeHasChanges` early-returns are impossible +because a fresh temp destdir has no `torch.hash.yml`. **Nothing in the ImGui progress bar +(`OTRGlobals.cpp:750-755`) needs to change.** `GetCurrentAssetName()` is available for a label. + +**The temp destdir is mandatory, not cosmetic.** `Process` unconditionally writes +`destdir/torch.hash.yml` (`Companion.cpp:1804`) and reads it back next run (`:708`). Extracting +straight into the app dir would litter it *and* make a second in-session extraction skip files it +thinks are unchanged. + +**MQ naming:** leave `config.yml` at `binary: oot.o2r` for all 17 hashes and rename on the way out +of the temp dir based on `IsMasterQuest()`. Torch offers no output-filename override +(`-d` sets only the directory; you'd get a *directory* named `oot-mq.o2r`), and this keeps SoH the +single source of truth for MQ-ness and preserves the harness's `/oot.o2r` expectation. + +**Delete:** `CallZapd` (Extract.cpp:640-696) and its decl in `Extract.h:66-67`, the `zapd_report` +extern, and `GetZapdVerStr()` (`:583`) — replaced by a `GetTorchVersionDir()` returning +`pal_gc`/`ntsc_u_gc`/… over the same CRC mapping. `Mkdtemp()` stays. + +--- + +## Phase 4 — The `soh.o2r` packer + +`tools/soh-o2r-packer/`, `target_link_libraries(soh-o2r-packer PRIVATE torch)`. Reusing `torch` +gets `ZWrapper` (miniz — the same zip writer that produces `oot.o2r`, so archives stay structurally +consistent), `Companion::ParseVersionString` for the 7-byte `portVersion`, and `stb_image` via the +vendored `N64Graphics`. No new dependency, no Python, works on Windows out of the box. + +Usage: `soh-o2r-packer `. Behaviour ported from +`OTRExporter/OTRExporter/Main.cpp:206-260`: + +1. **`..png`** (`rgba32|rgb5a1|i4|i8|ia4|ia8|ia16|ci4|ci8`) → convert; archive path is + the relative path with `..png` stripped. Payload = the 0x40 resource header + (endianness `0x00`, resType `Texture`, version 0, id `0xDEADBEEF…`, zero-padded) then + `u32 textureType, width, height, rawDataSize`, then the raw N64 bytes. + **Reproduce ZAPD's asymmetry:** the size field is the unpadded `GetRawDataSize()` while the + buffer written is `ALIGN8`-padded. +2. **Path contains `accessibility`** → `.json` added with full relative path; non-`.json` **skipped**. + All 12 files are `.json` today, so this is a no-op — replicate it anyway to avoid silent drift. +3. **Everything else** → raw byte copy, full relative path. +4. `portVersion` only. **No `version` file** — OTRExporter doesn't write one for `soh.o2r`, and + `sohArchiveVersionMatch` (`OTRGlobals.cpp:282-283`) only reads `portVersion`. + +**Critical: use ZAPD's quantisation, not n64graphics'.** `n64graphics`' `ia2raw` uses +`SCALE_8_4(x)` (`x*15/255`) where ZAPD uses `(x >> 4) & 0xF` — for `r = 0xF0` those give `0xE` +vs `0xF`, **different bytes**. Same for IA4. Decode with `stbi_load(path, &w, &h, &n, 4)`, then +hand-port ~80 lines from `ZAPDTR/ZAPD/ZTexture.cpp`: rgba32 (`:507`), ia16 (`:595`, `[r, a]`), +ia8 (`:578`, `((r>>4)&0xF)<<4 | ((a>>4)&0xF)`), ia4 (`:552`, `((r>>5)<<1) | (a != 0)`), plus +i4/i8/rgba16/ci4/ci8 for completeness so adding a texture later doesn't silently produce garbage. + +Only 31 of 1,041 files hit the conversion path (15 rgba32, 8 ia8, 5 ia16, 3 ia4); the rest — 923 +objects, 23 `.rgba16`, 22 json, 4 ttf, 2 xml, ~40 plain `.png` — are raw copies. + +**Acceptance:** sha256 of every one of the 1,041 entries against the Gate E manifest. + +**Long term:** the Torch PR is `Companion::Pack` gaining PNG conversion + a version file, after +which this collapses to `torch pack soh/assets/custom soh.o2r o2r -u 9.2.3`. Structure the format +table so it lifts into Torch wholesale. + +--- + +## Phase 5 — Deletions + +- **Submodules:** `ZAPDTR/` (1.9 MB), `OTRExporter/` (71 MB) — `.gitmodules` + `git rm` +- **`soh/assets/xml/`** — 7,680 files, 54 MB (**snapshot into the assets repo first**) +- `soh/assets/extractor/Config_*.xml` (14) and `TexturePool.xml` — ZAPD `-rconf` inputs, no runtime use +- `soh/assets/extractor/filelists/`, `soh/assets/extractor/symbols/` — **move** to the assets repo + (`extract_dma.py` needs filelists; symbols are referenced only from the deleted `Config_*.xml`, + with no runtime reader in `soh/soh` or `soh/src`) +- `copy-existing-otrs.cmake` +- CMake and source hunks listed in Phases 2–3 + +**Keep:** all 1,085 `soh/assets/**/*.h`, all 1,041 files under `soh/assets/custom/`. + +Net: −127 MB submodules, −54 MB XML, +111 MB YAML (submodule initially; ~7-15 MB packed). + +--- + +## Phase 6 — CI + +`.github/workflows/generate-builds.yml`: + +- **`generate-soh-otr` (L10-68)** does a full SoH configure — SDL2 from source, SDL2_net, + tinyxml2 from source, ccache, apt deps — purely to build `GenerateSohOtr`. The new packer has + no SDL/LUS dependency, so add a `SOH_TOOLS_ONLY` root guard that skips + `add_subdirectory(libultraship)`/`add_subdirectory(soh)` and configures only Torch + + `soh-o2r-packer`. Saves ~5-8 min per run and drops the `install-tinyxml2` dependency. Add a step + asserting the `soh.o2r` content manifest. +- **Platform build jobs** (`build-macos` L71, `build-linux`, `build-windows`, Switch, Wii U): + `submodules: true` already covers the new `assets/` submodule (it has no nested submodules). + Switch/Wii U must not build Torch — same `NintendoSwitch|CafeOS` guard as `ZAPDLib` today. + Package payload changes from `assets/xml/` to `assets/`. +- **`test-builds-on-distros.yml` L137-150** removes distro tinyxml2 and builds 10.0.0 from source. + **Leave it in this PR** — one variable at a time; delete in a follow-up once Torch's fetched + tinyxml2 is confirmed to satisfy LUS. +- `pr-artifacts.yml:51` already filters `soh.o2r`; no change. + +Optional follow-up: a CI job running Gate A/B against a checked-in manifest so Torch pin bumps are +validated automatically. Needs a ROM, so self-hosted or encrypted secret. + +--- + +## Risks, ranked + +1. **zlib double-link** — Torch links an unused zlib and `OVERRIDE_FIND_PACKAGE` hijacks the whole + build's `find_package(ZLIB)`. Most likely thing to break the link, especially on Windows/vcpkg-static. +2. **stb / StringHelper / StrHash64 duplicate symbols** — Torch's own guards for these are + `if(BUILD_UI)`, which we don't set. +3. **`soh.o2r` texture byte-identity** — libpng→stb_image plus the `SCALE_8_4` vs `>>4` trap. + Fully testable offline (Gate E); de-risk first, costs an afternoon. +4. **Windows build** — Torch forces `/MT`, adds `-DSTORMLIB_NO_AUTO_LINK`, `/bigobj`, and carries a + `cmake_minimum_required(3.12)` + policy shim under a CMake 4 host. The one platform not + predictable from reading. Build it early. +5. **Extraction wall-clock** — Torch parses ~1,450 YAMLs per run and the parse phase is serial + (export is explicitly serial for Binary too: *"Binary/code/header all share wrapper state"*, + `Companion.cpp:1748`). Measure vs ZAPD; if it regresses, the UI needs a "this takes N minutes" note. +6. **Re-entrancy across two in-session extractions** — `AliasManager` uncleaned plus file-scope + statics in `TextureFactory.cpp:14-15`, `CompressedTextureFactory.cpp:17-18`, + `DisplayListFactory.cpp:82`. Believed inert for OoT/Binary; only a real vanilla-then-MQ run proves it. +7. **spdlog global-state stomping** from Torch's worker thread. +8. **Header regeneration gap** until the follow-up Torch PR lands. +9. **111 MB submodule ergonomics** — clone time, and a flatten commit GitHub will refuse to render. +10. **`Companion::Instance` is a raw global** — `RunExtract` uses a 1-thread pool so it's serialised + today; add a comment/assert. + +--- + +## Verification + +**Archive level** +1. Gates A, B, A′, E (Phase 0) — 14/14, `0 failed / 0 not generated / 0 not in reference`. +2. Post-integration: run Shipwright's `soh-torch` over all 14 ROMs and feed the outputs to the + harness's `check.sh`/`test_assets.py`. This proves the *Shipwright-configured* Torch + (FetchContent, OoT-only, static lib, SoH's flags) matches — not just the harness build. +3. `soh.o2r`: full 1,041-file sha256 diff vs an OTRExporter archive from `develop` HEAD. +4. Diff the in-game extractor's `oot.o2r` against `soh-torch`'s for the same ROM — different code + paths (temp destdir, phase callback), must agree byte-for-byte. + +**Build level** +5. Clean configure + build on Linux (gcc, clang), macOS (universal), Windows (MSVC x64), Switch, + Wii U. Switch/Wii U build with Torch excluded. +6. Confirm exactly one tinyxml2 and one zlib in the final link (`nm`/`dumpbin` for duplicate + `inflate`; `cmake --graphviz` for target duplication). +7. Confirm `soh` links with the untouched checked-in `.h` files after ZAPDTR is gone. + +**Run the game** +8. Cold start with no archives: ROM picker → progress bar (must be **monotonic 0→100, not + sawtooth**) → "Extraction Complete" → boots. +9. **Two-ROM session** — extract vanilla, answer "Yes" to extract another, extract MQ in the same + process. Highest-value manual test: covers re-entrancy and both output names. +10. CLI path: `./soh ` → `ES_EXTRACT_ARGS`. +11. Failure paths: non-OoT ROM and a truncated ROM → clean popup, not a hang or half-written `.o2r`. +12. No litter: no `torch.hash.yml`, no temp dir surviving in the app directory. +13. Gameplay: boot vanilla and MQ saves, load a dungeon (scene/room/collision), open the pause menu, + check PNG-path textures (`gSun`, `gTriforcePiece`, `gDPad`), play a cutscene and a sequence, + and confirm the `ValidHashes` check at `OTRGlobals.cpp:941` accepts the new archives. +14. Build a CPack artifact per platform, unzip fresh, run from that directory, re-extract — catches + broken install rules for the assets payload. + +--- + +## Ordering + +``` +0. Torch PR: gate the zlib fetch, make stb/StringHelper guards unconditional [only if they fire] +1. Harness: Gates A, B, A′ +2. New assets repo: snapshot soh/assets/xml, publish YAML + generators +3. Gate E: soh.o2r manifest from current develop +--- single Shipwright PR from here --- +4. Add assets submodule; FetchContent Torch before libultraship +5. tools/torch-cli + tools/soh-o2r-packer +6. Replace ExtractAssets / GenerateSohOtr; delete ExtractAssetHeaders +7. TorchExtract.{h,cpp} + Extract.{h,cpp} rewrite +8. Delete ZAPDTR, OTRExporter, soh/assets/xml, extractor inputs +9. CI: generate-soh-otr slim-down, packaging payload +10. Full verification matrix +--- follow-ups --- +11. Torch PR: OoT Header exporters + `#define d` → restore ExtractAssetHeaders +12. Torch PR: PNG support in Companion::Pack → retire soh-o2r-packer +13. Zip the shipped YAML (111 MB → ~15 MB) +14. Flatten the assets submodule into the tree +``` + +## Critical files + +- `CMakeLists.txt` (178-179, 196-201, 208-216, 219-261, 292-297) +- `soh/CMakeLists.txt` (107-109, 325, 599-612, 624/642/662/704) +- `soh/soh/Extractor/Extract.cpp` (`CallZapd` 640-696, `GetZapdVerStr` 583, `Mkdtemp` 619), `Extract.h:66-67` +- `soh/soh/OTRGlobals.cpp` (`RunExtract` 398, extraction task 596-680, progress bar 750-755, ValidHashes 941) +- `OTRExporter/OTRExporter/Main.cpp:183-268` and `ZAPDTR/ZAPD/ZTexture.cpp:507-620` — the spec for Phase 4 +- Torch `src/Companion.{h,cpp}` (`Init` 185-395, `Process` 1299-1819, `Pack` 1821, `torch.hash.yml` + 708/1804, phase callback 1282-1293), `src/main.cpp:15`, `CMakeLists.txt` (21-37, 146-152, 228-235, + 402-428), `lib/n64graphics/CMakeLists.txt:13-31`