Shipwright/torch-migration/PHASE0.md

Phase 0 — Verification gates

Detail for PLAN.md Phase 0. Nothing in Shipwright changes until every gate here is green. All work happens in briaguya0/zapd-to-torch-test-harness.

---

Why there are gates at all

The harness proves 14/14 byte-for-byte parity — but for exactly one build of Torch, driven exactly one way:

	Harness measured	Shipwright will use
Build kind	executable (USE_STANDALONE=ON)	static lib (OFF)
Games compiled in	all 9 (defaults)	OoT only
Build type	Debug (-g)	Release (-O3)
Driver	main.cpp + CLI11	our own Companion calls
Extractions per process	one	up to two (vanilla, then MQ)
soh.o2r	n/a (OTRExporter makes it)	a new in-tree packer

Every row is an unproven variable. Each gate closes exactly one of them, one at a time, so a failure names its own cause. Running only the final combination would tell us "something broke" without saying what.

Reading the source says all six should be fine. Reading is how we got here; running is how we know.

---

Preconditions

• Everything builds inside the soh distrobox (distrobox enter soh) — cmake, ninja, and the compilers are not on the host PATH.
• Harness at ~/code/zapd-to-torch-test-harness, submodules initialised.
• roms/ populated: 19 ROM dumps → 14 version directories (ntsc_1-0, ntsc_1-1 and ntsc_1-2 have two dumps each; pal_mq_dbg has three). (PLAN.md says 17 hashes in a couple of places — it's 19. Corrected there.)
• assets/yml/ generated: 20,353 .yml / 119 MB across the 14 version dirs, plus the committed config.yml. Gitignored — regenerate with zapd_to_torch.py if absent.
• o2r/ holds all 19 OTRExporter reference archives (one per ROM dump, from Shipwright 95d8f7e).
• Existing baseline build at torch/build/ (USE_STANDALONE=ON, all games ON, Debug, PORT_VERSION_ENDIANNESS=ON) — leave it alone; it is the control.

---

How the gates are actually run

Do not use tools/test_assets.py for these gates. It hardcodes TORCH to torch/build/torch, and it copies a filtered subset of the yml into a scratch dir (1,320 of 1,450 files for pal_gc) — which is not the invocation SoH will make. It's the right tool for bisecting a single failing asset, not for proving a configuration.

Use the full-tree + check.sh path instead. It points Torch at the entire yml tree — the same shape as Phase 1's srcdir = assets/ — and diffs the two archives file-by-file:

OUT=$(mktemp -d)                                   # fresh every run — see below
"$TORCH_BIN" o2r -s assets/yml -d "$OUT" -u 9.2.3 roms/<rom>.z64
cp "$OUT/oot.o2r" o2r/torch.o2r
cp "o2r/<rom>.o2r" o2r/reference.o2r
./check.sh                                          # missing / extra / mismatched, all three

check.sh never invokes Torch, so it takes any binary — Gate A's driver drops straight in with no harness patching.

The fresh destdir is mandatory, not hygiene. Process() writes destdir/torch.hash.yml and reads it back on the next run to skip unchanged files. Reuse a destdir and the second run silently produces a partial archive. (This is the same constraint that forces Phase 3's Mkdtemp().)

New: tools/matrix.sh

One script, used by every gate:

tools/matrix.sh <torch-bin> <label>

• loops all 19 ROM dumps, fresh destdir each
• writes logs/matrix-<label>/<rom>.log
• prints a 19-row PASS/FAIL table and exits non-zero on any failure

Cost: ~20 s per ROM (19 s of that is Torch), so ~7 min per configuration. Four configurations plus preflight is well under an hour of wall-clock; the builds dominate.

---

Preflight — reproduce the known-good result

Before changing a single variable, re-run the current configuration on this machine, today:

tools/matrix.sh torch/build/torch baseline

Expect 19/19 PASS: All N files match!. Without this, a later gate failure is ambiguous between "the variable broke it" and "something drifted in the environment."

Also confirm the reference archives are what we think they are (the README claims this for pal_gc; spot-check one more):

./manifest.sh o2r/ntsc_u_gc_b82710.o2r /tmp/regen.json
diff <(jq -S . manifests/ntsc_u_gc.json) <(jq -S . /tmp/regen.json) && echo MATCH

---

Gate D — the upstream pin (already green)

HarbourMasters/Torch main and the harness's pinned fork commit 9422bf4 have identical trees (5d247f78…; git diff --stat = 0 lines). Nothing needs upstreaming for parity.

Action: record the merge SHA 4cae44160693e1beb562e39dc301bd42278be1f9 and re-run git diff 4cae4416 9422bf4 immediately before opening the Shipwright PR, in case main has moved. No build, no matrix run.

---

Gate A — static lib, driven the way SoH will drive it

Variable: USE_STANDALONE=OFF and the hand-rolled driver, together.

They're one gate on purpose. Testing "does the static lib link" without also testing "does driving Companion by hand produce the same bytes" would leave the more interesting half unproven — and the driver has to exist either way, because USE_STANDALONE=OFF compiles out int main.

The driver is a dry run of Phase 3

torchlib-driver/main.cpp is not throwaway scaffolding. It executes the exact sequence soh/soh/Extractor/TorchExtract.cpp will use, so if any assumption in Phase 3 is wrong, it fails here — offline, against a reference archive — instead of in the game with a progress bar spinning.

zapd-to-torch-test-harness/
  torchlib-driver/
    CMakeLists.txt
    main.cpp

CMakeLists.txt — note it deliberately does not set the BUILD_<game> flags or the build type; those are Gate B's and Gate C's variables and get passed on the command line:

cmake_minimum_required(VERSION 3.26)
project(torchlib-driver CXX)
set(CMAKE_CXX_STANDARD 20)

set(USE_STANDALONE          OFF CACHE BOOL "" FORCE)
set(PORT_VERSION_ENDIANNESS ON  CACHE BOOL "" FORCE)
set(ROM_CRC_BSWAP           OFF CACHE BOOL "" FORCE)
set(BUILD_UI                OFF CACHE BOOL "" FORCE)
set(BUILD_STORMLIB          OFF CACHE BOOL "" FORCE)

add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../torch ${CMAKE_BINARY_DIR}/torch)
add_executable(torchlib-driver main.cpp)
target_link_libraries(torchlib-driver PRIVATE torch)

torch exports src/, lib/, and the yaml-cpp includes as PUBLIC under USE_STANDALONE=OFF (torch/CMakeLists.txt:437-442), and spdlog becomes PUBLIC too — so nothing else is needed.

main.cpp — argv-compatible with torch o2r -s S -d D -u V <rom> so matrix.sh can't tell the two binaries apart:

#include <atomic>
#include <cstdio>
#include <filesystem>
#include <string>
#include "Companion.h"

static int RunOnce(const std::string& rom, const std::string& src,
                   const std::string& dest, const std::string& ver) {
    std::atomic<size_t> phases{ 0 };
    try {
        auto* c = new Companion(std::filesystem::path(rom), ArchiveType::O2R, false, src, dest);
        Companion::Instance = c;              // bare global; factories deref it unconditionally
        c->SetVersion(ver);
        c->SetPhaseCallback([&phases](int) { ++phases; });
        c->Init(ExportType::Binary);          // Init is the whole run; it calls Process() itself
    } catch (const std::exception& e) {
        fprintf(stderr, "driver: exception: %s\n", e.what());
        return 2;
    }
    if (!std::filesystem::exists(std::filesystem::path(dest) / "oot.o2r")) {
        fprintf(stderr, "driver: no oot.o2r produced\n");   // Process() returns void and has
        return 3;                                           // several log-and-return failure paths
    }
    fprintf(stderr, "driver: phases=%zu\n", phases.load());
    return 0;
}

…plus ~20 lines of argv parsing and, for Gate A2, a --second <rom> <dest> flag that calls RunOnce a second time in the same process.

Four Phase 3 claims this proves or refutes:

1. Companion::Instance can be assigned by hand. It's a raw global defined at src/main.cpp:15, outside the #if defined(STANDALONE) guard — so the static lib carries the storage but no entry point. Deliberate, but untested from outside.
2. try/catch is sufficient. 127 throw std::runtime_error sites across torch/src and nothing catches at the top level; an escape is std::terminate, not a popup.
3. Statting the output is necessary. Process() returns void and just logs-and-returns on missing config.yml (Companion.cpp:1304), unknown ROM hash (:1334), no config: node (:1390), bad GBI (:1484). A silent no-archive must be caught.
4. SetPhaseCallback is a usable progress denominator. One call site (gPhaseCallback(2), Companion.cpp:1282-1283), fired once per yml file at the parse→export transition. Compare the printed phases= against find assets/yml/<version> -name '*.yml' | wc -l (1,449–1,480 depending on version). If they match, the Phase 3 progress bar is monotonic 0→100; if they don't, the denominator theory is wrong and it's better to know now.

What Gate A is hunting

libgfxd. This is the one that could actually fail. torch/CMakeLists.txt:60-81 fetches and compiles libgfxd only if(USE_STANDALONE) — under OFF it isn't in the build at all. Reading the source says that's safe: every gfxd_* call sits inside #ifdef STANDALONE (DisplayListFactory.cpp:81-199 = DListCodeExporter + DebugDisplayList; DisplayListOverrides.cpp:18-224 = the GFXDOverride callbacks), while DListBinaryExporter (:239) and DListFactory::parse (:526) are outside it. Display lists are the largest asset class in the archive, so if that reading is wrong the gate fails immediately and unmistakably.

Other STANDALONE deltas, all expected inert:

• Companion.cpp:1524 gates enums: parsing — config.yml has no enums: key.
• DListCodeExporter isn't registered — we export Binary.
• yaml-cpp gains YAML_CPP_STATIC_DEFINE; spdlog moves PRIVATE→PUBLIC; the standalone-only find_package(ZLIB REQUIRED) disappears (zlibstatic from FetchContent remains).

One landmine worth writing down: if(NOT USE_STANDALONE AND EXISTS "/mnt/c/WINDOWS/system32/wsl.exe") (torch/CMakeLists.txt:295) FetchContents Microsoft GSL. It fires only inside WSL, and only in the static-lib configuration — i.e. exactly what Shipwright will use. Irrelevant in the distrobox; it will bite the first person who builds SoH under WSL after this lands.

Running it

cmake -S torchlib-driver -B torchlib-driver/build -GNinja -DCMAKE_BUILD_TYPE=Debug
cmake --build torchlib-driver/build -j
tools/matrix.sh torchlib-driver/build/torchlib-driver static-lib

Pass: 19/19 identical, and phases= matches the yml count for each version. Smoke-test pal_gc alone before committing to the full matrix.

---

Gate A2 — two extractions in one process

Variable: re-entrancy. Free to add once the driver exists, and it retires PLAN.md risk #6 offline instead of by hand in the game.

RunExtract can extract more than once per process (PS_FIRST → PS_SECOND, and ES_EXTRACT_ARGS loops over CLI args), so a real user extracting vanilla and then MQ in one session hits state SoH has never exercised:

• gProcessedFiles (Companion.h:300) is never cleared — a second Process() on the same instance skips every file. Phase 3's answer is a fresh Companion per call; this gate is what proves that's enough.
• gAddrMap / gParseResults / gWriteMap also accumulate.
• AliasManager::Instance isn't cleared at the end of Process (believed inert for OoT — no OoT factory calls Register — but unproven).
• AudioManager::Instance leaks per run (Companion.cpp:1584).
• File-scope statics in TextureFactory.cpp:14-15, CompressedTextureFactory.cpp:17-18, DisplayListFactory.cpp:82.

tools/matrix.sh --pair pal_gc_0227d7 pal_mq_f46239 <driver> reentrancy

One process, two fresh Companions, two destdirs; both archives compared against their own references. Pass: both identical — specifically, the second one must be identical, which is the whole point.

---

Gate B — OoT-only build

Variable: the eight non-OoT BUILD_<game> flags. Still USE_STANDALONE=ON, still Debug, so this is measured against the preflight baseline with exactly one thing changed.

cmake -S torch -B torch/build-oot -GNinja -DCMAKE_BUILD_TYPE=Debug \
      -DPORT_VERSION_ENDIANNESS=ON \
      -DBUILD_SM64=OFF -DBUILD_MK64=OFF -DBUILD_SF64=OFF -DBUILD_PM64=OFF \
      -DBUILD_FZERO=OFF -DBUILD_BK64=OFF -DBUILD_MARIO_ARTIST=OFF -DBUILD_NAUDIO=OFF
cmake --build torch/build-oot -j
tools/matrix.sh torch/build-oot/torch oot-only

Each BUILD_X=OFF drops -DX_SUPPORT and filters src/factories/x/* out of the source glob (torch/CMakeLists.txt:158-207). Two effects are real:

• BUILD_BK64=OFF removes BK64::TrySynthesizeRomConfig, Companion's fallback for a ROM hash absent from config.yml. All 19 of ours are present, so it's unreachable here — but it means an unknown dump behaves differently, which is precisely why Phase 3 must stat the output rather than trust the run.
• BUILD_NAUDIO=OFF removes Torch's only tinyxml2 consumers. It does not remove the tinyxml2 FetchContent — that's unconditional — so Phase 2's OVERRIDE_FIND_PACKAGE concern stands regardless.

Nothing else should touch a shared path. A cross-game factory registration or a shared header behind an #ifdef would show up as a diff, and that's what the gate is for.

---

Gate C — Release build

Variable: -DCMAKE_BUILD_TYPE=Release. Otherwise the baseline configuration.

This gate isn't in PLAN.md and should be. The harness's 14/14 is a Debug measurement (torch/build/CMakeCache.txt: CMAKE_BUILD_TYPE=Debug), and so is the local Shipwright build. Shipwright's CI and every release build Release — Torch sets CMAKE_CXX_FLAGS_RELEASE "-O3" for non-MSVC (torch/CMakeLists.txt:242). Any UB, uninitialised read, or unspecified evaluation order in Torch can change bytes between -g and -O3.

cmake -S torch -B torch/build-release -GNinja -DCMAKE_BUILD_TYPE=Release -DPORT_VERSION_ENDIANNESS=ON
cmake --build torch/build-release -j
tools/matrix.sh torch/build-release/torch release

Seven minutes to find out, versus finding out from a user whose release-build archive doesn't match their friend's. If it fails it's an upstream Torch bug and everything else stops until it's fixed.

---

Gate A′ — the shipping configuration

Variable: all of them at once — static lib + driver + OoT-only + Release.

Not implied by A ∧ A2 ∧ B ∧ C. #ifdef interactions are combinatorial, and this is the only configuration that will ever ship.

cmake -S torchlib-driver -B torchlib-driver/build-ship -GNinja -DCMAKE_BUILD_TYPE=Release \
      -DBUILD_SM64=OFF -DBUILD_MK64=OFF -DBUILD_SF64=OFF -DBUILD_PM64=OFF \
      -DBUILD_FZERO=OFF -DBUILD_BK64=OFF -DBUILD_MARIO_ARTIST=OFF -DBUILD_NAUDIO=OFF
cmake --build torchlib-driver/build-ship -j
tools/matrix.sh torchlib-driver/build-ship/torchlib-driver shipping
tools/matrix.sh --pair pal_gc_0227d7 pal_mq_f46239 torchlib-driver/build-ship/torchlib-driver shipping-pair

When A′ is green, that flag set is the spec. Phase 2's root-CMakeLists.txt block is a transcription of it, not a re-derivation — and main.cpp's RunOnce is a transcription into TorchExtract.cpp. Copy, don't rewrite.

---

Gate E — the soh.o2r baseline

Different in kind from the others: it doesn't test Torch at all. It captures the artifact the Phase 4 packer has to reproduce, from a pipeline that is about to be deleted. Do this while ZAPD still builds.

Current state: soh.o2r at the repo root, 1,042 entries = 1,041 files under soh/assets/custom/ + portVersion. No version file — matching sohArchiveVersionMatch (OTRGlobals.cpp:282-283), which only reads portVersion.

1. Rebuild rather than trust the file on disk, so provenance is unambiguous:

   cmake --build build --target GenerateSohOtr     # inside the soh distrobox
   

   GenerateSohOtr writes into the source tree: it rm -rfs and re-copies soh/assets/custom/shaders/ from libultraship/src/fast/shaders/ (3 files — directx/default.shader.hlsl, metal/default.shader.metal, opengl/default.shader.glsl). That directory is gitignored (soh/assets/.gitignore:8), which is why git status stays clean and why the on-disk count (1,041) exceeds git ls-files (1,038). Phase 4's packer must run after the same copy or the archives differ by exactly those 3 entries.

2. Record the output — manifest.sh already does exactly this job:

   ./manifest.sh /var/home/briaguya/code/Shipwright/soh.o2r manifests/soh_o2r.json
   

   Lives in the harness alongside the other 14 manifests. (Not in the yml repo — that holds yml and nothing else.)

3. Record the inputs too, or a future mismatch is ambiguous between "the packer is wrong" and "the assets changed":

   • sha256 of every file under soh/assets/custom/ → manifests/soh_custom_inputs.json
   • CMAKE_PROJECT_VERSION = 9.2.3 (CMakeLists.txt:9), the portVersion payload
   • the develop SHA the archive was built from

4. Call out the 31 files that matter. Only these hit the conversion path — everything else is a byte copy, and a byte copy that fails is a trivial bug:

   Format	Count
   .rgba32.png	15
   .ia8.png	8
   .ia16.png	5
   .ia4.png	3

   The other 1,010: 923 objects, 38 plain .png, 23 .rgba16, 22 json, 4 ttf, 3 shaders, 2 xml. The trap is quantisation, not decoding — n64graphics' SCALE_8_4(x) (x*15/255) vs ZAPD's (x >> 4) & 0xF disagree (0xF0 → 0xE vs 0xF). Phase 4 hand-ports ZAPD's arithmetic; this manifest is what catches it if the port is wrong.

---

Exit criteria

Gate	Pass condition
Preflight	19/19 identical with the existing baseline binary; one manifest re-derivation matches
D	pin SHA recorded; git diff vs upstream main still empty
A	19/19 identical; phases= matches the per-version yml count
A2	both archives in a two-extraction process identical — especially the second
B	19/19 identical
C	19/19 identical
A′	19/19 identical, plus the pair run
E	manifests/soh_o2r.json (1,042 entries) + input hashes + version + SHA committed

"Identical" means check.sh prints PASS: All N files match! — 0 missing, 0 extra, 0 mismatched, not "close enough".

Write the results up as a second table in the harness README, next to the existing 14/14 one. That table is the evidence the Shipwright PR points at.

---

Cost

	Build	Matrix
Preflight	—	~7 min
A	full Torch rebuild	~7 min
A2	— (reuses A)	~1 min
B	full Torch rebuild	~7 min
C	full Torch rebuild	~7 min
A′	full Torch rebuild	~8 min
E	Shipwright GenerateSohOtr	~1 min

An evening, dominated by four full Torch builds. Four extra build trees; the Debug Torch binary alone is 125 MB, so either build them sequentially and delete as you go or budget a few GB.

---

What Phase 0 deliberately does not cover

Being explicit so these don't get mistaken for retired risk:

• Linking Torch into Shipwright. The zlib / stb / StringHelper / StrHash64 / tinyxml2 / storm collisions in PLAN.md Phase 2 need SoH's own build to surface. A standalone driver links Torch against nothing else, so it can't see any of them. That's Phase 2's first day.
• spdlog global-state stomping. Torch calls spdlog::set_level/set_pattern on the default logger (Companion.cpp:193-194, 1810-1814), and our config.yml sets logging: CRITICAL — so this fires on every run, it just doesn't matter until there's a SoH logger to clobber.
• Windows and macOS. Every gate runs Linux/gcc in the distrobox. Torch forces /MT under MSVC, adds -DSTORMLIB_NO_AUTO_LINK and /bigobj, and carries a cmake_minimum_required(3.12) + CMAKE_POLICY_VERSION_MINIMUM shim under a CMake 4 host. If a Windows machine is available, running Gate A′ there is the cheapest possible insurance on the one platform that can't be predicted from reading — but it isn't a blocker for starting Phase 2.
• Anything in-game. The driver proves the bytes; it says nothing about the ImGui progress bar, the ROM picker, or ValidHashes. That's the Phase 3 manual matrix.