Two extractions in one process, vanilla then MQ, both byte-identical to their references. The second run's phases=1450 matches pal_mq's on-disk yml count exactly, which is the direct refutation of the gProcessedFiles risk -- a leak would have skipped files and truncated the archive. A fresh Companion per extraction is sufficient; PLAN.md risk #6 retired. Also records what the gate turned up on the way: torch writes 25 duplicate archive entries, benign because libultraship indexes by CRC64 of the path. Filed as Torch#233 item 5. Corrects the Gate A note that attributed the whole ~343 KB size delta to compression -- 76 KB of it is these duplicates.
30 KiB
Phase 0 — Verification gates
Detail for PLAN.md Phase 0. Nothing in Shipwright changes until every gate here is
green. All work happens in
briaguya0/zapd-to-torch-test-harness.
Why there are gates at all
The harness proves 14/14 byte-for-byte parity — but for exactly one build of Torch, driven exactly one way:
| Harness measured | Shipwright will use | |
|---|---|---|
| Build kind | executable (USE_STANDALONE=ON) |
static lib (OFF) |
| Games compiled in | all 9 (defaults) | OoT only |
| Build type | Debug (-g) |
Release (-O3) |
| Driver | main.cpp + CLI11 |
our own Companion calls |
| Extractions per process | one | up to two (vanilla, then MQ) |
soh.o2r |
n/a (OTRExporter makes it) | a new in-tree packer |
Every row is an unproven variable. Each gate closes exactly one of them, one at a time, so a failure names its own cause. Running only the final combination would tell us "something broke" without saying what.
Reading the source says all six should be fine. Reading is how we got here; running is how we know.
Preconditions
- Everything builds inside the
sohdistrobox (distrobox enter soh) —cmake,ninja, and the compilers are not on the hostPATH. - Harness at
~/code/zapd-to-torch-test-harness, submodules initialised. roms/populated: 19 ROM dumps → 14 version directories (ntsc_1-0,ntsc_1-1andntsc_1-2have two dumps each;pal_mq_dbghas three). (PLAN.mdsays 17 hashes in a couple of places — it's 19. Corrected there.)assets/yml/generated: 20,353.yml/ 119 MB across the 14 version dirs, plus the committedconfig.yml. Gitignored — regenerate withzapd_to_torch.pyif absent.o2r/holds all 19 OTRExporter reference archives (one per ROM dump, from Shipwright95d8f7e).- Existing baseline build at
torch/build/(USE_STANDALONE=ON, all games ON,Debug,PORT_VERSION_ENDIANNESS=ON) — leave it alone; it is the control.
How the gates are actually run
tools/test_assets.py, unfiltered — the same tool that produced the existing 14/14. Using
anything else would make the gate results not directly comparable to the baseline they're being
measured against.
An unfiltered run is a genuine full-tree, both-directions comparison:
list_assetswith no--category/--file/--typereturns every key inmanifests/<version>.json— all 35,386 forpal_gc.collect_yaml_filesreverse-maps those asset paths to yml files (1,320), thensetup_scratch'scopy_yaml_with_externalsrecursion followsexternal_files:and pulls in the remaining 130 — all 1,450 land in the scratch dir. Verified by replaying the logic:copied == on-disk, zero yml files omitted.- It hashes what Torch produced and diffs it against the manifest in both directions —
not generated(in the reference, missing from ours) andnot in reference(extras, computed over the whole zip, not just the filtered set). - The destdir is a fresh
mkdtempper run, which matters:Process()writesdestdir/torch.hash.ymland reads it back next run to skip unchanged files. Reuse a destdir and the second run silently produces a partial archive. (Same constraint that forces Phase 3'sMkdtemp().)
The
Copying 1320 YAMLs to scratch dirline in the logs is printed before theexternal_filesrecursion runs — it's the pre-recursion count, not what actually gets copied. Easy to misread as a filtered subset. It isn't one.
One harness patch is needed: the torch binary path is hardcoded in two places —
tools/test_assets.py:34 and lib.sh:12. Make both honour a TORCH_BIN environment override so
Gate A's driver can be dropped in without editing files between runs:
TORCH = os.environ.get("TORCH_BIN") or os.path.join(SOH_DIR, "torch", "build", "torch")
./check.sh stays as the secondary whole-archive check — it compares two extracted .o2r
directly, so it's the tool to reach for if a gate fails and you want the raw file-list diff rather
than a per-asset table.
New: tools/matrix.sh
There's no 19-ROM loop script today; every gate needs one. It's a thin wrapper, not new comparison logic:
TORCH_BIN=<binary> tools/matrix.sh <label>
- loops all 19 ROM dumps, mapping each to its
--rom-version - calls
test_assets.py <rom> --rom-version <ver> --failures-only - writes
logs/matrix-<label>/<rom>.log, prints a 19-row PASS/FAIL table, exits non-zero on any failure
Cost: ~20 s per ROM (19 s of that is Torch), so ~7 min per configuration. Four configurations plus preflight is well under an hour of wall-clock; the builds dominate.
Preflight — reproduce the known-good result
Before changing a single variable, re-run the current configuration on this machine, today:
TORCH_BIN=torch/build/torch tools/matrix.sh baseline
Expect 19/19 0 failed, 0 not generated, 0 not in reference. Without this, a later gate failure is ambiguous between
"the variable broke it" and "something drifted in the environment."
Also confirm the reference archives are what we think they are (the README claims this for
pal_gc; spot-check one more):
./manifest.sh o2r/ntsc_u_gc_b82710.o2r /tmp/regen.json
diff <(jq -S . manifests/ntsc_u_gc.json) <(jq -S . /tmp/regen.json) && echo MATCH
Gate D — the upstream pin (already green)
HarbourMasters/Torch main and the harness's pinned fork commit 9422bf4 have identical trees
(5d247f78…; git diff --stat = 0 lines). Nothing needs upstreaming for parity.
Action: record the merge SHA 4cae44160693e1beb562e39dc301bd42278be1f9 and re-run
git diff 4cae4416 9422bf4 immediately before opening the Shipwright PR, in case main has moved.
No build, no matrix run.
Gate A — static lib, driven the way SoH will drive it
Variable: USE_STANDALONE=OFF and the hand-rolled driver, together.
They're one gate on purpose. Testing "does the static lib link" without also testing "does driving
Companion by hand produce the same bytes" would leave the more interesting half unproven — and
the driver has to exist either way, because USE_STANDALONE=OFF compiles out int main.
The driver is a dry run of Phase 3
torchlib-driver/main.cpp is not throwaway scaffolding. It executes the exact sequence
soh/soh/Extractor/TorchExtract.cpp will use, so if any assumption in Phase 3 is wrong, it fails
here — offline, against a reference archive — instead of in the game with a progress bar spinning.
zapd-to-torch-test-harness/
torchlib-driver/
CMakeLists.txt
main.cpp
CMakeLists.txt — note it deliberately does not set the BUILD_<game> flags or the build
type; those are Gate B's and Gate C's variables and get passed on the command line:
cmake_minimum_required(VERSION 3.26)
project(torchlib-driver CXX)
set(CMAKE_CXX_STANDARD 20)
set(USE_STANDALONE OFF CACHE BOOL "" FORCE)
set(PORT_VERSION_ENDIANNESS ON CACHE BOOL "" FORCE)
set(ROM_CRC_BSWAP OFF CACHE BOOL "" FORCE)
set(BUILD_UI OFF CACHE BOOL "" FORCE)
set(BUILD_STORMLIB OFF CACHE BOOL "" FORCE)
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../torch ${CMAKE_BINARY_DIR}/torch)
add_executable(torchlib-driver main.cpp)
target_link_libraries(torchlib-driver PRIVATE torch)
torch exports src/, lib/, and the yaml-cpp includes as PUBLIC under USE_STANDALONE=OFF
(torch/CMakeLists.txt:437-442), and spdlog becomes PUBLIC too — so nothing else is needed.
main.cpp — must be argv-compatible with torch o2r -s S -d D -u V <rom>, because that's
exactly what test_assets.py:run_torch execs ([TORCH, "o2r", "-s", …, "-d", …, "-u", "9.2.3", rom]). Get that right and TORCH_BIN is the only thing that changes between the baseline and
every gate:
#include <atomic>
#include <cstdio>
#include <filesystem>
#include <string>
#include "Companion.h"
static int RunOnce(const std::string& rom, const std::string& src,
const std::string& dest, const std::string& ver) {
std::atomic<size_t> phases{ 0 };
try {
auto* c = new Companion(std::filesystem::path(rom), ArchiveType::O2R, false, src, dest);
Companion::Instance = c; // bare global; factories deref it unconditionally
c->SetVersion(ver);
c->SetPhaseCallback([&phases](int) { ++phases; });
c->Init(ExportType::Binary); // Init is the whole run; it calls Process() itself
} catch (const std::exception& e) {
fprintf(stderr, "driver: exception: %s\n", e.what());
return 2;
}
if (!std::filesystem::exists(std::filesystem::path(dest) / "oot.o2r")) {
fprintf(stderr, "driver: no oot.o2r produced\n"); // Process() returns void and has
return 3; // several log-and-return failure paths
}
fprintf(stderr, "driver: phases=%zu\n", phases.load());
return 0;
}
…plus ~20 lines of argv parsing and, for Gate A2, a --second <rom> <dest> flag that calls
RunOnce a second time in the same process.
Four Phase 3 claims this proves or refutes:
Companion::Instancecan be assigned by hand. It's a raw global defined atsrc/main.cpp:15, outside the#if defined(STANDALONE)guard — so the static lib carries the storage but no entry point. Deliberate, but untested from outside.try/catchis sufficient. 127throw std::runtime_errorsites acrosstorch/srcand nothing catches at the top level; an escape isstd::terminate, not a popup.- Statting the output is necessary.
Process()returnsvoidand just logs-and-returns on missingconfig.yml(Companion.cpp:1304), unknown ROM hash (:1334), noconfig:node (:1390), bad GBI (:1484). A silent no-archive must be caught. SetPhaseCallbackis a usable progress denominator. One call site (gPhaseCallback(2),Companion.cpp:1282-1283), fired once per yml file at the parse→export transition. Compare the printedphases=againstfind assets/yml/<version> -name '*.yml' | wc -l(1,449–1,480 depending on version). If they match, the Phase 3 progress bar is monotonic 0→100; if they don't, the denominator theory is wrong and it's better to know now.
What Gate A is hunting
libgfxd. This is the one that could actually fail. torch/CMakeLists.txt:60-81 fetches and
compiles libgfxd only if(USE_STANDALONE) — under OFF it isn't in the build at all. Reading
the source says that's safe: every gfxd_* call sits inside #ifdef STANDALONE
(DisplayListFactory.cpp:81-199 = DListCodeExporter + DebugDisplayList;
DisplayListOverrides.cpp:18-224 = the GFXDOverride callbacks), while DListBinaryExporter
(:239) and DListFactory::parse (:526) are outside it. Display lists are the largest asset
class in the archive, so if that reading is wrong the gate fails immediately and unmistakably.
Other STANDALONE deltas, all expected inert:
Companion.cpp:1524gatesenums:parsing —config.ymlhas noenums:key.DListCodeExporterisn't registered — we exportBinary.- yaml-cpp gains
YAML_CPP_STATIC_DEFINE; spdlog movesPRIVATE→PUBLIC; the standalone-onlyfind_package(ZLIB REQUIRED)disappears (zlibstaticfrom FetchContent remains).
One landmine worth writing down:
if(NOT USE_STANDALONE AND EXISTS "/mnt/c/WINDOWS/system32/wsl.exe") (torch/CMakeLists.txt:295)
FetchContents Microsoft GSL. It fires only inside WSL, and only in the static-lib
configuration — i.e. exactly what Shipwright will use. Irrelevant in the distrobox; it will bite
the first person who builds SoH under WSL after this lands.
Running it
cmake -S torchlib-driver -B torchlib-driver/build -GNinja -DCMAKE_BUILD_TYPE=Debug
cmake --build torchlib-driver/build -j
TORCH_BIN=torchlib-driver/build/torchlib-driver tools/matrix.sh static-lib
Pass: 19/19 identical, and phases= matches the yml count for each version.
Smoke-test pal_gc alone before committing to the full matrix.
Gate A2 — two extractions in one process
Variable: re-entrancy. Free to add once the driver exists, and it retires
PLAN.md risk #6 offline instead of by hand in the game.
RunExtract can extract more than once per process (PS_FIRST → PS_SECOND, and
ES_EXTRACT_ARGS loops over CLI args), so a real user extracting vanilla and then MQ in one
session hits state SoH has never exercised:
gProcessedFiles(Companion.h:300) is never cleared — a secondProcess()on the same instance skips every file. Phase 3's answer is a freshCompanionper call; this gate is what proves that's enough.gAddrMap/gParseResults/gWriteMapalso accumulate.AliasManager::Instanceisn't cleared at the end ofProcess(believed inert for OoT — no OoT factory callsRegister— but unproven).AudioManager::Instanceleaks per run (Companion.cpp:1584).- File-scope statics in
TextureFactory.cpp:14-15,CompressedTextureFactory.cpp:17-18,DisplayListFactory.cpp:82.
This is the one gate test_assets.py can't drive — it execs the binary once per ROM, so two
extractions in one process has to come from the driver itself. Hence the --second flag, and a
--pair mode in matrix.sh that invokes the driver directly and then hands each of the two
outputs to check.sh:
DEST_A=$(mktemp -d); DEST_B=$(mktemp -d)
"$TORCH_BIN" o2r -s assets/yml -d "$DEST_A" -u 9.2.3 roms/pal_gc_0227d7.z64 \
--second roms/pal_mq_f46239.z64 "$DEST_B"
# then, for each of (DEST_A, pal_gc_0227d7) and (DEST_B, pal_mq_f46239):
cp "$DEST/oot.o2r" o2r/torch.o2r; cp "o2r/<rom>.o2r" o2r/reference.o2r; ./check.sh
One process, two fresh Companions, two destdirs. Deliberately a vanilla ROM then an MQ one — the
same sequence a user hits when they answer "Yes" to "extract another". Pass: check.sh prints
PASS: All N files match! for both — specifically for the second, which is the whole point.
Gate B — OoT-only build
Variable: the eight non-OoT BUILD_<game> flags. Still USE_STANDALONE=ON, still Debug, so
this is measured against the preflight baseline with exactly one thing changed.
cmake -S torch -B torch/build-oot -GNinja -DCMAKE_BUILD_TYPE=Debug \
-DPORT_VERSION_ENDIANNESS=ON \
-DBUILD_SM64=OFF -DBUILD_MK64=OFF -DBUILD_SF64=OFF -DBUILD_PM64=OFF \
-DBUILD_FZERO=OFF -DBUILD_BK64=OFF -DBUILD_MARIO_ARTIST=OFF -DBUILD_NAUDIO=OFF
cmake --build torch/build-oot -j
TORCH_BIN=torch/build-oot/torch tools/matrix.sh oot-only
Each BUILD_X=OFF drops -DX_SUPPORT and filters src/factories/x/* out of the source glob
(torch/CMakeLists.txt:158-207). Two effects are real:
BUILD_BK64=OFFremovesBK64::TrySynthesizeRomConfig, Companion's fallback for a ROM hash absent fromconfig.yml. All 19 of ours are present, so it's unreachable here — but it means an unknown dump behaves differently, which is precisely why Phase 3 must stat the output rather than trust the run.BUILD_NAUDIO=OFFremoves Torch's only tinyxml2 consumers. It does not remove the tinyxml2FetchContent— that's unconditional — so Phase 2'sOVERRIDE_FIND_PACKAGEconcern stands regardless.
Nothing else should touch a shared path. A cross-game factory registration or a shared header
behind an #ifdef would show up as a diff, and that's what the gate is for.
Gate C — Release build
Variable: -DCMAKE_BUILD_TYPE=Release. Otherwise the baseline configuration.
This gate isn't in PLAN.md and should be. The harness's 14/14 is a Debug measurement
(torch/build/CMakeCache.txt: CMAKE_BUILD_TYPE=Debug), and so is the local Shipwright build.
Shipwright's CI and every release build Release — Torch sets CMAKE_CXX_FLAGS_RELEASE "-O3" for
non-MSVC (torch/CMakeLists.txt:242). Any UB, uninitialised read, or unspecified evaluation order
in Torch can change bytes between -g and -O3.
cmake -S torch -B torch/build-release -GNinja -DCMAKE_BUILD_TYPE=Release -DPORT_VERSION_ENDIANNESS=ON
cmake --build torch/build-release -j
TORCH_BIN=torch/build-release/torch tools/matrix.sh release
Seven minutes to find out, versus finding out from a user whose release-build archive doesn't match their friend's. If it fails it's an upstream Torch bug and everything else stops until it's fixed.
Gate A′ — the shipping configuration
Variable: all of them at once — static lib + driver + OoT-only + Release.
Not implied by A ∧ A2 ∧ B ∧ C. #ifdef interactions are combinatorial, and this is the only
configuration that will ever ship.
cmake -S torchlib-driver -B torchlib-driver/build-ship -GNinja -DCMAKE_BUILD_TYPE=Release \
-DBUILD_SM64=OFF -DBUILD_MK64=OFF -DBUILD_SF64=OFF -DBUILD_PM64=OFF \
-DBUILD_FZERO=OFF -DBUILD_BK64=OFF -DBUILD_MARIO_ARTIST=OFF -DBUILD_NAUDIO=OFF
cmake --build torchlib-driver/build-ship -j
export TORCH_BIN=torchlib-driver/build-ship/torchlib-driver
tools/matrix.sh shipping
tools/matrix.sh --pair pal_gc_0227d7 pal_mq_f46239 shipping-pair # Gate A2, shipping config
When A′ is green, that flag set is the spec. Phase 2's root-CMakeLists.txt block is a
transcription of it, not a re-derivation — and main.cpp's RunOnce is a transcription into
TorchExtract.cpp. Copy, don't rewrite.
Gate E — the soh.o2r baseline
Different in kind from the others: it doesn't test Torch at all. It captures the artifact the Phase 4 packer has to reproduce, from a pipeline that is about to be deleted. Do this while ZAPD still builds.
Current state: soh.o2r at the repo root, 1,042 entries = 1,041 files under
soh/assets/custom/ + portVersion. No version file — matching
sohArchiveVersionMatch (OTRGlobals.cpp:282-283), which only reads portVersion.
-
Rebuild rather than trust the file on disk, so provenance is unambiguous:
cmake --build build --target GenerateSohOtr # inside the soh distroboxGenerateSohOtrwrites into the source tree: itrm -rfs and re-copiessoh/assets/custom/shaders/fromlibultraship/src/fast/shaders/(3 files —directx/default.shader.hlsl,metal/default.shader.metal,opengl/default.shader.glsl). That directory is gitignored (soh/assets/.gitignore:8), which is whygit statusstays clean and why the on-disk count (1,041) exceedsgit ls-files(1,038). Phase 4's packer must run after the same copy or the archives differ by exactly those 3 entries. -
Record the output —
manifest.shalready does exactly this job:./manifest.sh /var/home/briaguya/code/Shipwright/soh.o2r manifests/soh_o2r.jsonLives in the harness alongside the other 14 manifests. (Not in the yml repo — that holds yml and nothing else.)
-
Record the inputs too, or a future mismatch is ambiguous between "the packer is wrong" and "the assets changed":
- sha256 of every file under
soh/assets/custom/→manifests/soh_custom_inputs.json CMAKE_PROJECT_VERSION= 9.2.3 (CMakeLists.txt:9), theportVersionpayload- the
developSHA the archive was built from
- sha256 of every file under
-
Call out the 31 files that matter. Only these hit the conversion path — everything else is a byte copy, and a byte copy that fails is a trivial bug:
Format Count .rgba32.png15 .ia8.png8 .ia16.png5 .ia4.png3 The other 1,010: 923 objects, 38 plain
.png, 23.rgba16, 22 json, 4 ttf, 3 shaders, 2 xml. The trap is quantisation, not decoding —n64graphics'SCALE_8_4(x)(x*15/255) vs ZAPD's(x >> 4) & 0xFdisagree (0xF0→0xEvs0xF). Phase 4 hand-ports ZAPD's arithmetic; this manifest is what catches it if the port is wrong.
Exit criteria
| Gate | Pass condition | Status |
|---|---|---|
| Preflight | 19/19 identical with the existing baseline binary; one manifest re-derivation matches | ✅ green — see below |
| D | pin SHA recorded; git diff vs upstream main still empty |
✅ green (trees equal) |
| A | 19/19 identical; phases= matches the per-version yml count |
✅ green — see below |
| A2 | both archives in a two-extraction process identical — especially the second | ✅ green — see below |
| B | 19/19 identical | ⬜ |
| C | 19/19 identical | ⬜ |
| A′ | 19/19 identical, plus the pair run | ⬜ |
| E | manifests/soh_o2r.json (1,042 entries) + input hashes + version + SHA committed |
⬜ |
"Identical" means test_assets.py reports 0 failed, 0 not generated, 0 not in reference —
not "close enough". --failures-only keeps the logs to the point; drop it when bisecting.
Write the results up as a second table in the harness README, next to the existing 14/14 one. That table is the evidence the Shipwright PR points at.
Preflight result — 2026-07-24
TORCH_BIN=torch/build/torch tools/matrix.sh baseline, harness phase0-gates off
bump-torch-9422bf4. 19 passed, 0 failed, of 19 ROM dumps, every one
0 failed, 0 not generated, 0 not in reference.
| Version | Dumps | Assets |
|---|---|---|
ntsc_1-0 |
ntsc_j_1-0_c892bb, ntsc_u_1-0_ad69c9 |
38,390 |
ntsc_1-1 |
ntsc_j_1-1_dbfc81, ntsc_u_1-1_d3ecb2 |
38,526 |
ntsc_1-2 |
ntsc_j_1-2_fa5f59, ntsc_u_1-2_41b3bd |
39,032 |
ntsc_j_gc |
ntsc_j_gc_0769c8 |
39,064 |
ntsc_j_gc_collection |
ntsc_j_gc_collection_2ce2d1 |
39,066 |
ntsc_j_mq |
ntsc_j_mq_dd14e1 |
39,026 |
ntsc_u_gc |
ntsc_u_gc_b82710 |
39,066 |
ntsc_u_mq |
ntsc_u_mq_8b5d13 |
39,029 |
pal_1-0 |
pal_1-0_328a1f |
35,362 |
pal_1-1 |
pal_1-1_cfbb98 |
35,362 |
pal_gc |
pal_gc_0227d7 |
35,386 |
pal_gc_dbg |
pal_gc_dbg_cee6bc |
35,646 |
pal_mq |
pal_mq_f46239 |
35,352 |
pal_mq_dbg |
pal_mq_dbg_079b85, _50bebe, _cfecfd |
35,612 |
Counts match the harness README's 14-target table exactly. The 19-dump matrix adds something
the 14-target one never showed: the duplicate dumps agree with each other — both 1.0 dumps produce
the same 38,390 assets against the same manifest, and all three pal_mq_dbg dumps produce the same
35,612. Since PLAN.md's MQ handling keys off IsMasterQuest() rather than the dump, that's worth
having on record.
Reference archives spot-checked as well — manifest.sh re-derivation of
o2r/ntsc_u_gc_b82710.o2r (39,066 entries) and o2r/pal_mq_dbg_079b85.o2r (35,612) reproduces the
committed manifests byte-for-byte. The baseline the gates measure against hasn't drifted.
Gate A result — 2026-07-24
TORCH_BIN=torchlib-driver/build/torchlib-driver tools/matrix.sh static-lib, harness
phase0-gates @ 504f086. 19 passed, 0 failed, of 19 ROM dumps — identical counts to the
preflight table above, every one 0 failed, 0 not generated, 0 not in reference.
USE_STANDALONE=OFF changes nothing about the bytes, and driving Companion by hand produces
exactly what the CLI does. Confirmed rather than assumed:
- libgfxd really is absent.
torchlib-driver/build/_deps/contains onlytinyxml2,yaml-cppandzlib— nolibgfxd-src, because theFetchContentsits insideif(USE_STANDALONE). The Binary export path is genuinely gfxd-free; all 19 archives extract identically without it. - The progress denominator is exact.
phases=matchedfind assets/yml/<version> -name '*.yml' | wc -lfor all three distinct counts:pal_gc1450/1450,ntsc_u_gc1449/1449,pal_gc_dbg1480/1480. So Phase 3's bar is monotonic 0→100 with a denominator counted off disk, andOTRGlobals.cpp:750-755needs no change. - Archive size matches the CLI build's (33,154,599 bytes for
pal_gc). It's ~343 KB larger than the OTRExporter reference: 76 KB of that is Torch's duplicate archive entries (see Gate A2 below) and the remaining ~267 KB is zip-compression difference, not content. Which is why comparison is always file-by-file inside the archive. - The
Companion::Instancehand-assignment,SetVersion,SetPhaseCallback,Init(Binary)sequence works from outside the library, soTorchExtract.cppcan be a transcription ofRunOnce().
Also learned, for Phase 2: zlib is fetched under USE_STANDALONE=OFF with
BUILD_STORMLIB=OFF — _deps/zlib-src exists despite zero zlib.h includes in torch/src,
because that FetchContent_Declare(... OVERRIDE_FIND_PACKAGE) is unconditional. Risk #1 is
observed, not hypothetical. Conversely spdlog was not fetched (find_package(spdlog QUIET) found
the distrobox's), so the second-copy ABI concern doesn't apply — only the global-logger stomping.
Gate A2 result — 2026-07-24 — ✅ green
TORCH_BIN=torchlib-driver/build/torchlib-driver tools/matrix.sh --pair pal_gc_0227d7 pal_mq_f46239 reentrancy. One process, vanilla then MQ, a fresh Companion and destdir each:
| assets | missing | extra | mismatched | |
|---|---|---|---|---|
pal_gc (first) |
35,386 / 35,386 | 0 | 0 | 0 |
pal_mq (second) |
35,352 / 35,352 | 0 | 0 | 0 |
The load-bearing number is the second run's phases=1450, exactly pal_mq's on-disk yml
count. Had gProcessedFiles leaked across runs the second pass would have skipped files it
believed already processed — a collapsed phase count and a truncated archive. It processed all
1,450 and reproduced the reference exactly.
A fresh Companion per extraction is sufficient. AliasManager::Instance not being cleared,
AudioManager::Instance leaking, and the file-scope statics in TextureFactory.cpp:14-15,
CompressedTextureFactory.cpp:17-18, DisplayListFactory.cpp:82 are all confirmed inert for
OoT/Binary. PLAN.md risk #6 is retired offline; the in-game two-ROM session (§9 of
the verification matrix) is now a confirmation, not a discovery.
Found along the way: Torch writes duplicate archive entries
The first attempt failed both comparisons — not on parity, but because check.sh never got to
compare. unzip had aborted mid-extract.
Torch writes some assets more than once under the same archive path. pal_mq: 35,377 physical
zip entries for 35,352 unique names; 24 names duplicated (23 cutscenes + link_home_room_0Back ground_002480, which appears three times) = 25 redundant entries. pal_gc: also 24. Every copy is
byte-identical to its siblings. All 19 OTRExporter reference archives have zero. Identical
under USE_STANDALONE ON and OFF, so it is not a static-lib artifact — pre-existing Torch
behavior, and Gate A's conclusions are unaffected.
Zip permits this: the format is a flat sequence of member records plus a central directory, with
the path as an ordinary string field and no uniqueness constraint. Only a reader materializing to
a real filesystem has to reconcile them — hence unzip prompting replace ...?, reading EOF from
a non-interactive stdin, and exiting non-zero into set -e.
Benign for SoH. O2rArchive::LoadFile resolves via zip_name_locate, which returns one entry
index, and the copies are byte-identical. O2rArchive::Open is the only place LUS walks entries by
raw index, and it only feeds Archive::IndexFile, which does (*mHashes)[CRC64(path)] = path — a
repeat insert is an idempotent overwrite. ListFiles/HasFile derive from that same map. So the
duplicates collapse at the archive's front door: no double-loading, no double-counting, no
duplicate list entries. Cost is 76 KB compressed on a 33 MB archive.
Filed as item 5 of HarbourMasters/Torch#233 for post-migration cleanup. Not a blocker and not something the migration needs to fix.
The lasting lesson is about the harness, not Torch: a comparison keyed on archive path — which is
both test_assets.py and check.sh — structurally cannot see duplicate entries, because they
collapse into one key. Asset parity genuinely holds (identical unique-path sets, identical bytes at
every path), but "19/19 identical" was never a statement about the zip container. check.sh now
extracts with unzip -o and prints a duplicate-name count for both archives so this can't go quiet
again.
Cost
| Build | Matrix | |
|---|---|---|
| Preflight | — | ~7 min |
| A | full Torch rebuild | ~7 min |
| A2 | — (reuses A) | ~1 min |
| B | full Torch rebuild | ~7 min |
| C | full Torch rebuild | ~7 min |
| A′ | full Torch rebuild | ~8 min |
| E | Shipwright GenerateSohOtr |
~1 min |
An evening, dominated by four full Torch builds. Four extra build trees; the Debug Torch binary
alone is 125 MB, so either build them sequentially and delete as you go or budget a few GB.
What Phase 0 deliberately does not cover
Being explicit so these don't get mistaken for retired risk:
- Linking Torch into Shipwright. The zlib / stb /
StringHelper/StrHash64/ tinyxml2 /stormcollisions inPLAN.mdPhase 2 need SoH's own build to surface. A standalone driver links Torch against nothing else, so it can't see any of them. That's Phase 2's first day. - spdlog global-state stomping. Torch calls
spdlog::set_level/set_patternon the default logger (Companion.cpp:193-194, 1810-1814), and ourconfig.ymlsetslogging: CRITICAL— so this fires on every run, it just doesn't matter until there's a SoH logger to clobber. - Windows and macOS. Every gate runs Linux/gcc in the distrobox. Torch forces
/MTunder MSVC, adds-DSTORMLIB_NO_AUTO_LINKand/bigobj, and carries acmake_minimum_required(3.12)+CMAKE_POLICY_VERSION_MINIMUMshim under a CMake 4 host. If a Windows machine is available, running Gate A′ there is the cheapest possible insurance on the one platform that can't be predicted from reading — but it isn't a blocker for starting Phase 2. - Anything in-game. The driver proves the bytes; it says nothing about the ImGui progress bar,
the ROM picker, or
ValidHashes. That's the Phase 3 manual matrix.