mirror of
https://github.com/HarbourMasters/Shipwright
synced 2026-08-12 20:22:37 -04:00
Add Torch migration plan
Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset pipeline, following HarbourMasters/Torch#219 ("oot support"). Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
briaguya
@@ -0,0 +1,458 @@
|
||||
# Migrate Shipwright from ZAPDTR/OTRExporter to Torch
|
||||
|
||||
## Context
|
||||
|
||||
SoH extracts OoT assets from the user's ROM with **ZAPDTR** (a ZAPD fork) driven by
|
||||
**OTRExporter**, described by 7,680 ZAPD XMLs under `soh/assets/xml/`. Both are git
|
||||
submodules, and ZAPD is additionally linked *into the game binary* so users can drag a
|
||||
ROM onto SoH and get an `oot.o2r` with a progress bar.
|
||||
|
||||
[HarbourMasters/Torch#219](https://github.com/HarbourMasters/Torch/pull/219) ("oot support",
|
||||
merged 2026-07-24) landed the OoT factories in Torch. `briaguya0/zapd-to-torch-test-harness`
|
||||
proves the swap is safe: **14 of 14 ROM targets extract byte-for-byte identical** to an
|
||||
OTRExporter reference O2R from Shipwright `95d8f7e` (35,386–39,066 assets per ROM,
|
||||
0 failed / 0 missing / 0 extra).
|
||||
|
||||
**Goal:** delete ZAPDTR and OTRExporter entirely; Torch becomes the only asset pipeline.
|
||||
**Shape:** one big PR — `develop` is never half-migrated.
|
||||
|
||||
Verified compatibility that makes this viable: Torch's 0x40-byte resource header
|
||||
(`src/factories/BaseFactory.cpp:3`) already matches `libultraship/src/ship/resource/ResourceLoader.cpp:246-259`,
|
||||
and Torch's OoT FourCC `ResourceType`s are byte-identical to `soh/soh/resource/type/SohResourceType.h`.
|
||||
|
||||
---
|
||||
|
||||
## Decisions
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| ZAPDTR / OTRExporter | Full deletion |
|
||||
| Torch consumption | `FetchContent`, pinned to `HarbourMasters/Torch` **`4cae44160693e1beb562e39dc301bd42278be1f9`** |
|
||||
| Asset YAML home | Shipwright, but a **separate repo consumed as a submodule** for PR reviewability; flatten into the tree post-merge |
|
||||
| In-game extraction | Keep it — link Torch as a **static lib** (`USE_STANDALONE=OFF`) and drive `Companion` directly |
|
||||
| `soh.o2r` | Small **in-tree C++ packer** (no Python — Windows devs). Long term: Torch PR adding PNG support to `Companion::Pack` |
|
||||
| `ExtractAssetHeaders` | **Dropped.** Freeze the 1,085 checked-in `.h` files; follow-up Torch PR (see Blocker below) |
|
||||
| Release payload | Ship the raw 111 MB YAML tree; zip it (~15 MB) in a follow-up |
|
||||
| Dep collisions | Try `FetchContent` as-is on Linux + Windows first, fix only what actually breaks |
|
||||
|
||||
**Pin note:** `git diff --stat 9422bf4 upstream/main` = 0 lines; the harness-pinned fork
|
||||
commit and `HarbourMasters/Torch` main have identical trees (`5d247f78…`). Nothing to upstream
|
||||
for parity. Re-check `git diff 4cae4416 <pin>` right before opening the PR in case main moves.
|
||||
|
||||
---
|
||||
|
||||
## Blocker resolved up front: headers
|
||||
|
||||
`torch header` **cannot** replace `ExtractAssetHeaders`, for two independent reasons:
|
||||
|
||||
1. **No OoT factory registers a Header exporter.** Every `REGISTER(` in `src/factories/oot/*.h`
|
||||
is `REGISTER(Binary, ...)`. `Companion::ParseNode` skips assets whose factory lacks an
|
||||
exporter for the active `ExportType` (`Companion.cpp:882-884`), so ~10k assets — skeletons,
|
||||
limbs, animations, collision, arrays, paths, cutscenes, text, scenes, rooms — would be
|
||||
silently omitted. `gLinkAdultSkel` simply would not appear.
|
||||
2. **No `#define d<sym>` line.** Torch emits only
|
||||
`static const ALIGN_ASSET(2) char sym[] = "__OTR__path";`. SoH's headers emit a pair, and
|
||||
soh source references the `d`-prefixed macros **24,698 times**. Also `#pragma once` vs
|
||||
`#ifndef OBJECTS_..._H`, no `#include "align_asset_macro.h"`, and a flat
|
||||
`headers/objects/foo.h` layout vs SoH's `objects/foo/foo.h`.
|
||||
|
||||
The load-bearing part — the `__OTR__objects/…/gSym` path string — already matches byte-for-byte
|
||||
(that's what the harness proves). What's missing is presentation plus ~14 mechanical Header
|
||||
exporters. **This PR deletes the target and keeps the `.h` files as ordinary checked-in source**
|
||||
(they already are, and contain no ZAPD-derived includes beyond `align_asset_macro.h`).
|
||||
`soh/assets/soh_assets.h` is hand-written and was never generated. Document loudly in the PR
|
||||
that header regeneration is temporarily unavailable; file the Torch PR in parallel.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Verification gates (no Shipwright changes)
|
||||
|
||||
All run in `zapd-to-torch-test-harness` against the committed `manifests/*.json`.
|
||||
|
||||
**Gate A — `USE_STANDALONE=OFF` is byte-identical.**
|
||||
The static-lib build has no CLI, so add a throwaway driver *in the harness*
|
||||
(`torchlib-driver/`: `add_subdirectory(../torch)` with `USE_STANDALONE=OFF`, ~40-line `main.cpp`
|
||||
constructing `Companion` and calling `Init(ExportType::Binary)`). Build both ways from the same
|
||||
tree with `-DPORT_VERSION_ENDIANNESS=ON`, run all 14 versions, compare via `check.sh`.
|
||||
Expected identical: the only reachable `#ifdef STANDALONE` difference is `enums:` parsing
|
||||
(`Companion.cpp:1524`) and the Code exporter — and `assets/yml/config.yml` has no `enums:` key.
|
||||
Smoke-test `pal_gc` first.
|
||||
|
||||
**Gate B — OoT-only build is byte-identical.**
|
||||
`-DBUILD_{SM64,MK64,SF64,PM64,FZERO,BK64,MARIO_ARTIST,NAUDIO}=OFF`. Run the 14-target matrix
|
||||
(`ROM_VERSION=<v> python3 tools/test_assets.py roms/<rom>.z64`). The 14/14 parity was measured
|
||||
with defaults ON, so this is a genuine change. Two real effects: `BUILD_BK64=OFF` drops the
|
||||
`BK64::TrySynthesizeRomConfig` fallback (irrelevant — all 17 hashes are in `config.yml`), and
|
||||
`BUILD_NAUDIO=OFF` removes Torch's only tinyxml2 consumers.
|
||||
|
||||
**Gate A′ — the shipping configuration.** Re-run Gate A *with* the Gate B flags. This exact
|
||||
combination is what ships.
|
||||
|
||||
**Gate E — `soh.o2r` baseline.** From current `develop`, build `GenerateSohOtr` and record a
|
||||
sha256 manifest of all 1,041 entries. This is the acceptance test for the new packer. All 31
|
||||
convertible PNGs are 8-bit / colour-type 6, so decode risk is low; the risk is quantisation
|
||||
(see Phase 4).
|
||||
|
||||
**Gate D — upstream pin.** Already satisfied (trees equal). Just record the SHA.
|
||||
|
||||
**Nothing in Shipwright changes until A, B, A′, E are green.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — The asset YAML repo
|
||||
|
||||
New repo (e.g. `HarbourMasters/soh-assets`), submoduled at **`assets/`** (top level, sibling of
|
||||
`soh/`) so the eventual flatten is one `git rm` + `git add` of a single directory.
|
||||
|
||||
```
|
||||
config.yml 17 ROM SHA1 → 14 version dirs (gbi: F3DEX2_OoT, sort: OFFSET,
|
||||
primary_virtual_segment: 0x80, strict_declarations: true)
|
||||
<14 version dirs>/ 20,353 .yml, 111 MB
|
||||
xml/ SNAPSHOT of soh/assets/xml before deletion ← most important item here
|
||||
filelists/ symbols/ moved from soh/assets/extractor (generator inputs)
|
||||
dma/ supplemental/ 17 JSON each (generator inputs)
|
||||
tools/ zapd_to_torch.py, extract_dma.py, generate_supplemental.py
|
||||
manifests/ 14 sha256 manifests from the OTRExporter reference
|
||||
README.md regeneration recipe + which SoH commit the reference came from
|
||||
```
|
||||
|
||||
Python lives **here**, not in Shipwright — SoH devs never need it; regeneration is a maintainer
|
||||
task. Regeneration: `extract_dma.py` → `generate_supplemental.py` → `zapd_to_torch.py` →
|
||||
`manifest.sh` → re-run the 14-target check.
|
||||
|
||||
**Snapshotting `soh/assets/xml/` into this repo before deleting it is non-negotiable** — it is
|
||||
the only input from which the YAML can be regenerated.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — CMake
|
||||
|
||||
### Root `CMakeLists.txt`
|
||||
|
||||
Add **before** `add_subdirectory(libultraship)` (~L196) — ordering matters, Torch declares
|
||||
`tinyxml2` and `zlib` with `OVERRIDE_FIND_PACKAGE`:
|
||||
|
||||
```cmake
|
||||
include(FetchContent)
|
||||
set(USE_STANDALONE OFF CACHE BOOL "" FORCE) # static lib, linkable into soh
|
||||
set(PORT_VERSION_ENDIANNESS ON CACHE BOOL "" FORCE) # 7-byte portVersion, OTRExporter parity
|
||||
set(ROM_CRC_BSWAP OFF CACHE BOOL "" FORCE)
|
||||
set(BUILD_UI OFF CACHE BOOL "" FORCE) # would FetchContent a 2nd libultraship
|
||||
set(BUILD_STORMLIB OFF CACHE BOOL "" FORCE) # MANDATORY: target-name clash with LUS's `storm`
|
||||
set(BUILD_OOT ON CACHE BOOL "" FORCE)
|
||||
foreach(g SM64 MK64 SF64 PM64 FZERO BK64 MARIO_ARTIST NAUDIO)
|
||||
set(BUILD_${g} OFF CACHE BOOL "" FORCE)
|
||||
endforeach()
|
||||
FetchContent_Declare(torch
|
||||
GIT_REPOSITORY https://github.com/HarbourMasters/Torch.git
|
||||
GIT_TAG 4cae44160693e1beb562e39dc301bd42278be1f9)
|
||||
FetchContent_MakeAvailable(torch)
|
||||
```
|
||||
|
||||
Remove: `add_subdirectory(ZAPDTR/ZAPD …)` (L199), `add_subdirectory(OTRExporter)` (L200), both
|
||||
`install(TARGETS ZAPD …)` (L209, L292), the `soh/assets/extractor` + `soh/assets/xml` install
|
||||
rules (L210-211, 215-216, 296-297), `find_package(Python3 …)` (L219), and the three custom
|
||||
targets (L222-261). Fix the stale L178 comment ("to use libgfxd from ZAPDTR" — LUS FetchContents
|
||||
libgfxd itself, so the Gfx debugger is unaffected).
|
||||
|
||||
### Two new in-tree tools
|
||||
|
||||
Torch can't be added twice, and the static-lib build has no `main` (`int main` is inside
|
||||
`#if defined(STANDALONE)`, while `Companion* Companion::Instance;` at `src/main.cpp:15` is
|
||||
deliberately *outside* it — so the lib gets the singleton storage and no entry point).
|
||||
|
||||
- **`tools/torch-cli/`** → target `soh-torch`. ~40 lines: parse `--src/--dest/--version/--rom`,
|
||||
construct `Companion`, `SetVersion`, `Init(ExportType::Binary)`. Links `torch`. One Torch build
|
||||
serves both the game link and the build-time targets; CI never needs the upstream CLI.
|
||||
- **`tools/soh-o2r-packer/`** → see Phase 4.
|
||||
|
||||
### Replacement targets
|
||||
|
||||
```cmake
|
||||
add_custom_target(ExtractAssets
|
||||
COMMAND ${CMAKE_COMMAND} -E rm -f oot.o2r oot-mq.o2r
|
||||
COMMAND $<TARGET_FILE:soh-torch> --src ${CMAKE_SOURCE_DIR}/assets
|
||||
--dest ${CMAKE_BINARY_DIR}/soh --version ${CMAKE_PROJECT_VERSION} --rom ${SOH_ROM_PATH}
|
||||
DEPENDS soh-torch)
|
||||
|
||||
add_custom_target(GenerateSohOtr
|
||||
COMMAND ${CMAKE_COMMAND} -E rm -r -f ${CMAKE_SOURCE_DIR}/soh/assets/custom/shaders/
|
||||
COMMAND ${CMAKE_COMMAND} -E copy_directory
|
||||
${CMAKE_SOURCE_DIR}/libultraship/src/fast/shaders/ ${CMAKE_SOURCE_DIR}/soh/assets/custom/shaders/
|
||||
COMMAND $<TARGET_FILE:soh-o2r-packer> ${CMAKE_SOURCE_DIR}/soh/assets/custom
|
||||
${CMAKE_BINARY_DIR}/soh/soh.o2r ${CMAKE_PROJECT_VERSION}
|
||||
DEPENDS soh-o2r-packer)
|
||||
# ExtractAssetHeaders: DELETED
|
||||
```
|
||||
|
||||
The LUS-shader copy must be preserved verbatim — it's why `soh/assets/custom/shaders/` exists and
|
||||
it currently lives inside both deleted targets.
|
||||
|
||||
### `soh/CMakeLists.txt`
|
||||
|
||||
- L107-109: drop the `ZAPDLib` `add_subdirectory` guard (Torch comes from the root).
|
||||
- L325: delete the dead `../ZAPDTR/ZAPD/resource/type` include — **that directory doesn't exist**.
|
||||
- L599-612: swap the `soh/assets/extractor` + `soh/assets/xml` POST_BUILD copies for a copy of
|
||||
`${CMAKE_SOURCE_DIR}/assets`; drop the ZAPD-only `assets/symbols` mkdir.
|
||||
- L624/642/662/704: `ZAPDLib` → `torch`, keeping the existing
|
||||
`NOT CMAKE_SYSTEM_NAME MATCHES "NintendoSwitch|CafeOS"` guard — **Switch and Wii U don't link
|
||||
the extractor today and must not link Torch either.**
|
||||
- Add `soh/soh/Extractor/TorchExtract.cpp`.
|
||||
|
||||
### Known collision landmines (fix only if they fire)
|
||||
|
||||
| Dep | Risk |
|
||||
|---|---|
|
||||
| **zlib** | Torch `FetchContent_Declare(zlib … OVERRIDE_FIND_PACKAGE)` unconditionally and links `zlibstatic` — but `grep zlib.h torch/src` = **0 hits**; it's only reachable via StormLib, which we disable. Redirects every later `find_package(ZLIB)` (libpng, libzip, SDL) and defines `zlibstatic` but not `ZLIB::ZLIB`. **Highest risk.** Best fix is a one-line Torch PR gating the fetch on `BUILD_STORMLIB OR BUILD_UI`. |
|
||||
| **stb** | `lib/n64graphics/CMakeLists.txt:13-31` renames `stbi_write_*` → `n64g_stbi_write_*` **only `if(BUILD_UI)`**. `stbi_load*` is never renamed. LUS links its own `stb`. |
|
||||
| **StringHelper / StrHash64** | Torch excludes its copies **only `if(BUILD_UI)`** (`CMakeLists.txt:146-152`), citing exactly this clash. SoH has the same files in `libultraship/include/ship/utils/`. |
|
||||
| **tinyxml2** | `OVERRIDE_FIND_PACKAGE` hijacks LUS's `find_package(tinyxml2 REQUIRED)`. Torch fetches 10.0.0 — the same version `.github/actions/install-tinyxml2` installs, so declaring Torch first is likely *beneficial* (one copy instead of two). All Torch tinyxml2 consumers are `naudio`, which we disable. |
|
||||
| **spdlog** | `find_package(spdlog QUIET)` runs first and SoH always has vcpkg spdlog, so no ABI clash. But Torch calls `spdlog::set_level`/`set_pattern` on the **global default logger** from the worker thread (`Companion.cpp:193-194, 1810-1814`), clobbering SoH's config at `OTRGlobals.cpp:808-815`. Save/restore around the call. |
|
||||
| **storm** | Hard CMake error if `BUILD_STORMLIB=ON` (LUS already creates a `storm` target, and `INCLUDE_MPQ_SUPPORT` is ON at root L186). Avoided by keeping it OFF; `.o2r` uses vendored miniz anyway. |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — In-game extractor rewrite
|
||||
|
||||
`Extractor::CallZapd` (`soh/soh/Extractor/Extract.cpp:641`) currently chdirs into a temp dir with
|
||||
`assets/` symlinked and hand-builds 22 argv entries for
|
||||
`extern "C" zapd_report(argc, argv, extractCount, totalExtract)`. Everything that needed the chdir
|
||||
becomes constructor arguments — `grep current_path torch/src` hits only `main.cpp` (STANDALONE) and
|
||||
`LusBackend.cpp` (BUILD_UI), so **Torch has no CWD dependence** when both `srcdir` and `destdir`
|
||||
are passed.
|
||||
|
||||
**Isolate Torch in its own TU.** `Companion.h` pulls in yaml-cpp, `n64/Cartridge.h`, a bare
|
||||
`CONTAINS` macro, and the `torch` target exports Torch's `src/` **and** `lib/` (CLI11, TinySHA1,
|
||||
`hj/`, `nposix/`) as PUBLIC includes. Don't mix that into `Extract.cpp` alongside game headers.
|
||||
|
||||
- `soh/soh/Extractor/TorchExtract.h` — declarations only, no Torch types:
|
||||
`SohTorch::CountAssetFiles(ymlDir)` and
|
||||
`SohTorch::Extract(rom, srcDir, destDir, portVersion, std::atomic<size_t>* progress)`.
|
||||
- `soh/soh/Extractor/TorchExtract.cpp` — the only file that includes `Companion.h`.
|
||||
|
||||
**Flow:** `srcDir = <installPath>/assets`; `destDir = Mkdtemp()` (reuse `Extract.cpp:619`);
|
||||
`totalExtract` = recursive count of `*.yml` under `srcDir/<versionDir>` minus `config.yml`;
|
||||
run; then `rename(destDir/oot.o2r → exportdir/(IsMasterQuest() ? "oot-mq.o2r" : "oot.o2r"))`;
|
||||
`remove_all(destDir)`.
|
||||
|
||||
**Driving Companion:**
|
||||
```cpp
|
||||
auto* c = new Companion(fs::path(rom), ArchiveType::O2R, /*debug*/false, srcDir, destDir);
|
||||
Companion::Instance = c; // bare singleton, no getter — factories deref it unconditionally
|
||||
c->SetVersion(portVersion);
|
||||
c->SetPhaseCallback([progress](int){ if (progress) ++*progress; });
|
||||
c->Init(ExportType::Binary); // Init IS the whole run; it calls Process() internally
|
||||
```
|
||||
|
||||
- **Must `try/catch`.** 127 `throw std::runtime_error` across `torch/src`; nothing catches at the
|
||||
top level, so an escape means `std::terminate`. The existing `extractionTask->get()` try/catch
|
||||
at `OTRGlobals.cpp:730-734` turns a rethrow into the "Extraction Crashed" popup.
|
||||
- **Must also stat the output.** `Process()` returns `void` and several fatal conditions just log
|
||||
and `return` — no `config.yml` (`Companion.cpp:1304`), no config entry for the ROM hash (`1334`),
|
||||
no `config:` node (`1390`), bad GBI (`1484`). A ROM dump outside the 17-hash table produces no
|
||||
archive and no exception.
|
||||
- **Fresh `Companion` per call.** `gProcessedFiles` (`Companion.h:300`) is never cleared — a second
|
||||
`Process()` on the same instance skips *every* file. `gAddrMap`/`gParseResults`/`gWriteMap` also
|
||||
accumulate. `RunExtract` can extract more than once per process (`PS_FIRST`→`PS_SECOND`,
|
||||
`ES_EXTRACT_ARGS` loops over CLI args), so allocate anew and reassign `Companion::Instance`.
|
||||
Watch: `AliasManager::Instance` is not cleared at end of `Process` (inert for OoT — no OoT factory
|
||||
calls `Register` — but unproven), and `AudioManager::Instance` is leaked per run (`Companion.cpp:1584`).
|
||||
|
||||
**Progress: use `SetPhaseCallback`, not `Init`'s atomic overload.** `ProcessFile`
|
||||
(`Companion.cpp:1282-1293`) resets `*gAssetCounter = 0` and recomputes the total **once per YAML
|
||||
file**, so with ~1,450 YAMLs a naive bar restarts 1,450 times. The phase callback has exactly one
|
||||
call site (`gPhaseCallback(2)` at `Companion.cpp:1282-1283`), fired once per YAML at the
|
||||
parse→export transition, after `NodeHasChanges` has already let the file through — a 1:1 match for
|
||||
a denominator computed by counting files on disk. Sub-files from `AddSubFileAsset` are virtual names
|
||||
never present on disk, so the mapping is exact. `NodeHasChanges` early-returns are impossible
|
||||
because a fresh temp destdir has no `torch.hash.yml`. **Nothing in the ImGui progress bar
|
||||
(`OTRGlobals.cpp:750-755`) needs to change.** `GetCurrentAssetName()` is available for a label.
|
||||
|
||||
**The temp destdir is mandatory, not cosmetic.** `Process` unconditionally writes
|
||||
`destdir/torch.hash.yml` (`Companion.cpp:1804`) and reads it back next run (`:708`). Extracting
|
||||
straight into the app dir would litter it *and* make a second in-session extraction skip files it
|
||||
thinks are unchanged.
|
||||
|
||||
**MQ naming:** leave `config.yml` at `binary: oot.o2r` for all 17 hashes and rename on the way out
|
||||
of the temp dir based on `IsMasterQuest()`. Torch offers no output-filename override
|
||||
(`-d` sets only the directory; you'd get a *directory* named `oot-mq.o2r`), and this keeps SoH the
|
||||
single source of truth for MQ-ness and preserves the harness's `<out>/oot.o2r` expectation.
|
||||
|
||||
**Delete:** `CallZapd` (Extract.cpp:640-696) and its decl in `Extract.h:66-67`, the `zapd_report`
|
||||
extern, and `GetZapdVerStr()` (`:583`) — replaced by a `GetTorchVersionDir()` returning
|
||||
`pal_gc`/`ntsc_u_gc`/… over the same CRC mapping. `Mkdtemp()` stays.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — The `soh.o2r` packer
|
||||
|
||||
`tools/soh-o2r-packer/`, `target_link_libraries(soh-o2r-packer PRIVATE torch)`. Reusing `torch`
|
||||
gets `ZWrapper` (miniz — the same zip writer that produces `oot.o2r`, so archives stay structurally
|
||||
consistent), `Companion::ParseVersionString` for the 7-byte `portVersion`, and `stb_image` via the
|
||||
vendored `N64Graphics`. No new dependency, no Python, works on Windows out of the box.
|
||||
|
||||
Usage: `soh-o2r-packer <custom-assets-dir> <out.o2r> <M.m.p>`. Behaviour ported from
|
||||
`OTRExporter/OTRExporter/Main.cpp:206-260`:
|
||||
|
||||
1. **`<name>.<fmt>.png`** (`rgba32|rgb5a1|i4|i8|ia4|ia8|ia16|ci4|ci8`) → convert; archive path is
|
||||
the relative path with `.<fmt>.png` stripped. Payload = the 0x40 resource header
|
||||
(endianness `0x00`, resType `Texture`, version 0, id `0xDEADBEEF…`, zero-padded) then
|
||||
`u32 textureType, width, height, rawDataSize`, then the raw N64 bytes.
|
||||
**Reproduce ZAPD's asymmetry:** the size field is the unpadded `GetRawDataSize()` while the
|
||||
buffer written is `ALIGN8`-padded.
|
||||
2. **Path contains `accessibility`** → `.json` added with full relative path; non-`.json` **skipped**.
|
||||
All 12 files are `.json` today, so this is a no-op — replicate it anyway to avoid silent drift.
|
||||
3. **Everything else** → raw byte copy, full relative path.
|
||||
4. `portVersion` only. **No `version` file** — OTRExporter doesn't write one for `soh.o2r`, and
|
||||
`sohArchiveVersionMatch` (`OTRGlobals.cpp:282-283`) only reads `portVersion`.
|
||||
|
||||
**Critical: use ZAPD's quantisation, not n64graphics'.** `n64graphics`' `ia2raw` uses
|
||||
`SCALE_8_4(x)` (`x*15/255`) where ZAPD uses `(x >> 4) & 0xF` — for `r = 0xF0` those give `0xE`
|
||||
vs `0xF`, **different bytes**. Same for IA4. Decode with `stbi_load(path, &w, &h, &n, 4)`, then
|
||||
hand-port ~80 lines from `ZAPDTR/ZAPD/ZTexture.cpp`: rgba32 (`:507`), ia16 (`:595`, `[r, a]`),
|
||||
ia8 (`:578`, `((r>>4)&0xF)<<4 | ((a>>4)&0xF)`), ia4 (`:552`, `((r>>5)<<1) | (a != 0)`), plus
|
||||
i4/i8/rgba16/ci4/ci8 for completeness so adding a texture later doesn't silently produce garbage.
|
||||
|
||||
Only 31 of 1,041 files hit the conversion path (15 rgba32, 8 ia8, 5 ia16, 3 ia4); the rest — 923
|
||||
objects, 23 `.rgba16`, 22 json, 4 ttf, 2 xml, ~40 plain `.png` — are raw copies.
|
||||
|
||||
**Acceptance:** sha256 of every one of the 1,041 entries against the Gate E manifest.
|
||||
|
||||
**Long term:** the Torch PR is `Companion::Pack` gaining PNG conversion + a version file, after
|
||||
which this collapses to `torch pack soh/assets/custom soh.o2r o2r -u 9.2.3`. Structure the format
|
||||
table so it lifts into Torch wholesale.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Deletions
|
||||
|
||||
- **Submodules:** `ZAPDTR/` (1.9 MB), `OTRExporter/` (71 MB) — `.gitmodules` + `git rm`
|
||||
- **`soh/assets/xml/`** — 7,680 files, 54 MB (**snapshot into the assets repo first**)
|
||||
- `soh/assets/extractor/Config_*.xml` (14) and `TexturePool.xml` — ZAPD `-rconf` inputs, no runtime use
|
||||
- `soh/assets/extractor/filelists/`, `soh/assets/extractor/symbols/` — **move** to the assets repo
|
||||
(`extract_dma.py` needs filelists; symbols are referenced only from the deleted `Config_*.xml`,
|
||||
with no runtime reader in `soh/soh` or `soh/src`)
|
||||
- `copy-existing-otrs.cmake`
|
||||
- CMake and source hunks listed in Phases 2–3
|
||||
|
||||
**Keep:** all 1,085 `soh/assets/**/*.h`, all 1,041 files under `soh/assets/custom/`.
|
||||
|
||||
Net: −127 MB submodules, −54 MB XML, +111 MB YAML (submodule initially; ~7-15 MB packed).
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — CI
|
||||
|
||||
`.github/workflows/generate-builds.yml`:
|
||||
|
||||
- **`generate-soh-otr` (L10-68)** does a full SoH configure — SDL2 from source, SDL2_net,
|
||||
tinyxml2 from source, ccache, apt deps — purely to build `GenerateSohOtr`. The new packer has
|
||||
no SDL/LUS dependency, so add a `SOH_TOOLS_ONLY` root guard that skips
|
||||
`add_subdirectory(libultraship)`/`add_subdirectory(soh)` and configures only Torch +
|
||||
`soh-o2r-packer`. Saves ~5-8 min per run and drops the `install-tinyxml2` dependency. Add a step
|
||||
asserting the `soh.o2r` content manifest.
|
||||
- **Platform build jobs** (`build-macos` L71, `build-linux`, `build-windows`, Switch, Wii U):
|
||||
`submodules: true` already covers the new `assets/` submodule (it has no nested submodules).
|
||||
Switch/Wii U must not build Torch — same `NintendoSwitch|CafeOS` guard as `ZAPDLib` today.
|
||||
Package payload changes from `assets/xml/` to `assets/`.
|
||||
- **`test-builds-on-distros.yml` L137-150** removes distro tinyxml2 and builds 10.0.0 from source.
|
||||
**Leave it in this PR** — one variable at a time; delete in a follow-up once Torch's fetched
|
||||
tinyxml2 is confirmed to satisfy LUS.
|
||||
- `pr-artifacts.yml:51` already filters `soh.o2r`; no change.
|
||||
|
||||
Optional follow-up: a CI job running Gate A/B against a checked-in manifest so Torch pin bumps are
|
||||
validated automatically. Needs a ROM, so self-hosted or encrypted secret.
|
||||
|
||||
---
|
||||
|
||||
## Risks, ranked
|
||||
|
||||
1. **zlib double-link** — Torch links an unused zlib and `OVERRIDE_FIND_PACKAGE` hijacks the whole
|
||||
build's `find_package(ZLIB)`. Most likely thing to break the link, especially on Windows/vcpkg-static.
|
||||
2. **stb / StringHelper / StrHash64 duplicate symbols** — Torch's own guards for these are
|
||||
`if(BUILD_UI)`, which we don't set.
|
||||
3. **`soh.o2r` texture byte-identity** — libpng→stb_image plus the `SCALE_8_4` vs `>>4` trap.
|
||||
Fully testable offline (Gate E); de-risk first, costs an afternoon.
|
||||
4. **Windows build** — Torch forces `/MT`, adds `-DSTORMLIB_NO_AUTO_LINK`, `/bigobj`, and carries a
|
||||
`cmake_minimum_required(3.12)` + policy shim under a CMake 4 host. The one platform not
|
||||
predictable from reading. Build it early.
|
||||
5. **Extraction wall-clock** — Torch parses ~1,450 YAMLs per run and the parse phase is serial
|
||||
(export is explicitly serial for Binary too: *"Binary/code/header all share wrapper state"*,
|
||||
`Companion.cpp:1748`). Measure vs ZAPD; if it regresses, the UI needs a "this takes N minutes" note.
|
||||
6. **Re-entrancy across two in-session extractions** — `AliasManager` uncleaned plus file-scope
|
||||
statics in `TextureFactory.cpp:14-15`, `CompressedTextureFactory.cpp:17-18`,
|
||||
`DisplayListFactory.cpp:82`. Believed inert for OoT/Binary; only a real vanilla-then-MQ run proves it.
|
||||
7. **spdlog global-state stomping** from Torch's worker thread.
|
||||
8. **Header regeneration gap** until the follow-up Torch PR lands.
|
||||
9. **111 MB submodule ergonomics** — clone time, and a flatten commit GitHub will refuse to render.
|
||||
10. **`Companion::Instance` is a raw global** — `RunExtract` uses a 1-thread pool so it's serialised
|
||||
today; add a comment/assert.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
**Archive level**
|
||||
1. Gates A, B, A′, E (Phase 0) — 14/14, `0 failed / 0 not generated / 0 not in reference`.
|
||||
2. Post-integration: run Shipwright's `soh-torch` over all 14 ROMs and feed the outputs to the
|
||||
harness's `check.sh`/`test_assets.py`. This proves the *Shipwright-configured* Torch
|
||||
(FetchContent, OoT-only, static lib, SoH's flags) matches — not just the harness build.
|
||||
3. `soh.o2r`: full 1,041-file sha256 diff vs an OTRExporter archive from `develop` HEAD.
|
||||
4. Diff the in-game extractor's `oot.o2r` against `soh-torch`'s for the same ROM — different code
|
||||
paths (temp destdir, phase callback), must agree byte-for-byte.
|
||||
|
||||
**Build level**
|
||||
5. Clean configure + build on Linux (gcc, clang), macOS (universal), Windows (MSVC x64), Switch,
|
||||
Wii U. Switch/Wii U build with Torch excluded.
|
||||
6. Confirm exactly one tinyxml2 and one zlib in the final link (`nm`/`dumpbin` for duplicate
|
||||
`inflate`; `cmake --graphviz` for target duplication).
|
||||
7. Confirm `soh` links with the untouched checked-in `.h` files after ZAPDTR is gone.
|
||||
|
||||
**Run the game**
|
||||
8. Cold start with no archives: ROM picker → progress bar (must be **monotonic 0→100, not
|
||||
sawtooth**) → "Extraction Complete" → boots.
|
||||
9. **Two-ROM session** — extract vanilla, answer "Yes" to extract another, extract MQ in the same
|
||||
process. Highest-value manual test: covers re-entrancy and both output names.
|
||||
10. CLI path: `./soh <rom.z64>` → `ES_EXTRACT_ARGS`.
|
||||
11. Failure paths: non-OoT ROM and a truncated ROM → clean popup, not a hang or half-written `.o2r`.
|
||||
12. No litter: no `torch.hash.yml`, no temp dir surviving in the app directory.
|
||||
13. Gameplay: boot vanilla and MQ saves, load a dungeon (scene/room/collision), open the pause menu,
|
||||
check PNG-path textures (`gSun`, `gTriforcePiece`, `gDPad`), play a cutscene and a sequence,
|
||||
and confirm the `ValidHashes` check at `OTRGlobals.cpp:941` accepts the new archives.
|
||||
14. Build a CPack artifact per platform, unzip fresh, run from that directory, re-extract — catches
|
||||
broken install rules for the assets payload.
|
||||
|
||||
---
|
||||
|
||||
## Ordering
|
||||
|
||||
```
|
||||
0. Torch PR: gate the zlib fetch, make stb/StringHelper guards unconditional [only if they fire]
|
||||
1. Harness: Gates A, B, A′
|
||||
2. New assets repo: snapshot soh/assets/xml, publish YAML + generators
|
||||
3. Gate E: soh.o2r manifest from current develop
|
||||
--- single Shipwright PR from here ---
|
||||
4. Add assets submodule; FetchContent Torch before libultraship
|
||||
5. tools/torch-cli + tools/soh-o2r-packer
|
||||
6. Replace ExtractAssets / GenerateSohOtr; delete ExtractAssetHeaders
|
||||
7. TorchExtract.{h,cpp} + Extract.{h,cpp} rewrite
|
||||
8. Delete ZAPDTR, OTRExporter, soh/assets/xml, extractor inputs
|
||||
9. CI: generate-soh-otr slim-down, packaging payload
|
||||
10. Full verification matrix
|
||||
--- follow-ups ---
|
||||
11. Torch PR: OoT Header exporters + `#define d<sym>` → restore ExtractAssetHeaders
|
||||
12. Torch PR: PNG support in Companion::Pack → retire soh-o2r-packer
|
||||
13. Zip the shipped YAML (111 MB → ~15 MB)
|
||||
14. Flatten the assets submodule into the tree
|
||||
```
|
||||
|
||||
## Critical files
|
||||
|
||||
- `CMakeLists.txt` (178-179, 196-201, 208-216, 219-261, 292-297)
|
||||
- `soh/CMakeLists.txt` (107-109, 325, 599-612, 624/642/662/704)
|
||||
- `soh/soh/Extractor/Extract.cpp` (`CallZapd` 640-696, `GetZapdVerStr` 583, `Mkdtemp` 619), `Extract.h:66-67`
|
||||
- `soh/soh/OTRGlobals.cpp` (`RunExtract` 398, extraction task 596-680, progress bar 750-755, ValidHashes 941)
|
||||
- `OTRExporter/OTRExporter/Main.cpp:183-268` and `ZAPDTR/ZAPD/ZTexture.cpp:507-620` — the spec for Phase 4
|
||||
- Torch `src/Companion.{h,cpp}` (`Init` 185-395, `Process` 1299-1819, `Pack` 1821, `torch.hash.yml`
|
||||
708/1804, phase callback 1282-1293), `src/main.cpp:15`, `CMakeLists.txt` (21-37, 146-152, 228-235,
|
||||
402-428), `lib/n64graphics/CMakeLists.txt:13-31`
|
||||
Reference in New Issue
Block a user