From bf4482b4c8d976d994d3e5f4c311671cb78fd240 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sun, 5 Jul 2026 11:04:47 -0700 Subject: [PATCH] docs: spec on-the-fly string translation / localization framework Implementation-ready design for capturing PSX source strings from memory and substituting translated bytes on the fly, so JP-only titles (Tsumu Light SLPS-02253 first) can be played in English or any language. Adapts the author's proven n64recomp prior art (Pocket Monsters Stadium JP): always-on capture ring keyed by FNV-1a64 content hash, a per-title multilingual table, and a draw-time apply hook that repoints the string arg at guest scratch so the original renderer draws the replacement (no regen, unbounded length, dynamic args preserved). Grounds the PSX design in real findings: Tsumu text is Shift-JIS (2-byte fullwidth + 1-byte halfwidth kana) with 0xFFFF-framed message records; in-EXE text block ~0x80070900-0x80072000 via pointer table at 0x80071474; menu labels at 0x80010C5C/0x80010C94; font already has fullwidth Latin (English renders with zero glyph injection). Hooks onto the psx_dispatch chokepoint (fntrace_record census + g_psx_bios_hle_hook precedent), so no per-function codegen and works in Release. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/STRING_TRANSLATION.md | 581 +++++++++++++++++++++++++++++++++++++ 1 file changed, 581 insertions(+) create mode 100644 docs/STRING_TRANSLATION.md diff --git a/docs/STRING_TRANSLATION.md b/docs/STRING_TRANSLATION.md new file mode 100644 index 00000000..e6eaa360 --- /dev/null +++ b/docs/STRING_TRANSLATION.md @@ -0,0 +1,581 @@ +# On-the-Fly String Translation / Localization for psxrecomp + +**Status:** SPEC (design only — no implementation yet). Reusable framework +feature; **Tsumu Light (SLPS-02253)** is the first consumer. + +**Goal.** Let a Japanese-only PSX title be played in English (or any target +language) by **capturing the game's source strings from memory and substituting +translated bytes on the fly**, so a non-Japanese speaker can navigate menus, +read tutorials/dialogue, and understand the game. **Ultimate success = every +string we can find renders in the target language.** Multilingual is a +first-class requirement (English is just the first target); the mechanism is a +**shared psxrecomp module** any title can adopt. + +This design is a direct PSX adaptation of the author's proven N64 prior art in +the `n64recomp` fork for **Pocket Monsters Stadium (Japan)** +(`F:\Projects\n64recomp\PocketMonstersStadiumRecomp\`). Section 1 documents that +mechanism; Section 2 documents Tsumu's concrete text reality; Sections 3+ define +the PSX framework. + +--- + +## 0. TL;DR architecture + +Three always-on subsystems hung off the **`psx_dispatch` call chokepoint** (the +one place every guest function call funnels through — see §3.1): + +1. **Capture ring (always-on).** On every dispatch, scan the argument registers + (and a few stack slots) for pointers to Shift-JIS text records. Hash each + distinct record (FNV-1a64 over its raw bytes), record `{hash, bytes, addr, + target-PC, call-site}`, persist immediately. This *enumerates every string to + translate* and *discovers which PCs draw text* — no arm-then-capture (per the + project's ring-buffer rule). +2. **Translation table (per-title, human-editable, multilingual).** + `TsumuRecomp/translations/tsumu.toml` — keyed by source-byte hash (+ optional + `src_addr`), one target string per language, with encoding/length metadata. + Hot-reloaded on mtime change. +3. **Apply hook (always-on).** A dispatch hook (sibling to the existing + `g_psx_bios_hle_hook`) that, when the target is a registered text-draw PC and + an arg points at a translated source record, writes the target-language bytes + into transient guest scratch (below `$sp`), repoints the arg register, and + lets the **original** draw routine render it. No regen, no engine change, + unbounded replacement length, dynamic format strings preserved. + +The only title-specific pieces are (a) the translation table and (b) the font +situation (§4). Everything else — capture ring, apply hook, config, coverage +tooling — is shared framework code. + +--- + +## 1. Prior art: the n64recomp PMS-J mechanism + +Files: `PocketMonstersStadiumRecomp/src/main/diagnostics.cpp` (the engine), +`include/trace.h` (the hook wiring), `tools/pms_build_translations.py` (authoring), +`translations.json` (the table). Verified by reading the source. + +### 1.1 Capture — always-on string inventory + text-PC census + +The recompiler emits a bare `TRACE_ENTRY()` at the top of **every** recompiled +function when `game.toml` has `trace_mode = true`. `trace.h` expands it to call, +with `rdram`/`ctx` in scope: + +```c +pkmnstadium_textdraw_probe(__func__, ra, a0, a1, a2, a3); // census + inventory +pkmnstadium_text_xlate(__func__, rdram, ctx); // apply +``` + +`pkmnstadium_textdraw_probe` (diagnostics.cpp:1235) is the **always-on capture** +(explicitly *no* env gate — "we never 'arm' capture and hope to catch the +window, we record every unique on-screen string as it is first drawn and persist +it immediately"): + +- **`read_text_arg()`** (1152) validates that an arg register points at a + NUL-terminated EUC-JP/ASCII run ≥2 bytes: every byte must be newline, + printable ASCII, or a EUC-JP lead/kana byte (`0x8E/0x8F/0xA1–0xFE`). The + RDRAM-range check is the real discriminator — coords/ints/non-pointers are + rejected before the byte scan. +- **Class-1 convention gate:** most text draws use `x=a0, y=a1, str=a2`; the + `a0<0x1000 && a1<0x1000` "looks like screen coords" test rejects binary + headers (`Yay0`/`PRESJPEG`) that decode as valid EUC-JP by chance. +- **`_Printf` chokepoint** (`kPrintfPC`): the universal vararg formatter is + special-cased — its format string (a2) is inventoried regardless of the coord + gate, so battle/overlay text built via `_Printf` is captured too. +- **`strinv_upsert()`** records every *distinct* string (content-hash keyed) into + `stringdump.log`, flushed on every newly-seen string so a crash never loses + captures. +- **`textpc_add()`** maintains a lock-free open-addressing set of PCs proven to + draw text (`~70` distinct), so the apply hook only pays the arg-scan cost on + real text PCs. + +There is also a capture-only **discovery probe** (`xlate_discovery`, gated by +`PMS_TEXTPROBE`) that finds *which register/stack slot* a known-untranslated +watch-string arrives in — used to learn non-standard conventions (battle labels, +Pokédex descriptions in `r7`/`r4`, the `%sポケモン` category classifier in `r5`). + +### 1.2 Translation table format + +`translations.json` next to the exe, hot-reloaded on mtime (1550): + +```json +{ "src_hex": "a5d0a5c8a5eb", "src_jp": "バトル", "target": "Battle", "orig_w": 3 } +``` + +- **`src_hex`** — the raw source bytes (EUC-JP) in hex. The load path converts + hex→bytes and keys the in-memory map by **`FNV-1a64` of those bytes** (robust + to RDRAM buffer reuse — the key is content, not address). +- **`target`** — the replacement (ASCII; may contain `%s`/`%d` specifiers to + preserve dynamic args, and `\n` for line breaks). +- **`orig_w`** — Japanese footprint in glyph units (documentation + default fit + budget). Optional **`fit_w`** overrides the width budget (use more/less space + than the original when there is room). `src_jp` is documentation only. + +Authoring workflow (`pms_build_translations.py`): `stringdump.log` → decode +EUC-JP → `strings_decoded.tsv` (persistent master across sessions) → author +`key_en.tsv` (`\t`) → `build` joins them into `translations.json`, +**merge-preserving** hand-authored entries. A `todo` mode lists still-untranslated +strings sorted by draw count. + +### 1.3 Apply — on-the-fly substitution + +`pkmnstadium_text_xlate` (2100) runs at every function entry; cheap no-ops +(`g_xlate_active` reentrancy guard, lock-free `g_xlate_armed` reject, reload +every 1024 calls). On a text PC it takes one of three paths: + +- **Pointer-swap (`write_guest_str` + repoint `ctx->r6`).** Content-hash the + source at the fmt arg; on a KV hit, write the replacement (ASCII, NUL-term) + into **transient scratch on the calling thread's guest stack** (`sp-0x800` for + format templates, `sp-0xC00` for literal args — two disjoint slots so a paired + `_Printf` swap can't self-clobber) and repoint the arg register. The + **original function then runs unchanged** — `_Printf` formats it (dynamic + `%d`/`%s` stay intact) and renders glyph-by-glyph. No engine change, no regen, + unbounded replacement length (never writes back into game RAM). +- **Proportional self-render (`self_render_static`, 1887).** For plain strings at + coordinate drawers, it draws the English itself by calling the game's glyph + routine (`FUN_8001a3e4`) per character with **per-glyph proportional advances** + (calibrated by measuring real ink width of each glyph tile in VRAM, + `calibrate_slot`/`measure_glyph`) and **auto-fit** condensing so English never + spills past the Japanese footprint (`fit_w` budget or per-line JP width). Then + it suppresses the caller's own draw with an empty fmt. +- **Register-specific paths** (`xlate_desc`, `xlate_general`) for the + discovered non-`r6` conventions, including a subtle `_Printf` writeproc + byte-count re-sync when a shorter replacement would otherwise embed a NUL and + truncate a composed string. + +### 1.4 Multilingual & font + +- **Multilingual** is table-driven: the KV value is per-language. (PMS shipped + English only, but the schema and hook are language-agnostic — selecting a + language means loading a different column/file.) +- **Font/glyph:** PMS-J's font sheet **already contained Latin glyphs**, so + English renders directly through the existing glyphs (the whole "no glyph + injection needed" premise). `pkmnstadium_fontdump` (2227) dumps the font-state + struct + glyph sheets (`.i8`) to answer exactly this gating question; the + fallback contingency (documented in comments/tools) is to inject an ASCII + glyph set if a game lacks Latin glyphs. The renderer measures glyph tiles + directly from VRAM to get proportional widths. + +### 1.5 What carries to PSX vs. what changes + +| Aspect | N64 (PMS-J) | PSX (Tsumu) | +|---|---|---| +| Hook point | `TRACE_ENTRY()` emitted per function (needs `trace_mode=true` regen) | **`psx_dispatch` chokepoint** — one install site, no regen, works in Release | +| Encoding | EUC-JP (JIS X 0208 via `A1/A3/A4/A5` leads) | **Shift-JIS** (2-byte fullwidth + 1-byte halfwidth kana `A1–DF`) + custom control framing | +| String delimiter | NUL | **NUL (menu labels) AND `0xFFFF` (framed message records)** | +| Memory swizzle | big-endian `^3` byte swap | **little-endian, no swizzle** (simpler) | +| Arg access | `ctx->r4..r7` | `cpu->gpr[4..7]` | +| Glyph routine | `FUN_8001a3e4` (known) | text-draw PC pinned at runtime by the census (Phase 0) | +| Latin glyphs | present in font | fullwidth Latin present (proven); halfwidth ASCII TBD (§4) | + +--- + +## 2. Tsumu's concrete text reality + +Evidence gathered from the headerless boot EXE +(`TsumuRecomp/ghidra/SLPS_022.53_no_header.bin`, 512 KB, loads at `0x80010000`, +entry `0x800691F0`) via direct byte analysis + Ghidra, plus `DISC.md`. + +### 2.1 Encoding — Shift-JIS (confirmed) + +The text is **Shift-JIS**, not ASCII and not EUC-JP (EUC was the N64 game). +Proven by decoding real records: + +| Address | Decoded | Notes | +|---|---|---| +| `0x80010C5C` | `ＴＳＵＭＵ　ｌｉｇｈｔ　プレイデータ` | fullwidth Latin + fullwidth space + katakana ("TSUMU light / Play Data") | +| `0x80010C94` | `ＴＳＵＭＵ　マップデータ` | "TSUMU / Map Data" | +| `0x80010190` | `？？？？？？？？` | fullwidth question marks (placeholder score row) | +| `0x80070AA8` | `それじゃ、がんばってね！`… | tutorial dialogue (hiragana) | + +The character mix is: + +- **2-byte fullwidth** (JIS X 0208): kanji, hiragana, katakana, **fullwidth + Latin `0x8260–0x8279` (Ａ–Ｚ), `0x8281–0x829A` (ａ–ｚ), fullwidth digits + `0x824F–0x8258`**, symbols. Lead byte `0x81–0x9F` / `0xE0–0xFC`, trail + `0x40–0x7E` / `0x80–0xFC`. +- **1-byte halfwidth katakana** `0xA1–0xDF` (e.g. `0xBB`=ｻ) — mixed inline with + 2-byte chars. This means the font has a **halfwidth grid**. +- **Custom control framing** interleaved with text (raw bytes, not SJIS): + - `0xFF 0xFF` — **message terminator** (record end). + - `0x82 0xFE 0xFF` / `0x81 …` sequences — **line break / formatting** markers. + - Menu labels (e.g. `0x80010C5C`) are simpler: plain 2-byte SJIS, **NUL**-terminated. + +**Consequence for the framework:** the PSX text validator must be Shift-JIS +aware (2-byte lead+trail, 1-byte halfwidth kana `A1–DF`, ASCII `20–7E`), must +**pass through control bytes**, and must treat a "string" as a record delimited +by **either** NUL (menu labels) **or** `0xFFFF` (framed messages). The hash key +is the raw record bytes up to (not including) the terminator. + +### 2.2 String storage + +- **In-EXE Shift-JIS text block:** a large body of tutorial/hint/system messages + lives in the EXE at roughly **`0x80070900`–`0x80072000`** (offsets + `0x60900`–`0x62000`), reached through a **pointer/index table**. +- **Text pointer table** at **`0x80071474`** (preceded by header word + `0x0000FC10` at `0x80071470`): a contiguous array of 32-bit little-endian + pointers into the text records, e.g. + `0x80070918, 0x80070958, 0x800709A4, 0x800709F0, … 0x80070AA8, 0x80070AC4, …`. + This is the message-ID → record-address map; the text system indexes it by ID. +- **Menu/UI labels** live earlier in `.rodata` as NUL-terminated SJIS + (`0x80010C5C`, `0x80010C94`, the `？？？？` row at `0x80010190`, memcard save + titles `BISLPS-02253 TSMLGT` / `TSMMP` for "TSUMU light" / "TSUMU MaP"). +- **Disc-streamed text/data:** `DISC.md` lists **`MOJI.BIN`** (文字 = "glyphs/ + characters" → the font sheet) and **`MONDAI.BIN`** (問題 = "problems/puzzles" → + puzzle text/data), plus `GRAPH1-4.BIN`, `SOUND.BIN`, FMV (`DOUGA.BIN`/ + `END.STR`), `TSUMU.XA`. Disc image + files are present locally under + `TsumuRecomp/tsumu/`. So some text is **static (patchable in the EXE)** and + some is **loaded/composed from disc at runtime** (must be caught at draw time). + +Static vs. dynamic: the in-EXE block and menu labels are static and could in +principle be patched at load, **but the framework deliberately intercepts at +draw time** (like PMS) so it uniformly covers disc-loaded and dynamically- +composed strings too, without touching game RAM. + +**Recompiler note (not blocking):** the recompiler's flow walker currently +mis-decodes the text/pointer-table region as code (it emitted +`func_80070300` over `0x80070300…`, decoding the pointer words as load-byte +instructions — classic data-walk over-reach). This does not affect the runtime +translation hook (which works on live memory), but a data/code separation pass +would clean the generated C. Track separately. + +### 2.3 Text render path + +`psx_dispatch` funnels every call, so the exact draw routine is **pinned at +runtime by the capture census** (Phase 0), exactly as PMS discovered its +`~70` text PCs. Static anchors already found to seed/confirm it: + +- The text-lookup routine loads base **`0x80071474`** and indexes it by message + ID to fetch a record pointer; that pointer flows (as `a0`/`a1`/`a2`) into a + glyph-draw routine that walks the SJIS bytes, maps each char to a glyph cell in + the font texture (uploaded from `MOJI.BIN` into VRAM), and issues textured + sprite primitives via the GPU. Menu labels reach the same drawer directly. +- Phase 0 confirms the exact draw PC(s), the arg register carrying the string + pointer (PSX convention is commonly `a0`=str or `a0`=x,`a1`=y,`a2`=str), and + the glyph-draw leaf (the PSX analogue of `FUN_8001a3e4`) if a self-render path + is wanted for fit control. + +(Full static disassembly of the drawer was not completed here because Ghidra +auto-analysis produced 0 functions on the raw dump and the region is +data-poisoned; the runtime census is the reliable, spec-sanctioned way to pin it +and is a Phase-0 deliverable.) + +### 2.4 Font / glyph availability + +- The game **renders fullwidth Latin and fullwidth digits today** (`ＴＳＵＭＵ + ｌｉｇｈｔ`, score digits), so its font sheet **already contains a fullwidth Latin + A–Z / a–z / 0–9 block** (SJIS `0x8260–0x8279`, `0x8281–0x829A`, + `0x824F–0x8258`). **English can therefore render with zero glyph injection** by + emitting fullwidth-Latin SJIS codes — the same "existing glyphs" premise that + made PMS work. +- The game also uses **halfwidth katakana** (`0xA1–0xDF`), so a **halfwidth grid + exists**. Whether a **halfwidth ASCII block (`0x20–0x7E`)** is also present — + which would let English render *compactly* (half width) instead of fullwidth — + is the **open question**, resolved in Phase 0 by dumping `MOJI.BIN` / the VRAM + font page. PSX SJIS fonts frequently include halfwidth ASCII, so this is + likely but must be confirmed. +- **Contingency** (if no compact Latin): inject an ASCII glyph set into the font + page — append a Latin row to the `MOJI.BIN` glyph upload or upload a + supplementary sheet into free VRAM and map ASCII codes to it (mirrors PMS's + documented injection fallback). + +### 2.5 Length / layout constraints + +- **Menu labels** sit in fixed-width boxes; **framed messages** are laid out with + the game's own `0xFFFF`/line-break control codes into a text window. +- English is usually longer than Japanese, and **fullwidth Latin doubles the + width**, so overflow is the primary layout risk. Mitigations (§3.4): prefer + halfwidth ASCII when available; re-insert the game's own line-break control + codes to reflow multi-line messages; optionally a self-render fit path + (PMS-style advance condensing) for coordinate drawers; and per-entry + `max_w`/`fit` metadata in the table to cap width. + +--- + +## 3. Framework design (PSX) + +Shared module, proposed location: `runtime/src/text_xlate.c` + +`runtime/include/text_xlate.h` (peers of `fntrace.*` and `bios_hle.*`). + +### 3.1 The hook point — `psx_dispatch` + +`psx_dispatch` / `psx_dispatch_impl` (declared `cpu_state.h:164`) is the single +chokepoint every guest call routes through. Two precedents already hang off it: + +- **`fntrace_record(cpu, target)`** (`fntrace.c:58`) — an **always-on ring** of + every dispatch (`{frame, target, ra, a0..a3, s3, sp}`), explicitly designed as + "record from boot, query the window after the fact" (the project ring-buffer + rule). This is the capture analogue. +- **`g_psx_bios_hle_hook`** (`bios_hle.h:54`) — a hook slot consulted at the top + of `psx_dispatch_impl` with the pre-normalize physical address; it can + intercept/replace the target (HLE call + boot shell-skip). This is the apply + analogue. + +The translation module adds the same two shapes, so it needs **no per-function +codegen** and works in Release. + +### 3.2 Capture subsystem (always-on ring) + +Add `text_capture_record(cpu, target)` called from `psx_dispatch` right beside +`fntrace_record` (or fold into it): + +1. For each candidate arg (`gpr[4..7]`, plus a few stack slots `sp+0x10..`), + run **`sjis_read_record(addr, out, &len, &term)`**: + - Reject if `addr` outside main RAM (`0x80000000–0x80200000` and mirrors). + - Walk bytes until NUL **or** `0xFFFF` terminator (cap `kSrcMax=512`), classi- + fying each: ASCII `0x20–0x7E`, halfwidth kana `0xA1–0xDF`, SJIS 2-byte + lead+trail, or known control byte. Reject on any non-textish byte (filters + structs/pointers). Require ≥1 real text char. +2. Hash the record bytes: **`h = fnv1a64(bytes, len)`**. +3. Upsert into an **always-on inventory ring** `{h, bytes[≤64 sample], full-len, + src_addr, target_pc, ra, first_frame, count}`; on first sight, append to a + persistent `tsumu_stringdump.log` (flush immediately — crash-safe, per the + ring-buffer rule). +4. `textpc_add(target)` into a lock-free set so the apply hook only scans args on + proven text PCs. + +This runs from boot for every player (no arm gate). "Enumerate every string" = +drive the game (§6) and read the ever-growing inventory; "find untranslated" = +query the ring for records with no table entry. + +**Coord/`_Printf` disambiguation:** reuse PMS's precision filters adapted to +PSX — a coordinate gate (`a0,a1 < 0x1000`) for the class-1 drawer, plus explicit +handling for any universal formatter PC once the census reveals it. + +### 3.3 Translation table (multilingual, human-editable) + +**Location:** `TsumuRecomp/translations/tsumu.toml` (single multilingual file; +TOML fits the project's config conventions — `game_options.toml`, +`SCPH1001.toml`). Loaded by the shared module from a path the runtime injects +(peer of the overlay-cache dir injection). Hot-reloaded on mtime change. + +**Schema:** + +```toml +# tsumu.toml — Tsumu Light string translations (multilingual) +schema = 1 +default_lang = "en" +langs = ["en", "es", "de", "fr"] # first-class multilingual; add columns freely + +[[entry]] +src_hex = "82bb82ea82b6..." # raw Shift-JIS record bytes (hex), the hash key +src_jp = "それじゃ、がんばってね！" # documentation (decoded), not used for matching +src_addr = 0x80070AA8 # optional disambiguator when two records collide on hash +term = "ffff" # terminator that delimited this record: "nul" | "ffff" +orig_w = 12 # JP footprint in glyph cells (fit budget / docs) +max_w = 12 # optional hard width cap (glyph cells) for overflow control +note = "tutorial: end-of-lesson encouragement" +en = "Well then, do your best!" +es = "¡Pues venga, esfuérzate!" +de = "Also dann, gib dein Bestes!" + +[[entry]] +src_hex = "8273827282758..." # ＴＳＵＭＵ　ｌｉｇｈｔ　プレイデータ +term = "nul" +en = "TSUMU light Play Data" +``` + +- **Key = `fnv1a64(src bytes)`** (content hash — robust to buffer reuse), with + `src_addr` as a tiebreak if two distinct records ever collide. +- Per-language target strings; **untranslated / missing-language falls back to + the source** (render the original Japanese) so the game is never broken by a + gap. +- Target strings are authored in **UTF-8**; the apply layer transcodes UTF-8 → + game glyph codes (§3.4). `\n` requests a line break (emitted as the game's own + line-break control code). `%d`/`%s`-style specifiers are preserved for dynamic + args if the census shows a formatter path. +- `orig_w`/`max_w` drive width fitting. + +**Authoring tools** (shared, `psxrecomp/tools/`, adapted from PMS): +`text_xlate_decode.py` (stringdump → decoded TSV master, Shift-JIS via Python +`cp932`), `text_xlate_build.py` (join `\tUTF-8` → `tsumu.toml`, +merge-preserving), `text_xlate_todo.py` (list untranslated by draw count → +coverage report). + +### 3.4 Apply subsystem (dispatch hook) + +Install `g_psx_text_xlate_hook` (sibling to `g_psx_bios_hle_hook`), consulted at +the top of `psx_dispatch_impl` **after** the BIOS-HLE hook, gated by a lock-free +`armed` flag (table non-empty) and the `textpc_contains(target)` set: + +1. Read the source record at the string-arg register (`sjis_read_record`). +2. `h = fnv1a64(...)`; look up `{lang}` target. Miss → **return without change** + (fallback to source). +3. **Transcode** the UTF-8 target → a byte sequence the game's drawer will + render: + - **Halfwidth-ASCII path** (preferred, if Phase 0 confirms halfwidth Latin + glyphs): emit ASCII `0x20–0x7E` directly — compact, closest to JP width. + - **Fullwidth-Latin path** (guaranteed): map each ASCII char to its fullwidth + SJIS code (`A→0x8260…`, `a→0x8281…`, `0→0x824F…`, space→`0x8140`). Always + works because those glyphs are proven present. Double width → rely on width + fitting. + - **Injected-glyph path** (contingency): map to injected ASCII glyph cells. + - Re-emit the record's framing: preserve/insert the game's `0xFFFF` terminator + and line-break control codes; convert `\n` to the game's line-break code. +4. **Write to guest scratch** below the caller's `$sp` (`gpr[29]`) using the + runtime store accessors (little-endian, no swizzle): pick a scratch slot with + stack headroom (e.g. `sp - 0x800`, `& ~7`), bounds-checked against RAM; NUL/ + `0xFFFF`-terminate. +5. **Repoint the string-arg register** at the scratch address. Return "not + handled" so the **original draw routine runs and renders the replacement** + (the whole no-regen premise). A reentrancy `thread_local` guard prevents the + hook re-triggering on any nested draw calls. + +**Length / layout handling:** +- Prefer halfwidth ASCII (compact). Use table `max_w` to cap width. +- For framed multi-line messages, reflow by inserting the game's own line-break + control codes at word boundaries to fit the text window. +- **Optional self-render fit path** (Phase 1+, mirrors PMS `self_render_static`): + once the glyph-draw leaf PC is known, draw the target ourselves per-glyph with + condensed advances and suppress the caller's draw — gives pixel-accurate + fit-to-JP-footprint. Not required for first light; the pointer-swap path is. + +### 3.5 Language selection + +- **`game.toml` / `settings.toml`:** add `[localization] language = "en"` (peer + of the existing renderer/aspect/FMV/turbo keys; `settings.toml` overrides + `game.toml`, CLI overrides both — per the project's settings precedence). + `"jp"`/`"off"` = passthrough (no translation). +- **RmlUi launcher:** a "Language" dropdown populated from the table's `langs` + list; writes the choice to `game_options.toml`. Runtime switch = reselect → + the module reloads the active language column (no restart needed; hot-reload + already covers table edits). +- Default = `default_lang` from the table (English for Tsumu). + +### 3.6 Coverage workflow + +The loop to reach "all strings target-language" (§0 goal): + +1. **Capture pass.** Fresh run; drive every screen/menu/tutorial/dialogue/ending + (self-driven via the debug port + input injection, or a scripted playthrough). + The always-on ring logs every distinct record to `tsumu_stringdump.log`. +2. **Decode + author.** `text_xlate_decode.py` → master TSV (Japanese visible). + Fill translations; `text_xlate_build.py` → `tsumu.toml`. +3. **Apply + verify.** Re-run with the table; the hook substitutes. **Verify + visually** (screenshot both windows — never infer from a counter), per the + project's verification rules. +4. **Re-capture to find gaps.** `text_xlate_todo.py` diffs the live inventory + against the table → coverage report (`N/total translated`, untranslated + sorted by draw count). Repeat until the untranslated set is empty across a + full playthrough. +5. **Coverage report** artifact: total distinct records seen, translated %, + per-screen breakdown, list of dynamic/composed strings needing special + handling. "We've found them all" = a full playthrough adds zero new records to + the inventory. + +### 3.7 Framework generality (what's shared vs. title-specific) + +| Shared (framework) | Title-specific | +|---|---| +| Capture ring + `sjis_read_record` + inventory persistence | The translation table (`.toml`) | +| `fnv1a64` keying, text-PC census set | The font situation (Latin present? inject?) | +| Apply dispatch hook + guest-scratch write + arg repoint | Encoding profile (SJIS here; another game might be custom-tiled) | +| UTF-8 → glyph transcoders (ASCII, fullwidth-Latin, injected) | The string-arg register convention (pinned by census) | +| Config keys + launcher wiring + hot-reload | Control-framing profile (terminators, line-break codes) | +| Authoring tools (decode/build/todo/coverage) | — | + +Encoding is abstracted behind an **`EncodingProfile`** (validator + terminator +rules + transcoders) selected per title in `game.toml` +(`[localization] encoding = "shift_jis"`), so a game using a custom +tile-index encoding plugs in its own profile without touching the capture/apply +core. + +--- + +## 4. Font/glyph plan — concrete Tsumu situation + +1. **Guaranteed today:** fullwidth Latin + fullwidth digits are in the font + (proven by `ＴＳＵＭＵ ｌｉｇｈｔ`). Ship English on the **fullwidth-Latin path** + for first light — zero asset work, guaranteed to render. +2. **Phase-0 check:** dump `MOJI.BIN` / the live VRAM font page and inspect for a + **halfwidth ASCII block** (`0x20–0x7E`). The presence of halfwidth katakana + (`0xA1–0xDF`) means a halfwidth grid exists; if ASCII is in it, switch to the + compact **halfwidth-ASCII path** (better width fit). +3. **If neither compact option:** inject an ASCII glyph set — extend the + `MOJI.BIN` upload or blit a supplementary Latin sheet into free VRAM and map + ASCII to those cells. Add a small `font_probe`/`font_dump` tool (PSX analogue + of `pkmnstadium_fontdump`) that reports the font page layout and which code + ranges have ink. + +--- + +## 5. Risks / open questions + +1. **Halfwidth ASCII glyphs present?** Determines compact vs. fullwidth (width + budget). Phase-0 blocker for layout quality (not for first light). +2. **Exact text-draw PC(s) + string-arg register.** Pinned by the Phase-0 + census; menu labels vs. framed messages may use different drawers/registers + (PMS had `~70` PCs and multiple conventions — expect the same spread). +3. **Control-code framing semantics.** `0xFFFF` (terminator) and the `0x81…`/ + `0x82 0xFE 0xFF` sequences (line break/format) must be decoded so we preserve + them across substitution and can reflow. Reverse the exact code set in + Phase 0. +4. **Dynamically composed strings** (scores, counts, names concatenated at + runtime, e.g. the `？？？？` placeholder row): the source bytes vary, so a + pure content-hash misses them. Handle via formatter-template entries + (`"%d"`-style) once the census reveals a composing routine, or per-fragment + translation — same class of problem PMS solved for `%sポケモン`. +5. **Length overflow / fixed boxes.** English (esp. fullwidth) overruns menu + boxes; needs `max_w` capping, reflow via line-break codes, and possibly the + self-render fit path. +6. **Disc-loaded text (`MONDAI.BIN`/`MOJI.BIN`).** Confirm puzzle/problem text is + caught at draw time (it should be — the capture is draw-time, not load-time); + if a puzzle *is* Japanese-character gameplay, translation may be + gameplay-affecting and should be scoped carefully (menus/UI/tutorials first). +7. **DTE / dictionary compression.** Not observed (text is plain SJIS in the + EXE), but confirm no compressed text table exists on disc before declaring + full coverage. +8. **Hot path cost.** The apply hook runs at every dispatch; keep the fast reject + (lock-free armed flag + text-PC set membership) ahead of any RAM scan, exactly + as PMS does, so non-text dispatch pays ~nothing. + +--- + +## 6. Phased implementation plan + +- **Phase 0 — Discovery (runtime census).** Add the capture ring + + `sjis_read_record` + inventory persistence + text-PC census. Drive Tsumu; dump + `tsumu_stringdump.log`. Pin the text-draw PC(s), the string-arg register, the + control-code set, and the font situation (`MOJI.BIN` dump). Deliver the encoding + profile. *(No visible change yet — pure observability.)* +- **Phase 1 — Apply (pointer-swap).** Install `g_psx_text_xlate_hook`; implement + the fullwidth-Latin transcoder + guest-scratch write + arg repoint. Author a + starter `tsumu.toml` (main menu). Verify menus render English visually. +- **Phase 2 — Encoding/layout polish.** Halfwidth-ASCII path if available; + reflow via line-break codes; `max_w` fitting; framed-message coverage. +- **Phase 3 — Multilingual + launcher.** Language dropdown, `game.toml`/ + `settings.toml` key, runtime switch, second language column to prove the + pipeline. +- **Phase 4 — Coverage to 100%.** Iterate capture→author→verify with the + coverage report until a full playthrough surfaces no untranslated records. + Handle dynamic/composed strings and any injected-glyph work. +- **Phase 5 — Generalize.** Lift the module to a clean framework API + (`EncodingProfile`, config-driven table path), document a "new title" recipe, + and (optionally) fix the data/code separation for the mis-decoded text region. + +--- + +## Appendix A — Key source references + +**n64recomp prior art** (`F:\Projects\n64recomp\PocketMonstersStadiumRecomp\`): +- `src/main/diagnostics.cpp` — capture (`pkmnstadium_textdraw_probe` 1235, + `read_text_arg` 1152, `strinv_upsert`/`textpc_add`), apply + (`pkmnstadium_text_xlate` 2100, `self_render_static` 1887, `xlate_desc`/ + `xlate_general`), `load_translations` 1550, `pkmnstadium_fontdump` 2227. +- `include/trace.h` — `TRACE_ENTRY()` hook wiring. +- `tools/pms_build_translations.py` — decode/build/todo authoring. +- `translations.json` — table format. + +**psxrecomp hook points** (`F:\Projects\psxrecomp\_wt-tsumu\psxrecomp\`): +- `runtime/include/fntrace.h` + `runtime/src/fntrace.c:58` — always-on dispatch + ring (capture analogue). +- `runtime/include/bios_hle.h:54` (`g_psx_bios_hle_hook`) + `runtime/src/bios_hle.c:297` + — dispatch hook slot precedent (apply analogue). +- `runtime/include/cpu_state.h:164` — `psx_dispatch` / `psx_dispatch_call`. + +**Tsumu evidence** (`F:\Projects\psxrecomp\TsumuRecomp\`): +- `ghidra/SLPS_022.53_no_header.bin` — boot EXE (load `0x80010000`, entry + `0x800691F0`). +- Menu labels `0x80010C5C`, `0x80010C94`, `0x80010190`; tutorial text block + `~0x80070900–0x80072000`; **text pointer table `0x80071474`** (header + `0x0000FC10` @ `0x80071470`). +- `DISC.md` — `MOJI.BIN` (font), `MONDAI.BIN` (puzzle text), disc file list. +- `game.toml` — `load_address`, `entry_pc`, `debug_port = 4510`.