docs: spec on-the-fly string translation / localization framework

Implementation-ready design for capturing PSX source strings from memory
and substituting translated bytes on the fly, so JP-only titles (Tsumu
Light SLPS-02253 first) can be played in English or any language.

Adapts the author's proven n64recomp prior art (Pocket Monsters Stadium
JP): always-on capture ring keyed by FNV-1a64 content hash, a per-title
multilingual table, and a draw-time apply hook that repoints the string
arg at guest scratch so the original renderer draws the replacement (no
regen, unbounded length, dynamic args preserved).

Grounds the PSX design in real findings: Tsumu text is Shift-JIS (2-byte
fullwidth + 1-byte halfwidth kana) with 0xFFFF-framed message records;
in-EXE text block ~0x80070900-0x80072000 via pointer table at 0x80071474;
menu labels at 0x80010C5C/0x80010C94; font already has fullwidth Latin
(English renders with zero glyph injection). Hooks onto the psx_dispatch
chokepoint (fntrace_record census + g_psx_bios_hle_hook precedent), so no
per-function codegen and works in Release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
2026-07-05 11:04:47 -07:00
parent c586badcb4
commit bf4482b4c8
+581
View File
@@ -0,0 +1,581 @@
# On-the-Fly String Translation / Localization for psxrecomp
**Status:** SPEC (design only — no implementation yet). Reusable framework
feature; **Tsumu Light (SLPS-02253)** is the first consumer.
**Goal.** Let a Japanese-only PSX title be played in English (or any target
language) by **capturing the game's source strings from memory and substituting
translated bytes on the fly**, so a non-Japanese speaker can navigate menus,
read tutorials/dialogue, and understand the game. **Ultimate success = every
string we can find renders in the target language.** Multilingual is a
first-class requirement (English is just the first target); the mechanism is a
**shared psxrecomp module** any title can adopt.
This design is a direct PSX adaptation of the author's proven N64 prior art in
the `n64recomp` fork for **Pocket Monsters Stadium (Japan)**
(`F:\Projects\n64recomp\PocketMonstersStadiumRecomp\`). Section 1 documents that
mechanism; Section 2 documents Tsumu's concrete text reality; Sections 3+ define
the PSX framework.
---
## 0. TL;DR architecture
Three always-on subsystems hung off the **`psx_dispatch` call chokepoint** (the
one place every guest function call funnels through — see §3.1):
1. **Capture ring (always-on).** On every dispatch, scan the argument registers
(and a few stack slots) for pointers to Shift-JIS text records. Hash each
distinct record (FNV-1a64 over its raw bytes), record `{hash, bytes, addr,
target-PC, call-site}`, persist immediately. This *enumerates every string to
translate* and *discovers which PCs draw text* — no arm-then-capture (per the
project's ring-buffer rule).
2. **Translation table (per-title, human-editable, multilingual).**
`TsumuRecomp/translations/tsumu.toml` — keyed by source-byte hash (+ optional
`src_addr`), one target string per language, with encoding/length metadata.
Hot-reloaded on mtime change.
3. **Apply hook (always-on).** A dispatch hook (sibling to the existing
`g_psx_bios_hle_hook`) that, when the target is a registered text-draw PC and
an arg points at a translated source record, writes the target-language bytes
into transient guest scratch (below `$sp`), repoints the arg register, and
lets the **original** draw routine render it. No regen, no engine change,
unbounded replacement length, dynamic format strings preserved.
The only title-specific pieces are (a) the translation table and (b) the font
situation (§4). Everything else — capture ring, apply hook, config, coverage
tooling — is shared framework code.
---
## 1. Prior art: the n64recomp PMS-J mechanism
Files: `PocketMonstersStadiumRecomp/src/main/diagnostics.cpp` (the engine),
`include/trace.h` (the hook wiring), `tools/pms_build_translations.py` (authoring),
`translations.json` (the table). Verified by reading the source.
### 1.1 Capture — always-on string inventory + text-PC census
The recompiler emits a bare `TRACE_ENTRY()` at the top of **every** recompiled
function when `game.toml` has `trace_mode = true`. `trace.h` expands it to call,
with `rdram`/`ctx` in scope:
```c
pkmnstadium_textdraw_probe(__func__, ra, a0, a1, a2, a3); // census + inventory
pkmnstadium_text_xlate(__func__, rdram, ctx); // apply
```
`pkmnstadium_textdraw_probe` (diagnostics.cpp:1235) is the **always-on capture**
(explicitly *no* env gate — "we never 'arm' capture and hope to catch the
window, we record every unique on-screen string as it is first drawn and persist
it immediately"):
- **`read_text_arg()`** (1152) validates that an arg register points at a
NUL-terminated EUC-JP/ASCII run ≥2 bytes: every byte must be newline,
printable ASCII, or a EUC-JP lead/kana byte (`0x8E/0x8F/0xA1–0xFE`). The
RDRAM-range check is the real discriminator — coords/ints/non-pointers are
rejected before the byte scan.
- **Class-1 convention gate:** most text draws use `x=a0, y=a1, str=a2`; the
`a0<0x1000 && a1<0x1000` "looks like screen coords" test rejects binary
headers (`Yay0`/`PRESJPEG`) that decode as valid EUC-JP by chance.
- **`_Printf` chokepoint** (`kPrintfPC`): the universal vararg formatter is
special-cased — its format string (a2) is inventoried regardless of the coord
gate, so battle/overlay text built via `_Printf` is captured too.
- **`strinv_upsert()`** records every *distinct* string (content-hash keyed) into
`stringdump.log`, flushed on every newly-seen string so a crash never loses
captures.
- **`textpc_add()`** maintains a lock-free open-addressing set of PCs proven to
draw text (`~70` distinct), so the apply hook only pays the arg-scan cost on
real text PCs.
There is also a capture-only **discovery probe** (`xlate_discovery`, gated by
`PMS_TEXTPROBE`) that finds *which register/stack slot* a known-untranslated
watch-string arrives in — used to learn non-standard conventions (battle labels,
Pokédex descriptions in `r7`/`r4`, the `%sポケモン` category classifier in `r5`).
### 1.2 Translation table format
`translations.json` next to the exe, hot-reloaded on mtime (1550):
```json
{ "src_hex": "a5d0a5c8a5eb", "src_jp": "バトル", "target": "Battle", "orig_w": 3 }
```
- **`src_hex`** — the raw source bytes (EUC-JP) in hex. The load path converts
hex→bytes and keys the in-memory map by **`FNV-1a64` of those bytes** (robust
to RDRAM buffer reuse — the key is content, not address).
- **`target`** — the replacement (ASCII; may contain `%s`/`%d` specifiers to
preserve dynamic args, and `\n` for line breaks).
- **`orig_w`** — Japanese footprint in glyph units (documentation + default fit
budget). Optional **`fit_w`** overrides the width budget (use more/less space
than the original when there is room). `src_jp` is documentation only.
Authoring workflow (`pms_build_translations.py`): `stringdump.log` → decode
EUC-JP → `strings_decoded.tsv` (persistent master across sessions) → author
`key_en.tsv` (`<fnv-key>\t<english>`) → `build` joins them into `translations.json`,
**merge-preserving** hand-authored entries. A `todo` mode lists still-untranslated
strings sorted by draw count.
### 1.3 Apply — on-the-fly substitution
`pkmnstadium_text_xlate` (2100) runs at every function entry; cheap no-ops
(`g_xlate_active` reentrancy guard, lock-free `g_xlate_armed` reject, reload
every 1024 calls). On a text PC it takes one of three paths:
- **Pointer-swap (`write_guest_str` + repoint `ctx->r6`).** Content-hash the
source at the fmt arg; on a KV hit, write the replacement (ASCII, NUL-term)
into **transient scratch on the calling thread's guest stack** (`sp-0x800` for
format templates, `sp-0xC00` for literal args — two disjoint slots so a paired
`_Printf` swap can't self-clobber) and repoint the arg register. The
**original function then runs unchanged** — `_Printf` formats it (dynamic
`%d`/`%s` stay intact) and renders glyph-by-glyph. No engine change, no regen,
unbounded replacement length (never writes back into game RAM).
- **Proportional self-render (`self_render_static`, 1887).** For plain strings at
coordinate drawers, it draws the English itself by calling the game's glyph
routine (`FUN_8001a3e4`) per character with **per-glyph proportional advances**
(calibrated by measuring real ink width of each glyph tile in VRAM,
`calibrate_slot`/`measure_glyph`) and **auto-fit** condensing so English never
spills past the Japanese footprint (`fit_w` budget or per-line JP width). Then
it suppresses the caller's own draw with an empty fmt.
- **Register-specific paths** (`xlate_desc`, `xlate_general`) for the
discovered non-`r6` conventions, including a subtle `_Printf` writeproc
byte-count re-sync when a shorter replacement would otherwise embed a NUL and
truncate a composed string.
### 1.4 Multilingual & font
- **Multilingual** is table-driven: the KV value is per-language. (PMS shipped
English only, but the schema and hook are language-agnostic — selecting a
language means loading a different column/file.)
- **Font/glyph:** PMS-J's font sheet **already contained Latin glyphs**, so
English renders directly through the existing glyphs (the whole "no glyph
injection needed" premise). `pkmnstadium_fontdump` (2227) dumps the font-state
struct + glyph sheets (`.i8`) to answer exactly this gating question; the
fallback contingency (documented in comments/tools) is to inject an ASCII
glyph set if a game lacks Latin glyphs. The renderer measures glyph tiles
directly from VRAM to get proportional widths.
### 1.5 What carries to PSX vs. what changes
| Aspect | N64 (PMS-J) | PSX (Tsumu) |
|---|---|---|
| Hook point | `TRACE_ENTRY()` emitted per function (needs `trace_mode=true` regen) | **`psx_dispatch` chokepoint** — one install site, no regen, works in Release |
| Encoding | EUC-JP (JIS X 0208 via `A1/A3/A4/A5` leads) | **Shift-JIS** (2-byte fullwidth + 1-byte halfwidth kana `A1–DF`) + custom control framing |
| String delimiter | NUL | **NUL (menu labels) AND `0xFFFF` (framed message records)** |
| Memory swizzle | big-endian `^3` byte swap | **little-endian, no swizzle** (simpler) |
| Arg access | `ctx->r4..r7` | `cpu->gpr[4..7]` |
| Glyph routine | `FUN_8001a3e4` (known) | text-draw PC pinned at runtime by the census (Phase 0) |
| Latin glyphs | present in font | fullwidth Latin present (proven); halfwidth ASCII TBD (§4) |
---
## 2. Tsumu's concrete text reality
Evidence gathered from the headerless boot EXE
(`TsumuRecomp/ghidra/SLPS_022.53_no_header.bin`, 512 KB, loads at `0x80010000`,
entry `0x800691F0`) via direct byte analysis + Ghidra, plus `DISC.md`.
### 2.1 Encoding — Shift-JIS (confirmed)
The text is **Shift-JIS**, not ASCII and not EUC-JP (EUC was the N64 game).
Proven by decoding real records:
| Address | Decoded | Notes |
|---|---|---|
| `0x80010C5C` | `TSUMU light プレイデータ` | fullwidth Latin + fullwidth space + katakana ("TSUMU light / Play Data") |
| `0x80010C94` | `TSUMU マップデータ` | "TSUMU / Map Data" |
| `0x80010190` | `????????` | fullwidth question marks (placeholder score row) |
| `0x80070AA8` | `それじゃ、がんばってね!`… | tutorial dialogue (hiragana) |
The character mix is:
- **2-byte fullwidth** (JIS X 0208): kanji, hiragana, katakana, **fullwidth
Latin `0x8260–0x8279` (A–Z), `0x8281–0x829A` (a–z), fullwidth digits
`0x824F–0x8258`**, symbols. Lead byte `0x81–0x9F` / `0xE0–0xFC`, trail
`0x40–0x7E` / `0x80–0xFC`.
- **1-byte halfwidth katakana** `0xA1–0xDF` (e.g. `0xBB`=サ) — mixed inline with
2-byte chars. This means the font has a **halfwidth grid**.
- **Custom control framing** interleaved with text (raw bytes, not SJIS):
- `0xFF 0xFF` — **message terminator** (record end).
- `0x82 0xFE 0xFF` / `0x81 …` sequences — **line break / formatting** markers.
- Menu labels (e.g. `0x80010C5C`) are simpler: plain 2-byte SJIS, **NUL**-terminated.
**Consequence for the framework:** the PSX text validator must be Shift-JIS
aware (2-byte lead+trail, 1-byte halfwidth kana `A1–DF`, ASCII `20–7E`), must
**pass through control bytes**, and must treat a "string" as a record delimited
by **either** NUL (menu labels) **or** `0xFFFF` (framed messages). The hash key
is the raw record bytes up to (not including) the terminator.
### 2.2 String storage
- **In-EXE Shift-JIS text block:** a large body of tutorial/hint/system messages
lives in the EXE at roughly **`0x80070900`–`0x80072000`** (offsets
`0x60900`–`0x62000`), reached through a **pointer/index table**.
- **Text pointer table** at **`0x80071474`** (preceded by header word
`0x0000FC10` at `0x80071470`): a contiguous array of 32-bit little-endian
pointers into the text records, e.g.
`0x80070918, 0x80070958, 0x800709A4, 0x800709F0, … 0x80070AA8, 0x80070AC4, …`.
This is the message-ID → record-address map; the text system indexes it by ID.
- **Menu/UI labels** live earlier in `.rodata` as NUL-terminated SJIS
(`0x80010C5C`, `0x80010C94`, the `????` row at `0x80010190`, memcard save
titles `BISLPS-02253 TSMLGT` / `TSMMP` for "TSUMU light" / "TSUMU MaP").
- **Disc-streamed text/data:** `DISC.md` lists **`MOJI.BIN`** (文字 = "glyphs/
characters" → the font sheet) and **`MONDAI.BIN`** (問題 = "problems/puzzles" →
puzzle text/data), plus `GRAPH1-4.BIN`, `SOUND.BIN`, FMV (`DOUGA.BIN`/
`END.STR`), `TSUMU.XA`. Disc image + files are present locally under
`TsumuRecomp/tsumu/`. So some text is **static (patchable in the EXE)** and
some is **loaded/composed from disc at runtime** (must be caught at draw time).
Static vs. dynamic: the in-EXE block and menu labels are static and could in
principle be patched at load, **but the framework deliberately intercepts at
draw time** (like PMS) so it uniformly covers disc-loaded and dynamically-
composed strings too, without touching game RAM.
**Recompiler note (not blocking):** the recompiler's flow walker currently
mis-decodes the text/pointer-table region as code (it emitted
`func_80070300` over `0x80070300…`, decoding the pointer words as load-byte
instructions — classic data-walk over-reach). This does not affect the runtime
translation hook (which works on live memory), but a data/code separation pass
would clean the generated C. Track separately.
### 2.3 Text render path
`psx_dispatch` funnels every call, so the exact draw routine is **pinned at
runtime by the capture census** (Phase 0), exactly as PMS discovered its
`~70` text PCs. Static anchors already found to seed/confirm it:
- The text-lookup routine loads base **`0x80071474`** and indexes it by message
ID to fetch a record pointer; that pointer flows (as `a0`/`a1`/`a2`) into a
glyph-draw routine that walks the SJIS bytes, maps each char to a glyph cell in
the font texture (uploaded from `MOJI.BIN` into VRAM), and issues textured
sprite primitives via the GPU. Menu labels reach the same drawer directly.
- Phase 0 confirms the exact draw PC(s), the arg register carrying the string
pointer (PSX convention is commonly `a0`=str or `a0`=x,`a1`=y,`a2`=str), and
the glyph-draw leaf (the PSX analogue of `FUN_8001a3e4`) if a self-render path
is wanted for fit control.
(Full static disassembly of the drawer was not completed here because Ghidra
auto-analysis produced 0 functions on the raw dump and the region is
data-poisoned; the runtime census is the reliable, spec-sanctioned way to pin it
and is a Phase-0 deliverable.)
### 2.4 Font / glyph availability
- The game **renders fullwidth Latin and fullwidth digits today** (`TSUMU
light`, score digits), so its font sheet **already contains a fullwidth Latin
A–Z / a–z / 0–9 block** (SJIS `0x8260–0x8279`, `0x8281–0x829A`,
`0x824F–0x8258`). **English can therefore render with zero glyph injection** by
emitting fullwidth-Latin SJIS codes — the same "existing glyphs" premise that
made PMS work.
- The game also uses **halfwidth katakana** (`0xA1–0xDF`), so a **halfwidth grid
exists**. Whether a **halfwidth ASCII block (`0x20–0x7E`)** is also present —
which would let English render *compactly* (half width) instead of fullwidth —
is the **open question**, resolved in Phase 0 by dumping `MOJI.BIN` / the VRAM
font page. PSX SJIS fonts frequently include halfwidth ASCII, so this is
likely but must be confirmed.
- **Contingency** (if no compact Latin): inject an ASCII glyph set into the font
page — append a Latin row to the `MOJI.BIN` glyph upload or upload a
supplementary sheet into free VRAM and map ASCII codes to it (mirrors PMS's
documented injection fallback).
### 2.5 Length / layout constraints
- **Menu labels** sit in fixed-width boxes; **framed messages** are laid out with
the game's own `0xFFFF`/line-break control codes into a text window.
- English is usually longer than Japanese, and **fullwidth Latin doubles the
width**, so overflow is the primary layout risk. Mitigations (§3.4): prefer
halfwidth ASCII when available; re-insert the game's own line-break control
codes to reflow multi-line messages; optionally a self-render fit path
(PMS-style advance condensing) for coordinate drawers; and per-entry
`max_w`/`fit` metadata in the table to cap width.
---
## 3. Framework design (PSX)
Shared module, proposed location: `runtime/src/text_xlate.c` +
`runtime/include/text_xlate.h` (peers of `fntrace.*` and `bios_hle.*`).
### 3.1 The hook point — `psx_dispatch`
`psx_dispatch` / `psx_dispatch_impl` (declared `cpu_state.h:164`) is the single
chokepoint every guest call routes through. Two precedents already hang off it:
- **`fntrace_record(cpu, target)`** (`fntrace.c:58`) — an **always-on ring** of
every dispatch (`{frame, target, ra, a0..a3, s3, sp}`), explicitly designed as
"record from boot, query the window after the fact" (the project ring-buffer
rule). This is the capture analogue.
- **`g_psx_bios_hle_hook`** (`bios_hle.h:54`) — a hook slot consulted at the top
of `psx_dispatch_impl` with the pre-normalize physical address; it can
intercept/replace the target (HLE call + boot shell-skip). This is the apply
analogue.
The translation module adds the same two shapes, so it needs **no per-function
codegen** and works in Release.
### 3.2 Capture subsystem (always-on ring)
Add `text_capture_record(cpu, target)` called from `psx_dispatch` right beside
`fntrace_record` (or fold into it):
1. For each candidate arg (`gpr[4..7]`, plus a few stack slots `sp+0x10..`),
run **`sjis_read_record(addr, out, &len, &term)`**:
- Reject if `addr` outside main RAM (`0x80000000–0x80200000` and mirrors).
- Walk bytes until NUL **or** `0xFFFF` terminator (cap `kSrcMax=512`), classi-
fying each: ASCII `0x20–0x7E`, halfwidth kana `0xA1–0xDF`, SJIS 2-byte
lead+trail, or known control byte. Reject on any non-textish byte (filters
structs/pointers). Require ≥1 real text char.
2. Hash the record bytes: **`h = fnv1a64(bytes, len)`**.
3. Upsert into an **always-on inventory ring** `{h, bytes[≤64 sample], full-len,
src_addr, target_pc, ra, first_frame, count}`; on first sight, append to a
persistent `tsumu_stringdump.log` (flush immediately — crash-safe, per the
ring-buffer rule).
4. `textpc_add(target)` into a lock-free set so the apply hook only scans args on
proven text PCs.
This runs from boot for every player (no arm gate). "Enumerate every string" =
drive the game (§6) and read the ever-growing inventory; "find untranslated" =
query the ring for records with no table entry.
**Coord/`_Printf` disambiguation:** reuse PMS's precision filters adapted to
PSX — a coordinate gate (`a0,a1 < 0x1000`) for the class-1 drawer, plus explicit
handling for any universal formatter PC once the census reveals it.
### 3.3 Translation table (multilingual, human-editable)
**Location:** `TsumuRecomp/translations/tsumu.toml` (single multilingual file;
TOML fits the project's config conventions — `game_options.toml`,
`SCPH1001.toml`). Loaded by the shared module from a path the runtime injects
(peer of the overlay-cache dir injection). Hot-reloaded on mtime change.
**Schema:**
```toml
# tsumu.toml — Tsumu Light string translations (multilingual)
schema = 1
default_lang = "en"
langs = ["en", "es", "de", "fr"] # first-class multilingual; add columns freely
[[entry]]
src_hex = "82bb82ea82b6..." # raw Shift-JIS record bytes (hex), the hash key
src_jp = "それじゃ、がんばってね!" # documentation (decoded), not used for matching
src_addr = 0x80070AA8 # optional disambiguator when two records collide on hash
term = "ffff" # terminator that delimited this record: "nul" | "ffff"
orig_w = 12 # JP footprint in glyph cells (fit budget / docs)
max_w = 12 # optional hard width cap (glyph cells) for overflow control
note = "tutorial: end-of-lesson encouragement"
en = "Well then, do your best!"
es = "¡Pues venga, esfuérzate!"
de = "Also dann, gib dein Bestes!"
[[entry]]
src_hex = "8273827282758..." # TSUMU light プレイデータ
term = "nul"
en = "TSUMU light Play Data"
```
- **Key = `fnv1a64(src bytes)`** (content hash — robust to buffer reuse), with
`src_addr` as a tiebreak if two distinct records ever collide.
- Per-language target strings; **untranslated / missing-language falls back to
the source** (render the original Japanese) so the game is never broken by a
gap.
- Target strings are authored in **UTF-8**; the apply layer transcodes UTF-8 →
game glyph codes (§3.4). `\n` requests a line break (emitted as the game's own
line-break control code). `%d`/`%s`-style specifiers are preserved for dynamic
args if the census shows a formatter path.
- `orig_w`/`max_w` drive width fitting.
**Authoring tools** (shared, `psxrecomp/tools/`, adapted from PMS):
`text_xlate_decode.py` (stringdump → decoded TSV master, Shift-JIS via Python
`cp932`), `text_xlate_build.py` (join `<key>\tUTF-8` → `tsumu.toml`,
merge-preserving), `text_xlate_todo.py` (list untranslated by draw count →
coverage report).
### 3.4 Apply subsystem (dispatch hook)
Install `g_psx_text_xlate_hook` (sibling to `g_psx_bios_hle_hook`), consulted at
the top of `psx_dispatch_impl` **after** the BIOS-HLE hook, gated by a lock-free
`armed` flag (table non-empty) and the `textpc_contains(target)` set:
1. Read the source record at the string-arg register (`sjis_read_record`).
2. `h = fnv1a64(...)`; look up `{lang}` target. Miss → **return without change**
(fallback to source).
3. **Transcode** the UTF-8 target → a byte sequence the game's drawer will
render:
- **Halfwidth-ASCII path** (preferred, if Phase 0 confirms halfwidth Latin
glyphs): emit ASCII `0x20–0x7E` directly — compact, closest to JP width.
- **Fullwidth-Latin path** (guaranteed): map each ASCII char to its fullwidth
SJIS code (`A→0x8260…`, `a→0x8281…`, `0→0x824F…`, space→`0x8140`). Always
works because those glyphs are proven present. Double width → rely on width
fitting.
- **Injected-glyph path** (contingency): map to injected ASCII glyph cells.
- Re-emit the record's framing: preserve/insert the game's `0xFFFF` terminator
and line-break control codes; convert `\n` to the game's line-break code.
4. **Write to guest scratch** below the caller's `$sp` (`gpr[29]`) using the
runtime store accessors (little-endian, no swizzle): pick a scratch slot with
stack headroom (e.g. `sp - 0x800`, `& ~7`), bounds-checked against RAM; NUL/
`0xFFFF`-terminate.
5. **Repoint the string-arg register** at the scratch address. Return "not
handled" so the **original draw routine runs and renders the replacement**
(the whole no-regen premise). A reentrancy `thread_local` guard prevents the
hook re-triggering on any nested draw calls.
**Length / layout handling:**
- Prefer halfwidth ASCII (compact). Use table `max_w` to cap width.
- For framed multi-line messages, reflow by inserting the game's own line-break
control codes at word boundaries to fit the text window.
- **Optional self-render fit path** (Phase 1+, mirrors PMS `self_render_static`):
once the glyph-draw leaf PC is known, draw the target ourselves per-glyph with
condensed advances and suppress the caller's draw — gives pixel-accurate
fit-to-JP-footprint. Not required for first light; the pointer-swap path is.
### 3.5 Language selection
- **`game.toml` / `settings.toml`:** add `[localization] language = "en"` (peer
of the existing renderer/aspect/FMV/turbo keys; `settings.toml` overrides
`game.toml`, CLI overrides both — per the project's settings precedence).
`"jp"`/`"off"` = passthrough (no translation).
- **RmlUi launcher:** a "Language" dropdown populated from the table's `langs`
list; writes the choice to `game_options.toml`. Runtime switch = reselect →
the module reloads the active language column (no restart needed; hot-reload
already covers table edits).
- Default = `default_lang` from the table (English for Tsumu).
### 3.6 Coverage workflow
The loop to reach "all strings target-language" (§0 goal):
1. **Capture pass.** Fresh run; drive every screen/menu/tutorial/dialogue/ending
(self-driven via the debug port + input injection, or a scripted playthrough).
The always-on ring logs every distinct record to `tsumu_stringdump.log`.
2. **Decode + author.** `text_xlate_decode.py` → master TSV (Japanese visible).
Fill translations; `text_xlate_build.py` → `tsumu.toml`.
3. **Apply + verify.** Re-run with the table; the hook substitutes. **Verify
visually** (screenshot both windows — never infer from a counter), per the
project's verification rules.
4. **Re-capture to find gaps.** `text_xlate_todo.py` diffs the live inventory
against the table → coverage report (`N/total translated`, untranslated
sorted by draw count). Repeat until the untranslated set is empty across a
full playthrough.
5. **Coverage report** artifact: total distinct records seen, translated %,
per-screen breakdown, list of dynamic/composed strings needing special
handling. "We've found them all" = a full playthrough adds zero new records to
the inventory.
### 3.7 Framework generality (what's shared vs. title-specific)
| Shared (framework) | Title-specific |
|---|---|
| Capture ring + `sjis_read_record` + inventory persistence | The translation table (`<title>.toml`) |
| `fnv1a64` keying, text-PC census set | The font situation (Latin present? inject?) |
| Apply dispatch hook + guest-scratch write + arg repoint | Encoding profile (SJIS here; another game might be custom-tiled) |
| UTF-8 → glyph transcoders (ASCII, fullwidth-Latin, injected) | The string-arg register convention (pinned by census) |
| Config keys + launcher wiring + hot-reload | Control-framing profile (terminators, line-break codes) |
| Authoring tools (decode/build/todo/coverage) | — |
Encoding is abstracted behind an **`EncodingProfile`** (validator + terminator
rules + transcoders) selected per title in `game.toml`
(`[localization] encoding = "shift_jis"`), so a game using a custom
tile-index encoding plugs in its own profile without touching the capture/apply
core.
---
## 4. Font/glyph plan — concrete Tsumu situation
1. **Guaranteed today:** fullwidth Latin + fullwidth digits are in the font
(proven by `TSUMU light`). Ship English on the **fullwidth-Latin path**
for first light — zero asset work, guaranteed to render.
2. **Phase-0 check:** dump `MOJI.BIN` / the live VRAM font page and inspect for a
**halfwidth ASCII block** (`0x20–0x7E`). The presence of halfwidth katakana
(`0xA1–0xDF`) means a halfwidth grid exists; if ASCII is in it, switch to the
compact **halfwidth-ASCII path** (better width fit).
3. **If neither compact option:** inject an ASCII glyph set — extend the
`MOJI.BIN` upload or blit a supplementary Latin sheet into free VRAM and map
ASCII to those cells. Add a small `font_probe`/`font_dump` tool (PSX analogue
of `pkmnstadium_fontdump`) that reports the font page layout and which code
ranges have ink.
---
## 5. Risks / open questions
1. **Halfwidth ASCII glyphs present?** Determines compact vs. fullwidth (width
budget). Phase-0 blocker for layout quality (not for first light).
2. **Exact text-draw PC(s) + string-arg register.** Pinned by the Phase-0
census; menu labels vs. framed messages may use different drawers/registers
(PMS had `~70` PCs and multiple conventions — expect the same spread).
3. **Control-code framing semantics.** `0xFFFF` (terminator) and the `0x81…`/
`0x82 0xFE 0xFF` sequences (line break/format) must be decoded so we preserve
them across substitution and can reflow. Reverse the exact code set in
Phase 0.
4. **Dynamically composed strings** (scores, counts, names concatenated at
runtime, e.g. the `????` placeholder row): the source bytes vary, so a
pure content-hash misses them. Handle via formatter-template entries
(`"%d"`-style) once the census reveals a composing routine, or per-fragment
translation — same class of problem PMS solved for `%sポケモン`.
5. **Length overflow / fixed boxes.** English (esp. fullwidth) overruns menu
boxes; needs `max_w` capping, reflow via line-break codes, and possibly the
self-render fit path.
6. **Disc-loaded text (`MONDAI.BIN`/`MOJI.BIN`).** Confirm puzzle/problem text is
caught at draw time (it should be — the capture is draw-time, not load-time);
if a puzzle *is* Japanese-character gameplay, translation may be
gameplay-affecting and should be scoped carefully (menus/UI/tutorials first).
7. **DTE / dictionary compression.** Not observed (text is plain SJIS in the
EXE), but confirm no compressed text table exists on disc before declaring
full coverage.
8. **Hot path cost.** The apply hook runs at every dispatch; keep the fast reject
(lock-free armed flag + text-PC set membership) ahead of any RAM scan, exactly
as PMS does, so non-text dispatch pays ~nothing.
---
## 6. Phased implementation plan
- **Phase 0 — Discovery (runtime census).** Add the capture ring +
`sjis_read_record` + inventory persistence + text-PC census. Drive Tsumu; dump
`tsumu_stringdump.log`. Pin the text-draw PC(s), the string-arg register, the
control-code set, and the font situation (`MOJI.BIN` dump). Deliver the encoding
profile. *(No visible change yet — pure observability.)*
- **Phase 1 — Apply (pointer-swap).** Install `g_psx_text_xlate_hook`; implement
the fullwidth-Latin transcoder + guest-scratch write + arg repoint. Author a
starter `tsumu.toml` (main menu). Verify menus render English visually.
- **Phase 2 — Encoding/layout polish.** Halfwidth-ASCII path if available;
reflow via line-break codes; `max_w` fitting; framed-message coverage.
- **Phase 3 — Multilingual + launcher.** Language dropdown, `game.toml`/
`settings.toml` key, runtime switch, second language column to prove the
pipeline.
- **Phase 4 — Coverage to 100%.** Iterate capture→author→verify with the
coverage report until a full playthrough surfaces no untranslated records.
Handle dynamic/composed strings and any injected-glyph work.
- **Phase 5 — Generalize.** Lift the module to a clean framework API
(`EncodingProfile`, config-driven table path), document a "new title" recipe,
and (optionally) fix the data/code separation for the mis-decoded text region.
---
## Appendix A — Key source references
**n64recomp prior art** (`F:\Projects\n64recomp\PocketMonstersStadiumRecomp\`):
- `src/main/diagnostics.cpp` — capture (`pkmnstadium_textdraw_probe` 1235,
`read_text_arg` 1152, `strinv_upsert`/`textpc_add`), apply
(`pkmnstadium_text_xlate` 2100, `self_render_static` 1887, `xlate_desc`/
`xlate_general`), `load_translations` 1550, `pkmnstadium_fontdump` 2227.
- `include/trace.h` — `TRACE_ENTRY()` hook wiring.
- `tools/pms_build_translations.py` — decode/build/todo authoring.
- `translations.json` — table format.
**psxrecomp hook points** (`F:\Projects\psxrecomp\_wt-tsumu\psxrecomp\`):
- `runtime/include/fntrace.h` + `runtime/src/fntrace.c:58` — always-on dispatch
ring (capture analogue).
- `runtime/include/bios_hle.h:54` (`g_psx_bios_hle_hook`) + `runtime/src/bios_hle.c:297`
— dispatch hook slot precedent (apply analogue).
- `runtime/include/cpu_state.h:164` — `psx_dispatch` / `psx_dispatch_call`.
**Tsumu evidence** (`F:\Projects\psxrecomp\TsumuRecomp\`):
- `ghidra/SLPS_022.53_no_header.bin` — boot EXE (load `0x80010000`, entry
`0x800691F0`).
- Menu labels `0x80010C5C`, `0x80010C94`, `0x80010190`; tutorial text block
`~0x80070900–0x80072000`; **text pointer table `0x80071474`** (header
`0x0000FC10` @ `0x80071470`).
- `DISC.md` — `MOJI.BIN` (font), `MONDAI.BIN` (puzzle text), disc file list.
- `game.toml` — `load_address`, `entry_pc`, `debug_port = 4510`.