formats.md §6: SaveLoadRoutine @0x8002B154 = multi-entry save/load blob driving the linked
PsyQ libmcrd primitives; saveHeaderTemplate @0x80072DF0 decoded — memcard filename
"BASLUS-00726MUSASHI", SJIS title "BRAVE FENCER MUSASHI", and a @+0x54 handler DISPATCH
TABLE of 0x8002Bxxx pointers (the "multi-entry blob" IS this table of save/load sub-handlers).
Static layout VERIFIED. TBD (needs Ghidra-on-EXE + a Drew-operated PCSX-Redux save trace +
PsyQ-memcard CRC research): the save-DATA serialization (player/progress state 0x80078Exx +
checksum) + matching the tractable handlers. ram_savescreen.bin holds the live header.
- memory-map.md: RESOLVE the "Script VM instruction set" open question — BFM has NO bytecode
VM. "Scripting" is compiled-MIPS state machines dispatched via function-pointer tables
indexed by game-state vars, proven by the matched resident dispatchers (D_800D3430[
D_800B99F0](), D_800D3480/3488/3490[state](), the func_800D2Fxx cluster) mirroring the EXE's
GameModeDispatch[gameMode](). Overlays = compiled MIPS that call the resident, not bytecode.
Basis for the gen2-roadmap criterion #2 amendment (T6).
- formats.md §5: SQV driver = semi-custom — thin Square wrappers/glue over stock PsyQ libsnd;
DsMix @0x800D1BD8 is a custom 2-line wrapper (NOT stock libsnd; the 4.7 sig was coincidental,
T1). Bytes matched in the harvest; deep playback semantics deferred (Gen3, not needed for the
byte-match).
The pass-4 functions that match_one matched standalone but the single-TU build rejected
on conflicting shared-symbol extern types are recovered by UNIFYING those types in
resident.c (widening definitions' return types where byte-identical — u16 f(){return
u16g} ≡ s32 f(){...} both lhu;jr; choosing asm-correct widths for globals; per-site
*(u16*)&g tricks where a neighbor pins the type). ~20 symbols reconciled
(func_800291B4/CFD68/D1714/D1078/801465C0/... + D_800AE6A8/80078E50/...).
20 of 21 recovered; func_800D00E4 stays an honest INCLUDE_ASM stub (its switch needs the
external jtbl_80113ED8 relocation, not a C-source fix — analysis preserved under #if 0).
REAL 102 -> 123 (85.62% byte-identical); resident 8e17e02f, main 143dbb89 (R22 clean).
22 stubs remain: ~20 documented compiler-internal residuals (permuter/§3a) + D00E4 + dedup.
- matching-cookbook.md §12: the parallel-draft + deterministic byte-gate + match_one
iteration loop that took the resident 1.4%->71.7% in one session. The pipeline (Workflow
draft -> harvest_verify.py gate -> redraft passes -> match_one.py iterate), retry-wave
resilience, the two TU gotchas (inline-typedef strip; standalone-MATCH-but-build-FAIL =
extern-type conflict -> unify types), and the idioms the swarm surfaced (void-dispatcher-
is-s32, struct-copy prologue, *10->2x+8x, masked-andi-range, callee-return-type-forces-cast).
- effort-map.md: standing instruction to PROACTIVELY prompt Drew to enable /effort ultracode
on breadth-shaped stretches during a Max session (with the Phase-12 evidence), keeping the
deep single-thread tasks at Max. Memories: ultracode-harvest-pattern, effort-prompt-
ultracode-on-breadth.
- tools/match_one.py (new): compile ONE function's C standalone (pinned triple), mask
relocations (jal/HI16/LO16 like psyq_identify), compare to the target bytes in its .s.
Isolated temp dir -> parallelizable; a real per-function asm-differ loop for agents to
iterate against without a shared build. Validated: a matched fn -> MATCH, residuals ->
informative diffs.
- harvest_verify.py + match_one.py: strip inline scalar-typedef redefinitions (some drafts
inlined `typedef unsigned char u8;` -> C89 redefinition error vs common.h = a COMPILE
fail, not a byte mismatch). Re-gate recovered func_800CFBCC + func_800D1938.
- REAL 89 (62.76% byte-identical); resident 8e17e02f, main 143dbb89 (R22 clean).
- 54 stubs remain (scheduling/regalloc residuals + big GPU/dispatch fns) -> match_one-driven
iteration pass next.
Ultracode harvest: a 16-agent workflow (resident-harvest-draft) drafted matching C for
the 135 unmatched resident functions (each agent: asm + cookbook + the 8 worked examples,
writing .run/drafts/<fn>.c, no builds/no Ghidra). Then tools/harvest_verify.py (new) byte-
gated every draft against 8e17e02f one at a time (chunk+bisect), keeping only the byte-
perfect ones and reverting the rest -- the build is the sole arbiter (G3/P9).
- 62 / 127 drafts verified byte-identical in one pass; 65 near/hard misses reverted to stubs.
- REAL 8 -> 70 (48.28%); byte-identical 49.66%. resident 8e17e02f, main 143dbb89 (R22 clean).
- cleaned 35 redundant '#include "common.h"' lines the drafts inlined (still byte-identical).
- remaining: 65 failed drafts (close-misses + the hard dispatch/GPU/big-fn tail) + 8 undrafted
(2 agent batches hit transient API 500s) -> second-pass workflow + manual hard tail.
- opt level: resident is -O2 (0x21f0a003 frame-ptr sig count = 0, vs the EXE's 54;
the EXE's 800 game module is also -O2). No per-file override needed.
- triple (Makefile default, no override): gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000
-mgas -msoft-float -fgnu-linker + maspsx --aspsx-version=2.56 --expand-div.
- 2 byte-exact resident matches confirm it across idiom classes:
* func_800CF854 — lbu D_800AE6BF; sltu $zero (unsigned-!=0 accessor)
* func_800CEE84 — 8-call init: branch-store (D_80127084=0; if(D_800B9A00) =1),
delay-slot-filled func_80010AE0(0x3E0), early-scheduled load
- R22 clean rebuild: resident 8e17e02f, main 143dbb89 — both byte-identical.
make report BINARY=resident: REAL 0 -> 2.
- R24: the "resident is 4.7" DetectPsyQ hit was a coincidental DsMix signature; the
resident's actual game-code triple == the EXE's (byte-confirmed). No candidate
ladder; the STOP-condition did not fire. Harvest (T3) de-risked.
- psyq_identify over 4.7 (conv47, already ELF) AND 4.0 libs vs the resident
(MAIN.CD/FILE_010/1.1, --vram-base 0x800CEDF8), code window + whole file:
NIL footprint — 4.7 libsnd 1/226 (4-ins coincidence), libspu 0/134, libgte
0/509, libgpu 0/61; 4.0 libsnd 2/163 (VM_DON/VM_DOFF 4-5 ins), libspu/gte 0.
All hits are the §9.5 short-object coincidental class (<=8 ins).
- tool sanity: 4.0 libsnd vs the EXE snd region = 35/163 (works, version-sensitive)
- Ghidra corroboration: DsMix = {FUN_800d1bf8(); return 1;} custom wrapper (NOT
stock libsnd; R13 tag refuted); resident makes 61 distinct EXE-range jal calls
vs 37 internal — it CALLS the EXE's SDK, doesn't embed it
- => the resident is ~143 funcs of custom engine code; nothing to link. The
DetectPsyQ 4.7 was one coincidental signature. Phase-12 plan revised
(Drew-approved): drop the library-integration tasks; full substantial harvest
of the resident engine by hand (REAL, not LINKED). Resident unchanged 8e17e02f.
- docs/psyq-worklist.md: corrected resident section with the byte-evidence +
regen command; phase-ends/CURRENT_PHASE.md: revised task plan
- tools/psyq/CHECKSUMS.sha256: integrity record for psyq-4.7-converted.zip (R20) + Phase-12
provenance (the resident is 4.7 -> link from conv47/, not the EXE's 4.0 libs; R24)
- cookbook §11: the cross-binary dedup & code-share workflow — source-level macro share (NOT an
object swap; game fns are interior to one object/binary, D1), the byte-gate, sig_image notes
(h_exact workhorse / self-consistent h_norm D2 / linear-partition+code-end overlay boundaries),
the cross-report as the Phase-12/13 work queue, per-binary SDK provenance (R24)
- SETUP §6.8 + tooling inventory: sig_image.py, dedup_integrate.py, dup_report --cross,
make sig-overlays, config/dedup.us.yaml + src/shared/ (R21)
- docs/psyq-worklist.md: resident PsyQ is 4.7 (Phase-12 linking note)
- README: Gen2 status — resident is the 2nd byte-identical binary; cross-binary dedup live
(~9000 cross-binary groups / ~28 MB collapsible; one engine fn byte-identical in all 134 overlays)
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) +
detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes
~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer
tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix
- Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s)
- dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed
source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals;
compact member sample + #bin column
- RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm
8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one
match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue
- dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob,
no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates)
- src/shared/clearTbl40.h: CLEAR_TBL40 macro = the matched byte-clear loop body, authored ONCE;
instantiated at both func_80037004 and func_80037334 in src/800.c (one source -> two vrams)
- matched first try; clean rebuild main -> 143dbb89 BYTE-IDENTICAL WITH the shared C AND WITHOUT
it (INCLUDE_ASM stub fallback) -> dual invariant proven (R22 clean rebuilds, both states)
- config/dedup.us.yaml: I0_clearTbl40 group registered (h_exact a0744d60…); dedup_integrate
--check validates it; negative tests (corrupt hash / wrong vram) fail-closed (exit 1)
- tools/progress.py: count dedup-shared members as REAL via the registry (the macro form isn't a
parseable function def); REAL 52 -> 54, byte-identical 50.24% -> 50.33%; honest measurement (P9)
- tools/dedup_integrate.py: display vram in hex in diagnostics
- the machinery half of the Phase-11 milestone is proven on the byte-verified EXE
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row
with its binary, buckets by h_exact then h_norm across binaries, splits
cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by
collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md
- Makefile: make report emits the cross report once (gated BINARY=main)
- per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged)
- EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the
byte-finding predicted); real cross groups arrive with overlays (T6)
- phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log
- PhaseEnd_Phase10.md: resident engine blob byte-identical from source (8e17e02f) at 100%
INCLUDE_ASM; the binary-agnostic toolchain proven on a real 2nd binary; main still 143dbb89
- docs/SETUP.md §6.7: the resident first-instantiation + the reusable flat-blob <bin> recipe
(per-binary OBJS prune, build_path=build, flat config, leading-data-word-as-rodata); 2 new
Ghidra tools in the inventory (R21)
- rule R24 (per-binary compiler/SDK provenance — resident is PsyQ 4.7 vs EXE 4.0)
- worklog archived CURRENT_PHASE.md -> phase-ends/logs/Phase10.md (R19)
- bumps project version 1.9.0 -> 1.10.0
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr +
PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for
Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8.
- tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points
(.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143);
this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted.
- FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range
PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to
Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7.
- config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11);
stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22),
main still 143dbb89 (no regression)
- Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl)
- dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder)
- ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op)
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
(build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
(Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
(52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
- matching-cookbook §9.6 "Scaling library linking to the whole EXE": the survey-first
approach, gen_lib_subsegs.py + the section-size boundary gotcha, the combined-region
pattern for interleaved libs (make_snd_used/make_apicard_used, alias dedup by byte-match),
scattered-.bss cross-object exclusion (by address), the incremental-.ld clean-rebuild
gotcha, and the LINKED honest-metric note
- SETUP.md: add the 3 new Phase-8 tools to the tool inventory (R21)
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
pick the byte-matching object per aliased address (link_object), exclude 4 addresses
that don't reconcile in-region (kept as byte-identical stubs):
0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T — scattered-.bss commons, cross-object (§9.1)
0x3D94C S_IH/UT_RON — false placement (inside libsnd SSSTART.o)
0x3FA64 VM_F (237 ins) — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
- splat: libc2 C stdlib in 2 blocks — main 16-obj run libc2_1 (BZERO/MEMCPY/STRCMP/
PRINTF/PRNT[jtbl ok]/.../SETJMP) split from 800c, + STRCAT.o libc2_2 split from
800c2; new game-code fragments 800c3 + regenerated 800c2
- region byte-verified 17/17; PRNT.o printf-format jtbl resolves via NOLOAD .rodata
- BOUNDARY GOTCHA found + fixed: SETJMP.o .text is 0x80 B (8-align pad), not the
0x78 psyq_identify reports (30 ins) -> the libc2_1/800c3 boundary was 8 B too low,
overlapping SETJMP's padded tail; the relink inserted +8 padding and shifted the
whole downstream image (56k diff bytes, +8 file len). Fix: boundary = last obj's
readelf .text size (0x5CE18), not ins-count. Lesson recorded in docs/psyq-worklist.md
- when a library block sits at a subseg start, rm the old .c so splat regenerates it
- dual byte-gate PASS: 143dbb89 with and without the libc2 objects
- LINKED 528 -> 590; REAL still 43; byte-identical 32.25%
- splat: split 800c into [800c][libmcrd1][800c2][libmcrd2]; LIBMCRD.o (2186 ins,
the 55 LIBMCRD_OBJ_* + _card_* memcard I/O) and USERFUNC.o (68 ins) are 2
non-adjacent blocks split by game/libc2/libapi/libcard code (800c2)
- Makefile LIBMCRD_* + gated psyq_integrate libmcrd1,libmcrd2 + -T
- region byte-verified: 2/2 objects, 27 .bss commons all recovered (not scattered)
- src/800c.c trimmed (0 real-C moved); src/{libmcrd1,800c2,libmcrd2}.c committed
- dual byte-gate PASS: 143dbb89 with and without the libmcrd objects
- LINKED 424 -> 528; REAL still 43; byte-identical 29.29%
- NB: these are the libmcrd SDK objects; the game SaveLoadRoutine/Q#5 is Phase 12
- splat: split 800b2 into [800b2][libgpu][800c]; libgpu block = EXT.o+PRIM.o
(vram 0x80058890-0x80059234, holds LoadClut/LoadClut2 + primitives)
- SYS.o (3109 ins) EXCLUDED — scattered-.bss commons (cookbook §9.1, the GS_001
class): SYS references .bss by section+offset but the original linker scattered
the commons across 0x80078xxx/0x800c5xxx, so no single NOLOAD base reproduces it.
Stays a byte-identical INCLUDE_ASM stub in 800c; documented in docs/psyq-worklist.md
- curated dir .run/obj40/libgpu_used = {EXT,PRIM} (validates the _used mechanism for
the later alias libraries); Makefile LIBGPU_* + gated psyq_integrate + -T
- src/800b2.c trimmed; src/libgpu.c + src/800c.c committed (stub records); the trim's
9 "moved" items were all splat-auto empties (regenerated identically, 0 real lost)
- dual byte-gate PASS: 143dbb89 with libgpu objects AND without (stub fallback)
- LINKED 375 -> 424; REAL still 43; byte-identical 24.32%
- docs/gen2-roadmap.md: approved Gen2 plan (phases 8-15; finish-EXE-first,
two-repo LATE public flip, substantial exit) + Backup & disaster-recovery policy
- back up irreplaceable / hard-to-re-source artifacts to the private remote (R20):
- track ghidra/ (the RE database; *.lock/tmp*.ps transients excluded)
- tools/psyq/ working libs/tools (minus the two >100MB raw source archives)
- tools/bin/*.tar.gz (old-gcc cc1 compiler sources; extracted binaries stay regenerable)
- tools/ghidra-ext/ (GhidrAssistMCP 2.8.0 + ghidra_psx_ldr 2026.06.04 installer zips)
- .claude/settings.json: SessionStart + SessionEnd hooks (auto-start / clean-save the
headless Ghidra MCP); moved out of the gitignored settings.local.json so they are backed up
- docs/SETUP.md: document the MCP server lifecycle + persistence model + session hooks (§2.8),
a tooling inventory, .run/ runtime scratch (§1a), and the backup posture (§ Backup) — R21
- .gitmodules: ignore=dirty on the 4 submodules (maspsx/asm-differ/m2c/decomp-permuter) —
silences the phantom "modified" state from generated pycache + cross-filesystem filemode;
gitlinks unchanged (still at the pinned commits)
- rules R20 (back up irreplaceable work + gathered tooling; loosens R8) and R21 (keep
SETUP.md current) recorded in memory; to be formalized at the next PhaseEnd
- LzssDecodeSector (0x80018730): asm-differ score 0, make check BYTE-IDENTICAL;
NON_MATCHING guard dropped (C is the default build)
- closed the session-E ~4 regalloc/scheduling residuals via the §3a research tier
(ground-truthed vs pinned gcc-2.7.2 reorg.c/jump.c/local-alloc.c) + a floor-free
.text object metric (the permuter could not measure it: rodata/jtbl score floor)
- 5 LOAD-BEARING constructs documented in src/800.c + cookbook §10:
(A) decoupled nh high-byte var + (code&0xFF)|(nh<<8) operand order [combine_regs]
(B) result set in save predecessors [reorg fill_simple_delay_slots]
(B/state-2) explicit register $2 local + early read-only input-asm pin [sched]
(B3) no switch default -> state>=5 reuses the sltiu result
(§5a) zero-byte cross-jump barrier (retained)
- cookbook §10 added (reusable regalloc/schedule-tail idioms + the .text metric),
§5a cross-linked; CURRENT_PHASE task + green-log updated (Gen1-exit LZSS gate MET)
- tools/ghidra_scripts/ImportPsyqGdt.java: opens psyq400.gdt and resolve()s
all 2599 PsyQ 4.0 types into the program DTM (205 -> 2609), registering
psyq400 as a SourceArchive; saved to ghidra/bfm.rep
- verified over MCP: types get DRAWENV -> full 92-byte struct (/LIBGPU.H),
SVECTOR (/LIBGTE.H) -> the .gdt "attach" is doable headlessly, no GUI
- docs/SETUP.md: ledger #2 RESOLVED; §2.5 step 5 answered (import types into
the program; MCP type tools then resolve them)
Move the original starting-point brief (now bannered: it holds corrected-wrong claims re GCC 2.6.3 / LaurieWired MCP / all-.CD-LZSS / recomp-first / Windows-hybrid), the architect template, the research output, and link notes out of repo root so no session reads them as guidance before the load order.
Drew T