- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index
(fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic,
source-derived per-binary table + fleet totals). Single-binary output byte-unchanged.
- fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%
across 136 binaries (main + resident + 134 overlays)
- dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES
entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes
to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups
/ 28.6M collapsible (matches the Phase-11 figure)
- Makefile: wire progress.py --fleet into make report (main-only block)
- docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory
- make report green; dedup-check 8 validated / 0 failed
- ghidra/ churn NOT staged (R23)
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in
src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing
that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in
config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X).
- proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all
18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test
(corrupt shared body) -> make check FAILS then restores
- Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a
shared header recompiles dependents — closes a stale-build/false-pass gap the negative
test exposed; output-neutral (R22 clean 18/18)
- structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate
can't catch under-application -> assert DEFINE present + stub gone
- FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays
(2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest
- cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory
- ghidra/ churn NOT staged (R23); build/asm/.run gitignored
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) +
detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes
~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer
tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix
- Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s)
- dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed
source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals;
compact member sample + #bin column
- RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm
8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one
match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue
- dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob,
no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates)
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
current sig hash drifted from the recorded hash (a stale share can never silently
mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
one object per binary so the linker can't swap them; the byte-gate is the existing
per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
-> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row
with its binary, buckets by h_exact then h_norm across binaries, splits
cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by
collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md
- Makefile: make report emits the cross report once (gated BINARY=main)
- per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged)
- EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the
byte-finding predicted); real cross groups arrive with overlays (T6)
- phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr +
PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for
Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8.
- tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points
(.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143);
this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted.
- FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range
PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to
Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7.
- config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11);
stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22),
main still 143dbb89 (no regression)
- Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl)
- dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder)
- ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op)
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
(build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
(Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
(52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
- Makefile: BINARIES += resident; resident_* var block (vram 0x800CEDF8, build/resident/,
config/*.resident.*); per-binary ASM_DIR/SRC_DIR + $(BINARIES)-derived OBJS prune-list so
main (asm/+src/) and resident (asm/resident/+src/resident/) object sets stay disjoint;
per-binary UNDEF_SYMS/UNDEF_FUNCS (resident writes under build/resident/)
- config/splat.resident.yaml: flat blob config (NO header, NO gp_value, single c seg @
vram 0x800CEDF8, stacked symbols [symbols.us.txt, symbols.resident.txt], per-binary
asm/src/build/undefined paths, EOF 0x5935C -> end vram 0x80128154)
- config/check.resident.sha (8e17e02f… = extracted MAIN.CD/FILE_010/1.1, 365,404 B, type-1)
- config/symbols.resident.txt (R13/R15 provenance banner; seeded empty, populated in T4)
- gates: (a) main clean rebuild 143dbb89… (refactor is a no-op on the byte-locked EXE);
(b) make extract BINARY=resident -> 144 INCLUDE_ASM stubs + .ld @0x800CEDF8;
(c) main OBJS=81 excludes resident even with resident on disk + main stays 143dbb89…
- src/resident/*.c left WIP-untracked (reshaped by T2 boundary carving)
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's
sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific
- Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no
rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly
- GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag);
negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded);
restore -> 143dbb89
- BINARIES/BINARY guard + main_* var set + selected-binary aliases; the EXE's
artifact paths (build/us/, config/splat.us.exe.yaml, check.us.sha) preserved
verbatim so its rebuild stays a byte-exact no-op
- 9 PsyQ SDK-integration blocks wrapped in ifeq(BINARY,main); SYMS/LD/OBJCOPY
stay outside (run for every binary); a 2nd binary skips the SDK block
- gate: make clean && extract && build -> 143dbb89 BYTE-IDENTICAL; make report
52/959/7/50.24%; BINARY=bogus errors via the $(filter) guard
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
pick the byte-matching object per aliased address (link_object), exclude 4 addresses
that don't reconcile in-region (kept as byte-identical stubs):
0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T — scattered-.bss commons, cross-object (§9.1)
0x3D94C S_IH/UT_RON — false placement (inside libsnd SSSTART.o)
0x3FA64 VM_F (237 ins) — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
- splat: libc2 C stdlib in 2 blocks — main 16-obj run libc2_1 (BZERO/MEMCPY/STRCMP/
PRINTF/PRNT[jtbl ok]/.../SETJMP) split from 800c, + STRCAT.o libc2_2 split from
800c2; new game-code fragments 800c3 + regenerated 800c2
- region byte-verified 17/17; PRNT.o printf-format jtbl resolves via NOLOAD .rodata
- BOUNDARY GOTCHA found + fixed: SETJMP.o .text is 0x80 B (8-align pad), not the
0x78 psyq_identify reports (30 ins) -> the libc2_1/800c3 boundary was 8 B too low,
overlapping SETJMP's padded tail; the relink inserted +8 padding and shifted the
whole downstream image (56k diff bytes, +8 file len). Fix: boundary = last obj's
readelf .text size (0x5CE18), not ins-count. Lesson recorded in docs/psyq-worklist.md
- when a library block sits at a subseg start, rm the old .c so splat regenerates it
- dual byte-gate PASS: 143dbb89 with and without the libc2 objects
- LINKED 528 -> 590; REAL still 43; byte-identical 32.25%
- splat: split 800c into [800c][libmcrd1][800c2][libmcrd2]; LIBMCRD.o (2186 ins,
the 55 LIBMCRD_OBJ_* + _card_* memcard I/O) and USERFUNC.o (68 ins) are 2
non-adjacent blocks split by game/libc2/libapi/libcard code (800c2)
- Makefile LIBMCRD_* + gated psyq_integrate libmcrd1,libmcrd2 + -T
- region byte-verified: 2/2 objects, 27 .bss commons all recovered (not scattered)
- src/800c.c trimmed (0 real-C moved); src/{libmcrd1,800c2,libmcrd2}.c committed
- dual byte-gate PASS: 143dbb89 with and without the libmcrd objects
- LINKED 424 -> 528; REAL still 43; byte-identical 29.29%
- NB: these are the libmcrd SDK objects; the game SaveLoadRoutine/Q#5 is Phase 12
- splat: split 800b2 into [800b2][libgpu][800c]; libgpu block = EXT.o+PRIM.o
(vram 0x80058890-0x80059234, holds LoadClut/LoadClut2 + primitives)
- SYS.o (3109 ins) EXCLUDED — scattered-.bss commons (cookbook §9.1, the GS_001
class): SYS references .bss by section+offset but the original linker scattered
the commons across 0x80078xxx/0x800c5xxx, so no single NOLOAD base reproduces it.
Stays a byte-identical INCLUDE_ASM stub in 800c; documented in docs/psyq-worklist.md
- curated dir .run/obj40/libgpu_used = {EXT,PRIM} (validates the _used mechanism for
the later alias libraries); Makefile LIBGPU_* + gated psyq_integrate + -T
- src/800b2.c trimmed; src/libgpu.c + src/800c.c committed (stub records); the trim's
9 "moved" items were all splat-auto empties (regenerated identically, 0 real lost)
- dual byte-gate PASS: 143dbb89 with libgpu objects AND without (stub fallback)
- LINKED 375 -> 424; REAL still 43; byte-identical 24.32%
Drew T