- fix tools/build_engine_types.py to be ADDITIVE: merge new source types with the
types already in engine_types.h (a re-run after a prior --strip was destructive —
it dropped the ~34 already-migrated types and broke the build; caught + reverted)
- engine_types.h 34 -> 42 named types (the 8 from wave-1's struct-using matches);
inline defs stripped from ov_SC01_077.c (byte-neutral)
- dedup_propagate --auto-from: 8 of 15 struct-blocked matches now self-contained ->
propagated across all 134 overlays (byte-gated); 7 remain (anonymous/typedef'd types)
- R22 CLEAN fleet rebuild: 136/136 byte-identical; dedup-check 1333/0
- zero agent tokens (deterministic)
- gen_harvest_targets.py: now also resolves canonical DATA-symbol declarations per target
(scan the .s for D_XXXX refs -> `extern <type> D_XXXX;` from banked code) in addition to
callee function sigs — the §14c(c) fix at the source, so agents declare both functions and
data correctly and avoid the dominant remaining conflict class.
- v2 harvest (callee+data-sig-aware) on the 591 small still-stub call-heavy fns + a gap-fill
agent for 6 targets dropped by a connection-closed-mid-response failure (the retry-wave
misses truncated-non-null results). Gate: 306 verified (ov_SC01_077 1554 -> 1248 stubs).
- dedup_propagate.py: skip any struct/union-touching body — agents named structs with colliding
generic names (`struct S`/`struct vec`) inline, so two macros' types redefine/conflict when
instantiated in one overlay (ov_SC01_000 abort). 34 skip; struct-using shared fns stay banked
in 077 (shared-types-header follow-up).
- 281 functions propagated fleet-wide (134 overlays each, every overlay byte-gated); dedup
795 -> 1076 groups, dedup_integrate --check 1076 validated / 0 failed.
- fleet REAL substantive 105,764 -> 143,322; byte-identical 31.22% -> 42.14%.
- R22 clean rebuild: 136/136 byte-identical, 0 extract failures.
- cookbook §12: gap-fill is now a MANDATORY post-step for every multi-agent run (reconcile
produced-vs-expected: missing + truncated drafts) before gating.
- tools/canon_draft_decls.py: rewrites a quarantined harvest draft's extern/data
declarations to the banked-consistent set (functions from engine_core.h DEFINE
bodies / extern decls / inline defs; data from existing `extern <type> D_XXXX;`),
so a draft whose BODY is byte-correct (passes match_one) but whose DECLARATION
collided with already-banked code can re-gate cleanly. The whole-binary
harvest_verify byte-gate stays the sole arbiter — a semantics-changing
canonicalization just fails and reverts.
- recovered 19 of the 90 match_one-MATCH-but-conflicting quarantined failures
(no API — immune to the rate limiting), propagated fleet-wide, all 134 overlays
byte-identical; dedup 776 -> 795 groups, 0 failed.
- fleet byte-identical 30.48% -> 31.22%; ov_SC01_077 1574 -> 1535 stubs.
- R22 clean rebuild: 136/136 byte-identical, 0 extract failures.
- the remaining ~70 match_one-MATCH residual needs definition-widening (§14c(d),
narrow-return) or true data-type resolution — queued for the next pass, with the
un-started medium band (430) and the 54 genuine instruction-misses (permuter/§3a).
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index
(fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic,
source-derived per-binary table + fleet totals). Single-binary output byte-unchanged.
- fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%
across 136 binaries (main + resident + 134 overlays)
- dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES
entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes
to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups
/ 28.6M collapsible (matches the Phase-11 figure)
- Makefile: wire progress.py --fleet into make report (main-only block)
- docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory
- make report green; dedup-check 8 validated / 0 failed
- ghidra/ churn NOT staged (R23)
- onboarded all 118 not-yet-onboarded overlays via tools/new_overlay.sh (0 failed)
-> fleet = 134 location overlays + main + resident = 136 binaries
- R22 CLEAN rebuild (make clean -> extract all 136 -> make check-all):
136 passed, 0 failed of 136 -- every overlay byte-identical at 100% INCLUDE_ASM
from a fully clean tree, 0 extract failures
- per-overlay config (splat.ov_*.yaml / check.ov_*.sha / symbols.ov_*.txt) +
config/overlays.mk var blocks + the 4 report/diff BINARIES dicts auto-registered
(sentinel-anchored, idempotent); src/ov_*/ stub records committed (harvest_verify
reverts via git checkout, so the records must be tracked before any harvest)
- ghidra/ churn NOT staged (R23); build/asm/assets/.run gitignored
- report-digest regen (fleet-wide duplicates.cross.md + progress.fleet.md) is T4
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in
src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing
that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in
config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X).
- proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all
18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test
(corrupt shared body) -> make check FAILS then restores
- Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a
shared header recompiles dependents — closes a stale-build/false-pass gap the negative
test exposed; output-neutral (R22 clean 18/18)
- structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate
can't catch under-application -> assert DEFINE present + stub gone
- FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays
(2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest
- cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory
- ghidra/ churn NOT staged (R23); build/asm/.run gitignored
- onboarded ov_{SC01_000,SC01_001,SC02_000,SC02_003,SC03_002,SC03_006,
SC04_000,SC04_018,SC04_019,SC05_000,SC06_000,SC07_000} via tools/new_overlay.sh
— spans all 7 SC areas + 2 full duplicate pairs (SC02_000≡003, SC04_018≡019)
- each byte-identical at 100% INCLUDE_ASM (non-4-aligned auto-handled); no
position-lock anomalies — the cross-shape risk is retired before the full onboard
- make check-all = 18/18 passed (main + resident + 4 prior + 12 new)
- config/overlays.mk + 4 report dicts auto-registered (sentinel, idempotent)
- ghidra/ DB churn NOT staged (R23); asm/build/assets/.run gitignored
- tools/match_one.py (new): compile ONE function's C standalone (pinned triple), mask
relocations (jal/HI16/LO16 like psyq_identify), compare to the target bytes in its .s.
Isolated temp dir -> parallelizable; a real per-function asm-differ loop for agents to
iterate against without a shared build. Validated: a matched fn -> MATCH, residuals ->
informative diffs.
- harvest_verify.py + match_one.py: strip inline scalar-typedef redefinitions (some drafts
inlined `typedef unsigned char u8;` -> C89 redefinition error vs common.h = a COMPILE
fail, not a byte mismatch). Re-gate recovered func_800CFBCC + func_800D1938.
- REAL 89 (62.76% byte-identical); resident 8e17e02f, main 143dbb89 (R22 clean).
- 54 stubs remain (scheduling/regalloc residuals + big GPU/dispatch fns) -> match_one-driven
iteration pass next.
Ultracode harvest: a 16-agent workflow (resident-harvest-draft) drafted matching C for
the 135 unmatched resident functions (each agent: asm + cookbook + the 8 worked examples,
writing .run/drafts/<fn>.c, no builds/no Ghidra). Then tools/harvest_verify.py (new) byte-
gated every draft against 8e17e02f one at a time (chunk+bisect), keeping only the byte-
perfect ones and reverting the rest -- the build is the sole arbiter (G3/P9).
- 62 / 127 drafts verified byte-identical in one pass; 65 near/hard misses reverted to stubs.
- REAL 8 -> 70 (48.28%); byte-identical 49.66%. resident 8e17e02f, main 143dbb89 (R22 clean).
- cleaned 35 redundant '#include "common.h"' lines the drafts inlined (still byte-identical).
- remaining: 65 failed drafts (close-misses + the hard dispatch/GPU/big-fn tail) + 8 undrafted
(2 agent batches hit transient API 500s) -> second-pass workflow + manual hard tail.
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) +
detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes
~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer
tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix
- Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s)
- dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed
source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals;
compact member sample + #bin column
- RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm
8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one
match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue
- dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob,
no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates)
- norm_stream(): mask address-sensitive fields so structurally-identical functions at different
addresses normalize equal — j/jal 26-bit targets, lui highs, and hi/lo-paired I-type address-los
(a consistent lui->reg pending-hi tracker); KEEP registers, true constants, PC-relative branch
offsets. h_seq = mnemonic sequence. Self-consistent within sig_image (the overlay fleet).
- validated on the resident vs the Ghidra oracle: h_exact equivalence classes 6/6 == Ghidra;
h_norm reproduces 7/9 Ghidra structural groups — the 2 disagreements are BENIGN (differ only in
masked address-low / jump targets; Ghidra's own reference analysis is inconsistent there). Any
h_norm candidate is byte-gated (its sole acceptance, per the roadmap).
- DEVIATION D2: self-consistent h_norm, NOT a Ghidra-byte-exact normToken replica. Byte-reality
makes cross-tool structural matching low-value (EXE<->resident share nothing; overlays share with
each other via h_exact at the same vram, and CALL not embed the resident). h_exact is the
format-independent cross-tool workhorse; full normToken replica is a documented future refinement.
Scope-guard pre-approved in the plan.
- tools/sig_image.py: linear rabbitizer disassembler that signs a flat image (overlay 0.4.dec /
resident 1.1) at a vram base, JSONL field-identical to DumpFunctionSignatures.java. h_exact =
SHA1 of raw bytes (format-independent — the cross-binary workhorse; overlays share the same vram
so same-offset dups are byte-identical). Seeded + jal-closure bootstrap entry discovery.
- boundary rule: a function ends at the first 'jr $ra'(+delay) at/after every forward branch/jump
target — handles early-return jr AND ignores the trailing orphan jr;nop (double-epilogue)
- VALIDATED vs the resident Ghidra oracle: 100% h_exact on the contiguous/non-GTE subset (140/140),
ZERO UNEXPLAINED (whenever nbytes agrees, h_exact agrees) — the byte pipeline is exact; 98.6%
overall. The 2 misses are non-contiguous Ghidra bodies (D5, inherent to a linear sweep)
- h_norm/h_seq emitted as conservative placeholders (= h_exact -> zero false structural matches)
until T5 calibrates the normToken/mnemonic replica; overlays aren't signed until T6 (post-T5)
- src/shared/clearTbl40.h: CLEAR_TBL40 macro = the matched byte-clear loop body, authored ONCE;
instantiated at both func_80037004 and func_80037334 in src/800.c (one source -> two vrams)
- matched first try; clean rebuild main -> 143dbb89 BYTE-IDENTICAL WITH the shared C AND WITHOUT
it (INCLUDE_ASM stub fallback) -> dual invariant proven (R22 clean rebuilds, both states)
- config/dedup.us.yaml: I0_clearTbl40 group registered (h_exact a0744d60…); dedup_integrate
--check validates it; negative tests (corrupt hash / wrong vram) fail-closed (exit 1)
- tools/progress.py: count dedup-shared members as REAL via the registry (the macro form isn't a
parseable function def); REAL 52 -> 54, byte-identical 50.24% -> 50.33%; honest measurement (P9)
- tools/dedup_integrate.py: display vram in hex in diagnostics
- the machinery half of the Phase-11 milestone is proven on the byte-verified EXE
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
current sig hash drifted from the recorded hash (a stale share can never silently
mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
one object per binary so the linker can't swap them; the byte-gate is the existing
per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
-> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row
with its binary, buckets by h_exact then h_norm across binaries, splits
cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by
collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md
- Makefile: make report emits the cross report once (gated BINARY=main)
- per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged)
- EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the
byte-finding predicted); real cross groups arrive with overlays (T6)
- phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr +
PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for
Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8.
- tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points
(.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143);
this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted.
- FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range
PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to
Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7.
- config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11);
stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22),
main still 143dbb89 (no regression)
- Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl)
- dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder)
- ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op)
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
(build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
(Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
(52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's
sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific
- Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no
rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly
- GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag);
negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded);
restore -> 143dbb89
- build_region/placement/classify (+ psyq_link_lib) take vram_base/exe, default to
the kept globals (transitional, removed T8); pass --vram-base/--exe to the
psyq_identify subprocess argv; argparse mains (window nargs=*)
- in-process callers from psyq_integrate (classify/placement) stay green via the
defaults until T5 passes them explicitly; VRAM_BASE import kept as the default
source (dropped at T8 with the rest)
- checks: link_lib libcd 18/18 byte-identical; wrong --vram-base 0x8000F900 -> 6/18
(threaded); link_region libcd per-object byte-identical True; build -> 143dbb89
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
pick the byte-matching object per aliased address (link_object), exclude 4 addresses
that don't reconcile in-region (kept as byte-identical stubs):
0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T — scattered-.bss commons, cross-object (§9.1)
0x3D94C S_IH/UT_RON — false placement (inside libsnd SSSTART.o)
0x3FA64 VM_F (237 ins) — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
- docs/gen2-roadmap.md: approved Gen2 plan (phases 8-15; finish-EXE-first,
two-repo LATE public flip, substantial exit) + Backup & disaster-recovery policy
- back up irreplaceable / hard-to-re-source artifacts to the private remote (R20):
- track ghidra/ (the RE database; *.lock/tmp*.ps transients excluded)
- tools/psyq/ working libs/tools (minus the two >100MB raw source archives)
- tools/bin/*.tar.gz (old-gcc cc1 compiler sources; extracted binaries stay regenerable)
- tools/ghidra-ext/ (GhidrAssistMCP 2.8.0 + ghidra_psx_ldr 2026.06.04 installer zips)
- .claude/settings.json: SessionStart + SessionEnd hooks (auto-start / clean-save the
headless Ghidra MCP); moved out of the gitignored settings.local.json so they are backed up
- docs/SETUP.md: document the MCP server lifecycle + persistence model + session hooks (§2.8),
a tooling inventory, .run/ runtime scratch (§1a), and the backup posture (§ Backup) — R21
- .gitmodules: ignore=dirty on the 4 submodules (maspsx/asm-differ/m2c/decomp-permuter) —
silences the phantom "modified" state from generated pycache + cross-filesystem filemode;
gitlinks unchanged (still at the pinned commits)
- rules R20 (back up irreplaceable work + gathered tooling; loosens R8) and R21 (keep
SETUP.md current) recorded in memory; to be formalized at the next PhaseEnd
- tools/ghidra_scripts/ImportPsyqGdt.java: opens psyq400.gdt and resolve()s
all 2599 PsyQ 4.0 types into the program DTM (205 -> 2609), registering
psyq400 as a SourceArchive; saved to ghidra/bfm.rep
- verified over MCP: types get DRAWENV -> full 92-byte struct (/LIBGPU.H),
SVECTOR (/LIBGTE.H) -> the .gdt "attach" is doable headlessly, no GUI
- docs/SETUP.md: ledger #2 RESOLVED; §2.5 step 5 answered (import types into
the program; MCP type tools then resolve them)
Drew T