Commit Graph

62 Commits

Author SHA1 Message Date
Drew T c4b0cef86a feat(phase-17): close — canonical-sig layer built; the wall is the compiler, not sigs; pivot to gcc research (v1.16.0)
- canonical-sig layer (session 4): tools/census_conflict_callees.py + derive_canonical_sigs.py
  -> a 20-extern byte-neutral block atop ov_SC01_077.c (LOCAL, not engine_core.h); census
  conflict callees 20->0, blocked targets 24->0; gate pipeline now draft -> sig_unify (MANDATORY)
  -> harvest_verify --chunk 1; fleet 136/136 byte-identical (R22), 55.51% (no regression)
- FINDING (R14/P9): the conflict wall is 7%-reach not ~2x; the 4 reach-134 circular targets are
  ALL gcc-quirk/regalloc/layout-bound (0 banked); the high-reach core IS the quirk tail; struct
  types are byte-neutral for matching (the wall is gcc codegen, not knowledge)
- leverage analysis: fleet % is function-count-weighted (size adds no %); "unblock many" = the
  layer (declaration, not matching); reach is the lever (already reach-sorted); 247 tractable
  reach-134 stubs ~ +3-4% projected
- GO/NO-GO: NO-GO on brute waves at the current ceiling; GO on a compiler-quirk research phase
  (read gcc-2.7.2 source + Xenogears + the §10/regalloc classes, R17) -> then resume the wave
- docs: cookbook §16 corrected + hand-matching-process.md §8 (the layer + the finding + handoff)
- worklog archived -> phase-ends/logs/Phase17.md (R19); bumps 1.15.0 -> 1.16.0
2026-06-20 09:35:18 -06:00
Drew T 0dd9bf6bac feat(phase-17): calibration harvest wave — fleet 55.04%->55.51% (+0.47%)
- Ultracode wave: 30 high-reach tractable residuals, parallel guided-hand-matching
  agents (m2c + Ghidra-C cache + asm + actor-struct + S1/S2/S3a playbook), self-
  validate match_one. 18/30 match_one MATCH (60%); 10/30 whole-binary verified (33%)
  after sig_unify recovered 2 -> all 10 propagated x134; 2 demo fns caught up
- 136/136 byte-identical (R22 clean check-all); REAL +1,610; 1421 dedup groups
- KEY FINDING: the match_one->whole-binary gap (60%->33%) is 100% SIG CONFLICTS
  (shared callees like func_80131CA8 declared inconsistently across parallel agents),
  NOT codegen. The canonical-sig layer (deferred as low-value) is the ESSENTIAL
  enabler -> would lift 33% toward 60%. 12/30 are the genuine gcc-quirk tail (DIFF).
- Ghidra pre-pass: tools/ghidra_scripts/DecompileFunctions.java (headless batch
  decompile -> .run/ghidra_c cache; no /mcp needed); 300 tractable targets sized
- harness: .run/harvest_wave_s3.js (embedded-targets Workflow; args channel doesn't
  transit arrays). cost ~1.89M tokens / 30 fns
2026-06-19 20:54:45 -06:00
Drew T 1431b523a4 feat(phase-17): T4+T5 — actor struct recovered (comprehension) but byte-neutral for matching
- recovered the engine actor/player struct cold via PCSX-Redux (R10/R11): base 0x80078E00,
  ~154 fields over 0x24C bytes, 32 pointer fields; HP/BP/gold/day/hour/position/BP-drain all
  live-verified and anchored to the Phase-3 player state + the walk diff. -> docs/actor-struct.md
- tools/ram_probe.py: PCSX-Redux RAM capture/diff/read/ptr-scan workhorse for field-typing
- FINDING (the matching verdict): feeding the struct to m2c as --context is BYTE-NEUTRAL.
  m2c adopts it (arg0->f_X) but structural-miss sample scores 0 better / 10 same / 2 worse vs
  m2c-alone (full code+RAM pointer set). The misses are 50-98% mismatched = m2c decompiler
  limits, not missing types. T4 fn-ptr context unblocks compilation but functions stay structural.
- conclusion: rich context (struct/type recovery) is a comprehension / decomp.me win, NOT a
  byte-match lever -> Phase-16 wall re-confirmed from the type angle. struct-core-pivot.md updated.
- emulator field-naming pass deferred (comprehension, off the matching path); NEXT: T6 permuter
  validation on the 146 genuine near-misses, then T7 go/no-go.
2026-06-19 13:03:14 -06:00
Drew T 070cf375c9 feat(phase-17): T3 — bank 42 sig_unify matches + propagate x134 (fleet 54.48%->55.00%); fix taxonomy classifier
- banked 42 gate-proven SIG_FIXABLE_KR matches in ov_SC01_077 (of 48 predicted; 6 honestly
  reverted by the whole-binary byte-gate, P9) + propagated 13 h_exact-shared x134 via
  dedup_propagate (one body -> 134 overlays, each byte-gated, fail-closed)
- fleet byte-identical-from-source 54.48% -> 55.00% (+0.52%); REAL +1,778; dedup groups
  1395 -> 1408; make check-all 136/136 byte-identical (R22); dedup-check 1408 validated/0 failed
- wall probe (R14): raw drafts 0/99 + sig_unify fails -> the K&R/sig_unify avenue is exhausted;
  the residual conflict-class is ARITY_WALL (the §14e param-count dead-end) or structural (T4/T5)
- FIX a T1 classifier bug surfaced by verifying the wall vs the bytes: tu_compile_error captured
  the baseline's pre-existing byte-NEUTRAL warnings ("makes integer from pointer", cc1 rc=0) and
  mislabeled 99 fns LOOSE_TYPING_WALL; now keeps only non-warning errors -> there is NO loose-typing
  wall, the real residual wall is ARITY_WALL=74. docs/wall-taxonomy.md amended with the correction.
- 134 overlay .c files carry the propagated DEFINE_func instantiations; a one-time full-fleet
  make extract was required for the per-overlay propagation gate + check-all
2026-06-19 12:13:38 -06:00
Drew T edb0360d30 feat(phase-17): T1 wall taxonomy — census the 957 residuals, size every avenue (skip T2)
- tools/wall_taxonomy.py: full-census classifier (m2c --valid-syntax -> match_one ->
  sig_unify + full-overlay-TU cc1) labels every unmatched ov_SC01_077 stub by blocker
  class; reproducible (~22s, --jobs 16); read-only (throwaway _wt_*.c, auto-removed)
- docs/wall-taxonomy.md: ranked taxonomy + per-lever aggregation + sequencing recommendation
  (957 residuals, 99.2% carry an actionable label)
- findings (byte-gate-validated, R14: SIG_FIXABLE_KR = exactly the 4/4 gate-passing set):
  - T4+T5 rich context (struct/jump-table/fn-ptr-table types) = 67% reach, incl. all 30 jtbl giants
  - T6 permuter = 146 candidates (101 <=4-mismatch near-certain)
  - T3 sig_unify = 48 gate-proven immediate wins
  - loose-typing WALL = 99 (10%); non-faithful defer = 16
  - T2 DATA-symbol byte-array header = 0 functions -> SKIP (the cookbook §14c-c lever is empty)
- recommended resequence: T3 -> T4+T5 (built together) -> T6 ; drop T2 (pending Drew's P5d nod)
- struct-core-pivot.md: Phase-17 quantification note; CURRENT_PHASE.md: T1 done + resequence
- all binaries byte-identical (T1 read-only; ov_SC01_077 d19c9580 clean rebuild)
2026-06-19 11:31:41 -06:00
Drew T 9062f0fcc0 docs(phase-16): PIVOT — m2c+permuter won't crack the loose-typed core; new plan + harness fixes
- docs/struct-core-pivot.md: findings + decision + new research directions. Root cause = the
  original engine is LOOSELY TYPED (K&R; same fn called with int/ptr, arg/no-arg across sites),
  so no single canonical signature exists -> m2c guesses inconsistently, permuter can't fix
  semantics, byte-gate (correctly) rejects. Yields ~3%, not the crack. New plan: emulator-recover
  the actor struct/types -> Ghidra global type propagation -> Ghidra-C -> permuter+gate.
- harness bug-fixes (REAL, kept): p16_permute output-0-only match (killed the false '42%'),
  base.c keeps callee externs, winner_to_draft line-strip; sig_unify canonicalizes m2c's
  no-extern prototypes; gen_engine_decls.py (documents why a global canonical header breaks
  loose-typed matches).
- a few byte-gated leaf matches banked in ov_SC01_077.c.
2026-06-19 10:05:58 -06:00
Drew T ed6314f090 feat(phase-16): wire the permuter into auto_driver (the near-miss yield lever)
- driver per-fn flow: m2c -> match_one prefilter -> if near-miss & --permute-secs>0, run
  decomp-permuter (time-boxed, reuses p16_permute setup/run) -> use the score-0 winner as the
  draft -> whole-binary byte-gate. The permuter is closing near-misses well in the overnight
  batch, so it's the driver's core yield step for the unattended run.
- +--permute-j (RAM-bound on the 15GiB box).
2026-06-18 23:41:40 -06:00
Drew T 5dd42ad703 feat(phase-16): permuter batch tester + known-answer improvement-loop workhorse
- tools/p16_permute.py: measure decomp-permuter close-rate on m2c near-misses (nested
  M2C_FIELD->cast expander + permuter setup/run/parse). Overnight batch validated the
  permuter CLOSES near-misses (func_8012A988 in 34s).
- tools/p16_improve.py: known-answer-guided improvement loop (Drew's method) — graduated
  difficulty bands, runs the real pipeline, CATEGORIZES every fail into an actionable class
  (compile:NULL / stack-var / undeclared / conflicting / near-miss), reports re-derivation
  ceiling. Friday's workhorse: fix classes until dry, track the ceiling.
- src/ov_SC01_077.c: +~6 byte-gated leaf matches from the macro+sig_unify+NULL gate.
2026-06-18 23:40:17 -06:00
Drew T c8c9fd20c1 feat(phase-16): S5 driver+supervisor+safe-exit + NULL fix + sig_unify pipeline
- common.h: +NULL (byte-neutral; recovers 54/106 m2c CC1-fails that were just NULL-undeclared)
- tools/auto_driver.py: never-stop worklist loop (m2c -> sig_unify -> byte-gate -> propagate ->
  commit), STOP-sentinel safe-exit at every fn boundary, heartbeat.json
- tools/auto_supervisor.sh (pure-bash babysitter, relaunch-on-crash, reap permuters, MCP-off),
  auto_stop.sh (touch .run/auto/STOP), auto_status.sh (remote check-in)
- p16_known_answer.py: +--gate (whole-binary capability test)
- YIELD REALITY (honest, P9): macro+sig_unify whole-binary = 4/16 (25%) on KNOWN-matchable,
  ~1% on the unmatched hard tail. The permuter is the untested differentiator -> overnight test.
- CURRENT_PHASE: full night's findings + Drew's never-stop/safe-exit/graduated-ladder refinements
2026-06-18 23:15:31 -06:00
Drew T 381b0652a1 feat(phase-16): pipeline proven — known-answer ladder 67% m2c-direct re-derivation
- tools/p16_known_answer.py: graduated known-answer validation (Drew's method) —
  stub already-matched fns -> re-extract .s -> run pipeline -> measure re-derivation,
  restore safely (never git-checkout mid-harvest, §14c). Auto-picks a difficulty ladder.
- RESULT: 8/12 known-answer fns re-derived DIRECTLY by m2c+macros (no permuter, no struct
  types). Remainder = permuter/struct/typing candidates. Overlay byte-restored (d19c9580).
- Infra proven end-to-end: m2c --valid-syntax + common.h macros -> compile -> byte-gate.
- CURRENT_PHASE: Drew refinements (graduated ladder, never-stop run, safe-exit sentinel).
2026-06-18 22:53:19 -06:00
Drew T a10f7f8a1a feat(phase-15): sig-unify recovery + find_site brace-fix — fleet 52.08% -> 54.44%
- tools/sig_unify.py (NEW): unifies a draft's FULL signature set to the banked-canonical
  decls — both callee externs AND the draft's OWN definition signature (return + param
  types, body param-names preserved). Recovery for the standalone-MATCH residual: a probe
  found 30/30 sampled failures were type conflicts, 0 false-positives. Gated 191 -> 32 verified.
- tools/dedup_propagate.py find_site FIX: the inline-def detector required the opening brace
  ON THE SAME LINE as the signature, silently dropping every next-line-brace def from
  propagation (sig_unify/permuter outputs + ~stragglers all session). Now accepts brace on
  the same OR next line -> unlocked a 61-function propagation backlog.
- propagated 61 fns fleet-wide (incl. 32 sig-unify + the permuter win func_801508F8 +
  earlier-dropped next-line-brace matches), all 134 byte-identical; dedup-check 1394/0
- R22 CLEAN fleet rebuild: 136/136 byte-identical. REAL 177501 -> 185646. Zero agent tokens.
2026-06-18 20:00:50 -06:00
Drew T 4b0f4d1003 feat(phase-15): #3 types-header propagation — +8 struct-using shared fns; fleet 51.77% -> 52.08%
- fix tools/build_engine_types.py to be ADDITIVE: merge new source types with the
  types already in engine_types.h (a re-run after a prior --strip was destructive —
  it dropped the ~34 already-migrated types and broke the build; caught + reverted)
- engine_types.h 34 -> 42 named types (the 8 from wave-1's struct-using matches);
  inline defs stripped from ov_SC01_077.c (byte-neutral)
- dedup_propagate --auto-from: 8 of 15 struct-blocked matches now self-contained ->
  propagated across all 134 overlays (byte-gated); 7 remain (anonymous/typedef'd types)
- R22 CLEAN fleet rebuild: 136/136 byte-identical; dedup-check 1333/0
- zero agent tokens (deterministic)
2026-06-18 18:03:44 -06:00
Drew T 5c5e9fb973 feat(phase-15): T6 struct follow-up — shared engine_types.h unblocks 37 struct-using fns; fleet 46.27% -> 47.70%
- tools/build_engine_types.py -> src/shared/engine_types.h: extract the 34 named struct/union
  types from ov_SC01_077.c (forward decls + defs in source order; consumes trailing
  __attribute__((packed)) + aliases through the ';'). 0 same-name-different-layout collisions.
- engine_core.h includes engine_types.h; the 34 inline defs stripped from ov_SC01_077.c
  (byte-neutral — type defs emit no code; verified byte-identical).
- dedup_propagate.py: relax the struct guard to skip only inline named-struct/typedef
  DEFINITIONS (header-type USAGES + anonymous local structs now propagate); add engine_types.h
  to the compile pre-filter so struct-using bodies resolve.
- 37 struct-using shared fns propagated fleet-wide (skip 41 -> 4 overlay-local typedef'd types),
  134 overlays byte-identical; dedup 1183 -> 1220 groups, 1220 validated / 0 failed.
- fleet REAL substantive 157,541 -> 162,462; byte-identical 46.27% -> 47.70%.
- R22 clean rebuild: 136/136 byte-identical, 0 extract failures.
2026-06-17 15:25:44 -06:00
Drew T bdfb86dc4f feat(phase-15): T6 v2 — data-sig-aware harvest + struct-guarded propagate; fleet 31.22% -> 42.14%
- gen_harvest_targets.py: now also resolves canonical DATA-symbol declarations per target
  (scan the .s for D_XXXX refs -> `extern <type> D_XXXX;` from banked code) in addition to
  callee function sigs — the §14c(c) fix at the source, so agents declare both functions and
  data correctly and avoid the dominant remaining conflict class.
- v2 harvest (callee+data-sig-aware) on the 591 small still-stub call-heavy fns + a gap-fill
  agent for 6 targets dropped by a connection-closed-mid-response failure (the retry-wave
  misses truncated-non-null results). Gate: 306 verified (ov_SC01_077 1554 -> 1248 stubs).
- dedup_propagate.py: skip any struct/union-touching body — agents named structs with colliding
  generic names (`struct S`/`struct vec`) inline, so two macros' types redefine/conflict when
  instantiated in one overlay (ov_SC01_000 abort). 34 skip; struct-using shared fns stay banked
  in 077 (shared-types-header follow-up).
- 281 functions propagated fleet-wide (134 overlays each, every overlay byte-gated); dedup
  795 -> 1076 groups, dedup_integrate --check 1076 validated / 0 failed.
- fleet REAL substantive 105,764 -> 143,322; byte-identical 31.22% -> 42.14%.
- R22 clean rebuild: 136/136 byte-identical, 0 extract failures.
- cookbook §12: gap-fill is now a MANDATORY post-step for every multi-agent run (reconcile
  produced-vs-expected: missing + truncated drafts) before gating.
2026-06-17 11:59:33 -06:00
Drew T 1d94f9ec96 feat(phase-15): T6d — deterministic decl-canonicalization recovers 19 more shared fns; fleet -> 31.22%
- tools/canon_draft_decls.py: rewrites a quarantined harvest draft's extern/data
  declarations to the banked-consistent set (functions from engine_core.h DEFINE
  bodies / extern decls / inline defs; data from existing `extern <type> D_XXXX;`),
  so a draft whose BODY is byte-correct (passes match_one) but whose DECLARATION
  collided with already-banked code can re-gate cleanly. The whole-binary
  harvest_verify byte-gate stays the sole arbiter — a semantics-changing
  canonicalization just fails and reverts.
- recovered 19 of the 90 match_one-MATCH-but-conflicting quarantined failures
  (no API — immune to the rate limiting), propagated fleet-wide, all 134 overlays
  byte-identical; dedup 776 -> 795 groups, 0 failed.
- fleet byte-identical 30.48% -> 31.22%; ov_SC01_077 1574 -> 1535 stubs.
- R22 clean rebuild: 136/136 byte-identical, 0 extract failures.
- the remaining ~70 match_one-MATCH residual needs definition-widening (§14c(d),
  narrow-return) or true data-type resolution — queued for the next pass, with the
  un-started medium band (430) and the 54 genuine instruction-misses (permuter/§3a).
2026-06-17 00:30:45 -06:00
Drew T ef1cf9f05e feat(phase-15): T6 pass-1 — callee-sig-aware harvest of the call-heavy shared core; fleet 25.38% -> 30.48%
- tools/gen_harvest_targets.py: callee-signature-aware target manifest (the §14b
  extern-conflict wall-breaker). Pre-resolves each callee's EXACT signature from
  engine_core.h DEFINE bodies + extern declarations + inline defs, so drafts never
  emit a conflicting extern. Lifted call-heavy gate yield from the §14b-naive 16%
  to 60-67%.
- ov_SC01_077: 142 functions banked (1716 -> 1574 INCLUDE_ASM stubs) via the
  Ultracode parallel-draft + harvest_verify byte-gate loop (cookbook §12). The
  draft Workflow was throttled by server-side API rate limiting (stopped at
  ~246/616); the incremental gate -> dedup_propagate loop is deterministic and
  throttle-immune, so the wins banked steadily regardless.
- dedup_propagate --auto-from ov_SC01_077: ~132 functions propagated fleet-wide
  (134 overlays each), every round per-overlay byte-gated; dedup 644 -> 776 groups,
  dedup_integrate --check 776 validated / 0 failed.
- fleet REAL 85,651 -> 103,217; byte-identical 25.38% -> 30.48%.
- R22 clean rebuild: make clean + extract all 136 + check-all = 136/136 byte-
  identical, 0 extract failures.
- cookbook §14c: the wall-breaker + the 4 conflict flavors (defined / declared /
  DATA-symbol / narrow-return-widening) + the never-git-checkout-mid-harvest
  hazard + the rate-limit-resilient incremental loop. Residual = the data-symbol
  conflict class (structural canonical-decls fix queued for next pass) + the
  un-started medium band (430).
2026-06-17 00:12:23 -06:00
Drew T aefe2eac18 feat(phase-15): T5a — bulk-propagate 553 shared functions fleet-wide (3.82% -> 22.14%)
- dedup_propagate --auto-from ov_SC01_077: propagated 553 already-matched, self-contained
  shared engine functions across all 134 overlays, each byte-gated. 9 functions skipped
  (use 077-local struct types -> not mechanically liftable; honest, P9). One R22 CLEAN
  rebuild: 136/136 byte-identical.
- fleet REAL 947 -> 74,527; byte-identical 13,132 -> 76,174 / 344,010 = 22.14%; 561 dedup
  groups, dedup-check 561 validated / 0 failed
- tool hardening for fleet scale: registry SHORTHAND (vram + binaries; dedup.us.yaml ~4k
  lines vs ~77k verbose); per-overlay apply (one read/write per file); gate-only-changed;
  skip already-registered (additive/resumable); compile pre-filter (body must build with
  common.h alone); group_members() reader (both forms) in dedup_integrate + progress;
  progress _DEDUP_CACHE (fleet 6m -> 7s)
- the first bulk attempt fail-closed-reverted on a local-typed body (byte-gate working) ->
  added the compile filter; cookbook 14a documents the 5 fleet-bulk lessons
- ghidra/ churn NOT staged (R23)
2026-06-16 20:35:39 -06:00
Drew T c716eb15e1 feat(phase-15): T4 — fleet progress roll-up + fix cross-report double-count
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index
  (fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic,
  source-derived per-binary table + fleet totals). Single-binary output byte-unchanged.
- fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%
  across 136 binaries (main + resident + 134 overlays)
- dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES
  entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes
  to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups
  / 28.6M collapsible (matches the Phase-11 figure)
- Makefile: wire progress.py --fleet into make report (main-only block)
- docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory
- make report green; dedup-check 8 validated / 0 failed
- ghidra/ churn NOT staged (R23)
2026-06-16 20:00:19 -06:00
Drew T 87136a024e feat(phase-15): T3 — onboard the remaining 118 overlays; full fleet 136/136 byte-identical
- onboarded all 118 not-yet-onboarded overlays via tools/new_overlay.sh (0 failed)
  -> fleet = 134 location overlays + main + resident = 136 binaries
- R22 CLEAN rebuild (make clean -> extract all 136 -> make check-all):
  136 passed, 0 failed of 136 -- every overlay byte-identical at 100% INCLUDE_ASM
  from a fully clean tree, 0 extract failures
- per-overlay config (splat.ov_*.yaml / check.ov_*.sha / symbols.ov_*.txt) +
  config/overlays.mk var blocks + the 4 report/diff BINARIES dicts auto-registered
  (sentinel-anchored, idempotent); src/ov_*/ stub records committed (harvest_verify
  reverts via git checkout, so the records must be tracked before any harvest)
- ghidra/ churn NOT staged (R23); build/asm/assets/.run gitignored
- report-digest regen (fleet-wide duplicates.cross.md + progress.fleet.md) is T4
2026-06-16 19:41:37 -06:00
Drew T 0c7619231d feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in
  src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing
  that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in
  config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X).
- proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all
  18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test
  (corrupt shared body) -> make check FAILS then restores
- Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a
  shared header recompiles dependents — closes a stale-build/false-pass gap the negative
  test exposed; output-neutral (R22 clean 18/18)
- structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate
  can't catch under-application -> assert DEFINE present + stub gone
- FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays
  (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest
- cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory
- ghidra/ churn NOT staged (R23); build/asm/.run gitignored
2026-06-16 19:24:08 -06:00
Drew T e5a2bd4083 feat(phase-15): T1 — onboard diverse wave-1 (12 overlays); fleet 18/18 byte-identical
- onboarded ov_{SC01_000,SC01_001,SC02_000,SC02_003,SC03_002,SC03_006,
  SC04_000,SC04_018,SC04_019,SC05_000,SC06_000,SC07_000} via tools/new_overlay.sh
  — spans all 7 SC areas + 2 full duplicate pairs (SC02_000≡003, SC04_018≡019)
- each byte-identical at 100% INCLUDE_ASM (non-4-aligned auto-handled); no
  position-lock anomalies — the cross-shape risk is retired before the full onboard
- make check-all = 18/18 passed (main + resident + 4 prior + 12 new)
- config/overlays.mk + 4 report dicts auto-registered (sentinel, idempotent)
- ghidra/ DB churn NOT staged (R23); asm/build/assets/.run gitignored
2026-06-16 19:02:14 -06:00
Drew T 67592993ea feat(phase-13): T3+T4 — new_overlay.sh + dup-pair & free-roam onboarded byte-identical
- tools/new_overlay.sh: one-command overlay onboarding (template instantiate +
  @TRAILING@ carve + overlays.mk + 4-dict sentinel-insert + extract/build/check);
  idempotent (re-run = no-op, no dupe entries)
- T3: ov_SC03_001 (free-roam, 815,887 B) byte-identical f8fd92f5
- T2 (onboard): ov_SC01_005 + ov_SC01_006 (byte-identical pair) both 56760dbe
- template tokens unquoted (correct YAML int types); SETUP §2.8 MCP-reconnect note (R21)
- 6 binaries now build byte-identical (main, resident, 4 overlays)
2026-06-16 16:44:01 -06:00
Drew T 144839f24b feat(phase-13): T1a/T1b — SC01/077 onboarded + all-asm byte-match (d19c9580)
- ov_SC01_077 scaffolded: config/overlays.mk (OVERLAY_BINARIES + var block),
  config/splat.ov_SC01_077.yaml (from template), check.sha, empty symbols, +
  ov_SC01_077 entry & sentinel anchor in the 4 Python BINARIES dicts
- GATE T1b: make check BINARY=ov_SC01_077 -> d19c9580 BYTE-IDENTICAL @ 100%
  INCLUDE_ASM; clean rebuild leaves main 143dbb89 + resident 8e17e02f unregressed
- reusable non-word-aligned-overlay handling (~75% of fleet) added to the
  template + Makefile: (1) [word_floor, bin, trailing] carve for the final 1-3
  bytes spimdisasm drops; (2) build/assets/%.o incbin rule (+ .data align=1) for
  splat bin assets (asset_path scoped per-alias); (3) objcopy end-align TRIM
  (shrink-only, <=3 B) removing the .ld's segment-end ALIGN(.,4) pad
- .gitignore /assets/ (regenerable splat output); clean removes assets/
2026-06-16 15:39:52 -06:00
Drew T 4c7cf48ebf feat(phase-12): T3 — +2 (typedef-strip recovery) + per-function match tool (REAL 87->89)
- tools/match_one.py (new): compile ONE function's C standalone (pinned triple), mask
  relocations (jal/HI16/LO16 like psyq_identify), compare to the target bytes in its .s.
  Isolated temp dir -> parallelizable; a real per-function asm-differ loop for agents to
  iterate against without a shared build. Validated: a matched fn -> MATCH, residuals ->
  informative diffs.
- harvest_verify.py + match_one.py: strip inline scalar-typedef redefinitions (some drafts
  inlined `typedef unsigned char u8;` -> C89 redefinition error vs common.h = a COMPILE
  fail, not a byte mismatch). Re-gate recovered func_800CFBCC + func_800D1938.
- REAL 89 (62.76% byte-identical); resident 8e17e02f, main 143dbb89 (R22 clean).
- 54 stubs remain (scheduling/regalloc residuals + big GPU/dispatch fns) -> match_one-driven
  iteration pass next.
2026-06-16 12:43:59 -06:00
Drew T c37332fa19 feat(phase-12): T3 batch 2 — 62 resident functions matched via parallel-draft + byte-gate (REAL 8->70)
Ultracode harvest: a 16-agent workflow (resident-harvest-draft) drafted matching C for
the 135 unmatched resident functions (each agent: asm + cookbook + the 8 worked examples,
writing .run/drafts/<fn>.c, no builds/no Ghidra). Then tools/harvest_verify.py (new) byte-
gated every draft against 8e17e02f one at a time (chunk+bisect), keeping only the byte-
perfect ones and reverting the rest -- the build is the sole arbiter (G3/P9).

- 62 / 127 drafts verified byte-identical in one pass; 65 near/hard misses reverted to stubs.
- REAL 8 -> 70 (48.28%); byte-identical 49.66%. resident 8e17e02f, main 143dbb89 (R22 clean).
- cleaned 35 redundant '#include "common.h"' lines the drafts inlined (still byte-identical).
- remaining: 65 failed drafts (close-misses + the hard dispatch/GPU/big-fn tail) + 8 undrafted
  (2 agent batches hit transient API 500s) -> second-pass workflow + manual hard tail.
2026-06-16 11:59:02 -06:00
Drew T f672c709c1 feat(phase-11): T6 — sign the 134 overlays; cross-report spans 136 binaries (milestone report half)
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) +
  detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes
  ~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer
  tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix
- Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s)
- dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed
  source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals;
  compact member sample + #bin column
- RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm
  8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one
  match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue
- dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob,
  no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates)
2026-06-16 01:26:49 -06:00
Drew T 81e33c7a1e feat(phase-11): T5 — sig_image self-consistent h_norm + h_seq (scope-guard D2)
- norm_stream(): mask address-sensitive fields so structurally-identical functions at different
  addresses normalize equal — j/jal 26-bit targets, lui highs, and hi/lo-paired I-type address-los
  (a consistent lui->reg pending-hi tracker); KEEP registers, true constants, PC-relative branch
  offsets. h_seq = mnemonic sequence. Self-consistent within sig_image (the overlay fleet).
- validated on the resident vs the Ghidra oracle: h_exact equivalence classes 6/6 == Ghidra;
  h_norm reproduces 7/9 Ghidra structural groups — the 2 disagreements are BENIGN (differ only in
  masked address-low / jump targets; Ghidra's own reference analysis is inconsistent there). Any
  h_norm candidate is byte-gated (its sole acceptance, per the roadmap).
- DEVIATION D2: self-consistent h_norm, NOT a Ghidra-byte-exact normToken replica. Byte-reality
  makes cross-tool structural matching low-value (EXE<->resident share nothing; overlays share with
  each other via h_exact at the same vram, and CALL not embed the resident). h_exact is the
  format-independent cross-tool workhorse; full normToken replica is a documented future refinement.
  Scope-guard pre-approved in the plan.
2026-06-16 01:15:07 -06:00
Drew T 685e35fdc6 feat(phase-11): T4 — sig_image.py, Ghidra-free h_exact signer + boundary detection
- tools/sig_image.py: linear rabbitizer disassembler that signs a flat image (overlay 0.4.dec /
  resident 1.1) at a vram base, JSONL field-identical to DumpFunctionSignatures.java. h_exact =
  SHA1 of raw bytes (format-independent — the cross-binary workhorse; overlays share the same vram
  so same-offset dups are byte-identical). Seeded + jal-closure bootstrap entry discovery.
- boundary rule: a function ends at the first 'jr $ra'(+delay) at/after every forward branch/jump
  target — handles early-return jr AND ignores the trailing orphan jr;nop (double-epilogue)
- VALIDATED vs the resident Ghidra oracle: 100% h_exact on the contiguous/non-GTE subset (140/140),
  ZERO UNEXPLAINED (whenever nbytes agrees, h_exact agrees) — the byte pipeline is exact; 98.6%
  overall. The 2 misses are non-contiguous Ghidra bodies (D5, inherent to a linear sweep)
- h_norm/h_seq emitted as conservative placeholders (= h_exact -> zero false structural matches)
  until T5 calibrates the normToken/mnemonic replica; overlays aren't signed until T6 (post-T5)
2026-06-16 01:07:28 -06:00
Drew T a8ae190fa3 feat(phase-11): T3 — intra-binary collapse PROOF (one body, two addresses, byte-identical)
- src/shared/clearTbl40.h: CLEAR_TBL40 macro = the matched byte-clear loop body, authored ONCE;
  instantiated at both func_80037004 and func_80037334 in src/800.c (one source -> two vrams)
- matched first try; clean rebuild main -> 143dbb89 BYTE-IDENTICAL WITH the shared C AND WITHOUT
  it (INCLUDE_ASM stub fallback) -> dual invariant proven (R22 clean rebuilds, both states)
- config/dedup.us.yaml: I0_clearTbl40 group registered (h_exact a0744d60…); dedup_integrate
  --check validates it; negative tests (corrupt hash / wrong vram) fail-closed (exit 1)
- tools/progress.py: count dedup-shared members as REAL via the registry (the macro form isn't a
  parseable function def); REAL 52 -> 54, byte-identical 50.24% -> 50.33%; honest measurement (P9)
- tools/dedup_integrate.py: display vram in hex in diagnostics
- the machinery half of the Phase-11 milestone is proven on the byte-verified EXE
2026-06-16 00:56:35 -06:00
Drew T b06d74c1be feat(phase-11): T2 — dedup.us.yaml registry + dedup_integrate byte-honesty validator
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
  source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
  current sig hash drifted from the recorded hash (a stale share can never silently
  mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
  instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
  one object per binary so the linker can't swap them; the byte-gate is the existing
  per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
  -> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
2026-06-16 00:47:20 -06:00
Drew T 7cc2911456 feat(phase-11): T1 — cross-binary dup_report --cross → duplicates.cross.md
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row
  with its binary, buckets by h_exact then h_norm across binaries, splits
  cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by
  collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md
- Makefile: make report emits the cross report once (gated BINARY=main)
- per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged)
- EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the
  byte-finding predicted); real cross groups arrive with overlays (T6)
- phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log
2026-06-16 00:37:18 -06:00
Drew T 03978ec4ba feat(phase-10): T4 — resident Ghidra program (2nd binary) + symbol seed + sig
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr +
  PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for
  Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8.
- tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points
  (.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143);
  this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted.
- FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range
  PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to
  Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7.
- config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11);
  stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22),
  main still 143dbb89 (no regression)
- Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl)
- dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder)
- ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op)
2026-06-15 23:25:00 -06:00
Drew T cc73951128 feat(phase-10): T3 — per-binary reports/diff/expected for the resident binary
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
  (build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
  write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
  (Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
  baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
  (52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
2026-06-15 23:06:25 -06:00
Drew T 1b9397c462 refactor(phase-9): T8b — remove transitional defaults across link pipeline (no-EXE-default end state)
Every binary-specific value is now a REQUIRED parameter — the phase's #1-risk
mitigation (no EXE default an overlay could silently inherit):
- psyq_link.py: removed module EXE/VRAM_BASE; recover_sym_addrs/unique_byte_vram
  vram_base now positional-required; link_object vram_base keyword-required (*,...);
  CLI --vram-base/--exe required
- psyq_identify.py: removed EXE/VRAM_BASE globals; --vram-base/--exe required
- psyq_link_region.py + psyq_link_lib.py: dropped 'VRAM_BASE' import + EXE; defaults
  removed (placement/classify/build_region required); CLI required
- psyq_integrate.py: dropped VRAM_BASE import + EXE; integrate() vram_base/exe_path/
  symbols_path keyword-required; CLI --vram-base/--exe/--symbols required
- PROOFS: psyq_link bare -> 'required: --vram-base, --exe' (loud); build with AND
  without SDK -> 143dbb89; wrong base -> cae22f7e; curated dirs regenerate identical
2026-06-15 21:00:36 -06:00
Drew T cb7a280994 refactor(phase-9): T8a — de-default the EXE-curation helpers (thread --exe/--vram-base)
- make_snd_used.py + make_apicard_used.py: argparse --exe/--vram-base (default the
  EXE's); thread into psyq_identify subprocess + link_object (required post-T8b).
  RLO/RHI/EXCLUDE_ADDR kept as the EXE's region data (EXE-specific, documented)
- gen_lib_subsegs.py: --vram-base/--exe (default EXE); threaded into psyq_identify
  and the pos->fileoff arithmetic (module VRAM_BASE constant removed)
- split_src_region.py: --symbols overrides SYMS_PATH (default the EXE's symbols.us.txt)
- checks: snd_used (60) + apicard_used (22) regenerate BYTE-IDENTICAL; gen_lib_subsegs
  libgte -> 22 blocks/53 objs; clean build -> 143dbb89
2026-06-15 20:55:28 -06:00
Drew T 5e0d80a887 refactor(phase-9): T7 — ld_interleave.py --front/--tail; gate extract call to BINARY=main
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's
  sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific
- Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no
  rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly
- GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag);
  negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded);
  restore -> 143dbb89
2026-06-15 20:49:45 -06:00
Drew T 22b20e8f31 refactor(phase-9): T6 — --binary selector for dup_report + difficulty; report threads BINARY
- dup_report.py + difficulty.py: argparse --binary (default main) + BINARIES table
  (main = current sig/src/asm/md/csv paths); overlay src/asm subtree deferred to P10
- Makefile report target threads --binary $(BINARY) to progress/difficulty/dup_report
- no-op proof: docs/progress.md + docs/duplicates.md UNCHANGED; difficulty.py old-code
  vs new-code output IDENTICAL (pure path-table refactor)
- docs/difficulty.md regenerated (the Phase-8-deferred regen: 9 harvested funcs left
  the unmatched queue — 12/12 row shift); build path untouched (still 143dbb89)
2026-06-15 20:46:48 -06:00
Drew T 06676e6ab1 refactor(phase-9): T5 — psyq_integrate flags + progress regex + 9 Makefile call sites
The build-path linchpin (one commit — integrate signature and the progress.py
Makefile regex must move in lockstep):
- psyq_integrate.py: argparse (--vram-base/--exe/--symbols flags BEFORE the 4
  positionals + nargs=* window); threads vram_base/exe into placement/classify
  and symbols_path into the stub-name->addr resolution (was hardcoded symbols.us.txt)
- Makefile: all 9 psyq_integrate calls pass --vram-base $(main_VRAM_BASE) --exe
  $(main_EXE) --symbols $(main_SYMBOLS) (flags before positionals)
- progress.py: stub-list regex gains (?:\s+--\S+\s+\S+)* to consume the leading
  flags; + a --binary selector (BINARIES table, default main = current paths)
- GATES: positive build -> 143dbb89; report 52/959/7/50.24% (regex still finds the
  9 stub lists); negative main_VRAM_BASE=0x8000F804 -> cae22f7e (NOT target);
  without-SDK fresh-clone fallback -> 143dbb89
2026-06-15 20:42:32 -06:00
Drew T 0a59e96ce4 refactor(phase-9): T4 — thread vram_base/exe through psyq_link_region + psyq_link_lib
- build_region/placement/classify (+ psyq_link_lib) take vram_base/exe, default to
  the kept globals (transitional, removed T8); pass --vram-base/--exe to the
  psyq_identify subprocess argv; argparse mains (window nargs=*)
- in-process callers from psyq_integrate (classify/placement) stay green via the
  defaults until T5 passes them explicitly; VRAM_BASE import kept as the default
  source (dropped at T8 with the rest)
- checks: link_lib libcd 18/18 byte-identical; wrong --vram-base 0x8000F900 -> 6/18
  (threaded); link_region libcd per-object byte-identical True; build -> 143dbb89
2026-06-15 20:36:32 -06:00
Drew T 4737d3ea4e refactor(phase-9): T3 — argparse --vram-base/--exe for psyq_identify.py
- migrate to argparse: elf_dir + optional [lo hi] window positionals + --vram-base
  /--exe flags defaulting to the kept EXE globals (TRANSITIONAL, removed T8)
- caller subprocess argv updates deferred to T4 (link_region/link_lib) and T8
  (make_snd/apicard/gen_lib_subsegs) — one touch per file; defaults keep them green
- negative control: narrow window 0x50000..0x60000 -> 0/25 located (window threaded);
  default + explicit -> 18/25; clean build -> 143dbb89 BYTE-IDENTICAL
2026-06-15 20:32:53 -06:00
Drew T d9517a013b refactor(phase-9): T2 — thread vram_base/exe through psyq_link.py
- recover_sym_addrs/unique_byte_vram/link_object take vram_base (+ link_object
  exe_path), defaulting to the kept EXE module globals (TRANSITIONAL — removed in
  T8 at the no-EXE-default end state, once every caller passes explicitly)
- argparse main() with --vram-base/--exe (in-process importers unaffected: the
  globals + defaults keep psyq_integrate/link_region/make_* green)
- negative control: TOC.o --vram-base 0x8000F900 -> FAIL (.rdata shifts +0x100),
  0x8000F800 -> PASS, bare defaults -> PASS; clean build -> 143dbb89 BYTE-IDENTICAL
2026-06-15 20:30:19 -06:00
Drew T 11a9bdc032 feat(phase-8): link libapi+libcard combined 800c2 region (T7+T8) — 22 objs / +24 fns
- libapi (BIOS syscall trampolines) + libcard interleave in 0x61F38..0x62888; linked as
  one combined region (4 blocks apicard1..4) via tools/make_apicard_used.py (C112.o shared,
  byte-match dedup, 0 exclusions)
- subsegs via gen_lib_subsegs.py; integrate window 0x61F38..0x62888; region verified 22/22
- rm'd src/800c2.c (region starts with a lib block) -> splat regenerates apicard1..4 + frags
- dual byte-gate PASS: 143dbb89 with and without the apicard objects
- LINKED 935 -> 959 (+24); REAL still 43; byte-identical 49.81%
- libapi's ~22 objects in the 800c3 region DEFERRED (lowest value); SSGM + the scattered-.bss
  exclusions documented in docs/psyq-worklist.md

LIBRARY CEILING: 8 PsyQ libraries linked byte-identical (959 LINKED fns); byte-identical
coverage 20.31% (libcd+libgs only) -> 49.81%. Remaining = documented low-value deferrals.
2026-06-15 18:59:14 -06:00
Drew T 8bbf2e8e4d feat(phase-8): link libspu+libsnd combined sound region (T9+T10) — 60 objs / +225 fns
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
  as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
  pick the byte-matching object per aliased address (link_object), exclude 4 addresses
  that don't reconcile in-region (kept as byte-identical stubs):
    0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T  — scattered-.bss commons, cross-object (§9.1)
    0x3D94C S_IH/UT_RON                      — false placement (inside libsnd SSSTART.o)
    0x3FA64 VM_F (237 ins)                   — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
  integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
  game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
2026-06-15 18:54:21 -06:00
Drew T c4a627b22b feat(phase-8): link libgte byte-identical (T11, value-first) — 53/58 objs / +120 fns
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
  multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
  gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
  interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
  0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
  stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
  multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
  with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
2026-06-15 18:43:27 -06:00
Drew T c27a4432eb feat(phase-8): library survey + progress.py LINKED category (T1-T2)
- T1: docs/psyq-worklist.md — byte-confirmed PsyQ footprint map (psyq_identify
  over the original EXE): 9 footprint libs (199 placements) ranked ascending
  difficulty; 3 skip (libmath/libc/libsn = 0 footprint). Findings: the libgs
  "gaps" are libgte objects; heavy game/library interleaving (800=game+libspu+
  libsnd+libetc, 800b~libgte, 800b2=game+libgpu/libc2/libapi/libmcrd/libcard);
  cross-library aliases (C112, UT_RON/S_IH, S_R/S_W)
- T2: tools/progress.py LINKED category — INCLUDE_ASM stubs in integrated library
  subsegs counted as a distinct byte-identical bucket; linked-subseg set sourced
  from the Makefile's psyq_integrate calls (single source of truth -> auto-updates
  per integration, no manifest drift). libcd+libgs: 340 stubs -> LINKED;
  byte-identical/matchable now 425/2093 = 20.31%. Build unchanged (143dbb89)
- phase-ends/CURRENT_PHASE.md: Phase 8 plan + per-task log
2026-06-15 18:00:56 -06:00
Drew T 6f248462dd chore(gen2-prep): Gen2 roadmap + private-repo backup hardening + SETUP/submodule fixes
- docs/gen2-roadmap.md: approved Gen2 plan (phases 8-15; finish-EXE-first,
  two-repo LATE public flip, substantial exit) + Backup & disaster-recovery policy
- back up irreplaceable / hard-to-re-source artifacts to the private remote (R20):
  - track ghidra/ (the RE database; *.lock/tmp*.ps transients excluded)
  - tools/psyq/ working libs/tools (minus the two >100MB raw source archives)
  - tools/bin/*.tar.gz (old-gcc cc1 compiler sources; extracted binaries stay regenerable)
  - tools/ghidra-ext/ (GhidrAssistMCP 2.8.0 + ghidra_psx_ldr 2026.06.04 installer zips)
- .claude/settings.json: SessionStart + SessionEnd hooks (auto-start / clean-save the
  headless Ghidra MCP); moved out of the gitignored settings.local.json so they are backed up
- docs/SETUP.md: document the MCP server lifecycle + persistence model + session hooks (§2.8),
  a tooling inventory, .run/ runtime scratch (§1a), and the backup posture (§ Backup) — R21
- .gitmodules: ignore=dirty on the 4 submodules (maspsx/asm-differ/m2c/decomp-permuter) —
  silences the phantom "modified" state from generated pycache + cross-filesystem filemode;
  gitlinks unchanged (still at the pinned commits)
- rules R20 (back up irreplaceable work + gathered tooling; loosens R8) and R21 (keep
  SETUP.md current) recorded in memory; to be formalized at the next PhaseEnd
2026-06-15 17:15:31 -06:00
Drew T 49db81d8ae feat(phase-7): Gen1 close-out — matching loop industrialized, PsyQ libs linked, LZSS matched (v1.7.0)
- reports (Task 3): tools/{progress,difficulty,dup_report}.py + make report/sig-refresh
  (deterministic digests); harvest to 43 REAL matches incl. LzssDecodeSector
- per-module -O0 split (Task 5.5): src/boot.c + Makefile per-file CC1FLAGS (cookbook §6)
- PsyQ SDK linking (sessions C-G): split/build/identify/link/integrate pipeline
  (tools/psyq_*.py, make_libgs.sh, ld_interleave.py, split_src_region.py) — libcd (18 obj)
  + FULL libgs (31 obj / 6 blocks) linked BYTE-IDENTICAL from real PsyQ 4.0 libs; byte-
  identical with or without the SDK objects (stub fallback); cookbook §7/§8/§9.1-9.5
- LZSS MATCHED byte-for-byte (session F): §5a cross-jump barrier + §10 regalloc/schedule
  idioms, ground-truthed against pinned gcc-2.7.2 source (the Gen1 LZSS gate)
- file-loader cluster: 6 matched + 5 NON_MATCHING-drafted (CdReadStateMachine,
  CdReadSectorReadyCB, StreamLoadStateMachine new this session, valid under -DNON_MATCHING);
  SaveLoadRoutine DEFERRED to Q#5 (save/memcard, documented stub)
- README Gen1-complete; worklog preserved -> phase-ends/logs/Phase7.md
- rules R17 (web-research compiler quirks), R18 (plain-English session summaries),
  R19 (preserve phase worklog out of load order)
- MILESTONE: Gen1 EXIT confirmed — byte-identical build, 43 matches incl LZSS, 7 zero-
  regression sessions, loader cluster matched/drafted, overlay map emulator-proven
- bumps project version 1.6.0 -> 1.7.0
2026-06-15 13:44:32 -06:00
Drew T f91a67a6fe feat(phase-7): libgs block 6 linked byte-identical (16 objs / 26 SDK funcs) — session F
- resegment 800b -> [800b pre][libgs6][800b2 post]; psyq_integrate the 16 contiguous
  block-6 libgs objects (GS_110/119/120/121/122/123/127/131/133/134/135/137, PRESET,
  PRESET3, OBJT, OBJT3) at vram 0x80053AF8-0x80057928; make check BYTE-IDENTICAL
- fix the resegment-triggered spimdisasm boundary instability: declare GsMulCoord2/3
  (real MATRIX.o funcs, verified via the PsyQ object, R15 Ghidra-mirror pending) +
  carve the s8 data descriptor table [0x53198,data,53198] (D_80062998 was shadowed by
  a mis-detected func_80062998); ld_interleave FRONT_DATA -> 53198.data.o
- psyq_integrate multi-library support: per-lib NOLOAD section namespacing (.nl_<tag>_*)
  so the 2nd integration's NOLOAD isn't skipped (was discarding OBJT3 .rdata/.bss);
  trial-link includes sibling *_externals.ld (kills the '31 UNRESOLVED' false alarm);
  global NOLOAD re-sort by vram (kills 'dot moved backwards')
- tools/make_libgs_block6.sh: reproducible curated-dir regen (disambiguated 16 objects)
- gated on [ -d .run/obj40/libgs_block6 ]: fresh-clone stub fallback verified byte-identical
- cookbook s9.4 (multi-library integration gotchas); CURRENT_PHASE updated
- next: full libgs integration (other 5 blocks), then Task 6 close-out + Task 7 PhaseEnd
2026-06-15 10:48:08 -06:00
Drew T 038e7de532 feat(phase-7): LZSS cross-jump barrier breakthrough + libgs byte-verified — session E checkpoint
- LZSS: defeated gcc 2.7.2 -O2 cross-jump-merge of the twin state-save tails with a
  zero-byte __asm__ __volatile__("" ::: "memory") barrier (find_cross_jump bails on
  ASM_INPUT; ground-truthed vs gcc-2.7.2.3 jump.c by a web-research subagent).
  LzssDecodeSector now 122 instructions (correct structure); ~3 regalloc/scheduling
  swaps remain -> C kept under #ifdef NON_MATCHING, default build = stub = BYTE-IDENTICAL
- surgical rodata carve (lean LZSS path, no libgs needed): splat carves ONLY
  jtbl_80072A38 ([0x63238,.rodata,800] bounded by [0x6324C,data,6324C]); ld_interleave
  wired into `make extract` as the .data->.rodata->.data sandwich (TAIL_DATA=6324C.data.o)
- libgs: 31/32 used objects byte-verified IDENTICAL to real PsyQ libgs 4.0 (0 conflicts,
  69 externals); GS_001 deferred (psyq-obj-parser .bss-common scattering); 6-block
  resegmentation wiring pending. psyq_identify.py now skips data-only objects (GLOBAL.o)
- cookbook: §3a (web-research compiler internals — escalation tier above the permuter)
  + §5a (the cross-jump barrier idiom) — both reusable
- build BYTE-IDENTICAL (143dbb89f34491258bbc27810d0a12ec8b43a8dd) throughout
2026-06-15 01:47:18 -06:00
Drew T 3788a89b3c feat(phase-7): libcd linked into byte-identical build (58 SDK funcs) — session D checkpoint
The first real PsyQ library is sourced from real SDK objects in the byte-identical
build, replacing the libcd-region asm stubs. Tasks 2'.1/2'.2/2'.3 — the whole
PsyQ-library-into-build mechanism, proven on the simplest library (libcd).

- psyq_link.py: per-object byte-link engine — recovers externals from the EXE's
  resolved relocations (R_MIPS_26 / HI16+LO16); weakens psyq-obj-parser's
  mislabelled common-style .bss globals so a strong --defsym wins
- psyq_link_lib.py: whole-library verify — 18/18 libcd objects byte-identical
- psyq_link_region.py: region link via NOLOAD data placement (no carving) +
  weaken-all-.bss; gap-aware per-object placement
- psyq_integrate.py: build wiring — splat resegment swap + per-object .text at
  vram + NOLOAD data; external resolution (func_<addr> aliases, symbols.us.txt
  jump-table names, recovered St*/CD_* globals, all weakened .bss commons);
  idempotent; conditional on tools/psyq/ (fresh clone builds via stubs)
- split_src_region.py: H5-safe src/*.c split at vram boundaries (preserves
  matched C + NON_MATCHING blocks)
- splat.us.exe.yaml: 5-way text split [pre][libcd1][gap][libcd2][post]
- src/800.c trimmed to <0x80043088; src/{800b,gap,libcd1,libcd2}.c splat stubs
- Makefile: libcd integration in the $(OUT) recipe (conditional + idempotent)
- cookbook §9.1/9.2/9.3 (the reusable mechanism for libgs/Gen2); CURRENT_PHASE
  session-D log; ≥3-session green bar A/B/C/+D
- make clean && extract && build && check -> 143dbb89... BYTE-IDENTICAL
2026-06-14 23:13:39 -06:00