Commit Graph

9 Commits

Author SHA1 Message Date
Drew T 757bd82a0f feat(phase-31): S79 #4 — scattered-.bss split at link-prepare (psyq_bss_split): SYS.o→libgpu2, VM_F.o→snd12, GS_001.o→libgs8 LINKED; libgpu_used retired
The §9.1 "scattered .bss commons" exclusion class (Phase 8 → P31) is closed 3/3. New
tools/psyq_bss_split.py (own ELF32 REL reader/writer) cuts an object's packed .bss into
per-base NOBITS pieces: bases derived from the game bytes per HI16/LO16 pair, references
walked in offset order into single-base runs, cuts snapped to symbol starts (the linker
scattered SYMBOLS), symbols moved, a LOCAL section symbol per piece inserted, relocs
retargeted with the addend rewritten in the immediates, self-diffed. It runs inside the one
prepare step shared by psyq_link.link_object / psyq_link_region.build_region /
psyq_integrate.integrate (prepare_object before classify), re-derived every build.

GS_001.o was certified "5 interleaved bases, NOT splittable" by the S77 probe, which grouped
by BASE; by RUN it is six symbol-aligned pieces. All seven cuts across the three objects are
confirmed by the other objects' by-name recoveries (_que 0x800C5510, _svm_sreg_buf
0x800B9B58, PSDBASEX/CLIP2/PSDBASEY/POSITION/GsDRAWENV). R39 negative control: 235 placed
objects across 9 curated dirs, 0 refusals, exactly 3 splits (a libcd .bss+size end pointer
refused the first build → reference problems are fatal only when a split is needed).

Wiring: yaml 800c→libgpu2, sgap_6→sgap_6+snd12, gsgap3→libgs8 (comments rewritten);
LIBGPU_ELF := .run/obj40/libgpu (curated libgpu_used retired); libgs 34 objs/8 blocks
(make_libgs.sh +GS_001); snd 63/12 (make_snd_used.py exclusions 4→3). src/800c.c and
src/gsgap3.c removed (Sony code hand-matched as REAL/verbatim), sgap_6.c keeps only
func_8003FA54; splat-emitted libgpu2.c/libgs8.c/snd12.c stubs for the no-SDK fallback.

Verified: main 143dbb89f34491258bbc27810d0a12ec8b43a8dd WITH the SDK objects and WITHOUT
them from a fresh extract; make tools-health OK; R22 fleet clean extract-all 212/212 +
check-all 213/213. Metrics: main REAL 886→839, LINKED 1,040→1,150, VERBATIM 85→29, stubs 29
(unchanged); game-code weighted 91.1% (40,895/44,870) — both terms lost the 3,667 SDK ins;
the remainder is still exactly the 3,975-ins open-stub sum. Verbatim manifest --update
200→33 rows (subtractive). Docs: cookbook §489 (+index), psyq-worklist rows + "S78 task #4",
SETUP S79 R21 table, decision-log S79 addendum, accelerators S79, CURRENT_PHASE S79 FINAL 🛑.
2026-09-04 17:19:29 -06:00
Drew T a85733a487 feat(phase-31): S78 #3 — 13 "game code" subsegs were PsyQ objects: wired LINKED (libgte 70/30, libgs 33/7, snd 62/11); main's game-code metric corrected to 91.8%
- exact tiles, 0 tokens: libgte23-26 (MSC01/02/05/09, SMP_00, FGO_01-06, PATCHGTE), libgte9 re-derived
  as SMP_05 NormalClip (SMP_06 NormalClipS = nested sub-pattern; psyq_integrate now drops nested
  placements), libgte27-30 (the libgs-gap MTX_05/07/11, REG03+REG11), libgs7 (2D_BG0+2D_BG1), snd10
  (VM_NO1), snd11 (VM_NOWON carved off sgap_8). LINKED 959->1040, REAL 912->886 (SDK inline-asm wrappers
  re-provenanced), VERBATIM 146->85, 13 TUs deleted; splat re-emits the stub records.
- main 143dbb89 WITH and WITHOUT the SDK objects. The no-SDK fallback had been red since S7x
  (CdReadyCallback called by its SDK name while the libcd stub carried func_800435B4) — curated
  CdReadyCallback = 0x800435B4, refs unified. R22 clean fleet 213/213; tools-health OK.
- METRIC CORRECTION (R35): progress.py's "MAIN game-code weighted" sig never excluded the LINKED
  objects (its comment said it did) — ~31k linked-SDK ins sat in the denominator as unmatched game
  code. Exclusion now derived LIVE from the Makefile stub lists + yaml ranges: 91.8% (44,562/48,537),
  not 59.8%; the 3,975-ins remainder equals the open-stub sum exactly.
- VM_F.o probed SPLITTABLE at .bss 0x50c (SYS.o's class -> task #4). cookbook §488; worklist S78 #3;
  decision-log + accelerators; SETUP rows.
2026-09-04 16:26:12 -06:00
Drew T a7394f44dc feat(phase-31): S78 #12 — the 800c3 "wall" band is LIBPAD 4.2.1 + LIBAPI 4.2: 46 names applied; integrate wired by subseg range; renames via ApplySymbols
- provenance: the psx loader's per-version PsyQ signature sets place PADENTRY/PADCMD/PADPORTD/
  PADSEQD (4.2), WAITRC2 (4.3), COUNTER/C114/FIRST/PAD/PATCH/CHCLRPAD (libapi 4.2) byte-exact in
  0x8005CE48-0x8005FC68 / 800c2 -> 12 of main's 29 stubs incl. all four §332 walls are Sony's
  DualShock library in reorder mode. 46 names -> symbols.us.txt (count 1081), band TUs, verbatim
  manifest, wave_exclude; firstfile/firstfile2 (4.2 naming); CdGetToc @0x800430B8 (was the Phase-21
  xdedup mislabel DecDCToutCallback). SETUP §5.1 corrected; psyq-worklist S78; cookbook §487;
  decision-log + accelerators S78; CHECKSUMS +Psy-Q_46.zip +PSYQ_SDevTC_v4.5.zip.
- psyq_integrate: --yaml maps stub<->objects by SUBSEG RANGE with an exact-tiling check and PRINTS
  the located-but-unwired residue (libgte: 13 objs / 1,264 ins) — main's LINKED build had been RED
  at HEAD since the S77 psyq_identify fix (22 libgte blocks merged to 3; gate worktrees take the
  stub fallback so it never showed); a library object's exported symbol whose recovered address the
  curated file names differently is --redefine-sym'd (R15; A66 firstfile->firstfile2).
- Ghidra: 47 MCP renames did NOT persist through the sentinel stop (R9 caught it) -> NEW
  tools/ghidra_scripts/ApplySymbols.java + tools/ghidra_apply_symbols.sh mirror the curated file
  headless with a real save: 73 renamed, R9-verified x4. SETUP inventory rows (R21).
- lint_symbol_refs: scans verbatim __asm__ bodies (`.ent\tfunc_X` is invisible to \b and to the
  string-masked scan); negative-controlled (red on the pre-fix TUs, green on the passing tree).
- R22: clean extract-all 212/212 + check-all green on the final config; main rebuilt byte-identical
  143dbb89 after the last src-only fix -> 213/213; tools-health OK.
2026-09-04 15:57:06 -06:00
Drew T 3ebbec9426 fix(psyq_integrate): main was RED on every incremental relink — make the externals file monotonic
THE TRUE IDENTITY OF THE LONG-STANDING 'main link defect' (2026-08-15). The extra C
function never broke the link; the RELINK it forced did.

integrate() derives each *_externals.ld from trial_undefined() against the CURRENT
ld_path, so its answer depends on how much of the linker script has ALREADY been
rewritten. On a virgin splat .ld the apicard region is still the stub object
(defining only firstfile2), so at the libmcrd stage 'firstfile' is undefined and
gets an entry. On an already-rewritten .ld, A66.o is present and defines
'firstfile' at 0x80062248, the trial no longer reports it undefined, and the entry
'firstfile = 0x80061FA8;' is DROPPED -- after which LIBMCRD's jal binds to A66.o
and main comes out 2 of 413,696 bytes different from retail (file 0x51674,
VA 0x80060E74, retail jal 0x80061FA8 vs built jal 0x80062248).

That is why main was green ONLY on the first build after a fresh extract, and it
is why NO main draft could ever bank through an incremental gate: the baseline was
already red before any draft was spliced.

integrate()'s own comment already CLAIMED this operation was idempotent ('a re-run
on an already-rewritten .ld only redoes syms'). This makes it true: the externals
map is merged with the file's prior contents, newly-derived values winning on a
name collision, names the new derivation no longer sees kept at their previous
address. The file becomes a function of the tree, not of how many times this ran.
It reports what it kept rather than doing it silently.

VERIFIED, three builds:
  fresh extract + build ...... GREEN (unchanged)
  INCREMENTAL relink ......... GREEN (was RED -- the failing case)
  third relink ............... GREEN (monotonic across repeats)
and the merge is observed firing: 'kept 6/15/2 extern(s) this re-run no longer saw
as undefined' across the integrate stages.

Root-caused by a Fable agent, verified here against the bytes.
2026-08-31 17:03:50 -06:00
Drew T 1b9397c462 refactor(phase-9): T8b — remove transitional defaults across link pipeline (no-EXE-default end state)
Every binary-specific value is now a REQUIRED parameter — the phase's #1-risk
mitigation (no EXE default an overlay could silently inherit):
- psyq_link.py: removed module EXE/VRAM_BASE; recover_sym_addrs/unique_byte_vram
  vram_base now positional-required; link_object vram_base keyword-required (*,...);
  CLI --vram-base/--exe required
- psyq_identify.py: removed EXE/VRAM_BASE globals; --vram-base/--exe required
- psyq_link_region.py + psyq_link_lib.py: dropped 'VRAM_BASE' import + EXE; defaults
  removed (placement/classify/build_region required); CLI required
- psyq_integrate.py: dropped VRAM_BASE import + EXE; integrate() vram_base/exe_path/
  symbols_path keyword-required; CLI --vram-base/--exe/--symbols required
- PROOFS: psyq_link bare -> 'required: --vram-base, --exe' (loud); build with AND
  without SDK -> 143dbb89; wrong base -> cae22f7e; curated dirs regenerate identical
2026-06-15 21:00:36 -06:00
Drew T 06676e6ab1 refactor(phase-9): T5 — psyq_integrate flags + progress regex + 9 Makefile call sites
The build-path linchpin (one commit — integrate signature and the progress.py
Makefile regex must move in lockstep):
- psyq_integrate.py: argparse (--vram-base/--exe/--symbols flags BEFORE the 4
  positionals + nargs=* window); threads vram_base/exe into placement/classify
  and symbols_path into the stub-name->addr resolution (was hardcoded symbols.us.txt)
- Makefile: all 9 psyq_integrate calls pass --vram-base $(main_VRAM_BASE) --exe
  $(main_EXE) --symbols $(main_SYMBOLS) (flags before positionals)
- progress.py: stub-list regex gains (?:\s+--\S+\s+\S+)* to consume the leading
  flags; + a --binary selector (BINARIES table, default main = current paths)
- GATES: positive build -> 143dbb89; report 52/959/7/50.24% (regex still finds the
  9 stub lists); negative main_VRAM_BASE=0x8000F804 -> cae22f7e (NOT target);
  without-SDK fresh-clone fallback -> 143dbb89
2026-06-15 20:42:32 -06:00
Drew T 49db81d8ae feat(phase-7): Gen1 close-out — matching loop industrialized, PsyQ libs linked, LZSS matched (v1.7.0)
- reports (Task 3): tools/{progress,difficulty,dup_report}.py + make report/sig-refresh
  (deterministic digests); harvest to 43 REAL matches incl. LzssDecodeSector
- per-module -O0 split (Task 5.5): src/boot.c + Makefile per-file CC1FLAGS (cookbook §6)
- PsyQ SDK linking (sessions C-G): split/build/identify/link/integrate pipeline
  (tools/psyq_*.py, make_libgs.sh, ld_interleave.py, split_src_region.py) — libcd (18 obj)
  + FULL libgs (31 obj / 6 blocks) linked BYTE-IDENTICAL from real PsyQ 4.0 libs; byte-
  identical with or without the SDK objects (stub fallback); cookbook §7/§8/§9.1-9.5
- LZSS MATCHED byte-for-byte (session F): §5a cross-jump barrier + §10 regalloc/schedule
  idioms, ground-truthed against pinned gcc-2.7.2 source (the Gen1 LZSS gate)
- file-loader cluster: 6 matched + 5 NON_MATCHING-drafted (CdReadStateMachine,
  CdReadSectorReadyCB, StreamLoadStateMachine new this session, valid under -DNON_MATCHING);
  SaveLoadRoutine DEFERRED to Q#5 (save/memcard, documented stub)
- README Gen1-complete; worklog preserved -> phase-ends/logs/Phase7.md
- rules R17 (web-research compiler quirks), R18 (plain-English session summaries),
  R19 (preserve phase worklog out of load order)
- MILESTONE: Gen1 EXIT confirmed — byte-identical build, 43 matches incl LZSS, 7 zero-
  regression sessions, loader cluster matched/drafted, overlay map emulator-proven
- bumps project version 1.6.0 -> 1.7.0
2026-06-15 13:44:32 -06:00
Drew T f91a67a6fe feat(phase-7): libgs block 6 linked byte-identical (16 objs / 26 SDK funcs) — session F
- resegment 800b -> [800b pre][libgs6][800b2 post]; psyq_integrate the 16 contiguous
  block-6 libgs objects (GS_110/119/120/121/122/123/127/131/133/134/135/137, PRESET,
  PRESET3, OBJT, OBJT3) at vram 0x80053AF8-0x80057928; make check BYTE-IDENTICAL
- fix the resegment-triggered spimdisasm boundary instability: declare GsMulCoord2/3
  (real MATRIX.o funcs, verified via the PsyQ object, R15 Ghidra-mirror pending) +
  carve the s8 data descriptor table [0x53198,data,53198] (D_80062998 was shadowed by
  a mis-detected func_80062998); ld_interleave FRONT_DATA -> 53198.data.o
- psyq_integrate multi-library support: per-lib NOLOAD section namespacing (.nl_<tag>_*)
  so the 2nd integration's NOLOAD isn't skipped (was discarding OBJT3 .rdata/.bss);
  trial-link includes sibling *_externals.ld (kills the '31 UNRESOLVED' false alarm);
  global NOLOAD re-sort by vram (kills 'dot moved backwards')
- tools/make_libgs_block6.sh: reproducible curated-dir regen (disambiguated 16 objects)
- gated on [ -d .run/obj40/libgs_block6 ]: fresh-clone stub fallback verified byte-identical
- cookbook s9.4 (multi-library integration gotchas); CURRENT_PHASE updated
- next: full libgs integration (other 5 blocks), then Task 6 close-out + Task 7 PhaseEnd
2026-06-15 10:48:08 -06:00
Drew T 3788a89b3c feat(phase-7): libcd linked into byte-identical build (58 SDK funcs) — session D checkpoint
The first real PsyQ library is sourced from real SDK objects in the byte-identical
build, replacing the libcd-region asm stubs. Tasks 2'.1/2'.2/2'.3 — the whole
PsyQ-library-into-build mechanism, proven on the simplest library (libcd).

- psyq_link.py: per-object byte-link engine — recovers externals from the EXE's
  resolved relocations (R_MIPS_26 / HI16+LO16); weakens psyq-obj-parser's
  mislabelled common-style .bss globals so a strong --defsym wins
- psyq_link_lib.py: whole-library verify — 18/18 libcd objects byte-identical
- psyq_link_region.py: region link via NOLOAD data placement (no carving) +
  weaken-all-.bss; gap-aware per-object placement
- psyq_integrate.py: build wiring — splat resegment swap + per-object .text at
  vram + NOLOAD data; external resolution (func_<addr> aliases, symbols.us.txt
  jump-table names, recovered St*/CD_* globals, all weakened .bss commons);
  idempotent; conditional on tools/psyq/ (fresh clone builds via stubs)
- split_src_region.py: H5-safe src/*.c split at vram boundaries (preserves
  matched C + NON_MATCHING blocks)
- splat.us.exe.yaml: 5-way text split [pre][libcd1][gap][libcd2][post]
- src/800.c trimmed to <0x80043088; src/{800b,gap,libcd1,libcd2}.c splat stubs
- Makefile: libcd integration in the $(OUT) recipe (conditional + idempotent)
- cookbook §9.1/9.2/9.3 (the reusable mechanism for libgs/Gen2); CURRENT_PHASE
  session-D log; ≥3-session green bar A/B/C/+D
- make clean && extract && build && check -> 143dbb89... BYTE-IDENTICAL
2026-06-14 23:13:39 -06:00