mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-01 07:40:42 -04:00
feat(phase-7): libcd linked into byte-identical build (58 SDK funcs) — session D checkpoint
The first real PsyQ library is sourced from real SDK objects in the byte-identical
build, replacing the libcd-region asm stubs. Tasks 2'.1/2'.2/2'.3 — the whole
PsyQ-library-into-build mechanism, proven on the simplest library (libcd).
- psyq_link.py: per-object byte-link engine — recovers externals from the EXE's
resolved relocations (R_MIPS_26 / HI16+LO16); weakens psyq-obj-parser's
mislabelled common-style .bss globals so a strong --defsym wins
- psyq_link_lib.py: whole-library verify — 18/18 libcd objects byte-identical
- psyq_link_region.py: region link via NOLOAD data placement (no carving) +
weaken-all-.bss; gap-aware per-object placement
- psyq_integrate.py: build wiring — splat resegment swap + per-object .text at
vram + NOLOAD data; external resolution (func_<addr> aliases, symbols.us.txt
jump-table names, recovered St*/CD_* globals, all weakened .bss commons);
idempotent; conditional on tools/psyq/ (fresh clone builds via stubs)
- split_src_region.py: H5-safe src/*.c split at vram boundaries (preserves
matched C + NON_MATCHING blocks)
- splat.us.exe.yaml: 5-way text split [pre][libcd1][gap][libcd2][post]
- src/800.c trimmed to <0x80043088; src/{800b,gap,libcd1,libcd2}.c splat stubs
- Makefile: libcd integration in the $(OUT) recipe (conditional + idempotent)
- cookbook §9.1/9.2/9.3 (the reusable mechanism for libgs/Gen2); CURRENT_PHASE
session-D log; ≥3-session green bar A/B/C/+D
- make clean && extract && build && check -> 143dbb89... BYTE-IDENTICAL
This commit is contained in:
Drew T
@@ -157,6 +157,15 @@ CHECK_SHA := config/check.us.sha
|
||||
UNDEF_SYMS := undefined_syms_auto.txt
|
||||
UNDEF_FUNCS := undefined_funcs_auto.txt
|
||||
|
||||
# Phase 7 (Task 2'): link the real PsyQ libcd SDK objects in place of the libcd-region asm stubs.
|
||||
# tools/psyq_integrate.py rewrites the splat .ld (swap stub objects -> build/psyq/libcd/*.o + NOLOAD
|
||||
# data placement, no carving) and emits the externals defsym fragment. Conditional on the SDK ELF
|
||||
# objects being present (gitignored, SDK-derived, via tools/psyq_build_libs.sh LIBCD); a fresh clone
|
||||
# without them builds byte-identically via the stubs.
|
||||
LIBCD_ELF := .run/obj40/libcd
|
||||
LIBCD_OBJDIR := build/psyq/libcd
|
||||
LIBCD_SYMS := build/psyq/libcd_externals.ld
|
||||
|
||||
# Assembler flags (docs/SETUP.md §6.2). -G0 is confirmed by the disassembly
|
||||
# (ledger #8: zero $gp-relative addressing). -no-pad-sections keeps section ends
|
||||
# un-padded so the link reproduces the original layout.
|
||||
@@ -220,8 +229,16 @@ build/src/boot.o: CC1FLAGS := -quiet -O0 -G0 -mips1 -mcpu=3000 -mgas -msoft-floa
|
||||
$(OUT): $(OBJS) $(LD_SCRIPT)
|
||||
@set -e
|
||||
mkdir -p $(dir $@)
|
||||
# Wire in the real libcd objects (after the build objects exist — the externals discovery
|
||||
# trial-links the whole image). Idempotent: re-running re-derives the externals only.
|
||||
if [ -d "$(LIBCD_ELF)" ]; then
|
||||
$(PYTHON) tools/psyq_integrate.py $(LIBCD_ELF) $(LD_SCRIPT) $(LIBCD_OBJDIR) $(LIBCD_SYMS) libcd1,libcd2
|
||||
else
|
||||
echo " (no $(LIBCD_ELF) — libcd region stays asm stubs; run tools/psyq_build_libs.sh LIBCD)"
|
||||
fi
|
||||
SYMS=""; [ -f "$(LIBCD_SYMS)" ] && SYMS="-T $(LIBCD_SYMS)"
|
||||
echo " LD $(ELF)"
|
||||
$(LD) -T $(LD_SCRIPT) -T $(UNDEF_SYMS) -T $(UNDEF_FUNCS) --no-check-sections -Map $(MAPFILE) -o $(ELF)
|
||||
$(LD) -T $(LD_SCRIPT) -T $(UNDEF_SYMS) -T $(UNDEF_FUNCS) $$SYMS --no-check-sections -Map $(MAPFILE) -o $(ELF)
|
||||
echo " OBJCOPY $@"
|
||||
$(OBJCOPY) -O binary $(ELF) $@
|
||||
|
||||
|
||||
@@ -73,6 +73,13 @@ segments:
|
||||
# INCLUDE_ASM (Phase-7 regression gate) — opt level only affects matched C, not stubs.
|
||||
# Boundary = func_800123F0 (vram 0x800123F0 -> file vram-0x8000F800 = 0x2BF0).
|
||||
- [0x800, c, boot] # -O0 boot module -> src/boot.c (vram 0x80010000-0x800123F0)
|
||||
- [0x2BF0, c, 800] # -O2 remainder -> src/800.c (vram 0x800123F0-0x800629DC); name "800" kept (legacy) so the 38 matched fns + their asm/nonmatchings/800 paths don't migrate; NB start is now 0x2BF0, not 0x800
|
||||
- [0x2BF0, c, 800] # -O2 game code -> src/800.c (vram 0x800123F0-0x80043088); name "800" kept (legacy) so the matched fns + their asm/nonmatchings/800 paths don't migrate
|
||||
# PsyQ libcd region (Phase 7 Task 2'): the real libcd objects are linked here in place of
|
||||
# stubs (the .ld swaps build/src/libcd{1,2}.o -> build/psyq/libcd/*.o + NOLOAD data). The
|
||||
# 18 objects form two contiguous blocks split by a 76-B non-libcd gap (stays a stub).
|
||||
- [0x33888, c, libcd1] # libcd block 1 -> src/libcd1.c (vram 0x80043088-0x80046980, 11 objs)
|
||||
- [0x37180, c, gap] # non-libcd gap -> src/gap.c (vram 0x80046980-0x800469CC, stub)
|
||||
- [0x371CC, c, libcd2] # libcd block 2 -> src/libcd2.c (vram 0x800469CC-0x8004787C, 7 objs)
|
||||
- [0x3807C, c, 800b] # -O2 game code -> src/800b.c (vram 0x8004787C-0x800629DC)
|
||||
- [0x531DC, data, 531DC] # data — psxexeinfo boundary; round-trips byte-identical
|
||||
- [0x65000]
|
||||
|
||||
@@ -260,3 +260,73 @@ vs PsyQ libcd; `PRESET_OBJ_*` ∈ `LIBGS.LIB`. Workflow (the decomp-standard psy
|
||||
.text` → byte-compare to the EXE. `.rdata`/`.data` vrams are found by searching the EXE for the section
|
||||
bytes (`objcopy --only-section`). Tools: `tools/psyq_lib_split.py`, `tools/psyq_build_libs.sh`,
|
||||
`tools/psyq_identify.py`.
|
||||
|
||||
### §9.1 Generalised per-object linker — `tools/psyq_link.py` (+ `psyq_link_lib.py`), 18/18 libcd byte-exact
|
||||
Session-D generalised the SYS.o recipe into a tool that links **every** used object of a library byte-identical.
|
||||
Two gotchas the one-object recipe didn't surface, both now handled:
|
||||
- **psyq-obj-parser MISLABELS common-style globals.** Uninitialised globals (PSYLINK `.comm`) get packed into each
|
||||
object's `.bss` with *sequential* `st_value`s, but the original linker SCATTERED them (e.g. libcd CDROM's
|
||||
`StRingAddr`→`0x800c7c94` and `StRingSize`→`0x800c7f00` are 0x26c apart, the ELF claims 8). **Trust no
|
||||
`st_value`** for placement.
|
||||
- **Robust model = recover-everything + selective override.** Place `.text` at its vram and the *real* initialised
|
||||
sections at their bases (byte-search; or, for `.bss`/reloc-bearing `.data`, the address the section symbol itself
|
||||
resolves to in the EXE). Resolve every symbol the `.text` references by the address read out of the EXE's already-
|
||||
linked relocations (`R_MIPS_26` jump field; `HI16`+`LO16` immediates; object addend subtracted, but PsyQ addends
|
||||
are 0). A symbol that is a *genuine* member of a placed section (recovered == base+st_value) is left to `ld`; a
|
||||
*mislabelled* one is **`--weaken-symbol`'d then `--defsym`'d** to its recovered address (a strong defsym beats the
|
||||
weak section def — `--strip-symbol` is refused on reloc-referenced symbols, weaken isn't). `.text` byte-compare is
|
||||
the check (G3).
|
||||
- **Tells:** `ld: 'X' referenced … defined in discarded section` = you discarded a section whose section-symbol the
|
||||
`.text` needs → place it instead. A 1–4 word residual in `lui/lw/sw` immediates (`3c0480xx`) = a mislabelled
|
||||
`.bss` common → weaken+defsym it.
|
||||
- **Externals split intra/extra-library.** Per-object "externals" (UND) include symbols defined in *sibling* objects
|
||||
(resolve internally in a whole-library link) vs truly external ones (other libs' funcs like `VSync`/`memcpy`, and
|
||||
module data globals like `St*`) — the latter feed `--defsym`/`symbols.us.txt` (R15). libcd: 82 union = 48 intra +
|
||||
34 extra. `tools/psyq_link_lib.py <elf_dir>` links all located objects, flags address conflicts, writes
|
||||
`.run/psyq_link.<lib>.json`. Same tooling will serve libgs/libspu/… (the +24 culprits).
|
||||
|
||||
### §9.2 Wire a library region into the build with NOLOAD — no data carving (`tools/psyq_link_region.py`)
|
||||
To replace the asm stubs of a library's functions with the real objects in the byte-identical build WITHOUT
|
||||
carving the flat `data` subsegment:
|
||||
- **Place `.text` LOADED at each object's exact vram; place `.data`/`.rdata`/`.bss` as NOLOAD at their vrams.** A
|
||||
NOLOAD section contributes its symbol addresses but **zero bytes** to `objcopy -O binary`, so the build's existing
|
||||
flat data subsegment still emits those bytes (no double-emit, no carve) while the hundreds of section-relative
|
||||
`.text` refs resolve via the NOLOAD placement. Set `.data`/`.rdata`/`.bss` align=4 first or a 4-but-not-8-aligned
|
||||
vram bumps +4 (same tell as §9). Needs `ld --no-check-sections` (NOLOAD overlaps the loaded flat blob's VMA).
|
||||
- **Weaken every `.bss`/`.sbss`-defined named symbol**, then `--defsym` it to its recovered address: the common-style
|
||||
globals are scattered (genuine `CD_*` and mislabelled `St*` alike, and a `.bss` symbol of object A may be referenced
|
||||
by object B), so a uniform strong-defsym-beats-weak-def resolves them all. Truly-undefined externals (other libs'
|
||||
funcs) surface from a probe link's `undefined reference` lines → defsym from the recovered map.
|
||||
- **Place each object at its EXACT vram, not by concatenation** — a library's objects are *mostly* contiguous but a
|
||||
non-library function can sit between them (libcd: a 76-B gap of non-libcd code between C_003 and C_004), so naive
|
||||
`*(.text)` concatenation drifts past the gap. The gap stays an asm stub in the build (split the splat code subseg
|
||||
into [pre][lib block 1][gap stub][lib block 2][post]). `tools/psyq_link_region.py <elf_dir> --emit <p>` verifies the
|
||||
region byte-identical per-object and emits `<p>.ld` (text + NOLOAD lines) + `<p>.syms`. libcd: 18 objects byte-exact,
|
||||
36 externals.
|
||||
|
||||
### §9.3 Make it the build: resegment + swap + resolve (`tools/psyq_integrate.py`, libcd DONE)
|
||||
Wiring a library region into `make build` byte-identical (libcd: 58 SDK funcs, full pipeline green):
|
||||
- **Resegment the splat text subseg into [pre][block1][gap][block2…][post]** at the library blocks
|
||||
(one `c` subseg per block + per non-library gap; vram→file = −0x8000F800). `make extract` regenerates
|
||||
the gap/post stubs; **`tools/split_src_region.py trim`** rewrites the curated pre-file (keeps items
|
||||
<lo, preserving real C and `#ifdef NON_MATCHING` blocks by brace/`#endif` matching) — splat will NOT
|
||||
overwrite an existing `.c`, so a stale one mis-places everything. splat-auto-empties (`void f(void){}`)
|
||||
≥hi regenerate identically — no move needed.
|
||||
- **`psyq_integrate.py` (run in the `$(OUT)` recipe, after objects compile — the externals discovery
|
||||
trial-links the whole image):** (1) prep objects (align=4 + weaken every `.bss` symbol) → `build/psyq/<lib>/`;
|
||||
(2) rewrite the splat `.ld` — replace each `build/src/<stub>.o(.text);` with the block's real
|
||||
`<obj>.o(.text);` (concatenation places them at their vrams since the pre-file ends exactly at the block
|
||||
start) AND **delete the stub object's other `(.rodata/.data/.bss)` lines** (else its stub symbols
|
||||
multiply-define the real ones); add per-object **NOLOAD** data sections sorted by vram (unsorted →
|
||||
"dot moved backwards"); (3) resolve externals via a **full trial link** (symbols still defined elsewhere
|
||||
never appear, so no double-def — no blanket exclude needed).
|
||||
- **External resolution, in order:** `func_<addr>` (external code/data calls a libcd fn by its splat
|
||||
address-name; the real object exports a PsyQ name) → that address; a `symbols.us.txt` name (jump-table /
|
||||
dispatch pointer in the flat `.data`, e.g. `BIOS_OBJ_3B8`) → its `symbols.us.txt` address; a recovered
|
||||
data/extern global (`St*`/`CD_*`) → recovered. **Always also defsym EVERY weakened `.bss` common** — the
|
||||
ones whose object's `.bss` is NOLOAD-placed resolve to that weak placement and never show as undefined
|
||||
(the 8-word `StMode` miss). Capture BOTH `undefined reference` AND `defined in discarded section` from the
|
||||
trial link.
|
||||
- **Byte-identical with OR without the SDK objects** (the stubs reproduce the same bytes), so gate the
|
||||
whole thing on `[ -d <elf_dir> ]` — a fresh clone without `tools/psyq/` builds via stubs. Idempotent
|
||||
(`build/psyq/<lib>` in the `.ld` ⇒ re-derive syms only).
|
||||
|
||||
@@ -34,15 +34,21 @@ rodata-island foundation + LZSS match are DEFERRED to a focused sub-project afte
|
||||
- [ ] **Task 7 — PhaseEnd_Phase7** (Gen1 synthesis, milestone gate). **Max · Tier 1.**
|
||||
|
||||
## Current task
|
||||
**Task 2′ — rodata-island / LZSS gate.** Mechanism now PROVEN end-to-end; a strategic pivot to PsyQ-library
|
||||
linking is in flight (see the PsyQ spike section below) — that fixes the library-half alignment AND gives
|
||||
~350 SDK functions byte-exact for free. Then LZSS, Task 6, Task 7.
|
||||
NOTE: Gen1 exit needs ≥3 SESSIONS of green `make check` — **now satisfied** (A, B, C below); Tasks 6/7 still pending.
|
||||
**Task 2′ — libcd-into-build DONE (session D); next = libgs + LZSS.** Per Drew's approved-plan ordering
|
||||
(libcd-first to prove the build-integration mechanism on the simplest library, then libgs+island+LZSS):
|
||||
- ✅ **libcd wired into the byte-identical build** (session D) — 58 SDK funcs from real objects; full
|
||||
build-integration mechanism proven + tooled (psyq_integrate.py, NOLOAD = no carving, splat resegment,
|
||||
H5-safe src split). Sub-tasks 2′.1 (ground)/2′.2 (18/18 verify)/2′.3 (build wiring) complete.
|
||||
- ▶ **NEXT: libgs** (the actual +24 culprit — PRESET/PRESET2/OBJT2/PRNT jtbls) via the same psyq_integrate
|
||||
path → fixes the +24 + banks ~69 libgs funcs; then **LZSS** (jtbl_80072A38) via the rodata-island
|
||||
migration+ld_interleave path; then Task 6 (Gen1 close) + Task 7 (PhaseEnd).
|
||||
NOTE: Gen1 exit needs ≥3 SESSIONS of green `make check` — **satisfied** (A, B, C, +D); Tasks 6/7 still pending.
|
||||
|
||||
## Per-session `make check` green log (≥3 sessions needed for the milestone)
|
||||
- 2026-06-14 (session A): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ — baseline restored + reproducibility fix, reports built, **38 real matches** (22 accessor leaves + ResourceGetCdLoc + LoaderResetReadState), build byte-identical throughout. [need ≥2 more sessions]
|
||||
- 2026-06-14 (session B): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ — **per-file -O0 split mechanism** (src/boot.c + Makefile per-file flags); **4 real matches** (GameModeDispatch, DebugMenuHandler, CdQueueBusy, CdReadRequest) → **42 real**; **PsyQ libcd.h infra** (CdlLOC/CdlFILE + 4 named symbols, unlocks the loader cluster); **LoaderInitFileTable + ResourceLoadStateMachine NON_MATCHING-drafted** (→ 4 NM) — **Task 5 non-jtbl loaders COMPLETE** (6 matched + 2 drafted); report tooling fixed (multi-file); cookbook §6/§7/T4. Build byte-identical throughout. [need ≥1 more session]
|
||||
- 2026-06-15 (session C): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ (full `clean && extract && build`, restored after the Task-2′ experiments). **≥3-session bar MET.** This session: fully diagnosed + built the **rodata-island mechanism** (works); root-caused the +24; **proved the PsyQ-library-linking GO** (see below). No new matches (architectural session). Build green at start and after restore.
|
||||
- 2026-06-14 (session D): `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL` ✓ — **libcd LINKED INTO THE BUILD** (Drew-approved push-through). The first real PsyQ library is now sourced from real SDK objects in the byte-identical build: **58 libcd SDK functions** linked (not stubs), replacing the libcd-region asm stubs. Idempotent; Makefile-automated; conditional (fresh clone w/o `tools/psyq/` builds via stubs). New committed tooling: `tools/psyq_link.py` (per-object byte-link engine — recovers externals from resolved relocs, weakens psyq-obj-parser's mislabelled `.bss` commons), `psyq_link_lib.py` (whole-lib verify, 18/18 libcd), `psyq_link_region.py` (region link via **NOLOAD** = no data carving), `psyq_integrate.py` (build wiring: splat resegment + .ld swap + external resolution), `split_src_region.py` (H5-safe src split). Cookbook §9.1/9.2/9.3 + R16. **No data carving** (NOLOAD data placement; flat data subseg unchanged). Build green throughout.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1698
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,7 @@
|
||||
#include "common.h"
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/gap", DsSyncCallback);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/gap", DsReadyCallback);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/gap", DsDataCallback);
|
||||
+213
@@ -0,0 +1,213 @@
|
||||
#include "common.h"
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", StSetRing);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800430B8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800430DC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", TOC_OBJ_220);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043300);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", EVENT_OBJ_84);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", def_cbsync);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", def_cbready);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", def_cbread);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043410);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043420);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdLastCom);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043440);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043450);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", SYS_OBJ_98);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800434BC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdSetDebug);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdComstr);
|
||||
|
||||
void SYS_OBJ_110(void) {
|
||||
}
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdIntstr);
|
||||
|
||||
void SYS_OBJ_144(void) {
|
||||
}
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004355C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004357C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004359C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800435B4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800435CC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043704);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043830);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", SYS_OBJ_538);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043974);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043994);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800439B4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800439D4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800439F8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdIntToPos);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdPosToInt);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80043B9C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_64);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_26C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_36C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_3B8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_43C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_4C0);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_570);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044124);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_6B4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_6E4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_7DC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800443A4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_93C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_96C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_AA4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044670);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_DAC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_DDC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_EC8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044A8C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044B14);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044BF4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044CE8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044D38);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80044F24);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_14A4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045090);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045190);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CD_set_test_parmnum);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", callback);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", BIOS_OBJ_1728);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdSearchFile);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_F8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_29C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_2A8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045640);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045660);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_5A4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045940);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_650);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800459E8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_7F4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_810);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", ISO9660_OBJ_8F4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80045C94);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", cb_read);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD_OBJ_A4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD_OBJ_1A4);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", cb_data);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800460CC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD_OBJ_5C0);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdReadBreak);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004637C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD_OBJ_6B8);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80046480);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD_OBJ_810);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004654C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80046564);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CdRead2);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", CDREAD2_OBJ_50);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", StCdInterrupt2);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_80046630);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", TYPE_OBJ_104);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_8004674C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", TYPE_OBJ_178);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", TYPE_OBJ_254);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", StClearRing);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd1", func_800468FC);
|
||||
@@ -0,0 +1,41 @@
|
||||
#include "common.h"
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", data_ready_callback);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", StGetBackloc);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_004_OBJ_DC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", StSetStream);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", StFreeRing);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_007_OBJ_AC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", init_ring_status);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", StGetNext);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", func_80046CFC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", StCdInterrupt);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_270);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_3D0);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_7BC);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_85C);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_900);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_960);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", mem2mem);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", dma_execute);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_A30);
|
||||
|
||||
INCLUDE_ASM("asm/nonmatchings/libcd2", C_011_OBJ_A90);
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Wire real PsyQ library objects into the splat build, replacing stub subsegments (Phase 7 2'.3).
|
||||
|
||||
Run after `make extract`. Given a library's ELF objects and the stub subsegment(s) splat emitted
|
||||
for the library's text region(s), this:
|
||||
1. prepares each object (objcopy: .text/.data/.rdata/.bss align=4; weaken every .bss/.sbss
|
||||
named symbol so a strong --defsym beats its scattered weak def) -> <objdir>/*.o
|
||||
2. rewrites the splat linker script: each `build/src/<stub>.o(.text);` line is replaced by the
|
||||
real objects' `<objdir>/<obj>.o(.text);` lines (the library's objects form one contiguous
|
||||
block per stub, so concatenation places them at their exact vrams); each object's
|
||||
.data/.rdata/.bss is added as a NOLOAD section at its vram (addresses only — the flat data
|
||||
subseg still supplies the bytes, no carve)
|
||||
3. writes <syms.ld>: `NAME = 0xADDR;` for every external the objects reference but no linked
|
||||
object defines (other libs' funcs + module data globals) — added to the link via -T.
|
||||
|
||||
Stub<->block mapping is by vram order: the i-th stub (in vram order) gets the i-th contiguous
|
||||
object block. Non-library gaps between blocks keep their own stub subsegment untouched.
|
||||
|
||||
Usage: psyq_integrate.py <elf_dir> <ld_path> <objdir> <syms_ld> <stub1>[,<stub2>,...]
|
||||
"""
|
||||
import os, re, subprocess, sys, tempfile
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from psyq_link import recover_sym_addrs, AS, sh, VRAM_BASE, DATA_SECTIONS
|
||||
from psyq_link_region import classify, placement
|
||||
|
||||
EXE = "extracted/retail/SLUS_007.26"
|
||||
|
||||
|
||||
def contiguous_blocks(order):
|
||||
"""Split vram-ordered objects into contiguous runs (a non-library gap starts a new block)."""
|
||||
blocks, cur, end = [], [], None
|
||||
for name, (vram, ins) in order:
|
||||
if end is not None and vram != end:
|
||||
blocks.append(cur)
|
||||
cur = []
|
||||
cur.append((name, vram, ins))
|
||||
end = vram + ins * 4
|
||||
if cur:
|
||||
blocks.append(cur)
|
||||
return blocks
|
||||
|
||||
|
||||
def trial_undefined(ld_path, extra_syms=None):
|
||||
"""Full-build link with the rewritten .ld; return the set of undefined symbol names."""
|
||||
td = tempfile.mkdtemp(dir=".run")
|
||||
elf = os.path.join(td, "trial.elf")
|
||||
cmd = [f"{AS}ld", "-T", ld_path, "-T", "undefined_syms_auto.txt",
|
||||
"-T", "undefined_funcs_auto.txt", "--no-check-sections", "-o", elf]
|
||||
if extra_syms:
|
||||
cmd += ["-T", extra_syms]
|
||||
p = subprocess.run(cmd, capture_output=True)
|
||||
err = p.stderr.decode()
|
||||
# a weakened .bss common referenced by another object whose .bss we discarded shows up not as
|
||||
# "undefined reference" but as "defined in discarded section" — capture both forms.
|
||||
return (set(re.findall(r"undefined reference to [`']([^`']+)'", err))
|
||||
| set(re.findall(r"[`']([^`']+)' referenced in section .*? defined in discarded section", err)))
|
||||
|
||||
|
||||
def integrate(elf_dir, ld_path, objdir, syms_path, stubs):
|
||||
exe = open(EXE, "rb").read()
|
||||
order = sorted(placement(elf_dir, None, None).items(), key=lambda kv: kv[1][0])
|
||||
recovered, weaken_by, bases_by = {}, {}, {}
|
||||
for name, (vram, _) in order:
|
||||
bases, weaken, sym_addr = classify(os.path.join(elf_dir, name), vram, exe)
|
||||
bases_by[name], weaken_by[name] = bases, weaken
|
||||
for s, a in sym_addr.items():
|
||||
if not s.startswith("."):
|
||||
recovered[s] = a
|
||||
|
||||
# 1. prepare objects (persistent)
|
||||
os.makedirs(objdir, exist_ok=True)
|
||||
for name, _ in order:
|
||||
args = []
|
||||
for S in (".text",) + DATA_SECTIONS:
|
||||
args += ["--set-section-alignment", f"{S}=4"]
|
||||
for w in weaken_by[name]:
|
||||
args += ["--weaken-symbol", w]
|
||||
sh(f"{AS}objcopy", *args, os.path.join(elf_dir, name), os.path.join(objdir, name))
|
||||
|
||||
blocks = contiguous_blocks(order)
|
||||
if len(blocks) != len(stubs):
|
||||
sys.exit(f"integrate: {len(blocks)} object blocks but {len(stubs)} stub(s) given "
|
||||
f"({[len(b) for b in blocks]} objs/block)")
|
||||
|
||||
# 2. rewrite the linker script
|
||||
ld = open(ld_path).read()
|
||||
already = objdir in ld # idempotent: a re-run on an already-rewritten .ld only redoes syms
|
||||
for stub, block in zip(stubs, blocks):
|
||||
textlines = "\n".join(f' "{objdir}/{nm}"(.text);' for nm, _, _ in block)
|
||||
pat = re.compile(r"^[ \t]*build/src/" + re.escape(stub) + r"\.o\(\.text\);[ \t]*$", re.M)
|
||||
if pat.search(ld):
|
||||
ld = pat.sub(textlines, ld, count=1)
|
||||
elif not already:
|
||||
sys.exit(f"integrate: stub text line for '{stub}' not found in {ld_path}")
|
||||
# drop the stub object's other (empty) section lines so it is not linked at all
|
||||
# (else its INCLUDE_ASM-stub symbols multiply-define the real objects' symbols)
|
||||
other = re.compile(r"^[ \t]*build/src/" + re.escape(stub)
|
||||
+ r"\.o\(\.(?:rodata|data|bss|sdata|sbss)\);[ \t]*\n", re.M)
|
||||
ld = other.sub("", ld)
|
||||
# NOLOAD data sections, sorted by vram (keeps ld's location counter monotonic), before /DISCARD/
|
||||
nol_items = sorted((b, name, S) for name, (_, _) in order
|
||||
for S, b in bases_by[name].items())
|
||||
nol = [f' .nl_{i} 0x{b:08X} (NOLOAD) : {{ "{objdir}/{name}"(.{S[1:]}) }}'
|
||||
for i, (b, name, S) in enumerate(nol_items)]
|
||||
if ".nl_0 " not in ld:
|
||||
ld = re.sub(r"^([ \t]*/DISCARD/ :)", "\n".join(nol) + r"\n\n\1", ld, count=1, flags=re.M)
|
||||
open(ld_path, "w").write(ld)
|
||||
|
||||
# 3. resolve every symbol the rewritten build ACTUALLY leaves undefined (a full link, so symbols
|
||||
# still defined elsewhere — e.g. VSync in the 800 region — never appear and can't be
|
||||
# double-defined). A removed libcd stub used to define each libcd-region symbol by its splat
|
||||
# name (func_<addr> or a symbols.us.txt name like BIOS_OBJ_3B8) and the data blob / external
|
||||
# code references it by that name; the real object defines a PsyQ name instead. So map each
|
||||
# back to its address: func_<addr> -> the address; a named symbol -> symbols.us.txt; a
|
||||
# recovered data/extern global (St*, CD_*) -> its recovered address.
|
||||
symu = {}
|
||||
for ln in open("config/symbols.us.txt"):
|
||||
m = re.match(r"(\w+)\s*=\s*0x([0-9A-Fa-f]+)", ln)
|
||||
if m:
|
||||
symu[m.group(1)] = int(m.group(2), 16)
|
||||
# every weakened .bss common must be defsym'd to its recovered (scattered) address: where its
|
||||
# object's .bss is NOLOAD-placed it otherwise resolves to that weak placement (wrong) and never
|
||||
# shows as undefined; the strong defsym overrides the weak def uniformly.
|
||||
weaken_all = {w for ws in weaken_by.values() for w in ws}
|
||||
externals = {s: recovered[s] for s in weaken_all if s in recovered}
|
||||
missing = []
|
||||
for s in sorted(set(trial_undefined(ld_path)) - set(externals)):
|
||||
m = re.fullmatch(r"func_([0-9A-Fa-f]{8})", s)
|
||||
if m:
|
||||
externals[s] = int(m.group(1), 16)
|
||||
elif s in symu:
|
||||
externals[s] = symu[s]
|
||||
elif s in recovered:
|
||||
externals[s] = recovered[s]
|
||||
else:
|
||||
missing.append(s)
|
||||
with open(syms_path, "w") as f:
|
||||
for s, a in sorted(externals.items(), key=lambda kv: kv[1]):
|
||||
f.write(f"{s} = 0x{a:08X};\n")
|
||||
print(f"integrate {os.path.basename(elf_dir)}: {len(order)} objects in {len(blocks)} block(s) "
|
||||
f"-> stubs {stubs}; {len(nol)} NOLOAD sections; {len(externals)} externals -> {syms_path}")
|
||||
if missing:
|
||||
print(f" !! {len(missing)} UNRESOLVED (not func_<addr>, not recovered): {missing[:16]}")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 6:
|
||||
sys.exit(__doc__)
|
||||
integrate(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5].split(","))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Link a single PsyQ ELF object so its `.text` is byte-identical to the BFM EXE.
|
||||
|
||||
This is the generalised form of the session-C SYS.o proof (cookbook §9): given a
|
||||
PsyQ-SDK object (a `.LIB` member converted to ELF by psyq-obj-parser) and the vram
|
||||
where the EXE links its `.text`, reproduce the EXE bytes exactly by:
|
||||
|
||||
1. section sizes — `objdump -h`
|
||||
2. locate .rdata/.data vram — unique byte-search of the EXE (objcopy --only-section)
|
||||
3. recover externals — read each undefined symbol's address straight out of the
|
||||
EXE's already-RESOLVED relocations (R_MIPS_26 jump field;
|
||||
HI16/LO16 immediate-field pair), minus the object addend
|
||||
4. alignment fix — objcopy --set-section-alignment .text/.rdata/.data = 4
|
||||
(psyq-obj-parser emits 2**3; the original is 4-aligned, so
|
||||
an 8-align bumps a 4-but-not-8 vram +4)
|
||||
5. link — ld -T <SECTIONS placing each section at its vram>
|
||||
--defsym NAME=ADDR per recovered external
|
||||
6. verify — objcopy -O binary --only-section .text -> byte-compare
|
||||
|
||||
The byte-compare of `.text` is the ground-truth check (G3/P9). Externals defined by *other*
|
||||
objects of the same library are recovered here too (single-object mode); the whole-library
|
||||
wiring (2'.3) links them together so those resolve internally and only true externals def(
|
||||
DMACallback, hardware) need --defsym.
|
||||
|
||||
Usage:
|
||||
psyq_link.py <obj.o> <text_vram> [--name NAME] [--quiet]
|
||||
-> prints PASS/FAIL + recovered externals; exit 0 on byte-identical .text.
|
||||
|
||||
Importable: link_object(obj, text_vram) -> dict(result).
|
||||
"""
|
||||
import struct, subprocess, sys, os, re, tempfile
|
||||
|
||||
EXE = "extracted/retail/SLUS_007.26"
|
||||
VRAM_BASE = 0x8000F800
|
||||
AS = "mipsel-linux-gnu-"
|
||||
|
||||
|
||||
def sh(*a, **k):
|
||||
return subprocess.run(a, check=True, capture_output=True, **k)
|
||||
|
||||
|
||||
def u32(buf, off):
|
||||
return struct.unpack_from("<I", buf, off)[0]
|
||||
|
||||
|
||||
def s16(x):
|
||||
x &= 0xFFFF
|
||||
return x - 0x10000 if x & 0x8000 else x
|
||||
|
||||
|
||||
def only_section(obj, sec):
|
||||
"""Raw bytes of one section, or b'' if absent/empty."""
|
||||
try:
|
||||
return sh(f"{AS}objcopy", "-O", "binary", "--only-section", sec, obj, "/dev/stdout").stdout
|
||||
except subprocess.CalledProcessError:
|
||||
return b""
|
||||
|
||||
|
||||
def section_table(obj):
|
||||
"""{name: (size, align_pow)} from objdump -h."""
|
||||
out = sh(f"{AS}objdump", "-h", obj).stdout.decode()
|
||||
t = {}
|
||||
for ln in out.splitlines():
|
||||
m = re.match(r"\s+\d+\s+(\.\S+)\s+([0-9a-f]+)\s+[0-9a-f]+\s+[0-9a-f]+\s+[0-9a-f]+\s+2\*\*(\d+)", ln)
|
||||
if m:
|
||||
t[m.group(1)] = (int(m.group(2), 16), int(m.group(3)))
|
||||
return t
|
||||
|
||||
|
||||
def undefined_syms(obj):
|
||||
out = sh(f"{AS}readelf", "-s", obj).stdout.decode()
|
||||
u = set()
|
||||
for ln in out.splitlines():
|
||||
p = ln.split()
|
||||
# Num: Value Size Type Bind Vis Ndx Name
|
||||
if len(p) >= 8 and p[6] == "UND" and p[7] and not p[7].startswith("."):
|
||||
u.add(p[7])
|
||||
return u
|
||||
|
||||
|
||||
def text_relocs(obj):
|
||||
"""Ordered (offset, type, sym) for .text relocations."""
|
||||
out = sh(f"{AS}readelf", "-r", obj).stdout.decode()
|
||||
relocs, in_text = [], False
|
||||
for ln in out.splitlines():
|
||||
if ln.startswith("Relocation section"):
|
||||
in_text = "'.rel.text'" in ln or '".rel.text"' in ln or ".rel.text" in ln
|
||||
continue
|
||||
if not in_text:
|
||||
continue
|
||||
m = re.match(r"\s*([0-9a-f]+)\s+[0-9a-f]+\s+(R_MIPS_\S+)\s+[0-9a-f]+\s+(\S+)", ln)
|
||||
if m:
|
||||
relocs.append((int(m.group(1), 16), m.group(2), m.group(3)))
|
||||
return relocs
|
||||
|
||||
|
||||
SECTION_IDX_RE = re.compile(r"\s*\[\s*(\d+)\]\s+(\.\S+)")
|
||||
DATA_SECTIONS = (".data", ".rdata", ".rodata", ".sdata", ".bss", ".sbss")
|
||||
|
||||
|
||||
def section_index_names(obj):
|
||||
"""{section index: name} from readelf -S."""
|
||||
out = sh(f"{AS}readelf", "-S", obj).stdout.decode()
|
||||
idx = {}
|
||||
for ln in out.splitlines():
|
||||
m = SECTION_IDX_RE.match(ln)
|
||||
if m:
|
||||
idx[int(m.group(1))] = m.group(2)
|
||||
return idx
|
||||
|
||||
|
||||
def symbol_table(obj):
|
||||
"""{name: (section_name|'UND'|'ABS', st_value)} from readelf -s."""
|
||||
idx = section_index_names(obj)
|
||||
out = sh(f"{AS}readelf", "-s", obj).stdout.decode()
|
||||
syms = {}
|
||||
for ln in out.splitlines():
|
||||
p = ln.split()
|
||||
# Num: Value Size Type Bind Vis Ndx Name
|
||||
if len(p) >= 8 and p[0].endswith(":") and re.fullmatch(r"[0-9a-f]+", p[1]):
|
||||
ndx, name = p[6], p[7]
|
||||
if name.startswith("."):
|
||||
continue
|
||||
if ndx in ("UND", "ABS"):
|
||||
sec = ndx
|
||||
elif ndx.isdigit():
|
||||
sec = idx.get(int(ndx), "?")
|
||||
else:
|
||||
continue
|
||||
syms[name] = (sec, int(p[1], 16))
|
||||
return syms
|
||||
|
||||
|
||||
def recover_sym_addrs(obj, text_vram, exe):
|
||||
"""Resolved EXE address of every symbol referenced by a .text relocation.
|
||||
|
||||
Works for section symbols (name == '.data' etc.) and named data/bss/extern
|
||||
symbols alike: reads the already-linked field(s) out of the EXE and subtracts
|
||||
the object's in-field addend. R_MIPS_26 (jump) and HI16/LO16 pairs.
|
||||
"""
|
||||
text = only_section(obj, ".text")
|
||||
addr, pending_hi = {}, {}
|
||||
for off, typ, sym in text_relocs(obj):
|
||||
if typ == "R_MIPS_26":
|
||||
if sym not in addr:
|
||||
site = text_vram + off
|
||||
exew = u32(exe, site - VRAM_BASE)
|
||||
objw = u32(text, off)
|
||||
A = (objw & 0x03FFFFFF) << 2
|
||||
addr[sym] = ((((exew & 0x03FFFFFF) << 2) | (site & 0xF0000000)) - A) & 0xFFFFFFFF
|
||||
elif typ == "R_MIPS_HI16":
|
||||
pending_hi.setdefault(sym, []).append(off)
|
||||
elif typ == "R_MIPS_LO16":
|
||||
his = pending_hi.get(sym)
|
||||
if his:
|
||||
hi_off = his[-1]
|
||||
if sym not in addr:
|
||||
ehi = u32(exe, text_vram + hi_off - VRAM_BASE) & 0xFFFF
|
||||
elo = u32(exe, text_vram + off - VRAM_BASE) & 0xFFFF
|
||||
ohi = u32(text, hi_off) & 0xFFFF
|
||||
olo = u32(text, off) & 0xFFFF
|
||||
addr[sym] = (((ehi << 16) + s16(elo)) - ((ohi << 16) + s16(olo))) & 0xFFFFFFFF
|
||||
pending_hi[sym] = his[:-1]
|
||||
return addr
|
||||
|
||||
|
||||
def unique_byte_vram(obj, sec, exe):
|
||||
raw = only_section(obj, sec)
|
||||
if not raw:
|
||||
return None
|
||||
hits, start = [], 0
|
||||
while True:
|
||||
j = exe.find(raw, start)
|
||||
if j < 0:
|
||||
break
|
||||
hits.append(j + VRAM_BASE)
|
||||
start = j + 1
|
||||
if len(hits) > 1:
|
||||
return None # ambiguous
|
||||
return hits[0] if hits else None
|
||||
|
||||
|
||||
def link_object(obj, text_vram, name=None, exe_bytes=None):
|
||||
"""Place .text at its EXE vram, --defsym every external it references, byte-verify.
|
||||
|
||||
The robust model (validated against the psyq-obj-parser .bss-mislabelling: it packs
|
||||
common-style globals into per-object .bss with sequential st_values the original
|
||||
linker did NOT honour): trust NO st_value. Place only .text; DISCARD every other
|
||||
section; resolve every symbol the .text references by its address recovered from the
|
||||
EXE's already-linked relocations (R_MIPS_26 / HI16+LO16). Internal .text labels are
|
||||
left for ld to place. Discarding the defining section makes a --defsym of a
|
||||
once-defined symbol conflict-free, so this is uniform for externals, data, and bss.
|
||||
|
||||
Section bases (.rdata/.data/.bss) are still computed — for the report and the
|
||||
whole-library build wiring (2'.3) — but they do not gate the .text verification.
|
||||
"""
|
||||
name = name or os.path.basename(obj)
|
||||
exe = exe_bytes if exe_bytes is not None else open(EXE, "rb").read()
|
||||
secs = section_table(obj)
|
||||
tsize = secs.get(".text", (0, 0))[0]
|
||||
res = {"name": name, "text_vram": text_vram, "tsize": tsize}
|
||||
|
||||
symtab = symbol_table(obj)
|
||||
sym_addr = recover_sym_addrs(obj, text_vram, exe)
|
||||
|
||||
def sym_section(s):
|
||||
if s in secs: # section symbol (.data/.rdata/.bss/…)
|
||||
return (s, 0)
|
||||
return symtab.get(s, ("UND", 0))
|
||||
|
||||
# section bases for placement: byte-search (initialised) or the section-symbol reloc.
|
||||
bases = {}
|
||||
for S in DATA_SECTIONS:
|
||||
if S not in secs or secs[S][0] == 0:
|
||||
continue
|
||||
b = unique_byte_vram(obj, S, exe) if S not in (".bss", ".sbss") else None
|
||||
if b is None:
|
||||
b = sym_addr.get(S) # set iff the object referenced the section symbol
|
||||
bases[S] = b
|
||||
placed = {S: b for S, b in bases.items() if b is not None}
|
||||
res["rdata_vram"] = bases.get(".rdata") or bases.get(".rodata")
|
||||
res["data_vram"] = bases.get(".data")
|
||||
res["bss_vram"] = bases.get(".bss")
|
||||
|
||||
# Classify each referenced symbol: resolve by section placement, or by --defsym to its
|
||||
# recovered address — weakening a placed-but-mislabelled definition first (psyq-obj-parser
|
||||
# packs common-style globals into .bss with st_values the original linker did not honour).
|
||||
defs, weaken = {}, []
|
||||
for s, a in sym_addr.items():
|
||||
sec, val = sym_section(s)
|
||||
if sec == ".text":
|
||||
continue # internal label -> ld placement
|
||||
if s in secs: # a section symbol (.data/.bss/…)
|
||||
if s not in placed:
|
||||
defs[s] = a # its section was discarded
|
||||
continue
|
||||
if sec in placed and a == (placed[sec] + val) & 0xFFFFFFFF:
|
||||
continue # genuine member -> ld placement
|
||||
defs[s] = a
|
||||
if sec in placed:
|
||||
weaken.append(s) # mislabelled inside a placed section
|
||||
externals = {s: a for s, a in defs.items() if sym_section(s)[0] == "UND"}
|
||||
res["externals"] = externals
|
||||
res["unrecovered"] = sorted(undefined_syms(obj) - set(externals))
|
||||
|
||||
with tempfile.TemporaryDirectory(dir=".run") as td:
|
||||
aligned = os.path.join(td, "a.o")
|
||||
align_args = ["--set-section-alignment", ".text=4"]
|
||||
for S in placed:
|
||||
align_args += ["--set-section-alignment", f"{S}=4"]
|
||||
for s in weaken:
|
||||
align_args += ["--weaken-symbol", s]
|
||||
sh(f"{AS}objcopy", *align_args, obj, aligned)
|
||||
ld = os.path.join(td, "link.ld")
|
||||
lines = ["SECTIONS {", f" . = 0x{text_vram:08X};", " .text : { *(.text) }"]
|
||||
for S, b in placed.items():
|
||||
lines += [f" . = 0x{b:08X};", f" {S} : {{ *({S}) }}"]
|
||||
lines += [" /DISCARD/ : { *(*) }", "}"]
|
||||
open(ld, "w").write("\n".join(lines) + "\n")
|
||||
cmd = [f"{AS}ld", "-T", ld, "-o", os.path.join(td, "out.elf"), aligned]
|
||||
for s, a in sorted(defs.items()):
|
||||
cmd += ["--defsym", f"{s}=0x{a:08X}"]
|
||||
p = subprocess.run(cmd, capture_output=True)
|
||||
if p.returncode != 0:
|
||||
res["ok"] = False
|
||||
res["error"] = "ld: " + p.stderr.decode().strip().split("\n")[-1]
|
||||
return res
|
||||
got = sh(f"{AS}objcopy", "-O", "binary", "--only-section", ".text",
|
||||
os.path.join(td, "out.elf"), "/dev/stdout").stdout
|
||||
|
||||
want = exe[text_vram - VRAM_BASE: text_vram - VRAM_BASE + tsize]
|
||||
res["ok"] = (got == want)
|
||||
if not res["ok"]:
|
||||
diffs = [i for i in range(0, min(len(got), len(want)), 4) if got[i:i+4] != want[i:i+4]]
|
||||
res["ndiff"] = len(diffs) + abs(len(got) - len(want)) // 4
|
||||
res["first_diff"] = (text_vram + diffs[0]) if diffs else None
|
||||
return res
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
sys.exit(__doc__)
|
||||
obj, text_vram = sys.argv[1], int(sys.argv[2], 0)
|
||||
name = None
|
||||
quiet = "--quiet" in sys.argv
|
||||
if "--name" in sys.argv:
|
||||
name = sys.argv[sys.argv.index("--name") + 1]
|
||||
r = link_object(obj, text_vram, name)
|
||||
tag = "PASS" if r["ok"] else "FAIL"
|
||||
print(f"[{tag}] {r['name']:14s} .text@0x{text_vram:08X} ({r['tsize']} B) "
|
||||
f".rdata@{r['rdata_vram'] if not isinstance(r['rdata_vram'],int) else hex(r['rdata_vram'])} "
|
||||
f".data@{r['data_vram'] if not isinstance(r['data_vram'],int) else hex(r['data_vram'])}")
|
||||
if r.get("error"):
|
||||
print(" ", r["error"])
|
||||
if not r["ok"] and "ndiff" in r:
|
||||
fd = r["first_diff"]
|
||||
print(f" {r['ndiff']} word(s) differ; first @ {hex(fd) if fd else '(size)'}")
|
||||
if not quiet:
|
||||
print(f" externals recovered ({len(r['externals'])}):")
|
||||
for s, a in sorted(r["externals"].items()):
|
||||
print(f" {s:14s} = 0x{a:08X}")
|
||||
if r["unrecovered"]:
|
||||
print(f" UNRECOVERED undefined: {', '.join(r['unrecovered'])}")
|
||||
sys.exit(0 if r["ok"] else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Link every used object of a PsyQ library and byte-verify each against the BFM EXE.
|
||||
|
||||
Driver over `psyq_identify.py` (placement) + `psyq_link.link_object` (per-object byte link).
|
||||
For a library's ELF-object dir it:
|
||||
1. runs psyq_identify to get {object: text_vram} for the objects the EXE actually links,
|
||||
2. links each object at its vram, recovering externals from the EXE relocations,
|
||||
3. byte-compares each `.text`, prints a PASS/FAIL table + summary,
|
||||
4. aggregates recovered externals across objects and flags any inconsistency (the same
|
||||
symbol recovered to two addresses = a bug to investigate),
|
||||
5. writes a JSON report to .run/psyq_link.<lib>.json (consumed by the build-wiring step 2'.3).
|
||||
|
||||
Usage: psyq_link_lib.py <elf_dir> [text_lo text_hi] e.g. .run/obj40/libcd
|
||||
"""
|
||||
import json, os, re, subprocess, sys
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from psyq_link import link_object # noqa: E402
|
||||
|
||||
EXE = "extracted/retail/SLUS_007.26"
|
||||
|
||||
|
||||
def placement(elf_dir, lo, hi):
|
||||
cmd = ["python3", "tools/psyq_identify.py", elf_dir]
|
||||
if lo and hi:
|
||||
cmd += [lo, hi]
|
||||
out = subprocess.check_output(cmd, text=True)
|
||||
placed = {}
|
||||
for ln in out.splitlines():
|
||||
m = re.match(r"\s+0x([0-9A-Fa-f]+)\s+(\S+\.o)\s+\((\d+) ins\)", ln)
|
||||
if m:
|
||||
placed[m.group(2)] = int(m.group(1), 16)
|
||||
return placed, out
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
sys.exit(__doc__)
|
||||
elf_dir = sys.argv[1]
|
||||
lo = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
hi = sys.argv[3] if len(sys.argv) > 3 else None
|
||||
lib = os.path.basename(elf_dir.rstrip("/"))
|
||||
|
||||
placed, idout = placement(elf_dir, lo, hi)
|
||||
print(idout.strip())
|
||||
print(f"\n=== linking {len(placed)} located {lib} objects ===")
|
||||
|
||||
exe = open(EXE, "rb").read()
|
||||
results, all_ext = [], {}
|
||||
conflicts = []
|
||||
npass = 0
|
||||
for name, vram in sorted(placed.items(), key=lambda kv: kv[1]):
|
||||
obj = os.path.join(elf_dir, name)
|
||||
r = link_object(obj, vram, name=name, exe_bytes=exe)
|
||||
results.append(r)
|
||||
tag = "PASS" if r["ok"] else "FAIL"
|
||||
if r["ok"]:
|
||||
npass += 1
|
||||
extra = ""
|
||||
if not r["ok"]:
|
||||
extra = " " + (r.get("error") or f"{r.get('ndiff','?')} word(s) diff @ "
|
||||
f"{hex(r['first_diff']) if r.get('first_diff') else '(size)'}")
|
||||
print(f" [{tag}] 0x{vram:08X} {name:12s} ({r['tsize']:5d} B){extra}")
|
||||
for s, a in r["externals"].items():
|
||||
if s in all_ext and all_ext[s] != a:
|
||||
conflicts.append((s, all_ext[s], a, name))
|
||||
all_ext.setdefault(s, a)
|
||||
|
||||
print(f"\n{npass}/{len(results)} objects byte-identical (.text)")
|
||||
if conflicts:
|
||||
print(" !! external address conflicts:")
|
||||
for s, a0, a1, nm in conflicts:
|
||||
print(f" {s}: 0x{a0:08X} vs 0x{a1:08X} (at {nm})")
|
||||
print(f" {len(all_ext)} distinct externals recovered across the library")
|
||||
|
||||
rep = {
|
||||
"lib": lib,
|
||||
"elf_dir": elf_dir,
|
||||
"objects": [
|
||||
{"name": r["name"], "text_vram": r["text_vram"], "tsize": r["tsize"],
|
||||
"rdata_vram": r["rdata_vram"] if isinstance(r["rdata_vram"], int) else None,
|
||||
"data_vram": r["data_vram"] if isinstance(r["data_vram"], int) else None,
|
||||
"ok": r["ok"], "externals": {s: a for s, a in r["externals"].items()},
|
||||
"unrecovered": r["unrecovered"]}
|
||||
for r in results
|
||||
],
|
||||
"externals_union": all_ext,
|
||||
"conflicts": [{"sym": s, "a": a0, "b": a1, "at": nm} for s, a0, a1, nm in conflicts],
|
||||
"npass": npass, "ntotal": len(results),
|
||||
}
|
||||
os.makedirs(".run", exist_ok=True)
|
||||
path = f".run/psyq_link.{lib}.json"
|
||||
json.dump(rep, open(path, "w"), indent=1)
|
||||
print(f" report -> {path}")
|
||||
sys.exit(0 if npass == len(results) else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Link a whole PsyQ library's objects into the build IN PLACE OF asm stubs — no data carving.
|
||||
|
||||
The wiring trick (cookbook §9.2): each object's `.text` is LOADED at its EXE vram (the asm
|
||||
stubs there are removed); each object's `.data`/`.rdata`/`.bss` is placed as a **NOLOAD**
|
||||
section at its vram — addresses only, zero bytes — so the build's existing flat `data`
|
||||
subsegment still supplies those bytes (no carve, no double-emit) while the ~hundreds of
|
||||
section-relative `.text` references resolve via the NOLOAD placement. Mislabelled common-style
|
||||
`.bss` globals are `--weaken-symbol`'d so a strong `--defsym` to their real (scattered) address
|
||||
wins (cookbook §9.1). Symbols that no linked object defines (other libraries' functions, module
|
||||
data globals) are emitted for `symbols.us.txt`.
|
||||
|
||||
This module both VERIFIES the region links byte-identical and EMITS the build artifacts:
|
||||
--emit <prefix> -> <prefix>.ld (text lines + NOLOAD sections, for the main linker script)
|
||||
<prefix>.syms (NAME = 0xADDR; external symbols for symbols.us.txt)
|
||||
|
||||
Usage: psyq_link_region.py <elf_dir> [text_lo text_hi] [--emit <prefix>]
|
||||
"""
|
||||
import json, os, re, subprocess, sys, tempfile
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from psyq_link import (section_table, symbol_table, recover_sym_addrs, unique_byte_vram,
|
||||
VRAM_BASE, DATA_SECTIONS, AS, sh)
|
||||
|
||||
EXE = "extracted/retail/SLUS_007.26"
|
||||
|
||||
|
||||
def placement(elf_dir, lo, hi):
|
||||
cmd = ["python3", "tools/psyq_identify.py", elf_dir] + ([lo, hi] if lo else [])
|
||||
placed = {}
|
||||
for ln in subprocess.check_output(cmd, text=True).splitlines():
|
||||
m = re.match(r"\s+0x([0-9A-Fa-f]+)\s+(\S+\.o)\s+\((\d+) ins\)", ln)
|
||||
if m:
|
||||
placed[m.group(2)] = (int(m.group(1), 16), int(m.group(3)))
|
||||
return placed
|
||||
|
||||
|
||||
def classify(obj, text_vram, exe):
|
||||
"""Per-object: NOLOAD section bases, and the .bss/.sbss symbols to weaken.
|
||||
|
||||
Every named symbol psyq-obj-parser put in .bss/.sbss is a common-style global the original
|
||||
linker scattered (their st_values are not honoured). Weaken them ALL so a strong --defsym to
|
||||
the recovered address wins over the (placed or discarded) weak definition — uniform across the
|
||||
genuine (CD_*) and mislabelled (St*) cases, and across objects (a .bss symbol of object A
|
||||
referenced by object B).
|
||||
"""
|
||||
secs = section_table(obj)
|
||||
symtab = symbol_table(obj)
|
||||
sym_addr = recover_sym_addrs(obj, text_vram, exe)
|
||||
|
||||
bases = {}
|
||||
for S in DATA_SECTIONS:
|
||||
if S in secs and secs[S][0] > 0:
|
||||
b = unique_byte_vram(obj, S, exe) if S not in (".bss", ".sbss") else None
|
||||
if b is None:
|
||||
b = sym_addr.get(S) # the object referenced the section symbol
|
||||
if b is not None:
|
||||
bases[S] = b
|
||||
weaken = [s for s, (sec, _) in symtab.items() if sec in (".bss", ".sbss")]
|
||||
return bases, weaken, sym_addr
|
||||
|
||||
|
||||
def defined_text_syms(obj):
|
||||
out = sh(f"{AS}readelf", "-s", obj).stdout.decode()
|
||||
names = set()
|
||||
secs = {}
|
||||
# map section index -> name to find .text-defined symbols
|
||||
for ln in sh(f"{AS}readelf", "-S", obj).stdout.decode().splitlines():
|
||||
m = re.match(r"\s*\[\s*(\d+)\]\s+(\.\S+)", ln)
|
||||
if m:
|
||||
secs[m.group(1)] = m.group(2)
|
||||
for ln in out.splitlines():
|
||||
p = ln.split()
|
||||
if len(p) >= 8 and p[0].endswith(":") and re.fullmatch(r"[0-9a-f]+", p[1]):
|
||||
if p[6].isdigit() and secs.get(p[6]) == ".text" and not p[7].startswith("."):
|
||||
names.add(p[7])
|
||||
return names
|
||||
|
||||
|
||||
def build_region(elf_dir, lo=None, hi=None, emit=None):
|
||||
exe = open(EXE, "rb").read()
|
||||
placed = placement(elf_dir, lo, hi)
|
||||
order = sorted(placed.items(), key=lambda kv: kv[1][0]) # by vram
|
||||
region_lo = order[0][1][0]
|
||||
region_hi = order[-1][1][0] + order[-1][1][1] * 4
|
||||
|
||||
recovered, weaken_by, bases_by = {}, {}, {}
|
||||
conflicts = []
|
||||
for name, (vram, _) in order:
|
||||
obj = os.path.join(elf_dir, name)
|
||||
bases, weaken, sym_addr = classify(obj, vram, exe)
|
||||
bases_by[name] = bases
|
||||
weaken_by[name] = weaken
|
||||
for s, a in sym_addr.items():
|
||||
if s.startswith("."):
|
||||
continue # section symbols are per-object (NOLOAD-placed)
|
||||
if s in recovered and recovered[s] != a:
|
||||
conflicts.append((s, recovered[s], a))
|
||||
recovered[s] = a
|
||||
|
||||
td = tempfile.mkdtemp(dir=".run")
|
||||
prepared = []
|
||||
for name, (vram, _) in order:
|
||||
dst = os.path.join(td, name)
|
||||
args = []
|
||||
for S in (".text",) + DATA_SECTIONS:
|
||||
args += ["--set-section-alignment", f"{S}=4"]
|
||||
for w in weaken_by[name]:
|
||||
args += ["--weaken-symbol", w]
|
||||
sh(f"{AS}objcopy", *args, os.path.join(elf_dir, name), dst)
|
||||
prepared.append((name, vram, dst))
|
||||
|
||||
# Each object's .text is placed at its EXACT vram (the region is two contiguous libcd
|
||||
# sub-blocks split by a 76-byte non-libcd gap, so naive concatenation would drift past it).
|
||||
def write_ld(path, objref):
|
||||
lines = ["SECTIONS {"]
|
||||
for i, (name, vram, dst) in enumerate(prepared):
|
||||
lines += [f" . = 0x{vram:08X};", f" .t{i} : {{ {objref(dst)}(.text) }}"]
|
||||
n = 0
|
||||
for name, _, dst in prepared:
|
||||
for S, b in bases_by[name].items():
|
||||
lines.append(f" .nl_{n} 0x{b:08X} (NOLOAD) : {{ {objref(dst)}(.{S[1:]}) }}")
|
||||
n += 1
|
||||
lines += [" /DISCARD/ : { *(*) }", "}"]
|
||||
open(path, "w").write("\n".join(lines) + "\n")
|
||||
|
||||
# ---- link: let ld report which symbols are unresolved, then --defsym exactly those ----
|
||||
# (symbols defined by some object's .text, or genuinely placed via NOLOAD, resolve internally.)
|
||||
weaken_all = {w for ws in weaken_by.values() for w in ws}
|
||||
ld = os.path.join(td, "verify.ld")
|
||||
write_ld(ld, lambda d: f'"{d}"')
|
||||
elf = os.path.join(td, "out.elf")
|
||||
base_cmd = [f"{AS}ld", "--no-check-sections", "-T", ld, "-o", elf]
|
||||
# weakened .bss commons resolve to their (wrong) weak def, so they never show as "undefined";
|
||||
# defsym them explicitly. Then a probe link surfaces the remaining true externals (other libs).
|
||||
forced = {s for s in weaken_all if s in recovered}
|
||||
probe = base_cmd + [arg for s in sorted(forced) for arg in ("--defsym", f"{s}=0x{recovered[s]:08X}")]
|
||||
p = subprocess.run(probe, capture_output=True)
|
||||
undef = set(re.findall(r"undefined reference to [`']([^`']+)'", p.stderr.decode()))
|
||||
externals = {s: recovered[s] for s in (forced | undef) if s in recovered}
|
||||
missing = sorted(s for s in undef if s not in recovered)
|
||||
cmd = base_cmd + [arg for s, a in sorted(externals.items())
|
||||
for arg in ("--defsym", f"{s}=0x{a:08X}")]
|
||||
p = subprocess.run(cmd, capture_output=True)
|
||||
ok = (p.returncode == 0)
|
||||
ndiff = badobjs = None
|
||||
if ok:
|
||||
# verify each object's .text bytes individually (gap bytes are not ours to provide)
|
||||
ndiff = 0
|
||||
badobjs = []
|
||||
for i, (name, vram, _) in enumerate(prepared):
|
||||
got = sh(f"{AS}objcopy", "-O", "binary", "--only-section", f".t{i}", elf, "/dev/stdout").stdout
|
||||
want = exe[vram - VRAM_BASE: vram - VRAM_BASE + len(got)]
|
||||
d = sum(1 for j in range(0, min(len(got), len(want)), 4) if got[j:j+4] != want[j:j+4])
|
||||
ndiff += d
|
||||
if d:
|
||||
badobjs.append(name)
|
||||
ok = (ndiff == 0)
|
||||
else:
|
||||
print("ld error:", p.stderr.decode().strip().split("\n")[-1])
|
||||
|
||||
weaken_all = {w for ws in weaken_by.values() for w in ws}
|
||||
print(f"region {os.path.basename(elf_dir)} [0x{region_lo:08X}..0x{region_hi:08X}] "
|
||||
f"{len(order)} objects | conflicts={len(conflicts)} | externals={len(externals)}")
|
||||
print(f" weakened commons: {sorted(weaken_all)}")
|
||||
if missing:
|
||||
print(f" !! {len(missing)} undefined symbols NOT in recovered map: {missing}")
|
||||
print(f" per-object .text byte-identical: {ok}"
|
||||
+ (f" ({ndiff} words differ in {badobjs})" if ndiff else ""))
|
||||
|
||||
if emit and ok:
|
||||
write_ld(emit + ".ld", lambda d: os.path.join("build/psyq", os.path.basename(elf_dir),
|
||||
os.path.basename(d)))
|
||||
with open(emit + ".syms", "w") as f:
|
||||
for s, a in sorted(externals.items(), key=lambda kv: kv[1]):
|
||||
f.write(f"{s} = 0x{a:08X};\n")
|
||||
print(f" emitted {emit}.ld + {emit}.syms ({len(externals)} externals)")
|
||||
return ok
|
||||
|
||||
|
||||
def main():
|
||||
a = [x for x in sys.argv[1:] if not x.startswith("--")]
|
||||
emit = None
|
||||
if "--emit" in sys.argv:
|
||||
emit = sys.argv[sys.argv.index("--emit") + 1]
|
||||
elf_dir = a[0]
|
||||
lo = a[1] if len(a) > 1 else None
|
||||
hi = a[2] if len(a) > 2 else None
|
||||
ok = build_region(elf_dir, lo, hi, emit)
|
||||
sys.exit(0 if ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Split a splat src/*.c at vram boundaries, preserving matched C and NON_MATCHING blocks (H5).
|
||||
|
||||
When a text subsegment is resegmented (e.g. to swap a library region's stubs for real objects),
|
||||
the existing curated .c must be trimmed and its matched functions relocated — splat will NOT
|
||||
overwrite an existing .c, so a stale .c would place functions at the wrong addresses. This tool
|
||||
partitions the file's top-level items (INCLUDE_ASM stubs, real C functions, and #ifdef
|
||||
NON_MATCHING blocks) by each item's vram:
|
||||
|
||||
trim <src.c> <lo> <hi> <move.frag>
|
||||
rewrite <src.c> keeping only items with vram < lo (the header is preserved verbatim);
|
||||
items in [lo, hi) are dropped (superseded by the linked objects); real-C items with
|
||||
vram >= hi are written to <move.frag> for injection into the post-region file.
|
||||
|
||||
inject <dst.c> <move.frag>
|
||||
in the freshly splat-generated <dst.c>, replace each moved function's INCLUDE_ASM stub
|
||||
with its real C body (matched by function name).
|
||||
|
||||
Item vram comes from a func_XXXXXXXX name, else a name looked up in config/symbols.us.txt.
|
||||
Brace matching is naive (counts {}); the build's SHA1 check is the backstop if an item with
|
||||
string-literal braces is mis-split.
|
||||
"""
|
||||
import re, sys
|
||||
|
||||
SYMS_PATH = "config/symbols.us.txt"
|
||||
|
||||
|
||||
def load_syms():
|
||||
s = {}
|
||||
for ln in open(SYMS_PATH):
|
||||
m = re.match(r"(\w+)\s*=\s*0x([0-9A-Fa-f]+)", ln)
|
||||
if m:
|
||||
s[m.group(1)] = int(m.group(2), 16)
|
||||
return s
|
||||
|
||||
|
||||
def item_name(text):
|
||||
m = re.search(r"INCLUDE_ASM\([^,]+,\s*(\w+)\)", text)
|
||||
if m:
|
||||
return m.group(1)
|
||||
m = re.search(r"^\s*(?:static\s+)?[\w\*]+[\s\*]+(\w+)\s*\(", text, re.M)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def item_addr(text, syms):
|
||||
name = item_name(text)
|
||||
if name:
|
||||
m = re.match(r"func_([0-9A-Fa-f]{8})$", name)
|
||||
if m:
|
||||
return int(m.group(1), 16)
|
||||
if name in syms:
|
||||
return syms[name]
|
||||
return None
|
||||
|
||||
|
||||
def parse(src):
|
||||
"""Return (header, [item_text, ...]) splitting top-level items."""
|
||||
lines = src.split("\n")
|
||||
n = len(lines)
|
||||
i = 0
|
||||
while i < n and (lines[i].startswith("#include") or lines[i].strip() == ""):
|
||||
i += 1
|
||||
header = "\n".join(lines[:i]).rstrip("\n")
|
||||
items = []
|
||||
while i < n:
|
||||
if lines[i].strip() == "":
|
||||
i += 1
|
||||
continue
|
||||
start = i
|
||||
# leading line/block comments belong to the following item
|
||||
while i < n and (lines[i].lstrip().startswith("//") or lines[i].lstrip().startswith("/*")
|
||||
or lines[i].lstrip().startswith("*")):
|
||||
if lines[i].lstrip().startswith("/*") and "*/" not in lines[i]:
|
||||
while i < n and "*/" not in lines[i]:
|
||||
i += 1
|
||||
i += 1
|
||||
if i >= n:
|
||||
break
|
||||
l = lines[i]
|
||||
if l.startswith("#ifdef NON_MATCHING"):
|
||||
while i < n and not lines[i].startswith("#endif"):
|
||||
i += 1
|
||||
i += 1
|
||||
elif l.lstrip().startswith("INCLUDE_ASM"):
|
||||
i += 1
|
||||
else:
|
||||
depth = 0
|
||||
seen = False
|
||||
while i < n:
|
||||
depth += lines[i].count("{") - lines[i].count("}")
|
||||
if "{" in lines[i]:
|
||||
seen = True
|
||||
i += 1
|
||||
if seen and depth <= 0:
|
||||
break
|
||||
items.append("\n".join(lines[start:i]))
|
||||
return header, items
|
||||
|
||||
|
||||
def is_real_c(text):
|
||||
return "INCLUDE_ASM" not in text or text.startswith("#ifdef NON_MATCHING")
|
||||
|
||||
|
||||
def trim(srcpath, lo, hi, movepath):
|
||||
syms = load_syms()
|
||||
header, items = parse(open(srcpath).read())
|
||||
keep, drop, move = [], 0, []
|
||||
for it in items:
|
||||
a = item_addr(it, syms)
|
||||
if a is None:
|
||||
sys.exit(f"trim: cannot resolve address of item:\n{it[:80]}")
|
||||
if a < lo:
|
||||
keep.append(it)
|
||||
elif a < hi:
|
||||
drop += 1
|
||||
else:
|
||||
if is_real_c(it):
|
||||
move.append((a, item_name(it), it))
|
||||
open(srcpath, "w").write(header + "\n\n" + "\n\n".join(keep) + "\n")
|
||||
move.sort()
|
||||
with open(movepath, "w") as f:
|
||||
f.write("\n\n".join(it for _, _, it in move) + "\n")
|
||||
print(f"trim {srcpath}: kept {len(keep)} (<0x{lo:X}), dropped {drop} stubs/superseded "
|
||||
f"[0x{lo:X},0x{hi:X}), moved {len(move)} real-C funcs (>=0x{hi:X}) -> {movepath}")
|
||||
for a, nm, _ in move:
|
||||
print(f" move 0x{a:08X} {nm}")
|
||||
|
||||
|
||||
def inject(dstpath, movepath):
|
||||
_, moved = parse(open(movepath).read())
|
||||
by_name = {item_name(it): it for it in moved}
|
||||
dst = open(dstpath).read()
|
||||
done = []
|
||||
for name, body in by_name.items():
|
||||
# replace the whole stub line (optionally wrapped) for this function
|
||||
pat = re.compile(r"^INCLUDE_ASM\([^,]+,\s*" + re.escape(name) + r"\);\s*$", re.M)
|
||||
if pat.search(dst):
|
||||
dst = pat.sub(lambda m: body, dst, count=1)
|
||||
done.append(name)
|
||||
open(dstpath, "w").write(dst)
|
||||
print(f"inject {dstpath}: replaced {len(done)} stubs with matched C: {done}")
|
||||
missing = [n for n in by_name if n not in done]
|
||||
if missing:
|
||||
sys.exit(f"inject: stubs not found for {missing}")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) >= 5 and sys.argv[1] == "trim":
|
||||
trim(sys.argv[2], int(sys.argv[3], 0), int(sys.argv[4], 0), sys.argv[5])
|
||||
elif len(sys.argv) == 4 and sys.argv[1] == "inject":
|
||||
inject(sys.argv[2], sys.argv[3])
|
||||
else:
|
||||
sys.exit(__doc__)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user