mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 21:36:06 -04:00
feat(phase-8): library survey + progress.py LINKED category (T1-T2)
- T1: docs/psyq-worklist.md — byte-confirmed PsyQ footprint map (psyq_identify over the original EXE): 9 footprint libs (199 placements) ranked ascending difficulty; 3 skip (libmath/libc/libsn = 0 footprint). Findings: the libgs "gaps" are libgte objects; heavy game/library interleaving (800=game+libspu+ libsnd+libetc, 800b~libgte, 800b2=game+libgpu/libc2/libapi/libmcrd/libcard); cross-library aliases (C112, UT_RON/S_IH, S_R/S_W) - T2: tools/progress.py LINKED category — INCLUDE_ASM stubs in integrated library subsegs counted as a distinct byte-identical bucket; linked-subseg set sourced from the Makefile's psyq_integrate calls (single source of truth -> auto-updates per integration, no manifest drift). libcd+libgs: 340 stubs -> LINKED; byte-identical/matchable now 425/2093 = 20.31%. Build unchanged (143dbb89) - phase-ends/CURRENT_PHASE.md: Phase 8 plan + per-task log
This commit is contained in:
Drew T
+4
-4
@@ -1,18 +1,18 @@
|
||||
# BFM matching progress (generated by tools/progress.py — authoritative)
|
||||
|
||||
REAL substantive matches : 43 <- the Gen1-exit >=25 bar counts THIS
|
||||
LINKED real PsyQ objects : 340 <- byte-identical via linked SDK objects
|
||||
NON_MATCHING (near-miss) : 7
|
||||
splat-auto empty no-ops : 42
|
||||
INCLUDE_ASM stubs : 2001
|
||||
INCLUDE_ASM stubs : 1661
|
||||
data blobs (excluded) : 2
|
||||
----------------------------------------
|
||||
matchable functions : 2093
|
||||
REAL / matchable : 43 / 2093 = 2.05%
|
||||
incl. empties : 85 / 2093 = 4.06%
|
||||
byte-identical/ matchable: 425 / 2093 = 20.31% (REAL+LINKED+empties)
|
||||
|
||||
LINKED subsegs: libcd1 libcd2 libgs1 libgs2 libgs3 libgs4 libgs5 libgs6 (340 fns)
|
||||
REAL matches: CdQueueBusy CdReadRequest DebugMenuHandler GameModeDispatch LoaderResetReadState LzssDecodeSector ResourceGetCdLoc func_80012AB0 func_800168B4 func_80018F20 func_80019018 func_80019198 func_80019378 func_80019388 func_80019398 func_800193A8 func_8001AA78 func_8001AA88 func_8001B22C func_8001B374 func_8001B384 func_8001B85C func_8001BE20 func_8001BFA0 func_8001BFE8 func_8001D0E8 func_80029254 func_80029264 func_80029504 func_80029514 func_8002953C func_8002954C func_8002957C func_8002958C func_80029FD4 func_8002A26C func_8002A27C func_8002A4B8 func_8002A4C8 func_8002A728 func_8002A738 func_8002A998 func_8002A9A8
|
||||
NON_MATCHING: CdReadSectorReadyCB CdReadStateMachine LoaderInitFileTable ResourceLoadStateMachine StreamLoadStateMachine func_80015A74 func_80016714
|
||||
|
||||
build SHA1: 143dbb89f34491258bbc27810d0a12ec8b43a8dd (byte-identical)
|
||||
|
||||
empties audit: 42/42 genuine jr;nop (all clean)
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# PsyQ library integration worklist (Phase 8)
|
||||
|
||||
> Byte-confirmed footprint map of the remaining PsyQ 4.0 libraries linked by `SLUS_007.26`.
|
||||
> Generated by `psyq_identify.py` over the **original** EXE (`extracted/retail/SLUS_007.26`,
|
||||
> text `0x80010000..0x800629DC`); raw per-object placements in `.run/survey/<lib>.txt` (regenerable:
|
||||
> `for L in …; do python3 tools/psyq_identify.py .run/obj40/$L; done`). Replaces the Phase-7 ~350-fn
|
||||
> *estimate* (R14). **Placement = candidate footprint; per-object byte-identity is confirmed at
|
||||
> integration time by `psyq_link_region.py --verify` before wiring (the real gate, G3/P9).**
|
||||
>
|
||||
> **DONE (Gen1/Phase 7):** libcd (18 objs) + libgs (31 objs) linked byte-identical.
|
||||
|
||||
## Footprint summary (remaining libraries)
|
||||
|
||||
| Order | Library | Placed / total | Region(s) | ~banked B | Difficulty notes |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | **libgpu** | 3 / 12 | 800b2 (`0x80058890`) | ~14.9k | EXT→PRIM→SYS **contiguous single block**; SYS.o=3109 ins. Cleanest. |
|
||||
| 2 | **libetc** | 5 / 7 | 800 tail (`0x8004239C`) | ~3.3k | VSYNC/INTR/INTR_VB/INTR_DMA/VMODE **contiguous**, ends exactly at libcd1 (`0x80043088`). Clean. |
|
||||
| 3 | **libmcrd** | 2 / 2 | 800b2 (`0x8005FC68`,`0x80062888`) | ~9.0k | LIBMCRD.o (2186 ins, huge, holds the 55 `LIBMCRD_OBJ_*`) + USERFUNC.o — **2 blocks**. `_card_*` h_norm dups collapse inside LIBMCRD. |
|
||||
| 4 | **libc2** | 17 / 46 | 800b2 (`0x8005C2C8`–`0x5CD98` + STRCAT `0x80061E90`) | ~3.1k | C stdlib (BZERO/MEMCPY/STRCMP/PRINTF/PRNT…); contiguous run + 1 outlier. **PRNT.o (418 ins) has an internal jtbl** — verify NOLOAD `.rodata` placement (the `PRNT_OBJ_24C` rodata note). |
|
||||
| 5 | **libcard** | 11 / 18 | 800b2 (`0x80061F38`–`0x80062808`) | ~0.5k | 11 short objects (mostly 4–46 ins). **C112.o alias with libapi** (both claim `0x80061F38`). Short objects → windowed placement. |
|
||||
| 6 | **libapi** | 33 / 90 | 800b2 (scattered `0x8005CE18`–`0x800626B8`) | ~0.5k | Mostly **4-ins BIOS syscall stubs** (A##/C##/L##); interleaved with libcard/libc2/libmcrd. 1 ambiguous (SEND), C112 alias. **Windowed + alias disambiguation.** |
|
||||
| 7 | **libspu** | 38 / 129 | 800 sound (`0x8003A444`–`0x800422E8`) | ~12k | SPU.o(741)/S_SRMP(318)/SR_SV(399)/S_SCA(229)…; **interleaved with libsnd**; aliases (S_R≡S_W @`0x8003C438`; S_GRMDT/FB/T @`0x8003D424`; S_IH @`0x8003D94C`≡libsnd UT_RON). 1 ambiguous (S_I). Windowed. |
|
||||
| 8 | **libsnd** | 32 / 163 | 800 (`0x8003D454`–`0x80042374` + SSGM `0x8001BD80` isolated) | ~12k | SSSTART(206)/MIDIREAD(445)/VS_VH(283)/UT_KEYV(259)…; **interleaved with libspu**; SSINIT_C≡SSINIT_H @`0x8003D518`; UT_RON @`0x8003D94C`. 6 ambiguous (PLAY/SSNOFF/SSQUIT/UT_ROFF/VM_DOFF/VM_VIB). Windowed + isolated SSGM block. |
|
||||
| 9 | **libgte** | 58 / 381 | 800b (`0x8004787C`–`0x5082C`) + **libgs gaps** | ~48k | Largest. MTX/GEO/COR/PRS/F/G/T series; **multiple blocks** (game code interleaved in 800b, e.g. between FGO_00 and PRS_F3). **Fills gsgap1=MTX_05, gsgap2≈MTX_07, gsgap4=MTX_11, gsgap5=REG03+REG11.** Multi-region, do last. |
|
||||
|
||||
**SKIP — zero footprint (recorded, not linked by the EXE):** `libmath` 0/48, `libc` 0/56, `libsn` 0/51 (2 tiny ambiguous). BFM links **libc2**, not libc; no libmath/libsn. (Also unbuilt/no-footprint: libcomb, libds, libgun, libsio, libtap, libpress — never converted, no symbols.)
|
||||
|
||||
## Region map (where each library lives, relative to the current splat subsegs)
|
||||
|
||||
```
|
||||
800 (0x800123F0–0x80043088) game code + libsnd(SSGM @1BD80; 3D454–42374) + libspu(3A444–422E8) + libetc(4239C–43088, tail)
|
||||
libcd (0x80043088–0x8004787C) DONE
|
||||
800b (0x8004787C–0x80051804) libgte(4787C–5082C, multi-block) + game-code tail
|
||||
libgs (0x80051804–0x80057928) DONE; gaps gsgap1/2/4/5 are actually libgte (MTX_05/07/11, REG03/11); gsgap3=GS_001 excluded
|
||||
800b2 (0x80057928–0x800629DC) game code + libgpu(58890–5Bxxx) + libc2(5C2C8–5CD98,+STRCAT 61E90) + libapi(5CE18–626B8 scattered) + libmcrd(5FC68 + 62888) + libcard(61F38–62808)
|
||||
```
|
||||
|
||||
## Integration order rationale (ascending difficulty)
|
||||
- **1–2 (libgpu, libetc):** clean single contiguous blocks, minimal interleaving → re-validate the §9 mechanism end-to-end on a fresh library before the hard ones.
|
||||
- **3–4 (libmcrd, libc2):** few blocks, contained in 800b2; libc2 introduces the PRNT internal-jtbl wrinkle.
|
||||
- **5–6 (libcard, libapi):** short syscall stubs + cross-library aliases + scatter in 800b2 → need windowed placement + curated `_used` dirs.
|
||||
- **7–8 (libspu, libsnd):** the interleaved sound pair in the 800 subseg (which still holds the 43 REAL game-code matches lower down) → `split_src_region.py` trims, windowed placement, alias disambiguation.
|
||||
- **9 (libgte):** largest, multi-region (800b + 4 libgs gaps), game-code-interleaved blocks → last.
|
||||
|
||||
## Per-library integration recipe (each = one byte-gated task, §9.3/§9.5)
|
||||
1. `psyq_link_region.py .run/obj40/<lib> <lo> <hi> --verify` → confirm per-object byte-identity + the block/gap structure (G1: derive blocks empirically, don't trust this table's addresses).
|
||||
2. Curate `.run/obj40/<lib>_used` (drop cross-library aliases / keep the variant whose `.data`/`.rdata` also matches) when aliases exist (make_libgs.sh pattern).
|
||||
3. Resegment `config/splat.us.exe.yaml` (one `c` subseg per contiguous block; non-library gaps stay stubs); `split_src_region.py trim` the affected game-code `.c`; `make extract`.
|
||||
4. Makefile: add `LIB<X>_ELF/OBJDIR/SYMS` + a gated `psyq_integrate.py … <stubs> [lo hi]` call + `-T $(LIB<X>_SYMS)` on the `ld` line.
|
||||
5. Add recovered SDK names to `config/symbols.us.txt` (mirror to Ghidra, R15/G6); update progress.py's LINKED config.
|
||||
6. **Dual byte-gate:** `make clean && extract && build && check` → `143dbb89…` WITH `.run/obj40/<lib>` present AND with it absent (stub fallback). `make report` LINKED up. Commit (R20).
|
||||
|
||||
**Cross-cutting:** windowed placement is mandatory for the short-object / interleaved libraries (the libgs GS_106 precedent); cross-library address collisions (C112 libapi/libcard; UT_RON/S_IH; S_R/S_W) mean total distinct objects < the 199 summed placements — byte-verify decides ownership. Any object that won't link byte-identical after §9.1/§9.4 stays a documented honest stub (does not block exit, P9).
|
||||
@@ -0,0 +1,34 @@
|
||||
# CURRENT PHASE — Phase 8: Finish the EXE — remaining PsyQ libraries + final harvest
|
||||
|
||||
> **Gen2, Phase 8** (first Gen2 phase). **Plan APPROVED by Drew 2026-06-15** (gate 1 passed).
|
||||
> Full plan: `/home/musashi/.claude/plans/plan-mode-enabled-max-wobbly-manatee.md` (mirrored here as the in-repo crash-recovery log, P3).
|
||||
> **Execution effort:** xHigh default; conditional Max if a library fights past cookbook §9; Max for the PhaseEnd.
|
||||
> **Owner decisions (2026-06-15):** progress metric = **add a LINKED category to progress.py**; exit bar = **full matching ceiling** (link every footprint-confirmed library; defer any that fight as honest documented stubs).
|
||||
> **Commit cadence:** per-session/per-task checkpoints, commit + (Drew) push irreplaceable work (R20, loosens R8).
|
||||
|
||||
## Goal / milestone (byte-checkable gate)
|
||||
1. `make clean && extract && build && check` → SHA1 `143dbb89…` byte-identical **with** all newly-linked SDK objects AND **without** them (`.run/obj40/` absent → stub fallback) — both green.
|
||||
2. `make report` shows REAL / **LINKED** / NON_MATCHING / stubs; LINKED grown by the newly-linked objects, stubs reduced accordingly, REAL up by the harvested game leaves; deterministic.
|
||||
3. **0 NON_MATCHING in the default build** preserved (G4).
|
||||
4. Every footprint-confirmed library is linked OR documented as an honest deferral (`docs/psyq-worklist.md`) — no silent gaps (P9).
|
||||
5. `git status` shows zero ROM-derived/generated bulk staged.
|
||||
|
||||
## Tasks
|
||||
- [x] **T0** — Write this `CURRENT_PHASE.md` (kickoff). *(Tier-3)*
|
||||
- [x] **T1** — Library placement survey + worklist → `docs/psyq-worklist.md`. **DONE:** 9 footprint libs (libgpu/libetc/libmcrd/libc2/libcard/libapi/libspu/libsnd/libgte = 199 placements) ranked ascending-difficulty; 3 SKIP (libmath/libc/libsn = 0 footprint); key findings: libgs "gaps" gsgap1/2/4/5 are libgte; heavy interleaving (800=game+libspu+libsnd+libetc, 800b≈libgte, 800b2=game+libgpu/libc2/libapi/libmcrd/libcard); cross-lib aliases (C112, UT_RON/S_IH, S_R/S_W).
|
||||
- [x] **T2** — `progress.py` LINKED category. **DONE:** linked subsegs sourced from the Makefile's `psyq_integrate` calls (single source of truth, auto-updates per integration); libcd+libgs **340** stubs→LINKED; byte-identical/matchable now **425/2093 = 20.31%**; build unchanged (143dbb89).
|
||||
- [ ] **T3…N** — Per-library integrate, ascending difficulty (resegment splat → trim src → Makefile gated psyq_integrate + `-T` syms → symbols+Ghidra mirror → dual byte-gate → report → commit). *(xHigh; Max-escalate if a lib fights)* **→ report #2 after first lib lands; #3 when last placed lib done**
|
||||
- [ ] **T(N+1)** — Deferred-library hard-case documentation (honest stub deferrals, structured comments, worklist notes). *(xHigh)*
|
||||
- [ ] **T(N+2)** — Game-code harvest (read cookbook+§5.4 first; sig-refresh → difficulty.md regen → cheap non-jtbl game leaves; bank, don't exhaust). *(xHigh; optional breadth)* **→ report #4**
|
||||
- [ ] **T(N+3)** — Headers (`include/psyq/*.h`) + cookbook §9 flywheel (R16) + SETUP/memory-map refresh (R21). *(xHigh)*
|
||||
- [ ] **T(N+4)** — Phase 8 close: verify checkboxes (P7) → demo milestone → Drew confirms (gate 2) → PhaseEnd_Phase8.md + archive worklog (R19) + commit + Ghidra push (R20) + plain-English recap (R18) + hard stop (P8). *(Tier-1 Max)*
|
||||
|
||||
## Current task pointer
|
||||
**T3 — Integrate libgpu** (3 contiguous objs EXT/PRIM/SYS @0x80058890, 800b2; cleanest first integration to re-validate the §9 mechanism). **→ progress report #2 after it lands.**
|
||||
|
||||
## Progress log
|
||||
- **2026-06-15 (session A, planning):** Read PROJECT_CONTEXT + all PhaseEnds + effort-map + gen2-roadmap. Byte-grounded the Phase-8 scope (R14): harvest is ~90% library objects; progress.py counts linked libs as stubs; machinery ready (20 `.LIB`s, 14 `.a`, 14 built `.run/obj40/*` dirs). Two owner decisions taken (LINKED category; full ceiling). Plan approved (gate 1). **T0 done** — this file written.
|
||||
- **2026-06-15 (session A, exec):** Baseline confirmed byte-identical (143dbb89). **T1 done** — `psyq_identify` survey → `docs/psyq-worklist.md` (9 footprint libs / 199 placements ranked; 3 skip). **T2 done** — progress.py LINKED category (340 fns; byte-identical 20.31%; build unchanged). Built the 16-task monitor list. Committed T1+T2 foundation checkpoint.
|
||||
|
||||
## Blockers
|
||||
- None.
|
||||
+37
-9
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env python3
|
||||
"""BFM matching-progress report (authoritative; supersedes the Phase-6 PhaseEnd estimate).
|
||||
|
||||
Classifies every matchable function in src/800.c and prints a deterministic summary
|
||||
Classifies every matchable function across all src/*.c and prints a deterministic summary
|
||||
(also written to docs/progress.md). Ghidra-free. The REAL count is the number the
|
||||
Gen1-exit "≥25 matched functions" bar counts (splat-auto empties do NOT count).
|
||||
|
||||
@@ -18,6 +18,25 @@ ASM_ROOT = ROOT / "asm" / "nonmatchings" # per-segment subdirs (boot/, 800/
|
||||
OUT = ROOT / "docs" / "progress.md"
|
||||
BUILD = ROOT / "build" / "us" / "SLUS_007.26"
|
||||
CHECK = ROOT / "config" / "check.us.sha"
|
||||
MAKEFILE = ROOT / "Makefile"
|
||||
|
||||
def linked_subsegs():
|
||||
"""Library subsegments swapped to real PsyQ objects at build time = the 5th positional arg
|
||||
of each `psyq_integrate.py` call in the Makefile (e.g. `libcd1,libcd2`). Parsing the Makefile
|
||||
keeps it the SINGLE source of truth: a new library integration adds its psyq_integrate call and
|
||||
its stubs are auto-counted LINKED (no separate manifest to drift). Stubs in these subsegs build
|
||||
byte-identically from the real SDK objects when present, and from the committed asm fallback
|
||||
otherwise (fresh clone) — so they are LINKED regardless of local .run/obj40 state (a property of
|
||||
the project, not the machine)."""
|
||||
if not MAKEFILE.exists():
|
||||
return set()
|
||||
segs = set()
|
||||
for m in re.finditer(r'psyq_integrate\.py\s+\S+\s+\S+\s+\S+\s+\S+\s+([A-Za-z0-9_,]+)',
|
||||
MAKEFILE.read_text()):
|
||||
segs.update(m.group(1).split(','))
|
||||
return segs
|
||||
|
||||
LINKED_SEGS = linked_subsegs()
|
||||
|
||||
def find_s(name):
|
||||
"""Locate <name>.s in any asm/nonmatchings/<seg>/ subdir (segments: boot, 800, ...)."""
|
||||
@@ -54,7 +73,7 @@ def asm_is_trivial(name):
|
||||
SIG = re.compile(r'^\s*[A-Za-z_][\w \t\*]*\b([A-Za-z_]\w*)\s*\(')
|
||||
|
||||
def classify():
|
||||
real, empty, nonmatching, stubs, blobs = [], [], [], [], []
|
||||
real, empty, nonmatching, stubs, blobs, linked = [], [], [], [], [], []
|
||||
for src in SRCS:
|
||||
lines = src.read_text().split('\n')
|
||||
n = len(lines); i = 0
|
||||
@@ -68,9 +87,16 @@ def classify():
|
||||
m = re.search(r'INCLUDE_ASM\("[^"]+",\s*(\w+)\)', '\n'.join(blk))
|
||||
if m: nonmatching.append(m.group(1))
|
||||
continue
|
||||
m = re.match(r'INCLUDE_ASM\("[^"]+",\s*(\w+)\)', s)
|
||||
m = re.match(r'INCLUDE_ASM\("([^"]+)",\s*(\w+)\)', s)
|
||||
if m:
|
||||
(blobs if is_data_blob(m.group(1)) else stubs).append(m.group(1)); i += 1; continue
|
||||
seg, fn = m.group(1).rstrip('/').rsplit('/', 1)[-1], m.group(2)
|
||||
if is_data_blob(fn):
|
||||
blobs.append(fn)
|
||||
elif seg in LINKED_SEGS: # subseg swapped to a real PsyQ object (byte-identical)
|
||||
linked.append(fn)
|
||||
else:
|
||||
stubs.append(fn)
|
||||
i += 1; continue
|
||||
if s.startswith('INCLUDE_RODATA'):
|
||||
i += 1; continue
|
||||
fm = SIG.match(lines[i])
|
||||
@@ -97,19 +123,20 @@ def classify():
|
||||
(real if strip_comments(body[a+1:b]).strip() else empty).append(fm.group(1))
|
||||
continue
|
||||
i += 1
|
||||
return real, empty, nonmatching, stubs, blobs
|
||||
return real, empty, nonmatching, stubs, blobs, linked
|
||||
|
||||
def main():
|
||||
audit = '--audit' in sys.argv
|
||||
check = '--check' in sys.argv
|
||||
real, empty, nonmatching, stubs, blobs = classify()
|
||||
matchable = len(real) + len(empty) + len(nonmatching) + len(stubs)
|
||||
incl = len(real) + len(empty)
|
||||
real, empty, nonmatching, stubs, blobs, linked = classify()
|
||||
matchable = len(real) + len(empty) + len(nonmatching) + len(stubs) + len(linked)
|
||||
byteident = len(real) + len(linked) + len(empty) # all byte-identical in the build
|
||||
|
||||
out = []
|
||||
out.append("# BFM matching progress (generated by tools/progress.py — authoritative)")
|
||||
out.append("")
|
||||
out.append(f"REAL substantive matches : {len(real):5d} <- the Gen1-exit >=25 bar counts THIS")
|
||||
out.append(f"LINKED real PsyQ objects : {len(linked):5d} <- byte-identical via linked SDK objects")
|
||||
out.append(f"NON_MATCHING (near-miss) : {len(nonmatching):5d}")
|
||||
out.append(f"splat-auto empty no-ops : {len(empty):5d}")
|
||||
out.append(f"INCLUDE_ASM stubs : {len(stubs):5d}")
|
||||
@@ -117,8 +144,9 @@ def main():
|
||||
out.append("-" * 40)
|
||||
out.append(f"matchable functions : {matchable:5d}")
|
||||
out.append(f"REAL / matchable : {len(real)} / {matchable} = {100*len(real)/matchable:.2f}%")
|
||||
out.append(f"incl. empties : {incl} / {matchable} = {100*incl/matchable:.2f}%")
|
||||
out.append(f"byte-identical/ matchable: {byteident} / {matchable} = {100*byteident/matchable:.2f}% (REAL+LINKED+empties)")
|
||||
out.append("")
|
||||
out.append("LINKED subsegs: " + " ".join(sorted(LINKED_SEGS)) + f" ({len(linked)} fns)")
|
||||
out.append("REAL matches: " + " ".join(sorted(real)))
|
||||
out.append("NON_MATCHING: " + " ".join(sorted(nonmatching)))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user