mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 14:02:04 -04:00
Drew T
329ab4cbe7
TWO REAL BUGS in classify(), and an HONEST CORRECTION of their blast radius (P9/R14).
- BUG 1 (under-count). classify() decides definition-vs-declaration by scanning to the first `{` or
`;`. A K&R definition puts its parameter declarations BEFORE the brace:
s32 func_8015AE2C(arg0)
s32 arg0; <- a `;` before the `{`
{ ... }
so it was read as a forward declaration and dropped into NO bucket — not REAL, not a stub,
invisible. And a K&R def is MANDATORY whenever a zero-arg engine_core.h thunk calls the function,
i.e. exactly the heavy-jr cores our own banking recipe produces: func_8015AE2C (562x134),
func_8015A3C8 (493x132), func_80166994 (369x134) were all compiled, linked and BYTE-IDENTICAL in
the shipped build while counting as zero. Fix: skip over K&R parameter declarations (a bare
`<type> <name>;` carrying no parens — that is what distinguishes it from a wrapped ANSI
prototype's continuation line, which always carries the `)`).
- BUG 2 (over-count). `real |= dedup_members(BINARY)` folded in EVERY registered dedup member without
checking it is actually instantiated. A member still sitting as an INCLUDE_ASM stub was counted
REAL *and* stayed in `stubs` — double-counting into `matchable` and inflating `byteident`
(532 phantom instances, per the scanner audit). Fix: subtract `stubs`. The registry is advisory;
the source tree is authoritative.
- COVERAGE ASSERTION (the rule ratified 2026-07-14): ground truth = every function splat emitted a
.s for. Anything classify() cannot place in ANY bucket is now reported LOUDLY (stderr + the .md),
because a silent skip is a defect, not a no-op. Currently: 0 unplaced.
- CORRECTION (this is the part that matters — I over-claimed and the bytes refuted me). The scanner
audit reported ~243k instructions "counted as nothing", and I repeated it. WRONG. weighted_metrics()
— which produces the HEADLINE instr-weighted and distinct-code numbers — does NOT call classify()
at all. It tests `func not in src_stubs(binary)`: since the fleet is 136/136 byte-identical,
anything not wrapped in INCLUDE_ASM must be compiled C emitting the exact original bytes. That test
never parses a definition, so it is IMMUNE to this bug. Verified: old-vs-new on the same tree gives
identical weighted numbers. The published 65.6% / 44.9% were CORRECT ALL ALONG; only the secondary
REAL count and fn-count % were wrong.
THE LESSON, sharper than the one we started with: a metric DERIVED FROM A PROVEN INVARIANT beats a
metric that RE-PARSES THE WORLD. weighted_metrics() leans on the byte-gate and inherits its
correctness; classify() re-derives the same fact by parsing C and inherited a bug instead. Prefer
the former wherever an invariant exists.
675 lines
52 KiB
Python
675 lines
52 KiB
Python
#!/usr/bin/env python3
|
||
"""BFM matching-progress report (authoritative; supersedes the Phase-6 PhaseEnd estimate).
|
||
|
||
Classifies every matchable function across all src/*.c and prints a deterministic summary
|
||
(also written to docs/progress.md). Ghidra-free. The REAL count is the number the
|
||
Gen1-exit "≥25 matched functions" bar counts (splat-auto empties do NOT count).
|
||
|
||
Usage:
|
||
tools/progress.py # print summary + write docs/progress.md
|
||
tools/progress.py --audit # also verify every empty no-op's asm is exactly {jr,nop}
|
||
tools/progress.py --check # also hash build/us/SLUS_007.26 vs config/check.us.sha
|
||
tools/progress.py --binary <alias> # report a non-default binary (default: main = the EXE)
|
||
"""
|
||
import re, sys, hashlib, pathlib
|
||
|
||
ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||
|
||
# Per-binary report config (Phase 9). `main` = the retail EXE; its paths are the originals
|
||
# (no-op default). A second binary (Phase 10) adds an entry — its build/check + src/asm tree;
|
||
# the overlay src/asm subtree LAYOUT is a Phase-10 decision, not invented here.
|
||
BINARIES = {
|
||
"main": dict(build="build/us/SLUS_007.26", check="config/check.us.sha",
|
||
src="src", asm="asm/nonmatchings", out="docs/progress.md"),
|
||
"resident": dict(build="build/resident/resident", check="config/check.resident.sha",
|
||
src="src/resident", asm="asm/resident/nonmatchings", out="docs/progress.resident.md"),
|
||
"ov_SC01_077": dict(build="build/ov_SC01_077/ov_SC01_077", check="config/check.ov_SC01_077.sha",
|
||
src="src/ov_SC01_077", asm="asm/ov_SC01_077/nonmatchings", out="docs/progress.ov_SC01_077.md"),
|
||
"ov_SC01_005": dict(build="build/ov_SC01_005/ov_SC01_005", check="config/check.ov_SC01_005.sha",
|
||
src="src/ov_SC01_005", asm="asm/ov_SC01_005/nonmatchings", out="docs/progress.ov_SC01_005.md"),
|
||
"ov_SC01_006": dict(build="build/ov_SC01_006/ov_SC01_006", check="config/check.ov_SC01_006.sha",
|
||
src="src/ov_SC01_006", asm="asm/ov_SC01_006/nonmatchings", out="docs/progress.ov_SC01_006.md"),
|
||
"ov_SC03_001": dict(build="build/ov_SC03_001/ov_SC03_001", check="config/check.ov_SC03_001.sha",
|
||
src="src/ov_SC03_001", asm="asm/ov_SC03_001/nonmatchings", out="docs/progress.ov_SC03_001.md"),
|
||
"ov_SC01_000": dict(build="build/ov_SC01_000/ov_SC01_000", check="config/check.ov_SC01_000.sha",
|
||
src="src/ov_SC01_000", asm="asm/ov_SC01_000/nonmatchings", out="docs/progress.ov_SC01_000.md"),
|
||
"ov_SC01_001": dict(build="build/ov_SC01_001/ov_SC01_001", check="config/check.ov_SC01_001.sha",
|
||
src="src/ov_SC01_001", asm="asm/ov_SC01_001/nonmatchings", out="docs/progress.ov_SC01_001.md"),
|
||
"ov_SC02_000": dict(build="build/ov_SC02_000/ov_SC02_000", check="config/check.ov_SC02_000.sha",
|
||
src="src/ov_SC02_000", asm="asm/ov_SC02_000/nonmatchings", out="docs/progress.ov_SC02_000.md"),
|
||
"ov_SC02_003": dict(build="build/ov_SC02_003/ov_SC02_003", check="config/check.ov_SC02_003.sha",
|
||
src="src/ov_SC02_003", asm="asm/ov_SC02_003/nonmatchings", out="docs/progress.ov_SC02_003.md"),
|
||
"ov_SC03_002": dict(build="build/ov_SC03_002/ov_SC03_002", check="config/check.ov_SC03_002.sha",
|
||
src="src/ov_SC03_002", asm="asm/ov_SC03_002/nonmatchings", out="docs/progress.ov_SC03_002.md"),
|
||
"ov_SC03_006": dict(build="build/ov_SC03_006/ov_SC03_006", check="config/check.ov_SC03_006.sha",
|
||
src="src/ov_SC03_006", asm="asm/ov_SC03_006/nonmatchings", out="docs/progress.ov_SC03_006.md"),
|
||
"ov_SC04_000": dict(build="build/ov_SC04_000/ov_SC04_000", check="config/check.ov_SC04_000.sha",
|
||
src="src/ov_SC04_000", asm="asm/ov_SC04_000/nonmatchings", out="docs/progress.ov_SC04_000.md"),
|
||
"ov_SC04_018": dict(build="build/ov_SC04_018/ov_SC04_018", check="config/check.ov_SC04_018.sha",
|
||
src="src/ov_SC04_018", asm="asm/ov_SC04_018/nonmatchings", out="docs/progress.ov_SC04_018.md"),
|
||
"ov_SC04_019": dict(build="build/ov_SC04_019/ov_SC04_019", check="config/check.ov_SC04_019.sha",
|
||
src="src/ov_SC04_019", asm="asm/ov_SC04_019/nonmatchings", out="docs/progress.ov_SC04_019.md"),
|
||
"ov_SC05_000": dict(build="build/ov_SC05_000/ov_SC05_000", check="config/check.ov_SC05_000.sha",
|
||
src="src/ov_SC05_000", asm="asm/ov_SC05_000/nonmatchings", out="docs/progress.ov_SC05_000.md"),
|
||
"ov_SC06_000": dict(build="build/ov_SC06_000/ov_SC06_000", check="config/check.ov_SC06_000.sha",
|
||
src="src/ov_SC06_000", asm="asm/ov_SC06_000/nonmatchings", out="docs/progress.ov_SC06_000.md"),
|
||
"ov_SC07_000": dict(build="build/ov_SC07_000/ov_SC07_000", check="config/check.ov_SC07_000.sha",
|
||
src="src/ov_SC07_000", asm="asm/ov_SC07_000/nonmatchings", out="docs/progress.ov_SC07_000.md"),
|
||
"ov_SC01_004": dict(build="build/ov_SC01_004/ov_SC01_004", check="config/check.ov_SC01_004.sha",
|
||
src="src/ov_SC01_004", asm="asm/ov_SC01_004/nonmatchings", out="docs/progress.ov_SC01_004.md"),
|
||
"ov_SC01_008": dict(build="build/ov_SC01_008/ov_SC01_008", check="config/check.ov_SC01_008.sha",
|
||
src="src/ov_SC01_008", asm="asm/ov_SC01_008/nonmatchings", out="docs/progress.ov_SC01_008.md"),
|
||
"ov_SC01_009": dict(build="build/ov_SC01_009/ov_SC01_009", check="config/check.ov_SC01_009.sha",
|
||
src="src/ov_SC01_009", asm="asm/ov_SC01_009/nonmatchings", out="docs/progress.ov_SC01_009.md"),
|
||
"ov_SC01_074": dict(build="build/ov_SC01_074/ov_SC01_074", check="config/check.ov_SC01_074.sha",
|
||
src="src/ov_SC01_074", asm="asm/ov_SC01_074/nonmatchings", out="docs/progress.ov_SC01_074.md"),
|
||
"ov_SC01_080": dict(build="build/ov_SC01_080/ov_SC01_080", check="config/check.ov_SC01_080.sha",
|
||
src="src/ov_SC01_080", asm="asm/ov_SC01_080/nonmatchings", out="docs/progress.ov_SC01_080.md"),
|
||
"ov_SC01_084": dict(build="build/ov_SC01_084/ov_SC01_084", check="config/check.ov_SC01_084.sha",
|
||
src="src/ov_SC01_084", asm="asm/ov_SC01_084/nonmatchings", out="docs/progress.ov_SC01_084.md"),
|
||
"ov_SC02_004": dict(build="build/ov_SC02_004/ov_SC02_004", check="config/check.ov_SC02_004.sha",
|
||
src="src/ov_SC02_004", asm="asm/ov_SC02_004/nonmatchings", out="docs/progress.ov_SC02_004.md"),
|
||
"ov_SC02_005": dict(build="build/ov_SC02_005/ov_SC02_005", check="config/check.ov_SC02_005.sha",
|
||
src="src/ov_SC02_005", asm="asm/ov_SC02_005/nonmatchings", out="docs/progress.ov_SC02_005.md"),
|
||
"ov_SC02_011": dict(build="build/ov_SC02_011/ov_SC02_011", check="config/check.ov_SC02_011.sha",
|
||
src="src/ov_SC02_011", asm="asm/ov_SC02_011/nonmatchings", out="docs/progress.ov_SC02_011.md"),
|
||
"ov_SC02_015": dict(build="build/ov_SC02_015/ov_SC02_015", check="config/check.ov_SC02_015.sha",
|
||
src="src/ov_SC02_015", asm="asm/ov_SC02_015/nonmatchings", out="docs/progress.ov_SC02_015.md"),
|
||
"ov_SC02_016": dict(build="build/ov_SC02_016/ov_SC02_016", check="config/check.ov_SC02_016.sha",
|
||
src="src/ov_SC02_016", asm="asm/ov_SC02_016/nonmatchings", out="docs/progress.ov_SC02_016.md"),
|
||
"ov_SC02_017": dict(build="build/ov_SC02_017/ov_SC02_017", check="config/check.ov_SC02_017.sha",
|
||
src="src/ov_SC02_017", asm="asm/ov_SC02_017/nonmatchings", out="docs/progress.ov_SC02_017.md"),
|
||
"ov_SC02_021": dict(build="build/ov_SC02_021/ov_SC02_021", check="config/check.ov_SC02_021.sha",
|
||
src="src/ov_SC02_021", asm="asm/ov_SC02_021/nonmatchings", out="docs/progress.ov_SC02_021.md"),
|
||
"ov_SC02_026": dict(build="build/ov_SC02_026/ov_SC02_026", check="config/check.ov_SC02_026.sha",
|
||
src="src/ov_SC02_026", asm="asm/ov_SC02_026/nonmatchings", out="docs/progress.ov_SC02_026.md"),
|
||
"ov_SC02_027": dict(build="build/ov_SC02_027/ov_SC02_027", check="config/check.ov_SC02_027.sha",
|
||
src="src/ov_SC02_027", asm="asm/ov_SC02_027/nonmatchings", out="docs/progress.ov_SC02_027.md"),
|
||
"ov_SC02_028": dict(build="build/ov_SC02_028/ov_SC02_028", check="config/check.ov_SC02_028.sha",
|
||
src="src/ov_SC02_028", asm="asm/ov_SC02_028/nonmatchings", out="docs/progress.ov_SC02_028.md"),
|
||
"ov_SC02_031": dict(build="build/ov_SC02_031/ov_SC02_031", check="config/check.ov_SC02_031.sha",
|
||
src="src/ov_SC02_031", asm="asm/ov_SC02_031/nonmatchings", out="docs/progress.ov_SC02_031.md"),
|
||
"ov_SC02_035": dict(build="build/ov_SC02_035/ov_SC02_035", check="config/check.ov_SC02_035.sha",
|
||
src="src/ov_SC02_035", asm="asm/ov_SC02_035/nonmatchings", out="docs/progress.ov_SC02_035.md"),
|
||
"ov_SC02_039": dict(build="build/ov_SC02_039/ov_SC02_039", check="config/check.ov_SC02_039.sha",
|
||
src="src/ov_SC02_039", asm="asm/ov_SC02_039/nonmatchings", out="docs/progress.ov_SC02_039.md"),
|
||
"ov_SC02_041": dict(build="build/ov_SC02_041/ov_SC02_041", check="config/check.ov_SC02_041.sha",
|
||
src="src/ov_SC02_041", asm="asm/ov_SC02_041/nonmatchings", out="docs/progress.ov_SC02_041.md"),
|
||
"ov_SC03_003": dict(build="build/ov_SC03_003/ov_SC03_003", check="config/check.ov_SC03_003.sha",
|
||
src="src/ov_SC03_003", asm="asm/ov_SC03_003/nonmatchings", out="docs/progress.ov_SC03_003.md"),
|
||
"ov_SC03_007": dict(build="build/ov_SC03_007/ov_SC03_007", check="config/check.ov_SC03_007.sha",
|
||
src="src/ov_SC03_007", asm="asm/ov_SC03_007/nonmatchings", out="docs/progress.ov_SC03_007.md"),
|
||
"ov_SC03_010": dict(build="build/ov_SC03_010/ov_SC03_010", check="config/check.ov_SC03_010.sha",
|
||
src="src/ov_SC03_010", asm="asm/ov_SC03_010/nonmatchings", out="docs/progress.ov_SC03_010.md"),
|
||
"ov_SC03_011": dict(build="build/ov_SC03_011/ov_SC03_011", check="config/check.ov_SC03_011.sha",
|
||
src="src/ov_SC03_011", asm="asm/ov_SC03_011/nonmatchings", out="docs/progress.ov_SC03_011.md"),
|
||
"ov_SC03_012": dict(build="build/ov_SC03_012/ov_SC03_012", check="config/check.ov_SC03_012.sha",
|
||
src="src/ov_SC03_012", asm="asm/ov_SC03_012/nonmatchings", out="docs/progress.ov_SC03_012.md"),
|
||
"ov_SC03_013": dict(build="build/ov_SC03_013/ov_SC03_013", check="config/check.ov_SC03_013.sha",
|
||
src="src/ov_SC03_013", asm="asm/ov_SC03_013/nonmatchings", out="docs/progress.ov_SC03_013.md"),
|
||
"ov_SC03_014": dict(build="build/ov_SC03_014/ov_SC03_014", check="config/check.ov_SC03_014.sha",
|
||
src="src/ov_SC03_014", asm="asm/ov_SC03_014/nonmatchings", out="docs/progress.ov_SC03_014.md"),
|
||
"ov_SC03_015": dict(build="build/ov_SC03_015/ov_SC03_015", check="config/check.ov_SC03_015.sha",
|
||
src="src/ov_SC03_015", asm="asm/ov_SC03_015/nonmatchings", out="docs/progress.ov_SC03_015.md"),
|
||
"ov_SC03_023": dict(build="build/ov_SC03_023/ov_SC03_023", check="config/check.ov_SC03_023.sha",
|
||
src="src/ov_SC03_023", asm="asm/ov_SC03_023/nonmatchings", out="docs/progress.ov_SC03_023.md"),
|
||
"ov_SC03_024": dict(build="build/ov_SC03_024/ov_SC03_024", check="config/check.ov_SC03_024.sha",
|
||
src="src/ov_SC03_024", asm="asm/ov_SC03_024/nonmatchings", out="docs/progress.ov_SC03_024.md"),
|
||
"ov_SC03_028": dict(build="build/ov_SC03_028/ov_SC03_028", check="config/check.ov_SC03_028.sha",
|
||
src="src/ov_SC03_028", asm="asm/ov_SC03_028/nonmatchings", out="docs/progress.ov_SC03_028.md"),
|
||
"ov_SC03_029": dict(build="build/ov_SC03_029/ov_SC03_029", check="config/check.ov_SC03_029.sha",
|
||
src="src/ov_SC03_029", asm="asm/ov_SC03_029/nonmatchings", out="docs/progress.ov_SC03_029.md"),
|
||
"ov_SC03_030": dict(build="build/ov_SC03_030/ov_SC03_030", check="config/check.ov_SC03_030.sha",
|
||
src="src/ov_SC03_030", asm="asm/ov_SC03_030/nonmatchings", out="docs/progress.ov_SC03_030.md"),
|
||
"ov_SC03_031": dict(build="build/ov_SC03_031/ov_SC03_031", check="config/check.ov_SC03_031.sha",
|
||
src="src/ov_SC03_031", asm="asm/ov_SC03_031/nonmatchings", out="docs/progress.ov_SC03_031.md"),
|
||
"ov_SC03_089": dict(build="build/ov_SC03_089/ov_SC03_089", check="config/check.ov_SC03_089.sha",
|
||
src="src/ov_SC03_089", asm="asm/ov_SC03_089/nonmatchings", out="docs/progress.ov_SC03_089.md"),
|
||
"ov_SC03_090": dict(build="build/ov_SC03_090/ov_SC03_090", check="config/check.ov_SC03_090.sha",
|
||
src="src/ov_SC03_090", asm="asm/ov_SC03_090/nonmatchings", out="docs/progress.ov_SC03_090.md"),
|
||
"ov_SC03_091": dict(build="build/ov_SC03_091/ov_SC03_091", check="config/check.ov_SC03_091.sha",
|
||
src="src/ov_SC03_091", asm="asm/ov_SC03_091/nonmatchings", out="docs/progress.ov_SC03_091.md"),
|
||
"ov_SC03_092": dict(build="build/ov_SC03_092/ov_SC03_092", check="config/check.ov_SC03_092.sha",
|
||
src="src/ov_SC03_092", asm="asm/ov_SC03_092/nonmatchings", out="docs/progress.ov_SC03_092.md"),
|
||
"ov_SC03_093": dict(build="build/ov_SC03_093/ov_SC03_093", check="config/check.ov_SC03_093.sha",
|
||
src="src/ov_SC03_093", asm="asm/ov_SC03_093/nonmatchings", out="docs/progress.ov_SC03_093.md"),
|
||
"ov_SC03_094": dict(build="build/ov_SC03_094/ov_SC03_094", check="config/check.ov_SC03_094.sha",
|
||
src="src/ov_SC03_094", asm="asm/ov_SC03_094/nonmatchings", out="docs/progress.ov_SC03_094.md"),
|
||
"ov_SC03_095": dict(build="build/ov_SC03_095/ov_SC03_095", check="config/check.ov_SC03_095.sha",
|
||
src="src/ov_SC03_095", asm="asm/ov_SC03_095/nonmatchings", out="docs/progress.ov_SC03_095.md"),
|
||
"ov_SC03_096": dict(build="build/ov_SC03_096/ov_SC03_096", check="config/check.ov_SC03_096.sha",
|
||
src="src/ov_SC03_096", asm="asm/ov_SC03_096/nonmatchings", out="docs/progress.ov_SC03_096.md"),
|
||
"ov_SC03_097": dict(build="build/ov_SC03_097/ov_SC03_097", check="config/check.ov_SC03_097.sha",
|
||
src="src/ov_SC03_097", asm="asm/ov_SC03_097/nonmatchings", out="docs/progress.ov_SC03_097.md"),
|
||
"ov_SC03_098": dict(build="build/ov_SC03_098/ov_SC03_098", check="config/check.ov_SC03_098.sha",
|
||
src="src/ov_SC03_098", asm="asm/ov_SC03_098/nonmatchings", out="docs/progress.ov_SC03_098.md"),
|
||
"ov_SC03_099": dict(build="build/ov_SC03_099/ov_SC03_099", check="config/check.ov_SC03_099.sha",
|
||
src="src/ov_SC03_099", asm="asm/ov_SC03_099/nonmatchings", out="docs/progress.ov_SC03_099.md"),
|
||
"ov_SC03_100": dict(build="build/ov_SC03_100/ov_SC03_100", check="config/check.ov_SC03_100.sha",
|
||
src="src/ov_SC03_100", asm="asm/ov_SC03_100/nonmatchings", out="docs/progress.ov_SC03_100.md"),
|
||
"ov_SC03_101": dict(build="build/ov_SC03_101/ov_SC03_101", check="config/check.ov_SC03_101.sha",
|
||
src="src/ov_SC03_101", asm="asm/ov_SC03_101/nonmatchings", out="docs/progress.ov_SC03_101.md"),
|
||
"ov_SC03_102": dict(build="build/ov_SC03_102/ov_SC03_102", check="config/check.ov_SC03_102.sha",
|
||
src="src/ov_SC03_102", asm="asm/ov_SC03_102/nonmatchings", out="docs/progress.ov_SC03_102.md"),
|
||
"ov_SC03_103": dict(build="build/ov_SC03_103/ov_SC03_103", check="config/check.ov_SC03_103.sha",
|
||
src="src/ov_SC03_103", asm="asm/ov_SC03_103/nonmatchings", out="docs/progress.ov_SC03_103.md"),
|
||
"ov_SC03_104": dict(build="build/ov_SC03_104/ov_SC03_104", check="config/check.ov_SC03_104.sha",
|
||
src="src/ov_SC03_104", asm="asm/ov_SC03_104/nonmatchings", out="docs/progress.ov_SC03_104.md"),
|
||
"ov_SC03_105": dict(build="build/ov_SC03_105/ov_SC03_105", check="config/check.ov_SC03_105.sha",
|
||
src="src/ov_SC03_105", asm="asm/ov_SC03_105/nonmatchings", out="docs/progress.ov_SC03_105.md"),
|
||
"ov_SC03_108": dict(build="build/ov_SC03_108/ov_SC03_108", check="config/check.ov_SC03_108.sha",
|
||
src="src/ov_SC03_108", asm="asm/ov_SC03_108/nonmatchings", out="docs/progress.ov_SC03_108.md"),
|
||
"ov_SC03_109": dict(build="build/ov_SC03_109/ov_SC03_109", check="config/check.ov_SC03_109.sha",
|
||
src="src/ov_SC03_109", asm="asm/ov_SC03_109/nonmatchings", out="docs/progress.ov_SC03_109.md"),
|
||
"ov_SC03_110": dict(build="build/ov_SC03_110/ov_SC03_110", check="config/check.ov_SC03_110.sha",
|
||
src="src/ov_SC03_110", asm="asm/ov_SC03_110/nonmatchings", out="docs/progress.ov_SC03_110.md"),
|
||
"ov_SC03_111": dict(build="build/ov_SC03_111/ov_SC03_111", check="config/check.ov_SC03_111.sha",
|
||
src="src/ov_SC03_111", asm="asm/ov_SC03_111/nonmatchings", out="docs/progress.ov_SC03_111.md"),
|
||
"ov_SC03_112": dict(build="build/ov_SC03_112/ov_SC03_112", check="config/check.ov_SC03_112.sha",
|
||
src="src/ov_SC03_112", asm="asm/ov_SC03_112/nonmatchings", out="docs/progress.ov_SC03_112.md"),
|
||
"ov_SC03_113": dict(build="build/ov_SC03_113/ov_SC03_113", check="config/check.ov_SC03_113.sha",
|
||
src="src/ov_SC03_113", asm="asm/ov_SC03_113/nonmatchings", out="docs/progress.ov_SC03_113.md"),
|
||
"ov_SC03_114": dict(build="build/ov_SC03_114/ov_SC03_114", check="config/check.ov_SC03_114.sha",
|
||
src="src/ov_SC03_114", asm="asm/ov_SC03_114/nonmatchings", out="docs/progress.ov_SC03_114.md"),
|
||
"ov_SC03_115": dict(build="build/ov_SC03_115/ov_SC03_115", check="config/check.ov_SC03_115.sha",
|
||
src="src/ov_SC03_115", asm="asm/ov_SC03_115/nonmatchings", out="docs/progress.ov_SC03_115.md"),
|
||
"ov_SC03_116": dict(build="build/ov_SC03_116/ov_SC03_116", check="config/check.ov_SC03_116.sha",
|
||
src="src/ov_SC03_116", asm="asm/ov_SC03_116/nonmatchings", out="docs/progress.ov_SC03_116.md"),
|
||
"ov_SC03_117": dict(build="build/ov_SC03_117/ov_SC03_117", check="config/check.ov_SC03_117.sha",
|
||
src="src/ov_SC03_117", asm="asm/ov_SC03_117/nonmatchings", out="docs/progress.ov_SC03_117.md"),
|
||
"ov_SC03_118": dict(build="build/ov_SC03_118/ov_SC03_118", check="config/check.ov_SC03_118.sha",
|
||
src="src/ov_SC03_118", asm="asm/ov_SC03_118/nonmatchings", out="docs/progress.ov_SC03_118.md"),
|
||
"ov_SC03_119": dict(build="build/ov_SC03_119/ov_SC03_119", check="config/check.ov_SC03_119.sha",
|
||
src="src/ov_SC03_119", asm="asm/ov_SC03_119/nonmatchings", out="docs/progress.ov_SC03_119.md"),
|
||
"ov_SC03_121": dict(build="build/ov_SC03_121/ov_SC03_121", check="config/check.ov_SC03_121.sha",
|
||
src="src/ov_SC03_121", asm="asm/ov_SC03_121/nonmatchings", out="docs/progress.ov_SC03_121.md"),
|
||
"ov_SC03_124": dict(build="build/ov_SC03_124/ov_SC03_124", check="config/check.ov_SC03_124.sha",
|
||
src="src/ov_SC03_124", asm="asm/ov_SC03_124/nonmatchings", out="docs/progress.ov_SC03_124.md"),
|
||
"ov_SC03_125": dict(build="build/ov_SC03_125/ov_SC03_125", check="config/check.ov_SC03_125.sha",
|
||
src="src/ov_SC03_125", asm="asm/ov_SC03_125/nonmatchings", out="docs/progress.ov_SC03_125.md"),
|
||
"ov_SC03_126": dict(build="build/ov_SC03_126/ov_SC03_126", check="config/check.ov_SC03_126.sha",
|
||
src="src/ov_SC03_126", asm="asm/ov_SC03_126/nonmatchings", out="docs/progress.ov_SC03_126.md"),
|
||
"ov_SC04_002": dict(build="build/ov_SC04_002/ov_SC04_002", check="config/check.ov_SC04_002.sha",
|
||
src="src/ov_SC04_002", asm="asm/ov_SC04_002/nonmatchings", out="docs/progress.ov_SC04_002.md"),
|
||
"ov_SC04_003": dict(build="build/ov_SC04_003/ov_SC04_003", check="config/check.ov_SC04_003.sha",
|
||
src="src/ov_SC04_003", asm="asm/ov_SC04_003/nonmatchings", out="docs/progress.ov_SC04_003.md"),
|
||
"ov_SC04_004": dict(build="build/ov_SC04_004/ov_SC04_004", check="config/check.ov_SC04_004.sha",
|
||
src="src/ov_SC04_004", asm="asm/ov_SC04_004/nonmatchings", out="docs/progress.ov_SC04_004.md"),
|
||
"ov_SC04_005": dict(build="build/ov_SC04_005/ov_SC04_005", check="config/check.ov_SC04_005.sha",
|
||
src="src/ov_SC04_005", asm="asm/ov_SC04_005/nonmatchings", out="docs/progress.ov_SC04_005.md"),
|
||
"ov_SC04_006": dict(build="build/ov_SC04_006/ov_SC04_006", check="config/check.ov_SC04_006.sha",
|
||
src="src/ov_SC04_006", asm="asm/ov_SC04_006/nonmatchings", out="docs/progress.ov_SC04_006.md"),
|
||
"ov_SC04_007": dict(build="build/ov_SC04_007/ov_SC04_007", check="config/check.ov_SC04_007.sha",
|
||
src="src/ov_SC04_007", asm="asm/ov_SC04_007/nonmatchings", out="docs/progress.ov_SC04_007.md"),
|
||
"ov_SC04_008": dict(build="build/ov_SC04_008/ov_SC04_008", check="config/check.ov_SC04_008.sha",
|
||
src="src/ov_SC04_008", asm="asm/ov_SC04_008/nonmatchings", out="docs/progress.ov_SC04_008.md"),
|
||
"ov_SC04_009": dict(build="build/ov_SC04_009/ov_SC04_009", check="config/check.ov_SC04_009.sha",
|
||
src="src/ov_SC04_009", asm="asm/ov_SC04_009/nonmatchings", out="docs/progress.ov_SC04_009.md"),
|
||
"ov_SC04_010": dict(build="build/ov_SC04_010/ov_SC04_010", check="config/check.ov_SC04_010.sha",
|
||
src="src/ov_SC04_010", asm="asm/ov_SC04_010/nonmatchings", out="docs/progress.ov_SC04_010.md"),
|
||
"ov_SC04_011": dict(build="build/ov_SC04_011/ov_SC04_011", check="config/check.ov_SC04_011.sha",
|
||
src="src/ov_SC04_011", asm="asm/ov_SC04_011/nonmatchings", out="docs/progress.ov_SC04_011.md"),
|
||
"ov_SC04_012": dict(build="build/ov_SC04_012/ov_SC04_012", check="config/check.ov_SC04_012.sha",
|
||
src="src/ov_SC04_012", asm="asm/ov_SC04_012/nonmatchings", out="docs/progress.ov_SC04_012.md"),
|
||
"ov_SC04_015": dict(build="build/ov_SC04_015/ov_SC04_015", check="config/check.ov_SC04_015.sha",
|
||
src="src/ov_SC04_015", asm="asm/ov_SC04_015/nonmatchings", out="docs/progress.ov_SC04_015.md"),
|
||
"ov_SC04_016": dict(build="build/ov_SC04_016/ov_SC04_016", check="config/check.ov_SC04_016.sha",
|
||
src="src/ov_SC04_016", asm="asm/ov_SC04_016/nonmatchings", out="docs/progress.ov_SC04_016.md"),
|
||
"ov_SC04_020": dict(build="build/ov_SC04_020/ov_SC04_020", check="config/check.ov_SC04_020.sha",
|
||
src="src/ov_SC04_020", asm="asm/ov_SC04_020/nonmatchings", out="docs/progress.ov_SC04_020.md"),
|
||
"ov_SC04_021": dict(build="build/ov_SC04_021/ov_SC04_021", check="config/check.ov_SC04_021.sha",
|
||
src="src/ov_SC04_021", asm="asm/ov_SC04_021/nonmatchings", out="docs/progress.ov_SC04_021.md"),
|
||
"ov_SC05_001": dict(build="build/ov_SC05_001/ov_SC05_001", check="config/check.ov_SC05_001.sha",
|
||
src="src/ov_SC05_001", asm="asm/ov_SC05_001/nonmatchings", out="docs/progress.ov_SC05_001.md"),
|
||
"ov_SC05_002": dict(build="build/ov_SC05_002/ov_SC05_002", check="config/check.ov_SC05_002.sha",
|
||
src="src/ov_SC05_002", asm="asm/ov_SC05_002/nonmatchings", out="docs/progress.ov_SC05_002.md"),
|
||
"ov_SC05_003": dict(build="build/ov_SC05_003/ov_SC05_003", check="config/check.ov_SC05_003.sha",
|
||
src="src/ov_SC05_003", asm="asm/ov_SC05_003/nonmatchings", out="docs/progress.ov_SC05_003.md"),
|
||
"ov_SC05_004": dict(build="build/ov_SC05_004/ov_SC05_004", check="config/check.ov_SC05_004.sha",
|
||
src="src/ov_SC05_004", asm="asm/ov_SC05_004/nonmatchings", out="docs/progress.ov_SC05_004.md"),
|
||
"ov_SC05_005": dict(build="build/ov_SC05_005/ov_SC05_005", check="config/check.ov_SC05_005.sha",
|
||
src="src/ov_SC05_005", asm="asm/ov_SC05_005/nonmatchings", out="docs/progress.ov_SC05_005.md"),
|
||
"ov_SC05_006": dict(build="build/ov_SC05_006/ov_SC05_006", check="config/check.ov_SC05_006.sha",
|
||
src="src/ov_SC05_006", asm="asm/ov_SC05_006/nonmatchings", out="docs/progress.ov_SC05_006.md"),
|
||
"ov_SC05_007": dict(build="build/ov_SC05_007/ov_SC05_007", check="config/check.ov_SC05_007.sha",
|
||
src="src/ov_SC05_007", asm="asm/ov_SC05_007/nonmatchings", out="docs/progress.ov_SC05_007.md"),
|
||
"ov_SC05_008": dict(build="build/ov_SC05_008/ov_SC05_008", check="config/check.ov_SC05_008.sha",
|
||
src="src/ov_SC05_008", asm="asm/ov_SC05_008/nonmatchings", out="docs/progress.ov_SC05_008.md"),
|
||
"ov_SC05_009": dict(build="build/ov_SC05_009/ov_SC05_009", check="config/check.ov_SC05_009.sha",
|
||
src="src/ov_SC05_009", asm="asm/ov_SC05_009/nonmatchings", out="docs/progress.ov_SC05_009.md"),
|
||
"ov_SC05_010": dict(build="build/ov_SC05_010/ov_SC05_010", check="config/check.ov_SC05_010.sha",
|
||
src="src/ov_SC05_010", asm="asm/ov_SC05_010/nonmatchings", out="docs/progress.ov_SC05_010.md"),
|
||
"ov_SC05_011": dict(build="build/ov_SC05_011/ov_SC05_011", check="config/check.ov_SC05_011.sha",
|
||
src="src/ov_SC05_011", asm="asm/ov_SC05_011/nonmatchings", out="docs/progress.ov_SC05_011.md"),
|
||
"ov_SC05_017": dict(build="build/ov_SC05_017/ov_SC05_017", check="config/check.ov_SC05_017.sha",
|
||
src="src/ov_SC05_017", asm="asm/ov_SC05_017/nonmatchings", out="docs/progress.ov_SC05_017.md"),
|
||
"ov_SC05_018": dict(build="build/ov_SC05_018/ov_SC05_018", check="config/check.ov_SC05_018.sha",
|
||
src="src/ov_SC05_018", asm="asm/ov_SC05_018/nonmatchings", out="docs/progress.ov_SC05_018.md"),
|
||
"ov_SC05_019": dict(build="build/ov_SC05_019/ov_SC05_019", check="config/check.ov_SC05_019.sha",
|
||
src="src/ov_SC05_019", asm="asm/ov_SC05_019/nonmatchings", out="docs/progress.ov_SC05_019.md"),
|
||
"ov_SC06_006": dict(build="build/ov_SC06_006/ov_SC06_006", check="config/check.ov_SC06_006.sha",
|
||
src="src/ov_SC06_006", asm="asm/ov_SC06_006/nonmatchings", out="docs/progress.ov_SC06_006.md"),
|
||
"ov_SC06_008": dict(build="build/ov_SC06_008/ov_SC06_008", check="config/check.ov_SC06_008.sha",
|
||
src="src/ov_SC06_008", asm="asm/ov_SC06_008/nonmatchings", out="docs/progress.ov_SC06_008.md"),
|
||
"ov_SC06_010": dict(build="build/ov_SC06_010/ov_SC06_010", check="config/check.ov_SC06_010.sha",
|
||
src="src/ov_SC06_010", asm="asm/ov_SC06_010/nonmatchings", out="docs/progress.ov_SC06_010.md"),
|
||
"ov_SC06_011": dict(build="build/ov_SC06_011/ov_SC06_011", check="config/check.ov_SC06_011.sha",
|
||
src="src/ov_SC06_011", asm="asm/ov_SC06_011/nonmatchings", out="docs/progress.ov_SC06_011.md"),
|
||
"ov_SC06_013": dict(build="build/ov_SC06_013/ov_SC06_013", check="config/check.ov_SC06_013.sha",
|
||
src="src/ov_SC06_013", asm="asm/ov_SC06_013/nonmatchings", out="docs/progress.ov_SC06_013.md"),
|
||
"ov_SC06_014": dict(build="build/ov_SC06_014/ov_SC06_014", check="config/check.ov_SC06_014.sha",
|
||
src="src/ov_SC06_014", asm="asm/ov_SC06_014/nonmatchings", out="docs/progress.ov_SC06_014.md"),
|
||
"ov_SC06_015": dict(build="build/ov_SC06_015/ov_SC06_015", check="config/check.ov_SC06_015.sha",
|
||
src="src/ov_SC06_015", asm="asm/ov_SC06_015/nonmatchings", out="docs/progress.ov_SC06_015.md"),
|
||
"ov_SC06_016": dict(build="build/ov_SC06_016/ov_SC06_016", check="config/check.ov_SC06_016.sha",
|
||
src="src/ov_SC06_016", asm="asm/ov_SC06_016/nonmatchings", out="docs/progress.ov_SC06_016.md"),
|
||
"ov_SC06_018": dict(build="build/ov_SC06_018/ov_SC06_018", check="config/check.ov_SC06_018.sha",
|
||
src="src/ov_SC06_018", asm="asm/ov_SC06_018/nonmatchings", out="docs/progress.ov_SC06_018.md"),
|
||
"ov_SC06_020": dict(build="build/ov_SC06_020/ov_SC06_020", check="config/check.ov_SC06_020.sha",
|
||
src="src/ov_SC06_020", asm="asm/ov_SC06_020/nonmatchings", out="docs/progress.ov_SC06_020.md"),
|
||
"ov_SC06_022": dict(build="build/ov_SC06_022/ov_SC06_022", check="config/check.ov_SC06_022.sha",
|
||
src="src/ov_SC06_022", asm="asm/ov_SC06_022/nonmatchings", out="docs/progress.ov_SC06_022.md"),
|
||
"ov_SC06_024": dict(build="build/ov_SC06_024/ov_SC06_024", check="config/check.ov_SC06_024.sha",
|
||
src="src/ov_SC06_024", asm="asm/ov_SC06_024/nonmatchings", out="docs/progress.ov_SC06_024.md"),
|
||
"ov_SC06_025": dict(build="build/ov_SC06_025/ov_SC06_025", check="config/check.ov_SC06_025.sha",
|
||
src="src/ov_SC06_025", asm="asm/ov_SC06_025/nonmatchings", out="docs/progress.ov_SC06_025.md"),
|
||
"ov_SC06_027": dict(build="build/ov_SC06_027/ov_SC06_027", check="config/check.ov_SC06_027.sha",
|
||
src="src/ov_SC06_027", asm="asm/ov_SC06_027/nonmatchings", out="docs/progress.ov_SC06_027.md"),
|
||
"ov_SC06_029": dict(build="build/ov_SC06_029/ov_SC06_029", check="config/check.ov_SC06_029.sha",
|
||
src="src/ov_SC06_029", asm="asm/ov_SC06_029/nonmatchings", out="docs/progress.ov_SC06_029.md"),
|
||
"ov_SC06_030": dict(build="build/ov_SC06_030/ov_SC06_030", check="config/check.ov_SC06_030.sha",
|
||
src="src/ov_SC06_030", asm="asm/ov_SC06_030/nonmatchings", out="docs/progress.ov_SC06_030.md"),
|
||
"ov_SC06_032": dict(build="build/ov_SC06_032/ov_SC06_032", check="config/check.ov_SC06_032.sha",
|
||
src="src/ov_SC06_032", asm="asm/ov_SC06_032/nonmatchings", out="docs/progress.ov_SC06_032.md"),
|
||
"ov_SC06_033": dict(build="build/ov_SC06_033/ov_SC06_033", check="config/check.ov_SC06_033.sha",
|
||
src="src/ov_SC06_033", asm="asm/ov_SC06_033/nonmatchings", out="docs/progress.ov_SC06_033.md"),
|
||
"ov_SC07_001": dict(build="build/ov_SC07_001/ov_SC07_001", check="config/check.ov_SC07_001.sha",
|
||
src="src/ov_SC07_001", asm="asm/ov_SC07_001/nonmatchings", out="docs/progress.ov_SC07_001.md"),
|
||
"ov_SC07_002": dict(build="build/ov_SC07_002/ov_SC07_002", check="config/check.ov_SC07_002.sha",
|
||
src="src/ov_SC07_002", asm="asm/ov_SC07_002/nonmatchings", out="docs/progress.ov_SC07_002.md"),
|
||
"ov_SC07_008": dict(build="build/ov_SC07_008/ov_SC07_008", check="config/check.ov_SC07_008.sha",
|
||
src="src/ov_SC07_008", asm="asm/ov_SC07_008/nonmatchings", out="docs/progress.ov_SC07_008.md"),
|
||
"ov_SC07_009": dict(build="build/ov_SC07_009/ov_SC07_009", check="config/check.ov_SC07_009.sha",
|
||
src="src/ov_SC07_009", asm="asm/ov_SC07_009/nonmatchings", out="docs/progress.ov_SC07_009.md"),
|
||
# <<< overlays: tools/new_overlay.sh inserts ov_* entries above this line (Phase 13) >>>
|
||
}
|
||
MAKEFILE = ROOT / "Makefile"
|
||
# Per-binary state (BINARY/_cfg/SRCS/ASM_ROOT/OUT/BUILD/CHECK/LINKED_SEGS) is set by
|
||
# set_binary() so report() can run over MANY binaries in one process (--fleet, Phase 15).
|
||
BINARY = _cfg = SRCS = ASM_ROOT = OUT = BUILD = CHECK = LINKED_SEGS = None
|
||
|
||
def set_binary(binary):
|
||
"""Point the module globals at <binary>'s src/asm/build/check tree + recompute LINKED_SEGS."""
|
||
global BINARY, _cfg, SRCS, ASM_ROOT, OUT, BUILD, CHECK, LINKED_SEGS
|
||
if binary not in BINARIES:
|
||
sys.exit(f"progress.py: unknown --binary '{binary}' (known: {', '.join(BINARIES)})")
|
||
BINARY = binary
|
||
_cfg = BINARIES[binary]
|
||
SRCS = sorted((ROOT / _cfg["src"]).glob("*.c")) # every c-segment (src/boot.c, src/800.c, ...)
|
||
ASM_ROOT = ROOT / _cfg["asm"] # per-segment subdirs (boot/, 800/, ...)
|
||
OUT, BUILD, CHECK = ROOT / _cfg["out"], ROOT / _cfg["build"], ROOT / _cfg["check"]
|
||
LINKED_SEGS = linked_subsegs()
|
||
global _S_INDEX # one directory walk per binary (not a glob per function — 340k globs over the fleet)
|
||
_S_INDEX = {p.stem: p for p in ASM_ROOT.glob("*/*.s")} if ASM_ROOT.exists() else {}
|
||
|
||
def linked_subsegs():
|
||
"""Library subsegments swapped to real PsyQ objects at build time = the 5th positional arg
|
||
of each `psyq_integrate.py` call in the Makefile (e.g. `libcd1,libcd2`). Parsing the Makefile
|
||
keeps it the SINGLE source of truth: a new library integration adds its psyq_integrate call and
|
||
its stubs are auto-counted LINKED (no separate manifest to drift). Stubs in these subsegs build
|
||
byte-identically from the real SDK objects when present, and from the committed asm fallback
|
||
otherwise (fresh clone) — so they are LINKED regardless of local .run/obj40 state (a property of
|
||
the project, not the machine).
|
||
|
||
Scoped to the active binary: every psyq_integrate call lives inside the Makefile's
|
||
`ifeq ($(BINARY),main)` block (PsyQ library linking is the EXE's layout — Phase 8), so only
|
||
`main` has LINKED subsegs. A second binary (e.g. resident) has none. When a future binary gains
|
||
its own gated integration, parse that gate here instead of the main-only shortcut."""
|
||
if BINARY != "main":
|
||
return set()
|
||
if not MAKEFILE.exists():
|
||
return set()
|
||
txt = MAKEFILE.read_text()
|
||
# stub lists may be passed inline (`libcd1,libcd2`) or via a make var (`$(LIBGTE_STUBS)` for the
|
||
# long multi-block ones) — collect `NAME := <comma,list>` defs so either form resolves.
|
||
mvars = dict(re.findall(r'^(\w+)\s*:=\s*([A-Za-z0-9_,]+)\s*$', txt, re.M))
|
||
segs = set()
|
||
# leading --flag value pairs (Phase 9: --vram-base/--exe/--symbols) precede the 4 positionals
|
||
# (elf_dir ld_path objdir syms_ld); the 5th positional is the stub list captured below.
|
||
for m in re.finditer(r'psyq_integrate\.py(?:\s+--\S+\s+\S+)*\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)', txt):
|
||
arg = m.group(1)
|
||
vm = re.fullmatch(r'\$\((\w+)\)', arg)
|
||
if vm:
|
||
arg = mvars.get(vm.group(1), '')
|
||
segs.update(s for s in arg.split(',') if re.fullmatch(r'[A-Za-z0-9_]+', s))
|
||
return segs
|
||
|
||
_DEDUP_CACHE = None # binary -> set(shared fn names); parsed once (the registry is large at fleet scale)
|
||
|
||
def dedup_members(binary):
|
||
"""Function names matched-once-and-shared via config/dedup.us.yaml for this binary. They are
|
||
hand-matched byte-identical C, but instantiated from a shared body (a macro in src/shared/),
|
||
so classify()'s function-definition scan does NOT recognize the macro form — count them REAL
|
||
via the registry instead (the source of truth for code shares, Phase 11). Returns a set."""
|
||
p = ROOT / "config/dedup.us.yaml"
|
||
if not p.exists():
|
||
return set()
|
||
global _DEDUP_CACHE
|
||
if _DEDUP_CACHE is None: # parse the registry ONCE, not per --fleet binary (136x)
|
||
_DEDUP_CACHE = {}
|
||
try:
|
||
import yaml
|
||
sys.path.insert(0, str(ROOT / "tools"))
|
||
from dedup_integrate import group_members # one parser for both verbose + shorthand forms
|
||
data = yaml.safe_load(p.read_text()) or {}
|
||
for g in (data.get("groups") or []):
|
||
for (b, _v, name) in group_members(g):
|
||
_DEDUP_CACHE.setdefault(b, set()).add(name)
|
||
except Exception:
|
||
_DEDUP_CACHE = {}
|
||
return _DEDUP_CACHE.get(binary, set())
|
||
|
||
_S_INDEX = {} # name -> .s path, rebuilt per binary in set_binary() (avoids a glob per function)
|
||
|
||
def find_s(name):
|
||
"""Locate <name>.s in any asm/nonmatchings/<seg>/ subdir (segments: boot, 800, ...)."""
|
||
return _S_INDEX.get(name)
|
||
|
||
INSTR = re.compile(r'^\s*/\*\s*[0-9A-Fa-f]+\s+[0-9A-Fa-f]+\s+[0-9A-Fa-f]+\s*\*/\s+[a-z]')
|
||
|
||
def strip_comments(s):
|
||
s = re.sub(r'/\*.*?\*/', '', s, flags=re.S)
|
||
return re.sub(r'//[^\n]*', '', s)
|
||
|
||
def is_data_blob(name):
|
||
"""A .s with a code label (glabel/jlabel) is a function; data-only (dlabel, no code) is a blob."""
|
||
p = find_s(name)
|
||
if p is None:
|
||
return False
|
||
txt = p.read_text()
|
||
return ('glabel' not in txt and 'jlabel' not in txt and 'dlabel' in txt)
|
||
|
||
def asm_is_trivial(name):
|
||
"""True iff the function's asm is exactly {jr, nop} (the empty-no-op shape splat emits void{} for)."""
|
||
p = find_s(name)
|
||
if p is None:
|
||
return None
|
||
mnem = []
|
||
for ln in p.read_text().splitlines():
|
||
m = re.match(r'^\s*/\*\s*[0-9A-Fa-f]+\s+[0-9A-Fa-f]+\s+[0-9A-Fa-f]+\s*\*/\s+([a-z0-9.]+)', ln)
|
||
if m:
|
||
mnem.append(m.group(1))
|
||
return set(mnem) <= {'jr', 'nop'} and len(mnem) <= 2
|
||
|
||
SIG = re.compile(r'^\s*[A-Za-z_][\w \t\*]*\b([A-Za-z_]\w*)\s*\(')
|
||
# A K&R parameter declaration sitting between the signature and the `{`: a bare `<type> <name>;`
|
||
# (optionally an array/pointer). It carries NO parens — that is what distinguishes it from the
|
||
# continuation line of a wrapped ANSI prototype, which always carries the closing `)`.
|
||
KR_PARAM = re.compile(r'^\s*[A-Za-z_][\w \t\*]*\s+\**\w+\s*(?:\[[^\]]*\])?\s*;\s*$')
|
||
|
||
def classify():
|
||
real, empty, nonmatching, stubs, blobs, linked = [], [], [], [], [], []
|
||
for src in SRCS:
|
||
lines = src.read_text().split('\n')
|
||
n = len(lines); i = 0
|
||
while i < n:
|
||
s = lines[i].strip()
|
||
if s.startswith('#ifdef NON_MATCHING'):
|
||
blk = []
|
||
while i < n and not lines[i].strip().startswith('#endif'):
|
||
blk.append(lines[i]); i += 1
|
||
i += 1
|
||
m = re.search(r'INCLUDE_ASM\("[^"]+",\s*(\w+)\)', '\n'.join(blk))
|
||
if m: nonmatching.append(m.group(1))
|
||
continue
|
||
m = re.match(r'INCLUDE_ASM\("([^"]+)",\s*(\w+)\)', s)
|
||
if m:
|
||
seg, fn = m.group(1).rstrip('/').rsplit('/', 1)[-1], m.group(2)
|
||
if is_data_blob(fn):
|
||
blobs.append(fn)
|
||
elif seg in LINKED_SEGS: # subseg swapped to a real PsyQ object (byte-identical)
|
||
linked.append(fn)
|
||
else:
|
||
stubs.append(fn)
|
||
i += 1; continue
|
||
if s.startswith('INCLUDE_RODATA'):
|
||
i += 1; continue
|
||
fm = SIG.match(lines[i])
|
||
if fm and '(' in lines[i]:
|
||
# Definition ({ ... }) vs forward declaration (ends ;)? Scan to the first { or ;.
|
||
# extern/prototype lines (e.g. `extern s32 CdQueueBusy(void);`) are NOT functions.
|
||
#
|
||
# K&R DEFINITIONS (Phase 26 session 8 — this silently erased ~190k banked instructions):
|
||
# s32 func_8015AE2C(arg0)
|
||
# s32 arg0; <- a `;` BEFORE the `{`
|
||
# { ... }
|
||
# The old scan hit that `;` and classified the function as a forward declaration, so it
|
||
# landed in NO bucket at all — not REAL, not a stub, invisible. And a K&R def is MANDATORY
|
||
# whenever a zero-arg engine_core.h thunk calls the function (it must be unprototyped), so
|
||
# the metric was blind to exactly the shape our own banking recipe requires: the heavy-jr
|
||
# cores. func_8015AE2C (562x134), func_8015A3C8 (493x132), func_80166994 (369x134) were all
|
||
# compiled, linked and BYTE-IDENTICAL in the shipped build while counting as zero.
|
||
# A K&R parameter declaration is a bare `<type> <name>;` — no parens (a wrapped ANSI
|
||
# prototype's continuation carries the `)`), so it is unambiguous to skip over.
|
||
j = i; kind = None
|
||
while j < n:
|
||
c = strip_comments(lines[j])
|
||
br = c.find('{'); sm = c.find(';')
|
||
if br != -1 and (sm == -1 or br < sm): kind = 'def'; break
|
||
if sm != -1:
|
||
if j > i and '(' not in c and ')' not in c and KR_PARAM.match(c):
|
||
j += 1; continue # K&R parameter declaration -> keep scanning for the `{`
|
||
kind = 'decl'; break
|
||
j += 1
|
||
if kind != 'def':
|
||
i = j + 1; continue # skip the declaration
|
||
start = i; depth = 0; opened = False
|
||
while i < n:
|
||
c = strip_comments(lines[i]); depth += c.count('{') - c.count('}')
|
||
if '{' in c: opened = True
|
||
i += 1
|
||
if opened and depth <= 0: break
|
||
body = '\n'.join(lines[start:i])
|
||
a, b = body.index('{'), body.rindex('}')
|
||
(real if strip_comments(body[a+1:b]).strip() else empty).append(fm.group(1))
|
||
continue
|
||
i += 1
|
||
return real, empty, nonmatching, stubs, blobs, linked
|
||
|
||
def report(binary, audit=False, write=True):
|
||
"""Classify one binary; write docs/progress.<binary>.md (if write) + print; return a stats dict
|
||
(for --fleet aggregation). Single-binary output is byte-for-byte the legacy format."""
|
||
set_binary(binary)
|
||
real, empty, nonmatching, stubs, blobs, linked = classify()
|
||
# Code-shared functions (dedup.us.yaml) are REAL byte-identical matches whose macro-instantiated
|
||
# form classify() doesn't parse — fold them in (dedup-safe) so the count stays honest (P9).
|
||
#
|
||
# ...but ONLY if they are actually instantiated. A member that is STILL an INCLUDE_ASM stub in this
|
||
# binary is not banked, whatever the registry says — the registry can go stale (a group whose
|
||
# DEFINE_ macro no longer exists in src/). Counting it REAL while it also sits in `stubs`
|
||
# double-counts it into `matchable` AND inflates `byteident`. Measured: 532 phantom instances
|
||
# (Phase 26 session 8 scanner audit). Subtracting `stubs` makes the registry advisory, and the
|
||
# source tree authoritative — which is the right precedence (P9: only what is in the build counts).
|
||
shared = sorted(dedup_members(BINARY) - set(real) - set(stubs))
|
||
real = sorted(set(real) | set(shared))
|
||
|
||
# ---- COVERAGE ASSERTION (the rule ratified 2026-07-14: a scanner over the corpus must assert its
|
||
# own coverage; a silent skip is a DEFECT, not a no-op). Ground truth = every function splat emitted
|
||
# a .s for. Anything classify() could not place in ANY bucket is a parsing hole, and it is exactly
|
||
# how the K&R blindness above hid ~190k banked instructions for 26 phases while the byte-gate stayed
|
||
# green (the gate compiles; this tool only reads text — they share no code, so the gate can never
|
||
# catch a miscount). Report it LOUDLY rather than silently under-reporting progress.
|
||
placed = set(real) | set(empty) | set(nonmatching) | set(stubs) | set(blobs) | set(linked)
|
||
unplaced = sorted(set(_S_INDEX) - placed)
|
||
matchable = len(real) + len(empty) + len(nonmatching) + len(stubs) + len(linked)
|
||
byteident = len(real) + len(linked) + len(empty) # all byte-identical in the build
|
||
|
||
out = []
|
||
out.append("# BFM matching progress (generated by tools/progress.py — authoritative)")
|
||
out.append("")
|
||
out.append(f"REAL substantive matches : {len(real):5d} <- the Gen1-exit >=25 bar counts THIS")
|
||
if shared:
|
||
out.append(f" (of which dedup-shared : {len(shared):5d} one body -> N sites, config/dedup.us.yaml)")
|
||
out.append(f"LINKED real PsyQ objects : {len(linked):5d} <- byte-identical via linked SDK objects")
|
||
out.append(f"NON_MATCHING (near-miss) : {len(nonmatching):5d}")
|
||
out.append(f"splat-auto empty no-ops : {len(empty):5d}")
|
||
out.append(f"INCLUDE_ASM stubs : {len(stubs):5d}")
|
||
out.append(f"data blobs (excluded) : {len(blobs):5d}")
|
||
if unplaced:
|
||
out.append(f"!! UNPLACED (parse hole) : {len(unplaced):5d} <- classify() could not bucket these; "
|
||
f"a SILENT SKIP is a defect, not a no-op. e.g. {', '.join(unplaced[:4])}")
|
||
print(f"[progress] COVERAGE DEFECT in {binary}: {len(unplaced)} function(s) splat emitted a .s for "
|
||
f"are in NO bucket — the count is WRONG. e.g. {', '.join(unplaced[:6])}", file=sys.stderr)
|
||
out.append("-" * 40)
|
||
out.append(f"matchable functions : {matchable:5d}")
|
||
out.append(f"REAL / matchable : {len(real)} / {matchable} = {100*len(real)/matchable:.2f}%")
|
||
out.append(f"byte-identical/ matchable: {byteident} / {matchable} = {100*byteident/matchable:.2f}% (REAL+LINKED+empties)")
|
||
out.append("")
|
||
out.append("LINKED subsegs: " + " ".join(sorted(LINKED_SEGS)) + f" ({len(linked)} fns)")
|
||
out.append("REAL matches: " + " ".join(sorted(real)))
|
||
out.append("NON_MATCHING: " + " ".join(sorted(nonmatching)))
|
||
|
||
if BUILD.exists():
|
||
h = hashlib.sha1(BUILD.read_bytes()).hexdigest()
|
||
want = CHECK.read_text().split()[0] if CHECK.exists() else ""
|
||
out.append("")
|
||
out.append(f"build SHA1: {h} ({'byte-identical' if h == want else 'MISMATCH'})")
|
||
|
||
if audit:
|
||
bad = [n for n in empty if asm_is_trivial(n) is False]
|
||
out.append("")
|
||
out.append(f"empties audit: {len(empty)-len(bad)}/{len(empty)} genuine jr;nop"
|
||
+ (f" !! SUSPICIOUS: {bad}" if bad else " (all clean)"))
|
||
|
||
if write:
|
||
text = "\n".join(out) + "\n"
|
||
print(text, end="")
|
||
OUT.write_text(text)
|
||
return dict(binary=binary, real=len(real), shared=len(shared), linked=len(linked),
|
||
nonmatching=len(nonmatching), empty=len(empty), stubs=len(stubs), blobs=len(blobs),
|
||
matchable=matchable, byteident=byteident)
|
||
|
||
|
||
def weighted_metrics():
|
||
"""Instruction/byte-weighted matching % from the committed sigs (.run/sig.*.jsonl) + src stubs.
|
||
Two framings, both against EXECUTABLE CODE only (not the ISO/audio/assets/data); MIPS instrs are
|
||
all 4 bytes so instruction% == byte%:
|
||
- fleet : sum(matched nins) / sum(total nins) over resident + every overlay (shared engine
|
||
counted PER-OVERLAY) — the number a decomp.dev/frogress per-binary aggregate displays.
|
||
- dedup : each distinct h_exact class ONCE, nins-weighted, matched if ANY overlay has it non-stub
|
||
— the 'distinct reverse-engineering' number (harsh: the ~74k x1 unique monsters dominate).
|
||
Covers resident + the 134 overlays; the main EXE has no image-sig (a small separate binary, excluded).
|
||
Returns a dict, or None if no sigs (run `make sig-overlays` first — keeps --fleet working without them)."""
|
||
import os, glob, json
|
||
paths = sorted(glob.glob(str(ROOT / ".run/sig.ov_*.jsonl")))
|
||
rp = ROOT / ".run/sig.resident.jsonl"
|
||
if rp.exists():
|
||
paths.append(str(rp))
|
||
if not paths:
|
||
return None
|
||
|
||
def src_stubs(binname):
|
||
m = set()
|
||
for cf in glob.glob(str(ROOT / f"src/{binname}/*.c")):
|
||
for a in re.findall(r'INCLUDE_ASM\([^)]*,\s*func_([0-9A-Fa-f]+)\)', open(cf).read()):
|
||
m.add(int(a, 16))
|
||
return m
|
||
|
||
fm = ft = 0
|
||
cls_nins, matched_cls = {}, set()
|
||
for p in paths:
|
||
b = os.path.basename(p).split("sig.")[1][:-6]
|
||
st = src_stubs(b)
|
||
for line in open(p):
|
||
r = json.loads(line)
|
||
a, n, hx = int(r["addr"], 16), r["nins"], r["h_exact"]
|
||
ft += n
|
||
cls_nins[hx] = n # h_exact-identical -> identical nins
|
||
if a not in st: # non-stub == matched (fleet is 136/136 byte-identical)
|
||
fm += n
|
||
matched_cls.add(hx)
|
||
ut = sum(cls_nins.values())
|
||
um = sum(n for hx, n in cls_nins.items() if hx in matched_cls)
|
||
return dict(fleet_m=fm, fleet_t=ft, fleet_pct=(100 * fm / ft if ft else 0.0),
|
||
dedup_m=um, dedup_t=ut, dedup_pct=(100 * um / ut if ut else 0.0),
|
||
nbins=len(paths), dedup_fns=len(matched_cls), dedup_total_fns=len(cls_nins))
|
||
|
||
|
||
def fleet():
|
||
"""Aggregate every binary into docs/progress.fleet.md — DETERMINISTIC (source-derived from the
|
||
committed src/*.c + config/dedup.us.yaml). The live byte-identity gate is `make check-all`;
|
||
collapsible-byte leverage lives in docs/duplicates.cross.md. Rows: main, resident, overlays sorted."""
|
||
order = [b for b in ("main", "resident") if b in BINARIES] + \
|
||
sorted(b for b in BINARIES if b not in ("main", "resident"))
|
||
rows = [report(b, write=False) for b in order]
|
||
tot = lambda k: sum(r[k] for r in rows)
|
||
REAL, SHARED, LINKED, EMPTY, NM, STUBS, MATCH, BYTE = (tot('real'), tot('shared'), tot('linked'),
|
||
tot('empty'), tot('nonmatching'), tot('stubs'), tot('matchable'), tot('byteident'))
|
||
# dedup registry totals (deterministic from the committed yaml)
|
||
ngroups = nmembers = 0
|
||
dp = ROOT / "config/dedup.us.yaml"
|
||
if dp.exists():
|
||
try:
|
||
import yaml
|
||
from dedup_integrate import group_members # expands verbose + shorthand alike
|
||
gs = (yaml.safe_load(dp.read_text()) or {}).get("groups") or []
|
||
ngroups = len(gs); nmembers = sum(1 for g in gs for _ in group_members(g))
|
||
except Exception:
|
||
pass
|
||
|
||
wm = weighted_metrics()
|
||
head = ["# BFM FLEET matching progress (generated by tools/progress.py --fleet — authoritative)",
|
||
f"# {len(rows)} binaries: main + resident + {len(rows)-2} location overlays. DETERMINISTIC,",
|
||
"# source-derived (committed src/*.c + config/dedup.us.yaml). Live byte gate: `make check-all`;",
|
||
"# cross-binary collapsible-byte leverage: docs/duplicates.cross.md.", "",
|
||
"# THREE progress metrics (all matter — see the labels):",
|
||
f"FLEET fn-count byte-ident: {BYTE:6d} / {MATCH} = {100*BYTE/MATCH:.2f}% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)"]
|
||
if wm:
|
||
head += [
|
||
f"FLEET instr-weighted : {wm['fleet_m']:7d} / {wm['fleet_t']} = {wm['fleet_pct']:.1f}% (shipped .text across resident+{wm['nbins']-1} overlays; the decomp.dev-DISPLAY number)",
|
||
f"FLEET distinct-code(uniq): {wm['dedup_m']:7d} / {wm['dedup_t']} = {wm['dedup_pct']:.1f}% ({wm['dedup_fns']}/{wm['dedup_total_fns']} unique fns; the DISTINCT-RE number; main EXE not sig'd)"]
|
||
else:
|
||
head += ["# (instr-weighted + distinct-code metrics need .run/sig.*.jsonl — run `make sig-overlays`)"]
|
||
head += ["",
|
||
f"FLEET REAL substantive : {REAL:6d} (of which dedup-shared {SHARED} via {ngroups} groups / {nmembers} instances)",
|
||
f"FLEET LINKED PsyQ objs : {LINKED:6d}",
|
||
f"FLEET NON_MATCHING : {NM:6d} (0 in any default build — G4)",
|
||
f"FLEET INCLUDE_ASM stubs : {STUBS:6d}",
|
||
f"FLEET matchable : {MATCH:6d}", ""]
|
||
table = ["| binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |",
|
||
"|---|---:|---:|---:|---:|---:|---:|"]
|
||
for r in rows:
|
||
pct = (100 * r['byteident'] / r['matchable']) if r['matchable'] else 0.0
|
||
table.append(f"| {r['binary']} | {r['real']} | {r['shared']} | {r['linked']} | "
|
||
f"{r['byteident']} | {r['matchable']} | {pct:.1f}% |")
|
||
text = "\n".join(head + table) + "\n"
|
||
print("\n".join(head[:-1])) # the summary block (drop the trailing blank)
|
||
(ROOT / "docs/progress.fleet.md").write_text(text)
|
||
|
||
|
||
def main():
|
||
if '--fleet' in sys.argv:
|
||
fleet(); return
|
||
if '--weighted' in sys.argv:
|
||
wm = weighted_metrics()
|
||
if not wm:
|
||
sys.exit("progress.py --weighted: no .run/sig.*.jsonl — run `make sig-overlays` first.")
|
||
print("BFM byte-weighted matching progress (executable code only; resident + %d overlays; main EXE excluded):"
|
||
% (wm['nbins'] - 1))
|
||
print(" instr-weighted (fleet, decomp.dev-display): %7d / %d = %.1f%%"
|
||
% (wm['fleet_m'], wm['fleet_t'], wm['fleet_pct']))
|
||
print(" distinct-code (dedup, distinct-RE) : %7d / %d = %.1f%% (%d/%d unique fns)"
|
||
% (wm['dedup_m'], wm['dedup_t'], wm['dedup_pct'], wm['dedup_fns'], wm['dedup_total_fns']))
|
||
return
|
||
audit = '--audit' in sys.argv
|
||
check = '--check' in sys.argv
|
||
binary = next((sys.argv[i + 1] for i, x in enumerate(sys.argv)
|
||
if x == "--binary" and i + 1 < len(sys.argv)), "main")
|
||
report(binary, audit=audit, write=True)
|
||
if check:
|
||
import subprocess
|
||
sys.exit(subprocess.run(["make", "-C", str(ROOT), "check"]).returncode)
|
||
|
||
if __name__ == "__main__":
|
||
main()
|