R35: fix the instrument before planting a flag on its denominator.
- #if 0 BLINDNESS: classify() knew `#ifdef NON_MATCHING` (:425) but not `#if 0`, so a dead
analysis body was read as a live definition AND its real INCLUDE_ASM stub counted separately —
the SAME function in BOTH `real` and `stubs`. Live case: resident.c:868-925 wraps a full
void func_800D00E4(s32){...} in #if 0 (its jtbl dossier) and re-declares the stub at :926.
Now the block is skipped entirely: dead code is neither matched nor stubbed.
- THE len()-SUM (the dual defect): `placed` was a set union, so it caught a function in NO bucket
— but `matchable` SUMMED len()s, so a function in TWO buckets counted twice and nothing
complained. matchable/byteident are now SET unions, plus a new OVER-coverage assertion that
fails loudly if any fn lands in multiple buckets. R32 means both directions: nothing missing,
nothing double-counted.
- NEGATIVE CONTROL (the fix must change an answer the old tool gave):
resident REAL 123 -> 122 | matchable 146 -> 145 | 85.62% -> 85.52% | func_800D00E4 no longer
double-counted. FLEET instr 68.9% UNCHANGED (no #if 0 in the overlays) — the fix is scoped.
- FINDING (logged for T7's audit-binaries, does NOT block the flag-plant): the two INDEPENDENT
oracles now agree exactly at 144 — corpus (21 stubs + 123 matched, derived from the tree) and
sig_image (the 2nd oracle) — with EMPTY set difference both ways. progress.py still reports 145
because it counts func_800CEDFC and func_800D33E0, which are DEFINED in resident.c but absent
from sig_image. 0x800CEDFC is the resident's vram base +4 (the first function, code starts at
file offset 0x4 after the leading data word), yet make audit-corpus reports 0 PHANTOM +
0 TRUNCATED. Either sig_image has a boundary blind spot or those defs are not image functions.
progress.py's text-scanning classify() is exactly the re-parsing R33 says should be DERIVED from
corpus instead — a real refactor, logged not rushed.
The flag-plant claim is unaffected: it rests on corpus.stubs('resident') == 21 (tree-derived,
verified 5 ways), not on the contested denominator.
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
The whole-binary byte-gate is structurally blind to code nobody onboarded (R34): check-all is
green over the onboarded set no matter what code sits unbuilt on the disc. This reconciles the
onboarded set against every code-bearing PAC payload.
- new_overlay.sh: optional [ENTRY] arg (default 0.4) reaches a non-0.4.dec payload. Onboarded
ov_SC07_{006,007,010,011} from 1.4.dec (they put graphics at PAC entry 0, the code overlay at
entry 1 — invisible to the 0.4 hardcode for a month). Each byte-identical (7ca772be / b3b95547 /
d7b5875d / 9885af74). FLEET 136 -> 140; check-all 140/140 (T2's pass==N re-baselined cleanly).
difficulty.py NOT in the insertion set anymore (it derives, T6) -> only 3 tool dicts touched.
- tools/disc_code_sweep.py: decode every payload (reusing sig_image.make_insn) and gate code on
BOTH valid>=0.90 AND jr_$ra density>=0.01. The jr_$ra gate is decisive: isValid() alone flags
389 false hits (type-0/2 structured data decodes ~100% valid but has ZERO returns); jr_$ra
separates code (~2.9-3.4%) from data (0.000%), validated on positive+negative controls.
- FINDING (docs/disc-completeness.md): type-4 location overlays are COMPLETE (138/138). All other
types are data EXCEPT type-1 = 40 code payloads, 1 onboarded (the resident), 39 HIDDEN
resident-class modules (mostly MAIN.CD/FILE_XXX/1.1). They load at UNKNOWN addresses (not the
shared overlay slot), so they are NOT mechanically onboardable — byte-verifying a build binary
needs its load address (P9), knowable only by runtime RE (the Phase-3 method). Deferred with
evidence, NOT force-onboarded at a guess.
- CONSEQUENCE: game-code TRUE 100% now spans 140 onboarded binaries PLUS ~39 type-1 modules
pending load-address RE. The roadmap assumed 136 — this is a real re-baselining (the +4 overlays
also add ~2.45 MB to the denominator; every family propagation is now x138). Flows to T10/T11.
- SETUP §6.3 tool inventory updated (R21).
TWO REAL BUGS in classify(), and an HONEST CORRECTION of their blast radius (P9/R14).
- BUG 1 (under-count). classify() decides definition-vs-declaration by scanning to the first `{` or
`;`. A K&R definition puts its parameter declarations BEFORE the brace:
s32 func_8015AE2C(arg0)
s32 arg0; <- a `;` before the `{`
{ ... }
so it was read as a forward declaration and dropped into NO bucket — not REAL, not a stub,
invisible. And a K&R def is MANDATORY whenever a zero-arg engine_core.h thunk calls the function,
i.e. exactly the heavy-jr cores our own banking recipe produces: func_8015AE2C (562x134),
func_8015A3C8 (493x132), func_80166994 (369x134) were all compiled, linked and BYTE-IDENTICAL in
the shipped build while counting as zero. Fix: skip over K&R parameter declarations (a bare
`<type> <name>;` carrying no parens — that is what distinguishes it from a wrapped ANSI
prototype's continuation line, which always carries the `)`).
- BUG 2 (over-count). `real |= dedup_members(BINARY)` folded in EVERY registered dedup member without
checking it is actually instantiated. A member still sitting as an INCLUDE_ASM stub was counted
REAL *and* stayed in `stubs` — double-counting into `matchable` and inflating `byteident`
(532 phantom instances, per the scanner audit). Fix: subtract `stubs`. The registry is advisory;
the source tree is authoritative.
- COVERAGE ASSERTION (the rule ratified 2026-07-14): ground truth = every function splat emitted a
.s for. Anything classify() cannot place in ANY bucket is now reported LOUDLY (stderr + the .md),
because a silent skip is a defect, not a no-op. Currently: 0 unplaced.
- CORRECTION (this is the part that matters — I over-claimed and the bytes refuted me). The scanner
audit reported ~243k instructions "counted as nothing", and I repeated it. WRONG. weighted_metrics()
— which produces the HEADLINE instr-weighted and distinct-code numbers — does NOT call classify()
at all. It tests `func not in src_stubs(binary)`: since the fleet is 136/136 byte-identical,
anything not wrapped in INCLUDE_ASM must be compiled C emitting the exact original bytes. That test
never parses a definition, so it is IMMUNE to this bug. Verified: old-vs-new on the same tree gives
identical weighted numbers. The published 65.6% / 44.9% were CORRECT ALL ALONG; only the secondary
REAL count and fn-count % were wrong.
THE LESSON, sharper than the one we started with: a metric DERIVED FROM A PROVEN INVARIANT beats a
metric that RE-PARSES THE WORLD. weighted_metrics() leans on the byte-gate and inherits its
correctness; classify() re-derives the same fact by parsing C and inherited a bug instead. Prefer
the former wherever an invariant exists.
- weighted_metrics() from .run/sig.*.jsonl + src stubs (executable code only, resident + 134
overlays; main EXE excluded). Two framings: fleet instr-weighted (per-overlay, the decomp.dev
-display number) + dedup distinct-code (each unique h_exact once, the distinct-RE number)
- --fleet now emits THREE labeled metrics into docs/progress.fleet.md: fn-count 74.48% (×134-
inflated), instr-weighted 56.8% (shipped .text), distinct-code 27.3% (of 84,996 unique fns)
- --weighted prints the two weighted numbers standalone; degrades gracefully if sigs absent
- corrects the stale "~30-35% byte-weighted" estimate: the giant campaign since Phase 19 raised
the fleet instr-weighted number to 56.8%; the distinct-code 27.3% is the unique-monster-tail truth
- SETUP §tooling row updated (R21)
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index
(fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic,
source-derived per-binary table + fleet totals). Single-binary output byte-unchanged.
- fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%
across 136 binaries (main + resident + 134 overlays)
- dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES
entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes
to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups
/ 28.6M collapsible (matches the Phase-11 figure)
- Makefile: wire progress.py --fleet into make report (main-only block)
- docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory
- make report green; dedup-check 8 validated / 0 failed
- ghidra/ churn NOT staged (R23)
- onboarded all 118 not-yet-onboarded overlays via tools/new_overlay.sh (0 failed)
-> fleet = 134 location overlays + main + resident = 136 binaries
- R22 CLEAN rebuild (make clean -> extract all 136 -> make check-all):
136 passed, 0 failed of 136 -- every overlay byte-identical at 100% INCLUDE_ASM
from a fully clean tree, 0 extract failures
- per-overlay config (splat.ov_*.yaml / check.ov_*.sha / symbols.ov_*.txt) +
config/overlays.mk var blocks + the 4 report/diff BINARIES dicts auto-registered
(sentinel-anchored, idempotent); src/ov_*/ stub records committed (harvest_verify
reverts via git checkout, so the records must be tracked before any harvest)
- ghidra/ churn NOT staged (R23); build/asm/assets/.run gitignored
- report-digest regen (fleet-wide duplicates.cross.md + progress.fleet.md) is T4
- onboarded ov_{SC01_000,SC01_001,SC02_000,SC02_003,SC03_002,SC03_006,
SC04_000,SC04_018,SC04_019,SC05_000,SC06_000,SC07_000} via tools/new_overlay.sh
— spans all 7 SC areas + 2 full duplicate pairs (SC02_000≡003, SC04_018≡019)
- each byte-identical at 100% INCLUDE_ASM (non-4-aligned auto-handled); no
position-lock anomalies — the cross-shape risk is retired before the full onboard
- make check-all = 18/18 passed (main + resident + 4 prior + 12 new)
- config/overlays.mk + 4 report dicts auto-registered (sentinel, idempotent)
- ghidra/ DB churn NOT staged (R23); asm/build/assets/.run gitignored
- src/shared/clearTbl40.h: CLEAR_TBL40 macro = the matched byte-clear loop body, authored ONCE;
instantiated at both func_80037004 and func_80037334 in src/800.c (one source -> two vrams)
- matched first try; clean rebuild main -> 143dbb89 BYTE-IDENTICAL WITH the shared C AND WITHOUT
it (INCLUDE_ASM stub fallback) -> dual invariant proven (R22 clean rebuilds, both states)
- config/dedup.us.yaml: I0_clearTbl40 group registered (h_exact a0744d60…); dedup_integrate
--check validates it; negative tests (corrupt hash / wrong vram) fail-closed (exit 1)
- tools/progress.py: count dedup-shared members as REAL via the registry (the macro form isn't a
parseable function def); REAL 52 -> 54, byte-identical 50.24% -> 50.33%; honest measurement (P9)
- tools/dedup_integrate.py: display vram in hex in diagnostics
- the machinery half of the Phase-11 milestone is proven on the byte-verified EXE
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
(build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
(Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
(52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
Drew T