The near-miss ledger (.run/backlog.jsonl) is append-only, so it filled with already-banked noise:
6,867 rows, ~98% banked. load_best()/render() already filtered on READ (docs/backlog.md was correct),
but the raw log drifted stale and every render re-scanned all 6,867 rows against the stub oracle.
- backlog.py: new `prune` subcommand — atomic rewrite (temp + os.replace) to load_best()'s output
(drop-now-matched P9 + best-per-addr collapse). Idempotent. 6,867 -> 1,704 open near-misses.
- Makefile: `backlog.py prune` wired into `make report` (BINARY=main block) so the ledger tracks
reality every cycle instead of drifting.
- Finding (Drew's question): crack waves DO log every non-byte-match to the backlog durably
(gate_stage copies best_draft -> .run/backlog_drafts/). BUT the `closeness` field is UNRELIABLE —
byte-correct drafts (match_one MATCH) are logged with closeness>0 (e.g. func_8012F49C logged 29,
actually MATCH). And a reach-N function's draft is overlay-SPECIFIC (per-location symbols), so the
backlog is a messy recovery source vs the fresh per-wave stranded drafts. Integration-recovery
should consume the fresh wave-dir strandeds, not re-derive from the backlog.
- ROOT CAUSE PINNED (the session-2 half-pin was INVERTED; both probes were vacuous, R35):
cc1 emits .align 3 before EVERY jump table; maspsx passes it VERBATIM (the :435 'drop' is
an inventory-only pass); as bakes the pad SECTION-RELATIVE; link placement was never guilty
(SUBALIGN(2) + ALIGN(.,4) place 4-mod-8 carve starts tight). Merging originally-separate
TUs fires an intra-TU align where the original packed tight -> +4 at rodata 0xCC ->
image-wide %lo shift. Honest probes persisted: .run/probe_jtbl/ (verdict.md + objdumps).
- NEW tools/jtbl_rodata_pads.py: post-maspsx filter replaces each rodata .align 3 with the
ORIGINAL's exact pad bytes per a JTBL_PADS spec; fail-loud on table-count drift /
non-align-3 / non-jtbl rodata content. Byte-proven: verbatim 0xE4 pad-at-0xCC ->
filtered 0xE0 tight (= the merged carve span).
- jtbl_carve.py: spec-aware same-subseg merge (gap 0 or 4-with-zero-payload-word; else
NON-CONTIGUOUS -> isolate), interval-arithmetic pad specs (committed values CARRIED,
never re-derived), JTBL_PADS target-var emission into overlays.mk + revert() restore +
stale-.o invalidation; the false 'maspsx drops .align' docstring corrected (H5).
- Makefile: $(if $(JTBL_PADS),| jtbl_rodata_pads.py ...) stage in build/src/%.o + file-scope
empty default (env-shield). jtbl_family_bank.stub_file: duplicate-stub fail-loud (the
earlier 'ladder failure' was a wrong-TU splice into a stale _a.c stub, byte-witnessed).
- R22 clean-fleet WITH the fix wired: 140/140 byte-identical, tools-health green
(dedup 1846/0, C1 234205/234205), ZERO new banks -- fleet-neutral by construction.
- cookbook §8e (the jtbl alignment law) + §8a/§8a-pad corrections; decision-log R31 entry;
SETUP.md tool row; .gitignore allowlist for the probe verdict artifacts.
R36: a newly-discovered binary is not real until every consumer knows it. Onboarding produces a
byte-CLEAN binary (check-all green) that is not yet a CITIZEN — the tools that enumerate binaries
can each be silently unaware of it, and the byte-gate is structurally blind to that (R34).
Not hypothetical: P27 onboarded 4 SC07 overlays byte-clean; P28 found FOUR consumers silently
ignoring them (family_remap.img_path, .run/family_hseq.json, config/dedup.us.yaml, and the overlays'
own .c), hiding ~6,400 already-matched bodies. Every failure was silent.
- tools/audit_binaries.py asserts, coverage-checked BOTH directions (R32), against the config the
BUILD reads (R33 — onboarded = main + resident + every config/splat.ov_*.yaml):
1. dup_report.BINARIES (what corpus/family_hseq/progress all derive from) EXACTLY equals the
onboarded set — a missing binary is invisible to every derived tool; a phantom is invented.
2. every onboarded binary has a byte-derived sig.
3. THE LOAD-BEARING SC07 CHECK: every onboarded OVERLAY's .c includes ../shared/engine_core.h,
or no shared body can ever reach it (main/resident have their own bodies, exempt).
4. every onboarded overlay is represented in the family map (warn — regenerable/may post-date).
INFO: dedup-group membership (0 = onboarded-but-un-harvested, a dedup_extend candidate).
- NEGATIVE CONTROL: stripping the shared include from ov_SC07_006.c makes the gate FAIL loudly and
exit 1 — it catches the exact bug that hid 6,400 bodies for a month. Restored clean.
- Wired into `make tools-health` (the pre-matching ritual) — cheap (config + text scans, no build),
so it sits in the fast lane. Passes today: 140 onboarded, all full citizens.
- Reads config/dedup.us.yaml as TEXT (never a YAML round-trip — the H5 lesson from T4).
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each
whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and
every earlier failure is swallowed. `dedup-check` "gated" purely by being last;
lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the
26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one)
biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a
report-invoked tool is swallowed on arrival.
- .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its
contract is accumulate-every-failure-and-report, which -e would truncate at the first
missing tool).
- check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e
treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did.
- check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker.
The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32).
- new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate
pre-matching ritual the roadmap's standing invariant names, and the dependent the two
derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl
cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21).
VERIFIED:
- NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under
the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical.
- the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation.
- check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already
-e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard.
- R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild
drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89.
- audit-corpus 7s / audit-cdecl green / tools-health wired.
The ONLY detector for the R22 rename-drift failure mode (a symbols.us.txt rename leaves a
func_<ADDR> ref dangling in committed src; a clean rebuild fails, an incremental build masks it
with a stale .o — undetected Phase 21→23). It was RED (262 false positives) and UNWIRED. The
audit's 3 blind spots, all fixed:
#1 globbed src/**/*.c only -> now ALSO scans src/shared/*.h, where engine_core.h's 10k+ func_/D_
tokens live and one dangling ref breaks EVERY clean build at once (negative-control-proven:
an injected bare ref in a shared header IS flagged).
#2 read 2 of 138 symbol files -> now reads every REAL stacked file (config/symbols*.txt), and
correctly EXCLUDES the R13 proto files (never stacked into a build; would invent phantoms).
#3 no __asm__("label") model -> ALL 262 false positives were this class: engine_core.h binds
func_8005C324 to memcpy via `... __asm__("memcpy")`, so the ref emits the LABEL and resolves
at link regardless of the rename. asm_labeled_addrs() now recognizes it.
VERIFIED: green on HEAD (exit 0); negative control (inject a bare func_800d1bd8 -> DsMix in a
shared header) FLAGS it (exit 1) then clears on removal — detection intact, not trivially green.
Wired as a fail-closed gate in `make report` (make -n confirms), beside dedup_integrate --check.
No src/build change (report-time integrity gate) -> the byte-identical build is unaffected.
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
R22 CLEAN-FLEET: make clean -> extract 136 -> build 136 -> check-all = 136 PASSED, 0 FAILED.
make audit-corpus: 0 PHANTOM + 0 TRUNCATED (was 193).
=== A4: a CORPUS defect the byte-gate could never have caught ===
config/symbols.us.txt:981 declared `listCdBuffer = 0x80180000` — a correct Phase-3 name for MAIN's
LIST.CD RAM buffer. But that address is OUTSIDE main's image and INSIDE the overlay slot, and every
overlay's splat config stacks symbols.us.txt. High RAM is REUSED: an address that is a buffer to main
is live CODE to an overlay. So splat saw a symbol boundary mid-code and, across 97 of 134 overlays:
* CUT 97 REAL FUNCTIONS IN HALF (a head ending on a `lui`, no return), and
* INVENTED 96 PHANTOM ONES (a tail beginning by reading the assembler temp $at).
193 slices NOBODY COULD EVER MATCH — not "hard", not "a compiler wall": unmatchable by construction.
They sat in the harvest queue as ordinary work, so agents would burn on them forever and the failures
would be filed as intrinsic compiler residuals.
The phantom listCdBuffer.s in ov_SC01_005 literally begins:
lw $ra, 0x10($sp) / addiu $sp, $sp, 0x18 / jr $ra
splat cut a function immediately before its EPILOGUE and called the epilogue a function.
AND IT HAD ALREADY CONTAMINATED REAL WORK: in ov_SC03_031 the cut landed where the epilogue was
exactly `jr $ra; nop`, so the Phase-26 x134 sweep innocently BANKED the phantom as
`void listCdBuffer(void) {}` — byte-correct, gate-green, entirely fictitious — while leaving
func_8017FFC4 permanently unmatchable. Removed.
WHY NO GATE CAUGHT IT, AND WHY THAT IS THE POINT: INCLUDE_ASM pastes the two .s halves back VERBATIM
in original order, so the image is byte-identical either way. The byte-gate was green the whole time
and always would have been. It is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle. No
assertion added INSIDE it could ever have found this. What found it was a SECOND, INDEPENDENT oracle:
tools/sig_image.py derives boundaries from the ORIGINAL bytes without splat, and DISAGREED with the
corpus (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements).
=> When one oracle is structurally blind to a class of error, the answer is not a better assertion
inside it. It is a SECOND ORACLE THAT CAN DISAGREE WITH IT. (`make audit-corpus` is now that.)
THE RULE (the mirror of R13/R15, never written down): a symbol whose address falls inside ANOTHER
binary's vram window must never enter that binary's symbol stack.
FIX: config/symbols.us.ram.txt — main-scoped symbols outside main's image — stacked ONLY by
config/splat.us.exe.yaml. Main keeps the name it needs (10 %hi / 11 %lo refs; 143dbb89 byte-identical);
the overlays never see it. Exactly one symbol was in scope fleet-wide; the resident window was clean.
AND A REAL FUNCTION THE ACCIDENT WAS HIDING: in ov_SC01_084 / ov_SC02_041 / ov_SC03_094 / ov_SC06_008
there IS a genuine function at 0x80180000 (111 / 35 / 28 / 74 ins), reachable ONLY via a fn-pointer
table (.word func_80180000) and never by `jal` — so splat cannot find it and needs the boundary
DECLARED. listCdBuffer had been supplying it by luck. Now declared honestly, per-overlay, in
config/symbols.<ov>.txt — exactly where R13/R15 says an overlay-scoped symbol belongs.
=== A5: the closeness oracle every crack agent trusts was lying on 155 functions ===
masked_diff._reloc_kind() knew 26/HI16/LO16. An over-approximating sweep of every reloc objdump emits
across all 3,367 build objects found FOUR: R_MIPS_26, HI16, LO16 — and R_MIPS_PC16 (211). PC16 fell
through to a FULL-WORD compare, but the object holds an UNRESOLVED PLACEHOLDER in the branch
displacement, so that compare can NEVER succeed.
DECISIVE TEST (derived from the invariant, not from reading the regex): INCLUDE_ASM pastes the
ORIGINAL asm, so for every stub diff_object_s() MUST be 0. Measured, coverage-asserted:
2,741 functions scored — old mask: 150 LIES; PC16 masked: 4 LIES.
(The 4 survivors are the separate length-delta defect.) A phantom non-zero sends an agent to grind at
a wall that is not there, and the wasted attempt is then booked as a MATCHING failure, feeding
reserved_walls() and PERMANENTLY BLACKLISTING a function that was never broken.
=== NEW FINDING (found by cutting the R22 corner): a STALE OBJECT CAN PRODUCE A FALSE PASS ===
`.o <- .s` is not a dependency make can see: assembly arrives via INCLUDE_ASM, expanded to a `.include`
consumed by maspsx/as AFTER cpp, while -MMD tracks headers only. Re-extract, build incrementally, and
make links a STALE object. This is not merely slow — INCLUDE_ASM pastes the ORIGINAL bytes, so a stale
object still yields the original image: SHA1 GOES GREEN while the split just changed is never exercised.
A broken config change can be "verified" by an incremental build. Live proof: 8 of 136 binaries linked
stale objects here; they failed LOUDLY ONLY BY LUCK (the dead symbol was an undefined reference) — a
merely-different-but-valid split would have gone green on all 136.
R22/H3 already legislate this, and I broke them. But a rule that needs a human to remember it is not a
gate. FIX: `extract` now invalidates the objects that include what it just rewrote (main's are top-level,
so -maxdepth 1 — verified it cannot clobber the other 1,605 objects). Structural, not advisory.
R14 self-catch, recorded: my first A5 test passed `fn=` to diff_object_s(), which takes two args; the
TypeError was swallowed by my own `except Exception: continue` and it reported 0 scored / 0 lies. I
wrote the exact bug I was auditing, inside the test for it. Caught only because 0 looked wrong. The
test now asserts its own coverage.
The 28 surviving audit findings collapse to ONE bug repeated ~10 times: a hand-maintained model of
the corpus layout (a file allowlist, a single-.c assumption, a func_-only symbol regex, a REGION_SUB
dict) sitting on top of a filesystem that already answers the question. The fix is not ten repaired
regexes — it is one DERIVED oracle and ten deleted scanners (R33).
WHAT IT DERIVES FROM
1. THE FILESYSTEM. Which .c files make up a binary, and where a function's .s lives, are FACTS OF
THE TREE THAT SPLAT ITSELF WROTE. The INCLUDE_ASM line is SELF-DESCRIBING — its first argument
IS the asm subdir — so there is nothing to guess and no dict to rot. A dict literal is strictly
worse than the filesystem AND it fails OPEN (silently yields a wrong path) instead of closed.
2. THE PROVEN INVARIANT. INCLUDE_ASM pastes the ORIGINAL asm and the build is byte-identical, so a
function NOT wrapped in it is byte-exact. `matched` is DERIVED as sig - stubs, never re-parsed
from C text. (progress.py learned this the hard way: weighted_metrics() derived and was right;
classify() re-parsed C and inherited a bug.)
VALIDATED against the real corpus:
* ov_SC01_077: 264 stubs across 14 files. The old 3-file allowlist saw 30.
* Fleet: 58,717 stubs vs the allowlist's 1,992 — 56,725 (96.6%) were INVISIBLE.
* Coverage-asserted (R32): every INCLUDE_ASM line must parse, every symbol must resolve (ANY C
identifier — a func_-only regex silently misses the 100 curated listCdBuffer stubs), every stub
must have a .s. A silent skip is a DEFECT, not a no-op.
THE SECOND ORACLE (`make audit-corpus`) — the real lesson of this audit.
The byte-gate is structurally BLIND to a bad function boundary: the .s halves are pasted back
verbatim in original order, so the image stays byte-identical and green. Only an oracle that can
DISAGREE can see it. sig_image is that oracle — Ghidra-free, derived from the ORIGINAL bytes,
independent of splat. corpus.audit() cross-checks the two and reports:
PHANTOM — a stub address the sig does not know: splat INVENTED a function.
TRUNCATED — a stub whose .s length != the sig's: splat MIS-SLICED one.
It reports 193 (96 + 97) — reproducing the A2 audit's number EXACTLY, from an independently written
tool. That is a third confirmation of the listCdBuffer defect (auditor -> skeptic -> this).
AND AN R14 SELF-CATCH, recorded because the near-miss is the lesson.
Run naively over all 136 binaries the same check reports 914 slices — 4.7x the truth. It is noise:
main/resident are signed by the GHIDRA dumper, whose boundaries are shorter than splat's by design
(and which never analysed the linked PsyQ subsegs at all), so the comparison measures GHIDRA'S limits,
not splat's errors. Only the overlays are signed by sig_image, the oracle actually validated at
58,524/58,621. sig_is_independent() now encodes that domain, with the reasoning, so nobody repeats it.
A check applied outside its valid domain does not become more thorough — it becomes noise.
`make audit-corpus` is RED by design until A4 removes the bad symbol line; then it becomes a gate.
- profiled the clean-fleet R22: extract-all ~6m11s (136 serial `splat split`) + check-all
~2m58s (136 serial builds) = ~9 min, all serial on a 32-core box
- the only shared WRITE is the 4 generated include/*.inc macros at extract time (identical
content per binary); everything else is per-binary-disjoint and include/ is read-only during
a build -> concurrent builds/extracts don't race
- Makefile: JOBS ?= 16 + `make extract-all` (seed main serial for the macros, then parallel-
extract the rest via xargs -P) + parallel `make check-all` (xargs -P), correctness-gated
- MEASURED: `make clean && make extract-all && make check-all` = ~50s, check-all 136/136
BYTE-IDENTICAL (== the serial result) -> ~10.5x. Compounds across the endgame (R22 per commit)
- new R22 recipe: `make clean && make extract-all && make check-all` (was the serial for-loop)
- overlay jr-functions can now bank as C: gcc switch jump tables form a .rodata island at
the overlay TAIL; carve a matched fn's jtbl into a dotted [.rodata, <code-subseg>] subseg
+ ld_interleave (data->rodata->data sandwich) places it byte-exact. cookbook §8a + SETUP.
- tools/ld_interleave.py: --section .<binary> param (derives the <binary>_TEXT/DATA/RODATA/
DATA2/BSS symbol prefix); default .main = the EXE, byte-identical (backward-compat proven)
- Makefile + config/overlays.mk: <bin>_JTBL_INTERLEAVE hook + a $(strip)-guarded extract
branch (gotcha caught: a trailing #comment on the := left whitespace -> non-empty -> the
branch misfired on resident with the EXE defaults)
- PoC: func_8012ACE0 (25-ins jr-fn in ov_SC01_077) reconciled (canon_sig_reconcile) + banked
BYTE-IDENTICAL d19c9580 -- the first overlay jr-function matched through the C pipeline
- R22 FULL-FLEET clean rebuild: 136 passed, 0 failed (main 143dbb89 unaffected by the
ld_interleave change); 0 NON_MATCHING in any default build (G4)
- P9 findings: func_80159C84/func_8015444C (the 2 carried Fable5 jr bodies) are rtu_match
FALSE-matches (incomplete jtbls: 52B vs 56B -> never bank); the maspsx "hang" scare was a
truncated experimental-file artifact (real pipeline builds in ~1s)
- metrics: distinct-code 39.1% (50,572 unique fns), instr-weighted 62.9%
- NEXT: the ×134 automation (generate the per-overlay carve + template the reconciled body)
The single biggest byte-weight lever (770 ins ×134 ~ +1.6% byte-weight), banked across all 134
overlays as -O0 compiled C (was ×1 in ov_SC01_077, commit:0463).
- ROLLOUT (tools/rollout_whale_o0.py): per single-file overlay, LINE-split <ov>.c at the whale
(splat emits in vram order), carve the yaml code subseg into before/<ov>_o0b(-O0)/<ov>_after,
thin <ov>_o0b.c #includes the shared src/shared/func_80144B9C.h (the whale C moved there — DRY,
TU-local typedefs, not a fragile 200-line DEFINE_ macro). Makefile WHALE_O0B_OBJS wildcard -O0.
- DEDUP: group E_func_80144B9C (134 members, h_exact 74186b97, source=the header; group_members
keys on binary+vram so a header-share validates like a macro-share).
- MILESTONE: clean fleet check-all 136/136 (R22); dedup-check 1811/0; FLEET REAL 225816->225949,
byte-identical 65.91->65.95% function-count (the gain is byte-weighted). Validated on ov_SC01_000
+ spot-checks SC02/SC03/SC07 before the full fleet.
- cookbook §38 + SETUP inventory.
- CRACK (cheap Opus, no Fable5/calls.c): the 770-ins reach-134 whale's 2-insn residual was a
STRUCT ASSIGN `D_80078E50 = *ent;` (sizeof(ENT)==0x24), NOT an explicit memcpy() call — gcc-2.7.2
-O0 expands a >MOVE_RATIO-word struct copy via emit_block_move->emit_library_call(memcpy),
precomputing dst/src into pseudos then addu into $a0/$a1 = the exact 2 missing moves.
- -O0 SPLIT: whale is a 2nd -O0 region (prologue 21F0A003) in the -O2 'after' segment; carved into
its own -O0 object o0b (splat.ov_SC01_077.yaml 3-way + Makefile CC1FLAGS=-O0; address-sorted
line-split preserves all banked C). ov_SC01_077 == d19c9580.
- memcpy: memcpy=0x8005C324 in symbols.resident.txt (overlays+resident only; main untouched vs its
MEMCPY.o) + __asm__("memcpy") on the engine_core.h block-copy macro (non-builtin C name -> no
built-in codegen, same call).
- MILESTONE: CLEAN fleet check-all 136/136 byte-identical (R22), dedup 1810/0. reach-134 -> ×134 (W9) next.
- config/splat.ov_SC01_077.yaml: 3-object code split (ov_SC01_077_a before /
ov_SC01_077_o0 -O0 cluster / ov_SC01_077 after). A single object's .text cannot be
split around a middle object, so before/after are distinct objects; the after-region
keeps the ov_SC01_077 name (bulk matched C + asm paths unchanged).
- Makefile: target-specific CC1FLAGS:=-O0 for ov_SC01_077_o0.o (src/boot.c precedent).
- src/ov_SC01_077/ov_SC01_077_a.c (new before-region) + ov_SC01_077.c (after-region) +
ov_SC01_077_o0.c (new -O0 cluster): 6/16 -O0 fns matched byte-perfect
(func_8013B568/B7F4/BD34/C360/C938/C964).
- DEFERRED to Phase 20 (Drew, on the discovered difficulty): the 10 remaining -O0 fns hit
an indexed-global %lo-folding codegen quirk (gcc-source research, R17) + the x134
per-overlay rollout (engine_core.h compiles -O2, cannot carry -O0 fns). Corrects the
Phase-18 backlog premise (afternoon/free-x134) per R14. Documented in ov_SC01_077_o0.c.
- verified clean: main 143dbb89, resident 8e17e02f, ov_SC01_077 d19c9580 + 2 overlays.
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index
(fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic,
source-derived per-binary table + fleet totals). Single-binary output byte-unchanged.
- fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82%
across 136 binaries (main + resident + 134 overlays)
- dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES
entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes
to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups
/ 28.6M collapsible (matches the Phase-11 figure)
- Makefile: wire progress.py --fleet into make report (main-only block)
- docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory
- make report green; dedup-check 8 validated / 0 failed
- ghidra/ churn NOT staged (R23)
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in
src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing
that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in
config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X).
- proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all
18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test
(corrupt shared body) -> make check FAILS then restores
- Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a
shared header recompiles dependents — closes a stale-build/false-pass gap the negative
test exposed; output-neutral (R22 clean 18/18)
- structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate
can't catch under-application -> assert DEFINE present + stub gone
- FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays
(2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest
- cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory
- ghidra/ churn NOT staged (R23); build/asm/.run gitignored
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) +
detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes
~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer
tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix
- Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s)
- dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed
source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals;
compact member sample + #bin column
- RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm
8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one
match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue
- dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob,
no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates)
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
current sig hash drifted from the recorded hash (a stale share can never silently
mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
one object per binary so the linker can't swap them; the byte-gate is the existing
per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
-> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row
with its binary, buckets by h_exact then h_norm across binaries, splits
cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by
collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md
- Makefile: make report emits the cross report once (gated BINARY=main)
- per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged)
- EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the
byte-finding predicted); real cross groups arrive with overlays (T6)
- phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr +
PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for
Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8.
- tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points
(.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143);
this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted.
- FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range
PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to
Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7.
- config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11);
stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22),
main still 143dbb89 (no regression)
- Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl)
- dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder)
- ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op)
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry
(build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs)
- dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent —
write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export)
- progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout
(Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs)
- Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling
baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings)
- reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED
(52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified)
- Makefile: BINARIES += resident; resident_* var block (vram 0x800CEDF8, build/resident/,
config/*.resident.*); per-binary ASM_DIR/SRC_DIR + $(BINARIES)-derived OBJS prune-list so
main (asm/+src/) and resident (asm/resident/+src/resident/) object sets stay disjoint;
per-binary UNDEF_SYMS/UNDEF_FUNCS (resident writes under build/resident/)
- config/splat.resident.yaml: flat blob config (NO header, NO gp_value, single c seg @
vram 0x800CEDF8, stacked symbols [symbols.us.txt, symbols.resident.txt], per-binary
asm/src/build/undefined paths, EOF 0x5935C -> end vram 0x80128154)
- config/check.resident.sha (8e17e02f… = extracted MAIN.CD/FILE_010/1.1, 365,404 B, type-1)
- config/symbols.resident.txt (R13/R15 provenance banner; seeded empty, populated in T4)
- gates: (a) main clean rebuild 143dbb89… (refactor is a no-op on the byte-locked EXE);
(b) make extract BINARY=resident -> 144 INCLUDE_ASM stubs + .ld @0x800CEDF8;
(c) main OBJS=81 excludes resident even with resident on disk + main stays 143dbb89…
- src/resident/*.c left WIP-untracked (reshaped by T2 boundary carving)
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's
sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific
- Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no
rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly
- GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag);
negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded);
restore -> 143dbb89
- BINARIES/BINARY guard + main_* var set + selected-binary aliases; the EXE's
artifact paths (build/us/, config/splat.us.exe.yaml, check.us.sha) preserved
verbatim so its rebuild stays a byte-exact no-op
- 9 PsyQ SDK-integration blocks wrapped in ifeq(BINARY,main); SYMS/LD/OBJCOPY
stay outside (run for every binary); a 2nd binary skips the SDK block
- gate: make clean && extract && build -> 143dbb89 BYTE-IDENTICAL; make report
52/959/7/50.24%; BINARY=bogus errors via the $(filter) guard
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
pick the byte-matching object per aliased address (link_object), exclude 4 addresses
that don't reconcile in-region (kept as byte-identical stubs):
0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T — scattered-.bss commons, cross-object (§9.1)
0x3D94C S_IH/UT_RON — false placement (inside libsnd SSSTART.o)
0x3FA64 VM_F (237 ins) — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a
multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary
gotcha so it can't recur). Reused for libspu/libsnd next.
- libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code
interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window
0x4787C..0x51804 so it sees exactly the 22 in-region blocks
- 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay
stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist
- progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long
multi-block lists are passed via a make var
- region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts
with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags
- dual byte-gate PASS: 143dbb89 with and without the libgte objects
- LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93%
- reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last)
- splat: libc2 C stdlib in 2 blocks — main 16-obj run libc2_1 (BZERO/MEMCPY/STRCMP/
PRINTF/PRNT[jtbl ok]/.../SETJMP) split from 800c, + STRCAT.o libc2_2 split from
800c2; new game-code fragments 800c3 + regenerated 800c2
- region byte-verified 17/17; PRNT.o printf-format jtbl resolves via NOLOAD .rodata
- BOUNDARY GOTCHA found + fixed: SETJMP.o .text is 0x80 B (8-align pad), not the
0x78 psyq_identify reports (30 ins) -> the libc2_1/800c3 boundary was 8 B too low,
overlapping SETJMP's padded tail; the relink inserted +8 padding and shifted the
whole downstream image (56k diff bytes, +8 file len). Fix: boundary = last obj's
readelf .text size (0x5CE18), not ins-count. Lesson recorded in docs/psyq-worklist.md
- when a library block sits at a subseg start, rm the old .c so splat regenerates it
- dual byte-gate PASS: 143dbb89 with and without the libc2 objects
- LINKED 528 -> 590; REAL still 43; byte-identical 32.25%
- splat: split 800c into [800c][libmcrd1][800c2][libmcrd2]; LIBMCRD.o (2186 ins,
the 55 LIBMCRD_OBJ_* + _card_* memcard I/O) and USERFUNC.o (68 ins) are 2
non-adjacent blocks split by game/libc2/libapi/libcard code (800c2)
- Makefile LIBMCRD_* + gated psyq_integrate libmcrd1,libmcrd2 + -T
- region byte-verified: 2/2 objects, 27 .bss commons all recovered (not scattered)
- src/800c.c trimmed (0 real-C moved); src/{libmcrd1,800c2,libmcrd2}.c committed
- dual byte-gate PASS: 143dbb89 with and without the libmcrd objects
- LINKED 424 -> 528; REAL still 43; byte-identical 29.29%
- NB: these are the libmcrd SDK objects; the game SaveLoadRoutine/Q#5 is Phase 12
- splat: split 800b2 into [800b2][libgpu][800c]; libgpu block = EXT.o+PRIM.o
(vram 0x80058890-0x80059234, holds LoadClut/LoadClut2 + primitives)
- SYS.o (3109 ins) EXCLUDED — scattered-.bss commons (cookbook §9.1, the GS_001
class): SYS references .bss by section+offset but the original linker scattered
the commons across 0x80078xxx/0x800c5xxx, so no single NOLOAD base reproduces it.
Stays a byte-identical INCLUDE_ASM stub in 800c; documented in docs/psyq-worklist.md
- curated dir .run/obj40/libgpu_used = {EXT,PRIM} (validates the _used mechanism for
the later alias libraries); Makefile LIBGPU_* + gated psyq_integrate + -T
- src/800b2.c trimmed; src/libgpu.c + src/800c.c committed (stub records); the trim's
9 "moved" items were all splat-auto empties (regenerated identically, 0 real lost)
- dual byte-gate PASS: 143dbb89 with libgpu objects AND without (stub fallback)
- LINKED 375 -> 424; REAL still 43; byte-identical 24.32%
Drew T