mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 05:56:00 -04:00
0fa386c14dfd5a4e2e65da2c11417bde3e6df86f
59 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
e18738c48e |
feat(phase-31 T5): THE FRONTIER ATLAS — 5,139 lever-labeled crack groups over all 12,058 open fns
- tools/atlas.py: cousin units baseline + T1.5 h_seqn merges + CALIBRATED warm tier (measured: li-norm metric holds ~99% recall to 0.55; rule = smallest t with neg-accept<=0.2% AND recall>=95% -> THRESH_WARM=0.70 @ 99.1%/0.18% — false merges waste exemplar cracks, misses only route cheaper) + seed sweep (65% of open skeletons carry a >=0.55 matched seed) + kNN graph + tiny-direct + evidence joins (audit/backlog/ledgers/cards; unparsable=fatal) + lever labels with confidence measured>ledger>tell>default>UNKNOWN - partition ASSERTED: 12,058 = progress stubs 12,051 + NM 7 EXACTLY (chased the +1: data blobs now excluded, reconciled against classify() buckets; T1 banks confirmed absent); every instance in exactly one group; main joins at the atlas layer only (family maps stay non-main — 4 silent-skip hazards) - warm tier merged 1,019; top group unifies 268 drifted per-location skeletons - lever table: head-crack 186.9k ins / UNKNOWN 138.6k (honest) / extend-tell 76.7k / redraft 46.8k / jtbl-carve 45.7k / integration 23.4k / seeded 23.4k / len-vein 16.8k / swaprepeat 9.2k / plumbing 8.1k / o0 6.6k / cc1 6.4k - atlas_features: li_norm_toks exported (shared with atlas, R33; hash-stable); mid_jr verifier fixed (compared ZERO rows — R32 silent no-op; now 6,444/6,444) - make atlas = full regen chain (~10-15 min, zero tokens); --targets emits crack slates (12/12 .s resolved); survey 92 s - SETUP rows (R21); docs/frontier-atlas.md committed |
||
|
Drew T
|
22eed7d78a |
feat(phase-31 T3): main enablement — sig-main at splat-true lengths + main streams
- sig_image: --seeds accepts '0xADDR NINS' (and jsonl nins); a seeded nins is
authoritative ([addr, addr+4*nins), bypasses func_end whose heuristic mis-sliced
3/40 main samples); R32 guard on seeded end > hi
- corpus: s_ins_count() factored from audit() (R33, one counter) + '--seed-ends'
CLI emitting per-stub splat-true lengths
- make sig-main: 2,002 main stubs signed -> .run/sig.main.jsonl; FULL word
cross-check 2,002/2,002 EXE slices == .s words (0 SLICE-SUSPECT; .s word field
is byte-order hex, not LE — first checker draft misread 1,999 false suspects).
Deliberately splat-SEEDED; main's independent second oracle stays deferred
(second-oracle.md; sig_is_independent(main) stays False)
- family_remap: vram_of/img_path special-case 'main' derived from splat.us.exe.yaml
(file0-vram = code-seg vram - start = 0x8000F800; target_path); stream_words
('main') verified 25/25 vs .s
- regression: sig-resident re-run byte-identical after the shared read_seeds change
- SETUP §6.3 rows (R21)
|
||
|
Drew T
|
d6ade08f3a |
feat(phase-31 T0): pivot log + instrument freshness + hygiene
- decision-log: the P31 re-charter entry (organize-before-grind; R37/R38/R39 ratified at gate-1) per R31 - harvest_verify.py: import guard — a bare import now RAISES loud instead of running a full gate (CLI unchanged, verified both directions) - sig-resident: bootstrap boundary artifacts fixed (fused +0 data word with func_800CEDFC; func_800D33E0 dropped past a glued tail) -> ELF-seeded per the S45 pattern, exactly 145 fns; true denominator confirmed 145 (progress was right); audit-corpus 0 PHANTOM + 0 TRUNCATED; all three oracles agree - family maps regenerated at HEAD commit:2161: 11,025 open non-main members reconciles EXACTLY with 12,059 - main 1,034 (102 stale phantoms cleared); adapt cards 704, aprop cards 204 (full emission) - main fuel-gap finding: 2,001/2,002 main stubs already have cached Ghidra-C (only func_80049600 missing) — the roadmap '0/2,096' note was stale - tools-health OK (dedup 2,063/0; C1 254,521/254,521; audit-digest green) |
||
|
Drew T
|
a0f07d629e |
chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/ rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java + VerifyOverlay.java (ghidra_import_raw.sh is the live path) - reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green - SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol- window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21); disc-completeness Reproduce marked retired |
||
|
Drew T
|
4cadac4e11 |
feat(phase-30 S45 II.1c): module batch dedup-banked + verified — 408 banks, R22 183/183, audit-disc 75->34 (parked-only)
- dedup measure (R37 probe): 69/1,113 module fns h_exact-match matched corpus (~6%, LOW as
planned — modules are novel frontier); dedup_extend inapplicable (same-vram group model) ->
family_sweep --hseq --band all over the 57 matched-exemplar families: 408 member-matches
banked (182 into modules, 226 into the big 3 — families Part I's --only scoping missed),
169 failed + 77 STRUCT = genuine per-member frontier
- R22 clean-fleet 183/183 BYTE-IDENTICAL; audit-disc UNCLAIMED 75->34 residue 0 (34 = 31
parked-for-L3 + SC03/53,54,56 — 3 rows Discovery-3 never tiered, now parked with evidence)
- three instrument fixes, each negative-control-proven:
- family_sweep --hseq stub map derives ov_*+md_*+resident (was sig.ov_* glob -> module
members silently 'not-stub', R32 class) [committed earlier as commit:1506]
- sig-modules seeds from the built ELF's func_* symbols (bootstrap GLUES adjacent fns
around jtbl dispatch -> 24 false TRUNCATED; perturbed-sig control still bites)
- corpus.audit counts CODE lines only (module .s carries its header jtbl as .word lines);
progress.py buckets INCLUDE_RODATA symbols as blobs (unbucketed R32 hole)
- NEW HONEST BASELINE (183 binaries): 94.0% instr / 95.96% fn-count / 87.6% distinct;
tools-health OK, audit-digest OK
|
||
|
Drew T
|
ec1a805766 |
feat(phase-30 S44 I.1b): Makefile learns the module class — modules.mk + sig-modules
- -include config/modules.mk (silent when absent, same contract as overlays.mk) and BINARIES += $(MODULE_BINARIES). Everything downstream of $(BINARIES) — prune, check-all, build-all, expected — is untouched and picks modules up automatically. - NEW sig-modules target: signs every module at ITS OWN vram with ITS OWN --text-lo (the §154 module-id-word law — bootstrap from offset 0 yields 0 functions on 75/78 payloads). Derived MODULE_SIG_JOBS from modules.mk (R33, the sig-overlays pattern). Wired into tools-health after sig-resident. Empty registry = clean no-op (verified). - NEGATIVE CONTROLS: make -n sig-modules iterates an empty list; main rebuilds 143dbb89 byte-identical with no modules.mk present. |
||
|
Drew T
|
03794d91cd |
feat(phase-30 S43): make audit-disc — the disc PARTITION holds at residue 0; 34 UNCLAIMED code payloads
L1 of Drew's definitive disc audit ("we really need a full audit that definitively lists ALL code
that we need to decomp"). THE INVARIANT (R32): every byte on the disc belongs to exactly ONE bucket,
the buckets SUM TO THE DISC, and residue is a DEFECT — a partition with an asserted residue of zero
is a completeness proof; a longer list is only a longer list.
- WALKS THE DISC IMAGE, NOT OUR CONFIGS, classifies WHOLE payloads (no window), and decodes BOTH the
raw and LZSS layers — the three shapes that produced the three "more code all along" surprises
(the 0.4.dec glob missing 4 SC07 overlays; disc_code_sweep blind to COMPRESSED code, its type-4
row vacuous for 138 known binaries; a 4,096-word window reading only payload heads).
- CLAIMED-BY IS DERIVED (R33): config/check.<bin>.sha IS the SHA1 of that binary's disc payload, so
payload->binary is a hash lookup against the build's own byte-identity gate. It cannot drift.
- RESULT, 416,021,760 bytes, 1,291 payloads, RESIDUE 0:
onboarded-code 32,564,876 (7.83%) · UNCLAIMED-CODE 1,700,049 (0.41%) ·
classified-data 150,631,480 · audio-video 184,338,000 · filesystem-metadata 46,787,355
34 UNCLAIMED code payloads — largest a 383,783 B type-1 in MAIN.CD, the rest small type-1 entries.
These are the "there was more code all along" surprises, now ENUMERATED instead of stumbled into.
- MY OWN FIRST RUN FAILED THE PARTITION by -49,709,520 B, and the fail-closed exit is what caught it:
.DA entries' LBAs point PAST track 1 into the CD-DA tracks (double-counted against the whole-track
audio total), and .STR/.XA are MODE2 FORM2 (2324 user bytes/sector, not 2048). Both fixed.
- NOT wired into tools-health: it needs disks/, which a fresh clone does not have (H1).
- KNOWN GAP, stated not hidden: LIST.CD fails the TOC walk (it IS the TOC cache, not a container)
and is booked as data — correct today, worth a real classifier when L2 lands.
|
||
|
Drew T
|
a0e499d8f1 |
feat(phase-30 S39): make audit-frontier — the reconciliation gate (Drew's MASTER_REMAINING, derived form)
Drew asked whether we should build a master list of all funcs, a banked list, and a MASTER_REMAINING = total - banked that we hand-edit on every bank. Assessment in docs/decision-log.md (2026-08-04): ADOPT THE GOAL, REJECT THE MECHANISM. The triple already exists and is DERIVED, not maintained: total = .run/sig.*.jsonl (sig_image over the ORIGINAL bytes, independent of splat) banked = sig - stubs (INCLUDE_ASM pastes the original asm => not-wrapped == byte-exact) remaining = corpus.stubs() (filesystem-derived, coverage-asserted) and "remove it when we bank it" already happens -- banking IS deleting the INCLUDE_ASM line. A hand-maintained file would drift SILENTLY and flatteringly, which is the exact failure R33 exists for (fuel_manifest recorded 130 live stubs when the truth was 30, hiding 91.6% of remaining gain). What was genuinely missing is CROSS-ASSERTION. Six artifacts answer "what's left" -- corpus.stubs, worklist, backlog, family_hseq, fuel_manifest, progress.fleet -- each individually derived, none ever compared to the others. That is what cost P30 T0 a hand-reconciliation (family_hseq 29,961 vs progress.py 28,296). R34: not a better assertion inside one oracle, but a second one that can argue. tools/audit_frontier.py takes corpus.stubs as the reference and checks every other view against it: rows/targets naming an already-banked function, and any view whose PUBLISHED count disagrees with a recount. On first run it immediately caught a real one: family_hseq publishes 11,456 unmatched instances; only 11,297 of its members are still open per the corpus (delta +159) -- the map predates tonight's 159 banks. Ranking work off it would have mis-scoped by that much. It also PRINTS ITS OWN SCOPE LIMIT, deliberately: agreement here does NOT mean the denominator is complete. Every view, and the byte-gate itself, is blind to never-onboarded code -- the 39 type-1 modules and main's missing independent boundary oracle stay open (R34/R36). DELIBERATELY NOT wired into tools-health (Drew said "dont do this now" about the master list; this is the additive half). --strict exits 1 for when he wants it binding; wiring is one line. |
||
|
Drew T
|
1576570271 |
fix(phase-30 S1e): the distinct-code "regression" was a STALE DIGEST — alias lever ungated
The S38 checkpoint gated the phase's best lever ("do NOT scale the alias lever") on
distinct-code falling 89.3 -> 89.2. It never fell.
PROOF (each commit's metric recomputed from its OWN committed tree, 0 unresolved):
commit:1426 TRUE : instr 12394533 distinct 5022306 (77895 uniq)
commit:1426 COMMITTED: instr 12402412 distinct 5029324 (78025 uniq) <- stale
HEAD TRUE == COMMITTED: instr 12405402 distinct 5025082 (77952 uniq)
=> true delta 843->HEAD: instr +10869, distinct +2776 ins / +57 uniq. ALL ROSE.
The 843 digest was generated from a working tree still holding work REVERTED before the
commit landed (+7,879 ins / +130 uniq overstated) and never regenerated, so the next
HONEST digest read as a fall. => THE ALIAS LEVER IS UNGATED (scale it, §61 small batches).
Both recorded leads were wrong (R14): progress.py:423's SIG regex feeds fn-count ONLY
(neither weighted metric sees a C identifier — both derive matched = sig - corpus.stubs),
and "the harvest reverted functions to INCLUDE_ASM" died on one grep (483 removed, 0 added).
The 3-grep proof: identical sigs + unchanged tools/ + zero +INCLUDE_ASM => HEAD's stub set
is a strict subset => both numerators are FORBIDDEN to fall.
THREE INSTRUMENT DEFECTS, all one class (a bare except around a fail-CLOSED oracle):
- progress.py stub_addrs wrapped corpus.stubs in `except Exception: return set()`. An empty
stub set means "could not answer", not "no stubs", so matched = sig - stubs credited EVERY
function. Byte-witnessed: instr 100.00% / distinct 100.00% in a tree with no asm/. Now
propagates.
- cast_call_sites.tu_for + reconcile_tu.tu_for had the identical swallow, falling back to the
default <ov>.c instead of the jr/-O0 split TU — silently reinstating the exact bug
cast_call_sites' own docstring says it exists to fix. A wrong-TU reconcile fails the gate,
and this phase's base rate is ~24k PLUMBING vs 4,917 DIFF, so it presents as a codegen wall.
Now propagate CorpusError; ValueError fallback for curated names preserved; derived-TU path
re-verified (a _jr_ split stub resolves correctly, both tools agree).
NEW GATE (R34 — the byte-gate is a null oracle for DOCUMENTS; check-all stays 140/140 over a
stale digest forever): tools/audit_digest.py + `make audit-digest`, wired into tools-health
after report. Recomputes the three headline metrics from the current tree and fails if the
committed digest disagrees. Compares INTEGERS, not percentages — the +7,879-instruction
staleness printed as "94.4%" on both sides. Negative-control-proven against the stale 843
digest (fails, exit 1) and green on HEAD.
Verified: make report exit 0 (dedup-check 1910 validated / 0 failed, C1 coverage
241216/241216); audit-digest OK; cookbook-index OK (398 sections); metrics unchanged by the
fix (94.40% / 89.18%). No src/ or config/ edits — no bytes touched, nothing banked.
cookbook §140 · decision-log 2026-08-04 · SETUP.md inventory (R21) · R14/R32/R34/R35.
|
||
|
Drew T
|
b9efe66f91 |
fix(phase-30): the JR-PAIR "wall" was TWO instrument defects — pair banked, class retired
S28 ledgered `JR-PAIR-IN-ONE-O0-OBJECT` (two jr fns matched in one -O0 object => a clean build that cannot link: `undefined reference to $L105` + `func_8013C938`) with §81 step 1 (isolate one into its own code subseg) as the untested escape. BOTH the class and the escape are REFUTED — no isolation, no compiler wall, both fns banked from a genuinely clean fleet. The 4th consecutive "structural wall" to resolve to our own tooling (§124/§125/§126/§131). - DEFECT 1 (tools/jtbl_carve.py): ov_SC01_077_o0's carve at 0xb01a4 predates the §8e `tables=` persistence and is a MERGED DOUBLE (func_8013C0F8 $L75 + func_8013C414 $L105); the 2nd owner is MATCHED so extract pruned the stub .s naming its table. The single-table- predecessor inference derived 3 starts where the object emits 4 tables -> JTBL_PADS 0,4,4 -> jtbl_rodata_pads refused mid-stream, correctly. FIX: R32 coverage assertion + payload recovery at the single choke point (spec_from_starts) — every zero word inside a span is an original `.align 3` pad (the tool's own axiom), so the word after it STARTS a table; recovered starts are logged. No-op where structure is known (the 134 sibling _o0c spans carry tables=+0x0,+0x70). Honest limit: tight (0-pad) boundaries stay unrecoverable but fail LOUD via the filter's count guard — never silent. - DEFECT 2 (Makefile): no .DELETE_ON_ERROR, so `as` (a pipeline consumer) left a TRUNCATED .o on disk — 12 of 16 T func_, undefined $L57/$L59/$L63/$L75/$L76 — newer than its .c, and the NEXT build linked the corpse. That IS the S28 link error, one build downstream of a loud, correct compile error. Negative-control-proven on a scratch invocation. - BANKED: func_8013B83C (272 ins) + func_8013BD74 (198 ins) in ov_SC01_077 (d19c9580). Byte proof: 4 tables 0x801D8254/828C/82FC/836C (13/27/27/27 entries, each zero-pad separated); span 0xb00fc..0xb0280 = 388 B = 52+4+108+4+108+4+108 exactly; spec 0,4,4,4. - R22 clean-fleet (make clean + extract-all + check-all): 140 passed, 0 failed of 140. The incremental result was NOT trusted (§130). Fleet 93.25% fn-count / 89.2% instr / 80.5% distinct; dedup 1905/0; 0 NON_MATCHING (G4). - cookbook §132 + index (356 sections): the mechanism, the fingerprint (an undefined $L<n> in a LINK error is a truncated object, never codegen), the 30-second standalone-TU ladder that named the 4th table owner before any build, and the transferable rule — a fail-loud guard is only as trustworthy as the artifact hygiene around it. |
||
|
Drew T
|
05f6293067 |
fix(phase-30): 137 tracked C sources held a raw NUL that made grep SILENTLY SKIP them
FOUND BY ACCIDENT, WHICH IS THE POINT. `grep -rn func_8013C08C src/` returned NOTHING for a function that is defined right there. The file held a RAW NUL byte inside a character literal — the source read `== '<NUL>'` where it should read `== '\0'`. It COMPILES (the fleet was byte-identical), so no byte-gate ever objected. But file(1) classifies such a file as `data`, and **grep treats a file containing NUL as BINARY and reports nothing, silently**. The whole file therefore vanished from every grep-based audit and every hand search. I burned real time chasing a phantom missing function before `file` gave it away. SCOPE, measured: 137 files — every `_o0c`/`_o0e` region created in THIS session. The templated bodies carried the NUL fleet-wide, so I propagated the defect today. All fixed (`'<NUL>'` -> `'\0'`); R22 CLEAN-FLEET 140 passed, 0 failed of 140 => byte-neutral. NEW ORACLE: tools/audit_text_sources.py + `make audit-text-sources`, wired into tools-health, coverage-asserting over all 3,887 tracked .c/.h files (R32). This is the SAME silent-skip family as SS124 (a scanner that cannot see something reports it is not there) and SS126a (a bare except swallowing a coverage assertion) — but one layer LOWER, in the tool everyone reaches for first. The byte-gate is structurally blind to it (R34): the bytes are correct, so it has nothing to say. It needs its own oracle. MY OWN ERROR, RECORDED: proving the new guard fires, I injected a NUL into the REAL tracked file and restored it through nested shell escaping. The restore left `'\\0'` — an escaped backslash, i.e. a multi-character constant, NOT a NUL — a genuine semantic change. R22 caught it (139/140) in one cycle, before any commit; repaired to `'\0'` (10465 -> 10466 bytes) and re-verified 140/140. The lesson is not "be careful": a negative control must corrupt a SCRATCH COPY under .run/, never the tracked file it is testing. Testing a guard must not risk introducing the defect the guard exists to catch. |
||
|
Drew T
|
45d26cc413 |
feat(phase-30): T2 carve-within-a-carve PROVEN end-to-end; Arm-A does not bite; 3 -O0 fns banked
T2's central unknown is resolved, and it is NOT what the phase plan predicted. The plan
named the Arm-A splat `%lo +0x20` defect as T2's real substance. Four probes, each
isolating ONE variable, SHA vs config/check from a clean tree (SS125 rules):
1. sub-split the jr object at ARBITRARY addresses, pure -O2 -> BYTE-IDENTICAL
** the re-carve is NEUTRAL; Arm-A does not bite here **
2. same split, middle region routed to -O0 -> DIVERGED (2 vars at once)
3. same NAME as probe 1, only the -O0 flag added -> DIVERGED
** therefore the FLAG, not the subseg name **
4. -O0 regions cut to EXCLUDE the matched bodies -> BYTE-IDENTICAL
** route PROVEN end-to-end **
THE REAL OBSTACLE: an address range is not an optimization region. Interleaved among the
15 -O0 stubs at 0x80183CF0..0x80184920 are TWO already-MATCHED functions (func_80184440,
func_801848E4) that expand from engine_core.h and compile at -O2. Flipping the FILE to -O0
recompiles them too. Cut around them and it is byte-clean.
MY EARLIER "15 contiguous -O0 fns, clean cut" WAS AN UNDER-COUNT (R14 on myself): I derived
it by scanning asm/**/*.s for the frame-pointer prologue, and a MATCHED function emits no
.s -- so the scan was structurally blind to exactly the bodies that break the flip. Same
shape as SS124. Any T2 driver must derive -O0 bounds as (address range MINUS already-matched
bodies), never from an asm scan.
BANKED (3, whole-binary gate the sole arbiter): func_801846E4 + siblings func_8018473C /
func_80184794. Each global DERIVED FROM THE ASM (%hi/%lo operands), not taken from the
draft's comment; 3/3 match_one --o0 MATCH (22 ins) with a -O2 control showing the mismatch;
3/3 verified through harvest_verify; bank truth read from the SOURCE (SS55b trap 4).
MAKEFILE: the -O0 glob widened `ov_*_o0b.c` -> `ov_*_o0?.c` so ANY lettered -O0 sub-split is
covered by one rule. A MISSED rule is silent -- the region would compile -O2 and every
residual it produced would be a pure artifact (SS116). corpus.o0_sources() re-verified: 137
sources, resolves `?` via glob, both pre-existing rules intact.
CONFIG: ov_SC03_014_jr_8017EB7C sub-split into 5 regions (pre / _o0c / matched-O2 /
_o0d / post), reusing jr_isolate_all's plan+build_new_config+validation verbatim so the
carve-repoint and source-repartition semantics are the proven ones.
R22 CLEAN-FLEET: extract-all 139/139 (+main) ; check-all 140 passed, 0 failed of 140.
|
||
|
Drew T
|
4570a5854b |
feat(phase-30): cookbook-index — a SYMPTOM-keyed index (226 sections, derived + coverage-asserted, in tools-health)
Wave-1 measured the tax: three agents each reported a 'NEW idiom' that was ALREADY documented — the asm-label alias (line ~2516, same 'address-of perturbs regalloc' mechanism) and the void->s32 non-neutrality (§41d, Phase 26; the agents cited the very entry §41d corrects). They consulted the cookbook as instructed and could not FIND them. 716 KB / 226 sections with no index = a discoverability failure, and every wave re-paying for prior waves' findings is the inverse of R16. docs/cookbook-index.md maps SYMPTOM (what you see in the diff) -> sections, 14 buckets, a section listed under every symptom it addresses. Derived by tools/cookbook_index.py (R33 — cannot drift), --check wired into tools-health. R32 on my own tool: the first regex required an em-dash separator and silently dropped 50 sections — including §1 (idiom catalog), §2, §5a (cross-jump, cited by an agent today). An index missing its most-cited entries turns 'I could not find it' into 'it is not there'. Now asserts extracted == candidate '§' headers and hard-exits on a gap. |
||
|
Drew T
|
ef85803b1f |
feat(phase-29): backlog prune — compact the append-only near-miss log + wire into make report
The near-miss ledger (.run/backlog.jsonl) is append-only, so it filled with already-banked noise: 6,867 rows, ~98% banked. load_best()/render() already filtered on READ (docs/backlog.md was correct), but the raw log drifted stale and every render re-scanned all 6,867 rows against the stub oracle. - backlog.py: new `prune` subcommand — atomic rewrite (temp + os.replace) to load_best()'s output (drop-now-matched P9 + best-per-addr collapse). Idempotent. 6,867 -> 1,704 open near-misses. - Makefile: `backlog.py prune` wired into `make report` (BINARY=main block) so the ledger tracks reality every cycle instead of drifting. - Finding (Drew's question): crack waves DO log every non-byte-match to the backlog durably (gate_stage copies best_draft -> .run/backlog_drafts/). BUT the `closeness` field is UNRELIABLE — byte-correct drafts (match_one MATCH) are logged with closeness>0 (e.g. func_8012F49C logged 29, actually MATCH). And a reach-N function's draft is overlay-SPECIFIC (per-location symbols), so the backlog is a messy recovery source vs the fresh per-wave stranded drafts. Integration-recovery should consume the fresh wave-dir strandeds, not re-derive from the backlog. |
||
|
Drew T
|
5e78dccf07 |
feat(phase-29 §8e): jtbl 8-align pad-spec filter — the 4-giant unblock tooling (fleet-neutral)
- ROOT CAUSE PINNED (the session-2 half-pin was INVERTED; both probes were vacuous, R35): cc1 emits .align 3 before EVERY jump table; maspsx passes it VERBATIM (the :435 'drop' is an inventory-only pass); as bakes the pad SECTION-RELATIVE; link placement was never guilty (SUBALIGN(2) + ALIGN(.,4) place 4-mod-8 carve starts tight). Merging originally-separate TUs fires an intra-TU align where the original packed tight -> +4 at rodata 0xCC -> image-wide %lo shift. Honest probes persisted: .run/probe_jtbl/ (verdict.md + objdumps). - NEW tools/jtbl_rodata_pads.py: post-maspsx filter replaces each rodata .align 3 with the ORIGINAL's exact pad bytes per a JTBL_PADS spec; fail-loud on table-count drift / non-align-3 / non-jtbl rodata content. Byte-proven: verbatim 0xE4 pad-at-0xCC -> filtered 0xE0 tight (= the merged carve span). - jtbl_carve.py: spec-aware same-subseg merge (gap 0 or 4-with-zero-payload-word; else NON-CONTIGUOUS -> isolate), interval-arithmetic pad specs (committed values CARRIED, never re-derived), JTBL_PADS target-var emission into overlays.mk + revert() restore + stale-.o invalidation; the false 'maspsx drops .align' docstring corrected (H5). - Makefile: $(if $(JTBL_PADS),| jtbl_rodata_pads.py ...) stage in build/src/%.o + file-scope empty default (env-shield). jtbl_family_bank.stub_file: duplicate-stub fail-loud (the earlier 'ladder failure' was a wrong-TU splice into a stale _a.c stub, byte-witnessed). - R22 clean-fleet WITH the fix wired: 140/140 byte-identical, tools-health green (dedup 1846/0, C1 234205/234205), ZERO new banks -- fleet-neutral by construction. - cookbook §8e (the jtbl alignment law) + §8a/§8a-pad corrections; decision-log R31 entry; SETUP.md tool row; .gitignore allowlist for the probe verdict artifacts. |
||
|
Drew T
|
f6f89781ff |
feat(phase-29 T2 Arm A): swing verdict = BANKED FACT (9/9 -O0 members on ov_SC07_010); fleet -O0 rollout deferred at the splat wall
- tools/rollout_o0_cluster.py (new) + Makefile O0_CLUSTER_OBJS -O0 wildcard: the -O0-cluster carve (0x13410..0x14834), adapting rollout_whale_o0.py to a 3-way <ov>/<ov>_o0/<ov>_o2b split - ov_SC07_010: carve byte-neutral -> family_sweep --hseq banked 9/9 -O0 exemplar-family members whole-binary (R22 clean-fleet 140/140). The Task-1 masked-MATCH swing verdict is now a BANKED FACT: -O0 cluster members DO bank at -O0 (§52b). Phase-20 'func_8013B7AC overlay-local' refuted. - THE WALL (byte-proven, TOOLING not compiler): the same carve on 006/007/011 byte-shifts the whole image (+0x20 %lo data-symbol shift, 34% diff) from a CLEAN build; boundaries verified as real fn-starts. Root cause = splat re-disassembly of a 3-way-split subseg that still holds INCLUDE_ASM stubs (the whale's stub-free _o0b shape avoids it). The Phase-20 '-O0 split infra' wall, root-caused. - DEFERRED (ROI): full -O0 fleet rollout (~1,233 / ~0.6pp) — 3/4 sampled walled + 134 jr-embedded + bigger levers (Task 3 core-cracks, Task 6 tiny-IMM ~5,566). decision-log R31 + cookbook §18-P29. - 140/140 byte-identical; dedup 1840/0; 0 NON_MATCHING (G4); main 143dbb89. Task 2 substantively done. |
||
|
Drew T
|
758c4365b6 |
feat(phase-28 T7): make audit-binaries — the R36 citizenship gate (R32 enforcement)
R36: a newly-discovered binary is not real until every consumer knows it. Onboarding produces a
byte-CLEAN binary (check-all green) that is not yet a CITIZEN — the tools that enumerate binaries
can each be silently unaware of it, and the byte-gate is structurally blind to that (R34).
Not hypothetical: P27 onboarded 4 SC07 overlays byte-clean; P28 found FOUR consumers silently
ignoring them (family_remap.img_path, .run/family_hseq.json, config/dedup.us.yaml, and the overlays'
own .c), hiding ~6,400 already-matched bodies. Every failure was silent.
- tools/audit_binaries.py asserts, coverage-checked BOTH directions (R32), against the config the
BUILD reads (R33 — onboarded = main + resident + every config/splat.ov_*.yaml):
1. dup_report.BINARIES (what corpus/family_hseq/progress all derive from) EXACTLY equals the
onboarded set — a missing binary is invisible to every derived tool; a phantom is invented.
2. every onboarded binary has a byte-derived sig.
3. THE LOAD-BEARING SC07 CHECK: every onboarded OVERLAY's .c includes ../shared/engine_core.h,
or no shared body can ever reach it (main/resident have their own bodies, exempt).
4. every onboarded overlay is represented in the family map (warn — regenerable/may post-date).
INFO: dedup-group membership (0 = onboarded-but-un-harvested, a dedup_extend candidate).
- NEGATIVE CONTROL: stripping the shared include from ov_SC07_006.c makes the gate FAIL loudly and
exit 1 — it catches the exact bug that hid 6,400 bodies for a month. Restored clean.
- Wired into `make tools-health` (the pre-matching ritual) — cheap (config + text scans, no build),
so it sits in the fast lane. Passes today: 140 onboarded, all full citizens.
- Reads config/dedup.us.yaml as TEXT (never a YAML round-trip — the H5 lesson from T4).
|
||
|
Drew T
|
427baba3bf |
feat(phase-27 T10): completion dashboard (main in the weighted metric) + the resident second oracle
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
|
||
|
Drew T
|
ebdef9012b |
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired. |
||
|
Drew T
|
d7d2613ae6 |
fix(phase-26a): A9c — lint_symbol_refs green + wired into make report
The ONLY detector for the R22 rename-drift failure mode (a symbols.us.txt rename leaves a func_<ADDR> ref dangling in committed src; a clean rebuild fails, an incremental build masks it with a stale .o — undetected Phase 21→23). It was RED (262 false positives) and UNWIRED. The audit's 3 blind spots, all fixed: #1 globbed src/**/*.c only -> now ALSO scans src/shared/*.h, where engine_core.h's 10k+ func_/D_ tokens live and one dangling ref breaks EVERY clean build at once (negative-control-proven: an injected bare ref in a shared header IS flagged). #2 read 2 of 138 symbol files -> now reads every REAL stacked file (config/symbols*.txt), and correctly EXCLUDES the R13 proto files (never stacked into a build; would invent phantoms). #3 no __asm__("label") model -> ALL 262 false positives were this class: engine_core.h binds func_8005C324 to memcpy via `... __asm__("memcpy")`, so the ref emits the LABEL and resolves at link regardless of the rename. asm_labeled_addrs() now recognizes it. VERIFIED: green on HEAD (exit 0); negative control (inject a bare func_800d1bd8 -> DsMix in a shared header) FLAGS it (exit 1) then clears on removal — detection intact, not trivially green. Wired as a fail-closed gate in `make report` (make -n confirms), beside dedup_integrate --check. No src/build change (report-time integrity gate) -> the byte-identical build is unaffected. |
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|
||
|
Drew T
|
af2f40d153 |
fix(phase-26a): A4/A5 — 193 unmatchable slices dissolved; the closeness oracle stops lying
R22 CLEAN-FLEET: make clean -> extract 136 -> build 136 -> check-all = 136 PASSED, 0 FAILED.
make audit-corpus: 0 PHANTOM + 0 TRUNCATED (was 193).
=== A4: a CORPUS defect the byte-gate could never have caught ===
config/symbols.us.txt:981 declared `listCdBuffer = 0x80180000` — a correct Phase-3 name for MAIN's
LIST.CD RAM buffer. But that address is OUTSIDE main's image and INSIDE the overlay slot, and every
overlay's splat config stacks symbols.us.txt. High RAM is REUSED: an address that is a buffer to main
is live CODE to an overlay. So splat saw a symbol boundary mid-code and, across 97 of 134 overlays:
* CUT 97 REAL FUNCTIONS IN HALF (a head ending on a `lui`, no return), and
* INVENTED 96 PHANTOM ONES (a tail beginning by reading the assembler temp $at).
193 slices NOBODY COULD EVER MATCH — not "hard", not "a compiler wall": unmatchable by construction.
They sat in the harvest queue as ordinary work, so agents would burn on them forever and the failures
would be filed as intrinsic compiler residuals.
The phantom listCdBuffer.s in ov_SC01_005 literally begins:
lw $ra, 0x10($sp) / addiu $sp, $sp, 0x18 / jr $ra
splat cut a function immediately before its EPILOGUE and called the epilogue a function.
AND IT HAD ALREADY CONTAMINATED REAL WORK: in ov_SC03_031 the cut landed where the epilogue was
exactly `jr $ra; nop`, so the Phase-26 x134 sweep innocently BANKED the phantom as
`void listCdBuffer(void) {}` — byte-correct, gate-green, entirely fictitious — while leaving
func_8017FFC4 permanently unmatchable. Removed.
WHY NO GATE CAUGHT IT, AND WHY THAT IS THE POINT: INCLUDE_ASM pastes the two .s halves back VERBATIM
in original order, so the image is byte-identical either way. The byte-gate was green the whole time
and always would have been. It is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle. No
assertion added INSIDE it could ever have found this. What found it was a SECOND, INDEPENDENT oracle:
tools/sig_image.py derives boundaries from the ORIGINAL bytes without splat, and DISAGREED with the
corpus (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements).
=> When one oracle is structurally blind to a class of error, the answer is not a better assertion
inside it. It is a SECOND ORACLE THAT CAN DISAGREE WITH IT. (`make audit-corpus` is now that.)
THE RULE (the mirror of R13/R15, never written down): a symbol whose address falls inside ANOTHER
binary's vram window must never enter that binary's symbol stack.
FIX: config/symbols.us.ram.txt — main-scoped symbols outside main's image — stacked ONLY by
config/splat.us.exe.yaml. Main keeps the name it needs (10 %hi / 11 %lo refs; 143dbb89 byte-identical);
the overlays never see it. Exactly one symbol was in scope fleet-wide; the resident window was clean.
AND A REAL FUNCTION THE ACCIDENT WAS HIDING: in ov_SC01_084 / ov_SC02_041 / ov_SC03_094 / ov_SC06_008
there IS a genuine function at 0x80180000 (111 / 35 / 28 / 74 ins), reachable ONLY via a fn-pointer
table (.word func_80180000) and never by `jal` — so splat cannot find it and needs the boundary
DECLARED. listCdBuffer had been supplying it by luck. Now declared honestly, per-overlay, in
config/symbols.<ov>.txt — exactly where R13/R15 says an overlay-scoped symbol belongs.
=== A5: the closeness oracle every crack agent trusts was lying on 155 functions ===
masked_diff._reloc_kind() knew 26/HI16/LO16. An over-approximating sweep of every reloc objdump emits
across all 3,367 build objects found FOUR: R_MIPS_26, HI16, LO16 — and R_MIPS_PC16 (211). PC16 fell
through to a FULL-WORD compare, but the object holds an UNRESOLVED PLACEHOLDER in the branch
displacement, so that compare can NEVER succeed.
DECISIVE TEST (derived from the invariant, not from reading the regex): INCLUDE_ASM pastes the
ORIGINAL asm, so for every stub diff_object_s() MUST be 0. Measured, coverage-asserted:
2,741 functions scored — old mask: 150 LIES; PC16 masked: 4 LIES.
(The 4 survivors are the separate length-delta defect.) A phantom non-zero sends an agent to grind at
a wall that is not there, and the wasted attempt is then booked as a MATCHING failure, feeding
reserved_walls() and PERMANENTLY BLACKLISTING a function that was never broken.
=== NEW FINDING (found by cutting the R22 corner): a STALE OBJECT CAN PRODUCE A FALSE PASS ===
`.o <- .s` is not a dependency make can see: assembly arrives via INCLUDE_ASM, expanded to a `.include`
consumed by maspsx/as AFTER cpp, while -MMD tracks headers only. Re-extract, build incrementally, and
make links a STALE object. This is not merely slow — INCLUDE_ASM pastes the ORIGINAL bytes, so a stale
object still yields the original image: SHA1 GOES GREEN while the split just changed is never exercised.
A broken config change can be "verified" by an incremental build. Live proof: 8 of 136 binaries linked
stale objects here; they failed LOUDLY ONLY BY LUCK (the dead symbol was an undefined reference) — a
merely-different-but-valid split would have gone green on all 136.
R22/H3 already legislate this, and I broke them. But a rule that needs a human to remember it is not a
gate. FIX: `extract` now invalidates the objects that include what it just rewrote (main's are top-level,
so -maxdepth 1 — verified it cannot clobber the other 1,605 objects). Structural, not advisory.
R14 self-catch, recorded: my first A5 test passed `fn=` to diff_object_s(), which takes two args; the
TypeError was swallowed by my own `except Exception: continue` and it reported 0 scored / 0 lies. I
wrote the exact bug I was auditing, inside the test for it. Caught only because 0 looked wrong. The
test now asserts its own coverage.
|
||
|
Drew T
|
f7b7399ebe |
feat(phase-26a): A3 — tools/corpus.py, ONE derived corpus oracle (+ a second oracle that can disagree)
The 28 surviving audit findings collapse to ONE bug repeated ~10 times: a hand-maintained model of
the corpus layout (a file allowlist, a single-.c assumption, a func_-only symbol regex, a REGION_SUB
dict) sitting on top of a filesystem that already answers the question. The fix is not ten repaired
regexes — it is one DERIVED oracle and ten deleted scanners (R33).
WHAT IT DERIVES FROM
1. THE FILESYSTEM. Which .c files make up a binary, and where a function's .s lives, are FACTS OF
THE TREE THAT SPLAT ITSELF WROTE. The INCLUDE_ASM line is SELF-DESCRIBING — its first argument
IS the asm subdir — so there is nothing to guess and no dict to rot. A dict literal is strictly
worse than the filesystem AND it fails OPEN (silently yields a wrong path) instead of closed.
2. THE PROVEN INVARIANT. INCLUDE_ASM pastes the ORIGINAL asm and the build is byte-identical, so a
function NOT wrapped in it is byte-exact. `matched` is DERIVED as sig - stubs, never re-parsed
from C text. (progress.py learned this the hard way: weighted_metrics() derived and was right;
classify() re-parsed C and inherited a bug.)
VALIDATED against the real corpus:
* ov_SC01_077: 264 stubs across 14 files. The old 3-file allowlist saw 30.
* Fleet: 58,717 stubs vs the allowlist's 1,992 — 56,725 (96.6%) were INVISIBLE.
* Coverage-asserted (R32): every INCLUDE_ASM line must parse, every symbol must resolve (ANY C
identifier — a func_-only regex silently misses the 100 curated listCdBuffer stubs), every stub
must have a .s. A silent skip is a DEFECT, not a no-op.
THE SECOND ORACLE (`make audit-corpus`) — the real lesson of this audit.
The byte-gate is structurally BLIND to a bad function boundary: the .s halves are pasted back
verbatim in original order, so the image stays byte-identical and green. Only an oracle that can
DISAGREE can see it. sig_image is that oracle — Ghidra-free, derived from the ORIGINAL bytes,
independent of splat. corpus.audit() cross-checks the two and reports:
PHANTOM — a stub address the sig does not know: splat INVENTED a function.
TRUNCATED — a stub whose .s length != the sig's: splat MIS-SLICED one.
It reports 193 (96 + 97) — reproducing the A2 audit's number EXACTLY, from an independently written
tool. That is a third confirmation of the listCdBuffer defect (auditor -> skeptic -> this).
AND AN R14 SELF-CATCH, recorded because the near-miss is the lesson.
Run naively over all 136 binaries the same check reports 914 slices — 4.7x the truth. It is noise:
main/resident are signed by the GHIDRA dumper, whose boundaries are shorter than splat's by design
(and which never analysed the linked PsyQ subsegs at all), so the comparison measures GHIDRA'S limits,
not splat's errors. Only the overlays are signed by sig_image, the oracle actually validated at
58,524/58,621. sig_is_independent() now encodes that domain, with the reasoning, so nobody repeats it.
A check applied outside its valid domain does not become more thorough — it becomes noise.
`make audit-corpus` is RED by design until A4 removes the bad symbol line; then it becomes a gate.
|
||
|
Drew T
|
6fb8d323bd |
perf(phase-26): parallelize R22 clean-fleet — ~9min -> ~50s (10.5x), 136/136 byte-identical
- profiled the clean-fleet R22: extract-all ~6m11s (136 serial `splat split`) + check-all ~2m58s (136 serial builds) = ~9 min, all serial on a 32-core box - the only shared WRITE is the 4 generated include/*.inc macros at extract time (identical content per binary); everything else is per-binary-disjoint and include/ is read-only during a build -> concurrent builds/extracts don't race - Makefile: JOBS ?= 16 + `make extract-all` (seed main serial for the macros, then parallel- extract the rest via xargs -P) + parallel `make check-all` (xargs -P), correctness-gated - MEASURED: `make clean && make extract-all && make check-all` = ~50s, check-all 136/136 BYTE-IDENTICAL (== the serial result) -> ~10.5x. Compounds across the endgame (R22 per commit) - new R22 recipe: `make clean && make extract-all && make check-all` (was the serial for-loop) |
||
|
Drew T
|
095a611e75 |
feat(phase-26): §8 jtbl-rodata tooling — overlay PoC proven (func_8012ACE0, R22 136/136)
- overlay jr-functions can now bank as C: gcc switch jump tables form a .rodata island at the overlay TAIL; carve a matched fn's jtbl into a dotted [.rodata, <code-subseg>] subseg + ld_interleave (data->rodata->data sandwich) places it byte-exact. cookbook §8a + SETUP. - tools/ld_interleave.py: --section .<binary> param (derives the <binary>_TEXT/DATA/RODATA/ DATA2/BSS symbol prefix); default .main = the EXE, byte-identical (backward-compat proven) - Makefile + config/overlays.mk: <bin>_JTBL_INTERLEAVE hook + a $(strip)-guarded extract branch (gotcha caught: a trailing #comment on the := left whitespace -> non-empty -> the branch misfired on resident with the EXE defaults) - PoC: func_8012ACE0 (25-ins jr-fn in ov_SC01_077) reconciled (canon_sig_reconcile) + banked BYTE-IDENTICAL d19c9580 -- the first overlay jr-function matched through the C pipeline - R22 FULL-FLEET clean rebuild: 136 passed, 0 failed (main 143dbb89 unaffected by the ld_interleave change); 0 NON_MATCHING in any default build (G4) - P9 findings: func_80159C84/func_8015444C (the 2 carried Fable5 jr bodies) are rtu_match FALSE-matches (incomplete jtbls: 52B vs 56B -> never bank); the maspsx "hang" scare was a truncated experimental-file artifact (real pipeline builds in ~1s) - metrics: distinct-code 39.1% (50,572 unique fns), instr-weighted 62.9% - NEXT: the ×134 automation (generate the per-overlay carve + template the reconciled body) |
||
|
Drew T
|
8cfbbf2d47 |
feat(phase-24): T7 whale func_80144B9C banked ×134 (-O0 reach-134 rollout) — clean fleet 136/136
The single biggest byte-weight lever (770 ins ×134 ~ +1.6% byte-weight), banked across all 134 overlays as -O0 compiled C (was ×1 in ov_SC01_077, commit:0463). - ROLLOUT (tools/rollout_whale_o0.py): per single-file overlay, LINE-split <ov>.c at the whale (splat emits in vram order), carve the yaml code subseg into before/<ov>_o0b(-O0)/<ov>_after, thin <ov>_o0b.c #includes the shared src/shared/func_80144B9C.h (the whale C moved there — DRY, TU-local typedefs, not a fragile 200-line DEFINE_ macro). Makefile WHALE_O0B_OBJS wildcard -O0. - DEDUP: group E_func_80144B9C (134 members, h_exact 74186b97, source=the header; group_members keys on binary+vram so a header-share validates like a macro-share). - MILESTONE: clean fleet check-all 136/136 (R22); dedup-check 1811/0; FLEET REAL 225816->225949, byte-identical 65.91->65.95% function-count (the gain is byte-weighted). Validated on ov_SC01_000 + spot-checks SC02/SC03/SC07 before the full fleet. - cookbook §38 + SETUP inventory. |
||
|
Drew T
|
3b0cc190b2 |
feat(phase-24): T7 whale func_80144B9C CRACKED + banked ×1 (-O0 struct-assign) + ×134 infra
- CRACK (cheap Opus, no Fable5/calls.c): the 770-ins reach-134 whale's 2-insn residual was a
STRUCT ASSIGN `D_80078E50 = *ent;` (sizeof(ENT)==0x24), NOT an explicit memcpy() call — gcc-2.7.2
-O0 expands a >MOVE_RATIO-word struct copy via emit_block_move->emit_library_call(memcpy),
precomputing dst/src into pseudos then addu into $a0/$a1 = the exact 2 missing moves.
- -O0 SPLIT: whale is a 2nd -O0 region (prologue 21F0A003) in the -O2 'after' segment; carved into
its own -O0 object o0b (splat.ov_SC01_077.yaml 3-way + Makefile CC1FLAGS=-O0; address-sorted
line-split preserves all banked C). ov_SC01_077 == d19c9580.
- memcpy: memcpy=0x8005C324 in symbols.resident.txt (overlays+resident only; main untouched vs its
MEMCPY.o) + __asm__("memcpy") on the engine_core.h block-copy macro (non-builtin C name -> no
built-in codegen, same call).
- MILESTONE: CLEAN fleet check-all 136/136 byte-identical (R22), dedup 1810/0. reach-134 -> ×134 (W9) next.
|
||
|
Drew T
|
be33b887fe |
feat(phase-19): T1 -O0 split lever — infra proven + 6 -O0 fns matched (ov_SC01_077)
- config/splat.ov_SC01_077.yaml: 3-object code split (ov_SC01_077_a before / ov_SC01_077_o0 -O0 cluster / ov_SC01_077 after). A single object's .text cannot be split around a middle object, so before/after are distinct objects; the after-region keeps the ov_SC01_077 name (bulk matched C + asm paths unchanged). - Makefile: target-specific CC1FLAGS:=-O0 for ov_SC01_077_o0.o (src/boot.c precedent). - src/ov_SC01_077/ov_SC01_077_a.c (new before-region) + ov_SC01_077.c (after-region) + ov_SC01_077_o0.c (new -O0 cluster): 6/16 -O0 fns matched byte-perfect (func_8013B568/B7F4/BD34/C360/C938/C964). - DEFERRED to Phase 20 (Drew, on the discovered difficulty): the 10 remaining -O0 fns hit an indexed-global %lo-folding codegen quirk (gcc-source research, R17) + the x134 per-overlay rollout (engine_core.h compiles -O2, cannot carry -O0 fns). Corrects the Phase-18 backlog premise (afternoon/free-x134) per R14. Documented in ov_SC01_077_o0.c. - verified clean: main 143dbb89, resident 8e17e02f, ov_SC01_077 d19c9580 + 2 overlays. |
||
|
Drew T
|
c716eb15e1 |
feat(phase-15): T4 — fleet progress roll-up + fix cross-report double-count
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index (fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic, source-derived per-binary table + fleet totals). Single-binary output byte-unchanged. - fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82% across 136 binaries (main + resident + 134 overlays) - dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups / 28.6M collapsible (matches the Phase-11 figure) - Makefile: wire progress.py --fleet into make report (main-only block) - docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory - make report green; dedup-check 8 validated / 0 failed - ghidra/ churn NOT staged (R23) |
||
|
Drew T
|
0c7619231d |
feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X). - proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all 18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test (corrupt shared body) -> make check FAILS then restores - Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a shared header recompiles dependents — closes a stale-build/false-pass gap the negative test exposed; output-neutral (R22 clean 18/18) - structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate can't catch under-application -> assert DEFINE present + stub gone - FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest - cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory - ghidra/ churn NOT staged (R23); build/asm/.run gitignored |
||
|
Drew T
|
e2200e17db |
feat(phase-13): T5 — make build-all / check-all (fleet PASS/FAIL)
Recursive $(MAKE) BINARY=<b> over $(BINARIES) (correct per-binary OBJS prune, not foreach), aggregated to one PASS/FAIL + per-binary .run/check.<b>.log. check-all: 6/6 [OK] exit 0; negative control (corrupt sha) -> [FAIL] + nonzero. |
||
|
Drew T
|
144839f24b |
feat(phase-13): T1a/T1b — SC01/077 onboarded + all-asm byte-match (d19c9580)
- ov_SC01_077 scaffolded: config/overlays.mk (OVERLAY_BINARIES + var block), config/splat.ov_SC01_077.yaml (from template), check.sha, empty symbols, + ov_SC01_077 entry & sentinel anchor in the 4 Python BINARIES dicts - GATE T1b: make check BINARY=ov_SC01_077 -> d19c9580 BYTE-IDENTICAL @ 100% INCLUDE_ASM; clean rebuild leaves main 143dbb89 + resident 8e17e02f unregressed - reusable non-word-aligned-overlay handling (~75% of fleet) added to the template + Makefile: (1) [word_floor, bin, trailing] carve for the final 1-3 bytes spimdisasm drops; (2) build/assets/%.o incbin rule (+ .data align=1) for splat bin assets (asset_path scoped per-alias); (3) objcopy end-align TRIM (shrink-only, <=3 B) removing the .ld's segment-end ALIGN(.,4) pad - .gitignore /assets/ (regenerable splat output); clean removes assets/ |
||
|
Drew T
|
50c5c9ea33 |
feat(phase-13): T0 — overlay splat template + overlays.mk indirection
- config/splat.us.overlay.template.yaml: reusable flat-blob splat template for location overlays (@TOKEN@ placeholders), modeled on splat.resident.yaml; code at file 0x0 (no leading-rodata trick), stacks symbols.resident.txt (overlays call the engine), vram 0x80128158 - Makefile: -include config/overlays.mk + BINARIES := main resident $(OVERLAY_BINARIES); the generated overlays.mk keeps new_overlay.sh out of the hand-maintained Makefile body; silent when absent (fresh-clone-safe) - GATE T0: clean rebuild main 143dbb89 + resident 8e17e02f BYTE-IDENTICAL (indirection proven a no-op) - phase-ends/CURRENT_PHASE.md: Phase-13 task log (plan approved gate 1) |
||
|
Drew T
|
f672c709c1 |
feat(phase-11): T6 — sign the 134 overlays; cross-report spans 136 binaries (milestone report half)
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) + detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes ~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix - Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s) - dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals; compact member sample + #bin column - RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm 8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue - dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob, no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates) |
||
|
Drew T
|
b06d74c1be |
feat(phase-11): T2 — dedup.us.yaml registry + dedup_integrate byte-honesty validator
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
current sig hash drifted from the recorded hash (a stale share can never silently
mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
one object per binary so the linker can't swap them; the byte-gate is the existing
per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
-> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
|
||
|
Drew T
|
7cc2911456 |
feat(phase-11): T1 — cross-binary dup_report --cross → duplicates.cross.md
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row with its binary, buckets by h_exact then h_norm across binaries, splits cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md - Makefile: make report emits the cross report once (gated BINARY=main) - per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged) - EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the byte-finding predicted); real cross groups arrive with overlays (T6) - phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log |
||
|
Drew T
|
03978ec4ba |
feat(phase-10): T4 — resident Ghidra program (2nd binary) + symbol seed + sig
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr + PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8. - tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points (.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143); this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted. - FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7. - config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11); stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22), main still 143dbb89 (no regression) - Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl) - dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder) - ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op) |
||
|
Drew T
|
cc73951128 |
feat(phase-10): T3 — per-binary reports/diff/expected for the resident binary
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry (build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs) - dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent — write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export) - progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout (Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs) - Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings) - reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED (52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified) |
||
|
Drew T
|
95ade84768 |
feat(phase-10): T1 — scaffold the resident second binary (Makefile no-op on main)
- Makefile: BINARIES += resident; resident_* var block (vram 0x800CEDF8, build/resident/, config/*.resident.*); per-binary ASM_DIR/SRC_DIR + $(BINARIES)-derived OBJS prune-list so main (asm/+src/) and resident (asm/resident/+src/resident/) object sets stay disjoint; per-binary UNDEF_SYMS/UNDEF_FUNCS (resident writes under build/resident/) - config/splat.resident.yaml: flat blob config (NO header, NO gp_value, single c seg @ vram 0x800CEDF8, stacked symbols [symbols.us.txt, symbols.resident.txt], per-binary asm/src/build/undefined paths, EOF 0x5935C -> end vram 0x80128154) - config/check.resident.sha (8e17e02f… = extracted MAIN.CD/FILE_010/1.1, 365,404 B, type-1) - config/symbols.resident.txt (R13/R15 provenance banner; seeded empty, populated in T4) - gates: (a) main clean rebuild 143dbb89… (refactor is a no-op on the byte-locked EXE); (b) make extract BINARY=resident -> 144 INCLUDE_ASM stubs + .ld @0x800CEDF8; (c) main OBJS=81 excludes resident even with resident on disk + main stays 143dbb89… - src/resident/*.c left WIP-untracked (reshaped by T2 boundary carving) |
||
|
Drew T
|
5e0d80a887 |
refactor(phase-9): T7 — ld_interleave.py --front/--tail; gate extract call to BINARY=main
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific - Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly - GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag); negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded); restore -> 143dbb89 |
||
|
Drew T
|
22b20e8f31 |
refactor(phase-9): T6 — --binary selector for dup_report + difficulty; report threads BINARY
- dup_report.py + difficulty.py: argparse --binary (default main) + BINARIES table (main = current sig/src/asm/md/csv paths); overlay src/asm subtree deferred to P10 - Makefile report target threads --binary $(BINARY) to progress/difficulty/dup_report - no-op proof: docs/progress.md + docs/duplicates.md UNCHANGED; difficulty.py old-code vs new-code output IDENTICAL (pure path-table refactor) - docs/difficulty.md regenerated (the Phase-8-deferred regen: 9 harvested funcs left the unmatched queue — 12/12 row shift); build path untouched (still 143dbb89) |
||
|
Drew T
|
06676e6ab1 |
refactor(phase-9): T5 — psyq_integrate flags + progress regex + 9 Makefile call sites
The build-path linchpin (one commit — integrate signature and the progress.py Makefile regex must move in lockstep): - psyq_integrate.py: argparse (--vram-base/--exe/--symbols flags BEFORE the 4 positionals + nargs=* window); threads vram_base/exe into placement/classify and symbols_path into the stub-name->addr resolution (was hardcoded symbols.us.txt) - Makefile: all 9 psyq_integrate calls pass --vram-base $(main_VRAM_BASE) --exe $(main_EXE) --symbols $(main_SYMBOLS) (flags before positionals) - progress.py: stub-list regex gains (?:\s+--\S+\s+\S+)* to consume the leading flags; + a --binary selector (BINARIES table, default main = current paths) - GATES: positive build -> 143dbb89; report 52/959/7/50.24% (regex still finds the 9 stub lists); negative main_VRAM_BASE=0x8000F804 -> cae22f7e (NOT target); without-SDK fresh-clone fallback -> 143dbb89 |
||
|
Drew T
|
170586d6bf |
refactor(phase-9): T1 — data-driven BINARIES Makefile (no-op on EXE)
- BINARIES/BINARY guard + main_* var set + selected-binary aliases; the EXE's artifact paths (build/us/, config/splat.us.exe.yaml, check.us.sha) preserved verbatim so its rebuild stays a byte-exact no-op - 9 PsyQ SDK-integration blocks wrapped in ifeq(BINARY,main); SYMS/LD/OBJCOPY stay outside (run for every binary); a 2nd binary skips the SDK block - gate: make clean && extract && build -> 143dbb89 BYTE-IDENTICAL; make report 52/959/7/50.24%; BINARY=bogus errors via the $(filter) guard |
||
|
Drew T
|
11a9bdc032 |
feat(phase-8): link libapi+libcard combined 800c2 region (T7+T8) — 22 objs / +24 fns
- libapi (BIOS syscall trampolines) + libcard interleave in 0x61F38..0x62888; linked as one combined region (4 blocks apicard1..4) via tools/make_apicard_used.py (C112.o shared, byte-match dedup, 0 exclusions) - subsegs via gen_lib_subsegs.py; integrate window 0x61F38..0x62888; region verified 22/22 - rm'd src/800c2.c (region starts with a lib block) -> splat regenerates apicard1..4 + frags - dual byte-gate PASS: 143dbb89 with and without the apicard objects - LINKED 935 -> 959 (+24); REAL still 43; byte-identical 49.81% - libapi's ~22 objects in the 800c3 region DEFERRED (lowest value); SSGM + the scattered-.bss exclusions documented in docs/psyq-worklist.md LIBRARY CEILING: 8 PsyQ libraries linked byte-identical (959 LINKED fns); byte-identical coverage 20.31% (libcd+libgs only) -> 49.81%. Remaining = documented low-value deferrals. |
||
|
Drew T
|
8bbf2e8e4d |
feat(phase-8): link libspu+libsnd combined sound region (T9+T10) — 60 objs / +225 fns
- libspu and libsnd interleave in 0x3A444..0x4239C (the 800-subseg tail), so they link
as ONE combined region rather than two tangled passes
- NEW tools/make_snd_used.py: build the curated combined dir — merge both libs by vram,
pick the byte-matching object per aliased address (link_object), exclude 4 addresses
that don't reconcile in-region (kept as byte-identical stubs):
0x3C438 S_R/S_W, 0x3D424 S_GRMDT/FB/T — scattered-.bss commons, cross-object (§9.1)
0x3D94C S_IH/UT_RON — false placement (inside libsnd SSSTART.o)
0x3FA64 VM_F (237 ins) — scattered-.bss commons (the one real loss)
- subsegs via gen_lib_subsegs.py (9 snd blocks + 8 sgap game/excluded-stub gaps);
integrate window 0x3A444..0x4239C; region byte-verified 60/60
- src/800.c trimmed at [0x3A444,0x4239C): kept 732 items < 0x3A444 (ALL matched C +
game + the deferred SSGM stub preserved), 0 real-C moved
- dual byte-gate PASS: 143dbb89 with and without the sound objects
- LINKED 710 -> 935 (+225); REAL still 43; byte-identical 48.66% (nearly half the EXE)
- deferred: SSGM.o @0x1BD80 (isolated, matched-C region, 8 ins)
|
||
|
Drew T
|
c4a627b22b |
feat(phase-8): link libgte byte-identical (T11, value-first) — 53/58 objs / +120 fns
- NEW tools/gen_lib_subsegs.py: generate splat subseg lines + integrate stub list for a multi-block library (section-size-correct block ends — bakes in the libc2/T6 boundary gotcha so it can't recur). Reused for libspu/libsnd next. - libgte = GTE math, 53 objects in 22 blocks across the old 800b region (game code interleaved as 800b/800b_2..800b_7). Resegmented from the generator; integrate window 0x4787C..0x51804 so it sees exactly the 22 in-region blocks - 5 libgs-gap libgte objects (MTX_05/07/11/REG03/REG11) DEFERRED — gsgap1/2/4/5 stay stubs (gsgap2 != MTX_07 exactly, needs a sub-split); documented in worklist - progress.py linked_subsegs() now resolves `$(VAR)` stub lists (LIBGTE_STUBS) — the long multi-block lists are passed via a make var - region byte-verified 58/58 (clean, no scattered .bss); rm'd src/800b.c (region starts with a lib block) so splat regenerates the fragments; 22 src/libgteN.c + 6 game frags - dual byte-gate PASS: 143dbb89 with and without the libgte objects - LINKED 590 -> 710 (+120); REAL still 43; byte-identical 37.93% - reordered remaining libs value-first (libspu/libsnd next; dense libcard/libapi last) |
||
|
Drew T
|
9dbae2681e |
feat(phase-8): link libc2 byte-identical (T6) — 17 objs / +62 fns
- splat: libc2 C stdlib in 2 blocks — main 16-obj run libc2_1 (BZERO/MEMCPY/STRCMP/ PRINTF/PRNT[jtbl ok]/.../SETJMP) split from 800c, + STRCAT.o libc2_2 split from 800c2; new game-code fragments 800c3 + regenerated 800c2 - region byte-verified 17/17; PRNT.o printf-format jtbl resolves via NOLOAD .rodata - BOUNDARY GOTCHA found + fixed: SETJMP.o .text is 0x80 B (8-align pad), not the 0x78 psyq_identify reports (30 ins) -> the libc2_1/800c3 boundary was 8 B too low, overlapping SETJMP's padded tail; the relink inserted +8 padding and shifted the whole downstream image (56k diff bytes, +8 file len). Fix: boundary = last obj's readelf .text size (0x5CE18), not ins-count. Lesson recorded in docs/psyq-worklist.md - when a library block sits at a subseg start, rm the old .c so splat regenerates it - dual byte-gate PASS: 143dbb89 with and without the libc2 objects - LINKED 528 -> 590; REAL still 43; byte-identical 32.25% |
||
|
Drew T
|
0d6380c845 |
feat(phase-8): link libmcrd byte-identical (T5) — 2 objs / +104 fns
- splat: split 800c into [800c][libmcrd1][800c2][libmcrd2]; LIBMCRD.o (2186 ins,
the 55 LIBMCRD_OBJ_* + _card_* memcard I/O) and USERFUNC.o (68 ins) are 2
non-adjacent blocks split by game/libc2/libapi/libcard code (800c2)
- Makefile LIBMCRD_* + gated psyq_integrate libmcrd1,libmcrd2 + -T
- region byte-verified: 2/2 objects, 27 .bss commons all recovered (not scattered)
- src/800c.c trimmed (0 real-C moved); src/{libmcrd1,800c2,libmcrd2}.c committed
- dual byte-gate PASS: 143dbb89 with and without the libmcrd objects
- LINKED 424 -> 528; REAL still 43; byte-identical 29.29%
- NB: these are the libmcrd SDK objects; the game SaveLoadRoutine/Q#5 is Phase 12
|
||
|
Drew T
|
ade73cce62 |
feat(phase-8): link libgpu EXT+PRIM byte-identical (T4); exclude SYS.o (scattered .bss)
- splat: split 800b2 into [800b2][libgpu][800c]; libgpu block = EXT.o+PRIM.o
(vram 0x80058890-0x80059234, holds LoadClut/LoadClut2 + primitives)
- SYS.o (3109 ins) EXCLUDED — scattered-.bss commons (cookbook §9.1, the GS_001
class): SYS references .bss by section+offset but the original linker scattered
the commons across 0x80078xxx/0x800c5xxx, so no single NOLOAD base reproduces it.
Stays a byte-identical INCLUDE_ASM stub in 800c; documented in docs/psyq-worklist.md
- curated dir .run/obj40/libgpu_used = {EXT,PRIM} (validates the _used mechanism for
the later alias libraries); Makefile LIBGPU_* + gated psyq_integrate + -T
- src/800b2.c trimmed; src/libgpu.c + src/800c.c committed (stub records); the trim's
9 "moved" items were all splat-auto empties (regenerated identically, 0 real lost)
- dual byte-gate PASS: 143dbb89 with libgpu objects AND without (stub fallback)
- LINKED 375 -> 424; REAL still 43; byte-identical 24.32%
|
||
|
Drew T
|
675872616d |
feat(phase-8): link libetc byte-identical (T3) — 5 objs / 35 fns
- splat: carve libetc subseg (vram 0x8004239C-0x80043088, VSYNC/INTR/INTR_VB/ INTR_DMA/VMODE) out of the 800-subseg tail (ends exactly at libcd1); 5 objs, 1 contiguous block, no placement window needed, no matched-C disturbed - Makefile: LIBETC_ELF/OBJDIR/SYMS + gated psyq_integrate call + -T on the ld line (same conditional/stub-fallback model as libcd/libgs) - src/800.c trimmed (split_src_region: dropped 35 libetc stubs, 0 real-C moved); src/libetc.c committed (35 INCLUDE_ASM stub records) - pre-wire gate: psyq_link_lib 5/5 objects byte-identical (.text) - dual byte-gate PASS: make check 143dbb89 WITH libetc objects AND without them (stub fallback) — both byte-identical - progress.py LINKED 340 -> 375 (+35), stubs -> 1626, byte-identical 21.98% (metric auto-updated from the new Makefile psyq_integrate call) |