mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-28 06:49:47 -04:00
2cedd19aaabc3395abfeeecbdb0ba3fcec3830b4
66 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
067f25f682 |
feat(build): §332b — per-object REORDER path for the 800c2/800c3 PsyQ island
Those two objects were originally assembled in REORDER mode (the assembler filled
the delay slots). maspsx force-emits `.set noreorder`, making that unreachable, so
a whole class there read as a permanent compiler wall (§332) when the property
belongs to the OBJECT, not the toolchain.
For REORDER_TUS only, swap maspsx for tools/reorder_passthrough.py + `as -O2` --
the pipeline tools/oracle_reorder.py already proved byte-exact (0 diffs on
func_80061FA8 where the pinned path gives 57). Everything else is untouched.
Verified:
* branch selection BOTH ways: 800c3 -> reorder_passthrough, 800.c -> maspsx
* tools/reorder_passthrough.py --selftest, incl. a negative control (a line
merely CONTAINING "move", e.g. `jal remove_thing`, must not be rewritten)
* BYTE-INERT: main rebuilds BYTE-IDENTICAL via verify_binary (§384, re-extracts)
Note the first patch used `ifeq ($(filter $*,...))`, which make evaluates at PARSE
time when $* is empty -- it would have silently always taken the maspsx branch.
`$(if ...)` expands per-target, which is why the rule already uses that form for
JTBL_PADS.
|
||
|
Drew T
|
91895afd3c |
fix(tools-health): audit-cdecl is a HEALTH check, not a regression suite — sample by default
Drew, correctly: "this is a tools health test, not a full regression test." audit-cdecl re-parsed every declaration in all 4,168 TUs and handed each to real gcc — ~787s of pure-Python collection before the first cc1 call. It made `make tools-health` unrunnable: >15 min, killed twice, never completed once. `--limit` already existed and its own help calls it "a fast smoke run"; nothing was using it. Sampled by default (CDECL_AUDIT_TUS ?= 60); the exhaustive form stays as `audit-cdecl-full` for when cdecl.py itself changes. audit-cdecl : >9 min -> 61s (4,777 declarations adjudicated, 0 rejected) tools-health: never completed -> 333s, rc=0, all green Known limit, recorded not hidden: --limit takes the FIRST N TUs, not a random sample, so the smoke run always exercises the same files. Randomising the sample (or rotating by seed) is the follow-up. |
||
|
Drew T
|
26ba449684 |
perf(tools-health): parallelise the sig targets; fix a latent probe-file race; MEASURE the real cost
Drew asked why `make tools-health` runs 15+ min. Measured per step rather than
guessed (I guessed wrong twice first, and both are recorded in the comments):
sig-overlays ~52s serial -> 3.9s wall / 51.8s user (xargs -P$(JOBS), 32 cores)
sig-modules 0s sig-resident 0s audit-corpus 17s
audit-cdecl >9 MINUTES <-- the actual bottleneck, and NOT the gcc probes:
the `[gcc] N distinct declarations` line never printed inside a
10-minute run, so not one cc1 call had happened. `tu_statements`
over 4,168 TUs is ~787s single-core, all of it before the probes.
SHIPPED
* sig-overlays / sig-modules: xargs -P$(JOBS), same pattern extract-all and
check-all already use in this file. sig_image has exactly one write path
(its own per-alias .jsonl), verified before fanning out. NEGATIVE CONTROL:
141/141 sig files BYTE-IDENTICAL to the serial output. Also adds the failure
detection the serial loops never had -- a sig_image crash used to vanish (R32).
* cdecl._gcc_probe: `probe_{tag}.c` was ONE FIXED FILENAME PER TAG, correct only
while _sift is serial. Now unique per call, so concurrent probes cannot
overwrite each other's source between write and compile and return a verdict
about another chunk's declarations.
* cdecl._sift: threads over chunks + over the bisection probes (gcc is a
subprocess, so the GIL is released), results written back BY INDEX so the
output stays deterministic. A/B on --limit 6: IDENTICAL verdicts (829/829).
NOT SHIPPED, and the measurement is left in the code
A ProcessPoolExecutor over the collection phase was tried and REVERTED: 12 TUs
yield 32,352 statements, so the full pass ships ~11M strings through IPC and the
pickling costs more than the parse it saves. The fix is to dedupe/filter INSIDE
the worker or memoise per-TU by content hash -- left measured, not guessed.
|
||
|
Drew T
|
d6e28fcb1a |
feat(tools): work_evidence — assert a tool ACTUALLY DID the work it reports
make tools-health audits DATA integrity (corpus/cdecl/binaries/digest/text) and
nothing audited TOOL BEHAVIOUR -- the gap all four S70 defects fell through. Each
reported success while doing nothing or doing harm, and none would have been found
by reading the source: a wrong instrument returns a plausible NUMBER, not an error.
tools/work_evidence.py, three assertions on OBSERVABLE CONSEQUENCE:
assert_inputs zero readable inputs is a DEFECT, not a zero-yield result. "0 of 0"
is a fact about the harness; "0 of 57" is a fact about the subject.
assert_floor work claiming a compile/gate cannot beat physics -- the ONLY tell on
the pgate defect was a 1-2s wall clock (§402).
assert_effect N claimed successes must show a persistent effect; verification is
not banking (§404).
Self-test is a negative control both directions (11/11): each assertion PASSES the
already-succeeded case and FAILS the known-bad one, and non-strict warns instead of
raising. Wired into `make tools-health` so it cannot rot (R54).
Wiring on the wave critical path:
* parallel_gate: per-worker wall-clock floor; a sub-floor worker is flagged
"BLIND SUSPECT" in the summary line instead of passing as a clean zero.
* gate_stage: the silent `if not draft_fns: return {...}` -- the exact point the
pgate defect flowed through -- is now loud and marks the result `refused`.
* harvest_verify: says at the point of confusion that "verified" is not "banked"
and names gate_stage as the entrypoint that persists.
Negative control: empty drafts dir -> loud + refused. Positive control: a real
2-draft dir still gates normally (drafts:2, no false refusal).
|
||
|
Drew T
|
7a969d1c61 |
feat(o0): md_MAIN_003 carve — the module-binary -O0 route opens, func_800D0D6C banked (345 ins)
The single-object module binaries could not be carved at all: o0_subsplit planned correctly and then jr_isolate_all refused with 'unaddressable content'. That blocked 9 of the 12 remaining -O0-in-an--O2-TU functions fleet-wide, including a byte-correct 345-instruction draft with nowhere to go. THREE ROOT CAUSES behind the refusal, all fixed here: * overlay_src_split.load_ov_syms: an interior YAML comment terminated the symbol-file list. md_MAIN_003's yaml annotates the list body, so only symbols.us.txt loaded and D_800D3200 resolved to None -> refusal. * jr_isolate_all._partition: a trailing content chunk (the verbatim-asm pair after the last addressable anchor) now attaches to the LAST region when every symbol it defines resolves at/after the last cut, instead of hard-refusing. * _file_scope_decls: bare tag forward decls (struct S_D2394;) exempted from the dedupe refusal; plus addr_of's D_<hex8> fallback. THEN A LINK FAILURE THE CARVE CAUSED, worth knowing: spimdisasm migrates rodata referenced by exactly one function into that function's .s ONLY within the same subseg. The carve moved func_800D30D0 into the jr subseg while the .rodata island stayed on md_MAIN_003, so three dlabel string blocks were SILENTLY DROPPED -> undefined reference to D_800CEE58/D_800CEE80. Adding INCLUDE_RODATA does not resurrect them (splat marks them migrated segment-wide and emits nothing). The fix is to rename the .rodata subseg to the jr object, where every island emitter lives. The regenerated func_800D30D0.s came back byte-identical to the pre-carve .s. Makefile: the -O0 glob widened to src/md_*/md_*_o0?.c. Without it the region file compiles -O2 -- byte-neutral while stub-only, but every -O0 draft banked into it would mystery-fail the gate (§362's trap class). This is why the Makefile and tool hunks MUST land with the carve: a fresh clone would otherwise lose the -O0 flag. VERIFIED INDEPENDENTLY of the agent that did it: sha1 dd1b32ecf1103c6f7cf1943d25546a3046e17b14 == config/check.md_MAIN_003.sha, from a rebuild I ran myself; md_MAIN_003 13 -> 12 stubs; func_800D0D6C absent from corpus.stubs. interleave_check's DRIFT on this binary is PRE-EXISTING (identical on a clean tree, verified before any change) -- md_MAIN_003 has no _JTBL_INTERLEAVE block and must not get one; forcing ALIGNED moves the leading rodata island after .text and shifts every address by 0xD8. config/overlays.mk untouched (R59/R60). 8 of the 9 md_MAIN_003 -O0 stubs remain: they need drafts and follow-on carves. |
||
|
Drew T
|
0b75da6a36 | feat(build+tools): module jtbl pads DERIVED at build time — jtbl_rodata_pads --derive <binary> --tu <tu> (walks the retail island with the emission stream: .s spans from their comments, C data anchors from D_ names, C tables lead/trailing zeros; refuses on any anchor miss); Makefile runs it for every md_* object (no stored spec, nothing to drift); harvest_verify hands module island walls to it instead of isolate/split; the modules.mk probe line removed; md_SC03_076 byte-identical from clean (P31 S62 T3a) | ||
|
Drew T
|
05a4aeaf41 |
feat(o0): md_MAIN_011 compiles -O0 — 21 functions / 4,321 ins unlocked, byte-proven
Its entire code subseg is the -O0 run and the .c is stub-only, so this is the boot
precedent: a whole-object CC1FLAGS override, no splat change, no carve, and none of
the 18-P29 re-disassembly risk. Proven byte-neutral by a CLEAN per-binary rebuild —
build dirs deleted, re-extract, rebuild:
sha1 80731bac0ddd6b3e354f43b2c179582b12590752 == config/check.md_MAIN_011.sha
Landed with the coupling fix it requires, or the 21 would have stayed invisible.
Three tools decided -O0-ness from the subseg NAME ('_o0' in it, or 'boot'); this
object keeps its plain name, so match_one would have warned 'cannot bank' about
functions that now bank and the wave draw would have kept refusing to draw them. All
three now ask corpus.o0_subseg(), which derives the answer from the Makefile itself
(R33: a name is a convention, the Makefile is ground truth).
Verified end to end: match_one compiles md_MAIN_011 targets at -O0 with the
cannot-bank warning correctly gone, the wave draw emits cards for them, and
test_o0_detect still passes 167/167 coverage with 0 false positives.
|
||
|
Drew T
|
e18738c48e |
feat(phase-31 T5): THE FRONTIER ATLAS — 5,139 lever-labeled crack groups over all 12,058 open fns
- tools/atlas.py: cousin units baseline + T1.5 h_seqn merges + CALIBRATED warm tier (measured: li-norm metric holds ~99% recall to 0.55; rule = smallest t with neg-accept<=0.2% AND recall>=95% -> THRESH_WARM=0.70 @ 99.1%/0.18% — false merges waste exemplar cracks, misses only route cheaper) + seed sweep (65% of open skeletons carry a >=0.55 matched seed) + kNN graph + tiny-direct + evidence joins (audit/backlog/ledgers/cards; unparsable=fatal) + lever labels with confidence measured>ledger>tell>default>UNKNOWN - partition ASSERTED: 12,058 = progress stubs 12,051 + NM 7 EXACTLY (chased the +1: data blobs now excluded, reconciled against classify() buckets; T1 banks confirmed absent); every instance in exactly one group; main joins at the atlas layer only (family maps stay non-main — 4 silent-skip hazards) - warm tier merged 1,019; top group unifies 268 drifted per-location skeletons - lever table: head-crack 186.9k ins / UNKNOWN 138.6k (honest) / extend-tell 76.7k / redraft 46.8k / jtbl-carve 45.7k / integration 23.4k / seeded 23.4k / len-vein 16.8k / swaprepeat 9.2k / plumbing 8.1k / o0 6.6k / cc1 6.4k - atlas_features: li_norm_toks exported (shared with atlas, R33; hash-stable); mid_jr verifier fixed (compared ZERO rows — R32 silent no-op; now 6,444/6,444) - make atlas = full regen chain (~10-15 min, zero tokens); --targets emits crack slates (12/12 .s resolved); survey 92 s - SETUP rows (R21); docs/frontier-atlas.md committed |
||
|
Drew T
|
22eed7d78a |
feat(phase-31 T3): main enablement — sig-main at splat-true lengths + main streams
- sig_image: --seeds accepts '0xADDR NINS' (and jsonl nins); a seeded nins is
authoritative ([addr, addr+4*nins), bypasses func_end whose heuristic mis-sliced
3/40 main samples); R32 guard on seeded end > hi
- corpus: s_ins_count() factored from audit() (R33, one counter) + '--seed-ends'
CLI emitting per-stub splat-true lengths
- make sig-main: 2,002 main stubs signed -> .run/sig.main.jsonl; FULL word
cross-check 2,002/2,002 EXE slices == .s words (0 SLICE-SUSPECT; .s word field
is byte-order hex, not LE — first checker draft misread 1,999 false suspects).
Deliberately splat-SEEDED; main's independent second oracle stays deferred
(second-oracle.md; sig_is_independent(main) stays False)
- family_remap: vram_of/img_path special-case 'main' derived from splat.us.exe.yaml
(file0-vram = code-seg vram - start = 0x8000F800; target_path); stream_words
('main') verified 25/25 vs .s
- regression: sig-resident re-run byte-identical after the shared read_seeds change
- SETUP §6.3 rows (R21)
|
||
|
Drew T
|
d6ade08f3a |
feat(phase-31 T0): pivot log + instrument freshness + hygiene
- decision-log: the P31 re-charter entry (organize-before-grind; R37/R38/R39 ratified at gate-1) per R31 - harvest_verify.py: import guard — a bare import now RAISES loud instead of running a full gate (CLI unchanged, verified both directions) - sig-resident: bootstrap boundary artifacts fixed (fused +0 data word with func_800CEDFC; func_800D33E0 dropped past a glued tail) -> ELF-seeded per the S45 pattern, exactly 145 fns; true denominator confirmed 145 (progress was right); audit-corpus 0 PHANTOM + 0 TRUNCATED; all three oracles agree - family maps regenerated at HEAD commit:2161: 11,025 open non-main members reconciles EXACTLY with 12,059 - main 1,034 (102 stale phantoms cleared); adapt cards 704, aprop cards 204 (full emission) - main fuel-gap finding: 2,001/2,002 main stubs already have cached Ghidra-C (only func_80049600 missing) — the roadmap '0/2,096' note was stale - tools-health OK (dedup 2,063/0; C1 254,521/254,521; audit-digest green) |
||
|
Drew T
|
a0f07d629e |
chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/ rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java + VerifyOverlay.java (ghidra_import_raw.sh is the live path) - reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green - SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol- window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21); disc-completeness Reproduce marked retired |
||
|
Drew T
|
4cadac4e11 |
feat(phase-30 S45 II.1c): module batch dedup-banked + verified — 408 banks, R22 183/183, audit-disc 75->34 (parked-only)
- dedup measure (R37 probe): 69/1,113 module fns h_exact-match matched corpus (~6%, LOW as
planned — modules are novel frontier); dedup_extend inapplicable (same-vram group model) ->
family_sweep --hseq --band all over the 57 matched-exemplar families: 408 member-matches
banked (182 into modules, 226 into the big 3 — families Part I's --only scoping missed),
169 failed + 77 STRUCT = genuine per-member frontier
- R22 clean-fleet 183/183 BYTE-IDENTICAL; audit-disc UNCLAIMED 75->34 residue 0 (34 = 31
parked-for-L3 + SC03/53,54,56 — 3 rows Discovery-3 never tiered, now parked with evidence)
- three instrument fixes, each negative-control-proven:
- family_sweep --hseq stub map derives ov_*+md_*+resident (was sig.ov_* glob -> module
members silently 'not-stub', R32 class) [committed earlier as commit:1506]
- sig-modules seeds from the built ELF's func_* symbols (bootstrap GLUES adjacent fns
around jtbl dispatch -> 24 false TRUNCATED; perturbed-sig control still bites)
- corpus.audit counts CODE lines only (module .s carries its header jtbl as .word lines);
progress.py buckets INCLUDE_RODATA symbols as blobs (unbucketed R32 hole)
- NEW HONEST BASELINE (183 binaries): 94.0% instr / 95.96% fn-count / 87.6% distinct;
tools-health OK, audit-digest OK
|
||
|
Drew T
|
ec1a805766 |
feat(phase-30 S44 I.1b): Makefile learns the module class — modules.mk + sig-modules
- -include config/modules.mk (silent when absent, same contract as overlays.mk) and BINARIES += $(MODULE_BINARIES). Everything downstream of $(BINARIES) — prune, check-all, build-all, expected — is untouched and picks modules up automatically. - NEW sig-modules target: signs every module at ITS OWN vram with ITS OWN --text-lo (the §154 module-id-word law — bootstrap from offset 0 yields 0 functions on 75/78 payloads). Derived MODULE_SIG_JOBS from modules.mk (R33, the sig-overlays pattern). Wired into tools-health after sig-resident. Empty registry = clean no-op (verified). - NEGATIVE CONTROLS: make -n sig-modules iterates an empty list; main rebuilds 143dbb89 byte-identical with no modules.mk present. |
||
|
Drew T
|
03794d91cd |
feat(phase-30 S43): make audit-disc — the disc PARTITION holds at residue 0; 34 UNCLAIMED code payloads
L1 of Drew's definitive disc audit ("we really need a full audit that definitively lists ALL code
that we need to decomp"). THE INVARIANT (R32): every byte on the disc belongs to exactly ONE bucket,
the buckets SUM TO THE DISC, and residue is a DEFECT — a partition with an asserted residue of zero
is a completeness proof; a longer list is only a longer list.
- WALKS THE DISC IMAGE, NOT OUR CONFIGS, classifies WHOLE payloads (no window), and decodes BOTH the
raw and LZSS layers — the three shapes that produced the three "more code all along" surprises
(the 0.4.dec glob missing 4 SC07 overlays; disc_code_sweep blind to COMPRESSED code, its type-4
row vacuous for 138 known binaries; a 4,096-word window reading only payload heads).
- CLAIMED-BY IS DERIVED (R33): config/check.<bin>.sha IS the SHA1 of that binary's disc payload, so
payload->binary is a hash lookup against the build's own byte-identity gate. It cannot drift.
- RESULT, 416,021,760 bytes, 1,291 payloads, RESIDUE 0:
onboarded-code 32,564,876 (7.83%) · UNCLAIMED-CODE 1,700,049 (0.41%) ·
classified-data 150,631,480 · audio-video 184,338,000 · filesystem-metadata 46,787,355
34 UNCLAIMED code payloads — largest a 383,783 B type-1 in MAIN.CD, the rest small type-1 entries.
These are the "there was more code all along" surprises, now ENUMERATED instead of stumbled into.
- MY OWN FIRST RUN FAILED THE PARTITION by -49,709,520 B, and the fail-closed exit is what caught it:
.DA entries' LBAs point PAST track 1 into the CD-DA tracks (double-counted against the whole-track
audio total), and .STR/.XA are MODE2 FORM2 (2324 user bytes/sector, not 2048). Both fixed.
- NOT wired into tools-health: it needs disks/, which a fresh clone does not have (H1).
- KNOWN GAP, stated not hidden: LIST.CD fails the TOC walk (it IS the TOC cache, not a container)
and is booked as data — correct today, worth a real classifier when L2 lands.
|
||
|
Drew T
|
a0e499d8f1 |
feat(phase-30 S39): make audit-frontier — the reconciliation gate (Drew's MASTER_REMAINING, derived form)
Drew asked whether we should build a master list of all funcs, a banked list, and a MASTER_REMAINING = total - banked that we hand-edit on every bank. Assessment in docs/decision-log.md (2026-08-04): ADOPT THE GOAL, REJECT THE MECHANISM. The triple already exists and is DERIVED, not maintained: total = .run/sig.*.jsonl (sig_image over the ORIGINAL bytes, independent of splat) banked = sig - stubs (INCLUDE_ASM pastes the original asm => not-wrapped == byte-exact) remaining = corpus.stubs() (filesystem-derived, coverage-asserted) and "remove it when we bank it" already happens -- banking IS deleting the INCLUDE_ASM line. A hand-maintained file would drift SILENTLY and flatteringly, which is the exact failure R33 exists for (fuel_manifest recorded 130 live stubs when the truth was 30, hiding 91.6% of remaining gain). What was genuinely missing is CROSS-ASSERTION. Six artifacts answer "what's left" -- corpus.stubs, worklist, backlog, family_hseq, fuel_manifest, progress.fleet -- each individually derived, none ever compared to the others. That is what cost P30 T0 a hand-reconciliation (family_hseq 29,961 vs progress.py 28,296). R34: not a better assertion inside one oracle, but a second one that can argue. tools/audit_frontier.py takes corpus.stubs as the reference and checks every other view against it: rows/targets naming an already-banked function, and any view whose PUBLISHED count disagrees with a recount. On first run it immediately caught a real one: family_hseq publishes 11,456 unmatched instances; only 11,297 of its members are still open per the corpus (delta +159) -- the map predates tonight's 159 banks. Ranking work off it would have mis-scoped by that much. It also PRINTS ITS OWN SCOPE LIMIT, deliberately: agreement here does NOT mean the denominator is complete. Every view, and the byte-gate itself, is blind to never-onboarded code -- the 39 type-1 modules and main's missing independent boundary oracle stay open (R34/R36). DELIBERATELY NOT wired into tools-health (Drew said "dont do this now" about the master list; this is the additive half). --strict exits 1 for when he wants it binding; wiring is one line. |
||
|
Drew T
|
1576570271 |
fix(phase-30 S1e): the distinct-code "regression" was a STALE DIGEST — alias lever ungated
The S38 checkpoint gated the phase's best lever ("do NOT scale the alias lever") on
distinct-code falling 89.3 -> 89.2. It never fell.
PROOF (each commit's metric recomputed from its OWN committed tree, 0 unresolved):
commit:1426 TRUE : instr 12394533 distinct 5022306 (77895 uniq)
commit:1426 COMMITTED: instr 12402412 distinct 5029324 (78025 uniq) <- stale
HEAD TRUE == COMMITTED: instr 12405402 distinct 5025082 (77952 uniq)
=> true delta 843->HEAD: instr +10869, distinct +2776 ins / +57 uniq. ALL ROSE.
The 843 digest was generated from a working tree still holding work REVERTED before the
commit landed (+7,879 ins / +130 uniq overstated) and never regenerated, so the next
HONEST digest read as a fall. => THE ALIAS LEVER IS UNGATED (scale it, §61 small batches).
Both recorded leads were wrong (R14): progress.py:423's SIG regex feeds fn-count ONLY
(neither weighted metric sees a C identifier — both derive matched = sig - corpus.stubs),
and "the harvest reverted functions to INCLUDE_ASM" died on one grep (483 removed, 0 added).
The 3-grep proof: identical sigs + unchanged tools/ + zero +INCLUDE_ASM => HEAD's stub set
is a strict subset => both numerators are FORBIDDEN to fall.
THREE INSTRUMENT DEFECTS, all one class (a bare except around a fail-CLOSED oracle):
- progress.py stub_addrs wrapped corpus.stubs in `except Exception: return set()`. An empty
stub set means "could not answer", not "no stubs", so matched = sig - stubs credited EVERY
function. Byte-witnessed: instr 100.00% / distinct 100.00% in a tree with no asm/. Now
propagates.
- cast_call_sites.tu_for + reconcile_tu.tu_for had the identical swallow, falling back to the
default <ov>.c instead of the jr/-O0 split TU — silently reinstating the exact bug
cast_call_sites' own docstring says it exists to fix. A wrong-TU reconcile fails the gate,
and this phase's base rate is ~24k PLUMBING vs 4,917 DIFF, so it presents as a codegen wall.
Now propagate CorpusError; ValueError fallback for curated names preserved; derived-TU path
re-verified (a _jr_ split stub resolves correctly, both tools agree).
NEW GATE (R34 — the byte-gate is a null oracle for DOCUMENTS; check-all stays 140/140 over a
stale digest forever): tools/audit_digest.py + `make audit-digest`, wired into tools-health
after report. Recomputes the three headline metrics from the current tree and fails if the
committed digest disagrees. Compares INTEGERS, not percentages — the +7,879-instruction
staleness printed as "94.4%" on both sides. Negative-control-proven against the stale 843
digest (fails, exit 1) and green on HEAD.
Verified: make report exit 0 (dedup-check 1910 validated / 0 failed, C1 coverage
241216/241216); audit-digest OK; cookbook-index OK (398 sections); metrics unchanged by the
fix (94.40% / 89.18%). No src/ or config/ edits — no bytes touched, nothing banked.
cookbook §140 · decision-log 2026-08-04 · SETUP.md inventory (R21) · R14/R32/R34/R35.
|
||
|
Drew T
|
b9efe66f91 |
fix(phase-30): the JR-PAIR "wall" was TWO instrument defects — pair banked, class retired
S28 ledgered `JR-PAIR-IN-ONE-O0-OBJECT` (two jr fns matched in one -O0 object => a clean build that cannot link: `undefined reference to $L105` + `func_8013C938`) with §81 step 1 (isolate one into its own code subseg) as the untested escape. BOTH the class and the escape are REFUTED — no isolation, no compiler wall, both fns banked from a genuinely clean fleet. The 4th consecutive "structural wall" to resolve to our own tooling (§124/§125/§126/§131). - DEFECT 1 (tools/jtbl_carve.py): ov_SC01_077_o0's carve at 0xb01a4 predates the §8e `tables=` persistence and is a MERGED DOUBLE (func_8013C0F8 $L75 + func_8013C414 $L105); the 2nd owner is MATCHED so extract pruned the stub .s naming its table. The single-table- predecessor inference derived 3 starts where the object emits 4 tables -> JTBL_PADS 0,4,4 -> jtbl_rodata_pads refused mid-stream, correctly. FIX: R32 coverage assertion + payload recovery at the single choke point (spec_from_starts) — every zero word inside a span is an original `.align 3` pad (the tool's own axiom), so the word after it STARTS a table; recovered starts are logged. No-op where structure is known (the 134 sibling _o0c spans carry tables=+0x0,+0x70). Honest limit: tight (0-pad) boundaries stay unrecoverable but fail LOUD via the filter's count guard — never silent. - DEFECT 2 (Makefile): no .DELETE_ON_ERROR, so `as` (a pipeline consumer) left a TRUNCATED .o on disk — 12 of 16 T func_, undefined $L57/$L59/$L63/$L75/$L76 — newer than its .c, and the NEXT build linked the corpse. That IS the S28 link error, one build downstream of a loud, correct compile error. Negative-control-proven on a scratch invocation. - BANKED: func_8013B83C (272 ins) + func_8013BD74 (198 ins) in ov_SC01_077 (d19c9580). Byte proof: 4 tables 0x801D8254/828C/82FC/836C (13/27/27/27 entries, each zero-pad separated); span 0xb00fc..0xb0280 = 388 B = 52+4+108+4+108+4+108 exactly; spec 0,4,4,4. - R22 clean-fleet (make clean + extract-all + check-all): 140 passed, 0 failed of 140. The incremental result was NOT trusted (§130). Fleet 93.25% fn-count / 89.2% instr / 80.5% distinct; dedup 1905/0; 0 NON_MATCHING (G4). - cookbook §132 + index (356 sections): the mechanism, the fingerprint (an undefined $L<n> in a LINK error is a truncated object, never codegen), the 30-second standalone-TU ladder that named the 4th table owner before any build, and the transferable rule — a fail-loud guard is only as trustworthy as the artifact hygiene around it. |
||
|
Drew T
|
05f6293067 |
fix(phase-30): 137 tracked C sources held a raw NUL that made grep SILENTLY SKIP them
FOUND BY ACCIDENT, WHICH IS THE POINT. `grep -rn func_8013C08C src/` returned NOTHING for a function that is defined right there. The file held a RAW NUL byte inside a character literal — the source read `== '<NUL>'` where it should read `== '\0'`. It COMPILES (the fleet was byte-identical), so no byte-gate ever objected. But file(1) classifies such a file as `data`, and **grep treats a file containing NUL as BINARY and reports nothing, silently**. The whole file therefore vanished from every grep-based audit and every hand search. I burned real time chasing a phantom missing function before `file` gave it away. SCOPE, measured: 137 files — every `_o0c`/`_o0e` region created in THIS session. The templated bodies carried the NUL fleet-wide, so I propagated the defect today. All fixed (`'<NUL>'` -> `'\0'`); R22 CLEAN-FLEET 140 passed, 0 failed of 140 => byte-neutral. NEW ORACLE: tools/audit_text_sources.py + `make audit-text-sources`, wired into tools-health, coverage-asserting over all 3,887 tracked .c/.h files (R32). This is the SAME silent-skip family as SS124 (a scanner that cannot see something reports it is not there) and SS126a (a bare except swallowing a coverage assertion) — but one layer LOWER, in the tool everyone reaches for first. The byte-gate is structurally blind to it (R34): the bytes are correct, so it has nothing to say. It needs its own oracle. MY OWN ERROR, RECORDED: proving the new guard fires, I injected a NUL into the REAL tracked file and restored it through nested shell escaping. The restore left `'\\0'` — an escaped backslash, i.e. a multi-character constant, NOT a NUL — a genuine semantic change. R22 caught it (139/140) in one cycle, before any commit; repaired to `'\0'` (10465 -> 10466 bytes) and re-verified 140/140. The lesson is not "be careful": a negative control must corrupt a SCRATCH COPY under .run/, never the tracked file it is testing. Testing a guard must not risk introducing the defect the guard exists to catch. |
||
|
Drew T
|
45d26cc413 |
feat(phase-30): T2 carve-within-a-carve PROVEN end-to-end; Arm-A does not bite; 3 -O0 fns banked
T2's central unknown is resolved, and it is NOT what the phase plan predicted. The plan
named the Arm-A splat `%lo +0x20` defect as T2's real substance. Four probes, each
isolating ONE variable, SHA vs config/check from a clean tree (SS125 rules):
1. sub-split the jr object at ARBITRARY addresses, pure -O2 -> BYTE-IDENTICAL
** the re-carve is NEUTRAL; Arm-A does not bite here **
2. same split, middle region routed to -O0 -> DIVERGED (2 vars at once)
3. same NAME as probe 1, only the -O0 flag added -> DIVERGED
** therefore the FLAG, not the subseg name **
4. -O0 regions cut to EXCLUDE the matched bodies -> BYTE-IDENTICAL
** route PROVEN end-to-end **
THE REAL OBSTACLE: an address range is not an optimization region. Interleaved among the
15 -O0 stubs at 0x80183CF0..0x80184920 are TWO already-MATCHED functions (func_80184440,
func_801848E4) that expand from engine_core.h and compile at -O2. Flipping the FILE to -O0
recompiles them too. Cut around them and it is byte-clean.
MY EARLIER "15 contiguous -O0 fns, clean cut" WAS AN UNDER-COUNT (R14 on myself): I derived
it by scanning asm/**/*.s for the frame-pointer prologue, and a MATCHED function emits no
.s -- so the scan was structurally blind to exactly the bodies that break the flip. Same
shape as SS124. Any T2 driver must derive -O0 bounds as (address range MINUS already-matched
bodies), never from an asm scan.
BANKED (3, whole-binary gate the sole arbiter): func_801846E4 + siblings func_8018473C /
func_80184794. Each global DERIVED FROM THE ASM (%hi/%lo operands), not taken from the
draft's comment; 3/3 match_one --o0 MATCH (22 ins) with a -O2 control showing the mismatch;
3/3 verified through harvest_verify; bank truth read from the SOURCE (SS55b trap 4).
MAKEFILE: the -O0 glob widened `ov_*_o0b.c` -> `ov_*_o0?.c` so ANY lettered -O0 sub-split is
covered by one rule. A MISSED rule is silent -- the region would compile -O2 and every
residual it produced would be a pure artifact (SS116). corpus.o0_sources() re-verified: 137
sources, resolves `?` via glob, both pre-existing rules intact.
CONFIG: ov_SC03_014_jr_8017EB7C sub-split into 5 regions (pre / _o0c / matched-O2 /
_o0d / post), reusing jr_isolate_all's plan+build_new_config+validation verbatim so the
carve-repoint and source-repartition semantics are the proven ones.
R22 CLEAN-FLEET: extract-all 139/139 (+main) ; check-all 140 passed, 0 failed of 140.
|
||
|
Drew T
|
4570a5854b |
feat(phase-30): cookbook-index — a SYMPTOM-keyed index (226 sections, derived + coverage-asserted, in tools-health)
Wave-1 measured the tax: three agents each reported a 'NEW idiom' that was ALREADY documented — the asm-label alias (line ~2516, same 'address-of perturbs regalloc' mechanism) and the void->s32 non-neutrality (§41d, Phase 26; the agents cited the very entry §41d corrects). They consulted the cookbook as instructed and could not FIND them. 716 KB / 226 sections with no index = a discoverability failure, and every wave re-paying for prior waves' findings is the inverse of R16. docs/cookbook-index.md maps SYMPTOM (what you see in the diff) -> sections, 14 buckets, a section listed under every symptom it addresses. Derived by tools/cookbook_index.py (R33 — cannot drift), --check wired into tools-health. R32 on my own tool: the first regex required an em-dash separator and silently dropped 50 sections — including §1 (idiom catalog), §2, §5a (cross-jump, cited by an agent today). An index missing its most-cited entries turns 'I could not find it' into 'it is not there'. Now asserts extracted == candidate '§' headers and hard-exits on a gap. |
||
|
Drew T
|
ef85803b1f |
feat(phase-29): backlog prune — compact the append-only near-miss log + wire into make report
The near-miss ledger (.run/backlog.jsonl) is append-only, so it filled with already-banked noise: 6,867 rows, ~98% banked. load_best()/render() already filtered on READ (docs/backlog.md was correct), but the raw log drifted stale and every render re-scanned all 6,867 rows against the stub oracle. - backlog.py: new `prune` subcommand — atomic rewrite (temp + os.replace) to load_best()'s output (drop-now-matched P9 + best-per-addr collapse). Idempotent. 6,867 -> 1,704 open near-misses. - Makefile: `backlog.py prune` wired into `make report` (BINARY=main block) so the ledger tracks reality every cycle instead of drifting. - Finding (Drew's question): crack waves DO log every non-byte-match to the backlog durably (gate_stage copies best_draft -> .run/backlog_drafts/). BUT the `closeness` field is UNRELIABLE — byte-correct drafts (match_one MATCH) are logged with closeness>0 (e.g. func_8012F49C logged 29, actually MATCH). And a reach-N function's draft is overlay-SPECIFIC (per-location symbols), so the backlog is a messy recovery source vs the fresh per-wave stranded drafts. Integration-recovery should consume the fresh wave-dir strandeds, not re-derive from the backlog. |
||
|
Drew T
|
5e78dccf07 |
feat(phase-29 §8e): jtbl 8-align pad-spec filter — the 4-giant unblock tooling (fleet-neutral)
- ROOT CAUSE PINNED (the session-2 half-pin was INVERTED; both probes were vacuous, R35): cc1 emits .align 3 before EVERY jump table; maspsx passes it VERBATIM (the :435 'drop' is an inventory-only pass); as bakes the pad SECTION-RELATIVE; link placement was never guilty (SUBALIGN(2) + ALIGN(.,4) place 4-mod-8 carve starts tight). Merging originally-separate TUs fires an intra-TU align where the original packed tight -> +4 at rodata 0xCC -> image-wide %lo shift. Honest probes persisted: .run/probe_jtbl/ (verdict.md + objdumps). - NEW tools/jtbl_rodata_pads.py: post-maspsx filter replaces each rodata .align 3 with the ORIGINAL's exact pad bytes per a JTBL_PADS spec; fail-loud on table-count drift / non-align-3 / non-jtbl rodata content. Byte-proven: verbatim 0xE4 pad-at-0xCC -> filtered 0xE0 tight (= the merged carve span). - jtbl_carve.py: spec-aware same-subseg merge (gap 0 or 4-with-zero-payload-word; else NON-CONTIGUOUS -> isolate), interval-arithmetic pad specs (committed values CARRIED, never re-derived), JTBL_PADS target-var emission into overlays.mk + revert() restore + stale-.o invalidation; the false 'maspsx drops .align' docstring corrected (H5). - Makefile: $(if $(JTBL_PADS),| jtbl_rodata_pads.py ...) stage in build/src/%.o + file-scope empty default (env-shield). jtbl_family_bank.stub_file: duplicate-stub fail-loud (the earlier 'ladder failure' was a wrong-TU splice into a stale _a.c stub, byte-witnessed). - R22 clean-fleet WITH the fix wired: 140/140 byte-identical, tools-health green (dedup 1846/0, C1 234205/234205), ZERO new banks -- fleet-neutral by construction. - cookbook §8e (the jtbl alignment law) + §8a/§8a-pad corrections; decision-log R31 entry; SETUP.md tool row; .gitignore allowlist for the probe verdict artifacts. |
||
|
Drew T
|
f6f89781ff |
feat(phase-29 T2 Arm A): swing verdict = BANKED FACT (9/9 -O0 members on ov_SC07_010); fleet -O0 rollout deferred at the splat wall
- tools/rollout_o0_cluster.py (new) + Makefile O0_CLUSTER_OBJS -O0 wildcard: the -O0-cluster carve (0x13410..0x14834), adapting rollout_whale_o0.py to a 3-way <ov>/<ov>_o0/<ov>_o2b split - ov_SC07_010: carve byte-neutral -> family_sweep --hseq banked 9/9 -O0 exemplar-family members whole-binary (R22 clean-fleet 140/140). The Task-1 masked-MATCH swing verdict is now a BANKED FACT: -O0 cluster members DO bank at -O0 (§52b). Phase-20 'func_8013B7AC overlay-local' refuted. - THE WALL (byte-proven, TOOLING not compiler): the same carve on 006/007/011 byte-shifts the whole image (+0x20 %lo data-symbol shift, 34% diff) from a CLEAN build; boundaries verified as real fn-starts. Root cause = splat re-disassembly of a 3-way-split subseg that still holds INCLUDE_ASM stubs (the whale's stub-free _o0b shape avoids it). The Phase-20 '-O0 split infra' wall, root-caused. - DEFERRED (ROI): full -O0 fleet rollout (~1,233 / ~0.6pp) — 3/4 sampled walled + 134 jr-embedded + bigger levers (Task 3 core-cracks, Task 6 tiny-IMM ~5,566). decision-log R31 + cookbook §18-P29. - 140/140 byte-identical; dedup 1840/0; 0 NON_MATCHING (G4); main 143dbb89. Task 2 substantively done. |
||
|
Drew T
|
758c4365b6 |
feat(phase-28 T7): make audit-binaries — the R36 citizenship gate (R32 enforcement)
R36: a newly-discovered binary is not real until every consumer knows it. Onboarding produces a
byte-CLEAN binary (check-all green) that is not yet a CITIZEN — the tools that enumerate binaries
can each be silently unaware of it, and the byte-gate is structurally blind to that (R34).
Not hypothetical: P27 onboarded 4 SC07 overlays byte-clean; P28 found FOUR consumers silently
ignoring them (family_remap.img_path, .run/family_hseq.json, config/dedup.us.yaml, and the overlays'
own .c), hiding ~6,400 already-matched bodies. Every failure was silent.
- tools/audit_binaries.py asserts, coverage-checked BOTH directions (R32), against the config the
BUILD reads (R33 — onboarded = main + resident + every config/splat.ov_*.yaml):
1. dup_report.BINARIES (what corpus/family_hseq/progress all derive from) EXACTLY equals the
onboarded set — a missing binary is invisible to every derived tool; a phantom is invented.
2. every onboarded binary has a byte-derived sig.
3. THE LOAD-BEARING SC07 CHECK: every onboarded OVERLAY's .c includes ../shared/engine_core.h,
or no shared body can ever reach it (main/resident have their own bodies, exempt).
4. every onboarded overlay is represented in the family map (warn — regenerable/may post-date).
INFO: dedup-group membership (0 = onboarded-but-un-harvested, a dedup_extend candidate).
- NEGATIVE CONTROL: stripping the shared include from ov_SC07_006.c makes the gate FAIL loudly and
exit 1 — it catches the exact bug that hid 6,400 bodies for a month. Restored clean.
- Wired into `make tools-health` (the pre-matching ritual) — cheap (config + text scans, no build),
so it sits in the fast lane. Passes today: 140 onboarded, all full citizens.
- Reads config/dedup.us.yaml as TEXT (never a YAML round-trip — the H5 lesson from T4).
|
||
|
Drew T
|
427baba3bf |
feat(phase-27 T10): completion dashboard (main in the weighted metric) + the resident second oracle
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
|
||
|
Drew T
|
ebdef9012b |
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired. |
||
|
Drew T
|
d7d2613ae6 |
fix(phase-26a): A9c — lint_symbol_refs green + wired into make report
The ONLY detector for the R22 rename-drift failure mode (a symbols.us.txt rename leaves a func_<ADDR> ref dangling in committed src; a clean rebuild fails, an incremental build masks it with a stale .o — undetected Phase 21→23). It was RED (262 false positives) and UNWIRED. The audit's 3 blind spots, all fixed: #1 globbed src/**/*.c only -> now ALSO scans src/shared/*.h, where engine_core.h's 10k+ func_/D_ tokens live and one dangling ref breaks EVERY clean build at once (negative-control-proven: an injected bare ref in a shared header IS flagged). #2 read 2 of 138 symbol files -> now reads every REAL stacked file (config/symbols*.txt), and correctly EXCLUDES the R13 proto files (never stacked into a build; would invent phantoms). #3 no __asm__("label") model -> ALL 262 false positives were this class: engine_core.h binds func_8005C324 to memcpy via `... __asm__("memcpy")`, so the ref emits the LABEL and resolves at link regardless of the rename. asm_labeled_addrs() now recognizes it. VERIFIED: green on HEAD (exit 0); negative control (inject a bare func_800d1bd8 -> DsMix in a shared header) FLAGS it (exit 1) then clears on removal — detection intact, not trivially green. Wired as a fail-closed gate in `make report` (make -n confirms), beside dedup_integrate --check. No src/build change (report-time integrity gate) -> the byte-identical build is unaffected. |
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|
||
|
Drew T
|
af2f40d153 |
fix(phase-26a): A4/A5 — 193 unmatchable slices dissolved; the closeness oracle stops lying
R22 CLEAN-FLEET: make clean -> extract 136 -> build 136 -> check-all = 136 PASSED, 0 FAILED.
make audit-corpus: 0 PHANTOM + 0 TRUNCATED (was 193).
=== A4: a CORPUS defect the byte-gate could never have caught ===
config/symbols.us.txt:981 declared `listCdBuffer = 0x80180000` — a correct Phase-3 name for MAIN's
LIST.CD RAM buffer. But that address is OUTSIDE main's image and INSIDE the overlay slot, and every
overlay's splat config stacks symbols.us.txt. High RAM is REUSED: an address that is a buffer to main
is live CODE to an overlay. So splat saw a symbol boundary mid-code and, across 97 of 134 overlays:
* CUT 97 REAL FUNCTIONS IN HALF (a head ending on a `lui`, no return), and
* INVENTED 96 PHANTOM ONES (a tail beginning by reading the assembler temp $at).
193 slices NOBODY COULD EVER MATCH — not "hard", not "a compiler wall": unmatchable by construction.
They sat in the harvest queue as ordinary work, so agents would burn on them forever and the failures
would be filed as intrinsic compiler residuals.
The phantom listCdBuffer.s in ov_SC01_005 literally begins:
lw $ra, 0x10($sp) / addiu $sp, $sp, 0x18 / jr $ra
splat cut a function immediately before its EPILOGUE and called the epilogue a function.
AND IT HAD ALREADY CONTAMINATED REAL WORK: in ov_SC03_031 the cut landed where the epilogue was
exactly `jr $ra; nop`, so the Phase-26 x134 sweep innocently BANKED the phantom as
`void listCdBuffer(void) {}` — byte-correct, gate-green, entirely fictitious — while leaving
func_8017FFC4 permanently unmatchable. Removed.
WHY NO GATE CAUGHT IT, AND WHY THAT IS THE POINT: INCLUDE_ASM pastes the two .s halves back VERBATIM
in original order, so the image is byte-identical either way. The byte-gate was green the whole time
and always would have been. It is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle. No
assertion added INSIDE it could ever have found this. What found it was a SECOND, INDEPENDENT oracle:
tools/sig_image.py derives boundaries from the ORIGINAL bytes without splat, and DISAGREED with the
corpus (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements).
=> When one oracle is structurally blind to a class of error, the answer is not a better assertion
inside it. It is a SECOND ORACLE THAT CAN DISAGREE WITH IT. (`make audit-corpus` is now that.)
THE RULE (the mirror of R13/R15, never written down): a symbol whose address falls inside ANOTHER
binary's vram window must never enter that binary's symbol stack.
FIX: config/symbols.us.ram.txt — main-scoped symbols outside main's image — stacked ONLY by
config/splat.us.exe.yaml. Main keeps the name it needs (10 %hi / 11 %lo refs; 143dbb89 byte-identical);
the overlays never see it. Exactly one symbol was in scope fleet-wide; the resident window was clean.
AND A REAL FUNCTION THE ACCIDENT WAS HIDING: in ov_SC01_084 / ov_SC02_041 / ov_SC03_094 / ov_SC06_008
there IS a genuine function at 0x80180000 (111 / 35 / 28 / 74 ins), reachable ONLY via a fn-pointer
table (.word func_80180000) and never by `jal` — so splat cannot find it and needs the boundary
DECLARED. listCdBuffer had been supplying it by luck. Now declared honestly, per-overlay, in
config/symbols.<ov>.txt — exactly where R13/R15 says an overlay-scoped symbol belongs.
=== A5: the closeness oracle every crack agent trusts was lying on 155 functions ===
masked_diff._reloc_kind() knew 26/HI16/LO16. An over-approximating sweep of every reloc objdump emits
across all 3,367 build objects found FOUR: R_MIPS_26, HI16, LO16 — and R_MIPS_PC16 (211). PC16 fell
through to a FULL-WORD compare, but the object holds an UNRESOLVED PLACEHOLDER in the branch
displacement, so that compare can NEVER succeed.
DECISIVE TEST (derived from the invariant, not from reading the regex): INCLUDE_ASM pastes the
ORIGINAL asm, so for every stub diff_object_s() MUST be 0. Measured, coverage-asserted:
2,741 functions scored — old mask: 150 LIES; PC16 masked: 4 LIES.
(The 4 survivors are the separate length-delta defect.) A phantom non-zero sends an agent to grind at
a wall that is not there, and the wasted attempt is then booked as a MATCHING failure, feeding
reserved_walls() and PERMANENTLY BLACKLISTING a function that was never broken.
=== NEW FINDING (found by cutting the R22 corner): a STALE OBJECT CAN PRODUCE A FALSE PASS ===
`.o <- .s` is not a dependency make can see: assembly arrives via INCLUDE_ASM, expanded to a `.include`
consumed by maspsx/as AFTER cpp, while -MMD tracks headers only. Re-extract, build incrementally, and
make links a STALE object. This is not merely slow — INCLUDE_ASM pastes the ORIGINAL bytes, so a stale
object still yields the original image: SHA1 GOES GREEN while the split just changed is never exercised.
A broken config change can be "verified" by an incremental build. Live proof: 8 of 136 binaries linked
stale objects here; they failed LOUDLY ONLY BY LUCK (the dead symbol was an undefined reference) — a
merely-different-but-valid split would have gone green on all 136.
R22/H3 already legislate this, and I broke them. But a rule that needs a human to remember it is not a
gate. FIX: `extract` now invalidates the objects that include what it just rewrote (main's are top-level,
so -maxdepth 1 — verified it cannot clobber the other 1,605 objects). Structural, not advisory.
R14 self-catch, recorded: my first A5 test passed `fn=` to diff_object_s(), which takes two args; the
TypeError was swallowed by my own `except Exception: continue` and it reported 0 scored / 0 lies. I
wrote the exact bug I was auditing, inside the test for it. Caught only because 0 looked wrong. The
test now asserts its own coverage.
|
||
|
Drew T
|
f7b7399ebe |
feat(phase-26a): A3 — tools/corpus.py, ONE derived corpus oracle (+ a second oracle that can disagree)
The 28 surviving audit findings collapse to ONE bug repeated ~10 times: a hand-maintained model of
the corpus layout (a file allowlist, a single-.c assumption, a func_-only symbol regex, a REGION_SUB
dict) sitting on top of a filesystem that already answers the question. The fix is not ten repaired
regexes — it is one DERIVED oracle and ten deleted scanners (R33).
WHAT IT DERIVES FROM
1. THE FILESYSTEM. Which .c files make up a binary, and where a function's .s lives, are FACTS OF
THE TREE THAT SPLAT ITSELF WROTE. The INCLUDE_ASM line is SELF-DESCRIBING — its first argument
IS the asm subdir — so there is nothing to guess and no dict to rot. A dict literal is strictly
worse than the filesystem AND it fails OPEN (silently yields a wrong path) instead of closed.
2. THE PROVEN INVARIANT. INCLUDE_ASM pastes the ORIGINAL asm and the build is byte-identical, so a
function NOT wrapped in it is byte-exact. `matched` is DERIVED as sig - stubs, never re-parsed
from C text. (progress.py learned this the hard way: weighted_metrics() derived and was right;
classify() re-parsed C and inherited a bug.)
VALIDATED against the real corpus:
* ov_SC01_077: 264 stubs across 14 files. The old 3-file allowlist saw 30.
* Fleet: 58,717 stubs vs the allowlist's 1,992 — 56,725 (96.6%) were INVISIBLE.
* Coverage-asserted (R32): every INCLUDE_ASM line must parse, every symbol must resolve (ANY C
identifier — a func_-only regex silently misses the 100 curated listCdBuffer stubs), every stub
must have a .s. A silent skip is a DEFECT, not a no-op.
THE SECOND ORACLE (`make audit-corpus`) — the real lesson of this audit.
The byte-gate is structurally BLIND to a bad function boundary: the .s halves are pasted back
verbatim in original order, so the image stays byte-identical and green. Only an oracle that can
DISAGREE can see it. sig_image is that oracle — Ghidra-free, derived from the ORIGINAL bytes,
independent of splat. corpus.audit() cross-checks the two and reports:
PHANTOM — a stub address the sig does not know: splat INVENTED a function.
TRUNCATED — a stub whose .s length != the sig's: splat MIS-SLICED one.
It reports 193 (96 + 97) — reproducing the A2 audit's number EXACTLY, from an independently written
tool. That is a third confirmation of the listCdBuffer defect (auditor -> skeptic -> this).
AND AN R14 SELF-CATCH, recorded because the near-miss is the lesson.
Run naively over all 136 binaries the same check reports 914 slices — 4.7x the truth. It is noise:
main/resident are signed by the GHIDRA dumper, whose boundaries are shorter than splat's by design
(and which never analysed the linked PsyQ subsegs at all), so the comparison measures GHIDRA'S limits,
not splat's errors. Only the overlays are signed by sig_image, the oracle actually validated at
58,524/58,621. sig_is_independent() now encodes that domain, with the reasoning, so nobody repeats it.
A check applied outside its valid domain does not become more thorough — it becomes noise.
`make audit-corpus` is RED by design until A4 removes the bad symbol line; then it becomes a gate.
|
||
|
Drew T
|
6fb8d323bd |
perf(phase-26): parallelize R22 clean-fleet — ~9min -> ~50s (10.5x), 136/136 byte-identical
- profiled the clean-fleet R22: extract-all ~6m11s (136 serial `splat split`) + check-all ~2m58s (136 serial builds) = ~9 min, all serial on a 32-core box - the only shared WRITE is the 4 generated include/*.inc macros at extract time (identical content per binary); everything else is per-binary-disjoint and include/ is read-only during a build -> concurrent builds/extracts don't race - Makefile: JOBS ?= 16 + `make extract-all` (seed main serial for the macros, then parallel- extract the rest via xargs -P) + parallel `make check-all` (xargs -P), correctness-gated - MEASURED: `make clean && make extract-all && make check-all` = ~50s, check-all 136/136 BYTE-IDENTICAL (== the serial result) -> ~10.5x. Compounds across the endgame (R22 per commit) - new R22 recipe: `make clean && make extract-all && make check-all` (was the serial for-loop) |
||
|
Drew T
|
095a611e75 |
feat(phase-26): §8 jtbl-rodata tooling — overlay PoC proven (func_8012ACE0, R22 136/136)
- overlay jr-functions can now bank as C: gcc switch jump tables form a .rodata island at the overlay TAIL; carve a matched fn's jtbl into a dotted [.rodata, <code-subseg>] subseg + ld_interleave (data->rodata->data sandwich) places it byte-exact. cookbook §8a + SETUP. - tools/ld_interleave.py: --section .<binary> param (derives the <binary>_TEXT/DATA/RODATA/ DATA2/BSS symbol prefix); default .main = the EXE, byte-identical (backward-compat proven) - Makefile + config/overlays.mk: <bin>_JTBL_INTERLEAVE hook + a $(strip)-guarded extract branch (gotcha caught: a trailing #comment on the := left whitespace -> non-empty -> the branch misfired on resident with the EXE defaults) - PoC: func_8012ACE0 (25-ins jr-fn in ov_SC01_077) reconciled (canon_sig_reconcile) + banked BYTE-IDENTICAL d19c9580 -- the first overlay jr-function matched through the C pipeline - R22 FULL-FLEET clean rebuild: 136 passed, 0 failed (main 143dbb89 unaffected by the ld_interleave change); 0 NON_MATCHING in any default build (G4) - P9 findings: func_80159C84/func_8015444C (the 2 carried Fable5 jr bodies) are rtu_match FALSE-matches (incomplete jtbls: 52B vs 56B -> never bank); the maspsx "hang" scare was a truncated experimental-file artifact (real pipeline builds in ~1s) - metrics: distinct-code 39.1% (50,572 unique fns), instr-weighted 62.9% - NEXT: the ×134 automation (generate the per-overlay carve + template the reconciled body) |
||
|
Drew T
|
8cfbbf2d47 |
feat(phase-24): T7 whale func_80144B9C banked ×134 (-O0 reach-134 rollout) — clean fleet 136/136
The single biggest byte-weight lever (770 ins ×134 ~ +1.6% byte-weight), banked across all 134 overlays as -O0 compiled C (was ×1 in ov_SC01_077, commit:0463). - ROLLOUT (tools/rollout_whale_o0.py): per single-file overlay, LINE-split <ov>.c at the whale (splat emits in vram order), carve the yaml code subseg into before/<ov>_o0b(-O0)/<ov>_after, thin <ov>_o0b.c #includes the shared src/shared/func_80144B9C.h (the whale C moved there — DRY, TU-local typedefs, not a fragile 200-line DEFINE_ macro). Makefile WHALE_O0B_OBJS wildcard -O0. - DEDUP: group E_func_80144B9C (134 members, h_exact 74186b97, source=the header; group_members keys on binary+vram so a header-share validates like a macro-share). - MILESTONE: clean fleet check-all 136/136 (R22); dedup-check 1811/0; FLEET REAL 225816->225949, byte-identical 65.91->65.95% function-count (the gain is byte-weighted). Validated on ov_SC01_000 + spot-checks SC02/SC03/SC07 before the full fleet. - cookbook §38 + SETUP inventory. |
||
|
Drew T
|
3b0cc190b2 |
feat(phase-24): T7 whale func_80144B9C CRACKED + banked ×1 (-O0 struct-assign) + ×134 infra
- CRACK (cheap Opus, no Fable5/calls.c): the 770-ins reach-134 whale's 2-insn residual was a
STRUCT ASSIGN `D_80078E50 = *ent;` (sizeof(ENT)==0x24), NOT an explicit memcpy() call — gcc-2.7.2
-O0 expands a >MOVE_RATIO-word struct copy via emit_block_move->emit_library_call(memcpy),
precomputing dst/src into pseudos then addu into $a0/$a1 = the exact 2 missing moves.
- -O0 SPLIT: whale is a 2nd -O0 region (prologue 21F0A003) in the -O2 'after' segment; carved into
its own -O0 object o0b (splat.ov_SC01_077.yaml 3-way + Makefile CC1FLAGS=-O0; address-sorted
line-split preserves all banked C). ov_SC01_077 == d19c9580.
- memcpy: memcpy=0x8005C324 in symbols.resident.txt (overlays+resident only; main untouched vs its
MEMCPY.o) + __asm__("memcpy") on the engine_core.h block-copy macro (non-builtin C name -> no
built-in codegen, same call).
- MILESTONE: CLEAN fleet check-all 136/136 byte-identical (R22), dedup 1810/0. reach-134 -> ×134 (W9) next.
|
||
|
Drew T
|
be33b887fe |
feat(phase-19): T1 -O0 split lever — infra proven + 6 -O0 fns matched (ov_SC01_077)
- config/splat.ov_SC01_077.yaml: 3-object code split (ov_SC01_077_a before / ov_SC01_077_o0 -O0 cluster / ov_SC01_077 after). A single object's .text cannot be split around a middle object, so before/after are distinct objects; the after-region keeps the ov_SC01_077 name (bulk matched C + asm paths unchanged). - Makefile: target-specific CC1FLAGS:=-O0 for ov_SC01_077_o0.o (src/boot.c precedent). - src/ov_SC01_077/ov_SC01_077_a.c (new before-region) + ov_SC01_077.c (after-region) + ov_SC01_077_o0.c (new -O0 cluster): 6/16 -O0 fns matched byte-perfect (func_8013B568/B7F4/BD34/C360/C938/C964). - DEFERRED to Phase 20 (Drew, on the discovered difficulty): the 10 remaining -O0 fns hit an indexed-global %lo-folding codegen quirk (gcc-source research, R17) + the x134 per-overlay rollout (engine_core.h compiles -O2, cannot carry -O0 fns). Corrects the Phase-18 backlog premise (afternoon/free-x134) per R14. Documented in ov_SC01_077_o0.c. - verified clean: main 143dbb89, resident 8e17e02f, ov_SC01_077 d19c9580 + 2 overlays. |
||
|
Drew T
|
c716eb15e1 |
feat(phase-15): T4 — fleet progress roll-up + fix cross-report double-count
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index (fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic, source-derived per-binary table + fleet totals). Single-binary output byte-unchanged. - fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82% across 136 binaries (main + resident + 134 overlays) - dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups / 28.6M collapsible (matches the Phase-11 figure) - Makefile: wire progress.py --fleet into make report (main-only block) - docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory - make report green; dedup-check 8 validated / 0 failed - ghidra/ churn NOT staged (R23) |
||
|
Drew T
|
0c7619231d |
feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X). - proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all 18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test (corrupt shared body) -> make check FAILS then restores - Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a shared header recompiles dependents — closes a stale-build/false-pass gap the negative test exposed; output-neutral (R22 clean 18/18) - structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate can't catch under-application -> assert DEFINE present + stub gone - FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest - cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory - ghidra/ churn NOT staged (R23); build/asm/.run gitignored |
||
|
Drew T
|
e2200e17db |
feat(phase-13): T5 — make build-all / check-all (fleet PASS/FAIL)
Recursive $(MAKE) BINARY=<b> over $(BINARIES) (correct per-binary OBJS prune, not foreach), aggregated to one PASS/FAIL + per-binary .run/check.<b>.log. check-all: 6/6 [OK] exit 0; negative control (corrupt sha) -> [FAIL] + nonzero. |
||
|
Drew T
|
144839f24b |
feat(phase-13): T1a/T1b — SC01/077 onboarded + all-asm byte-match (d19c9580)
- ov_SC01_077 scaffolded: config/overlays.mk (OVERLAY_BINARIES + var block), config/splat.ov_SC01_077.yaml (from template), check.sha, empty symbols, + ov_SC01_077 entry & sentinel anchor in the 4 Python BINARIES dicts - GATE T1b: make check BINARY=ov_SC01_077 -> d19c9580 BYTE-IDENTICAL @ 100% INCLUDE_ASM; clean rebuild leaves main 143dbb89 + resident 8e17e02f unregressed - reusable non-word-aligned-overlay handling (~75% of fleet) added to the template + Makefile: (1) [word_floor, bin, trailing] carve for the final 1-3 bytes spimdisasm drops; (2) build/assets/%.o incbin rule (+ .data align=1) for splat bin assets (asset_path scoped per-alias); (3) objcopy end-align TRIM (shrink-only, <=3 B) removing the .ld's segment-end ALIGN(.,4) pad - .gitignore /assets/ (regenerable splat output); clean removes assets/ |
||
|
Drew T
|
50c5c9ea33 |
feat(phase-13): T0 — overlay splat template + overlays.mk indirection
- config/splat.us.overlay.template.yaml: reusable flat-blob splat template for location overlays (@TOKEN@ placeholders), modeled on splat.resident.yaml; code at file 0x0 (no leading-rodata trick), stacks symbols.resident.txt (overlays call the engine), vram 0x80128158 - Makefile: -include config/overlays.mk + BINARIES := main resident $(OVERLAY_BINARIES); the generated overlays.mk keeps new_overlay.sh out of the hand-maintained Makefile body; silent when absent (fresh-clone-safe) - GATE T0: clean rebuild main 143dbb89 + resident 8e17e02f BYTE-IDENTICAL (indirection proven a no-op) - phase-ends/CURRENT_PHASE.md: Phase-13 task log (plan approved gate 1) |
||
|
Drew T
|
f672c709c1 |
feat(phase-11): T6 — sign the 134 overlays; cross-report spans 136 binaries (milestone report half)
- sig_image overlay discovery: linear partition (split contiguous code at jr+delay boundaries) + detect_code_end (first run of invalid instrs = the code->data transition; overlay code decodes ~100% valid, data drops to 43-95%). BFS dead-ended — overlays dispatch via function-pointer tables, not jal — so call-graph BFS found ~2 fns; linear partition recovers the whole code prefix - Makefile: sig-overlays: signed 134 overlays -> .run/sig.ov_*.jsonl signs all 134 SCxx 0.4.dec @ 0x80128158 -> .run/sig.ov_*.jsonl (27s) - dup_report --cross: ingest the overlay sigs (sig.ov_* convention) + main/resident; condensed source header (overlay set-sha for reproducibility); cap top-200 per subsection + state totals; compact member sample + #bin column - RESULT (docs/duplicates.cross.md): h_exact 9366 cross-binary groups / 28.5 MB collapsible; h_norm 8957 / 38.3 MB. Top group = a 770-instruction function BYTE-IDENTICAL in all 134 overlays (one match credits the whole fleet) — 'one match unlocks many' quantified; the Phase-12/13 work queue - dedup-check green; committed per-binary digests byte-stable; report deterministic (sorted glob, no Date/random); the EXE<->resident pair still shares nothing (overlay<->overlay dominates) |
||
|
Drew T
|
b06d74c1be |
feat(phase-11): T2 — dedup.us.yaml registry + dedup_integrate byte-honesty validator
- config/dedup.us.yaml: cross-binary code-share registry (group -> {id,tier,hash,
source,func,members[{binary,vram,name}]}); empty for now (T3 adds the first group)
- tools/dedup_integrate.py: --check validator. Fail-closed if a shared function's
current sig hash drifted from the recorded hash (a stale share can never silently
mislead, P9/G3); unsigned binary -> WARN (unvalidated), not a hard fail
- Makefile: dedup-check is the fail-closed last line of `make report` (gated BINARY=main)
- DEVIATION D1: game-code dedup is SOURCE-LEVEL (shared body in src/shared/<fn>.h,
instantiated per site), NOT a psyq_integrate object-swap — game funcs are interior to
one object per binary so the linker can't swap them; the byte-gate is the existing
per-binary make check. .ld interpose stays the library mechanism (Phase 8)
- verified: --check passes on empty registry; make report runs it; clean rebuild main
-> 143dbb89 BYTE-IDENTICAL (T2 is a proven build no-op, R22)
|
||
|
Drew T
|
7cc2911456 |
feat(phase-11): T1 — cross-binary dup_report --cross → duplicates.cross.md
- dup_report.py: additive --cross mode ingests all BINARIES sigs, tags each row with its binary, buckets by h_exact then h_norm across binaries, splits cross-binary (binaries>1, the Phase-12/13 work queue) vs intra-binary, ranks by collapsible bytes (count-1)*nins*4, one row per member -> docs/duplicates.cross.md - Makefile: make report emits the cross report once (gated BINARY=main) - per-binary duplicates.md/.resident.md stay byte-stable (per-binary path unchanged) - EXE<->resident cross-binary groups = 0 (4.0 vs 4.7 + different roles, as the byte-finding predicted); real cross groups arrive with overlays (T6) - phase-ends/CURRENT_PHASE.md: Phase 11 plan + per-task log |
||
|
Drew T
|
03978ec4ba |
feat(phase-10): T4 — resident Ghidra program (2nd binary) + symbol seed + sig
- tools/ghidra_import_raw.sh (NEW): raw-blob importer (BinaryLoader + --loader-baseAddr + PSX:LE:32:default) — the Gen2 counterpart to ghidra_import.sh (PS-X-EXE only); reusable for Phase-13 location overlays. Imports the resident blob as program 'resident' @0x800CEDF8. - tools/ghidra_scripts/DefineFunctions.java (NEW): seed splat's validated entry points (.run/<prog>_funcs.txt) — raw-binary auto-analysis finds only the reachable subset (23/143); this defines all 143 (created=120/existed=23/failed=0). R9-verified 143 funcs persisted. - FINDING: DetectPsyQ reports the resident is PsyQ 4.7.0 (the EXE is 4.0.0); the lone in-range PsyQ-signature hit is DsMix (libsnd -> resident holds the sound driver). 4.7 .LIBs DEFERRED to Phase 11 start (Drew); carried to PhaseEnd Notes. Phase 10 needs nothing from 4.7. - config/symbols.resident.txt: seed DsMix (R13 candidate, 4.0-sig vs 4.7 blob — confirm in Phase 11); stacked under symbols.us.txt, applied on re-extract -> resident still 8e17e02f BYTE-IDENTICAL (R22), main still 143dbb89 (no regression) - Makefile: per-binary GHIDRA_PROG -> 'make sig-refresh BINARY=resident' (.run/sig.resident.jsonl) - dup_report.resident now real (6 byte-identical intra-resident groups — Phase-11 dedup fodder) - ghidra DB committed (R23, MCP stopped): new resident program 00000003.* (+ main db.15->16 no-op) |
||
|
Drew T
|
cc73951128 |
feat(phase-10): T3 — per-binary reports/diff/expected for the resident binary
- diff_settings.py + progress.py/difficulty.py/dup_report.py: add the `resident` BINARIES entry (build/resident/, config/check.resident.sha, src/resident, asm/resident/nonmatchings, per-binary docs) - dup_report.py: degrade gracefully when the Ghidra sig (.run/sig.resident.jsonl) is absent — write a placeholder + exit 0 instead of crashing the whole `make report` (sig is a T4 export) - progress.py: scope linked_subsegs() to BINARY==main — PsyQ library linking is the EXE's layout (Phase 8, gated ifeq BINARY,main), so a second binary has 0 LINKED (was: listed main's libs) - Makefile expected: per-binary-safe — drop `rm -rf expected/build` (clobbered every sibling baseline); refresh only the active binary's image dir + merge-copy (cp never deletes siblings) - reports: resident = 0 REAL / 143 INCLUDE_ASM stubs / 100% stub (correct); main UNCHANGED (52 REAL / 959 LINKED / 50.24%); both expected/ baselines coexist (verified) |
||
|
Drew T
|
95ade84768 |
feat(phase-10): T1 — scaffold the resident second binary (Makefile no-op on main)
- Makefile: BINARIES += resident; resident_* var block (vram 0x800CEDF8, build/resident/, config/*.resident.*); per-binary ASM_DIR/SRC_DIR + $(BINARIES)-derived OBJS prune-list so main (asm/+src/) and resident (asm/resident/+src/resident/) object sets stay disjoint; per-binary UNDEF_SYMS/UNDEF_FUNCS (resident writes under build/resident/) - config/splat.resident.yaml: flat blob config (NO header, NO gp_value, single c seg @ vram 0x800CEDF8, stacked symbols [symbols.us.txt, symbols.resident.txt], per-binary asm/src/build/undefined paths, EOF 0x5935C -> end vram 0x80128154) - config/check.resident.sha (8e17e02f… = extracted MAIN.CD/FILE_010/1.1, 365,404 B, type-1) - config/symbols.resident.txt (R13/R15 provenance banner; seeded empty, populated in T4) - gates: (a) main clean rebuild 143dbb89… (refactor is a no-op on the byte-locked EXE); (b) make extract BINARY=resident -> 144 INCLUDE_ASM stubs + .ld @0x800CEDF8; (c) main OBJS=81 excludes resident even with resident on disk + main stays 143dbb89… - src/resident/*.c left WIP-untracked (reshaped by T2 boundary carving) |
||
|
Drew T
|
5e0d80a887 |
refactor(phase-9): T7 — ld_interleave.py --front/--tail; gate extract call to BINARY=main
- ld_interleave.py: argparse [ld] + --front/--tail (repeatable; default to the EXE's sandwich objects, transitional); the .data->.rodata->.data island is EXE-specific - Makefile extract: ld_interleave call wrapped in ifeq(BINARY,main) (overlays have no rodata island) and passes --front 53198.data.o --tail 6324C.data.o explicitly - GATES: positive extract+build -> 143dbb89 (tail_data=1 from the threaded flag); negative --tail BOGUS.data.o -> 'tail data object not found' exit=1 (threaded); restore -> 143dbb89 |
||
|
Drew T
|
22b20e8f31 |
refactor(phase-9): T6 — --binary selector for dup_report + difficulty; report threads BINARY
- dup_report.py + difficulty.py: argparse --binary (default main) + BINARIES table (main = current sig/src/asm/md/csv paths); overlay src/asm subtree deferred to P10 - Makefile report target threads --binary $(BINARY) to progress/difficulty/dup_report - no-op proof: docs/progress.md + docs/duplicates.md UNCHANGED; difficulty.py old-code vs new-code output IDENTICAL (pure path-table refactor) - docs/difficulty.md regenerated (the Phase-8-deferred regen: 9 harvested funcs left the unmatched queue — 12/12 row shift); build path untouched (still 143dbb89) |
||
|
Drew T
|
06676e6ab1 |
refactor(phase-9): T5 — psyq_integrate flags + progress regex + 9 Makefile call sites
The build-path linchpin (one commit — integrate signature and the progress.py Makefile regex must move in lockstep): - psyq_integrate.py: argparse (--vram-base/--exe/--symbols flags BEFORE the 4 positionals + nargs=* window); threads vram_base/exe into placement/classify and symbols_path into the stub-name->addr resolution (was hardcoded symbols.us.txt) - Makefile: all 9 psyq_integrate calls pass --vram-base $(main_VRAM_BASE) --exe $(main_EXE) --symbols $(main_SYMBOLS) (flags before positionals) - progress.py: stub-list regex gains (?:\s+--\S+\s+\S+)* to consume the leading flags; + a --binary selector (BINARIES table, default main = current paths) - GATES: positive build -> 143dbb89; report 52/959/7/50.24% (regex still finds the 9 stub lists); negative main_VRAM_BASE=0x8000F804 -> cae22f7e (NOT target); without-SDK fresh-clone fallback -> 143dbb89 |
||
|
Drew T
|
170586d6bf |
refactor(phase-9): T1 — data-driven BINARIES Makefile (no-op on EXE)
- BINARIES/BINARY guard + main_* var set + selected-binary aliases; the EXE's artifact paths (build/us/, config/splat.us.exe.yaml, check.us.sha) preserved verbatim so its rebuild stays a byte-exact no-op - 9 PsyQ SDK-integration blocks wrapped in ifeq(BINARY,main); SYMS/LD/OBJCOPY stay outside (run for every binary); a 2nd binary skips the SDK block - gate: make clean && extract && build -> 143dbb89 BYTE-IDENTICAL; make report 52/959/7/50.24%; BINARY=bogus errors via the $(filter) guard |