mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
85fb289db582d842fc41dc059fa187bb992e76ea
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
043532b475 |
feat(phase-30 S48-T6): wave 4 — 18 exemplars banked; + a 4th comment-blindness fix
Wave 4 (wf_05895a19-121, 75 agents, 7.9M tok): 40 targets -> 35 agent-MATCH,
0 refuted, 5 NEAR, 0 FAIL, 0 drafts lost. 18 banked so far on the whole-binary
gate across 9 binaries; the other 17 are re-gating on a clean tree (see below).
PRIOR-NOTES SEEDING HELD AT SCALE: 10 of 12 seeded targets confirmed (wave 3
was 7 of 9). func_8017C294 — the x16 family, the largest single item on the
board — is now NEAR at **2 ins** (18 -> 11 -> 2 across three seeded attempts).
THE 4th COMMENT-BLINDNESS DEFECT OF THE SESSION, and the first with blast
radius. A crack agent annotated a decl in its draft:
extern void func_801842DC(s32 a0); /* TU:4023 INCLUDE_ASM (no decl) */
`corpus._INCLUDE_ASM_CAND` only skips lines that BEGIN with a comment marker,
so it read `INCLUDE_ASM (` out of the trailing PROSE, found no quoted path, and
refused the whole binary's stub oracle — correctly, by its own R32 contract.
That then failed gate_stage for every LATER binary in the run, because they all
walk the corpus: 12 binaries banked, then 5 were blocked by one comment in a
13th. Fixed the same way as the other three today: decide candidacy on
cdecl._mask'ed text, PARSE FROM THE ORIGINAL (the mask blanks string content
and would erase the asm path). Verified on 4 binaries incl. main (2,002 stubs).
AND THE DAMAGE IT LEFT: gate_stage raised out of the CorpusError BEFORE its
revert, stranding failed drafts spliced in src/. The 17 solo re-gates that
followed all read 0/17 — they were building a POISONED TREE, not judging their
own drafts. Residue reverted here; the 17 re-gate clean next.
The pattern is now sharp enough to state: any scanner that greps C source for a
token must mask comments and strings FIRST — and agent-authored drafts make it
far likelier, because their prose mentions the exact tokens our tools hunt for.
|
||
|
Drew T
|
fa7b9d4c71 |
feat(phase-30 S45 L3): the emulator tour — all 28 script modules + MAIN/3 onboarded; fleet 212, R22 212/212
- THE TOUR (Drew driving the retail debug menu; mode-7 hammer over the Redux web API): all 28 script modules captured live at four byte-verified per-chapter slots (SC03/73-79 @0x801EF468 ch2-period, SC03/132-138 @0x801E25E8 ch3, SC04/24-30 @0x801E7B28, SC05/23-29 @0x801ED988); the routing law: debug-menu AREA selects the chapter, each CITY interior streams its own module (member k <-> interior k). md_MAIN_011/DISELECT byte-proven 24,236/24,240 in RAM; slots A/B/boot R34-verified live. - MAIN/3 DISCOVERED: the main-menu module (id 0x39, 121,884 B), mis-bucketed as data by BOTH audit oracles; live byte-proven @0x800CEDF8 (42,632-B exact prefix); onboarded. - 29 onboardings BYTE-IDENTICAL on first build -> fleet 212; R22 212/212 after three md_MAIN_003 catches: the A4 DsMix leak; an extract-order-sensitive splat boundary (bytes: a 1-word data sentinel in .text + fn at +4 -> pinned in symbols file); corpus.stubs now treats D_*/jtbl_* INCLUDE_ASM as blob includes (mirrors progress.py) - module-id census (offline, disc-wide): 77 id-law code payloads, 0 further misses; SC03/55 = confirmed DATA. audit-disc: UNCLAIMED 34 -> 6, residue 0 — the 6 carry byte-checked negative evidence; next tier = the CD-read tracer - docs: memory-map §S45 (slots + routing + debug-menu ops), disc-completeness S45 addendum, decision-log R31 entry, docs/debug-menu-list.txt (Drew's transcription) - .run/s45 evidence allowlisted (tour logs/scripts/rosters); 104 ram dumps LOCAL-ONLY - new baseline: 93.8% instr / 95.68% fn / 87.2% distinct over 212 |
||
|
Drew T
|
4cadac4e11 |
feat(phase-30 S45 II.1c): module batch dedup-banked + verified — 408 banks, R22 183/183, audit-disc 75->34 (parked-only)
- dedup measure (R37 probe): 69/1,113 module fns h_exact-match matched corpus (~6%, LOW as
planned — modules are novel frontier); dedup_extend inapplicable (same-vram group model) ->
family_sweep --hseq --band all over the 57 matched-exemplar families: 408 member-matches
banked (182 into modules, 226 into the big 3 — families Part I's --only scoping missed),
169 failed + 77 STRUCT = genuine per-member frontier
- R22 clean-fleet 183/183 BYTE-IDENTICAL; audit-disc UNCLAIMED 75->34 residue 0 (34 = 31
parked-for-L3 + SC03/53,54,56 — 3 rows Discovery-3 never tiered, now parked with evidence)
- three instrument fixes, each negative-control-proven:
- family_sweep --hseq stub map derives ov_*+md_*+resident (was sig.ov_* glob -> module
members silently 'not-stub', R32 class) [committed earlier as commit:1506]
- sig-modules seeds from the built ELF's func_* symbols (bootstrap GLUES adjacent fns
around jtbl dispatch -> 24 false TRUNCATED; perturbed-sig control still bites)
- corpus.audit counts CODE lines only (module .s carries its header jtbl as .word lines);
progress.py buckets INCLUDE_RODATA symbols as blobs (unbucketed R32 hole)
- NEW HONEST BASELINE (183 binaries): 94.0% instr / 95.96% fn-count / 87.6% distinct;
tools-health OK, audit-digest OK
|
||
|
Drew T
|
369dd14f4f |
fix(phase-30 S44 I.1d): the module class reaches every enumerating consumer
- family_hseq: widened from src/ov_*+sig.ov_* to every non-main binary (resident + md_*); the map now carries 139 binaries incl. resident (was overlays-only — which is exactly why the R36 gate's CHECK 4 could never see them). Self-count uses the SAME widened globs (cannot drift). - progress --weighted :647 + audit_frontier :57: + sig.md_* globs. - corpus.sig_is_independent: md_* sigs are sig_image-signed => independent (R34 trust). - backlog alias regex + prefetch_fleet (md_* derived from splat configs) + dedup_propagate (reads modules.mk alongside overlays.mk — excluding modules would re-create the SC07 invisible-work bug one class over). - VERIFIED: family map regenerated with resident (139 binaries); audit-binaries OK over 140; all six tools parse. |
||
|
Drew T
|
b3ab5c2253 |
feat(phase-29 Task-13A): deterministic residual classifier — the permuter's problem is TARGETING
The autopsy (hindsight-study §7) assumed the permuter loses for want of a mutation. Measured over the whole open backlog, it loses because it is aimed at work a search-closer provably cannot close. - NEW tools/residual_class.py: decide a near-miss's class FROM THE BYTES. Decodes each mismatching MIPS word -> (op-skeleton, register-fields, immediate); drift FIRST (one inserted insn inflates `closeness` by the tail length), then consistent-injective register map -> REGALLOC-PERM (§31 S11/RC-3), same-multiset-reorder -> SCHEDULE-REORDER, DELAY-SLOT, WIDTH/BRANCH-POLARITY/STRENGTH/ADDRESSING/IMM-OFFSET/IMM-VALUE. Every class routes to a BUCKET = which tool the failure wants. Uncovered opcode -> UNKNOWN, COUNTED (R32). 16 synthetic unit tests (test_residual_class.py). - NEW tools/autopsy.py: `collect` materialises the corpus Task-12's telemetry never filled (1 of 6,169 records had a residual) by recompiling every open draft through the EXISTING match_one path (R33) — 1,752 drafts in 21s at -j12. `report` -> docs/autopsy.md. - NEW corpus.o0_sources()/is_o0(): the opt-level oracle DERIVED from the Makefile's own -O0 rules, coverage-asserted. Scoring an -O0 target at -O2 makes the residual 100% artefact (the trap this phase hit four times). - R34 cross-check baked in: residual_class's closeness vs masked_diff.structured_diff's, asserted per row; 1,673/1,673 agree, 0 classifier errors. FINDING: of the 972 records the grinder's own filter admits, only 75 (7.7%) are permuter-shaped; 547 are structural and 348 are drafts that are not the function at all. ~92% of the daemon's CPU went where it could not win — the byte-grounded explanation of "7 banks all-time, all Phase 21, 0 since" (Phase-22 audit). grinder.candidates() now filters on the measured bucket (1,303 -> 78) and takes its directed profile from the measured class, not the logged label (91% carry none -> it ran on gcc defaults). Degrades to undirected if uncollected and says so; --no-targeting A/Bs it. Two measured corollaries (R14, not projections): - 699 records rank as near-misses at closeness up to 278 purely from a length artefact: un-attempted work misfiled as a backlog of hard functions -> new `redraft` bucket. - a 12-draft gate probe of the `integration` bucket banked 1/12 (11 PLUMBING), so the 306 prices Task 14's reconcile ladder rather than promising free banks. func_80167714 (104 ins, reach-134) banked x1, un-propagated by design (§55b). Two defects fixed forward: - masked_diff._common_typedefs() used ONE shared probe path, so parallel match_one processes clobbered each other: 14 of 1,752 drafts lost in a single 12-way run (0.8%), silently, in every parallel wave ever run. Now per-PID. - gate_stage.match_one_closeness never passed --o0 -> phantom residuals for every -O0 function, written straight into the backlog this autopsy reads. R22 clean-fleet: check-all 140 passed, 0 failed of 140; tools-health OK (dedup 1847/0, C1 234343/234343); 0 NON_MATCHING (G4). Flywheel captured in-session (R30/R31): cookbook §60, decision-log entry, SETUP.md inventory. |
||
|
Drew T
|
427baba3bf |
feat(phase-27 T10): completion dashboard (main in the weighted metric) + the resident second oracle
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
|
||
|
Drew T
|
f7b7399ebe |
feat(phase-26a): A3 — tools/corpus.py, ONE derived corpus oracle (+ a second oracle that can disagree)
The 28 surviving audit findings collapse to ONE bug repeated ~10 times: a hand-maintained model of
the corpus layout (a file allowlist, a single-.c assumption, a func_-only symbol regex, a REGION_SUB
dict) sitting on top of a filesystem that already answers the question. The fix is not ten repaired
regexes — it is one DERIVED oracle and ten deleted scanners (R33).
WHAT IT DERIVES FROM
1. THE FILESYSTEM. Which .c files make up a binary, and where a function's .s lives, are FACTS OF
THE TREE THAT SPLAT ITSELF WROTE. The INCLUDE_ASM line is SELF-DESCRIBING — its first argument
IS the asm subdir — so there is nothing to guess and no dict to rot. A dict literal is strictly
worse than the filesystem AND it fails OPEN (silently yields a wrong path) instead of closed.
2. THE PROVEN INVARIANT. INCLUDE_ASM pastes the ORIGINAL asm and the build is byte-identical, so a
function NOT wrapped in it is byte-exact. `matched` is DERIVED as sig - stubs, never re-parsed
from C text. (progress.py learned this the hard way: weighted_metrics() derived and was right;
classify() re-parsed C and inherited a bug.)
VALIDATED against the real corpus:
* ov_SC01_077: 264 stubs across 14 files. The old 3-file allowlist saw 30.
* Fleet: 58,717 stubs vs the allowlist's 1,992 — 56,725 (96.6%) were INVISIBLE.
* Coverage-asserted (R32): every INCLUDE_ASM line must parse, every symbol must resolve (ANY C
identifier — a func_-only regex silently misses the 100 curated listCdBuffer stubs), every stub
must have a .s. A silent skip is a DEFECT, not a no-op.
THE SECOND ORACLE (`make audit-corpus`) — the real lesson of this audit.
The byte-gate is structurally BLIND to a bad function boundary: the .s halves are pasted back
verbatim in original order, so the image stays byte-identical and green. Only an oracle that can
DISAGREE can see it. sig_image is that oracle — Ghidra-free, derived from the ORIGINAL bytes,
independent of splat. corpus.audit() cross-checks the two and reports:
PHANTOM — a stub address the sig does not know: splat INVENTED a function.
TRUNCATED — a stub whose .s length != the sig's: splat MIS-SLICED one.
It reports 193 (96 + 97) — reproducing the A2 audit's number EXACTLY, from an independently written
tool. That is a third confirmation of the listCdBuffer defect (auditor -> skeptic -> this).
AND AN R14 SELF-CATCH, recorded because the near-miss is the lesson.
Run naively over all 136 binaries the same check reports 914 slices — 4.7x the truth. It is noise:
main/resident are signed by the GHIDRA dumper, whose boundaries are shorter than splat's by design
(and which never analysed the linked PsyQ subsegs at all), so the comparison measures GHIDRA'S limits,
not splat's errors. Only the overlays are signed by sig_image, the oracle actually validated at
58,524/58,621. sig_is_independent() now encodes that domain, with the reasoning, so nobody repeats it.
A check applied outside its valid domain does not become more thorough — it becomes noise.
`make audit-corpus` is RED by design until A4 removes the bad symbol line; then it becomes a gate.
|