mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 14:02:04 -04:00
docs(phase-26a): A3 progress — the oracle is built; 4 of ~10 scanners deleted
corpus.py + build_fuel_manifest + wave_targets + harvest_verify/gate_stage landed. Remaining: family_manifest/family_hseq (62% of the endgame plan is phantom targets), DELETE census_conflict_callees (R33), exemplar_miner/difficulty, jr_isolate_all.
This commit is contained in:
@@ -66,7 +66,13 @@ The Phase-25 h_seq reframe: the "unique tail" is really per-location families
|
||||
**(1) The Phase-22 verdict "the permuter's fuel is exhausted" is UNSAFE.** `grinder` banks through `harvest_verify`, which sees one TU — **1,290 of 1,298 (99%) of the grinder's own queue could never have banked**, however good the permuter's output was. "0 banks since Phase 21" is equally consistent with *the tool could not bank*. **Re-test before repeating it (A12).**
|
||||
**(2) The Phase-25/26 endgame plan is majority-fiction.** `docs/family-manifest.md` advertises "2,758 multi-member families / 11.0 MB of hidden leverage." **1,071 of them / 6.80 MB are ALREADY FULLY MATCHED** — phantom targets, **62% of the advertised byte-weight** — because the matched-set oracle scans ONE overlay. The byte-weight *ranking*, the entire point of the file, is sorted mostly on dead work. True frontier: 1,475 families / 3.9 MB.
|
||||
**(3) A corpus defect the byte-gate is structurally BLIND to.** `config/symbols.us.txt:981` declares `listCdBuffer = 0x80180000` (a main-EXE **data** symbol) and every overlay's splat config loads that file — but in overlay space 0x80180000 is **code**. splat cuts **97 real functions in half** and invents **96 phantom ones**: **193 slices that can never be matched by anyone**, in 97 of 134 overlays. They sit in the harvest queue as ordinary work, so agents burn on them forever and the failure reads as an intrinsic compiler wall — **and the build stays byte-identical and green throughout**, because the `.s` halves are pasted back verbatim. *A perfect correctness oracle and a null coverage oracle.* What saved us: **`sig_image` was RIGHT** (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements). **A second independent oracle is the only reason this was visible at all.**
|
||||
- [ ] *(superseded — see A3/A4/A5 below, restructured around the root cause)* `jtbl_family_bank` (3 bugs found by hand, never audited) · the SELECTION tools `family_hseq`/`family_manifest`/`wave_targets`/`exemplar_miner`/`worklist`/`build_fuel_manifest` · the closeness oracle `masked_diff`/`match_one` that every crack agent trusts · the bank/gate path `gate_stage`/`harvest_verify`/`bank_exemplar` · `dup_report`/`difficulty`/`sig_image`/`canon_draft_decls`/`derive_canonical_sigs`/`census_conflict_callees`. **METHOD (do NOT audit by reading regexes — that is the failure mode that WROTE these bugs):** build a deliberately **over-approximating** candidate detector, run both over the real corpus, compute `gap = candidates − parsed`, and **classify every item in the gap** as a real silent skip or a justified exclusion. Pair each auditor with an **adversarial skeptic** told to refute it (in the first audit several magnitudes were wrong in *both* directions and one whole class was refuted). **R33 before R32.**
|
||||
- [~] **A3 — `tools/corpus.py`: ONE derived corpus oracle** `[Max]` — **IN PROGRESS. The oracle is built and 4 of ~10 scanners are deleted.**
|
||||
**`tools/corpus.py` (`commit:0589`)** — derives from the FILESYSTEM (which `.c` files make up a binary; where a function's `.s` lives — the `INCLUDE_ASM` line is **self-describing**, its first argument *is* the asm subdir) and from the PROVEN INVARIANT (`matched` = `sig − stubs`, *derived*, never re-parsed from C text). Coverage-asserted (R32): every `INCLUDE_ASM` line must parse, every symbol must resolve (**any** C identifier — a `func_`-only regex silently misses the 100 curated `listCdBuffer` stubs), every stub must have a `.s`.
|
||||
**The SECOND ORACLE (`make audit-corpus`)** — cross-checks splat's boundaries against `sig_image`'s independent ones. Reports **193** (96 PHANTOM + 97 TRUNCATED), **reproducing the A2 audit's number exactly from an independently-written tool** — a third confirmation. *RED by design until A4; then it becomes a gate.*
|
||||
**R14 self-catch, recorded because the near-miss IS the lesson:** run naively over all 136 binaries the same check reports **914** slices — 4.7× the truth. `main`/`resident` are signed by the **Ghidra** dumper (shorter boundaries by design; never analysed the linked PsyQ subsegs), so the comparison measures *Ghidra's* limits, not splat's errors. Only the overlays are signed by `sig_image`. `sig_is_independent()` now encodes that domain. **A check applied outside its valid domain does not become more thorough — it becomes noise.**
|
||||
**`build_fuel_manifest` + `wave_targets` (`commit:0590`)** — targets **30 → 263**; reach-134 targets **10 → 127** (the entire high-ROI band); remaining gain **83,305 → 994,633 ins** — *the audit's predicted figure to the unit* (a **fourth** independent confirmation). `wave_targets`' asm paths: **0 of 263 missing** (was 78 of 87 pointing at nothing, whose wasted attempts were booked as *matching* failures and fed `reserved_walls()` — a silent skip compounding into a **false wall**).
|
||||
**`harvest_verify` + `gate_stage` (`commit:0591`) — the byte-gate itself.** It could only see ONE TU: **4.9% → 100%** of ov_SC01_077 reachable; **96.6%** of fleet stubs were unreachable. *Safety argued explicitly:* only the SPLICE LOCATION is derived — the verdict (`make build` + SHA1) is untouched, and since `INCLUDE_ASM` pastes the original asm, a bug can make it **fail to bank but never falsely bank**. Verified by an **identity known-answer test** across three different split TUs → `d19c9580` BYTE-IDENTICAL, tree clean.
|
||||
**STILL TO DO in A3:** `family_manifest`/`family_hseq` (the matched-set oracle — **1,071 of 2,758 advertised families are already fully matched**, 62% of the endgame plan's byte-weight is phantom) · **DELETE `census_conflict_callees`** (R33 — `reconcile_tu` answers its question from the build) · `exemplar_miner`/`difficulty` (inherited holes) · `jr_isolate_all.jr_inventory` (reads its banked-roster from an **ephemeral gitignored scratch file**). `jtbl_family_bank` (3 bugs found by hand, never audited) · the SELECTION tools `family_hseq`/`family_manifest`/`wave_targets`/`exemplar_miner`/`worklist`/`build_fuel_manifest` · the closeness oracle `masked_diff`/`match_one` that every crack agent trusts · the bank/gate path `gate_stage`/`harvest_verify`/`bank_exemplar` · `dup_report`/`difficulty`/`sig_image`/`canon_draft_decls`/`derive_canonical_sigs`/`census_conflict_callees`. **METHOD (do NOT audit by reading regexes — that is the failure mode that WROTE these bugs):** build a deliberately **over-approximating** candidate detector, run both over the real corpus, compute `gap = candidates − parsed`, and **classify every item in the gap** as a real silent skip or a justified exclusion. Pair each auditor with an **adversarial skeptic** told to refute it (in the first audit several magnitudes were wrong in *both* directions and one whole class was refuted). **R33 before R32.**
|
||||
- [ ] **A3 — `tools/cdecl.py`: ONE coverage-asserting C-decl parser** `[Max]` *(blocked on A2)* — the same char-class disease (`[\w\s\*]` cannot hold `(`, `,`, `[N]`) is **independently re-implemented in 6+ scanners**; two tools in ONE pipeline already disagree about what a data decl *is*. Patching 6 regexes = 6 chances to diverge again. **R33:** one statement-oriented splitter + a **TOTAL** classifier (scalar/ptr/`[N]`/fn-ptr/fn-ptr-array/multi-declarator/wrapped/trailing-comment) with a built-in coverage assertion, consumed by all of them. Migrate consumers one at a time, each byte-gated. Kills ~15 of the 40 findings at the root.
|
||||
- [ ] **A4 — Wire `reconcile_tu.py`; retire the fleet-majority oracle** `[Max]` *(blocked on A2)* — `reconcile_decls.canonical_data_map` asks *"what does the FLEET call this symbol"* when C's constraint is **per-TU**: 36.8% of symbols carry ≥2 conflicting spellings, so one fleet-wide answer is **wrong for some TU by construction** — it returns an **actively WRONG** canon for **3,717** symbols and disagrees with the TU's own decl on **21.7%** of pairs (**every one of 678 TUs** has ≥1 broken answer). `reconcile_tu.py` is **written + validated + NOT WIRED** (`commit:0580`). Wire → `bank_exemplar` / `jtbl_family_bank` / `gate_stage`; byte-gate. Unblocks `func_8017A4AC` (287 KB), `func_8013F350`, `func_80131340`.
|
||||
- [ ] **A5 — Family engine: 93 phantom exemplars, 1,834 free members** `[Max]` *(blocked on A2)* — `family_remap.extract_unit` globs only `src/<ov>/<ov>*.c`, so it **cannot read a `DEFINE_func_*` body out of `engine_core.h`** → **93 of 218 (43%)** "matched" exemplars are PHANTOM and **1,834** still-stubbed, PURE/IMM-clean, unpinned member templates are **never attempted**. Plus `symbol_map` naming an address-taken **function** `D_<addr>` (3 families × 37 members silently no-op, booked as *byte* failures), `gather_externs`' multi-line externs (9), `family_sweep.stub_map`'s curated-name stubs (3 phantom families). Fix, assert, **re-run the sweep, harvest.**
|
||||
|
||||
Reference in New Issue
Block a user