mirror/BFM-decomp
1
0
Fork 0
mirror of https://github.com/Druthulu/BFM-decomp synced 2026-09-26 13:33:34 -04:00
Code Issues Packages Projects Releases Wiki Activity
4,051 Commits 1 Branch 0 Tags
583ec85d6e9fe56f5d4100b00a4fd27ad2b0680a
Commit Graph

6 Commits

Author SHA1 Message Date
Drew T 26ba449684 perf(tools-health): parallelise the sig targets; fix a latent probe-file race; MEASURE the real cost 
Drew asked why `make tools-health` runs 15+ min. Measured per step rather than
guessed (I guessed wrong twice first, and both are recorded in the comments):

  sig-overlays  ~52s serial  -> 3.9s wall / 51.8s user  (xargs -P$(JOBS), 32 cores)
  sig-modules   0s   sig-resident 0s   audit-corpus 17s
  audit-cdecl   >9 MINUTES  <-- the actual bottleneck, and NOT the gcc probes:
                the `[gcc] N distinct declarations` line never printed inside a
                10-minute run, so not one cc1 call had happened. `tu_statements`
                over 4,168 TUs is ~787s single-core, all of it before the probes.

SHIPPED
  * sig-overlays / sig-modules: xargs -P$(JOBS), same pattern extract-all and
    check-all already use in this file. sig_image has exactly one write path
    (its own per-alias .jsonl), verified before fanning out. NEGATIVE CONTROL:
    141/141 sig files BYTE-IDENTICAL to the serial output. Also adds the failure
    detection the serial loops never had -- a sig_image crash used to vanish (R32).
  * cdecl._gcc_probe: `probe_{tag}.c` was ONE FIXED FILENAME PER TAG, correct only
    while _sift is serial. Now unique per call, so concurrent probes cannot
    overwrite each other's source between write and compile and return a verdict
    about another chunk's declarations.
  * cdecl._sift: threads over chunks + over the bisection probes (gcc is a
    subprocess, so the GIL is released), results written back BY INDEX so the
    output stays deterministic. A/B on --limit 6: IDENTICAL verdicts (829/829).

NOT SHIPPED, and the measurement is left in the code
  A ProcessPoolExecutor over the collection phase was tried and REVERTED: 12 TUs
  yield 32,352 statements, so the full pass ships ~11M strings through IPC and the
  pickling costs more than the parse it saves. The fix is to dedupe/filter INSIDE
  the worker or memoise per-TU by content hash -- left measured, not guessed.
 2026-09-01 20:10:51 -06:00
Drew T a0f07d629e chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe 
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py
  (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the
  differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/
  rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java +
  VerifyOverlay.java (ghidra_import_raw.sh is the live path)
- reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs
  annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green
- SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol-
  window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21);
  disc-completeness Reproduce marked retired
 2026-08-06 13:34:50 -06:00
Drew T f142a6858f feat(phase-29): ov_SC06_018 crack-wave — func_801365B8 x138 (fresh-exemplar sweep CONFIRMED), thesis is family-specific 
- binary-aware crack wave (new tools/workflows/wave_binary.js): 8-target calibration
  over ov_SC06_018 substantial stubs, 7/8 match_one MATCH
- func_801365B8 (155, reach 133): cracked FRESH in ov_SC06_018, swept 132/132 siblings
  via family_sweep --hseq --source ov_SC06_018 --allow-pins -- SESSION-10 refused this
  family 0/133 from an ov077 exemplar. THESIS CONFIRMED (fresh exemplar unlocks it).
- func_80133AB0 (137, reach 137): cracked fresh + banked x1 (+ a byte-neutral s17a-1
  cast reconcile of banked caller func_801343C4), but the family sweep FAILED 0/136 even
  from the fresh exemplar (reverted clean) -- THESIS REFUTED for this family.
- FINDING (R14/R31 -> decision-log): the fresh-exemplar sweep is FAMILY-SPECIFIC, not a
  blanket mechanical x137. A fresh crack is necessary but not sufficient; the byte-gate
  arbitrates each family (~50% on this 2-family sample -> discount the ~1.5pp estimate).
- tooling (R33): family_sweep --source override now searches matched_members (a fresh
  member leaves 'members' after a sig-regen); cdecl._depth0_spans consumes backslash
  line-continuations so a raw-draft #define macro no longer trips audit-cdecl.
- R22 clean-fleet 140/140 byte-identical; tools-health green (dedup 1849/0, C1 234615);
  0 NON_MATCHING. fleet 78.4->78.5% instr / 67.1->67.5% distinct / 88.14->88.18% fn-count.
 2026-07-22 18:12:09 -06:00
Drew T e0a0becfaa feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr 
The plan named two defective regexes; the tree had SIX with complementary holes, each
silently recording the resulting compile failure as "not a match" — a plumbing error
wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32).

- cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs
  (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts
  -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits
  multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs;
  keeps draft-local types. lru_cached.
- harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) ->
  unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1
  stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… /
  CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer
  recorded byte-identically to a codegen miss.
- masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33,
  cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one +
  p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over
  whitespace. Unblocks B4's func_8015C32C (redefinition of 's16').
- canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate
  per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap).

VERIFIED:
- HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was
  CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that
  was being discarded over whitespace).
- unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped.
- classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly.
- all 5 edited tools import + AST-parse clean.
- R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248
  "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean
  rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.)
- SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the
  original asm; a wrong draft always changes bytes -> always fails SHA1).
 2026-07-15 18:11:43 -06:00
Drew T f4502f11bb fix(phase-26a): A3c — the recovery passes were reconciling 95% of drafts against the WRONG TU 
FIRST CONSUMER MIGRATION onto the cdecl oracle — and the compiler taught me two things I had
wrong, one of which reopens a wall that has been closed since Phase 15.

1. cdecl.compatible() — "will cc1 accept these two declarations of one name?"
   The predicate four tools each half-implement and get wrong: norm_sig / _norm_type collapse the
   int family to ONE token, so a SIGNEDNESS change reads as "already compatible" and gets no
   rewrite -- while cc1 REJECTS that redeclaration. Right about codegen, wrong about the front end,
   which never reaches codegen.

2. THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE (cookbook §51g LAW 9).
   I wrote the rules from the C standard, then let a compiler judge. It contradicted me -- and then
   the RIGHT compiler contradicted the first one. Three different answers:

       declarations in one TU        | standard | modern gcc | gcc-2.7.2 cc1
       typedef int X;  twice         | error    | ACCEPTS    | ERROR
       extern u16 X; + volatile u16 X| error    | error      | ACCEPTS
       void X(s16);  then  void X(); | error    | error      | ACCEPTS
       void X();     then  void X(s16)| error   | error      | ERROR

   --compat now adjudicates with tools/bin/gcc-2.7.2-psx/cc1, the front end that actually
   arbitrates the build: 1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.

3. THE PRIZE: the Phase-15 narrow-param wall rests on a false premise.
   The no-prototype rule is ORDER-DEPENDENT. `void X(s16); void X();` COMPILES; only the reverse
   fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as "no clean deterministic fix --
   it is simply C's default-promotion rule". cc1 does not enforce that rule in the direction the
   wall assumed. Four three-line probes, 90 seconds, zero tokens. -> A10 RE-TEST TARGET.
   Probe the compiler for FACTS; read its source only for LEVERS; byte-validate both. (We read
   gcc-papermario for five phases believing it was 2.7.2. It was 2.8.1.)

4. THE MIGRATION: cast_call_sites canonicalized 95.1% of drafts against a TU that would never
   compile them. `--src-file` is an OPTIONAL HAND-PASSED flag defaulting to src/<ov>/<ov>.c, and no
   caller knows about the Phase-26 _jr_<ADDR> carves: ov_SC01_077 has 263 open stubs across 12 TUs
   and only 13 are in the main .c -- while harvest_verify (A3) correctly splices into the real one.
   Now DERIVED from corpus.stubs() (the INCLUDE_ASM line is self-describing), with the canonical map
   derived from cdecl.tu_scope() (cpp -- so macro-injected DEFINE_func_* decls are finally visible).
   Callee-conflict repair reach: 8 -> 58 of 196 drafts (7x).

5. AND THE NULL RESULT, REPORTED AS SUCH (P9/R14). Those 58 banked ZERO functions. The historical
   draft tail fails on CODEGEN, not plumbing -- func_801387B8, which the audit blames on a single
   unparsed `[4]`, is really 67/100 instructions off with a $s0/$s1 swap (that claim does not
   reproduce on today's tree). The real gain is narrower and still worth having: 52 drafts moved
   from "won't compile" to "compiles, N instructions off" -- from an INVISIBLE failure that reads as
   a compiler wall into a SCORED near-miss the permuter and the §47/§48 dials can act on. That is
   the audit's thesis, not a bank. THREE times in one session a confirmed mechanism produced a null
   consequence.

Also: my own new audit printed "ALL ORACLES GREEN" while silently skipping 100% of its corpus (a
missing -Isrc). The exact bug class, in the tool written to hunt it. An unadjudicable check is not
a passed check.

  R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
  src/ untouched (0 changes)   make audit-cdecl: green   --compat: 1485/1485
  NEXT: sig_unify + reconcile_decls carry the SAME wrong-TU bug (same --src-file flag).
 2026-07-14 12:26:01 -06:00
Drew T f9742cf9c0 feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models 
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
    extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).

REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)

Two statement paths, because the inputs genuinely differ:
  * tu_statements()    - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
                         macro body declares NOTHING until the macro is invoked (the §8c law);
                         a raw scan is wrong in both directions. cpp answers it exactly, in
                         54 ms/TU (~20 s for the fleet, cacheable).
  * split_statements() - span-preserving raw split, for drafts (which get rewritten).

THREE ORACLES, whole corpus — a measurement, not a belief:
  * coverage      2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
  * the real gcc  50,405 distinct declarations compiled beside this parser's reconstruction
                  of each one -> 0 REJECTED
  * differential  0 file-scope symbols the incumbents see that cdecl misses; 26 in
                  engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope

Two ideas worth keeping (cookbook §51g, LAWS 4-8):
  * THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
    declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
    by the grammar, with no hand-maintained candidate regex to rot.
  * GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
    grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
    rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
    not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.

NEW findings (docs/tooling-audit.md):
  * reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
    and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
  * gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
    function body) as file-scope canonicals — the §8d `conflicting types` confusion.
  * tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
  * R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
    by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
    today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
    cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.

Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.

SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.

  R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
  make audit-corpus: 0 PHANTOM + 0 TRUNCATED    make audit-cdecl: ALL ORACLES GREEN (new gate)
 2026-07-14 11:37:13 -06:00