mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
main
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
db63fcc673 | tools(phase-35): T6 part 2 — the dead macro branches dropped from 12 live tools; the macro-form guard in tool_census --check (ast scan of every LIVE tool, FROZEN/retired/6 whitelisted detectors excluded; 0 LIVE tools reference the retired form; --guard-root negative control: the pre-T4 tree flags exactly the 12 fixed + 3 retired); progress.py: an empty-bodied included member is EMPTY, not folded into REAL (the R32 bucket assertion caught the double count) — the fleet denominator 363,221 -> 363,680 (+459: SC03_015's 219 single-site macros, SC03_118 +3, the never-extended members of five groups) and REAL 360,744 -> 350,533 with EMPTY up the same; progress.json + README block + fleet table regenerated; SETUP; kit corpus | ||
|
Drew T
|
6cb056c932 | tools(phase-35): T2 — the health chain learns the include-at-site and twin forms while the macro form still builds: one source-dir oracle (corpus.src_dirs/src_dir/twin_of from the Makefile), share_census.header_defs as the one reader of the include form, progress.classify counts include sites (fold keeps 255,632), overlay_src_split's include item kind + loud macro-table refusal, dedup_integrate C2a' (a header must DEFINE func), audit_binaries prelude + twin citizenship (CHECK 3b), cdecl differential not-applicable when the macro header is gone, recursive shared-header globs; the count correction 363,214 -> 363,221 (7 whale include sites never registered; ov_MAIN_012 2,401 -> 2,402) regenerated into progress.json + README | ||
|
Drew T
|
cd768d717a | tools(phase-34): cdecl --audit — create its scratch dir and refuse (never misgrade) when the gcc adjudication cannot run; the prune had removed .run/audit and every unparsed draft line read as a DEFECT | ||
|
Drew T
|
26ba449684 |
perf(tools-health): parallelise the sig targets; fix a latent probe-file race; MEASURE the real cost
Drew asked why `make tools-health` runs 15+ min. Measured per step rather than
guessed (I guessed wrong twice first, and both are recorded in the comments):
sig-overlays ~52s serial -> 3.9s wall / 51.8s user (xargs -P$(JOBS), 32 cores)
sig-modules 0s sig-resident 0s audit-corpus 17s
audit-cdecl >9 MINUTES <-- the actual bottleneck, and NOT the gcc probes:
the `[gcc] N distinct declarations` line never printed inside a
10-minute run, so not one cc1 call had happened. `tu_statements`
over 4,168 TUs is ~787s single-core, all of it before the probes.
SHIPPED
* sig-overlays / sig-modules: xargs -P$(JOBS), same pattern extract-all and
check-all already use in this file. sig_image has exactly one write path
(its own per-alias .jsonl), verified before fanning out. NEGATIVE CONTROL:
141/141 sig files BYTE-IDENTICAL to the serial output. Also adds the failure
detection the serial loops never had -- a sig_image crash used to vanish (R32).
* cdecl._gcc_probe: `probe_{tag}.c` was ONE FIXED FILENAME PER TAG, correct only
while _sift is serial. Now unique per call, so concurrent probes cannot
overwrite each other's source between write and compile and return a verdict
about another chunk's declarations.
* cdecl._sift: threads over chunks + over the bisection probes (gcc is a
subprocess, so the GIL is released), results written back BY INDEX so the
output stays deterministic. A/B on --limit 6: IDENTICAL verdicts (829/829).
NOT SHIPPED, and the measurement is left in the code
A ProcessPoolExecutor over the collection phase was tried and REVERTED: 12 TUs
yield 32,352 statements, so the full pass ships ~11M strings through IPC and the
pickling costs more than the parse it saves. The fix is to dedupe/filter INSIDE
the worker or memoise per-TU by content hash -- left measured, not guessed.
|
||
|
Drew T
|
a0f07d629e |
chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/ rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java + VerifyOverlay.java (ghidra_import_raw.sh is the live path) - reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green - SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol- window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21); disc-completeness Reproduce marked retired |
||
|
Drew T
|
f142a6858f |
feat(phase-29): ov_SC06_018 crack-wave — func_801365B8 x138 (fresh-exemplar sweep CONFIRMED), thesis is family-specific
- binary-aware crack wave (new tools/workflows/wave_binary.js): 8-target calibration over ov_SC06_018 substantial stubs, 7/8 match_one MATCH - func_801365B8 (155, reach 133): cracked FRESH in ov_SC06_018, swept 132/132 siblings via family_sweep --hseq --source ov_SC06_018 --allow-pins -- SESSION-10 refused this family 0/133 from an ov077 exemplar. THESIS CONFIRMED (fresh exemplar unlocks it). - func_80133AB0 (137, reach 137): cracked fresh + banked x1 (+ a byte-neutral s17a-1 cast reconcile of banked caller func_801343C4), but the family sweep FAILED 0/136 even from the fresh exemplar (reverted clean) -- THESIS REFUTED for this family. - FINDING (R14/R31 -> decision-log): the fresh-exemplar sweep is FAMILY-SPECIFIC, not a blanket mechanical x137. A fresh crack is necessary but not sufficient; the byte-gate arbitrates each family (~50% on this 2-family sample -> discount the ~1.5pp estimate). - tooling (R33): family_sweep --source override now searches matched_members (a fresh member leaves 'members' after a sig-regen); cdecl._depth0_spans consumes backslash line-continuations so a raw-draft #define macro no longer trips audit-cdecl. - R22 clean-fleet 140/140 byte-identical; tools-health green (dedup 1849/0, C1 234615); 0 NON_MATCHING. fleet 78.4->78.5% instr / 67.1->67.5% distinct / 88.14->88.18% fn-count. |
||
|
Drew T
|
e0a0becfaa |
feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr
The plan named two defective regexes; the tree had SIX with complementary holes, each silently recording the resulting compile failure as "not a match" — a plumbing error wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32). - cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs; keeps draft-local types. lru_cached. - harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) -> unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1 stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… / CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer recorded byte-identically to a codegen miss. - masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33, cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one + p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over whitespace. Unblocks B4's func_8015C32C (redefinition of 's16'). - canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap). VERIFIED: - HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that was being discarded over whitespace). - unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped. - classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly. - all 5 edited tools import + AST-parse clean. - R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248 "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.) - SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the original asm; a wrong draft always changes bytes -> always fails SHA1). |
||
|
Drew T
|
f4502f11bb |
fix(phase-26a): A3c — the recovery passes were reconciling 95% of drafts against the WRONG TU
FIRST CONSUMER MIGRATION onto the cdecl oracle — and the compiler taught me two things I had
wrong, one of which reopens a wall that has been closed since Phase 15.
1. cdecl.compatible() — "will cc1 accept these two declarations of one name?"
The predicate four tools each half-implement and get wrong: norm_sig / _norm_type collapse the
int family to ONE token, so a SIGNEDNESS change reads as "already compatible" and gets no
rewrite -- while cc1 REJECTS that redeclaration. Right about codegen, wrong about the front end,
which never reaches codegen.
2. THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE (cookbook §51g LAW 9).
I wrote the rules from the C standard, then let a compiler judge. It contradicted me -- and then
the RIGHT compiler contradicted the first one. Three different answers:
declarations in one TU | standard | modern gcc | gcc-2.7.2 cc1
typedef int X; twice | error | ACCEPTS | ERROR
extern u16 X; + volatile u16 X| error | error | ACCEPTS
void X(s16); then void X(); | error | error | ACCEPTS
void X(); then void X(s16)| error | error | ERROR
--compat now adjudicates with tools/bin/gcc-2.7.2-psx/cc1, the front end that actually
arbitrates the build: 1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.
3. THE PRIZE: the Phase-15 narrow-param wall rests on a false premise.
The no-prototype rule is ORDER-DEPENDENT. `void X(s16); void X();` COMPILES; only the reverse
fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as "no clean deterministic fix --
it is simply C's default-promotion rule". cc1 does not enforce that rule in the direction the
wall assumed. Four three-line probes, 90 seconds, zero tokens. -> A10 RE-TEST TARGET.
Probe the compiler for FACTS; read its source only for LEVERS; byte-validate both. (We read
gcc-papermario for five phases believing it was 2.7.2. It was 2.8.1.)
4. THE MIGRATION: cast_call_sites canonicalized 95.1% of drafts against a TU that would never
compile them. `--src-file` is an OPTIONAL HAND-PASSED flag defaulting to src/<ov>/<ov>.c, and no
caller knows about the Phase-26 _jr_<ADDR> carves: ov_SC01_077 has 263 open stubs across 12 TUs
and only 13 are in the main .c -- while harvest_verify (A3) correctly splices into the real one.
Now DERIVED from corpus.stubs() (the INCLUDE_ASM line is self-describing), with the canonical map
derived from cdecl.tu_scope() (cpp -- so macro-injected DEFINE_func_* decls are finally visible).
Callee-conflict repair reach: 8 -> 58 of 196 drafts (7x).
5. AND THE NULL RESULT, REPORTED AS SUCH (P9/R14). Those 58 banked ZERO functions. The historical
draft tail fails on CODEGEN, not plumbing -- func_801387B8, which the audit blames on a single
unparsed `[4]`, is really 67/100 instructions off with a $s0/$s1 swap (that claim does not
reproduce on today's tree). The real gain is narrower and still worth having: 52 drafts moved
from "won't compile" to "compiles, N instructions off" -- from an INVISIBLE failure that reads as
a compiler wall into a SCORED near-miss the permuter and the §47/§48 dials can act on. That is
the audit's thesis, not a bank. THREE times in one session a confirmed mechanism produced a null
consequence.
Also: my own new audit printed "ALL ORACLES GREEN" while silently skipping 100% of its corpus (a
missing -Isrc). The exact bug class, in the tool written to hunt it. An unadjudicable check is not
a passed check.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
src/ untouched (0 changes) make audit-cdecl: green --compat: 1485/1485
NEXT: sig_unify + reconcile_decls carry the SAME wrong-TU bug (same --src-file flag).
|
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|