mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
main
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
048666600b |
fix(oracle): masked_diff kept the opcode at R_MIPS_26 slots — a masked j/jal was swallowing whatever the other side held
mask_for(reloc_kind='26') returned 0, i.e. 'compare NOTHING at this position'. Both comparers pick the mask from ONE side (diff_object_s from mine, diff_object_object from the target's), so a j/jal there masked the OTHER side's instruction entirely. Reproduced on synthetic pairs of real encodings: my 'j 8017e248' (0805f892) vs target 'bne v0,v1' (14430002) -> 0; vs 'nop' (00000000) -> 0; my 'jal' vs target 'bne' -> 0; while the mirror (my 'bne' vs target 'j') -> 1. That asymmetry is the bug. _j_mismatch cannot cover it: it fires only when BOTH sides carry an internal-j target, which a j-vs-bne pair by definition does not. Fix: return 0xFC000000 — the 26-bit target field stays masked (it IS link-time), the opcode never is. diff_object_object's masked-slot test updated to match so the reloc symbol+addend check still fires there. R39 negative control (tools/stub_invariant_audit.py, the INCLUDE_ASM invariant): 2554 stubs, nonzero 3 before and 3 after — the same three known main length-delta survivors, same values. Zero new false positives, over a population that exercises the changed path (812 stubs carry internal-j .text relocs, 3164 such instructions). Found by a t5s drafting agent on func_8017EB30 (reported as a one-sided internal-j check); verified here to be broader than reported. No bank was ever at risk — the whole-binary gate is independent (G3) — but every crack agent and the permuter scorer read this number. R35/R14. |
||
|
Drew T
|
3a5caa65b1 | feat(tools)+docs(phase-31): T1 — resolver declfix wiring; masked_diff/rtu_match compare internal-j targets (jrel; §195 premise refuted, cookbook §301); diff_autopsy.sh + stub_invariant_audit.py promoted; frontier-s61 Addendum 2 (step-1 falsifier fired: 54 carve-refused, 2 DIFF autopsied+banked, 1 plumbing); S62 log | ||
|
Drew T
|
0d39cd97e1 |
feat(phase-31): S54 wave-V harvest — cookbook §195 (14 laws) + close the internal-j blind spot in masked_diff
32 agents over wave V's 67 index_gap reports: 14 CONFIRMED, 9 REJECTED, 76 already-covered (the third harvest this session; readers seeded with §193 AND §194 so neither could be re-derived). THE HEADLINE IS A DEFECT IN OUR OWN VERIFIER (§195-D). masked_diff.mask_for short-circuited on the OPCODE -- `if (word >> 26) in (2, 3): return 0` -- ahead of the reloc dispatch, so every `j` to a LOCAL label was dropped from the comparison. The assembler resolves those itself and emits no relocation, so there was nothing link-time about them. For a loop or switch arm, which label a `j` targets is the difference between `break` (fall into the shared tail and execute its calls) and `return` (skip them): byte-proven on ov_SC03_118:func_801825EC, where the banked `break;` and a call-skipping `return;` variant differ in exactly one word (0800003e vs 08000041) and BOTH reported MATCH. The blindness reached match_one, the permuter's MaskedScorer, family_cousins.tok and the atlas similarity tiers at once -- nothing between a draft and the whole-binary gate could see it. Now the 26-bit field is masked only when reloc_kind == "26" (the linker really does fill it). R39 control: 35/35 already-banked wave-V drafts still MATCH against their snapshot .s. Other confirmations of note: §195-A bounds §167-08 with a byte-proven false-negative class (an argument that DIES at the call is allocated straight into $aN, so its only def is a plain load and every use reads $aN -- there is no positive tell in either direction, only the two-arity A/B); §195-B a CALL_INSN does not start a basic block in gcc-2.7.2, so a call-crossing temp can be a LOCAL-alloc quantity; §195-M frame `vars` is a sequential bump-allocation, unifying §193-I's CEIL(aggregate,8) term with §165-03/§167-06's 8x-orphan term as one frame_offset walk. |
||
|
Drew T
|
d806766eef |
feat(phase-30 S47-W1c): crack the reach-57 exemplar; fix the instrument that called it a wall
func_801EDC18 (md_SC05_023) is the largest multiplier remaining — 57 members. The wave agent
abandoned it at "closeness 6" with class SIZE-MISMATCH [redraft]. It was two lines from correct.
THE CODE (cookbook §160a): the target copies 8 bytes with lwl/lwr + swl/swr — gcc-2.7.2's
emit_block_move for a type with ALIGNMENT 1. The draft used a u32 copy (aligned lw/sw), which is
wrong by construction. `typedef struct { char c[8]; } Blk8; buffer = D_801ED98C;` reproduces it.
Six spellings were tried in parallel; two independent agents converged on the same one.
THE INSTRUMENT (§160b) — this is the part worth more than the function. The target .s bundles a
leading `.section .rodata` block (D_801ED98C as two .word) ahead of .text. Those lines carry the
same `/* off vaddr HEX */` shape as instructions, so masked_diff.insns_from_s counted them as TARGET
instructions, while insns_from_object (objdump -j .text) can never emit them. A byte-perfect draft
therefore read `mine=26, target=28, 26 mismatched` — every position shifted by a constant +2 — and
got classified as needing a redraft. 116 of 12,583 .s files in the corpus have this shape, one at
-29 instructions. Every one of them would report a false wall to any agent that tried it.
Fixed: insns_from_s tracks .section and counts only .text. Full-corpus control: 12,467 unchanged,
116 corrected, 0 regressions. Same artifact class as §129a (post-carve jtbl inflation).
THE OWNERSHIP LAW (§160c) — my own error, corrected by the gate. Four sites declare
`extern short D_801ED98C;` and nothing in src/ defines it, so I shipped an extern-only draft. The
gate refuted it: `undefined reference`. The .s block the draft REPLACED was the definition. The
variant emitting `const Blk8 D_801ED98C = {{...}}` banks clean. Never infer ownership from externs.
R22 clean-fleet: check-all 213 passed / 0 failed of 213.
ALSO BANKED — the wave's idiom harvest, which had been sitting unwritten in workflow transcripts
(R16/R30 debt): §160d the ASYMMETRIC INDEX RELOAD (a just-stored narrow field read twice emits
reuse-then-reload; the C is deliberately asymmetric), §160e a stack-layout scheduling rule now
byte-proven on a SECOND independent function (promoting it from coincidence to rule), §160f the
address-only global store via array decl, §160g sibling-search keyed on the CALLEE SET as step 0 of
every wave prompt (one grep turned a 126-instruction crack into a copy-edit).
Cookbook index regenerated: 468 sections.
|
||
|
Drew T
|
b3ab5c2253 |
feat(phase-29 Task-13A): deterministic residual classifier — the permuter's problem is TARGETING
The autopsy (hindsight-study §7) assumed the permuter loses for want of a mutation. Measured over the whole open backlog, it loses because it is aimed at work a search-closer provably cannot close. - NEW tools/residual_class.py: decide a near-miss's class FROM THE BYTES. Decodes each mismatching MIPS word -> (op-skeleton, register-fields, immediate); drift FIRST (one inserted insn inflates `closeness` by the tail length), then consistent-injective register map -> REGALLOC-PERM (§31 S11/RC-3), same-multiset-reorder -> SCHEDULE-REORDER, DELAY-SLOT, WIDTH/BRANCH-POLARITY/STRENGTH/ADDRESSING/IMM-OFFSET/IMM-VALUE. Every class routes to a BUCKET = which tool the failure wants. Uncovered opcode -> UNKNOWN, COUNTED (R32). 16 synthetic unit tests (test_residual_class.py). - NEW tools/autopsy.py: `collect` materialises the corpus Task-12's telemetry never filled (1 of 6,169 records had a residual) by recompiling every open draft through the EXISTING match_one path (R33) — 1,752 drafts in 21s at -j12. `report` -> docs/autopsy.md. - NEW corpus.o0_sources()/is_o0(): the opt-level oracle DERIVED from the Makefile's own -O0 rules, coverage-asserted. Scoring an -O0 target at -O2 makes the residual 100% artefact (the trap this phase hit four times). - R34 cross-check baked in: residual_class's closeness vs masked_diff.structured_diff's, asserted per row; 1,673/1,673 agree, 0 classifier errors. FINDING: of the 972 records the grinder's own filter admits, only 75 (7.7%) are permuter-shaped; 547 are structural and 348 are drafts that are not the function at all. ~92% of the daemon's CPU went where it could not win — the byte-grounded explanation of "7 banks all-time, all Phase 21, 0 since" (Phase-22 audit). grinder.candidates() now filters on the measured bucket (1,303 -> 78) and takes its directed profile from the measured class, not the logged label (91% carry none -> it ran on gcc defaults). Degrades to undirected if uncollected and says so; --no-targeting A/Bs it. Two measured corollaries (R14, not projections): - 699 records rank as near-misses at closeness up to 278 purely from a length artefact: un-attempted work misfiled as a backlog of hard functions -> new `redraft` bucket. - a 12-draft gate probe of the `integration` bucket banked 1/12 (11 PLUMBING), so the 306 prices Task 14's reconcile ladder rather than promising free banks. func_80167714 (104 ins, reach-134) banked x1, un-propagated by design (§55b). Two defects fixed forward: - masked_diff._common_typedefs() used ONE shared probe path, so parallel match_one processes clobbered each other: 14 of 1,752 drafts lost in a single 12-way run (0.8%), silently, in every parallel wave ever run. Now per-PID. - gate_stage.match_one_closeness never passed --o0 -> phantom residuals for every -O0 function, written straight into the backlog this autopsy reads. R22 clean-fleet: check-all 140 passed, 0 failed of 140; tools-health OK (dedup 1847/0, C1 234343/234343); 0 NON_MATCHING (G4). Flywheel captured in-session (R30/R31): cookbook §60, decision-log entry, SETUP.md inventory. |
||
|
Drew T
|
eec96f76a4 |
feat(phase-29 Task-12): structured failure telemetry + durable permuter-winner save
The permuter-autopsy prerequisite (Drew-directed). Captures the WHAT/WHERE of every
non-match so the classifier/autopsy (Task-13) can mine it, not just a scalar closeness:
- masked_diff.structured_diff(): the per-instruction masked residual [(idx,mine,tgt),...]
- match_one --json: emits {status,closeness,nins,residual} (refactored to share the primitive)
- backlog FIELDS += residual, passes_tried
- gate_stage.match_one_closeness returns + logs the residual on every near/fail (verified
end-to-end: a near-miss's structured residual now lands in backlog.jsonl)
- grinder: durably save the winning C to .run/permuter-winners/ BEFORE gate_stage banks
(this session's lesson — 3 permuter wins were lost to a mid-flight revert)
|
||
|
Drew T
|
e0a0becfaa |
feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr
The plan named two defective regexes; the tree had SIX with complementary holes, each silently recording the resulting compile failure as "not a match" — a plumbing error wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32). - cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs; keeps draft-local types. lru_cached. - harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) -> unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1 stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… / CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer recorded byte-identically to a codegen miss. - masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33, cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one + p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over whitespace. Unblocks B4's func_8015C32C (redefinition of 's16'). - canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap). VERIFIED: - HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that was being discarded over whitespace). - unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped. - classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly. - all 5 edited tools import + AST-parse clean. - R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248 "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.) - SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the original asm; a wrong draft always changes bytes -> always fails SHA1). |
||
|
Drew T
|
af2f40d153 |
fix(phase-26a): A4/A5 — 193 unmatchable slices dissolved; the closeness oracle stops lying
R22 CLEAN-FLEET: make clean -> extract 136 -> build 136 -> check-all = 136 PASSED, 0 FAILED.
make audit-corpus: 0 PHANTOM + 0 TRUNCATED (was 193).
=== A4: a CORPUS defect the byte-gate could never have caught ===
config/symbols.us.txt:981 declared `listCdBuffer = 0x80180000` — a correct Phase-3 name for MAIN's
LIST.CD RAM buffer. But that address is OUTSIDE main's image and INSIDE the overlay slot, and every
overlay's splat config stacks symbols.us.txt. High RAM is REUSED: an address that is a buffer to main
is live CODE to an overlay. So splat saw a symbol boundary mid-code and, across 97 of 134 overlays:
* CUT 97 REAL FUNCTIONS IN HALF (a head ending on a `lui`, no return), and
* INVENTED 96 PHANTOM ONES (a tail beginning by reading the assembler temp $at).
193 slices NOBODY COULD EVER MATCH — not "hard", not "a compiler wall": unmatchable by construction.
They sat in the harvest queue as ordinary work, so agents would burn on them forever and the failures
would be filed as intrinsic compiler residuals.
The phantom listCdBuffer.s in ov_SC01_005 literally begins:
lw $ra, 0x10($sp) / addiu $sp, $sp, 0x18 / jr $ra
splat cut a function immediately before its EPILOGUE and called the epilogue a function.
AND IT HAD ALREADY CONTAMINATED REAL WORK: in ov_SC03_031 the cut landed where the epilogue was
exactly `jr $ra; nop`, so the Phase-26 x134 sweep innocently BANKED the phantom as
`void listCdBuffer(void) {}` — byte-correct, gate-green, entirely fictitious — while leaving
func_8017FFC4 permanently unmatchable. Removed.
WHY NO GATE CAUGHT IT, AND WHY THAT IS THE POINT: INCLUDE_ASM pastes the two .s halves back VERBATIM
in original order, so the image is byte-identical either way. The byte-gate was green the whole time
and always would have been. It is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle. No
assertion added INSIDE it could ever have found this. What found it was a SECOND, INDEPENDENT oracle:
tools/sig_image.py derives boundaries from the ORIGINAL bytes without splat, and DISAGREED with the
corpus (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements).
=> When one oracle is structurally blind to a class of error, the answer is not a better assertion
inside it. It is a SECOND ORACLE THAT CAN DISAGREE WITH IT. (`make audit-corpus` is now that.)
THE RULE (the mirror of R13/R15, never written down): a symbol whose address falls inside ANOTHER
binary's vram window must never enter that binary's symbol stack.
FIX: config/symbols.us.ram.txt — main-scoped symbols outside main's image — stacked ONLY by
config/splat.us.exe.yaml. Main keeps the name it needs (10 %hi / 11 %lo refs; 143dbb89 byte-identical);
the overlays never see it. Exactly one symbol was in scope fleet-wide; the resident window was clean.
AND A REAL FUNCTION THE ACCIDENT WAS HIDING: in ov_SC01_084 / ov_SC02_041 / ov_SC03_094 / ov_SC06_008
there IS a genuine function at 0x80180000 (111 / 35 / 28 / 74 ins), reachable ONLY via a fn-pointer
table (.word func_80180000) and never by `jal` — so splat cannot find it and needs the boundary
DECLARED. listCdBuffer had been supplying it by luck. Now declared honestly, per-overlay, in
config/symbols.<ov>.txt — exactly where R13/R15 says an overlay-scoped symbol belongs.
=== A5: the closeness oracle every crack agent trusts was lying on 155 functions ===
masked_diff._reloc_kind() knew 26/HI16/LO16. An over-approximating sweep of every reloc objdump emits
across all 3,367 build objects found FOUR: R_MIPS_26, HI16, LO16 — and R_MIPS_PC16 (211). PC16 fell
through to a FULL-WORD compare, but the object holds an UNRESOLVED PLACEHOLDER in the branch
displacement, so that compare can NEVER succeed.
DECISIVE TEST (derived from the invariant, not from reading the regex): INCLUDE_ASM pastes the
ORIGINAL asm, so for every stub diff_object_s() MUST be 0. Measured, coverage-asserted:
2,741 functions scored — old mask: 150 LIES; PC16 masked: 4 LIES.
(The 4 survivors are the separate length-delta defect.) A phantom non-zero sends an agent to grind at
a wall that is not there, and the wasted attempt is then booked as a MATCHING failure, feeding
reserved_walls() and PERMANENTLY BLACKLISTING a function that was never broken.
=== NEW FINDING (found by cutting the R22 corner): a STALE OBJECT CAN PRODUCE A FALSE PASS ===
`.o <- .s` is not a dependency make can see: assembly arrives via INCLUDE_ASM, expanded to a `.include`
consumed by maspsx/as AFTER cpp, while -MMD tracks headers only. Re-extract, build incrementally, and
make links a STALE object. This is not merely slow — INCLUDE_ASM pastes the ORIGINAL bytes, so a stale
object still yields the original image: SHA1 GOES GREEN while the split just changed is never exercised.
A broken config change can be "verified" by an incremental build. Live proof: 8 of 136 binaries linked
stale objects here; they failed LOUDLY ONLY BY LUCK (the dead symbol was an undefined reference) — a
merely-different-but-valid split would have gone green on all 136.
R22/H3 already legislate this, and I broke them. But a rule that needs a human to remember it is not a
gate. FIX: `extract` now invalidates the objects that include what it just rewrote (main's are top-level,
so -maxdepth 1 — verified it cannot clobber the other 1,605 objects). Structural, not advisory.
R14 self-catch, recorded: my first A5 test passed `fn=` to diff_object_s(), which takes two args; the
TypeError was swallowed by my own `except Exception: continue` and it reported 0 scored / 0 lies. I
wrote the exact bug I was auditing, inside the test for it. Caught only because 0 looked wrong. The
test now asserts its own coverage.
|
||
|
Drew T
|
5984749421 |
feat(phase-24): T2 — floor-free relocation-masked permuter scorer (-drz)
- tools/masked_diff.py: shared objdump -drz masking oracle (jal/j 26-bit + HI16/LO16 immediate mask; -z keeps nop runs -> no GTE under-count). object-vs-object (permuter, +reloc-operand equality) and object-vs-.s (match_one) modes. - match_one.py refactored onto masked_diff (-dr -> -drz); regression-clean on count-exact seeds (35/52/72/110, leaf-MATCHes), output format unchanged (gate_stage/grinder compatible) - tools/masked_scorer.py MaskedScorer: drop-in for decomp-permuter's Scorer; scores masked .text closeness (bottoms out at 0) not the stock mnemonic-diff floor - tools/permuter/run_masked.py: rebinds src.main.Scorer -> MaskedScorer (NO submodule edit, R3/R20); p16_permute.run_permuter wired to it - VALIDATED: masked_self=0; masked_cand~match_one; STOCK floor 1750-2100 vs masked 36-77 (the wander cause); live permuter base score = masked 77 (not stock 1930), descends to 74 - R14 SELF-CORRECTION: -drz confirms func_80132784 is 4/400 count-exact (T1's -dr 204 was a 16-nop-collapse artifact); backlog re-logged close 4. The flagship IS 4 ins away. - no build-input changed (136/136 untouched); 2 compile FAILs surfaced -> T3 (GTE asm, typedefs) |