mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-06 09:06:04 -04:00
583ec85d6e9fe56f5d4100b00a4fd27ad2b0680a
26 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
cb948a6bbc |
feat(decomp): the ov_SC01 reloc-only cluster + its 5th latent victim — 5 fns, 1,301 ins
S74 handed this forward as "1,116 instructions behind one question": family_remap
on ov_SC01_004/005/006/008 gated DIFF 4/4 against the banked exemplar
ov_SC01_009:func_8017EB08, and the class had been carried as a codegen wall since
S70. The four bodies were byte-identical to the exemplar the entire time.
Word-level classification vs the exemplar, computed independently twice (a Fable
agent's script, then mine from scratch against the retail images), identical:
nins=279 EQ 213 · RELOC-HI16 23 · RELOC-LO16 24 · INTERNAL-J 19 · CODEGEN 0
Zero register-allocation, instruction-selection or scheduling differences.
ROOT CAUSE — tools/jtbl_carve.py reserved ONE WORD TOO MANY per table:
* spimdisasm runs an island's LAST `jtbl_` dlabel one word into the following
NON-ZERO data (string bytes 0x696F760A / 0x000013FF / 0x62647020), so the
zero-word trim cannot see it; and
* the over-span clamp that would have caught it was guarded by
`len(sltiu_bounds) == 1` -- but `sltiu` is ALSO how gcc emits an unsigned
range check ((u32)(x-lo) < n, I1). These four carry five distinct sltiu
immediates, so the guard silently disabled itself on precisely the functions
that needed it.
0x2C reserved for a 0x28 table => image 4 bytes short => ~850 %lo immediates
shift => whole-binary DIFF about a function whose own bytes are perfect.
Fixed with a PER-TABLE bound: gcc-2.7.2's dispatch is a fixed idiom, so the
`sltiu` nearest ABOVE that table's own %hi(jtbl_X) is unambiguous whatever else
the function tests. Second defect stacked behind it: a carve span whose
JTBL_PADS line lacks a `tables=` comment lost its existing table's start on
merge and refused "table starts do not fit the span" -- which harvest_verify
then "repaired" with a needless jr_isolate_all that walked back into the first.
THE NEGATIVE CONTROL IS THE STORY. Run over every other open table-bearing stub
fleet-wide, the fixed bound changed exactly one more table: ov_SC06_022/
func_80185B80 (185 ins), a FIFTH victim nobody had drafted against. A guard that
disables itself on a common idiom does not fail once -- it fails quietly across
the whole corpus.
Banked, each with its own byte-gate verdict (--no-propagate, clean re-gate):
func_8017EB30 ov_SC01_004 279
func_8017F2D4 ov_SC01_005 279
func_8017F2D4 ov_SC01_006 279
func_8017EC68 ov_SC01_008 279
func_80185B80 ov_SC06_022 185
Also here:
* dedup_propagate: memoize find_site's mask (lru_cache) -- 54 ms of masking
per call over the whole source, recomputed though it depends only on the
text. 2x on that loop (58.3 -> 33.0 ms/call), NC identical on 120 addrs.
Scoped honestly: that loop is ~2.4 min of a 30-min run; the profiler puts
43% in family_remap._alias_decl_for, which is NOT fixed here.
* Makefile: `clean` says out loud that BINARY= is ignored and it is fleet-wide
(cookbook §445) -- it silently deleted asm/ for all 213 binaries this session.
* Cookbook §446 (the carve law: when a standalone-MATCH jtbl draft gates DIFF,
diff the carve extent against 4 x sltiu before touching the body), §445, and
SETUP rows for both tools (R21).
* CURRENT_PHASE: the S75 log, incl. the measured fleet dedup-hygiene census
(~2,073 fns / ~12,116 items, all ALREADY MATCHED -- cleanup, not work) and
Drew's decision to leave it and gate --no-propagate from here.
|
||
|
Drew T
|
ed53a68f18 |
feat(p31 s68): deferred propagation done honestly (2 banked) + seed_ref was offering DEAD TEXT
The S67 FINAL-3 OPEN item, plus the two defects found while doing it. * fix(dedup_propagate): the tool could not run AT ALL. S67's -j patch wrote `os.environ` at module level in the one module that imports `os as _os`, so every invocation died with NameError before doing any work. Propagation was not deferred, it was impossible. Import-checked the other 7 -j-patched tools. * propagation, honestly scoped: the real closable set is 11, not 32, derived two independent ways that agree (seed_ref exact+same_addr, and a direct corpus derivation). The 3,161-entry --auto-from plan over 53 overlays is dedup hygiene over already-matched code and closes almost no open stub. Applied: 2 banked byte-green (ov_SC04_018 func_80181270, func_80182AF8); 3 gate-refused and cleanly reverted; 6 blocked with named blockers (3 CARRY-FIXABLE, 3 func_80144B9C not-inline-def -> needs the o0 whale carve). R22 clean fleet: extract 212/212, check 213 passed 0 failed of 213, rc 0/0/0. Frontier 453 -> 451. * fix(seed_ref): REFUSE targets in LINKED subsegs. The playbook calls this tool "the fleet-wide answer" and it reported 82 open stubs with a banked twin -- 43 of them main stubs whose TUs the linker script never references. Any C written there compiles, links and leaves the SHA1 green WHETHER OR NOT IT IS CORRECT, so a mechanical twin lane fed from that list could have minted up to 43 gate-green FALSE matches the byte gate cannot see. draw_waves has refused these since S66; this oracle did not. The refusal is counted and printed, not silent. NC: guarded 39 subset of raw 82, all 43 dropped are main, the non-main population is identical. * wave drawn: .run/S68o1 (24 opus 187-770 ins) + .run/S68m1 (30 main), cards + packs + wave_args asserted, queue of 53. Drafting opened at concurrency 5. |
||
|
Drew T
|
675b4702b3 |
perf(gate): pass -j to the per-binary build — 6.1x on the inner loop of every gate
MEASURED on ov_SC03_010 (35 objects), clean each time, byte-verified against the locked SHA:
make build 7.18 s real / 6.84 s user <- SERIAL, one core, on a 32-thread box
make -j16 build 1.18 s real / 11.3 s user <- 6.1x, IDENTICAL bytes
Negative control at -j32 over ov_SC03_010 + ov_SC01_004 + md_MAIN_031: all rc=0, all byte-identical
to config/check.<bin>.sha.
WHY IT WAS MISSED: the Makefile's `JOBS ?= 16` is parallelism ACROSS binaries (`xargs -P`), which
parallel_gate already uses for extract-all/check-all. Parallelism WITHIN one binary's ~35 objects was
never passed by any tool, though docs/SETUP.md:416 documents `make -j$(nproc) build` as the form.
PATCHED the two hot sites:
* harvest_verify.py — the gate's build, run ONCE PER DRAFT (--chunk 1). Every gate in the project.
* dedup_propagate.py byte_gate — run once per propagation candidate, which is why a wide
propagation dominated a 33-minute gate this session.
Both honour BFM_BUILD_JOBS, else os.cpu_count().
SAFE BY CONSTRUCTION: these builds feed a locked-SHA comparison, so a bad parallel build FAILS the
gate rather than banking wrong bytes. The error direction is a false NEGATIVE, never a false bank;
G3/P9 remains the sole arbiter.
Correction recorded: I earlier extrapolated "9 serial binaries x 30 min" from ONE 33-minute
measurement. That was unfounded — propagation time scales with how many sites a body reaches, and
other gates today propagated x19/x8/x7 quickly. One slow binary is not a rate (R41).
|
||
|
Drew T
|
f6e48b60c5 |
perf(phase-30 S46-4): parallelise the propagation — 24min -> 11.4min, and +62 MORE instances
Drew: "make it more multi-threaded... I still see my cpu idle for far too long." Measured, fixed, and regression-tested against the S46-3 bank as a KNOWN ANSWER. - THE MEASUREMENT: 31s saturated (33 makes/48 cc1/load 27) then ~25s with ONE build alive while 31 cores idled, repeating. Causes: ex.map starts in list order so the giants land last, and apply/restore is single-threaded. - gate_all -> gate_failures: return EVERY failure the sweep already computed (~138 rounds -> 1). - Longest-first gate scheduling; results re-sorted into `changed` order so the verdict stays bit-identical to the serial loop's. - PER-OVERLAY INDEPENDENT SEARCH, IN PROCESSES. My first cut used threads and the box refuted it: 0-4 builds alive at load 3, because the work is regex over 15k-line files and 138 "parallel" searches all queued on the GIL. Same logic in a ProcessPoolExecutor: 14-29 builds, load 34.75, search phase ~100s. Safe because the shared header is written ONCE by the parent and each overlay owns its own .c files + build/<bin>/. Seeded with one in-process search first — a pool submitted at once gives every worker an empty suspect list and makes all 138 pay a full bisection. place_in_overlay extracted to module level so the worker and the in-process apply cannot drift (R33); compiles_standalone's fixed t.c is per-call now. - THE REGRESSION (the point, not the stopwatch): revert src/+config to pre-bank, re-run the identical command -> 29 functions (same), 141 overlays byte-identical, 682s vs ~1440s, and 285 exclusions vs ~350 => +62 MORE member instances (249,161). The old prefix-based necessity probe was OVER-EXCLUDING (charging 4 fns to 9 overlays that did not all need them); the per-overlay shrink minimises per overlay. The faster path is also more correct — a timing comparison would never have shown it. R22 213/213 + tools-health green. - STILL SERIAL, now the actual wall-clock (neither is a build): ~3min setup before the first gate (registered_addrs() yaml-parsing a 1949-group/249k-instance registry + 213 sig loads) and ~2.5min of sequential reconcile_caller_extern after the search. - Captured as defaults: docs/accelerators.md A8 + memory fleet-tool-parallelism-defaults. cookbook index regenerated (my §155c append left it stale — the gate caught it, exit 1). |
||
|
Drew T
|
91c64ce92c |
fix(phase-30 S46-1): dedup_propagate — no silent skips, no unproven REVERTs
The S45p9 blocker: `[FAIL] ov_MAIN_012: 0x80156600 not instantiated — REVERTED`
92 minutes into a --auto-from run, naming no mechanism.
- FIRST, the honest finding (R14/R35): it does NOT reproduce at HEAD. A replay of
apply_plan's per-file site resolution over the exact 30-fn plan resolves
0x80156600 as a stub at line 7505, and def-range/stub-line overlaps = 0 (the
splice-swallow hypothesis refuted). The failing input state was not the committed
tree — most likely a concurrent writer mid-run. So this commit does not "fix" that
run; it makes the next occurrence name itself.
- SILENT SKIP -> LOUD (R32): an address resolving as neither the sp-regex stub nor a
def just stayed in `remaining`. apply_plan now records gaps={ov:[addrs]} and the
caller fails FIRST with a per-site diagnosis (whole-overlay find_site verdict,
in-sig, file list) instead of struct_check's terse late message.
- CAPABILITY GAP that produces exactly that skip: find_site returning 'stub' was
ignored (apply_plan acted only on 'def'), so a stub whose INCLUDE_ASM asm-subdir
!= its file stem was invisible to the stem-anchored sp regex AND unhandled. Now
placed ('macro' treated as already-placed). find_site's stub match is an exact
stub_line(ov,addr) compare against THIS file's text — it cannot cross files/TUs.
- INCOMPLETE REVERT (the §156 class, different path): struct_check restored only
`touched`, leaking every kept Part-B reconcile. New _abort() undoes touched AND
every kept reconcile, then diffs the worktree against a start-of-run baseline and
reports any residue. A tree dirty in a way nobody knows about makes every later
byte-gate report `near` — that is how S45p7 lost two batches.
- NEGATIVE CONTROL: neuter ov_MAIN_012's stub -> [GAP] fires naming the exact
condition (find_site=None, in-sig=True) -> "[revert] tree restored to baseline;
no residue" -> exit 1 (fail-closed). Restore -> tree clean.
|
||
|
Drew T
|
e86e45320a |
chore(phase-30 S45p9): session close — 32-way parallel gate in dedup_propagate; banking deferred on a tool bug
PARALLEL GATE (landed, verdict-proven): dedup_propagate's byte-gate loop was serial --
one `make build BINARY=<ov>` at a time over up to 141 members per function. Measured: a
propagation ran 95 minutes at load 1.6 on a 32-core box (~5% utilisation). The Makefile
has parallelised extract-all/check-all since Phase 26 (xargs -P$(JOBS)), but this tool
predates that and drives the SINGLE-binary target from Python, so it never saw any of it.
- new gate_all(): ThreadPoolExecutor over distinct overlays, 32-way by default (JOBS env
overrides; deliberately NOT capped at the Makefile's conservative 16).
- SAFE by the same argument check-all relies on: byte_gate only runs `make build`, writing
solely to per-binary-disjoint build/<bin>/**; it mutates no source. Splice happens before,
restore after -- only the VERIFICATION is parallel.
- DETERMINISTIC: ThreadPoolExecutor.map preserves order, so the reported first failure is
the first in `changed` order -- identical verdict to the serial loop. Control run: same
verdict on a clean tree.
- Measured and NOT optimised: setup (sig load + registered_addrs) is 8.6s of a 5,700s run
= 0.15%. All the time is gating. Don't thread the setup.
BANKING DEFERRED on a genuine pre-existing tool bug (NOT the parallel change -- 0 gate
batches ran, it never reached that code):
[FAIL] ov_MAIN_012: 0x80156600 not instantiated -- REVERTED
Inputs verified sound at HEAD (in sig, find_site->stub, stub line matches), so the bug is
in apply_plan's multi-function edit path. Run #1 missed it because it launched before the
15 wave-3 banks were committed; they landed mid-flight, enlarging run #2's plan.
SECOND DEFECT: the failure exit printed REVERTED but left 38 files dirty incl.
src/shared/engine_core.h -- the same incomplete-restore class as the reconcile-ledger bug
(cookbook 156), on a different path. struct_check needs the same ledger treatment.
Not patching the fleet-shared writer at the end of a marathon session -- that is how the
next 141-binary incident happens. Tree clean, 44 banks safe, propagation is pure
multiplication and can run any time.
Checkpoint p9 carries: the fix-then-resume plan, the master-IDXTAB-map design (DESTPTR half
proven 14/14), wave guidance, and an 8-item error ledger with its single root cause.
|
||
|
Drew T
|
e0eaa16741 |
feat(phase-30 S45p7): Stage 1 complete — shared-state RW lock + 15 more banks (wave-3 revived)
STAGE 1 of docs/concurrency-design.md, landed and negative-control proven.
tools/shared_lock.py (NEW) — one reader/writer flock over the FLEET-SHARED state
(src/shared/*, config/overlays.mk, config/dedup.us.yaml, the overlay .c files
propagation rewrites). Per-binary resources keep gate_stage's existing per-binary flock.
- gate_stage takes it SHARED when the gate writes no shared state, EXCLUSIVE when it
does (propagate, or the arity pre-pass enabled) -- so distinct-binary gates still run
concurrently but can never overlap a writer.
- dedup_propagate and fix_arity_callers --apply take it EXCLUSIVE.
- NESTING-AWARE: gate_stage SPAWNS both writers, so a naive child lock would deadlock
against the parent. The holder exports BFM_SHARED_LOCK_HELD and children inherit.
NEGATIVE CONTROLS (all pass):
NC1 a held SHARED lock refuses a non-blocking exclusive writer, loudly, naming the lock
NC2 parent-holds/child-inherits does NOT deadlock (the real risk in this design)
NC3 two readers acquire concurrently (0.00s) -- phase-B parallelism preserved
bulk_harvest docstring CORRECTED: its "propagation is the ONLY writer of the shared
engine_core.h" claim was FALSE as written and had been asserted for phases (F1 -- the
arity pre-pass writes it from inside every worker). Now states what is actually true,
under which two conditions, plus the one-line assertion that detects a violation.
BANKS: wave-3's drafts re-gated on a CLEAN tree -> 15 of 20 banked. The same drafts
previously reported 0 banked / 20 near -- that verdict was 100% an artifact of the
broken tree, which is why they were held as UNJUDGED rather than accepted as failures.
check-all 213 passed / 0 failed. F1 bracketing assertion CLEAN.
Session total banked: 44 functions + func_8015C030 propagated x7.
|
||
|
Drew T
|
fe946595fd |
fix(phase-30 S45p7): dedup_propagate leaves no orphaned reconcile on failure + func_8015C030 propagated x7
ROOT CAUSE of the 141/213 breakage earlier this session (correctly derived this time;
my first attribution to F1 was WRONG -- no arity journal ever touched func_80146A6C and
the arity undo reported success):
dedup_propagate --recover's Part B reconciles a conflicting caller extern and
DELIBERATELY leaves the edit on disk when it buys the byte-match ("keep the reconcile
on disk"). Correct while the fn survives -- but a fn can still be dropped by a LATER
iteration against a different overlay, and when the plan finally emptied, the
"all candidates dropped" sys.exit fired with NO restore. Reconciles kept for
ov_SC07_001..009 were orphaned: no-proto'd caller externs for functions that were
never propagated -> ov_SC07_010 "passing arg 2 of func_80146A6C makes pointer from
integer" -> 141 of 213 binaries failed check-all.
The byte-gate never mis-banked (it fails closed). The real cost was VERDICT VOIDING:
every subsequent gate reported "near" against the broken tree, so two whole batches
(4/4 and 20/20) were mis-read as draft failures when they measured the tree (R35).
FIX: a reconcile LEDGER. Every kept reconcile is recorded against its fn, undone the
moment that fn leaves the plan, and ALL outstanding reconciles are restored before the
failure exit -- so a failed propagation leaves the tree exactly as it found it.
HONESTY: the fix is IMPLEMENTED AND REVIEWED BUT NOT YET PROVEN. The negative control
aimed at the exact failing propagation SUCCEEDED instead (different tree state), so the
guarded path never executed. A targeted test of the ledger is still owed.
Also lands the propagation that control performed: func_8015C030 x7 overlays
(func_80168B70 excluded from 4 SC07 overlays, survived elsewhere). check-all 213/213.
|
||
|
Drew T
|
369dd14f4f |
fix(phase-30 S44 I.1d): the module class reaches every enumerating consumer
- family_hseq: widened from src/ov_*+sig.ov_* to every non-main binary (resident + md_*); the map now carries 139 binaries incl. resident (was overlays-only — which is exactly why the R36 gate's CHECK 4 could never see them). Self-count uses the SAME widened globs (cannot drift). - progress --weighted :647 + audit_frontier :57: + sig.md_* globs. - corpus.sig_is_independent: md_* sigs are sig_image-signed => independent (R34 trust). - backlog alias regex + prefetch_fleet (md_* derived from splat configs) + dedup_propagate (reads modules.mk alongside overlays.mk — excluding modules would re-create the SC07 invisible-work bug one class over). - VERIFIED: family map regenerated with resident (139 binaries); audit-binaries OK over 140; all six tools parse. |
||
|
Drew T
|
b497ee1649 |
feat(phase-30 S33c): PROPAGATE head COMPLETE — func_801466F0 x137 took three fixes + a type-lift
Fleet 96.17 -> 96.21% fn-count / 93.8% instr / 88.0% distinct; dedup 1909 -> 1910
groups, 0 failed, C1 241216/241216. R22 clean-fleet: 140 passed, 0 failed of 140.
The head is now 5/5 classes, 18,545 templatable ins, all banked this session from
a standing start of 0.
func_801466F0 had sat since S6b behind THREE separate blockers, each of which
looked sufficient on its own to explain the failure:
1. Its definition is under a §37/§73 ASM-LABEL ALIAS (`aF801466F0` in C, bound to
the real symbol by `__asm__`), and dedup_propagate.find_site anchored its head
regex on the literal `func_<ADDR>` — structurally blind to the form, returning
None, which every caller reads as "not matched". Now reuses
family_remap._alias_decl_for rather than growing a second matcher (R33).
2. That matcher was itself blind to the WRAPPED (multi-line) declaration — the
§134 shape, third tool. Fixed by matching over the joined text and mapping the
offset back to the decl's FIRST line (extract_unit carries from there).
Regression control: the single-line form still resolves. Fleet census after:
2,768 of 2,768 alias sites resolve, 0 missed.
3. Its record type was a draft-local typedef, so the body failed
compiles_standalone. Lifted Rec801466F0 to src/shared/engine_types.h INSIDE
the include guard (the SESSION-19 double-include note) and switched both the
macro and the exemplar to it — byte-neutral, gate-proven.
Probed on ONE member before the fleet run: byte-identical 9052dc0e first try.
MEASURED, NOT INHERITED (R37): the S6b note frames the alias-regex gap as a CLASS
of missed work. It is ONE function — 91 distinct alias decls fleet-wide, the
per-line matcher resolved 90. Recording it so a future session does not scope a
phase against a class that does not exist.
cookbook §138 extended with the alias-form tool boundary and the three-blocker
story; index regenerated.
|
||
|
Drew T
|
62042f65ca |
fix(phase-30 S11): multi-line-comment blindness in dedup_propagate; func_8012A598 x138
Fleet 96.06 -> 96.10% fn-count / 93.7% instr / 88.0% distinct; dedup 1907 -> 1908 groups, 0 failed, C1 240807/240807. R22 clean-fleet: 140 passed, 0 failed of 140. func_8012A598 (3,288 templatable ins) was being written off as CARRY-FIXABLE. It took TWO fixes; either alone leaves it skipped. 1. TOOL (R33) — find_site's preamble backscan. The SESSION-18 fix handled blank, `//`, and SINGLE-LINE `/* … */` lines, but a MULTI-LINE block comment still halted the walk: its middle lines start with `*` and its last line ends `*/` without starting `/*`. So the three externs above the body were dropped and the body then failed compiles_standalone on now-undeclared data. This is the §134 multi-line-blindness class — S6b fixed the identical shape three times in family_remap (D1/D2/D5) and this copy was never reached. Fixed by deciding skippability on `cdecl._mask` — the project's ONE masking oracle — instead of on line syntax: it subsumes every comment form at once and cannot be fooled by a `/*` inside a string, with an R32 assertion on the length-preservation invariant it rests on. Strictly monotone (it can only carry MORE preamble), and dedup_propagate is a byte-gate feeder, so a bug here can fail to bank but never falsely bank. 2. EXEMPLAR — the body also declared a draft-local `struct BigCopy164` tag, which the tool refuses by design (two macros defining one tag would redefine it in a single TU). The shared `struct BigCopy` (engine_types.h L312) is the identical layout and is ALREADY used this exact way at engine_core.h:16158, so switching the exemplar to it is byte-neutral and drops the alias too. Probed on ONE member before scaling (R37/S29): byte-identical 9052dc0e first try; then 138 overlays byte-identical. PROPAGATE head accounting after this: 7,398 of 18,545 ins banked (func_80147364 4,110 + func_8012A598 3,288). Still open, each with a NAMED cause and none yet diagnosed against a build: func_8012f274 (3,973, dropped), func_8016ba68 (3,886, 4/138), func_801466f0 (3,288, the S6b D4 wrapped-alias gap). |
||
|
Drew T
|
e112eff601 |
fix(phase-29): find_site — a comment-only line halted the extern scan (§68); func_80174CB0 x1 -> x3
TWO mislabels in one tool, both found by making it print what the compiler actually said.
1) compiles_standalone() returned a bare False and the caller filed EVERY failure under
"overlay-local TYPE (the real cap)". The dominant real cause is undeclared FILE-SCOPE EXTERNS.
Now returns (ok, stderr) and the skip is classified by actual cc1 output.
2) find_site()'s backward walk over "preceding contiguous externs" skipped BLANK lines but not
COMMENT-ONLY lines, so a full-line /* ---- */ between two extern groups dropped every extern
above it. Comment lines are now skipped like blanks and filtered out of the emitted body so
make_macro never meets a `//`.
RESULT, measured honestly: func_80174CB0 went from "not self-contained" to a 138-member PLAN, but
--recover banked only x3 (ov_SC07_006/007/011); 135 overlays excluded. Those exclusions are NOT
byte divergence (all 138 share h_exact) -- they are the CARRIED EXTERNS colliding with each target
overlay's own decls. The carry is necessary but not sufficient: it must reconcile per-target-TU
(cdecl.compatible(), the shape reconcile_tu already uses). Spec updated in CURRENT_PHASE.md.
- R22 clean-fleet 140/140, 0 failed. dedup-check 1883 validated / 0 failed, C1 coverage complete.
- fleet instr 79.9% (10,501,384 / 13,141,652); +246 ins from the x3.
- WHY THIS MATTERS beyond the numbers: the Phase-21 backlog already prescribed "macro-extern-
injection frees them x134 (~+0.3%)" and it was never built, because the mislabel told every later
session these were the known-hard type wall. A wrong diagnostic label cost ~4 phases.
- cookbook §68. NOTE the exclusion message is ALSO mislabelled ("byte-diverge / irreconcilable"
conflates differing bytes with a non-compiling instantiation) -- logged to fix.
|
||
|
Drew T
|
00b6448f4e |
fix(phase-29): dedup_propagate — the "overlay-local TYPE (the real cap)" skip was a MISLABEL
compiles_standalone() returned a bare False and the caller attributed EVERY failure to the overlay-local type cap. The dominant real cause is undeclared FILE-SCOPE EXTERNS: the body references extern decls living outside the extracted def block (func_80174CB0: 22 of them; carrying them makes it compile cc1 rc=0). - compiles_standalone now returns (ok, stderr); the skip is classified by actual cause: "missing file-scope extern (CARRY-FIXABLE): <names>" vs "overlay-local TYPE (the real cap)". - FLEET SIZING (--auto-from ov_SC01_077 --check-only): 7 skipped, ALL 7 carry-fixable, 0 genuine type-cap. Three of them (0x80142B2C/0x801535F4/0x80155800) are on the Phase-21 backlog list whose note ALREADY said "macro-extern-injection frees them x134 (~+0.3%)" -- never built, because the mislabel told every later session they were the type wall. A wrong label cost ~4 phases. - also: func_80174CB0's local Mtx_/Svec_ typedefs swapped for the canonical shared MATRIX/SVECTOR (byte-identical layouts); ov_SC07_006 still BYTE-IDENTICAL 7ca772be. - value behind the real fix: the 7 (~+0.3pp) + func_80174CB0 x138 (16,974 ins, ~+0.13pp), ~0 tokens. - _carry_externs itself is SPEC'd but NOT built here: it writes 138 overlay files (§63 class) and wants a fresh session with an R22 budget. func_80174CB0 (banked x1) is the test case. |
||
|
Drew T
|
e393c320e6 |
feat(phase-29): VARIANT camps -> UNIQUIFY (not reconcile); validated on Buf, R22 140/140 (§64a)
MEASUREMENT CORRECTED THE PLAN. The checkpoint called for a "per-camp field-access reconcile";
measuring the camps refutes that: Vec8 = {s32 w[8]} (32B) in 180 files AND {s16 unk0..} (8B) in
139 files; MATRIX 48B/32B/32B; Buf 16B / 0x20+ / DrawEnv. These are DIFFERENT types sharing an
identifier across TUs of the same overlay — reconciling to a canonical layout MERGES them, the
same failure that broke 103 binaries on Prim. The right op is UNIQUIFY: rename the non-majority
camp (byte-neutral — a type name emits no code; TU-local by construction), which makes every camp
single-def and liftable by the existing lift_types rules.
- NEW tools/uniquify_type.py: deterministic camp ordering (file-count desc, then normalized text,
so re-runs assign the same suffixes); majority keeps the name, camp n -> <T>_c<n>; rewrites ONLY
files that DEFINE that camp (a file that merely USES the name gets it elsewhere and is untouched);
\bT\b word boundaries so `Buf` never matches `Buf80153978`.
- VALIDATED on Buf (578/6/1 files): 11 identifiers across 7 files -> 3 camps LIFTABLE -> lifted
(585 local copies stripped) -> R22 140/140 -> blocked core queue 13 -> 11 (0x8012ea90, 0x801749c8
freed). Propagated 0x8012EA90 ×138; 0x801749C8 dropped (straggler in ov_SC07_006).
- YIELD, HONESTLY (P9): ZERO new matched functions. fn-count 88.61% / instr 79.3% / stubs 40281 all
UNCHANGED; dedup 1867->1868, C1 +138. 0x8012EA90's members were ALREADY matched in all 138
overlays — the propagation consolidated duplication into one shared macro (DRY), not coverage.
The value is the PROVEN RECIPE + the queue moving 13->11, not the numbers.
- dedup_propagate (R32): the skip line printed a COUNT and no names, and aggregated three unrelated
causes into n_local — a body skipped merely for a `//` comment (macro-unsafe, 1-line fix) read
identically to one genuinely using an overlay-local type. Now named and split by cause.
- cookbook §64a (uniquify-vs-reconcile + the validated recipe + remaining camps by cost).
|
||
|
Drew T
|
82d79e7a32 |
fix(phase-26a): A6/A7 — the family engine could not see half its corpus; 17 fns banked x134 free
R22: check-all 136 PASSED / 0 FAILED. dedup-check 1823 validated / 0 failed (C1 coverage 224,933/224,933).
Fleet instr-weighted 66.5% -> 66.7%.
=== dedup_propagate: it was blind to HALF the corpus ===
overlay_files() used a hardcoded suffix allowlist ("_a","_o0","_o0b","_after") that predated the
Phase-26 jr carves -> 404 of the fleet's 811 overlay .c. The 407-file gap held 36,135 INCLUDE_ASM stubs
and ~32,000 inline defs, and overlay_files gates ALL of dedup_propagate (source_text / find_site /
apply_plan / struct_check / reconcile_caller_extern). Now a GLOB — never an allowlist, because the NEXT
split family would re-open it. The asm_subdir is always the file stem, an invariant the old four entries
already satisfied.
find_site's def-detector required the signature line to END in ')' and the next non-blank line to START
with '{'. It therefore silently dropped THREE shapes: K&R definitions (`s32 f(arg0)` / `s32 arg0;` / `{`),
multi-line signatures, and single-line bodies. K&R is the project's house style for exactly the biggest,
highest-reach functions — func_8015AE2C (562 ins), func_80166994, func_80133CD4, func_8015A3C8 — and they
live in the _jr_* files overlay_files could not even open. Fixing either alone would have been useless:
the glob exposes the files, and find_site would still drop their biggest prizes. Both fixed together.
* The signature's closing paren is now found by a real paren-walk, not line.count() or split(')')[-1]:
a single-line body containing a call (`void f(int a){ g(a); }`) has balanced parens of its own, so
both shortcuts land on the WRONG paren and then misread the body's ';' as a prototype terminator.
* AGREEMENT ASSERTION (the audit's): find_site vs family_remap.extract_unit -> 701 agree / 0 disagree.
Negative controls hold (a prototype+call is rejected; a 1-line body with a call is a def).
=== THE HARVEST (free work, byte-gated) ===
--auto-from ov_SC01_077 now nominates what it could never see: 20 planned, 17 propagated x134, 3 dropped
as cross-overlay stragglers. 134 overlays rebuilt BYTE-IDENTICAL; 17 new dedup groups.
Includes ALL FOUR functions A1 caught the registry lying about (func_80128ED8 / 8012C098 / 8012C0EC /
8012C750): 0 stubs remaining, real shared macros. THE LOOP CLOSES — A1 found the lie, and THIS is the
bug that had made it true (3 of the 4 are defined in ov_SC01_077_jr_8012ACE0.c, which the allowlist could
not open, so the propagation never ran and dedup_integrate greenlit the result).
=== family_remap: 96 PHANTOM exemplars -> 0 ===
extract_unit globbed only src/<ov>/<ov>*.c, so a function matched via a SHARED body had no source form
and read as NOT MATCHED. 93-96 of 218 h_seq "matched" exemplars were phantom, carrying 2,157 candidate
members of which 1,834 are still-stubbed, PURE/IMM-clean, symbol_map-clean and unpinned — staged and
gated today, dropped before the first build then. It is now TOTAL over BOTH shared-body mechanisms:
(1) the DEFINE_func_<ADDR>() macro — reconstructed as the exact INVERSE of dedup_propagate.make_macro
(derived from the generator, not re-guessed from the text);
(2) a DIRECT definition in a shared header, #included per overlay — the whale (func_80144B9C, 770 ins,
-O0), which the registry explicitly records as "NOT a DEFINE_ macro".
CENSUS: 216 matched exemplars, 216 real, 0 PHANTOM.
symbol_map named the symbol by HOW IT WAS LOADED, not by WHAT IT IS: reloc_targets labels every lui/%lo
pair "data", and a FUNCTION's address taken via lui/%lo (an address-taken callback) is exactly that shape
(splat's own .s: %lo(func_8017E1D4), 7 occurrences). The map got a D_<ADDR> key while the C writes
func_<ADDR>, so the word-bounded substitution matched NOTHING and silently no-op'd — the sibling kept the
EXEMPLAR's function pointer and the loss was booked as a BYTE failure, indistinguishable from a compiler
wall. Now emits both keys (addresses are unique; the pass is simultaneous, so the extra key is free).
gather_externs was line-oriented, so a WRAPPED comma extern was invisible in both directions (the first
line has no ';', the continuation has no `extern`). ov_SC01_077.c:271-272 declares NINE symbols that way,
and the exemplar referencing them (func_8013D178) is a 133-member family — every sibling was staged with
NO declaration, failed to compile, and bisect-stormed its whole gate group. Now statement-oriented, and
an unresolved symbol is REPORTED, never silently dropped.
=== family_sweep.stub_map / build_engine_types ===
stub_map: func_-only -> a curated-name stub read as "already matched" -> phantom exemplar. Now corpus-derived.
build_engine_types hard-exited on 1,070 of 1,470 type-bearing overlay .c (73%; the audit measured 573/709
= 81% on its narrower set) because 1,929 TAGGED-struct typedefs tripped a guard whose own comment asserts
"our source has only ANONYMOUS-struct typedefs" — true in Phase 20, false since the harvest agents started
writing tagged structs. inject_capped_externs routes every type-bearing body HERE as the type-heavy tail's
ONLY sanctioned unblocker, so the tail's unblocker could not run on the corpus the tail lives in.
A contained def (the typedef's span encloses the body) is liftable — it just must not be counted twice;
only a PARTIAL overlap is malformed. Verified on a file that used to hard-exit: 5 tagged typedefs folded +
forward-declared, 46 types written, exit 0.
** AND THE SHARPEST LESSON IN THE AUDIT: this one was never silent. It printed "[overlap] ... handle
manually" every single time. But the message reads like a rare edge case rather than a four-fifths
coverage failure, so nobody ever COUNTED it. A loud failure that nobody counts is exactly as
invisible as a silent one. R32 must be "assert your coverage", not merely "fail loud". **
R14 self-catches, recorded because I hit both while fixing them: my first shared-header scan read a macro
body's `extern void f(void); \` as a DEFINITION (the trailing continuation means the line does not end in
';', so the decl guard never fired) — the exact bug fixed at commit:0552, reintroduced by me and caught only
because the whale resolved from the WRONG file. Column-0 anchoring fixes it by construction. And my
phantom census returned 0/0 twice because I guessed the manifest schema instead of reading it.
|
||
|
Drew T
|
8e6658ee85 |
feat(phase-24): T7 §G — func_801770E0 banked ×134 (native DEFINE-macro path); fleet 65.95→65.99%
- func_801770E0 (152 ins, reach-134) propagated ×134 via dedup_propagate --recover. Chose the NATIVE DEFINE-macro path over a hand-rolled shared header: func_8014E048 (pins+asm) is already ×134 via a DEFINE macro, proving that path handles pin/asm -O2 giants (the whale needed a shared header only because it is -O0 -> separate object). Clean fleet check-all 136/136 (R22), dedup-check 1811->1812/0. - R14: func_8014E048 was ALREADY ×134 (T6 §A) — the whale-session handoff was stale; only func_801770E0 + func_801372B0 actually remained ×1. - tooling (reusable): dedup_propagate.overlay_files now also scans the whale-rollout _o0b/_after splits — post-whale-region fns (func_801770E0 in _after.c) were invisible for both source-def-find and stub-replacement. func_801770E0 extern block made contiguous (a comment between externs made find_site drop 5 externs; byte-neutral). |
||
|
Drew T
|
99ccc37480 |
feat(phase-24): T6.4 — find_site trailing-comment externs → func_8014E048 + func_80157580 ×134
- find_site extern-collection: allow a trailing `/* comment */` after the `;`. The comment-blind regex `^\s*extern\b.*;\s*$` stopped the backward scan at `extern u8 D_801152A8[]; /* canonical TU type */`, dropping every EARLIER extern → compiles_standalone failed on the now-undeclared callees/data (func_80135A4C, func_80133784, D_801152A8). R14: THIS — not "pin/asm" as the backlog framed it — was func_8014E048's real self-containment blocker. - propagated func_8014E048 (the T5b S11 pins+barrier crack) + func_80157580 ×134; dedup 1797→1799 groups (0 failed). CLEAN fleet check-all 136/136 BYTE-IDENTICAL (R22). Fleet byte-identical 65.40% → 65.48%. |
||
|
Drew T
|
5b3697553f |
feat(phase-24): T6 — 13 leaf-MATCH fns propagated ×134 (fleet 64.90→65.40%)
find_site + dedup_propagate --recover + build_engine_types fixes, then re-bank the 13 recover_integration leaf-MATCHes and propagate each across all 134 overlays. - find_site: match INDENTED inline defs (was column-0 only, silently dropping every recover_integration-banked def from propagation — T6 blocker 1). Unit-tested: indented defs match; indented call-exprs (if/assign/bare/return) correctly rejected. - dedup_propagate --recover: on a straggler byte-gate failure, FIRST no-proto that overlay's conflicting caller extern + re-gate (Part B, byte-neutral, same lever as fix_arity_callers --any-proto); else EXCLUDE only that overlay (Part A, ×N-1) rather than the historical all-or-nothing drop. Wired into gate_stage. - build_engine_types: comment-aware find_defs/find_typedefs (blank_comments). The generated header's own "...typedef lift" comment was captured as a bogus `typedef vec`, self-colliding and blocking every --strip. NOTE: full --strip still conflicts with the _a/_o0 split files, so split-file overlays need a TARGETED lift. - banked + propagated 13 fns ×134: func_8014F74C 801542A4 8015BE94 8015F380 80160F00 801653B8 80166244 8016E778 801732C4 8017331C 80173374 80174554 801745AC. func_8014F74C needed PosT/MoveT lifted to engine_types.h (targeted, byte-neutral). - dedup 1784→1797 groups (0 failed). CLEAN fleet check-all 136/136 BYTE-IDENTICAL (R22: make clean && extract-all && check-all). Fleet REAL 224073, byte-identical 65.40%. R14: the confounding stale-asm/ tree (13 missing .s) that masked the first --recover test is fixed by re-extract; the 3 pre-existing --auto-from stragglers (0x80174650/ 8012A018/80165CA0) are pin/asm + uncaptured-local-macro (SHB) bodies, correctly dropped. |
||
|
Drew T
|
5e5423e953 |
feat(phase-21): dedup_propagate drop-straggler retry — _a pipeline now hands-free
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure, isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates); byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical). Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3 clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe. |
||
|
Drew T
|
04c5c9b47f |
fix(phase-21): dedup_propagate skips -O0 overlay-local fns + gate_stage surfaces prop failures
The cont.4 root-cause fixes so future _a waves auto-realize ×134: - dedup_propagate --auto-from now excludes *_o0.c (-O0) defs: -O0 codegen embeds per-overlay %lo data, so masked h_exact falsely reports reach-134 (§18/§20). One such straggler (func_8013C360) reverted 10 clean ×134 matches under the all-or-nothing batch gate (cont.4). Detected via find_site on the -O0 split file; --addr still forces them. (6 fns excluded.) - gate_stage: capture dedup_propagate's exit; on a real byte-gate revert (not the benign 'nothing to propagate' no-op) write .run/auto/last_propagate_error.log + add prop_error to the summary + warn. A swallowed revert previously hid the gain silently. |
||
|
Drew T
|
cbf7e774ab |
feat(phase-21): split-aware dedup_propagate + cast_call_sites --src-file (unlock the _a.c vein)
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0); source-find scans all, member-loop edits the right file, structural-check spans all. Single-file overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert. - cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged. - FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same §16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe. - no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted. |
||
|
Drew T
|
a10f7f8a1a |
feat(phase-15): sig-unify recovery + find_site brace-fix — fleet 52.08% -> 54.44%
- tools/sig_unify.py (NEW): unifies a draft's FULL signature set to the banked-canonical decls — both callee externs AND the draft's OWN definition signature (return + param types, body param-names preserved). Recovery for the standalone-MATCH residual: a probe found 30/30 sampled failures were type conflicts, 0 false-positives. Gated 191 -> 32 verified. - tools/dedup_propagate.py find_site FIX: the inline-def detector required the opening brace ON THE SAME LINE as the signature, silently dropping every next-line-brace def from propagation (sig_unify/permuter outputs + ~stragglers all session). Now accepts brace on the same OR next line -> unlocked a 61-function propagation backlog. - propagated 61 fns fleet-wide (incl. 32 sig-unify + the permuter win func_801508F8 + earlier-dropped next-line-brace matches), all 134 byte-identical; dedup-check 1394/0 - R22 CLEAN fleet rebuild: 136/136 byte-identical. REAL 177501 -> 185646. Zero agent tokens. |
||
|
Drew T
|
5c5e9fb973 |
feat(phase-15): T6 struct follow-up — shared engine_types.h unblocks 37 struct-using fns; fleet 46.27% -> 47.70%
- tools/build_engine_types.py -> src/shared/engine_types.h: extract the 34 named struct/union types from ov_SC01_077.c (forward decls + defs in source order; consumes trailing __attribute__((packed)) + aliases through the ';'). 0 same-name-different-layout collisions. - engine_core.h includes engine_types.h; the 34 inline defs stripped from ov_SC01_077.c (byte-neutral — type defs emit no code; verified byte-identical). - dedup_propagate.py: relax the struct guard to skip only inline named-struct/typedef DEFINITIONS (header-type USAGES + anonymous local structs now propagate); add engine_types.h to the compile pre-filter so struct-using bodies resolve. - 37 struct-using shared fns propagated fleet-wide (skip 41 -> 4 overlay-local typedef'd types), 134 overlays byte-identical; dedup 1183 -> 1220 groups, 1220 validated / 0 failed. - fleet REAL substantive 157,541 -> 162,462; byte-identical 46.27% -> 47.70%. - R22 clean rebuild: 136/136 byte-identical, 0 extract failures. |
||
|
Drew T
|
bdfb86dc4f |
feat(phase-15): T6 v2 — data-sig-aware harvest + struct-guarded propagate; fleet 31.22% -> 42.14%
- gen_harvest_targets.py: now also resolves canonical DATA-symbol declarations per target (scan the .s for D_XXXX refs -> `extern <type> D_XXXX;` from banked code) in addition to callee function sigs — the §14c(c) fix at the source, so agents declare both functions and data correctly and avoid the dominant remaining conflict class. - v2 harvest (callee+data-sig-aware) on the 591 small still-stub call-heavy fns + a gap-fill agent for 6 targets dropped by a connection-closed-mid-response failure (the retry-wave misses truncated-non-null results). Gate: 306 verified (ov_SC01_077 1554 -> 1248 stubs). - dedup_propagate.py: skip any struct/union-touching body — agents named structs with colliding generic names (`struct S`/`struct vec`) inline, so two macros' types redefine/conflict when instantiated in one overlay (ov_SC01_000 abort). 34 skip; struct-using shared fns stay banked in 077 (shared-types-header follow-up). - 281 functions propagated fleet-wide (134 overlays each, every overlay byte-gated); dedup 795 -> 1076 groups, dedup_integrate --check 1076 validated / 0 failed. - fleet REAL substantive 105,764 -> 143,322; byte-identical 31.22% -> 42.14%. - R22 clean rebuild: 136/136 byte-identical, 0 extract failures. - cookbook §12: gap-fill is now a MANDATORY post-step for every multi-agent run (reconcile produced-vs-expected: missing + truncated drafts) before gating. |
||
|
Drew T
|
aefe2eac18 |
feat(phase-15): T5a — bulk-propagate 553 shared functions fleet-wide (3.82% -> 22.14%)
- dedup_propagate --auto-from ov_SC01_077: propagated 553 already-matched, self-contained shared engine functions across all 134 overlays, each byte-gated. 9 functions skipped (use 077-local struct types -> not mechanically liftable; honest, P9). One R22 CLEAN rebuild: 136/136 byte-identical. - fleet REAL 947 -> 74,527; byte-identical 13,132 -> 76,174 / 344,010 = 22.14%; 561 dedup groups, dedup-check 561 validated / 0 failed - tool hardening for fleet scale: registry SHORTHAND (vram + binaries; dedup.us.yaml ~4k lines vs ~77k verbose); per-overlay apply (one read/write per file); gate-only-changed; skip already-registered (additive/resumable); compile pre-filter (body must build with common.h alone); group_members() reader (both forms) in dedup_integrate + progress; progress _DEDUP_CACHE (fleet 6m -> 7s) - the first bulk attempt fail-closed-reverted on a local-typed body (byte-gate working) -> added the compile filter; cookbook 14a documents the 5 fleet-bulk lessons - ghidra/ churn NOT staged (R23) |
||
|
Drew T
|
0c7619231d |
feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X). - proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all 18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test (corrupt shared body) -> make check FAILS then restores - Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a shared header recompiles dependents — closes a stale-build/false-pass gap the negative test exposed; output-neutral (R22 clean 18/18) - structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate can't catch under-application -> assert DEFINE present + stub gone - FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest - cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory - ghidra/ churn NOT staged (R23); build/asm/.run gitignored |