mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
85fb289db582d842fc41dc059fa187bb992e76ea
23 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
f6e48b60c5 |
perf(phase-30 S46-4): parallelise the propagation — 24min -> 11.4min, and +62 MORE instances
Drew: "make it more multi-threaded... I still see my cpu idle for far too long." Measured, fixed, and regression-tested against the S46-3 bank as a KNOWN ANSWER. - THE MEASUREMENT: 31s saturated (33 makes/48 cc1/load 27) then ~25s with ONE build alive while 31 cores idled, repeating. Causes: ex.map starts in list order so the giants land last, and apply/restore is single-threaded. - gate_all -> gate_failures: return EVERY failure the sweep already computed (~138 rounds -> 1). - Longest-first gate scheduling; results re-sorted into `changed` order so the verdict stays bit-identical to the serial loop's. - PER-OVERLAY INDEPENDENT SEARCH, IN PROCESSES. My first cut used threads and the box refuted it: 0-4 builds alive at load 3, because the work is regex over 15k-line files and 138 "parallel" searches all queued on the GIL. Same logic in a ProcessPoolExecutor: 14-29 builds, load 34.75, search phase ~100s. Safe because the shared header is written ONCE by the parent and each overlay owns its own .c files + build/<bin>/. Seeded with one in-process search first — a pool submitted at once gives every worker an empty suspect list and makes all 138 pay a full bisection. place_in_overlay extracted to module level so the worker and the in-process apply cannot drift (R33); compiles_standalone's fixed t.c is per-call now. - THE REGRESSION (the point, not the stopwatch): revert src/+config to pre-bank, re-run the identical command -> 29 functions (same), 141 overlays byte-identical, 682s vs ~1440s, and 285 exclusions vs ~350 => +62 MORE member instances (249,161). The old prefix-based necessity probe was OVER-EXCLUDING (charging 4 fns to 9 overlays that did not all need them); the per-overlay shrink minimises per overlay. The faster path is also more correct — a timing comparison would never have shown it. R22 213/213 + tools-health green. - STILL SERIAL, now the actual wall-clock (neither is a build): ~3min setup before the first gate (registered_addrs() yaml-parsing a 1949-group/249k-instance registry + 213 sig loads) and ~2.5min of sequential reconcile_caller_extern after the search. - Captured as defaults: docs/accelerators.md A8 + memory fleet-tool-parallelism-defaults. cookbook index regenerated (my §155c append left it stale — the gate caught it, exit 1). |
||
|
Drew T
|
91c64ce92c |
fix(phase-30 S46-1): dedup_propagate — no silent skips, no unproven REVERTs
The S45p9 blocker: `[FAIL] ov_MAIN_012: 0x80156600 not instantiated — REVERTED`
92 minutes into a --auto-from run, naming no mechanism.
- FIRST, the honest finding (R14/R35): it does NOT reproduce at HEAD. A replay of
apply_plan's per-file site resolution over the exact 30-fn plan resolves
0x80156600 as a stub at line 7505, and def-range/stub-line overlaps = 0 (the
splice-swallow hypothesis refuted). The failing input state was not the committed
tree — most likely a concurrent writer mid-run. So this commit does not "fix" that
run; it makes the next occurrence name itself.
- SILENT SKIP -> LOUD (R32): an address resolving as neither the sp-regex stub nor a
def just stayed in `remaining`. apply_plan now records gaps={ov:[addrs]} and the
caller fails FIRST with a per-site diagnosis (whole-overlay find_site verdict,
in-sig, file list) instead of struct_check's terse late message.
- CAPABILITY GAP that produces exactly that skip: find_site returning 'stub' was
ignored (apply_plan acted only on 'def'), so a stub whose INCLUDE_ASM asm-subdir
!= its file stem was invisible to the stem-anchored sp regex AND unhandled. Now
placed ('macro' treated as already-placed). find_site's stub match is an exact
stub_line(ov,addr) compare against THIS file's text — it cannot cross files/TUs.
- INCOMPLETE REVERT (the §156 class, different path): struct_check restored only
`touched`, leaking every kept Part-B reconcile. New _abort() undoes touched AND
every kept reconcile, then diffs the worktree against a start-of-run baseline and
reports any residue. A tree dirty in a way nobody knows about makes every later
byte-gate report `near` — that is how S45p7 lost two batches.
- NEGATIVE CONTROL: neuter ov_MAIN_012's stub -> [GAP] fires naming the exact
condition (find_site=None, in-sig=True) -> "[revert] tree restored to baseline;
no residue" -> exit 1 (fail-closed). Restore -> tree clean.
|
||
|
Drew T
|
e86e45320a |
chore(phase-30 S45p9): session close — 32-way parallel gate in dedup_propagate; banking deferred on a tool bug
PARALLEL GATE (landed, verdict-proven): dedup_propagate's byte-gate loop was serial --
one `make build BINARY=<ov>` at a time over up to 141 members per function. Measured: a
propagation ran 95 minutes at load 1.6 on a 32-core box (~5% utilisation). The Makefile
has parallelised extract-all/check-all since Phase 26 (xargs -P$(JOBS)), but this tool
predates that and drives the SINGLE-binary target from Python, so it never saw any of it.
- new gate_all(): ThreadPoolExecutor over distinct overlays, 32-way by default (JOBS env
overrides; deliberately NOT capped at the Makefile's conservative 16).
- SAFE by the same argument check-all relies on: byte_gate only runs `make build`, writing
solely to per-binary-disjoint build/<bin>/**; it mutates no source. Splice happens before,
restore after -- only the VERIFICATION is parallel.
- DETERMINISTIC: ThreadPoolExecutor.map preserves order, so the reported first failure is
the first in `changed` order -- identical verdict to the serial loop. Control run: same
verdict on a clean tree.
- Measured and NOT optimised: setup (sig load + registered_addrs) is 8.6s of a 5,700s run
= 0.15%. All the time is gating. Don't thread the setup.
BANKING DEFERRED on a genuine pre-existing tool bug (NOT the parallel change -- 0 gate
batches ran, it never reached that code):
[FAIL] ov_MAIN_012: 0x80156600 not instantiated -- REVERTED
Inputs verified sound at HEAD (in sig, find_site->stub, stub line matches), so the bug is
in apply_plan's multi-function edit path. Run #1 missed it because it launched before the
15 wave-3 banks were committed; they landed mid-flight, enlarging run #2's plan.
SECOND DEFECT: the failure exit printed REVERTED but left 38 files dirty incl.
src/shared/engine_core.h -- the same incomplete-restore class as the reconcile-ledger bug
(cookbook 156), on a different path. struct_check needs the same ledger treatment.
Not patching the fleet-shared writer at the end of a marathon session -- that is how the
next 141-binary incident happens. Tree clean, 44 banks safe, propagation is pure
multiplication and can run any time.
Checkpoint p9 carries: the fix-then-resume plan, the master-IDXTAB-map design (DESTPTR half
proven 14/14), wave guidance, and an 8-item error ledger with its single root cause.
|
||
|
Drew T
|
e0eaa16741 |
feat(phase-30 S45p7): Stage 1 complete — shared-state RW lock + 15 more banks (wave-3 revived)
STAGE 1 of docs/concurrency-design.md, landed and negative-control proven.
tools/shared_lock.py (NEW) — one reader/writer flock over the FLEET-SHARED state
(src/shared/*, config/overlays.mk, config/dedup.us.yaml, the overlay .c files
propagation rewrites). Per-binary resources keep gate_stage's existing per-binary flock.
- gate_stage takes it SHARED when the gate writes no shared state, EXCLUSIVE when it
does (propagate, or the arity pre-pass enabled) -- so distinct-binary gates still run
concurrently but can never overlap a writer.
- dedup_propagate and fix_arity_callers --apply take it EXCLUSIVE.
- NESTING-AWARE: gate_stage SPAWNS both writers, so a naive child lock would deadlock
against the parent. The holder exports BFM_SHARED_LOCK_HELD and children inherit.
NEGATIVE CONTROLS (all pass):
NC1 a held SHARED lock refuses a non-blocking exclusive writer, loudly, naming the lock
NC2 parent-holds/child-inherits does NOT deadlock (the real risk in this design)
NC3 two readers acquire concurrently (0.00s) -- phase-B parallelism preserved
bulk_harvest docstring CORRECTED: its "propagation is the ONLY writer of the shared
engine_core.h" claim was FALSE as written and had been asserted for phases (F1 -- the
arity pre-pass writes it from inside every worker). Now states what is actually true,
under which two conditions, plus the one-line assertion that detects a violation.
BANKS: wave-3's drafts re-gated on a CLEAN tree -> 15 of 20 banked. The same drafts
previously reported 0 banked / 20 near -- that verdict was 100% an artifact of the
broken tree, which is why they were held as UNJUDGED rather than accepted as failures.
check-all 213 passed / 0 failed. F1 bracketing assertion CLEAN.
Session total banked: 44 functions + func_8015C030 propagated x7.
|
||
|
Drew T
|
fe946595fd |
fix(phase-30 S45p7): dedup_propagate leaves no orphaned reconcile on failure + func_8015C030 propagated x7
ROOT CAUSE of the 141/213 breakage earlier this session (correctly derived this time;
my first attribution to F1 was WRONG -- no arity journal ever touched func_80146A6C and
the arity undo reported success):
dedup_propagate --recover's Part B reconciles a conflicting caller extern and
DELIBERATELY leaves the edit on disk when it buys the byte-match ("keep the reconcile
on disk"). Correct while the fn survives -- but a fn can still be dropped by a LATER
iteration against a different overlay, and when the plan finally emptied, the
"all candidates dropped" sys.exit fired with NO restore. Reconciles kept for
ov_SC07_001..009 were orphaned: no-proto'd caller externs for functions that were
never propagated -> ov_SC07_010 "passing arg 2 of func_80146A6C makes pointer from
integer" -> 141 of 213 binaries failed check-all.
The byte-gate never mis-banked (it fails closed). The real cost was VERDICT VOIDING:
every subsequent gate reported "near" against the broken tree, so two whole batches
(4/4 and 20/20) were mis-read as draft failures when they measured the tree (R35).
FIX: a reconcile LEDGER. Every kept reconcile is recorded against its fn, undone the
moment that fn leaves the plan, and ALL outstanding reconciles are restored before the
failure exit -- so a failed propagation leaves the tree exactly as it found it.
HONESTY: the fix is IMPLEMENTED AND REVIEWED BUT NOT YET PROVEN. The negative control
aimed at the exact failing propagation SUCCEEDED instead (different tree state), so the
guarded path never executed. A targeted test of the ledger is still owed.
Also lands the propagation that control performed: func_8015C030 x7 overlays
(func_80168B70 excluded from 4 SC07 overlays, survived elsewhere). check-all 213/213.
|
||
|
Drew T
|
369dd14f4f |
fix(phase-30 S44 I.1d): the module class reaches every enumerating consumer
- family_hseq: widened from src/ov_*+sig.ov_* to every non-main binary (resident + md_*); the map now carries 139 binaries incl. resident (was overlays-only — which is exactly why the R36 gate's CHECK 4 could never see them). Self-count uses the SAME widened globs (cannot drift). - progress --weighted :647 + audit_frontier :57: + sig.md_* globs. - corpus.sig_is_independent: md_* sigs are sig_image-signed => independent (R34 trust). - backlog alias regex + prefetch_fleet (md_* derived from splat configs) + dedup_propagate (reads modules.mk alongside overlays.mk — excluding modules would re-create the SC07 invisible-work bug one class over). - VERIFIED: family map regenerated with resident (139 binaries); audit-binaries OK over 140; all six tools parse. |
||
|
Drew T
|
b497ee1649 |
feat(phase-30 S33c): PROPAGATE head COMPLETE — func_801466F0 x137 took three fixes + a type-lift
Fleet 96.17 -> 96.21% fn-count / 93.8% instr / 88.0% distinct; dedup 1909 -> 1910
groups, 0 failed, C1 241216/241216. R22 clean-fleet: 140 passed, 0 failed of 140.
The head is now 5/5 classes, 18,545 templatable ins, all banked this session from
a standing start of 0.
func_801466F0 had sat since S6b behind THREE separate blockers, each of which
looked sufficient on its own to explain the failure:
1. Its definition is under a §37/§73 ASM-LABEL ALIAS (`aF801466F0` in C, bound to
the real symbol by `__asm__`), and dedup_propagate.find_site anchored its head
regex on the literal `func_<ADDR>` — structurally blind to the form, returning
None, which every caller reads as "not matched". Now reuses
family_remap._alias_decl_for rather than growing a second matcher (R33).
2. That matcher was itself blind to the WRAPPED (multi-line) declaration — the
§134 shape, third tool. Fixed by matching over the joined text and mapping the
offset back to the decl's FIRST line (extract_unit carries from there).
Regression control: the single-line form still resolves. Fleet census after:
2,768 of 2,768 alias sites resolve, 0 missed.
3. Its record type was a draft-local typedef, so the body failed
compiles_standalone. Lifted Rec801466F0 to src/shared/engine_types.h INSIDE
the include guard (the SESSION-19 double-include note) and switched both the
macro and the exemplar to it — byte-neutral, gate-proven.
Probed on ONE member before the fleet run: byte-identical 9052dc0e first try.
MEASURED, NOT INHERITED (R37): the S6b note frames the alias-regex gap as a CLASS
of missed work. It is ONE function — 91 distinct alias decls fleet-wide, the
per-line matcher resolved 90. Recording it so a future session does not scope a
phase against a class that does not exist.
cookbook §138 extended with the alias-form tool boundary and the three-blocker
story; index regenerated.
|
||
|
Drew T
|
62042f65ca |
fix(phase-30 S11): multi-line-comment blindness in dedup_propagate; func_8012A598 x138
Fleet 96.06 -> 96.10% fn-count / 93.7% instr / 88.0% distinct; dedup 1907 -> 1908 groups, 0 failed, C1 240807/240807. R22 clean-fleet: 140 passed, 0 failed of 140. func_8012A598 (3,288 templatable ins) was being written off as CARRY-FIXABLE. It took TWO fixes; either alone leaves it skipped. 1. TOOL (R33) — find_site's preamble backscan. The SESSION-18 fix handled blank, `//`, and SINGLE-LINE `/* … */` lines, but a MULTI-LINE block comment still halted the walk: its middle lines start with `*` and its last line ends `*/` without starting `/*`. So the three externs above the body were dropped and the body then failed compiles_standalone on now-undeclared data. This is the §134 multi-line-blindness class — S6b fixed the identical shape three times in family_remap (D1/D2/D5) and this copy was never reached. Fixed by deciding skippability on `cdecl._mask` — the project's ONE masking oracle — instead of on line syntax: it subsumes every comment form at once and cannot be fooled by a `/*` inside a string, with an R32 assertion on the length-preservation invariant it rests on. Strictly monotone (it can only carry MORE preamble), and dedup_propagate is a byte-gate feeder, so a bug here can fail to bank but never falsely bank. 2. EXEMPLAR — the body also declared a draft-local `struct BigCopy164` tag, which the tool refuses by design (two macros defining one tag would redefine it in a single TU). The shared `struct BigCopy` (engine_types.h L312) is the identical layout and is ALREADY used this exact way at engine_core.h:16158, so switching the exemplar to it is byte-neutral and drops the alias too. Probed on ONE member before scaling (R37/S29): byte-identical 9052dc0e first try; then 138 overlays byte-identical. PROPAGATE head accounting after this: 7,398 of 18,545 ins banked (func_80147364 4,110 + func_8012A598 3,288). Still open, each with a NAMED cause and none yet diagnosed against a build: func_8012f274 (3,973, dropped), func_8016ba68 (3,886, 4/138), func_801466f0 (3,288, the S6b D4 wrapped-alias gap). |
||
|
Drew T
|
e112eff601 |
fix(phase-29): find_site — a comment-only line halted the extern scan (§68); func_80174CB0 x1 -> x3
TWO mislabels in one tool, both found by making it print what the compiler actually said.
1) compiles_standalone() returned a bare False and the caller filed EVERY failure under
"overlay-local TYPE (the real cap)". The dominant real cause is undeclared FILE-SCOPE EXTERNS.
Now returns (ok, stderr) and the skip is classified by actual cc1 output.
2) find_site()'s backward walk over "preceding contiguous externs" skipped BLANK lines but not
COMMENT-ONLY lines, so a full-line /* ---- */ between two extern groups dropped every extern
above it. Comment lines are now skipped like blanks and filtered out of the emitted body so
make_macro never meets a `//`.
RESULT, measured honestly: func_80174CB0 went from "not self-contained" to a 138-member PLAN, but
--recover banked only x3 (ov_SC07_006/007/011); 135 overlays excluded. Those exclusions are NOT
byte divergence (all 138 share h_exact) -- they are the CARRIED EXTERNS colliding with each target
overlay's own decls. The carry is necessary but not sufficient: it must reconcile per-target-TU
(cdecl.compatible(), the shape reconcile_tu already uses). Spec updated in CURRENT_PHASE.md.
- R22 clean-fleet 140/140, 0 failed. dedup-check 1883 validated / 0 failed, C1 coverage complete.
- fleet instr 79.9% (10,501,384 / 13,141,652); +246 ins from the x3.
- WHY THIS MATTERS beyond the numbers: the Phase-21 backlog already prescribed "macro-extern-
injection frees them x134 (~+0.3%)" and it was never built, because the mislabel told every later
session these were the known-hard type wall. A wrong diagnostic label cost ~4 phases.
- cookbook §68. NOTE the exclusion message is ALSO mislabelled ("byte-diverge / irreconcilable"
conflates differing bytes with a non-compiling instantiation) -- logged to fix.
|
||
|
Drew T
|
00b6448f4e |
fix(phase-29): dedup_propagate — the "overlay-local TYPE (the real cap)" skip was a MISLABEL
compiles_standalone() returned a bare False and the caller attributed EVERY failure to the overlay-local type cap. The dominant real cause is undeclared FILE-SCOPE EXTERNS: the body references extern decls living outside the extracted def block (func_80174CB0: 22 of them; carrying them makes it compile cc1 rc=0). - compiles_standalone now returns (ok, stderr); the skip is classified by actual cause: "missing file-scope extern (CARRY-FIXABLE): <names>" vs "overlay-local TYPE (the real cap)". - FLEET SIZING (--auto-from ov_SC01_077 --check-only): 7 skipped, ALL 7 carry-fixable, 0 genuine type-cap. Three of them (0x80142B2C/0x801535F4/0x80155800) are on the Phase-21 backlog list whose note ALREADY said "macro-extern-injection frees them x134 (~+0.3%)" -- never built, because the mislabel told every later session they were the type wall. A wrong label cost ~4 phases. - also: func_80174CB0's local Mtx_/Svec_ typedefs swapped for the canonical shared MATRIX/SVECTOR (byte-identical layouts); ov_SC07_006 still BYTE-IDENTICAL 7ca772be. - value behind the real fix: the 7 (~+0.3pp) + func_80174CB0 x138 (16,974 ins, ~+0.13pp), ~0 tokens. - _carry_externs itself is SPEC'd but NOT built here: it writes 138 overlay files (§63 class) and wants a fresh session with an R22 budget. func_80174CB0 (banked x1) is the test case. |
||
|
Drew T
|
e393c320e6 |
feat(phase-29): VARIANT camps -> UNIQUIFY (not reconcile); validated on Buf, R22 140/140 (§64a)
MEASUREMENT CORRECTED THE PLAN. The checkpoint called for a "per-camp field-access reconcile";
measuring the camps refutes that: Vec8 = {s32 w[8]} (32B) in 180 files AND {s16 unk0..} (8B) in
139 files; MATRIX 48B/32B/32B; Buf 16B / 0x20+ / DrawEnv. These are DIFFERENT types sharing an
identifier across TUs of the same overlay — reconciling to a canonical layout MERGES them, the
same failure that broke 103 binaries on Prim. The right op is UNIQUIFY: rename the non-majority
camp (byte-neutral — a type name emits no code; TU-local by construction), which makes every camp
single-def and liftable by the existing lift_types rules.
- NEW tools/uniquify_type.py: deterministic camp ordering (file-count desc, then normalized text,
so re-runs assign the same suffixes); majority keeps the name, camp n -> <T>_c<n>; rewrites ONLY
files that DEFINE that camp (a file that merely USES the name gets it elsewhere and is untouched);
\bT\b word boundaries so `Buf` never matches `Buf80153978`.
- VALIDATED on Buf (578/6/1 files): 11 identifiers across 7 files -> 3 camps LIFTABLE -> lifted
(585 local copies stripped) -> R22 140/140 -> blocked core queue 13 -> 11 (0x8012ea90, 0x801749c8
freed). Propagated 0x8012EA90 ×138; 0x801749C8 dropped (straggler in ov_SC07_006).
- YIELD, HONESTLY (P9): ZERO new matched functions. fn-count 88.61% / instr 79.3% / stubs 40281 all
UNCHANGED; dedup 1867->1868, C1 +138. 0x8012EA90's members were ALREADY matched in all 138
overlays — the propagation consolidated duplication into one shared macro (DRY), not coverage.
The value is the PROVEN RECIPE + the queue moving 13->11, not the numbers.
- dedup_propagate (R32): the skip line printed a COUNT and no names, and aggregated three unrelated
causes into n_local — a body skipped merely for a `//` comment (macro-unsafe, 1-line fix) read
identically to one genuinely using an overlay-local type. Now named and split by cause.
- cookbook §64a (uniquify-vs-reconcile + the validated recipe + remaining camps by cost).
|
||
|
Drew T
|
82d79e7a32 |
fix(phase-26a): A6/A7 — the family engine could not see half its corpus; 17 fns banked x134 free
R22: check-all 136 PASSED / 0 FAILED. dedup-check 1823 validated / 0 failed (C1 coverage 224,933/224,933).
Fleet instr-weighted 66.5% -> 66.7%.
=== dedup_propagate: it was blind to HALF the corpus ===
overlay_files() used a hardcoded suffix allowlist ("_a","_o0","_o0b","_after") that predated the
Phase-26 jr carves -> 404 of the fleet's 811 overlay .c. The 407-file gap held 36,135 INCLUDE_ASM stubs
and ~32,000 inline defs, and overlay_files gates ALL of dedup_propagate (source_text / find_site /
apply_plan / struct_check / reconcile_caller_extern). Now a GLOB — never an allowlist, because the NEXT
split family would re-open it. The asm_subdir is always the file stem, an invariant the old four entries
already satisfied.
find_site's def-detector required the signature line to END in ')' and the next non-blank line to START
with '{'. It therefore silently dropped THREE shapes: K&R definitions (`s32 f(arg0)` / `s32 arg0;` / `{`),
multi-line signatures, and single-line bodies. K&R is the project's house style for exactly the biggest,
highest-reach functions — func_8015AE2C (562 ins), func_80166994, func_80133CD4, func_8015A3C8 — and they
live in the _jr_* files overlay_files could not even open. Fixing either alone would have been useless:
the glob exposes the files, and find_site would still drop their biggest prizes. Both fixed together.
* The signature's closing paren is now found by a real paren-walk, not line.count() or split(')')[-1]:
a single-line body containing a call (`void f(int a){ g(a); }`) has balanced parens of its own, so
both shortcuts land on the WRONG paren and then misread the body's ';' as a prototype terminator.
* AGREEMENT ASSERTION (the audit's): find_site vs family_remap.extract_unit -> 701 agree / 0 disagree.
Negative controls hold (a prototype+call is rejected; a 1-line body with a call is a def).
=== THE HARVEST (free work, byte-gated) ===
--auto-from ov_SC01_077 now nominates what it could never see: 20 planned, 17 propagated x134, 3 dropped
as cross-overlay stragglers. 134 overlays rebuilt BYTE-IDENTICAL; 17 new dedup groups.
Includes ALL FOUR functions A1 caught the registry lying about (func_80128ED8 / 8012C098 / 8012C0EC /
8012C750): 0 stubs remaining, real shared macros. THE LOOP CLOSES — A1 found the lie, and THIS is the
bug that had made it true (3 of the 4 are defined in ov_SC01_077_jr_8012ACE0.c, which the allowlist could
not open, so the propagation never ran and dedup_integrate greenlit the result).
=== family_remap: 96 PHANTOM exemplars -> 0 ===
extract_unit globbed only src/<ov>/<ov>*.c, so a function matched via a SHARED body had no source form
and read as NOT MATCHED. 93-96 of 218 h_seq "matched" exemplars were phantom, carrying 2,157 candidate
members of which 1,834 are still-stubbed, PURE/IMM-clean, symbol_map-clean and unpinned — staged and
gated today, dropped before the first build then. It is now TOTAL over BOTH shared-body mechanisms:
(1) the DEFINE_func_<ADDR>() macro — reconstructed as the exact INVERSE of dedup_propagate.make_macro
(derived from the generator, not re-guessed from the text);
(2) a DIRECT definition in a shared header, #included per overlay — the whale (func_80144B9C, 770 ins,
-O0), which the registry explicitly records as "NOT a DEFINE_ macro".
CENSUS: 216 matched exemplars, 216 real, 0 PHANTOM.
symbol_map named the symbol by HOW IT WAS LOADED, not by WHAT IT IS: reloc_targets labels every lui/%lo
pair "data", and a FUNCTION's address taken via lui/%lo (an address-taken callback) is exactly that shape
(splat's own .s: %lo(func_8017E1D4), 7 occurrences). The map got a D_<ADDR> key while the C writes
func_<ADDR>, so the word-bounded substitution matched NOTHING and silently no-op'd — the sibling kept the
EXEMPLAR's function pointer and the loss was booked as a BYTE failure, indistinguishable from a compiler
wall. Now emits both keys (addresses are unique; the pass is simultaneous, so the extra key is free).
gather_externs was line-oriented, so a WRAPPED comma extern was invisible in both directions (the first
line has no ';', the continuation has no `extern`). ov_SC01_077.c:271-272 declares NINE symbols that way,
and the exemplar referencing them (func_8013D178) is a 133-member family — every sibling was staged with
NO declaration, failed to compile, and bisect-stormed its whole gate group. Now statement-oriented, and
an unresolved symbol is REPORTED, never silently dropped.
=== family_sweep.stub_map / build_engine_types ===
stub_map: func_-only -> a curated-name stub read as "already matched" -> phantom exemplar. Now corpus-derived.
build_engine_types hard-exited on 1,070 of 1,470 type-bearing overlay .c (73%; the audit measured 573/709
= 81% on its narrower set) because 1,929 TAGGED-struct typedefs tripped a guard whose own comment asserts
"our source has only ANONYMOUS-struct typedefs" — true in Phase 20, false since the harvest agents started
writing tagged structs. inject_capped_externs routes every type-bearing body HERE as the type-heavy tail's
ONLY sanctioned unblocker, so the tail's unblocker could not run on the corpus the tail lives in.
A contained def (the typedef's span encloses the body) is liftable — it just must not be counted twice;
only a PARTIAL overlap is malformed. Verified on a file that used to hard-exit: 5 tagged typedefs folded +
forward-declared, 46 types written, exit 0.
** AND THE SHARPEST LESSON IN THE AUDIT: this one was never silent. It printed "[overlap] ... handle
manually" every single time. But the message reads like a rare edge case rather than a four-fifths
coverage failure, so nobody ever COUNTED it. A loud failure that nobody counts is exactly as
invisible as a silent one. R32 must be "assert your coverage", not merely "fail loud". **
R14 self-catches, recorded because I hit both while fixing them: my first shared-header scan read a macro
body's `extern void f(void); \` as a DEFINITION (the trailing continuation means the line does not end in
';', so the decl guard never fired) — the exact bug fixed at commit:0552, reintroduced by me and caught only
because the whale resolved from the WRONG file. Column-0 anchoring fixes it by construction. And my
phantom census returned 0/0 twice because I guessed the manifest schema instead of reading it.
|
||
|
Drew T
|
8e6658ee85 |
feat(phase-24): T7 §G — func_801770E0 banked ×134 (native DEFINE-macro path); fleet 65.95→65.99%
- func_801770E0 (152 ins, reach-134) propagated ×134 via dedup_propagate --recover. Chose the NATIVE DEFINE-macro path over a hand-rolled shared header: func_8014E048 (pins+asm) is already ×134 via a DEFINE macro, proving that path handles pin/asm -O2 giants (the whale needed a shared header only because it is -O0 -> separate object). Clean fleet check-all 136/136 (R22), dedup-check 1811->1812/0. - R14: func_8014E048 was ALREADY ×134 (T6 §A) — the whale-session handoff was stale; only func_801770E0 + func_801372B0 actually remained ×1. - tooling (reusable): dedup_propagate.overlay_files now also scans the whale-rollout _o0b/_after splits — post-whale-region fns (func_801770E0 in _after.c) were invisible for both source-def-find and stub-replacement. func_801770E0 extern block made contiguous (a comment between externs made find_site drop 5 externs; byte-neutral). |
||
|
Drew T
|
99ccc37480 |
feat(phase-24): T6.4 — find_site trailing-comment externs → func_8014E048 + func_80157580 ×134
- find_site extern-collection: allow a trailing `/* comment */` after the `;`. The comment-blind regex `^\s*extern\b.*;\s*$` stopped the backward scan at `extern u8 D_801152A8[]; /* canonical TU type */`, dropping every EARLIER extern → compiles_standalone failed on the now-undeclared callees/data (func_80135A4C, func_80133784, D_801152A8). R14: THIS — not "pin/asm" as the backlog framed it — was func_8014E048's real self-containment blocker. - propagated func_8014E048 (the T5b S11 pins+barrier crack) + func_80157580 ×134; dedup 1797→1799 groups (0 failed). CLEAN fleet check-all 136/136 BYTE-IDENTICAL (R22). Fleet byte-identical 65.40% → 65.48%. |
||
|
Drew T
|
5b3697553f |
feat(phase-24): T6 — 13 leaf-MATCH fns propagated ×134 (fleet 64.90→65.40%)
find_site + dedup_propagate --recover + build_engine_types fixes, then re-bank the 13 recover_integration leaf-MATCHes and propagate each across all 134 overlays. - find_site: match INDENTED inline defs (was column-0 only, silently dropping every recover_integration-banked def from propagation — T6 blocker 1). Unit-tested: indented defs match; indented call-exprs (if/assign/bare/return) correctly rejected. - dedup_propagate --recover: on a straggler byte-gate failure, FIRST no-proto that overlay's conflicting caller extern + re-gate (Part B, byte-neutral, same lever as fix_arity_callers --any-proto); else EXCLUDE only that overlay (Part A, ×N-1) rather than the historical all-or-nothing drop. Wired into gate_stage. - build_engine_types: comment-aware find_defs/find_typedefs (blank_comments). The generated header's own "...typedef lift" comment was captured as a bogus `typedef vec`, self-colliding and blocking every --strip. NOTE: full --strip still conflicts with the _a/_o0 split files, so split-file overlays need a TARGETED lift. - banked + propagated 13 fns ×134: func_8014F74C 801542A4 8015BE94 8015F380 80160F00 801653B8 80166244 8016E778 801732C4 8017331C 80173374 80174554 801745AC. func_8014F74C needed PosT/MoveT lifted to engine_types.h (targeted, byte-neutral). - dedup 1784→1797 groups (0 failed). CLEAN fleet check-all 136/136 BYTE-IDENTICAL (R22: make clean && extract-all && check-all). Fleet REAL 224073, byte-identical 65.40%. R14: the confounding stale-asm/ tree (13 missing .s) that masked the first --recover test is fixed by re-extract; the 3 pre-existing --auto-from stragglers (0x80174650/ 8012A018/80165CA0) are pin/asm + uncaptured-local-macro (SHB) bodies, correctly dropped. |
||
|
Drew T
|
5e5423e953 |
feat(phase-21): dedup_propagate drop-straggler retry — _a pipeline now hands-free
The 2nd straggler class (cont.4 wave-2): cross-overlay loose-typing COMPILE errors (not just the -O0 byte-mismatch class). The -O0 exclusion alone was insufficient. Refactor the propagation into a drop-straggler retry loop: extract apply_plan(subplan, restrict=); on a byte-gate failure, isolate the culprit(s) for the failing overlay (per-fn trial), drop them (kept x1), retry the survivors. Handles BOTH straggler classes failing ANY overlay; plan strictly shrinks (terminates); byte-gate stays sole arbiter (a dropped fn never banks where it isn't byte-identical). Validated end-to-end: wave-2 --auto-from auto-dropped 2 loose-typing stragglers + propagated 3 clean x134 (commit:0271); fleet 62.70 -> 62.82%. Future _a waves now auto-realize x134, no manual probe. |
||
|
Drew T
|
04c5c9b47f |
fix(phase-21): dedup_propagate skips -O0 overlay-local fns + gate_stage surfaces prop failures
The cont.4 root-cause fixes so future _a waves auto-realize ×134: - dedup_propagate --auto-from now excludes *_o0.c (-O0) defs: -O0 codegen embeds per-overlay %lo data, so masked h_exact falsely reports reach-134 (§18/§20). One such straggler (func_8013C360) reverted 10 clean ×134 matches under the all-or-nothing batch gate (cont.4). Detected via find_site on the -O0 split file; --addr still forces them. (6 fns excluded.) - gate_stage: capture dedup_propagate's exit; on a real byte-gate revert (not the benign 'nothing to propagate' no-op) write .run/auto/last_propagate_error.log + add prop_error to the summary + warn. A swallowed revert previously hid the gain silently. |
||
|
Drew T
|
cbf7e774ab |
feat(phase-21): split-aware dedup_propagate + cast_call_sites --src-file (unlock the _a.c vein)
- dedup_propagate split-aware: overlay_files() handles ov_SC01_077's Phase-19 split (main + _a/_o0); source-find scans all, member-loop edits the right file, structural-check spans all. Single-file overlays unchanged (default path). Validated: source-find (--check-only 134 members) + fail-closed revert. - cast_call_sites --src-file: canonicalize callee decls against the file the draft lands in (cross-file loose typing — a callee declared differently in main vs _a.c). Default = main, unchanged. - FINDING (cookbook §24): the fresh reach-134 fuel (66+ fns) is in ov_SC01_077_a.c (a tooling gap, not difficulty); bodies are matchable (func_8012C098 cast-banked) BUT matching/propagation hits the same §16/§20 loose-typing wall (func_8012C098 cross-overlay def-conflict -> x1; func_8012F274 within-_a RotTransSV multi-sig). x134 yield uncertain -> a small measurement wave is the next probe. - no net banks this turn; fleet 62.31% unchanged; ov_SC01_077 verified d19c9580 (R22). sig_unify reverted. |
||
|
Drew T
|
a10f7f8a1a |
feat(phase-15): sig-unify recovery + find_site brace-fix — fleet 52.08% -> 54.44%
- tools/sig_unify.py (NEW): unifies a draft's FULL signature set to the banked-canonical decls — both callee externs AND the draft's OWN definition signature (return + param types, body param-names preserved). Recovery for the standalone-MATCH residual: a probe found 30/30 sampled failures were type conflicts, 0 false-positives. Gated 191 -> 32 verified. - tools/dedup_propagate.py find_site FIX: the inline-def detector required the opening brace ON THE SAME LINE as the signature, silently dropping every next-line-brace def from propagation (sig_unify/permuter outputs + ~stragglers all session). Now accepts brace on the same OR next line -> unlocked a 61-function propagation backlog. - propagated 61 fns fleet-wide (incl. 32 sig-unify + the permuter win func_801508F8 + earlier-dropped next-line-brace matches), all 134 byte-identical; dedup-check 1394/0 - R22 CLEAN fleet rebuild: 136/136 byte-identical. REAL 177501 -> 185646. Zero agent tokens. |
||
|
Drew T
|
5c5e9fb973 |
feat(phase-15): T6 struct follow-up — shared engine_types.h unblocks 37 struct-using fns; fleet 46.27% -> 47.70%
- tools/build_engine_types.py -> src/shared/engine_types.h: extract the 34 named struct/union types from ov_SC01_077.c (forward decls + defs in source order; consumes trailing __attribute__((packed)) + aliases through the ';'). 0 same-name-different-layout collisions. - engine_core.h includes engine_types.h; the 34 inline defs stripped from ov_SC01_077.c (byte-neutral — type defs emit no code; verified byte-identical). - dedup_propagate.py: relax the struct guard to skip only inline named-struct/typedef DEFINITIONS (header-type USAGES + anonymous local structs now propagate); add engine_types.h to the compile pre-filter so struct-using bodies resolve. - 37 struct-using shared fns propagated fleet-wide (skip 41 -> 4 overlay-local typedef'd types), 134 overlays byte-identical; dedup 1183 -> 1220 groups, 1220 validated / 0 failed. - fleet REAL substantive 157,541 -> 162,462; byte-identical 46.27% -> 47.70%. - R22 clean rebuild: 136/136 byte-identical, 0 extract failures. |
||
|
Drew T
|
bdfb86dc4f |
feat(phase-15): T6 v2 — data-sig-aware harvest + struct-guarded propagate; fleet 31.22% -> 42.14%
- gen_harvest_targets.py: now also resolves canonical DATA-symbol declarations per target (scan the .s for D_XXXX refs -> `extern <type> D_XXXX;` from banked code) in addition to callee function sigs — the §14c(c) fix at the source, so agents declare both functions and data correctly and avoid the dominant remaining conflict class. - v2 harvest (callee+data-sig-aware) on the 591 small still-stub call-heavy fns + a gap-fill agent for 6 targets dropped by a connection-closed-mid-response failure (the retry-wave misses truncated-non-null results). Gate: 306 verified (ov_SC01_077 1554 -> 1248 stubs). - dedup_propagate.py: skip any struct/union-touching body — agents named structs with colliding generic names (`struct S`/`struct vec`) inline, so two macros' types redefine/conflict when instantiated in one overlay (ov_SC01_000 abort). 34 skip; struct-using shared fns stay banked in 077 (shared-types-header follow-up). - 281 functions propagated fleet-wide (134 overlays each, every overlay byte-gated); dedup 795 -> 1076 groups, dedup_integrate --check 1076 validated / 0 failed. - fleet REAL substantive 105,764 -> 143,322; byte-identical 31.22% -> 42.14%. - R22 clean rebuild: 136/136 byte-identical, 0 extract failures. - cookbook §12: gap-fill is now a MANDATORY post-step for every multi-agent run (reconcile produced-vs-expected: missing + truncated drafts) before gating. |
||
|
Drew T
|
aefe2eac18 |
feat(phase-15): T5a — bulk-propagate 553 shared functions fleet-wide (3.82% -> 22.14%)
- dedup_propagate --auto-from ov_SC01_077: propagated 553 already-matched, self-contained shared engine functions across all 134 overlays, each byte-gated. 9 functions skipped (use 077-local struct types -> not mechanically liftable; honest, P9). One R22 CLEAN rebuild: 136/136 byte-identical. - fleet REAL 947 -> 74,527; byte-identical 13,132 -> 76,174 / 344,010 = 22.14%; 561 dedup groups, dedup-check 561 validated / 0 failed - tool hardening for fleet scale: registry SHORTHAND (vram + binaries; dedup.us.yaml ~4k lines vs ~77k verbose); per-overlay apply (one read/write per file); gate-only-changed; skip already-registered (additive/resumable); compile pre-filter (body must build with common.h alone); group_members() reader (both forms) in dedup_integrate + progress; progress _DEDUP_CACHE (fleet 6m -> 7s) - the first bulk attempt fail-closed-reverted on a local-typed body (byte-gate working) -> added the compile filter; cookbook 14a documents the 5 fleet-bulk lessons - ghidra/ churn NOT staged (R23) |
||
|
Drew T
|
0c7619231d |
feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X). - proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all 18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test (corrupt shared body) -> make check FAILS then restores - Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a shared header recompiles dependents — closes a stale-build/false-pass gap the negative test exposed; output-neutral (R22 clean 18/18) - structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate can't catch under-application -> assert DEFINE present + stub gone - FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest - cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory - ghidra/ churn NOT staged (R23); build/asm/.run gitignored |