From 9794b13ed25d34b4d09fd7c0d4bcda101d9e761e Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 14 Jul 2026 09:40:38 -0600 Subject: [PATCH] =?UTF-8?q?fix(phase-26a):=20A3=20=E2=80=94=20the=20endgam?= =?UTF-8?q?e=20plan=20was=202.8x=20too=20big;=20the=20matched=20set=20is?= =?UTF-8?q?=20now=20DERIVED?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/family-manifest.md is the document the whole Phase-25/26 structural-family endgame was planned from. Its matched-set oracle asked ov_SC01_077 ALONE: matched := {h_exact of that one overlay's non-stub fns} | dedup hashes. So a function ABSENT from that overlay — or stubbed there but matched in the other 133 — came out "unmatched" and was ranked as live work. advertised real (derived) multi-member families 2,758 -> 1,495 "hidden leverage" 11.0 MB -> 3.9 MB matched-free lever 235/5.9 MB -> 57/1.0 MB 7.1 MB of the advertised leverage was DEAD WORK. And because `instances` counted every overlay carrying a function — including the ones where it was already banked — the byte-weight RANKING (the file's entire purpose: "draft these first") was sorted mostly on already-finished code, with the real targets buried underneath. The A2 audit predicted "true frontier: 1,475 families / 3.9 MB"; derived independently here it is 1,495 / 3.9 MB. R33: the invariant answers this with no oracle at all — an h_exact class is WORK iff at least ONE of its instances is still an INCLUDE_ASM stub. That also makes the dedup-hash union redundant (a dedup-shared member is by definition not a stub), so the `hash:` regex over config/dedup.us.yaml is DELETED. `instances` now counts only the members still to bank, so the leverage is the real x-N. family_hseq: stub scan -> corpus (+100 curated-name stubs the func_-only regex could not see; they had made 3 still-stubbed functions look like MATCHED exemplars, which every sweep then re-nominates, produces nothing from, and books as a silent skip). Its hardcoded "expect ~663/~186/~1.85M" self-check was a stale 2026-07-11 snapshot — 38 banking commits have landed since — and is now labelled a point-in-time reference, not an invariant. (Verified my change can only GROW the frontier: +100 stubs.) census_conflict_callees: scoped to src//.c alone, so it saw 13 of 264 stubs and reported "wave scope: 2 still-stub" when the truth is 57 — every downstream percentage computed against a denominator 96% too small. Now 0/57 (the audit's exact figure). Its 0-conflict answer was right BY LUCK; it is now right for a reason. MARKED FOR DELETION (R33): it re-derives from C text what reconcile_tu.py answers from the build, and its parse holes fail in the UNSAFE direction (an unknown callee is silently bucketed "conflict-free"). Delete once reconcile_tu is wired into its only consumer. R14 near-miss, recorded: my first census patch handed collect_stubs() a set of NAMES where it wanted ADDRESSES, so the membership test was always false and it printed 0/0. Caught only because 0 contradicted the audit's expected 57. A scanner that returns 0 is indistinguishable from a scanner that found nothing — which is the entire thesis of this audit, and it very nearly bit me while fixing it. --- docs/family-hseq.md | 106 +++++++++++++------------- docs/family-manifest.md | 124 +++++++++++++++---------------- tools/census_conflict_callees.py | 28 ++++++- tools/family_hseq.py | 21 ++++-- tools/family_manifest.py | 56 +++++++++----- 5 files changed, 191 insertions(+), 144 deletions(-) diff --git a/docs/family-hseq.md b/docs/family-hseq.md index ef330a638b..d303e13e54 100644 --- a/docs/family-hseq.md +++ b/docs/family-hseq.md @@ -2,64 +2,64 @@ > Generated by `tools/family_hseq.py` from the 134 overlay sigs + per-overlay src stubs. Ranked by TEMPLATABLE byte-weight (PURE+IMM members × nins × 4). The byte-gate is the arbiter. -**Fleet (overlays):** 82.5% fn / 63.6% instr / 40.5% distinct-code matched. Unmatched: 59,680 instances / 4,626,128 ins (34,021 distinct classes). +**Fleet (overlays):** 82.8% fn / 66.5% instr / 46.8% distinct-code matched. Unmatched: 58,621 instances / 4,257,491 ins (33,145 distinct classes). -**Tail cross-check (Phase-25 close):** 42,139 tail fns / 2,299,092 ins → 601 h_seq families ≥2, **179 substantial (nins≥80) / 1,699,870 ins**. +**Tail cross-check (Phase-25 close):** 41,366 tail fns / 1,972,283 ins → 597 h_seq families ≥2, **175 substantial (nins≥80) / 1,372,887 ins**. -**Full frontier (all unmatched by h_seq):** 2730 target families (≥2 members or a matched sibling) + 3553 singletons (Step-D residue). Substantial: **574 families / 3,015,589 templatable ins**, 29 with a matched sibling (zero-crack). Substantial member classes: 16,918 PURE · 39 IMM · 6 STRUCT-excluded. +**Full frontier (all unmatched by h_seq):** 2724 target families (≥2 members or a matched sibling) + 3554 singletons (Step-D residue). Substantial: **568 families / 2,646,704 templatable ins**, 30 with a matched sibling (zero-crack). Substantial member classes: 15,855 PURE · 39 IMM · 6 STRUCT-excluded. ## Top substantial families (by templatable byte-weight) | # | nins | members (P/I/S) | #addr/#ov | tag | class | matched | jr | exemplar | templ. ins | |--:|--:|--|--|--|--|--:|:-:|--|--:| -| 1 | 890 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80178d40 draft-ov077 | 119,260 | -| 2 | 952 | 113 (107/6/0) | 21/113 | scattered | IMM | 0 | Y | 0x8017bebc modal | 107,576 | -| 3 | 536 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8017a4ac draft-ov077 | 71,824 | -| 4 | 493 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015a3c8 draft-ov077 | 66,062 | -| 5 | 490 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013f350 draft-ov077 | 65,660 | -| 6 | 438 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x801380e0 draft-ov077 | 58,692 | -| 7 | 424 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80131340 draft-ov077 | 56,816 | -| 8 | 371 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80176734 draft-ov077 | 49,714 | -| 9 | 363 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015444c draft-ov077 | 48,642 | -| 10 | 337 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80159c84 draft-ov077 | 45,158 | -| 11 | 329 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013c414 draft-ov077 | 44,086 | -| 12 | 327 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80176218 draft-ov077 | 43,818 | -| 13 | 312 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013faf8 draft-ov077 | 41,808 | -| 14 | 304 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8014d820 draft-ov077 | 40,736 | -| 15 | 289 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80135eb0 draft-ov077 | 38,726 | -| 16 | 279 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x801670e4 draft-ov077 | 37,386 | -| 17 | 272 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013b83c draft-ov077 | 36,448 | -| 18 | 271 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015b950 draft-ov077 | 36,314 | -| 19 | 260 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80140958 draft-ov077 | 34,840 | -| 20 | 249 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013cb84 draft-ov077 | 33,366 | -| 21 | 240 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013d53c draft-ov077 | 32,160 | -| 22 | 231 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80175da8 draft-ov077 | 30,954 | -| 23 | 226 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8012956c draft-ov077 | 30,284 | -| 24 | 222 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015c32c draft-ov077 | 29,748 | -| 25 | 213 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013ffd8 draft-ov077 | 28,542 | -| 26 | 209 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8016cbc0 draft-ov077 | 28,006 | -| 27 | 204 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015ea3c draft-ov077 | 27,336 | -| 28 | 201 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015d738 draft-ov077 | 26,934 | -| 29 | 198 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013bd74 draft-ov077 | 26,532 | -| 30 | 198 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x801412a8 draft-ov077 | 26,532 | -| 31 | 188 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015f448 draft-ov077 | 25,192 | -| 32 | 188 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8016ab6c draft-ov077 | 25,192 | -| 33 | 188 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80175ab8 draft-ov077 | 25,192 | -| 34 | 93 | 268 (268/0/0) | 2/134 | cross-address | PURE | 0 | · | 0x8014ffdc draft-ov077 | 24,924 | -| 35 | 183 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8014032c draft-ov077 | 24,522 | -| 36 | 181 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80135a4c draft-ov077 | 24,254 | -| 37 | 174 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8017ae2c draft-ov077 | 23,316 | -| 38 | 173 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80154c24 draft-ov077 | 23,182 | -| 39 | 165 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80178004 draft-ov077 | 22,110 | -| 40 | 162 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80169bbc draft-ov077 | 21,708 | -| 41 | 161 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80160534 draft-ov077 | 21,574 | -| 42 | 161 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x8016dc20 matched-ov077 | 21,413 | -| 43 | 158 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x801299c8 draft-ov077 | 21,172 | -| 44 | 156 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x8016df5c matched-ov077 | 20,748 | -| 45 | 154 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013c0f8 draft-ov077 | 20,636 | -| 46 | 155 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x801365b8 matched-ov077 | 20,615 | -| 47 | 154 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x80144090 matched-ov077 | 20,482 | -| 48 | 148 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8016d1d8 draft-ov077 | 19,832 | -| 49 | 147 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80177b5c draft-ov077 | 19,698 | -| 50 | 141 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8014f4c0 draft-ov077 | 18,894 | +| 1 | 952 | 112 (106/6/0) | 21/112 | scattered | IMM | 1 | Y | 0x8017bebc matched | 106,624 | +| 2 | 536 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8017a4ac draft-ov077 | 71,824 | +| 3 | 490 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013f350 draft-ov077 | 65,660 | +| 4 | 424 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80131340 draft-ov077 | 56,816 | +| 5 | 371 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80176734 draft-ov077 | 49,714 | +| 6 | 337 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80159c84 draft-ov077 | 45,158 | +| 7 | 329 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013c414 draft-ov077 | 44,086 | +| 8 | 327 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80176218 draft-ov077 | 43,818 | +| 9 | 312 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013faf8 draft-ov077 | 41,808 | +| 10 | 304 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8014d820 draft-ov077 | 40,736 | +| 11 | 289 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80135eb0 draft-ov077 | 38,726 | +| 12 | 279 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x801670e4 draft-ov077 | 37,386 | +| 13 | 272 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013b83c draft-ov077 | 36,448 | +| 14 | 271 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015b950 draft-ov077 | 36,314 | +| 15 | 260 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80140958 draft-ov077 | 34,840 | +| 16 | 249 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013cb84 draft-ov077 | 33,366 | +| 17 | 240 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8013d53c draft-ov077 | 32,160 | +| 18 | 231 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80175da8 draft-ov077 | 30,954 | +| 19 | 226 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8012956c draft-ov077 | 30,284 | +| 20 | 222 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8015c32c draft-ov077 | 29,748 | +| 21 | 209 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8016cbc0 draft-ov077 | 28,006 | +| 22 | 204 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015ea3c draft-ov077 | 27,336 | +| 23 | 201 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015d738 draft-ov077 | 26,934 | +| 24 | 198 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013bd74 draft-ov077 | 26,532 | +| 25 | 198 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x801412a8 draft-ov077 | 26,532 | +| 26 | 188 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8015f448 draft-ov077 | 25,192 | +| 27 | 188 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80175ab8 draft-ov077 | 25,192 | +| 28 | 93 | 268 (268/0/0) | 2/134 | cross-address | PURE | 0 | · | 0x8014ffdc draft-ov077 | 24,924 | +| 29 | 183 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8014032c draft-ov077 | 24,522 | +| 30 | 181 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80135a4c draft-ov077 | 24,254 | +| 31 | 174 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8017ae2c draft-ov077 | 23,316 | +| 32 | 173 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80154c24 draft-ov077 | 23,182 | +| 33 | 165 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80178004 draft-ov077 | 22,110 | +| 34 | 162 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80169bbc draft-ov077 | 21,708 | +| 35 | 161 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80160534 draft-ov077 | 21,574 | +| 36 | 161 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x8016dc20 matched-ov077 | 21,413 | +| 37 | 158 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x801299c8 draft-ov077 | 21,172 | +| 38 | 156 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x8016df5c matched-ov077 | 20,748 | +| 39 | 154 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x8013c0f8 draft-ov077 | 20,636 | +| 40 | 155 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x801365b8 matched-ov077 | 20,615 | +| 41 | 154 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x80144090 matched-ov077 | 20,482 | +| 42 | 148 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8016d1d8 draft-ov077 | 19,832 | +| 43 | 147 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80177b5c draft-ov077 | 19,698 | +| 44 | 141 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8014f4c0 draft-ov077 | 18,894 | +| 45 | 141 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x8013d9b0 matched-ov077 | 18,753 | +| 46 | 139 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x8014ade0 draft-ov077 | 18,626 | +| 47 | 136 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x80135260 draft-ov077 | 18,224 | +| 48 | 137 | 133 (133/0/0) | 1/133 | per-location | PURE | 1 | · | 0x80133ab0 matched-ov077 | 18,221 | +| 49 | 131 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | Y | 0x801555f4 draft-ov077 | 17,554 | +| 50 | 129 | 134 (134/0/0) | 1/134 | per-location | PURE | 0 | · | 0x80137dd4 draft-ov077 | 17,286 | diff --git a/docs/family-manifest.md b/docs/family-manifest.md index 548121936b..ff94ff150a 100644 --- a/docs/family-manifest.md +++ b/docs/family-manifest.md @@ -2,69 +2,69 @@ > Generated by `tools/family_manifest.py` from the 134 overlay sigs + ov_SC01_077 matched set + `config/dedup.us.yaml`. One exemplar per h_norm family, drafted from its **ov_SC01_077 member** where present. h_norm families are reloc-only by construction; whether one C body matches all members is confirmed per-family by the byte-gate (T3 magnitude / T7 banks). **Leverage = `inst` (fleet-wide member count), NOT `hexR` (h_exact-reach, ~1-3 for byte-shattered families).** -**Families:** 44101 · **multi-member (hidden leverage):** 2758 / 11.0 MB · **with an ov_SC01_077 draftable exemplar:** 42. +**Families:** 23633 · **multi-member (hidden leverage):** 1495 / 3.9 MB · **with an ov_SC01_077 draftable exemplar:** 17. -**Endgame levers** (multi-member families × ov_SC01_077 membership): **draftable** 42 / 3.0 MB (unmatched ov077 member → draft the exemplar, T5/T7) · **matched-free** 235 / 5.9 MB (matched ov077 member + unmatched siblings → free `--tier h_norm` propagation, T3) · **absent** 2481 / 2.1 MB (no ov077 member → draft from another overlay). +**Endgame levers** (multi-member families × ov_SC01_077 membership): **draftable** 17 / 1.1 MB (unmatched ov077 member → draft the exemplar, T5/T7) · **matched-free** 57 / 1.0 MB (matched ov077 member + unmatched siblings → free `--tier h_norm` propagation, T3) · **absent** 1421 / 1.9 MB (no ov077 member → draft from another overlay). | # | rep addr | ov077 exemplar | nins | hexcls | inst | hexR | byteMB | bucket | |--:|----------|----------------|-----:|-------:|-----:|-----:|-------:|--------| -| 1 | 0x80133cd4 | 0x80133cd4 | 399 | 127 | 134 | 3 | 0.21 | GIANT | -| 2 | 0x80166994 | 0x80166994 | 369 | 127 | 134 | 3 | 0.20 | GIANT | -| 3 | 0x8013c414 | 0x8013c414 | 329 | 127 | 134 | 3 | 0.18 | GIANT | -| 4 | 0x80150480 | 0x8014fbc0 | 22 | 1891 | 1996 | 3 | 0.18 | small | -| 5 | 0x8014d820 | 0x8014d820 | 304 | 126 | 134 | 3 | 0.16 | GIANT | -| 6 | 0x8016a290 | — | 284 | 125 | 133 | 3 | 0.15 | GIANT | -| 7 | 0x801670e4 | 0x801670e4 | 279 | 126 | 134 | 3 | 0.15 | GIANT | -| 8 | 0x80130d48 | — | 266 | 125 | 133 | 3 | 0.14 | GIANT | -| 9 | 0x80135480 | 0x80135480 | 258 | 127 | 134 | 3 | 0.14 | GIANT | -| 10 | 0x80163ec8 | 0x80163ec8 | 234 | 127 | 134 | 3 | 0.13 | GIANT | -| 11 | 0x80134c20 | — | 230 | 126 | 133 | 3 | 0.12 | GIANT | -| 12 | 0x8016cbc0 | 0x8016cbc0 | 209 | 127 | 134 | 3 | 0.11 | GIANT | -| 13 | 0x80133784 | — | 203 | 126 | 133 | 3 | 0.11 | GIANT | -| 14 | 0x8013bd74 | 0x8013bd74 | 198 | 127 | 134 | 3 | 0.11 | GIANT | -| 15 | 0x80166690 | 0x80166690 | 193 | 127 | 134 | 3 | 0.10 | GIANT | -| 16 | 0x80150170 | 0x8014fe60 | 95 | 2 | 268 | 134 | 0.10 | med | -| 17 | 0x8013b274 | — | 189 | 125 | 133 | 3 | 0.10 | GIANT | -| 18 | 0x8013dd68 | — | 187 | 126 | 133 | 3 | 0.10 | GIANT | -| 19 | 0x80167ae0 | — | 183 | 125 | 133 | 3 | 0.10 | GIANT | -| 20 | 0x80161e08 | — | 175 | 126 | 133 | 3 | 0.09 | GIANT | -| 21 | 0x80167dbc | — | 173 | 125 | 133 | 3 | 0.09 | GIANT | -| 22 | 0x8016c49c | — | 172 | 126 | 133 | 3 | 0.09 | GIANT | -| 23 | 0x8016cf04 | 0x8016cf04 | 166 | 127 | 134 | 3 | 0.09 | GIANT | -| 24 | 0x80136334 | 0x80136334 | 161 | 126 | 134 | 3 | 0.09 | GIANT | -| 25 | 0x801347a0 | — | 162 | 125 | 133 | 3 | 0.09 | GIANT | -| 26 | 0x8016dc20 | — | 161 | 126 | 133 | 3 | 0.09 | GIANT | -| 27 | 0x80162438 | — | 160 | 126 | 133 | 3 | 0.09 | GIANT | -| 28 | 0x8016df5c | 0x8016df5c | 156 | 127 | 134 | 3 | 0.08 | GIANT | -| 29 | 0x801365b8 | 0x801365b8 | 155 | 126 | 134 | 3 | 0.08 | GIANT | -| 30 | 0x80136824 | — | 156 | 125 | 133 | 3 | 0.08 | GIANT | -| 31 | 0x8013c0f8 | 0x8013c0f8 | 154 | 127 | 134 | 3 | 0.08 | GIANT | -| 32 | 0x8016d1d8 | 0x8016d1d8 | 148 | 127 | 134 | 3 | 0.08 | large | -| 33 | 0x8012c890 | — | 149 | 125 | 133 | 3 | 0.08 | large | -| 34 | 0x8013d9b0 | 0x8013d9b0 | 141 | 127 | 134 | 3 | 0.08 | large | -| 35 | 0x80133ab0 | — | 137 | 126 | 133 | 3 | 0.07 | large | -| 36 | 0x80140f00 | — | 128 | 125 | 133 | 3 | 0.07 | large | -| 37 | 0x80145cec | — | 127 | 125 | 133 | 3 | 0.07 | large | -| 38 | 0x8017d960 | — | 3338 | 5 | 5 | 1 | 0.07 | GIANT | -| 39 | 0x8013eb7c | — | 124 | 125 | 133 | 3 | 0.07 | large | -| 40 | 0x80145a2c | — | 62 | 250 | 266 | 3 | 0.07 | med | -| 41 | 0x8013ad38 | — | 122 | 125 | 133 | 3 | 0.06 | large | -| 42 | 0x8013d178 | — | 110 | 126 | 133 | 3 | 0.06 | large | -| 43 | 0x8013221c | 0x80132144 | 27 | 506 | 539 | 3 | 0.06 | small | -| 44 | 0x8013ef88 | — | 108 | 125 | 133 | 3 | 0.06 | large | -| 45 | 0x80168828 | — | 108 | 125 | 133 | 3 | 0.06 | large | -| 46 | 0x80134a74 | — | 107 | 126 | 133 | 3 | 0.06 | large | -| 47 | 0x801345f8 | — | 106 | 126 | 133 | 3 | 0.06 | large | -| 48 | 0x80141100 | — | 106 | 125 | 133 | 3 | 0.06 | large | -| 49 | 0x8017b614 | — | 101 | 126 | 133 | 3 | 0.05 | large | -| 50 | 0x8017b490 | 0x8017b490 | 97 | 127 | 134 | 3 | 0.05 | med | -| 51 | 0x8013dbe4 | — | 97 | 126 | 133 | 3 | 0.05 | med | -| 52 | 0x801596f0 | — | 97 | 125 | 133 | 3 | 0.05 | med | -| 53 | 0x8016bfd0 | — | 95 | 125 | 133 | 3 | 0.05 | med | -| 54 | 0x8013ee10 | — | 94 | 125 | 133 | 3 | 0.05 | med | -| 55 | 0x801418f8 | 0x801418f8 | 90 | 127 | 134 | 3 | 0.05 | med | -| 56 | 0x8013d3d4 | — | 90 | 126 | 133 | 3 | 0.05 | med | -| 57 | 0x8013e6ac | — | 90 | 125 | 133 | 3 | 0.05 | med | -| 58 | 0x80131b14 | 0x80131b14 | 89 | 126 | 134 | 3 | 0.05 | med | -| 59 | 0x80135004 | — | 89 | 126 | 133 | 3 | 0.05 | med | -| 60 | 0x801397b0 | — | 89 | 125 | 133 | 3 | 0.05 | med | +| 1 | 0x8013c414 | 0x8013c414 | 329 | 127 | 134 | 3 | 0.18 | GIANT | +| 2 | 0x8014d820 | 0x8014d820 | 304 | 126 | 134 | 3 | 0.16 | GIANT | +| 3 | 0x801670e4 | 0x801670e4 | 279 | 126 | 134 | 3 | 0.15 | GIANT | +| 4 | 0x8016cbc0 | 0x8016cbc0 | 209 | 127 | 134 | 3 | 0.11 | GIANT | +| 5 | 0x8013bd74 | 0x8013bd74 | 198 | 127 | 134 | 3 | 0.11 | GIANT | +| 6 | 0x8016dc20 | — | 161 | 126 | 133 | 3 | 0.09 | GIANT | +| 7 | 0x8016df5c | — | 156 | 126 | 133 | 3 | 0.08 | GIANT | +| 8 | 0x8013c0f8 | 0x8013c0f8 | 154 | 127 | 134 | 3 | 0.08 | GIANT | +| 9 | 0x801365b8 | — | 155 | 125 | 133 | 3 | 0.08 | GIANT | +| 10 | 0x8016d1d8 | 0x8016d1d8 | 148 | 127 | 134 | 3 | 0.08 | large | +| 11 | 0x8013d9b0 | — | 141 | 126 | 133 | 3 | 0.08 | large | +| 12 | 0x80133ab0 | — | 137 | 126 | 133 | 3 | 0.07 | large | +| 13 | 0x8017d960 | — | 3338 | 5 | 5 | 1 | 0.07 | GIANT | +| 14 | 0x8013d178 | — | 110 | 126 | 133 | 3 | 0.06 | large | +| 15 | 0x8013ef88 | — | 108 | 125 | 133 | 3 | 0.06 | large | +| 16 | 0x80134a74 | — | 107 | 126 | 133 | 3 | 0.06 | large | +| 17 | 0x8017b490 | — | 97 | 126 | 133 | 3 | 0.05 | med | +| 18 | 0x801418f8 | — | 90 | 126 | 133 | 3 | 0.05 | med | +| 19 | 0x80143d28 | — | 80 | 125 | 133 | 3 | 0.04 | med | +| 20 | 0x801457a4 | 0x801457a4 | 79 | 126 | 134 | 3 | 0.04 | med | +| 21 | 0x8016c188 | — | 79 | 126 | 133 | 3 | 0.04 | med | +| 22 | 0x80141a60 | — | 76 | 125 | 133 | 3 | 0.04 | med | +| 23 | 0x80156044 | — | 74 | 125 | 133 | 3 | 0.04 | med | +| 24 | 0x8012e364 | 0x8012e364 | 67 | 127 | 134 | 3 | 0.04 | med | +| 25 | 0x80165240 | 0x80165240 | 63 | 126 | 134 | 3 | 0.03 | med | +| 26 | 0x8016d688 | 0x8016d688 | 60 | 127 | 134 | 3 | 0.03 | med | +| 27 | 0x801801e8 | — | 195 | 34 | 36 | 2 | 0.03 | GIANT | +| 28 | 0x8013d8fc | — | 45 | 126 | 133 | 3 | 0.02 | small | +| 29 | 0x8017da90 | — | 227 | 25 | 26 | 2 | 0.02 | GIANT | +| 30 | 0x80186b78 | — | 236 | 20 | 20 | 1 | 0.02 | GIANT | +| 31 | 0x801549f8 | 0x801549f8 | 31 | 127 | 134 | 3 | 0.02 | small | +| 32 | 0x8017cd9c | — | 793 | 5 | 5 | 1 | 0.02 | GIANT | +| 33 | 0x80180a64 | — | 766 | 3 | 5 | 2 | 0.02 | GIANT | +| 34 | 0x8017d8b8 | 0x8017c294 | 246 | 6 | 15 | 5 | 0.01 | GIANT | +| 35 | 0x8017cdd8 | — | 113 | 28 | 32 | 2 | 0.01 | large | +| 36 | 0x80180324 | — | 328 | 11 | 11 | 1 | 0.01 | GIANT | +| 37 | 0x80187e4c | — | 237 | 15 | 15 | 1 | 0.01 | GIANT | +| 38 | 0x8017c9bc | — | 253 | 13 | 14 | 2 | 0.01 | GIANT | +| 39 | 0x80164e40 | 0x80164e40 | 25 | 127 | 134 | 3 | 0.01 | small | +| 40 | 0x801808e0 | — | 92 | 11 | 36 | 10 | 0.01 | med | +| 41 | 0x80180034 | — | 90 | 11 | 36 | 10 | 0.01 | med | +| 42 | 0x8017c43c | — | 293 | 10 | 11 | 2 | 0.01 | GIANT | +| 43 | 0x8013bc7c | — | 24 | 125 | 133 | 3 | 0.01 | small | +| 44 | 0x8017efb0 | — | 223 | 14 | 14 | 1 | 0.01 | GIANT | +| 45 | 0x8017f510 | — | 1511 | 2 | 2 | 1 | 0.01 | GIANT | +| 46 | 0x8013bcdc | — | 22 | 125 | 133 | 3 | 0.01 | small | +| 47 | 0x8017ff08 | — | 263 | 11 | 11 | 1 | 0.01 | GIANT | +| 48 | 0x80185150 | — | 185 | 13 | 15 | 2 | 0.01 | GIANT | +| 49 | 0x8017f9bc | — | 386 | 6 | 7 | 2 | 0.01 | GIANT | +| 50 | 0x80188e6c | — | 168 | 15 | 16 | 2 | 0.01 | GIANT | +| 51 | 0x801849ac | — | 177 | 13 | 15 | 2 | 0.01 | GIANT | +| 52 | 0x8017e1dc | — | 96 | 25 | 26 | 2 | 0.01 | med | +| 53 | 0x8017d1a0 | — | 10 | 237 | 243 | 3 | 0.01 | tiny | +| 54 | 0x8013b7f4 | — | 18 | 126 | 133 | 3 | 0.01 | tiny | +| 55 | 0x8017cdd8 | — | 17 | 116 | 137 | 3 | 0.01 | tiny | +| 56 | 0x80188c28 | — | 145 | 15 | 16 | 2 | 0.01 | large | +| 57 | 0x80188440 | — | 154 | 15 | 15 | 1 | 0.01 | GIANT | +| 58 | 0x80180474 | — | 164 | 14 | 14 | 1 | 0.01 | GIANT | +| 59 | 0x80188850 | — | 147 | 15 | 15 | 1 | 0.01 | large | +| 60 | 0x80188200 | — | 144 | 15 | 15 | 1 | 0.01 | large | diff --git a/tools/census_conflict_callees.py b/tools/census_conflict_callees.py index c1532cbafb..e29a46745d 100644 --- a/tools/census_conflict_callees.py +++ b/tools/census_conflict_callees.py @@ -21,6 +21,9 @@ import argparse, json, os, importlib.util from collections import defaultdict REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +import sys +sys.path.insert(0, os.path.join(REPO, 'tools')) +import corpus # the derived corpus oracle (Phase 26-A) _spec = importlib.util.spec_from_file_location('ght', os.path.join(REPO, 'tools/gen_harvest_targets.py')) _ght = importlib.util.module_from_spec(_spec) @@ -36,15 +39,32 @@ def main(): args = ap.parse_args() src = args.source - c_path = os.path.join(REPO, f'src/{src}/{src}.c') ec = os.path.join(REPO, 'src/shared/engine_core.h') + # ⚠ DEPRECATED — SCHEDULED FOR DELETION (Phase 26-A audit, R33). + # This tool re-derives, by re-parsing C text, the question "for this TU, which callees are + # defined / declared / stubbed / external?" — which is exactly and only what tools/reconcile_tu.py + # answers FROM THE BUILD. A parse hole in reconcile_tu makes it CONSERVATIVE; a parse hole here + # makes it WRONG IN THE UNSAFE DIRECTION (an unknown callee is silently bucketed "conflict-free"). + # Delete this file once reconcile_tu is wired into derive_canonical_sigs (its only consumer). + # + # Until then, at least stop the 96% under-report: the corpus was `src//.c` ALONE, so it + # saw 13 of ov_SC01_077's 264 stubs and printed "wave scope: 2 still-stub" when the truth is 57 — + # every downstream percentage was computed against a denominator 96% too small. The 0-conflict + # answer it gives today is right BY LUCK (those callees happen to be banked); point it at a fresh + # overlay and it would see ~14 of ~600 stubs and confidently report zero conflicts into a wave + # riddled with them — the exact failure that cost 60%-vs-33% in the Phase-17 calibration. + c_paths = [str(p) for p in corpus.src_files(src)] + c_path = c_paths[0] if c_paths else os.path.join(REPO, f'src/{src}/{src}.c') + src_sig = _ght.load_sig(os.path.join(REPO, f'.run/sig.{src}.jsonl')) # addr-int -> {calls,nins,h_exact,reach?} - stubs = _ght.collect_stubs(c_path) # current INCLUDE_ASM set + stubs = set(corpus.stubs(src)) # ADDRESSES (ints) — ALL TUs, ANY symbol name define_sigs = _ght.collect_define_sigs(ec) - inline_sigs = _ght.collect_inline_sigs(c_path) + inline_sigs = {} + for cp in c_paths: + inline_sigs.update(_ght.collect_inline_sigs(cp)) defined = {**define_sigs, **inline_sigs} - extern_sigs = _ght.collect_extern_sigs([ec, c_path]) + extern_sigs = _ght.collect_extern_sigs([ec] + c_paths) s3 = json.load(open(os.path.join(REPO, args.targets))) tgt_addrs = [int(t['addr'], 16) for t in s3] diff --git a/tools/family_hseq.py b/tools/family_hseq.py index 3f929f5d6b..952ce17e85 100644 --- a/tools/family_hseq.py +++ b/tools/family_hseq.py @@ -20,7 +20,7 @@ arbiter (G3/P9): this survey RANKS and CLASSES; it never asserts a match. Ground truth = the sigs (`make sig-overlays`) + src stubs. Companion to tools/family_manifest.py (h_norm). """ -import json, glob, re, collections, sys +import os, sys, json, glob, re, collections, sys sys.path.insert(0, "tools") import family_remap as FR @@ -29,13 +29,18 @@ TINY = 16 # nins < this is the coincidental-h_seq-collision band EX_OV = "ov_SC01_077" # the canonical drafting overlay (cached Ghidra-C) +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import corpus # the derived corpus oracle (Phase 26-A) + + def load(): """-> instances[(ov, addr, nins, h_exact, h_norm, h_seq, matched)], and nins/matched maps.""" - stubs = {} - for ovdir in sorted(glob.glob("src/ov_*")): - ov = ovdir.split("/")[-1] - txt = "".join(open(f).read() for f in glob.glob(f"{ovdir}/*.c")) - stubs[ov] = set(int(m, 16) for m in re.findall(r'INCLUDE_ASM\([^)]*,\s*func_([0-9A-Fa-f]+)\)', txt)) + # DERIVED from tools/corpus.py (Phase 26-A). The old scan globbed every .c correctly but its + # regex only matched `func_` symbols — so the 100 CURATED-name stubs (listCdBuffer) were + # invisible, and family_hseq therefore labelled 3 still-stubbed functions as MATCHED exemplars. + # A phantom exemplar is re-nominated by every sweep, produces nothing, and books a silent skip. + stubs = {ov: set(corpus.stubs(ov)) for ov in + (d.split("/")[-1] for d in sorted(glob.glob("src/ov_*")))} inst = [] for p in sorted(glob.glob(".run/sig.ov_*.jsonl")): ov = f"ov_{p.split('sig.ov_')[1][:-6]}" @@ -197,7 +202,9 @@ def main(): print(f"fleet: {metrics['fn_count_matched_pct']}% fn / {metrics['instr_weighted_matched_pct']}% instr / " f"{metrics['distinct_matched_pct']}% distinct") print(f"tailcheck: {tailcheck['hseq_families_ge2']} families ≥2, {tailcheck['substantial_families']} " - f"substantial / {tailcheck['substantial_ins']:,} ins (expect ~663 / ~186 / ~1.85M)") + f"substantial / {tailcheck['substantial_ins']:,} ins" + f" [Task-2 snapshot 2026-07-11: 663 / 186 / 1.85M — this number SHRINKS as banking " + f"proceeds; it is a point-in-time reference, NOT an invariant to match]") print(f"frontier: {len(multi)} target families ({len(subst)} substantial, {len(with_matched)} w/ matched sib) " f"+ {len(singles)} singletons") print(f"-> .run/family_hseq.json + docs/family-hseq.md") diff --git a/tools/family_manifest.py b/tools/family_manifest.py index 89210a9d56..97ee91ca21 100644 --- a/tools/family_manifest.py +++ b/tools/family_manifest.py @@ -15,7 +15,10 @@ many overlays carry it), NOT h_exact-reach (which is ~1-3 for a byte-shattered f Ground truth = the sigs (they sign the ORIGINAL bytes, stable across matching); reproducible via `make sig-overlays`. Companion to the frontier surveys .run/{analyze_frontier,probe_hnorm}.py. """ -import json, glob, re, collections +import json, glob, os, re, sys, collections + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import corpus # the derived corpus oracle (Phase 26-A) sig_by_ov = {} for p in sorted(glob.glob(".run/sig.ov_*.jsonl")): @@ -29,33 +32,50 @@ for p in sorted(glob.glob(".run/sig.ov_*.jsonl")): recs[int(d["addr"], 16)] = (d["nins"], d["h_exact"], d["h_norm"]) sig_by_ov[ov] = recs -# matched set (h_exact): ov_SC01_077's non-stub functions + every dedup-registered hash -src077 = "".join(open(cf).read() for cf in glob.glob("src/ov_SC01_077/*.c")) -stub077 = set(int(m, 16) for m in re.findall(r'INCLUDE_ASM\([^)]*,\s*func_([0-9A-Fa-f]+)\)', src077)) +# ---- THE MATCHED SET, DERIVED (Phase 26-A audit, HIGH — R33) -------------------------------------- +# The old oracle asked ov_SC01_077 ALONE: matched := {h_exact of its non-stub fns} | dedup hashes. +# That is the wrong question. A function absent from that one overlay — or stubbed there but matched +# in the 133 others — came out "unmatched", so it was ranked as live work. +# +# MEASURED CONSEQUENCE: docs/family-manifest.md, the document the whole Phase-25/26 endgame is planned +# from, advertised "2,758 multi-member families / 11.0 MB of hidden leverage". 1,071 of them (6.80 MB, +# **62% of the advertised byte-weight**) were ALREADY FULLY MATCHED — pure phantom targets. A further +# 253 had their byte-weight inflated by already-matched members. The byte-weight RANKING — the entire +# purpose of the file, "draft these first" — was therefore sorted mostly on dead work, with the real +# targets buried underneath. +# +# The invariant answers it exactly, per overlay, with no oracle at all: +# an h_exact class is WORK iff at least ONE of its instances is still an INCLUDE_ASM stub somewhere. +# (And the dedup-hash union is now redundant: a dedup-shared member is by definition not a stub. One +# more scanner deleted — the `hash:` regex over config/dedup.us.yaml is gone.) +stub_by_ov = {ov: set(corpus.stubs(ov)) for ov in sig_by_ov} sig077 = sig_by_ov.get("ov_SC01_077", {}) -matched_hex = {hx for v, (n, hx, hn) in sig077.items() if v not in stub077} -matched_hex |= set(re.findall(r'hash:\s*([0-9a-f]{40})', open("config/dedup.us.yaml").read())) +stub077 = stub_by_ov.get("ov_SC01_077", set()) -# per h_exact class: {ovs, nins, rep_vram (ov_SC01_077's if present), h_norm} +# per h_exact class: which overlays carry it, and in which is it STILL A STUB hexcls = {} for ov, recs in sig_by_ov.items(): + st = stub_by_ov[ov] for v, (n, hx, hn) in recs.items(): - e = hexcls.setdefault(hx, [set(), n, v, hn]) - e[0].add(ov) + e = hexcls.setdefault(hx, {"ovs": set(), "stub_ovs": set(), "n": n, "rep": v, "hn": hn}) + e["ovs"].add(ov) + if v in st: + e["stub_ovs"].add(ov) if ov == "ov_SC01_077": - e[2] = v + e["rep"] = v # UNMATCHED h_exact classes -> group by h_norm (the structural family) fam = collections.defaultdict(lambda: {"hexclasses": 0, "instances": 0, "nins": 0, "bw": 0, "rep": 0, "maxhexreach": 0}) -for hx, (ovs, n, v, hn) in hexcls.items(): - if hx in matched_hex: - continue - r = len(ovs) +for hx, e in hexcls.items(): + if not e["stub_ovs"]: + continue # fully matched fleet-wide -> NOT work (was: counted as work) + n, hn, v = e["n"], e["hn"], e["rep"] + r = len(e["stub_ovs"]) # the REAL x-N leverage: only the members still to bank f = fam[hn] f["hexclasses"] += 1 - f["instances"] += r # fleet-wide member count = the x-N propagation leverage + f["instances"] += r f["bw"] += r * n * 4 - f["maxhexreach"] = max(f["maxhexreach"], r) + f["maxhexreach"] = max(f["maxhexreach"], len(e["ovs"])) if n >= f["nins"] or f["rep"] == 0: f["rep"] = v f["nins"] = n @@ -63,7 +83,7 @@ for hx, (ovs, n, v, hn) in hexcls.items(): # the draftable exemplar per family: the largest UNMATCHED ov_SC01_077 stub whose h_norm == the family ov077_ex = {} # h_norm -> (vram, nins) for v, (n, hx, hn) in sig077.items(): - if v not in stub077 or hx in matched_hex: + if v not in stub077: continue if hn not in ov077_ex or n > ov077_ex[hn][1]: ov077_ex[hn] = (v, n) @@ -95,7 +115,7 @@ multi = [r for r in rows if r["multi_member"]] with_ex = [r for r in multi if r["ov077_exemplar"]] # classify the multi-member families by ov_SC01_077 membership -> the endgame's 3 levers -ov077_matched_hn = {hn for v, (n, hx, hn) in sig077.items() if v not in stub077 or hx in matched_hex} +ov077_matched_hn = {hn for v, (n, hx, hn) in sig077.items() if v not in stub077} lev = {"draftable": [0, 0], "matched_free": [0, 0], "absent": [0, 0]} for r in multi: k = "draftable" if r["ov077_exemplar"] else ("matched_free" if r["hnorm"] in ov077_matched_hn else "absent")