mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-04 00:20:10 -04:00
T2: check-casts and residue gates, kept-cast macro set, standards, task log and summary
This commit is contained in:
Drew T
@@ -280,7 +280,7 @@
|
||||
}
|
||||
},
|
||||
"coverage_ok": true,
|
||||
"elapsed_s": 25.4,
|
||||
"elapsed_s": 31.7,
|
||||
"generated": "2026-10-01",
|
||||
"gte_levers": {
|
||||
"direct": 356,
|
||||
@@ -291,7 +291,7 @@
|
||||
"via_macro": 94,
|
||||
"what": "GTE ops whose clobbers exceed the canonical macro's (a scheduling steer): class-B levers INSIDE the headline number since T5 (2026-09-09), marked, 0 at the close"
|
||||
},
|
||||
"head": "b5e18e5f83",
|
||||
"head": "7845da8f03",
|
||||
"headers": 3181,
|
||||
"levers_AB": {
|
||||
"asm": 1861,
|
||||
@@ -473,7 +473,7 @@
|
||||
"volatile_qualified": 0,
|
||||
"zero": 54
|
||||
},
|
||||
"src_stamp": "287e6f521d908e6d",
|
||||
"src_stamp": "64f7b70e3eb5b45d",
|
||||
"tus": 4121,
|
||||
"unclassified": 0,
|
||||
"union_AD": {
|
||||
|
||||
@@ -12,7 +12,7 @@ lever_census: 218 binaries · 4,121 TUs + 3,181 headers · coverage OK · unclas
|
||||
E asm-label 5986 898 5088 730 127 2010 0 5986 {'asm-label': 5986}
|
||||
F builtin 302 302 0 285 25 285 0 302 {'builtin': 302}
|
||||
G attribute 26 0 26 0 0 2 0 26 {'attribute': 26}
|
||||
UNION A–D: 5,603 bodies · 1,168 distinct (addresses normalized) · 107 multi-copy classes holding 4,542 bodies · by kind {'ov': 5214, 'md': 105, 'main': 120, 'resident': 6, 'shared': 158}
|
||||
UNION A–D: 5,603 bodies · 1,168 distinct (addresses normalized) · 107 multi-copy classes holding 4,542 bodies · by kind {'ov': 5214, 'shared': 158, 'main': 120, 'md': 105, 'resident': 6}
|
||||
THE PHASE'S NUMBER (pins + asm statements, GTE excluded): 3,729 sites in 1,837 bodies (661 distinct) · marked !FAKE 3,729 · UNMARKED 0
|
||||
orphan !FAKE markers (no pin/asm site on the line nor below): 0
|
||||
marked ordinary-C fakes kept by Drew's S104 ruling (a) (`do { } while (0)`, dead initialisers; NOT levers): 53
|
||||
@@ -29,4 +29,4 @@ lever_census: 218 binaries · 4,121 TUs + 3,181 headers · coverage OK · unclas
|
||||
src/shared/ov/func_80178004.h pins got 8 expected 8 OK
|
||||
ov_SC03_006 func_801896EC bare-name pins got 2 expected 2 OK
|
||||
engine_prelude.h asm sites (a macro definition only) got 0 expected 0 OK
|
||||
elapsed 25.4 s
|
||||
elapsed 31.7 s
|
||||
|
||||
File diff suppressed because one or more lines are too long
+1176
-19
File diff suppressed because it is too large
Load Diff
@@ -1,14 +1,14 @@
|
||||
type_census — 2026-10-01T16:47:07Z at 0cb2b6b522 (src stamp 287e6f521d908e6d, tool 17a2928ffd); 7306 files (4121 TUs + 3181 headers), 99109 function definitions; walk 42.1 s, total 224.1 s
|
||||
type_census — 2026-10-01T17:32:55Z at 7845da8f03 (src stamp 64f7b70e3eb5b45d, tool a018054562); 7306 files (4121 TUs + 3181 headers), 99109 function definitions; walk 57.6 s, total 276.0 s
|
||||
|
||||
DEFINITIONS 1650 struct/union bodies (+0 enums): canonical header 1021 · .c file-scope 0 · .c block-scope 600 · per-function shared headers 29 · other headers 0 · include/ 326
|
||||
1024 distinct names · 523 distinct layouts (0 unresolved) · 53 duplicate classes covering 496 names (tier 1 0 same-named, tier 2 53 opaque; 44 layout-twin classes / 472 names kept apart) · 42 exact-text classes / 546 copies · 0 VARIANT names (>1 layout) · 56 dead canonical names · 0 canonical names redefined in .c · 4300 scalar/alias typedefs · 145 forward decls
|
||||
CAST SITES deref 502983 = P 408974 + I 60666 + X 13800 + M 19543 (+ A address-of 18892; + C typed cast-member 36853) in 69492 bodies · coverage OK (raw {'P': 408974, 'I': 60666, 'X': 13800, 'A': 18892, 'C': 36853, 'M': 19543}) · refused 0 · absolute 0 · readability regex over raw text 411816
|
||||
by base: param 276840 · local 149900 · gaddr 40123 · nested 24076 · global 8861 · other 1677 · addr-of 1350 · ident 156
|
||||
by width: s4 145146 · u2 142772 · s2 131805 · u1 43174 · agg 13475 · p4 12018 · u4 11164 · s1 3429
|
||||
CAST SITES deref 502988 = P 408979 + I 60666 + X 13800 + M 19543 (+ A address-of 18892; + C typed cast-member 36853) in 69492 bodies · coverage OK (raw {'P': 408979, 'I': 60666, 'X': 13800, 'A': 18892, 'C': 36853, 'M': 19543, 'K': 12}) · refused 0 · absolute 0 · readability regex over raw text 411816
|
||||
by base: param 276840 · local 149900 · gaddr 40123 · nested 24076 · global 8861 · other 1677 · addr-of 1350 · ident 161
|
||||
by width: s4 145146 · u2 142772 · s2 131805 · u1 43174 · agg 13480 · p4 12018 · u4 11164 · s1 3429
|
||||
top bases: a0(param) 92255 · param_1(param) 81578 · arg0(param) 67704 · p(local) 18039 · p(param) 11784 · s0(local) 10368 · cach(local) 8246 · s1(local) 7137 · buf(local) 7060 · flag(local) 5453 · *(s32 *)(a0 + 0x20)(nested) 5215 · iVar2(local) 5160 · dst(local) 4874 · iVar1(local) 4836 · iVar3(local) 4693 · e(local) 3918
|
||||
DECLARATIONS 99109 function definitions (1894 K&R) · 4279260 extern function declarations over 12088 names (3773 spelled >1 way) · 6413 names with >1 body text · 1078766 data externs over 55542 symbols (18498 typed >1 way) · asm-label aliases 5148 (159 names) · builtins {'memcpy': 301, 'abs': 1} · attributes {'packed': 26, 'aligned': 6}
|
||||
lying declarations 229 over 24 callees {'K&R-empty': 216, 'narrow': 13} · K&R sites (marked, counted apart) 5293 over 200 callees · cross-binary (the fleet's definitions disagree, this binary has none) 418 over 96 callees
|
||||
STRUCT MAP 19122 types explain 498863/502983 deref sites (99.2 %); unassigned 4120; 4046 types span ≥2 bodies (440088 sites), 15076 single-body; types with conflicts 857, suspect 11
|
||||
STRUCT MAP 19123 types explain 498868/502988 deref sites (99.2 %); unassigned 4120; 4046 types span ≥2 bodies (440088 sites), 15077 single-body; types with conflicts 857, suspect 11
|
||||
merges by evidence {'param': 38862, 'assign': 16777, 'ret-of': 4334, 'arg': 10823, 'return': 1524, 'assign&': 1183, 'arg&': 270, 'field': 9896}; vetoed {'arg-conduit': 115274, 'ret-of-multisource': 6678, 'assign-multisource': 33062, 'arg&-conduit': 14636, 'field-multisource': 7008, 'ret-of': 675, 'assign&-multisource': 12102, 'arg&-no-overlap': 3455, 'assign': 347, 'arg-no-overlap': 43617, 'assign&': 337, 'arg': 182, 'return': 12, 'field': 3224}
|
||||
Unkstruct_800B5CB8 sites 107580 bodies 11184 fields 134 size≥0x10c width-conflicts 50 sign-mixed 55 merges {'param': 9704, 'assign': 8078, 'arg': 7080, 'field': 2622} at D_800B5CB8,D_800D3888,D_80181448 ptr D_8006386C,D_80074778,D_80074780
|
||||
Unkstruct_80126B58 sites 65397 bodies 17047 fields 175 size≥0x24c width-conflicts 50 sign-mixed 60 merges {'param': 2102, 'arg': 1087, 'assign&': 106, 'field': 51} at D_80126B58 ptr D_80126B58
|
||||
@@ -29,5 +29,5 @@ CONTROLS 4/4: player_block_widths OK (D_80078E8C:4/4 D_80078EB2:2/2 D_80078E
|
||||
duplicate-layout classes (top 12): 4B×56 names (0 defs): B4_80165140, B4_8017F154, B4_8018750C, B4_8018AD9C… | 8B×46 names (0 defs): Blk8S16, Blk8_80126940, Blk8_80180A38, D_80195AF4_t… | 8B×45 names (0 defs): B8, B8_17F474, B8_80165140, B8_801851A8… | 8B×34 names (0 defs): Args_80182194, Blk8017FD54, Blk8_801A6AD0, Blk8_801AD914… | 20B×32 names (0 defs): Blk20_8017D110, Buf8017BF34_8017BF34, Buf8017BF50_8017C234, Loc_80180E10… | 16B×28 names (0 defs): Blk16X, Blk16_8017ECD4, Blk16_8018A4C4, Blk16_8018A8B8… | 6B×25 names (0 defs): D80190448_t, Pt6_8017D65C, ResMapCol, S6… | 32B×25 names (0 defs): Blk20_8016DF5C, Blk20_80180B78, Blk20_80184694, Blk32L_80133298… | 8B×22 names (22 defs): B8014, Blk8_80133CD4, Blk8_801463A0, Blk8_8017FF80… | 12B×19 names (19 defs): Blk12, Prim126, PrimC59C, PrimC974… | 8B×14 names (0 defs): CdFileLoc, Cmd_801805C8, M8_8016B6BC, Pair8_8017DDC4… | 4B×14 names (0 defs): D_801D0660_s, Ent_80186AE4, P4, Pair4_801B02DC…
|
||||
VARIANT names (top 12):
|
||||
|
||||
type_census: 1650 definitions (523 layouts, 53 duplicate classes, 0 variant camps) · 502983 cast sites in 69492 bodies over 8 base classes (4 forms + A, coverage OK) · 229 lying declarations / 24 callees (+5293 K&R sites, +418 cross-binary apart) · controls 4/4
|
||||
struct_map: 19122 types over 19122 clusters explain 498863/502983 sites (99.2 %); 4120 unassigned sites; 857 types with conflicts
|
||||
type_census: 1650 definitions (523 layouts, 53 duplicate classes, 0 variant camps) · 502988 cast sites in 69492 bodies over 8 base classes (4 forms + A, coverage OK) · 229 lying declarations / 24 callees (+5293 K&R sites, +418 cross-binary apart) · controls 4/4
|
||||
struct_map: 19123 types over 19123 clusters explain 498868/502988 sites (99.2 %); 4120 unassigned sites; 857 types with conflicts
|
||||
|
||||
@@ -59,6 +59,17 @@ failure modes are different:
|
||||
4. **Types come from the code, names from observation.** Structure is recovered from access patterns the compiler locked into
|
||||
the bytes (many functions reading a `u16` at one offset from one base); meaning comes from observation. The 1,232 struct
|
||||
definitions are unified before any field is named; a duplicate type is a defect, a wrong name is a worse one.
|
||||
5. **Every raw cast goes, or names its cause.** (Phase 39 T2, 2026-10-01.) A `*(T *)(p + k)` is a member access the drafter
|
||||
could not spell; finished code spells it as the member. Where the byte gate refuses the member (cookbook §458: `/s` moves
|
||||
the scheduler), the cast is kept through ONE of the kept-cast macros in `include/common.h`, each expanding to the identical
|
||||
cast and naming one cause class — `CAST_ALIAS` SCHED-ALIAS, `CAST_SIGN` SIGN, `CAST_WIDTH` WIDTH (overlap loser),
|
||||
`CAST_MISALIGNED` MISALIGNED, `CAST_NONSTRUCT` NON-STRUCT base, and `LOBU/LOH/HIH/LOHU/HIHU/LOW/LOWU` REINTERP (an lvalue
|
||||
read at another width or sign). A macro site is honest only with its restruct ledger row (rung S, site verdict
|
||||
`KEPT(<class>)`, keyed tu + function per R99); `tools/type_census.py --check-casts` counts raw sites by form, macro sites
|
||||
by name against the ledger, and lying declarations against `config/lying_exceptions.tsv`, and fails on any raw site, any
|
||||
unbacked macro site or any unledgered lie. `M2C_FIELD` is a raw form, never a kept spelling. Each macro's byte proof (one
|
||||
kept site respelled, `make check` BYTE-IDENTICAL) is recorded in the Phase 39 T2 summary. Levers follow the same reading:
|
||||
`tools/lever_census.py --strict --residue config/lever_residue.tsv` is the gate, the residue file the ledger of what stays.
|
||||
|
||||
Every step is checked the only way this project checks anything: 218 binaries rebuild identical.
|
||||
|
||||
@@ -109,7 +120,8 @@ The order, each step gated by the 218 hashes and each sized by a probe before it
|
||||
every residual.
|
||||
|
||||
**Definition of done for Gen3 (proposed; the Gen3 plan ratifies):** 0 pins or each one `// !FAKE:`-marked with its reason; 0
|
||||
macro bodies (**met at Phase 35**); one definition per structure; every renamed symbol with a recorded basis; clang-format clean; and, unchanged,
|
||||
macro bodies (**met at Phase 35**); one definition per structure; 0 raw casts, every kept cast a `CAST_*`/`REINTERP` macro
|
||||
site with its ledger row (`type_census --check-casts` green, Phase 39 T2); every renamed symbol with a recorded basis; clang-format clean; and, unchanged,
|
||||
218 of 218 byte-identical from a clean rebuild.
|
||||
|
||||
## 5. Related
|
||||
|
||||
@@ -1106,6 +1106,46 @@ void func_80128500(void) {
|
||||
'''
|
||||
|
||||
|
||||
RESIDUE_COLS = ("file", "func", "kind", "pass", "instrument", "cause")
|
||||
|
||||
|
||||
def load_residue(path):
|
||||
"""rows of the residue TSV (config/lever_residue.tsv; reading A, Phase 39 T2): `#` lines, blank lines and the header row skipped."""
|
||||
rows = []
|
||||
for ln in (REPO / path).read_text().splitlines():
|
||||
if not ln.strip() or ln.lstrip().startswith("#"):
|
||||
continue
|
||||
f = ln.split("\t")
|
||||
if f[0] == "file":
|
||||
continue
|
||||
rows.append(dict(zip(RESIDUE_COLS, f + [""] * (len(RESIDUE_COLS) - len(f)))))
|
||||
return rows
|
||||
|
||||
|
||||
def strict_failures(sites, pt, gte_header, residue_rows):
|
||||
"""The --strict failure set. Returns (fails, residue_hit, counts).
|
||||
P37 (S106): --strict reads its own words — "0 asm statements outside the GTE header" — so a DIRECT GTE statement in a body
|
||||
(kind gte / gte-unsigned: Sony's idiom spelled inline instead of as the header's macro call) is a violation here, while the
|
||||
--check headline keeps the P36 definition (the series' continuity). verbatim-body stays excluded by the manifest.
|
||||
P39 T2 (reading A): a site matching a residue row on (file==tu, func==fn, kind) leaves the failures but stays in every count.
|
||||
Per-TU asm macro definitions arrive as a count only (pt): they fail as one synthetic entry and no residue row can match them."""
|
||||
ab = [x for x in sites if x["cls"] in "AB" and x["kind"] not in NON_LEVER_KINDS and x["tu"] != gte_header]
|
||||
direct_gte = [x for x in sites if x["cls"] == "B" and x["kind"] in ("gte", "gte-unsigned") and x["tu"] != gte_header]
|
||||
keys = {(r["file"], r["func"], r["kind"]) for r in residue_rows}
|
||||
used, fails, residue_hit = set(), [], 0
|
||||
for x in ab + direct_gte:
|
||||
k = (x["tu"], x["fn"], x["kind"])
|
||||
if k in keys:
|
||||
used.add(k); residue_hit += 1
|
||||
else:
|
||||
fails.append(x)
|
||||
if pt:
|
||||
fails.append(dict(tu="(per-TU asm macro definitions)", fn="", line=0, cls="B", kind="asm-macro", detail=f"{pt} definition(s)", marked=False))
|
||||
counts = dict(asm=sum(1 for x in ab if x["cls"] == "B" and x["kind"] != GTE_LEVER_KIND), direct_gte=len(direct_gte), ab=len(ab), pt=pt,
|
||||
residue_unused=[r for r in residue_rows if (r["file"], r["func"], r["kind"]) not in used])
|
||||
return fails, residue_hit, counts
|
||||
|
||||
|
||||
def selftest():
|
||||
r = walk_file(FIXTURE, "src/fx/x.c", False, global_names={"ENGINE_SHB": "launder"})
|
||||
got = collections.Counter((s["cls"], s["kind"]) for s in r["sites"])
|
||||
@@ -1151,6 +1191,22 @@ def selftest():
|
||||
print("selftest: instruction mnemonics not recorded"); ok = False
|
||||
if sorted(d["name"] for d in r["defs"]) != ["func_80128218", "func_80128500"]:
|
||||
print(f"selftest: defs {[d['name'] for d in r['defs']]}"); ok = False
|
||||
# strict_failures (P39 T2, reading A): residue rows leave the failures, unmatched rows are reported
|
||||
pin = dict(tu="src/fx/x.c", fn="func_1", line=3, cls="A", kind="pin", detail="$16", marked=True)
|
||||
gte = dict(tu="src/fx/x.c", fn="func_2", line=9, cls="B", kind="gte", detail="rtps", marked=False)
|
||||
row = lambda f, fn, k: dict(file=f, func=fn, kind=k, **{"pass": "sched2", "instrument": "pin", "cause": "fx"})
|
||||
f1, h1, c1 = strict_failures([pin], 0, GTE_HEADER_DEFAULT, [])
|
||||
if f1 != [pin] or h1 != 0:
|
||||
print("selftest: strict (i) pin without residue not in fails"); ok = False
|
||||
f2, h2, c2 = strict_failures([pin], 0, GTE_HEADER_DEFAULT, [row("src/fx/x.c", "func_1", "pin")])
|
||||
if f2 or h2 != 1 or c2["residue_unused"]:
|
||||
print(f"selftest: strict (ii) residue row did not exclude the pin ({len(f2)} fails, hit {h2})"); ok = False
|
||||
f3, h3, c3 = strict_failures([], 0, GTE_HEADER_DEFAULT, [row("src/fx/y.c", "func_9", "pin")])
|
||||
if f3 or h3 or len(c3["residue_unused"]) != 1:
|
||||
print(f"selftest: strict (iii) residue-unused {len(c3['residue_unused'])} (want 1)"); ok = False
|
||||
f4, h4, c4 = strict_failures([gte], 0, GTE_HEADER_DEFAULT, [row("src/fx/x.c", "func_1", "pin")])
|
||||
if f4 != [gte] or h4 or len(c4["residue_unused"]) != 1:
|
||||
print("selftest: strict (iv) direct gte statement escaped the failures"); ok = False
|
||||
print(f"selftest: {'OK' if ok else 'FAIL'} — {len(r['sites'])} sites, {len(r['defs'])} defs, {len(r['macro_defs'])} asm macros")
|
||||
return ok
|
||||
|
||||
@@ -1160,6 +1216,8 @@ def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
ap.add_argument("--check", action="store_true", help="exit 1 if any pin / asm statement (GTE excluded) lacks a `// !FAKE:` marker")
|
||||
ap.add_argument("--strict", action="store_true", help="with --check: exit 1 if ANY pin / asm statement remains outside the GTE header")
|
||||
ap.add_argument("--residue", default=None, metavar="PATH",
|
||||
help="with --strict: TSV (file, func, kind, pass, instrument, cause) of sites excluded from the failures, still counted")
|
||||
ap.add_argument("--gte-header", default=GTE_HEADER_DEFAULT)
|
||||
ap.add_argument("--out-dir", default=OUT_DIR_DEFAULT)
|
||||
ap.add_argument("--sites", action="store_true", help="also write every site to lever_sites.jsonl (scratch)")
|
||||
@@ -1188,18 +1246,19 @@ def main():
|
||||
ab = [x for x in sites if x["cls"] in "AB" and x["kind"] not in NON_LEVER_KINDS and x["tu"] != a.gte_header]
|
||||
un = [x for x in ab if not x["marked"]]
|
||||
if a.strict:
|
||||
# P37 (S106): --strict reads its own words — "0 asm statements outside the GTE header" — so a DIRECT GTE statement in a body
|
||||
# (kind gte / gte-unsigned: Sony's idiom spelled inline instead of as the header's macro call) is a violation here, while the
|
||||
# --check headline keeps the P36 definition (the series' continuity). verbatim-body stays excluded by the manifest.
|
||||
direct_gte = [x for x in sites if x["cls"] == "B" and x["kind"] in ("gte", "gte-unsigned") and x["tu"] != a.gte_header]
|
||||
residue_rows = load_residue(a.residue) if a.residue else []
|
||||
pt = s.get("per_tu_asm_macro_definitions", {}).get("total", 0)
|
||||
fails, residue_hit, c = strict_failures(sites, pt, a.gte_header, residue_rows)
|
||||
gl = s.get("gte_levers", {}).get("sites", 0)
|
||||
print(f"lever_census --check --strict: pins {s['pins']['sites']}, asm {sum(1 for x in ab if x['cls'] == 'B' and x['kind'] != GTE_LEVER_KIND)}, "
|
||||
f"gte-levers {gl}, direct GTE statements in bodies {len(direct_gte)}, per-TU asm macro definitions {pt}, "
|
||||
f"volatile-needed {s['classes']['C']['sites']}, register-needed {s['classes']['D']['sites']} — "
|
||||
f"{'OK' if not ab and not pt and not direct_gte else 'FAIL'}")
|
||||
if ab or pt or direct_gte:
|
||||
for x in (ab + direct_gte)[:10]:
|
||||
print(f"lever_census --check --strict: pins {s['pins']['sites']}, asm {c['asm']}, "
|
||||
f"gte-levers {gl}, direct GTE statements in bodies {c['direct_gte']}, per-TU asm macro definitions {pt}, "
|
||||
f"volatile-needed {s['classes']['C']['sites']}, register-needed {s['classes']['D']['sites']}, "
|
||||
f"residue {residue_hit}, residue-unused {len(c['residue_unused'])} — "
|
||||
f"{'OK' if not fails else 'FAIL'}")
|
||||
for r in c["residue_unused"][:10]:
|
||||
print(f" residue-unused: {r['file']} {r['func']} {r['kind']}")
|
||||
if fails:
|
||||
for x in fails[:10]:
|
||||
print(f" {x['tu']}:{x['line']} {x['fn']} {x['cls']}/{x['kind']} {x['detail']}")
|
||||
rc = 1
|
||||
else:
|
||||
|
||||
@@ -137,6 +137,13 @@ FORM_X = re.compile(r"\(\s*\(\s*" + TYPE_IN_CAST + r"\s*\)\s*([^()\[\]]+?)\s*\)\
|
||||
FORM_M = re.compile(r"\bM2C_FIELD\s*\(")
|
||||
FORM_A = re.compile(r"(?<![\w\)\]\*])\(\s*" + TYPE_IN_CAST + r"\s*\)\s*\((?=[^()]*[+-])") # (T *)(… + …) not deref'd
|
||||
FORM_C = re.compile(r"\(\s*\(\s*" + TYPE_IN_CAST + r"\s*\)\s*([^()\[\]]+?)\s*\)\s*->\s*([A-Za-z_]\w*)") # ((T *)e)->f (cast-then-member)
|
||||
# K: a kept-cast / reinterpretation macro site (P39 T2; include/common.h): counted by name, never raw — --check-casts demands its
|
||||
# restruct ledger row (rung S, site verdict KEPT(<class>)). Not in RAW_FORMS: by_form P/I/X/M and deref_total are unchanged.
|
||||
KEPT_MACROS = {"CAST_ALIAS": "SCHED-ALIAS", "CAST_SIGN": "SIGN", "CAST_WIDTH": "WIDTH", "CAST_MISALIGNED": "MISALIGNED",
|
||||
"CAST_NONSTRUCT": "NON-STRUCT", **{n: "REINTERP" for n in ("LOBU", "LOH", "HIH", "LOHU", "HIHU", "LOW", "LOWU")}}
|
||||
FORM_K = re.compile(r"\b(" + "|".join(KEPT_MACROS) + r")\s*\(")
|
||||
DEFLINE_FILE = "include/common.h" # the macros' own `#define` lines there are definitions, not sites (--check-casts drops them)
|
||||
DEFLINE_RX = re.compile(r"^[ \t]*#[ \t]*define[ \t]+([A-Za-z_]\w*)")
|
||||
ABS_ADDR = re.compile(r"^0x80[0-9A-Fa-f]{6}$")
|
||||
ASSIGN_OP = re.compile(r"^\s*(?:(?:\+|-|\*|/|%|&|\||\^|<<|>>)?=(?!=)|\+\+|--)")
|
||||
IDENT = re.compile(r"^[A-Za-z_]\w*$")
|
||||
@@ -365,6 +372,25 @@ def find_sites(masked, rel, span_of_line, line_of, params_of):
|
||||
if k is None:
|
||||
rec["index"] = _strip_parens(args[2])[:60]
|
||||
sites.append(rec)
|
||||
# K: CAST_*(T, p, k) / LOBU..LOWU(x) — a kept-cast macro site, by name and class (3-arg split as M; REINTERP keyed on its line)
|
||||
for m in FORM_K.finditer(masked):
|
||||
raw_counts["K"] += 1
|
||||
inner, close = _paren_body(masked, m.end() - 1)
|
||||
ln, d = fn_ctx(m.start())
|
||||
name = m.group(1)
|
||||
rec = dict(form="K", tu=rel, fn=(d["name"] if d else None), line=ln, pos=m.start(), end=(close + 1 if inner is not None else m.end()),
|
||||
name=name, cls=KEPT_MACROS[name], ctype=None, stars=0, base=None, bclass="other", off=0)
|
||||
args = _split_top(inner, ",") if inner is not None else []
|
||||
if name.startswith("CAST_") and len(args) == 3:
|
||||
b = parse_base(args[1])
|
||||
b["bclass"] = classify(b["bclass"], b.get("base"), d)
|
||||
k = _int(_strip_parens(args[2]))
|
||||
ctype = _norm_type(args[0])
|
||||
rec.update(base=b.get("base"), bclass=b["bclass"], text=b.get("text"), ctype=ctype, stars=1,
|
||||
off=(b.get("off") or 0) + (k if k is not None else 0), access=_store_kind(_after(masked, close + 1)))
|
||||
if k is None:
|
||||
rec["index"] = _strip_parens(args[2])[:60]
|
||||
sites.append(rec)
|
||||
# A: (T *)(base + k) — an address, not a dereference
|
||||
for m in FORM_A.finditer(masked):
|
||||
raw_counts["A"] += 1
|
||||
@@ -602,8 +628,14 @@ def walk_file(raw, rel):
|
||||
definitions, aliases_td, fwd = find_definitions(masked, rel, span_of_line, line_of)
|
||||
fndefs, extern_fns, extern_data, asm_aliases, builtins, attrs, flows, params_of = find_decls_and_flows(masked, rel, defs, span_of_line, line_of)
|
||||
sites, raw_counts = find_sites(masked, rel, span_of_line, line_of, params_of)
|
||||
if rel == DEFLINE_FILE: # P39 T2: a macro's own `#define` line (kept-cast set, M2C_FIELD) is a definition, not a site
|
||||
deflines = {i + 1 for i, ln in enumerate(raw.split("\n"))
|
||||
if (mdl := DEFLINE_RX.match(ln)) and (mdl.group(1) in KEPT_MACROS or mdl.group(1) == "M2C_FIELD")}
|
||||
for s in sites:
|
||||
if s["line"] in deflines:
|
||||
s["defline"] = True
|
||||
# every site records the classification of its base in the function's parameter list
|
||||
abs_casts = sum(1 for s in sites if s.get("bclass") == "abs")
|
||||
abs_casts = sum(1 for s in sites if s.get("bclass") == "abs" and s["form"] != "K")
|
||||
# identifier counts for the dead-name test (type names are identifiers; the canonical header is excluded by the caller)
|
||||
ident_counts = collections.Counter(m.group(0) for m in re.finditer(r"\b[A-Za-z_]\w*\b", masked))
|
||||
# T5.1: the tokens the filter below drops, kept apart so the --check-structs dead test drops no identifier by length or prefix
|
||||
@@ -1168,6 +1200,14 @@ def def_layout_hash(res, d, lay):
|
||||
return hashlib.sha1(json.dumps([lay[0], lay[2], "align", al]).encode()).hexdigest()[:12]
|
||||
|
||||
|
||||
def check_casts_counts(sites, raw_counts):
|
||||
"""P39 T2: the --check-casts inputs — ({P,I,X,M: raw count}, [macro sites]) without include/common.h's own `#define` lines."""
|
||||
deflined = collections.Counter(s["form"] for s in sites if s.get("defline"))
|
||||
check_raw = {f: raw_counts.get(f, 0) - deflined[f] for f in ("P", "I", "X", "M")}
|
||||
macro_sites = [dict(tu=s["tu"], fn=s["fn"], line=s["line"], pos=s["pos"], name=s["name"], cls=s["cls"], base=s.get("base"),
|
||||
bclass=s.get("bclass"), off=s.get("off")) for s in sites if s["form"] == "K" and not s.get("defline")]
|
||||
return check_raw, macro_sites
|
||||
|
||||
def run_census(jobs, use_cache=True, out_dir=OUT_DIR_DEFAULT, want_sites=False):
|
||||
t0 = time.time()
|
||||
w = walk_all(jobs, use_cache=use_cache, out_dir=out_dir)
|
||||
@@ -1324,6 +1364,8 @@ def run_census(jobs, use_cache=True, out_dir=OUT_DIR_DEFAULT, want_sites=False):
|
||||
top_bases = collections.Counter((s["bclass"], s["base"]) for s in sites_all if s["form"] in RAW_FORMS and s.get("base")).most_common(40)
|
||||
deref_sites = [s for s in sites_all if s["form"] in RAW_FORMS]
|
||||
bodies_with_sites = len({(s["tu"], s["fn"]) for s in deref_sites if s.get("fn")})
|
||||
check_raw, macro_sites = check_casts_counts(sites_all, raw_counts)
|
||||
macros = collections.Counter(s["name"] for s in macro_sites)
|
||||
# the readability series' own regex over the RAW text (continuity with docs/readability-progress.tsv)
|
||||
import readability_progress as rp
|
||||
raw_readability = 0
|
||||
@@ -1424,12 +1466,14 @@ def run_census(jobs, use_cache=True, out_dir=OUT_DIR_DEFAULT, want_sites=False):
|
||||
casts=dict(by_form=dict(by_form), raw_counts=dict(raw_counts), coverage_ok=coverage_ok, refused=len(refused),
|
||||
deref_total=len(deref_sites), addr_form=by_form["A"], typed_cast_member=by_form["C"], bodies=bodies_with_sites,
|
||||
by_bclass=dict(by_bclass), by_width=dict(by_width.most_common()), readability_raw_regex=raw_readability,
|
||||
abs_casts=by_bclass.get("abs", 0), top_bases=[[bc, b, n] for ((bc, b), n) in top_bases]),
|
||||
abs_casts=by_bclass.get("abs", 0), top_bases=[[bc, b, n] for ((bc, b), n) in top_bases],
|
||||
macros=dict(macros), check_raw=check_raw, macro_sites=macro_sites),
|
||||
decls=dict(fn_definitions=len(fndefs), kr_definitions=sum(1 for f in fndefs if f["kr"]), extern_fn_decls=len(extern_fns),
|
||||
extern_data_decls=len(extern_data), declared_fn_names=len(spellings), multi_spelled_callees=multi_spelled,
|
||||
multi_body_names=multi_body, data_symbols_declared=len(data_types), multi_typed_data_symbols=multi_typed_data,
|
||||
asm_label_aliases=len(asm_aliases), asm_label_alias_names=len({a["name"] for a in asm_aliases}),
|
||||
builtins=dict(builtins), attributes=dict(attrs),
|
||||
lying_rows=[dict(callee=r["callee"], defined_in=r.get("defined_in"), kind=r["kind"]) for r in lying],
|
||||
lying=len(lying), lying_callees=lying_callees, lying_kinds=dict(lying_kinds),
|
||||
kr_sites=len(kr_sites), kr_callees=kr_callees,
|
||||
cross_binary=len(cross), cross_callees=len({r["callee"] for r in cross})),
|
||||
@@ -1838,6 +1882,43 @@ def selftest():
|
||||
bad_ctrl = dict(zero, controls=dict(a=dict(ok=False), ok=0, n=1))
|
||||
rc2, l2 = check_structs_verdict(bad_ctrl, 0, 0)
|
||||
checks.append(("check-structs controls a<b FAIL", rc2 == 1 and "controls 0/1" in l2[0] and "a=FAIL" in l2[2]))
|
||||
# P39 T2: --check-casts — each raw form counted (positive) and its near-miss not (negative); the macro sites; the define lines
|
||||
def cc(body, rel="src/ov_TEST/k.c"):
|
||||
w = walk_file("void g(s32 a0, s32 *p, s32 *q, s32 k, s32 x) {\n " + body + "\n}\n", rel)
|
||||
return w, check_casts_counts(w["sites"], w["raw_counts"])
|
||||
for f, pos, negs in (("P", "*(u16 *)(a0 + 4) = 1;", ("p->f = 0;", "CAST_ALIAS(u16, a0, 4) = 2;")),
|
||||
("I", "*(u8 *)D_80078EB4 = 1;", ("k = (u8)x;",)),
|
||||
("X", "((s16 *)a0)[2] = 0;", ("q[k] = 0;",)),
|
||||
("M", "M2C_FIELD(a0, u8 *, 1) = 0;", ("CAST_ALIAS(u16, a0, 4) = 2;",))):
|
||||
checks.append((f"check-casts {f} counted", cc(pos)[1][0] == {g: int(g == f) for g in ("P", "I", "X", "M")}))
|
||||
for ng in negs:
|
||||
checks.append((f"check-casts {f} near-miss {ng!r} not counted", cc(ng)[1][0] == dict(P=0, I=0, X=0, M=0)))
|
||||
_, (_, ks) = cc("CAST_ALIAS(u16, a0, 4) = 2;\n LOH(x) = 1;")
|
||||
checks.append(("check-casts K sites", [(s["name"], s["cls"]) for s in ks] == [("CAST_ALIAS", "SCHED-ALIAS"), ("LOH", "REINTERP")]
|
||||
and (ks[0]["bclass"], ks[0]["base"], ks[0]["off"]) == ("param", "a0", 4)))
|
||||
defs_txt = ("#define CAST_ALIAS(T, p, k) (*(T *)((p) + (k)))\n#define M2C_FIELD(expr, type_ptr, offset) (*(type_ptr)((s8 *)(expr) + (offset)))\n"
|
||||
"#define LOH(x) (*(s16 *)&(x))\n")
|
||||
dw = walk_file(defs_txt, DEFLINE_FILE)
|
||||
dr, dk = check_casts_counts(dw["sites"], dw["raw_counts"])
|
||||
ow = walk_file(defs_txt, "src/ov_TEST/d.c")
|
||||
orw, ok_ = check_casts_counts(ow["sites"], ow["raw_counts"])
|
||||
checks.append(("check-casts common.h #define lines dropped", dr == dict(P=0, I=0, X=0, M=0) and dk == []
|
||||
and orw["P"] == 1 and orw["M"] == 1 and len(ok_) == 2))
|
||||
site = dict(tu="src/ov_TEST/k.c", fn="g", line=2, pos=0, name="CAST_ALIAS", cls="SCHED-ALIAS", base="a0", bclass="param", off=4)
|
||||
row = {("S", "src/ov_TEST/k.c", "g"): dict(verdict="KEPT", sites=[dict(form="P", line=2, base="param:a0", off=4, verdict="KEPT(SCHED-ALIAS)")])}
|
||||
wrong = {("S", "src/ov_TEST/k.c", "g"): dict(verdict="KEPT", sites=[dict(form="P", line=2, base="param:a0", off=4, verdict="KEPT(SIGN)")])}
|
||||
z = dict(P=0, I=0, X=0, M=0)
|
||||
checks.append(("check-casts macro backed", check_casts_verdict(z, [site], row, [], set())[1].endswith("macros CAST_ALIAS=1 (unbacked 0); lying=0 (ledgered 0)")
|
||||
and check_casts_verdict(z, [site], row, [], set())[0] == 0))
|
||||
checks.append(("check-casts macro unbacked", check_casts_verdict(z, [site], {}, [], set()) [0] == 1 and "(unbacked 1)" in check_casts_verdict(z, [site], {}, [], set())[1]))
|
||||
checks.append(("check-casts macro wrong class", "(unbacked 1)" in check_casts_verdict(z, [site], wrong, [], set())[1]))
|
||||
lie = [dict(callee="func_80001000", defined_in="src/ov_TEST/k.c", kind="arity")]
|
||||
rl1, ll1 = check_casts_verdict(z, [], {}, lie, {("func_80001000", "src/ov_TEST/k.c", "arity")})
|
||||
rl0, ll0 = check_casts_verdict(z, [], {}, lie, set())
|
||||
checks.append(("check-casts lying ledgered", rl1 == 0 and ll1.endswith("lying=1 (ledgered 1)") and rl0 == 1 and ll0.endswith("lying=1 (ledgered 0)")))
|
||||
rz, lz = check_casts_verdict(z, [], {}, [], set())
|
||||
checks.append(("check-casts all-zero OK + line", rz == 0 and lz == "check-casts: raw P=0 I=0 X=0 M=0; macros none (unbacked 0); lying=0 (ledgered 0)"))
|
||||
checks.append(("check-casts raw FAIL", check_casts_verdict(dict(z, X=1), [], {}, [], set())[0] == 1))
|
||||
bad = [n for n, ok in checks if not ok]
|
||||
print(f"type_census --selftest: {len(checks) - len(bad)}/{len(checks)} checks OK" + (f"; FAILED: {bad}" if bad else ""))
|
||||
return 0 if not bad else 1
|
||||
@@ -1875,6 +1956,56 @@ def check_structs_verdict(summary, pad_names, parse_error_decls):
|
||||
+ f" doc_disputed={sum(v.get('detail', '').count('[doc-disputed') for v in c.values() if isinstance(v, dict))}"]
|
||||
return (1 if viol else 0), lines
|
||||
|
||||
|
||||
# --check-casts (P39 T2): the cast-and-lying gate. A macro site is backed only by its restruct ledger row (rung S, read through
|
||||
# restruct.ledger_latest — R100: no second done-filter); a lying row only by a config/lying_exceptions.tsv row.
|
||||
LYING_EXCEPTIONS = "config/lying_exceptions.tsv"
|
||||
BACKED_S_VERDICTS = ("KEPT", "S2", "KEPT-ALL")
|
||||
|
||||
|
||||
def check_casts_inputs():
|
||||
"""(latest ledger {(rung, tu, unit): row}, {(callee, def_file, kind)} exceptions). Lazy import: restruct imports this module."""
|
||||
import restruct as rs
|
||||
latest = rs.ledger_latest(rs.load_ledger())
|
||||
exc = set()
|
||||
p = REPO / LYING_EXCEPTIONS
|
||||
if p.exists():
|
||||
for ln in p.read_text().splitlines():
|
||||
if not ln.strip() or ln.startswith("#") or ln.startswith("callee\t"):
|
||||
continue
|
||||
f = ln.split("\t")
|
||||
if len(f) >= 3:
|
||||
exc.add((f[0], f[1], f[2]))
|
||||
return latest, exc
|
||||
|
||||
|
||||
def check_casts_verdict(raw_by_form, macro_sites, latest, lying_rows, exceptions):
|
||||
"""(rc, line) — rc 1 iff any raw form > 0, any macro site unbacked, or any lying row not ledgered in the exceptions TSV."""
|
||||
unbacked = 0
|
||||
for s in macro_sites:
|
||||
row = latest.get(("S", s["tu"], s["fn"]))
|
||||
ok = False
|
||||
if row and row.get("verdict") in BACKED_S_VERDICTS:
|
||||
want = f"KEPT({s['cls']})"
|
||||
for e in row.get("sites") or []:
|
||||
if e.get("verdict") != want:
|
||||
continue
|
||||
if s["cls"] == "REINTERP" and e.get("line") == s["line"]:
|
||||
ok = True
|
||||
elif s["cls"] != "REINTERP" and e.get("off") == s["off"] and e.get("base") == f"{s['bclass']}:{s['base']}":
|
||||
ok = True
|
||||
if ok:
|
||||
break
|
||||
unbacked += not ok
|
||||
lying = len(lying_rows)
|
||||
unledgered = sum(1 for r in lying_rows if (r["callee"], r.get("defined_in"), r["kind"]) not in exceptions)
|
||||
macros = collections.Counter(s["name"] for s in macro_sites)
|
||||
mtxt = " ".join(f"{n}={macros[n]}" for n in KEPT_MACROS if macros[n] > 0) or "none"
|
||||
raw = [raw_by_form.get(f, 0) for f in ("P", "I", "X", "M")]
|
||||
line = (f"check-casts: raw P={raw[0]} I={raw[1]} X={raw[2]} M={raw[3]}; macros {mtxt} (unbacked {unbacked}); "
|
||||
f"lying={lying} (ledgered {lying - unledgered})")
|
||||
return (1 if any(raw) or unbacked or unledgered else 0), line
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------------
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
@@ -1884,6 +2015,7 @@ def main():
|
||||
ap.add_argument("--sites", action="store_true", help="also write sites.jsonl")
|
||||
ap.add_argument("--check", action="store_true", help="T8's gate (the invariants; exit 1 on a violation)")
|
||||
ap.add_argument("--check-structs", action="store_true", help="the struct-unification gate (P38 T2; exit 1 on a violation)")
|
||||
ap.add_argument("--check-casts", action="store_true", help="the cast-and-lying gate (P39 T2; exit 1 on a raw cast, an unbacked macro site or an unledgered lie)")
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
ap.add_argument("--quiet", action="store_true")
|
||||
a = ap.parse_args()
|
||||
@@ -1924,6 +2056,12 @@ def main():
|
||||
print(f"conflicting_types={conflicting} tu_conflict={tu_conflict} types_floor_lying={summary['decls']['lying']} audit_other={other} "
|
||||
f"twins_listed={st['twins_listed']} dead_kept={st['dead_kept']} block_frames={st['block_frames']} (not gating)")
|
||||
rc = rc or rc_s
|
||||
if a.check_casts:
|
||||
latest, exc = check_casts_inputs()
|
||||
rc_c, line = check_casts_verdict(summary["casts"]["check_raw"], summary["casts"]["macro_sites"], latest,
|
||||
summary["decls"]["lying_rows"], exc)
|
||||
print(line)
|
||||
rc = rc or rc_c
|
||||
sys.exit(rc)
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+13
-1
@@ -59,6 +59,17 @@ failure modes are different:
|
||||
4. **Types come from the code, names from observation.** Structure is recovered from access patterns the compiler locked into
|
||||
the bytes (many functions reading a `u16` at one offset from one base); meaning comes from observation. The 1,232 struct
|
||||
definitions are unified before any field is named; a duplicate type is a defect, a wrong name is a worse one.
|
||||
5. **Every raw cast goes, or names its cause.** (Phase 39 T2, 2026-10-01.) A `*(T *)(p + k)` is a member access the drafter
|
||||
could not spell; finished code spells it as the member. Where the byte gate refuses the member (cookbook §458: `/s` moves
|
||||
the scheduler), the cast is kept through ONE of the kept-cast macros in `include/common.h`, each expanding to the identical
|
||||
cast and naming one cause class — `CAST_ALIAS` SCHED-ALIAS, `CAST_SIGN` SIGN, `CAST_WIDTH` WIDTH (overlap loser),
|
||||
`CAST_MISALIGNED` MISALIGNED, `CAST_NONSTRUCT` NON-STRUCT base, and `LOBU/LOH/HIH/LOHU/HIHU/LOW/LOWU` REINTERP (an lvalue
|
||||
read at another width or sign). A macro site is honest only with its restruct ledger row (rung S, site verdict
|
||||
`KEPT(<class>)`, keyed tu + function per R99); `tools/type_census.py --check-casts` counts raw sites by form, macro sites
|
||||
by name against the ledger, and lying declarations against `config/lying_exceptions.tsv`, and fails on any raw site, any
|
||||
unbacked macro site or any unledgered lie. `M2C_FIELD` is a raw form, never a kept spelling. Each macro's byte proof (one
|
||||
kept site respelled, `make check` BYTE-IDENTICAL) is recorded in the Phase 39 T2 summary. Levers follow the same reading:
|
||||
`tools/lever_census.py --strict --residue config/lever_residue.tsv` is the gate, the residue file the ledger of what stays.
|
||||
|
||||
Every step is checked the only way this project checks anything: 218 binaries rebuild identical.
|
||||
|
||||
@@ -109,7 +120,8 @@ The order, each step gated by the 218 hashes and each sized by a probe before it
|
||||
every residual.
|
||||
|
||||
**Definition of done for Gen3 (proposed; the Gen3 plan ratifies):** 0 pins or each one `// !FAKE:`-marked with its reason; 0
|
||||
macro bodies (**met at Phase 35**); one definition per structure; every renamed symbol with a recorded basis; clang-format clean; and, unchanged,
|
||||
macro bodies (**met at Phase 35**); one definition per structure; 0 raw casts, every kept cast a `CAST_*`/`REINTERP` macro
|
||||
site with its ledger row (`type_census --check-casts` green, Phase 39 T2); every renamed symbol with a recorded basis; clang-format clean; and, unchanged,
|
||||
218 of 218 byte-identical from a clean rebuild.
|
||||
|
||||
## 5. Related
|
||||
|
||||
+3
-3
@@ -1,8 +1,8 @@
|
||||
# The struct map — the types the fleet's cast sites need (Phase 37 T1)
|
||||
|
||||
> **Generated by `tools/type_census.py` at `0cb2b6b522` (2026-10-01T16:47:07Z); never typed (R75).** Every raw pointer-cast dereference in `src/` was assigned to a base (a parameter, a local, a global by value or by address, an absolute address) and the bases were clustered by EVIDENCE only — the same global symbol, an argument passed to a callee's parameter, an assignment or return, a pointer field read, a shared body's fleet-wide identity — never by a similar offset signature. Each cluster is one type the tree needs. Names are placeholders (`Unkstruct_<addr>`: the lowest global instance's address, else the lowest member function's); fields are placeholders (`unk<HEX>`) until Phase 38 cites evidence.
|
||||
> **Generated by `tools/type_census.py` at `7845da8f03` (2026-10-01T17:32:55Z); never typed (R75).** Every raw pointer-cast dereference in `src/` was assigned to a base (a parameter, a local, a global by value or by address, an absolute address) and the bases were clustered by EVIDENCE only — the same global symbol, an argument passed to a callee's parameter, an assignment or return, a pointer field read, a shared body's fleet-wide identity — never by a similar offset signature. Each cluster is one type the tree needs. Names are placeholders (`Unkstruct_<addr>`: the lowest global instance's address, else the lowest member function's); fields are placeholders (`unk<HEX>`) until Phase 38 cites evidence.
|
||||
|
||||
**19122 types explain 498863 of 502983 dereference sites (99.2 %); 4120 sites have a base the census could not classify; 857 types carry a width conflict.** Merges by evidence: param 38862, assign 16777, arg 10823, field 9896, ret-of 4334, return 1524, assign& 1183, arg& 270.
|
||||
**19123 types explain 498868 of 502988 dereference sites (99.2 %); 4120 sites have a base the census could not classify; 857 types carry a width conflict.** Merges by evidence: param 38862, assign 16777, arg 10823, field 9896, ret-of 4334, return 1524, assign& 1183, arg& 270.
|
||||
|
||||
| # | type | sites | bodies | fields | size ≥ | width conflicts (+sign mixes) | instance globals | pointer globals |
|
||||
|---:|---|---:|---:|---:|---:|---:|---|---|
|
||||
@@ -207,4 +207,4 @@
|
||||
| 199 | `Unkstruct_76f4299b` | 262 | 131 | 1 | 0x4a | 0 (+0 sign) | | |
|
||||
| 200 | `Unkstruct_3816caab` | 260 | 130 | 2 | 0x1c | 0 (+0 sign) | | |
|
||||
|
||||
*19122 types in all; the table shows the 200 largest by site count. The full map with every type's layout (offset, width, sign, site count per field) and conflicts is `.run/P37/census/struct_map.json`.*
|
||||
*19123 types in all; the table shows the 200 largest by site count. The full map with every type's layout (offset, width, sign, site count per field) and conflicts is `.run/P37/census/struct_map.json`.*
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# T2 log — cast and lying gate instruments, reinterpret-macro set (Phase 3.39)
|
||||
|
||||
Expert: expert-fable (Fable 5.1, effort high). Start commit 0830f6374d.
|
||||
|
||||
## Timeline
|
||||
1. Read card slice, plan Context/Interfaces/Cookbook/Research, T2 entry, tasks/T1.md. Read include/common.h (3.4 KB), docs/gen3-standards.md (11 KB), cookbook/C0507.md.
|
||||
2. Finding against plan Context: the legacy reinterpret macros LOBU/LOH/HIH/LOHU/HIHU/LOW/LOWU DO exist, include/common.h:55-61 (Phase 37 T3), unused in src (grep: only the defines). The retriever note "none in include/" was wrong.
|
||||
3. Finding: `.run/P37/restruct/ledger.jsonl` holds 0 rung-S rows and 0 `KEPT(` sites (173,459 rows: D 173,185, L 274). No existing kept site is ledgered; byte proofs had to use raw sites chosen by cause class.
|
||||
4. Three retriever-code questions (no reports written; answers inline): type_census internals (forms :133-140, find_sites :275, selftest :1767, main :1879, TOOL_STAMP :55 keys the walk cache, lazy `import restruct` :1853); restruct ledger schema (rung-S row: tu, unit=fn, verdict S2/KEPT-ALL, sites[] {form,line,base "<bclass>:<base>",off,width,sign,access,type,field,verdict "KEPT(<hint>)"}, `pass_hint` classes :563-591 incl. SCHED-ALIAS/ALIGNMENT/WIDTH, no SIGN/NON-STRUCT; `macro_probe()` :4245 proves LOBU..LOWU only under `--selftest --real`); lever_census internals (no pass/instrument parse exists; --strict :1190-1204 fails on every A/B site regardless of marker, pt>0, direct GTE; selftest is an `ok` flag, not a checks list).
|
||||
5. Hook blocked `sed` on tools/*.py ("learn a tool from --help, never its source"); source facts came from retriever-code instead.
|
||||
6. Picked byte-proof sites with a scratch filter over .run/P39/census/sites.jsonl (scratchpad pick.py, not kept): SIGN `ov_SC01_000_after.c:2682` (same base+off read s16 and u16 in func_801523F4), WIDTH `ov_SC01_000_jr_801734BC.c:2582` (two widths at param_1+8 in func_801749C8), MISALIGNED `ov_SC01_000_jr_8012ACE0.c:2624` (`*(int *)(param_1 + 6)`), NON-STRUCT `ov_SC01_000_jr_8012ACE0.c:795` (bclass other), SCHED-ALIAS `ov_SC05_010_jr_80180F84.c:3006` (func_801814AC `+0x34` store, cookbook §458 exemplar).
|
||||
7. Coder c1 (common.h ONE edit + 5 proofs + R22) and c2 (lever_census --residue) in parallel; c3 (type_census --check-casts) after c1's commit.
|
||||
8. Expert edit: docs/gen3-standards.md §2 rule 5 + definition-of-done clause (one file, ~15 lines).
|
||||
9. Verify (two calls, > 285 s together): .run/logs/t2_verify_a.log (selftests + check-structs, rc 0), .run/logs/t2_verify_b.log (check-casts, rc 1 as required). kit-corpus + tools-health: .run/logs/t2_kit.log.
|
||||
|
||||
## Decisions (judgment items)
|
||||
- Raw = forms P/I/X/M (M2C_FIELD stays raw). Backed = a site spelled by a registered macro whose `restruct.ledger_latest[("S", tu, fn)]` row has verdict KEPT/S2/KEPT-ALL and a sites[] entry `KEPT(<class>)` matching off + "<bclass>:<base>" (REINTERP: line). One filter, R100.
|
||||
- Cause classes: SCHED-ALIAS, SIGN, WIDTH (overlap loser), MISALIGNED, NON-STRUCT, plus REINTERP for the legacy lvalue set. Macro names CAST_ALIAS/CAST_SIGN/CAST_WIDTH/CAST_MISALIGNED/CAST_NONSTRUCT, all `(T, p, k)` → `(*(T *)((p) + (k)))`.
|
||||
- Legacy LOBU..LOWU: reinstated as the set's sixth class rather than retired (a form-I lvalue reinterpretation has no other honest spelling); byte proof = restruct's macro_probe (selftest --real 82/82 at b5e18e5f83, T1), no new site proof.
|
||||
- Definition lines of registered macros in include/common.h (incl. M2C_FIELD:43) are not sites: M reads 19,542 (T1 19,543 − common.h's own define); P stays 408,974 despite five new `*(T *)(` define lines.
|
||||
- Byte-proof sites were restored after IDENTICAL (tree keeps 0 macro sites, unbacked 0): keeping them would have left 5 unledgered macro sites for the next task; the proof lines live in logs/T2.c1.md.
|
||||
- Reading A: --residue rows excluded from --strict failures only, counted in every total; residue-unused rows printed (stale rows visible).
|
||||
|
||||
## Coder briefs
|
||||
- T2.c1 → 247422bb83 (logs/T2.c1.md): common.h:62-78; proofs `make check BINARY=ov_SC01_000` → BYTE-IDENTICAL sha1 9052dc0e…, `make check BINARY=ov_SC05_010` → BYTE-IDENTICAL sha1 9b5a70f9…; R22 `make clean && make extract-all JOBS=16 && make check-all JOBS=16` → 218 passed 0 failed (.run/logs/t2_r22.log). Deviation: 4 src files restored, not 3; discussions/INDEX.md (pre-modified) auto-staged into the commit.
|
||||
- T2.c2 → 313ee50403 (logs/T2.c2.md): lever_census.py `load_residue`, `strict_failures()` :1109-1148, selftest controls i-iv :1194-1209, `--residue` :1219, main :1249-1265; config/lever_residue.tsv header. Live: `pins 1868, asm 1411, gte-levers 450, direct GTE 6717, per-TU asm macro defs 314, …, residue 0, residue-unused 0 — FAIL` rc 1 (.run/logs/t2c2_strict.log). Deviation: per-TU asm macro defs not matchable by residue (mdefs_all local to run_census) → one synthetic `asm-macro` fail entry while pt>0.
|
||||
- T2.c3 → 7845da8f03 (logs/T2.c3.md): type_census.py KEPT_MACROS/FORM_K :141-147, form-K pass :374-394, defline drop :612-620, `check_casts_inputs`/`check_casts_verdict`, `--check-casts`; selftest 32 → 50 checks; config/lying_exceptions.tsv header. Live: `check-casts: raw P=408974 I=60666 X=13800 M=19542; macros none (unbacked 0); lying=229 (ledgered 0)` rc 1 (.run/logs/t2c3_casts.log). Caches: .run/P37/census/cache/walk_cache.json and .run/P39/census/cache removed; rebuilt under the new TOOL_STAMP.
|
||||
|
||||
## Hypotheses rejected
|
||||
- "Legacy LOBU..LOWU absent from include/" (plan Context) — false, present at common.h:55-61.
|
||||
- "Existing kept sites in the ledger can serve as proofs" — ledger has no rung-S rows; proofs used class-picked raw sites.
|
||||
|
||||
## Commands (.run/logs)
|
||||
t2_proof_sc01, t2_proof_sc05, t2_r22 (c1); t2c2_strict (c2); t2c3_structs, t2c3_casts (c3); t2_verify_a, t2_verify_b, t2_kit (expert).
|
||||
@@ -1,3 +1,4 @@
|
||||
# Task summaries -- this phase
|
||||
# id | status | title | tags | summary | log | research
|
||||
T1 | done | phase-start health, carry cleanup and baselines | - | tasks/T1.md | logs/T1.md | R3.39-001
|
||||
T2 | done | cast and lying gate instruments, reinterpret-macro set | - | tasks/T2.md | logs/T2.md | -
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# T2 — cast and lying gate instruments, reinterpret-macro set
|
||||
|
||||
Status: done | expert: expert-fable | ctx-at-completion: n/a | commit: see COMMIT | coder runs: c1 opus55 done (247422bb83), c2 opus55 done (313ee50403), c3 opus55 done (7845da8f03)
|
||||
Done: `type_census --check-casts` gate (raw by form, macro sites vs restruct ledger, lying vs TSV), kept-cast macro set in include/common.h with five make-check proofs + one R22, `lever_census --strict --residue`, both selftests extended, caches invalidated, check-structs still 0.
|
||||
Files:
|
||||
- include/common.h:62-78 — CAST_ALIAS / CAST_SIGN / CAST_WIDTH / CAST_MISALIGNED / CAST_NONSTRUCT, each `(T, p, k)` → `(*(T *)((p) + (k)))`, one cause class each; LOBU..LOWU (:55-61, already present since P37 T3) kept as class REINTERP; M2C_FIELD unchanged (raw form M) (c1)
|
||||
- tools/type_census.py — KEPT_MACROS registry + FORM_K :141-147; form-K sites (tu, fn, line, pos, name, cls, base/bclass/off) :374-394; registered `#define` lines in include/common.h dropped from counts :612-620; `check_casts_inputs()` (restruct.ledger_latest, config/lying_exceptions.tsv), `check_casts_verdict()`, `--check-casts`; selftest 32 → 50 checks (c3)
|
||||
- tools/lever_census.py — `load_residue`, `strict_failures(sites, pt, gte_header, residue)` :1109-1148, `--residue PATH` :1219, main --strict prints `residue <n>, residue-unused <n>` + first 10 unused rows; selftest controls i-iv :1194-1209 (c2)
|
||||
- config/lying_exceptions.tsv — header only: `callee def_file kind cause task` (ledgered = row matches callee + defined_in + kind)
|
||||
- config/lever_residue.tsv — header only: `file func kind pass instrument cause` (match on file + func + kind; pass/instrument carried, R113)
|
||||
- docs/gen3-standards.md — §2 rule 5 (every raw cast goes or names its cause; the set, the gate, the ledger reading) + definition-of-done clause (expert)
|
||||
- decomp-architect/corpus/**, docs/struct-map.md, .run/P36/census/lever_census.*, .run/P37/census/* — regenerated by kit-corpus and the census runs
|
||||
Interfaces (the exact shapes later tasks read):
|
||||
- `type_census --check-casts` → `check-casts: raw P=<n> I=<n> X=<n> M=<n>; macros <name>=<n> …|none (unbacked <n>); lying=<n> (ledgered <n>)`; exit 1 on raw > 0, unbacked > 0, lying − ledgered > 0.
|
||||
- Backed macro site: `restruct.ledger_latest[("S", tu, fn)]` row with verdict in {KEPT, S2, KEPT-ALL} and a `sites[]` entry `verdict == "KEPT(<class>)"` for the macro's class matching `off` and `base == "<bclass>:<base>"`; REINTERP macros match on `line`. restruct does not yet write macro respells or `KEPT(SIGN|MISALIGNED|NON-STRUCT|REINTERP)` hints (pass_hint :563-591 has SCHED-ALIAS, ALIGNMENT, WIDTH only).
|
||||
- Classes: SCHED-ALIAS, SIGN, WIDTH (overlap loser), MISALIGNED, NON-STRUCT, REINTERP.
|
||||
Decisions:
|
||||
- Legacy LOBU/LOH/HIH/LOHU/HIHU/LOW/LOWU reinstated as the sixth class REINTERP (lvalue read at another width/sign), not retired; their byte proof is restruct's macro_probe (`--selftest --real` 82/82 at b5e18e5f83, T1).
|
||||
- Registered macros' `#define` lines in include/common.h are definitions, not sites: M = 19,542 (T1's 19,543 included common.h:43 itself); P unchanged at 408,974.
|
||||
- binding: a kept cast is spelled only by a registered macro of include/common.h and is honest only with its ledger row; `--check-casts` is the campaign's gate, `--check` stays the broader (non-gating on block frames) reading.
|
||||
- binding: reading A for levers: `lever_census --check --strict --residue config/lever_residue.tsv` is the gate; residue rows are excluded from failures and counted in every total.
|
||||
Deviations:
|
||||
- Byte-proof respells were restored after BYTE-IDENTICAL (tree has 0 macro sites, unbacked 0) so the next task inherits no unledgered macro sites; proof lines in logs/T2.c1.md.
|
||||
- `--strict` only runs inside `--check` (unchanged tool shape), so the gate spelling is `--check --strict --residue …`.
|
||||
- Residue cannot name a per-TU asm macro definition (mdefs_all is local to run_census): while pt > 0 strict fails with one synthetic `asm-macro` entry; a later task exposes mdefs if residue must cover them.
|
||||
- No restruct/delever apply was run, so no recalibration was needed; `restruct --selftest --real` is still red after these commits (T1 gotcha) — the next apply task recalibrates first.
|
||||
- commit_task auto-staged the pre-modified phase-ends/current/discussions/INDEX.md into 247422bb83 (one router line).
|
||||
Findings:
|
||||
- Byte proofs (all BYTE-IDENTICAL, .run/logs/t2_proof_sc01.log, t2_proof_sc05.log): CAST_ALIAS src/ov_SC05_010/ov_SC05_010_jr_80180F84.c:3006 (func_801814AC `+0x34` store, cookbook §458); CAST_SIGN src/ov_SC01_000/ov_SC01_000_after.c:2682 (func_801523F4); CAST_WIDTH src/ov_SC01_000/ov_SC01_000_jr_801734BC.c:2582 (func_801749C8); CAST_MISALIGNED src/ov_SC01_000/ov_SC01_000_jr_8012ACE0.c:2624 (func_801345F8 `*(int *)(param_1 + 6)`); CAST_NONSTRUCT same file :795 (func_8012E364). R22 after the common.h edit: 218 passed 0 failed (.run/logs/t2_r22.log).
|
||||
- Ledger .run/P37/restruct/ledger.jsonl: 173,459 rows, rung S = 0, `KEPT(` = 0 — the gate's macro half starts empty.
|
||||
- Live gate on this tree: `check-casts: raw P=408974 I=60666 X=13800 M=19542; macros none (unbacked 0); lying=229 (ledgered 0)` rc 1; `lever_census --check --strict --residue`: pins 1868, asm 1411 (non-GTE), gte-levers 450, direct GTE 6717, per-TU asm macro defs 314, residue 0, residue-unused 0 — FAIL rc 1.
|
||||
Gotchas:
|
||||
- harness: a PreToolUse hook denies `sed`/`cat` on tools/*.py for the expert even when the tool is the task's deliverable; source facts go through retriever-code.
|
||||
- harness: `tools/run.sh --wait --max 280` inside one Bash call plus the --bg launch exceeds the 285 s call budget when the census runs cold; launch and wait in separate calls.
|
||||
- generalizable: a macro definition line that textually matches a site regex is counted as a site by a text census; a gate must exclude the registry's own `#define` lines by name, not by file.
|
||||
- workflow: a tools/ edit → `make kit-corpus` → `make tools-health` before the task commit (corpus copies are tracked).
|
||||
Research: none new (three retriever-code answers inline, no report files).
|
||||
Next task needs: restruct must learn to (a) respell a KEPT site with its class macro and (b) write `KEPT(<class>)` with the four new hints (SIGN, MISALIGNED, NON-STRUCT, REINTERP) before any macro site can be backed; recalibrate (`delever_oracle --snapshot-baseline` + `--calibrate …`) before the first apply; denominators: P 408,974 I 60,666 X 13,800 M 19,542, lying 229/24 callees, levers 3,729.
|
||||
Verified: `PY tools/type_census.py --selftest && PY tools/lever_census.py --selftest && PY tools/type_census.py --check-structs --quiet` → 50/50 checks OK; selftest OK — 24 sites; check-structs OK, conflicting_types=0, rc 0 (.run/logs/t2_verify_a.log); `PY tools/type_census.py --check-casts` → the live line above, rc 1 as the verify requires (.run/logs/t2_verify_b.log)
|
||||
Full log: phase-ends/current/logs/T2.md
|
||||
Reference in New Issue
Block a user