From 40477281ce1554ff0074a95a03d2c7d3abddcef3 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 14 Jul 2026 21:21:02 -0600 Subject: [PATCH] =?UTF-8?q?fix(phase-26a):=20A9d=20=E2=80=94=20retire=20th?= =?UTF-8?q?e=20dead=20Phase-17=20canonical-sig=20chain=20(R33)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DELETE tools/census_conflict_callees.py + tools/derive_canonical_sigs.py. - census_conflict_callees: audit-CONFIRMED marked-for-deletion (commit:0593; decision-log 836). It re-derives from C text the per-TU "defined/declared/ stubbed/external?" question that reconcile_tu (Phase 26) answers FROM THE BUILD — and does it WRONG in the unsafe direction (unknown -> conflict-free). - derive_canonical_sigs (census's ONLY consumer): genuinely dead — last touched Phase-17 (commit:0140), output .run/canonical_sigs.json read by nothing (no Makefile/workflow/import), no-ops on the 2-byte [] input, asm-arity heuristic 36% wrong vs byte-exact banked C. Its purpose was retired in A3d (fleet-majority oracle -> reconcile_tu's per-TU oracle). Deleting census orphans it, so the whole dead chain ceases to exist (R33: the best outcome is a DELETED SCANNER, not a fixed regex). Byte-neutral by construction (neither tool is in any build/report path): module-import smoke over the 13 importable harvest/bank/report/reconcile tools = all clean; bank_exemplar is a run-only script (indexes sys.argv at module scope), imports neither deleted module. No src/config change -> no byte moves. Doc-pointer hygiene: hand-matching-process.md 8a, matching-cookbook.md (canonical-sig-layer entry), tooling-audit.md (ledger row + derive entry) all annotated DELETED/historical so nothing points at a nonexistent tool. --- docs/hand-matching-process.md | 3 + docs/matching-cookbook.md | 2 +- docs/tooling-audit.md | 9 +- phase-ends/CURRENT_PHASE.md | 26 ++++++ tools/census_conflict_callees.py | 142 ----------------------------- tools/derive_canonical_sigs.py | 148 ------------------------------- 6 files changed, 37 insertions(+), 293 deletions(-) delete mode 100644 tools/census_conflict_callees.py delete mode 100644 tools/derive_canonical_sigs.py diff --git a/docs/hand-matching-process.md b/docs/hand-matching-process.md index 706231634..a54515ca1 100644 --- a/docs/hand-matching-process.md +++ b/docs/hand-matching-process.md @@ -342,6 +342,9 @@ big wave.** Targets: `.run/harvest_targets_s3.json` (300, relocs≤5, reach-sort gcc-quirk tail, so the next lever is understanding gcc-2.7.2 (R17 research, Phase 18), NOT more brute waves.** ### 8a. What was built (committed, byte-neutral, reusable) +> **DELETED in Phase 26-A (R33):** both tools below were removed — `reconcile_tu`/`cdecl` answer their +> question (a TU's *visible* declarations) **from the build**, not by re-parsing C text. This section is +> retained as the historical Phase-17 record of the (now-retired) fleet-canonical-sig approach. - `tools/census_conflict_callees.py` — the accurate conflict predicate: an undeclared-`stub` callee with `decl_sources = n_callers + is_target >= 2` is a sig-conflict risk (a `declared`/`defined`/`extern` callee is conflict-free; gen_harvest_targets feeds the one sig). Writes `.run/conflict_callees.json`. diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index f6e689791..c1dea1c67 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -1257,7 +1257,7 @@ callees inconsistently → `conflicting types` in the one-big-TU; 100% compile-e SURGICAL per-callee canonical-sig layer: `tools/census_conflict_callees.py` (the conflict predicate: undeclared-stub callee with `decl_sources = n_callers + is_target >= 2`) + `tools/derive_canonical_sigs.py` (byte-neutral `s32 func_X(s32...)`, arity from Ghidra-C + asm read-before-write `$a0-$a3`) → a 20-extern -block at the TOP of `ov_SC01_077.c` (LOCAL, not engine_core.h — reach-1 names differ across overlays). +block at the TOP of `ov_SC01_077.c` (LOCAL, not engine_core.h — reach-1 names differ across overlays). *(Both `census_conflict_callees.py` and `derive_canonical_sigs.py` were **DELETED in Phase 26-A** — R33; the fleet-canonical-sig approach was superseded by `reconcile_tu`'s per-TU oracle. Historical record.)* `gen_harvest_targets` + `sig_unify` auto-read it; **the gate pipeline is now draft → `sig_unify` (MANDATORY) → `harvest_verify --chunk 1`** (the accumulating baseline now carries the file-top block, so a raw draft's guessed extern would clash without sig_unify). **SIZING CORRECTION (R14):** for the *remaining 270*, the diff --git a/docs/tooling-audit.md b/docs/tooling-audit.md index 8dbbb53f1..e614b67eb 100644 --- a/docs/tooling-audit.md +++ b/docs/tooling-audit.md @@ -716,7 +716,7 @@ scanners** — the "best outcome is a deleted scanner" rule (R33), applied at sc | A3 | `harvest_verify` + `gate_stage` | **the byte-gate could see ONE TU**: 4.9% of ov_SC01_077, **96.6% of fleet stubs unreachable**, **1,290 of the grinder's own 1,298 queued fns unbankable** | **100%**; each draft spliced into the TU that holds its stub; verdict untouched | `commit:0591` | | A3 | `family_manifest` | matched-set from **one overlay** → the endgame plan advertised **2,758 families / 11.0 MB**; **1,071 (62% of the byte-weight) were ALREADY MATCHED** | **1,495 / 3.9 MB** — derived from the invariant; the dedup-hash scanner **deleted** | `commit:0593` | | A3 | `family_hseq` | `func_`-only stub regex → 3 phantom "matched" exemplars | corpus-derived (+100 curated stubs); stale hardcoded baseline labelled | `commit:0593` | -| A3 | `census_conflict_callees` | wave scope **2** when the truth is **57** (96% under-report) | **0/57**; **MARKED FOR DELETION** (R33 — `reconcile_tu` answers it from the build) | `commit:0593` | +| A3 | `census_conflict_callees` | wave scope **2** when the truth is **57** (96% under-report) | **0/57**; **DELETED** with `derive_canonical_sigs` (its only consumer) — R33, the dead Phase-17 chain ceases to exist; `reconcile_tu` answers it from the build | `commit:0593`→A9d | | A3 | **`tools/cdecl.py`** *(new)* + `make audit-cdecl` | **fifteen** tools each carried their own regex model of "what is a C declaration", and they disagreed; all fifteen were blind to fn-ptr / sized-array / multi-declarator decls | ONE recursive-descent parser of the C **declarator grammar** — total by construction, not by shape enumeration. **2,952,246 statements → 2,731,521 declarators, 0 parser defects; 50,405 declarations round-tripped through the real cross-gcc, 0 rejected.** Cookbook **§51g** (LAWS 4–8) | *(this commit)* | ### What `cdecl` measured that the audit had not (all new, all reproducible via `make audit-cdecl`) @@ -1174,7 +1174,12 @@ CLEAN RESULT worth recording: the regex's hardcoded NAME list is COMPLETE — in - **assertion (R32):** R32: after stripping, assert NO typedef of an include/-provided name survives anywhere in the source text — i.e. re-scan the stripped source and hard-fail if any of the 15 known names is still being typedef'd. That is a direct coverage assertion (candidates vs parsed) and it catches every future line-shape the regex has not anticipated, permanently. Separately, assert the hardcoded name set == the typedef names parsed out of include/common.h. - **skeptic:** RAN: (a) my own scanner over .run/drafts*/*.c (10,731 files) -> candidates=5638, SCALAR_TYPEDEF_RE matches=4719, gap=919 — exact reproduction. (b) My own independent classifier of the 919 (split each line on ';', check whether any declarator's terminal identifier is one of the 15 names include/common.h really defines) -> 55 collide / 864 justified, across 42 files — exact reproduction; the 864 are genuinely justified (multi-line `typedef struct {` openers, draft-local names like Blob8/S801563EC). (c) Shape audit: 55/55 skip lines have >1 semicolon, 9 distinct shapes, all multi-typedef-per-line — the claimed anchor mechanism is the only one present. (d) Cited locations verified: masked_diff.py:36-40 (regex), match_one.py:51 (the .sub call), and all three example file:lines exist verbatim. (e) Consequence executed against the real pinned cc1 (tools/bin/gcc-2.7.2-psx/cc1, match_one's exact flags): exit=33, "redefinition of `u8'" / "include/common.h:16: `u8' previously declared here" -> match_one.py:64 prints CC1 FAIL, exit(1). (f) THE COUNTERFACTUAL that defeated my skepticism: .run/drafts-t3b1-fb/func_8015C030.c, with NEWLINES as the only edit (split the multi-typedef lines), goes from CC1 FAIL to "MATCH (23 ins)" — and func_8015C030 is STILL unbanked (134 INCLUDE_ASM stubs, e.g. src/ov_SC06_008/ov_SC06_008_jr_8015AE2C.c:1556). A masked-MATCH draft for a live 134-site family target is being discarded over whitespace. BOUNDS I VERIFIED (kept it at MEDIUM, not higher): 0 of the 42 files sit in the default auto-searched dirs (.run/drafts3|2|), so damage needs an explicit --c; silence is caller-dependent (auto_driver.py:93 books it ('fail',-1) and ab_score.py:56 books it 'compilefail' — truly silent; api_draft.py:199 feeds the CC1 FAIL text back to the drafter, self-healing; interactive is loud); and it is a false NEGATIVE only — no wrong bytes can pass through it, no false green. Counts, mechanism, evidence, and consequence all reproduce; I could not shrink it. -### [MEDIUM] `derive_canonical_sigs.py` — DOWNGRADED +### [MEDIUM] `derive_canonical_sigs.py` — DOWNGRADED → **DELETED (A9d, R33)** +> Deleted together with `census_conflict_callees` (its only input generator). No live consumer (output +> `.run/canonical_sigs.json` read by nothing; no Makefile/workflow; last touched Phase-17). The +> canonical-sig purpose was retired in A3d (the fleet-majority oracle → `reconcile_tu`'s per-TU oracle), +> and the asm-arity heuristic is 36% wrong vs byte-exact banked C — so per R33 the heuristic **ceases to +> exist** rather than being fixed. Original DOWNGRADED analysis retained below as the evidence trail. - **scanner:** tools/derive_canonical_sigs.py:100 asm_arity() via `asm_dir = asm/{source}/nonmatchings/{source}` (:130) — a hardcoded subdir; AND tools/derive_canonical_sigs.py:77 a_role() `src = set(a_in_order) - dest` - **counts:** candidates **264** / parsed **13** / real skips **251** - **evidence:** TWO independent holes in the arity oracle — the same class as the callee-signature hole that once made nine byte-exact functions look like a compiler wall. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index bc5a9500e..cfa15450d 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -659,6 +659,32 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag ## Log +- **2026-07-14 (session 13, A9d — retire the dead Phase-17 canonical-sig chain; Max):** R33 applied to a + two-tool dead chain. **DELETED `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py`.** + `census` was audit-CONFIRMED MARKED-FOR-DELETION (`commit:0593`; decision-log 836 "in its entirety gets + deleted"): it re-derives from C text the question — "for this TU, which callees are defined/declared/ + stubbed/external?" — that `reconcile_tu` (Phase 26) answers FROM THE BUILD; a parse hole in `reconcile_tu` + is conservative, a parse hole here is WRONG in the unsafe direction (unknown→"conflict-free"). Its output + fed only `derive_canonical_sigs`, which the audit DOWNGRADED but which is genuinely **dead**: last touched + Phase-17 (`commit:0140`), output `.run/canonical_sigs.json` read by **nothing** (no Makefile target, no + workflow, no module import — all verified by grep), no-ops on the 2-byte `[]` input, and its asm-arity + heuristic is **36% wrong vs byte-exact banked C** (audit skeptic, ground-truth run). Its canonical-sig + purpose was itself retired in **A3d** (fleet-majority oracle → `reconcile_tu`'s per-TU oracle). So the + whole dead chain **ceases to exist** rather than leaving an orphaned, non-runnable, wrong-by-36% scanner + (R33: "the best outcome is a DELETED SCANNER"). **Byte-neutral BY CONSTRUCTION** — neither tool is in any + build/report path; verification = module-import smoke over the 13 importable harvest/bank/report/reconcile + tools (`corpus`/`cdecl`/`reconcile_tu`/`canon_sig_reconcile`/`gen_harvest_targets`/`gate_stage`/ + `jtbl_family_bank`/`family_sweep`/`dedup_propagate`/`sig_unify`/`cast_call_sites`/`scope_data_externs`/ + `family_remap`) = **all clean** (`bank_exemplar` is a run-only script that indexes `sys.argv` at module + scope → not importable by design, imports neither deleted module — a test-method false alarm, not + breakage). No `src/`/`config/` change ⇒ no byte can move ⇒ full R22 not warranted (P9-honest: a 136-binary + rebuild would prove only what the absence of a build-path edit already guarantees). **Doc-pointer hygiene** + (so nothing points at a nonexistent tool): `docs/hand-matching-process.md` §8a, `docs/matching-cookbook.md` + (canonical-sig-layer entry), and `docs/tooling-audit.md` (ledger row + the `derive_canonical_sigs` entry) + all annotated DELETED/historical. **NEXT: A9e — wire `reconcile_tu` into `bank_exemplar`** (its stage + ladder is `raw→scoped→recovered→reconciled`; `recovered`=`fb.recover`, `reconciled`=`canon_sig_reconcile` + — add a `reconcile_tu` stage, byte-gate); then A9f (overlay_src_split force_decl latch), A9g (jr_inventory + ephemeral read), A10 (wall re-test), A11 (distill + close → resume Phase 26 Task 7). - **2026-07-14 (session 12, A9a+A9b — canon_sig_reconcile fn-ptr fix + wall re-test; Max):** Continued the tool-hygiene audit. **A9a — `canon_sig_reconcile` sees fn-ptr dispatch tables (`commit:0609`).** The def-side-wall recovery tool, live on the ×134 economic-engine paths (jtbl_family_bank, family_sweep --reconcile-raw, diff --git a/tools/census_conflict_callees.py b/tools/census_conflict_callees.py deleted file mode 100644 index e29a46745..000000000 --- a/tools/census_conflict_callees.py +++ /dev/null @@ -1,142 +0,0 @@ -#!/usr/bin/env python3 -"""Census the shared callees that block the parallel hand-matching wave (Phase 17 canonical-sig layer). - -The calibration wave (hand-matching-process.md §7c) hit 60% match_one MATCH but only 33% whole-binary; -the entire gap was SIG CONFLICTS — parallel agents each declare an undeclared-stub shared callee (e.g. -func_80131CA8) with a different guessed signature, which then clash in the one-big-TU ov_SC01_077.c -(`conflicting types for func_X`). 100% compile-errors, ZERO codegen mismatches. - -A callee that is already `defined` (engine_core.h DEFINE / inline body) or already `declared` (an extern -written somewhere) is conflict-FREE — gen_harvest_targets resolves it and every draft reuses the one sig. -The conflict set is exactly the `stub` callees (no body, no extern anywhere) that >=2 still-stub wave -targets call. Declaring ONE canonical extern for each (the canonical-sig layer) turns them `declared` -> -the wave stops conflicting on them. - -This is read-only. Output: the ranked conflict-callee table + the call-edge status totals. - -Usage: - tools/census_conflict_callees.py [--source ov_SC01_077] [--targets .run/harvest_targets_s3.json] -""" -import argparse, json, os, importlib.util -from collections import defaultdict - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -import sys -sys.path.insert(0, os.path.join(REPO, 'tools')) -import corpus # the derived corpus oracle (Phase 26-A) - -_spec = importlib.util.spec_from_file_location('ght', os.path.join(REPO, 'tools/gen_harvest_targets.py')) -_ght = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(_ght) - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument('--source', default='ov_SC01_077') - ap.add_argument('--targets', default='.run/harvest_targets_s3.json') - ap.add_argument('--min-callers', type=int, default=2) - ap.add_argument('--out', default='.run/conflict_callees.json') - args = ap.parse_args() - - src = args.source - ec = os.path.join(REPO, 'src/shared/engine_core.h') - - # ⚠ DEPRECATED — SCHEDULED FOR DELETION (Phase 26-A audit, R33). - # This tool re-derives, by re-parsing C text, the question "for this TU, which callees are - # defined / declared / stubbed / external?" — which is exactly and only what tools/reconcile_tu.py - # answers FROM THE BUILD. A parse hole in reconcile_tu makes it CONSERVATIVE; a parse hole here - # makes it WRONG IN THE UNSAFE DIRECTION (an unknown callee is silently bucketed "conflict-free"). - # Delete this file once reconcile_tu is wired into derive_canonical_sigs (its only consumer). - # - # Until then, at least stop the 96% under-report: the corpus was `src//.c` ALONE, so it - # saw 13 of ov_SC01_077's 264 stubs and printed "wave scope: 2 still-stub" when the truth is 57 — - # every downstream percentage was computed against a denominator 96% too small. The 0-conflict - # answer it gives today is right BY LUCK (those callees happen to be banked); point it at a fresh - # overlay and it would see ~14 of ~600 stubs and confidently report zero conflicts into a wave - # riddled with them — the exact failure that cost 60%-vs-33% in the Phase-17 calibration. - c_paths = [str(p) for p in corpus.src_files(src)] - c_path = c_paths[0] if c_paths else os.path.join(REPO, f'src/{src}/{src}.c') - - src_sig = _ght.load_sig(os.path.join(REPO, f'.run/sig.{src}.jsonl')) # addr-int -> {calls,nins,h_exact,reach?} - stubs = set(corpus.stubs(src)) # ADDRESSES (ints) — ALL TUs, ANY symbol name - define_sigs = _ght.collect_define_sigs(ec) - inline_sigs = {} - for cp in c_paths: - inline_sigs.update(_ght.collect_inline_sigs(cp)) - defined = {**define_sigs, **inline_sigs} - extern_sigs = _ght.collect_extern_sigs([ec] + c_paths) - - s3 = json.load(open(os.path.join(REPO, args.targets))) - tgt_addrs = [int(t['addr'], 16) for t in s3] - remaining = [a for a in tgt_addrs if a in stubs] # still-stub wave targets - - def status(caddr): - if caddr in defined: return 'defined' - if caddr in extern_sigs: return 'declared' - if caddr in stubs: return 'stub' - return 'extern' # resident/EXE, conflict-free - - # reach map (how many overlays hold a byte-identical copy) for prioritization - reach = {int(t['addr'], 16): t.get('reach', 1) for t in s3} - - callee_callers = defaultdict(set) # callee_addr -> {target addrs calling it (excl. self)} - edge_status = defaultdict(int) # status -> # call edges (over remaining targets) - for a in remaining: - rec = src_sig.get(a) - if not rec: - continue - for ch in set(rec.get('calls', [])): - caddr = int(ch, 16) - edge_status[status(caddr)] += 1 - if caddr != a: - callee_callers[caddr].add(a) - - # THE conflict predicate (hand-matching-process.md §7c): a sig conflict needs >=2 INDEPENDENT - # declarations of an UNDECLARED stub in the one-big-TU build. Declaration sources for callee X = - # one extern per drafted caller + one definition if X is itself a drafted target (def-vs-extern). - # decl_sources = n_callers + (1 if X is a remaining target else 0) - # An already-`declared`/`defined` callee is conflict-FREE (gen_harvest_targets feeds the one sig). - conflicts = [] - for caddr, callers in callee_callers.items(): - if status(caddr) != 'stub': - continue - is_t = caddr in remaining - decl_sources = len(callers) + (1 if is_t else 0) - if decl_sources < 2: - continue - crec = src_sig.get(caddr, {}) - conflicts.append({ - 'callee': f'func_{caddr:08X}', 'addr': f'{caddr:08X}', - 'n_callers': len(callers), 'is_target': is_t, 'decl_sources': decl_sources, - 'kind': 'match-first' if is_t else 'derive-declare', - 'callee_nins': crec.get('nins'), 'callee_ncalls': len(crec.get('calls', [])), - 'callee_reach': reach.get(caddr, crec.get('reach')), - }) - conflicts.sort(key=lambda x: (-x['decl_sources'], -(x['callee_reach'] or 0))) - - blocked = {a for a in remaining - if any(int(ch, 16) in {int(c['addr'], 16) for c in conflicts} - for ch in src_sig.get(a, {}).get('calls', []))} - blk_reach = sum(reach.get(a, 1) for a in blocked) - tot_reach = sum(reach.get(a, 1) for a in remaining) - - json.dump(conflicts, open(os.path.join(REPO, args.out), 'w'), indent=1) - - print(f'wave scope: {len(s3)} s3 targets, {len(remaining)} still-stub (remaining)') - print('call-edge status totals (over remaining targets): ' - + ' '.join(f'{k}:{edge_status[k]}' for k in ('defined', 'declared', 'stub', 'extern'))) - mf = sum(1 for c in conflicts if c['is_target']) - print(f'\nCONFLICT CALLEES (undeclared stub, decl_sources>=2): {len(conflicts)} ' - f'[{mf} match-first / {len(conflicts)-mf} derive-declare]') - print(f'{"callee":>16} {"callers":>7} {"isTgt":>5} {"srcs":>4} {"nins":>5} {"ncalls":>6} {"reach":>5}') - for c in conflicts: - print(f'{c["callee"]:>16} {c["n_callers"]:>7} {("Y" if c["is_target"] else ""):>5} ' - f'{c["decl_sources"]:>4} {str(c["callee_nins"] or "-"):>5} ' - f'{str(c["callee_ncalls"] or "-"):>6} {str(c["callee_reach"] or "-"):>5}') - print(f'\nwave targets blocked by >=1 conflict callee: {len(blocked)}/{len(remaining)} ' - f'(reach-weighted {blk_reach}/{tot_reach} = {100*blk_reach//max(tot_reach,1)}% of wave reach)') - print(f'wrote {args.out}') - - -if __name__ == '__main__': - main() diff --git a/tools/derive_canonical_sigs.py b/tools/derive_canonical_sigs.py deleted file mode 100644 index 539d42197..000000000 --- a/tools/derive_canonical_sigs.py +++ /dev/null @@ -1,148 +0,0 @@ -#!/usr/bin/env python3 -"""Derive a byte-neutral canonical signature for each Phase-17 conflict callee (the canonical-sig layer). - -Input: .run/conflict_callees.json (from census_conflict_callees.py) — the undeclared-stub callees that -parallel hand-matching agents would declare inconsistently (hand-matching-process.md §7c). For each we -emit ONE canonical `extern s32 func_X(s32 a0, ...);` to seed src/shared/engine_core.h, so gen_harvest_targets -feeds every drafting agent the SAME signature and the one-big-TU build stops conflicting. - -Canonical form = WIDEST byte-neutral (hand-matching-process.md §3a): - - return `s32` : void->s32 is byte-neutral (no explicit return => identical epilogue); s32 is REQUIRED - where a caller uses $v0. So s32 is universally safe. - - params `s32` : widest scalar; a matched body casts int->ptr (`*(T*)(a0+off)`, the demo idiom) and a - caller narrows in the call expression. s32 never blocks a match; the byte-gate validates. - - ARITY is the only value that must be exact (a wrong count => "too few/many arguments" at a caller, or a - def/extern arity clash for a circular target). Derived two ways and cross-checked: - (G) Ghidra-C cache .run/ghidra_c/func_.c (FUN_(...) param count) — the static oracle (G1). - (A) asm read-before-write of $a0..$a3 in asm//nonmatchings//func_.s — an a-reg whose - FIRST-touching instruction uses it as a SOURCE is an incoming param; dest-only first touch - (lui/lw/move/ALU-dest) = scratch, not a param. Arity = highest param index + 1 (contiguous). - -The whole-binary harvest_verify byte-gate remains the sole arbiter (G3/P9): a wrong arity just fails the -gate and is fixed per-callee. This only shapes the drafting / seeds the canonical decls. - -Usage: - tools/derive_canonical_sigs.py [--source ov_SC01_077] [--in .run/conflict_callees.json] -""" -import argparse, json, os, re, glob - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -AREGS = ['a0', 'a1', 'a2', 'a3'] - -# instruction operand roles restricted to what we need: is the FIRST a-reg touch a source (=> param)? -LOAD = {'lw', 'lh', 'lhu', 'lb', 'lbu', 'lwl', 'lwr', 'll', 'lwc1', 'lwc2', 'ldc1', 'ldc2'} # rt=dest, base=src -STORE = {'sw', 'sh', 'sb', 'swl', 'swr', 'sc', 'swc1', 'swc2', 'sdc1', 'sdc2'} # rt=src, base=src -DEST_FIRST = {'lui', 'li', 'move', 'addu', 'addiu', 'subu', 'and', 'andi', 'or', 'ori', 'xor', # rd/rt=dest, rest=src - 'xori', 'nor', 'slt', 'sltu', 'slti', 'sltiu', 'sll', 'srl', 'sra', 'sllv', 'srlv', - 'srav', 'mul', 'mult', 'negu', 'neg', 'not', 'mflo', 'mfhi', 'movn', 'movz', 'sub', - 'add', 'rotr', 'clz', 'seb', 'seh', 'mfc1', 'mfc2', 'la'} -SRC_ALL = {'beq', 'bne', 'beqz', 'bnez', 'blez', 'bgtz', 'bltz', 'bgez', 'bgezal', 'bltzal', # all regs are src - 'jr', 'jalr', 'multu', 'divu', 'div', 'mtlo', 'mthi', 'mtc1', 'mtc2', 'teq', 'tne', - 'beql', 'bnel', 'cache'} - - -def reg_tokens(operand_str): - return re.findall(r'\$([a-z0-9]+)', operand_str) - - -def a_role(mnem, ops): - """return (a_sources, a_dest) restricted to a0..a3 for one instruction.""" - regs = reg_tokens(ops) - a_in_order = [r for r in regs if r in AREGS] - if not a_in_order: - return set(), set() - if mnem in LOAD: - dest = {regs[0]} & set(AREGS) if regs else set() - src = set(a_in_order) - dest - elif mnem in STORE: - dest, src = set(), set(a_in_order) - elif mnem in DEST_FIRST: - dest = {regs[0]} & set(AREGS) if regs else set() - src = set(a_in_order) - dest - elif mnem in SRC_ALL: - dest, src = set(), set(a_in_order) - else: # unknown: be conservative -> treat as sources (flags a param) - dest, src = set(), set(a_in_order) - return src, dest - - -INSN_RE = re.compile(r'\*/\s+([a-z][a-z0-9.]*)\s+(.*)$') # after the `... XXXX */` comment - - -def asm_arity(s_path): - """highest read-before-write a-reg index +1; returns (arity, note).""" - if not os.path.exists(s_path): - return None, 'no-asm' - first = {} # areg -> 'param' | 'scratch' - for line in open(s_path): - m = INSN_RE.search(line) - if not m: - continue - mnem, ops = m.group(1), m.group(2) - src, dest = a_role(mnem, ops) - for r in AREGS: - if r in first: - continue - if r in src: - first[r] = 'param' - elif r in dest: - first[r] = 'scratch' - arity = 0 - for i, r in enumerate(AREGS): - if first.get(r) == 'param': - arity = i + 1 - # contiguity note: a param above a scratch/untouched gap (loose-typed) -> flag - gap = any(first.get(AREGS[j]) != 'param' for j in range(arity - 1)) if arity else False - return arity, ('gap' if gap else 'ok') - - -GHIDRA_SIG_RE = re.compile(r'^\s*[A-Za-z_].*\bFUN_[0-9a-f]+\s*\((.*?)\)\s*$', re.M) - - -def ghidra_arity(addr): - p = os.path.join(REPO, f'.run/ghidra_c/func_{addr}.c') - if not os.path.exists(p): - return None - m = GHIDRA_SIG_RE.search(open(p).read()) - if not m: - return None - params = m.group(1).strip() - if params in ('', 'void'): - return 0 - return len([x for x in params.split(',') if x.strip()]) - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument('--source', default='ov_SC01_077') - ap.add_argument('--in', dest='infile', default='.run/conflict_callees.json') - args = ap.parse_args() - asm_dir = os.path.join(REPO, f'asm/{args.source}/nonmatchings/{args.source}') - conf = json.load(open(os.path.join(REPO, args.infile))) - - print(f'{"callee":>16} {"kind":>14} {"ghidra":>6} {"asm":>4} {"note":>6} {"->arity":>7} canonical') - rows = [] - for c in conf: - addr = c['addr'] - g = ghidra_arity(addr) - a, note = asm_arity(os.path.join(asm_dir, f'func_{addr}.s')) - # reconcile: prefer asm (callee's own consumption); if asm gap or asm14} {str(g):>6} {str(a):>4} {note:>6} {arity:>7} {sig}') - - out = os.path.join(REPO, '.run/canonical_sigs.json') - json.dump(rows, open(out, 'w'), indent=1) - print(f'\nwrote {out} ({len(rows)} canonical sigs)') - print('disagreements (G!=A) to eyeball:', - ', '.join(r['callee'] for r in rows if r['note'].startswith('G')) or 'none') - - -if __name__ == '__main__': - main()