mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-28 06:49:47 -04:00
fix(phase-26a): A9d — retire the dead Phase-17 canonical-sig chain (R33)
DELETE tools/census_conflict_callees.py + tools/derive_canonical_sigs.py. - census_conflict_callees: audit-CONFIRMED marked-for-deletion (commit:0593; decision-log 836). It re-derives from C text the per-TU "defined/declared/ stubbed/external?" question that reconcile_tu (Phase 26) answers FROM THE BUILD — and does it WRONG in the unsafe direction (unknown -> conflict-free). - derive_canonical_sigs (census's ONLY consumer): genuinely dead — last touched Phase-17 (commit:0140), output .run/canonical_sigs.json read by nothing (no Makefile/workflow/import), no-ops on the 2-byte [] input, asm-arity heuristic 36% wrong vs byte-exact banked C. Its purpose was retired in A3d (fleet-majority oracle -> reconcile_tu's per-TU oracle). Deleting census orphans it, so the whole dead chain ceases to exist (R33: the best outcome is a DELETED SCANNER, not a fixed regex). Byte-neutral by construction (neither tool is in any build/report path): module-import smoke over the 13 importable harvest/bank/report/reconcile tools = all clean; bank_exemplar is a run-only script (indexes sys.argv at module scope), imports neither deleted module. No src/config change -> no byte moves. Doc-pointer hygiene: hand-matching-process.md 8a, matching-cookbook.md (canonical-sig-layer entry), tooling-audit.md (ledger row + derive entry) all annotated DELETED/historical so nothing points at a nonexistent tool.
This commit is contained in:
Drew T
@@ -342,6 +342,9 @@ big wave.** Targets: `.run/harvest_targets_s3.json` (300, relocs≤5, reach-sort
|
||||
gcc-quirk tail, so the next lever is understanding gcc-2.7.2 (R17 research, Phase 18), NOT more brute waves.**
|
||||
|
||||
### 8a. What was built (committed, byte-neutral, reusable)
|
||||
> **DELETED in Phase 26-A (R33):** both tools below were removed — `reconcile_tu`/`cdecl` answer their
|
||||
> question (a TU's *visible* declarations) **from the build**, not by re-parsing C text. This section is
|
||||
> retained as the historical Phase-17 record of the (now-retired) fleet-canonical-sig approach.
|
||||
- `tools/census_conflict_callees.py` — the accurate conflict predicate: an undeclared-`stub` callee with
|
||||
`decl_sources = n_callers + is_target >= 2` is a sig-conflict risk (a `declared`/`defined`/`extern` callee
|
||||
is conflict-free; gen_harvest_targets feeds the one sig). Writes `.run/conflict_callees.json`.
|
||||
|
||||
@@ -1257,7 +1257,7 @@ callees inconsistently → `conflicting types` in the one-big-TU; 100% compile-e
|
||||
SURGICAL per-callee canonical-sig layer: `tools/census_conflict_callees.py` (the conflict predicate:
|
||||
undeclared-stub callee with `decl_sources = n_callers + is_target >= 2`) + `tools/derive_canonical_sigs.py`
|
||||
(byte-neutral `s32 func_X(s32...)`, arity from Ghidra-C + asm read-before-write `$a0-$a3`) → a 20-extern
|
||||
block at the TOP of `ov_SC01_077.c` (LOCAL, not engine_core.h — reach-1 names differ across overlays).
|
||||
block at the TOP of `ov_SC01_077.c` (LOCAL, not engine_core.h — reach-1 names differ across overlays). *(Both `census_conflict_callees.py` and `derive_canonical_sigs.py` were **DELETED in Phase 26-A** — R33; the fleet-canonical-sig approach was superseded by `reconcile_tu`'s per-TU oracle. Historical record.)*
|
||||
`gen_harvest_targets` + `sig_unify` auto-read it; **the gate pipeline is now draft → `sig_unify` (MANDATORY)
|
||||
→ `harvest_verify --chunk 1`** (the accumulating baseline now carries the file-top block, so a raw draft's
|
||||
guessed extern would clash without sig_unify). **SIZING CORRECTION (R14):** for the *remaining 270*, the
|
||||
|
||||
@@ -716,7 +716,7 @@ scanners** — the "best outcome is a deleted scanner" rule (R33), applied at sc
|
||||
| A3 | `harvest_verify` + `gate_stage` | **the byte-gate could see ONE TU**: 4.9% of ov_SC01_077, **96.6% of fleet stubs unreachable**, **1,290 of the grinder's own 1,298 queued fns unbankable** | **100%**; each draft spliced into the TU that holds its stub; verdict untouched | `commit:0591` |
|
||||
| A3 | `family_manifest` | matched-set from **one overlay** → the endgame plan advertised **2,758 families / 11.0 MB**; **1,071 (62% of the byte-weight) were ALREADY MATCHED** | **1,495 / 3.9 MB** — derived from the invariant; the dedup-hash scanner **deleted** | `commit:0593` |
|
||||
| A3 | `family_hseq` | `func_`-only stub regex → 3 phantom "matched" exemplars | corpus-derived (+100 curated stubs); stale hardcoded baseline labelled | `commit:0593` |
|
||||
| A3 | `census_conflict_callees` | wave scope **2** when the truth is **57** (96% under-report) | **0/57**; **MARKED FOR DELETION** (R33 — `reconcile_tu` answers it from the build) | `commit:0593` |
|
||||
| A3 | `census_conflict_callees` | wave scope **2** when the truth is **57** (96% under-report) | **0/57**; **DELETED** with `derive_canonical_sigs` (its only consumer) — R33, the dead Phase-17 chain ceases to exist; `reconcile_tu` answers it from the build | `commit:0593`→A9d |
|
||||
| A3 | **`tools/cdecl.py`** *(new)* + `make audit-cdecl` | **fifteen** tools each carried their own regex model of "what is a C declaration", and they disagreed; all fifteen were blind to fn-ptr / sized-array / multi-declarator decls | ONE recursive-descent parser of the C **declarator grammar** — total by construction, not by shape enumeration. **2,952,246 statements → 2,731,521 declarators, 0 parser defects; 50,405 declarations round-tripped through the real cross-gcc, 0 rejected.** Cookbook **§51g** (LAWS 4–8) | *(this commit)* |
|
||||
|
||||
### What `cdecl` measured that the audit had not (all new, all reproducible via `make audit-cdecl`)
|
||||
@@ -1174,7 +1174,12 @@ CLEAN RESULT worth recording: the regex's hardcoded NAME list is COMPLETE — in
|
||||
- **assertion (R32):** R32: after stripping, assert NO typedef of an include/-provided name survives anywhere in the source text — i.e. re-scan the stripped source and hard-fail if any of the 15 known names is still being typedef'd. That is a direct coverage assertion (candidates vs parsed) and it catches every future line-shape the regex has not anticipated, permanently. Separately, assert the hardcoded name set == the typedef names parsed out of include/common.h.
|
||||
- **skeptic:** RAN: (a) my own scanner over .run/drafts*/*.c (10,731 files) -> candidates=5638, SCALAR_TYPEDEF_RE matches=4719, gap=919 — exact reproduction. (b) My own independent classifier of the 919 (split each line on ';', check whether any declarator's terminal identifier is one of the 15 names include/common.h really defines) -> 55 collide / 864 justified, across 42 files — exact reproduction; the 864 are genuinely justified (multi-line `typedef struct {` openers, draft-local names like Blob8/S801563EC). (c) Shape audit: 55/55 skip lines have >1 semicolon, 9 distinct shapes, all multi-typedef-per-line — the claimed anchor mechanism is the only one present. (d) Cited locations verified: masked_diff.py:36-40 (regex), match_one.py:51 (the .sub call), and all three example file:lines exist verbatim. (e) Consequence executed against the real pinned cc1 (tools/bin/gcc-2.7.2-psx/cc1, match_one's exact flags): exit=33, "redefinition of `u8'" / "include/common.h:16: `u8' previously declared here" -> match_one.py:64 prints CC1 FAIL, exit(1). (f) THE COUNTERFACTUAL that defeated my skepticism: .run/drafts-t3b1-fb/func_8015C030.c, with NEWLINES as the only edit (split the multi-typedef lines), goes from CC1 FAIL to "MATCH (23 ins)" — and func_8015C030 is STILL unbanked (134 INCLUDE_ASM stubs, e.g. src/ov_SC06_008/ov_SC06_008_jr_8015AE2C.c:1556). A masked-MATCH draft for a live 134-site family target is being discarded over whitespace. BOUNDS I VERIFIED (kept it at MEDIUM, not higher): 0 of the 42 files sit in the default auto-searched dirs (.run/drafts3|2|), so damage needs an explicit --c; silence is caller-dependent (auto_driver.py:93 books it ('fail',-1) and ab_score.py:56 books it 'compilefail' — truly silent; api_draft.py:199 feeds the CC1 FAIL text back to the drafter, self-healing; interactive is loud); and it is a false NEGATIVE only — no wrong bytes can pass through it, no false green. Counts, mechanism, evidence, and consequence all reproduce; I could not shrink it.
|
||||
|
||||
### [MEDIUM] `derive_canonical_sigs.py` — DOWNGRADED
|
||||
### [MEDIUM] `derive_canonical_sigs.py` — DOWNGRADED → **DELETED (A9d, R33)**
|
||||
> Deleted together with `census_conflict_callees` (its only input generator). No live consumer (output
|
||||
> `.run/canonical_sigs.json` read by nothing; no Makefile/workflow; last touched Phase-17). The
|
||||
> canonical-sig purpose was retired in A3d (the fleet-majority oracle → `reconcile_tu`'s per-TU oracle),
|
||||
> and the asm-arity heuristic is 36% wrong vs byte-exact banked C — so per R33 the heuristic **ceases to
|
||||
> exist** rather than being fixed. Original DOWNGRADED analysis retained below as the evidence trail.
|
||||
- **scanner:** tools/derive_canonical_sigs.py:100 asm_arity() via `asm_dir = asm/{source}/nonmatchings/{source}` (:130) — a hardcoded subdir; AND tools/derive_canonical_sigs.py:77 a_role() `src = set(a_in_order) - dest`
|
||||
- **counts:** candidates **264** / parsed **13** / real skips **251**
|
||||
- **evidence:** TWO independent holes in the arity oracle — the same class as the callee-signature hole that once made nine byte-exact functions look like a compiler wall.
|
||||
|
||||
@@ -659,6 +659,32 @@ On approval → `/model opus` + `/effort xHigh` (Tasks 0–4; ALL Fable5 via `Ag
|
||||
|
||||
## Log
|
||||
|
||||
- **2026-07-14 (session 13, A9d — retire the dead Phase-17 canonical-sig chain; Max):** R33 applied to a
|
||||
two-tool dead chain. **DELETED `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py`.**
|
||||
`census` was audit-CONFIRMED MARKED-FOR-DELETION (`commit:0593`; decision-log 836 "in its entirety gets
|
||||
deleted"): it re-derives from C text the question — "for this TU, which callees are defined/declared/
|
||||
stubbed/external?" — that `reconcile_tu` (Phase 26) answers FROM THE BUILD; a parse hole in `reconcile_tu`
|
||||
is conservative, a parse hole here is WRONG in the unsafe direction (unknown→"conflict-free"). Its output
|
||||
fed only `derive_canonical_sigs`, which the audit DOWNGRADED but which is genuinely **dead**: last touched
|
||||
Phase-17 (`commit:0140`), output `.run/canonical_sigs.json` read by **nothing** (no Makefile target, no
|
||||
workflow, no module import — all verified by grep), no-ops on the 2-byte `[]` input, and its asm-arity
|
||||
heuristic is **36% wrong vs byte-exact banked C** (audit skeptic, ground-truth run). Its canonical-sig
|
||||
purpose was itself retired in **A3d** (fleet-majority oracle → `reconcile_tu`'s per-TU oracle). So the
|
||||
whole dead chain **ceases to exist** rather than leaving an orphaned, non-runnable, wrong-by-36% scanner
|
||||
(R33: "the best outcome is a DELETED SCANNER"). **Byte-neutral BY CONSTRUCTION** — neither tool is in any
|
||||
build/report path; verification = module-import smoke over the 13 importable harvest/bank/report/reconcile
|
||||
tools (`corpus`/`cdecl`/`reconcile_tu`/`canon_sig_reconcile`/`gen_harvest_targets`/`gate_stage`/
|
||||
`jtbl_family_bank`/`family_sweep`/`dedup_propagate`/`sig_unify`/`cast_call_sites`/`scope_data_externs`/
|
||||
`family_remap`) = **all clean** (`bank_exemplar` is a run-only script that indexes `sys.argv` at module
|
||||
scope → not importable by design, imports neither deleted module — a test-method false alarm, not
|
||||
breakage). No `src/`/`config/` change ⇒ no byte can move ⇒ full R22 not warranted (P9-honest: a 136-binary
|
||||
rebuild would prove only what the absence of a build-path edit already guarantees). **Doc-pointer hygiene**
|
||||
(so nothing points at a nonexistent tool): `docs/hand-matching-process.md` §8a, `docs/matching-cookbook.md`
|
||||
(canonical-sig-layer entry), and `docs/tooling-audit.md` (ledger row + the `derive_canonical_sigs` entry)
|
||||
all annotated DELETED/historical. **NEXT: A9e — wire `reconcile_tu` into `bank_exemplar`** (its stage
|
||||
ladder is `raw→scoped→recovered→reconciled`; `recovered`=`fb.recover`, `reconciled`=`canon_sig_reconcile`
|
||||
— add a `reconcile_tu` stage, byte-gate); then A9f (overlay_src_split force_decl latch), A9g (jr_inventory
|
||||
ephemeral read), A10 (wall re-test), A11 (distill + close → resume Phase 26 Task 7).
|
||||
- **2026-07-14 (session 12, A9a+A9b — canon_sig_reconcile fn-ptr fix + wall re-test; Max):** Continued the
|
||||
tool-hygiene audit. **A9a — `canon_sig_reconcile` sees fn-ptr dispatch tables (`commit:0609`).** The def-side-wall
|
||||
recovery tool, live on the ×134 economic-engine paths (jtbl_family_bank, family_sweep --reconcile-raw,
|
||||
|
||||
@@ -1,142 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Census the shared callees that block the parallel hand-matching wave (Phase 17 canonical-sig layer).
|
||||
|
||||
The calibration wave (hand-matching-process.md §7c) hit 60% match_one MATCH but only 33% whole-binary;
|
||||
the entire gap was SIG CONFLICTS — parallel agents each declare an undeclared-stub shared callee (e.g.
|
||||
func_80131CA8) with a different guessed signature, which then clash in the one-big-TU ov_SC01_077.c
|
||||
(`conflicting types for func_X`). 100% compile-errors, ZERO codegen mismatches.
|
||||
|
||||
A callee that is already `defined` (engine_core.h DEFINE / inline body) or already `declared` (an extern
|
||||
written somewhere) is conflict-FREE — gen_harvest_targets resolves it and every draft reuses the one sig.
|
||||
The conflict set is exactly the `stub` callees (no body, no extern anywhere) that >=2 still-stub wave
|
||||
targets call. Declaring ONE canonical extern for each (the canonical-sig layer) turns them `declared` ->
|
||||
the wave stops conflicting on them.
|
||||
|
||||
This is read-only. Output: the ranked conflict-callee table + the call-edge status totals.
|
||||
|
||||
Usage:
|
||||
tools/census_conflict_callees.py [--source ov_SC01_077] [--targets .run/harvest_targets_s3.json]
|
||||
"""
|
||||
import argparse, json, os, importlib.util
|
||||
from collections import defaultdict
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
import sys
|
||||
sys.path.insert(0, os.path.join(REPO, 'tools'))
|
||||
import corpus # the derived corpus oracle (Phase 26-A)
|
||||
|
||||
_spec = importlib.util.spec_from_file_location('ght', os.path.join(REPO, 'tools/gen_harvest_targets.py'))
|
||||
_ght = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_ght)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--source', default='ov_SC01_077')
|
||||
ap.add_argument('--targets', default='.run/harvest_targets_s3.json')
|
||||
ap.add_argument('--min-callers', type=int, default=2)
|
||||
ap.add_argument('--out', default='.run/conflict_callees.json')
|
||||
args = ap.parse_args()
|
||||
|
||||
src = args.source
|
||||
ec = os.path.join(REPO, 'src/shared/engine_core.h')
|
||||
|
||||
# ⚠ DEPRECATED — SCHEDULED FOR DELETION (Phase 26-A audit, R33).
|
||||
# This tool re-derives, by re-parsing C text, the question "for this TU, which callees are
|
||||
# defined / declared / stubbed / external?" — which is exactly and only what tools/reconcile_tu.py
|
||||
# answers FROM THE BUILD. A parse hole in reconcile_tu makes it CONSERVATIVE; a parse hole here
|
||||
# makes it WRONG IN THE UNSAFE DIRECTION (an unknown callee is silently bucketed "conflict-free").
|
||||
# Delete this file once reconcile_tu is wired into derive_canonical_sigs (its only consumer).
|
||||
#
|
||||
# Until then, at least stop the 96% under-report: the corpus was `src/<ov>/<ov>.c` ALONE, so it
|
||||
# saw 13 of ov_SC01_077's 264 stubs and printed "wave scope: 2 still-stub" when the truth is 57 —
|
||||
# every downstream percentage was computed against a denominator 96% too small. The 0-conflict
|
||||
# answer it gives today is right BY LUCK (those callees happen to be banked); point it at a fresh
|
||||
# overlay and it would see ~14 of ~600 stubs and confidently report zero conflicts into a wave
|
||||
# riddled with them — the exact failure that cost 60%-vs-33% in the Phase-17 calibration.
|
||||
c_paths = [str(p) for p in corpus.src_files(src)]
|
||||
c_path = c_paths[0] if c_paths else os.path.join(REPO, f'src/{src}/{src}.c')
|
||||
|
||||
src_sig = _ght.load_sig(os.path.join(REPO, f'.run/sig.{src}.jsonl')) # addr-int -> {calls,nins,h_exact,reach?}
|
||||
stubs = set(corpus.stubs(src)) # ADDRESSES (ints) — ALL TUs, ANY symbol name
|
||||
define_sigs = _ght.collect_define_sigs(ec)
|
||||
inline_sigs = {}
|
||||
for cp in c_paths:
|
||||
inline_sigs.update(_ght.collect_inline_sigs(cp))
|
||||
defined = {**define_sigs, **inline_sigs}
|
||||
extern_sigs = _ght.collect_extern_sigs([ec] + c_paths)
|
||||
|
||||
s3 = json.load(open(os.path.join(REPO, args.targets)))
|
||||
tgt_addrs = [int(t['addr'], 16) for t in s3]
|
||||
remaining = [a for a in tgt_addrs if a in stubs] # still-stub wave targets
|
||||
|
||||
def status(caddr):
|
||||
if caddr in defined: return 'defined'
|
||||
if caddr in extern_sigs: return 'declared'
|
||||
if caddr in stubs: return 'stub'
|
||||
return 'extern' # resident/EXE, conflict-free
|
||||
|
||||
# reach map (how many overlays hold a byte-identical copy) for prioritization
|
||||
reach = {int(t['addr'], 16): t.get('reach', 1) for t in s3}
|
||||
|
||||
callee_callers = defaultdict(set) # callee_addr -> {target addrs calling it (excl. self)}
|
||||
edge_status = defaultdict(int) # status -> # call edges (over remaining targets)
|
||||
for a in remaining:
|
||||
rec = src_sig.get(a)
|
||||
if not rec:
|
||||
continue
|
||||
for ch in set(rec.get('calls', [])):
|
||||
caddr = int(ch, 16)
|
||||
edge_status[status(caddr)] += 1
|
||||
if caddr != a:
|
||||
callee_callers[caddr].add(a)
|
||||
|
||||
# THE conflict predicate (hand-matching-process.md §7c): a sig conflict needs >=2 INDEPENDENT
|
||||
# declarations of an UNDECLARED stub in the one-big-TU build. Declaration sources for callee X =
|
||||
# one extern per drafted caller + one definition if X is itself a drafted target (def-vs-extern).
|
||||
# decl_sources = n_callers + (1 if X is a remaining target else 0)
|
||||
# An already-`declared`/`defined` callee is conflict-FREE (gen_harvest_targets feeds the one sig).
|
||||
conflicts = []
|
||||
for caddr, callers in callee_callers.items():
|
||||
if status(caddr) != 'stub':
|
||||
continue
|
||||
is_t = caddr in remaining
|
||||
decl_sources = len(callers) + (1 if is_t else 0)
|
||||
if decl_sources < 2:
|
||||
continue
|
||||
crec = src_sig.get(caddr, {})
|
||||
conflicts.append({
|
||||
'callee': f'func_{caddr:08X}', 'addr': f'{caddr:08X}',
|
||||
'n_callers': len(callers), 'is_target': is_t, 'decl_sources': decl_sources,
|
||||
'kind': 'match-first' if is_t else 'derive-declare',
|
||||
'callee_nins': crec.get('nins'), 'callee_ncalls': len(crec.get('calls', [])),
|
||||
'callee_reach': reach.get(caddr, crec.get('reach')),
|
||||
})
|
||||
conflicts.sort(key=lambda x: (-x['decl_sources'], -(x['callee_reach'] or 0)))
|
||||
|
||||
blocked = {a for a in remaining
|
||||
if any(int(ch, 16) in {int(c['addr'], 16) for c in conflicts}
|
||||
for ch in src_sig.get(a, {}).get('calls', []))}
|
||||
blk_reach = sum(reach.get(a, 1) for a in blocked)
|
||||
tot_reach = sum(reach.get(a, 1) for a in remaining)
|
||||
|
||||
json.dump(conflicts, open(os.path.join(REPO, args.out), 'w'), indent=1)
|
||||
|
||||
print(f'wave scope: {len(s3)} s3 targets, {len(remaining)} still-stub (remaining)')
|
||||
print('call-edge status totals (over remaining targets): '
|
||||
+ ' '.join(f'{k}:{edge_status[k]}' for k in ('defined', 'declared', 'stub', 'extern')))
|
||||
mf = sum(1 for c in conflicts if c['is_target'])
|
||||
print(f'\nCONFLICT CALLEES (undeclared stub, decl_sources>=2): {len(conflicts)} '
|
||||
f'[{mf} match-first / {len(conflicts)-mf} derive-declare]')
|
||||
print(f'{"callee":>16} {"callers":>7} {"isTgt":>5} {"srcs":>4} {"nins":>5} {"ncalls":>6} {"reach":>5}')
|
||||
for c in conflicts:
|
||||
print(f'{c["callee"]:>16} {c["n_callers"]:>7} {("Y" if c["is_target"] else ""):>5} '
|
||||
f'{c["decl_sources"]:>4} {str(c["callee_nins"] or "-"):>5} '
|
||||
f'{str(c["callee_ncalls"] or "-"):>6} {str(c["callee_reach"] or "-"):>5}')
|
||||
print(f'\nwave targets blocked by >=1 conflict callee: {len(blocked)}/{len(remaining)} '
|
||||
f'(reach-weighted {blk_reach}/{tot_reach} = {100*blk_reach//max(tot_reach,1)}% of wave reach)')
|
||||
print(f'wrote {args.out}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,148 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Derive a byte-neutral canonical signature for each Phase-17 conflict callee (the canonical-sig layer).
|
||||
|
||||
Input: .run/conflict_callees.json (from census_conflict_callees.py) — the undeclared-stub callees that
|
||||
parallel hand-matching agents would declare inconsistently (hand-matching-process.md §7c). For each we
|
||||
emit ONE canonical `extern s32 func_X(s32 a0, ...);` to seed src/shared/engine_core.h, so gen_harvest_targets
|
||||
feeds every drafting agent the SAME signature and the one-big-TU build stops conflicting.
|
||||
|
||||
Canonical form = WIDEST byte-neutral (hand-matching-process.md §3a):
|
||||
- return `s32` : void->s32 is byte-neutral (no explicit return => identical epilogue); s32 is REQUIRED
|
||||
where a caller uses $v0. So s32 is universally safe.
|
||||
- params `s32` : widest scalar; a matched body casts int->ptr (`*(T*)(a0+off)`, the demo idiom) and a
|
||||
caller narrows in the call expression. s32 never blocks a match; the byte-gate validates.
|
||||
- ARITY is the only value that must be exact (a wrong count => "too few/many arguments" at a caller, or a
|
||||
def/extern arity clash for a circular target). Derived two ways and cross-checked:
|
||||
(G) Ghidra-C cache .run/ghidra_c/func_<A>.c (FUN_<a>(...) param count) — the static oracle (G1).
|
||||
(A) asm read-before-write of $a0..$a3 in asm/<src>/nonmatchings/<src>/func_<A>.s — an a-reg whose
|
||||
FIRST-touching instruction uses it as a SOURCE is an incoming param; dest-only first touch
|
||||
(lui/lw/move/ALU-dest) = scratch, not a param. Arity = highest param index + 1 (contiguous).
|
||||
|
||||
The whole-binary harvest_verify byte-gate remains the sole arbiter (G3/P9): a wrong arity just fails the
|
||||
gate and is fixed per-callee. This only shapes the drafting / seeds the canonical decls.
|
||||
|
||||
Usage:
|
||||
tools/derive_canonical_sigs.py [--source ov_SC01_077] [--in .run/conflict_callees.json]
|
||||
"""
|
||||
import argparse, json, os, re, glob
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
AREGS = ['a0', 'a1', 'a2', 'a3']
|
||||
|
||||
# instruction operand roles restricted to what we need: is the FIRST a-reg touch a source (=> param)?
|
||||
LOAD = {'lw', 'lh', 'lhu', 'lb', 'lbu', 'lwl', 'lwr', 'll', 'lwc1', 'lwc2', 'ldc1', 'ldc2'} # rt=dest, base=src
|
||||
STORE = {'sw', 'sh', 'sb', 'swl', 'swr', 'sc', 'swc1', 'swc2', 'sdc1', 'sdc2'} # rt=src, base=src
|
||||
DEST_FIRST = {'lui', 'li', 'move', 'addu', 'addiu', 'subu', 'and', 'andi', 'or', 'ori', 'xor', # rd/rt=dest, rest=src
|
||||
'xori', 'nor', 'slt', 'sltu', 'slti', 'sltiu', 'sll', 'srl', 'sra', 'sllv', 'srlv',
|
||||
'srav', 'mul', 'mult', 'negu', 'neg', 'not', 'mflo', 'mfhi', 'movn', 'movz', 'sub',
|
||||
'add', 'rotr', 'clz', 'seb', 'seh', 'mfc1', 'mfc2', 'la'}
|
||||
SRC_ALL = {'beq', 'bne', 'beqz', 'bnez', 'blez', 'bgtz', 'bltz', 'bgez', 'bgezal', 'bltzal', # all regs are src
|
||||
'jr', 'jalr', 'multu', 'divu', 'div', 'mtlo', 'mthi', 'mtc1', 'mtc2', 'teq', 'tne',
|
||||
'beql', 'bnel', 'cache'}
|
||||
|
||||
|
||||
def reg_tokens(operand_str):
|
||||
return re.findall(r'\$([a-z0-9]+)', operand_str)
|
||||
|
||||
|
||||
def a_role(mnem, ops):
|
||||
"""return (a_sources, a_dest) restricted to a0..a3 for one instruction."""
|
||||
regs = reg_tokens(ops)
|
||||
a_in_order = [r for r in regs if r in AREGS]
|
||||
if not a_in_order:
|
||||
return set(), set()
|
||||
if mnem in LOAD:
|
||||
dest = {regs[0]} & set(AREGS) if regs else set()
|
||||
src = set(a_in_order) - dest
|
||||
elif mnem in STORE:
|
||||
dest, src = set(), set(a_in_order)
|
||||
elif mnem in DEST_FIRST:
|
||||
dest = {regs[0]} & set(AREGS) if regs else set()
|
||||
src = set(a_in_order) - dest
|
||||
elif mnem in SRC_ALL:
|
||||
dest, src = set(), set(a_in_order)
|
||||
else: # unknown: be conservative -> treat as sources (flags a param)
|
||||
dest, src = set(), set(a_in_order)
|
||||
return src, dest
|
||||
|
||||
|
||||
INSN_RE = re.compile(r'\*/\s+([a-z][a-z0-9.]*)\s+(.*)$') # after the `... XXXX */` comment
|
||||
|
||||
|
||||
def asm_arity(s_path):
|
||||
"""highest read-before-write a-reg index +1; returns (arity, note)."""
|
||||
if not os.path.exists(s_path):
|
||||
return None, 'no-asm'
|
||||
first = {} # areg -> 'param' | 'scratch'
|
||||
for line in open(s_path):
|
||||
m = INSN_RE.search(line)
|
||||
if not m:
|
||||
continue
|
||||
mnem, ops = m.group(1), m.group(2)
|
||||
src, dest = a_role(mnem, ops)
|
||||
for r in AREGS:
|
||||
if r in first:
|
||||
continue
|
||||
if r in src:
|
||||
first[r] = 'param'
|
||||
elif r in dest:
|
||||
first[r] = 'scratch'
|
||||
arity = 0
|
||||
for i, r in enumerate(AREGS):
|
||||
if first.get(r) == 'param':
|
||||
arity = i + 1
|
||||
# contiguity note: a param above a scratch/untouched gap (loose-typed) -> flag
|
||||
gap = any(first.get(AREGS[j]) != 'param' for j in range(arity - 1)) if arity else False
|
||||
return arity, ('gap' if gap else 'ok')
|
||||
|
||||
|
||||
GHIDRA_SIG_RE = re.compile(r'^\s*[A-Za-z_].*\bFUN_[0-9a-f]+\s*\((.*?)\)\s*$', re.M)
|
||||
|
||||
|
||||
def ghidra_arity(addr):
|
||||
p = os.path.join(REPO, f'.run/ghidra_c/func_{addr}.c')
|
||||
if not os.path.exists(p):
|
||||
return None
|
||||
m = GHIDRA_SIG_RE.search(open(p).read())
|
||||
if not m:
|
||||
return None
|
||||
params = m.group(1).strip()
|
||||
if params in ('', 'void'):
|
||||
return 0
|
||||
return len([x for x in params.split(',') if x.strip()])
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--source', default='ov_SC01_077')
|
||||
ap.add_argument('--in', dest='infile', default='.run/conflict_callees.json')
|
||||
args = ap.parse_args()
|
||||
asm_dir = os.path.join(REPO, f'asm/{args.source}/nonmatchings/{args.source}')
|
||||
conf = json.load(open(os.path.join(REPO, args.infile)))
|
||||
|
||||
print(f'{"callee":>16} {"kind":>14} {"ghidra":>6} {"asm":>4} {"note":>6} {"->arity":>7} canonical')
|
||||
rows = []
|
||||
for c in conf:
|
||||
addr = c['addr']
|
||||
g = ghidra_arity(addr)
|
||||
a, note = asm_arity(os.path.join(asm_dir, f'func_{addr}.s'))
|
||||
# reconcile: prefer asm (callee's own consumption); if asm gap or asm<ghidra, trust the larger
|
||||
cand = [x for x in (g, a) if x is not None]
|
||||
arity = max(cand) if cand else 0
|
||||
if g is not None and a is not None and g != a:
|
||||
note = f'G{g}/A{a}'
|
||||
params = 'void' if arity == 0 else ', '.join('s32 a%d' % i for i in range(arity))
|
||||
sig = f'extern s32 func_{addr}({params});'
|
||||
rows.append({'callee': c['callee'], 'addr': addr, 'arity': arity, 'kind': c['kind'],
|
||||
'ghidra': g, 'asm': a, 'note': note, 'sig': sig})
|
||||
print(f' func_{addr} {c["kind"]:>14} {str(g):>6} {str(a):>4} {note:>6} {arity:>7} {sig}')
|
||||
|
||||
out = os.path.join(REPO, '.run/canonical_sigs.json')
|
||||
json.dump(rows, open(out, 'w'), indent=1)
|
||||
print(f'\nwrote {out} ({len(rows)} canonical sigs)')
|
||||
print('disagreements (G!=A) to eyeball:',
|
||||
', '.join(r['callee'] for r in rows if r['note'].startswith('G')) or 'none')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user