mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-01 23:52:03 -04:00
feat(phase-29): fix_header_decl.py — automate the fresh-138 def-side header reconcile (§63)
Builds + self-tests the tool that reconciles a shared-header forward-decl to a function's byte-true signature — the integration half of the proven func_8014CD80 x138 pipeline. - tools/fix_header_decl.py: --fn --draft [--check|--apply]. Parses the byte-true def sig, canon-compares (typedef-aware int==s32, unsigned short*==u16* -> skip ALREADY-OK), REFUSES ABI-changing rewrites (param-count / ptr<->scalar / scalar class), preserves the macro `\` continuation, snapshots + prints the git-restore (§61: undo = restore, validate fleet-wide by R22). - 4 self-tests pass: (1) idempotent on the already-fixed func_8014CD80; (2) correct rewrite + `\` preserved; (3) REFUSE on a 2-vs-3-param ABI mismatch; (4) end-to-end --apply on func_8014D12C turned the `conflicting types` PLUMBING into a clean codegen DIFF (plumbing dissolved; its body is a near-miss so no bank), reverted clean. The self-test CAUGHT two fleet-corrupting bugs before any apply (a dropped macro `\`, and non-idempotent int/s32). - cookbook §63 + CURRENT_PHASE. NEXT: fresh-family wave over the 38-family market -> fix_header_decl --apply -> harvest_verify -> dedup_propagate -> R22.
This commit is contained in:
Drew T
@@ -4852,3 +4852,34 @@ fleet-shared-state restore trap:** `fix_arity_callers --funcs <fn in engine_core
|
||||
engine_core.h` fleet-wide; a `git checkout HEAD -- src/<ov>/` restore MISSES it (wrong directory), and the
|
||||
per-overlay build stays byte-identical so nothing flags the leak. Restore `src/shared/` too, and R22
|
||||
clean-fleet after any fix_arity probe (this bit me exactly as §61 warns — caught by a full `git status`).
|
||||
|
||||
## §63 — The fresh-138 DEF-SIDE blocker: fix the HEADER decl, not the draft (`fix_header_decl.py`, Phase 29 SESSION-13, 2026-07-23)
|
||||
|
||||
**The blocker.** A genuinely fresh reach-138 family (live in ~138 overlays, matched nowhere, no
|
||||
`DEFINE_func_*` macro) with a byte-perfect universal draft still fails the gate as
|
||||
`conflicting types for func_X` (cc1 exit 33). Cause: a shared caller macro in `src/shared/engine_core.h`
|
||||
forward-declares it with a SIMPLIFIED, caller-derived signature — `extern void func_X(s32 a0, void *a1,
|
||||
void *a2);` — that conflicts with the byte-true def `int func_X(s32, u16*, u16*)` the moment the def is
|
||||
spliced into an overlay TU that `#include`s the header. `gate_stage`'s §61 arity pre-pass is
|
||||
param-COUNT-only (misses return type + pointer element type); §54 `reconcile_def_sig` rewrites the DRAFT
|
||||
to match the header (wrong direction — the header is the simplified one, matching it can DCE the return).
|
||||
|
||||
**The fix — proven ×138 (func_8014CD80).** Rewrite the HEADER decl to the byte-true def. It is byte-neutral:
|
||||
(a) return `void`↔`intN` — the true fn always sets `$v0`; a caller that declared it `void` never read `$v0`
|
||||
(unchanged), and no caller USES the return; (b) pointer element type `void*`↔`T*` — register-passed
|
||||
regardless. One edit → `harvest_verify` banks ×1 BYTE-IDENTICAL → `dedup_propagate --addr` fills 138/138
|
||||
byte-identical → R22 140/140. Fleet +0.1pp instr off ONE family.
|
||||
|
||||
**`tools/fix_header_decl.py`** automates it: `--fn --draft [--check|--apply]`. Parses the byte-true def sig,
|
||||
canon-compares (typedef-aware: `int`≡`s32`, `unsigned short*`≡`u16*`) to skip ALREADY-OK decls, REFUSES any
|
||||
rewrite that changes param COUNT / flips pointer↔scalar / changes scalar class (ABI change — left to a
|
||||
human), and preserves the macro line's trailing `\` continuation (dropping it corrupts the DEFINE macro
|
||||
fleet-wide — a self-test caught this). Snapshots + prints the `git checkout` restore (§61: undo = restore,
|
||||
never inverse; validate FLEET-WIDE by R22, never the per-binary gate).
|
||||
|
||||
**Market (SESSION-13 scan):** of the 75 fresh (≥100-live) families, **46 carry an engine_core.h caller
|
||||
forward-decl, 38 SIMPLIFIED** = this pattern → each a candidate ×138 (≈+1.5–2.8pp instr). RESIDUAL RISK: the
|
||||
header is only the INTEGRATION half — each family still needs a byte-true UNIVERSAL body from a wave; a
|
||||
non-universal body (overlay-local `D_*` refs) is a genuine per-member wall regardless (`func_80165CA0`,
|
||||
0/135). Distinguish: h_exact=1 does NOT — the BODY's universality does. Pipeline: wave (`build_wave_args.py
|
||||
--rank live --min-live 100`) → `fix_header_decl --apply` → `harvest_verify` → `dedup_propagate --addr` → R22.
|
||||
|
||||
@@ -1548,12 +1548,17 @@ conditional) · main-EXE/B9 + GLM/B6 + resident's 14 walls (P30) · behemoths B7
|
||||
> **Quantified market: 38 of the 75 fresh (≥100-live) families carry the exact pattern** (engine_core.h caller
|
||||
> forward-decl, simplified `void`/`void*`) → each a candidate ×138 ≈ **+1.5–2.8pp instr**. → decision-log.
|
||||
>
|
||||
> **▶ THE SINGLE NEXT TASK (supersedes the options above for the FRESH pool):** build **`tools/fix_header_decl.py`**
|
||||
> — given a fn with a MATCH draft (or banked def), parse the byte-true sig, rewrite EVERY engine_core.h
|
||||
> forward-decl of that fn to match (byte-neutral: ptr-width-compatible + return-ignored callers), then hand to
|
||||
> the existing `harvest_verify → dedup_propagate --addr → R22` chain. It edits fleet-shared engine_core.h → MUST
|
||||
> use the §61 snapshot-undo (restore, never inverse) + validate FLEET-WIDE (R22, not the per-binary gate).
|
||||
> Then run a **fresh-family wave** (`build_wave_args.py --rank live --min-live 100`, needs agent budget — reset
|
||||
> ~1:40am Denver) over the 38, gate each through fix_header_decl. Expected: ~half-to-most bank ×138 (each
|
||||
> draft's BODY must be universal + byte-correct — that's the residual risk, not the header). This is option (b)
|
||||
> reopened as the campaign's best lever. **The onboarded-tail low-ROI read still stands; only the FRESH read flipped.**
|
||||
> **✅ `tools/fix_header_decl.py` BUILT + SELF-TESTED (SESSION-13, cookbook §63).** `--fn --draft
|
||||
> [--check|--apply]`: parses the byte-true sig, canon-compares (typedef-aware, skips ALREADY-OK), REFUSES
|
||||
> ABI-changing rewrites (param-count / ptr↔scalar / scalar-class), preserves the macro `\` continuation,
|
||||
> snapshots + prints the git-restore (§61). 4 self-tests pass: idempotent on func_8014CD80; correct rewrite
|
||||
> +`\` preserved; REFUSE on a 2-vs-3 param mismatch; **end-to-end `--apply` on func_8014D12C turned the
|
||||
> `conflicting types` PLUMBING into a clean codegen DIFF** (plumbing dissolved; func_8014D12C's body is a
|
||||
> near-miss so it didn't bank — the tool is the INTEGRATION half, reverted clean).
|
||||
>
|
||||
> **▶ THE SINGLE NEXT TASK (needs agent budget — wave reset ~1:40am Denver):** run the fresh-family pipeline
|
||||
> over the 38-family market. `build_wave_args.py --rank live --min-live 100 --binary ov_SC07_006` → wave the
|
||||
> fresh families (byte-true UNIVERSAL bodies) → `fix_header_decl --apply` per fn → `harvest_verify` →
|
||||
> `dedup_propagate --addr` → R22. Expect ~half-to-most bank ×138 (residual risk = a non-universal body, a
|
||||
> per-member wall like func_80165CA0, NOT the header). Option (b) reopened as the campaign's best lever
|
||||
> (≈+1.5–2.8pp instr). **The onboarded-tail low-ROI read still stands; only the FRESH read flipped.**
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env python3
|
||||
"""fix_header_decl.py — reconcile shared-header forward-decls to a fn's BYTE-TRUE signature.
|
||||
|
||||
THE FRESH-138 DEF-SIDE BLOCKER (Phase-29 SESSION-13, cookbook §63). A shared caller macro in
|
||||
`src/shared/engine_core.h` forward-declares an engine fn with a SIMPLIFIED, caller-derived signature
|
||||
(`extern void func_X(s32 a0, void *a1, void *a2);`) that conflicts with the fn's BYTE-TRUE definition
|
||||
(`int func_X(s32, u16*, u16*)`). When a wave drafts that fn, splicing its true def into an overlay TU
|
||||
that `#include`s the header throws `conflicting types for func_X` (cc1 exit 33) — so the fn can NEVER
|
||||
bank, even with a byte-perfect body. `gate_stage`'s arity pre-pass is param-COUNT-only (misses return
|
||||
type + pointer element type); §54 `reconcile_def_sig` fixes the DRAFT to the header (wrong direction —
|
||||
the header is the simplified one). This tool fixes the HEADER to the byte-true def.
|
||||
|
||||
PROVEN (func_8014CD80, SESSION-13): one byte-neutral header edit -> harvest_verify banks x1
|
||||
BYTE-IDENTICAL -> `dedup_propagate --addr` fills 138/138 overlays byte-identical -> R22 140/140.
|
||||
|
||||
BYTE-NEUTRALITY (why rewriting the decl cannot change any shipped byte):
|
||||
- return void <-> intN: the true fn always sets $v0; a caller that declared it `void` simply never
|
||||
read $v0 (unchanged), and no current caller USES the return (it was declared void). Widening the
|
||||
decl to the true return is a pure type-check change.
|
||||
- pointer element type void* <-> T*: pointers are register-passed regardless of element type.
|
||||
REFUSED (would change the ABI, left to a human / permuter): a param-COUNT change, or a param that
|
||||
flips pointer<->scalar, or a scalar-CLASS change (int<->float — different register file).
|
||||
The whole-binary byte-gate (harvest_verify / dedup_propagate / R22) is the FINAL arbiter regardless;
|
||||
this tool only rewrites the decl, and never trusts its own byte-neutrality judgement.
|
||||
|
||||
FLEET-SHARED SAFETY (§61): editing engine_core.h touches all 138 overlays. --apply snapshots every
|
||||
touched file first and prints the exact `git checkout` restore; the undo is ALWAYS a restore, never an
|
||||
inverse transform, and correctness is validated FLEET-WIDE by R22, never by the per-binary gate that
|
||||
authorised the bank (the SESSION-12 incident).
|
||||
|
||||
USAGE
|
||||
# dry-run: show the rewrite the tool would make
|
||||
fix_header_decl.py --fn func_8014CD80 --draft .run/drafts-X/func_8014CD80.c --check
|
||||
# apply the header rewrite (snapshots first), then bank+propagate yourself:
|
||||
fix_header_decl.py --fn func_8014D12C --draft .run/drafts-X/func_8014D12C.c --apply
|
||||
# -> python tools/harvest_verify.py --binary <ov> --drafts <draftdir>
|
||||
# -> python tools/dedup_propagate.py --addr 0x<addr> --recover
|
||||
# -> make clean && make extract-all && make check-all (R22, MANDATORY — fleet-shared edit)
|
||||
"""
|
||||
import argparse, os, re, sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
HEADERS = ["src/shared/engine_core.h", "src/shared/engine_types.h"]
|
||||
|
||||
# canonical project spellings (byte-identical typedefs; used for the rewritten decl's style)
|
||||
CANON = {
|
||||
"int": "s32", "unsigned int": "u32", "unsigned": "u32",
|
||||
"short": "s16", "unsigned short": "u16",
|
||||
"char": "s8", "unsigned char": "u8", "signed char": "s8",
|
||||
"long": "s32", "unsigned long": "u32",
|
||||
}
|
||||
INT_CLASS = {"s8", "u8", "s16", "u16", "s32", "u32", "int", "unsigned", "short", "long", "char",
|
||||
"unsigned int", "unsigned short", "unsigned char", "signed char", "unsigned long"}
|
||||
|
||||
|
||||
def canon_base(base):
|
||||
base = " ".join(base.split())
|
||||
return CANON.get(base, base)
|
||||
|
||||
|
||||
def parse_param(p):
|
||||
"""'unsigned short *param_2' -> (base='unsigned short', is_ptr=True, name='param_2')."""
|
||||
p = " ".join(p.replace("*", " * ").split())
|
||||
if p in ("", "void"):
|
||||
return None
|
||||
is_ptr = "*" in p
|
||||
toks = [t for t in p.split() if t != "*"]
|
||||
TYPEWORDS = {"unsigned", "signed", "short", "long", "int", "char", "void", "float", "double",
|
||||
"u8", "u16", "u32", "s8", "s16", "s32", "f32", "const", "struct", "void"}
|
||||
name = ""
|
||||
if toks and toks[-1] not in TYPEWORDS:
|
||||
name = toks[-1]
|
||||
toks = toks[:-1]
|
||||
return " ".join(toks), is_ptr, name
|
||||
|
||||
|
||||
def parse_sig(sig_body):
|
||||
"""sig_body = the text between ( and ). Returns list of (base, is_ptr, name)."""
|
||||
sig_body = sig_body.strip()
|
||||
if sig_body in ("", "void"):
|
||||
return []
|
||||
return [parse_param(p) for p in sig_body.split(",")]
|
||||
|
||||
|
||||
def byte_true_sig(draft_path, fn):
|
||||
"""Parse the fn's DEFINITION (has a { body) from the draft. Returns (ret, [params])."""
|
||||
text = open(draft_path).read()
|
||||
m = re.search(rf"(?:^|\n)[ \t]*([A-Za-z_][\w \t\*]*?)\b{re.escape(fn)}[ \t]*\(([^;{{]*?)\)[ \t\n]*\{{",
|
||||
text)
|
||||
if not m:
|
||||
return None
|
||||
return m.group(1).strip(), parse_sig(m.group(2))
|
||||
|
||||
|
||||
DECL_RE_T = r"(extern\s+)([A-Za-z_][\w \t\*]*?)\b{fn}[ \t]*\(([^;]*?)\)([ \t]*;)"
|
||||
|
||||
|
||||
def find_decls(fn):
|
||||
"""Return [(hdr_path, lineno, full_line, ret, [params])] for every forward-decl of fn."""
|
||||
out = []
|
||||
rx = re.compile(DECL_RE_T.format(fn=re.escape(fn)))
|
||||
for h in HEADERS:
|
||||
path = os.path.join(REPO, h)
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
for i, line in enumerate(open(path), 1):
|
||||
m = rx.search(line)
|
||||
if m:
|
||||
out.append((h, i, line, m.group(2).strip(), parse_sig(m.group(3)), m))
|
||||
return out
|
||||
|
||||
|
||||
def compatible(true_ret, true_params, decl_ret, decl_params):
|
||||
"""Byte-neutral iff same param count, ptr-ness matches per param, and scalar params stay in the
|
||||
integer class (no int<->float flip). Return (ok, reason)."""
|
||||
if len(true_params) != len(decl_params):
|
||||
return False, f"param COUNT differs (true {len(true_params)} vs decl {len(decl_params)}) — ABI change, REFUSED"
|
||||
for i, (tp, dp) in enumerate(zip(true_params, decl_params)):
|
||||
if tp is None or dp is None:
|
||||
return False, f"param {i} unpar4seable"
|
||||
tb, tptr, _ = tp
|
||||
db, dptr, _ = dp
|
||||
if tptr != dptr:
|
||||
return False, f"param {i} flips pointer<->scalar ({db}{'*' if dptr else ''} -> {tb}{'*' if tptr else ''}) — ABI change, REFUSED"
|
||||
if not tptr: # scalar: both must be integer-class
|
||||
if canon_base(tb) not in INT_CLASS or canon_base(db) not in INT_CLASS:
|
||||
if canon_base(tb) != canon_base(db):
|
||||
return False, f"param {i} scalar class change ({db} -> {tb}) — REFUSED"
|
||||
return True, "ok"
|
||||
|
||||
|
||||
def canon_param_str(p):
|
||||
if p is None:
|
||||
return "void"
|
||||
base, is_ptr, _ = p
|
||||
return canon_base(base) + ("*" if is_ptr else "")
|
||||
|
||||
|
||||
def canon_sig(ret, params):
|
||||
"""Typedef-normalized (ret, param-string) — the C-type identity, so int==s32, unsigned short*==u16*.
|
||||
Two decls with the same canon_sig do NOT `conflicting types`; different ones DO."""
|
||||
r = "void" if ret.strip() == "void" else canon_base(ret)
|
||||
ps = ",".join(canon_param_str(p) for p in params) if params else "void"
|
||||
return r, ps
|
||||
|
||||
|
||||
def build_decl(m, true_ret, true_params):
|
||||
"""Return just the rewritten `extern <ret> func_X(...);` span (the caller splices it back into the
|
||||
line, preserving leading whitespace + any trailing macro `\\` continuation)."""
|
||||
extern = m.group(1)
|
||||
decl_params = parse_sig(m.group(3))
|
||||
ret = canon_base(true_ret) if true_ret.strip() != "void" else "void"
|
||||
parts = []
|
||||
for i, tp in enumerate(true_params):
|
||||
tb, tptr, _ = tp
|
||||
name = decl_params[i][2] if i < len(decl_params) and decl_params[i] and decl_params[i][2] else ""
|
||||
base = canon_base(tb) if tb else "void"
|
||||
parts.append((f"{base} *{name}" if tptr else f"{base} {name}").rstrip())
|
||||
inner = ", ".join(parts) if parts else "void"
|
||||
return f"{extern}{ret} func_{FN_ADDR}({inner});"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--fn", required=True, help="func_XXXXXXXX")
|
||||
ap.add_argument("--draft", required=True, help="draft .c holding the byte-true DEFINITION")
|
||||
ap.add_argument("--check", action="store_true", help="dry-run: show the rewrite, touch nothing")
|
||||
ap.add_argument("--apply", action="store_true", help="snapshot + rewrite the header decls")
|
||||
args = ap.parse_args()
|
||||
global FN_ADDR
|
||||
FN_ADDR = args.fn.replace("func_", "")
|
||||
|
||||
os.chdir(REPO)
|
||||
true = byte_true_sig(args.draft, args.fn)
|
||||
if not true:
|
||||
print(f"[fix_header_decl] FATAL: no byte-true DEFINITION of {args.fn} in {args.draft}", file=sys.stderr)
|
||||
return 2
|
||||
true_ret, true_params = true
|
||||
pstr = ", ".join(f"{b}{'*' if p else ''}" for (b, p, _) in true_params) or "void"
|
||||
print(f"byte-true def: {true_ret} {args.fn}({pstr})")
|
||||
|
||||
decls = find_decls(args.fn)
|
||||
if not decls:
|
||||
print(f"[fix_header_decl] no shared-header forward-decl of {args.fn} — nothing to reconcile "
|
||||
f"(the blocker is elsewhere: callee/data/struct/DIFF).")
|
||||
return 1
|
||||
changes = []
|
||||
for h, ln, line, dret, dparams, m in decls:
|
||||
already = (canon_sig(true_ret, true_params) == canon_sig(dret, dparams))
|
||||
ok, why = compatible(true_ret, true_params, dret, dparams)
|
||||
# splice the rewritten decl span back into the line, preserving leading ws + trailing `\`
|
||||
new_line = line[:m.start()] + build_decl(m, true_ret, true_params) + line[m.end():]
|
||||
status = "ALREADY-OK" if already else ("SAFE" if ok else "REFUSE")
|
||||
print(f" {h}:{ln} [{status}]")
|
||||
print(f" - {line.rstrip()}")
|
||||
if already:
|
||||
continue
|
||||
if not ok:
|
||||
print(f" ! {why}")
|
||||
continue
|
||||
print(f" + {new_line.rstrip()}")
|
||||
changes.append((h, line, new_line))
|
||||
|
||||
if not changes:
|
||||
print("no change needed (decls already byte-true, or all REFUSED).")
|
||||
return 0
|
||||
if args.check:
|
||||
print(f"[--check] {len(changes)} decl(s) WOULD be rewritten. Re-run with --apply.")
|
||||
return 0
|
||||
if not args.apply:
|
||||
print("pass --apply to write (or --check to dry-run).", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
# §61: snapshot every touched header before writing; undo is a RESTORE (git checkout), never inverse.
|
||||
touched = sorted({h for h, _, _ in changes})
|
||||
for h in touched:
|
||||
path = os.path.join(REPO, h)
|
||||
text = open(path).read()
|
||||
for _, old, new in [c for c in changes if c[0] == h]:
|
||||
assert text.count(old) >= 1, f"decl line vanished in {h}"
|
||||
text = text.replace(old, new, 1)
|
||||
open(path, "w").write(text)
|
||||
print(f"[--apply] rewrote {len(changes)} decl(s) in {len(touched)} header(s).")
|
||||
print(f" UNDO (if the bank fails / R22 is not 140/140): git checkout -- {' '.join(touched)}")
|
||||
print(f" NEXT: harvest_verify --binary <ov> --drafts <dir> ; dedup_propagate --addr 0x{FN_ADDR.lower()} --recover ; R22 (make clean && extract-all && check-all)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user