From 1ea54fa594aeee0c359db59f7d4c129b23d6ee6e Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Fri, 24 Jul 2026 14:01:57 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-29):=20T16.1=20=E2=80=94=20the=20two?= =?UTF-8?q?-oracle=20blocker=20probe=20(read-only);=20rtu=5Fmatch=20full-s?= =?UTF-8?q?tderr=20+=20multi-line=20//@EDIT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 16 (the integration-recovery pass) T1. The SESSION-15 audit measured the wave bottleneck as INTEGRATION (~92% of drafts byte-correct, ~27% bank); 36 stranded byte-correct reach-138 drafts are the fuel. Before building any recovery, measure the REAL blocker per draft. - tools/rtu_match.py: --stderr-out (atexit flush, covers every sys.exit path) + \n in //@EDIT replacements (T6 needs a multi-line macro expansion). The inline tail is truncated and these TUs emit hundreds of benign warnings -- on the first real run it was 100% warnings while the actual errors sat ~180 lines earlier (the §58 red-herring, one level down). - tools/blocker_probe.py (NEW, read-only, two oracles R34): static (cdecl.compatible -- never text equality, which is what made the deleted scanner report u8-vs-unsigned-char as a conflict) and real cc1 via rtu_match (ONE compile implementation, R33). Leads with the DISAGREEMENT table. - DELETED .run/diag_plumbing.py (R3 tooling under tools/; R33 net -1 scanner). - Two build-forced corrections: (1) cdecl.tu_scope runs real cpp, so it already expands instantiated DEFINE_func_* macros -- the macro scan's job is ATTRIBUTION (tu-text vs shared-header macro body: different transforms, same T1 tier), byte-checked against engine_core.h:7533 for func_80161374; (2) blockers STACK and cc1 reveals only the first, so a function's tier is the MAX over blockers. - Smoke test 3 fns: oracles agree 3/3. src/ untouched (write-set asserted). Not yet population evidence -- that is T2 / KILL GATE A. --- phase-ends/CURRENT_PHASE.md | 51 ++++++ tools/blocker_probe.py | 301 ++++++++++++++++++++++++++++++++++++ tools/rtu_match.py | 53 ++++++- 3 files changed, 398 insertions(+), 7 deletions(-) create mode 100644 tools/blocker_probe.py diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index b75940a44..affaf5a33 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -178,6 +178,22 @@ arithmetic scales (R14/R35).** Every prior "structural wall" (B2, SC07, pin-cras backlog drafts via grinder/`permuter_ils`. - [ ] **Task 7 — Burn-down tracker + velocity/3-metric report + ROI-gated close [Max]** — close when per-session yield across both tracks floors out; Roadmap delta; hand P30 an honest frontier. +- [~] **Task 16 — THE INTEGRATION-RECOVERY PASS (measurement-first, blast-radius-typed)** + SESSION-16, plan approved 2026-07-24 (plan mirror + `~/.claude/plans/max-effort-set-plan-reactive-gem.md`). The SESSION-15 audit measured the wave + bottleneck as INTEGRATION (~92% of drafts byte-correct, ~27% bank) — 36 stranded byte-correct + reach-138 drafts sit in `.run/drafts-s15` (18) + `.run/drafts-s14r` (18), all `ov_SC07_006`. + Sub-tasks: **T1** instrument the probe ✅ · **T2** S0 measure all 36 [KILL GATE A] · + **T3** reproduce the ladder as a baseline · **T4** extend `recover_integration.py` into the + tiered driver · **T5** run the T0/T1 stages [KILL GATE B] · **T6** class-1 de-macroize probe + [KILL GATE E] · **T7** bank/propagate/R22/distill. + **The design spine — a BLAST-RADIUS TAXONOMY that makes §61 structural, not remembered:** + T0 draft-only → per-binary gate · T1 binary-local (`src//**`) → per-binary gate is + SUFFICIENT (the write set cannot reach another binary) · T2 fleet-shared (`src/shared/**`) → + **R22 mandatory** (§63 UPDATE: the per-binary gate is necessary-not-sufficient). Every stage + declares its tier; the driver MEASURES the write set and asserts containment. + Drew's Phase-Start decisions: probe-then-continue with kill gates; class-1 de-macroize IN + scope as a bounded ×1 probe (the ×138 variant is OUT of scope). **OUT of scope:** parallel gate farm (bulk_harvest already is it) · family-adapt fine-tune (bounded, conditional) · main-EXE/B9 + GLM/B6 + resident's 14 walls (P30) · behemoths B7 + walls B8 (P31). @@ -2082,3 +2098,38 @@ conditional) · main-EXE/B9 + GLM/B6 + resident's 14 walls (P30) · behemoths B7 > **CARRIED:** the 16 s15 nears + 18 s14 nears (byte-correct, integration-blocked) in the wave dirs — the > recovery tool's first fuel. The 2 s15 genuine nears (func_80175820/func_801758FC, phantom-frame) → permuter. > **DO NOT close P29 on ROI** — burn-down floor undetermined. + +- **✅ 2026-07-24 (SESSION-16, Max) — Task 16 / T1: the two-oracle blocker probe BUILT. Read-only; `src/` untouched.** + **`tools/rtu_match.py` (2 additions).** `--stderr-out PATH` persists EVERY stage's stderr via an + `atexit` flush (so all `sys.exit` paths are covered); `//@EDIT` replacements now accept `\n` + (multi-line), which T6's macro expansion needs. **The `--stderr-out` change earned itself on its + first run:** the tail rtu_match prints inline for `func_80161374` is **100% warnings**, while the + real errors sit ~180 lines earlier. Nobody diagnosing from that tail could have seen the cause — + §58's red-herring one level down, and the reason every prior per-function verdict for these + drafts was inferred rather than read. + **`tools/blocker_probe.py` (NEW, read-only, two oracles — R34).** *Static:* `cdecl.parse` + + **`cdecl.compatible`** (never text equality — `.run/diag_plumbing.py` compared decl TEXT, so + `extern u8 D_X;` vs `extern unsigned char D_X;` read as a conflict though `common.h` makes them + the same type; that is very likely the whole "~10 data-extern co-blockers" figure). *Real cc1:* + shells out to `rtu_match` (ONE implementation of the compile, R33) → `CC1-FAIL:` / `MATCH` / `DIFF`. Output leads with the DISAGREEMENT table, not the agreement rate. + **DELETED `.run/diag_plumbing.py`** (R3 tooling belongs in `tools/`; R33 — net −1 scanner). + **Two corrections the build forced (both would have mis-routed the recovery):** + **(1) `cdecl.tu_scope` runs REAL cpp** (`tu_statements`, the §8c law) — so it ALREADY expands an + instantiated `DEFINE_func_*` macro body and is authoritative about whether a conflict exists. My + first cut added a redundant macro scanner for DETECTION. Its real job is **ATTRIBUTION**: a decl + in the TU's own text is rewritten in place (`normalize_self_decls`), one inside an instantiated + shared-header macro is escaped by de-macroizing that single instantiation (§63's route). Both are + T1 — but they are different edits, and the first version mis-labelled the second as the first. + Byte-checked against the tree: `func_80161374` has NO literal decl in its TU (only the + `INCLUDE_ASM` at :3861 and `DEFINE_func_80161278()` at :3853); the conflicting + `extern void func_80161374(void *a0, s32 a1);` is `engine_core.h:7533`, inside the macro body. + The probe now reports exactly that. + **(2) BLOCKERS STACK, and cc1 only ever reveals the FIRST.** All 3 smoke-test functions carry + 2–5 independent blockers (e.g. `func_80161374`: 4 callee_decl + 1 self_decl_hdr). So the tier a + function needs is the **MAX** over its blockers, not the first one's — and the static oracle's + COMPLETE list is the routing signal, with cc1 confirming only "yes, it is a decl conflict". + **Smoke test (3 fns, `-v`): both oracles AGREE 3/3, 0 static-only, 0 cc1-only.** Not yet evidence + about the population — that is T2. + + diff --git a/tools/blocker_probe.py b/tools/blocker_probe.py new file mode 100644 index 000000000..c225867e6 --- /dev/null +++ b/tools/blocker_probe.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +"""blocker_probe.py — WHY does a byte-correct draft fail the whole-binary gate? Read-only. + +A crack wave produces drafts that `match_one` calls MATCH but the whole-binary gate rejects +(~92% draft-correct, ~27% bank — the SESSION-15 efficiency audit). Those are byte-correct +bodies stranded by a DECLARATION conflict, not wrong instructions. This says which one, per +function, and it does so with TWO INDEPENDENT ORACLES because a single oracle that is +structurally blind to a class of error cannot be fixed by making it stricter (R34): + + ORACLE A — static. Parses the draft and everything the TU declares (`cdecl`, the + coverage-asserting C-declaration parser) and asks cc1's OWN compatibility question via + `cdecl.compatible`. It is fast, it names the exact pair of declarations, and it is the + only oracle that can attribute a conflict to a SOURCE (the TU's own text vs a + `DEFINE_func_*` macro body in the shared header) — which is what decides the blast + radius of any fix. It CANNOT see: anything order-dependent that cc1 resolves + differently, and any error that is not a declaration conflict. + ORACLE B — the real cc1. Shells out to `tools/rtu_match.py`, which compiles the WHOLE split + TU with the draft spliced and `INCLUDE_ASM` neutralised. It is the ground truth for + "does this compile", and it additionally reports MATCH/DIFF for the spliced body in its + REAL ambient context. It CANNOT see: link-time errors (undefined reference — the §58a + Ghidra-name class), and it is relocation-masked, so it can over-claim MATCH. + +Neither is the arbiter. The whole-binary byte-gate is (G3/P9). What this tool produces is the +CEILING and the routing: which recovery transform each stranded draft needs, and — from the +disagreement column — how much the static classifier can be trusted at all. A prior static +classifier (`.run/diag_plumbing.py`, deleted by this tool) compared declaration TEXT, so +`extern u8 D_X;` vs `extern unsigned char D_X;` read as a conflict though `common.h` makes them +the same type and cc1 accepts both silently. Text equality is never the question; `compatible` +is. + + tools/blocker_probe.py --binary ov_SC07_006 --drafts .run/drafts-s15,.run/drafts-s14r + tools/blocker_probe.py --binary ov_SC07_006 --drafts .run/drafts-s15 --fns func_80161374 -v + +Writes NOTHING outside --work (and --json). Never touches src/. +""" +import argparse, concurrent.futures, json, os, re, subprocess, sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.dirname(HERE) +sys.path.insert(0, HERE) +import cdecl +import corpus + +ENGINE_CORE = os.path.join(REPO, 'src/shared/engine_core.h') +DRAFT_RE = re.compile(r'^func_[0-9A-Fa-f]{8}\.c$') # wave dirs also hold scratch: _b.c, try2.c … +PY = os.path.join(REPO, '.venv/bin/python') + +# A blocker class -> the recovery tier it implies. This is the routing the pass exists to produce. +# T0 draft-only : rewrite the draft; per-binary gate is sufficient +# T1 binary-local : edit src//**; per-binary gate is sufficient (cannot reach another binary) +# T2 fleet-shared : edit src/shared/**; R22 clean-fleet is MANDATORY (§61, §63 UPDATE) +TIER = { + 'callee_decl': 'T0', # cast_call_sites / reconcile_tu — rewrite the draft's own externs + 'data_decl': 'T0', + 'local_type': 'T0', # rename/drop the draft's duplicate type (the LIFT is only for propagation) + 'self_decl_tu': 'T1', # normalize_self_decls on the overlay's own TU + 'self_decl_hdr': 'T1', # de-macroize the one instantiation in the overlay's own TU (§63 escape) + 'parse': '--', +} + + +# --------------------------------------------------------------------------------------------- +# ORACLE A — static +# --------------------------------------------------------------------------------------------- +def macro_scope(tu_text, header_path=ENGINE_CORE): + """{name: (Declarator, macro)} contributed by the `DEFINE_func_*` macros this TU instantiates. + + ATTRIBUTION ONLY — not detection. `cdecl.tu_scope` runs real cpp (the §8c law), so it already + expands these instantiations and is authoritative about whether a conflict EXISTS. What it + cannot tell you is WHERE the conflicting declaration lives, and that decides which transform + applies: a declaration in the TU's own text is rewritten in place (normalize_self_decls), + whereas one inside an instantiated shared-header macro body is escaped by de-macroizing that + single instantiation (§63's unexplored "per-overlay-local decl"). Both are binary-local; they + are not the same edit. Mis-attributing one as the other routes the fix at the wrong file.""" + inst = set(re.findall(r'\b(DEFINE_func_[0-9A-Fa-f]+)\s*\(', tu_text)) + if not inst: + return {} + text = open(header_path, errors='replace').read() + out = {} + for m in re.finditer(r'^#define\s+(DEFINE_func_[0-9A-Fa-f]+)\s*\(', text, re.M): + name = m.group(1) + if name not in inst: + continue + i, body = m.end(), [] # gather the \-continued logical line + while i < len(text): + nl = text.find('\n', i) + if nl < 0: + body.append(text[i:]); break + line = text[i:nl] + body.append(line.rstrip('\\')) + if not line.rstrip().endswith('\\'): + break + i = nl + 1 + for stmt in re.findall(r'\bextern\b[^;{}]*;', '\n'.join(body)): + try: + for d in cdecl.parse(stmt): + out.setdefault(d.name, (d, name)) + except cdecl.CDeclError: + continue # a macro body line cdecl cannot read + return out + + +def draft_types(text): + """Type names the draft DEFINES (typedef aliases + struct/union tags).""" + names = set(re.findall(r'\btypedef\b[^;]*?\b(\w+)\s*(?:\[[^\]]*\])?\s*;', text, re.S)) + names |= set(re.findall(r'\b(?:struct|union)\s+(\w+)\s*\{', text)) + return {n for n in names if n} + + +def static_verdict(fn, stub, draft_text): + """[(klass, detail)] — every declaration in the draft that cc1 would refuse, named.""" + tu_path = os.path.join(REPO, stub.path) + try: + ddecls = cdecl.declarations(cdecl.split_statements(draft_text), path='') + except cdecl.CDeclError as e: + return [('parse', 'draft: %s' % str(e).split('\n')[0][:110])] + try: + tu = cdecl.tu_scope(tu_path) # full scope: a decl BELOW conflicts too + except cdecl.CDeclError as e: + return [('parse', 'tu: %s' % str(e).split('\n')[0][:110])] + + tu_text = open(tu_path, errors='replace').read() + macro = macro_scope(tu_text) + + def whence(name): + """'tu' | 'hdr:' — where the conflicting declaration physically lives. A literal + declaration in the TU's own text wins: it is the one an in-place rewrite can reach.""" + if re.search(r'^[^/\n]*\b%s\b[^;\n]*;' % re.escape(name), tu_text, re.M): + return 'tu' + return 'hdr:' + macro[name][1] if name in macro else 'tu' + + out = [] + for d in ddecls: + other = tu.get(d.name) + if other is None or (cdecl.compatible(other, d) and cdecl.compatible(d, other)): + continue + src = whence(d.name) + if d.is_definition and d.name == fn: + out.append(('self_decl_hdr' if src.startswith('hdr') else 'self_decl_tu', + '%s declares %s | def is %s' % (src, other.type, d.type))) + else: + klass = 'callee_decl' if d.kind == 'func' else 'data_decl' + out.append((klass, '%s: %s declares %s | draft %s' % (d.name, src, other.type, d.type))) + + dup = draft_types(draft_text) & (set(cdecl.typedef_names(tu_path)) + | set(re.findall(r'\b(?:struct|union)\s+(\w+)\s*\{', tu_text))) + for t in sorted(dup): + out.append(('local_type', 'draft redefines type %r the TU already defines' % t)) + return out + + +# --------------------------------------------------------------------------------------------- +# ORACLE B — the real cc1, over the whole TU +# --------------------------------------------------------------------------------------------- +_WARN = re.compile(r'\bwarning:') +_ERR = re.compile(r'\berror\b|conflicting types|redefinition of|redeclar|parse error|' + r'storage size|undefined|has no type or storage class', re.I) + + +def first_error(errfile): + """The FIRST non-warning error line. These TUs emit hundreds of benign `type mismatch with + previous external decl' warnings, so the tail rtu_match prints inline is usually 100% warnings + and the real cause is invisible in it (§58's red-herring, one level down).""" + try: + lines = open(errfile, errors='replace').read().splitlines() + except OSError: + return '' + for ln in lines: + if not _WARN.search(ln) and _ERR.search(ln): + return re.sub(r'^\S*t\.c:', 'tu:', ln.strip())[:120] + return '' + + +def cc1_verdict(fn, stub, draft_path, work): + """('MATCH'|'DIFF'|'CC1-FAIL'|'ERR', detail) from the real whole-TU compile.""" + split = os.path.splitext(os.path.basename(stub.path))[0] + wd = os.path.join(work, fn) + err = os.path.join(wd, 'err.txt') + cmd = [PY, os.path.join(HERE, 'rtu_match.py'), fn, + '--split', split, '--source', stub.path.split('/')[1], + '--c', draft_path, '--work', wd, '--stderr-out', err, '--maxdiff', '4'] + if corpus.is_o0(stub.path): + cmd.append('--o0') + p = subprocess.run(cmd, capture_output=True, cwd=REPO, text=True) + o = p.stdout + if 'MATCH (' in o: + return 'MATCH', re.search(r'MATCH \((\d+) ins\)', o).group(1) + ' ins' + if o.startswith('DIFF'): + m = re.search(r'mine=(\d+) ins, target=(\d+) ins, (\d+) mismatched', o) + return 'DIFF', ('%s/%s ins, %s mismatched' % (m.group(1), m.group(2), m.group(3)) if m else '') + if 'CC1 FAIL' in o: + return 'CC1-FAIL', first_error(err) or o.splitlines()[-1][:120] + return 'ERR', (o.strip().splitlines() or [''])[0][:120] + + +# --------------------------------------------------------------------------------------------- +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--binary', required=True) + ap.add_argument('--drafts', required=True, help='comma-separated wave dirs') + ap.add_argument('--fns', default=None, help='comma-separated subset') + ap.add_argument('--work', default='.run/recover/probe') + ap.add_argument('--jobs', type=int, default=8) + ap.add_argument('--json', default=None) + ap.add_argument('-v', '--verbose', action='store_true', help='every blocker, not just the first') + a = ap.parse_args() + + stubs = {s.symbol: s for s in corpus.stubs(a.binary).values()} # R33: derived, never a hand map + want = set(a.fns.split(',')) if a.fns else None + + drafts, dup = {}, [] + for d in a.drafts.split(','): + for b in sorted(os.listdir(os.path.join(REPO, d))): + if not DRAFT_RE.match(b): + continue + fn = b[:-2] + if want and fn not in want: + continue + if fn in drafts: + dup.append(fn) + drafts[fn] = os.path.join(REPO, d, b) + + # R32: name what is excluded and why. A stranded draft that quietly vanishes from the + # denominator is exactly how a recovery rate gets manufactured. + stranded = {fn: p for fn, p in drafts.items() if fn in stubs} + banked = sorted(set(drafts) - set(stranded)) + print('drafts=%d stranded(still INCLUDE_ASM)=%d already-banked(excluded)=%d%s' + % (len(drafts), len(stranded), len(banked), ' dup-across-dirs=%d' % len(dup) if dup else '')) + if banked: + print(' excluded as banked: ' + ' '.join(banked)) + if not stranded: + return 1 + + work = os.path.join(REPO, a.work) + os.makedirs(work, exist_ok=True) + rows = {} + + def one(fn): + stub = stubs[fn] + text = open(stranded[fn], errors='replace').read() + return fn, static_verdict(fn, stub, text), cc1_verdict(fn, stub, stranded[fn], work) + + with concurrent.futures.ThreadPoolExecutor(max_workers=a.jobs) as ex: + for fn, st, (cc, det) in ex.map(one, sorted(stranded)): + # Blockers STACK (cc1 only ever reveals the first). The tier a function needs is the + # MAX over all of them: one T1 blocker among four T0s still makes the fix binary-local. + tiers = sorted(TIER.get(k, '--') for k, _ in st) if st else [] + rows[fn] = {'static': st, 'cc1': cc, 'cc1_detail': det, 'tu': stubs[fn].path, + 'tier': tiers[-1] if tiers else '--'} + + # ---- the disagreement table (the finding), not the agreement rate ------------------------- + print('\n%-16s %-4s %-14s %-26s %s' % ('fn', 'tier', 'static', 'real cc1', 'agree')) + print('-' * 108) + agree = dis_static_only = dis_cc1_only = 0 + for fn in sorted(rows): + r = rows[fn] + sk = r['static'][0][0] if r['static'] else 'none' + blocked_static = bool(r['static']) and sk != 'parse' + blocked_cc1 = r['cc1'] == 'CC1-FAIL' + if blocked_static == blocked_cc1: + verdict, _ = 'yes', 0 + agree += 1 + elif blocked_static: + verdict = 'STATIC-ONLY' + dis_static_only += 1 + else: + verdict = 'CC1-ONLY' + dis_cc1_only += 1 + print('%-16s %-4s %-14s %-26s %s' % (fn, r['tier'], sk, '%s %s' % (r['cc1'], r['cc1_detail'])[:26], + verdict)) + if a.verbose: + for k, d in r['static']: + print('%22s %-14s %s' % ('', k, d)) + + n = len(rows) + ccf = sum(1 for r in rows.values() if r['cc1'] == 'CC1-FAIL') + mat = sum(1 for r in rows.values() if r['cc1'] == 'MATCH') + dif = sum(1 for r in rows.values() if r['cc1'] == 'DIFF') + print('-' * 108) + print('n=%d real cc1: CC1-FAIL=%d DIFF=%d MATCH=%d' % (n, ccf, dif, mat)) + print('agreement=%d STATIC-ONLY(false positives)=%d CC1-ONLY(static blind)=%d' + % (agree, dis_static_only, dis_cc1_only)) + by = {} + for r in rows.values(): + for k, _ in (r['static'] or []): + by[k] = by.get(k, 0) + 1 + if by: + print('static blockers by class: ' + ' '.join('%s=%d(%s)' % (k, v, TIER.get(k, '--')) + for k, v in sorted(by.items()))) + print('\nCEILING: a draft that CC1-FAILs is recoverable if its blocker class has a transform;\n' + ' a draft that already DIFFs in its real TU is NOT an integration problem (G3/P9).') + + if a.json: + json.dump(rows, open(os.path.join(REPO, a.json), 'w'), indent=1, sort_keys=True) + print('wrote %s' % a.json) + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/rtu_match.py b/tools/rtu_match.py index bad11e33b..c14afd5e8 100644 --- a/tools/rtu_match.py +++ b/tools/rtu_match.py @@ -14,8 +14,14 @@ than match_one's. STILL finish on the real `make build` whole-binary SHA gate (G tools/rtu_match.py func_80164930 --split ov_SC01_077_after --c cand.c # candidate may carry file-scope TU pre-edits as leading directive lines: # //@EDIT old_text||new_text (applied to the split .c before splicing; e.g. the s16->u16 flip) + # the replacement may be MULTI-LINE: a literal backslash-n in new_text becomes a newline + # (needed to expand a DEFINE_func_* macro instantiation in place -- the §63 per-overlay-local + # decl override). Use \\n in the directive if you want a literal backslash-n. + # --stderr-out PATH persists the FULL stderr of every stage. The inline prints are tail-truncated, + # and these TUs emit hundreds of benign `type mismatch with previous external decl' warnings, so a + # truncated tail can hide the real first error entirely (the §58 red-herring, one level down). """ -import subprocess, re, sys, os, argparse +import subprocess, re, sys, os, argparse, atexit sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import masked_diff @@ -28,8 +34,24 @@ ap.add_argument('--asm-subdir', default=None) ap.add_argument('--work', default='.run/crack3/rtu') ap.add_argument('--o0', action='store_true') ap.add_argument('--maxdiff', type=int, default=60) +ap.add_argument('--stderr-out', default=None, + help='write the FULL stderr of every stage here (the inline prints are tail-truncated)') a = ap.parse_args() +# Every stage's stderr, verbatim, flushed on ANY exit path (including sys.exit). A consumer that +# classifies a failure needs the FIRST non-warning error line; a truncated tail cannot supply it. +_errlog = [] +def _flush_errlog(): + if not a.stderr_out: + return + d = os.path.dirname(a.stderr_out) + if d: + os.makedirs(d, exist_ok=True) + with open(a.stderr_out, 'w') as f: + for stage, err in _errlog: + f.write('### %s\n%s\n' % (stage, err)) +atexit.register(_flush_errlog) + SPLIT_SRC = 'src/%s/%s.c' % (a.source, a.split) ASM_SUBDIR = a.asm_subdir or ('asm/%s/nonmatchings/%s' % (a.source, a.split)) CPP='mipsel-linux-gnu-cpp'; CC1='tools/bin/gcc-2.7.2-psx/cc1' @@ -40,13 +62,27 @@ ASFLAGS='-Iinclude -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0'.split() split_txt = open(SPLIT_SRC).read() cand_raw = open(a.c).read() + +def _unescape_nl(s): + """`\\n` -> newline, `\\\\` -> a literal backslash. A //@EDIT directive is one LINE, so a + multi-line replacement (expanding a DEFINE_func_* macro in place) can only be written escaped.""" + out, i = [], 0 + while i < len(s): + if s[i] == '\\' and i + 1 < len(s) and s[i + 1] in 'n\\': + out.append('\n' if s[i + 1] == 'n' else '\\') + i += 2 + continue + out.append(s[i]) + i += 1 + return ''.join(out) + # extract //@EDIT directives (file-scope pre-edits), strip them from the spliced body edits = [] body_lines = [] for ln in cand_raw.split('\n'): m = re.match(r'\s*//@EDIT\s+(.*?)\|\|(.*)$', ln) if m: - edits.append((m.group(1), m.group(2))) + edits.append((_unescape_nl(m.group(1)), _unescape_nl(m.group(2)))) else: body_lines.append(ln) body = '\n'.join(body_lines).rstrip('\n') @@ -66,15 +102,18 @@ else: wd = '%s/%s' % (a.work, a.fn); os.makedirs(wd, exist_ok=True) open('%s/t.c' % wd, 'w').write(split_txt) -def pipe(cmd, data=None): return subprocess.run(cmd, input=data, capture_output=True) +def pipe(stage, cmd, data=None): + p = subprocess.run(cmd, input=data, capture_output=True) + _errlog.append((stage, p.stderr.decode('utf-8', 'replace'))) + return p # -Isrc/ so the split's relative `#include "../shared/..."` resolves from the temp dir -p = pipe([CPP]+CPPFLAGS+['-Isrc/%s'%a.source, '-DINCLUDE_ASM(a,b)=', '%s/t.c'%wd]) +p = pipe('CPP', [CPP]+CPPFLAGS+['-Isrc/%s'%a.source, '-DINCLUDE_ASM(a,b)=', '%s/t.c'%wd]) if p.returncode: print('CPP FAIL\n'+p.stderr.decode()[-1500:]); sys.exit(1) -p = pipe([CC1]+CC1FLAGS, p.stdout) +p = pipe('CC1', [CC1]+CC1FLAGS, p.stdout) if p.returncode: print('CC1 FAIL\n'+p.stderr.decode()[-2000:]); sys.exit(1) -p = pipe([PY, MASPSX, '--aspsx-version=2.56', '--expand-div'], p.stdout) +p = pipe('MASPSX', [PY, MASPSX, '--aspsx-version=2.56', '--expand-div'], p.stdout) if p.returncode: print('MASPSX FAIL\n'+p.stderr.decode()[-1500:]); sys.exit(1) -p = pipe([AS]+ASFLAGS+['-o', '%s/t.o'%wd], p.stdout) +p = pipe('AS', [AS]+ASFLAGS+['-o', '%s/t.o'%wd], p.stdout) if p.returncode: print('AS FAIL\n'+p.stderr.decode()[-1500:]); sys.exit(1) mine = masked_diff.insns_from_object('%s/t.o'%wd, a.fn)