mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired.
This commit is contained in:
@@ -8,6 +8,13 @@
|
||||
|
||||
SHELL := /bin/bash
|
||||
.ONESHELL:
|
||||
# FAIL-CLOSED BY DEFAULT (Phase-27 T2). Without `-e`, .ONESHELL sends the WHOLE recipe to one
|
||||
# `bash -c`, so a recipe's exit status is its LAST command's only — every earlier failure is
|
||||
# silently swallowed. That made `report`'s lint_symbol_refs / progress --audit / difficulty /
|
||||
# dup_report non-gates (dedup-check "worked" purely by being last), i.e. exactly the defect the
|
||||
# 26-A audit exists to kill: a loud failure nobody counts is as invisible as a silent one (R32).
|
||||
# `-e` makes every recipe line load-bearing. Deliberate opt-out: `check-env` (see its recipe).
|
||||
.SHELLFLAGS := -ec
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
# --- paths & tooling ---------------------------------------------------------
|
||||
@@ -125,7 +132,7 @@ CC1_SMOKE_FLAGS := -quiet -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-lin
|
||||
BINUTILS_WARN_MAJOR := 2
|
||||
BINUTILS_WARN_MINOR := 38
|
||||
|
||||
.PHONY: help check-env extract build check expected clean report sig-refresh sig-overlays build-all check-all audit-corpus audit-cdecl
|
||||
.PHONY: help check-env extract build check expected clean report sig-refresh sig-overlays build-all check-all audit-corpus audit-cdecl tools-health
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
help:
|
||||
@@ -168,6 +175,18 @@ audit-corpus:
|
||||
audit-cdecl:
|
||||
$(VENV_PY) tools/cdecl.py --audit --gcc
|
||||
|
||||
# The tool-health ritual (Phase-27 T2). Before the 26-A audit the two oracles above had NO dependent
|
||||
# — nothing invoked them, so "run the audits" was a manual habit, and a habit nobody automates is a
|
||||
# gate nobody counts (R32). This is that dependent: `make tools-health` runs both derived oracles and
|
||||
# the report gates (lint_symbol_refs + dedup-check) together, and under the global -e ANY one failing
|
||||
# aborts it. It is deliberately NOT a prerequisite of `report`/`build` — audit-cdecl cross-compiles
|
||||
# every C declaration through real gcc (minutes), so it belongs to a deliberate pre-matching ritual,
|
||||
# not the inner harvest loop. Matches the roadmap's standing invariant (audit-corpus · audit-cdecl ·
|
||||
# report green before matching).
|
||||
tools-health: audit-corpus audit-cdecl
|
||||
$(MAKE) --no-print-directory report BINARY=main
|
||||
echo "tools-health: OK — corpus + cdecl + report(lint+dedup) all green."
|
||||
|
||||
report:
|
||||
$(VENV_PY) tools/progress.py --binary $(BINARY) --audit
|
||||
$(VENV_PY) tools/difficulty.py --binary $(BINARY)
|
||||
@@ -214,7 +233,13 @@ sig-overlays:
|
||||
# stop at the first failure) so the report is complete, then exits nonzero if any
|
||||
# hard check failed. binutils >= 2.38 is a WARN, never a FAIL (§4.5).
|
||||
check-env:
|
||||
@fail=0
|
||||
# DELIBERATE opt-out from the global `-e` (.SHELLFLAGS, Phase-27 T2). This recipe's contract is
|
||||
# "run EVERY preflight check, print EVERY [FAIL], exit with the accumulated status" — it manages
|
||||
# its own `fail` and exits 1 at the end. Under `-e` a probe assignment (e.g. `pyver=$$(python3
|
||||
# ...)` on a box without python3) would abort at the FIRST problem and hide the rest, turning a
|
||||
# diagnostic into a stop-on-first-error. Accumulate-and-report is correct here; nowhere else.
|
||||
@set +e
|
||||
fail=0
|
||||
echo "== BFM-decomp environment preflight (Phase 4 check-env) =="
|
||||
echo
|
||||
# 1) Python >= 3.12 (system python3 drives tooling + the EXE-hash import)
|
||||
@@ -596,20 +621,41 @@ JOBS ?= 16 # parallel binary builds/extracts (override: `make check-all JOBS=
|
||||
# (+ build/psyq) exist before the parallel fan-out; then extract the rest in parallel.
|
||||
extract-all:
|
||||
@mkdir -p .run; : > .run/extract-all.txt
|
||||
# Under the global `-e` a failing main extract now aborts here. It previously did NOT: its status
|
||||
# was swallowed by .ONESHELL, and the closing `! grep -q` then passed regardless — a seed failure
|
||||
# could sail through as green.
|
||||
$(MAKE) --no-print-directory extract BINARY=main
|
||||
echo "$(filter-out main,$(BINARIES))" | tr ' ' '\n' | xargs -P$(JOBS) -I{} sh -c \
|
||||
'$(MAKE) --no-print-directory extract BINARY={} >.run/extract.{}.log 2>&1 || echo "[EXTRACT FAIL] {}"' \
|
||||
'$(MAKE) --no-print-directory extract BINARY={} >.run/extract.{}.log 2>&1 && echo "[ OK ] {}" || echo "[EXTRACT FAIL] {}"' \
|
||||
| tee .run/extract-all.txt
|
||||
! grep -q "EXTRACT FAIL" .run/extract-all.txt
|
||||
pass=$$(grep -c "^\[ OK \]" .run/extract-all.txt || true)
|
||||
fail=$$(grep -c "^\[EXTRACT FAIL\]" .run/extract-all.txt || true)
|
||||
want=$$(( $(words $(BINARIES)) - 1 ))
|
||||
echo "extract-all: $$pass extracted, $$fail failed of $$want (+ main, serial)"
|
||||
# Assert COVERAGE (pass == N-1), not the absence of a marker (R32) — `! grep -q "EXTRACT FAIL"`
|
||||
# was a vacuous pass on an empty pipeline.
|
||||
if [ "$$pass" -ne "$$want" ]; then
|
||||
echo "[FAIL] extract-all: expected $$want extracted, got $$pass (failed=$$fail)"; exit 1
|
||||
fi
|
||||
|
||||
check-all:
|
||||
@mkdir -p .run; : > .run/check-all.txt
|
||||
echo "$(BINARIES)" | tr ' ' '\n' | xargs -P$(JOBS) -I{} sh -c \
|
||||
'$(MAKE) --no-print-directory check BINARY={} >.run/check.{}.log 2>&1 && echo "[ OK ] {}" || { echo "[FAIL] {}"; tail -3 .run/check.{}.log >&2; }' \
|
||||
| tee .run/check-all.txt
|
||||
pass=$$(grep -c "^\[ OK \]" .run/check-all.txt); fail=$$(grep -c "^\[FAIL\]" .run/check-all.txt)
|
||||
echo "check-all: $$pass passed, $$fail failed of $(words $(BINARIES))"
|
||||
[ "$$fail" -eq 0 ]
|
||||
# `|| true`: grep -c EXITS 1 when the count is 0, and under `-e` a failing command substitution
|
||||
# aborts the assignment — so the bare form would make check-all FAIL exactly when nothing failed.
|
||||
pass=$$(grep -c "^\[ OK \]" .run/check-all.txt || true)
|
||||
fail=$$(grep -c "^\[FAIL\]" .run/check-all.txt || true)
|
||||
want=$(words $(BINARIES))
|
||||
echo "check-all: $$pass passed, $$fail failed of $$want"
|
||||
# Assert COVERAGE (pass == N), not merely the absence of a failure marker (R32). `fail -eq 0`
|
||||
# was a VACUOUS PASS: if the xargs pipeline emitted nothing at all, pass=0 fail=0 -> [ 0 -eq 0 ]
|
||||
# -> green while checking NOTHING. The byte-gate is a correctness oracle with a null coverage
|
||||
# dimension; this line is the coverage half.
|
||||
if [ "$$pass" -ne "$$want" ]; then
|
||||
echo "[FAIL] check-all: expected $$want passing, got $$pass (failed=$$fail)"; exit 1
|
||||
fi
|
||||
build-all: check-all
|
||||
|
||||
# check: SHA1 of the build vs the committed original hash. The definition of "build OK".
|
||||
|
||||
@@ -444,6 +444,12 @@ Modern cpp preprocesses → **vintage cc1** compiles to asm → **maspsx** emula
|
||||
|
||||
**As-built (Phase 5, 2026-06-14):** all five implemented in the root Makefile. The code is **100% assembly** (the phase's "all-asm byte-match"; the cpp→cc1→maspsx→as `c` path is wired-but-dormant until Phase 6). `make extract && make build && make check` → `build/us/SLUS_007.26` **SHA1-identical** to the original. Config `config/splat.us.exe.yaml` (platform psx, compiler PSYQ, subalign 2, gp_value 0x80074750, main segment `align: 4` so the text→data boundary isn't 16-byte-padded); committed checksum `config/check.us.sha`. Build chain = `as -march=r3000 -mtune=r3000 -no-pad-sections -O1 -G0` → `ld -T <splat .ld> -T undefined_syms_auto.txt -T undefined_funcs_auto.txt --no-check-sections` → `objcopy -O binary`.
|
||||
|
||||
**Fail-closed recipes (Phase-27 T2, 2026-07-15).** The Makefile sets **`.SHELLFLAGS := -ec`** — because `.ONESHELL` sends each whole recipe to ONE `bash -c`, so without `-e` a recipe's exit status is its LAST command's only, and every earlier failure is silently swallowed. That had made `make report`'s middle gates (`lint_symbol_refs`, `progress --audit`, `difficulty`, `dup_report`) into non-gates — the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure. Consequences of the flag, now standing:
|
||||
- **`make report` is genuinely fail-closed** — any of its gates failing exits non-zero (verified by a negative control: the same broken gate exits 0 under the old `-c`, non-zero under `-ec`).
|
||||
- **`make check-all` / `extract-all` assert COVERAGE (`pass == N`), not the absence of a failure marker** — the old `fail == 0` form was a *vacuous pass* on an empty pipeline. `check-all`'s `pass=$(grep -c …)` carries `|| true` (grep -c exits 1 on zero matches, which `-e` would otherwise treat as fatal — it would fail check-all exactly when nothing failed).
|
||||
- **`make check-env` opts OUT** (`set +e` at the top of its recipe) — its contract is accumulate-every-failure-and-report, which `-e` would truncate at the first missing tool. It is the **only** intended opt-out; add `set +e` to a recipe only with the same justification.
|
||||
- **`make tools-health`** (new) = `audit-corpus` + `audit-cdecl` + `report` run together, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names. Deliberately NOT a prerequisite of `report`/`build` (audit-cdecl cross-compiles every C declaration through real gcc, ~minutes). `audit-cdecl` ≈ several minutes; `audit-corpus` ≈ 7 s.
|
||||
|
||||
### §6.4 asm-differ + baseline discipline
|
||||
|
||||
```bash
|
||||
|
||||
@@ -36,7 +36,7 @@ Phase 26 closed on an honest pivot — the mechanical/templating harvest is byte
|
||||
## Task checklist (effort per R7 · one commit per completed task after this file is updated, Drew pushes — R6/R20)
|
||||
|
||||
- [ ] ▶ **Task 1 — Fable5 discovery sprint** `[orchestration xHigh · agents model:fable · distillation Max]` — **PERISHABLE, background.** *(Task 3 was pulled ahead of it — 5 minutes to make the sprint's inputs non-destroyable; see the Log.)* Wave 1 (parallel-isolated): the 3 recon-done pin-free seeds `func_8014D820` (304), `func_8016CBC0` (209), `func_801670E4` (279, close=23); seeds at `.run/giants/*.opus.{c,md}`. **Distill idioms into cookbook §31/§52 + `docs/gcc-2.7.2-map/` IN-SESSION (R30)** — the value is the idiom, not the bank (§52: a *failed* Fable5 pass still fed 670 cheap-Opus instances). Wave 2, informed by wave 1: `0x80176734` (371) + **the pin-crash cc1 SIGABRT characterization** (gates P31's pin-×1 endgame; harness works at `.run/fable_80178004/{runorc.sh,oracle2.gdb}`; cause is currently **hypothesis-only** — no abort site, assert identity, backtrace, or minimal repro exists). `func_80178004`'s `qty_n_refs` = wave-2 filler only (decision-log prices grinding it low-EV). **Gate: idioms distilled, not functions banked.** Verify: whole-binary byte-gate per crack; `family_sweep` propagate; R22 clean-fleet.
|
||||
- [ ] **Task 2 — Makefile fail-closed (the enabling fix)** `[xHigh]` — `.SHELLFLAGS := -ec` + **sweep every target** for recipes relying on ignored non-zero exits (`check-all:610`'s `grep -c` returns 1 on zero matches → would abort). `fail == 0` → **`pass == N`** in `check-all`/`extract-all`. Give `audit-corpus`/`audit-cdecl` a dependent. Verify (known-answer): break `lint_symbol_refs` → `make report` MUST exit non-zero; empty the `check-all` pipeline → must fail; `make check-all` still 136/136.
|
||||
- [x] **Task 2 — Makefile fail-closed (the enabling fix)** `[xHigh]` — **DONE.** `.SHELLFLAGS := -ec` (global fail-closed) with ONE documented opt-out: `check-env` (`set +e` — its contract is accumulate-every-failure). Fixed the `check-all:610` `grep -c` landmine (`|| true` — grep -c exits 1 on 0 matches, which `-e` would treat as fatal → check-all would fail when nothing failed). Strengthened `check-all`/`extract-all` from `fail == 0` → **`pass == N`** (coverage assertion, R32 — the old form was a vacuous pass on an empty pipeline). Gave the two audit oracles a dependent: **new `make tools-health`** = `audit-corpus` + `audit-cdecl` + `report`, fail-closed (NOT a `report`/`build` prereq — audit-cdecl is ~minutes). SETUP §6.3 documents it (R21). **VERIFIED:** (1) known-answer — a broken `lint_symbol_refs` makes `make report` exit non-zero, and a **negative control** proves it: the *identical* break exits **0** under old `.SHELLFLAGS=-c`, **2** under `-ec`; (2) the `grep -c` landmine and the vacuous-pass both reproduced + fixed in isolation; (3) `check-env` still exits 0 (opt-out works); (4) **`make check-all` → 136/136 byte-identical**, and a forced `main` re-extract+rebuild exercised the full splat→cpp→cc1→maspsx→as→ld→objcopy→check pipeline under `-e` → `143dbb89…`; (5) `audit-corpus` (7s) + `audit-cdecl` (green) + `tools-health` dry-run all wired. Recipe scan found the Makefile was already `-e`-aware (`set -o pipefail`, explicit `|| true`, guarded `@` lines) — line 610 was the only real hazard. *(completes with this commit)*
|
||||
- [x] **Task 3 — Curated `.run/` preservation** `[xHigh]` — **DONE** (pulled ahead of Task 1 — it de-risks the sprint's inputs). `.gitignore` `/.run/` → contents-exclude form (`/.run/*` + `!` exceptions, the `/tools/bin/*.sha256` precedent). **Refined at execution against the bytes:** the naive "commit the dirs" would have been **12.3 MB of regenerable gcc RTL scratch**; the genuinely irreplaceable set is **~2.2 MB / 31 files** — the 6 Phase-25 `*.opus.{c,md}` seed recons (49K), the `func_80178004` gdb-on-cc1 **harness + `ORACLE_PROOF.md` + the v00–v07 draft ladder + the sched/combine `.lst` evidence** (~110K), and the two frontier ledgers (`backlog.jsonl` 1.9M, `fuel_manifest.json` 67K). `dumps_v00..v07/` + `d_pf*.i.*` stay ignored — **regenerable via `runorc.sh` + the `.gdb` scripts** (R33: commit what a rerun cannot reproduce). **VERIFIED:** `git add --dry-run .run/` stages exactly the 30 intended files, 0 bulk; negative control — `.run/ghidra-mcp.log`, `dumps_v00`, `d_pf.i.sched`, `d_pf.s` all still `IGNORED`; no `db.*.gbf` staged (R23). *(completes with this commit)*
|
||||
- [ ] **Task 4 — The cdecl strip primitive + surface cc1 stderr** `[xHigh]` — one primitive replacing two regexes with **complementary holes**; strip-set **per-TU** via `cdecl.tu_scope` (NOT a global `engine_types.h` parse — 213 TUs never include it; NOT a longer name list — `tooling-audit.md:1652`); split multi-typedef lines via `cdecl.split_statements`. **Surface cc1 stderr** in `harvest_verify.build()` (currently `capture_output=True` throws it away → a plumbing failure is recorded byte-identically to a compiler wall). Unblocks the 39 `_TD`-blocked fns (8 already FAILED), the 42 discarded masked-MATCH drafts, and **B4's `func_8015C32C`**. Verify (known-answer): `.run/drafts-t3b1-fb/func_8015C030.c` must reach `MATCH (23 ins)` **unedited** (today it CC1-FAILs; newlines alone fix it).
|
||||
- [ ] **Task 5 — `extract_unit` macro-carry → the `0x8017BEBC` probe** `[Max]` — carry file-scope macro deps; then `--stage-only` sizing (minutes) → a **bounded ~10-member gate sample** for the rate → leave the full ~112 gated builds to P28's gate farm. Verify: the staged unit compiles standalone (today it cannot); report the realized bank-rate as a measured number. **A 0% here is only meaningful AFTER the macro fix.**
|
||||
@@ -61,4 +61,5 @@ Phase 26 closed on an honest pivot — the mechanical/templating harvest is byte
|
||||
|
||||
- **2026-07-15 · Task 3 — Curated `.run/` preservation.** **Pulled ahead of Task 1** (a 5-minute deviation from plan order, P3 autonomy): Task 1's Fable5 agents work *inside* `.run/`, and its Phase-25 seed recons were untracked — an agent overwriting `.run/giants/func_8014D820.opus.c` would have destroyed irreplaceable input. Five minutes out of a four-day window is a trivial price for removing that. **Execution refined the plan against the bytes (R33):** the plan said "track `.run/giants/*.opus.{c,md}` + `.run/fable_80178004/`", but those directories are **8.5M and 3.8M — almost entirely gcc RTL dump scratch** (`d_pf.i.combine/.sched/.lreg`, `dumps_v00..v07`) that `runorc.sh` + the `.gdb` scripts regenerate. The irreplaceable core is **~2.2 MB**: 49K of seed recon, ~110K of oracle harness + proof + draft ladder, and the two ledgers. Committed that; left the regenerable bulk ignored. Verified both directions (intended set stages; bulk still `IGNORED`). **Carried to Task 1:** the sprint's *outputs* must be added to the allowlist as they land — the same reasoning that motivated this task.
|
||||
|
||||
- **2026-07-15 · Task 2 — Makefile fail-closed.** The roadmap §5 asserted `make report` is fail-closed; it was not (`.ONESHELL` + no `-e` → only the last command's exit survives; `dedup-check` "gated" purely by being last). Set `.SHELLFLAGS := -ec` globally + `check-env` opt-out; fixed the `grep -c` landmine; upgraded `check-all`/`extract-all` to coverage assertions (`pass == N`); added `make tools-health` as the audits' dependent. The **negative control** is the proof that mattered — same broken gate, exit 0 under `-c` vs exit 2 under `-ec` — turning "the swallow is real" from a claim into a measurement (R14 discipline applied to my own fix). Full clean-fleet R22 held (136/136 + a forced main rebuild under `-e`). **This unblocks every downstream R32 assertion**: until now, any gate added to a report-invoked tool was swallowed on arrival. SETUP §6.3 updated (R21).
|
||||
- **2026-07-15 · Task 3 (addendum) — the allowlist was still too narrow; cookbook §45 cites untracked files.** While reading the seeds for Task 1 I hit a real defect: **cookbook §45 names `.run/giants/func_80133CD4.fable.c` as its worked example and `.run/giants/fable_cd4/` as the flagship's gdb oracle — and BOTH were untracked.** The documentation cites artifacts that were not in git. Widened the allowlist by **file type rather than directory** (`.run/giants/*.{c,md,sh}` + `fable_cd4/*.{c,md,sh,gdb,txt}`), adding **49 files / 460K**: the flagship crack + its oracle, the byte-verified `pf*.c` regression ladder (the seeds' own "Method/reproducibility" cites it), the `dump.sh`/`mon*.sh` harnesses, and the banked giants' drafts. `d_pf*.i.*`, `*.s`, `dumps_m*/`, and the ILS/permuter `.log`s stay ignored (regenerable via `dump.sh`). Negative control re-verified. **Lesson (→ R31/decision-log): a doc that cites a path is an untested claim about the repo — the citation and the file were four days out of sync, and only reading the seed for an unrelated reason caught it. Candidate for a lint (cookbook path citations must resolve to tracked files).**
|
||||
|
||||
Reference in New Issue
Block a user