mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 14:02:04 -04:00
ebdef9012bd658df2df537e6825c978f0fb5bcc8
632 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
ebdef9012b |
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired. |
||
|
Drew T
|
6e99157bcf |
chore(phase-27 T3 addendum): widen the .run/giants allowlist — cookbook §45 cited untracked files
Found while reading the seeds for T1: cookbook §45 names
.run/giants/func_80133CD4.fable.c as its worked example and .run/giants/fable_cd4/
as the flagship's gdb oracle — BOTH were untracked. The docs cite artifacts that
were not in the repo.
- .gitignore: widen by FILE TYPE, not directory — .run/giants/*.{c,md,sh} +
fable_cd4/*.{c,md,sh,gdb,txt}. +49 files / 460K.
- Now preserved: the flagship func_80133CD4 crack + its gdb oracle (§45's cited
worked example); the byte-verified pf*.c regression ladder (the seeds' own
Method/reproducibility section cites it: pf2 78, pf_c2 30, pf_d1 35, pf_h1 280);
the dump.sh/mon*.sh RTL harnesses; the banked giants' drafts (80135480, 80163EC8,
80166994).
- Still ignored (regenerable via dump.sh, R33): d_pf*.i.*, *.s, dumps_m*/, and the
ILS/permuter .log files. Negative control re-verified: all 5 probes IGNORED, no
db.*.gbf staged (R23).
Lesson (R31 candidate): a doc that cites a path is an untested claim about the repo.
The §45 citation and its file were 4 days out of sync; only reading the seed for an
unrelated reason caught it. Candidate lint: cookbook path citations must resolve to
tracked files.
|
||
|
Drew T
|
2351c43e66 |
chore(phase-27 T3): preserve the irreplaceable .run/ recon (R20) — 2.2M, not 12.3M
Pulled ahead of T1: the Fable5 sprint's agents work inside .run/, and its Phase-25
seed recons were untracked — an agent overwriting .run/giants/*.opus.c would have
destroyed irreplaceable input. 5 minutes to remove that risk.
- .gitignore: /.run/ -> contents-exclude form (/.run/* + ! exceptions), following the
/tools/bin/*.sha256 precedent. Resolves R20 (commit irreplaceable RE work) vs R12
(.run/ is scratch) by splitting the directory on the real axis: what a rerun CANNOT
reproduce.
- PRESERVED (~2.2M / 31 files): the 6 Phase-25 *.opus.{c,md} giant seed recons (49K);
the func_80178004 gdb-on-cc1 harness + ORACLE_PROOF.md + the v00-v07 draft ladder +
the sched/combine .lst evidence (~110K — the distilled output of a 477k-token Fable5
pass, and the method §52 lever 6 depends on); backlog.jsonl (1.9M) + fuel_manifest.json.
- STILL IGNORED (regenerable, R33): dumps_v00..v07/ and d_pf*.i.* gcc RTL scratch —
12.3M reproducible via runorc.sh + the .gdb scripts; the MCP log; draft scratch.
The plan said "track the dirs"; the bytes said the dirs are 96% regenerable.
- VERIFIED both directions: git add --dry-run stages exactly the 30 intended files and
0 bulk; negative control — ghidra-mcp.log / dumps_v00 / d_pf.i.sched / d_pf.s all
still IGNORED. No db.*.gbf staged (R23 restart-noise).
|
||
|
Drew T
|
002f6d7c7b |
chore(phase-27 T0): Phase Start — plan approved (gate 1) + the P27 verification corrections
- CURRENT_PHASE.md: the approved plan (11 tasks, effort-annotated per R7), the
Planning-verification section, blockers, and the Task-0 log entry
- harness task list built before work (R28); deps enforced T6->T7, T2->T8,
T2+T7->T10, T5->T9, all->T11
VERIFICATION (R14 at planning scale — the roadmap's own §0 mandate): 3 read-only
agents checked every P27 premise against the repo. ~28 specifics corrected. The
three that reshaped the plan:
- the 0x8017BEBC probe ("possibly the largest cheap win left") would fail 112/112
today on a TOOLING defect — extract_unit drops the 8 file-scope gte_* macros the
banked exemplar needs; 0 of 112 member TUs define them. It would have been logged
as a 4th 0% exhaustion probe: the 26-A audit's thesis, about to recur.
- `make report` is NOT fail-closed (roadmap §5 asserts it is): .ONESHELL + no -e in
.SHELLFLAGS => only the last command's exit survives; lint_symbol_refs/progress
--audit/difficulty/dup_report are swallowed. check-all/extract-all assert fail==0,
not pass==N => an empty pipeline is a vacuous pass.
- 4 code-bearing SC07 payloads (~2.45 MB, 98.0% plausible-opcode) are invisible to
every tool: they sit at PAC entry index 1 while new_overlay.sh hardcodes 0.4.dec.
Zero mentions in docs/ or config/.
DROPPED with reasons: 0x8013C414 (x134 contested by an explicit verified_reach:1;
already drafted) · func_801549F8 (from the SUPERSEDED megaplan; Phase 26 walled it
17/31 and wrote "do NOT hand-grind it"; inside the 0/958 re-gate) · func_8012E364
(the "stale closeness" label is itself stale — close=23) · jtbl_carve "fix first,
load-bearing for B5" (the audit measured it twice: 0 of 5043 jtbl ends differ).
B4 dissolves into T4 — its remedy already ran (A9b, 1/7) and the 1 is banked.
Owner decisions (Drew, 2026-07-15): curated .run/ preservation (R20 vs .gitignore —
every Fable5-sprint input is currently untracked) · full disc audit incl. the
type-sweep, accepting the denominator expansion · Fable5 in 2 waves, distill between.
|
||
|
Drew T
|
6344a2c424 |
docs(roadmap): adopt the Road-to-100 endgame roadmap (Phase 27+) — supersedes the family-endgame megaplan
- docs/roadmap-to-100.md: the adopted P27-P32 roadmap to game-code TRUE 100% + public flip + Gen2 exit. Contract (Drew 2026-07-15): true 100% (walls re-attacked until they fall), PsyQ LINKED = complete (libs-from-source = far-future note), flip AT 100% (standing per-phase velocity checkpoint keeps the timing falsifiable), Fable5 window ~7/19 (P27 discovery sprint runs FIRST). Measured baseline from committed post-audit artifacts only; residue buckets B1-B12 incl. the never-probed 0x8017BEBC IMM family (~106k ins) and the main/resident second-oracle gap; foreseen-tooling table (member_adapt, gate farm, fleet Ghidra-C prefetch, diff_regions, cc1_probe, assert-partition); authority rules (docs-layer advisory; every PhaseEnd carries a 'Roadmap delta' line; constitution + latest PhaseEnd win) - docs/decision-log.md (R31): the adoption entry — Drew's four contract decisions, the flip-timing tension + checkpoint, the 16-defect red-team pass that purged numbers carried past their invalidation events (R14 at planning scale) - docs/family-endgame-megaplan.md: SUPERSEDED banner (content preserved; h_seq reframe survives, the templating thesis is byte-proven spent per PhaseEnd_Phase26) - memory: roadmap-to-100 pointer added; structural-family-mechanical-remap corrected with the Phase-26 probe results (outside the repo tree) - Phase 27 NOT started (fresh session, plan mode, per the constitution) |
||
|
Drew T
|
1bbab78c65 |
feat(phase-26): close — family engine + the tooling-integrity audit + the §52 discovery-flywheel; mechanical harvest byte-proven exhausted; fleet 58.2->68.9% instr (v1.25.0)
- FAMILY ENGINE (Tasks 1-6): family_remap (extended reloc tracker + single-pass subst) + family_hseq (the h_seq reframe) + family_sweep (crack-one -> template-x134 -> byte-gate) + canon_sig_reconcile v3.2 + rtu_match. Sessions 6-8 cracked 13 cores (58.2->66.5% instr). - 26-A TOOLING-INTEGRITY AUDIT (inserted half-phase, A0-A11): the tools WERE several of the walls. Fixed ~15; DELETED decaying scanners (R33); built corpus.py + cdecl.py (derived, coverage-asserted oracles) + make audit-corpus (a SECOND, disagreeing oracle, R34); the listCdBuffer 193-slice corpus defect -> 0; masked_diff 150 closeness-lies -> 4; the stale- object false-pass closed. Payoff 66.5->68.6% instr. docs/tooling-audit.md AUDIT-CLOSE LEDGER. - 52 DISCOVERY-FLYWHEEL (Task 7): single Fable5 on func_80178004 = intrinsic 3-integer regalloc wall, BUT distilled the walker-family idiom (52); two cheap-Opus waves applied it -> 5 pin-free cores banked x134 = 670 instances (68.6->68.9%). 4 named wall classes; 52/52a/52b. pin-guard comment false-positive fixed; family_sweep --allow-pins. - FINDING (R14/P9, 3 probes 0%): the matched-sib mechanical harvest is EXHAUSTED; the manifest's ~13k "templatable" members are an h_seq prediction the byte-gate refuses. Phase-26's templating thesis is spent -> close, open Phase 27 with byte-gate-honest re-scans. - R22 clean-fleet 136/136 BYTE-IDENTICAL throughout; dedup 1840/0; 0 NON_MATCHING (G4). - rules R32 (coverage assertion) / R33 (derive, don't re-derive) / R34 (a second, disagreeing oracle). worklog -> phase-ends/logs/Phase26.md (R19). bumps 1.24.0 -> 1.25.0. |
||
|
Drew T
|
d40711fe5f |
chore(phase-26 T7): family_sweep --allow-pins flag + honest matched-sib finding
- family_sweep: --allow-pins bypasses the §42e pinned-exemplar skip (template WITH pins, byte-gate arbitrates) — added to TEST the func_8017A4AC pinned-×134 precedent. - FINDING (3 probes, all 0% banked): the matched-sib harvest is TAPPED. tiny-IMM 0/241, PURE reach-134 0/134, pinned-PURE-with-pins 0/133. The manifest's ~13,075 'templatable' member-slots are an over-prediction the whole-binary byte-gate refuses (collision/drift/ pin-crash). A3h + the wave propagations already banked everything cleanly templatable. - regenerated family-hseq.md. |
||
|
Drew T
|
ed09ee749f |
feat(phase-26 T7): §52 sibling wave 2 — 3 more cores banked ×134 (402 instances)
Second cheap-Opus §52 wave over the close=0 regalloc cluster (armed with §52a):
- BANKED ×134: func_801379FC (97), func_801497A8 (47), func_801495C4 (34) —
3 exemplars + 399 members = 402 function-instances, 0 gate failures.
- 2 whole-binary-near (func_8012E138, func_8012F40C — match_one MATCH, A10 gap),
1 new wall (func_8012B4B8 — symbol-address-base wins-low-needs-high, a 3rd class).
- §52b: new verified de-pin levers (per-loop pseudos for register role-swap; the
RC-7 second-set dial to defeat rematerialization; value-barriers dissolve the
CSE-stack-address-common wall) + the new wall class + the match_one→whole-binary
gap-at-scale finding.
- TOOL FIX: family_sweep §42e pin-guard was a FALSE POSITIVE — it matched
'__asm__("$N")' inside COMMENTS that document a REMOVED pin (recovered
func_801495C4's 133 members). Now strips comments before the pin check.
- R22 clean-fleet 136/136 BYTE-IDENTICAL; dedup 1840/0.
- Wave 1+2 combined: 5 pin-free cracks -> 670 instances, from the walled flagship's idiom.
|
||
|
Drew T
|
06e43873e9 |
feat(phase-26 T7): §52 regalloc sibling wave — 2 cores banked ×134 (268 instances)
The Fable5 walker-family idiom (§52, from the func_80178004 wall) applied by a 6-agent cheap-Opus wave over the regalloc-order reach-134 cluster: - BANKED ×134: func_80171FFC (40 ins), func_801775E0 (67 ins) — 2 exemplars + 266 members = 268 function-instances, 0 gate failures (family_sweep byte-gate). - 4 precisely-characterized walls (P9), each yielding a byte-verified lever: func_80167714 (whole-binary near), func_80177AD4 (non-coalescing delay-slot copy), func_80169228 (NEW caller-saved priority-first-fit wall), func_80131A34 (save-order/ load-hoist tension; new const-unchanging-load lever). - §52a: the wave's new banking levers (pass-real-args/RC-10, store-base-both-arms, copy-chain-direction, pp-decl-schedule, const-unchanging-load/RC-3) + the two new intrinsic-wall classes. Fable5 DISCOVERS, cheap-Opus APPLIES. - fn-count 84.27→84.35% (+268), instr-weighted 68.6→68.7% (+14,338 ins), distinct +2. - R22 clean-fleet 136/136 BYTE-IDENTICAL; dedup 1840/0. |
||
|
Drew T
|
8f3e4a12ba |
docs(phase-26): T7 re-baseline + §52 walker-family regalloc idiom (func_80178004 wall)
- re-baseline the target frontier from the fixed tools (R14): 228 ov077 stubs (broken manifest saw 30); worklist/family-hseq regenerated. 68.6% instr / 49.2% distinct. - R14 corrections to the audit handoff: the '~1,200 type-heavy' was already banked by A3h (+2,675); the real remaining lever is 128 matched-sib families (~2.64M templ ins), and register pins are NOT a banking blocker (func_8017A4AC banked ×134 with pins). - Task 7 is near-miss CLOSING (close 1-5 re-gate = 0/13 through the fixed pipeline), not fresh crack. - §52: single Fable5 on func_80178004 (regalloc-order class exemplar) = honest wall (P9), structure-exact 163/165, residual = 3 compiler-internal register-alloc integers, likely intrinsic; byte-verified 126/165. Correction: the 'pinned MATCH' was a myth (never matched). 6 byte-proven walker-family levers + the skeleton idiom transfer to the 11 regalloc-order siblings (Fable5 discovers, cheap-Opus applies). decision-log entry (R31). |
||
|
Drew T
|
a33c6f85f5 |
docs(phase-26a): A11 — distill + close the tooling-integrity audit (26-A COMPLETE)
Closed the inserted half-phase. tooling-audit.md: DIAGNOSIS -> AUDIT-CLOSE LEDGER (A1-A10 outcomes + the payoff 66.5->68.6% instr + remaining/handoff); the "two rules" -> R32/R33/R34 crisp for P10 ratification at the Phase-26 PhaseEnd. decision-log: the A10 wall-re-test verdict (R31 -- the broken tools WERE the walls; the payoff was banked by the fixes; the closeness-0 residual is genuine; the real deliverable is the 3 rules + the derived-oracle pattern). SETUP: the A9d-A10 tool changes (R21). Cookbook §51 verified complete; LAW 3 tagged R34. Observables green: final R22 clean-fleet 136/136 BYTE-IDENTICAL; make report EXIT 0 (dedup 1840/0, C1 227211/227211 signed, lint_symbol_refs wired + passing); audit-corpus 0 slices; audit-cdecl green. Zero src/config changes this session. Phase 26 resumes at Task 7 (fresh session). |
||
|
Drew T
|
e9038a04ea |
docs(phase-26a): A10 COMPLETE — wall re-test verdict on all 5 walls
The audit thesis is CONFIRMED: the tooling-walls were dissolved by the FIXES and the payoff banked there (A3f/g/h + A9a/b, 66.5->68.6% instr), while the re-tests confirm the residual walls are real. - fuel/closeness-0: CONFIRMED REAL (0/958 bank at fleet scale). - arity (Phase-15 dead-end): was tooling (A3c order-dependent rule); 13/18 banked. - def-side loose-typing: was partly tooling; A9b banked func_8017A4AC x134. - type-heavy: blocking tool build_engine_types was broken (A7 fixed it, now runs); the ~1,200-member family harvest is Task-8 integration, not a pure re-test. - 780 h_seq callee-oracle rejections: was tooling; A3h banked +2,675. A re-confirmed wall is as valuable as a dissolved one (P9). Next: A11 close. |
||
|
Drew T
|
b1c58d7668 |
docs(phase-26a): A10 wave 1 — closeness-0 wall re-test CONFIRMED REAL (0/958 bank)
Re-gated all 958 closeness-0 open-stub backlog drafts through the FIXED gate across 135 binaries in parallel: banked=0, near=957, failed=71. The closeness-0 backlog is genuine whole-binary near-misses, NOT tooling misses -- match_one's isolated closeness==0 systematically overstates whole-binary bankability, and the repaired gate recovers none. P9: a re-confirmed wall is as valuable as a dissolved one. (The audit's tooling-walls were already banked by A3f/g/h + A9b, +2.1% instr.) backlog.py: env-gated BACKLOG_NO_RENDER so parallel workers skip the render race (append is atomic) -- backward-compatible parallel-safety. backlog.md refreshed with the re-test's whole-binary-informed scores. |
||
|
Drew T
|
f2b2a9d778 |
docs(phase-26a): A10 Max-phase — wall re-test scoping + arity measurement (fan-out pending R27)
Scoped the 5 walls. The audit fixes already dissolved the easy wall (A3f/g/h + A9b, 66.5->68.6% instr). Remaining re-test pool = 2,218 open-stub backlog drafts (958 at closeness 0). #2 arity: 13/18 Phase-15 arity-conflict fns already banked (wall largely fell). Confirmed the remaining re-test is breadth (serial gate_stage timed out). R27 boundary reached: prompting Drew for /effort ultracode before the parallel re-gate fan-out. Tree clean (HEAD commit:0618 before this). |
||
|
Drew T
|
ea20bdf9f3 |
fix(phase-26a): A9g — jr_inventory: retire the ephemeral roster, derive banked from the image (R33)
jr_inventory's `banked` set was filtered by an EPHEMERAL, gitignored .run/banked_func_*.json roster: a `rm -rf .run` / fresh clone would blind ALL banked jr at once, cross-address siblings (roster named after the exemplar) were structurally invisible, and non-leader banked jr were missed. "The purest R33 case in the group" (audit). FIX (the audit's exact prescription): delete the roster glob + `cand` filter; `banked` is DERIVED FROM THE IMAGE — a real-C def/define fn is a banked jr iff family_remap.reloc_targets shows it references a committed .rodata carve offset (config + image, both durable; cross-address- and non-leader-immune). R32 assertion: every committed carve must resolve to EXACTLY ONE owner or abort (a stranded/duplicated carve is the §8b func_801734BC incident, never silent). Also fixed the adjacent finding: the asm_jr scan's func_-fullmatch dropped the curated-name listCdBuffer jr; now resolved via oss.addr_of(). (The --only path's own fullmatch is left — it parses user input, not the corpus.) Perf: read the overlay image ONCE and pass it to reloc_targets(..., data=) — a new backward-compatible param on family_remap (regression: 0/80 mismatch vs the re-read path). Verified: data-param behavior-identical; the R33 win — ov_SC02_000 now finds the cross-address sibling func_8017FCB0 the roster missed; full-fleet parallel run = 134/134 OK, 0 false aborts, 1336 banked jr == 1336 carves -> 1:1 ownership holds fleet-wide. Byte-safe: jr_isolate_all is not in the make build/extract path (R22-neutral); the change makes future isolations strictly more correct. |
||
|
Drew T
|
96e025a324 |
fix(phase-26a): A9f — overlay_src_split swallowed 2 real definitions; the selftest was blind
scan_construct's force_decl latched from the FIRST token and returned at the
first depth-0 `;`, so a definition sharing a physical line with leading externs
(`extern A; extern B; void f(){...}`) was never anchored — absorbed into the
next anchor's preamble. The parser jr_isolate_all rewrites source from was short
two functions in the exemplar overlay. The round-trip selftest is a SERIALISATION
check (a miss lands in a preamble -> round-trip still exact BY CONSTRUCTION), so
it was structurally incapable of seeing this.
FIX (byte-safe): force_decl no longer survives a same-line `;` with trailing
code — re-classify from the remainder and keep scanning so the def anchors (its
leading externs stay in its whole-line item text -> round-trip byte-identical).
Rejected the audit's "split into 3 constructs": round-trip joins whole-line
chunks with `\n`, so sub-line splitting would insert a newline where a space was.
def_name now names the LAST top-level header before `{` (the definition, not the
first same-line extern; byte-identical on every single-def construct).
R32: hidden_definitions() coverage oracle wired into selftest — an independent
detector of `func_XXXX(...){` bodies not anchored. The selftest is now a coverage
check, not just serialisation.
Verified: 2 swallowed -> 0; regression over 1738 overlay .c = 0 round-trip fails,
0 non-monotonic, 0 non-additive changes, +2 anchored defs. Byte-safe: tool not in
the build path (R22-neutral); ov_SC01_077 rebuilds d19c9580; neither def straddles
a committed subseg boundary. Audit ledger line refs were stale (src rewritten);
real cases are ov_SC01_077_after.c:2020 + ov_SC01_077_jr_8015444C.c:1495.
|
||
|
Drew T
|
68d29ba8af |
fix(phase-26a): A9e — reconcile_tu already wired into bank_exemplar (A3d); document the ladder
NULL RESULT (P9/R14): the session-12 "wire reconcile_tu into bank_exemplar"
handoff item was stale — A3d (commit:0601) already wired reconcile_tu into
jtbl_family_bank.recover() ("on BOTH banking paths"), and bank_exemplar's
`recovered` stage delegates to fb.recover = cast_call_sites + reconcile_tu.
Proven working by A9b (func_8017A4AC banked at the recovered stage, reconcile_tu
resolving its struct + fn-ptr conflicts). No live tool references the RETIRED
reconcile_decls (only docstrings + the audit-cdecl differential harness).
No code change warranted. Byte-neutral hardening only: document the
raw/scoped/recovered/reconciled fallback-ladder composition inline in
bank_exemplar so a future session does not re-run this "is it wired?" trace.
|
||
|
Drew T
|
40477281ce |
fix(phase-26a): A9d — retire the dead Phase-17 canonical-sig chain (R33)
DELETE tools/census_conflict_callees.py + tools/derive_canonical_sigs.py. - census_conflict_callees: audit-CONFIRMED marked-for-deletion (commit:0593; decision-log 836). It re-derives from C text the per-TU "defined/declared/ stubbed/external?" question that reconcile_tu (Phase 26) answers FROM THE BUILD — and does it WRONG in the unsafe direction (unknown -> conflict-free). - derive_canonical_sigs (census's ONLY consumer): genuinely dead — last touched Phase-17 (commit:0140), output .run/canonical_sigs.json read by nothing (no Makefile/workflow/import), no-ops on the 2-byte [] input, asm-arity heuristic 36% wrong vs byte-exact banked C. Its purpose was retired in A3d (fleet-majority oracle -> reconcile_tu's per-TU oracle). Deleting census orphans it, so the whole dead chain ceases to exist (R33: the best outcome is a DELETED SCANNER, not a fixed regex). Byte-neutral by construction (neither tool is in any build/report path): module-import smoke over the 13 importable harvest/bank/report/reconcile tools = all clean; bank_exemplar is a run-only script (indexes sys.argv at module scope), imports neither deleted module. No src/config change -> no byte moves. Doc-pointer hygiene: hand-matching-process.md 8a, matching-cookbook.md (canonical-sig-layer entry), tooling-audit.md (ledger row + derive entry) all annotated DELETED/historical so nothing points at a nonexistent tool. |
||
|
Drew T
|
181191b6af | docs(phase-26a): log A9c (lint_symbol_refs green + wired into make report) | ||
|
Drew T
|
d7d2613ae6 |
fix(phase-26a): A9c — lint_symbol_refs green + wired into make report
The ONLY detector for the R22 rename-drift failure mode (a symbols.us.txt rename leaves a func_<ADDR> ref dangling in committed src; a clean rebuild fails, an incremental build masks it with a stale .o — undetected Phase 21→23). It was RED (262 false positives) and UNWIRED. The audit's 3 blind spots, all fixed: #1 globbed src/**/*.c only -> now ALSO scans src/shared/*.h, where engine_core.h's 10k+ func_/D_ tokens live and one dangling ref breaks EVERY clean build at once (negative-control-proven: an injected bare ref in a shared header IS flagged). #2 read 2 of 138 symbol files -> now reads every REAL stacked file (config/symbols*.txt), and correctly EXCLUDES the R13 proto files (never stacked into a build; would invent phantoms). #3 no __asm__("label") model -> ALL 262 false positives were this class: engine_core.h binds func_8005C324 to memcpy via `... __asm__("memcpy")`, so the ref emits the LABEL and resolves at link regardless of the rename. asm_labeled_addrs() now recognizes it. VERIFIED: green on HEAD (exit 0); negative control (inject a bare func_800d1bd8 -> DsMix in a shared header) FLAGS it (exit 1) then clears on removal — detection intact, not trivially green. Wired as a fail-closed gate in `make report` (make -n confirms), beside dedup_integrate --check. No src/build change (report-time integrity gate) -> the byte-identical build is unaffected. |
||
|
Drew T
|
49004f5b06 | docs(phase-26a): log A9a (canon_sig_reconcile fn-ptr fix) + A9b (func_8017A4AC ×134 wall re-test) | ||
|
Drew T
|
3509acf4b7 |
feat(phase-26a): A9b — func_8017A4AC banked ×134 (536-ins giant, wall re-test payoff)
The A10 re-test payoff. func_8017A4AC (536 ins, reach-134) — "blocked on plumbing" since session 8 — banks now that the audit repaired the recover path (A3d reconcile_tu / A3e gate). jtbl_family_bank --raw swept all 133 siblings (per-sibling isolate → jtbl carve → remap_hseq + canon_sig_reconcile → whole-binary gate): 133/133 BANKED, 0 failed. 0 still-stub overlays. R22 CLEAN-FLEET (make clean + extract-all + check-all): 136 passed, 0 failed of 136. dedup-check 1840/0 (jtbl sweep banks are per-overlay src, not registry). DELTA: instr-weighted 68.1% -> 68.6% (+0.5%, ~71,824 shipped .text instructions) distinct-code 48.0% -> 49.2% (+1.2% — the siblings are per-location byte-variants) The audit thesis, demonstrated: a giant "wall" that stood for many phases was our TOOLING (the recover path could not resolve its struct/fn-ptr conflicts), not an intrinsic compiler residual. Once the oracle was fixed, the wall dissolved and banked ×134. |
||
|
Drew T
|
97d86fae69 |
feat(phase-26a): A9b — bank func_8017A4AC exemplar ×1 (wall re-test payoff)
The 536-ins reach-134 giant listed "blocked on plumbing" since session 8. Re-tested through bank_exemplar after the audit's recover-path fixes (A3d reconcile_tu wiring / A3e gate): BANKED at the `recovered` stage (fb.recover / reconcile_tu resolves the D_80126B58 struct + D_801DA75C fn-ptr conflicts the raw/scoped stages hit). Lazy-isolated into its own jr subseg + jtbl carve. HONEST ATTRIBUTION (R14): this bank is the payoff of the A3 recover path, NOT A9a — it banked at `recovered`, before the `reconciled` (canon_sig_reconcile) stage was reached. A9a's fn-ptr classifier fix is a correctness fix that did NOT independently unblock a bank in the 7-candidate re-test (the reconciled stage failed on func_8015B950's func-conflicts; the rest hit K&R / scalar-typedef / non-ov077 / non-contiguous-carve blockers) — the same null-immediate-banking pattern as A3c/A3d/A3e; its value is protecting all future dispatch-table banking. R22 clean-fleet: 136 passed, 0 failed of 136. Exemplar ×1 (+536 ins); the ×134 family sweep follows. |
||
|
Drew T
|
abbedcee8b |
fix(phase-26a): A9a — canon_sig_reconcile sees fn-ptr dispatch tables (cdecl supplement)
The def-side-wall recovery tool, live on the ×134 economic-engine paths (jtbl_family_bank,
family_sweep --reconcile-raw, bank_exemplar, t7_bank, scope_data_externs, family_remap). The
audit's two HIGH findings: its tu_ambient/visible_above classifier regexes and _reconcile_data's
_DATA_EXTERN_RE have a type class `[\w \*]` that cannot hold a `(`, so they are 100% blind to
fn-ptr DATA decls — `extern void (*D_x[])(void);`, the per-overlay jump-table DISPATCH arrays
jtbl_family_bank exists to bank. A fn-ptr symbol thus landed in NO bucket, fell out of `visible`,
and _reconcile_data block-moved the draft's extern into a guaranteed `conflicting types`.
FIX (R33-adjacent, but SUPPLEMENT not wholesale-replace — the proven regex stays byte-identical):
- tu_ambient / visible_above: keep the v3.2 regex classification UNCHANGED, then supplement from
cdecl.tu_scope with ONLY the fn-ptr symbols the regex drops (_fnptr_data). A full delegation was
tried and rejected — cdecl normalizes funcs formatting (void*→void *, param names, and a real
return-ptr hazard u8*→u8) which rippled 728k value-changes into the callee string-compare; the
audit flagged fn-ptr DATA, not funcs, so the blast radius is held to exactly that.
- _reconcile_data: a fn-ptr pre-pass (_FNPTR_DATA_RE) — visible ⇒ STRIP (ambient serves; a
call-through `D_x[i]()` is decl-INDEPENDENT indirect codegen, so NO access-cast: casting a
fn-ptr would mangle it into `((u8*)D_x)[i]()`, the dormant transform reconcile_tu documents);
not visible ⇒ block-move verbatim. Plus the F2 fix (bail on a `(` in the CODE, not a comment).
- R32 coverage assertion: a fn-ptr data extern may never survive the pass.
PROVEN ADDITIVE (regression baseline .run/audit/a9a_*):
- classifier snapshot over 1683 TUs: REGRESSIONS 0, ADDITIONS 69,798 (all data fn-ptr).
- reconcile() OLD-vs-NEW over 434 real drafts against home TUs: 0 regressions.
- src/ UNTOUCHED (a tool change moves no bytes); the whole-binary byte-gate (G3/P9) remains the
sole arbiter — a wrong reconcile fails to bank, never falsely banks.
|
||
|
Drew T
|
959cc9b7c1 | docs(phase-26a): log A3h — standing-lead harvest (measured, banked +1.3% instr, Bucket X honest-stop) | ||
|
Drew T
|
6139800e6d |
feat(phase-26a): A3h — family_sweep --hseq re-harvest, +2,675 member banks (post-audit-fix)
Re-ran the mechanical h_seq family sweep after the audit tool-fixes (A3–A8) + the 14 new
Bucket-P exemplars regenerated the manifest. family_sweep --hseq --band all templated each
matched exemplar's still-stubbed members per sibling (reloc/immediate remap) and byte-gated:
BANKED 2,675 member-matches / 9,698 failed (22% — the hard residual; the easy bands were
harvested in earlier sessions, so what's left fails the whole-binary gate on jtbl / type /
plumbing, correctly rejected by G3/P9). Skipped: 3,319 pinned-exemplar (×1-only, cc1-crash
siblings, §42e), 212 unresolved-immediates, 137 STRUCT.
223 families with a matched ov077 exemplar contributed. R22 CLEAN-FLEET (make clean +
extract-all + check-all): 136 passed, 0 failed of 136. dedup-check 1840/0 (unchanged — hseq
banks are per-overlay src, not registry).
DELTA:
instr-weighted 67.4% -> 68.1% (+0.7%)
distinct-code 46.8% -> 48.0% (+1.2%, +1,435 unique fns — templated members are distinct
byte-variants, so this moves BOTH metrics unlike propagation)
876 overlay .c files + progress.fleet.md + family-hseq.md
Only 33 of the 567 substantial frontier families have a matched exemplar — the rest need
CRACKING (Phase-26 Task 7, Fable5). Remaining standing lead: Bucket X (~905 absent-from-ov077
byte-exact drafts) via per-overlay gate — next.
|
||
|
Drew T
|
2f38e31e76 |
feat(phase-26a): A3h — propagate 14 fleet-wide byte-exact stubs ×134 (Bucket P)
The standing-lead harvest (A3f/A3g continuation), measured precisely first (R14). Of the
~1,060 still-open byte-exact functions in the backlog:
- Bucket G (67 open in ov_SC01_077): re-gated through the A3e-fixed gate_stage
--no-propagate -> 0 banked. HONEST: A3f already took the bankable 33; the residual is
the known hard classes (jtbl-rodata / register-pins / struct-collision) + stale backlog
rows whose LATEST state is a WAVE mismatch. Correct G3/P9 rejection.
- Bucket P (88 matched in ov077, open in siblings): the clean lead. dedup_propagate --addr
(A3g primitive) skipped 70 as h_exact reach<2 (per-location byte VARIANTS -> family_sweep
territory, not plain propagation) and propagated the 14 genuine PURE fleet families:
5 top (func_80129C40/8012A6D0/80130A18/80131D68/80136DFC) + 9 more; 2 stragglers
dropped all-or-nothing (0x80173A60, 0x8014C568 -> --recover candidates).
Each propagated x~133 (dedup_propagate internal gate: 134 overlays byte-identical).
R22 CLEAN-FLEET (make clean + extract-all + check-all): 136 passed, 0 failed of 136.
dedup-check: 1826 -> 1840 validated, 0 failed | C1 227211/227211.
DELTA:
instr-weighted 66.8% -> 67.4% (+~1,862 member instantiations shipped from C)
distinct-code 46.8% -> 46.8% (flat: propagation adds MEMBERS, not new distinct code)
673 files (671 overlay .c instantiations + engine_core.h) + dedup.us.yaml + progress.fleet.md
Remaining standing lead: the ~72 variant Bucket-P + ~905 Bucket-X (absent from ov077) fns,
all latest-row closeness==0 -> route through family_sweep --hseq (per-sibling remap), next.
|
||
|
Drew T
|
7a3921a3f3 | docs(phase-26a): log A3f+A3g (harvest + propagation) + disk-hygiene note | ||
|
Drew T
|
60e26e07f8 |
feat(phase-26a): A3g — propagate the 3 fleet-wide banks ×134 (bounded, gated, R22-clean)
The 3 of A3f's 33 banks that are shared fleet-wide, stamped across all 134 overlays. Done the way
the earlier run should have been: TARGETED (--addr, not --auto-from), dry-run-sized first
(3 functions × 134 members = ~400 gates, not an unbounded fleet sweep), on a clean tree at HEAD.
func_80130650 (31 ins) · func_80149450 (13 ins) · func_80174684 (9 ins) — each ×134.
dedup_propagate internal gate : 134 overlays byte-identical, 3 groups registered
R22 CLEAN-FLEET (the real proof, not the tool's incremental check that lied during the crash):
make clean + extract-all + check-all -> 136 passed, 0 failed of 136
dedup-check: 1823 -> 1826 validated, 0 failed | C1 coverage 225335/225335
DELTA (reconciles exactly):
functions byte-identical 284,559 -> 284,958 (+399 = 3 fns × 133 other overlays)
instr-weighted 66.7% -> 66.8% (+13,167 shipped .text instructions)
distinct-code 46.8% -> 46.8% (flat: propagation adds MEMBERS, not new distinct
code — the 3 bodies were counted at A3f)
403 src files (3 ×134 instantiations + engine_core.h) + config/dedup.us.yaml
The other 30 of A3f's 33 are overlay-unique (×1) and need no propagation. The larger prize remains
the ~310 byte-exact stubs in the OTHER overlays (A3e), not yet attempted.
|
||
|
Drew T
|
8ffcf9646e |
feat(phase-26a): A3f — 33 functions banked that the project had written off as compiler walls
The payoff of A3e, byte-verified. These 33 sat in the backlog at closeness==0 -- match_one said
their bodies were BYTE-EXACT -- and the whole-binary gate rejected them, so they were logged as
`near`/`failed`, i.e. AS MATCHING PROBLEMS, and filed as intrinsic compiler residuals.
They were not hard. They were UNREACHABLE. gate_stage passed `--src src/<ov>/<ov>.c`
unconditionally, which restricts the byte-gate to ONE translation unit -- and every one of these
functions has its stub in a SPLIT TU. Look at where they landed:
src/ov_SC01_077/ov_SC01_077_a.c
src/ov_SC01_077/ov_SC01_077_after.c
src/ov_SC01_077/ov_SC01_077_jr_8012ACE0.c
src/ov_SC01_077/ov_SC01_077_jr_8015AE2C.c
src/ov_SC01_077/ov_SC01_077_jr_8016AB6C.c
src/ov_SC01_077/ov_SC01_077_jr_801734BC.c
src/ov_SC01_077/ov_SC01_077_jr_80178D40.c
src/ov_SC01_077/ov_SC01_077_jr_80182268.c
8 files. SEVEN of them are _jr_/_a/_after carves. NOT ONE is the main .c -- the only file the gate
was ever allowed to look at.
gate: 63 drafts -> banked 33, near 30, FAILED 0 (--no-propagate; the gate and the propagation
are different jobs, and letting an experiment tow an unbounded fleet-wide propagation is
what broke the tree an hour ago)
R22 CLEAN-FLEET: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
dedup-check: 1823 validated, 0 failed | C1 coverage 224933/224933
METRICS, ×1, HONESTLY (no propagation yet -- the multiplier is still ahead):
functions byte-identical 284,526 -> 284,559 (+33)
instructions 8,470,381 -> 8,471,912 (+1,531)
fn-count % 82.79% -> 82.80%
instr-weighted % 66.7% -> 66.7% (flat: ×1 banks do not move the headline)
WHAT THIS MEASURES, beyond the 33: the backlog holds 1,588 entries at closeness==0. 1,215 have been
banked since by other paths. 373 ARE STILL OPEN STUBS WITH BYTE-EXACT BODIES. 63 of them were in
ov_SC01_077 and 33 banked -- a 52% rate on functions the ledger calls unrecoverable. The other 310
are spread across the remaining overlays: same class, same fix, not yet attempted.
Cookbook §51g LAW 11 -- a fix is not landed until its caller stops overriding it. And the reason
this hid for 26 phases, which belongs in the posterity doc: A TOOL THAT CANNOT BANK A FUNCTION IS
INDISTINGUISHABLE, IN EVERY LOG THIS PROJECT KEEPS, FROM A FUNCTION THAT CANNOT BE BANKED.
|
||
|
Drew T
|
b89fcc2edc |
fix(phase-26a): A3e — gate_stage pinned the byte-gate back to 4.9%, OF A3'S OWN FIX
THE WORST DEFECT IN THE AUDIT IS NOT IN A SCANNER. It is one default argument in the CALLER of a
scanner we had already fixed.
# tools/gate_stage.py:315
summary = run_gate(a.drafts, binary=b, src=a.src or f"src/{b}/{b}.c", ...) # ALWAYS the main .c
`src` RESTRICTS the byte-gate to ONE translation unit, and _gate1 does `if src: cmd += ["--src", src]`
-- always truthy. A3 had just taught harvest_verify to DERIVE each draft's home TU *when --src is
omitted*, lifting the byte-gate's reach from 4.9% to 100%. gate_stage NEVER OMITS IT. The fix was
neutralised by its own caller's default, and the PRIMARY BANKING PATH -- every wave, the grinder, the
orchestrator, bulk_harvest -- remained structurally unable to bank 250 of ov_SC01_077's 263 stubs.
WHY IT SURVIVED 26 PHASES: harvest_verify cannot splice a draft whose stub is not in the TU it was
pointed at, so the draft never verifies -- and is then logged as near/failed, i.e. AS A MATCHING
PROBLEM. The wave reports a poor close-rate; the function goes to the backlog as a compiler residual.
A tool that CANNOT bank a function is indistinguishable, in every log this project keeps,
from a function that CANNOT BE banked.
PROOF, same draft / same gate / same second: gate_stage rejected func_80129C40; harvest_verify run
directly (no --src) VERIFIED it byte-identical and banked it.
AND A COUNTING BUG THAT HID THE HIDING (gate_stage:261): when match_one says MATCH but the whole-binary
gate rejects, the record is logged status="near" and THE COUNTER IS NEVER INCREMENTED. A 63-draft run
printed `banked 0, near 0, failed 0` -- three zeros that do not sum to 63 -- for phases. Nobody ever
added them up. (The number was not wrong. It was ABSENT.)
ALSO FIXED, sig_unify (the same disease, one level down): it SILENTLY DROPPED 190 of 196 drafts (97%).
`cur_stubs` was read from the main .c (13 of 263 stubs), so any draft whose stub lives in a _jr_ carve
hit `if fn not in cur_stubs: continue` -- dropped BEFORE THE WRITE: never copied to --out, never gated,
never logged, while the summary printed "drafts unified: 6" and read like success. THIS IS GATE_STAGE'S
STAGE-2 RECOVERY -- the pass whose whole job is to rescue the stage-1 failures -- and it has been a
no-op for nearly every draft it was meant to save. Now: TU derived per draft (corpus.stubs), canon
derived from cdecl.tu_scope (cpp -- macro-injected decls finally visible), and _keep() so an
already-acceptable decl is left alone (the §19 "sig_unify regresses canonical drafts" failure mode).
Reach: 6 -> 196 drafts; callee-externs rewritten 2 -> 90; own def-sig 2 -> 86.
MEASURED, all three consumers migrated (196 never-banked drafts):
near 5 -> 116 failed 190 -> 17
=> 173 of 190 "failures" were PLUMBING, not codegen: now compiling and SCORED instead of invisible.
THE PRIZE (measured, not claimed): the backlog holds 1,588 entries at closeness==0 -- body byte-exact
per match_one, whole-binary gate rejected. 1,215 have been banked since by other paths. 373 ARE STILL
OPEN STUBS WHOSE BODIES ARE ALREADY BYTE-EXACT, sitting in a ledger that calls them unrecoverable.
⚠ THE HARVEST ITSELF IS NOT IN THIS COMMIT, AND IS NOT CLAIMED (P9). Gating the 63 ov_SC01_077 ones
dragged `dedup_propagate --auto-from --recover` behind it; it ran >1h and hit its timeout -- its
first-ever run over the FULL corpus (A6/A7 unblocked the 407 files it could never see). It MUTATES THE
TREE BEFORE IT GATES, so the kill left 859 files + engine_core.h (+544 lines) written and UN-GATED with
the registry never updated. R22 on that tree: 44 passed / 92 FAILED -> `git checkout -- src/ config/`,
fleet restored to 136/136. Nothing lost (H4: the tree was clean, so the revert was one command).
Two real lessons, recorded: dedup_propagate is NOT crash-safe and must never run under a timeout it can
hit; and a 63-draft experiment must not drag an unbounded fleet-wide propagation behind it.
R22 clean-fleet after revert: 136 passed, 0 failed of 136. src/ and config/ clean.
cookbook §51g LAW 11: A FIX IS NOT LANDED UNTIL ITS CALLER STOPS OVERRIDING IT. After fixing a
scanner, grep every call site and ask whether a caller's default re-disables it. An audit that stops
at the callee is half an audit.
|
||
|
Drew T
|
4aae5e7589 |
fix(phase-26a): A3d — retire the fleet-majority oracle: it was WRONG for the TU 16% of the time, on both banking paths
R33 applied to the worst finding in the audit: this oracle was not fixed, it was RETIRED.
reconcile_decls asks "what does the FLEET call this symbol?"
C asks "what does THIS TRANSLATION UNIT declare?"
The engine is loosely typed -- the same address is legitimately declared with incompatible types in
different overlays -- so a single fleet-wide answer is WRONG FOR SOME TU BY CONSTRUCTION. And it is
worse than a silent skip: it writes an ACTIVELY WRONG declaration into the draft, which then
collides with the very TU it was meant to conform to.
MEASURED across ov_SC01_077's 12 TUs, against what cpp says each TU really declares:
the fleet oracle AGREES with the TU ................ 2883
the fleet oracle CONFLICTS with it (cc1 REJECTS) .. 548 <- 16%
the TU declares it, the oracle has NO answer ...... 357
and it was LIVE ON BOTH BANKING PATHS:
* gate_stage -- rewrote 60 of 196 drafts in the current batch
* jtbl_family_bank -- EVERY SIBLING of the ×134 family sweep, the project's economic engine.
A poisoned decl means that sibling silently does not bank, and the loss is invisible: the sweep
simply reports a smaller number. The irony is exact -- that function's own docstring already
knew the conflicting symbols are PER-OVERLAY, which is precisely why a FLEET oracle could never
have been right.
reconcile_tu.py (written in Phase 26 but NEVER WIRED) now supersedes it, rebuilt on cdecl:
* ask cpp what the TU declares (macro-injected DEFINE_func_* externs included -- a raw scan
cannot see them, §8c / §51g LAW 7);
* ask cc1 whether the draft's decl can coexist (cdecl.compatible, validated against the real
gcc-2.7.2 front end on 1,485 live pairs -- NOT the C standard, NOT modern gcc; §51g LAW 9);
* NOT declared -> leave the draft alone (its extern types are load-bearing: %lo-folding, access
width, alignment); compatible -> nothing; CONFLICTING -> the TU wins + cast at every USE so the
draft's intended access survives byte-for-byte;
* derives WHICH TU from corpus.stubs() rather than a hand-passed --src-file (§51g LAW 10).
* handles the fn-ptr kind NATIVELY -- which is why it supersedes rather than patches: teaching
reconcile_decls' parser to see `extern void (*D_x[])(void);` would have ARMED its fn-ptr-blind
data_access_subs to rewrite a call-through `D_x[i]()` into `((u8 *)D_x)[i]()`. Fixing the regex
would have detonated a dormant bug.
AND THE NULL RESULT, AGAIN, REPORTED AS SUCH (P9/R14): on the 196 never-banked historical drafts the
new oracle banks EXACTLY AS MANY AS THE OLD ONE -- zero. That tail fails on CODEGEN, not on decl
plumbing. The two disagree on 45 of 196 drafts and the outcome does not move. This is a CORRECTNESS
fix (548 wrong declarations removed from two live pipelines, protecting all FUTURE drafts and every
future family sweep), not a banking win, and it is not being sold as one. Three nulls in one session.
reconcile_decls.py is kept as EVIDENCE, marked RETIRED, with no live caller.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
src/ untouched (0 changes) reconcile_tu: 0 coverage defects over 196 drafts
NOTE: the family-sweep path gets its real exercise at Task 8 -- watch the per-sibling bank rate.
|
||
|
Drew T
|
f4502f11bb |
fix(phase-26a): A3c — the recovery passes were reconciling 95% of drafts against the WRONG TU
FIRST CONSUMER MIGRATION onto the cdecl oracle — and the compiler taught me two things I had
wrong, one of which reopens a wall that has been closed since Phase 15.
1. cdecl.compatible() — "will cc1 accept these two declarations of one name?"
The predicate four tools each half-implement and get wrong: norm_sig / _norm_type collapse the
int family to ONE token, so a SIGNEDNESS change reads as "already compatible" and gets no
rewrite -- while cc1 REJECTS that redeclaration. Right about codegen, wrong about the front end,
which never reaches codegen.
2. THE ADJUDICATOR MUST BE THE COMPILER THAT COMPILES YOUR CODE (cookbook §51g LAW 9).
I wrote the rules from the C standard, then let a compiler judge. It contradicted me -- and then
the RIGHT compiler contradicted the first one. Three different answers:
declarations in one TU | standard | modern gcc | gcc-2.7.2 cc1
typedef int X; twice | error | ACCEPTS | ERROR
extern u16 X; + volatile u16 X| error | error | ACCEPTS
void X(s16); then void X(); | error | error | ACCEPTS
void X(); then void X(s16)| error | error | ERROR
--compat now adjudicates with tools/bin/gcc-2.7.2-psx/cc1, the front end that actually
arbitrates the build: 1,485/1,485 live corpus pairs agree, 0 disagree, 0 skipped.
3. THE PRIZE: the Phase-15 narrow-param wall rests on a false premise.
The no-prototype rule is ORDER-DEPENDENT. `void X(s16); void X();` COMPILES; only the reverse
fails. Phase 15 closed "the 159 arity/narrow-param conflicts" as "no clean deterministic fix --
it is simply C's default-promotion rule". cc1 does not enforce that rule in the direction the
wall assumed. Four three-line probes, 90 seconds, zero tokens. -> A10 RE-TEST TARGET.
Probe the compiler for FACTS; read its source only for LEVERS; byte-validate both. (We read
gcc-papermario for five phases believing it was 2.7.2. It was 2.8.1.)
4. THE MIGRATION: cast_call_sites canonicalized 95.1% of drafts against a TU that would never
compile them. `--src-file` is an OPTIONAL HAND-PASSED flag defaulting to src/<ov>/<ov>.c, and no
caller knows about the Phase-26 _jr_<ADDR> carves: ov_SC01_077 has 263 open stubs across 12 TUs
and only 13 are in the main .c -- while harvest_verify (A3) correctly splices into the real one.
Now DERIVED from corpus.stubs() (the INCLUDE_ASM line is self-describing), with the canonical map
derived from cdecl.tu_scope() (cpp -- so macro-injected DEFINE_func_* decls are finally visible).
Callee-conflict repair reach: 8 -> 58 of 196 drafts (7x).
5. AND THE NULL RESULT, REPORTED AS SUCH (P9/R14). Those 58 banked ZERO functions. The historical
draft tail fails on CODEGEN, not plumbing -- func_801387B8, which the audit blames on a single
unparsed `[4]`, is really 67/100 instructions off with a $s0/$s1 swap (that claim does not
reproduce on today's tree). The real gain is narrower and still worth having: 52 drafts moved
from "won't compile" to "compiles, N instructions off" -- from an INVISIBLE failure that reads as
a compiler wall into a SCORED near-miss the permuter and the §47/§48 dials can act on. That is
the audit's thesis, not a bank. THREE times in one session a confirmed mechanism produced a null
consequence.
Also: my own new audit printed "ALL ORACLES GREEN" while silently skipping 100% of its corpus (a
missing -Isrc). The exact bug class, in the tool written to hunt it. An unadjudicable check is not
a passed check.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
src/ untouched (0 changes) make audit-cdecl: green --compat: 1485/1485
NEXT: sig_unify + reconcile_decls carry the SAME wrong-TU bug (same --src-file flag).
|
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|
||
|
Drew T
|
c7772bc452 |
docs(phase-26a): SESSION-9 CLOSE — cookbook §51 (the tooling-integrity laws) + handoff
R30/R16: the context-dependent artifacts, written while the context is live.
cookbook §51 — the SILENT SKIP: the bug class, why the byte-gate cannot see it, the
over-approximating-detector method, and FOUR LAWS:
1. Derive, don't re-derive — the best outcome is a DELETED SCANNER (28 findings -> one
derived oracle + ~10 deleted scanners). A derived fact cannot rot; a hand-maintained
copy of it is a liability that grows with every structural change.
2. Assert your COVERAGE, not merely your correctness. *** A LOUD FAILURE THAT NOBODY
COUNTS IS EXACTLY AS INVISIBLE AS A SILENT ONE *** — build_engine_types printed
'[overlap] handle manually' every single time for four phases while dead on 81% of its
own corpus. This CORRECTS the first draft of R32 ('fail loud'), which was not enough.
3. When an oracle is structurally blind to a class of error, add a SECOND ORACLE THAT CAN
DISAGREE WITH IT — not a better assertion inside it. We had two all along and never made
them argue. (And scope the comparison to where the second oracle is genuinely independent:
the same check run outside its domain reports 914 slices when the truth is 193.)
4. A rule that needs a human to remember it is not a gate. Make it structural.
+ the FALSE-WALL PIPELINE (a silent skip -> a wasted draft -> a backlog 'matching failure'
-> reserved_walls() PERMANENTLY blacklists a function that was never attempted), and a
checklist for any new corpus-scanning tool.
CURRENT_PHASE: session-9 handoff — what is done, what remains (each with its spec on disk),
and the R32-corrected / R33 / R34-new rule candidates for P10 ratification.
|
||
|
Drew T
|
2086b15b48 |
fix(phase-26a): A8 — jr_isolate_all silently dropped 683 prototypes: a LATENT BYTE-CHANGER
_file_scope_decls() hoists a region's file-scope decls into the carried layer. Its _SAFE_TYPE guard
only ever whitelisted BUILTIN base types — so a decl naming a carried file-local type was recognised
by _HOIST_RE as hoistable and then SILENTLY DROPPED.
THE COMMENT ON _SAFE_TYPE DESCRIBED A FIX THAT WAS NEVER APPLIED TO THE CODE. Verbatim: "one naming a
FILE-LOCAL type is only safe once that type is carried too — which file_scope_types() now does, so such
decls ride along after their typedef." The predicate never implemented it.
MEASURED (audit: 4,040 fleet-wide; independently re-measured here over 4 overlays: 189 drops, 32 of
them function PROTOTYPES):
* The 3,357 dropped DATA externs are LOUD — an undeclared identifier is a compile error.
* The 683 dropped function PROTOTYPES are NOT. In C89 an undeclared function is implicitly `int f()`,
so the TU still COMPILES — with the wrong return type and lost pointer-ness. And this project has
BYTE-PROVEN that the return type drives codegen (cookbook: "schedule — delay-slot fill via void
return type"; ov_SC01_077_after.c carries an `extern int`->`extern void` flip described as
byte-neutral precisely because the return type moves the delay slot).
=> A DROPPED PROTOTYPE IS A SILENT BYTE-CHANGER, armed to fire on the next carve.
Today's split is green only because the source redundantly re-declares externs per fn-group, so most
items happen to carry their own decl. That is luck, not design.
Two of the rejected base types were not even file-local: `uint` (139 drops) and `code_fn` (21) are
DEFINED IN src/shared/engine_types.h, which engine_core.h pulls into every region — the predicate was
rejecting INCLUDE-PROVIDED types it had no reason to reject. `volatile` (3) fell off because the
qualifier group had `const` but not `volatile`.
FIX: implement what the comment promised. A decl is hoistable if its base type is a builtin, OR is
carried by this layer's own file_scope_types, OR is provided by the shared headers (108 type names
parsed from engine_types.h + common.h — including FN-PTR typedefs, whose name sits inside the parens
and which every name-before-';' pattern misses; those were exactly the 5 residual drops).
COVERAGE ASSERTION (R32): a line _HOIST_RE recognises as hoistable but that cannot be placed is now a
HARD FAILURE with the base-type histogram printed, not a silent no-op. Verified safe: 0 residual drops
across 5 overlays. This one check would have surfaced all 4,040 the day the first split shipped.
No build impact (the isolator runs only when carving); --dry-run clean; tree unchanged.
|
||
|
Drew T
|
82d79e7a32 |
fix(phase-26a): A6/A7 — the family engine could not see half its corpus; 17 fns banked x134 free
R22: check-all 136 PASSED / 0 FAILED. dedup-check 1823 validated / 0 failed (C1 coverage 224,933/224,933).
Fleet instr-weighted 66.5% -> 66.7%.
=== dedup_propagate: it was blind to HALF the corpus ===
overlay_files() used a hardcoded suffix allowlist ("_a","_o0","_o0b","_after") that predated the
Phase-26 jr carves -> 404 of the fleet's 811 overlay .c. The 407-file gap held 36,135 INCLUDE_ASM stubs
and ~32,000 inline defs, and overlay_files gates ALL of dedup_propagate (source_text / find_site /
apply_plan / struct_check / reconcile_caller_extern). Now a GLOB — never an allowlist, because the NEXT
split family would re-open it. The asm_subdir is always the file stem, an invariant the old four entries
already satisfied.
find_site's def-detector required the signature line to END in ')' and the next non-blank line to START
with '{'. It therefore silently dropped THREE shapes: K&R definitions (`s32 f(arg0)` / `s32 arg0;` / `{`),
multi-line signatures, and single-line bodies. K&R is the project's house style for exactly the biggest,
highest-reach functions — func_8015AE2C (562 ins), func_80166994, func_80133CD4, func_8015A3C8 — and they
live in the _jr_* files overlay_files could not even open. Fixing either alone would have been useless:
the glob exposes the files, and find_site would still drop their biggest prizes. Both fixed together.
* The signature's closing paren is now found by a real paren-walk, not line.count() or split(')')[-1]:
a single-line body containing a call (`void f(int a){ g(a); }`) has balanced parens of its own, so
both shortcuts land on the WRONG paren and then misread the body's ';' as a prototype terminator.
* AGREEMENT ASSERTION (the audit's): find_site vs family_remap.extract_unit -> 701 agree / 0 disagree.
Negative controls hold (a prototype+call is rejected; a 1-line body with a call is a def).
=== THE HARVEST (free work, byte-gated) ===
--auto-from ov_SC01_077 now nominates what it could never see: 20 planned, 17 propagated x134, 3 dropped
as cross-overlay stragglers. 134 overlays rebuilt BYTE-IDENTICAL; 17 new dedup groups.
Includes ALL FOUR functions A1 caught the registry lying about (func_80128ED8 / 8012C098 / 8012C0EC /
8012C750): 0 stubs remaining, real shared macros. THE LOOP CLOSES — A1 found the lie, and THIS is the
bug that had made it true (3 of the 4 are defined in ov_SC01_077_jr_8012ACE0.c, which the allowlist could
not open, so the propagation never ran and dedup_integrate greenlit the result).
=== family_remap: 96 PHANTOM exemplars -> 0 ===
extract_unit globbed only src/<ov>/<ov>*.c, so a function matched via a SHARED body had no source form
and read as NOT MATCHED. 93-96 of 218 h_seq "matched" exemplars were phantom, carrying 2,157 candidate
members of which 1,834 are still-stubbed, PURE/IMM-clean, symbol_map-clean and unpinned — staged and
gated today, dropped before the first build then. It is now TOTAL over BOTH shared-body mechanisms:
(1) the DEFINE_func_<ADDR>() macro — reconstructed as the exact INVERSE of dedup_propagate.make_macro
(derived from the generator, not re-guessed from the text);
(2) a DIRECT definition in a shared header, #included per overlay — the whale (func_80144B9C, 770 ins,
-O0), which the registry explicitly records as "NOT a DEFINE_ macro".
CENSUS: 216 matched exemplars, 216 real, 0 PHANTOM.
symbol_map named the symbol by HOW IT WAS LOADED, not by WHAT IT IS: reloc_targets labels every lui/%lo
pair "data", and a FUNCTION's address taken via lui/%lo (an address-taken callback) is exactly that shape
(splat's own .s: %lo(func_8017E1D4), 7 occurrences). The map got a D_<ADDR> key while the C writes
func_<ADDR>, so the word-bounded substitution matched NOTHING and silently no-op'd — the sibling kept the
EXEMPLAR's function pointer and the loss was booked as a BYTE failure, indistinguishable from a compiler
wall. Now emits both keys (addresses are unique; the pass is simultaneous, so the extra key is free).
gather_externs was line-oriented, so a WRAPPED comma extern was invisible in both directions (the first
line has no ';', the continuation has no `extern`). ov_SC01_077.c:271-272 declares NINE symbols that way,
and the exemplar referencing them (func_8013D178) is a 133-member family — every sibling was staged with
NO declaration, failed to compile, and bisect-stormed its whole gate group. Now statement-oriented, and
an unresolved symbol is REPORTED, never silently dropped.
=== family_sweep.stub_map / build_engine_types ===
stub_map: func_-only -> a curated-name stub read as "already matched" -> phantom exemplar. Now corpus-derived.
build_engine_types hard-exited on 1,070 of 1,470 type-bearing overlay .c (73%; the audit measured 573/709
= 81% on its narrower set) because 1,929 TAGGED-struct typedefs tripped a guard whose own comment asserts
"our source has only ANONYMOUS-struct typedefs" — true in Phase 20, false since the harvest agents started
writing tagged structs. inject_capped_externs routes every type-bearing body HERE as the type-heavy tail's
ONLY sanctioned unblocker, so the tail's unblocker could not run on the corpus the tail lives in.
A contained def (the typedef's span encloses the body) is liftable — it just must not be counted twice;
only a PARTIAL overlap is malformed. Verified on a file that used to hard-exit: 5 tagged typedefs folded +
forward-declared, 46 types written, exit 0.
** AND THE SHARPEST LESSON IN THE AUDIT: this one was never silent. It printed "[overlap] ... handle
manually" every single time. But the message reads like a rare edge case rather than a four-fifths
coverage failure, so nobody ever COUNTED it. A loud failure that nobody counts is exactly as
invisible as a silent one. R32 must be "assert your coverage", not merely "fail loud". **
R14 self-catches, recorded because I hit both while fixing them: my first shared-header scan read a macro
body's `extern void f(void); \` as a DEFINITION (the trailing continuation means the line does not end in
';', so the decl guard never fired) — the exact bug fixed at commit:0552, reintroduced by me and caught only
because the whale resolved from the WRONG file. Column-0 anchoring fixes it by construction. And my
phantom census returned 0/0 twice because I guessed the manifest schema instead of reading it.
|
||
|
Drew T
|
a2a507d079 | docs(phase-26a): A4/A5 + the stale-object false-pass hole recorded | ||
|
Drew T
|
af2f40d153 |
fix(phase-26a): A4/A5 — 193 unmatchable slices dissolved; the closeness oracle stops lying
R22 CLEAN-FLEET: make clean -> extract 136 -> build 136 -> check-all = 136 PASSED, 0 FAILED.
make audit-corpus: 0 PHANTOM + 0 TRUNCATED (was 193).
=== A4: a CORPUS defect the byte-gate could never have caught ===
config/symbols.us.txt:981 declared `listCdBuffer = 0x80180000` — a correct Phase-3 name for MAIN's
LIST.CD RAM buffer. But that address is OUTSIDE main's image and INSIDE the overlay slot, and every
overlay's splat config stacks symbols.us.txt. High RAM is REUSED: an address that is a buffer to main
is live CODE to an overlay. So splat saw a symbol boundary mid-code and, across 97 of 134 overlays:
* CUT 97 REAL FUNCTIONS IN HALF (a head ending on a `lui`, no return), and
* INVENTED 96 PHANTOM ONES (a tail beginning by reading the assembler temp $at).
193 slices NOBODY COULD EVER MATCH — not "hard", not "a compiler wall": unmatchable by construction.
They sat in the harvest queue as ordinary work, so agents would burn on them forever and the failures
would be filed as intrinsic compiler residuals.
The phantom listCdBuffer.s in ov_SC01_005 literally begins:
lw $ra, 0x10($sp) / addiu $sp, $sp, 0x18 / jr $ra
splat cut a function immediately before its EPILOGUE and called the epilogue a function.
AND IT HAD ALREADY CONTAMINATED REAL WORK: in ov_SC03_031 the cut landed where the epilogue was
exactly `jr $ra; nop`, so the Phase-26 x134 sweep innocently BANKED the phantom as
`void listCdBuffer(void) {}` — byte-correct, gate-green, entirely fictitious — while leaving
func_8017FFC4 permanently unmatchable. Removed.
WHY NO GATE CAUGHT IT, AND WHY THAT IS THE POINT: INCLUDE_ASM pastes the two .s halves back VERBATIM
in original order, so the image is byte-identical either way. The byte-gate was green the whole time
and always would have been. It is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle. No
assertion added INSIDE it could ever have found this. What found it was a SECOND, INDEPENDENT oracle:
tools/sig_image.py derives boundaries from the ORIGINAL bytes without splat, and DISAGREED with the
corpus (58,524/58,621 agreement with spimdisasm; correct on all 97 disagreements).
=> When one oracle is structurally blind to a class of error, the answer is not a better assertion
inside it. It is a SECOND ORACLE THAT CAN DISAGREE WITH IT. (`make audit-corpus` is now that.)
THE RULE (the mirror of R13/R15, never written down): a symbol whose address falls inside ANOTHER
binary's vram window must never enter that binary's symbol stack.
FIX: config/symbols.us.ram.txt — main-scoped symbols outside main's image — stacked ONLY by
config/splat.us.exe.yaml. Main keeps the name it needs (10 %hi / 11 %lo refs; 143dbb89 byte-identical);
the overlays never see it. Exactly one symbol was in scope fleet-wide; the resident window was clean.
AND A REAL FUNCTION THE ACCIDENT WAS HIDING: in ov_SC01_084 / ov_SC02_041 / ov_SC03_094 / ov_SC06_008
there IS a genuine function at 0x80180000 (111 / 35 / 28 / 74 ins), reachable ONLY via a fn-pointer
table (.word func_80180000) and never by `jal` — so splat cannot find it and needs the boundary
DECLARED. listCdBuffer had been supplying it by luck. Now declared honestly, per-overlay, in
config/symbols.<ov>.txt — exactly where R13/R15 says an overlay-scoped symbol belongs.
=== A5: the closeness oracle every crack agent trusts was lying on 155 functions ===
masked_diff._reloc_kind() knew 26/HI16/LO16. An over-approximating sweep of every reloc objdump emits
across all 3,367 build objects found FOUR: R_MIPS_26, HI16, LO16 — and R_MIPS_PC16 (211). PC16 fell
through to a FULL-WORD compare, but the object holds an UNRESOLVED PLACEHOLDER in the branch
displacement, so that compare can NEVER succeed.
DECISIVE TEST (derived from the invariant, not from reading the regex): INCLUDE_ASM pastes the
ORIGINAL asm, so for every stub diff_object_s() MUST be 0. Measured, coverage-asserted:
2,741 functions scored — old mask: 150 LIES; PC16 masked: 4 LIES.
(The 4 survivors are the separate length-delta defect.) A phantom non-zero sends an agent to grind at
a wall that is not there, and the wasted attempt is then booked as a MATCHING failure, feeding
reserved_walls() and PERMANENTLY BLACKLISTING a function that was never broken.
=== NEW FINDING (found by cutting the R22 corner): a STALE OBJECT CAN PRODUCE A FALSE PASS ===
`.o <- .s` is not a dependency make can see: assembly arrives via INCLUDE_ASM, expanded to a `.include`
consumed by maspsx/as AFTER cpp, while -MMD tracks headers only. Re-extract, build incrementally, and
make links a STALE object. This is not merely slow — INCLUDE_ASM pastes the ORIGINAL bytes, so a stale
object still yields the original image: SHA1 GOES GREEN while the split just changed is never exercised.
A broken config change can be "verified" by an incremental build. Live proof: 8 of 136 binaries linked
stale objects here; they failed LOUDLY ONLY BY LUCK (the dead symbol was an undefined reference) — a
merely-different-but-valid split would have gone green on all 136.
R22/H3 already legislate this, and I broke them. But a rule that needs a human to remember it is not a
gate. FIX: `extract` now invalidates the objects that include what it just rewrote (main's are top-level,
so -maxdepth 1 — verified it cannot clobber the other 1,605 objects). Structural, not advisory.
R14 self-catch, recorded: my first A5 test passed `fn=` to diff_object_s(), which takes two args; the
TypeError was swallowed by my own `except Exception: continue` and it reported 0 scored / 0 lies. I
wrote the exact bug I was auditing, inside the test for it. Caught only because 0 looked wrong. The
test now asserts its own coverage.
|
||
|
Drew T
|
9794b13ed2 |
fix(phase-26a): A3 — the endgame plan was 2.8x too big; the matched set is now DERIVED
docs/family-manifest.md is the document the whole Phase-25/26 structural-family endgame was planned
from. Its matched-set oracle asked ov_SC01_077 ALONE: matched := {h_exact of that one overlay's
non-stub fns} | dedup hashes. So a function ABSENT from that overlay — or stubbed there but matched
in the other 133 — came out "unmatched" and was ranked as live work.
advertised real (derived)
multi-member families 2,758 -> 1,495
"hidden leverage" 11.0 MB -> 3.9 MB
matched-free lever 235/5.9 MB -> 57/1.0 MB
7.1 MB of the advertised leverage was DEAD WORK. And because `instances` counted every overlay
carrying a function — including the ones where it was already banked — the byte-weight RANKING (the
file's entire purpose: "draft these first") was sorted mostly on already-finished code, with the
real targets buried underneath. The A2 audit predicted "true frontier: 1,475 families / 3.9 MB";
derived independently here it is 1,495 / 3.9 MB.
R33: the invariant answers this with no oracle at all —
an h_exact class is WORK iff at least ONE of its instances is still an INCLUDE_ASM stub.
That also makes the dedup-hash union redundant (a dedup-shared member is by definition not a stub),
so the `hash:` regex over config/dedup.us.yaml is DELETED. `instances` now counts only the members
still to bank, so the leverage is the real x-N.
family_hseq: stub scan -> corpus (+100 curated-name stubs the func_-only regex could not see; they
had made 3 still-stubbed functions look like MATCHED exemplars, which every sweep then re-nominates,
produces nothing from, and books as a silent skip). Its hardcoded "expect ~663/~186/~1.85M" self-check
was a stale 2026-07-11 snapshot — 38 banking commits have landed since — and is now labelled a
point-in-time reference, not an invariant. (Verified my change can only GROW the frontier: +100 stubs.)
census_conflict_callees: scoped to src/<ov>/<ov>.c alone, so it saw 13 of 264 stubs and reported
"wave scope: 2 still-stub" when the truth is 57 — every downstream percentage computed against a
denominator 96% too small. Now 0/57 (the audit's exact figure). Its 0-conflict answer was right BY
LUCK; it is now right for a reason. MARKED FOR DELETION (R33): it re-derives from C text what
reconcile_tu.py answers from the build, and its parse holes fail in the UNSAFE direction (an unknown
callee is silently bucketed "conflict-free"). Delete once reconcile_tu is wired into its only consumer.
R14 near-miss, recorded: my first census patch handed collect_stubs() a set of NAMES where it wanted
ADDRESSES, so the membership test was always false and it printed 0/0. Caught only because 0
contradicted the audit's expected 57. A scanner that returns 0 is indistinguishable from a scanner
that found nothing — which is the entire thesis of this audit, and it very nearly bit me while
fixing it.
|
||
|
Drew T
|
3be734bde8 |
docs(phase-26a): A3 progress — the oracle is built; 4 of ~10 scanners deleted
corpus.py + build_fuel_manifest + wave_targets + harvest_verify/gate_stage landed. Remaining: family_manifest/family_hseq (62% of the endgame plan is phantom targets), DELETE census_conflict_callees (R33), exemplar_miner/difficulty, jr_isolate_all. |
||
|
Drew T
|
70a99aa9f7 |
fix(phase-26a): A3 — the byte-gate could only see ONE translation unit (96.6% of stubs unreachable)
harvest_verify is the sole arbiter (G3/P9) and has never accepted a wrong match. It also could not
REACH most of the work: it scanned the single file the caller passed as --src and silently dropped
every draft whose stub lived elsewhere. An overlay's source spans up to 14 .c files, so:
open overlay stubs it could not see : 56,742 of 58,717 (96.6%)
ov_SC01_077 reachable by the gate : 13 of 264 (4.9%) -> 264 of 264 (100%)
THREE of the six callers passed no --src at all (orchestrator.py, grinder.py, idiom_hunt.py) and so
inherited gate_stage's `src = src or f"src/{binary}/{binary}.c"` default. For grinder.py that means
1,290 of its own 1,298 QUEUED FUNCTIONS COULD NEVER BANK, however good the permuter's output was.
=> Phase-22's "the permuter's fuel is exhausted" was never a safe conclusion. Re-test (A12).
gate_stage knew the right answer and then handed the gate the wrong file: its negative control
ALREADY globs every split .c to build bin_stubs. The default is now removed; --src is passed only
when a caller deliberately restricts the gate to one TU.
WHAT CHANGED, PRECISELY: only the SPLICE LOCATION. Each draft is now spliced into whichever TU
actually holds its stub, derived from tools/corpus.py. Every TU links into the same image, so ONE
`make build BINARY=<bin>` still gates them all — correct AND strictly fewer builds than the
per-split re-gate it replaces.
SAFETY (this is the byte-gate, so the argument is explicit): the VERDICT is untouched — `make build`
+ SHA1 == the locked hash. INCLUDE_ASM pastes the ORIGINAL assembly, so a wrong draft always changes
the bytes and always fails SHA1. A bug in the splice can therefore make the tool FAIL TO BANK; it
CANNOT make it falsely bank. The failure mode is conservative by construction.
VERIFIED end-to-end (2 real builds, tree clean before and after):
* discovery: 264 live stubs across 12 TUs (was: only those in the single --src file)
* IDENTITY known-answer test: 3 drafts whose stubs live in THREE DIFFERENT split TUs
(_jr_801734BC, _after, _jr_8012ACE0) — all discovered, spliced into their own files, built,
SHA-matched, committed, restored. Final SHA d19c9580 BYTE-IDENTICAL. Under the old code all
three were silently dropped as "not stubbed".
* `git checkout -- src/` recovers, exactly as the docstring promises.
Also derived rather than defaulted: --good-sha now reads config/check.<bin>.sha (a caller that
passed --binary but forgot --good-sha used to gate an overlay against RESIDENT's SHA), and
match_one_closeness resolves the asm subdir PER FUNCTION — one subdir for a whole batch is the same
single-TU bug, and pointing match_one at the wrong one scores a draft against a DIFFERENT function's
asm, producing a phantom non-zero closeness that lands in the backlog and feeds reserved_walls().
No committed source or config changed, so no build artifact can have moved; the byte-gate was
exercised twice and returned BYTE-IDENTICAL both times.
|
||
|
Drew T
|
a302908f24 |
fix(phase-26a): A3 — target selection was blind to 91.6% of the remaining work; now derived
The audit's CRITICAL finding, fixed at the root. Both tools now derive the corpus from
tools/corpus.py instead of keeping their own decaying copy of the tree layout.
build_fuel_manifest.live_stubs() — a hardcoded 3-file dict {<ov>.c, _a.c, _o0.c}. ov_SC01_077 has
FOURTEEN .c files, so it saw 30 of 264 stubs AND REPORTED SUCCESS. Everything downstream consumes
this manifest — worklist.py (100% of its rows), wave_targets.py (100% of its pools) — so:
targets 30 -> 263
reach-134 targets 10 -> 127 (the ENTIRE high-ROI band was invisible)
remaining gain 83,305 -> 994,633 instructions
994,633 is the A2 audit's predicted figure TO THE UNIT — a fourth independent confirmation
(auditor -> skeptic -> corpus.py -> this). Four of the five highest-leverage functions in the whole
project sit in split regions no tool could see; the top one, func_80178004 (165 ins x reach 134 =
22,110), had never been nominated by anything.
It rotted SILENTLY: .run/fuel_manifest.json (Jul 8) recorded 130 stubs; the same code today returns
30, because the Phase-26 jr splits moved ~100 stubs out from under a dict literal last edited in
Phase 22. Nobody noticed, because a target that is never nominated produces SILENCE, not an error.
wave_targets.REGION_SUB / asm_for() — a 3-entry dict with a silent fallback to the main subdir.
ov_SC01_077 has TWELVE asm subdirs, so 78 of the 87 targets any --class wave emitted handed a
drafter an asm path THAT DOES NOT EXIST. The drafter then drafts against nothing, and the wasted
attempt is booked in the backlog as a *matching* failure — which feeds reserved_walls() and
PERMANENTLY BLACKLISTS a function that was never actually attempted. A silent skip compounding into
a false wall. Now 263/263 asm paths resolve, 0 missing; asm_for() raises rather than guess.
Also: --region's 3-value whitelist defaulted to 'main', which sees 13 of 264 stubs even with a
correct manifest -> default 'any', free-form.
R33 throughout: the INCLUDE_ASM line is SELF-DESCRIBING (its first argument IS the asm subdir,
because splat wrote it there), so both dicts were second copies of a fact the tree already states.
A dict literal is strictly worse than the filesystem AND it fails OPEN. Never re-introduce one.
No build impact (selection/report tools only); docs/worklist.md regenerated with the honest numbers.
|
||
|
Drew T
|
f7b7399ebe |
feat(phase-26a): A3 — tools/corpus.py, ONE derived corpus oracle (+ a second oracle that can disagree)
The 28 surviving audit findings collapse to ONE bug repeated ~10 times: a hand-maintained model of
the corpus layout (a file allowlist, a single-.c assumption, a func_-only symbol regex, a REGION_SUB
dict) sitting on top of a filesystem that already answers the question. The fix is not ten repaired
regexes — it is one DERIVED oracle and ten deleted scanners (R33).
WHAT IT DERIVES FROM
1. THE FILESYSTEM. Which .c files make up a binary, and where a function's .s lives, are FACTS OF
THE TREE THAT SPLAT ITSELF WROTE. The INCLUDE_ASM line is SELF-DESCRIBING — its first argument
IS the asm subdir — so there is nothing to guess and no dict to rot. A dict literal is strictly
worse than the filesystem AND it fails OPEN (silently yields a wrong path) instead of closed.
2. THE PROVEN INVARIANT. INCLUDE_ASM pastes the ORIGINAL asm and the build is byte-identical, so a
function NOT wrapped in it is byte-exact. `matched` is DERIVED as sig - stubs, never re-parsed
from C text. (progress.py learned this the hard way: weighted_metrics() derived and was right;
classify() re-parsed C and inherited a bug.)
VALIDATED against the real corpus:
* ov_SC01_077: 264 stubs across 14 files. The old 3-file allowlist saw 30.
* Fleet: 58,717 stubs vs the allowlist's 1,992 — 56,725 (96.6%) were INVISIBLE.
* Coverage-asserted (R32): every INCLUDE_ASM line must parse, every symbol must resolve (ANY C
identifier — a func_-only regex silently misses the 100 curated listCdBuffer stubs), every stub
must have a .s. A silent skip is a DEFECT, not a no-op.
THE SECOND ORACLE (`make audit-corpus`) — the real lesson of this audit.
The byte-gate is structurally BLIND to a bad function boundary: the .s halves are pasted back
verbatim in original order, so the image stays byte-identical and green. Only an oracle that can
DISAGREE can see it. sig_image is that oracle — Ghidra-free, derived from the ORIGINAL bytes,
independent of splat. corpus.audit() cross-checks the two and reports:
PHANTOM — a stub address the sig does not know: splat INVENTED a function.
TRUNCATED — a stub whose .s length != the sig's: splat MIS-SLICED one.
It reports 193 (96 + 97) — reproducing the A2 audit's number EXACTLY, from an independently written
tool. That is a third confirmation of the listCdBuffer defect (auditor -> skeptic -> this).
AND AN R14 SELF-CATCH, recorded because the near-miss is the lesson.
Run naively over all 136 binaries the same check reports 914 slices — 4.7x the truth. It is noise:
main/resident are signed by the GHIDRA dumper, whose boundaries are shorter than splat's by design
(and which never analysed the linked PsyQ subsegs at all), so the comparison measures GHIDRA'S limits,
not splat's errors. Only the overlays are signed by sig_image, the oracle actually validated at
58,524/58,621. sig_is_independent() now encodes that domain, with the reasoning, so nobody repeats it.
A check applied outside its valid domain does not become more thorough — it becomes noise.
`make audit-corpus` is RED by design until A4 removes the bad symbol line; then it becomes a gate.
|
||
|
Drew T
|
ffb6f1a40f |
docs(phase-26a): A2 — the full audit; 28 findings survive; the endgame plan was majority-fiction
38 agents / 2.24M tok / 0 err. 32 findings raised -> 28 SURVIVED adversarial verification
(4 REFUTED, 16 downgraded). 40 scanners measured CLEAN. Full write-up: docs/tooling-audit.md ROUND 2.
THE ROOT CAUSE — one bug, ~10 times: a hand-maintained model of the corpus layout (a file
allowlist, a single-.c assumption, a func_-only regex, a REGION_SUB dict) sitting on top of a
filesystem that already answers the question. Every TU split silently widened it.
DECAY PROVEN: .run/fuel_manifest.json (Jul 8) recorded 130 stubs; the same tool today returns 30.
The Phase-26 splits moved ~100 stubs out from under a dict literal last edited in Phase 22 — and
nobody noticed, because an un-nominated target produces SILENCE, not an error.
MEASURED: 91.6% of ALL remaining project gain is invisible to target selection (true 994,633 ins;
the manifest sees 83,305). 117 of 127 reach-134 fns never nominated. harvest_verify cannot see
56,742 of 58,717 (96.6%) open stubs. wave_targets hands 78 of 87 targets a nonexistent asm path.
THREE RESULTS OVERTURN SETTLED CONCLUSIONS:
1. Phase-22's 'the permuter's fuel is exhausted' is UNSAFE. grinder banks through harvest_verify,
which sees ONE TU — 1,290 of its own 1,298 queued fns live in another. 99% could never have
banked. '0 banks since Phase 21' is equally consistent with 'the tool could not bank'.
2. The Phase-25/26 endgame plan is MAJORITY-FICTION. family-manifest.md advertises 2,758
multi-member families / 11.0 MB; 1,071 of them / 6.80 MB (62% of the byte-weight) are ALREADY
FULLY MATCHED. The ranking — the file's whole purpose — is sorted mostly on dead work.
3. A CORPUS defect the byte-gate is structurally blind to: symbols.us.txt:981 puts a main-EXE DATA
symbol (listCdBuffer = 0x80180000) into every overlay's symbol stack, but in overlay space that
address is CODE. splat cuts 97 real functions in half and invents 96 phantom ones = 193 slices
NOBODY CAN EVER MATCH, in 97 of 134 overlays — and the build stays byte-identical and green,
because the .s halves are pasted back verbatim. A perfect correctness oracle, a null coverage
oracle. What saved us: sig_image was RIGHT (58,524/58,621 vs spimdisasm; correct on all 97
disagreements). A SECOND INDEPENDENT ORACLE is the only reason it was visible at all.
FIX RESTRUCTURED around the root cause: ONE derived corpus oracle (A3) + ~10 DELETED scanners —
not ten fixed regexes. Plus the listCdBuffer corpus fix (A4) and the closeness oracle (A5, which
lies on 155 functions, feeding false walls into reserved_walls()).
decision-log (R31): the why, and the design lesson — a derived fact cannot rot; a hand-maintained
copy of it is a liability that grows with every structural change. We had no instrument that could
report ABSENCE: every gate we owned answered 'is this right?', none answered 'is this all?'
|
||
|
Drew T
|
92a695ef93 |
docs(phase-26a): ORDER CORRECTED — the full 18-tool audit runs BEFORE the fix campaign
Drew, mid-session: 'I thought the last session said there were some 15 tools we need to audit.' He was right, and my ordering was wrong. I had put the 18-tool audit near the END (as A9). docs/tooling-audit.md prescribes the opposite: dedup_integrate -> jtbl_family_bank -> the SELECTION tools -> masked_diff/match_one -> THEN the 40 measured findings. The reason is the one that matters: A hole in a SELECTION tool makes work invisible to PLANNING — the worst kind, because you never know to look. Fixing on top of unaudited selection tooling means re-running every fix when the audit later finds the hole. So: A2 is now the full audit; A3-A9 (the fix campaign) are blocked on it. Tool coverage, stated plainly: A1 (1) + A2 (18) + the fix campaign (~17 already-measured) = ~36 tools — not 82. The filter, from the audit doc: does it PARSE something, and does it GATE or SELECT work? The remaining ~46 are dead LLM-tier scripts. A1's result recorded in-file (the three false greens, the causal chain, the null-result blast radius that confirms R33). |
||
|
Drew T
|
bb65d36341 |
fix(phase-26a): A1 — dedup_integrate was a gate that could print a FALSE GREEN
The audit's priority #1: a fail-closed byte-honesty validator whose silent skips nothing downstream can catch. Three false-green paths, all measured, all now fail-closed with negative controls. R33 FIRST (derive, don't re-derive). The registry makes two claims; the tool only ever checked one, and mis-described that one: C1 EQUIVALENCE ("these vrams hold the same code in the ORIGINAL") — checked against the sigs, which sign the ORIGINAL bytes. KEPT. But the docstring claimed it also caught SOURCE drift: it cannot. A sig is a property of the ROM, immutable w.r.t. src/. Source drift is caught by the BUILD. Docstring corrected (P9). C2 BANK ("matched once in the source header, instantiated at every member") — NEVER CHECKED. Now DERIVED from the build invariant: INCLUDE_ASM pastes the ORIGINAL asm, so a member NOT wrapped in it is byte-exact, and one that IS wrapped is not banked — whatever the registry says. C2a: the group's macro token must occur in its source file. C2b: no member may still be an INCLUDE_ASM stub. THE THREE FALSE GREENS 1. 1808 groups claimed a DEFINE_func_* macro; only 1801 exist. The 7 ghosts printed [ OK ] — hiding 532 member-instances / 22,344 instructions of REAL, UNBANKED work (4 fns matched in ov_SC01_077, still INCLUDE_ASM in the other 133 overlays). 2. An absent .run/sig.<bin>.jsonl degraded to "0 validated, 0 failed" and EXIT 0. On a fresh clone the gate validated NOTHING and passed. Now fails; --allow-unsigned is the escape. 3. The bank claim was never checked at all. THE CAUSAL CHAIN (the audit's thesis in one example). 3 of the 4 hidden fns are defined in ov_SC01_077_jr_8012ACE0.c — a _jr_* split file. dedup_propagate.overlay_files allowlists only ("_a","_o0","_o0b","_after"), so the propagator could not SEE them; the group was registered anyway; dedup_integrate greenlit the lie. TWO silent-skip bugs compounding: one created the hole, the other hid it. Harvest fuel -> .run/audit/a1_harvest_fuel.json, banked in A5. BLAST RADIUS, MEASURED NOT PREDICTED (R14). Headline metrics UNCHANGED to the decimal (instr-weighted 66.5%, distinct-code 46.8%) — weighted_metrics() derives from the invariant and was structurally immune to the lying registry. FLEET REAL substantive unchanged (282,466): progress.py had already been taught to distrust it (commit:0574). Only dedup_integrate still believed it. A null result that CONFIRMS R33: the tool that refused to re-derive was the one that was right. - registry repaired: 1813 -> 1806 groups (7 ghosts removed; instances 223,725 -> 222,787) - make report GREEN end-to-end: 1806 validated, 0 failed | C1 coverage 222,787/222,787 signed - negative controls: stubbed member -> exit 1; missing sig -> exit 1; --allow-unsigned -> exit 0 - report-only tool: no compiled artifact depends on it, so no R22 clean-fleet is owed here |
||
|
Drew T
|
978ef703ac |
docs(phase-26a): A0 — the tooling-integrity audit, as an INSERTED HALF-PHASE (Drew's call)
- Drew (2026-07-14, gate 1): run the audit inside Phase 26, then resume at Task 7. Declined the alternative (close Phase 26 early on an unmet milestone -> Phase 27): the audit is a PREREQUISITE to structural completion, not a successor to it — the tooling that MEASURES the milestone is the thing at fault. Phase-3.5 precedent. - CURRENT_PHASE.md: the Phase 26-A block (A0-A11), built FROM docs/tooling-audit.md (40 measured findings), R33-before-R32 ordering — the best outcome is a DELETED scanner, not a fixed regex. - decision-log (R31): the why, the structural blind spot (a scanner extracts N, the true count is M > N, and nobody ever compared N to M — the byte-gate is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle), and A1's first finding. - harness task list built (R28). |
||
|
Drew T
|
b757de73a8 |
docs(phase-26): checkpoint now POINTS AT docs/tooling-audit.md as the audit phase's input document
The 40 measured findings were living only in an ephemeral workflow journal outside the repo; the checkpoint carried my SUMMARY of the audit, not the audit. Now the fresh session is routed to the evidence, with the priority order (dedup_integrate FIRST — it can print a false green), the R33-before-R32 method (the best outcome is a DELETED scanner), and the real prize: re-test the walls that were diagnosed on top of the broken 10% callee oracle (the def-side loose-typing wall, the 159 arity conflicts, the type-heavy tail). |
||
|
Drew T
|
9a94e9ba46 |
docs(phase-26): docs/tooling-audit.md — the 40 measured findings, made DURABLE (R30)
The audit's evidence (6 agents + 6 skeptics, 1.2M tokens, 40 findings with file:line proof and measured candidate/parsed/skip counts) existed ONLY in a workflow journal OUTSIDE the repo. A fresh session would have inherited my SUMMARY of the audit, not the audit — exactly the R30 failure mode (capture context-dependent artifacts DURING the session that produced them). Drew caught it. Now committed as the plannable input to the audit phase, with: - the method (measure found-vs-candidates against an OVER-approximating detector; never "review the regex" — that is the failure mode that wrote these bugs); - why it gates the matching work (the byte-gate is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle: green since Phase 5 at 0% decompiled, so it is compatible with ANY decomp %); - THE QUESTION IT ANSWERS: how many walls we have already "byte-proven" across 26 phases were lookup misses wearing a wall's clothes? (the def-side loose-typing wall, the 159 arity conflicts, the type-heavy tail were ALL diagnosed on top of the 10% callee-oracle hole); - R32 (coverage assertion) + R33 (derive, don't re-derive — apply FIRST: the best outcome is a DELETED scanner, not a fixed regex); - the priority order (dedup_integrate FIRST — a fail-closed validator that can print a FALSE GREEN); - the 7 bugs already fixed (do not redo) and the 63 tools not yet audited, with the filter for which matter. |