feat(phase-29): backlog prune — compact the append-only near-miss log + wire into make report

The near-miss ledger (.run/backlog.jsonl) is append-only, so it filled with already-banked noise:
6,867 rows, ~98% banked. load_best()/render() already filtered on READ (docs/backlog.md was correct),
but the raw log drifted stale and every render re-scanned all 6,867 rows against the stub oracle.

- backlog.py: new `prune` subcommand — atomic rewrite (temp + os.replace) to load_best()'s output
  (drop-now-matched P9 + best-per-addr collapse). Idempotent. 6,867 -> 1,704 open near-misses.
- Makefile: `backlog.py prune` wired into `make report` (BINARY=main block) so the ledger tracks
  reality every cycle instead of drifting.
- Finding (Drew's question): crack waves DO log every non-byte-match to the backlog durably
  (gate_stage copies best_draft -> .run/backlog_drafts/). BUT the `closeness` field is UNRELIABLE —
  byte-correct drafts (match_one MATCH) are logged with closeness>0 (e.g. func_8012F49C logged 29,
  actually MATCH). And a reach-N function's draft is overlay-SPECIFIC (per-location symbols), so the
  backlog is a messy recovery source vs the fresh per-wave stranded drafts. Integration-recovery
  should consume the fresh wave-dir strandeds, not re-derive from the backlog.
This commit is contained in:
Drew T
2026-07-24 11:36:54 -06:00
parent bf307f2827
commit ef85803b1f
4 changed files with 3422 additions and 8563 deletions
+1687 -6850
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -216,6 +216,10 @@ ifeq ($(BINARY),main)
$(VENV_PY) tools/dup_report.py --cross
# Fleet roll-up (Phase 15): deterministic per-binary table + fleet totals -> docs/progress.fleet.md.
$(VENV_PY) tools/progress.py --fleet
# Backlog compaction (Phase 29): the near-miss log is append-only, so it fills with already-banked
# noise (measured 6,867 rows, 98% banked). prune rewrites .run/backlog.jsonl to the open near-misses
# (drop-now-matched P9 + best-per-addr) so the ledger tracks reality instead of drifting stale.
$(VENV_PY) tools/backlog.py prune
# Rename-drift gate (Phase 26-A): fail-closed if a symbols.us.txt rename left a func_<ADDR>
# ref dangling in committed src/ or src/shared/*.h — the R22 failure mode an incremental build
# masks (stale .o) but a genuinely-clean rebuild fails on. The ONLY detector for it.
+1705 -1713
View File
File diff suppressed because it is too large Load Diff
+26
View File
@@ -150,6 +150,29 @@ def render():
return len(recs)
def prune():
"""Compact .run/backlog.jsonl to reality: keep ONE record per addr (the best), and DROP every
entry whose function is no longer an open stub in its binary (banked since it was logged, P9).
The raw log is append-only, so it grows without bound and fills with already-banked noise
(measured 2026-07-24: 6,867 rows, ~98% already banked). `render()`/`load_best()` already filter
on READ, so docs/backlog.md was correct — but every render re-scanned all 6,867 rows and re-hit
the stub oracle, and the file itself misrepresented the real open count. prune rewrites the log
to load_best()'s output so the file matches what the tools already compute.
Atomic (temp + os.replace) so an interrupt never truncates the log. Idempotent."""
before = sum(1 for _ in open(JSONL)) if os.path.exists(JSONL) else 0
kept = load_best() # already: drop-now-matched (P9) + best-per-addr
tmp = JSONL + ".tmp"
with open(tmp, "w") as f:
for r in kept:
f.write(json.dumps({k: r.get(k) for k in FIELDS}) + "\n")
os.replace(tmp, JSONL)
print(f"backlog prune: {before} rows -> {len(kept)} open near-misses "
f"(dropped {before - len(kept)} banked/superseded)")
return len(kept)
def main():
ap = argparse.ArgumentParser()
sub = ap.add_subparsers(dest="cmd", required=True)
@@ -159,6 +182,7 @@ def main():
for fl in ("reach", "nins", "closeness"):
lg.add_argument(f"--{fl}", type=int, default=None)
sub.add_parser("render")
sub.add_parser("prune")
sh = sub.add_parser("show"); sh.add_argument("-n", type=int, default=40)
a = ap.parse_args()
if a.cmd == "log":
@@ -169,6 +193,8 @@ def main():
print(f"logged {rec.get('name') or rec.get('addr')}; backlog open={n}")
elif a.cmd == "render":
print(f"docs/backlog.md: {render()} open near-misses")
elif a.cmd == "prune":
prune(); render()
elif a.cmd == "show":
for i, r in enumerate(sorted(load_best(), key=_rank_key)[:a.n], 1):
print(f"{i:3} {r.get('name'):16} reach={r.get('reach')} {r.get('klass'):7} "