tools(phase-35): T7 — the S1 invariant wired into tools-health (share_census --selftest + --check --strict-macros --strict-text, by exit code), C2c/C2d in dedup_integrate --check (one source under src/shared/ defining one function; every member's site includes it — from the census's per-instance forms), progress.py: unique_function_bodies 105,007 / duplicate_source_copies 160 in 51 ledgered classes + the dated corrections list + the README sentence; the second oracle refined (distinct TUs; deferred vs pending vs violation): 380 texts deferred inside cross-address classes, 38 same-address texts / 2,030 sites byte-variant per binary (PENDING the owner), 0 violations; negative control in place: one reverted include -> S1 FAIL naming the class + C2d naming the member; tools-health OK (551 s)

This commit is contained in:
Drew T
2026-09-08 21:46:05 -06:00
parent 79eb66fa07
commit c8224e657f
17 changed files with 7153 additions and 34 deletions
+5
View File
@@ -0,0 +1,5 @@
.venv/bin/python tools/tool_census.py --all
tool_census: 295 tool files (find == git ls-files) + 38 retired; classes {'LIVE': 236, 'ORPHAN': 30, 'REFERENCED': 29, 'RETIRED': 38}; dictionary rows 333
tool_census: wrote docs/tool-index.md
tool_census: wrote decomp-architect/tools/MANIFEST.md
tool_census: corpus materialised — 361 verbatim copies, 30 pointers, INDEX.md
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -31,7 +31,7 @@
},
"duplicate_text_classes": 6429,
"duplicate_text_sites": 19811,
"elapsed_s": 112.4,
"elapsed_s": 36.8,
"flags": {
"ALIAS": 32,
"E": 3826,
+1 -1
View File
@@ -37,4 +37,4 @@ share_census: 218 binaries · 362,389 sig instances · 362,389 classified · 0 U
B E,F 11852 x 2 0x80014554,0x8001455C,0x8001513C func_80014554,func_8001455c
A - 141 x 167 0x80163C2C func_80163c2c
A - 141 x 165 0x80178004 func_80178004
macro sites 0 · duplicate-text classes 6,429 (19,811 sites) · elapsed 112.4 s
macro sites 0 · duplicate-text classes 6,429 (19,811 sites) · elapsed 36.8 s
+6
View File
@@ -293,6 +293,12 @@ tools-health:
$(MAKE) --no-print-directory audit-cdecl
$(MAKE) --no-print-directory audit-binaries
$(MAKE) --no-print-directory audit-text-sources
# Phase 35 T7: the S1 invariant — one source per unique function — as a gate (R36): the census self-test (7 verdicts), then the
# strict check by exit code (R97): no same-address class unshared unless ledgered in config/dedup_exceptions.tsv, no DEFINE_func_
# site under src/ (--strict-macros), and the sig-blind second oracle (--strict-text). Runs BEFORE report so progress.py reads a
# fresh .run/P35/census/share_census.json for its duplicate-copy fields.
$(VENV_PY) tools/share_census.py --selftest
$(VENV_PY) tools/share_census.py --check --strict-macros --strict-text --quiet
$(MAKE) --no-print-directory report BINARY=main
# AFTER report (which regenerates the digest), so this asserts the freshly-written digest agrees
# with the tree — and, on a tree whose digest was committed stale, says so instead of staying green.
+2
View File
@@ -25,6 +25,8 @@ originals; nothing "functionally equivalent" counts. What the repository claims
218 binaries rebuild byte-identical from source · 350,533 functions in C (249,873 of them shared bodies via 3,135 dedup groups) · 1,256 Sony PsyQ library functions linked from the SDK objects, not our C · 5 hand-written-assembly bodies kept verbatim · 0 assembly stubs left · 0 non-matching functions.
One source per unique function (Phase 35): 105,007 function bodies written once in C (3,135 of them shared headers instantiated 260,543 times); 160 duplicate copies remain in 51 ledgered classes (declaration conflicts left for the types phase).
_Generated by `tools/progress.py --readme` from `docs/progress.json` — numbers are never typed by hand._
<!-- progress:end -->
@@ -39,6 +39,12 @@ tools-health:
$(MAKE) --no-print-directory audit-cdecl
$(MAKE) --no-print-directory audit-binaries
$(MAKE) --no-print-directory audit-text-sources
# Phase 35 T7: the S1 invariant — one source per unique function — as a gate (R36): the census self-test (7 verdicts), then the
# strict check by exit code (R97): no same-address class unshared unless ledgered in config/dedup_exceptions.tsv, no DEFINE_func_
# site under src/ (--strict-macros), and the sig-blind second oracle (--strict-text). Runs BEFORE report so progress.py reads a
# fresh .run/P35/census/share_census.json for its duplicate-copy fields.
$(VENV_PY) tools/share_census.py --selftest
$(VENV_PY) tools/share_census.py --check --strict-macros --strict-text --quiet
$(MAKE) --no-print-directory report BINARY=main
# AFTER report (which regenerates the digest), so this asserts the freshly-written digest agrees
# with the tree — and, on a tree whose digest was committed stale, says so instead of staying green.
@@ -642,6 +642,8 @@ def render_table(classes, summary, cov_notes, ctrl):
# the second oracle: duplicate definition TEXT across translation units (sig-blind)
# ----------------------------------------------------------------------------------------------------------------------
def text_duplicates(forms):
"""ANY-address duplicates: a name-blind normalized definition text present in >1 TU (the cross-address ones are the names
phase's — reported as a count, never a violation here)."""
by_text = collections.defaultdict(set)
for a, per in forms.items():
for addr, rec in per.items():
@@ -650,6 +652,19 @@ def text_duplicates(forms):
return {h: sorted(v) for h, v in by_text.items() if len({(a, tu) for a, tu, _ in v}) > 1}
def text_duplicates_same_addr(forms, excepted_sites=frozenset()):
"""S1's SECOND, sig-blind oracle (R34; --strict-text): the same normalized definition text, at the SAME address, still a private
`def` in >1 TU — exactly what the h_exact join must have shared and did not. Ledgered sites (the exception ledger's classes) are
excluded, as they are from the first oracle. Keyed by (text, addr) so the deferred cross-address duplicates never count."""
by_key = collections.defaultdict(set)
for a, per in forms.items():
for addr, rec in per.items():
if rec["form"] == "def" and rec.get("text_hash") and rec.get("nlines", 0) >= 2 and (a, addr) not in excepted_sites:
by_key[(rec["text_hash"], addr)].add((a, rec["tu"]))
# distinct TUs, not distinct aliases: a twin's instance resolves to its primary's TU (one source already)
return {k: sorted(v) for k, v in by_key.items() if len({tu for _, tu in v}) > 1}
# ----------------------------------------------------------------------------------------------------------------------
# self-test fixture (R39): every verdict and flag, in memory
# ----------------------------------------------------------------------------------------------------------------------
@@ -760,7 +775,7 @@ def main():
groups = load_groups()
exceptions = load_exceptions(a.exceptions)
verb = verbatim_instances()
classes, cov = classify(sigs, forms, groups, spaces, dirs, twins, twin_of, verb, exceptions, scope=a.scope)
classes, cov = classify(sigs, forms, groups, spaces, dirs, twins, twin_of, verb, exceptions, scope=a.scope, keep_instances=True)
summary = summarize(classes, cov, notes, aliases)
ctrl = controls(classes, sigs, forms)
macro_tokens = 0
@@ -801,8 +816,44 @@ def main():
bad = len(v)
if a.strict_macros and macro_tokens:
print(f"share_census: S1 — {macro_tokens} DEFINE_func_ site(s) remain under src/ (--strict-macros)"); bad += 1
if a.strict_text and dup_text:
print(f"share_census: S1 — {len(dup_text)} definition text(s) duplicated across TUs (--strict-text)"); bad += 1
if a.strict_text:
# the second oracle mirrors the first's scope: ledgered classes AND the gate-1 deferral (E/F cross-address classes — an
# empty body's hash spans every address, so its same-address pairs sit inside a deferred class) are excluded; the deferred
# same-address duplicates are PUBLISHED as the names phase's inheritance, never silently dropped (R41)
exc_sites = {(i_alias, int(addr_s, 16)) for c in classes if c["excepted"] for addr_s in c["addrs"] for i_alias in c["aliases"]}
def_sites = {(i_alias, int(addr_s, 16)) for c in classes if ("E" in c["flags"] or "F" in c["flags"]) for addr_s in c["addrs"] for i_alias in c["aliases"]}
dup_deferred = text_duplicates_same_addr(forms, exc_sites)
dup_rest = text_duplicates_same_addr(forms, exc_sites | def_sites)
n_def = len(dup_deferred) - len(dup_rest)
summary["same_address_text_duplicates_deferred"] = {"texts": n_def, "sites": sum(len(v) for v in dup_deferred.values()) - sum(len(v) for v in dup_rest.values())}
print(f"share_census: second oracle — {n_def:,} same-address duplicated texts sit inside the deferred cross-address classes "
f"({summary['same_address_text_duplicates_deferred']['sites']:,} sites; the names phase's inheritance)")
# what is left is either a class the byte join has (a VIOLATION the first oracle missed — must be 0) or the same text at
# the same address compiling to DIFFERENT bytes per binary (singleton h_exact classes: data addresses differ per overlay);
# the latter is one source the byte tier cannot register — published as PENDING (a decision, not a pass)
# a violation only when two DISTINCT TUs hold private copies of the SAME byte class at that address (a twin pair is one
# TU); every other same-address text duplicate is byte-variant across its TUs — the byte tier cannot register it
site_h = {(i["alias"], i["addr"]): c["h"] for c in classes for i in c["insts"]}
def _shared_class_across_tus(k, v):
by_h = {}
for a, tu in v:
h = site_h.get((a, k[1]))
if h:
by_h.setdefault(h, set()).add(tu)
return any(len(tus) > 1 for tus in by_h.values())
dup_same = {k: v for k, v in dup_rest.items() if _shared_class_across_tus(k, v)}
dup_pending = {k: v for k, v in dup_rest.items() if k not in dup_same}
summary["same_address_text_duplicates_pending"] = {"texts": len(dup_pending), "sites": sum(len(v) for v in dup_pending.values())}
if dup_pending:
print(f"share_census: second oracle — PENDING {len(dup_pending):,} same-address definition texts ({summary['same_address_text_duplicates_pending']['sites']:,} "
f"sites) are duplicated across TUs but compile to DIFFERENT bytes per binary (singleton h_exact classes) — one source the"
f" byte tier cannot register; a decision for the owner (an h_norm/text tier or the names phase)")
if dup_same:
print(f"share_census: S1 second oracle — {len(dup_same)} definition text(s) duplicated at the SAME address across TUs and not "
f"ledgered (--strict-text): " + "; ".join(f"0x{addr:08X} in {[a for a, _ in v][:4]}" for (_, addr), v in list(dup_same.items())[:6])); bad += 1
else:
print(f"share_census: S1 second oracle — 0 same-address definition texts duplicated across TUs (any-address duplicates "
f"{len(dup_text):,}, the names phase's)")
exc = sum(1 for c in classes if c["excepted"])
twinc = sum(1 for c in classes if "TWIN-COVERED" in c["flags"] and c["verdict"] != "A")
print(f"S1: one source per unique function — {len(classes):,} classes, {len(classes) - len(v):,} satisfied "
@@ -44,7 +44,7 @@ sharing is source-level. The .ld interpose stays the library mechanism.
Usage: tools/dedup_integrate.py [--check] [--binary <alias>] [--allow-unsigned]
[--dedup config/dedup.us.yaml]
"""
import argparse, json, pathlib, re, sys
import argparse, json, os, pathlib, re, sys
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "tools"))
@@ -168,6 +168,19 @@ def check(groups, binary_filter=None, allow_unsigned=False):
failures = unvalidated = validated = 0
members_seen = members_c1 = 0
# C2d (Phase 35 T7): the per-instance source FORMS from the census (one scan per TU, cached by mtime/size) — a member's site must
# be an include of THIS group's source; a surviving private copy is the registry running ahead of the source.
_forms = {}
def site_form(binary, vram):
if not _forms:
sys.path.insert(0, str(ROOT / "tools"))
import share_census as _sc
aliases, dirs = _sc.fleet_and_dirs()
f, _notes = _sc.build_forms(aliases, dirs, os.cpu_count() or 4, use_cache=True)
_forms.update(f)
_forms.setdefault("__loaded__", {})
return _forms.get(binary, {}).get(vram)
for g in groups:
gid = g.get("id", "?")
@@ -195,14 +208,20 @@ def check(groups, binary_filter=None, allow_unsigned=False):
# ---- C2a′ (Phase 35 T2): a PLAIN-C header source must DEFINE `func` (a token occurrence is not a body) --------
# The parameterized form (SHARED_FN) defines through cpp and keeps the token check above; every other source is a
# plain-C header read by share_census.header_defs — the one reader of that form (R33).
if not PARAM_FUNC_RE.match(fn):
sys.path.insert(0, str(ROOT / "tools"))
import share_census
defined = {n for n, _ in share_census.header_defs(ROOT / src)}
if fn not in defined:
print(f"[FAIL] {gid}: `func: {fn}` occurs in {src} but that header does not DEFINE it "
f"(defines: {sorted(defined)[:4]}) — a declaration is not a shared body")
failures += 1; continue
sys.path.insert(0, str(ROOT / "tools"))
import share_census
defined = {n for n, _ in share_census.header_defs(ROOT / src)}
if not PARAM_FUNC_RE.match(fn) and fn not in defined:
print(f"[FAIL] {gid}: `func: {fn}` occurs in {src} but that header does not DEFINE it "
f"(defines: {sorted(defined)[:4]}) — a declaration is not a shared body")
failures += 1; continue
# ---- C2c (Phase 35 T7): ONE source, under src/shared/, defining exactly ONE function — the token the group names ----------
if not src.startswith("src/shared/"):
print(f"[FAIL] {gid}: source {src} is not under src/shared/ — a shared body has its one source there (S1)")
failures += 1; continue
if len(defined) != 1:
print(f"[FAIL] {gid}: {src} defines {len(defined)} functions {sorted(defined)[:4]} — a shared header defines exactly one (C2c)")
failures += 1; continue
tier, want = g["tier"], g.get("hash")
if not want:
@@ -218,6 +237,18 @@ def check(groups, binary_filter=None, allow_unsigned=False):
f"INCLUDE_ASM stub — the registry is claiming work that was never done")
failures += 1; ok = False; continue
# ---- C2d (Phase 35 T7): the member's SITE includes this group's source — no private copy survives ----------------
rec = site_form(b, vram)
if rec is None:
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) has no source form in the census (coverage, R32)")
failures += 1; ok = False; continue
if rec.get("form") not in ("include", "param-include"):
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) is listed as a member but its site is a {rec.get('form')} in {rec.get('tu')} — "
f"the registry runs ahead of the source (C2d)")
failures += 1; ok = False; continue
if rec.get("header") and rec["header"] != src:
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) includes {rec['header']}, not this group's source {src} (C2d)")
failures += 1; ok = False; continue
# ---- C1: equivalence, against the ORIGINAL bytes ------------------------------------
idx = sig_for(b)
if idx is None:
+25 -1
View File
@@ -1139,6 +1139,14 @@ def fleet():
nverb = len(vm.get("rows", vm) if isinstance(vm, dict) else vm)
except Exception:
nverb = 0
_census = {}
try:
import json as _j
_cj = ROOT / ".run/P35/census/share_census.json"
if _cj.exists():
_census = _j.loads(_cj.read_text()).get("same_vram_unregistered", {})
except Exception:
_census = {}
return {
"schema": 1,
"note": "generated by tools/progress.py --fleet/--json from the committed sources — every number carries its denominator; regenerate, never edit",
@@ -1156,7 +1164,19 @@ def fleet():
},
"counts": {"real_c_functions": REAL, "dedup_shared_of_real": SHARED, "linked_psyq_objects": LINKED,
"verbatim_asm_bodies": nverb, "include_asm_stubs": STUBS, "non_matching": NM, "matchable": MATCH,
"dedup_groups": ngroups, "dedup_instances": nmembers},
"dedup_groups": ngroups, "dedup_instances": nmembers,
# Phase 35 T7 — "one source per unique function": bodies written ONCE in C = every instantiated function (REAL + EMPTY)
# minus the shared instances plus their one header each (derived from this run's own totals, R33); the duplicate
# copies still in source come from the census (.run/P35/census/share_census.json — every one is ledgered, S1)
"unique_function_bodies": REAL + EMPTY - nmembers + ngroups,
"duplicate_source_copies": _census.get("copies"), "duplicate_source_classes": _census.get("classes")},
"corrections": [ # dated snapshots of published-number moves with their cause and the command that shows them (R75)
{"date": "2026-09-08", "phase": "P35 T2", "metric": "fn_count.total", "delta": 7,
"cause": "seven overlays including the whale header but never registered were counted by neither classify nor the registry fold", "command": "tools/progress.py --fleet"},
{"date": "2026-09-08", "phase": "P35 T6", "metric": "fn_count.total", "delta": 459,
"cause": "macro sites invisible to the classifier became includes at T4: ov_SC03_015's 219 single-site macros, ov_SC03_118's 3, and the never-extended members of five under-listed groups", "command": "tools/progress.py --fleet"},
{"date": "2026-09-08", "phase": "P35 T6", "metric": "counts.real_c_functions", "delta": -10211,
"cause": "empty-bodied shared functions were folded into REAL under the macro form; the include form classifies them EMPTY (the instruction-weighted metrics are unchanged)", "command": "tools/progress.py --fleet"}],
"per_binary": [{"binary": r['binary'], "real": r['real'], "shared": r['shared'], "linked": r['linked'],
"byte_identical": r['byteident'], "matchable": r['matchable'],
"instr_matched": (wm['per_bin'].get(r['binary'], [None, None])[0] if wm else None),
@@ -1214,6 +1234,10 @@ def readme_block(d):
f"{c['linked_psyq_objects']:,} Sony PsyQ library functions linked from the SDK objects, not our C · "
f"{c['verbatim_asm_bodies']} hand-written-assembly bodies kept verbatim · {c['include_asm_stubs']} assembly stubs left · "
f"{c['non_matching']} non-matching functions.",
"", (f"One source per unique function (Phase 35): {c['unique_function_bodies']:,} function bodies written once in C "
f"({c['dedup_groups']:,} of them shared headers instantiated {c['dedup_instances']:,} times); "
+ (f"{c['duplicate_source_copies']:,} duplicate copies remain in {c['duplicate_source_classes']:,} ledgered classes "
f"(declaration conflicts left for the types phase)." if c.get('duplicate_source_copies') is not None else "the duplicate-copy census was not available in this run.")),
"", f"_Generated by `tools/progress.py --readme` from `docs/progress.json` — numbers are never typed by hand._",
README_END]
return "\n".join(out)
+30 -1
View File
@@ -39,8 +39,37 @@
"non_matching": 0,
"matchable": 363680,
"dedup_groups": 3135,
"dedup_instances": 260543
"dedup_instances": 260543,
"unique_function_bodies": 105007,
"duplicate_source_copies": 160,
"duplicate_source_classes": 51
},
"corrections": [
{
"date": "2026-09-08",
"phase": "P35 T2",
"metric": "fn_count.total",
"delta": 7,
"cause": "seven overlays including the whale header but never registered were counted by neither classify nor the registry fold",
"command": "tools/progress.py --fleet"
},
{
"date": "2026-09-08",
"phase": "P35 T6",
"metric": "fn_count.total",
"delta": 459,
"cause": "macro sites invisible to the classifier became includes at T4: ov_SC03_015's 219 single-site macros, ov_SC03_118's 3, and the never-extended members of five under-listed groups",
"command": "tools/progress.py --fleet"
},
{
"date": "2026-09-08",
"phase": "P35 T6",
"metric": "counts.real_c_functions",
"delta": -10211,
"cause": "empty-bodied shared functions were folded into REAL under the macro form; the include form classifies them EMPTY (the instruction-weighted metrics are unchanged)",
"command": "tools/progress.py --fleet"
}
],
"per_binary": [
{
"binary": "main",
+1 -1
View File
@@ -89,6 +89,6 @@
| 2026-09-05 | 83 | P31 (v1.30.0) | 218 | 100.0% | 100.0% | 100.0% | 97.7% | 4 | 787 / 2091 |
| 2026-09-06 | 36 | P32 (v1.31.0) | 218 | 100.0% | 100.0% | 100.0% | 100.0% | 0 | 789 / 2091 |
| 2026-09-07 | 61 | P33 (v1.32.0) | 218 | 100.0% | 100.0% | 100.0% | 100.0% | 0 | — |
| 2026-09-08 | 67 | P33.5 (v1.32.1), P34 (v2.0.0) | 218 | 100.0% | 100.0% | 100.0% | 100.0% | 0 | — |
| 2026-09-08 | 69 | P33.5 (v1.32.1), P34 (v2.0.0) | 218 | 100.0% | 100.0% | 100.0% | 100.0% | 0 | — |
74 dated rows · phase ticks from the 35 PhaseEnds · the chart: `docs/story-timeline.svg`.
+1 -1
View File
@@ -328,7 +328,7 @@ then what proved it. The same data generates the day-one kit's manifest and its
| flag address-named references whose address now has a curated name | `lint_symbol_refs.py` | Flags address-named references in committed sources whose address now has a curated name | .github/workflows/no-rom.yml, Makefile | repo symbol/src paths | LIVE |
| give each conflicting camp of a same-named type its own name | `uniquify_type.py` | Gives each conflicting camp of a same-named type its own name so every camp becomes liftable | — | repo src layout | REFERENCED |
| guard that every inline-assembly body still reproduces its target bytes | `verbatim_check.py` | Regression guard that every inline-assembly body still reproduces its target bytes | .github/workflows/no-rom.yml, verbatim_target_s.py, verbatim_to_stub.py | repo src layout | LIVE |
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | audit_binaries.py, dedup_integrate.py, gen_harvest_targets.py, macro_to_header.py (+4) | repo paths, the registry and signature schemas | LIVE |
| measure duplicate function bodies across the fleet and assert one source per unique function | `share_census.py` | The census of byte-identical function classes across every binary with their source forms and verdicts, plus the S1 invariant check with its exception ledger and a fixture self-test | Makefile, audit_binaries.py, dedup_integrate.py, gen_harvest_targets.py (+5) | repo paths, the registry and signature schemas | LIVE |
| mirror the curated symbol file into the analysis database with a real save | `ghidra_apply_symbols.sh` | Mirrors the curated symbol file into the analysis program headlessly, with a real save | — | repo symbol path, project name | REFERENCED |
| refuse, from one place, the command line of a tool the project has frozen | `frozen.py` | The one refusal a FROZEN tool prints from its main(): the tool, the successor and why; imports never exit (libraries stay usable) | aprop_autodraft.py, blocker_probe.py, canon_sig_reconcile.py, conform_decls.py (+10) | nothing | LIVE |
| regenerate a splitter-format target disassembly for a function no longer stubbed | `verbatim_target_s.py` | Regenerates a splitter-format target disassembly for a function that is no longer a stub; --gas emits the assemblable gas-syntax form ($-registers, .L labels, noreorder) that a web diff service accepts as a pasted target | decompme_replica.sh | repo build/asm layout | LIVE |
+28 -3
View File
@@ -64,8 +64,10 @@
tools' dead macro branches dropped, the `ast` macro-form guard in `tool_census --check` (0 LIVE hits; the pre-T4 tree flags 15 —
the negative control), `progress.py`'s empty-shared fold corrected (+459 denominator, REAL −10,211 → EMPTY), kit corpus, SETUP;
`tools-health: OK`.
- ☐ **T7** — S1 strict + `--selftest` + C2c/C2d in `make tools-health`; `progress.py` fields + README block; the +219 correction stated;
the gate negative-controlled in a worktree.
- ☑ **T7** (S96) — S1 strict (`--strict-macros --strict-text`) + `--selftest` in `make tools-health`; C2c/C2d in `dedup_integrate --check`;
`progress.py` fields (105,007 bodies written once; 160 / 51 duplicate copies, all ledgered) + the dated corrections (+7, +459, REAL
−10,211) + the README sentence; the gate negative-controlled in place (exit 1 naming the class; C2d naming the member); the second
oracle's finding published: 380 texts deferred inside cross-address classes, 38 texts / 2,030 sites byte-variant — PENDING Drew.
- ☐ **T8** — the record: wiki (4 pages), how-to ch.10, README:117, the charter rows as dated snapshots, the cookbook section, decision log
(R31), accelerators, DIGEST §4, sunset rows, the memory rewritten; doc_links/wiki_render/cookbook_index/kit_coverage green.
- ☐ **T9** — close (Tier 1): R22 → 218/218; tools-health OK; the metrics table; the reviewer sequence; PhaseEnd + DIGEST + log archived; v2.1.0.
@@ -509,6 +511,29 @@
484 s, exit 0) with `macro-form guard: 0 LIVE tools reference the retired form (217 scanned, 14 frozen, 6 whitelisted detectors, 38
retired)`; the timeline regenerated by the chain against the committed digest (74 rows, self-check OK). **T6 ☑.**
- **S96 — T7: the invariants wired + every published number regenerated.** Makefile `tools-health`: after `audit-text-sources`, before
`report BINARY=main`, two rungs — `share_census.py --selftest` (7/7) and `share_census.py --check --strict-macros --strict-text --quiet`
(exit code, R97) — so `progress.py` reads a fresh census json. `dedup_integrate --check` gained **C2c** (one source under `src/shared/`
defining exactly one function) and **C2d** (every member's site is an include of THIS group's source — from the census's per-instance
forms, one cached scan; a `def` site = the registry ahead of the source). `progress.py`: `counts.unique_function_bodies` (REAL + EMPTY −
shared instances + their one header each = **105,007** bodies written once in C), `duplicate_source_copies` / `_classes` (**160 / 51**,
from the census json, all ledgered), a dated `corrections` list (T2 +7; T6 +459; T6 REAL −10,211) and the README sentence "One source
per unique function (Phase 35): …". **The second oracle (R34) disagreed with the byte oracle and the disagreement is measured:** of
4,312 same-address definition texts duplicated across TUs, 3,495 were a twin and its primary (one TU — the oracle now counts distinct
TUs), 28 are ledgered, **380 texts (1,668 sites) sit inside the E/F-deferred cross-address classes** (an empty body's hash spans every
address, so gate 1's deferral took its same-address pairs along — published as the names phase's inheritance), and **38 texts (2,030
sites) are the same C at the same address compiling to DIFFERENT bytes per binary** (singleton h_exact classes — e.g. `func_8012AAAC`,
138 copies, 133 byte patterns, h_norm differs in 5 ways): one source the byte tier cannot register — published as PENDING, a decision
for Drew (an `h_text` tier now, or the names phase); 0 texts a shared byte class holds across two TUs — the first oracle has no hole.
**Negative control (R39), in place on the probe class `S_func_801681FC` (2 members):** ov_SC05_009's include reverted to the 19-line
private copy → `S1 … 10,179 satisfied … 1 VIOLATION(S) — FAIL` naming `B - h=c1c085b28d … ['ov_SC04_008', 'ov_SC05_009']`, exit 1;
`[FAIL] S_func_801681FC: ov_SC05_009:0x801681fc … its site is a def in … (C2d)`, `dedup-check: 3134 validated, 1 failed`; restored →
`0 VIOLATION(S) — OK`, exit 0. Instrument slips on the way, each caught by the run: a missing `os` import (C2d's first run), the
`--check` path not keeping per-instance records, twin pairs counted as two TUs, "classed" mis-defined as any classed site (fixed to
"one class across two TUs"). **Verify:** `make tools-health` → `tools-health: OK — …` (`.run/P35/baseline/tools_health_t7.log`, 551 s,
exit 0) with `S1: … 10,180 satisfied … 0 VIOLATION(S) — OK`, `dedup-check: 3135 validated, 0 failed | C1 coverage 260543/260543`,
`macro-form guard: 0 LIVE …`, `progress.py --check: … fresh`, `timeline --check: fresh (74 rows)`. **T7 ☑.**
## Approved plan (verbatim, gate 1 — 2026-09-08)
# Phase 35 — Gen3 opens: the dedup phase, "one source per unique function" (v2.0.0 → v2.1.0)
@@ -808,7 +833,7 @@ snapshot" (share_body's bisect wiped the previous batch's uncommitted shares; R4
negative-controlled against the compiler's real message forms, not against the word error" (gcc 2.7.2 prints errors without it;
254 of 303 rejection lines read `Error 33`).
## 🛑 SESSION CHECKPOINT — S95 recovery, refreshed S96 (2026-09-08): Phase 35 OPEN at gate 1; T0–T4 ☑; T5 ☑ (S1 10,180/10,180, 0 violations; backlog 1,099 → 51 ledgered classes; registry 3,135 groups); T6 ☑ (14 frozen / 4 retired / the guard 0 LIVE / tools-health OK); NEXT = T7 (S1 strict + C2c/C2d in tools-health; progress fields; the corrections published); the old sequence for reference: fix the tool (§2 step 2) → repair the registry → replay the suspect rejections → decide E_func_80168B70 → bucket `new`
## 🛑 SESSION CHECKPOINT — S95 recovery, refreshed S96 (2026-09-08): Phase 35 OPEN at gate 1; T0–T4 ☑; T5 ☑ (S1 10,180/10,180, 0 violations; backlog 1,099 → 51 ledgered classes; registry 3,135 groups); T6 ☑ (14 frozen / 4 retired / the guard 0 LIVE); T7 ☑ (S1 strict + C2c/C2d in tools-health, OK; 38 byte-variant same-address texts PENDING Drew); NEXT = T8 (the record); the old sequence for reference: fix the tool (§2 step 2) → repair the registry → replay the suspect rejections → decide E_func_80168B70 → bucket `new`
### 0. How to use this block
A fresh session (S96) reads CLAUDE.md's load order, replays THIS block verbatim, and resumes at §2 step 1. This block was written by S95, a
+40 -9
View File
@@ -44,7 +44,7 @@ sharing is source-level. The .ld interpose stays the library mechanism.
Usage: tools/dedup_integrate.py [--check] [--binary <alias>] [--allow-unsigned]
[--dedup config/dedup.us.yaml]
"""
import argparse, json, pathlib, re, sys
import argparse, json, os, pathlib, re, sys
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "tools"))
@@ -168,6 +168,19 @@ def check(groups, binary_filter=None, allow_unsigned=False):
failures = unvalidated = validated = 0
members_seen = members_c1 = 0
# C2d (Phase 35 T7): the per-instance source FORMS from the census (one scan per TU, cached by mtime/size) — a member's site must
# be an include of THIS group's source; a surviving private copy is the registry running ahead of the source.
_forms = {}
def site_form(binary, vram):
if not _forms:
sys.path.insert(0, str(ROOT / "tools"))
import share_census as _sc
aliases, dirs = _sc.fleet_and_dirs()
f, _notes = _sc.build_forms(aliases, dirs, os.cpu_count() or 4, use_cache=True)
_forms.update(f)
_forms.setdefault("__loaded__", {})
return _forms.get(binary, {}).get(vram)
for g in groups:
gid = g.get("id", "?")
@@ -195,14 +208,20 @@ def check(groups, binary_filter=None, allow_unsigned=False):
# ---- C2a′ (Phase 35 T2): a PLAIN-C header source must DEFINE `func` (a token occurrence is not a body) --------
# The parameterized form (SHARED_FN) defines through cpp and keeps the token check above; every other source is a
# plain-C header read by share_census.header_defs — the one reader of that form (R33).
if not PARAM_FUNC_RE.match(fn):
sys.path.insert(0, str(ROOT / "tools"))
import share_census
defined = {n for n, _ in share_census.header_defs(ROOT / src)}
if fn not in defined:
print(f"[FAIL] {gid}: `func: {fn}` occurs in {src} but that header does not DEFINE it "
f"(defines: {sorted(defined)[:4]}) — a declaration is not a shared body")
failures += 1; continue
sys.path.insert(0, str(ROOT / "tools"))
import share_census
defined = {n for n, _ in share_census.header_defs(ROOT / src)}
if not PARAM_FUNC_RE.match(fn) and fn not in defined:
print(f"[FAIL] {gid}: `func: {fn}` occurs in {src} but that header does not DEFINE it "
f"(defines: {sorted(defined)[:4]}) — a declaration is not a shared body")
failures += 1; continue
# ---- C2c (Phase 35 T7): ONE source, under src/shared/, defining exactly ONE function — the token the group names ----------
if not src.startswith("src/shared/"):
print(f"[FAIL] {gid}: source {src} is not under src/shared/ — a shared body has its one source there (S1)")
failures += 1; continue
if len(defined) != 1:
print(f"[FAIL] {gid}: {src} defines {len(defined)} functions {sorted(defined)[:4]} — a shared header defines exactly one (C2c)")
failures += 1; continue
tier, want = g["tier"], g.get("hash")
if not want:
@@ -218,6 +237,18 @@ def check(groups, binary_filter=None, allow_unsigned=False):
f"INCLUDE_ASM stub — the registry is claiming work that was never done")
failures += 1; ok = False; continue
# ---- C2d (Phase 35 T7): the member's SITE includes this group's source — no private copy survives ----------------
rec = site_form(b, vram)
if rec is None:
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) has no source form in the census (coverage, R32)")
failures += 1; ok = False; continue
if rec.get("form") not in ("include", "param-include"):
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) is listed as a member but its site is a {rec.get('form')} in {rec.get('tu')} — "
f"the registry runs ahead of the source (C2d)")
failures += 1; ok = False; continue
if rec.get("header") and rec["header"] != src:
print(f"[FAIL] {gid}: {b}:0x{vram:08x} ({name}) includes {rec['header']}, not this group's source {src} (C2d)")
failures += 1; ok = False; continue
# ---- C1: equivalence, against the ORIGINAL bytes ------------------------------------
idx = sig_for(b)
if idx is None:
+25 -1
View File
@@ -1139,6 +1139,14 @@ def fleet():
nverb = len(vm.get("rows", vm) if isinstance(vm, dict) else vm)
except Exception:
nverb = 0
_census = {}
try:
import json as _j
_cj = ROOT / ".run/P35/census/share_census.json"
if _cj.exists():
_census = _j.loads(_cj.read_text()).get("same_vram_unregistered", {})
except Exception:
_census = {}
return {
"schema": 1,
"note": "generated by tools/progress.py --fleet/--json from the committed sources — every number carries its denominator; regenerate, never edit",
@@ -1156,7 +1164,19 @@ def fleet():
},
"counts": {"real_c_functions": REAL, "dedup_shared_of_real": SHARED, "linked_psyq_objects": LINKED,
"verbatim_asm_bodies": nverb, "include_asm_stubs": STUBS, "non_matching": NM, "matchable": MATCH,
"dedup_groups": ngroups, "dedup_instances": nmembers},
"dedup_groups": ngroups, "dedup_instances": nmembers,
# Phase 35 T7 — "one source per unique function": bodies written ONCE in C = every instantiated function (REAL + EMPTY)
# minus the shared instances plus their one header each (derived from this run's own totals, R33); the duplicate
# copies still in source come from the census (.run/P35/census/share_census.json — every one is ledgered, S1)
"unique_function_bodies": REAL + EMPTY - nmembers + ngroups,
"duplicate_source_copies": _census.get("copies"), "duplicate_source_classes": _census.get("classes")},
"corrections": [ # dated snapshots of published-number moves with their cause and the command that shows them (R75)
{"date": "2026-09-08", "phase": "P35 T2", "metric": "fn_count.total", "delta": 7,
"cause": "seven overlays including the whale header but never registered were counted by neither classify nor the registry fold", "command": "tools/progress.py --fleet"},
{"date": "2026-09-08", "phase": "P35 T6", "metric": "fn_count.total", "delta": 459,
"cause": "macro sites invisible to the classifier became includes at T4: ov_SC03_015's 219 single-site macros, ov_SC03_118's 3, and the never-extended members of five under-listed groups", "command": "tools/progress.py --fleet"},
{"date": "2026-09-08", "phase": "P35 T6", "metric": "counts.real_c_functions", "delta": -10211,
"cause": "empty-bodied shared functions were folded into REAL under the macro form; the include form classifies them EMPTY (the instruction-weighted metrics are unchanged)", "command": "tools/progress.py --fleet"}],
"per_binary": [{"binary": r['binary'], "real": r['real'], "shared": r['shared'], "linked": r['linked'],
"byte_identical": r['byteident'], "matchable": r['matchable'],
"instr_matched": (wm['per_bin'].get(r['binary'], [None, None])[0] if wm else None),
@@ -1214,6 +1234,10 @@ def readme_block(d):
f"{c['linked_psyq_objects']:,} Sony PsyQ library functions linked from the SDK objects, not our C · "
f"{c['verbatim_asm_bodies']} hand-written-assembly bodies kept verbatim · {c['include_asm_stubs']} assembly stubs left · "
f"{c['non_matching']} non-matching functions.",
"", (f"One source per unique function (Phase 35): {c['unique_function_bodies']:,} function bodies written once in C "
f"({c['dedup_groups']:,} of them shared headers instantiated {c['dedup_instances']:,} times); "
+ (f"{c['duplicate_source_copies']:,} duplicate copies remain in {c['duplicate_source_classes']:,} ledgered classes "
f"(declaration conflicts left for the types phase)." if c.get('duplicate_source_copies') is not None else "the duplicate-copy census was not available in this run.")),
"", f"_Generated by `tools/progress.py --readme` from `docs/progress.json` — numbers are never typed by hand._",
README_END]
return "\n".join(out)
+54 -3
View File
@@ -642,6 +642,8 @@ def render_table(classes, summary, cov_notes, ctrl):
# the second oracle: duplicate definition TEXT across translation units (sig-blind)
# ----------------------------------------------------------------------------------------------------------------------
def text_duplicates(forms):
"""ANY-address duplicates: a name-blind normalized definition text present in >1 TU (the cross-address ones are the names
phase's — reported as a count, never a violation here)."""
by_text = collections.defaultdict(set)
for a, per in forms.items():
for addr, rec in per.items():
@@ -650,6 +652,19 @@ def text_duplicates(forms):
return {h: sorted(v) for h, v in by_text.items() if len({(a, tu) for a, tu, _ in v}) > 1}
def text_duplicates_same_addr(forms, excepted_sites=frozenset()):
"""S1's SECOND, sig-blind oracle (R34; --strict-text): the same normalized definition text, at the SAME address, still a private
`def` in >1 TU — exactly what the h_exact join must have shared and did not. Ledgered sites (the exception ledger's classes) are
excluded, as they are from the first oracle. Keyed by (text, addr) so the deferred cross-address duplicates never count."""
by_key = collections.defaultdict(set)
for a, per in forms.items():
for addr, rec in per.items():
if rec["form"] == "def" and rec.get("text_hash") and rec.get("nlines", 0) >= 2 and (a, addr) not in excepted_sites:
by_key[(rec["text_hash"], addr)].add((a, rec["tu"]))
# distinct TUs, not distinct aliases: a twin's instance resolves to its primary's TU (one source already)
return {k: sorted(v) for k, v in by_key.items() if len({tu for _, tu in v}) > 1}
# ----------------------------------------------------------------------------------------------------------------------
# self-test fixture (R39): every verdict and flag, in memory
# ----------------------------------------------------------------------------------------------------------------------
@@ -760,7 +775,7 @@ def main():
groups = load_groups()
exceptions = load_exceptions(a.exceptions)
verb = verbatim_instances()
classes, cov = classify(sigs, forms, groups, spaces, dirs, twins, twin_of, verb, exceptions, scope=a.scope)
classes, cov = classify(sigs, forms, groups, spaces, dirs, twins, twin_of, verb, exceptions, scope=a.scope, keep_instances=True)
summary = summarize(classes, cov, notes, aliases)
ctrl = controls(classes, sigs, forms)
macro_tokens = 0
@@ -801,8 +816,44 @@ def main():
bad = len(v)
if a.strict_macros and macro_tokens:
print(f"share_census: S1 — {macro_tokens} DEFINE_func_ site(s) remain under src/ (--strict-macros)"); bad += 1
if a.strict_text and dup_text:
print(f"share_census: S1 — {len(dup_text)} definition text(s) duplicated across TUs (--strict-text)"); bad += 1
if a.strict_text:
# the second oracle mirrors the first's scope: ledgered classes AND the gate-1 deferral (E/F cross-address classes — an
# empty body's hash spans every address, so its same-address pairs sit inside a deferred class) are excluded; the deferred
# same-address duplicates are PUBLISHED as the names phase's inheritance, never silently dropped (R41)
exc_sites = {(i_alias, int(addr_s, 16)) for c in classes if c["excepted"] for addr_s in c["addrs"] for i_alias in c["aliases"]}
def_sites = {(i_alias, int(addr_s, 16)) for c in classes if ("E" in c["flags"] or "F" in c["flags"]) for addr_s in c["addrs"] for i_alias in c["aliases"]}
dup_deferred = text_duplicates_same_addr(forms, exc_sites)
dup_rest = text_duplicates_same_addr(forms, exc_sites | def_sites)
n_def = len(dup_deferred) - len(dup_rest)
summary["same_address_text_duplicates_deferred"] = {"texts": n_def, "sites": sum(len(v) for v in dup_deferred.values()) - sum(len(v) for v in dup_rest.values())}
print(f"share_census: second oracle — {n_def:,} same-address duplicated texts sit inside the deferred cross-address classes "
f"({summary['same_address_text_duplicates_deferred']['sites']:,} sites; the names phase's inheritance)")
# what is left is either a class the byte join has (a VIOLATION the first oracle missed — must be 0) or the same text at
# the same address compiling to DIFFERENT bytes per binary (singleton h_exact classes: data addresses differ per overlay);
# the latter is one source the byte tier cannot register — published as PENDING (a decision, not a pass)
# a violation only when two DISTINCT TUs hold private copies of the SAME byte class at that address (a twin pair is one
# TU); every other same-address text duplicate is byte-variant across its TUs — the byte tier cannot register it
site_h = {(i["alias"], i["addr"]): c["h"] for c in classes for i in c["insts"]}
def _shared_class_across_tus(k, v):
by_h = {}
for a, tu in v:
h = site_h.get((a, k[1]))
if h:
by_h.setdefault(h, set()).add(tu)
return any(len(tus) > 1 for tus in by_h.values())
dup_same = {k: v for k, v in dup_rest.items() if _shared_class_across_tus(k, v)}
dup_pending = {k: v for k, v in dup_rest.items() if k not in dup_same}
summary["same_address_text_duplicates_pending"] = {"texts": len(dup_pending), "sites": sum(len(v) for v in dup_pending.values())}
if dup_pending:
print(f"share_census: second oracle — PENDING {len(dup_pending):,} same-address definition texts ({summary['same_address_text_duplicates_pending']['sites']:,} "
f"sites) are duplicated across TUs but compile to DIFFERENT bytes per binary (singleton h_exact classes) — one source the"
f" byte tier cannot register; a decision for the owner (an h_norm/text tier or the names phase)")
if dup_same:
print(f"share_census: S1 second oracle — {len(dup_same)} definition text(s) duplicated at the SAME address across TUs and not "
f"ledgered (--strict-text): " + "; ".join(f"0x{addr:08X} in {[a for a, _ in v][:4]}" for (_, addr), v in list(dup_same.items())[:6])); bad += 1
else:
print(f"share_census: S1 second oracle — 0 same-address definition texts duplicated across TUs (any-address duplicates "
f"{len(dup_text):,}, the names phase's)")
exc = sum(1 for c in classes if c["excepted"])
twinc = sum(1 for c in classes if "TWIN-COVERED" in c["flags"] and c["verdict"] != "A")
print(f"S1: one source per unique function — {len(classes):,} classes, {len(classes) - len(v):,} satisfied "