diff --git a/.github/workflows/no-rom.yml b/.github/workflows/no-rom.yml index 36db3cefcf..397a169f3e 100644 --- a/.github/workflows/no-rom.yml +++ b/.github/workflows/no-rom.yml @@ -4,7 +4,7 @@ # original disc (`make check-all`, docs/verification.md). That needs the game disc, which is never in the repo # or in CI. So CI proves everything that does NOT need the disc: # * audits — the tracked tree is public-clean (tools/audit_public.py: no ROM-derived bytes, no purge path, -# nothing > 50 MiB), every source is text with portable includes, the verbatim-asm manifest has +# nothing > 100 MiB), every source is text with portable includes, the verbatim-asm manifest has # no drift, the cookbook index / Ghidra roster / work-evidence self-tests are fresh, the LZSS # decoder's unit tests pass, no stale symbol references; # * compile-only — the committed C still compiles with the pinned vintage toolchain (gcc-2.7.2 cc1 from the @@ -38,7 +38,7 @@ jobs: python-version: "3.12" - name: Python deps (the pinned venv set) run: python -m pip install --quiet -r requirements-python.txt - - name: audit_public — no ROM-derived bytes, no purge path, nothing > 50 MiB among tracked files + - name: audit_public — no ROM-derived bytes, no purge path, nothing > 100 MiB among tracked files run: python tools/audit_public.py - name: audit_text_sources — every source is text, every #include portable run: python tools/audit_text_sources.py diff --git a/docs/SETUP.md b/docs/SETUP.md index f8cf2f9835..4417675599 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -816,7 +816,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data///` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. | | | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) | | | `.venv/bin/git-filter-repo` 2.47.0 | (P33 C1) `pip install git-filter-repo==2.47.0` (in `requirements-python.txt`); used through its module API by `run_filter.py`. | -| | `tools/audit_public.py [--paths …]` | **(P33 B7; extended P33.5 task 8) The first-push gate**, four checks: (1) no tracked file under `tools/public_rewrite/purge_set.txt` (the C1 rewrite's own input, filter-repo syntax) OR under the audit-only sibling `tools/public_rewrite/untracked_after_rewrite.txt` (paths untracked after the rewrite without a second rewrite — kept out of the purge set because `gate_scan.py` reads that as the history census; both files refuse to be empty, R43); (2) none whose SHA1 is ROM-derived (DERIVED set: every `sha1` in `extracted/retail/manifest.jsonl` + `config/check.*.sha` + the redump Track-1 SHA1; zero-length files exempt — the empty-file SHA1 is also SC04/SC05 `FILE_029/1.6`'s); (3) none > 50 MiB; (4) **CONTENT** — no tracked text file with ≥ 64 CONTIGUOUS disassembly-shaped lines (asm-differ incl. operand-less `nop`, objdump, splat `/* ADDR HEX hex */`, `glabel`) — the class-3 case path+hash cannot see. Names every offender, exits 1; prints the three longest runs. ≈2 s over 6,443 paths. Controls: S87 the purge set; P33.5 the two `fable_cd4` listings fail check 4 at 398/179 lines by `--paths`, the 40-line xsig fixtures pass at 14. | +| | `tools/audit_public.py [--paths …]` | **(P33 B7; extended P33.5 task 8) The first-push gate**, four checks: (1) no tracked file under `tools/public_rewrite/purge_set.txt` (the C1 rewrite's own input, filter-repo syntax) OR under the audit-only sibling `tools/public_rewrite/untracked_after_rewrite.txt` (paths untracked after the rewrite without a second rewrite — kept out of the purge set because `gate_scan.py` reads that as the history census; both files refuse to be empty, R43); (2) none whose SHA1 is ROM-derived (DERIVED set: every `sha1` in `extracted/retail/manifest.jsonl` + `config/check.*.sha` + the redump Track-1 SHA1; zero-length files exempt — the empty-file SHA1 is also SC04/SC05 `FILE_029/1.6`'s); (3) none > 100 MiB (GitHub's hard limit; 50 MiB until 2026-09-12); (4) **CONTENT** — no tracked text file with ≥ 64 CONTIGUOUS disassembly-shaped lines (asm-differ incl. operand-less `nop`, objdump, splat `/* ADDR HEX hex */`, `glabel`) — the class-3 case path+hash cannot see. Names every offender, exits 1; prints the three longest runs. ≈2 s over 6,443 paths. Controls: S87 the purge set; P33.5 the two `fable_cd4` listings fail check 4 at 398/179 lines by `--paths`, the 40-line xsig fixtures pass at 14. | | | `tools/compile_only.py \| --all [-j N] [--list]` | **(P33 B7)** cpp → cc1 → maspsx → as on every eligible TU with the Makefile's flags PARSED at run time; TUs per binary from `_SRC_DIR` with nested-binary pruning (= the Makefile's `C_SRCS`); skips main's 70 LINKED tiles (`progress._main_linked_segs_from_makefile`) and the 47 `INCLUDE_ASM(`/`INCLUDE_RODATA(` TUs (they `.include` asm/); -O0 TUs (`corpus.o0_sources`) compile at -O0. Coverage line with every denominator. Measured: PR scope 54 of 124 TUs in 1.6 s; fleet 4,170 of 4,287 in 123 s at -j32 (≈50 CPU-min). | | | `tools/public_rewrite/purge_set.txt` | **(P33 B7)** THE purge set (Drew's decisions 3+11): the EXE at both historical paths, `glob:dumps/*.bin`, `ghidra/`, `tools/psyq/`, `session archive/`, `glob:tools/ghidra-ext/*.zip`, `tools/brave-CUE/brave.exe`. Read by audit_public now and by C1's `git filter-repo --paths-from-file` later. | | | `ExportAnnotations.java` + `tools/ghidra_export_annotations.sh` | **(P33 B5) The read-only TEXT export of a program** — byte-stable JSONL (fixed key order, sorted, `0x%08x`): `program`/`block`/`archive` container rows, LOCAL-archive types, every function signature (params/locals/storage/sources/comment), defined data, the 5 comment kinds, bookmarks, equates, labels not in the symbol files. `tools/ghidra_export_annotations.sh [PROG…]` → `.run/ghidra_export/.jsonl` (no arg = all programs in one `-readOnly` run; 129 in 18.5 s). MCP must be STOPPED. | diff --git a/docs/wiki/The-ROM-firewall.md b/docs/wiki/The-ROM-firewall.md index ad407d375c..664d148543 100644 --- a/docs/wiki/The-ROM-firewall.md +++ b/docs/wiki/The-ROM-firewall.md @@ -129,7 +129,7 @@ Thumbs.db 2. **The audit** — `tools/audit_public.py`, the first-push gate reused as CI. It refuses any tracked file under a purged path; any tracked file whose SHA1 appears in the ROM-hash set, which it *derives* from the extraction manifest, every per-binary contract and the disc's own track hash rather than from a typed list (rule R33); anything over GitHub's - 50 MiB threshold; and, since Phase 33.5, any tracked text file with a long contiguous run of disassembly-shaped + 100 MiB threshold (GitHub's hard limit; raised from its 50 MiB warning on 2026-09-12); and, since Phase 33.5, any tracked text file with a long contiguous run of disassembly-shaped lines (an assembler listing, an objdump, a splat `/* ADDR HEX */` block) — the class-3 case the path and hash checks cannot see. It asserts its own coverage (a missing manifest, zero contracts or a short manifest is a failure, rule R32), refuses rules it cannot interpret rather than mishandling them (R43), prints every count with its denominator diff --git a/docs/wiki/Verification-and-progress.md b/docs/wiki/Verification-and-progress.md index 0bff884b07..44ed46b7a5 100644 --- a/docs/wiki/Verification-and-progress.md +++ b/docs/wiki/Verification-and-progress.md @@ -55,7 +55,7 @@ every scanner in the repository prints its denominator. [`.github/workflows/no-rom.yml`](../../.github/workflows/no-rom.yml) runs on every push and pull request without the disc: the tracked tree is public-clean ([`tools/audit_public.py`](../../tools/audit_public.py): no tracked file's hash -appears in the ROM manifest or the contracts, no purge path, nothing over 50 MiB), every source is text with portable +appears in the ROM manifest or the contracts, no purge path, nothing over 100 MiB), every source is text with portable includes, the verbatim manifest has no drift, the derived indexes are fresh, the LZSS decoder's unit tests pass, and every eligible translation unit compiles with the pinned `cc1` → maspsx → `as` and the Makefile's exact flags ([`tools/compile_only.py`](../../tools/compile_only.py): four representative binaries per PR, the whole fleet weekly). diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 0372d52c7d..02f97bbd35 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -355,6 +355,15 @@ bucketed by what each bucket needs before any plan is proposed. **Candidates for that push. Not touched (P36's evidence; a decision): recommendation below the checkpoint. Also: 19,343 stale scratch files under `.run/P36/delever/obj/` (ignored; hygiene, not correctness). +- **S107 — Drew's decision on the red `no-rom` CI (the P36 ledger's size):** *"did we accidentally push rom info? or is that just a file >50 + failing our test? if so, just relax the size gate to 100MB per github limit"* → evidence shown: `audit_public`'s checks 1 (purge paths), 2 + (1,422 ROM-derived SHA1s vs 15,203 tracked files) and 4 (disassembly-shaped runs; longest 14 lines, cap 64) all CLEAN — only check 3 (SIZE) + fired; the ledger is 29,649 rows of our own tool's verdicts on our own C (0 eight-hex-digit words outside the hash fields). **No ROM content + was pushed.** The gate raised to GitHub's HARD limit: `tools/audit_public.py` `SIZE_CAP = 100 MiB` (+ a `WARN` above 80 MiB, so a file + approaching the limit is announced before a push can be rejected — R54), the docstring, `.github/workflows/no-rom.yml`, SETUP's row, + `docs/wiki/The-ROM-firewall.md` and `docs/wiki/Verification-and-progress.md` (the wiki's live copy needs Drew's sync). `audit_public: OK — 0 + offenders among 15219 tracked paths`. (The ledger is 74.8 MiB today; P36's rows are frozen — only P37's S+A rows are appended to it.) + ## 🛑 SESSION CHECKPOINT — S107 (2026-09-12, FINAL — written for a FRESH session; the session's last commit follows this): gate 1 APPROVED, **T0 ☑ T1 ☑ T2 ☑ T3 ☑** (baseline · the census + map · the probe · THE ENGINE) — 🛑 **T4 NEXT (the declaration layer over the fleet by symbol space — rung D in unattended cycles; xHigh) — and FIRST the P6 rules check (four tasks complete)** | R22 `check-all: 218 passed, 0 failed of 218` at `42a57f576`+`common.h` (`r22_t3c.log`; the close commit adds no build input) | HEAD after this commit is the checkpoint's commit; tree clean; nothing pushed after `79b2f6f15` (6 commits ahead: `0a55cb0fd` … this one — Drew pushes, R6) **Replay this block into the chat at the next session start (R64); it is the ONLY in-phase context the next session inherits. Everything below @@ -375,11 +384,7 @@ readability series), then rewrite this block, commit, recap (R18). because a parameter cannot be typed while its body still adds offsets to it (pointer arithmetic scales). **A session-start owed check: the P6 rules check** (T0–T3 = four tasks) — re-read CLAUDE.md's rules and PROJECT_CONTEXT.md's, state "Rules check — re-read complete. Continuing with T4." Commits this session: `21d2ccc14` (T3 bank 1: the tools + batch t3d1), `42a57f576` (batch t3d2), then this checkpoint's commit (the T3 - close: cycle script, macros, the record). **For Drew (found S107, untouched — a decision):** `tools/audit_public.py` is RED on HEAD: - `.run/P36/delever/ledger.jsonl` is tracked at 78,396,681 bytes (> the 50 MiB cap) since P36 S105 `31cf8695f` — the public CI's `no-rom` job has - been red since that push. Recommendation: freeze the P36 rows as `.run/P36/delever/ledger.p36.jsonl.gz` (tracked, ~6 MB), untrack the live - `ledger.jsonl` (ignored-but-present; `delever.py` keeps appending to it; `--status` learns to read both), one commit; tools-health must be OK - before any PhaseEnd. Also 19,343 stale scratch files under the ignored `.run/P36/delever/obj/` (hygiene). + close: cycle script, macros, the record). **Resolved S107 (Drew):** the red `no-rom` CI was the SIZE check alone — `.run/P36/delever/ledger.jsonl` (74.8 MiB, our own tool's rows, no ROM content; checks 1/2/4 clean) — and the gate is now GitHub's 100 MiB hard limit (`SIZE_CAP`, a WARN above 80 MiB); `audit_public: OK`. The two edited wiki pages await Drew's wiki sync. Also 19,343 stale scratch files under the ignored `.run/P36/delever/obj/` (hygiene). 2. **The counters (T1, `tools/type_census.py`, coverage OK, controls 4/4; every verify line in the T1/T2 log entries).** Raw dereferences **503,016** = P `*(T *)(…)` 409,007 + I `*(T *)ident|&D_|0x80…` 60,666 + X `((T *)e)[i]` 13,800 + M `M2C_FIELD` 19,543, diff --git a/tools/audit_public.py b/tools/audit_public.py index c5b239f471..19715f925d 100644 --- a/tools/audit_public.py +++ b/tools/audit_public.py @@ -13,7 +13,9 @@ Four checks, each derived from something the repo already asserts (R33), never f 2. ROM CONTENT — no tracked file's SHA1 is a known ROM-derived hash: every `sha1` in extracted/retail/manifest.jsonl (the manifest IS the list of ROM-derived artifacts), every binary's SHA1 in config/check.*.sha, and the redump Track-1 SHA1 (tools/bfm_extract/extract_exe.REDUMP_TRACK1_SHA1). A renamed copy is caught by content. - 3. SIZE — no tracked file over 50 MiB (GitHub's warning threshold; nothing legitimately tracked is near it). + 3. SIZE — no tracked file over 100 MiB (GitHub's HARD limit: a push with a larger file is rejected; 50 MiB is only its + warning). Raised from 50 MiB on 2026-09-12 (Drew, P37 S107) when the Phase-36 judgement ledger — our own tool's rows, no + ROM content — reached 78 MB and the cap alone turned the public CI red. 4. CONTENT — no tracked TEXT file carries a long contiguous run of disassembly-shaped lines (an assembler listing, an objdump, a splat `/* ADDR HEX hex */` block, a glabel block): the class-3 case a path-and-hash audit cannot see — a notes file that pastes a function's instructions is ROM-derived even when the tracked C reproduces the bytes. @@ -38,7 +40,8 @@ REPO = pathlib.Path(__file__).resolve().parent.parent PURGE = REPO / "tools" / "public_rewrite" / "purge_set.txt" UNTRACKED_AFTER = REPO / "tools" / "public_rewrite" / "untracked_after_rewrite.txt" MANIFEST = REPO / "extracted" / "retail" / "manifest.jsonl" -SIZE_CAP = 50 * 1024 * 1024 +SIZE_CAP = 100 * 1024 * 1024 +SIZE_WARN = 80 * 1024 * 1024 # a tracked file this close to GitHub's hard limit is announced before a push can be rejected (R54) RUN_CAP = 64 # contiguous disassembly-shaped lines that make a text file an offender DISASM_RES = [ # asm-differ: `12: addiu $sp, $sp, -0x40`, `44: nop`, `50: j .L80133D40` (an operand-less mnemonic and a @@ -147,6 +150,7 @@ def main(argv): u_prefixes, u_globs = read_rules(UNTRACKED_AFTER, "untracked-after-rewrite") hashes, n_manifest, n_checks = rom_hashes() offenders = [] + warnings = [] n_hashed = n_text = 0 top_runs = [] for rel in files: @@ -161,7 +165,9 @@ def main(argv): continue size = p.stat().st_size if size > SIZE_CAP: - offenders.append((rel, f"{size:,} bytes > 50 MiB")) + offenders.append((rel, f"{size:,} bytes > 100 MiB")) + elif size > SIZE_WARN: + warnings.append((rel, f"{size:,} bytes — within 20 MiB of GitHub's 100 MiB hard limit")) if size == 0: # the empty-file SHA1 is also a zero-length disc payload's (SC05/029/1.6) — not ROM bytes continue h = sha1_of(p) @@ -179,14 +185,16 @@ def main(argv): print(f"audit_public: {len(files)} tracked paths, {n_hashed} files hashed against {len(hashes)} ROM hashes " f"({n_manifest} manifest rows + {n_checks} check.*.sha + redump), " f"{len(p_prefixes) + len(p_globs)} purge rules + {len(u_prefixes) + len(u_globs)} untracked-after-rewrite rules, " - f"cap 50 MiB; {n_text} text files scanned for disassembly runs (cap {RUN_CAP} lines), " + f"cap 100 MiB; {n_text} text files scanned for disassembly runs (cap {RUN_CAP} lines), " f"longest runs: {', '.join(f'{r} {p}' for r, p in top_runs[:3]) or 'none'}") + for rel, why in warnings: + print(f" WARN {rel}: {why}") if offenders: for rel, why in offenders: print(f" OFFENDER {rel}: {why}") print(f"audit_public: FAIL — {len(offenders)} offender(s) among {len(files)} tracked paths") return 1 - print(f"audit_public: OK — 0 offenders among {len(files)} tracked paths") + print(f"audit_public: OK — 0 offenders among {len(files)} tracked paths" + (f" ({len(warnings)} size warning(s))" if warnings else "")) return 0