mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-07 09:27:55 -04:00
chore(phase-37): the size gate = GitHub's 100 MiB hard limit (was its 50 MiB warning) — the red no-rom CI was SIZE alone on the P36 judgement ledger (74.8 MiB, our own rows; ROM checks 1/2/4 clean), Drew's decision; audit_public WARNs above 80 MiB; workflow, wiki and SETUP wording follow | audit_public: OK — 0 offenders among 15219 tracked paths | 🛑 T4 NEXT
This commit is contained in:
@@ -4,7 +4,7 @@
|
||||
# original disc (`make check-all`, docs/verification.md). That needs the game disc, which is never in the repo
|
||||
# or in CI. So CI proves everything that does NOT need the disc:
|
||||
# * audits — the tracked tree is public-clean (tools/audit_public.py: no ROM-derived bytes, no purge path,
|
||||
# nothing > 50 MiB), every source is text with portable includes, the verbatim-asm manifest has
|
||||
# nothing > 100 MiB), every source is text with portable includes, the verbatim-asm manifest has
|
||||
# no drift, the cookbook index / Ghidra roster / work-evidence self-tests are fresh, the LZSS
|
||||
# decoder's unit tests pass, no stale symbol references;
|
||||
# * compile-only — the committed C still compiles with the pinned vintage toolchain (gcc-2.7.2 cc1 from the
|
||||
@@ -38,7 +38,7 @@ jobs:
|
||||
python-version: "3.12"
|
||||
- name: Python deps (the pinned venv set)
|
||||
run: python -m pip install --quiet -r requirements-python.txt
|
||||
- name: audit_public — no ROM-derived bytes, no purge path, nothing > 50 MiB among tracked files
|
||||
- name: audit_public — no ROM-derived bytes, no purge path, nothing > 100 MiB among tracked files
|
||||
run: python tools/audit_public.py
|
||||
- name: audit_text_sources — every source is text, every #include portable
|
||||
run: python tools/audit_text_sources.py
|
||||
|
||||
+1
-1
@@ -816,7 +816,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data/<project>/<version>/` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. |
|
||||
| | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) |
|
||||
| | `.venv/bin/git-filter-repo` 2.47.0 | (P33 C1) `pip install git-filter-repo==2.47.0` (in `requirements-python.txt`); used through its module API by `run_filter.py`. |
|
||||
| | `tools/audit_public.py [--paths …]` | **(P33 B7; extended P33.5 task 8) The first-push gate**, four checks: (1) no tracked file under `tools/public_rewrite/purge_set.txt` (the C1 rewrite's own input, filter-repo syntax) OR under the audit-only sibling `tools/public_rewrite/untracked_after_rewrite.txt` (paths untracked after the rewrite without a second rewrite — kept out of the purge set because `gate_scan.py` reads that as the history census; both files refuse to be empty, R43); (2) none whose SHA1 is ROM-derived (DERIVED set: every `sha1` in `extracted/retail/manifest.jsonl` + `config/check.*.sha` + the redump Track-1 SHA1; zero-length files exempt — the empty-file SHA1 is also SC04/SC05 `FILE_029/1.6`'s); (3) none > 50 MiB; (4) **CONTENT** — no tracked text file with ≥ 64 CONTIGUOUS disassembly-shaped lines (asm-differ incl. operand-less `nop`, objdump, splat `/* ADDR HEX hex */`, `glabel`) — the class-3 case path+hash cannot see. Names every offender, exits 1; prints the three longest runs. ≈2 s over 6,443 paths. Controls: S87 the purge set; P33.5 the two `fable_cd4` listings fail check 4 at 398/179 lines by `--paths`, the 40-line xsig fixtures pass at 14. |
|
||||
| | `tools/audit_public.py [--paths …]` | **(P33 B7; extended P33.5 task 8) The first-push gate**, four checks: (1) no tracked file under `tools/public_rewrite/purge_set.txt` (the C1 rewrite's own input, filter-repo syntax) OR under the audit-only sibling `tools/public_rewrite/untracked_after_rewrite.txt` (paths untracked after the rewrite without a second rewrite — kept out of the purge set because `gate_scan.py` reads that as the history census; both files refuse to be empty, R43); (2) none whose SHA1 is ROM-derived (DERIVED set: every `sha1` in `extracted/retail/manifest.jsonl` + `config/check.*.sha` + the redump Track-1 SHA1; zero-length files exempt — the empty-file SHA1 is also SC04/SC05 `FILE_029/1.6`'s); (3) none > 100 MiB (GitHub's hard limit; 50 MiB until 2026-09-12); (4) **CONTENT** — no tracked text file with ≥ 64 CONTIGUOUS disassembly-shaped lines (asm-differ incl. operand-less `nop`, objdump, splat `/* ADDR HEX hex */`, `glabel`) — the class-3 case path+hash cannot see. Names every offender, exits 1; prints the three longest runs. ≈2 s over 6,443 paths. Controls: S87 the purge set; P33.5 the two `fable_cd4` listings fail check 4 at 398/179 lines by `--paths`, the 40-line xsig fixtures pass at 14. |
|
||||
| | `tools/compile_only.py <aliases…> \| --all [-j N] [--list]` | **(P33 B7)** cpp → cc1 → maspsx → as on every eligible TU with the Makefile's flags PARSED at run time; TUs per binary from `<alias>_SRC_DIR` with nested-binary pruning (= the Makefile's `C_SRCS`); skips main's 70 LINKED tiles (`progress._main_linked_segs_from_makefile`) and the 47 `INCLUDE_ASM(`/`INCLUDE_RODATA(` TUs (they `.include` asm/); -O0 TUs (`corpus.o0_sources`) compile at -O0. Coverage line with every denominator. Measured: PR scope 54 of 124 TUs in 1.6 s; fleet 4,170 of 4,287 in 123 s at -j32 (≈50 CPU-min). |
|
||||
| | `tools/public_rewrite/purge_set.txt` | **(P33 B7)** THE purge set (Drew's decisions 3+11): the EXE at both historical paths, `glob:dumps/*.bin`, `ghidra/`, `tools/psyq/`, `session archive/`, `glob:tools/ghidra-ext/*.zip`, `tools/brave-CUE/brave.exe`. Read by audit_public now and by C1's `git filter-repo --paths-from-file` later. |
|
||||
| | `ExportAnnotations.java` + `tools/ghidra_export_annotations.sh` | **(P33 B5) The read-only TEXT export of a program** — byte-stable JSONL (fixed key order, sorted, `0x%08x`): `program`/`block`/`archive` container rows, LOCAL-archive types, every function signature (params/locals/storage/sources/comment), defined data, the 5 comment kinds, bookmarks, equates, labels not in the symbol files. `tools/ghidra_export_annotations.sh [PROG…]` → `.run/ghidra_export/<prog>.jsonl` (no arg = all programs in one `-readOnly` run; 129 in 18.5 s). MCP must be STOPPED. |
|
||||
|
||||
@@ -129,7 +129,7 @@ Thumbs.db
|
||||
2. **The audit** — `tools/audit_public.py`, the first-push gate reused as CI. It refuses any tracked file under a purged
|
||||
path; any tracked file whose SHA1 appears in the ROM-hash set, which it *derives* from the extraction manifest, every
|
||||
per-binary contract and the disc's own track hash rather than from a typed list (rule R33); anything over GitHub's
|
||||
50 MiB threshold; and, since Phase 33.5, any tracked text file with a long contiguous run of disassembly-shaped
|
||||
100 MiB threshold (GitHub's hard limit; raised from its 50 MiB warning on 2026-09-12); and, since Phase 33.5, any tracked text file with a long contiguous run of disassembly-shaped
|
||||
lines (an assembler listing, an objdump, a splat `/* ADDR HEX */` block) — the class-3 case the path and hash checks
|
||||
cannot see. It asserts its own coverage (a missing manifest, zero contracts or a short manifest is a failure, rule
|
||||
R32), refuses rules it cannot interpret rather than mishandling them (R43), prints every count with its denominator
|
||||
|
||||
@@ -55,7 +55,7 @@ every scanner in the repository prints its denominator.
|
||||
|
||||
[`.github/workflows/no-rom.yml`](../../.github/workflows/no-rom.yml) runs on every push and pull request without the
|
||||
disc: the tracked tree is public-clean ([`tools/audit_public.py`](../../tools/audit_public.py): no tracked file's hash
|
||||
appears in the ROM manifest or the contracts, no purge path, nothing over 50 MiB), every source is text with portable
|
||||
appears in the ROM manifest or the contracts, no purge path, nothing over 100 MiB), every source is text with portable
|
||||
includes, the verbatim manifest has no drift, the derived indexes are fresh, the LZSS decoder's unit tests pass, and
|
||||
every eligible translation unit compiles with the pinned `cc1` → maspsx → `as` and the Makefile's exact flags
|
||||
([`tools/compile_only.py`](../../tools/compile_only.py): four representative binaries per PR, the whole fleet weekly).
|
||||
|
||||
@@ -355,6 +355,15 @@ bucketed by what each bucket needs before any plan is proposed. **Candidates for
|
||||
that push. Not touched (P36's evidence; a decision): recommendation below the checkpoint. Also: 19,343 stale scratch files under
|
||||
`.run/P36/delever/obj/` (ignored; hygiene, not correctness).
|
||||
|
||||
- **S107 — Drew's decision on the red `no-rom` CI (the P36 ledger's size):** *"did we accidentally push rom info? or is that just a file >50
|
||||
failing our test? if so, just relax the size gate to 100MB per github limit"* → evidence shown: `audit_public`'s checks 1 (purge paths), 2
|
||||
(1,422 ROM-derived SHA1s vs 15,203 tracked files) and 4 (disassembly-shaped runs; longest 14 lines, cap 64) all CLEAN — only check 3 (SIZE)
|
||||
fired; the ledger is 29,649 rows of our own tool's verdicts on our own C (0 eight-hex-digit words outside the hash fields). **No ROM content
|
||||
was pushed.** The gate raised to GitHub's HARD limit: `tools/audit_public.py` `SIZE_CAP = 100 MiB` (+ a `WARN` above 80 MiB, so a file
|
||||
approaching the limit is announced before a push can be rejected — R54), the docstring, `.github/workflows/no-rom.yml`, SETUP's row,
|
||||
`docs/wiki/The-ROM-firewall.md` and `docs/wiki/Verification-and-progress.md` (the wiki's live copy needs Drew's sync). `audit_public: OK — 0
|
||||
offenders among 15219 tracked paths`. (The ledger is 74.8 MiB today; P36's rows are frozen — only P37's S+A rows are appended to it.)
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — S107 (2026-09-12, FINAL — written for a FRESH session; the session's last commit follows this): gate 1 APPROVED, **T0 ☑ T1 ☑ T2 ☑ T3 ☑** (baseline · the census + map · the probe · THE ENGINE) — 🛑 **T4 NEXT (the declaration layer over the fleet by symbol space — rung D in unattended cycles; xHigh) — and FIRST the P6 rules check (four tasks complete)** | R22 `check-all: 218 passed, 0 failed of 218` at `42a57f576`+`common.h` (`r22_t3c.log`; the close commit adds no build input) | HEAD after this commit is the checkpoint's commit; tree clean; nothing pushed after `79b2f6f15` (6 commits ahead: `0a55cb0fd` … this one — Drew pushes, R6)
|
||||
|
||||
**Replay this block into the chat at the next session start (R64); it is the ONLY in-phase context the next session inherits. Everything below
|
||||
@@ -375,11 +384,7 @@ readability series), then rewrite this block, commit, recap (R18).
|
||||
because a parameter cannot be typed while its body still adds offsets to it (pointer arithmetic scales). **A session-start owed check: the P6
|
||||
rules check** (T0–T3 = four tasks) — re-read CLAUDE.md's rules and PROJECT_CONTEXT.md's, state "Rules check — re-read complete. Continuing with
|
||||
T4." Commits this session: `21d2ccc14` (T3 bank 1: the tools + batch t3d1), `42a57f576` (batch t3d2), then this checkpoint's commit (the T3
|
||||
close: cycle script, macros, the record). **For Drew (found S107, untouched — a decision):** `tools/audit_public.py` is RED on HEAD:
|
||||
`.run/P36/delever/ledger.jsonl` is tracked at 78,396,681 bytes (> the 50 MiB cap) since P36 S105 `31cf8695f` — the public CI's `no-rom` job has
|
||||
been red since that push. Recommendation: freeze the P36 rows as `.run/P36/delever/ledger.p36.jsonl.gz` (tracked, ~6 MB), untrack the live
|
||||
`ledger.jsonl` (ignored-but-present; `delever.py` keeps appending to it; `--status` learns to read both), one commit; tools-health must be OK
|
||||
before any PhaseEnd. Also 19,343 stale scratch files under the ignored `.run/P36/delever/obj/` (hygiene).
|
||||
close: cycle script, macros, the record). **Resolved S107 (Drew):** the red `no-rom` CI was the SIZE check alone — `.run/P36/delever/ledger.jsonl` (74.8 MiB, our own tool's rows, no ROM content; checks 1/2/4 clean) — and the gate is now GitHub's 100 MiB hard limit (`SIZE_CAP`, a WARN above 80 MiB); `audit_public: OK`. The two edited wiki pages await Drew's wiki sync. Also 19,343 stale scratch files under the ignored `.run/P36/delever/obj/` (hygiene).
|
||||
|
||||
2. **The counters (T1, `tools/type_census.py`, coverage OK, controls 4/4; every verify line in the T1/T2 log entries).**
|
||||
Raw dereferences **503,016** = P `*(T *)(…)` 409,007 + I `*(T *)ident|&D_|0x80…` 60,666 + X `((T *)e)[i]` 13,800 + M `M2C_FIELD` 19,543,
|
||||
|
||||
+13
-5
@@ -13,7 +13,9 @@ Four checks, each derived from something the repo already asserts (R33), never f
|
||||
2. ROM CONTENT — no tracked file's SHA1 is a known ROM-derived hash: every `sha1` in extracted/retail/manifest.jsonl
|
||||
(the manifest IS the list of ROM-derived artifacts), every binary's SHA1 in config/check.*.sha, and the redump
|
||||
Track-1 SHA1 (tools/bfm_extract/extract_exe.REDUMP_TRACK1_SHA1). A renamed copy is caught by content.
|
||||
3. SIZE — no tracked file over 50 MiB (GitHub's warning threshold; nothing legitimately tracked is near it).
|
||||
3. SIZE — no tracked file over 100 MiB (GitHub's HARD limit: a push with a larger file is rejected; 50 MiB is only its
|
||||
warning). Raised from 50 MiB on 2026-09-12 (Drew, P37 S107) when the Phase-36 judgement ledger — our own tool's rows, no
|
||||
ROM content — reached 78 MB and the cap alone turned the public CI red.
|
||||
4. CONTENT — no tracked TEXT file carries a long contiguous run of disassembly-shaped lines (an assembler listing, an
|
||||
objdump, a splat `/* ADDR HEX hex */` block, a glabel block): the class-3 case a path-and-hash audit cannot see —
|
||||
a notes file that pastes a function's instructions is ROM-derived even when the tracked C reproduces the bytes.
|
||||
@@ -38,7 +40,8 @@ REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
PURGE = REPO / "tools" / "public_rewrite" / "purge_set.txt"
|
||||
UNTRACKED_AFTER = REPO / "tools" / "public_rewrite" / "untracked_after_rewrite.txt"
|
||||
MANIFEST = REPO / "extracted" / "retail" / "manifest.jsonl"
|
||||
SIZE_CAP = 50 * 1024 * 1024
|
||||
SIZE_CAP = 100 * 1024 * 1024
|
||||
SIZE_WARN = 80 * 1024 * 1024 # a tracked file this close to GitHub's hard limit is announced before a push can be rejected (R54)
|
||||
RUN_CAP = 64 # contiguous disassembly-shaped lines that make a text file an offender
|
||||
DISASM_RES = [
|
||||
# asm-differ: `12: addiu $sp, $sp, -0x40`, `44: nop`, `50: j .L80133D40` (an operand-less mnemonic and a
|
||||
@@ -147,6 +150,7 @@ def main(argv):
|
||||
u_prefixes, u_globs = read_rules(UNTRACKED_AFTER, "untracked-after-rewrite")
|
||||
hashes, n_manifest, n_checks = rom_hashes()
|
||||
offenders = []
|
||||
warnings = []
|
||||
n_hashed = n_text = 0
|
||||
top_runs = []
|
||||
for rel in files:
|
||||
@@ -161,7 +165,9 @@ def main(argv):
|
||||
continue
|
||||
size = p.stat().st_size
|
||||
if size > SIZE_CAP:
|
||||
offenders.append((rel, f"{size:,} bytes > 50 MiB"))
|
||||
offenders.append((rel, f"{size:,} bytes > 100 MiB"))
|
||||
elif size > SIZE_WARN:
|
||||
warnings.append((rel, f"{size:,} bytes — within 20 MiB of GitHub's 100 MiB hard limit"))
|
||||
if size == 0: # the empty-file SHA1 is also a zero-length disc payload's (SC05/029/1.6) — not ROM bytes
|
||||
continue
|
||||
h = sha1_of(p)
|
||||
@@ -179,14 +185,16 @@ def main(argv):
|
||||
print(f"audit_public: {len(files)} tracked paths, {n_hashed} files hashed against {len(hashes)} ROM hashes "
|
||||
f"({n_manifest} manifest rows + {n_checks} check.*.sha + redump), "
|
||||
f"{len(p_prefixes) + len(p_globs)} purge rules + {len(u_prefixes) + len(u_globs)} untracked-after-rewrite rules, "
|
||||
f"cap 50 MiB; {n_text} text files scanned for disassembly runs (cap {RUN_CAP} lines), "
|
||||
f"cap 100 MiB; {n_text} text files scanned for disassembly runs (cap {RUN_CAP} lines), "
|
||||
f"longest runs: {', '.join(f'{r} {p}' for r, p in top_runs[:3]) or 'none'}")
|
||||
for rel, why in warnings:
|
||||
print(f" WARN {rel}: {why}")
|
||||
if offenders:
|
||||
for rel, why in offenders:
|
||||
print(f" OFFENDER {rel}: {why}")
|
||||
print(f"audit_public: FAIL — {len(offenders)} offender(s) among {len(files)} tracked paths")
|
||||
return 1
|
||||
print(f"audit_public: OK — 0 offenders among {len(files)} tracked paths")
|
||||
print(f"audit_public: OK — 0 offenders among {len(files)} tracked paths" + (f" ({len(warnings)} size warning(s))" if warnings else ""))
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user