mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 05:56:00 -04:00
feat(phase-33): D3 progress publishing — per-binary instruction totals in docs/progress.json, docs/badges/*.json (shields endpoint), --check covers JSON + README block + badges (in make audit-digest), tools/objdiff_report.py (report.proto v2; validated with objdiff-cli 3.8.1 incl. a mutated-unit control), tools/frogress_upload.py (dry-run default), .github/workflows/progress.yml (artifact SLUS_007.26_report, no rebuild); SETUP rows + P33 D1–D3 section; checkpoint -> NEXT = D4
This commit is contained in:
Drew T
@@ -0,0 +1,37 @@
|
||||
# .github/workflows/progress.yml — publish the progress report for decomp.dev (P33 D3).
|
||||
#
|
||||
# CI has no game disc, so it never rebuilds the game. The numbers come from the COMMITTED docs/progress.json, which
|
||||
# `make report` regenerates from a local clean rebuild (and `make audit-digest` asserts describes the tree). This
|
||||
# workflow converts that file to objdiff's report format (tools/objdiff_report.py, validated with objdiff-cli) and
|
||||
# uploads it as the artifact decomp.dev reads: `<VERSION>_report` = SLUS_007.26_report.
|
||||
name: progress
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
report:
|
||||
name: objdiff report from docs/progress.json (no rebuild)
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: false
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Convert the committed progress.json to an objdiff report
|
||||
run: python tools/objdiff_report.py --in docs/progress.json --out report.json
|
||||
- name: Print the headline
|
||||
run: python -c "import json; m=json.load(open('report.json'))['measures']; print(f\"code {m['matched_code']}/{m['total_code']} = {m['matched_code_percent']:.2f}% functions {m['matched_functions']}/{m['total_functions']} units {m['complete_units']}/{m['total_units']}\")"
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: SLUS_007.26_report
|
||||
path: report.json
|
||||
if-no-files-found: error
|
||||
@@ -228,6 +228,9 @@ audit-disc:
|
||||
# phantom. R34: the byte-gate is a null oracle for documents, so this is a second one that disagrees.
|
||||
audit-digest:
|
||||
$(VENV_PY) tools/audit_digest.py
|
||||
# P33 D3: the published DATA must describe the current tree too — docs/progress.json, the README's
|
||||
# generated block and the badge files (the same numbers as the digest; R51: never typed)
|
||||
$(VENV_PY) tools/progress.py --json --readme --check
|
||||
|
||||
# P30 S39 (Drew's MASTER_REMAINING proposal, derived form — docs/decision-log.md 2026-08-04):
|
||||
# "what's left" is answered by six artifacts, each individually derived and NONE ever checked
|
||||
|
||||
@@ -749,6 +749,9 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/<prog>_funcs.txt`) — completes a raw-blob program's function set (Phase 10). |
|
||||
| | `ApplySymbols.java` + `tools/ghidra_apply_symbols.sh` | **(P31 S78) The Ghidra MIRROR of the curated symbol file (R15/G6), headless with a real save.** `tools/ghidra_apply_symbols.sh [PROG] [symbols files…]` (defaults `SLUS_007.26 config/symbols.us.txt`; MCP must be STOPPED first) reads `name = 0xADDR;` rows and sets every function/label to its curated name; a name held by another address is moved to that address's own curated name first (`firstfile`/`firstfile2`), else to `<name>__at_<addr>`. Idempotent; prints `BFMAPPLY renamed_funcs=… unchanged=…`; R9-verify with `ghidra_mcp_verify.sh`. **Use this, not MCP `rename_symbol`/`batch_rename`, for renames:** S78 observed 47 MCP renames NOT persisting through the sentinel stop ("Save succeeded", DB grew, names gone — R9 caught it; cause not yet isolated), while the postScript path persisted 73/73 on the first run. |
|
||||
| **Public flip / CI** | `.github/workflows/no-rom.yml` | **(P33 B7) The ROM-free CI**: job `audits` (audit_public, audit_text_sources, verbatim_check --strict, cookbook_index --check, ghidra_roster --check, work_evidence --selftest, test_lzss, lint_symbol_refs — ≈45 s of checks) + job `compile-only` (binutils-mipsel + `cpp-mipsel-linux-gnu` from apt, cc1 from the tracked tarball sha256-checked, maspsx submodule; PR scope `main resident ov_SC01_077 md_MAIN_013`; `--all` weekly Mon 06:17 UTC + `workflow_dispatch`). Byte-identity is NOT proven in CI (needs the disc) — `docs/verification.md`. |
|
||||
| **Publishing** | `tools/progress.py --json \| --readme [--check]` | **(P33 D1/D3)** The same numbers as DATA: `--json` → `docs/progress.json` (schema 1: the four metrics with numerator/denominator/pct, the counts, 218 per-binary rows incl. instruction totals; no run date) + `docs/badges/{fleet_instr,fleet_fn,distinct,binaries}.json` (shields endpoint format; the README references `fleet_instr` + `binaries` by name); `--readme` rewrites the README's `<!-- progress:begin/end -->` block (refuses a README without the markers); `--check` asserts JSON + block + badges are fresh (in `make audit-digest`). Run by `make report BINARY=main`. |
|
||||
| | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). |
|
||||
| | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data/<project>/<version>/` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. |
|
||||
| | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. |
|
||||
| | `.venv/bin/git-filter-repo` 2.47.0 | (P33 C1) `pip install git-filter-repo==2.47.0` (in `requirements-python.txt`); used through its module API by `run_filter.py`. |
|
||||
| | `tools/audit_public.py [--paths …]` | **(P33 B7) The first-push gate**: no tracked file under `tools/public_rewrite/purge_set.txt` (the C1 rewrite's own input, filter-repo syntax), none whose SHA1 is ROM-derived (DERIVED set: every `sha1` in `extracted/retail/manifest.jsonl` + `config/check.*.sha` + the redump Track-1 SHA1; zero-length files exempt — the empty-file SHA1 is also SC04/SC05 `FILE_029/1.6`'s), none > 50 MiB. Names every offender, exits 1. ≈1 s over 6,798 paths. |
|
||||
@@ -1132,6 +1135,20 @@ fills fast). Nothing is leaking — but the host does not get the memory back on
|
||||
`tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh <addr> <name>
|
||||
[PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`.
|
||||
|
||||
### P33 D1–D3 (S87, 2026-09-07) — the README and the published numbers, generated never typed
|
||||
- **D1 README:** rewritten from the Phase-19 text (every number was stale). Its numbers block is generated: `tools/progress.py
|
||||
--json` writes `docs/progress.json` (every metric with its denominator, R41; no run date so `make report` never churns it)
|
||||
and `--readme` rewrites the block between `<!-- progress:begin -->` / `<!-- progress:end -->`; `--check` asserts both (and the
|
||||
badges) are fresh and sits in `make audit-digest`; `make report BINARY=main` regenerates all of it after the fleet digest.
|
||||
- **D2:** `LICENSE` (AGPL-3.0 verbatim from gnu.org, 661 lines), `src/NOTICE.md` (no license asserted over `src/`; never the
|
||||
phrase "clean-room"), `THIRD_PARTY.md` (every license from the upstream LICENSE file or the GitHub API's license metadata:
|
||||
old-gcc and ghidra_psx_ldr publish NONE — download-only rows), `tools/README.md` (the one AGPL statement).
|
||||
- **D3:** badges (shields endpoint JSON, colour by percentage), `tools/objdiff_report.py` (validated: `objdiff-cli report
|
||||
changes report.json report.json` parses and diffs it; a mutated unit shows as a change), `tools/frogress_upload.py`
|
||||
(dry-run default), `.github/workflows/progress.yml` (artifact `SLUS_007.26_report`). CI never rebuilds the game: the report
|
||||
is the committed `docs/progress.json` from a local clean rebuild. Outward actions after the flip (Drew): decomp.dev
|
||||
registration, the frogress slug + key. The `objdiff-cli` binary used for validation lives in scratch (`.run/objdiff/`).
|
||||
|
||||
### P33 C1 (S87, 2026-09-07) — the history-rewrite tooling, measured before the irreversible run
|
||||
- **Design points.** The scrub replaces a hex token only when the WHOLE token is a prefix (≥ 7) of an old commit hash —
|
||||
so a 16-char sig hash can never be mistaken for a commit; only 7–8-char tokens carry any false-positive risk (≈1.6e-5
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion": 1, "label": "byte-identical binaries", "message": "218/218", "color": "brightgreen"}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion": 1, "label": "distinct code matched", "message": "100.0%", "color": "brightgreen"}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion": 1, "label": "matched (functions)", "message": "100.00%", "color": "brightgreen"}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion": 1, "label": "matched (instructions)", "message": "100.0%", "color": "brightgreen"}
|
||||
+654
-218
File diff suppressed because it is too large
Load Diff
+23
-10
@@ -73,7 +73,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo
|
||||
- [ ] **C10** Support purge → probe gate → FLIP (Drew) — decision Max
|
||||
- [x] **D1** README rewrite — Max — see Log 2026-09-07 D1
|
||||
- [x] **D2** LICENSE + `src/NOTICE.md` + `THIRD_PARTY.md` — Max wording — see Log 2026-09-07 D2
|
||||
- [ ] **D3** Progress publishing (progress.json, badges, frogress_upload, objdiff_report + progress.yml) — xHigh
|
||||
- [x] **D3** Progress publishing (progress.json, badges, frogress_upload, objdiff_report + progress.yml) — xHigh — see Log 2026-09-07 D3
|
||||
- [ ] **D4** SETUP.md public-clean pass — xHigh
|
||||
- [ ] **D5** Governing-docs consistency pass + `tools/doc_links.py` — xHigh
|
||||
- [ ] **F1** timeline.py + story — xHigh tool / Max narrative
|
||||
@@ -374,14 +374,26 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact
|
||||
never distributed; the AP world cited as facts). `tools/README.md` (new): the one AGPL statement for `tools/`, no per-file
|
||||
SPDX headers. **For Drew:** `docs/history/project_architect_v1.3.0.md` is KEPT as the historical record of the framework
|
||||
version used (THIRD_PARTY row links the repo) — say if you prefer link-only. Commit: see below.
|
||||
- **2026-09-07 (S87) — D3 progress publishing.** `progress.py`: `weighted_metrics` now returns per-binary instruction totals
|
||||
(`per_bin`, from the existing loop), `--json` rows carry `instr_matched/instr_total` (main 45,150/45,150, resident
|
||||
4,483/4,483 — known-true check), and `docs/badges/{fleet_instr,fleet_fn,distinct,binaries}.json` are written in the shields
|
||||
endpoint format; `--check` also asserts the badges (control: a mutated badge → STALE). `make audit-digest` runs
|
||||
`progress.py --json --readme --check`. `tools/objdiff_report.py`: report.proto v2 (snake_case) — 218 units all complete,
|
||||
code 53,968,452/53,968,452 bytes (= 13,492,113 × 4, reconciles), functions 363,214/363,214, categories `game-code` +
|
||||
`linked-sony-objects`; **validated with the real `objdiff-cli` 3.8.1** (`report changes a a` parses and reports no change;
|
||||
a mutated unit is reported — control) — its `changes` takes POSITIONAL previous/current. `tools/frogress_upload.py`: dry run
|
||||
prints the payload (git_hash, timestamp, the measures with denominators); `--push` needs `FROGRESS_API_SECRET`.
|
||||
`.github/workflows/progress.yml`: converts the committed JSON, uploads `SLUS_007.26_report` (no rebuild in CI). SETUP: 3
|
||||
inventory rows + a P33 D1–D3 section (R21). Outward (Drew, after the flip): decomp.dev registration, frogress slug + key.
|
||||
Commit: see below.
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D2 ✓; C10 IN PROGRESS ON DREW'S SIDE (the Support ticket + the daily probe); NEXT = D3 (2026-09-07 ~07:50 UTC, written by session fa49faf3 "S87" after the D2 commit; SUPERSEDES the earlier blocks)
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D3 ✓; C10 IN PROGRESS ON DREW'S SIDE (the Support ticket + the daily probe); NEXT = D4 (2026-09-07 ~08:20 UTC, written by session fa49faf3 "S87" after the D3 commit; SUPERSEDES the earlier blocks)
|
||||
|
||||
### 0. How to use this block
|
||||
You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file,
|
||||
and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest
|
||||
(R1–R73), then WAIT for Drew. **NEXT = two tracks.** **Drew (C10):** open the GitHub Support ticket (text in `docs/public-flip-runbook.md` §11), then run `tools/public_rewrite/probe_github.sh` daily (gh is authenticated in WSL; 31 of 33 sampled old hashes were alive at S87's baseline) until it prints PASS — only then the visibility flip, and only after Blocks D/E/F have landed. **Claude (NEXT = D3, xHigh + one Max pass on the metric→schema map):** `progress.json` exists (D1); add `docs/badges/fleet_instr.json` + `docs/badges/binaries.json` in shields endpoint format (`{"schemaVersion":1,"label","message","color"}`, written by `progress.py --json`, the README already references exactly these names); extend `make audit-digest` with `progress.py --json --readme --check`; `tools/frogress_upload.py` (stdlib, `--dry-run` default, `FROGRESS_API_SECRET` env; frogress projects are admin-created — Drew requests slug `bfm` + key later); `tools/objdiff_report.py` (progress.json → objdiff `report.json`, categories `game-code` vs `linked-sony-objects`, metadata "from a local clean rebuild; CI does not rebuild") + `.github/workflows/progress.yml` uploading artifact `SLUS_007.26_report` (verify the JSON parses with a downloaded `objdiff-cli` first — X2 data). Then D4 → D5 → F1 → F2 → F3 → E3 → E4 → E5 → E6 on the still-private repo; E1/E2 and D3's outward actions after the flip. **Every commit cites NEW hashes only; `.run/public_rewrite/` (scratch: dict, mailmap, old-to-new, bundle 556 MB, the rewritten bare clone) holds the old ones — keep until the probe passes, then delete (C11).**
|
||||
one TaskCreate per plan item A1…G2, 40 items, mark A1–A5 + B1–B9/C3 + C1–C9 + D1–D2 completed; R28). The SessionStart hook restarts the headless
|
||||
(R1–R73), then WAIT for Drew. **NEXT = two tracks.** **Drew (C10):** open the GitHub Support ticket (text in `docs/public-flip-runbook.md` §11), then run `tools/public_rewrite/probe_github.sh` daily (gh is authenticated in WSL; 31 of 33 sampled old hashes were alive at S87's baseline) until it prints PASS — only then the visibility flip, and only after Blocks D/E/F have landed. **Claude (NEXT = D4, xHigh): the SETUP.md public-clean pass** — header date; §2.3/§2.4 → download-only wording (drop the "vendored zip" lines the plan cites at 16-17/861-862 — grep `vendored`); §4.4 a generic dump path; §4.7 the tarballs; §4.8 + §5.6 PsyQ = user-supplied; §6.5 → the decomp.me preset (E1); the "Backup & private-repo posture" section → the PUBLIC posture (R1/R20 historical; the archive repo; `config/ghidra/` as R20's home — the P33 paragraph is already there, the private-era prose above it is not); ledger rows 11/14 closed; an R21 row for every P33 tool (audit against `git log --diff-filter=A --name-only <P33 range> -- tools/`); every `gitignored|private|curated public mirror|two-repo` mention reviewed line by line. Then D5 (governing docs + `tools/doc_links.py`) → F1 → F2 → F3 → E3 → E4 → E5 → E6; E1/E2 and D3's outward actions after the flip. **Every commit cites NEW hashes only; `.run/public_rewrite/` (scratch: dict, mailmap, old-to-new, bundle 556 MB, the rewritten bare clone) holds the old ones — keep until the probe passes, then delete (C11).**
|
||||
one TaskCreate per plan item A1…G2, 40 items, mark A1–A5 + B1–B9/C3 + C1–C9 + D1–D3 completed; R28). The SessionStart hook restarts the headless
|
||||
MCP server when `ghidra/bfm.rep` exists (it did not stay up in S87 — `ss -tln` showed nothing on :8080; harmless): B6/B7/B8
|
||||
need no Ghidra; run `tools/ghidra_mcp_stop.sh` before any headless step (R23).
|
||||
|
||||
@@ -417,15 +429,16 @@ still stand for the tasks ahead — restate them, Drew decides (R7/R27).
|
||||
1. **C10** (Drew; decision Max): the Support ticket (text in `docs/public-flip-runbook.md` §11), then `tools/public_rewrite/
|
||||
probe_github.sh` daily until PASS (gh auth done in S87; `old-to-new.tsv` + `old_tag_tip.txt` in scratch); the flip only
|
||||
after the probe passes AND Blocks D/E/F have landed.
|
||||
2. **D3 — progress publishing** (see the NEXT line above): badges JSON (2 files, exact names), audit-digest extension,
|
||||
frogress_upload.py, objdiff_report.py + progress.yml. Log, tick, refresh, commit. Then **D4** (SETUP.md public-clean
|
||||
pass: header date; §2.3/§2.4 download-only wording — the "vendored zip" lines; §4.4 generic dump path; §4.7; §4.8/§5.6
|
||||
PsyQ user-supplied; §6.5 → the decomp.me preset; the "Backup & private-repo posture" section → the public posture;
|
||||
ledger rows 11/14; R21 rows for every P33 tool) and **D5** (governing-docs consistency + `tools/doc_links.py`).
|
||||
2. **D4 — SETUP.md public-clean pass** (xHigh; see the NEXT line above). Then **D5**: CLAUDE.md:61 wording (true again),
|
||||
`.gitignore` header, `dumps/INDEX.md:8`, `docs/roadmap-to-100.md` §1 → a dated "status at P33" block, `docs/gen2-roadmap.md`
|
||||
Phase-14 → a "SUPERSEDED at P33" banner, `phase-ends/DIGEST.md` §0/§1/§3 (H1 re-tightened, R1/R20 historical, the
|
||||
PROJECT_CONTEXT corrections, R65–R73 full text — already there), `docs/backlog.md` (0 rows — done); NEW `tools/doc_links.py`
|
||||
(relative link/path checker over README, THIRD_PARTY, SETUP, verification, the runbook, the wiki; in tools-health) + a
|
||||
line-by-line grep review for `gitignored|private|curated public mirror|two-repo`.
|
||||
3. Aftercare reminders (C11): `gh auth logout` in WSL after the flip; `git remote remove archive`; other clones reset, never
|
||||
pull; `.run/public_rewrite/` deleted after the probe passes (keep the bundle off-machine if wanted).
|
||||
### 4. Files S87 touched
|
||||
D2: `LICENSE` (new), `src/NOTICE.md` (new), `THIRD_PARTY.md` (new), `tools/README.md` (new). D1: `README.md` (rewritten), `docs/progress.json` (new, generated), `tools/progress.py` (`--json`/`--readme`/`--check`), `Makefile` (report wiring). C9: `tools/public_rewrite/probe_github.sh` (422 = gone), `docs/public-flip-runbook.md` §10/§12; `.git` (the archive remote removed, 12 worktrees removed, repacked to one 80 MB pack). C8: `.run/P33/verify/*` (the recorded run, tracked), `docs/verification.md` §2, `docs/family-hseq.md`, `tools/verify_contract.sh` (the step-00 fix). C4–C7: `docs/commit-map.tsv` (new), the 98 token-resolved files (see the C7 entry), `tools/public_rewrite/resolve_tokens.py` (the skip rule), `.git/config` (repo-local noreply identity). Scratch `.run/public_rewrite/`: `pre-rewrite.bundle`, `repo.git` (the rewritten bare clone, keep until C9), `c4c5.{sh,log}`, `old-to-new.tsv`, `unchanged_commits.txt`, `old_tag_tip.txt`, `c5_commit-map.tsv`, `ids_after.txt`, `refs_*.txt`. C1: `tools/public_rewrite/{common,hash_dict,scrub,gate_scan,run_filter,verify_rewrite,build_commit_map,resolve_tokens,absent_scan}.py`, `probe_github.sh`, `expected_offenders.txt` (all new), `requirements-python.txt`, `docs/SETUP.md`, `docs/public-flip-runbook.md`. Scratch `.run/public_rewrite/`: `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old_tag_tip.txt`, `trial*.{sh,log}`, `trial_commit-map.tsv`, `trial_old-to-new.tsv`, `unchanged_commits.txt`, `nonpurge_blob_ids.txt`, `filter.log`. B9/C3: 251 index removals (files on disk), `docs/public-flip-runbook.md` (new), `docs/SETUP.md` §2.3/§2.4, `docs/verification.md`, `CLAUDE.md`, `docs/decision-log.md`. A5: `tools/verify_contract.sh` (new), `tools/audit_frontier.py` (derived denominator lines), `.gitignore` (the `.run/P33/verify/` allowlist), `.run/P33/verify/*` (tracked evidence), `docs/verification.md` §2, regenerated `docs/family-hseq.md` + `docs/progress*.md`/`duplicates*.md`. B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
|
||||
D3: `tools/progress.py` (per_bin, badges, --check), `docs/badges/*.json` (4, new), `docs/progress.json`, `tools/objdiff_report.py` (new), `tools/frogress_upload.py` (new), `.github/workflows/progress.yml` (new), `Makefile` (audit-digest), `docs/SETUP.md`. Scratch: `.run/objdiff/` (the objdiff-cli 3.8.1 validator + reports). D2: `LICENSE` (new), `src/NOTICE.md` (new), `THIRD_PARTY.md` (new), `tools/README.md` (new). D1: `README.md` (rewritten), `docs/progress.json` (new, generated), `tools/progress.py` (`--json`/`--readme`/`--check`), `Makefile` (report wiring). C9: `tools/public_rewrite/probe_github.sh` (422 = gone), `docs/public-flip-runbook.md` §10/§12; `.git` (the archive remote removed, 12 worktrees removed, repacked to one 80 MB pack). C8: `.run/P33/verify/*` (the recorded run, tracked), `docs/verification.md` §2, `docs/family-hseq.md`, `tools/verify_contract.sh` (the step-00 fix). C4–C7: `docs/commit-map.tsv` (new), the 98 token-resolved files (see the C7 entry), `tools/public_rewrite/resolve_tokens.py` (the skip rule), `.git/config` (repo-local noreply identity). Scratch `.run/public_rewrite/`: `pre-rewrite.bundle`, `repo.git` (the rewritten bare clone, keep until C9), `c4c5.{sh,log}`, `old-to-new.tsv`, `unchanged_commits.txt`, `old_tag_tip.txt`, `c5_commit-map.tsv`, `ids_after.txt`, `refs_*.txt`. C1: `tools/public_rewrite/{common,hash_dict,scrub,gate_scan,run_filter,verify_rewrite,build_commit_map,resolve_tokens,absent_scan}.py`, `probe_github.sh`, `expected_offenders.txt` (all new), `requirements-python.txt`, `docs/SETUP.md`, `docs/public-flip-runbook.md`. Scratch `.run/public_rewrite/`: `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old_tag_tip.txt`, `trial*.{sh,log}`, `trial_commit-map.tsv`, `trial_old-to-new.tsv`, `unchanged_commits.txt`, `nonpurge_blob_ids.txt`, `filter.log`. B9/C3: 251 index removals (files on disk), `docs/public-flip-runbook.md` (new), `docs/SETUP.md` §2.3/§2.4, `docs/verification.md`, `CLAUDE.md`, `docs/decision-log.md`. A5: `tools/verify_contract.sh` (new), `tools/audit_frontier.py` (derived denominator lines), `.gitignore` (the `.run/P33/verify/` allowlist), `.run/P33/verify/*` (tracked evidence), `docs/verification.md` §2, regenerated `docs/family-hseq.md` + `docs/progress*.md`/`duplicates*.md`. B8: `docs/verification.md` (new), `docs/SETUP.md` (§4.4/§4.6/§4.8/§6.3 pointers, backup posture). B7: `.github/workflows/no-rom.yml`, `tools/audit_public.py`, `tools/compile_only.py`, `tools/public_rewrite/purge_set.txt` (all new), `docs/SETUP.md`. B6: `dumps/CHECKSUMS.sha1` (new), `dumps/INDEX.md`, `docs/memory-map.md`. B5: `tools/ghidra_scripts/ImportAnnotations.java` (3 compile fixes + the `/undefined` resolver), `tools/ghidra_rebuild.sh`
|
||||
(`.proof` markers; dies unless `failed=0`), `tools/ghidra_annotations_delta.py` (the three drift classes), new
|
||||
`tools/ghidra_roster.py`, `tools/ghidra_mcp_start.sh` (silent no-op guard), `.claude/settings.json` (relative hooks),
|
||||
`Makefile` (roster check in tools-health), `docs/SETUP.md` (P33 B5 section, 5 inventory rows, §2.8), `config/ghidra/*`
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
"""frogress_upload.py — push docs/progress.json to a frogress instance (progress.deco.mp) (P33 D3).
|
||||
|
||||
tools/frogress_upload.py # --dry-run is the default: prints the payload, sends nothing
|
||||
tools/frogress_upload.py --push --project bfm --version us # needs FROGRESS_API_SECRET in the environment
|
||||
|
||||
frogress projects are created by the frogress maintainers (a slug + an API key are issued to the project owner);
|
||||
the upload is `POST https://progress.deco.mp/data/<project>/<version>/` with a JSON body
|
||||
`{"api_key": …, "entries": [{"git_hash": …, "timestamp": …, "categories": {"default": {measure: value, …}}}]}`
|
||||
(schema as documented by decomp.me's frogress; treat the endpoint's answer as data, X2). Measures sent, every one with
|
||||
its denominator (R41): functions / instructions / distinct code / main game code matched and total, binaries
|
||||
byte-identical and total. Stdlib only. The secret is never written anywhere; a missing secret with --push is an error.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def payload(d, git_hash, ts, key):
|
||||
f, c = d["fleet"], d["counts"]
|
||||
ok_bins = sum(1 for r in d["per_binary"] if r["matchable"] and r["byte_identical"] == r["matchable"])
|
||||
m = {
|
||||
"fns_matched": f["fn_count"]["matched"], "fns_total": f["fn_count"]["total"],
|
||||
"instr_matched": f["instr_weighted"]["matched"] if f["instr_weighted"] else 0,
|
||||
"instr_total": f["instr_weighted"]["total"] if f["instr_weighted"] else 0,
|
||||
"distinct_matched": f["distinct_code"]["matched"] if f["distinct_code"] else 0,
|
||||
"distinct_total": f["distinct_code"]["total"] if f["distinct_code"] else 0,
|
||||
"main_matched": f["main_game_code"]["matched"] if f["main_game_code"] else 0,
|
||||
"main_total": f["main_game_code"]["total"] if f["main_game_code"] else 0,
|
||||
"binaries_ok": ok_bins, "binaries_total": d["binaries"],
|
||||
"linked_sony_functions": c["linked_psyq_objects"], "verbatim_asm_bodies": c["verbatim_asm_bodies"],
|
||||
}
|
||||
return {"api_key": key, "entries": [{"git_hash": git_hash, "timestamp": ts, "categories": {"default": m}}]}
|
||||
|
||||
|
||||
def main(argv):
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--push", action="store_true", help="actually POST (default: dry run)")
|
||||
ap.add_argument("--project", default="bfm")
|
||||
ap.add_argument("--version", default="us")
|
||||
ap.add_argument("--url", default="https://progress.deco.mp")
|
||||
ap.add_argument("--in", dest="inp", default=str(REPO / "docs/progress.json"))
|
||||
a = ap.parse_args(argv)
|
||||
d = json.loads(pathlib.Path(a.inp).read_text(encoding="utf-8"))
|
||||
git_hash = subprocess.run(["git", "-C", str(REPO), "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
|
||||
ts = int(subprocess.run(["git", "-C", str(REPO), "log", "-1", "--format=%ct"], capture_output=True, text=True).stdout.strip() or 0)
|
||||
key = os.environ.get("FROGRESS_API_SECRET", "")
|
||||
body = payload(d, git_hash, ts, key if a.push else "<FROGRESS_API_SECRET>")
|
||||
endpoint = f"{a.url.rstrip('/')}/data/{a.project}/{a.version}/"
|
||||
if not a.push:
|
||||
print(f"frogress_upload: DRY RUN — would POST to {endpoint}:")
|
||||
print(json.dumps(body, indent=1))
|
||||
return 0
|
||||
if not key:
|
||||
sys.exit("frogress_upload: --push needs FROGRESS_API_SECRET in the environment (never in a file)")
|
||||
req = urllib.request.Request(endpoint, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}, method="POST")
|
||||
with urllib.request.urlopen(req, timeout=60) as resp:
|
||||
text = resp.read().decode("utf-8", "replace")
|
||||
print(f"frogress_upload: HTTP {resp.status} {text[:300]}")
|
||||
return 0 if 200 <= resp.status < 300 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env python3
|
||||
"""objdiff_report.py — docs/progress.json → an objdiff progress report (report.json) for decomp.dev (P33 D3).
|
||||
|
||||
tools/objdiff_report.py [--out report.json] [--in docs/progress.json]
|
||||
|
||||
decomp.dev reads a GitHub Actions artifact in objdiff's report format (objdiff-core/protos/report.proto, `Report`
|
||||
version 2): top-level `measures`, one `units` entry per binary with its own measures and metadata, and `categories`.
|
||||
Field names are the proto's (snake_case); the output was validated with `objdiff-cli report changes` (S87).
|
||||
|
||||
What the numbers ARE here (every one carries its denominator in docs/progress.json, R41):
|
||||
* code = instructions × 4 bytes (MIPS): `total_code` / `matched_code` per binary from the build-derived function
|
||||
boundaries; main's code is its GAME CODE only (the linked Sony objects are a separate functions-only category);
|
||||
* functions = byte-identical / matchable per binary (REAL C + LINKED Sony objects + empties);
|
||||
* `fuzzy_match_percent` == the exact matched percentage (this project has no fuzzy state: a function is byte-identical
|
||||
or it is an assembly stub);
|
||||
* categories: `game-code` (every binary's game code) and `linked-sony-objects` (the 1,256 PsyQ functions of the main
|
||||
executable — byte-identical by link, not our C; counted in functions only, their byte size is not measured here).
|
||||
The report is derived from a LOCAL clean rebuild recorded in docs/progress.json (`make report`); CI does not rebuild the
|
||||
game (it has no disc) — the workflow only converts the committed JSON and uploads the artifact.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
REPO = pathlib.Path(__file__).resolve().parent.parent
|
||||
VERSION = 2
|
||||
|
||||
|
||||
def measures(code_m, code_t, fn_m, fn_t, units_t=0, units_c=0):
|
||||
pct = lambda m, t: (100.0 * m / t) if t else 0.0
|
||||
return {
|
||||
"fuzzy_match_percent": round(pct(code_m, code_t), 4),
|
||||
"total_code": int(code_t), "matched_code": int(code_m), "matched_code_percent": round(pct(code_m, code_t), 4),
|
||||
"total_data": 0, "matched_data": 0, "matched_data_percent": 0.0,
|
||||
"total_functions": int(fn_t), "matched_functions": int(fn_m), "matched_functions_percent": round(pct(fn_m, fn_t), 4),
|
||||
"complete_code": int(code_m), "complete_code_percent": round(pct(code_m, code_t), 4),
|
||||
"complete_data": 0, "complete_data_percent": 0.0,
|
||||
"total_units": int(units_t), "complete_units": int(units_c),
|
||||
}
|
||||
|
||||
|
||||
def build(d):
|
||||
units, cm_all, ct_all, fm_all, ft_all, complete = [], 0, 0, 0, 0, 0
|
||||
game_fm = game_ft = 0
|
||||
for r in d["per_binary"]:
|
||||
im, it = r.get("instr_matched"), r.get("instr_total")
|
||||
if im is None or it is None:
|
||||
sys.exit(f"objdiff_report: {r['binary']} has no instruction totals in progress.json — regenerate it with the sigs present (make report BINARY=main)")
|
||||
code_m, code_t = im * 4, it * 4
|
||||
fn_m, fn_t = r["byte_identical"], r["matchable"]
|
||||
game_fn_t = fn_t - r["linked"]
|
||||
game_fn_m = fn_m - r["linked"] # every linked object is byte-identical by construction
|
||||
done = bool(fn_t) and fn_m == fn_t
|
||||
units.append({
|
||||
"name": r["binary"],
|
||||
"measures": measures(code_m, code_t, fn_m, fn_t, 1, 1 if done else 0),
|
||||
"sections": [{"name": ".text", "size": code_t, "fuzzy_match_percent": round(100.0 * code_m / code_t, 4) if code_t else 0.0}],
|
||||
"functions": [],
|
||||
"metadata": {"complete": done, "module_name": r["binary"],
|
||||
"source_path": "src" if r["binary"] == "main" else f"src/{r['binary']}",
|
||||
"progress_categories": ["game-code"] + (["linked-sony-objects"] if r["linked"] else []),
|
||||
"auto_generated": False},
|
||||
})
|
||||
cm_all += code_m; ct_all += code_t; fm_all += fn_m; ft_all += fn_t; complete += int(done)
|
||||
game_fm += game_fn_m; game_ft += game_fn_t
|
||||
linked = d["counts"]["linked_psyq_objects"]
|
||||
return {
|
||||
"version": VERSION,
|
||||
"measures": measures(cm_all, ct_all, fm_all, ft_all, len(units), complete),
|
||||
"units": units,
|
||||
"categories": [
|
||||
{"id": "game-code", "name": "Game code (our C)", "measures": measures(cm_all, ct_all, game_fm, game_ft, len(units), complete)},
|
||||
{"id": "linked-sony-objects", "name": "Sony PsyQ library objects (linked, not our C)",
|
||||
"measures": measures(0, 0, linked, linked, 1, 1)},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main(argv):
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--in", dest="inp", default=str(REPO / "docs/progress.json"))
|
||||
ap.add_argument("--out", default="report.json")
|
||||
a = ap.parse_args(argv)
|
||||
d = json.loads(pathlib.Path(a.inp).read_text(encoding="utf-8"))
|
||||
if d.get("schema") != 1:
|
||||
sys.exit(f"objdiff_report: unexpected progress.json schema {d.get('schema')!r} (R43)")
|
||||
rep = build(d)
|
||||
pathlib.Path(a.out).write_text(json.dumps(rep, indent=1) + "\n", encoding="utf-8")
|
||||
m = rep["measures"]
|
||||
print(f"objdiff_report: {len(rep['units'])} units ({m['complete_units']} complete), code {m['matched_code']:,}/{m['total_code']:,} bytes "
|
||||
f"= {m['matched_code_percent']:.2f}%, functions {m['matched_functions']:,}/{m['total_functions']:,} -> {a.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+45
-5
@@ -925,16 +925,20 @@ def weighted_metrics():
|
||||
|
||||
fm = ft = 0
|
||||
cls_nins, matched_cls = {}, set()
|
||||
per_bin = {} # P33 D3: binary -> [matched ins, total ins] (the objdiff units)
|
||||
for p in paths:
|
||||
st = stub_addrs(_sig_binary(p))
|
||||
b = _sig_binary(p)
|
||||
st = stub_addrs(b)
|
||||
bm = bt = 0
|
||||
for line in open(p):
|
||||
r = json.loads(line)
|
||||
a, n, hx = int(r["addr"], 16), r["nins"], r["h_exact"]
|
||||
ft += n
|
||||
ft += n; bt += n
|
||||
cls_nins[hx] = n # h_exact-identical -> identical nins
|
||||
if a not in st: # non-stub == matched (fleet is byte-identical)
|
||||
fm += n
|
||||
fm += n; bm += n
|
||||
matched_cls.add(hx)
|
||||
per_bin[b] = [bm, bt]
|
||||
ut = sum(cls_nins.values())
|
||||
um = sum(n for hx, n in cls_nins.items() if hx in matched_cls)
|
||||
|
||||
@@ -994,7 +998,8 @@ def weighted_metrics():
|
||||
# main's open stubs in `frontier_classify` (a known-true-case check).
|
||||
fleet_m_all, fleet_t_all = fm + mm, ft + mt
|
||||
dedup_m_all, dedup_t_all = um + mm, ut + mt # main's fns are unique — no h_exact sharing
|
||||
return dict(fleet_m=fleet_m_all, fleet_t=fleet_t_all,
|
||||
per_bin["main"] = [mm, mt]
|
||||
return dict(fleet_m=fleet_m_all, fleet_t=fleet_t_all, per_bin=per_bin,
|
||||
fleet_pct=(100 * fleet_m_all / fleet_t_all if fleet_t_all else 0.0),
|
||||
dedup_m=dedup_m_all, dedup_t=dedup_t_all,
|
||||
dedup_pct=(100 * dedup_m_all / dedup_t_all if dedup_t_all else 0.0),
|
||||
@@ -1101,10 +1106,40 @@ def fleet():
|
||||
"verbatim_asm_bodies": nverb, "include_asm_stubs": STUBS, "non_matching": NM, "matchable": MATCH,
|
||||
"dedup_groups": ngroups, "dedup_instances": nmembers},
|
||||
"per_binary": [{"binary": r['binary'], "real": r['real'], "shared": r['shared'], "linked": r['linked'],
|
||||
"byte_identical": r['byteident'], "matchable": r['matchable']} for r in rows],
|
||||
"byte_identical": r['byteident'], "matchable": r['matchable'],
|
||||
"instr_matched": (wm['per_bin'].get(r['binary'], [None, None])[0] if wm else None),
|
||||
"instr_total": (wm['per_bin'].get(r['binary'], [None, None])[1] if wm else None)} for r in rows],
|
||||
}
|
||||
|
||||
|
||||
def badge(label, message, pct):
|
||||
color = "brightgreen" if pct >= 99.95 else "green" if pct >= 75 else "yellow" if pct >= 50 else "orange"
|
||||
return {"schemaVersion": 1, "label": label, "message": message, "color": color}
|
||||
|
||||
|
||||
def badge_files(d):
|
||||
"""{name: json text} for docs/badges/*.json in the shields.io endpoint format (README:
|
||||
img.shields.io/endpoint?url=…). Names are fixed: the README references exactly these files."""
|
||||
import json as _json
|
||||
f, c = d["fleet"], d["counts"]
|
||||
ok_bins = sum(1 for r in d["per_binary"] if r["matchable"] and r["byte_identical"] == r["matchable"])
|
||||
files = {
|
||||
"fleet_instr.json": badge("matched (instructions)", f"{f['instr_weighted']['pct']:.1f}%", f['instr_weighted']['pct']) if f['instr_weighted'] else None,
|
||||
"fleet_fn.json": badge("matched (functions)", f"{f['fn_count']['pct']:.2f}%", f['fn_count']['pct']),
|
||||
"distinct.json": badge("distinct code matched", f"{f['distinct_code']['pct']:.1f}%", f['distinct_code']['pct']) if f['distinct_code'] else None,
|
||||
"binaries.json": badge("byte-identical binaries", f"{ok_bins}/{d['binaries']}", 100.0 * ok_bins / d['binaries']),
|
||||
}
|
||||
return {n: _json.dumps(v) + "\n" for n, v in files.items() if v is not None}
|
||||
|
||||
|
||||
def write_badges(d):
|
||||
bd = ROOT / "docs/badges"; bd.mkdir(exist_ok=True)
|
||||
files = badge_files(d)
|
||||
for name, text in files.items():
|
||||
(bd / name).write_text(text)
|
||||
return sorted(files)
|
||||
|
||||
|
||||
README_BEGIN, README_END = "<!-- progress:begin -->", "<!-- progress:end -->"
|
||||
|
||||
|
||||
@@ -1140,8 +1175,13 @@ def write_json_and_readme(check=False):
|
||||
if check:
|
||||
if not jp.exists() or jp.read_text() != text:
|
||||
sys.exit("progress.py --json --check: docs/progress.json is STALE — run `make report BINARY=main`")
|
||||
for name, want in badge_files(d).items():
|
||||
bp = ROOT / "docs/badges" / name
|
||||
if not bp.exists() or bp.read_text() != want:
|
||||
sys.exit(f"progress.py --check: docs/badges/{name} is STALE or missing — run `make report BINARY=main`")
|
||||
else:
|
||||
jp.write_text(text)
|
||||
write_badges(d)
|
||||
rp = ROOT / "README.md"
|
||||
if not rp.exists():
|
||||
sys.exit("progress.py --readme: README.md missing")
|
||||
|
||||
Reference in New Issue
Block a user