mirror/BFM-decomp
1
0
Fork 0
mirror of https://github.com/Druthulu/BFM-decomp synced 2026-10-01 15:43:06 -04:00
Code Issues Packages Projects Releases Wiki Activity

T1.c1: tools-health off docs/retired: census reads ops inventory, links fixed, cookbook_index off

Browse Source
This commit is contained in:
Drew T
2026-09-29 19:42:44 -06:00
parent 7e6e5dda0d
commit db68150f5e
19 changed files with 72 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+6 -1
View File
@@ -314,7 +314,12 @@ tools-health:
$(MAKE) --no-print-directory audit-digest
# The cookbook index is DERIVED (R33) and self-asserts its coverage (R32). Stale = agents can't
# find documented idioms and re-derive them at full token cost (measured, P30 wave 1).
$(VENV_PY) tools/cookbook_index.py --check
# DISABLED: cookbook_index.py --check — 2026-09-29 / Phase 38 T1
# Original intent: assert docs/cookbook-index.md is fresh against docs/matching-cookbook.md.
# Why disabled: both files moved to docs/retired/ at the PA3 migration (b8a6cbc54); the check crashed on the missing
# source. Successor: cookbook/INDEX.md, kept by tools/cookbook_add.sh.
# Re-enable if: docs/matching-cookbook.md becomes a live source again.
# $(VENV_PY) tools/cookbook_index.py --check
$(VENV_PY) tools/gccmap_cites.py --check
# P33 B5: the Ghidra roster is DERIVED from config/ghidra/*.jsonl (R33); a stale roster misreports
# which programs' RE work is tracked as text. Pure text check, no Ghidra needed.
README.md
+1 -1
View File
@@ -105,7 +105,7 @@ The reverse-engineering side ran on Ghidra driven over MCP, with a PCSX-Redux ru
byte gate over the whole binary, a permuter, a compiler-idiom cookbook, and — decisively — reading the gcc 2.7.2 source
to explain the bytes the compiler emits. Everything that was learned is written down where it can be reused:
- [`docs/matching-cookbook.md`](docs/matching-cookbook.md) — the idioms, each byte-proven, and the compiler-internals laws
- [`cookbook/INDEX.md`](cookbook/INDEX.md) — the idioms, each byte-proven, and the compiler-internals laws
- [`docs/gcc-2.7.2-map/`](docs/gcc-2.7.2-map/) — the codegen map of the pinned compiler, by pass
- [`docs/decision-log.md`](docs/decision-log.md) — every strategic pivot with its measurement and its hindsight
- [`docs/accelerators.md`](docs/accelerators.md) — what would have made earlier phases faster
docs/gcc-2.7.2-map/README.md
+1 -1
View File
@@ -6,7 +6,7 @@ that moves it — each lever **byte-proven** on a named function with the diff b
Phase 23 by four frontier-class agents reading the gcc 2.7.2 passes directly, after six phases in which the same
residuals had been called "unsteerable"; it is the artifact that lets a mid-tier agent, or a local model, apply
compiler-internal levers without reading 80,000 lines of source. The index and triage table live in the cookbook
([`docs/matching-cookbook.md`](../matching-cookbook.md) §31); the method is chapter 07 of the how-to
([`cookbook/INDEX.md`](../../cookbook/INDEX.md) §31); the method is chapter 07 of the how-to
([`docs/how-to-ai-decomp/07-compiler-source.md`](../how-to-ai-decomp/07-compiler-source.md)).
## The files
docs/how-to-ai-decomp/00-README.md
+1 -1
View File
@@ -67,7 +67,7 @@ the machine that builds it is universal.** That is the whole point of these chap
- [`docs/accelerators.md`](../accelerators.md) — what was learned late that day one should know, each entry dated with when it *could* have been found
- [`docs/retrospective.md`](../retrospective.md) and [`docs/story.md`](../story.md) — what we believed, what failed, what it cost; and the narrative
- [`docs/wave-playbook.md`](../wave-playbook.md) — the running procedure, each guard paired with the measurement that earned it
- [`docs/decision-log.md`](../decision-log.md) — every pivot with its measurement and hindsight; [`docs/effort-map.md`](../effort-map.md) — the reasoning-effort doctrine
- [`docs/decision-log.md`](../decision-log.md) — every pivot with its measurement and hindsight; [`docs/retired/effort-map.md`](../retired/effort-map.md) — the reasoning-effort doctrine
- [`phase-ends/DIGEST.md`](../../phase-ends/DIGEST.md) — every phase's synopsis and the full text of every rule cited here as `R<n>`
The framework the governance chapter describes is published separately as
docs/how-to-ai-decomp/01-governance.md
+1 -1
View File
@@ -86,7 +86,7 @@ The last thirty sessions of the project resumed from the digest and the checkpoi
## Effort, and who toggles it
The orchestrating agent's reasoning depth is a setting the human controls, and the project mapped it per task
([`docs/effort-map.md`](../effort-map.md)): the deepest setting for phase plans, PhaseEnds, architectural forks and
([`docs/retired/effort-map.md`](../retired/effort-map.md)): the deepest setting for phase plans, PhaseEnds, architectural forks and
non-obvious debugging; the highest *persistent* setting for execution whose design is already settled; multi-agent
breadth (many independent items) for surveys, audits and bulk matching. The rule that made this workable is R27:
**every transition is prompted and waited for** — the agent never launches a fan-out on a verbal yes, and never rolls
docs/how-to-ai-decomp/06-knowledge-base.md
+2 -2
View File
@@ -8,8 +8,8 @@ behaviour; yours will be about your compiler. The machine that builds it is what
| Layer | BFM's instance | What it holds |
|---|---|---|
| **The cookbook** | [`docs/matching-cookbook.md`](../matching-cookbook.md) (3.5 MB; grep it by section, never read it whole) | one section per idiom or law, each **byte-proven** on a named function with the diff before and after; the compiler-internals laws; the integration classes; the wall proofs and their later refutations |
| **The symptom index** | [`docs/cookbook-index.md`](../cookbook-index.md), derived by a self-asserting tool | *symptom → section*: what a residual looks like in the diff, and where the lever is |
| **The cookbook** | [`cookbook/INDEX.md`](../../cookbook/INDEX.md) (3.5 MB; grep it by section, never read it whole) | one section per idiom or law, each **byte-proven** on a named function with the diff before and after; the compiler-internals laws; the integration classes; the wall proofs and their later refutations |
| **The symptom index** | [`cookbook/INDEX.md`](../../cookbook/INDEX.md), derived by a self-asserting tool | *symptom → section*: what a residual looks like in the diff, and where the lever is |
| **The codegen map** | [`docs/gcc-2.7.2-map/`](../gcc-2.7.2-map/) by compiler pass | pass → residual pattern → C-level lever, from reading the compiler's source ([chapter 07](07-compiler-source.md)) |
| **The pinned triple and the permuter recipes** | the build, `tools/permuter/` | the executable form of the same knowledge: what the tools try before an agent is asked |
| **The journal** | per-function past-attempt notes | what every earlier agent tried on *this* function — pack fuel |
docs/how-to-ai-decomp/08-models-and-budgets.md
+1 -1
View File
@@ -46,7 +46,7 @@ differently.
## The orchestrator's effort — depth vs. breadth
The orchestrating agent's reasoning depth is a human-controlled setting, and it maps onto two axes
([`docs/effort-map.md`](../effort-map.md)):
([`docs/retired/effort-map.md`](../retired/effort-map.md)):
| | Depth — how hard one agent thinks | Breadth — how many angles at once |
|---|---|---|
docs/ops/tooling-inventory.md
+1 -1
View File
@@ -18,7 +18,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
| | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). |
| | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/<prog>_funcs.txt`) — completes a raw-blob program's function set (Phase 10). |
| | `ApplySymbols.java` + `tools/ghidra_apply_symbols.sh` | **(P31 S78) The Ghidra MIRROR of the curated symbol file (R15/G6), headless with a real save.** `tools/ghidra_apply_symbols.sh [PROG] [symbols files…]` (defaults `SLUS_007.26 config/symbols.us.txt`; MCP must be STOPPED first) reads `name = 0xADDR;` rows and sets every function/label to its curated name; a name held by another address is moved to that address's own curated name first (`firstfile`/`firstfile2`), else to `<name>__at_<addr>`. Idempotent; prints `BFMAPPLY renamed_funcs=… unchanged=…`; R9-verify with `ghidra_mcp_verify.sh`. **Use this, not MCP `rename_symbol`/`batch_rename`, for renames:** S78 observed 47 MCP renames NOT persisting through the sentinel stop ("Save succeeded", DB grew, names gone — R9 caught it; cause not yet isolated), while the postScript path persisted 73/73 on the first run. |
| **Public flip / CI** | `.github/workflows/no-rom.yml` | **(P33 B7) The ROM-free CI**: job `audits` (audit_public, audit_text_sources, verbatim_check --strict, cookbook_index --check, ghidra_roster --check, work_evidence --selftest, test_lzss, lint_symbol_refs — ≈45 s of checks) + job `compile-only` (binutils-mipsel + `cpp-mipsel-linux-gnu` from apt, cc1 from the tracked tarball sha256-checked, maspsx submodule; PR scope `main resident ov_SC01_077 md_MAIN_013`; `--all` weekly Mon 06:17 UTC + `workflow_dispatch`). Byte-identity is NOT proven in CI (needs the disc) — `docs/verification.md`. |
| **Public flip / CI** | `.github/workflows/no-rom.yml` | **(P33 B7) The ROM-free CI**: job `audits` (audit_public, audit_text_sources, verbatim_check --strict, cookbook_index --check (retired from `make tools-health` at P38 T1: its sources moved to `docs/retired/`; successor `cookbook/INDEX.md` via `tools/cookbook_add.sh`), ghidra_roster --check, work_evidence --selftest, test_lzss, lint_symbol_refs — ≈45 s of checks) + job `compile-only` (binutils-mipsel + `cpp-mipsel-linux-gnu` from apt, cc1 from the tracked tarball sha256-checked, maspsx submodule; PR scope `main resident ov_SC01_077 md_MAIN_013`; `--all` weekly Mon 06:17 UTC + `workflow_dispatch`). Byte-identity is NOT proven in CI (needs the disc) — `docs/verification.md`. |
| | `tools/timeline.py [--check]` | **(P33 F1)** The progress timeline from the repository's own committed digests: every commit touching `docs/progress.fleet.md` / `docs/progress.md` (both historical formats parsed), one row per DATE (never keyed by hash — the rewrite changed them), PhaseEnd ticks from the headers, commits per day → `docs/story-timeline.md` + `docs/story-timeline.svg` (three polylines, phase ticks, the 07-22 denominator step annotated from the rows); self-check: the last row == `docs/progress.json`; `--check` = stale detection. 72 rows, 1.5 s. **Wired P33.5 task 7:** regenerated by `make report BINARY=main` (after `progress.py --json`), asserted fresh by `make audit-digest` (it had been wired to nothing and sat stale at the Phase-33 close). |
| | `tools/mine_hindsight.py [--out …]` | **(P33 F2)** Gathers every recorded hindsight with `file:line` anchors — the decision-log's `Hindsight` bullets and `### Hindsight` sections (19 over 79 entries), the PhaseEnds' "What we believed…" sections (2), every Deviations table (237 rows / 32 PhaseEnds) → `.run/P33/hindsight.md` (scratch; `docs/retrospective.md` cites the sources, never the working set). Prints the census. |
| **Docs** | `tools/doc_links.py [--strict] [--disk] [FILE…]` | **(P33 D5; extended P33.5 task 7)** Six checks, each printed with its denominator: (1) every relative Markdown link in the public-facing docs (the DEFAULT set + every wiki page and how-to chapter) resolves; targets listed in `docs/doc_links_pending.txt` (`path<TAB>creating task`) count as PENDING, not broken — `--strict` (gate 2) refuses any pending entry; (2) nothing links into `docs/sunset/`; (3) a wiki page/chapter links into `docs/` only at a target the Reference index or the README links (the allow-list is DERIVED from those two pages; the two index pages are exempt); (4) every tracked `docs/` file outside wiki/how-to/sunset is covered the same way; (5) backticked `docs/…` / `.run/…` citations are TRACKED or UNTRACKED by `git ls-files` (never the disk) — a wiki page citing an UNTRACKED path fails, elsewhere it is counted; `--disk` adds the PRIVATE/DANGLING split for the maintainer; (6) wiki-first WARNINGS (exit 0) for a non-wiki document linking a `docs/` file whose topic has a wiki page. In `tools-health`. Controls: a broken link → rc 1; the P33.5 cookbook control (`--disk docs/matching-cookbook.md` listed the stale `.run/` citations before they were fixed). |
docs/wiki/Archive-index.md
+5 -5
View File
@@ -21,7 +21,7 @@ Records that stay — the PhaseEnds, the decision log, the cookbook — keep the
| `frontier-p30.md`, `frontier-s39.md` | Point-in-time frontier reports (2026-07-30, 2026-08-05), never re-baselined by design | The frontier they described emptied | `PhaseEnd_Phase30/31.md`; `docs/progress.fleet.md` (218 of 218) |
| `frontier-p32.md` | The last 21 functions and the five parked payloads, by blocker (2026-09-05) — the Phase-32 jumping-off document | All 21 banked and all 5 payloads onboarded; PhaseEnd_Phase31 asked that it be kept forever, so it is archived, never deleted | `PhaseEnd_Phase32.md`; the decision log P32 entries |
| `recovery-queue-s65.md` | 69 unbanked wave targets classified into four lanes, "build it, run it next session" | Absorbed by the P31–P32 finish; 0 open near-misses at the P32 close | `docs/backlog.md` (0 open); `PhaseEnd_Phase31.md` |
| `next-session-triage-ladder.md` | The triage-ladder specification and its three corrections | Built and shipped (`tools/triage_ladder.py`); the corrections are cookbook §376/§378 | `docs/wave-playbook.md` §4b; `docs/SETUP.md`'s tool row |
| `next-session-triage-ladder.md` | The triage-ladder specification and its three corrections | Built and shipped (`tools/triage_ladder.py`); the corrections are cookbook §376/§378 | `docs/wave-playbook.md` §4b; `docs/ops/INDEX.md`'s tool row |
## Studies and syntheses superseded by the wiki
@@ -45,8 +45,8 @@ Records that stay — the PhaseEnds, the decision log, the cookbook — keep the
| File (`docs/sunset/…`) | What it was | What came of it | Where it lives now |
|---|---|---|---|
| `distill/atbhbkbl.md` | 165 candidate idiom notes from waves at/bh/bk/bl, each disposed of with its A/B evidence | Landed as cookbook §265 (the verbatim-asm bank lane), §266 (the inert-rider law) and §267 (the addenda) — §267 cites this ledger as its evidence | `docs/matching-cookbook.md` §265–§267 |
| `distill/axbm.md` | 82 candidate notes from waves ax/bm | Landed as cookbook §268 (the register-pin law on call-clobbered registers) and §269 | `docs/matching-cookbook.md` §268–§269 |
| `distill/atbhbkbl.md` | 165 candidate idiom notes from waves at/bh/bk/bl, each disposed of with its A/B evidence | Landed as cookbook §265 (the verbatim-asm bank lane), §266 (the inert-rider law) and §267 (the addenda) — §267 cites this ledger as its evidence | `cookbook/INDEX.md` §265–§267 |
| `distill/axbm.md` | 82 candidate notes from waves ax/bm | Landed as cookbook §268 (the register-pin law on call-clobbered registers) and §269 | `cookbook/INDEX.md` §268–§269 |
## Session tool designs and analyses (Phase 31, sessions 58–61)
@@ -81,8 +81,8 @@ documents (0 open stubs), so the last populated snapshot is kept instead of bein
| File (`docs/sunset/history/…`) | What it was | Where it lives now |
|---|---|---|
| `Fable5-ultracode-Full research findings-w42n9cp9t.output` | 216 KB of raw multi-agent research JSON from the first day (the sotn workflow, the Ghidra/PSX stack, WSL) | Everything actionable went into `docs/SETUP.md` and `docs/formats.md` in June 2026 |
| `debug menu link.txt`, `ghidra psx plugin link.txt`, `sotn-decomp wiki link.txt` | The original three reference URLs | `docs/SETUP.md` §2 and the memory map's source index |
| `Fable5-ultracode-Full research findings-w42n9cp9t.output` | 216 KB of raw multi-agent research JSON from the first day (the sotn workflow, the Ghidra/PSX stack, WSL) | Everything actionable went into `docs/ops/INDEX.md` and `docs/formats.md` in June 2026 |
| `debug menu link.txt`, `ghidra psx plugin link.txt`, `sotn-decomp wiki link.txt` | The original three reference URLs | `docs/ops/INDEX.md` §2 and the memory map's source index |
## Tools retired at Phase 33.5 (`tools/sunset/`, v1.32.1)
docs/wiki/Contributing-and-the-no-ROM-policy.md
+1 -1
View File
@@ -33,7 +33,7 @@ The matching frontier is empty, so new matches are not the need. The work that h
are the type-lift, call-site cast and canonical-signature tools under `tools/`; the constraint is below.
- **Shiftability** — a build whose addresses can move; the LZSS repack (recompression is not byte-stable, so this is a
Gen3 problem) and asset tooling.
- **Tooling and documentation** — every tool has a row in [`docs/SETUP.md`](../SETUP.md)'s inventory; a tool change
- **Tooling and documentation** — every tool has a row in [`docs/ops/INDEX.md`](../ops/INDEX.md)'s inventory; a tool change
ships with its siblings wired to use it and its docs in the same change.
## AI use — conduct
docs/wiki/Docs-and-scratch-conventions.md
+3 -3
View File
@@ -12,12 +12,12 @@ durable result it produced goes into one of these files, and the note is archive
| Kind of knowledge | Where it goes | The rule behind it |
|---|---|---|
| A compiler idiom proven on the bytes (the residual, the mechanism, the lever, the byte proof) | a numbered section of `docs/matching-cookbook.md`, and its symptom index `docs/cookbook-index.md` (regenerated, never edited) | R16 — consult the knowledge base before a match, feed the lesson back after |
| A compiler idiom proven on the bytes (the residual, the mechanism, the lever, the byte proof) | an entry in `cookbook/INDEX.md`, added with `tools/cookbook_add.sh` (never hand-edited) | R16 — consult the knowledge base before a match, feed the lesson back after |
| A strategic pivot: what was believed, what failed, the measurement, the hindsight | an entry in `docs/decision-log.md`, written while fresh | R31 |
| The post-100 % story (Gen3): the narrative of the readability phases | `docs/story.md` §10 and `docs/retrospective.md` §7, advanced at the END OF EVERY SESSION from that session's decision-log entry; `tools/timeline.py` draws the lever and readability series as the chart's lower panel | the owner's rule, 2026-09-12 |
| A late discovery that would have sped up an earlier phase — what it is, when it was found, when it *could* have been found, what it would have saved | `docs/accelerators.md` | the ledger a future project starts from |
| A procedure people run | a runbook — `docs/wave-playbook.md` for the matching campaign, `docs/public-flip-runbook.md` for the publication; a superseded runbook carries a banner at the top and is then archived | R21 for the ops reference |
| An environment or tool fact (a version, a flag, a hook, a row per tool) | `docs/SETUP.md`, in the same change as the tool | R21 |
| An environment or tool fact (a version, a flag, a hook, a row per tool) | `docs/ops/INDEX.md`, in the same change as the tool | R21 |
| An address, with its source and its verification status | `docs/memory-map.md` | G5 |
| A file or container format | `docs/formats.md` | — |
| The state of the open phase; then the phase's synthesis; then the one-page digest every session starts from | `phase-ends/CURRENT_PHASE.md` → `phase-ends/PhaseEnd_Phase<N>.md` → `phase-ends/DIGEST.md` | R64 |
@@ -40,7 +40,7 @@ on its first line; edits go to the generator, not the file.
|---|---|---|
| `docs/progress.md`, `docs/progress.fleet.md`, `docs/progress.json`, the README's progress block, `docs/badges/*.json` | `tools/progress.py` | `make report BINARY=main`; `make audit-digest` asserts freshness |
| `docs/difficulty.md`, `docs/duplicates.md`, `docs/duplicates.cross.md`, `docs/backlog.md` | `tools/difficulty.py`, `tools/dup_report.py`, `tools/backlog.py` | `make report BINARY=main` |
| `docs/cookbook-index.md` | `tools/cookbook_index.py` | `make tools-health` (`--check`) |
| `cookbook/INDEX.md` | `tools/cookbook_add.sh` | none (`cookbook_index.py --check` retired from `make tools-health`, P38 T1) |
| `docs/disc-ledger.md` | `tools/disc_audit.py` | `make audit-disc` |
| `docs/story-timeline.md`, `docs/story-timeline.svg` | `tools/timeline.py` | `make report BINARY=main`; `--check` in `make audit-digest` |
| `docs/commit-map.tsv` | `tools/public_rewrite/build_commit_map.py` | one-shot, at the history rewrite |
docs/wiki/Reference-index.md
+6 -4
View File
@@ -11,10 +11,13 @@ in the Archive index, not here. The conventions behind this page: [Docs and scra
| File | What it is | How to read it | Written by |
|---|---|---|---|
| [`SETUP.md`](../SETUP.md) | The environment reference: every tool's install row, the pinned compiler triple (§5.4), the daily command crib, the MCP lifecycle, the scratch conventions (§1a). 270 KB. | Grep by `§` or by tool name; §5.4 for the triple; the `## Tooling inventory` table for a tool's one-line row. | authored; kept current in the same change as any tool (R21) |
| [`ops/INDEX.md`](../ops/INDEX.md) | The environment reference: every tool's install row, the pinned compiler triple (§5.4), the daily command crib, the MCP lifecycle, the scratch conventions (§1a). 270 KB. | Grep by `§` or by tool name; §5.4 for the triple; the `## Tooling inventory` table for a tool's one-line row. | authored; kept current in the same change as any tool (R21) |
| [`ops/`](../ops/) | The environment reference split one page per section at the PA3 migration (2026-09); `ops/INDEX.md` lists every page, `ops/tooling-inventory.md` is the per-tool table. | Open `ops/INDEX.md`, then one page. | authored; kept current in the same change as any tool (R21) |
| [`retired/`](../retired/) | Files retired at the PA3 migration: the monolithic `SETUP.md`, `matching-cookbook.md` and its index, `effort-map.md`, the PA2-era files. Kept for history; never in an agent's load order. | Grep only. | frozen |
| [`project-architect.md`](../project-architect.md) | The Project Architect 3.0 methodology this repository runs under. | Read whole. | authored |
| [`verification.md`](../verification.md) | The byte-identity contract and the recorded run that proved it (218 of 218 from a clean rebuild, both SDK legs). | Read whole; §2 quotes the run's summary from `.run/P33/verify/`. | authored; quotes `tools/verify_contract.sh`'s output |
| [`second-oracle.md`](../second-oracle.md) | The independent boundary oracle (R34): what it covers, where it disagrees with the primary one by design. | Read whole. | authored; read by the Makefile and four tools |
| [`effort-map.md`](../effort-map.md) | The reasoning-effort doctrine for the AI agent: depth vs breadth, the per-phase map, the decision rule. Governs where `CLAUDE.md`'s tier language differs. | Read whole. | authored |
| [`retired/effort-map.md`](../retired/effort-map.md) | The reasoning-effort doctrine for the AI agent: depth vs breadth, the per-phase map, the decision rule. Governs where `CLAUDE.md`'s tier language differs. | Read whole. | authored |
| [`doc_links_pending.txt`](../doc_links_pending.txt) | Link targets promised by a document and created by a later task of the same phase (R80). Empty at every phase close. | One `path<TAB>task` per line. | authored per task |
| [`public-flip-runbook.md`](../public-flip-runbook.md) | The history-rewrite and publication procedure, its controls, its risk register, and what the host kept serving afterwards (§11). | By section; §11 for the purge and the probe. | authored |
@@ -33,8 +36,7 @@ in the Archive index, not here. The conventions behind this page: [Docs and scra
| File | What it is | How to read it | Written by |
|---|---|---|---|
| [`matching-cookbook.md`](../matching-cookbook.md) | Some 500 compiler idioms, each with its residual, mechanism, lever and byte proof. **3.5 MB — never read whole.** | Grep by `§` number or by symptom; the index below maps symptoms to sections. | authored, one section per proven idiom (R16) |
| [`cookbook-index.md`](../cookbook-index.md) | The symptom → section index of the cookbook. 560 KB. | Grep the symptom. | **generated** — `tools/cookbook_index.py`, checked in `make tools-health` |
| [`cookbook/INDEX.md`](../../cookbook/INDEX.md) | The cookbook since the PA3 migration: one line per technique, each a `cookbook/C<nnnn>.md` file (the monolith is in `docs/retired/`). Some 500 compiler idioms, each with its residual, mechanism, lever and byte proof. **3.5 MB — never read whole.** | Grep by `§` number or by symptom; the index below maps symptoms to sections. | authored, one section per proven idiom (R16) |
| [`docs/tool-index.md`](../tool-index.md) | every tool, keyed by the NEED it answers, with what proved it (its consumers) and what it hard-codes; the retired ones last | generated by `tools/tool_census.py` from the tree and `config/tool_dictionary.tsv` (the authored phase/need/verdict); `--check` in tools-health; the same data generates the day-one kit's manifest and its verbatim tool corpus | read it by need: find the phrase that matches what you are trying to do |
| [`gcc-2.7.2-map/`](../gcc-2.7.2-map/) | The codegen map of the pinned compiler by pass (scheduling, register allocation, loop, CSE/expression), with a citation audit; its README is the catalogue. | Start at the README's pass → residual → lever table. | authored; `tools/gccmap_cites.py --check` in tools-health |
| [`wave-playbook.md`](../wave-playbook.md) | The matching-campaign procedure: draw, cards, packs, draft, gate, harvest, checkpoint — each guard paired with the measurement that produced it. | By numbered step; the S80 addenda for the one-agent-per-function shape. | authored |
docs/wiki/Repository-layout.md
+1 -1
View File
@@ -6,7 +6,7 @@
|---|---|
| [`src/`](../../src/) | **The C.** One directory per binary: `src/` itself is the main executable (`SLUS_007.26`), `src/resident/` the always-resident engine, `src/ov_*/` the 141 overlays, `src/md_*/` the 75 code modules. [`src/shared/`](../../src/shared/) holds the bodies shared across binaries by [the dedup engine](The-dedup-engine.md) (`engine_core.h`, ~8.8 MB of `DEFINE_func_…()` macros — every translation unit that shares code includes it). Sony's library regions in main are `INCLUDE_ASM` tiles (`src/lib*.c`, `src/apicard*.c`); the five original hand-written routines are verbatim `__asm__` bodies. [`src/NOTICE.md`](../../src/NOTICE.md) states the licensing position. `src/shared/` holds every shared function body ONCE as a plain-C header (`ov/`, `slot_*/`, `main/`, Phase 35), included at each member's site — see [The dedup engine](The-dedup-engine.md). |
| [`config/`](../../config/) | The build definition: one splat yaml per binary (`splat.<alias>.yaml`), the symbol files (`symbols.us*.txt`, curated and build-valid — rule R15), the per-binary SHA1 contracts (`check.<alias>.sha`), the overlay and module registries ([`overlays.mk`](../../config/overlays.mk), [`modules.mk`](../../config/modules.mk) — generated by the onboarding scripts, never edited by hand), the dedup registry ([`dedup.us.yaml`](../../config/dedup.us.yaml)), the verbatim-assembly manifest ([`verbatim_manifest.json`](../../config/verbatim_manifest.json)), the Ghidra annotations as text ([`config/ghidra/`](../../config/ghidra/)). |
| [`tools/`](../../tools/) | Everything the project built (about 236 Python tools, 25 shell tools, 12 Ghidra scripts): the disc extractor, the byte gates, the matching harness, the dedup and family engines, the campaign machinery, the audits, the public-flip rewrite package. The pinned submodules live here too (`maspsx`, `asm-differ`, `decomp-permuter`, `m2c`) and the tracked `cc1` tarballs (`tools/bin/`). [`tools/README.md`](../../tools/README.md) carries the one AGPL statement; [`docs/SETUP.md`](../SETUP.md)'s tooling inventory has a one-line row for every tool. |
| [`tools/`](../../tools/) | Everything the project built (about 236 Python tools, 25 shell tools, 12 Ghidra scripts): the disc extractor, the byte gates, the matching harness, the dedup and family engines, the campaign machinery, the audits, the public-flip rewrite package. The pinned submodules live here too (`maspsx`, `asm-differ`, `decomp-permuter`, `m2c`) and the tracked `cc1` tarballs (`tools/bin/`). [`tools/README.md`](../../tools/README.md) carries the one AGPL statement; [`docs/ops/INDEX.md`](../ops/INDEX.md)'s tooling inventory has a one-line row for every tool. |
| [`docs/`](../) | The knowledge base and the record: the matching cookbook and its index, the gcc-2.7.2 codegen map, the memory map (addresses with provenance), the formats, the decision log, the accelerators, the wave playbook, the runbooks, the story, the retrospective, this wiki's sources (`docs/wiki/`, `docs/how-to-ai-decomp/`), the generated progress digests, badges and timeline, and the [Archive index](Archive-index.md) of closed-phase documents (their files left the tree at Phase 34; the history keeps them). Where each kind of knowledge goes, which files are generated, and how links are checked: [Docs and scratch conventions](Docs-and-scratch-conventions.md). |
| [`phase-ends/`](../../phase-ends/) | The governance record: `PhaseEnd_Phase<N>.md` for every phase, [`DIGEST.md`](../../phase-ends/DIGEST.md) (every phase's synopsis and every rule's full text), `CURRENT_PHASE.md` while a phase is open, `logs/` (each closed phase's working log, on demand only). |
| [`include/`](../../include/) | Headers, including the four generic splat preset headers (`include_asm.h`, `macro.inc`, `labels.inc`, `gte_macros.inc`). |
docs/wiki/The-matching-workflow.md
+2 -2
View File
@@ -129,8 +129,8 @@ earlier inline match never propagated — and a propagated body ships its signat
## The knowledge base, and feeding it
Before each match, consult [`docs/matching-cookbook.md`](../matching-cookbook.md) (grep it by symptom through
[`docs/cookbook-index.md`](../cookbook-index.md)); after each match, write the generalizable lesson back into the
Before each match, consult [`cookbook/INDEX.md`](../../cookbook/INDEX.md) (grep it by symptom through
[`cookbook/INDEX.md`](../../cookbook/INDEX.md)); after each match, write the generalizable lesson back into the
cookbook *and* into the tooling, so the next similar function is a one-shot (rule R16). Two laws learned at cost: only
harvest lessons from byte-proven results, and strip a credited lever from the accepted body and recompile before it
enters the base — one in three credited levers was byte-inert.
docs/wiki/Toolchain-setup.md
+1 -1
View File
@@ -4,7 +4,7 @@
Everything runs inside one Linux environment (the project uses WSL2 Ubuntu 24.04 on a single ext4 clone; builds never
run on a Windows-mounted path — the 9P bridge is slow and breaks file watching). The authoritative, evolvable reference
for every version, path and command is [`docs/SETUP.md`](../SETUP.md); this page is the map of it.
for every version, path and command is [`docs/ops/INDEX.md`](../ops/INDEX.md); this page is the map of it.
## The pinned triple — and why it is pinned
phase-ends/current/logs/T1.c1.md
+30
View File
@@ -0,0 +1,30 @@
# T1.c1 — tools-health: repoint off docs/retired/ (PA3 migration b8a6cbc54)
## Changed
- `tools/tool_census.py:14,42-43,51,143,331` — setup_rows() reads `docs/ops/tooling-inventory.md` (same `| group | tool | purpose |` shape, cells[2]); docstring updated. Beyond brief (needed for --check to get past the crash): COOKBOOK_SOURCES -> `docs/retired/{matching-cookbook,cookbook-index}.md`, RECORD_SOURCES effort-map -> `docs/retired/effort-map.md`; corpus_plan keeps the old corpus dest names (`replace("docs/retired/","docs/",1)`). Moves were pure renames (0 lines changed per `git diff --stat -M b8a6cbc54^ b8a6cbc54`).
- `Makefile:317-322` — `cookbook_index.py --check` commented out under a house DISABLED block (tool kept; CI `.github/workflows/no-rom.yml` still runs it, untouched).
- `tools/doc_links.py:38,54` — DEFAULT + TOPIC_PAGE entry `docs/SETUP.md` -> `docs/ops/INDEX.md` (kept, not dropped: Toolchain-setup.md still covers the topic).
- `tools/wiki_render.py:112` — selftest case g `/docs/SETUP.md` -> `/docs/ops/INDEX.md`.
- 36 BROKEN links retargeted: README.md:108, docs/gcc-2.7.2-map/README.md, docs/wiki/{Archive-index,Contributing-and-the-no-ROM-policy,Docs-and-scratch-conventions,Reference-index,Repository-layout,The-matching-workflow,Toolchain-setup}.md, docs/how-to-ai-decomp/{00-README,01-governance,06-knowledge-base,08-models-and-budgets}.md. Mapping per brief. Reference-index: the matching-cookbook row now describes cookbook/INDEX.md; the cookbook-index row dropped (would duplicate it). Docs-and-scratch-conventions rows reworded to cookbook/INDEX.md + cookbook_add.sh.
- `docs/wiki/Reference-index.md` — 3 new rows (`ops/`, `retired/`, `project-architect.md`): after the links were fixed doc_links still rc 1 on 60 UNCOVERED docs files (53 docs/ops/*, 6 docs/retired/*, docs/project-architect.md); a directory link covers its files (covered(), doc_links.py:128).
- `docs/ops/tooling-inventory.md:21` — note cookbook_index retired from tools-health (no own row exists; the CI row is the only one naming it).
- docs/wiki/Home.md: no `SETUP` link present at HEAD (grep); nothing to change. The dead link was the wiki_render selftest case.
## Verified
- `.venv/bin/python tools/doc_links.py` -> rc 0, `0 broken`, coverage 126 of 126, `doc_links: OK`
- `.venv/bin/python tools/wiki_render.py --selftest` -> rc 0, 12 cases 0 failed
- `.venv/bin/python tools/kit_lint.py` -> rc 0 `kit_lint: OK`
- `.venv/bin/python tools/kit_coverage.py` -> rc 0 `kit_coverage: OK`
- `.venv/bin/python tools/tool_census.py --check` -> rc 1, 65 gaps (NOT green; see Blocker)
- `make -n tools-health >/dev/null` -> rc 2 at `Makefile:516: sdk-dual`; control with HEAD Makefile (`make -n -f .run/t1c1/Makefile.head tools-health`) fails identically -> pre-existing, not this change.
## Blocker: tool_census --check
Past the SETUP crash, 65 gaps, all PA3-migration drift outside the brief:
- 22 `no dictionary row` — new PA3 tools (card.py, commit_task.sh, cookbook_add.sh, plan_edit.py, run.sh, outline.py, launch.py, ...) lack rows in `config/tool_dictionary.tsv` (authored data, 8 cols each).
- 43 corpus gaps in `decomp-architect/corpus/` (generated): missing copies for those tools, stale copies of changed tools/docs, missing `record/phase-ends/PhaseEnd_Phase37{,.5}.md`. Fix = `tools/tool_census.py --corpus` after the rows exist; that copies the untracked `phase-ends/PhaseEnd_Phase37.md` into the kit (brief forbids touching it) -> expert decision.
## Incident (expert must know)
`make -n tools-health` is NOT a dry run here: `.ONESHELL` + a recipe containing `$(MAKE)` makes make execute the whole recipe; sub-makes stay dry (-n in MAKEFLAGS) but plain shell lines run. Within sdk-dual that ran `rm -rf build/psyq` and two `cp $(main_MAPFILE) .run/P33/verify/main_{with,no}_sdk.map` (19:40, after the in-flight tools-health1 had printed `sdk-dual: OK`). Ran 3x (twice via tools-health, once `make -n sdk-dual` while diagnosing).
- `.run/P33/verify/main_no_sdk.map` (tracked) was clobbered with the WITH map -> restored with `git checkout --` (clean now).
- `build/psyq/` is gone. Restore = leg 3 of sdk-dual: `make extract BINARY=main && make check BINARY=main` (not run: brief forbids make builds and a run is in flight). The in-flight tools-health1 was past sdk-dual (at `report`).
- generalizable: `make -n` on a `.ONESHELL` target whose recipe calls `$(MAKE)` executes the recipe's non-make lines; the brief's `make -n tools-health` check is destructive on an SDK machine.
tools/doc_links.py
+2 -2
View File
@@ -35,7 +35,7 @@ import subprocess
import sys
REPO = pathlib.Path(__file__).resolve().parent.parent
DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/README.md", "docs/SETUP.md",
DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/README.md", "docs/ops/INDEX.md",
"docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md",
"docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md",
"docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md",
@@ -51,7 +51,7 @@ INDEX_PAGES = ("docs/wiki/Reference-index.md", "docs/wiki/Archive-index.md")
ARCHIVE_DIR = "docs/sunset/"
# the wiki page that covers a docs/ file's topic — for the wiki-first WARNING only (a hand-kept map; small on purpose)
TOPIC_PAGE = {
"docs/SETUP.md": "docs/wiki/Toolchain-setup.md",
"docs/ops/INDEX.md": "docs/wiki/Toolchain-setup.md",
"docs/verification.md": "docs/wiki/Verification-and-progress.md",
"docs/public-flip-runbook.md": "docs/wiki/The-ROM-firewall.md",
"docs/gen3-standards.md": "docs/wiki/Where-the-project-goes-next.md",
tools/tool_census.py
+6 -5
View File
@@ -11,7 +11,7 @@ two verbatim corpora, all from one hand-kept dictionary (P33.5 task 13.5; `--che
Two kinds of fact, deliberately separated (R33):
DERIVED on every run from the tree — the file set (two independent enumerations that must agree: `find` and `git ls-files`,
R34), each tool's docstring line, its SETUP row (`docs/SETUP.md`, matched by file name), its CONSUMERS (Makefile + config/*.mk
R34), each tool's docstring line, its SETUP row (`docs/ops/tooling-inventory.md`, matched by file name), its CONSUMERS (Makefile + config/*.mk
targets, the CI workflows, the wave playbook, and every other tool that imports it or names it), and hence its class:
LIVE (a runtime consumer), REFERENCED (only a SETUP/doc row), ORPHAN (neither).
AUTHORED once, in `config/tool_dictionary.tsv` (tab-separated, LF line endings, the eight columns below) — the ladder phase, the portability (PORTABLE / ADAPT / PROJECT-ONLY), the NEED
@@ -39,7 +39,8 @@ KIT = REPO / "decomp-architect"
MANIFEST = KIT / "tools" / "MANIFEST.md"
CORPUS_TOOLS = KIT / "corpus" / "tools"
CORPUS_COOK = KIT / "corpus" / "cookbook"
COOKBOOK_SOURCES = ["docs/matching-cookbook.md", "docs/cookbook-index.md"] + [p.relative_to(REPO).as_posix() for p in sorted((REPO / "docs" / "gcc-2.7.2-map").glob("*")) if p.is_file()]
# the two cookbook files moved to docs/retired/ at the PA3 migration (b8a6cbc54, bytes unchanged); the corpus keeps their old names
COOKBOOK_SOURCES = ["docs/retired/matching-cookbook.md", "docs/retired/cookbook-index.md"] +[p.relative_to(REPO).as_posix() for p in sorted((REPO / "docs" / "gcc-2.7.2-map").glob("*")) if p.is_file()]
# P33.5 task 14.5 (Drew: "the whole of our experience"): the RECORD as the third dictionary — the distilled records and the
# phase-by-phase record, verbatim. The phase worklogs (phase-ends/logs/) are deliberately NOT here (R19: on-demand archives, 30k
# lines); their distillation is the task-14.5 mining pass, banked into the kernels/accelerators. Layout under corpus/record/:
@@ -47,7 +48,7 @@ COOKBOOK_SOURCES = ["docs/matching-cookbook.md", "docs/cookbook-index.md"] + [p.
CORPUS_RECORD = KIT / "corpus" / "record"
RECORD_SOURCES = ([p.relative_to(REPO).as_posix() for p in sorted((REPO / "docs" / "how-to-ai-decomp").glob("*.md"))]
+ ["docs/decision-log.md", "docs/accelerators.md", "docs/retrospective.md", "docs/story.md", "docs/wave-playbook.md",
"docs/effort-map.md", "docs/gen3-standards.md", "docs/gen3-handoff.md", "phase-ends/DIGEST.md"]
"docs/retired/effort-map.md", "docs/gen3-standards.md", "docs/gen3-handoff.md", "phase-ends/DIGEST.md"]
+ [p.relative_to(REPO).as_posix() for p in sorted((REPO / "phase-ends").glob("PhaseEnd_Phase*.md"),
key=lambda q: [float(x) if x.replace(".", "").isdigit() else x for x in re.split(r"(\d+(?:\.\d+)?)", q.stem)])])
@@ -139,7 +140,7 @@ def docstring_line(path):
def setup_rows():
"""file name -> the SETUP row's purpose cell (first 160 chars)."""
out = {}
for ln in (REPO / "docs" / "SETUP.md").read_text(encoding="utf-8", errors="replace").splitlines():
for ln in (REPO / "docs" / "ops" / "tooling-inventory.md").read_text(encoding="utf-8", errors="replace").splitlines():
if not ln.startswith("|"):
continue
cells = [c.strip() for c in ln.strip().strip("|").split("|")]
@@ -327,7 +328,7 @@ def corpus_plan(recs):
dest = CORPUS_TOOLS / r["phase"] / r["path"].removeprefix("tools/")
copies.append((REPO / r["path"], dest))
for s in COOKBOOK_SOURCES:
copies.append((REPO / s, CORPUS_COOK / pathlib.Path(s).relative_to("docs")))
copies.append((REPO / s, CORPUS_COOK / pathlib.Path(s.replace("docs/retired/", "docs/", 1)).relative_to("docs")))
for s in RECORD_SOURCES:
copies.append((REPO / s, record_dest(s)))
return copies, pointers
tools/wiki_render.py
+1 -1
View File
@@ -109,7 +109,7 @@ def selftest():
("[d](../verification.md)", f"[d]({GH}/blob/main/docs/verification.md)"),
("[e](../../README.md#license)", f"[e]({GH}/blob/main/README.md#license)"),
("[f](../../tools/)", f"[f]({GH}/tree/main/tools)"),
("[g](/docs/SETUP.md)", f"[g]({GH}/blob/main/docs/SETUP.md)"),
("[g](/docs/ops/INDEX.md)", f"[g]({GH}/blob/main/docs/ops/INDEX.md)"),
("![h](../story-timeline.svg)", f"![h]({RAW}/docs/story-timeline.svg)"),
("[i](https://example.org/x.md)", "[i](https://example.org/x.md)"),
("[j](#anchor)", "[j](#anchor)"),