diff --git a/docs/doc_links_pending.txt b/docs/doc_links_pending.txt index fe3b326c6..d96fb2281 100644 --- a/docs/doc_links_pending.txt +++ b/docs/doc_links_pending.txt @@ -1,5 +1,4 @@ # docs/doc_links_pending.txt — link targets promised by a document and created by a later task of the SAME phase. # Format: . tools/doc_links.py reports these as PENDING instead of # BROKEN; `--strict` (gate 2) refuses any entry, so this file must be EMPTY before a PhaseEnd. -docs/wiki/Reference-index.md Phase 33.5 task 4 (the Reference index — the only sanctioned place for wiki→docs links) docs/wiki/Archive-index.md Phase 33.5 task 6 (the Archive index — one row per sunset document) diff --git a/docs/gen3-handoff.md b/docs/gen3-handoff.md index e55917d6c..8e7680fe3 100644 --- a/docs/gen3-handoff.md +++ b/docs/gen3-handoff.md @@ -204,11 +204,11 @@ PCSX-Redux, then the disc rebuild with recompressed payloads. ## 8. Governance for Gen3 The framework carries over unchanged: the constitution, the session protocol, the digest, the two gates, one task at -a time, the rules R1–R73 plus the Phase-33 candidates (a)–(h) once ratified. A new generation starts with a fresh +a time, the rules R1–R83 (the Phase-33 candidates were ratified as R74–R83 at the Phase-33.5 gate). A new generation starts with a fresh plan in plan mode at Max. Three things the record says to do first: 1. **Measure the shape before choosing** (`docs/hindsight-study.md` §2.1): which structures own most of the 61,898 - data symbols, and which families own most of the 44,243 pins — a census with a self-asserting scanner, checked + data symbols, and which families own most of the 43,925 pins — a census with a self-asserting scanner, checked against a case whose answer is known. 2. **Build the differential harness for the new question** before the campaign: "is this rename/type change byte-neutral?" has two paths — the per-binary gate and the fleet run — and a scanner that asserts every reference diff --git a/docs/wiki/Contributing-and-the-no-ROM-policy.md b/docs/wiki/Contributing-and-the-no-ROM-policy.md index a5ec8bce4..f2769db70 100644 --- a/docs/wiki/Contributing-and-the-no-ROM-policy.md +++ b/docs/wiki/Contributing-and-the-no-ROM-policy.md @@ -36,6 +36,26 @@ The matching frontier is empty, so new matches are not the need. The work that h - **Tooling and documentation** — every tool has a row in [`docs/SETUP.md`](../SETUP.md)'s inventory; a tool change ships with its siblings wired to use it and its docs in the same change. +## AI use — conduct + +The project was carried out by an AI coding agent and says so once, at the project level; that is the disclosure. What +keeps it from being a liability is a short set of conduct rules, which apply to any contribution made with an agent's +help (the charter is [`docs/gen3-standards.md`](../gen3-standards.md) §3; the direction they serve is +[Where the project goes next](Where-the-project-goes-next.md)): + +1. **The byte gate is the only claim of success.** No match is reported that the gate has not proven; "the compiler ran" + and "looks equivalent" are not results, and every "done" is written from a tool's printed success line. +2. **Names and types are evidence-based, never guessed.** A function or variable is renamed only on a recorded basis — a + string it prints, a cross-reference chain, the debug menu, a live-RAM datapoint, a community label with provenance. No + evidence, no name: `func_80xxxxxx` is honest and greppable. A model may propose; it may never assert. +3. **Outward text is written by a person.** Issues, pull requests, posts and notes to other projects are written by the + maintainer the way a developer writes — short and plain — never a model draft with the tells removed; and the target + project's own AI policy is read first (sotn-decomp, for one, requires pull requests made without autonomous tooling). +4. **No automated traffic against community infrastructure.** decomp.me asks that nobody scrape it or hook an LLM to it; + the project's use of it is a person in a browser, and its local replica of that toolchain exists so proofs run at home. +5. **Agents assist; a person owns.** Every change is justifiable by a person from the record — the phase logs, the + decision log, the cookbook's byte proofs. + ## The one invariant Every change to `src/`, `config/` or `include/` is held to the byte gate: `make check BINARY=` stays green for diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index 7c643702c..eb597c652 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -8,14 +8,18 @@ on every build. The repository is [github.com/Druthulu/BFM-decomp](../../README. The claim is narrow and machine-checkable: *matching* means byte-identical output, nothing "functionally equivalent" counts, and what the repository claims is exactly what `make check-all` proves. The live numbers are generated into the -README's progress block and [`docs/progress.json`](../progress.json), never typed by hand; at the Phase-33 close -(September 2026) all three metrics — functions, instructions, distinct code — read 100%, with two things deliberately -not our C and stated as such: Sony's PsyQ library objects linked into the main executable (1,256 functions) and five -hand-written assembly routines kept verbatim. +README's progress block and `docs/progress.json`, never typed by hand; at the Phase-33 close (September 2026) all three +metrics — functions, instructions, distinct code — read 100%, with two things deliberately not our C and stated as such: +Sony's PsyQ library objects linked into the main executable (1,256 functions) and five hand-written assembly routines +kept verbatim. The project was carried out end to end by an AI coding agent (Claude Code) under a written constitution and a two-gate phase system, in twelve weeks. How that was done — and what it cost — is the second half of this wiki. +**This wiki is the source of truth for the project's documentation.** The records behind it (the cookbook, the decision +log, the memory map, the environment reference) live under `docs/` and are listed, with how to read each one, in the +[Reference index](Reference-index.md); when a page and a record disagree, the record wins and the page gets fixed. + ## Using the project | Page | What it answers | @@ -28,9 +32,22 @@ phase system, in twelve weeks. How that was done — and what it cost — is the | [Overlays and modules](Overlays-and-modules.md) | The disc, the 218 binaries, where each loads and how every load address was proven | | [Ghidra rebuild from text](Ghidra-rebuild-from-text.md) | The reverse-engineering database is not in git; here is how it regenerates from text + the disc, and the proof that it does | | [Verification and progress](Verification-and-progress.md) | The contract run, the three metrics and how they are computed, what CI proves without a disc, the published numbers | -| [Contributing and the no-ROM policy](Contributing-and-the-no-ROM-policy.md) | What may never enter the repository, what a useful contribution looks like now that the frontier is empty, the license split | +| [Contributing and the no-ROM policy](Contributing-and-the-no-ROM-policy.md) | What may never enter the repository, what a useful contribution looks like now that the frontier is empty, the AI-use conduct rules, the license split | | [Tools from this project](Tools-from-this-project.md) | xsig, the permuter driver, the codegen map, the decomp.me replica, the drafter write-up — what stands on its own for other projects | +## Working conventions + +| Page | What it answers | +|---|---| +| [Docs and scratch conventions](Docs-and-scratch-conventions.md) | Where each kind of knowledge goes, which files are generated, how links are checked, what the archive is, what may be tracked under `.run/` | +| [The ROM firewall](The-ROM-firewall.md) | The rule with no private exemption, the nine classes of ROM-derived content, a copyable `.gitignore`, the audit and CI, what a history rewrite costs | + +## Reference + +The [Reference index](Reference-index.md): every live reference and generated file under `docs/` — the environment +reference, the formats and the memory map, the cookbook and the codegen map, the progress files, the releases, the record, +the Gen3 inputs — with what each is, how to read it and who writes it. + ## How to AI-decomp [How to AI-decomp](How-to-AI-decomp.md) — the transferable part: thirteen chapters on running a byte-exact decompilation @@ -39,16 +56,17 @@ gate, the bootstrap order, oracles and instruments, cards/lanes/waves, the knowl models and budgets, the economics as measured, integration and propagation, publishing, and a museum of the failures that looked right at the time. -## The records behind the wiki +## Where the project goes next -Everything here is a summary of files kept *as it happened*; when a page and a record disagree, the record wins. +[Where the project goes next](Where-the-project-goes-next.md): the flip and the Gen2 exit (Phase 34), then Gen3 — +readability on a byte-exact floor: the style bar, the measured gap, the one invariant, the order of work, the levers. -- [`docs/story.md`](../story.md) — the narrative, first commit to public flip; [`docs/retrospective.md`](../retrospective.md) — what we believed, what failed, what it cost -- [`docs/matching-cookbook.md`](../matching-cookbook.md) (some 500 compiler idioms, each byte-proven; grep it by section — it is 3.5 MB) and its symptom index [`docs/cookbook-index.md`](../cookbook-index.md); [`docs/gcc-2.7.2-map/`](../gcc-2.7.2-map/) — the codegen map of the pinned compiler, by pass -- [`docs/decision-log.md`](../decision-log.md) — every strategic pivot with its measurement and its hindsight; [`docs/accelerators.md`](../accelerators.md) — what would have made earlier phases faster -- [`phase-ends/`](../../phase-ends/) — one PhaseEnd per phase, the digest of every phase and every rule ([`DIGEST.md`](../../phase-ends/DIGEST.md)), the per-phase logs -- [`docs/SETUP.md`](../SETUP.md) — the environment reference, every tool's row; [`docs/verification.md`](../verification.md) — the contract and the last recorded run -- [`docs/memory-map.md`](../memory-map.md) and [`docs/formats.md`](../formats.md) — addresses with provenance; the disc, container, archive and LZSS formats +## History + +The narrative and the retrospective are in the Reference index (`docs/story.md`, `docs/story-timeline.md`, +`docs/retrospective.md`); the phase-by-phase record is `phase-ends/` ([Repository layout](Repository-layout.md)); the +documents that served one phase and were then retired are in the [Archive index](Archive-index.md), each with what came +of it and where its information lives now. *Brave Fencer Musashi is © 1998 Square. This project is not affiliated with or endorsed by Square Enix. `src/` asserts no license ([`src/NOTICE.md`](../../src/NOTICE.md)); the project's own tools and documents are AGPL-3.0 ([`LICENSE`](../../LICENSE)).* diff --git a/docs/wiki/Reference-index.md b/docs/wiki/Reference-index.md new file mode 100644 index 000000000..52678f96d --- /dev/null +++ b/docs/wiki/Reference-index.md @@ -0,0 +1,92 @@ +# Reference index + +Every live reference and generated file under `docs/`, in one place: what it is, how to read it, and who writes it. +This page is the **only** place a wiki page links into `docs/` — every other page links a wiki page for its topic, and +the link checker (`tools/doc_links.py`) derives its allow-list of `docs/` targets from the rows below (a directory row +covers everything under it). The two index pages, this one and the [Archive index](Archive-index.md), are exempt from +that rule; the README keeps its own handful of direct links. Files that left `docs/` at the Phase-33.5 consolidation are +in the Archive index, not here. The conventions behind this page: [Docs and scratch conventions](Docs-and-scratch-conventions.md). + +## Environment and verification + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`SETUP.md`](../SETUP.md) | The environment reference: every tool's install row, the pinned compiler triple (§5.4), the daily command crib, the MCP lifecycle, the scratch conventions (§1a). 270 KB. | Grep by `§` or by tool name; §5.4 for the triple; the `## Tooling inventory` table for a tool's one-line row. | authored; kept current in the same change as any tool (R21) | +| [`verification.md`](../verification.md) | The byte-identity contract and the recorded run that proved it (218 of 218 from a clean rebuild, both SDK legs). | Read whole; §2 quotes the run's summary from `.run/P33/verify/`. | authored; quotes `tools/verify_contract.sh`'s output | +| [`second-oracle.md`](../second-oracle.md) | The independent boundary oracle (R34): what it covers, where it disagrees with the primary one by design. | Read whole. | authored; read by the Makefile and four tools | +| [`effort-map.md`](../effort-map.md) | The reasoning-effort doctrine for the AI agent: depth vs breadth, the per-phase map, the decision rule. Governs where `CLAUDE.md`'s tier language differs. | Read whole. | authored | +| [`doc_links_pending.txt`](../doc_links_pending.txt) | Link targets promised by a document and created by a later task of the same phase (R80). Empty at every phase close. | One `pathtask` per line. | authored per task | +| [`public-flip-runbook.md`](../public-flip-runbook.md) | The history-rewrite and publication procedure, its controls, its risk register, and what the host kept serving afterwards (§11). | By section; §11 for the purge and the probe. | authored | + +## Formats and addresses + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`formats.md`](../formats.md) | The disc, the `.CD` containers, the PAC archives, the LZSS codec (game semantics), the SQV split. Anything not byte-verified is marked. | By section. | authored | +| [`memory-map.md`](../memory-map.md) | Every address with its source and verification status (G5): the loader, the overlay slots, player state, the RAM-dump datapoints, the community labels. | By section (`§S…` = the session that established it); Ghidra is authoritative once an address is imported. | authored | +| [`idxtab-map.md`](../idxtab-map.md) | The master load map: every payload's index-table and destination-pointer entries. | Read whole. | authored from `tools/idxtab_map.py` | +| [`debug-menu-list.txt`](../debug-menu-list.txt) | The retail debug menu's AREA/SCENE list, transcribed from the live menu. | Read whole. | authored (primary data) | +| [`disc-completeness.md`](../disc-completeness.md) | The disc-completeness audit: every payload claimed by a binary or explicitly excluded, with evidence. | Read whole. | authored; reproduce with `tools/disc_code_sweep.py` | +| [`disc-ledger.md`](../disc-ledger.md) | The completeness partition, residue 0. | Read the summary line; the table is per payload. | **generated** — `tools/disc_audit.py`, `make audit-disc` | + +## The knowledge base + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`matching-cookbook.md`](../matching-cookbook.md) | Some 500 compiler idioms, each with its residual, mechanism, lever and byte proof. **3.5 MB — never read whole.** | Grep by `§` number or by symptom; the index below maps symptoms to sections. | authored, one section per proven idiom (R16) | +| [`cookbook-index.md`](../cookbook-index.md) | The symptom → section index of the cookbook. 560 KB. | Grep the symptom. | **generated** — `tools/cookbook_index.py`, checked in `make tools-health` | +| [`gcc-2.7.2-map/`](../gcc-2.7.2-map/) | The codegen map of the pinned compiler by pass (scheduling, register allocation, loop, CSE/expression), with a citation audit; its README is the catalogue. | Start at the README's pass → residual → lever table. | authored; `tools/gccmap_cites.py --check` in tools-health | +| [`wave-playbook.md`](../wave-playbook.md) | The matching-campaign procedure: draw, cards, packs, draft, gate, harvest, checkpoint — each guard paired with the measurement that produced it. | By numbered step; the S80 addenda for the one-agent-per-function shape. | authored | +| [`accelerators.md`](../accelerators.md) | What was learned late that a future project should know on day one: each entry with when it was found, when it could have been, what it would have saved. | Read whole; the entries are short. | authored | +| [`generic-decomp-package.md`](../generic-decomp-package.md) | The thesis for the next project: what it inherits and does before cracking — the knowledge base seeded from the compiler's source and sibling projects, the structural tooling, the similarity join as a band not a point, the differential-oracle harness, the widening review. | Read whole. | authored | +| [`decision-log.md`](../decision-log.md) | Every strategic pivot with its context, what failed, the measurement and the hindsight (R31). 280 KB, append-only. | By date heading. | authored | +| [`psyq-worklist.md`](../psyq-worklist.md) | The per-object record of the 1,256 Sony library functions linked into the executable, and the named residue. | By library. | authored from `tools/psyq_identify.py`; complete | +| [`psyq-sample-idioms.md`](../psyq-sample-idioms.md) | Style notes mined from the SDK's sample sources. | Read whole. | authored (a seed) | +| [`concurrency-design.md`](../concurrency-design.md) | The concurrency contract for overlapping draft, gate, propagate and fleet-verify runs; implemented in the gate snapshot layer. | Read whole. | authored | +| [`tooling-audit.md`](../tooling-audit.md) | The Phase-26 tooling-integrity audit, finding by finding — the evidence that several "walls" were the instruments. 400 KB. | By finding number. | authored | + +## Verification and progress (generated) + +| File | What it is | Written by | +|---|---|---| +| [`progress.md`](../progress.md), [`progress.fleet.md`](../progress.fleet.md), [`progress.json`](../progress.json) | The executable's digest, the fleet roll-up, and the published data the README block, the badges and CI read. | `tools/progress.py`, `make report BINARY=main`; freshness asserted by `make audit-digest` | +| [`badges/`](../badges/) | The four shields endpoints. | `tools/progress.py` | +| [`difficulty.md`](../difficulty.md), [`duplicates.md`](../duplicates.md), [`duplicates.cross.md`](../duplicates.cross.md) | The executable's harvest queue and self-duplicates; the cross-binary duplicate groups. | `tools/difficulty.py`, `tools/dup_report.py`, `make report BINARY=main` | +| [`backlog.md`](../backlog.md) | The near-miss ledger (0 open). | `tools/backlog.py`, `make report BINARY=main` | +| [`story-timeline.md`](../story-timeline.md), [`story-timeline.svg`](../story-timeline.svg) | The dated digest timeline and its chart. | `tools/timeline.py`; `--check` in `make audit-digest` | +| [`commit-map.tsv`](../commit-map.tsv) | The ordinal → hash map for commits cited in historical documents after the history rewrite. | `tools/public_rewrite/build_commit_map.py`, one-shot | + +## Releases and outreach + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`permuter-ils.md`](../permuter-ils.md) | The permuter recipe, the masked scorer and the local-search driver, and the upstream offer's outcome. | Read whole. | authored | +| [`matching-drafter-pipeline.md`](../matching-drafter-pipeline.md) | The local drafter model's pipeline, its measured arc, and what is not published. | Read whole. | authored | +| [`gen2-mips-matching-model.md`](../gen2-mips-matching-model.md) | The design and every measurement behind that write-up. | By section. | authored | +| [`community-matching-model-plan.md`](../community-matching-model-plan.md) | The parked plan to publish the drafter's weights and dataset (waits on a licensing decision). | Read whole. | authored | +| [`decompme-preset.md`](../decompme-preset.md) | The decomp.me preset for the pinned compiler, its local proof, and the post-flip request procedure. | §5 for the procedure. | authored | +| [`outreach/`](../outreach/) | The notes to neighbouring projects, written for the maintainer to send after the flip. | Read whole. | authored | + +## The record + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`story.md`](../story.md) | The narrative, first commit to public flip. | Read whole. | authored | +| [`retrospective.md`](../retrospective.md) | What was believed, what failed, what it cost, what to do sooner. | Read whole; §4 is the list for the next project. | authored; assembled with `tools/mine_hindsight.py` | +| [`phase34-seed.md`](../phase34-seed.md) | The plan seed for the flip phase (Phase 34): every task, actor, effort and carrying document. | Read whole. | authored | +| [`history/`](../history/) | The provenance archive: the original brief (superseded, with its corrected claims), the methodology version the constitution was generated from, an early experiment. Nothing current. | Its README. | authored | +| `phase-ends/` | Not under `docs/`, but the record proper: one PhaseEnd per phase, the digest of every phase and every rule, the working logs. | [Repository layout](Repository-layout.md). | authored per phase | + +## Gen3 inputs + +| File | What it is | How to read it | Written by | +|---|---|---|---| +| [`gen3-standards.md`](../gen3-standards.md) | The charter for the readability generation: accuracy met, the style bar (sotn's guide plus four rules), the AI-use conduct, the measured gap, a definition of done. | Read whole. | authored | +| [`gen3-handoff.md`](../gen3-handoff.md) | Where Gen2 ends, the starter census with its commands, the one invariant, the levers, shiftability scoped, the parked ideas. | §3 for the census, §5 for the levers. | authored | +| [`gen3-parking-lot.md`](../gen3-parking-lot.md) | The asset-export / native-rebuild survey, with the decisive probes. | Read whole. | authored | +| [`actor-struct.md`](../actor-struct.md) | The recovered player/actor structure (~154 fields, live-verified) and the byte-neutrality finding. | Read whole. | authored | +| [`struct-core-pivot.md`](../struct-core-pivot.md) | What struct recovery can and cannot do for matching — the Phase-16 wall and the pivot. | Read whole. | authored | +| [`wall-taxonomy.md`](../wall-taxonomy.md) | The census of residual stubs by blocker class, validated against the bytes. | By class. | authored from `tools/wall_taxonomy.py` | + +Not listed: `docs/wiki/` and `docs/how-to-ai-decomp/` are this wiki's sources; `docs/sunset/` is the archive, indexed by +the [Archive index](Archive-index.md). diff --git a/docs/wiki/Where-the-project-goes-next.md b/docs/wiki/Where-the-project-goes-next.md new file mode 100644 index 000000000..4cd94c628 --- /dev/null +++ b/docs/wiki/Where-the-project-goes-next.md @@ -0,0 +1,97 @@ +# Where the project goes next + +The matching is finished: 218 binaries rebuild byte for byte and the frontier is empty. What remains is in two parts. +The first is short and mostly administrative — the public flip and the close of the second generation. The second is the +third generation: making code that *matches* into code a maintainer would call *finished*. + +## Phase 34 — the flip and the Gen2 exit + +The repository's history was rewritten before publication to remove every game-derived byte and force-pushed in +place. The flip itself waits on one thing outside the project's control: GitHub Support purging the old objects, which +the host's Activity view still serves for every pre-force-push tip ([The ROM firewall](The-ROM-firewall.md)). A probe +of the old hashes runs daily; the flip is gated on it printing PASS (or, if the purge stalls, on recreating the +repository under the same name). + +After that, in order: the visibility change and the checks that follow it (the probe after the flip, the Actions tab +green, the wiki enabled — GitHub offers wikis only on public repositories on the free plan); the outward actions the +maintainer does in his own words — the decomp.me preset request, a note to the neighbouring Archipelago project whose +RAM map cited this one, the progress boards (decomp.dev, frogress), the wiki push, the tool announcement and the +topics on the released tool repositories; the aftercare (other clones reset to the rewritten history, the rewrite +scratch deleted); and the Phase-34 close at **v2.0.0**, which is the Gen2 exit. The seed for that phase, with every +task, actor and effort level, is [`docs/phase34-seed.md`](../phase34-seed.md). + +## Gen3 — readability, on a byte-exact floor + +**The floor.** Accuracy is not a Gen3 concern; it is the floor Gen3 stands on. "Matched" means instruction-identical +including register allocation *and* a whole-binary hash equal to the original, checked inside every build of every +binary, with a clean rebuild after every batch. Nothing functionally equivalent has ever been counted, and what is not +the project's own C is stated: 1,256 Sony library objects linked byte-identical into the executable, five hand-written +assembly routines kept verbatim. + +**The bar.** The community's expectations are written down, and the project adopts sotn-decomp's style guide as its +baseline: `camelCase` locals, `g_PascalCase` globals, `PascalCase` types and functions, the project's own `u8…u32` +types, clang-format at 4 spaces and 80 columns, decimal for counts and hexadecimal for addresses and masks, +`// !FAKE:` on anything that exists only to force a match, functions in assembly order in the file of the system they +implement — and, ranked first, *"if you are not sure what something does, it is better to leave it unnamed than name it +wrongly."* To that the project adds four rules of its own, because its code was produced by a machine at scale: + +1. **Every name carries evidence** — a string, a cross-reference chain, the debug menu, a live-RAM datapoint, a community + label with provenance — recorded the way addresses are. A model may propose a name; it may never assert one. +2. **Every register pin goes**, or stays with a `// !FAKE:` line naming the residual. The count is a published metric until + it is zero. +3. **Shared engine functions live as C, not as macros** — the dedup engine's macro bodies become `.c` files per system, + byte-neutrally, proven per family. +4. **Types come from the code, names from observation** — structure from the access patterns the compiler locked into the + bytes; the 1,232 struct definitions unified before any field is named. + +**The gap, as measured.** The starter census below was derived on 2026-09-07 with the commands in +[`docs/gen3-handoff.md`](../gen3-handoff.md) §3 — re-derive it, do not trust it: + +| Quantity | Value | Note | +|---|---|---| +| Raw address casts `*(T *)0x80…` | 143 | the last of a class the type-lift tools already handle | +| Distinct `D_80xxxxxx` data symbols | 61,898 | most are fields of a handful of structures the engine indexes | +| Distinct `func_80xxxxxx` names | 16,335 | across 4,287 C files | +| Register-pin declarations | 43,925 | `grep -rhoE 'register [^;/]*__asm__\("\$?[a-z0-9]+"\)' src --include=*.c --include=*.h \| wc -l`; 43,857 in the `"$reg"` form, 68 in the bare form | +| Shared bodies living as macros | 5,147 | in one 8.4 MB header | +| Struct definitions | 1,232 | many drafter-invented variants of one type | +| Curated names in the symbol files | 1,083 | the names the build already knows | + +**The one invariant.** Every Gen3 edit is gated exactly like a match was: the per-binary check for every binary a change +touches, and the clean fleet run after anything that touches a shared body, a shared header or the executable. A rename +is a symbol-file change mirrored into Ghidra by the headless script, never a hand edit of assembly. A shared body is one +source and up to 138 binaries, so the gate runs on all of them. Types are a comprehension lever, not a byte lever — the +compiler does not care what a field is called — which is exactly why struct work is safe and exactly why it still goes +through the gate. + +**The order of work:** pins off → the macro bodies into shared C files → struct unification → names with evidence → +formatting and comments. Three things the record says to do first: measure the shape (which structures own most of the +data symbols, which families own most of the pins — with a self-asserting scanner checked against a known case), build +the differential harness for the new question ("is this edit byte-neutral?" has two paths, and a scanner must assert +every reference to a renamed symbol was rewritten), and batch by leverage — a pin in a shared body comes off for 138 +binaries at once. + +**The levers Gen3 inherits** are tools already in the tree: the fleet-wide type lift, the call-site cast and canonical- +signature tools, the declaration layer of the reconcile ladder, the allocation-order and RTL-dump readers to consult +before touching a pin, the propagation and registry machinery that keeps 2,220 dedup groups consistent through a rename, +the headless symbol mirror into Ghidra, and the atlas for grouping candidates. The table with each lever's purpose is +[`docs/gen3-handoff.md`](../gen3-handoff.md) §5. + +**The use of AI — conduct.** The project is openly machine-carried and says so once, at the project level. The rules that +keep that from being a liability: the byte gate is the only claim of success; names and types are evidence-based, never +guessed; outward text is written by a person, the way a developer writes, after reading the target project's own AI +policy; no automated traffic against community infrastructure; agents assist, a person owns. They are stated in full on +the [Contributing](Contributing-and-the-no-ROM-policy.md) page and in [`docs/gen3-standards.md`](../gen3-standards.md) §3. + +**Shiftability, honestly scoped.** Overlays are position-locked at one slot by the loader; LZSS recompression is not +byte-stable, so a rebuilt disc verifies by booting, not by hash; the executable's Sony regions are linked objects. +The order that follows: symbolic references first (byte-neutral, gated), then a shifted overlay that boots in the +emulator, then the disc rebuild. Parked beyond that, each with its state recorded in the handoff: asset export and a +native rebuild, a PC port, randomizer-grade tooling, the community matching model (pipeline published; data and weights +wait on a licensing decision), the Japanese release and the prototypes as extra versions, the Sony libraries from source. + +## What a new project takes from this + +The order this project would use if it started again — and the firewall, the conventions and the conduct rules above — +are packaged as a day-one kit for the next decompilation, described on the wiki page *Start a new decomp project* once +Phase 33.5 ships it. diff --git a/docs/wiki/_Sidebar.md b/docs/wiki/_Sidebar.md index 8b258067d..ace14e2fe 100644 --- a/docs/wiki/_Sidebar.md +++ b/docs/wiki/_Sidebar.md @@ -16,6 +16,9 @@ - [Docs and scratch conventions](Docs-and-scratch-conventions.md) - [The ROM firewall](The-ROM-firewall.md) +**Reference** +- [Reference index](Reference-index.md) + **How to AI-decomp** - [Index](How-to-AI-decomp.md) - [00 Read me first](../how-to-ai-decomp/00-README.md) @@ -31,3 +34,9 @@ - [10 Integration and propagation](../how-to-ai-decomp/10-integration-and-propagation.md) - [11 Publishing](../how-to-ai-decomp/11-publishing.md) - [12 The failure museum](../how-to-ai-decomp/12-failure-museum.md) + +**Where the project goes next** +- [Where the project goes next](Where-the-project-goes-next.md) + +**History** +- [Archive index](Archive-index.md) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 73dd47edd..9bb70136c 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -42,7 +42,7 @@ in-tree links to `docs/wiki/.md`. 8. `.run/`: only what git tracks; no sca - [x] **1** Unblock, code side: tool docstrings re-pointed (`decl_from_use.py`, `integration_resolver.py`, `jtbl_carve.py`, `autopsy.py`, `residual_class.py`); CLAUDE.md lines 11 / 61 / the effort-default sentence — xHigh — see Log 2026-09-07 Task 1 - [x] **2** Unblock, doc side: banners (`concurrency-design`, `gen2-mips-matching-model`, `psyq-worklist`, `distill/*`, `hindsight-study` HISTORICAL + supersession table); SETUP §1a rewritten; the pin count re-derived (one dated number + its command, both gen3 docs) — xHigh — see Log 2026-09-07 Task 2 - [x] **3** Wiki — conventions + firewall: `Docs-and-scratch-conventions.md`, `The-ROM-firewall.md` (the gitignore template block); sidebar rows; Repository-layout/Contributing/ch.11 links; forward links in `doc_links_pending.txt` — Max — see Log 2026-09-07 Task 3 -- [ ] **4** Wiki — reference + direction: `Reference-index.md` (list-driven), `Where-the-project-goes-next.md`, Contributing "AI use — conduct", Home rewritten, sidebar sections — xHigh — **then P6 rules check** +- [x] **4** Wiki — reference + direction: `Reference-index.md` (list-driven), `Where-the-project-goes-next.md`, Contributing "AI use — conduct", Home rewritten, sidebar sections — xHigh — see Log 2026-09-07 Task 4 — **P6 rules check done after it** - [ ] **5** Integrate the six (hand-matching-process; wave-metrics + the s37w block → wave-playbook; portable-decomp-workflow vs the 13 chapters; generic-decomp-package attributed; hindsight §2.1/§7 → ch.03 + permuter-ils; the gen3 docs → the direction page); re-point every referrer incl. `retrospective.md:10–11` — xHigh - [ ] **6** Sunset moves (~61 files incl. the 6 frozen snapshots) with the referrer census by command; `docs/sunset/README.md`; `Archive-index.md` rows (backticked paths + `v1.32.1`); DIGEST §4 + `docs/history/README.md` refreshed; ledger coverage: every `git ls-files docs` path outside wiki/how-to/sunset is KEEP in the ledger — xHigh - [ ] **7** Checkers: `doc_links.py` (sunset refusal, index-derived allowlist, wiki-first warnings, TRACKED/UNTRACKED citation classifier + `--disk`, coverage ⊆); reachability in `wiki_render --selftest`; `timeline.py` regen in `report BINARY=main` + `--check` in `audit-digest`; the 14 dangling cookbook cites; the gitignore-template diff; tools-health wiring; SETUP rows — xHigh @@ -259,57 +259,71 @@ one-list rule with the `untracked_after_rewrite.txt` sibling, the rewrite lesson layout (`docs/` row, `.run/` row, the not-in-repo section, R1…R83), Contributing (the firewall paragraph; "83 numbered rules"), how-to ch.11 (the day-one section + the proposed-rules heading now says "ratified as R74–R80"). Forward links registered as PENDING: `Reference-index.md` (task 4), `Archive-index.md` (task 6). Verify: `wiki_render --selftest` OK; `doc_links` 0 broken / 2 pending. +Commit `5d10a0d12`. -## 🛑 SESSION CHECKPOINT — Tasks 0–3 ✓; NEXT = task 4 (wiki: Reference-index + Where-the-project-goes-next + Contributing conduct + Home/Sidebar; xHigh — prompt Drew to drop back, R27; then the P6 rules check) +### 2026-09-07 — Task 4 — Wiki: the Reference index, the direction page, Contributing's conduct section, Home + sidebar (xHigh) +`Reference-index.md`: 47 rows in seven groups (environment and verification; formats and addresses; the knowledge base; verification +and progress (generated); releases and outreach; the record; Gen3 inputs) + four directory rows (`gcc-2.7.2-map/`, `badges/`, +`outreach/`, `history/`) — each with what it is, how to read it (the 3.5 MB / 560 KB warnings), who writes it; the page states it is the +only sanctioned wiki→`docs/` link site and that the two index pages are exempt. `Where-the-project-goes-next.md`: Phase 34 (the +probe-gated flip, the outward actions in the maintainer's words, aftercare, v2.0.0), Gen3 (the floor, the bar — sotn's guide + the four +rules —, the census table with the ONE pin count and its command, the invariant, the order of work, the levers, the conduct rules, +shiftability scoped, the parked ideas), and the day-one-kit pointer in prose. Contributing gained "AI use — conduct" (the five rules of +gen3-standards §3, linking the direction page). `Home.md` rewritten: the source-of-truth statement + sections Using / Working conventions +/ Reference / How to AI-decomp / Where next / History; `_Sidebar.md` gained Reference, Where-next and History (Archive index, pending). +`gen3-handoff.md` §8: "R1–R73 plus candidates" → R1–R83; the last 44,243 → 43,925. `doc_links_pending.txt`: the Reference-index line +removed (1 entry left: Archive-index, task 6). **Coverage check** (a Python one-liner over `doc_links`' LINK_RE — the task-7 checker's +prototype): 132 `docs/` files outside wiki/how-to; index ∪ README covers 66; the 66 uncovered are EXACTLY the task-6 leavers (33 root +sunset + 4 integrate + 6 frozen + 12 per-binary snapshots + 11 tool-designs + 2 distill + 4 history files with spaces in their names) +plus one KEEP file the first draft omitted — `generic-decomp-package.md`, added as a row (how-to 00/06 link it). Verify: +`wiki_render --selftest` 12/12; `doc_links` 0 broken / 1 pending. + +**P6 rules check — re-read complete** (CLAUDE.md's Mandatory Behavior + fail-safes, DIGEST §3 R1–R83): one task at a time, one commit +per task after this log; never overwrite blind (H5 — every banner was added, none replaced); never commit ROM content (the two +`fable_cd4` listings leave at task 8, before any push Drew makes of this phase would matter — they are already in the pushed history); +never `git clean -x`; a match is byte-for-byte (no build input has changed). Continuing with task 5. + +## 🛑 SESSION CHECKPOINT — Tasks 0–4 ✓ (P6 check done); NEXT = task 5 (integrate the six documents into the wiki; xHigh) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase31/32/33.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest (R1–R83), then -WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking tasks 0–3 completed and task 4 in progress. +WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking tasks 0–4 completed and task 5 in progress. ### 1. Where we are **Phase 33.5** (sub-phase; v1.32.0 → v1.32.1), gate 1 approved 2026-09-07 by Drew in plan mode at Max; effort follows the plan's -column (Max for tasks 3, 10, 12, 15 — Drew toggled Max for task 3; xHigh otherwise; prompt at each transition, R27). Baseline HEAD -`80d45b29b`; task 0 = `39d524991`; task 1 = `a0cf302e5`; task 2 = `d06923a06`; task 3 = the commit after it (the two wiki pages). -Nothing under `src/`, `config/` or the `Makefile` has changed; the fleet is 218/218 at the Phase-33 close. `docs/doc_links_pending.txt` -carries TWO entries (`docs/wiki/Reference-index.md` → task 4; `docs/wiki/Archive-index.md` → task 6) — remove each when its page lands. -Wiki pages now: the 12 original + `Docs-and-scratch-conventions.md` + `The-ROM-firewall.md`; `_Sidebar.md` has a "Working conventions" -section between "Using the project" and "How to AI-decomp". +column (Max for tasks 10, 12, 15 — prompt at each transition, R27; xHigh now). Baseline HEAD `80d45b29b`; task 0 = `39d524991`; +task 1 = `a0cf302e5`; task 2 = `d06923a06`; task 3 = `5d10a0d12`; task 4 = the commit after it. Nothing under `src/`, `config/` or the +`Makefile` has changed; the fleet is 218/218 at the Phase-33 close. `docs/doc_links_pending.txt` carries ONE entry +(`docs/wiki/Archive-index.md` → task 6). Wiki pages now: 12 original + Docs-and-scratch-conventions, The-ROM-firewall, Reference-index, +Where-the-project-goes-next (16 + the 13 how-to chapters); the sidebar has sections Using / Working conventions / Reference / How to +AI-decomp / Where next / History (Archive index, pending). The coverage one-liner in the task-4 log entry is the prototype of task 7's +check: the 66 files it lists as uncovered are the task-6 leavers. -### 2. What NEXT does (task 4, xHigh) — exact steps -1. `docs/wiki/Reference-index.md` — list-driven from ledger L1 (this file): one row per KEEP-LIVE / KEEP-GEN file, grouped: environment - (`SETUP.md`, `effort-map.md`, `verification.md`, `second-oracle.md`, `doc_links_pending.txt`) · formats and addresses (`formats.md`, - `memory-map.md`, `idxtab-map.md`, `debug-menu-list.txt`, `disc-completeness.md`, `disc-ledger.md`) · the knowledge base - (`matching-cookbook.md`, `cookbook-index.md`, `gcc-2.7.2-map/` as a directory row, `wave-playbook.md`, `accelerators.md`, - `decision-log.md`, `psyq-sample-idioms.md`, `psyq-worklist.md`, `concurrency-design.md`, `tooling-audit.md`) · verification and - progress (`progress.md/.json/.fleet.md`, `difficulty.md`, `duplicates.md/.cross.md`, `backlog.md`, `badges/` directory row, - `story-timeline.md/.svg`, `commit-map.tsv`, `public-flip-runbook.md`) · releases and outreach (`permuter-ils.md`, - `matching-drafter-pipeline.md`, `gen2-mips-matching-model.md`, `community-matching-model-plan.md`, `decompme-preset.md`, - `outreach/` directory row) · the record (`story.md`, `retrospective.md`, `phase34-seed.md`, `history/` directory row) · Gen3 inputs - (`gen3-standards.md`, `gen3-handoff.md`, `gen3-parking-lot.md`, `actor-struct.md`, `struct-core-pivot.md`, `wall-taxonomy.md`). - Columns: file (a relative link `../`) · what it is · how to read it (grep by §, the size warning for the 3.5 MB cookbook and - the 566 KB index) · generated by (tool + target) or "authored". The INTEGRATE→SUNSET four (`hand-matching-process`, `wave-metrics`, - `portable-decomp-workflow`, `hindsight-study`) are NOT rows — they leave in task 6; the 6 frozen snapshots are NOT rows. The - directory rows (`gcc-2.7.2-map/`, `badges/`, `outreach/`, `history/`) are prefixes for the task-7 allow-list. State at the top that - this page is the only sanctioned place for wiki→`docs/` links and that the index pages are exempt from that rule. -2. `docs/wiki/Where-the-project-goes-next.md` — Phase 34 (from `docs/phase34-seed.md` §1: the daily probe, the flip, the outward - actions, aftercare, v2.0.0 = Gen2 exit; the Support ticket as the gate) and Gen3 (from `gen3-standards.md` §1–§3 in summary — - accuracy met, the readability bar, the AI-use conduct — and `gen3-handoff.md`: the census with the ONE pin count 43,925 + its - command + date, the invariant "every Gen3 edit is gated like a match", the levers table condensed, the order of work - pins → macro bodies → struct unification → names → formatting, the parked ideas). Link both docs via the Reference index row form. -3. `Contributing-and-the-no-ROM-policy.md` — a new section "AI use — conduct" after "What a useful contribution looks like": the five - rules of `gen3-standards.md` §3 (the byte gate is the only claim of success; names/types evidence-based never guessed — unnamed - beats wrong; outward text is written by a person, the target project's AI policy read first; no automated traffic against - community infrastructure; agents assist, a person owns), linking the direction page. -4. `Home.md` rewritten: the lead as is; sections "Using the project" (table as is + the two conventions rows), "Working conventions", - "Reference" (one paragraph → the index), "How to AI-decomp" (as is), "Where the project goes next" (one paragraph), "History" - (story / timeline / retrospective / the Archive index — pending) ; the "records behind the wiki" list becomes the index pointer. - `_Sidebar.md`: add "Reference" (Reference index), "Where the project goes next", "History" (Story and timeline → via index? — no: - sidebar rows must be wiki pages; History = Archive index (pending) + the direction page); keep the how-to list. -5. Remove the `Reference-index.md` line from `docs/doc_links_pending.txt`. Verify: `wiki_render --selftest`; `doc_links` 0 broken / - 1 pending; the list-driven check — every L1 KEEP path appears as a link target or under a directory row (a small Python one-liner - over the page with `doc_links.LINK_RE`). Log + checkpoint; commit by explicit path. **Then the P6 rules check** (re-read CLAUDE.md's - Mandatory Behavior + the fail-safes; state "Rules check — re-read complete. Continuing with task 5"). Task 5 is xHigh. +### 2. What NEXT does (task 5, xHigh) — exact steps +Fold the six INTEGRATE documents into the wiki/how-to and re-point every referrer, so task 6 can move the four that leave: +1. `docs/hand-matching-process.md` (32 KB, P17 Ghidra-driven hand-matching guide) → read it; fold what the wiki lacks into + `The-matching-workflow.md` (a "Matching by hand" section: the Ghidra-driven loop, the differ discipline, the register-pin toolkit + pointer §17, when to hand the near-miss to the permuter) and, for the transferable parts, how-to `07-compiler-source.md`; record + the "unique claims folded" list in this log. +2. `docs/wave-metrics.md` (20 KB, one row per crack wave) → the measurement source of how-to `09-economics.md` and `05-cards-lanes- + waves.md`: fold the per-wave table's headline numbers (agents, tokens, wall, banked per wave; the 76%→77%→100% bank-rate arc) where + the chapters cite economics without a table; move its two `.run/` prose citations FIRST: line 62's "Use `.run/s37w.js`'s execution + block for every future wave" → copy the execution block itself into `docs/wave-playbook.md` (a short fenced block in §5 Draft) and + line 153's `.run/s6f_gate.py` assertion → name it as archived. Then the file can leave. +3. `docs/portable-decomp-workflow.md` (14 KB, §0–§11) → diff each § against the 13 chapters (this is the breadth-shaped part — three + read-only agents may take §0–§4 / §5–§8 / §9–§11); fold any unique claim into ch.03 (bootstrap order), ch.05 (lanes/cards), ch.09 + (economics), ch.04 (refuse work the pipeline cannot bank — R43/R45 framing); re-point how-to `00-README.md:59`'s link to the chapters. +4. `docs/generic-decomp-package.md` (KEEP) → attribute it in ch.03 and ch.06 where its claims are used without citation (the twin BAND + 3.4× widening; "rank by work, not distance"; the widening review); it stays and is in the Reference index. +5. `docs/hindsight-study.md` → re-point how-to `00-README.md:60`, `02-byte-gate.md:16`, `docs/retrospective.md:10–11` and + `docs/gen3-handoff.md` §8 item 1 (which cites its §2.1: "measure the shape before choosing") to the chapters/the retrospective; + add to ch.03 Phase 2b the "mine the permuter's failures" paragraph (its §7 → `tools/autopsy.py` + `residual_class.py`; the + docstrings of both tools already cite ch.03 Phase 2b) and to `docs/permuter-ils.md` a one-line pointer to the autopsy. +6. `docs/gen3-standards.md` / `gen3-handoff.md` (KEEP) → already summarised in the direction page (task 4); nothing more. +7. ch.04 gains a citation of `docs/tooling-audit.md` as its evidence base (one sentence + a Reference-index-form link). +8. Referrer census for the four leavers (the D3 command) → must be 0 outside PhaseEnds/logs before task 6. Verify `doc_links` 0 broken; + log the folded-claims checklist; commit by explicit path. Task 6 is xHigh. ### 3. Standing facts for every task of this phase - One commit per task, after this file's log line (R8/R42 form); commit by explicit path; no trailers (R5); Drew pushes (R6).