From 7eac53da5173f10be88c366857124b73becedce2 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 30 Sep 2026 20:07:04 -0600 Subject: [PATCH] T10: publish P38 census: story 11 Gen3 row, kit corpus copy, log and summary --- decomp-architect/corpus/record/docs/story.md | 93 ++++++++++++++++++-- docs/story.md | 2 +- phase-ends/current/logs/T10.md | 32 +++++++ phase-ends/current/tasks/INDEX.md | 1 + phase-ends/current/tasks/T10.md | 28 ++++++ 5 files changed, 150 insertions(+), 6 deletions(-) create mode 100644 phase-ends/current/logs/T10.md create mode 100644 phase-ends/current/tasks/T10.md diff --git a/decomp-architect/corpus/record/docs/story.md b/decomp-architect/corpus/record/docs/story.md index db5e03649c..d541472ed1 100644 --- a/decomp-architect/corpus/record/docs/story.md +++ b/decomp-architect/corpus/record/docs/story.md @@ -170,11 +170,13 @@ revision, old hashes in historical documents replaced by inert tokens and mapped would have corrupted history), proven pair by pair, and force-pushed. This document, the retrospective, the wiki and the tooling releases are the last block; the visibility flip waits for GitHub to purge the old objects. -## 10. After 100 %: making the code say what it means (Gen3 — Phases 35, 36, 37, 2026-09-08 →) +## 10. After 100 %: making the code say what it means (Gen3 — Phases 35, 36, 37, 38, 2026-09-08 →) -*From `PhaseEnd_Phase35.md`, `PhaseEnd_Phase36.md`, `phase-ends/CURRENT_PHASE.md` (Phase 37), decision log "P35", "P36 S99/S101/S104/S105", -"P37 S106", `docs/levers.md`, `docs/readability.md`. This chapter is advanced at the end of every session (the owner's rule, 2026-09-12); the -numbers are the two Gen3 series'.* +*From `PhaseEnd_Phase35.md`, `PhaseEnd_Phase36.md`, `PhaseEnd_Phase37.md` and its log (`phase-ends/logs/Phase37.md`), decision log "P35", +"P36 S99/S101/S104/S105", "P37 S106", `docs/levers.md`, `docs/readability.md`; for the migration and Phase 38, the Phase 38 plan's change +lines, its task summaries (T1, T5, T5.1, T6, T7) and logs, and git. This chapter was advanced at the end of every session (the owner's rule, +2026-09-12) until the migration of 2026-09-29 dropped that duty; since 2026-09-30 it is advanced by every phase's closing work (rule R121). +The numbers are the two Gen3 series'.* A hundred percent is a byte statement, not a code statement. The tree that rebuilt 218 binaries byte for byte was still the tree a machine had drafted at speed: every shared engine function lived as a macro instantiated per level, tens of thousands of register pins and inline-asm @@ -269,6 +271,87 @@ compiler from stealing an instruction into a branch's delay slot. The levers cou hundred and thirty. One mistake of the day was mine: a restore run against a judge that had not yet finished stopping, which split a batch's bookkeeping — erased cleanly, and the rule written down: wait for the exit. +That night the declaration layer was called at its floor and the phase's fourth task closed: about 107,000 declarations made to say +what their definitions say, 1,796 lying declarations left over 76 callees (98,648 over 1,609 at the phase's start), 5,262 call sites +counted apart as the original's own calling convention and 418 as cross-binary calls, 133 of 149 alias-defined functions back under +their names, ten of twelve parked signatures landed, and the levers at 3,730. Of the 1,796, 1,537 were judged a floor only the types +could lower: a declaration cannot name a type its file cannot see. The checkpoint said the next task — one definition per struct type — +would open in a fresh session. It did not open for seventeen days. The record says only that the phase sat at that boundary with a clean +tree; it does not say why. On 2026-09-29 the owner re-scoped Phase 37 to the work already done (its tasks T0–T4) and carried the six +unstarted tasks to the next phase word for word, so that the repository could change its working method between phases rather than in +the middle of one. Phase 37 closed as v2.3.0 with the 218-binary gate green, and no tag. + +**The migration (2026-09-29, recorded as Phase 37.5).** The new method, Project Architect 3.0, replaced the long session-start reading with +a written plan per phase, tasks that each run in a fresh context and end in a log and a short summary, and a one-page card of standing +facts. One commit of 777 files installed it and moved the old operating documents — the setup reference, the 3.5-megabyte matching +cookbook and its index, the effort map — into `docs/retired/`, where nothing loads them. Four things broke, and each was found only when +something used it. The project's own tools still read the retired documents: the health chain's tool census took its tool table from the +old setup file, a link checker and the wiki renderer pointed at it, and 36 links in the wiki, the README and the how-to pages led nowhere; +the next phase's first task found four red health items with this one cause and repaired them as one defect. The new launcher, which +sorts the closing reports by phase number, crashed: the half-phase from the first week is filed as `3_5`, with an underscore, beside +`33.5` and `37.5` with dots, and the sort ended up comparing a number with a string. The migration had normalised none of the ids; the +router could not build its starting state after the next plan was approved, and the fix — a sort that splits every id the same way — was +made outside the task loop and reported upstream. The migration's ignore file gained a bare `.run/` line, which overrides the older lines +that let chosen per-phase evidence under `.run/` be committed; from then on a census snapshot could be committed only by forcing it +(the next phase met this as a commit helper's refusal). And the duty to advance this chapter and the retrospective at the end of every +session had lived in the old method's memory and checkpoint routine; the migration carried it nowhere, and the next phase ran to its +milestone with neither document touched. It came back the day after as rule R121, which hands it to every phase's closing work; this +part of the chapter was written under it. + +**Phase 38 (2026-09-29 → 09-30) took up the carried task: one canonical definition per struct type.** Its first task repaired the +migration (above), proved the gate green, re-took the byte oracle's baseline and measured the ground: 7,255 struct definitions — 1,179 +in the canonical header, 4,085 at file scope in `.c` files, 1,904 inside functions, 87 in other headers — in 206 classes of duplicates +over 2,823 names, 40 "variant camps" (one name, several layouts), 141 canonical names nothing used, 26 legacy padding names, and the +1,796 lying declarations. The second built the instrument the milestone would be read by, a `--check-structs` mode of the census, and +its known-true case was that the starting tree fails with exactly those counts; it did, adding 1,106 declarations the engine had kept +because a type was not visible. The third and fourth designed the three largest types — the entity, a 0x10C-byte block and the player +block — into a new canonical header and folded their local copies onto it, every batch judged on the bytes. + +The fifth task, the long tail, did the mechanical part in one night — 1,308 types lifted into the canonical header, the padding names to +0, the 40 variant camps to 0 by renaming each divergent local copy, 314 dead definitions deleted — and then stopped and asked, because +three of the gate's counts could not reach zero without breaking the plan's own words. The duplicate count included every opaque layout +twin: two placeholder-named structs with the same shape and no evidence that they are the same thing. Reaching 0 would have meant +folding, for one example, 293 opaque eight-byte structs into one — "one name per layout", which the project's naming law forbids — while +the same task said such twins stay apart and are listed. The outside-the-canonical-files count included 1,774 structs declared together +with a variable inside one function (a stack frame, local by nature), and the lifting tool could reach only one kind of file scope. The +dead-name count ignored uses inside the canonical header itself, so 114 of its 134 "dead" names were members of live types; deleting one +single-letter type had broken a level. The census defined the milestone's zero as impossible, and the plan had asked for it anyway: the +second task's known-true case had proved that the gate fails on the starting tree, and no one had asked whether it could pass on the +intended one. + +The question went to the plan's critic, which reopened the task as T5.1 with the gate reading fixed, and the owner set that reading on +09-30: a duplicate class still counts if its names are meaningful, or if it is a placeholder class not listed in a new record, +`docs/struct-twins.md`, with per-class evidence from the struct map; a stack frame declared with its variable is reported, not counted; +a canonical type used only inside another canonical type is alive. The milestone's own line stayed as written. The reasoning between the +question and the answer was spoken rather than written: the plan keeps the decision as one change line, the fifth task's log keeps the +conflict, and the discussion record opened that day is an empty template. What is said here is what those support. + +T5.1 took two attempts (the first handed off without leaving its progress file on disk) and ten coder runs, one of them red and +reverted, and three things that had looked right failed on the way. The census counted a struct's tag and its typedef as two names, +because C keeps them in two namespaces; the fifth task had split nine such pairs to satisfy it, and T5.1 taught the census that one +definition's tag and typedef are one name. The layout hash that defines "the same layout" ignored alignment attributes, and the gate +said so twice: folding a class whose members differed only by `packed` or `aligned` turned 142 of the 218 binaries red, and folding a +packed four-byte struct onto its natural twin turned 148 red; the hash now carries those attributes, and two definitions that differ only +in them are different types. And a fold by layout alone was not safe even then: a member name the canonical type spells at another +offset compiles without a word from the compiler and reads the wrong field — a trial fold of that kind failed 142 binaries, and only the +byte gate saw it. At the close every definition outside the canonical files had moved in (1,684 to 0), every meaningful-name class was +folded, 942 placeholder names whose uses showed them to be one type were folded, and 53 placeholder classes were listed — 44 with +evidence that they are separate, 9 with no use-site evidence either way, among them the shapes of Sony's `RECT`, `SVECTOR` and `DVECTOR`. +Six layouts took Sony's own SDK names, three unused names were kept with their reason, and the 0x10C block's split was confirmed, with no +unions. + +The sixth task put the declarations on the canonical types: nine redraw cycles, each through the gate, took the 1,106 hidden-type +declarations to 0 and the "conflicting types" from 160 to 18 — all 18 stale rows of the engine's ledger whose source is already right, +because the engine writes no new row for a unit it finds nothing to fix in. The shared-body conflicts went from 44 (the plan's 45 had +counted a header line) to 40, each with its diagnostic. The seventh ran the gate on a fresh census on 09-30: exit 0, and 218 of 218 +binaries identical. From the phase's start to its end, struct definitions went from 7,255 to 1,650; duplicate classes from 206 to 53, all +of them listed twins; file-scope definitions in `.c` files from 4,085 to 0; variant camps from 40 to 0; dead canonical names from 141 to 0 +(three kept with their cause); legacy padding names from 26 to 0; hidden-type declarations from about 1,106 to 0. Lying declarations fell +from 1,796 over 76 callees to 229 over 24 — the census's `decls.lying` field at both ends, which the new gate prints as +`types_floor_lying`; it is not the 98,648-to-1,796 series quoted above for Phase 37, nor the plan's 1,537 "types floor", and it is a +different count from the hidden-type declarations, which are read from the engine's ledger. The levers stayed at 3,730: the phase moved no +compiler pins, and none was asked of it. + ## 11. By the numbers | | | @@ -280,7 +363,7 @@ batch's bookkeeping — erased cleanly, and the rule written down: wait for the | Instructions | 13,492,113 / 13,492,113 (100.0%); distinct code 5,820,205 / 5,820,205; main game code 45,150 / 45,150 | | Knowledge base | the matching cookbook (§1–§501 and sub-sections, 3.5 MB), the gcc-2.7.2 codegen map, 79 decision-log entries, 73 numbered rules (+ the constitution's 25) | | Tooling | ≈235 Python tools, 25 shell tools, 12 Ghidra scripts — the extractor, the byte gate, the dedup engine, the family engine, the atlas, the lanes, the rewrite package | -| **After 100 % (Gen3, snapshot 2026-09-12)** | dedup: 0 macro bodies, 3,175 shared headers, one source per unique function (P35) · levers: 53,234 → 4,010 marked sites, +6,717 inline GTE statements counted from P37 (`docs/levers.md`) · struct debt: 503,016 raw dereferences in 69,497 functions, 7,255 struct definitions over 527 layouts, 98,648 lying declarations (`docs/readability.md`) · the chart's lower panel | +| **After 100 % (Gen3, snapshot at the P38 close)** | dedup: 0 macro bodies, 3,175 shared headers, one source per unique function (P35) · levers: 53,234 → 3,729 marked sites, +6,717 inline GTE statements counted from P37 (`docs/levers.md`) · struct debt (the P38 type census): 502,983 raw dereferences in 69,492 functions, 1,650 struct definitions (7,255 at the P38 start) over 523 layouts, 53 duplicate classes all listed with a cause in `docs/struct-twins.md`, 229 lying declarations over 24 callees (1,796 at the P37 T4 close) · the chart's lower panel | *Every number above is generated or counted from the repository; the timeline behind the chart is [`docs/story-timeline.md`](story-timeline.md).* diff --git a/docs/story.md b/docs/story.md index 5453e42685..d541472ed1 100644 --- a/docs/story.md +++ b/docs/story.md @@ -363,7 +363,7 @@ compiler pins, and none was asked of it. | Instructions | 13,492,113 / 13,492,113 (100.0%); distinct code 5,820,205 / 5,820,205; main game code 45,150 / 45,150 | | Knowledge base | the matching cookbook (§1–§501 and sub-sections, 3.5 MB), the gcc-2.7.2 codegen map, 79 decision-log entries, 73 numbered rules (+ the constitution's 25) | | Tooling | ≈235 Python tools, 25 shell tools, 12 Ghidra scripts — the extractor, the byte gate, the dedup engine, the family engine, the atlas, the lanes, the rewrite package | -| **After 100 % (Gen3, snapshot 2026-09-12)** | dedup: 0 macro bodies, 3,175 shared headers, one source per unique function (P35) · levers: 53,234 → 4,010 marked sites, +6,717 inline GTE statements counted from P37 (`docs/levers.md`) · struct debt: 503,016 raw dereferences in 69,497 functions, 7,255 struct definitions over 527 layouts, 98,648 lying declarations (`docs/readability.md`) · the chart's lower panel | +| **After 100 % (Gen3, snapshot at the P38 close)** | dedup: 0 macro bodies, 3,175 shared headers, one source per unique function (P35) · levers: 53,234 → 3,729 marked sites, +6,717 inline GTE statements counted from P37 (`docs/levers.md`) · struct debt (the P38 type census): 502,983 raw dereferences in 69,492 functions, 1,650 struct definitions (7,255 at the P38 start) over 523 layouts, 53 duplicate classes all listed with a cause in `docs/struct-twins.md`, 229 lying declarations over 24 callees (1,796 at the P37 T4 close) · the chart's lower panel | *Every number above is generated or counted from the repository; the timeline behind the chart is [`docs/story-timeline.md`](story-timeline.md).* diff --git a/phase-ends/current/logs/T10.md b/phase-ends/current/logs/T10.md new file mode 100644 index 0000000000..a75f5bef84 --- /dev/null +++ b/phase-ends/current/logs/T10.md @@ -0,0 +1,32 @@ +# T10 log — publish the P38 type census + +## Timeline +- Start: card slice, plan Context/Interfaces/Cookbook/Research, T10 entry, Changes lines 112/114; summaries T7, T9. +- Hook denied reading tool sources (tools/progress.py, tools/timeline.py) → all tool edits delegated to coder. +- README.md:32 Types paragraph is generated (`tools/progress.py --readme` from docs/progress.json); readability-progress.tsv already carries `struct_defs` column; `.gitignore:595` `.run/` blanket ignore → census files need `git add -f`. +- Coder T10.c1 briefed (census path rule, README Types counts, readability row, timeline struct-defs column, track census files, tools-health). Returned blocked: changes committed (67c6710fb1, 14e100d723, 6980aed89f) but tools-health FAIL `tool_census --check: 19 gap(s)` (3 from this task's tool edits, 16 pre-existing: owed `make kit-corpus` from P37 + 2 missing dictionary rows). Also lever census re-run moved Levers 3,730 → 3,729. +- Expert: docs/story.md §11 Gen3 row refreshed (P38 census figures; levers 3,729). +- Coder T10.c2 briefed: kit-corpus refresh + 2 dictionary rows (head_types, struct_twins), find vanished lever, commit regenerated tree, tools-health green. Returned done, c70f76f891. +- Expert: after the story.md edit, `make kit-corpus` (run.sh log `kitcorpus`) → 407 copies + 30 pointers; `tool_census --check` → 0 gaps OK. + +## Coder briefs +- T10.c1: progress.py census path (newest .run/P/census), README Types counts, readability P38 row, timeline struct_defs column, force-add .run/P38/census/type_census.{json,txt}, tools-health. Log logs/T10.c1.md. +- T10.c2: make kit-corpus + dictionary rows, lever 3,730→3,729 evidence, regenerate + commit, tools-health. Log logs/T10.c2.md. + +## Findings +- Lever 3,730 → 3,729: commit 83b75954b9 (T6.c2 share_body) replaced func_801759D8's body in src/ov_SC07_006/ov_SC07_006_jr_80171B4C.c with the shared header, dropping one `!FAKE: barrier memory — NEEDED` asm; shared body has none; R22 stayed 218/218 at T7. Only FAKE line removed between 465d915bd and HEAD (git log -S). +- readability_progress.py also hard-coded P37; same newest-census rule applied (c1). +- pad / parse-error counts are not in type_census.json; progress.py takes them from `type_census.check_structs_inputs()` (c1). +- commit_task.sh `git add` refuses ignored paths: force-added files must be staged with `git add -f` first. +- timeline.py --check counts commits per date: goes stale after any later same-day commit. + +## Hypotheses rejected +- none. + +## Retrievers +- none; no report ids. + +## Verification +- make tools-health (c2, run.sh tools-health) → exit 0, `tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` +- After expert story edit: make kit-corpus + tool_census --check → 0 gaps OK. +- doc_links docs/story.md → OK. diff --git a/phase-ends/current/tasks/INDEX.md b/phase-ends/current/tasks/INDEX.md index 735fd82a8f..b8736f77c1 100644 --- a/phase-ends/current/tasks/INDEX.md +++ b/phase-ends/current/tasks/INDEX.md @@ -10,3 +10,4 @@ T6 | done | declaration layer on the canonical types | - | tasks/T6.md | logs/T6 T7 | done | milestone gate and record | - | tasks/T7.md | logs/T7.md | - T8 | done | docs/formats.md: LIST.CD verified on US, runtime reader, repack note, LZSS ring + terminator rules | - | tasks/T8.md | logs/T8.md | - T9 | review | story §10 + retrospective §7 for P37 T5-close, PA3 migration and P38 | - | tasks/T9.md | logs/T9.md | - +T10 | done | publish the P38 type census in README and timeline | - | tasks/T10.md | logs/T10.md | - diff --git a/phase-ends/current/tasks/T10.md b/phase-ends/current/tasks/T10.md new file mode 100644 index 0000000000..ff1bb73c4d --- /dev/null +++ b/phase-ends/current/tasks/T10.md @@ -0,0 +1,28 @@ +# T10 — publish the P38 type census in README and timeline + +Status: done | expert: expert-opus55 | ctx-at-completion: n/a | commit: see COMMIT | coder runs: 2 (T10.c1 blocked on tools-health, T10.c2 done) +Done: progress.py and readability_progress.py read the newest `.run/P/census/type_census.json` (no phase literal); README Types paragraph shows the P38 close counts; readability series has a P38 row; timeline has a struct-definitions column; .run/P38/census type_census.{json,txt} tracked; story §11 Gen3 row refreshed; kit corpus refreshed; tools-health OK. +Files: +- tools/progress.py, tools/readability_progress.py — newest-census rule; Types paragraph counts (c1) +- tools/timeline.py — struct_defs column (c1) +- README.md, docs/progress.json, docs/progress.fleet.md, docs/readability.md, docs/readability-progress.tsv, docs/story-timeline.{md,svg} — generated +- .run/P38/census/type_census.{json,txt} — force-added; .run/P35, .run/P36 census tracked files — tools-health regenerated +- config/tool_dictionary.tsv — rows for head_types, struct_twins (c2) +- decomp-architect/corpus/**, decomp-architect/tools/MANIFEST.md, docs/tool-index.md — `make kit-corpus` (owed from P37) +- docs/story.md — §11 Gen3 row: P38 census figures (1,650 definitions, 53 listed twins, 229 lying declarations, levers 3,729) +Decisions: +- Kit corpus refresh (owed from P37) done here: tools-health cannot be green without it. +Deviations: +- Levers read 3,729, not the brief's 3,730: the lever census re-run by tools-health counts one fewer `!FAKE` barrier, removed by 83b75954b9 (T6.c2 share_body on func_801759D8, ov_SC07_006); generated count published, nothing hand-typed. +- Census files staged with `git add -f` before commit_task.sh (it refuses ignored paths). +Findings: +- README Types: 1,650 struct definitions (canon 1,021, include/ 326, block-scope 600, fn headers 29, file-scope 0) over 1,024 names / 523 layouts; 53 duplicate classes over 496 names, all listed in docs/struct-twins.md, 0 gating; 0 variants; 0 dead (3 kept); 0 pad; 0 parse-error; 229 lying declarations over 24 callees. +- T6.c2's share_body dropped a NEEDED-marked barrier with bytes still identical (R22 218/218 at T7): the NEEDED marker was context-dependent. +Gotchas: +- harness: commit_task.sh's `git add` refuses gitignored tracked paths under .run/; stage them with `git add -f` first. +- harness: `timeline.py --check` counts commits per date, so it reads stale after any later same-day commit; regenerate after the day's last commit. +- generalizable: any edit to a tool or doc mirrored in decomp-architect/corpus makes tools-health red until `make kit-corpus`; run it in the same task. +- harness: the tool-source read hook blocks an expert from reading tools it must change; the coder reads them. +Next task needs: timeline regenerate after the phase's last commit. +Verified: `make tools-health` → exit 0 `tools-health: OK — sigs fresh; corpus(+resident) + cdecl + binaries + report(lint+dedup) + cookbook-index all green.` (run.sh tools-health, T10.c2); after the story edit `make kit-corpus` + `tool_census.py --check` → 0 gaps OK +Full log: phase-ends/current/logs/T10.md