mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-08 17:51:46 -04:00
fix+docs: make every consumer aware of main's new TU layout (R36)
The split created two new TUs and a shared header; four consumers still described main's
game code as one file:
* tools/reconcile_slate.py — HARDCODED open('src/800.c'), so after the split it saw a
THIRD of main's typedefs while reporting success (silently-narrowed scope, R32).
Measured: 133 visible before the fix, 187 after, 0 lost. Now globs
corpus.src_files('main') + src/800_shared.h, so a future split is already handled.
* docs/wave-playbook.md 1c — still said spans B/C/D were NOT drawable and that drawing
one is an R45 violation. That is now false and would have STOPPED a future session
from drawing the very targets this work unlocked.
* cookbook §426 — its 'the remaining spans need src/800.c split' paragraph now records
that it was done the same session, and points at §431 for the method.
* config/dedup.us.yaml + src/shared/clearTbl40.h — both said dedup group I0 is
instantiated 'at both sites in src/800.c'; both sites are above 0x80035270 and are now
in src/800_c.c.
Byte-neutral: dedup.us.yaml parses, gate_main --assert-baseline BYTE-IDENTICAL.
SETUP.md gains a row describing the layout and the rule it implies: never hardcode
src/800.c, glob corpus.src_files('main').
This commit is contained in:
Drew T
@@ -49,7 +49,7 @@ version: 1
|
||||
groups:
|
||||
# I0 — Phase-11 intra-binary proof: the byte-identical pair func_80037004 / func_80037334
|
||||
# (9-ins byte-clear loop) matched ONCE in src/shared/clearTbl40.h, instantiated at both sites
|
||||
# in src/800.c. Clean rebuild of `main` stays 143dbb89 with the shared C and with INCLUDE_ASM
|
||||
# in src/800_c.c (both vrams are above the S72 split at 0x80035270). Clean rebuild of `main` stays 143dbb89 with the shared C and with INCLUDE_ASM
|
||||
# stubs (both byte-identical). Proves the source-level share machinery + the byte-honesty gate.
|
||||
- id: I0_clearTbl40
|
||||
tier: h_exact
|
||||
|
||||
@@ -753,6 +753,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/claude_wave_packs.py` · `tools/workflows/claude_wave_draft.js` · `tools/wave_judge.py` | **(Phase 31 S62 T4)** The Claude-subagent wave harness that ran the model-ladder probe: identical packs from `api_agent` (SYS + user_msg + law-1c warm start + gate feedback) → a Workflow of drafting agents per (arm, fn) with CLI equivalents of grep/read_file/match_one/submit, writing only under `.run/<wave>/<arm>/` → `wave_judge.py` gates each arm with `sweep_parallel` (banked = removed INCLUDE_ASM lines in `git diff`; tree reset between arms; `--union` leaves the cheapest winning drafts in the tree). `tools/workflows/claude_wave_distill.js` = the R16 distill (extract per byte-proven transcript → verify vs the cookbook). Routing rule from T4: ≤50 Sonnet, 51–120 Sonnet, >120 Opus, Opus on residue; haiku retired; DeepSeek optional at ~$0.03/fn. |
|
||||
| | `tools/t5_targets.py` · `tools/t5_cards.py` · `tools/wave_card_fuel.py` · `tools/t5_bank.sh` | **(Phase 31 S63 T5)** The wave *draw* and the wave *bank* halves around the T4 harness: `t5_targets.py --wave .run/t5x --n 48 [--residue .run/t5w] [--main-only]` draws a routed slate (open per `corpus.stubs`, ledger `.run/t5/drawn.json` keyed (binary,fn), ≤120 ins → sonnet / >120 → opus, residue → opus once, stratified by band, ≤4 per binary, one fn NAME per wave — the harness keys packs/drafts by bare name, so `claude_wave_packs`/`wave_judge` now REFUSE duplicates, R43/R48); then **`t5_cards.py --wave .run/t5x`** BUILDS the target's OWN card fuel (`tu_ref` = in-TU banked neighbours §194-E, `decl_prior` = fleet signatures §196) from `wave_card_fuel.py` — those helpers extracted VERBATIM out of `build_wave_atlas.py` (which parses argv at import and so was unusable as a library) so both consumers share ONE oracle (R33); measured 88% tu_ref / 98% decl_prior on a fresh K-class slate, against 0% correct fuel before the fix. Then `claude_wave_packs.py --cards <wave>/cards.json` → Workflow `tools/workflows/claude_wave_draft.js` `args={wave, targets}` (one agent per target, `model=target.arm`) → `tools/t5_bank.sh .run/t5x sonnet opus` (refuses if a lane/gate runs or src/ config/ is dirty; `wave_judge --union`; then a RECOVERY pass — every un-banked draft through `fix_tu_ret_decls`, which retypes a destination TU's `extern void` decl on a value-returning function and re-gates (wave t5a: 2/2 banked, both byte-exact bodies refused for that decl alone); then the R22 clean fleet sweep parsed `N passed, 0 failed of N`; commit `— N banked`). |
|
||||
| | `tools/jtbl_rodata_pads.py --derive` | **(Phase 31 S62 T3a, cookbook §303)** Module path of the §8e pads filter: the Makefile runs `--derive $(BINARY) --tu <tu>` for every `md_*` object **and for `main` (P31 S72)** — jump-table pads derived at build time from the retail island + the emission stream (trailing pads `0t1`, table-aware, const data passes through). No stored spec; an anchor miss fails the build with the offset. **S72:** `--derive` now serves main too — `_file0_vram` returns the code segment's `vram - start` (the PS-X EXE's 0x800 header), which makes both `raw[a - vram]` and `vram + <yaml offset>` correct for the EXE and leaves flat overlays byte-identical; `_splat_yaml` resolves main to `splat.us.exe.yaml`. |
|
||||
| | **main's TU layout (P31 S72)** | main's game code is no longer one file. `src/800.c` was split at the jtbl-span TU boundaries into **`src/800.c`** (vram 0x800123F0-0x8002B0B4, owns `.rodata` span A 0x80072A38-0x80072C70), **`src/800_b.c`** (0x8002B0B4-0x80035270, span B 0x80072E44-0x80073140) and **`src/800_c.c`** (0x80035270-0x8003A444, span C 0x800732A0-0x8007344C), plus **`src/800_shared.h`** for the 19 typedefs + 2 includes that cross. Reason: one code object contributes exactly ONE contiguous `.rodata` run, so each jump-table span needs its own object or every switch function in it double-emits its table (cookbook §426/§431). main's island is now a 7-piece sandwich and `ld_interleave` runs with `--order`, not `--front/--tail`. **Any tool that reads "main's source" must glob `corpus.src_files('main')`, never hardcode `src/800.c`** — `reconcile_slate` did, and saw 133 of 187 typedefs after the split. |
|
||||
| | `tools/main_diff_locate.py` | **(P31 S72, cookbook §426/§427)** Turns a RED main gate — whose whole output is two SHA1s — into a NAMED list of divergent symbols: byte-diffs `build/us/SLUS_007.26` against `extracted/retail/SLUS_007.26`, coalesces the differing bytes into runs, and attributes them PER BYTE to symbols via `build/us/SLUS_007.26.map` (file-offset mapping DERIVED from each output section's `load address`, not the 0x800 header constant). `--focus <fn>` prints the routing verdict: **BODY REJECT** (divergence confined to the function) vs **PLUMBING REJECT** (the function is byte-identical, everything differs elsewhere → the §376/§378 chain) vs **MIXED**. `--self-test <addr>` is the negative control: flips one byte at a known address and asserts the containing symbol is named, plus the identical-pair-reports-zero direction. `gate_main` calls it automatically and preserves the red image + map under `.run/gate_main_fail/` BEFORE the R40 baseline control rebuilds over them. |
|
||||
| | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. |
|
||||
| | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). |
|
||||
|
||||
+2
-2
@@ -1,9 +1,9 @@
|
||||
# Unmatched difficulty inventory (generated by tools/difficulty.py — harvest queue)
|
||||
|
||||
unmatched functions : 1009
|
||||
unmatched functions : 1004
|
||||
trivial (<=5 ins) : 215
|
||||
non-jtbl leaves : 663 (best harvest targets)
|
||||
jump-table funcs : 51 (deferred — need the rodata-island workflow, Task 2')
|
||||
jump-table funcs : 46 (deferred — need the rodata-island workflow, Task 2')
|
||||
|
||||
## Easiest 120 unmatched (score asc) — the work queue
|
||||
| score | name | nins | br | calls | jtbl | leaf |
|
||||
|
||||
@@ -33928,6 +33928,10 @@ ordered** (span A owners < span B owners < span C owners): the jtbl spans ARE th
|
||||
translation units' rodata, so splitting `src/800.c` at those two boundaries is both the fix and a
|
||||
recovery of the game's real TU structure.
|
||||
|
||||
**DONE, SAME SESSION — see §431 for the method.** `src/800.c` was split at `0x8002B0B4` and
|
||||
`0x80035270`, byte-identical with nothing banked, and **7 of the session's 14 main banks were span
|
||||
B/C functions that could not have banked the day before.**
|
||||
|
||||
## §427 ★★ — A HASH IS A CORRECTNESS ORACLE WITH ZERO DIAGNOSTIC CONTENT; PRESERVE THE RED ARTIFACT BEFORE ANYTHING REBUILDS OVER IT (P31 S72)
|
||||
|
||||
`[FAIL] got <sha> want <sha>` cannot distinguish *"your body is wrong"* from *"your body is perfect
|
||||
|
||||
@@ -4,21 +4,21 @@
|
||||
# cross-binary collapsible-byte leverage: docs/duplicates.cross.md.
|
||||
|
||||
# THREE progress metrics (all matter — see the labels):
|
||||
FLEET fn-count byte-ident: 362814 / 362954 = 99.96% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 13468535 / 13523865 = 99.6% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5797343 / 5851972 = 99.1% (90847/90929 unique fns; the DISTINCT-RE number)
|
||||
MAIN game-code weighted : 41187 / 79510 = 51.8% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
|
||||
FLEET fn-count byte-ident: 362819 / 362954 = 99.96% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 13468946 / 13523865 = 99.6% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5797754 / 5851972 = 99.1% (90847/90929 unique fns; the DISTINCT-RE number)
|
||||
MAIN game-code weighted : 41598 / 79510 = 52.3% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
|
||||
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 13427348 / 13444355 = 99.9%)
|
||||
|
||||
FLEET REAL substantive : 360644 (of which dedup-shared 255632 via 2220 groups / 255708 instances)
|
||||
FLEET REAL substantive : 360649 (of which dedup-shared 255632 via 2220 groups / 255708 instances)
|
||||
FLEET LINKED PsyQ objs : 959
|
||||
FLEET NON_MATCHING : 2 (0 in any default build — G4)
|
||||
FLEET INCLUDE_ASM stubs : 138
|
||||
FLEET INCLUDE_ASM stubs : 133
|
||||
FLEET matchable : 362954
|
||||
|
||||
| binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| main | 859 | 2 | 959 | 1861 | 1913 | 97.3% |
|
||||
| main | 864 | 2 | 959 | 1866 | 1913 | 97.5% |
|
||||
| resident | 138 | 0 | 0 | 140 | 145 | 96.6% |
|
||||
| md_MAIN_001 | 11 | 0 | 0 | 11 | 11 | 100.0% |
|
||||
| md_MAIN_003 | 43 | 0 | 0 | 43 | 47 | 91.5% |
|
||||
|
||||
+5
-5
File diff suppressed because one or more lines are too long
+11
-9
@@ -328,17 +328,19 @@ carved** — the image grows and 238 symbols shift, which reads as a codegen rej
|
||||
`config/splat.us.exe.yaml` currently carves ONE span:
|
||||
|
||||
```
|
||||
0x80072A38-0x80072C70 span A — LZSS + 11 game tables, 8 frontier owners CARVED, drawable
|
||||
0x80072E44-0x80073140 span B — 14 tables, ~10 owners (SaveLoadRoutine, func_8003388C) NOT drawable
|
||||
0x800732A0-0x8007344C span C — 8 tables, ~8 owners (StreamLoadStateMachine) NOT drawable
|
||||
0x80073494-0x80073514 span D — 4 tables NOT drawable
|
||||
0x80072A38-0x80072C70 span A — LZSS + 11 game tables -> src/800.c CARVED, drawable
|
||||
0x80072E44-0x80073140 span B — 14 tables -> src/800_b.c CARVED, drawable
|
||||
0x800732A0-0x8007344C span C — 8 tables -> src/800_c.c CARVED, drawable
|
||||
0x80073494-0x80073514 span D — 4 tables -> snd2, not game code
|
||||
```
|
||||
|
||||
Spans B–D each need their OWN code object (one object contributes exactly ONE contiguous `.rodata`
|
||||
run, and `800.o`'s is span A), i.e. `src/800.c` split at the span owners' address boundaries — which
|
||||
are disjoint and ordered, because the spans ARE the original TUs' rodata. Until that split lands,
|
||||
**drawing a span-B/C/D function is an R45 violation: the pipeline cannot bank it.** Census the class
|
||||
with the `jr $rN` (N != `ra`) detector — never `jr $ra`, which ends every function (§401).
|
||||
**All three game spans are carved as of S72** — `src/800.c` was split into three TUs at
|
||||
`0x8002B0B4` / `0x80035270` so each span gets its own code object (one object contributes exactly
|
||||
ONE contiguous `.rodata` run). 7 of the 14 main functions banked that session were span B/C, i.e.
|
||||
impossible the day before. **Every main jtbl function is now drawable**; the R45 refusal that used to
|
||||
apply here is gone. Census the class with the `jr $rN` (N != `ra`) detector — never `jr $ra`, which
|
||||
ends every function (§401). If a FOURTH span ever appears (a newly-matched switch whose table sits
|
||||
outside A/B/C), it needs its own object too: split again at that span's owner range (§431).
|
||||
|
||||
### 5b. GATE THE DIRECTORY, NEVER THE VERDICT LIST (P31 S72)
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/* src/shared/clearTbl40.h — shared body for the byte-identical pair func_80037004 / func_80037334
|
||||
* (Phase 11 dedup proof, group I0). The body is authored ONCE here as a macro and instantiated at
|
||||
* each member site in src/800.c, so the same bytes land at both vrams (0x80037004 + 0x80037334).
|
||||
* each member site in src/800_c.c (both vrams are above the P31 S72 TU split at 0x80035270), so the same bytes land at both vrams (0x80037004 + 0x80037334).
|
||||
* Registered in config/dedup.us.yaml; byte-honesty validated by tools/dedup_integrate.py.
|
||||
*
|
||||
* Clears 5 bytes of a table: D_80076251[i] = 0 for i = 0x40,0x30,0x20,0x10,0x00 (down-counting). */
|
||||
|
||||
@@ -219,7 +219,18 @@ def main():
|
||||
a = ap.parse_args()
|
||||
|
||||
slate = json.load(open(a.slate))
|
||||
tu_bodies = _bodies(open('src/800.c').read())
|
||||
# EVERY TU OF main, NOT src/800.c ALONE (P31 S72). main's game code used to be one file, so
|
||||
# a hardcoded `src/800.c` was the whole world; the jtbl-span split made it src/800.c +
|
||||
# src/800_b.c + src/800_c.c + src/800_shared.h, and this read would have silently seen a THIRD
|
||||
# of main's typedefs while reporting success — the silently-narrowed-scope shape (R32). The
|
||||
# glob is the project's own oracle for "which files make up a binary" (corpus.src_files), so a
|
||||
# future split is already handled; the shared header is added because typedefs MOVED there.
|
||||
tu_bodies = {}
|
||||
for _f in corpus.src_files('main') + [os.path.join(corpus.REPO, 'src', '800_shared.h')]:
|
||||
try:
|
||||
tu_bodies.update(_bodies(open(_f, errors='replace').read()))
|
||||
except OSError:
|
||||
pass
|
||||
refusals, fixed_total = [], []
|
||||
|
||||
for rnd in range(1, a.max_rounds + 1):
|
||||
|
||||
Reference in New Issue
Block a user