mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-06 09:06:04 -04:00
fix(phase-32): T1b (1) — jr_isolate_all keys a bodiless typedef struct Tag Alias; by the ALIAS (§497); ov_SC02_017 dry-run REFUSED -> CLEAN, no source rename
- _type_names returned the TAG for `typedef struct Rec801806C8_s Rec801806C8;`, so the typedef block and the tag's own packed struct definition collided under one key with different bodies and the R43 "CONFLICTING bodies — a rename is needed" refusal fired on legal C. Now keyed by the alias (_TYPEDEF_TAG_ALIAS); the `carried` set learns the alias; `typedef struct X X;` (alias == tag) keeps the old key so a second one still dedupes/refuses. Unit control on 7 block shapes PASS; ov_SC02_017 --only func_80186C64 --dry-run: 2 region files, no carve repoints. cookbook §497; SETUP row.
This commit is contained in:
Drew T
@@ -781,6 +781,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/dedup_propagate.py` — `_split_masked` memo | **(P31 S75)** `find_site` re-ran `cdecl._mask` (4 regex passes, one `re.S`) + two `splitlines()` over the ENTIRE concatenated source on EVERY call, though the mask is a pure function of the text and the caller loops over every function in the binary. Measured on `ov_SC01_005` (2.50 MB, 2,503 fns): 6.5 + 38.9 + 4.6 = **54 ms per call before any searching**. Memoized via `lru_cache(maxsize=8)` — callers already hold one text object per binary and CPython caches a str's hash, so a repeat lookup is a pointer compare. **2x on that loop (58.3 -> 33.0 ms/call), NC identical results on 120 addresses.** Honest scope: that loop is only ~2.4 min of a 30-min run; the profiler puts 43% of `--check-only` in `family_remap._alias_decl_for` (107 s / 1,312 calls), which is the real target and is NOT fixed here. |
|
||||
| | `tools/main_diff_locate.py` — `classify()` | **(P31 S75, cookbook §447)** The `TABLE REJECT` class was **unreachable for main** and demanded purity. (a) It summed bytes whose object string contains `(.rodata)`, but main's `section_order` is `[.rodata, .text, .data, .bss]` — its rodata sits BELOW `.text` and its jump tables live in `.data` objects, so the test could never fire on the one binary with the most jump-table functions left. Now matches `(.data)` OR `(.rodata)`. (b) It required `ro == outside`, so a few bytes of perturbed code dropped the verdict through to `PLUMBING REJECT` and its §376 declaration advice. Now **dominance-based** (≥60%), reporting the split and naming which part is the table problem and which the declaration problem. Measured on `SaveLoadRoutine` (1,165 ins, the §434 wall): body BYTE-IDENTICAL, 3,787 of 3,989 differing bytes (94.9%) in `.data` jump tables, 202 (5.1%) in `.text`, built image 4 bytes SHORT — verdict moved `PLUMBING REJECT` → `TABLE REJECT (MIXED)`. The §376 chain had been run on it twice and fixed nothing, because it addresses the 5%. NC over all five pre-existing verdict shapes: 5 of 6 unchanged. |
|
||||
| | **`tools/asm_in_c.py`** (NEW) | **(P31 S75, cookbook §448)** Finds every function that is **assembly posing as C** — a §265 file-scope `__asm__` body (class A) or a C function whose body is only asm statements (class B) — while correctly EXCLUDING the §3a cross-jump barrier, which is what `__asm__` means in 3,182 of the 4,224 sources. Measured: **199 functions, 154 of them GAME CODE, 171 in main**, largest `SaveLoadRoutine` (1,165 ins). These were in NO `progress.py` bucket, so main's REAL% was overstated (45.88% → **42.15%** once counted; `progress.py` gained a `VERBATIM __asm__ bodies` line that counts them byte-identical but NEVER as REAL). Design is the point: **three independent detectors that must agree**, disagreement reported as a defect (it caught `jtbl_*` being claimed as functions); **SDK-ness derived from the 14 shipped PsyQ archives via `nm`** (2,227 symbols), not a hand list (which had mis-classified `VectorNormalSS`/`SquareRoot12`/`OuterProduct12` as game code); coverage asserted; and **`--selftest`** with a known-true case of EVERY spelling — hand counts went 116→112→108→178→199 because the sources use both `".ent\tNAME\n"` and `".ent NAME\n"`, and a bare `".ent\t"` fragment yields a phantom function called `t`. Run `--selftest` before believing the number. |
|
||||
| | `tools/jr_isolate_all.py` — `typedef struct Tag Alias;` keyed by the ALIAS | **(P32 T1b, cookbook §497)** The carried-type dedupe keyed every block by the names its regexes found; for a bodiless `typedef struct Rec801806C8_s Rec801806C8;` that was the TAG, so the typedef block and the tag's own `struct Rec801806C8_s {...} __attribute__((packed, aligned(1)));` definition collided under one key with different bodies and the R43 "CONFLICTING bodies — a rename is needed" refusal fired on legal C (ov_SC02_017). `_type_names` now keys such a line by the alias (`_TYPEDEF_TAG_ALIAS`), and the `carried` set learns the alias too; `typedef struct X X;` (alias == tag) keeps the old key so a second one still dedupes/refuses. Unit control on 7 block shapes; the refusing overlay's dry-run is CLEAN (2 region files) with no source rename. |
|
||||
| | `tools/jr_isolate_all.py` — include-derived provided types | **(P32 T1a, cookbook §496)** The carried-type test consulted `_engine_types()` (engine_types.h + common.h) for EVERY TU, assuming each region `#include`s engine_core.h. Overlays do; the resident, the `md_*` modules and main's TUs include only `common.h`. A resident file-local typedef whose NAME engine_types.h also defines (`CdFileLoc`) was therefore NOT carried into the new regions ("the shared headers already define it") → `parse error before cdFileLocTable` in both region TUs, build rc 2 while the stale binary on disk read GREEN (R53). Now `_provided_types(header)` derives the provided set from the TU's own `#include` lines (engine_core.h ⇒ engine_types.h + common.h — engine_core's OWN typedefs live inside `DEFINE_func_*` macro bodies and are provided only where invoked; `common.h` ⇒ common.h only) and `_file_scope_decls(items, provided)` uses it at both decision points; `_engine_types()` kept for legacy callers. R39 controls: an overlay header yields exactly the legacy set (1,197 names); the resident header's set lacks `CdFileLoc`. Positive control: the resident 3-region split builds byte-identical (`8e17e02f…`). |
|
||||
| | `tools/jr_isolate_all.py` — boundary derivation | **(P31 S74, cookbook §441)** `_region_emit_start()`: the yaml offset for a region is derived from the region's **CONTENT** — min of item addresses and of every `.globl`/`.ent` its text names that resolves inside the object — and taken as `min(cut, emit)`, so a boundary can only move DOWN. Reason: **a §265 verbatim `__asm__` body is not one of `parse_overlay_c`'s four addressed-anchor forms, so it attaches to the NEXT anchor as PREAMBLE — and preamble is assumed byte-neutral when it emits bytes.** A cut at `func_800D0268` would have moved 0x168 bytes of three other functions into the new object while the yaml claimed the region started higher. Where no verbatim asm is in play it equals the cut, so every existing isolate is unchanged. Also: an item-less CLOSING region used to emit a duplicate `- [off, c, …]` line (the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing region look non-empty). |
|
||||
| | `tools/jtbl_rodata_pads.py --derive` | **(Phase 31 S62 T3a, cookbook §303)** Module path of the §8e pads filter: the Makefile runs `--derive $(BINARY) --tu <tu>` for every `md_*` object **and for `main` (P31 S72)** — jump-table pads derived at build time from the retail island + the emission stream (trailing pads `0t1`, table-aware, const data passes through). No stored spec; an anchor miss fails the build with the offset. **S72:** `--derive` now serves main too — `_file0_vram` returns the code segment's `vram - start` (the PS-X EXE's 0x800 header), which makes both `raw[a - vram]` and `vram + <yaml offset>` correct for the EXE and leaves flat overlays byte-identical; `_splat_yaml` resolves main to `splat.us.exe.yaml`. |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
|
||||
>
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1164 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1165 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
|
||||
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
|
||||
|
||||
@@ -450,7 +450,7 @@
|
||||
- **§430** — ★★★ — A GOTO INTO A LOOP IS FINE; HAND-HOIST THE CONSTANTS IT COSTS YOU (P31 S73 — **this section previously said the OPPOSITE and was wrong; the refutation is kept below**) <sub>L34073</sub>
|
||||
- **§472** — 🔴 §148-A's HOIST THRESHOLD IS 29, NOT 58, WHEN THE LOOP CONTAINS A CALL <sub>L35489</sub>
|
||||
|
||||
### structs, block moves & memcpy (91)
|
||||
### structs, block moves & memcpy (92)
|
||||
|
||||
- **§3-T2** — Source statement order drives instruction scheduling <sub>L78</sub>
|
||||
- **§5** — Known hard-residual classes (instruction-identical, one byte-exact blocker) <sub>L199</sub>
|
||||
@@ -543,8 +543,9 @@
|
||||
- **§442** — ★★★ — A RECOVERY RUNG THAT REWRITES A LEGAL CONSTRUCT INTO AN ILLEGAL ONE READS EXACTLY LIKE A CODEGEN WALL (P31 S75; `reconcile_tu`, 344 ins unblocked) <sub>L34647</sub>
|
||||
- **§468** — THE `%lo`-FOLD EXTENDS TO STORES ONLY VIA `extern Struct SYM[]`, AND MASKING HID THE OPERAND ORDER <sub>L35380</sub>
|
||||
- **§469** — THE `MEM_IN_STRUCT_P` ALIAS UNLOCK (and §463's spill law, independently confirmed) <sub>L35414</sub>
|
||||
- **§497** — ★ — A BODILESS `typedef struct Tag Alias;` DEFINES THE ALIAS, NOT THE TAG: THE CARRIER'S FALSE "CONFLICTING BODIES" REFUSAL (P32 T1b; ov_SC02_017 `func_80186C64` isolation) <sub>L36695</sub>
|
||||
|
||||
### types, signedness & load/store width (97)
|
||||
### types, signedness & load/store width (98)
|
||||
|
||||
- **§3-I1** — Unsigned range check: `(x - lo) < (hi-lo)` → `addiu`+`sltiu` <sub>L41</sub>
|
||||
- **§3-I2** — Byte mask forces `andi` even after `lbu` <sub>L47</sub>
|
||||
@@ -643,6 +644,7 @@
|
||||
- **§423** — ★★★ — "MATCH IN ISOLATION + GATE REJECTS + CAUSE NOT DETERMINED" ⇒ GREP THE TU FOR A FILE-SCOPE TYPEDEF THE DRAFT ALSO CARRIES (P31 S71; byte-proven `ov_SC03_092/func_8017FA74`) <sub>L33780</sub>
|
||||
- **§481** — ★★★ — `conflicting types` IS A SAME-SCOPE ERROR; ACROSS SCOPES IT IS ONLY A WARNING (P31 S77, `main:func_8001FC08`, 400 ins) <sub>L35901</sub>
|
||||
- **§496** — ★★ — A CARRIED-TYPE TEST THAT ASSUMES THE OVERLAY INCLUDE SET SILENTLY DROPS A RESIDENT TYPEDEF (P32 T1a; byte-proven, resident `func_800D128C` isolation) <sub>L36661</sub>
|
||||
- **§497** — ★ — A BODILESS `typedef struct Tag Alias;` DEFINES THE ALIAS, NOT THE TAG: THE CARRIER'S FALSE "CONFLICTING BODIES" REFUSAL (P32 T1b; ov_SC02_017 `func_80186C64` isolation) <sub>L36695</sub>
|
||||
|
||||
### declarations, prototypes & K&R (122)
|
||||
|
||||
@@ -2956,6 +2958,7 @@
|
||||
- **§494** — ★★★ — TEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79/S80 #9): THE IDIOMS, THE PLUMBING, AND THE THREE WAYS AN AGENT'S "MATCH" WAS NOT ONE <sub>L36480</sub>
|
||||
- **§495** — ★★★ — THE VERBATIM END-STATE (P31 S80 #10): TWO DEF-SIDE DECLARATION WALLS, A "BANK" THAT WAS THE ASSEMBLY, AND A GATE THAT DROPPED A BANK ON EXIT 0 <sub>L36620</sub>
|
||||
- **§496** — ★★ — A CARRIED-TYPE TEST THAT ASSUMES THE OVERLAY INCLUDE SET SILENTLY DROPS A RESIDENT TYPEDEF (P32 T1a; byte-proven, resident `func_800D128C` isolation) <sub>L36661</sub>
|
||||
- **§497** — ★ — A BODILESS `typedef struct Tag Alias;` DEFINES THE ALIAS, NOT THE TAG: THE CARRIER'S FALSE "CONFLICTING BODIES" REFUSAL (P32 T1b; ov_SC02_017 `func_80186C64` isolation) <sub>L36695</sub>
|
||||
|
||||
|
||||
---
|
||||
@@ -4132,3 +4135,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
|
||||
| L36480 | §494 | ★★★ — TEN BANKS FROM ONE-AGENT-PER-FUNCTION DRAFTING (P31 S79/S80 #9): THE IDIOMS, THE PLU |
|
||||
| L36620 | §495 | ★★★ — THE VERBATIM END-STATE (P31 S80 #10): TWO DEF-SIDE DECLARATION WALLS, A "BANK" THAT |
|
||||
| L36661 | §496 | ★★ — A CARRIED-TYPE TEST THAT ASSUMES THE OVERLAY INCLUDE SET SILENTLY DROPS A RESIDENT TY |
|
||||
| L36695 | §497 | ★ — A BODILESS `typedef struct Tag Alias;` DEFINES THE ALIAS, NOT THE TAG: THE CARRIER'S F |
|
||||
|
||||
@@ -36691,3 +36691,26 @@ isolated region = a type the carrier believed the headers provide. Ask WHICH hea
|
||||
before believing any "the header has it" guard; the resident/md_/main TUs are the `common.h`-only class.
|
||||
Same class as §323 (attributes hid the name) and §321 (re-emission): the carrier's blind spots are always
|
||||
"a type-name recogniser or a provenance assumption", never the C.
|
||||
|
||||
#### §497 ★ — A BODILESS `typedef struct Tag Alias;` DEFINES THE ALIAS, NOT THE TAG: THE CARRIER'S FALSE "CONFLICTING BODIES" REFUSAL (P32 T1b; ov_SC02_017 `func_80186C64` isolation)
|
||||
|
||||
**The refusal.** `jr_isolate_all ov_SC02_017 --only func_80186C64` refused (R43): *1 carried type name(s) have CONFLICTING
|
||||
bodies — a rename is needed, not a dedupe: `Rec801806C8_s` — A: `struct Rec801806C8_s { s32 f0; s32 f1; s32 f2; }
|
||||
__attribute__((packed, aligned(1)));` B: `typedef struct Rec801806C8_s Rec801806C8;`*. Both lines are legal C in one TU
|
||||
(a tag definition, then a typedef naming it), so `docs/frontier-p32.md` had costed a source RENAME.
|
||||
|
||||
**The mechanism.** The dedupe in `_file_scope_decls` keys a type block by `_type_names(block)`, whose recogniser
|
||||
`\b(?:struct|union|enum)\s+(\w+)` returns the TAG for a bodiless `typedef struct Tag Alias;`. The tag's definition block
|
||||
yields the same single name, so two DIFFERENT blocks shared one key, their normalised bodies differed, and the refusal
|
||||
that exists for a genuine redefinition fired. The line itself declares `Rec801806C8` (ordinary namespace) and only
|
||||
REFERENCES the tag — C's tag and ordinary namespaces are distinct, and the carrier had collapsed them.
|
||||
|
||||
**The fix (tool, not rename).** `_TYPEDEF_TAG_ALIAS = ^\s*typedef\s+(struct|union|enum)\s+(\w+)\s+(\w+)\s*;\s*$`; when it
|
||||
matches and alias ≠ tag, `_type_names` returns `{alias}` and the `carried` set learns the alias. `typedef struct X X;`
|
||||
(alias == tag) keeps the old key on purpose: two of those in one TU ARE a redefinition and must still collapse or
|
||||
refuse. Unit control on seven block shapes (tag+body, typedef-anon, typedef-tag-body, fn-ptr typedef, `struct X;`);
|
||||
the overlay's dry-run went REFUSED → CLEAN (2 region files) with the source untouched.
|
||||
|
||||
**Tell + generalization.** A "CONFLICTING bodies" refusal whose two bodies are a struct DEFINITION and a bodiless
|
||||
`typedef struct <same tag> <other name>;` is this class — no rename, the carrier is wrong. Sibling of §496 (the
|
||||
provided-type set assumed the overlay include set): the carrier's failures are recogniser/namespace assumptions.
|
||||
|
||||
@@ -684,6 +684,10 @@ def _strip_attrs(block):
|
||||
return _ATTR.sub(' ', block)
|
||||
|
||||
|
||||
# `typedef struct Tag Alias;` — a bodiless typedef of an existing tag (see _type_names, §497)
|
||||
_TYPEDEF_TAG_ALIAS = re.compile(r'^\s*typedef\s+(?:struct|union|enum)\s+([A-Za-z_]\w*)\s+([A-Za-z_]\w*)\s*;\s*$')
|
||||
|
||||
|
||||
def _file_scope_decls(items, provided=None):
|
||||
"""[(line, [syms])] for every decl that stood at FILE SCOPE in the original TU, in item
|
||||
order. TWO sources — the second is the §8b scoping-wall fix:
|
||||
@@ -716,6 +720,9 @@ def _file_scope_decls(items, provided=None):
|
||||
block = _strip_attrs(block)
|
||||
for a, b in re.findall(r'\}\s*([A-Za-z_]\w*)\s*;|\b(?:struct|union|enum)\s+([A-Za-z_]\w*)', block):
|
||||
carried.add(a or b)
|
||||
m = _TYPEDEF_TAG_ALIAS.match(block.strip()) # the alias of `typedef struct Tag Alias;` (§497)
|
||||
if m:
|
||||
carried.add(m.group(2))
|
||||
carried |= set(re.findall(r'typedef\s+[^;{}]*?\(\s*\*\s*([A-Za-z_]\w*)\s*\)\s*\([^;]*\)\s*;', block))
|
||||
if provided is None: # legacy callers: the overlay assumption
|
||||
provided = _engine_types()
|
||||
@@ -738,6 +745,15 @@ def _file_scope_decls(items, provided=None):
|
||||
|
||||
def _type_names(block):
|
||||
block = _strip_attrs(block)
|
||||
# `typedef struct Tag Alias;` (no body) DEFINES the alias and merely REFERENCES the tag (P32 T1b,
|
||||
# cookbook §497). Keying it by the tag made it collide with the tag's own `struct Tag {...};`
|
||||
# definition block — same key, different bodies — and the R43 refusal fired on legal C
|
||||
# (ov_SC02_017: `struct Rec801806C8_s {...} __attribute__((packed, aligned(1)));` +
|
||||
# `typedef struct Rec801806C8_s Rec801806C8;`). A typedef whose alias EQUALS the tag
|
||||
# (`typedef struct X X;`) keeps the old key so it still dedupes/refuses against a second one.
|
||||
m = _TYPEDEF_TAG_ALIAS.match(block.strip())
|
||||
if m and m.group(1) != m.group(2):
|
||||
return {m.group(2)}
|
||||
names = {a or b for a, b in
|
||||
re.findall(r'\}\s*([A-Za-z_]\w*)\s*;|\b(?:struct|union|enum)\s+([A-Za-z_]\w*)', block)}
|
||||
names |= set(re.findall(r'typedef\s+[^;{}]*?\(\s*\*\s*([A-Za-z_]\w*)\s*\)\s*\([^;]*\)\s*;', block))
|
||||
|
||||
Reference in New Issue
Block a user