mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 05:56:00 -04:00
feat(tools): asm_in_c.py — 154 game functions are assembly wearing a .c extension
A .c file in src/ looks decompiled. 199 functions are not: they are the target
assembly pasted into a C string literal (§265), byte-identical BY CONSTRUCTION
and completely unexplained. 45 are PsyQ/CRT routines where that is defensible;
154 are GAME CODE, 171 of the 199 in main, the largest being SaveLoadRoutine at
1,165 instructions.
They were invisible because progress.py's classify() matched INCLUDE_ASM,
INCLUDE_RODATA and C definitions, and a file-scope __asm__ block is none of
those -- so each landed in NO bucket, either swallowed by a surrounding
construct or surfacing as the single `UNPLACED (parse hole)` line the tool has
been printing all along.
progress.py gains a VERBATIM __asm__ bodies line: counted byte-identical (it is,
by construction) but NEVER as REAL. main's headline moves 45.88% -> 42.15%.
Nothing regressed and no work was lost -- the denominator was missing 173
functions that are real remaining work.
THE COUNTING LESSON IS THE REUSABLE PART. Counting these by hand went
116 -> 112 -> 108 -> 178 -> 199 across five attempts in one session, every
intermediate number reported confidently. All five errors were one shape, a
pattern narrower than the claim it supported:
* the sources use BOTH ".ent\tNAME\n" and ".ent NAME\n" -- anchoring on either
silently drops every instance of the other;
* a bare ".ent\t" fragment yields a phantom function literally named `t`, six
times, which is the only reason the error was noticed;
* __asm__ appears in 3,182 of 4,224 sources, almost all the §3a barrier, so
counting files or counting __asm__ measures nothing;
* `.globl NAME` + `NAME:` proves EXPORT, not CODE -- the first real run
reported jtbl_80072ED4/EEC/F0C/F24 as four "functions";
* a hand-written SDK name list reported 170 game functions because it did not
know VectorNormalSS / SquareRoot12 / OuterProduct12 are libgte.
So the tool does not trust one regex: THREE independent detectors that must
agree with disagreement reported as a defect (R34 -- that is what caught the
jump tables); SDK-ness DERIVED from the 14 shipped PsyQ archives via nm (2,227
symbols) rather than a list (R33); coverage asserted so a definition-shaped
block no detector claims fails loudly (R32/R43); and --selftest carrying a
known-true case of every spelling plus the phantom `t` and the jtbl regression.
Cookbook §448, SETUP row. Law: when a count comes from a text pattern, the
pattern has a denominator too -- validate it against one known-true case of
every FORM the corpus contains before quoting the number.
This commit is contained in:
Drew T
@@ -774,6 +774,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/jtbl_carve.py` — per-table `sltiu` bound | **(P31 S75, cookbook §446)** Two silent defects that made a byte-identical `.text` gate DIFF for five sessions across four 279-ins siblings. (a) **The over-span clamp disabled itself on the functions that needed it.** spimdisasm runs an island's LAST `jtbl_` dlabel one word into the following NON-ZERO data (string bytes), so the zero-word trim cannot see it; the clamp that would have caught it was guarded by `len(sltiu_bounds)==1`, and `sltiu` is ALSO how gcc emits an unsigned range check (`(u32)(x-lo) < n`, I1) — the four siblings carry five distinct `sltiu` immediates, so the guard stood down. A 0x28 table got 0x2C reserved: image 4 bytes short, ~850 `%lo` shifted, whole-binary DIFF. Fixed with `_table_bound()` — gcc-2.7.2's dispatch is a fixed idiom, so the `sltiu` nearest ABOVE that table's own `%hi(jtbl_X)` is unambiguous per table. (b) **A carve span whose `JTBL_PADS` line has no `tables=` comment** (written by `jtbl_pads_fix`) fell through both merge branches and lost its existing table's start, refusing with "table starts do not fit the span" — which `harvest_verify` then 'repaired' with a needless `jr_isolate_all` that walked back into (a); the merge now seeds from the span start unconditionally, per the invariant the validator already asserts (every carve span begins with a table). NC over every other open table-bearing stub fleet-wide: 11 tables, 10 unchanged, 1 changed — a FIFTH latent victim (`ov_SC06_022:func_80185B80`). All five banked byte-identical. |
|
||||
| | `tools/dedup_propagate.py` — `_split_masked` memo | **(P31 S75)** `find_site` re-ran `cdecl._mask` (4 regex passes, one `re.S`) + two `splitlines()` over the ENTIRE concatenated source on EVERY call, though the mask is a pure function of the text and the caller loops over every function in the binary. Measured on `ov_SC01_005` (2.50 MB, 2,503 fns): 6.5 + 38.9 + 4.6 = **54 ms per call before any searching**. Memoized via `lru_cache(maxsize=8)` — callers already hold one text object per binary and CPython caches a str's hash, so a repeat lookup is a pointer compare. **2x on that loop (58.3 -> 33.0 ms/call), NC identical results on 120 addresses.** Honest scope: that loop is only ~2.4 min of a 30-min run; the profiler puts 43% of `--check-only` in `family_remap._alias_decl_for` (107 s / 1,312 calls), which is the real target and is NOT fixed here. |
|
||||
| | `tools/main_diff_locate.py` — `classify()` | **(P31 S75, cookbook §447)** The `TABLE REJECT` class was **unreachable for main** and demanded purity. (a) It summed bytes whose object string contains `(.rodata)`, but main's `section_order` is `[.rodata, .text, .data, .bss]` — its rodata sits BELOW `.text` and its jump tables live in `.data` objects, so the test could never fire on the one binary with the most jump-table functions left. Now matches `(.data)` OR `(.rodata)`. (b) It required `ro == outside`, so a few bytes of perturbed code dropped the verdict through to `PLUMBING REJECT` and its §376 declaration advice. Now **dominance-based** (≥60%), reporting the split and naming which part is the table problem and which the declaration problem. Measured on `SaveLoadRoutine` (1,165 ins, the §434 wall): body BYTE-IDENTICAL, 3,787 of 3,989 differing bytes (94.9%) in `.data` jump tables, 202 (5.1%) in `.text`, built image 4 bytes SHORT — verdict moved `PLUMBING REJECT` → `TABLE REJECT (MIXED)`. The §376 chain had been run on it twice and fixed nothing, because it addresses the 5%. NC over all five pre-existing verdict shapes: 5 of 6 unchanged. |
|
||||
| | **`tools/asm_in_c.py`** (NEW) | **(P31 S75, cookbook §448)** Finds every function that is **assembly posing as C** — a §265 file-scope `__asm__` body (class A) or a C function whose body is only asm statements (class B) — while correctly EXCLUDING the §3a cross-jump barrier, which is what `__asm__` means in 3,182 of the 4,224 sources. Measured: **199 functions, 154 of them GAME CODE, 171 in main**, largest `SaveLoadRoutine` (1,165 ins). These were in NO `progress.py` bucket, so main's REAL% was overstated (45.88% → **42.15%** once counted; `progress.py` gained a `VERBATIM __asm__ bodies` line that counts them byte-identical but NEVER as REAL). Design is the point: **three independent detectors that must agree**, disagreement reported as a defect (it caught `jtbl_*` being claimed as functions); **SDK-ness derived from the 14 shipped PsyQ archives via `nm`** (2,227 symbols), not a hand list (which had mis-classified `VectorNormalSS`/`SquareRoot12`/`OuterProduct12` as game code); coverage asserted; and **`--selftest`** with a known-true case of EVERY spelling — hand counts went 116→112→108→178→199 because the sources use both `".ent\tNAME\n"` and `".ent NAME\n"`, and a bare `".ent\t"` fragment yields a phantom function called `t`. Run `--selftest` before believing the number. |
|
||||
| | `tools/jr_isolate_all.py` — boundary derivation | **(P31 S74, cookbook §441)** `_region_emit_start()`: the yaml offset for a region is derived from the region's **CONTENT** — min of item addresses and of every `.globl`/`.ent` its text names that resolves inside the object — and taken as `min(cut, emit)`, so a boundary can only move DOWN. Reason: **a §265 verbatim `__asm__` body is not one of `parse_overlay_c`'s four addressed-anchor forms, so it attaches to the NEXT anchor as PREAMBLE — and preamble is assumed byte-neutral when it emits bytes.** A cut at `func_800D0268` would have moved 0x168 bytes of three other functions into the new object while the yaml claimed the region started higher. Where no verbatim asm is in play it equals the cut, so every existing isolate is unchanged. Also: an item-less CLOSING region used to emit a duplicate `- [off, c, …]` line (the empty-region skip covered only region 0, and `_partition`'s empty `footer` made the closing region look non-empty). |
|
||||
| | `tools/jtbl_rodata_pads.py --derive` | **(Phase 31 S62 T3a, cookbook §303)** Module path of the §8e pads filter: the Makefile runs `--derive $(BINARY) --tu <tu>` for every `md_*` object **and for `main` (P31 S72)** — jump-table pads derived at build time from the retail island + the emission stream (trailing pads `0t1`, table-aware, const data passes through). No stored spec; an anchor miss fails the build with the offset. **S72:** `--derive` now serves main too — `_file0_vram` returns the code segment's `vram - start` (the PS-X EXE's 0x800 header), which makes both `raw[a - vram]` and `vram + <yaml offset>` correct for the EXE and leaves flat overlays byte-identical; `_splat_yaml` resolves main to `splat.us.exe.yaml`. |
|
||||
| | `config/wave_exclude.txt` | **(P31 S72)** THE canonical wave exclude list — **tracked**, and **regenerated, never hand-edited**: `tools/exclude_audit.py config/wave_exclude.txt --write <new>`. Consumed as `draw_waves --exclude-file config/wave_exclude.txt`, which AUDITS it as a PREREQUISITE and refuses to draw on a stale one (`--exclude-stale-ok` overrides, loudly). Two classes: **CARVE-BLOCKED** (derived from `split_indicator`; disappears when the subseg is split — **EMPTY since P31 S74**, all four overlays split) and **WALL** (curated, cannot be re-derived — the `# WALL:` annotation is a PIN that survives regeneration, and its note is the refutation list to beat before reopening the entry). It replaces the nine `.run/S*_exclude.txt` snapshots and seven walls ledgers, none of which was authoritative; measured on its predecessor, 88 of 107 entries were stale one day after it was written, 46 of them open drawable work totalling 12,750 instructions. |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
|
||||
>
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1116 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1117 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
|
||||
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
|
||||
|
||||
@@ -1385,7 +1385,7 @@
|
||||
- **§430** — ★★★ — A GOTO INTO A LOOP IS FINE; HAND-HOIST THE CONSTANTS IT COSTS YOU (P31 S73 — **this section previously said the OPPOSITE and was wrong; the refutation is kept below**) <sub>L34073</sub>
|
||||
- **§439** — ★★ — LEVER SET FROM THE S74 WAVE (each entry is one measured crack, not a hypothesis) <sub>L34555</sub>
|
||||
|
||||
### (unbucketed — title matched no symptom vocabulary) (331)
|
||||
### (unbucketed — title matched no symptom vocabulary) (332)
|
||||
|
||||
- **§3-How** — to use this <sub>L30</sub>
|
||||
- **§1** — Idiom catalog (asm pattern → C that produces it) <sub>L39</sub>
|
||||
@@ -1718,6 +1718,7 @@
|
||||
- **§441** — ★★ — THREE MORE INSTRUMENT DEFECTS FROM THE SAME SESSION, ALL OF WHICH BLAME THE SUBJECT (P31 S74) <sub>L34617</sub>
|
||||
- **§443** — ★★ — A DERIVED DEPENDENCY MUST BE PROVISIONED BY *EVERY* PROVISIONER, AND WE HAVE TWO (P31 S69 → S74, the same defect twice) <sub>L34702</sub>
|
||||
- **§445** — ★★ — `make clean BINARY=<x>` IS FLEET-WIDE: THE VARIABLE IS ACCEPTED AND IGNORED (P31 S75) <sub>L34778</sub>
|
||||
- **§448** — ★★★ — ASSEMBLY POSING AS C: 154 GAME FUNCTIONS THE REPORTS COUNTED AS DONE (P31 S75) <sub>L34898</sub>
|
||||
|
||||
|
||||
## All sections, in order
|
||||
@@ -2838,6 +2839,7 @@
|
||||
- **§445** — ★★ — `make clean BINARY=<x>` IS FLEET-WIDE: THE VARIABLE IS ACCEPTED AND IGNORED (P31 S75) <sub>L34778</sub>
|
||||
- **§446** — ★★★ — "RELOC-ONLY REMAP GATED DIFF" IS A CARVE VERDICT, NOT A CODEGEN VERDICT, UNTIL YOU DIFF THE `.text` (P31 S75; 4 siblings + 1 latent, ~1,300 ins) <sub>L34807</sub>
|
||||
- **§447** — ★★★ — THE BIGGEST FUNCTION LEFT IS A CARVE, AND THE VERDICT TOOL SAID "DECLARATIONS" (P31 S75; SaveLoadRoutine, 1,165 ins) <sub>L34854</sub>
|
||||
- **§448** — ★★★ — ASSEMBLY POSING AS C: 154 GAME FUNCTIONS THE REPORTS COUNTED AS DONE (P31 S75) <sub>L34898</sub>
|
||||
|
||||
|
||||
---
|
||||
@@ -3966,3 +3968,4 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
|
||||
| L34778 | §445 | ★★ — `make clean BINARY=<x>` IS FLEET-WIDE: THE VARIABLE IS ACCEPTED AND IGNORED (P31 S75) |
|
||||
| L34807 | §446 | ★★★ — "RELOC-ONLY REMAP GATED DIFF" IS A CARVE VERDICT, NOT A CODEGEN VERDICT, UNTIL YOU D |
|
||||
| L34854 | §447 | ★★★ — THE BIGGEST FUNCTION LEFT IS A CARVE, AND THE VERDICT TOOL SAID "DECLARATIONS" (P31 |
|
||||
| L34898 | §448 | ★★★ — ASSEMBLY POSING AS C: 154 GAME FUNCTIONS THE REPORTS COUNTED AS DONE (P31 S75) |
|
||||
|
||||
@@ -34894,3 +34894,46 @@ class that does not exist: it converts "I don't know" into confident, specific,
|
||||
verdict tool names a subsystem, check that the named subsystem owns the MAJORITY OF THE BYTES before
|
||||
you act on it. Here 95% of the evidence pointed one way and the recommendation pointed the other,
|
||||
and nothing in the pipeline compared the two.
|
||||
|
||||
## §448 ★★★ — ASSEMBLY POSING AS C: 154 GAME FUNCTIONS THE REPORTS COUNTED AS DONE (P31 S75)
|
||||
|
||||
A `.c` file in `src/` looks decompiled. **199 functions in this tree are not**: they are the target
|
||||
assembly pasted into a C string literal (§265), byte-identical BY CONSTRUCTION and completely
|
||||
unexplained. 45 are PsyQ/CRT routines where that is defensible; **154 are game code**, 171 of the 199
|
||||
in `main` alone, the largest being `SaveLoadRoutine` at 1,165 instructions.
|
||||
|
||||
They were invisible because `progress.py`'s `classify()` matched `INCLUDE_ASM`, `INCLUDE_RODATA` and
|
||||
C definitions — and a file-scope `__asm__` block is none of those, so each one landed in NO bucket:
|
||||
either swallowed by a surrounding construct or surfacing as the single `UNPLACED (parse hole)` line.
|
||||
Main's headline moved **45.88% → 42.15%** once they were counted; nothing regressed, the denominator
|
||||
had simply been missing 173 functions that are real work.
|
||||
|
||||
**THE COUNTING LESSON, WHICH IS THE REUSABLE PART.** Counting them by hand went
|
||||
**116 → 112 → 108 → 178 → 199** across five attempts in one session, every intermediate number
|
||||
reported with confidence. Every error was the same shape — a pattern narrower than the claim it
|
||||
supported:
|
||||
|
||||
* the sources use **BOTH** `".ent\tNAME\n"` (escaped tab) **and** `".ent NAME\n"` (literal space);
|
||||
a pattern anchored on either one silently drops every instance of the other;
|
||||
* a bare fragment `".ent\t"` also occurs, and an optional-`\t` pattern captured the literal name
|
||||
**`t`** from it, six times, which is the only reason the error was noticed at all;
|
||||
* `__asm__` appears in **3,182 files**, almost all of it the §3a zero-byte cross-jump barrier, so
|
||||
counting files — or counting `__asm__` — measures nothing;
|
||||
* a `.globl NAME` + `NAME:` pair proves the symbol is **EXPORTED, not CODE**: the first real run
|
||||
reported `jtbl_80072ED4/EEC/F0C/F24` as four "functions", when they are jump tables sitting in the
|
||||
same block;
|
||||
* a hand-written SDK name list reported 170 game functions because it did not know `VectorNormalSS`,
|
||||
`SquareRoot12` and `OuterProduct12` are libgte — an incomplete list inflates exactly the number
|
||||
that matters.
|
||||
|
||||
`tools/asm_in_c.py` is the answer to all five, and its design is the lesson: **three independent
|
||||
detectors that must AGREE, with disagreement reported as a defect** (R34 — that is what caught the
|
||||
jump tables); **SDK-ness derived from the 14 shipped PsyQ archives via `nm`** (2,227 symbols) rather
|
||||
than a list (R33); **coverage asserted**, so a definition-shaped block no detector claims fails loudly
|
||||
(R32/R43); and a **`--selftest` carrying a known-true case of every spelling plus the phantom `t` and
|
||||
the jump-table regression**, so the next spelling change fails loudly instead of quietly returning a
|
||||
smaller number.
|
||||
|
||||
**The law: when a count comes from a text pattern, the pattern has a denominator too.** Validate it
|
||||
against one known-true case of every FORM the corpus contains before quoting the number — and if you
|
||||
cannot enumerate the forms, you do not yet know what you are counting.
|
||||
|
||||
+10
-10
@@ -4,24 +4,24 @@
|
||||
# cross-binary collapsible-byte leverage: docs/duplicates.cross.md.
|
||||
|
||||
# THREE progress metrics (all matter — see the labels):
|
||||
FLEET fn-count byte-ident: 362891 / 362959 = 99.98% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 13484569 / 13523865 = 99.7% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5813098 / 5851972 = 99.3% (90902/90929 unique fns; the DISTINCT-RE number)
|
||||
MAIN game-code weighted : 44903 / 79510 = 56.5% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
|
||||
FLEET fn-count byte-ident: 363002 / 363068 = 99.98% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 13484747 / 13523865 = 99.7% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5813276 / 5851972 = 99.3% (90902/90929 unique fns; the DISTINCT-RE number)
|
||||
MAIN game-code weighted : 45081 / 79510 = 56.7% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
|
||||
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 13439666 / 13444355 = 100.0%)
|
||||
|
||||
FLEET REAL substantive : 360721 (of which dedup-shared 255632 via 2220 groups / 255708 instances)
|
||||
FLEET REAL substantive : 360722 (of which dedup-shared 255632 via 2220 groups / 255708 instances)
|
||||
FLEET LINKED PsyQ objs : 959
|
||||
FLEET NON_MATCHING : 0 (0 in any default build — G4)
|
||||
FLEET INCLUDE_ASM stubs : 68
|
||||
FLEET matchable : 362959
|
||||
FLEET INCLUDE_ASM stubs : 66
|
||||
FLEET matchable : 363068
|
||||
|
||||
| binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| main | 880 | 2 | 959 | 1882 | 1918 | 98.1% |
|
||||
| main | 881 | 2 | 959 | 1991 | 2025 | 98.3% |
|
||||
| resident | 141 | 0 | 0 | 143 | 145 | 98.6% |
|
||||
| md_MAIN_001 | 11 | 0 | 0 | 11 | 11 | 100.0% |
|
||||
| md_MAIN_003 | 46 | 0 | 0 | 46 | 47 | 97.9% |
|
||||
| md_MAIN_003 | 46 | 0 | 0 | 47 | 48 | 97.9% |
|
||||
| md_MAIN_008 | 6 | 0 | 0 | 6 | 6 | 100.0% |
|
||||
| md_MAIN_011 | 20 | 0 | 0 | 21 | 21 | 100.0% |
|
||||
| md_MAIN_013 | 16 | 0 | 0 | 16 | 16 | 100.0% |
|
||||
@@ -193,7 +193,7 @@ FLEET matchable : 362959
|
||||
| ov_SC05_002 | 2439 | 1811 | 0 | 2442 | 2442 | 100.0% |
|
||||
| ov_SC05_003 | 2479 | 1811 | 0 | 2480 | 2481 | 100.0% |
|
||||
| ov_SC05_004 | 2462 | 1811 | 0 | 2464 | 2464 | 100.0% |
|
||||
| ov_SC05_005 | 2489 | 1825 | 0 | 2490 | 2490 | 100.0% |
|
||||
| ov_SC05_005 | 2489 | 1825 | 0 | 2491 | 2491 | 100.0% |
|
||||
| ov_SC05_006 | 2430 | 1813 | 0 | 2430 | 2430 | 100.0% |
|
||||
| ov_SC05_007 | 2477 | 1812 | 0 | 2482 | 2482 | 100.0% |
|
||||
| ov_SC05_008 | 2541 | 1812 | 0 | 2543 | 2543 | 100.0% |
|
||||
|
||||
+7
-6
File diff suppressed because one or more lines are too long
@@ -0,0 +1,382 @@
|
||||
#!/usr/bin/env python3
|
||||
"""asm_in_c.py — find every function that is ASSEMBLY POSING AS C.
|
||||
|
||||
WHY THIS EXISTS (P31 S75). A `.c` file in `src/` looks decompiled. Some of it is not: a §265
|
||||
verbatim body is the target assembly pasted into a C string literal, byte-identical BY CONSTRUCTION
|
||||
and completely unexplained. Those functions are real work still to do, and until now nothing counted
|
||||
them — `progress.py`'s `classify()` matched `INCLUDE_ASM`, `INCLUDE_RODATA` and C definitions, and a
|
||||
file-scope `__asm__` block matched none of them, so it fell into the `UNPLACED (parse hole)` line or
|
||||
was swallowed by a surrounding construct.
|
||||
|
||||
THE MEASUREMENT THIS TOOL EXISTS TO GET RIGHT. Counting them by hand went 116 -> 112 -> 108 -> 178
|
||||
across four attempts in one session, every number confidently reported, because:
|
||||
|
||||
* the sources use BOTH `".ent\\tNAME\\n"` (tab) and `".ent NAME\\n"` (space) — a pattern anchored
|
||||
on one silently drops every instance of the other;
|
||||
* a bare fragment `".ent\\t"` also occurs, and an optional-`\\t` pattern captured the literal name
|
||||
`t` from it, six times;
|
||||
* `__asm__` appears in 3,182 files, almost all of it the §3a zero-byte cross-jump barrier
|
||||
(`__asm__ __volatile__("" ::: "memory")`) — counting files, or counting `__asm__`, is meaningless.
|
||||
|
||||
So this tool does NOT trust one regex. It runs THREE independent detectors and makes them ARGUE
|
||||
(R34): a disagreement is reported as a DEFECT, not silently resolved. It asserts its own coverage
|
||||
(R32): every `__asm__` block is classified as function-defining or not, and an unclassifiable one
|
||||
fails loudly. And `--selftest` checks it against known-true cases of EACH spelling before you are
|
||||
allowed to believe its number.
|
||||
|
||||
THREE CLASSES, and only the first two are "posing":
|
||||
|
||||
A FILE-SCOPE VERBATIM — a file-scope `__asm__("...")` that DEFINES a function
|
||||
(`.ent NAME` / `.globl NAME` + `NAME:` / `.type NAME, @function`).
|
||||
No C at all. This is §265.
|
||||
B ASM-BODIED C — a C function whose body contains nothing but asm statements. It has a
|
||||
real signature and prototype, so it reads as decompiled in every report,
|
||||
and it is not.
|
||||
C LEGITIMATE INLINE — a C function with real C plus some inline asm (the §3a barrier, register
|
||||
pins from §17). NOT posing, NOT reported. Excluding these correctly is
|
||||
most of the tool's value: they outnumber the real hits by ~20x.
|
||||
|
||||
Usage:
|
||||
tools/asm_in_c.py # report, all binaries
|
||||
tools/asm_in_c.py --binary main # one binary
|
||||
tools/asm_in_c.py --json out.json # machine-readable
|
||||
tools/asm_in_c.py --selftest # prove the detectors on known-true cases (DO THIS FIRST)
|
||||
tools/asm_in_c.py --list # bare "binary function" lines, for piping
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# PsyQ library subsegs: their stubs are byte-identical via LINKED SDK objects, not our source.
|
||||
LINKED_SEGS = set(
|
||||
'apicard1 apicard2 apicard3 apicard4 libc2_1 libc2_2 libcd1 libcd2 libetc libgpu libgs1 libgs2 '
|
||||
'libgs3 libgs4 libgs5 libgs6 libgte1 libgte10 libgte11 libgte12 libgte13 libgte14 libgte15 '
|
||||
'libgte16 libgte17 libgte18 libgte19 libgte2 libgte20 libgte21 libgte22 libgte3 libgte4 libgte5 '
|
||||
'libgte6 libgte7 libgte8 libgte9 libmcrd1 libmcrd2 snd1 snd2 snd3 snd4 snd5 snd6 snd7 snd8 '
|
||||
'snd9'.split())
|
||||
|
||||
# SDK vs GAME CODE — DERIVED FROM THE ACTUAL PsyQ ARCHIVES, not a hand-written list (R33).
|
||||
#
|
||||
# The first cut used a hardcoded set of ~35 names and reported "170 game code". Wrong: the run
|
||||
# immediately surfaced `VectorNormalSS`, `SquareRoot12`, `OuterProduct12`, `OuterProduct0` as
|
||||
# "game code" when every one is a libgte routine that simply was not on my list. A hand list cannot
|
||||
# be complete and its incompleteness silently INFLATES the number that matters most here (how much
|
||||
# real decompilation work is hiding). We ship the SDK archives, so ask them.
|
||||
#
|
||||
# Falls back to a small literal set only if the archives or `nm` are unavailable, and SAYS SO in the
|
||||
# report rather than pretending the classification is authoritative.
|
||||
_PSYQ_DIR = os.path.join(REPO, 'tools/psyq/lib40_elf')
|
||||
_NM = 'mipsel-linux-gnu-nm'
|
||||
_FALLBACK = set('DisableEvent WaitEvent ReturnFromException SysDeqIntRP GsSortBg GsSortClear '
|
||||
'MatrixNormal VectorNormal __main start _clr _drs _dws'.split())
|
||||
|
||||
|
||||
def sdk_names():
|
||||
"""Every function symbol DEFINED by the shipped PsyQ archives -> (set, provenance string)."""
|
||||
import subprocess
|
||||
if not os.path.isdir(_PSYQ_DIR):
|
||||
return _FALLBACK, 'FALLBACK literal list (no tools/psyq/lib40_elf) — classification is INDICATIVE'
|
||||
archives = sorted(f for f in os.listdir(_PSYQ_DIR) if f.lower().endswith('.a'))
|
||||
if not archives:
|
||||
return _FALLBACK, 'FALLBACK literal list (no archives found) — classification is INDICATIVE'
|
||||
names = set()
|
||||
for a in archives:
|
||||
try:
|
||||
r = subprocess.run([_NM, '--defined-only', os.path.join(_PSYQ_DIR, a)],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return _FALLBACK, f'FALLBACK literal list ({_NM} unavailable) — classification is INDICATIVE'
|
||||
for ln in r.stdout.splitlines():
|
||||
parts = ln.split()
|
||||
if len(parts) >= 3 and parts[1] in ('T', 't', 'W'):
|
||||
names.add(parts[2])
|
||||
if not names:
|
||||
return _FALLBACK, 'FALLBACK literal list (archives defined no T symbols) — INDICATIVE'
|
||||
return names, f'derived from {len(archives)} PsyQ archive(s) in tools/psyq/lib40_elf ({len(names)} symbols)'
|
||||
|
||||
|
||||
SDK_NAMES, SDK_PROVENANCE = sdk_names()
|
||||
|
||||
# --- the three detectors -------------------------------------------------------------------------
|
||||
# Each takes ONE __asm__ block's text and returns the set of function names it believes are DEFINED.
|
||||
# They are deliberately different shapes so that agreement is evidence and disagreement is a defect.
|
||||
#
|
||||
# NOTE the `(?:\\t|\s)+` in every one: the sources use BOTH the escaped tab and a literal space, and
|
||||
# requiring the `\n` terminator is what stops a bare `".ent\t"` fragment from yielding the name `t`.
|
||||
RE_ENT = re.compile(r'\.ent(?:\\t|[ \t])+(\w+)\\n')
|
||||
RE_TYPE = re.compile(r'\.type(?:\\t|[ \t])+(\w+)\s*,\s*@function')
|
||||
RE_GLOBL = re.compile(r'\.globl(?:\\t|[ \t])+(\w+)\\n')
|
||||
RE_LABEL = re.compile(r'"(\w+):\\n"')
|
||||
|
||||
|
||||
def detect_ent(txt):
|
||||
"""`.ent NAME` — the MIPS function-start directive. The most direct evidence."""
|
||||
return set(RE_ENT.findall(txt))
|
||||
|
||||
|
||||
def detect_type(txt):
|
||||
"""`.type NAME, @function` — the ELF symbol-type directive."""
|
||||
return set(RE_TYPE.findall(txt))
|
||||
|
||||
|
||||
# DATA-SHAPED NAMES. A `.globl NAME` + `NAME:` pair proves the symbol is EXPORTED, not that it is
|
||||
# CODE — and jump tables and data blobs are exported too. Caught on the tool's first real run:
|
||||
# `jtbl_80072ED4/EEC/F0C/F24` were reported as four "functions posing as C" when they are
|
||||
# SaveLoadRoutine's jump tables sitting in the same §265 block. `.ent` and `.type @function` are
|
||||
# code-specific and need no such filter; only the weakest detector does (R34 — the disagreement
|
||||
# between detectors is what exposed this, which is the whole point of running three).
|
||||
RE_DATA_NAME = re.compile(r'^(?:jtbl_|D_|_?LC?\d|\$L)')
|
||||
|
||||
|
||||
def detect_globl_label(txt):
|
||||
"""`.globl NAME` AND a `NAME:` label — a definition without the directives.
|
||||
|
||||
Exported-DATA names are excluded (see RE_DATA_NAME): this detector proves export, not code."""
|
||||
both = set(RE_GLOBL.findall(txt)) & set(RE_LABEL.findall(txt))
|
||||
return {n for n in both if not RE_DATA_NAME.match(n)}
|
||||
|
||||
|
||||
DETECTORS = (('ent', detect_ent), ('type', detect_type), ('globl+label', detect_globl_label))
|
||||
|
||||
# A block with any of these and NO definition is inline asm inside a function, not a definition.
|
||||
RE_BARRIER = re.compile(r'__asm__\s+__volatile__\s*\(\s*""')
|
||||
|
||||
|
||||
def asm_blocks(text):
|
||||
"""[(start_line, end_line, block_text, is_file_scope)] for every __asm__/asm statement.
|
||||
|
||||
Brace-depth tracking decides file scope, so a block INSIDE a C function (class B or C) is
|
||||
distinguished from a file-scope one (class A) without guessing from indentation."""
|
||||
lines = text.split('\n')
|
||||
out, depth, i, n = [], 0, 0, len(lines)
|
||||
while i < n:
|
||||
raw = lines[i]
|
||||
code = re.sub(r'//.*$', '', raw)
|
||||
if re.match(r'\s*(__asm__|asm)\b', code) or re.search(r'\b(__asm__|asm)\s*\(', code):
|
||||
start, d2, opened = i, 0, False
|
||||
while i < n:
|
||||
c = re.sub(r'//.*$', '', lines[i])
|
||||
d2 += c.count('(') - c.count(')')
|
||||
if '(' in c:
|
||||
opened = True
|
||||
i += 1
|
||||
if opened and d2 <= 0:
|
||||
break
|
||||
blk = '\n'.join(lines[start:i])
|
||||
out.append((start + 1, i, blk, depth == 0))
|
||||
# a statement inside a function still moves brace depth
|
||||
depth += blk.count('{') - blk.count('}')
|
||||
continue
|
||||
depth += code.count('{') - code.count('}')
|
||||
i += 1
|
||||
return out
|
||||
|
||||
|
||||
C_SIG = re.compile(r'^[A-Za-z_][\w \*]*\b(\w+)\s*\([^;]*\)\s*\{?\s*$')
|
||||
|
||||
|
||||
def c_functions(text):
|
||||
"""[(name, start, end, body)] for each C function DEFINITION, by brace matching."""
|
||||
lines = text.split('\n')
|
||||
out, i, n = [], 0, len(lines)
|
||||
while i < n:
|
||||
m = C_SIG.match(re.sub(r'//.*$', '', lines[i]).rstrip())
|
||||
if m and '(' in lines[i] and not lines[i].lstrip().startswith(('return', 'if', 'while', 'for', 'switch')):
|
||||
j, kind = i, None
|
||||
while j < n:
|
||||
c = re.sub(r'//.*$', '', lines[j])
|
||||
br, sm = c.find('{'), c.find(';')
|
||||
if br != -1 and (sm == -1 or br < sm):
|
||||
kind = 'def'
|
||||
break
|
||||
if sm != -1:
|
||||
kind = 'decl'
|
||||
break
|
||||
j += 1
|
||||
if kind == 'def':
|
||||
start, depth, opened, k = i, 0, False, j
|
||||
while k < n:
|
||||
c = re.sub(r'//.*$', '', lines[k])
|
||||
depth += c.count('{') - c.count('}')
|
||||
if '{' in c:
|
||||
opened = True
|
||||
k += 1
|
||||
if opened and depth <= 0:
|
||||
break
|
||||
out.append((m.group(1), start + 1, k, '\n'.join(lines[start:k])))
|
||||
i = k
|
||||
continue
|
||||
i += 1
|
||||
return out
|
||||
|
||||
|
||||
def strip_comments_and_strings(s):
|
||||
"""Blank comments and string literals so a body can be tested for REAL C content."""
|
||||
s = re.sub(r'/\*.*?\*/', ' ', s, flags=re.S)
|
||||
s = re.sub(r'//[^\n]*', ' ', s)
|
||||
s = re.sub(r'"(?:\\.|[^"\\])*"', '""', s)
|
||||
return s
|
||||
|
||||
|
||||
def scan_file(path, binary):
|
||||
"""-> (rows, defects). rows are the POSING functions; defects are coverage failures."""
|
||||
text = open(path, errors='ignore').read()
|
||||
rows, defects = [], []
|
||||
if '__asm__' not in text and 'asm(' not in text:
|
||||
return rows, defects
|
||||
|
||||
blocks = asm_blocks(text)
|
||||
cfuncs = c_functions(text)
|
||||
|
||||
for (ln0, ln1, blk, file_scope) in blocks:
|
||||
votes = {name: fn(blk) for name, fn in DETECTORS}
|
||||
union = set().union(*votes.values())
|
||||
|
||||
if file_scope and union:
|
||||
# CLASS A — a file-scope block that defines functions. Detectors must agree.
|
||||
disagree = {n: sorted(v) for n, v in votes.items() if v and v != union}
|
||||
for fname in sorted(union):
|
||||
agreeing = [n for n, v in votes.items() if fname in v]
|
||||
rows.append(dict(binary=binary, fn=fname, cls='A-FILE-SCOPE-VERBATIM',
|
||||
path=os.path.relpath(path, REPO), line=ln0,
|
||||
detectors=agreeing, lines=blk.count('\n') + 1))
|
||||
if disagree:
|
||||
defects.append(f"{os.path.relpath(path, REPO)}:{ln0}: detectors DISAGREE on which "
|
||||
f"functions this block defines: {votes} — resolve before trusting the count")
|
||||
elif file_scope and not union:
|
||||
# A file-scope asm block that defines nothing is data or a directive island. Only a
|
||||
# defect if it looks like it MEANT to define something (R32: never skip silently).
|
||||
if re.search(r'\.(ent|globl|type)\b', blk) or RE_LABEL.search(blk):
|
||||
defects.append(f"{os.path.relpath(path, REPO)}:{ln0}: file-scope __asm__ carries "
|
||||
f"definition-shaped directives but NO detector claimed a function — "
|
||||
f"a spelling this tool does not know (R43). Inspect it.")
|
||||
|
||||
# CLASS B — a C function whose body is ONLY asm statements.
|
||||
for (fname, s0, s1, body) in cfuncs:
|
||||
inner = body[body.find('{') + 1:body.rfind('}')]
|
||||
has_asm = re.search(r'\b(__asm__|asm)\s*\(', inner)
|
||||
if not has_asm:
|
||||
continue
|
||||
# remove every asm statement, then ask whether any C remains
|
||||
rest = re.sub(r'\b(?:__asm__|asm)\b(?:\s+__volatile__)?\s*\([^;]*\)\s*;', ' ',
|
||||
inner, flags=re.S)
|
||||
rest = strip_comments_and_strings(rest)
|
||||
# declarations alone are not "real C content" for this purpose; a return of a constant is.
|
||||
meaningful = re.sub(r'\b(register|volatile|const|static|unsigned|signed|struct|union|'
|
||||
r'char|short|int|long|float|double|void|[su](?:8|16|32|64)|f32)\b', ' ', rest)
|
||||
meaningful = re.sub(r'[\s;{}()\[\],*]|(?<![\w.])\w+(?![\w.])', ' ', meaningful).strip()
|
||||
if not meaningful and not RE_BARRIER.search(inner):
|
||||
rows.append(dict(binary=binary, fn=fname, cls='B-ASM-BODIED-C',
|
||||
path=os.path.relpath(path, REPO), line=s0,
|
||||
detectors=['c-body-is-only-asm'], lines=s1 - s0 + 1))
|
||||
return rows, defects
|
||||
|
||||
|
||||
def sources(binary=None):
|
||||
out = []
|
||||
for root, dirs, files in os.walk(os.path.join(REPO, 'src')):
|
||||
for f in files:
|
||||
if not f.endswith('.c') or f[:-2] in LINKED_SEGS:
|
||||
continue
|
||||
rel = os.path.relpath(root, os.path.join(REPO, 'src'))
|
||||
b = 'main' if rel == '.' else rel.split(os.sep)[0]
|
||||
if binary and b != binary:
|
||||
continue
|
||||
out.append((os.path.join(root, f), b))
|
||||
return sorted(out)
|
||||
|
||||
|
||||
def selftest():
|
||||
"""Prove the detectors on known-true cases of EVERY spelling before anyone believes a number.
|
||||
|
||||
These are the exact cases that broke four hand counts in one session. If the sources change
|
||||
spelling again, this fails LOUDLY rather than silently returning a smaller number."""
|
||||
cases = [
|
||||
('tab-spelled .ent', '__asm__(\n ".ent\\tSaveLoadRoutine\\n"\n "SaveLoadRoutine:\\n");', {'SaveLoadRoutine'}),
|
||||
('space-spelled .ent', '__asm__(\n ".ent func_80047D3C\\n"\n "func_80047D3C:\\n");', {'func_80047D3C'}),
|
||||
('.type @function', '__asm__(\n ".type\\tfoo, @function\\n");', {'foo'}),
|
||||
('globl + label', '__asm__(\n ".globl\\tbar\\n"\n "bar:\\n");', {'bar'}),
|
||||
('BARE FRAGMENT (phantom)','__asm__(\n ".ent\\t"\n);', set()),
|
||||
('barrier is NOT a def', '__asm__ __volatile__("" ::: "memory");', set()),
|
||||
# regression: a jump table is EXPORTED but is not a function (first real run, S75)
|
||||
('jtbl is data, not a fn', '__asm__(\n ".globl\\tjtbl_80072ED4\\n"\n "jtbl_80072ED4:\\n");', set()),
|
||||
('D_ is data, not a fn', '__asm__(\n ".globl\\tD_80073140\\n"\n "D_80073140:\\n");', set()),
|
||||
]
|
||||
ok = True
|
||||
print('SELFTEST — the detectors against known-true cases:')
|
||||
for label, txt, want in cases:
|
||||
got = set().union(*[fn(txt) for _, fn in DETECTORS])
|
||||
good = got == want
|
||||
ok &= good
|
||||
print(f" [{'PASS' if good else 'FAIL'}] {label:26s} want={sorted(want) or '-'} got={sorted(got) or '-'}")
|
||||
print('SELFTEST:', 'ALL PASS' if ok else '*** FAILURE — do not trust this tool until fixed ***')
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--binary')
|
||||
ap.add_argument('--json')
|
||||
ap.add_argument('--list', action='store_true', help='bare "<binary> <function>" lines')
|
||||
ap.add_argument('--selftest', action='store_true')
|
||||
ap.add_argument('--strict', action='store_true', help='exit non-zero if any coverage defect')
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.selftest:
|
||||
sys.exit(selftest())
|
||||
|
||||
rows, defects, files = [], [], 0
|
||||
for path, b in sources(a.binary):
|
||||
r, d = scan_file(path, b)
|
||||
rows += r
|
||||
defects += d
|
||||
files += 1
|
||||
|
||||
if a.list:
|
||||
for r in sorted(rows, key=lambda x: (x['binary'], x['fn'])):
|
||||
print(f"{r['binary']} {r['fn']}")
|
||||
sys.exit(0)
|
||||
|
||||
by_cls = collections.Counter(r['cls'] for r in rows)
|
||||
by_bin = collections.Counter(r['binary'] for r in rows)
|
||||
sdk = [r for r in rows if r['fn'] in SDK_NAMES]
|
||||
game = [r for r in rows if r['fn'] not in SDK_NAMES]
|
||||
|
||||
print(f"ASSEMBLY POSING AS C — {len(rows)} function(s) across {len(by_bin)} binaries "
|
||||
f"({files} sources scanned)\n")
|
||||
for c, n in sorted(by_cls.items()):
|
||||
print(f" {c:24s} {n:5d}")
|
||||
print(f"\n PsyQ/CRT SDK routines : {len(sdk):5d} [{SDK_PROVENANCE}]")
|
||||
print(f" GAME CODE — real decompilation work remaining : {len(game):5d}")
|
||||
print(f"\n by binary:")
|
||||
for b, n in by_bin.most_common():
|
||||
print(f" {b:16s} {n:5d}")
|
||||
|
||||
# every row carries WHICH detectors claimed it — a row claimed by only one is weaker evidence
|
||||
weak = [r for r in rows if len(r['detectors']) == 1 and r['cls'] == 'A-FILE-SCOPE-VERBATIM']
|
||||
if weak:
|
||||
print(f"\n {len(weak)} row(s) claimed by a SINGLE detector (weaker evidence — spot-check these):")
|
||||
for r in weak[:8]:
|
||||
print(f" {r['binary']:14s} {r['fn']:24s} {r['detectors']} {r['path']}:{r['line']}")
|
||||
|
||||
if defects:
|
||||
print(f"\n!! {len(defects)} COVERAGE DEFECT(S) — a silent skip is a defect, not a no-op (R32):")
|
||||
for d in defects[:12]:
|
||||
print(f" {d}")
|
||||
else:
|
||||
print(f"\n coverage: every file-scope __asm__ block classified; 0 defects.")
|
||||
|
||||
if a.json:
|
||||
json.dump(rows, open(a.json, 'w'), indent=1)
|
||||
print(f"\n-> {a.json}")
|
||||
if a.strict and defects:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+51
-8
@@ -588,7 +588,7 @@ SIG = re.compile(r'^\s*[A-Za-z_][\w \t\*]*\b([A-Za-z_]\w*)\s*\(')
|
||||
KR_PARAM = re.compile(r'^\s*[A-Za-z_][\w \t\*]*\s+\**\w+\s*(?:\[[^\]]*\])?\s*;\s*$')
|
||||
|
||||
def classify():
|
||||
real, empty, nonmatching, stubs, blobs, linked = [], [], [], [], [], []
|
||||
real, empty, nonmatching, stubs, blobs, linked, verbatim = [], [], [], [], [], [], []
|
||||
for src in SRCS:
|
||||
lines = src.read_text().split('\n')
|
||||
n = len(lines); i = 0
|
||||
@@ -662,6 +662,42 @@ def classify():
|
||||
if m:
|
||||
blobs.append(m.group(1))
|
||||
i += 1; continue
|
||||
# §265 VERBATIM-ASM BODY — bytes, NOT a decompile, and it needs its OWN bucket (P31 S75).
|
||||
#
|
||||
# A file-scope `__asm__("...")` block that DEFINES a function (`.ent NAME` / `.globl NAME`
|
||||
# + `NAME:`) matches none of the patterns above, so it landed in NO bucket and surfaced as
|
||||
# `UNPLACED (parse hole)` — which is R32 doing its job, but leaves the operator with an
|
||||
# error instead of a number. Counting it REAL would be worse: it is the assembly pasted
|
||||
# into a C string, byte-identical BY CONSTRUCTION and completely unexplained.
|
||||
#
|
||||
# Measured fleet-wide at introduction: 116 such bodies (114 main, 1 md_MAIN_003,
|
||||
# 1 ov_SC05_005), the largest being SaveLoadRoutine at 1,165 instructions. Several main
|
||||
# ones are PsyQ library routines where verbatim is defensible; the point of the bucket is
|
||||
# that the GAP between "byte-identical binary" and "decompiled source" is now visible in
|
||||
# the headline instead of hiding inside REAL or inside a parse hole.
|
||||
if s.startswith('__asm__') and '(' in s:
|
||||
blk, j, depth, opened = [], i, 0, False
|
||||
while j < n:
|
||||
c = lines[j]
|
||||
blk.append(c)
|
||||
depth += c.count('(') - c.count(')')
|
||||
if '(' in c: opened = True
|
||||
j += 1
|
||||
if opened and depth <= 0: break
|
||||
txt = '\n'.join(blk)
|
||||
# BOTH SPELLINGS, AND REQUIRE THE `\n` TERMINATOR (P31 S75, second cut).
|
||||
# The sources use `".ent\tNAME\n"` AND `".ent NAME\n"`, and a bare fragment
|
||||
# `".ent\t"` also occurs. A pattern with an OPTIONAL `\t` and no terminator matched
|
||||
# the fragment and captured the literal name `t`, while a pattern requiring `\t`
|
||||
# missed every space-spelled block: three of my own counts (116 / 112 / 108) were
|
||||
# all wrong before this was validated against one known-true case of EACH spelling
|
||||
# (SaveLoadRoutine = tab, func_80047D3C = space). True count: 178.
|
||||
names = set(re.findall(r'\.ent(?:\\t|\s)+(\w+)\\n', txt))
|
||||
if not names:
|
||||
g = set(re.findall(r'\.globl(?:\\t|\s)+(\w+)\\n', txt))
|
||||
names = g & set(re.findall(r'"(\w+):\\n"', txt))
|
||||
verbatim.extend(sorted(names))
|
||||
i = j; continue
|
||||
fm = SIG.match(lines[i])
|
||||
if fm and '(' in lines[i]:
|
||||
# Definition ({ ... }) vs forward declaration (ends ;)? Scan to the first { or ;.
|
||||
@@ -702,13 +738,13 @@ def classify():
|
||||
(real if strip_comments(body[a+1:b]).strip() else empty).append(fm.group(1))
|
||||
continue
|
||||
i += 1
|
||||
return real, empty, nonmatching, stubs, blobs, linked
|
||||
return real, empty, nonmatching, stubs, blobs, linked, verbatim
|
||||
|
||||
def report(binary, audit=False, write=True):
|
||||
"""Classify one binary; write docs/progress.<binary>.md (if write) + print; return a stats dict
|
||||
(for --fleet aggregation). Single-binary output is byte-for-byte the legacy format."""
|
||||
set_binary(binary)
|
||||
real, empty, nonmatching, stubs, blobs, linked = classify()
|
||||
real, empty, nonmatching, stubs, blobs, linked, verbatim = classify()
|
||||
# Code-shared functions (dedup.us.yaml) are REAL byte-identical matches whose macro-instantiated
|
||||
# form classify() doesn't parse — fold them in (dedup-safe) so the count stays honest (P9).
|
||||
#
|
||||
@@ -727,7 +763,8 @@ def report(binary, audit=False, write=True):
|
||||
# how the K&R blindness above hid ~190k banked instructions for 26 phases while the byte-gate stayed
|
||||
# green (the gate compiles; this tool only reads text — they share no code, so the gate can never
|
||||
# catch a miscount). Report it LOUDLY rather than silently under-reporting progress.
|
||||
placed = set(real) | set(empty) | set(nonmatching) | set(stubs) | set(blobs) | set(linked)
|
||||
placed = (set(real) | set(empty) | set(nonmatching) | set(stubs) | set(blobs)
|
||||
| set(linked) | set(verbatim))
|
||||
unplaced = sorted(set(_S_INDEX) - placed)
|
||||
|
||||
# OVER-coverage is the DUAL defect, and the assertion above is blind to it (Phase-28 T5). `placed`
|
||||
@@ -736,7 +773,8 @@ def report(binary, audit=False, write=True):
|
||||
# hole (fixed above) put func_800D00E4 in BOTH `real` and `stubs`, reporting the resident as
|
||||
# 123/146 = 85.62% when the truth is 122/145 = 85.5% — and the flag-plant target is that very
|
||||
# denominator. R32 means assert coverage in BOTH directions: nothing missing, nothing counted twice.
|
||||
_buckets = {"real": real, "empty": empty, "nonmatching": nonmatching, "stubs": stubs, "linked": linked}
|
||||
_buckets = {"real": real, "empty": empty, "nonmatching": nonmatching, "stubs": stubs,
|
||||
"linked": linked, "verbatim": verbatim}
|
||||
_seen = {}
|
||||
for _name, _b in _buckets.items():
|
||||
for _fn in _b:
|
||||
@@ -745,8 +783,10 @@ def report(binary, audit=False, write=True):
|
||||
assert not _multi, ("progress.py: %d function(s) landed in MULTIPLE buckets — a MISCOUNT, not a "
|
||||
"no-op (R32): %s" % (len(_multi), dict(list(_multi.items())[:5])))
|
||||
|
||||
matchable = len(set(real) | set(empty) | set(nonmatching) | set(stubs) | set(linked)) # SET, not a len() sum
|
||||
byteident = len(set(real) | set(linked) | set(empty)) # all byte-identical in the build
|
||||
matchable = len(set(real) | set(empty) | set(nonmatching) | set(stubs) | set(linked)
|
||||
| set(verbatim)) # SET, not a len() sum
|
||||
# VERBATIM counts as byte-identical (it is, by construction) but NEVER as REAL.
|
||||
byteident = len(set(real) | set(linked) | set(empty) | set(verbatim))
|
||||
|
||||
out = []
|
||||
out.append("# BFM matching progress (generated by tools/progress.py — authoritative)")
|
||||
@@ -755,6 +795,9 @@ def report(binary, audit=False, write=True):
|
||||
if shared:
|
||||
out.append(f" (of which dedup-shared : {len(shared):5d} one body -> N sites, config/dedup.us.yaml)")
|
||||
out.append(f"LINKED real PsyQ objects : {len(linked):5d} <- byte-identical via linked SDK objects")
|
||||
if verbatim:
|
||||
out.append(f"VERBATIM __asm__ bodies : {len(verbatim):5d} <- §265: BYTES, NOT A DECOMPILE "
|
||||
f"(byte-identical by construction, unexplained; NOT counted in REAL)")
|
||||
out.append(f"NON_MATCHING (near-miss) : {len(nonmatching):5d}")
|
||||
out.append(f"splat-auto empty no-ops : {len(empty):5d}")
|
||||
out.append(f"INCLUDE_ASM stubs : {len(stubs):5d}")
|
||||
@@ -767,7 +810,7 @@ def report(binary, audit=False, write=True):
|
||||
out.append("-" * 40)
|
||||
out.append(f"matchable functions : {matchable:5d}")
|
||||
out.append(f"REAL / matchable : {len(real)} / {matchable} = {100*len(real)/matchable:.2f}%")
|
||||
out.append(f"byte-identical/ matchable: {byteident} / {matchable} = {100*byteident/matchable:.2f}% (REAL+LINKED+empties)")
|
||||
out.append(f"byte-identical/ matchable: {byteident} / {matchable} = {100*byteident/matchable:.2f}% (REAL+LINKED+empties+VERBATIM)")
|
||||
out.append("")
|
||||
out.append("LINKED subsegs: " + " ".join(sorted(LINKED_SEGS)) + f" ({len(linked)} fns)")
|
||||
out.append("REAL matches: " + " ".join(sorted(real)))
|
||||
|
||||
Reference in New Issue
Block a user