Christopher Williams
7090663275
MERGE 10: +4 bodies -> 639 bodies / 648 regions. All four re-verified by me from fresh --work dirs
with --symbols first: 0x80099E34 (40 B), 0x800AC9D8 (56 B), 0x800460AC (40 B), 0x80045540 (56 B).
Gate: c_regions=648 differing_bytes=0 result=MATCH, SHA-1 unchanged. make check exit 0.
The tool, and why it exists. The Phase 12 re-dispatch put all four workers on the ~193-row
classified-negatives index. I generated those pools with a one-off script, and that script was
wrong in FIVE ways. Every one of them shipped, and **every one was found by a worker, not by me**:
1. The index documents its own schema on line 2 -- `# Columns: address<TAB>size<TAB>status<TAB>class`
-- and my script **never read `status`**. 5 rows `blocked` + 1 `blocked,deferred` are rows
charter section 7 forbids attempting (trapping arithmetic, the 0x80012xxx primitive-init family,
the maspsx rare-epilogue mutual exclusion). **Six forbidden rows went out at prio 1, the top of
the list.** Worker C found three in its own pool and refused them.
2. `status` is not enough either: `0x80012A48` is status `near-match` with class `primitive-init
scheduler-bound family`, so it reached prio 1 as well. Blocked families are now matched on the
CHARTER's own wording, in both the status and the class column -- 7 rows dropped.
Deliberately NOT blocked: `rare-epilogue-ORDER`. Worker A drew that line precisely (the order is
work; only the maspsx mutual EXCLUSION is blocked), and cookbook 140/147/165 shipped
`maspsx=epilogue` for it.
3. The Makefile's `NAMED_EXCLUSIONS` (12 rows) was invisible to my script: **8 of the 12 leaked back**
as fresh work -- $gp-switch thunk halves, a fragment, false extent starts. Worker A found two and
asked whether they should be filtered. Eight were leaking, not two. The list now lives in ONE
place (`NAMED_EXCLUSIONS`), and both `worklist` and the new `negpool` target are given it.
4. The workers' own `negatives.tsv` staging was invisible: **94 of the 115 rows they had already
classified were served as fresh work, at the TOP of the prio-1 order.** Worker C: "they are the
first four rows in the file's own prio-1 order, so a worker starting at the top spends its first
hours re-deriving my floor." 58 rows dropped as classified-with-no-mechanism (a second worker
re-deriving a row that already yielded nothing is pure duplication), and a worker's OWN
classification is dropped from that worker's own pool (12 rows) while still being dispatched to
the others, who get the recorded mechanism as a lead.
5. `named = class not in ('-', '')` is a test for a NON-EMPTY STRING, not for a mechanism. Worker D's
staging labels 56 of its 92 rows `no-mechanism-yet` and 6 more `no-extent`, so **62 rows were
ranked prio 1, "cheapest, mechanism already named", when their label says the exact opposite.**
That is why worker A's pool led with rows it could not close. `names_a_mechanism()` now rejects
absences, numbers and bare status words, and accepts prose (workers A and C write their class as
a sentence, so a token-only rule would have missed their mechanisms).
This is a SCHEMA error, not a logic error, and it is worth naming as such: five separate bugs all
trace to generating a dispatch file from a table whose header I had not read.
One coordinator aside that belongs in the record: I had already "fixed" defect 3 by hand, adding
`--exclude 0x8001DC20` to the `worklist` target -- and **the counter caught me**:
`excluded_named_exclusion` did not move, because that row was already excluded from the worklist
under a different rule and had reached the worker through the negatives pool instead. Fixing one row
by hand while the generator leaked was treating a symptom, and the tool said so.
Pools: 176 unregistered -> **93 rows** (a 24 / b 23 / c 28 / d 18), every exclusion and blocked row
gone. C's pool now leads with `index-only` rows, i.e. genuinely fresh work.
Tests: 17 new, `tools/tests/test_sf3_negpool.py`. Fail-first is demonstrated rather than asserted --
the two INTEGRATION tests (which read `NAMED_EXCLUSIONS` out of the Makefile rather than hard-coding
addresses, because the whole defect was two lists that must agree living in different files) were run
against the pools the workers were holding at that moment and **failed, naming all 8 leaked rows**.
Two further bugs found in the process: my own `names_a_mechanism` accepted the numeric dashboard
columns (`'1'` is not a mechanism), which made the rule vacuous and dropped 0 rows instead of ~62; and
one of my tests asserted output that its own `--quiet` flag suppressed.
199 lines
9.0 KiB
Makefile
199 lines
9.0 KiB
Makefile
# Phase 3 all-assembly baseline orchestration.
|
|
# All original and generated ROM-derived inputs/outputs remain ignored.
|
|
|
|
SHELL := /usr/bin/bash
|
|
|
|
EXE := extracted/SCUS_946.40;1
|
|
MANIFEST := extracted/MANIFEST.tsv
|
|
SPLAT := tools/splat/.venv/bin/splat
|
|
AS := tools/mipsel-none-elf-binutils/prefix/usr/bin/mipsel-none-elf-as
|
|
LD := tools/mipsel-none-elf-binutils/prefix/usr/bin/mipsel-none-elf-ld
|
|
OBJCOPY := tools/mipsel-none-elf-binutils/prefix/usr/bin/mipsel-none-elf-objcopy
|
|
|
|
ASM_ROOT := asm
|
|
SPLAT_CONFIG := $(ASM_ROOT)/scus_946_40.yaml
|
|
SPLAT_OUTPUT := $(ASM_ROOT)/generated
|
|
LINKER_SCRIPT := $(ASM_ROOT)/scus_946_40.ld
|
|
BUILD := build
|
|
ELF := $(BUILD)/scus_946_40.elf
|
|
REBUILT := $(BUILD)/scus_946_40.rebuilt
|
|
INPUT_REPORT := .run/p3-exe-info.tsv
|
|
|
|
# Phase 5 ordered code build: C regions listed in the tracked registry are
|
|
# compiled with the identified toolchain and linked in address order between
|
|
# data fallbacks taken from the original executable. An empty registry
|
|
# reproduces the all-payload data baseline exactly.
|
|
REGIONS := config/regions.tsv
|
|
SYMBOLS := config/symbols.tsv
|
|
INVENTORY := config/function_inventory.tsv
|
|
EXTENTS := config/function_extents.tsv
|
|
DUPES := config/duplicate_bodies.tsv
|
|
NEGATIVES := config/near_match_negatives.tsv
|
|
WORKLIST := config/match_worklist.tsv
|
|
MATCH := tools/sf3_match
|
|
EXTENTS_TOOL := tools/sf3_extents
|
|
DUPES_TOOL := tools/sf3_dupes
|
|
TRIAGE_TOOL := tools/sf3_triage
|
|
NEGPOOL_TOOL := tools/sf3_negpool
|
|
|
|
# Rows deliberately excluded from dispatch, kept in ONE place: a previous phase excluded each for a
|
|
# recorded reason (false extent start, shared tail, unmatchable $gp-thunk half, fragment), and the
|
|
# reason is in the comment block above `worklist`. Both `worklist` and `negpool` are given this
|
|
# list, because the Phase 12 pools were first built by a one-off script that did not know about it
|
|
# and **8 of these 12 rows were delivered straight back to the workers as fresh work**.
|
|
NAMED_EXCLUSIONS := 0x8005DEF8 0x800F3160 0x80108034 0x8010804C 0x80012A10 0x80012AE0 \
|
|
0x80012CFC 0x80012B20 0x8008A198 0x8010080C 0x800C3490 0x8001DC20
|
|
|
|
# Worker staging for the classified-negatives pools. Empty in a clean checkout, which is correct:
|
|
# the pools then contain only the rows the tracked index names.
|
|
NEGPOOL_DIR := .run/p12
|
|
NEGPOOL_CLASSIFIED := $(wildcard .run/p12/w-*/negatives.tsv)
|
|
CODE_OUT := build/code
|
|
EXPECTED_SHA1 := e173426c157384ebf1b6caf8c6fea18a85a14af9
|
|
|
|
.PHONY: all validate split assemble link binary code gate extents extents-verify dupes worklist negpool test check clean
|
|
|
|
all: binary
|
|
|
|
validate:
|
|
@test -f "$(EXE)"
|
|
@test -f "$(MANIFEST)"
|
|
@mkdir -p .run
|
|
@./tools/sf3_extract psx-exe-info "$(EXE)" "$(MANIFEST)" > "$(INPUT_REPORT)"
|
|
|
|
split: validate
|
|
@test -f "$(SPLAT)"
|
|
@test -f "$(SPLAT_CONFIG)"
|
|
@cd "$(ASM_ROOT)" && ../"$(SPLAT)" split "$$(basename "$(SPLAT_CONFIG)")" --disassemble-all
|
|
|
|
assemble: split
|
|
@test -d "$(SPLAT_OUTPUT)" && test ! -L "$(SPLAT_OUTPUT)"
|
|
@test -f "$(AS)"
|
|
@test ! -L "$(BUILD)"
|
|
@mkdir -p "$(BUILD)"
|
|
@while IFS= read -r source; do \
|
|
rel="$${source#$(ASM_ROOT)/}"; \
|
|
object="$(BUILD)/$${rel}.o"; \
|
|
mkdir -p "$$(dirname "$$object")"; \
|
|
"$(AS)" -march=r3000 -G0 -I "$(ASM_ROOT)/include" -o "$$object" "$$source"; \
|
|
done < <(find "$(SPLAT_OUTPUT)" -type f -name '*.s' -print | sort)
|
|
|
|
link: assemble
|
|
@test -f "$(LINKER_SCRIPT)"
|
|
@test -f "$(LD)"
|
|
@rm -f "$(ELF)"
|
|
@"$(LD)" -T "$(LINKER_SCRIPT)" -o "$(ELF)"
|
|
|
|
binary: link
|
|
@test -f "$(OBJCOPY)"
|
|
@rm -f "$(REBUILT)"
|
|
@"$(OBJCOPY)" -O binary "$(ELF)" "$(REBUILT)"
|
|
|
|
# Ordered code build and full-binary gate (Phase 5). `code` builds only;
|
|
# `gate` builds and then requires the whole executable to be byte-identical.
|
|
code: validate
|
|
@test -f "$(MATCH)"
|
|
@test -f "$(REGIONS)"
|
|
@test -f "$(SYMBOLS)"
|
|
@rm -rf "$(CODE_OUT)"
|
|
@"$(MATCH)" build --exe "$(EXE)" --regions "$(REGIONS)" --symbols "$(SYMBOLS)" --out "$(CODE_OUT)"
|
|
|
|
gate: validate
|
|
@test -f "$(MATCH)"
|
|
@test -f "$(REGIONS)"
|
|
@test -f "$(SYMBOLS)"
|
|
@rm -rf "$(CODE_OUT)"
|
|
@"$(MATCH)" gate --exe "$(EXE)" --regions "$(REGIONS)" --symbols "$(SYMBOLS)" --out "$(CODE_OUT)" \
|
|
--expect-sha1 "$(EXPECTED_SHA1)"
|
|
|
|
# Function extents (Phase 7). `extents` regenerates the tracked table from the
|
|
# inventory; `extents-verify` re-derives it and requires every registered region
|
|
# to agree, so a region can never silently disagree with the derived extent.
|
|
extents: validate
|
|
@test -f "$(EXTENTS_TOOL)"
|
|
@test -f "$(INVENTORY)"
|
|
@"$(EXTENTS_TOOL)" scan --exe "$(EXE)" --inventory "$(INVENTORY)" --out "$(EXTENTS)" --force
|
|
|
|
extents-verify: validate
|
|
@test -f "$(EXTENTS_TOOL)"
|
|
@test -f "$(INVENTORY)"
|
|
@test -f "$(EXTENTS)"
|
|
@"$(EXTENTS_TOOL)" verify --exe "$(EXE)" --inventory "$(INVENTORY)" \
|
|
--extents "$(EXTENTS)" --regions "$(REGIONS)"
|
|
|
|
# Duplicate-body census (Phase 7): groups byte-identical bodies so a shared body
|
|
# is matched once and registered once per address.
|
|
dupes: validate
|
|
@test -f "$(DUPES_TOOL)"
|
|
@test -f "$(EXTENTS)"
|
|
@"$(DUPES_TOOL)" census --exe "$(EXE)" --extents "$(EXTENTS)" --out "$(DUPES)" --force
|
|
|
|
# Ranked match worklist (Phase 7). The --exclude addresses are recorded
|
|
# near-misses or known-bad rows deferred by developer direction; naming them here
|
|
# keeps the exclusion visible instead of burying it in the tool.
|
|
# 0x8010080C was added in Phase 10 as a FALSE EXTENT START (worker C): its first
|
|
# instruction is `beq s0,zero` with s0 never written in range, and the walk split
|
|
# one real 252-byte function (0x801007E0..0x801008DC) at a spurious `jal`-target
|
|
# boundary. It is now caught earlier by sf3_triage's general `restores_unsaved`
|
|
# rule (worker A's independent scan re-derived the same row from a different
|
|
# signal), so this row is retained only as the provenance record for that defect.
|
|
#
|
|
# 0x800C3490 was added in Phase 12 as a FRAGMENT, not a function (cookbook 114,
|
|
# found by worker A and re-confirmed by the coordinator against the registry). It
|
|
# starts mid-expression, its body is a SHARED TAIL (`addiu sp,sp,48; jr ra`) that
|
|
# also appears at 0x800C3470-0x800C348C, and it cannot be matched standalone. The
|
|
# extents table still grades it `exact` with `term=jr_ra`, because the boundary walk
|
|
# sees a well-formed terminal -- so the tool cannot catch this one. Excluded rather
|
|
# than left in the dispatch files, where it costs every fresh-band worker a reading
|
|
# budget before it is recognised again. Worker A identified it for the second time in
|
|
# Phase 12 and skipped it rather than spending that budget.
|
|
#
|
|
# 0x8001DC20 was added in Phase 12 as the SECOND instance of the same class, and it
|
|
# gives the class a CHECKABLE predicate instead of a vibe. Its first instruction is
|
|
# `lw t0,0(t5)` (0x8DA80000) and **$13 is written ZERO times in the whole 72-byte
|
|
# extent** -- t5 is read but never established, so the extent is the tail of a larger
|
|
# function. Worker D found it and refused to spend GTE spellings on it; the
|
|
# coordinator re-derived the same conclusion from the bytes before excluding it, so
|
|
# this row is a two-worker confirmation like 0x800C3490 is.
|
|
# The general predicate -- "a caller-saved temporary ($8-$15) is READ before any
|
|
# instruction in the extent WRITES it" -- holds for BOTH rows (`0x800C3490` also
|
|
# writes $13 zero times) and is sound, because $8-$15 are never incoming o32
|
|
# arguments and no compiler emits a read of an uninitialised temporary. It is NOT
|
|
# yet implemented as an automatic triage rule; until it is, fragments are caught by
|
|
# a worker reading the row. See the Phase 12 ledger for that open item.
|
|
worklist: validate
|
|
@test -f "$(TRIAGE_TOOL)"
|
|
@test -f "$(EXTENTS)"
|
|
@test -f "$(DUPES)"
|
|
@"$(TRIAGE_TOOL)" plan --exe "$(EXE)" --extents "$(EXTENTS)" --inventory "$(INVENTORY)" \
|
|
--census "$(DUPES)" --regions "$(REGIONS)" --negatives "$(NEGATIVES)" \
|
|
$(foreach a,$(NAMED_EXCLUSIONS),--exclude $(a)) --out "$(WORKLIST)" --force
|
|
|
|
# Classified-negatives worker pools. Regenerating these is cheap and idempotent; handing a worker a
|
|
# pool that a row was ALREADY removed from is not, so this target owns both filters:
|
|
# * $(NAMED_EXCLUSIONS) -- rows dispatch must never offer (same list as `worklist`)
|
|
# * the workers' own classification -- a row already read this phase, kept at prio 1 only if
|
|
# it NAMES a mechanism, dropped otherwise.
|
|
# See the tools/sf3_negpool docstring for why both rules exist and what they cost when absent.
|
|
negpool: validate
|
|
@test -f "$(NEGPOOL_TOOL)"
|
|
@test -f "$(EXTENTS)"
|
|
@excl=""; for a in $(NAMED_EXCLUSIONS); do excl="$$excl --exclude $$a"; done; \
|
|
cls=""; for f in $(NEGPOOL_CLASSIFIED); do cls="$$cls --classified $$f"; done; \
|
|
"$(NEGPOOL_TOOL)" --negatives "$(NEGATIVES)" --regions "$(REGIONS)" --extents "$(EXTENTS)" \
|
|
--exe "$(EXE)" --out-dir "$(NEGPOOL_DIR)" --prefix negatives --workers 4 \
|
|
$$excl $$cls
|
|
|
|
# Verification gates. `test` is synthetic-only and needs no game input;
|
|
# `check` adds the extents check and the full-binary byte gate (which do).
|
|
test:
|
|
@PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests
|
|
|
|
check: test extents-verify gate
|
|
|
|
clean:
|
|
@if test -e "$(BUILD)"; then \
|
|
test -d "$(BUILD)" && test ! -L "$(BUILD)"; \
|
|
rm -rf "$(BUILD)"; \
|
|
fi
|