MERGE 10: +4 bodies -> 639 bodies / 648 regions. All four re-verified by me from fresh --work dirs
with --symbols first: 0x80099E34 (40 B), 0x800AC9D8 (56 B), 0x800460AC (40 B), 0x80045540 (56 B).
Gate: c_regions=648 differing_bytes=0 result=MATCH, SHA-1 unchanged. make check exit 0.
The tool, and why it exists. The Phase 12 re-dispatch put all four workers on the ~193-row
classified-negatives index. I generated those pools with a one-off script, and that script was
wrong in FIVE ways. Every one of them shipped, and **every one was found by a worker, not by me**:
1. The index documents its own schema on line 2 -- `# Columns: address<TAB>size<TAB>status<TAB>class`
-- and my script **never read `status`**. 5 rows `blocked` + 1 `blocked,deferred` are rows
charter section 7 forbids attempting (trapping arithmetic, the 0x80012xxx primitive-init family,
the maspsx rare-epilogue mutual exclusion). **Six forbidden rows went out at prio 1, the top of
the list.** Worker C found three in its own pool and refused them.
2. `status` is not enough either: `0x80012A48` is status `near-match` with class `primitive-init
scheduler-bound family`, so it reached prio 1 as well. Blocked families are now matched on the
CHARTER's own wording, in both the status and the class column -- 7 rows dropped.
Deliberately NOT blocked: `rare-epilogue-ORDER`. Worker A drew that line precisely (the order is
work; only the maspsx mutual EXCLUSION is blocked), and cookbook 140/147/165 shipped
`maspsx=epilogue` for it.
3. The Makefile's `NAMED_EXCLUSIONS` (12 rows) was invisible to my script: **8 of the 12 leaked back**
as fresh work -- $gp-switch thunk halves, a fragment, false extent starts. Worker A found two and
asked whether they should be filtered. Eight were leaking, not two. The list now lives in ONE
place (`NAMED_EXCLUSIONS`), and both `worklist` and the new `negpool` target are given it.
4. The workers' own `negatives.tsv` staging was invisible: **94 of the 115 rows they had already
classified were served as fresh work, at the TOP of the prio-1 order.** Worker C: "they are the
first four rows in the file's own prio-1 order, so a worker starting at the top spends its first
hours re-deriving my floor." 58 rows dropped as classified-with-no-mechanism (a second worker
re-deriving a row that already yielded nothing is pure duplication), and a worker's OWN
classification is dropped from that worker's own pool (12 rows) while still being dispatched to
the others, who get the recorded mechanism as a lead.
5. `named = class not in ('-', '')` is a test for a NON-EMPTY STRING, not for a mechanism. Worker D's
staging labels 56 of its 92 rows `no-mechanism-yet` and 6 more `no-extent`, so **62 rows were
ranked prio 1, "cheapest, mechanism already named", when their label says the exact opposite.**
That is why worker A's pool led with rows it could not close. `names_a_mechanism()` now rejects
absences, numbers and bare status words, and accepts prose (workers A and C write their class as
a sentence, so a token-only rule would have missed their mechanisms).
This is a SCHEMA error, not a logic error, and it is worth naming as such: five separate bugs all
trace to generating a dispatch file from a table whose header I had not read.
One coordinator aside that belongs in the record: I had already "fixed" defect 3 by hand, adding
`--exclude 0x8001DC20` to the `worklist` target -- and **the counter caught me**:
`excluded_named_exclusion` did not move, because that row was already excluded from the worklist
under a different rule and had reached the worker through the negatives pool instead. Fixing one row
by hand while the generator leaked was treating a symptom, and the tool said so.
Pools: 176 unregistered -> **93 rows** (a 24 / b 23 / c 28 / d 18), every exclusion and blocked row
gone. C's pool now leads with `index-only` rows, i.e. genuinely fresh work.
Tests: 17 new, `tools/tests/test_sf3_negpool.py`. Fail-first is demonstrated rather than asserted --
the two INTEGRATION tests (which read `NAMED_EXCLUSIONS` out of the Makefile rather than hard-coding
addresses, because the whole defect was two lists that must agree living in different files) were run
against the pools the workers were holding at that moment and **failed, naming all 8 leaked rows**.
Two further bugs found in the process: my own `names_a_mechanism` accepted the numeric dashboard
columns (`'1'` is not a mechanism), which made the rule vacuous and dropped 0 rows instead of ~62; and
one of my tests asserted output that its own `--quiet` flag suppressed.
Worker D found it while sweeping the GTE batch, refused to spend spellings on it, and reported it
rather than grinding. I re-derived the conclusion from the bytes before acting: its first
instruction is lw t0,0(t5) (0x8DA80000) and $13 is written ZERO times in the whole 72-byte extent,
so t5 is read but never established and the extent is the tail of a larger function. The
already-excluded 0x800C3490 has the identical signature (also zero writes to $13).
That gives the fragment class a CHECKABLE predicate instead of a vibe: 'a caller-saved temporary
($8-$15) is READ before any instruction in the extent WRITES it.' It is sound because $8-$15 are
never incoming o32 arguments and no compiler emits a read of an uninitialised temporary. It is not
yet an automatic triage rule -- that needs a fail-first test, so it is recorded as an open item
rather than half-implemented at the context cap. Until then fragments are caught by a worker
reading the row, which has now cost two workers a reading budget each.
Excluded rather than left in the dispatch files: 0x800C3490 was re-identified twice by two separate
workers before this.
Worker A's third claim, verified from a fresh --work dir against the exact md5 in the claim,
merged to a candidate, gated whole-binary, promoted only on result=MATCH. It is the largest body
any worker has taken this phase (572 B).
sf3_match gate c_regions=630 differing_bytes=0 result=MATCH
sha1 e173426c157384ebf1b6caf8c6fea18a85a14af9 (unchanged)
make check exit 0 make extents-verify regions=630 disagreements=0 AGREE
make worklist listed=989, excluded_already_registered=630
registry audit 630 rows, 621 distinct sources, 0 missing, 3 carry cc1bin
**ADJACENCY IS NOW 3-FOR-3 FOR WORKER A, ACROSS THREE BANDS.** 0x800297F4 (204 B, band 0) ->
0x800298C0 (392 B, band 1) -> 0x80029A48 (572 B, band 2): three CONSECUTIVE bodies, 1168 bytes,
four spellings total, with the density ranker not involved in any of the three picks. The chain
ends where no row starts at the next boundary. That is the strongest single piece of dispatch
evidence in the phase, and it is why the fresh-band files were re-cut to put adjacency above
density inside a band.
Worker A's lever, recorded because it is an operator RECOVERY rule rather than a spelling tip: the
original is `if ((x1 < 0 && x2 > 0) || (x1 > 0 && x2 < 0))`, and the truth-table-equivalent
if/else ladder is EXACTLY ONE INSTRUCTION SHORT. The emitted stream is `bgez x1` / `bgtz x2` /
`blez x1` / `bgez x2` -- each term's first test branches over its own second test -- and the ladder
has no `bgez x1` to emit. So the four BRANCH SENSES let you write the operator down without
guessing; cookbook 83's "branch direction distinguishes && from ||" turned into a recovery rule.
And `(x1 ^ x2) < 0` is the same predicate with the wrong codegen: the original compares.
**0x800C3490 IS EXCLUDED FROM THE WORKLIST.** Cookbook 114 (worker A, Phase 10) records it as a
FRAGMENT: it starts mid-expression, its body is a SHARED TAIL (`addiu sp,sp,48; jr ra`) that also
appears at 0x800C3470-0x800C348C, and it cannot be matched standalone. The extents table still
grades it `exact` with `term=jr_ra` because the boundary walk sees a well-formed terminal, so the
tool cannot catch it -- which is exactly why it needs a recorded exclusion rather than a tool rule.
Worker A recognised it for the SECOND time in Phase 12 and skipped it instead of spending reading
budget, which is the signal that the exclusion belongs in the Makefile: a row that has to be
re-identified by hand every phase is a row the dispatch should not be offering.
Added with its provenance in the Makefile comment, the same mechanism Phase 10 used for
0x8010080C's false extent start. excluded_named_exclusion 8 -> 9.
Ledger updated with merges 2-5, the cc1bin lever and its gate, the mechanism correction, the gp
pair rewrite and both of my failures in it, the dispatch measurement (369 of 993 rows abut SOME
region = noise; only 5 abut a PHASE-12 match = the signal), and the partition-membership fix.
+2 bodies (worker C claims 19-20) and 3 gp symbol rows (D_80121F90,
D_8012196E, D_80121964). Candidate gate MATCH before promotion.
NEW EXCLUSION CLASS `restores_unsaved` (worker A, coordinator-verified). A body
that restores a callee-saved register it never saves cannot be a whole function:
the register it restores was established by an enclosing prologue that the derived
extent cut off. These are jal targets INSIDE a real function, so the walk began
mid-body — distinct from bad_extent_start, which flags starts that are not
function entries at all.
Verified disjoint from the matched corpus before acting, as the project requires:
0 of 462 registered regions trip the rule; 11 worklist rows do.
excluded_restores_unsaved=11; worklist 1265 -> 1253 rows. 5 new tests; suite
229 -> 237, OK. make check green: regions=462 AGREE, differing_bytes=0 MATCH.
Independent corroboration worth recording: the new rule re-derives 0x8010080C,
the false extent start worker C reported earlier via a completely different signal
(the first instruction reads a register the range never defines). Two independent
detections of one defect class. bad_extent_start is False for that row, confirming
C's observation that the older rule missed it. Its Makefile --exclude entry is
retained only as the provenance record for that defect.
+8 bodies from 432 (440 total, from 400). All gated on the candidate whole-binary
before promotion. SHA-1 e173426c157384ebf1b6caf8c6fea18a85a14af9 stable.
Registry option granted: gp=-D_801226F4 on 0x8002D364 (worker B). The first
candidate gate for this batch DIFFed (differing_bytes=1117197) because worker B
staged the option in a separate request file rather than in the claim row, so the
region merged without it; the tracked registry was never touched and the gate
caught it. Folded the option into the claim and re-gated to MATCH. Recorded as a
protocol event: a worker's region-option request must be folded into the claim
row before merging.
Evidence model: 0x8010080C is a FALSE EXTENT START (worker C, coordinator-verified
against the payload). Its first instruction is beq s0,zero and s0 is never written
in the range, so it is mid-body. The walk split one real 252-byte function
(0x801007E0..0x801008DC) at a spurious boundary because 0x8010080C is a jal target
from 0x8011419C. The row was unclaimable and is now a named exclusion
(excluded_named_exclusion 8 -> 9) with the reason in the Makefile and the
extents-defect record in the ledger. Fixing sf3_extents itself is left as a
bounded candidate, not attempted here.
Worklist 1304 rows, excluded_already_registered=446 (= registry size).
Partitions refiltered against it: a=441 b=432 c=431, disjoint, union==worklist.
Fourth failure (0x80012CFC, 54 bytes — worst residual of the cycle) against
two successes in one family confirms C's scheduler-bound diagnosis. Family
is a toolchain limit, not a source problem: 0x80012A10/0x80012AE0 (earlier
excluded) plus now 0x80012CFC/0x80012B20. Extensive arithmetic checks
(5 stack args fix a 9-parameter signature; lui immediates 0x08000000 and
0x38000000 are high halves) confirm the model without the bytes.
Worker B measured that recorded negatives cluster at the head of every
partition: 39 of the first 40 worklist rows and 57 of 271 tier-1 rows (21%)
were addresses an earlier session had already attempted and recorded. The
triage tool never read the negatives index, so every closed negative
returned to the queue head and workers rediscovered them.
Fix: sf3_triage reads the tracked config/near_match_negatives.tsv (a new
--negatives input) and excludes those rows as recorded_negative=69. The
tracked index was grown from 38 Phase-8 rows to 94 by importing 56 Phase-9
negatives from the workers' reports and the coordinator's absorption record
(filtered to unregistered; tracked rows win on conflict). Worklist
regenerated: the head is now genuinely fresh (0x800F9134, 0x80042D88,
0x800450C4, ...). 3 new synthetic tests; plan-level tests wired for the new
argument. The phase plan said extract negatives only at close; B's
measurement showed the harm is per-cycle, so the index is now a living
tracked input. Note for P9-T8: keep importing worker negatives each cycle.
31 new regions merged and gated MATCH (whole-binary, SHA-1
e173426c157384ebf1b6caf8c6fea18a85a14af9), make check green
(regions=246 disagreements=0 AGREE, c_regions=246 MATCH).
Worker A took the tier-2 lead the rotation pointed at: 29 new framed
wrapper/call bodies incl. jalr-indirect rows, argument-swap wrappers, and a
statement-order-around-side-effect fix (0x800198C0). New levers recorded:
lui/lw symbol address arithmetic check (carry-adjusted %hi), named-local
ordering around a side-effect store.
Worker C closed two retry rows from the negatives index: 0x80036AD8
(commutative-operand-order, the 0x8002DEB4-class) and 0x800B5CB4 (deferred
--g_80122068 family hypothesis, using the gp symbols staged in cycle 1).
Worklist regenerated with the 0x80012A10-family exclusion (worker C
recommendation): --exclude 0x80012A10 --exclude 0x80012AE0, pool 1,666,
named exclusions 5; partitions re-proved disjoint (556/555/555).
Worker B rotated out (budget) with a clean handoff: 13 promoted claims, 18
negative entries in report.tsv, four named cheap closes for fresh context,
and a P9-T5 lead (trapping arithmetic in un-extented gaps, verified by the
coordinator: 275 gap words vs 661 in-extent words). No replacement session
available; the coordinator absorbs partition B per the plan's fallback.
Control record updated with both verified cycles, the two per-region overrides
that came out of worker findings (maspsx=off and gp=-NAME, with the byte evidence
for each of the six anomaly-class regions), and the coordinator findings: the
stale worklist caught before dispatch, the candidate-first merge that contained a
bad cycle, Ghidra's COP2 pseudo-op collapsing, and the two $gp thunk halves.
Also excludes 0x80108034 and 0x8010804C from the worklist: they are the two halves
of a $gp-switch thunk and are not matchable as C regions.
Milestone met at 109 distinct bodies (target 70).
Phase 6 chose match targets by eye from the boundary inventory. tools/sf3_triage
now ranks every eligible candidate by (tier, size, address) from tracked inputs
alone and records why everything else was excluded.
Eligibility: an exact or fallthrough extent, a non-degenerate body, not already
registered, not the header entry, not named by --exclude. indirect, escape,
outside, runaway, contained and standalone are excluded and counted.
Tiers: 0 duplicate-group representative (one match, several addresses), 1 exact
leaf (no cross-references, so no symbol rows), 2 exact non-leaf, 3 fallthrough.
Result: 1916 listed (tier 0: 9, tier 1: 509, tier 2: 1394, tier 3: 4), with 252
degenerate bodies, 88 low-confidence grades, 12 registered, 1 header entry and
2 named near-misses excluded. The nine tier-0 entries are the real duplicate
groups: matching those nine bodies registers 22 function addresses.
The two deferred near-misses are excluded by name in the Makefile so the
exclusion stays visible rather than buried in the tool. 160 synthetic tests pass.
Matching conventions require a duplicate check before registering, because a
shared body is matched once and registered once per address. Phase 6 did that
check by hand and found one 12-byte pair. tools/sf3_dupes now hashes every
derived extent body and groups exact duplicates.
Results: 2284 extents, 65 multi-address groups, 2104 singletons. Only 10 groups
contain code (24 addresses, all exact-graded); 55 are all-zero bodies. The
hand-found pair 0x800262E0/0x800262EC is reproduced as g0002, which is the check
that the census measures what it claims. The largest real groups are 712 bytes
(0x8001084C/0x800189E8) and 436 bytes.
The zero groups are a real finding: 252 extents have all-zero bodies, 245 inside
the zero band 0x80147000..0x80170000. The cause is the inventory's jal grade,
which decodes every word as an instruction -- in a data region a word with
opcode 3 is graded as a call whose target lands in the zero band. The census
flags those groups rather than hiding them, and the worklist must exclude
degenerate bodies.
The census is tracked rather than ignored as the plan said, because it holds
addresses, sizes and grades only (the same class as the tracked inventory and
extents tables) and the worklist must be reproducible from tracked inputs. The
content hash is computed and never written.
Phase 6 graded function starts and left every end to be derived by hand. This
adds tools/sf3_extents, which explores all reachable control flow from each hard
start (jal/entry) and reports an extent plus how far it can be trusted.
Measured decisions, not stylistic ones:
- Soft starts are not walk boundaries. A body's second instruction can satisfy
the prologue grade exactly (0x800152AC is lw v1,8(gp) / addiu sp,sp,-176, so
0x800152B0 looks like a start). Enforcing soft boundaries stopped 155 of 416
walks inside a real body.
- The walk is a full reachability computation, not a first-terminal search: a
function whose paths return at different addresses must report the whole body.
Grades: exact 1940 (1666 packed, gap=0), fallthrough 256, indirect 73,
escape 15, contained 153, standalone 438; 63.8% of the payload covered.
Verification: all 12 registered regions reproduce exactly (make extents-verify,
now part of make check), 29 new synthetic tests (115 total), byte-identical
across two runs, and Ghidra's independent body for FUN_80017ad4 agrees. Two
defects were caught by writing the tests first and are recorded: reach had to be
an exclusive end, and a terminal j's delay slot must not continue linearly.
The table holds addresses, sizes, grades and site addresses only -- no bytes.
P5-T6. Adds docs/MATCHING_COOKBOOK.md (nine byte-proven compiler/assembler
findings, each with basis and limit), docs/MATCHING_CONVENTIONS.md (what counts
as a match, registry format, source naming, duplicate sharing, symbols, the
verification procedure, the firewall) and docs/PHASE5_VERIFICATION.md (outcome,
clean gates, bounded negatives, milestone request).
Tooling: make test and make check targets; region objects are now
symbol-localized so the documented duplicate-sharing mechanism (N registry rows
-> 1 source) links, covered by a synthetic test (53 tests).
Clean gates: 53/53 tests; make clean/all/cmp/SHA-1 green; make gate with one C
region -> c_regions=1, 0 differing bytes, SHA-1 e173426c...; 57 tracked files, 0
under any prohibited root. Phase 5 is not closed until the milestone is
confirmed.
P5-T4: tools/sf3_match provides range/plan/build/gate. It compiles a C candidate
with the identified toolchain (egcs-2.91.66 psx, -O2 -G0 -mno-split-addresses),
compares the exact instruction range byte-for-byte against the original, and
builds the address-ordered executable from the tracked registry config/regions.tsv
(header 0x800 + data gaps + C regions, LMA 0x800, metadata discarded).
Promotes the Phase 4 ordered workflow from an ignored experiment to tracked
tooling. Verified: make gate with an empty registry and with one real C region
both reproduce SHA-1 e173426c...; the Phase 3 baseline is unchanged and green;
the synthetic suite is now 50 tests including deliberate-corruption failures and
safe refusal of an existing destination. No ROM-derived material is tracked.
Christopher Williams