mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
8d548ec2604fcb1efe7141f087900aeb3fccd94a
71 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
0488903e57 |
feat(tools): decl_from_use — negative-controlled extern inference for the no-seed-decl class (Fable agent build, verified); SETUP.md rows for it + the baseline-red gate input (R21)
42-case first run: 39 were one uncompilable TU (ov_SC04_018_jr_8017AE2C.c), not draft defects — the resolver's 'undeclared' classification needs a TU-alone compile probe first (open follow-up). |
||
|
Drew T
|
433c7cd8ef |
feat(resolver): the integration-resolver lane — zero-token re-judging of the ledgers' shape-correct stock at the real TU (P31 S61 T10+/S61-1)
frontier-analysis-s60 §4 measured that ~571 open functions had FINISHED drafting (closeness-0 backlog rows / reloc shape-MATCH rejects) and were being re-drafted wave after wave. tools/integration_resolver.py treats those ledgers as an index: still-open? -> rtu_match at the real split TU (CC1: the gate ladder's draft-side transforms, one retry) -> reloc_identity as the disagreeing oracle (rtu masks reloc fields) -> aprop_symfix on MISMATCH/shape-MATCH -> stage -> sweep_parallel (whole-binary SHA, sole arbiter) -> commit at once (R42). Refuses main by name (gate_main owns it), //@EDIT drafts, dirty trees, collapsed registries; every drop is counted (R32); a negative control over recently-banked functions must pass N/N before any verdict is trusted (R35/R39 — its first form picked carve moves as banks, 9/12 FAIL, and was fixed before a single stock verdict was read). Ledger .run/resolver/verdicts.jsonl keyed by (binary, fn, draft-sha, split-TU-sha) so unchanged rejects are never re-judged. First pass (commit:2991): 1,352 nominated -> 901 already banked, 27 main -> 424 judged in 41 s -> 245 staged (57.8%; 242 raw, 3 via transforms) -> 63 banked (net INCLUDE_ASM delta; that commit's subject says 72 = gross incl. 9 carve moves), 182 gate-refused, zero model tokens, ~10 min total. Lane wrapper tools/lanes/resolver_lane.sh (holds .run/auto/draw.lock for judge+gate: rtu reads the TUs a gate splices into). |
||
|
Drew T
|
6f5eb90844 |
docs(ops): correct the truncation numbers — the denominator was wrong
I reported 240 of 244 turn-finishes truncated in wave bk and called it ~100%. That count came from grepping lines containing 'finish=', which api_agent only prints when a turn ends WITHOUT a tool call — so the denominator was not all turns, it was all anomalous turns. I compared a subset against itself. Counting every turn: bk (8k / 420s): 240 truncated of 3,222 turns = 7.4% bt (16k / 700s): 16 truncated of 1,210 turns = 1.3% Still a ~6x improvement and the change stands on its own evidence, but it is a tax reduction, not the collapse I described. Recorded alongside: a truncated turn is one turn of 24, not a lost agent — the logs show the agent emitting its tool call on the very next turn. Both docs keep the wrong figure explicitly, with why it was wrong, so the next reader does not re-derive it from the same grep. |
||
|
Drew T
|
b6ba6ab7df |
docs(ops): SETUP + runbook carry the probed campaign constants
SETUP gains rows for recover_rejects.py (free recovery of the 45% of drafts that never reach the gate, 13% of which are a deterministic symbol rebase), restart_main_lane_when_idle.sh, and a campaign-constants row recording MAXTOK 16000 / HTTP_TIMEOUT 700 with the measurements behind them: reasoning_tokens=0 so the output cap was the reasoning cap, 240 of 244 turn-finishes truncated at 8k, an uncapped hard prompt wanting 8,067 tokens, ~30 tok/s, and the 1M-context / 131,072-max-completion model ceiling that makes 16k our choice rather than a limit. The runbook gains the same table plus the ordering rule (generation < HTTP_TIMEOUT < stallguard's 1200s kill), the evidence that turn caps are NOT binding on the default lane, one-lane-one-band with the size table that retired the 120-2000 slot, the maintenance lane's new recovery job, and the main-lane restart helper. |
||
|
Drew T
|
df5c1a1b15 |
docs(ops): rewrite the automation runbook for the campaign as it actually runs
The runbook was dated 2026-06-22 and described the reach-1 grinder pivot — it named no lane that exists today, two months and an entire toolchain later. Rewritten around what is actually running: the six lanes and their restart rules, the OpenRouter drafting toolchain (cards, LEVER_CRIB, per-lane budgets, the draw's admit/refuse census, quotas as floor AND ceiling), the banking toolchain by binary class (sweep_parallel vs gate_main vs the gate-time jtbl carve vs -O0 objects), the main lane, the distill lane and the flywheel's measured yield, the rate/credit numbers with their denominators, and recovery. Two ops laws are stated where they will be read rather than rediscovered: bash parses a while-loop up front (so code, args and draw-defaults each take effect differently), and never pkill -f a lane by a bare name because it matches the harness's own wrapper. SETUP.md gains rows for main_lane, the distill lane and the three restart helpers. |
||
|
Drew T
|
9e44612c2a |
feat(jtbl): the carve→draft→bank pipeline is automated at the gate — byte-proven ×3, one per class (S59)
The carve stays INSIDE the byte-gate (harvest_verify._jtbl_prep_one, the §61b-proven
order); everything new routes work to it:
* jtbl_carve: island_probe (read-only classifier: tail/covered/island-end/island-blocked/
island-pads/main-manual), --island-split (the one-line §260 insert, end-adjacent only),
and apply() now recognizes a completed island split as a no-op success instead of the
historical refusal.
* harvest_verify: _ISLAND_WALLS branch — on the §154-A refusal the gate isolates (body
still spliced), inserts the split line, re-extracts, re-carves. Snapshot-restore covers it.
* jtbl_lane.py (new): probe → draft (--draft-dir or api_agent) → gate via the exact
sweep_parallel worker call, HOLDING the campaign draw lock across gate+commit → commit
named per-binary paths. One jtbl target per gate invocation (§61c).
* build_wave_atlas: probe filter + one-jtbl-card-per-binary cap (inert unless
--levers jtbl-carve). idiom_serial: refuses the jtbl-carve lever (R43; its pre-carve ran
the refuted order — S58: 8 attempts, 0 banks).
Proven end to end with the live campaign running (banks committed separately):
commit:2661 ov_SC03_014/func_8017DCC0 tail: §8b adjacent merge + §8e pad recovery,
jr_8017AE2C.o .rodata 0x14→0x28 TIGHT, sha d84b01a2 green
commit:2663 md_SC03_076/func_801F218C covered: §260 STAGE 2 — sha 9a165e36 identical
with the table COMPILER-EMITTED; the first md_* jr bank ever
commit:2664 md_SC03_135/func_801E5358 island-end: the FULL split done BY THE GATE on a
virgin module (~1.1s, R40-checked), sha b901fda5, md.o 0x27c→0x268 + jr.o 0x14
Census (245/245 members probed, R32): 181 members / 26,445 ins reachable unattended;
main 47 parked (gate_main cadence), island-blocked 10 (stack order), island-pads 6
(needs §8e pads for modules.mk), no-jtbl 1 (atlas mislabel).
Design + failure semantics + campaign hook: docs/tool-designs/jtbl-automation-s59.md;
cookbook §260-A.
|
||
|
Drew T
|
e9ac4db9ba |
fix(phase-31): wire the S56 findings into the tools, SETUP.md and memory
Audit after "did you update tooling for those findings" found three gaps
beyond the four committed fixes:
- R21: docs/SETUP.md had no record of any of it. Adds the "four flow traps"
table — each trap, its measured cost, and where it is now caught.
- family_sweep --only had no coverage assertion, so the 3-vs-50 mis-scope
could recur silently. It now resolves member addrs to their family, always
prints the coverage line, and refuses when it resolves to zero families.
- pregate_check modelled the banking driver's typedef strip but never checked
the consequence. Adds [DROP-RISK] §203 USE-BEFORE-TYPEDEF. R39 NC: flags
the known-bad splice, 0 false positives on the post-hoist file and all 7
other wave-Z TUs (first draft read a typedef named in its own comment).
Also corrects §203 and the phase log: the text that banked was the RENAMED
variant, because gate_stage's backlog.save_draft() overwrote the original on
the failed attempt. Bytes are correct (R22 213/213); the claim "draft
byte-unchanged" was not.
|
||
|
Drew T
|
7ba1ae96d4 |
chore(phase-31): wire gap_triage into the harvest prompt, SETUP.md and the wave-closing memory
Caught by Drew: the tool was committed and nothing called it — not the harvest prompt, not the memory, not SETUP.md. A tool nobody invokes is the same defect this session keeps finding in the CARDS (§193-A, §194-E, §196: the answer was already computed and nothing carried it), wearing tooling clothes. Now: step 0 of the harvest reader prompt runs it; docs/SETUP.md carries a row with the measured threshold rationale (R21); the wave-closing memory names it in step 6. All three restate the same caveat because it is the part that gets forgotten — a candidate is a STARTING POINT, and an empty candidate list is NOT evidence of novelty, since dumb term overlap misses paraphrase entirely. |
||
|
Drew T
|
e18738c48e |
feat(phase-31 T5): THE FRONTIER ATLAS — 5,139 lever-labeled crack groups over all 12,058 open fns
- tools/atlas.py: cousin units baseline + T1.5 h_seqn merges + CALIBRATED warm tier (measured: li-norm metric holds ~99% recall to 0.55; rule = smallest t with neg-accept<=0.2% AND recall>=95% -> THRESH_WARM=0.70 @ 99.1%/0.18% — false merges waste exemplar cracks, misses only route cheaper) + seed sweep (65% of open skeletons carry a >=0.55 matched seed) + kNN graph + tiny-direct + evidence joins (audit/backlog/ledgers/cards; unparsable=fatal) + lever labels with confidence measured>ledger>tell>default>UNKNOWN - partition ASSERTED: 12,058 = progress stubs 12,051 + NM 7 EXACTLY (chased the +1: data blobs now excluded, reconciled against classify() buckets; T1 banks confirmed absent); every instance in exactly one group; main joins at the atlas layer only (family maps stay non-main — 4 silent-skip hazards) - warm tier merged 1,019; top group unifies 268 drifted per-location skeletons - lever table: head-crack 186.9k ins / UNKNOWN 138.6k (honest) / extend-tell 76.7k / redraft 46.8k / jtbl-carve 45.7k / integration 23.4k / seeded 23.4k / len-vein 16.8k / swaprepeat 9.2k / plumbing 8.1k / o0 6.6k / cc1 6.4k - atlas_features: li_norm_toks exported (shared with atlas, R33; hash-stable); mid_jr verifier fixed (compared ZERO rows — R32 silent no-op; now 6,444/6,444) - make atlas = full regen chain (~10-15 min, zero tokens); --targets emits crack slates (12/12 .s resolved); survey 92 s - SETUP rows (R21); docs/frontier-atlas.md committed |
||
|
Drew T
|
22eed7d78a |
feat(phase-31 T3): main enablement — sig-main at splat-true lengths + main streams
- sig_image: --seeds accepts '0xADDR NINS' (and jsonl nins); a seeded nins is
authoritative ([addr, addr+4*nins), bypasses func_end whose heuristic mis-sliced
3/40 main samples); R32 guard on seeded end > hi
- corpus: s_ins_count() factored from audit() (R33, one counter) + '--seed-ends'
CLI emitting per-stub splat-true lengths
- make sig-main: 2,002 main stubs signed -> .run/sig.main.jsonl; FULL word
cross-check 2,002/2,002 EXE slices == .s words (0 SLICE-SUSPECT; .s word field
is byte-order hex, not LE — first checker draft misread 1,999 false suspects).
Deliberately splat-SEEDED; main's independent second oracle stays deferred
(second-oracle.md; sig_is_independent(main) stays False)
- family_remap: vram_of/img_path special-case 'main' derived from splat.us.exe.yaml
(file0-vram = code-seg vram - start = 0x8000F800; target_path); stream_words
('main') verified 25/25 vs .s
- regression: sig-resident re-run byte-identical after the shared read_seeds change
- SETUP §6.3 rows (R21)
|
||
|
Drew T
|
fa425f89a5 |
feat(phase-31 T2): reference expansion — PsyQ SDK sample source extracted + gcc-2.7.2 completed
- PsyQ 4.0 dev-CD Track-1 walked with the frozen iso9660.py (R33): 2,374 files / 231.6 MB -> tools/reference/psyq-sdk/ (gitignored); 400 C sources incl. the full PSX/SAMPLE tree (373) — drafting-agent style priors, esp. for main - provenance find: GNU/SNGNUVER.TXT = SN's gcc build history (2.7.2.SN32.3.7.0002) naming SN's exact patches vs vanilla; only UNROLL.C codegen-relevant - gcc-2.7.2 reference +6 files (calls.c + caller-save.c — §172-cited, previously missing — + integrate/optabs/varasm/recog), tarball sha256 recorded - docs/psyq-sample-idioms.md seeded; SETUP §5.6 rows (R21) |
||
|
Drew T
|
e903713b71 |
feat(phase-30 S50): the mechanical A-prop draft — 256 members banked with no agent in the loop
Final S50 state: 307 instances banked, stubs 12,468 -> 12,161, fleet 95.3% instr / 90.0% distinct / 96.65% fn-count. R22 clean rebuild 4x, check-all 213/213 every time. - tools/aprop_autodraft.py + tools/draft_prechecks.py: seed body + symbol_map + a MINIMAL synthesized preamble. The seed's decl layer never travels — that layer is family_sweep's dominant failure (331 of 458 S49 verdicts). 256 banked at zero agent tokens, against the ~20M the same work would have cost as a wave. - Macro seeds (567 of 1196 members, all 3737 de-macroize) take the DEFINITION only; the block stays the decl source. Pasting it whole measured 28% vs inline's 68% — func_8016AB6C's macro is 1,891 lines of which 108 are the function. - IMM is a second engine, not a wall: T2a's imm_map_tier1 resolves a per-location LITERAL like symbol_map resolves a per-location SYMBOL. 131 of 275 IMM members resolve. - draft_prechecks negative-controlled against ALL 205 banked drafts: zero false positives, catches 39 of 67 known failures. That control found two bugs in the checks themselves — C89 `f()` declares UNSPECIFIED parameters (not zero), and a member's own definition read as a call to itself. Conservative by design: a pre-check that discards good drafts is worse than one that lets a few builds fail. - The A-prop pool is now priced exactly: PURE 437/37,376 ins, IMM 275/8,849, STRUCT 238/4,259. - Cookbook §171a; SETUP rows; CURRENT_PHASE S50 FINAL checkpoint. |
||
|
Drew T
|
2f7e06564b |
feat(phase-30 S50): the stale seed symbol — A-prop's 57% was ours, not the lane's (§171)
- REFUTES §170's open hypothesis (batched cards concentrate members into one TU ⇒ §169
collision): 5-draft groups banked 5/5; 11 of 35 unbanked drafts were already one-per-TU;
and the two "concentrated" groups banked 12/12 and 10/10 once the real defect was fixed.
- The cause: a per-location data symbol carried out of the seed body unrebased. match_one
compares instruction ENCODINGS and is blind to a relocation's target NAME, so it scores
MATCH standalone and dies at link in the host TU. 24 of 24 concentrated failures, all 1:1
rewritable at one constant vram delta (0x4128).
- tools/aprop_symfix.py: audit + --fix, emits a gate_lane-shaped slate; deterministic and
build-free, so it runs BEFORE the gate. The R34 second oracle for the class match_one
cannot see.
- family_cousins.py --aprop-cards: members now carry sym_map, the explicit {seed -> member}
renames, read from the seed's C BODY (a matched seed has no .s of its own) vs the member's
.s. Two case-mismatch defects fixed while wiring it (sig lowercase vs splat uppercase).
- 23/24 banked. Stubs 12,468 -> 12,445. Fleet 95.2% instr / 89.9% distinct / 96.57% fn.
R22 clean rebuild: check-all 213 passed, 0 failed of 213. dedup 2,043/0.
- A-prop's true conversion is 87% (79/91); the 320 batched members are unblocked.
- Cookbook §171 + §170 struck in place; SETUP row; decision-log (R31).
|
||
|
Drew T
|
b3713cc3ca |
feat(phase-30 S49): the cousin tier — family_cousins.py similarity map + seeded wave-7 slate (§168)
- FINDING (Drew's smell, byte-verified): the '4,513 unique singletons' picture is substantially an h_seq exact-hash artifact — 86/120 near-pairs in the 0.85-0.99 band differ by PURE insertion/deletion (li-expansion tell in 25). Specimen: ov_SC06_010:0x8017bebc (753 ins, 'singleton') is 0.987-similar to a MATCHED fn in the same binary. - NEW tools/family_cousins.py: distinct open skeletons -> shingle index -> >=0.85 union-find -> matched-seed attachment -> .run/family_cousins.json + docs/family-cousins.md. R32 BOTH ways (independent stub recount fails loud on a stale map — negative-control-proven; partition assert). Reproduced the probe within +-1%; totals EXACT (11,627 inst / 584,448 ins). - Unit table: A-prop 197u/68,729ins · seeded 418u/50,422 · cousin-multi 1,552u/249,799 · cold 3,240u/215,498 — the genuinely-unique tail is 37% of the remainder, not 90%. Main's 'structurally barren' HOLDS at the similarity tier (94% mass <0.70). - --targets wave slate: .run/wave7_targets.json = 40 targets / 33,304 unit ins (+33% vs family-ranked), 9 resolved seed C paths, size-routed 2 haiku/20 sonnet/18 opus. - LAWS (§168): a cousin is a SEEDED CRACK never a remap; rank waves by UNIT weight; discount short-fn similarity. Byte-gate stays the sole arbiter (G3/P9). - docs/family-hseq.md: this session's frontier regen (post-S48 propagations) rides along. - cookbook §168 + SETUP inventory row (R16/R21/R30); CURRENT_PHASE S49 entry. |
||
|
Drew T
|
9351b17f48 |
feat(phase-30 S46-2): the master IDXTAB/DESTPTR load map — and the tracker blind spot that hid it
Drew's S45 idea, delivered fleet-wide + wired into the permanent references.
- THE BLOCKER WAS OUR INSTRUMENT (R35, the 3rd time): the S45 plan ("require a
register-verified reference to the run's address") returns ZERO for both byte-proved
tables. They are read by gcc's indexed global-array form —
lui $at,0x8019 ; addu $at,$at,$a0 ; lh $v0,-0x2844($at) -> 0x8018D7BC
— where the address exists only as (lui imm, LOAD offset) with the index add between.
find_addr_refs killed the lui register at the addu, so the halves never rejoined and
the tables looked unreachable. Now it carries the hi half through the index add (still
strictly register-tracked, never window-paired) and labels those hits `-indexed`.
- tools/idxtab_map.py (NEW): fleet-wide payload -> owning binary -> load address.
Controls-gated (refuses to emit unless ov_SC01_000 0x8017EEC8/37 + *0x801A3234, and
ov_SC03_001 0x8018D7BC/5 + *0x801EBC68 reproduce from the images alone). Index space
DERIVED from the extracted tree (reproduces §S44's table independently). Process-pooled.
Rejects all-zero and majority-zero runs (132 of the first pass's 452 "tables" were that).
- RESULT: 213 binaries -> 143 with a referenced table (294), 141 with a DESTPTR (141/141
resolved from the binary's OWN image), 61 payloads. The two dominant tables are
fleet-wide CONSTANTS (5-entry and 37-entry, identical in all 141 overlays); the
per-binary variable is the destination (134 distinct).
- CORRECTION 1 (R14): §S45 p6's "the SC03 trio are owned by ov_SC03_001" is refuted —
that 5-entry table is identical in ALL 141 overlays. The byte-observed parts stand.
- CORRECTION 2 (P9): this route CANNOT settle MAIN/7+9. They are absent from all 294
tables — but so are MAIN/13/20/34/42/44, which are byte-proved to load. Absence here
means "not on this route", nothing more. Recorded so it is not re-derived as a finding.
- Confidence is stated per-claim in docs/idxtab-map.md: proven (controls) / high (283
fleet-wide-class tables) / low (3 named rare rows) / UNMEASURED (recall — no oracle
for "all tables" exists beyond the 2 controls).
- Wired in permanently: docs/idxtab-map.md (the how/when/limits), memory-map.md §S46,
cookbook §155c (the generalizable law: "no code references X" is a claim about your
DECODER until it is shown to recognise the forms the compiler emits), SETUP.md
tooling inventory (R21).
|
||
|
Drew T
|
a0f07d629e |
chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/ rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java + VerifyOverlay.java (ghidra_import_raw.sh is the live path) - reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green - SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol- window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21); disc-completeness Reproduce marked retired |
||
|
Drew T
|
1576570271 |
fix(phase-30 S1e): the distinct-code "regression" was a STALE DIGEST — alias lever ungated
The S38 checkpoint gated the phase's best lever ("do NOT scale the alias lever") on
distinct-code falling 89.3 -> 89.2. It never fell.
PROOF (each commit's metric recomputed from its OWN committed tree, 0 unresolved):
commit:1426 TRUE : instr 12394533 distinct 5022306 (77895 uniq)
commit:1426 COMMITTED: instr 12402412 distinct 5029324 (78025 uniq) <- stale
HEAD TRUE == COMMITTED: instr 12405402 distinct 5025082 (77952 uniq)
=> true delta 843->HEAD: instr +10869, distinct +2776 ins / +57 uniq. ALL ROSE.
The 843 digest was generated from a working tree still holding work REVERTED before the
commit landed (+7,879 ins / +130 uniq overstated) and never regenerated, so the next
HONEST digest read as a fall. => THE ALIAS LEVER IS UNGATED (scale it, §61 small batches).
Both recorded leads were wrong (R14): progress.py:423's SIG regex feeds fn-count ONLY
(neither weighted metric sees a C identifier — both derive matched = sig - corpus.stubs),
and "the harvest reverted functions to INCLUDE_ASM" died on one grep (483 removed, 0 added).
The 3-grep proof: identical sigs + unchanged tools/ + zero +INCLUDE_ASM => HEAD's stub set
is a strict subset => both numerators are FORBIDDEN to fall.
THREE INSTRUMENT DEFECTS, all one class (a bare except around a fail-CLOSED oracle):
- progress.py stub_addrs wrapped corpus.stubs in `except Exception: return set()`. An empty
stub set means "could not answer", not "no stubs", so matched = sig - stubs credited EVERY
function. Byte-witnessed: instr 100.00% / distinct 100.00% in a tree with no asm/. Now
propagates.
- cast_call_sites.tu_for + reconcile_tu.tu_for had the identical swallow, falling back to the
default <ov>.c instead of the jr/-O0 split TU — silently reinstating the exact bug
cast_call_sites' own docstring says it exists to fix. A wrong-TU reconcile fails the gate,
and this phase's base rate is ~24k PLUMBING vs 4,917 DIFF, so it presents as a codegen wall.
Now propagate CorpusError; ValueError fallback for curated names preserved; derived-TU path
re-verified (a _jr_ split stub resolves correctly, both tools agree).
NEW GATE (R34 — the byte-gate is a null oracle for DOCUMENTS; check-all stays 140/140 over a
stale digest forever): tools/audit_digest.py + `make audit-digest`, wired into tools-health
after report. Recomputes the three headline metrics from the current tree and fails if the
committed digest disagrees. Compares INTEGERS, not percentages — the +7,879-instruction
staleness printed as "94.4%" on both sides. Negative-control-proven against the stale 843
digest (fails, exit 1) and green on HEAD.
Verified: make report exit 0 (dedup-check 1910 validated / 0 failed, C1 coverage
241216/241216); audit-digest OK; cookbook-index OK (398 sections); metrics unchanged by the
fix (94.40% / 89.18%). No src/ or config/ edits — no bytes touched, nothing banked.
cookbook §140 · decision-log 2026-08-04 · SETUP.md inventory (R21) · R14/R32/R34/R35.
|
||
|
Drew T
|
19766a6dc3 |
feat(phase-30): T0.5 — prefetch_fleet.py, the fleet Ghidra-C batch orchestrator (+ SETUP row, R21; fuel manifest ride-along)
One representative per remaining h_seq distinct class + all main/resident stubs -> .run/ghidra_c/. Resumable (skips cached); serial on the exclusive project lock; auto-stops a serving MCP (R23); imports missing overlay programs on demand via ghidra_import_raw.sh (blob derived via family_remap.img_path, vram from the splat yaml — R33, never guessed); R32 per-program outcome report, continues past failures. Dry-run: 126 programs / 7,966 uncached representatives. |
||
|
Drew T
|
f285de46e8 |
feat(phase-29): T53 — fold the T51 lever into the gate; kill gather_externs' false positive; close a revert gap
Items 1-2 off T52's list, plus a third defect found by T53's own testing. TOOLING ONLY — banks
nothing; metrics unchanged by design (85.7% instr / 76.4% distinct / 90.65% fn-count).
1. THE T51 PRE-PASS IS A jtbl_family_bank STAGE (cookbook §103, AUTOMATED)
Order: raw -> scoped -> tu-scoped -> recovered -> reconciled. After the non-invasive stages (it
edits the TU outside the spliced body); BEFORE the recovery stages deliberately — those bend the
DRAFT and T48 measured both at +3 ins for this class, so they cannot succeed here. The stage
re-runs scope_data_fix against the SCOPED TU rather than reusing the raw body: composition-correct,
since the contested symbols no longer have a file-scope decl to be dropped against.
COUNTERFACTUAL, byte-gated on a reproduced blocker (ov_SC01_000 restored to its pre-T51 TU):
raw -> compile error (conflicting types)
scoped -> compiles, FAILS the byte check (§8d drops the decl -> the u8 CSE costs +3)
tu-scoped -> BANKED
That is the evidence the stage does the work — not T52's sweep, which ran on TUs T51 had already
scoped by hand.
Refactor note: a stage editing outside the spliced body must RE-FIND the splice point (the stub
offset indexes the ORIGINAL TU). Each stage now carries its base; every pre-existing stage passes
`orig`, where the re-search returns the identical span — same operation as before, by construction.
2. gather_externs' COMMENT-SCANNING FALSE POSITIVE — FIXED (cookbook §104)
It scanned RAW text, so a symbol named only in the draft's PROSE counted as referenced: the
func_80135D20 warning that fired on 137/137 and was right 0 times. Fix is a two-text discipline —
MATCH on cdecl._mask'ed text, EMIT by span from the ORIGINAL (a masked decl is all blanks, so
"just mask it" would splice whitespace). Same change closes a second, unobserved defect of the
class: a COMMENTED-OUT extern could be selected as the carried decl and spliced in as live code.
MEASURED as a no-op on output (R14): 20 (exemplar, sibling) draft pairs across 4 families, old vs
new -> 20 identical / 0 differing. Only the false warning changed.
3. UNPLANNED — A REVERT THAT DID NOT SURVIVE AN EXCEPTION (cookbook §105)
A wrong exemplar made remap_hseq raise AFTER the carve rewrote config/ and jr_isolate created a
region file; the exception propagated out of bank(), the revert never ran, and the tree kept a
rewritten carve config plus an UNTRACKED region file (git checkout -- src/ does not remove it).
In a 132-member sweep that residue rides into the next member's build. bank() is now a
revert-guaranteed wrapper around _bank(). Negative-control proven: the crashing invocation now
reports {'exception': 2} and leaves git status -- config/ src/ at 0.
"Revert on failure" != "revert on every exit"; the exits are success, gate-fail, refusal, and the throw.
GATES: R22 clean-fleet 140 passed, 0 failed of 140. tools-health OK (corpus 0 PHANTOM + 0 TRUNCATED,
cdecl, audit-binaries, report/lint/dedup 1886/0). 0 NON_MATCHING (G4).
HONEST COVERAGE GAP: no live end-to-end BANK through the refactored loop — all three big families are
137/137 and the only family with live stubs (0x80191c50) has no banked exemplar, so it refuses. The
counterfactual byte-gated the exact splice on all three candidates and the 2-member run exercised
construction/refusal/revert/tally; the next real family sweep is the true end-to-end validation.
MY ERRORS: invoked the sweep with a wrong exemplar+address for a cross-address family (an unmeasured
guess about a members file I had not read — it is what surfaced defect 3); and deleted last_err's
initializer while refactoring, which would have raised NameError on the first clean gate-fail.
cookbook §103 (AUTOMATED) + §104 + §105; SETUP inventory row updated (R21).
|
||
|
Drew T
|
a524142eb5 |
feat(phase-29): T51 — the fleet-wide decl-scoping tool; func_80135260's 132 siblings unblocked
Item 1 off the SESSION-23 list. T48 proved the lever by hand on the exemplar, T50 located the same
blocker in every sibling; this builds the tool, measures the population, applies it fleet-wide, and
gates it. It BANKS NOTHING — it removes the blocker. The sweep is the next task.
MEASURED BEFORE BUILDING (R35). The blocker census over all 132 still-stubbed siblings is perfectly
uniform: 132/132 carry it, 3 contested symbols each, EXACTLY ONE file-scope decl statement per
(TU, sym), ZERO file-scope references below the decl (so the deletion is always safe), 660
block-scope re-declarations needed.
THE TOOL: tools/scope_tu_externs.py — the TU-side complement of scope_data_externs.py (§8d). §8d
fixes the incoming DRAFT and has a give-up branch that DROPS the draft's own decl when the TU already
declares the symbol at file scope. Right when the types agree; fatal when the byte-true draft needs a
different one — which is exactly how 132 byte-true siblings gate-failed wearing a codegen wall's
costume. This moves the TU's OWN file-scope decl down into every later function that references the
symbol and lacks its own block-scope decl, then deletes the file-scope line.
FILE(u8 D_x) ... then BLOCK(u16 *D_x) below it -> conflicting types (the 132 failures)
(no file-scope decl) ... BLOCK(u8) ... BLOCK(u16 *) -> builds; each fn owns its own view
- contested set DERIVED, never hand-listed (R33): the remapped draft's block-scope D_ externs
intersected with the TU's file-scope decls above the splice point; --family does it per sibling
- built on cdecl.split_statements/_mask (R33), not a 7th regex: depth-0 spans (a fn definition
flushes at its closing '}' — a column-0 test is NOT a file-scope test, m2c emits goto labels at
column 0 inside bodies) + length-preserving comment/string masking. That masking is what kills the
comment-scanning false-positive class still open as item 3.
- REFUSES LOUDLY, never skips silently (R32): >1 file-scope decl above the splice point; a
file-scope statement below the decl referencing the symbol; an unlocatable body brace
- coverage asserted as a DELTA (R32): file-scope -1, block-scope +len(consumers). An absolute
"a block-scope decl exists" check would have passed VACUOUSLY — these TUs already carry ~18
legitimate block-scope decls of the same symbols
VERIFIED IN TWO STEPS (T48's structure — why a 132-file edit was safe to make):
1. the move ALONE on ov_SC01_000 -> make build -> 9052dc0e BYTE-IDENTICAL, then reverted
2. fleet-wide -> R22 clean-fleet (make clean && extract-all && check-all) -> 140 passed, 0 failed
of 140; make tools-health OK (corpus 0 PHANTOM + 0 TRUNCATED, cdecl, audit-binaries,
report/lint/dedup 1886/0). Metrics UNCHANGED at 85.5% instr / 76.1% distinct / 90.62% fn-count
— the correct result for a declaration-only change.
The diff is uniform to the line: all 132 files +11/-3. A second --family run reports 132
nothing-to-do, 0 refused (idempotent).
Deliberately NOT done: wiring this as an automatic jtbl_family_bank stage. That waits until the
sweep measures the payoff — folding an unproven pre-pass into the gate is the same unmeasured
premise this phase keeps catching.
Also preserves .run/near6/g5260_a.c (the T48 raw crack body, allowlisted) — the sweep may need it
for --raw.
cookbook §103 + SETUP tool-inventory row (R21).
|
||
|
Drew T
|
b8e7250c78 |
feat(phase-29): tools/symcheck.py — the pre-gate symbol-set guard (§67a)
Builds the guard SESSION-17 left as a TODO after the func_801463A0 `_s`-alias trap, where a draft invented extern aliases no symbol table defines, read MATCH under rtu_match, and could never bank. - diffs the symbols a draft's object references (reloc records) against the target .s's %hi/%lo/jal set; reports MISSING (invented-alias signature) and INVENTED separately. - fills a real hole: match_one/masked_diff compare relocation-MASKED words (object-vs-.s is symbol-agnostic BY CONSTRUCTION) and rtu_match COMPILES WITHOUT LINKING -- so both are structurally blind to this class. R34: a second oracle that can disagree with the first. - NEGATIVE-CONTROL PROVEN: with one data extern renamed to an invented alias, match_one reports the SAME 14 mismatched as the correct draft; symcheck exits 1 naming both symbols. - --c compiles via match_one so the pinned triple/flags can never drift (R33); or --obj. - applied to the live func_8014D820 close=14 draft: 12/12 symbols agree, so a match there will link cleanly -- the §65c class is ruled out for it in advance. - cookbook §67a + SETUP tooling-inventory row (R21). Necessary condition, NOT a match oracle: still finish on the whole-binary byte-gate (G3/P9). |
||
|
Drew T
|
ffca2e4803 |
feat(phase-29): T16.10 — the driver's SUCCESS path verified (free re-bank test); 2 defects fixed
- THE FREE TEST (cookbook §66): reverted func_801778A8's bank to its INCLUDE_ASM stub (stub state rebuilds BYTE-IDENTICAL 7ca772be — a faithful revert proves itself; needs `make extract` first, the R22 corollary) and re-banked it THROUGH recover_integration.py --commit --r22. pass1 1/1 -> exact restore -> pass2 1/1 -> commit commit:0928 -> R22 140/140 -> report.json. Bank confirmed from SOURCE (stub gone), never the report (§55b trap 4). EQUIVALENCE: git diff vs the pre-revert commit = ONE blank line (mine) -> the driver reproduced SESSION-16's state exactly. - DEFECT 1 (SAFETY, found by reading before firing): PROPAGATION is a fleet-tier write (dedup_propagate --auto-from -> src/shared/engine_core.h + up to 138 overlay .c) that was both UNDECLARED and the DEFAULT, so --max-tier binary still permitted the widest write in the toolchain. assert_write_set cannot catch it (it runs before the gate; under --commit git status is clean). FIXED up front: propagate now requires --max-tier fleet AND --r22, and is REFUSED after a demacroize stage (those banks are x1 by construction; --auto-from would re-macroize and undo them). Both refusals negative-control-tested, exit 1. The "standing hazard" is now a refusal. - DEFECT 2 (METRIC): gate_stage scraped the fleet % via a progress.py label that no longer exists -> fp=None -> 50 gate commits recorded "fleet None%". Now reads FLEET instr-weighted (legacy fallback + loud stderr warning if neither matches); parses 79.6. - STALE DIGEST (R14): docs/progress.fleet.md at HEAD disagreed with HEAD's own source by 45 in the dedup-shared column — generated during the §65g local_type trial whose edits were then reverted. Regenerated (reproduced identically in-gate + standalone); headline %s unaffected. - cookbook §66/§66a/§66b distilled in-session (R30); SETUP.md gains the missing recover_integration row (R21 debt). tools-health OK: corpus 0/0, cdecl green, audit-binaries 140 citizens, lint OK, dedup-check 1879/0. Fleet unchanged 79.6% instr / 67.7% distinct / 88.86% fn-count. |
||
|
Drew T
|
f1f6238251 |
feat(phase-29): T16.6/T16.7 — 14th bank via the generalized de-macroize; §65 distilled; R21 inventory debt cleared
- func_8012F40C banked (the callee-conflict variant): relaxing demacroize from "the draft's own function" to "any decl the DRAFT declares incompatibly" reaches macros that declare a CALLEE differently than the draft does (RotTransPers/RotTransSV). 14 banks total, R22 140/140. - THE ONE FAILURE, kept honest: func_8012F49C was rtu-MATCH but the whole-binary gate REJECTED it. rtu_match is relocation-masked, so a wrong call TARGET is invisible to it -- and this was a callee case, exactly where the mask hides the error. Trust rtu MATCH for self-decl corrections, distrust it for callee ones (§65c). Reverted its edits and re-banked only the winner rather than leave byte-neutral churn on matched code (§57a-4). - DISTILLED IN-SESSION (R30/R16/R31/R21): cookbook §65 + §65a-§65e (blast-radius tiers; the de-macroize escape and the §20 refutation; the rtu-vs-gate divergence; the existing-ladder baseline; two-oracle practice); decision-log entry with the HONEST multiple (~2.3x, not the projected 3.7x, and it lands on distinct-code not the display number); calibration.md measured table; SETUP.md rows for blocker_probe + demacroize PLUS the three the inventory was missing (lift_types, uniquify_type, fix_header_decl-as-retired). - Carried and NAMED, not dropped: 10 match_one-MATCH drafts still blocked by stacked classes, and the 11 `near` drafts which are unfinished drafts, not recovery fuel. |
||
|
Drew T
|
b3ab5c2253 |
feat(phase-29 Task-13A): deterministic residual classifier — the permuter's problem is TARGETING
The autopsy (hindsight-study §7) assumed the permuter loses for want of a mutation. Measured over the whole open backlog, it loses because it is aimed at work a search-closer provably cannot close. - NEW tools/residual_class.py: decide a near-miss's class FROM THE BYTES. Decodes each mismatching MIPS word -> (op-skeleton, register-fields, immediate); drift FIRST (one inserted insn inflates `closeness` by the tail length), then consistent-injective register map -> REGALLOC-PERM (§31 S11/RC-3), same-multiset-reorder -> SCHEDULE-REORDER, DELAY-SLOT, WIDTH/BRANCH-POLARITY/STRENGTH/ADDRESSING/IMM-OFFSET/IMM-VALUE. Every class routes to a BUCKET = which tool the failure wants. Uncovered opcode -> UNKNOWN, COUNTED (R32). 16 synthetic unit tests (test_residual_class.py). - NEW tools/autopsy.py: `collect` materialises the corpus Task-12's telemetry never filled (1 of 6,169 records had a residual) by recompiling every open draft through the EXISTING match_one path (R33) — 1,752 drafts in 21s at -j12. `report` -> docs/autopsy.md. - NEW corpus.o0_sources()/is_o0(): the opt-level oracle DERIVED from the Makefile's own -O0 rules, coverage-asserted. Scoring an -O0 target at -O2 makes the residual 100% artefact (the trap this phase hit four times). - R34 cross-check baked in: residual_class's closeness vs masked_diff.structured_diff's, asserted per row; 1,673/1,673 agree, 0 classifier errors. FINDING: of the 972 records the grinder's own filter admits, only 75 (7.7%) are permuter-shaped; 547 are structural and 348 are drafts that are not the function at all. ~92% of the daemon's CPU went where it could not win — the byte-grounded explanation of "7 banks all-time, all Phase 21, 0 since" (Phase-22 audit). grinder.candidates() now filters on the measured bucket (1,303 -> 78) and takes its directed profile from the measured class, not the logged label (91% carry none -> it ran on gcc defaults). Degrades to undirected if uncollected and says so; --no-targeting A/Bs it. Two measured corollaries (R14, not projections): - 699 records rank as near-misses at closeness up to 278 purely from a length artefact: un-attempted work misfiled as a backlog of hard functions -> new `redraft` bucket. - a 12-draft gate probe of the `integration` bucket banked 1/12 (11 PLUMBING), so the 306 prices Task 14's reconcile ladder rather than promising free banks. func_80167714 (104 ins, reach-134) banked x1, un-propagated by design (§55b). Two defects fixed forward: - masked_diff._common_typedefs() used ONE shared probe path, so parallel match_one processes clobbered each other: 14 of 1,752 drafts lost in a single 12-way run (0.8%), silently, in every parallel wave ever run. Now per-PID. - gate_stage.match_one_closeness never passed --o0 -> phantom residuals for every -O0 function, written straight into the backlog this autopsy reads. R22 clean-fleet: check-all 140 passed, 0 failed of 140; tools-health OK (dedup 1847/0, C1 234343/234343); 0 NON_MATCHING (G4). Flywheel captured in-session (R30/R31): cookbook §60, decision-log entry, SETUP.md inventory. |
||
|
Drew T
|
5e78dccf07 |
feat(phase-29 §8e): jtbl 8-align pad-spec filter — the 4-giant unblock tooling (fleet-neutral)
- ROOT CAUSE PINNED (the session-2 half-pin was INVERTED; both probes were vacuous, R35): cc1 emits .align 3 before EVERY jump table; maspsx passes it VERBATIM (the :435 'drop' is an inventory-only pass); as bakes the pad SECTION-RELATIVE; link placement was never guilty (SUBALIGN(2) + ALIGN(.,4) place 4-mod-8 carve starts tight). Merging originally-separate TUs fires an intra-TU align where the original packed tight -> +4 at rodata 0xCC -> image-wide %lo shift. Honest probes persisted: .run/probe_jtbl/ (verdict.md + objdumps). - NEW tools/jtbl_rodata_pads.py: post-maspsx filter replaces each rodata .align 3 with the ORIGINAL's exact pad bytes per a JTBL_PADS spec; fail-loud on table-count drift / non-align-3 / non-jtbl rodata content. Byte-proven: verbatim 0xE4 pad-at-0xCC -> filtered 0xE0 tight (= the merged carve span). - jtbl_carve.py: spec-aware same-subseg merge (gap 0 or 4-with-zero-payload-word; else NON-CONTIGUOUS -> isolate), interval-arithmetic pad specs (committed values CARRIED, never re-derived), JTBL_PADS target-var emission into overlays.mk + revert() restore + stale-.o invalidation; the false 'maspsx drops .align' docstring corrected (H5). - Makefile: $(if $(JTBL_PADS),| jtbl_rodata_pads.py ...) stage in build/src/%.o + file-scope empty default (env-shield). jtbl_family_bank.stub_file: duplicate-stub fail-loud (the earlier 'ladder failure' was a wrong-TU splice into a stale _a.c stub, byte-witnessed). - R22 clean-fleet WITH the fix wired: 140/140 byte-identical, tools-health green (dedup 1846/0, C1 234205/234205), ZERO new banks -- fleet-neutral by construction. - cookbook §8e (the jtbl alignment law) + §8a/§8a-pad corrections; decision-log R31 entry; SETUP.md tool row; .gitignore allowlist for the probe verdict artifacts. |
||
|
Drew T
|
427baba3bf |
feat(phase-27 T10): completion dashboard (main in the weighted metric) + the resident second oracle
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
|
||
|
Drew T
|
264fe6c115 |
feat(phase-27 T7): disc-completeness audit — onboard 4 hidden SC07 overlays (136->140) + the type sweep
The whole-binary byte-gate is structurally blind to code nobody onboarded (R34): check-all is
green over the onboarded set no matter what code sits unbuilt on the disc. This reconciles the
onboarded set against every code-bearing PAC payload.
- new_overlay.sh: optional [ENTRY] arg (default 0.4) reaches a non-0.4.dec payload. Onboarded
ov_SC07_{006,007,010,011} from 1.4.dec (they put graphics at PAC entry 0, the code overlay at
entry 1 — invisible to the 0.4 hardcode for a month). Each byte-identical (7ca772be / b3b95547 /
d7b5875d / 9885af74). FLEET 136 -> 140; check-all 140/140 (T2's pass==N re-baselined cleanly).
difficulty.py NOT in the insertion set anymore (it derives, T6) -> only 3 tool dicts touched.
- tools/disc_code_sweep.py: decode every payload (reusing sig_image.make_insn) and gate code on
BOTH valid>=0.90 AND jr_$ra density>=0.01. The jr_$ra gate is decisive: isValid() alone flags
389 false hits (type-0/2 structured data decodes ~100% valid but has ZERO returns); jr_$ra
separates code (~2.9-3.4%) from data (0.000%), validated on positive+negative controls.
- FINDING (docs/disc-completeness.md): type-4 location overlays are COMPLETE (138/138). All other
types are data EXCEPT type-1 = 40 code payloads, 1 onboarded (the resident), 39 HIDDEN
resident-class modules (mostly MAIN.CD/FILE_XXX/1.1). They load at UNKNOWN addresses (not the
shared overlay slot), so they are NOT mechanically onboardable — byte-verifying a build binary
needs its load address (P9), knowable only by runtime RE (the Phase-3 method). Deferred with
evidence, NOT force-onboarded at a guess.
- CONSEQUENCE: game-code TRUE 100% now spans 140 onboarded binaries PLUS ~39 type-1 modules
pending load-address RE. The roadmap assumed 136 — this is a real re-baselining (the +4 overlays
also add ~2.45 MB to the denominator; every family propagation is now x138). Flows to T10/T11.
- SETUP §6.3 tool inventory updated (R21).
|
||
|
Drew T
|
e0a0becfaa |
feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr
The plan named two defective regexes; the tree had SIX with complementary holes, each silently recording the resulting compile failure as "not a match" — a plumbing error wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32). - cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs; keeps draft-local types. lru_cached. - harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) -> unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1 stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… / CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer recorded byte-identically to a codegen miss. - masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33, cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one + p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over whitespace. Unblocks B4's func_8015C32C (redefinition of 's16'). - canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap). VERIFIED: - HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that was being discarded over whitespace). - unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped. - classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly. - all 5 edited tools import + AST-parse clean. - R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248 "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.) - SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the original asm; a wrong draft always changes bytes -> always fails SHA1). |
||
|
Drew T
|
ebdef9012b |
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired. |
||
|
Drew T
|
a33c6f85f5 |
docs(phase-26a): A11 — distill + close the tooling-integrity audit (26-A COMPLETE)
Closed the inserted half-phase. tooling-audit.md: DIAGNOSIS -> AUDIT-CLOSE LEDGER (A1-A10 outcomes + the payoff 66.5->68.6% instr + remaining/handoff); the "two rules" -> R32/R33/R34 crisp for P10 ratification at the Phase-26 PhaseEnd. decision-log: the A10 wall-re-test verdict (R31 -- the broken tools WERE the walls; the payoff was banked by the fixes; the closeness-0 residual is genuine; the real deliverable is the 3 rules + the derived-oracle pattern). SETUP: the A9d-A10 tool changes (R21). Cookbook §51 verified complete; LAW 3 tagged R34. Observables green: final R22 clean-fleet 136/136 BYTE-IDENTICAL; make report EXIT 0 (dedup 1840/0, C1 227211/227211 signed, lint_symbol_refs wired + passing); audit-corpus 0 slices; audit-cdecl green. Zero src/config changes this session. Phase 26 resumes at Task 7 (fresh session). |
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|
||
|
Drew T
|
1ab9905368 |
feat(phase-26): §8d scope_data_externs — the ×133 sweep blocker fixed; func_8015AE2C banked ×134
- ROOT CAUSE (R14 — the session-7 diagnosis was half right): the isolated region builds [ OK ] WITHOUT the body, so §8b isolation was never implicated. `family_remap.gather_externs` prepends carried decls at FILE scope; D_801812A4 is a fn-ptr dispatch table the sibling declares FOUR incompatible ways at BLOCK scope inside its own later functions, so the carried file-scope decl ESTABLISHES A GLOBAL THE TU NEVER HAD and every later block-scope extern must now agree with it. Byte-proven asymmetry: BLOCK(int)->BLOCK(struct*)->FILE(void*) builds; FILE(void*)->BLOCK(int) errors. It was the ONLY hard error in the build — all 27 carried function externs were fine raw. - THE FIX (demote, don't reconcile): tools/scope_data_externs.py emits a carried D_ extern at BLOCK scope inside the function body when the TU has no file-scope decl of it above the insertion point. Byte-neutral (an extern emits no code; type + access opcodes unchanged) and never worse than raw, so it needs no oracle, no type comparator, no fn-ptr parser. Restores fidelity — the original declares these symbols at block scope in exactly this way. Wired into jtbl_family_bank as the `scoped` stage: raw -> scoped -> recovered -> reconciled (scoped is the base for the later stages). - reconcile_decls is the WRONG instrument for this class, twice: its oracle answers "what does the FLEET call this symbol" when the question is "what can THIS TU see", and its DATA_DECL_LINE_RE cannot parse `extern void (*D_x[])(void *);` — silently skipping the very symbols that were failing (the phase's third silent-skip bug, after find_site braces + overlay_files splits). - R17 TRIAGE RULE, first real test, held: `conflicting types` = the compiler REFUSED TO COMPILE = a C front-end diagnostic = our Python. Reading cse.c/global.c would have taught nothing. - RESULT: func_8015AE2C (562 ins, reach 134) swept 133/133 siblings, 0 failures. R22 clean-fleet 136/136 BYTE-IDENTICAL (534 changed src files); dedup-check 1813 validated / 0 failed; 0 NON_MATCHING (G4). instr-weighted 63.0 -> 63.6%; distinct-code 39.1 -> 40.5% (+256 unique fns / +79,957 ins) — one core, ~0 agent tokens. - knowledge captured during the producing session (R30/R31/R21): cookbook §8d, decision-log 2026-07-13 session 8, SETUP tool-inventory row; CURRENT_PHASE session-8 checkpoint. |
||
|
Drew T
|
095a611e75 |
feat(phase-26): §8 jtbl-rodata tooling — overlay PoC proven (func_8012ACE0, R22 136/136)
- overlay jr-functions can now bank as C: gcc switch jump tables form a .rodata island at the overlay TAIL; carve a matched fn's jtbl into a dotted [.rodata, <code-subseg>] subseg + ld_interleave (data->rodata->data sandwich) places it byte-exact. cookbook §8a + SETUP. - tools/ld_interleave.py: --section .<binary> param (derives the <binary>_TEXT/DATA/RODATA/ DATA2/BSS symbol prefix); default .main = the EXE, byte-identical (backward-compat proven) - Makefile + config/overlays.mk: <bin>_JTBL_INTERLEAVE hook + a $(strip)-guarded extract branch (gotcha caught: a trailing #comment on the := left whitespace -> non-empty -> the branch misfired on resident with the EXE defaults) - PoC: func_8012ACE0 (25-ins jr-fn in ov_SC01_077) reconciled (canon_sig_reconcile) + banked BYTE-IDENTICAL d19c9580 -- the first overlay jr-function matched through the C pipeline - R22 FULL-FLEET clean rebuild: 136 passed, 0 failed (main 143dbb89 unaffected by the ld_interleave change); 0 NON_MATCHING in any default build (G4) - P9 findings: func_80159C84/func_8015444C (the 2 carried Fable5 jr bodies) are rtu_match FALSE-matches (incomplete jtbls: 52B vs 56B -> never bank); the maspsx "hang" scare was a truncated experimental-file artifact (real pipeline builds in ~1s) - metrics: distinct-code 39.1% (50,572 unique fns), instr-weighted 62.9% - NEXT: the ×134 automation (generate the per-overlay carve + template the reconciled body) |
||
|
Drew T
|
c0379f0738 |
feat(phase-25): progress.py --weighted — byte/instruction-weighted metrics (the honest headline numbers)
- weighted_metrics() from .run/sig.*.jsonl + src stubs (executable code only, resident + 134 overlays; main EXE excluded). Two framings: fleet instr-weighted (per-overlay, the decomp.dev -display number) + dedup distinct-code (each unique h_exact once, the distinct-RE number) - --fleet now emits THREE labeled metrics into docs/progress.fleet.md: fn-count 74.48% (×134- inflated), instr-weighted 56.8% (shipped .text), distinct-code 27.3% (of 84,996 unique fns) - --weighted prints the two weighted numbers standalone; degrades gracefully if sigs absent - corrects the stale "~30-35% byte-weighted" estimate: the giant campaign since Phase 19 raised the fleet instr-weighted number to 56.8%; the distinct-code 27.3% is the unique-monster-tail truth - SETUP §tooling row updated (R21) |
||
|
Drew T
|
8fa29f2dda |
feat(phase-25): T7 F-band wave 3 part 2 — corrected rtu_match fan-out, 7/9 cracked + 266 swept; fleet 73.89->73.97%, R22 136/136
- NEW tools/rtu_match.py: real-TU-faithful, PARALLEL-SAFE per-fn match check. Compiles the WHOLE split TU (candidate spliced, INCLUDE_ASM neutralized via -DINCLUDE_ASM(a,b)= + -Isrc/<source>) in a per-fn temp dir -> no asm/, no shared overlay build. Closes match_one's isolation blind spot (in-TU decl/global-type/memcpy-builtin drift) so a MATCH HOLDS at the whole-binary gate. - Corrected Ultracode fan-out (wf_80762f2c-822, 9 xHigh workers): 7/9 real-TU MATCH -> all 7 banked byte-identical (individual + combined d19c9580), ZERO drift (vs wave-2's ~50% attrition). Banked: func_8012FCC4 func_80133AB0 func_80134A74 func_80136824 func_8014DD8C func_80168828 func_8016C188. func_8012FCC4's "irreducible" delay-slot was a wrong-callee-arity bug. - Swept x134: 266 siblings banked / 266 failed (the 2 //@EDIT families func_80133AB0/80136824 can't per-sibling-reconcile via family_sweep -> backlog: a --edit-remap enhancement). Batch = 7 exemplars + 266 = 273 fns. - func_8014DD8C: engine_core.h DEFINE_func_8014D790 extern void->s32 flip (byte-neutral fleet-wide, caller discards return) -- R22-confirmed neutral across all 136. - 7 durable levers -> cookbook §42c (callee-arity delay-slot, pointer-global *(T**)&, array-decay CSE, §17 zero-reg copy, void->s32, register-arg capture, free-floating load hoist). rtu_match -> SETUP §6. - 2 DIFF -> permuter (func_801670E4 70->48, func_80185BA4 c=65), seeds .run/crack3/wave3/. - R22 clean-fleet 136/136 BYTE-IDENTICAL from a fully clean tree; NON_MATCHING 7 (0 in default build, G4). |
||
|
Drew T
|
8cfbbf2d47 |
feat(phase-24): T7 whale func_80144B9C banked ×134 (-O0 reach-134 rollout) — clean fleet 136/136
The single biggest byte-weight lever (770 ins ×134 ~ +1.6% byte-weight), banked across all 134 overlays as -O0 compiled C (was ×1 in ov_SC01_077, commit:0463). - ROLLOUT (tools/rollout_whale_o0.py): per single-file overlay, LINE-split <ov>.c at the whale (splat emits in vram order), carve the yaml code subseg into before/<ov>_o0b(-O0)/<ov>_after, thin <ov>_o0b.c #includes the shared src/shared/func_80144B9C.h (the whale C moved there — DRY, TU-local typedefs, not a fragile 200-line DEFINE_ macro). Makefile WHALE_O0B_OBJS wildcard -O0. - DEDUP: group E_func_80144B9C (134 members, h_exact 74186b97, source=the header; group_members keys on binary+vram so a header-share validates like a macro-share). - MILESTONE: clean fleet check-all 136/136 (R22); dedup-check 1811/0; FLEET REAL 225816->225949, byte-identical 65.91->65.95% function-count (the gain is byte-weighted). Validated on ov_SC01_000 + spot-checks SC02/SC03/SC07 before the full fleet. - cookbook §38 + SETUP inventory. |
||
|
Drew T
|
994368fbcd |
feat(phase-24): T7b/T5 — 2nd giant func_80138ED0 MATCHED + banked ×134; fleet 65.52→65.56%; §34 idioms
- func_80138ED0 (159 ins, reach-134): Opus applying §32 -> close=21 (all semantics/control-flow/ constants exact, +the giv-init fence lever) -> Fable5Max reading the vanilla gcc-2.7.2 source -> MATCH (every regalloc/sched class C-reachable, no permuter). - banked via the standard pipeline (validates it on a real un-reconciled sibling): cast_call_sites reconciled func_8013914C (u8*,u16*)->(s32,s32)+cast; reconcile_decls a NO-OP (2 data bases giant-local -> no fleet conflict, confirms no false-positive); harvest_verify ×1 d19c9580 -> dedup_propagate --recover ×134 (pins/asm body propagates fine). clean fleet check-all 136/136, dedup 1800->1801/0, 0 NON_MATCHING. - NEW §31 idioms (cookbook §34 + regalloc.md RC-11 + loop.md): gcc-2.7.2's 3-qty local-alloc SORT BUG (local-alloc.c:1441 — <=3 qtys allocate in creation not density order; decoy-qty fix) + the zero-byte asm allocation toolkit (input-only/multi-input/def+use dummies) + the giv-init fence + gdb-on-cc1. gcc-2.7.2 reference tree completed (18 .c); SETUP §5.6. - T5 satisfied: reconcile_decls proven on 2 real giants (func_80129CF8 reconcile-path byte-proof + func_80138ED0 no-op/full-pipeline). CURRENT_PHASE T5 logged. |
||
|
Drew T
|
787b756e6d |
docs(phase-24): T7b — distill reconcile_decls (cookbook §33, SETUP, design-doc RESOLVED); R14-correct §32 wall
- cookbook §33: the reconcile_decls mechanism (oracle + byte-neutral cast taxonomy + pipeline
placement + the func_80129CF8 byte-proof); §32 'BANKING WALL' note R14-corrected (×134 was free,
the wall was only the ×1 loose-draft reconcile).
- SETUP.md: reconcile_decls tool-inventory row.
- canonical-decl-reconcile-design.md: ✅ RESOLVED banner — the premise reframe (automation not
unlock), the answered open questions, T4 no-op, the validation.
- CURRENT_PHASE: T7b/T2+T3 progress-log entry; NEXT = T5 sibling (func_80138ED0) via Fable5Max.
|
||
|
Drew T
|
0cda15c6de |
fix(phase-24): T5c — restore main's clean-rebuild: propagate the Phase-21 PsyQ renames to 62 stub refs
Pre-existing latent breakage (Phase-21 close commit:0292, "+62 PsyQ names"), found during T5b's R22 fleet verify. A genuinely-clean `make clean` + re-extract + build of `main` failed with dozens of `can't open asm/nonmatchings/800c3/func_*.s` — clean-rebuild broken since Phase 21, masked all along by incremental builds reusing stale pre-rename .s/.o (the exact R22 failure mode). - Root cause (byte-proven): Phase-21 xdedup renamed 62 PsyQ library functions to their proper names (InitHeap, FlushCache, GetTPage, SysEnqIntRP, SpuWrite, CdMix, __main …) in symbols.us.txt, but never regenerated/renamed the committed src/*.c stub refs that call them by the OLD func_<ADDR> name. splat's FRESH regeneration of a stub .c uses the CURATED names (move src/800c3.c aside -> splat writes INCLUDE_ASM(InitHeap) + emits InitHeap.s); the committed stubs were simply stale. - Fix: rename all 62 INCLUDE_ASM(func_<ADDR>) -> the curated name across 12 files (800c.c 1, 800c3.c 22, apicard1/2/4 4/6/2, boot.c 1, libcd1.c 8, libetc.c 6, libgpu.c 3, sgap.c 1, snd1.c 7, snd2.c 1). Pure rename to match splat's canonical output; byte-neutral. - VERIFIED: main clean-builds 143dbb89 from a fully clean tree; full clean fleet check-all 136/136. - Lesson (cookbook): a symbols.us.txt rename must be propagated to (a) shared-macro bodies (engine_core.h — the T5b GetTPage fix) AND (b) INCLUDE_ASM stub refs (this), AND verified by a genuinely-clean check-all (make clean + full re-extract), never incremental. Added a lint check. |
||
|
Drew T
|
0eadf313ee |
feat(phase-24): T5 — §31-directed permuter mutation + grinder input-changed gating
- tools/permuter_weights.py (NEW): classify(klass,where) -> regalloc|schedule|cse|None + render_settings_toml emitting the [weight_overrides] table decomp-permuter merges over the gcc defaults (main.py:336 / helpers.py per-key replace). Biases pass-selection toward each class's §31 levers (perm_reorder_decls RC-1/3, perm_reorder_stmts RC-2/S1, perm_temp_for_expr S2, perm_commutative cse); value/type noise -> ~0.1. NO submodule edit. - p16_permute.setup(..., klass=, where=) writes it (+--klass, backlog auto-lookup); grinder.py auto-threads klass/where_stuck. klass=None -> gcc defaults (undirected superset). - grinder fix (R14): replace the blind tried.clear() idle churn with input-changed gating (draft_sig=(best_draft mtime, closeness) — re-open a fn only when the worker improved it). - VALIDATED: regalloc profile drove func_8014E048 masked 36->29 (match_one -drz 35->28) where the undirected search stalled; re-logged (28, improved draft promoted). Flagship S11/RC-6 count-exact seeds improve but do not gate (intrinsic, §31 "two probes dont grind") -> T5b fuel. - distill: cookbook §3b + SETUP inventory (R16/R30). No build-input changed -> 136/136 untouched. |
||
|
Drew T
|
3b6f2c5f94 |
docs(phase-23): §31 — the gcc-2.7.2 codegen map (4 Fable5 agents read the source)
- docs/gcc-2.7.2-map/{sched,regalloc,loop,cse_expr}.md: source-cited, byte-proven
pass -> residual -> C-lever catalogs (935 lines). cookbook §31 = the index + triage.
- WALLS BROKEN (byte-proven steerable, were "CONFIRMED unsteerable"): §10/§20
hoist-vs-remat (cross-call address-caching), store-vs-load (/s), dbr delay-slot
(D1+S2 fresh-local: func_801770E0 53->49), birthing-boost both directions.
Incidental bank: func_80149374 x134 (fleet 64.86%).
- GENUINE walls -> permuter: S3 chain-priority sink, S11 LUID(x)alloc coupling,
RC-6 pressure-lock, cse 1000-insn table flush.
- CORRECTION (loop agent): tools/reference/gcc-papermario is gcc 2.8.1 NOT 2.7.2
(behavioral biv-elim diff). Vanilla gcc-2.7.2 -> tools/reference/gcc-2.7.2/
(gitignored); SETUP §5.6 + §31 flag it. All banked levers stand (match_one-validated
vs the real cc1). Also corrected: spill-slot = declaration order; §25 4th rank rule.
- the payoff: the cheap tier (Opus agents + local model) can now apply compiler-internal
levers by triage-table lookup, without reading 80k lines of source.
|
||
|
Drew T
|
e9f199530e |
docs(phase-23): T10 harvester built + measured — 65% bank-rate, parallel gate ~75x, vLLM justified
- gen2-mips-matching-model.md: T10 RESULT (bulk_harvest phase-separation; 52/80=65% on fresh SC03 <=15-ins; gate 0.4s/fn/8w vs ~30s/fn serial; drafting now the sole bottleneck -> vLLM) - SETUP.md: bulk_harvest.py inventory row + gate_stage/harvest_verify parallel-safe params (R21) - CURRENT_PHASE.md: T10 done (built+measured), T10.5 vLLM split out as Drew-gated; progress log |
||
|
Drew T
|
205fc202e9 |
feat(phase-23): macro-extern-injection lever — 8 reach-134 fns freed ×134 (fleet 63.82→64.09%)
- tools/inject_capped_externs.py (new): source each propagation-capped reach≥2 inline match's externs from ov_SC01_077's own file-scope decls, inject them block-scope so the lifted macro is self-contained (the Phase-20-backlog macro-extern-injection lever; byte-neutral, fail-safe via compiles_standalone) - 8 reach-134 fns freed from ov_SC01_077.c, 077 byte-identical; dedup_propagate --auto-from registered 7 new shared groups ×134 (1678->1685), check-all 136/136 - the LLM reach≥2 measurement run already banked +7 (v3 banks reach≥2: 6/15 in 005 vs v2's 0/15) — committed earlier (commit:0359, commit:0360) - cookbook §28d + SETUP tooling inventory (R16/R21/R30) - fleet 63.82% -> 64.09%, 0 NON_MATCHING, dedup-check 1685/0 failed |
||
|
Drew T
|
1eadbc73e9 |
docs(phase-23): 8-hour autonomous run — corpus-v3 + v3 + local serving; fleet 63.67->63.82%
The run (Drew away): built tools/serve_local.py (Unsloth GPU serving, replaces ejected LM Studio); the prompt fix (translate-every-instruction, 0/3->2/3 on the empty-leaf class); corpus-v3 (mine the 1623 engine_core.h DEFINE_func macros + struct types: 1312->2891 pairs, 2534+291 trainable); v3 trained (loss 1.275->0.085) eval 23/40 (57.5%); v3 production batch banked ~352 fns + 45 new shared groups -> fleet 63.67->63.82% (+502 byte-identical), 136/136 byte-clean, 25 auto-commits, $0. v3 banks the empty-leaf/setter class v2 couldn't. - docs/gen2-mips-matching-model.md: corpus-v3 + serving + prompt-fix + v3 eval + harvest - docs/SETUP.md: LLM matching tier added to the Tooling inventory (R21) - phase-ends/CURRENT_PHASE.md: run progress + next levers (reach>=2 targeting, dedup-collapse) |
||
|
Drew T
|
1f4aac62fb |
docs(phase-21): cont.8 — collaboration sweep (clean verdicts) + 62 PsyQ names + SETUP -G fix
- reach-1 idiom loop: +16 ×1 matches (commit:0289, commit:0291) + chained-store-fanout idiom (commit:0290); giants pivot 0 auto-banks (2× = the byte-proven wall) -> 6 ranked hand-finish near-misses in backlog (2 close: func_8015126C close=2, func_8014EE14 close=6) - collaboration sweep CLOSED with clean verdicts (5 scouts): parity research (we're SOTA; no neural/GPU/diffusion shortcut), decomp.dev survey (4 gcc-2.7.2-psx twins; VS=CC0 the clean pool partner), census (no Square symbols; Croc=SDK-naming only), VS corpus (76 byte-verified pairs, low-priority asset), dedup probe (CLEAN NEGATIVE — BFM shares only PsyQ with sisters, AKAO refuted, 0 engine code) -> no external shortcut to the gcc wall - config/symbols.us.txt: +62 PsyQ library/BIOS names from cross-project byte-signature dedup (xdedup vs VS/Xenogears); byte-safe (main still 143dbb89, R22-verified) - docs/SETUP.md §5.3: corrected stale "-G TBD" marker -> RESOLVED Phase 5 (-G0) (R21) - docs/backlog.md: giant near-misses logged with true worker closeness (cont.7d-safe) - CURRENT_PHASE.md cont.8: session log + 3 parked R&D items + process lessons - fleet 63.18%, 136/136 byte-identical, 0 NON_MATCHING |
||
|
Drew T
|
6c5bd1c4a9 |
feat(phase-20): cast_call_sites.py — the §17a-1 callee-cast cap tool; 6 recovered, fleet 58.63%->58.82%
- tools/cast_call_sites.py: per draft, callee whose canonical TU sig != draft's intended sig -> decl->canonical (kills in-TU conflicting types, keeps symbol in scope) + call-site cast ((ret(*)(args))func_X)(args) (codegen-neutral: gcc folds the cast of a known symbol -> direct jal). Pure --in/--out; whole-binary gate is the arbiter (G3/P9). Pipeline: canon_resident_calls -> cast_call_sites -> sig_unify -> harvest_verify --chunk 1 - recovered 6 of T6 batch-1's 33 gate-fails byte-identical (func_80153C44/8015CF58/ 801711FC/80161BE0/801683D8/8015F948); 5 propagated x134 + 1 local; check-all 136/136 - R14 CORRECTION of cookbook §20's '~33' projection (byte-proven): the batch is NOT mostly callee-conflicts — ~6 callee (cast-recovered), ~18 DEF-side loose-typing wall (def's matchable sig conflicts with the unchangeable caller-canonical; caller-side fix BLOCKED because INCLUDE_ASM declares no symbol -> a shared macro's internal extern is load-bearing in the 133 stub overlays), + a small DATA tail. The def-side needs re-drafting under the caller-canonical sig (future wave), not a text transform - cookbook §20 RESOLUTION + SETUP tool inventory (R16/R21/R30); CURRENT_PHASE log (R28) - fleet 58.63% -> 58.82%, dedup 1501 -> 1508, 0 NON_MATCHING (G4) |
||
|
Drew T
|
2070cdb000 |
docs(phase-20): T4 — distill the codegen map (cookbook §20) + R14 loop-guard correction
- cookbook §20: the wave-at-scale GATE CAP (match_one over-predicts; the in-TU loose-typing call-graph wall; all recovery levers fail; the only fix is automating §17a-1 call-site casts = Phase-21 cap lever) + the stale-.o diagnostic lesson - CONFIRMED-irreducible classes w/ byte-evidence: §10 store-vs-load (8014F2E0/80150528), §10 hoist-vs-remat (80149374/801493D0); NEW classes: IV-combine combine_givs (80177AD4), hoisted-invariant prologue order (80177F84), the -O1 build-infra class (80161A90) - operational gotchas: Workflow args-as-array, harvest_verify --chunk 1 - R14 correction to §17a: loop-guard (8012C2D0) is STRENGTH-REDUCTION/IV-final-value addressing, NOT operand-order - SETUP §tooling: exemplar_miner.py (T2) + build_engine_types.py typedef-lift (T1) (R21) - T3c cited (written up), T4 done; CURRENT_PHASE handoff refreshed |
||
|
Drew T
|
de35505a63 |
feat(phase-19): scale the toolkit — 2 waves (88%/92%), fleet 56.64%->58.00%, propagation is the cap (v1.18.0)
- T1 (prior session, commit:0158/commit:0159): per-file -O0 split infra + 6 -O0 matches; deferred the 10 %lo-folding-quirk fns + the ×134 rollout (R14 corrected "free ×134") - T2 (commit:0161): recovery tooling. R14: the 3 Phase-18 "implicit-int" near-misses are NOT implicit-int -> resident-callee link-miss (NEW canon_resident_calls.py) + shared-caller arity (no-proto caller-extern fix). 3 matches; cookbook §17a-3 corrected - T3 (commit:0162/commit:0163/commit:0164): 2 Ultracode waves over reach-134 tractable residuals. batch-1 44/50 match_one (30 banked, 25 propagated ×134); batch-2 35/38 (25 banked, 10 propagated). NEW fix_arity_callers.py (no-proto, ALL 18 batch-2 failures were the arity class); canon-first 2-stage gate (sig_unify REGRESSES canonical drafts); gen_harvest_targets garbled-sig fix (0/749, 88%->92%); build_engine_types --strip lift (+4 propagated) - FINDING: propagation (not matching) caps the fleet-% yield — typedef/local-type bodies can't auto-propagate (-> Phase-20 type-lift lever). decomp.dev-comparable byte-weighted est ~30% - check-all 136/136 BYTE-IDENTICAL (R22); dedup 1450->1485/0 failed; 0 NON_MATCHING (G4); fleet 56.64% -> 58.00% (+1.36%). worklog -> phase-ends/logs/Phase19.md (R19) - bumps 1.17.0 -> 1.18.0 |
||
|
Drew T
|
85afa0dc81 |
feat(phase-19): T2 recovery tooling — canon_resident_calls + no-proto caller fix (3 matches)
- R14 finding: the 3 Phase-18 "implicit-int / propagate-first" near-misses are NOT
implicit-int (reproduced each through the byte-gate). Real classes:
(a) resident-callee LINK-miss: draft calls func_<ADDR> but the addr has a curated
name (0x8004CFEC = ratan2) -> "undefined reference to func_8004CFEC". NEW
tools/canon_resident_calls.py rewrites func_<ADDR> -> curated name (pure draft
text, body bytes unchanged) -> run FIRST in the recovery pipeline.
(b) shared-caller ARITY conflict: an engine_core.h caller macro declares the callee
file-scope `extern void func_X(void)` then calls func_X(); the real def takes an
arg -> "conflicting types". FIX = caller extern -> no-proto `extern void func_X();`
(byte-neutral for the caller; compatible with promotion-safe params int/long/ptr).
- 3 matches banked in ov_SC01_077: func_8017209C (ratan2 rewrite), func_80147514 (s32),
func_80168F40 (void*); 2 no-proto lines in src/shared/engine_core.h
- recovery pipeline: draft -> canon_resident_calls -> sig_unify -> harvest_verify --chunk 1
- make check-all 136/136 BYTE-IDENTICAL (full extract, R22); dedup 1450 validated/0 failed;
0 NON_MATCHING in any default build (G4); fleet 56.64% (the 3 are ov_SC01_077-local,
x134 propagation rides T3/Phase-20)
- cookbook §17a-3 corrected (R14/R16); SETUP tool inventory (R21); CURRENT_PHASE T2 done
|
||
|
Drew T
|
d20b20c64b |
feat(phase-18): T0 — setup + exemplar set; match_one validated as the §10 oracle
- clone gcc-papermario (a6afc2af) + xenogears-decomp (f27c0768) -> tools/reference/ (gitignored research refs; SHAs pinned in docs/SETUP.md §5.6, R20/R21) - apparatus byte-validated: match_one cleanly measures the §10 residual on both quirk exemplars (the floor-free oracle Phase 17 failed to use; permuter score was polluted): - func_8012B8E4: 75=75 ins, 24 mismatched -> $s0/$s1 regalloc swap + reassociation (§10 A) - func_8012B4B8: 94 vs 84 ins, 88 mismatched -> extra-reg spill + struct-copy/reload (§10 B) - CURRENT_PHASE.md crash-recovery log; harness task list (R28); no build-affecting change |