mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 05:56:00 -04:00
0f73618fa82876e81ca7b015dfabc0dbc242f94c
55 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew T
|
1576570271 |
fix(phase-30 S1e): the distinct-code "regression" was a STALE DIGEST — alias lever ungated
The S38 checkpoint gated the phase's best lever ("do NOT scale the alias lever") on
distinct-code falling 89.3 -> 89.2. It never fell.
PROOF (each commit's metric recomputed from its OWN committed tree, 0 unresolved):
commit:1426 TRUE : instr 12394533 distinct 5022306 (77895 uniq)
commit:1426 COMMITTED: instr 12402412 distinct 5029324 (78025 uniq) <- stale
HEAD TRUE == COMMITTED: instr 12405402 distinct 5025082 (77952 uniq)
=> true delta 843->HEAD: instr +10869, distinct +2776 ins / +57 uniq. ALL ROSE.
The 843 digest was generated from a working tree still holding work REVERTED before the
commit landed (+7,879 ins / +130 uniq overstated) and never regenerated, so the next
HONEST digest read as a fall. => THE ALIAS LEVER IS UNGATED (scale it, §61 small batches).
Both recorded leads were wrong (R14): progress.py:423's SIG regex feeds fn-count ONLY
(neither weighted metric sees a C identifier — both derive matched = sig - corpus.stubs),
and "the harvest reverted functions to INCLUDE_ASM" died on one grep (483 removed, 0 added).
The 3-grep proof: identical sigs + unchanged tools/ + zero +INCLUDE_ASM => HEAD's stub set
is a strict subset => both numerators are FORBIDDEN to fall.
THREE INSTRUMENT DEFECTS, all one class (a bare except around a fail-CLOSED oracle):
- progress.py stub_addrs wrapped corpus.stubs in `except Exception: return set()`. An empty
stub set means "could not answer", not "no stubs", so matched = sig - stubs credited EVERY
function. Byte-witnessed: instr 100.00% / distinct 100.00% in a tree with no asm/. Now
propagates.
- cast_call_sites.tu_for + reconcile_tu.tu_for had the identical swallow, falling back to the
default <ov>.c instead of the jr/-O0 split TU — silently reinstating the exact bug
cast_call_sites' own docstring says it exists to fix. A wrong-TU reconcile fails the gate,
and this phase's base rate is ~24k PLUMBING vs 4,917 DIFF, so it presents as a codegen wall.
Now propagate CorpusError; ValueError fallback for curated names preserved; derived-TU path
re-verified (a _jr_ split stub resolves correctly, both tools agree).
NEW GATE (R34 — the byte-gate is a null oracle for DOCUMENTS; check-all stays 140/140 over a
stale digest forever): tools/audit_digest.py + `make audit-digest`, wired into tools-health
after report. Recomputes the three headline metrics from the current tree and fails if the
committed digest disagrees. Compares INTEGERS, not percentages — the +7,879-instruction
staleness printed as "94.4%" on both sides. Negative-control-proven against the stale 843
digest (fails, exit 1) and green on HEAD.
Verified: make report exit 0 (dedup-check 1910 validated / 0 failed, C1 coverage
241216/241216); audit-digest OK; cookbook-index OK (398 sections); metrics unchanged by the
fix (94.40% / 89.18%). No src/ or config/ edits — no bytes touched, nothing banked.
cookbook §140 · decision-log 2026-08-04 · SETUP.md inventory (R21) · R14/R32/R34/R35.
|
||
|
Drew T
|
19766a6dc3 |
feat(phase-30): T0.5 — prefetch_fleet.py, the fleet Ghidra-C batch orchestrator (+ SETUP row, R21; fuel manifest ride-along)
One representative per remaining h_seq distinct class + all main/resident stubs -> .run/ghidra_c/. Resumable (skips cached); serial on the exclusive project lock; auto-stops a serving MCP (R23); imports missing overlay programs on demand via ghidra_import_raw.sh (blob derived via family_remap.img_path, vram from the splat yaml — R33, never guessed); R32 per-program outcome report, continues past failures. Dry-run: 126 programs / 7,966 uncached representatives. |
||
|
Drew T
|
f285de46e8 |
feat(phase-29): T53 — fold the T51 lever into the gate; kill gather_externs' false positive; close a revert gap
Items 1-2 off T52's list, plus a third defect found by T53's own testing. TOOLING ONLY — banks
nothing; metrics unchanged by design (85.7% instr / 76.4% distinct / 90.65% fn-count).
1. THE T51 PRE-PASS IS A jtbl_family_bank STAGE (cookbook §103, AUTOMATED)
Order: raw -> scoped -> tu-scoped -> recovered -> reconciled. After the non-invasive stages (it
edits the TU outside the spliced body); BEFORE the recovery stages deliberately — those bend the
DRAFT and T48 measured both at +3 ins for this class, so they cannot succeed here. The stage
re-runs scope_data_fix against the SCOPED TU rather than reusing the raw body: composition-correct,
since the contested symbols no longer have a file-scope decl to be dropped against.
COUNTERFACTUAL, byte-gated on a reproduced blocker (ov_SC01_000 restored to its pre-T51 TU):
raw -> compile error (conflicting types)
scoped -> compiles, FAILS the byte check (§8d drops the decl -> the u8 CSE costs +3)
tu-scoped -> BANKED
That is the evidence the stage does the work — not T52's sweep, which ran on TUs T51 had already
scoped by hand.
Refactor note: a stage editing outside the spliced body must RE-FIND the splice point (the stub
offset indexes the ORIGINAL TU). Each stage now carries its base; every pre-existing stage passes
`orig`, where the re-search returns the identical span — same operation as before, by construction.
2. gather_externs' COMMENT-SCANNING FALSE POSITIVE — FIXED (cookbook §104)
It scanned RAW text, so a symbol named only in the draft's PROSE counted as referenced: the
func_80135D20 warning that fired on 137/137 and was right 0 times. Fix is a two-text discipline —
MATCH on cdecl._mask'ed text, EMIT by span from the ORIGINAL (a masked decl is all blanks, so
"just mask it" would splice whitespace). Same change closes a second, unobserved defect of the
class: a COMMENTED-OUT extern could be selected as the carried decl and spliced in as live code.
MEASURED as a no-op on output (R14): 20 (exemplar, sibling) draft pairs across 4 families, old vs
new -> 20 identical / 0 differing. Only the false warning changed.
3. UNPLANNED — A REVERT THAT DID NOT SURVIVE AN EXCEPTION (cookbook §105)
A wrong exemplar made remap_hseq raise AFTER the carve rewrote config/ and jr_isolate created a
region file; the exception propagated out of bank(), the revert never ran, and the tree kept a
rewritten carve config plus an UNTRACKED region file (git checkout -- src/ does not remove it).
In a 132-member sweep that residue rides into the next member's build. bank() is now a
revert-guaranteed wrapper around _bank(). Negative-control proven: the crashing invocation now
reports {'exception': 2} and leaves git status -- config/ src/ at 0.
"Revert on failure" != "revert on every exit"; the exits are success, gate-fail, refusal, and the throw.
GATES: R22 clean-fleet 140 passed, 0 failed of 140. tools-health OK (corpus 0 PHANTOM + 0 TRUNCATED,
cdecl, audit-binaries, report/lint/dedup 1886/0). 0 NON_MATCHING (G4).
HONEST COVERAGE GAP: no live end-to-end BANK through the refactored loop — all three big families are
137/137 and the only family with live stubs (0x80191c50) has no banked exemplar, so it refuses. The
counterfactual byte-gated the exact splice on all three candidates and the 2-member run exercised
construction/refusal/revert/tally; the next real family sweep is the true end-to-end validation.
MY ERRORS: invoked the sweep with a wrong exemplar+address for a cross-address family (an unmeasured
guess about a members file I had not read — it is what surfaced defect 3); and deleted last_err's
initializer while refactoring, which would have raised NameError on the first clean gate-fail.
cookbook §103 (AUTOMATED) + §104 + §105; SETUP inventory row updated (R21).
|
||
|
Drew T
|
a524142eb5 |
feat(phase-29): T51 — the fleet-wide decl-scoping tool; func_80135260's 132 siblings unblocked
Item 1 off the SESSION-23 list. T48 proved the lever by hand on the exemplar, T50 located the same
blocker in every sibling; this builds the tool, measures the population, applies it fleet-wide, and
gates it. It BANKS NOTHING — it removes the blocker. The sweep is the next task.
MEASURED BEFORE BUILDING (R35). The blocker census over all 132 still-stubbed siblings is perfectly
uniform: 132/132 carry it, 3 contested symbols each, EXACTLY ONE file-scope decl statement per
(TU, sym), ZERO file-scope references below the decl (so the deletion is always safe), 660
block-scope re-declarations needed.
THE TOOL: tools/scope_tu_externs.py — the TU-side complement of scope_data_externs.py (§8d). §8d
fixes the incoming DRAFT and has a give-up branch that DROPS the draft's own decl when the TU already
declares the symbol at file scope. Right when the types agree; fatal when the byte-true draft needs a
different one — which is exactly how 132 byte-true siblings gate-failed wearing a codegen wall's
costume. This moves the TU's OWN file-scope decl down into every later function that references the
symbol and lacks its own block-scope decl, then deletes the file-scope line.
FILE(u8 D_x) ... then BLOCK(u16 *D_x) below it -> conflicting types (the 132 failures)
(no file-scope decl) ... BLOCK(u8) ... BLOCK(u16 *) -> builds; each fn owns its own view
- contested set DERIVED, never hand-listed (R33): the remapped draft's block-scope D_ externs
intersected with the TU's file-scope decls above the splice point; --family does it per sibling
- built on cdecl.split_statements/_mask (R33), not a 7th regex: depth-0 spans (a fn definition
flushes at its closing '}' — a column-0 test is NOT a file-scope test, m2c emits goto labels at
column 0 inside bodies) + length-preserving comment/string masking. That masking is what kills the
comment-scanning false-positive class still open as item 3.
- REFUSES LOUDLY, never skips silently (R32): >1 file-scope decl above the splice point; a
file-scope statement below the decl referencing the symbol; an unlocatable body brace
- coverage asserted as a DELTA (R32): file-scope -1, block-scope +len(consumers). An absolute
"a block-scope decl exists" check would have passed VACUOUSLY — these TUs already carry ~18
legitimate block-scope decls of the same symbols
VERIFIED IN TWO STEPS (T48's structure — why a 132-file edit was safe to make):
1. the move ALONE on ov_SC01_000 -> make build -> 9052dc0e BYTE-IDENTICAL, then reverted
2. fleet-wide -> R22 clean-fleet (make clean && extract-all && check-all) -> 140 passed, 0 failed
of 140; make tools-health OK (corpus 0 PHANTOM + 0 TRUNCATED, cdecl, audit-binaries,
report/lint/dedup 1886/0). Metrics UNCHANGED at 85.5% instr / 76.1% distinct / 90.62% fn-count
— the correct result for a declaration-only change.
The diff is uniform to the line: all 132 files +11/-3. A second --family run reports 132
nothing-to-do, 0 refused (idempotent).
Deliberately NOT done: wiring this as an automatic jtbl_family_bank stage. That waits until the
sweep measures the payoff — folding an unproven pre-pass into the gate is the same unmeasured
premise this phase keeps catching.
Also preserves .run/near6/g5260_a.c (the T48 raw crack body, allowlisted) — the sweep may need it
for --raw.
cookbook §103 + SETUP tool-inventory row (R21).
|
||
|
Drew T
|
b8e7250c78 |
feat(phase-29): tools/symcheck.py — the pre-gate symbol-set guard (§67a)
Builds the guard SESSION-17 left as a TODO after the func_801463A0 `_s`-alias trap, where a draft invented extern aliases no symbol table defines, read MATCH under rtu_match, and could never bank. - diffs the symbols a draft's object references (reloc records) against the target .s's %hi/%lo/jal set; reports MISSING (invented-alias signature) and INVENTED separately. - fills a real hole: match_one/masked_diff compare relocation-MASKED words (object-vs-.s is symbol-agnostic BY CONSTRUCTION) and rtu_match COMPILES WITHOUT LINKING -- so both are structurally blind to this class. R34: a second oracle that can disagree with the first. - NEGATIVE-CONTROL PROVEN: with one data extern renamed to an invented alias, match_one reports the SAME 14 mismatched as the correct draft; symcheck exits 1 naming both symbols. - --c compiles via match_one so the pinned triple/flags can never drift (R33); or --obj. - applied to the live func_8014D820 close=14 draft: 12/12 symbols agree, so a match there will link cleanly -- the §65c class is ruled out for it in advance. - cookbook §67a + SETUP tooling-inventory row (R21). Necessary condition, NOT a match oracle: still finish on the whole-binary byte-gate (G3/P9). |
||
|
Drew T
|
ffca2e4803 |
feat(phase-29): T16.10 — the driver's SUCCESS path verified (free re-bank test); 2 defects fixed
- THE FREE TEST (cookbook §66): reverted func_801778A8's bank to its INCLUDE_ASM stub (stub state rebuilds BYTE-IDENTICAL 7ca772be — a faithful revert proves itself; needs `make extract` first, the R22 corollary) and re-banked it THROUGH recover_integration.py --commit --r22. pass1 1/1 -> exact restore -> pass2 1/1 -> commit commit:0928 -> R22 140/140 -> report.json. Bank confirmed from SOURCE (stub gone), never the report (§55b trap 4). EQUIVALENCE: git diff vs the pre-revert commit = ONE blank line (mine) -> the driver reproduced SESSION-16's state exactly. - DEFECT 1 (SAFETY, found by reading before firing): PROPAGATION is a fleet-tier write (dedup_propagate --auto-from -> src/shared/engine_core.h + up to 138 overlay .c) that was both UNDECLARED and the DEFAULT, so --max-tier binary still permitted the widest write in the toolchain. assert_write_set cannot catch it (it runs before the gate; under --commit git status is clean). FIXED up front: propagate now requires --max-tier fleet AND --r22, and is REFUSED after a demacroize stage (those banks are x1 by construction; --auto-from would re-macroize and undo them). Both refusals negative-control-tested, exit 1. The "standing hazard" is now a refusal. - DEFECT 2 (METRIC): gate_stage scraped the fleet % via a progress.py label that no longer exists -> fp=None -> 50 gate commits recorded "fleet None%". Now reads FLEET instr-weighted (legacy fallback + loud stderr warning if neither matches); parses 79.6. - STALE DIGEST (R14): docs/progress.fleet.md at HEAD disagreed with HEAD's own source by 45 in the dedup-shared column — generated during the §65g local_type trial whose edits were then reverted. Regenerated (reproduced identically in-gate + standalone); headline %s unaffected. - cookbook §66/§66a/§66b distilled in-session (R30); SETUP.md gains the missing recover_integration row (R21 debt). tools-health OK: corpus 0/0, cdecl green, audit-binaries 140 citizens, lint OK, dedup-check 1879/0. Fleet unchanged 79.6% instr / 67.7% distinct / 88.86% fn-count. |
||
|
Drew T
|
f1f6238251 |
feat(phase-29): T16.6/T16.7 — 14th bank via the generalized de-macroize; §65 distilled; R21 inventory debt cleared
- func_8012F40C banked (the callee-conflict variant): relaxing demacroize from "the draft's own function" to "any decl the DRAFT declares incompatibly" reaches macros that declare a CALLEE differently than the draft does (RotTransPers/RotTransSV). 14 banks total, R22 140/140. - THE ONE FAILURE, kept honest: func_8012F49C was rtu-MATCH but the whole-binary gate REJECTED it. rtu_match is relocation-masked, so a wrong call TARGET is invisible to it -- and this was a callee case, exactly where the mask hides the error. Trust rtu MATCH for self-decl corrections, distrust it for callee ones (§65c). Reverted its edits and re-banked only the winner rather than leave byte-neutral churn on matched code (§57a-4). - DISTILLED IN-SESSION (R30/R16/R31/R21): cookbook §65 + §65a-§65e (blast-radius tiers; the de-macroize escape and the §20 refutation; the rtu-vs-gate divergence; the existing-ladder baseline; two-oracle practice); decision-log entry with the HONEST multiple (~2.3x, not the projected 3.7x, and it lands on distinct-code not the display number); calibration.md measured table; SETUP.md rows for blocker_probe + demacroize PLUS the three the inventory was missing (lift_types, uniquify_type, fix_header_decl-as-retired). - Carried and NAMED, not dropped: 10 match_one-MATCH drafts still blocked by stacked classes, and the 11 `near` drafts which are unfinished drafts, not recovery fuel. |
||
|
Drew T
|
b3ab5c2253 |
feat(phase-29 Task-13A): deterministic residual classifier — the permuter's problem is TARGETING
The autopsy (hindsight-study §7) assumed the permuter loses for want of a mutation. Measured over the whole open backlog, it loses because it is aimed at work a search-closer provably cannot close. - NEW tools/residual_class.py: decide a near-miss's class FROM THE BYTES. Decodes each mismatching MIPS word -> (op-skeleton, register-fields, immediate); drift FIRST (one inserted insn inflates `closeness` by the tail length), then consistent-injective register map -> REGALLOC-PERM (§31 S11/RC-3), same-multiset-reorder -> SCHEDULE-REORDER, DELAY-SLOT, WIDTH/BRANCH-POLARITY/STRENGTH/ADDRESSING/IMM-OFFSET/IMM-VALUE. Every class routes to a BUCKET = which tool the failure wants. Uncovered opcode -> UNKNOWN, COUNTED (R32). 16 synthetic unit tests (test_residual_class.py). - NEW tools/autopsy.py: `collect` materialises the corpus Task-12's telemetry never filled (1 of 6,169 records had a residual) by recompiling every open draft through the EXISTING match_one path (R33) — 1,752 drafts in 21s at -j12. `report` -> docs/autopsy.md. - NEW corpus.o0_sources()/is_o0(): the opt-level oracle DERIVED from the Makefile's own -O0 rules, coverage-asserted. Scoring an -O0 target at -O2 makes the residual 100% artefact (the trap this phase hit four times). - R34 cross-check baked in: residual_class's closeness vs masked_diff.structured_diff's, asserted per row; 1,673/1,673 agree, 0 classifier errors. FINDING: of the 972 records the grinder's own filter admits, only 75 (7.7%) are permuter-shaped; 547 are structural and 348 are drafts that are not the function at all. ~92% of the daemon's CPU went where it could not win — the byte-grounded explanation of "7 banks all-time, all Phase 21, 0 since" (Phase-22 audit). grinder.candidates() now filters on the measured bucket (1,303 -> 78) and takes its directed profile from the measured class, not the logged label (91% carry none -> it ran on gcc defaults). Degrades to undirected if uncollected and says so; --no-targeting A/Bs it. Two measured corollaries (R14, not projections): - 699 records rank as near-misses at closeness up to 278 purely from a length artefact: un-attempted work misfiled as a backlog of hard functions -> new `redraft` bucket. - a 12-draft gate probe of the `integration` bucket banked 1/12 (11 PLUMBING), so the 306 prices Task 14's reconcile ladder rather than promising free banks. func_80167714 (104 ins, reach-134) banked x1, un-propagated by design (§55b). Two defects fixed forward: - masked_diff._common_typedefs() used ONE shared probe path, so parallel match_one processes clobbered each other: 14 of 1,752 drafts lost in a single 12-way run (0.8%), silently, in every parallel wave ever run. Now per-PID. - gate_stage.match_one_closeness never passed --o0 -> phantom residuals for every -O0 function, written straight into the backlog this autopsy reads. R22 clean-fleet: check-all 140 passed, 0 failed of 140; tools-health OK (dedup 1847/0, C1 234343/234343); 0 NON_MATCHING (G4). Flywheel captured in-session (R30/R31): cookbook §60, decision-log entry, SETUP.md inventory. |
||
|
Drew T
|
5e78dccf07 |
feat(phase-29 §8e): jtbl 8-align pad-spec filter — the 4-giant unblock tooling (fleet-neutral)
- ROOT CAUSE PINNED (the session-2 half-pin was INVERTED; both probes were vacuous, R35): cc1 emits .align 3 before EVERY jump table; maspsx passes it VERBATIM (the :435 'drop' is an inventory-only pass); as bakes the pad SECTION-RELATIVE; link placement was never guilty (SUBALIGN(2) + ALIGN(.,4) place 4-mod-8 carve starts tight). Merging originally-separate TUs fires an intra-TU align where the original packed tight -> +4 at rodata 0xCC -> image-wide %lo shift. Honest probes persisted: .run/probe_jtbl/ (verdict.md + objdumps). - NEW tools/jtbl_rodata_pads.py: post-maspsx filter replaces each rodata .align 3 with the ORIGINAL's exact pad bytes per a JTBL_PADS spec; fail-loud on table-count drift / non-align-3 / non-jtbl rodata content. Byte-proven: verbatim 0xE4 pad-at-0xCC -> filtered 0xE0 tight (= the merged carve span). - jtbl_carve.py: spec-aware same-subseg merge (gap 0 or 4-with-zero-payload-word; else NON-CONTIGUOUS -> isolate), interval-arithmetic pad specs (committed values CARRIED, never re-derived), JTBL_PADS target-var emission into overlays.mk + revert() restore + stale-.o invalidation; the false 'maspsx drops .align' docstring corrected (H5). - Makefile: $(if $(JTBL_PADS),| jtbl_rodata_pads.py ...) stage in build/src/%.o + file-scope empty default (env-shield). jtbl_family_bank.stub_file: duplicate-stub fail-loud (the earlier 'ladder failure' was a wrong-TU splice into a stale _a.c stub, byte-witnessed). - R22 clean-fleet WITH the fix wired: 140/140 byte-identical, tools-health green (dedup 1846/0, C1 234205/234205), ZERO new banks -- fleet-neutral by construction. - cookbook §8e (the jtbl alignment law) + §8a/§8a-pad corrections; decision-log R31 entry; SETUP.md tool row; .gitignore allowlist for the probe verdict artifacts. |
||
|
Drew T
|
427baba3bf |
feat(phase-27 T10): completion dashboard (main in the weighted metric) + the resident second oracle
The metrics contract (roadmap §1) wants all three metrics WITH main in the denominators, and the
second, independent boundary oracle (R34) extended beyond the overlays. Both had landmines.
10a — main into the weighted metric, safely:
- weighted_metrics off the func_-only src_stubs regex onto corpus.stubs (R33). THE LANDMINE IS
REAL: src_stubs("SLUS_007.26") globs src/SLUS_007.26/*.c -> 0 files -> every row "matched" ->
main 100% + fleet % silently inflates. Routing through corpus.stubs is a PROVEN 0.000pp no-op on
the existing fleet (overlays are all func_) and closes the curated-name leak.
- a SEPARATE "MAIN game-code weighted" line (0.7%): main's Ghidra sig excludes the LINKED PsyQ
objects (Ghidra never analysed them), which is exactly right for a game-code metric (LINKED is
complete, counted in fn-count). Reported un-folded and caveated (month-stale sig, PROVISIONAL) —
folding a stale/incomplete value into the decomp.dev headline would mislead the flip checkpoint.
10b — the resident second oracle:
- make sig-resident: sig_image on the resident flat blob (byte-derived, not Ghidra). corpus.
sig_is_independent now covers resident -> audit-corpus checks its boundaries too. Probed clean
BEFORE wiring (144 fns, all 21 stubs present, 0 phantom), verified 0 phantom + 0 truncated.
- sig-overlays now derives its payload list from config/overlays.mk, not a 0.4.dec glob that
silently dropped the 4 SC07 index-1 overlays (the audit's own silent-skip class). tools-health
regenerates sig-overlays + sig-resident first so the audit never crashes on an absent sig.
10c — main's second oracle: docs/second-oracle.md. sig_image can't sign the PS-X EXE yet (0x800
header offset, interleaved data/linked islands, one text range); seeding from splat would destroy
independence for the PHANTOM class specifically. Honest deferral + scoped design, not a fake oracle.
- docs/progress.fleet.md regenerated: 140 binaries · fn-count 82.16% · instr-weighted 67.0%
(the honest post-T7 drop from 68.9%) · distinct 47.8% · MAIN game-code 0.7% (separate).
- SETUP §6.3 updated (R21).
|
||
|
Drew T
|
264fe6c115 |
feat(phase-27 T7): disc-completeness audit — onboard 4 hidden SC07 overlays (136->140) + the type sweep
The whole-binary byte-gate is structurally blind to code nobody onboarded (R34): check-all is
green over the onboarded set no matter what code sits unbuilt on the disc. This reconciles the
onboarded set against every code-bearing PAC payload.
- new_overlay.sh: optional [ENTRY] arg (default 0.4) reaches a non-0.4.dec payload. Onboarded
ov_SC07_{006,007,010,011} from 1.4.dec (they put graphics at PAC entry 0, the code overlay at
entry 1 — invisible to the 0.4 hardcode for a month). Each byte-identical (7ca772be / b3b95547 /
d7b5875d / 9885af74). FLEET 136 -> 140; check-all 140/140 (T2's pass==N re-baselined cleanly).
difficulty.py NOT in the insertion set anymore (it derives, T6) -> only 3 tool dicts touched.
- tools/disc_code_sweep.py: decode every payload (reusing sig_image.make_insn) and gate code on
BOTH valid>=0.90 AND jr_$ra density>=0.01. The jr_$ra gate is decisive: isValid() alone flags
389 false hits (type-0/2 structured data decodes ~100% valid but has ZERO returns); jr_$ra
separates code (~2.9-3.4%) from data (0.000%), validated on positive+negative controls.
- FINDING (docs/disc-completeness.md): type-4 location overlays are COMPLETE (138/138). All other
types are data EXCEPT type-1 = 40 code payloads, 1 onboarded (the resident), 39 HIDDEN
resident-class modules (mostly MAIN.CD/FILE_XXX/1.1). They load at UNKNOWN addresses (not the
shared overlay slot), so they are NOT mechanically onboardable — byte-verifying a build binary
needs its load address (P9), knowable only by runtime RE (the Phase-3 method). Deferred with
evidence, NOT force-onboarded at a guess.
- CONSEQUENCE: game-code TRUE 100% now spans 140 onboarded binaries PLUS ~39 type-1 modules
pending load-address RE. The roadmap assumed 136 — this is a real re-baselining (the +4 overlays
also add ~2.45 MB to the denominator; every family propagation is now x138). Flows to T10/T11.
- SETUP §6.3 tool inventory updated (R21).
|
||
|
Drew T
|
e0a0becfaa |
feat(phase-27 T4): one cdecl typedef-strip primitive (was six regexes) + surface cc1 stderr
The plan named two defective regexes; the tree had SIX with complementary holes, each silently recording the resulting compile failure as "not a match" — a plumbing error wearing a compiler wall's clothes, the exact class the 26-A audit exists to end (R32). - cdecl.py: the canonical primitive — typedef_names(tu_path) + strip_provided_typedefs (draft, provided). Built on tu_statements (robust) NOT tu_scope (which coverage-asserts -> would crash the byte-gate on any unrelated unparseable file-scope statement). Splits multi-typedef lines (split_statements, depth-aware); covers scalar AND struct typedefs; keeps draft-local types. lru_cached. - harvest_verify.py: strips PER-TU (cdecl.typedef_names of the draft's real target TU) -> unblocks the 39 struct-typedef drafts the scalar-only _TD dropped. And SURFACES cc1 stderr: build() stashes it; a single-draft failure is classified DIFF / PLUMBING:… / CC1-FAIL / SKIP -> .run/harvest_failed.classified.txt. A `redefinition` is no longer recorded byte-identically to a codegen miss. - masked_diff.py: strip_scalar_typedefs() (common.h set derived from the header once, R33, cached) replaces SCALAR_TYPEDEF_RE.sub for the ISOLATED compile; wired into match_one + p16_permute. Fixes the multi-typedef-LINE skip that discarded 42 masked-MATCH drafts over whitespace. Unblocks B4's func_8015C32C (redefinition of 's16'). - canon_sig_reconcile / eval_lora / format_finetune keep their own copies — migrate per-bank, byte-gated (the audit-prescribed cadence, not a big-bang swap). VERIFIED: - HEADLINE known-answer: func_8015C030 -> MATCH (23 ins) UNEDITED via match_one (was CC1-FAIL; the multi-line typedef split alone fixes it — a live x134-family draft that was being discarded over whitespace). - unit: 7/7 scalars stripped; a local struct KEPT; a TU-provided Blk16 stripped. - classifier unit: DIFF / PLUMBING:… / CC1-FAIL / SKIP all label correctly. - all 5 edited tools import + AST-parse clean. - R22 clean-fleet: check-all 136/136; main clean-rebuild 143dbb89. (A mid-test c4546248 "mismatch" was a stale-incremental artifact from concurrent compiles, cleared by a clean rebuild — the R22 lesson; edits touch only tools/, src/ stayed git-clean.) - SAFETY: a strip bug can only fail-to-bank, never falsely bank (INCLUDE_ASM pastes the original asm; a wrong draft always changes bytes -> always fails SHA1). |
||
|
Drew T
|
ebdef9012b |
feat(phase-27 T2): make the Makefile fail-closed — the enabling fix for every downstream gate
The roadmap §5 asserted `make report` is fail-closed. It was NOT: .ONESHELL sends each whole recipe to one `bash -c`, so with no -e only the LAST command's exit survives and every earlier failure is swallowed. `dedup-check` "gated" purely by being last; lint_symbol_refs / progress --audit / difficulty / dup_report were non-gates. That is the 26-A audit's own thesis (a loud failure nobody counts is as invisible as a silent one) biting the audit's infrastructure — and until it's fixed, any R32 assertion added to a report-invoked tool is swallowed on arrival. - .SHELLFLAGS := -ec (global fail-closed). ONE documented opt-out: check-env (set +e — its contract is accumulate-every-failure-and-report, which -e would truncate at the first missing tool). - check-all:610 grep -c landmine fixed (|| true): grep -c exits 1 on zero matches, which -e treats as fatal in a command substitution -> check-all would FAIL exactly when nothing did. - check-all / extract-all: assert COVERAGE (pass == N), not the absence of a failure marker. The old `fail == 0` / `! grep -q` form was a VACUOUS PASS on an empty pipeline (R32). - new `make tools-health` = audit-corpus + audit-cdecl + report, fail-closed — the deliberate pre-matching ritual the roadmap's standing invariant names, and the dependent the two derived oracles never had (nothing invoked them). NOT a report/build prereq — audit-cdecl cross-compiles every C decl through gcc (~minutes). SETUP §6.3 documents it (R21). VERIFIED: - NEGATIVE CONTROL (the proof): a broken lint_symbol_refs makes `make report` exit 0 under the old .SHELLFLAGS=-c and exit 2 under -ec. The swallow was real, not theoretical. - the grep -c landmine + the vacuous-pass both reproduced and fixed in isolation. - check-env still exits 0 (the opt-out works); recipe sweep found the Makefile already -e-aware (set -o pipefail, explicit || true) — line 610 was the only real hazard. - R22 clean-fleet: make check-all -> 136/136 byte-identical; a forced main re-extract+rebuild drove the full splat->cpp->cc1->maspsx->as->ld->objcopy->check pipeline under -e -> 143dbb89. - audit-corpus 7s / audit-cdecl green / tools-health wired. |
||
|
Drew T
|
a33c6f85f5 |
docs(phase-26a): A11 — distill + close the tooling-integrity audit (26-A COMPLETE)
Closed the inserted half-phase. tooling-audit.md: DIAGNOSIS -> AUDIT-CLOSE LEDGER (A1-A10 outcomes + the payoff 66.5->68.6% instr + remaining/handoff); the "two rules" -> R32/R33/R34 crisp for P10 ratification at the Phase-26 PhaseEnd. decision-log: the A10 wall-re-test verdict (R31 -- the broken tools WERE the walls; the payoff was banked by the fixes; the closeness-0 residual is genuine; the real deliverable is the 3 rules + the derived-oracle pattern). SETUP: the A9d-A10 tool changes (R21). Cookbook §51 verified complete; LAW 3 tagged R34. Observables green: final R22 clean-fleet 136/136 BYTE-IDENTICAL; make report EXIT 0 (dedup 1840/0, C1 227211/227211 signed, lint_symbol_refs wired + passing); audit-corpus 0 slices; audit-cdecl green. Zero src/config changes this session. Phase 26 resumes at Task 7 (fresh session). |
||
|
Drew T
|
f9742cf9c0 |
feat(phase-26a): A3b — cdecl.py, THE C-declaration oracle: one grammar, fifteen deleted models
Fifteen tools each carried their own regex model of "what is a C declaration", and they
disagreed — two tools in ONE pipeline disagree today about whether `extern s32 D_a, D_b;`
is a declaration at all. All fifteen shared one character class,
extern\s+([A-Za-z_][\w\s\*]*?\bD_[0-9A-Fa-f]+\s*(?:\[\s*\])?)\s*;
which cannot hold '(', ',', or a non-empty [N] — so three whole shapes were invisible to
every one of them: fn-ptr/jump-table arrays, sized arrays (one unparsed `[4]` has blocked
func_801387B8 in 134 TUs), and multi-declarators (the WHOLE line dropped, not just #2..N).
REJECTED the audit's own prescription (a shape-aware alternation per tool, ~15 coordinated
regex edits) on R33 grounds: fifteen hand-maintained models are exactly what diverged, and
an alternation only ever covers the shapes somebody remembered. The thing being scanned HAS
A GRAMMAR. C's declarator grammar is small, closed and TOTAL — it describes fn-ptr arrays,
sized/2-D arrays, multi-declarators, fn-ptr params and K&R identifier-lists without being
told they exist. ~250 lines of recursive descent: LESS code than the regexes it deletes, and
exhaustive by construction rather than by memory. (decision-log 2026-07-14.)
Two statement paths, because the inputs genuinely differ:
* tu_statements() - a TU's file scope, derived from cpp. A decl inside a DEFINE_func_*
macro body declares NOTHING until the macro is invoked (the §8c law);
a raw scan is wrong in both directions. cpp answers it exactly, in
54 ms/TU (~20 s for the fleet, cacheable).
* split_statements() - span-preserving raw split, for drafts (which get rewritten).
THREE ORACLES, whole corpus — a measurement, not a belief:
* coverage 2,952,246 depth-0 statements -> 2,731,521 declarators, 0 PARSER DEFECTS
* the real gcc 50,405 distinct declarations compiled beside this parser's reconstruction
of each one -> 0 REJECTED
* differential 0 file-scope symbols the incumbents see that cdecl misses; 26 in
engine_core.h they cannot see; 6 they wrongly promote from BLOCK scope
Two ideas worth keeping (cookbook §51g, LAWS 4-8):
* THE CANDIDATE SET IS DERIVED TOO (R33 applied to R32). At file scope C admits nothing but
declarations, so R32's over-approximating detector is *every depth-0 statement* — supplied
by the grammar, with no hand-maintained candidate regex to rot.
* GCC ADJUDICATES MY OWN COVERAGE GAP. Deciding for myself which failures "don't count" is
grading my own homework — the habit that wrote the fifteen bugs. A statement gcc ALSO
rejects is not C (my rejection is correct, the INPUT is corrupt); one gcc ACCEPTS and I do
not is MY defect. All 33 residual: NOT-C, all dead .run/drafts* scratch, none in src/.
NEW findings (docs/tooling-audit.md):
* reconcile_decls.DATA_DECL_LINE_RE finds ZERO decls in engine_core.h — it is line-anchored
and every decl there ends in a '\'. Its "authoritative tier" has ALWAYS been empty.
* gen_harvest_targets + sig_unify count BLOCK-SCOPE externs (6, byte-proven inside a macro's
function body) as file-scope canonicals — the §8d `conflicting types` confusion.
* tu_ambient's func regex ([^()]* params) drops ANY callee with a fn-ptr parameter.
* R14 near-miss: 33 drafts contain `extern if ((func_80029178(0x119) & 0xFF) != 0);`, written
by a RECOVERY TOOL — but the source bug was already fixed in Phase 19 (0 garbage / 300 sigs
today). Mechanism confirmed, consequence nil. Note what it cost while live: a draft that
cannot compile fails the byte-gate and reads downstream as an INTRINSIC COMPILER WALL.
Bugs the oracles caught in ME (and would otherwise have shipped): `extern s32 (*D_801274D0)(s32);`
parsed the BASE TYPE as the name; a K&R declaration-list flushes as SEVERAL spans, so the body
attached to the wrong one and leaked the K&R parameter names into file scope as fake globals.
SCOPE, deliberate: NO consumer is migrated here, so this cannot move a byte. The audit warns
that making the parser see more ARMS dormant transforms (reconcile_decls.data_access_subs would
mangle `D_1[i]()` -> `((u8 *)D_1)[i]()` the moment fn-ptr decls become visible to it). Migration
is one tool at a time, each byte-gated.
R22 clean-fleet: make clean + extract-all + check-all -> 136 passed, 0 failed of 136
make audit-corpus: 0 PHANTOM + 0 TRUNCATED make audit-cdecl: ALL ORACLES GREEN (new gate)
|
||
|
Drew T
|
1ab9905368 |
feat(phase-26): §8d scope_data_externs — the ×133 sweep blocker fixed; func_8015AE2C banked ×134
- ROOT CAUSE (R14 — the session-7 diagnosis was half right): the isolated region builds [ OK ] WITHOUT the body, so §8b isolation was never implicated. `family_remap.gather_externs` prepends carried decls at FILE scope; D_801812A4 is a fn-ptr dispatch table the sibling declares FOUR incompatible ways at BLOCK scope inside its own later functions, so the carried file-scope decl ESTABLISHES A GLOBAL THE TU NEVER HAD and every later block-scope extern must now agree with it. Byte-proven asymmetry: BLOCK(int)->BLOCK(struct*)->FILE(void*) builds; FILE(void*)->BLOCK(int) errors. It was the ONLY hard error in the build — all 27 carried function externs were fine raw. - THE FIX (demote, don't reconcile): tools/scope_data_externs.py emits a carried D_ extern at BLOCK scope inside the function body when the TU has no file-scope decl of it above the insertion point. Byte-neutral (an extern emits no code; type + access opcodes unchanged) and never worse than raw, so it needs no oracle, no type comparator, no fn-ptr parser. Restores fidelity — the original declares these symbols at block scope in exactly this way. Wired into jtbl_family_bank as the `scoped` stage: raw -> scoped -> recovered -> reconciled (scoped is the base for the later stages). - reconcile_decls is the WRONG instrument for this class, twice: its oracle answers "what does the FLEET call this symbol" when the question is "what can THIS TU see", and its DATA_DECL_LINE_RE cannot parse `extern void (*D_x[])(void *);` — silently skipping the very symbols that were failing (the phase's third silent-skip bug, after find_site braces + overlay_files splits). - R17 TRIAGE RULE, first real test, held: `conflicting types` = the compiler REFUSED TO COMPILE = a C front-end diagnostic = our Python. Reading cse.c/global.c would have taught nothing. - RESULT: func_8015AE2C (562 ins, reach 134) swept 133/133 siblings, 0 failures. R22 clean-fleet 136/136 BYTE-IDENTICAL (534 changed src files); dedup-check 1813 validated / 0 failed; 0 NON_MATCHING (G4). instr-weighted 63.0 -> 63.6%; distinct-code 39.1 -> 40.5% (+256 unique fns / +79,957 ins) — one core, ~0 agent tokens. - knowledge captured during the producing session (R30/R31/R21): cookbook §8d, decision-log 2026-07-13 session 8, SETUP tool-inventory row; CURRENT_PHASE session-8 checkpoint. |
||
|
Drew T
|
095a611e75 |
feat(phase-26): §8 jtbl-rodata tooling — overlay PoC proven (func_8012ACE0, R22 136/136)
- overlay jr-functions can now bank as C: gcc switch jump tables form a .rodata island at the overlay TAIL; carve a matched fn's jtbl into a dotted [.rodata, <code-subseg>] subseg + ld_interleave (data->rodata->data sandwich) places it byte-exact. cookbook §8a + SETUP. - tools/ld_interleave.py: --section .<binary> param (derives the <binary>_TEXT/DATA/RODATA/ DATA2/BSS symbol prefix); default .main = the EXE, byte-identical (backward-compat proven) - Makefile + config/overlays.mk: <bin>_JTBL_INTERLEAVE hook + a $(strip)-guarded extract branch (gotcha caught: a trailing #comment on the := left whitespace -> non-empty -> the branch misfired on resident with the EXE defaults) - PoC: func_8012ACE0 (25-ins jr-fn in ov_SC01_077) reconciled (canon_sig_reconcile) + banked BYTE-IDENTICAL d19c9580 -- the first overlay jr-function matched through the C pipeline - R22 FULL-FLEET clean rebuild: 136 passed, 0 failed (main 143dbb89 unaffected by the ld_interleave change); 0 NON_MATCHING in any default build (G4) - P9 findings: func_80159C84/func_8015444C (the 2 carried Fable5 jr bodies) are rtu_match FALSE-matches (incomplete jtbls: 52B vs 56B -> never bank); the maspsx "hang" scare was a truncated experimental-file artifact (real pipeline builds in ~1s) - metrics: distinct-code 39.1% (50,572 unique fns), instr-weighted 62.9% - NEXT: the ×134 automation (generate the per-overlay carve + template the reconciled body) |
||
|
Drew T
|
c0379f0738 |
feat(phase-25): progress.py --weighted — byte/instruction-weighted metrics (the honest headline numbers)
- weighted_metrics() from .run/sig.*.jsonl + src stubs (executable code only, resident + 134 overlays; main EXE excluded). Two framings: fleet instr-weighted (per-overlay, the decomp.dev -display number) + dedup distinct-code (each unique h_exact once, the distinct-RE number) - --fleet now emits THREE labeled metrics into docs/progress.fleet.md: fn-count 74.48% (×134- inflated), instr-weighted 56.8% (shipped .text), distinct-code 27.3% (of 84,996 unique fns) - --weighted prints the two weighted numbers standalone; degrades gracefully if sigs absent - corrects the stale "~30-35% byte-weighted" estimate: the giant campaign since Phase 19 raised the fleet instr-weighted number to 56.8%; the distinct-code 27.3% is the unique-monster-tail truth - SETUP §tooling row updated (R21) |
||
|
Drew T
|
8fa29f2dda |
feat(phase-25): T7 F-band wave 3 part 2 — corrected rtu_match fan-out, 7/9 cracked + 266 swept; fleet 73.89->73.97%, R22 136/136
- NEW tools/rtu_match.py: real-TU-faithful, PARALLEL-SAFE per-fn match check. Compiles the WHOLE split TU (candidate spliced, INCLUDE_ASM neutralized via -DINCLUDE_ASM(a,b)= + -Isrc/<source>) in a per-fn temp dir -> no asm/, no shared overlay build. Closes match_one's isolation blind spot (in-TU decl/global-type/memcpy-builtin drift) so a MATCH HOLDS at the whole-binary gate. - Corrected Ultracode fan-out (wf_80762f2c-822, 9 xHigh workers): 7/9 real-TU MATCH -> all 7 banked byte-identical (individual + combined d19c9580), ZERO drift (vs wave-2's ~50% attrition). Banked: func_8012FCC4 func_80133AB0 func_80134A74 func_80136824 func_8014DD8C func_80168828 func_8016C188. func_8012FCC4's "irreducible" delay-slot was a wrong-callee-arity bug. - Swept x134: 266 siblings banked / 266 failed (the 2 //@EDIT families func_80133AB0/80136824 can't per-sibling-reconcile via family_sweep -> backlog: a --edit-remap enhancement). Batch = 7 exemplars + 266 = 273 fns. - func_8014DD8C: engine_core.h DEFINE_func_8014D790 extern void->s32 flip (byte-neutral fleet-wide, caller discards return) -- R22-confirmed neutral across all 136. - 7 durable levers -> cookbook §42c (callee-arity delay-slot, pointer-global *(T**)&, array-decay CSE, §17 zero-reg copy, void->s32, register-arg capture, free-floating load hoist). rtu_match -> SETUP §6. - 2 DIFF -> permuter (func_801670E4 70->48, func_80185BA4 c=65), seeds .run/crack3/wave3/. - R22 clean-fleet 136/136 BYTE-IDENTICAL from a fully clean tree; NON_MATCHING 7 (0 in default build, G4). |
||
|
Drew T
|
8cfbbf2d47 |
feat(phase-24): T7 whale func_80144B9C banked ×134 (-O0 reach-134 rollout) — clean fleet 136/136
The single biggest byte-weight lever (770 ins ×134 ~ +1.6% byte-weight), banked across all 134 overlays as -O0 compiled C (was ×1 in ov_SC01_077, commit:0463). - ROLLOUT (tools/rollout_whale_o0.py): per single-file overlay, LINE-split <ov>.c at the whale (splat emits in vram order), carve the yaml code subseg into before/<ov>_o0b(-O0)/<ov>_after, thin <ov>_o0b.c #includes the shared src/shared/func_80144B9C.h (the whale C moved there — DRY, TU-local typedefs, not a fragile 200-line DEFINE_ macro). Makefile WHALE_O0B_OBJS wildcard -O0. - DEDUP: group E_func_80144B9C (134 members, h_exact 74186b97, source=the header; group_members keys on binary+vram so a header-share validates like a macro-share). - MILESTONE: clean fleet check-all 136/136 (R22); dedup-check 1811/0; FLEET REAL 225816->225949, byte-identical 65.91->65.95% function-count (the gain is byte-weighted). Validated on ov_SC01_000 + spot-checks SC02/SC03/SC07 before the full fleet. - cookbook §38 + SETUP inventory. |
||
|
Drew T
|
994368fbcd |
feat(phase-24): T7b/T5 — 2nd giant func_80138ED0 MATCHED + banked ×134; fleet 65.52→65.56%; §34 idioms
- func_80138ED0 (159 ins, reach-134): Opus applying §32 -> close=21 (all semantics/control-flow/ constants exact, +the giv-init fence lever) -> Fable5Max reading the vanilla gcc-2.7.2 source -> MATCH (every regalloc/sched class C-reachable, no permuter). - banked via the standard pipeline (validates it on a real un-reconciled sibling): cast_call_sites reconciled func_8013914C (u8*,u16*)->(s32,s32)+cast; reconcile_decls a NO-OP (2 data bases giant-local -> no fleet conflict, confirms no false-positive); harvest_verify ×1 d19c9580 -> dedup_propagate --recover ×134 (pins/asm body propagates fine). clean fleet check-all 136/136, dedup 1800->1801/0, 0 NON_MATCHING. - NEW §31 idioms (cookbook §34 + regalloc.md RC-11 + loop.md): gcc-2.7.2's 3-qty local-alloc SORT BUG (local-alloc.c:1441 — <=3 qtys allocate in creation not density order; decoy-qty fix) + the zero-byte asm allocation toolkit (input-only/multi-input/def+use dummies) + the giv-init fence + gdb-on-cc1. gcc-2.7.2 reference tree completed (18 .c); SETUP §5.6. - T5 satisfied: reconcile_decls proven on 2 real giants (func_80129CF8 reconcile-path byte-proof + func_80138ED0 no-op/full-pipeline). CURRENT_PHASE T5 logged. |
||
|
Drew T
|
787b756e6d |
docs(phase-24): T7b — distill reconcile_decls (cookbook §33, SETUP, design-doc RESOLVED); R14-correct §32 wall
- cookbook §33: the reconcile_decls mechanism (oracle + byte-neutral cast taxonomy + pipeline
placement + the func_80129CF8 byte-proof); §32 'BANKING WALL' note R14-corrected (×134 was free,
the wall was only the ×1 loose-draft reconcile).
- SETUP.md: reconcile_decls tool-inventory row.
- canonical-decl-reconcile-design.md: ✅ RESOLVED banner — the premise reframe (automation not
unlock), the answered open questions, T4 no-op, the validation.
- CURRENT_PHASE: T7b/T2+T3 progress-log entry; NEXT = T5 sibling (func_80138ED0) via Fable5Max.
|
||
|
Drew T
|
0cda15c6de |
fix(phase-24): T5c — restore main's clean-rebuild: propagate the Phase-21 PsyQ renames to 62 stub refs
Pre-existing latent breakage (Phase-21 close commit:0292, "+62 PsyQ names"), found during T5b's R22 fleet verify. A genuinely-clean `make clean` + re-extract + build of `main` failed with dozens of `can't open asm/nonmatchings/800c3/func_*.s` — clean-rebuild broken since Phase 21, masked all along by incremental builds reusing stale pre-rename .s/.o (the exact R22 failure mode). - Root cause (byte-proven): Phase-21 xdedup renamed 62 PsyQ library functions to their proper names (InitHeap, FlushCache, GetTPage, SysEnqIntRP, SpuWrite, CdMix, __main …) in symbols.us.txt, but never regenerated/renamed the committed src/*.c stub refs that call them by the OLD func_<ADDR> name. splat's FRESH regeneration of a stub .c uses the CURATED names (move src/800c3.c aside -> splat writes INCLUDE_ASM(InitHeap) + emits InitHeap.s); the committed stubs were simply stale. - Fix: rename all 62 INCLUDE_ASM(func_<ADDR>) -> the curated name across 12 files (800c.c 1, 800c3.c 22, apicard1/2/4 4/6/2, boot.c 1, libcd1.c 8, libetc.c 6, libgpu.c 3, sgap.c 1, snd1.c 7, snd2.c 1). Pure rename to match splat's canonical output; byte-neutral. - VERIFIED: main clean-builds 143dbb89 from a fully clean tree; full clean fleet check-all 136/136. - Lesson (cookbook): a symbols.us.txt rename must be propagated to (a) shared-macro bodies (engine_core.h — the T5b GetTPage fix) AND (b) INCLUDE_ASM stub refs (this), AND verified by a genuinely-clean check-all (make clean + full re-extract), never incremental. Added a lint check. |
||
|
Drew T
|
0eadf313ee |
feat(phase-24): T5 — §31-directed permuter mutation + grinder input-changed gating
- tools/permuter_weights.py (NEW): classify(klass,where) -> regalloc|schedule|cse|None + render_settings_toml emitting the [weight_overrides] table decomp-permuter merges over the gcc defaults (main.py:336 / helpers.py per-key replace). Biases pass-selection toward each class's §31 levers (perm_reorder_decls RC-1/3, perm_reorder_stmts RC-2/S1, perm_temp_for_expr S2, perm_commutative cse); value/type noise -> ~0.1. NO submodule edit. - p16_permute.setup(..., klass=, where=) writes it (+--klass, backlog auto-lookup); grinder.py auto-threads klass/where_stuck. klass=None -> gcc defaults (undirected superset). - grinder fix (R14): replace the blind tried.clear() idle churn with input-changed gating (draft_sig=(best_draft mtime, closeness) — re-open a fn only when the worker improved it). - VALIDATED: regalloc profile drove func_8014E048 masked 36->29 (match_one -drz 35->28) where the undirected search stalled; re-logged (28, improved draft promoted). Flagship S11/RC-6 count-exact seeds improve but do not gate (intrinsic, §31 "two probes dont grind") -> T5b fuel. - distill: cookbook §3b + SETUP inventory (R16/R30). No build-input changed -> 136/136 untouched. |
||
|
Drew T
|
3b6f2c5f94 |
docs(phase-23): §31 — the gcc-2.7.2 codegen map (4 Fable5 agents read the source)
- docs/gcc-2.7.2-map/{sched,regalloc,loop,cse_expr}.md: source-cited, byte-proven
pass -> residual -> C-lever catalogs (935 lines). cookbook §31 = the index + triage.
- WALLS BROKEN (byte-proven steerable, were "CONFIRMED unsteerable"): §10/§20
hoist-vs-remat (cross-call address-caching), store-vs-load (/s), dbr delay-slot
(D1+S2 fresh-local: func_801770E0 53->49), birthing-boost both directions.
Incidental bank: func_80149374 x134 (fleet 64.86%).
- GENUINE walls -> permuter: S3 chain-priority sink, S11 LUID(x)alloc coupling,
RC-6 pressure-lock, cse 1000-insn table flush.
- CORRECTION (loop agent): tools/reference/gcc-papermario is gcc 2.8.1 NOT 2.7.2
(behavioral biv-elim diff). Vanilla gcc-2.7.2 -> tools/reference/gcc-2.7.2/
(gitignored); SETUP §5.6 + §31 flag it. All banked levers stand (match_one-validated
vs the real cc1). Also corrected: spill-slot = declaration order; §25 4th rank rule.
- the payoff: the cheap tier (Opus agents + local model) can now apply compiler-internal
levers by triage-table lookup, without reading 80k lines of source.
|
||
|
Drew T
|
e9f199530e |
docs(phase-23): T10 harvester built + measured — 65% bank-rate, parallel gate ~75x, vLLM justified
- gen2-mips-matching-model.md: T10 RESULT (bulk_harvest phase-separation; 52/80=65% on fresh SC03 <=15-ins; gate 0.4s/fn/8w vs ~30s/fn serial; drafting now the sole bottleneck -> vLLM) - SETUP.md: bulk_harvest.py inventory row + gate_stage/harvest_verify parallel-safe params (R21) - CURRENT_PHASE.md: T10 done (built+measured), T10.5 vLLM split out as Drew-gated; progress log |
||
|
Drew T
|
205fc202e9 |
feat(phase-23): macro-extern-injection lever — 8 reach-134 fns freed ×134 (fleet 63.82→64.09%)
- tools/inject_capped_externs.py (new): source each propagation-capped reach≥2 inline match's externs from ov_SC01_077's own file-scope decls, inject them block-scope so the lifted macro is self-contained (the Phase-20-backlog macro-extern-injection lever; byte-neutral, fail-safe via compiles_standalone) - 8 reach-134 fns freed from ov_SC01_077.c, 077 byte-identical; dedup_propagate --auto-from registered 7 new shared groups ×134 (1678->1685), check-all 136/136 - the LLM reach≥2 measurement run already banked +7 (v3 banks reach≥2: 6/15 in 005 vs v2's 0/15) — committed earlier (commit:0359, commit:0360) - cookbook §28d + SETUP tooling inventory (R16/R21/R30) - fleet 63.82% -> 64.09%, 0 NON_MATCHING, dedup-check 1685/0 failed |
||
|
Drew T
|
1eadbc73e9 |
docs(phase-23): 8-hour autonomous run — corpus-v3 + v3 + local serving; fleet 63.67->63.82%
The run (Drew away): built tools/serve_local.py (Unsloth GPU serving, replaces ejected LM Studio); the prompt fix (translate-every-instruction, 0/3->2/3 on the empty-leaf class); corpus-v3 (mine the 1623 engine_core.h DEFINE_func macros + struct types: 1312->2891 pairs, 2534+291 trainable); v3 trained (loss 1.275->0.085) eval 23/40 (57.5%); v3 production batch banked ~352 fns + 45 new shared groups -> fleet 63.67->63.82% (+502 byte-identical), 136/136 byte-clean, 25 auto-commits, $0. v3 banks the empty-leaf/setter class v2 couldn't. - docs/gen2-mips-matching-model.md: corpus-v3 + serving + prompt-fix + v3 eval + harvest - docs/SETUP.md: LLM matching tier added to the Tooling inventory (R21) - phase-ends/CURRENT_PHASE.md: run progress + next levers (reach>=2 targeting, dedup-collapse) |
||
|
Drew T
|
1f4aac62fb |
docs(phase-21): cont.8 — collaboration sweep (clean verdicts) + 62 PsyQ names + SETUP -G fix
- reach-1 idiom loop: +16 ×1 matches (commit:0289, commit:0291) + chained-store-fanout idiom (commit:0290); giants pivot 0 auto-banks (2× = the byte-proven wall) -> 6 ranked hand-finish near-misses in backlog (2 close: func_8015126C close=2, func_8014EE14 close=6) - collaboration sweep CLOSED with clean verdicts (5 scouts): parity research (we're SOTA; no neural/GPU/diffusion shortcut), decomp.dev survey (4 gcc-2.7.2-psx twins; VS=CC0 the clean pool partner), census (no Square symbols; Croc=SDK-naming only), VS corpus (76 byte-verified pairs, low-priority asset), dedup probe (CLEAN NEGATIVE — BFM shares only PsyQ with sisters, AKAO refuted, 0 engine code) -> no external shortcut to the gcc wall - config/symbols.us.txt: +62 PsyQ library/BIOS names from cross-project byte-signature dedup (xdedup vs VS/Xenogears); byte-safe (main still 143dbb89, R22-verified) - docs/SETUP.md §5.3: corrected stale "-G TBD" marker -> RESOLVED Phase 5 (-G0) (R21) - docs/backlog.md: giant near-misses logged with true worker closeness (cont.7d-safe) - CURRENT_PHASE.md cont.8: session log + 3 parked R&D items + process lessons - fleet 63.18%, 136/136 byte-identical, 0 NON_MATCHING |
||
|
Drew T
|
6c5bd1c4a9 |
feat(phase-20): cast_call_sites.py — the §17a-1 callee-cast cap tool; 6 recovered, fleet 58.63%->58.82%
- tools/cast_call_sites.py: per draft, callee whose canonical TU sig != draft's intended sig -> decl->canonical (kills in-TU conflicting types, keeps symbol in scope) + call-site cast ((ret(*)(args))func_X)(args) (codegen-neutral: gcc folds the cast of a known symbol -> direct jal). Pure --in/--out; whole-binary gate is the arbiter (G3/P9). Pipeline: canon_resident_calls -> cast_call_sites -> sig_unify -> harvest_verify --chunk 1 - recovered 6 of T6 batch-1's 33 gate-fails byte-identical (func_80153C44/8015CF58/ 801711FC/80161BE0/801683D8/8015F948); 5 propagated x134 + 1 local; check-all 136/136 - R14 CORRECTION of cookbook §20's '~33' projection (byte-proven): the batch is NOT mostly callee-conflicts — ~6 callee (cast-recovered), ~18 DEF-side loose-typing wall (def's matchable sig conflicts with the unchangeable caller-canonical; caller-side fix BLOCKED because INCLUDE_ASM declares no symbol -> a shared macro's internal extern is load-bearing in the 133 stub overlays), + a small DATA tail. The def-side needs re-drafting under the caller-canonical sig (future wave), not a text transform - cookbook §20 RESOLUTION + SETUP tool inventory (R16/R21/R30); CURRENT_PHASE log (R28) - fleet 58.63% -> 58.82%, dedup 1501 -> 1508, 0 NON_MATCHING (G4) |
||
|
Drew T
|
2070cdb000 |
docs(phase-20): T4 — distill the codegen map (cookbook §20) + R14 loop-guard correction
- cookbook §20: the wave-at-scale GATE CAP (match_one over-predicts; the in-TU loose-typing call-graph wall; all recovery levers fail; the only fix is automating §17a-1 call-site casts = Phase-21 cap lever) + the stale-.o diagnostic lesson - CONFIRMED-irreducible classes w/ byte-evidence: §10 store-vs-load (8014F2E0/80150528), §10 hoist-vs-remat (80149374/801493D0); NEW classes: IV-combine combine_givs (80177AD4), hoisted-invariant prologue order (80177F84), the -O1 build-infra class (80161A90) - operational gotchas: Workflow args-as-array, harvest_verify --chunk 1 - R14 correction to §17a: loop-guard (8012C2D0) is STRENGTH-REDUCTION/IV-final-value addressing, NOT operand-order - SETUP §tooling: exemplar_miner.py (T2) + build_engine_types.py typedef-lift (T1) (R21) - T3c cited (written up), T4 done; CURRENT_PHASE handoff refreshed |
||
|
Drew T
|
de35505a63 |
feat(phase-19): scale the toolkit — 2 waves (88%/92%), fleet 56.64%->58.00%, propagation is the cap (v1.18.0)
- T1 (prior session, commit:0158/commit:0159): per-file -O0 split infra + 6 -O0 matches; deferred the 10 %lo-folding-quirk fns + the ×134 rollout (R14 corrected "free ×134") - T2 (commit:0161): recovery tooling. R14: the 3 Phase-18 "implicit-int" near-misses are NOT implicit-int -> resident-callee link-miss (NEW canon_resident_calls.py) + shared-caller arity (no-proto caller-extern fix). 3 matches; cookbook §17a-3 corrected - T3 (commit:0162/commit:0163/commit:0164): 2 Ultracode waves over reach-134 tractable residuals. batch-1 44/50 match_one (30 banked, 25 propagated ×134); batch-2 35/38 (25 banked, 10 propagated). NEW fix_arity_callers.py (no-proto, ALL 18 batch-2 failures were the arity class); canon-first 2-stage gate (sig_unify REGRESSES canonical drafts); gen_harvest_targets garbled-sig fix (0/749, 88%->92%); build_engine_types --strip lift (+4 propagated) - FINDING: propagation (not matching) caps the fleet-% yield — typedef/local-type bodies can't auto-propagate (-> Phase-20 type-lift lever). decomp.dev-comparable byte-weighted est ~30% - check-all 136/136 BYTE-IDENTICAL (R22); dedup 1450->1485/0 failed; 0 NON_MATCHING (G4); fleet 56.64% -> 58.00% (+1.36%). worklog -> phase-ends/logs/Phase19.md (R19) - bumps 1.17.0 -> 1.18.0 |
||
|
Drew T
|
85afa0dc81 |
feat(phase-19): T2 recovery tooling — canon_resident_calls + no-proto caller fix (3 matches)
- R14 finding: the 3 Phase-18 "implicit-int / propagate-first" near-misses are NOT
implicit-int (reproduced each through the byte-gate). Real classes:
(a) resident-callee LINK-miss: draft calls func_<ADDR> but the addr has a curated
name (0x8004CFEC = ratan2) -> "undefined reference to func_8004CFEC". NEW
tools/canon_resident_calls.py rewrites func_<ADDR> -> curated name (pure draft
text, body bytes unchanged) -> run FIRST in the recovery pipeline.
(b) shared-caller ARITY conflict: an engine_core.h caller macro declares the callee
file-scope `extern void func_X(void)` then calls func_X(); the real def takes an
arg -> "conflicting types". FIX = caller extern -> no-proto `extern void func_X();`
(byte-neutral for the caller; compatible with promotion-safe params int/long/ptr).
- 3 matches banked in ov_SC01_077: func_8017209C (ratan2 rewrite), func_80147514 (s32),
func_80168F40 (void*); 2 no-proto lines in src/shared/engine_core.h
- recovery pipeline: draft -> canon_resident_calls -> sig_unify -> harvest_verify --chunk 1
- make check-all 136/136 BYTE-IDENTICAL (full extract, R22); dedup 1450 validated/0 failed;
0 NON_MATCHING in any default build (G4); fleet 56.64% (the 3 are ov_SC01_077-local,
x134 propagation rides T3/Phase-20)
- cookbook §17a-3 corrected (R14/R16); SETUP tool inventory (R21); CURRENT_PHASE T2 done
|
||
|
Drew T
|
d20b20c64b |
feat(phase-18): T0 — setup + exemplar set; match_one validated as the §10 oracle
- clone gcc-papermario (a6afc2af) + xenogears-decomp (f27c0768) -> tools/reference/ (gitignored research refs; SHAs pinned in docs/SETUP.md §5.6, R20/R21) - apparatus byte-validated: match_one cleanly measures the §10 residual on both quirk exemplars (the floor-free oracle Phase 17 failed to use; permuter score was polluted): - func_8012B8E4: 75=75 ins, 24 mismatched -> $s0/$s1 regalloc swap + reassociation (§10 A) - func_8012B4B8: 94 vs 84 ins, 88 mismatched -> extra-reg spill + struct-copy/reload (§10 B) - CURRENT_PHASE.md crash-recovery log; harness task list (R28); no build-affecting change |
||
|
Drew T
|
f0dd9351e4 |
docs(phase-17): flywheel + handoff up to date — canonical-sig wall is the scaling lever
- CURRENT_PHASE.md: new session-4 START HERE (build the canonical-sig layer, then scale the wave); old demo-protocol START HERE marked superseded - hand-matching-process.md S7: the Ultracode harvest wave (pipeline, calibration result 60% match_one / 33% whole-binary / +0.47%) + THE CANONICAL-SIG WALL (the 60->33 gap is 100% sig conflicts; surgical per-callee canonicalization = ~2x lever) - matching-cookbook.md S16: guided-hand-matching breakthrough + new idioms (mask-local, shared-ret0 goto, v0/v1 coalescing) + the wave/canonical-sig finding (R16 flywheel) - SETUP.md: DecompileFunctions.java (headless batch Ghidra-C pre-pass) in the inventory (R21) |
||
|
Drew T
|
964cec8b61 |
docs(phase-15): close — PhaseEnd v1.14.0 (fleet 54.48%), continuation backlog -> Phase 16
- PhaseEnd_Phase15.md: synthesis of the open-ended overlay long-tail phase (136 binaries byte-identical from clean; fleet 3.82% -> 54.48%); CURRENT_PHASE.md archived -> logs/Phase15.md (R19) - the final-session finding: deterministic recovery (sig_unify + find_site fix) beat a 50-agent wave ~7x/token (+2.67% vs +0.36%); cookbook §14d (the recovery) + §14e (() / m2c-struct dead-ends) - Continuation Backlog -> Phase 16 'Crack the struct-heavy shared core': the actor-struct -> m2c-context -> permuter compute pipeline (+ multi-day autonomous run), the 28 giants, decomp.me, emulator capture, ML/CUDA wildcard, the arity-conflict dead-end - SETUP tool inventory (sig_unify + recovery tools); gen2-roadmap Phase-15 CLOSED + Phase-16 added - bumps project version 1.13.0 -> 1.14.0 |
||
|
Drew T
|
c716eb15e1 |
feat(phase-15): T4 — fleet progress roll-up + fix cross-report double-count
- progress.py: refactor to report(binary) + set_binary() + a per-binary .s index (fleet run 6m38s -> 7s); add --fleet -> docs/progress.fleet.md (deterministic, source-derived per-binary table + fleet totals). Single-binary output byte-unchanged. - fleet now: 947 REAL / 959 LINKED / 13132 byte-identical / 344010 matchable = 3.82% across 136 binaries (main + resident + 134 overlays) - dup_report --cross: HONESTY FIX (R14) — an onboarded overlay is both a named BINARIES entry AND a .run/sig.ov_* glob hit; it was counted twice, inflating collapsible bytes to 58.7M once the fleet was onboarded. Dedupe by alias -> accurate 9366 h_exact groups / 28.6M collapsible (matches the Phase-11 figure) - Makefile: wire progress.py --fleet into make report (main-only block) - docs: progress.fleet.md (new), duplicates.cross.md regenerated, SETUP inventory - make report green; dedup-check 8 validated / 0 failed - ghidra/ churn NOT staged (R23) |
||
|
Drew T
|
0c7619231d |
feat(phase-15): T2 — dedup_propagate.py (match-once -> propagate-many) proven on wave
- tools/dedup_propagate.py: lift a matched body -> DEFINE_func_<ADDR>() macro in src/shared/engine_core.h -> instantiate in place at every onboarded overlay sharing that h_exact -> snapshot + per-overlay byte-gate (fail-closed revert) -> register in config/dedup.us.yaml. Keyed by addr-int (sig lowercase vs splat func_%08X). - proof: 4 medium funcs propagated across all 16 onboarded overlays; clean check-all 18/18 byte-identical; dedup_integrate 8 validated/0 failed; idempotent; negative test (corrupt shared body) -> make check FAILS then restores - Makefile: header-dependency tracking (cpp -MMD -MP + -include ) so editing a shared header recompiles dependents — closes a stale-build/false-pass gap the negative test exposed; output-neutral (R22 clean 18/18) - structural self-check: a leftover INCLUDE_ASM stub is byte-identical, so the byte-gate can't catch under-application -> assert DEFINE present + stub gone - FINDING: 577 of ov_SC01_077's 785 matches are h_exact across all 134 overlays (2.19 MB collapsible, already matched) -> T5 = bulk-propagate these + harvest the rest - cookbook §14 (the propagation runbook + gotchas), SETUP tool inventory - ghidra/ churn NOT staged (R23); build/asm/.run gitignored |
||
|
Drew T
|
52514b5f45 |
docs(phase-13): T6 docs — cookbook §13 overlay runbook + SETUP/gen2-roadmap + digests
- cookbook §13: the canonical add-an-overlay runbook (new_overlay.sh, the flat-blob config, the non-4-aligned bin-carve/asset/trim gotcha, A->B->C workflow, dedup-credit two shapes, build-all/check-all) — the Phase-15 fleet recipe - SETUP tooling inventory: tools/new_overlay.sh (R21) - gen2-roadmap Phase-13 milestone: ACHIEVED amendment (6 binaries byte-identical, 704 harvested, dup-pair collapse, T7 effort finding) - regenerated per-overlay progress/difficulty/duplicates digests + cross report |
||
|
Drew T
|
67592993ea |
feat(phase-13): T3+T4 — new_overlay.sh + dup-pair & free-roam onboarded byte-identical
- tools/new_overlay.sh: one-command overlay onboarding (template instantiate + @TRAILING@ carve + overlays.mk + 4-dict sentinel-insert + extract/build/check); idempotent (re-run = no-op, no dupe entries) - T3: ov_SC03_001 (free-roam, 815,887 B) byte-identical f8fd92f5 - T2 (onboard): ov_SC01_005 + ov_SC01_006 (byte-identical pair) both 56760dbe - template tokens unquoted (correct YAML int types); SETUP §2.8 MCP-reconnect note (R21) - 6 binaries now build byte-identical (main, resident, 4 overlays) |
||
|
Drew T
|
190afe10f0 |
docs(phase-11): T7 close-out — 4.7 sha-record, cookbook §11, SETUP §6.8, README/worklist
- tools/psyq/CHECKSUMS.sha256: integrity record for psyq-4.7-converted.zip (R20) + Phase-12 provenance (the resident is 4.7 -> link from conv47/, not the EXE's 4.0 libs; R24) - cookbook §11: the cross-binary dedup & code-share workflow — source-level macro share (NOT an object swap; game fns are interior to one object/binary, D1), the byte-gate, sig_image notes (h_exact workhorse / self-consistent h_norm D2 / linear-partition+code-end overlay boundaries), the cross-report as the Phase-12/13 work queue, per-binary SDK provenance (R24) - SETUP §6.8 + tooling inventory: sig_image.py, dedup_integrate.py, dup_report --cross, make sig-overlays, config/dedup.us.yaml + src/shared/ (R21) - docs/psyq-worklist.md: resident PsyQ is 4.7 (Phase-12 linking note) - README: Gen2 status — resident is the 2nd byte-identical binary; cross-binary dedup live (~9000 cross-binary groups / ~28 MB collapsible; one engine fn byte-identical in all 134 overlays) |
||
|
Drew T
|
702b94fd7d |
docs(phase-10): close-out — PhaseEnd_Phase10 + flat-blob recipe (v1.10.0)
- PhaseEnd_Phase10.md: resident engine blob byte-identical from source (8e17e02f) at 100% INCLUDE_ASM; the binary-agnostic toolchain proven on a real 2nd binary; main still 143dbb89 - docs/SETUP.md §6.7: the resident first-instantiation + the reusable flat-blob <bin> recipe (per-binary OBJS prune, build_path=build, flat config, leading-data-word-as-rodata); 2 new Ghidra tools in the inventory (R21) - rule R24 (per-binary compiler/SDK provenance — resident is PsyQ 4.7 vs EXE 4.0) - worklog archived CURRENT_PHASE.md -> phase-ends/logs/Phase10.md (R19) - bumps project version 1.9.0 -> 1.10.0 |
||
|
Drew T
|
6345b5012a |
docs(phase-9): T10 — binary-agnostic toolchain docs + final milestone proof (R16/R21)
- SETUP.md §6.7: the binary-agnostic toolchain (make build BINARY=<alias>, the required --vram-base/--exe/--symbols flags, BFM_BINARY for asm-differ, and the future-binary path convention config/splat.<bin>.yaml / build/<bin>/ etc.) - matching-cookbook.md §9.7: the reusable refactor pattern for Gen2 (required params/ no defaults, --vram-base scalar vs window, transitional-default + leaf-first technique, the negative control, the progress.py regex lockstep gotcha) - psyq-worklist.md: per-library recipe updated with the now-required flags - README.md: status -> 52 REAL / 959 LINKED / ~50% byte-identical / Gen2 underway (binary-agnostic toolchain) - MILESTONE (all green): EXE rebuilds 143dbb89 with AND without SDK objects; report 52/959/7/50.24%; wrong --vram-base -> cae22f7e (negative control); expected/ refreshed |
||
|
Drew T
|
20a0d7a4d2 |
docs(phase-8): cookbook §9.6 + SETUP tool inventory (R16/R21 flywheel)
- matching-cookbook §9.6 "Scaling library linking to the whole EXE": the survey-first approach, gen_lib_subsegs.py + the section-size boundary gotcha, the combined-region pattern for interleaved libs (make_snd_used/make_apicard_used, alias dedup by byte-match), scattered-.bss cross-object exclusion (by address), the incremental-.ld clean-rebuild gotcha, and the LINKED honest-metric note - SETUP.md: add the 3 new Phase-8 tools to the tool inventory (R21) |
||
|
Drew T
|
6f248462dd |
chore(gen2-prep): Gen2 roadmap + private-repo backup hardening + SETUP/submodule fixes
- docs/gen2-roadmap.md: approved Gen2 plan (phases 8-15; finish-EXE-first, two-repo LATE public flip, substantial exit) + Backup & disaster-recovery policy - back up irreplaceable / hard-to-re-source artifacts to the private remote (R20): - track ghidra/ (the RE database; *.lock/tmp*.ps transients excluded) - tools/psyq/ working libs/tools (minus the two >100MB raw source archives) - tools/bin/*.tar.gz (old-gcc cc1 compiler sources; extracted binaries stay regenerable) - tools/ghidra-ext/ (GhidrAssistMCP 2.8.0 + ghidra_psx_ldr 2026.06.04 installer zips) - .claude/settings.json: SessionStart + SessionEnd hooks (auto-start / clean-save the headless Ghidra MCP); moved out of the gitignored settings.local.json so they are backed up - docs/SETUP.md: document the MCP server lifecycle + persistence model + session hooks (§2.8), a tooling inventory, .run/ runtime scratch (§1a), and the backup posture (§ Backup) — R21 - .gitmodules: ignore=dirty on the 4 submodules (maspsx/asm-differ/m2c/decomp-permuter) — silences the phantom "modified" state from generated pycache + cross-filesystem filemode; gitlinks unchanged (still at the pinned commits) - rules R20 (back up irreplaceable work + gathered tooling; loosens R8) and R21 (keep SETUP.md current) recorded in memory; to be formalized at the next PhaseEnd |
||
|
Drew T
|
5f937bdda5 |
feat(phase-6): compiler pinned + matching loop proven (14 matches) — LZSS to Phase 7
- splat.us.exe.yaml: text subseg asm->c (INCLUDE_ASM scaffold); include/common.h (committed prelude); Makefile c-rule cpp->cc1->maspsx->as + -Map - PIN (G8): gcc-2.7.2-psx -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker + maspsx --aspsx-version=2.56 --expand-div; byte-exact on func_80018F20 + instruction-identical across 3 idiom classes; --expand-div REQUIRED for div/rem (SETUP.md §5.4 PINNED, Makefile MASPSX_FLAGS) - harness: diff_settings.py (asm-differ mipsel object mode), tools/decompile.py (m2c wrapper), tools/permuter/ (decomp-permuter harness + objdump shim), expected/ - 14 real functions matched (asm-differ score 0): func_80018F20 + 13 accessors; +14 splat-auto empty no-ops; make check BYTE-IDENTICAL throughout - 2 instruction-identical near-misses guarded NON_MATCHING (func_80015A74 scheduling, func_80016714 phantom frame) -> decomp-permuter candidates - flywheel: docs/matching-cookbook.md, memory + CLAUDE.md step 5 (consult+evolve), R16 - requirements-python.txt: venv freeze (pycparser<3.0 pinned — permuter needs plyparser) - LZSS spike: needs jump-table-in-rodata workflow (102 tables, mixed blob) -> Phase 7 - venv: asm-differ + permuter deps installed - bumps project version 1.5.0 -> 1.6.0 |
||
|
Drew T
|
d55882b4b8 |
feat(phase-5): splat config + all-asm byte-identical build of SLUS_007.26
- config/splat.us.exe.yaml: splat PSX config (platform psx, compiler PSYQ, subalign 2, gp_value 0x80074750, base_path .., main segment align 4); one asm text seg [0x800] + data seg [0x531DC], header via splat's header segtype - Makefile: implement extract (splat split) / build (as -> ld(splat .ld + undefined_*_auto) -> objcopy) / check (SHA1 vs config/check.us.sha, auto-run) / expected (asm-differ baseline) / clean; check-env untouched - config/check.us.sha: committed build-check hash (validation stage 2) - config/symbols.us.txt: now hand-curated — 15 2D_* -> gfx2D_* (illegal leading digit broke as), +D_800629D4 label; header flags re-merge + Ghidra-mirror (G6) - .gitignore: ignore regenerable splat outputs (asm/, include macros, undefined_*_auto.txt); refine the earlier commit-asm/ note - docs: SETUP.md §4.5 binutils 2.42 = byte-clean (no 2.35 downgrade), §5.3 + ledger #8 RESOLVED -G0, §6.3 as-built; memory-map.md gp0 -> -G0 + text/data split - MILESTONE: make extract && make build && make check -> build/us/SLUS_007.26 SHA1 143dbb89... == original (BYTE-IDENTICAL), 100% asm, check fails on corruption - rule R15 (symbol file = curated, valid-identifier build input, Ghidra-mirrored) - bumps project version 1.4.0 -> 1.5.0 |
||
|
Drew T
|
ffe2352db2 |
feat(phase-4): matching-build toolchain (WSL) — make check-env green
- apt: build-essential, binutils-mipsel-linux-gnu 2.42 (+gcc-mipsel 12.x), ninja
1.11.1, python3-pip/venv, libelf-dev, clang-format, p7zip, bchunk, bsdmainutils
- .venv + splat64 0.41.0 ([mips]: spimdisasm 1.41.0, rabbitizer 1.16.2, PyYAML 6.0.3)
- 4 pinned submodules under tools/: maspsx@874855c5, decomp-permuter@b44b0622,
asm-differ@2ad4a4a4, m2c@4266cc28 (.gitmodules)
- old-gcc 0.17 cc1 candidates -> tools/bin/gcc-2.7.2-{psx,cdk}/ (32-bit i386 static;
tarballs are flat + same-named -> per-subdir extraction); CHECKSUMS.sha256 committed
(gitignore exception !/tools/bin/*.sha256); psx cc1 self-id "GNU C 2.7.2 [AL 1.1,
MM 40] Sony Playstation"
- Makefile: check-env (.ONESHELL bash) asserts python>=3.12, venv import splat, both
cc1 (stdin smoke), maspsx, mipsel as/ld/objcopy, sha1(EXE)==EXPECTED_EXE_SHA1;
binutils>=2.38 -> WARN (verdict deferred to Phase 5, §4.5); extract/build/check/
expected/clean are loud-failing Phase-5 stubs (names per §6.3)
- docs/SETUP.md: as-built versions; §4.7 corrected (i386, flat-tarball fix); §4.8
deferred to Phase 6; ledger #5 confirmed (Ubuntu 24.04.4), #6 (2.42), #7 (sha256)
- MILESTONE: make check-env exits 0; zero ROM-derived bytes staged
- bumps project version 1.3.1 -> 1.4.0
|
||
|
Drew T
|
bd5e21b366 |
feat(phase-3): file-loader & overlay-map RE + PCSX-Redux runtime oracle
- RE'd the full file-loader/overlay subsystem in Ghidra via MCP: CdReadRequest -> CdReadStateMachine (hand-rolled CdControl SeekL/ReadN reader, NOT PsyQ CdRead); LIST.CD RAM cache cdFileLocTable + ResourceGetCdLoc/resourceIdMap lookup; LZSS staging buffer 0x80079A70; loadDestPtrTable; ~50 symbols named (T1-T5,T8) - T6b MILESTONE: resident blob @0x800CEDF8 + location overlay @0x80128158 proven byte-identical (sha1-equal contiguous prefixes) vs live PCSX-Redux RAM; blobs imported into Ghidra (ImportOverlay.java); overlay-map table -> memory-map.md §4.3 - PCSX-Redux runtime oracle: Windows-native build, WSL reads RAM via the web API at 172.17.208.1:8081; ledger #4 resolved - live-verified player/time state: gold(/10) HP BP day hour tiredness boss-HP; save-header template (Q#5) + debug-menu dispatch (Q#9: DEBUG.BIN path is dead) - overlay map: 13 area-states across SC01/02/03, 2 duplicate pairs, location x chapter keying, free-roam vs menu-hub granularity - resolves Open Q#1/#3/#4/#5/#6/#8/#9/#12; corrects SLUS-90029 (FF8 demo, not BFM) - 2 US prototypes acquired + CRC-verified (Aug-31 1369DE07, Sep-8 5C24728E) - rules R9 (verify persistence), R10 (>=3-datapoint live verification), R11 (runtime oracle Windows-native, bridged to WSL) - bumps project version 1.2.0 -> 1.3.0 |
||
|
Drew T
|
b6ea3a2747 |
feat(phase-2): deterministic disc/.CD/PAC/LZSS extraction pipeline
- extract.py: one-command full-disc extraction (27 files: 24 Track-1 verbatim
+ 3 .DA as raw CD-DA audio from tracks 2-4) over the frozen iso9660.py
- cd_archive.py + pac.py: .CD TOC walker + PAC splitter, {index}.{type} naming
(no collisions; 447 sub-files, 1189 entries), SQV detection (98)
- lzss.py: game-semantics decoder (pos==0 terminator, -1 bias) + 5 unit tests;
138 type-4 decoded clean (0 warnings)
- manifest.py: deterministic JSON-Lines manifest (1801 artifacts) + --verify
- crosscheck.py + tools/brave-CUE/posix_shim.h: build CUE's brave from source
(GPL untouched), byte-compare; 1484 raw identical, 138/138 type-4 F6
- docs/formats.md: F5 counts (SC02=43), F2 refuted, F6 resolved, F1 type-7=139
- docs/effort-map.md + CLAUDE.md: Effort-map check rule (R7), Ultracode fix
- .gitignore: /extracted/* + !manifest/EXE exceptions; brave _build/
- install gcc 13.3.0 for the oracle
- milestone: one command extracts the full disc; MAIN.CD=49 sub-files; decoder
agrees byte-for-byte with CUE up to the game pos==0 terminator; zero ROM
staged; project version 1.1.0 -> 1.2.0
|