Commit Graph

15 Commits

Author SHA1 Message Date
Drew T 80b89d8a2d feat(phase-23): LLM matching tier + the Fable5 wall-breaker + the gcc-2.7.2 codegen map (v1.22.0)
- LLM TIER (T1-T10.9): tools/serve_local + api_draft + lora_grind + bulk_harvest + the LoRA
  pipeline (v3 model). Local v3 saturated <=15 (~1300 banks, $0). Frontier GLM-5.2 (~$4)
  confirmed the def-side wall INTRINSIC + the idiom well DRY at the leaf (both angles).
- THE PIVOT (07-02): Fable5Max reading the gcc-2.7.2 SOURCE matched func_8014EE14, a §20/§10
  store-vs-load giant "CONFIRMED unsteerable" for 22 phases -> the "wall is intrinsic" verdict
  is MODEL-RELATIVE. Generalized to Opus agents applying §30 (2 more giants, 3-5x cheaper).
- §31 THE CODEGEN MAP: 4 Fable5 agents read the whole gcc-2.7.2 source -> docs/gcc-2.7.2-map/
  {sched,regalloc,loop,cse_expr}.md (935 lines, byte-proven residual->lever catalogs) +
  cookbook §31 (index+triage). Broke hoist-vs-remat/delay-slot/coalescing walls. Correction:
  gcc-papermario is 2.8.1 not 2.7.2 -> tools/reference/gcc-2.7.2/ (SETUP §5.6).
- THE WAVE (8 reach-134 near-misses, Opus + §31): 3 leaf-MATCH by lookup (47-137k tok vs
  200-250k walls) + 5 tight permuter seeds (func_80132784 240->4-off!), zero dead-ends.
- 5 giants/hard-fns banked x134 (func_8014EE14/F2E0/150528/149374/144090); fleet 63.66->64.86%,
  136/136 byte-identical, 0 NON_MATCHING. cookbook §30/§30a/§31. No new governance rules.
- FINDING: matching is SOLVED by the map; bottlenecks are now the permuter (S11 seeds) +
  whole-binary integration (auto-bank leaf-MATCHes) -> Phase 24. worklog -> logs/Phase23.md.
2026-07-02 21:51:25 -06:00
Drew T 76054ef383 chore(phase-23): gitignore the unused .venv-serve (llama-cpp SIGILLs; serve via .venv-train) 2026-06-30 07:37:10 -06:00
Drew T bff03dbdb5 fix(phase-23): T7 — lora_grind broad-rotation banking (two gate bugs); ov_SC01_000 7/15
The 500-fn calibration banked 0/222 across the binary rotation. Root-caused (R14, by
reading the code + the run's own backlog — resolving a flat contradiction between two
scout agents) to TWO independent bugs in lora_grind's use of gate_stage.run_gate, NOT
model quality:
- Bug A: good_sha() passed the sha1sum line "<sha>  <name>" vs harvest_verify's bare
  sha1() -> 0 banks for EVERY binary incl. 077 (so the "0/12" was a bug artifact, not
  an exhausted tail)
- Bug B: the gate call left src/asm/out at the hardcoded ov_SC01_077 defaults -> non-077
  drafts dropped at the 077 stub-filter, silently (and the asm mis-resolution contaminated
  the backlog near-miss classification)

Fix (tools/gate_stage.py): run_gate resolves src/asm/out/good_sha from `binary` when unset
(binary-agnostic, no silent ov_SC01_077 default an overlay inherits; good_sha bare-hash
normalized) + a loud negative-control guard (0-overlap binary/src mismatch warns, so a 0
can never again masquerade as 'nothing matched'). tools/lora_grind.good_sha fixed at source.
Byte-neutral: make check-all 136/136.

Proof: ov_SC01_000 spot-run banked 7/15 (47%) byte-identical (@commit:0322); reach-2
func_8017CE24 propagated x2. ROI finding: 6/7 banks are reach-1 (overlay-unique) -> broad
rotation is high bank-RATE / low fleet-% ROI; the fleet lever is reach>=2 targeting (T9) +
corpus-v3 (T8). Backlog now correctly classified (4x close=1 = grinder fuel).

- docs/gen2-mips-matching-model.md: T7 RESULT section
- phase-ends/CURRENT_PHASE.md: T7 done; next = T8 corpus-v3 / T9 reach>=2 selection
2026-06-29 23:44:12 -06:00
Drew T ad535ad1ab feat(phase-22): gen2 matching-model design + export_pairs.py corpus miner
docs/gen2-mips-matching-model.md: the BFM/gcc-2.7.2 matching-specialist idea (LoRA on our own
gate-verified pairs — the corpus off-the-shelf RE LLMs lack). export_pairs.py mines 1307 banked
(asm<->C) pairs from build objects (asm/ is gitignored, so disasm the ROM-identical build, splat-like
format) + src defs -> datasets/match_pairs/{pairs,train,test}.jsonl (gitignored, 1174/133 split).
api_draft.py: TEMP env-tunable. .gitignore: datasets/ models/ weights.
2026-06-29 14:27:19 -06:00
Drew T d20b20c64b feat(phase-18): T0 — setup + exemplar set; match_one validated as the §10 oracle
- clone gcc-papermario (a6afc2af) + xenogears-decomp (f27c0768) -> tools/reference/
  (gitignored research refs; SHAs pinned in docs/SETUP.md §5.6, R20/R21)
- apparatus byte-validated: match_one cleanly measures the §10 residual on both quirk
  exemplars (the floor-free oracle Phase 17 failed to use; permuter score was polluted):
  - func_8012B8E4: 75=75 ins, 24 mismatched -> $s0/$s1 regalloc swap + reassociation (§10 A)
  - func_8012B4B8: 94 vs 84 ins, 88 mismatched -> extra-reg spill + struct-copy/reload (§10 B)
- CURRENT_PHASE.md crash-recovery log; harness task list (R28); no build-affecting change
2026-06-20 11:08:21 -06:00
Drew T 144839f24b feat(phase-13): T1a/T1b — SC01/077 onboarded + all-asm byte-match (d19c9580)
- ov_SC01_077 scaffolded: config/overlays.mk (OVERLAY_BINARIES + var block),
  config/splat.ov_SC01_077.yaml (from template), check.sha, empty symbols, +
  ov_SC01_077 entry & sentinel anchor in the 4 Python BINARIES dicts
- GATE T1b: make check BINARY=ov_SC01_077 -> d19c9580 BYTE-IDENTICAL @ 100%
  INCLUDE_ASM; clean rebuild leaves main 143dbb89 + resident 8e17e02f unregressed
- reusable non-word-aligned-overlay handling (~75% of fleet) added to the
  template + Makefile: (1) [word_floor, bin, trailing] carve for the final 1-3
  bytes spimdisasm drops; (2) build/assets/%.o incbin rule (+ .data align=1) for
  splat bin assets (asset_path scoped per-alias); (3) objcopy end-align TRIM
  (shrink-only, <=3 B) removing the .ld's segment-end ALIGN(.,4) pad
- .gitignore /assets/ (regenerable splat output); clean removes assets/
2026-06-16 15:39:52 -06:00
Drew T d380b13fc5 feat(phase-9): close-out — PhaseEnd_Phase9 + binary-agnostic toolchain (v1.9.0)
- PhaseEnd_Phase9.md written; CURRENT_PHASE.md archived -> phase-ends/logs/Phase9.md (R19)
- MILESTONE (gate-2 confirmed): the toolchain is binary-agnostic — EXE rebuilds
  143dbb89 through the parameterized path WITH and WITHOUT SDK objects; make report
  52/959/7/50.24%; every binary-specific value a required param (no EXE default);
  wrong --vram-base -> cae22f7e (negative control). Landed as 11 per-tool checkpoints.
- R23: stop the Ghidra MCP + commit the DB at phase-end/RE-checkpoint (lock won't
  release until MCP closes; SessionEnd is too late for a mid-session commit). This
  commit reconciles the Ghidra program DB (db.12 -> db.15, 99% identical: Phase 9 did
  zero RE writes) after a clean ghidra_mcp_stop.sh save.
- .gitignore: ignore /ghidra/**/*.lock~ (the lock-backup that churned every session)
- bumps project version 1.8.0 -> 1.9.0
2026-06-15 22:01:17 -06:00
Drew T 6f248462dd chore(gen2-prep): Gen2 roadmap + private-repo backup hardening + SETUP/submodule fixes
- docs/gen2-roadmap.md: approved Gen2 plan (phases 8-15; finish-EXE-first,
  two-repo LATE public flip, substantial exit) + Backup & disaster-recovery policy
- back up irreplaceable / hard-to-re-source artifacts to the private remote (R20):
  - track ghidra/ (the RE database; *.lock/tmp*.ps transients excluded)
  - tools/psyq/ working libs/tools (minus the two >100MB raw source archives)
  - tools/bin/*.tar.gz (old-gcc cc1 compiler sources; extracted binaries stay regenerable)
  - tools/ghidra-ext/ (GhidrAssistMCP 2.8.0 + ghidra_psx_ldr 2026.06.04 installer zips)
- .claude/settings.json: SessionStart + SessionEnd hooks (auto-start / clean-save the
  headless Ghidra MCP); moved out of the gitignored settings.local.json so they are backed up
- docs/SETUP.md: document the MCP server lifecycle + persistence model + session hooks (§2.8),
  a tooling inventory, .run/ runtime scratch (§1a), and the backup posture (§ Backup) — R21
- .gitmodules: ignore=dirty on the 4 submodules (maspsx/asm-differ/m2c/decomp-permuter) —
  silences the phantom "modified" state from generated pycache + cross-filesystem filemode;
  gitlinks unchanged (still at the pinned commits)
- rules R20 (back up irreplaceable work + gathered tooling; loosens R8) and R21 (keep
  SETUP.md current) recorded in memory; to be formalized at the next PhaseEnd
2026-06-15 17:15:31 -06:00
Drew T d55882b4b8 feat(phase-5): splat config + all-asm byte-identical build of SLUS_007.26
- config/splat.us.exe.yaml: splat PSX config (platform psx, compiler PSYQ,
  subalign 2, gp_value 0x80074750, base_path .., main segment align 4); one
  asm text seg [0x800] + data seg [0x531DC], header via splat's header segtype
- Makefile: implement extract (splat split) / build (as -> ld(splat .ld +
  undefined_*_auto) -> objcopy) / check (SHA1 vs config/check.us.sha, auto-run)
  / expected (asm-differ baseline) / clean; check-env untouched
- config/check.us.sha: committed build-check hash (validation stage 2)
- config/symbols.us.txt: now hand-curated — 15 2D_* -> gfx2D_* (illegal leading
  digit broke as), +D_800629D4 label; header flags re-merge + Ghidra-mirror (G6)
- .gitignore: ignore regenerable splat outputs (asm/, include macros,
  undefined_*_auto.txt); refine the earlier commit-asm/ note
- docs: SETUP.md §4.5 binutils 2.42 = byte-clean (no 2.35 downgrade), §5.3 +
  ledger #8 RESOLVED -G0, §6.3 as-built; memory-map.md gp0 -> -G0 + text/data split
- MILESTONE: make extract && make build && make check -> build/us/SLUS_007.26
  SHA1 143dbb89... == original (BYTE-IDENTICAL), 100% asm, check fails on corruption
- rule R15 (symbol file = curated, valid-identifier build input, Ghidra-mirrored)
- bumps project version 1.4.0 -> 1.5.0
2026-06-14 12:05:38 -06:00
Drew T ffe2352db2 feat(phase-4): matching-build toolchain (WSL) — make check-env green
- apt: build-essential, binutils-mipsel-linux-gnu 2.42 (+gcc-mipsel 12.x), ninja
  1.11.1, python3-pip/venv, libelf-dev, clang-format, p7zip, bchunk, bsdmainutils
- .venv + splat64 0.41.0 ([mips]: spimdisasm 1.41.0, rabbitizer 1.16.2, PyYAML 6.0.3)
- 4 pinned submodules under tools/: maspsx@874855c5, decomp-permuter@b44b0622,
  asm-differ@2ad4a4a4, m2c@4266cc28 (.gitmodules)
- old-gcc 0.17 cc1 candidates -> tools/bin/gcc-2.7.2-{psx,cdk}/ (32-bit i386 static;
  tarballs are flat + same-named -> per-subdir extraction); CHECKSUMS.sha256 committed
  (gitignore exception !/tools/bin/*.sha256); psx cc1 self-id "GNU C 2.7.2 [AL 1.1,
  MM 40] Sony Playstation"
- Makefile: check-env (.ONESHELL bash) asserts python>=3.12, venv import splat, both
  cc1 (stdin smoke), maspsx, mipsel as/ld/objcopy, sha1(EXE)==EXPECTED_EXE_SHA1;
  binutils>=2.38 -> WARN (verdict deferred to Phase 5, §4.5); extract/build/check/
  expected/clean are loud-failing Phase-5 stubs (names per §6.3)
- docs/SETUP.md: as-built versions; §4.7 corrected (i386, flat-tarball fix); §4.8
  deferred to Phase 6; ledger #5 confirmed (Ubuntu 24.04.4), #6 (2.42), #7 (sha256)
- MILESTONE: make check-env exits 0; zero ROM-derived bytes staged
- bumps project version 1.3.1 -> 1.4.0
2026-06-14 03:56:07 -06:00
Drew T 3207d52491 feat(phase-3.5): prototype research spike — go/no-go (protos don't accelerate Gen1 matching)
- extract_proto_exe.py: subdir-aware proto main-EXE extraction (reuses iso9660); extracted
  sep8 SLUS_007.26 (413,696 B) + aug31 MUSASHI/USA_DEMO.EXE (415,744 B, base 0x80018000)
- ghidra_import.sh: reusable headless import; both protos imported into the bfm project
  (sep8 1726 funcs, aug31 1737 funcs, both PsyQ 4.0.0), R9-verified
- DumpFunctionSignatures.java + match_protos.py: 3-tier (exact/norm/seq) cross-binary
  correspondence; MIPS normalizer validated by anchors
- FINDINGS: (T2) no debug symbols in either proto — Hidden Palace "no symbols" VERIFIED;
  recon "Sep-8 less-stripped" REFUTED by per-file byte compare. (T3) Sep-8 99.6% byte-identical
  to retail (only 3 funcs differ: DebugMenuHandler, CdReadSectorReadyCB, SaveLoadRoutine);
  Aug-31 66% norm-identical, 862 1:1 correspondences. (T4) demo shares retail's 18-entry
  game-mode dispatch architecture but the handler code diverged — Q#10 resolved
- VERDICT (docs/proto-correspondence.md): NO-GO as a Gen1 label/symbol accelerator; GO to retain
  Sep-8 (Phase-6 compiler corroborant) + both (Gen2 assets); proto-side labels in
  config/symbols.proto-{sep8,demo}.txt (R13-tagged, never feed symbols.us.txt)
- reorg extracted/ into per-ROM subfolders (extracted/retail/, extracted/proto/); nested
  gitignore allowlist; extractor defaults updated; manifest --verify PASS
- memory-map.md: Phase-3.5 block + demo dispatch anchors; Q#10 RESOLVED; §5 proto notes VERIFIED
- ghidra_mcp_start.sh: PROG arg to serve a prototype
- rules R13 (proto-provenance/confidence tagging), R14 (verify recon counts vs bytes)
- bumps project version 1.3.0 -> 1.3.1
2026-06-14 03:19:23 -06:00
Drew T bd5e21b366 feat(phase-3): file-loader & overlay-map RE + PCSX-Redux runtime oracle
- RE'd the full file-loader/overlay subsystem in Ghidra via MCP: CdReadRequest ->
  CdReadStateMachine (hand-rolled CdControl SeekL/ReadN reader, NOT PsyQ CdRead);
  LIST.CD RAM cache cdFileLocTable + ResourceGetCdLoc/resourceIdMap lookup; LZSS
  staging buffer 0x80079A70; loadDestPtrTable; ~50 symbols named (T1-T5,T8)
- T6b MILESTONE: resident blob @0x800CEDF8 + location overlay @0x80128158 proven
  byte-identical (sha1-equal contiguous prefixes) vs live PCSX-Redux RAM; blobs
  imported into Ghidra (ImportOverlay.java); overlay-map table -> memory-map.md §4.3
- PCSX-Redux runtime oracle: Windows-native build, WSL reads RAM via the web API
  at 172.17.208.1:8081; ledger #4 resolved
- live-verified player/time state: gold(/10) HP BP day hour tiredness boss-HP;
  save-header template (Q#5) + debug-menu dispatch (Q#9: DEBUG.BIN path is dead)
- overlay map: 13 area-states across SC01/02/03, 2 duplicate pairs, location x
  chapter keying, free-roam vs menu-hub granularity
- resolves Open Q#1/#3/#4/#5/#6/#8/#9/#12; corrects SLUS-90029 (FF8 demo, not BFM)
- 2 US prototypes acquired + CRC-verified (Aug-31 1369DE07, Sep-8 5C24728E)
- rules R9 (verify persistence), R10 (>=3-datapoint live verification),
  R11 (runtime oracle Windows-native, bridged to WSL)
- bumps project version 1.2.0 -> 1.3.0
2026-06-14 02:04:28 -06:00
Drew T b6ea3a2747 feat(phase-2): deterministic disc/.CD/PAC/LZSS extraction pipeline
- extract.py: one-command full-disc extraction (27 files: 24 Track-1 verbatim
  + 3 .DA as raw CD-DA audio from tracks 2-4) over the frozen iso9660.py
- cd_archive.py + pac.py: .CD TOC walker + PAC splitter, {index}.{type} naming
  (no collisions; 447 sub-files, 1189 entries), SQV detection (98)
- lzss.py: game-semantics decoder (pos==0 terminator, -1 bias) + 5 unit tests;
  138 type-4 decoded clean (0 warnings)
- manifest.py: deterministic JSON-Lines manifest (1801 artifacts) + --verify
- crosscheck.py + tools/brave-CUE/posix_shim.h: build CUE's brave from source
  (GPL untouched), byte-compare; 1484 raw identical, 138/138 type-4 F6
- docs/formats.md: F5 counts (SC02=43), F2 refuted, F6 resolved, F1 type-7=139
- docs/effort-map.md + CLAUDE.md: Effort-map check rule (R7), Ultracode fix
- .gitignore: /extracted/* + !manifest/EXE exceptions; brave _build/
- install gcc 13.3.0 for the oracle
- milestone: one command extracts the full disc; MAIN.CD=49 sub-files; decoder
  agrees byte-for-byte with CUE up to the game pos==0 terminator; zero ROM
  staged; project version 1.1.0 -> 1.2.0
2026-06-13 19:47:42 -06:00
Drew T ca51930605 docs: convert to all-in-WSL architecture; relax H1 ROM policy while private
Surgically correct PROJECT_CONTEXT.md env sections (H1/H2, Quick Ref, Architecture diagram, Assumptions/Risks, Phase 1 & 4 roadmap) to Linux-first single-clone; all rules/roadmap content otherwise preserved. SETUP.md rewritten Linux-first with all facts intact + redump disc-provenance added. .gitignore permits ROM-derived content (asm/assets/extracted) while private; raw dump + build churn stay ignored.
2026-06-10 23:43:31 -06:00
Drew T d4334fc1ca plan files created 2026-06-10 22:02:07 -06:00