phase5: cookbook, conventions, and verification record
P5-T6. Adds docs/MATCHING_COOKBOOK.md (nine byte-proven compiler/assembler findings, each with basis and limit), docs/MATCHING_CONVENTIONS.md (what counts as a match, registry format, source naming, duplicate sharing, symbols, the verification procedure, the firewall) and docs/PHASE5_VERIFICATION.md (outcome, clean gates, bounded negatives, milestone request). Tooling: make test and make check targets; region objects are now symbol-localized so the documented duplicate-sharing mechanism (N registry rows -> 1 source) links, covered by a synthetic test (53 tests). Clean gates: 53/53 tests; make clean/all/cmp/SHA-1 green; make gate with one C region -> c_regions=1, 0 differing bytes, SHA-1 e173426c...; 57 tracked files, 0 under any prohibited root. Phase 5 is not closed until the milestone is confirmed.
This commit is contained in:
Christopher Williams
@@ -28,7 +28,7 @@ MATCH := tools/sf3_match
|
||||
CODE_OUT := build/code
|
||||
EXPECTED_SHA1 := e173426c157384ebf1b6caf8c6fea18a85a14af9
|
||||
|
||||
.PHONY: all validate split assemble link binary code gate clean
|
||||
.PHONY: all validate split assemble link binary code gate test check clean
|
||||
|
||||
all: binary
|
||||
|
||||
@@ -81,6 +81,13 @@ gate: validate
|
||||
@"$(MATCH)" gate --exe "$(EXE)" --regions "$(REGIONS)" --out "$(CODE_OUT)" \
|
||||
--expect-sha1 "$(EXPECTED_SHA1)"
|
||||
|
||||
# Verification gates. `test` is synthetic-only and needs no game input;
|
||||
# `check` adds the full-binary byte gate (which does).
|
||||
test:
|
||||
@PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests
|
||||
|
||||
check: test gate
|
||||
|
||||
clean:
|
||||
@if test -e "$(BUILD)"; then \
|
||||
test -d "$(BUILD)" && test ! -L "$(BUILD)"; \
|
||||
|
||||
@@ -99,7 +99,7 @@ verify against the original SDK compiler — the proprietary PsyQ SDK plus the `
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `tools/` | The project's own tooling: `sf3_extract` (disc/extraction), `sf3_probe` (structural probe), `sf3_fingerprint_probe`, and the synthetic test suite in `tools/tests/` |
|
||||
| `docs/` | The format/address records, per-phase investigation and verification records, setup and toolchain provenance, and the Phase 5 toolchain fingerprint |
|
||||
| `docs/` | The format/address records, per-phase investigation and verification records, setup and toolchain provenance, the Phase 5 toolchain fingerprint, and the [matching cookbook](docs/MATCHING_COOKBOOK.md) and [matching conventions](docs/MATCHING_CONVENTIONS.md) |
|
||||
| `phase-ends/` | The project's governance record: the phase digest, one PhaseEnd per closed phase, each phase plan, and the active `CURRENT_PHASE.md` |
|
||||
| `Makefile` | The all-assembly rebuild: validate → split → assemble → link → binary |
|
||||
| `AGENTS.md`, `PROJECT_CONTEXT.md` | The standing rules and the permanent project constitution |
|
||||
@@ -121,6 +121,11 @@ match oracle is a clean rebuild plus `cmp` and SHA-1 over the whole binary. Comp
|
||||
reached by differential fingerprinting against the real executable, never from a version label — the
|
||||
Phase 5 record documents one such conclusion that was found wrong and corrected.
|
||||
|
||||
What has been learned about the compiler is written down where it can be reused:
|
||||
[`docs/MATCHING_COOKBOOK.md`](docs/MATCHING_COOKBOOK.md) (byte-proven findings, each with its basis
|
||||
and limit) and [`docs/MATCHING_CONVENTIONS.md`](docs/MATCHING_CONVENTIONS.md) (what counts as a match
|
||||
and how one is registered).
|
||||
|
||||
## Standards, and the no-ROM policy
|
||||
|
||||
**Accuracy.** A function is matched only when its compiled instructions are identical to the
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# Matching Conventions
|
||||
|
||||
How a function becomes a match in this project, and what evidence it must carry. These are the
|
||||
rules the tooling enforces; they exist so a match claim is never a matter of opinion.
|
||||
|
||||
## What counts as a match
|
||||
|
||||
A function is matched only when **both** hold:
|
||||
|
||||
1. `tools/sf3_match range` reports `differing_bytes=0` for the function's exact instruction range; and
|
||||
2. `make gate` reports `differing_bytes=0` and the rebuilt executable's SHA-1 equals
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`.
|
||||
|
||||
Functional equivalence, a plausible-looking decompilation, or a diff that is "only register
|
||||
allocation" is **not** a match. If the bytes differ, the function stays fallback.
|
||||
|
||||
## Registering a match
|
||||
|
||||
Add one row to the tracked registry `config/regions.tsv`:
|
||||
|
||||
```
|
||||
# start<TAB>end<TAB>source
|
||||
0x80017AD4 0x80017AE8 src/func_80017AD4.c
|
||||
```
|
||||
|
||||
Rules the harness enforces:
|
||||
|
||||
- Regions must be ordered, non-overlapping and inside the declared payload.
|
||||
- A region's compiled `.text` must be **exactly** the declared length; anything else is
|
||||
`LENGTH-MISMATCH` and fails. A size disagreement is a failure, never something to widen the region
|
||||
for.
|
||||
- The source must exist and be a regular file.
|
||||
|
||||
## Source layout and naming
|
||||
|
||||
- One function per file under `src/`, named `func_XXXXXXXX.c` using the function's address in
|
||||
uppercase hex, until evidence supports a real name. Names are earned, not guessed.
|
||||
- Every reconstructed type, field and function name is a **hypothesis**. State that in the file
|
||||
header, and state the limits, exactly as `src/func_80017AD4.c` does.
|
||||
- Do not invent semantics that the bytes do not support. Where a struct's other fields are unknown,
|
||||
pad or leave them unnamed rather than inventing a layout.
|
||||
|
||||
## Unmatched code stays fallback
|
||||
|
||||
Unmatched code remains raw data taken from the original. It does not appear as `NON_MATCHING` C, and
|
||||
it never enters the default build. A failed candidate is recorded as a bounded negative, not hidden
|
||||
behind a guard.
|
||||
|
||||
## Duplicate bodies
|
||||
|
||||
A body shared by several addresses is matched **once** and registered several times against the same
|
||||
source:
|
||||
|
||||
```
|
||||
0x80017AD4 0x80017AE8 src/func_80017AD4.c
|
||||
0x80012DAC 0x80012DC0 src/func_80017AD4.c
|
||||
```
|
||||
|
||||
The harness compiles each region object and then **localizes its symbols**
|
||||
(`objcopy --keep-global-symbol=<sentinel>`), so the same source can be instantiated repeatedly without
|
||||
a duplicate-symbol clash. This is safe because region objects are placed by the linker script rather
|
||||
than by symbol name, and cross-references are supplied as absolute `--defsym` values — so symbol
|
||||
visibility cannot affect the emitted bytes. The behaviour is covered by a synthetic test.
|
||||
|
||||
Before registering any match, check for duplicates: search the payload for the function's exact byte
|
||||
body. A shared *tail* is not a duplicate function — verify the candidate address is a real function
|
||||
start before sharing.
|
||||
|
||||
## Symbols
|
||||
|
||||
Cross-references to functions and globals that are not themselves matched are supplied as absolute
|
||||
assembler definitions (`--defsym NAME=0xADDR`). The harness currently accepts these **globally** for a
|
||||
whole build; per-region symbols are not yet supported, which is why a symbol-referencing match such as
|
||||
`0x8002D2A0` is verified but not yet registered.
|
||||
|
||||
## Verification procedure
|
||||
|
||||
```bash
|
||||
make test # synthetic-only suite; needs no game input
|
||||
make check # make test + the full-binary byte gate (needs the extracted executable)
|
||||
```
|
||||
|
||||
`make gate` prints `c_regions=<n>` and, when the registry is empty, an explicit note that the build
|
||||
contains no C. A `MATCH` with `c_regions=0` means the data baseline was reproduced, not that anything
|
||||
was matched.
|
||||
|
||||
## Firewall
|
||||
|
||||
Never commit: the disc image, extracted files, generated disassembly or assembly, build outputs,
|
||||
expected binaries, the Ghidra database, memory dumps, or proprietary SDK material. All of those live
|
||||
in ignored paths (`disks/`, `extracted/`, `asm/`, `build/`, `ghidra/`, `dumps/`, `assets/`,
|
||||
`tools/psyq/`, `tools/splat/`, `tools/maspsx/`, `tools/old-gcc/`, `tools/mipsel-none-elf-binutils/`,
|
||||
`tools/wibo/`). Review `git status` before every commit. `git clean -x` and `git clean -fdx` are
|
||||
forbidden.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Matching Cookbook
|
||||
|
||||
Compiler and assembler behaviour that this project has **proved on this executable**, recorded so it
|
||||
is not rediscovered. Every entry states its basis and its limit. Nothing here is copied from another
|
||||
project; each claim was observed here first.
|
||||
|
||||
## The toolchain
|
||||
|
||||
| Role | Identity | Basis |
|
||||
|---|---|---|
|
||||
| Compiler | `egcs-2.91.66 19990314` (egcs-1.1.2), target `mips-sony-psx` — PsyQ 4.5 `CC1PSX` | SDK README banner; the real binary executed and compared |
|
||||
| Open substitute | `decompals/old-gcc` 0.17 `gcc-2.91.66-psx` | Instruction-identical to the real `CC1PSX` across ~990 probe instructions |
|
||||
| Assembler | `ASPSX` 2.81 (Sony) | SDK banner |
|
||||
| Working invocation | `cc1 -quiet -O2 -G0 -mno-split-addresses`, then GNU `as -march=r3000 -G0` | Byte-identical ranges; `make gate` |
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. `-mno-split-addresses` is mandatory
|
||||
|
||||
`cc1` has two address-materialisation modes, and only one of them matches.
|
||||
|
||||
- **`-msplit-addresses`** (the real `CC1PSX` default) emits explicit sequences such as
|
||||
`lui $3,%hi(sym)` / `lw $2,%lo(sym)($3)`. **Does not match.**
|
||||
- **`-mno-split-addresses`** emits assembler **macros** (`lw $2,sym`, `sw $4,sym`, `la $2,sym`) that
|
||||
the assembler expands. **Matches.**
|
||||
|
||||
*Basis:* oracle ranges `0x80085B80`, `0x800F3160` — byte-identical only under `-mno-split-addresses`.
|
||||
*Limit:* proven for symbol loads, symbol stores and `la`; not yet exhaustively for every construct.
|
||||
|
||||
### 2. Symbol load — the macro expands into the destination register
|
||||
|
||||
`lw rt,sym` cannot need a separate scratch, so the assembler expands it as
|
||||
`lui rt,%hi(sym)` + `lw rt,%lo(sym)(rt)` — the **same** register twice.
|
||||
|
||||
*Basis:* `0x80085B80` (`lui v0,0x8014` / `lw v0,-31652(v0)`) byte-identical.
|
||||
*Limit:* single observation class; `lhu`/`lbu` assumed to behave the same, unverified.
|
||||
|
||||
### 3. Symbol store — the macro expands through `$at`
|
||||
|
||||
`sw rt,sym` must not clobber its source register, so the assembler expands it through `$at`:
|
||||
`lui $at,%hi(sym)` + `sw rt,%lo(sym)($at)`.
|
||||
|
||||
*Basis:* `0x800F3160` (`lui at,0x8014` / `jr ra` / `sw a0,11996(at)`) byte-identical.
|
||||
*Limit:* as above.
|
||||
|
||||
### 4. `la` differs between assemblers — GNU `as` uses `ori`, ASPSX uses `addiu`
|
||||
|
||||
This is the project's current sharpest edge.
|
||||
|
||||
- GNU `as` expands `la rt,sym` with **`ori`** (zero-extended low half).
|
||||
- The original — assembled by **ASPSX 2.81** — uses **`addiu`** (sign-extended `%lo`).
|
||||
|
||||
So any function that materialises a symbol **address** (as opposed to loading/storing through one)
|
||||
will mismatch until `maspsx` is wired into the harness.
|
||||
|
||||
*Basis:* the original's clear loop (`lui v0,0x8012` / `addiu v0,v0,8664`) versus GNU `as` output for
|
||||
the same `la` under the same flags.
|
||||
*Limit:* `maspsx` is not yet exercised here; the exact `--aspsx-version` is not chosen.
|
||||
|
||||
### 5. A literal address is not a symbol address
|
||||
|
||||
A **literal** address in the C is materialised as `lui`+`ori`; a **symbol** address uses the
|
||||
macro/`%hi`/`%lo` split, which the assembler turns into `addiu`. A probe written with a literal
|
||||
address therefore cannot reproduce the original's `addiu` form.
|
||||
|
||||
*Basis:* the two entry clear-loop experiments (literal vs `extern`).
|
||||
*Limit:* observed for constant-vs-symbol; interaction with `-G` is untested.
|
||||
|
||||
### 6. `cc1` consumes preprocessed input only
|
||||
|
||||
`cc1` is the compiler proper, not the driver. It rejects `/* … */` comments and `#` directives with a
|
||||
parse error, so a `cpp` stage must run first. The harness always preprocesses.
|
||||
|
||||
*Basis:* direct parse failures on commented sources.
|
||||
*Limit:* none; this is a hard requirement.
|
||||
|
||||
### 7. `char` is unsigned by default
|
||||
|
||||
The PlayStation target defaults plain `char` to **unsigned**: a `char` load emits `lbu`, and an
|
||||
`unsigned char` parameter is masked with `andi …,0xff`.
|
||||
|
||||
*Basis:* `v_strlen`-style probe (`lbu`) and `func_80017AD4` (`andi a2,a2,0xff` before `sltu`).
|
||||
*Limit:* explicit `signed char` is assumed to emit `lb`; the original contains both `lb` and `lbu`,
|
||||
so per-variable signedness still has to be read from the bytes.
|
||||
|
||||
### 8. Delay-slot filling
|
||||
|
||||
- `jr ra` takes the last independent instruction; a trailing store commonly lands in the slot.
|
||||
- Branch delay slots take `nop` when no independent instruction is available.
|
||||
|
||||
*Basis:* `0x80017AD4` (store in the `jr ra` slot), `0x80017AE8` (three stores, last in the slot),
|
||||
the entry's clear loop (`nop` after `bnez`).
|
||||
*Limit:* scheduling is optimisation-level dependent; only `-O2` is characterised.
|
||||
|
||||
### 9. Magic-division scratch register
|
||||
|
||||
Signed and unsigned division by a non-power-of-two constant is expanded into a multiply by a magic
|
||||
constant plus `mfhi`. The `mfhi` destination is a scratch register chosen by register pressure — in
|
||||
this executable overwhelmingly `t0`.
|
||||
|
||||
*Basis:* 74 magic-constant sites censused; `mfhi` histogram `t0` ×25, never `v0` in complex code.
|
||||
*Limit:* the register choice is context-dependent, so it cannot be used to identify the compiler; it
|
||||
only matters when reproducing a specific function.
|
||||
|
||||
## Open questions
|
||||
|
||||
- The exact `-G` small-data threshold (the executable clearly uses `gp`-relative data in places).
|
||||
- Whether `-mgpopt` was passed.
|
||||
- `maspsx`'s `--aspsx-version` for this SDK (ASPSX 2.81 is reported by the SDK banner).
|
||||
- One reconstructed function (`0x8005DEF8`) whose constant multiply the original emits as a real
|
||||
`mult` while every compiler tested synthesises it — the reconstruction is the likely cause.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Phase 5 Verification Record
|
||||
|
||||
**Scope:** the consolidated Phase 5 outcome and the clean gates for the phase milestone.
|
||||
**Task:** P5-T6
|
||||
**Status:** complete pending developer milestone confirmation. The phase is not closed.
|
||||
|
||||
## Outcome summary
|
||||
|
||||
| Task | Result |
|
||||
|---|---|
|
||||
| P5-T1 | Baseline revalidated and toolchain evidence inventoried (`docs/PHASE5_TOOLCHAIN_EVIDENCE.md`). |
|
||||
| P5-T2 | The `decompals/old-gcc` 0.17 candidate ladder acquired with provenance (`docs/SETUP.md`). |
|
||||
| P5-T3 | Compiler identified as **`egcs-2.91.66` (egcs-1.1.2), target `mips-sony-psx` — PsyQ 4.5 `CC1PSX`**; the open `gcc-2.91.66-psx` is instruction-identical to the real SDK binary (`docs/PHASE5_TOOLCHAIN_FINGERPRINT.md`). |
|
||||
| P5-T4 | Tracked matching harness and ordered-layout gate (`docs/PHASE5_MATCH_HARNESS.md`). |
|
||||
| P5-T5 | The entry determined to be CRT startup, not C; **first byte-identical C match** `func_80017AD4` (`docs/PHASE5_FIRST_MATCH.md`). |
|
||||
| P5-T6 | Cookbook, conventions, verification record, milestone request (`docs/MATCHING_COOKBOOK.md`, `docs/MATCHING_CONVENTIONS.md`). |
|
||||
|
||||
**The Phase 5 milestone is met:** the original toolchain is identified with byte evidence, and one C
|
||||
function is matched instruction-for-instruction with a green clean full-binary rebuild.
|
||||
|
||||
## Clean verification gates (last run)
|
||||
|
||||
1. **Synthetic suite** — `PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools/tests` —
|
||||
exit 0; **53 tests pass** (25 new this phase).
|
||||
2. **Phase 3 baseline** — `make clean` exit 0; `make all` exit 0;
|
||||
`cmp -s build/scus_946_40.rebuilt 'extracted/SCUS_946.40;1'` exit 0; both files SHA-1
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`.
|
||||
3. **Ordered gate with one C region** — `make gate`: `c_regions=1`, `differing_bytes=0`,
|
||||
`result=MATCH`, rebuilt SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`.
|
||||
4. **Firewall / Git review** — 57 tracked files; 0 tracked paths under any prohibited root
|
||||
(`disks/`, `extracted/`, `asm/`, `assets/`, `expected/`, `build/`, `ghidra/`, `dumps/`,
|
||||
`tools/psyq/`, `tools/splat/`, `tools/maspsx/`, `tools/mipsel-none-elf-binutils/`,
|
||||
`tools/old-gcc/`, `tools/wibo/`); `git diff --check` exit 0.
|
||||
|
||||
## Match evidence
|
||||
|
||||
`func_80017AD4`, `0x80017AD4..0x80017AE8` (20 bytes), `src/func_80017AD4.c`:
|
||||
|
||||
- `sf3_match range` — 0 differing bytes.
|
||||
- `make gate` — 0 differing bytes and SHA-1 identical.
|
||||
- Duplicate review — the body occurs exactly once; one caller at `0x80014C14`.
|
||||
|
||||
Everything else in the executable remains the raw data fallback. The registry contains exactly one
|
||||
region, and `make gate` reports `c_regions=1` so the single match cannot be confused with the
|
||||
baseline.
|
||||
|
||||
## Bounded negatives and open items
|
||||
|
||||
Recorded, not hidden:
|
||||
|
||||
- The entry `[0x800FB368, 0x800FB410)` is **not** compiler output and no C is claimed for it.
|
||||
- `0x8005DEF8` — the original emits a real `mult` for a constant multiply that every compiler tested
|
||||
(including the real `CC1PSX`) synthesises into shifts. The reconstruction is the likely cause.
|
||||
- `0x8002D2A0` and `0x80017AE8` are verified byte-identical by `sf3_match range` but not registered
|
||||
(the first needs symbol support; the second was deliberately left for the next batch).
|
||||
- GNU `as` expands the `la` macro with `ori` while ASPSX uses `addiu`, so `la`-using functions need
|
||||
`maspsx` (cookbook finding 4).
|
||||
- The `-G` small-data threshold and `-mgpopt` remain undetermined.
|
||||
|
||||
## Rules added this phase
|
||||
|
||||
None. No rule in `AGENTS.md` was added, changed or removed.
|
||||
|
||||
## Requested action
|
||||
|
||||
The developer is asked to confirm the Phase 5 milestone:
|
||||
|
||||
- the original compiler is identified with byte evidence as `egcs-2.91.66` (PSX) / PsyQ 4.5
|
||||
`CC1PSX`, with the open `gcc-2.91.66-psx` verified instruction-identical; and
|
||||
- one C function (`func_80017AD4`) is matched instruction-for-instruction, with the clean full-binary
|
||||
rebuild passing exact `cmp` and SHA-1.
|
||||
|
||||
On confirmation, a PhaseEnd record follows, `phase-ends/DIGEST.md` is updated, `CURRENT_PHASE.md` is
|
||||
archived to `phase-ends/logs/Phase5.md`, and the phase closes. Do not begin Phase 6 in this session.
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
**Phase:** Phase 5 — Original Toolchain Identification and First C Match
|
||||
**Plan:** `phase-ends/Phase5_PLAN.md`
|
||||
**Status:** active; P5-T5 complete — the entry is CRT startup (not C) and the first C match is byte-identical
|
||||
**Active task:** P5-T5 complete; next is P5-T6 (cookbook, registry conventions, phase gate)
|
||||
**Status:** active; all six tasks complete — awaiting developer milestone confirmation
|
||||
**Active task:** P5-T6 complete; phase is not closed until the milestone is confirmed
|
||||
|
||||
> Approval note: `Phase5_PLAN.md` was authored as a DRAFT requiring explicit developer
|
||||
> approval. The developer's instruction to begin Phase 5 is being treated as that approval
|
||||
@@ -69,10 +69,31 @@ The Phase 4 ordered workflow is now tracked tooling rather than an ignored exper
|
||||
- Six codegen findings were recorded, including the key one: GNU `as` expands the `la` macro with
|
||||
`ori` while the original assembler (ASPSX 2.81) uses `addiu`, so `la`-using functions need `maspsx`.
|
||||
|
||||
## Next task
|
||||
## P5-T6 completed evidence
|
||||
|
||||
P5-T6 — matching cookbook, registry/verification conventions, and the phase gate. Known gaps to
|
||||
close: symbol support in the region registry, `maspsx` integration, and a per-region flag override.
|
||||
- `docs/MATCHING_COOKBOOK.md` — nine byte-proven compiler/assembler findings, each with its basis and
|
||||
limit, plus the open questions.
|
||||
- `docs/MATCHING_CONVENTIONS.md` — what counts as a match, registry format, source naming, the
|
||||
duplicate-sharing mechanism, symbol handling, the verification procedure and the firewall.
|
||||
- `docs/PHASE5_VERIFICATION.md` — the consolidated outcome, the clean gates, the bounded negatives,
|
||||
and the milestone request.
|
||||
- Tooling: `make test` and `make check` targets added; region objects are now symbol-localized, which
|
||||
makes the documented duplicate-sharing mechanism (N registry rows -> 1 source) work and is covered
|
||||
by a synthetic test (53 tests total).
|
||||
|
||||
## Clean gates (last run)
|
||||
|
||||
Synthetic suite 53/53 pass; `make clean`, `make all`, `cmp` and SHA-1 green; `make gate` with one C
|
||||
region gives `c_regions=1`, 0 differing bytes, SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`; 57
|
||||
tracked files and 0 under any prohibited root.
|
||||
|
||||
## Awaiting developer milestone confirmation
|
||||
|
||||
The Phase 5 milestone is ready: the original compiler is identified with byte evidence as
|
||||
`egcs-2.91.66` (PSX) / PsyQ 4.5 `CC1PSX` (open substitute verified instruction-identical), and one C
|
||||
function is matched with a green clean full-binary rebuild. On confirmation: write the PhaseEnd,
|
||||
update `phase-ends/DIGEST.md`, archive this file to `phase-ends/logs/Phase5.md`, commit, and stop
|
||||
without beginning Phase 6.
|
||||
|
||||
## P5-T2 completed evidence
|
||||
|
||||
|
||||
@@ -64,7 +64,7 @@ Identify and verify the original Syphon Filter 3 (USA) build toolchain — compi
|
||||
- Iterate only on comparison-supported root causes; stop after two distinct unexplained failures.
|
||||
- **Verify:** the direct instruction-identical comparison AND the clean full-binary `cmp` + SHA-1 check both pass before C is enabled by default. On failure, keep fallback, record the bounded result, and do not claim a match.
|
||||
|
||||
- [ ] **P5-T6 — Cookbook, registry, verification record, and phase gate** *(xHigh)*
|
||||
- [x] **P5-T6 — Cookbook, registry, verification record, and phase gate** *(xHigh)* — complete; `docs/MATCHING_COOKBOOK.md`, `docs/MATCHING_CONVENTIONS.md`, `docs/PHASE5_VERIFICATION.md`; awaiting developer milestone confirmation.
|
||||
- If P5-T5 succeeds: record reusable compiler/codegen findings in a matching cookbook, add ROM-safe registration and verification conventions for matched functions, and share verified duplicate bodies through the documented mechanism.
|
||||
- Otherwise: document the evidence-backed blocker.
|
||||
- Complete the clean test, baseline/full-binary comparison, firewall, and Git-review gates, then request milestone confirmation.
|
||||
@@ -72,4 +72,4 @@ Identify and verify the original Syphon Filter 3 (USA) build toolchain — compi
|
||||
|
||||
## Milestone
|
||||
|
||||
The original build toolchain is identified with byte evidence and used to produce at least one instruction-identical C function match whose clean full-binary rebuild passes exact `cmp` and SHA-1, **or** Phase 5 records the evidence-backed blocker that prevents a reproducible toolchain and leaves all content as assembly/data fallback. Phase 5 closes only after the developer confirms the milestone; write a PhaseEnd record, update `phase-ends/DIGEST.md`, archive `CURRENT_PHASE.md` to `phase-ends/logs/Phase5.md`, commit, and stop without beginning Phase 6.
|
||||
The original build toolchain is identified with byte evidence and used to produce at least one instruction-identical C function match whose clean full-binary rebuild passes exact `cmp` and SHA-1, **or** Phase 5 records the evidence-backed blocker that prevents a reproducible toolchain and leaves all content as assembly/data fallback. **Met:** compiler identified as `egcs-2.91.66` (PSX) / PsyQ 4.5 `CC1PSX`; first C match `func_80017AD4` gated byte-identical. Phase 5 closes only after the developer confirms the milestone; write a PhaseEnd record, update `phase-ends/DIGEST.md`, archive `CURRENT_PHASE.md` to `phase-ends/logs/Phase5.md`, commit, and stop without beginning Phase 6.
|
||||
|
||||
@@ -317,6 +317,22 @@ def extract_text_bytes(object_path: Path, destination: Path, tools: Toolchain) -
|
||||
return destination.read_bytes()
|
||||
|
||||
|
||||
# A symbol that never exists: `--keep-global-symbol` then localizes everything else.
|
||||
LOCALIZE_SENTINEL = "__sf3_keep_no_global_symbol"
|
||||
|
||||
|
||||
def localize_symbols(object_path: Path, tools: Toolchain) -> None:
|
||||
"""Make a region object's symbols local.
|
||||
|
||||
Region objects are placed by the linker script, not by symbol name, and any
|
||||
cross-reference is supplied as an absolute `--defsym`. Localizing therefore
|
||||
changes nothing about the emitted bytes, and it lets one shared source file
|
||||
be instantiated for several regions without a duplicate-symbol clash.
|
||||
"""
|
||||
run([str(tools.objcopy), f"--keep-global-symbol={LOCALIZE_SENTINEL}",
|
||||
str(object_path)])
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# subcommands
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -411,6 +427,7 @@ def _build(args: argparse.Namespace) -> tuple[Path, Path]:
|
||||
assemble_asm(item.source, object_path, tools)
|
||||
else:
|
||||
compile_c(item.source, object_path, work, tools)
|
||||
localize_symbols(object_path, tools)
|
||||
objects.append(item.object_name)
|
||||
|
||||
script = out / "link.ld"
|
||||
|
||||
@@ -253,6 +253,28 @@ class EndToEndTests(unittest.TestCase):
|
||||
"--out", str(root / "out")])
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
def test_two_regions_can_share_one_source(self) -> None:
|
||||
"""The registry's documented duplicate-sharing mechanism: N rows -> 1 source."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
exe, code_length = self._build_synthetic(root, corrupt=False)
|
||||
# Place a second, identical copy of the body later in the payload.
|
||||
payload = bytearray(exe.read_bytes())
|
||||
body = payload[0x800:0x800 + code_length]
|
||||
payload[0x800 + 0x20:0x800 + 0x20 + code_length] = body
|
||||
exe.write_bytes(bytes(payload))
|
||||
regions = root / "regions.tsv"
|
||||
regions.write_text(
|
||||
f"0x{PAYLOAD:X} 0x{PAYLOAD + code_length:X} {root / 'synthetic.c'}\n"
|
||||
f"0x{PAYLOAD + 0x20:X} 0x{PAYLOAD + 0x20 + code_length:X} {root / 'synthetic.c'}\n",
|
||||
encoding="ascii",
|
||||
)
|
||||
out = root / "out"
|
||||
rc = sf3_match.main(["gate", "--exe", str(exe), "--regions", str(regions),
|
||||
"--out", str(out)])
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual((out / "scus_946_40.rebuilt").read_bytes(), exe.read_bytes())
|
||||
|
||||
def test_refuses_an_existing_destination(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
|
||||
Reference in New Issue
Block a user