goldeneye_src/scripts/IMAGE_BUILD_SYSTEM.md

# Image Build System

## Overview

The image build system synchronizes data between `imagelist.u.csv` (ROM offsets/sizes) and `assets/images.def` (image names and properties). The system:
- Uses ROM offsets and sizes from imagelist.u.csv as ground truth for positioning
- Uses image names from images.def for proper file naming
- Maintains original ROM structure for byte-perfect matching
- Pads combined.bin to 16-byte (0x10) boundary as required by ROM

## Key Files

- **`imagelist.u.csv`** - Ground truth for ROM offsets and sizes
  - Format: `offset,size,path,0,1`
  - Contains absolute ROM offsets (starting at 9403888 / 0x8F6D50)
  - 2699 entries total (2698 images + 1 dummy for padding calculation)

- **`assets/images.def`** - Ground truth for image names and properties
  - Format: `IMAGE(NAME, SIZE, HIT_SOUND, HIT_TYPE, F3, F4, F5, F6)`
  - Order in this file must match imagelist.u.csv order
  - SIZE field should match CSV but CSV is authoritative
  - 2698 IMAGE() entries

- **`assets/images/split/*.bin`** - Individual texture files
  - Named using IMAGE_ enum name (e.g., `COPYICON.bin`)
  - Must match names in images.def
  - Extracted from ROM using imagelist.u.csv offsets/sizes

- **`build/u/imagelist.csv`** - Build-time generated file
  - Auto-generated by syncing imagelist.u.csv with images.def
  - Combines ROM offsets/sizes with proper file names
  - Format: `offset,size,path,0,1`

## Scripts

### `scripts/make/sync_imagelist_with_def.py`

Primary script that synchronizes imagelist.u.csv with images.def.

**Features:**
- Reads image names from images.def (in order)
- Reads ROM offsets from imagelist.u.csv (for ROM images only)
- Recalculates ALL sizes from actual .bin files on disk
- Generates build imagelist with proper names, ROM offsets, and current sizes
- Validates that entry counts match (warns if mismatch)
- CSV offsets preserved for ROM layout, but sizes always reflect actual files

**Usage:**
```bash
python3 scripts/make/sync_imagelist_with_def.py build/u/imagelist.csv
```

**Output Format:**
```
9403888,1876,assets/images/split/COPYICON.bin,0,1
9405764,362,assets/images/split/X.bin,0,1
...
```

### `scripts/make/combine_images_named.sh`

Combines individual .bin files into single combined.bin.

**Features:**
- Reads imagelist.csv for correct ordering
- Concatenates files in exact order
- Pads output to 16-byte (0x10) boundary
- Warns about missing files (skips them)

**Usage:**
```bash
scripts/make/combine_images_named.sh build/u/imagelist.csv assets/images/combined
```

**Padding:**
The script automatically pads combined.bin to the next 16-byte boundary, which is required for proper ROM alignment.

## Makefile Integration

```makefile
# Generate imagelist by syncing imagelist.u.csv (ROM offsets/sizes) with images.def (names)
$(BUILD_DIR)/imagelist.csv: imagelist.u.csv assets/images.def
	@mkdir -p $(BUILD_DIR)
	python3 scripts/make/sync_imagelist_with_def.py $@

assets/images/combined/combined.bin: $(BUILD_DIR)/imagelist.csv
	scripts/make/combine_images_named.sh $(BUILD_DIR)/imagelist.csv assets/images/combined
```

**Dependency Chain:**
1. `imagelist.u.csv` (ROM offsets/sizes) + `assets/images.def` (names)
2. → `build/u/imagelist.csv` (synced build list)
3. → `assets/images/combined/combined.bin` (final combined texture with 0x10 padding)

## Workflow

### Initial ROM Extraction

To extract images from the original ROM:

```bash
# This will extract all images with proper names using ROM offsets
./scripts/extract_baserom.u.sh
```

The extraction script:
1. Runs `sync_imagelist_with_def.py` to generate imagelist with proper names
2. Extracts each image from ROM using the offset/size from imagelist.u.csv
3. Creates named .bin files in `assets/images/split/`

### Adding a New Image

**New images can be added by simply updating `images.def`!** The CSV is only used for initial ROM extraction.

1. Add entry to `assets/images.def` (at end or desired position):
   ```c
   IMAGE(MY_NEW_TEXTURE, 0x1000, HIT_DEFAULT, HIT_DEFAULT, 0, 0, 0, 0)
   ```
   - SIZE will be auto-detected from actual .bin file
   - Position in file determines position in combined binary

2. Create the .bin file:
   ```bash
   cp my_texture.bin assets/images/split/MY_NEW_TEXTURE.bin
   ```

3. Rebuild:
   ```bash
   make assets/images/combined/combined.bin
   ```

The build system will:
- Use ROM offsets from `imagelist.u.csv` for original 2698 images
- **Recalculate ALL sizes** from actual .bin files (not from CSV)
- Calculate new offsets for additional images based on actual file sizes
- Automatically detect the file size if the .bin exists
- Use a default size (0x1000) if the .bin doesn't exist yet
- CSV offsets are preserved (for ROM layout) but sizes are always current

**Adding images after ROM images:**
- Original ROM images: Use data from imagelist.u.csv (positions 0-2697)
- New images: Automatically positioned after ROM images with calculated offsets
- Combined.bin will be larger, breaking ROM matching (expected for mods)

**Inserting images within ROM range:**
To insert a new image within the original 2698 ROM images, you must update `imagelist.u.csv` to adjust offsets. Simply appending new images to the end is much easier.

### Updating an Existing Image (Same Size)

1. Replace the .bin file:
   ```bash
   cp new_version.bin assets/images/split/COPYICON.bin
   ```

2. Rebuild:
   ```bash
   make assets/images/combined/combined.bin
   ```

If the replacement has the **exact same size**, the ROM will remain matching.

### Updating an Existing Image (Different Size)

**Important:** Changing image sizes breaks ROM matching.

1. Update size in `imagelist.u.csv` for the image
2. Update all subsequent image offsets in imagelist.u.csv
3. Update SIZE in corresponding IMAGE() entry in images.def
4. Replace the .bin file
5. Rebuild

This is only for development/modding purposes.

### Checking for Issues

The sync_imagelist_with_def.py script will report:
- Entry count mismatches between CSV and .def
- Missing .bin files during combine (warns but continues)
- Total combined size and padding

### Verifying ROM Matching

To verify the image system produces byte-perfect matching:

```bash
# Extract image section from ROM
dd if=baserom.u.z64 bs=1 skip=9403888 count=3075864 2>/dev/null | sha1sum

# Compare with built combined.bin (excluding padding)
head -c 3075864 assets/images/combined/combined.bin | sha1sum
```

Both should produce: `044fca472bf6ef6691fa02ff1b65ff474d86a9fa`

## Key Features

**Current System:**
- Uses ROM offsets/sizes from imagelist.u.csv (preserves original ROM structure)
- Uses proper names from images.def (better than image0.bin numbering)
- Automatically syncs both sources during build
- Pads combined.bin to 0x10 boundary as required by ROM
- Maintains byte-perfect ROM matching
- Named images (COPYICON.bin, DELICON.bin, etc.)

**Design Philosophy:**
- `imagelist.u.csv` = ROM offset database (tracks baserom addresses, never modified during build)
- `images.def` = Complete image database (names, properties, and all images including new ones)
- Build system uses CSV offsets for ROM images, recalculates ALL sizes from actual files
- CSV sizes are informational/historical only - actual file sizes always used
- New images can be added to images.def without touching CSV
- Extraction uses ROM data to create named .bin files
- Modding-friendly: just add entries to images.def and provide .bin files
- File modifications automatically reflected in build (sizes recalculated every time)

## C Code Integration

The `images.def` file is included in C code to generate:

1. **Image ID enum** (`src/bondconstants.h`):
   ```c
   #define IMAGE(NAME, SZ, HS, HT, F3, F4, F5, F6) IMAGE_ ## NAME,
   typedef enum IMAGEIDS {
       #include <assets/images.def>
       IMAGEIDS_COUNT
   } IMAGEIDS;
   ```

2. **Image size enum** (`src/bondconstants.h`):
   ```c
   #define IMAGE(NAME, SZ, HS, HT, F3, F4, F5, F6) IMAGE_ ## NAME ## _SIZE = SZ,
   typedef enum IMGAGESIZES {
       #include <assets/images.def>
       IMGAGESIZES_COUNT
   } IMGAGESIZES;
   ```

3. **Image properties table** (`src/game/image.c`):
   ```c
   #define IMAGE(NAME, SZ, HS, HT, F3, F4, F5, F6) {HS, HT, SZ, F3, F4, F5, F6 },
   struct image_entry g_Textures[] = {
       #include <assets/images.def>
       {HIT_DEFAULT, HIT_DEFAULT,0xFFFF,0,0,0,0}
   };
   ```

This is why the SIZE field must remain in images.def (it's used by C code), but it's now automatically maintained.

## Testing

To verify the system works:

```bash
# Clean and rebuild
make dataclean
make assets/images/combined/combined.bin

# Check output
ls -lh assets/images/combined/combined.bin
head build/u/imagelist.csv
tail build/u/imagelist.csv
```

Expected output:
- combined.bin should be 3075872 bytes (3075864 + 8 padding to 0x10 boundary)
- imagelist.csv should show ROM offsets starting at 9403888
- imagelist.csv should have 2698 entries with proper names
- No errors about missing files (unless images are actually missing)

Example imagelist.csv entries:
```
9403888,1876,assets/images/split/COPYICON.bin,0,1
9405764,362,assets/images/split/X.bin,0,1
...
12477898,513,assets/images/split/2696.bin,0,1
12478411,1341,assets/images/split/2697.bin,0,1
```

## File Sizes

- **ROM image section:** 3075864 bytes (0x8F6D50 to 0xBE5C18)
- **Combined.bin (with padding):** 3075872 bytes (0x2EEF20)
- **Padding:** 8 bytes to align to 0x10 boundary
- **Number of images:** 2698
- **CSV entries:** 2699 (2698 images + 1 dummy for offset calculation)