diff --git a/doc/build_system.md b/doc/build_system.md new file mode 100644 index 0000000000..f23168dd66 --- /dev/null +++ b/doc/build_system.md @@ -0,0 +1,77 @@ +# Build System +Game Build System. The build system should allow us to +- automatically extract large asset files from the game so we don't have to keep them in this repository +- support modifications (adding new files, modifying files, etc) +- have checks that the resulting data matches the original if expected, assuming no modifications are applied + +Intermediate files should go somewhere in `data` and final files should go in `out`. + +## 1. C++ Build +All C++ code should be built through `cmake`, including the compiler, decompiler, runtime, and any tools. It's expected to be built in the `build` folder. + +## 2. Unpack Assets +This step extracts data from asset files from the game. An asset file is any file that's not a CGO/DGO. All files on the DVD will be in `iso_data/`, and all unpacked data should go in `data/` + +## 3. Unpack DGOs +This step extracts data from CGO/DGO files and unpacks it. All unpacked data should go in `data/` + +## 4. Pack Assets +This step creates final asset files from data in `data/`. The final files should do in `out`. + +## 5. Compile GOAL Code / Data +This step creates all the data for the CGOs/DGOs. The compiler will have a big list of files which it will compile in order. Each `.gc` file contains GOAL code, which is compiled to a GOAL code object (`.o` file) stored in `data/obj`. Each `.gd` file is a "GOAL data description file" which contains a short script to launch the appropriate tool to produce the data object. The objects will also be stored in `data/obj`, but will have a `.go` extension. + +## 6. Create CGO/DGOs +This step takes the output from step 5, (stored in `data/obj`) and creates all the DGO/CGO files, which go in `out`. The list of what goes in the DGOs is in `goal_src/build/dgos.txt`. + +## 7. Asset Check +There might be some asset files which we expect to be unchanged between the version in `iso_data` and in `out`. At this point we should check if these are actually the same. + +## 8. CGO/DGO check +The data objects contained in CGO/DGO files should be the same between the original version and the ported version. We should go through each CGO/DGO file, look at each object, and if it's a data object, make sure it matches. + + +# What can be worked on now? +## C++ Build +Getting more things working on Windows. + +### Compiler +The compiler right now only supports GOOS, the macro language. It may be useful to play around with this. There may be missing features or bugs in the language. + + +## Unpack Assets +I imagine that each type of asset file will have its own tool to unpack things. We'll need a master "unpacker" tool which runs the correct tool on each file. + +This unpacker could take a config file, maybe JSON?, that says what to do for each file: +``` +[ +{"filename":"MUS/SNOW.MUS", "unpack_tool":"music_unpacker", "destination":"data/music/snow/"... arguments to music_unpacker}, +... +] +``` +and then would run the correct tool on each file. For starters, we could make a simple unpacking tool named `copy` that just directly copies the asset file into the `data` folder. + +We could also look into particular files and start writing unpacking tools. + +## Unpack DGOs +Again, there will be a main tool which runs through each DGO/CGO file, finds data objects, then runs an appropriate unpacking tools based on the type of data object (level, texture, art-group,...). I imagine there would a config file (json maybe?) that looks something like this: +``` +{"dgo":"MAI.DGO", "objects":[{"tpage1234", "texture-unpacker", ...}, {"spider", "art-group-unpacker", ...}]} +``` + +My plan for duplicate data objects: +- When there are duplicates, only one will actually be unpacked +- Ones that aren't unpacked will be checked if they are the same as the unpacked one to make sure we don't make a mistake +- The list of which objects should be unpacked vs. checked, and what name the unpacking should get will be determined by the decompiler. +- If there is an object not in the config file, or a missing object, then there is an error + +The actual unpacking tools probably can't be developed until I package up the GOAL disassembler into a usable library, and we probably won't be able to understand the data format until later. We could write a very simple `copy` unpacker which just copies the object data directly into `data` and doesn't unpack it at all. + +## Pack Assets +Again, there will be a main tool which runs the appropriate packer for each file. We could write this main tool, and a `copy` packer which just copies the exact file in `data/` to `out/`. + +## Asset Check +Just a tool with a list of asset files to check, does a comparison and reports an error if there is a difference. We might expect some files to change slightly in the PC version, and these will be excluded from the check. + +## CGO/DGO Check +Iterates through objects in the original / rebuilt CGO/DGOs. Code objects are not checked, but data objects are checked. Should also check that the order of objects is the same, and all are present. \ No newline at end of file