diff --git a/.gitignore b/.gitignore index 9a5df39548..d722b67441 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ cmake-build-debug/* .idea/* build/* +/.vs \ No newline at end of file diff --git a/CMakeLists.txt b/CMakeLists.txt index 45706cfa7f..ce4228908f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -5,10 +5,25 @@ project(jak) set(CMAKE_CXX_STANDARD 14) # optimization level can be set here. Note that game/ overwrites this for building game C++ code. -set(CMAKE_CXX_FLAGS "-O0 -ggdb -Wall \ --Wextra -Wcast-align -Wcast-qual -Wdisabled-optimization -Wformat=2 \ --Winit-self -Wmissing-include-dirs -Woverloaded-virtual \ --Wredundant-decls -Wshadow -Wsign-promo ") +set(CMAKE_CXX_FLAGS "-O3 -ggdb") + +# Set default compile flags for GCC +if(CMAKE_COMPILER_IS_GNUCXX) + message(STATUS "GCC detected, adding compile flags") + set(CMAKE_CXX_FLAGS + "${CMAKE_CXX_FLAGS} + -Wall-Winit-self + -Wextra + -Wcast-align + -Wcast-qual + -Wdisabled-optimization + -Wformat=2 + -Wmissing-include-dirs + -Woverloaded-virtual + -Wredundant-decls + -Wshadow + -Wsign-promo") +endif(CMAKE_COMPILER_IS_GNUCXX) # includes relative to top level jak-project folder include_directories(./) diff --git a/README.md b/README.md index 556a7f5924..47dc3c4de1 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,77 @@ -Project Structure ----------------------- -Requirements: +# Jak Project + +## Table of Contents + + + +- [Jak Project](#jak-project) + - [Table of Contents](#table-of-contents) + - [Requirements](#requirements) + - [Getting Started - Linux (Ubuntu)](#getting-started---linux-ubuntu) + - [Getting Started - Windows](#getting-started---windows) + - [Project Layout](#project-layout) + - [Design](#design) + - [Current State](#current-state) + - [Coding Guidelines](#coding-guidelines) + - [TODOs](#todos) + - [Project Description](#project-description) + - [GOAL Decompiler](#goal-decompiler) + - [GOAL Runtime](#goal-runtime) + - [GOAL Compiler](#goal-compiler) + - [Asset Extraction Tool](#asset-extraction-tool) + - [Asset Packing Tool](#asset-packing-tool) + + + +## Requirements + - `cmake` for build system - `clang-format` for formatting code (there is already a `.clang-format` provided) - `gtest` for testing. (Run `git submodule update --init --recursive` to check out the repository) -- `nasm` for assembling x86. There isn't much x86 assembly so if there's a better way to do this for windows, we can change. +- `nasm` for assembling x86 - Third party libraries (`nlohmann/json`, `minilzo`, and `linenoise`) are provided in the `third-party` folder -Setup (for Ubuntu): -``` +## Getting Started - Linux (Ubuntu) + +Install Packages and Init Repository + +```bash sudo apt install gcc make cmake build-essential g++ nasm clang-format +git submodule update --init --recursive ``` -Layout: +Compile + +```bash +mkdir build && cd build && cmake .. && make -j +``` + +Run Tests + +```bash +./test.sh +``` + +## Getting Started - Windows + +Install Visual Studio 2019 and get the C++ and CMake tools via the Visual Studio Installer + +On Windows, it's recommended to get Scoop to use as a package manager, making the follow steps _much_ easier. Follow the steps on the bottom of the homepage here https://scoop.sh/ + +Once Scoop is installed, run the following command: + +```ps +scoop install llvm nasm +``` + +Open the project as a CMake project, browse for the root level `CMakeLists.txt`: + +![](./doc/imgs/open-cmake-vs.png) + +TODO - more steps to follow as we actually figure it out! + +## Project Layout + - `goalc` is the GOAL compiler - `gs` contains GOOS code for parts of GOOS implemented in GOOS - `gc` contains GOAL code for parts of GOAL implemented in GOAL (must generate no machine code, just defining macros) @@ -29,8 +88,10 @@ Layout: - `tests` will contain all tests - `asset_tool` will contain the asset packer/unpacker -Design: +## Design + (if anybody has better ideas, feel free to suggest improvements! This is just a rough plan for now) + - All C++ code should build from the top-level `cmake`. - All C++ applications (GOAL compiler, asset extractor, asset packer, runtime, test) should have a script in the top level which launches them. - All file paths should be relative to the `jak` folder. @@ -48,11 +109,13 @@ Design: - `./gc.sh` : run the compiler in interactive mode - `./gs.sh` : run a goos interpreter in interactive mode - `./decomp.sh : run the decompiler - -Current state: + +## Current State + - GOAL compiler just implements the GOOS Scheme Macro Language. Running `./gc.sh` just loads the GOOS library (`goalc/gs/goos-lib.gs`) and then goes into an interactive mode. Use `(exit)` to exit. - `./test.sh` runs tests for some game C++ code, for GOOS, for the reader, for the listener connection, and for some early emitter stuff. -- The runtime boots in `fakeiso` mode which will load some dummy files. Then the C Kernel (`game/kernel`) will load the `KERNEL.CGO` and `GAME.CGO` files, which are from the "proof of concept" GOAL compiler. If you run `./gk.sh`, you should see it load stuff, then print: +- The runtime boots in `fakeiso` mode which will load some dummy files. Then the C Kernel (`game/kernel`) will load the `KERNEL.CGO` and `GAME.CGO` files, which are from the "proof of concept" GOAL compiler. If you run `./gk.sh`, you should see it load stuff, then print: + ``` calling play! ~~ HACK ~~ : fake play has been called @@ -61,13 +124,16 @@ InitCheckListener kernel: machine started ``` + where the `~~ HACK ~~` message is from code in `KERNEL.CGO`. -Code Guidelines: +## Coding Guidelines + - Avoid warnings - Use asserts over throwing exceptions in game code (throwing exceptions from C code called by GOAL code is sketchy) -TODOS: +## TODOs + - Build on Windows! - Networking - File paths @@ -82,8 +148,8 @@ TODOS: - Clean up possible duplicate code in compiler/decompiler `util` folder, consider a common utility library - Clean up header guard names (or just use `#pragma once`?) - Investigate a better config format - - The current JSON library seems to have issues with comments, which I really like -- Clean up use of namespaces + - The current JSON library seems to have issues with comments, which I really like +- Clean up use of namespaces - Clean up the print message when `gk` starts. - Finish commenting runtime stuff - Runtime document @@ -95,11 +161,10 @@ TODOS: - Clean up decompiler print spam, finish up the CFG stuff - Decompiler document - -Project Description ------------------------ +## Project Description This project is to port Jak 1 (NTSC, "black label" version) to PC. The strategy is to: + - recompile for x86 to get much better performance than emulation - create human-reabable GOAL source code that can be modified - create a GOAL compiler for x86-64 which supports live patching of code like the original @@ -107,6 +172,7 @@ This project is to port Jak 1 (NTSC, "black label" version) to PC. The strategy - unpack assets in a format that can be modified There are 6 components to this project + - GOAL decompiler. The result will be manually cleaned up for running on a PC. - GOAL compiler for x86-64. - Game source code, made from cleaning up the result of the GOAL decompiler. @@ -115,25 +181,27 @@ There are 6 components to this project - Asset packing tool. The process to build the port will be + - Build data extraction tool, GOAL compiler, and GOAL runtime library (all written in C++) - Run the GOAL compiler on the game source code to build the game engine - Run asset extraction on the game disc to get level data, textures, geometry data, music... - Run the asset packing tool to combine the unpacked assets with the compiled game engine to create the game! Some statistics: + - Estimated ~500k lines of GOAL code - 10410 functions - 5451 functions with no control flow (no branching, loops, if/else, short-circuiting boolean operators, gotos, etc) -The rough timeline is to finish sometime in 2022. If it looks like this is impossible, the project will be abandoned. But I have already spent about 4 months preparing to start this and seems doable. I also have some background in compilers, and familiarity with PS2 (worked on DobieStation PS2 emulator) / MIPS in general (wrote a PS1 emulator). I think the trick will be making good automated tools - the approach taken for SM64 and other N64 decompilations is way too labor-intensive to work. +The rough timeline is to finish sometime in 2022. If it looks like this is impossible, the project will be abandoned. But I have already spent about 4 months preparing to start this and seems doable. I also have some background in compilers, and familiarity with PS2 (worked on DobieStation PS2 emulator) / MIPS in general (wrote a PS1 emulator). I think the trick will be making good automated tools - the approach taken for SM64 and other N64 decompilations is way too labor-intensive to work. +### GOAL Decompiler -GOAL Decompiler ------------------- -The decompiler is in progress, at +The decompiler is in progress, at https://github.com/water111/jak-disassembler Here is the plan for writing the decompiler: + - [x] Decode the CGO/DGO format. - [x] Decode the linking data format. - [x] Identify all code and disassemble @@ -147,10 +215,10 @@ Here is the plan for writing the decompiler: - [ ] Variable map and scoping - [ ] S-expression construction (expression stack) +### GOAL Runtime + +The "runtime" will be a replacement for all of the C/C++ code of the original game. There is C/C++ code that runs on the main processor (EE) and the separate I/O processor (IOP). -GOAL Runtime --------------- -The "runtime" will be a replacement for all of the C/C++ code of the original game. There is C/C++ code that runs on the main processor (EE) and the separate I/O processor (IOP). - The "C Kernel", which runs on the EE and contains - [ ] File I/O (for debugging, not used by retail game) - [x] Initialization to boostrap the GOAL Kernel and start the game engine @@ -180,17 +248,18 @@ The "Sony libraries" are a simple wrapper around my `system` library, which impl Likely there will be sound/graphics code in here at some point, but this part is not fully planned yet. -GOAL Compiler ---------------- +### GOAL Compiler + The GOAL compiler will target x86-64. At first just Linux. There is a macro language called GOOS which is basically just Scheme but with a few bonus features. -The compiler will reuse a significant amount of code from my existing LISP compiler for x86-64. I have a very bad WIP combination which is capable of building a modified `gkernel.gc` for x86 as a proof of concept. It can create and run functions in threads. +The compiler will reuse a significant amount of code from my existing LISP compiler for x86-64. I have a very bad WIP combination which is capable of building a modified `gkernel.gc` for x86 as a proof of concept. It can create and run functions in threads. -An important part of the compiler is the test suite. Without tests the compiler will be full of bugs. So every feature should have a good set of tests. +An important part of the compiler is the test suite. Without tests the compiler will be full of bugs. So every feature should have a good set of tests. The major components are - [ ] GOAL-IR, a typed linear intermediate representation for GOAL code + - [ ] "Environment" - [ ] "Ref" - [ ] Constant propagation of integers/floats @@ -198,6 +267,7 @@ The major components are - [ ] Ref `in_gpr` - [ ] The type system + - [ ] Type inheritance - [ ] Integer/float/pointer types (value semantics) - [ ] Reference types @@ -216,41 +286,41 @@ The major components are - [ ] Built-in types in the GOAL runtime/C Kernel - [x] The GOOS Macro Language + - [x] S-expression parser (the "Reader") - [x] Reader text db (for error messages that point to a specific line) - [x] Scheme interpreter - + - [ ] Front-end (convert s-expressions (a tree structure) to GOAL-IR (a linear representation)) - - [ ] Parsing helpers - - [ ] Macro expansion - - [ ] Control flow/block forms - - [ ] Type definitions - - [ ] Inline assembly forms - - [ ] Function/method call - - [ ] Math forms - - [ ] Lexical scoping (immediate application of `lambda`) - - [ ] Function inlining (slightly different scoping rules of immediate `lambda`) - - [ ] Function/macro definition - - [ ] Static Objects +- [ ] Parsing helpers +- [ ] Macro expansion +- [ ] Control flow/block forms +- [ ] Type definitions +- [ ] Inline assembly forms +- [ ] Function/method call +- [ ] Math forms +- [ ] Lexical scoping (immediate application of `lambda`) +- [ ] Function inlining (slightly different scoping rules of immediate `lambda`) +- [ ] Function/macro definition +- [ ] Static Objects - [ ] Regsiter allocation - - [ ] Analysis - - [ ] Allocation - - [ ] Stack spilling - - [ ] `xmm` and `gpr` promotion/demotions for EE 128-bit register usage - +- [ ] Analysis +- [ ] Allocation +- [ ] Stack spilling +- [ ] `xmm` and `gpr` promotion/demotions for EE 128-bit register usage - [ ] Codegen / Emitter (convert GOAL-IR + register allocations to x86 object file format) - - [ ] Emitter (convert GOAL-IR to instructions) - - [ ] x86-64 instruction generation (actually generate the machine code) - - [ ] Linking data - - [ ] 64-bit GPR - - [ ] 32-bit float - - [ ] 128-bit GPR - - [ ] 32-bit float x4 vector register - - [ ] function prologue/epilogue - - [ ] stack spilling - - [ ] static object and static object links +- [ ] Emitter (convert GOAL-IR to instructions) +- [ ] x86-64 instruction generation (actually generate the machine code) +- [ ] Linking data +- [ ] 64-bit GPR +- [ ] 32-bit float +- [ ] 128-bit GPR +- [ ] 32-bit float x4 vector register +- [ ] function prologue/epilogue +- [ ] stack spilling +- [ ] static object and static object links - [ ] Listener/REPL - [ ] Network connection @@ -260,14 +330,14 @@ The major components are - [ ] Expand single macro debugging feature - [ ] Interface for running gtests +### Asset Extraction Tool -Asset Extraction Tool ------------------------ -Not started yet. The simplest version of this tool is just to use the decompiler logic to turn the level/art-group/texture/TXT files into GOAL source code, and copy all STR/sound/visibility files, as these don't need to be modified. +Not started yet. The simplest version of this tool is just to use the decompiler logic to turn the level/art-group/texture/TXT files into GOAL source code, and copy all STR/sound/visibility files, as these don't need to be modified. Eventually this should export to a more useful format. File formats: + - [ ] Art group (a GOAL object format) - There may be more formats related to art groups. - [ ] Texture page (a GOAL object format) @@ -282,12 +352,8 @@ File formats: - [ ] Loading screen image - [ ] save game icon (I do not care about this) +### Asset Packing Tool -Asset Packing Tool ------------------------ -Packs together all assets/compiled code/runtime into a format that can be played. The simplest version to go with the simplest extraction tool will just pass the level/art-group/texture/TXT files to the compiler, and copy STR/sound/visbility files into the fakeiso. Then pack in CGOs/DGOs. +Packs together all assets/compiled code/runtime into a format that can be played. The simplest version to go with the simplest extraction tool will just pass the level/art-group/texture/TXT files to the compiler, and copy STR/sound/visbility files into the fakeiso. Then pack in CGOs/DGOs. It's important that the asset extraction/packing can be automated so we can avoid distributing the assets, which are large and probably not supposed to be distributed. - - - diff --git a/doc/imgs/cmake-target-view.png b/doc/imgs/cmake-target-view.png new file mode 100644 index 0000000000..2a69df167a Binary files /dev/null and b/doc/imgs/cmake-target-view.png differ diff --git a/doc/imgs/open-cmake-vs.png b/doc/imgs/open-cmake-vs.png new file mode 100644 index 0000000000..914a2ff7fa Binary files /dev/null and b/doc/imgs/open-cmake-vs.png differ