mirror of
https://github.com/open-goal/jak-project
synced 2026-07-09 14:55:51 -04:00
Format and start adding Windows steps
This commit is contained in:
Tyler Wilding
@@ -2,3 +2,4 @@
|
||||
cmake-build-debug/*
|
||||
.idea/*
|
||||
build/*
|
||||
/.vs
|
||||
+19
-4
@@ -5,10 +5,25 @@ project(jak)
|
||||
set(CMAKE_CXX_STANDARD 14)
|
||||
|
||||
# optimization level can be set here. Note that game/ overwrites this for building game C++ code.
|
||||
set(CMAKE_CXX_FLAGS "-O0 -ggdb -Wall \
|
||||
-Wextra -Wcast-align -Wcast-qual -Wdisabled-optimization -Wformat=2 \
|
||||
-Winit-self -Wmissing-include-dirs -Woverloaded-virtual \
|
||||
-Wredundant-decls -Wshadow -Wsign-promo ")
|
||||
set(CMAKE_CXX_FLAGS "-O3 -ggdb")
|
||||
|
||||
# Set default compile flags for GCC
|
||||
if(CMAKE_COMPILER_IS_GNUCXX)
|
||||
message(STATUS "GCC detected, adding compile flags")
|
||||
set(CMAKE_CXX_FLAGS
|
||||
"${CMAKE_CXX_FLAGS}
|
||||
-Wall-Winit-self
|
||||
-Wextra
|
||||
-Wcast-align
|
||||
-Wcast-qual
|
||||
-Wdisabled-optimization
|
||||
-Wformat=2
|
||||
-Wmissing-include-dirs
|
||||
-Woverloaded-virtual
|
||||
-Wredundant-decls
|
||||
-Wshadow
|
||||
-Wsign-promo")
|
||||
endif(CMAKE_COMPILER_IS_GNUCXX)
|
||||
|
||||
# includes relative to top level jak-project folder
|
||||
include_directories(./)
|
||||
|
||||
@@ -1,18 +1,77 @@
|
||||
Project Structure
|
||||
----------------------
|
||||
Requirements:
|
||||
# Jak Project
|
||||
|
||||
## Table of Contents
|
||||
|
||||
<!-- toc -->
|
||||
|
||||
- [Jak Project](#jak-project)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Requirements](#requirements)
|
||||
- [Getting Started - Linux (Ubuntu)](#getting-started---linux-ubuntu)
|
||||
- [Getting Started - Windows](#getting-started---windows)
|
||||
- [Project Layout](#project-layout)
|
||||
- [Design](#design)
|
||||
- [Current State](#current-state)
|
||||
- [Coding Guidelines](#coding-guidelines)
|
||||
- [TODOs](#todos)
|
||||
- [Project Description](#project-description)
|
||||
- [GOAL Decompiler](#goal-decompiler)
|
||||
- [GOAL Runtime](#goal-runtime)
|
||||
- [GOAL Compiler](#goal-compiler)
|
||||
- [Asset Extraction Tool](#asset-extraction-tool)
|
||||
- [Asset Packing Tool](#asset-packing-tool)
|
||||
|
||||
<!-- tocstop -->
|
||||
|
||||
## Requirements
|
||||
|
||||
- `cmake` for build system
|
||||
- `clang-format` for formatting code (there is already a `.clang-format` provided)
|
||||
- `gtest` for testing. (Run `git submodule update --init --recursive` to check out the repository)
|
||||
- `nasm` for assembling x86. There isn't much x86 assembly so if there's a better way to do this for windows, we can change.
|
||||
- `nasm` for assembling x86
|
||||
- Third party libraries (`nlohmann/json`, `minilzo`, and `linenoise`) are provided in the `third-party` folder
|
||||
|
||||
Setup (for Ubuntu):
|
||||
```
|
||||
## Getting Started - Linux (Ubuntu)
|
||||
|
||||
Install Packages and Init Repository
|
||||
|
||||
```bash
|
||||
sudo apt install gcc make cmake build-essential g++ nasm clang-format
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
Layout:
|
||||
Compile
|
||||
|
||||
```bash
|
||||
mkdir build && cd build && cmake .. && make -j
|
||||
```
|
||||
|
||||
Run Tests
|
||||
|
||||
```bash
|
||||
./test.sh
|
||||
```
|
||||
|
||||
## Getting Started - Windows
|
||||
|
||||
Install Visual Studio 2019 and get the C++ and CMake tools via the Visual Studio Installer
|
||||
|
||||
On Windows, it's recommended to get Scoop to use as a package manager, making the follow steps _much_ easier. Follow the steps on the bottom of the homepage here https://scoop.sh/
|
||||
|
||||
Once Scoop is installed, run the following command:
|
||||
|
||||
```ps
|
||||
scoop install llvm nasm
|
||||
```
|
||||
|
||||
Open the project as a CMake project, browse for the root level `CMakeLists.txt`:
|
||||
|
||||
|
||||
|
||||
TODO - more steps to follow as we actually figure it out!
|
||||
|
||||
## Project Layout
|
||||
|
||||
- `goalc` is the GOAL compiler
|
||||
- `gs` contains GOOS code for parts of GOOS implemented in GOOS
|
||||
- `gc` contains GOAL code for parts of GOAL implemented in GOAL (must generate no machine code, just defining macros)
|
||||
@@ -29,8 +88,10 @@ Layout:
|
||||
- `tests` will contain all tests
|
||||
- `asset_tool` will contain the asset packer/unpacker
|
||||
|
||||
Design:
|
||||
## Design
|
||||
|
||||
(if anybody has better ideas, feel free to suggest improvements! This is just a rough plan for now)
|
||||
|
||||
- All C++ code should build from the top-level `cmake`.
|
||||
- All C++ applications (GOAL compiler, asset extractor, asset packer, runtime, test) should have a script in the top level which launches them.
|
||||
- All file paths should be relative to the `jak` folder.
|
||||
@@ -48,11 +109,13 @@ Design:
|
||||
- `./gc.sh` : run the compiler in interactive mode
|
||||
- `./gs.sh` : run a goos interpreter in interactive mode
|
||||
- `./decomp.sh : run the decompiler
|
||||
|
||||
Current state:
|
||||
|
||||
## Current State
|
||||
|
||||
- GOAL compiler just implements the GOOS Scheme Macro Language. Running `./gc.sh` just loads the GOOS library (`goalc/gs/goos-lib.gs`) and then goes into an interactive mode. Use `(exit)` to exit.
|
||||
- `./test.sh` runs tests for some game C++ code, for GOOS, for the reader, for the listener connection, and for some early emitter stuff.
|
||||
- The runtime boots in `fakeiso` mode which will load some dummy files. Then the C Kernel (`game/kernel`) will load the `KERNEL.CGO` and `GAME.CGO` files, which are from the "proof of concept" GOAL compiler. If you run `./gk.sh`, you should see it load stuff, then print:
|
||||
- The runtime boots in `fakeiso` mode which will load some dummy files. Then the C Kernel (`game/kernel`) will load the `KERNEL.CGO` and `GAME.CGO` files, which are from the "proof of concept" GOAL compiler. If you run `./gk.sh`, you should see it load stuff, then print:
|
||||
|
||||
```
|
||||
calling play!
|
||||
~~ HACK ~~ : fake play has been called
|
||||
@@ -61,13 +124,16 @@ InitCheckListener
|
||||
kernel: machine started
|
||||
|
||||
```
|
||||
|
||||
where the `~~ HACK ~~` message is from code in `KERNEL.CGO`.
|
||||
|
||||
Code Guidelines:
|
||||
## Coding Guidelines
|
||||
|
||||
- Avoid warnings
|
||||
- Use asserts over throwing exceptions in game code (throwing exceptions from C code called by GOAL code is sketchy)
|
||||
|
||||
TODOS:
|
||||
## TODOs
|
||||
|
||||
- Build on Windows!
|
||||
- Networking
|
||||
- File paths
|
||||
@@ -82,8 +148,8 @@ TODOS:
|
||||
- Clean up possible duplicate code in compiler/decompiler `util` folder, consider a common utility library
|
||||
- Clean up header guard names (or just use `#pragma once`?)
|
||||
- Investigate a better config format
|
||||
- The current JSON library seems to have issues with comments, which I really like
|
||||
- Clean up use of namespaces
|
||||
- The current JSON library seems to have issues with comments, which I really like
|
||||
- Clean up use of namespaces
|
||||
- Clean up the print message when `gk` starts.
|
||||
- Finish commenting runtime stuff
|
||||
- Runtime document
|
||||
@@ -95,11 +161,10 @@ TODOS:
|
||||
- Clean up decompiler print spam, finish up the CFG stuff
|
||||
- Decompiler document
|
||||
|
||||
|
||||
Project Description
|
||||
-----------------------
|
||||
## Project Description
|
||||
|
||||
This project is to port Jak 1 (NTSC, "black label" version) to PC. The strategy is to:
|
||||
|
||||
- recompile for x86 to get much better performance than emulation
|
||||
- create human-reabable GOAL source code that can be modified
|
||||
- create a GOAL compiler for x86-64 which supports live patching of code like the original
|
||||
@@ -107,6 +172,7 @@ This project is to port Jak 1 (NTSC, "black label" version) to PC. The strategy
|
||||
- unpack assets in a format that can be modified
|
||||
|
||||
There are 6 components to this project
|
||||
|
||||
- GOAL decompiler. The result will be manually cleaned up for running on a PC.
|
||||
- GOAL compiler for x86-64.
|
||||
- Game source code, made from cleaning up the result of the GOAL decompiler.
|
||||
@@ -115,25 +181,27 @@ There are 6 components to this project
|
||||
- Asset packing tool.
|
||||
|
||||
The process to build the port will be
|
||||
|
||||
- Build data extraction tool, GOAL compiler, and GOAL runtime library (all written in C++)
|
||||
- Run the GOAL compiler on the game source code to build the game engine
|
||||
- Run asset extraction on the game disc to get level data, textures, geometry data, music...
|
||||
- Run the asset packing tool to combine the unpacked assets with the compiled game engine to create the game!
|
||||
|
||||
Some statistics:
|
||||
|
||||
- Estimated ~500k lines of GOAL code
|
||||
- 10410 functions
|
||||
- 5451 functions with no control flow (no branching, loops, if/else, short-circuiting boolean operators, gotos, etc)
|
||||
|
||||
The rough timeline is to finish sometime in 2022. If it looks like this is impossible, the project will be abandoned. But I have already spent about 4 months preparing to start this and seems doable. I also have some background in compilers, and familiarity with PS2 (worked on DobieStation PS2 emulator) / MIPS in general (wrote a PS1 emulator). I think the trick will be making good automated tools - the approach taken for SM64 and other N64 decompilations is way too labor-intensive to work.
|
||||
The rough timeline is to finish sometime in 2022. If it looks like this is impossible, the project will be abandoned. But I have already spent about 4 months preparing to start this and seems doable. I also have some background in compilers, and familiarity with PS2 (worked on DobieStation PS2 emulator) / MIPS in general (wrote a PS1 emulator). I think the trick will be making good automated tools - the approach taken for SM64 and other N64 decompilations is way too labor-intensive to work.
|
||||
|
||||
### GOAL Decompiler
|
||||
|
||||
GOAL Decompiler
|
||||
------------------
|
||||
The decompiler is in progress, at
|
||||
The decompiler is in progress, at
|
||||
https://github.com/water111/jak-disassembler
|
||||
|
||||
Here is the plan for writing the decompiler:
|
||||
|
||||
- [x] Decode the CGO/DGO format.
|
||||
- [x] Decode the linking data format.
|
||||
- [x] Identify all code and disassemble
|
||||
@@ -147,10 +215,10 @@ Here is the plan for writing the decompiler:
|
||||
- [ ] Variable map and scoping
|
||||
- [ ] S-expression construction (expression stack)
|
||||
|
||||
### GOAL Runtime
|
||||
|
||||
The "runtime" will be a replacement for all of the C/C++ code of the original game. There is C/C++ code that runs on the main processor (EE) and the separate I/O processor (IOP).
|
||||
|
||||
GOAL Runtime
|
||||
--------------
|
||||
The "runtime" will be a replacement for all of the C/C++ code of the original game. There is C/C++ code that runs on the main processor (EE) and the separate I/O processor (IOP).
|
||||
- The "C Kernel", which runs on the EE and contains
|
||||
- [ ] File I/O (for debugging, not used by retail game)
|
||||
- [x] Initialization to boostrap the GOAL Kernel and start the game engine
|
||||
@@ -180,17 +248,18 @@ The "Sony libraries" are a simple wrapper around my `system` library, which impl
|
||||
|
||||
Likely there will be sound/graphics code in here at some point, but this part is not fully planned yet.
|
||||
|
||||
GOAL Compiler
|
||||
---------------
|
||||
### GOAL Compiler
|
||||
|
||||
The GOAL compiler will target x86-64. At first just Linux. There is a macro language called GOOS which is basically just Scheme but with a few bonus features.
|
||||
|
||||
The compiler will reuse a significant amount of code from my existing LISP compiler for x86-64. I have a very bad WIP combination which is capable of building a modified `gkernel.gc` for x86 as a proof of concept. It can create and run functions in threads.
|
||||
The compiler will reuse a significant amount of code from my existing LISP compiler for x86-64. I have a very bad WIP combination which is capable of building a modified `gkernel.gc` for x86 as a proof of concept. It can create and run functions in threads.
|
||||
|
||||
An important part of the compiler is the test suite. Without tests the compiler will be full of bugs. So every feature should have a good set of tests.
|
||||
An important part of the compiler is the test suite. Without tests the compiler will be full of bugs. So every feature should have a good set of tests.
|
||||
|
||||
The major components are
|
||||
|
||||
- [ ] GOAL-IR, a typed linear intermediate representation for GOAL code
|
||||
|
||||
- [ ] "Environment"
|
||||
- [ ] "Ref"
|
||||
- [ ] Constant propagation of integers/floats
|
||||
@@ -198,6 +267,7 @@ The major components are
|
||||
- [ ] Ref `in_gpr`
|
||||
|
||||
- [ ] The type system
|
||||
|
||||
- [ ] Type inheritance
|
||||
- [ ] Integer/float/pointer types (value semantics)
|
||||
- [ ] Reference types
|
||||
@@ -216,41 +286,41 @@ The major components are
|
||||
- [ ] Built-in types in the GOAL runtime/C Kernel
|
||||
|
||||
- [x] The GOOS Macro Language
|
||||
|
||||
- [x] S-expression parser (the "Reader")
|
||||
- [x] Reader text db (for error messages that point to a specific line)
|
||||
- [x] Scheme interpreter
|
||||
|
||||
|
||||
- [ ] Front-end (convert s-expressions (a tree structure) to GOAL-IR (a linear representation))
|
||||
- [ ] Parsing helpers
|
||||
- [ ] Macro expansion
|
||||
- [ ] Control flow/block forms
|
||||
- [ ] Type definitions
|
||||
- [ ] Inline assembly forms
|
||||
- [ ] Function/method call
|
||||
- [ ] Math forms
|
||||
- [ ] Lexical scoping (immediate application of `lambda`)
|
||||
- [ ] Function inlining (slightly different scoping rules of immediate `lambda`)
|
||||
- [ ] Function/macro definition
|
||||
- [ ] Static Objects
|
||||
- [ ] Parsing helpers
|
||||
- [ ] Macro expansion
|
||||
- [ ] Control flow/block forms
|
||||
- [ ] Type definitions
|
||||
- [ ] Inline assembly forms
|
||||
- [ ] Function/method call
|
||||
- [ ] Math forms
|
||||
- [ ] Lexical scoping (immediate application of `lambda`)
|
||||
- [ ] Function inlining (slightly different scoping rules of immediate `lambda`)
|
||||
- [ ] Function/macro definition
|
||||
- [ ] Static Objects
|
||||
|
||||
- [ ] Regsiter allocation
|
||||
- [ ] Analysis
|
||||
- [ ] Allocation
|
||||
- [ ] Stack spilling
|
||||
- [ ] `xmm` and `gpr` promotion/demotions for EE 128-bit register usage
|
||||
|
||||
- [ ] Analysis
|
||||
- [ ] Allocation
|
||||
- [ ] Stack spilling
|
||||
- [ ] `xmm` and `gpr` promotion/demotions for EE 128-bit register usage
|
||||
|
||||
- [ ] Codegen / Emitter (convert GOAL-IR + register allocations to x86 object file format)
|
||||
- [ ] Emitter (convert GOAL-IR to instructions)
|
||||
- [ ] x86-64 instruction generation (actually generate the machine code)
|
||||
- [ ] Linking data
|
||||
- [ ] 64-bit GPR
|
||||
- [ ] 32-bit float
|
||||
- [ ] 128-bit GPR
|
||||
- [ ] 32-bit float x4 vector register
|
||||
- [ ] function prologue/epilogue
|
||||
- [ ] stack spilling
|
||||
- [ ] static object and static object links
|
||||
- [ ] Emitter (convert GOAL-IR to instructions)
|
||||
- [ ] x86-64 instruction generation (actually generate the machine code)
|
||||
- [ ] Linking data
|
||||
- [ ] 64-bit GPR
|
||||
- [ ] 32-bit float
|
||||
- [ ] 128-bit GPR
|
||||
- [ ] 32-bit float x4 vector register
|
||||
- [ ] function prologue/epilogue
|
||||
- [ ] stack spilling
|
||||
- [ ] static object and static object links
|
||||
|
||||
- [ ] Listener/REPL
|
||||
- [ ] Network connection
|
||||
@@ -260,14 +330,14 @@ The major components are
|
||||
- [ ] Expand single macro debugging feature
|
||||
- [ ] Interface for running gtests
|
||||
|
||||
### Asset Extraction Tool
|
||||
|
||||
Asset Extraction Tool
|
||||
-----------------------
|
||||
Not started yet. The simplest version of this tool is just to use the decompiler logic to turn the level/art-group/texture/TXT files into GOAL source code, and copy all STR/sound/visibility files, as these don't need to be modified.
|
||||
Not started yet. The simplest version of this tool is just to use the decompiler logic to turn the level/art-group/texture/TXT files into GOAL source code, and copy all STR/sound/visibility files, as these don't need to be modified.
|
||||
|
||||
Eventually this should export to a more useful format.
|
||||
|
||||
File formats:
|
||||
|
||||
- [ ] Art group (a GOAL object format)
|
||||
- There may be more formats related to art groups.
|
||||
- [ ] Texture page (a GOAL object format)
|
||||
@@ -282,12 +352,8 @@ File formats:
|
||||
- [ ] Loading screen image
|
||||
- [ ] save game icon (I do not care about this)
|
||||
|
||||
### Asset Packing Tool
|
||||
|
||||
Asset Packing Tool
|
||||
-----------------------
|
||||
Packs together all assets/compiled code/runtime into a format that can be played. The simplest version to go with the simplest extraction tool will just pass the level/art-group/texture/TXT files to the compiler, and copy STR/sound/visbility files into the fakeiso. Then pack in CGOs/DGOs.
|
||||
Packs together all assets/compiled code/runtime into a format that can be played. The simplest version to go with the simplest extraction tool will just pass the level/art-group/texture/TXT files to the compiler, and copy STR/sound/visbility files into the fakeiso. Then pack in CGOs/DGOs.
|
||||
|
||||
It's important that the asset extraction/packing can be automated so we can avoid distributing the assets, which are large and probably not supposed to be distributed.
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Reference in New Issue
Block a user