diff --git a/README.md b/README.md index 4c50465411..39e8df598f 100644 --- a/README.md +++ b/README.md @@ -53,16 +53,16 @@ We support both Linux and Windows on x86-64. We have a Discord server where we discuss development. https://discord.gg/V82sTJGEAs ## Current Status -So far, we've decompiled around 341,233 lines of GOAL code, out of an estimated 500,000 total lines and we've started work on an OpenGL renderer. Currently, the main display process (`*dproc*`) runs and sends data to our renderer. We can load textures, text files, and level files. Using keyboard controls, we can open the debug menu and turn on some simple debug visualizations. +So far, we've decompiled around 400,000 lines of GOAL code, out of an estimated 500,000 total lines. We have a working OpenGL renderer which renders most of the game world and foreground. Levels are fully playable, and you can finish the game with 100% completion! There is currently *no* audio. Here are some screenshots of the renderer: -![](./docs/markdown/imgs/screenshot_hut.png) -![](./docs/markdown/imgs/screenshot_vi1.png) +![](./docs/markdown/imgs/screenshot_hut_new_small.png) +![](./docs/markdown/imgs/screenshot_jungle1_small.png) YouTube playlist: https://www.youtube.com/playlist?list=PLWx9T30aAT50cLnCTY1SAbt2TtWQzKfXX -To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where the original code had type casts until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have around 300,000 lines (380 files) that pass. +To help with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where we believe the original code had type casts (or where they feel appropriate) until the decompiler succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in `goal_src`. Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is tested as part of our unit tests, and so far we have over 300,000 lines (460 files) that pass. We don't save any assets from the game - you must bring your own copy of the game and use the decompiler to extract assets. @@ -146,9 +146,7 @@ nix-build -A packages.x86_64-linux.jak-asan # package with Clang ASan build ## Getting Started - Windows -Install Visual Studio 2022 and get the `Desktop development with C++` workload during the installation process. - -> if you already have visual studio and don't have this installed - open your `Visual Studio Installer` and modify the installation +If you do not have it already, install Visual Studio 2019 or 2022 and get the `Desktop development with C++` workload during the installation process. If you already have Visual Studio installed and don't have this, open your `Visual Studio Installer` and modify the installation. On Windows, it's recommended to get Scoop to use as a package manager, making the follow steps _much_ easier. Follow the steps on the bottom of the homepage here https://scoop.sh/ @@ -168,8 +166,9 @@ Open the project as a CMake project. ![](./docs/markdown/imgs/windows/open-project.png) -Then build the entire project +Then build the entire project as "Release". You can also press Ctrl+Shift+B as a hotkey for Build All. +![](./docs/markdown/imgs/windows/release-build.png) ![](./docs/markdown/imgs/windows/build-all.png) ## Building and Running the Game @@ -186,13 +185,13 @@ Running the decompiler on the entire game is slow and not needed, so it is recom "decompile_code": false, // change this to false, don't decompile code ``` -Place a copy of the game's files in `iso_data` (on Windows, make a `jak1` subfolder and place the files there), then run the decompiler with the `scripts/shell/decomp.sh` (Linux) or `scripts/batch/decomp-jak1.bat` (Windows) script. +Place a copy of the game's files in `iso_data/jak1/`, then run the decompiler with the `scripts/shell/decomp.sh` (Linux) or `scripts/batch/decomp-jak1.bat` (Windows) script. ### Build Game -Run the OpenGOAL compiler `build/goalc/goalc`, or use one of the `gc` scripts. Enter `(mi)` to build the `"iso"` target, which contains everything we have placed in the build system so far. +Run the OpenGOAL compiler `out/build/goalc/goalc` (keep in mind that the build directories may vary by system, on Windows the equivalent is similar to `out/build/Release/bin/goalc`), or use one of the `gc` scripts. Enter `(mi)` to build the `"iso"` target, which contains everything we have placed in the build system so far. ### Run Game -In a separate terminal, start the runtime with `build/game/gk -fakeiso -debug`. Then, in the OpenGOAL window, run `(mi)` to create the data for the game and give the REPL information for running code, `(lt)` to connect, `(lg)` to load the game engine and `(test-play)` to start the game engine. If it all works right, it will look something like this: +In a separate terminal, start the runtime with `out/build/game/gk -boot -fakeiso -debug`, or instead run the `gk-display.bat` script (Windows). The game should boot up automatically! If you want to connect the REPL to the live game and use it for inspecting code or changing things around, rhen, in the OpenGOAL compiler window (REPL), run `(mi)` to create the data for the game and give the REPL information for running code and `(lt)` to connect. If you want to start the runtime but not boot up the game, remove the `-boot` argument or run the regular `gk.bat` script, and after connecting to the runtime from the REPL, run `(lg)` to load the game engine and `(test-play)` to start it. Assuming it all works right, it will look something like this: ``` g > (lt) [Listener] Socket connected established! (took 0 tries). Waiting for version... @@ -208,16 +207,20 @@ gc> (test-play) gc> ``` -Then, in the graphics window, you can use the period key to bring up the debug menu. Controllers also work, using the same mapping as the original game. +In the graphics window, you can use the period key to bring up the debug menu. Controllers also work, using the same mapping as the original game. -Check out the `pc_debug` and `examples` folder under `goal_src` for some examples of GOAL code we wrote. They have instructions for how to run them. +Check out the `pc_debug`, `examples` and `engine/pc/` folders under `goal_src` for some examples of GOAL code we wrote. The debug files have instructions for how to run them if they are not loaded automatically by the engine. ## Project Layout There are four main components to the project. The first is `goalc`, which is a GOAL compiler for x86-64. Our implementation of GOAL is called OpenGOAL. All of the compiler source code is in `goalc`. To run the compiler on Linux, there is a script `gc.sh`. On Windows, there is a `gc.bat` scripts and a `gc-no-lt.bat` script, the latter of which will not attempt to automatically attach to a running target. The compiler is controlled through a prompt which can be used to enter commands to compile, connect to a running GOAL program for interaction, run the OpenGOAL debugger, or, if you are connected to a running GOAL program, can be used as a REPL to run code interactively. In addition to compiling code files, the compiler has features to pack and build data files. -The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD into the `iso_data` folder. Then run `decomp.sh` (Linux) to run the decompiler. For Windows, it is the `decomp-jak1.bat` file, and it wants your game's DVD files in a `jak1` folder inside `iso_data`. The decompile will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler. +The second component to the project is the decompiler. You must have a copy of the PS2 game and place all files from the DVD inside a folder corresponding to the game within `iso_data` folder (`jak1` for Jak 1 Black Label, etc.), as seen in this picture: + +![](./docs/markdown/imgs/iso_data-help.png) + +Then run `decomp.sh` (Linux) or `decomp-jak1.bat` (Windows) to run the decompiler. The decompiler will extract assets to the `assets` folder. These assets will be used by the compiler when building the port, and you may want to turn asset extraction off after running it once. The decompiler will output code and other data intended to be inspected by humans in the `decompiler_out` folder. Stuff in this folder will not be used by the compiler. The third is the game source code, written in OpenGOAL. This is located in `goal_src`. All GOAL and GOOS code should be in this folder. Right now most of this is placeholders or incomplete, but you can take a look at `kernel/gcommon.gc` or `goal-lib.gc` to see some in-progress source code. diff --git a/decompiler/config.cpp b/decompiler/config.cpp index 7c6ff9c74c..01bf6cdedc 100644 --- a/decompiler/config.cpp +++ b/decompiler/config.cpp @@ -29,6 +29,7 @@ Config read_config_file(const std::string& path_to_config_file) { config.game_version = cfg.at("game_version").get(); config.text_version = cfg.at("text_version").get(); + config.game_name = cfg.at("game_name").get(); auto inputs_json = read_json_file_from_config(cfg, "inputs_file"); config.dgo_names = inputs_json.at("dgo_names").get>(); @@ -211,7 +212,8 @@ Config read_config_file(const std::string& path_to_config_file) { config.merged_objects.insert(x); } - config.levels_to_extract = cfg.at("levels_to_extract").get>(); + config.levels_to_extract = inputs_json.at("levels_to_extract").get>(); + config.levels_extract = cfg.at("levels_extract").get(); return config; } diff --git a/decompiler/config.h b/decompiler/config.h index c2d6b8bf36..c17012de70 100644 --- a/decompiler/config.h +++ b/decompiler/config.h @@ -111,7 +111,7 @@ struct Config { bool generate_symbol_definition_map = false; bool is_pal = false; - + std::string game_name; GameTextVersion text_version = GameTextVersion::JAK1_V1; std::unordered_set allowed_objects; @@ -133,6 +133,7 @@ struct Config { std::unordered_map bad_format_strings; std::vector levels_to_extract; + bool levels_extract; DecompileHacks hacks; }; diff --git a/decompiler/config/jak1_ntsc_black_label.jsonc b/decompiler/config/jak1_ntsc_black_label.jsonc index 6c599c4b63..9d69a27120 100644 --- a/decompiler/config/jak1_ntsc_black_label.jsonc +++ b/decompiler/config/jak1_ntsc_black_label.jsonc @@ -1,6 +1,7 @@ { "game_version": 1, "text_version": 10, + "game_name": "jak1", // if you want to filter to only some object names. // it will make the decompiler much faster. @@ -84,33 +85,8 @@ // LEVEL EXTRACTION //////////////////////////// + // turn this on to extract level background graphics data + "levels_extract": true, // turn this on if you want extracted levels to be saved out as .obj files - "levels_convert_to_obj": false, - // list of level DGOs to extract background graphics for - "levels_to_extract":[ - "BEA.DGO", - "CIT.DGO", - "DAR.DGO", - "DEM.DGO", - "FIN.DGO", - "INT.DGO", - "JUB.DGO", - "JUN.DGO", - "FIC.DGO", - "LAV.DGO", - "MAI.DGO", - "MIS.DGO", - "OGR.DGO", - "ROB.DGO", - "ROL.DGO", - "SNO.DGO", - "SUB.DGO", - "SUN.DGO", - "SWA.DGO", - "TIT.DGO", - "TRA.DGO", - "VI1.DGO", - "VI2.DGO", - "VI3.DGO" - ] + "levels_convert_to_obj": false } diff --git a/decompiler/config/jak1_ntsc_black_label/inputs.jsonc b/decompiler/config/jak1_ntsc_black_label/inputs.jsonc index 5132abcca8..a42b8980da 100644 --- a/decompiler/config/jak1_ntsc_black_label/inputs.jsonc +++ b/decompiler/config/jak1_ntsc_black_label/inputs.jsonc @@ -262,5 +262,32 @@ //"audio_dir_file_name": "jak1/VAG", "audio_dir_file_name": "", - "streamed_audio_file_names": ["VAGWAD.ENG", "VAGWAD.JAP"] + "streamed_audio_file_names": ["VAGWAD.ENG", "VAGWAD.JAP"], + + "levels_to_extract":[ + "BEA.DGO", + "CIT.DGO", + "DAR.DGO", + "DEM.DGO", + "FIN.DGO", + "INT.DGO", + "JUB.DGO", + "JUN.DGO", + "FIC.DGO", + "LAV.DGO", + "MAI.DGO", + "MIS.DGO", + "OGR.DGO", + "ROB.DGO", + "ROL.DGO", + "SNO.DGO", + "SUB.DGO", + "SUN.DGO", + "SWA.DGO", + "TIT.DGO", + "TRA.DGO", + "VI1.DGO", + "VI2.DGO", + "VI3.DGO" + ] } diff --git a/decompiler/main.cpp b/decompiler/main.cpp index 988ad9682d..2e585fe3db 100644 --- a/decompiler/main.cpp +++ b/decompiler/main.cpp @@ -40,8 +40,8 @@ int main(int argc, char** argv) { return 1; } - std::string in_folder = argv[2]; - std::string out_folder = argv[3]; + std::string in_folder = file_util::combine_path(argv[2], config.game_name); + std::string out_folder = file_util::combine_path(argv[3], config.game_name); std::vector dgos, objs, strs; for (const auto& dgo_name : config.dgo_names) { @@ -149,8 +149,10 @@ int main(int argc, char** argv) { } } - for (auto& lev : config.levels_to_extract) { - extract_from_level(db, tex_db, lev, config.hacks, config.rip_levels); + if (config.levels_extract) { + for (auto& lev : config.levels_to_extract) { + extract_from_level(db, tex_db, lev, config.hacks, config.rip_levels); + } } fmt::print("[Mem] After extraction: {} MB\n", get_peak_rss() / (1024 * 1024)); diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/bones-see-light_2022-01-28.png b/docs/gh-pages-proj/src/assets/gallery/jak1/bones-see-light_2022-01-28.png new file mode 100644 index 0000000000..04207b0979 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/bones-see-light_2022-01-28.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/entity-debugging_2022-01-31.png b/docs/gh-pages-proj/src/assets/gallery/jak1/entity-debugging_2022-01-31.png new file mode 100644 index 0000000000..bfb650eb79 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/entity-debugging_2022-01-31.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/jak-and-daxter-are-stunned_2022-02-04.png b/docs/gh-pages-proj/src/assets/gallery/jak1/jak-and-daxter-are-stunned_2022-02-04.png new file mode 100644 index 0000000000..6f54bca9f3 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/jak-and-daxter-are-stunned_2022-02-04.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/light-cave_2022-02-02.png b/docs/gh-pages-proj/src/assets/gallery/jak1/light-cave_2022-02-02.png new file mode 100644 index 0000000000..314ed2e7c4 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/light-cave_2022-02-02.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/lurker-chilling_2022-02-04.png b/docs/gh-pages-proj/src/assets/gallery/jak1/lurker-chilling_2022-02-04.png new file mode 100644 index 0000000000..8e7f49e408 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/lurker-chilling_2022-02-04.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/robotboss-appears_2022-02-04.png b/docs/gh-pages-proj/src/assets/gallery/jak1/robotboss-appears_2022-02-04.png new file mode 100644 index 0000000000..3a032276aa Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/robotboss-appears_2022-02-04.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/starry-sky_2022-01-11.png b/docs/gh-pages-proj/src/assets/gallery/jak1/starry-sky_2022-01-11.png new file mode 100644 index 0000000000..2dd9ec14f6 Binary files /dev/null and b/docs/gh-pages-proj/src/assets/gallery/jak1/starry-sky_2022-01-11.png differ diff --git a/docs/gh-pages-proj/src/assets/gallery/jak1/videos.json b/docs/gh-pages-proj/src/assets/gallery/jak1/videos.json index 87ab0d2eeb..8f32480e24 100644 --- a/docs/gh-pages-proj/src/assets/gallery/jak1/videos.json +++ b/docs/gh-pages-proj/src/assets/gallery/jak1/videos.json @@ -1,4 +1,8 @@ [ + { + "link": "https://www.youtube.com/embed/ZO7A22btJc0", + "timestamp": "2022-02-04" + }, { "link": "https://www.youtube.com/watch?v=6dJhyVqqq5Q", "timestamp": "2022-01-15" diff --git a/docs/markdown/imgs/iso_data-help.png b/docs/markdown/imgs/iso_data-help.png new file mode 100644 index 0000000000..bace9d6ad1 Binary files /dev/null and b/docs/markdown/imgs/iso_data-help.png differ diff --git a/docs/markdown/imgs/screenshot_hut_new_small.png b/docs/markdown/imgs/screenshot_hut_new_small.png new file mode 100644 index 0000000000..bcf79184fe Binary files /dev/null and b/docs/markdown/imgs/screenshot_hut_new_small.png differ diff --git a/docs/markdown/imgs/screenshot_jungle1_small.png b/docs/markdown/imgs/screenshot_jungle1_small.png new file mode 100644 index 0000000000..ccfa520cf0 Binary files /dev/null and b/docs/markdown/imgs/screenshot_jungle1_small.png differ diff --git a/docs/markdown/imgs/windows/release-build.png b/docs/markdown/imgs/windows/release-build.png new file mode 100644 index 0000000000..c6d2aaf3c3 Binary files /dev/null and b/docs/markdown/imgs/windows/release-build.png differ diff --git a/goal_src/game.gp b/goal_src/game.gp index 9630bc1668..c7c0e37a9c 100644 --- a/goal_src/game.gp +++ b/goal_src/game.gp @@ -103,10 +103,12 @@ (fmt #f "tpage-{}.go" id) ) + +(define *game-directory* (get-environment-variable "OPENGOAL_DECOMP_DIR" :default "jak1/")) + (defmacro copy-texture (tpage-id) "Copy a texture from the game, using the given tpage ID" - (let* ((folder (get-environment-variable "OPENGOAL_DECOMP_DIR" :default "")) - (path (string-append "decompiler_out/" folder "raw_obj/" (tpage-name tpage-id)))) + (let* ((path (string-append "decompiler_out/" *game-directory* "raw_obj/" (tpage-name tpage-id)))) `(defstep :in ,path :tool 'copy :out '(,(string-append "out/obj/" (tpage-name tpage-id)))))) @@ -118,8 +120,7 @@ ) (defmacro copy-go (name) - (let* ((folder (get-environment-variable "OPENGOAL_DECOMP_DIR" :default "")) - (path (string-append "decompiler_out/" folder "raw_obj/" name ".go"))) + (let* ((path (string-append "decompiler_out/" *game-directory* "raw_obj/" name ".go"))) `(defstep :in ,path :tool 'copy :out '(,(string-append "out/obj/" name ".go"))))) @@ -135,8 +136,7 @@ `(begin ,@(apply (lambda (x) `(copy-str ,x)) strs))) (defun copy-str (name) - (let* ((folder (get-environment-variable "OPENGOAL_DECOMP_DIR" :default "")) - (path (string-append "iso_data/" folder "STR/" name ".STR")) + (let* ((path (string-append "iso_data/" *game-directory* "STR/" name ".STR")) (out-file (string-append "out/iso/" name ".STR"))) (defstep :in path :tool 'copy @@ -148,8 +148,7 @@ `(begin ,@(apply (lambda (x) `(copy-vis-file ,x)) files))) (defun copy-vis-file (name) - (let* ((folder (get-environment-variable "OPENGOAL_DECOMP_DIR" :default "")) - (path (string-append "iso_data/" folder "VIS/" name ".VIS")) + (let* ((path (string-append "iso_data/" *game-directory* "VIS/" name ".VIS")) (out-name (string-append "out/iso/" name ".VIS"))) (defstep :in path :tool 'copy @@ -219,22 +218,22 @@ ) ;; the TWEAKVAL file -(defstep :in "iso_data/MUS/TWEAKVAL.MUS" +(defstep :in (string-append "iso_data/" *game-directory* "MUS/TWEAKVAL.MUS") :tool 'copy :out '("out/iso/TWEAKVAL.MUS")) ;; the VAGDIR file -(defstep :in "iso_data/VAG/VAGDIR.AYB" +(defstep :in (string-append "iso_data/" *game-directory* "VAG/VAGDIR.AYB") :tool 'copy :out '("out/iso/VAGDIR.AYB")) ;; the save icon file -(defstep :in "iso_data/DRIVERS/SAVEGAME.ICO" +(defstep :in (string-append "iso_data/" *game-directory* "DRIVERS/SAVEGAME.ICO") :tool 'copy :out '("out/iso/SAVEGAME.ICO")) ;; the loading screen file -(defstep :in "iso_data/DRIVERS/SCREEN1.USA" +(defstep :in (string-append "iso_data/" *game-directory* "DRIVERS/SCREEN1.USA") :tool 'copy :out '("out/iso/SCREEN1.USA")) diff --git a/scripts/batch/decomp-jak1.bat b/scripts/batch/decomp-jak1.bat index fdddbdeb04..9a4828663b 100644 --- a/scripts/batch/decomp-jak1.bat +++ b/scripts/batch/decomp-jak1.bat @@ -1,4 +1,4 @@ @echo off cd ..\.. -out\build\Release\bin\decompiler decompiler\config\jak1_ntsc_black_label.jsonc iso_data\jak1 decompiler_out\jak1 +out\build\Release\bin\decompiler decompiler\config\jak1_ntsc_black_label.jsonc iso_data decompiler_out\ pause