- Adds `capstone` as a disassembling library that could eventually
replace Zydis (for now left that alone)
- Replicates all of the existing x86 tests to ARM64, fixed a bunch of
underlying issues along the way
- There are a very small handful of tests remaining that need to be
enabled / fixed
This PR adds various fixes to the GLTF exporter:
- Merc and TIE envmaps now properly extract
- Changed joint offset in `convert_per_vertex_data` since we now export
the `align` bone
- The bone accessor for merc models now uses
`TINYGLTF_COMPONENT_TYPE_UNSIGNED_BYTE`, fixing a decompiler crash when
attempting to reimport an extracted model into the game without first
importing it into Blender and re-exporting
- All model buffers now get merged into one large buffer before export,
saving some space and increasing parsing speed of GLB files
- Tfrags, TIEs and shrubs are now no longer stored in one giant mesh per
tree and get grouped into categories:
- Tfrags will be grouped by the tfrag tree name they come from (normal,
trans, dirt, ice, etc.) and get separated by the texture their material
uses (tfrags are not instanced, so this is the next best solution)
- TIEs and shrubs get grouped by tree index and separated by their proto
name (TIE wind will also be grouped separately)
<img width="1630" height="1716" alt="image"
src="https://github.com/user-attachments/assets/2ce31795-a445-4a0d-982d-413fd51833b2"
/>
<img width="497" height="291" alt="image"
src="https://github.com/user-attachments/assets/a8b44073-a33e-4c40-a5d7-2c7a328a86b4"
/>
<img width="484" height="750" alt="image"
src="https://github.com/user-attachments/assets/15392b94-60f3-4fee-97c5-cc6a50d298d5"
/>
<img width="525" height="684" alt="image"
src="https://github.com/user-attachments/assets/e3814105-7538-4028-9129-63b21d432528"
/>
<img width="1221" height="1298" alt="image"
src="https://github.com/user-attachments/assets/7aefa662-890a-4be8-9540-3e63fff52f28"
/>
This adds a debug server to `goalc` that sends JSON over the socket to
communicate with an external debugger using the Debug Adapter Protocol.
This lets us debug GOAL code in a proper debugger with breakpoints, step
over, step in and step out per line, stack frames and supports watches
for global symbols, registers and local variables (local variables only
work within the most recent stack frame). Special registers (`r13`,
`r14`, `r15`, argument registers, etc.) are tracked separately and the
current process register even displays the type of the current `pp` if
possible.
Watches that track addresses holding a reference type generate a list of
field names according to the object's type. All fields will show their
name, type and value and, depending on the type, will try to infer extra
info like symbol names/values, function names for `function` fields,
enum values and more. This also works nested, so any field that is also
a reference type can also be accessed and display its fields, etc.
Dynamic arrays are also supported where possible, e.g. in
`inline-array-class` children and boxed arrays, it will figure out the
value of the `length` field and access the memory up to that point so
all the elements can be accessed and viewed from the `data` field.
Our VS Code extension implements the DAP in
open-goal/opengoal-vscode#375.
Using it is as simple as connecting a REPL to a running game instance
with `(lt)`, compiling with `(mi)` and, with the extension installed,
pressing F5 in VS Code to start the debugger. By default, it will try to
connect to the game that the active `.gc` file is from, the socket port
is different per game (8128 for Jak 1, 8129 for Jak 2, 8130 for Jak 3).
The `launch.json` was updated with two entries for this, the second
entry lets you pick the game/port manually if desired.
~~Not tested on Windows.~~ Only supports x86 for now.
Fixes another long-standing subtitle translation bug.
There is a `merge` flag on each subtitle line, the idea being that we
would merge in the text from the base game if this was set. However in
jak 2 and beyond:
- The subtitles for english (base) are not predefined for the cutscenes,
only the hints. The rest are in the game files and we do not
simultaneously load both languages when the game is running so there is
no trivial fallback path.
- However, every subtitle language by default inherits the base games
metadata, where all `merge` flags are set to `true`
This means that all custom cutscene translations were being ignored
unless this flag was set to false in the metadata file for each line.
This is what finnish does since it changes the number of lines and their
timings, and is why it works fine.
This flag can probably be completely removed, as per the above, it
serves no actual purpose. All it does in our code is if it's set, we
skip the line when writing out the subtitle file.
But for now i simply just iterate through and flip these flags
automatically if the language overrode the cutscene. Tested on
english/finnish/pt-BR
<img width="648" height="467" alt="image"
src="https://github.com/user-attachments/assets/292e3dfe-df66-4d23-942a-b41f2b157bf8"
/>
basically initial commit to add support for polish base game text
needs work before you could push to master
im a little rusty so testing along side me if everything is working
would be appreciated
<img width="1919" height="1107" alt="jak2 pl menu"
src="https://github.com/user-attachments/assets/862dcbe0-1c6a-4b9e-9565-7aa91b0a9115"
/>
Namely all I did was include the normals from MercVertex in with the glb
exporter. I also converted parts of MercVertex to use union structs for
faster casting. The union structs should just work on most compilers
from what I researched, if not I can revert that and go a different
route for that.
Co-authored-by: Hat Kid <6624576+Hat-Kid@users.noreply.github.com>
Less than 100 instructions left to implement, with the vast vast
majority being load-and-stores. These will likely be knocked out quickly
but they require a more involved implementation than just simply
translating the instructions (several need multiple instructions, others
may need reserved registers (x16 or x17 are common for this purpose))
This is a good milestone to get something pushed to master.
This adds the blerc data from merc models to the GLB export as shape
keys, also including the data from animations. Blender is unfortunately
a bit weird about this and only really lets you change what animated
weights are used via the NLA track editor by selecting both the
corresponding armature animation and then the one for the shape keys.
Additionally, due to the way the Blender GLB import works when you have
animated weights on a model + animations as actions for an armature, any
models that have blerc data get an extra useless empty, but this seems
to be harmless.
This PR also fixes materials that use texture animations so they now
have their base texture in the export, rather than using the default
material. Materials now also get named after their base texture for
easier recognition.
This PR does the following:
- Designs a mechanism by which arm64 instructions can be encoded and
emitted
- Dispatch our higher-level instruction emitting calls to either x86 or
arm64 instructions depending on what the compiler is set to (defaults to
x86)
- Bare minimum scaffolding to get the arm64 instructions successfully
executing atleast on apple silicon
- Implement enough instructions to get the codetester test suite passing
on arm
This attempts to get into master whatever work was done in this PR /
it's earlier PR https://github.com/open-goal/jak-project/pull/3965
I don't want this work to be lost / floating around in massive PRs.
However the changes are:
- switch to ntsc_v1 instead of PAL as the development target, as we have
done for all other games
- remove most of the copied-from-jak2/3 changes as they need to be
confirmed during the decompilation process not just assumed
- avoids committing any changes to `game/kernel/common` as it was not
clear to me if these were changes made in jak x's kernel that were not
properly broken out into it's own functions. We don't want to
accidentally introduce bugs into jak1-3's kernel code.
- in other words, if the change in the kernel only happens in jak x...it
should likely be specific to jak x's kernel, not common.
---------
Co-authored-by: VodBox <dillon@vodbox.io>
Co-authored-by: yodah <greenboyyodah@gmail.com>
Massive overhaul of Finnish translations
I guess it's time to finally push this out
I'll go insane if I have to proofread one more time :P
Every time I just realize my translations sucked and I keep tweaking
everything
---------
Co-authored-by: Tyler Wilding <xtvaser@gmail.com>
During the iso creation process it's possible that this file in will be
internally renamed to `WATER_AN.CGO`, despite the external file name
displaying `WATER-AN.CGO`. We expect the file to be named
`WATER-AN.CGO`.
We can't control how our users create their iso files. We can alleviate
future friction regarding this matter by adopting this simple change.
---------
Co-authored-by: Tyler Wilding <xtvaser@gmail.com>
Resolves#3075
TODO before merge:
- [x] Properly draw non-korean strings while in korean mode (language
selection)
- [x] Check jak 3
- [x] Translation scaffolding (allow korean characters, add to Crowdin,
fix japanese locale, etc)
- [x] Check translation of text lines
- [x] Check translation of subtitle lines
- [x] Cleanup PR / some performance optimization (it's take a bit too
long to build the text and it shouldn't since the information is in a
giant lookup table)
- [x] Wait until release is cut
I confirmed the font textures are identical between Jak 2 and Jak 3, so
thank god for that.
Some examples of converting the korean encoding to utf-8. These show off
all scenarios, pure korean / korean with ascii and japanese / korean
with replacements (flags):
<img width="316" height="611" alt="Screenshot 2025-07-26 191511"
src="https://github.com/user-attachments/assets/614383ba-8049-4bf4-937e-24ad3e605d41"
/>
<img width="254" height="220" alt="Screenshot 2025-07-26 191529"
src="https://github.com/user-attachments/assets/1f6e5a6c-8527-4f98-a988-925ec66e437d"
/>
And it working in game. `Input Options` is a custom not-yet-translated
string. It now shows up properly instead of a disgusting block of
glyphs, and all the original strings are hopefully the same
semantically!:
<img width="550" height="493" alt="Screenshot 2025-07-26 202838"
src="https://github.com/user-attachments/assets/9ebdf6c0-f5a3-4a30-84a1-e5840809a1a2"
/>
Quite the challenge. The crux of the problem is -- Naughty Dog came up
with their own encoding for representing korean syllable blocks, and
that source information is lost so it has to be reverse engineered.
Instead of trying to figure out their encoding from the text -- I went
at it from the angle of just "how do i draw every single korean
character using their glyph set".
One might think this is way too time consuming but it's important to
remember:
- Korean letters are designed to be composable from a relatively small
number of glyphs (more on this later)
- Someone at naughty dog did basically this exact process
- There is no other way! While there are loose patterns, there isn't an
overarching rhyme or reason, they just picked the right glyph for the
writing context (more on this later). And there are even situations
where there IS NO good looking glyph, or the one ND chose looks awful
and unreadable (we could technically fix this by adjusting the
positioning of the glyphs but....no more)!
Information on their encoding that gets passed to `convert-korean-text`:
- It's a raw stream of bytes
- It can contain normal font letters
- Every syllable block begins with: `0x04 <num_glyphs> <...the glyph
bytes...>`
- DO NOT confuse `num_glyphs` with num jamo, because some glyphs can
have multiple jamo!
- Every section of normal text starts with `0x03`. For example a space
would be `0x03 0x20`
- There are a very select few number of jamo glyphs on a secondary
texture page, these glyph bytes are preceeded with a `0x05`. These jamo
are a variant of some of the final vowels, moving them as low down as
possible.
Crash course on korean writing:
- Nice resource as this is basically what we are doing -
https://glyphsapp.com/learn/creating-a-hangeul-font
- Korean syllable blocks have either 2 or 3 jamo. Jamo are basically
letters and are the individual pieces that make up the syllable blocks.
- The jamo are split up into "initial", "medial" and "final" categories.
Within the "medial" category there are obvious visual variants:
- Horizontal
- Vertical
- Combination (horizontal + a vertical)
- These jamo are laid out in 6 main pre-defined "orientations":
- initial + vertical medial
- initial + horizontal medial
- initial + combination
- initial + vertical medial + final
- initial + horizontal medial + final
- initial + combination + final
- Sometimes, for stylistic reasons, jamo will be written in different
ways (ie. if there is nothing below a vertical vowel will be extended).
- Annoying, and ND's glyph set supports this stylistic choice!
- There are some combination of jamo that are never used, and some that
are only used for a single word in the entire language!
With all that in mind, my basic process was:
- Scan the game's entire corpus of korean text, that includes subtitles.
It's very easy to look at the font texture's glyphs and assign them to
their respective jamo
- This let me construct a mapping and see which glyphs were used under
which context
- I then shoved this information into a 2-D matrix in excel, and created
an in-game tool to check every single jamo permutation to fill in the
gaps / change them if naughty dogs was bad. Most of the time, ND's
encoding was fine.
-
https://docs.google.com/spreadsheets/d/e/2PACX-1vTtyMeb5-mL5rXseS9YllVj32BGCISOGZFic6nkRV5Er5aLZ9CLq1Hj_rTY7pRCn-wrQDH1rvTqUHwB/pubhtml?gid=886895534&single=true
anything in red is an addition / modification on my part.
- This was the most lengthy part but not as long as you may think, you
can do a lot of pruning. For example if you are checking a 3-jamo
variant (the ones with the most permutations) and you've verified that
the medial jamo is as far up vertically as it can be, and you are using
the lowest final jamo that are available -- there is nothing to check or
improve -- for better or worse! So those end up being the permutations
between the initial and medial instead of a three-way permutation
nightmare.
- Also, while it is a 2d matrix, there's a lot of pruning even within
that. For example, for the first 3 orientations, you dont have to care
about final vowels at all.
- At the end, I'm left with a lookup table that I can use the encode the
best looking korean syllable blocks possible given the context of the
jamo combination.
Compilation with the NO_ASSERT flag set results in errors.
This happens because some the macros are not defined in the else branch
of #ifndef NO_ASSERT.
Fixes#3909
Long standing issue in Jak 2's subtitles, it's actually kinda incredible
that finnish didn't run into this problem. Later in this code it
accesses the subtitle based on the language-id, and finnish is always
out of bounds -- but it never caused a catastrophic issue.
Polish on the otherhand is much more out of bounds and always causes an
access violation / crash.
Jak 2's subtitle code has no fallback (ie. use english if not
translated) for cutscenes, since they are not defined in the base file
like in Jak 1. This means that an untranslated language will render no
subtitles during cutscenes. Translations are welcome!.
Tested a cutscene in:
- english
- italian (base game with subtitles)
- finnish (new language, actually is translated)
- polish (new language, not translated)
Custom levels for Jak 2/3 now support envmapped TIE geometry. The TIE
extract was also changed to ignore materials that have the specular flag
set, but are missing a roughness texture.
Jak 2/3 now also support the `build-actor` tool.
The `build-custom-level` and `build-actor` macros now have a few new
options:
- Both now have a `force-run` option (`#f` by default) that, when set to
`#t`, will always run level/art group generation even if the output
files are up to date.
- `build-custom-level` has a `gen-fr3` option (`#t` by default) that,
when set to `#f`, will skip generating the FR3 file for the custom level
and only generate the GOAL level file to skip the potentially slow
process of finding and adding art groups and textures. Useful for when
you want to temporarily edit only the GOAL side of the level (such as
entity placement, etc.).
- `build-actor` has a `texture-bucket` option (default 0) which will
determine what DMA sink group the model will be placed in, which is
useful to determine the draw order of the model. Previously, this was
omitted, resulting in shadows not drawing over custom actors because the
actors were put in a bucket that is drawn after shadows (this behavior
can be restored with `:texture-bucket #f`).
Two new flags were added to the Blender plugin to allow reusing the mod
and/or eye draws of the original model that is being replaced. Works
pretty well for eyes, but the blerc draws can cause some Z-fighting with
the non-moving parts of the model.
Also a small refactor to the merc replacement code to de-duplicate some
code by moving stuff to `gltf_util.cpp`.
When a material in Blender has its IOR level changed to anything other
than the default value of 0.5, the `KHR_materials_specular` extension is
applied during the glTF export, which is what we use to check for
envmaps in custom models. If an envmap is undesired, but the IOR value
was accidentally changed, the program would assert during model
processing if there is no metallic roughness texture attached to the
material.
Since this is an easy mistake to make and is hard to spot, this adds a
better error message for such cases.
Fix compiler warnings, and a bug where the `snd_handle` of
`SoundBankInfo` was never set, leading to sound banks never unloading.
The game relies on unloading soundbanks to make sure certain sounds
don't play, like the blue gun 1 fire noise when using blue gun 2.
---------
Co-authored-by: water111 <awaterford1111445@gmail.com>
DMA sizes are 16-bits, not 12-bits. This would cause rare crashes when
the ocean far renderer uses more than this.
I'm not really sure what the ocean far rendering is doing at this time,
but it happens as stuff is loading in.
There's a chance this fixes crashes in jak 2 as well, since we used to
see errors that would be explained by this.
Co-authored-by: water111 <awaterford1111445@gmail.com>
- Bug fix to KD tree splitting, should fix cases with bad vertex
colors/alphas.
- Normalize normals instead of asserting if they are the wrong length.
**the fact that blender exports normals incorrectly is a bug and I doubt
their implementation is correct if you've scaled things on only on
axis.**
- Automatically resize metallic texture (envmap strength) if it doesn't
match the size of the rgb texture instead of asserting
Co-authored-by: water111 <awaterford1111445@gmail.com>
I hope this is everything I needed, and nothing I didn't.
## What's Changed
This update adds a command-line parameter to goalc, `--iso-path`.
Providing a path to a directory like
`D:\Files\Repositories\ArchipelaGOAL\iso_data\jak1` will inform the
compiler to use that directory instead.
## Why is this useful?
When combined with `--proj-path`, the compiler can be pointed to a
completely different project folder, given the `(mi)` command, and
immediately begin building from that directory, with everything it
needs. This eliminates the need to copy `iso_data` to multiple `data`
directories.
If a subsequent change to the Launcher is made, each mod could be passed
an --iso-path pointing to a single shared folder, allowing mods to each
run their own REPL _without_ requiring a copy of `iso_data` in a
subfolder.
## Independent testing required!
My local repositories are a little suspect, with a mod, a fork of
mod-base, and a fork of jak-project, all on the same drive. My
decompiler_out and iso_data folders are in the mod repo, not mod-base
nor jak-project. So what I did was make the change in the mod-base fork,
point `--proj-path and --iso-path` to the mod folders, and then ran
`(mi)`. The output showed a build starting with no errors.
Then I had to create this PR, which my fork of mod-base is unable to do,
so I created a patch file, forked jak-project, then applied the patch
there.
All this is to say that it would be preferable if someone could apply
this code to their own installation and see if it works. Even I wouldn't
take my own word for this.
---------
Co-authored-by: Tyler Wilding <xtvaser@gmail.com>
This only applies to the background for now:
- support for alpha for vertex colors in custom levels
- switch time of day palette generation from octree to k-d tree
- support for alpha masking in custom levels
- support for transparent textures
- support for envmap in custom levels
---------
Co-authored-by: water111 <awaterford1111445@gmail.com>
Tyler Wilding