This PR does the following:
- Designs a mechanism by which arm64 instructions can be encoded and
emitted
- Dispatch our higher-level instruction emitting calls to either x86 or
arm64 instructions depending on what the compiler is set to (defaults to
x86)
- Bare minimum scaffolding to get the arm64 instructions successfully
executing atleast on apple silicon
- Implement enough instructions to get the codetester test suite passing
on arm
This attempts to get into master whatever work was done in this PR /
it's earlier PR https://github.com/open-goal/jak-project/pull/3965
I don't want this work to be lost / floating around in massive PRs.
However the changes are:
- switch to ntsc_v1 instead of PAL as the development target, as we have
done for all other games
- remove most of the copied-from-jak2/3 changes as they need to be
confirmed during the decompilation process not just assumed
- avoids committing any changes to `game/kernel/common` as it was not
clear to me if these were changes made in jak x's kernel that were not
properly broken out into it's own functions. We don't want to
accidentally introduce bugs into jak1-3's kernel code.
- in other words, if the change in the kernel only happens in jak x...it
should likely be specific to jak x's kernel, not common.
---------
Co-authored-by: VodBox <dillon@vodbox.io>
Co-authored-by: yodah <greenboyyodah@gmail.com>
This sets up the C Kernel for Jak 3, and makes it possible to build and
load code built with `goalc --jak3`.
There's not too much interesting here, other than they switched to a
system where symbol IDs (unique numbers less than 2^14) are generated at
compile time, and those get included in the object file itself.
This is kind of annoying, since it means all tools that produce a GOAL
object file need to work together to assign unique symbol IDs. And since
the symbol IDs can't conflict, and are only a number between 0 and 2^14,
you can't just hash and hope for no collisions.
We work around this by ignoring the IDs and re-assigning our own. I
think this is very similar to what the C Kernel did on early builds of
Jak 3 which supported loading old format level files, which didn't have
the IDs included.
As far as I can tell, this shouldn't cause any problems. It defeats all
of their fancy tricks to save memory by not storing the symbol string,
but we don't care.
Trying to make up for some of the startup speed lost in the SDL
transition. This saves about 1s from start (from ~3s), and about 500 MB
of RAM.
- Faster TIE unpack by merging matrix groups, more efficient vertex
transforms, and skipping normal transforms on groups with no normals.
- Refactor generic merc and merc to use a single renderer with multiple
interfaces, rather than many renderers. Removed "LightningRenderer" as a
special thing, but Warp is still special
- Add more profiling stuff to startup and the loader.
- Remove `SDL_INIT_HAPTIC` - this turned out to be needed for
force-feedback steering wheels, and not needed for controller vibration
- Switched `vag-player` to use quicksort instead of the default GOAL
sort (very slow)
Definitely needs a clean up pass, but I think the functionality is very
close.
There's a few "hacks" still:
- I am using the emerc logic for environment mapping, which doesn't care
about the length of the normals. I can't figure out how the normal
scaling worked in etie. I want to do a little bit more experimentation
with this before merging.
- There is some part about adgifs for TIE and ETIE that I don't
understand. The clearly correct behavior of TIE/ETIE is that the alpha
settings from the adgif shader are overwritten by the settings from the
renderer. But I can't figure out how this happens in all cases.
- Fade out is completely disabled. I think this is fine because the
performance difference isn't bad. But if you are comparing screenshots
with PCSX2, it will make things look a tiny bit different.
My mistake -- testing focused too much on preserving the existing
behaviour I clearly forgot to make sure the new stuff worked properly.
Just had to early out and not modify the args if they were in the new
format.
An attempt to cleanup the last CLI interface we have left to cleanup.
- `gk` args now follow the typical convention ie. `--proj-path` instead
of `-proj-path`.
- args that are passed through to the rest of the application / the
game's runtime use the typical convention of following a `--`
- I'm thinking some args shouldn't be handled at this level ie
(`-nodisplay`, `-vm`, `-novm` or `-jak2`) These could be better
documented as legitimate flags and passed in via a nice struct. They
don't seem to be used in `InitParams` but I'll triple check.
There's a temporary shim here so there is no coupled release with the
launcher (right now it executes `gk` with a few args). So I just change
the old args into the new format. After one release cycle, I can change
it in the launcher and delete it here.
I am unsure if this will break the bash shellscript usages -- not sure
which args were usually passed into `$@`
- make sure bsp is processed on `l` levels in extraction (caused missing
remaps)
- clean up a few prints in extraction
- handle the <15 byte differences in art group files automatically (no
more errors about file naming)
- fix potential exception thrown by merc2 in a few ways: fixed bad data
in FR3's, check texture index just in case, and handle exceptions a
little bit better (still a crash, but at least you get a print)
- fix mips2 ocean stuff causing ocean far crashes
Notable things:
- This assert is hit when trying to save the pc-settings file, NYI
https://github.com/open-goal/jak-project/blob/e630b506903d7f7028dff89702be7f586c2120e7/game/kernel/common/Symbol4.h#L14
so right now settings aren't persisted. But RPC defaults to on
- The existing functions can probably be made generic based off the game
version, but I didn't spend time refactoring them yet as they aren't
really ready to be used in jak 2 yet (we have no screenshots for the
levels for example)
Running reference tests/decompiler should now be possible on macos
(arm). Most of the changes were just cleaning up places where we were
sloppy with ifdefs, but there were two interesting ones:
- `Printer.cpp` was updated to not use a recursive function for printing
lists, to avoid stack overflow
- I replaced xxhash with another version of the same library that
supports arm (the one that comes in zstd). The interface is C instead of
C++ but it's not bad to use. I confirmed that the extractor succeeds on
jak 1 iso so it looks like this gives us the same results as the old
library.
This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular
The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it
https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
* Use sleepthread in RPC loop
* Keep a pointer to current IOP thread
* Implement IOP thread scheduling based on priority
And implement DelayThread as an actual delay.
* Run IOP flat out
* Use information from scheduler in wait_run_iop
* Lock sif mutex in set_rpc_queue
* always use kernel dispatch with wait_run
* Loop in dispatch until no thread is ready
* Use timestamp for next wakeup
instead of duration
* Wrap IOP thread entrypoints for safety
Libco threads are not supposed to return from their entrypoint
* Use a queue for IOP thread wakeups from EE thread
* IopThreadRecord -> IopThread
* add libco
* Use cooperative threading for IOP threads
* Ugly solution for overlord start
Needs to run in a thread
* Clean out thread shutdown logic
* Update comments
* [cleanup] remove merc1, fix memory bugs, reduce memory usage
* change compiler log to see actual info from build_level
* save file
* editing text is hard
* fix IOP getting stuck on music load
* fix regression? and clang
* fix a decomp
* fix another regression
* another
* fix "all actors"
* another regression!
* goalc: cleanup goalc's main method and add nrepl listener socket
* deps: add standalone ASIO for sockets
* lint: formatting
* common: make a common interface for creating a server socket
* goalc: setup new repl server
* deps: remove asio
* goalc: debug issues, nrepl is working again
* git: rename files
* attempt to fix linux function call
* test
* scripts: make the error message even more obvious....
* goalc: make suggested changes, still can't reconnect properly
* game: pull out single-client logic from XSocketServer
* nrepl: supports multiple clients and disconnection/reconnects
* goalc: some minor fixes for tests
* goalc: save repl history when the compiler reloads
* common: add include for linux networking
* a few small changes to fix tests
* is it the assert?
* change thread start order and add a print to an assert
Co-authored-by: water <awaterford111445@gmail.com>
* some changes to make the `anim-tester` kinda work
* types for `lava` (who cares)
* fix divide by zero macros
* cleanup filenames
* fix tests + minor bug
* also add a new sprite renderer
* claaaang
* goal build fix
* fix tests, add stack singleton option
* make all event-message-blocks the same
* diskboot
* temp
* temp
* wip
* more progress on the instance asm
* first half of tie extraction, up to dma lists
* more tie extraction
* first part figured out maybe
* bp1 loop seems to work, bp2 loop does not
* bp1 and bp2 appear working. sadly ip is needed
* ip1 outline, not working ip2
* just kidding, ip2 seems to work
* extraction seems to work
* basic rendering working
* tie fixes
* performance optimization of tie renderer
* hook up tie to engine
* fix more bugs
* cleanup and perf improvements
* fix tests
* ref tests
* mm256i for gcc
* CLANG
* windows
* more compile fixes
* fix fast time of day
* small fixes
* fix after merge
* clang
* wip, taking a break to work on asm stuff first
* the goal code for sparticle
* mips2c the first sparticle asm function
* temp
* particle processing no longer crashing
* temp
* working texture cache for vi1 and hud textures
* sprites
* cleanup 1
* temp
* temp
* add zstd library
* temp
* working
* tests
* include fix
* uncomment
* better decomp of sparticle stuff, part 1
* update references
* revamp gfx and display systems a bit
* Use some fancy c++ pointers instead of just raw pointers
* Tidy some things up.
* clang
* clang 2
* fixes
* fixesss
* error detection when making display
* use `std::function` for this because that's what you're meant to do
* [memdump] Analyze fields of type `(array basic)`
* fix test
* Massively simplify things.
* update VS launch target params
* remove redundant VS launch option
* Add a debug PS2 VM to the runtime, currently only for the DMAC
* Formatting
* remove broken assert
* Avoid weird buffer overflow bug
* Test on `VIF0_DMA_BANK`!
* Add a docstring
* patch pointers for the other dma channels
* patch DMAC pointer
* remove dead leftover code
* Change default return value for `get_vm_ptr`
* Implement runtime display (test)
* Update runtime.cpp
* [game display] add "-nodisplay" argument
* style fixes
* Update gfx.cpp
* [deci2server] fix deadlock when killing a Deci2Server
* add libxrandr to linux github test
* correct package name to libxrandr-dev
* set g_main_thread_id in exec_runtime
* add libxinerama to linux test packages
* correct the name
* add libxcursor1 package
* Update linux-workflow.yaml
* add libxi-dev
* fix constructor for g_main_thread_id
* fix submodules + use -nodisplay during tests
* move the gfx loop to its own function and use a lambda for exit conditions
* fix include
* fix include
* fix includes (for real this time)
Tyler Wilding
