Overhaul the build system (#234)

* wip

* fix

* add disassembler

* Disasm builds OK

* Variable addends

* More wip

* Rodata migration implemented

* Cleanup old tools

* Try fix submodule -> subrepo merge

* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "602e609"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "602e609"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Builds again but assets are totally broken

* git subrepo pull --force tools/asm-processor

subrepo:
  subdir:   "tools/asm-processor"
  merged:   "1ffdb08a"
upstream:
  origin:   "https://github.com/simonlindholm/asm-processor.git"
  branch:   "master"
  commit:   "1ffdb08a"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables

* rm z64compress in preparation for subrepo

* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "eb11085c"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "eb11085c"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Fix asset extraction

* Fix diff-init make rule

* Split code bss

* Split assumed linker bug padding from assembly files

* add filelists for mm.us.rev1

* Maybe working, but I'm not sure

* add overlays to spec

* Add rodata to actos

* Everything compiles

* Make a lot of C files for code

* Add almost every file in code to spec

* whoops

* 3 code files left

* add scenes to spec

* More progress on progress.py

* Fix skelanime in spec

* audio files!

* Fix merge issues

* Fix some C files in code

* Fix remaining code files

* Use existing O1 C files in spec

* reorder boot order in spec

* update spec

* fault.c

* Convert relocs on completed actors, fixbaserom uses current rom name

* more boot files

* Add VT macros and script

* finish already existing boot files

* most of  libultra

* fix 64bits libultra files

* Use C files for libultra, wrap some functions in NON_MATCHING

* Remove duplicate of OS_CLOCK_RATE from fault.c

* C files for fbdemos

* delete dumb files

* bootstrap C files, still need to add them to the spec

* update fixbaserom

* boot OK?

* I forgot to commit the spec

* C for gamestates

* C for kaleido

* Change all includes to ""

* copy actor sizes script from oot

* I forgot to delete those files

* Basic C files for effects

* Add effects initvars names

* Remove mislabelled boot functions from header/txt

* Begin porting bootstrap_fx, some sizes

* Fix <>

* Fix enum

* Fix diff.py

* fix libultra stuff

* update regconvert

* update setup warnings

* add some missing ;

* Fix some makefile stuff and other fixes on some non_matching functions

* add executable flag in extract_baserom and fixbaserom

* fix relative path

* copy assist from oot

* fix map path

* another assist path fix

* Delete C files for handwritten files

* add code_801A51F0 to spec

* add gfxbuffers to spec

* Move rodata to top of each file when possible

* UNK_TYPEs for func_801A51F0

* Remove kaleido rodata from spec

* Update spec and undefined_syms for recent merge

* GCC warnings and fix errors in nonmatchings,

* round percentage numbers

* progress script: format changes

* progress: error on non-existing files

* fix warning in z_scene_table

* Match 2 nonmatchings in z_actor

* Warnings in lightswitch and invadepoh

* Fix warning in z_actor_dlftbls

* I though I fixed this one

* whoops

* Comment out CC_CHECK

* Removed redundant ultra64.h includes

* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms

* Completed gamestates bootstrap

* Split kaleido_scope

* Remove section.h and segment.h, move keep object externs to a common location in variables.h

* Completed effects bootstrap

* Segmented address externs for effects, fbdemos, gamestates and kaleido

* Move actor data externs out of the if 0

* Segmented address externs for actors

* Prepare actionfunc detection

* fix script, how did it even work before

* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler

* Automated actionFunc detection in actors

* Segmented addresses from player .text

* rm old segment addrs script and fix build

* Move sizes folder to tools

* Make build.py executable

* New Jenkinsfile Prayge

* Remove numpy dependencies

* Add warnings_disasm_current.txt

* my bad

* Update spec and undefined_syms

* Add z_eff_ss_hahen to pametfrog

* git subrepo pull (merge) --force tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "163ca2af"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "163ca2af"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Make z64compress print to stdout

* sneeky commit to update warnings tooling

* test

* Another test

* Mark fixing overlay reloc generating as a TODO

* Update warnings stuff

* Communicate the return code from running z64compress back to the Makefile through the wrapper

* Run formatter, remove extra commented copy of function

* Re-fix some includes

* Convert atan to hex to conform to decided style

* Some tidying up, remove c for fp and the other two handwritten code files

* BSS in z_collision_check & z_scene_proc

* add static back in

* Fix timerintr bss, add file to spec, some cleanup

* Remove externs

* Newline

* Readd enums

* Typo

* Colours

* Comments for hitmark enum values

Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

* Improvements and suggestions

* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py

Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

src/boot_O2/__osMalloc.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 #define FILL_ALLOCBLOCK (1 << 0)
 #define FILL_FREEBLOCK (1 << 1)
@@ -16,6 +15,8 @@
 
 extern OSMesg sArenaLockMsg[1];
 
+#pragma GLOBAL_ASM("asm/non_matchings/boot/__osMalloc/D_80099110.s")
+
 void ArenaImpl_LockInit(Arena* arena) {
     osCreateMesgQueue(&arena->lock, sArenaLockMsg, ARRAY_COUNT(sArenaLockMsg));
 }
@@ -28,7 +29,7 @@ void ArenaImpl_Unlock(Arena* arena) {
     osRecvMesg(&arena->lock, NULL, OS_MESG_BLOCK);
 }
 
-ArenaNode* heap_get_tail(Arena* arena) {
+ArenaNode* ArenaImpl_GetLastBlock(Arena* arena) {
     ArenaNode* last;
     ArenaNode* iter;
 
@@ -69,7 +70,7 @@ void __osMallocAddBlock(Arena* arena, void* start, s32 size) {
             firstNode->isFree = 1;
             firstNode->magic = NODE_MAGIC;
             ArenaImpl_Lock(arena);
-            lastNode = heap_get_tail(arena);
+            lastNode = ArenaImpl_GetLastBlock(arena);
             if (lastNode == NULL) {
                 arena->head = firstNode;
                 arena->start = start;
@@ -141,7 +142,7 @@ void* __osMallocR(Arena* arena, u32 size) {
 
     size = ALIGN16(size);
     ArenaImpl_Lock(arena);
-    iter = heap_get_tail(arena);
+    iter = ArenaImpl_GetLastBlock(arena);
 
     while (iter != NULL) {
         if (iter->isFree && iter->size >= size) {
@@ -216,7 +217,7 @@ end:
     ArenaImpl_Unlock(arena);
 }
 
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/__osMalloc/__osRealloc.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/__osMalloc/__osRealloc.s")
 
 void __osAnalyzeArena(Arena* arena, u32* outMaxFree, u32* outFree, u32* outAlloc) {
     ArenaNode* iter;
@@ -244,4 +245,4 @@ void __osAnalyzeArena(Arena* arena, u32* outMaxFree, u32* outFree, u32* outAlloc
     ArenaImpl_Unlock(arena);
 }
 
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/__osMalloc/__osCheckArena.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/__osMalloc/__osCheckArena.s")

src/boot_O2/assert.c

@@ -1,7 +1,6 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
-void assert_fail(const char* file, u32 lineNum) {
+void __assert(const char* file, u32 lineNum) {
     osGetThreadId(NULL);
     Fault_AddHungupAndCrash(file, lineNum);
 }

src/boot_O2/boot_0x800862E0.c

@@ -1,20 +0,0 @@
-#include <ultra64.h>
-#include <global.h>
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/StartHeap_AllocMin1.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/StartHeap_FreeNull.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_8008633C.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_800863AC.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_8008641C.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_800864EC.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_80086588.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/StartHeap_Init.asm")
-
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/boot_0x800862E0/func_80086620.asm")

src/boot_O2/boot_80084940.c

@@ -0,0 +1,15 @@
+#include "global.h"
+
+void func_80084940(void) {
+    u32 spStatus = __osSpGetStatus();
+    u32 dpStatus = osDpGetStatus();
+
+    if (spStatus && dpStatus) {} // stubbed debug prints
+}
+
+void func_80084968(void) {
+    func_80084940();
+    osDpSetStatus(0x28);
+    __osSpSetStatus(0x4082);
+    func_80084940();
+}

src/boot_O2/boot_800862E0.c

@@ -0,0 +1,17 @@
+#include "global.h"
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/StartHeap_AllocMin1.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/StartHeap_FreeNull.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/func_8008633C.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/func_800863AC.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/func_8008641C.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/func_800864EC.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/func_80086588.s")
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/boot_800862E0/StartHeap_Init.s")

src/boot_O2/gfxprint.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 extern u16 sGfxPrintFontTLUT[64];
 extern u16 sGfxPrintUnkTLUT[16];
@@ -109,10 +108,10 @@ void GfxPrint_PrintCharImpl(GfxPrint* this, u8 c) {
     this->posX += 32;
 }
 #else
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/gfxprint/GfxPrint_PrintCharImpl.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/gfxprint/GfxPrint_PrintCharImpl.s")
 #endif
 
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/gfxprint/GfxPrint_PrintChar.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/gfxprint/GfxPrint_PrintChar.s")
 
 void GfxPrint_PrintStringWithSize(GfxPrint* this, const void* buffer, size_t charSize, size_t charCount) {
     const char* str = (const char*)buffer;
@@ -173,7 +172,7 @@ Gfx* GfxPrint_Close(GfxPrint* this) {
 }
 
 void GfxPrint_VPrintf(GfxPrint* this, const char* fmt, va_list args) {
-    func_80087900(&this->callback, fmt, args);
+    PrintUtils_VPrintf(&this->callback, fmt, args);
 }
 
 void GfxPrint_Printf(GfxPrint* this, const char* fmt, ...) {

src/boot_O2/loadfragment2.c

@@ -1,12 +1,11 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 UNK_TYPE4 D_80096C30 = 2;
 
 #ifdef NON_MATCHING
 // This needs lots of work. Mostly regalloc and getting the address of D_80096C30 placed in s5 at the beginning of the
 // function
-void Load2_Relocate(u32 allocatedVRamAddr, OverlayBlockSizes* overlayInfo, u32 vRamStart) {
+void Load2_Relocate(u32 allocatedVRamAddr, OverlayRelocationSection* overlayInfo, u32 vRamStart) {
     s32 sectionLocations[4];
     u32* regReferences[32];
     u32 regValues[32];
@@ -22,7 +21,7 @@ void Load2_Relocate(u32 allocatedVRamAddr, OverlayBlockSizes* overlayInfo, u32 v
     sectionLocations[1] = allocatedVRamAddr;
     sectionLocations[2] = overlayInfo->textSize + allocatedVRamAddr;
     sectionLocations[3] = sectionLocations[2] + overlayInfo->dataSize;
-    for (i = 0, relocationIndex = 0; i < overlayInfo->amountOfRelocations; relocationIndex++) {
+    for (i = 0, relocationIndex = 0; i < overlayInfo->nRelocations; relocationIndex++) {
         relocation = overlayInfo->relocations[relocationIndex];
         i++;
         inst = (u32*)(sectionLocations[relocation >> 0x1e] + (relocation & 0xffffff));
@@ -59,7 +58,7 @@ void Load2_Relocate(u32 allocatedVRamAddr, OverlayBlockSizes* overlayInfo, u32 v
     }
 }
 #else
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/loadfragment2/Load2_Relocate.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/loadfragment2/Load2_Relocate.s")
 #endif
 
 #ifdef NON_MATCHING
@@ -69,7 +68,7 @@ s32 Load2_LoadOverlay(u32 vRomStart, u32 vRomEnd, u32 vRamStart, u32 vRamEnd, u3
     u32 pad;
     u32 size;
     void* end;
-    OverlayBlockSizes* overlayInfo;
+    OverlayRelocationSection* overlayInfo;
 
     size = vRomEnd - vRomStart;
 
@@ -80,7 +79,7 @@ s32 Load2_LoadOverlay(u32 vRomStart, u32 vRomEnd, u32 vRamStart, u32 vRamEnd, u3
     DmaMgr_SendRequest0(allocatedVRamAddr, vRomStart, size);
 
     end = (void*)(allocatedVRamAddr + size);
-    overlayInfo = (OverlayBlockSizes*)((int)end - *(int*)((int)end + -4));
+    overlayInfo = (OverlayRelocationSection*)((int)end - *(int*)((int)end + -4));
 
     if (1) {
         ;
@@ -99,7 +98,7 @@ s32 Load2_LoadOverlay(u32 vRomStart, u32 vRomEnd, u32 vRamStart, u32 vRamEnd, u3
     return nbytes;
 }
 #else
-#pragma GLOBAL_ASM("./asm/non_matchings/boot/loadfragment2/Load2_LoadOverlay.asm")
+#pragma GLOBAL_ASM("asm/non_matchings/boot/loadfragment2/Load2_LoadOverlay.s")
 #endif
 
 void* Load2_AllocateAndLoad(u32 vRomStart, u32 vRomEnd, u32 vRamStart, u32 vRamEnd) {

src/boot_O2/mtxuty-cvt.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 void MtxConv_F2L(MatrixInternal* m1, MtxF* m2) {
     s32 i;

src/boot_O2/padsetup.c

@@ -0,0 +1,3 @@
+#include "global.h"
+
+#pragma GLOBAL_ASM("asm/non_matchings/boot/padsetup/func_80086620.s")

src/boot_O2/padutils.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 void PadUtils_Init(Input* input) {
     bzero(input, sizeof(Input));

src/boot_O2/rand.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 #define RAND_MULTIPLIER 1664525
 #define RAND_INCREMENT 1013904223

src/boot_O2/stackcheck.c

@@ -1,5 +1,4 @@
-#include <ultra64.h>
-#include <global.h>
+#include "global.h"
 
 StackEntry* sStackInfoListStart = NULL;
 StackEntry* sStackInfoListEnd = NULL;