diff --git a/docs/api-docs.html b/docs/api-docs.html index dfb65fe9b5..02dd537388 100644 --- a/docs/api-docs.html +++ b/docs/api-docs.html @@ -33,6 +33,9 @@ .search a { color: #c5ccd4 !important; } + .sidebar li > p { + color: #ffe301; + } @@ -43,10 +46,11 @@ repo: "https://github.com/water111/jak-project", basePath: "./markdown/", loadSidebar: true, - subMaxLevel: 2, + subMaxLevel: 3, logo: "./markdown/imgs/logo-text-colored.png", + notFoundPage: true, + auto2top: true, search: "auto", // default - // complete configuration parameters search: { maxAge: 86400000, // Expiration time, the default one day @@ -57,10 +61,38 @@ depth: 6, hideOtherSidebarContent: false, // whether or not to hide other sidebar content }, + plugins: [ + function(hook, vm) { + hook.beforeEach(function(html) { + if (/githubusercontent\.com/.test(vm.route.file)) { + url = vm.route.file + .replace('raw.githubusercontent.com', 'github.com') + .replace(/\/master/, '/blob/master'); + } else { + url = + 'https://github.com/water111/jak-project/blob/master/docs' + + vm.route.file; + } + var editHtml = '[:memo: Edit Document](' + url + ')\n'; + return ( + editHtml + + html + ); + }) + }, + ], }; + + + + + + + + diff --git a/docs/markdown/README.md b/docs/markdown/README.md index 340091393c..26d2b995c0 100644 --- a/docs/markdown/README.md +++ b/docs/markdown/README.md @@ -1,258 +1,29 @@ -# OpenGOAL Document -This is the main documentation for the OpenGOAL language. It's designed to be read in order to learn OpenGOAL. It does not explain the OpenGOAL kernel or state system. +# An Overview -The syntax description uses these rules: +This is the main documentation for the OpenGOAL language. It's designed to be read in order to learn OpenGOAL. **It does not explain the OpenGOAL kernel or state system.** + +The syntax descriptions uses these rules: - Something `[in-brackets]` is optional and can be left out. -- Something like `[:type type-name]` means there is an optional named argument. It can be used like `:type type-name`, replacing `type-name` with what you want, or left out entirely. +- Something like `[:type type-name]` means there is an optional named argument. + - It can be used like `:type type-name`, replacing `type-name` with what you want, or left out entirely. - When there are multiple choices, they are separated by `|`. Example: `#t|#f` is either `#t` or `#f`. -- A `...` means more of the thing before can be included. Example `(f arg...)` can have multiple arguments. +- `...` means more of the thing before can be included. Example `(f arg...)` can have multiple arguments. -# All Forms -Documented forms are crossed out. -- ~~asm-file~~ -- ~~m~~ -- ~~ml~~ -- ~~md~~ -- ~~build-game~~ -- ~~build-data~~ -- ~~blg~~ -- tc -- ~~e~~ -- db (debug mode) -- #when -- #unless -- ~~lt~~ -- ~~r~~ -- ~~shutdown-target~~ -- db (disassemble byte) -- dh -- dw -- dd -- df -- segfault -- fpe -- let -- let* -- ~~defun~~ -- while -- until -- dotimes -- protect -- +! -- ~~if~~ -- ~~when~~ -- ~~unless~~ -- and -- or -- +1 -- +! -- -! -- *! -- 1- -- zero? -- &+! -- &- -- &-> -- basic? -- pair? -- binteger? -- rtype-of -- cons -- list -- null? -- caar -- object-new -- expect-eq -- expect-true -- expect-false -- start-test -- finish-test -- top-level -- ~~begin~~ -- ~~block~~ -- ~~return-from~~ -- ~~label~~ -- ~~goto~~ -- gs -- :exit -- ~~asm-file~~ -- ~~asm-data-file~~ -- listen-to-target -- reset-target -- :status -- ~~in-package~~ -- ~~#cond~~ -- ~defglobalconstant~ -- ~~seval~~ -- ~~cond~~ -- ~~when-goto~~ -- ~~define~~ -- ~~define-extern~~ -- ~~set!~~ -- dbs -- dbg -- :cont -- :break -- :dump-all-mem -- :pm -- :di -- :disasm -- :bp -- :ubp -- deftype -- ~~defmethod~~ -- -> -- & -- the-as -- the -- print-type -- new -- car -- cdr -- method -- declare-type -- none -- ~~lambda~~ -- ~~declare~~ -- ~~inline~~ -- ~~quote~~ -- ~~mlet~~ -- defconstant -- ~~+~~ -- ~~-~~ -- ~~*~~ -- ~~/~~ -- ~~shlv~~ -- ~~shrv~~ -- ~~sarv~~ -- ~~mod~~ -- ~~logior~~ -- ~~logxor~~ -- ~~logand~~ -- ~~lognot~~ -- ~~=~~ -- ~~!=~~ -- ~~eq?~~ -- ~~neq?~~ -- ~~not~~ -- ~~<=~~ -- ~~>=~~ -- ~~<~~ -- ~~>~~ -- &+ -- ~~build-dgos~~ -- ~~set-config!~~ -- rlet -- .ret -- .sub -- .push -- .pop -- set-config! +## Language Basics -# Language Basics OpenGOAL is a compiled language. Source code is stored in `.gc` files. Each `.gc` file is compiled into a `.o` file. These `.o` files are then loaded by the game. When they are loaded, it has the effect of running every "top level" expression in the file. Usually these are function, type, and method declarations, but you can also use this for initialization code. For example, it is common to first define types, functions, and methods, then set up global instances. There are effectively three different "languages": -1. OpenGOAL - the normal compiled language. -2. OpenGOAL compiler commands - simple commands to run the compiler, listener, and debugger. These run in the compiler only. -3. GOOS macro language. This is used in OpenGOAL macros and runs at compile-time. These macros generate OpenGOAL compiler commands or OpenGOAL source which is then processed. These run in the compiler only. +1. **OpenGOAL** - the normal compiled language. +2. **OpenGOAL compiler commands** - simple commands to run the compiler, listener, and debugger. These run in the compiler only. +3. **GOOS** macro language. This is used in OpenGOAL macros and runs at compile-time. These macros generate OpenGOAL compiler commands or OpenGOAL source which is then processed. These run in the compiler only. The OpenGOAL language uses a LISP syntax, but on the inside is closer to C or C++. There is no protection against use-after-free or other common pointer bugs. Unlike a C/C++ compiler, the OpenGOAL compiler has a state. It remembers functions/methods/types/macros/constants/enums/etc defined in previous files. -# Compiler REPL - When you start the OpenGOAL compiler, you'll see a prompt like this: - ``` - OpenGOAL Compiler 0.2 -g > - ``` -The `g` indicates that you can input OpenGOAL compiler commands. For example: +## Type System Introduction -*** -### `(e)` - Compiler Command -Exit Compiler -```lisp -(e) -``` -Exit the compiler once the current REPL command is finished. Takes no arguments. If we are connected to a target through the listener, attempts to reset the target. -*** -### `(:exit)` - Compiler Command -Exit Compiler -```lisp -(:exit) -``` -Same as `(e)`, just requires more typing. `(e)` is actually a macro for `(:exit)`. Takes no arguments. -*** -### `(lt)` - Compiler Command -Listen to Target -```lisp -(lt ["ip address"] [port-number]) -``` -Connect the listener to a running target. The IP address defaults to `"127.0.0.1"` and the port to `8112` (`DECI2_PORT` in `listener_common.h`). These defaults are usually what you want, so you can just run `(lt)` to connect. - -Example: -```lisp -g > (lt) -[Listener] Socket connected established! (took 0 tries). Waiting for version... -Got version 0.2 OK! -[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2000000000, tid = 1296302 -gc > -``` -*** -### `r` -Reset the target. -```lisp -(r ["ip address"] [port-number]) -``` -Regardless of the current state, attempt to reset the target and reconnect. After this, the target will have nothing loaded. Like with `(lt)`, the default IP and port are probably what you want. - -Note: `r` is actually a macro. -*** -### `shutdown-target` -If the target is connected, make it exit. -```lisp -(shutdown-target) -``` -The target will print -``` -GOAL Runtime Shutdown (code 2) -``` -before it exits. -*** -### `:status` -Ping the target. -```lisp -(:status) -``` -Send a ping-like message to the target. Requires the target to be connected. If successful, prints nothing. Will time-out and display and error message if the GOAL kernel or code dispatched by the kernel is stuck in an infinite loop. Unlikely to be used often. -*** - - ## Connecting To Target Example -```lisp -;; we cannot execute OpenGOAL code unless we connect the listener -g > (+ 1 2 3) -REPL Error: Compilation generated code, but wasn't supposed to - -;; connect to the target -g > (lt) -[Listener] Socket connected established! (took 0 tries). Waiting for version... -Got version 0.2 OK! -[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2000000000, tid = 1297692 - -;; execute OpenGOAL code -gc > (+ 1 2 3) -6 - -;; quit the compiler and reset the target for next time. -gc > (e) -[Listener] Closed connection to target -``` -Once we are connected, we see that there is a `gc` prompt. This indicates that the listener has an open socket connection. Now the REPL will accept both compiler commands and OpenGOAL source code. All `(format #t ...` debugging prints (like `printf`) will show up in this REPL. Each time you run something in the REPL, the result is printed as a decimal number. If the result doesn't make sense to print as a decimal, or there is no result, you will get some random number. - -In the future there will be a fancier printer here. - -# OpenGOAL Type System OpenGOAL has a type system. Every expression and object in OpenGOAL has a type. With the exception of three special types (`none`, `_varags_`, and `_types_`), every type has a parent type, and the root of all types is `object`. Types themselves are objects in the runtime that contain some basic information and their method table. One annoying detail of OpenGOAL is that the type system has some slightly different types for variables and places in memory, and automatic conversions between them. @@ -263,1644 +34,4 @@ Some types are boxed vs. unboxed. If you have an object of boxed type, it is pos Some types are value or reference. A value type means it has value semantics, it is passed by value everywhere. A reference type is like a C/C++ pointer or reference, where there is memory allocated for the data somewhere, and the you just pass around a reference to this memory. -## Built-in Types -There are a number of built-in types. I use "abstract" type to refer to a type that is only a parent type . - -### `none` -This is a special type that represents "no information". This is the return type of a function which returns nothing, and also the return type of an expression that doesn't return anything. For example, the expression `(goto x)` does not produce a value, so its type is `none`. - -### `object` -This is the parent type of all types. This is an abstract class. In a variable, this is always `object`, and can hold any `object`. In memory, this is either `object32` or `object64`. The `object32` can hold everything except for `binteger` and 64-bit integers. This type is neither boxed nor unboxed and is neither value nor reference. - -### `structure` (child of `object`) -This is the parent type of all types with fields. This is an abstract class and a reference class. A `structure` can hold any `structure`, both in memory and in a variable. It is unboxed. - -### `basic` (child of `structure`) -This is the "boxed" version of `structure`. The first field of a basic is `type`, which contains the `type` of the object. It is boxed and a reference. A `basic` can hold any `basic`, both in memory and in a variable. - -### `symbol` (child of `basic`) -A symbol has a name and a value. The name is a string, and the value is an `object32`. Note that the value is an `object32` so you cannot store a 64-bit integer in a symbol. It is considered "bad" to store unboxed objects in symbols, though you can get away with it sometimes. - -All `symbol`s are stored in the global symbol table, which is a hash table. As a result, you cannot have multiple symbols with the same name. A name is enough to uniquely determine the symbol. To get a symbol, use the syntax `'symbol-name`. To get the value, use `symbol-name`. - -Each global variable, type, and named global function has a symbol for it which has the variable, type, or function as its value. The linker is able to perform symbol table lookups at link time and patch the code so you don't have to do a hash table lookup every time you access a global variable, function, or type. - -You can also use symbols as a efficient way to represent a enum. For example, a function may return `'error` or `'complete` as a status. The compiler is able to compare symbols for equality very efficiently (just a pointer comparison, as symbols are a reference type). - -### `type` (child of `basic`) -A `type` stores information about an OpenGOAL type, including its size, parent, and name (stored as a `symbol`). It also stores the method table. Some OpenGOAL types (children of integers, bitfield types, enums, compounds types) do not have runtime types, and instead become the parent/base type. But these types cannot have runtime type information or methods and are pretty rare. It is a reference type, is boxed, and is dynamically sized (the method table's size is not fixed). - -### `string` (child of `basic`) -A string. The string is null terminated and also stores the buffer size. This type is a reference type, is boxed, and is also dynamically sized. - -### `function` (child of `basic`) -A function. Boxed and reference. It is a reference to a function, so it's like a C/C++ function pointer type. - -### `kheap` (child of `structure`) -A simple bump-allocated heap. Doesn't store the heap memory, just metadata. Supports allocating from either the top or the bottom. This is used as the memory allocation strategy for the global, debug, and level heaps. Unboxed, reference, not dynamic. - -### `array` (child of `basic`) -A "fancy" array. It is not yet implemented in OpenGOAL. - -### `pair` (child of `object`) -A pair. It is boxed. You should not make child types of `pair`. The two objects stored by the pair are `object32`s. - -### `pointer` (child of `object`) -It is a 32-bit value type containing a pointer. Not boxed, value type. See section on compound types for more information. - -### `number` (child of `object`) -Abstract type for all numbers. Value type. 64-bits. - -### `float` (child of `number`) -4-byte, single precision floating point number. Value type. - -### `integer` (child of `number`) -Abstract class for integer numbers. Child classes are `sinteger` (signed integer), `uinteger` (unsigned integer), and `binteger` (boxed-integer, always signed). These are all 64-bit types. - -Children of `sinteger` and `uinteger` are `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`. These are the size you expect, value types, and not boxed. These only exist as memory types. In a variable, there is only `int` and `uint`. These are both 64-bit types. All integer operations (math, logical, shifts) are 64-bit. - -The `binteger` is a boxed integer. It is a 61 bit signed integer (the other three bits are lost due to the number being boxed). There may actually be a `buinteger` (or `ubinteger`) but it doesn't exist in OpenGOAL at this point. - -### Weird Built-in types that aren't supported yet. -- `vu-function` -- `link-block` -- `connectable` -- `file-stream` -- `inline-array` (class) - -## Compound Types -A compound type is a type like "a pointer to an int64" or "a function which takes int as an argument and returns a string". These exist only at compile time, and get simplified at runtime. For example, all pointers become `pointer` and all functions become `function`. (The one exception to this seems to be `inline-array-class` stuff, but this is not yet supported in OpenGOAL). - -### Pointer -Pointers work like you would expect. They can only point to memory types - you can't have a `(pointer int)`, instead you must have a `(pointer int32)` (for example). Note that a `(pointer basic)` is like a C++ `basic**` as `basic` is already like a C++ pointer to struct. You can nest these, like `(pointer (pointer int64))`. If you want a pointer with no type, (like C++ `void*`) just use a plain `pointer`. The `(pointer none)` type is invalid. - -Like in C/C++, you can use array indexing with a pointer. One thing to note is that a `(pointer basic)` (or pointer to any reference type) is like a C++ "array of pointers to structs". To get the C++ "array of structs", you need an `inline-array`. - -### Inline Array -These are only valid for reference types. They refer to an array of the actual data (like C array of structs) rather than an array of reference (like C array of pointers to structs, or GOAL `(pointer structure)`). At runtime, `inline-array` becomes pointer. - -For an inline array of basics, elements are 16-byte aligned. For `structure`s that aren't `basic`, the alignment is usually the minimum alignment of all members of the structure, but there is an option to make it 16-byte aligned if needed. - -For information about how to create these arrays, see `deftype` (fields in a type) and `new` (create just an array) sections. - -### Function -Function compound types look like this `(function arg0-type arg1-type... return-type)`. There can be no arguments. The `return-type` must always be specified, and should be `none` if there is no return value. The argument types themselves can be compound types. In order to call a function, you must have a compound function type - a `function` by itself cannot be called. - - -## Field Definitions -GOAL field definitions look like this: - -`(name type-name [optional stuff])` - -where optional stuff can include these, in any order: - -- `:inline #t` (default is false), to mark field as inline. This can only be done for a reference type, and indicates that the data should be stored inline, in the type, rather than just storing a reference to data stored elsewhere. -- `:dynamic #t` (default is false), to mark field as dynamically-sized array (must be the last field in the type) -- a number, to give an array size. -- `:offset x` where x is a number, to manually specify where the field is located - -There are many combinations of reference/value, dynamic/not-dynamic, inline/not-inline, array-size/no-array-size, and it can be confusing. This list explains all that are valid. - -- Value type, no modifiers: a single value is stored in the field. The field type is the value type. -- Value type, `:dynamic #t`: the field marks the beginning of an array (of unknown size). Field type is `(pointer your-type)` -- Value type, with array size: the field marks the beginning of an array (of known size). Field type is `(pointer your-type)` -- Value type, with `:inline #t`: invalid in all cases. -- Reference type, no modifiers: a single reference is stored in the type. Type of field is `your-type` (a C++ pointer). -- Reference type, `:inline #t`: a single object is stored inside the type. Type of field is `your-type` still (a C++ pointer). The access logic is different to make this work. -- Reference type, `:dynamic #t` or array size: the field marks the beginning of an **array of references**. Field type is `(pointer your-type)`. Like C array of pointers. -- Reference type, `:inline #t` and (`:dynamic #t` or array size): the field marks the beginning of an **array of inline objects**. Field type is `(inline-array your-type)`. Like C array of structs. - -Bonus ones, for where the array is stored _outside_ of the type: -- A dynamically typed GOAL array, stored outside your type (think `std::vector`): use `(name (array your-type))` -- A dynamically type GOAL array, stored inside your type: Not allowed, `array` is dynamic! -- An array of value types, stored outside your type: use `(name (pointer your-type))` -- An array of references (C++ array of pointers), stored outside your type: use `(name (pointer your-ref-type))` -- An array of objects of reference type (C++ array of structs), stored outside your type: use `(name (inline-array your-ref-type))` - -Of course, you can combine these, to get even more confusing types! But this seems uncommon. - -## Dynamic Size Types -Any type which ends with a dynamic array as the last field is dynamic. For these, it's a good idea to implement the `asize-of` method. - -# OpenGOAL Method System -OpenGOAL has a virtual method system. This means that child types can override parent methods. The first argument to a method is always the object the method is being called on, except for `new`. - -## Special Method Type: `_type_` -Methods have the same type as `function`. But they are allowed to use the special type `_type_`, which means "the compile-time type of the object the method is being called on". The type system is flexible with allowing you to use `_type_` in the method declaration in `deftype`, but not using `_type_` in the actual `defmethod`. - -## Built in Methods -All types have these 9 methods. They have reasonable defaults if you don't provide anything. - -### `new` -The new method is a very special method used to construct a new object, like a constructor. Note that some usages of the `new` keyword __do not__ end up calling the new method. See the `new` section for more details. Unlike C++, fields of a type and elements in an array are not constructed either. - -The first argument is an "allocation", indicating where the object should be constructed. It can be -- The symbol `'global` or `'debug`, indicating the global or debug heaps -- The symbols `'process-level-heap` or `'loading-level`, indicating whatever heaps are stored in those symbols. -- `'process`, indicating the allocation should occur on the current process heap. -- `'scratch`, for allocating on the scratchpad. This is unused. -- Otherwise it's treated as a 16-byte aligned address and used for in place construction (it zeros the memory first) - -The second argument is the "type to make". It might seem stupid at first, but it allows child classes to use the same `new` method as the parent class. - -The remaining arguments can be used for whatever you want. - -When writing your own `new` methods, you should ignore the `allocation` argument and use the `object-new` macro to actually do the allocation. This takes care of all the details for getting the memory (and setting up runtime type information if its a basic). See the section on `object-new` for more details. - -### `delete` -This method isn't really used very much. Unlike a C++ destructor it's never called automatically. In some cases, it's repurposed as a "clean up" type function but it doesn't actually free any memory. It takes no arguments. The default implementations call `kfree` on what the allocation, but there are two issues: -1. The implementation is sometimes wrong, likely confusing doing pointer math (steps by array stride) with address math (steps by one byte). -2. The `kfree` function does nothing. - -The `kheap` system doesn't really support freeing objects unless you free in the opposite order you allocate, so it makes sense that `delete` doesn't really work. - -### `print` -This method should print out a short description of the object (with no newlines) and return the object. The printing should be done with `(format #t ...)` (see the section on `format`) for more information. If you call `print` by itself, it'll make this description show up in the REPL. (Note that there is some magic involved to add a newline here... there's actually a function named `print` that calls the `print` method and adds a newline) - -The default short description looks like this: `#` for printing an object of type `test-type`. Of course, you can override it with a better version. Built-in types like string, type, boxed integer, pair, have reasonable overrides. - -This method is also used to print out the object with `format`'s `~A` format option. - -### `inspect` -This method should print out a detailed, multi-line description. By default, `structure`s and `basic`s will have an auto-generated method that prints out the name and value of all fields. For example: - -``` -gc > (inspect *kernel-context*) -[00164b44] kernel-context - prevent-from-run: 65 - require-for-run: 0 - allow-to-run: 0 - next-pid: 2 - fast-stack-top: 1879064576 - current-process: #f - relocating-process: #f - relocating-min: 0 - relocating-max: 0 - relocating-offset: 0 - low-memory-message: #t -``` - -In some cases this method is overridden to provide nicer formatting. - -### `length` -This method should return a "length". The default method for this just returns 0, but for things like strings or buffers, it could be used to return the number of characters or elements in use. It's usually used to refer to how many are used, rather than the capacity. - -### `asize-of` -This method should return the size of the object. Including the 4 bytes of type info for a `basic`. - -By default this grabs the value from the object's `type`, which is only correct for non-dynamic types. For types like `string` or other dynamic types, this method should be overridden. If you intend to store dynamically sized objects of a given type on a process heap, you __must__ implement this method accurately. - -### `copy` -Creates a copy of the object. I don't think this used very much. Just does a `memcpy` to duplicate by default. - -### `relocate` -The exact details are still unknown, but is used to update internal data structures after an object is moved in memory. This must be support for objects allocated in process heaps of processes allocated on the actor heap or debug actor heap. - -It's also called on objects loaded from a GOAL data object file. - -### `mem-usage` -Not much is known yet, but used for computing memory usage statistics. - - -## Details on the Order of Overrides -The order in which you `defmethod` and `deftype` matters. - -When you `deftype`, you copy all methods from the parent. When you `defmethod`, you always set a method in that type. You may also override methods in a child if: the child hasn't modified that method already, and if you are in a certain mode. This is a somewhat slow process that involves iterating over the entire symbol table and every type in the runtime, so I believe it was disabled when loading level code, and you just had to make sure to `deftype` and `defmethod` in order. - -Assume you have the type hierarchy where `a` is the parent of `b`, which is the parent of `c`. - -If you first define the three types using `deftype`, then override a method from `a` on `c`, then override that same method on `b`, then `c` won't use the override from `b`. - -If you first define the three types using `deftype`, then override a method on `b`, it will _sometimes_ do the override on `c`. This depends on the value of the global variable `*enable-method-set*`, and some other confusing options. It may also print a warning but still do the override in certain cases. - -# Syntax Basics -An "atom" in Lisp is a form that can't be broken down into smaller forms. For example `1234` is an atom, but `(1234 5678)` is not. OpenGOAL supports the following atoms: - -## Integers -All integers are by default `int`, a signed 64-bit integer. You can use: -- decimal: Like `123` or `-232`. The allowable range is `INT64_MIN` to `INT64_MAX`. -- hex: Like `#x123`. The allowable range is `0` to `UINT64_MAX`. Values over `INT64_MAX` will wrap around. -- binary: Like `#b10101010`. The range is the same as hex. -- character: - - Most can be written like `#\c` for the character `c`. - - Space is `#\\s` - - New Line is `#\\n` - - Tab is `#\\t` - - -## String -A string generates a static string constant. Currently the "const" of this string "constant" isn't enforced. Creating two identical string constants creates two different string objects, which is different from GOAL and should be fixed at some point. - -The string data is in quotes, like in C. The following escapes are supported: -- Newline: `\n` -- Tab: `\t` -- The `\` character: `\\` -- The `"` character: `\"` -- Any character: `\cXX` where `XX` is the hex number for the character. - -## Float -Any number constant with a decimal in it. The trailing and leading zeros and negative sign is flexible, so you can do any of these: -- `1.`, `1.0`, `01.`, `01.0` -- `.1`, `0.1`, `.10`, `0.10` -- `-.1`, `-0.1`, `-.10`, `-0.10` - -Like string, it creates a static floating point constant. In later games the float was inlined instead of being a static constant. - -## Symbol -Use `symbol-name` to get the value of a symbol and `'symbol-name` to get the symbol object. - -## Comments -Use `;` for line comments and `#|` and `|#` for block comments. - - - - -# Compiler Forms - Block Related - -## `begin` -Execute forms in order. -```lisp -(begin form...) -``` -A `begin` form is just a list of other forms which are executed in order. A `begin` form evaluates to the value of the last form. - -Example: -```lisp -(begin - (print "hello ") - (print "world!") - 7 - ) -``` -will print `hello world!` and the value of the entire form is `7`. - -The `begin` form is used a lot in macros, but not that much in code. It's generally used when you want to execute multiple things, but fit it into a single form. - - -## `block` -A `block` form is pretty similar to a `begin`, except the `block` has a name. You can then "return" from the block early with `return-from`. -```lisp -(block block-name form...) -``` - -Example: -```lisp -(block my-block - (print "hello ") - (return-from my-block 7) - (print "world") - "test" - ) -``` -will print `hello ` only and the value of the entire `block` form is `7`. The type of the `block` is the most specific type that describes all of the possible return values from any `return-from` or from reaching the end (even if its technically not possible to reach the end). In the case above, the possible return types are `int` and `string`, so the return type of the whole block is `object`, the lowest common ancestor type of `int` and `string`. - -Block is used rarely, and possibly almost never? - -## `return-from` -Exit a `block` or function early. -```lisp -(return-from block-name value) -``` - -Looks up the block and exits from it with the value. You can exit out nested blocks. If you are enclosed in multiple blocks with the same name, exits from the inner-most one with a matching name. Everything in a function is wrapped in a block named `#f`, so you can use `(return-from #f x)` to return early from a function with `x`. When using this form, it may change the return type of the function or block. The return type will be the lowest common ancestor type of all written return paths. If there is an unreachable return path, it will still be considered. - -Example -```lisp -(if (is-a-match? x) - (return-from #f x) - ) -``` -if `x` is a match, returns `x` from the function (not shown) immediately. - -The `return-from` form is very rarely used to return from a block, but sometimes used to return from a function. - -## `return` -Exit a function early. -```lisp -(return value) -``` -Has the same behavior as `(return-from #f value)`. - -## `label` -Create a named label for `goto` or `goto-when`. -```lisp -(label label-name) -``` -The label spaces are per-function and not nested. You can't jump from function to function. You can't jump in or out of functions which end up getting inlined. You can't jump in or out of an anonymous lambda function. You can jump in and out of `let`s. - -See `goto` for an example. - -Labels are used extremely rarely. Usually only in inline assembly and part of macros for `while` loops and similar. - -## `goto` -Jump to a label. -```lisp -(goto label-name) -``` -The label must be in the current label space. You can jump forward or backward. - -Example: -```lisp -(if skip-code? - (goto end) - ) - -;; code here runs only if skip-code is false. - -(label end) -``` - -The `goto` form used very rarely outside of macros and inline assembly. Try to avoid using `goto`. - -## `top-level` -This form is reserved by the compiler. Internally all forms in a file are grouped under a `top-level` form, so you may see this in error messages. Do not name anything `top-level`. - -# Compiler Forms - Compiler Commands -These forms are used to control the GOAL compiler, and are usually entered at the GOAL REPL, or as part of a macro that's executed at the GOAL REPL. These shouldn't be used in GOAL source code. - -## `reload` -Reload the GOAL compiler -```lisp -(reload) -``` -Disconnect from the target and reset all compiler state. This is equivalent to exiting the compiler and opening it again. - -## `get-info` -Get information about something. -```lisp -(get-info ) -``` -Use `get-info` to see what something is and where it is defined. - -For example: -```lisp -;; get info about a global variable: -g > (get-info *kernel-context*) -[Global Variable] Type: kernel-context Defined: text from goal_src/kernel/gkernel.gc, line: 88 -(define *kernel-context* (new 'static 'kernel-context - -;; get info about a function. This particular function is forward declared, so there's an entry for that too. -;; global functions are also global variables, so there's a global variable entry as well. -g > (get-info fact) -[Forward-Declared] Name: fact Defined: text from goal_src/kernel/gcommon.gc, line: 1098 -(define-extern fact (function int int)) - -[Function] Name: fact Defined: text from kernel/gcommon.gc, line: 1099 -(defun fact ((x int)) - -[Global Variable] Type: (function int int) Defined: text from goal_src/kernel/gcommon.gc, line: 1099 -(defun fact ((x int)) - -;; get info about a type -g > (get-info kernel-context) -[Type] Name: kernel-context Defined: text from goal_src/kernel/gkernel-h.gc, line: 114 -(deftype kernel-context (basic) - -;; get info about a method -g > (get-info reset!) -[Method] Type: collide-sticky-rider-group Method Name: reset! Defined: text from goal_src/engine/collide/collide-shape-h.gc, line: 48 -(defmethod reset! collide-sticky-rider-group ((obj collide-sticky-rider-group)) -[Method] Type: collide-overlap-result Method Name: reset! Defined: text from goal_src/engine/collide/collide-shape-h.gc, line: 94 -(defmethod reset! collide-overlap-result ((obj collide-overlap-result)) -[Method] Type: load-state Method Name: reset! Defined: text from goal_src/engine/level/load-boundary.gc, line: 9 -(defmethod reset! load-state ((obj load-state)) - -;; get info about a constant -g > (get-info TWO_PI) -[Constant] Name: TWO_PI Value: (the-as float #x40c90fda) Defined: text from goal_src/engine/math/trigonometry.gc, line: 34 -(defconstant TWO_PI (the-as float #x40c90fda)) - -;; get info about a built-in form -g > (get-info asm-file) -[Built-in Form] asm-file -``` - -## `autocomplete` -Preview the results of the REPL autocomplete: -```lisp -(autcomplete ) -``` - -For example: -```lisp -g > (autocomplete *) - * - *16k-dead-pool* - *4k-dead-pool* -... -Autocomplete: 326/1474 symbols matched, took 1.29 ms -``` - -## `seval` -Execute GOOS code. -```lisp -(seval form...) -``` -Evaluates the forms in the GOOS macro language. The result is not returned in any way, so it's only useful for getting side effects. It's not really used other than to bootstrap some GOAL macros for creating macros. - -## `asm-file` -Compile a file. -```lisp -(asm-file "file-name" [:color] [:write] [:load] [:no-code]) -``` -This runs the compiler on a given file. The file path is relative to the `jak-project` folder. These are the options: -- `:color`: run register allocation and code generation. Can be omitted if you don't want actually generate code. Usually you want this option. -- `:write`: write the object file to the `out/obj` folder. You must also have `:color` on. You must do this to include this file in a DGO. -- `:load`: send the object file to the target with the listener. Requires `:color` but not `:write`. There may be issues with `:load`ing very large object files (believed fixed). -- `:disassemble`: prints a disassembly of the code by function. Currently data is not disassebmled. This code is not linked so references to symbols will have placeholder values like `0xDEADBEEF`. The IR is printed next to each instruction so you can see what symbol is supposed to be linked. Requires `:color`. -- `:no-code`: checks that the result of processing the file generates no code or data. This will be true if your file contains only macros / constant definition. The `goal-lib.gc` file that is loaded by the compiler automatically when it starts must generate no code. You can use `(asm-file "goal_src/goal-lib.gc" :no-code)` to reload this file and double check that it doesn't generate code. - -To reduce typing, there are some useful macros: -- `(m "filename")` is "make" and does a `:color` and `:write`. -- `(ml "filename")` is "make and load" and does a `:color` and `:write` and `:load`. This effectively replaces the previous version of file in the currently running game with the one you just compiled, and is a super useful tool for quick debugging/iterating. -- `(md "filename")` is "make debug" and does a `:color`, `:write`, and `:disassemble`. It is quite useful for working on the compiler and seeing what code is output. -- `(build-game)` does `m` on all game files and rebuilds DGOs -- `(blg)` (build and load game) does `build-game` then sends commands to load KERNEL and GAME CGOs. The load is done through DGO loading, not `:load`ing individual object files. - -## `asm-data-file` -Build a data file. -```lisp -(asm-data-file tool-name "file-name") -``` -The `tool-name` refers to which data building tool should be used. For example, this should be `game-text` when building the game text data files. - -There's a macro `(build-data)` which rebuilds everything. - -## `gs` -Enter a GOOS REPL. -```lisp -(gs) -``` -Example: -``` -g> (gs) -goos> (+ 1 2 3) -6 -goos> (exit) -() -``` -mainly useful for debugging/trying things out in GOOS. The GOOS REPL shares its environment with the GOOS interpreter used by the compiler, so you can inspect/modify things for debugging with this. Likely not used much outside of initial debugging. - -## `set-config!` -```lisp -(set-config! config-name config-value) -``` -Used to set compiler configuration. This is mainly for debugging the compiler and enabling print statements. There is a `(db)` macro which sets all the configuration options for the compiler to print as much debugging info as possible. Not used often. - -## `in-package` -```lisp -(in-package stuff...) -``` -The compiler ignores this. GOAL files evidently start with this for some reason related to emacs. - -## `build-dgos` -```lisp -(build-dgos "path to dgos description file") -``` -Builds all the DGO files described in the DGO description file. See `goal_src/builds/dgos.txt` for an example. This just packs existing things into DGOs - you must have already built all the dependencies. - -In the future, this may become part of `asm-data-file`. - -## `add-macro-to-autocomplete` -```lisp -(add-macro-to-autocomplete macro-name) -``` - -Makes the given name show up as a macro in the GOAL REPL. Generating macros may be done programmatically using GOOS and this form can be used to make these show up in the autocomplete. This also makes the macro known to `get-info` which will report that the macro was defined at the location where the macro which expanded to `add-macro-to-autocomplete` is located in GOAL code. This is used internally by `defmacro`. - -# Compiler Forms - Control Flow - -## GOAL "Two Element" Conditions -These are `!=`, `eq?`, `neq?`, `=`, `>`, `<`, `>=`, `<=`. The default is to compare the two objects as unsigned 64-bit integers, unless a special case is hit. The special case is determined by the type of the __first__ argument: -- Floating point: if second argument is a number, convert to floating point. Use floating point comparisons -- Binteger: convert first argument to integer. If second argument is a number, convert to integer. Use signed integer comparisons -- Integer: If second argument is a number, convert to integer. Use signed integer comparison -- Unsigned Integer: If second argument is a number, convert to integer. Use unsigned integer comparison - -Note that comparing structures just checks if they refer to the same memory, not if the contents are the same. - -Currently there is absolutely no type checking done on comparisons, which makes it quite easy to get things wrong. - -When a two-element condition is used as a condition for an `if`, `cond`, `while`, `goto-when`, or any other branching condition, there is an optimization that avoids computing a GOAL boolean, then checking that GOAL boolean as a branch condition. Instead the comparison operation directly sets the x86-64 flags for a jump. Original GOAL has a similar optimization. - -There are a lot of unknown details -- What order are things evaluated / converted? -- How does the order work when there's a `not`? -- Type Checking is probably needed. -- Currently `=` and `eq?` are exactly the same. Likely `=` should only work on numbers and `eq?` should work on references? - -## `not` -```lisp -(not value) -``` -If value is `#f`, return `#t`. Otherwise return `#f`. - -Using `not` on a "two element condition" will be optimized to modifying that condition. - -## `when-goto` -```lisp -(when-goto value-or-condition destination) -``` -Examples: -```lisp -(when-goto (> x y) some-label) -(when-goto (is-player-dead?) some-label) -``` -Jump to `destination` if the condition is truthy (not `#f`). This ends up generating much better code than `(if condition (goto x))`. - -Like normal `goto`, this isn't used much outside of macros. - -## `cond` -A normal Lisp/Scheme `cond`. -```lisp -(cond (test clause...)...) -``` -Evaluates `test`s until one is truthy. Then evaluates all `clause`s in that case and returns the value of the last one. Optionally there can be a case with `else` as the test which runs if no other cases match. This must be the last case. - -If there is no `else`, and no cases match, return `#f`. (is it always `#f`?) - -The return type is the lowest common ancestor type of all cases. Note that for a `cond` without `else`, the possible return value of `#f` is __not__ considered when determining the return type. So: -```lisp -(print-type - (cond (test1 "hi") (test2 "bye")) - ) -``` -will print `[TYPE] string`, even though `cond` may return `#f`. - -This behavior seems correct if the return value is a `basic`, but less clear if the return value is supposed to be a number. - -Note that GOAL will load `#f` into a register for an else case even if the value of the `cond` is never used. Also, even if the value of the `cond` is never used, all cases will store their result into a common `cond` return-register. This is incredibly helpful for the decompiler! - -## `if` -A normal Lisp/Scheme `if`. -```lisp -(if test true-case [false-case]) -``` -The value of the entire statement is the value of the taken case. The false case can be left out. If the condition evaluates to false and the false case is left out, `#f` is returned. The return type follows the same logic as `cond`, as `if` is a macro using `cond`. - -## `when` -A normal Lisp/Scheme `when`. -```lisp -(when test forms...) -``` -If `test` is truthy, evaluate all forms and return the value of the last one. If `test` isn't true, return `#f`. - -## `unless` -A normal Lisp/Scheme `unless`. -```lisp -(unless test forms...) -``` -If `test` is false, evaluate all forms and return the value of the last one. If `test` isn't false, return `#f`. - -# Compiler Forms - Definition - - - -## `set!` -Set a value! -```lisp -(set! place value) -``` - -Sets `place` to `value`. The `place` can be once of: -- A lexical variable defined by `let`, or an argument of a function -- A global symbol -- A `car` or `cdr` of a pair -- A field of an object -- A dereferenced pointer -- An element in an array -- A bitfield within any of the above (not yet implemented) - -## `define` -Define a global symbol -```lisp -(define symbol-name value) -``` -Kind of like `set!`, but works exclusively on global symbols. Also sets the type of the global symbol. If the global symbol already has a type, and the `define` tries to change it, there will be an error or warning. (to be determined) - -## `define-extern` -Inform the compiler about the type of a global symbol -```lisp -(define-extern symbol-name type) -``` -Useful to forward declare functions or to let the compiler know about things in the C Kernel. See `goal-externs.gc` for where all the C Kernel stuff is declared. - -# Compiler Forms - Functions - -## `defun` -Define a new named global function! -```lisp -(defun name arg-list ["doc-string"] body...) -arg-list = (arg...) -arg = (arg-name arg-type)|arg-name -``` -Example: -``` -(defun 1/ ((x float)) - "Compute 1.0 / x" - (/ 1.0 x) - ) -``` -Define a new function with the given name. Note that until the `defun` itself executes, the function __cannot be used!__. So don't call functions before the `defun`. This is unlike C, where you can forward declare a function and use it before the actual definition. - -There is an optional docstring. Currently the docstring is just thrown away but in the future we could save them and and generate documentation or something. - -## `defmethod` -Define a method! -```lisp -(defmethod method-name type-name arg-list ["doc-string"] body...) -``` -In many ways is similar to `defun`. See section on methods for more details. - -You can only `defmethod` if one of these two is true: -- You are overriding a parent method -- You have declared this method in the type's `deftype` - -Like `defun`, the method only exists after the `defmethod` executes. Before that the method may be undefined or the parent's method. - -## `inline` -Make this function call inlined. -``` -(inline function-name) -``` -Example: inline call `my-function` -``` -((inline my-function) my-arg) -``` -Attempts to make the function call inline. If it is not possible, it will throw an error. You can't save the value of `(inline my-function)`, only use it immediately in a function call. You must provide a name of a global function as the function, not a function pointer. - -Methods cannot be inlined. (Maybe new methods can be in original GOAL?) - -## `lambda` -Create a GOAL Lambda. -```lisp -(lambda [:name name] [:inline-only #f|#t] [:segment main|debug] arg-list body-form..) -arg-list = (arg...) -arg = (arg-name arg-type)|arg-name -``` - -The `:name` option is only used for debugging the compiler - it cannot be accessed or referenced outside of internal compiler debugging features. It should be ignored unless you are debugging the compiler. - -The `:inline-only` flag defaults to `#f`, but can be set to `#t` to force the compiler to not generate x86-64 code for the function. In this case, the lambda cannot be used as a function pointer/GOAL `function` object. This is used in the `let` macro and not really useful for general programming. If the flag is `#f` (default value), it will generate and store an x86-64 function, regardless of if the function is ever called. - -The `:segment` option defaults to `main`, unless the `lambda` is defined in a function in the `debug` segment, in which case it defaults to `debug`. It can be overridden by this flag. This picks the segment of the object file where the function will be stored. - -The arguments default to type of `object` if no type is provided. In the case where a lambda is used immediately (a `let`) the compiler will propagate more specific types. This is why you don't have to give types when using `let`. - -A GOAL lambda has three purposes: -1. Generate a function as a "real" x86-64 function. -2. Save the code so the function can later be inlined -3. Immediately use the lambda in the head of a list as part of a `let` or `let*` macro - -__Example of 1:__ -``` -(sort my-list (lambda ((x int) (y int) (< x y)))) -``` -We create a real x86-64 function to pass to the `sort` function. In this case you should specify types - the default of `object` is not suitable. - -The other two cases are handled by `let` and `defun` macros, and shouldn't show up in other places, so they are left out. - -## `declare` -Set options for a function or method -```lisp -(declare [(inline)] [(allow-inline)] [(disallow-inline)] [(asm-func return-typespec)] [(print-asm)]) -``` -If used, this should be the first thing inside of a `defun`/`defmethod`. Don't use it anywhere else. -Example: -```lisp -(defun my-function () - (declare (inline)) - ... - ) -``` - -- `inline` means "inline whenever possible". See function inlining section for why inlining may be impossible in some cases. -- `allow-inline` or `disallow-inline`. You can control if inlining is allowed, though it is not clear why I thought this would be useful. Currently the default is to allow always. -- `print-asm` if codegen runs on this function (`:color #t`), disassemble the result and print it. This is intended for compiler debugging. -- `asm-func` will disable the prologue and epilogue from being generated. You need to include your own `ret` instruction or similar. The compiler will error if it needs to use the stack for a stack variable or a spilled register. The coloring system will not use callee saved registers and will error if you force it to use one. As a result, complicated GOAL expression may fail inside an `asm-func` function. The intent is to use it for context switching routines inside in the kernel, where you may not be able to use the stack, or may not want to return with `ret`. The return type of an `asm-func` must manually be specified as the compiler doesn't automatically put the result in the return register and cannot do type analysis to figure out the real return type. -- `allow-saved-regs` allows an `asm-func`'s coloring to use saved registers without an error. Stacks spills are still an error. The compiler will not automatically put things in a saved register, you must do this yourself. The move eliminator may still be used on your variables which use saved registers, so you should be careful if you really care about where saved variables are used. - -This form will probably get more options in the future. - -## `local-vars` -Declare variables local to a function, without an initial value. This will be used by the decompiler before `let` has been fully implemented. -```lisp -(local-vars (name type-spec)...) -``` - -The name can be any valid symbol. The scope of the variable is _always_ the function scope. Other scopes inside a function will always hide variables declared with `local-vars`. The type can be any GOAL typespec. If you use `float`, you get a floating point register, otherwise you get a normal GPR. - -It's recommended to avoid using this form. - -# Compiler Forms - Macro Forms - -## `#cond` -This is like an `#ifdef`, but the language for the conditions is GOOS and it's like a `cond`. -```lisp -(#cond (test clause...)...) -``` -The `#cond` is executed at compile time. The cases which don't match aren't compiled. There are `#when` and `#unless` macros that work like you would expect. - -Example: -```lisp -(#when PRINT_DEBUG_INFO - (format #t "debug info~%") - ) -``` - -## `quote` / `'` -The reader will expand `'thing` to `(quote thing)` - -```lisp -(quote thing) -``` -Currently `quote` supports: -- `'a-symbol` - will return the symbol `a-symbol` -- `'()` - will return the empty list. - -In the future, it should support quoted lists of static data. - -## `defglobalconstant` -Define a GOOS and GOAL constant. -```lisp -(defconstant name value) -``` - -The constant is available in both GOOS and GOAL. You can use it in `#cond` expressions (GOOS) and in GOAL code. The constant inserts the `value` as an s-expression in the code. This means it's a "code" constant, and doesn't have a type. - -There are a few restrictions: -- Currently constants do not work in `deftype`. (this should be fixed eventually) -- Constants may not work in places where the compiler is expecting a symbol which isn't actually compiled. So `(+ MY_CONSTANT 2)` is fine, but `(defun MY_CONSTANT () ...)` isn't. Don't use constants for type names, function names, or symbol names. It is fine to have a constant with a value of a symbol, but don't `(define MY_CONSTANT)` or `(set! MY_CONSTANT)` or `(new MY_CONSTANT)` or things like that. -- Don't use constants with `method` form. -- You can modify constants created with `defglobalconstant` in GOOS (effect won't be seen in GOAL) with `(set!)` -- There is no warning if one `defglobalconstant` changes the value of an existing `defglobalconstant` - -In general constants should have `UPPERCASE` names otherwise things get very confusing when there are name conflicts. - -The recommendation is to use constants for things like numbers or expressions like `(* NUM_THINGS SIZE_OF_EACH_THING)`. - -## `mlet` -Scoped constants in GOAL. Syntax is like a `let`. This feature has all the restrictions of `defglobalconstant`. -Avoid using `mlet` because it's confusing and not useful. -```lisp -(mlet ((constant-name constant-value)...) - body... - ) -``` - -Example: -```lisp -(mlet ((NUM-THINGS 12) - (THING-NAME "test")) - ; in here, NUM-THING and THING-NAME are constants - ) -``` - -# Compiler Forms - Math Forms - -Math forms will look at the type of the first argument to determine the "mode". So if you have `(+ 1 1.2)`, it will convert the `1.2` to an integer, do the add, and return an integer. - -## `+` -Addition. Can take 1 or more arguments. `(+ 1)` will give you `1`, like you'd expect. -```lisp -(+ form...) -``` -Works on integers and floats. Integer add is 64-bit and wrapping. - -## `-` -Subtraction or negative. Can take 1 or more arguments. -```lisp -(- form...) -``` -`(- 1)` gives you `-1`, but `(- 1 3)` gives you `-2`, which may be slightly confusing at first. Works on integers and floats. Integer subtract is 64-bit and wrapping. - -## `*` -Multiplication. Can take 1 or more arguments. -```lisp -(* form...) -``` -`(* 7)` will become `7`, as you'd expect. Works on integers and floats. The exact behavior for integers needs to be explored more because the EE is weird. GOAL generates different code for multiplication of signed vs. unsigned integers. - -## `/` -Division. Takes exactly two arguments -```lisp -(/ dividend divisor) -``` -Works on floats and integers. The exact behavior for integers needs to be explored more because the EE is weird. GOAL generates different code for division of signed vs. unsigned integers. - -## `mod` -Modulo operation. Takes exactly two arguments. -```lisp -(/ dividend divisor) -``` -Works on integers only. The exact behavior needs to be explored because the EE is weird. It is unknown if the same code is generate for signed/unsigned mod. - -## `slhv`, `sarv`, `shrv` -```lisp -(shlv value shift-amount) -``` -The exact behavior of GOAL shifts isn't fully understood, so these are temporary wrappers around x86-64 variable shift instructions. -- `shlv` shift left variable -- `sarv` shift arithmetic right variable -- `shrv` shift logical right variable -64-bit operation. - -## `logand` -Bitwise And -```lisp -(logand a b) -``` -64-bit operation. - -## `logior` -Bitwise Inclusive Or (normal Or) -```lisp -(logior a b) -``` -64-bit operation. - -## `logxor` -Bitwise Exclusive Or (Xor -```lisp -(logxor a b) -``` -64-bit operation. - -## `lognot` -Bitwise Not -```lisp -(lognot a) -``` -64-bit operation. - -# Compiler Forms - Type Forms - -## `defmethod` - -## `deftype` - - -## `method-of-object` -Get a method of an object. - -``` -(method-of-object object method-name) -``` - -This form takes an object and gets the method from it. If the object has runtime type information, will consult the method table at runtime to get a possibly more specific method than what is available at compile time. This uses the same lookup logic as method calling - see the section on method calls for more information. - -## `method-of-type` -Get a method of a type or an object. -``` -(method-of-type type method-name) -``` - -The first form of this takes a type name and method name and returns a GOAL `function` for this method. For example: -``` -(method string inspect) -``` -will return the `inspect` method of `string`. - -## `car` and `cdr` -Get element from pair -```lisp -(car some-pair) -(cdr some-pair) -``` - -The type of the result is always `object`, as pairs can hold any `object`. The type-check for `car` and `cdr` is relaxed - it allows it to be applied to any `pair` or `object`. The reason for allowing `object` is so you can write `(car (car x))` instead of `(car (the pair (car x)))`. However, if the argument to `car` is not a `pair`, you will get garbage or a crash. - -## `new` -```lisp -(new [allocation] [new-type-specification] [args]) -``` -See section on creating new GOAL objects. - -## `print-type` -Print the type of some GOAL expression at compile time. -```lisp -(print-type form) -``` -This is mainly used to debug the compiler or figure out why some code is failing a type check. The thing inside is compiled fully and used as the result of `print-type`. Example: -```lisp -(print-type "apples") ;; [TYPE] string -(print-type (+ 12 1.2)) ;; [TYPE] int -(print-type (the float 12)) ;; [TYPE] float -``` - -## `the` -Convert between types, doing the expected thing for number conversions. -```lisp -(the type thing) -``` - -If the `type` and the `thing` are both numbers, it will automatically convert between the different numeric types as needed. In all other cases, it does a dangerous `reinterpret_cast`. cppreference.com explains this idea clearly with "Converts between types by reinterpreting the underlying bit pattern." - -If the `thing` is a number: -- If `type` is `binteger`, convert the number to a `binteger` -- If `type` is `int` or `uint` (or some user-defined child type of these), convert the number to an integer. -- If `type` is `float`, convert the number to a `float` - -In all other cases, directly use the 64-bit value in the register as the value of the desired `type`. - - -Example of number conversions: -``` -(the binteger 12) ;; boxed integer for 12 -(the float 1) ;; floating point 1.0 -(the int 1.234) ;; signed integer 1 -``` - -Examples of common casts: -``` -(the uint 1) ;; unsigned integer, doesn't do any modification if input is already an int. -(the (pointer uint8) x) ;; like C (uint8_t*)x -(the string (car x)) ;; if you know (car x) is a string, do this. -``` - -Examples of dangerous things you can do but probably shouldn't: -``` -(the string 1.234) ;; take the binary representation of 1.234 and treat it as a GOAL memory address to string data. -(the float 'bean) ;; take the GOAL memory address of the "bean" symbol and treat it as the binary representation of a float. -``` - -There are some weird edge cases with `the` that are worth mentioning, if you try to set the value of something you've used `the` on. In the future the compiler should block you from doing something bad, but this is not yet implemented. - -``` -(set! (-> (the (pointer uint8) x)) 1) ;; OK, not casting the actual value -(set! (the int (-> obj x)) 1) ;; NOT OK. Int is a value type -;; Will actually work if x is already an int/uint, and with non-numeric values types like pointer. Avoid just to be safe. -(set! (the string (-> obj x)) "test") ;; OK, string is a basic, which is a reference type. -``` - -This becomes more clear if we look at the C equivalent: -``` -*(uint8_t*)(a->x) = 1; // makes sense - -(int)(a->x) = 1; // doesn't make sense - -*(thing*)(a->x) = thing(); // makes sense -``` - - -## `the-as` -Convert between types always using `reinterpret_cast` -```lisp -(the-as type thing) -``` -cppreference.com explains this idea clearly with "Converts between types by reinterpreting the underlying bit pattern." - -The recommendation is to prefer `the` in almost all cases, unless you want to set an exact bit pattern of a `float`, or want to examine the bits in a `float`. - -Example: -``` -(the-as int 1.234) ;; result happens to be 1067316150 -``` - -None of the edge cases of `the` apply to `the-as`. - -## `size-of` -Get the size of a type, in bytes. -```lisp -(size-of ) -``` -Get the size of a type, by name. The type must be a plain name, like `pointer` or `dma-bucket`. Compound types are not supported. It works on value and structure types and returns the size in memory. For dynamic types, it returns the size as if the dynamic part has 0 size. For weird types like `none` it throws an error. - -This value can be used in most places where the compiler is expecting a constant integer as well, such as the size of a stack array, which must be known at compile time. - -Example: -```lisp -(size-of dma-bucket) -``` -Returns the size of the `dma-bucket` type which is 16 bytes. - - - -## Pointer Math -Not implemented well yet. - -# Compiler Forms - Assembly - -## `rlet` -```lisp -(rlet ((var-name [:reg reg-name] [:class reg-class] [:type typespec] [:reset-here #t|#f])...) - body... - ) -``` -Create register variables. You can optionally specify a register with the `:reg` option and a register name like `rax` or `xmm3`. The initial value of the register is not set. If you don't specify a register, a GPR will be chosen for you by the coloring system and it will behave like a `let`. If you don't specify a register, you can specify a register class (`gpr`, a normal 64-bit integer register; `fpr`, a 32-bit single precision float; or `vf`, and 128-bit floating point vector register) and the compiler will pick a GPR or XMM for you. - -If you pick a callee-saved register and use it within the coloring system, the compiler will back it up for you in the prologue and restore it in the epilogue. -If you pick a special register like `rsp`, it won't be backed up. - -Inside the `rlet`, all uses of `var-name` will always be in the given register. If the variable goes dead (or is never live), the compiler may reuse the register as it wants. The compiler may also spill the variable onto the stack. Of course, if you are in an `asm-func`, the stack will never be used. Be extremely careful about using "normal" registers without the coloring system and with higher-level code as the compiler may use your "normal" register as a temporary. If you read the value of a register and use the coloring system, the variable will then be alive starting at the beginning of the function, and will make that register unavailable to the compiler and other `rlet`s that occur before. This is useful to preserve the value of a temporary register if needed, but can also be undesirable in other cases. If you add the `:reset-here #t` flag, it will make the variable dead until the start of the `rlet`. It "resets" the value of the register in the coloring system at the start of the `rlet`. The default value is false. It is recommended to keep the default value when accessing specific registers that are also normally used by the compiler. For special registers like `rsp`, `r15`, `r14`, and `r13`, if you plan to use them with the coloring system, it is recommended to set the `reset-here` flag. - -Here is an example of using an `rlet` to access registers: -```lisp -(defun get-goal-rsp-2 () - "Get the stack pointer as a GOAL pointer" - (rlet ((rsp :reg rsp :type uint) - (off :reg r15 :type uint)) - (the pointer (- rsp off)) - ) - ) -``` - -## General assembly forms -In general, assembly forms have a name that begins with a `.`. They all evaluate to `none` and copy the form of an x86-64 instruction. For example `(.sub dst src)`. A destination must be a settable register (ok if it's spilled). So you can't do something like `(.sub (-> obj field) x)`. Instead, do `(set! temp (-> obj field))`, `(.sub temp x)`, `(set! (-> obj field) temp)`. The sources can be any expression, or a register. This allows you to mix high-level code with assembly easily, like `(.mov rax (-> obj field))` or `(.push (+ 1 (-> obj field)))`. - -By default, assembly forms work with the coloring system. This means that assembly and high level expression can be mixed together without clobbering each other. It also means use of callee-saved registers will cause them to be backed up/restored in the function prologue and epilogue. Use of weird registers like `r15`, `r14`, and `rsp` works as you would expect with the coloring system. - -But you can also request to skip this with `:color #f` option, like `(.push my-reg-var :color #f)`. Be very careful with this. The `:color #f` option will only work with register variables from `rlet` which have a manually specified register. It will entirely bypass the coloring system and use this register. Use of this near high level GOAL variables is extremely dangerous and should be done very carefully or avoided, as the GOAL compiler will not know that you could be modifying its registers. In a form with `:color #f`, you cannot use higher level code or variables - all variables must be defined in `rlet`s. This is because higher level expressions and variables cannot be used without the coloring system. - -## `.sub` -```lisp -(.sub dest src [:color #t|#f]) -``` -x86-64 subtraction (64-bit). If coloring is on (the default), the `dest` must be a settable register (`rlet` var, `let` var, function argument, ...). It can't be a place like a symbol, field, stack variable, etc. If coloring is off, both `src` and `dest` must be registers defined and constrained in an enclosing `rlet`. - -Example: -``` -(defun get-goal-rsp () - (declare (asm-func uint)) - (rlet ((rsp :reg rsp :type uint) - (off :reg r15 :type uint) - (ret :reg rax :type uint) - ) - - ;; mov rax, rsp - (set! ret rsp) - ;; sub rax, r15 - (.sub ret off) - ;; ret - (.ret) - ) - ) -``` - -## `.add` -```lisp -(.add dest src [:color #t|#f]) -``` -Addition (64-bit). Similar to subtraction. - -## `.jr` -```lisp -(.jr addres-reg [:color #t|#f]) -``` -Jump-register. Jumps to the address given. The address is treated as a 64-bit pointer, not a GOAL pointer. - -## `.load-sym` -```lisp -(.load-sym dest symbol-name [:sext #t|#f] [:color #t|#f]) -``` -Load the value of a symbol into a register. By default, it will look at the type of the symbol to determine if it should be sign extended or not. You can override this with the `:sext` option if needed. The symbol must be known to the type system. - -## `.push` -```lisp -(.push src [:color #t|#f]) -``` - -The x86-64 push instruction. Does a 64-bit GPR. The `src` can be any expression that can be put in a gpr if color is on. Otherwise it must be a register defined and constrained in an enclosing `rlet`. - -## `.pop` -```lisp -(.pop dst [:color #t|#f]) -``` - -The x86-64 pop instruction. Does a 64-bit GPR. The `dst` can be any expression which evaluates to a settable register if color is on. Otherwise it must be a register defined and constrained in an enclosing `rlet`. - -## `.ret` -```lisp -(.ret [:color #t|#f]) -``` - -The x86-64 ret instruction. The color option does nothing. This is not recognized as a control flow instruction by the coloring system. It does not touch the return register `rax`. - -## `.mov` -```lisp -(.mov dst src [:color #t|#f]) -``` -Move between two registers. The `dst` should be a register (either `rlet` or `let` variable), and the `src` can be a register or any expression. The following moves are supported: -- `gpr` to `gpr` -- `fpr` to `fpr` (only moves lower 32-bits of the xmms, uses `movss`) -- `vf` to `vf` (moves all 128-bits of the xmms, uses `vmovaps`) -- `gpr` to `fpr` (only moves 32-bits, uses `movd`) -- `fpr` to `gpr` (only moves 32-bits, upper 32-bits are zero, uses `movd`) -This code generation is identical to using a `(set! dst src)` form. - -## `.nop.vf` -```lisp -(.nop.vf) -``` - -Inserts a `FNOP` assembly instruction, which is fundamentally the same as a `NOP`. It is a 2-byte instruction. - -## `.nop` or `(nop!)` -```lisp -(.nop) -;; or -(nop!) -``` - -Inserts a single-byte `nop`. - -## `.wait.vf` -```lisp -(.wait.vf) -``` - -Inserts a `FWAIT` assembly instruction, x86 does not require as much synchronization as the PS2's VU registers did, but it has a purpose in rare cases. It is a 2-byte instruction. - -## `.lvf` -```lisp -(.lvf dst-reg src-loc [:color #t|#f] [:offset ]) -``` -Load a vector float register from `src-loc`. The `dst-reg` must be a vector float register. The `src-loc` can be a gpr containing a GOAL pointer or expression which gives a GOAL pointer. There is no type checking on the `src-loc` so be careful. The load uses `vmovaps`, so the source must be 16-byte aligned. - -If the source is in the form `base-reg + constant-offset`, like from a `(&-> my-object my-inline-vector-field)`, the constant offset will be folded into the load instruction like `vmovaps xmm1, [r15 + rax + 12]`. An explicit offset can be provided via the `:offset` keyword, and will be used if applicable. - -If the source is an immediate `(new 'static ...)` form that results in a statically allocated variable, it will use `RIP` relative addressing (32-bit immediate) form. This means that the code: -```lisp -(.lvf vf1 (new 'static 'vector :x 1.2 :y 2.3 :z 3.4 :w 5.6)) -``` -will be just a single instruction to do a `vmovaps xmm1, [rip + XXX]`. - -## `.svf` -```lisp -(.svf dst-loc src-reg [:color #t|#f] [:offset ]) -``` -Store a vector float. Works similarly to the `lvf` form, but there is no optimized case for storing into a static because this isn't allowed in GOAL. - -## Three operand vector float operations. -```lisp -(.[.].vf dst src0 src1 [:color #t|#f] [:mask #b<0-15>]) -``` -All the three operand forms work similarly. You can do something like `(.add.vf vf1 vf2 vf3)`. All operations use the similarly named `vps` instruction, xmm128 VEX encoding. We support the following `op-name`s: -- `xor` -- `add` -- `sub` -- `mul` -- `min` -- `max` - -An optional `:mask` value can be provided as a binary number between 0-15 (inclusive). This determines _which_ of the resulting elements will be committed to the destination vector. For example, `:mask #b1011` means that the `w`, `y` and `x` results will be committed. Note that the components are defined left-to-right which may be a little counter-intuitive -- `w` is the left-most, `x` is the right-most. This aligns with the PS2's VU implementation. - -Additionally, all of these operations support defining a single `broadcast-element`. This can be one of the 4 vector components `x|y|z|w`. Take the following for an example: `(.add.x.xyzw vf10, vf20, vf30)`, translates into: - -```cpp -vf10[x] = vf20[x] + vf30[x] -vf10[y] = vf20[y] + vf30[x] -vf10[z] = vf20[z] + vf30[x] -vf10[w] = vf20[w] + vf30[x] -``` - -## Three operand vector float operations with the accumulator -```lisp -(.[.].vf dst src0 src1 acc [:color #t|#f] [:mask #b<0-15>]) -``` -There are a few functions that will perform multiple operations involving the accumulator. We support the following `op-name`s: -- `add.mul` - Calculate the product of `src0` and `src1` and add it to the value of `acc` => `acc + (src0 * src1)` -- `sub.mul` - Calculate the product of `src0` and `src1` and subtract it from the value of `acc` => `acc - (src0 * src1)` - -An optional `:mask` value can be provided as a binary number between 0-15 (inclusive). This determines _which_ of the resulting elements will be committed to the destination vector. For example, `:mask #b1011` means that the `w`, `y` and `x` results will be committed. Note that the components are defined left-to-right which may be a little counter-intuitive -- `w` is the left-most, `x` is the right-most. This aligns with the PS2's VU implementation. - -Additionally, all of these operations support defining a single `broadcast-element`. This can be one of the 4 vector components `x|y|z|w`. - -## `.abs.vf` -```lisp -(.abs.vf dst src [:color #t|#f] [:mask #b<0-15>]) -``` - -Calculates the absolute value of the `src` vector, and stores in the `dst` vector. - -## `.div.vf` and `.sqrt.vf` -```lisp -(.div.vf dst src1 src2 :ftf #b<0-3> :fsf #b<0-3> [:color #t|#f]) -``` - -Calculates the quotient of _one_ of `src1`'s components specified by `fsf` _one_ of `src2`'s components specified by `ftf` and stores in every component of `dst` - -```lisp -(.sqrt.vf dst src :ftf #b<0-3> [:color #t|#f]) -``` - -Calculates the square-root of _one_ of `src`'s components specified by `ftf` and stores in every component of `dst` - - -These instructions are interesting as they behave differently than the other math operations. In the original VU, results were stored in a seperate `Q` register, which was _NOT_ 128-bit. Instead it was a 32-bit register, meaning you have to pick which component from `src` you want to use. `:fsf` and `:ftf` are used to accomplish this, as usual, this is through bit flags -- `00` will select `x` and `11` will select `w`. - -As `dst` is just yet another vector / xmm register in x86, things are kept simple and the quotient is copied to _all_ packed single-float positions. This allows: -- Selecting any of the resulting vector slots will be equal to the quotient. -- Since the low-floating-point (X) is defined, the xmm register should function as expected for normal math operations - -## `.outer.product.vf` -```lisp -(.outer.product.vf dst src1 src2 [:color #t|#f]) -``` - -Calculates the outer-product of `src1` and `src2` and stores the result in `dst`. _ONLY_ the x,y,z components are considered, and `dst`'s `w` component will be untouched. The following example illustrates what the outer-product is: - -Given 2 vectors `V1 = <1,2,3,4>` and `V2 = <5,6,7,8>` and assume `VDEST = <0, 0, 0, 999>` -The outer product is computed like so (only x,y,z components are operated on): - -`x = (V1y * V2z) - (V2y * V1z) => (2 * 7) - (6 * 3) => -4` - -`y = (V1z * V2x) - (V2z * V1x) => (3 * 5) - (7 * 1) => 8` - -`z = (V1x * V2y) - (V2x * V1y) => (1 * 6) - (5 * 2) => -4` - -`w = N/A, left alone => 999` - -> `VDEST = <-4, 8, -4, 999>` - -## `.blend.vf` -```lisp -(.blend.vf dst src0 src1 mask [:color #t|#f]) -``` -Wrapper around `vblendps` (VEX xmm128 version) instruction. The `mask` must evaluate to a constant integer at compile time. The integer must be in the range of 0-15. - -## `.itof.vf` and `.ftoi.vf` -``` -(.itof.vf dst src [:mask mask-val] [:color #t|#f]) -(.ftoi.vf dst src [:mask mask-val] [:color #t|#f]) -``` - -Wrapper around `vcvtdq2ps` and `vcvtps2dq` to convert packed 32-bit signed integers to packed 32-bit floats and back. The `mask` and `color` arguments behave like other assembly operations. - -## `.pw.sra`, `.pw.srl`, and `pw.sll` -``` -(.pw.sra dst src shift-amount [:mask mask-val] [:color #t|#f]) -(.pw.srl dst src shift-amount [:mask mask-val] [:color #t|#f]) -(.pw.sll dst src shift-amount [:mask mask-val] [:color #t|#f]) -``` - -Wrapper around `vpsrld`, `vpsrad`, and `vpslld`. Does shifts on each of the 4 32-bit integers in the register. - -## `.pextlw`, `.pextuw`, `.pcpyud`, `.pcpyld`, `.pceqw`, `.ppach` -``` -(.pextlw dst src0 src1 [:color #t|#f]) -(.pextuw dst src0 src1 [:color #t|#f]) -(.pcpyud dst src0 src1 [:color #t|#f]) -(.pcpyld dst src0 src1 [:color #t|#f]) -(.pceqw dst src0 src1 [:color #t|#f]) -(.ppach dest src0 src1) -``` - -Equivalents of the EE's MMI instructions with the same name. These can only be used on 128-bit variables. Most map to single x86 instructions: -- `pextlw` is `VPUNPCKLDQ` (sources swapped) -- `pextuw` is `VPUNPCKHDQ` (sources swapped) -- `pcpyld` is `VPUNPCKLQDQ` (sources swapped) -- `pcpyud` is `VPUNPCKHQDQ` (sources _not_ swapped) -- `pceqw` is `VPCMPEQD` - -Some map to multiple instructions. These must use the coloring system. -- `ppach` is a sequence of 7 instructions (`VPSHUFLW`, `VPSHUFHW`, `VPSRLDQ`, `VPUNPCKLQDQ`). - -# Compiler Forms - Unsorted - -## `let` -```lisp -(let ((var-name value)...) body...) -``` -Define local variables. The variables are automatically assigned a type, like C++ `auto`. - -Example: -```lisp -(let ((count-1 (+ a b)) ;; count_1 = a + b - (count-2 (+ c d))) ;; count_2 = c + d - (if (count > 10) - ;; print a message if count > 10. - (format #t "count = ~D, ~D~%" count-1 count-2) - ) - ) -``` - -Note, if you define multiple variables in a `let`, you cannot refer to variables defined previously in the same `let`. In the example above, `count-2`'s value couldn't be defined in terms of `count-1` for example. See `let*`. - -## `let*` -```lisp -(let ((var-name value)...) body...) -``` - -Define local variables. If you define multiple in a single `let*`, you can refer to previous variables. - -Example: -```lisp -(let* ((count-1 (+ a b)) - (count-2 (+ count-1 d))) ;; okay because we used let* - (if (count > 10) - ;; print a message if count > 10. - (format #t "count = ~D, ~D~%" count-1 count-2) - ) - ) -``` - -## Things related to enums -Not yet implemented - -## `defmacro` - -## Loop forms - -## `&` - -## `->` - -## Type - -## Compile-Time Size stuff - -## `object-new` - - -# Compiler Features - -## Compiling a list -When the compiler encounters a list like `(a b c)` it attempts to parse in multiple ways in this order: -1. A compiler form -2. A GOOS macro -3. An enum (not yet implemented) -4. A function or method call - -## Compiling an integer -Integers can be specified as -- decimal: `1` or `-1234` (range of `INT64_MIN` to `INT64_MAX`) -- hex: `#x123`, `#xbeef` (range of `0` to `UINT64_MAX`) -- binary: `#b101010` (range of `0` to `UINT64_MAX`) - -All integers are converted to the signed "integer in variable" type called `int`, regardless of how they are specified. -Integer "constant"s are not stored in memory but instead are generated by code, so there's no way to modify them. - -## Compiling a string -A string constant can be specified by just putting it in quotes. Like `"this is a string constant"`. -There is an escape code `\` for string: -- `\n` newline -- `\t` tab character -- `\\` the `\` character -- `\"` the `"` character -- `\cXX` where `XX` is a two character hex number: insert this character. -- Any other character following a `\` is an error. - -OpenGOAL stores strings in the same segment of the function which uses the string. I believe GOAL does the same. - -In GOAL, string constants are pooled per object file (or perhaps per segment)- if the same string appears twice, it is only included once. OpenGOAL currently does not pool strings. If any code is found that modifies a string "constant", or if repeated strings take up too much memory, string pooling will be added. - -For now I will assume that string constants are never modified. - - -## Compiling a float -A floating point constant is distinguished from an integer by a decimal point. Leading/trailing zeros are optional. Examples of floats: `1.0, 1., .1, -.1, -0.2`. Floats are stored in memory, so it may be possible to modify a float constant. For now I will assume that float constants are never modified. It is unknown if they are pooled like strings. - -Trivia: Jak 2 realized that it's faster to store floats inline in the code. - -## Compiling a symbol -A `symbol` appearing in code is compiled by trying each of these in the following order -1. Is it `none`? (see section on `none`) -2. Try `mlet` symbols -3. Try "lexical" variables (defined in `let`) -4. Try global constants -5. Try global variables (includes named functions and all types) - -## The Special `none` type -Anything which doesn't return anything has a return type of `none`, indicating the return value can't be used. This is similar to C's `void`. - -## GOAL Structs vs. C Structs -There is one significant difference between C and GOAL when it comes to structs/classes - GOAL variables can only be references to structs. - -As an example, consider a GOAL type `my-type` and a C type `my_type`. In C/C++, a variable of type `my_type` represents an entire copy of a `my_type` object, and a `my_type*` is like a reference to an existing `my_type` object. In GOAL, an object of `my-type` is a reference to an existing `my-type` object, like a C `my_type*`. There is no equivalent to a C/C++ `my_type`. - -As a result you cannot pass or return a structure by value. - -Another way to explain this is that GOAL structures (including `pair`) always have reference semantics. All other GOAL types have value semantics. - -## Pointers -GOAL pointers work a lot like C/C++ pointers, but have some slight differences: -- A C `int32_t*` is a GOAL `(pointer int32)` -- A C `void*` is a GOAL `pointer` -- In C, if `x` is a `int32_t*`, `x + 1` is equivalent to `uintptr_t(x) + sizeof(int32_t)`. In GOAL, all pointer math is done in units of bytes. -- In C, you can't do pointer math on a `void*`. In GOAL you can, and all math is done in units of bytes. - -In both C and GOAL, there is a connection between arrays and pointers. A GOAL array field will have a pointer-to-element type, and a pointer can be accessed as an array. - -One confusing thing is that a `(pointer int32)` is a C `int32_t*`, but a `(pointer my-structure-type)` is a C `my_structure_type**`, because a GOAL `my-structure-type` is like a C `my_structure_type*`. - -## Inline Arrays -One limitation of the system above is that an array of `my_structure_type` is actually an array of references to structures (C `object*[]`). It would be more efficient if instead we had an array of structures, laid out together in memory (C `object[]`). - -GOAL has a "inline array" to represent this. A GOAL `(inline-array thing)` is like a C `thing[]`. The inline-array can only be used on structure types, as these are the only reference types. - - -## Fields in Structs -For a field with a reference type (structure/basic) -- `(data thing)` is like C `Thing* data;` -- `(data thing :inline #t)` is like C `Thing data;` -- `(data thing 12)` is like C `Thing* data[12];`. The field has `(pointer thing)` type. -- `(data thing 12 :inline #t)` is like `Thing data[12];`. The field has `(inline-array thing)` type - -For a field with a value type (integer, etc) -- `(data int32)` is like C `int32_t data;` -- `(data int32 12)` is like `int32_t data[12];`. The field has `(array int32)` type. - -Using the `:inline #t` option on a value type is not allowed. - -## Dynamic Structs -GOAL structure can be dynamically sized, which means their size isn't determined at compile time. Instead the user should implement `asize-of` to return the actual size. - -This works by having the structure end in an array of unknown size at compile time. In a dynamic structure definition, the last field of the struct should be an array with an unspecified size. To create this, add a `:dynamic #t` option to the field and do not specify an array size. This can be an array of value types, an array of reference types, or an inline-array of reference types. - -### Unknown -Is the `size` of a dynamic struct: -- size assuming the dynamic array has 0 elements (I think it's this) -- size assuming the dynamic array doesn't - -These can differ by padding for alignment. - - -## Methods - -## Method `_type_` type - -## Calling Methods - -## Built-in Methods - -## New - How To Create GOAL Objects -GOAL has several different ways to create objects, all using the `new` form. - -### Heap Allocated Objects -A new object can be allocated on a heap with `(new 'global 'obj-type [new-method-arguments])`. -This simply calls the `new` method of the given type. You can also replace `'global` with `'debug` to allocate on the debug heap. -Currently these are the only two heaps supported, in the future you will be able to call the new method with other arguments -to allow you to do an "in place new" or allocate on a different heap. - -This will only work on structures and basics. If you want a heap allocated float/integer/pointer, create an array of size 1. -This will work on dynamically sized items. - -### Heap Allocated Arrays -You can construct a heap array with `(new 'global 'inline-array 'obj-type count)` or `(new 'global 'array 'obj-type count)`. -These objects are not initialized. Note that the `array` version creates a `(pointer obj-type)` plain array, -__not__ a GOAL `array` type fancy array. In the future this may change because it is confusing. - -Because these objects are uninitialized, you cannot provide constructor arguments. -You cannot use this on dynamically sized member types. However, the array size can be determined at runtime. - -### Static Objects -You can create a static object with `(new 'static 'obj-type [field-def]...)`. It can be a structure, basic, bitfield, array, boxed array, or inline array. -Each field def looks like `:field-name field-value`. The `field-value` is evaluated at compile time. Fields -can be integers, floats, symbols, pairs, strings, or other statics. These field values may come from macros or GOAL constants. - -For bitfields, there is an exception, and fields can be set to expression that are not known at compile time. The compiler will generate the appropriate code to combine the values known at compile time and run time. This exception does not apply to a bitfield inside of another `(new 'static ...)`. - -Fields which aren't explicitly initialized are zeroed, except for the type field of basics, which is properly initialized to the correct type. - -This does not work on dynamically sized structures. - -### Stack Allocated Arrays -Currently only arrays of integers, floats, or pointers can be stack allocated. -For example, use `(new 'stack ''array 'int32 1)` to get a `(pointer int32)`. Unlike heap allocated arrays, these stack arrays -must have a size that can be determined at compile time. The objects are uninitialized. - -### Stack Allocated Structures -Works like heap allocated, the objects are initialized with the constructor. The constructor must support "stack mode". Using `object-new` supports stack mode so usually you don't have to worry about this. The structure's memory will be memset to 0 with `object-new` automatically. - -## Defining a `new` Method - -## Integer Type -GOAL has some weird behavior when it comes to integers. It may seem complicated to describe, but it really makes the implementation simpler - the integer types are designed around the available MIPS instructions. - -Integers that are used as local variables (defined with `let`), function arguments, function return values, and intermediate values when combining these are called "register integers", as the values will be stored in CPU registers. - -Integers that are stored in memory as a field of a `structure`/`basic`, an element in an array, or accessed through a `pointer` are "memory integers", as the values will need to be loaded/stored from memory to access them. - -The "register integer" types are `int` and `uint`. They are 64-bit and mostly work exactly like you'd expect. Multiplication, division, and mod, are a little weird and are documented separately. - -The "memory integer" types are `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, and `uint64`. - -Conversions between these types are completely automatic - as soon as you access a "memory integer", it will be converted to a "register integer", and trying to store a "register integer" will automatically convert it to the appropriate "memory integer". It (should be) impossible to accidentally get this wrong. - -### Side Note -- It's not clear what types `(new 'static 'integer)` or `(new 'stack 'integer)` are, though I would assume both are memory. - -- If there aren't enough hardware registers, "register integers" can be spilled to stack, but keep their "register integer" types. This process should be impossible to notice, so you don't have to worry about it. - -## Array Spacing -In general, all GOAL objects are 16-byte aligned and the boxing system requires this. All heap memory allocations are 16-byte aligned too, so this is usually not an issue. - -## Truth -Everything is true except for `#f`. This means `0` is true, and `'()` is true. -The value of `#f` can be used like `nullptr`, at least for any `basic` object. It's unclear if `#f` can/should be used as a null for other types, including `structure`s or numbers or pointers. - -Technical note: the hex number `0x147d24` is considered false in Jak 1 NTSC due to where the symbol table happened to be allocated. However, checking numbers for true/false shouldn't be done, you should use `(zero? x)` instead. - -## Empty Pair - -# Built-in Types - -## `structure` -A structure is the parent type of all types with fields. - -## `basic` -A basic is a structure with runtime type information. The first field is named `type` and contains the `type` of the basic. - -# Implemented in Runtime Language Functions - -## `string->symbol` - -## `symbol->string` - -## `format` -GOAL's `printf`. -```lisp -(format destination format-string args...) -``` - -The following destinations are currently supported: -- `#t` - print to the listener. After the current game frame is over, this will be sent to the compiler and you can see it at the GOAL prompt. -- `0` - print to `stdout` in the runtime immediately. This won't be visible from within the compiler - you must look at the runtime to see it. This is useful for debugging if the runtime crashes before flushing the normal print buffer. -- `#f` - print to a new string allocated on the global heap. -- a GOAL `string` object - append to an existing string -- a GOAL `file-stream` object - currently unsupported but eventually may allow printing into a file somewhere - -The global variable `*print-column*` can be used to automatically print at a certain indentation. The very first thing printed during a frame will not have the indentation applied. - -The format string escape sequences all start with `~`, then have arguments (possibly none), then have a single character code. The arguments look like: -- `~A`, the `A` code with no arguments -- `~12A`, the `A` code with an integer argument of `12` -- `~'zA`, the `A` code with a character argument of `z` -- ``` ~`hello`A```, the `A` code with a string argument of `"hello"` -- `~12,'bA`, the `A` code with two arguments, integer `12` and character `b` - - -The escape codes are - -### `~%` -Replace with a newline character. - -### `~~` -Replace with a `~` character - -### `~G` or `~g` -Replace with a C-style string given in `args`. Note that this will not work on a GOAL string. - -### `~A` or `~a` -Print a boxed object given in `args`. Uses the `print` method. Can take optional arguments for length and padding character. If the `print` method gives something shorter than the length argument, it will be padded on the left with the padding character (default is space). If the `print` method gives something too long, it will be truncated on the right. The last character printed will be changed to `~` to indicate it was truncated here. - -There is an option to left justify instead but I don't think there's a way to turn it on. - -### `~S` or `~s` -Very similar to `~A`, but a GOAL string will be printed without quotes around it. - -### `~C` or `~c` -Print a single character (GOAL `uint8` or `int8` type) - -### `~P` or `~p` -Print an object, similar to `~A`, but does not have optional arguments for length or padding. Instead, there is an optional argument to give a type name, then use that types `print` method. This is useful for printing non-boxed objects. If no type name is given, behaves like `~A`. - -### `~I` or `~i` -Like `~P`, but uses the inspect method instead. - -### `~Q` or `~q` -Likely was supposed to be for printing 128-bit integers stored in registers, but will always prints `0`. The `format` function is written in C using varargs, which doesn't support 128-bit register values. - -### `~B` or `~b` -Print integer as binary. Optional arguments for pad-length and pad-character (default is 0). Won't truncate. - -### `~D` or `~d` -Print integer as decimal (signed). Optional arguments for pad-length and pad-character (default is space). Won't truncate. - -### `~X` or `~x` -Print integer as hex. Optional arguments for pad-length and pad-character (default is 0). Won't truncate. - -### `~F` -Print floating point. Will print 4 digits after the decimal and pad with space to a width of 12 and does not accept any options. - -### `~f` -Print floating point. Takes optional arguments for pad-length (default don't pad), pad-character (default space), and precision (default 4). - -## `~R` or `~r` -Like `~f` but scales in-game rotation units to degrees. The float `65536.0` is 360 degrees. - -### `~M` or `~m` -Like `~f` but scales in-game distance units to meters. The float `4096.0` is 1 meter. - -### `~E` or `~e` -Like `~f` for argument, but takes an integer time-code as an input and scales time-code units to seconds. There are 300 ticks / second, which is the smallest number which has an integer number of ticks per NTSC and PAL frame. Something very weird happens when the input is negative? - -### `~T` or `~t` -Insert a tab character. - -### Pass Through Codes -Some codes will be passed through automatically, along with any arguments. This will also pass through arguments of `+` or `-`. I believe that these options could later be interpreted by the code for printing on-screen characters. - -The pass through codes are `H,J,K,L,N,V,W,Y,Z,h,j,k,l,n,v,w,y,z`. - -Any other codes will result in an error message being printed. - - - -## Load Stuff - -# GOOS -# Built-in Functions -# Compiler Start Procedure -# Inline Functions +For more information on the type system, see [the following](type_system.md) \ No newline at end of file diff --git a/docs/markdown/_404.md b/docs/markdown/_404.md new file mode 100644 index 0000000000..b21c14655b --- /dev/null +++ b/docs/markdown/_404.md @@ -0,0 +1 @@ +# 404 - Doc Page not Found! \ No newline at end of file diff --git a/docs/markdown/_sidebar.md b/docs/markdown/_sidebar.md index 6bd8ec53e9..9d5baba8ac 100644 --- a/docs/markdown/_sidebar.md +++ b/docs/markdown/_sidebar.md @@ -1,9 +1,20 @@ -* [Home](/README.md) -* [Compiler Example](compiler_example.md) -* [Type System](type_system.md) -* [Emitter](emitter.md) -* [GOAL Debugging](goal_dbg_doc.md) -* [GOOS](goos.md) -* [Object File Generation](object_file_generation.md) -* [Porting to x86](porting_to_x86.md) -* [Registers](registers.md) +- OpenGOAL Reference + - [Overview](/README.md) + - [Type System](/type_system.md) + - [Method System](/method_system.md) + - [Language Syntax & Features](/syntax.md) + - [Standard Library](/lib.md) + - [The Reader](/reader.md) + - [Macro Support](/goos.md) + - [Object File Formats](/object_file_formats.md) + +- Working with OpenGOAL + - [The REPL](/repl.md) + - [Debugging](/debugging.md) + - [Editor Configuration](/editor_setup.md) + +- Developing OpenGOAL + - [Compiler Walkthrough](/compiler_example.md) + - [Assembly Emitter](/asm_emitter.md) + - [Porting to x86](/porting_to_x86.md) + - [Register Handling](/registers.md) diff --git a/docs/markdown/emitter.md b/docs/markdown/asm_emitter.md similarity index 99% rename from docs/markdown/emitter.md rename to docs/markdown/asm_emitter.md index 060a44ee65..e2aa787fcf 100644 --- a/docs/markdown/emitter.md +++ b/docs/markdown/asm_emitter.md @@ -1,4 +1,5 @@ -# Emitter +# Assembly Emitter + x86-64 has a lot of instructions. They are described in Volume 2 of the 5 Volume "Intel® 64 and IA-32 Architectures Software Developer’s Manual". Just this volume alone is over 2000 pages, which would take forever to fully implement. As a result, we will use only a subset of these instructions. This the rough plan: - Most instructions like `add` will only be implemented with `r64 r64` versions. diff --git a/docs/markdown/compiler_example.md b/docs/markdown/compiler_example.md index a56a20cf92..6bab146bdc 100644 --- a/docs/markdown/compiler_example.md +++ b/docs/markdown/compiler_example.md @@ -1,18 +1,43 @@ # Compiler Example -This describes how the compiler works on a small piece of code in `example_goal.gc`. + +This describes how the compiler works using the following code snippet, saved in a file named `example_goal.gc`. + +```lisp +(defun factorial-iterative ((x integer)) + (let ((result 1)) + (while (!= x 1) + (set! result (* result x)) + (set! x (- x 1)) + ) + result + ) + ) + +;; until we load KERNEL.CGO automatically, we have to do this to +;; make format work correctly. +(define-extern _format function) +(define format _format) + +(let ((x 10)) + (format #t "The value of ~D factorial is ~D~%" x (factorial-iterative x)) + ) +``` + To run this yourself, start the compiler and runtime, then run: -``` +```lisp (lt) (asm-file "doc/example_goal.gc" :color :load) ``` And you should see: + ``` The value of 10 factorial is 3628800 ``` -# Overview +## Overview + The code to read from the GOAL REPL is in `Compiler.cpp`, in `Compiler::execute_repl`. Compiling an `asm-file` form will call `Compiler::compile_asm_file` in `CompilerControl.cpp`, which is where we'll start. I've divided the process into these steps: @@ -25,31 +50,35 @@ I've divided the process into these steps: 7. __Linking__: The runtime links the code so it can be run, and runs it. 8. __Result__: The code prints the message which is sent back to the REPL. -# Reader -The reader converts GOAL/GOOS source into a `goos::Object`. One of the core ideas of lisp is that "code is data", so GOAL code is represented as GOOS data. This makes it easy for GOOS macros to operate on GOAL code. A GOOS object can represent a number, string, pair, etc. This strips out comments/whitespace. +## Reader + +The reader converts GOAL/GOOS source into a `goos::Object`. One of the core ideas of lisp is that "code is data", so GOAL code is represented as GOOS data. This makes it easy for GOOS macros to operate on GOAL code. A GOOS object can represent a number, string, pair, etc. This strips out comments/whitespace. The reader is run with this code: -``` + +```cpp auto code = m_goos.reader.read_from_file({filename}); ``` + If you were to `code.print()`, you would get: -``` + +```lisp (top-level (defun factorial-iterative ((x integer)) (let ((result 1)) (while (!= x 1) (set! result (* result x)) (set! x (- x 1))) result)) (define-extern _format function) (define format _format) (let ((x 10)) (format #t "The value of ~D factorial is ~D~%" x (factorial-iterative x)))) ``` - There are a few details worth mentioning about this process: - The reader will expand `'my-symbol` to `(quote my-symbol)` - The reader will throw errors on syntax errors (mismatched parentheses, bad strings/numbers, etc.) - Using `read_from_file` adds information about where each thing came from to a map stored in the reader. This map is used to determine the source file/line for compiler errors. -# IR Pass +## IR Pass + This pass converts code (represented as a `goos::Object`) into intermediate representation. This is stored in an `Env*`, a tree structure. At the top is a `GlobalEnv*`, then an `FileEnv` for each file compiled, then a `FunctionEnv` for each function in the file. There are environments within `FunctionEnv` that are used for lexical scoping and the other types of GOAL scopes. The Intermediate Representation (IR) is a list per function that's built up in order as the compiler goes through the function. Note that the IR is a list of instructions and doesn't have a tree or more complicated structure. Here's an example of the IR for the example function: -``` +```nasm Function: function-factorial-iterative - mov-ic igpr-2, 1 - mov igpr-3, igpr-2 + mov-ic igpr-2, 1 + mov igpr-3, igpr-2 goto label-10 mov igpr-4, igpr-3 imul igpr-4, igpr-0 @@ -84,37 +113,46 @@ Function: function-top-level mov igpr-16, igpr-11 ret igpr-17 igpr-16 ``` -The `function-top-level` is the "top level" function, which is everything not in a function. In the example, this is just defining the function, defining `format`, and calling `format`. + +The `function-top-level` is the "top level" function, which is everything not in a function. In the example, this is just defining the function, defining `format`, and calling `format`. You'll notice that there are a ton of `mov`s between `igpr`. The compiler inserts tons of moves. Because this is all done in a single pass, there's a lot of cases where the compiler can't know if a move is needed or not. But the register allocator can figure it out and will remove most unneeded moves. Adding moves can also prevent stack spills. For example, consider the case where you want to get the return value of function `a`, then call function `b`, then call function `c` with the return value of function `a`. If there are a lot of moves, the register allocator can figure out a way to temporarily stash the value in a saved register instead of spilling to the stack. Another thing to notice is that GOAL nested function calls suck. Example: -``` + +```lisp (format #t "The value of ~D factorial is ~D~%" x (factorial-iterative x)) ``` + requires loading `format`, `#t`, the string, and `x` into registers, then calling `(factorial-iterative x)`, then calling `format`. This has to be done this way, just in case the `factorial-iterative` call modifies the value of `format`, `#t`, or `x`. -## IR Pass Implementation +### IR Pass Implementation + An important type in the compiler is `Val`, which is a specification on how to get a value. A `Val` has an associated GOAL type (`TypeSpec`) and the IR Pass should take care of all type checking. A `Val` can represent a constant, a memory location (relative to pointer, or static data, etc), a spot in an array, a register etc. A `Val` representing a register is a `RegVal`, which contains an `IRegister`. An `IRegister` is a register that's not yet mapped to the hardware, and instead has a unique integer to identify itself. The IR assumes there are infinitely many `IRegister`s, and a later stage maps `IRegister`s to real hardware registers. -The general process starts with a `compile` function, which dispatches other `compile_` functions as needed. These generally take in `goos::Object` as a code input, emit IR into an `Env`/or modify things in the `Env`, and return a `Val*` describing the result. +The general process starts with a `compile` function, which dispatches other `compile_` functions as needed. These generally take in `goos::Object` as a code input, emit IR into an `Env`/or modify things in the `Env`, and return a `Val*` describing the result. -In general, GOAL is very greedy and `compile` functions emit IR to do things, then put the result in a register, and return a `RegVal`. +In general, GOAL is very greedy and `compile` functions emit IR to do things, then put the result in a register, and return a `RegVal`. However, there is an exception for memory related things. Consider -``` + +```lisp (-> my-object my-field) ; my_object->my_field in C -``` -This shouldn't return a `Val` for a register containing the value of `my_object->my_field`, but should instead return something that represents "the memory location of my_field in my_object". This way you can do ``` + +This shouldn't return a `Val` for a register containing the value of `my_object->my_field`, but should instead return something that represents "the memory location of my_field in my_object". This way you can do + +```lisp (set! (-> my-object my-field) val) ;; or (& (-> my-object my-field)) ;; &my_object->my_field ``` -and the compiler will have enough information to figure out the memory address. + +and the compiler will have enough information to figure out the memory address. If the compiler actually needs the value of something, and wants to be sure its a value in a register, it will use the `to_reg` method of `Val`. This will emit IR into the current function to get the value in a register, then return a `Val` that represents this register. Example `to_reg` implementation for integer constants: -``` + +```cpp RegVal* IntegerConstantVal::to_reg(Env* fe) { auto rv = fe->make_gpr(m_ts); fe->emit(std::make_unique(rv, m_value)); @@ -122,29 +160,29 @@ RegVal* IntegerConstantVal::to_reg(Env* fe) { } ``` - Note that `to_reg` can emit code like reading from memory, where the order of operations really matters, so you have to be very careful. It's extremely dangerous to let a memory reference `Val` propagate too far. Consider this example: -``` +```lisp (let ((x (-> my-object my-field))) (set! (-> my-object my-field) 12) x ) ``` + Where `x` should be the old value of `my-field`. The `Val` for `x` needs to be `to_reg`ed _before_ getting inside the `let`. There's also some potential confusion around the order that you compile and `to_gpr` things. In a case where you need a bunch of values in gprs, you should do the `to_gpr` immediately after compiling to match the exact behavior of the original GOAL. For example -``` +```lisp (+ (-> my-array (* x y)) (some-function)) ;; like c++ my_array[x*y] + some_function() ``` -When we `compile` the `(-> my-array (* x y))`, it will emit code to calculate the `(*x y)`, but won't actually do the memory access until we call `to_reg` on the result. This memory access should happen __before__ `some-function` is called. +When we `compile` the `(-> my-array (* x y))`, it will emit code to calculate the `(*x y)`, but won't actually do the memory access until we call `to_reg` on the result. This memory access should happen __before__ `some-function` is called. In general, each time you `compile` something, you should immediately `to_gpr` it, _before_ `compile`ing the next thing. Many places will only accept a `RegVal` as an input to help with this. Also, the result for almost all compilation functions should be `to_reg`ed. The only exceptions are forms which deal with memory references (address of operator, dereference operator) or math. -Another important thing is that compilation functions should _never_ modify any existing `IRegister`s or `Val`s, unless that function is `set!`, which handles the situation correctly. Instead, create new `IRegister`s and move into those. I am planning to implement a `settable` flag to help reduce errors. +Another important thing is that compilation functions should _never_ modify any existing `IRegister`s or `Val`s, unless that function is `set!`, which handles the situation correctly. Instead, create new `IRegister`s and move into those. I am planning to implement a `settable` flag to help reduce errors. For example: - `RegVal` storing a local variable: is `settable`, you can modify local variables by writing to the register they use. @@ -153,29 +191,36 @@ For example: The only settable `RegVal` is one corresponding to a local variable. -## Following the Code +### Following the Code This pass runs from here: -``` + +```cpp auto obj_file = compile_object_file(obj_file_name, code, !no_code); ``` + That function sets up a `FileEnv*`, then runs -``` + +```cpp file_env->add_top_level_function( compile_top_level_function("top-level", std::move(code), compilation_env)); -``` -which compiles the body of the function with: ``` + +which compiles the body of the function with: + +```cpp auto result = compile_error_guard(code, fe.get()); ``` The `compile_error_guard` function takes in code (as a `goos::Object`) and a `Env*`, and returns a `Val` representing the return value of the code. It calls the `compile` function, but wraps it in a `try catch` block to catch any compilation errors and print an error message. In the case where there's no error, it just does: -``` + +```cpp return compile(code, env); ``` The `compile` function is pretty simple: -``` + +```cpp /*! * Highest level compile function */ @@ -197,8 +242,10 @@ Val* Compiler::compile(const goos::Object& code, Env* env) { return get_none(); } ``` + In our case, the code starts with `(defun..`, which is actually a GOOS macro. It throws away the docstring, creates a lambda, then stores the function in a symbol: -``` + +```lisp ;; Define a new function (defmacro defun (name bindings &rest body) (if (and @@ -214,7 +261,8 @@ In our case, the code starts with `(defun..`, which is actually a GOOS macro. It ``` The compiler notices this is a macro in `compile_pair`: -``` + +```cpp if (head.is_symbol()) { // ... @@ -225,10 +273,11 @@ The compiler notices this is a macro in `compile_pair`: // ... } +``` -``` The `compile_goos_macro` function sets up a GOOS environment and interprets the GOOS macro to generate more GOAL code: -``` + +```cpp Val* Compiler::compile_goos_macro(const goos::Object& o, const goos::Object& macro_obj, const goos::Object& rest, @@ -243,10 +292,12 @@ Val* Compiler::compile_goos_macro(const goos::Object& o, return compile_error_guard(goos_result, env); // compile resulting GOAL code } ``` -and the last line of that function compiles the result of macro expansion in GOAL. + +and the last line of that function compiles the result of macro expansion in GOAL. As an example, I'm going to look at `compile_add`, which handles the `+` form, and is representative of typical compiler code for this part. We start by checking that the arguments look valid: -``` + +```cpp Val* Compiler::compile_add(const goos::Object& form, const goos::Object& rest, Env* env) { auto args = get_va(form, rest); // get arguments to + in a list if (!args.named.empty() || args.unnamed.empty()) { @@ -255,19 +306,23 @@ Val* Compiler::compile_add(const goos::Object& form, const goos::Object& rest, E ``` Then we compile the first thing in the `(+ ...` form, get its type, and pick a math mode (int, float): -``` + +```cpp auto first_val = compile_error_guard(args.unnamed.at(0), env); auto first_type = first_val->type(); auto math_type = get_math_mode(first_type); ``` + In the integer case, we first create a new variable in the IR called an `IRegister` that must be in a GPR (as opposed to an XMM floating point register), and then emit an IR instruction that sets this result register to the first argument. -``` + +```cpp auto result = env->make_gpr(first_type); env->emit(std::make_unique(result, first_val->to_gpr(env))); ``` Then, for each of the remaining arguments, we do: -``` + +```cpp for (size_t i = 1; i < args.unnamed.size(); i++) { env->emit(std::make_unique( IntegerMathKind::ADD_64, result, @@ -275,16 +330,18 @@ Then, for each of the remaining arguments, we do: ->to_gpr(env))); } ``` + which emits an IR to add the value to the sum. The `to_math_type` will emit any code needed to convert this to the correct numeric type (returns either a numeric constant or a `RegVal` containing the value). An important detail is that we create a new register which will hold the result. This may seem inefficient in cases, but a later compile pass will try to make this new register be the same register as `first_val` if possible, and will eliminate the `IR_RegSet`. -# Register Allocation +## Register Allocation This step figures out how to match up `IRegister`s to real hardware registers. In the case where there aren't enough hardware registers, it figures out how to "spill" variables onto the stack. The current implementation is a very greedy one, so it doesn't always succeed at doing things perfectly. The stack spilling is also not handled very efficiently, but the hope is that most functions won't require stack spilling. This step is run from `compile_asm_function` on the line: -``` + +```cpp color_object_file(obj_file); void Compiler::color_object_file(FileEnv* env) { @@ -305,15 +362,16 @@ void Compiler::color_object_file(FileEnv* env) { ``` The actual algorithm is too complicated to describe here, but it figures out a mapping from `IRegister`s to hardware registers. It also figures out how much space on the stack is needed for any stack spills, which saved registers will be used, and deals with aligning the stack. - -# Code Generation +## Code Generation This part actually generates the static data and x86 instructions and stores them in an `ObjectGenerator`. See `CodeGenerator::do_function`. It emits the function prologue and epilogue, as well as any extra loads/stores from the stack that the register allocator added. Each `IR` gets to emit instructions with: -``` + +```cpp ir->do_codegen(&m_gen, allocs, i_rec); ``` Each IR has its own `do_codegen` that emits the right instruction, and also any linking data that's needed. For example, instructions that access the symbol table are patched by the runtime to directly access the correct slot of the hash table, so the `do_codegen` also lets the `ObjectGenerator` know about this link: -``` + +```cpp // IR_GetSymbolValue::do_codegen // look at register allocation result to determine hw register @@ -326,19 +384,23 @@ auto instr = gen->add_instr(IGen::load32u_gpr64_gpr64_plus_gpr64_plus_s32( // add link info gen->link_instruction_symbol_mem(instr, m_src->name()); ``` + here `0xbadbeef` is used as a placeholder offset - the runtime should patch this to the actual offset of the symbol. There's a ton of book-keeping to figure out the correct offsets for `rip`-relative addressing, or how to deal with jumps to/from IR which become multiple (or zero!) x86-64 instructions. It should all be handled by `ObjectFileGenerator`, and not in `do_codegen` or `CodeGenerator`. -# Object File Generation +## Object File Generation Once the `CodeGenerator` is done going through all functions and static data, it runs: -``` + +```cpp return m_gen.generate_data_v3().to_vector(); ``` + This actually lays out everything in memory. It takes a few passes because x86 instructions are variable length (may even change based on which registers are used!), so it's a little bit tricky to figure out offsets between different instructions or instructions and data. Finally it generates link data tables, which efficiently pack together links to the same symbols into a single entry, to avoid duplicated symbol names. The link table also contains information about linking references in between different segments, as different parts of the object file may be loaded into different spots in memory, and will need to reference each other. This is the final result for top-level function (stored in top-level segment) -``` + +```nasm ;; prologue push rbx push rbp @@ -400,11 +462,13 @@ pop rbp pop rbx ret ``` + and the factorial function (stored in main segment) -``` + +```nasm mov eax,0x1 jmp 0x18 -mul eax,edi +mul eax,edi movsxd rax,eax mov ecx,0x1 sub rdi,rcx @@ -414,11 +478,14 @@ jne 0xa ret ``` -# Sending and Receiving +## Sending and Receiving + The result of `codegen_object_file` is sent with: -``` + +```cpp m_listener.send_code(data); ``` + which adds a message header then just sends the code over the socket. The receive process is complicated, so this is just a quick summary @@ -427,14 +494,15 @@ The receive process is complicated, so this is just a quick summary - Once fully received, `WaitForMessageAndAck` will return a pointer to the message. The name of this function is totally wrong, it doesn't wait for a message, and it doesn't ack the message. - The main loop in `KernelCheckAndDispatch` will see this message and run `ProcessListenerMessage` - `ProcessListenerMessage` sees that it has code, copies the message to the debug heap, and links it. -``` + +```cpp auto buffer = kmalloc(kdebugheap, MessCount, 0, "listener-link-block"); memcpy(buffer.c(), msg.c(), MessCount); ListenerLinkBlock->value = buffer.offset + 4; ListenerFunction->value = link_and_exec(buffer, "*listener*", 0, kdebugheap, LINK_FLAG_FORCE_DEBUG).offset; ``` + - The `link_and_exec` function doesn't actually execute anything because it doesn't have the `LINK_FLAG_EXECUTE` set, it just links things. It moves the top level function and linking data to the top of the heap (temporary storage for the kernel) and keep both the main segment and debug segment of the code on the debug heap. It'll move them together and eliminate gaps before linking. After linking, the `ListenerFunction->value` will contain a pointer to the top level function, which is stored in the top temp area of the heap. This `ListenerFunction` is the GOAL `*listener-function*` symbol. - The next time the GOAL kernel runs, it will notice that `*listener-function*` is set, then call this function, then set it to `#f` to indicate it called the function. -- This - After this, `ClearPending()` is called, which sends all of the `print` messages with the `Deci2Server` back to the compiler. - Because the GOAL kernel changed `ListenerFunction` to `#f`, it does a `SendAck()` to send a special `ACK` message to the compiler, saying "I got the function, ran it, and didn't crash. Now I'm ready for more messages." diff --git a/docs/markdown/goal_dbg_doc.md b/docs/markdown/debugging.md similarity index 88% rename from docs/markdown/goal_dbg_doc.md rename to docs/markdown/debugging.md index 3ef4f60a48..cc48b7d7d7 100644 --- a/docs/markdown/goal_dbg_doc.md +++ b/docs/markdown/debugging.md @@ -1,10 +1,15 @@ # OpenGOAL Debugger + Currently the debugger only works on Linux. All the platform specific stuff is in `xdbg.cpp`. -## `(dbs)` +## Commands + +### `(dbs)` + Print the status of the debugger and listener. The listener status is whether or not there is a socket connection open between the compiler and the target. The "debug context" is information that the runtime sends to the compiler so it can find the correct thread to debug. In order to debug, you need both. -## `(dbg)` +### `(dbg)` + Attach the debugger. This will stop the target. Example of connecting to the target for debugging: @@ -37,28 +42,35 @@ gc> (dbs) Context: valid = true, s7 = 0x147d24, base = 0x2000000000, tid = 1062568 ``` -## `(:cont)` +### `(:cont)` + Continue the target if it has been stopped. -## `(:break)` +### `(:break)` + Immediately stop the target if it is running. Will print some registers. -## `(:dump-all-mem )` +### `(:dump-all-mem )` + Dump all GOAL memory to a file. Must be stopped. -``` + +```lisp (:dump-all-mem "mem.bin") ``` + The path is relative to the Jak project folder. The file will be the exact size of `EE_MAIN_MEM_SIZE`, but the first `EE_LOW_MEM_PROTECT` bytes are zero, as these cannot be written or read. ## Address Spec + Anywhere an address can be used, you can also use an "address spec", which gives you easier ways to input addresses. For now, the address spec is pretty simple, but there will be more features in the future. - `(sym-val )`. Get the address stored in the symbol with the given name. Currently there's no check to see if the symbol actually stores an address or not. This is like "evaluate ``, then treat the value as an address" - `(sym )`. Get the address of the symbol object itself, including the basic offset. Example to show the difference: + ```lisp ;; the symbol is at 0x142d1c @@ -88,30 +100,31 @@ gc> (inspect *kernel-context*) ;; break, so we can debug gc> (:break) Read symbol table (159872 bytes, 226 reads, 225 symbols, 1.96 ms) -rax: 0xfffffffffffffdfc rcx: 0x00007f745b508361 rdx: 0x00007f745b3ffca0 rbx: 0x0000000000147d24 -rsp: 0x00007f745b3ffc40 rbp: 0x00007f745b3ffcc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000000000 r9: 0x0000000000000008 r10: 0x00007f745b3ffca0 r11: 0x0000000000000293 -r12: 0x0000000000147d24 r13: 0x00007ffdff32cfaf r14: 0x00007ffdff32cfb0 r15: 0x00007f745b3fffc0 +rax: 0xfffffffffffffdfc rcx: 0x00007f745b508361 rdx: 0x00007f745b3ffca0 rbx: 0x0000000000147d24 +rsp: 0x00007f745b3ffc40 rbp: 0x00007f745b3ffcc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000000000 r9: 0x0000000000000008 r10: 0x00007f745b3ffca0 r11: 0x0000000000000293 +r12: 0x0000000000147d24 r13: 0x00007ffdff32cfaf r14: 0x00007ffdff32cfb0 r15: 0x00007f745b3fffc0 rip: 0x00007f745b508361 ;; reads the symbol's memory: ;; at 0x142d1c there is the value 0x164a84 gc> (dw (sym *kernel-context*) 1) - 0x00142d1c: 0x00164a84 + 0x00142d1c: 0x00164a84 ;; treat the symbol's value as an address and read the memory there. ;; notice that the 0x41 in the first word is decimal 65, the first field of the kernel-context. gc> (dw (sym-val *kernel-context*) 10) - 0x00164a84: 0x00000041 0x00000000 0x00000000 0x00000002 - 0x00164a94: 0x70004000 0x00147d24 0x00147d24 0x00000000 - 0x00164aa4: 0x00000000 0x00000000 + 0x00164a84: 0x00000041 0x00000000 0x00000000 0x00000002 + 0x00164a94: 0x70004000 0x00147d24 0x00147d24 0x00000000 + 0x00164aa4: 0x00000000 0x00000000 ``` -## `(:pm)` +### `(:pm)` + Print memory -``` +```lisp (:pm elt-size addr elt-count [:print-mode mode]) ``` @@ -119,7 +132,6 @@ The element size is the size of each word to print. It can be 1, 2, 4, 8 current There are some useful macros inspired by the original PS2 TOOL debugger (`dsedb`) for the different sizes. They are `db`, `dh`, `dw`, and `dd` for 1, 2, 4, and 8 byte hex prints which follows the naming convention of MIPS load/stores. There is also a `df` for printing floats. See the example below. - ```lisp OpenGOAL Compiler 0.1 @@ -144,28 +156,28 @@ gc> (set! (-> x 2) 2.0) ;; attach the debugger (halts the target) gc> (dbg) [Debugger] PTRACE_ATTACHED! Waiting for process to stop... -rax: 0xfffffffffffffdfc rcx: 0x00007f6b94964361 rdx: 0x00007f6b8fffeca0 rbx: 0x0000000000147d24 -rsp: 0x00007f6b8fffec40 rbp: 0x00007f6b8fffecc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000000000 r9: 0x000000000000000b r10: 0x00007f6b8fffeca0 r11: 0x0000000000000293 -r12: 0x0000000000147d24 r13: 0x00007ffd16fb117f r14: 0x00007ffd16fb1180 r15: 0x00007f6b8fffefc0 +rax: 0xfffffffffffffdfc rcx: 0x00007f6b94964361 rdx: 0x00007f6b8fffeca0 rbx: 0x0000000000147d24 +rsp: 0x00007f6b8fffec40 rbp: 0x00007f6b8fffecc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000000000 r9: 0x000000000000000b r10: 0x00007f6b8fffeca0 r11: 0x0000000000000293 +r12: 0x0000000000147d24 r13: 0x00007ffd16fb117f r14: 0x00007ffd16fb1180 r15: 0x00007f6b8fffefc0 rip: 0x00007f6b94964361 Debugger connected. ;; print memory as 10 bytes gc> (db 1452224 10) - 0x001628c0: 0x00 0x00 0x80 0x3f 0x00 0x00 0x00 0x00 0x00 0x00 + 0x001628c0: 0x00 0x00 0x80 0x3f 0x00 0x00 0x00 0x00 0x00 0x00 ;; print memory as 10 words (32-bit words) gc> (dw 1452224 10) - 0x001628c0: 0x3f800000 0x00000000 0x40000000 0x00000000 - 0x001628d0: 0x00000000 0x00000000 0x00000000 0x00000000 - 0x001628e0: 0x00000000 0x00000000 + 0x001628c0: 0x3f800000 0x00000000 0x40000000 0x00000000 + 0x001628d0: 0x00000000 0x00000000 0x00000000 0x00000000 + 0x001628e0: 0x00000000 0x00000000 ;; print memory as 10 floats gc> (df 1452224 10) - 0x001628c0: 1.0000 0.0000 2.0000 0.0000 - 0x001628d0: 0.0000 0.0000 0.0000 0.0000 - 0x001628e0: 0.0000 0.0000 + 0x001628c0: 1.0000 0.0000 2.0000 0.0000 + 0x001628d0: 0.0000 0.0000 0.0000 0.0000 + 0x001628e0: 0.0000 0.0000 ;; set some more values, must unbreak first gc> (:cont) @@ -174,31 +186,32 @@ gc> (set! (-> x 1) (the-as float -12)) ;; break and print as decimal gc> (:break) -rax: 0xfffffffffffffdfc rcx: 0x00007f6b94964361 rdx: 0x00007f6b8fffeca0 rbx: 0x0000000000147d24 -rsp: 0x00007f6b8fffec40 rbp: 0x00007f6b8fffecc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000000000 r9: 0x0000000000000004 r10: 0x00007f6b8fffeca0 r11: 0x0000000000000293 -r12: 0x0000000000147d24 r13: 0x00007ffd16fb117f r14: 0x00007ffd16fb1180 r15: 0x00007f6b8fffefc0 +rax: 0xfffffffffffffdfc rcx: 0x00007f6b94964361 rdx: 0x00007f6b8fffeca0 rbx: 0x0000000000147d24 +rsp: 0x00007f6b8fffec40 rbp: 0x00007f6b8fffecc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000000000 r9: 0x0000000000000004 r10: 0x00007f6b8fffeca0 r11: 0x0000000000000293 +r12: 0x0000000000147d24 r13: 0x00007ffd16fb117f r14: 0x00007ffd16fb1180 r15: 0x00007f6b8fffefc0 rip: 0x00007f6b94964361 gc> (:pm 4 1452224 10 :print-mode unsigned-dec) - 0x001628c0: 1065353216 4294967284 1073741824 0 - 0x001628d0: 0 0 0 0 - 0x001628e0: 0 0 + 0x001628c0: 1065353216 4294967284 1073741824 0 + 0x001628d0: 0 0 0 0 + 0x001628e0: 0 0 gc> (:pm 4 1452224 10 :print-mode signed-dec) - 0x001628c0: 1065353216 -12 1073741824 0 - 0x001628d0: 0 0 0 0 - 0x001628e0: 0 0 + 0x001628c0: 1065353216 -12 1073741824 0 + 0x001628d0: 0 0 0 0 + 0x001628e0: 0 0 ``` +### `(:disasm)` -## `(:disasm)` Disassembly instructions in memory -``` +```lisp (:disasm addr len) ``` Example (after doing a `(lt)`, `(blg)`, `(dbg)`): -```asm + +```nasm gc> (:disasm (sym-val basic-type?) 80) [0x2000162ae4] mov eax, [r15+rdi*1-0x04] [0x2000162ae9] mov ecx, [r15+r14*1+0x38] @@ -216,14 +229,13 @@ gc> (:disasm (sym-val basic-type?) 80) [0x2000162b24] jnz 0x0000002000162AF4 [0x2000162b2a] mov eax, [r15+r14*1] [0x2000162b32] ret - ``` For now, the disassembly is pretty basic, but it should eventually support GOAL symbols. ## Breakpoints -``` +```lisp OpenGOAL Compiler 0.1 ;; first, connect to the target @@ -247,10 +259,10 @@ gc > (dbg) [Debugger] PTRACE_ATTACHED! Waiting for process to stop... Target has stopped. Run (:di) to get more information. Read symbol table (146816 bytes, 124 reads, 123 symbols, 2.02 ms) -rax: 0x000000000000000a rcx: 0x0000000000000005 rdx: 0x0000000000000000 rbx: 0x0000002000000000 -rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 -r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 +rax: 0x000000000000000a rcx: 0x0000000000000005 rdx: 0x0000000000000000 rbx: 0x0000002000000000 +rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 +r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 rip: 0x0000002007ffbf3b [0x2007ffbf1b] add [rax], al [0x2007ffbf1d] add [rcx+0x02], bh @@ -292,10 +304,10 @@ Target has stopped. Run (:di) to get more information. ;; get some info: gcs> (:di) Read symbol table (146816 bytes, 124 reads, 123 symbols, 1.46 ms) -rax: 0x0000000000000015 rcx: 0x0000000000000007 rdx: 0x0000000000000000 rbx: 0x0000002000000000 -rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 -r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 +rax: 0x0000000000000015 rcx: 0x0000000000000007 rdx: 0x0000000000000000 rbx: 0x0000002000000000 +rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 +r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 rip: 0x0000002007ffbf4c [0x2007ffbf2c] add eax, ecx [0x2007ffbf2e] mov ecx, 0x04 @@ -334,16 +346,16 @@ gcs> (:ubp #x2007ffbf4b) ;; continue, it stays running gcs> (:cont) -gcr> +gcr> ;; break and check, the code is back to normal! gcr> (:break) Target has stopped. Run (:di) to get more information. Read symbol table (146816 bytes, 124 reads, 123 symbols, 1.28 ms) -rax: 0x0000000000000015 rcx: 0x0000000000000007 rdx: 0x0000000000000000 rbx: 0x0000002000000000 -rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 - r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 -r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 +rax: 0x0000000000000015 rcx: 0x0000000000000007 rdx: 0x0000000000000000 rbx: 0x0000002000000000 +rsp: 0x00007fddcde75c58 rbp: 0x00007fddcde75cc0 rsi: 0x0000000000000000 rdi: 0x0000000000000000 + r8: 0x0000000000147d24 r9: 0x0000002000000000 r10: 0x00007fddcde75ca0 r11: 0x0000000000000000 +r12: 0x0000000000147d24 r13: 0x0000002007ffbf14 r14: 0x0000000000147d24 r15: 0x0000002000000000 rip: 0x0000002007ffbf4b [0x2007ffbf2b] add rax, rcx [0x2007ffbf2e] mov ecx, 0x04 @@ -375,11 +387,11 @@ rip: 0x0000002007ffbf4b [0x2007ffbf86] add [rbx], al [0x2007ffbf88] add [rax], al -gcs> +gcs> ;; we can still properly exit from the target, even in this state! gcs> (e) Tried to reset a halted target, detaching... Error - target has timed out. If it is stuck in a loop, it must be manually killed. [Listener] Closed connection to target -``` \ No newline at end of file +``` diff --git a/docs/markdown/emacs_config.el b/docs/markdown/editor_setup.md similarity index 87% rename from docs/markdown/emacs_config.el rename to docs/markdown/editor_setup.md index 29c10ae5cc..2d727956cf 100644 --- a/docs/markdown/emacs_config.el +++ b/docs/markdown/editor_setup.md @@ -1,3 +1,10 @@ +# Editor Configuration + +## EMacs + +The following EMacs config file should get you started and configure OpenGOAL's formatting style + +```lisp ;; make gc files use lisp-mode (add-to-list 'auto-mode-alist '("\\.gc\\'" . lisp-mode)) ;; run setup-goal when we enter lisp mode @@ -22,3 +29,5 @@ (setq-default indent-tabs-mode nil) ) ) + +``` \ No newline at end of file diff --git a/docs/markdown/example_goal.gc b/docs/markdown/example_goal.gc deleted file mode 100644 index 2c1b2724b4..0000000000 --- a/docs/markdown/example_goal.gc +++ /dev/null @@ -1,18 +0,0 @@ -(defun factorial-iterative ((x integer)) - (let ((result 1)) - (while (!= x 1) - (set! result (* result x)) - (set! x (- x 1)) - ) - result - ) - ) - -;; until we load KERNEL.CGO automatically, we have to do this to -;; make format work correctly. -(define-extern _format function) -(define format _format) - -(let ((x 10)) - (format #t "The value of ~D factorial is ~D~%" x (factorial-iterative x)) - ) diff --git a/docs/markdown/goos.md b/docs/markdown/goos.md index bf59bce834..9570166a9d 100644 --- a/docs/markdown/goos.md +++ b/docs/markdown/goos.md @@ -1,10 +1,10 @@ -Goos ------ -GOOS is a macro language for GOAL. It is a separate language. Files written in GOAL end in `.gc` and files written in GOOS end in `.gs`. The REPL will display a `goos>` prompt for GOOS and `goal>` for GOAL. +# GOOS + +GOOS is a macro language for GOAL. It is a separate language. Files written in GOAL end in `.gc` and files written in GOOS end in `.gs`. The REPL will display a `goos>` prompt for GOOS and `gc >` for GOAL. There is a special namespace shared between GOOS and GOAL containing the names of the macros (written in GOOS) which can be used in GOAL code. -To access a GOOS REPL, run `(goos)` from the `goal>` prompt (note, currently this happens by default as GOAL is not implemented). +To access a GOOS REPL, run `(goos)` from the `gc >` prompt. This document assumes some familiarity with the Scheme programming language. It's recommended to read a bit about Scheme first. @@ -13,28 +13,34 @@ Note that most Scheme things will work in GOOS, with the following exceptions: - The short form for defining functions is `(desfun function-name (arguments) body...)` - GOOS does not have tail call optimization and prefers looping to recursion (there is a `while` form) +## Special Forms -Special Forms ---------------- Most forms in Scheme have a name, and list of arguments. Like: -``` + +```scheme (my-operation first-argument second-argument ...) ``` + Usually, each argument is evaluated, then passed to the operation, and the resulting value is returned. However, there are cases where not all arguments are evaluated. For example: -``` + +```scheme (if (want-x?) (do-x) (do-y) ) ``` + In this case, only one of `(do-x)` and `(do-y)` are executed. This doesn't follow the pattern of "evaluate all arguments...", so it is a *SPECIAL FORM*. It's not possible for a function call to be a special form - GOOS will automatically evaluate all arguments. It is possible to build macros which act like special forms. There are some special forms built-in to the GOOS interpreter, which are documented in this section. ### define + This is used to define a value in the current lexical environment. For example: -``` + +```scheme (define x 10) ``` + will define `x` as a variable equal to `10` in the inner-most lexical environment. (Note, I'm not sure this is actually how Scheme works) There is an optional keyword argument to pick the environment for definition, but this is used rarely. The only named environments are: @@ -42,15 +48,17 @@ There is an optional keyword argument to pick the environment for definition, bu - `*global-env*` Example: -``` + +```scheme (define :env *global-env* x 10) ``` -will define `x` in the global (outer-most) environment, regardless of where the `define` is written. +will define `x` in the global (outer-most) environment, regardless of where the `define` is written. ### quote This form simply returns its argument without evaluating it. There's a reader shortcut for this: -``` + +```scheme (quote x) ;; reader shortcut @@ -58,7 +66,8 @@ This form simply returns its argument without evaluating it. There's a reader s ``` It's often used to get a symbol, but you can quote complex things like lists, pairs, and arrays. -``` + +```scheme goos> (cdr '(1 . 2)) 2 goos> (cdr '(1 2)) @@ -69,16 +78,19 @@ goos> '#(1 2 3) ### set! Set is used to search for a variable in the enclosing environments, then set it to a value. -``` + +```scheme (set! x (+ 1 2)) ``` + will set the lexically closest `x` to 3. It's an error if there's no variable named `x` in an enclosing scope. ### lambda See any Lisp/Scheme tutorial for a good explanation of `lambda`. Note that GOOS has some extensions for arguments. You can have a "rest" argument at the end, like this: -``` + +```scheme (lambda (a b &rest c) ...) ;; c is the rest arg (lambda (&rest a) ...) ;; a is the rest ``` @@ -86,13 +98,15 @@ Note that GOOS has some extensions for arguments. You can have a "rest" argumen The rest argument will contain a list of all extra arguments passed to the function. If there are no extra arguments, the rest argument will be the empty list. There are also keyword arguments, which contain a `&key` before the argument name. -``` + +```scheme (lambda (a b &key c) ...) ;; b is a keyword argument, a and c are not. (lambda (&key a &key b) ...) ;; a and b are keyword arguments ``` These keyword arguments _must_ be specified by name. So to call the two functions above: -``` + +```scheme (f 1 2 :c 3) ;; a = 1, b = 2, c = 3 (f :a 1 : b 2) ;; a = 1, b = 2 ``` @@ -100,7 +114,8 @@ These keyword arguments _must_ be specified by name. So to call the two function Note that it is not required to put keyword arguments last, but it is a good idea to do it for clarity. There are also keyword default arguments, which are like keyword arguments, but can be omitted from the call. In this case a default value is used instead. -``` + +```scheme (lambda (&key (c 12)) ...) (f :c 2) ;; c = 2 (f) ;; c = 12 @@ -123,7 +138,7 @@ Short circuiting `or`. If nothing is truthy, `#f`. Otherwise returns first truth Short circuiting `or`. If not all truthy, `#f`. Otherwise returns last truthy. ### macro -Kind of like `lambda`, but for a macro. +Kind of like `lambda`, but for a macro. A lambda: - Evaluate the arguments @@ -149,7 +164,8 @@ Special while loop form for GOOS. `(while condition body...)` To add together `[0, 100)`: -``` + +```scheme (define count 0) (define sum 0) @@ -161,16 +177,18 @@ To add together `[0, 100)`: sum ``` -Not Special Built-In Forms ---------------------------- +## Not Special Built-In Forms + +TODO - None at this time -Namespace Details ------------------- +## Namespace Details + The GOOS `define` form accepts an environment for definition. For example: -``` + +```scheme (define :env *goal-env* x 10) ``` + will define `x` in the `*goal-env*`. Any macros defined in the `*goal-env*` can be used as macros from within GOAL code. Things that aren't macros in the `*goal-env*` cannot be accessed. - diff --git a/docs/markdown/lib.md b/docs/markdown/lib.md new file mode 100644 index 0000000000..3b8acb2cd0 --- /dev/null +++ b/docs/markdown/lib.md @@ -0,0 +1,1235 @@ +# Standard Library + +## Runtime Functions + +### `string->symbol` + +### `symbol->string` + +### `format` + +GOAL's `printf`. + +```lisp +(format destination format-string args...) +``` + +The following destinations are currently supported: +- `#t` - print to the listener. After the current game frame is over, this will be sent to the compiler and you can see it at the GOAL prompt. +- `0` - print to `stdout` in the runtime immediately. This won't be visible from within the compiler - you must look at the runtime to see it. This is useful for debugging if the runtime crashes before flushing the normal print buffer. +- `#f` - print to a new string allocated on the global heap. +- a GOAL `string` object - append to an existing string +- a GOAL `file-stream` object - currently unsupported but eventually may allow printing into a file somewhere + +The global variable `*print-column*` can be used to automatically print at a certain indentation. The very first thing printed during a frame will not have the indentation applied. + +The format string escape sequences all start with `~`, then have arguments (possibly none), then have a single character code. The arguments look like: +- `~A`, the `A` code with no arguments +- `~12A`, the `A` code with an integer argument of `12` +- `~'zA`, the `A` code with a character argument of `z` +- ``` ~`hello`A```, the `A` code with a string argument of `"hello"` +- `~12,'bA`, the `A` code with two arguments, integer `12` and character `b` + + +The escape codes are: + +#### `~%` + +Replace with a newline character. + +#### `~~` + +Replace with a `~` character + +#### `~G` or `~g` + +Replace with a C-style string given in `args`. Note that this will not work on a GOAL string. + +#### `~A` or `~a` + +Print a boxed object given in `args`. Uses the `print` method. Can take optional arguments for length and padding character. If the `print` method gives something shorter than the length argument, it will be padded on the left with the padding character (default is space). If the `print` method gives something too long, it will be truncated on the right. The last character printed will be changed to `~` to indicate it was truncated here. + +There is an option to left justify instead but I don't think there's a way to turn it on. + +#### `~S` or `~s` + +Very similar to `~A`, but a GOAL string will be printed without quotes around it. + +#### `~C` or `~c` + +Print a single character (GOAL `uint8` or `int8` type) + +#### `~P` or `~p` + +Print an object, similar to `~A`, but does not have optional arguments for length or padding. Instead, there is an optional argument to give a type name, then use that types `print` method. This is useful for printing non-boxed objects. If no type name is given, behaves like `~A`. + +#### `~I` or `~i` + +Like `~P`, but uses the inspect method instead. + +#### `~Q` or `~q` + +Likely was supposed to be for printing 128-bit integers stored in registers, but will always prints `0`. The `format` function is written in C using varargs, which doesn't support 128-bit register values. + +#### `~B` or `~b` + +Print integer as binary. Optional arguments for pad-length and pad-character (default is 0). Won't truncate. + +#### `~D` or `~d` + +Print integer as decimal (signed). Optional arguments for pad-length and pad-character (default is space). Won't truncate. + +#### `~X` or `~x` + +Print integer as hex. Optional arguments for pad-length and pad-character (default is 0). Won't truncate. + +#### `~F` + +Print floating point. Will print 4 digits after the decimal and pad with space to a width of 12 and does not accept any options. + +#### `~f` + +Print floating point. Takes optional arguments for pad-length (default don't pad), pad-character (default space), and precision (default 4). + +#### `~R` or `~r` + +Like `~f` but scales in-game rotation units to degrees. The float `65536.0` is 360 degrees. + +#### `~M` or `~m` + +Like `~f` but scales in-game distance units to meters. The float `4096.0` is 1 meter. + +#### `~E` or `~e` + +Like `~f` for argument, but takes an integer time-code as an input and scales time-code units to seconds. There are 300 ticks / second, which is the smallest number which has an integer number of ticks per NTSC and PAL frame. Something very weird happens when the input is negative? + +#### `~T` or `~t` + +Insert a tab character. + +#### Pass Through Codes + +Some codes will be passed through automatically, along with any arguments. This will also pass through arguments of `+` or `-`. I believe that these options could later be interpreted by the code for printing on-screen characters. + +The pass through codes are `H,J,K,L,N,V,W,Y,Z,h,j,k,l,n,v,w,y,z`. + +Any other codes will result in an error message being printed. + +## Block Forms + +### `begin` + +Execute forms in order. + +```lisp +(begin form...) +``` + +A `begin` form is just a list of other forms which are executed in order. A `begin` form evaluates to the value of the last form. + +Example: + +```lisp +(begin + (print "hello ") + (print "world!") + 7 + ) +``` + +will print `hello world!` and the value of the entire form is `7`. + +The `begin` form is used a lot in macros, but not that much in code. It's generally used when you want to execute multiple things, but fit it into a single form. + +### `block` + +A `block` form is pretty similar to a `begin`, except the `block` has a name. You can then "return" from the block early with `return-from`. + +```lisp +(block block-name form...) +``` + +Example: + +```lisp +(block my-block + (print "hello ") + (return-from my-block 7) + (print "world") + "test" + ) +``` + +will print `hello ` only and the value of the entire `block` form is `7`. The type of the `block` is the most specific type that describes all of the possible return values from any `return-from` or from reaching the end (even if its technically not possible to reach the end). In the case above, the possible return types are `int` and `string`, so the return type of the whole block is `object`, the lowest common ancestor type of `int` and `string`. + +Block is used rarely, and possibly almost never? + +### `return-from` + +Exit a `block` or function early. + +```lisp +(return-from block-name value) +``` + +Looks up the block and exits from it with the value. You can exit out nested blocks. If you are enclosed in multiple blocks with the same name, exits from the inner-most one with a matching name. Everything in a function is wrapped in a block named `#f`, so you can use `(return-from #f x)` to return early from a function with `x`. When using this form, it may change the return type of the function or block. The return type will be the lowest common ancestor type of all written return paths. If there is an unreachable return path, it will still be considered. + +Example + +```lisp +(if (is-a-match? x) + (return-from #f x) + ) +``` + +if `x` is a match, returns `x` from the function (not shown) immediately. + +The `return-from` form is very rarely used to return from a block, but sometimes used to return from a function. + +### `return` + +Exit a function early. + +```lisp +(return value) +``` + +Has the same behavior as `(return-from #f value)`. + +### `label` + +Create a named label for `goto` or `goto-when`. + +```lisp +(label label-name) +``` + +The label spaces are per-function and not nested. You can't jump from function to function. You can't jump in or out of functions which end up getting inlined. You can't jump in or out of an anonymous lambda function. You can jump in and out of `let`s. + +See `goto` for an example. + +Labels are used extremely rarely. Usually only in inline assembly and part of macros for `while` loops and similar. + +### `goto` + +Jump to a label. + +```lisp +(goto label-name) +``` + +The label must be in the current label space. You can jump forward or backward. + +Example: +```lisp +(if skip-code? + (goto end) + ) + +;; code here runs only if skip-code is false. + +(label end) +``` + +The `goto` form used very rarely outside of macros and inline assembly. Try to avoid using `goto`. + +### `top-level` + +This form is reserved by the compiler. Internally all forms in a file are grouped under a `top-level` form, so you may see this in error messages. Do not name anything `top-level`. + +## Control Flow Forms + +### GOAL "Two Element" Conditions + +These are `!=`, `eq?`, `neq?`, `=`, `>`, `<`, `>=`, `<=`. The default is to compare the two objects as unsigned 64-bit integers, unless a special case is hit. The special case is determined by the type of the __first__ argument: +- Floating point: if second argument is a number, convert to floating point. Use floating point comparisons +- Binteger: convert first argument to integer. If second argument is a number, convert to integer. Use signed integer comparisons +- Integer: If second argument is a number, convert to integer. Use signed integer comparison +- Unsigned Integer: If second argument is a number, convert to integer. Use unsigned integer comparison + +Note that comparing structures just checks if they refer to the same memory, not if the contents are the same. + +Currently there is absolutely no type checking done on comparisons, which makes it quite easy to get things wrong. + +When a two-element condition is used as a condition for an `if`, `cond`, `while`, `goto-when`, or any other branching condition, there is an optimization that avoids computing a GOAL boolean, then checking that GOAL boolean as a branch condition. Instead the comparison operation directly sets the x86-64 flags for a jump. Original GOAL has a similar optimization. + +There are a lot of unknown details +- What order are things evaluated / converted? +- How does the order work when there's a `not`? +- Type Checking is probably needed. +- Currently `=` and `eq?` are exactly the same. Likely `=` should only work on numbers and `eq?` should work on references? + +### `not` + +```lisp +(not value) +``` + +If value is `#f`, return `#t`. Otherwise return `#f`. + +Using `not` on a "two element condition" will be optimized to modifying that condition. + +### `when-goto` + +```lisp +(when-goto value-or-condition destination) +``` + +Examples: + +```lisp +(when-goto (> x y) some-label) +(when-goto (is-player-dead?) some-label) +``` + +Jump to `destination` if the condition is truthy (not `#f`). This ends up generating much better code than `(if condition (goto x))`. + +Like normal `goto`, this isn't used much outside of macros. + +### `cond` + +A normal Lisp/Scheme `cond`. + +```lisp +(cond (test clause...)...) +``` + +Evaluates `test`s until one is truthy. Then evaluates all `clause`s in that case and returns the value of the last one. Optionally there can be a case with `else` as the test which runs if no other cases match. This must be the last case. + +If there is no `else`, and no cases match, return `#f`. (is it always `#f`?) + +The return type is the lowest common ancestor type of all cases. Note that for a `cond` without `else`, the possible return value of `#f` is __not__ considered when determining the return type. So: + +```lisp +(print-type + (cond (test1 "hi") (test2 "bye")) + ) +``` + +will print `[TYPE] string`, even though `cond` may return `#f`. + +This behavior seems correct if the return value is a `basic`, but less clear if the return value is supposed to be a number. + +Note that GOAL will load `#f` into a register for an else case even if the value of the `cond` is never used. Also, even if the value of the `cond` is never used, all cases will store their result into a common `cond` return-register. This is incredibly helpful for the decompiler! + +### `if` + +A normal Lisp/Scheme `if`. + +```lisp +(if test true-case [false-case]) +``` + +The value of the entire statement is the value of the taken case. The false case can be left out. If the condition evaluates to false and the false case is left out, `#f` is returned. The return type follows the same logic as `cond`, as `if` is a macro using `cond`. + +### `when` + +A normal Lisp/Scheme `when`. + +```lisp +(when test forms...) +``` + +If `test` is truthy, evaluate all forms and return the value of the last one. If `test` isn't true, return `#f`. + +### `unless` + +A normal Lisp/Scheme `unless`. + +```lisp +(unless test forms...) +``` + +If `test` is false, evaluate all forms and return the value of the last one. If `test` isn't false, return `#f`. + +## Definition Forms + +### `set!` + +Set a value! + +```lisp +(set! place value) +``` + +Sets `place` to `value`. The `place` can be once of: +- A lexical variable defined by `let`, or an argument of a function +- A global symbol +- A `car` or `cdr` of a pair +- A field of an object +- A dereferenced pointer +- An element in an array +- A bitfield within any of the above (not yet implemented) + +### `define` + +Define a global symbol + +```lisp +(define symbol-name value) +``` + +Kind of like `set!`, but works exclusively on global symbols. Also sets the type of the global symbol. If the global symbol already has a type, and the `define` tries to change it, there will be an error or warning. (to be determined) + +### `define-extern` + +Inform the compiler about the type of a global symbol + +```lisp +(define-extern symbol-name type) +``` + +Useful to forward declare functions or to let the compiler know about things in the C Kernel. See `goal-externs.gc` for where all the C Kernel stuff is declared. + +### `let` + +```lisp +(let ((var-name value)...) body...) +``` + +Define local variables. The variables are automatically assigned a type, like C++ `auto`. + +Example: + +```lisp +(let ((count-1 (+ a b)) ;; count_1 = a + b + (count-2 (+ c d))) ;; count_2 = c + d + (if (count > 10) + ;; print a message if count > 10. + (format #t "count = ~D, ~D~%" count-1 count-2) + ) + ) +``` + +Note, if you define multiple variables in a `let`, you cannot refer to variables defined previously in the same `let`. In the example above, `count-2`'s value couldn't be defined in terms of `count-1` for example. See `let*`. + +### `let*` + +```lisp +(let ((var-name value)...) body...) +``` + +Define local variables. If you define multiple in a single `let*`, you can refer to previous variables. + +Example: + +```lisp +(let* ((count-1 (+ a b)) + (count-2 (+ count-1 d))) ;; okay because we used let* + (if (count > 10) + ;; print a message if count > 10. + (format #t "count = ~D, ~D~%" count-1 count-2) + ) + ) +``` + +## Functions + +### `defun` + +Define a new named global function! + +```lisp +(defun name arg-list ["doc-string"] body...) +arg-list = (arg...) +arg = (arg-name arg-type)|arg-name +``` + +Example: + +```lisp +(defun 1/ ((x float)) + "Compute 1.0 / x" + (/ 1.0 x) + ) +``` + +Define a new function with the given name. Note that until the `defun` itself executes, the function __cannot be used!__. So don't call functions before the `defun`. This is unlike C, where you can forward declare a function and use it before the actual definition. + +There is an optional docstring. Currently the docstring is just thrown away but in the future we could save them and and generate documentation or something. + +### `defmethod` + +Define a method! + +```lisp +(defmethod method-name type-name arg-list ["doc-string"] body...) +``` + +In many ways is similar to `defun`. See section on methods for more details. + +You can only `defmethod` if one of these two is true: +- You are overriding a parent method +- You have declared this method in the type's `deftype` + +Like `defun`, the method only exists after the `defmethod` executes. Before that the method may be undefined or the parent's method. + +### `inline` + +Make this function call inlined. + +```lisp +(inline function-name) +``` + +Example: inline call `my-function` + +```lisp +((inline my-function) my-arg) +``` + +Attempts to make the function call inline. If it is not possible, it will throw an error. You can't save the value of `(inline my-function)`, only use it immediately in a function call. You must provide a name of a global function as the function, not a function pointer. + +Methods cannot be inlined. (Maybe new methods can be in original GOAL?) + +### `lambda` + +Create a GOAL Lambda. + +```lisp +(lambda [:name name] [:inline-only #f|#t] [:segment main|debug] arg-list body-form..) +arg-list = (arg...) +arg = (arg-name arg-type)|arg-name +``` + +The `:name` option is only used for debugging the compiler - it cannot be accessed or referenced outside of internal compiler debugging features. It should be ignored unless you are debugging the compiler. + +The `:inline-only` flag defaults to `#f`, but can be set to `#t` to force the compiler to not generate x86-64 code for the function. In this case, the lambda cannot be used as a function pointer/GOAL `function` object. This is used in the `let` macro and not really useful for general programming. If the flag is `#f` (default value), it will generate and store an x86-64 function, regardless of if the function is ever called. + +The `:segment` option defaults to `main`, unless the `lambda` is defined in a function in the `debug` segment, in which case it defaults to `debug`. It can be overridden by this flag. This picks the segment of the object file where the function will be stored. + +The arguments default to type of `object` if no type is provided. In the case where a lambda is used immediately (a `let`) the compiler will propagate more specific types. This is why you don't have to give types when using `let`. + +A GOAL lambda has three purposes: +1. Generate a function as a "real" x86-64 function. +2. Save the code so the function can later be inlined +3. Immediately use the lambda in the head of a list as part of a `let` or `let*` macro + +__Example of 1:__ +``` +(sort my-list (lambda ((x int) (y int) (< x y)))) +``` +We create a real x86-64 function to pass to the `sort` function. In this case you should specify types - the default of `object` is not suitable. + +The other two cases are handled by `let` and `defun` macros, and shouldn't show up in other places, so they are left out. + +### `declare` + +Set options for a function or method + +```lisp +(declare [(inline)] [(allow-inline)] [(disallow-inline)] [(asm-func return-typespec)] [(print-asm)]) +``` + +If used, this should be the first thing inside of a `defun`/`defmethod`. Don't use it anywhere else. +Example: + +```lisp +(defun my-function () + (declare (inline)) + ... + ) +``` + +- `inline` means "inline whenever possible". See function inlining section for why inlining may be impossible in some cases. +- `allow-inline` or `disallow-inline`. You can control if inlining is allowed, though it is not clear why I thought this would be useful. Currently the default is to allow always. +- `print-asm` if codegen runs on this function (`:color #t`), disassemble the result and print it. This is intended for compiler debugging. +- `asm-func` will disable the prologue and epilogue from being generated. You need to include your own `ret` instruction or similar. The compiler will error if it needs to use the stack for a stack variable or a spilled register. The coloring system will not use callee saved registers and will error if you force it to use one. As a result, complicated GOAL expression may fail inside an `asm-func` function. The intent is to use it for context switching routines inside in the kernel, where you may not be able to use the stack, or may not want to return with `ret`. The return type of an `asm-func` must manually be specified as the compiler doesn't automatically put the result in the return register and cannot do type analysis to figure out the real return type. +- `allow-saved-regs` allows an `asm-func`'s coloring to use saved registers without an error. Stacks spills are still an error. The compiler will not automatically put things in a saved register, you must do this yourself. The move eliminator may still be used on your variables which use saved registers, so you should be careful if you really care about where saved variables are used. + +This form will probably get more options in the future. + +### `local-vars` + +Declare variables local to a function, without an initial value. This will be used by the decompiler before `let` has been fully implemented. + +```lisp +(local-vars (name type-spec)...) +``` + +The name can be any valid symbol. The scope of the variable is _always_ the function scope. Other scopes inside a function will always hide variables declared with `local-vars`. The type can be any GOAL typespec. If you use `float`, you get a floating point register, otherwise you get a normal GPR. + +It's recommended to avoid using this form. + +## Macro Forms + +### `#cond` + +This is like an `#ifdef`, but the language for the conditions is GOOS and it's like a `cond`. + +```lisp +(#cond (test clause...)...) +``` + +The `#cond` is executed at compile time. The cases which don't match aren't compiled. There are `#when` and `#unless` macros that work like you would expect. + +Example: + +```lisp +(#when PRINT_DEBUG_INFO + (format #t "debug info~%") + ) +``` + +### `quote` / `'` + +The reader will expand `'thing` to `(quote thing)` + +```lisp +(quote thing) +``` + +Currently `quote` supports: +- `'a-symbol` - will return the symbol `a-symbol` +- `'()` - will return the empty list. + +In the future, it should support quoted lists of static data. + +### `defglobalconstant` + +Define a GOOS and GOAL constant. + +```lisp +(defconstant name value) +``` + +The constant is available in both GOOS and GOAL. You can use it in `#cond` expressions (GOOS) and in GOAL code. The constant inserts the `value` as an s-expression in the code. This means it's a "code" constant, and doesn't have a type. + +There are a few restrictions: +- Currently constants do not work in `deftype`. (this should be fixed eventually) +- Constants may not work in places where the compiler is expecting a symbol which isn't actually compiled. So `(+ MY_CONSTANT 2)` is fine, but `(defun MY_CONSTANT () ...)` isn't. Don't use constants for type names, function names, or symbol names. It is fine to have a constant with a value of a symbol, but don't `(define MY_CONSTANT)` or `(set! MY_CONSTANT)` or `(new MY_CONSTANT)` or things like that. +- Don't use constants with `method` form. +- You can modify constants created with `defglobalconstant` in GOOS (effect won't be seen in GOAL) with `(set!)` +- There is no warning if one `defglobalconstant` changes the value of an existing `defglobalconstant` + +In general constants should have `UPPERCASE` names otherwise things get very confusing when there are name conflicts. + +The recommendation is to use constants for things like numbers or expressions like `(* NUM_THINGS SIZE_OF_EACH_THING)`. + +### `mlet` + +Scoped constants in GOAL. Syntax is like a `let`. This feature has all the restrictions of `defglobalconstant`. +Avoid using `mlet` because it's confusing and not useful. + +```lisp +(mlet ((constant-name constant-value)...) + body... + ) +``` + +Example: + +```lisp +(mlet ((NUM-THINGS 12) + (THING-NAME "test")) + ; in here, NUM-THING and THING-NAME are constants + ) +``` + +## Math Forms + +Math forms will look at the type of the first argument to determine the "mode". So if you have `(+ 1 1.2)`, it will convert the `1.2` to an integer, do the add, and return an integer. + +### `+` + +Addition. Can take 1 or more arguments. `(+ 1)` will give you `1`, like you'd expect. + +```lisp +(+ form...) +``` + +Works on integers and floats. Integer add is 64-bit and wrapping. + +### `-` + +Subtraction or negative. Can take 1 or more arguments. + +```lisp +(- form...) +``` + +`(- 1)` gives you `-1`, but `(- 1 3)` gives you `-2`, which may be slightly confusing at first. Works on integers and floats. Integer subtract is 64-bit and wrapping. + +### `*` + +Multiplication. Can take 1 or more arguments. + +```lisp +(* form...) +``` + +`(* 7)` will become `7`, as you'd expect. Works on integers and floats. The exact behavior for integers needs to be explored more because the EE is weird. GOAL generates different code for multiplication of signed vs. unsigned integers. + +### `/` + +Division. Takes exactly two arguments + +```lisp +(/ dividend divisor) +``` + +Works on floats and integers. The exact behavior for integers needs to be explored more because the EE is weird. GOAL generates different code for division of signed vs. unsigned integers. + +### `mod` + +Modulo operation. Takes exactly two arguments. + +```lisp +(/ dividend divisor) +``` + +Works on integers only. The exact behavior needs to be explored because the EE is weird. It is unknown if the same code is generate for signed/unsigned mod. + +### `slhv`, `sarv`, `shrv` + +```lisp +(shlv value shift-amount) +``` + +The exact behavior of GOAL shifts isn't fully understood, so these are temporary wrappers around x86-64 variable shift instructions. +- `shlv` shift left variable +- `sarv` shift arithmetic right variable +- `shrv` shift logical right variable +64-bit operation. + +### `logand` + +Bitwise And + +```lisp +(logand a b) +``` + +64-bit operation. + +### `logior` + +Bitwise Inclusive Or (normal Or) + +```lisp +(logior a b) +``` + +64-bit operation. + +### `logxor` + +Bitwise Exclusive Or (Xor + +```lisp +(logxor a b) +``` + +64-bit operation. + +### `lognot` + +Bitwise Not + +```lisp +(lognot a) +``` + +64-bit operation. + +## Type Forms + +### `defmethod` + +### `deftype` + +### `method-of-object` + +Get a method of an object. + +```lisp +(method-of-object object method-name) +``` + +This form takes an object and gets the method from it. If the object has runtime type information, will consult the method table at runtime to get a possibly more specific method than what is available at compile time. This uses the same lookup logic as method calling - see the section on method calls for more information. + +### `method-of-type` + +Get a method of a type or an object. + +```lisp +(method-of-type type method-name) +``` + +The first form of this takes a type name and method name and returns a GOAL `function` for this method. For example: + +```lisp +(method string inspect) +``` + +will return the `inspect` method of `string`. + +### `car` and `cdr` + +Get element from pair + +```lisp +(car some-pair) +(cdr some-pair) +``` + +The type of the result is always `object`, as pairs can hold any `object`. The type-check for `car` and `cdr` is relaxed - it allows it to be applied to any `pair` or `object`. The reason for allowing `object` is so you can write `(car (car x))` instead of `(car (the pair (car x)))`. However, if the argument to `car` is not a `pair`, you will get garbage or a crash. + +### `new` + +```lisp +(new [allocation] [new-type-specification] [args]) +``` + +See section on creating new GOAL objects. + +### `print-type` + +Print the type of some GOAL expression at compile time. + +```lisp +(print-type form) +``` + +This is mainly used to debug the compiler or figure out why some code is failing a type check. The thing inside is compiled fully and used as the result of `print-type`. Example: + +```lisp +(print-type "apples") ;; [TYPE] string +(print-type (+ 12 1.2)) ;; [TYPE] int +(print-type (the float 12)) ;; [TYPE] float +``` + +### `the` + +Convert between types, doing the expected thing for number conversions. + +```lisp +(the type thing) +``` + +If the `type` and the `thing` are both numbers, it will automatically convert between the different numeric types as needed. In all other cases, it does a dangerous `reinterpret_cast`. cppreference.com explains this idea clearly with "Converts between types by reinterpreting the underlying bit pattern." + +If the `thing` is a number: +- If `type` is `binteger`, convert the number to a `binteger` +- If `type` is `int` or `uint` (or some user-defined child type of these), convert the number to an integer. +- If `type` is `float`, convert the number to a `float` + +In all other cases, directly use the 64-bit value in the register as the value of the desired `type`. + +Example of number conversions: + +```lisp +(the binteger 12) ;; boxed integer for 12 +(the float 1) ;; floating point 1.0 +(the int 1.234) ;; signed integer 1 +``` + +Examples of common casts: + +```lisp +(the uint 1) ;; unsigned integer, doesn't do any modification if input is already an int. +(the (pointer uint8) x) ;; like C (uint8_t*)x +(the string (car x)) ;; if you know (car x) is a string, do this. +``` + +Examples of dangerous things you can do but probably shouldn't: + +```lisp +(the string 1.234) ;; take the binary representation of 1.234 and treat it as a GOAL memory address to string data. +(the float 'bean) ;; take the GOAL memory address of the "bean" symbol and treat it as the binary representation of a float. +``` + +There are some weird edge cases with `the` that are worth mentioning, if you try to set the value of something you've used `the` on. In the future the compiler should block you from doing something bad, but this is not yet implemented. + +```lisp +(set! (-> (the (pointer uint8) x)) 1) ;; OK, not casting the actual value +(set! (the int (-> obj x)) 1) ;; NOT OK. Int is a value type +;; Will actually work if x is already an int/uint, and with non-numeric values types like pointer. Avoid just to be safe. +(set! (the string (-> obj x)) "test") ;; OK, string is a basic, which is a reference type. +``` + +This becomes more clear if we look at the C equivalent: + +```lisp +*(uint8_t*)(a->x) = 1; // makes sense + +(int)(a->x) = 1; // doesn't make sense + +*(thing*)(a->x) = thing(); // makes sense +``` + +### `the-as` + +Convert between types always using `reinterpret_cast` + +```lisp +(the-as type thing) +``` + +cppreference.com explains this idea clearly with "Converts between types by reinterpreting the underlying bit pattern." + +The recommendation is to prefer `the` in almost all cases, unless you want to set an exact bit pattern of a `float`, or want to examine the bits in a `float`. + +Example: + +```lisp +(the-as int 1.234) ;; result happens to be 1067316150 +``` + +None of the edge cases of `the` apply to `the-as`. + +### `size-of` + +Get the size of a type, in bytes. + +```lisp +(size-of ) +``` + +Get the size of a type, by name. The type must be a plain name, like `pointer` or `dma-bucket`. Compound types are not supported. It works on value and structure types and returns the size in memory. For dynamic types, it returns the size as if the dynamic part has 0 size. For weird types like `none` it throws an error. + +This value can be used in most places where the compiler is expecting a constant integer as well, such as the size of a stack array, which must be known at compile time. + +Example: + +```lisp +(size-of dma-bucket) +``` + +Returns the size of the `dma-bucket` type which is 16 bytes. + +### Pointer Math + +Not implemented well yet. + +## Inline Assembly Forms + +In general, assembly forms have a name that begins with a `.`. They all evaluate to `none` and copy the form of an x86-64 instruction. For example `(.sub dst src)`. A destination must be a settable register (ok if it's spilled). So you can't do something like `(.sub (-> obj field) x)`. Instead, do `(set! temp (-> obj field))`, `(.sub temp x)`, `(set! (-> obj field) temp)`. The sources can be any expression, or a register. This allows you to mix high-level code with assembly easily, like `(.mov rax (-> obj field))` or `(.push (+ 1 (-> obj field)))`. + +By default, assembly forms work with the coloring system. This means that assembly and high level expression can be mixed together without clobbering each other. It also means use of callee-saved registers will cause them to be backed up/restored in the function prologue and epilogue. Use of weird registers like `r15`, `r14`, and `rsp` works as you would expect with the coloring system. + +But you can also request to skip this with `:color #f` option, like `(.push my-reg-var :color #f)`. Be very careful with this. The `:color #f` option will only work with register variables from `rlet` which have a manually specified register. It will entirely bypass the coloring system and use this register. Use of this near high level GOAL variables is extremely dangerous and should be done very carefully or avoided, as the GOAL compiler will not know that you could be modifying its registers. In a form with `:color #f`, you cannot use higher level code or variables - all variables must be defined in `rlet`s. This is because higher level expressions and variables cannot be used without the coloring system. + +### `rlet` + +```lisp +(rlet ((var-name [:reg reg-name] [:class reg-class] [:type typespec] [:reset-here #t|#f])...) + body... + ) +``` + +Create register variables. You can optionally specify a register with the `:reg` option and a register name like `rax` or `xmm3`. The initial value of the register is not set. If you don't specify a register, a GPR will be chosen for you by the coloring system and it will behave like a `let`. If you don't specify a register, you can specify a register class (`gpr`, a normal 64-bit integer register; `fpr`, a 32-bit single precision float; or `vf`, and 128-bit floating point vector register) and the compiler will pick a GPR or XMM for you. + +If you pick a callee-saved register and use it within the coloring system, the compiler will back it up for you in the prologue and restore it in the epilogue. +If you pick a special register like `rsp`, it won't be backed up. + +Inside the `rlet`, all uses of `var-name` will always be in the given register. If the variable goes dead (or is never live), the compiler may reuse the register as it wants. The compiler may also spill the variable onto the stack. Of course, if you are in an `asm-func`, the stack will never be used. Be extremely careful about using "normal" registers without the coloring system and with higher-level code as the compiler may use your "normal" register as a temporary. If you read the value of a register and use the coloring system, the variable will then be alive starting at the beginning of the function, and will make that register unavailable to the compiler and other `rlet`s that occur before. This is useful to preserve the value of a temporary register if needed, but can also be undesirable in other cases. If you add the `:reset-here #t` flag, it will make the variable dead until the start of the `rlet`. It "resets" the value of the register in the coloring system at the start of the `rlet`. The default value is false. It is recommended to keep the default value when accessing specific registers that are also normally used by the compiler. For special registers like `rsp`, `r15`, `r14`, and `r13`, if you plan to use them with the coloring system, it is recommended to set the `reset-here` flag. + +Here is an example of using an `rlet` to access registers: + +```lisp +(defun get-goal-rsp-2 () + "Get the stack pointer as a GOAL pointer" + (rlet ((rsp :reg rsp :type uint) + (off :reg r15 :type uint)) + (the pointer (- rsp off)) + ) + ) +``` + +### `.sub` + +```lisp +(.sub dest src [:color #t|#f]) +``` + +x86-64 subtraction (64-bit). If coloring is on (the default), the `dest` must be a settable register (`rlet` var, `let` var, function argument, ...). It can't be a place like a symbol, field, stack variable, etc. If coloring is off, both `src` and `dest` must be registers defined and constrained in an enclosing `rlet`. + +Example: + +```lisp +(defun get-goal-rsp () + (declare (asm-func uint)) + (rlet ((rsp :reg rsp :type uint) + (off :reg r15 :type uint) + (ret :reg rax :type uint) + ) + + ;; mov rax, rsp + (set! ret rsp) + ;; sub rax, r15 + (.sub ret off) + ;; ret + (.ret) + ) + ) +``` + +### `.add` + +```lisp +(.add dest src [:color #t|#f]) +``` + +Addition (64-bit). Similar to subtraction. + +### `.jr` + +```lisp +(.jr addres-reg [:color #t|#f]) +``` + +Jump-register. Jumps to the address given. The address is treated as a 64-bit pointer, not a GOAL pointer. + +### `.load-sym` + +```lisp +(.load-sym dest symbol-name [:sext #t|#f] [:color #t|#f]) +``` + +Load the value of a symbol into a register. By default, it will look at the type of the symbol to determine if it should be sign extended or not. You can override this with the `:sext` option if needed. The symbol must be known to the type system. + +### `.push` + +```lisp +(.push src [:color #t|#f]) +``` + +The x86-64 push instruction. Does a 64-bit GPR. The `src` can be any expression that can be put in a gpr if color is on. Otherwise it must be a register defined and constrained in an enclosing `rlet`. + +### `.pop` + +```lisp +(.pop dst [:color #t|#f]) +``` + +The x86-64 pop instruction. Does a 64-bit GPR. The `dst` can be any expression which evaluates to a settable register if color is on. Otherwise it must be a register defined and constrained in an enclosing `rlet`. + +### `.ret` + +```lisp +(.ret [:color #t|#f]) +``` + +The x86-64 ret instruction. The color option does nothing. This is not recognized as a control flow instruction by the coloring system. It does not touch the return register `rax`. + +### `.mov` + +```lisp +(.mov dst src [:color #t|#f]) +``` + +Move between two registers. The `dst` should be a register (either `rlet` or `let` variable), and the `src` can be a register or any expression. The following moves are supported: +- `gpr` to `gpr` +- `fpr` to `fpr` (only moves lower 32-bits of the xmms, uses `movss`) +- `vf` to `vf` (moves all 128-bits of the xmms, uses `vmovaps`) +- `gpr` to `fpr` (only moves 32-bits, uses `movd`) +- `fpr` to `gpr` (only moves 32-bits, upper 32-bits are zero, uses `movd`) +This code generation is identical to using a `(set! dst src)` form. + +### `.nop.vf` + +```lisp +(.nop.vf) +``` + +Inserts a `FNOP` assembly instruction, which is fundamentally the same as a `NOP`. It is a 2-byte instruction. + +### `.nop` or `(nop!)` + +```lisp +(.nop) +;; or +(nop!) +``` + +Inserts a single-byte `nop`. + +### `.wait.vf` + +```lisp +(.wait.vf) +``` + +Inserts a `FWAIT` assembly instruction, x86 does not require as much synchronization as the PS2's VU registers did, but it has a purpose in rare cases. It is a 2-byte instruction. + +### `.lvf` + +```lisp +(.lvf dst-reg src-loc [:color #t|#f] [:offset ]) +``` + +Load a vector float register from `src-loc`. The `dst-reg` must be a vector float register. The `src-loc` can be a gpr containing a GOAL pointer or expression which gives a GOAL pointer. There is no type checking on the `src-loc` so be careful. The load uses `vmovaps`, so the source must be 16-byte aligned. + +If the source is in the form `base-reg + constant-offset`, like from a `(&-> my-object my-inline-vector-field)`, the constant offset will be folded into the load instruction like `vmovaps xmm1, [r15 + rax + 12]`. An explicit offset can be provided via the `:offset` keyword, and will be used if applicable. + +If the source is an immediate `(new 'static ...)` form that results in a statically allocated variable, it will use `RIP` relative addressing (32-bit immediate) form. This means that the code: + +```lisp +(.lvf vf1 (new 'static 'vector :x 1.2 :y 2.3 :z 3.4 :w 5.6)) +``` + +will be just a single instruction to do a `vmovaps xmm1, [rip + XXX]`. + +### `.svf` + +```lisp +(.svf dst-loc src-reg [:color #t|#f] [:offset ]) +``` + +Store a vector float. Works similarly to the `lvf` form, but there is no optimized case for storing into a static because this isn't allowed in GOAL. + +### Three operand vector float operations. + +```lisp +(.[.].vf dst src0 src1 [:color #t|#f] [:mask #b<0-15>]) +``` + +All the three operand forms work similarly. You can do something like `(.add.vf vf1 vf2 vf3)`. All operations use the similarly named `vps` instruction, xmm128 VEX encoding. We support the following `op-name`s: +- `xor` +- `add` +- `sub` +- `mul` +- `min` +- `max` + +An optional `:mask` value can be provided as a binary number between 0-15 (inclusive). This determines _which_ of the resulting elements will be committed to the destination vector. For example, `:mask #b1011` means that the `w`, `y` and `x` results will be committed. Note that the components are defined left-to-right which may be a little counter-intuitive -- `w` is the left-most, `x` is the right-most. This aligns with the PS2's VU implementation. + +Additionally, all of these operations support defining a single `broadcast-element`. This can be one of the 4 vector components `x|y|z|w`. Take the following for an example: `(.add.x.xyzw vf10, vf20, vf30)`, translates into: + +```cpp +vf10[x] = vf20[x] + vf30[x] +vf10[y] = vf20[y] + vf30[x] +vf10[z] = vf20[z] + vf30[x] +vf10[w] = vf20[w] + vf30[x] +``` + +### Three operand vector float operations with the accumulator + +```lisp +(.[.].vf dst src0 src1 acc [:color #t|#f] [:mask #b<0-15>]) +``` + +There are a few functions that will perform multiple operations involving the accumulator. We support the following `op-name`s: +- `add.mul` - Calculate the product of `src0` and `src1` and add it to the value of `acc` => `acc + (src0 * src1)` +- `sub.mul` - Calculate the product of `src0` and `src1` and subtract it from the value of `acc` => `acc - (src0 * src1)` + +An optional `:mask` value can be provided as a binary number between 0-15 (inclusive). This determines _which_ of the resulting elements will be committed to the destination vector. For example, `:mask #b1011` means that the `w`, `y` and `x` results will be committed. Note that the components are defined left-to-right which may be a little counter-intuitive -- `w` is the left-most, `x` is the right-most. This aligns with the PS2's VU implementation. + +Additionally, all of these operations support defining a single `broadcast-element`. This can be one of the 4 vector components `x|y|z|w`. + +### `.abs.vf` + +```lisp +(.abs.vf dst src [:color #t|#f] [:mask #b<0-15>]) +``` + +Calculates the absolute value of the `src` vector, and stores in the `dst` vector. + +### `.div.vf` and `.sqrt.vf` + +```lisp +(.div.vf dst src1 src2 :ftf #b<0-3> :fsf #b<0-3> [:color #t|#f]) +``` + +Calculates the quotient of _one_ of `src1`'s components specified by `fsf` _one_ of `src2`'s components specified by `ftf` and stores in every component of `dst` + +```lisp +(.sqrt.vf dst src :ftf #b<0-3> [:color #t|#f]) +``` + +Calculates the square-root of _one_ of `src`'s components specified by `ftf` and stores in every component of `dst` + +These instructions are interesting as they behave differently than the other math operations. In the original VU, results were stored in a seperate `Q` register, which was _NOT_ 128-bit. Instead it was a 32-bit register, meaning you have to pick which component from `src` you want to use. `:fsf` and `:ftf` are used to accomplish this, as usual, this is through bit flags -- `00` will select `x` and `11` will select `w`. + +As `dst` is just yet another vector / xmm register in x86, things are kept simple and the quotient is copied to _all_ packed single-float positions. This allows: +- Selecting any of the resulting vector slots will be equal to the quotient. +- Since the low-floating-point (X) is defined, the xmm register should function as expected for normal math operations + +### `.outer.product.vf` + +```lisp +(.outer.product.vf dst src1 src2 [:color #t|#f]) +``` + +Calculates the outer-product of `src1` and `src2` and stores the result in `dst`. _ONLY_ the x,y,z components are considered, and `dst`'s `w` component will be untouched. The following example illustrates what the outer-product is: + +Given 2 vectors `V1 = <1,2,3,4>` and `V2 = <5,6,7,8>` and assume `VDEST = <0, 0, 0, 999>` +The outer product is computed like so (only x,y,z components are operated on): + +`x = (V1y * V2z) - (V2y * V1z) => (2 * 7) - (6 * 3) => -4` + +`y = (V1z * V2x) - (V2z * V1x) => (3 * 5) - (7 * 1) => 8` + +`z = (V1x * V2y) - (V2x * V1y) => (1 * 6) - (5 * 2) => -4` + +`w = N/A, left alone => 999` + +> `VDEST = <-4, 8, -4, 999>` + +### `.blend.vf` + +```lisp +(.blend.vf dst src0 src1 mask [:color #t|#f]) +``` + +Wrapper around `vblendps` (VEX xmm128 version) instruction. The `mask` must evaluate to a constant integer at compile time. The integer must be in the range of 0-15. + +### `.itof.vf` and `.ftoi.vf` + +```lisp +(.itof.vf dst src [:mask mask-val] [:color #t|#f]) +(.ftoi.vf dst src [:mask mask-val] [:color #t|#f]) +``` + +Wrapper around `vcvtdq2ps` and `vcvtps2dq` to convert packed 32-bit signed integers to packed 32-bit floats and back. The `mask` and `color` arguments behave like other assembly operations. + +### `.pw.sra`, `.pw.srl`, and `pw.sll` + +```lisp +(.pw.sra dst src shift-amount [:mask mask-val] [:color #t|#f]) +(.pw.srl dst src shift-amount [:mask mask-val] [:color #t|#f]) +(.pw.sll dst src shift-amount [:mask mask-val] [:color #t|#f]) +``` + +Wrapper around `vpsrld`, `vpsrad`, and `vpslld`. Does shifts on each of the 4 32-bit integers in the register. + +### `.pextlw`, `.pextuw`, `.pcpyud`, `.pcpyld`, `.pceqw`, `.ppach` + +```lisp +(.pextlw dst src0 src1 [:color #t|#f]) +(.pextuw dst src0 src1 [:color #t|#f]) +(.pcpyud dst src0 src1 [:color #t|#f]) +(.pcpyld dst src0 src1 [:color #t|#f]) +(.pceqw dst src0 src1 [:color #t|#f]) +(.ppach dest src0 src1) +``` + +Equivalents of the EE's MMI instructions with the same name. These can only be used on 128-bit variables. Most map to single x86 instructions: +- `pextlw` is `VPUNPCKLDQ` (sources swapped) +- `pextuw` is `VPUNPCKHDQ` (sources swapped) +- `pcpyld` is `VPUNPCKLQDQ` (sources swapped) +- `pcpyud` is `VPUNPCKHQDQ` (sources _not_ swapped) +- `pceqw` is `VPCMPEQD` + +Some map to multiple instructions. These must use the coloring system. +- `ppach` is a sequence of 7 instructions (`VPSHUFLW`, `VPSHUFHW`, `VPSRLDQ`, `VPUNPCKLQDQ`). + +## TODO + +### Things related to enums +Not yet implemented + +### `defmacro` + +### Loop forms + +### `&` + +### `->` + +### Type + +### Compile-Time Size stuff + +### `object-new` \ No newline at end of file diff --git a/docs/markdown/method_system.md b/docs/markdown/method_system.md new file mode 100644 index 0000000000..2ee3265fe0 --- /dev/null +++ b/docs/markdown/method_system.md @@ -0,0 +1,147 @@ +# OpenGOAL's Method System + +OpenGOAL has a virtual method system. This means that child types can override parent methods. The first argument to a method is always the object the method is being called on, except for `new`. + +All types have methods. Objects have access to all of their parents methods, and may override parent methods. All types have these 9 methods: + +- `new` - like a constructor, returns a new object. It's not used in all cases, and on all types, and needs more documentation on when specifically it is used. +- `delete` - basically unused, but like a destructor. Often calls `kfree`, which does nothing. +- `print` - prints a short, one line representation of the object to the `PrintBuffer` +- `inspect` - prints a multi-line description of the object to the `PrintBuffer`. Usually auto-generated by the compiler and prints out the name and value of each field. +- `length` - Returns a length if the type has something like a length (number of characters in string, etc). Otherwise returns 0. Usually returns the number of filled slots, instead of the total number of allocated slots, when there is possibly a difference. +- `asize-of` - Gets the size in memory of the entire object. Usually this just looks this up from the appropriate `type`, unless it's dynamically sized. +- `copy` - Create a copy of this object on the given heap. Not used very much? +- `relocate` - Some GOAL objects will be moved in memory by the kernel as part of the compacting actor heap system. After being moved, the `relocate` method will be called with the offset of the move, and the object should fix up any internal pointers which may point to the old location. It's also called on v2 objects loaded by the linker when they are first loaded into memory. +- `memusage` - Not understood yet, but probably returns how much memory in bytes the object uses. Not supported by all objects. + +Usually a method which overrides a parent method must have the same argument and return types. The only exception is `new` methods, which can have different argument/return types from the parent. (Dee the later section on `_type_` for another exception) + +The compiler's implementation for calling a method is: +- Is the type a basic? + - If so, look up the type using runtime type information + - Get the method from the vtable +- Is the type not a basic? + - Get the method from the vtable of the compile-time type + - Note that this process isn't very efficient - instead of directly linking to the slot in the vtable (one deref) it first looks up the `type` by symbol, then the slot (two derefs). I have no idea why it's done this way. + +In general, I suspect that the method system was modified after GOAL was first created. There is some evidence that types were once stored in the symbol table, but were removed because the symbol table became full. This could explain some of the weirdness around method calls/definition rules, and the disaster `method-set!` function. + +All type definitions should also define all the methods, in the order they appear in the vtable. I suspect GOAL had this as well because the method ordering otherwise seems random, and in some cases impossible to get right unless (at least) the number of methods was specified in the type declaration. + +## Special `_type_` Type + +The first argument of a method always contains the object that the method is being called on. It also must have the type `_type_`, which will be substituted by the type system (at compile time) using the following rules: + +- At method definition: replace with the type that the method is being defined for. +- At method call: replace with the compile-time type of the object the method is being called on. + +The type system is flexible with allowing you to use `_type_` in the method declaration in `deftype`, but not using `_type_` in the actual `defmethod`. + +A method can have other arguments or a return value that's of type `_type_`. This special "type" will be replaced __at compile time__ with the type which is defining or calling the method. No part of this exists at runtime. It may seem weird, but there are two uses for this. + +The first is to allow children to specialize methods and have their own child type as an argument type. For example, say you have a method `is-same-shape`, which compares two objects and sees if they are the same shape. Suppose you first defined this for type `square` with +``` +(defmethod square is-same-shape ((obj1 square) (obj2 square)) + (= (-> obj1 side-length) (-> obj2 side-length)) + ) +``` + +Then, if you created a child class of `square` called `rectangle` (this is a terrible way to use inheritance, but it's just an example), and overrode the `is-same-shape` method, you would have to have arguments that are `square`s, which blocks you from accessing `rectangle`-specific fields. The solution is to define the original method with type `_type_` for the first two arguments. Then, the method defined for `rectangle` also will have arguments of type `_type_`, which will expand to `rectangle`. + + +The second use is for a return value. For example, the `print` and `inspect` methods both return the object that is passed to them, which will always be the same type as the argument passed in. If `print` was define as `(function object object)`, then `(print my-square)` would lose the information that the return object is a `square`. If `print` is a `(function _type_ _type_)`, the type system will know that `(print my-square)` will return a `square`. + +## Details on the Order of Overrides + +The order in which you `defmethod` and `deftype` matters. + +When you `deftype`, you copy all methods from the parent. When you `defmethod`, you always set a method in that type. You may also override methods in a child if: the child hasn't modified that method already, and if you are in a certain mode. This is a somewhat slow process that involves iterating over the entire symbol table and every type in the runtime, so I believe it was disabled when loading level code, and you just had to make sure to `deftype` and `defmethod` in order. + +Assume you have the type hierarchy where `a` is the parent of `b`, which is the parent of `c`. + +If you first define the three types using `deftype`, then override a method from `a` on `c`, then override that same method on `b`, then `c` won't use the override from `b`. + +If you first define the three types using `deftype`, then override a method on `b`, it will _sometimes_ do the override on `c`. This depends on the value of the global variable `*enable-method-set*`, and some other confusing options. It may also print a warning but still do the override in certain cases. + +## Built in Methods + +All types have these 9 methods. They have reasonable defaults if you don't provide anything. + +### `new` + +The new method is a very special method used to construct a new object, like a constructor. Note that some usages of the `new` keyword __do not__ end up calling the new method. See the `new` section for more details. Unlike C++, fields of a type and elements in an array are not constructed either. + +The first argument is an "allocation", indicating where the object should be constructed. It can be +- The symbol `'global` or `'debug`, indicating the global or debug heaps +- The symbols `'process-level-heap` or `'loading-level`, indicating whatever heaps are stored in those symbols. +- `'process`, indicating the allocation should occur on the current process heap. +- `'scratch`, for allocating on the scratchpad. This is unused. +- Otherwise it's treated as a 16-byte aligned address and used for in place construction (it zeros the memory first) + +The second argument is the "type to make". It might seem stupid at first, but it allows child classes to use the same `new` method as the parent class. + +The remaining arguments can be used for whatever you want. + +When writing your own `new` methods, you should ignore the `allocation` argument and use the `object-new` macro to actually do the allocation. This takes care of all the details for getting the memory (and setting up runtime type information if its a basic). See the section on `object-new` for more details. + +### `delete` + +This method isn't really used very much. Unlike a C++ destructor it's never called automatically. In some cases, it's repurposed as a "clean up" type function but it doesn't actually free any memory. It takes no arguments. The default implementations call `kfree` on what the allocation, but there are two issues: +1. The implementation is sometimes wrong, likely confusing doing pointer math (steps by array stride) with address math (steps by one byte). +2. The `kfree` function does nothing. + +The `kheap` system doesn't really support freeing objects unless you free in the opposite order you allocate, so it makes sense that `delete` doesn't really work. + +### `print` + +This method should print out a short description of the object (with no newlines) and return the object. The printing should be done with `(format #t ...)` (see the section on `format`) for more information. If you call `print` by itself, it'll make this description show up in the REPL. (Note that there is some magic involved to add a newline here... there's actually a function named `print` that calls the `print` method and adds a newline) + +The default short description looks like this: `#` for printing an object of type `test-type`. Of course, you can override it with a better version. Built-in types like string, type, boxed integer, pair, have reasonable overrides. + +This method is also used to print out the object with `format`'s `~A` format option. + +### `inspect` + +This method should print out a detailed, multi-line description. By default, `structure`s and `basic`s will have an auto-generated method that prints out the name and value of all fields. For example: + +```lisp +gc > (inspect *kernel-context*) +[00164b44] kernel-context + prevent-from-run: 65 + require-for-run: 0 + allow-to-run: 0 + next-pid: 2 + fast-stack-top: 1879064576 + current-process: #f + relocating-process: #f + relocating-min: 0 + relocating-max: 0 + relocating-offset: 0 + low-memory-message: #t +``` + +In some cases this method is overridden to provide nicer formatting. + +### `length` + +This method should return a "length". The default method for this just returns 0, but for things like strings or buffers, it could be used to return the number of characters or elements in use. It's usually used to refer to how many are used, rather than the capacity. + +### `asize-of` + +This method should return the size of the object. Including the 4 bytes of type info for a `basic`. + +By default this grabs the value from the object's `type`, which is only correct for non-dynamic types. For types like `string` or other dynamic types, this method should be overridden. If you intend to store dynamically sized objects of a given type on a process heap, you __must__ implement this method accurately. + +### `copy` + +Creates a copy of the object. I don't think this used very much. Just does a `memcpy` to duplicate by default. + +### `relocate` + +The exact details are still unknown, but is used to update internal data structures after an object is moved in memory. This must be support for objects allocated in process heaps of processes allocated on the actor heap or debug actor heap. + +It's also called on objects loaded from a GOAL data object file. + +### `mem-usage` + +Not much is known yet, but used for computing memory usage statistics. \ No newline at end of file diff --git a/docs/markdown/object_file_generation.md b/docs/markdown/object_file_formats.md similarity index 88% rename from docs/markdown/object_file_generation.md rename to docs/markdown/object_file_formats.md index 149efe80fa..ccfa374ca9 100644 --- a/docs/markdown/object_file_generation.md +++ b/docs/markdown/object_file_formats.md @@ -1,11 +1,15 @@ -# CGO/DGO Files -The CGO/DGO file format is exactly the same - the only difference is the name of the file. The DGO name indicates that the file contains all the data for a level. The engine will load these files into a level heap, which can then be cleared and replaced with a different level. +# Object File Formats + +## CGO/DGO Files + +The CGO/DGO file format is exactly the same - the only difference is the name of the file. The DGO name indicates that the file contains all the data for a level. The engine will load these files into a level heap, which can then be cleared and replaced with a different level. I suspect that the DGO file name came first, as a package containing all the data in the level which can be loaded very quickly. Names in the code all say `dgo`, and the `MakeFileName` system shows that both CGO and DGO files are stored in the `game/dgo` folder. Probably the engine and kernel were packed into a CGO file after the file format was created for loading levels. -Each CGO/DGO file contains a bunch of individual object files. Each file has a name. There are some duplicate names - sometimes the two files with the same names are very different (example, code for an enemy, art for an enemy), and other times they are very similar (tiny differences in code/data). The files come in two versions, v4 and v3, and both CGOs and DGOs contain both versions. If an object file has code in it, it is always a v3. It is possible to have a v3 file with just data, but usually the data is pretty small. The v4 files tend to have a lot of data in them. My theory is that the compiler creates v3 files out of GOAL source code files, and that other tools for creating things like textures/levels/art-groups generate v4 objects. There are a number of optimizations in the loading process for v4 objects that are better suited for larger files. To stay at 60 fps always, a v3 object must be smaller than around 750 kB. A v4 object does not have this limitation. - -# The V3 format +Each CGO/DGO file contains a bunch of individual object files. Each file has a name. There are some duplicate names - sometimes the two files with the same names are very different (example, code for an enemy, art for an enemy), and other times they are very similar (tiny differences in code/data). The files come in two versions, v4 and v3, and both CGOs and DGOs contain both versions. If an object file has code in it, it is always a v3. It is possible to have a v3 file with just data, but usually the data is pretty small. The v4 files tend to have a lot of data in them. My theory is that the compiler creates v3 files out of GOAL source code files, and that other tools for creating things like textures/levels/art-groups generate v4 objects. There are a number of optimizations in the loading process for v4 objects that are better suited for larger files. To stay at 60 fps always, a v3 object must be smaller than around 750 kB. A v4 object does not have this limitation. + +### The V3 format + The v3 format is divided into three segments: 1. Main: this contains all of the functions/data that will be used by the game. 2. Debug: this is only loaded in debug mode, and is always stored on a separate `kdebugheap`. @@ -17,30 +21,33 @@ This format will be different between the PC and PS2 versions, as linking data f Each segment can contain functions and data. The top-level segment must start with a function which will be run to initialize the object. All the data here goes through the GOAL compiler and type system. -# The V4 format +### The V4 format + The V4 format contains just data. Like v3, the data is GOAL objects, but was probably generated by a tool that wasn't the compiler. A V4 object has no segments, but must start with a `basic` object. After being linked, the `relocate` method of this `basic` will be called, which should do any additional linking required for the specific object. -Because this is just data, there's no reason for the PC version to change this format. This means we can also check the +Because this is just data, there's no reason for the PC version to change this format. This means we can also check the Note: you may see references to v2 format in the code. I believe v4 format is identical to v2, except the linking data is stored at the end, to enable a "don't copy the last object" optimization. The game's code uses the `work_v2` function on v4 objects as a result, and some of my comments may refer to v2, when I really mean v4. I believe there are no v2 objects in any games. -# Plan +### The Plan + - Create a library for generating obj files in V3/V4 format - V4 should match game exactly. Doesn't support code. - V3 is our own thing. Must support code. - + We'll eventually create tools which use the library in V4 mode to generate object files for rebuilding levels and textures. We may need to wait until more about these formats is understood before trying this. The compiler will use the library in V3 mode to generate object files for each `gc` (GOAL source code) file. -# CGO files -The only CGO files read are `KERNEL.CGO` and `GAME.CGO`. - - The `KERNEL.CGO` file contains the GOAL kernel and some very basic libraries (`gcommon`, `gstring`, `gstate`, ...). I believe that `KERNEL` was always loaded on boot during development, as its required for the Listener to function. +### CGO files -The `GAME.CGO` file combines the contents of the `ENGINE`, `COMMON` and `ART` CGO files. `ENGINE` contains the game engine code, `COMMON` contains level-specific code (outside of the game engine) that is always loaded. If code is used in basically all the levels, it makes sense to put in in `COMMON`, so it doesn't have to be loaded for each currently active level. The `ART` CGO contains common art/textures/models, like Jak and his animations. +The only CGO files read are `KERNEL.CGO` and `GAME.CGO`. -The `JUNGLE.CGO`, `MAINCAVE.CGO`, `SUNKEN.CGO` file contains some copies of files used in the jungle, cave, LPC levels. Some are a tiny bit different. I believe it is unused. + The `KERNEL.CGO` file contains the GOAL kernel and some very basic libraries (`gcommon`, `gstring`, `gstate`, ...). I believe that `KERNEL` was always loaded on boot during development, as its required for the Listener to function. + +The `GAME.CGO` file combines the contents of the `ENGINE`, `COMMON` and `ART` CGO files. `ENGINE` contains the game engine code, `COMMON` contains level-specific code (outside of the game engine) that is always loaded. If code is used in basically all the levels, it makes sense to put in in `COMMON`, so it doesn't have to be loaded for each currently active level. The `ART` CGO contains common art/textures/models, like Jak and his animations. + +The `JUNGLE.CGO`, `MAINCAVE.CGO`, `SUNKEN.CGO` file contains some copies of files used in the jungle, cave, LPC levels. Some are a tiny bit different. I believe it is unused. The `L1.CGO` file contains basically all the level-specific code/Jak animations and some textures. It doesn't seem to contain any 3D models. It's unused, but I'm still interested in understanding its format, as the Jak 1 demos have this file. @@ -50,10 +57,11 @@ The `VILLAGEP.CGO` file contains common code shared in village levels, which isn The `WATER-AN.CGO` file contains some small code/data for water animations. Unused. The same data appears in the levels as needed. -# CGO/DGO Loading Process +### CGO/DGO Loading Process + A CGO/DGO file is loaded onto a "heap", which is just a chunk of contiguous memory. The loading process is designed to be fast, and also able to fill the entire heap, and allow each object to allocate memory after it is loaded. The process works like this: -1. Two temporary buffers are allocated at the end of the heap. These are sized so that they can fit the largest object file, not including the last object file. +1. Two temporary buffers are allocated at the end of the heap. These are sized so that they can fit the largest object file, not including the last object file. 2. The IOP begins loading, and is permitted to load the first two object files to the two temporary buffers 3. The main CPU waits for the first object file to be loaded. 4. While the second object file being loaded, the first object is "linked". The first step to linking is to copy the object file data from the temporary buffer to the bottom of the heap, kicking out all the other data in the process. The linking data is checked to see if it is in the top of the heap, and is moved there if it isn't already. The run-once initialization code is copied to another temporary allocation on top of the heap and the debug data is copied to the debug heap. @@ -66,13 +74,12 @@ A CGO/DGO file is loaded onto a "heap", which is just a chunk of contiguous memo 11. The last object will be loaded directly onto the bottom of the heap, as there may not be enough memory to use the temporary buffers and load the last object. The temporary buffers are freed. 12. If the last object is a v3, its linking data will be moved to the top-level, and the object data will be moved to fill in the gap left behind. If the last object is a v2, the main data will be at the beginning of the object data, so there is an optimization that will avoid copying the object data to save time, if the data is already close to being in the right place. - Generally the last file in a level DGO will be the largest v4 object. You can only have one file larger than a temporary buffer, and it must come last. The last file also doesn't have to be copied after being loaded into memory if it is a v4. -V3 max size: - A V3 object is copied all at once with a single `ultimate-memcpy`. Usually linking gets to run for around 3 to 5% of a total frame. The `ultimate-memcpy` routine does a to/from scratchpad transfer. In practice, mem/spr transfers are around 1800 MB/sec, and the data has to be copied twice, so the effective bandwidth is 900 MB/sec. - - `900 MB / second * (0.04 * 0.0167 seconds) = 601 kilobytes` - - This estimate is backed up by the the chunk size of the v4 copy routine, which copies one chunk per frame. It picks 524 kB as the maximum amount that's safe to copy per frame. - \ No newline at end of file +#### V3 max size + +A V3 object is copied all at once with a single `ultimate-memcpy`. Usually linking gets to run for around 3 to 5% of a total frame. The `ultimate-memcpy` routine does a to/from scratchpad transfer. In practice, mem/spr transfers are around 1800 MB/sec, and the data has to be copied twice, so the effective bandwidth is 900 MB/sec. + +`900 MB / second * (0.04 * 0.0167 seconds) = 601 kilobytes` + +This estimate is backed up by the the chunk size of the v4 copy routine, which copies one chunk per frame. It picks 524 kB as the maximum amount that's safe to copy per frame. diff --git a/docs/markdown/porting_to_x86.md b/docs/markdown/porting_to_x86.md index 81e9c36031..a446d02637 100644 --- a/docs/markdown/porting_to_x86.md +++ b/docs/markdown/porting_to_x86.md @@ -1,12 +1,14 @@ # Porting to x86 + This document will keep track of stuff that needs to be ported or modified significantly for x86. Anything that uses PS2-specific hardware or relies on stuff in the C Kernel will need to be ported. ## Basic Info + Most of the game is written in GOAL. All this source lives in `goal_src`. The "runtime" is all the support code written in C++. It's located in `game/`. Sometimes the "runtime" + GOAL code together is called the "target". -Most of the code in "runtime" is reverse engineered code from the real game, with small tweaks to make it work on x86 and with OpenGOAL. +Most of the code in "runtime" is reverse engineered code from the real game, with small tweaks to make it work on x86 and with OpenGOAL. The code in `game/system` is **not** from the game and is an implementation of system functions that are implemented by Sony in the PS2 game. It's stuff like threading, I/O, etc. @@ -23,6 +25,7 @@ To give an idea of the size of these (counted by `wc -l`): - System is 1294 lines, and still has some to implement ## Math Libraries + I think most of the math libraries can be decompiled, but there are a few that will need manual work. These are also a great place to do tests as the math functions have very few dependencies and we know what the right answer should be for many of them. - `bounding-box` (only some stuff) @@ -34,11 +37,12 @@ I think most of the math libraries can be decompiled, but there are a few that w At some point we may want to re-implement some of these to be more efficient. ## The IOP (I/O Processor) Framework + This is already implemented. The IOP was a separate I/O Processor on the PS2. It runs a cooperative multi-tasking kernel developed by Sony. In OpenGOAL it is implemented in `game/system/IOP_Kernel.h`. The IOP Kernel is managed by the IOP runtime thread (`game/system/iop_thread.h`). -The library in `game/sce/iop.h` wraps the `IOP_Kernel` in an interface that looks like the Sony libraries used by the game so the IOP code can be ported directly. +The library in `game/sce/iop.h` wraps the `IOP_Kernel` in an interface that looks like the Sony libraries used by the game so the IOP code can be ported directly. There are a few stub functions that are hardcoded to return the correct values for stuff like CD drive initialization. The main features currently supported are: - Threads (create, wakeup, start, sleep, delay, get ID) @@ -49,6 +53,7 @@ There are a few stub functions that are hardcoded to return the correct values f All this stuff is currently used for loading DGOs, which is tested and working. ## OVERLORD Framework + This is already implemented. The OVERLORD is the code written by Naughty Dog that runs on the IOP. It is responsible for sound and loading data. It's surprisingly complicated and some parts of it are extremely poorly written, especially the thread synchronization stuff. My implementation of OVERLORD is in `game/overlord`. It's not complete yet, but the basics are there and it does enough to load DGOs. @@ -58,6 +63,7 @@ The framework for OVERLORD is already implemented. The C Kernel calls a Sony lib Once `start_overlord` returns, the initial call to `sceSifLoadModule` returns and the runtime keeps initializing. ## OVERLORD ISO Thread + This is partially implemented. This thread is responsible for controlling the DVD drive and the small DVD data buffers used in the IOP. It has a big loop in `ISOThread()` in `iso.cpp` that looks for pending reads, executes them, waits for data to be read, then calls a callback. This code is unbelievably confusing. @@ -68,11 +74,12 @@ To interact with the DVD drive, it uses an `IsoFS` abstraction, which is a struc It also has some sound stuff in it for managing VAG audio streams, but this isn't implemented yet. -The other threads in OVERLORD are "RPC" threads. They sit in a loop waiting for the main runtime thread (EE thread) to send a remote procedure call (RPC). Then they do something (like maybe sending a message to the ISO thread), maybe wait for something to happen, and then return. +The other threads in OVERLORD are "RPC" threads. They sit in a loop waiting for the main runtime thread (EE thread) to send a remote procedure call (RPC). Then they do something (like maybe sending a message to the ISO thread), maybe wait for something to happen, and then return. -From the GOAL/EE side of things, RPC calls can be blocking or non-blocking. They can be issued from GOAL (with `rpc-call`) or from the C Kernel (`RpcCall`). Per "channel" (corresponds to an IOP thread), there can only be one RPC call happening at a time. The `rpc-busy?` command can be used to check if an RPC is complete. +From the GOAL/EE side of things, RPC calls can be blocking or non-blocking. They can be issued from GOAL (with `rpc-call`) or from the C Kernel (`RpcCall`). Per "channel" (corresponds to an IOP thread), there can only be one RPC call happening at a time. The `rpc-busy?` command can be used to check if an RPC is complete. ## IOP PLAY (6) + This is unimplemented. The `PLAY` RPC appears to be relatively simple and plays/stops/pauses/queues a VAG audio stream. It can either use the "AnimationName" system or another system to get the name of the audio stream. I don't know what sound effects in the game are streamed, but I believe there are some. @@ -80,22 +87,25 @@ The `PLAY` RPC appears to be relatively simple and plays/stops/pauses/queues a V I suspect the GOAL side code for this is in `gsound` and `gsound-h`. ## IOP STR (5) + This is unimplemented. This is an RPC for streaming data back to the EE. I think this is used to control animation streaming. ## IOP DGO (4) + This is implemented. This is the RPC for loading DGO files. The DGO loading is super complicated, but the basic idea is that loading / linking are double buffered. In order to allow linking files to allocate memory, the currently loading file goes in a temporary buffer on the top of the heap. (There are actually two temp buffers that rotate, one for loading out of and one for linking, as the "copy to heap" step is done as part of linking, not loading) The final chunk is not double buffered. This is so it can be loaded directly into its final location in the heap. This has three advantages: you don't need to copy it out of a temporary buffer, you can have a file larger than the temp buffer and you can also entirely fill the heap this way (the temp buffers are freed so you don't have to worry about that). -The IOP side state machine for this is in `iso.cpp`, implemented inside of the DGO load buffer complete callback and is somewhat complicated because DGO info may be split between multiple buffers, and you have to deal with getting partial info. The EE side is in `kdgo.cpp`. +The IOP side state machine for this is in `iso.cpp`, implemented inside of the DGO load buffer complete callback and is somewhat complicated because DGO info may be split between multiple buffers, and you have to deal with getting partial info. The EE side is in `kdgo.cpp`. The DGO synchronization is pretty confusing but I believe I have it working. It may be worth documenting it more (I thought I did already, but where did I put it?). ## IOP Server/Ramdisk (3) + This is implemented, but so far unused and untested. This RPC is used to store files in RAM on the IOP. There's a buffer of around 800 kB. I believe it's used for a few different things, in particular the level visibility data. The EE requests data to be loaded from a file on DVD into the "ramdisk" (just a buffer on the IOP), then can request chunks of this file. Of course it is not as fast as storing the file in the EE RAM, but it is much faster than reading from the DVD again. @@ -104,16 +114,19 @@ This is what Andy Gavin refers to when they said they did "things they weren't s ## IOP Loader (2) + This is unimplemented. This is used to control the loading of music and soundbanks. I haven't touched it yet. Music and soundbanks are loaded into IOP memory when you switch levels. ## IOP Player (1) + This is unimplemented. This is used to control the playing of sound, and goes with Loader. Like PLAY it can play VAG audio streams. I'm not sure which one is actually used for streaming audio, maybe both? ## IOP VBlank Handler + This is unimplemented. The IOP's kernel will call `VBlank_Handler` on each vblank. This is once per frame, and I don't know where it is, or if its tied to the actual HW vblank or framebuffer swap, if it happens at 30/60 fps (or even/odd frames if 30 fps). I suspect it's the real vblank at 60 fps but I don't know. @@ -124,8 +137,8 @@ The EE first has to do some set up to tell the IOP where to copy the data, which We'll also need to add some stuff to `system` and `sce/iop` to set this up, which will have to work with frame timing stuff so it happens at the right part of the frame. - ## Sound Library + This is a pretty big one. To actually make sounds, OVERLORD code uses a third-party sound library called 989SND. Internally 989SND uses the SPU2 (Sound Processor) to actually do the "sound math" to decode ADPCM, do ADSR for the sampled sounds, and do reverb/mixing. I think the lowest effort sound implementation is to try to reimplement 989SND + the SPU as a single library. This could be tested and developed in isolation from everything else. @@ -133,6 +146,7 @@ I think the lowest effort sound implementation is to try to reimplement 989SND + We'll also need to pick a library for doing audio on PC and a design for how to keep the audio in sync. My gut feeling is to let the IOP side audio stuff just run totally independent from everything else, like the real game does. Let the audio sampling be driven by the sound device so you never have any crackling/interpolation artifacts. This is why the audio keeps going even after the game crashes on PS2. ## GOAL Kernel + The GOAL kernel needs some modification to work on x86. It implements userspace threading and needs to know the details of how to back up the current CPU state and restore it. It also needs to work with the compiler to make sure that the kernel and compiler agree on what registers may not be preserved across a thread suspend There are also some CPU specific details on how to do dynamic throw/catches, unwinding stack frames, and passing initial arguments to a thread. In OpenGOAL, the `rsp` is a "real" pointer and all other pointers are "GOAL pointer"s (offset from base of GOAL memory), so there are some details needed to correctly save/restore stacks. @@ -140,9 +154,10 @@ In OpenGOAL, the `rsp` is a "real" pointer and all other pointers are "GOAL poin A final detail is we will probably want/need the ability to increase the default size of stack that can be backed up on suspend. The default is 256 bytes so if our compiler does worse than the original and we use more stack space, we could run out. There's a check for this so it shouldn't be hard to detect. ## Jak Graphics Basics -The PS2 has some special hardware that's used for graphics. These are the DMAC, the VU1, and the GS. -The DMAC is a sophisticated DMA controller. It runs separately from the EE and can copy data from one place to another at pretty high speed. If it is not stalled for any reason it can reach 2.4 GB/sec. The main RAM is only good for around 1.2 GB/sec so in practice "big" things don't move around any faster than 1.2 GB/sec on average. It's used to send graphics data from main memory to the other components. It can be configured, but it's not programmable. It can do simple transfers, like "copy this block of data from here to there", and more complicated things, like following linked lists. +The PS2 has some special hardware that's used for graphics. These are the DMAC, the VU1, and the GS. + +The DMAC is a sophisticated DMA controller. It runs separately from the EE and can copy data from one place to another at pretty high speed. If it is not stalled for any reason it can reach 2.4 GB/sec. The main RAM is only good for around 1.2 GB/sec so in practice "big" things don't move around any faster than 1.2 GB/sec on average. It's used to send graphics data from main memory to the other components. It can be configured, but it's not programmable. It can do simple transfers, like "copy this block of data from here to there", and more complicated things, like following linked lists. The VU1 takes the role of vertex shaders. It can be programmed, but only in assembly, and it is extremely challenging and confusing. It has an extremely small memory (16 kB), but this memory is extremely fast. It's role is usually to do vertex transformations and lighting, then generate a list of commands to send to the GS. The `XGKICK` instruction on VU1 is used to send data from the VU1 memory to the GS. @@ -153,7 +168,7 @@ The GS is the actual GPU. It has VRAM and receives commands from a few different The GS is like pixel shaders but it's very simple - it's not programmable and only can do a few fixed things. The GS also has the VRAM, which can contain frame buffers, z buffers, textures, and scratch area for effects. -My understanding is that during a frame, the EE generates a long list of things to draw. These are a DMA "chain" - basically a complicated linked-list like data structure that the PS2's DMA knows how to handle. I believe some graphics calculations are done on the EE - particularly the environment mapping. +My understanding is that during a frame, the EE generates a long list of things to draw. These are a DMA "chain" - basically a complicated linked-list like data structure that the PS2's DMA knows how to handle. I believe some graphics calculations are done on the EE - particularly the environment mapping. ## DMA @@ -209,4 +224,4 @@ My understanding is that during a frame, the EE generates a long list of things ## Ocean -## Navigate \ No newline at end of file +## Navigate diff --git a/docs/markdown/progress-notes/documented-forms.md b/docs/markdown/progress-notes/documented-forms.md new file mode 100644 index 0000000000..b04ee24a20 --- /dev/null +++ b/docs/markdown/progress-notes/documented-forms.md @@ -0,0 +1,141 @@ +# All Forms +Documented forms are crossed out. +- ~~asm-file~~ +- ~~m~~ +- ~~ml~~ +- ~~md~~ +- ~~build-game~~ +- ~~build-data~~ +- ~~blg~~ +- tc +- ~~e~~ +- db (debug mode) +- #when +- #unless +- ~~lt~~ +- ~~r~~ +- ~~shutdown-target~~ +- db (disassemble byte) +- dh +- dw +- dd +- df +- segfault +- fpe +- let +- let* +- ~~defun~~ +- while +- until +- dotimes +- protect +- +! +- ~~if~~ +- ~~when~~ +- ~~unless~~ +- and +- or +- +1 +- +! +- -! +- *! +- 1- +- zero? +- &+! +- &- +- &-> +- basic? +- pair? +- binteger? +- rtype-of +- cons +- list +- null? +- caar +- object-new +- expect-eq +- expect-true +- expect-false +- start-test +- finish-test +- top-level +- ~~begin~~ +- ~~block~~ +- ~~return-from~~ +- ~~label~~ +- ~~goto~~ +- gs +- :exit +- ~~asm-file~~ +- ~~asm-data-file~~ +- listen-to-target +- reset-target +- :status +- ~~in-package~~ +- ~~#cond~~ +- ~defglobalconstant~ +- ~~seval~~ +- ~~cond~~ +- ~~when-goto~~ +- ~~define~~ +- ~~define-extern~~ +- ~~set!~~ +- dbs +- dbg +- :cont +- :break +- :dump-all-mem +- :pm +- :di +- :disasm +- :bp +- :ubp +- deftype +- ~~defmethod~~ +- -> +- & +- the-as +- the +- print-type +- new +- car +- cdr +- method +- declare-type +- none +- ~~lambda~~ +- ~~declare~~ +- ~~inline~~ +- ~~quote~~ +- ~~mlet~~ +- defconstant +- ~~+~~ +- ~~-~~ +- ~~*~~ +- ~~/~~ +- ~~shlv~~ +- ~~shrv~~ +- ~~sarv~~ +- ~~mod~~ +- ~~logior~~ +- ~~logxor~~ +- ~~logand~~ +- ~~lognot~~ +- ~~=~~ +- ~~!=~~ +- ~~eq?~~ +- ~~neq?~~ +- ~~not~~ +- ~~<=~~ +- ~~>=~~ +- ~~<~~ +- ~~>~~ +- &+ +- ~~build-dgos~~ +- ~~set-config!~~ +- rlet +- .ret +- .sub +- .push +- .pop +- set-config! \ No newline at end of file diff --git a/docs/markdown/reader.md b/docs/markdown/reader.md index 507c9bc290..d78e226944 100644 --- a/docs/markdown/reader.md +++ b/docs/markdown/reader.md @@ -1,13 +1,17 @@ # Reader + GOOS and GOAL both use the same reader, which converts text files to S-Expressions and allows these s-expressions to be mapped back to a line in a source file for error messages. This document explains the syntax of the reader. Note that these rules do not explain the syntax of the language (for instance, GOAL has a much more complicated system of integers and many more restrictions), but rather the rules of how your program source must look. ## Integer Input + Integers handled by the reader are 64-bits. Any overflow is considered an error. An integer can be specified as a decimal, like `0` or `-12345`; in hex, like `#xbeef`; or in binary, like `#b101001`. All three representations can be used anywhere an integer is used. Hex numbers do not care about the case of the characters. Decimal numbers are signed, and wrapping from a large positive number to a negative number will generate an error. The valid input range for decimals is `INT64_MIN` to `INT64_MAX`. Hex and binary are unsigned and do not support negative signs, but allow large positive numbers to wrap to negative. Their input range is `0` to `UINT64_MAX`. For example, `-1` can be entered as `-1` or `#xffffffffffffffff`, but not as `UINT64_MAX` in decimal. ## Floating Point Input + Floating point values handled by the reader are implemented with `double`. Weird numbers (denormals, NaN, infinity) are invalid and not handled by the reader directly. A number _must_ have a decimal point to be interpreted as floating point. Otherwise, it will be an integer. Leading/trailing zeros are optional. ## Character Input + Characters are used to represent characters that are part of text. The character `c` is represented by `#\c`. This representation is used for all ASCII characters between `!` and `~`. There are three special characters which have a non-standard representation: - Space : `#\\s` - New Line: `#\\n` @@ -15,18 +19,19 @@ Characters are used to represent characters that are part of text. The characte All other characters are invalid. -## String +## Strings + A string is a sequence of characters, surrounding by double quotes. The ASCII characters from ` ` to `~` excluding `"` can be entered directly. Strings have the following escape codes: - `\\` : insert a backslash - `\n` : insert a new line - `\t` : insert a tab - `\"` : insert a double quote - ## Comments + The reader supports line comments with `;` and multi-line comments with `#| |#`. For example -``` +```lisp (print "hi") ; prints hi #| @@ -36,6 +41,7 @@ this is a multi-line comment! ``` ## Array + The reader supports arrays with the following syntax: ``` ; array of 1, 2, 3, 4 @@ -45,17 +51,21 @@ The reader supports arrays with the following syntax: Arrays can be nested with lists, pairs, and other arrays. ## Pair + The reader supports pairs with the following syntax: -``` + +```lisp ; pair of a, b (a . b) ``` + Pairs can be nested with lists, pairs, and arrays. ## List + The reader supports lists. Lists are just an easier way of constructing a linked list of pairs, terminated with the empty list. The empty list is a special list written like `()`. -``` +```lisp ; list of 1, 2, 3 (1 2 3) ; actually the same as @@ -63,8 +73,10 @@ The reader supports lists. Lists are just an easier way of constructing a linked ``` ## Symbol + A symbol is a sequence of characters containing no whitespace, and not matching any other data type. (Note: this is not a very good definition). Typically symbols are lower case, and words are separated by a `-`. Examples: -``` + +```lisp this-is-a-symbol ; you can have weird symbols too: #f @@ -76,9 +88,9 @@ __WEIRDLY-NamedSymbol ; this is weird, but OK. ``` ## Reader Macros + The reader has some default macros which are common in Scheme/LISP: - `'x` will be replaced with `(quote x)` - `` `x`` will be replaced with `(quasiquote x)` - `,x` will be replaced with `(unquote x)` - `,@` will be replaced with `(unquote-splicing x)` - diff --git a/docs/markdown/registers.md b/docs/markdown/registers.md index d0ff404620..1045d421ac 100644 --- a/docs/markdown/registers.md +++ b/docs/markdown/registers.md @@ -1,8 +1,9 @@ -## Registers +# Registers + Although modern computers are much faster than the PS2, and we could probably get away with a really inefficient register allocation scheme, I think it's worth it to get this right. - ## Register differences between MIPS and x86-64 + The PS2's MIPS processor has these categories of register: - General Purpose. They are 128-bit, but usually only lower 64 bits are used. 32 registers, each 128-bits. - Floating point registers. 32 registers, each for a 32-bit float. @@ -75,8 +76,8 @@ And the x86-64 GPR map - `r14`: symbol table - `r15`: offset pointer - ### Plan for Memory Access + The PS2 uses 32-bit pointers, and changing the pointer size is likely to introduce bugs, so we will keep using 32-bit pointers. Also, GOAL has some hardcoded checks on the value for pointers, so we need to make sure the memory appears to the program at the correct address. To do this, we have separate "GOAL Pointers" and "real pointers". The "real pointers" are just normal x86-64 pointers, and the "GOAL Pointer" is an offset into a main memory array. A "real pointer" to the main memory array is stored in `r15` (offset pointer) when GOAL code is executing, and the GOAL compiler will automatically add this to all memory accesses. @@ -96,6 +97,7 @@ The other registers are less clear. The process pointer can probably be a real Right now I'm leaning toward 2, but it shouldn't be a huge amount of work to change if I'm wrong. ### Plan for Function Call and Arguments + In GOAL for MIPS, function calls are weird. Functions are always called by register using `t9`. There seems to be a different register allocator for function pointers, as nested function calls have really wacky register allocation. In GOAL-x86-64, this restriction will be removed, and a function can be called from any register. (see next section for why we can do this) Unfortunately, GOAL's 128-bit function arguments present a big challenge. When calling a function, we can't know if the function we're calling is expecting an integer, float, or 128-bit integer. In fact, the caller may not even know if it has an integer, float, or 128-bit integer. The easy and foolproof way to get this right is to use 128-bit `xmm` registers for all arguments and return values, but this will cause a massive performance hit and increase code size, as we'll have to move values between register types constantly. The current plan is this: @@ -104,8 +106,10 @@ Unfortunately, GOAL's 128-bit function arguments present a big challenge. When - We'll compromise for 128-bit function calls. When the compiler can figure out that the function being called expects or returns a 128-bit value, it will use the 128-bit calling convention. In all other cases, it will use 64-bit. There aren't many places where 128-bit integer are used outside of inline assembly, so I suspect this will just work. If there are more complicated instances (call a function pointer and get either a 64 or 128-bit result), we will need to special case them. ### Plan for Static Data + The original GOAL implementation always called functions by using the `t9` register. So, on entry to a function, the `t9` register contains the address of the function. If the function needs to access static data, it will move this `fp`, then do `fp` relative addressing to load data. Example: -``` + +```nasm function-start: daddiu sp, sp, -16 ;; allocate space on stack sd fp, 8(sp) ;; back up old fp on stack @@ -116,8 +120,9 @@ function-start: To copy this exactly on x86 would require reserving two registers equivalent to `t9` and `gp`. A better approach for x86-64 is to use "RIP relative addressing". This can be used to load memory relative to the current instruction pointer. This addressing mode can be used with "load effective address" (`lea`) to create pointers to static data as well. ### Plan for Memory + Access memory by GOAL pointer in `rx` with constant offset (optionally zero): -``` + +```nasm mov rdest, [roff + rx + offset] ``` - diff --git a/docs/markdown/repl.md b/docs/markdown/repl.md new file mode 100644 index 0000000000..b4ef060bd2 --- /dev/null +++ b/docs/markdown/repl.md @@ -0,0 +1,304 @@ +# Compiler REPL + +When you start the OpenGOAL compiler, you'll see a prompt like this: + +```lisp +OpenGOAL Compiler 0.2 +g > +``` + +The `g` indicates that you can input OpenGOAL compiler commands. For a listing of common commands run: + +```lisp +(repl-help) +``` + +## Connecting To Target Example + +In order to execute OpenGOAL code, you must connect to the listener. + +```lisp +;; we cannot execute OpenGOAL code unless we connect the listener +g > (+ 1 2 3) +REPL Error: Compilation generated code, but wasn't supposed to + +;; connect to the target +g > (lt) +[Listener] Socket connected established! (took 0 tries). Waiting for version... +Got version 0.2 OK! +[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2000000000, tid = 1297692 + +;; execute OpenGOAL code +gc > (+ 1 2 3) +6 + +;; quit the compiler and reset the target for next time. +gc > (e) +[Listener] Closed connection to target +``` + +Once we are connected, we see that there is a `gc` prompt. This indicates that the listener has an open socket connection. Now the REPL will accept both compiler commands and OpenGOAL source code. All `(format #t ...` debugging prints (like `printf`) will show up in this REPL. Each time you run something in the REPL, the result is printed as a decimal number. If the result doesn't make sense to print as a decimal, or there is no result, you will get some random number. + +In the future there will be a fancier printer here. + +## General Command Listing + +### `(e)` + +```lisp +(e) +``` + +Exit the compiler once the current REPL command is finished. Takes no arguments. If we are connected to a target through the listener, attempts to reset the target. + +### `(:exit)` + +Exit Compiler + +```lisp +(:exit) +``` + +Same as `(e)`, just requires more typing. `(e)` is actually a macro for `(:exit)`. Takes no arguments. + +### `(lt)` + +Listen to Target + +```lisp +(lt ["ip address"] [port-number]) +``` + +Connect the listener to a running target. The IP address defaults to `"127.0.0.1"` and the port to `8112` (`DECI2_PORT` in `listener_common.h`). These defaults are usually what you want, so you can just run `(lt)` to connect. + +Example: + +```lisp +g > (lt) +[Listener] Socket connected established! (took 0 tries). Waiting for version... +Got version 0.2 OK! +[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2000000000, tid = 1296302 +gc > +``` + +### `r` + +Reset the target. + +```lisp +(r ["ip address"] [port-number]) +``` + +Regardless of the current state, attempt to reset the target and reconnect. After this, the target will have nothing loaded. Like with `(lt)`, the default IP and port are probably what you want. + +Note: `r` is actually a macro. + +### `shutdown-target` + +If the target is connected, make it exit. + +```lisp +(shutdown-target) +``` + +The target will print + +``` +GOAL Runtime Shutdown (code 2) +``` + +before it exits. + +### `:status` + +Ping the target. + +```lisp +(:status) +``` + +Send a ping-like message to the target. Requires the target to be connected. If successful, prints nothing. Will time-out and display and error message if the GOAL kernel or code dispatched by the kernel is stuck in an infinite loop. Unlikely to be used often. + +## Compiler Forms - Compiler Commands + +These forms are used to control the GOAL compiler, and are usually entered at the GOAL REPL, or as part of a macro that's executed at the GOAL REPL. These shouldn't be used in GOAL source code. + +### `reload` + +Reload the GOAL compiler +```lisp +(reload) +``` + +Disconnect from the target and reset all compiler state. This is equivalent to exiting the compiler and opening it again. + +### `get-info` + +Get information about something. + +```lisp +(get-info ) +``` + +Use `get-info` to see what something is and where it is defined. + +For example: + +```lisp +;; get info about a global variable: +g > (get-info *kernel-context*) +[Global Variable] Type: kernel-context Defined: text from goal_src/kernel/gkernel.gc, line: 88 +(define *kernel-context* (new 'static 'kernel-context + +;; get info about a function. This particular function is forward declared, so there's an entry for that too. +;; global functions are also global variables, so there's a global variable entry as well. +g > (get-info fact) +[Forward-Declared] Name: fact Defined: text from goal_src/kernel/gcommon.gc, line: 1098 +(define-extern fact (function int int)) + +[Function] Name: fact Defined: text from kernel/gcommon.gc, line: 1099 +(defun fact ((x int)) + +[Global Variable] Type: (function int int) Defined: text from goal_src/kernel/gcommon.gc, line: 1099 +(defun fact ((x int)) + +;; get info about a type +g > (get-info kernel-context) +[Type] Name: kernel-context Defined: text from goal_src/kernel/gkernel-h.gc, line: 114 +(deftype kernel-context (basic) + +;; get info about a method +g > (get-info reset!) +[Method] Type: collide-sticky-rider-group Method Name: reset! Defined: text from goal_src/engine/collide/collide-shape-h.gc, line: 48 +(defmethod reset! collide-sticky-rider-group ((obj collide-sticky-rider-group)) +[Method] Type: collide-overlap-result Method Name: reset! Defined: text from goal_src/engine/collide/collide-shape-h.gc, line: 94 +(defmethod reset! collide-overlap-result ((obj collide-overlap-result)) +[Method] Type: load-state Method Name: reset! Defined: text from goal_src/engine/level/load-boundary.gc, line: 9 +(defmethod reset! load-state ((obj load-state)) + +;; get info about a constant +g > (get-info TWO_PI) +[Constant] Name: TWO_PI Value: (the-as float #x40c90fda) Defined: text from goal_src/engine/math/trigonometry.gc, line: 34 +(defconstant TWO_PI (the-as float #x40c90fda)) + +;; get info about a built-in form +g > (get-info asm-file) +[Built-in Form] asm-file +``` + +### `autocomplete` + +Preview the results of the REPL autocomplete: + +```lisp +(autocomplete ) +``` + +For example: + +```lisp +g > (autocomplete *) + * + *16k-dead-pool* + *4k-dead-pool* +... +Autocomplete: 326/1474 symbols matched, took 1.29 ms +``` + +### `seval` + +Execute GOOS code. + +```lisp +(seval form...) +``` + +Evaluates the forms in the GOOS macro language. The result is not returned in any way, so it's only useful for getting side effects. It's not really used other than to bootstrap some GOAL macros for creating macros. + +### `asm-file` + +Compile a file. + +```lisp +(asm-file "file-name" [:color] [:write] [:load] [:no-code]) +``` + +This runs the compiler on a given file. The file path is relative to the `jak-project` folder. These are the options: +- `:color`: run register allocation and code generation. Can be omitted if you don't want actually generate code. Usually you want this option. +- `:write`: write the object file to the `out/obj` folder. You must also have `:color` on. You must do this to include this file in a DGO. +- `:load`: send the object file to the target with the listener. Requires `:color` but not `:write`. There may be issues with `:load`ing very large object files (believed fixed). +- `:disassemble`: prints a disassembly of the code by function. Currently data is not disassebmled. This code is not linked so references to symbols will have placeholder values like `0xDEADBEEF`. The IR is printed next to each instruction so you can see what symbol is supposed to be linked. Requires `:color`. +- `:no-code`: checks that the result of processing the file generates no code or data. This will be true if your file contains only macros / constant definition. The `goal-lib.gc` file that is loaded by the compiler automatically when it starts must generate no code. You can use `(asm-file "goal_src/goal-lib.gc" :no-code)` to reload this file and double check that it doesn't generate code. + +To reduce typing, there are some useful macros: +- `(m "filename")` is "make" and does a `:color` and `:write`. +- `(ml "filename")` is "make and load" and does a `:color` and `:write` and `:load`. This effectively replaces the previous version of file in the currently running game with the one you just compiled, and is a super useful tool for quick debugging/iterating. +- `(md "filename")` is "make debug" and does a `:color`, `:write`, and `:disassemble`. It is quite useful for working on the compiler and seeing what code is output. +- `(build-game)` does `m` on all game files and rebuilds DGOs +- `(blg)` (build and load game) does `build-game` then sends commands to load KERNEL and GAME CGOs. The load is done through DGO loading, not `:load`ing individual object files. + +### `asm-data-file` + +Build a data file. + +```lisp +(asm-data-file tool-name "file-name") +``` + +The `tool-name` refers to which data building tool should be used. For example, this should be `game-text` when building the game text data files. + +There's a macro `(build-data)` which rebuilds everything. + +### `gs` + +Enter a GOOS REPL. + +```lisp +(gs) +``` + +Example: + +```scheme +g> (gs) +goos> (+ 1 2 3) +6 +goos> (exit) +() +``` + +mainly useful for debugging/trying things out in GOOS. The GOOS REPL shares its environment with the GOOS interpreter used by the compiler, so you can inspect/modify things for debugging with this. Likely not used much outside of initial debugging. + +### `set-config!` + +```lisp +(set-config! config-name config-value) +``` + +Used to set compiler configuration. This is mainly for debugging the compiler and enabling print statements. There is a `(db)` macro which sets all the configuration options for the compiler to print as much debugging info as possible. Not used often. + +### `in-package` + +```lisp +(in-package stuff...) +``` + +The compiler ignores this. GOAL files evidently start with this for some reason related to emacs. + +### `build-dgos` + +```lisp +(build-dgos "path to dgos description file") +``` + +Builds all the DGO files described in the DGO description file. See `goal_src/builds/dgos.txt` for an example. This just packs existing things into DGOs - you must have already built all the dependencies. + +In the future, this may become part of `asm-data-file`. + +### `add-macro-to-autocomplete` + +```lisp +(add-macro-to-autocomplete macro-name) +``` + +Makes the given name show up as a macro in the GOAL REPL. Generating macros may be done programmatically using GOOS and this form can be used to make these show up in the autocomplete. This also makes the macro known to `get-info` which will report that the macro was defined at the location where the macro which expanded to `add-macro-to-autocomplete` is located in GOAL code. This is used internally by `defmacro`. diff --git a/docs/markdown/syntax.md b/docs/markdown/syntax.md new file mode 100644 index 0000000000..11a0b74e20 --- /dev/null +++ b/docs/markdown/syntax.md @@ -0,0 +1,239 @@ +# OpenGOAL Syntax & Examples + +## The Basics + +### Atoms + +An "atom" in Lisp is a form that can't be broken down into smaller forms. For example `1234` is an atom, but `(1234 5678)` is not. OpenGOAL supports the following atoms: + +### Integers + +All integers are by default `int`, a signed 64-bit integer. You can use: +- decimal: Like `123` or `-232`. The allowable range is `INT64_MIN` to `INT64_MAX`. +- hex: Like `#x123`. The allowable range is `0` to `UINT64_MAX`. Values over `INT64_MAX` will wrap around. +- binary: Like `#b10101010`. The range is the same as hex. +- character: + - Most can be written like `#\c` for the character `c`. + - Space is `#\\s` + - New Line is `#\\n` + - Tab is `#\\t` + +GOAL has some weird behavior when it comes to integers. It may seem complicated to describe, but it really makes the implementation simpler - the integer types are designed around the available MIPS instructions. + +Integers that are used as local variables (defined with `let`), function arguments, function return values, and intermediate values when combining these are called "register integers", as the values will be stored in CPU registers. + +Integers that are stored in memory as a field of a `structure`/`basic`, an element in an array, or accessed through a `pointer` are "memory integers", as the values will need to be loaded/stored from memory to access them. + +The "register integer" types are `int` and `uint`. They are 64-bit and mostly work exactly like you'd expect. Multiplication, division, and mod, are a little weird and are documented separately. + +The "memory integer" types are `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, and `uint64`. + +Conversions between these types are completely automatic - as soon as you access a "memory integer", it will be converted to a "register integer", and trying to store a "register integer" will automatically convert it to the appropriate "memory integer". It (should be) impossible to accidentally get this wrong. + +#### Side Note + +- It's not clear what types `(new 'static 'integer)` or `(new 'stack 'integer)` are, though I would assume both are memory. +- If there aren't enough hardware registers, "register integers" can be spilled to stack, but keep their "register integer" types. This process should be impossible to notice, so you don't have to worry about it. + +### String + +A string generates a static string constant. Currently the "const" of this string "constant" isn't enforced. Creating two identical string constants creates two different string objects, which is different from GOAL and should be fixed at some point. + +The string data is in quotes, like in C. The following escapes are supported: +- Newline: `\n` +- Tab: `\t` +- The `\` character: `\\` +- The `"` character: `\"` +- Any character: `\cXX` where `XX` is the hex number for the character. + +### Float + +Any number constant with a decimal in it. The trailing and leading zeros and negative sign is flexible, so you can do any of these: +- `1.`, `1.0`, `01.`, `01.0` +- `.1`, `0.1`, `.10`, `0.10` +- `-.1`, `-0.1`, `-.10`, `-0.10` + +Like string, it creates a static floating point constant. In later games the float was inlined instead of being a static constant. + +### Symbol + +Use `symbol-name` to get the value of a symbol and `'symbol-name` to get the symbol object. + +### Comments + +Use `;` for line comments and `#|` and `|#` for block comments. + +## Compiling a list + +When the compiler encounters a list like `(a b c)` it attempts to parse in multiple ways in this order: +1. A compiler form +2. A GOOS macro +3. An enum (not yet implemented) +4. A function or method call + +## Compiling an integer + +Integers can be specified as +- decimal: `1` or `-1234` (range of `INT64_MIN` to `INT64_MAX`) +- hex: `#x123`, `#xbeef` (range of `0` to `UINT64_MAX`) +- binary: `#b101010` (range of `0` to `UINT64_MAX`) + +All integers are converted to the signed "integer in variable" type called `int`, regardless of how they are specified. +Integer "constant"s are not stored in memory but instead are generated by code, so there's no way to modify them. + +## Compiling a string + +A string constant can be specified by just putting it in quotes. Like `"this is a string constant"`. +There is an escape code `\` for string: +- `\n` newline +- `\t` tab character +- `\\` the `\` character +- `\"` the `"` character +- `\cXX` where `XX` is a two character hex number: insert this character. +- Any other character following a `\` is an error. + +OpenGOAL stores strings in the same segment of the function which uses the string. I believe GOAL does the same. + +In GOAL, string constants are pooled per object file (or perhaps per segment)- if the same string appears twice, it is only included once. OpenGOAL currently does not pool strings. If any code is found that modifies a string "constant", or if repeated strings take up too much memory, string pooling will be added. + +For now I will assume that string constants are never modified. + +## Compiling a float + +A floating point constant is distinguished from an integer by a decimal point. Leading/trailing zeros are optional. Examples of floats: `1.0, 1., .1, -.1, -0.2`. Floats are stored in memory, so it may be possible to modify a float constant. For now I will assume that float constants are never modified. It is unknown if they are pooled like strings. + +Trivia: Jak 2 realized that it's faster to store floats inline in the code. + +## Compiling a symbol + +A `symbol` appearing in code is compiled by trying each of these in the following order +1. Is it `none`? (see section on `none`) +2. Try `mlet` symbols +3. Try "lexical" variables (defined in `let`) +4. Try global constants +5. Try global variables (includes named functions and all types) + +## The Special `none` type + +Anything which doesn't return anything has a return type of `none`, indicating the return value can't be used. This is similar to C's `void`. + +## GOAL Structs vs. C Structs + +There is one significant difference between C and GOAL when it comes to structs/classes - GOAL variables can only be references to structs. + +As an example, consider a GOAL type `my-type` and a C type `my_type`. In C/C++, a variable of type `my_type` represents an entire copy of a `my_type` object, and a `my_type*` is like a reference to an existing `my_type` object. In GOAL, an object of `my-type` is a reference to an existing `my-type` object, like a C `my_type*`. There is no equivalent to a C/C++ `my_type`. + +As a result you cannot pass or return a structure by value. + +Another way to explain this is that GOAL structures (including `pair`) always have reference semantics. All other GOAL types have value semantics. + +## Pointers + +GOAL pointers work a lot like C/C++ pointers, but have some slight differences: +- A C `int32_t*` is a GOAL `(pointer int32)` +- A C `void*` is a GOAL `pointer` +- In C, if `x` is a `int32_t*`, `x + 1` is equivalent to `uintptr_t(x) + sizeof(int32_t)`. In GOAL, all pointer math is done in units of bytes. +- In C, you can't do pointer math on a `void*`. In GOAL you can, and all math is done in units of bytes. + +In both C and GOAL, there is a connection between arrays and pointers. A GOAL array field will have a pointer-to-element type, and a pointer can be accessed as an array. + +One confusing thing is that a `(pointer int32)` is a C `int32_t*`, but a `(pointer my-structure-type)` is a C `my_structure_type**`, because a GOAL `my-structure-type` is like a C `my_structure_type*`. + +## Inline Arrays + +One limitation of the system above is that an array of `my_structure_type` is actually an array of references to structures (C `object*[]`). It would be more efficient if instead we had an array of structures, laid out together in memory (C `object[]`). + +GOAL has a "inline array" to represent this. A GOAL `(inline-array thing)` is like a C `thing[]`. The inline-array can only be used on structure types, as these are the only reference types. + +## Fields in Structs + +For a field with a reference type (structure/basic) +- `(data thing)` is like C `Thing* data;` +- `(data thing :inline #t)` is like C `Thing data;` +- `(data thing 12)` is like C `Thing* data[12];`. The field has `(pointer thing)` type. +- `(data thing 12 :inline #t)` is like `Thing data[12];`. The field has `(inline-array thing)` type + +For a field with a value type (integer, etc) +- `(data int32)` is like C `int32_t data;` +- `(data int32 12)` is like `int32_t data[12];`. The field has `(array int32)` type. + +Using the `:inline #t` option on a value type is not allowed. + +## Dynamic Structs + +GOAL structure can be dynamically sized, which means their size isn't determined at compile time. Instead the user should implement `asize-of` to return the actual size. + +This works by having the structure end in an array of unknown size at compile time. In a dynamic structure definition, the last field of the struct should be an array with an unspecified size. To create this, add a `:dynamic #t` option to the field and do not specify an array size. This can be an array of value types, an array of reference types, or an inline-array of reference types. + +### Unknown + +Is the `size` of a dynamic struct: +- size assuming the dynamic array has 0 elements (I think it's this) +- size assuming the dynamic array doesn't + +These can differ by padding for alignment. + + +## How To Create GOAL Objects - `new` + +GOAL has several different ways to create objects, all using the `new` form. + +### Heap Allocated Objects + +A new object can be allocated on a heap with `(new 'global 'obj-type [new-method-arguments])`. +This simply calls the `new` method of the given type. You can also replace `'global` with `'debug` to allocate on the debug heap. +Currently these are the only two heaps supported, in the future you will be able to call the new method with other arguments +to allow you to do an "in place new" or allocate on a different heap. + +This will only work on structures and basics. If you want a heap allocated float/integer/pointer, create an array of size 1. +This will work on dynamically sized items. + +### Heap Allocated Arrays + +You can construct a heap array with `(new 'global 'inline-array 'obj-type count)` or `(new 'global 'array 'obj-type count)`. +These objects are not initialized. Note that the `array` version creates a `(pointer obj-type)` plain array, +__not__ a GOAL `array` type fancy array. In the future this may change because it is confusing. + +Because these objects are uninitialized, you cannot provide constructor arguments. +You cannot use this on dynamically sized member types. However, the array size can be determined at runtime. + +### Static Objects + +You can create a static object with `(new 'static 'obj-type [field-def]...)`. It can be a structure, basic, bitfield, array, boxed array, or inline array. +Each field def looks like `:field-name field-value`. The `field-value` is evaluated at compile time. Fields +can be integers, floats, symbols, pairs, strings, or other statics. These field values may come from macros or GOAL constants. + +For bitfields, there is an exception, and fields can be set to expression that are not known at compile time. The compiler will generate the appropriate code to combine the values known at compile time and run time. This exception does not apply to a bitfield inside of another `(new 'static ...)`. + +Fields which aren't explicitly initialized are zeroed, except for the type field of basics, which is properly initialized to the correct type. + +This does not work on dynamically sized structures. + +### Stack Allocated Arrays + +Currently only arrays of integers, floats, or pointers can be stack allocated. +For example, use `(new 'stack ''array 'int32 1)` to get a `(pointer int32)`. Unlike heap allocated arrays, these stack arrays +must have a size that can be determined at compile time. The objects are uninitialized. + +### Stack Allocated Structures + +Works like heap allocated, the objects are initialized with the constructor. The constructor must support "stack mode". Using `object-new` supports stack mode so usually you don't have to worry about this. The structure's memory will be memset to 0 with `object-new` automatically. + +### Defining a `new` Method + +TODO + +## Array Spacing + +In general, all GOAL objects are 16-byte aligned and the boxing system requires this. All heap memory allocations are 16-byte aligned too, so this is usually not an issue. + +## Truth + +Everything is true except for `#f`. This means `0` is true, and `'()` is true. +The value of `#f` can be used like `nullptr`, at least for any `basic` object. It's unclear if `#f` can/should be used as a null for other types, including `structure`s or numbers or pointers. + +Technical note: the hex number `0x147d24` is considered false in Jak 1 NTSC due to where the symbol table happened to be allocated. However, checking numbers for true/false shouldn't be done, you should use `(zero? x)` instead. + +## Empty Pair + +TODO \ No newline at end of file diff --git a/docs/markdown/type_system.md b/docs/markdown/type_system.md index 6d8c343938..9238c0f86d 100644 --- a/docs/markdown/type_system.md +++ b/docs/markdown/type_system.md @@ -1,63 +1,104 @@ -Type System --------------- -This document explains the GOAL type system. The GOAL type system supports runtime typing, single inheritance, virtual methods, and dynamically sized structures. - - Everything in GOAL has a type at compile time. A subset of compile-time types are also available in the runtime as objects with the same name as the type. For example, there is a `string` type, and at runtime there is a global object named `string` which is an object of type `type` containing information about the `string` type. - - Some objects have runtime type information, and others don't. Objects which have runtime type information can have their type identified at runtime, and are called "boxed objects". Objects without runtime type information are called "unboxed objects". An unboxed object cannot reliably be detected as a unboxed object - you can't write a function that takes an arbitrary object and tells you if its boxed or not. However, boxed objects can always be recognized as boxed. - - All types have a parent type, and all types descend from the parent type `object`, except for the special type `none` (and maybe `_type_`, but more on this later). The `none` type doesn't exist in the runtime and is used to represent an invalid value that the compiler should not use. For example, the return type of a function which doesn't return anything is `none`, and attempting to use this value should cause an error. - - Here are some important special types: - - `object` - the parent of all types - - `structure` - parent type of any type with fields - - `basic` - parent type of any `structure` with runtime type information. - - All types have methods. Objects have access to all of their parents methods, and may override parent methods. All types have these 9 methods: - - - `new` - like a constructor, returns a new object. It's not used in all cases, and on all types, and needs more documentation on when specifically it is used. - - `delete` - basically unused, but like a destructor. Often calls `kfree`, which does nothing. - - `print` - prints a short, one line representation of the object to the `PrintBuffer` - - `inspect` - prints a multi-line description of the object to the `PrintBuffer`. Usually auto-generated by the compiler and prints out the name and value of each field. - - `length` - Returns a length if the type has something like a length (number of characters in string, etc). Otherwise returns 0. Usually returns the number of filled slots, instead of the total number of allocated slots, when there is possibly a difference. - - `asize-of` - Gets the size in memory of the entire object. Usually this just looks this up from the appropriate `type`, unless it's dynamically sized. - - `copy` - Create a copy of this object on the given heap. Not used very much? - - `relocate` - Some GOAL objects will be moved in memory by the kernel as part of the compacting actor heap system. After being moved, the `relocate` method will be called with the offset of the move, and the object should fix up any internal pointers which may point to the old location. It's also called on v2 objects loaded by the linker when they are first loaded into memory. - - `memusage` - Not understood yet, but probably returns how much memory in bytes the object uses. Not supported by all objects. - -Usually a method which overrides a parent method must have the same argument and return types. The only exception is `new` methods, which can have different argument/return types from the parent. (Dee the later section on `_type_` for another exception) - -The compiler's implementation for calling a method is: -- Is the type a basic? - - If so, look up the type using runtime type information - - Get the method from the vtable -- Is the type not a basic? - - Get the method from the vtable of the compile-time type - - Note that this process isn't very efficient - instead of directly linking to the slot in the vtable (one deref) it first looks up the `type` by symbol, then the slot (two derefs). I have no idea why it's done this way. - - -In general, I suspect that the method system was modified after GOAL was first created. There is some evidence that types were once stored in the symbol table, but were removed because the symbol table became full. This could explain some of the weirdness around method calls/definition rules, and the disaster `method-set!` function. - -GOAL Value Types --------------------- +# OpenGOAL's Type System + +This document explains the GOAL type system. The GOAL type system supports runtime typing, single inheritance, virtual methods, and dynamically sized structures. + +There's a single type system library, located in `common/type_system`. It will be used in both the decompiler and compiler. The plan is to have a single `all_types.gc` file which contains all type information (type definitions and types of globals). The decompiler will help generate this, but some small details may need to be filled in manually for some types. Later versions of the decompiler can use this information to figure out what fields of types are being accessed. We can also add a test to make sure that types defined in the decompiled game match `all_types.gc`. + +The main features are: + +- `TypeSystem` stores all type information and provides a convenient way to add new types or request information about existing types. +- `Type` information about a GOAL Type. A "base GOAL type" is identified by a single unique string. Examples: `function`, `string`, `vector3h`. +- `TypeSpec` a way to specify either `Type` or a "compound type". Compound types are used to create types which represent specific function types (function which takes two integer arguments and returns a string), or specific pointer/array types (pointer to an integer). These can be represented as (possibly nested) lists, like `(pointer integer)` or `(function (integer integer) string)`. +- Type Checking for compiler +- Parsing of type definitions for compiler +- Lowest common ancestor implementation for compiler to figure out return types for branching forms. +- Logic to catch multiple incompatible type definitions for both compiler warnings and decompiler sanity checks + +The type system will store: +- The types of all global variables (this includes functions) +- Information about all types: + - Fields/specific details on how to load from memory, alignment, sign extension, size in arrays, etc... + - Parent type + - Methods not defined for the parent. + +It's important that all of the type-related info is stored/calculated in a single location. The proof of concept compiler did not have the equivalent of `TypeSystem` and scattered field/array access logic all over the place. This was extremely confusing to get right. + +If type information is specified multiple times, and is also inconsistent, the TypeSystem can be configured to either throw an exception or print a warning. + +This will be a big improvement over the "proof of concept" compiler which did not handle this situation well. When debugging GOAL you will often put the same file through the compiler again and again, changing functions, but not types. In this case, there should be no warnings. If the type does change, it should warn (as old code may exist that uses the old type layout), but shouldn't cause the compiler to abort, error, or do something very unexpected. + +## Compile-time vs Run-time + +The types in the runtime are only a subset of the compile time types. Here are the rules I've discovered so far +- Any compound types become just the first type. So `(pointer my-type)` becomes `pointer`. +- The `inline-array` class just becomes `pointer`. +- Some children of integers disappear, but others don't. The rules for this aren't known yet. + +## Types of Types + +Everything in GOAL has a type at compile time. A subset of compile-time types are also available in the runtime as objects with the same name as the type. For example, there is a `string` type, and at runtime there is a global object named `string` which is an object of type `type` containing information about the `string` type. + +Some objects have runtime type information, and others don't. Objects which have runtime type information can have their type identified at runtime, and are called "boxed objects". Objects without runtime type information are called "unboxed objects". An unboxed object cannot reliably be detected as a unboxed object - you can't write a function that takes an arbitrary object and tells you if its boxed or not. However, boxed objects can always be recognized as boxed. + +All types have a parent type, and all types descend from the parent type `object`, except for the special type `none` (and maybe `_type_`, but more on this later). The `none` type doesn't exist in the runtime and is used to represent an invalid value that the compiler should not use. For example, the return type of a function which doesn't return anything is `none`, and attempting to use this value should cause an error. + +Here are some important special types: +- `object` - the parent of all types +- `structure` - parent type of any type with fields +- `basic` - parent type of any `structure` with runtime type information. + +### Value Types + Some GOAL types are "value types", meaning they are passed by value when used as arguments to functions, return values from functions, local variables, and when using `set!`. These are always very small and fit directly into the CPU registers. Some example value types: - Floating point numbers - Integers +### Reference Types -GOAL Reference Types --------------------- Other GOAL types are "reference types", meaning they act like a reference to data when used as arguments to functions, return values from functions, local variables, and when using `set!`. The data can be allocated on a heap, on the stack, or as part of static data included when loading code (which is technically also on a heap). All structure/basic types are reference types. You can think of these like C/C++ pointers or references, which is how it is implemented. The difference is that there's no special notation for this. A GOAL `string` object is like a C/C++ `string*` or `string&`. A GOAL "pointer to reference type" is like a C/C++ `my_type**`. Note - this is quite a bit different from C/C++. In C++ you can have a structure with value semantics (normal), or reference semantics (C++ reference or pointer). In GOAL, there is no value semantics for structures! This is great because it means function arguments/variables always fit into registers. - +### Dynamic Size Types +Any type which ends with a dynamic array as the last field is dynamic. For these, it's a good idea to implement the `asize-of` method. + +### Compound Types + +A compound type is a type like "a pointer to an int64" or "a function which takes int as an argument and returns a string". These exist only at compile time, and get simplified at runtime. For example, all pointers become `pointer` and all functions become `function`. (The one exception to this seems to be `inline-array-class` stuff, but this is not yet supported in OpenGOAL). + +#### Pointer + +Pointers work like you would expect. They can only point to memory types - you can't have a `(pointer int)`, instead you must have a `(pointer int32)` (for example). Note that a `(pointer basic)` is like a C++ `basic**` as `basic` is already like a C++ pointer to struct. You can nest these, like `(pointer (pointer int64))`. If you want a pointer with no type, (like C++ `void*`) just use a plain `pointer`. The `(pointer none)` type is invalid. + +Like in C/C++, you can use array indexing with a pointer. One thing to note is that a `(pointer basic)` (or pointer to any reference type) is like a C++ "array of pointers to structs". To get the C++ "array of structs", you need an `inline-array`. + +#### Arrays + +For value types, arrays work as you expect. They have type `(pointer your-type)`. Arrays of references come in two versions: +- Array of references: `(pointer your-type)`, like a C array of pointers +- Array of inline objects: `(inline-array your-type)`, like a C array of structs + +The default alignment of structs is 16 bytes, which is also the minimum alignment of `kmalloc`, and the minimum alignment used when using a reference type as an inline field. However, it's possible to violate this rule in a `(inline-array your-type)` to be more efficient. The `your-type` can set a flag indicating it should be packed in an inline array. + +I believe the alignment then becomes the maximum of the minimum alignment of the `your-type` fields. So if you have a type with two `uint32`s (alignment 4 bytes), an `(inline-array your-type)` can then have spacing of 8 bytes, instead of the usual minimum 16. The behavior of a `(field-name your-type :inline #t)` is unchanged and will still align at the minimum of 16 bytes. I _believe_ that the first element of the array will still have an alignment of 16. + +##### Inline Arrays + +These are only valid for reference types. They refer to an array of the actual data (like C array of structs) rather than an array of reference (like C array of pointers to structs, or GOAL `(pointer structure)`). At runtime, `inline-array` becomes pointer. + +For an inline array of basics, elements are 16-byte aligned. For `structure`s that aren't `basic`, the alignment is usually the minimum alignment of all members of the structure, but there is an option to make it 16-byte aligned if needed. + +For information about how to create these arrays, see `deftype` (fields in a type) and `new` (create just an array) sections. + +#### Function + +Function compound types look like this `(function arg0-type arg1-type... return-type)`. There can be no arguments. The `return-type` must always be specified, and should be `none` if there is no return value. The argument types themselves can be compound types. In order to call a function, you must have a compound function type - a `function` by itself cannot be called. + +### Field Definitions -GOAL Fields --------------- GOAL field definitions look like this: `(name type-name [optional stuff])` @@ -75,7 +116,7 @@ There are many combinations of reference/value, dynamic/not-dynamic, inline/not- - Value type, `:dynamic #t`: the field marks the beginning of an array (of unknown size). Field type is `(pointer your-type)` - Value type, with array size: the field marks the beginning of an array (of known size). Field type is `(pointer your-type)` - Value type, with `:inline #t`: invalid in all cases. -- Reference type, no modifiers: a single reference is stored in the type. Type of field is `your-type` (a C++ pointer). +- Reference type, no modifiers: a single reference is stored in the type. Type of field is `your-type` (a C++ pointer). - Reference type, `:inline #t`: a single object is stored inside the type. Type of field is `your-type` still (a C++ pointer). The access logic is different to make this work. - Reference type, `:dynamic #t` or array size: the field marks the beginning of an **array of references**. Field type is `(pointer your-type)`. Like C array of pointers. - Reference type, `:inline #t` and (`:dynamic #t` or array size): the field marks the beginning of an **array of inline objects**. Field type is `(inline-array your-type)`. Like C array of structs. @@ -89,10 +130,10 @@ Bonus ones, for where the array is stored _outside_ of the type: Of course, you can combine these, to get even more confusing types! But this seems uncommon. -GOAL Field Placement ---------------------- +#### Field Placement + The exact rules for placing fields in GOAL types is unknown, but the simple approach of "place the next field as close as possible to the end of the last field" seems to get it right almost all the time. However, we need to be extra certain that we lay out type fields correctly because many GOAL types have overlapping fields. - + The theory I'm going with for now is: - The order of fields in the `inspect` method is the order fields are listed in in the type definition - In the rare cases this is wrong, this is due to somebody manually specifying an offset. @@ -102,92 +143,88 @@ As a result, we should specify offsets like this: - If we think a field was automatically placed, use `:offset-assert` to inform the compiler where we expect it to be. In this case it will still place the field automatically, but if the result is different from the `:offset-assert`, it will throw an error. - Avoid defining any fields without `:offset` or `:offset-assert` +## Built-in Types -GOAL Arrays ---------------- -For value types, arrays work as you expect. They have type `(pointer your-type)`. Arrays of references come in two versions: -- Array of references: `(pointer your-type)`, like a C array of pointers -- Array of inline objects: `(inline-array your-type)`, like a C array of structs +There are a number of built-in types. I use "abstract" type to refer to a type that is only a parent type. -The default alignment of structs is 16 bytes, which is also the minimum alignment of `kmalloc`, and the minimum alignment used when using a reference type as an inline field. However, it's possible to violate this rule in a `(inline-array your-type)` to be more efficient. The `your-type` can set a flag indicating it should be packed in an inline array. - - I believe the alignment then becomes the maximum of the minimum alignment of the `your-type` fields. So if you have a type with two `uint32`s (alignment 4 bytes), an `(inline-array your-type)` can then have spacing of 8 bytes, instead of the usual minimum 16. The behavior of a `(field-name your-type :inline #t)` is unchanged and will still align at the minimum of 16 bytes. I _believe_ that the first element of the array will still have an alignment of 16. +### `none` +This is a special type that represents "no information". This is the return type of a function which returns nothing, and also the return type of an expression that doesn't return anything. For example, the expression `(goto x)` does not produce a value, so its type is `none`. -There's a single type system library, located in `common/type_system`. It will be used in both the decompiler and compiler. The plan is to have a single `all_types.gc` file which contains all type information (type definitions and types of globals). The decompiler will help generate this, but some small details may need to be filled in manually for some types. Later versions of the decompiler can use this information to figure out what fields of types are being accessed. We can also add a test to make sure that types defined in the decompiled game match `all_types.gc`. - - The main features are: +### `object` +This is the parent type of all types. This is an abstract class. In a variable, this is always `object`, and can hold any `object`. In memory, this is either `object32` or `object64`. The `object32` can hold everything except for `binteger` and 64-bit integers. This type is neither boxed nor unboxed and is neither value nor reference. -- `TypeSystem` stores all type information and provides a convenient way to add new types or request information about existing types. -- `Type` information about a GOAL Type. A "base GOAL type" is identified by a single unique string. Examples: `function`, `string`, `vector3h`. -- `TypeSpec` a way to specify either `Type` or a "compound type". Compound types are used to create types which represent specific function types (function which takes two integer arguments and returns a string), or specific pointer/array types (pointer to an integer). These can be represented as (possibly nested) lists, like `(pointer integer)` or `(function (integer integer) string)`. -- Type Checking for compiler -- Parsing of type definitions for compiler -- Lowest common ancestor implementation for compiler to figure out return types for branching forms. -- Logic to catch multiple incompatible type definitions for both compiler warnings and decompiler sanity checks +### `structure` (child of `object`) +This is the parent type of all types with fields. This is an abstract class and a reference class. A `structure` can hold any `structure`, both in memory and in a variable. It is unboxed. -Compile Time vs. Run Time types ------------------------- -The types in the runtime are only a subset of the compile time types. Here are the rules I've discovered so far -- Any compound types become just the first type. So `(pointer my-type)` becomes `pointer`. -- The `inline-array` class just becomes `pointer`. -- Some children of integers disappear, but others don't. The rules for this aren't known yet. +### `basic` (child of `structure`) +This is the "boxed" version of `structure`. The first field of a basic is `type`, which contains the `type` of the object. It is boxed and a reference. A `basic` can hold any `basic`, both in memory and in a variable. -Special `_type_` for methods ----------------------------- -The first argument of a method always contains the object that the method is being called on. It also must have the type `_type_`, which will be substituted by the type system (at compile time) using the following rules: +### `symbol` (child of `basic`) +A symbol has a name and a value. The name is a string, and the value is an `object32`. Note that the value is an `object32` so you cannot store a 64-bit integer in a symbol. It is considered "bad" to store unboxed objects in symbols, though you can get away with it sometimes. -- At method definition: replace with the type that the method is being defined for. -- At method call: replace with the compile-time type of the object the method is being called on. +All `symbol`s are stored in the global symbol table, which is a hash table. As a result, you cannot have multiple symbols with the same name. A name is enough to uniquely determine the symbol. To get a symbol, use the syntax `'symbol-name`. To get the value, use `symbol-name`. -A method can have other arguments or a return value that's of type `_type_`. This special "type" will be replaced __at compile time__ with the type which is defining or calling the method. No part of this exists at runtime. It may seem weird, but there are two uses for this. +Each global variable, type, and named global function has a symbol for it which has the variable, type, or function as its value. The linker is able to perform symbol table lookups at link time and patch the code so you don't have to do a hash table lookup every time you access a global variable, function, or type. -The first is to allow children to specialize methods and have their own child type as an argument type. For example, say you have a method `is-same-shape`, which compares two objects and sees if they are the same shape. Suppose you first defined this for type `square` with -``` -(defmethod square is-same-shape ((obj1 square) (obj2 square)) - (= (-> obj1 side-length) (-> obj2 side-length)) - ) -``` +You can also use symbols as a efficient way to represent a enum. For example, a function may return `'error` or `'complete` as a status. The compiler is able to compare symbols for equality very efficiently (just a pointer comparison, as symbols are a reference type). -Then, if you created a child class of `square` called `rectangle` (this is a terrible way to use inheritance, but it's just an example), and overrode the `is-same-shape` method, you would have to have arguments that are `square`s, which blocks you from accessing `rectangle`-specific fields. The solution is to define the original method with type `_type_` for the first two arguments. Then, the method defined for `rectangle` also will have arguments of type `_type_`, which will expand to `rectangle`. +### `type` (child of `basic`) +A `type` stores information about an OpenGOAL type, including its size, parent, and name (stored as a `symbol`). It also stores the method table. Some OpenGOAL types (children of integers, bitfield types, enums, compounds types) do not have runtime types, and instead become the parent/base type. But these types cannot have runtime type information or methods and are pretty rare. It is a reference type, is boxed, and is dynamically sized (the method table's size is not fixed). +### `string` (child of `basic`) +A string. The string is null terminated and also stores the buffer size. This type is a reference type, is boxed, and is also dynamically sized. -The second use is for a return value. For example, the `print` and `inspect` methods both return the object that is passed to them, which will always be the same type as the argument passed in. If `print` was define as `(function object object)`, then `(print my-square)` would lose the information that the return object is a `square`. If `print` is a `(function _type_ _type_)`, the type system will know that `(print my-square)` will return a `square`. +### `function` (child of `basic`) +A function. Boxed and reference. It is a reference to a function, so it's like a C/C++ function pointer type. + +### `kheap` (child of `structure`) +A simple bump-allocated heap. Doesn't store the heap memory, just metadata. Supports allocating from either the top or the bottom. This is used as the memory allocation strategy for the global, debug, and level heaps. Unboxed, reference, not dynamic. + +### `array` (child of `basic`) +A "fancy" array. It is not yet implemented in OpenGOAL. + +### `pair` (child of `object`) +A pair. It is boxed. You should not make child types of `pair`. The two objects stored by the pair are `object32`s. + +### `pointer` (child of `object`) +It is a 32-bit value type containing a pointer. Not boxed, value type. See section on compound types for more information. + +### `number` (child of `object`) +Abstract type for all numbers. Value type. 64-bits. + +### `float` (child of `number`) +4-byte, single precision floating point number. Value type. + +### `integer` (child of `number`) +Abstract class for integer numbers. Child classes are `sinteger` (signed integer), `uinteger` (unsigned integer), and `binteger` (boxed-integer, always signed). These are all 64-bit types. + +Children of `sinteger` and `uinteger` are `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`. These are the size you expect, value types, and not boxed. These only exist as memory types. In a variable, there is only `int` and `uint`. These are both 64-bit types. All integer operations (math, logical, shifts) are 64-bit. + +The `binteger` is a boxed integer. It is a 61 bit signed integer (the other three bits are lost due to the number being boxed). There may actually be a `buinteger` (or `ubinteger`) but it doesn't exist in OpenGOAL at this point. + +#### Weird Built-in types that aren't supported yet. +- `vu-function` +- `link-block` +- `connectable` +- `file-stream` +- `inline-array` (class) + +## Unknown Areas + +### Inline Array Class -Inline Array Class --------------------- There's a weird `inline-array-class` type that's not fully understood yet. It uses `heap-base`. -Heap Base --------------- +### Heap Base + This is a field in `type`. What does it mean? It's zero for most types (at least the early types). -Second Size Field -------------------- +### Second Size Field + There are two fields in `type` for storing the size. The first one stores the exact size, and by default the second stores the size rounded up to the nearest 16 bytes. Why? Who uses it? Does it ever get changed? -The Type System -------------------- -The type system will store: -- The types of all global variables (this includes functions) -- Information about all types: - - Fields/specific details on how to load from memory, alignment, sign extension, size in arrays, etc... - - Parent type - - Methods not defined for the parent. - -It's important that all of the type-related info is stored/calculated in a single location. The proof of concept compiler did not have the equivalent of `TypeSystem` and scattered field/array access logic all over the place. This was extremely confusing to get right. +## TODO -If type information is specified multiple times, and is also inconsistent, the TypeSystem can be configured to either throw an exception or print a warning. - -This will be a big improvement over the "proof of concept" compiler which did not handle this situation well. When debugging GOAL you will often put the same file through the compiler again and again, changing functions, but not types. In this case, there should be no warnings. If the type does change, it should warn (as old code may exist that uses the old type layout), but shouldn't cause the compiler to abort, error, or do something very unexpected. - - -Method System --------------- -All type definitions should also define all the methods, in the order they appear in the vtable. I suspect GOAL had this as well because the method ordering otherwise seems random, and in some cases impossible to get right unless (at least) the number of methods was specified in the type declaration. - - -Todo ------ - [ ] Kernel types that are built-in - [ ] Signed/unsigned for a few built-in type fields - [ ] Tests for field placement logic (probably a full compiler test?) @@ -196,4 +233,3 @@ Todo - [ ] Stuff for decompiler - [ ] What field is here? - [ ] Export all deftypes - \ No newline at end of file