Dependency graph work - Part 1 - Preliminary work (#3505)

Relates to #1353 

This adds no new functionality or overhead to the compiler, yet. This is
the preliminary work that has:
- added code to the compiler in several spots to flag when something is
used without being properly required/imported/whatever (disabled by
default)
- that was used to generate project wide file dependencies (some
circulars were manually fixed)
- then that graph underwent a transitive reduction and the result was
written to all `jak1` source files.

The next step will be making this actually produce and use a dependency
graph. Some of the reasons why I'm working on this:
- eliminates more `game.gp` boilerplate. This includes the `.gd` files
to some extent (`*-ag` files and `tpage` files will still need to be
handled) this is the point of the new `bundles` form. This should make
it even easier to add a new file into the source tree.
- a build order that is actually informed from something real and
compiler warnings that tell you when you are using something that won't
be available at build time.
- narrows the search space for doing LSP actions -- like searching for
references. Since it would be way too much work to store in the compiler
every location where every symbol/function/etc is used, I have to do
ad-hoc searches. By having a dependency graph i can significantly reduce
that search space.
- opens the doors for common shared code with a legitimate pattern.
Right now jak 2 shares code from the jak 1 folder. This is basically a
hack -- but by having an explicit require syntax, it would be possible
to reference arbitrary file paths, such as a `common` folder.

Some stats:
- Jak 1 has about 2500 edges between files, including transitives
- With transitives reduced at the source code level, each file seems to
have a modest amount of explicit requirements.

Known issues:
- Tracking the location for where `defmacro`s and virtual state
definitions were defined (and therefore the file) is still problematic.
Because those forms are in a macro environment, the reader does not
track them. I'm wondering if a workaround could be to search the
reader's text_db by not just the `goos::Object` but by the text
position. But for the purposes of finishing this work, I just statically
analyzed and searched the code with throwaway python code.

goalc/compiler/compilation/CompilerControl.cpp

@@ -311,6 +311,26 @@ Val* Compiler::compile_in_package(const goos::Object& form, const goos::Object&
   return get_none();
 }
 
+/*!
+ * TODO - this form will denote which .DGO/.CGO the file should be bundled within
+ */
+Val* Compiler::compile_bundles(const goos::Object& form, const goos::Object& rest, Env* env) {
+  (void)form;
+  (void)rest;
+  (void)env;
+  return get_none();
+}
+
+/*!
+ * TODO - this form will denote a dependency for the given file
+ */
+Val* Compiler::compile_require(const goos::Object& form, const goos::Object& rest, Env* env) {
+  (void)form;
+  (void)rest;
+  (void)env;
+  return get_none();
+}
+
 /*!
  * Build dgo files. Takes a string argument pointing to the DGO description file, which is read
  * and parsed here.
@@ -583,7 +603,6 @@ Val* Compiler::compile_update_macro_metadata(const goos::Object& form,
   auto& name = args.unnamed.at(0).as_symbol().name_ptr;
 
   auto arg_spec = m_goos.parse_arg_spec(form, args.unnamed.at(2));
-  m_macro_specs[name] = arg_spec;
   m_symbol_info.add_macro(name, arg_spec, form, args.unnamed.at(1).as_string()->data);
   return get_none();
 }
@@ -757,7 +776,8 @@ Compiler::generate_per_file_symbol_info() {
       macro_doc.name = sym_info->m_name;
       macro_doc.description = sym_info->m_docstring;
       macro_doc.def_location = def_loc;
-      const auto& arg_spec = m_macro_specs[macro_doc.name];
+      // TODO - rewrite all of this to use the new symbol map, make sure macros work
+      /*const auto& arg_spec = m_macro_specs[macro_doc.name];
       for (const auto& arg : arg_spec.unnamed) {
         macro_doc.args.push_back(arg);
       }
@@ -770,7 +790,7 @@ Compiler::generate_per_file_symbol_info() {
       }
       if (!arg_spec.rest.empty()) {
         macro_doc.variadic_arg = arg_spec.rest;
-      }
+      }*/
       file_doc.macros.push_back(macro_doc);
     } else if (sym_info->m_kind == symbol_info::Kind::METHOD) {
       Docs::MethodDocumentation method_doc;
@@ -856,6 +876,25 @@ Val* Compiler::compile_gen_docs(const goos::Object& form, const goos::Object& re
   return get_none();
 }
 
+Val* Compiler::compile_export_requires(const goos::Object& form, const goos::Object& rest, Env*) {
+  auto args = get_va(form, rest);
+  va_check(form, args, {goos::ObjectType::STRING}, {});
+
+  const auto& export_path = fs::path(args.unnamed.at(0).as_string()->data);
+  lg::info("Saving require map to: {}", export_path.string());
+
+  json export_map;
+  for (const auto& file : m_global_env->get_files()) {
+    export_map[file->name()] = file->m_missing_required_files;
+  }
+
+  file_util::write_text_file(
+      export_path / fmt::format("{}-missing-requires.json", version_to_game_name(m_version)),
+      export_map.dump());
+
+  return get_none();
+}
+
 Val* Compiler::compile_gc_text(const goos::Object&, const goos::Object&, Env*) {
   m_goos.reader.db.clear_info();
   return get_none();