offline-test: Partition by DGO and colorize/condense output (#2045)

This solves two main problems:
- the looming threat of running out of memory since every thread would
consume duplicate (and probably not needed) resources
- though I will point out, jak 2's offline tests seem to hardly use any
memory even with 400+ files, duplicated across many threads. Where as
jak 1 does indeed use tons more memory. So I think there is something
going on besides just the source files
- condense the output so it's much easier to see what is happening / how
close the test is to completing.
- one annoying thing about the multiple thread change was errors were
typically buried far in the middle of the output, this fixes that
- refactors the offline test code in general to be a lot more modular

The pretty printing is not enabled by default, run with `-p` or
`--pretty-print` if you want to use it


https://user-images.githubusercontent.com/13153231/205513212-a65c20d4-ce36-44f6-826a-cd475505dbf9.mp4
This commit is contained in:
Tyler Wilding
2022-12-22 13:41:33 -05:00
committed by GitHub
parent 74d1074eef
commit 9c631e11fe
39 changed files with 8619 additions and 7375 deletions
@@ -125,6 +125,8 @@ void ObjectFileDB::process_object_file_data(
void ObjectFileDB::analyze_functions_ir2(
const fs::path& output_dir,
const Config& config,
const std::optional<std::function<void(std::string)>> prefile_callback,
const std::optional<std::function<void()>> postfile_callback,
const std::unordered_set<std::string>& skip_functions,
const std::unordered_map<std::string, std::unordered_set<std::string>>& skip_states) {
int total_file_count = 0;
@@ -133,8 +135,14 @@ void ObjectFileDB::analyze_functions_ir2(
}
int file_idx = 1;
for_each_obj([&](ObjectFileData& data) {
if (prefile_callback) {
prefile_callback.value()(data.to_unique_name());
}
lg::info("[{:3d}/{}]------ {}", file_idx++, total_file_count, data.to_unique_name());
process_object_file_data(data, output_dir, config, skip_functions, skip_states);
if (postfile_callback) {
postfile_callback.value()();
}
});
lg::info("{}", stats.let.print());