Let a thread resume at the instruction after a syscall (#210)

A syscall can hand control back to the scheduler before the instruction
after it runs. SetSyscall lets the guest install its own handler for a
syscall number; dispatchSyscallOverride then suspends the calling
thread and queues that handler as a GuestInvocation. When the
invocation finishes, EeScheduler resumes the parent thread at the
address the generated code stored just before calling handleSyscall --
the instruction right after the syscall.

The analyzer never marked that address as an entry point. It queues
resume entries for JAL and JALR only, so no generated function could be
re-entered there, EeScheduler's hasFunction() check failed, and the
thread was made dormant instead of resumed. The thread simply stops;
because the scheduler then drains normally and run() returns, it looks
like a clean shutdown rather than a fault, which makes it awkward to
recognise.

This is reachable during early boot on a real title. Dragon Quest VIII
hits it in crt0: the Metrowerks startup code installs a handler for
syscall 0x83 and immediately issues it, and execution ends there,
roughly ten functions into the binary.

Note the offset is +4, not the +8 used for JAL and JALR -- syscall has
no delay slot.

(cherry picked from commit becb2be5bd)

ps2xRecomp/src/lib/control_flow_analyzer.cpp

@@ -135,6 +135,14 @@ namespace ps2recomp
 
         for (const auto &inst : instructions)
         {
+            // A guest-installed syscall handler runs as a separate invocation,
+            // so the scheduler resumes this thread at syscall+4 and needs an
+            // entry point there. +4, not +8: syscall has no delay slot.
+            if (inst.opcode == OPCODE_SPECIAL && inst.function == SPECIAL_SYSCALL)
+            {
+                queueResumeEntryTarget(inst.address + 4u);
+            }
+
             bool isStaticJump = (inst.opcode == OPCODE_J || inst.opcode == OPCODE_JAL);
             if (inst.isBranch && inst.opcode != OPCODE_J && inst.opcode != OPCODE_JAL)
             {

ps2xTest/src/code_generator_tests.cpp

@@ -144,6 +144,17 @@ static Instruction makeJr(uint32_t address, uint8_t rs)
     return inst;
 }
 
+static Instruction makeSyscall(uint32_t address)
+{
+    Instruction inst{};
+    inst.address = address;
+    inst.opcode = OPCODE_SPECIAL;
+    inst.function = SPECIAL_SYSCALL;
+    inst.hasDelaySlot = false;
+    inst.raw = (OPCODE_SPECIAL << 26) | SPECIAL_SYSCALL;
+    return inst;
+}
+
 static void printGeneratedCode(const std::string& name, const std::string& code)
 {
 #ifdef PRINT_GENERATED_CODE
@@ -556,6 +567,35 @@ void register_code_generator_tests()
                   "unresolved JALR should not pretend it has a resolved local jump table");
     });
 
+    tc.Run("syscall marks the following instruction as a resume entry", [](TestCase &t) {
+        // Shape of a real SDK syscall wrapper:
+        //   addiu $v1, $zero, <num> ; syscall ; jr $ra ; <delay slot>
+        Function func;
+        func.name = "syscall_wrapper";
+        func.start = 0x4000;
+        func.end = 0x4010;
+        func.isRecompiled = true;
+        func.isStub = false;
+
+        std::vector<Instruction> instructions{
+            makeAddiu(0x4000, 3, 0, 0x83),
+            makeSyscall(0x4004),
+            makeJr(0x4008, 31),
+            makeNop(0x400C),
+        };
+
+        CodeGenerator gen({}, {});
+        CodeGenerator::AnalysisResult analysis = gen.collectInternalBranchTargets(func, instructions);
+
+        // +4, not +8: syscall has no delay slot.
+        t.IsTrue(analysis.resumeEntryPoints.contains(0x4008u),
+                 "syscall should mark the next instruction as resumable");
+        t.IsTrue(analysis.entryPoints.contains(0x4008u),
+                 "syscall resume pc should emit a label in the owner");
+        t.IsFalse(analysis.resumeEntryPoints.contains(0x400Cu),
+                  "syscall must not claim a delay slot it does not have");
+    });
+
     tc.Run("resume entry targets emit a top-level pc switch in the owner wrapper", [](TestCase &t) {
         Function func;
         func.name = "resume_owner";