Let a thread resume at the instruction after a syscall

A syscall can hand control back to the scheduler before the instruction
after it runs. SetSyscall lets the guest install its own handler for a
syscall number; dispatchSyscallOverride then suspends the calling
thread and queues that handler as a GuestInvocation. When the
invocation finishes, EeScheduler resumes the parent thread at the
address the generated code stored just before calling handleSyscall --
the instruction right after the syscall.

The analyzer never marked that address as an entry point. It queues
resume entries for JAL and JALR only, so no generated function could be
re-entered there, EeScheduler's hasFunction() check failed, and the
thread was made dormant instead of resumed. The thread simply stops;
because the scheduler then drains normally and run() returns, it looks
like a clean shutdown rather than a fault, which makes it awkward to
recognise.

This is reachable during early boot on a real title. Dragon Quest VIII
hits it in crt0: the Metrowerks startup code installs a handler for
syscall 0x83 and immediately issues it, and execution ends there,
roughly ten functions into the binary.

Note the offset is +4, not the +8 used for JAL and JALR -- syscall has
no delay slot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ps2xRecomp/src/lib/control_flow_analyzer.cpp

@@ -135,6 +135,23 @@ namespace ps2recomp
 
         for (const auto &inst : instructions)
         {
+            // A syscall can hand control back to the scheduler before the
+            // instruction after it runs: SetSyscall lets the guest install its
+            // own handler, and dispatchSyscallOverride then suspends the
+            // calling thread and queues that handler as a GuestInvocation. When
+            // the invocation completes, the scheduler resumes the parent thread
+            // at the address the generated code stored before calling
+            // handleSyscall -- i.e. right here. Without an entry point there,
+            // EeScheduler's hasFunction() check fails and the thread is made
+            // dormant instead of resumed.
+            //
+            // Note the offset is +4, not the +8 used for JAL/JALR: syscall has
+            // no delay slot.
+            if (inst.opcode == OPCODE_SPECIAL && inst.function == SPECIAL_SYSCALL)
+            {
+                queueResumeEntryTarget(inst.address + 4u);
+            }
+
             bool isStaticJump = (inst.opcode == OPCODE_J || inst.opcode == OPCODE_JAL);
             if (inst.isBranch && inst.opcode != OPCODE_J && inst.opcode != OPCODE_JAL)
             {

ps2xTest/src/code_generator_tests.cpp

@@ -144,6 +144,17 @@ static Instruction makeJr(uint32_t address, uint8_t rs)
     return inst;
 }
 
+static Instruction makeSyscall(uint32_t address)
+{
+    Instruction inst{};
+    inst.address = address;
+    inst.opcode = OPCODE_SPECIAL;
+    inst.function = SPECIAL_SYSCALL;
+    inst.hasDelaySlot = false;
+    inst.raw = (OPCODE_SPECIAL << 26) | SPECIAL_SYSCALL;
+    return inst;
+}
+
 static void printGeneratedCode(const std::string& name, const std::string& code)
 {
 #ifdef PRINT_GENERATED_CODE
@@ -567,6 +578,40 @@ void register_code_generator_tests()
                   "unresolved JALR should not pretend it has a resolved local jump table");
     });
 
+    tc.Run("syscall marks the following instruction as a resume entry", [](TestCase &t) {
+        // Shape of a real SDK syscall wrapper:
+        //   addiu $v1, $zero, <num> ; syscall ; jr $ra ; <delay slot>
+        // SetSyscall lets a guest install its own handler for a syscall number,
+        // and the runtime then suspends the calling thread to run that handler
+        // as a separate invocation. The parent thread's saved pc is the address
+        // after the syscall, so the scheduler needs an entry point there to
+        // resume it -- otherwise the thread is made dormant and silently dies.
+        Function func;
+        func.name = "syscall_wrapper";
+        func.start = 0x4000;
+        func.end = 0x4010;
+        func.isRecompiled = true;
+        func.isStub = false;
+
+        std::vector<Instruction> instructions{
+            makeAddiu(0x4000, 3, 0, 0x83),
+            makeSyscall(0x4004),
+            makeJr(0x4008, 31),
+            makeNop(0x400C),
+        };
+
+        CodeGenerator gen({}, {});
+        CodeGenerator::AnalysisResult analysis = gen.collectInternalBranchTargets(func, instructions);
+
+        // +4, not +8: syscall has no delay slot.
+        t.IsTrue(analysis.resumeEntryPoints.contains(0x4008u),
+                 "syscall should mark the next instruction as resumable");
+        t.IsTrue(analysis.entryPoints.contains(0x4008u),
+                 "syscall resume pc should emit a label in the owner");
+        t.IsFalse(analysis.resumeEntryPoints.contains(0x400Cu),
+                  "syscall must not claim a delay slot it does not have");
+    });
+
     tc.Run("resume entry targets emit a top-level pc switch in the owner wrapper", [](TestCase &t) {
         Function func;
         func.name = "resume_owner";