Codex remote sessions can remain stuck in `turnInFlight` after a native subagent completes

[teekalpha]

What version of HAPI is running?

0.25.1

What platform is your computer?

Linux 6.6.87.2-microsoft-standard-WSL2 x86_64

What issue are you seeing?

Codex remote sessions sometimes get stuck after a native subagent (spawn_agent) finishes. The UI continues showing the session as "thinking", but there is no further output and the turn does not complete. Sending another user message makes the session responsive again.

I saw this more than five times during one session of roughly 80 minutes. Each occurrence stayed silent until I sent another message, usually after waiting 60–233 seconds.

This appears to happen only on turns that use native subagents. I have not seen the same behavior on normal turns without subagent activity.

What steps can reproduce the bug?

1. Start a Codex session in remote mode:

   hapi --hapi-starting-mode remote

2. Use a model that supports native subagents, such as gpt-5.6-terra.

3. Give it a task complex enough that it calls Codex's native spawn_agent tool.

4. Wait for the child agent to finish.

It is not completely deterministic, but it reproduced frequently in my sessions.

Here is one occurrence from the debug log in ~/.hapi/logs/:

10:00:50.866  [Codex] Routing event from non-active thread into agent trace;
               type=task_complete, eventThreadId=<child>, activeThread=<main>
10:00:50.866  [DiffProcessor] Resetting diff state
               <100 seconds with no log output or events>
10:02:30.915  [Codex] User message received: "处理完了之后，为什么没有给我后续的下文？"
10:02:30.945  thinking started

The session became responsive immediately after the new user message. Another session had the same pattern and remained silent for 233 seconds before I intervened.

Sessions where I reproduced this:

• 4ad92d1c-e2b8-4549-8af4-89b03a557c20 — stalled around 10:00:50; I sent another message around 10:02:30
• 57fee58b-7b36-4f68-9064-676acec43942 — five stalls between 09:21 and 10:13

What is the expected behavior?

The parent turn should either continue after the subagent finishes or transition to a recoverable error state. It should not remain in the "thinking" state indefinitely and require another user message to wake it up.

Investigation so far

startTurn() in codexAppServerClient.ts returns after the turn is created. The rest of the turn lifecycle is driven by app-server notifications.

For the active thread, a terminal notification such as task_complete clears turnInFlight, stops the thinking state, and calls wakeLoop(). Events belonging to a child thread are correctly routed into the child-agent trace and do not complete the parent turn.

In the affected logs, the last notification I can see is the child's task_complete. I do not see a later terminal notification for the parent, so the main loop eventually waits here while turnInFlight remains true:

if (!pending && turnInFlight && session.queue.size() === 0) {
    await waitForTurnOrRecovery(this.abortController.signal);
    continue;
}

waitForTurnOrRecovery() has no timeout or other bounded recovery path. This explains why the HAPI session can remain stuck once the expected parent terminal event is not observed.

I cannot tell from the current logs whether the parent notification is never sent by Codex app-server, is lost during conversion/dispatch, or is blocked elsewhere in notification handling. I did not see messages such as Ignoring terminal event or Dropping unscoped around these stalls.

Possible direction

It may be useful to add stall detection for an active turn, but a fixed timeout followed by an unconditional interrupt could also stop a legitimately long-running turn.

A safer recovery flow might be:

1. Track the last parent-thread event time while a turn is active.
2. After an extended period without parent activity, log the active thread/turn IDs and child-turn state.
3. If possible, query or otherwise verify the app-server's current turn status.
4. If the turn is no longer active, or the app-server is disconnected/unresponsive, move the launcher through the normal terminal/recovery cleanup path.
5. Fall back to interrupting the active turn only when its state cannot be recovered safely.

Even without automatic recovery, additional diagnostics around the active parent turn would help determine where the terminal event is being lost.

Additional information

• Permission mode was safe-yolo, so this did not appear to be waiting for a permission approval.
• The issue has only occurred for me on turns involving native subagents.
• Avoiding native subagents may be a temporary workaround.

[github-actions]

Thanks for the detailed report. Code review confirms the recovery gap you identified in HAPI 0.25.1: after turn/start returns, the remote launcher depends on notifications to finish the turn. A child-thread terminal event is correctly routed to the child trace and returns without changing the parent turn. Only an accepted parent terminal event clears turnInFlight, stops the thinking state, and wakes the loop.

waitForTurnOrRecovery() has no timer or status poll; it resolves only when explicitly woken or aborted. Therefore, if the parent terminal notification is absent or discarded after turn/start succeeds, the current code can remain in the thinking state indefinitely. This confirms the HAPI-side failure mode, but the available evidence does not yet show whether Codex app-server omitted the parent event or HAPI lost/rejected it.

One additional verified diagnostic point: wrapped terminal events without a turn_id are discarded by the event converter. I cannot determine from the provided excerpt whether such an event occurred here.

Relevant code:

• cli/src/codex/codexAppServerClient.ts:288 - startTurn() waits only for the turn/start response.
• cli/src/codex/codexRemoteLauncher.ts:2400 - non-active-thread events are routed to the child-agent trace and return before parent lifecycle handling.
• cli/src/codex/codexRemoteLauncher.ts:2626 - terminal events must match the active turn context.
• cli/src/codex/codexRemoteLauncher.ts:2740 - an accepted terminal event clears turnInFlight, stops thinking, and calls wakeLoop().
• cli/src/codex/codexRemoteLauncher.ts:1878 - the active-turn wait has no bounded recovery path.
• cli/src/codex/codexRemoteLauncher.ts:3551 - the main loop waits there while the turn remains active and the message queue is empty.
• cli/src/codex/utils/appServerEventConverter.ts:540 - wrapped terminal events without turn_id are ignored.

Need more info:

• What exact Codex version does codex --version report?
• Could you attach the complete DEBUG log slice from about 30 seconds before the child task_complete through the next user message, including any [CodexAppServer] and [AppServerEventConverter] lines? Please redact message contents or secrets as needed.
• Please include the [CodexAppServer] Resolved codex command line from the same run, which identifies the selected app-server executable/source.

---

HAPI Bot