fix(workflow): clear resume token when reset_agent_context fires without live execution - #2765
Conversation
…out live execution A step on_enter with reset_agent_context was silently skipped when no in-memory agent execution existed for the session — the early return omitted all persisted state clearing (ACP session ID, resume token, context window). This left a stale resume token on the executors_running row that the lazy resume path (applyRunningRecordToResumeRequest) reads directly, so any subsequent lazy launch would reconnect to the pre-reset ACP conversation and the agent would start its first turn with the old, stale context — effectively ignoring the reset. Fix: extract a shared clearPersistedResetState helper that clears the same durable state whether or not a live execution is present, and call it from both the no-in-memory-execution early path and the live-reset path. The call to clearResumeToken was also missing from the live-execution path; it previously cleared only the ACP session ID and context window metadata but never removed the resume token, which could also cause a stale reconnect on re-entry within the same provider session lifespan. Regression test: TestProcessOnEnterResetAgentContext_ClearsLazyResume TokenWithoutLiveExecution seeds a session with a stopped executor row carrying a resume token, triggers processOnEnter with a step that declares resetAgentContext, and asserts the resume token is cleared. Evidence finding — Review↔QA loop (task 6e0fc028): Session 8c401f9f completed at 04:26:31 UTC and was re-entered at ~04:54. The re-entry likely crossed a workflow step with reset_agent_context on_enter; with no live execution, the reset was skipped and the stale resume token survived. The lazy launch then reconnected to the old ACP conversation, giving the agent stale context and causing the Review↔QA loop (already_signaled at 04:56:06). This IS the same root cause. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: QUIET Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan includes up to 4 reviews per rolling hour; 3 remain after this review. 📝 WalkthroughSummary by CodeRabbit
WalkthroughThe workflow reset path now clears persisted ACP session, resume-token, and context-window state when no live agent execution exists. A test verifies that reset-on-enter removes a stopped executor’s lazy-resume token without restarting the executor. ChangesAgent context reset
Estimated code review effort: 2 (Simple) | ~15 minutes Merge Risk: ⚪ Minimal · up to This localized workflow fix clears stale persisted resume state during agent-context resets, including lazy resumes without a live execution. The change is merge-ready after normal checks, with no actionable merge-blocking risk remaining. Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
| Filename | Overview |
|---|---|
| apps/backend/internal/orchestrator/event_handlers_workflow.go | Consolidates durable reset-state cleanup and invokes it on both successful live resets and confirmed no-live-execution resets. |
| apps/backend/internal/orchestrator/event_handlers_workflow_lazy_resume_test.go | Adds focused regression coverage proving that a no-live-execution workflow reset clears the persisted lazy-resume token. |
Sequence Diagram
sequenceDiagram
participant W as Workflow on_enter
participant O as Orchestrator
participant M as Agent manager
participant DB as Persisted session state
participant R as Lazy resume
W->>O: reset_agent_context
O->>M: Lookup live execution
alt Live execution exists
O->>M: Reset provider context
M-->>O: Reset succeeded
else No live execution
M-->>O: No execution
end
O->>DB: Clear ACP session ID
O->>DB: Clear resume token
O->>DB: Clear context window
R->>DB: Read recovery state
DB-->>R: No stale resume token
R->>R: Start a fresh conversation
Reviews (1): Last reviewed commit: "fix(workflow): clear resume token when r..." | Re-trigger Greptile
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b95b75df68
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Add TestResetAgentContext_ClearsThenAllowsFreshStoreResumeToken to verify that the live-execution reset path: 1. Clears the stale resume token via clearPersistedResetState 2. Allows a subsequent storeResumeToken (simulating the async ACP session.created event from the new session) to write the fresh token This proves the concern about clearPersistedResetState racing with concurrent storeResumeToken is structurally impossible: - ResetAgentContext is synchronous and does NOT publish OnACPSessionCreated (only fresh-session and workspace-rebind paths do) - clearPersistedResetState runs on the same goroutine after ResetAgentContext returns — the clear deterministically precedes the async streaming event - The ACP session.created streaming event arrives later on a separate WebSocket reader goroutine, so storeResumeToken always writes AFTER the clear Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t path The live-execution path of resetAgentContext had a race window: after ResetAgentContext returned (creating a new ACP session on the same execution), clearResumeToken ran unconditionally and could erase the fresh resume token written by the async ACP session.created event. Since storeResumeToken's CAS keyed on agent_execution_id cannot distinguish ACP session generations within the same execution (ResetSession does not rotate the execution), the async event's CAS would succeed — then clearResumeToken would delete the fresh token. Fix: split clearResumeToken out of clearPersistedResetState and call it BEFORE ResetAgentContext in the live-execution path. The new ACP session doesn't exist yet at that point, so there is nothing for the async event to write and nothing for clearResumeToken to race with. Affected paths: - Live-execution reset: clearResumeToken runs before ResetAgentContext; clearPersistedResetState (now without clearResumeToken) runs after. - No-in-memory-execution reset (lazy resume): both clearResumeToken and clearPersistedResetState run on the same path, but there is no ResetAgentContext call and therefore no async event to race with. A pre-existing concern remains (tested by InterleavingC): a stale ACP session.created event from the OLD session can overwrite the fresh token via the same execution-ID CAS. The old code had the same window for acp_session_id metadata. Full fix requires a generation counter on executors_running (outside this change's scope). Tests: - InterleavingA (store before clear): PASS (previously failed) - InterleavingB (clear before store): PASS (unchanged) - InterleavingC (stale old event after store): SKIP (pre-existing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks for the contribution. I pushed
The focused orchestrator tests pass. |
The reconcile added in 9346651 had one test, covering only the partial-success shape (live ACP session moved, so the token is rewritten). Three neighbouring shapes decided the same branch's behavior with nothing asserting them: - Failure where the live ACP session did NOT move. The reconcile rewrites the same value, so kdlbs#2765's guarantee that a failed reset keeps a usable recovery token still holds, but nothing pinned it, and dropping the non-empty guard would have blanked a valid token silently. - Failure arriving from an execution the executors_running row has already rotated past. The reconcile must lose to storeResumeToken's CAS instead of stamping a defunct execution's session over the live one's token. - The ACP session moving again between resetAgentContext reading the live id and storeResumeToken re-reading it for the stale-generation guard. The reconcile must lose to the newer generation. Each test was mutation-checked: blanking the token, bypassing the CAS, and disabling the generation guard each make exactly the corresponding test fail. Tests only; no production change.
Problem
When a workflow step's
on_enterdeclaresreset_agent_contextbut no in-memory agent execution exists for the session (e.g. the session completed/stopped and is being lazily resumed), the reset was silently skipped. The early return omitted all persisted state clearing — the ACP session ID, resume token on theexecutors_runningrow, and context window metadata all survived.The lazy resume path (
applyRunningRecordToResumeRequest) reads the resume token directly from the persisted row, so a stale token would let the next lazy launch reconnect to the pre-reset ACP conversation. The agent would start its first turn with old, stale context — effectively ignoring the reset.Fix
Extracted
clearResumeTokenout ofclearPersistedResetStateand call it BEFOREResetAgentContextin the live-execution path. The new ACP session does not exist yet, so the asyncsession.createdevent (which callsstoreResumeToken) cannot race with the clear — eliminating the store-before-clear interleaving.For the no-in-memory-execution path (lazy resume),
clearResumeTokenis called separately before the unchangedclearPersistedResetState. There is noResetAgentContextcall on this path, so no async event exists to race with.clearPersistedResetStatenow handles only ACP session metadata and context-window cleanup, without touching the resume token. This prevents it from erasing a fresh token written by the concurrentstoreResumeTokenon the live-execution path.A pre-existing concern remains: a stale ACP
session.createdevent from the OLD session (same execution ID, old ACP session ID) can overwrite the fresh token via the execution-ID CAS, sinceResetSessiondoes not rotate the execution identity. The old code had the same window foracp_session_idmetadata. Full fix requires a generation counter onexecutors_running(outside this scope).Regression tests
TestProcessOnEnterResetAgentContext_ClearsLazyResumeTokenWithoutLiveExecution— seeds a session with a stopped executor row carrying a resume token, triggersprocessOnEnterwith a step that declaresresetAgentContext, and asserts the resume token is cleared. (Original regression test, unchanged.)TestResetAgentContext_InterleavingA_StoreBeforeClear— simulates the dangerous interleaving wherestoreResumeTokenfires beforeclearResumeToken. With the fix (clear before reset), the fresh token survives. PASS.TestResetAgentContext_InterleavingB_ClearBeforeStore— the favorable interleaving: clear runs before the async event writes the fresh token. PASS.TestResetAgentContext_InterleavingC_StaleOldEventOverwritesFresh— demonstrates the pre-existing concern where a stale old-session event overwrites the fresh token via the same execution-ID CAS. SKIP (pre-existing, requires generation counter).Evidence finding — Review↔QA loop
Session
8c401f9f(task6e0fc028) completed at 04:26:31 UTC and was re-entered at ~04:54. The observedalready_signaledat 04:56:06 is mechanistically consistent with a stale agent context — the agent would see old conversation history and loop rather than proceeding. However, direct production evidence does not prove that the re-entry crossed areset_agent_contexton_enterwith no live execution: there is no saved record of the specific branch, execution state, or workflow step that triggered the re-entry. The mechanism fixed here is a plausible downstream cause for the loop, but the practical observation cannot be definitively attributed to this bug without that execution trace. The terminal-session revival path (which shares the same no-in-memory-execution gap for a different entry point) is independently fixed by PR #2766.Validation
go test ./internal/orchestrator/...)