Skip to content

Let a long-running Step Execute run beside other work without stalling the Flow at its operation limit #646

Closed as not planned

Description

@longquanzheng

Problem

A long-running Step Execute holds back the rest of its Flow once the Flow reaches its operation limit (continue_as_new_threshold).

When the limit is met, the interpreter stops starting new Steps and waits for every running thread to drain before it continues in a new run. Pending WaitFor waits (Timer, Channel, SubFlow) carry into the new run, but an in-flight Execute does not. It must finish first.

A Flow that runs a long poll in one Step beside ongoing work therefore stalls that work until the poll ends. An example is an interactive loop of many short Steps. Raising the operation limit only postpones the stall.

The only current workaround is application-level: move the long poll into its own SubFlow, so the parent holds only a WaitFor. Applications then also need a stop protocol for the child, for example racing the SubFlow in AnyOf against a stop Channel and calling Client.StopFlow(SubFlowID). This workaround was drafted as SDK skill guidance (superdurable/dex-skills#93, closed without merging). We do not want every application with a long-running Step to restructure itself around a SubFlow.

Goal

A Flow can run a long-running Execute beside other Steps without the rest of the Flow stalling when the operation limit is reached. No SubFlow restructuring is required in the application.

Code to read

  • server/service/interpreter/continueAsNewer.go: DrainThreads / allThreadsDrained wait until GetThreadCount() == 0 and no in-flight update operations.
  • server/service/interpreter/cont/continueAsNewCounter.go: IsThresholdMet.
  • server/service/interpreter/signalReceiver.go, timers/greedyTimerScheduler.go, attributeSynchronizer.go: waits that release on IsThresholdMet().
  • ContinueAsNewer.activeStepMovements / StepExecutionToResumeMap: what is carried into the next run today.

Possible directions (hypotheses, not verified)

  1. Carry an in-flight Execute across the run boundary instead of draining it. The new run would resume waiting for the same Step execution's result, as it already does for WaitFor. This needs a way for the new run to receive the result of an Execute that the previous run started.
  2. Run Steps that are marked or detected as long-running in an internally managed child execution, so the parent's drain only waits on a lightweight handle.
  3. Let the parent keep starting new Steps while it waits for long Executes to drain, with a separate hard limit that bounds history growth.

Each direction must keep the existing Step guarantees (retry semantics, Step state, StepExecutionID stability, ordering of Step results).

Acceptance criteria

  • An integration test in server/integ/ reproduces the stall on current main. One Step's Execute runs much longer than the time a sibling loop of short Steps needs to hit a low continue_as_new_threshold. The test shows the sibling loop stops progressing.
  • With the fix, the same test shows the sibling loop keeps progressing across run boundaries while the long Execute is still running, and the long Execute's result is delivered exactly as before.
  • The test passes on both Temporal and Cadence integration backends (make -C server temporalIntegTests / cadenceIntegTests).
  • No application code change is needed for existing Flows. Persisted data and wire formats stay compatible.
  • If the fix changes application-visible behavior, the SDK skill guidance under the Dex skills is updated per the repository rule.

Activity

  1. longquanzheng commented on Nov 9, 2026

    @longquanzheng
    ContributorAuthor

    server-zd has no Continue-As-New or operation-limit drain, so a long-running Execute does not stall the Flow. Closing as obsolete (Go Server removed in #1059).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priorityneeds-decisionNeeds an owner decision before implementationsize:LA week or more, or needs designtype:featureNew capability or API

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions