Skip to content

Flaky TestAnyCommandCombinationFlow: fixed 1s sleep before checking COMPLETED #662

Closed as not planned

Description

@longquanzheng

Observed

TestAnyCommandCombinationFlowCadence failed twice under heavy local load: once in a full Cadence run and once when rerun in isolation. It passes in CI on clean runners (#661).

  • Assertion: server/integ/any_command_combination_test.go:161 expects FLOW_STATUS_COMPLETED, but the Flow is still FLOW_STATUS_RUNNING.
  • Cause, partly verified: the test calls PublishToChannel, then time.Sleep(1 * time.Second), then describes the Flow once. Under load, Step S2 is still processing after 1 second.
  • This was observed on branch codex/rt-delete (multi-tenancy removal), which does not touch this code path. The local machine's load average was 12–20.

Where to look

server/integ/any_command_combination_test.go, around the PublishToChannel → sleep → describe sequence near line 161.

Acceptance criteria

  • Replace the fixed sleep with convergence, for example require.Eventually on the Flow status, or WaitForFlow with a bounded timeout. Keep the existing assertions about the final state and history. Do not relax them.
  • The repo rule says no time.Sleep in tests except inside the behavior under test.
  • The test passes 10 times in a row on both Temporal and Cadence while the machine is loaded, for example with a parallel CPU burner or GOMAXPROCS=2.

Activity

  1. longquanzheng commented on Oct 15, 2026

    @longquanzheng
    ContributorAuthor

    A related timing-margin flake was seen in the same local runs. TestWaitForStateCompletionTemporal, handler-timeout subscenario: WaitForStepCompletion uses a fixed 12-second client deadline after an intentional 10-second timer, so the next Step started at about 12 seconds on a loaded host (load average 12–20) and hit the deadline. This reproduced in isolation with GOMAXPROCS=2. It passes in CI on clean runners. Acceptance: give the deadline a margin proportional to the timer, or assert convergence instead of a fixed total. Do not relax the semantic assertions. The test must pass 10 times in a row under load.

  2. added
    type:testTest coverage, flaky tests, or test infrastructure
    P2Normal priority
    size:SAbout a day or less
    codex-readySelf-contained; can be handed to a Codex worker as is
    on Oct 26, 2026
  3. longquanzheng commented on Nov 9, 2026

    @longquanzheng
    ContributorAuthor

    The Go Server integ suite was removed in #1059. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal prioritycodex-readySelf-contained; can be handed to a Codex worker as issize:SAbout a day or lesstype:testTest coverage, flaky tests, or test infrastructure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions