Flaky TestWaitUntilSearchAttributesOptimizationWorkflowTemporal: expected 2, actual 3 #654

Observed

TestWaitUntilSearchAttributesOptimizationWorkflowTemporal failed once in Server CI and passed on rerun without code changes.

• PR: Add rotating Temporal credentials and TLS configuration #653 (the change does not touch the default plaintext, no-key Temporal connection path the test uses)
• Run: https://github.com/superdurable/dex/actions/runs/38658689758, job "Temporal integration (partition 0)", first attempt, 2026-10-15T00:18Z
• Failure:

wait_until_search_attributes_optimization_test.go:117:
    Error Trace: server/integ/wait_until_search_attributes_optimization_test.go:117
                 server/integ/wait_until_search_attributes_optimization_test.go:46
    Error:       Not equal:
                 expected: 2
                 actual  : 3
    Test:        TestWaitUntilSearchAttributesOptimizationWorkflowTemporal

Shortly before the failure, the worker log shows Cached state staled, new task has unexpected events ... CachedPreviousStartedEventID 25 for the same workflow, followed by Steps S1-3, S1-4 and S1-5. WaitForFlow took 10.3s.

Where to look

• server/integ/wait_until_search_attributes_optimization_test.go: lines 46 and 117, and what the asserted count measures.
• The Temporal interpreter path that decides whether the optimization applies (search-attribute loading for WaitFor). Hypothesis, not verified: the count depends on workflow task timing. A sticky-cache eviction or a late signal can add one more WaitFor/Execute round, so the assertion is timing-sensitive rather than a product bug.

Evidence

The full job log is available from the run above (gh run view 38658689758 --log --job 116174497364). Download it before log retention expires.

Acceptance criteria

• Root cause identified: either a product bug in the optimization, or a test assumption that depends on task timing.
• Fix the product code or make the assertion deterministic (for example, assert the invariant the optimization guarantees rather than an exact count that depends on timing). Do not skip the test.
• The test passes repeatedly. For example, run it 20 times in a loop via the Temporal integ Makefile target with -run '^TestWaitUntilSearchAttributesOptimizationWorkflowTemporal$' -count=20. Include the result in the PR.