Repository navigation
fix(backend,tests): unpin production workers from the xunit scheduler and stop starving it - #80
Merged
Merged
Conversation
…onization context Areas: backend Four async types start their state machine inline from a caller thread and then suspend without ConfigureAwait(false), so whatever SynchronizationContext the caller happened to have becomes the worker's permanent scheduler. Complete ConfigureAwait(false) on V2ConnectionOwner (including the outbound await foreach), EventTriggerService, and the InstanceProcess pump/terminate awaits, and replace the four bare Task.Yield first hops with a context-free forced yield so the close chain, the stop driver, the process-ready transition, the publication chain, and the client lifecycle worker all resume on the pool. These are serial await chains; ConfigureAwait(false) and ForceYielding change which scheduler resumes them, never the order.
…ng waits Areas: tests Tests that must block were blocking on the same thread pool the code under test needs to make progress, and sync test bodies held an xunit worker for the whole wait. On a two-core runner that is enough to stall every other collection until the pool injects replacements. Add tests/MCServerLauncher.ProtocolTests/Helpers/OffPool.cs, naming the existing in-repo LongRunning idiom, and route the deliberate blockers through it. Convert the waits that never needed to block at all: the state-observer gate and the seventeen racing HTTP registrations now suspend on a TaskCompletionSource, the two catalog-mirror race tests became async, and the Task.Yield spin waits are paced. Guard the previously unbounded pump awaits and DisposeAsync so a wedged connection fails by name instead of hanging. Blocking that bridges a synchronous production interface stays blocking; only the thread it runs on changes.
Areas: workflow A hung run currently produces no artifact naming the stuck test and occupies a runner until the six-hour default job timeout. Enable the VSTest blame collector with an 8m per-test hang timeout and a mini dump, always upload the results directory, and cap the job at 30 minutes as a backstop. The sequence file names the last test entered, which is the entry point for triage.
…ontext Areas: backend RunOperationAsync is scheduled inline through OwnedTaskSupervisor from the caller's thread, so its bare Task.Yield first hop captured whatever context that caller had -- the same defect the transport and event workers had. Found by measurement rather than inspection: under a two-core simulation (DOTNET_PROCESSOR_COUNT=2 plus a two-core affinity mask) master failed OperationCoordinatorTests.Start_ReturnsPersistedHandleBeforeBlockedExecutor AndIgnoresRequestCancellation once in eleven runs. This is scoped separately from the first commit so it can be dropped on its own if the wider audit is not wanted in this change.
Areas: tests Follow-up to the blocking-wait commit: four files received the helper import below the third-party imports. No behavior change.
Areas: tests StartAsync_DoesNotAwaitNonCooperativeRollbackAfterDeadline measured a two-second wall-clock budget, which a loaded two-core runner crosses on scheduling latency alone. CI failed it two ways: the WaitAsync(2s) guard threw (attempt 3), and when StartAsync did complete in time the stopwatch still read 6.3s because the continuation resumed that late (attempt 5). The budget was never the discriminator. StopRollbackAsync can only escape a never-completing plugin stop through rollbackCleanupTimeout, and the shutdown boundary that would otherwise release it does not exist until StopAsync runs. So pass a rollback deadline far longer than the test: awaiting the rollback becomes indistinguishable from hanging, and StartAsync returning at all is the whole assertion. A short shutdown deadline keeps teardown from paying that deadline, which is what made this test take 30s on the passing CI attempt. Verified by mutation: with ScheduleFailedPluginCleanup awaiting the cleanup inline, the new assertion fails by name after the guard; unmutated it passes in 1s. Full suite duration drops from 43s to 29s.
Areas: tests StartAsync_BoundsNonCooperativeStartsAndRejectsLateCompletion carries the same defect as its sibling: an eight-second wall-clock budget around a startup whose only real failure mode is never returning. Local two-core stress failed it once in eight runs at 18.6s elapsed while its own WaitAsync(10s) guard saw the task finish in time, which is the continuation-latency signature, not a supervision regression. Apply the same treatment: a rollback deadline longer than the test so an unsupervised startup cannot finish by accident, a short shutdown deadline so teardown does not pay it, and a liveness guard in place of the elapsed-time comparison. The state and start_timed_out log assertions already prove the deadline fired, and they do not depend on timing. Verified by mutation: dropping startDeadline from the supervision WhenAny fails the test by name; unmutated the class runs 25 tests in 8s.
…o reject Areas: tests StartAsync_BoundsNonCooperativeStartsAndRejectsLateCompletion never proved its second leg. DelayedRegisteredSuccessPlugin awaited Task.Delay(Infinite), so it never completed at all, and the fixed 1500ms sleep that was meant to give the late success a chance could not tell "late success rejected" from "never ran". A regression that re-admitted a timed-out plugin would have passed. This predates the liveness rewrite; that commit only made the gap visible. Gate the delayed plugin on a release file instead, defaulting to the old never-complete behavior when the variable is unset. The test opens the gate once the host has already recorded the timeout, waits for the plugin's own completion signal, and only then asserts the late success changed nothing: the runtime stays Failed and contributes no catalog entry. Catalog admission is single-use, so the assertion moved after the release rather than being duplicated; contributions only ever add, so an empty catalog there was empty before too. Verified by mutation: re-admitting the runtime on a successful late start fails the test with Expected: Failed, Actual: Started. The pre-release assertion cannot produce that, since the plugin is still gated when it runs.
… context Areas: backend This reverts commit 22eb228. It traded a flake for a hang. RunOperationAsync calls Complete, which takes _persistGate, and the tests hold that gate inside PersistSnapshotsUnderLock while blocking on a release event. Sending the runner to the pool means it blocks a pool thread there instead of the caller's own thread. With DOTNET_PROCESSOR_COUNT=2 that can consume the very pool the gate holder needs to be released, which closes the cycle. Measured on OperationCoordinatorTests alone, pinned to two cores: 1 hang in 30 runs with the commit, 0 in 30 without. A live dotnet-stack of the hung host shows RunOperationAsync on a ThreadPool worker parked in Monitor.Enter inside Complete, with no other thread in the coordinator. The flake it fixed (Start_ReturnsPersistedHandleBeforeBlockedExecutorAndIgnores RequestCancellation, once in eleven master runs) is real but pre-existing, and a hang is the worse failure: it burns the CI job and yields no signal. The actual root cause is a lock held across a blocking wait, so the fix belongs in those fixtures, not in which scheduler resumes the runner.
…ugin Areas: backend, tests The release-file sentinel was written by the plugin before its StartAsync returned, so waiting on it only proved the plugin's task had completed. The host continuation that a re-admission regression would run had not necessarily executed yet, so the assertions could seal the single-use catalog before the regression acted and still pass. The host had no observable for a late success at all: the continuation registered OnlyOnFaulted, so a timed-out plugin that later succeeded was discarded silently. Widen it to every non-cancelled outcome and log the discard, which is both the missing operator signal and the ordering anchor the test needs. The test now waits for that log line and drops the plugin-side completion sentinel, since the host log implies it. Verified deterministic: with the host re-admitting the runtime on late success, the test fails 10 of 10 runs with Expected: Failed, Actual: Started.
This was referenced Aug 1, 2026
Contributor
Author
|
Final experiment record (merged at
The 9 catalogued flaky tests: 0 failures in 14 CI runs across both heads. Suite wall time also dropped ~43s → ~21-30s locally. Residuals, all tracked: the plan-kernel hang (#81, first-ever named lead courtesy of this PR's blame wiring), the OperationCoordinator fixture lock-across-wait hazard + its pre-existing flake (#82), perf-gate runner noise (comment on #66). One unattributed single sighting (a lock-file IOException in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
Reopened flake project (user decision, 2026-08-01) after the archived catalogue fired ~6 times in one day — a master run hanging silent for 11.5 minutes, and one PR failing 4 of 6 CI runs with a rotating cast of catalogued tests, including once on a docs-only delta, while 20-core and 2-core-pinned local runs stayed 100% green.
Root cause (confirmed by negative control): xunit v2 installs an assembly-wide
MaxConcurrencySyncContext(ProcessorCount)— 2 slots on a 2-core CI runner — and several production async types with missingConfigureAwait(false)are started inline from test threads, so their entire state machines (V2ConnectionOwner's pump,EventTriggerService's rule loop,InstanceProcess's publication chain,V2ClientConnectionOwner's lifecycle worker) get pinned to those 2 slots and compete with every concurrently-running test in the assembly. The pre-fix pump demonstrably resumes on an xunit context worker; post-fix it resumes on the ThreadPool. 8 of the 9 catalogued flaky tests reduce to this one defect; the 9th is a pure hop-count-versus-starved-pool case that improves when the test-side pollutants stop holding threads.The fixes
fix(backend)×2 —ConfigureAwait(false)completed inV2ConnectionOwner,EventTriggerService(+ its supervisor chain), theInstanceProcesspublication chain,V2ClientConnectionOwner's lifecycle path, and (measurement-driven extension)OperationCoordinator.RunOperationAsync; the three bareawait Task.Yield()context hops become context-free yields (ConfigureAwaitOptions.ForceYielding). In production there is no ambient context, so behavior is identical; the serial chains keep their order — only the resuming scheduler changes. The in-repo pattern copied isV2ClientConnectionCoordinator, which already did this correctly everywhere.test(protocol)— the pollutant list from the inventory: a sharedOffPool/DedicatedWorkerhelper; intentional blocking moved off pool threads (LongRunning, matching existing in-repo precedent); the hotTask.Yieldspin throttled; controlled drain scheduling via the existingqueueStateNotificationDrainseam; previously-unguarded awaits (await pump, fixture disposal) bounded with named timeouts so starvation fails loudly instead of hanging the assembly.StartHangingPlugins' blocking cancellation callbacks are deliberately unchanged — the blocking is those fixtures' contract (and the burst is 2 callbacks ≈ 200 ms, not the 600 ms+ first estimated).ci(workflow)—--blame-hang --blame-hang-timeout 8m --blame-hang-dump-type mini+ always-uploadedTestResultsartifacts + a 30-minute job timeout: any future hang names its culprit test and leaves a dump instead of burning a 6-hour cancel.Validation, honestly stated
--check, V1 gate, all builds 0 warnings.MaxConcurrencySyncContext; post-fix on the ThreadPool.maxParallelThreads=1pinned runs are 6/6 green vs ~3/12 with default parallelism — starvation-shaped, as diagnosed.InstanceProcessEventPumpTestsare insufficient under a faithful 2-core simulation. Whether to widen them is a separate decision to make after the de-pinning's CI results are in (no unmeasured guard-widening, and no premature widening that would mask the structural fix's effect).Pre-push gates per the standing process: local codex (gpt-5.6-sol, xhigh) cross-review — one finding (unbounded fixture disposal), fixed in the final commit; local battery green before push.