Skip to content

fix(backend,tests): unpin production workers from the xunit scheduler and stop starving it - #80

Merged
AresConnor merged 11 commits into
masterfrom
fix/protocoltests-scheduler-pinning
Aug 1, 2026
Merged

AresConnor merged 11 commits into
masterfrom
fix/protocoltests-scheduler-pinning

Conversation

@AresConnor

Copy link
Copy Markdown
Contributor

The defect

Reopened flake project (user decision, 2026-08-01) after the archived catalogue fired ~6 times in one day — a master run hanging silent for 11.5 minutes, and one PR failing 4 of 6 CI runs with a rotating cast of catalogued tests, including once on a docs-only delta, while 20-core and 2-core-pinned local runs stayed 100% green.

Root cause (confirmed by negative control): xunit v2 installs an assembly-wide MaxConcurrencySyncContext(ProcessorCount) — 2 slots on a 2-core CI runner — and several production async types with missing ConfigureAwait(false) are started inline from test threads, so their entire state machines (V2ConnectionOwner's pump, EventTriggerService's rule loop, InstanceProcess's publication chain, V2ClientConnectionOwner's lifecycle worker) get pinned to those 2 slots and compete with every concurrently-running test in the assembly. The pre-fix pump demonstrably resumes on an xunit context worker; post-fix it resumes on the ThreadPool. 8 of the 9 catalogued flaky tests reduce to this one defect; the 9th is a pure hop-count-versus-starved-pool case that improves when the test-side pollutants stop holding threads.

The fixes

  • fix(backend) ×2 — ConfigureAwait(false) completed in V2ConnectionOwner, EventTriggerService (+ its supervisor chain), the InstanceProcess publication chain, V2ClientConnectionOwner's lifecycle path, and (measurement-driven extension) OperationCoordinator.RunOperationAsync; the three bare await Task.Yield() context hops become context-free yields (ConfigureAwaitOptions.ForceYielding). In production there is no ambient context, so behavior is identical; the serial chains keep their order — only the resuming scheduler changes. The in-repo pattern copied is V2ClientConnectionCoordinator, which already did this correctly everywhere.
  • test(protocol) — the pollutant list from the inventory: a shared OffPool/DedicatedWorker helper; intentional blocking moved off pool threads (LongRunning, matching existing in-repo precedent); the hot Task.Yield spin throttled; controlled drain scheduling via the existing queueStateNotificationDrain seam; previously-unguarded awaits (await pump, fixture disposal) bounded with named timeouts so starvation fails loudly instead of hanging the assembly. StartHangingPlugins' blocking cancellation callbacks are deliberately unchanged — the blocking is those fixtures' contract (and the burst is 2 callbacks ≈ 200 ms, not the 600 ms+ first estimated).
  • ci(workflow) — --blame-hang --blame-hang-timeout 8m --blame-hang-dump-type mini + always-uploaded TestResults artifacts + a 30-minute job timeout: any future hang names its culprit test and leaves a dump instead of burning a 6-hour cancel.

Validation, honestly stated

  • Full battery green: ProtocolTests 1406/1406, ApiTests 116/116, ProtocolDocs --check, V1 gate, all builds 0 warnings.
  • Mechanism negative control: pre-fix the outbound pump resumes on the xunit MaxConcurrencySyncContext; post-fix on the ThreadPool.
  • Consistency probe: maxParallelThreads=1 pinned runs are 6/6 green vs ~3/12 with default parallelism — starvation-shaped, as diagnosed.
  • Caveat: local 2-core stress reproduces a different flake population than the CI catalogue (none of the 9 catalogued tests failed locally on either master or this branch across 29 runs), so local stress cannot confirm or refute the fix for the catalogued set. CI is the experiment: the plan is to re-run this PR's protocol-tests job repeatedly and compare against the 2026-08-01 baseline of 4 failures in 6 runs.
  • Measured but deliberately deferred: the 56 three-second guard literals in InstanceProcessEventPumpTests are insufficient under a faithful 2-core simulation. Whether to widen them is a separate decision to make after the de-pinning's CI results are in (no unmeasured guard-widening, and no premature widening that would mask the structural fix's effect).

Pre-push gates per the standing process: local codex (gpt-5.6-sol, xhigh) cross-review — one finding (unbounded fixture disposal), fixed in the final commit; local battery green before push.

…onization context

Areas: backend

Four async types start their state machine inline from a caller thread and
then suspend without ConfigureAwait(false), so whatever SynchronizationContext
the caller happened to have becomes the worker's permanent scheduler.

Complete ConfigureAwait(false) on V2ConnectionOwner (including the outbound
await foreach), EventTriggerService, and the InstanceProcess pump/terminate
awaits, and replace the four bare Task.Yield first hops with a context-free
forced yield so the close chain, the stop driver, the process-ready transition,
the publication chain, and the client lifecycle worker all resume on the pool.

These are serial await chains; ConfigureAwait(false) and ForceYielding change
which scheduler resumes them, never the order.
…ng waits

Areas: tests

Tests that must block were blocking on the same thread pool the code under
test needs to make progress, and sync test bodies held an xunit worker for the
whole wait. On a two-core runner that is enough to stall every other
collection until the pool injects replacements.

Add tests/MCServerLauncher.ProtocolTests/Helpers/OffPool.cs, naming the
existing in-repo LongRunning idiom, and route the deliberate blockers through
it. Convert the waits that never needed to block at all: the state-observer
gate and the seventeen racing HTTP registrations now suspend on a
TaskCompletionSource, the two catalog-mirror race tests became async, and the
Task.Yield spin waits are paced. Guard the previously unbounded pump awaits
and DisposeAsync so a wedged connection fails by name instead of hanging.

Blocking that bridges a synchronous production interface stays blocking; only
the thread it runs on changes.
Areas: workflow

A hung run currently produces no artifact naming the stuck test and occupies a
runner until the six-hour default job timeout.

Enable the VSTest blame collector with an 8m per-test hang timeout and a mini
dump, always upload the results directory, and cap the job at 30 minutes as a
backstop. The sequence file names the last test entered, which is the entry
point for triage.
…ontext

Areas: backend

RunOperationAsync is scheduled inline through OwnedTaskSupervisor from the
caller's thread, so its bare Task.Yield first hop captured whatever context
that caller had -- the same defect the transport and event workers had.

Found by measurement rather than inspection: under a two-core simulation
(DOTNET_PROCESSOR_COUNT=2 plus a two-core affinity mask) master failed
OperationCoordinatorTests.Start_ReturnsPersistedHandleBeforeBlockedExecutor
AndIgnoresRequestCancellation once in eleven runs.

This is scoped separately from the first commit so it can be dropped on its
own if the wider audit is not wanted in this change.
Areas: tests

Follow-up to the blocking-wait commit: four files received the helper import
below the third-party imports. No behavior change.
Areas: tests

StartAsync_DoesNotAwaitNonCooperativeRollbackAfterDeadline measured a
two-second wall-clock budget, which a loaded two-core runner crosses on
scheduling latency alone. CI failed it two ways: the WaitAsync(2s) guard threw
(attempt 3), and when StartAsync did complete in time the stopwatch still read
6.3s because the continuation resumed that late (attempt 5).

The budget was never the discriminator. StopRollbackAsync can only escape a
never-completing plugin stop through rollbackCleanupTimeout, and the shutdown
boundary that would otherwise release it does not exist until StopAsync runs.
So pass a rollback deadline far longer than the test: awaiting the rollback
becomes indistinguishable from hanging, and StartAsync returning at all is the
whole assertion. A short shutdown deadline keeps teardown from paying that
deadline, which is what made this test take 30s on the passing CI attempt.

Verified by mutation: with ScheduleFailedPluginCleanup awaiting the cleanup
inline, the new assertion fails by name after the guard; unmutated it passes in
1s. Full suite duration drops from 43s to 29s.
Areas: tests

StartAsync_BoundsNonCooperativeStartsAndRejectsLateCompletion carries the same
defect as its sibling: an eight-second wall-clock budget around a startup whose
only real failure mode is never returning. Local two-core stress failed it once
in eight runs at 18.6s elapsed while its own WaitAsync(10s) guard saw the task
finish in time, which is the continuation-latency signature, not a supervision
regression.

Apply the same treatment: a rollback deadline longer than the test so an
unsupervised startup cannot finish by accident, a short shutdown deadline so
teardown does not pay it, and a liveness guard in place of the elapsed-time
comparison. The state and start_timed_out log assertions already prove the
deadline fired, and they do not depend on timing.

Verified by mutation: dropping startDeadline from the supervision WhenAny fails
the test by name; unmutated the class runs 25 tests in 8s.
…o reject

Areas: tests

StartAsync_BoundsNonCooperativeStartsAndRejectsLateCompletion never proved its
second leg. DelayedRegisteredSuccessPlugin awaited Task.Delay(Infinite), so it
never completed at all, and the fixed 1500ms sleep that was meant to give the
late success a chance could not tell "late success rejected" from "never ran".
A regression that re-admitted a timed-out plugin would have passed. This
predates the liveness rewrite; that commit only made the gap visible.

Gate the delayed plugin on a release file instead, defaulting to the old
never-complete behavior when the variable is unset. The test opens the gate
once the host has already recorded the timeout, waits for the plugin's own
completion signal, and only then asserts the late success changed nothing: the
runtime stays Failed and contributes no catalog entry.

Catalog admission is single-use, so the assertion moved after the release
rather than being duplicated; contributions only ever add, so an empty catalog
there was empty before too.

Verified by mutation: re-admitting the runtime on a successful late start fails
the test with Expected: Failed, Actual: Started. The pre-release assertion
cannot produce that, since the plugin is still gated when it runs.
… context

Areas: backend

This reverts commit 22eb228. It traded a flake for a hang.

RunOperationAsync calls Complete, which takes _persistGate, and the tests hold
that gate inside PersistSnapshotsUnderLock while blocking on a release event.
Sending the runner to the pool means it blocks a pool thread there instead of
the caller's own thread. With DOTNET_PROCESSOR_COUNT=2 that can consume the
very pool the gate holder needs to be released, which closes the cycle.

Measured on OperationCoordinatorTests alone, pinned to two cores: 1 hang in 30
runs with the commit, 0 in 30 without. A live dotnet-stack of the hung host
shows RunOperationAsync on a ThreadPool worker parked in Monitor.Enter inside
Complete, with no other thread in the coordinator.

The flake it fixed (Start_ReturnsPersistedHandleBeforeBlockedExecutorAndIgnores
RequestCancellation, once in eleven master runs) is real but pre-existing, and
a hang is the worse failure: it burns the CI job and yields no signal. The
actual root cause is a lock held across a blocking wait, so the fix belongs in
those fixtures, not in which scheduler resumes the runner.
…ugin

Areas: backend, tests

The release-file sentinel was written by the plugin before its StartAsync
returned, so waiting on it only proved the plugin's task had completed. The
host continuation that a re-admission regression would run had not necessarily
executed yet, so the assertions could seal the single-use catalog before the
regression acted and still pass.

The host had no observable for a late success at all: the continuation
registered OnlyOnFaulted, so a timed-out plugin that later succeeded was
discarded silently. Widen it to every non-cancelled outcome and log the
discard, which is both the missing operator signal and the ordering anchor the
test needs. The test now waits for that log line and drops the plugin-side
completion sentinel, since the host log implies it.

Verified deterministic: with the host re-admitting the runtime on late success,
the test fails 10 of 10 runs with Expected: Failed, Actual: Started.
@AresConnor
AresConnor merged commit c5ed316 into master Aug 1, 2026
9 of 10 checks passed
@AresConnor

Copy link
Copy Markdown
Contributor Author

Final experiment record (merged at bb6ae9bf → master c5ed3160):

Sample Runs Failures Failure identity
Baseline, pre-fix (PR #79 head + master, 2026-08-01) 6 4 Rotating catalogued tests (QuickExit, HaltAfterCrash ×2, InstanceBase_FailedOsStart, MatchingRules), incl. one failure on a docs-only delta; plus two genuine hangs that day (11.5-min silent cancel; 8-min blame-hang)
PR #80, first head 7 3 Converged on one non-catalogued test (dead-zone stopwatch assert, root-caused and rewritten structurally) + 1 genuine hang (blame collector named its culprit — now #81)
PR #80, final head 7 1 A performance-gate ratio assert (PerfGate_PersistenceReadWriteRepresentative, 5% threshold on a shared runner) — different category, pre-existing, noted on #66

The 9 catalogued flaky tests: 0 failures in 14 CI runs across both heads. Suite wall time also dropped ~43s → ~21-30s locally.

Residuals, all tracked: the plan-kernel hang (#81, first-ever named lead courtesy of this PR's blame wiring), the OperationCoordinator fixture lock-across-wait hazard + its pre-existing flake (#82), perf-gate runner noise (comment on #66). One unattributed single sighting (a lock-file IOException in LoadFailureAfterAdapterConstruction... during one local 2-core run, never reproduced in 20 class-only runs per side) is recorded here for the archaeology.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant