Add experimental MTP-mode CTS config#14355
Draft
jankratochvilcz wants to merge 2 commits into
Draft
Conversation
Adds scripts/cts/cts.config.mtp.json so CTS can drive this repo's tests in their native Microsoft.Testing.Platform (xUnit.v3) form, driving the regular *.UnitTests DLLs directly instead of the *.VSTest wrappers. It sets RunConfiguration.TestHostResponseTimeoutSeconds to tune the CTS per-batch wedged-host timeout that the CTS driver-resilience fix adds. This is opt-in and not wired into CI: with xUnit.v3 3.2.2 a specific batch can still deterministically deadlock the test host in MTP server mode, which the CTS timeout+retry bounds but does not fully cure. VSTest mode remains the CI path and is left untouched (cts.config.json, *.ps1, azure-pipelines/cts-*.yml). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The CTS-side timeout changed from a fixed per-batch cap to an opt-in, inactivity-based window (reset on every host update). Update the MTP-mode README so cts.config.mtp.json's TestHostResponseTimeoutSeconds=120 is described as an inactivity window rather than a total-batch cap, and note the CTS default is 0 (disabled) so this config is what opts in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a41f0f3c-f165-414e-a8c5-efb8deecd9c8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental, opt-in CTS (Clever Test Selection) configuration that drives this repo's tests in their native Microsoft.Testing.Platform (MTP) form (the default xUnit.v3 runner), instead of the
*.VSTestwrappers that VSTest-mode CTS uses today.scripts/cts/cts.config.mtp.json— targets the regular*.UnitTestsMTP DLLs (**/artifacts/bin/*.UnitTests/Debug/net10.0/*.UnitTests.dll) and setsRunConfiguration.TestHostResponseTimeoutSecondsto tune the CTS per-batch wedged-host timeout.scripts/cts/README.md— new "MTP mode (experimental)" section documenting usage and the caveat.The VSTest path is intentionally left untouched (
scripts/cts/cts.config.json,scripts/cts/*.ps1,azure-pipelines/cts-*.yml) and remains the CI mode.Background
CTS could not previously drive MSBuild's tests in MTP form because the CTS↔host JSON-RPC session could hang indefinitely (the reason the
*.VSTestwrappers exist). That hang was root-caused to two independent issues in the CTS tool's Testing Platform driver, both fixed on the CTS side (separate PR in thevs-code-coveragerepo):WaitForExitAsyncthen waits for a pipe EOF that never arrives.The CTS fix waits for real process exit + bounded output drain, and adds a per-batch timeout that kills a wedged host and retries the affected tests on a fresh process. That timeout is what
TestHostResponseTimeoutSecondsin this config tunes.Why experimental / draft
With xUnit.v3 3.2.2, a specific batch can still deterministically deadlock the test host in MTP server mode (the auto-generated entry point blocks sync-over-async on its run task — captured stack:
Monitor.Wait ← TaskAwaiter.GetResult() ← XunitAutoGeneratedEntryPoint.Main). The CTS timeout+retry converts that from an infinite hang into a bounded outcome, but the deadlocking batch can still surface as a spurious failure. Fully green MTP-mode CTS is therefore gated on an upstream xUnit.v3 fix. Until then, VSTest mode remains the CI path and this config is opt-in only (not wired into any pipeline).Testing
Validated end-to-end with the fixed CTS driver against real
StringTools.UnitTestsMTP tests: the driver now always terminates and writes a baseline (previously: infinite hang). The config-driven timeout was confirmed to be honored from the JSON file.