Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions scripts/cts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ surface stays in sync; only the runner stack differs.
| -------------------------- | ---------------------------------------------------------- |
| `projects.json` | Registry of `*.UnitTests.VSTest` projects + demo files |
| `cts.config.json` | CTS configuration (Modules / SourceCodeFiles / Filter) |
| `cts.config.mtp.json` | Experimental MTP-mode config (see [MTP mode](#mtp-mode-experimental)) |
| `_Common.ps1` | Tiny PS helpers (paths, build, registry lookup) |
| `Collect-Local.ps1` | `cts collect vstest --coverage` → baseline in `.cts/` |
| `Run-Local.ps1` | `cts apply vstest --local-development` → impacted-only run |
Expand Down Expand Up @@ -104,9 +105,44 @@ silently skip the other. It is wired into `azure-pipelines/cts-apply.yml` as
a **non-blocking** PR-time step, and can also be run locally:
`pwsh ./scripts/cts/Check-SlnxParity.ps1`.

## Notes
## MTP mode (experimental)

> The VSTest path above remains the supported, non-blocking CI mode. The MTP
> path below is opt-in and **not yet wired into CI** — see the caveat.

CTS originally could not drive this repo's tests in their native
**Microsoft.Testing.Platform (MTP)** form (the default xUnit.v3 runner) because
the CTS↔host JsonRpc session could hang indefinitely — the reason the VSTest
wrappers exist. That hang has two independent causes, both now addressed in the
CTS tool itself (driver resilience: wait for real host exit + bounded output
drain, and an **opt-in inactivity timeout** that kills a host which has gone
completely silent and lets CTS retry it on a fresh process):

1. A Windows stdout/stderr pipe-inheritance leak past host exit.
2. A wedged JSON-RPC handshake with no timeout.

`cts.config.mtp.json` drives the **regular `*.UnitTests` MTP DLLs directly**
(no `.VSTest` wrapper): `Filter.Include` targets
`**/artifacts/bin/*.UnitTests/Debug/net10.0/*.UnitTests.dll`, and
`RunConfiguration.TestHostResponseTimeoutSeconds` (120s here) opts into the new
inactivity timeout. This is **not** a cap on total batch time — it is the
maximum time the host may go without producing *any* activity (a test-node
update or a client log) before CTS treats it as wedged. Every update from the
host resets the countdown, so a batch that keeps streaming progress runs as long
as it needs; only a host that falls silent for the whole window is killed and
retried. The CTS default is `0` (disabled), so the timeout only engages because
this config sets it. Point `cts collect testingplatform` /
`cts apply testingplatform` at this config instead of `cts.config.json`.

**Caveat (why this is still experimental):** with xUnit.v3 3.2.2, a specific
batch of tests can deterministically deadlock the test host in MTP *server*
mode (the auto-generated entry point blocks sync-over-async on its run task).
The CTS timeout+retry converts that from an infinite hang into a bounded
outcome, but the deadlocking batch can still surface as a spurious failure.
Fully green MTP-mode CTS is therefore gated on an upstream xUnit.v3 fix; until
then, VSTest mode remains the CI path.


* Baseline + logs live at `<repo>/.cts/` (gitignored).
* The `.VSTest.csproj` wrappers output to
`artifacts/bin/<MSBuildProjectName>/Debug/net10.0/` — distinct from the
default MTP variant.
Expand Down
54 changes: 54 additions & 0 deletions scripts/cts/cts.config.mtp.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
{
"SourceCodeFiles": {
"Include": [
"src/**/*.cs"
],
"Exclude": [
"**/obj/**",
"**/bin/**",
"**/TestAssets/**"
]
},
"Files": {
"Exclude": [
"**/*.md",
"**/*.yml",
"**/*.yaml",
"**/*.png",
"documentation/**"
]
},
"Modules": {
"Include": [],
"Exclude": [
"**/Microsoft.Testing.*.dll",
"**/Microsoft.TestPlatform*.dll",
"**/Microsoft.VisualStudio.TestPlatform*.dll",
"**/Microsoft.VisualStudio.CodeCoverage*.dll",
"**/Microsoft.DotNet.*.dll",
"**/xunit.*.dll",
"**/Xunit.*.dll",
"**/testhost*",
"**/Microsoft.Bcl.*.dll",
"**/Microsoft.ApplicationInsights.dll",
"**/Newtonsoft.Json.dll",
"**/Shouldly.dll",
"**/AwesomeAssertions.dll",
"**/FakeItEasy.dll",
"**/Verify.*.dll",
"**/System*.dll",
"**/runtimes/**"
]
},
"Filter": {
"Include": [
"**/artifacts/bin/*.UnitTests/Debug/net10.0/*.UnitTests.dll"
],
"Exclude": [
"**/*.resources.dll"
]
},
"RunConfiguration": {
"TestHostResponseTimeoutSeconds": 120
}
}
Loading