feat(teamcity): report the exact status, assertion count, suite taxonomy and test count - #278
Merged
Conversation
The standard protocol collapses eight outcomes into three shapes, so `Flaky` was indistinguishable from `Passed` and `Risky` from a clean pass. Every `testFinished` now carries the exact `Status` as a lowercased `status` attribute, and `testSuiteFinished` the aggregated one for a suite, a case and a DataProvider batch. Standard parsers ignore the attribute. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Assert plugin records how many assertions a test performed under the `assertions` metric; the attribute passes it straight through, so a consumer sees an unasserted pass for what it is. Omitted when nothing counted them — no assertion plugin active — which `assertions='0'` would misstate. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Located tests are known before the first one runs, so each suite now opens with the count IntelliJ-based IDEs use to size their progress bar (the TeamCity server ignores the message). Counts accumulate there, so one per suite is the intended shape. A DataProvider test counts once but reports a node per data set, making the number a lower bound. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`RectorFixtureInterceptor` dispatched the data set events but built each fixture's `TestInfo` with `arguments` alone, so every one kept the batch's `TestIdentity`. The terminal groups by `pipelineId` and did not care, but TeamCity addresses tree nodes by `nodeId`, which comes from the identity: all fixtures claimed the batch node's id and hung beside it rather than under it, collapsing into a single node with no data sets in sight. A fixture now occupies the data set slot of the address, as it does for inline tests and data providers, which also gives the location hint the `:0:N` coordinates `--filter` and the IDE take back. `DataPointer` is honoured accordingly, so those coordinates select one fixture. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both interceptors give an attribute the provider slot of the address with a single data set inside it, so `:N:0` is the only valid coordinate pair for one case. `InlineInterceptor` compared the provider index alone, so `--filter=method:1:7` reached case 1 through a coordinate naming nothing, and `BenchInterceptor` read no `DataPointer` at all — one benchmark case could not be selected out of several. The empty-selection branch comes with it for bench: unreachable while the loop could not skip a candidate, and without it `Summary::combine([])` would report a filtered-out test as a clean pass. `BenchInterceptor` had no tests; the new ones mirror the inline suite. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`testSuiteStarted` and `testStarted` now carry `testSuite` and `testType`, the two things `--suite` and `--type` select on, so a consumer offers the same slicing without parsing a name or a path. A suite of the run states only `testSuite` — it holds cases of several types and has none of its own. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Such a case names no class, so `fqn()` is null and the node came out with no `locationHint` at all — the only one in the tree nobody could click. It now answers with `file://<path>`: the file it groups holds several functions rather than one to name, and IDEs resolve that scheme to a whole file. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
CancelTest and SkipTest carry the reason why the test stopped, but the testIgnored message showed a fixed "Test cancelled" and nothing at all for a skip. Take the message from the exception, as the JUnit writer already does; the generic wording stays as a fallback for a cancel thrown without one. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
The TeamCity output now says what the standard protocol cannot express. Every
testFinishedcarries the exactStatusas a lowercasedstatusattribute (passed,failed,skipped,error,risky,flaky,cancelled,aborted) and the number ofassertionsthe test performed;testSuiteFinishedcarries the aggregated status for a suite, a case and a DataProvider batch. Every opening node —testSuiteStartedandtestStarted— statestestSuiteandtestType, the two things--suiteand--typeselect on. Each suite opens withtestCount, the number of located tests, which IntelliJ-based IDEs accumulate to size their progress bar. Finally, a case of free functions no longer comes out without alocationHint: it points at its file underfile://.See commit history for details.
Why?
Three of the eight statuses were invisible to a consumer: the protocol distinguishes only ignored, failed and everything else, so
Flakywas byte-identical toPassedandRiskyto a clean pass. The same applies to the rest — an IDE plugin reading the stream had no way to tell a cancelled test from a skipped one, to offer suite/type slicing without parsing names and paths, to show progress before the run ended, or to click through to a file of free functions.Every attribute is additive and unknown to standard parsers, which ignore them and keep reading the run exactly as before.
Checklist