Surface exit-code verdict and per-assembly clarity in GitHubActionsReport by Evangelink · Pull Request #9974 · microsoft/testfx

Evangelink · 2026-07-15T14:28:41Z

What

Follows up on feedback in #9003 for the Microsoft.Testing.Extensions.GitHubActionsReport extension:

Exit-code awareness (@Ramstad's ask): the extension previously only reflected per-test pass/fail. A run that failed for a non-test reason — a --minimum-expected-tests violation, a run that discovered zero tests, an aborted session, a --maximum-failed-tests stop, adapter/host failures — showed a misleading green ✅.
Multi-assembly (dotnet test) clarity: make each assembly's contribution to the job summary carry its own verdict.

Changes

New GitHubActionsExitCode helper maps every documented MTP exit code to a friendly, localized reason and classifies test-result vs. non-test-result outcomes.
Step summary (--report-gh-step-summary): flips the header to ❌ when the run failed even with 0 failed tests, and renders a > [!WARNING] callout naming the exit code + reason.
Annotations (--report-gh-annotations): the annotation reporter is now an ITestSessionLifetimeHandler and emits a single run-level ::error for non-test-result failures, so e.g. a --minimum-expected-tests violation surfaces in the Annotations tab even when no individual test failed. Plain per-test failures are unchanged (no redundant error).
Multi-assembly: each assembly already appends its own labelled GITHUB_STEP_SUMMARY section; those now each carry their own verdict/exit-code callout, so you can see which assembly failed and why. (A true cross-process grand total would require heavy cross-process coordination and was intentionally left out — noted in PACKAGE.md.)
ITestApplicationProcessExitCode wired into both reporters, new localized resources (+ regenerated .xlf), updated InternalAPI.Unshipped.txt and PACKAGE.md.

Tests

Updated GitHubActionsSummaryReporterTests (new signature + callout coverage) and added GitHubActionsExitCodeTests.
Extension + Microsoft.Testing.Extensions.UnitTests build clean; full suite 676 tests, 672 passed, 4 skipped, 0 failed.

Relates to #9003.

Addresses feedback on issue #9003 for the GitHub Actions report extension: - Map MTP exit codes (zero tests, --minimum-expected-tests, aborted, --maximum-failed-tests, adapter/host failures) to friendly reasons. - Step summary now flips to failure icon and renders an exit-code callout when the run failed without any failed test. - Annotation reporter emits a run-level ::error for non-test-result failures so they surface in the Annotations tab. - Each assembly's step-summary section carries its own verdict for multi-assembly (dotnet test) clarity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Pull request overview

Adds exit-code-aware verdicts to GitHub Actions summaries and annotations, including clearer per-assembly reporting.

Changes:

Maps MTP exit codes to localized failure reasons.
Adds run-level summary callouts and annotations.
Updates documentation, API tracking, and tests.

Show a summary per file

File	Description
`GitHubActionsSummaryReporter.cs`	Adds exit-code verdicts and warnings.
`GitHubActionsAnnotationReporter.cs`	Emits run-level failure annotations.
`GitHubActionsExitCode.cs`	Maps exit codes to names and reasons.
`GitHubActionsExtensions.cs`	Injects exit-code service and registers lifecycle handling.
`InternalAPI/InternalAPI.Unshipped.txt`	Tracks changed internal APIs.
`PACKAGE.md`	Documents exit-code and multi-assembly behavior.
`Resources/GitHubActionsResources.resx`	Adds exit-code messages.
`Resources/xlf/GitHubActionsResources.cs.xlf`	Adds Czech localization entries.
`Resources/xlf/GitHubActionsResources.de.xlf`	Adds German localization entries.
`Resources/xlf/GitHubActionsResources.es.xlf`	Adds Spanish localization entries.
`Resources/xlf/GitHubActionsResources.fr.xlf`	Adds French localization entries.
`Resources/xlf/GitHubActionsResources.it.xlf`	Adds Italian localization entries.
`Resources/xlf/GitHubActionsResources.ja.xlf`	Adds Japanese localization entries.
`Resources/xlf/GitHubActionsResources.ko.xlf`	Adds Korean localization entries.
`Resources/xlf/GitHubActionsResources.pl.xlf`	Adds Polish localization entries.
`Resources/xlf/GitHubActionsResources.pt-BR.xlf`	Adds Portuguese localization entries.
`Resources/xlf/GitHubActionsResources.ru.xlf`	Adds Russian localization entries.
`Resources/xlf/GitHubActionsResources.tr.xlf`	Adds Turkish localization entries.
`Resources/xlf/GitHubActionsResources.zh-Hans.xlf`	Adds Simplified Chinese entries.
`Resources/xlf/GitHubActionsResources.zh-Hant.xlf`	Adds Traditional Chinese entries.
`GitHubActionsExitCodeTests.cs`	Tests exit-code classification and formatting.
`GitHubActionsSummaryReporterTests.cs`	Tests summary verdicts and callouts.

Review details

Files reviewed: 22/22 changed files
Comments generated: 3
Review effort level: Medium

…verage - Qualify PACKAGE.md and code comments: the exit-code callout/annotation covers session-derived outcomes (ZeroTests, MinimumExpectedTests, TestAdapterTestSessionFailure, MaxFailedTests). A hard abort/cancellation short-circuits end-of-session reporting and early/late-host codes are out of reach, so those are no longer over-claimed. - Add GitHubActionsReportTests acceptance tests driving the extension through a real MTP session (GITHUB_ACTIONS=true + temp GITHUB_STEP_SUMMARY) for pass / fail / zero-tests / minimum-expected-tests, asserting emission and suppression of the callout and run-level annotation end to end. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 4
Review effort level: Medium

…nown-reason test Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 4
Review effort level: Medium

…aims, fix markdownlint - Add UTF-8 BOM to GitHubActionsExitCodeTests.cs and GitHubActionsReportTests.cs per .editorconfig utf-8-bom requirement for *.cs. - Remove the aborted-session / host-failure examples from GitHubActionsExitCode class doc and the summary callout comment, matching the documented reachable scope (session-derived outcomes only). - Fix PACKAGE.md markdownlint (MD032 list spacing, MD060 table pipe spacing). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 1
Review effort level: Medium

Addresses review r3588595379: the shared GITHUB_STEP_SUMMARY file was opened with FileMode.Append + FileShare.ReadWrite. Append only seeks to EOF at open (no atomic OS append), so concurrent test-host processes in a multi-assembly dotnet test run could interleave or clobber each other's section. AppendStepSummaryWithRetryAsync now opens with FileShare.Read (denies other writers) and retries on the sharing-violation IOException with bounded backoff, serializing the per-assembly append so every assembly's section is preserved. Added unit tests for first-attempt write, retry-then-succeed, and give-up-and- rethrow (best-effort warning). Acceptance suite (12 tests) still green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 1
Review effort level: Medium

Addresses review r3593645686: the retry previously wrapped open+write+flush, so a mid-write failure (e.g. disk full) after a partial append would be retried and re-append the full section, corrupting the summary. Contention only occurs while acquiring the exclusive append handle, so retries are now scoped to the open; any failure once writing has started propagates to the best-effort warning path. Added a unit test proving a post-acquire write failure throws after a single attempt (no retry). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 0 new
Review effort level: Medium

Addresses code-scanning nit r3594649673: expand the test helper's Position setter to explicitly consume the assigned value (discard) so the 'value ignored' analysis no longer fires. Test-only helper; behavior unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot

Review details

Files reviewed: 23/23 changed files
Comments generated: 0 new
Review effort level: Medium

github-actions · 2026-07-16T11:45:06Z

🧪 Test quality grade — PR #9974

Grade	Test	Notes
C (70–79)	new `GitHubActionsExitCodeTests. GetReason_ ForUnknownCode_ ReturnsGenericNonSuccessReason`	Only asserts the reason is non-empty — consider asserting a meaningful substring of the fallback text to verify content, not just presence.
A (90–100)	new `GitHubActionsExitCodeTests. GetExitCodeAnnotation_ ForZeroTests_ EmitsErrorWorkflowCommand`	Three targeted string assertions cover prefix format, exit code, and enum name — clear and complete.
A (90–100)	new `GitHubActionsExitCodeTests. GetName_ ReturnsEnumNameForKnownCodeAndUnknownOtherwise`	No issues found.
A (90–100)	new `GitHubActionsExitCodeTests. GetReason_ ForKnownCode_ MentionsRelevantOption`	No issues found.
A (90–100)	new `GitHubActionsExitCodeTests. IndicatesFailure_ OnlySuccessIsNotFailure`	Three assertions prove a coherent invariant together; compact and well-named.
A (90–100)	new `GitHubActionsExitCodeTests. IsTestResultOutcome_ ForNonTestResultCodes_ ReturnsFalse`	Parameterised negative path via `[DataRow]`; assertion directly checks the contract.
A (90–100)	new `GitHubActionsExitCodeTests. IsTestResultOutcome_ ForTestResultCodes_ ReturnsTrue`	Parameterised positive path via `[DataRow]`; concise and focused.
A (90–100)	new `GitHubActionsSummaryReporterTests. AppendStepSummaryWithRetryAsync_ DoesNotRetry_ WhenWriteFailsAfterHandleAcquired`	Exception assertion combined with `Times.Once` mock verification proves the no-retry contract precisely.
A (90–100)	new `GitHubActionsSummaryReporterTests. AppendStepSummaryWithRetryAsync_ Rethrows_ WhenAllAttemptsFail`	Exact exception type plus retry-count verification; named constant for attempt limit is a nice touch.
A (90–100)	new `GitHubActionsSummaryReporterTests. AppendStepSummaryWithRetryAsync_ RetriesOnSharingViolation_ ThenSucceeds`	Content equality plus `Times.Exactly(2)` captures both the retry and the eventual success.
A (90–100)	new `GitHubActionsSummaryReporterTests. AppendStepSummaryWithRetryAsync_ WritesContent_ OnFirstAttempt`	UTF-8 content equality plus `Times.Once` mock verification; BOM comment adds clarity.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ AllPassing_ UsesSuccessIconAndTotals`	Positive and negative Contains checks cover icon, totals row, and absence of failure section — well-rounded.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ EmitsSlowestTestsSortedByDuration`	Index comparison verifies sort order, not just presence — a strong structural assertion.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ MinimumExpectedTestsExitCode_ EmitsCalloutEvenWhenTestsPassed`	Comment clarifies the non-obvious scenario; assertions cover icon, code name, and option name.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ NoTests_ StillEmitsHeaderAndZeroTotals`	No issues found.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ WithFailures_ UsesFailureIconAndListsFailures`	Checks failure icon, failure-count header, failure item, and absence of warning callout — comprehensive.
A (90–100)	new `GitHubActionsSummaryReporterTests. BuildMarkdown_ ZeroTestsExitCode_ UsesFailureIconAndEmitsCallout`	No issues found.
A (90–100)	new `GitHubActionsReportTests. WhenMinimumExpectedTestsViolated_ EmitsExitCodeAnnotationAndCalloutEvenThoughTestPassed`	End-to-end coverage of a subtle edge case; multi-layer assertions on exit code, annotation, and summary are appropriate for acceptance tests.
A (90–100)	new `GitHubActionsReportTests. WhenRunPasses_ NoExitCodeCalloutAndSummaryUsesSuccessIcon`	Positive and negative assertions on both stdout and the step-summary file — thorough end-to-end coverage.
A (90–100)	new `GitHubActionsReportTests. WhenTestFails_ EmitsPerTestAnnotationButNoExitCodeCallout`	Distinguishes per-test annotation from run-level annotation cleanly; negative assertion on warning callout is key.
A (90–100)	new `GitHubActionsReportTests. WhenZeroTestsRan_ EmitsExitCodeAnnotationAndCallout`	Covers all three output surfaces (exit code, stdout annotation, summary callout) with matching assertions.

_{20 of 21 tests earn A — the suite is exceptionally well-written. The single C-grade test (GetReason_ForUnknownCode_ReturnsGenericNonSuccessReason) would improve by asserting a meaningful substring of the fallback message rather than only its non-emptiness. This advisory comment was generated automatically. Grades are heuristic and informational — they do not block merging. Re-run with /grade-tests.}

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🤖 Automated content by GitHub Copilot. Generated by the Grade Tests on PR (on open / sync) workflow. · 55.3 AIC · ⌖ 7.38 AIC · ⊞ 8.9K · [◷]( · ◷)

Copilot AI review requested due to automatic review settings July 15, 2026 14:28

Copilot started reviewing on behalf of Evangelink July 15, 2026 14:29 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Copilot AI review requested due to automatic review settings July 15, 2026 14:55

Copilot started reviewing on behalf of Evangelink July 15, 2026 14:56 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Expert review round 1: rename param, tighten PACKAGE.md note, add Unk…

928c35c

…nown-reason test Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b1e3944b-d2cf-4ab7-ac53-c8ace4ddc260

Copilot AI review requested due to automatic review settings July 15, 2026 15:04

Copilot started reviewing on behalf of Evangelink July 15, 2026 15:05 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

Copilot AI review requested due to automatic review settings July 15, 2026 15:20

Copilot started reviewing on behalf of Evangelink July 15, 2026 15:21 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/PACKAGE.md

This comment has been minimized.

Sign in to view

Copilot AI review requested due to automatic review settings July 16, 2026 07:58

Copilot started reviewing on behalf of Evangelink July 16, 2026 07:59 View session

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Comment thread src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsSummaryReporter.cs

This comment has been minimized.

Sign in to view

Evangelink enabled auto-merge (squash) July 16, 2026 10:16

Evangelink added the state/needs-review Awaiting review from the team. label Jul 16, 2026

Copilot AI review requested due to automatic review settings July 16, 2026 10:20

Copilot started reviewing on behalf of Evangelink July 16, 2026 10:22 View session

0101 approved these changes Jul 16, 2026

View reviewed changes

Copilot AI reviewed Jul 16, 2026

View reviewed changes

github-code-quality Bot found potential problems Jul 16, 2026

View reviewed changes

Comment thread test/UnitTests/Microsoft.Testing.Extensions.UnitTests/GitHubActionsSummaryReporterTests.cs Fixed

This comment has been minimized.

Sign in to view

Copilot AI review requested due to automatic review settings July 16, 2026 11:35

Copilot started reviewing on behalf of Evangelink July 16, 2026 11:36 View session

0101 approved these changes Jul 16, 2026

View reviewed changes

Copilot AI reviewed Jul 16, 2026

View reviewed changes

Evangelink disabled auto-merge July 16, 2026 13:46

Evangelink merged commit ff73a7e into main Jul 16, 2026
45 of 47 checks passed

Evangelink deleted the dev/amauryleve/github-report-multiasm-exitcode branch July 16, 2026 13:47

Uh oh!

Conversation

Evangelink commented Jul 15, 2026

What

Changes

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

github-actions Bot commented Jul 16, 2026

🧪 Test quality grade — PR #9974

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants