-
Notifications
You must be signed in to change notification settings - Fork 307
Add artifact merge engines and Kind metadata flow (RFC 018 phase 1) #9805
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
e1a52d8
Add artifact merge engines and Kind metadata flow (RFC 018 phase 1)
Evangelink 03364a8
Merge remote-tracking branch 'origin/main' into dev/amauryleve/miniat…
Evangelink 3d61cd4
Address review: TRX merge fidelity + out-of-process Kind
Evangelink 3c482cd
Merge remote-tracking branch 'origin/main' into dev/amauryleve/miniat…
Evangelink 46448d7
Address review: harden TRX attachment relocation
Evangelink 4ea04cc
Update .xlf files to be in sync with .resx
Evangelink a05a088
Merge remote-tracking branch 'origin/main' into dev/amauryleve/miniat…
Evangelink 3a712c1
Address review: destination reparse hardening, Kind assertion, revert…
Evangelink a5dab5b
Address review: ancestor-symlink hardening + producer-to-wire Kind tests
Evangelink a468d2e
Address review: CTRF tool/env attribution + TRX cancellation & path-f…
Evangelink 1c197f4
Address review: TRX nested-overlap relocation + CTRF full tool identity
Evangelink 1400070
Merge branch 'main' into dev/amauryleve/miniature-broccoli
Evangelink 0bf2a16
Merge branch 'main' into dev/amauryleve/miniature-broccoli
Copilot 011ce7c
Merge branch 'main' into dev/amauryleve/miniature-broccoli
Evangelink 579f8b2
Address review: CTRF test-level timing fallback + TRX nested staging …
Evangelink 97dcbb0
Address review: never delete input trees during TRX attachment reloca…
Evangelink 92aae3a
Fix IDE0008 CI break + address review: relocation confinement, stagin…
Evangelink 109ab51
Make report merges deterministic and non-destructive (RFC 018 review)
Evangelink df5c7c6
Fix TRX per-test ResultFile relocation, naming, and IDE0008 (RFC 018 …
Evangelink c14a6f9
Fix IDE0007 in merge test (use var for apparent type)
Evangelink 02c0bef
Harden report-merge filesystem handling (RFC 018 review)
Evangelink bac64fb
Copy only referenced attachments in TRX merge (RFC 018 review)
Evangelink 2b7a860
Drop unrelocated references and detect symlinked-parent output aliase…
Evangelink 31f43de
Fix merge semantics: outcome, definition dedup, environment, referenc…
Evangelink d582d47
Fix UriAttachment schema validity, dangling dest symlinks, netstandar…
Evangelink ecdc9f4
Harden path containment, hardlink copy, top-level bail, and env unani…
Evangelink e68ed42
Preserve rooted attachments (RFC 018), normalize separators, validate…
Evangelink 6cdc1b1
Require CTRF reportFormat, carry run-level ResultFiles, preserve root…
Evangelink d1b8007
Merge run-level Output, reject empty input early, filesystem-aware al…
Evangelink f378b5d
Probe output-path case sensitivity at its own directory, not temp (re…
Evangelink f91fcb2
Extract shared merge file I/O + alias checks into MergeOutputFileHelp…
Evangelink bd927b5
Fix macOS symlink-alias canonicalization and address merge review com…
Evangelink File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
220 changes: 220 additions & 0 deletions
220
src/Platform/Microsoft.Testing.Extensions.CtrfReport/CtrfReportMerger.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| // Copyright (c) Microsoft Corporation. All rights reserved. | ||
| // Licensed under the MIT license. See LICENSE file in the project root for full license information. | ||
|
|
||
| using System.Text.Json; | ||
| using System.Text.Json.Nodes; | ||
|
|
||
| using Microsoft.Testing.Platform; | ||
|
|
||
| namespace Microsoft.Testing.Extensions.CtrfReport; | ||
|
|
||
| /// <summary> | ||
| /// Merges several already-produced CTRF JSON reports into a single CTRF document. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// This is a pure, invocation-agnostic JSON-level merge (no I/O, no clock) that mirrors the | ||
| /// TRX and JUnit mergers, demonstrating that the same post-processing shape fits a JSON format: | ||
| /// <list type="bullet"> | ||
| /// <item><description><c>results.tests[]</c> arrays are concatenated as-is.</description></item> | ||
| /// <item><description><c>results.summary</c> counters are re-derived by counting the merged <c>tests[]</c> (so <c>summary.tests</c> always matches the array length); <c>start</c>/<c>stop</c> use the earliest/latest across inputs, <c>duration</c> is the resulting span.</description></item> | ||
| /// <item><description><c>reportFormat</c>, <c>specVersion</c>, <c>tool</c> and <c>environment</c> are taken from the first report; <c>reportId</c> is freshly generated.</description></item> | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
| /// </list> | ||
| /// </remarks> | ||
| internal static class CtrfReportMerger | ||
| { | ||
| internal static string Merge(IReadOnlyList<string> inputReports) | ||
| { | ||
| if (inputReports is null) | ||
| { | ||
| throw new ArgumentNullException(nameof(inputReports)); | ||
| } | ||
|
|
||
| if (inputReports.Count == 0) | ||
| { | ||
| throw new ArgumentException("At least one CTRF report is required to merge.", nameof(inputReports)); | ||
| } | ||
|
|
||
| JsonObject? first = null; | ||
| var mergedTests = new JsonArray(); | ||
|
|
||
| long? earliestStart = null; | ||
| long? latestStop = null; | ||
|
|
||
| foreach (string reportJson in inputReports) | ||
| { | ||
| if (JsonNode.Parse(reportJson) is not JsonObject root) | ||
| { | ||
| continue; | ||
| } | ||
|
Evangelink marked this conversation as resolved.
|
||
|
|
||
| first ??= root; | ||
|
|
||
| JsonNode? results = root["results"]; | ||
| if (results?["tests"] is JsonArray testArray) | ||
| { | ||
| foreach (JsonNode? test in testArray) | ||
| { | ||
| mergedTests.Add(test?.DeepClone()); | ||
| } | ||
| } | ||
|
|
||
| JsonNode? summary = results?["summary"]; | ||
| if (summary is not null) | ||
| { | ||
| if (TryReadLong(summary, "start", out long start) && (earliestStart is null || start < earliestStart)) | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
| { | ||
| earliestStart = start; | ||
| } | ||
|
|
||
| if (TryReadLong(summary, "stop", out long stop) && (latestStop is null || stop > latestStop)) | ||
| { | ||
| latestStop = stop; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (first is null) | ||
| { | ||
| throw new ArgumentException("None of the provided inputs were valid CTRF reports.", nameof(inputReports)); | ||
| } | ||
|
|
||
| long startMs = earliestStart ?? 0; | ||
| long stopMs = latestStop ?? startMs; | ||
|
|
||
| // Counters are derived from the merged tests[] rather than trusting each input's summary, so | ||
| // summary.tests always equals the array length even when an input omitted or under-reported | ||
| // its summary. | ||
| long passed = 0, failed = 0, skipped = 0, pending = 0, other = 0, flaky = 0; | ||
| foreach (JsonNode? test in mergedTests) | ||
| { | ||
| if (test is null) | ||
| { | ||
| continue; | ||
| } | ||
|
|
||
| switch ((string?)test["status"]) | ||
| { | ||
| case "passed": passed++; break; | ||
| case "failed": failed++; break; | ||
| case "skipped": skipped++; break; | ||
| case "pending": pending++; break; | ||
| default: other++; break; | ||
| } | ||
|
|
||
| if (test["flaky"] is JsonValue flakyValue && flakyValue.TryGetValue(out bool isFlaky) && isFlaky) | ||
| { | ||
| flaky++; | ||
| } | ||
| } | ||
|
|
||
| var summaryObject = new JsonObject | ||
| { | ||
| ["tests"] = mergedTests.Count, | ||
| ["passed"] = passed, | ||
| ["failed"] = failed, | ||
| ["skipped"] = skipped, | ||
| ["pending"] = pending, | ||
| ["other"] = other, | ||
| ["flaky"] = flaky, | ||
| ["start"] = startMs, | ||
| ["stop"] = stopMs, | ||
| ["duration"] = Math.Max(0, stopMs - startMs), | ||
| }; | ||
|
|
||
| var resultsObject = new JsonObject(); | ||
| if (first["results"]?["tool"] is JsonNode tool) | ||
| { | ||
| resultsObject["tool"] = tool.DeepClone(); | ||
| } | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
|
|
||
| resultsObject["summary"] = summaryObject; | ||
|
|
||
| if (first["results"]?["environment"] is JsonNode environment) | ||
| { | ||
| resultsObject["environment"] = environment.DeepClone(); | ||
| } | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
|
|
||
| resultsObject["tests"] = mergedTests; | ||
|
|
||
| var merged = new JsonObject | ||
| { | ||
| ["reportFormat"] = first["reportFormat"]?.DeepClone() ?? "CTRF", | ||
| ["specVersion"] = first["specVersion"]?.DeepClone() ?? "0.0.0", | ||
| ["reportId"] = Guid.NewGuid().ToString("D"), | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
| ["timestamp"] = DateTimeOffset.FromUnixTimeMilliseconds(stopMs).ToString("O", CultureInfo.InvariantCulture), | ||
| ["results"] = resultsObject, | ||
| }; | ||
|
|
||
| if (first["generatedBy"] is JsonNode generatedBy) | ||
| { | ||
| merged["generatedBy"] = generatedBy.DeepClone(); | ||
| } | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
|
|
||
| return merged.ToJsonString(new JsonSerializerOptions { WriteIndented = true }); | ||
| } | ||
|
|
||
| internal static async Task MergeToFileAsync( | ||
| IReadOnlyList<string> inputPaths, | ||
| string outputPath, | ||
| CancellationToken cancellationToken) | ||
| { | ||
| if (inputPaths is null) | ||
| { | ||
| throw new ArgumentNullException(nameof(inputPaths)); | ||
| } | ||
|
|
||
| if (outputPath is null) | ||
| { | ||
| throw new ArgumentNullException(nameof(outputPath)); | ||
| } | ||
|
|
||
| var reports = new List<string>(inputPaths.Count); | ||
| foreach (string inputPath in inputPaths) | ||
| { | ||
| cancellationToken.ThrowIfCancellationRequested(); | ||
| #if NETCOREAPP | ||
| reports.Add(await File.ReadAllTextAsync(inputPath, cancellationToken).ConfigureAwait(false)); | ||
| #else | ||
| reports.Add(File.ReadAllText(inputPath)); | ||
| #endif | ||
| } | ||
|
|
||
| string merged = Merge(reports); | ||
|
|
||
| string? outputDirectory = Path.GetDirectoryName(outputPath); | ||
| if (!RoslynString.IsNullOrEmpty(outputDirectory)) | ||
| { | ||
| Directory.CreateDirectory(outputDirectory); | ||
| } | ||
|
|
||
| #if NETCOREAPP | ||
| await File.WriteAllTextAsync(outputPath, merged, cancellationToken).ConfigureAwait(false); | ||
| #else | ||
| File.WriteAllText(outputPath, merged); | ||
|
Evangelink marked this conversation as resolved.
Outdated
|
||
| await Task.CompletedTask.ConfigureAwait(false); | ||
| #endif | ||
| } | ||
|
|
||
| private static bool TryReadLong(JsonNode summary, string propertyName, out long value) | ||
| { | ||
| value = 0; | ||
| if (summary[propertyName] is not JsonValue jsonValue) | ||
| { | ||
| return false; | ||
| } | ||
|
|
||
| if (jsonValue.TryGetValue(out long longValue)) | ||
| { | ||
| value = longValue; | ||
| return true; | ||
| } | ||
|
|
||
| if (jsonValue.TryGetValue(out double doubleValue)) | ||
| { | ||
| value = (long)doubleValue; | ||
| return true; | ||
| } | ||
|
|
||
| return false; | ||
| } | ||
| } | ||
4 changes: 4 additions & 0 deletions
4
src/Platform/Microsoft.Testing.Extensions.CtrfReport/InternalAPI/InternalAPI.Unshipped.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,5 @@ | ||
| #nullable enable | ||
| virtual Microsoft.Testing.Extensions.ReportGeneratorBase<TGenerator, TCapturedTestResult>.ArtifactKind.get -> string? | ||
| Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger | ||
| static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.Merge(System.Collections.Generic.IReadOnlyList<string!>! inputReports) -> string! | ||
| static Microsoft.Testing.Extensions.CtrfReport.CtrfReportMerger.MergeToFileAsync(System.Collections.Generic.IReadOnlyList<string!>! inputPaths, string! outputPath, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! |
1 change: 1 addition & 0 deletions
1
src/Platform/Microsoft.Testing.Extensions.HangDump/InternalAPI/InternalAPI.Unshipped.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| #nullable enable | ||
| const Microsoft.Testing.Platform.IPC.FileArtifactMessageFieldsId.Kind = 7 -> ushort | ||
| static Microsoft.Testing.Platform.IPC.Serializers.BaseSerializer.ReadFields(System.IO.Stream! stream, System.Func<ushort, int, bool>! tryReadField) -> void | ||
| static Microsoft.Testing.Platform.IPC.Serializers.BaseSerializer.WriteListPayload<T>(System.IO.Stream! stream, ushort fieldId, T[]? list, System.Action<System.IO.Stream!, T>! writeItem) -> void |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
src/Platform/Microsoft.Testing.Extensions.HtmlReport/InternalAPI/InternalAPI.Unshipped.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,2 @@ | ||
| #nullable enable | ||
| virtual Microsoft.Testing.Extensions.ReportGeneratorBase<TGenerator, TCapturedTestResult>.ArtifactKind.get -> string? |
4 changes: 4 additions & 0 deletions
4
src/Platform/Microsoft.Testing.Extensions.JUnitReport/InternalAPI/InternalAPI.Unshipped.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,5 @@ | ||
| #nullable enable | ||
| virtual Microsoft.Testing.Extensions.ReportGeneratorBase<TGenerator, TCapturedTestResult>.ArtifactKind.get -> string? | ||
| Microsoft.Testing.Extensions.JUnitReport.JUnitReportMerger | ||
| static Microsoft.Testing.Extensions.JUnitReport.JUnitReportMerger.Merge(System.Collections.Generic.IReadOnlyList<System.Xml.Linq.XDocument!>! inputReports, string! reportName) -> System.Xml.Linq.XDocument! | ||
| static Microsoft.Testing.Extensions.JUnitReport.JUnitReportMerger.MergeToFileAsync(System.Collections.Generic.IReadOnlyList<string!>! inputPaths, string! outputPath, string! reportName, System.Threading.CancellationToken cancellationToken) -> System.Threading.Tasks.Task! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.