action: select the baseline by the PR base commit and explain it in the report

CHANGELOG.md

@@ -6,6 +6,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 - Remove the documented limitation about packages whose name differs from their directory, which was never an actual limitation (fgrosse/go-coverage-report#13)
+- **Behavior change:** By default, the baseline coverage is now taken from the latest successful run for the base commit
+  of the pull request (new `baseline-sha` input, defaults to `github.event.pull_request.base.sha`) instead of the latest
+  successful run on the target branch. If there is no such run, the action falls back to the latest successful run on the
+  target branch and emits a warning. Set `baseline-sha` to `""` to restore the previous behavior (fgrosse/go-coverage-report#110)
+- Add `baseline-run-id` input to use the coverage of an exact workflow run as baseline (fgrosse/go-coverage-report#110)
+- Improve diagnostics when no baseline coverage is available: warnings and the pull request comment now distinguish between
+  no successful baseline run being found and the coverage artifact of the selected run not being downloadable, and the
+  selected baseline run is logged with its ID, commit and age (fgrosse/go-coverage-report#110)
+- Show a caution callout at the top of the pull request comment if the report does not compare against the base commit
+  of the pull request (fgrosse/go-coverage-report#110)
+- Name the baseline commit and run in the details section of the report via the new `-baseline-commit`, `-baseline-run-id` and
+  `-baseline-run-url` flags (fgrosse/go-coverage-report#110)
+- Simplify the details section of the report: rename it to "Coverage details" and drop its headings to avoid mixing many
+  text sizes (fgrosse/go-coverage-report#110)
 
 ## [v1.4.0] - 2026-09-20
 - Also report packages whose coverage changed without any of their files being changed, e.g. when running integration tests with `go test -coverpkg` (fgrosse/go-coverage-report#102)

README.md

@@ -23,7 +23,7 @@ Example of a pull request comment created by `go-coverage-report`:
 
 ![Example of a pull request comment created by go-coverage-report](screenshot.png)
 
-Please note that by default, the "Coverage by file" section is collapsed so the focus
+Please note that by default, the "Coverage details" section is collapsed so the focus
 is more on the overall impact on the coverage _per package_.
 
 There is no indicator of the _total_ coverage of the project because the impact by
@@ -189,8 +189,46 @@ inputs:
       For other event types (e.g. push, schedule), set this explicitly.
     default: ${{ github.base_ref }}
     required: false
+
+  baseline-run-id:
+    description: |
+      The ID of the GitHub Actions workflow run whose coverage artifact should be used as
+      baseline. If set, this exact run is used and no other lookup happens. This is intended
+      for scripting, e.g. to feed the baseline run ID from another workflow into this action.
+    default: ""
+    required: false
+
+  baseline-sha:
+    description: |
+      The commit whose coverage should be used as baseline. The action uses the latest
+      successful run of the baseline workflow for this commit on the target branch.
+      If there is no such run, it falls back to the latest successful run on the target branch.
+      Must be a full commit SHA. Defaults to the base commit of the pull request.
+      Set to "" to disable matching by commit and always use the latest run on the target branch.
+    default: ${{ github.event.pull_request.base.sha }}
+    required: false
 ```
 
+### Baseline selection
+
+The coverage of the pull request is compared against the coverage artifact of a successful
+run of the baseline workflow (see `github-baseline-workflow-ref` and `event-name`). The run is
+selected as follows:
+
+1. If `baseline-run-id` is set, exactly this run is used.
+2. Otherwise, if `baseline-sha` is set (by default the base commit of the pull request), the
+   latest successful run for this commit on the `target-branch` is used.
+3. If there is no such run, or `baseline-sha` is empty, the latest successful run on the
+   `target-branch` is used. If a `baseline-sha` was given, the action emits a warning because
+   the coverage changes may then include commits that are not part of the pull request.
+
+The selected run (ID, commit, age and URL) is logged, and the "Coverage details" section of the
+pull request comment names the baseline commit and run. If the report does not compare against
+the base commit of the pull request, a caution callout at the top of the comment explains why:
+either the latest run on the `target-branch` was used instead, no successful baseline run was
+found, or the coverage artifact of the selected run could not be downloaded (e.g. because it
+expired). In the latter two cases, the comment only shows the current coverage of the changed files.
+
 ### Outputs
 
 This action provides the following outputs:
@@ -225,11 +263,12 @@ This action provides the following outputs:
 
 - Currently, code coverage profiles are uploaded as GitHub artifacts which automatically expire after 90 days.
   In a repository which receives changes only infrequently, this might lead to issues when trying to compare
-  the code coverage of a pull request with the code coverage of the main branch (see fgrosse/go-coverage-report#5).  
+  the code coverage of a pull request with the code coverage of the main branch (see fgrosse/go-coverage-report#5).
+  In this case, the report only shows the current coverage of the changed files and a note explains why.
 - Support **for forks** is limited since the necessary `GITHUB_TOKEN` permissions don't allow to post comments to the
   pull request of the base repository (see fgrosse/go-coverage-report#15). If forks are important for you, this action
   might not be the best solution.
-- The "Coverage by file" section only lists changed files. Packages that are listed only because
+- The "Coverage details" section only lists changed files. Packages that are listed only because
   their coverage changed (e.g. when using `go test -coverpkg`) have no per-file breakdown.
 - Requires `actions/upload-artifact` >= **v4** (see this [issue][upload-artifacts-issues]).
 

action.yml

@@ -76,6 +76,24 @@ inputs:
     default: ${{ github.base_ref }}
     required: false
 
+  baseline-run-id:
+    description: |
+      The ID of the GitHub Actions workflow run whose coverage artifact should be used as
+      baseline. If set, this exact run is used and no other lookup happens. This is intended
+      for scripting, e.g. to feed the baseline run ID from another workflow into this action.
+    default: ""
+    required: false
+
+  baseline-sha:
+    description: |
+      The commit whose coverage should be used as baseline. The action uses the latest
+      successful run of the baseline workflow for this commit on the target branch.
+      If there is no such run, it falls back to the latest successful run on the target branch.
+      Must be a full commit SHA. Defaults to the base commit of the pull request.
+      Set to "" to disable matching by commit and always use the latest run on the target branch.
+    default: ${{ github.event.pull_request.base.sha }}
+    required: false
+
 outputs:
   coverage_report:
     description: 'The generated coverage report in Markdown format.'
@@ -139,3 +157,5 @@ runs:
         TRIM_PACKAGE: ${{ inputs.trim }}
         EXCLUDE: ${{ inputs.exclude }}
         EVENT_NAME: ${{ inputs.event-name }}
+        REQUESTED_BASELINE_RUN_ID: ${{ inputs.baseline-run-id }}
+        REQUESTED_BASELINE_SHA: ${{ inputs.baseline-sha }}

cmd/go-coverage-report/main.go

@@ -38,6 +38,7 @@ type options struct {
 	format      string
 	exclude     *regexp.Regexp
 	metricsFile string
+	baseline    Baseline
 }
 
 func main() {
@@ -53,6 +54,9 @@ func main() {
 	flag.String("format", "markdown", "output format (currently only 'markdown' is supported)")
 	flag.String("exclude", "", "exclude files matching the given regular expression from the report")
 	flag.String("metrics-file", "", "write key=value coverage metrics to this file for GitHub Actions outputs")
+	flag.String("baseline-commit", "", "the commit SHA of the old coverage, shown in the markdown report (optional)")
+	flag.String("baseline-run-id", "", "the ID of the workflow run that produced the old coverage, shown in the markdown report (optional)")
+	flag.String("baseline-run-url", "", "the URL of the workflow run that produced the old coverage (optional)")
 
 	err := run(programArgs())
 	if err != nil {
@@ -77,6 +81,11 @@ func programArgs() (oldCov, newCov, changedFile string, opts options) {
 		trim:        flag.Lookup("trim").Value.String(),
 		format:      flag.Lookup("format").Value.String(),
 		metricsFile: flag.Lookup("metrics-file").Value.String(),
+		baseline: Baseline{
+			Commit: flag.Lookup("baseline-commit").Value.String(),
+			RunID:  flag.Lookup("baseline-run-id").Value.String(),
+			RunURL: flag.Lookup("baseline-run-url").Value.String(),
+		},
 	}
 
 	if s := flag.Lookup("exclude").Value.String(); s != "" {
@@ -118,6 +127,9 @@ func run(oldCovPath, newCovPath, changedFilesPath string, opts options) error {
 	changedFiles = excludeFiles(changedFiles, opts.exclude)
 
 	report := NewReport(oldCov, newCov, changedFiles)
+	if opts.baseline.Commit != "" {
+		report.Baseline = &opts.baseline
+	}
 	if opts.trim != "" {
 		report.TrimPrefix(opts.trim)
 	}

cmd/go-coverage-report/report.go

@@ -14,6 +14,14 @@ type Report struct {
 	Old, New        *Coverage
 	ChangedFiles    []string
 	ChangedPackages []string
+	Baseline        *Baseline `json:",omitempty"`
+}
+
+// Baseline describes where the old coverage profile of the report came from.
+type Baseline struct {
+	Commit string // full or abbreviated commit SHA
+	RunID  string // ID of the workflow run that produced the old coverage (optional)
+	RunURL string // URL of that workflow run (optional)
 }
 
 func NewReport(oldCov, newCov *Coverage, changedFiles []string) *Report {
@@ -147,14 +155,16 @@ func (r *Report) impactedPackages(oldCovPkgs, newCovPkgs map[string]*Coverage) [
 }
 
 func (r *Report) addDetails(report *strings.Builder) {
-	fmt.Fprintln(report, "---")
-	fmt.Fprintln(report)
 	fmt.Fprintln(report, "<details>")
 	fmt.Fprintln(report)
 
-	fmt.Fprintln(report, "<summary>Coverage by file</summary>")
+	fmt.Fprintln(report, "<summary>Coverage details</summary>")
 	fmt.Fprintln(report)
 
+	if r.Baseline != nil && r.Baseline.Commit != "" {
+		fmt.Fprintf(report, "<sub>%s</sub>\n\n", r.Baseline.Markdown())
+	}
+
 	var codeFiles, unitTestFiles []string
 	for _, f := range r.ChangedFiles {
 		if strings.HasSuffix(f, "_test.go") {
@@ -174,9 +184,26 @@ func (r *Report) addDetails(report *strings.Builder) {
 	fmt.Fprint(report, "</details>")
 }
 
+// Markdown returns a single line describing the baseline, e.g.
+// "Compared to commit 2eb52af (run [#8221109494](https://…))".
+func (b *Baseline) Markdown() string {
+	commit := b.Commit
+	if len(commit) > 7 {
+		commit = commit[:7]
+	}
+
+	var run string
+	switch {
+	case b.RunID != "" && b.RunURL != "":
+		run = fmt.Sprintf(" (run [#%s](%s))", b.RunID, b.RunURL)
+	case b.RunID != "":
+		run = fmt.Sprintf(" (run #%s)", b.RunID)
+	}
+
+	return fmt.Sprintf("Compared to commit %s%s", commit, run)
+}
+
 func (r *Report) addCodeFileDetails(report *strings.Builder, files []string) {
-	fmt.Fprintln(report, "### Changed files (no unit tests)")
-	fmt.Fprintln(report)
 	fmt.Fprintln(report, "| Changed File | Coverage Δ | Total | Covered | Missed | :robot: |")
 	fmt.Fprintln(report, "|--------------|------------|-------|---------|--------|---------|")
 
@@ -225,8 +252,7 @@ func (r *Report) addCodeFileDetails(report *strings.Builder, files []string) {
 }
 
 func (r *Report) addTestFileDetails(report *strings.Builder, files []string) {
-	fmt.Fprintln(report, "### Changed unit test files")
-	fmt.Fprintln(report)
+	fmt.Fprintln(report, "Changed unit test files:")
 
 	for _, name := range files {
 		fmt.Fprintf(report, "- %s\n", name)

cmd/go-coverage-report/report_test.go

@@ -29,13 +29,9 @@ func TestReport_Markdown(t *testing.T) {
 |-------------------|------------|---------|
 | github.com/fgrosse/prioqueue | 90.20% (**-9.80%**) | :thumbsdown: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
-
-### Changed files (no unit tests)
+<summary>Coverage details</summary>
 
 | Changed File | Coverage Δ | Total | Covered | Missed | :robot: |
 |--------------|------------|-------|---------|--------|---------|
@@ -90,14 +86,11 @@ func TestReport_Markdown_OnlyChangedUnitTests(t *testing.T) {
 |-------------------|------------|---------|
 | github.com/fgrosse/prioqueue | 99.02% (**+8.82%**) | :thumbsup: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
-
-### Changed unit test files
+<summary>Coverage details</summary>
 
+Changed unit test files:
 - github.com/fgrosse/prioqueue/min_heap_test.go
 
 </details>`
@@ -127,14 +120,11 @@ func TestReport_Markdown_DeletedUnitTestFile(t *testing.T) {
 |-------------------|------------|---------|
 | github.com/fgrosse/prioqueue | 90.20% (**-8.82%**) | :thumbsdown: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
-
-### Changed unit test files
+<summary>Coverage details</summary>
 
+Changed unit test files:
 - github.com/fgrosse/prioqueue/min_heap_test.go
 
 </details>`
@@ -164,14 +154,11 @@ func TestReport_Markdown_CoverPkg(t *testing.T) {
 |-------------------|------------|---------|
 | example.com/demo/calc | 83.33% (**+50.00%**) | :star2: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
-
-### Changed unit test files
+<summary>Coverage details</summary>
 
+Changed unit test files:
 - example.com/demo/integration/integration_test.go
 
 </details>`
@@ -202,11 +189,9 @@ func TestReport_Markdown_CoverPkgExcluded(t *testing.T) {
 |-------------------|------------|---------|
 | example.com/demo/calc | 83.33% (**+50.00%**) | :star2: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
+<summary>Coverage details</summary>
 
 </details>`
 	assert.Equal(t, expected, actual)
@@ -238,13 +223,9 @@ func TestReport_Markdown_PackageNameDiffersFromDirectory(t *testing.T) {
 | github.com/owner/project/baz | 100.00% (**+100.00%**) | :star2: |
 | github.com/owner/project/foo-bar | 100.00% (**+33.33%**) | :star2: |
 
----
-
 <details>
 
-<summary>Coverage by file</summary>
-
-### Changed files (no unit tests)
+<summary>Coverage details</summary>
 
 | Changed File | Coverage Δ | Total | Covered | Missed | :robot: |
 |--------------|------------|-------|---------|--------|---------|
@@ -253,8 +234,7 @@ func TestReport_Markdown_PackageNameDiffersFromDirectory(t *testing.T) {
 
 _Please note that the "Total", "Covered", and "Missed" counts above refer to ***code statements*** instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code._
 
-### Changed unit test files
-
+Changed unit test files:
 - github.com/owner/project/baz/baz_test.go
 - github.com/owner/project/foo-bar/foo_bar_test.go
 
@@ -334,3 +314,57 @@ func parseCoverageString(t *testing.T, profile string) *Coverage {
 
 	return New(profiles)
 }
+
+func TestReport_Markdown_Baseline(t *testing.T) {
+	oldCov, err := ParseCoverage("testdata/02-old-coverage.txt", nil)
+	require.NoError(t, err)
+
+	newCov, err := ParseCoverage("testdata/02-new-coverage.txt", nil)
+	require.NoError(t, err)
+
+	changedFiles, err := ParseChangedFiles("testdata/02-changed-files.json", "github.com/fgrosse/prioqueue")
+	require.NoError(t, err)
+
+	report := NewReport(oldCov, newCov, changedFiles)
+	report.Baseline = &Baseline{
+		Commit: "2eb52af2e3c0c7d6b1d1a8e0f7e4f9d8c1b2a3f4",
+		RunID:  "8221109494",
+		RunURL: "https://github.com/fgrosse/prioqueue/actions/runs/8221109494",
+	}
+	actual := report.Markdown()
+
+	expected := `### Merging this branch will **increase** overall coverage
+
+| Impacted Packages | Coverage Δ | :robot: |
+|-------------------|------------|---------|
+| github.com/fgrosse/prioqueue | 99.02% (**+8.82%**) | :thumbsup: |
+
+<details>
+
+<summary>Coverage details</summary>
+
+<sub>Compared to commit 2eb52af (run [#8221109494](https://github.com/fgrosse/prioqueue/actions/runs/8221109494))</sub>
+
+Changed unit test files:
+- github.com/fgrosse/prioqueue/min_heap_test.go
+
+</details>`
+	assert.Equal(t, expected, actual)
+}
+
+func TestBaseline_Markdown(t *testing.T) {
+	cases := map[string]struct {
+		baseline Baseline
+		expected string
+	}{
+		"commit only":        {Baseline{Commit: "2eb52af"}, "Compared to commit 2eb52af"},
+		"run without url":    {Baseline{Commit: "2eb52af", RunID: "42"}, "Compared to commit 2eb52af (run #42)"},
+		"full sha shortened": {Baseline{Commit: "2eb52af2e3c0", RunID: "42", RunURL: "https://x/42"}, "Compared to commit 2eb52af (run [#42](https://x/42))"},
+	}
+
+	for name, c := range cases {
+		t.Run(name, func(t *testing.T) {
+			assert.Equal(t, c.expected, c.baseline.Markdown())
+		})
+	}
+}