CI: publish perf and coverage metrics as a sticky PR comment#105
Merged
Conversation
The build-gcc-bsp job benchmarks parse and dry-run on the built gcc example (perf stat + peak RSS); the coverage job condenses gcovr's per-file line coverage into overall/median/min/max. A final job merges both fragments into one marker-keyed PR comment, updated in place on every push, and mirrors them to the step summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U5KUoGDhJEZYoiK9CkeXtz
PR metricsPerformance (gcc example, Linux)
Deterministic signals: page faults, peak RSS, and the cachegrind D1/LL miss rates (simulated cache, exact across runs). CPU time is the stable compute signal on shared runners; instructions read n/a on GitHub-hosted runners (virtualized, no PMU). Test coverage (lines)
95 files · 13778/16220 lines covered Updated for 7169376 |
GitHub-hosted runners are virtualized without a PMU, so cycles and instructions come back '<not supported>'; render those as n/a and add task-clock and page-faults as the stable software-counter signals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U5KUoGDhJEZYoiK9CkeXtz
Hosted runners have no PMU, so hardware cache counters are unavailable; cachegrind's simulated rates are deterministic to the last reference and fill that gap. --cache-sim=yes is required since valgrind 3.21 turned the simulation off by default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U5KUoGDhJEZYoiK9CkeXtz
Generators now emit TSV; compose_metrics.py renders the markdown and annotates deltas (percent for counts, percentage points for rates, MB for RSS) against baseline TSVs when present. The comment job fetches the baseline artifacts from the most recent successful push run on main and falls back to current-only when none exist — main runs before this change carry no TSV artifacts, so the first delta appears one merge after this lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U5KUoGDhJEZYoiK9CkeXtz
This was referenced Jul 18, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
pr-metrics-commentjob that posts one continuously-updated comment per PR with:Performance (Linux, measured in
build-gcc-bspagainst the fully built gcc example):parseand dry-run (-n) workloadsperf stat -r 3, user-space counters; runner needskernel.perf_event_paranoidlowered, done via sudo)/usr/bin/time -vCoverage (from the existing
coveragejob's gcovrsummary.json):Mechanics: each producing job uploads a small markdown fragment as an artifact and mirrors it to its step summary; the comment job (
permissions: pull-requests: write, PR events only) merges the fragments and upserts a comment keyed by an HTML marker, so pushes edit the existing comment instead of stacking new ones. Untrusted-input rule respected: only PR number, head SHA, and repo slug reach the shell, viaenv:.Both generators are repo scripts validated locally:
.github/scripts/bench_metrics.shproduced the table against a local gcc-example tree, and.github/scripts/coverage_stats.pyagainst a real gcovr summary (84.0% overall, median 91.5%, min 0.0%src/platform/path-posix.cpp, max 100%include/pup/core/arena.hpp). This PR's own checks exercise the workflow end-to-end — the comment below this description is the acceptance test.🤖 Generated with Claude Code
https://claude.ai/code/session_01U5KUoGDhJEZYoiK9CkeXtz