-
Notifications
You must be signed in to change notification settings - Fork 84
CI status watch #2091
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
CI status watch #2091
Changes from 11 commits
18f2678
d27c73a
862b485
60bfafb
70434a8
0d36b2d
a613459
f05d69d
163bc41
980739c
a9080ce
67999a2
ce0c647
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| name: CI status watch | ||
|
|
||
| # Posts a Teams message ONLY when one of the README badge workflows on the default | ||
| # branch turns red (green -> red). For each it compares the two latest runs with gh; a | ||
| # change is reported only if the newest run finished within the look-back window, so a | ||
| # workflow that stays red is not repeated every day. | ||
|
hohwille marked this conversation as resolved.
Outdated
|
||
| # Requires the repository secret TEAMS_WEBHOOK_URL. | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: '0 7 * * 1-5' # 07:00 UTC, Mon-Fri (= 09:00 CEST / 08:00 CET) | ||
|
|
||
| permissions: | ||
| actions: read | ||
|
|
||
| jobs: | ||
| watch: | ||
| runs-on: ubuntu-latest | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| WEBHOOK: ${{ secrets.TEAMS_WEBHOOK_URL }} | ||
| REPO: ${{ github.repository }} | ||
| BRANCH: ${{ github.ref_name }} | ||
| steps: | ||
| - name: Detect newly red badge workflows and notify Teams | ||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| # The workflows backing the README badges. Add a line here if a badge is added. | ||
| WORKFLOWS=" | ||
| .github/workflows/build.yml | ||
| .github/workflows/update-urls.yml | ||
| .github/workflows/nightly-build.yml | ||
| .github/workflows/integration-tests.yml | ||
| " | ||
|
|
||
| # Monday catches up the weekend (Fri-Sun); other weekdays look back 24h. | ||
| if [ "$(date -u +%u)" -eq 1 ]; then | ||
| SINCE=$(date -u -d '3 days ago' +%Y-%m-%dT%H:%M:%SZ) | ||
| else | ||
| SINCE=$(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ) | ||
| fi | ||
| echo "Checking badge workflows for red since $SINCE" | ||
|
hohwille marked this conversation as resolved.
Outdated
|
||
| echo "Resolved branch: $BRANCH" | ||
|
|
||
| is_red() { [ "$1" = "failure" ] || [ "$1" = "timed_out" ] || [ "$1" = "startup_failure" ]; } | ||
|
|
||
| # Compare each workflow's two latest runs; report it only if it just turned red | ||
| # and that run finished inside the window (so a lasting red is reported once). | ||
| RED=$(for WF_PATH in $WORKFLOWS; do | ||
| ROW=$(gh run list --repo "$REPO" --workflow "$WF_PATH" --branch "$BRANCH" \ | ||
| --status completed --limit 2 --json conclusion,url,updatedAt,workflowName \ | ||
| --jq '[(.[0].conclusion // ""), (.[0].updatedAt // ""), (.[0].url // ""), (.[1].conclusion // ""), (.[0].workflowName // "")] | @tsv' || true) | ||
| [ -z "$ROW" ] && continue | ||
| IFS=$'\t' read -r CUR CURTIME URL PREV NAME <<< "$ROW" | ||
| [ -z "$CUR" ] && continue | ||
| # lexicographic string comparison sufficient due to ISO-8601 format | ||
| [[ "$CURTIME" > "$SINCE" ]] || continue | ||
|
hohwille marked this conversation as resolved.
Outdated
hohwille marked this conversation as resolved.
Outdated
|
||
| if is_red "$CUR" && { [ "$PREV" = "success" ] || [ -z "$PREV" ]; }; then | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A genuinely red run is swallowed when the previous run was cancelled: for runs
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @quando632 thanks for this finding that I agree to. 👍
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually I was checking this again:
I would therefore suggest to go for KISS and simply send the notification whenever the workflow is failed. FYI: |
||
| jq -n --arg n "$NAME" --arg u "$URL" '{name:$n, url:$u}' | ||
| fi | ||
| done | jq -s '.') | ||
|
hohwille marked this conversation as resolved.
Outdated
|
||
|
|
||
| COUNT=$(echo "$RED" | jq 'length') | ||
| if [ "$COUNT" -eq 0 ]; then | ||
| echo "No badge workflow turned red. Skipping Teams notification." | ||
| exit 0 | ||
| fi | ||
| echo "$COUNT badge workflow(s) turned red." | ||
|
hohwille marked this conversation as resolved.
Outdated
|
||
|
|
||
| if [ -z "${WEBHOOK:-}" ]; then | ||
| echo "No Teams webhook set, skipping notification."; exit 0 | ||
| fi | ||
|
|
||
| CARD=$(echo "$RED" | jq --argjson count "$COUNT" '{ | ||
| type: "AdaptiveCard", | ||
| "$schema": "http://adaptivecards.io/schemas/adaptive-card.json", | ||
| version: "1.4", | ||
| body: ( | ||
| [ | ||
| { type: "TextBlock", size: "Large", weight: "Bolder", wrap: true, color: "attention", | ||
| text: "🔴 Workflow rot geworden" }, | ||
| { type: "TextBlock", spacing: "None", isSubtle: true, wrap: true, | ||
| text: (if $count == 1 then "1 Workflow · grün → rot" else "\($count) Workflows · grün → rot" end) } | ||
| ] | ||
| + [ .[] | { | ||
| type: "Container", | ||
| separator: true, | ||
| spacing: "Medium", | ||
| items: [ | ||
| { type: "TextBlock", weight: "Bolder", wrap: true, text: "🔴 \(.name)" }, | ||
| { type: "TextBlock", spacing: "None", isSubtle: true, size: "Small", wrap: true, | ||
| text: "[Run-Log öffnen](\(.url))" } | ||
|
hohwille marked this conversation as resolved.
Outdated
|
||
| ] | ||
| } ] | ||
| ) | ||
| }') | ||
|
|
||
| curl -sS --fail -X POST -H "Content-Type: application/json" -d "$CARD" "$WEBHOOK" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR description says the workflow lists the currently red workflows and keeps them listed until they turn green again. This header comment and the code actually implement a green-to-red transition that is deduplicated (a workflow that stays red is reported once, not every day). The comment here is the accurate one, so please update the PR description to match the implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe it would be even smarter if we keep posting to our channel every night if something is broken so we get spammed and want to get rid of the spam by fixing the problem and nobody can excuse that he missed the notification ;)