Skip to content

fix(ci): reject incompatible Kubernetes and Talos pins - #3583

Merged
devantler merged 7 commits into
mainfrom
codex/platform-kubernetes-talos-compatibility-3536
Sep 4, 2026
Merged

fix(ci): reject incompatible Kubernetes and Talos pins#3583
devantler merged 7 commits into
mainfrom
codex/platform-kubernetes-talos-compatibility-3536

Conversation

@devantler

Copy link
Copy Markdown
Contributor

🤖 Generated by the Agentic Engineer

Summary

  • Validate the explicit Kubernetes/Talos pins in ksail.prod.yaml using Talos's versioned KubernetesVersion.SupportedWith predicate, also used by its runtime configuration validation.
  • Run the check in the existing required Talos validation job and select that job when production pins, validator sources, the Talos installer, or Go dependencies change.
  • Fail closed for unknown Talos release families, missing/malformed pins, and ambiguous YAML. The diagnostic names both versions and the separate Talos upgrade prerequisite.

Closes #3536. Related to #3534, which remains blocked independently until the staged Talos upgrade has completed. This does not change production versions, Renovate automerge policy, or merge-queue/deployment behavior.

Why this catches the failure

The existing offline talosctl validate, even with --strict, accepts Kubernetes v1.37.0 with Talos v1.13.9. It does not execute the runtime compatibility predicate. Calling the pinned upstream machinery library directly exercises the same version-range decision without a cluster, credentials, generated machine secrets, or a copied compatibility table.

Verification

  • RED: the v1.37.0 / v1.13.9 regression failed against an accepting implementation; workflow selection tests failed while production/dependency inputs were absent from the Talos filter.
  • GREEN: focused Go tests and vet pass. Current v1.36.4 / v1.13.9 and an in-range v1.36.5 patch pass; v1.37.0 / v1.13.9 and unknown Talos releases are rejected.
  • Ran the CLI against the actual production config (success) and a synthetic chore(deps): update dependency kubernetes/kubernetes to v1.37.0 #3534 pairing (exit 1 with both pins and Talos-upgrade guidance).
  • The existing patched machine-config render/validation step passes for production and local control-plane/worker configs with the checksum-verified pinned Talos CLI.
  • actionlint .github/workflows/ci.yaml and git diff --check pass.
  • The full go test ./... suite passes with its required network fixtures available.

The PR remains a draft until current-head CI and substantive review complete. A passing declared pairing is not evidence that an OS rollout has finished and does not clear #3534 for merging.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@devantler I will review the changes.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Team

Run ID: 085c4662-5bd4-4460-8f55-2be779466cc1

📥 Commits

Reviewing files that changed from the base of the PR and between 902bdfb and 9adb4ef.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (8)
  • .github/workflows/cd.yaml
  • .github/workflows/ci.yaml
  • .github/workflows/validate-main.yaml
  • go.mod
  • scripts/validate-dr-signing/main_test.go
  • scripts/validate-talos-kubernetes-compatibility/README.md
  • scripts/validate-talos-kubernetes-compatibility/main.go
  • scripts/validate-talos-kubernetes-compatibility/main_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2026-08-04T13:06:25.700Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2944
File: scripts/validate-flux-verify/main_test.go:300-305
Timestamp: 2026-08-04T13:06:25.700Z
Learning: For validator acceptance tests under scripts/**/main_test.go, fixed repository-relative paths passed to os.ReadFile do not require //nolint:gosec: golangci-lint does not run gosec on these test-file calls. Apply gosec G304 suppressions only to non-test Go code. Use scripts/validate-dr-signing/main_test.go as the reference analogue.

Applied to files:

  • scripts/validate-talos-kubernetes-compatibility/main_test.go
🪛 golangci-lint (2.13.2)
scripts/validate-talos-kubernetes-compatibility/main.go

[error] 60-60: comparing with != will fail on wrapped errors. Use errors.Is to check for a specific error

(errorlint)

scripts/validate-talos-kubernetes-compatibility/main_test.go

[error] 204-204: QF1001: could apply De Morgan's law

(staticcheck)

🪛 OSV Scanner (2.5.0)
go.mod

[LOW] 15-15: golang.org/x/net 0.55.0: Parsing an invalid SVCB or HTTPS RR can panic in golang.org/x/net/dns/dnsmessage

(GO-2026-5942)


[LOW] 17-17: golang.org/x/text 0.37.0: Infinite loop on invalid input in golang.org/x/text

(GO-2026-5970)


[HIGH] 19-19: google.golang.org/grpc 1.81.0: Vulnerabilities in the xDS RBAC authorization engine and the HTTP/2 transport server implementation in google.golang.org/grpc

(GO-2026-6061)


[HIGH] 19-19: google.golang.org/grpc 1.81.0: gRPC-Go: xDS RBAC and HTTP/2 Vulnerabilities

(GHSA-hrxh-6v49-42gf)


[HIGH] 19-19: google.golang.org/grpc 1.81.0: gRPC-Go: Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation

(GHSA-vp52-pcj8-j9qc)

🔇 Additional comments (8)
go.mod (1)

5-21: LGTM!

scripts/validate-talos-kubernetes-compatibility/main.go (1)

1-82: LGTM!

scripts/validate-talos-kubernetes-compatibility/main_test.go (1)

1-207: LGTM!

scripts/validate-talos-kubernetes-compatibility/README.md (1)

1-30: LGTM!

.github/workflows/ci.yaml (1)

457-461: LGTM!

Also applies to: 1630-1636, 1650-1659, 1804-1804, 1903-1914

.github/workflows/validate-main.yaml (1)

140-169: LGTM!

.github/workflows/cd.yaml (1)

63-87: LGTM!

Also applies to: 285-285

scripts/validate-dr-signing/main_test.go (1)

192-192: LGTM!

Also applies to: 194-194, 1559-1559, 1567-1567


📝 Walkthrough

Walkthrough

The change adds a Go validator that reads Kubernetes and Talos version pins from ksail.prod.yaml and checks them with Talos machinery compatibility rules. It validates version formats, malformed YAML, duplicate documents, unknown Talos releases, and incompatible pairings. Tests cover supported and rejected versions. CI and deployment workflows now run the validator for relevant changes, merge groups, main-branch pushes, manual deployments, and recovery deployments.

Merge Risk: ⚪ Minimal · up to 9adb4

This adds pre-deployment validation for Kubernetes and Talos version pins across production paths. Unsupported or malformed combinations are blocked before deployment, with no remaining concrete merge-readiness risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 3 files. (5 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes satisfy issue #3536 [#3536]. The validator derives compatibility from Talos's SupportedWith predicate, rejects incompatible or unknown versions, reports both pins and the required lockstep…
Out of Scope Changes check ✅ Passed The workflow, deployment-path, documentation, dependency, and test changes are related to the linked issue and stated objective of preventing incompatible Kubernetes/Talos pins from reaching productio…
Title check ✅ Passed The title clearly summarizes the primary change: CI now rejects incompatible Kubernetes and Talos version pins.
Description check ✅ Passed The description directly explains the validator, workflow integration, failure behavior, verification, and deployment-gating objectives.
Full details: Docstring Coverage

Explanation

Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 3 files. (5 skipped: 5 unsupported.)


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…points

The sibling validator scripts in scripts/ document their functions (6/8 to
52/54); this one documented 0/2, which CodeRabbit's current-head pre-merge
docstring check reported. Comment-only: no behaviour change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Readiness evaluation — exercised as a user, not reasoned about.

Built the validator from this branch and ran the real binary against real inputs. Observed behaviour, exit codes as printed:

Input Result Exit
the actual ksail.prod.yaml (K8s v1.36.4 / Talos v1.13.9) pinned Kubernetes/Talos versions are compatible 0
the exact #3534 pairing — K8s v1.37.0 / Talos v1.13.9 Kubernetes v1.37.0 is not verified compatible with Talos v1.13.9: version of Kubernetes 1.37.0 is too new to be used with Talos 1.13.9 + the Talos-upgrade guidance 1
malformed pin (kubernetesVersion: latest) explicit vMAJOR.MINOR.PATCH Talos and Kubernetes pins are required (got Talos "v1.13.9", Kubernetes "latest") 1
unknown future Talos family (v1.99.0) compatibility with version 1.99.0 is not supported — fails closed rather than inheriting an old ceiling 1

The middle row is the one that matters: this is the pairing #3534 proposes, and the validator rejects it with both versions named and the upgrade prerequisite spelled out. Judged as the CI consumer, the diagnostic says what to do next without needing the source.

Also run at this head: focused go test (pass), go vet (clean), gofmt -l (empty).

One review finding fixed since the green. CodeRabbit's current-head pre-merge docstring check reported 16.67% against an 80% threshold. Assessed on merit rather than waved: the sibling validators under scripts/ document 6/8, 25/33, 52/54, 12/15, 28/33 and 14/15 of their functions, so documenting is this repository's actual convention and this script's 0/2 was the outlier. Fixed in 1baa2838, comment-only — verified as comment-only by asserting the diff adds no non-comment line and removes nothing; behaviour re-checked identical after the change (rows 1 and 2 above re-run against the rebuilt binary).

That push stales the green review at cd91b5ab, so the lane restarts at CodeRabbit on the new head per the review-request order.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@devantler I will perform a full review of pull request #3583.

⚠️ Action not completed

Review rate limited.


Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 16 minutes.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

CodeRabbit refused this round at 1baa2838. Its reply (15:37:41Z, updated 15:38:00Z) postdates this round's request marker (15:37:07Z), so the refusal belongs to this round rather than to the spent one already sitting at this head: ⚠️ Action not completed — Review rate limited, next included review ~16 minutes out.

Recorded as cr:no-gate@1baa2838. Advancing to Codex per the lane order.

CI is green at this head — 🧪 Validate Manifests passed, 0 failing of 25 — and the PR carries 0 review threads and 0 outstanding findings, so the only thing between it and merge is one current-head green.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-04T15:42:50.948880Z 1baa283 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1baa2838c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yaml
ci.yaml's validate-talos job is pull_request-only, so a pairing pushed
straight to main reached the manually dispatched CD route unchecked.
Mirror the compatibility check onto the push event in validate-main.yaml,
the same way validate-shared-publish-pin mirrors its PR gate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Current commit and changed files =='
git rev-parse HEAD
git diff --name-status HEAD^ HEAD

printf '%s\n' '== Diff =='
git diff --unified=80 HEAD^ HEAD -- \
  .github/workflows/ci.yaml \
  go.mod \
  go.sum \
  scripts/validate-talos-kubernetes-compatibility

printf '%s\n' '== Validator structure =='
ast-grep outline scripts/validate-talos-kubernetes-compatibility --items all

printf '%s\n' '== Relevant workflow context =='
rg -n -C 12 'validate-talos|talos:|ksail\.prod\.yaml|setup-go|validate-talos-kubernetes-compatibility' .github/workflows/ci.yaml

printf '%s\n' '== Production pins =='
rg -n -C 3 'kubernetesVersion|talosVersion' ksail.prod.yaml

printf '%s\n' '== Sibling validator entry points =='
fd -t f 'main\.go|README\.md|main_test\.go' scripts -d 3 | sort

Length of output: 21784


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Pull request metadata =='
gh pr view 3583 --repo devantler-tech/platform --json baseRefName,baseRefOid,headRefName,headRefOid,changedFiles,commits

base_oid="$(gh pr view 3583 --repo devantler-tech/platform --json baseRefOid --jq .baseRefOid)"
head_oid="$(git rev-parse HEAD)"

printf '%s\n' '== Full pull request changed files =='
git diff --name-status "${base_oid}...${head_oid}"

printf '%s\n' '== Full pull request diff for review targets =='
git diff --unified=100 "${base_oid}...${head_oid}" -- \
  .github/workflows/ci.yaml \
  .github/workflows/validate-main.yaml \
  go.mod \
  go.sum \
  scripts/validate-talos-kubernetes-compatibility

printf '%s\n' '== Validator implementation =='
cat -n scripts/validate-talos-kubernetes-compatibility/main.go

printf '%s\n' '== Validator tests =='
cat -n scripts/validate-talos-kubernetes-compatibility/main_test.go

printf '%s\n' '== Current-head validate-main workflow change =='
git diff --unified=80 HEAD^ HEAD -- .github/workflows/validate-main.yaml

printf '%s\n' '== Full validate-main job context =='
rg -n -C 30 'validate-talos|CI - Required Checks|workflow_call|workflow_dispatch|merge_group|pull_request' .github/workflows/validate-main.yaml

Length of output: 88151


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== CI workflow triggers and merge-group consumers =='
sed -n '1,80p' .github/workflows/ci.yaml
rg -n -C 12 'merge_group|deploy-prod|validate-talos|ci-required-checks' .github/workflows/ci.yaml

printf '%s\n' '== Production Talos pin and surrounding schema =='
rg -n -C 12 '^[[:space:]]*talos:|^[[:space:]]*version:[[:space:]]*v[0-9]' ksail.prod.yaml

printf '%s\n' '== Deployment and main-validation workflow references =='
rg -n -C 10 'validate-talos-kubernetes-compatibility|validate-talos|workflow_dispatch|ksail\.prod\.yaml' \
  .github/workflows/cd.yaml \
  .github/workflows/validate-main.yaml \
  .github/workflows/ci.yaml

printf '%s\n' '== Existing workflow-wiring tests and validators =='
rg -n -C 8 'validate-main|validate-shared-publish-pin|workflow.*job|job.*workflow|paths-filter' \
  scripts --glob '*.go' --glob '*.sh' --glob '*.yaml'

printf '%s\n' '== Talos compatibility API source, if available in module cache =='
mod_dir="$(go env GOMODCACHE 2>/dev/null || true)"
if [ -n "$mod_dir" ] && [ -d "$mod_dir" ]; then
  fd -HI 'compatibility.*\.go|version.*\.go' "$mod_dir/github.com/siderolabs/talos/pkg/machinery@v1.13.9" \
    -x sh -c 'printf "\n-- %s --\n" "$1"; rg -n -C 5 "func .*SupportedWith|func ParseTalosVersion|func ParseKubernetesVersion" "$1"' sh {} \; \
    2>/dev/null || true
fi

Length of output: 224538


@devantler Full review completed. I found two blocking coverage gaps.

  1. P1 — Run the compatibility validator on the merge-group revision.
    .github/workflows/ci.yaml Line 1636 limits validate-talos to pull_request. The deploy-prod job runs on merge_group and does not depend on this validator. Therefore, a merge-group revision can deploy an incompatible effective pair without executing KubernetesVersion.SupportedWith. Run the validator when github.event_name == 'merge_group' and make deploy-prod depend on its result.

  2. P1 — Gate the manual CD deployment with the validator.
    .github/workflows/validate-main.yaml Line 150 runs only after a direct push reaches main. .github/workflows/cd.yaml can be manually dispatched independently and has no dependency on that run. A user can start CD before, or despite, the post-push validation result. Add the same validator as a gate in cd.yaml, then add it to deploy-prod.needs.

You are interacting with an AI system.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Two P1s reported by CodeRabbit at 06c122c2 — both assessed VALID and both left OPEN. This PR is NEEDS-FIX, not ready.

P1-1 — merge-group revision is unguarded. Confirmed: ci.yaml's validate-talos carries if: github.event_name == 'pull_request', while deploy-prod runs on merge_group and does not list the validator in its needs. A merge-group revision can therefore deploy a pairing the validator would reject. I had already flagged this residual in the thread above before CodeRabbit reported it independently; it agrees, and rates it P1 rather than a follow-up. Reclassifying it as in-scope for this PR.

P1-2 — my 06c122c2 fix is necessary but NOT sufficient. Confirmed: cd.yaml is workflow_dispatch-only and can be dispatched independently of the post-push run in validate-main.yaml, so nothing sequences them — a manual CD can start before, or despite, that validation. The push-side mirror closes detection on main; it does not gate the deploy route. The repository already shows the right shape: cd.yaml gates its own jobs on validator jobs (needs: [validate-shared-publish-pin], needs: [validate-pvc-prune-safety, validate-rgd-templates]), so the validator belongs there as a job with deploy-prod depending on it.

Recording rather than fixing: both are real workflow-wiring changes across two files with a deploy path attached, and this run is out of time. Fixing them tired at the end of a run is how a deploy gate acquires a quiet hole. Next run picks this up with the two remedies above already specified.

Method note, second occurrence today. This finding arrived through a CodeRabbit reply that again opened with "For best results, initiate chat on the files or code changes" and an 🧩 Analysis chain — the chat-misparse shape — this time in response to @coderabbitai full review, which the earlier misparse on #3587 had been fixed by. So neither trigger form is reliably parsed as a review, and in both cases the misparsed reply still carried genuine P1/P2 findings. A misparse is not a failed request and must not be discarded as one.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@devantler The assessment is correct.

validate-talos must run on the merge-group revision, and deploy-prod must require its successful result.

cd.yaml must run the same compatibility validator, and its deploy-prod job must require the validator.

Both P1 findings remain open. Pull request #3583 is not ready to merge.

The response formatting does not invalidate its findings. A chat-oriented CodeRabbit response can still contain substantive review results and must be assessed on its technical evidence.

🐇⚠️


✏️ Learnings added
Learnt from: devantler
URL: https://github.com/devantler-tech/platform/pull/3583

Timestamp: 2026-09-04T15:51:40.389Z
Learning: In GitHub pull request reviews, a CodeRabbit response that uses chat-oriented formatting such as “For best results, initiate chat on the files or code changes” and an analysis chain can still contain substantive P1 or P2 review findings. Assess the technical findings on their evidence; do not treat this formatting as a failed review request.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Resolved both current-head P1s from CodeRabbit's review output in 0fecb0545afb99d31745fa26b522a5fefcaea754.

  • Merge queue: validate-talos now runs on every merge_group revision, and deploy-prod requires that job. An unsupported Kubernetes/Talos pairing therefore stops before the speculative production deploy.
  • Manual CD: cd.yaml now runs the same production-pin validator in its own pre-secret job, and deploy-prod requires it. A direct-push recovery dispatch can no longer bypass compatibility validation.
  • Regression coverage: added workflow-graph tests for both routes and updated the existing DR-signing ablations so their mutations still exercise the live dependency lists. The new tests failed against the previous wiring for the two expected missing-gate reasons, then passed after the workflow changes.

Validation at this head: actionlint .github/workflows/ci.yaml .github/workflows/cd.yaml, go vet ./..., go test ./..., gofmt -l (empty), and git diff --check all pass. The full suite's pinned remote Kustomize reads and temporary safety-test plumbing were run under the required host boundary.

This push also merges Platform main at 902bdfb41a1031f4e84570c36f8011725f1d8de7. The PR remains draft while exact-head CI and a fresh substantive review run; this note records the fixes, not readiness.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== PR metadata =='
gh pr view 3583 --repo devantler-tech/platform \
  --json baseRefName,baseRefOid,headRefName,headRefOid,changedFiles,commits

base_oid="$(gh pr view 3583 --repo devantler-tech/platform --json baseRefOid --jq .baseRefOid)"
head_oid="$(git rev-parse HEAD)"
printf '\n== Checked out HEAD ==\n%s\n' "$head_oid"

printf '%s\n' '== Full changed-file list =='
git diff --name-status "${base_oid}...${head_oid}"

printf '%s\n' '== Full PR diff =='
git diff --check "${base_oid}...${head_oid}"
git diff --unified=100 "${base_oid}...${head_oid}" -- \
  .github/workflows/ci.yaml \
  .github/workflows/cd.yaml \
  .github/workflows/validate-main.yaml \
  go.mod \
  go.sum \
  scripts/validate-talos-kubernetes-compatibility

printf '%s\n' '== Changed source outlines =='
ast-grep outline scripts/validate-talos-kubernetes-compatibility --items all

printf '%s\n' '== CI workflow gate context =='
rg -n -C 35 '^[[:space:]]*(validate-talos|deploy-prod):|merge_group|pull_request|needs:|validate-talos-kubernetes-compatibility' .github/workflows/ci.yaml

printf '%s\n' '== CD workflow gate context =='
rg -n -C 35 '^[[:space:]]*(validate-talos|deploy-prod):|workflow_dispatch|needs:|validate-talos-kubernetes-compatibility' .github/workflows/cd.yaml

printf '%s\n' '== Validator implementation and tests =='
cat -n scripts/validate-talos-kubernetes-compatibility/main.go
cat -n scripts/validate-talos-kubernetes-compatibility/main_test.go

printf '%s\n' '== Production pin context =='
rg -n -C 8 'kubernetesVersion|talosVersion|^[[:space:]]*talos:' ksail.prod.yaml

Length of output: 150979


🏁 Script executed:

#!/bin/bash
set -euo pipefail

base_oid="$(gh pr view 3583 --repo devantler-tech/platform --json baseRefOid --jq .baseRefOid)"
head_oid="$(git rev-parse HEAD)"

printf '%s\n' '== k8s path filter =='
sed -n '150,460p' .github/workflows/ci.yaml

printf '%s\n' '== Changed DR-signing tests =='
git diff --unified=120 "${base_oid}...${head_oid}" -- scripts/validate-dr-signing/main_test.go
ast-grep outline scripts/validate-dr-signing/main_test.go --items all

printf '%s\n' '== Production deploy workflow references =='
rg -n -C 12 'deploy-prod|deploy.*[Pp]rod|workflow_dispatch|merge_group' .github/workflows

printf '%s\n' '== Compatibility-validator uses and workflow graph tests =='
rg -n -C 8 'validate-talos-kubernetes-compatibility|TestMergeGroupDeployRequiresCompatibilityValidation|TestManualDeployRequiresCompatibilityValidation' \
  .github scripts --glob '*.yaml' --glob '*.go' --glob '*.md'

printf '%s\n' '== Workflow YAML parser behavior in repository tests =='
rg -n -C 6 'yaml\.Unmarshal|workflowJob|jobNeeds|repositoryWorkflow' scripts --glob '*_test.go'

printf '%s\n' '== Check production configuration selection by k8s filter =='
python3 - <<'PY'
from pathlib import Path
import re

text = Path(".github/workflows/ci.yaml").read_text()
match = re.search(r'(?ms)^            k8s:\n(?P<body>.*?)(?=^            [A-Za-z_][A-Za-z0-9_]*:\n)', text)
if not match:
    raise SystemExit("could not locate k8s filter")
patterns = re.findall(r"^\s+- '([^']+)'", match.group("body"))
print("\n".join(patterns))
print("ksail.prod.yaml explicitly selected:", "ksail.prod.yaml" in patterns)
PY

Length of output: 180928


@devantler Full review completed. I found two findings.

  1. P1 — Validate the main revision before heal-prod-on-failure deploys it.
    .github/workflows/ci.yaml Line 1894 checks out the current main, but Line 1942 deploys it without running validate-talos or the compatibility validator against that checkout. validate-talos validated the speculative merge-group revision, not the potentially newer main revision. A direct push can place incompatible pins on main. If a later merge-group deployment fails, the heal job can then deploy those incompatible pins to production. Add a compatibility-validation step after the main checkout and before the deploy action. Extend the workflow-graph regression test to cover this production route.

  2. P2 — Correct the stale merge-group statement in validate-main.yaml.
    .github/workflows/validate-main.yaml Lines 141-143 state that ci.yaml runs the compatibility predicate on pull_request only and “not even on merge_group.” .github/workflows/ci.yaml Line 1636 now runs validate-talos on merge_group. Update the comment so it states that direct pushes trigger neither CI event.

The normal merge-queue route and the manual CD route now have the required compatibility gate.

You are interacting with an AI system.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Resolved both current-head findings from CodeRabbit's review output in 8e6d75fe2e0f8c3159244d3e0b5309c697a9bd4d.

  • Heal route: after checking out the current main, heal-prod-on-failure now sets up Go and runs the Kubernetes/Talos compatibility validator before invoking the production deploy composite. The workflow-graph regression test pins checkout → validation → deploy ordering.
  • Push-route documentation: validate-main.yaml now correctly states that CI validates both pull_request and merge_group, while a direct push to main fires neither event.

RED→GREEN: the new heal-route test first failed because the validator was absent, then passed after the gate was added. Validation at this head: full go test ./... -count=1, go vet ./..., actionlint on all three affected workflows, gofmt -l (empty), and git diff --check pass.

The PR remains draft while exact-head CI and a fresh substantive review run; the previous review is stale after this push.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Both P1s are closed by 0fecb054/8e6d75fe — verified, not assumed. I built the same fix in parallel and lost the race; abandoning mine and reporting the one finding the comparison surfaced. What I checked on this head: go test ./scripts/validate-dr-signing and ./scripts/validate-talos-kubernetes-compatibility both pass, so the byte-exact ablation literals updated in main_test.go still bite; actionlint is clean; the merge-group route has no skip mismatch (validate-talos now runs unconditionally on merge_group, so it can never skip while deploy-prod runs); and cd.yaml's new job is unconditional, so it genuinely gates the manual route.

One finding — the validate-talos comment now states the opposite of what the job does.

The condition became:

if: (github.event_name == 'pull_request' && needs.changes.outputs.talos == 'true') || github.event_name == 'merge_group'

while the comment above it still reads:

…runs talosctl validate on the result — on the PR event, so a broken patch (or an env expansion that would render empty, like #2462's ${WG_SERVER_PRIVATE_KEY}) fails the PR instead of surfacing in the merge group's deploy and evicting the queued PR.

That rationale is now false in both halves: the render+validate does run in the merge group, and because deploy-prod requires it, a broken patch there now does evict the queued PR. This is the deliberate and correct trade-off for closing P1-1 — the merge-group revision is what deploys, so it has to be checked — but the comment is the thing a future maintainer reads when deciding whether to narrow the condition back to pull_request, and as written it argues for exactly that.

Suggested replacement for the rationale clause: state that it runs on the PR event for fast feedback and on the merge-group revision because that is the revision production deploys, so a patch that only breaks under the speculative merge is caught before deploy-prod rather than after.

Not a blocker on the fix itself — AGENTS.md's "describe the as-is" rule is what makes it worth a one-line correction before merge.

Deliberately not raised as findings, having checked them: the heal job's new validation step sits before its re-deploy, but that job already validated main the same way (RGD templates, isolated chart namespaces), so this follows the established pattern rather than introducing a new failure mode; and the talos path filter already covers ksail.prod.yaml, go.mod/go.sum and the validator itself, so the PR-event coverage has no gap.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Resolved the remaining as-is documentation finding from this current-head review in 9adb4efffc847f25adf3fd9cc3045ba5f0dbb39c. The validate-talos rationale now records both actual routes: pull requests provide fast feedback, and the merge-group revision is validated because that is what production deploys.

actionlint .github/workflows/ci.yaml, the compatibility and publication-contract test packages, and git diff --check pass. The PR remains draft while this final head runs CI and review.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@devantler I will perform a full review of pull request #3583.

✅ Action performed

Full review finished.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Agentic Engineer

Genuine-readiness evaluation for 9adb4efffc847f25adf3fd9cc3045ba5f0dbb39c.

  • Programmatic: exact-head CI run 33909914508 is terminal green (21 passed, 0 failed), including compatibility, production authorization, manifest, CodeQL, zizmor, dependency review, and Kubescape checks. Local full go test ./... -count=1, go vet ./..., workflow actionlint, formatting, and diff checks pass.
  • Review: CodeRabbit's current-head review covers base 902bdfb41a1031f4e84570c36f8011725f1d8de7 through this exact head and reports no actionable comments with minimal merge risk. The complete review-thread connection has no unresolved threads.
  • User evaluation: exercised the shipped workflow graphs through the repository tests. The new heal-route test rejected the pre-fix graph because current main was not validated, then accepted checkout → compatibility validation → production deploy after the repair; the merge-group and manual-CD route tests also pass. actionlint parsed the exact shipped workflows. Dispatching the recovery or manual workflow would mutate production, so these contract tests plus the exact-head Actions run are the cheapest safe surface that observes the intended routing before the merge queue performs the real deployment.

The head is clean against the live base and the Conventional Commit title remains fix(ci): reject incompatible Kubernetes and Talos pins. This evidence supports self-promotion; it is not a merge or deployment claim.

@devantler
devantler marked this pull request as ready for review September 4, 2026 19:34
@devantler
devantler added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 487027c Sep 4, 2026
27 checks passed
@devantler
devantler deleted the codex/platform-kubernetes-talos-compatibility-3536 branch September 4, 2026 19:46
@github-project-automation github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

A Kubernetes pin above the Talos ceiling passes every PR check and is caught only by a failed prod deploy

1 participant