Skip to content

Improve CI stability: make kubeconform schema validation resilient to transient download failures - #705

Merged
lhotari merged 1 commit into
apache:masterfrom
lhotari:lh-kubeconform-schema-cache
Jul 6, 2026
Merged

Improve CI stability: make kubeconform schema validation resilient to transient download failures#705
lhotari merged 1 commit into
apache:masterfrom
lhotari:lh-kubeconform-schema-cache

Conversation

@lhotari

@lhotari lhotari commented Jul 6, 2026

Copy link
Copy Markdown
Member

Motivation

The chart-testing lint job on master is failing intermittently because kubeconform cannot download JSON schemas from raw.githubusercontent.com:

stdin - Role release-name-pulsar-broker-role failed validation: failed downloading schema at https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/v1.27.0-standalone-strict/role-rbac-v1.json: ... giving up after 3 attempt(s)

Two consecutive attempts of https://github.com/apache/pulsar-helm-chart/actions/runs/28790178682 failed this way (different schema files each time), while all actual chart validation passed (Invalid: 0). The root cause is likely throttling of GitHub Actions runner requests to raw.githubusercontent.com, aggravated by the fact that each kubeconform invocation re-downloads the same schemas: the lint job validates ~21 values files × 10 k8s versions, so the same schema files get downloaded hundreds of times per run.

Modifications

  • Cache downloaded schemas with kubeconform's -cache option (new hack::kubeconform_with_retries helper in hack/common.sh), so each schema is downloaded at most once per run.
  • Persist the schema cache across CI runs with actions/cache (restore + save split so the cache is saved even when validation fails, avoiding re-downloads on the next attempt).
  • Retry kubeconform up to 5 times when the failure is a schema download error. Already-cached schemas are not re-downloaded on retry, so each retry only fetches the missing ones. Genuine validation failures fail immediately without retrying.
  • Render helm template output to a file so the kubeconform validation can be retried without re-running helm.

Verification

Tested locally:

  • warm cache run performs zero downloads (0.3s vs 2.1s cold per validation)
  • unreachable schema host: 5 attempts with 5s backoff, then fails with exit code 1
  • invalid resource (-strict violation): fails immediately with no retries

… transient download failures

The chart-testing lint job intermittently fails when kubeconform cannot
download schemas from raw.githubusercontent.com (throttling of GitHub
Actions runners). Each kubeconform invocation re-downloaded the same
schemas, ~21 values files x 10 k8s versions per run.

- cache downloaded schemas with kubeconform's -cache option so each
  schema is downloaded at most once per run
- persist the schema cache across runs with actions/cache
- retry kubeconform up to 5 times when the failure is a schema download
  error; cached schemas are not re-downloaded on retry, and genuine
  validation failures fail immediately without retrying
- render helm template output to a file so validation can be retried
  without re-running helm
@lhotari
lhotari merged commit 29f357c into apache:master Jul 6, 2026
83 of 86 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant