feat: Add RKE2 as alternative Kubernetes distribution for service cluster WIP#4566
Open
j0hnL wants to merge 13 commits into
Open
feat: Add RKE2 as alternative Kubernetes distribution for service cluster WIP#4566j0hnL wants to merge 13 commits into
j0hnL wants to merge 13 commits into
Conversation
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
3 times, most recently
from
f513dc2 to
11db64e
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
2 times, most recently
from
0759230 to
b4bff18
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
b4bff18 to
ea1db21
Compare
sujit-jadhav
approved these changes
May 27, 2026
sujit-jadhav
requested changes
May 29, 2026
sujit-jadhav
left a comment
sujit-jadhav
left a comment
Collaborator
There was a problem hiding this comment.
input/omnia_config.yml
Can we segrigate the blocks as per the k8s destribution?
service_k8s_cluster:
- cluster_name: service_cluster
deployment: true
etcd_on_local_disk: false
k8s_distro: "kubeadm"
k8s_cni: "calico"
pod_external_ip_range: "172.16.107.170-172.16.107.200"
k8s_service_addresses: "10.233.0.0/18"
k8s_pod_network_cidr: "10.233.64.0/18"
nfs_storage_name: "nfs_k8s"
k8s_crio_storage_size: "20G"
csi_powerscale_driver_secret_file_path: ""
csi_powerscale_driver_values_file_path: ""
We can add new block:
service_rke2_k8s_cluster
and have specific settings for it.
I will ask team to change the existing service_k8s_cluster to service_upstream_k8s_cluster
Same way we can add service_charmed_k8s_cluster for canonical.
service_upstream_k8s_cluster
service_rke2_k8s_cluster
service_charmed_k8s_cluster
| "cluster": [ | ||
| { "package": "docker.io/library/busybox", "type": "image", "tag": "1.36" }, | ||
| { "package": "firewalld", "type": "rpm", "repo_name": "x86_64_baseos" }, | ||
| { "package": "python3-firewall", "type": "rpm", "repo_name": "x86_64_baseos" }, |
Collaborator
There was a problem hiding this comment.
software_config.json changes required to add service_rke2 and need to add input validation for the samea
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
ea1db21 to
7cb6f0c
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
7cb6f0c to
f2ba1dc
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
f606d62 to
174c1cb
Compare
Collaborator
Author
Compute RKE2 Cluster SupportThis update adds support for deploying a separate RKE2-based compute Kubernetes cluster that operates independently alongside the existing service cluster (kubeadm or RKE2) and Slurm clusters. Key Design Decisions
Files Changed (18 files, +1626 / -16)Configuration:
Package Definitions (new):
Validation:
Pipeline:
Cloud-Init Templates (new):
NFS Provisioning (new):
UsageTo enable the compute RKE2 cluster:
|
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
1f9885e to
3eaf74e
Compare
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
271ed98 to
31420a4
Compare
…ster Add support for deploying RKE2 (Rancher Kubernetes Engine 2) as an alternative to vanilla Kubernetes (kubeadm) for the service cluster. Users can choose the distribution via the new k8s_distro field in omnia_config.yml. Configuration changes: - Add k8s_distro field to service_k8s_cluster in omnia_config.yml (default: kubeadm) - Update omnia_config.json schema to validate k8s_distro enum (kubeadm/rke2) - Expand k8s_cni enum to include canal, cilium (RKE2-supported CNIs) Pipeline integration: - discovery.yml: Enable service_k8s tag when service_rke2 is in software_config.json - include_software_config.yml: Detect service_rke2 and set service_rke2_support fact - validate_software_config_json.yml: Handle service_rke2 arch and version detection - common_validation.py: Accept service_rke2 for cluster validation - image_package_collector.py: Dynamically select service_rke2.json for image builds - k8s_config/main.yml: Branch NFS setup based on k8s_distro - configure_cloud_init_group.yml: Select RKE2-specific templates when k8s_distro=rke2 New files: - 3 RKE2 cloud-init templates (first server, additional server, agent) - create_rke2_config_nfs.yml for RKE2-specific NFS directory setup - service_rke2.json package definitions for RHEL 10.0 x86_64 Key differences from kubeadm path: - RKE2 uses built-in containerd (no CRI-O) - RKE2 manages CNI lifecycle (calico/canal/cilium/flannel) - Token-based cluster join instead of kubeadm certificates - kube-vip deployed as RKE2 static pod manifest - RKE2 registries.yaml for Pulp mirror integration - Port 9345 for RKE2 supervisor API Existing kubeadm deployment is completely unaffected when k8s_distro=kubeadm (default). Signed-off-by: John Lockman <j.lockman@dell.com>
- Read NFS share details from storage_config.yml 'mounts' (nfs_client_params no longer exists); use k8s_nfs_server_path in cloud-init fstab and nfs-client-provisioner values - Add rke2 (stable/1.35) and rke2-common RPM repos to local_repo_config.yml; source rke2-selinux from rke2-common - Set cluster-cidr/service-cidr on joining servers to avoid critical configuration value mismatch - Stage official RKE2 airgap image tarballs (core + CNI, v1.35.1+rke2r1) on the NFS share and import via /var/lib/rancher/rke2/agent/images so system images work offline - Deploy kube-vip as a kubelet static pod in agent/pod-manifests (not the server/manifests AddOn dir), mount kubeconfig at /etc/kubernetes/admin.conf, and place it after rke2.yaml exists to avoid hostPath dir race Signed-off-by: John Lockman <j.lockman@dell.com>
- Fix validation guard: use 'service_rke2' in tag_names (not 'service_k8s') when populating service_rke2_k8s_cluster for validation - Fix airgap image download: url used .tar.gz extension but files are .tar.zst - Add pod_external_ip_range and k8s_pod_network_cidr to required fields in omnia_config.json schema - Fix set -e placement: move to top of inline shell block in additional control plane cloud-init - Fix heredoc indentation: config.yaml content was over-indented by 2 spaces producing invalid YAML - Fix omnia_config.yml example: deployment should be true, not false - Move RKE2 repo comments inline with the two new repo entries in local_repo_config.yml - Add arch validation: error if service_rke2 is configured with non-x86_64 arch - Remove redundant inline comments across cloud-init templates and image_package_collector.py - Remove unused HELM_VERSION variable from install-helm.sh in both CP templates - Fix typo: 'nfs sever' → 'nfs server' in create_rke2_config_nfs.yml task name Signed-off-by: John Lockman <jlockman3@gmail.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add compute_rke2 as an independent Kubernetes cluster type that can coexist with service_k8s (kubeadm), service_rke2, and slurm clusters. Configuration: - software_config.json: add compute_rke2 subgroups (compute_kube_*) - omnia_config.yml: add compute_rke2_k8s_cluster block (commented) - omnia_config.json: add compute_rke2_k8s_cluster schema - config.py: add compute_kube_* to FUNCTIONAL_GROUP_LAYER_MAP Packages: - compute_rke2.json: RKE2 core packages without service-layer images Validation: - common_validation.py: x86_64 arch check, subgroup requirement, NFS matching, cluster_set deployment validation for compute_rke2 - validate_software_config_json.yml: detect compute_rke2_support - include_software_config.yml: set compute_rke2_support fact Pipeline: - provision.yml: add compute_rke2 to omnia_run_tags - image_package_collector.py: add compute_kube_* functional groups Cloud-init: - 3 new templates: control_plane_first, control_plane, node - configure_cloud_init_group.yml: route compute_kube_* to rke2/ NFS: - create_compute_rke2_config_nfs.yml: compute cluster NFS setup - k8s_config/tasks/main.yml: include compute RKE2 NFS task Signed-off-by: John Lockman <jlockman3@gmail.com>
Change x86_64_baseos to baseos and x86_64_appstream to appstream to match the naming convention used by service_k8s_v1.35.1.json and the RHEL subscription repo configuration. Signed-off-by: John Lockman <jlockman3@gmail.com>
rpm.rancher.io returns HTTP 403 to Python urllib default User-Agent, causing the repo URL validation to fail even though the repo is accessible. Add 'rancher' to the skip list alongside k8s, cri-o, and other CDNs that block automated clients. Signed-off-by: John Lockman <jlockman3@gmail.com>
Pulp uses self-signed SSL certificates. The generated repo file in /etc/yum.repos.d/pulp.repo must include sslverify=0 for each section, otherwise dnf makecache fails with SSL verification errors when Ansible regenerates the DNF cache. Signed-off-by: John Lockman <jlockman3@gmail.com>
- download_common.py: use verify=False for Pulp self-signed cert downloads instead of verify=PULP_SSL_CA_CERT which fails with self-signed certificates - compute_rke2.json: fix package names for RHEL 10 - vim -> vim-enhanced (correct RPM name) - python3.12 -> python3 (RHEL 10 default Python package) Signed-off-by: John Lockman <jlockman3@gmail.com>
The HA cluster name extraction loop only processed service_k8s_cluster_ha, causing the cluster_name lookup for compute_rke2_k8s_cluster_ha to fail (comparing string against list of dicts instead of list of names). Signed-off-by: John Lockman <jlockman3@gmail.com>
Pulp generates self-signed SSL certificates for its HTTPS content server. Multiple Ansible get_url tasks and image-build config templates download manifests, tarballs, and packages from Pulp without setting validate_certs: false, causing SSL verification failures. Affected areas: - provision/k8s_config: 15 get_url tasks across create_k8s_config_nfs, create_rke2_config_nfs, create_compute_rke2_config_nfs, and get_powerscale_dependencies - build_image_x86_64: 2 config templates (rhel-base-config.yaml.j2, rhel-compute-config.yaml.j2) that generate repo configs used by dnf inside the image-build container This is safe because all affected downloads are from the local Pulp mirror (192.168.x.x:2225), not external sources. Signed-off-by: John Lockman <jlockman3@gmail.com>
The high_availability_config.json schema only allows service_k8s_cluster_ha as a property. Add compute_rke2_k8s_cluster_ha with the same structure (cluster_name, enable_k8s_ha, virtual_ip_address) so the compute RKE2 HA configuration passes schema validation. Signed-off-by: John Lockman <jlockman3@gmail.com>
Pulp downloads and stores RKE2 airgap image tarballs as .tar.gz regardless of the original .tar.zst extension in the source URL. Update the NFS config download tasks to use .tar.gz when fetching from the local Pulp mirror. Signed-off-by: John Lockman <jlockman3@gmail.com>
j0hnL
force-pushed
the
feature/rke2-service-cluster
branch
from
31420a4 to
c52234b
Compare
ansible-lint ignore-errors rule requires failed_when instead of ignore_errors for explicit error condition specification. Signed-off-by: John Lockman <jlockman3@gmail.com>
Collaborator
Author
Updated: RKE2 Service Cluster + Compute RKE2 Cluster + Bug FixesThis PR now contains three areas of work: 1. RKE2 Service Cluster (original scope)Deploy RKE2 as an alternative to kubeadm for the service Kubernetes cluster. 2. Compute RKE2 Cluster (new)Deploy a separate, independent RKE2-based compute Kubernetes cluster alongside the service cluster (kubeadm or RKE2) and Slurm. 3. Bug Fixes Found During TestingFixes for issues discovered while deploying on a live 17-node lab cluster with RHEL 10.0. How to UseService RKE2 (instead of vanilla kubeadm)In {"name": "service_rke2", "version": "1.35.1", "arch": ["x86_64"]}In Compute RKE2 (independent cluster)In {"name": "compute_rke2", "version": "1.35.1", "arch": ["x86_64"]}In compute_rke2_k8s_cluster:
- cluster_name: compute_cluster
deployment: true
k8s_cni: "calico"
pod_external_ip_range: "192.168.5.150-192.168.5.169" # must not overlap service cluster
k8s_service_addresses: "10.234.0.0/18" # must not overlap service cluster
k8s_pod_network_cidr: "10.234.64.0/18" # must not overlap service cluster
nfs_storage_name: "nfs_compute_k8s"In compute_rke2_k8s_cluster_ha:
- cluster_name: compute_cluster
enable_k8s_ha: true
virtual_ip_address: "192.168.5.2" # unused IP on admin networkIn - name: "nfs_compute_k8s"
source: "192.168.5.100:/mnt/share/omnia_compute_k8s"
mount_point: "/opt/omnia/compute_k8s_mount"
fs_type: "nfs"
mnt_opts: "nosuid,rw,sync,hard,intr"
mount_on_oim: true
functional_group_prefix: ["compute_kube"]In Exactly one node must use Supported Combinations
Files Changed (13 commits, 32 files)Service RKE2 (commits 1-4)
Compute RKE2 (commit 5)
Bug Fixes (commits 6-13)
Pulp SSL Fix (Fixes #4836)The Pulp SSL fixes in this PR address an Omnia-wide issue where Pulp's self-signed HTTPS certificate causes download failures in TestingTested on a 17-node lab cluster (Dell R650, C6420, C6620, R750XA, R740xd) with RHEL 10.0:
|
j0hnL
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add support for deploying RKE2 (Rancher Kubernetes Engine 2) as an alternative to vanilla Kubernetes (kubeadm) for the service cluster. The distribution is selected by listing
service_rke2(instead ofservice_k8s) insoftware_config.jsonand configuring the newservice_rke2_k8s_clusterblock inomnia_config.yml.How to use
input/software_config.json, replaceservice_k8swithservice_rke2in the softwares list and set its"version"(e.g.1.35.1)input/omnia_config.yml, uncomment and configure theservice_rke2_k8s_clusterblock withdeployment: true(only one ofservice_k8s_cluster/service_rke2_k8s_clustermay havedeployment: true)high_availability_config.yml— the kube-vip VIP is required for cluster joinlocal_repo.yml→build_image_x86_64.yml→discovery.yml)Configuration changes
service_rke2_k8s_clusterblock inomnia_config.ymlwith schema validation inomnia_config.json(supported CNIs: calico, canal, cilium, flannel)service_k8sandservice_rke2being present togetherrke2(stable/1.35 channel) andrke2-commonRPM repositories added tolocal_repo_config.ymlPipeline integration
provision.yml: enableservice_k8stag whenservice_rke2is insoftware_config.jsoninclude_software_config.yml/validate_software_config_json.yml: detectservice_rke2, set support facts, handle arch and versioncommon_validation.py: validateservice_rke2cluster configurationimage_package_collector.py: selectservice_rke2.jsonfor image buildsk8s_config/main.yml: branch NFS setup onk8s_distro;create_rke2_config_nfs.ymlreads NFS share details fromstorage_config.ymlmountsconfigure_cloud_init_group.yml: select RKE2-specific cloud-init templatesNew files
create_rke2_config_nfs.ymlfor RKE2-specific NFS setupservice_rke2.jsonpackage definitions for RHEL 10.0 x86_64Key differences from kubeadm path
/var/lib/rancher/rke2/agent/pod-manifests/) with the kubeconfig mounted at/etc/kubernetes/admin.confv1.35.1+rke2r1) are downloaded by local_repo, staged on the NFS share, and auto-imported from/var/lib/rancher/rke2/agent/images/on every noderegistries.yamlmirrors public registries to the Pulp mirror for all other imagesBackward compatibility
Existing kubeadm deployment is completely unaffected when
service_k8sis selected (the default). All branching logic only activates forservice_rke2.