Release soperator [1.22] Can't install slurm-cluster via helm chart.

[JeffYFHuang]

Output as following.

I0915 08:48:50.039135 172349 warnings.go:110] "Warning: unknown field "spec.pause""
I0915 08:48:50.039158 172349 warnings.go:110] "Warning: unknown field "spec.populateJail.priorityClass""
I0915 08:48:50.039167 172349 warnings.go:110] "Warning: unknown field "spec.sConfigController.node.priorityClass""
I0915 08:48:50.039175 172349 warnings.go:110] "Warning: unknown field "spec.sConfigController.reconfigurePollInterval""
I0915 08:48:50.039179 172349 warnings.go:110] "Warning: unknown field "spec.sConfigController.reconfigureWaitTimeout""
I0915 08:48:50.039187 172349 warnings.go:110] "Warning: unknown field "spec.sConfigController.runAsGid""
I0915 08:48:50.039190 172349 warnings.go:110] "Warning: unknown field "spec.sConfigController.runAsUid""
I0915 08:48:50.039199 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.accounting.priorityClass""
I0915 08:48:50.039203 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.controller.priorityClass""
I0915 08:48:50.039206 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.exporter.collectionInterval""
I0915 08:48:50.039214 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.exporter.priorityClass""
I0915 08:48:50.039218 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.login.priorityClass""
I0915 08:48:50.039226 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.rest.priorityClass""
I0915 08:48:50.039230 172349 warnings.go:110] "Warning: unknown field "spec.slurmNodes.worker.maxUnavailable""
I0915 08:48:50.039237 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[1].createPVC""
I0915 08:48:50.039240 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[1].size""
I0915 08:48:50.039248 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[1].storageClassName""
I0915 08:48:50.039255 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[2].createPVC""
I0915 08:48:50.039262 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[2].size""
I0915 08:48:50.039269 172349 warnings.go:110] "Warning: unknown field "spec.volumeSources[2].storageClassName""
Error: INSTALLATION FAILED: 1 error occurred:
* SlurmCluster.slurm.nebius.ai "slurm1" is invalid: [spec.periodicChecks: Required value, : Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]

helm install command as below.

helm install slurm-cluster helm/slurm-cluster/ --namespace slurm-cluster --createnamespace

[Uburro]

helm install slurm-cluster oci://cr.eu-north1.nebius.cloud/soperator/helm-soperator --namespace slurm-cluster --createnamespace

[JeffYFHuang]

For below logs. What steps that I haven't do? Thanks so much.

kubectl describe pod slurm1-populate-jail-wv855 -n slurm-cluster

Name: slurm1-populate-jail-wv855
Namespace: slurm-cluster
Priority: 0
Service Account: default
Node:
Labels: app.kubernetes.io/component=populate-jail
app.kubernetes.io/instance=slurm1
app.kubernetes.io/managed-by=slurm-operator
app.kubernetes.io/name=slurmcluster
app.kubernetes.io/part-of=slurm-operator
batch.kubernetes.io/controller-uid=6af9d1a1-8072-4b2f-85f4-6a6c2830e7ca
batch.kubernetes.io/job-name=slurm1-populate-jail
controller-uid=6af9d1a1-8072-4b2f-85f4-6a6c2830e7ca
job-name=slurm1-populate-jail
Annotations: container.apparmor.security.beta.kubernetes.io/populate-jail: unconfined
Status: Pending
IP:
IPs:
Controlled By: Job/slurm1-populate-jail
Containers:
populate-jail:
Image: cr.eu-north1.nebius.cloud/soperator/populate_jail:1.21.13-jammy-slurm24.11.5
Port:
Host Port:
Environment:
OVERWRITE: 0
SLURM_CLUSTER_TYPE: gpu
Mounts:
/mnt/jail from jail (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzwzw (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
jail:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: jail-pvc
ReadOnly: false
kube-api-access-dzwzw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
nvidia.com/gpu:NoSchedule op=Exists
Events:
Type Reason Age From Message

---

Warning FailedScheduling 3m16s (x20 over 98m) default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

[JeffYFHuang]

kubectl label node soperator nebius.com/node-group-id=node-group-id-here

got events after exec previously command.

Events:
Type Reason Age From Message

---

Warning FailedScheduling 8m50s default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 4m3s default-scheduler 0/1 nodes are available: 1 node(s) had volume node affinity conflict. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

kubectl get pv jail-pv -o yaml -n slurm-cluster

apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
meta.helm.sh/release-name: slurm-cluster-storage
meta.helm.sh/release-namespace: slurm-cluster
creationTimestamp: "2025-09-16T00:34:59Z"
finalizers:

• kubernetes.io/pv-protection
  labels:
  app.kubernetes.io/managed-by: Helm
  name: jail-pv
  resourceVersion: "1621786"
  uid: 1375fe7b-33e3-4d00-a96e-640895619d88
  spec:
  accessModes:
• ReadWriteMany
  capacity:
  storage: 100Gi
  claimRef:
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: jail-pvc
  namespace: slurm-cluster
  resourceVersion: "1590814"
  uid: 357a8b0e-7384-48c8-83c3-1b64e467fce5
  local:
  path: /mnt/jail
  mountOptions:
• rw
• relatime
• exec
• dev
  nodeAffinity:
  required:
  nodeSelectorTerms:
  • {}
    persistentVolumeReclaimPolicy: Retain
    storageClassName: slurm-local-pv
    volumeMode: Filesystem
    status:
    lastPhaseTransitionTime: "2025-09-16T00:34:59Z"
    phase: Bound

[JeffYFHuang]

As the screenshot, I have installed the slurm-cluster on kubernetes. But next, could you help to guide me how to create a slurm cluster and run a slurm job over it. Or point out something for me. Thanks.

Any instruction for checking slurm.nebius.ai/SlurmCluster resource be available. Thanks.

[JeffYFHuang]

kubectl describe SlurmCluster slurm1 -n slurm-cluster

Name: slurm1
Namespace: slurm-cluster
Labels: app.kubernetes.io/instance=slurm-cluster
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=slurm1
app.kubernetes.io/version=1.21.13
helm.sh/chart=helm-slurm-cluster-1.21.13
Annotations: meta.helm.sh/release-name: slurm-cluster
meta.helm.sh/release-namespace: slurm-cluster
API Version: slurm.nebius.ai/v1
Kind: SlurmCluster
Metadata:
Creation Timestamp: 2025-09-18T03:23:11Z
Generation: 1
Resource Version: 2294455
UID: aff3289d-64d8-46c1-ba2e-d08c52d0f5a8
Spec:
Cluster Type: gpu
Cr Version: 1.21.13
Health Check Config:
Health Check Interval: 300
Health Check Node State:
State: NONDRAINED_IDLE
State: ALLOC
State: MIXED
State: CYCLE
Health Check Program: /opt/slurm_scripts/hc_program.sh
k8sNodeFilters:
Affinity:
Node Affinity:
Required During Scheduling Ignored During Execution:
Node Selector Terms:
Match Expressions:
Key: nebius.com/node-group-id
Operator: In
Values:
node-group-id-here
Name: gpu
Tolerations:
Effect: NoSchedule
Key: nvidia.com/gpu
Operator: Exists
Affinity:
Node Affinity:
Required During Scheduling Ignored During Execution:
Node Selector Terms:
Match Expressions:
Key: nebius.com/node-group-id
Operator: In
Values:
node-group-id-here
Name: no-gpu
Maintenance: none
Mpi Config:
Pmix Env: OMPI_MCA_btl_tcp_if_include=eth0
Nccl Settings:
Topology Data:
Topology Type: auto
Partition Configuration:
Config Type: default
Raw Config:
Periodic Checks:
Nccl Benchmark:
Active Deadline Seconds: 1800
App Armor Profile: unconfined
Enabled: false
Failed Jobs History Limit: 24
Failure Actions:
Set Slurm Node Drain State: true
Image: cr.eu-north1.nebius.cloud/soperator/nccl_benchmark:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
k8sNodeFilterName: no-gpu
Nccl Arguments:
Max Bytes: 8Gb
Min Bytes: 512Mb
Step Factor: 2
Threshold More Than: 0
Timeout: 20:00
Use Infiniband: true
Schedule: 0 */3 * * *
Successful Jobs History Limit: 3
Plug Stack Config:
Custom Plugins:
Nccl Debug:
Enabled: true
Log Level: INFO
Output Directory: /opt/soperator-outputs/nccl_logs
Output To File: true
Output To Std Out: false
Required: false
Pyxis:
Container Image Save: /var/cache/enroot-container-images/
Required: true
Populate Jail:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/populate_jail:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
k8sNodeFilterName: gpu
Overwrite: false
S Config Controller:
Container:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/sconfigcontroller:1.21.13
Image Pull Policy: IfNotPresent
Resources:
Cpu: 250m
Ephemeral - Storage: 500Mi
Memory: 256Mi
Jail Slurm Config Path: /mnt/jail/etc/slurm
Node:
k8sNodeFilterName: no-gpu
Size: 1
Secrets:
Slurm Config:
Complete Wait: 5
Def Cpu Per GPU: 4
Def Mem Per Node: 1048576
Epilog: /opt/slurm_scripts/epilog.sh
Max Job Count: 10000
Message Timeout: 60
Min Job Age: 86400
Prolog: /opt/slurm_scripts/prolog.sh
Task Plugin Param:
Topology Param: SwitchAsNodeRank
Topology Plugin: topology/tree
Slurm Nodes:
Accounting:
Custom Init Containers:
Enabled: true
k8sNodeFilterName: no-gpu
Mariadb Operator:
App Armor Profile: unconfined
Enabled: true
Image: docker-registry1.mariadb.com/library/mariadb:11.4.3
Image Pull Policy: IfNotPresent
Metrics:
Enabled: false
Pod Security Context:
Protected Secret: false
Replicas: 1
Replication:
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Security Context:
Storage:
Munge:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/munge:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Slurm Config:
Accounting Storage TRES: CPU,Mem,Node,VMem,Gres/gpu
Job Acct Gather Frequency: 30
Job Acct Gather Type: jobacct_gather/cgroup
Priority Weight Age: 0
Priority Weight Fairshare: 0
Priority Weight QOS: 0
Slurmdbd:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/controller_slurmdbd:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Port: 6819
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Slurmdbd Config:
Archive Events: yes
Archive Jobs: yes
Archive Resvs: yes
Archive Steps: no
Archive Suspend: no
Archive TXN: no
Archive Usage: yes
Debug Level: info
Purge Event After: 1month
Purge Job After: 12month
Purge Resv After: 1month
Purge Step After: 1month
Purge Suspend After: 1month
Purge TXN After: 12month
Purge Usage After: 24month
Tcp Timeout: 2
Controller:
Custom Init Containers:
k8sNodeFilterName: no-gpu
Munge:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/munge:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Size: 1
Slurmctld:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/controller_slurmctld:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Port: 6817
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Volumes:
Jail:
Volume Source Name: jail
Spool:
Volume Source Name: controller-spool
Exporter:
Enabled: true
Exporter Container:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/soperator-exporter:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
k8sNodeFilterName: gpu
Pod Monitor Config:
Interval: 30s
Job Label: slurm-exporter
Scrape Timeout: 20s
Size: 1
Login:
Custom Init Containers:
k8sNodeFilterName: no-gpu
Munge:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/munge:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 500Mi
Size: 1
Ssh Root Public Keys:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKzxkjzPQ4EyZSjan4MLGFSA18idpZicoKW7Hfff username1
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICL8scMKnwu+Y9S6XDACacZ54+qu+YRo2y4Ieddd username2
Sshd:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/login_sshd:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Port: 22
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Sshd Service Annotations:
Sshd Service Load Balancer IP: 10.42.0.1
Sshd Service Node Port: 0
Sshd Service Type: LoadBalancer
Volumes:
Custom Mounts:
Mount Path: /opt/slurm_scripts/
Name: slurm-scripts
Read Only: false
Sub Path:
Volume Source Name: slurm-scripts
Mount Path: /mnt/jail.upper/opt/slurm_scripts/
Name: slurm-scripts-jail
Read Only: false
Sub Path:
Volume Source Name: slurm-scripts
Jail:
Volume Source Name: jail
Jail Sub Mounts:
Rest:
Custom Init Containers:
Enabled: false
k8sNodeFilterName: no-gpu
Max Connections: 10
Rest:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/slurmrestd:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Resources:
Cpu: 500m
Ephemeral - Storage: 500Mi
Memory: 1Gi
Size: 1
Thread Count: 3
Worker:
Cgroup Version: v2
Custom Init Containers:
Enable GDR Copy: false
k8sNodeFilterName: gpu
Munge:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/munge:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
Priority Class: slurm1
Size: 1
Slurmd:
App Armor Profile: unconfined
Image: cr.eu-north1.nebius.cloud/soperator/worker_slurmd:1.21.13-jammy-slurm24.11.5
Image Pull Policy: IfNotPresent
Port: 6818
Resources:
Cpu: 500m
Ephemeral - Storage: 1Gi
Memory: 1Gi
nvidia.com/gpu: 1
Volumes:
Custom Mounts:
Mount Path: /opt/slurm_scripts/
Name: slurm-scripts
Read Only: false
Sub Path:
Volume Source Name: slurm-scripts
Mount Path: /mnt/jail.upper/opt/slurm_scripts/
Name: slurm-scripts-jail
Read Only: false
Sub Path:
Volume Source Name: slurm-scripts
Jail:
Volume Source Name: jail
Shared Memory Size: 64Gi
Spool:
Volume Claim Template Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Storage Class Name: slurm-local-pv
Wait For Controller: true
Use Default App Armor Profile: false
Volume Sources:
Config Map:
Default Mode: 500
Name: slurm-scripts
Name: slurm-scripts
Name: controller-spool
Persistent Volume Claim:
Claim Name: controller-spool-pvc
Read Only: false
Name: jail
Persistent Volume Claim:
Claim Name: jail-pvc
Read Only: false
Status:
Conditions:
Last Transition Time: 2025-09-18T03:23:11Z
Message: Reconciling Slurm common resources
Reason: Reconciling
Status: Unknown
Type: CommonAvailable
Last Transition Time: 2025-09-18T03:23:11Z
Message: Reconciling Slurm Controllers
Reason: Reconciling
Status: Unknown
Type: ControllersAvailable
Last Transition Time: 2025-09-18T03:23:11Z
Message: Reconciling Slurm Workers
Reason: Reconciling
Status: Unknown
Type: WorkersAvailable
Last Transition Time: 2025-09-18T03:23:11Z
Message: Reconciling Slurm Login
Reason: Reconciling
Status: Unknown
Type: LoginAvailable
Last Transition Time: 2025-09-18T03:23:11Z
Message: Reconciling Slurm Accounting
Reason: Reconciling
Status: Unknown
Type: AccountingAvailable
Phase: Reconciling
Events:

Any suggestions for me! Thank you!