Failing to run pyxis/enroot GPU jobs on a fresh Soperator cluster

[roulbac]

Summary

On a fresh Soperator cluster in driverfull mode (not relying on GPU Operator), all Pyxis/enroot jobs that request a GPU fail at runtime with something such as :

NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.

The NVIDIA libraries are physically present in the shared jail (/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.<version> exists), but the jail's /etc/ld.so.cache contains no NVIDIA entries, so nvidia-container-cli (which reads the cache to decide what to inject) concludes every lib is missing and Pyxis injects nothing.

My cluster has identical drivers on every worker and still hits this.

Environment

• Nebius Soperator 3.0.x, driverfull mode (GPU Operator disabled)
• Two identical H200 workers, single node group
• Ubuntu 24.04, NVIDIA driver 580.126.09, CUDA 13.0
• Shared jail on NFS-backed PVC (Soperator default)
• Kubernetes 1.32

Reproduction

On a fresh Soperator cluster bootstrapped from nebius-solutions-library, run:

srun -N1 -n1 --gpus=1 --cpu-bind=none -w worker-0 \
  --container-image="nvidia/cuda:12.4.1-base-ubuntu22.04" \
  nvidia-smi

Result:

NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure
that the NVIDIA Display Driver is properly installed and present in your system.

On the host nvidia-smi works normally, so the driver there is fine.

Diagnosis

Inside the jail on any GPU worker:

# Files exist
ls -l /usr/lib/x86_64-linux-gnu/libnvidia-ml*
#  ... libnvidia-ml.so.580.126.09
#  ... libnvidia-ml.so.1 -> libnvidia-ml.so.580.126.09

# But the linker cache doesn't know about them
ldconfig -p | grep -E 'libnvidia-ml|libcuda\.so'
# (no output)

# Consequently nvidia-container-cli reports every NVIDIA lib as missing
nvidia-container-cli -k -d /dev/tty list 2>&1 | grep -E 'libnvidia-ml|libcuda\.so' | wc -l
# ...
# ... W0422 01:09:20.152295 9659 nvc_info.c:416] missing compat32 library libnvidia-ml.so
# ... W0422 01:09:20.152298 9659 nvc_info.c:416] missing compat32 library libnvidia-cfg.so
# ... W0422 01:09:20.152300 9659 nvc_info.c:416] missing compat32 library libnvidia-nscq.so
# ... W0422 01:09:20.152304 9659 nvc_info.c:416] missing compat32 library libcuda.so
# ... W0422 01:09:20.152306 9659 nvc_info.c:416] missing compat32 library libcudadebugger.so
# ... W0422 01:09:20.152308 9659 nvc_info.c:416] missing compat32 library libnvidia-opencl.so
# ... W0422 01:09:20.152311 9659 nvc_info.c:416] missing compat32 library libnvidia-gpucomp.so
# ...

Running ldconfig manually inside the jail fully resolves the issue for the whole cluster (the jail is shared, so one rebuild of /etc/ld.so.cache fixes every worker):

ssh worker-0 ldconfig
# From the login node:
srun -N1 -n1 --gpus=1 --container-image="nvidia/cuda:12.4.1-base-ubuntu22.04" nvidia-smi
# Works, lists the H200s normally.

Suspected source

I suspect something wrong is happening in images/common/scripts/complement_jail.sh.

See below:

# For worker node only
if [ -n "$worker" ]; then
    echo "Update linker cache inside the jail"
    ...
    ldconfig_rc=0
    flock --nonblock etc/complement_jail_ldconfig.lock -c \
        "chroot \"${jaildir}\" /usr/sbin/ldconfig" || ldconfig_rc=$?
    if [ "$ldconfig_rc" -eq 0 ]; then
        echo "ldconfig completed successfully (got flock)"
    elif [ "$ldconfig_rc" -eq 1 ]; then
        echo "ldconfig SKIPPED (flock busy)"
    else
        echo "ldconfig FAILED with exit code ${ldconfig_rc}"
    fi
    ...
fi

Workaround for anyone hitting this

Shell into any worker and run ldconfig inside the jail. The jail is shared, so one run unblocks the whole cluster until the next pod restart.

[Uburro]

HI @roulbac . Yes, this is a bug. We’re thinking about how to resolve it. Thanks.