where is the SPANK plugin logs & question about it's operation

[thien7902]

I am facing some issues about mountpoints like /mnt/jail/dev and /mnt/jail/sys, sometimes they are missing at the container level after job execution failure.
I want to check the logs of the chroot plugin, but I don't know where Soperator placed them.

[thien7902]

Has anyone faced the issue with mounting? Please help me :(

[thien7902]

I think we should not depend too much on jail's folders like /mnt/jail/sys or /mnt/jail/dev, we can just mount them directly from the root of the container to the new namespace of the job, like this in choot.c

in old chroot.c plugin, we reuse some mountpoint from jail, like /mnt/jail/sys for new job namespaces. If that mount is broken at the slurmd container namespace, all running jobs' namespaces will miss /sys, leading to cgroup inconsistency. That's serious.

int setup_mounts() {
    slurm_debug("chroot: setup_mounts: Bind-mount virtual filesystems");
    if (mount("sysfs", "sys", "sysfs", 0, NULL) != 0) {
        fprintf(stderr, "mount -t sysfs sysfs sys/: %s\n", strerror(errno));
        return 70;
    }
    if (mount("/mnt/host/dev", "dev", NULL, MS_BIND | MS_REC, NULL) != 0) {
        fprintf(stderr, "mount --rbind /mnt/host/dev dev/: %s\n", strerror(errno));
        return 80;
    }
    if (mount("/mnt/host/run", "run", NULL, MS_BIND | MS_REC, NULL) != 0) {
        fprintf(stderr, "mount --rbind /mnt/host/run run/: %s\n", strerror(errno));
        return 90;
    }

    slurm_debug("chroot: setup_mounts: Bind-mount cgroup filesystem");
    if (mount("/mnt/host/sys/fs/cgroup", "sys/fs/cgroup", NULL, MS_BIND | MS_REC, NULL) != 0) {
        fprintf(stderr, "mount --rbind /mnt/host/sys/fs/cgroup sys/fs/cgroup: %s\n", strerror(errno));
        return 100;
    }

    slurm_debug("chroot: setup_mounts: Remount /tmp");
    if (mount("tmpfs", "tmp", "tmpfs", 0, NULL) != 0) {
        fprintf(stderr, "mount -t tmpfs tmpfs tmp/: %s\n", strerror(errno));
        return 110;
    }

    slurm_debug("chroot: setup_mounts: Bind-mount /var/log because it should be node-local");
    if (mount("/mnt/host/var/log", "var/log", NULL, MS_BIND, NULL) != 0) {
        fprintf(stderr, "mount --bind /mnt/host/var/log var/log: %s\n", strerror(errno));
        return 120;
    }

    slurm_debug("chroot: setup_mounts: Bind-mount DNS configuration");
    if (mount("/mnt/host/etc/resolv.conf", "etc/resolv.conf", NULL, MS_BIND, NULL) != 0) {
        fprintf(stderr, "mount --bind /mnt/host/etc/resolv.conf etc/resolv.conf: %s\n", strerror(errno));
        return 130;
    }

    slurm_debug("chroot: setup_mounts: Bind-mount /etc/hosts");
    if (mount("/mnt/host/etc/hosts", "etc/hosts", NULL, MS_BIND, NULL) != 0) {
        fprintf(stderr, "mount --bind /mnt/host/etc/hosts etc/hosts: %s\n", strerror(errno));
        return 140;
    }

    return 0;
}

[Uburro]

I want to check the logs of the chroot plugin, but I don't know where Soperator placed them.

In case of an error, it goes to stdout. Without errors, you can't see the logs.

[Uburro]

How can we reproduce an error with a broken mount?

[thien7902]

@Uburro As for now, I found that my Jail PVC was full/almost full at the time the mountpoint broke. I freed that volume space and trace more.
btw i can not explain the relationship between diskfull & mountpoint dissappear :(

[thien7902]

hmm @Uburro seems that the disk full is not the root cause. The mount point disappearance still happens after jail-pvc is cleaned up. I also checked that, when mountpoints like /dev or /sys disappear on one container (like a pod), it also disappears on other pods (like sshd). I ran both the login pod and the worker pod on the same host. and use NFS as storage.

[thien7902]

More info, the error happened when I killed jobs by scancel, then job can not be killed within the specified time for sig term.