Repository navigation
Symbol relocation errors when used with more than 1 layer on squashfs or dwarfs filesystems #432
Description
Activity
The commit 6a0de4a is the culprit behind this issue, reverting to the one previous to it with
git reset --hard 918e19cfixes the symbol issues and krita opens.I'm seeing something very similar. However, reverting the commit suggested above didn't help. I haven't tried going back to the commit suggested.
$ cat test-chroot.sh #!/bin/bash unshare --mount -r /bin/bash <<LXEOF ./test.sh LXEOF exit 0 $ cat test.sh #!/bin/bash set -Eeuo pipefail TRACEX="set -x" #TRACEX= export PS4='+${BASH_SOURCE:-}:${LINENO:-}:${FUNCNAME:-} $ ' ${TRACEX} HOMEDIR=$( readlink -f "$( dirname "${BASH_SOURCE[0]}" )" ) # shellcheck disable=SC2317 # called from a trap cleanup() { set +x cd "${HOMEDIR}/" umount "${BUILDCHROOT}" umount "${SCRATCHDIR}/lower0" umount "${SCRATCHDIR}/lower1" [[ -z ${SCRATCHDIR:-} ]] || rm -fr --one-file-system "${SCRATCHDIR}" } trap cleanup EXIT build_deb() { local dist=${1} local arch=${2} SCRATCHDIR=$( mktemp -d -p "/tmp" scratch.XXXXXXXXXX ) BUILDCHROOT=${SCRATCHDIR}/build.chroot mkdir -p "${BUILDCHROOT}" mkdir "${SCRATCHDIR}/overlay" mkdir "${SCRATCHDIR}/work" mkdir "${SCRATCHDIR}/lower0" mkdir "${SCRATCHDIR}/lower1" squashfuse "/mnt/mirror/ftp/mirror/bootstrap/tarfiles/${dist}.${arch}.minimal.sqfs" "${SCRATCHDIR}/lower0" squashfuse "/mnt/mirror/ftp/mirror/bootstrap/tarfiles/${dist}.${arch}.build-deb.sqfs" "${SCRATCHDIR}/lower1" fuse-overlayfs -o lowerdir="${SCRATCHDIR}/lower1:${SCRATCHDIR}/lower0,upperdir=${SCRATCHDIR}/overlay,workdir=${SCRATCHDIR}/work" "${BUILDCHROOT}" # find ${BUILDCHROOT}/etc/apt -xdev -type d >& /dev/null touch "${BUILDCHROOT}/etc/apt/x" /sbin/chroot "${BUILDCHROOT}" /bin/bash <<CHEOF if [[ ${arch} = arm64 ]]; then export QEMU_CPU=max,pauth=off fi set -Eeuo pipefail ${TRACEX} export PS4='+chroot:${BASH_SOURCE:-}:${LINENO:-}:${FUNCNAME:-}:\${BASH_SOURCE:-}:\$(( ${LINENO} + \${LINENO:-} )):\${FUNCHAME:-} $ ' cd /build cat >x.c <<EOF int main (void) { return 0; } EOF cc -o x x.c CHEOF } build_deb "bookworm" "amd64" # fails #build_deb "bookworm" "arm64" # fails #build_deb "trixie" "amd64" # fails #build_deb "trixie" "arm64" # works echo finished at "$( date )" exit 0 $ ./test-chroot.sh ... +chroot:./test.sh:46:build_deb::61: $ cc -o x x.c /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_audit_symbind_alt@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__nptl_change_stack_perm@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_deallocate_tls@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__rseq_size@GLIBC_2.35' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_argv@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_allocate_tls@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_find_dso_for_object@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__libc_enable_secure@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_fatal_printf@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_exception_create@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__tunable_get_val@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_allocate_tls_init@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_rtld_di_serinfo@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_rtld_global@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__tls_get_addr@GLIBC_2.3' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `__libc_stack_end@GLIBC_2.2.5' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_dl_audit_preinit@GLIBC_PRIVATE' /usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: undefined reference to `_rtld_global_ro@GLIBC_PRIVATE' collect2: error: ld returned 1 exit statusIf I comment out the touch or I uncomment the find then it compiles successfully.
$ fuse-overlayfs -V fuse-overlayfs: version 1.16 FUSE library version 3.14.0 using FUSE kernel interface version 7.31 fusermount3 version: 3.14.0Found it!
e.attr.st_ino and e.ino are not consistent. st_ino typically gets set from tmp_ino which comes from the lower level while e.ino is set via node_to_inode;
The primary problem then occurs when the file is mmapped because this violates the VFS invariant and pages can be returned from the wrong file because it has two identities.
diff --git a/main.c b/main.c index 4df64ec..0175349 100644 --- a/main.c +++ b/main.c @@ -957,8 +957,7 @@ rpl_stat (fuse_req_t req, struct ovl_node *node, int fd, const char *path, struc st->st_uid = find_mapping (st->st_uid, data, true, true); st->st_gid = find_mapping (st->st_gid, data, true, false); - st->st_ino = node->tmp_ino; - st->st_dev = node->tmp_dev; + st->st_ino = node_to_inode(node); if (node->loaded && node->n_links > 0) st->st_nlink = node->n_links; @@ -2554,8 +2553,7 @@ ovl_do_readdir (fuse_req_t req, fuse_ino_t ino, size_t size, /* From the 'stbuf' argument the st_ino field and bits 12-15 of the * st_mode field are used. The other fields are ignored. */ - st->st_ino = node->tmp_ino; - st->st_dev = node->tmp_dev; + st->st_ino = node_to_inode(node); st->st_mode = node->ino->mode; entsize = fuse_add_direntry (req, p, remaining, name, st, offset + 1);There is no need to set st_dev - it gets replaced by the kernel with the correct st_dev for the fuse filesystem.
I've only done some very basic checks using this patch so far but all seems good.
In particular, the problem goes like this:
We have two files on the two different layers that have the same st_ino but different st_dev.
fuse-overlayfs returns st_dev and st_ino, which is unique, but the kernel overwrites st_dev with the st_dev for the mount point.
So now we have two completely unrelated files that have the same st_dev/st_ino identity which breaks mmap.Still needs work but much improved from the quick hack above here:
https://github.com/tjwoodall/fuse-overlayfs/tree/unique-inodesI've tried to roughly follow what the kernel overlayfs does.
#!/bin/bash #set -x mkdir -p lower mkdir -p upper mkdir -p mnt touch lower/lower touch upper/upper /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o lowerdir=lower:upper mnt echo echo "All on same fs - ino passthrough - stable ino" echo -n "lower/lower " stat -c %i lower/lower echo -n "upper/upper " stat -c %i upper/upper echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt rm -f img truncate -s 10M img /sbin/mke2fs -t ext2 img >& /dev/null mkdir -p ext2 fuse2fs -o fakeroot,uid=$(id -u),gid=$(id -u) img ext2 mkdir -p ext2/work mkdir -p ext2/upper mkdir -p ext2/mnt # mount -t overlay overlay -o lowerdir=fuseoverlayfs-lower,upperdir=fuseoverlayfs-upper,workdir=fuseoverlayfs-work fuseoverlayfs-mount echo echo "Upper on different fs - non-stable ino across mounts" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o lowerdir=lower:upper,upperdir=ext2/upper,workdir=ext2/work mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt echo "==" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o lowerdir=lower:upper,upperdir=ext2/upper,workdir=ext2/work mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt echo echo "Upper on different fs xino - stable ino across mounts" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o xino,lowerdir=lower:upper,upperdir=ext2/upper,workdir=ext2/work mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt echo "==" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o xino,lowerdir=lower:upper,upperdir=ext2/upper,workdir=ext2/work mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt echo echo "kernel fuse - disables xino - non-stable ino across mounts" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o xino,lowerdir=lower:upper:fuseoverlayfs-mount mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt echo "==" /mnt/nobackup/build/fuse-overlayfs/overlay/build/fuse-overlayfs/fuse-overlayfs -o xino,lowerdir=lower:upper:fuseoverlayfs-mount mnt echo -n " mnt/lower " stat -c %i mnt/lower echo -n " mnt/upper " stat -c %i mnt/upper umount mnt umount ext2 rm -fr lower upper mnt img ext2$ ./test-passthrough.sh All on same fs - ino passthrough - stable ino lower/lower 3416920 upper/upper 3416921 mnt/lower 3416920 mnt/upper 3416921 Upper on different fs - non-stable ino across mounts mnt/lower 94353636742544 mnt/upper 94353636742800 == mnt/lower 94220651775376 mnt/upper 94220651775632 Upper on different fs xino - stable ino across mounts mnt/lower 1428392772843533875 mnt/upper 742672234944029848 == mnt/lower 1428392772843533875 mnt/upper 742672234944029848 kernel fuse - disables xino - non-stable ino across mounts mnt/lower 93855548077344 mnt/upper 93855548077600 == mnt/lower 94202806027552 mnt/upper 94202806027808I've now got a small testcase that I'll add to that branch:
cat test-mmap.sh #!/bin/bash set -e BIN=../fuse-overlayfs BIN=fuse-overlayfs cleanup() { umount mnt || true umount ext1 ext2 || true rm -fr mnt img1 img2 ext1 ext2 mupper mlower echo echo "FAILED" echo } trap cleanup EXIT rm -fr mnt img1 img2 ext1 ext2 # override any other settings. export LC_ALL=C echo echo "Testing mmap where files have the same inodes across layers." rm -f img1 truncate -s 10M img1 /sbin/mke2fs -t ext2 img1 >& /dev/null rm -f img2 truncate -s 10M img2 /sbin/mke2fs -t ext2 img2 >& /dev/null mkdir -p ext1 mkdir -p ext2 fuse2fs -o fakeroot,uid=0,gid=0 img1 ext1 fuse2fs -o fakeroot,uid=0,gid=0 img2 ext2 # We assume that the files will get the same inode number on both layers, this is checked in test-passthrough.sh for a single file. dst=ext1 for i in $( ldd $( gcc -print-prog-name=cc1 ) | awk '{print $3}' ); do cp $i $dst if [[ $dst == ext1 ]]; then dst=ext2 else dst=ext1 fi done mkdir mnt $BIN -o lowerdir=ext2:ext1 mnt echo "int main(void) { return 0; }" >test.c LD_LIBRARY_PATH=mnt/ gcc test.c echo "PASSED" umount mnt umount ext1 umount ext2 rm -fr mnt img1 img2 ext1 ext2 trap - EXIT echo echo "FINISHED" echoWhich fails like this:
./test-mmap.sh Testing mmap where files have the same inodes across layers. unknown argument ignored: lazytime gcc: mnt/libm.so.6: version `GLIBC_2.33' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.11' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.3.4' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.6' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.34' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.3' not found (required by gcc) gcc: mnt/libm.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libfmt.so.10) gcc: mnt/libm.so.6: version `GLIBC_2.3.4' not found (required by /lib/x86_64-linux-gnu/libfmt.so.10) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by /lib/x86_64-linux-gnu/libfmt.so.10) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by /lib/x86_64-linux-gnu/libxxhash.so.0) gcc: mnt/libm.so.6: version `GLIBC_2.3.4' not found (required by mnt/libzstd.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.34' not found (required by mnt/libzstd.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.8' not found (required by mnt/libzstd.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by mnt/libzstd.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.3.2' not found (required by mnt/libzstd.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by /lib/x86_64-linux-gnu/libhiredis.so.1.1.0) gcc: mnt/libm.so.6: version `GLIBC_2.3.4' not found (required by /lib/x86_64-linux-gnu/libhiredis.so.1.1.0) gcc: mnt/libm.so.6: version `GLIBC_2.3' not found (required by /lib/x86_64-linux-gnu/libhiredis.so.1.1.0) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.6' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.17' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.3' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.36' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.3.2' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.36' not found (required by mnt/libm.so.6) gcc: mnt/libm.so.6: version `GLIBC_ABI_DT_RELR' not found (required by mnt/libm.so.6) gcc: mnt/libm.so.6: version `GLIBC_PRIVATE' not found (required by mnt/libm.so.6) gcc: mnt/libm.so.6: version `GLIBC_2.14' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.34' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1) gcc: mnt/libm.so.6: version `GLIBC_2.3.2' not found (required by /lib/x86_64-linux-gnu/libgcc_s.so.1) FAILEDand with the changes in that branch:
./test-mmap.sh Testing mmap where files have the same inodes across layers. PASSED FINISHEDThe issues I was seeing with the above MR appear to be a consequence of older userspace that did not always correctly handle 64 bit inodes. The branch now includes an option
-o ino32_twhich masks out the high order bits and my basic tests suggest that all is ok.I have a variety of setups I can test this on all the way back to debian jessie on amd64, i386, arm64 and armel, once all my tests are completed I'll reopen the MR.
I didn't investigate the issue in detail but when running gcc as
/usr/lib/ccache/gccon an i386 buster system, it appears to get into what is most likely an infinite recursion doing stat on/eventually leading to SIGSEGV./sbin/chroot mount /bin/bash root@dirac:/# dpkg --print-architecture i386 root@dirac:/# cat /etc/debian_version 10.13 root@dirac:/# gcc Segmentation fault root@dirac:/# which gcc /usr/lib/ccache/gcc root@dirac:/# strace -f gcc |& cat -n 53 openat(AT_FDCWD, "/etc/ccache.conf", O_RDONLY) = 3 54 fstat64(3, {st_mode=S_IFREG|0664, st_size=20, ...}) = 0 55 read(3, "cache_dir = /ccache\n", 8192) = 20 56 read(3, "", 8192) = 0 57 close(3) = 0 58 openat(AT_FDCWD, "/ccache/ccache.conf", O_RDONLY) = -1 ENOENT (No such file or directory) 59 stat64("/ccache", {st_mode=S_IFDIR|0775, st_size=0, ...}) = 0 60 stat64("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 ... 65502 stat64("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 65503 stat64("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 65504 stat64("/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 65505 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xff0fefa0} --- 65506 +++ killed by SIGSEGV +++using
-o ino32_tfixes.
Greetings,
I've found an issue where if overlayfs is used with more than one layer it breaks system libraries when used with squashfs or dwarfs. This is shown by
symbol not founderrors:If I copy the libraries to the upper directory, it works again. Here is a script to replicate the issue, it quickly sets up two layers, one with an xorg base and another with krita and its dependencies, the issue arises when krita (layer-1) requires libraries from layer-0. If everything is on the same layer this issue does not happen.
Note that this is not a proot issue, since this also breaks in bwrap. I tested with bwrap new
--overlayoptions, and they do not present the issue found here.