Date: Thu, 18 Apr 2019 16:33:19 -0700 From: Jim Long <freebsd-questions@umpquanet.com> To: freebsd-questions@freebsd.org Subject: ZFS is auto-mounting in wrong order Message-ID: <20190418233319.GA28238@g5.umpquanet.com>
next in thread | raw e-mail | index | archive | help
I have a system that boots from ZFS. I recently upgraded it from FreeBSD 11.2-STABLE to 12.0-STABLE: FreeBSD electron 12.0-STABLE FreeBSD 12.0-STABLE r346293 GENERIC amd64 I'm seeing certain directories as empty, when they should have lots of content. The problem can be worked around by unmounting key filesystems, and re-mounting them in the proper order. Troubleshooting leads me to believe that the ZFS system is not mounting them in the proper order, but I don't know of any way to either log the order that they are mounted in at boot time, nor to inspect on a hypothetical "dry-run" basis what ZFS thinks the correct mount order is. I did not have this problem prior to the 11 -> 12 upgrade, if that matters. Here are my auto-mountable filesystems: # zfs get -rt filesystem canmount | grep -vw off NAME PROPERTY VALUE SOURCE electron canmount on default electron/data canmount on local electron/data/backup canmount on local electron/data/backup/jimsdesk canmount on default electron/system canmount on default electron/system/ROOT canmount on default electron/system/ROOT/default canmount on local electron/system/home canmount on default electron/system/tmp canmount on default electron/system/usr canmount on default electron/system/usr/local canmount on default electron/system/var canmount on default And their mountpoints: # zfs get -Hrt filesystem canmount | grep -vw off | while read FS b; do echo $FS; done | xargs zfs list NAME USED AVAIL REFER MOUNTPOINT electron 80.0T 403G 201K none electron/data 46.4T 34.0T 238K /data electron/data/backup 6.67T 34.0T 4.81T /data/backup electron/data/backup/jimsdesk 267G 34.0T 267G /data/backup/jimsdesk electron/system 13.4G 403G 201K none electron/system/ROOT 560M 403G 201K none electron/system/ROOT/default 373M 403G 279M / electron/system/home 292K 403G 292K /home electron/system/tmp 15.5M 403G 13.8M /tmp electron/system/usr 12.3G 403G 10.6G /usr electron/system/usr/local 1007M 403G 569M /usr/local electron/system/var 555M 403G 516M /var Let's look at electron/data and its children, although I suspect the same problem is happening with electron/system/usr and its child electron/system/usr/local. None of the electron/data tree is currently mounted: # zfs mount electron/system/ROOT/default / electron/system/tmp /tmp electron/system/var /var electron/system/home /home electron/system/usr /usr electron/system/usr/local /usr/local Since they all mount under /data, the /data directory is currently empty: # find /data -ls 1587 1 drwxr-xr-x 2 root wheel 2 Apr 18 15:33 /data Now let's mount the remaining auto-mount filesystems: # zfs mount -a; zfs mount electron/system/ROOT/default / electron/system/tmp /tmp electron/system/var /var electron/system/home /home electron/system/usr /usr electron/system/usr/local /usr/local electron/data /data electron/data/backup /data/backup electron/data/backup/jimsdesk /data/backup/jimsdesk Inspect /data and /data/backup: # ls /data backup/ slide/ xqueue/ spare/ vince/ mendel/ rain/ # ls /data/backup jimsdesk/ There should be a lot more stuff in /data/backup than that. It's only showing the mountpoint for electron/data/backup/jimsdesk. Here's the work-around to get it to show up: # zfs umount electron/data/backup/jimsdesk # zfs umount electron/data/backup # zfs mount electron/data/backup # find /data/backup -maxdepth 1 /data/backup /data/backup/.zfs /data/backup/rayleigh.CentOS-7 /data/backup/horton /data/backup/lannister /data/backup/rayleigh.archived /data/backup/Thunder /data/backup/luna /data/backup/bioinformatics /data/backup/shiny /data/backup/salmon /data/backup/rayleigh.CentOS-6 /data/backup/stark /data/backup/hoh2 /data/backup/jimsdesk /data/backup/love /data/backup/rayleigh So clearly the filesystem contents are still there, they were just invisible before. Since /data is a mountpoint for electron/data, we would expect it to be empty, and indeed we verified above that it WAS empty, prior to doing the 'zfs mount -a'. But instead I find: # zfs umount electron/data/backup # zfs umount electron/data # find /data /data /data/backup # df -m /data/backup Filesystem 1M-blocks Used Avail Capacity Mounted on electron/system/ROOT/default 413020 278 412742 0% / /data/backup should not exist in electron/system/ROOT/default. So I rmdir'ed it and rebooted. After the reboot, and after unmounting electron/data/backup/jimsdesk, electron/data/backup and electron/data it again was present. This seems like strong evidence that /data/backup is getting mounted before /data gets mounted, and ZFS is auto-creating the mountpoint directory. How can I definitively SEE the order in which ZFS will mount filesystems, either in a "dry-run" mode, or have it logged in real time during the boot process? Even more to the point, how can I ensure that /data gets mounted before /data/backup gets mounted before /data/backup/jimsdesk, and likewise /usr gets mounted before /usr/local? Thank you! Jim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190418233319.GA28238>