Date: Fri, 8 Nov 2019 08:03:52 +0100 From: Peter Eriksson <pen@lysator.liu.se> To: Chris Watson <bsdunix44@gmail.com>, Jan Behrens <jbe-mlist@magnetkern.de>, =?utf-8?Q?Karli_Sj=C3=B6berg_via_freebsd-fs?= <freebsd-fs@freebsd.org> Subject: Re: ZFS snapdir readability (Crosspost) Message-ID: <C5B9E02E-BC08-4077-81CD-5CABB67F422B@lysator.liu.se> In-Reply-To: <65AE896D-A32E-451A-B9D0-EC40D438BB03@gmail.com> References: <FBB088B0-CE5C-45DC-8F2F-0D0AA2703846@lysator.liu.se> <65AE896D-A32E-451A-B9D0-EC40D438BB03@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Yes, and no. We use =E2=80=98refquotas=E2=80=99 on the user filesystems = to prevent a user from using up all the free space in the zpool.=20 However, what happens is that ZFS will decrease the =E2=80=9Ctransaction = size=E2=80=9D of writes when a filesystem is nearly at quota in order to = prevent a transaction to write more than the assigned quota (since it = might write more due to compression/snapshots etc (or whatever)). Now = this is (apparently) by design and it works.=20 The effect is that a user trying to write to a nearly full (at quota) = filesystem will =E2=80=9Csee=E2=80=9D an extremely slow fileserver (from = MB/s to KB/s (or B/s if _really_ full) which makes for interesting bug = reports and problem diagnosing (since other users at the same time will = see no such slow-downs). _However_ - other effects are that creating snapshots, or destroying = snapshots _also_ can take longer than usual. Or if you delete a lot of = snapshots at the same time as a user has deleted a lot of files and you = then reboot the server - then when it starts up again and attempts to = mount all filesystems you will notice that it might take =E2=80=9Cforever=E2= =80=9D and it might look like =E2=80=9Czfs mount -a=E2=80=9D is = =E2=80=9Cstuck=E2=80=9D. Luckily for us when this happened we had = previously modified the system startup scripts so all zfs mounting is = done in the background and in parallell - so we got a login prompt and = could login and run some commands (normally all filesystems are mounted = first before allowing system logins=E2=80=A6.) So we logged in and noticed that it was doing a lot of writes (zpool = iostat) and was =E2=80=9Cstuck=E2=80=9D at attempting to mount a single = specific filesystem. A filesystem that happened to have 0 bytes free = refquota. When I added some 10G more =E2=80=98refquota=E2=80=99 to that = filesystem things speeded up a lot :-). (First time this happened we let the server finish mounting by itself - = which took about 2.5 days=E2=80=A6) - Peter > On 8 Nov 2019, at 01:31, Chris Watson <bsdunix44@gmail.com> wrote: >=20 > Peter, on your last point about 100% utilization, don=E2=80=99t you = use quotas/user quotas to prevent that? >=20 > Chris >=20 > Sent from my iPhone >=20 >> On Nov 7, 2019, at 4:06 PM, Peter Eriksson <pen@lysator.liu.se> = wrote: >>=20 >> =EF=BB=BFThe =E2=80=9Ceasy=E2=80=9D solution is to give each user (or = group / project) their own ZFS filesystem. Then the =E2=80=9C.zfs=E2=80=9D= directory would be inside the users own $HOME and you can set $=08HOME = to 0700=E2=80=A6. >>=20 >> That is what we are doing. Granted it generates a =E2=80=9Cfew=E2=80=9D= filesystems (like some 20000 per server (we have around 120k users), = and then add hourly snapshots to each as =E2=80=9Cicing=E2=80=9D on the = cake). Mounting all those takes a bit of time - but luckily with the = latest FreeBSD release things are much faster these days :-) >>=20 >> There are some other issues with that - like 100% full filesystems = causing severe system slowdown during writes=E2=80=A6 So you really = wanna have some monitoring system that warns for that. >>=20 >> - Peter >>=20 >>=20 >>>=20 >>> I recently noticed that all ZFS filesystems in FreeBSD allow access = to >>> the .zfs directory (snapdir) for all users of the system. It is >>> possible to hide that directory using the snapdir option: >>=20 >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C5B9E02E-BC08-4077-81CD-5CABB67F422B>