Date: Thu, 3 Mar 2011 20:23:39 +0100 From: Fabian Keil <freebsd-listen@fabiankeil.de> To: Alexander Leidinger <Alexander@Leidinger.net> Cc: freebsd-fs@FreeBSD.org, freebsd-current@FreeBSD.org, Pawel Jakub Dawidek <pjd@FreeBSD.org> Subject: Re: HEADS UP: ZFSv28 is in! Message-ID: <20110303202339.585b0c0b@r500.local> In-Reply-To: <20110303135147.20096fpe1ntz75k8@webmail.leidinger.net> References: <20110227202957.GD1992@garage.freebsd.pl> <20110228192129.119cac0c@r500.local> <20110228214847.0000078c@unknown> <20110303130130.29a066a1@r500.local> <20110303135147.20096fpe1ntz75k8@webmail.leidinger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/rAZvbCaHCnRG73fv8zdj+23 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Alexander Leidinger <Alexander@Leidinger.net> wrote: > Quoting Fabian Keil <freebsd-listen@fabiankeil.de> (from Thu, 3 Mar =20 > 2011 13:01:30 +0100): >=20 > > Alexander Leidinger <Alexander@Leidinger.net> wrote: > > > >> On Mon, 28 Feb 2011 19:21:29 +0100 Fabian Keil > >> <freebsd-listen@fabiankeil.de> wrote: > >> > >> > Pawel Jakub Dawidek <pjd@FreeBSD.org> wrote: > >> > > >> > > I just committed ZFSv28 to HEAD. > >> > > >> > I updated the system without removing the tuning for ZFSv15 > >> > first, and somehow this completely messed up the performance. > >> > Booting the system took more than ten minutes and even once > >> > it was up it was next to unresponsive. > >> > > >> > I'm not sure which sysctl was to blame, but after removing > >> > all but vfs.zfs.arc_max=3D"800M" and rebooting, the problem > >> > was gone. > >> > >> When you add the tuning back, does it take minutes again to boot? If > >> not, I assume it was cleaning up some leftovers the old version was not > >> able to cleanup. > > > > I haven't tried that yet, but as I didn't upgrade the system's > > storage pool I don't think ZFS is supposed to do any such clean-ups. >=20 > AFAIK the new code knows how to remove some superfluous parts in your =20 > pool (no matter at which version the pool is), which the old code just =20 > skipped over. Such leftovers may not be in all pools, they show up =20 > just in some use cases. For this reason I asked to verify by adding =20 > the tuning back to this system (if possible). I don't have an exact list of sysctls I used earlier, but after commenting in all the zfs sysctls in loader.conf (some of which must have been commented out for quite some time) the problem was back. I interrupted the boot process after 14 minutes at which point the ezjail rc script was running for several minutes already, but still busy with the first jail. Usually this takes only a few seconds. The sysctls used were: #vfs.zfs.txg.timeout=3D5 Seems to be the default now. # vfs.zfs.zil_disable=3D1 No longer supported. # vfs.zfs.prefetch_disable=3D0 The default seems to be 1. # vfs.zfs.write_limit_override=3D15 Clearly the value makes no sense, so this may not have been active at the time of the update. I had a back-ported patch to add the sysctl, so at least in theory it should have caused problems with v15, too, unless there was a sanity check to ignore obviously incorrect values. The auto-tuned write-limit values are: vfs.zfs.write_limit_max: 258863616 vfs.zfs.write_limit_min: 33554432 # vfs.zfs.vdev.max_pending=3D15 The auto-tuned value is 10. vfs.zfs.arc_max=3D"800M" # vfs.zfs.arc_min=3D"500M" # vfs.zfs.vdev.cache.size=3D"5M" The auto-tuned value is 10485760 which seems close enough. # vfs.zfs.txg.synctime=3D1 This sysctl doesn't seem to exist (anymore). #vfs.zfs.cache_flush_disable=3D1 The default is 0. # vfs.zfs.txg.write_limit_override=3D134217728 Doesn't seem to exist (anymore) either. #vfs.zfs.vdev.max_pending=3D2 #vfs.zfs.vdev.min_pending=3D1 The auto-tuned values are vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 10 > If it is not a production-like system which does not accept downtime, =20 > this verification consumes less resources than sending out a developer =20 > hunting for a problem which may not even exist. It wasn't my intention to send anyone hunting. Fabian --Sig_/rAZvbCaHCnRG73fv8zdj+23 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) iEYEARECAAYFAk1v6sEACgkQBYqIVf93VJ1ryACgsaHxQhqQqQLB594YBLCtmZlS K/kAn0Cf3I6HqwPrhQJBE/+t+E8H+gz5 =WKq1 -----END PGP SIGNATURE----- --Sig_/rAZvbCaHCnRG73fv8zdj+23--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110303202339.585b0c0b>