Date: Fri, 4 Mar 2011 02:56:08 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Matthias Gamsjager <mgamsjager@gmail.com> Cc: freebsd-fs@freebsd.org, =?iso-8859-1?Q?Micka=EBl_Can=E9vet?= <canevet@embl.fr> Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE Message-ID: <20110304105608.GA23887@icarus.home.lan> In-Reply-To: <AANLkTikQiTi25TR6uDD2umRZQrOL8YZzEC960oWf4wax@mail.gmail.com> References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <AANLkTikQiTi25TR6uDD2umRZQrOL8YZzEC960oWf4wax@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Mar 04, 2011 at 11:30:04AM +0100, Matthias Gamsjager wrote: > > # Disable ZFS prefetching > > # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html > > # Increases overall speed of ZFS, but when disk flushing/writes occur, > > # system is less responsive (due to extreme disk I/O). > > # NOTE: Systems with 8GB of RAM or more have prefetch enabled by > > # default. > > vfs.zfs.prefetch_disable="1" > > > > > > I wonder if this is still the case since the website you referring to was > from 2008 and I guess a lot has changed since then. Yes, this is still the case. The website I linked in the comment is dated, but still applicable in my experience (at least to FreeBSD). On Solaris it doesn't appear applicable (every x86 Solaris 10 system at my workplace uses ZFS exclusively, and we do not have to disable prefetching; the performance is fine). We do have one Sol10 system though that's IMAP-centric, uses a 3-disk raidz1, but has severe performance (I/O) issues sometimes, but I'm chalking that up to being it running Dovecot 1.0.0. Circling back to FreeBSD: if you want further confirmations, you can read about my findings with 8.0-RC1 here -- scroll down to about the half way mark: http://koitsu.wordpress.com/2009/10/12/testing-out-freebsd-8-0-rc1/ As far as I can tell from following commits since 8.0-RC1, nothing has changed about the prefetch mechanism in FreeBSD (developers please correct me if I'm wrong here). I should also note that I think my "# NOTE:" comment is inaccurate; I believe it should read "Systems with more than 4GB usable RAM default to having prefetch enabled". I'll have to update my /boot/loader.conf files. > For example I get horrible performance with prefech disabled. (running on 4 > disk striped mirror) Every system I manage (personally; unrelated to my above workplace) has 8GB of RAM in it and run amd64. Some contain 2-disk mirrors, some contain single-disk pools, and a few are 3-disk raidz1. *All* of those systems have historically seen abysmal I/O performance when prefetching was enabled. The hardware is mostly the same (different CPU models, but the same SATA controllers (ICH9R)), so it's not an issue with a single system. If you get better performance -- really, truly, honestly -- with prefetch enabled on your system, then I strongly recommend you keep it enabled. However, for what it's worth (probably not much), this is the first I've ever heard of a FreeBSD system performing better with prefetch enabled. To be completely fair: I should probably dig out my test/stress system and re-test performance with prefetch enabled, keeping in mind the other tunables I use for ZFS (some loader, some sysctl). But those settings are performance-related, and the initial topic of discussion was kmem exhaustion. I don't want to get off-topic for the OP's sake. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110304105608.GA23887>