Date: Mon, 3 Oct 2011 21:56:07 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Aristedes Maniatis <ari@ish.com.au> Cc: freebsd-fs@freebsd.org Subject: Re: vm.kmem_size_scale recommendation for ZFS Message-ID: <20111004045607.GA8611@icarus.home.lan> In-Reply-To: <4E8A8740.100@ish.com.au> References: <4E8A8740.100@ish.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 04, 2011 at 03:10:40PM +1100, Aristedes Maniatis wrote: > Having just suffered from a FreeBSD 8.2 machine hit with the "kmem_map too small" error [1], what is the current recommendation for a machine running RELEASE-8.2? > > A. vm.kmem_size_scale=1 (this is the new default in 8-STABLE but 8.2-RELEASE had it set to 3) > B. vm.kmem_size_scale=0.66 (this is aligned with Pawel's recommendation of setting vm.kmem_size to 150% of your RAM. [2] > C. Leaving vm.kmem_size_scale alone and manually setting vm.kmem_size directly as Pawel is doing > > [1] http://freebsd.1045724.n5.nabble.com/kmem-map-too-small-with-ZFS-and-8-2-RELEASE-td4029979.html > [2] http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration Even if you were to mess about with vm.kmem_size_scale (which IMO should be set to 1 -- it defaults to that in 8.2-STABLE as well as other upcoming releases), you will still experience problems with kmem map exhaustion **even if** you limit the ARC. There are fixes in 8.2-STABLE that address this problem (there were multiple fixes put in place). So if you're already experiencing kmem map exhaustion, I strongly recommend you upgrade to 8.2-STABLE and then adjust one single tunable: vfs.zfs.arc_max You didn't disclose how much RAM your machine has, nor what other daemons are running on it, so it's very hard to give you an estimate. I tend to tell people to set vfs.zfs.arc_max to about 60% of their memory. E.g. if the machine has 8GB RAM, set vfs.zfs.arc_max="5120M". We use this on all of our servers with 8GB, including ones that run mysqld with some tunings. Example machine in question: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 60030 mysql 16 76 0 785M 294M sigwai 0 15:38 0.00% [mysqld] And relevant bits from /boot/loader.conf: kern.maxdsiz="2560M" kern.dfldsiz="2560M" kern.maxssiz="256M" vfs.zfs.arc_max="5120M" You need to keep in mind that ZFS can still use more than what you limit the ARC to. The mailing list has explanations regarding this; I believe it has to do with fragmentation or something like that. So don't go thinking you're being smart by setting it to something like "7500M"; there is a very good chance you will still experience the same problem. So my advice is to "start small" and work your way up after multiple weeks (not days!) of utilising the filesystems on ZFS, to really stress the ARC. Also be aware there may still be problems with applications that use sendfile(2) -- known programs which use this are ftpd (not adjustable), Apache (you can disable it), nginx (I think you can disable it), and Samba (you can disable it). There are some sendfile(2) fixes were which committed, but I'm not 100% sure if all situations have been accounted for. P.S. -- I am assuming you're using amd64. If you're using i386 this whole situation becomes much, much more complex. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111004045607.GA8611>