Date: Thu, 25 Sep 2008 08:43:57 -0700 From: Jeremy Chadwick <koitsu@FreeBSD.org> To: Pete French <petefrench@ticketswitch.com> Cc: freebsd-stable@freebsd.org, admin@kkip.pl Subject: Re: vm.kmem_size settings doesn't affect loader? Message-ID: <20080925154357.GA16220@icarus.home.lan> In-Reply-To: <E1KisTn-0000xy-AG@dilbert.ticketswitch.com> References: <20080925145154.GA15486@icarus.home.lan> <E1KisTn-0000xy-AG@dilbert.ticketswitch.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Sep 25, 2008 at 04:10:07PM +0100, Pete French wrote: > > These are the tuning settings I use: > > > > vm.kmem_size="1536M" > > vm.kmem_size_max="1536M" > > vfs.zfs.arc_min="16M" > > vfs.zfs.arc_max="64M" > > > > The entire copying process took almost 2 hours. Not once did I > > experience kmem exhaustion. I can *guarantee* that I would have crashed > > the box numerous times had I not tuned the machine with the values > > above. > > I am interested in the last two values you have there - I also use ZFS and > tune to prevent memory exhaustion. I tune kmem_size, but I have not bothered > toi set the arc_min - and the default appears to be 16M anyway. The arc_max > value I do set, but I got dreadful performance when I had it at 6M, so > I upped it to 128M. ARC tuning is what's necessary to induce ZFS memory constraints. If you only set kmem_size/max, you will likely run into cases where the ARC grows to such a size that kmem exhaustion happens. Even the Solaris ZFS "do not tune ZFS" article outlines this fact (read it all, including the RFEs): http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache Just keep in mind this is specific to Solaris, but the problems conceptually are identical to FreeBSD. > This has not resulted in any panics, and gives much better > performance. As I understand it there is only one ARC, and it comes from > kernel space too, so setting it to an extra 64 meg should not use that much > more memory, yes ? The ARC uses kmem. "Should not use that much more memory" is a matter of opinion; if an additional 64MB given to ARC causes kmem exhaustion (there's no global standard for this; keep reading, I explain it near the bottom of my mail), then decreasing the ARC max is necesary. On the bright side, it's very easy to determine how much memory is being dedicated to the ARC. sysctl kstat.zfs.misc will disclose this data and I believe the following statistics are the ones you have to examine for this issue: kstat.zfs.misc.arcstats.p kstat.zfs.misc.arcstats.c kstat.zfs.misc.arcstats.c_min kstat.zfs.misc.arcstats.c_max kstat.zfs.misc.arcstats.size On my system (the one using the tuning parameters you're questioning): kstat.zfs.misc.arcstats.p: 60487680 kstat.zfs.misc.arcstats.c: 67108864 kstat.zfs.misc.arcstats.c_min: 16777216 kstat.zfs.misc.arcstats.c_max: 67108864 kstat.zfs.misc.arcstats.size: 67092480 For those wanting to read about the ARC and what it's for: http://en.wikipedia.org/wiki/Adaptive_Replacement_Cache > My setup is: > > vm.kmem_size="1024M" > vm.kmem_size_max="1024M" > vfs.zfs.arc_max="128M" > vfs.zfs.prefetch_disable=1 > > This is on a 2gb amd64 machine, and works beautifully, with no out of > memory errors and panics. I have been trying to panic this for a few > days. I will probably chnage the kmem size to 1536 which is what I use on > other machines though. Disabling prefetch shouldn't have any effect on stability -- many of us (including FreeBSD core members) have found that disabling prefetch increases system responsiveness during heavy I/O, and results in a less "bursty" system. I disable prefetch, but it should have no bearing on the issue we're discussing. I'm sure the memory tuning values work fine for you, but they do not for me; I can absolutely panic the machine with a kmem size of 1GB. How? Because it's entirely dependent upon the type of ZFS pools/setup a person has, how much actual load they put on ZFS, and probably a hundred other things. Let's adhere to the K.I.S.S. concept, and not turn this thread into a "tuning micro-management" thread. What end-users want is ZFS that "just works", which on Solaris 10 it does. But I have no qualms with tuning ZFS on FreeBSD, though others seem to. A "tuning guide" cannot easily be written because there are too many factors to take into consideration. Besides, such a guide (documenting "how it all interconnects, memory-wise") would end up being tailored to the technical, which won't help end-users. The existing "ZFS tuning guide" for FreeBSD keeps things vague for a good reason -- I'm absolutely 100% positive an extensive document would just confuse the living daylights out of people. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080925154357.GA16220>