Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 28 Sep 2010 04:50:47 -0700
From:      Jeremy Chadwick <freebsd@jdc.parodius.com>
To:        Willem Jan Withagen <wjw@digiware.nl>
Cc:        stable@freebsd.org, fs@freebsd.org
Subject:   Re: Still getting kmem exhausted panic
Message-ID:  <20100928115047.GA62142@icarus.home.lan>
In-Reply-To: <4CA1D06C.9050305@digiware.nl>
References:  <4CA1D06C.9050305@digiware.nl>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote:
> This is with stable as of yesterday,but with an un-tunned ZFS box I
> was still able to generate a kmem exhausted panic.
> Hard panic, just 3 lines.
> 
> The box contains 12Gb memory, runs on a 6 core (with HT) xeon.
> 6* 2T WD black caviar in raidz2 with 2*512Mb mirrored log.
> 
> The box died while rsyncing 5.8T from its partnering system.
> (that was the only activity on the box)

It would help if you could provide output from the following commands
(even after the box has rebooted):

$ sysctl -a | egrep ^vm.kmem
$ sysctl -a | egrep ^vfs.zfs.arc
$ sysctl kstat.zfs.misc.arcstats

> So the obvious would to conclude that auto-tuning voor ZFS on
> 8.1-Stable is not yet quite there.
> 
> So I guess that we still need tuning advice even for 8.1.
> And thus prevent a hard panic.

Andriy Gapon provides this general recommendation:

http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059114.html

The advice I've given for RELENG_8 (as of the time of this writing),
8.1-STABLE, and 8.1-RELEASE, is that for amd64 you'll need to tune:

vm.kmem_size
vfs.zfs.arc_max

An example machine: amd64, with 4GB physical RAM installed (3916MB
available for use (verified via dmesg)) uses values:

vm.kmem_size="4096M"
vfs.zfs.arc_max="3584M"

Another example machine: amd64, with 8GB physical RAM installed (7875MB
available for use) uses values:

vm.kmem_size="8192M"
vfs.zfs.arc_max="6144M"

I believe the trick -- Andriy, please correct me if I'm wrong -- is the
tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high
watermark".

However, I believe there have been occasional reports of exhaustion
panics despite both of these being set[1].  Those reports are being
investigated on an individual basis.

I set some other ZFS-related parameters as well (disabling prefetch,
adjusting txg.timeout, etc.), but those shouldn't be necessary to gain
stability at this point in time.

I can't provide tuning advice for i386.


[1]: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059109.html

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100928115047.GA62142>