Date: Tue, 28 Sep 2010 14:40:00 -0400 From: Ben Kelly <ben@wanderview.com> To: Andriy Gapon <avg@icyb.net.ua> Cc: stable@freebsd.org, fs@freebsd.org Subject: Re: Still getting kmem exhausted panic Message-ID: <F244BA6D-3347-4D76-BAFB-D8B975783877@wanderview.com> In-Reply-To: <4CA22337.2010900@icyb.net.ua> References: <4CA1D06C.9050305@digiware.nl> <20100928115047.GA62142__15392.0458550148$1285675457$gmane$org@icarus.home.lan> <4CA1DDE9.8090107@icyb.net.ua> <20100928132355.GA63149@icarus.home.lan> <4CA1EF69.4040402@icyb.net.ua> <FE116FEC-714D-4BF5-86D8-E29BFA713C69@wanderview.com> <4CA21809.7090504@icyb.net.ua> <71D54408-4B97-4F7A-BD83-692D8D23461A@wanderview.com> <4CA22337.2010900@icyb.net.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sep 28, 2010, at 1:17 PM, Andriy Gapon wrote: > on 28/09/2010 19:46 Ben Kelly said the following: >> Hmm. My server is currently idle with no I/O happening: >>=20 >> kstat.zfs.misc.arcstats.c: 25165824 >> kstat.zfs.misc.arcstats.c_max: 46137344 >> kstat.zfs.misc.arcstats.size: 91863156 >>=20 >> If what you say is true, this shouldn't happen, should it? This = system is an i386 machine with kmem max at 800M and arc set to 40M. = This is running head from April 6, 2010, so it is a bit old, though. >=20 > Well, your system is a bit old indeed. > And the branch is unknown, so I can't really see what sources you = have. > And I am not sure if I'll be able to say anything about those sources. Quite old. I've been intending to update, but haven't found the time = lately. I'll try to do the upgrade this weekend and see if it changes = anything. > As to the numbers - yes, with current code I'd expect arcstats.size to = go down to > arcstats.c when there is no I/O. arc_reclaim_thread should do that. Thats what I thought as well, but when I debugged it a year or two ago I = found that the buffers were still referenced and thus could not be = reclaimed. As far as I can remember they needed a vfs/vnops like = zfs_vnops_inactive or zfs_vnops_reclaim to be executed in order to free = the reference. What is responsible for making those calls? >=20 >> At one point I had patches running on my system that triggered the = pagedaemon based on arc load and it did allow me to keep my arc below = the max. Or at least I thought it did. >>=20 >> In any case, I've never really been able to wrap my head around the = VFS layer and how it interacts with zfs. So I'm more than willing to = believe I'm confused. Any insights are greatly appreciated. >=20 > ARC is a ZFS private cache. > ZFS doesn't use unified buffer/page cache. > So ARC is not directly affected by pagedaemon. > But this is not exactly VFS layer thing. Can you explain the difference in how the vfs/vnode operations are = called or used for those two situations? I thought that the buffer cache was used by filesystems to implement = these operations. So that the buffer cache was below the vfs/vnops = layer. So while zfs implemented its operations in terms of the arc, = things like UFS implemented vfs/vnops in terms of the buffer cache. I = thought the layers further up the chain like the page daemon did not = distinguish that much between these two implementation due to the VFS = interface layer. (Although there seems to be a layering violation in = that the buffer cache signals directly to the upper page daemon layer to = trigger page reclamation.) The old (ancient) patch I tried previously to help reduce the arc = working set and allow it to shrink is here: http://www.wanderview.com/svn/public/misc/zfs/zfs_kmem_limit.diff Unfortunately, there are a couple ideas on fighting fragmentation mixed = into that patch. See the part about arc_reclaim_pages(). This patch = did seem to allow my arc to stay under the target maximum even when = under load that previously caused the system to exceed the maximum. = When I update this weekend I'll try a stripped down version of the patch = to see if it helps or not with the latest zfs. Thanks for your help in understanding this stuff! - Ben=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F244BA6D-3347-4D76-BAFB-D8B975783877>