From owner-freebsd-stable@FreeBSD.ORG Tue Sep 28 13:23:58 2010 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3B3D81065672 for ; Tue, 28 Sep 2010 13:23:58 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 20C338FC0A for ; Tue, 28 Sep 2010 13:23:57 +0000 (UTC) Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51]) by qmta06.emeryville.ca.mail.comcast.net with comcast id CCPM1f00116AWCUA6DPxLX; Tue, 28 Sep 2010 13:23:57 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta06.emeryville.ca.mail.comcast.net with comcast id CDPw1f0013LrwQ28SDPwRH; Tue, 28 Sep 2010 13:23:57 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id EC1849B418; Tue, 28 Sep 2010 06:23:55 -0700 (PDT) Date: Tue, 28 Sep 2010 06:23:55 -0700 From: Jeremy Chadwick To: Andriy Gapon Message-ID: <20100928132355.GA63149@icarus.home.lan> References: <4CA1D06C.9050305@digiware.nl> <20100928115047.GA62142__15392.0458550148$1285675457$gmane$org@icarus.home.lan> <4CA1DDE9.8090107@icyb.net.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4CA1DDE9.8090107@icyb.net.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: stable@freebsd.org, Willem Jan Withagen , fs@freebsd.org Subject: Re: Still getting kmem exhausted panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Sep 2010 13:23:58 -0000 On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: > on 28/09/2010 14:50 Jeremy Chadwick said the following: > > I believe the trick -- Andriy, please correct me if I'm wrong -- is the > > Wouldn't hurt to CC me, so that I could do it :-) > > > tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high > > watermark". > > Not sure what you mean here. > What is hard limit, what is high watermark, what is the difference and when is > "now"? :-) There was some speculation on the part of users a while back which lead to this understanding. Folks were seeing actual ARC usage higher than what vfs.zfs.arc_max was set to (automatically or administratively). I believe it started here: http://www.mailinglistarchive.com/freebsd-current@freebsd.org/msg28884.html With the "high-water mark" statements being here: http://www.mailinglistarchive.com/freebsd-current@freebsd.org/msg28887.html http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2010-04/msg00129.html The term implies that there is not an explicitly hard limit on the ARC utilisation/growth. As stated in the unix.derkeiler.com URL above, this behaviour was in fact changed. Why/when/how? I had to go digging up the commits -- this took me some time. Here they are, labelled r197816, for RELENG_8 and RELENG_7 respectively. These were both committed on 2010/01/08 UTC: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.22.2.2 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.15.2.6 In HEAD/CURRENT (yet to be MFC'd), it looks like above code got removed on 2010/09/17 UTC, citing they should be "enforced by actual calculations of delta": http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.46 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.45 So what's this "delta" code piece that's mentioned? That appears to be have been committed to RELENG_8 on 2010/05/24 UTC (thus, between the above two dates): http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c#rev1.22.2.4 (Side note: the "delta stuff" was never committed to RELENG_7 -- and that's fine. I'm pointing this out not out of retaliation or insult, but because people will almost certainly Google, find this post, and wonder if their 7.x machines might be affected.) This situation with the ARC, and all its changes over time, is one of the reasons why I rant aggressively about the need for more communication transparency (re: what the changes actually affect). Most SAs and users don't follow commits. > I believe that "the trick" is to set vm.kmem_size high enough, eitehr using this > tunable or vm.kmem_size_scale. Thanks for the clarification. I just wish I knew how vm.kmem_size_scale fit into the picture (meaning what it does, etc.). The sysctl description isn't very helpful. Again, my lack of VM knowledge... > > However, I believe there have been occasional reports of exhaustion > > panics despite both of these being set[1]. Those reports are being > > investigated on an individual basis. > > I don't believe that the report that you quote actually demonstrates what you say > it does. > Two quotes from it: > "During these panics no tuning or /boot/loader.conf values where present." > "Only after hitting this behaviour yesterday i created boot/loader.conf" > > > [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059109.html You're right -- the report I'm quoting is not the one I thought it was. I'll see if I can dig up the correct mail/report. It could be that I'm thinking of something quite old (pre-ARC-changes (see above paragraphs)). I can barely keep track of all the changes going on. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |