From owner-freebsd-stable@FreeBSD.ORG Thu Feb 9 01:32:07 2012 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA237106566B for ; Thu, 9 Feb 2012 01:32:07 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 59AD28FC12 for ; Thu, 9 Feb 2012 01:32:07 +0000 (UTC) Received: (qmail 12500 invoked by uid 0); 9 Feb 2012 01:05:25 -0000 Received: from smtp.bway.net (216.220.96.25) by xena.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 9 Feb 2012 01:05:25 -0000 Received: (qmail 12484 invoked by uid 90); 9 Feb 2012 01:05:25 -0000 Received: from unknown (HELO ?10.3.2.40?) (spork@96.57.144.66) by smtp.bway.net with (AES128-SHA encrypted) SMTP; 9 Feb 2012 01:05:25 -0000 Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Charles Sprickman In-Reply-To: <4F330F38.3010806@quip.cz> Date: Wed, 8 Feb 2012 20:05:24 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4F30E284.8080905@norma.perm.ru> <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> <4F32DB30.6020600@FreeBSD.org> <4F330F38.3010806@quip.cz> To: Miroslav Lachman <000.fbsd@quip.cz> X-Mailer: Apple Mail (2.1084) Cc: "Eugene M. Zheganin" , freebsd-stable , Andriy Gapon Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Feb 2012 01:32:07 -0000 On Feb 8, 2012, at 7:11 PM, Miroslav Lachman wrote: > Andriy Gapon wrote: >> on 08/02/2012 12:31 Eugene M. Zheganin said the following: >>> Hi. >>>=20 >>> On 08.02.2012 02:17, Andriy Gapon wrote: >>>> [output snipped] >>>>=20 >>>> Thank you. I don't see anything suspicious/unusual there. >>>> Just case, do you have ZFS dedup enabled by a chance? >>>>=20 >>>> I think that examination of vmstat -m and vmstat -z outputs may = provide some >>>> clues as to what got all that memory wired. >>>>=20 >>> Nope, I don't have deduplication feature enabled. >>=20 >> OK. So, did you have a chance to inspect vmstat -m and vmstat -z? >>=20 >>> By the way, today, after eating another 100M of wired memory this = server hanged >>> out with multiple non-stopping messages >>>=20 >>> swap_pager: indefinite wait buffer >>>=20 >>> Since it's swapping on zvol, it looks to me like it could be the = mentioned in >>> another thread here ("Swap on zvol - recommendable?") resource = starvation issue; >>> may be it happens faster when the ARC isn't limited. >>=20 >> It could be very well possible that swap on zvol doesn't work well = when the >> kernel itself is starved on memory. >>=20 >>> So I want to ask - how to report it and what should I include in = such pr ? >>=20 >> I am leaving swap-on-zvol issue aside. Your original problem doesn't = seem to be >> ZFS-related. I suspect that you might be running into some kernel = memory leak. >> If you manage to reproduce the high wired value again, then vmstat = -m and >> vmstat -z may provide some useful information. >>=20 >> In this vein, do you use any out-of-tree kernel modules? >> Also, can you try to monitor your system to see when wired count = grows? >=20 > I am seeing something similar on one of our machine. This is old 7.3 = with ZFS v13, that's why I did not reported it. >=20 > The machine is used as storage for backups made by rsync. All is = running fine for about 107 days. Then backups are slower and slower = because of some strange memory situation. >=20 > Mem: 15M Active, 17M Inact, 3620M Wired, 420K Cache, 48M Buf, 1166M = Free >=20 > ARC Size: > Current Size: 1769 MB (arcsize) > Target Size (Adaptive): 512 MB (c) > Min Size (Hard Limit): 512 MB (zfs_arc_min) > Max Size (Hard Limit): 3584 MB (zfs_arc_max) >=20 > The target size is going down to the min size and after few more days, = the system is so slow, that I must reboot the machine. Then it is = running fine for about 107 days and then it all repeat again. >=20 > You can see more on MRTG graphs > http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/ > You can see links to other useful informations on top of the page = (arc_summary, top, dmesg, fs usage, loader.conf) >=20 > There you can see nightly backups (higher CPU load started at 01:13), = otherwise the machine is idle. >=20 > It coresponds with ARC target size lowering in last 5 days > = http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_arcst= ats_size.html >=20 > And with ARC metadata cache overflowing the limit in last 5 days > = http://freebsd.quip.cz/ext/2012/2012-02-08-kiwi-mrtg-12-15/local_zfs_vfs_m= eta.html I'm not having luck finding it, but there's some known issue that exists = even in 8.2 where some 32-bit counter overflows or something. I don't = truly remember the logic in it, but when you hit it, it's around 110 = days or so. Before it gets really bad (to the point where you either = reboot or get some memory exhaustion panic), you can see zfs "evict = skips" incrementing rapidly. Looking at that graph, that would be my = guess as to what's happening to you. It's easy to check - run one of = the arc stats scripts, look for "evict_skips", note the number and then = run it a few minutes later. If it increases by more than a few hundred, = you've hit the bug. You'll find at that point the kernel is no longer = "evicting" ARC from the kernel and it will just continue to grow until = bad things happen. Charles >=20 > I don't know what's going on and I don't know if it is something know = / fixed in newer releases. We are running a few more ZFS systems on 8.2 = without this issue. But those systems are in different roles. >=20 > Miroslav Lachman > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"