From owner-freebsd-stable@FreeBSD.ORG Wed Feb 8 20:29:48 2012 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97595106564A for ; Wed, 8 Feb 2012 20:29:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id DE31B8FC14 for ; Wed, 8 Feb 2012 20:29:47 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id WAA17155; Wed, 08 Feb 2012 22:29:38 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1RvE94-0001yu-GI; Wed, 08 Feb 2012 22:29:38 +0200 Message-ID: <4F32DB30.6020600@FreeBSD.org> Date: Wed, 08 Feb 2012 22:29:36 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0) Gecko/20120202 Thunderbird/10.0 MIME-Version: 1.0 To: "Eugene M. Zheganin" References: <4F30E284.8080905@norma.perm.ru> <4F310115.3070507@FreeBSD.org> <4F310C5A.6070400@norma.perm.ru> <4F310E75.7090301@FreeBSD.org> <4F3144A9.2000505@norma.perm.ru> <4F314892.50806@FreeBSD.org> <4F314B5B.100@norma.perm.ru> <4F3186C6.8000904@FreeBSD.org> <4F324F10.2060508@norma.perm.ru> In-Reply-To: <4F324F10.2060508@norma.perm.ru> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable Subject: Re: zfs arc and amount of wired memory X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2012 20:29:48 -0000 on 08/02/2012 12:31 Eugene M. Zheganin said the following: > Hi. > > On 08.02.2012 02:17, Andriy Gapon wrote: >> [output snipped] >> >> Thank you. I don't see anything suspicious/unusual there. >> Just case, do you have ZFS dedup enabled by a chance? >> >> I think that examination of vmstat -m and vmstat -z outputs may provide some >> clues as to what got all that memory wired. >> > Nope, I don't have deduplication feature enabled. OK. So, did you have a chance to inspect vmstat -m and vmstat -z? > By the way, today, after eating another 100M of wired memory this server hanged > out with multiple non-stopping messages > > swap_pager: indefinite wait buffer > > Since it's swapping on zvol, it looks to me like it could be the mentioned in > another thread here ("Swap on zvol - recommendable?") resource starvation issue; > may be it happens faster when the ARC isn't limited. It could be very well possible that swap on zvol doesn't work well when the kernel itself is starved on memory. > So I want to ask - how to report it and what should I include in such pr ? I am leaving swap-on-zvol issue aside. Your original problem doesn't seem to be ZFS-related. I suspect that you might be running into some kernel memory leak. If you manage to reproduce the high wired value again, then vmstat -m and vmstat -z may provide some useful information. In this vein, do you use any out-of-tree kernel modules? Also, can you try to monitor your system to see when wired count grows? -- Andriy Gapon