From owner-freebsd-current@FreeBSD.ORG Tue Jan 14 20:30:41 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF150244 for ; Tue, 14 Jan 2014 20:30:41 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3808315D8 for ; Tue, 14 Jan 2014 20:30:40 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id WAA28205; Tue, 14 Jan 2014 22:30:38 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1W3AdB-000BMp-W3; Tue, 14 Jan 2014 22:30:38 +0200 Message-ID: <52D59E36.9040405@FreeBSD.org> Date: Tue, 14 Jan 2014 22:29:42 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Vladimir Sharun , Current FreeBSD Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> <52CA9963.1050507@FreeBSD.org> <1389676958.516993176.oq4lbgg7@frv45.ukr.net> In-Reply-To: <1389676958.516993176.oq4lbgg7@frv45.ukr.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 20:30:41 -0000 on 14/01/2014 07:27 Vladimir Sharun said the following: > Dear Andriy and FreeBSD community, > >> I am not sure if the buffers are leaked somehow or if they are actually in use. >> It's one of the very few places where data buffers are allocated without >> charging ARC. In all other places it's quite easy to match allocations and >> deallocations. But in L2ARC it is not obvious that all buffers get freed or >> when that happens. > > After one week under load I think we figure out the cause: it's L2ARC. > Here's the top's header for 7d17h of the runtime: > > last pid: 46409; load averages: 0.37, 0.62, 0.70 up 7+17:14:01 07:24:10 > 173 processes: 1 running, 171 sleeping, 1 zombie > CPU: 2.0% user, 0.0% nice, 3.5% system, 0.4% interrupt, 94.2% idle > Mem: 8714M Active, 14G Inact, 96G Wired, 1929M Cache, 3309M Buf, 3542M Free > ARC: 85G Total, 2558M MFU, 77G MRU, 28M Anon, 1446M Header, 4802M Other > > ARC related tunables: > > vm.kmem_size="110G" > vfs.zfs.arc_max="90G" > vfs.zfs.arc_min="42G" > > For more than 7 days of hard runtime the picture clearly shows: > Wired minus ARC = 11..12Gb, ARC grow and shrinks in 80-87Gb range and the > system runs just fine. > > So what shall we do with L2ARC leakage ? Could you please try this patch http://cr.illumos.org/~webrev/skiselkov/3995/illumos-gate.patch ? -- Andriy Gapon