From owner-freebsd-current@FreeBSD.ORG Wed Jan 15 11:16:08 2014 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 59ECCCE2 for ; Wed, 15 Jan 2014 11:16:08 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 87CB9167E for ; Wed, 15 Jan 2014 11:16:07 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA13035; Wed, 15 Jan 2014 13:15:58 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1W3ORy-000FEV-H2; Wed, 15 Jan 2014 13:15:58 +0200 Message-ID: <52D66DB6.7030807@FreeBSD.org> Date: Wed, 15 Jan 2014 13:15:02 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Vitalij Satanivskij Subject: Re: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> <52CA9963.1050507@FreeBSD.org> <1389676958.516993176.oq4lbgg7@frv45.ukr.net> <52D59E36.9040405@FreeBSD.org> <20140115102837.GA98983@hell.ukr.net> In-Reply-To: <20140115102837.GA98983@hell.ukr.net> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Vladimir Sharun , Current FreeBSD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2014 11:16:08 -0000 on 15/01/2014 12:28 Vitalij Satanivskij said the following: > Dear Andriy and FreeBSD community, > > Andriy Gapon wrote: > AG> on 14/01/2014 07:27 Vladimir Sharun said the following: > AG> > Dear Andriy and FreeBSD community, > AG> > > AG> >> I am not sure if the buffers are leaked somehow or if they are actually in use. > AG> >> It's one of the very few places where data buffers are allocated without > AG> >> charging ARC. In all other places it's quite easy to match allocations and > AG> >> deallocations. But in L2ARC it is not obvious that all buffers get freed or > AG> >> when that happens. > AG> > > AG> > After one week under load I think we figure out the cause: it's L2ARC. > AG> > Here's the top's header for 7d17h of the runtime: > AG> > > AG> > last pid: 46409; load averages: 0.37, 0.62, 0.70 up 7+17:14:01 07:24:10 > AG> > 173 processes: 1 running, 171 sleeping, 1 zombie > AG> > CPU: 2.0% user, 0.0% nice, 3.5% system, 0.4% interrupt, 94.2% idle > AG> > Mem: 8714M Active, 14G Inact, 96G Wired, 1929M Cache, 3309M Buf, 3542M Free > AG> > ARC: 85G Total, 2558M MFU, 77G MRU, 28M Anon, 1446M Header, 4802M Other > AG> > > AG> > ARC related tunables: > AG> > > AG> > vm.kmem_size="110G" > AG> > vfs.zfs.arc_max="90G" > AG> > vfs.zfs.arc_min="42G" > AG> > > AG> > For more than 7 days of hard runtime the picture clearly shows: > AG> > Wired minus ARC = 11..12Gb, ARC grow and shrinks in 80-87Gb range and the > AG> > system runs just fine. > AG> > > AG> > So what shall we do with L2ARC leakage ? > AG> > AG> > AG> Could you please try this patch > AG> http://cr.illumos.org/~webrev/skiselkov/3995/illumos-gate.patch ? > AG> > > While applying path to curent version of arc.c (r260622) I'm found next truble with compilation > > olaris/uts/common/fs/zfs/arc.c -o arc.o > /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4628:18: error: use of > undeclared identifier 'abl2' > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr, > ^ > 1 error generated. > *** Error code 1 > > > the code is - > > if (zio->io_error != 0) { > /* > * Error - drop L2ARC entry. > */ > list_remove(buflist, ab); > ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); > ab->b_l2hdr = NULL; > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr, > ab->b_size, 0); > kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t)); > ARCSTAT_INCR(arcstat_l2_size, -ab->b_size); > } > > > Looks like it's part is freebsd specific changes. > Can somebody help with this part of code ? > The first hunk of the patch is renaming of abl2 to l2hdr. -- Andriy Gapon