From owner-freebsd-current@FreeBSD.ORG Thu Jan 30 12:18:58 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 151568D8 for ; Thu, 30 Jan 2014 12:18:58 +0000 (UTC) Received: from frv199.fwdcdn.com (frv199.fwdcdn.com [212.42.77.199]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD63C1562 for ; Thu, 30 Jan 2014 12:18:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-Id:To:Subject:From:Date; bh=IVQPbHUHUP2vTQNiGz0x220+OfyOC8ZGH9mO6fK0YWE=; b=vjlE6/b6tJ2GM8LIIk2/HcBW0fxk7wqMYvtKD62HSZfLvrLmeBs0DsBzP5SXRORMm1PR23C40qfCa/QbwmTI9RIL9WPgmX0Lc+yD3Pk0aOhFZb0pkfAZVbESY3Ju42h2I6BqvrucDiDdAhclasfXPkbOsdftVdRMTilhhP2BozE=; Received: from [10.10.10.45] (helo=frv45.ukr.net) by frv199.fwdcdn.com with smtp ID 1W8qa4-000DST-8F for freebsd-current@freebsd.org; Thu, 30 Jan 2014 14:18:52 +0200 Date: Thu, 30 Jan 2014 14:18:51 +0200 From: Vladimir Sharun Subject: Re[2]: ARC "pressured out", how to control/stabilize ? (reformatted to text/plain) To: Current FreeBSD X-Mailer: mail.ukr.net 5.0 Message-Id: <1391083826.948700370.cmzf8475@frv45.ukr.net> In-Reply-To: <52EA3459.3070300@FreeBSD.org> References: <1388839805.123581691.q97ijp8l@frv45.ukr.net> <52C93E4D.1050100@FreeBSD.org> <1389005433.815055146.2dcjke36@frv45.ukr.net> <52CA9963.1050507@FreeBSD.org> <1389676958.516993176.oq4lbgg7@frv45.ukr.net> <52D59E36.9040405@FreeBSD.org> <20140115102837.GA98983@hell.ukr.net> <52D66DB6.7030807@FreeBSD.org> <1390900795.258244476.v35k1338@frv45.ukr.net> <52EA3459.3070300@FreeBSD.org> MIME-Version: 1.0 Received: from atz@ukr.net by frv45.ukr.net; Thu, 30 Jan 2014 14:18:51 +0200 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jan 2014 12:18:58 -0000 Dear Andriy and FreeBSD community, L2ARC temporarily turned off by setting secondarycache=none everywhere it was enabled, so no more leak for one particular day. Here's the top header: last pid: 89916; load averages: 2.49, 2.91, 2.89 up 5+19:21:42 14:09:12 561 processes: 2 running, 559 sleeping CPU: 5.7% user, 0.0% nice, 14.0% system, 1.0% interrupt, 79.3% idle Mem: 23G Active, 1017M Inact, 98G Wired, 1294M Cache, 3285M Buf, 1997M Free ARC: 69G Total, 3498M MFU, 59G MRU, 53M Anon, 1651M Header, 4696M Other Swap: Here's the calculated vmstat -z (mean all of the allocations, which exceeds 100*1024^2 printed): UMA Slabs: 199,915M VM OBJECT: 207,354M 32: 205,558M 64: 901,122M 128: 215,211M 256: 242,262M 4096: 2316,01M range_seg_cache: 205,396M zio_buf_512: 1103,31M zio_buf_16384: 15697,9M zio_data_buf_16384: 348,297M zio_data_buf_24576: 129,352M zio_data_buf_32768: 104,375M zio_data_buf_36864: 163,371M zio_data_buf_53248: 100,496M zio_data_buf_57344: 105,93M zio_data_buf_65536: 101,75M zio_data_buf_73728: 111,938M zio_data_buf_90112: 104,414M zio_data_buf_106496: 100,242M zio_data_buf_131072: 61652,5M dnode_t: 3203,98M dmu_buf_impl_t: 797,695M arc_buf_hdr_t: 1498,76M arc_buf_t: 105,802M zfs_znode_cache: 352,61M zio_data_buf_131072 (61652M) + zio_buf_16384 (15698M) = 77350M easily exceeds ARC total (70G) Here's the same calculations from exact the same system where L2 was disabled before reboot: last pid: 63407; load averages: 2.35, 2.71, 2.73 up 8+19:42:54 14:17:33 527 processes: 1 running, 526 sleeping CPU: 4.8% user, 0.0% nice, 6.6% system, 1.1% interrupt, 87.4% idle Mem: 21G Active, 1460M Inact, 99G Wired, 1748M Cache, 3308M Buf, 952M Free ARC: 87G Total, 4046M MFU, 76G MRU, 37M Anon, 2026M Header, 4991M Other Swap: and the vmstat -z filtered: UMA Slabs: 208,004M VM OBJECT: 207,392M 32: 172,831M 64: 752,226M 128: 210,024M 256: 244,204M 4096: 2249,02M range_seg_cache: 245,711M zio_buf_512: 1145,25M zio_buf_16384: 15170,1M zio_data_buf_16384: 422,766M zio_data_buf_20480: 120,742M zio_data_buf_24576: 148,641M zio_data_buf_28672: 112,848M zio_data_buf_32768: 117,375M zio_data_buf_36864: 185,379M zio_data_buf_45056: 103,168M zio_data_buf_53248: 105,32M zio_data_buf_57344: 122,828M zio_data_buf_65536: 109,25M zio_data_buf_69632: 100,406M zio_data_buf_73728: 126,844M zio_data_buf_77824: 101,086M zio_data_buf_81920: 100,391M zio_data_buf_86016: 101,391M zio_data_buf_90112: 112,836M zio_data_buf_98304: 100,688M zio_data_buf_102400: 106,543M zio_data_buf_106496: 108,875M zio_data_buf_131072: 63190,5M dnode_t: 3437,36M dmu_buf_impl_t: 840,62M arc_buf_hdr_t: 1870,88M arc_buf_t: 114,942M zfs_znode_cache: 353,055M Everything seems within ARC total range. We will try patch attached within few days and will come back with the result. Thank you for your help. > on 28/01/2014 11:28 Vladimir Sharun said the following: > > Dear Andriy and FreeBSD community, > > > > After applying this path one of the systems runs fine (disk subsystem load low to moderate > > - 10-20% busy sustained), > > > > Then I saw this patch was merged to the HEAD and we apply it to the one of the systems > > with moderate to high disk load: 30-60% busy (11.0-CURRENT #7 r261118: Fri Jan 24 17:25:08 EET 2014) > > > > Within 4 days we experiencing the same leak(?) as without patch: > > > > last pid: 53841; load averages: 4.47, 4.18, 3.78 up 3+16:37:09 11:24:39 > > 543 processes: 6 running, 537 sleeping > > CPU: 8.7% user, 0.0% nice, 14.6% system, 1.4% interrupt, 75.3% idle > > Mem: 22G Active, 1045M Inact, 98G Wired, 1288M Cache, 3284M Buf, 2246M Free > > ARC: 73G Total, 3763M MFU, 62G MRU, 56M Anon, 1887M Header, 4969M Other > > Swap: > > > > The ARC is populated within 30mins under load to the max (90Gb) then start decreasing. > > > > The delta between Wiread and ARC total start growing from typical 10-12Gb without L2 enabled > > to the 25Gb with L2 enabled and counting (4 hours ago was 22Gb delta). > > First, have you checked that vmstat -z output contains the same anomaly as for > in your original report? > > If yes, the please try to reproduce the problem with the following debugging patch: > http://people.freebsd.org/~avg/l2arc-b_tmp_cdata-diag.patch > Please make sure to compile your kernel (and modules) with INVARIANTS. > > -- > Andriy Gapon > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"