From owner-freebsd-stable@freebsd.org Tue Feb 12 19:18:32 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5975E14BD03B for ; Tue, 12 Feb 2019 19:18:32 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DB02A7637B; Tue, 12 Feb 2019 19:18:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x1CJIGTq073220 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 12 Feb 2019 20:18:17 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: lev@FreeBSD.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x1CJIGas053236 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 13 Feb 2019 02:18:16 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: 11.2-STABLE kernel wired memory leak To: lev@FreeBSD.org, Kurt Jaeger References: <201902121757.x1CHve0h056876@hergotha.csail.mit.edu> <7125e053-5adf-929a-bde6-a64fceae2aaa@grosbein.net> <8a9361fd-4701-4e33-33f0-e4800af7637c@grosbein.net> <20190212182915.GM2748@home.opsec.eu> <80e414af-058c-5f84-0f9b-36529d5e06b4@grosbein.net> <331bbb5f-6627-d73c-d174-0865234d296f@FreeBSD.org> Cc: freebsd-stable@freebsd.org, Garrett Wollman From: Eugene Grosbein Message-ID: <3d7f4ac4-67e4-b010-64c6-0d84a4451246@grosbein.net> Date: Wed, 13 Feb 2019 02:18:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <331bbb5f-6627-d73c-d174-0865234d296f@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: DB02A7637B X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; MX_INVALID(0.50)[cached]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.93)[-0.928,0]; IP_SCORE(-1.37)[ip: (-2.23), ipnet: 2a01:4f8::/29(-2.39), asn: 24940(-2.23), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 19:18:32 -0000 13.02.2019 1:50, Lev Serebryakov wrote: > I'm have same problem. > > According to top(1) I have 29G Wired, but only 17G Total ARC (12G > difference! System has 32G of RAM), and this statistic shows: > > 5487.5 zio_data_buf_524288 > 920.125 zio_data_buf_131072 > 626 zio_buf_131072 > 468 zio_data_buf_1048576 > 398.391 zio_buf_16384 > 305.464 dnode_t > 227.989 zio_buf_512 > 171.5 zio_data_buf_458752 > 141.75 zio_data_buf_393216 > 116.456 dmu_buf_impl_t > > So, more than 6G (!) is not used in ARC, but hold by ZFS anyway. dnode_t and dmu_buf_impl_t are parts of ZFS too, so these numbers represent about 9G, not 6G. Do you have/had some memory pressure here? Growth of swap usage?