From owner-freebsd-stable@freebsd.org Tue Feb 12 21:02:17 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFBB514D0E63 for ; Tue, 12 Feb 2019 21:02:16 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B36A382444; Tue, 12 Feb 2019 21:02:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 3685714B57; Tue, 12 Feb 2019 21:02:15 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:71eb:28be:92e:f722]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 25213BB71; Wed, 13 Feb 2019 00:02:08 +0300 (MSK) Date: Wed, 13 Feb 2019 00:02:07 +0300 From: Lev Serebryakov Reply-To: Lev Serebryakov Organization: FreeBSD Message-ID: <1294306375.20190213000207@serebryakov.spb.ru> To: Eugene Grosbein , Kurt Jaeger CC: freebsd-stable@freebsd.org, Garrett Wollman Subject: Re: 11.2-STABLE kernel wired memory leak In-Reply-To: <3d7f4ac4-67e4-b010-64c6-0d84a4451246@grosbein.net> References: <201902121757.x1CHve0h056876@hergotha.csail.mit.edu> <7125e053-5adf-929a-bde6-a64fceae2aaa@grosbein.net> <8a9361fd-4701-4e33-33f0-e4800af7637c@grosbein.net> <20190212182915.GM2748@home.opsec.eu> <80e414af-058c-5f84-0f9b-36529d5e06b4@grosbein.net> <331bbb5f-6627-d73c-d174-0865234d296f@FreeBSD.org> <3d7f4ac4-67e4-b010-64c6-0d84a4451246@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: B36A382444 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.93 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.94)[-0.938,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Feb 2019 21:02:17 -0000 Hello Eugene, Tuesday, February 12, 2019, 10:18:09 PM, you wrote: >> I'm have same problem. >> >> According to top(1) I have 29G Wired, but only 17G Total ARC (12G >> difference! System has 32G of RAM), and this statistic shows: >> >> 5487.5 zio_data_buf_524288 >> 920.125 zio_data_buf_131072 >> 626 zio_buf_131072 >> 468 zio_data_buf_1048576 >> 398.391 zio_buf_16384 >> 305.464 dnode_t >> 227.989 zio_buf_512 >> 171.5 zio_data_buf_458752 >> 141.75 zio_data_buf_393216 >> 116.456 dmu_buf_impl_t >> >> So, more than 6G (!) is not used in ARC, but hold by ZFS anyway. > dnode_t and dmu_buf_impl_t are parts of ZFS too, > so these numbers represent about 9G, not 6G. > Do you have/had some memory pressure here? Growth of swap usage? I don't have memory pressure right now, but according to my previous experience, ARC will not grow anymore even under heavy disk load (I don't have vfs.zfs.arc_max set). Before new ARC (vfs.zfs.abd_scatter_enabled) I had typically ALL memory occuped by ARC, Wired memory was almost exactly equal to ARC, and ARC hitrate was higher (but I have not exact numbers, unfortunately). Now I have "vfs.zfs.abd_scatter_enabled=0", but still Wired is much larger that ARC under any disk load (it is mostly torrent box). -- Best regards, Lev mailto:lev@FreeBSD.org