From owner-freebsd-stable@freebsd.org Fri Mar 29 14:38:46 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04F52156B0C4 for ; Fri, 29 Mar 2019 14:38:46 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.49.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C42296889 for ; Fri, 29 Mar 2019 14:38:45 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from mather.gromit23.net (c-98-244-101-97.hsd1.va.comcast.net [98.244.101.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id EFBC3212; Fri, 29 Mar 2019 10:38:43 -0400 (EDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: 11.2-STABLE kernel wired memory leak From: Paul Mather In-Reply-To: Date: Fri, 29 Mar 2019 10:38:43 -0400 Cc: FreeBSD Stable Content-Transfer-Encoding: quoted-printable Message-Id: <89A58F93-12DD-4F66-BA08-7A8C462459AC@gromit.dlib.vt.edu> References: To: Robert Schulze X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 2C42296889 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=vt.edu (policy=none) X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[vt.edu : No valid SPF, No valid DKIM,none]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.50)[ip: (-4.21), ipnet: 128.173.0.0/16(-2.11), asn: 1312(-1.12), country: US(-0.07)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: chumby.dlib.vt.edu]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; R_SPF_NA(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[97.101.244.98.zen.spamhaus.org : 127.0.0.10]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1312, ipnet:128.173.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2019 14:38:46 -0000 On Mar 29, 2019, at 5:52 AM, Robert Schulze wrote: > Hi, >=20 > I just want to report a similar issue here with 11.2-RELEASE-p8. >=20 > The affected machine has 64 GB ram and does daily backups from several > machines in the night, at daytime there a parallel runs of clamav on a > specific dataset. >=20 > One symtom is basic I/O-performance: After upgrading from 11.1 to 11.2 > backup times have increased, and are even still increasing. After one > week of operation, backup times have doubled - without having changed > anything else. >=20 > Then there is this wired memory and way too lazy reclaim of memory for > user processes: The clamav scans start at 10:30 and get swapped out > immediatly. Although vfs.zfs.arc_max=3D48G, wired is at 62 GB before = the > scans and it takes about 10 minutes for the scan processes to actually > run on system ram, not swap. >=20 > There is obviously something broken, as there are several threads with > similar observations. I am using FreeBSD 12 (both -RELEASE and -STABLE) and your comment about = "way too lazy reclaim of memory" struck a chord with me. On one system = I regularly have hundreds of MB identified as being in the "Laundry" = queue but FreeBSD hardly ever seems to do the laundry. I see the same = total for days. When does FreeBSD decide to do its laundry? Right now "top" is showing = 835M in "Laundry" and the system is >99% idle. How can I get the system to be more proactive about doing its = housekeeping when it has idle time? It would be much nicer to have it do laundry during a calm time rather = than get all flustered when it's down to its last pair of socks = (metaphorically speaking) and page even more stuff out to swap. :-) Cheers, Paul.