From owner-freebsd-stable@freebsd.org Thu Jun 7 06:39:42 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08711FFACE0 for ; Thu, 7 Jun 2018 06:39:42 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E4CD80A96 for ; Thu, 7 Jun 2018 06:39:40 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id w576dV4C069178 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 7 Jun 2018 16:39:36 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id w576dO3D096179 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 7 Jun 2018 16:39:25 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id w576dOGW096178 for freebsd-stable@freebsd.org; Thu, 7 Jun 2018 16:39:24 +1000 (AEST) (envelope-from peter) Date: Thu, 7 Jun 2018 16:39:24 +1000 From: Peter Jeremy To: freebsd-stable@freebsd.org Subject: ZFS+find(1) wiring all RAM Message-ID: <20180607063924.GG1697@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="24zk1gE8NUlDmwG9" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.9.5 (2018-04-13) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jun 2018 06:39:42 -0000 --24zk1gE8NUlDmwG9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've noticed that 11-stable/amd64 has been wiring seemingly excessive amounts of RAM for some time (the problem goes back at least 6 months). This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap errors killing processes on a low-memory system. I'm not sure when it started by it seems to hawe gotten worse between r331535 and r334494. I can see the "excessive wired memory" on my main home system with 32GB RAM but haven't seen it completely run out of RAM. After some gentle use and a nightly run, there is 10GB more wired RAM than ARC. My "low memory" system is a Google GCE f1-micro instance[*] (600MB RAM) with about 723k inodes used and the following ZFS tuning: vfs.zfs.arc_max=3D"128M" vfs.zfs.arc_meta_limit=3D"50M" vfs.zfs.arc_min=3D"25M" The following numbers were gatherer by looking at top(1). Running r334494, after booting, to multi-user, the system has about 187MB wired (94MB ARC). If I then run /etc/periodic/security/100.chksetuid, wired RAM increases to about 580MB, with 380MB ARC, dropping to 467MB and 217MB ARC when the script exits (this is still nearly twice arc_max). Free memory can drop to <10KB whilst the find(1) is running. I have several issues with this behaviour: 0) ARC usage can significantly exceed arc_max. I understand that arc_max is a soft limit but IMO, 3x is unreasonable - especially when the system is under extreme memory pressuse. 1) Significant amounts of wired memory are in use but I can't find anything in "vmstat -mz" that would explain where it's going. Does anyone have any suggestions for digging into this? [1] I get the same behaviour using a VBox instance with similar dimensioning and the same tuning) --=20 Peter Jeremy --24zk1gE8NUlDmwG9 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAlsY0xxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQFVhAAnUQsXhEenLbBMtduCz29j89IXGzPUeyk7z0oJDKT8esBcxZWBxBfsxlR hbM785z5z+cRfUzhMtja3LEJpNkrT0zFEVwwCjAMINi3U15J82A+uT9z4xWb1AwZ ftq91T+LPNNMyQUVu2FfClrXTIVEILu408ez5j+MKEYRH6GNc4Y9jGM4fUxPPmXp /PC3y6OEEqqnUwVOqoyNxEK1VhMoA8Y7mgO1XRyyhLcz6wta8mpRC5MwitTlhRAE F4n0rrfKqT9UQy35UWpmukK2rtwt5oyMFG+qucMPD53qnuCuzEOYdrIKqKJ+kSQm ymHWqmjkLUqPN7B2tl8/DaIwZanzDoBOX2vkTv//ZZw1pWFhQFCW31XSz9IEAYSB JUNv8K1R4DEG6mBGfW6lAkOhOaQx7hPsQTHaE1YPhuk8oJkYq67A5YwPD3VrCBSk 8juQk5XmiXms0SKDBRHWJPtPU+TZRRzBKFyUZvJPrSF1mbhaQwL2CMoex6WXVPRu GiXOQK6irmQnsZ48I/wA5PoSpOI0lQfJhA6tRMan7O2HeQApZPvPpPiJe+NtW8jC zsikDKcS45izNnkvLpEe1sd++CI5biHkXcDYL3zFvjulmjlVcfkIYlH2mW4Y17e4 v22Ppu0vBkKv15vjh47ikocwJgsBaV6TDN77PsY28/tD/1jR5Uo= =I5tr -----END PGP SIGNATURE----- --24zk1gE8NUlDmwG9--