Date: Thu, 7 Jun 2018 16:39:24 +1000 From: Peter Jeremy <peter@rulingia.com> To: freebsd-stable@freebsd.org Subject: ZFS+find(1) wiring all RAM Message-ID: <20180607063924.GG1697@server.rulingia.com>
next in thread | raw e-mail | index | archive | help
--24zk1gE8NUlDmwG9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I've noticed that 11-stable/amd64 has been wiring seemingly excessive amounts of RAM for some time (the problem goes back at least 6 months). This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap errors killing processes on a low-memory system. I'm not sure when it started by it seems to hawe gotten worse between r331535 and r334494. I can see the "excessive wired memory" on my main home system with 32GB RAM but haven't seen it completely run out of RAM. After some gentle use and a nightly run, there is 10GB more wired RAM than ARC. My "low memory" system is a Google GCE f1-micro instance[*] (600MB RAM) with about 723k inodes used and the following ZFS tuning: vfs.zfs.arc_max=3D"128M" vfs.zfs.arc_meta_limit=3D"50M" vfs.zfs.arc_min=3D"25M" The following numbers were gatherer by looking at top(1). Running r334494, after booting, to multi-user, the system has about 187MB wired (94MB ARC). If I then run /etc/periodic/security/100.chksetuid, wired RAM increases to about 580MB, with 380MB ARC, dropping to 467MB and 217MB ARC when the script exits (this is still nearly twice arc_max). Free memory can drop to <10KB whilst the find(1) is running. I have several issues with this behaviour: 0) ARC usage can significantly exceed arc_max. I understand that arc_max is a soft limit but IMO, 3x is unreasonable - especially when the system is under extreme memory pressuse. 1) Significant amounts of wired memory are in use but I can't find anything in "vmstat -mz" that would explain where it's going. Does anyone have any suggestions for digging into this? [1] I get the same behaviour using a VBox instance with similar dimensioning and the same tuning) --=20 Peter Jeremy --24zk1gE8NUlDmwG9 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAlsY0xxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQFVhAAnUQsXhEenLbBMtduCz29j89IXGzPUeyk7z0oJDKT8esBcxZWBxBfsxlR hbM785z5z+cRfUzhMtja3LEJpNkrT0zFEVwwCjAMINi3U15J82A+uT9z4xWb1AwZ ftq91T+LPNNMyQUVu2FfClrXTIVEILu408ez5j+MKEYRH6GNc4Y9jGM4fUxPPmXp /PC3y6OEEqqnUwVOqoyNxEK1VhMoA8Y7mgO1XRyyhLcz6wta8mpRC5MwitTlhRAE F4n0rrfKqT9UQy35UWpmukK2rtwt5oyMFG+qucMPD53qnuCuzEOYdrIKqKJ+kSQm ymHWqmjkLUqPN7B2tl8/DaIwZanzDoBOX2vkTv//ZZw1pWFhQFCW31XSz9IEAYSB JUNv8K1R4DEG6mBGfW6lAkOhOaQx7hPsQTHaE1YPhuk8oJkYq67A5YwPD3VrCBSk 8juQk5XmiXms0SKDBRHWJPtPU+TZRRzBKFyUZvJPrSF1mbhaQwL2CMoex6WXVPRu GiXOQK6irmQnsZ48I/wA5PoSpOI0lQfJhA6tRMan7O2HeQApZPvPpPiJe+NtW8jC zsikDKcS45izNnkvLpEe1sd++CI5biHkXcDYL3zFvjulmjlVcfkIYlH2mW4Y17e4 v22Ppu0vBkKv15vjh47ikocwJgsBaV6TDN77PsY28/tD/1jR5Uo= =I5tr -----END PGP SIGNATURE----- --24zk1gE8NUlDmwG9--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180607063924.GG1697>