Date: Wed, 13 Feb 2019 01:14:54 +0700 From: Eugene Grosbein <eugen@grosbein.net> To: Garrett Wollman <wollman@bimajority.org>, freebsd-stable@freebsd.org Subject: Re: 11.2-STABLE kernel wired memory leak Message-ID: <7125e053-5adf-929a-bde6-a64fceae2aaa@grosbein.net> In-Reply-To: <201902121757.x1CHve0h056876@hergotha.csail.mit.edu> References: <d8c7abc0-3ba1-40e4-22b1-1b30d28ced14@grosbein.net> <201902121757.x1CHve0h056876@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
13.02.2019 0:57, Garrett Wollman wrote:
> In article <d8c7abc0-3ba1-40e4-22b1-1b30d28ced14@grosbein.net>
> eugen@grosbein.net writes:
>
>> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel
>> wired memory over 81 days uptime
>> out of 8GB total RAM.
>
> Not a whole lot of evidence yet, but anecdotally I'm seeing the same
> thing on some huge-memory NFS servers running releng/11.2. They seem
> to run fine for a few weeks, then mysteriously start swapping
> continuously, a few hundred pages a second. The continues for hours
> at a time, and then stops just as mysteriously. Over time the total
> memory dedicated to ZFS ARC goes down but there's no decrease in wired
> memory. I've tried disabling swap, but this seems to make the server
> unstable. I have yet to find any obvious commonality (aside from the
> fact that these are all large-memory NFS servers which don't do much
> of anything else -- the only software running on them is related to
> managing and monitoring the NFS service).
I started to understand the issue. FreeBSD 11 has uma(9) zone allocator
for kernel subsystems and vmstat -z shows some stats for UMA zones.
When some subsystem using UMA frees its memory (including networking mbufs or ZFS ARC),
some kernel memory blocks are moved from USED to FREE category
inside corresponding UMA zone (see vmstat -z again) but this memory stays
unavailable to other consumers including userland applications
until pagedaemon reclaims this "FREE" memory back to global free pool.
This part seems to be broken in 11.2-STABLE.
Use following command to see how much memory is wasted in your case:
vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7125e053-5adf-929a-bde6-a64fceae2aaa>