Date: Fri, 17 Sep 2010 14:30:14 +0200 From: Andre Oppermann <andre@freebsd.org> To: Andriy Gapon <avg@freebsd.org> Cc: freebsd-hackers@freebsd.org, Jeff Roberson <jeff@freebsd.org> Subject: Re: zfs + uma Message-ID: <4C935F56.4030903@freebsd.org> In-Reply-To: <4C93236B.4050906@freebsd.org> References: <4C93236B.4050906@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 17.09.2010 10:14, Andriy Gapon wrote: > > I've been investigating interaction between zfs and uma for a while. > You might remember that there is a noticeable fragmentation in zfs uma zones > when uma use is not enabled for actual data/metadata buffers. > > I also noticed that when uma use is enabled for data/metadata buffers > (zio.use_uma=1) amount of memory reserved in free items of zfs uma zones becomes > really huge. And this is despite the fact that the vast majority of the > data/metadata zone have items with sizes that are multiples of page size. > This couldn't really be because of fragmentation. > > Further checks show that the free items are accumulated in per-cpu cache > buckets. uz_count for those buckets starts with 1, but over time, during bursts > of activity, it grows up to maximum of 128. > Problem with those buckets is that they are not drained on low memory conditions > and uz_count never goes down. > > So, after a while, I observe about 300 free items (on a mere two core system) > cached in 4 per-cpu buckets for a single zone with 128KB item size. > That's 30MB right there. > For all data and metadata zones the number goes as high as 500MB on my machine > with 4GB physical RAM. > This seems like a bit too much to me. > > Although keeping free items around improves performance, it does consume memory > too. And the fact that that memory is not freed on lowmem condition makes the > situation worse. Interesting. We may run into related issues with excessive mbuf (cluster) caching in the per-cpu buckets as well. Having a general solutions for that is appreciated. Maybe the size of the free per-cpu buckets should be specified when setting up the UMA zone. Of certain frequently re-used elements we may want to cache more, other less. -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C935F56.4030903>