Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 29 Jan 2014 15:11:21 -0800
From:      Navdeep Parhar <nparhar@gmail.com>
To:        Adrian Chadd <adrian@freebsd.org>
Cc:        Garrett Wollman <wollman@csail.mit.edu>, FreeBSD Net <freebsd-net@freebsd.org>
Subject:   Re: Big physically contiguous mbuf clusters
Message-ID:  <20140129231121.GA18434@ox>
In-Reply-To: <CAJ-VmomC5Ge3JwfUsgMrJ_rGqiYxfxR4wWzn5A-KAu7HBsueMw@mail.gmail.com>
References:  <21225.20047.947384.390241@khavrinen.csail.mit.edu> <CAJ-VmomC5Ge3JwfUsgMrJ_rGqiYxfxR4wWzn5A-KAu7HBsueMw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 29, 2014 at 02:21:21PM -0800, Adrian Chadd wrote:
> Hi,
> 
> On 29 January 2014 10:54, Garrett Wollman <wollman@csail.mit.edu> wrote:
> > Resolved: that mbuf clusters longer than one page ought not be
> > supported.  There is too much physical-memory fragmentation for them
> > to be of use on a moderately active server.  9k mbufs are especially
> > bad, since in the fragmented case they waste 3k per allocation.
> 
> I've been wondering whether it'd be feasible to teach the physical
> memory allocator about >page sized allocations and to create zones of
> slightly more physically contiguous memory.

I think this would be very useful.  For example, a zone_jumbo32 would
hit a sweet spot -- enough to fit 3 jumbo frames and some loose change
for metadata.  I'd like to see us improve our allocators and VM system
to work better with larger contiguous allocations, rather than
deprecating the larger zones.  It seems backwards to push towards
smaller allocation units when installed physical memory in a typical
system continues to rise.

Allocating 3 x 4K instead of 1 x 9K for a jumbo means 3x the number of
vtophys translations, 3x the phys_addr/len traffic on the PCIe bus
(scatter list has to be fed to the chip and now it's 3x what it has to
be), 3x the number of "wrapper" mbuf allocations (one for each 4K
cluster) which will then be stitched together to form a frame, etc. etc.

Regards,
Navdeep

> 
> For servers with lots of memory we could then keep these around and
> only dip into them for temporary allocations (eg not VM pages that may
> be held for some unknown amount of time.)
> 
> Question is - can we enforce that kind of behaviour?
> 
> 
> 
> -a
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140129231121.GA18434>