Date: Thu, 11 Jul 2013 14:47:03 +0200 From: Andre Oppermann <andre@freebsd.org> To: Steven Hartland <killing@multiplay.co.uk> Cc: stable@freebsd.org, re@freebsd.org, nonesuch@longcount.org, Alfred Perlstein <alfred@ixsystems.com> Subject: Re: status of autotuning freebsd for 9.2 Message-ID: <51DEA947.6060605@freebsd.org> In-Reply-To: <F438F05872C24151BE51F1F8ED3871C2@multiplay.co.uk> References: <51D90B9B.9080209@ixsystems.com> <51D92826.1070707@freebsd.org> <51D9B24B.8070303@ixsystems.com> <51DACE93.9050608@freebsd.org> <51DE6255.5000304@freebsd.org> <F438F05872C24151BE51F1F8ED3871C2@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11.07.2013 11:08, Steven Hartland wrote: > ----- Original Message ----- From: "Andre Oppermann" <andre@freebsd.org> > >> On 08.07.2013 16:37, Andre Oppermann wrote: >>> On 07.07.2013 20:24, Alfred Perlstein wrote: >>>> On 7/7/13 1:34 AM, Andre Oppermann wrote: >>>>> Can you help me with with testing? >>>>> >>>> Yes. Please give me your proposed changes and I'll stand up a machine and give feedback. >>> >>> http://people.freebsd.org/~andre/mfc-autotuning-20130708.diff >> >> Any feedback from testers on this? The MFC window is closing soon. > > Few things I've noticed most of which look like issues against the original > patch and not the MFC but worth mentioning. > > 1. You've introduced a new tunable kern.maxmbufmem which is autosized but > doesnt seem to be exposed via a sysctl so it looks like there is no way > to determine what its actually set to? Good point. I've made it global and exposed as kern.ipc.maxmbufmem (RDTUN). > 2. There's a missmatch between the tuneable kern.ipc.nmbufs in tunable_mbinit > and the sysctl kern.ipc.nmbuf i.e. no 's'. That's a typo, fixed. > 3. Should kern.maxmbufmem be kern.ipc.maxmbufmem to sit along side all of > the other sysctls? Yes, see above. > 4. style issues: > * @@ -178,11 +202,13 @@ > ... > if (newnmbjumbo9 > nmbjumbo9&& Thanks. All fixed in r253204. > Finally out of interest what made us arrive at the various defaults for each > type as it looks like the ratios have changed? Before it was an arbitrary mess. Mbufs were not limited at all and the others to some random multiple of maxusers with the net limit ending up at some 25,000 mbuf clusters by default. Now default overall limit is set at 50% of all available min(physical, kmem_map) memory to prevent mbufs from monopolizing kernel memory and leave some space for other kernel structures and buffers as well as user-space programs. It can be raised to 3/4 of available memory by the tunable. 2K and 4K (page size) mbuf clusters can each go up to 25% of this mbuf memory. The former is dominantly used on the receive path and the latter in the send path. 9K and 16K jumbo mbuf clusters can each go up to 12.5% of mbuf memory. They are only used in the receive path if large jumbo MTUs on a network interface are active. Both are special in that their memory is contiguous in KVM and physical memory. This becomes problematic due to memory fragmentation after a short amount of heavy system use. I hope to deprecate them for 10.0. Network interfaces should use 4K clusters instead and chain them together for larger packets. All modern NICs support that. Only the early and limited DMA engines without scatter-gather capabilities required contiguous physical memory. They are long gone by now. The limit for mbufs itselfs is 12.5% of mbuf memory and should be at least as many as the sum of the cluster types. Each cluster requires an mbuf to which it is attached. Two examples on the revised mbuf sizing limits: 1GB KVM: 512MB limit for mbufs 419,430 mbufs 65,536 2K mbuf clusters 32,768 4K mbuf clusters 9,709 9K mbuf clusters 5,461 16K mbuf clusters 16GB RAM: 8GB limit for mbufs 33,554,432 mbufs 1,048,576 2K mbuf clusters 524,288 4K mbuf clusters 155,344 9K mbuf clusters 87,381 16K mbuf clusters These defaults should be sufficient for even the most demanding network loads. For additional information see: http://svnweb.freebsd.org/changeset/base/243631 -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51DEA947.6060605>