Date: Sun, 30 Jun 2002 18:13:08 -0400 From: Don Bowman <don@sandvine.com> To: 'Doug White' <dwhite@resnet.uoregon.edu>, Don Bowman <don@sandvine.com> Cc: "'freebsd-stable@freebsd.org'" <freebsd-stable@FreeBSD.ORG> Subject: RE: number mbufs / cluster Message-ID: <FE045D4D9F7AED4CBFF1B3B813C85337676648@mail.sandvine.com>
next in thread | raw e-mail | index | archive | help
Doug White wrote: > On Sat, 29 Jun 2002, Don Bowman wrote: > > > > > I have an application where there is lots of connections, > > but packet sizes aren't that large. I'm running into the > > problem where I need an enormous amount of clusters, > > but the number of mbufs needed isn't as big: > > > > $ netstat -m > > 141164/141200/512000 mbufs in use (current/peak/max): > > 141163 mbufs allocated to data > > 1 mbufs allocated to fragment reassembly queue headers > > 127997/128000/128000 mbuf clusters in use (current/peak/max) > > 291300 Kbytes allocated to network (10% of mb_map in use) > > 121192 requests for memory denied > > holy cow. How much memory does this thing have and what did you do to > abuse it soo much? :-) I didn't think you could jack those values that > high and not get panics. > I tried changing MCLSHIFT to 10. Although this would seem like the right thing to do, some trouble ensues... Some things like nfs don't seem to work right (just UDP traffic as far as I could see). (10 would yield a 1K cluster). Are there any assumptions somewhere that a cluster is >= MTU size? Also, how does one change the number of mbufs down? The sysctl value is read-only. # Mbufs seems to be a constant 4x # clusters. My socket buffer sizes are set to 8k (xmit and rx). I can't really make them any smaller for this application. I'm trying to manage about ~40K sockets. I have 1GB of physical memory. I'm running into the limits shown above at about 4K-5K sockets (but each doing 10x the work I expect in the 40K case). Also, I'm not really sure why the number of clusters I'm using is so large. There shouldn't be any packet loss on this GE link (at least not due to congestion), I'm only running it at about 2-7 hundred Mbps (depending on the input I feed it). Data flow is fairly bidirectional. O yeah, as to the comment about panics... Sure, I'm getting them, which is why I'm doing the tuning :) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FE045D4D9F7AED4CBFF1B3B813C85337676648>