Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 30 Jun 2002 19:20:48 -0400
From:      Don Bowman <don@sandvine.com>
To:        "'freebsd-stable@freebsd.org'" <freebsd-stable@FreeBSD.ORG>
Subject:   RE: number mbufs / cluster
Message-ID:  <FE045D4D9F7AED4CBFF1B3B813C8533767664B@mail.sandvine.com>

next in thread | raw e-mail | index | archive | help
 
Doug White wrote:

> On Sat, 29 Jun 2002, Don Bowman wrote:
> 
> >
> > I have an application where there is lots of connections,
> > but packet sizes aren't that large. I'm running into the
> > problem where I need an enormous amount of clusters,
> > but the number of mbufs needed isn't as big:
> >
> > $ netstat -m
> > 141164/141200/512000 mbufs in use (current/peak/max):
> >         141163 mbufs allocated to data
> >         1 mbufs allocated to fragment reassembly queue headers
> > 127997/128000/128000 mbuf clusters in use (current/peak/max)
> > 291300 Kbytes allocated to network (10% of mb_map in use)
> > 121192 requests for memory denied
> 
> holy cow. How much memory does this thing have and what did you do to
> abuse it soo much? :-) I didn't think you could jack those values that
> high and not get panics.
> 

I tried changing MCLSHIFT to 10. Although this would seem like
the right thing to do, some trouble ensues... Some things like nfs
don't seem to work right (just UDP traffic as far as I could see). 
(10 would yield a 1K cluster). Are there
any assumptions somewhere that a cluster is >= MTU size?
Also, how does one change the number of mbufs down? The sysctl
value is read-only. # Mbufs seems to be a constant 4x # clusters.

My socket buffer sizes are set to 8k (xmit and rx). I can't really
make them any smaller for this application. I'm trying to manage
about ~40K sockets. I have 1GB of physical memory. I'm running
into the limits shown above at about 4K-5K sockets (but each
doing 10x the work I expect in the 40K case).

Also, I'm not really sure why the number of clusters I'm using is
so large.

There shouldn't be any packet loss on this GE link (at least not
due to congestion), I'm only running it at about 2-7 hundred Mbps
(depending on the input I feed it). Data flow is fairly bidirectional.

O yeah, as to the comment about panics... Sure, I'm getting them,
which is why I'm doing the tuning :)


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FE045D4D9F7AED4CBFF1B3B813C8533767664B>