Date: Sat, 18 Dec 1999 12:41:07 -0800 From: "Ronald F. Guilmette" <rfg@monkeys.com> To: Kevin Day <toasty@dragondata.com> Cc: hackers@FreeBSD.ORG Subject: Re: Practical limit for number of TCP connections? Message-ID: <42829.945549667@monkeys.com> In-Reply-To: Your message of Sat, 18 Dec 1999 13:44:59 -0600. <199912181944.NAA68434@celery.dragondata.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <199912181944.NAA68434@celery.dragondata.com>, you wrote:
>What's the practical number of TCP connections per server?
I've gotten over 8,000 at one time on one FreeBSD box.
>Is there an easy guideline for how {much} ram the kernel will be taking per
>connection/route/socket/fd/etc?
Not that I am aware of.
The biggest memory issue for a box that's handling a lot of TCP con-
nections at a time is the socket I/O buffer sizes. Unless you take
steps, programatically, to reduce these, you will get (I think) one
4KB buffer for input and another 4KB buffer for output. This is for
EACH active TCP connection.
This can add up to a substantial amount of memory if you have a lot of
connections.
The way to solve that is to include calls to setsockopt() in your server
that will have the effect of reducing the per-connection I/O buffer sizes
just after you accept() each new connection.
Quite a lot of memory (either virtual or real) will also get sucked up
*if* you have a separate and independent process handling each separate
connection. A simple experiment I did awhile back indicated that on
recent-vintage versions of FreeBSD, the absolute minimum per-process
overhead was 12KB. That is a *minimum*, e.g. for a process that contains
essentially no code and no data. But you will probably never see that in
practice, which is to say your minimum per-process overhead is going to
be bigger than that.
If this part of the equation (i.e. per-process overhead) causes you grief
(as it did in my case) then you might consider writing your server as a
sort of multi-user monitor sort-of thing, i.e. a single process that will
service multiple connections. (That's what I did.)
There are _potentially_ a couple of ways of doing this, but at the present
time, and with current FreeBSD kernels, you probably don't have a lot of
choice, and I think there's only one way of doing it.
The _clean_ way of doing it would be to write your multi-user server using
threads, and to assign one thread to each connection. If you can do that,
then the logic in the program becomes quite simple. Each thread just sits
there, blocked on a call to read(), until something comes in, and then it
just parses the command, does whatever it is supposed to do in response to
that command, and then goes back to the read() again.
But as I understand it, there is not yet sufficient threads support in the
FreeBSD kernel to make this work well/properly. (I may perhaps be misinformed
about that, but that's what I have been told anyway.)
The other way is just have your server be a single thread/process, and to
have it keep one big list of all of the connections (i.e. socket fds) that
it has open at present. Then it just executes mail loop, over and over
again. At the top of the main look is one big honkin' call to select()
in which you find out which of your connections is ready to be read or
written. Then you go off and read/write those as appropriate, and then
just come back and do the big select() again. (You could do this using
calls to poll() instead of calls to select(), and that might be a bit
more efficient.)
The only real problem with doing things this way is that you have to diddle
some things to make sure the bit arrays that you pass to select() are big
enough to handle the maximum number of connections you ever anticipate
having to service/maintain at one time. Ideally, all you really have to
do in order to get this to happen is to redefine the pre-processor symbol
FD_SETSIZE to your max connections number before you include <sys/types.h>,
but then your are stuck selecting some other compile-time fixed number
(in place of the default 1024), and personally, I prefer to use a somewhat
different approach that allows me to use a dynamic number, i.e. one I get
from querying the kernel for the current hard limit on simultaneously open
fds.
>My next interview will be Sunday, but to a much smaller audience than
>normal, so I'll be able to do some experimenting. Can anyone recommend
>specific things to watch for, wrt limits and memory use? I'll be watching
>vmstat carefully, at least.
If you want a lot of connections, start by increasing the values of
"maxusers" and NMBCLUSTERS in your kernel config file. Then build and
install a new kernel. How much is enough for these parameters? Beats
me. If you find that you are running out of resources, increase them
some more and then try again.
After installing your new kernel, you'll probably want to add commands
like the following to your /etc/rc.local file:
# Increase the max # of open files, systemwide
/sbin/sysctl -w kern.maxfiles=16384
# Increase the max # of open sockets, systemwide (use only on older kernels)
#/sbin/sysctl -w kern.somaxconn=16384
# Increase max # of open files per process
/sbin/sysctl -w kern.maxfilesperproc=16384
Last but not least, make sure that whatever account your server will run
under is associated with some login class (see /etc/login.conf) that has
an appropriately high setting for openfiles-cur and/or openfiles-max.
If you would like to see an example of a very simple multi-connection server
that runs as a single process (written in C) as described above, let me know.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42829.945549667>