Date: Sat, 18 Dec 1999 14:58:09 -0600 (CST) From: Kevin Day <toasty@dragondata.com> To: rfg@monkeys.com (Ronald F. Guilmette) Cc: toasty@dragondata.com (Kevin Day), hackers@FreeBSD.ORG Subject: Re: Practical limit for number of TCP connections? Message-ID: <199912182058.OAA42531@celery.dragondata.com> In-Reply-To: <42829.945549667@monkeys.com> from "Ronald F. Guilmette" at Dec 18, 1999 12:41:07 PM
next in thread | previous in thread | raw e-mail | index | archive | help
Wow, thanks for such a detailed reply. :) > > > In message <199912181944.NAA68434@celery.dragondata.com>, you wrote: > > >What's the practical number of TCP connections per server? > > I've gotten over 8,000 at one time on one FreeBSD box. Yeah, best case, I've had several thousand myself, but not really doing anything. :) I'm going to try to come up with a test suite for this sorta thing, unless someone pipes in that it already exists. > The biggest memory issue for a box that's handling a lot of TCP con- > nections at a time is the socket I/O buffer sizes. Unless you take > steps, programatically, to reduce these, you will get (I think) one > 4KB buffer for input and another 4KB buffer for output. This is for > EACH active TCP connection. I hadn't considered this much buffering, but it makes sense. > > This can add up to a substantial amount of memory if you have a lot of > connections. > > The way to solve that is to include calls to setsockopt() in your server > that will have the effect of reducing the per-connection I/O buffer sizes > just after you accept() each new connection. Speaking of accepting... What's the upper limit on listen queues? Something around 64, correct? > > Quite a lot of memory (either virtual or real) will also get sucked up > *if* you have a separate and independent process handling each separate > connection. A simple experiment I did awhile back indicated that on > recent-vintage versions of FreeBSD, the absolute minimum per-process > overhead was 12KB. That is a *minimum*, e.g. for a process that contains > essentially no code and no data. But you will probably never see that in > practice, which is to say your minimum per-process overhead is going to > be bigger than that. Yeah, I don't plan on doing things the apache way. :) One process per client seems silly here, since nearly every client will be getting the exact same data. > The _clean_ way of doing it would be to write your multi-user server using > threads, and to assign one thread to each connection. If you can do that, > then the logic in the program becomes quite simple. Each thread just sits > there, blocked on a call to read(), until something comes in, and then it > just parses the command, does whatever it is supposed to do in response to > that command, and then goes back to the read() again. > > But as I understand it, there is not yet sufficient threads support in the > FreeBSD kernel to make this work well/properly. (I may perhaps be misinformed > about that, but that's what I have been told anyway.) I believe this is how ConferenceRoom works, so it seems ok, but I remember the comments that FreeBSD was their least preferred platform because of thread problems. > The other way is just have your server be a single thread/process, and to > have it keep one big list of all of the connections (i.e. socket fds) that > it has open at present. Then it just executes mail loop, over and over > again. At the top of the main look is one big honkin' call to select() > in which you find out which of your connections is ready to be read or > written. Then you go off and read/write those as appropriate, and then > just come back and do the big select() again. (You could do this using > calls to poll() instead of calls to select(), and that might be a bit > more efficient.) This is how traditional ircd's handle it, and how I was planning to. It seems the most efficient, especially since 99.999% of the clients will be receiving the same data, it'll make transmission pretty easy. > If you want a lot of connections, start by increasing the values of > "maxusers" and NMBCLUSTERS in your kernel config file. Then build and > install a new kernel. How much is enough for these parameters? Beats > me. If you find that you are running out of resources, increase them > some more and then try again. I really wish more of those options were dynamically tunable, instead of a magic formula that maxusers controls. :) Thanks again, Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199912182058.OAA42531>