Date: Sat, 25 May 1996 18:58:42 -0600 From: Nate Williams <nate@sri.MT.net> To: "Karl Denninger, MCSNet" <karl@mcs.com> Cc: davidg@Root.COM, hackers@FreeBSD.ORG Subject: Re: Grrr.. is this is a FreeBSD problem (TIME_WAIT again) Message-ID: <199605260058.SAA22658@rocky.sri.MT.net> In-Reply-To: <m0uNTm7-000IDOC@venus.mcs.com> References: <199605252237.PAA23150@Root.COM> <m0uNTm7-000IDOC@venus.mcs.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>>>If the caller and callee are on the SAME machine, I get sockets in
>>> TIME_WAIT for 2 minutes each
....
> >
> > Based on what you've said thus far, it's working as it is supposed to.
> > There is a good discussion of the 2MSL wait ("TIME_WAIT") in "TCP/IP
> > Illustrated Volume 1", page 242, by W. Richard Stevens.
> > Core-team/Principal Architect, The FreeBSD Project
>
> I understand the purpose behind the 2MSL wait, but my understanding
> was that this was imposed for non-cleanly closed connections to
> prevent a collision (and possibly delivering data to the "wrong"
> client).
Nope, it's for *all* clients, even for 'successful' connections. It's
part of the specification.
> The problem is two-fold:
>
> 1) The TIME_WAIT sockets are no big deal in and of themselves (I have
> lots of Mbuf resources on the machine under consideration).....
>
> 2) BUT, once there get to be a dozen or so of these sockets in
> TIME_WAIT, a NEW connection trying to bind to the server (on the
> "connect()" call) gets blocked until one of the slots frees up!
Yep. This is expected behavior.
> For a high-volume transaction server this is murderous, as it means that the
> processing limit is (outstanding TIME_WAIT sockets MAX / 2) transactions per
> minute.
Tell me about it. I'm dealing with a system right now that the military
uses. (Shh, don't tell anyone.)
For sending data, it wants to use TCP sockets for 'reliability'.
However, someone had a screw loose. To send a 'packet' of data you must
do the following:
1) socket/connect
2) write out the data to the socket
3) close the socket
In this manner, they are guaranteeing (not really, but it's a pretty
good bet) that *all* the data will be sent in one shot *AND* that if you
have problems sending data you will know about it when the sockets
close.
The problems are many
- It assumes slow data rates, or large packets.
For the *original* purpose of the system, the former is true.
However, the military wants to use it for some of the new stuff
we're developing for them. We need to send out 14 packets/sec, so I
know have to play tricks with pre-buffering up data in order to get
around this and build *big* packets, and it's non-trivial.
- It assumes that the data is transferred if the close succeeds. This
is dependent on the how the OS handles data. You can play tricks with
the sockets to help, but there is no guarantee that the data
*actually* got to the other side even if the close succeeds.
- It assumes that a single write/read will give handle an entire packet.
Again, depending on the sending/receiving OS, the lower layers may
give you a partial packet read, *especially* if the data is
transferred over a long-haul line.
So, on my end I'm forced to set SOCK_REUSEADDR because I *must* have
multiple connections going to the remote site within the 2-minute time
period, but the remote site isn't setup that way. In any case, I'm
doing everything *exactly* as spec'd, including using shutdown() on the
sockets, but the TCP/IP protocol says the 2-minute time-out exists on
*all* stream sockets unless SO_REUSEADDR is set.
> I am being very careful to insure that (1) all the data is out of the pipe
> before I close it and (2) the stream is correctly closed in both server and
> client before either calls exit(). I would think that the system could
> (should?) immediately release the socket under these circumstances to
> prevent the blocking condition, or that the blocking condition shouldn't
> exist in the first place. It also doesn't explain why I don't get the same
> condition when I have the server and client on different machines.
??? I see it connecting combinations of FreeBSD, SCO, SunOS, and HP/UX
machines.
> I can't use UDP because the data going across the link is encrypted with a
> stream cipher that is position dependant (thus, a lost or corrupted packet
> would lead to the cipher getting hosed).
Unless you want to do the 'reliability' protocol by yourself. C'mon,
doing a sliding windows is something *everyone* should do. *grin*
> SO_REUSEADDR doesn't do me any good, as I'm not trying to reuse a tuple
> (again, I am allowing the system to grab a random port on the client
> side).
But the client is still 'connecting' to the same port on the remote
side, and thus *might* get data on that port back from the remote site.
(Why it does this for 'successful' sockets is probably explained in
Steven's book)
> How do you get around this if you need to have many (dozens or more) TCP
> connections made to a given client (and dropped) in a short period of time?
>
> It sounds to me like the goal is impossible to meet.
int on = 1;
/* Set it so we can re-use the local/remote port again quickly */
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (char *)&on, sizeof on);
Nate
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199605260058.SAA22658>