Date: Wed, 16 Jan 2002 13:39:54 -0800 From: Terry Lambert <tlambert2@mindspring.com> To: Chad David <davidc@acns.ab.ca> Cc: current@freebsd.org Subject: Re: socket shutdown delay? Message-ID: <3C45F32A.5B517F7E@mindspring.com> References: <20020116070908.A803@colnta.acns.ab.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Chad David wrote: > Has anyone noticed (or fixed) a bug in -current where socket connections > on the local machine do not shutdown properly? During stress testing > I'm seeing thousands (2316 right now) of these: > > tcp4 0 0 192.168.1.2.8080 192.168.1.2.2215 FIN_WAIT_2 > tcp4 0 0 192.168.1.2.2215 192.168.1.2.8080 LAST_ACK > > Both the client and the server are dead, but the connections stay in this > state. > > I tested with the server on -current and the client on another box, and > all of the server sockets end up in TIME_WAIT. Is there something delaying > the last ack on local connections? A connection goes into FIN_WAIT_2 when it has received the ACK of the FIN, but not received a FIN (or sent an ACK) itself, thus permitting it to enter TIME_WAIT state for 2MSL before proceeding to the CLOSED state, as a result of a server initiated close. A connection goes into LAST_ACK when it has sent a FIN and not received the ACK of the FIN before proceeding to the CLOSED state, as a result of a client initiated close. Since it's showing IP addresses, you appear to be using real network connections, rather than loopback connections. There are basically several ways to cause this: 1) You have something on your network, like a dummynet, that is deteministically dropping the the ACK to the client when the server goes from FIN_WAIT_1, so that the server goes to CLOSING instead of going to FIN_WAIT_2 (client closes first), or the FIN in the other direction so that the server doesn't go to TIME_WAIT from FIN_WAIT_2 (server closes first). 2) You have intentionally disabled KEEPALIVE, so that a close results in an RST instead of a normal shutdown of the TCP connection (I can't tell if you are doing a real call to "shutdown(2)", or if you are just relying on the OS resource tracking behaviour that is implicit to "close(2)" (but only if you don't set KEEPALIVE, and have disabled the sysctl default of always doing KEEPALIVE on every connection). In this case, it's possible that the RST was lost on the wire, and since RSTs are not retransmitted, you have shot yourself in the foot. Note: You often see this type of foolish foot shooting when running MAST, WAST, or webbench, which try to factor out response speed and measure connection speed, so that they benchmark the server, not the FS or other OS latencies in the document delivery path (which is why these tools suck as real world benchmarks go). You could also cause this (unlikely) with a bad firewall rule. 3) You've exhausted your mbufs before you've exhausted the number of simultaneous connections you are permitted, because you have incorrectly tuned your kernel, and therefore all your connections are sitting in a starvation deadlock, waiting for packets that can never be sent because there are no mbufs available. 4) You've got local hacks that your aren't telling us about (shame on you!). 5) You have found an introduced bug in -current. Note: I personally think this one is unlikely. 6) Maybe something I haven't thought of... Note: I personally think this one is unlikely, too... ;^) See RFC 793 (or Stevens) for details on the state machine for both ends of the connection, and you will see how your machine got into this mess in the first place. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C45F32A.5B517F7E>