Date: Thu, 29 May 2008 16:24:33 -0400 From: Robert Blayzor <rblayzor.bulk@inoc.net> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: freebsd-stable@freebsd.org Subject: Re: Sockets stuck in FIN_WAIT_1 Message-ID: <0C827F66-09CE-476D-86E9-146AB255926B@inoc.net> In-Reply-To: <200805291930.m4TJUeGX025815@apollo.backplane.com> References: <B42F9BDF-1E00-45FF-BD88-5A07B5B553DC@inoc.net> <1A19ABA2-61CD-4D92-A08D-5D9650D69768@mac.com> <23C02C8B-281A-4ABD-8144-3E25E36EDAB4@inoc.net> <483DE2E0.90003@FreeBSD.org> <B775700E-7494-42C1-A9B2-A600CE176ACB@inoc.net> <483E36CE.3060400@FreeBSD.org> <483E3C26.3060103@paradise.net.nz> <483E4657.9060906@FreeBSD.org> <483EA513.4070409@earthlink.net> <96AFE8D3-7EAC-4A4A-8EFF-35A5DCEC6426@inoc.net> <483EAED1.2050404@FreeBSD.org> <200805291912.m4TJCG56025525@apollo.backplane.com> <14DA211A-A9C5-483A-8CB9-886E5B19A840@inoc.net> <200805291930.m4TJUeGX025815@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On May 29, 2008, at 3:30 PM, Matthew Dillon wrote: > It is quite possible that the other ends of the connection are > still > live and that the issue could very well be a timeout setting in the > server config file instead of something in the TCP stack. I think we're onto something here, but for some reason it doesn't make any sense. I have keepalives turned OFF in Apache: In the OS I have: net.inet.tcp.keepidle: 900000 net.inet.tcp.keepintvl: 30000 net.inet.tcp.keepinit: 75000 net.inet.tcp.always_keepalive: 1 I see FIN_WAIT_1 connections collecting like crazy again. I managed to investigate one that's been hung around for several minutes. 1.1.1.1 = our server ip 2.2.2.2 = remote client ip [mirror0:/usr/local/etc/apache22] netstat -an | grep 2.2.2.2 tcp4 0 25861 1.1.1.1.80 2.2.2.2.33379 FIN_WAIT_1 When I tcpdump this, I see something sending ack's back and forth every 60 seconds, but what? Apache? I'm not sure why. I don't see any timeouts in Apache for ~60 seconds. As you can see, sometimes we send an ack, but never see a reply. I'm gathering the OS level keepalives don't come into play because this session is not considered idle? 0:13:07.640426 IP 1.1.1.1.80 > 2.2.2.2.33379: . 4208136508:4208136509(1) ack 1471446041 win 520 <nop,nop,timestamp 3019088951 5004131> 20:13:07.736505 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5022148 3019088951> 20:14:07.702647 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019148951 5022148> 20:15:07.764920 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019208951 5022148> 20:15:07.860988 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5058183 3019208951> 20:16:07.827262 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019268951 5058183> 20:16:07.923341 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5076200 3019268951> 20:17:07.889690 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019328951 5076200> 20:17:07.984770 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5094217 3019328951> 20:18:07.952167 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019388951 5094217> 20:18:08.048249 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5112234 3019388951> 20:19:08.014715 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019448951 5112234> 20:19:08.110803 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5130252 3019448951> 20:20:08.077321 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019508951 5130252> 20:21:08.139995 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019568951 5130252> 20:21:08.236063 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5166286 3019568951> 20:22:08.202435 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019628951 5166286> 20:22:08.297499 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5184303 3019628951> 20:23:08.264631 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019688951 5184303> 20:23:08.360700 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5202321 3019688951> I'm finding several of these sessions doing the same exact thing.... -- Robert Blayzor, BOFH INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0C827F66-09CE-476D-86E9-146AB255926B>