From owner-freebsd-stable Tue Mar 19 10:22:45 2002 Delivered-To: freebsd-stable@freebsd.org Received: from haruchai.esc.pike.il.us (esc.adams.net [216.138.0.234]) by hub.freebsd.org (Postfix) with SMTP id 1F6D237B4B4 for ; Tue, 19 Mar 2002 10:20:22 -0800 (PST) Received: (qmail 23039 invoked by uid 501); 19 Mar 2002 18:19:44 -0000 To: freebsd-stable@FreeBSD.ORG Subject: Re: Network stalls with 4.5 References: From: Aaron Baugher Date: 19 Mar 2002 12:19:44 -0600 In-Reply-To: Message-ID: Lines: 73 User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG To add to the previous info, here's a chunk of a tcpdump. I logged in through ssh, and ran 'top' to generate some traffic. My home machine (Mandrake Linux, connected with a modem) is 11.22.33.44, and the FreeBSD machine having the problem is 55.66.77.88. (I should stress that the problem occurs from several different client machines and OS's, so it's definitely caused at the server end.) 09:41:14.331731 55.66.77.88.22 > 11.22.33.44.4257: P 32772:32912(140) ack 161 win 65535 [tos 0x10] 09:41:14.609531 11.22.33.44.4257 > 55.66.77.88.22: . ack 32912 win 51360 (DF) [tos 0x10] 09:41:16.341753 55.66.77.88.22 > 11.22.33.44.4257: P 32912:33028(116) ack 161 win 65535 [tos 0x10] 09:41:16.625846 11.22.33.44.4257 > 55.66.77.88.22: . ack 33028 win 51360 (DF) [tos 0x10] 09:41:18.351793 55.66.77.88.22 > 11.22.33.44.4257: P 33028:33168(140) ack 161 win 65535 [tos 0x10] 09:41:18.611113 11.22.33.44.4257 > 55.66.77.88.22: . ack 33168 win 51360 (DF) [tos 0x10] 09:41:20.361833 55.66.77.88.22 > 11.22.33.44.4257: P 33168:33324(156) ack 161 win 65535 [tos 0x10] 09:41:20.628234 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] Up to this point it looks pretty normal. The server sends my machine a packet, and my machine sends back an ack. Then about here's where it stalled on me: 09:41:22.371874 55.66.77.88.22 > 11.22.33.44.4257: P 33324:33472(148) ack 161 win 65535 [tos 0x10] 09:41:23.371716 55.66.77.88.22 > 11.22.33.44.4257: P 33324:33472(148) ack 161 win 65535 [tos 0x10] That's the same packet sent twice, right? 09:41:24.381831 55.66.77.88.22 > 11.22.33.44.4257: P 33472:33628(156) ack 161 win 65535 [tos 0x10] 09:41:24.629822 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] 09:41:25.371763 55.66.77.88.22 > 11.22.33.44.4257: P 33324:33628(304) ack 161 win 65535 [tos 0x10] Now here's the same number (33324) as the doubled packet above, but with a different second number and size. 09:41:26.391896 55.66.77.88.22 > 11.22.33.44.4257: P 33628:33744(116) ack 161 win 65535 [tos 0x10] 09:41:26.630980 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] 09:41:28.401934 55.66.77.88.22 > 11.22.33.44.4257: P 33744:33868(124) ack 161 win 65535 [tos 0x10] 09:41:28.663837 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] 09:41:29.371842 55.66.77.88.22 > 11.22.33.44.4257: P 33324:33868(544) ack 161 win 65535 [tos 0x10] Now here it is yet again, but with an even larger size. 09:41:30.411974 55.66.77.88.22 > 11.22.33.44.4257: P 33868:34024(156) ack 161 win 65535 [tos 0x10] 09:41:30.664541 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] And about here somewhere is where it started moving normally again. 09:41:32.422153 55.66.77.88.22 > 11.22.33.44.4257: P 34024:34180(156) ack 161 win 65535 [tos 0x10] 09:41:32.665336 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] 09:41:34.432024 55.66.77.88.22 > 11.22.33.44.4257: P 34180:34336(156) ack 161 win 65535 [tos 0x10] 09:41:34.666565 11.22.33.44.4257 > 55.66.77.88.22: . ack 33324 win 51360 (DF) [tos 0x10] 09:41:34.666621 55.66.77.88.22 > 11.22.33.44.4257: P 33324:34336(1012) ack 161 win 65535 [tos 0x10] 09:41:35.147334 11.22.33.44.4257 > 55.66.77.88.22: . ack 34336 win 51360 (DF) [tos 0x10] So the packet numbered 33324 was sent several times, with the size increasing every time, and my system sending back acks for each one except that initial quick double packet. Here are all the sysctl changes I've made so far to try to fix the problem: net.inet.tcp.rfc1323: 0 net.inet.tcp.sendspace: 32768 net.inet.tcp.recvspace: 32768 net.inet.tcp.delayed_ack: 0 net.inet.tcp.path_mtu_discovery: 0 net.inet.tcp.newreno: 0 net.inet.tcp.syncookies: 0 None of those have made any difference. Any other ideas? Thanks, Aaron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message