From owner-freebsd-net Fri Oct 18 14:20:24 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 060B937B401 for ; Fri, 18 Oct 2002 14:20:18 -0700 (PDT) Received: from mail.wrs.com (unknown-1-11.windriver.com [147.11.1.11]) by mx1.FreeBSD.org (Postfix) with ESMTP id 107A043E6A for ; Fri, 18 Oct 2002 14:20:17 -0700 (PDT) (envelope-from stephenm@windriver.com) Received: from brisbane (brisbane [147.11.38.9]) by mail.wrs.com (8.9.3/8.9.1) with ESMTP id OAA25087; Fri, 18 Oct 2002 14:19:16 -0700 (PDT) Message-Id: <200210182119.OAA25087@mail.wrs.com> X-Mailer: exmh version 2.0.2 2/24/98 To: cjclark@alum.mit.edu Cc: net@freebsd.org, stephenm@mail.wrs.com Subject: Re: Strange FTP TCP Window Problem In-reply-to: Your message of "Fri, 18 Oct 2002 10:10:55 PDT." <20021018171055.GA45449@blossom.cjclark.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 18 Oct 2002 14:20:07 -0700 From: Stephen Macmanus Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org This type of behavior typically indicates a problem with the application itself, above the socket layer, not TCP or the other protocols. The window size continually decreases as data arrives since TCP is storing the incoming data in the receive buffer, but the target application is not reading it. I have seen cases where this behavior occurs due to subtle failures like CPU starvation due to scheduling priorities or incorrect mutual exclusion. I have also encountered simpler causes, such as creating a socket for the purpose of sending data that uses a port which also matches incoming data. In that case, the application simply never read the unexpected data from the socket. Setting the receive buffer to zero allowed TCP to discard the data and avoid the problem. This possibility may not apply in your case, since the client successfully handles other file transfers, but in my experience the application is the most likely cause of the problem. Stephen -- ------------------ Stephen Macmanus #include Technical Staff Wind River Networks stephenm@wrs.com > This has got me stumped. I must be missing something obvious. I am > trying to download some files via FTP. There are four files. Three > download just fine, one just won't go. Since the three work fine, I > assume it's not PORT vs. PASV, NAT, or other firewall issues, that is, > the usual suspects have been eliminated. But what is really strange is > that it looks like _my client_ is the one causing the headaches. > > Here is a tcpdump(8) of the data connection. Keep an eye on the window > values my client is sending back, > > 09:15:11.097800 a.b.c.136.29826 > d.e.f.17.18747: S [tcp sum ok] 2410623756:2410623756(0) win 16384 (ttl 64, id 50031, len 64) > 09:15:11.118335 d.e.f.17.18747 > a.b.c.136.29826: S [tcp sum ok] 2932757104:2932757104(0) ack 2410623757 win 16384 (ttl 56, id 24108, len 60) > 09:15:11.118443 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 1 win 16500 (ttl 64, id 43640, len 52) > 09:15:11.156424 d.e.f.17.18747 > a.b.c.136.29826: . 1:501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24111, len 552) > 09:15:11.159468 d.e.f.17.18747 > a.b.c.136.29826: . 501:1001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24112, len 552) > 09:15:11.227371 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 1001 win 15500 [tos 0x8] (ttl 64, id 61006, len 52) > 09:15:11.258205 d.e.f.17.18747 > a.b.c.136.29826: . 1001:1501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24113, len 552) > 09:15:11.261225 d.e.f.17.18747 > a.b.c.136.29826: . 1501:2001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24114, len 552) > 09:15:11.279923 d.e.f.17.18747 > a.b.c.136.29826: . 2001:2501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24115, len 552) > 09:15:11.427395 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 2501 win 14000 [tos 0x8] (ttl 64, id 40196, len 52) > 09:15:11.457865 d.e.f.17.18747 > a.b.c.136.29826: . 2501:3001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24116, len 552) > 09:15:11.461105 d.e.f.17.18747 > a.b.c.136.29826: . 3001:3501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24117, len 552) > 09:15:11.480485 d.e.f.17.18747 > a.b.c.136.29826: . 3501:4001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24118, len 552) > 09:15:11.483702 d.e.f.17.18747 > a.b.c.136.29826: . 4001:4501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24119, len 552) > 09:15:11.627707 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 4501 win 12000 [tos 0x8] (ttl 64, id 44302, len 52) > 09:15:11.648692 d.e.f.17.18747 > a.b.c.136.29826: . 4501:5001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24120, len 552) > 09:15:11.659731 d.e.f.17.18747 > a.b.c.136.29826: . 5001:5501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24121, len 552) > 09:15:11.670448 d.e.f.17.18747 > a.b.c.136.29826: . 5501:6001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24122, len 552) > 09:15:11.673581 d.e.f.17.18747 > a.b.c.136.29826: . 6001:6501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24123, len 552) > 09:15:11.692780 d.e.f.17.18747 > a.b.c.136.29826: . 6501:7001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24124, len 552) > 09:15:11.827393 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 7001 win 9500 [tos 0x8] (ttl 64, id 38068, len 52) > 09:15:11.855900 d.e.f.17.18747 > a.b.c.136.29826: . 7001:7501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24125, len 552) > 09:15:11.860077 d.e.f.17.18747 > a.b.c.136.29826: . 7501:8001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24126, len 552) > 09:15:11.878834 d.e.f.17.18747 > a.b.c.136.29826: . 8001:8501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24127, len 552) > 09:15:11.881856 d.e.f.17.18747 > a.b.c.136.29826: . 8501:9001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24128, len 552) > 09:15:11.892382 d.e.f.17.18747 > a.b.c.136.29826: . 9001:9501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24129, len 552) > 09:15:11.903415 d.e.f.17.18747 > a.b.c.136.29826: . 9501:10001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24130, len 552) > 09:15:12.027385 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 10001 win 6500 [tos 0x8] (ttl 64, id 60342, len 52) > 09:15:12.052757 d.e.f.17.18747 > a.b.c.136.29826: . 10001:10501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24131, len 552) > 09:15:12.063589 d.e.f.17.18747 > a.b.c.136.29826: . 10501:11001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24132, len 552) > 09:15:12.074515 d.e.f.17.18747 > a.b.c.136.29826: . 11001:11501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24133, len 552) > 09:15:12.078151 d.e.f.17.18747 > a.b.c.136.29826: . 11501:12001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24134, len 552) > 09:15:12.096930 d.e.f.17.18747 > a.b.c.136.29826: . 12001:12501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24135, len 552) > 09:15:12.099980 d.e.f.17.18747 > a.b.c.136.29826: . 12501:13001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24136, len 552) > 09:15:12.110647 d.e.f.17.18747 > a.b.c.136.29826: . 13001:13501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24137, len 552) > 09:15:12.227383 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 13501 win 3000 [tos 0x8] (ttl 64, id 51576, len 52) > 09:15:12.264837 d.e.f.17.18747 > a.b.c.136.29826: . 13501:14001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24138, len 552) > 09:15:12.268095 d.e.f.17.18747 > a.b.c.136.29826: . 14001:14501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24139, len 552) > 09:15:12.278653 d.e.f.17.18747 > a.b.c.136.29826: . 14501:15001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24140, len 552) > 09:15:12.289849 d.e.f.17.18747 > a.b.c.136.29826: . 15001:15501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24141, len 552) > 09:15:12.293592 d.e.f.17.18747 > a.b.c.136.29826: . 15501:16001(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24142, len 552) > 09:15:12.427402 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16001 win 500 [tos 0x8] (ttl 64, id 41001, len 52) > 09:15:17.202100 d.e.f.17.18747 > a.b.c.136.29826: . 16001:16501(500) ack 1 win 16384 [tos 0x8] (ttl 56, id 24157, len 552) > 09:15:17.227476 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 59052, len 52) > 09:15:22.186435 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 24183, len 53) > 09:15:22.186625 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 55942, len 52) > 09:15:30.194393 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 24221, len 53) > 09:15:30.195913 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 14020, len 52) > 09:15:46.198335 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 24333, len 53) > 09:15:46.198583 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 16087, len 52) > 09:16:18.201164 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 24500, len 53) > 09:16:18.201409 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 30338, len 52) > 09:17:18.194176 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 25061, len 53) > 09:17:18.194430 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 20615, len 52) > 09:18:18.193763 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 25447, len 53) > 09:18:18.194005 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 12107, len 52) > 09:19:18.186418 d.e.f.17.18747 > a.b.c.136.29826: . [tcp sum ok] 16501:16502(1) ack 1 win 16384 [tos 0x8] (ttl 56, id 25862, len 53) > 09:19:18.187970 a.b.c.136.29826 > d.e.f.17.18747: . [tcp sum ok] 1:1(0) ack 16501 win 0 [tos 0x8] (ttl 64, id 51084, len 52) > > And this just hangs here forever. > > I can't see why the client is refusing to accept more data. Like I > said, I'll bet I'm not seeing something obvious, but all of the SEQ > and ACK numbers look good to me. I should note that this isn't just an > issue with the FreeBSD ftp client. I get the same result with Windows, > Solaris, Cygwin, and OpenBSD too. > > Why is it happening and how do I get around this? > -- > Crist J. Clark | cjclark@alum.mit.edu > | cjclark@jhu.edu > http://people.freebsd.org/~cjc/ | cjc@freebsd.org > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-net" in the body of the message -- ------------------ Stephen Macmanus #include Technical Staff Wind River Networks stephenm@wrs.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message