From owner-freebsd-stable@FreeBSD.ORG Thu Feb 14 05:10:38 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 440F2830; Thu, 14 Feb 2013 05:10:38 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-pa0-f53.google.com (mail-pa0-f53.google.com [209.85.220.53]) by mx1.freebsd.org (Postfix) with ESMTP id 1A9A9648; Thu, 14 Feb 2013 05:10:37 +0000 (UTC) Received: by mail-pa0-f53.google.com with SMTP id bg4so1144210pad.40 for ; Wed, 13 Feb 2013 21:10:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=L/EdWVFJ8sRbCS/Jib2kKuVEju7L2bk/D3RoN4088gA=; b=ZkFCfAkM8XdlyyUy5kOzNTmQex4x7gcDpdyHriw9GTlA4vQFHDC7dYDAQcOFKOtgjr qA4YhJa8V8zLlSj/OhAHEbrVSbHDd61kpIism2ncwGUrcaF1mny0eUJbuyjjYy0NpmNX DQBNBMBlgJAoZsYVOPaarjr1awE8+yCLdvEcTIUYiDwMnIqGLsRL5YCXVlUmWpP0qQpJ NzF4d7gH7Cap9qSig0sinrm9Jvx5pr3xO83BcFEAv71yewJ/xO+MGFkdrT/iRvwSin4P SyVPqciXB1Xkr1KvMhaucKTYRiLNaFwmLJRYER1DOQdtzWT/xQ5DC5tVHWyJYmjeOz2U wqNA== MIME-Version: 1.0 X-Received: by 10.68.255.161 with SMTP id ar1mr257916pbd.17.1360818637115; Wed, 13 Feb 2013 21:10:37 -0800 (PST) Received: by 10.67.2.65 with HTTP; Wed, 13 Feb 2013 21:10:36 -0800 (PST) In-Reply-To: <20130214013723.GB2945@michelle.cdnetworks.com> References: <96AE8BD1-79C2-4743-854F-B8386C54E4A1@lafn.org> <511B6B21.5030606@rdtc.ru> <20130213130059.GA57337@icarus.home.lan> <20130214013723.GB2945@michelle.cdnetworks.com> Date: Wed, 13 Feb 2013 21:10:36 -0800 Message-ID: Subject: Re: Unusual TCP/IP Packet Size From: Kevin Oberman To: pyunyh@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Jeremy Chadwick , freebsd-stable@freebsd.org, Eugene Grosbein , yongari@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Feb 2013 05:10:38 -0000 On Wed, Feb 13, 2013 at 5:37 PM, YongHyeon PYUN wrote: > On Wed, Feb 13, 2013 at 05:00:59AM -0800, Jeremy Chadwick wrote: >> On Wed, Feb 13, 2013 at 05:29:53PM +0700, Eugene Grosbein wrote: >> > 13.02.2013 17:25, Doug Hardie ??????????: >> > > Monitoring a tcpdump between two systems, a FreeBSD 9.1 system has t= he following interface: >> > > >> > > msk0: flags=3D8843 metric 0 = mtu 1500 >> > > options=3Dc011b >> > > ether 00:11:2f:2a:c7:03 >> > > inet 10.0.1.199 netmask 0xffffff00 broadcast 10.0.1.255 >> > > inet6 fe80::211:2fff:fe2a:c703%msk0 prefixlen 64 scopeid 0x1 >> > > nd6 options=3D29 >> > > media: Ethernet autoselect (100baseTX ) >> > > status: active >> > > >> > > >> > > It sent the following packet: (data content abbreviated) >> > > >> > > 02:14:42.081617 IP 10.0.1.199.443 > 10.0.1.2.61258: Flags [P.], seq = 930:4876, ack 846, win 1040, options [nop,nop,TS val 401838072 ecr 92011018= 3], length 3946 >> > > 0x0000: 4500 0f9e ea89 4000 4006 2a08 0a00 01c7 E.....@.@.*..... >> > > 0x0010: 0a00 0102 01bb ef4a ece1 680b ae37 1bbc .......J..h..7.. >> > > 0x0020: 8018 0410 3407 0000 0101 080a 17f3 8ff8 ....4...??????. >> > > >> > > >> > > The indicated packet length is 3946 and the load of data shown is th= at size. The MTU on both interfaces is 1500. The receiving system receive= d 3 packets. There is a router and switch between them. One of them fragm= ented that packet. This is part of a SSL/TLS exchange and one side or the o= ther is hanging on this and just dropping the connection. I suspect the pa= cket size is the issue. ssldump complains about the packet too and stops mo= nitoring. Could this possibly be related to the hardware checksums? >> > >> > You have TSO enabled on the interface, so large outgoing TCP packet is= pretty normal. >> > It will be split by the NIC. Disable TSO with ifconfig if it interfere= s with your ssldump. >> >> This is not the behaviour I see with em(4) on a 82573E with all defaults >> used (which includes TSO4). Note that Doug is using msk(4). >> >> I can provide packet captures on both ends of a LAN segment using both >> tcpdump (on the FreeBSD side) and Wireshark (on the Windows side) that >> show a difference in behaviour compared to what Doug sees. > > This is strange. tcpdump sees a (big) TCP segment right before > controller actually transmits it. So if TSO is active for the TCP > segment, you should see a series of small TCP packets on receiver > side(i.e. 3 TCP packets in Doug's case). If you don't see a big TCP > segment with tcpdump on TX path, probably TSO was not used for the > TCP segment. > It's possible for controller to corrupt the TCP segment during > segmentation but Doug's tcpdump looks completely normal to me since > tcpdump sees the segment before TCP segmentation. > >> >> What I see on the FreeBSD side with tcpdump is repeated "bad-len 0" >> messages for payloads which are chunked or segmented as a result of TSO. >> I do not see a 1:1 ratio of "bad-len" entries to chunked payloads; I >> only see one "bad-len" entry for all chunks (up until the next ACK or >> PSH+ACK of course). >> > > I vaguely recall that some users reported similar TSO issues on > various drivers. The root cause of the issue was not identified > though. Personally I couldn't reproduce the issue at that time. > It could be a driver or network stack bug. Beware TSO. It can significantly improve throughput on high speed networks, but it really has issues. TSO segments the data and transmits all of them back-to-back with no delay beyond IFG (the 802.3 mandated space between frames) TSO does not understand congestion control. If there is congestion and TSO sends several frames in a row, it is entirely possible that a queue is full or getting close enough to full to start dropping packets and these segmented frames are excellent candidates. --=20 R. Kevin Oberman, Network Engineer E-mail: kob6558@gmail.com