From owner-freebsd-net@FreeBSD.ORG Tue Sep 3 19:27:36 2013 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7D34E317; Tue, 3 Sep 2013 19:27:36 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4975721E3; Tue, 3 Sep 2013 19:27:35 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.7/8.14.7) with ESMTP id r83JRYxu019517; Tue, 3 Sep 2013 12:27:34 -0700 (PDT) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.7/8.14.7/Submit) id r83JRYoV019516; Tue, 3 Sep 2013 12:27:34 -0700 (PDT) (envelope-from david) Date: Tue, 3 Sep 2013 12:27:34 -0700 From: David Wolfskill To: FreeBSD Net Subject: Re: TSO and FreeBSD vs Linux Message-ID: <20130903192734.GA19406@albert.catwhisker.org> References: <520A6D07.5080106@freebsd.org> <5214F506.3070706@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3V7upXqbjpZ4EhLz" Content-Disposition: inline In-Reply-To: <5214F506.3070706@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Sep 2013 19:27:36 -0000 --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > On 13.08.2013 19:29, Julian Elischer wrote: > > I have been tracking down a performance embarrassment on AMAZON EC2 and= have found it I think. > > Our OS cousins over at Linux land have implemented some interesting beh= aviour when TSO is in use. >=20 > There used to be a different problem with EC2 and FreeBSD TSO. The Xen h= ypervisor > doesn't like large 64K TSO bursts we generate, the drivers drops the whol= e TSO chain, > TCP gets upset and turns off TSO alltogether leaving the connection going= at one > packet a time as in the old days. > ... My apologies for jumping in so late -- I'm not subscribed to -net@. At work, I received a new desktop machine a few months ago; here's a recent history of what it has been running: FreeBSD 9.2-PRERELEASE #4 r254801M/254827:902501: Sun Aug 25 05:15:29 PDT = 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 FreeBSD 9.2-PRERELEASE #5 r255066M/255091:902503: Sat Aug 31 11:58:53 PDT = 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 FreeBSD 9.2-PRERELEASE #5 r255104M/255115:902503: Sun Sep 1 05:02:12 PDT = 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 Now, I like to have a "private playground" for doing things with machines, so I make use of both em(4) NICs on the machine: em0 connects to the rest of the work network; em1 is connected to a switch I brought in from home, and to which I connect "other things" (such as my laptop). And because I'm fairly comfortable with them, I use IPFW & natd. For some folks here, none of that should come as a surprise. :-}) For reference, the em(4) devices in question are: em0@pci0:0:25:0: class=3D0x020000 card=3D0x060d15d9 chip=3D0x10ef808= 6 rev=3D0x06 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82578DM Gigabit Network Connection' and em1@pci0:3:0:0: class=3D0x020000 card=3D0x060d15d9 chip=3D0x10d38086 rev=3D= 0x00 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82574L Gigabit Network Connection' I noticed that when I tried to write files to NFS, I could write small files OK, but larger ones seemed to ... hang. Note: We don't use jumbo frames. (Work IT is convinced that they don't help. I'm trying to better-understand their reasoning.) Further poking around showed that (under the above conditions): * natd CPU% was climbing as more of the file was copied, up to 2^21 bytes. (At that point, nothing further was saved on NFS.) * dhcpd CPU% was also climbing. I tried killing that, but doing so didn't affect the other results. (Killing natd made connectivity cease, given the IPFW rules in effect.) * Performing a tcpdump while trying to copy a file of length 117709618 showed lots of TCP retransmissions. In fact, I'd hazard that every TCP packet was getting retransmitted. * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. * "sysctl net.inet.tcp.tso" showed "1" -- enabled. As soon as I issued "sudo net.inet.tcp.tso=3D0" ... the copy worked without a hitch or a whine. And I was able to copy all 117709618 bytes, not just 2097152 (2^21). Is the above expected? It came rather as a surprise to me. Peace, david --=20 David H. Wolfskill david@catwhisker.org Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --3V7upXqbjpZ4EhLz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.21 (FreeBSD) iEYEARECAAYFAlImOCUACgkQmprOCmdXAD1PlwCfRI35ikbwABwsF9L8S91GH+LM 0K4AnRYK8kglQA2RzY5R39gSnP/NmYKI =/Nw8 -----END PGP SIGNATURE----- --3V7upXqbjpZ4EhLz--