From owner-freebsd-net@FreeBSD.ORG Tue Mar 25 01:04:20 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 619E68F; Tue, 25 Mar 2014 01:04:20 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 1024D15F; Tue, 25 Mar 2014 01:04:19 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEAFTVMFODaFve/2dsb2JhbABZg0FXgwe4KoZkUYE4dIIlAQEBAwEBAQEgBCcgCxsYAgINGQIjBgEJJgYIBwQBHAEDh0QDCQgNrEWbEg2HMheBKYs0gUwBARsBMweCb4FJBJVzaoMfizaFSYNJITGBBDk X-IronPort-AV: E=Sophos;i="4.97,724,1389762000"; d="scan'208";a="108794804" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 24 Mar 2014 21:04:18 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id D85A0B3F4B; Mon, 24 Mar 2014 21:04:18 -0400 (EDT) Date: Mon, 24 Mar 2014 21:04:18 -0400 (EDT) From: Rick Macklem To: Markus Gebert Message-ID: <1236110257.2510701.1395709458870.JavaMail.root@uoguelph.ca> In-Reply-To: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch> Subject: Re: 9.2 ixgbe tx queue hang MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Christopher Forgeron X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 01:04:20 -0000 Markus Gebert wrote: >=20 > On 24.03.2014, at 16:21, Christopher Forgeron > wrote: >=20 > > This is regarding the TSO patch that Rick suggested earlier. (With > > many > > thanks for his time and suggestion) > >=20 > > As I mentioned earlier, it did not fix the issue on a 10.0 system. > > It did > > make it less of a problem on 9.2, but either way, I think it's not > > needed, > > and shouldn't be considered as a patch for testing/etc. > >=20 > > Patching TSO to anything other than a max value (and by default the > > code > > gives it IP_MAXPACKET) is confusing the matter, as the packet > > length > > ultimately needs to be adjusted for many things on the fly like TCP > > Options, etc. Using static header sizes won't be a good idea. > >=20 > > Additionally, it seems that setting nic TSO will/may be ignored by > > code > > like this in sys/netinet/tcp_output.c: > >=20 > > 10.0 Code: > >=20 > > 780 if (len > tp->t_tsomax - hdrlen) > > { !! > > 781 len =3D tp->t_tsomax - > > hdrlen; !! > > 782 sendalot =3D > > 1; > > 783 } > >=20 > >=20 > > I've put debugging here, set the nic's max TSO as per Rick's patch > > ( set to > > say 32k), and have seen that tp->t_tsomax =3D=3D IP_MAXPACKET. It's > > being set > > someplace else, and thus our attempts to set TSO on the nic may be > > in vain. > >=20 > > It may have mattered more in 9.2, as I see the code doesn't use > > tp->t_tsomax in some locations, and may actually default to what > > the nic is > > set to. > >=20 > > The NIC may still win, I didn't walk through the code to confirm, > > it was > > enough to suggest to me that setting TSO wouldn't fix this issue. >=20 >=20 > I just applied Rick=E2=80=99s ixgbe TSO patch and additionally wanted to = be > able to easily change the value of hw_tsomax, so I made a sysctl out > of it. >=20 > While doing that, I asked myself the same question. Where and how > will this value actually be used and how comes that tcp_output() > uses that other value in struct tcpcb. >=20 > The only place tcpcb->t_tsomax gets set, that I have found so far, is > in tcp_input.c=E2=80=99s tcp_mss() function. Some subfunctions get called= : >=20 > tcp_mss() -> tcp_mss_update() -> tcp_maxmtu() >=20 > Then tcp_maxmtu() indeed uses the interface=E2=80=99s hw_tsomax value: >=20 > 1746 cap->tsomax =3D ifp->if_hw_tsomax; >=20 > It get=E2=80=99s passed back to tcp_mss() where it is set on the connect= ion > level which will be used in tcp_output() later on. >=20 > tcp_mss() gets called from multiple places, I=E2=80=99ll look into that > later. I will let you know if I find out more. >=20 >=20 > Markus >=20 Well, if tp->t_tsomax isn't set to a value of 65518, then the ixgbe.patch isn't doing what I thought it would. The only explanation I can think of for this is that there might be another net interface driver stacked on top of the ixgbe.c one and that the setting doesn't get propagated up. Does this make any sense? IP_MAXPACKET can't be changed from 65535, but I can see an argument for setting the default value of if_hw_tsomax to a smaller value. For example, in sys/net/if.c change it from: 657 if (ifp->if_hw_tsomax =3D=3D 0) 658 =09ifp->if_hw_tsomax =3D IP_MAXPACKET; to 657 if (ifp->if_hw_tsomax =3D=3D 0) 658 =09ifp->if_hw_tsomax =3D 65536 - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN)= ; This is a slightly smaller default which won't have much impact unless the hardware device can only handle 32 mbuf clusters for transmit of a segment and there are several of those. Christopher, can you do your test run with IP_MAXPACKET set to 65518, which should be the same as the above. If that gets rid of all the EFBIG error replies, then I think the above patch will have the same effect. Thanks, rick >=20 > > However, this is still a TSO related issue, it's just not one > > related to > > the setting of TSO's max size. > >=20 > > A 10.0-STABLE system with tso disabled on ix0 doesn't have a single > > packet > > over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer > > to > > increase confidence in this assertion, but I don't want to waste > > time on > > this when I could be logging problem packets on a system with TSO > > enabled. > >=20 > > Comments are very welcome.. > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org" > >=20 >=20 >=20