Date: Mon, 24 Mar 2014 21:04:18 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Markus Gebert <markus.gebert@hostpoint.ch> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Garrett Wollman <wollman@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, Christopher Forgeron <csforgeron@gmail.com> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <1236110257.2510701.1395709458870.JavaMail.root@uoguelph.ca> In-Reply-To: <0BC10908-2081-45AC-A1C8-14220D81EC0A@hostpoint.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Markus Gebert wrote:
>=20
> On 24.03.2014, at 16:21, Christopher Forgeron <csforgeron@gmail.com>
> wrote:
>=20
> > This is regarding the TSO patch that Rick suggested earlier. (With
> > many
> > thanks for his time and suggestion)
> >=20
> > As I mentioned earlier, it did not fix the issue on a 10.0 system.
> > It did
> > make it less of a problem on 9.2, but either way, I think it's not
> > needed,
> > and shouldn't be considered as a patch for testing/etc.
> >=20
> > Patching TSO to anything other than a max value (and by default the
> > code
> > gives it IP_MAXPACKET) is confusing the matter, as the packet
> > length
> > ultimately needs to be adjusted for many things on the fly like TCP
> > Options, etc. Using static header sizes won't be a good idea.
> >=20
> > Additionally, it seems that setting nic TSO will/may be ignored by
> > code
> > like this in sys/netinet/tcp_output.c:
> >=20
> > 10.0 Code:
> >=20
> > 780 if (len > tp->t_tsomax - hdrlen)
> > { !!
> > 781 len =3D tp->t_tsomax -
> > hdrlen; !!
> > 782 sendalot =3D
> > 1;
> > 783 }
> >=20
> >=20
> > I've put debugging here, set the nic's max TSO as per Rick's patch
> > ( set to
> > say 32k), and have seen that tp->t_tsomax =3D=3D IP_MAXPACKET. It's
> > being set
> > someplace else, and thus our attempts to set TSO on the nic may be
> > in vain.
> >=20
> > It may have mattered more in 9.2, as I see the code doesn't use
> > tp->t_tsomax in some locations, and may actually default to what
> > the nic is
> > set to.
> >=20
> > The NIC may still win, I didn't walk through the code to confirm,
> > it was
> > enough to suggest to me that setting TSO wouldn't fix this issue.
>=20
>=20
> I just applied Rick=E2=80=99s ixgbe TSO patch and additionally wanted to =
be
> able to easily change the value of hw_tsomax, so I made a sysctl out
> of it.
>=20
> While doing that, I asked myself the same question. Where and how
> will this value actually be used and how comes that tcp_output()
> uses that other value in struct tcpcb.
>=20
> The only place tcpcb->t_tsomax gets set, that I have found so far, is
> in tcp_input.c=E2=80=99s tcp_mss() function. Some subfunctions get called=
:
>=20
> tcp_mss() -> tcp_mss_update() -> tcp_maxmtu()
>=20
> Then tcp_maxmtu() indeed uses the interface=E2=80=99s hw_tsomax value:
>=20
> 1746 cap->tsomax =3D ifp->if_hw_tsomax;
>=20
> It get=E2=80=99s passed back to tcp_mss() where it is set on the connect=
ion
> level which will be used in tcp_output() later on.
>=20
> tcp_mss() gets called from multiple places, I=E2=80=99ll look into that
> later. I will let you know if I find out more.
>=20
>=20
> Markus
>=20
Well, if tp->t_tsomax isn't set to a value of 65518, then the ixgbe.patch
isn't doing what I thought it would.
The only explanation I can think of for this is that there might be
another net interface driver stacked on top of the ixgbe.c one and
that the setting doesn't get propagated up.
Does this make any sense?
IP_MAXPACKET can't be changed from 65535, but I can see an argument
for setting the default value of if_hw_tsomax to a smaller value.
For example, in sys/net/if.c change it from:
657 if (ifp->if_hw_tsomax =3D=3D 0)
658 =09ifp->if_hw_tsomax =3D IP_MAXPACKET;
to
657 if (ifp->if_hw_tsomax =3D=3D 0)
658 =09ifp->if_hw_tsomax =3D 65536 - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN)=
;
This is a slightly smaller default which won't have much impact unless
the hardware device can only handle 32 mbuf clusters for transmit of
a segment and there are several of those.
Christopher, can you do your test run with IP_MAXPACKET set to 65518,
which should be the same as the above. If that gets rid of all the
EFBIG error replies, then I think the above patch will have the same
effect.
Thanks, rick
>=20
> > However, this is still a TSO related issue, it's just not one
> > related to
> > the setting of TSO's max size.
> >=20
> > A 10.0-STABLE system with tso disabled on ix0 doesn't have a single
> > packet
> > over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit longer
> > to
> > increase confidence in this assertion, but I don't want to waste
> > time on
> > this when I could be logging problem packets on a system with TSO
> > enabled.
> >=20
> > Comments are very welcome..
> > _______________________________________________
> > freebsd-net@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to
> > "freebsd-net-unsubscribe@freebsd.org"
> >=20
>=20
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1236110257.2510701.1395709458870.JavaMail.root>