Date: Sun, 23 Mar 2014 12:19:25 -0300 From: Christopher Forgeron <csforgeron@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Garrett Wollman <wollman@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, Markus Gebert <markus.gebert@hostpoint.ch> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <CAB2_NwB%2BL5T1iTaxZtt00SL22_1H0S9X%2BM_LOGfkoJmy_TJokw@mail.gmail.com> In-Reply-To: <1752303953.1405506.1395524483238.JavaMail.root@uoguelph.ca> References: <CAB2_NwADUfs%2BbKV9QE_C4B1vchnzGWr1TK4C7wP8Fh8m94_mHA@mail.gmail.com> <1752303953.1405506.1395524483238.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Mar 22, 2014 at 6:41 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> Christopher Forgeron wrote:
> > #if defined(INET) || defined(INET6)
> > /* Initialize to max value. */
> > if (ifp->if_hw_tsomax == 0)
> > ifp->if_hw_tsomax = IP_MAXPACKET;
> > KASSERT(ifp->if_hw_tsomax <= IP_MAXPACKET &&
> > ifp->if_hw_tsomax >= IP_MAXPACKET / 8,
> > ("%s: tsomax outside of range", __func__));
> > #endif
> >
> >
> > Should this be the location where it's being set rather than in
> > ixgbe? I would assume that other drivers could fall prey to this
> > issue.
> >
> All of this should be prepended with "I'm an NFS guy, not a networking
> guy, so I might be wrong".
>
> Other drivers (and ixgbe for the 82598 chip) can handle a packet that
> is in more than 32 mbufs. (I think the 82598 handles 100, grep for SCATTER
> in *.h in sys/dev/ixgbe.)
>
[...]
Yes, I agree we have to be careful about the limitations of other drivers,
but I'm thinking setting tso to IP_MAXPACKET is a bad idea, unless all of
the header subtractions are happening elsewhere. Then again, perhaps every
other driver (and possibly ixgbe.. i need to look more) does a maxtso -
various_headers to set a limit for data packets.
I'm not familiar with the Freebsd network conventions/styles - I'm just
asking questions, something I have a bad habit for, but I'm in charge of
code stability issues at my work so it's hard to stop.
>
> Now, since several drivers do have this 32 mbufs limit, I can see an
> argument
> for making the default a little smaller to make these work, since the
> driver can override the default. (About now someone usually jumps in and
> says
> something along the lines of "You can't do that until all the drivers that
> can handle IP_MAXPACKET are fixed to set if_hw_tsomax" and since I can't
> fix
> drivers I can't test, that pretty much puts a stop on it.)
>
>
Testing is a problem isn't it? I once again offer my stack of network cards
and systems for some sort of testing.. I still have coax and token ring
around. :-)
> You see the problem isn't that IP_MAXPACKET is too big, but that the
> hardware
> has a limit of 32 non-contiguous chunks (mbufs)/packet and 32 * MCLBYTES =
> 64K.
> (Hardware/network drivers that can handle 35 or more chunks (they like to
> call
> them transmit segments, although ixgbe uses the term scatter) shouldn't
> have
> any problems.)
>
> I have an untested patch that adds a tsomaxseg count to use along with
> tsomax
> bytes so that a driver could inform tcp_output() it can only handle 32
> mbufs
> and then tcp_output() would limit a TSO segment using both, but I can't
> test
> it, so who knows when/if that might happen.
>
>
I think you give that to me in the next email - if not, please send.
> I also have a patch that modifies NFS to use pagesize clusters (reducing
> the
> mbuf count in the list), but that one causes grief when testing on an i386
> (seems to run out of kernel memory to the point where it can't allocate
> something
> called "boundary tags" and pretty well wedges the machine at that point.)
> Since I don't know how to fix this (I thought of making the patch "amd64
> only"),
> I can't really commit this to head, either.
>
>
Send me that one too. I love NFS patches.
> As such, I think it's going to be "fix the drivers one at a time" and tell
> folks to "disable TSO or limit rsize,wsize to 32K" when they run into
> trouble.
> (As you might have guessed, I'd rather just be "the NFS guy", but since NFS
> "triggers the problem" I\m kinda stuck with it;-)
>
> I know in some circumstances disabling TSO can be a benefit, but in
general you'd want it on a modern system with heavy data load.
> Also should we not also subtract ETHER_VLAN_ENCAP_LEN from tsomax to
> > make sure VLANs fit?
> >
> No idea. (I wouldn't know a VLAN if it jumped up and tried to
> bite me on the nose.;-) So, I have no idea what does this, but
> if it means the total ethernet header size can be > 14bytes, then I'd
> agree.
>
> Yeah, you need another 4 bytes for VLAN header if you're not using
hardware that strips it before the TCP stack gets it. I have a mix of
hardware and software VLANs running on our backbone, mostly due to a mixed
FreeBSD/OpenBSD/Windows environment.
> > Perhaps there is something in the newer network code that is filling
> > up the frames to the point where they are full - thus a TSO =
> > IP_MAXPACKET is just now causing problems.
> >
> Yea, I have no idea why this didn't bite running 9.1. (Did 9.1 have
> TSO enabled by default?)
>
I believe 9.0 has TSO on by default.. I seem to recall it always being
there, but I can't easily confirm it now. My last 9.0-STABLE doesn't have
an ixgbe card in it.
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB2_NwB%2BL5T1iTaxZtt00SL22_1H0S9X%2BM_LOGfkoJmy_TJokw>