Date: Sun, 1 Sep 2013 13:51:17 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Barney Cordoba <barney_cordoba@yahoo.com> Cc: Andre Oppermann <andre@freebsd.org>, Alan Somers <asomers@freebsd.org>, "net@freebsd.org" <net@freebsd.org>, Jack F Vogel <jfv@freebsd.org>, "Justin T. Gibbs" <gibbs@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it>, "T.C. Gubatayao" <tgubatayao@barracuda.com> Subject: Re: Flow ID, LACP, and igb Message-ID: <CAJ-VmomEKxJ5zz3Gw1b-HizDJ03_Mn=6uZVYR07QFTqwBzNsCg@mail.gmail.com> In-Reply-To: <1378050319.62710.YahooMailNeo@web121601.mail.ne1.yahoo.com> References: <D01A0CB2-B1E3-4F4B-97FA-4C821C0E3FD2@FreeBSD.org> <521BBD21.4070304@freebsd.org> <CAOtMX2jvKGY==t9i-a_8RtMAPH2p1VDj950nMHHouryoz3nbsA@mail.gmail.com> <521EE8DA.3060107@freebsd.org> <BCC2C62D4FE171479E2F1C2593FE508B0BE24383@BN-SCL-MBX03.Cudanet.local> <CAOtMX2h5SGh5eYV50y%2BQB_s367V9iattGU862wwXcONDV%2BTG8g@mail.gmail.com> <CA%2BhQ2%2BhgTaK1ZCOLGVFjSPY8nyNPHK4waSecyRQxR1gQcyjztg@mail.gmail.com> <1377952913.44129.YahooMailNeo@web121605.mail.ne1.yahoo.com> <BCC2C62D4FE171479E2F1C2593FE508B0BE2440B@BN-SCL-MBX03.Cudanet.local> <1378001733.36695.YahooMailNeo@web121606.mail.ne1.yahoo.com> <CA%2BhQ2%2Bj-DDuEX1KCDYioCactjL71p-d4AtusPUfePrswDyUpog@mail.gmail.com> <1378050319.62710.YahooMailNeo@web121601.mail.ne1.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Yo, LRO is an interesting hack that seems to do a good trick of hiding the ridiculous locking and unfriendly cache behaviour that we do per-packet. It helps with LAN test traffic where things are going out in batches from the TCP layer so the RX layer "sees" these frames in-order and can do LRO. When you disable it, I don't easily get 10GE LAN TCP performance. That has to be fixed. Given how fast the CPU cores, bus interconnect and memory interconnects are, I don't think there should be any reason why we can't hit 10GE traffic on a LAN with LRO disabled (in both software and hardware.) Now that I have the PMC sandy bridge stuff working right (but no PEBS, I have to talk to Intel about that in a bit more detail before I think about hacking that in) we can get actual live information about this stuff. But the last time I looked, there's just too much per-packet latency going on. The root cause looks like it's a toss up between scheduling, locking and just lots of code running to completion per-frame. As I said, that all has to die somehow. 2c, -adrian On 1 September 2013 08:45, Barney Cordoba <barney_cordoba@yahoo.com> wrote: > > > Comcast sends packets OOO. With any decent number of internet hops you're > likely to encounter a load > balancer or packet shaper that sends packets OOO, so you just can't be > worried about it. In fact, your > designs MUST work with OOO packets. > > Getting balance on your load balanced lines is certainly a bigger upside > than the additional CPU used. > You can buy a faster processor for your "stack" for a lot less than you > can buy bandwidth. > > Frankly my opinion of LRO is that it's a science project suitable for labs > only. It's a trick to get more bandwidth > than your bus capacity; the answer is to not run PCIe2 if you need pcie3. > You can use it internally if you have > control of all of the machines. When I modify a driver the first thing > that I do is rip it out. > > BC > > > ________________________________ > From: Luigi Rizzo <rizzo@iet.unipi.it> > To: Barney Cordoba <barney_cordoba@yahoo.com> > Cc: Andre Oppermann <andre@freebsd.org>; Alan Somers <asomers@freebsd.org>; > "net@freebsd.org" <net@freebsd.org>; Jack F Vogel <jfv@freebsd.org>; > Justin T. Gibbs <gibbs@freebsd.org>; T.C. Gubatayao < > tgubatayao@barracuda.com> > Sent: Saturday, August 31, 2013 10:27 PM > Subject: Re: Flow ID, LACP, and igb > > > On Sun, Sep 1, 2013 at 4:15 AM, Barney Cordoba <barney_cordoba@yahoo.com > >wrote: > > > ... > > > > [your point on testing with realistic assumptions is surely a valid one] > > > > > > Of course there's nothing really wrong with OOO packets. We had this > > discussion before; lots of people > > have round robin dual homing without any ill effects. It's just not an > > issue. > > > > It depends on where you are. > It may not be an issue if the reordering is not large enough to > trigger retransmissions, but even then it is annoying as it causes > more work in the endpoint -- it prevents LRO from working, and even > on the host stack it takes more work to sort where an out of order > segment goes than appending an in-order one to the socket buffer. > > cheers > luigi > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmomEKxJ5zz3Gw1b-HizDJ03_Mn=6uZVYR07QFTqwBzNsCg>