Date: Thu, 10 Dec 2015 17:42:34 -0200 From: Denis Pearson <dennix.pearson@gmail.com> To: Adrian Chadd <adrian.chadd@gmail.com> Cc: "Eggert, Lars" <lars@netapp.com>, "Pieper, Jeffrey E" <jeffrey.e.pieper@intel.com>, Kevin Oberman <rkoberman@gmail.com>, Daniel Engberg <daniel.engberg.lists@pyret.net>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: ixl 40G bad performance? Message-ID: <CAFC3-mTLmFijXNBdu_HKn2Q=AMLUAVH1AwZJ=XD=yMwSPX--dQ@mail.gmail.com> In-Reply-To: <CAJ-Vmo=cQQjfKhKB_D6Uc2e9TPnQwK2EXBka_sACwCkOCdYvgA@mail.gmail.com> References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <CAN6yY1t9Tw0j=uwaw1GK47r5=F-zeuz2hps_Ez3Y_QC-QSAGKA@mail.gmail.com> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> <A546ABEA-D495-461F-9441-31F70AACC146@netapp.com> <E21A5504-7780-4D84-AA5B-7A3F6F968FC7@netapp.com> <CAFC3-mQ2d1GKCRCEF8%2BC-u1-EVy094w0zCDv7QHeYnaHK-NHrw@mail.gmail.com> <CAJ-Vmo=cQQjfKhKB_D6Uc2e9TPnQwK2EXBka_sACwCkOCdYvgA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Dec 10, 2015 at 4:40 PM, Adrian Chadd <adrian.chadd@gmail.com> wrote: > On 10 December 2015 at 10:29, Denis Pearson <dennix.pearson@gmail.com> > wrote: > > On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars <lars@netapp.com> wrote: > > > >> On 2015-10-26, at 18:40, Eggert, Lars <lars@netapp.com> wrote: > >> > On 2015-10-26, at 17:08, Pieper, Jeffrey E < > jeffrey.e.pieper@intel.com> > >> wrote: > >> >> As a caveat, this was using default netperf message sizes. > >> > > >> > I get the same ~3 Gb/s with the default netperf sizes and driver > 1.4.5. > >> > >> Now there is version 1.4.8 on the Intel website, but it doesn't change > >> things for me. > >> > > > > I had the opportunity to see similar numbers and behavior while using > XL710 > > 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as > of > > r292035 was providing expected numbers. While removing rxcsum/txcsum did > > not provide differences, fully removing RSS + disabling rx/cxsum support > > provided better numbers. > > Can someone debug this a bit more? (My kit with ixl NICs in it is > still not up and available. :( ) > > Device RSS, even without kernel RSS enabled, shouldn't cause a massive > performance drop. If it is then something else odd is going on. > Do you have a diff where you removed things? > I can probably find out a snapshot with the code at the time and extract a diff, yes. I just don't know how it worths wasting the time when the problem is not reproducible on the current 1.4.8 driver which will hopefully get into -CURRENT (if it's not already there?). And it's much more specific, the performance drop happened on dpdk poll mode, not the usual kernel operation so a simple diff only pointing out the changes for the driver to actually build and run without rss will still require a testlab and different ways to generate traffic. This is why I suggested a transceiver change or replug first. Anyway RSS performance dropping problem is far from a FreeBSD specific problem, while researching I could find the exact same complaints on Windows users starting from windows 8 while having RSS@4 or RSS@16 or RSS completely disabled, some times with acceptable results only when it was disabled (despiste the fact that MiniportInterruptDPC was using a whole CPU when RSS was off results were still better). So I guess this is just a side effect of when it's just good to have NIC features turned off. The reason, I'm not an engineer to answer, but I would guess it's related to other NIC features also doing something with the packet or any sort of errors netstat or driver status may not tell. I was able to see the problem even with low pps rates and big packet sizes, as well as avg pkt size of 768bytes so I don't think it's any sort of card resource starvation. I can manage to have the whole lab up and running by the weekend if you want to investigate and compare, just ping me off list. > > -adrian > > > However now with driver 1.4.8 and the same set of hardware setup, except > > for a different transceiver, I can get 36Gbps/24Mpps with no further > > tweaks, so if you can replace your transceiver, shall be a different test > > as a starting point. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFC3-mTLmFijXNBdu_HKn2Q=AMLUAVH1AwZJ=XD=yMwSPX--dQ>