Date: Tue, 28 May 2013 16:14:20 +0200 From: Andre Oppermann <andre@freebsd.org> To: afischer@marvell.com Cc: freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: Low Tx-Rx performance with 10Gb NICs Message-ID: <51A4BBBC.8020405@freebsd.org> In-Reply-To: <1369406798.20748.30.camel@EL-DT095.site> References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <CADWvR2g_VfZcCrXH1TcOVtPKT8GLbSKaqcH46piFBUjoOzXWUA@mail.gmail.com> <1369406798.20748.30.camel@EL-DT095.site>
next in thread | previous in thread | raw e-mail | index | archive | help
On 24.05.2013 16:46, Axel Fischer wrote: > Hi Igor, > > my name is Axel Fischer. working at Marvell SC. Hi Axel, > In addition to your reply to my colleague Lino > Sanfilippo I did some performance measurements > on FreeBSD 9 with a commercial 10 GBit network > card. Which driver? > Unlike on other OS the FreeBSD performance > for duplex rx/tx operation never exceeded the > limit about 9.5 GBit/s. Normally a performance > of at least 16 GBit (up to line speed 20 GBit > in duplex mode) is expected. > As Lino already mentioned the CPU/bus system > (in general the HW) does not set a limit. > Furthermore I noticed that the CPU(s) load > is not very high, about 30 %. Your RX/TX is probably serialized in the driver and you can only make use of one core. > Here is an overview of the measurements: > > netperf rx-tx 4 streams / 60s > > 1768.16 Mb/s Port=2001 RX > 999.33 Mb/s Port=2002 RX > 72.16 Mb/s Port=1001 TX > 61.49 Mb/s Port=1002 TX > 2302.76 Mb/s Port=2003 RX > 73.48 Mb/s Port=1003 TX > 2416.23 Mb/s Port=2004 RX > 76.02 Mb/s Port=1004 TX > ==== > RX+TX Total Result: Mb/s 7769.63 > > > CPU load: > > last pid: 1739; load averages: 0.97, 0.49, 0.21 up 0+00:02:26 > 11:02:52 > 46 processes: 2 running, 44 sleeping > CPU 0: 2.0% user, 0.0% nice, 23.2% system, 0.4% interrupt, 74.4% idle > CPU 1: 1.2% user, 0.0% nice, 19.7% system, 0.4% interrupt, 78.7% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 80.7% interrupt, 19.3% idle > CPU 3: 0.0% user, 0.0% nice, 0.8% system, 1.6% interrupt, 97.6% idle > CPU 4: 2.4% user, 0.0% nice, 25.6% system, 0.0% interrupt, 72.0% idle > CPU 5: 3.1% user, 0.0% nice, 25.6% system, 0.0% interrupt, 71.3% idle > CPU 6: 0.0% user, 0.0% nice, 0.4% system, 32.7% interrupt, 66.9% idle > CPU 7: 0.4% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.0% idle > Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free > Swap: 4096M Total, 4096M Free > > Additionally I noticed the following TCP errors > with netstat -s ...: > > 1186 data packets (1717328 bytes) retransmitted This may happen and is typically not cause for concern on a loaded system. > 6847875 window update packets Normal. > 2319 duplicate acks Related to the retransmits. > 25831 out-of-order packets (37403288 bytes) This is unusual. What kind of test setup do you have, back-to-back cards or a switch in between? Out of order normally shouldn't happen unless over the internet. > 3733 discarded due to memory problems (drops) > 1186 segment rexmits in SACK recovery episodes > 1717328 byte rexmits in SACK recovery episodes Again related to retransmits. > My questions: > > - What is the max. performance (duplex) on > FreeBSD 9 that you have measured with a 10 GBit > NIC ? > (Expected > 16 GBit/s on appropriate HW) Certain large CDN are known to push more than 20Gbit/s production traffic per machine. Please also my other message from today to hackers@ "Re: preemptive kernel" with Message-ID: <51A4B991.3070805@freebsd.org>. -- Andre > Thank you in advance, > Axel > > > -------- Weitergeleitete Nachricht -------- > Von: Igor Mozolevsky <igor@hybrid-lab.co.uk> > An: Lino Sanfilippo <lsanfil@marvell.com> > Kopie: Hackers freeBSD <freebsd-hackers@freebsd.org>, Axel Fischer > <afischer@marvell.com>, Ralf Assmann <rassmann@marvell.com>, Markus > Althoff <malthoff@marvell.com> > Betreff: Re: Low Tx-Rx performance with 10Gb NICs > Datum: Thu, 23 May 2013 11:21:08 -0700 > > On 23 May 2013 19:00, Lino Sanfilippo <lsanfil@marvell.com> wrote: > >> Is there a known issue concerning high traffic on Tx and Rx paths? Are there any system >> settings I could adjust to get the expected performance? Any hints are very appreciated. > > check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting > ierrs on em chips, but none on fxp chips (connected to the same > wire/switch); might be unrelated to yours, but worth a check... > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51A4BBBC.8020405>