Date: Wed, 12 Aug 2015 01:14:57 +0200 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@cochard.me> To: Maxim Sobolev <sobomax@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org>, freebsd@intel.com, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com> Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA@mail.gmail.com> In-Reply-To: <CAH7qZftMB34PM1CqNhdg7AWhsq6YknUDgc60ASfT2Z0L1z8XCQ@mail.gmail.com> References: <CAH7qZftMB34PM1CqNhdg7AWhsq6YknUDgc60ASfT2Z0L1z8XCQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Aug 11, 2015 at 11:18 PM, Maxim Sobolev <sobomax@freebsd.org> wrote= : > Hi folks, > > =E2=80=8BHi, =E2=80=8B > We've trying to migrate some of our high-PPS systems to a new hardware th= at > has four X540-AT2 10G NICs and observed that interrupt time goes through > roof after we cross around 200K PPS in and 200K out (two ports in LACP). > The previous hardware was stable up to about 350K PPS in and 350K out. I > believe the old one was equipped with the I350 and had the identical LACP > configuration. The new box also has better CPU with more cores (i.e. 24 > cores vs. 16 cores before). CPU itself is 2 x E5-2690 v3. > =E2=80=8B200K PPS, and even 350K PPS are very low value indeed. On a Intel Xeon L5630 (4 cores only) with one X540-AT2=E2=80=8B =E2=80=8B(then 2 10Gigabit ports)=E2=80=8B I've reached about 1.8Mpps (fast= forwarding enabled) [1]. But my setup didn't use lagg(4): Can you disable lagg configuration and re-measure your performance without lagg ? Do you let Intel NIC drivers using 8 queues for port too? In my use case (forwarding smallest UDP packet size), I obtain better behaviour by limiting NIC queues to 4 (hw.ix.num_queues or hw.ixgbe.num_queues, don't remember) if my system had 8 cores. And this with Gigabit Intel[2] or Chelsio NIC [3]. Don't forget to disable TSO and LRO too. =E2=80=8BRegards, Olivier [1] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ib= m_system_x3550_m3_with_10-gigabit_intel_x540-at2#graphs [2] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_sup= erserver_5018a-ftn4#graph1 [3] http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_= proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr#reduci= ng_nic_queues
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcqq1SGofhwA9PBmkcNrWsiyY32Ha9tfFM8ROwXmmvX%2BrA>