From owner-freebsd-net@freebsd.org Thu Mar 29 12:46:10 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 086DDF73310 for ; Thu, 29 Mar 2018 12:46:10 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5FF326BC5A for ; Thu, 29 Mar 2018 12:46:09 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w2TCk6iQ062960; Thu, 29 Mar 2018 05:46:06 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w2TCk67R062959; Thu, 29 Mar 2018 05:46:06 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201803291246.w2TCk67R062959@pdx.rh.CN85.dnsmgr.net> Subject: Re: Current state of Intel XL710 40G NIC ixl performance In-Reply-To: <3bd4855f-3b5a-5db5-c004-845fd86e8191@spam-fetish.org> To: "Muenz, Michael" Date: Thu, 29 Mar 2018 05:46:06 -0700 (PDT) CC: freebsd-net@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2018 12:46:10 -0000 > Am 28.03.2018 um 18:38 schrieb Rodney W. Grimes: > >> Am 28.03.2018 um 06:11 schrieb christian russell: > >>> I am having trouble getting an Intel XL710-DA2 NIC to get even close to > >>> line rate. It is a 4x10 Gbps card. The box is running FreeBSD 11 (FreeNAS > >>> in particular). > >>> > >>> We have tried both 1.7 and 1.9 driver revisions with similar results. The > >>> NVM version is 5.05. The card is in a confirmed 8x slot on a SuperMicro > >>> X10DRL-i with two Xeon E5-2600 processors and 256 GB DDR4 RAM. After > >>> upping the interrupt threshold to 9000 dmesg doesn't log anything unusual. > >>> > >>> We have added the tunes that are standard for 10 Gbps configurations. > >>> > >>> On a single-client basis the fastest rates we see are around 5 Gbps. > >>> Hitting this server from multiple boxes we see peaks of 20 Gbps at the very > >>> highest. More frequently things top off around 13 Gbps. These numbers are > >>> coming from iperf tests. We are seeing similar numbers with direct > >>> point-to-point as well as switched topologies. > >>> > >>> These threads from 2015 describe similar issues but fizzled out: > >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-May/042273.html > >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-October/043584.html > >>> > >>> Is there very particular tuning required to get these cards working at > >>> proper speed? Any insights? > >>> > >>> >From Googling around it appears frustration with this card and FreeBSD is > >>> pretty common. > >>> > >>> Thanks in advance. > >>> > >>> Christian > >> I can't deliver any special insights but we had many problems with X710 > >> (without L) and Linux. > >> Did some testing a while ago with OPNsense (based on 11.1) and got line > >> rate with iperf and single client. > >> ixl0 in and ixl1 out. So this should be fine. If you like I can send you > >> the sysctl values to compare. > > I would be interested in your sysctl values. > > > > Hm, perhaps I misinterpreted your post. I recreated the lab and from the > history I saw > that it was single client but with 10 streams in parallel. If I reduce > to 1 stream I also > don't come over the 5,6G. > > If you're still interested I can send you the output. We specifically are interested in the sysctl settings you used to obtain a "line rate" results using ixl0 in and ixl1 out, it does not matter if that was 1 stream or 10 stream, we would just like to see what values you found to work well for you to compare to what values others may be using/suggesting. -- Rod Grimes rgrimes@freebsd.org