From owner-freebsd-net@freebsd.org Thu Mar 29 07:01:10 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6698CF537CF for ; Thu, 29 Mar 2018 07:01:10 +0000 (UTC) (envelope-from m.muenz@spam-fetish.org) Received: from mailout-02.maxonline.de (mailout-02.maxonline.de [81.24.66.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00A127E3E2 for ; Thu, 29 Mar 2018 07:01:09 +0000 (UTC) (envelope-from m.muenz@spam-fetish.org) Received: from web03-01.max-it.de (web03-01.max-it.de [81.24.64.215]) by mailout-02.maxonline.de (Postfix) with ESMTPS id 5AF912A for ; Thu, 29 Mar 2018 09:01:08 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by web03-01.max-it.de (Postfix) with ESMTP id 5135628A2F8 for ; Thu, 29 Mar 2018 09:01:08 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at web03-01.max-it.de Received: from web03-01.max-it.de ([127.0.0.1]) by localhost (web03-01.max-it.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4o6eoYBim9c6 for ; Thu, 29 Mar 2018 09:01:08 +0200 (CEST) Received: from [81.24.66.132] (unknown [81.24.66.132]) (Authenticated sender: m.muenz@spam-fetish.org) by web03-01.max-it.de (Postfix) with ESMTPA id 12A1C28A0BB for ; Thu, 29 Mar 2018 09:01:08 +0200 (CEST) Subject: Re: Current state of Intel XL710 40G NIC ixl performance To: freebsd-net@freebsd.org References: <201803281638.w2SGcWNp057893@pdx.rh.CN85.dnsmgr.net> From: "Muenz, Michael" Message-ID: <3bd4855f-3b5a-5db5-c004-845fd86e8191@spam-fetish.org> Date: Thu, 29 Mar 2018 09:01:07 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <201803281638.w2SGcWNp057893@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2018 07:01:10 -0000 Am 28.03.2018 um 18:38 schrieb Rodney W. Grimes: >> Am 28.03.2018 um 06:11 schrieb christian russell: >>> I am having trouble getting an Intel XL710-DA2 NIC to get even close to >>> line rate. It is a 4x10 Gbps card. The box is running FreeBSD 11 (FreeNAS >>> in particular). >>> >>> We have tried both 1.7 and 1.9 driver revisions with similar results. The >>> NVM version is 5.05. The card is in a confirmed 8x slot on a SuperMicro >>> X10DRL-i with two Xeon E5-2600 processors and 256 GB DDR4 RAM. After >>> upping the interrupt threshold to 9000 dmesg doesn't log anything unusual. >>> >>> We have added the tunes that are standard for 10 Gbps configurations. >>> >>> On a single-client basis the fastest rates we see are around 5 Gbps. >>> Hitting this server from multiple boxes we see peaks of 20 Gbps at the very >>> highest. More frequently things top off around 13 Gbps. These numbers are >>> coming from iperf tests. We are seeing similar numbers with direct >>> point-to-point as well as switched topologies. >>> >>> These threads from 2015 describe similar issues but fizzled out: >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-May/042273.html >>> https://lists.freebsd.org/pipermail/freebsd-net/2015-October/043584.html >>> >>> Is there very particular tuning required to get these cards working at >>> proper speed? Any insights? >>> >>> >From Googling around it appears frustration with this card and FreeBSD is >>> pretty common. >>> >>> Thanks in advance. >>> >>> Christian >> I can't deliver any special insights but we had many problems with X710 >> (without L) and Linux. >> Did some testing a while ago with OPNsense (based on 11.1) and got line >> rate with iperf and single client. >> ixl0 in and ixl1 out. So this should be fine. If you like I can send you >> the sysctl values to compare. > I would be interested in your sysctl values. > Hm, perhaps I misinterpreted your post. I recreated the lab and from the history I saw that it was single client but with 10 streams in parallel. If I reduce to 1 stream I also don't come over the 5,6G. If you're still interested I can send you the output. Michael