From owner-freebsd-stable@freebsd.org Mon Aug 17 21:50:22 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9F8D9BB836; Mon, 17 Aug 2015 21:50:22 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 6384511F2; Mon, 17 Aug 2015 21:50:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:oguZsh8vk9kE7f9uRHKM819IXTAuvvDOBiVQ1KB82uMcTK2v8tzYMVDF4r011RmSDd6dt6kMotGVmp6jcFRI2YyGvnEGfc4EfD4+ouJSoTYdBtWYA1bwNv/gYn9yNs1DUFh44yPzahANS47AblHf6ke/8SQVUk2mc1ElfaKpQcb7tIee6aObw9XreQJGhT6wM/tZDS6dikHvjPQQmpZoMa0ryxHE8TNicuVSwn50dxrIx06vru/5xpNo8jxRtvQ97IYAFPyiJ+VrBYFeFyksZmAp+NXw516ESQqU+mBaXH8bnxBTD07C9h69W57wti7zsK152TKGPMv4Svc6Qzmv5bxnDQT0gS0DOm0E9nrKgJlwkL5Du0Dm4Bh+2JLPJo+POfd0Za+beskVAm9IX8JUXioGBoKnc4oJAe1GM/xVooPmqx4VsRK0AQT/OOS65jZOh3LylYcg2uIgChqOiAApGdQfmH/P6tXoNqZUWOvzza2enhvZaPYD4zb268DtexsipfyJFeZqdMPayk0iEivYiVqNpIj9P3We37Je4CCg8+N8WLf32CYcoAZrr23qn590hw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BFAgBUGNJV/61jaINdDoNhaQaDHrpKAQmBawqFL0oCgWgUAQEBAQEBAQGBCYIdggYBAQEDAQEBASAEJyALBQsCAQgOCgICDRYDAgIhBgEJFRECBAgHBAEcBId4AwoIDbsukB0NhVcBAQEBAQEEAQEBAQEBGASBIoowgk+BaAEBBxUBMweCaYFDBYcijXuFBIUGdYM3kSeDT4NlAiaDP1oiMwd/CBcjgQQBAQE X-IronPort-AV: E=Sophos;i="5.15,697,1432612800"; d="scan'208";a="233068245" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 17 Aug 2015 17:49:11 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id DD2B815F565; Mon, 17 Aug 2015 17:49:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id jAkikwheNEJB; Mon, 17 Aug 2015 17:49:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 05CB815F56D; Mon, 17 Aug 2015 17:49:11 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 6mbRortKRG-f; Mon, 17 Aug 2015 17:49:10 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id D914115F565; Mon, 17 Aug 2015 17:49:10 -0400 (EDT) Date: Mon, 17 Aug 2015 17:49:10 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Cc: FreeBSD Net , Slawa Olhovchenkov , FreeBSD stable , Christopher Forgeron Message-ID: <805850043.24018217.1439848150695.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <20150817094145.GB3158@zxy.spb.ru> <17871443-E105-4434-80B1-6939306A865F@cs.huji.ac.il> <473274181.23263108.1439814072514.JavaMail.zimbra@uoguelph.ca> <7F892C70-9C04-4468-9514-EDBFE75CF2C6@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: eHOLT35UpIGat4XSkuNV2057nk2L2g== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Aug 2015 21:50:23 -0000 Daniel Braniss wrote: >=20 > > On Aug 17, 2015, at 3:21 PM, Rick Macklem wrote: > >=20 > > Daniel Braniss wrote: > >>=20 > >>> On Aug 17, 2015, at 1:41 PM, Christopher Forgeron > >>> wrote: > >>>=20 > >>> FYI, I can regularly hit 9.3 Gib/s with my Intel X520-DA2's and FreeB= SD > >>> 10.1. Before 10.1 it was less. > >>>=20 > >>=20 > >> this is NOT iperf/3 where i do get close to wire speed, > >> it=E2=80=99s NFS writes, i.e., almost real work :-) > >>=20 > >>> I used to tweak the card settings, but now it's just stock. You may w= ant > >>> to > >>> check your settings, the Mellanox may just have better defaults for y= our > >>> switch. > >>>=20 > > Have you tried disabling TSO for the Intel? With TSO enabled, it will b= e > > copying > > every transmitted mbuf chain to a new chain of mbuf clusters via. > > m_defrag() when > > TSO is enabled. (Assuming you aren't an 82598 chip. Most seem to be the > > 82599 chip > > these days?) > >=20 >=20 > hi Rick >=20 > how can i check the chip? >=20 Haven't a clue. Does "dmesg" tell you? (To be honest, since disabling TSO h= elped, I'll bet you don't have a 82598.) > > This has been fixed in the driver very recently, but those fixes won't = be > > in 10.1. > >=20 > > rick > > ps: If you could test with 10.2, it would be interesting to see how the= ix > > does with > > the current driver fixes in it? >=20 > I new TSO was involved! > ok, firstly, it=E2=80=99s 10.2 stable. > with TSO enabled, ix is bad, around 64MGB/s. > disabling TSO it=E2=80=99s better, around 130 >=20 Hmm, could you check to see of these lines are in sys/dev/ixgbe/if_ix.c at = around line#2500? /* TSO parameters */ 2572 =09 =09 ifp->if_hw_tsomax =3D 65518; 2573 =09 =09 ifp->if_hw_tsomaxsegcount =3D IXGBE_82599_SCATTER; 2574 =09 =09 ifp->if_hw_tsomaxsegsize =3D 2048; They are in stable/10. I didn't look at releng/10.2. (And if they're in a #= ifdef for FreeBSD11, take the #ifdef away.) If they are there and not ifdef'd, I can't explain why disabling TSO would = help. Once TSO is fixed so that it handles the 64K transmit segments without copy= ing all the mbufs, I suspect you might get better perf. with it enabled? Good luck with it, rick > still, mlxen0 is about 250! with and without TSO >=20 >=20 > >=20 > >>> On Mon, Aug 17, 2015 at 6:41 AM, Slawa Olhovchenkov >>> > wrote: > >>> On Mon, Aug 17, 2015 at 10:27:41AM +0300, Daniel Braniss wrote: > >>>=20 > >>>> hi, > >>>> I have a host (Dell R730) with both cards, connected to an HP82= 00 > >>>> switch at 10Gb. > >>>> when writing to the same storage (netapp) this is what I get: > >>>> ix0: ~130MGB/s > >>>> mlxen0 ~330MGB/s > >>>> this is via nfs/tcpv3 > >>>>=20 > >>>> I can get similar (bad) performance with the mellanox if I incr= ease > >>>> the file size > >>>> to 512MGB. > >>>=20 > >>> Look like mellanox have internal beffer for caching and do ACK > >>> acclerating. > >>>=20 > >>>> so at face value, it seems the mlxen does a better use of resou= rces > >>>> than the intel. > >>>> Any ideas how to improve ix/intel's performance? > >>>=20 > >>> Are you sure about netapp performance? > >>> _______________________________________________ > >>> freebsd-net@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>> > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > >>> " > >>>=20 > >>=20 > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.o= rg" >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"