From owner-freebsd-stable@freebsd.org Mon Aug 24 12:25:20 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CECE9C2C0F; Mon, 24 Aug 2015 12:25:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 1B6272ED; Mon, 24 Aug 2015 12:25:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:998oWRF0O/4N2d8MBfZYGZ1GYnF86YWxBRYc798ds5kLTJ75oMuwAkXT6L1XgUPTWs2DsrQf27GQ7v2rBTVIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/njKbvptaPOk1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGCeG4HoRVi08iBNOAhPepEX2V5H3owPxrax9xSube8T9C7EwD2eM9aBuHSXpgyRPEjcy82Xaj4QklqdSqxGlqhlX3onbfYyRLPo4daqLLoBSfnZIQssED38JOYi7dYZaSrNZZes= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BJAgC3DNtV/61jaINdDoNhaQaDH7pLAQmBbQqCQ4JuSgKBZRQBAQEBAQEBAYEJgh2CBgEBAQMBAQEBIAQnIAsFCwIBCA4KAgINGQICJwEJJgIECAcEARwEiAUIDbEVlTIBAQEBAQEBAwEBAQEBARgEgSKKNYQyBgEBBRcBMweCaYFDBZU0hQaFCoQth0yNT4NoAiaDQFoiMwd+AQYCFyOBBAEBAQ X-IronPort-AV: E=Sophos;i="5.15,737,1432612800"; d="scan'208";a="234429859" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 24 Aug 2015 08:25:12 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 94B2915F55D; Mon, 24 Aug 2015 08:25:12 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id vR_Dc3DDNiPK; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id DE95315F563; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id npe3rZJYJb-L; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id BD72615F55D; Mon, 24 Aug 2015 08:25:11 -0400 (EDT) Date: Mon, 24 Aug 2015 08:25:11 -0400 (EDT) From: Rick Macklem To: Daniel Braniss Cc: Hans Petter Selasky , pyunyh@gmail.com, FreeBSD Net , FreeBSD stable , Gleb Smirnoff Message-ID: <2112273205.29795512.1440419111720.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> References: <1D52028A-B39F-4F9B-BD38-CB1D73BF5D56@cs.huji.ac.il> <1153838447.28656490.1440193567940.JavaMail.zimbra@uoguelph.ca> <15D19823-08F7-4E55-BBD0-CE230F67D26E@cs.huji.ac.il> <818666007.28930310.1440244756872.JavaMail.zimbra@uoguelph.ca> <49173B1F-7B5E-4D59-8651-63D97B0CB5AC@cs.huji.ac.il> <1815942485.29539597.1440370972998.JavaMail.zimbra@uoguelph.ca> <55DAC623.60006@selasky.org> <62C7B1A3-CC6B-41A1-B254-6399F19F8FF7@cs.huji.ac.il> Subject: Re: ix(intel) vs mlxen(mellanox) 10Gb performance MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: ix(intel) vs mlxen(mellanox) 10Gb performance Thread-Index: ++D1Njh8EEvTZp84OstiEvfCeIwEzQ== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Aug 2015 12:25:20 -0000 Daniel Braniss wrote: >=20 > > On 24 Aug 2015, at 10:22, Hans Petter Selasky wrote: > >=20 > > On 08/24/15 01:02, Rick Macklem wrote: > >> The other thing is the degradation seems to cut the rate by about half > >> each time. > >> 300-->150-->70 I have no idea if this helps to explain it. > >=20 > > Might be a NUMA binding issue for the processes involved. > >=20 > > man cpuset > >=20 > > --HPS >=20 > I can=E2=80=99t see how this is relevant, given that the same host, using= the > mellanox/mlxen > behave much better. Well, the "ix" driver has a bunch of tunables for things like "number of qu= eues" and although I'll admit I don't understand how these queues are used, I thi= nk they are related to CPUs and their caches. There is also something called I= XGBE_FDIR, which others have recommended be disabled. (The code is #ifdef IXGBE_FDIR, = but I don't know if it defined for your kernel?) There are also tunables for interrupt = rate and something called hw.ixgbe_tx_process_limit, which appears to limit the numb= er of packets to send or something like that? (I suspect Hans would understand this stuff much better than I do, since I = don't understand it at all.;-) At a glance, the mellanox driver looks very different. > I=E2=80=99m getting different results with the intel/ix depending who is = the nfs > server >=20 Who knows until you figure out what is actually going on. It could just be = the timing of handling the write RPCs or when the different servers send acks for the TCP= segments or ... that causes this for one server and not another. One of the principals used when investigating airplane accidents is to "nev= er assume anything" and just try to collect the facts until the pieces of the puzzle fall in pl= ace. I think the same principal works for this kind of stuff. I once had a case where a specific read of one NFS file would fail on certa= in machines. I won't bore you with the details, but after weeks we got to the point wher= e we had a lab of identical machines (exactly the same hardware and exactly the same softw= are loaded on them) and we could reproduce this problem on about half the machines and not the = other half. We (myself and the guy I worked with) finally noticed the failing machines wer= e on network ports for a given switch. We moved the net cables to another switch and the probl= em went away. --> This particular network switch was broken in such a way that it would g= arble one specific packet consistently, but worked fine for everything else. My point here is that, if someone had suggested the "network switch might b= e broken" at the beginning of investigating this, I would have probably dismissed it, based = on "the network is working just fine", but in the end, that was the problem. --> I am not suggesting you have a broken network switch, just "don't take = anything off the table until you know what is actually going on". And to be honest, you may never know, but it is fun to try and solve these = puzzles. Beyond what I already suggested, I'd look at the "ix" driver's stats and tu= nables and see if any of the tunables has an effect. (And, yes, it will take time to w= ork through these.) Good luck with it, rick >=20 > danny >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"