Date: Fri, 22 Jul 2016 11:23:26 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Chris Dunbar <chris@dunbar.net> Cc: freebsd-net <freebsd-net@freebsd.org> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Message-ID: <CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ@mail.gmail.com> In-Reply-To: <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <CA%2Bb0zg-mXiDZzKmcomfLNxKbpb_R1F50k=vo%2B32sxQwLkRNGvg@mail.gmail.com> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <CAEW%2BogZwHKhXAb2ra2eq14UteUwzZLyEpPiLEywzaen0nCppbw@mail.gmail.com> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jul 22, 2016 at 6:48 AM, Chris Dunbar <chris@dunbar.net> wrote: > Hi Sami, > > I haven't actually fixed anything yet. I have only demonstrated that the > poor performance does not appear to happen between two FreeBSD boxes and > possibly between a Linux and FreeBSD, but I am going to confirm that now.= I > have also seen good performance between the Windows box and Linux so that > doesn't quite add up either. I may have to break out Wireshark and make > some packet captures to see if I can tell what's going on. If I find > anything, I will be sure to share it. > > Regards, > Chris > > > From: "Sami Halabi" <sodynet1@gmail.com> > To: "chris" <chris@dunbar.net> > Cc: "freebsd-net" <freebsd-net@freebsd.org> > Sent: Friday, July 22, 2016 8:34:30 AM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > > > hi, > would you share what was wrong in the windows side and how you solved it? > > Sami > > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 20= 16 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7= =91: > > > Hello again, > > I have good news and bad news: > > The bad news first: I am an idiot and I have wasted some of your time for > which I apologize. > > The good news: Testing now between two FreeBSD 10.3 systems, I am > achieving blistering speeds with iperf3. I apparently fell into the trap = of > assuming the new thing (FreeBSD is new to me) was broken. Now I see that = I > was assuming Windows was working fine and focusing all my attention on > FreeBSD. Looking back over everything I have done to troubleshoot this > situation I must conclude that the performance issue was on the Windows > side and not the FreeBSD side. I am less concerned about that because my > ultimate goal is to install my three X540s into one FreeBSD server and tw= o > VMware ESXi hosts. I am now fairly confident performance will be great. > > Many thanks for your collective attention and the suggestions I received > from Eric and others. > > Regards, > Chris > > ----- Original Message ----- > From: "chris" < chris@dunbar.net > > To: "freebsd-net" < freebsd-net@freebsd.org > > Sent: Thursday, July 21, 2016 4:53:40 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > Eric, et al: > > I haven't tried netperf yet, but I do have some new information to share. > I have two systems that I am using for testing: the new server and an old= er > (not too old) desktop PC. I installed CentOS on the new server again > because I know it can achieve >9 GB/s with the X540. I replaced Windows o= n > the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 > again. I was able to achieve >9 GB/s so I know the problem isn't the X540 > and I know the problem isn't anything with the default installation of > FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (o= r > elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal > slow performance? > > Regards, > Chris > > > From: "Eric Joyner" < erj@freebsd.org > > To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org > > > Sent: Thursday, July 21, 2016 1:27:10 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > (Replying-all this time) > > Did you try to set these settings that ESnet recommends? > https://fasterdata.es.net/host-tuning/freebsd/ > > We don't use iperf3 here at Intel (we use netperf instead), so I'm not > sure I can be much help diagnosing what's wrong. > > On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: > > > Hello, > > I am new to FreeBSD and recently built a file server out of new component= s > running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am > experiencing what I consider to be slow transfer speeds. I am using iperf= 3 > to measure the speed and test the results of modifications. So far nothin= g > I have done has made a noticeable difference. If I run iperf3 -s on the > FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run > iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I > replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds > surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box th= at > also has an Intel X540-T2 installed. > > I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) include= s > a slightly older Intel driver (v3.1.13-k). I managed to build a custom > kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me > to manually load the latest 3.1.14 driver downloaded from Intel's web sit= e. > Unfortunately that did not produce any improvements. I am working my way > through man tuning() and some other articles on network performance. So f= ar > nothing I tweak makes a noticeable difference. I'm increasingly skeptical > that I am going to find a setting or two that more than doubles the speed= I > am currently experiencing. > > I am open to any and all suggestions at this point. > > Thank you! > Chris > This sort of problem can be very tricky to diagnose. I'd like to suggest that one of the tool you use should be SIFTR. It does kernel level collection of network statistics and is a loadable module. By default it i IPv4 only. It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" uncommented in /sys/modules/siftr/Makefile for IPv6 support. It starts, stops, amd manages collection under the control of 4 sysctls. I have found it invaluable for analysis of netork performance issues, but seems to not be widely known. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ>