Date: Fri, 22 Jul 2016 14:36:01 -0400 (EDT) From: Chris Dunbar <chris@dunbar.net> To: freebsd-net <freebsd-net@freebsd.org> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Message-ID: <1366880114.715041.1469212561939.JavaMail.zimbra@dunbar.net> In-Reply-To: <CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ@mail.gmail.com> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <CA%2Bb0zg-mXiDZzKmcomfLNxKbpb_R1F50k=vo%2B32sxQwLkRNGvg@mail.gmail.com> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <CAEW%2BogZwHKhXAb2ra2eq14UteUwzZLyEpPiLEywzaen0nCppbw@mail.gmail.com> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> <CAN6yY1tp29nUc2=18jEYkhgfMpQ0Rg=dB_OtzB0ZpSrXRgN_bQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Thank you - I will check that out.=20 From: "Kevin Oberman" <rkoberman@gmail.com>=20 To: "chris" <chris@dunbar.net>=20 Cc: "freebsd-net" <freebsd-net@freebsd.org>=20 Sent: Friday, July 22, 2016 2:23:26 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 On Fri, Jul 22, 2016 at 6:48 AM, Chris Dunbar < chris@dunbar.net > wrote:= =20 Hi Sami,=20 I haven't actually fixed anything yet. I have only demonstrated that the po= or performance does not appear to happen between two FreeBSD boxes and poss= ibly between a Linux and FreeBSD, but I am going to confirm that now. I hav= e also seen good performance between the Windows box and Linux so that does= n't quite add up either. I may have to break out Wireshark and make some pa= cket captures to see if I can tell what's going on. If I find anything, I w= ill be sure to share it.=20 Regards,=20 Chris=20 From: "Sami Halabi" < sodynet1@gmail.com >=20 To: "chris" < chris@dunbar.net >=20 Cc: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Friday, July 22, 2016 8:34:30 AM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 hi,=20 would you share what was wrong in the windows side and how you solved it?= =20 Sami=20 =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 2016= 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7=91:= =20 Hello again,=20 I have good news and bad news:=20 The bad news first: I am an idiot and I have wasted some of your time for w= hich I apologize.=20 The good news: Testing now between two FreeBSD 10.3 systems, I am achieving= blistering speeds with iperf3. I apparently fell into the trap of assuming= the new thing (FreeBSD is new to me) was broken. Now I see that I was assu= ming Windows was working fine and focusing all my attention on FreeBSD. Loo= king back over everything I have done to troubleshoot this situation I must= conclude that the performance issue was on the Windows side and not the Fr= eeBSD side. I am less concerned about that because my ultimate goal is to i= nstall my three X540s into one FreeBSD server and two VMware ESXi hosts. I = am now fairly confident performance will be great.=20 Many thanks for your collective attention and the suggestions I received fr= om Eric and others.=20 Regards,=20 Chris=20 ----- Original Message -----=20 From: "chris" < chris@dunbar.net >=20 To: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Thursday, July 21, 2016 4:53:40 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 Eric, et al:=20 I haven't tried netperf yet, but I do have some new information to share. I= have two systems that I am using for testing: the new server and an older = (not too old) desktop PC. I installed CentOS on the new server again becaus= e I know it can achieve >9 GB/s with the X540. I replaced Windows on the de= sktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I wa= s able to achieve >9 GB/s so I know the problem isn't the X540 and I know t= he problem isn't anything with the default installation of FreeBSD 10.3. So= , what in the world might be nutty in my BIOS settings (or elsewhere) that = would cause the new server + FreeBSD 10.3 + X540 to equal slow performance?= =20 Regards,=20 Chris=20 From: "Eric Joyner" < erj@freebsd.org >=20 To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org >= =20 Sent: Thursday, July 21, 2016 1:27:10 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 (Replying-all this time)=20 Did you try to set these settings that ESnet recommends? https://fasterdata= .es.net/host-tuning/freebsd/=20 We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure= I can be much help diagnosing what's wrong.=20 On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote:=20 Hello,=20 I am new to FreeBSD and recently built a file server out of new components = running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experie= ncing what I consider to be slow transfer speeds. I am using iperf3 to meas= ure the speed and test the results of modifications. So far nothing I have = done has made a noticeable difference. If I run iperf3 -s on the FreeBSD se= rver, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in c= lient mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD= with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s.= The other end of my iperf3 test is a Windows 10 box that also has an Intel= X540-T2 installed.=20 I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes = a slightly older Intel driver (v3.1.13-k). I managed to build a custom kern= el that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to ma= nually load the latest 3.1.14 driver downloaded from Intel's web site. Unfo= rtunately that did not produce any improvements. I am working my way throug= h man tuning() and some other articles on network performance. So far nothi= ng I tweak makes a noticeable difference. I'm increasingly skeptical that I= am going to find a setting or two that more than doubles the speed I am cu= rrently experiencing.=20 I am open to any and all suggestions at this point.=20 Thank you!=20 Chris=20 This sort of problem can be very tricky to diagnose. I'd like to suggest th= at one of the tool you use should be SIFTR. It does kernel level collection= of network statistics and is a loadable module. By default it i IPv4 only.= It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" uncommented in /= sys/modules/siftr/Makefile for IPv6 support. It starts, stops, amd manages = collection under the control of 4 sysctls.=20 I have found it invaluable for analysis of netork performance issues, but s= eems to not be widely known.=20 --=20 Kevin Oberman, Part time kid herder and retired Network Engineer=20 E-mail: rkoberman@gmail.com=20 PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1366880114.715041.1469212561939.JavaMail.zimbra>