Date: Sat, 19 Oct 2013 23:58:35 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Konstantin Kuzvesov <kuzvesov@list.ru> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Re[5]: Assymetric NIC performance problem Message-ID: <CAN6yY1vYmi%2BA5Yt1VMvspBWb7r5eP5s12xyoQxZy1t-n6t%2B94A@mail.gmail.com> In-Reply-To: <1381227544.227894604@f404.i.mail.ru> References: <1381160691.954872513@f116.i.mail.ru> <1381227544.227894604@f404.i.mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 8, 2013 at 3:19 AM, Konstantin Kuzvesov <kuzvesov@list.ru>wrote: > On Mon, Oct 7, 2013 at 16:57 -07:00, Kevin Oberman <rkoberman@gmail.com> > wrote: > > On Mon, Oct 7, 2013 at 8:44 AM, Konstantin Kuzvesov <kuzvesov@list.ru<https://e.mail.ru/sentmsg?mailto=mailto%3akuzvesov@list.ru>; > > wrote: > > > I've got a FreeBSD file server running Samba, file upload speeds are okay, > but downloads are too slow. > First, I decided it is Samba's fault, but then I ran iperf tests... > > On a windows machine with gigabit NIC: > Z:\iperf>iperf -c 192.168.0.1 > ------------------------------------------------------------ > Client connecting to 192.168.0.1, TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.0.2 port 1064 connected with 192.168.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.2 sec 12.4 MBytes 10.2 Mbits/sec > > Z:\iperf>iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1 port 41220 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 716 MBytes 600 Mbits/sec > > And on a another with FastEthernet NIC: > C:\iperf>iperf.exe -c 192.168.0.1 > ------------------------------------------------------------ > Client connecting to 192.168.0.1, TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.0.5 port 4756 connected with 192.168.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.1 sec 11.4 MBytes 9.42 Mbits/sec > > C:\iperf>iperf.exe -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.0.5 port 5001 connected with 192.168.0.1 port 18558 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 106 MBytes 88.5 Mbits/sec > > Both tests show server's NIC performance degradation to around 10Mbit/s > when traffic goes from server to client. And everything works fine in other > direction. > > I verified the cables and hub by simply connecting server and a test > machine with a new short patch cord. I tried to change server's NIC from > D-Link DGE-528T to Planet ENW-9604. And it became even worse - > using Planet NIC > speeds slowed down to around 500Mbit/s to server and the same 10Mbit/s to > client. I tried to change NIC's media to 100BaseTX, it didn't help too. > What else should I try to fix the problem? Maybe my system requires is just > misconfigured? > > System configuration: > OS: FreeBSD 9.2-release > Kernel: generic > Firewall: none > /boot/loader.conf - load zfs modules only > /etc/sysctl.conf - empty > NIC: D-Link DGE-528T / Planet ENW-9604 > > -- > Konstantin Kuzvesov > > > Output from ifconfig would probably be helpful, but I'd also suggest that > you capture packets (or, at least headers) for at least the start of the > transfer and look at them with wireshark or some similar tool. > > wireshark can tell you quite a bit and, if you feed the capture into > tcptrace, you can really see what is happening. (Note, wireshark provides > indications of problems that can make sense to people not conversant with > the gory details of TCP, though some issues may not be obvious. tcptrace > output can be utterly cryptic if you don't understand a lot of the details > of TCP and congestion control. > > Both wireshark and tcptrace are in ports and are best installed on a > workstation. The tcpdump output can be used as input to both. ("tcpdump -pw > FILE -i INTERFACE host ADDRESS" can do the job. Then copy the capture to > the right place for analysis. But start with configuration and counters for > the interface (netstat -i). > -- > R. Kevin Oberman, Network Engineer > E-mail: rkoberman@gmail.com<https://e.mail.ru/sentmsg?mailto=mailto%3arkoberman@gmail.com>; > > > Sorry, I didn't know that UDP bandwidth must be specified manually, > otherwise it defaults to 1.0Mbit/s. > New UDP tests show that there must be some TCP misconfiguration: > > > #iperf -s -u > ------------------------------------------------------------ > Server listening on UDP port 5001 > Receiving 1470 byte datagrams > UDP buffer size: 41.1 KByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.0.1 port 5001 connected with 192.168.0.2 port 1039 > > [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams > [ 3] 0.0-10.0 sec 626 MBytes 525 Mbits/sec 0.062 ms 13181/459995 (2.9%) > > [ 3] 0.0-10.0 sec 1 datagrams received out-of-order > > #iperf -c 192.168.0.2 -u -b 1000M > ------------------------------------------------------------ > Client connecting to 192.168.0.2, UDP port 5001 > Sending 1470 byte datagrams > UDP buffer size: 9.00 KByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.0.1 port 15347 connected with 192.168.0.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 882 MBytes 740 Mbits/sec > [ 3] Sent 632673 datagrams > [ 3] WARNING: did not receive ack of last datagram after 10 tries. > > The only problem wireshark showed to me is wrong IP header checksums in > packets originated from server. > > -- > Konstantin Kuzvesov > > Sorry for the long break, but my wife and I have been traveling and I am just catching up on things. Ugh! 740 Mbit/sec is pretty bad. UDP at .95G should not be a problem. As you probably know, the checksum fails since checksums are off-loaded. I have no experience with your card. I can say that RealTek based cards are generally not the best, so that may be an issue, but I can't be sure. 740M is really pretty poor, though. I am assuming that these systems are within a very few microseconds of each other. I also assume the systems are both running the same version of FreeBSD and the same hardware with the same configurations. If this is not the case, please let me know as it might change things. Can you examine the inter-packet delay? Plotting it would be interesting and can be done from the packet capture with any of several tools. The UDP rate should be VERY consistent for UDP. Perhaps less so for TCP. I suspect that at least the TCP timings will be inconsistent. The trick is figuring out why. If the UDP times are not consistent, then something very bad is going on in the transmitting system. Even if they are consistent, something is not right. Can you provide the packet capture of the TCP test? It need only be the headers (the first 40 bytes). iperf data is pretty boring. :-) I think that wireshark can "edit" the capture if you need to obfuscate the address. I also realized that the FreeBSD port is still iperf-2.03. Version 3 has been out for quite a while. I'll drop the maintainer a note asking if there is a reason for not updating the port. This should not be relevant to any testing you are doing other than to eliminate some bogus messages. It does clean up the code and adds a few nice capabilities. The people who maintain iperf work in the tools and research groups where I work (at least until the end of November), so I am probably more aware then most of development work. I will also ask Jon about any issues the code might have on FreeBSD since Linux is now used for development of our tools. (Not my idea.) -- R. Kevin Oberman, Network Engineer E-mail: rkoberman@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1vYmi%2BA5Yt1VMvspBWb7r5eP5s12xyoQxZy1t-n6t%2B94A>