Date: Mon, 21 May 2007 11:29:30 -0700 From: "Kip Macy" <kip.macy@gmail.com> To: root <root@jim-liesl.org>, freebsd-net@freebsd.org Subject: Re: asymetric speeds over gigE link Message-ID: <b1fa29170705211129s40b6a1baua5ed2d21de1bca71@mail.gmail.com> In-Reply-To: <4651DB6E.5090102@jim-liesl.org> References: <4651DB6E.5090102@jim-liesl.org>
next in thread | previous in thread | raw e-mail | index | archive | help
- Original message -
Summary: Using iperf to measure TCP net speed between...
This is not terribly surprising rx is currently a more expensive
operation than tx on FreeBSD. My testing on 10GigE indicates that
there is ample room for improvement.
-Kip
On 5/21/07, root <root@jim-liesl.org> wrote:
> Summary: Using iperf to measure TCP net speed between a linux and
> freebsd box over gigE, I see significant speed difference depending on
> the data direction. Pushing data from the freebsd box to the linux box,
> I average about 500Gb/s. Pushing data from the linux box to the freebsd
> box, I see about 300Gb/s. It would seem that there is something
> limiting either the transmit side of the linux box or the recv side of
> the freebsd box. Any suggestions on how to narrow down who's tuning
> needs more work would also be welcome. Just for grins, I tried upping
> the txqueuelen on the linux box to 1500, but no help.
>
> I'd like to get closer to the theoretical speed of this link (I'd be
> happy if I could get 600-700 both ways), but the real puzzler is the
> difference. The client is the linux box @192.168.1.104 and the iperf
> server is the freebsd box @ 192.168.1.15 . Neither box is cpu or net
> busy during testing. Both systems only have one nic. I played with
> larger window sizes, but it only had a minor affect. I went with the
> Intel Pro/1000's based on their good rep in both the linux and freebsd
> communities.
>
> Client connecting to 192.168.1.15, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001
> [ 6] 0.0-10.0 sec 369 MBytes 309 Mbits/sec
> [ 5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963
> [ 5] 0.0-10.0 sec 597 MBytes 500 Mbits/sec
>
>
> Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are
> Belkin 5e), both systems are on this switch.
>
> FreeBSD box:
> FreeBSD 6.1-RELEASE-p10
> 1.9Ghz Athlon / 1 gig of main mem
> Abit/nforce2 MB+chipset (onboard nic disabled in bios)
> Intel Pro/1000GT NIC
> sysctl.conf:
> kern.ipc.maxsockbuf=8192000
> net.inet.tcp.sendspace=262144
> net.inet.tcp.recvspace=262144
> sysctl reports:
> kern.ipc.nmbjumbo16: 0
> kern.ipc.nmbjumbo9: 0
> kern.ipc.nmbjumbop: 0
> kern.ipc.nmbclusters: 25600
> net.inet.tcp.rfc1323: 1
>
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> options=b<RXCSUM,TXCSUM,VLAN_MTU>
> inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1
> inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
> ether 00:0e:0c:da:1a:3c
> media: Ethernet autoselect (1000baseTX <full-duplex>)
> status: active
>
> Linux box:
> Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686
> GNU/Linux
> Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem.
> I run tcp_tune.sh at boot on the linux box
> #!/bin/bash
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem
> echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem
>
> /proc/sys/net/ipv4/tcp_window_scaling is 1
> /proc/sys/net/ipv4/tcp_timestamps is 1
>
> eth1 Link encap:Ethernet HWaddr 00:0E:0C:DA:1A:3B
> inet addr:192.168.1.104 Bcast:192.168.1.255 Mask:255.255.255.0
> inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0
> TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:147638720 (140.7 MiB) TX bytes:1109677958 (1.0 GiB)
> Base address:0xd800 Memory:e2020000-e2040000
>
> Settings for eth1:
> Supported ports: [ TP ]
> Supported link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Supports auto-negotiation: Yes
> Advertised link modes: 10baseT/Half 10baseT/Full
> 100baseT/Half 100baseT/Full
> 1000baseT/Full
> Advertised auto-negotiation: Yes
> Speed: 1000Mb/s
> Duplex: Full
> Port: Twisted Pair
> PHYAD: 0
> Transceiver: internal
> Auto-negotiation: on
> Supports Wake-on: umbg
> Wake-on: g
> Current message level: 0x00000007 (7)
> Link detected: yes
>
>
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b1fa29170705211129s40b6a1baua5ed2d21de1bca71>