Date: Fri, 26 Oct 2007 09:10:55 -0700 From: Matthew Reimer <mattjreimer@gmail.com> To: current@freebsd.org Subject: Network throughput problems in RELENG_7 Message-ID: <1193415055.3378.1.camel@bilbo.vpop.net>
next in thread | raw e-mail | index | archive | help
I'm seeing a problem where a much faster quad-core host running RELENG_7
serves many fewer netrate/http requests per second (175/sec) than an
old, busy, UP 6.0 host (828/sec). The problem seems to be related to
latency and connection setup, as it shows up dramatically over a link
with 50-60 ms latency. Can you help?
"Gandalf" is the problem RELENG_7 host connected via em0 on a 100baseTX
full-duplex switch port.
"Boromir" is a 6.0 host for reference, connected via bge0 to another
100baseTX full-duplex port on the same switch.
"Theoden" is a 6.2 host remote (50-60 ms away) from gandalf and boromir.
Running tools/netrate/http against tools/netrate/httpd on gandalf and
boromir, serving a document with 35 bytes of content, I get this
(details including tcpdumps following my sig):
dest dest
source boromir gandalf
------ ------- -------
theoden 828 175
boromir - 18092
gandalf 11493 -
The problem occurs using ab as well, but is much less pronounced when -k
is used (enabling keepalive and presumably HTTP pipelining). This makes
me wonder if the problem might have something to do with connection
setups.
I've verified that the switch port and the interface are configured
correctly (100baseTX full-duplex). I've also tried toggling polling and
the following sysctls, with no significant change:
net.inet.tcp.tso
net.inet.tcp.newreno
net.inet.tcp.sack.enable
net.inet.tcp.inflight.enable
net.inet.tcp.syncookies_only
net.inet.tcp.syncookies
What else can I try?
Thanks for any help you can give.
Matt
theoden-gandalf
===============
theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2
216.193.240.4
1280 /short.html
187 transfers/second
0 errors/second
tcpdump:
http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate.dump.gz
---
theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.4 1280 /short.html
175 transfers/second
0 errors/second
tcpdump:
http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate-n100.dump.gz
theoden-boromir
===============
theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2
216.193.240.2
1280 /short.html
272 transfers/second
0 errors/second
tcpdump:
http://boromir.vpop.net/~mreimer/theoden-boromir-netrate.dump.gz
---
theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.2 1280 /short.html
828 transfers/second
0 errors/second
tcpdump:
http://boromir.vpop.net/~mreimer/theoden-boromir-netrate-n100.dump.gz
boromir-gandalf
===============
boromir: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.4 1280 /short.html
18092 transfers/second
0 errors/second
gandalf-boromir
===============
gandalf: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.2 1280 /short.html
11493 transfers/second
0 errors/second
boromir sysctl and ifconfig:
http://boromir.vpop.net/~mreimer/boromir.sysctl.gz
# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=1a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
inet6 fe80::2e0:81ff:fe60:3925%bge0 prefixlen 64 scopeid 0x1
inet 216.193.240.33 netmask 0xffffffff broadcast 216.193.240.33
inet 216.193.240.34 netmask 0xffffffff broadcast 216.193.240.34
inet 216.193.240.35 netmask 0xffffffff broadcast 216.193.240.35
inet 216.193.240.36 netmask 0xffffffff broadcast 216.193.240.36
inet 216.193.240.37 netmask 0xffffffff broadcast 216.193.240.37
inet 216.193.240.17 netmask 0xffffffff broadcast 216.193.240.17
inet 216.193.240.38 netmask 0xffffffff broadcast 216.193.240.38
inet 216.193.240.39 netmask 0xffffffff broadcast 216.193.240.39
inet 216.193.240.2 netmask 0xfffff800 broadcast 216.193.247.255
inet 216.193.240.40 netmask 0xffffffff broadcast 216.193.240.40
ether 00:e0:81:60:39:25
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
gandalf sysctl and ifconfig:
http://gandalf.vpop.net/~mreimer/gandalf.sysctl.gz
# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
options=1cb<RXCSUM,TXCSUM,VLAN_MTU,POLLING,VLAN_HWCSUM,TSO4>
ether 00:30:48:33:96:a4
inet 216.193.240.4 netmask 0xfffff800 broadcast 216.193.247.255
inet 216.193.240.65 netmask 0xffffffff broadcast 216.193.240.65
inet 216.193.240.66 netmask 0xffffffff broadcast 216.193.240.66
inet 216.193.240.67 netmask 0xffffffff broadcast 216.193.240.67
inet 216.193.240.68 netmask 0xffffffff broadcast 216.193.240.68
media: Ethernet 100baseTX <full-duplex>
status: active
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1193415055.3378.1.camel>