From owner-freebsd-performance@FreeBSD.ORG Sat Oct 17 20:36:42 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 559C61065692 for ; Sat, 17 Oct 2009 20:36:42 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from proxy.meer.net (proxy.meer.net [64.13.141.13]) by mx1.freebsd.org (Postfix) with ESMTP id ED0978FC0C for ; Sat, 17 Oct 2009 20:36:41 +0000 (UTC) Received: from mail.meer.net (mail.meer.net [64.13.141.3]) by proxy.meer.net (8.14.3/8.14.3) with ESMTP id n9HKsZTd003384 for ; Sat, 17 Oct 2009 13:54:35 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id n9HKMmCs051552; Sat, 17 Oct 2009 13:22:48 -0700 (PDT) (envelope-from gnn@neville-neil.com) Received: from [172.16.1.4] (cpe-68-175-77-169.nyc.res.rr.com [68.175.77.169]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.3) with ESMTP id n9HKMlDF087767 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 17 Oct 2009 13:22:48 -0700 (PDT) (envelope-from gnn@neville-neil.com) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: George Neville-Neil In-Reply-To: <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Date: Sat, 17 Oct 2009 16:22:47 -0400 Content-Transfer-Encoding: 7bit Message-Id: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> To: Chuck Swiger X-Mailer: Apple Mail (2.1076) X-Spam-Score: undef - spam scanning disabled X-CanIt-Geo: ip=64.13.141.3; country=US; region=CA; city=Mountain View; postalcode=94039; latitude=37.3974; longitude=-122.0732; metrocode=807; areacode=650; http://maps.google.com/maps?q=37.3974,-122.0732&z=6 X-CanItPRO-Stream: default X-Canit-Stats-ID: Bayes signature not available X-Scanned-By: CanIt (www . roaringpenguin . com) on 64.13.141.13 X-Mailman-Approved-At: Sun, 18 Oct 2009 03:30:56 +0000 Cc: Steve Dong , freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2009 20:36:42 -0000 Hi, Trying to chime in with a few pointers here. Things to check when doing a TCP benchmark on FreeBSD. In particular make sure to adjust theses: net.inet.tcp.recvbuf_max: 262144 net.inet.tcp.recvbuf_inc: 16384 net.inet.tcp.recvbuf_auto: 1 net.inet.tcp.sendbuf_max: 262144 net.inet.tcp.sendbuf_inc: 8192 net.inet.tcp.sendbuf_auto: 1 Leave the auto on, but increase the max values and you should probably also change the inc (increment) values as well. Make sure that if you increase the buffer sizes you increase your number of mbufs and clusters as well. See kern.ipc.nmbclusters, which is a kernel tunable that can be set in /boot/loader.conf . Make sure that both of the systems you're testing have the same low level hardware support such as TCP Segment Offload (TSO) and TCP Checkusm Offload are turned on. Also you might want to turn this off: net.inet.tcp.inflight.enable: 1 This page http://fasterdata.es.net/TCP-tuning/FreeBSD.html claims that it can harm high speed connections. Those are the basics to start with. A search of "Tuning FreeBSD TCP" turns up some decent pages as well. Best, George From owner-freebsd-performance@FreeBSD.ORG Sun Oct 18 04:44:44 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0FAD7106566C for ; Sun, 18 Oct 2009 04:44:44 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-98-108-131-11.ptldor.fios.verizon.net [98.108.131.11]) by mx1.freebsd.org (Postfix) with ESMTP id E3BF28FC0A for ; Sun, 18 Oct 2009 04:44:42 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id 82479B653; Sat, 17 Oct 2009 15:15:20 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id EAA21373; Sun, 18 Oct 2009 04:40:38 GMT Message-Id: <200910180440.EAA21373@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Tue, 06 Oct 2009 18:03:16 +1100." <20091006174121.V25604@delplex.bde.org> Date: Sat, 17 Oct 2009 21:40:38 PDT From: Dieter Subject: Re: tuning FFS for large files Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2009 04:44:44 -0000 > > I found a clue! The problem occurs with my big data partitions, > > which are newfs-ed with options intended to improve things. > > > > Reading a large file from the normal ad4s5b partition only delays other > > commands slightly, as expected. Reading a large file from the tuned > > ad4s11 partition yields the delay of minutes for other i/o. > > ... > > Here is the newfs command used for creating large data partitions: > > newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition > > Any block size above the default (16K) tends to thrash and fragment buffer > cache virtual memory. This is obviously a good pessimization with lots of > small files, and apparently, as you have found, it is a good pessimization > with a few large files too. I think severe fragmentation can easily take > several seconds to recover from. The worst case for causing fragmentaion > is probably a mixture of small and large files. Is there any way to avoid the "thrash and fragment buffercache virtual memory" problem other than keeping the block size 16K or smaller? > Some users fear fs consistency bugs with block sizes >= 16K, but I've never > seen them cause any bugs ecept performance ones. Yep, many TB of files on filesystems created with above newfs command and no corruption/consistency problems. > > And they have way more inodes than needed. (IIRC it doesn't actually > > use -i 67108864) > > It has to have at least 1 inode per cg, and may as well have a full block > of them, which gives a fairly large number of inodes especially if the > block size is too large (64K), so the -i ratio is limited. I converted a few filesystems to the default. In addition to losing space, fsck time went through the roof. So back to playing with newfs options. For some reason, larger block/frag sizes allow fewer cylinder groups, which reduces the number of inodes more than the larger block size increases it. From my reading of the newfs man page, -c only allows making cylinder groups smaller, not larger, and that appears to be the case in practice. default: newfs -U /dev/ad14s4 /dev/ad14s4: 431252.6MB (883205320 sectors) block size 16384, fragment size 2048 using 2348 cylinder groups of 183.72MB, 11758 blks, 23552 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 417678 0 384263 0% 2 55300092 0% fsck -fp: real 0m37.165s Attempt to reduce number of inodes: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 /dev/ad14s4 density reduced from 134217728 to 3676160 /dev/ad14s4: 431252.6MB (883205320 sectors) block size 16384, fragment size 2048 using 1923 cylinder groups of 224.38MB, 14360 blks, 64 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431162 0 396669 0% 2 123068 0% fsck -fp: real 0m32.687s Bigger block size: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 -b 65536 /dev/ad14s4 increasing fragment size from 2048 to block size / 8 (8192) density reduced from 134217728 to 14860288 /dev/ad14s4: 431252.6MB (883205312 sectors) block size 65536, fragment size 8192 using 119 cylinder groups of 3628.00MB, 58048 blks, 256 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431230 0 396731 0% 2 30460 0% fsck -fp: real 0m3.144s Bigger block size and bigger frag size: newfs -U -i 134217728 -g 134217728 -h 16 -e 261129 -b 65536 -f 65536 /dev/ad14s4 density reduced from 134217728 to 66846720 /dev/ad14s4: 431252.6MB (883205248 sectors) block size 65536, fragment size 65536 using 27 cylinder groups of 16320.56MB, 261129 blks, 512 inodes. Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad14s4 431245 0 396745 0% 2 13820 0% fsck -fp: real 0m0.369s With -b 65536 -f 65536 I'm finally approaching a reasonable number of inodes (even less would be better). The fsck time varies by a factor of over 100, and results are roughly similar on filesystems with files in them. From owner-freebsd-performance@FreeBSD.ORG Sun Oct 18 14:55:02 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4002E1065705 for ; Sun, 18 Oct 2009 14:55:02 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f202.google.com (mail-pz0-f202.google.com [209.85.222.202]) by mx1.freebsd.org (Postfix) with ESMTP id 13B4B8FC1B for ; Sun, 18 Oct 2009 14:55:01 +0000 (UTC) Received: by pzk40 with SMTP id 40so2733526pzk.7 for ; Sun, 18 Oct 2009 07:55:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=ejE04OtxJRQDj4Sf9DqaASury5yTuaezXbh/Gcqz5ok=; b=YoLxIHE7hFI7/9AZZ8SEluT+UJ+aMqCfgDg2TWDTKjP+gC+poE7NI9iDOBF6/8hooq w0Dcppq8k5ePXuv8Kd0iD5lwsKe/bDAw6+iPZyIhp/ktO6P5dxpu5fVrljh2iYwhwALs Tmkzg4vTRrbeG80JEBggPOpzv+eDj3KRrHgbs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=SLrhrqAm46iWSqi7eICbtyH0rccP7z+11iWJ0m8u0o4AcFWx1DolUSN0obcy6KLpY6 uH5qy43b+m+zFHFpwhwUa5KcQyAEIQPKqV3b8daHbYd1BoA5ZPSMn1XTXWu6Dx9hWgra +eYsVm3mJY5RzD4N2FMGjYrXWdAZmpr1e1xIQ= MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.115.99.11 with SMTP id b11mr3911654wam.17.1255877701511; Sun, 18 Oct 2009 07:55:01 -0700 (PDT) In-Reply-To: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Sun, 18 Oct 2009 22:55:01 +0800 X-Google-Sender-Auth: 1f5f7e24aeefc048 Message-ID: From: Adrian Chadd To: Hongtao Yin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2009 14:55:02 -0000 FYI, I installed netperf on my local p4-D test boxes that I use for other testing. 128 byte send/receive buffers on the client side: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 128 128 10.00 426.17 1kbyte send/receive buffers: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 1024 1024 10.00 903.39 8kbyte send/receive buffers: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 8192 8192 10.00 913.71 Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some onboard flavour of the intel e1000 NIC: device =3D '82573E Intel Corporation 82573E Gigabit Ethernet Controller (Copper)' They are connected via a Cisco 3750G L3 switch. In fact, the traffic is routed, rather than switched. My /etc/sysctl.conf: net.inet.icmp.icmplim=3D0 net.inet.icmp.icmplim_output=3D0 net.inet.tcp.msl=3D3000 net.inet.tcp.sendspace=3D8192 net.inet.tcp.recvspace=3D8192 kern.maxfilesperproc=3D65536 kern.maxfiles=3D262144 kern.ipc.maxsockets=3D32768 kern.ipc.somaxconn=3D1024 kern.ipc.nmbclusters=3D131072 net.inet.ip.fw.enable=3D0 kern.ipc.somaxconn=3D10240 2c, Adrian 2009/10/15 Hongtao Yin : > Hi, > > > > I compared TCP performance between FreeBSD and Linux by running test tool= s > Netperf and Iperf with Intel NIC. > > The kernels are full version and default values are used in the testing > except TCP Congestion Control algorithm set to Reno. > > >From the test results we can see Linux TCP performance in throughput is > better than FreeBSD. The worst case (send msg size 128) shows that FreeBS= D > throughput is only 43% of Linux's. > > > > I like to get some feedback if anyone did similar comparison test, or kno= ws > any issues with kernels or drivers. Thanks lot. > > > > FreeBSD and Linux Sysctl captures are attached for reference. > > > > Regards, > > Hongtao > > > > > > > > Test Environments: > > PC: Dell Precision T3400 (same 4 PCs) > > CPU: Intel Core 2 Duo CPU E4600@2.4Ghz > > FreeBSD: V7.1 (full version) (TCP CC: newReno) > > Linux: V2.6.31.1 (full version) (TCP CC: Reno) > > Ethernet card: Intel Pro/1000 PWLA8492 MT Dual Port Server Adapter (Gigab= it) > chip 82546EB (only one port used for each PC) > > Switch: Netgear ProSafe 8 port Gigabit Switch (model GS108) > > Iperf: V2.0.4 > > Netperf: V2.4.4 > > > > Setup: > > > > > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0------= ---- > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0| swit= ch | > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0------= ---- > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 ---------------------| | | |-----------------= --- > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= | | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0--------| |-----= --- =A0 =A0 =A0 =A0 =A0 =A0 | > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 | > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0| =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 | > > =A0 =A0 =A0 =A0-------------- =A0-------------- =A0-------------- =A0----= ---------- > > =A0 =A0 =A0 =A0| =A0 =A0 PC1 =A0 =A0| =A0| =A0 =A0PC2 =A0 =A0 | =A0| =A0 = =A0PC3 =A0 =A0 | =A0| =A0 =A0PC4 =A0 =A0 | > > =A0 =A0 =A0 =A0| =A0 FreeBSD =A0| =A0| =A0 FreeBSD =A0| =A0| =A0 Linux = =A0 =A0| =A0| =A0 Linux =A0 =A0| > > =A0 =A0 =A0 =A0|192.168.1.10| =A0|192.168.1.20| =A0|192.168.1.30| =A0|192= .168.1.40| > > =A0 =A0 =A0 =A0-------------- =A0-------------- =A0-------------- =A0----= ---------- > > > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > =A0 =A0 Netperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > > > > TCP Throughput Test > > ------------------- > > > > PC2/4: #netserver -p 22113 > > PC1/3: #netperf -H 192.168.1.20 -p 22113 -l 10 > > > > =A0 =A0 =A0 =A0 =A0Recv =A0 =A0 =A0 =A0 =A0 =A0 Send =A0 =A0 =A0 =A0 =A0 = =A0Send =A0 =A0 =A0 =A0 =A0 Elapsed > Throughput > > =A0 =A0 =A0 =A0 =A0Socket =A0 =A0 =A0 =A0 =A0 Socket =A0 =A0 =A0 =A0 =A0M= essage =A0 =A0 =A0 =A0Time > 10^6 bits/sec > > =A0 =A0 =A0 =A0 =A0Size =A0 =A0 =A0 =A0 =A0 =A0 Size =A0 =A0 =A0 =A0 =A0 = =A0Size =A0 =A0 =A0 =A0 =A0 Sec. > > =A0 =A0 =A0 =A0 =A0bytes =A0 =A0 =A0 =A0 =A0 =A0bytes =A0 =A0 =A0 =A0 =A0= bytes > > FreeBSD: =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0 32768 = =A0 =A0 =A0 =A0 =A010.34 > 598.11 > > Linux: =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0 1638= 4 =A0 =A0 =A0 =A0 =A010.04 > 779.02 > > > > > > PC1/3: #netperf -t TCP_STREAM -H 192.168.1.20 -p 22113 -- -m > 64/128/256/512/1024/2048/4096 > > > > =A0 =A0 =A0 =A0 =A0Recv =A0 =A0 =A0 =A0 =A0 =A0 Send =A0 =A0 =A0 =A0 =A0 = =A0Send =A0 =A0 =A0 =A0 =A0 Elapsed > Throughput > > =A0 =A0 =A0 =A0 =A0Socket =A0 =A0 =A0 =A0 =A0 Socket =A0 =A0 =A0 =A0 =A0M= essage =A0 =A0 =A0 =A0Time > 10^6 bits/sec > > =A0 =A0 =A0 =A0 =A0Size =A0 =A0 =A0 =A0 =A0 =A0 Size =A0 =A0 =A0 =A0 =A0 = =A0Size =A0 =A0 =A0 =A0 =A0 Sec. > > =A0 =A0 =A0 =A0 =A0bytes =A0 =A0 =A0 =A0 =A0 =A0bytes =A0 =A0 =A0 =A0 =A0= bytes > > FreeBSD: =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0 64 =A0= =A0 =A0 =A0 =A0 =A0 =A010.19 > 417.10 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 128 =A0 =A0 =A0 =A0 =A0 =A0 10.35 > 336.63 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 256 =A0 =A0 =A0 =A0 =A0 =A0 10.36 > 576.99 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 512 =A0 =A0 =A0 =A0 =A0 =A0 10.35 > 569.79 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 1024 =A0 =A0 =A0 =A0 =A0 =A010.35 > 553.70 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 2048 =A0 =A0 =A0 =A0 =A0 =A010.35 > 584.20 > > =A0 =A0 =A0 =A0 =A065536 =A0 =A0 =A0 =A0 =A0 =A032768 =A0 =A0 =A0 =A0 =A0= 4096 =A0 =A0 =A0 =A0 =A0 =A010.35 > 602.45 > > > > Linux: =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0 64 = =A0 =A0 =A0 =A0 =A0 =A0 =A010.03 > 778.21 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 128 =A0 =A0 =A0 =A0 =A0 =A0 10.03 > 779.72 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 256 =A0 =A0 =A0 =A0 =A0 =A0 10.04 > 780.16 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 512 =A0 =A0 =A0 =A0 =A0 =A0 10.03 > 776.85 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 1024 =A0 =A0 =A0 =A0 =A0 =A010.04 > 777.52 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 2048 =A0 =A0 =A0 =A0 =A0 =A010.04 > 777.83 > > =A0 =A0 =A0 =A0 =A087380 =A0 =A0 =A0 =A0 =A0 =A016384 =A0 =A0 =A0 =A0 =A0= 4096 =A0 =A0 =A0 =A0 =A0 =A010.03 > 780.17 > > > > > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > =A0 =A0 =A0Iperf Test Results > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > > > > Bandwidth Test > > -------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 > > > > =A0 =A0 =A0 =A0 =A0 Interval =A0 =A0 =A0 =A0Transfer =A0 =A0 =A0 =A0Bandw= idth > > =A0 =A0 =A0 =A0 =A0 sec =A0 =A0 =A0 =A0 =A0 =A0 MBytes =A0 =A0 =A0 =A0 = =A0Mbits/sec > > FreeBSD: =A0 0.0-10.3 =A0 =A0 =A0 =A0740 =A0 =A0 =A0 =A0 =A0 =A0 600 > > Linux: =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A0972 =A0 =A0 =A0 =A0 =A0 =A0 815 > > > > > > PC1/3: #iperf -c 192.168.1.20 -d > > > > =A0 =A0 =A0 =A0 =A0 Interval =A0 =A0 =A0 =A0Transfer =A0 =A0 =A0 =A0Bandw= idth > > =A0 =A0 =A0 =A0 =A0 sec =A0 =A0 =A0 =A0 =A0 =A0 MBytes =A0 =A0 =A0 =A0 = =A0Mbits/sec > > FreeBSD: =A0 0.0-10.0 =A0 =A0 =A0 =A0402 =A0 =A0 =A0 =A0 =A0 =A0 337 > > =A0 =A0 =A0 =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A0404 =A0 =A0 =A0 =A0 =A0 =A0 3= 38 > > > > Linux: =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A0926 =A0 =A0 =A0 =A0 =A0 =A0 776 > > =A0 =A0 =A0 =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A044.1 =A0 =A0 =A0 =A0 =A0 =A03= 6.9 > > > > > > Parallel Test > > ------------- > > > > PC2/4: #iperf -s > > PC1/3: #iperf -c 192.168.1.20 -P 2 > > > > =A0 =A0 =A0 =A0 =A0 Interval =A0 =A0 =A0 =A0Transfer =A0 =A0 =A0 =A0Bandw= idth > > =A0 =A0 =A0 =A0 =A0 sec =A0 =A0 =A0 =A0 =A0 =A0 MBytes =A0 =A0 =A0 =A0 = =A0Mbits/sec > > FreeBSD: =A0 0.0-10.3 =A0 =A0 =A0 =A0370 =A0 =A0 =A0 =A0 =A0 =A0 300 > > =A0 =A0 =A0 =A0 =A0 0.0-10.3 =A0 =A0 =A0 =A0370 =A0 =A0 =A0 =A0 =A0 =A0 3= 00 > > =A0 =A0 =A0SUM: 0.0-10.3 =A0 =A0 =A0 =A0739 =A0 =A0 =A0 =A0 =A0 =A0 600 > > > > Linux: =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A0479 =A0 =A0 =A0 =A0 =A0 =A0 402 > > =A0 =A0 =A0 =A0 =A0 0.0-10.0 =A0 =A0 =A0 =A0473 =A0 =A0 =A0 =A0 =A0 =A0 3= 96 > > =A0 =A0 =A0SUM: 0.0-10.0 =A0 =A0 =A0 =A0952 =A0 =A0 =A0 =A0 =A0 =A0 797 > > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" > From owner-freebsd-performance@FreeBSD.ORG Sun Oct 18 20:35:30 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A45B2106566C; Sun, 18 Oct 2009 20:35:30 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from mail-px0-f186.google.com (mail-px0-f186.google.com [209.85.216.186]) by mx1.freebsd.org (Postfix) with ESMTP id 7FB348FC1A; Sun, 18 Oct 2009 20:35:30 +0000 (UTC) Received: by pxi16 with SMTP id 16so2161894pxi.3 for ; Sun, 18 Oct 2009 13:35:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.142.3.35 with SMTP id 35mr249229wfc.205.1255896323541; Sun, 18 Oct 2009 13:05:23 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Sun, 18 Oct 2009 13:05:23 -0700 Message-ID: From: Brent Jones To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Hongtao Yin , freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2009 20:35:30 -0000 On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: > FYI, I installed netperf on my local p4-D test boxes that I use for > other testing. > > 128 byte send/receive buffers on the client side: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv =A0 Send =A0 =A0Send > Socket Socket =A0Message =A0Elapsed > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec > > =A08192 =A0 =A0128 =A0 =A0128 =A0 =A010.00 =A0 =A0 426.17 > > 1kbyte send/receive buffers: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv =A0 Send =A0 =A0Send > Socket Socket =A0Message =A0Elapsed > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec > > =A08192 =A0 1024 =A0 1024 =A0 =A010.00 =A0 =A0 903.39 > > 8kbyte send/receive buffers: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv =A0 Send =A0 =A0Send > Socket Socket =A0Message =A0Elapsed > Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput > bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec > > =A08192 =A0 8192 =A0 8192 =A0 =A010.00 =A0 =A0 913.71 > > Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some > onboard flavour of the intel e1000 NIC: > > =A0 =A0device =A0 =A0 =3D '82573E Intel Corporation 82573E Gigabit Ethern= et > Controller (Copper)' > > They are connected via a Cisco 3750G L3 switch. In fact, the traffic > is routed, rather than switched. > > My /etc/sysctl.conf: > > net.inet.icmp.icmplim=3D0 > net.inet.icmp.icmplim_output=3D0 > net.inet.tcp.msl=3D3000 > net.inet.tcp.sendspace=3D8192 > net.inet.tcp.recvspace=3D8192 > kern.maxfilesperproc=3D65536 > kern.maxfiles=3D262144 > kern.ipc.maxsockets=3D32768 > kern.ipc.somaxconn=3D1024 > kern.ipc.nmbclusters=3D131072 > net.inet.ip.fw.enable=3D0 > kern.ipc.somaxconn=3D10240 > > 2c, > > > Adrian > > 2009/10/15 Hongtao Yin : >> Hi, >> >> Can you try with 64K and up tp 1MB buffers? I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and cheap Netgear switches. I'm not sure how the original tester got such poor numbers, when my setup is relatively low end, and sustaining Gbit speeds is no major feat. --=20 Brent Jones brent@servuhome.net From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 01:36:37 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86F64106566B for ; Mon, 19 Oct 2009 01:36:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f202.google.com (mail-pz0-f202.google.com [209.85.222.202]) by mx1.freebsd.org (Postfix) with ESMTP id 5855E8FC14 for ; Mon, 19 Oct 2009 01:36:37 +0000 (UTC) Received: by pzk40 with SMTP id 40so2933330pzk.7 for ; Sun, 18 Oct 2009 18:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=lm0lzEZp5H+M+bXqnnsbCEYa/t32SL8xGL0gf7hI9dM=; b=YEeDsA7VWu0BzwZVXwX1VpZfmUGyFJ6DX/wvu5OHu7oizxzsOaycVvmyN/eQhqKZ7m CPjG1FaGgDcsdFJdEHicFdGJ9o3LvnXpoVJEDq7Sa7kL5tM6eyJ+FvWFygkB+hdng8rs qGeaFl/wdA7Sygp2g80UtVbaN0C6oAGD7gvwE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=W2MiE9fpXXaKK3vVTk9pSL6r9ShJpUESiOJz912wQHjRUaH3WwGD0PHQM0lK1HX0G5 OmWFdhoR6+1oWC7qDy1fR4/b9vCxNUIzhN4Ik3SGHZh+gsIKzx0/LtJ4Gc/ETDKXuw1Y CnLht2Q/bg9j3L8kQsRFR3OsQA2npk8X9DPzY= MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.115.113.9 with SMTP id q9mr4858618wam.224.1255916196870; Sun, 18 Oct 2009 18:36:36 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Mon, 19 Oct 2009 09:36:36 +0800 X-Google-Sender-Auth: 38f9c9dd18bd5be1 Message-ID: From: Adrian Chadd To: Brent Jones Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Hongtao Yin , freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 01:36:37 -0000 uhm: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 65536 65536 10.00 862.48 1 megabyte socket buffers threw an error. I'll see why later. Now, as for why 64k socket buffers gave a slower result than 8k socket buffers... ah. If I change the sending end to use 64k socket buffers: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 65536 65536 65536 10.00 916.23 Adrian 2009/10/19 Brent Jones : > On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: >> FYI, I installed netperf on my local p4-D test boxes that I use for >> other testing. >> >> 128 byte send/receive buffers on the client side: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv =A0 Send =A0 =A0Send >> Socket Socket =A0Message =A0Elapsed >> Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput >> bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec >> >> =A08192 =A0 =A0128 =A0 =A0128 =A0 =A010.00 =A0 =A0 426.17 >> >> 1kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv =A0 Send =A0 =A0Send >> Socket Socket =A0Message =A0Elapsed >> Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput >> bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec >> >> =A08192 =A0 1024 =A0 1024 =A0 =A010.00 =A0 =A0 903.39 >> >> 8kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv =A0 Send =A0 =A0Send >> Socket Socket =A0Message =A0Elapsed >> Size =A0 Size =A0 =A0Size =A0 =A0 Time =A0 =A0 Throughput >> bytes =A0bytes =A0 bytes =A0 =A0secs. =A0 =A010^6bits/sec >> >> =A08192 =A0 8192 =A0 8192 =A0 =A010.00 =A0 =A0 913.71 >> >> Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some >> onboard flavour of the intel e1000 NIC: >> >> =A0 =A0device =A0 =A0 =3D '82573E Intel Corporation 82573E Gigabit Ether= net >> Controller (Copper)' >> >> They are connected via a Cisco 3750G L3 switch. In fact, the traffic >> is routed, rather than switched. >> >> My /etc/sysctl.conf: >> >> net.inet.icmp.icmplim=3D0 >> net.inet.icmp.icmplim_output=3D0 >> net.inet.tcp.msl=3D3000 >> net.inet.tcp.sendspace=3D8192 >> net.inet.tcp.recvspace=3D8192 >> kern.maxfilesperproc=3D65536 >> kern.maxfiles=3D262144 >> kern.ipc.maxsockets=3D32768 >> kern.ipc.somaxconn=3D1024 >> kern.ipc.nmbclusters=3D131072 >> net.inet.ip.fw.enable=3D0 >> kern.ipc.somaxconn=3D10240 >> >> 2c, >> >> >> Adrian >> >> 2009/10/15 Hongtao Yin : >>> Hi, >>> >>> > > Can you try with 64K and up tp 1MB buffers? > > I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and > cheap Netgear switches. > I'm not sure how the original tester got such poor numbers, when my > setup is relatively low end, and sustaining Gbit speeds is no major > feat. > > > > -- > Brent Jones > brent@servuhome.net > From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 10:34:51 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C40F106568D; Mon, 19 Oct 2009 10:34:51 +0000 (UTC) (envelope-from prvs=15431f176b=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id CE0408FC19; Mon, 19 Oct 2009 10:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1255947856; x=1256552656; q=dns/txt; h=Received: Message-ID:From:To:Cc:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=hrx4fJ+tpq5J5cTmMvthf hkikvJmAnP5NnN+z9k/whA=; b=FkY7NwB9lJ6PJGVsu1EJhEGmFjanjpeXCmglj mgvS2s+f+82VuibWzzVWO0kZLS36+2rXWv4NdX+dxe4ArjzjEQBSDJ71XmC+01Nh AQJvw/ZkfPbOGkMBgScy3dCgn7x4U8+AId3qoqtgq7P5s+xyPeTNPsA03ReO4gWj OaeKfA= X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 19 Oct 2009 11:24:16 +0100 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008391556.msg; Mon, 19 Oct 2009 11:24:15 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 19 Oct 2009 11:24:15 +0100 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=15431f176b=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Adrian Chadd" , "Brent Jones" References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Mon, 19 Oct 2009 11:24:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Cc: Hongtao Yin , freebsd-performance@freebsd.org Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 10:34:51 -0000 Try with something like this, which is the standard set we use on our file serving machines. net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 Regards Steve ----- Original Message ----- From: "Adrian Chadd" To: "Brent Jones" Cc: "Hongtao Yin" ; Sent: Monday, October 19, 2009 2:36 AM Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance uhm: kristy# netperf -H 192.168.10.2 -p 22113 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 8192 65536 65536 10.00 862.48 1 megabyte socket buffers threw an error. I'll see why later. Now, as for why 64k socket buffers gave a slower result than 8k socket buffers... ah. If I change the sending end to use 64k socket buffers: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 (192.168.10.2) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 65536 65536 65536 10.00 916.23 Adrian 2009/10/19 Brent Jones : > On Sun, Oct 18, 2009 at 7:55 AM, Adrian Chadd wrote: >> FYI, I installed netperf on my local p4-D test boxes that I use for >> other testing. >> >> 128 byte send/receive buffers on the client side: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 128 128 10.00 426.17 >> >> 1kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 1024 1024 10.00 903.39 >> >> 8kbyte send/receive buffers: >> >> kristy# netperf -H 192.168.10.2 -p 22113 -l 10 >> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 >> (192.168.10.2) port 0 AF_INET >> Recv Send Send >> Socket Socket Message Elapsed >> Size Size Size Time Throughput >> bytes bytes bytes secs. 10^6bits/sec >> >> 8192 8192 8192 10.00 913.71 >> >> Both boxes are 7.2-REL amd64 boxes on 3.4GHz Pentium-D CPUs using some >> onboard flavour of the intel e1000 NIC: >> >> device = '82573E Intel Corporation 82573E Gigabit Ethernet >> Controller (Copper)' >> >> They are connected via a Cisco 3750G L3 switch. In fact, the traffic >> is routed, rather than switched. >> >> My /etc/sysctl.conf: >> >> net.inet.icmp.icmplim=0 >> net.inet.icmp.icmplim_output=0 >> net.inet.tcp.msl=3000 >> net.inet.tcp.sendspace=8192 >> net.inet.tcp.recvspace=8192 >> kern.maxfilesperproc=65536 >> kern.maxfiles=262144 >> kern.ipc.maxsockets=32768 >> kern.ipc.somaxconn=1024 >> kern.ipc.nmbclusters=131072 >> net.inet.ip.fw.enable=0 >> kern.ipc.somaxconn=10240 >> >> 2c, >> >> >> Adrian >> >> 2009/10/15 Hongtao Yin : >>> Hi, >>> >>> > > Can you try with 64K and up tp 1MB buffers? > > I see ~1Gbit speeds with my FreeBSD boxes using Broadcom NIC's and > cheap Netgear switches. > I'm not sure how the original tester got such poor numbers, when my > setup is relatively low end, and sustaining Gbit speeds is no major > feat. > > > > -- > Brent Jones > brent@servuhome.net > _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 11:44:00 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA184106568D for ; Mon, 19 Oct 2009 11:44:00 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 8480D8FC14 for ; Mon, 19 Oct 2009 11:44:00 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1Mzqar-0001Pq-FX for freebsd-performance@freebsd.org; Mon, 19 Oct 2009 13:40:05 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:40:05 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:40:05 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 19 Oct 2009 13:35:57 +0200 Lines: 16 Message-ID: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) In-Reply-To: <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> Sender: news Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 11:44:00 -0000 Steve Dong wrote: > It looks the jpeg attachments were somehow dropped. Trying again with pdf > attachment. Hopefully it works this time. > Hi, I haven't tried comparing this sort of performance with Linux so your conclusion still might be right, but the fact that you couldn't saturate 1 Gbps on either system even with big packets suggests that there might be an external problem - a bad network card or a bad driver for the network card, or a switch whose line discipline is a bit in conflict with the NIC or the driver. I have previously successfully (and rather trivially) saturated 1 Gbps links with Broadcom cards with FreeBSD 7.x, so it *is* possible. From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 11:48:11 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E77431065676 for ; Mon, 19 Oct 2009 11:48:11 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 7345F8FC1C for ; Mon, 19 Oct 2009 11:48:11 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1Mzqfg-0004Xz-7c for freebsd-performance@freebsd.org; Mon, 19 Oct 2009 13:45:04 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:45:04 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:45:04 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 19 Oct 2009 13:42:31 +0200 Lines: 28 Message-ID: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> <4831593800614E6796A45F20BA4B818E@china.huawei.com> <001301ca4e23$b96e35b0$3322c10a@china.huawei.com> <001c01ca4e24$f10f6e70$3322c10a@china.huawei.com> <20091016075336.03eb17f2.wmoran@collaborativefusion.com> <000001ca4f3c$78dc3550$6501a8c0@china.huawei.com> <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) In-Reply-To: <0D9BF9F3-FDA1-4111-9E6C-733E1FD972F5@mac.com> Sender: news Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 11:48:12 -0000 Chuck Swiger wrote: > Hi, Steve-- > > On Oct 17, 2009, at 8:14 AM, Steve Dong wrote: >> If there's a better/lighter way to show these graphics, I'd like to know. > > Sure-- put 'em on a webserver somewhere, and put links to them in your > email to this mailing list. > > If you wanted to do even better than that, set up a simple webpage > describing what you are doing in your comparison, have a link to the > dmesg/boot output for each platform as a .txt file and a description of > any system tweaks & tuning, have a link that points to a description of > the test setup (ie, your ASCII diagram of the switch and 4 machines), > then your graphs, then the raw data (or links to it, depending). You > can then throw in netstat -s output, or NIC driver stats from sysctl, or > switch stats, etc-- anything else that adds useful context. > > There are a fair number of posts in the list archives which describe how > to benchmark reliably, and the people who are most likely to be making > code changes to FreeBSD also tend to like to know whether you've > collected enough data, in a controlled fashion, to have an idea as to > whether your measurements are reproducible. I'm not a purist, and I Also, the OP should take a look at some previous benchmarks and the link to benchmark advices here: http://wiki.freebsd.org/BenchmarkMatrix From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 11:50:17 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85FE11065693 for ; Mon, 19 Oct 2009 11:50:17 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 3F9AF8FC15 for ; Mon, 19 Oct 2009 11:50:17 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1MzqkV-0007vV-SY for freebsd-performance@freebsd.org; Mon, 19 Oct 2009 13:50:03 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:50:03 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 19 Oct 2009 13:50:03 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 19 Oct 2009 13:44:59 +0200 Lines: 11 Message-ID: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) In-Reply-To: Sender: news Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 11:50:17 -0000 Steven Hartland wrote: > Try with something like this, which is the standard set we use on our > file serving machines. > > net.inet.tcp.inflight.enable=0 > net.inet.tcp.sendspace=65536 > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 16 MB network buffers? What kind of % impact do you see from them? From owner-freebsd-performance@FreeBSD.ORG Mon Oct 19 13:41:05 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D35E0106566B for ; Mon, 19 Oct 2009 13:41:05 +0000 (UTC) (envelope-from prvs=15431f176b=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 400B18FC0C for ; Mon, 19 Oct 2009 13:41:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1255959662; x=1256564462; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=yKNQzAoktXAL36TYXIqu9 WNnxi49Z/vlxNS3bL/CTuM=; b=MroyX3eote8uVsC4SYLXiZnZM+GNPuQJIFXe9 u/2ZUpLFzB2R0Ugz7c/PxxIJCnRpSMb6RWy+6p3y/vae0hR+pAcHKrPrypl2DhXO F4TjaA0q0pK6xcCzLdzroduI7ttz219fbhDlKhmraIsOtC44W3ovnNW0wnVzKwXA XVw9lE= X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 19 Oct 2009 14:41:02 +0100 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008392450.msg; Mon, 19 Oct 2009 14:41:02 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 19 Oct 2009 14:41:02 +0100 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=15431f176b=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: , "Ivan Voras" References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Mon, 19 Oct 2009 14:40:26 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Cc: Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2009 13:41:05 -0000 Its max not default, so relies on your configuring each app you want to have high performance to take advantage of it. In our case that means our large transfers easily saturate Gig. Regards Steve ----- Original Message ----- From: "Ivan Voras" To: Sent: Monday, October 19, 2009 12:44 PM Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance > Steven Hartland wrote: >> Try with something like this, which is the standard set we use on our >> file serving machines. >> >> net.inet.tcp.inflight.enable=0 >> net.inet.tcp.sendspace=65536 >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 > > 16 MB network buffers? What kind of % impact do you see from them? ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue Oct 20 00:17:00 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E23CC1065676; Tue, 20 Oct 2009 00:17:00 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id 493D78FC19; Tue, 20 Oct 2009 00:16:59 +0000 (UTC) Received: by fxm6 with SMTP id 6so5492409fxm.43 for ; Mon, 19 Oct 2009 17:16:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=D2Db1XjVmYiV4RNznZKdT0OuHpQ6RD60460J+icjiTU=; b=YHvMU/c49LpPGrDZfRD7N5uEmxy7806YKhOK+sIGlcxTgpbqJUzclfestRyFuT7Rv8 AkwMz10KtEa24kIYAyCHjP5oYTTRp527OTIlbz/k2qHGEuuUbF3IHLWdJuEspNcgEKCF /lVKw9PM++rVmu7NFVOcTRrKs9h29ZEW62HmA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=KqiQfckK4ap9uCN2FZL6G3fDaifPb/WnHaxvtSZnEhy8zKgK8JS2Y9vNdDukc+8ZO7 VxLncOT7t3SVrmqQuuM3pwjy4OOO1rv6Ls/rbJFQrkUNIB/XsjolyK2rrOwuWoITdqod kH9Nn9DbYgTtRC/dli9jQZsuGkN35mFZLgptU= MIME-Version: 1.0 Received: by 10.204.34.203 with SMTP id m11mr5600848bkd.79.1255997817892; Mon, 19 Oct 2009 17:16:57 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Tue, 20 Oct 2009 01:16:57 +0100 Message-ID: From: =?UTF-8?Q?Istv=C3=A1n?= To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Hongtao Yin , freebsd-performance@freebsd.org, Brent Jones Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2009 00:17:01 -0000 On Mon, Oct 19, 2009 at 2:36 AM, Adrian Chadd wrote: > uhm: > > kristy# netperf -H 192.168.10.2 -p 22113 -l 10 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 8192 65536 65536 10.00 862.48 > > 1 megabyte socket buffers threw an error. I'll see why later. > > Now, as for why 64k socket buffers gave a slower result than 8k socket > buffers... ah. If I change the sending end to use 64k socket buffers: > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.2 > (192.168.10.2) port 0 AF_INET > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 65536 65536 65536 10.00 916.23 > > > > Adrian > > therefore i like netpipe runs you can see the performance and the latency as well using the packet size as your "x" axis, i think it makes more sense then just 1 number -- the sun shines for all http://l1xl1x.blogspot.com From owner-freebsd-performance@FreeBSD.ORG Tue Oct 20 02:39:54 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 874841065670 for ; Tue, 20 Oct 2009 02:39:54 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yw0-f178.google.com (mail-yw0-f178.google.com [209.85.211.178]) by mx1.freebsd.org (Postfix) with ESMTP id 386648FC18 for ; Tue, 20 Oct 2009 02:39:54 +0000 (UTC) Received: by ywh8 with SMTP id 8so4525184ywh.3 for ; Mon, 19 Oct 2009 19:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=JhGEWTJtD9CsOPmG9Zh0zufE2Hmx53DXj4EAwPNfjf8=; b=Lb5NsvI0TIIjWP8MwRV37QKFBVqN7NBioiR0fISOv221aCtXkXBcRJtlIOnSHHkOTs ZeUfHiISacDzefk+A4U9FnIWZTHfrF1S0t9gTXMDJirYQjZD21aK5ebtK3rLSkH+8aG/ aXrXECFxjZPD3m79Hy7MpJUtPGpTYqEYqeXSE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=DsU/NXyE0toHTqiEZ0O597k1WenZJq4EWvDGxNZZXRefTqr3hHBqfj6LN7oOy+bpEE QDDX8EO+7aH4GCiG98v3QjfRkB317NJnEWdzGiLkU7TLMJ6qhNSenv4OL1G/vsV9pIbx KpC0drlLOxDo5jlrOyX4Fgwbn2VSRDfImWNio= MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.101.147.1 with SMTP id z1mr3717259ann.140.1256006393526; Mon, 19 Oct 2009 19:39:53 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Tue, 20 Oct 2009 10:39:53 +0800 X-Google-Sender-Auth: f439659a3c80c593 Message-ID: From: Adrian Chadd To: =?ISO-8859-1?B?SXN0duFu?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Hongtao Yin , freebsd-performance@freebsd.org, Brent Jones Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2009 02:39:54 -0000 2009/10/20 Istv=E1n : > therefore i like netpipe runs you can see the performance and the latency= as > well using the packet size as your "x" axis, i think it makes more sense > then just 1 number My point was to demonstrate that saturating gigabit ethernet is very doable with FreeBSD, and his limitation is more likely somewhere other than "TCP". I've told him privately to check CPU utilisation. I'll do the same on my boxes when I get some time; I'd like to know why I'm only seeing ~ 800mbit with large buffers. From owner-freebsd-performance@FreeBSD.ORG Tue Oct 20 07:05:39 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 361651065670 for ; Tue, 20 Oct 2009 07:05:39 +0000 (UTC) (envelope-from pathiaki2@yahoo.com) Received: from web110506.mail.gq1.yahoo.com (web110506.mail.gq1.yahoo.com [67.195.8.254]) by mx1.freebsd.org (Postfix) with SMTP id 0272C8FC18 for ; Tue, 20 Oct 2009 07:05:38 +0000 (UTC) Received: (qmail 91079 invoked by uid 60001); 20 Oct 2009 06:38:58 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1256020738; bh=lRCcBh6+Vx0mjoWdUp640PjZMZFs+cjdN7PLBMYnjVU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=X9cQil3fRJm0PpE3gU8vqEaVdZjp89LEg0+WWI6vgZ7PHztbFruz8dXypN0KSWLkA91L1Y3PLH43q5znSK64zrKNb8uGPpoBVy3PyxSAzhLqGB8ockd+AhxJZQh7loRr0TOoK1wdiaCrTwcd8Cp95EWFoUYBpq6G2WwIiYHV+V0= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=UCjccpijyxGd90LoB4y6b0ncLn3nEM6XE1PuIq0op+zWAbM5wQU/pd0VCxDRr0RY7YPStE/yg/WnrcBES0Xk2cc4P7m3huBaVzUCcDvRprgGrBKWvqmrvmwwG1eWUu23fxp5SMqxhW2sUzmKoVXLA8FZgxYSJwxwvNHFo66ZbR0=; Message-ID: <687096.90163.qm@web110506.mail.gq1.yahoo.com> X-YMail-OSG: aW9neU4VM1lJBI6sk4bISJ2mC_ptcU7qPo4Q1sw_A2HBzfndWmm9LgFs Received: from [71.174.61.120] by web110506.mail.gq1.yahoo.com via HTTP; Mon, 19 Oct 2009 23:38:58 PDT X-Mailer: YahooMailRC/182.10 YahooMailWebService/0.7.347.3 References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Mon, 19 Oct 2009 23:38:58 -0700 (PDT) From: Paul Pathiakis To: Adrian Chadd , =?iso-8859-1?Q?Istv=E1n?= In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Hongtao Yin , freebsd-performance@freebsd.org, Brent Jones Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2009 07:05:39 -0000 Hi,=0A=0Agoing to chime in on this one....just trying to help.=0A=0AThere's= some simple things to get Gb, jumbo frames (MTU > 1500 on both the switch = port and the card) is a simple way.=0A=0AHowever, I'd have to read back on = this thread as I haven't had time of late. Basically, and I've seen this o= n many, many Gb cards, chipsets and Drives make the world of difference. = =0A=0AI tried for a few days to try and get an HP DL360 with it's dual on-b= oard Broadcom bge NIC to get to 1 Gb.... just plain no way. If anyone has = settings for that, I'd like to know them. Also, this is the same chip set = that a lot of vendors use and it is cheap and inexpensive. When I couldn't= get the thing to go beyond 720Mb, I tried something simple. I ordered an = Intel dual Gb port card and put that in. WITHOUT tuning, this thing starte= d at almost 800 Mb throughput and I almost got it to 850 Mb within a few ho= urs.=0A=0AI wish I could send those settings to this list but it was well o= ver a year ago that I did this.=0A=0ASadly, most large vendors start with B= roadcom chipsets and don't want to spent the extra $10 for the Intel chipse= t. (No, I am not a fan boy of Intel, more of AMD if anything, but their NI= Cs rock.)=0A=0AP.=0A=0A=0A=0A=0A________________________________=0AFrom: Ad= rian Chadd =0ATo: Istv=E1n =0ACc: Ho= ngtao Yin ; freebsd-performance@freebsd.org; Brent Jones = =0ASent: Mon, October 19, 2009 10:39:53 PM=0ASubject: = Re: Comparison of FreeBSD/Linux TCP Throughput performance=0A=0A2009/10/20 = Istv=E1n :=0A=0A=0A> therefore i like netpipe runs you c= an see the performance and the latency as=0A> well using the packet size as= your "x" axis, i think it makes more sense=0A> then just 1 number=0A=0AMy = point was to demonstrate that saturating gigabit ethernet is very=0Adoable = with FreeBSD, and his limitation is more likely somewhere other=0Athan "TCP= ".=0A=0AI've told him privately to check CPU utilisation. I'll do the same = on=0Amy boxes when I get some time; I'd like to know why I'm only seeing ~= =0A800mbit with large buffers.=0A__________________________________________= _____=0Afreebsd-performance@freebsd.org mailing list=0Ahttp://lists.freebsd= .org/mailman/listinfo/freebsd-performance=0ATo unsubscribe, send any mail t= o "freebsd-performance-unsubscribe@freebsd.org"=0A=0A=0A=0A From owner-freebsd-performance@FreeBSD.ORG Tue Oct 20 08:13:07 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 235671065672; Tue, 20 Oct 2009 08:13:07 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id 7F7498FC19; Tue, 20 Oct 2009 08:13:06 +0000 (UTC) Received: by fxm6 with SMTP id 6so5796453fxm.43 for ; Tue, 20 Oct 2009 01:13:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=sTv/7T8hHj+50sTFFkvQiUcnJ1veyplreir7Egnf4vo=; b=P80RfcbdZTafBrRLgVkT6SJPxQCQimgwLNgK68MTa50XV1Q/kopvKoKkkpqqwnrcpq CaPgnweRfZlodbFqVALAZZQODXRsU48wfw9b8ud5txrQ8O0dTrOqJDE369xwd0aIOZja jN1j4CMfr1bB9aeYsn3/i8sZrPguYlEYTtDBk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=B7XEPtp84f351HRToDZmOcJQzSEJffjW9sRr6SfGkH/e98uL9HgwzXxFNrDSoXKE0n xRJPYMnqrTMX8TEDcqwtk0gs5dbB8TX1CVBkl7tWOFl3PJe9tMha2EsChCtoyxT7ndKa NWzdSuJoUW4IAPmJZWgwSjnXCfKjAOrMS7snU= MIME-Version: 1.0 Received: by 10.204.160.65 with SMTP id m1mr6123494bkx.193.1256026385231; Tue, 20 Oct 2009 01:13:05 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Tue, 20 Oct 2009 09:13:05 +0100 Message-ID: From: =?UTF-8?Q?Istv=C3=A1n?= To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Hongtao Yin , freebsd-performance@freebsd.org, Brent Jones Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2009 08:13:07 -0000 i see but there was no debt that it is possible. at least from my side :_) On Tue, Oct 20, 2009 at 3:39 AM, Adrian Chadd wrote: > 2009/10/20 Istv=C3=A1n : > > > > therefore i like netpipe runs you can see the performance and the laten= cy > as > > well using the packet size as your "x" axis, i think it makes more sens= e > > then just 1 number > > My point was to demonstrate that saturating gigabit ethernet is very > doable with FreeBSD, and his limitation is more likely somewhere other > than "TCP". > > I've told him privately to check CPU utilisation. I'll do the same on > my boxes when I get some time; I'd like to know why I'm only seeing ~ > 800mbit with large buffers. > --=20 the sun shines for all http://l1xl1x.blogspot.com From owner-freebsd-performance@FreeBSD.ORG Tue Oct 20 08:14:30 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 847B5106566B; Tue, 20 Oct 2009 08:14:30 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id DE5318FC1C; Tue, 20 Oct 2009 08:14:29 +0000 (UTC) Received: by fxm6 with SMTP id 6so5797636fxm.43 for ; Tue, 20 Oct 2009 01:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=hmPmg+d9b+8WL73aRWGacTCZb4Ae69K3T1tsqRCUe/8=; b=DUdIYEdUKmgKKtLmlB73860VtAdRUyxE1z0VwEl/xu000H1IkgoB4WWRjdt+Io76WW 5nP5qgDaa9CbW8LEXDYMQv8rN7VOb1Hh8LvGTRLJ1pm6Wr5hEwJc6ljWHSXzIRPrf7wV gUsk5Ck1tGRDW+6O0uuIHW9nL7XyX6Nt58RgU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OA75P5SqlJ7h/Mqp9aJQLAXXrT/RokEuNOjf38dwAhSat+F/Mu4VUcDifdfS2T20AC J3DAfu7jJBmsA9NQ7v2Np0JAZ8asX3/mwr79a6HLUqsl1fr8n7O0ObKdoKAWjBoigMZU eMk6V9HLi7xVnvuqB4SkHR+L0v9uTlxhHC0QU= MIME-Version: 1.0 Received: by 10.204.33.194 with SMTP id i2mr5960630bkd.146.1256026467968; Tue, 20 Oct 2009 01:14:27 -0700 (PDT) In-Reply-To: References: <78DB4AE8EF5F4A1EBD3992D7404B2725@china.huawei.com> Date: Tue, 20 Oct 2009 09:14:27 +0100 Message-ID: From: =?UTF-8?Q?Istv=C3=A1n?= To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Hongtao Yin , freebsd-performance@freebsd.org, Brent Jones Subject: Re: Comparison of FreeBSD/Linux TCP Throughput performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2009 08:14:30 -0000 i mean doubt, shouldn't do mailing before the first coffee :))) On Tue, Oct 20, 2009 at 9:13 AM, Istv=C3=A1n wrote: > i see > but there was no debt that it is possible. at least from my side :_) > > > On Tue, Oct 20, 2009 at 3:39 AM, Adrian Chadd wrote: > >> 2009/10/20 Istv=C3=A1n : >> >> >> > therefore i like netpipe runs you can see the performance and the >> latency as >> > well using the packet size as your "x" axis, i think it makes more sen= se >> > then just 1 number >> >> My point was to demonstrate that saturating gigabit ethernet is very >> doable with FreeBSD, and his limitation is more likely somewhere other >> than "TCP". >> >> I've told him privately to check CPU utilisation. I'll do the same on >> my boxes when I get some time; I'd like to know why I'm only seeing ~ >> 800mbit with large buffers. >> > > > > -- > the sun shines for all > > http://l1xl1x.blogspot.com > --=20 the sun shines for all http://l1xl1x.blogspot.com