From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 12:43:36 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 071531065695 for ; Thu, 10 Dec 2009 12:43:36 +0000 (UTC) (envelope-from noisex@apollo.lv) Received: from smtp1.apollo.lv (smtp1.apollo.lv [80.232.168.211]) by mx1.freebsd.org (Postfix) with ESMTP id 4C2258FC1B for ; Thu, 10 Dec 2009 12:43:34 +0000 (UTC) X-Virusscan: Clamd Received: from [195.122.14.116] (HELO VECTOR) by smtp1.apollo.lv (CommuniGate Pro SMTP 5.2.10) with ESMTP id 644670455 for freebsd-performance@freebsd.org; Mon, 07 Dec 2009 14:42:05 +0200 From: "Noisex" To: References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> In-Reply-To: Date: Mon, 7 Dec 2009 14:41:55 +0200 Message-ID: <056c01ca773a$a88f69e0$f9ae3da0$@lv> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acpxs88IRD8Wm3IRTcmTdnq5YVSP/gFhOnqw Content-Language: lv x-cr-hashedpuzzle: lXE= AmjK BJ8X B9fT CRlg DQTS DmOg H+tE Imx4 Itfq Jrb8 Ju5+ KssF LDaJ MNNn M8dK; 1; ZgByAGUAZQBiAHMAZAAtAHAAZQByAGYAbwByAG0AYQBuAGMAZQBAAGYAcgBlAGUAYgBzAGQALgBvAHIAZwA=; Sosha1_v1; 7; {CD3E0749-CB82-4A08-B143-BCBBB45C6D4C}; bgBvAGkAcwBlAHgAQABhAHAAbwBsAGwAbwAuAGwAdgA=; Mon, 07 Dec 2009 12:41:52 GMT; RgByAGUAZQBCAFMARAAgAFQAQwBQACAAdAB1AG4AaQBuAGcAIABhAG4AZAAgAHAAZQByAGYAbwByAG0AYQBuAGMAZQA= x-cr-puzzleid: {CD3E0749-CB82-4A08-B143-BCBBB45C6D4C} Subject: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 12:43:36 -0000 Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net = i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)). Currently i use = FBSD 7.1 AMD64. The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb = ram, raid 5 SAS). For net benchmark i used nuttcp and iperf. The servers (client-server) are in 1 VLAN. The results on 1Gbps (down & up): 63.4375 MB / 1.00 sec =3D 532.1332 Mbps 64.3750 MB / 1.00 sec =3D 540.0426 Mbps 62.8125 MB / 1.00 sec =3D 526.8963 Mbps 64.5625 MB / 1.00 sec =3D 541.6318 Mbps 63.9375 MB / 1.00 sec =3D 536.3595 Mbps 63.7500 MB / 1.00 sec =3D 534.7566 Mbps 63.0000 MB / 1.00 sec =3D 528.5003 Mbps 63.5000 MB / 1.00 sec =3D 532.7150 Mbps 64.0000 MB / 1.00 sec =3D 536.8586 Mbps 63.5625 MB / 1.00 sec =3D 533.2452 Mbps =20 637.6688 MB / 10.02 sec =3D 533.9108 Mbps 9 %TX 9 %RX 9 = host-retrans 0.67 msRTT 25.5625 MB / 1.00 sec =3D 214.3916 Mbps 30.8750 MB / 1.00 sec =3D 259.0001 Mbps 29.9375 MB / 1.00 sec =3D 251.1347 Mbps 27.1875 MB / 1.00 sec =3D 228.0669 Mbps 30.5000 MB / 1.00 sec =3D 255.8533 Mbps 30.2500 MB / 1.00 sec =3D 253.7551 Mbps 26.8125 MB / 1.00 sec =3D 224.9211 Mbps 30.3750 MB / 1.00 sec =3D 254.8047 Mbps 30.3750 MB / 1.00 sec =3D 254.8050 Mbps 30.0625 MB / 1.00 sec =3D 252.1835 Mbps 292.2155 MB / 10.02 sec =3D 244.6825 Mbps 10 %TX 12 %RX 0 = host-retrans 0.71 msRTT As you can see down is littlebit more than half of full link speed. And = upload is only 20-25% of full link. I tried to change a lot sysctl params but without a big results. = Currenlty my entries in /etc/sysctl.conf which regarding to TCP: #kernel tuning, tcp kern.ipc.somaxconn=3D2048 kern.ipc.nmbclusters=3D32768 kern.ipc.maxsockbuf=3D8388608 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.inflight.enable=3D0 net.inet.tcp.sendspace=3D65536 net.inet.tcp.recvspace=3D65536 net.inet.udp.recvspace=3D65536 net.inet.tcp.inflight.enable=3D0 net.inet.tcp.rfc1323=3D1 net.inet.tcp.sack.enable=3D1 net.inet.tcp.path_mtu_discovery=3D1 net.inet.tcp.sendbuf_auto=3D1 net.inet.tcp.sendbuf_inc=3D16384 net.inet.tcp.recvbuf_auto=3D1 net.inet.tcp.recvbuf_inc=3D524288 Do you have some kind suggestion what i could to change to increase the = performance of TCP?=20 Besides when i make the benchamrks i run the sniffer to see whats = happening with network..sometimes i saw that window size is 0...does it = mean that server can't handle something or recieve buffer size is to = small? p.s sory for my bad english :) Noisex From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 13:30:53 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81297106566C for ; Thu, 10 Dec 2009 13:30:53 +0000 (UTC) (envelope-from prvs=15957c31ed=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0C31A8FC15 for ; Thu, 10 Dec 2009 13:30:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1260451193; x=1261055993; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=L4FLHMKda383B+z0HaYbO i31P9ksF0eNJg+RNmUf4+o=; b=M5E5svepAMXbSHuaRKOUflCL/FEC0Q1pj0GaZ k5kOJXbapgujq/hMLcPuEeGaNwCaBD7p+YDiQKSvrXcSX3+/ev075oe5JQHoyw0m srnV7C82fbnI5oRLa45tlX68LvIjfxbwU87+yZBo0MV+2Evb95hJ5k9el9E7pdQl YTtd5o= X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 13:19:53 +0000 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008840761.msg for ; Thu, 10 Dec 2009 13:19:52 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 13:19:52 +0000 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=15957c31ed=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org Message-ID: <4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> From: "Steven Hartland" To: "Noisex" , References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv> Date: Thu, 10 Dec 2009 13:19:46 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Cc: Subject: Re: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 13:30:53 -0000 What app are you using there and is it setting the send / receive buffers correctly? ----- Original Message ----- From: "Noisex" To: Sent: Monday, December 07, 2009 12:41 PM Subject: FreeBSD TCP tuning and performance Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)). Currently i use FBSD 7.1 AMD64. The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb ram, raid 5 SAS). For net benchmark i used nuttcp and iperf. The servers (client-server) are in 1 VLAN. The results on 1Gbps (down & up): 63.4375 MB / 1.00 sec = 532.1332 Mbps 64.3750 MB / 1.00 sec = 540.0426 Mbps 62.8125 MB / 1.00 sec = 526.8963 Mbps 64.5625 MB / 1.00 sec = 541.6318 Mbps 63.9375 MB / 1.00 sec = 536.3595 Mbps 63.7500 MB / 1.00 sec = 534.7566 Mbps 63.0000 MB / 1.00 sec = 528.5003 Mbps 63.5000 MB / 1.00 sec = 532.7150 Mbps 64.0000 MB / 1.00 sec = 536.8586 Mbps 63.5625 MB / 1.00 sec = 533.2452 Mbps 637.6688 MB / 10.02 sec = 533.9108 Mbps 9 %TX 9 %RX 9 host-retrans 0.67 msRTT 25.5625 MB / 1.00 sec = 214.3916 Mbps 30.8750 MB / 1.00 sec = 259.0001 Mbps 29.9375 MB / 1.00 sec = 251.1347 Mbps 27.1875 MB / 1.00 sec = 228.0669 Mbps 30.5000 MB / 1.00 sec = 255.8533 Mbps 30.2500 MB / 1.00 sec = 253.7551 Mbps 26.8125 MB / 1.00 sec = 224.9211 Mbps 30.3750 MB / 1.00 sec = 254.8047 Mbps 30.3750 MB / 1.00 sec = 254.8050 Mbps 30.0625 MB / 1.00 sec = 252.1835 Mbps 292.2155 MB / 10.02 sec = 244.6825 Mbps 10 %TX 12 %RX 0 host-retrans 0.71 msRTT As you can see down is littlebit more than half of full link speed. And upload is only 20-25% of full link. I tried to change a lot sysctl params but without a big results. Currenlty my entries in /etc/sysctl.conf which regarding to TCP: #kernel tuning, tcp kern.ipc.somaxconn=2048 kern.ipc.nmbclusters=32768 kern.ipc.maxsockbuf=8388608 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.udp.recvspace=65536 net.inet.tcp.inflight.enable=0 net.inet.tcp.rfc1323=1 net.inet.tcp.sack.enable=1 net.inet.tcp.path_mtu_discovery=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 Do you have some kind suggestion what i could to change to increase the performance of TCP? Besides when i make the benchamrks i run the sniffer to see whats happening with network..sometimes i saw that window size is 0...does it mean that server can't handle something or recieve buffer size is to small? p.s sory for my bad english :) Noisex _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 13:56:13 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75DF81065670 for ; Thu, 10 Dec 2009 13:56:13 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 4F5648FC18 for ; Thu, 10 Dec 2009 13:56:12 +0000 (UTC) Received: from localhost (overdrive.ws.pitbpa0.priv.collaborativefusion.com [192.168.2.162]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Thu, 10 Dec 2009 08:56:12 -0500 id 00056426.000000004B20FDFC.0000048E Date: Thu, 10 Dec 2009 08:56:12 -0500 From: Bill Moran To: "Noisex" Message-Id: <20091210085612.098f8aae.wmoran@collaborativefusion.com> In-Reply-To: <056c01ca773a$a88f69e0$f9ae3da0$@lv> References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv> Organization: Collaborative Fusion Inc. X-Mailer: Sylpheed 2.7.1 (GTK+ 2.16.6; i386-portbld-freebsd7.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 13:56:13 -0000 In response to "Noisex" : > Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)). Currently i use FBSD 7.1 AMD64. > > The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb ram, raid 5 SAS). > > For net benchmark i used nuttcp and iperf. > > The servers (client-server) are in 1 VLAN. If this is on a switch shared with other (busy) systems, you might be measuring the saturation/capacity of the switch (even if you have those two units on a dedicated vlan). Try the test with a crossover cable to eliminate that possibility. > The results on 1Gbps (down & up): > > 63.4375 MB / 1.00 sec = 532.1332 Mbps > 64.3750 MB / 1.00 sec = 540.0426 Mbps > 62.8125 MB / 1.00 sec = 526.8963 Mbps > 64.5625 MB / 1.00 sec = 541.6318 Mbps > 63.9375 MB / 1.00 sec = 536.3595 Mbps > 63.7500 MB / 1.00 sec = 534.7566 Mbps > 63.0000 MB / 1.00 sec = 528.5003 Mbps > 63.5000 MB / 1.00 sec = 532.7150 Mbps > 64.0000 MB / 1.00 sec = 536.8586 Mbps > 63.5625 MB / 1.00 sec = 533.2452 Mbps > > 637.6688 MB / 10.02 sec = 533.9108 Mbps 9 %TX 9 %RX 9 host-retrans 0.67 msRTT > > 25.5625 MB / 1.00 sec = 214.3916 Mbps > 30.8750 MB / 1.00 sec = 259.0001 Mbps > 29.9375 MB / 1.00 sec = 251.1347 Mbps > 27.1875 MB / 1.00 sec = 228.0669 Mbps > 30.5000 MB / 1.00 sec = 255.8533 Mbps > 30.2500 MB / 1.00 sec = 253.7551 Mbps > 26.8125 MB / 1.00 sec = 224.9211 Mbps > 30.3750 MB / 1.00 sec = 254.8047 Mbps > 30.3750 MB / 1.00 sec = 254.8050 Mbps > 30.0625 MB / 1.00 sec = 252.1835 Mbps > > 292.2155 MB / 10.02 sec = 244.6825 Mbps 10 %TX 12 %RX 0 host-retrans 0.71 msRTT > > As you can see down is littlebit more than half of full link speed. And upload is only 20-25% of full link. I'm not familiar with that program, but can you increase the test sample size? 65M isn't a lot of data to push over a 1gps link for testing purposes, and you might be seeing startup overhead. > I tried to change a lot sysctl params but without a big results. Currenlty my entries in /etc/sysctl.conf which regarding to TCP: > > #kernel tuning, tcp > kern.ipc.somaxconn=2048 > kern.ipc.nmbclusters=32768 > > kern.ipc.maxsockbuf=8388608 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.inet.udp.recvspace=65536 > net.inet.tcp.inflight.enable=0 > net.inet.tcp.rfc1323=1 > net.inet.tcp.sack.enable=1 > net.inet.tcp.path_mtu_discovery=1 > net.inet.tcp.sendbuf_auto=1 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_auto=1 > net.inet.tcp.recvbuf_inc=524288 > > Do you have some kind suggestion what i could to change to increase the performance of TCP? > > Besides when i make the benchamrks i run the sniffer to see whats happening with network..sometimes i saw that window size is 0...does it mean that server can't handle something or recieve buffer size is to small? If the window size drops to 0, it means the receive buffer on the receiving system is full and waiting to be flushed by the application. Considering the fact that you're sending 65M per second, a 16M buffer might not be large enough. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 14:32:20 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48ABE10656A3 for ; Thu, 10 Dec 2009 14:32:20 +0000 (UTC) (envelope-from noisex@apollo.lv) Received: from smtp5.apollo.lv (smtp5.apollo.lv [80.232.168.197]) by mx1.freebsd.org (Postfix) with ESMTP id C21D68FC19 for ; Thu, 10 Dec 2009 14:32:19 +0000 (UTC) Received: from VECTOR (unknown [195.122.14.116]) by smtp5.apollo.lv (Postfix) with ESMTP id 79F1F3402BE for ; Thu, 10 Dec 2009 16:28:43 +0200 (EET) From: "Noisex" To: References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv> <4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> In-Reply-To: <4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> Date: Thu, 10 Dec 2009 16:28:10 +0200 Message-ID: <066b01ca79a5$04ab9980$0e02cc80$@lv> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acp5nRrNv2c6plx0ScuxdjY6VvZ3GgABhO8w Content-Language: lv X-Lattelecom-MailScanner-Information: Please contact the ISP for more information X-Lattelecom-MailScanner-ID: 79F1F3402BE.48745 X-Lattelecom-MailScanner: Found to be clean X-Lattelecom-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-1.148, required 5, BAYES_00 -2.60, RDNS_NONE 0.10, TVD_RCVD_SINGLE 1.35) X-Lattelecom-MailScanner-From: noisex@apollo.lv X-Spam-Status: No Subject: RE: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 14:32:20 -0000 This server is basically used for bandwidth testing (Our company is = internet service provider) and we need and our clients something to = measure a internet speed. Actually on that server is hosted MyConnection SpeedServer = (http://www.visualware.com/) for bandwidth tests. Some month ago we = started to clients offer GPON with speed 500Mbit/500mbit. While we offer DSL technology and speeds was up to 100Mbit it was pretty = good results...but now we can't measure a speed on 1Gbps link...the = results is very pure (almost half from real on FreeBSD). This results aren't only with MyConnection..also with nuttcp and = iperf...i have feeling, that FreBSD can't handle window size = send/receive buffer p.s maybe I should enable Network Pooling/disable interrupts on network = card etc things? What could be a recommendations to set up for maximum = performance TCP settings on 1/10Gbps interface with sysctl parameters? Noisex -----Original Message----- From: owner-freebsd-performance@freebsd.org = [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Steven = Hartland Sent: ceturtdiena, 2009. gada 10. decembr=C4=AB 15:20 To: Noisex; freebsd-performance@freebsd.org Subject: Re: FreeBSD TCP tuning and performance What app are you using there and is it setting the send / receive = buffers correctly? ----- Original Message -----=20 From: "Noisex" To: Sent: Monday, December 07, 2009 12:41 PM Subject: FreeBSD TCP tuning and performance Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net = i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)).=20 Currently i use FBSD 7.1 AMD64. The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb = ram, raid 5 SAS). For net benchmark i used nuttcp and iperf. The servers (client-server) are in 1 VLAN. The results on 1Gbps (down & up): 63.4375 MB / 1.00 sec =3D 532.1332 Mbps 64.3750 MB / 1.00 sec =3D 540.0426 Mbps 62.8125 MB / 1.00 sec =3D 526.8963 Mbps 64.5625 MB / 1.00 sec =3D 541.6318 Mbps 63.9375 MB / 1.00 sec =3D 536.3595 Mbps 63.7500 MB / 1.00 sec =3D 534.7566 Mbps 63.0000 MB / 1.00 sec =3D 528.5003 Mbps 63.5000 MB / 1.00 sec =3D 532.7150 Mbps 64.0000 MB / 1.00 sec =3D 536.8586 Mbps 63.5625 MB / 1.00 sec =3D 533.2452 Mbps 637.6688 MB / 10.02 sec =3D 533.9108 Mbps 9 %TX 9 %RX 9 = host-retrans 0.67 msRTT 25.5625 MB / 1.00 sec =3D 214.3916 Mbps 30.8750 MB / 1.00 sec =3D 259.0001 Mbps 29.9375 MB / 1.00 sec =3D 251.1347 Mbps 27.1875 MB / 1.00 sec =3D 228.0669 Mbps 30.5000 MB / 1.00 sec =3D 255.8533 Mbps 30.2500 MB / 1.00 sec =3D 253.7551 Mbps 26.8125 MB / 1.00 sec =3D 224.9211 Mbps 30.3750 MB / 1.00 sec =3D 254.8047 Mbps 30.3750 MB / 1.00 sec =3D 254.8050 Mbps 30.0625 MB / 1.00 sec =3D 252.1835 Mbps 292.2155 MB / 10.02 sec =3D 244.6825 Mbps 10 %TX 12 %RX 0 = host-retrans 0.71 msRTT As you can see down is littlebit more than half of full link speed. And = upload is only 20-25% of full link. I tried to change a lot sysctl params but without a big results. = Currenlty my entries in /etc/sysctl.conf which regarding to TCP: #kernel tuning, tcp kern.ipc.somaxconn=3D2048 kern.ipc.nmbclusters=3D32768 kern.ipc.maxsockbuf=3D8388608 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.inflight.enable=3D0 net.inet.tcp.sendspace=3D65536 net.inet.tcp.recvspace=3D65536 net.inet.udp.recvspace=3D65536 net.inet.tcp.inflight.enable=3D0 net.inet.tcp.rfc1323=3D1 net.inet.tcp.sack.enable=3D1 net.inet.tcp.path_mtu_discovery=3D1 net.inet.tcp.sendbuf_auto=3D1 net.inet.tcp.sendbuf_inc=3D16384 net.inet.tcp.recvbuf_auto=3D1 net.inet.tcp.recvbuf_inc=3D524288 Do you have some kind suggestion what i could to change to increase the = performance of TCP? Besides when i make the benchamrks i run the sniffer to see whats = happening with network..sometimes i saw that window size is=20 0...does it mean that server can't handle something or recieve buffer = size is to small? p.s sory for my bad english :) Noisex _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This e.mail is private and confidential between Multiplay (UK) Ltd. and = the person or entity to whom it is addressed. In the event of = misdirection, the recipient is prohibited from using, copying, printing = or otherwise disseminating it or any information contained in it.=20 In the event of misdirection, illegible or incomplete transmission = please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 15:05:40 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 989AB106568F for ; Thu, 10 Dec 2009 15:05:40 +0000 (UTC) (envelope-from prvs=15957c31ed=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 240848FC25 for ; Thu, 10 Dec 2009 15:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1260457538; x=1261062338; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=G5qp3kRr658BLHSo1sRd6 gjIvsWa0fDX4/wo0hxbGc0=; b=Qgm4Ueznh8hpTEUFcWtQYi8BiyonBkqV7I4eJ J6L5GJVt1cXa2t6GOJZ+hhIxWrHTDQiBV4NbrqMqz7PbmUZXM/o7mf5qnNh8lS4p M9QUQfAtylipkA2c/ufulXKxvuBiZojd2H4sa61gpbdX0K98Va500qCjxzWZtRm8 FIhPZg= X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 15:05:38 +0000 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008841342.msg for ; Thu, 10 Dec 2009 15:05:37 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 15:05:37 +0000 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=15957c31ed=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org Message-ID: <1376FE3FC7444CA89D2950E7DB913986@multiplay.co.uk> From: "Steven Hartland" To: "Noisex" , References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv><4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> <066b01ca79a5$04ab9980$0e02cc80$@lv> Date: Thu, 10 Dec 2009 15:05:29 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Cc: Subject: Re: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 15:05:40 -0000 What are your iperf command lines? A standard iperf cmd line won't give line rate, the ones we use here are:- == Server == iperf -s -w 2.5M -l 2.5M == Client == iperf -i 10 -t 20 -c -w 2.5M -l 2.5M == Tuning == We use the following tuning on our machines to achieve line rate Gig on 7.0 amd64 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 Out of curiousity I just tried this very test on an 8.0 box we have here and was only able to achieve similar performance to yourself. So it may be the case that there has been a significant regression since 7.0, I'll have to do some more tests when I have time. For reference the machines we have tested and get line rate on have the following nic's == Machine #1 7.0-RELEASE amd64 == em0: port 0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 em0: Using MSI interrupt em0: Ethernet address: ..... em0: [FILTER] == Machine #2 7.0-RELEASE amd64 == bge0: mem 0xfc9f0000-0xfc9fffff irq 26 at device 5.0 on pci3 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: .... bge0: [ITHREAD] The machine which is currently under performing == Machine #3 8.0-RELEASE amd64 == em0: port 0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:33:ec:44 Regards Steve ----- Original Message ----- From: "Noisex" To: Sent: Thursday, December 10, 2009 2:28 PM Subject: RE: FreeBSD TCP tuning and performance This server is basically used for bandwidth testing (Our company is internet service provider) and we need and our clients something to measure a internet speed. Actually on that server is hosted MyConnection SpeedServer (http://www.visualware.com/) for bandwidth tests. Some month ago we started to clients offer GPON with speed 500Mbit/500mbit. While we offer DSL technology and speeds was up to 100Mbit it was pretty good results...but now we can't measure a speed on 1Gbps link...the results is very pure (almost half from real on FreeBSD). This results aren't only with MyConnection..also with nuttcp and iperf...i have feeling, that FreBSD can't handle window size send/receive buffer p.s maybe I should enable Network Pooling/disable interrupts on network card etc things? What could be a recommendations to set up for maximum performance TCP settings on 1/10Gbps interface with sysctl parameters? Noisex -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Steven Hartland Sent: ceturtdiena, 2009. gada 10. decembrī 15:20 To: Noisex; freebsd-performance@freebsd.org Subject: Re: FreeBSD TCP tuning and performance What app are you using there and is it setting the send / receive buffers correctly? ----- Original Message ----- From: "Noisex" To: Sent: Monday, December 07, 2009 12:41 PM Subject: FreeBSD TCP tuning and performance Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)). Currently i use FBSD 7.1 AMD64. The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb ram, raid 5 SAS). For net benchmark i used nuttcp and iperf. The servers (client-server) are in 1 VLAN. The results on 1Gbps (down & up): 63.4375 MB / 1.00 sec = 532.1332 Mbps 64.3750 MB / 1.00 sec = 540.0426 Mbps 62.8125 MB / 1.00 sec = 526.8963 Mbps 64.5625 MB / 1.00 sec = 541.6318 Mbps 63.9375 MB / 1.00 sec = 536.3595 Mbps 63.7500 MB / 1.00 sec = 534.7566 Mbps 63.0000 MB / 1.00 sec = 528.5003 Mbps 63.5000 MB / 1.00 sec = 532.7150 Mbps 64.0000 MB / 1.00 sec = 536.8586 Mbps 63.5625 MB / 1.00 sec = 533.2452 Mbps 637.6688 MB / 10.02 sec = 533.9108 Mbps 9 %TX 9 %RX 9 host-retrans 0.67 msRTT 25.5625 MB / 1.00 sec = 214.3916 Mbps 30.8750 MB / 1.00 sec = 259.0001 Mbps 29.9375 MB / 1.00 sec = 251.1347 Mbps 27.1875 MB / 1.00 sec = 228.0669 Mbps 30.5000 MB / 1.00 sec = 255.8533 Mbps 30.2500 MB / 1.00 sec = 253.7551 Mbps 26.8125 MB / 1.00 sec = 224.9211 Mbps 30.3750 MB / 1.00 sec = 254.8047 Mbps 30.3750 MB / 1.00 sec = 254.8050 Mbps 30.0625 MB / 1.00 sec = 252.1835 Mbps 292.2155 MB / 10.02 sec = 244.6825 Mbps 10 %TX 12 %RX 0 host-retrans 0.71 msRTT As you can see down is littlebit more than half of full link speed. And upload is only 20-25% of full link. I tried to change a lot sysctl params but without a big results. Currenlty my entries in /etc/sysctl.conf which regarding to TCP: #kernel tuning, tcp kern.ipc.somaxconn=2048 kern.ipc.nmbclusters=32768 kern.ipc.maxsockbuf=8388608 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.udp.recvspace=65536 net.inet.tcp.inflight.enable=0 net.inet.tcp.rfc1323=1 net.inet.tcp.sack.enable=1 net.inet.tcp.path_mtu_discovery=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 Do you have some kind suggestion what i could to change to increase the performance of TCP? Besides when i make the benchamrks i run the sniffer to see whats happening with network..sometimes i saw that window size is 0...does it mean that server can't handle something or recieve buffer size is to small? p.s sory for my bad english :) Noisex _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 21:00:04 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7002C106568F for ; Thu, 10 Dec 2009 21:00:04 +0000 (UTC) (envelope-from sfourman@gmail.com) Received: from mail-pz0-f185.google.com (mail-pz0-f185.google.com [209.85.222.185]) by mx1.freebsd.org (Postfix) with ESMTP id 3933C8FC08 for ; Thu, 10 Dec 2009 21:00:04 +0000 (UTC) Received: by pzk15 with SMTP id 15so178665pzk.3 for ; Thu, 10 Dec 2009 13:00:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=c5luqUTfk4mTUK/hHarg4qArimAgFafZ094m3DDjpTw=; b=pPK8WDjgIIeDCrAGAMVeS5SuYDxSXRPu8oXQhzf+6067sGpV3wnYNb2Dm0H3M/DOWO eK3JDdKaDLvV+XYNRvHSGlVXq3C2CjRhIEENIKvIDgy2UpPAwl4GdaYkTvJ2/+l7BuMr 4soBXFI+ZTiXsAd2YrqXMgJ13eFCm1Gwy/WN8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=r/4fxx+T/OpplUPW5x29Ly2I2HjvGebEBOejhh61fIQhyBAyNszvp3JpGOrbF429Yw schsIRq8L8H4bCBqCLYrto6lUckq61Er3G9zBACkH65nhH6mlXIMMVqeQyBEoo/LksG8 6+lVh/ulP4DMsZpehOn2GMyqME5mURR2KThJI= MIME-Version: 1.0 Received: by 10.143.24.39 with SMTP id b39mr260803wfj.51.1260477063294; Thu, 10 Dec 2009 12:31:03 -0800 (PST) In-Reply-To: <1376FE3FC7444CA89D2950E7DB913986@multiplay.co.uk> References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv> <4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> <066b01ca79a5$04ab9980$0e02cc80$@lv> <1376FE3FC7444CA89D2950E7DB913986@multiplay.co.uk> Date: Thu, 10 Dec 2009 14:31:03 -0600 Message-ID: <11167f520912101231v34433e1dh6c5fbeee507e3118@mail.gmail.com> From: "Sam Fourman Jr." To: Steven Hartland Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-performance@freebsd.org, Noisex Subject: Re: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 21:00:04 -0000 On Thu, Dec 10, 2009 at 9:05 AM, Steven Hartland wrote: > What are your iperf command lines? using FreeBSD RELENG_8 amd64 on a low traffic dell 5224 switch I was able to see these results, and I don't see a problem wile running iperf I brought up top -P on my desktop machine here is what it said last pid: 27659; load averages: 0.62, 0.34, 0.22 up 1+20:29:22 14:28:43 166 processes: 3 running, 163 sleeping CPU 0: 0.0% user, 0.0% nice, 91.6% system, 0.0% interrupt, 8.4% idle CPU 1: 3.8% user, 0.0% nice, 9.6% system, 0.0% interrupt, 86.5% idle CPU 2: 7.7% user, 0.0% nice, 18.7% system, 0.6% interrupt, 72.9% idle CPU 3: 1.3% user, 0.0% nice, 26.5% system, 0.6% interrupt, 71.6% idle Mem: 587M Active, 706M Inact, 376M Wired, 6556K Cache, 271M Buf, 2241M Free Swap: 3072M Total, 168K Used, 3072M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 24268 sfourman 1 46 0 368M 43060K CPU2 1 87:54 2.29% npviewer.bin 24263 sfourman 10 44 0 357M 186M ucond 1 25:21 1.07% firefox-bin 1396 sfourman 1 44 0 3245M 98416K select 1 47:19 0.00% Xorg 24271 sfourman 1 44 0 368M 43060K futex 1 15:35 0.00% npviewer.bin 24272 sfourman 1 44 0 368M 43060K futex 2 12:26 0.00% npviewer.bin 24273 sfourman 1 44 0 368M 43060K futex 3 8:14 0.00% npviewer.bin 24284 sfourman 1 44 0 368M 43060K pcmwrv 2 1:07 0.00% npviewer.bin 22460 sfourman 1 44 0 33336K 25280K select 0 1:02 0.00% wowmatrix 1657 sfourman 2 51 0 112M 18880K piperd 1 0:52 0.00% gnome-terminal 1368 root 1 44 0 12536K 1688K select 1 0:50 0.00% hald-addon-storage 1366 root 1 44 0 12536K 1684K select 1 0:50 0.00% hald-addon-storage 1412 sfourman 3 44 0 212M 19304K ucond 1 0:40 0.00% gnome-settings-daem 705 root 1 44 0 8036K 1184K select 3 0:37 0.00% moused 1437 sfourman 2 44 0 198M 41172K ucond 1 0:32 0.00% nautilus 1329 root 1 44 0 12536K 1628K select 3 0:19 0.00% hald-addon-storage 18652 sfourman 2 56 0 112M 18736K piperd 2 0:18 0.00% gnome-terminal 1321 haldaemon 1 44 0 24380K 4920K select 2 0:18 0.00% hald 1435 sfourman 2 44 0 164M 29048K ucond 2 0:17 0.00% gnome-panel 1434 sfourman 1 44 0 110M 16348K select 1 0:15 0.00% metacity 17410 sfourman 1 44 0 13000K 2336K select 3 0:15 0.00% gam_server 1506 sfourman 2 44 0 172M 22000K ucond 2 0:14 0.00% clock-applet 1469 sfourman 2 44 0 98440K 14148K ucond 3 0:12 0.00% gnome-screensaver 1476 sfourman 1 44 0 135M 18960K select 3 0:10 0.00% wnck-applet 1482 sfourman 2 44 0 25104K 4200K ucond 1 0:10 0.00% gvfsd-trash 1488 sfourman 3 47 0 28204K 4628K piperd 0 0:10 0.00% gvfs-hal-volume-mon 1407 sfourman 1 44 0 26764K 7252K select 1 0:10 0.00% gconfd-2 27653 root 3 44 0 15588K 4512K ucond 1 0:07 0.00% iperf 19711 sfourman 2 44 0 113M 20304K piperd 3 0:07 0.00% Thunar 1546 root 1 44 0 13000K 2116K select 3 0:06 0.00% gam_server 24852 sfourman 10 44 0 249M 51160K select 0 0:03 0.00% vlc 22153 sfourman 1 60 16 66140K 13548K select 2 0:03 0.00% trackerd 25373 sfourman 10 44 0 251M 54412K select 1 0:02 0.00% vlc 1450 sfourman 1 44 0 126M 17076K select 2 0:02 0.00% gnome-power-manager FreeBSD Sam.PuffyBSD.Com 8.0-STABLE FreeBSD 8.0-STABLE #1: Mon Nov 30 21:25:50 CST 2009 sfourman@Sam.PuffyBSD.Com:/usr/obj/usr/src/sys/GENERIC amd64 nfe0: flags=8843 metric 0 mtu 1500 options=19b ether 00:23:54:96:dd:8d inet 192.168.12.117 netmask 0xffffff00 broadcast 192.168.12.255 media: Ethernet autoselect (1000baseT ) status: active Sam# iperf -i 1 -t 60 -c 192.168.12.188 -w 2.5M -l 2.5M ------------------------------------------------------------ Client connecting to 192.168.12.188, TCP port 5001 TCP window size: 32.5 KByte (WARNING: requested 2.50 MByte) ------------------------------------------------------------ [ 3] local 192.168.12.117 port 46609 connected with 192.168.12.188 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 110 MBytes 923 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 1.0- 2.0 sec 110 MBytes 923 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 2.0- 3.0 sec 112 MBytes 944 Mbits/sec [ ID] Interval Transfer Bandwidth From owner-freebsd-performance@FreeBSD.ORG Fri Dec 11 15:16:10 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DAA39106566B for ; Fri, 11 Dec 2009 15:16:10 +0000 (UTC) (envelope-from prvs=1596b04d6d=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5FBCD8FC08 for ; Fri, 11 Dec 2009 15:16:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1260543927; x=1261148727; q=dns/txt; h=Received: Message-ID:From:To:Subject:Date:MIME-Version:Content-Type: Content-Transfer-Encoding; bh=DzqoRA0YNasI9VxTsPPqqMCiWa4NL3Z/ZW wWnlnwEbs=; b=bEe/DEB2mvY20jvs3+7joCjCgPsGt6YSBn5EjbMc8soEgPQfvn LcEDl1rXFXlXp4pG5siVr9UEHR4CAn7ZjHPLE7lIDzuBy1YONyv7pTx1d4bHIWsO hemiHWrxgQIiSmHEhmUB+rDMWYOZ9tL7LpOSSi3AnM5S6oPcTK/MCCBtI= X-MDAV-Processed: mail1.multiplay.co.uk, Fri, 11 Dec 2009 15:05:27 +0000 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008847478.msg for ; Fri, 11 Dec 2009 15:05:26 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Fri, 11 Dec 2009 15:05:26 +0000 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=1596b04d6d=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org Message-ID: From: "Steven Hartland" To: Date: Fri, 11 Dec 2009 15:05:04 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Subject: Significant performance regression for increased maxsockbuf on 8.0-RELEASE X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Dec 2009 15:16:11 -0000 As noted in the FreeBSD TCP tuning and performance thread here: http://lists.freebsd.org/pipermail/freebsd-performance/2009-December/003909.html There seems to be a significant performance drop when using 8.0 vs 7.0 after digging around this seems to be caused by the use of increased kern.ipc.maxsockbuf Using the default value ( 262144 ) I see the following iperf results: iperf -t 10 -c server1 -w 2.5M -l 2.5M ------------------------------------------------------------ Client connecting to db3, TCP port 5001 TCP window size: 65.0 KByte (WARNING: requested 2.50 MByte) ------------------------------------------------------------ [ 3] local 85.236.96.66 port 50848 connected with 85.236.96.42 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.09 GBytes 935 Mbits/sec With anything larger than that the rate drops off significantly e.g. 282144 gives:- iperf -t 10 -c db3 -w 2.5M -l 2.5M ------------------------------------------------------------ Client connecting to db3, TCP port 5001 TCP window size: 65.0 KByte (WARNING: requested 2.50 MByte) ------------------------------------------------------------ [ 3] local 85.236.96.66 port 25867 connected with 85.236.96.42 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 778 MBytes 651 Mbits/sec and 302144 we're down to just under half the baseline / line rate iperf -t 10 -c db3 -w 2.5M -l 2.5M ------------------------------------------------------------ Client connecting to db3, TCP port 5001 TCP window size: 65.0 KByte (WARNING: requested 2.50 MByte) ------------------------------------------------------------ [ 3] local 85.236.96.66 port 12506 connected with 85.236.96.42 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 588 MBytes 492 Mbits/sec Any ideas why using window sizes above 65K results in such poor performance on 8.0? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.