From owner-freebsd-performance@FreeBSD.ORG Thu Dec 10 15:05:40 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 989AB106568F for ; Thu, 10 Dec 2009 15:05:40 +0000 (UTC) (envelope-from prvs=15957c31ed=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 240848FC25 for ; Thu, 10 Dec 2009 15:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=multiplay.co.uk; s=Multiplay; t=1260457538; x=1261062338; q=dns/txt; h=Received: Message-ID:From:To:References:Subject:Date:MIME-Version: Content-Type:Content-Transfer-Encoding; bh=G5qp3kRr658BLHSo1sRd6 gjIvsWa0fDX4/wo0hxbGc0=; b=Qgm4Ueznh8hpTEUFcWtQYi8BiyonBkqV7I4eJ J6L5GJVt1cXa2t6GOJZ+hhIxWrHTDQiBV4NbrqMqz7PbmUZXM/o7mf5qnNh8lS4p M9QUQfAtylipkA2c/ufulXKxvuBiZojd2H4sa61gpbdX0K98Va500qCjxzWZtRm8 FIhPZg= X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 15:05:38 +0000 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50008841342.msg for ; Thu, 10 Dec 2009 15:05:37 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 10 Dec 2009 15:05:37 +0000 (not processed: message from trusted or authenticated source) X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 213.123.247.160 X-Return-Path: prvs=15957c31ed=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org Message-ID: <1376FE3FC7444CA89D2950E7DB913986@multiplay.co.uk> From: "Steven Hartland" To: "Noisex" , References: <4B108A18.207@truschinski.de> <584ec6bb0911291330o11fba282y400e0abf121f5e7f@mail.gmail.com> <056c01ca773a$a88f69e0$f9ae3da0$@lv><4ADA4DF6401E47F6A0FE26890B50FBB1@multiplay.co.uk> <066b01ca79a5$04ab9980$0e02cc80$@lv> Date: Thu, 10 Dec 2009 15:05:29 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="utf-8"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5843 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Cc: Subject: Re: FreeBSD TCP tuning and performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2009 15:05:40 -0000 What are your iperf command lines? A standard iperf cmd line won't give line rate, the ones we use here are:- == Server == iperf -s -w 2.5M -l 2.5M == Client == iperf -i 10 -t 20 -c -w 2.5M -l 2.5M == Tuning == We use the following tuning on our machines to achieve line rate Gig on 7.0 amd64 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 Out of curiousity I just tried this very test on an 8.0 box we have here and was only able to achieve similar performance to yourself. So it may be the case that there has been a significant regression since 7.0, I'll have to do some more tests when I have time. For reference the machines we have tested and get line rate on have the following nic's == Machine #1 7.0-RELEASE amd64 == em0: port 0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 em0: Using MSI interrupt em0: Ethernet address: ..... em0: [FILTER] == Machine #2 7.0-RELEASE amd64 == bge0: mem 0xfc9f0000-0xfc9fffff irq 26 at device 5.0 on pci3 miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bge0: Ethernet address: .... bge0: [ITHREAD] The machine which is currently under performing == Machine #3 8.0-RELEASE amd64 == em0: port 0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:33:ec:44 Regards Steve ----- Original Message ----- From: "Noisex" To: Sent: Thursday, December 10, 2009 2:28 PM Subject: RE: FreeBSD TCP tuning and performance This server is basically used for bandwidth testing (Our company is internet service provider) and we need and our clients something to measure a internet speed. Actually on that server is hosted MyConnection SpeedServer (http://www.visualware.com/) for bandwidth tests. Some month ago we started to clients offer GPON with speed 500Mbit/500mbit. While we offer DSL technology and speeds was up to 100Mbit it was pretty good results...but now we can't measure a speed on 1Gbps link...the results is very pure (almost half from real on FreeBSD). This results aren't only with MyConnection..also with nuttcp and iperf...i have feeling, that FreBSD can't handle window size send/receive buffer p.s maybe I should enable Network Pooling/disable interrupts on network card etc things? What could be a recommendations to set up for maximum performance TCP settings on 1/10Gbps interface with sysctl parameters? Noisex -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Steven Hartland Sent: ceturtdiena, 2009. gada 10. decembrī 15:20 To: Noisex; freebsd-performance@freebsd.org Subject: Re: FreeBSD TCP tuning and performance What app are you using there and is it setting the send / receive buffers correctly? ----- Original Message ----- From: "Noisex" To: Sent: Monday, December 07, 2009 12:41 PM Subject: FreeBSD TCP tuning and performance Hi! I have a problem with TCP performance on FBSD boxes with 1Gbps net i-faces (Broadcom NetXtreme II BCM5708 1000Base-T (B2)). Currently i use FBSD 7.1 AMD64. The test lab: 2 x (server-client) HP Proliant G5 DL360 (quad-core/8gb ram, raid 5 SAS). For net benchmark i used nuttcp and iperf. The servers (client-server) are in 1 VLAN. The results on 1Gbps (down & up): 63.4375 MB / 1.00 sec = 532.1332 Mbps 64.3750 MB / 1.00 sec = 540.0426 Mbps 62.8125 MB / 1.00 sec = 526.8963 Mbps 64.5625 MB / 1.00 sec = 541.6318 Mbps 63.9375 MB / 1.00 sec = 536.3595 Mbps 63.7500 MB / 1.00 sec = 534.7566 Mbps 63.0000 MB / 1.00 sec = 528.5003 Mbps 63.5000 MB / 1.00 sec = 532.7150 Mbps 64.0000 MB / 1.00 sec = 536.8586 Mbps 63.5625 MB / 1.00 sec = 533.2452 Mbps 637.6688 MB / 10.02 sec = 533.9108 Mbps 9 %TX 9 %RX 9 host-retrans 0.67 msRTT 25.5625 MB / 1.00 sec = 214.3916 Mbps 30.8750 MB / 1.00 sec = 259.0001 Mbps 29.9375 MB / 1.00 sec = 251.1347 Mbps 27.1875 MB / 1.00 sec = 228.0669 Mbps 30.5000 MB / 1.00 sec = 255.8533 Mbps 30.2500 MB / 1.00 sec = 253.7551 Mbps 26.8125 MB / 1.00 sec = 224.9211 Mbps 30.3750 MB / 1.00 sec = 254.8047 Mbps 30.3750 MB / 1.00 sec = 254.8050 Mbps 30.0625 MB / 1.00 sec = 252.1835 Mbps 292.2155 MB / 10.02 sec = 244.6825 Mbps 10 %TX 12 %RX 0 host-retrans 0.71 msRTT As you can see down is littlebit more than half of full link speed. And upload is only 20-25% of full link. I tried to change a lot sysctl params but without a big results. Currenlty my entries in /etc/sysctl.conf which regarding to TCP: #kernel tuning, tcp kern.ipc.somaxconn=2048 kern.ipc.nmbclusters=32768 kern.ipc.maxsockbuf=8388608 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.inflight.enable=0 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.inet.udp.recvspace=65536 net.inet.tcp.inflight.enable=0 net.inet.tcp.rfc1323=1 net.inet.tcp.sack.enable=1 net.inet.tcp.path_mtu_discovery=1 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 Do you have some kind suggestion what i could to change to increase the performance of TCP? Besides when i make the benchamrks i run the sniffer to see whats happening with network..sometimes i saw that window size is 0...does it mean that server can't handle something or recieve buffer size is to small? p.s sory for my bad english :) Noisex _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.