From owner-freebsd-performance@FreeBSD.ORG Tue Jul 1 15:50:28 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C251106564A for ; Tue, 1 Jul 2008 15:50:28 +0000 (UTC) (envelope-from david.kwan@isilon.com) Received: from seaxch07.isilon.com (seaxch07.isilon.com [74.85.160.23]) by mx1.freebsd.org (Postfix) with ESMTP id E52018FC15 for ; Tue, 1 Jul 2008 15:50:27 +0000 (UTC) (envelope-from david.kwan@isilon.com) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Tue, 1 Jul 2008 08:38:27 -0700 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. Thread-Index: AcjbkIFABezCESyvRBm4kh4UjoEFBg== From: "David Kwan" To: X-Mailman-Approved-At: Tue, 01 Jul 2008 15:52:17 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 15:50:28 -0000 =20 I have a few questions regarding the TCP: =20 I have a situation with clients on a 100MB network connecting to servers on a Gigabit network where the client read speeds are very slow from the FreeBSD server and fast from the Linux server. Write speeds from the clients to both servers are fast. (Clients on the gigabit network work fine with blazing read and write speeds). The network traces shows congestion packets for both servers when doing reads from the clients (dup acks and retransmissions), but the Linux server seem to handle the congestion better. ECN is not enabled on the network and I don't see any congestion windowing or clients window changing. The 100/1G switch is dropping packets. I double checked the network configuration and also swapped swithports for the servers to use the others to make sure the switch configuration are the same and the Linux always does better than FreeBSD. Assuming that the network configuration is a constant for all clients and servers (speed, duplex, and etc...), the only variable is the servers themselves (Linux and FreeBSD). I have tried a couple of FreeBSD machines with 6.1 and 7.0 and they show the same problem, with no luck matching the speed and network utilization of Linux (2 years old). The read speed test I'm referring is doing transferring of a 100MB file (cifs, nfs, and ftp), and the Linux server does it constantly in around 10 sec (line speed) with a constant network utilization chart, while the FreeBSD servers are magnitudes slower with erratic network utilization chart. I've attempted to tweak some network sysctl options on the FreeBSD, and the only ones that helped were disabling TSO and inflight; which leads me to think that the inter-packet gap was slightly increased to partially relieve congestion on the switch; not a long term solution. =20 My questions are:=20 1. Have you heard of this problem before with 100MB clients to Gigabit servers? 2. Are you aware of any Linux fix/patch the TCP stack to better handling congestion than FreeBSD? I'm looking to address this issue in the FreeBSD, but wondering if the Linux stack did something special that can help with the FreeBSD performance. =20 David K. =20 From owner-freebsd-performance@FreeBSD.ORG Wed Jul 2 03:09:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 861461065685 for ; Wed, 2 Jul 2008 03:09:55 +0000 (UTC) (envelope-from hwahing@smartteam.net) Received: from mozart.smartteam.net (mozart.smartteam.net [202.186.153.96]) by mx1.freebsd.org (Postfix) with ESMTP id 510A68FC1A for ; Wed, 2 Jul 2008 03:09:53 +0000 (UTC) (envelope-from hwahing@smartteam.net) Received: from mozart.smartteam.net (localhost [127.0.0.1]) by mozart.smartteam.net (8.13.8/8.13.8) with ESMTP id m622K30v001186; Wed, 2 Jul 2008 10:20:04 +0800 Received: from [192.168.1.101] ( [60.53.52.153]) by mozart.smartteam.net (Scalix SMTP Relay 11.3.0.11339) via ESMTP; Wed, 02 Jul 2008 10:20:03 +0800 (MYT) Date: Wed, 2 Jul 2008 10:22:48 +0800 From: Hwa Hing To: David Kwan Message-ID: <486AE678.8050206@smartteam.net> In-Reply-To: References: x-scalix-Hops: 1 User-Agent: Thunderbird 2.0.0.14 (X11/20080505) X-Enigmail-Version: 0.95.6 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline Cc: freebsd-performance@freebsd.org Subject: Re: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 03:09:55 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I also had such problem with FreeBSD-7 Stable-SMP with 4 port Intel 1GBit NIC em Driver and with Packet Filter, ALTQ enabled. I tested with iperf from network a to network b. I found freebsd droping packets and the transfer speed in routed mode is only 2 mbit/s compare to linux router which able to forward the traffic at almost full speed. David Kwan wrote: > > > I have a few questions regarding the TCP: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server. Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see > any congestion windowing or clients window changing. The 100/1G switch > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple > of FreeBSD machines with 6.1 and 7.0 and they show the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > constantly in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to > Gigabit servers? > > 2. Are you aware of any Linux fix/patch the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkhq5ngACgkQn94C8abtJZqjGQCgguGT6NG4LknjD3El+dnamlBv Z1cAniBSYCJ7n2wi/MpWgRcdVhBWcMVT =6jzX -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Thu Jul 3 11:12:23 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5DCC1065674 for ; Thu, 3 Jul 2008 11:12:23 +0000 (UTC) (envelope-from tosha@inbox.com) Received: from WM28.inbox.com (wm28.inbox.com [64.135.83.28]) by mx1.freebsd.org (Postfix) with SMTP id 953F18FC14 for ; Thu, 3 Jul 2008 11:12:23 +0000 (UTC) (envelope-from tosha@inbox.com) Received: from inbox.com (127.0.0.1:25) by inbox.com with [InBox.Com SMTP Server] id <807030003325.WM28> for from ; Thu, 3 Jul 2008 3:02:14 AM -0800 Mime-Version: 1.0 Date: Thu, 3 Jul 2008 03:02:14 -0800 Message-ID: From: "Ing. Todor Colakov" Sender: tosha@inbox.com To: freebsd-performance@freebsd.org X-Mailer: INBOX.COM X-Originating-IP: 194.125.8.250 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-IWM-ACU: J2V6KCbjef1ZBeIt4AsP9ywkifDRDyLGTkP6XxSDyy6TbSIc2qIcnG9GVoLd uDIyRwqAbqc093JqUAMNqHLZx9z5ddO0gcrotlGVw8Dr4Io8gUNF0hCkBtV_ dq9J68Qs@ Subject: how to measure performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 11:12:23 -0000 Hello, I'm trying to measure performance of my servers. I would like to have (at = least rough) estimation, how much of punishment it can take, so I've = crawled web for some howtos.. i've founded some, it gave me some start, = but what i've miss in all of them was simple guide in the style... If you are setting new server, you are interested at THIS, THIS and THIS = (say how much packets incomming packets can it handle at one time) because = of THAT, THAT and THAT (because if you plan to put it into enviroment of = some magnitudes higher it wont do) and you can meassure it with tool A B C = ( dunno :( ) Can somebody help me compile this list? I would like address specially = experienced admins. I would like to have this list mainly for general = server (without specialization), mailserver, webserver, DNS and NFS server = - because it address services the most of newbiee admins work with. Thank to all=20 Todor From owner-freebsd-performance@FreeBSD.ORG Thu Jul 3 13:11:34 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42C27106568B for ; Thu, 3 Jul 2008 13:11:34 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id EA5E08FC17; Thu, 3 Jul 2008 13:11:15 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <486CCFEF.4080000@FreeBSD.org> Date: Thu, 03 Jul 2008 15:11:11 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: "Ing. Todor Colakov" References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: how to measure performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 13:11:34 -0000 Ing. Todor Colakov wrote: > Hello, > > I'm trying to measure performance of my servers. I would like to have (at least rough) estimation, how much of punishment it can take, so I've crawled web for some howtos.. i've founded some, it gave me some start, but what i've miss in all of them was simple guide in the style... > > If you are setting new server, you are interested at THIS, THIS and THIS (say how much packets incomming packets can it handle at one time) because of THAT, THAT and THAT (because if you plan to put it into enviroment of some magnitudes higher it wont do) and you can meassure it with tool A B C ( dunno :( ) > > Can somebody help me compile this list? I would like address specially experienced admins. I would like to have this list mainly for general server (without specialization), mailserver, webserver, DNS and NFS server - because it address services the most of newbiee admins work with. > Thank to all > Todor Perhaps the answer is because there is no single thing as "performance". Rather, "performance" can mean many things depending on what your requirements are. What do you use your server for? Kris From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 12:58:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 260921065680 for ; Fri, 4 Jul 2008 12:58:55 +0000 (UTC) (envelope-from tosha@inbox.com) Received: from WM28.inbox.com (wm28.inbox.com [64.135.83.28]) by mx1.freebsd.org (Postfix) with SMTP id B79928FC15 for ; Fri, 4 Jul 2008 12:58:54 +0000 (UTC) (envelope-from tosha@inbox.com) Received: from inbox.com (127.0.0.1:25) by inbox.com with [InBox.Com SMTP Server] id <807040003011.WM28> for from ; Fri, 4 Jul 2008 4:58:50 AM -0800 Mime-Version: 1.0 Date: Fri, 4 Jul 2008 04:58:50 -0800 Message-ID: From: "Ing. Todor Colakov" Sender: tosha@inbox.com To: freebsd-performance@freebsd.org X-Mailer: INBOX.COM X-Originating-IP: 194.125.8.250 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-IWM-ACU: b57bf1sBZ6DPHqlVnRzEyOTIeC07m7_g9N_zWKC5Iy0UjTYdOm2uN6G5JTUH QJzqMsLUuLnwlde-1m9OD394mxoOf6lLDO87iTSPWx7K1bgSUgraZD6_EL-u snRaMJRA@ Subject: re: performance meassuring X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 12:58:55 -0000 I have mailserver, webserver, DNS and NFS server. I know there is no = specific performance value, because of that I wanted make separated lists for those = basic types (Maybe my english wasn't descriptive enough for that :( ). And also I = wanted know why that or other value is important for the concrete type of server... I know it = all stands on the knowledges of HW architectures, kernel, FS and alike... but I don't have = these knowdleges (yet) and I need to meassure server. Because I need to have feedback, when = changing setting of server. Todor P.S. Can the number of packets per second be the optimistic assumption of = raw network processing power of the server? From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 15:00:02 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1EFDB1065678 for ; Fri, 4 Jul 2008 15:00:02 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.184]) by mx1.freebsd.org (Postfix) with ESMTP id 9D0308FC12 for ; Fri, 4 Jul 2008 15:00:01 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: by nf-out-0910.google.com with SMTP id h3so450566nfh.33 for ; Fri, 04 Jul 2008 08:00:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:subject:from:to:cc :in-reply-to:references:content-type:date:message-id:mime-version :x-mailer; bh=huUmysDDvlmbyMsJel+3kJ4czWW6/FPuldiiZEyNhDI=; b=ihyH0f7zMNrt/Y8DNcKwGX2eSLsL+6AFp9+WeGxSyNPWKvpqdoSvct+5nomSRN269s 8K5Mub0cEAd8h1TMufkCEsQmRkFAD8Fb7lydc7ChFVP2zJPSaz2bQyHMdEPJlz17JAC6 hS0G5zWkbFaCuMdkCI1cKXPjc0JFfakDBaDas= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=subject:from:to:cc:in-reply-to:references:content-type:date :message-id:mime-version:x-mailer; b=AdfbNpiEoLmMhviAWi1veaJ4EZYEZc3FT+vD7N6E03PmL1H8DPa1XUhwYtHq1UTocx bR3bTK838cL3tCseDzyEJD2SJnj49k/g7KdheZe7olRQRdWiBsrcbSTsJMTDnONoxpon +UZ5znafbbE8Zp/vjq9EZR9DxuR1egM9gQt5w= Received: by 10.210.21.6 with SMTP id 6mr389622ebu.184.1215181998442; Fri, 04 Jul 2008 07:33:18 -0700 (PDT) Received: from ?127.0.0.1? ( [217.206.187.80]) by mx.google.com with ESMTPS id j8sm505572gvb.1.2008.07.04.07.33.16 (version=SSLv3 cipher=RC4-MD5); Fri, 04 Jul 2008 07:33:16 -0700 (PDT) From: Tom Evans To: "Ing. Todor Colakov" In-Reply-To: References: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-M6NHUPBm08l6WpJj6i9Q" Date: Fri, 04 Jul 2008 15:33:14 +0100 Message-Id: <1215181994.35536.58.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 FreeBSD GNOME Team Port Cc: freebsd-performance@freebsd.org Subject: re: performance meassuring X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 15:00:02 -0000 --=-M6NHUPBm08l6WpJj6i9Q Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, 2008-07-04 at 04:58 -0800, Ing. Todor Colakov wrote: > I have mailserver, webserver, DNS and NFS server. I know there is no spec= ific > performance value, because of that I wanted make separated lists for thos= e basic types > (Maybe my english wasn't descriptive enough for that :( ). And also I wan= ted know why that > or other value is important for the concrete type of server... I know it = all stands on the > knowledges of HW architectures, kernel, FS and alike... but I don't have = these knowdleges > (yet) and I need to meassure server. Because I need to have feedback, whe= n changing > setting of server. > Todor > P.S. Can the number of packets per second be the optimistic assumption of= raw network > processing power of the server? PPS of what? If the server is a forwarding router, then PPS of 64 byte packets forwarded would be a good metric. If its a database server, then PPS is largely irrelevant. Generally speaking, work out what you want to change, find a good metric to measure the performance change (Eg, change filesystem/tuning options, run bonnie++), measure before and after. Even then, its not that simple, as bonnie++ doesnt accurately represent my FS-related workloads. The ideal way is to find a way of measuring $APPLICATION, since that is what the server is for. Tom --=-M6NHUPBm08l6WpJj6i9Q Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEABECAAYFAkhuNKYACgkQlcRvFfyds/fevwCffmCnm5b9dvHkhxnzMpeDSUre djkAn03v1AVOd/v2QJvEfmEzHExEz5Gf =oGvv -----END PGP SIGNATURE----- --=-M6NHUPBm08l6WpJj6i9Q-- From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 16:53:13 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FAA0106567A for ; Fri, 4 Jul 2008 16:53:13 +0000 (UTC) (envelope-from rabing@omc.net) Received: from office.omc.net (office.omc.net [212.77.224.22]) by mx1.freebsd.org (Postfix) with ESMTP id B91008FC15 for ; Fri, 4 Jul 2008 16:53:12 +0000 (UTC) (envelope-from rabing@omc.net) Received: from xoffice.omc.net (xoffice.omc.net [212.77.224.172]) by office.omc.net (8.14.1/8.14.1) with ESMTP id m64GDf4T088896 for ; Fri, 4 Jul 2008 18:13:41 +0200 (CEST) (envelope-from rabing@omc.net) Received: from [212.77.224.50] (lutz.omc.net [212.77.224.50]) by xoffice.omc.net (8.14.2/8.14.2) with ESMTP id m64GDfOm078941 for ; Fri, 4 Jul 2008 18:13:41 +0200 (CEST) (envelope-from rabing@omc.net) Message-ID: <486E4C20.7050102@omc.net> Date: Fri, 04 Jul 2008 18:13:20 +0200 From: Lutz Rabing User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-performance@freebsd.org X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 04 Jul 2008 17:20:01 +0000 Subject: low sysbench scores on 8 core server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rabing@omc.net List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 16:53:13 -0000 hi, I did some testing an a supermicro 2 x 4 core xeon server under 64bit "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load (2000 apache threads) the system performed bad compare to other dual core systems under the same workload. the 8 core system had 0% idle time and almost 100% system load. I could not find out what the load was. disk IO was close to zero during that time. because of that I checked the sysbench results with this test: sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run here are the sysbench results: - 1 thread -------------------------------------------------------- OLTP test statistics: queries performed: read: 354774 write: 0 other: 50682 total: 405456 transactions: 25341 (422.34 per sec.) deadlocks: 0 (0.00 per sec.) read/write requests: 354774 (5912.82 per sec.) other operations: 50682 (844.69 per sec.) - 8 threads ------------------------------------------------------- OLTP test statistics: queries performed: read: 145166 write: 0 other: 20738 total: 165904 transactions: 10369 (172.75 per sec.) deadlocks: 0 (0.00 per sec.) read/write requests: 145166 (2418.54 per sec.) other operations: 20738 (345.51 per sec.) -------------------------------------------------------------------- during the 8 thread test "top" looks like this: ----------------------------------------------------------------------------- last pid: 30104; load averages: 2.50, 0.65, 0.52 up 0+01:49:44 17:52:14 69 processes: 7 running, 62 sleeping CPU: 5.6% user, 0.0% nice, 64.0% system, 0.0% interrupt, 30.4% idle Mem: 44M Active, 296M Inact, 277M Wired, 84K Cache, 214M Buf, 15G Free Swap: 16G Total, 16G Free ----------------------------------------------------------------------------- software versions used: - mysql-server-5.1.25 - sysbench-0.4.8 any ideas what I should look for or what to change? I could supply more info if needed. dmesg appended. thanks, lutz ================================================================================================== FreeBSD 7.0-STABLE #1: Fri Jul 4 01:41:13 CEST 2008 root@www5.4eins.de:/usr/obj/usr/src/sys/OMC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (2333.35-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7 Features=0xbfebfbff Features2=0x4e3bd AMD Features=0x20100800 AMD Features2=0x1 Cores per package: 4 usable memory = 17166041088 (16370 MB) avail memory = 16626790400 (15856 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 2.0 on pci0 pci1: on pcib1 pcib2: irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 0.0 on pci2 pci3: on pcib3 pcib4: at device 2.0 on pci2 pci4: on pcib4 em0: port 0x2000-0x201f mem 0xda000000-0xda01ffff irq 18 at device 0.0 on pci4 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:30:48:79:cb:ba em1: port 0x2020-0x203f mem 0xda020000-0xda03ffff irq 19 at device 0.1 on pci4 em1: Using MSI interrupt em1: [FILTER] em1: Ethernet address: 00:30:48:79:cb:bb pcib5: at device 0.3 on pci1 pci5: on pcib5 pcib6: at device 4.0 on pci0 pci6: on pcib6 pcib7: at device 0.0 on pci6 pci7: on pcib7 pcib8: at device 0.2 on pci6 pci8: on pcib8 pcib9: at device 6.0 on pci0 pci9: on pcib9 pcib10: at device 0.0 on pci9 pci10: on pcib10 3ware device driver for 9000 series storage controllers, version: 3.70.05.001 twa0: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem 0xd8000000-0xd9ffffff,0xda300000-0xda300fff irq 18 at device 1.0 on pci10 twa0: [ITHREAD] twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue: twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SXU-4LP, 4 ports, Firmware FE9X 3.04.00.005, BIOS BE9X 3.04.00.002 pcib11: at device 0.2 on pci9 pci11: on pcib11 pci0: at device 8.0 (no driver attached) uhci0: port 0x1800-0x181f irq 17 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: on uhci0 usb0: USB revision 1.0 uhub0: on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: port 0x1820-0x183f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: on uhci1 usb1: USB revision 1.0 uhub1: on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: port 0x1840-0x185f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: on uhci2 usb2: USB revision 1.0 uhub2: on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: mem 0xda600400-0xda6007ff irq 17 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: on ehci0 usb3: USB revision 2.0 uhub3: on usb3 uhub3: 6 ports with 6 removable, self powered pcib12: at device 30.0 on pci0 pci12: on pcib12 vgapci0: port 0x4000-0x40ff mem 0xd0000000-0xd7ffffff,0xda200000-0xda20ffff irq 18 at device 1.0 on pci12 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xda600800-0xda600bff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI Version 01.10 controller with 6 ports detected ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] pci0: at device 31.3 (no driver attached) cpu0: on acpi0 est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 p4tcc1: on cpu1 cpu2: on acpi0 est2: on cpu2 p4tcc2: on cpu2 cpu3: on acpi0 est3: on cpu3 p4tcc3: on cpu3 cpu4: on acpi0 est4: on cpu4 p4tcc4: on cpu4 cpu5: on acpi0 est5: on cpu5 p4tcc5: on cpu5 cpu6: on acpi0 est6: on cpu6 p4tcc6: on cpu6 cpu7: on acpi0 est7: on cpu7 p4tcc7: on cpu7 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 acpi_button0: on acpi0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcc7ff on isa0 ppc0: cannot reserve I/O port range sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging limited to 100 packets/entry by default acd0: CDRW at ata0-slave UDMA33 SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #5 Launched! da0 at twa0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 100.000MB/s transfers da0: 1430490MB (2929643520 512 byte sectors: 255H 63S/T 182361C) Trying to mount root from ufs:/dev/da0s1a From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 18:25:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 124201065671 for ; Fri, 4 Jul 2008 18:25:05 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 594E48FC19; Fri, 4 Jul 2008 18:25:04 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <486E6B00.10208@FreeBSD.org> Date: Fri, 04 Jul 2008 20:25:04 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: rabing@omc.net References: <486E4C20.7050102@omc.net> In-Reply-To: <486E4C20.7050102@omc.net> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: low sysbench scores on 8 core server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 18:25:05 -0000 Lutz Rabing wrote: > hi, > > I did some testing an a supermicro 2 x 4 core xeon server under 64bit > "7.0-STABLE #1: Fri Jul 4". when we first tested the system under load > (2000 apache threads) the system performed bad compare to other dual > core systems under the same workload. > > the 8 core system had 0% idle time and almost 100% system load. I could > not find out what the load was. disk IO was close to zero during that time. > > because of that I checked the sysbench results with this test: > > sysbench --test=oltp --mysql-socket=/tmp/mysql.sock --mysql-user=root \ > --max-requests=0 --max-time=60 --oltp-read -only=on --num-threads=$1 run See my tuning notes at: http://people.freebsd.org/~kris/scaling/mysql.html > software versions used: > - mysql-server-5.1.25 In my tests mysql 5.1 has much worse performance than 5.0. Kris From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 18:30:37 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CA531065679 for ; Fri, 4 Jul 2008 18:30:37 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 863288FC1E; Fri, 4 Jul 2008 18:30:35 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <486E6C4B.6060902@FreeBSD.org> Date: Fri, 04 Jul 2008 20:30:35 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Hwa Hing References: <486AE678.8050206@smartteam.net> In-Reply-To: <486AE678.8050206@smartteam.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, David Kwan Subject: Re: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 18:30:37 -0000 Hwa Hing wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > I also had such problem with FreeBSD-7 Stable-SMP with 4 port Intel > 1GBit NIC em Driver and with Packet Filter, ALTQ enabled. But it doesn't sound like you are testing the same thing at all. > I tested with iperf from network a to network b. I found freebsd droping > packets and the transfer speed in routed mode is only 2 mbit/s compare > to linux router which able to forward the traffic at almost full speed. Check that the software isn't making invalid Linux-specific assumptions about e.g. socket buffer sizes. Kris From owner-freebsd-performance@FreeBSD.ORG Fri Jul 4 18:31:49 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F4BE1065678 for ; Fri, 4 Jul 2008 18:31:49 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D3D258FC20; Fri, 4 Jul 2008 18:31:48 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <486E6C95.1020509@FreeBSD.org> Date: Fri, 04 Jul 2008 20:31:49 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: David Kwan References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: TCP stack in FreeBSD poor performance in 100MB to Gigabit environment. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 18:31:49 -0000 David Kwan wrote: > > > I have a few questions regarding the TCP: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server. Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see > any congestion windowing or clients window changing. The 100/1G switch > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple > of FreeBSD machines with 6.1 and 7.0 and they show the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > constantly in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to > Gigabit servers? The key point seems to be that the switch is dropping packets. If you have packet loss then TCP is not going to perform well. Kris