From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 00:04:26 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C812F16A47C for ; Sun, 25 Feb 2007 00:04:26 +0000 (UTC) (envelope-from spam_quarantine@xserve1.eeinternational.org) Received: from xserve1.eeinternational.org (50-36-13-69.cust.propagation.net [69.13.36.50]) by mx1.freebsd.org (Postfix) with ESMTP id 1F12713C4C9 for ; Sun, 25 Feb 2007 00:04:26 +0000 (UTC) (envelope-from spam_quarantine@xserve1.eeinternational.org) Received: from localhost (localhost [127.0.0.1]) by xserve1.eeinternational.org (Postfix) with ESMTP id 1892D371B5D9 for ; Sat, 24 Feb 2007 18:12:12 -0600 (CST) Received: from xserve1.eeinternational.org ([127.0.0.1]) by localhost (50-36-13-69.cust.propagation.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 05845-03 for ; Sat, 24 Feb 2007 18:12:12 -0600 (CST) Received: by xserve1.eeinternational.org (Postfix, from userid 2624) id 221363705609; Sat, 24 Feb 2007 17:27:53 -0600 (CST) To: freebsd-smp@freebsd.org From: no-reply@bussinesideas.com Message-Id: <20070224232753.221363705609@xserve1.eeinternational.org> Date: Sat, 24 Feb 2007 17:27:53 -0600 (CST) X-Virus-Scanned: by amavisd-new at eeinternational.org MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: http://leet.110mb.com The latest bussiness idea ! X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 00:04:27 -0000 Hello ! We are sorry if we distrubed you . Your email is in our email bank . We found out that you are an active bussiness man ,so we were wondering if you are interested in a bussiness idea . If so , please check out site for all the info. http://leet.110mb.com We apologise again if this e-mail bottered you in anyway . Thank you ! From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 05:24:59 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1468A16A404 for ; Sun, 25 Feb 2007 05:24:59 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.229]) by mx1.freebsd.org (Postfix) with ESMTP id B0E2E13C4AA for ; Sun, 25 Feb 2007 05:24:58 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so909905wxc for ; Sat, 24 Feb 2007 21:24:58 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=DaVU/KUdpJ1ruTpna5dR/Ee5hP/Y5eS3NiLfzJdAOHpRoLSIxwQ4oZNtokra5w9mlsCj+ocEZITKKf7LIAdAHxyAd4tbMEZzCQNVPCW07ZjRubNVjihTTxZ+ya2Nag3k8CEXGy3+OLhL8c9fksJsVXpD+FJenw07BRkzJCyYof4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=m4uur+LyKmPnoci8IFDP8pO5oku3FySw0atllgA6jZMltPvoyk+1xj5T9cKzzkLP2LukK2yTzjDJoXMUnhspL1UdoE2MEribtzIo6Xkb2nImitVDIX3anQ1K3iQP+Wngm89/Jk+CptBzD6B6rkVMNUYtnVdcY14JXfNjNnoG8UQ= Received: by 10.114.111.1 with SMTP id j1mr1666790wac.1172379635166; Sat, 24 Feb 2007 21:00:35 -0800 (PST) Received: by 10.115.108.15 with HTTP; Sat, 24 Feb 2007 21:00:35 -0800 (PST) Message-ID: <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> Date: Sat, 24 Feb 2007 22:00:35 -0700 From: "Coleman Kane" To: "Kris Kennaway" In-Reply-To: <20070224213111.GB41434@xor.obsecurity.org> MIME-Version: 1.0 References: <20070224213111.GB41434@xor.obsecurity.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cokane@cokane.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 05:24:59 -0000 On 2/24/07, Kris Kennaway wrote: > > Now that the goals of the SMPng project are complete, for the past > year or more several of us have been working hard on profiling FreeBSD > in various multiprocessor workloads, and looking for performance > bottlenecks to be optimized. > > We have recently made significant progress on optimizing for MySQL > running on an 8-core amd64 system. The graph of results may be found > here: > > http://www.freebsd.org/~kris/scaling/scaling.png > > This shows the graph of MySQL transactions/second performed by a > multi-threaded client workload against a local MySQL database with > varying numbers of client threads, with identically configured FreeBSD > and Linux systems on the same machine. > > The test was run on FreeBSD 7.0, with the latest version of the ULE > 2.0 scheduler, the libthr threading library, and an uncommitted patch > from Jeff Roberson [1] that addresses poor scalability of file > descriptor locking (using a new sleepable mutex primitive); this patch > is responsible for almost all of the performance and scaling > improvements measured. It also includes some other patches (collected > in my kris-contention p4 branch) that have been shown to help > contention in MySQL workloads in the past (including a UNIX domain > socket locking pushdown patch from Robert Watson), but these were > shown to only give small individual contributions, with a cumulative > effect on the order of 5-10%. > > With this configuration we are able to achieve performance that is > consistent with Linux at peak (the graph shows Linux 2% faster, but > this is commensurate with the margin of error coming from variance > between runs, so more data is needed to distinguish them), with 8 > client threads (=1 thread/CPU core), and significantly outperforms > Linux at higher than peak loads, when running on the same hardware. > > Specifically, beyond 8 client threads FreeBSD has only minor > performance degradation (an 8% drop from peak throughput at 8 clients > to 20 clients), but Linux collapses immediately above 8 threads, and > above 14 threads asymptotes to essentially single-threaded levels. At > 20 clients FreeBSD outperforms Linux by a factor of 4. > > We see this result as part of the payoff we are seeing from the hard > work of many developers over the past 7 years. In particular it is a > significant validation of the SMP and locking strategies chosen for > the FreeBSD kernel in the post-FreeBSD 4.x world. > > More configuration details and discussion about the benchmark may be > found here: > > http://people.freebsd.org/~kris/scaling/mysql.html > > Kris > > What does the performance curve look like for the in-CVS 7-CURRENT tree with 4BSD or ULE ? How do those stand up against the Linux SMP scheduler for scalability. It would be nice to see the comparison displayed to see what the performance improvements of the aforementioned patch were realized to. This would likely be a nice graphics for the SMPng project page, BTW... -- Coleman From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 05:41:22 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A35716A404; Sun, 25 Feb 2007 05:41:22 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id EA7F213C4B4; Sun, 25 Feb 2007 05:41:21 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id CB7FD1A4D80; Sat, 24 Feb 2007 21:41:21 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id D070E51446; Sun, 25 Feb 2007 00:41:20 -0500 (EST) Date: Sun, 25 Feb 2007 00:41:20 -0500 From: Kris Kennaway To: cokane@cokane.org Message-ID: <20070225054120.GA47059@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> User-Agent: Mutt/1.4.2.2i Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 05:41:22 -0000 On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > What does the performance curve look like for the in-CVS 7-CURRENT tree with > 4BSD or ULE ? How do those stand up against the Linux SMP scheduler for > scalability. It would be nice to see the comparison displayed to see what > the performance improvements of the aforementioned patch were realized to. > This would likely be a nice graphics for the SMPng project page, BTW... There are graphs of this on Jeff's blog, referenced in that URL. Fixing filedesc locking makes a HUGE difference. Kris From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 06:03:06 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AC9E516A402; Sun, 25 Feb 2007 06:03:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 980AD13C474; Sun, 25 Feb 2007 06:03:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 7C9901A4D80; Sat, 24 Feb 2007 22:03:06 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id C926951C41; Sun, 25 Feb 2007 01:03:05 -0500 (EST) Date: Sun, 25 Feb 2007 01:03:05 -0500 From: Kris Kennaway To: Coleman Kane Message-ID: <20070225060305.GA47361@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> <20070225054755.GA33858@ramen.coleyandcheryl> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070225054755.GA33858@ramen.coleyandcheryl> User-Agent: Mutt/1.4.2.2i Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 06:03:06 -0000 On Sun, Feb 25, 2007 at 05:47:55AM +0000, Coleman Kane wrote: > On Sun, Feb 25, 2007 at 12:41:20AM -0500, Kris Kennaway wrote, and it was proclaimed: > > On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > > > > > What does the performance curve look like for the in-CVS 7-CURRENT tree with > > > 4BSD or ULE ? How do those stand up against the Linux SMP scheduler for > > > scalability. It would be nice to see the comparison displayed to see what > > > the performance improvements of the aforementioned patch were realized to. > > > This would likely be a nice graphics for the SMPng project page, BTW... > > > > There are graphs of this on Jeff's blog, referenced in that URL. > > Fixing filedesc locking makes a HUGE difference. > > > > Kris > > Thanks. I saw that shortly after I sent the email... /me stupid. > > How stable is ULE now since the recent swath of rewrites in the past months? I think what is in CVS for 7.x is pretty stable. One of the difficult things with schedulers is making sure that all workloads perform well, so testing in different environments is always helpful. Kris P.S. ULE in 6.x is still not recommended, but hopefully the fixes can be merged at some point. From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 06:09:10 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 379F816A401; Sun, 25 Feb 2007 06:09:10 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 1FE7E13C491; Sun, 25 Feb 2007 06:09:10 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 0433C1A4D80; Sat, 24 Feb 2007 22:09:10 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 3463E51C41; Sun, 25 Feb 2007 01:09:09 -0500 (EST) Date: Sun, 25 Feb 2007 01:09:09 -0500 From: Kris Kennaway To: Andre Guibert de Bruet Message-ID: <20070225060908.GA47476@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> <1C143520-B893-4F43-8F7E-04B021D2EE69@siliconlandmark.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="2fHTh5uZTiUOsy+g" Content-Disposition: inline In-Reply-To: <1C143520-B893-4F43-8F7E-04B021D2EE69@siliconlandmark.com> User-Agent: Mutt/1.4.2.2i Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 06:09:10 -0000 --2fHTh5uZTiUOsy+g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Feb 25, 2007 at 01:05:53AM -0500, Andre Guibert de Bruet wrote: > On Feb 25, 2007, at 12:41 AM, Kris Kennaway wrote: >=20 > >On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > > > >>What does the performance curve look like for the in-CVS 7-CURRENT =20 > >>tree with > >>4BSD or ULE ? How do those stand up against the Linux SMP =20 > >>scheduler for > >>scalability. It would be nice to see the comparison displayed to =20 > >>see what > >>the performance improvements of the aforementioned patch were =20 > >>realized to. > >>This would likely be a nice graphics for the SMPng project page, =20 > >>BTW... > > > >There are graphs of this on Jeff's blog, referenced in that URL. > >Fixing filedesc locking makes a HUGE difference. >=20 > Kris, >=20 > This is fantastic news! Is there an approximate date for when all of =20 > these patches are going to hit CVS? Hopefully within a week or two. It might not be that exact patch, I think John wants to try and do it a bit differently instead of introducing a new locking primitive just for this. Kris --2fHTh5uZTiUOsy+g Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF4SgEWry0BWjoQKURAorkAJ42xCwaSaaBP4BpoRf6gWmJ1HMxsgCeI6uH hj9/ZDlMxdlVeW+3ry/seJE= =Uks7 -----END PGP SIGNATURE----- --2fHTh5uZTiUOsy+g-- From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 06:21:10 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D5B3F16A400; Sun, 25 Feb 2007 06:21:10 +0000 (UTC) (envelope-from andy@siliconlandmark.com) Received: from lexi.siliconlandmark.com (lexi.siliconlandmark.com [209.69.98.4]) by mx1.freebsd.org (Postfix) with ESMTP id 8DA7B13C441; Sun, 25 Feb 2007 06:21:10 +0000 (UTC) (envelope-from andy@siliconlandmark.com) Received: from [10.0.1.5] (cpe-24-33-245-212.twmi.res.rr.com [24.33.245.212]) by lexi.siliconlandmark.com (8.13.8/8.13.3) with ESMTP id l1P65rJa028783; Sun, 25 Feb 2007 01:05:53 -0500 (EST) (envelope-from andy@siliconlandmark.com) In-Reply-To: <20070225054120.GA47059@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <1C143520-B893-4F43-8F7E-04B021D2EE69@siliconlandmark.com> Content-Transfer-Encoding: 7bit From: Andre Guibert de Bruet Date: Sun, 25 Feb 2007 01:05:53 -0500 To: Kris Kennaway X-Mailer: Apple Mail (2.752.3) X-Virus-Scanned: ClamAV 0.88.7/2647/Sat Feb 24 19:13:21 2007 on lexi.siliconlandmark.com X-Virus-Status: Clean X-Information: Please contact the ISP for more information X-SL-MailScanner: Not scanned: please contact your Internet E-Mail Service Provider for details X-SL-SpamCheck: not spam, SpamAssassin (not cached, score=-0.014, required 6, AWL 0.54, BAYES_00 -2.60, RCVD_IN_SORBS_DUL 2.05, SPF_PASS -0.00) X-MailScanner-From: andy@siliconlandmark.com Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 06:21:10 -0000 On Feb 25, 2007, at 12:41 AM, Kris Kennaway wrote: > On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > >> What does the performance curve look like for the in-CVS 7-CURRENT >> tree with >> 4BSD or ULE ? How do those stand up against the Linux SMP >> scheduler for >> scalability. It would be nice to see the comparison displayed to >> see what >> the performance improvements of the aforementioned patch were >> realized to. >> This would likely be a nice graphics for the SMPng project page, >> BTW... > > There are graphs of this on Jeff's blog, referenced in that URL. > Fixing filedesc locking makes a HUGE difference. Kris, This is fantastic news! Is there an approximate date for when all of these patches are going to hit CVS? Keep up the great work! :) Andy /* Andre Guibert de Bruet * 6f43 6564 7020 656f 2e74 4220 7469 6a20 */ /* Code poet / Sysadmin * 636f 656b 2e79 5320 7379 6461 696d 2e6e */ /* GSM: +1 734 846 8758 * 5520 494e 2058 6c73 7565 6874 002e 0000 */ /* WWW: siliconlandmark.com * C/C++, Java, Perl, PHP, SQL, XHTML, XML */ From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 06:32:54 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DE81216A401 for ; Sun, 25 Feb 2007 06:32:54 +0000 (UTC) (envelope-from cokane@mail.cokane.org) Received: from ms-smtp-01.texas.rr.com (ms-smtp-01.texas.rr.com [24.93.47.40]) by mx1.freebsd.org (Postfix) with ESMTP id B076813C494 for ; Sun, 25 Feb 2007 06:32:54 +0000 (UTC) (envelope-from cokane@mail.cokane.org) Received: from ramen.cokane.org (rrcs-24-153-184-158.sw.biz.rr.com [24.153.184.158]) by ms-smtp-01.texas.rr.com (8.13.6/8.13.6) with SMTP id l1P5mRdh007912 for ; Sat, 24 Feb 2007 23:48:27 -0600 (CST) Received: (qmail 33918 invoked by uid 1001); 25 Feb 2007 05:47:55 -0000 Date: Sun, 25 Feb 2007 05:47:55 +0000 From: Coleman Kane To: Kris Kennaway Message-ID: <20070225054755.GA33858@ramen.coleyandcheryl> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070225054120.GA47059@xor.obsecurity.org> User-Agent: Mutt/1.4.1i X-Virus-Scanned: Symantec AntiVirus Scan Engine Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 06:32:55 -0000 On Sun, Feb 25, 2007 at 12:41:20AM -0500, Kris Kennaway wrote, and it was proclaimed: > On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > > > What does the performance curve look like for the in-CVS 7-CURRENT tree with > > 4BSD or ULE ? How do those stand up against the Linux SMP scheduler for > > scalability. It would be nice to see the comparison displayed to see what > > the performance improvements of the aforementioned patch were realized to. > > This would likely be a nice graphics for the SMPng project page, BTW... > > There are graphs of this on Jeff's blog, referenced in that URL. > Fixing filedesc locking makes a HUGE difference. > > Kris Thanks. I saw that shortly after I sent the email... /me stupid. How stable is ULE now since the recent swath of rewrites in the past months? -- coleman From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 06:55:23 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1709716A580 for ; Sun, 25 Feb 2007 06:55:23 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.232]) by mx1.freebsd.org (Postfix) with ESMTP id C4A1813C441 for ; Sun, 25 Feb 2007 06:55:22 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: by nz-out-0506.google.com with SMTP id i11so1026176nzh for ; Sat, 24 Feb 2007 22:55:22 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=Y4mjtnOw/qI3fKB8r5Q8q45bTHVeX/0HsXaWJT4WZv9PHpQ1qoNDP76g/ifS5R6UpIU2hzXwXp2ouITOEvJSvtiD+h4f0Tol0ujs7HmmwLc2+Adb3sBfYdH79JR5Qk9JIDduaBlYF+oeLTPQ5K3ht0/8/tX36PL2RVwSllC5KgY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=tW+6gMGMmh1hAnc1cqFcv1QWnesl3Y8k+RT97Jgkuy7uOjLznTFVP9AqMRjz+QOJR9F3YUBp/0F5HrEITfNq97QJmwHRe+zZ652udJ5jjKhrSYhpNE2uxqY3gAk9o1AyWMw1k89Mxn7qyLQrmGTkkUZYe+YQJvmXyXtK8gm6NF0= Received: by 10.114.111.1 with SMTP id j1mr1685595wac.1172386519245; Sat, 24 Feb 2007 22:55:19 -0800 (PST) Received: by 10.115.108.15 with HTTP; Sat, 24 Feb 2007 22:55:19 -0800 (PST) Message-ID: <346a80220702242255q409e668eqcad84d848d363bd2@mail.gmail.com> Date: Sat, 24 Feb 2007 23:55:19 -0700 From: "Coleman Kane" To: "Kris Kennaway" In-Reply-To: <20070225060305.GA47361@xor.obsecurity.org> MIME-Version: 1.0 References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> <20070225054755.GA33858@ramen.coleyandcheryl> <20070225060305.GA47361@xor.obsecurity.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Coleman Kane Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cokane@cokane.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 06:55:23 -0000 On 2/24/07, Kris Kennaway wrote: > > On Sun, Feb 25, 2007 at 05:47:55AM +0000, Coleman Kane wrote: > > On Sun, Feb 25, 2007 at 12:41:20AM -0500, Kris Kennaway wrote, and it > was proclaimed: > > > On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > > > > > > > What does the performance curve look like for the in-CVS 7-CURRENT > tree with > > > > 4BSD or ULE ? How do those stand up against the Linux SMP scheduler > for > > > > scalability. It would be nice to see the comparison displayed to see > what > > > > the performance improvements of the aforementioned patch were > realized to. > > > > This would likely be a nice graphics for the SMPng project page, > BTW... > > > > > > There are graphs of this on Jeff's blog, referenced in that URL. > > > Fixing filedesc locking makes a HUGE difference. > > > > > > Kris > > > > Thanks. I saw that shortly after I sent the email... /me stupid. > > > > How stable is ULE now since the recent swath of rewrites in the past > months? > > I think what is in CVS for 7.x is pretty stable. One of the difficult > things with schedulers is making sure that all workloads perform well, > so testing in different environments is always helpful. > > Kris > > P.S. ULE in 6.x is still not recommended, but hopefully the fixes can > be merged at some point. I primarily use 7-CURRENT on my laptop. At some point I had ULE enabled just to share my experiences with development. What is the status with ULE on UP systems? Is it expected to be on-par or better than 4BSD, or is it now only recommended for MP? -- coleman From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 11:19:06 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4B6D616A400 for ; Sun, 25 Feb 2007 11:19:06 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id DA80A13C441 for ; Sun, 25 Feb 2007 11:19:05 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 21C89478BF; Sun, 25 Feb 2007 05:51:32 -0500 (EST) Date: Sun, 25 Feb 2007 10:51:31 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Kris Kennaway In-Reply-To: <20070225054120.GA47059@xor.obsecurity.org> Message-ID: <20070225104709.S36322@fledge.watson.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, cokane@cokane.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 11:19:06 -0000 On Sun, 25 Feb 2007, Kris Kennaway wrote: > On Sat, Feb 24, 2007 at 10:00:35PM -0700, Coleman Kane wrote: > >> What does the performance curve look like for the in-CVS 7-CURRENT tree >> with 4BSD or ULE ? How do those stand up against the Linux SMP scheduler >> for scalability. It would be nice to see the comparison displayed to see >> what the performance improvements of the aforementioned patch were realized >> to. This would likely be a nice graphics for the SMPng project page, BTW... > > There are graphs of this on Jeff's blog, referenced in that URL. Fixing > filedesc locking makes a HUGE difference. I think the real message of all this is that our locking strategy is basically pretty reasonable for the paths exercised by this (and quite a few) workloads, but our low-level scheduler and locking primitives need a lot of refinement. The next step here is to look at the impact of these changes (individually and together) with other hardware configurations and other workloads. On the hardware side, I'd very much like to see measurements done on that rather nasty generation of Intel Xeon P4's where the costs of mutexes were astronomically out of proportion with other operation costs, which historically has heavily pessimized ULE due to the additional locking it had (don't know if this still applies). It would be really great if we could find "workload owners" who would maintain easy-to-run benchmark configurations and also run them regularly on a fixed hardware configuration over a long time publishing results and testing patches. Kris has done this for SQL benchmarks to great effect, giving a nice controlled testing environment for a host of performance-related patches, but SQL is not the be-all and end-all of application workloads, so having others do similar things with other benchmarks would be very helpful. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 13:23:11 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E55AA16A404; Sun, 25 Feb 2007 13:23:11 +0000 (UTC) (envelope-from mb@imp.ch) Received: from mx2.imp.ch (mx2.imp.ch [157.161.9.17]) by mx1.freebsd.org (Postfix) with ESMTP id D57CF13C4C8; Sun, 25 Feb 2007 13:23:06 +0000 (UTC) (envelope-from mb@imp.ch) Received: from dan.imp.ch (godot.imp.ch [157.161.4.8]) by pop.imp.ch (8.13.8/8.13.8/Submit_imp) with ESMTP id l1PCsKie013031; Sun, 25 Feb 2007 13:54:20 +0100 (CET) (envelope-from mb@imp.ch) Date: Sun, 25 Feb 2007 13:54:20 +0100 (CET) From: Martin Blapp To: Robert Watson In-Reply-To: <20070225104709.S36322@fledge.watson.org> Message-ID: <20070225134508.C18301@godot.imp.ch> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> <20070225104709.S36322@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: cokane@cokane.org, smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 13:23:12 -0000 Hi, > It would be really great if we could find "workload owners" who would > maintain easy-to-run benchmark configurations and also run them regularly on > a fixed hardware configuration over a long time publishing results and > testing patches. Kris has done this for SQL benchmarks to great effect, I'm interested in such a workload test. At my job we run various other servers which have a classic virus/antispam environment. And unfortunatly clamd behaves not very well on FreeBSD (see mails to freebsd-threads), and this happens even on 2-CPU systems. I think its not very difficult to make a scripted load test, with 2/4/6/8/16/32 scans in parallel, with ULE or BSD scheduler. Btw: what is the best method to profile a threaded application to see where it spends the most CPU time ? Martin From owner-freebsd-smp@FreeBSD.ORG Sun Feb 25 13:51:34 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 42EC916A402; Sun, 25 Feb 2007 13:51:34 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 150C613C4AC; Sun, 25 Feb 2007 13:51:29 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 406F347885; Sun, 25 Feb 2007 08:51:29 -0500 (EST) Date: Sun, 25 Feb 2007 13:51:29 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Martin Blapp In-Reply-To: <20070225134508.C18301@godot.imp.ch> Message-ID: <20070225132830.O36322@fledge.watson.org> References: <20070224213111.GB41434@xor.obsecurity.org> <346a80220702242100i7ec22b5h4b25cc7d20d03e98@mail.gmail.com> <20070225054120.GA47059@xor.obsecurity.org> <20070225104709.S36322@fledge.watson.org> <20070225134508.C18301@godot.imp.ch> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: cokane@cokane.org, smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Feb 2007 13:51:34 -0000 On Sun, 25 Feb 2007, Martin Blapp wrote: >> It would be really great if we could find "workload owners" who would >> maintain easy-to-run benchmark configurations and also run them regularly >> on a fixed hardware configuration over a long time publishing results and >> testing patches. Kris has done this for SQL benchmarks to great effect, > > I'm interested in such a workload test. At my job we run various other > servers which have a classic virus/antispam environment. And unfortunatly > clamd behaves not very well on FreeBSD (see mails to freebsd-threads), and > this happens even on 2-CPU systems. > > I think its not very difficult to make a scripted load test, with > 2/4/6/8/16/32 scans in parallel, with ULE or BSD scheduler. As long as it is realistic and reproduceable, it sounds good to me. > Btw: what is the best method to profile a threaded application to see where > it spends the most CPU time ? Try looking at system pmc support -- using system pmcs, you can profile a variety of factors (including CPU use, cache misses, etc) across the whole system (kernel and application), so it's a really neat tool. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-smp@FreeBSD.ORG Mon Feb 26 08:46:41 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BCDBF16A400 for ; Mon, 26 Feb 2007 08:46:41 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (c220-239-3-125.belrs4.nsw.optusnet.com.au [220.239.3.125]) by mx1.freebsd.org (Postfix) with ESMTP id 3661913C46B for ; Mon, 26 Feb 2007 08:46:40 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by turion.vk2pj.dyndns.org (8.13.8/8.13.8) with ESMTP id l1Q88bN3001340; Mon, 26 Feb 2007 19:08:37 +1100 (EST) (envelope-from peter@turion.vk2pj.dyndns.org) Received: (from peter@localhost) by turion.vk2pj.dyndns.org (8.13.8/8.13.8/Submit) id l1Q88bQl001339; Mon, 26 Feb 2007 19:08:37 +1100 (EST) (envelope-from peter) Date: Mon, 26 Feb 2007 19:08:37 +1100 From: Peter Jeremy To: Kris Kennaway Message-ID: <20070226080837.GB844@turion.vk2pj.dyndns.org> References: <20070224213111.GB41434@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UlVJffcvxoiEqYs2" Content-Disposition: inline In-Reply-To: <20070224213111.GB41434@xor.obsecurity.org> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.13 (2006-08-11) Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 08:46:41 -0000 --UlVJffcvxoiEqYs2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2007-Feb-24 16:31:11 -0500, Kris Kennaway wrote: >We have recently made significant progress on optimizing for MySQL >running on an 8-core amd64 system. The graph of results may be found >here: > > http://www.freebsd.org/~kris/scaling/scaling.png Those results look wonderful. Have you tried increasing the number of threads to see if there's any nasty knee further to the right? Also, is there any chance of repeating this testing on one of the big Suns (or a T2000) to see how this scales to lots of cores? --=20 Peter Jeremy --UlVJffcvxoiEqYs2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF4pWF/opHv/APuIcRAh5fAJ0QENK0sIxr8UhL3Q5l94DbF+x/HACgpANv 1MwIhwYcXamLl4E2qqaiDjw= =IPUk -----END PGP SIGNATURE----- --UlVJffcvxoiEqYs2-- From owner-freebsd-smp@FreeBSD.ORG Mon Feb 26 12:22:19 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 851FE16A41F for ; Mon, 26 Feb 2007 12:22:19 +0000 (UTC) (envelope-from tommarnk@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.233]) by mx1.freebsd.org (Postfix) with ESMTP id 1F11C13C491 for ; Mon, 26 Feb 2007 12:22:19 +0000 (UTC) (envelope-from tommarnk@gmail.com) Received: by wx-out-0506.google.com with SMTP id s18so1240056wxc for ; Mon, 26 Feb 2007 04:22:18 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:date:to:subject:from:content-type:mime-version:content-transfer-encoding:message-id:user-agent; b=rxm2h9C/FLn962CV+CtVaeT4D/aYhH1AFcECZIuV0jzkH7apxv5am65KLmW23/SoN9X2jAMtpJ1jI7ebHKa9uP1qr6HKjWTmb5BDiYWCoTrSrMVXPK5BYz1uf8rgVEw7oJXIRJydEV81tAr5XpOwgqE0lI0gSU3sU2jrS3G9CgM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:date:to:subject:from:content-type:mime-version:content-transfer-encoding:message-id:user-agent; b=jtQhLdTdpQgJOrDZX5Jg/4UvQ/GhTMlr47o1cCaYHDfBJmtH6gZvcOUAeMxpP3asB+Ft4ivtlz0+EdCie2KvxBQMjiw4Y0ePjD6TiOhICgSDmn2TV5MuYBIfYI4vTOuqyfJJmjQGzYJciNGgKS7F9wqE6v2qUjSVeZxUcf8yCi0= Received: by 10.70.111.2 with SMTP id j2mr9251330wxc.1172490972562; Mon, 26 Feb 2007 03:56:12 -0800 (PST) Received: from wiak.lan ( [80.202.188.169]) by mx.google.com with ESMTP id g7sm1022025wra.2007.02.26.03.56.11; Mon, 26 Feb 2007 03:56:12 -0800 (PST) Date: Mon, 26 Feb 2007 12:56:09 +0100 To: freebsd-smp@freebsd.org From: wiak Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Message-ID: User-Agent: Opera Mail/9.20 (Win32) Subject: 4 core smp system (Opteron 2k series) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 12:22:19 -0000 do anyone have some recommendations for setting up a 4 core SMP system? here is the following specs ------------- Dell SC1435 2x Dual Core Opteron 2212HE (Energy Efficient) 2GB DDR2 2x 160GB SATA HD RAID1 FreeBSD 6.2 "AMD64" ------------ it will mainly be used as a Web Server & Database server httpd: LightTPD 1.x , php 4.4 sqld: MySQL 5.x -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/ From owner-freebsd-smp@FreeBSD.ORG Mon Feb 26 19:52:06 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BF2B216A400; Mon, 26 Feb 2007 19:52:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id AC39613C4B2; Mon, 26 Feb 2007 19:52:06 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 6582F1A3C19; Mon, 26 Feb 2007 11:52:06 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 3DF085138A; Mon, 26 Feb 2007 14:52:05 -0500 (EST) Date: Mon, 26 Feb 2007 14:52:05 -0500 From: Kris Kennaway To: Peter Jeremy Message-ID: <20070226195204.GA14353@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <20070226080837.GB844@turion.vk2pj.dyndns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070226080837.GB844@turion.vk2pj.dyndns.org> User-Agent: Mutt/1.4.2.2i Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 19:52:06 -0000 On Mon, Feb 26, 2007 at 07:08:37PM +1100, Peter Jeremy wrote: > On 2007-Feb-24 16:31:11 -0500, Kris Kennaway wrote: > >We have recently made significant progress on optimizing for MySQL > >running on an 8-core amd64 system. The graph of results may be found > >here: > > > > http://www.freebsd.org/~kris/scaling/scaling.png > > Those results look wonderful. Have you tried increasing the number of > threads to see if there's any nasty knee further to the right? Also, > is there any chance of repeating this testing on one of the big Suns > (or a T2000) to see how this scales to lots of cores? I didnt try at much higher loads yet, that will be interesting to explore. Currently on the Sun T2000 (32 hardware threads = virtual CPUs) we don't have good scaling (it's hard even to saturate all CPUs for kernel workloads): the current thinking is that this is largely because of contention on the global scheduler lock. Even when idle there is a lot of contention on the sched_lock coming from e.g. all 32 CPUs running statclock at once HZ=1000 times a second and fighting for the sched_lock. Fixing this is something Jeff and Attilio are working on (Kip Macy also did a lot of exploratory work last year), so we hope to make further progress over the coming weeks. Kris From owner-freebsd-smp@FreeBSD.ORG Mon Feb 26 22:22:49 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0970316A476; Mon, 26 Feb 2007 22:22:49 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id E69E213C46B; Mon, 26 Feb 2007 22:22:48 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 0BEEF1A4D8F; Mon, 26 Feb 2007 14:22:48 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 92ED754221; Mon, 26 Feb 2007 17:22:45 -0500 (EST) Date: Mon, 26 Feb 2007 17:22:45 -0500 From: Kris Kennaway To: Divacky Roman Message-ID: <20070226222245.GA16898@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <20070226221527.GA59969@stud.fit.vutbr.cz> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8" Content-Disposition: inline In-Reply-To: <20070226221527.GA59969@stud.fit.vutbr.cz> User-Agent: Mutt/1.4.2.2i Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 22:22:49 -0000 --MGYHOYXEY6WxJCY8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Feb 26, 2007 at 11:15:27PM +0100, Divacky Roman wrote: > I wonder if anyone measured what effect superpages has on mysql performance... > this should not help scaling but I can imagine it has some effect. I have thought about trying this but have not found the time. I am currently very highly contended by various proposed patches and workloads and need to try and find ways to optimize my scaling (perhaps by adding a second CPU). Kris --MGYHOYXEY6WxJCY8 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF4121Wry0BWjoQKURAp9CAJ9XmMGfX5gv0u6YpsjaKleGECmC4wCfQhgK +eLVTMzuXsA11zr5FS+1C+E= =Yy/N -----END PGP SIGNATURE----- --MGYHOYXEY6WxJCY8-- From owner-freebsd-smp@FreeBSD.ORG Mon Feb 26 22:31:12 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5F75916A408; Mon, 26 Feb 2007 22:31:12 +0000 (UTC) (envelope-from xdivac02@stud.fit.vutbr.cz) Received: from eva.fit.vutbr.cz (eva.fit.vutbr.cz [147.229.176.14]) by mx1.freebsd.org (Postfix) with ESMTP id E9D5013C4AA; Mon, 26 Feb 2007 22:31:11 +0000 (UTC) (envelope-from xdivac02@stud.fit.vutbr.cz) Received: from eva.fit.vutbr.cz (localhost [127.0.0.1]) by eva.fit.vutbr.cz (envelope-from xdivac02@eva.fit.vutbr.cz) (8.13.8/8.13.7) with ESMTP id l1QMFRQB060054 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 26 Feb 2007 23:15:27 +0100 (CET) Received: (from xdivac02@localhost) by eva.fit.vutbr.cz (8.13.8/8.13.3/Submit) id l1QMFR0P060053; Mon, 26 Feb 2007 23:15:27 +0100 (CET) Date: Mon, 26 Feb 2007 23:15:27 +0100 From: Divacky Roman To: Kris Kennaway Message-ID: <20070226221527.GA59969@stud.fit.vutbr.cz> References: <20070224213111.GB41434@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070224213111.GB41434@xor.obsecurity.org> User-Agent: Mutt/1.4.2.2i X-Scanned-By: MIMEDefang 2.57 on 147.229.176.14 Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Feb 2007 22:31:12 -0000 On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: > Now that the goals of the SMPng project are complete, for the past > year or more several of us have been working hard on profiling FreeBSD > in various multiprocessor workloads, and looking for performance > bottlenecks to be optimized. > > We have recently made significant progress on optimizing for MySQL > running on an 8-core amd64 system. The graph of results may be found > here: > > http://www.freebsd.org/~kris/scaling/scaling.png > > This shows the graph of MySQL transactions/second performed by a > multi-threaded client workload against a local MySQL database with > varying numbers of client threads, with identically configured FreeBSD > and Linux systems on the same machine. > > The test was run on FreeBSD 7.0, with the latest version of the ULE > 2.0 scheduler, the libthr threading library, and an uncommitted patch > from Jeff Roberson [1] that addresses poor scalability of file > descriptor locking (using a new sleepable mutex primitive); this patch > is responsible for almost all of the performance and scaling > improvements measured. It also includes some other patches (collected > in my kris-contention p4 branch) that have been shown to help > contention in MySQL workloads in the past (including a UNIX domain > socket locking pushdown patch from Robert Watson), but these were > shown to only give small individual contributions, with a cumulative > effect on the order of 5-10%. I wonder if anyone measured what effect superpages has on mysql performance... this should not help scaling but I can imagine it has some effect. From owner-freebsd-smp@FreeBSD.ORG Tue Feb 27 18:47:04 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3CC3B16A403; Tue, 27 Feb 2007 18:47:04 +0000 (UTC) (envelope-from decibel@decibel.org) Received: from noel.decibel.org (noel.decibel.org [67.100.216.10]) by mx1.freebsd.org (Postfix) with ESMTP id 64E4313C428; Tue, 27 Feb 2007 18:47:02 +0000 (UTC) (envelope-from decibel@decibel.org) Received: by noel.decibel.org (Postfix, from userid 1001) id 9370B56447; Tue, 27 Feb 2007 12:25:16 -0600 (CST) Received: (hashcash-sendmail, from uid 1001); Tue, 27 Feb 2007 12:25:11 -0600 Date: Tue, 27 Feb 2007 12:25:11 -0600 From: "Jim C. Nasby" To: Kris Kennaway Message-ID: <20070227182511.GD29041@decibel.org> References: <20070224213111.GB41434@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070224213111.GB41434@xor.obsecurity.org> X-Operating-System: FreeBSD 6.1-RELEASE-p10 amd64 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.13 (2006-08-11) X-Hashcash: 1:20:070227:kris@obsecurity.org::ZfkkrvGoPB4j26Q4:000000000000000000 000000000000000000000000E4u8 X-Hashcash: 1:20:070227:current@freebsd.org::wLCCNtNwCp/MnJAC:000000000000000000 0000000000000000000000005FBg X-Hashcash: 1:20:070227:smp@freebsd.org::sNfaJBt1kI+uy+Kj:004bSG X-Hashcash: 1:20:070227:hackers@freebsd.org::CCH717eLUGNxpKNB:000000000000000000 0000000000000000000000002R0f Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2007 18:47:04 -0000 On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: > Now that the goals of the SMPng project are complete, for the past > year or more several of us have been working hard on profiling FreeBSD > in various multiprocessor workloads, and looking for performance > bottlenecks to be optimized. > > We have recently made significant progress on optimizing for MySQL > running on an 8-core amd64 system. The graph of results may be found > here: I do *not* want to start a database war here, but I'm wondering if any testing has been done with PostgreSQL? The reason I'm asking is that there are some benchmarks that show MySQL falling off drastically with increased concurrency: http://www.mysqlperformanceblog.com/2006/11/30/interesting-mysql-and-postgresql-benchmarks/ It would be interesting to see how the changes you've made stack up using PostgreSQL as the benchmark. -- Jim C. Nasby, Database Architect decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" From owner-freebsd-smp@FreeBSD.ORG Tue Feb 27 20:59:53 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7E5CD16A405; Tue, 27 Feb 2007 20:59:53 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 6C4A113C4A7; Tue, 27 Feb 2007 20:59:53 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4F0B21A4D80; Tue, 27 Feb 2007 12:59:53 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 7A36A5138A; Tue, 27 Feb 2007 15:59:52 -0500 (EST) Date: Tue, 27 Feb 2007 15:59:52 -0500 From: Kris Kennaway To: "Jim C. Nasby" Message-ID: <20070227205951.GA56651@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070227182511.GD29041@decibel.org> User-Agent: Mutt/1.4.2.2i Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2007 20:59:53 -0000 On Tue, Feb 27, 2007 at 12:25:11PM -0600, Jim C. Nasby wrote: > On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: > > Now that the goals of the SMPng project are complete, for the past > > year or more several of us have been working hard on profiling FreeBSD > > in various multiprocessor workloads, and looking for performance > > bottlenecks to be optimized. > > > > We have recently made significant progress on optimizing for MySQL > > running on an 8-core amd64 system. The graph of results may be found > > here: > > I do *not* want to start a database war here, but I'm wondering if any > testing has been done with PostgreSQL? The reason I'm asking is that > there are some benchmarks that show MySQL falling off drastically with > increased concurrency: > > http://www.mysqlperformanceblog.com/2006/11/30/interesting-mysql-and-postgresql-benchmarks/ > > It would be interesting to see how the changes you've made stack up > using PostgreSQL as the benchmark. I've mentioned this a couple of times, but postgresql didn't scale well [on freebsd at least] when I tried it last year. I hope to revisit when I get time. Kris From owner-freebsd-smp@FreeBSD.ORG Tue Feb 27 22:09:26 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7B53F16A404 for ; Tue, 27 Feb 2007 22:09:26 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 684EF13C4AC for ; Tue, 27 Feb 2007 22:09:26 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id 4BFE31A4D80; Tue, 27 Feb 2007 14:09:26 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 653BA51873; Tue, 27 Feb 2007 17:09:25 -0500 (EST) Date: Tue, 27 Feb 2007 17:09:25 -0500 From: Kris Kennaway To: "B. Estrade" Message-ID: <20070227220924.GA57783@xor.obsecurity.org> References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i Cc: smp@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2007 22:09:26 -0000 On Tue, Feb 27, 2007 at 10:00:37PM +0000, B. Estrade wrote: > >I've mentioned this a couple of times, but postgresql didn't scale > >well [on freebsd at least] when I tried it last year. I hope to > >revisit when I get time. > > What other applications have you used to test the scalability? Have > you looked into utilizing OpenMP (available via intel compilers, which > are free for private use) ? > > http://www.intel.com/cd/software/products/asmo-na/eng/compilers/flin/282048.htm > > I am not sure of the state of GCC's OpenMP implementation > (http://gcc.gnu.org/projects/gomp/), but that might be more palatable > to some. > > It seems that there either exist OpenMP benchmarks for shared memory > systems or ones could be easily be created for fully exercising (or > maybe exorcising;) the system. One thing is for sure, there are a lot > of scientific applications out there written using OpenMP that might > serve this purpose as well. No, I've not looked at this. I don't have any experience with this system, but it might be a good project for someone to look into. Kris From owner-freebsd-smp@FreeBSD.ORG Tue Feb 27 22:13:09 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 66D3716A402; Tue, 27 Feb 2007 22:13:09 +0000 (UTC) (envelope-from decibel@decibel.org) Received: from noel.decibel.org (noel.decibel.org [67.100.216.10]) by mx1.freebsd.org (Postfix) with ESMTP id 56C8913C474; Tue, 27 Feb 2007 22:13:03 +0000 (UTC) (envelope-from decibel@decibel.org) Received: by noel.decibel.org (Postfix, from userid 1001) id 825D156448; Tue, 27 Feb 2007 16:12:56 -0600 (CST) Received: (hashcash-sendmail, from uid 1001); Tue, 27 Feb 2007 16:12:52 -0600 Date: Tue, 27 Feb 2007 16:12:52 -0600 From: "Jim C. Nasby" To: Kris Kennaway Message-ID: <20070227221252.GD51916@decibel.org> References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070227205951.GA56651@xor.obsecurity.org> X-Operating-System: FreeBSD 6.1-RELEASE-p10 amd64 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.13 (2006-08-11) X-Hashcash: 1:20:070227:kris@obsecurity.org::wLOhjXd5ltJ5IpVt:000000000000000000 000000000000000000000000Eto9 X-Hashcash: 1:20:070227:smp@freebsd.org::9BQbC1VrW17/px+a:000cyW X-Hashcash: 1:20:070227:hackers@freebsd.org::l4G+UwZc9EVfZ30P:000000000000000000 0000000000000000000000000MVl X-Hashcash: 1:20:070227:current@freebsd.org::SAkcIpT47EVw7M9K:000000000000000000 0000000000000000000000004zQf Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2007 22:13:09 -0000 On Tue, Feb 27, 2007 at 03:59:52PM -0500, Kris Kennaway wrote: > On Tue, Feb 27, 2007 at 12:25:11PM -0600, Jim C. Nasby wrote: > > On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: > > > Now that the goals of the SMPng project are complete, for the past > > > year or more several of us have been working hard on profiling FreeBSD > > > in various multiprocessor workloads, and looking for performance > > > bottlenecks to be optimized. > > > > > > We have recently made significant progress on optimizing for MySQL > > > running on an 8-core amd64 system. The graph of results may be found > > > here: > > > > I do *not* want to start a database war here, but I'm wondering if any > > testing has been done with PostgreSQL? The reason I'm asking is that > > there are some benchmarks that show MySQL falling off drastically with > > increased concurrency: > > > > http://www.mysqlperformanceblog.com/2006/11/30/interesting-mysql-and-postgresql-benchmarks/ > > > > It would be interesting to see how the changes you've made stack up > > using PostgreSQL as the benchmark. > > I've mentioned this a couple of times, but postgresql didn't scale > well [on freebsd at least] when I tried it last year. I hope to > revisit when I get time. Let me know if you need help when you get to that point. Keep in mind that PostgreSQL's out-of-the-box configuration is pretty conservative, so you won't get good numbers that way. -- Jim C. Nasby, Database Architect decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" From owner-freebsd-smp@FreeBSD.ORG Tue Feb 27 22:27:13 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E2CDE16A402 for ; Tue, 27 Feb 2007 22:27:13 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.170]) by mx1.freebsd.org (Postfix) with ESMTP id CC7BB13C494 for ; Tue, 27 Feb 2007 22:27:12 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so1149780ugh for ; Tue, 27 Feb 2007 14:27:11 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=E+RunK/tqE1QEFZpm/JQBRaACFIWg3giAO+y23QIXgxwjC+C5diiT6EHC2YKVr+zPcRoZsiwSLH3uihUy8QA43MDR9bHyjwYEtYO39hNCr45FgXKX7A1R7kNheF12GhbXxXUcy+XVbAnAOdwfxeZSdyWCBiHH3CwV672LjdN9Ao= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=lKGlP0IwPS4VHzctW64cYy3ArJHzGlWSYZ/DLGUaMX2h1rTisJd7J4eSiccec3u7EStZ+ZlNRgbLiL/j1gNtexFVEGkcvYMGE98l6xznAaBA+AU/b8lJ2dLn0dVGzWrxYCQI7ozTJEzReZ6I4OXqB0/FfyfK4YLkiLalN/cWF+M= Received: by 10.115.107.1 with SMTP id j1mr553569wam.1172613637873; Tue, 27 Feb 2007 14:00:37 -0800 (PST) Received: by 10.114.148.11 with HTTP; Tue, 27 Feb 2007 14:00:37 -0800 (PST) Message-ID: Date: Tue, 27 Feb 2007 22:00:37 +0000 From: "B. Estrade" Sender: estrabd@gmail.com To: "Kris Kennaway" In-Reply-To: <20070227205951.GA56651@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> X-Google-Sender-Auth: f8370664a7df8998 Cc: smp@freebsd.org Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Feb 2007 22:27:14 -0000 > I've mentioned this a couple of times, but postgresql didn't scale > well [on freebsd at least] when I tried it last year. I hope to > revisit when I get time. What other applications have you used to test the scalability? Have you looked into utilizing OpenMP (available via intel compilers, which are free for private use) ? http://www.intel.com/cd/software/products/asmo-na/eng/compilers/flin/282048.htm I am not sure of the state of GCC's OpenMP implementation (http://gcc.gnu.org/projects/gomp/), but that might be more palatable to some. It seems that there either exist OpenMP benchmarks for shared memory systems or ones could be easily be created for fully exercising (or maybe exorcising;) the system. One thing is for sure, there are a lot of scientific applications out there written using OpenMP that might serve this purpose as well. Cheers, Brett From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 05:40:55 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0CB3416A403 for ; Thu, 1 Mar 2007 05:40:55 +0000 (UTC) (envelope-from netslists@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.168]) by mx1.freebsd.org (Postfix) with ESMTP id 64E7313C4A3 for ; Thu, 1 Mar 2007 05:40:51 +0000 (UTC) (envelope-from netslists@gmail.com) Received: by ug-out-1314.google.com with SMTP id 71so290137ugh for ; Wed, 28 Feb 2007 21:40:51 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=Gga+nURMcnWK/F6ONvWVmfVaQpkzHzcXm4xwk3at3s2yx1nEepRnxmPoe6jijicGsFYGiyNxmVlUr//czUuHldNK21hddvcT+T5a+qS//8+zfkLnmEPwFrzUsrsFWF6tVBvFnVKeeoJvuVfVoEcUD1+OtwFJFMEHI1/aHsh9Eb8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=KqKHa4524h58bRUpEub2Y9v9GcOORW+gZge8hDLJKAudm1auMows87qvlGePYvzUrOSpqwrN06VbxPiV5w4eahJgLjml65y4dkrD5lAqwuze6PPea4dJwSO5QxKMIXs5qOcJCSsRGlT1lzWtjpi9D5m6yxzIkVwDF9sHh+5L0TU= Received: by 10.67.119.13 with SMTP id w13mr1720222ugm.1172725976059; Wed, 28 Feb 2007 21:12:56 -0800 (PST) Received: from ?192.168.11.11? ( [85.164.1.86]) by mx.google.com with ESMTP id s1sm1151795uge.2007.02.28.21.12.54; Wed, 28 Feb 2007 21:12:55 -0800 (PST) Message-ID: <45E660CE.6010600@gmail.com> Date: Thu, 01 Mar 2007 06:12:46 +0100 From: Sten Daniel Soersdal User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: "Jim C. Nasby" References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> <20070227221252.GD51916@decibel.org> In-Reply-To: <20070227221252.GD51916@decibel.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 05:40:55 -0000 Jim C. Nasby wrote: > On Tue, Feb 27, 2007 at 03:59:52PM -0500, Kris Kennaway wrote: >> On Tue, Feb 27, 2007 at 12:25:11PM -0600, Jim C. Nasby wrote: >>> On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: >>>> Now that the goals of the SMPng project are complete, for the past >>>> year or more several of us have been working hard on profiling FreeBSD >>>> in various multiprocessor workloads, and looking for performance >>>> bottlenecks to be optimized. >>>> >>>> We have recently made significant progress on optimizing for MySQL >>>> running on an 8-core amd64 system. The graph of results may be found >>>> here: >>> I do *not* want to start a database war here, but I'm wondering if any >>> testing has been done with PostgreSQL? The reason I'm asking is that >>> there are some benchmarks that show MySQL falling off drastically with >>> increased concurrency: >>> >>> http://www.mysqlperformanceblog.com/2006/11/30/interesting-mysql-and-postgresql-benchmarks/ >>> >>> It would be interesting to see how the changes you've made stack up >>> using PostgreSQL as the benchmark. >> I've mentioned this a couple of times, but postgresql didn't scale >> well [on freebsd at least] when I tried it last year. I hope to >> revisit when I get time. > > Let me know if you need help when you get to that point. Keep in mind > that PostgreSQL's out-of-the-box configuration is pretty conservative, > so you won't get good numbers that way. Just a me 2 for postgresql tests: I would be interrested in postgresql numbers too as i have servers with 2 x dual core (xeon, dell 2850ies) currently running 6.1. I'm basically looking for something like a benchmark which would justify upgrading (or even experiment with 7.x) to my boss. I am aware that it's not your job to spend your valuable time doing obscure tests for us, so consider this rant as another "vote" for postgresql performance benchmarks. -- Sten Daniel Soersdal From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 15:11:52 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A233A16A403 for ; Thu, 1 Mar 2007 15:11:52 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.226]) by mx1.freebsd.org (Postfix) with ESMTP id 5668713C474 for ; Thu, 1 Mar 2007 15:11:52 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so524726nza for ; Thu, 01 Mar 2007 07:11:51 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=i4v9a4hne+SJyHNM1boav+thsrZL8pshpHU54HMQHnxvpP/L2fOXIDVLXQaHt5baudCJeP3ZwvOOaIx1/UH/eXqxM+1evlrwGTOD8ZZYqw3kkN43taM7StUdA0cTWx90VWEeI6BjihZWdhiUz1O3Xg5Iv2PxCsgfR0ssCyjLcnU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=Drx3eBcMwu6sYJ+TAsutIH2uGCIDk9fSCSZHcr4ACgOgJi2XLVVDfvU2ju9JmzvwkXeBY1Do7L9Br95NBFFcI1jBhvcEI4WFTojnXSfgd7o86DyOcsnTPM3005gw8Me44UzT5BiPRWfCgm3rVpJ+tQMrZjAng9ROzr8Hat9o5NE= Received: by 10.65.206.7 with SMTP id i7mr3179997qbq.1172760354972; Thu, 01 Mar 2007 06:45:54 -0800 (PST) Received: by 10.64.201.3 with HTTP; Thu, 1 Mar 2007 06:45:54 -0800 (PST) Message-ID: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> Date: Thu, 1 Mar 2007 09:45:54 -0500 From: "Keith Arner" Sender: keith.arner@gmail.com To: freebsd-smp@freebsd.org MIME-Version: 1.0 X-Google-Sender-Auth: a893e1d06f40a209 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: INP_INFO_WLOCK(&tcbinfo) bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 15:11:52 -0000 I'm working on a project that needs high throughput, low latency TCP traffic. I'd like to be able to parallelize as much of the processing as possible, across several cores. I've done some early investigation in FreeBSD with direct dispatch enabled (`sysctl net.isr.direct=1`). What I've discovered is that the system tends to bottleneck with a lot of contention for INP_INFO_WLOCK(&tcbinfo). Looking at the source code, it's apparent why this is; both tcp_input() and tcp_usr_send() hold this mutex for the majority of their processing, which effectively means that TCP input and output becomes single threaded. (Though I see that this is relaxed in the output path in -CURRENT.) I see from Robert Watson's todo list (at http://wiki.freebsd.org/NetworkTodo ) that using a read/write lock over the pcb list is a work in progress. Are there any patches available that reflect whatever work has been done so far? Or even any descriptions of what has been tried, and what the hard parts are? Keith From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 15:20:04 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1B1FE16A404 for ; Thu, 1 Mar 2007 15:20:04 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.233]) by mx1.freebsd.org (Postfix) with ESMTP id CA80813C467 for ; Thu, 1 Mar 2007 15:20:03 +0000 (UTC) (envelope-from zombyfork@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so526939nza for ; Thu, 01 Mar 2007 07:20:03 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=HT5GzjRpJqLTOcKLG61s6nelMg52ONi/HKn6VKa0TtPvLxMHQORd7bKSt+n4HOJEwYLv3PY6eVediLhpOYWZ8zDzESvW1gyapjAUdMZe144LM6kYpu70HpVPOxsYWIel8yNPc80yPoV52hWIquOudd24OEaJ/70WiBc8YHT+ZsQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:references; b=A8kfAyOYLatDVMcPGAdVmlOkWmwLK3W3IX+AWbcQLGmEedXMYEFU5cEfTS52mYpTmN3Gkh//3If/IUn5//jyRRLRkjuysU2902NkGTncpWuhLHMBAYYogFXLZHu0BgyKjw7Xqdr7Tq4va8VcgAV6XEfhDraxNX6AUJFM8wxQ8XA= Received: by 10.114.157.1 with SMTP id f1mr179214wae.1172762387197; Thu, 01 Mar 2007 07:19:47 -0800 (PST) Received: by 10.115.108.15 with HTTP; Thu, 1 Mar 2007 07:19:46 -0800 (PST) Message-ID: <346a80220703010719k5fcd8d31l94b4dcef3f830a60@mail.gmail.com> Date: Thu, 1 Mar 2007 08:19:46 -0700 From: "Coleman Kane" To: "Sten Daniel Soersdal" In-Reply-To: <45E660CE.6010600@gmail.com> MIME-Version: 1.0 References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> <20070227221252.GD51916@decibel.org> <45E660CE.6010600@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Kris Kennaway , smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, "Jim C. Nasby" Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cokane@cokane.org List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 15:20:04 -0000 On 2/28/07, Sten Daniel Soersdal wrote: > > Jim C. Nasby wrote: > > On Tue, Feb 27, 2007 at 03:59:52PM -0500, Kris Kennaway wrote: > >> On Tue, Feb 27, 2007 at 12:25:11PM -0600, Jim C. Nasby wrote: > >>> On Sat, Feb 24, 2007 at 04:31:11PM -0500, Kris Kennaway wrote: > >>>> Now that the goals of the SMPng project are complete, for the past > >>>> year or more several of us have been working hard on profiling > FreeBSD > >>>> in various multiprocessor workloads, and looking for performance > >>>> bottlenecks to be optimized. > >>>> > >>>> We have recently made significant progress on optimizing for MySQL > >>>> running on an 8-core amd64 system. The graph of results may be found > >>>> here: > >>> I do *not* want to start a database war here, but I'm wondering if any > >>> testing has been done with PostgreSQL? The reason I'm asking is that > >>> there are some benchmarks that show MySQL falling off drastically with > >>> increased concurrency: > >>> > >>> > http://www.mysqlperformanceblog.com/2006/11/30/interesting-mysql-and-postgresql-benchmarks/ > >>> > >>> It would be interesting to see how the changes you've made stack up > >>> using PostgreSQL as the benchmark. > >> I've mentioned this a couple of times, but postgresql didn't scale > >> well [on freebsd at least] when I tried it last year. I hope to > >> revisit when I get time. > > > > Let me know if you need help when you get to that point. Keep in mind > > that PostgreSQL's out-of-the-box configuration is pretty conservative, > > so you won't get good numbers that way. > > Just a me 2 for postgresql tests: > > I would be interrested in postgresql numbers too as i have servers with > 2 x dual core (xeon, dell 2850ies) currently running 6.1. I'm basically > looking for something like a benchmark which would justify upgrading (or > even experiment with 7.x) to my boss. I am aware that it's not your job > to spend your valuable time doing obscure tests for us, so consider this > rant as another "vote" for postgresql performance benchmarks. > > -- > Sten Daniel Soersdal I wouldn't recommend upgrading your production servers to FreeBSD 7.x yet, no matter what the results say. -- Coleman From owner-freebsd-smp@FreeBSD.ORG Thu Mar 1 20:37:42 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 338F616A404 for ; Thu, 1 Mar 2007 20:37:41 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.244]) by mx1.freebsd.org (Postfix) with ESMTP id A37B013C4BC for ; Thu, 1 Mar 2007 20:37:41 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by an-out-0708.google.com with SMTP id c24so463960ana for ; Thu, 01 Mar 2007 12:37:41 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=irymc1AZNeOK6TiuVVjoCcaOS9UPiA5iRIgM+A8Gm8+HlOF/zeydYckpF5M0e88ezoeEKx/UOU4myffwWHcESHkpkuEj04El1CxziXDZmhMfMpR5/GJP7QCFMPEMZ9/4qBCqhgw2Xdl7EXaq7KAiKEOtfQxfFmejGeTnvpcngnk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=LKYkE6wBhdTQjvT5E3PqjCABMramA96JMprXeoy74g0XQJOTjQkg3UlHOMkto8+twTLYmdjWl6wGUvczfT8niOrNmxudkZv3pYtWyY7Gu8nvCR++Mwqx8m3++5b7702xq3Dem1e9zvl8Z6ARmw+u5nqWDTSrQq/dRRcCmLL3ERg= Received: by 10.100.57.14 with SMTP id f14mr1795254ana.1172779866030; Thu, 01 Mar 2007 12:11:06 -0800 (PST) Received: by 10.100.191.1 with HTTP; Thu, 1 Mar 2007 12:11:05 -0800 (PST) Message-ID: <3bbf2fe10703011211i6b0ad2b6gf75067ca17d0ac3b@mail.gmail.com> Date: Thu, 1 Mar 2007 21:11:05 +0100 From: "Attilio Rao" Sender: asmrookie@gmail.com To: "Keith Arner" In-Reply-To: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> X-Google-Sender-Auth: 89d8169ea68659df Cc: freebsd-smp@freebsd.org Subject: Re: INP_INFO_WLOCK(&tcbinfo) bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Mar 2007 20:37:42 -0000 2007/3/1, Keith Arner : > I'm working on a project that needs high throughput, low latency TCP > traffic. > I'd like to be able to parallelize as much of the processing as possible, > across > several cores. > > I've done some early investigation in FreeBSD with direct dispatch enabled > (`sysctl net.isr.direct=1`). What I've discovered is that the system tends > to bottleneck with a lot of contention for INP_INFO_WLOCK(&tcbinfo). > Looking at the source code, it's apparent why this is; both tcp_input() > and tcp_usr_send() hold this mutex for the majority of their processing, > which effectively means that TCP input and output becomes single threaded. > (Though I see that this is relaxed in the output path in -CURRENT.) > > I see from Robert Watson's todo list (at http://wiki.freebsd.org/NetworkTodo > ) > that using a read/write lock over the pcb list is a work in progress. Are > there > any patches available that reflect whatever work has been done so far? Or > even any descriptions of what has been tried, and what the hard parts are? > > Keith Currently, the main problem for this is that somewhere (TM) there mutex gets recursed and recursion is not handled alredy for rwlocks, so kernel starts panicing. Patch for recursion in rwlock is not trivial, but there are ongoing discussions on it. Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-smp@FreeBSD.ORG Fri Mar 2 03:03:45 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7E57716A402; Fri, 2 Mar 2007 03:03:45 +0000 (UTC) (envelope-from craig@tobuj.gank.org) Received: from ion.gank.org (ion.gank.org [69.55.238.164]) by mx1.freebsd.org (Postfix) with ESMTP id 624ED13C46B; Fri, 2 Mar 2007 03:03:45 +0000 (UTC) (envelope-from craig@tobuj.gank.org) Received: by ion.gank.org (Postfix, from userid 1001) id B843411A7F; Thu, 1 Mar 2007 20:45:35 -0600 (CST) Date: Thu, 1 Mar 2007 20:45:33 -0600 From: Craig Boston To: "Jim C. Nasby" Message-ID: <20070302024533.GA64754@nowhere> Mail-Followup-To: Craig Boston , "Jim C. Nasby" , Kris Kennaway , smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org References: <20070224213111.GB41434@xor.obsecurity.org> <20070227182511.GD29041@decibel.org> <20070227205951.GA56651@xor.obsecurity.org> <20070227221252.GD51916@decibel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070227221252.GD51916@decibel.org> User-Agent: Mutt/1.4.2.2i Cc: smp@FreeBSD.org, hackers@FreeBSD.org, current@FreeBSD.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Mar 2007 03:03:45 -0000 On Tue, Feb 27, 2007 at 04:12:52PM -0600, Jim C. Nasby wrote: > On Tue, Feb 27, 2007 at 03:59:52PM -0500, Kris Kennaway wrote: > > > > I've mentioned this a couple of times, but postgresql didn't scale > > well [on freebsd at least] when I tried it last year. I hope to > > revisit when I get time. > > Let me know if you need help when you get to that point. Keep in mind > that PostgreSQL's out-of-the-box configuration is pretty conservative, > so you won't get good numbers that way. I was kind of wondering that myself, especially as PostgreSQL uses a multi-process model rather than threads. It seems like it would benefit more from optimization of the Sys-V semaphores and shared memory. Craig From owner-freebsd-smp@FreeBSD.ORG Fri Mar 2 14:37:33 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D53CF16A403 for ; Fri, 2 Mar 2007 14:37:33 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.235]) by mx1.freebsd.org (Postfix) with ESMTP id 7FDE013C491 for ; Fri, 2 Mar 2007 14:37:33 +0000 (UTC) (envelope-from keith.arner@gmail.com) Received: by nz-out-0506.google.com with SMTP id r28so890638nza for ; Fri, 02 Mar 2007 06:37:33 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=KGZsgqSV8oMLKTDERyorZOHjJTLy5YneLPyfDKqzu1ENPNNKqZ2eqbFwglNj0mJgqmegBForbQ4JfU7irlNDv+IJ4FIgEfzRyQRaBt6pBYuujxkUP1l59ntuBSXjpx/kaMSuWTkhPUnZh1qgXZ2Hv+xaUBGBclCS1LQXkcYv3oc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=Q2mjx/oH6dAVVFn7Dj/YjXXfAub6pknkMKRiABemRz4hPiUFDSwN6SkOfJZQ2tGvwO9ESK04JfIVaz/2K5uSe2iTfV2gscsCWP+x0aQ/BzsnJZ4PrPa0vdIUGnr0uK0fmRmpCkKIIbD36fLKUvIyeOQIgTu7+PVBDqfzlNqCkOI= Received: by 10.64.10.2 with SMTP id 2mr5266109qbj.1172846249825; Fri, 02 Mar 2007 06:37:29 -0800 (PST) Received: by 10.64.201.3 with HTTP; Fri, 2 Mar 2007 06:37:29 -0800 (PST) Message-ID: <8e552a500703020637l2e1d6036h4a975736bdc48914@mail.gmail.com> Date: Fri, 2 Mar 2007 09:37:29 -0500 From: "Keith Arner" Sender: keith.arner@gmail.com To: "Attilio Rao" In-Reply-To: <3bbf2fe10703011211i6b0ad2b6gf75067ca17d0ac3b@mail.gmail.com> MIME-Version: 1.0 References: <8e552a500703010645x61d9b064w21c475ecc00a0e0e@mail.gmail.com> <3bbf2fe10703011211i6b0ad2b6gf75067ca17d0ac3b@mail.gmail.com> X-Google-Sender-Auth: 8cd9a1fad296262d Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-smp@freebsd.org Subject: Re: INP_INFO_WLOCK(&tcbinfo) bottleneck X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Mar 2007 14:37:34 -0000 On 3/1/07, Attilio Rao wrote: > > > Currently, the main problem for this is that somewhere (TM) there > mutex gets recursed and recursion is not handled alredy for rwlocks, > so kernel starts panicing. > > Patch for recursion in rwlock is not trivial, but there are ongoing > discussions on it. > Well, even if the INP_INFO_[RW]LOCK macros were changed to use shared/exclusive locks rather than a mutex, and the sx locks were fixed to be taken recursively, I'm not sure that would really do the trick for parallelism. In tcp_input(), the code is using INP_INFO_WLOCK(), to take an exclusive lock, so only one thread would be able to be in the TCP input code path anyway. Keith