From owner-freebsd-smp@FreeBSD.ORG Tue Feb 20 17:10:06 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 88C1C16BA94 for ; Tue, 20 Feb 2007 17:10:06 +0000 (UTC) (envelope-from dbrewer@pixelfish.com) Received: from mail01.vmatrixmail.com (mail01.vmatrixmail.com [216.219.244.230]) by mx1.freebsd.org (Postfix) with ESMTP id 475BF13C4B4 for ; Tue, 20 Feb 2007 17:10:06 +0000 (UTC) (envelope-from dbrewer@pixelfish.com) Received: (vmatrix@mail01.vmatrixmail.com) by vmatrixmail.com id S6071110AbXBTQkL for ; Tue, 20 Feb 2007 08:40:11 -0800 To: freebsd-smp@freebsd.org MIME-Version: 1.0 X-Mailer: Rich Media Mail V4. Vmatrix, (C) 2003 From: "David Brewer" Sender: "David Brewer" Message-Id: Date: Tue, 20 Feb 2007 08:40:11 -0800 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Your Recent Trade Show Results X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dbrewer@pixelfish.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2007 17:10:06 -0000 Greetings!

Are you happy with your results from Social Linux Expo 2007? Did you have what it takes to make the difference between creating excitement and blending in with the competition, between lots of hot leads and a few hard sells ... between success and failure?

Did you have video?

Why video? Video vividly demonstrates the features and benefits of your products. Video captures the praises of your most enthusiastic customers. Video instills interest, reaction and trust. Video sells.

In just 45 days, PixelFish creates marketing videos that become an integral part of your marketing and sales efforts when it streams from your Web site, launches from multimedia email newsletters, plays from CD video brochures and loops from a DVD at your tradeshow booth.

We are PixelFish. We deliver “The Evolution of Video”. And we guarantee results. Click on the videos to the right to see samples of our work.

Contact us today for a free evaluation of your video marketing needs.

David Brewer
PixelFish, Inc.
800.503.3020 x102
dbrewer@pixelfish.com
http://www.pixelfish.com
RedMan Video
Pentax Video
JEM Video
------------------------------------------------ Unsubscribe to safely remove yourself from this email list, please send email to info@pixelfish.com. Powered by Pixelfish http://www.pixelfish.com From owner-freebsd-smp@FreeBSD.ORG Tue Feb 20 17:15:35 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE61D16B117 for ; Tue, 20 Feb 2007 17:15:35 +0000 (UTC) (envelope-from dbrewer@pixelfish.com) Received: from mail01.vmatrixmail.com (mail01.vmatrixmail.com [216.219.244.230]) by mx1.freebsd.org (Postfix) with ESMTP id 3525513C48E for ; Tue, 20 Feb 2007 17:15:35 +0000 (UTC) (envelope-from dbrewer@pixelfish.com) Received: (vmatrix@mail01.vmatrixmail.com) by vmatrixmail.com id S6071031AbXBTQkH for ; Tue, 20 Feb 2007 08:40:07 -0800 To: smp@FreeBSD.org MIME-Version: 1.0 X-Mailer: Rich Media Mail V4. Vmatrix, (C) 2003 From: "David Brewer" Sender: "David Brewer" Message-Id: Date: Tue, 20 Feb 2007 08:40:07 -0800 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Your Recent Trade Show Results X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dbrewer@pixelfish.com List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2007 17:15:35 -0000 Greetings!

Are you happy with your results from Social Linux Expo 2007? Did you have what it takes to make the difference between creating excitement and blending in with the competition, between lots of hot leads and a few hard sells ... between success and failure?

Did you have video?

Why video? Video vividly demonstrates the features and benefits of your products. Video captures the praises of your most enthusiastic customers. Video instills interest, reaction and trust. Video sells.

In just 45 days, PixelFish creates marketing videos that become an integral part of your marketing and sales efforts when it streams from your Web site, launches from multimedia email newsletters, plays from CD video brochures and loops from a DVD at your tradeshow booth.

We are PixelFish. We deliver “The Evolution of Video”. And we guarantee results. Click on the videos to the right to see samples of our work.

Contact us today for a free evaluation of your video marketing needs.

David Brewer
PixelFish, Inc.
800.503.3020 x102
dbrewer@pixelfish.com
http://www.pixelfish.com
RedMan Video
Pentax Video
JEM Video
------------------------------------------------ Unsubscribe to safely remove yourself from this email list, please send email to info@pixelfish.com. Powered by Pixelfish http://www.pixelfish.com From owner-freebsd-smp@FreeBSD.ORG Thu Feb 22 02:01:25 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E08D116A402; Thu, 22 Feb 2007 02:01:25 +0000 (UTC) (envelope-from aaron.kunze@intel.com) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx1.freebsd.org (Postfix) with ESMTP id C440D13C481; Thu, 22 Feb 2007 02:01:25 +0000 (UTC) (envelope-from aaron.kunze@intel.com) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by mga01.intel.com with ESMTP; 21 Feb 2007 17:32:50 -0800 Received: from orsmsx335.jf.intel.com ([10.22.226.40]) by fmsmga001.fm.intel.com with ESMTP; 21 Feb 2007 17:32:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: i="4.14,203,1170662400"; d="scan'208"; a="202690950:sNHT21998389" Received: from orsmsx416.amr.corp.intel.com ([10.22.226.46]) by orsmsx335.jf.intel.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 21 Feb 2007 17:32:49 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 21 Feb 2007 17:32:49 -0800 Message-ID: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Re: Setting CPU affinity to process( Freebsd smp kernel) thread-index: AcdWIVzDXVZyVVzJRTKTq9pCuzWVVA== From: "Kunze, Aaron" To: , X-OriginalArrivalTime: 22 Feb 2007 01:32:49.0270 (UTC) FILETIME=[5CB62960:01C75621] Cc: atmblr@gmail.com Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2007 02:01:26 -0000 Hello! Does anyone know if this will change any time soon? For example, is anyone working on exposing affinity to user-space applications via extensions of the pthreads interface? =20 Sorry to reply to such an old thread... Aaron --------------- On Monday 20 November 2006 06:24, Ashok TM wrote: > Hello, >=20 > Is there a way to set a process's CPU affinity with freebsd smp kernel. > By doing so a process is bound to and will only run on the processor (where > the affinity was set ) even though the other processors are free. (similar > to sched_setaffinity available on Linux kernel) . >=20 > Does freebsd smp scheduler supports setting affinity? Yes, but currently only in the kernel. --=20 John Baldwin -------------------------- Aaron Kunze aaron.kunze@intel.com --PGP Key ID: 0x81124B7C-- --NOT SPEAKING FOR INTEL-- -------GO BOILERS!!------- From owner-freebsd-smp@FreeBSD.ORG Thu Feb 22 23:00:46 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 17F0816A402 for ; Thu, 22 Feb 2007 23:00:46 +0000 (UTC) (envelope-from sean@digital-jungle.com) Received: from smtp.digital-jungle.com (mail-server.digital-jungle.com [69.90.192.252]) by mx1.freebsd.org (Postfix) with ESMTP id EED0613C4AA for ; Thu, 22 Feb 2007 23:00:45 +0000 (UTC) (envelope-from sean@digital-jungle.com) Received: from smtp.digital-jungle.com (localhost.digital-jungle.com [127.0.0.1]) by smtp.digital-jungle.com (Postfix) with ESMTP id 0F9FB15ED77; Thu, 22 Feb 2007 22:45:41 +0000 (UTC) Received: from [192.168.1.125] (S0106000f66b26ddf.vc.shawcable.net [24.82.170.103]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.digital-jungle.com (Postfix) with ESMTP id CFD3C15ED75; Thu, 22 Feb 2007 22:45:40 +0000 (UTC) Mime-Version: 1.0 (Apple Message framework v752.2) To: freebsd-smp@freebsd.org Message-Id: <3B5A38D6-46E2-4370-B1F9-9455FD792FBF@digital-jungle.com> From: Sean Benoit Date: Thu, 22 Feb 2007 14:45:10 -0800 X-Mailer: Apple Mail (2.752.2) X-Virus-Scanned: ClamAV using ClamSMTP Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Remko Lodder Subject: Kernel Update To 6.2 Stable Causes Dell PowerEdge 1750 (SMP) To Freeze On Boot X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2007 23:00:46 -0000 Hello, Last fall, while updating a 6.1 Stable kernel to 6.2 Pre-Release, I noted that the then new kernel caused all our previously functioning Dell PowerEdge 1750s (dual processor) to freeze on boot at the point in the boot process where the second processor was being detected. Disabling the second processor in the BIOS served as a temporary workaround and enabled the systems to boot successfully. At that time I submitted a PR via the website and a patch was issued within a few hours. The patch resolved the problem. I didn't keep that original correspondence because I assumed that all was well moving forward. Last weekend I attempted to update the same systems for the first time since last fall from the patched 6.2 Pre-Release kernel to the new 6.2 Stable kernel. I was alarmed to see that the EXACT same issue had reappeared. The symptoms are exactly the same - as is the workaround (disable the second processor in the BIOS). I have once again submitted a PR (109267) via the website (details at http://www.freebsd.org/cgi/query-pr.cgi?pr=109267); however, I have been advised that I should post the issue to this mailing list instead/prior. Not sure what to do from here... Thanks in advance for any help, Sean Benoit Digital Jungle Systems Inc. http://www.digital-jungle.com 604-639-7882 / 1-877-639-7882 From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 15:42:28 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 360F216A400 for ; Fri, 23 Feb 2007 15:42:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0F07D13C467 for ; Fri, 23 Feb 2007 15:42:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 197B04759D; Fri, 23 Feb 2007 10:13:53 -0500 (EST) Date: Fri, 23 Feb 2007 15:13:52 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Kunze, Aaron" In-Reply-To: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> Message-ID: <20070223151158.Q88189@fledge.watson.org> References: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: atmblr@gmail.com, freebsd-smp@freebsd.org Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 15:42:28 -0000 On Wed, 21 Feb 2007, Kunze, Aaron wrote: > Does anyone know if this will change any time soon? For example, is anyone > working on exposing affinity to user-space applications via extensions of > the pthreads interface? > > Sorry to reply to such an old thread... I know of no work along these lines currently, but it's something a lot of people would like to see happen. There's a potential for conflict between the kernel's use of pinning and binding for kernel synchronization and the user space affinity model, which will be entirely avoided if done right. :-) For now, it's quite easy to add a sysctl/syscall that allows user space to send the kernel scheduler's notion of thread binding, but this isn't really the right approach. As I understand it, some systems support setting CPU affinity for a thread as a set of CPUs it is willing to run on ? Robert N M Watson Computer Laboratory University of Cambridge > > Aaron > > --------------- > > On Monday 20 November 2006 06:24, Ashok TM wrote: >> Hello, >> >> Is there a way to set a process's CPU affinity with freebsd smp > kernel. >> By doing so a process is bound to and will only run on the processor > (where >> the affinity was set ) even though the other processors are free. > (similar >> to sched_setaffinity available on Linux kernel) . >> >> Does freebsd smp scheduler supports setting affinity? > > Yes, but currently only in the kernel. > > -- > John Baldwin > > > -------------------------- > Aaron Kunze > aaron.kunze@intel.com > --PGP Key ID: 0x81124B7C-- > --NOT SPEAKING FOR INTEL-- > -------GO BOILERS!!------- > > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" > From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 15:42:28 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB33916A401 for ; Fri, 23 Feb 2007 15:42:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id BB79013C471 for ; Fri, 23 Feb 2007 15:42:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id C719B478AD; Fri, 23 Feb 2007 10:19:25 -0500 (EST) Date: Fri, 23 Feb 2007 15:19:25 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Sean Benoit In-Reply-To: <3B5A38D6-46E2-4370-B1F9-9455FD792FBF@digital-jungle.com> Message-ID: <20070223151628.B88189@fledge.watson.org> References: <3B5A38D6-46E2-4370-B1F9-9455FD792FBF@digital-jungle.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Remko Lodder , freebsd-smp@freebsd.org Subject: Re: Kernel Update To 6.2 Stable Causes Dell PowerEdge 1750 (SMP) To Freeze On Boot X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 15:42:29 -0000 On Thu, 22 Feb 2007, Sean Benoit wrote: > Last fall, while updating a 6.1 Stable kernel to 6.2 Pre-Release, I noted > that the then new kernel caused all our previously functioning Dell > PowerEdge 1750s (dual processor) to freeze on boot at the point in the boot > process where the second processor was being detected. Disabling the second > processor in the BIOS served as a temporary workaround and enabled the > systems to boot successfully. > > At that time I submitted a PR via the website and a patch was issued within > a few hours. The patch resolved the problem. I didn't keep that original > correspondence because I assumed that all was well moving forward. > > Last weekend I attempted to update the same systems for the first time since > last fall from the patched 6.2 Pre-Release kernel to the new 6.2 Stable > kernel. I was alarmed to see that the EXACT same issue had reappeared. The > symptoms are exactly the same - as is the workaround (disable the second > processor in the BIOS). > > I have once again submitted a PR (109267) via the website (details at > http://www.freebsd.org/cgi/query-pr.cgi?pr=109267); however, I have been > advised that I should post the issue to this mailing list instead/prior. > > Not sure what to do from here... Does this relate to PR 103697 which seems to be a similar correspondence: http://www.freebsd.org/cgi/query-pr.cgi?pr=103697 Are you using the ULE scheduler? Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 16:36:18 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1E38A16A403 for ; Fri, 23 Feb 2007 16:36:18 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id A6BC413C474 for ; Fri, 23 Feb 2007 16:36:17 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.14.0/8.14.0/NETPLEX) with ESMTP id l1NG9PT0006588; Fri, 23 Feb 2007 11:09:25 -0500 (EST) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-3.0 (mail.ntplx.net [204.213.176.10]); Fri, 23 Feb 2007 11:09:25 -0500 (EST) Date: Fri, 23 Feb 2007 11:09:25 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Robert Watson In-Reply-To: <20070223151158.Q88189@fledge.watson.org> Message-ID: References: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> <20070223151158.Q88189@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-smp@freebsd.org, atmblr@gmail.com, "Kunze, Aaron" Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 16:36:18 -0000 On Fri, 23 Feb 2007, Robert Watson wrote: > On Wed, 21 Feb 2007, Kunze, Aaron wrote: > >> Does anyone know if this will change any time soon? For example, is anyone >> working on exposing affinity to user-space applications via extensions of >> the pthreads interface? >> >> Sorry to reply to such an old thread... > > I know of no work along these lines currently, but it's something a lot of > people would like to see happen. There's a potential for conflict between > the kernel's use of pinning and binding for kernel synchronization and the > user space affinity model, which will be entirely avoided if done right. :-) > For now, it's quite easy to add a sysctl/syscall that allows user space to > send the kernel scheduler's notion of thread binding, but this isn't really > the right approach. As I understand it, some systems support setting CPU > affinity for a thread as a set of CPUs it is willing to run on ? I know Solaris has processor_bind(2) and pset_bind(2): http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=expand#P -- DE From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 16:36:21 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D7DC316A403 for ; Fri, 23 Feb 2007 16:36:21 +0000 (UTC) (envelope-from aaron.kunze@intel.com) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx1.freebsd.org (Postfix) with ESMTP id A959C13C478 for ; Fri, 23 Feb 2007 16:36:21 +0000 (UTC) (envelope-from aaron.kunze@intel.com) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by mga01.intel.com with ESMTP; 23 Feb 2007 08:36:21 -0800 Received: from orsmsx334.jf.intel.com ([10.22.226.45]) by fmsmga001.fm.intel.com with ESMTP; 23 Feb 2007 08:36:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: i="4.14,212,1170662400"; d="scan'208"; a="203364185:sNHT23588166" Received: from orsmsx416.amr.corp.intel.com ([10.22.226.46]) by orsmsx334.jf.intel.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 23 Feb 2007 08:36:18 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Fri, 23 Feb 2007 08:35:58 -0800 Message-ID: <07DDDFCFB8BE0A43BCA52E743373DBDC03102190@orsmsx416.amr.corp.intel.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Setting CPU affinity to process( Freebsd smp kernel) thread-index: AcdXZQQl/6foEZcqTH+NbboIQZM3dwAAt75w References: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> <20070223151158.Q88189@fledge.watson.org> From: "Kunze, Aaron" To: "Daniel Eischen" , "Robert Watson" X-OriginalArrivalTime: 23 Feb 2007 16:36:18.0226 (UTC) FILETIME=[BE2E2920:01C75768] Cc: atmblr@gmail.com, freebsd-smp@freebsd.org Subject: RE: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 16:36:21 -0000 Thanks for the info. The Linux equivalent would be sched_setaffinity which takes a bitmask as input, allowing the user to define which processors will run a particular thread. Here's a link: http://ibm5.ma.utexas.edu/cgi-bin/man-cgi?sched_setaffinity+2 > > There's a potential for=20 > > conflict between the kernel's use of pinning and binding for kernel=20 > > synchronization and the user space affinity model, which will be=20 Can you elaborate on this? Some of my colleagues and I are considering tackling this and would like to avoid such pitfalls, if possible. Aaron -------------------------- Aaron Kunze Advanced Visual Computing aaron.kunze@intel.com --PGP Key ID: 0x81124B7C-- --NOT SPEAKING FOR INTEL-- -------GO BOILERS!!------- =20 > -----Original Message----- > From: Daniel Eischen [mailto:deischen@freebsd.org]=20 > Sent: Friday, February 23, 2007 8:09 AM > To: Robert Watson > Cc: Kunze, Aaron; atmblr@gmail.com; freebsd-smp@freebsd.org > Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) >=20 > On Fri, 23 Feb 2007, Robert Watson wrote: >=20 > > On Wed, 21 Feb 2007, Kunze, Aaron wrote: > > > >> Does anyone know if this will change any time soon? For=20 > example, is=20 > >> anyone working on exposing affinity to user-space applications via=20 > >> extensions of the pthreads interface? > >>=20 > >> Sorry to reply to such an old thread... > > > > I know of no work along these lines currently, but it's something a=20 > > lot of people would like to see happen. There's a potential for=20 > > conflict between the kernel's use of pinning and binding for kernel=20 > > synchronization and the user space affinity model, which will be=20 > > entirely avoided if done right. :-) For now, it's quite=20 > easy to add a=20 > > sysctl/syscall that allows user space to send the kernel=20 > scheduler's=20 > > notion of thread binding, but this isn't really the right=20 > approach. =20 > > As I understand it, some systems support setting CPU=20 > affinity for a thread as a set of CPUs it is willing to run on ? >=20 > I know Solaris has processor_bind(2) and pset_bind(2): >=20 > http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=3Dexpand#P >=20 > -- > DE >=20 From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 19:12:32 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D756116A401 for ; Fri, 23 Feb 2007 19:12:32 +0000 (UTC) (envelope-from sean@digital-jungle.com) Received: from smtp.digital-jungle.com (mail-server.digital-jungle.com [69.90.192.252]) by mx1.freebsd.org (Postfix) with ESMTP id B866C13C4AA for ; Fri, 23 Feb 2007 19:12:32 +0000 (UTC) (envelope-from sean@digital-jungle.com) Received: from smtp.digital-jungle.com (localhost.digital-jungle.com [127.0.0.1]) by smtp.digital-jungle.com (Postfix) with ESMTP id 9CA1515EDD0; Fri, 23 Feb 2007 19:10:08 +0000 (UTC) Received: from [192.168.1.125] (S0106000f66b26ddf.vc.shawcable.net [24.82.170.103]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.digital-jungle.com (Postfix) with ESMTP id EE64815EDD7; Fri, 23 Feb 2007 19:05:05 +0000 (UTC) In-Reply-To: <20070223151628.B88189@fledge.watson.org> References: <3B5A38D6-46E2-4370-B1F9-9455FD792FBF@digital-jungle.com> <20070223151628.B88189@fledge.watson.org> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Sean Benoit Date: Fri, 23 Feb 2007 11:05:04 -0800 To: Robert Watson X-Mailer: Apple Mail (2.752.2) X-Virus-Scanned: ClamAV using ClamSMTP Cc: Remko Lodder , freebsd-smp@freebsd.org Subject: Re: Kernel Update To 6.2 Stable Causes Dell PowerEdge 1750 (SMP) To Freeze On Boot X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 19:12:32 -0000 Hello Robert, First, thanks for the quick response and yes, PR 103697 was in fact the initial PR generated by me last fall referred to in my original message below. The current issue is identical with the same symptoms. I have confirmed that our kernels were compiled with the ULE scheduler enabled. Your response in addition to some Googling and a quick look in the GENERIC kernconf and I can see that the 4BSD scheduler should be enabled by default. I have recompiled and installed the kernel using the 4BSD scheduler and this enables the systems to boot correctly with both processors enabled in the BIOS. Thanks again, Sean On 23-Feb-07, at 7:19 AM, Robert Watson wrote: > > On Thu, 22 Feb 2007, Sean Benoit wrote: > >> Last fall, while updating a 6.1 Stable kernel to 6.2 Pre-Release, >> I noted that the then new kernel caused all our previously >> functioning Dell PowerEdge 1750s (dual processor) to freeze on >> boot at the point in the boot process where the second processor >> was being detected. Disabling the second processor in the BIOS >> served as a temporary workaround and enabled the systems to boot >> successfully. >> >> At that time I submitted a PR via the website and a patch was >> issued within a few hours. The patch resolved the problem. I >> didn't keep that original correspondence because I assumed that >> all was well moving forward. >> >> Last weekend I attempted to update the same systems for the first >> time since last fall from the patched 6.2 Pre-Release kernel to >> the new 6.2 Stable kernel. I was alarmed to see that the EXACT >> same issue had reappeared. The symptoms are exactly the same - as >> is the workaround (disable the second processor in the BIOS). >> >> I have once again submitted a PR (109267) via the website (details >> at http://www.freebsd.org/cgi/query-pr.cgi?pr=109267); however, I >> have been advised that I should post the issue to this mailing >> list instead/prior. >> >> Not sure what to do from here... > > Does this relate to PR 103697 which seems to be a similar > correspondence: > > http://www.freebsd.org/cgi/query-pr.cgi?pr=103697 > > Are you using the ULE scheduler? > > Robert N M Watson > Computer Laboratory > University of Cambridge From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 20:34:34 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 365B316A405 for ; Fri, 23 Feb 2007 20:34:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outI.internet-mail-service.net (outI.internet-mail-service.net [216.240.47.232]) by mx1.freebsd.org (Postfix) with ESMTP id 0B52F13C4A3 for ; Fri, 23 Feb 2007 20:34:34 +0000 (UTC) (envelope-from julian@elischer.org) Received: from mx0.idiom.com (HELO idiom.com) (216.240.32.160) by out.internet-mail-service.net (qpsmtpd/0.32) with ESMTP; Fri, 23 Feb 2007 11:56:40 -0800 Received: from [10.251.22.38] (nat.ironport.com [63.251.108.100]) by idiom.com (Postfix) with ESMTP id DEC89125B04; Fri, 23 Feb 2007 12:21:14 -0800 (PST) Message-ID: <45DF4CBA.1010906@elischer.org> Date: Fri, 23 Feb 2007 12:21:14 -0800 From: Julian Elischer User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: "Kunze, Aaron" References: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> <20070223151158.Q88189@fledge.watson.org> <07DDDFCFB8BE0A43BCA52E743373DBDC03102190@orsmsx416.amr.corp.intel.com> In-Reply-To: <07DDDFCFB8BE0A43BCA52E743373DBDC03102190@orsmsx416.amr.corp.intel.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Daniel Eischen , freebsd-smp@freebsd.org, atmblr@gmail.com, Robert Watson Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 20:34:34 -0000 Kunze, Aaron wrote: > Thanks for the info. The Linux equivalent would be sched_setaffinity > which takes a bitmask as input, allowing the user to define which > processors will run a particular thread. Here's a link: > > http://ibm5.ma.utexas.edu/cgi-bin/man-cgi?sched_setaffinity+2 > >>> There's a potential for >>> conflict between the kernel's use of pinning and binding for kernel >>> synchronization and the user space affinity model, which will be > > Can you elaborate on this? Some of my colleagues and I are considering > tackling this and would like to avoid such pitfalls, if possible. [...] >> >> I know Solaris has processor_bind(2) and pset_bind(2): >> >> http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=expand#P I prefer the linux model but it does limit you to some set number of procesors. it looks however that the solaris interface doesn't allow 'sets' of processors but just allows you to specify a single processor. I would postulate the following syscall copied from Linux. As in Linux, the 'pid_t' can specify a TID in which case the give thread is changed I would suggest that it would have to be a thread in the current process however. (you don't allow threads to be manipulated from outside the process) It's not obvious from the man page if a pid of 0 afects the current process or thread. I read it as being process, with no way of specifying the current thread. I would add a PID/TID of (-1) to indicate 'current (kernel) thread'. In addition it is possible that a UTS mailbox might specify a set I guess for kse type threads. int sched_getaffinity(pid_t pid, unsigned int cpusetsize, cpu_set_t *mask); void CPU_CLR(int cpu, cpu_set_t *set); int CPU_ISSET(int cpu, cpu_set_t *set); void CPU_SET(int cpu, cpu_set_t *set); void CPU_ZERO(cpu_set_t *set); >> >> -- >> DE >> > _______________________________________________ > freebsd-smp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-smp > To unsubscribe, send any mail to "freebsd-smp-unsubscribe@freebsd.org" From owner-freebsd-smp@FreeBSD.ORG Fri Feb 23 21:18:36 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7758316A407; Fri, 23 Feb 2007 21:18:36 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.ntplx.net (mail.ntplx.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id 355EC13C4B7; Fri, 23 Feb 2007 21:18:36 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.ntplx.net (8.14.0/8.14.0/NETPLEX) with ESMTP id l1NLIXIn008700; Fri, 23 Feb 2007 16:18:33 -0500 (EST) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.ntplx.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-3.0 (mail.ntplx.net [204.213.176.10]); Fri, 23 Feb 2007 16:18:33 -0500 (EST) Date: Fri, 23 Feb 2007 16:18:33 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Julian Elischer In-Reply-To: <45DF4CBA.1010906@elischer.org> Message-ID: References: <07DDDFCFB8BE0A43BCA52E743373DBDC030C5D5A@orsmsx416.amr.corp.intel.com> <20070223151158.Q88189@fledge.watson.org> <07DDDFCFB8BE0A43BCA52E743373DBDC03102190@orsmsx416.amr.corp.intel.com> <45DF4CBA.1010906@elischer.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-smp@freebsd.org, Robert Watson , atmblr@gmail.com, "Kunze, Aaron" Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Feb 2007 21:18:36 -0000 On Fri, 23 Feb 2007, Julian Elischer wrote: > Kunze, Aaron wrote: >> Thanks for the info. The Linux equivalent would be sched_setaffinity >> which takes a bitmask as input, allowing the user to define which >> processors will run a particular thread. Here's a link: >> >> http://ibm5.ma.utexas.edu/cgi-bin/man-cgi?sched_setaffinity+2 >> >>>> There's a potential for conflict between the kernel's use of pinning and >>>> binding for kernel synchronization and the user space affinity model, >>>> which will be >> >> Can you elaborate on this? Some of my colleagues and I are considering >> tackling this and would like to avoid such pitfalls, if possible. > > [...] > >>> >>> I know Solaris has processor_bind(2) and pset_bind(2): >>> >>> http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=expand#P > > I prefer the linux model but it does limit you to some set number of > procesors. > it looks however that the solaris interface doesn't allow 'sets' of > processors but > just allows you to specify a single processor. Why do you say that? Solaris has pset_bind(2) as well as other pset_foo(). -- DE From owner-freebsd-smp@FreeBSD.ORG Sat Feb 24 03:04:20 2007 Return-Path: X-Original-To: freebsd-smp@freebsd.org Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D3BA16A4C0; Sat, 24 Feb 2007 03:04:20 +0000 (UTC) (envelope-from tlambert2@mindspring.com) Received: from elasmtp-spurfowl.atl.sa.earthlink.net (elasmtp-spurfowl.atl.sa.earthlink.net [209.86.89.66]) by mx1.freebsd.org (Postfix) with ESMTP id 0E47E13C442; Sat, 24 Feb 2007 03:04:20 +0000 (UTC) (envelope-from tlambert2@mindspring.com) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dk20050327; d=mindspring.com; b=dYlL9iLH+5ZMcWoAWgOz61ikRDBwNM9ixfLG+jFkMm1IKZ46f0bm4zZ/1l7y8ztI; h=Message-ID:Date:From:Reply-To:To:Subject:Cc:Mime-Version:Content-Type:Content-Transfer-Encoding:X-Mailer:X-ELNK-Trace:X-Originating-IP; Received: from [209.86.224.24] (helo=mswamui-andean.atl.sa.earthlink.net) by elasmtp-spurfowl.atl.sa.earthlink.net with asmtp (Exim 4.34) id 1HKn0h-0008Pa-CB; Fri, 23 Feb 2007 21:51:43 -0500 Received: from 17.202.40.209 by webmail.atl.earthlink.net with HTTP; Fri, 23 Feb 2007 21:51:43 -0500 Message-ID: <18488133.1172285503375.JavaMail.root@mswamui-andean.atl.sa.earthlink.net> Date: Fri, 23 Feb 2007 18:51:43 -0800 (GMT-08:00) From: Terry Lambert To: Julian Elischer , "Kunze, Aaron" Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Mailer: EarthLink Zoo Mail 1.0 X-ELNK-Trace: b1a02af9316fbb217a47c185c03b154d40683398e744b8a41afe7d5046e86c325f246f2d090d8730667c3043c0873f7e350badd9bab72f9c350badd9bab72f9c X-Originating-IP: 209.86.224.24 Cc: Daniel Eischen , Robert Watson , atmblr@gmail.com, freebsd-smp@freebsd.org Subject: Re: Setting CPU affinity to process( Freebsd smp kernel) X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Terry Lambert List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2007 03:04:20 -0000 From: Julian Elischer >Kunze, Aaron wrote: >> Thanks for the info. The Linux equivalent would be sched_setaffinity >> which takes a bitmask as input, allowing the user to define which >> processors will run a particular thread. Here's a link: >> >> http://ibm5.ma.utexas.edu/cgi-bin/man-cgi?sched_setaffinity+2 >> >>>> There's a potential for >>>> conflict between the kernel's use of pinning and binding for kernel >>>> synchronization and the user space affinity model, which will be >> >> Can you elaborate on this? Some of my colleagues and I are considering >> tackling this and would like to avoid such pitfalls, if possible. > >[...] > >>> >>> I know Solaris has processor_bind(2) and pset_bind(2): >>> >>> http://docs.sun.com/app/docs/doc/816-5167/6mbb2jaeu?a=expand#P > >I prefer the linux model but it does limit you to some set number of procesors. >it looks however that the solaris interface doesn't allow 'sets' of processors but >just allows you to specify a single processor. There's a lot bad to be said about the Linux interface, though it limits you to a certain number of processors is not one of them. It has the same limitations on number of bits as FreeBSD's select(2) call, which is "none". One big weakness of the Linux model is that it requires the application to pick the affinity (to get use out of it, you effectively have to have a helper parent that is knowledgable, and sets things up for a child process). The good thing about the Solaris interface is that it's more or less starting to become industry standard between vendors making machines with large numbers of processors; HP also supports it. It's likely to make standards track at some point in the future, unless someone comes up with something better, and gets one or more other people to implement it. The documentation on the Solaris site is pretty bad, in that it doesn't tell you where baby processor sets come from, initial state, or other issues that you'd need to know in order to judge it fairly; the HP documentation is much better, and the psrset(1M) manula page is most informative; here's a link: The most useful information is that you can iterate the processsors in a set, there's always a processor set 0, and processor 0 always live in processor set 0, and at least one processor in processor set zero has to be willing to take I/O interrupts and be available for scheduling LWPs (KSEs). You can read the rest at the link above for more details, but it should be clear that you can build any set of CPUs and move them from set to set using this interface. This is basically the old Mach "Processor Set" abstraction, revivified. The major weakness of the Solaris/HP approach is that it does not permit any migration of threads between processor sets (there is no hierarchy - Linux has the same problem with its API). If you consider a modern system with SMT ("hyprethreading"), multiple cores on a die with shared L2, multiple dies in a package, and a shared L3 or memory controller in the package, it becomes very obvious that any scalable approach with a future needs to be able to have a hierarchy, let you control migration between members in a set of sets, and let you control both negaffinity and affinity in threads in a process (preferrably, both in the same process). -- Terry From owner-freebsd-smp@FreeBSD.ORG Sat Feb 24 21:31:13 2007 Return-Path: X-Original-To: smp@FreeBSD.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0768E16A400; Sat, 24 Feb 2007 21:31:13 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id EBB7B13C441; Sat, 24 Feb 2007 21:31:12 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id CF1911A4D8E; Sat, 24 Feb 2007 13:31:12 -0800 (PST) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 1B57151426; Sat, 24 Feb 2007 16:31:12 -0500 (EST) Date: Sat, 24 Feb 2007 16:31:11 -0500 From: Kris Kennaway To: current@FreeBSD.org, smp@FreeBSD.org, hackers@FreeBSD.org Message-ID: <20070224213111.GB41434@xor.obsecurity.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UugvWAfsgieZRqgk" Content-Disposition: inline User-Agent: Mutt/1.4.2.2i Cc: Subject: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2007 21:31:13 -0000 --UugvWAfsgieZRqgk Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Now that the goals of the SMPng project are complete, for the past year or more several of us have been working hard on profiling FreeBSD in various multiprocessor workloads, and looking for performance bottlenecks to be optimized. We have recently made significant progress on optimizing for MySQL running on an 8-core amd64 system. The graph of results may be found here: http://www.freebsd.org/~kris/scaling/scaling.png This shows the graph of MySQL transactions/second performed by a multi-threaded client workload against a local MySQL database with varying numbers of client threads, with identically configured FreeBSD and Linux systems on the same machine. The test was run on FreeBSD 7.0, with the latest version of the ULE 2.0 scheduler, the libthr threading library, and an uncommitted patch from Jeff Roberson [1] that addresses poor scalability of file descriptor locking (using a new sleepable mutex primitive); this patch is responsible for almost all of the performance and scaling improvements measured. It also includes some other patches (collected in my kris-contention p4 branch) that have been shown to help contention in MySQL workloads in the past (including a UNIX domain socket locking pushdown patch from Robert Watson), but these were shown to only give small individual contributions, with a cumulative effect on the order of 5-10%. With this configuration we are able to achieve performance that is consistent with Linux at peak (the graph shows Linux 2% faster, but this is commensurate with the margin of error coming from variance between runs, so more data is needed to distinguish them), with 8 client threads (=1 thread/CPU core), and significantly outperforms Linux at higher than peak loads, when running on the same hardware. Specifically, beyond 8 client threads FreeBSD has only minor performance degradation (an 8% drop from peak throughput at 8 clients to 20 clients), but Linux collapses immediately above 8 threads, and above 14 threads asymptotes to essentially single-threaded levels. At 20 clients FreeBSD outperforms Linux by a factor of 4. We see this result as part of the payoff we are seeing from the hard work of many developers over the past 7 years. In particular it is a significant validation of the SMP and locking strategies chosen for the FreeBSD kernel in the post-FreeBSD 4.x world. More configuration details and discussion about the benchmark may be found here: http://people.freebsd.org/~kris/scaling/mysql.html Kris --UugvWAfsgieZRqgk Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFF4K6eWry0BWjoQKURAmflAKCkuhNrJS6JziErZLMLeFMxUy3EjACgzuze N42WM6TxJ2XJzxrNiocKMgE= =ymdU -----END PGP SIGNATURE----- --UugvWAfsgieZRqgk-- From owner-freebsd-smp@FreeBSD.ORG Sat Feb 24 23:13:58 2007 Return-Path: X-Original-To: smp@freebsd.org Delivered-To: freebsd-smp@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AA89316A404; Sat, 24 Feb 2007 23:13:58 +0000 (UTC) (envelope-from matt@ixsystems.com) Received: from mail.iXsystems.com (newknight.ixsystems.net [206.40.55.70]) by mx1.freebsd.org (Postfix) with ESMTP id 7DB8413C467; Sat, 24 Feb 2007 23:13:56 +0000 (UTC) (envelope-from matt@ixsystems.com) Received: from localhost (localhost [127.0.0.1]) by mail.iXsystems.com (Postfix) with ESMTP id EA3DFBE3B; Sat, 24 Feb 2007 14:44:58 -0800 (PST) Received: from mail.iXsystems.com ([127.0.0.1]) by localhost (mail.ixsystems.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 38327-02; Sat, 24 Feb 2007 14:44:56 -0800 (PST) Received: from [192.168.0.100] (c-24-6-31-249.hsd1.ca.comcast.net [24.6.31.249]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTP id 05C01BD4B; Sat, 24 Feb 2007 14:44:56 -0800 (PST) From: Matt Olander Organization: iXsystems To: freebsd-hackers@freebsd.org Date: Sat, 24 Feb 2007 14:44:54 -0800 User-Agent: KMail/1.9.5 References: <20070224213111.GB41434@xor.obsecurity.org> In-Reply-To: <20070224213111.GB41434@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200702241444.54960.matt@ixsystems.com> X-Virus-Scanned: Maia Mailguard Cc: smp@freebsd.org, hackers@freebsd.org, current@freebsd.org, Kris Kennaway Subject: Re: Progress on scaling of FreeBSD on 8 CPU systems X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Feb 2007 23:13:58 -0000 On Saturday 24 February 2007 1:31 pm, Kris Kennaway wrote: > Now that the goals of the SMPng project are complete, for the past > year or more several of us have been working hard on profiling > FreeBSD in various multiprocessor workloads, and looking for > performance bottlenecks to be optimized. > > We have recently made significant progress on optimizing for MySQL > running on an 8-core amd64 system. The graph of results may be found > here: > > http://www.freebsd.org/~kris/scaling/scaling.png > > This shows the graph of MySQL transactions/second performed by a > multi-threaded client workload against a local MySQL database with > varying numbers of client threads, with identically configured > FreeBSD and Linux systems on the same machine. > > The test was run on FreeBSD 7.0, with the latest version of the ULE > 2.0 scheduler, the libthr threading library, and an uncommitted patch > from Jeff Roberson [1] that addresses poor scalability of file > descriptor locking (using a new sleepable mutex primitive); this > patch is responsible for almost all of the performance and scaling > improvements measured. It also includes some other patches > (collected in my kris-contention p4 branch) that have been shown to > help contention in MySQL workloads in the past (including a UNIX > domain socket locking pushdown patch from Robert Watson), but these > were shown to only give small individual contributions, with a > cumulative effect on the order of 5-10%. > > With this configuration we are able to achieve performance that is > consistent with Linux at peak (the graph shows Linux 2% faster, but > this is commensurate with the margin of error coming from variance > between runs, so more data is needed to distinguish them), with 8 > client threads (=1 thread/CPU core), and significantly outperforms > Linux at higher than peak loads, when running on the same hardware. > > Specifically, beyond 8 client threads FreeBSD has only minor > performance degradation (an 8% drop from peak throughput at 8 clients > to 20 clients), but Linux collapses immediately above 8 threads, and > above 14 threads asymptotes to essentially single-threaded levels. > At 20 clients FreeBSD outperforms Linux by a factor of 4. > > We see this result as part of the payoff we are seeing from the hard > work of many developers over the past 7 years. In particular it is a > significant validation of the SMP and locking strategies chosen for > the FreeBSD kernel in the post-FreeBSD 4.x world. > > More configuration details and discussion about the benchmark may be > found here: > > http://people.freebsd.org/~kris/scaling/mysql.html Well done! Nice work guys!!! FYI, we're working with Intel to get an engineering sample of the quad core, quad CPU system for some further SMP testing. Kris, I'll let you know as soon as I have word when we're getting one and if we can keep it, we'll send it up to be included in the cluster in Canada. Yay!!!!!!! This is just fabulous. -matt > > Kris -- Matt Olander CTO, iXsystems - "Servers for Open Source" http://www.iXsystems.com Public Relations, The FreeBSD Project http://www.FreeBSD.org BSD on the Desktop! http://www.pcbsd.org Phone: (408)943-4100 ext. 113 Fax: (408)943-4101