From owner-freebsd-arch@FreeBSD.ORG Sun Dec 23 06:33:04 2007 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D7B516A417; Sun, 23 Dec 2007 06:33:04 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id E43D513C447; Sun, 23 Dec 2007 06:33:03 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id lBN6Ww4a017018; Sun, 23 Dec 2007 01:32:59 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Sat, 22 Dec 2007 20:34:16 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Andre Oppermann In-Reply-To: <476A4DCC.4040206@freebsd.org> Message-ID: <20071222203120.A899@desktop> References: <20071219211025.T899@desktop> <476A4DCC.4040206@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: Linux compatible setaffinity. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2007 06:33:04 -0000 On Thu, 20 Dec 2007, Andre Oppermann wrote: > Jeff Roberson wrote: >> I have implemented a linux compatible sched_setaffinity() call which is >> somewhat crippled. This allows a userspace process to supply a bitmask of >> processors which it will run on. I have copied the linux interface such >> that it should be api compatible because I believe it is a sensible >> interface and they beat us to it by 3 years. > > The Linux (and Solaris) style setaffinity is rather low level and > any user of it has to make many assumptions based on incomplete > knowledge of the underlying hardware and its architecture (buses, > caches, latency between cores, etc). > > In practical use I'd rather have a function to bind myself to the > current CPU or CPU number X, and then to specify that new threads > or forked processes should emerge on another, but not this CPU. > Pepper that with a few hints like latency and cache affinity (important > or not important) the kernel can act on appropriately and it becomes > much more powerful and simpler to use. Taking it even further an > application may want to specify that it would like to run on a number > X of cores that are close (latency/cache) together, be permanently > bound to it and to repel any other such requests. This way I can > run my database server on socket 1 cores 1-4, and the webserver on > socket 2 cores 5-8 more or less automagically. sched_setaffinity > requires a lot of operator involvement and architecture knowledge > to make that happen. > > Not that I'm against a Linux compatible sched_setaffinity(), it's > just not as practical to use as other constructs. > > Food for thought. Well my hope is that the kernel scheduler has all of the required information about the processor to make these kinds of decisions for the general case. Right now we need better topology information in the kernel, but I think userspace only uses setaffinity in very special cases. I'd hate for it to become the norm in applications to start looking at cpu topology and making decisions based on that. Not that I would argue if someone were to implement this. I just want us to get it right often enough in the scheduler that it's not necessary. The uses for setaffinity that I have seen so far have been very special purpose. Or quite often just spawning one thread per cpu and pinning it in place for various purposes. Jeff > > -- > Andre >