From owner-freebsd-arch@FreeBSD.ORG Wed Feb 20 11:10:02 2008 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0E8E16A408; Wed, 20 Feb 2008 11:10:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 7FCAE13C4F0; Wed, 20 Feb 2008 11:10:02 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id B8D8546BC2; Wed, 20 Feb 2008 06:10:01 -0500 (EST) Date: Wed, 20 Feb 2008 11:10:01 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Jeff Roberson In-Reply-To: <20080220005030.Y920@desktop> Message-ID: <20080220105333.G44565@fledge.watson.org> References: <20071219211025.T899@desktop> <18311.49715.457070.397815@grasshopper.cs.duke.edu> <20080112182948.F36731@fledge.watson.org> <20080112170831.A957@desktop> <20080112194521.I957@desktop> <20080219234101.D920@desktop> <20080220101348.D44565@fledge.watson.org> <20080220005030.Y920@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Daniel Eischen , arch@FreeBSD.org, Andrew Gallatin Subject: Re: Linux compatible setaffinity. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Feb 2008 11:10:03 -0000 On Wed, 20 Feb 2008, Jeff Roberson wrote: >> So perhaps this means a slightly more complex API, but not much more >> complex. How about: >> >> int cpuaffinity_get(scope, id, length, mask) >> int cpuaffinity_getmax(scope, id, length, mask) >> int cpuaffinity_set(scope, id, length, mask) >> int cpuaffinity_setmax(scope, id, length, mask) >> >> Scope would be something on the order of process (representing individual >> processes or process groups, potentially), id would be the id in that scope >> namespace, length and mask would be as you propose. You could imagine >> adding a further field to indicate whether it's the current affinity or the >> maximum affinity, but I'm not sure the details matter all that much. Here >> might be some application logic, though: > > Well I'm not sure about the max. How about just a cpuaffinity_get with a > scope that specifies what cpus are available to you? If the set is > restricted by a jail or some other mechanism it would be returned in avail. > Otherwise all cpus would be returned. The thread probably wouldn't directly > mainpulate its max, rather it would be set by changing the jail or cpu group > it belonged to. I think the actual details don't matter too much as long as we can express what we need to, so I'm ok with a special scope for that. You do raise an interesting point, though, on the nature of scope: presumably we'd like to be able to query a few different "maximums" in the interest of having a maximally debuggable system -- be it the hardware limit, the administrative limit, etc. So perhaps the scopes are something more like: #define CPUAFF_SCOPE_HARDWARE 1 /* Hardware limits */ #define CPUAFF_SCOPE_SYSTEM 2 /* System usage mask */ #define CPUAFF_SCOPE_PROCESS 3 /* Processes and process groups */ #define CPUAFF_SCOPE_THREAD 4 /* Threads in process */ CPUAFF_SCOPE_HARDWARE would be what the kernel has probed at the physical layer. This would be get-only, and available simply to provide a consistent interface when dealing with CPU masks. CPUAFF_SCOPE_SYSTEM would be what are exposed by the kernel for use by the application. This would be get-only without privilege, and presumably not exceed CPUAFF_SCOPE_HARDWARE. This would be the practical upper bound on what a process could use, and would likely be how we implement Jail scoping of CPUs. This might or might not be how one tries to discourage userspace use of a particular CPU. CPUAFF_SCOPE_PROCESS would be the process affinity for the process as a whole. It would be get-set without privilege, but limited to CPUAFF_SCOPE_SYSTEM. I'm not sure we want that limitation to be something overridden with privilege, but I guess we can think about that. CPUAFF_SCOPE_THREAD would be the thread affinity for an individual thread. It would be get-set without privilege. Do we limit it to CPUAFF_SCOPE_SYSTEM or CPUAFF_SCOPE_PROCESS? Since this involves some access control now, a pondering on access control: When a process sets the affinity of another thread or process, is it limited to the system scope of the current process, or of the target process? This may be a practical question if we're talking about how to deal with a process outside of a jail setting the affinity of a process inside jail. It's easy to implement the right thing in the kernel, but does that then imply that the system scope query should take a process ID so that a user tool can figure out what valid choices are? :-) I think it's useful to play out a few of these scenarios a bit and see what works; we won't be locked into any particular model until it hits a -STABLE branch, but getting the underlying primitive right will make it a lot easier to implement some of the more mature services we have in mind, making them more likely to happen. Robert N M Watson Computer Laboratory University of Cambridge