Date: Fri, 2 Feb 2007 08:25:20 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: John Baldwin <jhb@freebsd.org> Cc: arch@freebsd.org, Scott Long <scottl@samsco.org>, Pawel Jakub Dawidek <pjd@freebsd.org>, Nate Lawson <nate@root.org> Subject: Re: cvs commit: src/sys/geom/eli g_eli.c Message-ID: <20070202075421.A91177@fledge.watson.org> In-Reply-To: <200701311308.16810.jhb@freebsd.org> References: <20070128202917.5B67916A5A6@hub.freebsd.org> <200701301636.38175.jhb@freebsd.org> <20070130233238.GB94650@garage.freebsd.pl> <200701311308.16810.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
(commit mailing lists dropped, just on arch now) On Wed, 31 Jan 2007, John Baldwin wrote: > I wouldn't sell yourself so short actually. :) You're probably more > competent than you think. I'm not really an expert on SMP or schedulers > either (I've not had any formal training, and if you were to review some of > my earlier SMP stuff it shows), but ended up working on it because it needed > to be done. Most of what I've learned is by just working on stuff. > Anyways, maybe this issue will bubble up to someone's todo list at BSDCan to > settle how to handle CPU's arriving and departing. One big thing that > worries me is how to handle pinned and bound threads when a CPU goes away. > Also, it may be more useful to think of CPUs not as just present/not > present, but more in terms of: > > - not present > - present but offline > - present and online > > And most code would really only care about offline/online events. We could > maybe allow pinned and bound threads to still run on an offline CPU (and the > idlethread for that CPU of course) but require that there be no pinned bound > threads to completely detach a CPU (in the case of systems with removable > CPUs). It would be useful to at least handle taking CPUs offline and then > back online though. Well, I think bound threads are the easy case, as they involve an explicit CPU-use strategy for a kernel subsystem. Any subsystem using CPU binding must register with the event handler to provide a facility for unbinding (perhaps binding to another CPU, perhaps exiting). Depending on what the thread is doing, the event handler might have to sleep waiting for the thread to migrate by itself (part of a CPU rotation that will start skipping that CPU once it's removed from the list of available CPUs), or perhaps flag, signal, or otherwise take action before waiting. We should expect shutdown of bound services to potentially take "quite a while" -- i.e., wait on a disk I/O or sx lock to become available. Not so sure about the pinned case -- we may need to evaluate what sort of things are currently covered by pinning. Are they all "short sleep" activities, such as might be safe under a mutex, in which case perhaps we simply wait for the number of threads pinned on the CPU to hit zero, having decided no longer to schedule new threads there? Or do we need a more explicit strategy? Regardless, pinning is a strategy used by subsystems that do not own the thread they are running in, so can't use sched_bind(), etc. Pinning may, for example, be used by a subsystem invoked in a thread that is already bound, since pinning is similar to a critical section. Either way, it sounds like there might well be a multiphase shutdown on the order of the following: - The CPU is marked as "going down", so that new services starting no longer get assigned to the CPU, and any further events in the shutdown can know that the CPU is going away. - Eventhandler runs on the CPU to cancel bound CPU services -- things like per-CPU crypto or worker functions, which may take some time to shut down. Each subsystem prevents the eventhandler from returning until it is shut down, so this is a synchronous shutdown process. This would include making sure that device driver interrupts stop going to the CPU. - "Critical" subsystems, such as UMA, shut down. Since memory allocation is depended on by most other subsystems, shutting down UMA on the CPU needs to happen very close to the CPU going offline, perhaps as the last stage before the scheduler goes offline. - Scheduler shutdown, which arranges to make sure no further clock interrupts are disabled, that it is the last thing running, and a panic if there is anything bound to the CPU still. And a similar, but reversed, startup, in which UMA is started, the scheduler becomes happy, interrupts are brought up, it is marked as available for use, and event handlers are run so that subsystems can start pushing load to it by creating per-CPU workers (netisr, crypto, etc). Robert N M Watson Computer Laboratory University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070202075421.A91177>