Date: Thu, 29 Jun 2000 14:52:59 -0600 (MDT) From: Kevin Van Maren <vanmaren@fast.cs.utah.edu> To: cp@bsdi.com, dillon@apollo.backplane.com Cc: dg@root.com, freebsd-smp@FreeBSD.ORG Subject: Re: Stepping on Toes Message-ID: <200006292052.OAA14920@fast.cs.utah.edu>
next in thread | raw e-mail | index | archive | help
> :} Even with interrupt threads we have the GiantMutex issue... the same > :} issue that we have with our current MP implementation. We cannot remove > :} SPL's until we remove the GiantMutex, and we cannot remove GiantMutex > :} without major modifications to just about every single source file in sys/ > :In general this isn't true. If you get to the point where > : > :1) All entrance to unsafe code is proteced by Giant. > :2) Tsleep and friend if any release Giant when they > : a process is suspended and re-acquire it on exit > :3) Interrupts have a context to run in. > :4) You have one or more scheduling locks. > : > :Then you can just turn spls into a nop. There is lots of hand waving > :in regards to details at this point. BSD/OS SMPng is the existance > :proof. > :It seems like one of the major problems in retaining spls during > :the change over period is that they don't much useful, and effectively > :push everything under Giant. > : > : Grabbing a spl will only block interrupts, it will not give > : any protection against an interrupt thread which has already > : started. > Right this moment the requirement is that only someone holding Giant > is allowed to mess with spl*()'s (the cpl variable can only be messed > with by people holding Giant). > > At this moment, without interrupt threads, interrupts can share Giant > with the curproc they interrupted. This is how our existing MP stuff > worked already. > > When Greg moves interrupts to their own threads, and obtains Giant to > run those interrupts, no more sharing will occur and just the fact > that the interrupt is holding Giant guarentees that nobody else will > be messing with SPLs, thus the SPLs can be removed entirely. > > -Matt Hi all, I'm really glad to see FreeBSD/SMP taking off again. It will be a great day when FreeBSD scales well on the Unisys ES7000 (which is what I work on for my day job these days). Forget those puny 8X servers... I do run FreeBSD on my dual P-II, so I'm excited about what will be in FreeBSD 5.0 Anyway, for my MS Thesis, I ran Linux/BSD device dirvers on top of a preemptive, multi-threaded kernel. The approach I took is similar to what is proposed here, so I thought it might be of general interest. Basically, I reduced the synchronization problem down to two mutexes: a process lock (giant), and an interrupt lock. Interrupts executed in their own threads, and acquired and released the interrupt lock. Process-level requests acquired the process lock (giant), and disabled interrupts (spl disables/enables all interrupts in this simple mapping) by acquiring the interrupt lock; this either caused the interrupt thread to block, or the process thread blocked until the interrupt handler completed. To preserve the atomicity of interrupts (which was assumed in *many* places, at least in Linux), I required the interrupt lock to be held to acquire or release "giant", and had the interrupt thread "stop" the process-level thread. sleep is handled as mentioned above, by releasing/acquiring the process lock. The solution here uses only a single lock for interrupts and process- level activity. This was not a solution for me because in Linux drivers would spin (on jiffies...) at process level until the interrupt handler executed. The biggest problem with my approach was that if the interrupt thread had to stop a process-level thread, that was pretty expensive. Also, a process-level thread could not be started until the interrupt handler completed -- I should have used a separate lock to protect it for the period of time the interrupt handler needed the info to be "valid". Removing the assumtion that interrupts are processed atomically is difficult (since the code doesn't state that it makes the assumption) and may be the cause of some hairy race conditions, but is critical for better performance (eliminating the extra synchronization to prevent process and interrupt threads from executing in parallel). A copy of my MS thesis may be downloaded from: http://www.cs.utah.edu/flux/papers/vanmaren-thesis-base.html I also have a bit of advice: cache lines are more important under SMP than on a uni-processor. Highly-contended spin-locks should be in their own cache line, but less-contended locks should be in the same cache line as the data they protect. It can be expensive to shuffle dirty cache lines around, especially in larger systems (8X up through the ES7000) where the processor buss is not shared by all the CPUs. Watching the data layout in structures would be a good idea, but can be tuned after everything "works".. Kevin Van Maren To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200006292052.OAA14920>