Date: Wed, 28 May 2008 05:25:04 -0700 (PDT) From: Alexander Popov <aopopov@yahoo.com> To: Benjamin Close <Benjamin.Close@clearchain.com> Cc: freebsd-drivers@freebsd.org Subject: Re: Synchronization in drivers (after SMP improvements) Message-ID: <600706.33129.qm@web51403.mail.re2.yahoo.com>
next in thread | raw e-mail | index | archive | help
Hi, Benjamin, also Ilya, Thanks for your quick responses. mtx_sleep() would definitely help. Does it mean that user process executing in kernel space is guaranteed not to be preempted between mtx_lock() and mtx_sleep()? Because then I would get a sleeping thread with non-sleepable mutex anyway... or have I been in application development for too long? :-) Regards, Alexander. ----- Original Message ---- From: Benjamin Close <Benjamin.Close@clearchain.com> To: Alexander Popov <aopopov@yahoo.com> Cc: freebsd-drivers@freebsd.org Sent: Wednesday, May 28, 2008 1:09:07 AM Subject: Re: Synchronization in drivers (after SMP improvements) Alexander Popov wrote: > Hello all, > > I'm currently writing a FreeBSD USB driver and have a few questions about synchronization between bottom and top half of the driver. Before posting here, I have already done quite extensive research, starting from "The Design and Implementation of the 4.4 BSD Operating System" by McKusick et al, to reading about FreeBSD kernel improvements that have been done in the scope of SMP project, i.e. http://www.lemis.com/~grog/SMPng/, and so on, but I am still puzzled about how the synchronization shall be done. > > Previously, i.e. in 4.4 BSD, the following statements were true: > > 1. When interrupt handler was executing in the bottom half of the driver, no processes could run in kernel mode; > 2. To synchronize top half of the driver code (i.e. to maintain integrity across interrupt calls) a set of splxxx functions were used to manipulate interrupt priorities (i.e. disabling certain interrupts in critical sections of the driver). > > This is the strategy that is followed by most of the (USB) drivers currently. However, after the SMP improvements have been implemented, the splxxx family of functions has been successfully stubbed out, so now these functions do absolutely nothing, which (correct me if am wrong) leaves most of the existing USB drivers without any synchronization with their bottom halves whatsoever. > > Next, in the scope of SMP project, interrupt handlers have been replaced with (lightweight?) threads, which makes it possible to use synchronization primitives in the interrupt handlers to synchronize them with driver's top half. However, this does not apply to USB stack, which, as I understand, still uses Giant lock. So, from this perspective, I can not make interrupts in my USB driver MPSAFE. > > It has been suggested everywhere that mutexes shall be used to protect top and bottom halves of the driver. So I've tried. The following is the pseudo-code that I've used: > int driver_read(...) > { > mtx_lock(&sc->mtx); > > if (no data is available) => tsleep(..) > > mtx_unlock(&sc->mtx); > } > > First it worked very well, but under relatively heavy load I started getting kernel panic, all the time related to one error: a thread holding a non-sleepable lock. Which means that user process has been suspended somewhere during execution in the read() function (probably awaiting data, but not necessary) and its thread has been holding mutex that I've used for synchronization, but that it apparently not allowed (look in man page on mutex). > > Then I found that in SMP project an additional flag was added to mutex - MTX_SPEEPABLE, which I guess would allow me to have sleeping threads in kernel that hold mutexes. But, apparently, this flag has been recently removed from kernel... > > The bottom line is: > 1. I can not make interrupt handlers MPSAFE (USB framework has not been redesigned yet to support this); > 2. I can not synchronize bottom half with top half of the driver using splxxx functions - these are deprecated and stubbed; > 3. I can not use standard mutexes because user processes can get preempted while holding the mutex, which is not allowed. > > Question is: what is now a synchronization model for modern kernels, i.e. FreeBSD 7.0? How shall I properly implement synchronization in my driver? > > Of course, I might be missing something here, please help. > > Thanks in advance, > > Hi Alexander, As you've noticed, sleeping using tlseep whilst holding a mutex can cause you grief due to preemption. This is why mtx_sleep was introduced. It drops the lock before the sleep and reaquires it before continuing. Hence you can then be preempted without issues. Under 7 and above, man 9 locking is a very useful page. In general drivers tend to use mutexes or in some cases rwlocks. Be aware though, you should not sleep in an interrupt handler. Cheers, Benjamin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?600706.33129.qm>