Date: Sat, 31 May 2008 20:57:22 +0300 From: Ilya Bobir <ilya@po4ta.com> To: Alexander Popov <aopopov@yahoo.com> Cc: freebsd-drivers@FreeBSD.org Subject: Re: Synchronization in drivers (after SMP improvements) Message-ID: <48419182.1030200@po4ta.com> In-Reply-To: <987362.36986.qm@web51404.mail.re2.yahoo.com> References: <987362.36986.qm@web51404.mail.re2.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alexander Popov wrote:
> Hi, Warner,
>
> Thanks, I appreciate your help.
> Yes, indeed I am writing a USB device driver. I have done some tests,
> and results are interesting. First of all, when I am trying to use default mutex in top-haft of the driver, under heavy load it panics anyway, even though there is no tsleep or alike functions used while holding the mutex. Basically what I do is this:
>
> void driver_open()
> {
> mtx_lock(&sc->mtx)
> // open USB pipes to device, etc.
> mtx_unlock(&sc->mtx)
> }
>
> void driver_write(...)
> {
> mtx_lock(&sc->mtx)
> // write to device (I do not sleep here!)
> mtx_unlock(&sc->mtx)
> }
>
> then, one process opens device, and starts repeatedly write to it, constantly acquiring and leaving mutex; then, I start another process, that simply tries to open the same device => if it happens to be while first process is holding the mutex => kernel panics with message "sleeping thread with non-sleepable lock".
The only spot in the source were I was able to find a message like this
is in src/sys/kern/subr_turnstile.c:propagate_priority(struct thread
*td)
(http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/subr_turnstile.c?rev=1.173;content-type=text%2Fplain
search for "non-sleepable"). IIUC, this message means that a thread
that is sleeping on a lock also holds a spin lock. Another thread
(current thread) is trying to acquire this spin lock but finds out that
the current owner of the spin lock is sleeping. As sleeping while
holding a spin lock is prohibited as it may cause a deadlock the
function panics. Note that the misbehaving thread's id is present in
the message. Chapter 11 of the "FreeBSD Developers' Handbook" explains
how you can get a thread stack trace from a kernel dump or a running
kernel. But you probably already know it.
> When I change default mutex to spin type, it works well, and I was not able to crash it. As Ilya pointed out, locking(9) says that Giant lock must be acquired before any other locks, which might be the reason why my code crashes with default mutex - USB drivers are Giant locked.
> It is interesting, because I've seen USB drivers in the default FreeBSD distribution that use default mutexes...
>
I think you would get a lock order reversal message or some other kind
of message if you are locking Giant after some other lock. "Sleeping
thread (tid %d, pid %d) owns a non-sleepable lock\n" is not about Giant.
> However, when I try to lock spin mutex from interrupt handler, everything blocks and after a while I get panic with a message that "spin mutex has been locked for too long". Could not find explanation for this one yet...
> [...]
It probably means that the mutex is already locked. You can drop into a
debugger and try do debug it as described in "11.9 Debugging Deadlocks"
in the "FreeBSD Developers' Handbook".
P.S. Note, that sleeping and preemption are different. A thread is
sleeping when it is waiting on a sleepable lock. While waiting on a
non-sleepable lock or in some other cases a thread may be preempted but
it is not marked as sleeping.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48419182.1030200>