Date: Tue, 27 Jan 2009 20:09:21 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Julian Elischer <julian@elischer.org> Cc: arch@freebsd.org, Poul-Henning Kamp <phk@phk.freebsd.dk> Subject: Re: need for another mutex type/flag? Message-ID: <alpine.BSF.2.00.0901272001310.51605@fledge.watson.org> In-Reply-To: <497F51D6.1000903@elischer.org> References: <23211.1232871523@critter.freebsd.dk> <497C249C.5060507@elischer.org> <alpine.BSF.2.00.0901271509290.5592@fledge.watson.org> <497F51D6.1000903@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 27 Jan 2009, Julian Elischer wrote:
> Robert Watson wrote:
>> On Sun, 25 Jan 2009, Julian Elischer wrote:
>>
>>> Even purely documentary would be good but given the option, I'd like it to
>>> scream when Witness is enabled and you try get another mutex....
>>>
>>> there are certain contexts (e.g. in most netgraph nodes) where we really
>>> want the authors to only take such locks and taking more unpredictable
>>> mutexes plays havoc with netgraph response times as a system as a whole,
>>> since if one node blocks, teh thread doesn't get to go off and service
>>> other nodes.
>>
>> I thought lots of Netgraph nodes liked to do things like call m_pullup()
>> and m_prepend()? Disallowing acquiring mutexes will prevent them from
>> doing that.
>
> I think I mentioned that in another mail. The problem I see is that some
> module writers are tempted to just use a mutex in their node without
> thinking about what the result will be. My understanding is that the mbuf
> allocation code has been tuned to within an inch of its life to try keep
> it's waits to a minimum. I am worried about it, but I am more worried about
> a netgraph node waiting on something that is depending on some piece of
> hardware such as what I think HPS had in his suggested patch for the
> bluetooth code..
Right, but what I'm saying is: if we have a MTX_LEAFNODE flag for mtx_init(9),
it won't work for any code that holds the lock over a call to the mbuf
routines. I am happy with us adding a MTX_LEAFNODE flag and would use it
myself, I just not sure it will work for Netgraph node mutexes.
The case you're talking about is generally problematic for mutexes anyway --
in principle we divide the world of sleeping into two categories: sleeps of
strictly "bounded" length that generall correspond to the sleeps associated
with mutex or rwlock acquire, and sleeps of potentially "unbounded" length,
such as those associated with msleep(9), cv_wait(9), sx(9), lockmgr(9), etc.
If you try to perform an unbounded sleep while holding a mutex, then WITNESS
will already complain about sleeping while holding a lock.
The tricky case is the "may sleep" case, in which case we already have a
WITNESS call you can do to say this code may sleep, even though it doesn't in
the common case, so warn if this is called with a mutex held so that we can
catch that happening". For example, mb_reclaim does this so that any attempt
to call it with a mutex held will throw up a red flag:
/*
* This is the protocol drain routine.
*
* No locks should be held when this is called. The drain routines have to
* presently acquire some locks which raises the possibility of lock order
* reversal.
*/
static void
mb_reclaim(void *junk)
{
struct domain *dp;
struct protosw *pr;
WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK | WARN_PANIC, NULL,
"mb_reclaim()");
for (dp = domains; dp != NULL; dp = dp->dom_next)
for (pr = dp->dom_protosw; pr < dp->dom_protoswNPROTOSW; pr++)
if (pr->pr_drain != NULL)
(*pr->pr_drain)();
}
If there is a common or unconditional call to msleep(9) in code called by
Netgraph with a mutex held, however, then it should already be displaying a
warning when that happens. If you want to simulate this effect without a lock
necessarily be held, you can do what the softclock() code does:
THREAD_NO_SLEEPING();
c_func(c_arg);
THREAD_SLEEPING_OK();
This causes WITNESS to complain loudly if the callout handler attempts to
perform an unbounded sleep (i.e., msleep(), cv_wait(), etc). Splitting the
world into bounded an unbounded sleeps is quite useful, and is entirely about
the sort of deadlock/cascading delay scenario you have in mind: code inside a
mutex-protected block is essentially a critical section, and should never wait
a long time, especially given that ithreads may acquire mutexes leading to
potential deadlocks if msleep() is effectively waiting on an interrupt that
will never be delivered -- if we allow ithreads to wait on msleep(), which we
don't because of the above mechanisms.
Robert N M Watson
Computer Laboratory
University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.0901272001310.51605>