Date: Sat, 17 Mar 2007 12:26:36 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Aniruddha Bohra <bohra@nec-labs.com> Cc: freebsd-net@FreeBSD.org Subject: Re: ether_input question Message-ID: <20070317121920.D7579@fledge.watson.org> In-Reply-To: <45FAA00F.4070208@nec-labs.com> References: <45FA98DD.3080205@cs.rutgers.edu> <20070316141836.J60288@fledge.watson.org> <45FAA00F.4070208@nec-labs.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 16 Mar 2007, Aniruddha Bohra wrote: > Robert Watson wrote: >> >> I can't speak to the details of the above, but can speak generally on the >> issue of the link layer input path and locking. There is no assumption >> that the caller will provide serialization, and fully concurrent input from >> multiple threads is supported. The reason the drivers drop their locks is >> that the network stack frequently holds locks over calls to driver output >> routines. As a result, driver locks tend to follow network stack locks in >> the lock order--at least, for drivers that have a single lock covering both >> send and receive paths (quite common). As a result, the driver must drop >> the driver lock before calling into the stack to avoid a lock order >> reversal. > > So, if I have a queue shared between ether_input() and another thread, I > need to ensure mutual exclusion. In such scenarios, should spinlocks or > default mutexes be used? I'm not sure I completely understand the scenario you are suggesting -- could you be specific? Normally, from the perspective of a device driver author, you simply drop your driver-specific locks and call ifp->if_input() to hand the mbuf chain up to the link layer. No locks need to be held around the call to the input routine. On the other hand, if you are modifying the link layer itself (i.e., hooking into it in one of a number of ways), this means that you must provide any synchronization necessary to make your code operate correctly in the presence of concurrency. Many instances of ether_input() may run at the same time on various CPUs -- typically one per input source, since they run in the ithread of the device driver. In general, you should use default mutexes in preference to spin mutexes unless you know your code will run the fast interrupt path (rather than the ithread path). Universally, the network stack assumes it will not run in the fast interrupt path, so unless you're doing something quite low level and special involving a device driver, default mutexes (or rwlocks if you need shared locking) are the right way to go. > The driver locks themselves are usually MTX_DEF, whereas in netgraph for > example, (the scenario above), a spinlock is used. Is there a rule of thumb, > for example, never use blocking locks in the network interrupt path? I am not sure why Netgraph uses spin locks -- it probably shouldn't be doing so. In the kernel we have several different notions of "sleeping", and unfortunately the terminology is not entirely clear. The best way I've found to explain it is using the term "bounded". Sleeping associated with mutexes and rwlocks is bounded sleeping, whereas sleeping associated with condition variables, wait channels, and sx locks is unbounded sleeping. This distinction is important because you don't want, for example, an interrupt thread performing an unbounded sleep waiting on something that may not happen for a very long (unbounded) period of time, such as waiting for keyboard input or disk I/O to return. If you run with INVARIANTS and WITNESS, a debugging kernel should warn you if you try to acquire the wrong type of lock in the wrong context. A locking(9) man page talking about some of the selection choices was recently added to 7-CURRENT, FYI. Robert N M Watson Computer Laboratory University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070317121920.D7579>