Date: Mon, 30 Aug 2004 12:01:24 -0600 (MDT) From: "M. Warner Losh" <imp@bsdimp.com> To: scottl@freebsd.org Cc: freebsd-arch@freebsd.org Subject: Re: splxxx level? Message-ID: <20040830.120124.28086427.imp@bsdimp.com> In-Reply-To: <4133682D.3000403@freebsd.org> References: <20040830.102606.130865377.imp@bsdimp.com> <20040830.111428.56562495.imp@bsdimp.com> <4133682D.3000403@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In message: <4133682D.3000403@freebsd.org>
Scott Long <scottl@freebsd.org> writes:
: M. Warner Losh wrote:
:
: > In message: <20040830.102606.130865377.imp@bsdimp.com>
: > "M. Warner Losh" <imp@bsdimp.com> writes:
: > : In message: <41334C3B.4070101@freebsd.org>
: > : Scott Long <scottl@freebsd.org> writes:
: > : : Sam wrote:
: > : :
: > : : > Hello -
: > : : >
: > : : > I'm almost to testing on my AoE driver for 4.x and have
: > : : > a question about interrupt priority levels.
: > : : >
: > : : > There are currently three entry points into the driver:
: > : : >
: > : : > a) strategy routine
: > : : > b) network frame reception routine
: > : : > c) timer rexmit routine
: > : : >
: > : : > Any of the three can diddle with the device structure
: > : : > and thusly I need to ensure they're not running simultaneously.
: > : : > For example, the network reception can cause a buf to be completed
: > : : > and the rexmit timer can cause a buf to be failed.
: > : : >
: > : : > So, what kind of contexts are the callout, strategy, and
: > : : > network soft interrupt called in? Which splxxx will give
: > : : > one of them exclusive access to whatever they need?
: > : : >
: > : : > Just as a reality check -- I am thinking about this correct, right?
: > : : >
: > : : > Cheers,
: > : : >
: > : : > Sam
: > : : >
: > : :
: > : : With 4.x, only one CPU can be in the kernel at a time. You won't have
: > : : to worry about multiple processes trying to get into strategy at the
: > : : same time and whatnot. However, you can be preempted by your interrupt
: > : : handler or by a timeout or by a software interrupt like the netisr. I
: > : : don't remember if your driver is for a specific piece of hardware or if
: > : : it's a generic layer that sits in between the network interface and the
: > : : block layer. If it's for dedicated hardware then you'll need to define
: > : : a interrupt type in bus_setup_intr() and use that type for the spl
: > : : (i.e. INTR_TYPE_NET translates to splnet(), INTR_TYPE_BIO translates to
: > : : splbio(), etc).
: > : :
: > : : The safe way to go is to protect all of your critical code sections with
: > : : the appropriate spl level regardless. spls are very cheap and can be
: > : : set/reset from an interrupt context so there is little penalty in using
: > : : them liberally at first and then narrowing them down later. Just make
: > : : sure that you don't leak an spl references, and don't hold an spl for so
: > : : long that it creates priority inversions. Since the only interrupts and
: > : : timeouts that you'll likely be dealing with are network related,
: > : : splnet() is probably the right one to use.
: > :
: > : splimp() is what you want to use, not splnet(). Yes, this is
: > : confusing, but it appears to be what all the other network drivers
: > : use. None of them are using splnet() that I could find. splimp() is
: > : also used by the mbuf routines to protect mbuf operations.
: > :
: > : splnet() is a list of the software interrupts that are network
: > : drivers.
: > :
: > : splimp() is splnet() plus the hardware interrupts, so is more
: > : appropriate to block things called from the driver. Especially one
: > : that's described as having timeouts. If it is a network driver, you
: > : might consider using the timeout functionality in the net stack as
: > : opposed to the callout functions. This makes it possible to have
: > : almost the entire driver w/o doing any spls (most of the network
: > : drivers in 4 don't do spl at all, except for entry points that are
: > : outside the scope of the network/interrupt entry points, eg
: > : device_suspend).
: >
: > Ah, just saw 'AoE' in the reply.
: >
: > This likely means that you are writing a network stack that's glued
: > into the system with the strategy routine. Inside the strategy
: > routine, splbio() is likely what you need to hold while dealing with
: > the struct buf that's passed into that routine. I'm not sure what
: > spl level you need to be at to call into the network code. I'm
: > thinking it is splimp(), but I'm not 100% sure about that. Stevens
: > will likely be a good resource for 4.x.
: >
: > You may have to define a software interrupt to pass the packets to the
: > network code to make the spls work out correctly.
: >
: > Warner
:
: No, splbio() is not explicitely needed in this case. You'll be
: dealing with a bioq that might need protection, but you have total
: control over that. There is nothing else in the block layer that
: will interrupt you, not like the netisr in the network stack or the
: camisr in the CAM layer.
Why does da use splbio in its strategy routine then:
static void
dastrategy(struct buf *bp)
{
...
/*
* Mask interrupts so that the pack cannot be invalidated until
* after we are in the queue. Otherwise, we might not properly
* clean up one of the buffers.
*/
s = splbio();
...(error cases)
/*
* Place it in the queue of disk activities for this disk
*/
bufqdisksort(&softc->buf_queue, bp);
splx(s);
/*
* Schedule ourselves for performing the work.
*/
xpt_schedule(periph, /* XXX priority */1);
return;
... (error cases)
}
Is that due to the ISR routine from the cam layer? Or is the
softc->buf_queue the 'bioq' that you were talking about?
Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040830.120124.28086427.imp>