Date: Fri, 18 Nov 2011 12:20:04 -0500 From: John Baldwin <jhb@freebsd.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: Matteo Landi <matteo@matteolandi.net>, freebsd-current@freebsd.org Subject: Re: ixgbe and fast interrupts Message-ID: <201111181220.04846.jhb@freebsd.org> In-Reply-To: <20111118170615.GA9762@onelab2.iet.unipi.it> References: <CALJ8J_HPZewO12uanb=kctQYwepMssr63E0DQh9CqV6PGaC=JA@mail.gmail.com> <201111180800.06593.jhb@freebsd.org> <20111118170615.GA9762@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday, November 18, 2011 12:06:15 pm Luigi Rizzo wrote: > On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote: > > On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote: > > > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway. > > > > > > Why do you say that? Is MSI-X faster than INTx in terms of interrupt > > > latency? When should I use MSI-X, instead of fast filters interrupts > > > (fast interrupt?), instead of ithread interrupts? Thanks in advace. > > > > With MSI-X you can have more than one interrupt and those interrupts can be > > distributed across CPUs. This means you can (somewhat) tie each queue on your > > NIC to a different CPU. > > > > MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X > > interrupts are not shared, and require no interrupt masking in hardware (they > > are effectively edge-triggered), so using a filter for MSI is rather pointless > > and only adds needless complexity. For MSI I would just use a theraded > > interrupt handler. For INTx, I would only use a fast interrupt handler if > > there is a really good reason to do so (e.g. em(4) does so to work around > > broken Intel Host-PCI bridges). > > A bit more context: Matteo is looking at the latency of RPCs across > the network involving userspace processes, and possibly using the > netmap API. As we understand it: > > if you are not using a filter, the interrupt calls a "predefined" > filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes > up the handler thread which in turn wakes up the user process. This > means two scheduler invocations on each side. Yes, but if you use a filter you still have to do that as your filter would just be queueing a task to on a taskqueue which would then do the actual selwakeup() from a taskqueue thread. Filters are typically used to avoid masking the interrupt in the PIC, or to limit the handlers executed on a shared interrupt. > In the case of netmap, all the handler needs to do is a selwakeup() > of the user thread blocked on the file descriptor, so if this > can be done in the filter we can save an extra step through the > scheduler. You can't call selwakeup() from a filter, it is not safe since it uses mutexes, etc. There are only a few things you can do from a filter. You could do a plain wakeup() if you let userland use a custom ioctl to block on the filter, but not selwakeup(). -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201111181220.04846.jhb>