Date: Tue, 28 May 2013 16:05:05 +0200 From: Andre Oppermann <andre@freebsd.org> To: Orit Moskovich <oritm@mellanox.com> Cc: Konstantin Belousov <kostikbel@gmail.com>, "freebsd-hackers@freebsd.org" <freebsd-hackers@FreeBSD.org>, Andriy Gapon <avg@FreeBSD.org> Subject: Re: preemptive kernel Message-ID: <51A4B991.3070805@freebsd.org> In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> <51A34EEA.9050609@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 27.05.2013 14:29, Orit Moskovich wrote:
>>From what I've read in subr_taskqueue.c taskqueue_swi, taskqueue_swi_giant and taskqueue_fast are all implemented using swi_add which calls ithread_create().
> Is there any performance difference between them. Is one of the above or ithread given to bus_setup_intr preferable on the other?
It depends on what you intend to do. If it is a network packet
based work load then you want to avoid locking in the driver RX
path. With MSI-X and a dedicated RX vector (per queue if you have
more than one, RSS) you can do a lock-free hybrid interrupt/polling
and life-lock avoiding RX model.
The RX handler thereby is split into two parts: 1) the interrupt
acknowledger quieting the interrupt until re-enabled (many modern
cards also support fire-once interrupts and you can omit this step)
and schedule the RX ithread/task; 2) run the dedicated (one per RX
queue) ithread/task and pull the packets out of the RX ring in a
while(1) loop with a maybe_yield() every so many packets to check
your quantum and send them up the stack. Re-enable the interrupt
once no more packets are available. When packets contiguously come
in you never get an interrupt and always stay in the RX loop. The
maybe_yield() lets other processes run if you've used a full quantum
and returns when your next quantum is available.
To completely omit the locking on the RX path you must make sure to
prevent the driver from going away while the RX ithread/task is
running. Also you have to make sure that nothing else is touching
the RX ring and other directly related data structures. To stop
the driver the ithread/task and the interrupt handler have to be
torn down and drained. When that function returns both are gone
and you're safe.
Pseudo-code:
error = bus_setup_intr(dev, sc->drv_irq, INTR_TYPE_NET | INTR_MPSAFE,
dev_intr_handler, dev_intr_task, sc, &sc->drv_intrcookie);
int
drv_intr_handler(void *arg) {
struct drv_softc *sc = arg;
uint32_t status;
status = CSR_READ_4(sc, DRV_INTR_STAUS); /* not necessary on MSI-X */
if (!status)
return (FILTER_STRAY); /* Was not for us, try next handler on shared IRQ */
CSR_WRITE_4(sc, DRV_INTR_SUPPRESS); /* only if not fire-once */
return (FILTER_SCHEDULE_THREAD); /* Run ithread/task */
}
void
drv_intr_task(void *arg) { /* look mom, no locks! */
struct drv_softc *sc = arg;
bus_dmamap_sync(...);
while (rx_pkts) {
bus_dmamap_sync(...);
bus_dmamap_unload(...);
...
(*ifp->if_input)(ifp, m);
maybe_yield();
if (i++ > 32) {
drv_rx_refill(sc, ...); /* Amortize over a number of RX packets */
i = 0;
}
}
CSR_WRITE_4(sc, DRV_INTR_ENABLE);
}
bus_teardown_intr(dev, sc->drv_irq, &sc->drv_intrcookie);
I'm soon starting work to revamp the lower stack / driver interface,
to document the expected behavior of both sides and the best common
practices. Also the proliferation of code-duplication for certain
advanced features will be tackled. Both the RX and the TX side are
part of this project.
--
Andre
> -----Original Message-----
> From: Andriy Gapon [mailto:avg@FreeBSD.org]
> Sent: Monday, May 27, 2013 03:18 PM
> To: Orit Moskovich
> Cc: Konstantin Belousov; freebsd-hackers@freebsd.org
> Subject: Re: preemptive kernel
>
> on 27/05/2013 10:21 Orit Moskovich said the following:
>> What is actually the difference between deferring a filter routine's work using an ithread given to bus_setup_intr, or using the global taskqueue_swi (implemented using interrupt thread)?
>
> I think you mean taskqueue_fast.
> The difference is only in how much code you need to write. I do not think there is any significant difference in the resulting functionality.
>
>> What do you mean that the functionality is locked under INTR_FILTER?
>
> Please see the code. You have to use option INTR_FILTER to get the behavior I described earlier.
>
>> -----Original Message-----
>> From: Andriy Gapon [mailto:avg@FreeBSD.org]
>> Sent: Monday, May 27, 2013 10:11 AM
>> To: Konstantin Belousov
>> Cc: Orit Moskovich; freebsd-hackers@freebsd.org
>> Subject: Re: preemptive kernel
>>
>> on 27/05/2013 09:34 Konstantin Belousov said the following:
>>> Having both filter and ithread for the same interrupt is apparently
>>> possible but weird. I do not see anything which would prevent
>>> interrupt filter from being executed while the ithread is running.
>>> But again, this is very unusual setup.
>>
>> I wouldn't call it weird, but, yes, it is rare. It's a pretty normal configuration when the filter acts as a filter and the handler acts as a handler (in ithread). In other words, it would be a replacement for a configuration where a filter is used and the filter offloads actual work to non-interrupt context via a e.g. taskqueue.
>> But, hmm, this functionality is probably locked under INTR_FILTER option.
>>
>> --
>> Andriy Gapon
>>
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51A4B991.3070805>