Date: Mon, 08 Jun 2009 12:15:24 +0300 From: Alexander Motin <mav@FreeBSD.org> To: FreeBSD-Current <freebsd-current@freebsd.org>, freebsd-arch@freebsd.org Subject: Multiple MSI on SMP, misrouting or misunderstanding? Message-ID: <4A2CD6AC.80407@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------010407070706090009080901 Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Hi. While experimenting with using multiple MSIs support on AHCI controller I have got the problem. When system boots as UP - everything is fine, driver allocates all available 16 MSIs and works. But when system booted as SMP, interrupts begin to behave strange: I didn't receive expected AHCI IRQs, but instead receive IRQ1 interrupts of atkbd0, while I have no PS/2 keyboard/mouse attached. As I have found, problem appears due to IRQ rebalancing between CPUs. As I have got, MSI requires that all vectors from the same group to be allocated sequentially, but IRQ rebalancing breaks correct order, that happed during initial allocation. I was quite surprised by this issue. If multiple MSI vectors of the same device have to be allocated sequentially and bound to the same CPU, then they will be unable to give any SMP scalability benefits. Am I right, or there is some special technique expected to be used to somehow distribute grouped MSI vectors between CPUs which we don't have? I have made small patch that denies rebalancing for grouped MSIs, to make them work at least somehow. It works fine for me, but I am not sure that it is the best solution. -- Alexander Motin --------------010407070706090009080901 Content-Type: text/plain; name="msi.c.reassign.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="msi.c.reassign.patch" --- msi.c.prev 2009-06-08 11:30:13.000000000 +0300 +++ msi.c 2009-06-08 11:30:06.000000000 +0300 @@ -210,6 +210,8 @@ msi_assign_cpu(struct intsrc *isrc, u_in old_id = msi->msi_cpu; if (old_vector && old_id == apic_id) return; + if (old_vector && !msi->msi_msix && msi->msi_first->msi_count > 1) + return; /* Allocate IDT vector on this cpu. */ vector = apic_alloc_vector(apic_id, msi->msi_irq); if (vector == 0) --------------010407070706090009080901--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A2CD6AC.80407>