Date: Wed, 17 Jun 2009 11:47:26 -0400 From: John Baldwin <jhb@freebsd.org> To: Jeff Roberson <jeff@freebsd.org> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r187880 - in head/sys: amd64/amd64 amd64/include i386/i386 i386/include Message-ID: <200906171147.26968.jhb@freebsd.org> In-Reply-To: <200901290922.n0T9MuJ4047351@svn.freebsd.org> References: <200901290922.n0T9MuJ4047351@svn.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday 29 January 2009 4:22:56 am Jeff Roberson wrote: > Author: jeff > Date: Thu Jan 29 09:22:56 2009 > New Revision: 187880 > URL: http://svn.freebsd.org/changeset/base/187880 > > Log: > - Allocate apic vectors on a per-cpu basis. This allows us to allocate > more irqs as we have more cpus. This is principally useful on systems > with msi devices which may want many irqs per-cpu. > > Discussed with: jhb > Sponsored by: Nokia It turns out that this completely breaks devices using multiple MSI (not MSI-X) messages. In MSI-X each interrupt has a separate address and data register, so each one can have an independent IDT vector and CPU. This is not the case with MSI. With MSI you only have a single address and data register set. If you have multiple messages, the OS is required to leave the bottom N (log_2 (count)) bits set to zero and the individual message is encoded by via those bottom N bits. This requires that multiple MSI messages be treated as a single group that all go to the same CPU. Furthermore, the block of IDT vectors for the group has to be aligned on an N bit boundary. This was the purpose of the apic_alloc_vectors() function you removed. I think that part of the change will need to be reverted. msi_alloc() will probably need to preallocate vectors again rather than deferring them to msi_enable_intr(). Also, we can move MSI groups around. I had changes in a p4 tree to basically fail to bind messages that weren't the first message in such a group. Those can be revived for this purpose. I think it is currently accidentally working for mav@'s AHCI tests because he does a bus_setup_intr() of all the interrupts in sequence during attach. That causes the IDT vectors to happen to all be contiguous on CPU 0 during boot. It would fail if the driver was kldloaded. Also, he must currently just be very lucky that the first IDT vector he allocates is properly aligned. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200906171147.26968.jhb>