From owner-freebsd-drivers@FreeBSD.ORG Mon Apr 25 18:08:32 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1D631065670 for ; Mon, 25 Apr 2011 18:08:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 77BD58FC15 for ; Mon, 25 Apr 2011 18:08:32 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 2792046B1A; Mon, 25 Apr 2011 14:08:32 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B7FF38A027; Mon, 25 Apr 2011 14:08:31 -0400 (EDT) From: John Baldwin To: freebsd-drivers@freebsd.org Date: Mon, 25 Apr 2011 14:08:30 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201104251408.30816.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 25 Apr 2011 14:08:31 -0400 (EDT) Cc: Bret Ketchum Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2011 18:08:32 -0000 On Thursday, April 21, 2011 7:51:32 am Bret Ketchum wrote: > As an experiment I made align equal to count in apic_alloc_vectors to > mimic the way MSIX vectors are assigned and all my issue went away.. Hmm, it shouldn't be working then. Rather, I think you are making the system violate the spec. :) When a PCI device uses MSI (not MSI-X) with multiple vectors, it is required that the low N bits of the 'data' register all be zeros, and that the device will set the low N bits to the individual message number. Can you do 'show lapic' at a DDB prompt (you can just use Ctrl-Atl-Break or the like to drop into the debugger) and reply with the output? It's a bit odd that you don't have an aligned block of 16 vectors available. > On Tue, Apr 12, 2011 at 8:16 AM, Bret Ketchum wrote: > > > > > I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts > > for PCI-E HBAs where one or more will be installed in a off-the-shelf amd64 > > pizza box. The driver is using bus_setup_intr() and depending upon the slots > > the HBAs are install I see log messages from apic_alloc_vectors(), for > > example: > > > > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI > > vectors (16 supported) > > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 IRQs > > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to > > lapic 0 vector 59 > > > > Using vmstat -ia: > > > > interrupt total rate > > irq37: xxxpci10 74 0 > > > > The problem appears to be that HBA interrupts are not being delivered > > to the driver. If I swap cards around in slots I can eliminate the message > > and: > > > > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0 > > vector 80 > > > > And interrupts appear to be delivered properly. Before I dive in, can > > anyone explain this behavior? > > > > Thanks in advance. > > > > Dr. > > > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > -- John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Mon Apr 25 18:58:37 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 65B6E1065676 for ; Mon, 25 Apr 2011 18:58:37 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 34DA18FC08 for ; Mon, 25 Apr 2011 18:58:37 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id A8FE546B98; Mon, 25 Apr 2011 14:58:36 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 319408A01B; Mon, 25 Apr 2011 14:58:36 -0400 (EDT) From: John Baldwin To: Bret Ketchum Date: Mon, 25 Apr 2011 14:55:57 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: <201104151211.49148.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201104251455.57383.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 25 Apr 2011 14:58:36 -0400 (EDT) Cc: freebsd-drivers@freebsd.org Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2011 18:58:37 -0000 On Wednesday, April 20, 2011 3:43:32 pm Bret Ketchum wrote: > My response, which included dmesg output along with verbose boot > messages was rejected by the mailing list moderator as too large so I'll > reply with some of the interesting bits. Here is the MSI related mesages > grepped out of the verbose boot messages: > > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 1 message, 64 bit, vector masks > MSI supports 1 message, 64 bit, vector masks > igb0: attempting to allocate 5 MSI-X vectors (10 supported) > msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 > msi: routing MSI-X IRQ 257 to local APIC 0 vector 50 > msi: routing MSI-X IRQ 258 to local APIC 0 vector 51 > msi: routing MSI-X IRQ 259 to local APIC 0 vector 52 > msi: routing MSI-X IRQ 260 to local APIC 0 vector 53 > igb0: Using MSIX interrupts with 5 vectors > igb1: attempting to allocate 5 MSI-X vectors (10 supported) > msi: routing MSI-X IRQ 261 to local APIC 0 vector 54 > msi: routing MSI-X IRQ 262 to local APIC 0 vector 55 > msi: routing MSI-X IRQ 263 to local APIC 0 vector 56 > msi: routing MSI-X IRQ 264 to local APIC 0 vector 57 > msi: routing MSI-X IRQ 265 to local APIC 0 vector 58 > igb1: Using MSIX interrupts with 5 vectors > cmlpci0: attempting to allocate 1 MSI vectors (32 supported) > msi: routing MSI IRQ 266 to local APIC 0 vector 80 > cmlpci1: attempting to allocate 1 MSI vectors (32 supported) > msi: routing MSI IRQ 267 to local APIC 0 vector 112 > cmlpci2: attempting to allocate 1 MSI vectors (32 supported) > msi: routing MSI IRQ 268 to local APIC 0 vector 144 > cmlpci3: attempting to allocate 1 MSI vectors (32 supported) > msi: routing MSI IRQ 269 to local APIC 0 vector 176 > cmlpci4: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 270 to local APIC 0 vector 64 > cmlpci5: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 271 to local APIC 0 vector 96 > cmlpci6: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 272 to local APIC 0 vector 160 > cmlpci7: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 273 to local APIC 0 vector 192 > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59 > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60 > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61 > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62 > ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63 > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > MSI supports 2 messages, vector masks > cmlpci8: attempting to allocate 1 MSI vectors (2 supported) > msi: routing MSI IRQ 274 to local APIC 0 vector 68 > MSI supports 4 messages, 64 bit, vector masks > MSI supports 4 messages, 64 bit, vector masks > MSI supports 4 messages, 64 bit, vector masks > MSI supports 4 messages, 64 bit, vector masks > MSI supports 16 messages, 64 bit, vector masks > MSI supports 16 messages, 64 bit, vector masks > cmlpci9: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 275 to local APIC 0 vector 208 > cmlpci10: attempting to allocate 1 MSI vectors (16 supported) > msi: routing MSI IRQ 276 to local APIC 0 vector 224 > MSI supports 16 messages, 64 bit, vector masks > MSI supports 16 messages, 64 bit, vector masks > cmlpci11: attempting to allocate 1 MSI vectors (16 supported) > APIC: Couldn't find APIC vectors for 1 IRQs > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67 > cmlpci12: attempting to allocate 1 MSI vectors (16 supported) > APIC: Couldn't find APIC vectors for 1 IRQs > ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69 > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70 > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71 > ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72 > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48 > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48 > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48 > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49 > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49 > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49 > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50 > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50 > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50 > msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51 > msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52 > msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53 > msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51 > msi: Assigning MSI IRQ 266 to local APIC 6 vector 51 > msi: Assigning MSI IRQ 268 to local APIC 2 vector 54 > msi: Assigning MSI IRQ 269 to local APIC 4 vector 52 > msi: Assigning MSI IRQ 270 to local APIC 6 vector 52 > msi: Assigning MSI IRQ 272 to local APIC 2 vector 55 > msi: Assigning MSI IRQ 273 to local APIC 4 vector 53 > msi: Assigning MSI IRQ 274 to local APIC 6 vector 53 > msi: Assigning MSI IRQ 276 to local APIC 2 vector 56 > > vectors are allocated starting at 48. The first 48 (beginning at 0) are > reserved IDT I/O interrupts. This is an amd64 build. The maximum that will > be allocated is 192. Note that when requesting a single MSI interrupts > apic_alloc_vectors actually looks for a range of free vectors based upon the > number of vectors the PCI HBA supports. Yet subsequent calls to > apic_alloc_vectors returns vectors within the already allocated range: > > msi: routing MSI IRQ 270 to local APIC 0 vector 64 > cmlpci5: attempting to allocate 1 MSI vectors (16 supported) > .... > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 > > Question, why limit the allocation of vectors to a range supported by a > PCI device when vectors within that range may be allocated in subsequent > requests? Ah, I just replied to another e-mail on the list with the reason. Hmm. After rechecking this, the spec doesn't require the behavior of fully aligning based on the device's size, but IIRC, we encountered some hardware that did behave that way. Perhaps some bge(4) devices (I think the hardware actually fixes the low N bits of the message data register as 0, so if you wrote a value with any low N bits non-zero they were lost). Perhaps we can use a quirk to work around that behavior now though instead. -- John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Wed Apr 27 13:45:14 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51B60106566B; Wed, 27 Apr 2011 13:45:14 +0000 (UTC) (envelope-from bcketchum@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 69F168FC18; Wed, 27 Apr 2011 13:45:12 +0000 (UTC) Received: by bwz12 with SMTP id 12so1984901bwz.13 for ; Wed, 27 Apr 2011 06:45:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=lEqAPZYqWoJ36gfXZ7fTz3e4KaupoBIr3EYp9K9omc4=; b=VPStKbQoEU0zmHXwikN9PzvHjkhkkP85Sw18tdhUl+Hz5kxCTHS9GBH0PAGoThqCOs Ql7rOX9vdDN7ZT0rlr7YgY0fQGvNuyD448dLyp0wKWSAb8kpc0vsH5dHi31s5ciDptFE N/YVZwU+GAfre2ZvUx8Min0uggqR3vmzoQ2vI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=RLIc4f0SpwajekephmDYW90DXmMiJWjw88iEGlLywuCUzpU/Rvstnbjd5X26dXPNeq GhVTcaRllZNKXGjXXpAedEbqHi1cLzZahCUOgoC83Jw7gWHxNWU69ACJzocFQ0WzEyxn NL88JLN8notZHvtqCmKe5w9cYMrU3vc+ZyKnY= MIME-Version: 1.0 Received: by 10.204.189.74 with SMTP id dd10mr1803586bkb.174.1303911912151; Wed, 27 Apr 2011 06:45:12 -0700 (PDT) Received: by 10.204.10.11 with HTTP; Wed, 27 Apr 2011 06:45:12 -0700 (PDT) In-Reply-To: <201104251455.57383.jhb@freebsd.org> References: <201104151211.49148.jhb@freebsd.org> <201104251455.57383.jhb@freebsd.org> Date: Wed, 27 Apr 2011 08:45:12 -0500 Message-ID: From: Bret Ketchum To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-drivers@freebsd.org Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2011 13:45:14 -0000 Do you still want to see the output of 'show lapic'? What quirk do you have in mind? I'd be glad to do the testing. On Mon, Apr 25, 2011 at 1:55 PM, John Baldwin wrote: > On Wednesday, April 20, 2011 3:43:32 pm Bret Ketchum wrote: > > My response, which included dmesg output along with verbose boot > > messages was rejected by the mailing list moderator as too large so I'll > > reply with some of the interesting bits. Here is the MSI related mesages > > grepped out of the verbose boot messages: > > > > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 1 message, 64 bit, vector masks > > MSI supports 1 message, 64 bit, vector masks > > igb0: attempting to allocate 5 MSI-X vectors (10 supported) > > msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 > > msi: routing MSI-X IRQ 257 to local APIC 0 vector 50 > > msi: routing MSI-X IRQ 258 to local APIC 0 vector 51 > > msi: routing MSI-X IRQ 259 to local APIC 0 vector 52 > > msi: routing MSI-X IRQ 260 to local APIC 0 vector 53 > > igb0: Using MSIX interrupts with 5 vectors > > igb1: attempting to allocate 5 MSI-X vectors (10 supported) > > msi: routing MSI-X IRQ 261 to local APIC 0 vector 54 > > msi: routing MSI-X IRQ 262 to local APIC 0 vector 55 > > msi: routing MSI-X IRQ 263 to local APIC 0 vector 56 > > msi: routing MSI-X IRQ 264 to local APIC 0 vector 57 > > msi: routing MSI-X IRQ 265 to local APIC 0 vector 58 > > igb1: Using MSIX interrupts with 5 vectors > > cmlpci0: attempting to allocate 1 MSI vectors (32 supported) > > msi: routing MSI IRQ 266 to local APIC 0 vector 80 > > cmlpci1: attempting to allocate 1 MSI vectors (32 supported) > > msi: routing MSI IRQ 267 to local APIC 0 vector 112 > > cmlpci2: attempting to allocate 1 MSI vectors (32 supported) > > msi: routing MSI IRQ 268 to local APIC 0 vector 144 > > cmlpci3: attempting to allocate 1 MSI vectors (32 supported) > > msi: routing MSI IRQ 269 to local APIC 0 vector 176 > > cmlpci4: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 270 to local APIC 0 vector 64 > > cmlpci5: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 271 to local APIC 0 vector 96 > > cmlpci6: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 272 to local APIC 0 vector 160 > > cmlpci7: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 273 to local APIC 0 vector 192 > > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59 > > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60 > > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61 > > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62 > > ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63 > > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 > > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > MSI supports 2 messages, vector masks > > cmlpci8: attempting to allocate 1 MSI vectors (2 supported) > > msi: routing MSI IRQ 274 to local APIC 0 vector 68 > > MSI supports 4 messages, 64 bit, vector masks > > MSI supports 4 messages, 64 bit, vector masks > > MSI supports 4 messages, 64 bit, vector masks > > MSI supports 4 messages, 64 bit, vector masks > > MSI supports 16 messages, 64 bit, vector masks > > MSI supports 16 messages, 64 bit, vector masks > > cmlpci9: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 275 to local APIC 0 vector 208 > > cmlpci10: attempting to allocate 1 MSI vectors (16 supported) > > msi: routing MSI IRQ 276 to local APIC 0 vector 224 > > MSI supports 16 messages, 64 bit, vector masks > > MSI supports 16 messages, 64 bit, vector masks > > cmlpci11: attempting to allocate 1 MSI vectors (16 supported) > > APIC: Couldn't find APIC vectors for 1 IRQs > > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67 > > cmlpci12: attempting to allocate 1 MSI vectors (16 supported) > > APIC: Couldn't find APIC vectors for 1 IRQs > > ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69 > > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70 > > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71 > > ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72 > > ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48 > > ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48 > > ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48 > > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49 > > ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49 > > ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49 > > ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50 > > ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50 > > ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50 > > msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51 > > msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52 > > msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53 > > msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51 > > msi: Assigning MSI IRQ 266 to local APIC 6 vector 51 > > msi: Assigning MSI IRQ 268 to local APIC 2 vector 54 > > msi: Assigning MSI IRQ 269 to local APIC 4 vector 52 > > msi: Assigning MSI IRQ 270 to local APIC 6 vector 52 > > msi: Assigning MSI IRQ 272 to local APIC 2 vector 55 > > msi: Assigning MSI IRQ 273 to local APIC 4 vector 53 > > msi: Assigning MSI IRQ 274 to local APIC 6 vector 53 > > msi: Assigning MSI IRQ 276 to local APIC 2 vector 56 > > > > vectors are allocated starting at 48. The first 48 (beginning at 0) > are > > reserved IDT I/O interrupts. This is an amd64 build. The maximum that > will > > be allocated is 192. Note that when requesting a single MSI interrupts > > apic_alloc_vectors actually looks for a range of free vectors based upon > the > > number of vectors the PCI HBA supports. Yet subsequent calls to > > apic_alloc_vectors returns vectors within the already allocated range: > > > > msi: routing MSI IRQ 270 to local APIC 0 vector 64 > > cmlpci5: attempting to allocate 1 MSI vectors (16 supported) > > .... > > ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 > > ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 > > > > Question, why limit the allocation of vectors to a range supported by > a > > PCI device when vectors within that range may be allocated in subsequent > > requests? > > Ah, I just replied to another e-mail on the list with the reason. > > Hmm. After rechecking this, the spec doesn't require the behavior of fully > aligning based on the device's size, but IIRC, we encountered some hardware > that did behave that way. Perhaps some bge(4) devices (I think the > hardware > actually fixes the low N bits of the message data register as 0, so if you > wrote a value with any low N bits non-zero they were lost). Perhaps we can > use a quirk to work around that behavior now though instead. > > -- > John Baldwin > From owner-freebsd-drivers@FreeBSD.ORG Wed Apr 27 15:45:31 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DB0310656D4 for ; Wed, 27 Apr 2011 15:45:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 559818FC23 for ; Wed, 27 Apr 2011 15:45:31 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 0B3C246B9D; Wed, 27 Apr 2011 11:45:30 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7FD568A027; Wed, 27 Apr 2011 11:45:29 -0400 (EDT) From: John Baldwin To: Bret Ketchum Date: Wed, 27 Apr 2011 11:33:33 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: <201104251455.57383.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201104271133.34148.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Wed, 27 Apr 2011 11:45:29 -0400 (EDT) Cc: freebsd-drivers@freebsd.org Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Apr 2011 15:45:31 -0000 On Wednesday, April 27, 2011 9:45:12 am Bret Ketchum wrote: > Do you still want to see the output of 'show lapic'? No, this output was enough for me to understand what was happening. > What quirk do you have in mind? I'd be glad to do the testing. Hmm, actually, I'm digging around in my old e-mail and it may be that I added this requirement based on misunderstanding someone else's e-mail. I'll let you know what I find out. -- John Baldwin