From owner-freebsd-drivers@FreeBSD.ORG Mon Apr 25 18:08:32 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1D631065670 for ; Mon, 25 Apr 2011 18:08:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 77BD58FC15 for ; Mon, 25 Apr 2011 18:08:32 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 2792046B1A; Mon, 25 Apr 2011 14:08:32 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B7FF38A027; Mon, 25 Apr 2011 14:08:31 -0400 (EDT) From: John Baldwin To: freebsd-drivers@freebsd.org Date: Mon, 25 Apr 2011 14:08:30 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110325; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201104251408.30816.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 25 Apr 2011 14:08:31 -0400 (EDT) Cc: Bret Ketchum Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Apr 2011 18:08:32 -0000 On Thursday, April 21, 2011 7:51:32 am Bret Ketchum wrote: > As an experiment I made align equal to count in apic_alloc_vectors to > mimic the way MSIX vectors are assigned and all my issue went away.. Hmm, it shouldn't be working then. Rather, I think you are making the system violate the spec. :) When a PCI device uses MSI (not MSI-X) with multiple vectors, it is required that the low N bits of the 'data' register all be zeros, and that the device will set the low N bits to the individual message number. Can you do 'show lapic' at a DDB prompt (you can just use Ctrl-Atl-Break or the like to drop into the debugger) and reply with the output? It's a bit odd that you don't have an aligned block of 16 vectors available. > On Tue, Apr 12, 2011 at 8:16 AM, Bret Ketchum wrote: > > > > > I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts > > for PCI-E HBAs where one or more will be installed in a off-the-shelf amd64 > > pizza box. The driver is using bus_setup_intr() and depending upon the slots > > the HBAs are install I see log messages from apic_alloc_vectors(), for > > example: > > > > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI > > vectors (16 supported) > > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 IRQs > > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to > > lapic 0 vector 59 > > > > Using vmstat -ia: > > > > interrupt total rate > > irq37: xxxpci10 74 0 > > > > The problem appears to be that HBA interrupts are not being delivered > > to the driver. If I swap cards around in slots I can eliminate the message > > and: > > > > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0 > > vector 80 > > > > And interrupts appear to be delivered properly. Before I dive in, can > > anyone explain this behavior? > > > > Thanks in advance. > > > > Dr. > > > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > -- John Baldwin