From owner-freebsd-drivers@FreeBSD.ORG Sat Apr 16 12:38:15 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 945331065675 for ; Sat, 16 Apr 2011 12:38:15 +0000 (UTC) (envelope-from cy6ergn0m@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 52DDF8FC1C for ; Sat, 16 Apr 2011 12:38:15 +0000 (UTC) Received: by iyj12 with SMTP id 12so3639174iyj.13 for ; Sat, 16 Apr 2011 05:38:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=zFMFYii75hMD6ZNEASB6BReLDuBwo97g3KTPrt2DQ+I=; b=RRZPbaC1Ryq8tLmzFS1a5KuALArgGzaWZPjvyfjevuRntkzvIcHA3BFeYaONKaqOIu kh0Q3JjZOyGNoyu667NZ8ru/eikRu3d+l1y4+KATlbBP8YMV3DLUdRJzTUY3U1Tz6sBH CR/pStVrjn+yfAiFAmUfRBbiOPoFgIimbjkG8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=csqsUXCxWKf+6qabuPYOtuKq2PcvM8baisuNldYsYIdKOHdDgkHg2dZBONFC5r0abU RZoJgV8IDBsuDFd+qAN4ezxRm4lERYdtGuEHk70JX0zIl2dllzHTCsFzbg5xFFnzcfSX tC2uQn1xkjr0jAXHPBNI8CS6KRYnrs7cfbvwk= MIME-Version: 1.0 Received: by 10.231.117.36 with SMTP id o36mr2423623ibq.64.1302956139522; Sat, 16 Apr 2011 05:15:39 -0700 (PDT) Received: by 10.231.200.140 with HTTP; Sat, 16 Apr 2011 05:15:39 -0700 (PDT) Received: by 10.231.200.140 with HTTP; Sat, 16 Apr 2011 05:15:39 -0700 (PDT) In-Reply-To: References: Date: Sat, 16 Apr 2011 16:15:39 +0400 Message-ID: From: cyberGn0m To: freebsd-drivers@freebsd.org X-Mailman-Approved-At: Sun, 17 Apr 2011 04:55:18 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Marvell Yukon 2 driver issue X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Apr 2011 12:38:15 -0000 Hi all Yesterday I've installed freebsd 8.2 to my Sony vaio laptop. Unfortunately the network adapter driver works just a few seconds after connect and then nothing transferred. I have expirience with linux driver development but I've never worked with BSD-systems so The question is: where should I look for related code and debug points? Can you recommend something for debug? - CG From owner-freebsd-drivers@FreeBSD.ORG Mon Apr 18 16:11:56 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA4D51065678 for ; Mon, 18 Apr 2011 16:11:56 +0000 (UTC) (envelope-from cy6ergn0m@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7DFA48FC1F for ; Mon, 18 Apr 2011 16:11:56 +0000 (UTC) Received: by iyj12 with SMTP id 12so5376455iyj.13 for ; Mon, 18 Apr 2011 09:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=RyCUsx6SUjzD+i+efQ4TXjcOkAJJC+Q8LzYZLKEnXBs=; b=hm4Aq1RA7c//u/iTD6YTOAThJHs8jkob1wsMdVlIiMBHynFZf9Pl47DNlG2KtmydZX Y4C6Jx167OzvBW9UZuwk2UFtS2V1HAoss/oohAgVnj9eqMmawJ27S4DkazCW20PLzLx9 DbdeZZmzTnenQrasog/AfLR3hrI2xNomg00o8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=gaJJKJ7rkVV5BnXU1BQ+5wh1P+Q0tA1yAcgK6+IiqpHlTwPSCKQQoy3mzK58BjdTMY jvrHcf41JeJu1ezNpcr+MMfIeOZABJw7LJySPtmSq5AWlxcI5AksK+LsyNt3RqPb1GCx mVSmMDx/CsOqIzew8KDNqCbVi4PFfLZb+VJFw= MIME-Version: 1.0 Received: by 10.231.66.146 with SMTP id n18mr4021039ibi.14.1303143115600; Mon, 18 Apr 2011 09:11:55 -0700 (PDT) Received: by 10.231.200.140 with HTTP; Mon, 18 Apr 2011 09:11:55 -0700 (PDT) In-Reply-To: References: Date: Mon, 18 Apr 2011 20:11:55 +0400 Message-ID: From: cyberGn0m To: freebsd-drivers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Fwd: Marvell Yukon 2 driver issue X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Apr 2011 16:11:56 -0000 Resending original message because of issue in mailing list engine ---------- Forwarded message ---------- From: cyberGn0m Date: 2011/4/16 Subject: Marvell Yukon 2 driver issue To: freebsd-drivers@freebsd.org Hi all Yesterday I've installed freebsd 8.2 to my Sony vaio laptop. Unfortunately the network adapter driver works just a few seconds after connect and then nothing transferred. I have expirience with linux driver development but I've never worked with BSD-systems so The question is: where should I look for related code and debug points? Can you recommend something for debug? - CG From owner-freebsd-drivers@FreeBSD.ORG Wed Apr 20 19:43:34 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8032C106567D; Wed, 20 Apr 2011 19:43:34 +0000 (UTC) (envelope-from bcketchum@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id A11B08FC08; Wed, 20 Apr 2011 19:43:33 +0000 (UTC) Received: by bwz12 with SMTP id 12so1223036bwz.13 for ; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=ZRVxhfGU3G2wz71f9T2SngxojP8PO5OR5bieb+35fto=; b=i3dj1WZl+YEg9iZsVHaKA4jmxJcXnUpUEFdwhkHTxkZ/pomlm9P7483W/WQ1lMMj7G fHat6jdyf2gc+o+mBqI4H9sXI7sPB7BoHADckCz26s3OiXzt4SUZ6QIPdUQeHwZbVEO+ fop2YkfiIhK9dSS2/nhPGOJI43oR0njc5TSvs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=nm2xxB5XgR3cBVfUemsb4MZ40ME80Vn5mxh4u5altUG8G1pgeoiVD0H2VVt/UN6GZ1 u9Kor5/3MAf7fnFXjlwJvfwrr7U8YNxSDQ+DKlInd9lnPqvUqPygpbakGAJvcKGlN8uw dXlBLzrJnRYa4gfKbrtSLfP4+LQepYlsUMRTE= MIME-Version: 1.0 Received: by 10.204.8.141 with SMTP id h13mr379409bkh.64.1303328612368; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) Received: by 10.204.36.77 with HTTP; Wed, 20 Apr 2011 12:43:32 -0700 (PDT) In-Reply-To: <201104151211.49148.jhb@freebsd.org> References: <201104151211.49148.jhb@freebsd.org> Date: Wed, 20 Apr 2011 14:43:32 -0500 Message-ID: From: Bret Ketchum To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-drivers@freebsd.org Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2011 19:43:34 -0000 My response, which included dmesg output along with verbose boot messages was rejected by the mailing list moderator as too large so I'll reply with some of the interesting bits. Here is the MSI related mesages grepped out of the verbose boot messages: ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 0 vector 48 MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 1 message, 64 bit, vector masks MSI supports 1 message, 64 bit, vector masks igb0: attempting to allocate 5 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 256 to local APIC 0 vector 49 msi: routing MSI-X IRQ 257 to local APIC 0 vector 50 msi: routing MSI-X IRQ 258 to local APIC 0 vector 51 msi: routing MSI-X IRQ 259 to local APIC 0 vector 52 msi: routing MSI-X IRQ 260 to local APIC 0 vector 53 igb0: Using MSIX interrupts with 5 vectors igb1: attempting to allocate 5 MSI-X vectors (10 supported) msi: routing MSI-X IRQ 261 to local APIC 0 vector 54 msi: routing MSI-X IRQ 262 to local APIC 0 vector 55 msi: routing MSI-X IRQ 263 to local APIC 0 vector 56 msi: routing MSI-X IRQ 264 to local APIC 0 vector 57 msi: routing MSI-X IRQ 265 to local APIC 0 vector 58 igb1: Using MSIX interrupts with 5 vectors cmlpci0: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 266 to local APIC 0 vector 80 cmlpci1: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 267 to local APIC 0 vector 112 cmlpci2: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 268 to local APIC 0 vector 144 cmlpci3: attempting to allocate 1 MSI vectors (32 supported) msi: routing MSI IRQ 269 to local APIC 0 vector 176 cmlpci4: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 270 to local APIC 0 vector 64 cmlpci5: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 271 to local APIC 0 vector 96 cmlpci6: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 272 to local APIC 0 vector 160 cmlpci7: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 273 to local APIC 0 vector 192 ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 0 vector 59 ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 0 vector 60 ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 0 vector 61 ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 0 vector 62 ioapic0: routing intpin 19 (PCI IRQ 19) to lapic 0 vector 63 ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks MSI supports 2 messages, vector masks cmlpci8: attempting to allocate 1 MSI vectors (2 supported) msi: routing MSI IRQ 274 to local APIC 0 vector 68 MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 4 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks cmlpci9: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 275 to local APIC 0 vector 208 cmlpci10: attempting to allocate 1 MSI vectors (16 supported) msi: routing MSI IRQ 276 to local APIC 0 vector 224 MSI supports 16 messages, 64 bit, vector masks MSI supports 16 messages, 64 bit, vector masks cmlpci11: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 0 vector 67 cmlpci12: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs ioapic2: routing intpin 5 (PCI IRQ 53) to lapic 0 vector 69 ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 0 vector 70 ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 0 vector 71 ioapic0: routing intpin 1 (ISA IRQ 1) to lapic 0 vector 72 ioapic0: routing intpin 3 (ISA IRQ 3) to lapic 2 vector 48 ioapic0: routing intpin 4 (ISA IRQ 4) to lapic 4 vector 48 ioapic0: routing intpin 9 (ISA IRQ 9) to lapic 6 vector 48 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 2 vector 49 ioapic0: routing intpin 16 (PCI IRQ 16) to lapic 4 vector 49 ioapic0: routing intpin 18 (PCI IRQ 18) to lapic 6 vector 49 ioapic0: routing intpin 21 (PCI IRQ 21) to lapic 2 vector 50 ioapic0: routing intpin 23 (PCI IRQ 23) to lapic 4 vector 50 ioapic2: routing intpin 3 (PCI IRQ 51) to lapic 6 vector 50 msi: Assigning MSI-X IRQ 258 to local APIC 2 vector 51 msi: Assigning MSI-X IRQ 260 to local APIC 2 vector 52 msi: Assigning MSI-X IRQ 263 to local APIC 2 vector 53 msi: Assigning MSI-X IRQ 265 to local APIC 4 vector 51 msi: Assigning MSI IRQ 266 to local APIC 6 vector 51 msi: Assigning MSI IRQ 268 to local APIC 2 vector 54 msi: Assigning MSI IRQ 269 to local APIC 4 vector 52 msi: Assigning MSI IRQ 270 to local APIC 6 vector 52 msi: Assigning MSI IRQ 272 to local APIC 2 vector 55 msi: Assigning MSI IRQ 273 to local APIC 4 vector 53 msi: Assigning MSI IRQ 274 to local APIC 6 vector 53 msi: Assigning MSI IRQ 276 to local APIC 2 vector 56 vectors are allocated starting at 48. The first 48 (beginning at 0) are reserved IDT I/O interrupts. This is an amd64 build. The maximum that will be allocated is 192. Note that when requesting a single MSI interrupts apic_alloc_vectors actually looks for a range of free vectors based upon the number of vectors the PCI HBA supports. Yet subsequent calls to apic_alloc_vectors returns vectors within the already allocated range: msi: routing MSI IRQ 270 to local APIC 0 vector 64 cmlpci5: attempting to allocate 1 MSI vectors (16 supported) .... ioapic0: routing intpin 14 (ISA IRQ 14) to lapic 0 vector 65 ioapic0: routing intpin 15 (ISA IRQ 15) to lapic 0 vector 66 Question, why limit the allocation of vectors to a range supported by a PCI device when vectors within that range may be allocated in subsequent requests? In this case enough vectors are allocated that a supported range cannot be found: cmlpci11: attempting to allocate 1 MSI vectors (16 supported) APIC: Couldn't find APIC vectors for 1 IRQs On Fri, Apr 15, 2011 at 11:11 AM, John Baldwin wrote: > On Tuesday, April 12, 2011 9:16:16 am Bret Ketchum wrote: > > I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts > for > > PCI-E HBAs where one or more will be installed in a off-the-shelf amd64 > > pizza box. The driver is using bus_setup_intr() and depending upon the > slots > > the HBAs are install I see log messages from apic_alloc_vectors(), for > > example: > > > > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI > > vectors (16 supported) > > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 > IRQs > > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to > > lapic 0 vector 59 > > > > Using vmstat -ia: > > > > interrupt total rate > > irq37: xxxpci10 74 0 > > > > The problem appears to be that HBA interrupts are not being delivered > to > > the driver. If I swap cards around in slots I can eliminate the message > and: > > > > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0 > > vector 80 > > > > And interrupts appear to be delivered properly. Before I dive in, can > > anyone explain this behavior? > > Hmm, can you capture messages with bootverbose enabled? A full boot -v > dmesg > might be useful as well. > > -- > John Baldwin > From owner-freebsd-drivers@FreeBSD.ORG Thu Apr 21 11:51:34 2011 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 031E81065672 for ; Thu, 21 Apr 2011 11:51:34 +0000 (UTC) (envelope-from bcketchum@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 856AA8FC0C for ; Thu, 21 Apr 2011 11:51:33 +0000 (UTC) Received: by bwz12 with SMTP id 12so1873017bwz.13 for ; Thu, 21 Apr 2011 04:51:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Uj5/OmfaqXwq76iugxqexCNjWaVexlzChicpOXbeUQU=; b=Cic4RGy5dKSKLlOltzVZ86oUYKPvegL72WDBMEtprxz6XEZekAgeTMAGuEAmpK4udp vXhJHJXFL40AJVgGc84Zo48+md0aCybvWFz4uduUhn4ANZIzUxR7+efprCPmiEadVw7+ F2UgI8ym2WU1XG0xU4VW+VWsXwpdHx4RA/Rok= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=NfPoaAc4L+18/gUgOKu5A3+4M/pBDoEYRiyC3oCGWcrvL9RUlEkPm3vlnhr1HVgCeC h23bFN8aZhlXIVxhzqVYb9qqQJfR64wj1xhts8OABy27swJk3/XOq36Ru1al1YiDuVJl WzFznAczR4a5fDUW4hgRRddHm2TT8iEuyZuKo= MIME-Version: 1.0 Received: by 10.204.8.141 with SMTP id h13mr1105272bkh.64.1303386692272; Thu, 21 Apr 2011 04:51:32 -0700 (PDT) Received: by 10.204.36.77 with HTTP; Thu, 21 Apr 2011 04:51:32 -0700 (PDT) In-Reply-To: References: Date: Thu, 21 Apr 2011 06:51:32 -0500 Message-ID: From: Bret Ketchum To: freebsd-drivers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: MSI interrupts. X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Apr 2011 11:51:34 -0000 As an experiment I made align equal to count in apic_alloc_vectors to mimic the way MSIX vectors are assigned and all my issue went away.. On Tue, Apr 12, 2011 at 8:16 AM, Bret Ketchum wrote: > > I've a roll-your-own driver for FreeBSD 8.x that uses MSI interrupts > for PCI-E HBAs where one or more will be installed in a off-the-shelf amd64 > pizza box. The driver is using bus_setup_intr() and depending upon the slots > the HBAs are install I see log messages from apic_alloc_vectors(), for > example: > > Apr 12 06:44:15 mfsbsd kernel: xxxpci10: attempting to allocate 1 MSI > vectors (16 supported) > Apr 12 06:44:15 mfsbsd kernel: APIC: Couldn't find APIC vectors for 1 IRQs > Apr 12 06:44:15 mfsbsd kernel: ioapic1: routing intpin 13 (PCI IRQ 37) to > lapic 0 vector 59 > > Using vmstat -ia: > > interrupt total rate > irq37: xxxpci10 74 0 > > The problem appears to be that HBA interrupts are not being delivered > to the driver. If I swap cards around in slots I can eliminate the message > and: > > Apr 12 06:44:15 mfsbsd kernel: msi: routing MSI IRQ 266 to local APIC 0 > vector 80 > > And interrupts appear to be delivered properly. Before I dive in, can > anyone explain this behavior? > > Thanks in advance. > > Dr. >