From owner-freebsd-stable@FreeBSD.ORG Wed Oct 24 11:00:36 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 66E21D28 for ; Wed, 24 Oct 2012 11:00:36 +0000 (UTC) (envelope-from h.schmalzbauer@omnilan.de) Received: from host.omnilan.net (s1.omnilan.net [62.245.232.135]) by mx1.freebsd.org (Postfix) with ESMTP id AE6D88FC16 for ; Wed, 24 Oct 2012 11:00:35 +0000 (UTC) Received: from titan.inop.wdn.omnilan.net (titan.inop.wdn.omnilan.net [172.21.3.1]) (authenticated bits=0) by host.omnilan.net (8.13.8/8.13.8) with ESMTP id q9OB1tkR067256 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Oct 2012 13:01:56 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Message-ID: <5087CA50.6090102@omnilan.de> Date: Wed, 24 Oct 2012 13:00:32 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Marius Strobl Subject: Re: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation] References: <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> <20121023211205.GA21019@alchemy.franken.de> In-Reply-To: <20121023211205.GA21019@alchemy.franken.de> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB0B54448E64B4896BB34B3BC" Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 11:00:36 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB0B54448E64B4896BB34B3BC Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable schrieb Marius Strobl am 23.10.2012 23:12 (localtime): > On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote: >> schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): >>> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): >>>> Hello, >>>> >>>> when using igb as module, no packet is received. >>>> If I send out anything, I see the packet with tcpdump, also the swi= tch >>>> learns the MAC address, but nothing comes back in - total silenc, no= >>>> boradcasts, nothing. >>>> If I unload the module and load it again, everything works as expect= ed! >>>> No matter if I load it by 4th loader, or later, I always have tio un= load >>>> first then load it again. >>>> I'ts late here, I'll see tomorrow if things change when compieled in= to >>>> kernel. >> It doesn't matter if igb is loaded as module or compiled into kernel. >> >>>> Maby somebody has an idea what the source of the problem could be. >>>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and= >>>> nics are pci-passthrough. >>> I found one possibly relevant difference: >>> >>> Non-Working state: dev.igb.0.link_irq: 0 >>> Working state: dev.igb.0.link_irq: 2 >> This is only true with msi-x!!! >> If I disable mis-x, the problem itself vanishes. igb just works fine >> from the initial loading (with dev.igb.0.link_irq=3D0!). >> So dev.igb.0.link_irq is only relevant with msi-x. >> But what makes me curious is why it also works mith mis-x enabled afte= r >> the second kldload!?! > Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so > it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge > VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases > observed it didn't even work with a single MSI-X message). So unless > they've changed the ID of the fictitious bridge, igb(4) should fail > to allocate a MSI/MSI-X in the first place. Could you please provide > the output of `pciconf -lv` on that "system"? Hello, here's the output: hostb0@pci0:0:0:0: class=3D0x060000 card=3D0x197615ad chip=3D0x71908= 086 rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '440BX/ZX/DX - 82443BX/ZX/DX Host bridge' class =3D bridge subclass =3D HOST-PCI pcib1@pci0:0:1:0: class=3D0x060400 card=3D0x00000000 chip=3D0x71918= 086 rev=3D0x01 hdr=3D0x01 vendor =3D 'Intel Corporation' device =3D '440BX/ZX/DX - 82443BX/ZX/DX AGP bridge' class =3D bridge subclass =3D PCI-PCI isab0@pci0:0:7:0: class=3D0x060100 card=3D0x197615ad chip=3D0x71108= 086 rev=3D0x08 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82371AB/EB/MB PIIX4 ISA' class =3D bridge subclass =3D PCI-ISA atapci0@pci0:0:7:1: class=3D0x01018e card=3D0x197615ad chip=3D0x71118= 086 rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82371AB/EB/MB PIIX4 IDE' class =3D mass storage subclass =3D ATA intsmb0@pci0:0:7:3: class=3D0x068000 card=3D0x197615ad chip=3D0x71138= 086 rev=3D0x08 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82371AB/EB/MB PIIX4 ACPI' class =3D bridge none0@pci0:0:7:7: class=3D0x088000 card=3D0x074015ad chip=3D0x07401= 5ad rev=3D0x10 hdr=3D0x00 vendor =3D 'VMware' device =3D 'Virtual Machine Communication Interface' class =3D base peripheral vgapci0@pci0:0:15:0: class=3D0x030000 card=3D0x040515ad chip=3D0x04051= 5ad rev=3D0x00 hdr=3D0x00 vendor =3D 'VMware' device =3D 'SVGA II Adapter' class =3D display subclass =3D VGA pcib2@pci0:0:17:0: class=3D0x060401 card=3D0x079015ad chip=3D0x07901= 5ad rev=3D0x02 hdr=3D0x01 vendor =3D 'VMware' device =3D 'PCI bridge' class =3D bridge subclass =3D PCI-PCI pcib3@pci0:0:21:0: class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01= 5ad rev=3D0x01 hdr=3D0x01 vendor =3D 'VMware' device =3D 'PCI Express Root Port' class =3D bridge subclass =3D PCI-PCI pcib4@pci0:0:22:0: class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01= 5ad rev=3D0x01 hdr=3D0x01 vendor =3D 'VMware' device =3D 'PCI Express Root Port' class =3D bridge subclass =3D PCI-PCI pcib5@pci0:0:23:0: class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01= 5ad rev=3D0x01 hdr=3D0x01 vendor =3D 'VMware' device =3D 'PCI Express Root Port' class =3D bridge subclass =3D PCI-PCI pcib6@pci0:0:24:0: class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01= 5ad rev=3D0x01 hdr=3D0x01 vendor =3D 'VMware' device =3D 'PCI Express Root Port' class =3D bridge subclass =3D PCI-PCI mpt0@pci0:3:0:0: class=3D0x010700 card=3D0x197615ad chip=3D0x00541= 000 rev=3D0x01 hdr=3D0x00 vendor =3D 'LSI Logic / Symbios Logic' device =3D 'SAS1068 PCI-X Fusion-MPT SAS' class =3D mass storage subclass =3D SAS mps0@pci0:4:0:0: class=3D0x010700 card=3D0x30201000 chip=3D0x00721= 000 rev=3D0x03 hdr=3D0x00 vendor =3D 'LSI Logic / Symbios Logic' device =3D 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]' class =3D mass storage subclass =3D SAS igb0@pci0:5:0:0: class=3D0x020000 card=3D0xa03c8086 chip=3D0x10c98= 086 rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82576 Gigabit Network Connection' class =3D network subclass =3D ethernet igb1@pci0:6:0:0: class=3D0x020000 card=3D0xa03c8086 chip=3D0x10c98= 086 rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82576 Gigabit Network Connection' class =3D network subclass =3D ethernet 0x079015ad is used for PCI-devices afaik, no matter if they're passthrough or virtual. PCIe devices gets a 0x07a015ad slot, again no matter if they're passtrhough or virtual. I limited the number of functions on each slot, to have control over irq assigning. Usually all devices would be found on pcib3, where the virtual mpt sits. In my setup, each PCIe device gets it's own slot to avoid forced irq sharing. Thanks, -Harry --------------enigB0B54448E64B4896BB34B3BC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlCHylAACgkQLDqVQ9VXb8iwhwCeIHAUhaEBopKS0Ufol+fSuKv+ bywAn2qAIe4sqvbvZlwU78jqbJ2FK2xI =wRxJ -----END PGP SIGNATURE----- --------------enigB0B54448E64B4896BB34B3BC--