Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 24 Oct 2012 13:00:32 +0200
From:      Harald Schmalzbauer <h.schmalzbauer@omnilan.de>
To:        Marius Strobl <marius@alchemy.franken.de>
Cc:        FreeBSD Stable <freebsd-stable@freebsd.org>
Subject:   Re: msi-x enabled igb works only if module loaded twice [Was: Re: kldload if_igb twice needed to bring nic into operation]
Message-ID:  <5087CA50.6090102@omnilan.de>
In-Reply-To: <20121023211205.GA21019@alchemy.franken.de>
References:  <50859F7F.2080308@omnilan.de> <5085A323.8030501@omnilan.de> <50866839.5090204@omnilan.de> <20121023211205.GA21019@alchemy.franken.de>

next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigB0B54448E64B4896BB34B3BC
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

 schrieb Marius Strobl am 23.10.2012 23:12 (localtime):
> On Tue, Oct 23, 2012 at 11:49:45AM +0200, Harald Schmalzbauer wrote:
>>  schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime):
>>>  schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime):
>>>>  Hello,
>>>>
>>>> when using igb as module, no packet is received.
>>>> If I send out anything, I see the packet with tcpdump,  also the swi=
tch
>>>> learns the MAC address, but nothing comes back in - total silenc, no=

>>>> boradcasts, nothing.
>>>> If I unload the module and load it again, everything works as expect=
ed!
>>>> No matter if I load it by 4th loader, or later, I always have tio un=
load
>>>> first then load it again.
>>>> I'ts late here, I'll see tomorrow if things change when compieled in=
to
>>>> kernel.
>> It doesn't matter if igb is loaded as module or compiled into kernel.
>>
>>>> Maby somebody has an idea what the source of the problem could be.
>>>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and=

>>>> nics are pci-passthrough.
>>> I found one possibly relevant difference:
>>>
>>> Non-Working state:    dev.igb.0.link_irq: 0
>>> Working state:           dev.igb.0.link_irq: 2
>> This is only true with msi-x!!!
>> If I disable mis-x, the problem itself vanishes. igb just works fine
>> from the initial loading (with dev.igb.0.link_irq=3D0!).
>> So dev.igb.0.link_irq is only relevant with msi-x.
>> But what makes me curious is why it also works mith mis-x enabled afte=
r
>> the second kldload!?!
> Hrm, this is strange; in r231621 (MFC'ed to stable/9 in r232092, so
> it's part of 9.1-RC2) I've blacklisted the fictitious PCI-PCI bridge
> VMware uses in case of PCI pass-through for MSI/MSI-X (as in the cases
> observed it didn't even work with a single MSI-X message). So unless
> they've changed the ID of the fictitious bridge, igb(4) should fail
> to allocate a MSI/MSI-X in the first place. Could you please provide
> the output of `pciconf -lv` on that "system"?

Hello,

here's the output:
hostb0@pci0:0:0:0:      class=3D0x060000 card=3D0x197615ad chip=3D0x71908=
086
rev=3D0x01 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '440BX/ZX/DX - 82443BX/ZX/DX Host bridge'
    class      =3D bridge
    subclass   =3D HOST-PCI
pcib1@pci0:0:1:0:       class=3D0x060400 card=3D0x00000000 chip=3D0x71918=
086
rev=3D0x01 hdr=3D0x01
    vendor     =3D 'Intel Corporation'
    device     =3D '440BX/ZX/DX - 82443BX/ZX/DX AGP bridge'
    class      =3D bridge
    subclass   =3D PCI-PCI
isab0@pci0:0:7:0:       class=3D0x060100 card=3D0x197615ad chip=3D0x71108=
086
rev=3D0x08 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '82371AB/EB/MB PIIX4 ISA'
    class      =3D bridge
    subclass   =3D PCI-ISA
atapci0@pci0:0:7:1:     class=3D0x01018e card=3D0x197615ad chip=3D0x71118=
086
rev=3D0x01 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '82371AB/EB/MB PIIX4 IDE'
    class      =3D mass storage
    subclass   =3D ATA
intsmb0@pci0:0:7:3:     class=3D0x068000 card=3D0x197615ad chip=3D0x71138=
086
rev=3D0x08 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '82371AB/EB/MB PIIX4 ACPI'
    class      =3D bridge
none0@pci0:0:7:7:       class=3D0x088000 card=3D0x074015ad chip=3D0x07401=
5ad
rev=3D0x10 hdr=3D0x00
    vendor     =3D 'VMware'
    device     =3D 'Virtual Machine Communication Interface'
    class      =3D base peripheral
vgapci0@pci0:0:15:0:    class=3D0x030000 card=3D0x040515ad chip=3D0x04051=
5ad
rev=3D0x00 hdr=3D0x00
    vendor     =3D 'VMware'
    device     =3D 'SVGA II Adapter'
    class      =3D display
    subclass   =3D VGA
pcib2@pci0:0:17:0:      class=3D0x060401 card=3D0x079015ad chip=3D0x07901=
5ad
rev=3D0x02 hdr=3D0x01
    vendor     =3D 'VMware'
    device     =3D 'PCI bridge'
    class      =3D bridge
    subclass   =3D PCI-PCI
pcib3@pci0:0:21:0:      class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01=
5ad
rev=3D0x01 hdr=3D0x01
    vendor     =3D 'VMware'
    device     =3D 'PCI Express Root Port'
    class      =3D bridge
    subclass   =3D PCI-PCI
pcib4@pci0:0:22:0:      class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01=
5ad
rev=3D0x01 hdr=3D0x01
    vendor     =3D 'VMware'
    device     =3D 'PCI Express Root Port'
    class      =3D bridge
    subclass   =3D PCI-PCI
pcib5@pci0:0:23:0:      class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01=
5ad
rev=3D0x01 hdr=3D0x01
    vendor     =3D 'VMware'
    device     =3D 'PCI Express Root Port'
    class      =3D bridge
    subclass   =3D PCI-PCI
pcib6@pci0:0:24:0:      class=3D0x060400 card=3D0x07a015ad chip=3D0x07a01=
5ad
rev=3D0x01 hdr=3D0x01
    vendor     =3D 'VMware'
    device     =3D 'PCI Express Root Port'
    class      =3D bridge
    subclass   =3D PCI-PCI
mpt0@pci0:3:0:0:        class=3D0x010700 card=3D0x197615ad chip=3D0x00541=
000
rev=3D0x01 hdr=3D0x00
    vendor     =3D 'LSI Logic / Symbios Logic'
    device     =3D 'SAS1068 PCI-X Fusion-MPT SAS'
    class      =3D mass storage
    subclass   =3D SAS
mps0@pci0:4:0:0:        class=3D0x010700 card=3D0x30201000 chip=3D0x00721=
000
rev=3D0x03 hdr=3D0x00
    vendor     =3D 'LSI Logic / Symbios Logic'
    device     =3D 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
    class      =3D mass storage
    subclass   =3D SAS
igb0@pci0:5:0:0:        class=3D0x020000 card=3D0xa03c8086 chip=3D0x10c98=
086
rev=3D0x01 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '82576 Gigabit Network Connection'
    class      =3D network
    subclass   =3D ethernet
igb1@pci0:6:0:0:        class=3D0x020000 card=3D0xa03c8086 chip=3D0x10c98=
086
rev=3D0x01 hdr=3D0x00
    vendor     =3D 'Intel Corporation'
    device     =3D '82576 Gigabit Network Connection'
    class      =3D network
    subclass   =3D ethernet

0x079015ad is used for PCI-devices afaik, no matter if they're
passthrough or virtual. PCIe devices gets a 0x07a015ad slot, again no
matter if they're passtrhough or virtual.
I limited the number of functions on each slot, to have control over irq
assigning. Usually all devices would be found on pcib3, where the
virtual mpt sits. In my setup, each PCIe device gets it's own slot to
avoid forced irq sharing.

Thanks,

-Harry



--------------enigB0B54448E64B4896BB34B3BC
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (FreeBSD)

iEYEARECAAYFAlCHylAACgkQLDqVQ9VXb8iwhwCeIHAUhaEBopKS0Ufol+fSuKv+
bywAn2qAIe4sqvbvZlwU78jqbJ2FK2xI
=wRxJ
-----END PGP SIGNATURE-----

--------------enigB0B54448E64B4896BB34B3BC--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5087CA50.6090102>