Date: Mon, 20 May 2013 06:28:01 GMT From: Jason Shi <legumen@foxmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/178782: [ixgbe] 82599EB SFP does not work with passthrough under KVM. Message-ID: <201305200628.r4K6S14J077860@oldred.FreeBSD.org> Resent-Message-ID: <201305200630.r4K6U0MO017313@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 178782
>Category: kern
>Synopsis: [ixgbe] 82599EB SFP does not work with passthrough under KVM.
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon May 20 06:30:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator: Jason Shi
>Release: 9.1-PRERELEASE
>Organization:
Arraynetworks
>Environment:
Host OS:
OpenSUSE 12.2
Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux
QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard
Guest OS:
FreeBSD freebsd9.1 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
The 10G Intel 82599EB SFP NIC could not work with passthrough on KVM.
I did some tests with different Hypervisors and Guest OS, which make me think it's a FreeBSD problem.
1. OpenXen + FreeBSD9.1 guest os. -- [OK]
2. OpenXen + Linux guest os (e.g. Fedora 12). -- [OK]
3. KVM + Linux guest os (e.g. Fedora 12). -- [OK]
4. KVM + FreeBSD9.1 guest os. -- [Failed]
>How-To-Repeat:
1.Make the pci device of 82599SFP assignale. From lspci on OpenSUSE:
0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: QLogic, Corp. Device 00e4
Flags: fast devsel, IRQ 25
Memory at f8fe0000 (64-bit, prefetchable) [size=128K]
I/O ports at cc00 [size=32]
Memory at f8fdc000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: pci-stub
2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file
<domain type='kvm'>
<name>FreeBSD9.1</name>
<uuid>4d56613e-400c-f37d-2a9a-e0332d81ca63</uuid>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<clock offset='utc'>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='default'/>
<source file='/home/vm/FreeBSD9.1/FreeBSD9.1.raw'/>
<target dev='hdc'/>
</disk>
<interface type='bridge'>
<source bridge='br0'/>
<mac address='52:54:00:0d:92:6f'/>
<model type='rtl8139'/>
</interface>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
</source>
</hostdev>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
</devices>
</domain>
3.The device could be probed and attached by FreeBSD. From dmesg on FreeBSD:
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xc020-0xc03f mem 0xfe000000-0xfe01ffff,0xfe020000-0xfe023fff irq 11 at device 4.0 on pci0
ix0: Using MSIX interrupts with 2 vectors
ix0: Ethernet address: 00:0c:bd:05:7d:4a
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
But no interrupt could be found. From "vmstat -i":
interrupt total rate
irq1: atkbd0 143 0
irq4: uart0 177 1
irq15: ata1 1593 10
cpu0:timer 7140 46
Total 9053 58
And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping). From "ifconfig":
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:0c:bd:05:7d:4a
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201305200628.r4K6S14J077860>