Date: Sat, 13 Feb 2021 02:49:55 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 253473] Possible bug in iflib, observed with vmxnet3 driver on ESXI servers Message-ID: <bug-253473-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253473 Bug ID: 253473 Summary: Possible bug in iflib, observed with vmxnet3 driver on ESXI servers Product: Base System Version: 12.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: shrikanth07@gmail.com Attachment #222401 text/plain mime type: Created attachment 222401 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D222401&action= =3Dedit Boot log from Juniper VSRX box Possible bug with iflib code observed with VMXNET3 driver. On a 12.1 RELEASE FreeBSD VM running on ESXi server, doing a "ifconfig vmx0 media autoselect" renders the interface unusable. You can have a ping going to the interface = from outside of VM, it stops when you do a media autoselect using ifconfig.=20 My real objective was to track the way iflib processes media change so the ifconfig command wasn't a specific need for me. The original instance of th= is problem was on a Juniper VSRX box running Junos running FreeBSD stable/12 (12.1) underneath. I actually reached out to bryanv@freebsd.org on the prob= lem before I verified it on pristine FreeBSD stable/12 (Junos customizes FreeBSD drivers to work with it's network stack).=20 I guess when iflib layer is processing iflib_media_change() it calls iflib_init_locked() while the driver is running and does not bother to check the state while it attempts some of the initialization again which leaves t= he interface in an inconsistent state.=20 root@freebsd:~ #ifconfig vmx0 media autoselect vmx0: device enable command failed! To recover you have to flap the interface, ifconfig vmx0 down and then up recovers the interface. original observation of the issue and the email I sent to Bryan Venteicher -------------------------------------------------------------------------- Hello Bryan, I need to clarify a certain behavior with vmxnet3 driver and iflib on FreeB= SD stable/12. The background on this is during vmxnet3 driver initialization probe/attach happens fine and interface initialization is all good. A subsequent reconfi= gure on the interface does the following call flow iflib_if_init =E2=86=92 iflib_stop =E2=86=92 vmxnet3_stop =E2=86=92 iflib_i= nit_locked=20 What that does underneath is a proper VMXNET3_CMD_DISABLE, VMXNET3_CMD_RESET and eventually VMXNET3_CMD_ENABLE. No problem there. But on a subsequent ioctl (SIOCSIFMEDIA) it does a VMXNET3_CMD_ENABLE again= on an already enabled device which fails and renders the interface in a non working state. The call flow seems to be=20 Ifmedia_ioctl =E2=86=92 ifm->ifm_change =E2=86=92 iflib_media_change =E2=86= =92 vmxnet3_media_change. vmxnet3_media_change return 0=E2=86=92 iflib_init_locked =E2=86=92 vmxnet3_= init =E2=86=92vmxnet3_enable_device The problem seems iflib_init_locked() does *not* care to do a iflib_stop() = and vmxnet3_media_change() being a dummy function just returns 0 indicating suc= cess to iflib_media_change so it proceeds to call iflib_init_locked() without calling iflib_stop().=20 Herein lies the problem and my query, iflib_init_locked ends up calling vmxnet3_init() =E2=86=92 vmxnet3_enable_device() which means doing VMXNET3_= CMD_ENABLE a second time, the subsequent one resulting in a failure as indicated by the = log =E2=80=9Cdevice enable command failed!=E2=80=9D. (Refer the logs below capt= ured from a boot). The workaround to recover the interface is to flap it i.e ifconfig fxp0 dow= n / up to recover. Another experiment was to do a VMXNET3_CMD_DISABLE in vmxnet3_media_change = to =E2=80=9Csupport=E2=80=9D the subsequent VMXNET3_CMD_ENABLE from iflib_init= _locked. I tried looking around for documentation/datasheet to confirm this behavior but sad= ly no direct hits. Possibly look at the Linux side of things, not sure but reaching out to you for help. Note that the environment is Junos (Juniper OS) based on FreeBSD stable/12 = for one of our products which is an instance running on ESXI servers. Below are the logs from the boot of a VSRX device. The ones marked with ###= are some device_printfs added to capture the call flow along with IFMEDIA_DEBUG enabled.=20 My question is should there be a sequence of VMXNET3_CMD_DISABLE / VMXNET3_CMD_RESET before calling vmxnet3_media_change? Vmxnet3 logs from boot (complete boot log is attached) ------------------------------------------------------ vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem 0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at device 0.0 on pci3 vmx0: Using 512 TX descriptors and 512 RX descriptors vmx0: Using 4 RX queues 4 TX queues vmx0: failed to allocate 5 MSI-X vectors, err: 6 vmx0: Using an MSI interrupt vmx0: LRO is disabled Adding entry for Ethernet autoselect ifmedia_set: target Ethernet autoselect ifmedia_set: setting to Ethernet autoselect Reconfiguring vmx0 to fxp0 ... vmx0: ### iflib_if_init: called vmx0: ### iflib_if_init_locked: called vmx0: ### iflib_stop: called vmx0: ### vmxnet3_stop: called! vmx0: ### iflib_init_locked: called vmx0: ### vmxnet3_init: called! vmx0: ### vmxnet3_reinit_shared_data: mtu: 1514. vmx0: ### vmxnet3_reinit_queues: called! vmx0: ### vmxnet3_txinit: called! vmx0: ### vmxnet3_rxinit: called! vmx0: ### vmxnet3_enable_device: called! vmx0: ### vmxnet3_reinit_rxfilters: called! vmx0: ### iflib_if_ioctl: called: command: 2152753457 vmx0: ### vmxnet3_multi_set: called! vmx0: ### iflib_if_ioctl: called: command: 3226495287 ifmedia_ioctl: switching fxp0 to Ethernet autoselect vmx0: ### iflib_media_change: called vmx0: ### vmxnet3_media_change: called! vmx0: ### iflib_init_locked: called vmx0: ### vmxnet3_init: called! vmx0: ### vmxnet3_reinit_shared_data: mtu: 1514. vmx0: ### vmxnet3_reinit_queues: called! vmx0: ### vmxnet3_txinit: called! vmx0: ### vmxnet3_rxinit: called! vmx0: ### vmxnet3_reinit_rxfilters: called! vmx0: device enable command failed! vmx0: ### iflib_if_ioctl: called: command: 3227543864 vmx0: ### vmxnet3_update_admin_status: called! vmx0: ### vmxnet3_media_status: called! vmx0: ### iflib_if_ioctl: called: command: 2152753424 vmx0: ### iflib_if_ioctl: called: command: 2152753457 vmx0: ### vmxnet3_multi_set: called! --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-253473-227>