From owner-svn-src-all@freebsd.org Sat Feb 2 21:17:24 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72C3A14CFF14; Sat, 2 Feb 2019 21:17:24 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EAAEB6A04F; Sat, 2 Feb 2019 21:17:23 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: by mail-pg1-f179.google.com with SMTP id v28so4578029pgk.10; Sat, 02 Feb 2019 13:17:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=l3e8BTip52CWEQ/8GXygkVdw8/Ufvsg1NRNm0nj2SpQ=; b=o+GwH94PdMxAvQT7sDFpPfs7xmaio7ZzEkuiQSjSrghu0Az+YvjCSiOkwwcjTmOyxj VpIjspAeygL6P8F1CcYPSf4IyHruX/hRtvA9W8ZfLFefdZdK73EBIF7UreaYfFlwUVET cUETblYhf2adfzpG6Xs8oAz1Z9cf0NzAeahHjnNwfmMyawcXPCEbtIpWjjb7je3GwCuE V1YxuLLK07iLH9iqTqCWyXa1gx/0osVCJNMfaFxHYklGdRO3Tz0kzhYg49+qvS2fYdod /+LQytG3An+wOvc4hUY03X/K4j3WAexRNKeD3tLg8E+OEQzeANRZVCAYd7r/WzeKGiDu +Sdw== X-Gm-Message-State: AJcUukcQW6tV6hpkQBC2AGrjcif0vghWAXCDWYgWWbSnM25FJTSsD45U zC63sIx0Nxufd81xm49lFHVP+KGEGY0DM4l7sb1P6Y3g X-Google-Smtp-Source: ALg8bN4U0c3gJILpJi11yl/EXLGXS/7Afi2Q6t6u98fLvuG8Kmgk+HTBWuPONj+GEcw+FtQvQfTPmAfElMGetpPPPIs= X-Received: by 2002:a62:28c9:: with SMTP id o192mr45792715pfo.57.1549142236634; Sat, 02 Feb 2019 13:17:16 -0800 (PST) MIME-Version: 1.0 References: <201901220111.x0M1BHS1025063@repo.freebsd.org> <2c0e0179-63e7-1e2f-ba7f-6a373c927d88@yuripv.net> In-Reply-To: From: Patrick Kelsey Date: Sat, 2 Feb 2019 16:17:03 -0500 Message-ID: Subject: Re: svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net To: Yuri Pankov Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org X-Rspamd-Queue-Id: EAAEB6A04F X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.86 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.86)[-0.857,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2019 21:17:24 -0000 On Sat, Feb 2, 2019 at 3:29 PM Patrick Kelsey wrote: > > > On Sat, Feb 2, 2019 at 9:28 AM Yuri Pankov wrote: > >> Patrick Kelsey wrote: >> > Author: pkelsey >> > Date: Tue Jan 22 01:11:17 2019 >> > New Revision: 343291 >> > URL: https://svnweb.freebsd.org/changeset/base/343291 >> > >> > Log: >> > onvert vmx(4) to being an iflib driver. >> > >> > Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add >> > iflib_dma_alloc_align() to the iflib API. >> > >> > Performance is generally better with the tunable/sysctl >> > dev.vmx..iflib.tx_abdicate=1. >> > >> > Reviewed by: shurd >> > MFC after: 1 week >> > Relnotes: yes >> > Sponsored by: RG Nets >> > Differential Revision: https://reviews.freebsd.org/D18761 >> >> This breaks vmx interfaces for me in ESXi 6.7 (output below). The >> review mentions setting hw.pci.honor_msi_blacklist="0" and it helps >> indeed -- worth mentioning in UPDATING? >> >> vmx0: port 0x3000-0x300f mem >> 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe900000-0xfe901fff at >> device 0.0 on pci3 >> vmx0: Using 512 tx descriptors and 256 rx descriptors >> vmx0: msix_init qsets capped at 8 >> vmx0: intr CPUs: 20 queue msgs: 24 admincnt: 1 >> vmx0: Using 8 rx queues 8 tx queues >> vmx0: attempting to allocate 9 MSI-X vectors (25 supported) >> vmx0: failed to allocate 9 MSI-X vectors, err: 6 - using MSI >> vmx0: attempting to allocate 1 MSI vectors (1 supported) >> msi: routing MSI IRQ 25 to local APIC 6 vector 48 >> vmx0: using IRQ 25 for MSI >> vmx0: Using an MSI interrupt >> msi: Assigning MSI IRQ 25 to local APIC 25 vector 48 >> msi: Assigning MSI IRQ 25 to local APIC 24 vector 48 >> vmx0: bpf attached >> vmx0: Ethernet address: 00:00:00:00:00:33 >> vmx0: netmap queues/slots: TX 1/512, RX 1/512 >> vmx0: device enable command failed! >> vmx0: link state changed to UP >> vmx0: device enable command failed! >> >> > Setting hw.pci.honor_msi_blacklist="0" should only be necessary if you > want to operate with more than one queue. If > hw.pci.honor_msi_blacklist="0" is not set, then MSI-X will not be > available, and MSI will be used, which reduces the number of queues that > can be configured for use to 1. This case should work correctly. > > I am able to reproduce the behavior you described above on ESXi 6.7 using > the latest snapshot release (based on r343598). The error that appears in > the ESXi logs will be similar to: > > 2019-02-02T15:14:02.986Z| vcpu-1| I125: VMXNET3 user: failed to activate > 'Ethernet0', status: 0xbad0001 > > which vaguely means 'the device did not like something about the > configuration it was given'. I will see if I can determine the root > cause. Given that enabling MSI-X seems to work around the problem, and > based on other issues I encountered during development, I currently suspect > there is a problem with the interrupt index that is being configured for > the transmit queue in the device configuration structure when using MSI. > > Indeed, the interrupt index for the tx queue in MSI mode was the problem. This is now fixed in r343688 ( https://svnweb.freebsd.org/changeset/base/343688). Thanks for reporting the issue! -Patrick