Date: Fri, 7 Oct 2011 14:14:52 -0700 From: Jason Wolfe <nitroboost@gmail.com> To: Mike Tancsa <mike@sentex.net>, freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> In-Reply-To: <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Oct 7, 2011 at 12:55 PM, Arnaud Lacombe <lacombar@gmail.com> wrote: > Hi, > > On Fri, Oct 7, 2011 at 3:24 PM, Mike Tancsa <mike@sentex.net> wrote: > > When you disable MSI-X, you mean via hw.pci.enable_msix=0 across the > > board, or you disable multi-queue for the NIC, so it uses just one > > interrupt, rather than separate ones for xmit and recv ? > > > As noticed it was the prior, all were disabled via hw.em.enable_msix=0 > em(4)'s multiqueue is misleading. By default, with MSI-X enabled, > before AFAIK, April 2010 it used 2 (RX+TX) queue + 1, ie. 5 MSI-X > vectors[0]. After April 2010, it uses 1 * (RX+TX) queue + 1, ie. 3 > MSI-X vectors. There is no logic for the driver to use 1 vector with > MSI-X enabled. > > As a side note, the only gain of EM_MULTIQUEUE, now, is to allow the > driver to use the buf_ring(9) lockless queue API, compared to the > locked ifq. Today, em(4) should waste about 16k of memory for when > !EM_MULTIQUEUE. This is the memory, 4096 * sizeof(void *), allocated > for the buf_ring(9) structure which is not used in the !EM_MULTIQUEUE > case. > > > Also, what is the purpose of > > hw.pci.do_power_nodriver=3 vs 0 (3 means put absolutely everything > > in D3 state.) > 0 is the default state and means no 'power management' is done. My stripped down kernels don't have usb support so '3' just reduces the consumption by not powering those devices. If you load up the modules power will be supplied and the devices function. > > > > net.link.ifqmaxlen 1024 vs 50 (does anything else need to be adjusted of > > this value is increased?) > I haven't seen any evidence that there are other supporting adjustments for this, this was actually something I stumbled across while researching the issue. > > > He might as well try to enable EM_MULTIQUEUE. > > > hw.em.rxd="2048" > > hw.em.txd="2048" > > > As it starts to be well known here, I am not a fan of bumping a limit > to hide a bug. So I'd rather lower this to 512 or 256, and hope it > triggers the issue more often, so that it could be diagnosticed and > fixed for good. > > - Arnaud > > [0]: actually it depends on a field in the chip NVM, which can be up > to 4 (0 based accounting, this would translate in 5 vectors), but > happened to be 2 (3 vector) in 82574 I've got access to. Last time I > checked, this setting could not be seen with the standard NVM dump > sysctl, which limit the output's size. On those chip, the > pre-April-2010 code would falls back on MSI even if 3 were available. > > > Have you tried leaving these two at the default on 7.2.3 ? > > if_em.h implies 1024 for each. > > > > ---Mike > Bumping rx/tx descriptors to 2048 was actually for performance reasons and not to try to get around the issue. I did some fairly in depth testing and found under heavy load it performed the best with those settings. As mentioned on the other thread I'll re enable MSI-X on a few servers here and collect uptime and the kernel msgbuf in addition. I'll bump the descriptors down to 512 to try and increase our chances and compile the driver with EM_MULTIQUEUE also. Jason
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA>