Date: Wed, 26 Oct 2011 15:04:43 -0700 From: Jason Wolfe <nitroboost@gmail.com> To: Mike Tancsa <mike@sentex.net> Cc: freebsd-net@freebsd.org, Hooman Fazaeli <fazaeli@sepehrs.com>, Jack Vogel <jfvogel@gmail.com> Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled Message-ID: <CAAAm0r0Ve%2BP4wx%2BuLHm%2ByVMThFkpPMp2CPZOpF6G8AUCkcw4WQ@mail.gmail.com> In-Reply-To: <4EA80818.3030504@sentex.net> References: <CAAAm0r0RXEJo4UiKS=Ui0e5OQTg6sg-xcYf3mYB5%2Bvk8i8557w@mail.gmail.com> <4E8F157A.40702@sentex.net> <CAAAm0r2JH43Rct7UxQK2duH1p43Nepnj5mpb6bXo==DPayhJLg@mail.gmail.com> <4E8F51D4.1060509@sentex.net> <CACqU3MVwLaepFymZJkaVk6p=SpykGhqs=VYFjLh9fP9S=AxDhg@mail.gmail.com> <CAAAm0r1DKvoL9=Ket9up=4%2B5xiCzTTZJK99FhF9jcCA28B0M%2BA@mail.gmail.com> <CAAAm0r3XdsMHZh%2BP_NF-txZasdExzwZ8ymmGQgGhJQds0fOiBQ@mail.gmail.com> <CAAAm0r1iS3z-7CBJ=xYDf%2BJOA1Q2nU0O54Twbyb7FjvgWHjKVw@mail.gmail.com> <4EA7E203.3020306@sepehrs.com> <CAAAm0r3Nr2t8cCetPkFnLQ-3KwqHw_0SpqbtvYPRUkSP=9n8CA@mail.gmail.com> <4EA80818.3030504@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 26, 2011 at 6:16 AM, Mike Tancsa <mike@sentex.net> wrote: > On 10/26/2011 7:33 AM, Jason Wolfe wrote: > > Hooman, > > > > I have run with dev.em.X.flow_control=0, which should have the same > result > > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the > > remaining options would be able to produce the scenario I'm seeing, but > I'm > > open to giving it a try with no options on the interfaces. I've also > added > > ifconfig output to the collection. > > > > > options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC> > > ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol > > options=88<VLAN_MTU,VLAN_HWCSUM> > > > > It's always TX, but these servers push ~12x what they receive, so I'm > > guessing it could happen to either buffer given the right traffic > patterns. > > While looking through commits I also found a patch to add a couple > sysctls > > for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 > > Can you provide some more details as to how traffic flows on these servers > ? Are they going through them, or are they generating the traffic ? I > wonder given a smaller CPU, it would be easier to trigger the condition > somehow. We recently got a Soekris 6501 which has onboard 4 such em nics, > but has just a 1Ghz Atom. > > > em0@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa1000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa1020000, size 16384, > enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em1@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa2000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa2020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em2@pci0:10:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa3000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x4000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa3020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em3@pci0:11:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa4000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x5000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa4020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > > soekris6501# vmstat -i > interrupt total rate > irq4: uart0 4993 0 > cpu0:timer 84947808 71 > irq256: ahci0 59195 0 > irq257: em0:rx 0 1946546 1 > irq258: em0:tx 0 629707 0 > irq259: em0:link 2 0 > Total 87588251 73 > soekris6501# > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > Mike, There is actually a mix of traffic going through the machine and traffic originating from the box itself, though it is ~75% coming from the local disk. Let us know if you have any issues with that Soekris once it's loaded up. Though in my case the hang is still somewhat rare, it works out to roughly once every 4 months, though each box has 2 x 82574L. I've just set aside enough boxes to reproduce about once a week, which gives them time to settle in and not have the quest take up too large a chunk of my time :) Jason
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAAm0r0Ve%2BP4wx%2BuLHm%2ByVMThFkpPMp2CPZOpF6G8AUCkcw4WQ>